fix(server): properly unwrap signature blob per RFC 8332

[Luwdo]

Problem

The server fails to verify public key signatures from clients that put ssh-rsa in the userauth packet's algorithm field but embed rsa-sha2-256 (or rsa-sha2-512) as the algorithm inside the signature blob. This is permitted by RFC 8332 Section 3:

When the "rsa-sha2-256" or "rsa-sha2-512" signature algorithms are used in the "publickey" authentication method, the "algorithm name" in the signature blob MUST be "rsa-sha2-256" or "rsa-sha2-512" (respectively), while the "public key algorithm name" field in the overall userauth request MAY be "ssh-rsa", "rsa-sha2-256", or "rsa-sha2-512".

The current signature header stripping logic only removes the wrapper when the signature blob's embedded algorithm exactly matches keyAlgo (the packet-level algorithm). When they differ, the header bytes are left intact in the signature buffer, causing verify() to fail on corrupted input.

Example client exhibiting this behavior: TablePlus (which uses libssh internally). It sends ssh-rsa as the packet algorithm but signs with rsa-sha2-256, embedding that in the signature blob. This works against OpenSSH servers (which handle the mismatch internally) but fails against ssh2-based servers.

Fix

Instead of comparing the signature blob's algorithm against keyAlgo, the fix reads the actual algorithm identifier from the signature blob and strips the header whenever a known SSH algorithm is detected (ssh-rsa, rsa-sha2-256, rsa-sha2-512, ssh-dss, ssh-ed25519, ecdsa-sha2-*).

Additionally, when hashAlgo was not derived from the packet-level algorithm (because the packet said ssh-rsa), the fix now derives it from the signature blob's embedded algorithm:

• rsa-sha2-256 → sha256
• rsa-sha2-512 → sha512

This ensures ctx.hashAlgo is correctly populated for the server's verify() call.

Applied to both publickey and hostbased authentication handlers.

Changes

• lib/protocol/handlers.misc.js: Replaced the signature header stripping logic for both publickey and hostbased auth methods
• test/test-userauth-rfc8332.js: Added regression tests for RSA auth paths through the new unwrapping logic

Testing

• All existing tests pass (npm test)
• New test-userauth-rfc8332.js verifies both rsa-sha2-256 and plain ssh-rsa auth succeed through the new logic
• Verified with TablePlus desktop client connecting to an ssh2-based server (previously failed, now succeeds)

References

• RFC 8332 Section 3 — Use of RSA Keys with SHA-256 and SHA-512
• Related real-world clients affected: TablePlus (libssh), potentially other libssh-based tools