alkssh/docs/research/poc-2-vault-keys.md

POC-2 findings — client-side at-rest key handling (vault-backed storage)

Status: Complete (phase 0). POC source: /workspace/alkssh-poc2 (standalone crate, disposable — never merges). All hypotheses in pocs.md §POC-2 were confirmed; the doc below records the evidence, the inner-format comparison, the Signer-seam dismissal (now with runtime + source evidence instead of the earlier reasoning-only withdrawal), several API facts that correct the phase 0 assumptions, and the OQ seeds this POC produced.

Setup exercised

• russh 0.64.1 (crates.io, ring backend) + alkvault 0.1.0 (crates.io; its pinned surface: EncryptedData, VaultServiceHandle, CURRENT_KEY_VERSION), ssh-key 0.7.0-rc.11 exact-pinned and re-exported through russh::keys::ssh_key (identical tree to POC-1's — the two POCs exercised the same dependency graph).
• Server side: POC-1's loopback shape verbatim (fingerprint registry fed from ssh-keygen -lf output, stub IdentityProvider, into_stream echo server, no wait() before into_stream).
• The adapter-shaped seam under test (vaultkey module in the POC):
  • KeySource { blob: Option<EncryptedData> } — the assembly-layer storage stand-in (None = absent material);
  • VaultKeyStore::load(&KeySource) -> Result<DialKey, LoadError> — decrypt-at-dial: vault.decrypt(blob) → plaintext wrapped in Zeroizing<String> at the seam → one-time russh::keys::decode_secret_key(&plain, None) → fingerprint computed in the same step. Every error shape maps to a distinct fail-closed variant (Absent / Locked / Vault(_) / Decode).
• A vault-backed custom russh::Signer (VaultSigner, decrypt-per- sign) for the seam verdict; a second standalone vault (own mnemonic) for the wrong-mnemonic row of the failure matrix.

Validated: all checks pass (39/39)

1. At-rest shape — OpenSSH-encoded key inside alkvault EncryptedData confirmed; the &str-only API constraint is decisive

alkvault's encrypt/decrypt surface is &str in, String out (VaultServiceHandle::encrypt(plaintext: &str, key_version: u32) -> Result<EncryptedData, _>, decrypt(&EncryptedData) -> Result<String, _>). Any non-text inner format needs a base64 (or hex) layer on both ends. The four inner-format candidates, baked and dialed:

inner format	plaintext	blob ciphertext	key reconstruction at use	verdict
OpenSSH PEM (unencrypted)	387 B	540 B	decode_secret_key(plain, None) — one call	winner
raw ed25519 seed (base64)	32 B	80 B	Ed25519Keypair::from_seed(&seed) -> PrivateKey (From exists)	works, smallest blob; drops comment/public metadata (pub is re-derived from seed — fine)
ssh-key self-encrypted PEM (passphrase, bcrypt KDF)	449 B	616 B	two decryptions: vault, then decode_secret_key(pem, Some(pass))	works but pointless — the passphrase is itself a stored secret, so the model just moves the same problem one level down; double-wrapping buys nothing
PKCS#8 PEM (unencrypted)	246 B	252 B	encode_pkcs8_pem (russh) at rest; decode_secret_key(plain, None) at use	works; loses the native comment/model round-trip; adds nothing over OpenSSH PEM

(All four rows produced the correct fingerprint through the full decrypt→decode→dial pipeline; the seed and PKCS#8 rows were checked to reproduce the same registered fingerprint as the PEM inner, proving inner and outer representation choices are orthogonal to the key identity the registry sees.)

The OpenSSH PEM inner wins on honest grounds: ASCII natively (no encode layer), self-describing (algorithm + public key + comment embedded), vanilla-OpenSSH-tool-compatible at every boundary, and exactly one decode call from russh. The seed variant is a legitimate size optimization the ADR can note (80 B vs 540 B blobs) — it couples alkssh's storage to a fixed key shape though, whereas "a PEM file's bytes" works for every algorithm ssh-key can decode. This mirrors the alkhttp ADR-014 analogy exactly as hypothesized in pocs.md.

At-rest hygiene, verified end to end:

• EncryptedData { key_version: 2, salt, iv, data } serializes to stable JSON (serde), roundtrips, and decrypts to the byte-identical original PEM. The salt field is 32 B (44 b64 chars), the IV 12 B (16 b64 chars) — pure wire-format confirmation against the type's documented shape.
• AES-256-GCM opacity: the ciphertext's data field contains no recognizable substrings of the key (checked OPENSSH, PRIVATE, ssh-ed25519, and a 30-byte interior slice of the PEM — none found in ciphertext or the serialized JSON).
• Fresh IV + fresh salt per encrypt: re-encrypting the identical plaintext produces entirely different blobs (no determinism trap for naive dedup code to stumble into).

2. Dial lifecycle — decrypt-at-dial, zeroization bounds, lock/unlock mid-session

• The full dial through the adapter works: load → authenticate_publickey → Success, the server's auth_publickey fires post-signature-verification and resolves the fingerprint against the assembly-registered identity, and the fingerprint matches the ssh-keygen-sourced registry value exactly (POC-1's parity, re-proven through the vault pipeline).
• The vault is needed only at dial time. vault.lock() mid-session: the already-established session kept streaming (echo roundtrip after lock), while a new load on the same blob failed. This is the decode-once model behaving as designed: the decoded PrivateKey is the dial's lifetime, not a vault-coupled lease.
• Re-unlock with the same mnemonic → decrypt works again, same fingerprint. Lock/unlock cycles are side-effect-free on the key.
• Zeroization bounds (source-verified in ssh-key 0.7.0-rc.11):
  • Ed25519PrivateKey zeroizes on Drop (private/ed25519.rs — impl Drop { self.0.zeroize() }), so the decoded key's secret bytes are cleaned up by ssh-key itself when the PrivateKey (or clones of it — the type is Clone with zeroizing storage) drops;
  • alkvault zeroizes its own key material (EncryptionKey #[zeroize(drop)], mnemonic phrase via Zeroizing, lock() purges seed + cache);
  • the gap: VaultServiceHandle::decrypt returns a plain String. The decrypted plaintext key sits in unfenced heap memory from the vault's return until the adapter takes ownership. The adapter shape proven here wraps it in Zeroizing<String> at the seam (owning the move), so the adapter-bound bytes zeroize on drop — but the pre-wrap window is the vault's API shape to close. → OQ seed below.

3. Failure matrix — every failure mode observed; all fail closed

disturbed condition	observed	adapter mapping
vault locked	VaultServiceError::VaultLocked	LoadError::Locked — no dial
blob absent at source (assembly layer has no record)	no vault call needed	LoadError::Absent — no dial
wrong mnemonic (unrelated seed)	VaultServiceError::Encryption("decryption error: aead::Error") — a decryption failure, distinguishable from Locked	no dial
tampered ciphertext (one flipped byte)	GCM integrity failure (same decrypt-error family)	no dial
key_version = 0 (reserved/legacy)	VaultServiceError::InvalidPath — the vault refuses before touching crypto	no dial
key_version = 7 (future, undecodable)	derivation-path error	no dial
healthy rotation v2→v3	rotate() → new blob stamped key_version: 3, decrypt yields the identical PEM, decoded fingerprint identical	dial succeeds

The wrong-mnemonic/tampered cases land in VaultServiceError::Encryption as strings — the error taxonomy at the adapter is ours to shape (distinct variants), but the root causes are distinguishable today via the variant + message. The InvalidPath behavior for version < 2 is deliberate (v1 is the TS_PBKDF2 legacy the vault cannot decrypt; rotation re-encryption is the migration path, per alkvault's own docs) — consistent with the ADR-021-style version-indexed derivation the alkssh ADR will cite for rotation.

4. Signer-seam dismissal — confirmed, now with runtime and source evidence

pocs.md expected the answer "no — recorded so the ADR can dismiss it with evidence". Evidence assembled:

• The trait exists but has no provided implementation: russh 0.64.1 re-exports Signer at the crate root (russh::Signer, auth::Signer shape: auth_sign(&AgentIdentity, hash_alg, to_sign)), and Handle::authenticate_publickey_with(user, ssh_key::PublicKey, hash_alg, &mut S) accepts it. The doc comment claims "Currently, this crate only provides an implementation for an SSH agent" — that is stale: there is no impl Signer for AgentClient (or anything else) anywhere in the 0.64.1 tree. A custom signer must be built by hand.
• The wire contract is non-obvious and hand-rolled: the flow is probe (has_sig=0, write_auth_request) → server USERAUTH_PK_OK → client builds session_id || userauth-prefix (client_make_to_sign) → Reply::SignRequest carries the full buffer → the signer returns to_sign || string{signature} — the original buffer with the signature length-prefixed as an SSH string — and the client splices buffer[i0..] as the packet payload. (First attempt appended the raw signature without the outer length prefix: the server silently misparsed and declined — USERAUTH_FAILURE — without the auth_publickey hook ever firing. Signature-verification failures happen inside russh before any handler hook, which is also a clean fail-closed property worth noting for the mapping docs.)
• Runtime proof that the seam works anyway: VaultSigner (decrypt-per-sign against the vault) authenticated successfully end-to-end via authenticate_publickey_with — the splice convention replicated, PrivateKey: signature::Signer::try_sign + .encoded() (i.e. sig → SSH wire blob) then length-prefixed into the returned buffer.
• Locked-vault fail-closed on the Signer path, too: with the vault locked, the sign-time decrypt fails and the auth call itself errors (Err from authenticate_publickey_with) — no half-auth, no fallback.
• Verdict: dismissed for the vault path. The vault is in-process: decrypted key bytes necessarily exist client-side at either dial time or sign time — the Signer seam buys no custody improvement, and decrypt-per-sign merely multiplies GCM+parse work per signature while keeping more plaintext copies alive (per-signature) rather than fewer. The seam's genuine home is external signer processes (ssh-agent), which the family treats as out of scope. The ADR can record: one-time decode_secret_key at dial; Signer not taken.

5. Host keys ride the same model — server bootstrap + vanilla compatibility

• The server's host key was generated in-POC, vault-encrypted, and decrypted-at-construction into Config::keys through the same VaultKeyStore::load the client dial uses — one model, both sides. Vanilla OpenSSH authenticated against that server, the known_hosts fingerprint (written by the vanilla client) matched the vault host key's fingerprint exactly, and the russh client's check_server_key observed the same fingerprint.
• The client's pin — the expected host fingerprint — is public config, not vault material (check_server_key policy + the known-hosts entry); only the server's private host key is held secret. The ADR's "assembly configures Config::keys" note can now cite the exact decrypt-at-boot path.
• Vanilla-tool compatibility of the plaintext: the decrypted client-key PEM written to a 0600 temp file was accepted by vanilla ssh -i directly (auth succeeded; Permission denied never appeared — exit 255 only because the POC server intentionally serves no exec). The at-rest bytes are, at rest and in motion, plain OpenSSH key material.
• Locked vault at server bootstrap → LoadError::Locked → no server is constructed — the fail-closed posture extends to the host-key boot path.

Hypothesis verdicts (pocs.md §POC-2)

• "OpenSSH private-key encoding as the inner format; alkvault encrypt/decrypt (EncryptedData) as the outer wrapper" — confirmed, and strengthened by the 4-way comparison table (the seed-only variant as a size optimization; the ssh-key self-encrypted variant recorded as a double-wrapping antipattern).
• "decrypt-at-dial → one-time decode_secret_key → zeroized handling for the dial's lifetime" — confirmed (ssh-key zeroizes key bytes on drop; adapter owns plaintext zeroization at the seam; vault in-memory state self-purges on lock).
• "what happens locked / absent / rotation" — confirmed as fail-closed with distinct, mappable error shapes; healthy rotation (v2→v3) preserves key identity.
• "Signer-shaped seam still worth anything — expected: no" — dismissed with evidence (no provided impl; hand-rolled splice convention; in-process vault gains nothing; runtime proof the seam works anyway).
• "Server and client host-key material riding the same model" — confirmed end to end, including vanilla-tool compatibility and locked-at-bootstrap fail-closed.

API facts for the ADRs (all verified against crates.io sources)

• alkvault 0.1.0: VaultServiceHandle is a sync, cloneable handle (interior Arc<RwLock>; short-held locks — no tokio coupling); unlock_new(word_count) returns the phrase as Zeroizing<String>; encrypt(&str, u32) / decrypt(&EncryptedData) -> Result<String> / rotate(&EncryptedData, to_version); current version 2, versions derive on per-version paths; decrypt errors distinguish VaultLocked / Encryption(…) / InvalidPath.
• russh 0.64.1: keys::decode_secret_key(&str, Option<&str>) handles OpenSSH (incl. passphrase/bcrypt via the enabled ssh-key/encryption feature), PuTTY PPK, PKCS#5 and PKCS#8 (+ the BEGIN PRIVATE KEY / BEGIN ENCRYPTED PRIVATE KEY PEM cases); keys::encode_pkcs8_pem[_encrypted] exist for writers (PKCS#8 support lives in russh, not in ssh-key — ssh-key rc11 has no pkcs8 feature; the research note about ssh-key-own-PKCS#8 does not apply at this pin). PrivateKey: signature::Signer<Signature>, Signature::encoded() via signature::SignatureEncoding; russh's own signing (helpers::sign_with_hash_alg) is exactly try_sign(keypair, msg)?.encoded() — a vault signer can mirror it, but helpers (and its EncodedExt) is crate-private; the public path is russh::keys::signature::Signer + russh::keys::ssh_encoding::Encode.
• Server-side auth pipeline: a failed/malformed signature produces USERAUTH_FAILURE without calling auth_publickey — handler hooks only fire on cryptographically valid signatures (POC-1 established the "after verification" half; POC-2 adds "invalid sigs never reach the handler at all").
• Client-side event routing: check_server_key events are client- handler facts, so anything asserting on them must read the client's event surface, not the server session's (POC logistics note for future bake-offs).

OQ seeds carried forward

1. alkvault decrypt returns plain String — a zeroization gap at the seam (the vault's own key material is zeroized; the decrypted plaintext is not until the adapter wraps it). Candidate fixes: alkvault returns Zeroizing<String>, or the alkssh ADR documents the adapter-side Zeroizing obligation as the family convention. POC-2's adapter shape does the latter.
2. Key-record schema: alkssh's vault-backed key record should treat EncryptedData as opaque and never interpret key_version itself (the vault owns version semantics; InvalidPath for unknown versions is the fail-closed behavior). Rotation (at-rest version bump, key constant) is a storage-side operation — proven to preserve key identity.
3. One vault, many roles? The POC held client dial keys and the server host key in one vault (one mnemonic). Whether the assembly layer should run separate vault instances per role (client vs server custody) is an assembly policy question, not an alkssh design constraint — recorded so phase 1 doesn't bake it in.
4. Never persist decrypted plaintext: the vanilla-tool check wrote a decrypted key to a 0600 temp file (then zeroized and deleted it) — that is a test-only demonstration of the byte format. Adapters do not spill plaintext to disk; the vanilla-compatible PEM is handed to processes only via the assembly layer if ever.

Reference run

Full output of a clean run at findings time: 39 checks, 5 groups (at-rest shape 11, dial lifecycle 8, failure matrix 10, Signer seam 3, host key + vanilla 7) — all PASS. overall=true, exit 0.

References

• pocs.md §POC-2 (plan), auth-bridge.md §The client side (design — updated with a pointer to these verdicts), russh.md (client::Handle authenticate methods — updated with the Signer-verdict pointer)
• POC-1: poc-1-auth-bridge.md + /workspace/alkssh-poc1 (server loopback shape reused)
• russh 0.64.1, alkvault 0.1.0, ssh-key 0.7.0-rc.11 (crates.io, ring backend), OpenSSH 9.6p1 (vanilla tooling)