alkssh/docs/architecture/decisions/005-at-rest-key-custody.md

status, last_updated

status	last_updated
reviewed	2026-10-28

ADR-005: At-rest key custody (vault-wrapped OpenSSH-PEM, decrypt-at-use)

Status

Accepted (2026-10-26). Evidence-backed (POC-2, 39/39, including the 4-way inner-format bake and the failure matrix); amendable only via a new ADR.

Context

On the client direction alkssh holds secret material (dial keys); on the server direction it holds the host key. Family rules in force:

• Secrets come from alkvault-backed sources, never env vars, never plaintext at rest (family ADR-002 decision 5; family-wide standing posture).
• The vault is a secret store on this path, not a key generator or custodian: its derivation surface (derive_ed25519, SLIP-0010) is a different feature and is not the SSH-keys path (auth-bridge.md §The gap). Held SSH keys are bearer-token-class secrets: opaque bytes, vault-encrypted at rest, decrypted at use.
• The vault is never involved in resolving clients' fingerprints; that path is the assembly-layer IdentityProvider (scope note in pocs.md §POC-2).

alkvault's encrypt/decrypt surface is &str in, Zeroizing<String> out since 0.2.0 (0.1.0 returned a plain String; the EncryptedData wrapper unchanged), a decisive constraint on any binary inner format (POC-2 measured: it would force base64 layers on both ends).

Decision

1. Inner format: OpenSSH private-key PEM (unencrypted). Winner on POC-2's 4-way evidence: natively ASCII (no encode layer over the vault's &str API), self-describing (algorithm + public key + comment round-trip), vanilla-OpenSSH-tool-compatible at every boundary (ssh -i accepted the decrypted bytes directly), and exactly one decode call (decode_secret_key(plain, None)) from russh. Rejected alternatives, recorded for the trail: ssh-key's own encrypted-key format (double-wrapping antipattern: the passphrase is itself a stored secret, moving the problem one level down), raw-seed base64 (80 B vs 540 B blobs, a legitimate size win, rejected because it couples storage to a fixed key shape; "the bytes of a PEM file" survives every algorithm ssh-key can decode), PKCS#8 PEM (parity, loses the native round-trip, adds nothing).
2. At-use lifecycle: decrypt-at-dial, one-time decode. vault.decrypt(blob) → plaintext wrapped in Zeroizing<String> at the adapter seam → one-time decode_secret_key → decoded PrivateKey lives for the dial's lifetime. The vault is needed only at dial time: locking it mid-session keeps live sessions streaming while new dials fail (POC-2 verified); unlock is side-effect-free on key identity.
3. Zeroization ownership (the gap, closed upstream). ssh-key zeroizes ed25519 key bytes on Drop (decoded keys, and their clones, clean themselves). The decrypted plaintext was the gap: alkvault's decrypt once returned a plain String, leaving the plaintext in unfenced heap between the vault's return and the adapter's wrap; the adapter owned the wrap then (proven in POC-2; that was the contract on this path) and the vault-side improvement (a Zeroizing<String>-out API) was an upstream ask filed in alkvault's own tracker. The ask shipped in alkvault 0.2.0 (2026-11-04, the tracker's review 001; this ADR is the origin it cites): decrypt returns Zeroizing<String>, the buffer zeroizes on every path (including the UTF-8 error path), and rotate transits zeroized output too. The adapter seam wrap is thereby dropped (the alkssh 0.2.0 adoption): decrypt_and_decode consumes the vault's zeroized return directly, since the double wrap would be dead posture. Decision 3's ownership now sits entirely with alkvault; the adapter's obligation shrinks to consuming the zeroized value in one pass (the decode_key_arm signature keeps riding the Zeroizing<String> type as the proof).
4. Failure modes are all fail-closed, with distinct error shapes. Locked vault → no dial (and no server: a host key that can't load prevents server construction, verified at bootstrap). Absent blob → no dial. Wrong mnemonic / tampered ciphertext → decryption failure, no dial. Unknown key_version → vault's InvalidPath refuse-before-crypto. Rotation (at-rest version bump, key constant) preserves key identity (v2→v3 proven). The adapter maps each to a distinct variant; the vault's error taxonomy is the adapter's to reshape, never to collapse.
5. EncryptedData is opaque to alkssh. The crate never interprets key_version or any record internals; the vault owns version semantics. Rotation is a storage-side operation.
6. Host keys ride the same model. Server host keys are decrypt-at-construction into Config::keys through the same load path; the client's host-pin is public config, never vault material (OQ-05's two-case policy handles pinning). The two-case host-key policy seam is the crate's trust-decision contract: the crate is policy-blind (fail-closed default; a permissive case-2 policy is an assembly choice), and the contract lives in auth.md/client.md.
7. Plaintext never persists. No disk spill on this path, ever; the vanilla-tool PEM compatibility is for the assembly layer to exploit if ever (e.g. handing a key to an external agent process, itself out of family scope).
8. The Signer seam is dismissed, with evidence. POC-2's verdict: no impl Signer exists anywhere in the 0.64.1 tree (the doc comment claiming an SSH-agent implementation is stale); a hand-rolled signer must replicate the splice convention (to_sign || string{sig}; an initial omission was silently misparsed); an in-process vault gains nothing over one-time decode (decrypted bytes exist client-side at either dial time or sign time; decrypt-per-sign keeps more plaintext copies alive while multiplying GCM+parse work). Recorded so a future revisit starts from evidence, not instinct.

Consequences

• Positive: one model covers client dial keys, host keys, and any future held material; every failure mode is proven fail-closed; the at-rest bytes are plain OpenSSH key files, so alkvault stores and rotates them with no SSH-specific logic.
• Negative: decrypted plaintext has one unfenced window (before the adapter's wrap) until alkvault changes its API; the gap is resolved (alkvault 0.2.0's Zeroizing<String> return; the adapter wrap dropped, OQ-07 amended). No plaintext window remains on this ADR's path.
• Neutral: the seed-variant size optimization remains noted as a legitimate future option if blob size ever matters; adopting it would be an ADR amendment (it changes what a stored record is).

References

• auth-bridge.md §The client side + §Host keys
• docs/research/poc-2-vault-keys.md (§1 the 4-way table, §2 the lifecycle + zeroization bounds, §3 the failure matrix, §4 the Signer dismissal, §5 host keys end to end)
• alkcall ADR-010 + alkhttp ADR-014 (the secret-material-flow family pattern; ADR-010 is alkhttp ADR-014's alkcall analog)
• Family ADR-002 decision 5, family ADR-003 decision 2
• OQ-05, OQ-06, OQ-07, OQ-13 (all resolved)