alkssh/docs/architecture/decisions/005-at-rest-key-custody.md

status, last_updated

status	last_updated
reviewed	2026-10-28

ADR-005: At-rest key custody — vault-wrapped OpenSSH-PEM, decrypt-at-use

Status

Accepted — 2026-10-26. Evidence-backed (POC-2, 39/39, including the 4-way inner-format bake and the failure matrix); amendable only via a new ADR.

Context

On the client direction alkssh holds secret material (dial keys); on the server direction it holds the host key. Family rules in force:

• Secrets come from alkvault-backed sources — never env vars, never plaintext at rest (family ADR-002 decision 5; family-wide standing posture).
• The vault is a secret store on this path, not a key generator or custodian: its derivation surface (derive_ed25519, SLIP-0010) is a different feature and is not the SSH-keys path (auth-bridge.md §The gap). Held SSH keys are bearer-token-class secrets: opaque bytes, vault-encrypted at rest, decrypted at use.
• The vault is never involved in resolving clients' fingerprints — that path is the assembly-layer IdentityProvider (scope note in pocs.md §POC-2).

alkvault 0.1.0's encrypt/decrypt surface is &str in, String out (EncryptedData as the wrapper) — a decisive constraint on any binary inner format (POC-2 measured: it would force base64 layers on both ends).

Decision

1. Inner format: OpenSSH private-key PEM (unencrypted). Winner on POC-2's 4-way evidence: natively ASCII (no encode layer over the vault's &str API), self-describing (algorithm + public key + comment round-trip), vanilla-OpenSSH-tool-compatible at every boundary (ssh -i accepted the decrypted bytes directly), and exactly one decode call (decode_secret_key(plain, None)) from russh. Rejected alternatives, recorded for the trail: ssh-key's own encrypted-key format (double-wrapping antipattern — the passphrase is itself a stored secret, moving the problem one level down), raw-seed base64 (80 B vs 540 B blobs — legitimate size win, rejected because it couples storage to a fixed key shape; "the bytes of a PEM file" survives every algorithm ssh-key can decode), PKCS#8 PEM (parity, loses the native round-trip, adds nothing).
2. At-use lifecycle: decrypt-at-dial, one-time decode. vault.decrypt(blob) → plaintext wrapped in Zeroizing<String> at the adapter seam → one-time decode_secret_key → decoded PrivateKey lives for the dial's lifetime. The vault is needed only at dial time: locking it mid-session keeps live sessions streaming while new dials fail (POC-2 verified); unlock is side-effect-free on key identity.
3. Zeroization ownership (the known gap). ssh-key zeroizes ed25519 key bytes on Drop (decoded keys, and their clones, clean themselves). The decrypted plaintext is the gap: alkvault's decrypt returns a plain String, so from the vault's return to the adapter's wrap the plaintext sits in unfenced heap. The adapter owns the wrap (proven in POC-2) — that is the contract on this path; the vault-side improvement (a Zeroizing<String>-out API) is an upstream ask filed in alkvault's own tracker, not an alkssh obligation — when it lands, this ADR's pointer updates as a one-line doc touch (alkssh OQ-07 is closed at the adapter-convention level).
4. Failure modes are all fail-closed, with distinct error shapes. Locked vault → no dial (and no server: a host key that can't load prevents server construction — verified at bootstrap). Absent blob → no dial. Wrong mnemonic / tampered ciphertext → decryption failure, no dial. Unknown key_version → vault's InvalidPath refuse-before-crypto. Rotation (at-rest version bump, key constant) preserves key identity (v2→v3 proven). The adapter maps each to a distinct variant; the vault's error taxonomy is the adapter's to reshape, never to collapse.
5. EncryptedData is opaque to alkssh. The crate never interprets key_version or any record internals — the vault owns version semantics. Rotation is a storage-side operation.
6. Host keys ride the same model. Server host keys are decrypt-at-construction into Config::keys through the same load path; the client's host-pin is public config, never vault material (OQ-05's two-case policy handles pinning). The two-case host-key policy seam is the crate's trust-decision contract — the crate is policy-blind (fail-closed default; a permissive case-2 policy is an assembly choice) — the contract lives in auth.md/client.md.
7. Plaintext never persists. No disk spill on this path, ever; the vanilla-tool PEM compatibility is for the assembly layer to exploit if ever (e.g. handing a key to an external agent process — itself out of family scope).
8. The Signer seam is dismissed, with evidence. POC-2's verdict: no impl Signer exists anywhere in the 0.64.1 tree (the doc comment claiming an SSH-agent implementation is stale); a hand-rolled signer must replicate the splice convention (to_sign || string{sig} — initial omission was silently misparsed); an in-process vault gains nothing over one-time decode (decrypted bytes exist client-side at either dial time or sign time; decrypt-per-sign keeps more plaintext copies alive while multiplying GCM+parse work). Recorded so a future revisit starts from evidence, not instinct.

Consequences

• Positive: one model covers client dial keys, host keys, and any future held material; every failure mode is proven fail-closed; the at-rest bytes are plain OpenSSH key files, so alkvault stores and rotates them with no SSH-specific logic.
• Negative: decrypted plaintext has one unfenced window (before the adapter's wrap) until alkvault changes its API — the gap is recorded, owned, and filed with its owner (the alkvault tracker; OQ-07 resolved at the adapter level), not silently absorbed.
• Neutral: the seed-variant size optimization remains noted as a legitimate future option if blob size ever matters; adopting it would be an ADR amendment (it changes what a stored record is).

References

• auth-bridge.md §The client side + §Host keys
• docs/research/poc-2-vault-keys.md (§1 the 4-way table, §2 the lifecycle + zeroization bounds, §3 the failure matrix, §4 the Signer dismissal, §5 host keys end to end)
• alkcall ADR-010 + alkhttp ADR-014 (the secret-material-flow family pattern — ADR-010 is alkhttp ADR-014's alkcall analog)
• Family ADR-002 decision 5, family ADR-003 decision 2
• OQ-05, OQ-06, OQ-07, OQ-13 (all resolved)