alkssh/docs/research/poc-2-vault-keys.md

# POC-2 findings — client-side at-rest key handling (vault-backed storage)

Status: Complete (phase 0). POC source: `/workspace/alkssh-poc2`
(standalone crate, disposable — never merges). All hypotheses in
`pocs.md` §POC-2 were confirmed; the doc below records the evidence,
the inner-format comparison, the Signer-seam dismissal (now with
runtime + source evidence instead of the earlier reasoning-only
withdrawal), several API facts that correct the phase 0 assumptions,
and the OQ seeds this POC produced.

## Setup exercised

- russh **0.64.1** (crates.io, ring backend) + **alkvault 0.1.0**
  (crates.io; its pinned surface: `EncryptedData`, `VaultServiceHandle`,
  `CURRENT_KEY_VERSION`), ssh-key **0.7.0-rc.11** exact-pinned and
  re-exported through `russh::keys::ssh_key` (identical tree to POC-1's
  — the two POCs exercised the same dependency graph).
- Server side: POC-1's loopback shape verbatim (fingerprint registry
  fed from `ssh-keygen -lf` output, stub `IdentityProvider`,
  `into_stream` echo server, no `wait()` before `into_stream`).
- The adapter-shaped seam under test (`vaultkey` module in the POC):
  - `KeySource { blob: Option<EncryptedData> }` — the assembly-layer
    storage stand-in (`None` = absent material);
  - `VaultKeyStore::load(&KeySource) -> Result<DialKey, LoadError>` —
    decrypt-at-dial: `vault.decrypt(blob)` → plaintext wrapped in
    `Zeroizing<String>` at the seam → one-time
    `russh::keys::decode_secret_key(&plain, None)` → fingerprint
    computed in the same step. Every error shape maps to a distinct
    fail-closed variant (`Absent` / `Locked` / `Vault(_)` / `Decode`).
- A vault-backed custom `russh::Signer` (`VaultSigner`, decrypt-per-
  sign) for the seam verdict; a second standalone vault (own mnemonic)
  for the wrong-mnemonic row of the failure matrix.

## Validated: all checks pass (39/39)

### 1. At-rest shape — OpenSSH-encoded key inside alkvault `EncryptedData` confirmed; the `&str`-only API constraint is decisive

alkvault's `encrypt`/`decrypt` surface is **`&str` in, `String` out**
(`VaultServiceHandle::encrypt(plaintext: &str, key_version: u32) ->
Result<EncryptedData, _>`, `decrypt(&EncryptedData) ->
Result<String, _>`). Any non-text inner format needs a base64 (or hex)
layer on *both* ends. The four inner-format candidates, baked and
dialed:

| inner format | plaintext | blob ciphertext | key reconstruction at use | verdict |
|---|---|---|---|---|
| **OpenSSH PEM** (unencrypted) | 387 B | 540 B | `decode_secret_key(plain, None)` — one call | **winner** |
| raw ed25519 seed (base64) | 32 B | 80 B | `Ed25519Keypair::from_seed(&seed) -> PrivateKey` (`From` exists) | works, smallest blob; drops comment/public metadata (pub is re-derived from seed — fine) |
| ssh-key self-encrypted PEM (passphrase, bcrypt KDF) | 449 B | 616 B | two decryptions: vault, then `decode_secret_key(pem, Some(pass))` | works but pointless — the passphrase is *itself* a stored secret, so the model just moves the same problem one level down; double-wrapping buys nothing |
| PKCS#8 PEM (unencrypted) | 246 B | 252 B | `encode_pkcs8_pem` (russh) at rest; `decode_secret_key(plain, None)` at use | works; loses the native comment/model round-trip; adds nothing over OpenSSH PEM |

(All four rows produced the correct fingerprint through the full
decrypt→decode→dial pipeline; the seed and PKCS#8 rows were checked to
reproduce the *same* registered fingerprint as the PEM inner, proving
inner and outer representation choices are orthogonal to the key
identity the registry sees.)

The **OpenSSH PEM inner wins** on honest grounds: ASCII natively (no
encode layer), self-describing (algorithm + public key + comment
embedded), vanilla-OpenSSH-tool-compatible at every boundary, and
exactly one decode call from russh. The seed variant is a legitimate
size optimization the ADR can note (80 B vs 540 B blobs) — it couples
alkssh's storage to a fixed key shape though, whereas "a PEM file's
bytes" works for every algorithm ssh-key can decode. This mirrors the
alkhttp ADR-014 analogy exactly as hypothesized in `pocs.md`.

At-rest hygiene, verified end to end:

- `EncryptedData { key_version: 2, salt, iv, data }` serializes to
  stable JSON (serde), roundtrips, and decrypts to the byte-identical
  original PEM. The salt field is 32 B (44 b64 chars), the IV 12 B
  (16 b64 chars) — pure wire-format confirmation against the type's
  documented shape.
- AES-256-GCM opacity: the ciphertext's `data` field contains no
  recognizable substrings of the key (checked `OPENSSH`, `PRIVATE`,
  `ssh-ed25519`, and a 30-byte interior slice of the PEM — none found
  in ciphertext or the serialized JSON).
- Fresh IV + fresh salt per encrypt: re-encrypting the identical
  plaintext produces entirely different blobs (no determinism trap for
  naive dedup code to stumble into).

### 2. Dial lifecycle — decrypt-at-dial, zeroization bounds, lock/unlock mid-session

- The full dial through the adapter works: `load` →
  `authenticate_publickey` → `Success`, the server's
  `auth_publickey` fires post-signature-verification and resolves the
  fingerprint against the assembly-registered identity, and the
  fingerprint matches the `ssh-keygen`-sourced registry value exactly
  (POC-1's parity, re-proven through the vault pipeline).
- **The vault is needed only at dial time.** `vault.lock()` mid-session:
  the already-established session kept streaming (echo roundtrip after
  lock), while a *new* `load` on the same blob failed. This is the
  decode-once model behaving as designed: the decoded `PrivateKey` is
  the dial's lifetime, not a vault-coupled lease.
- Re-`unlock` with the same mnemonic → decrypt works again, same
  fingerprint. Lock/unlock cycles are side-effect-free on the key.
- **Zeroization bounds** (source-verified in ssh-key 0.7.0-rc.11):
  - `Ed25519PrivateKey` zeroizes on `Drop` (`private/ed25519.rs` —
    `impl Drop { self.0.zeroize() }`), so the decoded key's secret
    bytes are cleaned up by ssh-key itself when the `PrivateKey` (or
    clones of it — the type is `Clone` with zeroizing storage) drops;
  - alkvault zeroizes its own key material (`EncryptionKey`
    `#[zeroize(drop)]`, mnemonic phrase via `Zeroizing`, `lock()`
    purges seed + cache);
  - **the gap:** `VaultServiceHandle::decrypt` returns a *plain*
    `String`. The decrypted plaintext key sits in unfenced heap memory
    from the vault's return until the adapter takes ownership. The
    adapter shape proven here wraps it in `Zeroizing<String>` at the
    seam (owning the move), so the adapter-bound bytes zeroize on
    drop — but the pre-wrap window is the vault's API shape to close.
    → OQ seed below.

### 3. Failure matrix — every failure mode observed; all fail closed

| disturbed condition | observed | adapter mapping |
|---|---|---|
| vault locked | `VaultServiceError::VaultLocked` | `LoadError::Locked` — no dial |
| blob absent at source (assembly layer has no record) | no vault call needed | `LoadError::Absent` — no dial |
| wrong mnemonic (unrelated seed) | `VaultServiceError::Encryption("decryption error: aead::Error")` — a *decryption* failure, distinguishable from `Locked` | no dial |
| tampered ciphertext (one flipped byte) | GCM integrity failure (same decrypt-error family) | no dial |
| `key_version = 0` (reserved/legacy) | `VaultServiceError::InvalidPath` — the vault refuses before touching crypto | no dial |
| `key_version = 7` (future, undecodable) | derivation-path error | no dial |
| healthy rotation v2→v3 | `rotate()` → new blob stamped `key_version: 3`, decrypt yields the identical PEM, decoded fingerprint identical | dial succeeds |

The wrong-mnemonic/tampered cases land in
`VaultServiceError::Encryption` *as strings* — the error taxonomy at
the adapter is ours to shape (distinct variants), but the root causes
are distinguishable today via the variant + message. The `InvalidPath`
behavior for version < 2 is deliberate (v1 is the TS_PBKDF2 legacy the
vault cannot decrypt; rotation re-encryption is the migration path,
per alkvault's own docs) — consistent with the ADR-021-style
version-indexed derivation the alkssh ADR will cite for rotation.

### 4. Signer-seam dismissal — confirmed, now with runtime and source evidence

`pocs.md` expected the answer "no — recorded so the ADR can dismiss it
with evidence". Evidence assembled:

- **The trait exists but has no provided implementation**: russh
  0.64.1 re-exports `Signer` at the crate root
  (`russh::Signer`, `auth::Signer` shape: `auth_sign(&AgentIdentity,
  hash_alg, to_sign)`), and `Handle::authenticate_publickey_with(user,
  ssh_key::PublicKey, hash_alg, &mut S)` accepts it. The doc comment
  claims "Currently, this crate only provides an implementation for an
  SSH agent" — **that is stale**: there is no `impl Signer` for
  `AgentClient` (or anything else) anywhere in the 0.64.1 tree. A
  custom signer must be built by hand.
- **The wire contract is non-obvious and hand-rolled**: the flow is
  probe (`has_sig=0`, `write_auth_request`) → server `USERAUTH_PK_OK`
  → client builds `session_id || userauth-prefix` (`client_make_to_sign`)
  → `Reply::SignRequest` carries the full buffer → the signer returns
  **`to_sign || string{signature}`** — the original buffer with the
  signature length-prefixed as an SSH string — and the client splices
  `buffer[i0..]` as the packet payload. (First attempt appended the raw
  signature without the outer length prefix: the server silently
  misparsed and declined — `USERAUTH_FAILURE` — *without the
  `auth_publickey` hook ever firing*. Signature-verification failures
  happen inside russh before any handler hook, which is also a clean
  fail-closed property worth noting for the mapping docs.)
- **Runtime proof that the seam works anyway**: `VaultSigner`
  (decrypt-per-sign against the vault) authenticated successfully
  end-to-end via `authenticate_publickey_with` — the splice convention
  replicated, `PrivateKey: signature::Signer::try_sign` +
  `.encoded()` (i.e. `sig` → SSH wire blob) then length-prefixed into
  the returned buffer.
- **Locked-vault fail-closed on the Signer path, too**: with the vault
  locked, the sign-time decrypt fails and the auth call itself errors
  (`Err` from `authenticate_publickey_with`) — no half-auth, no
  fallback.
- **Verdict: dismissed for the vault path.** The vault is in-process:
  decrypted key bytes necessarily exist client-side at *either* dial
  time or sign time — the Signer seam buys no custody improvement, and
  decrypt-per-sign merely multiplies GCM+parse work per signature while
  keeping *more* plaintext copies alive (per-signature) rather than
  fewer. The seam's genuine home is *external* signer processes
  (ssh-agent), which the family treats as out of scope. The ADR can
  record: one-time `decode_secret_key` at dial; `Signer` not taken.

### 5. Host keys ride the same model — server bootstrap + vanilla compatibility

- The server's host key was generated in-POC, vault-encrypted, and
  decrypted-at-construction into `Config::keys` through the same
  `VaultKeyStore::load` the client dial uses — one model, both sides.
  Vanilla OpenSSH authenticated against that server, the
  `known_hosts` fingerprint (written by the vanilla client) matched
  the vault host key's fingerprint exactly, and the russh client's
  `check_server_key` observed the same fingerprint.
- The client's *pin* — the expected host fingerprint — is **public
  config, not vault material** (`check_server_key` policy + the
  known-hosts entry); only the server's *private* host key is held
  secret. The ADR's "assembly configures `Config::keys`" note can now
  cite the exact decrypt-at-boot path.
- **Vanilla-tool compatibility of the plaintext**: the decrypted
  client-key PEM written to a 0600 temp file was accepted by vanilla
  `ssh -i` directly (auth succeeded; `Permission denied` never
  appeared — exit 255 only because the POC server intentionally serves
  no exec). The at-rest bytes are, at rest *and* in motion, plain
  OpenSSH key material.
- Locked vault at server bootstrap → `LoadError::Locked` → **no
  server is constructed** — the fail-closed posture extends to the
  host-key boot path.

## Hypothesis verdicts (pocs.md §POC-2)

- "OpenSSH private-key encoding as the inner format; alkvault
  `encrypt`/`decrypt` (`EncryptedData`) as the outer wrapper" —
  **confirmed**, and strengthened by the 4-way comparison table (the
  seed-only variant as a size optimization; the ssh-key self-encrypted
  variant recorded as a double-wrapping antipattern).
- "decrypt-at-dial → one-time `decode_secret_key` → zeroized handling
  for the dial's lifetime" — **confirmed** (ssh-key zeroizes key bytes
  on drop; adapter owns plaintext zeroization at the seam; vault
  in-memory state self-purges on `lock`).
- "what happens locked / absent / rotation" — **confirmed as
  fail-closed** with distinct, mappable error shapes; healthy rotation
  (v2→v3) preserves key identity.
- "Signer-shaped seam still worth anything — expected: no" —
  **dismissed with evidence** (no provided impl; hand-rolled splice
  convention; in-process vault gains nothing; runtime proof the seam
  works anyway).
- "Server and client host-key material riding the same model" —
  **confirmed** end to end, including vanilla-tool compatibility and
  locked-at-bootstrap fail-closed.

## API facts for the ADRs (all verified against crates.io sources)

- **alkvault 0.1.0**: `VaultServiceHandle` is a sync, cloneable handle
  (interior `Arc<RwLock>`; short-held locks — no tokio coupling);
  `unlock_new(word_count)` returns the phrase as `Zeroizing<String>`;
  `encrypt(&str, u32)` / `decrypt(&EncryptedData) -> Result<String>` /
  `rotate(&EncryptedData, to_version)`; current version 2, versions
  derive on per-version paths; decrypt errors distinguish
  `VaultLocked` / `Encryption(…)` / `InvalidPath`.
- **russh 0.64.1**: `keys::decode_secret_key(&str, Option<&str>)`
  handles OpenSSH (incl. passphrase/bcrypt via the enabled
  `ssh-key/encryption` feature), PuTTY PPK, PKCS#5 and PKCS#8 (+ the
  `BEGIN PRIVATE KEY` / `BEGIN ENCRYPTED PRIVATE KEY` PEM cases);
  `keys::encode_pkcs8_pem[_encrypted]` exist for writers (PKCS#8
  support lives in *russh*, not in ssh-key — ssh-key rc11 has no
  `pkcs8` feature; the research note about ssh-key-own-PKCS#8 does not
  apply at this pin). `PrivateKey: signature::Signer<Signature>`,
  `Signature::encoded()` via `signature::SignatureEncoding`; russh's
  own signing (`helpers::sign_with_hash_alg`) is exactly
  `try_sign(keypair, msg)?.encoded()` — a vault signer can mirror it,
  but `helpers` (and its `EncodedExt`) is crate-private; the public
  path is `russh::keys::signature::Signer` + `russh::keys::ssh_encoding::Encode`.
- **Server-side auth pipeline**: a failed/malformed signature produces
  `USERAUTH_FAILURE` *without* calling `auth_publickey` — handler
  hooks only fire on cryptographically valid signatures (POC-1
  established the "after verification" half; POC-2 adds "invalid sigs
  never reach the handler at all").
- **Client-side event routing**: `check_server_key` events are client-
  handler facts, so anything asserting on them must read the client's
  event surface, not the server session's (POC logistics note for
  future bake-offs).

## OQ seeds carried forward

1. **alkvault `decrypt` returns plain `String`** — a zeroization gap at
   the seam (the vault's own key material is zeroized; the decrypted
   *plaintext* is not until the adapter wraps it). Candidate fixes:
   alkvault returns `Zeroizing<String>`, or the alkssh ADR documents
   the adapter-side `Zeroizing` obligation as the family convention.
   POC-2's adapter shape does the latter.
2. **Key-record schema**: alkssh's vault-backed key record should treat
   `EncryptedData` as opaque and never interpret `key_version` itself
   (the vault owns version semantics; `InvalidPath` for unknown
   versions is the fail-closed behavior). Rotation (at-rest version
   bump, key constant) is a storage-side operation — proven to
   preserve key identity.
3. **One vault, many roles?** The POC held client dial keys *and* the
   server host key in one vault (one mnemonic). Whether the assembly
   layer should run separate vault instances per role (client vs
   server custody) is an assembly policy question, not an alkssh
   design constraint — recorded so phase 1 doesn't bake it in.
4. **Never persist decrypted plaintext**: the vanilla-tool check
   wrote a decrypted key to a 0600 temp file (then zeroized and
   deleted it) — that is a test-only demonstration of the byte format.
   Adapters do not spill plaintext to disk; the vanilla-compatible PEM
   is handed to processes only via the assembly layer if ever.

## Reference run

Full output of a clean run at findings time: **39 checks, 5 groups**
(at-rest shape 11, dial lifecycle 8, failure matrix 10, Signer seam 3,
host key + vanilla 7) — all PASS. `overall=true`, exit 0.

## References

- `pocs.md` §POC-2 (plan), `auth-bridge.md` §The client side (design —
  updated with a pointer to these verdicts), `russh.md`
  (`client::Handle` authenticate methods — updated with the
  Signer-verdict pointer)
- POC-1: `poc-1-auth-bridge.md` + `/workspace/alkssh-poc1`
  (server loopback shape reused)
- russh 0.64.1, alkvault 0.1.0, ssh-key 0.7.0-rc.11 (crates.io,
  ring backend), OpenSSH 9.6p1 (vanilla tooling)