docs(phase-0): POC-2 findings — vault-backed key custody confirmed

- OpenSSH-PEM inner + alkvault EncryptedData outer confirmed as the
  at-rest shape (4-way inner-format comparison recorded; alkvault's
  &str-only encrypt/decrypt surface is the decisive API constraint)
- decrypt-at-dial lifecycle proven: mid-session vault lock leaves live
  sessions intact; locked/absent/tampered/rotate all fail closed with
  distinct error shapes
- Signer seam dismissed with evidence: no provided impl in russh
  0.64.1 (doc claim stale), splice convention hand-rolled, in-process
  vault gains nothing over one-time decode
- host keys ride the same model end to end incl. vanilla-tool
  compatibility; 39/39 checks pass
- OQ seeds: decrypt-returns-plain-String zeroization gap, opaque
  EncryptedData handling, vault-per-role policy

Verification: POC run 39 checks / 5 groups all PASS (source at
/workspace/alkssh-poc2, standalone, never merges)

docs/research/auth-bridge.md

@@ -165,6 +165,17 @@ work:
   an alknet-`main`-shaped `KeySource` seam
   (`{ File, Memory }`-equivalent) backed by vault-encrypted material
   at the assembly layer, mirroring alkhttp's capability injection.
+- **POC-2 verdicts (see `poc-2-vault-keys.md`)**: OpenSSH-PEM inner in
+  alkvault `EncryptedData` confirmed as the at-rest winner (4-way
+  comparison; note alkvault's `encrypt`/`decrypt` speak `String` only,
+  so binary formats need a base64 layer); one-time
+  `decode_secret_key` at dial confirmed; ssh-key zeroizes ed25519 key
+  bytes on drop but the vault's `decrypt` returns a plain `String` —
+  the adapter owns plaintext zeroization at the seam (`Zeroizing`
+  wrap) [OQ seed]; the `Signer` seam is dismissed with evidence (no
+  provided impl in 0.64.1; works when hand-rolled; in-process vault
+  gains nothing). Locked/absent/tampered/rotation all fail closed
+  through distinct, mappable error shapes.
 - **Scope note**: the vault is **client-side only**. The server side
   never touches the vault for *peer* identity — it receives clients'
   public keys and resolves fingerprints through the assembly layer's

docs/research/poc-2-vault-keys.md

@@ -0,0 +1,297 @@
+# POC-2 findings — client-side at-rest key handling (vault-backed storage)
+
+Status: Complete (phase 0). POC source: `/workspace/alkssh-poc2`
+(standalone crate, disposable — never merges). All hypotheses in
+`pocs.md` §POC-2 were confirmed; the doc below records the evidence,
+the inner-format comparison, the Signer-seam dismissal (now with
+runtime + source evidence instead of the earlier reasoning-only
+withdrawal), several API facts that correct the phase 0 assumptions,
+and the OQ seeds this POC produced.
+
+## Setup exercised
+
+- russh **0.64.1** (crates.io, ring backend) + **alkvault 0.1.0**
+  (crates.io; its pinned surface: `EncryptedData`, `VaultServiceHandle`,
+  `CURRENT_KEY_VERSION`), ssh-key **0.7.0-rc.11** exact-pinned and
+  re-exported through `russh::keys::ssh_key` (identical tree to POC-1's
+  — the two POCs exercised the same dependency graph).
+- Server side: POC-1's loopback shape verbatim (fingerprint registry
+  fed from `ssh-keygen -lf` output, stub `IdentityProvider`,
+  `into_stream` echo server, no `wait()` before `into_stream`).
+- The adapter-shaped seam under test (`vaultkey` module in the POC):
+  - `KeySource { blob: Option<EncryptedData> }` — the assembly-layer
+    storage stand-in (`None` = absent material);
+  - `VaultKeyStore::load(&KeySource) -> Result<DialKey, LoadError>` —
+    decrypt-at-dial: `vault.decrypt(blob)` → plaintext wrapped in
+    `Zeroizing<String>` at the seam → one-time
+    `russh::keys::decode_secret_key(&plain, None)` → fingerprint
+    computed in the same step. Every error shape maps to a distinct
+    fail-closed variant (`Absent` / `Locked` / `Vault(_)` / `Decode`).
+- A vault-backed custom `russh::Signer` (`VaultSigner`, decrypt-per-
+  sign) for the seam verdict; a second standalone vault (own mnemonic)
+  for the wrong-mnemonic row of the failure matrix.
+
+## Validated: all checks pass (39/39)
+
+### 1. At-rest shape — OpenSSH-encoded key inside alkvault `EncryptedData` confirmed; the `&str`-only API constraint is decisive
+
+alkvault's `encrypt`/`decrypt` surface is **`&str` in, `String` out**
+(`VaultServiceHandle::encrypt(plaintext: &str, key_version: u32) ->
+Result<EncryptedData, _>`, `decrypt(&EncryptedData) ->
+Result<String, _>`). Any non-text inner format needs a base64 (or hex)
+layer on *both* ends. The four inner-format candidates, baked and
+dialed:
+
+| inner format | plaintext | blob ciphertext | key reconstruction at use | verdict |
+|---|---|---|---|---|
+| **OpenSSH PEM** (unencrypted) | 387 B | 540 B | `decode_secret_key(plain, None)` — one call | **winner** |
+| raw ed25519 seed (base64) | 32 B | 80 B | `Ed25519Keypair::from_seed(&seed) -> PrivateKey` (`From` exists) | works, smallest blob; drops comment/public metadata (pub is re-derived from seed — fine) |
+| ssh-key self-encrypted PEM (passphrase, bcrypt KDF) | 449 B | 616 B | two decryptions: vault, then `decode_secret_key(pem, Some(pass))` | works but pointless — the passphrase is *itself* a stored secret, so the model just moves the same problem one level down; double-wrapping buys nothing |
+| PKCS#8 PEM (unencrypted) | 246 B | 252 B | `encode_pkcs8_pem` (russh) at rest; `decode_secret_key(plain, None)` at use | works; loses the native comment/model round-trip; adds nothing over OpenSSH PEM |
+
+(All four rows produced the correct fingerprint through the full
+decrypt→decode→dial pipeline; the seed and PKCS#8 rows were checked to
+reproduce the *same* registered fingerprint as the PEM inner, proving
+inner and outer representation choices are orthogonal to the key
+identity the registry sees.)
+
+The **OpenSSH PEM inner wins** on honest grounds: ASCII natively (no
+encode layer), self-describing (algorithm + public key + comment
+embedded), vanilla-OpenSSH-tool-compatible at every boundary, and
+exactly one decode call from russh. The seed variant is a legitimate
+size optimization the ADR can note (80 B vs 540 B blobs) — it couples
+alkssh's storage to a fixed key shape though, whereas "a PEM file's
+bytes" works for every algorithm ssh-key can decode. This mirrors the
+alkhttp ADR-014 analogy exactly as hypothesized in `pocs.md`.
+
+At-rest hygiene, verified end to end:
+
+- `EncryptedData { key_version: 2, salt, iv, data }` serializes to
+  stable JSON (serde), roundtrips, and decrypts to the byte-identical
+  original PEM. The salt field is 32 B (44 b64 chars), the IV 12 B
+  (16 b64 chars) — pure wire-format confirmation against the type's
+  documented shape.
+- AES-256-GCM opacity: the ciphertext's `data` field contains no
+  recognizable substrings of the key (checked `OPENSSH`, `PRIVATE`,
+  `ssh-ed25519`, and a 30-byte interior slice of the PEM — none found
+  in ciphertext or the serialized JSON).
+- Fresh IV + fresh salt per encrypt: re-encrypting the identical
+  plaintext produces entirely different blobs (no determinism trap for
+  naive dedup code to stumble into).
+
+### 2. Dial lifecycle — decrypt-at-dial, zeroization bounds, lock/unlock mid-session
+
+- The full dial through the adapter works: `load` →
+  `authenticate_publickey` → `Success`, the server's
+  `auth_publickey` fires post-signature-verification and resolves the
+  fingerprint against the assembly-registered identity, and the
+  fingerprint matches the `ssh-keygen`-sourced registry value exactly
+  (POC-1's parity, re-proven through the vault pipeline).
+- **The vault is needed only at dial time.** `vault.lock()` mid-session:
+  the already-established session kept streaming (echo roundtrip after
+  lock), while a *new* `load` on the same blob failed. This is the
+  decode-once model behaving as designed: the decoded `PrivateKey` is
+  the dial's lifetime, not a vault-coupled lease.
+- Re-`unlock` with the same mnemonic → decrypt works again, same
+  fingerprint. Lock/unlock cycles are side-effect-free on the key.
+- **Zeroization bounds** (source-verified in ssh-key 0.7.0-rc.11):
+  - `Ed25519PrivateKey` zeroizes on `Drop` (`private/ed25519.rs` —
+    `impl Drop { self.0.zeroize() }`), so the decoded key's secret
+    bytes are cleaned up by ssh-key itself when the `PrivateKey` (or
+    clones of it — the type is `Clone` with zeroizing storage) drops;
+  - alkvault zeroizes its own key material (`EncryptionKey`
+    `#[zeroize(drop)]`, mnemonic phrase via `Zeroizing`, `lock()`
+    purges seed + cache);
+  - **the gap:** `VaultServiceHandle::decrypt` returns a *plain*
+    `String`. The decrypted plaintext key sits in unfenced heap memory
+    from the vault's return until the adapter takes ownership. The
+    adapter shape proven here wraps it in `Zeroizing<String>` at the
+    seam (owning the move), so the adapter-bound bytes zeroize on
+    drop — but the pre-wrap window is the vault's API shape to close.
+    → OQ seed below.
+
+### 3. Failure matrix — every failure mode observed; all fail closed
+
+| disturbed condition | observed | adapter mapping |
+|---|---|---|
+| vault locked | `VaultServiceError::VaultLocked` | `LoadError::Locked` — no dial |
+| blob absent at source (assembly layer has no record) | no vault call needed | `LoadError::Absent` — no dial |
+| wrong mnemonic (unrelated seed) | `VaultServiceError::Encryption("decryption error: aead::Error")` — a *decryption* failure, distinguishable from `Locked` | no dial |
+| tampered ciphertext (one flipped byte) | GCM integrity failure (same decrypt-error family) | no dial |
+| `key_version = 0` (reserved/legacy) | `VaultServiceError::InvalidPath` — the vault refuses before touching crypto | no dial |
+| `key_version = 7` (future, undecodable) | derivation-path error | no dial |
+| healthy rotation v2→v3 | `rotate()` → new blob stamped `key_version: 3`, decrypt yields the identical PEM, decoded fingerprint identical | dial succeeds |
+
+The wrong-mnemonic/tampered cases land in
+`VaultServiceError::Encryption` *as strings* — the error taxonomy at
+the adapter is ours to shape (distinct variants), but the root causes
+are distinguishable today via the variant + message. The `InvalidPath`
+behavior for version < 2 is deliberate (v1 is the TS_PBKDF2 legacy the
+vault cannot decrypt; rotation re-encryption is the migration path,
+per alkvault's own docs) — consistent with the ADR-021-style
+version-indexed derivation the alkssh ADR will cite for rotation.
+
+### 4. Signer-seam dismissal — confirmed, now with runtime and source evidence
+
+`pocs.md` expected the answer "no — recorded so the ADR can dismiss it
+with evidence". Evidence assembled:
+
+- **The trait exists but has no provided implementation**: russh
+  0.64.1 re-exports `Signer` at the crate root
+  (`russh::Signer`, `auth::Signer` shape: `auth_sign(&AgentIdentity,
+  hash_alg, to_sign)`), and `Handle::authenticate_publickey_with(user,
+  ssh_key::PublicKey, hash_alg, &mut S)` accepts it. The doc comment
+  claims "Currently, this crate only provides an implementation for an
+  SSH agent" — **that is stale**: there is no `impl Signer` for
+  `AgentClient` (or anything else) anywhere in the 0.64.1 tree. A
+  custom signer must be built by hand.
+- **The wire contract is non-obvious and hand-rolled**: the flow is
+  probe (`has_sig=0`, `write_auth_request`) → server `USERAUTH_PK_OK`
+  → client builds `session_id || userauth-prefix` (`client_make_to_sign`)
+  → `Reply::SignRequest` carries the full buffer → the signer returns
+  **`to_sign || string{signature}`** — the original buffer with the
+  signature length-prefixed as an SSH string — and the client splices
+  `buffer[i0..]` as the packet payload. (First attempt appended the raw
+  signature without the outer length prefix: the server silently
+  misparsed and declined — `USERAUTH_FAILURE` — *without the
+  `auth_publickey` hook ever firing*. Signature-verification failures
+  happen inside russh before any handler hook, which is also a clean
+  fail-closed property worth noting for the mapping docs.)
+- **Runtime proof that the seam works anyway**: `VaultSigner`
+  (decrypt-per-sign against the vault) authenticated successfully
+  end-to-end via `authenticate_publickey_with` — the splice convention
+  replicated, `PrivateKey: signature::Signer::try_sign` +
+  `.encoded()` (i.e. `sig` → SSH wire blob) then length-prefixed into
+  the returned buffer.
+- **Locked-vault fail-closed on the Signer path, too**: with the vault
+  locked, the sign-time decrypt fails and the auth call itself errors
+  (`Err` from `authenticate_publickey_with`) — no half-auth, no
+  fallback.
+- **Verdict: dismissed for the vault path.** The vault is in-process:
+  decrypted key bytes necessarily exist client-side at *either* dial
+  time or sign time — the Signer seam buys no custody improvement, and
+  decrypt-per-sign merely multiplies GCM+parse work per signature while
+  keeping *more* plaintext copies alive (per-signature) rather than
+  fewer. The seam's genuine home is *external* signer processes
+  (ssh-agent), which the family treats as out of scope. The ADR can
+  record: one-time `decode_secret_key` at dial; `Signer` not taken.
+
+### 5. Host keys ride the same model — server bootstrap + vanilla compatibility
+
+- The server's host key was generated in-POC, vault-encrypted, and
+  decrypted-at-construction into `Config::keys` through the same
+  `VaultKeyStore::load` the client dial uses — one model, both sides.
+  Vanilla OpenSSH authenticated against that server, the
+  `known_hosts` fingerprint (written by the vanilla client) matched
+  the vault host key's fingerprint exactly, and the russh client's
+  `check_server_key` observed the same fingerprint.
+- The client's *pin* — the expected host fingerprint — is **public
+  config, not vault material** (`check_server_key` policy + the
+  known-hosts entry); only the server's *private* host key is held
+  secret. The ADR's "assembly configures `Config::keys`" note can now
+  cite the exact decrypt-at-boot path.
+- **Vanilla-tool compatibility of the plaintext**: the decrypted
+  client-key PEM written to a 0600 temp file was accepted by vanilla
+  `ssh -i` directly (auth succeeded; `Permission denied` never
+  appeared — exit 255 only because the POC server intentionally serves
+  no exec). The at-rest bytes are, at rest *and* in motion, plain
+  OpenSSH key material.
+- Locked vault at server bootstrap → `LoadError::Locked` → **no
+  server is constructed** — the fail-closed posture extends to the
+  host-key boot path.
+
+## Hypothesis verdicts (pocs.md §POC-2)
+
+- "OpenSSH private-key encoding as the inner format; alkvault
+  `encrypt`/`decrypt` (`EncryptedData`) as the outer wrapper" —
+  **confirmed**, and strengthened by the 4-way comparison table (the
+  seed-only variant as a size optimization; the ssh-key self-encrypted
+  variant recorded as a double-wrapping antipattern).
+- "decrypt-at-dial → one-time `decode_secret_key` → zeroized handling
+  for the dial's lifetime" — **confirmed** (ssh-key zeroizes key bytes
+  on drop; adapter owns plaintext zeroization at the seam; vault
+  in-memory state self-purges on `lock`).
+- "what happens locked / absent / rotation" — **confirmed as
+  fail-closed** with distinct, mappable error shapes; healthy rotation
+  (v2→v3) preserves key identity.
+- "Signer-shaped seam still worth anything — expected: no" —
+  **dismissed with evidence** (no provided impl; hand-rolled splice
+  convention; in-process vault gains nothing; runtime proof the seam
+  works anyway).
+- "Server and client host-key material riding the same model" —
+  **confirmed** end to end, including vanilla-tool compatibility and
+  locked-at-bootstrap fail-closed.
+
+## API facts for the ADRs (all verified against crates.io sources)
+
+- **alkvault 0.1.0**: `VaultServiceHandle` is a sync, cloneable handle
+  (interior `Arc<RwLock>`; short-held locks — no tokio coupling);
+  `unlock_new(word_count)` returns the phrase as `Zeroizing<String>`;
+  `encrypt(&str, u32)` / `decrypt(&EncryptedData) -> Result<String>` /
+  `rotate(&EncryptedData, to_version)`; current version 2, versions
+  derive on per-version paths; decrypt errors distinguish
+  `VaultLocked` / `Encryption(…)` / `InvalidPath`.
+- **russh 0.64.1**: `keys::decode_secret_key(&str, Option<&str>)`
+  handles OpenSSH (incl. passphrase/bcrypt via the enabled
+  `ssh-key/encryption` feature), PuTTY PPK, PKCS#5 and PKCS#8 (+ the
+  `BEGIN PRIVATE KEY` / `BEGIN ENCRYPTED PRIVATE KEY` PEM cases);
+  `keys::encode_pkcs8_pem[_encrypted]` exist for writers (PKCS#8
+  support lives in *russh*, not in ssh-key — ssh-key rc11 has no
+  `pkcs8` feature; the research note about ssh-key-own-PKCS#8 does not
+  apply at this pin). `PrivateKey: signature::Signer<Signature>`,
+  `Signature::encoded()` via `signature::SignatureEncoding`; russh's
+  own signing (`helpers::sign_with_hash_alg`) is exactly
+  `try_sign(keypair, msg)?.encoded()` — a vault signer can mirror it,
+  but `helpers` (and its `EncodedExt`) is crate-private; the public
+  path is `russh::keys::signature::Signer` + `russh::keys::ssh_encoding::Encode`.
+- **Server-side auth pipeline**: a failed/malformed signature produces
+  `USERAUTH_FAILURE` *without* calling `auth_publickey` — handler
+  hooks only fire on cryptographically valid signatures (POC-1
+  established the "after verification" half; POC-2 adds "invalid sigs
+  never reach the handler at all").
+- **Client-side event routing**: `check_server_key` events are client-
+  handler facts, so anything asserting on them must read the client's
+  event surface, not the server session's (POC logistics note for
+  future bake-offs).
+
+## OQ seeds carried forward
+
+1. **alkvault `decrypt` returns plain `String`** — a zeroization gap at
+   the seam (the vault's own key material is zeroized; the decrypted
+   *plaintext* is not until the adapter wraps it). Candidate fixes:
+   alkvault returns `Zeroizing<String>`, or the alkssh ADR documents
+   the adapter-side `Zeroizing` obligation as the family convention.
+   POC-2's adapter shape does the latter.
+2. **Key-record schema**: alkssh's vault-backed key record should treat
+   `EncryptedData` as opaque and never interpret `key_version` itself
+   (the vault owns version semantics; `InvalidPath` for unknown
+   versions is the fail-closed behavior). Rotation (at-rest version
+   bump, key constant) is a storage-side operation — proven to
+   preserve key identity.
+3. **One vault, many roles?** The POC held client dial keys *and* the
+   server host key in one vault (one mnemonic). Whether the assembly
+   layer should run separate vault instances per role (client vs
+   server custody) is an assembly policy question, not an alkssh
+   design constraint — recorded so phase 1 doesn't bake it in.
+4. **Never persist decrypted plaintext**: the vanilla-tool check
+   wrote a decrypted key to a 0600 temp file (then zeroized and
+   deleted it) — that is a test-only demonstration of the byte format.
+   Adapters do not spill plaintext to disk; the vanilla-compatible PEM
+   is handed to processes only via the assembly layer if ever.
+
+## Reference run
+
+Full output of a clean run at findings time: **39 checks, 5 groups**
+(at-rest shape 11, dial lifecycle 8, failure matrix 10, Signer seam 3,
+host key + vanilla 7) — all PASS. `overall=true`, exit 0.
+
+## References
+
+- `pocs.md` §POC-2 (plan), `auth-bridge.md` §The client side (design —
+  updated with a pointer to these verdicts), `russh.md`
+  (`client::Handle` authenticate methods — updated with the
+  Signer-verdict pointer)
+- POC-1: `poc-1-auth-bridge.md` + `/workspace/alkssh-poc1`
+  (server loopback shape reused)
+- russh 0.64.1, alkvault 0.1.0, ssh-key 0.7.0-rc.11 (crates.io,
+  ring backend), OpenSSH 9.6p1 (vanilla tooling)

docs/research/russh.md

@@ -146,7 +146,12 @@ variants (`authenticate_publickey_with(user, PublicKey, hash_alg,
 storage model (held keys are bearer-token-class secrets decrypted at
 use — see `auth-bridge.md`) it is not presumed to be the vault
 integration shape, but it exists and POC-2 evaluates it against the
-simple decrypt-and-decode flow), `authenticate_gssapi_with_mic`,
+simple decrypt-and-decode flow; POC-2 verdict: dismissed — no
+provided `impl Signer` exists anywhere in 0.64.1 (the doc's "SSH
+agent" implementation claim is stale), a custom signer must
+replicate the splice convention (`to_sign || string{sig}`), and an
+in-process vault gains nothing over one-time decode at dial —
+see `poc-2-vault-keys.md` §4), `authenticate_gssapi_with_mic`,
 `best_supported_rsa_hash` (waits ≤1s for EXT_INFO `server-sig-algs`).
 
 Channel opening: only the four typed opens exist —