docs(research): phase 0 closed — pocs.md refreshed, reverse-runner pattern extracted

- pocs.md: marked complete; corrected the stale 0.5x-era accept-defaults
  hypothesis wording (0.64.1: only auth_publickey_offered defaults to
  accept), stage D recorded as ran-in-full, completion gate updated with
  per-question resolution status
- added client-side scope section: no dedicated POC needed (client API
  exercised throughout POC-1/POC-3) + the acceptor-default fact (russh
  client accept-and-drop for unhandled forwarded opens is ours to fix)
- new reverse-runner.md: alkminer pattern extraction (concepts only,
  never code) — hub-dials-runner surface inventory, both tunnel legs,
  the OAI-proxy secret-hygiene shape, and the client-direction lessons
  (keepalive defaults, reconnect-restore, acceptor disposition)
- vision.md: scope-question resolution addendum + whitespace typo fix

docs/research/pocs.md

@@ -1,8 +1,11 @@
 # POCs — phase 0 validation plan
 
-Status: Draft (phase 0 research). Deliverables of each POC are
-findings docs (`docs/research/poc-N-*.md` when run); POC source is
-disposable and never merges to main. Mode for all: **standalone**
+Status: **Complete** (phase 0). All three POCs ran against the current
+published russh (0.64.1) and passed; the findings docs
+(`poc-1-auth-bridge.md`, `poc-2-vault-keys.md`,
+`poc-3-mapping-bakeoff.md`) are canonical and supersede the planned
+hypothesis wording below wherever they record corrections. POC source
+is disposable and never merges to main. Mode for all: **standalone**
 (repos have no code to build on yet) — scratch cargo projects at
 `/workspace/<poc-name>` unless a POC explicitly builds on repo code
 (then branch mode per `docs/sdd_process.md`).
@@ -38,9 +41,11 @@ keys; russh client available as a secondary client for
 - `auth_publickey_offered` disposition (accept-and-let-decide) behaves
   with OpenSSH's probe-then-sign flow.
 - Decline matrix: `auth_none`/`password`/`keyboard-interactive`
-  declined (default impls are *accept* — every default must be
-  overridden and proven); `Config::methods` advertises `publickey`
-  only.
+   declined (0.64.1 correction: the *default impls already reject* at
+   this russh — only `auth_publickey_offered` defaults to accept;
+   `poc-1-auth-bridge.md` §3. The family contract still requires
+   implementing every hook explicitly — the pin is a one-way door);
+   `Config::methods` advertises `publickey` only.
 - `ChannelOpenHandle` drop = auto-reject semantics; unfeatured
   channel-type declines observable by vanilla `ssh`
   (`ssh host` session-open refused, `-L` refused, clean diagnostics).
@@ -127,11 +132,14 @@ forms). russh server loopback + vanilla clients; per stage:
   real `ChannelClient` vs local-dial stand-in). Whichever route the
   POC exercises informs the phase 1 ADR on the presence/absence of a
   serve-over-stream entry in alktunnels.
-- **Stage D — session/pty mapping**, last and optional (a real PTY
-  backend needs alktty's `local` feature; the wire-level questions are
-  already largely answered by A/B/C): shell over
-  `drive_session` with the locally-constructed `NegotiateRequest`,
-  window-change and signal mapping.
+- **Stage D — session/pty mapping**, final stage (ran in full): shell,
+  exec, window-change and signal mapping against a real PTY
+  (`LocalTtyBackend`, alktty `local`). The "optional" hedge in the
+  original plan was unnecessary — stage D completed 13/13 and produced
+  the load-bearing session-row corrections: the alktty bridge is a
+  **translation bridge** (chunk framing both directions, not a byte
+  pump) and the alktty exit chunk carries no signal identity
+  (`poc-3-mapping-bakeoff.md` §Stage D).
 
 **Validates:** the one-seam-per-mapping hypothesis; the parse-then-
 route dispatch; stderr semantics in pipe vs PTY mode; EOF/close/signal
@@ -142,18 +150,47 @@ channels reassembly cap on the bridged route).
 fixups recorded; stage D may surface PTY-merge stderr surprises that
 reshape the request-level mapping rows.
 
+## Client-side scope (post-POC addition)
+
+The three POCs are server-door-centric by design, but the crate has a
+**client direction** of equal scope: a family app dials an *upstream*
+vanilla-SSH server and consumes its surface (exec/pty channels,
+`direct-tcpip` opens, the `tcpip_forward` + accepted-`forwarded-tcpip`
+listener flow). No dedicated POC was needed: the POCs exercised the
+client API throughout (POC-1's russh-client auth leg, POC-3 §D.3's
+client-side request mechanics — `Channel::exec()`, the `wait()`-drain
+trap, fire-and-forget `window_change`/`signal` before `into_stream()`).
+
+Two client-side facts phase 1 designs from:
+
+1. **The acceptor rule.** The only server-initiated channel type in the
+   pattern is `forwarded-tcpip`, and russh's client default for an
+   unhandled forwarded open is *accept-and-drop* — which is not the
+   family fail-closed posture. The client-side accepted-open
+   disposition (accept into a registered handler vs decline) is
+   alkssh's contract to define.
+2. **A real consumer pattern exists**: the alkminer hub dialing rented
+   GPU containers (exec + job management, SFTP binary deploy, `-L` and
+   `-R` tunnel legs in both directions, keepalives tuned for NAT'd
+   hosts). Extracted (concepts only, never code) in
+   `reverse-runner.md`.
+
 ## Sequencing and gates
 
 1. POC-1 first — everything downstream presumes its auth plumbing.
 2. POC-2 after POC-1 (reuses POC-1's server loopback).
 3. POC-3 last; stage A gates phase 1's mapping ADR work — it is the
    path alkforge phase 0 consumes first.
-4. Phase 0 completes when the findings docs exist and the OQ list in
-   `vision.md` §scope questions is either answered by findings or
-   carried into phase 1's `open-questions.md` as seeded OQs. POC-1/2/3
-   findings docs now exist; scope question 5 (mapping-table details)
-   is answered per `poc-3-mapping-bakeoff.md` §hypothesis verdicts,
-   and the POC-seeded OQs are listed in each findings doc.
+4. Phase 0 completed when the findings docs existed and the OQ list in
+    `vision.md` §scope questions was either answered by findings or
+    carried into phase 1's `open-questions.md` as seeded OQs. Done:
+    POC-1/2/3 findings docs exist; scope question 5 (mapping-table
+    details) is answered per `poc-3-mapping-bakeoff.md` §hypothesis
+    verdicts; scope question 4 (auth dispositions + client-side vault
+    keys) is answered per POC-1/POC-2; scope question 2 (russh pin
+    facts) is verified; questions 1 (the `alk/ssh` topology) and 3
+    (feature naming) are decision territory carried as phase 1 OQs —
+    the seeded OQ list is in each findings doc.
 
 ## References
 

docs/research/reverse-runner.md

@@ -0,0 +1,109 @@
+# Reverse runner — client-direction pattern extraction (alkminer)
+
+Status: phase 0 post-POC extraction. **Reading aid only — concepts and
+wire shapes, never code** (the source is not open-sourced and stays
+that way; per repo license hygiene, reference code is for reading and
+facts). Source:
+`/workspace/@alkminer/alkminer` — specifically
+`crates/alkminer-host/src/remote/{ssh,exec,forward,manager,sftp}.rs`
+and `src/oai/mod.rs`. Also old (russh 0.62 era) and POC-grade; its
+anti-patterns are as instructive as its patterns, and are named below
+as anti-patterns, not precedents.
+
+## The direction
+
+The reverse-runner pattern inverts the typical runner flow: the
+**hub/coordinator is the SSH client** and dials out to vanilla `sshd`
+on rented containers (vast.ai, runpod — any box that exposes SSH). The
+runner exposes nothing but the wire; the hub is the active party and
+pushes all work and all channel opens.
+
+Security shape (why this beat the alternatives in practice):
+
+- **No inbound surface on runners** — only `sshd` with one key.
+- **Secrets never leave the hub** — the pattern's killer use was an
+  OAI-compatible proxy served *by the hub* (`oai/mod.rs`, alkminer
+  ADR-016): the instance points its LLM tooling at a localhost URL
+  that rides the tunnel to the hub's proxy, which injects the real
+  provider key. The runner authenticates to the proxy with a scoped
+  dev token — least privilege, no upstream-secret custody.
+- **Reachback both ways, deliberately**: hub→runner services via `-L`,
+  runner→hub services via `-R` (the two legs below were both used,
+  sometimes chained: instance port ‑L→ hub port ‑L→ dev workstation).
+
+## Per-connection surface (what the hub actually drove)
+
+One authenticated SSH connection per instance carried:
+
+| Surface | alkminer use | alkssh client-direction translation |
+|---|---|---|
+| blocking `exec` | provisioning commands, health checks | session channel open + `exec()`; alktty pipe-mode seam when the runner is family-built |
+| background jobs (`start_job`/`job_status`/`job_logs`/`stop_job`) | mining worker lifecycle; pid files on the runner, job rows in hub-side storage | assembly-level job supervision over exec channels; the alkminer pid-file approach is the ad-hoc analog of POC-3 §D's pipe-mode signal gap (alktty exit-chunk signal identity OQ) |
+| SFTP | binary deploy to the instance | **future alkfs client half** — real consumer evidence that the `sftp` feature's deferred implementation needs both directions (alkforge CI artifact deploy is the consumer) |
+| `-L` client-side tunnel | opencode dev tunnel (instance:4096 → hub) | client opens `direct-tcpip` per accepted local connection and pumps (POC-1's `into_stream` plumbing, reversed) |
+| `-R` client-side tunnel | hub-service reach tunnel (instance → hub's HTTP/OAI proxy) | `tcpip_forward` global request (returns the bound port — no mpsc sink param, matching `russh.md`'s client inventory) then **accept the server-initiated `forwarded-tcpip` opens** and relay each to the hub-local target |
+
+## Client-direction facts that phase 1 designs from
+
+1. **The acceptor row is unavoidable and its disposition is ours.** russh's
+   client handler receives `server_channel_open_forwarded_tcpip` for
+   `-R` data. In alkminer's 0.62-era code (and unchanged in kind at
+   0.64.1), an accepted forwarded channel with no handler installed is
+   **accept-and-drop** — a fail-open-ish default. alkssh's client
+   contract: only channel types/features the assembly registered are
+   served; otherwise decline cleanly. This is the client side of the
+   feature-graph symmetry rule; unlike the server auth defaults (which
+   0.64.1 already has rejecting), the client acceptor default is
+   *ours to enforce*.
+2. **The routing sink installs post-connect.** alkminer's shape: the
+   handler exposes a late-bound handler slot
+   (`install_forward_sink`) set after `connect` by the consumer that
+   owns the forward. The family translation is an assembly-injected
+   acceptor-handlers registry (per channel type / per feature), not a
+   static config — but the *late-install* dynamic is real: consumers
+   turn capabilities on/off over a live connection.
+3. **Keepalives are load-bearing on cloud runners.** The hub needed
+   `keepalive_interval` 15 s + `keepalive_max` 3 (OpenSSH
+   `ServerAliveInterval=15`/`ServerAliveCountMax=3` parity) — with
+   russh's default (no keepalives), dead NAT'd connections sat
+   unnoticed until a health-check exec failed. Evidence for the
+   client-side bounded-resources defaults (the server side has
+   symmetric knobs; cloud edges need both).
+4. **Reconnect-and-restore semantics.** Tunnel ports were persisted
+   (hub storage) and re-bound identically on reconnect; forwards were
+   tracked per connection with idempotent stop handles. This is
+   assembly policy (session/forward lifecycle), not wire contract —
+   kept in view so phase 1 doesn't accidentally bake reconnect policy
+   into the crate.
+5. **Host-key pinning anti-pattern.** alkminer's `check_server_key`
+   returned accept-all (TOFU-without-the-F, 0.62 default posture
+   overridden to the permissive extreme for rental fleets). The family
+   rule (no TOFU; expected host fingerprints pinned at the assembly
+   layer) is right, and this code is the working exhibit of why. The
+   *convenience* need is real (fleets churn); the resolution is
+   pinning policy at the assembly, not accept-all in the crate.
+6. **Per-runner key registration.** Keys were minted per instance and
+   registered hub-side (their provisioning flow, including a real
+   gitea client for dev workspaces) — the deployment precedent for
+   alkforge's per-runner `IdentityProvider` rows.
+
+## Not carried
+
+- No code extraction in either direction (concepts/wire shapes only).
+- The job-supervision storage model (hub-side SQLite job ledger, pid
+  files on the runner) belongs to the assembly layer in a family
+  deployment; the crate's surface is channels and tunnels.
+- No per-target scoping of proxied traffic existed (dev token proved
+  "a valid dev instance", nothing more) — a real least-privilege gap
+  the family model fixes via per-identity ACLs on the service side.
+
+## References
+
+- `pocs.md` §Client-side scope (the scope item this doc grounds)
+- `mapping-table.md` (server-direction rows; POC-3 stage C is the
+  server side of the same bridge legs), `poc-1-auth-bridge.md` §7
+  (the channel↔stream plumbing both leg styles reuse),
+  `poc-3-mapping-bakeoff.md` §D.3 (client-request mechanics)
+- `auth-bridge.md` §Host keys and `check_server_key` (the pinning
+  rule this doc's anti-pattern motivates)
+- alkminer ADR-016 (the OAI proxy pattern — secrets stay at the hub)

docs/research/vision.md

@@ -50,7 +50,7 @@ not protocol invention.
    exactly one point per mapping. No channel-type knowledge flows
    upstream into the protocol crates.
 5. **Identity is the family's, not SSH's.** SSH user names, `~/.ssh/`
-   conventions, and password auth    are not the alk family's identity
+   conventions, and password auth are not the alk family's identity
    model. A verified key's fingerprint → `Identity` via alkcall's
    `IdentityProvider` is the only resolution path; secrets an alkssh
    *client* holds come from alkvault-backed sources — vault-encrypted
@@ -102,6 +102,15 @@ phase 1's `open-questions.md`:
    the family seam, the parameter mapping, and the decline contract
    (see `mapping-table.md`); POC validation per `pocs.md`.
 
+Resolution status (post-POC): question 4 is answered by
+POC-1/POC-2 evidence, question 5 by POC-3 (three mapping-row
+corrections recorded), and question 2's *facts* are verified (the pin
+decision itself is phase 1 ADR territory). Questions 1 (the `alk/ssh`
+topology) and 3 (feature naming) are phase 1 decisions, carried as
+seeded OQs / ADR territory per `pocs.md` §Sequencing and gates. A
+client-direction consumer pattern (hub-dials-runner) is extracted in
+`reverse-runner.md` and shapes the phase 1 client surface.
+
 ## References
 
 - family ADR-003 (`/workspace/@alkdev/alkfamily/docs/architecture/decisions/003-alkssh-russh-adapter.md`)