alkssh/docs/research/reverse-runner.md

Reverse runner — client-direction pattern extraction (alkminer)

Status: phase 0 post-POC extraction. Reading aid only — concepts and wire shapes, never code (the source is not open-sourced and stays that way; per repo license hygiene, reference code is for reading and facts). Source: /workspace/@alkminer/alkminer — specifically crates/alkminer-host/src/remote/{ssh,exec,forward,manager,sftp}.rs and src/oai/mod.rs. Also old (russh 0.62 era) and POC-grade; its anti-patterns are as instructive as its patterns, and are named below as anti-patterns, not precedents.

The direction

The reverse-runner pattern inverts the typical runner flow: the hub/coordinator is the SSH client and dials out to vanilla sshd on rented containers (vast.ai, runpod — any box that exposes SSH). The runner exposes nothing but the wire; the hub is the active party and pushes all work and all channel opens.

Security shape (why this beat the alternatives in practice):

• No inbound surface on runners — only sshd with one key.
• Secrets never leave the hub — the pattern's killer use was an OAI-compatible proxy served by the hub (oai/mod.rs, alkminer ADR-016): the instance points its LLM tooling at a localhost URL that rides the tunnel to the hub's proxy, which injects the real provider key. The runner authenticates to the proxy with a scoped dev token — least privilege, no upstream-secret custody.
• Reachback both ways, deliberately: hub→runner services via -L, runner→hub services via -R (the two legs below were both used, sometimes chained: instance port ‑L→ hub port ‑L→ dev workstation).

Per-connection surface (what the hub actually drove)

One authenticated SSH connection per instance carried:

Surface	alkminer use	alkssh client-direction translation
blocking exec	provisioning commands, health checks	session channel open + exec(); alktty pipe-mode seam when the runner is family-built
background jobs (start_job/job_status/job_logs/stop_job)	mining worker lifecycle; pid files on the runner, job rows in hub-side storage	assembly-level job supervision over exec channels; the alkminer pid-file approach is the ad-hoc analog of POC-3 §D's pipe-mode signal gap (alktty exit-chunk signal identity OQ)
SFTP	binary deploy to the instance	future alkfs client half — real consumer evidence that the sftp feature's deferred implementation needs both directions (alkforge CI artifact deploy is the consumer)
-L client-side tunnel	opencode dev tunnel (instance:4096 → hub)	client opens direct-tcpip per accepted local connection and pumps (POC-1's into_stream plumbing, reversed)
-R client-side tunnel	hub-service reach tunnel (instance → hub's HTTP/OAI proxy)	tcpip_forward global request (returns the bound port — no mpsc sink param, matching russh.md's client inventory) then accept the server-initiated forwarded-tcpip opens and relay each to the hub-local target

Client-direction facts that phase 1 designs from

1. The acceptor row is unavoidable and its disposition is ours. russh's client handler receives server_channel_open_forwarded_tcpip for -R data. In alkminer's 0.62-era code (and unchanged in kind at 0.64.1), an accepted forwarded channel with no handler installed is accept-and-drop — a fail-open-ish default. alkssh's client contract: only channel types/features the assembly registered are served; otherwise decline cleanly. This is the client side of the feature-graph symmetry rule; unlike the server auth defaults (which 0.64.1 already has rejecting), the client acceptor default is ours to enforce.
2. The routing sink installs post-connect. alkminer's shape: the handler exposes a late-bound handler slot (install_forward_sink) set after connect by the consumer that owns the forward. The family translation is an assembly-injected acceptor-handlers registry (per channel type / per feature), not a static config — but the late-install dynamic is real: consumers turn capabilities on/off over a live connection.
3. Keepalives are load-bearing on cloud runners. The hub needed keepalive_interval 15 s + keepalive_max 3 (OpenSSH ServerAliveInterval=15/ServerAliveCountMax=3 parity) — with russh's default (no keepalives), dead NAT'd connections sat unnoticed until a health-check exec failed. Evidence for the client-side bounded-resources defaults (the server side has symmetric knobs; cloud edges need both).
4. Reconnect-and-restore semantics. Tunnel ports were persisted (hub storage) and re-bound identically on reconnect; forwards were tracked per connection with idempotent stop handles. This is assembly policy (session/forward lifecycle), not wire contract — kept in view so phase 1 doesn't accidentally bake reconnect policy into the crate.
5. Host-key pinning anti-pattern. alkminer's check_server_key returned accept-all (TOFU-without-the-F, 0.62 default posture overridden to the permissive extreme for rental fleets). The family rule: no trust decisions inside the crate — default fail-closed, trust via an assembly-provided policy seam. The lesson is sharper than "pin everything", though: for ephemeral fleets strict TOFU is unusable anyway (fresh container each rent → fresh host key; the churn is exactly why accept-all won), so the real security anchor there is hub-held key material (per-instance dial keys make a captured hub credential worthless) plus what rides the tunnels; host pinning is the right control only where anchors can exist (family-built runners, whose host key the assembly mints and therefore knows beforehand). The fleet churn is the reason the two-case policy seam (OQ-05) is the correct shape rather than "pin everything".
6. Per-runner key registration. Keys were minted per instance and registered hub-side (their provisioning flow, including a real gitea client for dev workspaces) — the deployment precedent for alkforge's per-runner IdentityProvider rows.

Not carried

• No code extraction in either direction (concepts/wire shapes only).
• The job-supervision storage model (hub-side SQLite job ledger, pid files on the runner) belongs to the assembly layer in a family deployment; the crate's surface is channels and tunnels.
• No per-target scoping of proxied traffic existed (dev token proved "a valid dev instance", nothing more) — a real least-privilege gap the family model fixes via per-identity ACLs on the service side.

References

• pocs.md §Client-side scope (the scope item this doc grounds)
• mapping-table.md (server-direction rows; POC-3 stage C is the server side of the same bridge legs), poc-1-auth-bridge.md §7 (the channel↔stream plumbing both leg styles reuse), poc-3-mapping-bakeoff.md §D.3 (client-request mechanics)
• auth-bridge.md §Host keys and check_server_key (the pinning rule this doc's anti-pattern motivates)
• alkminer ADR-016 (the OAI proxy pattern — secrets stay at the hub)