# Reverse runner — client-direction pattern extraction (alkminer)

Status: phase 0 post-POC extraction. **Reading aid only — concepts and
wire shapes, never code** (the source is not open-sourced and stays
that way; per repo license hygiene, reference code is for reading and
facts). Source:
`/workspace/@alkminer/alkminer` — specifically
`crates/alkminer-host/src/remote/{ssh,exec,forward,manager,sftp}.rs`
and `src/oai/mod.rs`. Also old (russh 0.62 era) and POC-grade; its
anti-patterns are as instructive as its patterns, and are named below
as anti-patterns, not precedents.

## The direction

The reverse-runner pattern inverts the typical runner flow: the
**hub/coordinator is the SSH client** and dials out to vanilla `sshd`
on rented containers (vast.ai, runpod — any box that exposes SSH). The
runner exposes nothing but the wire; the hub is the active party and
pushes all work and all channel opens.

Security shape (why this beat the alternatives in practice):

- **No inbound surface on runners** — only `sshd` with one key.
- **Secrets never leave the hub** — the pattern's killer use was an
  OAI-compatible proxy served *by the hub* (`oai/mod.rs`, alkminer
  ADR-016): the instance points its LLM tooling at a localhost URL
  that rides the tunnel to the hub's proxy, which injects the real
  provider key. The runner authenticates to the proxy with a scoped
  dev token — least privilege, no upstream-secret custody.
- **Reachback both ways, deliberately**: hub→runner services via `-L`,
  runner→hub services via `-R` (the two legs below were both used,
  sometimes chained: instance port ‑L→ hub port ‑L→ dev workstation).

## Per-connection surface (what the hub actually drove)

One authenticated SSH connection per instance carried:

| Surface | alkminer use | alkssh client-direction translation |
|---|---|---|
| blocking `exec` | provisioning commands, health checks | session channel open + `exec()`; alktty pipe-mode seam when the runner is family-built |
| background jobs (`start_job`/`job_status`/`job_logs`/`stop_job`) | mining worker lifecycle; pid files on the runner, job rows in hub-side storage | assembly-level job supervision over exec channels; the alkminer pid-file approach is the ad-hoc analog of POC-3 §D's pipe-mode signal gap (alktty exit-chunk signal identity OQ) |
| SFTP | binary deploy to the instance | **future alkfs client half** — real consumer evidence that the `sftp` feature's deferred implementation needs both directions (alkforge CI artifact deploy is the consumer) |
| `-L` client-side tunnel | opencode dev tunnel (instance:4096 → hub) | client opens `direct-tcpip` per accepted local connection and pumps (POC-1's `into_stream` plumbing, reversed) |
| `-R` client-side tunnel | hub-service reach tunnel (instance → hub's HTTP/OAI proxy) | `tcpip_forward` global request (returns the bound port — no mpsc sink param, matching `russh.md`'s client inventory) then **accept the server-initiated `forwarded-tcpip` opens** and relay each to the hub-local target |

## Client-direction facts that phase 1 designs from

1. **The acceptor row is unavoidable and its disposition is ours.** russh's
   client handler receives `server_channel_open_forwarded_tcpip` for
   `-R` data. In alkminer's 0.62-era code (and unchanged in kind at
   0.64.1), an accepted forwarded channel with no handler installed is
   **accept-and-drop** — a fail-open-ish default. alkssh's client
   contract: only channel types/features the assembly registered are
   served; otherwise decline cleanly. This is the client side of the
   feature-graph symmetry rule; unlike the server auth defaults (which
   0.64.1 already has rejecting), the client acceptor default is
   *ours to enforce*.
2. **The routing sink installs post-connect.** alkminer's shape: the
   handler exposes a late-bound handler slot
   (`install_forward_sink`) set after `connect` by the consumer that
   owns the forward. The family translation is an assembly-injected
   acceptor-handlers registry (per channel type / per feature), not a
   static config — but the *late-install* dynamic is real: consumers
   turn capabilities on/off over a live connection.
3. **Keepalives are load-bearing on cloud runners.** The hub needed
   `keepalive_interval` 15 s + `keepalive_max` 3 (OpenSSH
   `ServerAliveInterval=15`/`ServerAliveCountMax=3` parity) — with
   russh's default (no keepalives), dead NAT'd connections sat
   unnoticed until a health-check exec failed. Evidence for the
   client-side bounded-resources defaults (the server side has
   symmetric knobs; cloud edges need both).
4. **Reconnect-and-restore semantics.** Tunnel ports were persisted
   (hub storage) and re-bound identically on reconnect; forwards were
   tracked per connection with idempotent stop handles. This is
   assembly policy (session/forward lifecycle), not wire contract —
   kept in view so phase 1 doesn't accidentally bake reconnect policy
   into the crate.
5. **Host-key pinning anti-pattern.** alkminer's `check_server_key`
   returned accept-all (TOFU-without-the-F, 0.62 default posture
   overridden to the permissive extreme for rental fleets). The family
   rule (no TOFU; expected host fingerprints pinned at the assembly
   layer) is right, and this code is the working exhibit of why. The
   *convenience* need is real (fleets churn); the resolution is
   pinning policy at the assembly, not accept-all in the crate.
6. **Per-runner key registration.** Keys were minted per instance and
   registered hub-side (their provisioning flow, including a real
   gitea client for dev workspaces) — the deployment precedent for
   alkforge's per-runner `IdentityProvider` rows.

## Not carried

- No code extraction in either direction (concepts/wire shapes only).
- The job-supervision storage model (hub-side SQLite job ledger, pid
  files on the runner) belongs to the assembly layer in a family
  deployment; the crate's surface is channels and tunnels.
- No per-target scoping of proxied traffic existed (dev token proved
  "a valid dev instance", nothing more) — a real least-privilege gap
  the family model fixes via per-identity ACLs on the service side.

## References

- `pocs.md` §Client-side scope (the scope item this doc grounds)
- `mapping-table.md` (server-direction rows; POC-3 stage C is the
  server side of the same bridge legs), `poc-1-auth-bridge.md` §7
  (the channel↔stream plumbing both leg styles reuse),
  `poc-3-mapping-bakeoff.md` §D.3 (client-request mechanics)
- `auth-bridge.md` §Host keys and `check_server_key` (the pinning
  rule this doc's anti-pattern motivates)
- alkminer ADR-016 (the OAI proxy pattern — secrets stay at the hub)