--- id: w5-e2e-server name: "Wave 5: real-tool e2e, server direction (OpenSSH / git / curl against the door)" status: completed depends_on: [review-wave-4] scope: moderate risk: medium impact: phase level: implementation --- ## Description The server-direction real-tool e2e, with the POC-3 gates re-landed against the shipped crate (`docs/plans/implementation.md` §Wave 5, the real-tool e2e bullet's server half). The wave-3 smokes proved one clone, one `-L`, one pty session; this task lands the **full** server-direction e2e set in a dedicated home (decide + record the placement, where a new `tests/e2e_server.rs` binary reusing `tests/common` is the recommended posture, keeping the wave-3/4 mapping binaries single-writer for `w5-decline-matrix`), under the established check-gated **skipped-with-reason** posture (never silently green): - **OpenSSH against the door**: pty/shell interactive session (the wave-3 smoke's composition, re-homed or extended per the placement decision), plain exec (`ssh host "cmd"` through the exec wrapper), `-L` local forward through the tunnel mapping, **`-R` remote forward** (the `w3-tunnel-listen` flow: listen request, bound port, the server-initiated `forwarded-tcpip` arrival pumped through the fabric, the leg the in-process smokes covered only through the alkcall fabric double), and **`-D`**: curl through the SOCKS proxy with the door as the server (the `direct-tcpip` leg end to end, POC-3 correction 2's posture). - **Decline observability**: a declined open/session surfaces a clean diagnostic to the real client (POC-1 §5 posture: channel-scoped decline, auth intact, no hangs): an unfeatured channel type from real `ssh -L`/`-R`/pty attempts in a feature-limited build, the sftp subsystem decline from a real `sftp`/`ssh -s` attempt, the unknown-repo collapsed generic denial from a real `git clone` of a nonexistent repo. - **git 2.43+ through the git mapping**: clone (re-land) **and push** (the wave-3 smoke was clone-only; receive-pack through the real client is the POC-3 §B gate re-landed). - **The harness posture**: real processes (`ssh`/`git`/`curl`), the adapter mounted on a real TCP listener over the in-memory doubles (the existing smoke composition); every skip reasoned (tool absent / version short), every finding fixed in-task or booked for the gate, never absorbed silently. ## Acceptance Criteria - [ ] The e2e home decided + recorded (new binary recommended); the check-gated skipped-with-reason posture held for every row - [ ] OpenSSH rows green (or reasoned-skipped): pty/shell, exec, `-L`, `-R` (listen + arrival + data leg), `-D` + curl through the proxy - [ ] Decline-observability rows green: unfeatured declines from a real client (clean diagnostic, no hang), the sftp decline, the unknown-repo generic denial - [ ] git clone **and** push through the git mapping green (or reasoned-skipped) - [ ] Every finding fixed in-task or booked for the gate (recorded) - [ ] `cargo test` (+ the builds the new binary rides), clippy ×2 (`-D warnings`), fmt, `cargo doc --no-deps` green ## References - docs/plans/implementation.md §Wave 5 (the real-tool e2e bullet) - docs/architecture/mapping.md (the rows the real tools exercise), docs/research/poc-3-mapping-bakeoff.md (stages A–D, the gates re-landed), docs/research/poc-1-auth-bridge.md §5 (decline observability) - tests/git_mapping.rs / tests/tty_mapping.rs / tests/tunnel_mapping.rs (the wave-3 smoke compositions this task re-lands/extends), tests/common/ (the doubles + harness) ## Notes **The e2e home decided**: `tests/e2e_server.rs` (the new binary), reusing `tests/common`, so the mapping binaries stay single-writer per the decomposition's file map; this task edits only its own binary. Its fixtures live in-file: a `support` module (tool gates, the fixture-key/option builders, the registered identity provider, the port allocator, the real-TCP mount) and a tunnel-gated `fabric` module (the trimmed leg-1/leg-2 fabric double, TCP-only and happy-path, with the decline and collision machinery stays the mapping binaries'). Everything is `#[cfg(unix)]` with the body feature-gated per row group; no src/ changes were needed; every observed issue was e2e-graph local. **Row inventory** (all check-gated, **skipped-with-reason** when a tool is absent / short, never silently green; each row's real-process interaction is deadline-bounded and asserts the door's decline event through the capture's deadline-bounded poll): | Row | Build(s) | What it pins | |---|---|---| | `git_rows::real_git_clones_the_empty_repo_through_the_door` | git (+all) | the wave-3 clone smoke re-landed: real `git clone ssh://…` → unborn-HEAD advertisement, normalized resolved id | | `git_rows::real_git_push_creates_the_ref_through_the_door` | git (+all) | **push** (the wave-3 smoke's gap): receive-pack through the real client, rev parity (`refs/heads/main` new-oid == local `rev-parse HEAD`), CAS-application row | | `session_rows::real_openssh_pty_shell_session_serves_through_the_door` | session (+all) | the wave-3 pty smoke re-landed + extended: the `echo` marker rides the real PTY round trip, exit-status(0) before close | | `session_rows::real_openssh_pipe_exec_separates_stdout_and_stderr_through_the_door` | session (+all) | `ssh host "cmd"` through the exec wrapper (`wrapper ++ [cmd]`): stdout rides SSH data, stderr rides `extended_data(1)`, POC-3 §D.1's pipe split end to end | | `tunnel_rows::real_ssh_local_forward_serves_the_direct_tcpip_leg` | tunnel (+all) | `-L` smoke re-landed: route-1 bridge + real `tcp_dial`, body parity | | `tunnel_rows::real_ssh_remote_forward_serves_the_arrival_and_data_legs` | tunnel (+all) | `-R`: listen, fabric-bound port, the server-initiated `forwarded-tcpip` arrival on the real wire, the client-dialed data leg, the leg the in-process smokes covered only through the fabric double | | `tunnel_rows::real_ssh_dynamic_forward_serves_curl_through_the_direct_tcpip_leg` | tunnel (+all) | **`-D`**: curl `--socks5-hostname` through vanilla OpenSSH's client-side SOCKS → the door's `direct-tcpip` leg (POC-3 correction 2 end to end); two requests pin the per-connection channel opens | | `unfeatured_session_decline::real_openssh_pty_attempt_declines_cleanly_in_a_feature_limited_build` | not-session (none/git/tunnel/sftp) | real `ssh -tt … echo` → nonzero exit, clean diagnostic, no hang; the decline event pins the build's class (`pty_request` "unfeatured" request-decline in the git build, `session` "unfeatured" open-decline otherwise) | | `unfeatured_tunnel_declines::real_openssh_remote_forward_attempt_declines_cleanly` | not-tunnel | real `ssh -N -R -o ExitOnForwardFailure=yes` → exit + OpenSSH's `forwarding failed` diagnostic; `tcpip_forward` global-request decline event | | `unfeatured_tunnel_declines::real_openssh_local_forward_connections_decline_per_connection` | not-tunnel | real `ssh -N -L` + repeated curls: every connection fails promptly (exit 7-class, no hang); `direct-tcpip` open-decline event | | `subsystem_decline::real_sftp_subsystem_attempt_declines_with_reason` | any(session,git) | the OQ-13 placeholder from a real `sftp` client: nonzero exit, clean diagnostic; `subsystem_request` request-decline event (`unimplemented`) | | `unknown_repo_decline::real_git_clone_of_an_unknown_repo_collapses_to_the_generic_denial` | git (+all) | real `git clone` of a nonexistent repo → exit 128, the collapsed `access denied` band naming the normalized id, **no existence oracle** | **Findings fixed in-task** (no src/ changes; all e2e-graph local): 1. **Parallel-row flake (the shared key workdir)**: the three tunnel rows initially shared one `write_key` workdir, so concurrent rows raced on the same key file and one row's cleanup deleted the dir under the others (observed as refused forwards / curl exit 7). Fixed: per-row workdirs (`tunnel-l` / `tunnel-r` / `tunnel-d`). 2. **Parallel-row flake (port collisions)**: the probe-and-drop ephemeral port picker let parallel rows (and under load, re-entries) collide on named binds. Fixed: an in-process monotonic allocator over the 2000–16999 window (below the kernel's ephemeral range, so `bind(0)` can never steal one), probe-verified per allocation. 3. **Fabric-registration race**: the leg-2 fabric's openables register on a spawned task; an `ssh -R` listen dial ahead of registration is refused, not retried (`curl never got through` exit 7). Fixed: both fabric constructors settle on the direct/listen openables' registration before returning (`settle`, bounded poll on `OperationRegistry::registration`). 4. **Event-window flake under CI load**: the decline rows' fixed 10 s `await_event` window can expire on a loaded host. Fixed: `event_capture::captured_with` (additive, non-blocking peek) + `support::decline_event`, so the rows poll within the row deadline instead of a fixed window. ## Summary Completed 2026-10-30. The server-direction real-tool e2e landed as `tests/e2e_server.rs` (the placement decision above): 12 rows over the six build shapes: OpenSSH pty/shell + pipe exec (stderr separation), `-L`/`-R`/`-D`+curl (with curl's two-request per-connection pin), git clone **and** push (rev parity + CAS row pinned through the recording quartet), and the six decline-observability rows (unfeatured pty/`-L`/`-R` from real clients, the sftp subsystem decline, the unknown-repo collapsed generic denial, each also pinned by its door-side decline event). Every finding fixed in-task (four, listed in Notes; flake hardening only, no src/ changes). Verification: full `scripts/matrix.sh` run green: `cargo test` per build (none / git / session / tunnel / sftp / --all-features; the new binary rides each build compiling and running only its rows' builds), both clippy runs `-D warnings` clean, fmt clean, `cargo doc --no-deps` clean; plus 10 consecutive all-features e2e runs green after the flake fixes (the parallel-port + shared-workdir + fabric-settle + event-poll hardening). Skips are reasoned prints only (tool absence; on this host Git 2.43 / OpenSSH 9.6 / curl 8.5 / sftp are present, so every row ran).