alkgit/docs/research/poc3-findings.md

POC-3 findings: smart-http shape through alkhttp

Status: complete — proceed Date: 2026-09-21 Mode: branch (poc3-smart-http, per pocs.md); the POC code lives at /workspace/alkgit-poc3 (scratch crate — branch mode's exercise of the repo skeleton turned out to be pure-HttpAdapter consumption, which the published alkhttp 0.5 already serves, so no repo code changes were needed). The branch carries no code deltas and is dropped after this document merges; it exists only as the POC's declared mode. Depends on: POC-1 (pkt-line over alkcall BiStream), POC-2 (streaming pack generation)

Hypothesis (restated)

alkhttp can serve GET /<repo>/info/refs and stream a POST body (ingest without full buffering) for the smart-http services; real git clone http://… completes against it, and the request-body API's streaming-vs- buffering behavior is measured (not guessed).

Result

Confirmed, with two course corrections (both git-protocol facts, not alkhttp facts). Real git 2.43 clones, fetches, and tag-checkouts over http:// against a service whose entire wire path is:

tokio TcpStream
  -> alkcall Connection::from_bidi(stream, "http/1.1", addr)   [alkcall 0.8.0]
  -> alkhttp HttpAdapter::handle(conn, auth)                    [alkhttp 0.5.0]
  -> accept_bi() -> BiStream
  -> hyper-util auto builder (h1/h2) -> axum Router
  -> our git routes mounted via HttpAdapter::with_extra_routes
     GET  /{repo}/info/refs?service=git-upload-pack   (V2 advertisement)
     POST /{repo}/git-upload-pack                     (V2 stateless commands)
  -> request body: axum Body -> BodyDataStream -> BodyReader
     (futures-io adapter, chunk-at-a-time, no accumulation)
     -> gix-packetline StreamingPeekableIter (stateless command parse)
  -> response: pkt-lines / band-1 sideband -> bounded mpsc (8 slots)
     -> axum Body::from_stream (chunked, back-pressured)

Check	Result
GET /info/refs?service=git-upload-pack	200, correct smart prefix + V2 advertisement, application/x-git-upload-pack-advertisement, no-cache
git ls-remote http://… (V2)	all 5 refs, clean exit
git clone (V2, full handshake)	clone completes, git fsck --strict passes
git checkout v1.0	annotated tag served over http, checks out
git fetch (incremental)	ref updated, fsck clean
git clone with http.postBuffer=65536 (forces chunked/streamed POST)	clone completes, fsck clean
git clone with http.version=HTTP/2 (h2c prior knowledge)	clone completes, fsck clean
10k-commit / 60k-object clone over http	5.33 MB pack, 37 s wall (debug build), fsck clean
in-process body-streaming probe (dribbled chunks)	chunks consumed as they arrive (gaps preserved), not buffered
slow-reader pack fetch (32 KB / 50 ms ≈ 650 KB/s)	server RSS stays ~33.8 MB peak for the whole 5.33 MB pack; drops back after

The measured answer: request bodies stream, not buffer

alkhttp's request-body API streams. axum::body::Body::into_data_stream() yields BodyDataStream (an http_body-driven Stream<Item = io::Result<Bytes>>); each poll_next produces one hyper-read chunk. The POC's BodyReader (a ~40-line futures_io::AsyncRead adapter) serves reads from the current chunk's cursor and pulls the next chunk only when the cursor drains — gix-packetline's StreamingPeekableIter reads through it unmodified.

Evidence, not guesswork:

1. Dribble probe (POST /poc3/probe-upload): a client that sends 3 chunks with 150 ms gaps sees the server consume them as they arrive — the response reports chunks: 3, first_gap_ms: 149, max_gap_ms: 150. A buffering server would report all-at-once (gaps ≈ 0 at the end).
2. git's own chunked path: with http.postBuffer=65536 git forces the large-request path (chunked POST, no Content-Length), and the full clone handshake completes through the same handler. This is the shape receive-pack will see for large pushes — it works and streams.
3. Response-side back-pressure is equally real: fetching the 60k-object pack with a slow reader keeps server RSS flat — peak ≈ 33.9 MB total process (≈ 8 MB above idle) while 5.33 MB trickles out at 650 KB/s; RSS returns to baseline (~26 MB) after completion. No whole-pack buffering anywhere: pack generation (POC-2's spawn_blocking pipeline) parks on the bounded mpsc (capacity 8) whenever the HTTP client lags.

What streaming response bodies look like in axum (the POC's composition)

• Body::from_stream(ReceiverStream<Bytes>) gives a chunked response body with real back pressure. The blocking-side pack generator writes band-1 pkt-lines through an io::Write sink that does mpsc::Sender::blocking_send — the sink parks under pressure, exactly the POC-2 SidebandSink shape with the pkt-line writer swapped for channel sends.
• The single 65000-byte sideband chunk size from POC-2 carries over unchanged; each chunk becomes one Bytes item, one HTTP chunk each.
• hyper/axum add no buffering on top: the first band-1 chunk left the server ~8 ms into generation in the fast-reader measurement.

Course corrections (git smart-http protocol facts for git-protocol.md)

1. V2-over-http stateless framing: what the client actually sends and requires

Captured against git 2.43 (remote-curl.c stateless_connect + ground truth from git http-backend behind a local mirror):

• The first POST carries the cached capability dump. remote-curl pipes the entire info/refs response it received (including the # service=… smart prefix, its flush, the V2 capability lines, and their flush) ahead of the client's command in the body of the first POST. A server's command parser must skip both flush-delimited sections before looking for command=…. Subsequent POSTs (negotiation rounds) carry only the command request.
• Capabilities ride before the delimiter, command args after. command=<name>, agent=…, object-format=… precede the 0001 delim; peel/symrefs/want/have/done/ref-prefix follow it. Parsing that treats pre-delim lines as args breaks fetch (the object-format line would shadow the real args).
• Responses end with a flush, never a response-end packet. The 0002 response-end pkt that protocol V2 requires (and that fetch-pack checks via check_stateless_delimiter) is synthesized by remote-curl, written to the child process's pipe after every POST response. The HTTP response body itself must end at the flush: remote-curl's check_pktline dies on 0002 anywhere in an HTTP response body (remote-curl: unexpected response end packet). Verified against git http-backend ground truth: info/refs, ls-refs response, and fetch packfile response all end at 0000; no 0002 anywhere. This contradicts a natural reading of the protocol docs' grammar and is the POC's most load-bearing correction for alkgit-http.
• The flush is one-shot per response. A second flush pkt in one response breaks the client (clone< 0000 then "expected response end packet"). The POC's first sink version emitted the flush on every flush() call; the fix is the same idempotent-flush guard POC-2 already had (flushed flag). FromEntriesIter flushes its writer at the pack trailer, and the handler's explicit flush() is then a no-op — the guard is mandatory, not cosmetic.
• A flush-only POST is a probe, not an error. remote-curl's probe_rpc (large-request path) POSTs a 4-byte 0000 body before the real request; git's http-backend answers 200 with an empty body (upload-pack exits without output). Answering 400 breaks the whole clone: RPC failed; HTTP 400 → "expected flush after ref listing". The POC mirrors the 200-empty-body shape.

2. alkhttp fit: everything needed exists; three small frictions noted

• with_extra_routes is the whole extension surface and it works: a Router<()> with internal state (.with_state(...)) merges under the gateway's auth layer; reserved-path collision checks pass for /{repo}/info/refs and /{repo}/git-upload-pack.
• HttpAdapter is not Clone — the POC wraps it in Arc per accept task. ProtocolHandler::handle takes &self, so Arc is the natural shape; worth documenting (or #[derive(Clone)] if the internals ever allow) in the assembly layer's notes.
• Request-body streaming is by-construction (see above) but there is no body-size limit on custom routes: the gateway installs its own 2 MiB cap (GW-15) but with_extra_routes routes get hyper's unbounded stream. The git receive-pack route will need an explicit budget (bounded-resources invariant) — a phase-1 note for alkgit-http, not a blocker.
• One POC-side bug is worth recording as a transport-design lesson: an error-shaped read (EOF mid-parse) that loops with reset() spins a tokio worker at 100% — StreamingPeekableIter returns Some(Err(io)) (not None) on EOF-mid-line, so an error branch that resets and retries never terminates. The parse loop must break on error unconditionally. Encode this in the alkgit-transport request-reader type (POC-1 follow-up 2).

What the POC does NOT settle

• receive-pack (git push): not exercised. The request-streaming answer (above) plus POC-2's bundle::write finding cover its two halves, but the push protocol itself (unpack/ack/report-status over http) is phase-1 work.
• Auth: the POC serves anonymous git over alkhttp's permissive routes (the gateway's bearer layer is not applied to extra routes' handlers unless registered before it — the POC relies on the route-local permissiveness). The visible-surface = authorized-surface invariant needs AccessControl wiring at assembly time; out of POC scope by design.
• TLS: plaintext h2c/h1 over local tcp. alktls composition is Connection::from_bidi(stream, alpn) with a TlsStream — unchanged from POC-1's analysis, not re-verified here.
• h2 keep-alive interaction with long pack streams: the h2c clone worked; long-lived streaming under h2 with hyper's 30 s/10 s ping knobs (SRV-04) deserves one soak test in phase 1.

What it changes in the research docs

• alk-stack.md §"What to verify" item 2: resolved — response bodies stream with back pressure; request bodies stream (measured); smart-http V2 works end-to-end through alkhttp's custom-route surface.
• git-protocol.md: add the V2-over-http framing facts above (capability dump in first POST, flush-only responses, probe_rpc 200-empty, no 0002 on the wire).
• pocs.md: POC-3 outcome recorded; phase 0 gates all pass.

Follow-ups for phase 1 (architecture input)

1. alkgit-http route shapes: GET /{repo}/info/refs + POST /{repo}/git-upload-pack|git-receive-pack as the canonical handlers, mounted via with_extra_routes; the POC's httpservice.rs is the seed. Response composition (sideband sink → mpsc → Body::from_stream) is exactly reusable.
2. The request-body futures-io adapter (BodyReader) belongs in alkgit-transport next to the request-reader type; both POC-1's reset() discipline and this POC's error-branch rule encode there.
3. Body-size budgets: receive-pack POST bodies must carry a server- configurable limit (GW-15-style counting stream) — unbounded bodies are a bounded-resources violation.
4. The probe/empty-body response convention (200 + empty body for flush-only POSTs) and the no-0002-on-http rule should be encoded in the transport crate's http adapter so handlers can't get them wrong.
5. HttpAdapter non-Clone: assembly layers will hold Arc<HttpAdapter> per accept task; document as the intended usage (or derive Clone).

Verification transcript (2026-09-21)

$ cargo run -- fixture /tmp/opencode/poc3          # small fixture (5 refs)
$ cargo run -- serve 9445 /tmp/opencode/poc3/fixture.git
$ git -c protocol.version=2 ls-remote http://127.0.0.1:9445/repo.git
    -> 5 refs, clean exit
$ git -c protocol.version=2 clone http://127.0.0.1:9445/repo.git fv
    -> completes; git fsck --strict OK
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/d
    -> * [new branch] dev -> d; git fsck --strict OK
$ git checkout v1.0                                # annotated tag OK
$ git -c protocol.version=2 -c http.postBuffer=65536 clone … cfinal2
    -> chunked-POST path; completes; git fsck --strict OK
$ git -c protocol.version=2 -c http.version=HTTP/2 clone … ch2
    -> h2c; completes; git fsck --strict OK
$ cargo run -- fixture-large /tmp/opencode/poc3 10000
$ … serve 9444 …/fixture-large.git
$ git clone http://… clonelarge                    # 60k objects, 5.33 MB
    -> completes (37.8 s wall, debug build); git fsck --strict OK
$ slow-reader RSS probe: peak 33.9 MB total / baseline 26 MB
  while streaming 5.33 MB at ~650 KB/s (back-pressure parks the generator)
$ dribble probe: chunks consumed as they arrive
    -> {"chunks":3,"bytes":300,"first_gap_ms":149,"max_gap_ms":150}

Traces: GIT_TRACE_PACKET=1 shows the V2 handshake over http — the advertisement from info/refs, command=ls-refs and command=fetch POSTs (with the capability dump prefixing the first POST), and the packfile section over band-1 with flush termination. GIT_CURL_VERBOSE captures the chunked-transfer large-request path. Ground truth for every framing decision was cross-checked against git http-backend behind a local CGI bridge serving the same fixture.