alkgit/docs/research/poc3-findings.md

# POC-3 findings: smart-http shape through alkhttp

**Status**: complete — **proceed**
**Date**: 2026-09-21
**Mode**: branch (`poc3-smart-http`, per `pocs.md`); the POC code lives at
`/workspace/alkgit-poc3` (scratch crate — branch mode's exercise of the repo
skeleton turned out to be pure-`HttpAdapter` consumption, which the published
alkhttp 0.5 already serves, so no repo code changes were needed). The branch
carries no code deltas and is dropped after this document merges; it exists
only as the POC's declared mode.
**Depends on**: POC-1 (pkt-line over alkcall BiStream), POC-2 (streaming pack
generation)

## Hypothesis (restated)

alkhttp can serve `GET /<repo>/info/refs` and stream a POST body (ingest
without full buffering) for the smart-http services; real `git clone
http://…` completes against it, and the request-body API's streaming-vs-
buffering behavior is measured (not guessed).

## Result

**Confirmed, with two course corrections (both git-protocol facts, not
alkhttp facts).** Real `git` 2.43 clones, fetches, and tag-checkouts over
`http://` against a service whose entire wire path is:

```
tokio TcpStream
  -> alkcall Connection::from_bidi(stream, "http/1.1", addr)   [alkcall 0.8.0]
  -> alkhttp HttpAdapter::handle(conn, auth)                    [alkhttp 0.5.0]
  -> accept_bi() -> BiStream
  -> hyper-util auto builder (h1/h2) -> axum Router
  -> our git routes mounted via HttpAdapter::with_extra_routes
     GET  /{repo}/info/refs?service=git-upload-pack   (V2 advertisement)
     POST /{repo}/git-upload-pack                     (V2 stateless commands)
  -> request body: axum Body -> BodyDataStream -> BodyReader
     (futures-io adapter, chunk-at-a-time, no accumulation)
     -> gix-packetline StreamingPeekableIter (stateless command parse)
  -> response: pkt-lines / band-1 sideband -> bounded mpsc (8 slots)
     -> axum Body::from_stream (chunked, back-pressured)
```

| Check | Result |
|---|---|
| `GET /info/refs?service=git-upload-pack` | 200, correct smart prefix + V2 advertisement, `application/x-git-upload-pack-advertisement`, no-cache |
| `git ls-remote http://…` (V2) | all 5 refs, clean exit |
| `git clone` (V2, full handshake) | clone completes, `git fsck --strict` passes |
| `git checkout v1.0` | annotated tag served over http, checks out |
| `git fetch` (incremental) | ref updated, fsck clean |
| `git clone` with `http.postBuffer=65536` (forces chunked/streamed POST) | clone completes, fsck clean |
| `git clone` with `http.version=HTTP/2` (h2c prior knowledge) | clone completes, fsck clean |
| 10k-commit / 60k-object clone over http | 5.33 MB pack, 37 s wall (debug build), fsck clean |
| in-process body-streaming probe (dribbled chunks) | chunks consumed as they arrive (gaps preserved), not buffered |
| slow-reader pack fetch (32 KB / 50 ms ≈ 650 KB/s) | server RSS stays ~33.8 MB peak for the whole 5.33 MB pack; drops back after |

## The measured answer: request bodies stream, not buffer

**alkhttp's request-body API streams.** `axum::body::Body::into_data_stream()`
yields `BodyDataStream` (an `http_body`-driven `Stream<Item = io::Result<Bytes>>`);
each `poll_next` produces one hyper-read chunk. The POC's `BodyReader` (a
~40-line `futures_io::AsyncRead` adapter) serves reads from the current
chunk's cursor and pulls the next chunk only when the cursor drains —
gix-packetline's `StreamingPeekableIter` reads through it unmodified.

Evidence, not guesswork:

1. **Dribble probe** (`POST /poc3/probe-upload`): a client that sends
   3 chunks with 150 ms gaps sees the server consume them as they arrive —
   the response reports `chunks: 3, first_gap_ms: 149, max_gap_ms: 150`.
   A buffering server would report all-at-once (gaps ≈ 0 at the end).
2. **git's own chunked path**: with `http.postBuffer=65536` git forces the
   large-request path (chunked POST, no Content-Length), and the full clone
   handshake completes through the same handler. This is the shape
   receive-pack will see for large pushes — it works and streams.
3. **Response-side back-pressure is equally real**: fetching the 60k-object
   pack with a slow reader keeps server RSS flat — peak ≈ 33.9 MB total
   process (≈ 8 MB above idle) while 5.33 MB trickles out at 650 KB/s;
   RSS returns to baseline (~26 MB) after completion. No whole-pack
   buffering anywhere: pack generation (POC-2's `spawn_blocking` pipeline)
   parks on the bounded mpsc (capacity 8) whenever the HTTP client lags.

## What streaming response bodies look like in axum (the POC's composition)

- `Body::from_stream(ReceiverStream<Bytes>)` gives a chunked response body
  with real back pressure. The blocking-side pack generator writes
  band-1 pkt-lines through an `io::Write` sink that does
  `mpsc::Sender::blocking_send` — the sink parks under pressure, exactly
  the POC-2 `SidebandSink` shape with the pkt-line writer swapped for
  channel sends.
- The single 65000-byte sideband chunk size from POC-2 carries over
  unchanged; each chunk becomes one `Bytes` item, one HTTP chunk each.
- hyper/axum add no buffering on top: the first band-1 chunk left the
  server ~8 ms into generation in the fast-reader measurement.

## Course corrections (git smart-http protocol facts for `git-protocol.md`)

### 1. V2-over-http stateless framing: what the client actually sends and requires

Captured against git 2.43 (`remote-curl.c` `stateless_connect` + ground
truth from `git http-backend` behind a local mirror):

- **The first POST carries the cached capability dump.** remote-curl pipes
  the *entire* `info/refs` response it received (including the
  `# service=…` smart prefix, its flush, the V2 capability lines, and
  their flush) ahead of the client's command in the body of the first
  POST. A server's command parser must skip both flush-delimited sections
  before looking for `command=…`. Subsequent POSTs (negotiation rounds)
  carry only the command request.
- **Capabilities ride before the delimiter, command args after.**
  `command=<name>`, `agent=…`, `object-format=…` precede the `0001` delim;
  `peel`/`symrefs`/`want`/`have`/`done`/`ref-prefix` follow it. Parsing
  that treats pre-delim lines as args breaks `fetch` (the `object-format`
  line would shadow the real args).
- **Responses end with a flush, never a response-end packet.** The `0002`
  response-end pkt that protocol V2 requires (and that `fetch-pack`
  checks via `check_stateless_delimiter`) is **synthesized by
  remote-curl**, written to the child process's pipe after every POST
  response. The HTTP response body itself must end at the flush:
  remote-curl's `check_pktline` **dies on `0002` anywhere in an HTTP
  response body** (`remote-curl: unexpected response end packet`). Verified
  against `git http-backend` ground truth: `info/refs`, `ls-refs` response,
  and `fetch` packfile response all end at `0000`; no `0002` anywhere.
  This contradicts a natural reading of the protocol docs' grammar and is
  the POC's most load-bearing correction for `alkgit-http`.
- **The flush is one-shot per response.** A second flush pkt in one
  response breaks the client (`clone< 0000` then "expected response end
  packet"). The POC's first sink version emitted the flush on every
  `flush()` call; the fix is the same idempotent-flush guard POC-2 already
  had (`flushed` flag). `FromEntriesIter` flushes its writer at the pack
  trailer, and the handler's explicit `flush()` is then a no-op — the guard
  is mandatory, not cosmetic.
- **A flush-only POST is a probe, not an error.** remote-curl's
  `probe_rpc` (large-request path) POSTs a 4-byte `0000` body before the
  real request; git's `http-backend` answers **200 with an empty body**
  (upload-pack exits without output). Answering 400 breaks the whole
  clone: `RPC failed; HTTP 400` → "expected flush after ref listing".
  The POC mirrors the 200-empty-body shape.

### 2. alkhttp fit: everything needed exists; three small frictions noted

- **`with_extra_routes` is the whole extension surface and it works**: a
  `Router<()>` with internal state (`.with_state(...)`) merges under the
  gateway's auth layer; reserved-path collision checks pass for
  `/{repo}/info/refs` and `/{repo}/git-upload-pack`.
- **`HttpAdapter` is not `Clone`** — the POC wraps it in `Arc` per accept
  task. `ProtocolHandler::handle` takes `&self`, so `Arc` is the natural
  shape; worth documenting (or `#[derive(Clone)]` if the internals ever
  allow) in the assembly layer's notes.
- **Request-body streaming is by-construction** (see above) but there is
  **no body-size limit on custom routes**: the gateway installs its own
  2 MiB cap (GW-15) but `with_extra_routes` routes get hyper's unbounded
  stream. The git receive-pack route will need an explicit budget
  (bounded-resources invariant) — a phase-1 note for `alkgit-http`, not a
  blocker.
- One POC-side bug is worth recording as a transport-design lesson: an
  error-shaped read (EOF mid-parse) that loops with `reset()` **spins a
  tokio worker at 100%** — `StreamingPeekableIter` returns `Some(Err(io))`
  (not `None`) on EOF-mid-line, so an error branch that resets and retries
  never terminates. The parse loop must break on error unconditionally.
  Encode this in the `alkgit-transport` request-reader type (POC-1
  follow-up 2).

## What the POC does NOT settle

- **receive-pack (`git push`)**: not exercised. The request-streaming
  answer (above) plus POC-2's `bundle::write` finding cover its two
  halves, but the push protocol itself (unpack/ack/report-status over
  http) is phase-1 work.
- **Auth**: the POC serves anonymous git over alkhttp's permissive routes
  (the gateway's bearer layer is not applied to extra routes' handlers
  unless registered before it — the POC relies on the route-local
  permissiveness). The visible-surface = authorized-surface invariant
  needs `AccessControl` wiring at assembly time; out of POC scope by
  design.
- **TLS**: plaintext h2c/h1 over local tcp. alktls composition is
  `Connection::from_bidi(stream, alpn)` with a `TlsStream` — unchanged
  from POC-1's analysis, not re-verified here.
- **h2 keep-alive interaction with long pack streams**: the h2c clone
  worked; long-lived streaming under h2 with hyper's 30 s/10 s ping knobs
  (SRV-04) deserves one soak test in phase 1.

## What it changes in the research docs

- `alk-stack.md` §"What to verify" item 2: **resolved** — response bodies
  stream with back pressure; request bodies stream (measured); smart-http
  V2 works end-to-end through alkhttp's custom-route surface.
- `git-protocol.md`: add the V2-over-http framing facts above (capability
  dump in first POST, flush-only responses, probe_rpc 200-empty, no `0002`
  on the wire).
- `pocs.md`: POC-3 outcome recorded; phase 0 gates all pass.

## Follow-ups for phase 1 (architecture input)

1. `alkgit-http` route shapes: `GET /{repo}/info/refs` +
   `POST /{repo}/git-upload-pack|git-receive-pack` as the canonical
   handlers, mounted via `with_extra_routes`; the POC's `httpservice.rs`
   is the seed. Response composition (sideband sink → mpsc →
   `Body::from_stream`) is exactly reusable.
2. The request-body `futures-io` adapter (`BodyReader`) belongs in
   `alkgit-transport` next to the request-reader type; both POC-1's
   `reset()` discipline and this POC's error-branch rule encode there.
3. Body-size budgets: receive-pack POST bodies must carry a server-
   configurable limit (GW-15-style counting stream) — unbounded bodies are
   a bounded-resources violation.
4. The probe/empty-body response convention (200 + empty body for
   flush-only POSTs) and the no-`0002`-on-http rule should be encoded in
   the transport crate's http adapter so handlers can't get them wrong.
5. `HttpAdapter` non-`Clone`: assembly layers will hold `Arc<HttpAdapter>`
   per accept task; document as the intended usage (or derive `Clone`).

## Verification transcript (2026-09-21)

```
$ cargo run -- fixture /tmp/opencode/poc3          # small fixture (5 refs)
$ cargo run -- serve 9445 /tmp/opencode/poc3/fixture.git
$ git -c protocol.version=2 ls-remote http://127.0.0.1:9445/repo.git
    -> 5 refs, clean exit
$ git -c protocol.version=2 clone http://127.0.0.1:9445/repo.git fv
    -> completes; git fsck --strict OK
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/d
    -> * [new branch] dev -> d; git fsck --strict OK
$ git checkout v1.0                                # annotated tag OK
$ git -c protocol.version=2 -c http.postBuffer=65536 clone … cfinal2
    -> chunked-POST path; completes; git fsck --strict OK
$ git -c protocol.version=2 -c http.version=HTTP/2 clone … ch2
    -> h2c; completes; git fsck --strict OK
$ cargo run -- fixture-large /tmp/opencode/poc3 10000
$ … serve 9444 …/fixture-large.git
$ git clone http://… clonelarge                    # 60k objects, 5.33 MB
    -> completes (37.8 s wall, debug build); git fsck --strict OK
$ slow-reader RSS probe: peak 33.9 MB total / baseline 26 MB
  while streaming 5.33 MB at ~650 KB/s (back-pressure parks the generator)
$ dribble probe: chunks consumed as they arrive
    -> {"chunks":3,"bytes":300,"first_gap_ms":149,"max_gap_ms":150}
```

Traces: `GIT_TRACE_PACKET=1` shows the V2 handshake over http — the
advertisement from `info/refs`, `command=ls-refs` and `command=fetch`
POSTs (with the capability dump prefixing the first POST), and the
`packfile` section over band-1 with flush termination. `GIT_CURL_VERBOSE`
captures the chunked-transfer large-request path. Ground truth for every
framing decision was cross-checked against `git http-backend` behind a
local CGI bridge serving the same fixture.