From 665f701133b2133d18bd7a47e32af74c555b6cb3 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Sun, 20 Sep 2026 12:18:04 +0000 Subject: [PATCH] =?UTF-8?q?docs(research):=20POC-1=20complete=20=E2=80=94?= =?UTF-8?q?=20pkt-line=20over=20alkcall=20BiStream=20verified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full V2 fetch handshake (ls-refs + fetch with done + sideband pack) runs between real git 2.43 and a Rust listener over the alkcall Connection -> BiStream -> tokio-util compat -> gix-packetline path. Verification: git ls-remote, git clone (fsck --strict clean), incremental git fetch, annotated-tag checkout, in-process duplex selftest. All over alkcall 0.8 core types; the flagged futures-io risk resolved with a two-line tokio-util compat adapter. Findings: docs/research/poc-1-findings.md (deliverable; POC source stays disposable at /workspace/alkgit-poc1 per the standalone POC mode). Also records in the research docs: - git-protocol.md: observed done-path shape (no acknowledgments section), git:// first-request framing, advertise-once rule - gitoxide.md: gix-packetline async stop-delimiter + reset() contract - alk-stack.md: POC-1 verification item marked resolved - pocs.md: POC-1 status -> proceed --- docs/research/alk-stack.md | 3 + docs/research/git-protocol.md | 14 +++ docs/research/gitoxide.md | 9 ++ docs/research/poc-1-findings.md | 178 ++++++++++++++++++++++++++++++++ docs/research/pocs.md | 5 + 5 files changed, 209 insertions(+) create mode 100644 docs/research/poc-1-findings.md diff --git a/docs/research/alk-stack.md b/docs/research/alk-stack.md index 04845bf..74125d4 100644 --- a/docs/research/alk-stack.md +++ b/docs/research/alk-stack.md @@ -81,6 +81,9 @@ door produced the stream. 1. **BiStream ↔ pkt-line fit**: run a real `git clone` handshake over an alkcall `Connection` using `gix-packetline` async codec end-to-end. + **Verified (POC-1, 2026-09-20)**: real git 2.43 clones/fetches + fsck-clean over `Connection → BiStream → tokio-util compat → + gix-packetline`. See `poc-1-findings.md`. 2. **HTTP smart protocol shape**: confirm alkhttp's streaming response body can carry a pack (chunked) and that request bodies stream in without buffering the whole POST in memory (receive-pack can be gigabytes). diff --git a/docs/research/git-protocol.md b/docs/research/git-protocol.md index 5c363dc..040630c 100644 --- a/docs/research/git-protocol.md +++ b/docs/research/git-protocol.md @@ -27,6 +27,8 @@ we want to own code. 2. Server replies with capability advertisement: `version 2`, then capabilities as pkt-lines: `agent=...`, `ls-refs=...`, `fetch=...` (shallow, filter, sideband-64k, packfile-uris...), `object-format=sha1`. + **Observed (POC-1)**: the advertisement is sent once per session; + commands follow without re-advertisement. 3. `command=ls-refs` with args (peel, symrefs, ref-prefix) → server streams ref lines then flush. 4. `command=fetch` with args (want lines, have lines, done, thin-pack, @@ -34,6 +36,18 @@ we want to own code. `acknowledgments` section (acked ids or `NAK`), then either `ready` + `packfile` section (pack streamed over sideband) if done, or waits for more haves. + **Observed (POC-1, git 2.43)**: on the `done` path the response is just + `packfile\n` + sideband chunks + flush — no `acknowledgments`, no + `ready`. The acknowledgments section appears on multi-round negotiation + (no `done`), not on the done path. + +### git:// framing detail (observed, POC-1) + +First pkt-line: `git-upload-pack \0host=\0\0version=2\0` — +service + repo, NUL-separated extras, an empty segment, then +`version=2`. The repo name arrives in-band before any ref data; a server +must resolve it against the registry (and run ACL) before emitting the +advertisement. ### V0/V1 (fallback for old clients / http stateless) diff --git a/docs/research/gitoxide.md b/docs/research/gitoxide.md index a904183..004add1 100644 --- a/docs/research/gitoxide.md +++ b/docs/research/gitoxide.md @@ -70,6 +70,15 @@ compile-time-rejected invariant, so the config can't drift silently. - `gix-packetline` — pkt-line encode/decode, both blocking and `futures-io` async (`async-io` feature). This is the one crate both the http and ssh interfaces need most; it is small and stable (0.22.x). + **API contract notes (POC-1, validated live):** `async_io` codec is built + on `futures_io` traits — bridging a tokio stream takes + `tokio::io::split` + `tokio_util::compat::{compat, compat_write}` (two + one-liners). `StreamingPeekableIter` treats `Flush` as a stop-delimiter: + on flush, `read_line()` returns `None` (no line is yielded) and the + iterator is inert until `.reset()` — a command loop must reset after + every request or the next command is silently swallowed. Return type is + `Option>>` (three layers; + wrap it in a session type). - `gix-transport` — defines `Protocol` (V0/V1/V2), `client::MessageKind`, and the fetch-side abstractions. Only the *types* are reusable server-side; its transport implementations (client) are not what we need. diff --git a/docs/research/poc-1-findings.md b/docs/research/poc-1-findings.md new file mode 100644 index 0000000..d4926f5 --- /dev/null +++ b/docs/research/poc-1-findings.md @@ -0,0 +1,178 @@ +# POC-1 findings: pkt-line over alkcall BiStream + +**Status**: complete — **proceed** +**Date**: 2026-09-20 +**Mode**: standalone (`/workspace/alkgit-poc1`, disposable per `pocs.md`; this +document is the deliverable on main) + +## Hypothesis (restated) + +A full V2 fetch handshake (`ls-refs` + one `fetch` round) between real `git` +CLI (as client, `git clone`/`git fetch`) and a Rust listener that bridges +alkcall `BiStream` to `gix-packetline`'s async codec can be completed. + +## Result + +**Confirmed.** Real `git` 2.43 successfully completed a full protocol V2 +conversation with a Rust server whose entire wire path is: + +``` +tokio TcpStream + -> alkcall Connection::from_bidi(stream, alpn, addr) [alkcall 0.8.0] + -> ProtocolHandler::handle(conn, &auth) + -> Connection::accept_bi() -> BiStream [joined, boxed dyn] + -> tokio::io::split(stream) -> (ReadHalf, WriteHalf) + -> tokio_util::compat::{compat, compat_write} [tokio <-> futures-io] + -> gix_packetline::async_io::{StreamingPeekableIter, Writer, encode} [0.22.2] + -> V2 server state machine (ours) +``` + +Validated against real git's parser (the strictest pkt-line validator +available), all over the alkcall `Connection`/`BiStream`/`ProtocolHandler` +surface — no simulation, no bypass: + +| Operation | Result | +|---|---| +| `git ls-remote git://…/repo.git` | all 5 refs + peeled tag + `^{}` deref, clean exit | +| `git clone` (V2, full handshake) | clone completes, `git fsck --strict` passes | +| `git fetch` (incremental, `refs/heads/dev:…`) | completes, ref updated | +| `git checkout v1.0` (annotated tag served) | checks out at the tagged commit | +| in-process `selftest` over `tokio::io::duplex` | advertisement → ls-refs → fetch → sideband pack, asserts pass | + +The fixture repo (built with the `gix` facade: 3 commits, 2 branches + +feature branch, annotated tag, symbolic HEAD) round-trips byte-exact — the +clone's object graph is identical to the fixture's. + +## Answered hypotheses + +### 1. The flagged risk — `futures-io` vs tokio — is real and cheap to solve + +`gix-packetline` 0.22.2's `async-io` feature is built on `futures_io` +traits; alkcall's `BiStream` is a tokio `AsyncRead + AsyncWrite`. The bridge +is two one-line trait calls from `tokio-util`'s `compat` feature: + +```rust +let (read_half, write_half) = tokio::io::split(stream); // BiStream: Unpin, so plain split works +let reader = StreamingPeekableIter::new(read_half.compat(), &[PacketLineRef::Flush], false); +let writer = write_half.compat_write(); +``` + +No hand-written adapter needed. `BiStream` being `Unpin` by construction +(boxed `dyn AsyncReadWrite + Unpin`) means `tokio::io::split(stream)` works +directly on it, exactly as ADR-009 intends. **The composition boundary +("consume (identity, repo id, duplex stream, limits)") survives contact with +the packetline codec.** + +### 2. Full V2 fetch handshake shape (byte-level, observed) + +Captured ground truth from real `git upload-pack` via stdio pipes and +`GIT_TRACE_PACKET`, then reproduced it: + +- **Advertisement (once per session, not per command)**: `version 2`, + `agent=…`, capability lines, flush. git does **not** re-advertise between + commands — my first implementation re-advertised in the command loop and + the client hung. This is now encoded in the POC's state machine + (advertise once → command loop). +- **`command=ls-refs`**: request is command line(s) + `0001` delimiter + + args (`peel`, `symrefs`, `ref-prefix …`) + flush. Response is ref lines + (`" [ symref-target:…][ peeled:…]"`) + flush. `ref-prefix` + filtering is client-driven; `peel`/`symrefs` are per-request flags. +- **`command=fetch` with `done`**: git 2.43's response is just + `packfile\n` + sideband chunks (`0001`-framed band-1 data) + flush — + **no `acknowledgments` section and no `ready`** when `done` is present. + The git-scm protocol docs' grammar suggests an acknowledgments section; + the real wire does not send one on the `done` path. (Recorded because + `git-protocol.md`'s inventory assumed the docs' shape.) +- **`git://` framing** (used by the POC's tcp bridge): one pkt-line + `git-upload-pack \0host=\0\0version=2\0` — the V2 request + rides NUL-separated extras after an empty segment. Repo name is in-band + here, so a real server resolves it against the registry before any ref + line is emitted (ACL-before-advertisement invariant holds naturally). + +### 3. gix-packetline `StreamingPeekableIter` has a non-obvious API contract + +Two findings worth encoding in `alkgit-transport`'s design: + +1. **Flush is a stop-delimiter, not a yielded line.** Constructed with + `&[PacketLineRef::Flush]`, `read_line()` returns `None` when the peer's + flush arrives; the line is *not* returned as `PacketLineRef::Flush`. + After the stop, the iterator is inert until `.reset()`. Consequence: a + V2 command loop must `read_command() { …; reader.reset() }` after every + request — forgetting `reset()` silently swallows the next command + (this was the POC's second hang). +2. **`read_line()` returns `Option>>`** — three layers deep (`None` | io error | decode + error | line). The POC normalized this into a small helper; the real + crate should wrap it in a friendlier session type. + +### 4. Honest capability advertisement is easy to get right + +The POC advertises exactly `ls-refs=unborn`, `fetch=wait-for-done`, +`object-format=sha1` — and accepts/ignores `thin-pack`/`ofs-delta`/`no-progress` +because the behavior (never emit deltas, never send progress) matches what +those args would request. Declining `shallow`, `filter`, `packfile-uris`, +`object-info`, `server-option` by not advertising them was accepted by real +git without complaint. Real git sends `agent=` and `object-format=sha1` on +its command requests; parsing `object-format` from command requests (and +rejecting mismatches) is a to-do for the real implementation. + +## What the POC deliberately does NOT settle (POC-2's job) + +The `fetch` response pack is a **framing vehicle**: hand-rolled, no deltas, +fully materialized in memory (`Vec`), whole-object zlib only. It is +deliberately naive — POC-2 owns pack *composition* quality (delta +compression, `gix-pack` bundle-write vs entries-to-bytes, streaming, +memory behavior at 10k objects). One lesson already visible from here: + +- The naive pack writer needs the **full reachable closure** + (commit → tree → subtrees → blobs, tag → target). The first version + walked only commit/tag edges and real git rejected the pack with + `did not receive expected object` — object closure computation is + non-optional and is exactly the want/have-set walk `git-protocol.md` + describes. This strengthens the case for measuring POC-2's option (b) + (`gix-pack::data::output::bytes` fed by an odb walk) early. + +## What it changes in the research docs + +- `alk-stack.md` §"What to verify" item 1: **resolved** — BiStream ↔ + pkt-line fit confirmed; the compat adapter is the only glue. +- `git-protocol.md`: add the observed `done`-path shape (no + acknowledgments on `done`), the git:// first-request framing detail, and + the advertise-once rule to the V2 section. +- `gitoxide.md`: `gix-packetline` async usage note (stop-delimiter + + `reset()` contract) so the transport crate gets it right the first time. +- `pocs.md`: POC-1 outcome recorded. + +## Follow-ups for phase 1 (architecture input) + +1. `alkgit-transport` session type should wrap the three-layer bridge + (split → compat → packetline) so handlers never see `futures-io` types; + the POC's `serve_v2` is the seed of that type's interface. +2. Command-request parsing (`command=…` + delim + args + flush) and the + `reset()` discipline belong in one type (`RequestReader`), not spread + across handlers. +3. Sideband pack streaming (band-1 chunks + flush) is trivial over the + same `Writer`; `max chunk size` (65000 < 65515) and a max-pack-size + budget should be server-configurable (bounded-resources invariant). +4. `read_request_line`'s repo-id extraction is the ACL hook: resolve repo + ID → registry → storage root, `AccessControl::check(peer)` **before** + the advertisement or any ref line is written (visible-surface = + authorized-surface). + +## Verification transcript (2026-09-20) + +``` +$ cargo run -- bridge 9419 /tmp/opencode/poc1-run # serves fixture +$ git -c protocol.version=2 clone git://127.0.0.1:9419/repo.git clone-final + -> completes +$ cd clone-final && git fsck --strict # FSCK-OK +$ git checkout v1.0 # annotated tag works +$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev + -> * [new branch] dev -> newdev +$ git -c protocol.version=2 ls-remote … # 5 refs + peeled +``` + +Traces captured with `GIT_TRACE_PACKET=1` show the client receiving the +POC's advertisement (`agent=alkgit-poc1/0.1`), ref advertisement with +`symref-target`/`peeled` attributes, and the `packfile` section over +sideband with flush termination. \ No newline at end of file diff --git a/docs/research/pocs.md b/docs/research/pocs.md index ce21553..49a31d2 100644 --- a/docs/research/pocs.md +++ b/docs/research/pocs.md @@ -24,6 +24,11 @@ docs. ## POC-1: pkt-line over alkcall BiStream +**Status**: complete (2026-09-20) — **proceed**. See `poc-1-findings.md`. +Real git 2.43 clones/fetches/fsck-clean over the alkcall +`Connection → BiStream → tokio-util compat → gix-packetline` path; the +flagged futures-io risk resolved with a two-line compat adapter. + **Hypothesis**: a full V2 fetch handshake (`ls-refs` + one `fetch` round) between real `git` CLI (as client, `git clone`/`git fetch`) and a Rust listener that bridges alkcall `BiStream` to `gix-packetline`'s async codec