docs(research): POC-1 complete — pkt-line over alkcall BiStream verified
Full V2 fetch handshake (ls-refs + fetch with done + sideband pack) runs between real git 2.43 and a Rust listener over the alkcall Connection -> BiStream -> tokio-util compat -> gix-packetline path. Verification: git ls-remote, git clone (fsck --strict clean), incremental git fetch, annotated-tag checkout, in-process duplex selftest. All over alkcall 0.8 core types; the flagged futures-io risk resolved with a two-line tokio-util compat adapter. Findings: docs/research/poc-1-findings.md (deliverable; POC source stays disposable at /workspace/alkgit-poc1 per the standalone POC mode). Also records in the research docs: - git-protocol.md: observed done-path shape (no acknowledgments section), git:// first-request framing, advertise-once rule - gitoxide.md: gix-packetline async stop-delimiter + reset() contract - alk-stack.md: POC-1 verification item marked resolved - pocs.md: POC-1 status -> proceed
This commit is contained in:
1 parent
b9ab5e4c9e
commit
665f701133
5 files changed
+209
No files matched your search
@@ -81,6 +81,9 @@ door produced the stream.
|
||||
|
||||
1. **BiStream ↔ pkt-line fit**: run a real `git clone` handshake over an
|
||||
alkcall `Connection` using `gix-packetline` async codec end-to-end.
|
||||
**Verified (POC-1, 2026-09-20)**: real git 2.43 clones/fetches
|
||||
fsck-clean over `Connection → BiStream → tokio-util compat →
|
||||
gix-packetline`. See `poc-1-findings.md`.
|
||||
2. **HTTP smart protocol shape**: confirm alkhttp's streaming response body
|
||||
can carry a pack (chunked) and that request bodies stream in without
|
||||
buffering the whole POST in memory (receive-pack can be gigabytes).
|
||||
|
||||
@@ -27,6 +27,8 @@ we want to own code.
|
||||
2. Server replies with capability advertisement: `version 2`, then
|
||||
capabilities as pkt-lines: `agent=...`, `ls-refs=...`, `fetch=...`
|
||||
(shallow, filter, sideband-64k, packfile-uris...), `object-format=sha1`.
|
||||
**Observed (POC-1)**: the advertisement is sent once per session;
|
||||
commands follow without re-advertisement.
|
||||
3. `command=ls-refs` with args (peel, symrefs, ref-prefix) → server streams
|
||||
ref lines then flush.
|
||||
4. `command=fetch` with args (want lines, have lines, done, thin-pack,
|
||||
@@ -34,6 +36,18 @@ we want to own code.
|
||||
`acknowledgments` section (acked ids or `NAK`), then either
|
||||
`ready` + `packfile` section (pack streamed over sideband) if done, or
|
||||
waits for more haves.
|
||||
**Observed (POC-1, git 2.43)**: on the `done` path the response is just
|
||||
`packfile\n` + sideband chunks + flush — no `acknowledgments`, no
|
||||
`ready`. The acknowledgments section appears on multi-round negotiation
|
||||
(no `done`), not on the done path.
|
||||
|
||||
### git:// framing detail (observed, POC-1)
|
||||
|
||||
First pkt-line: `git-upload-pack <repo>\0host=<host>\0\0version=2\0` —
|
||||
service + repo, NUL-separated extras, an empty segment, then
|
||||
`version=2`. The repo name arrives in-band before any ref data; a server
|
||||
must resolve it against the registry (and run ACL) before emitting the
|
||||
advertisement.
|
||||
|
||||
### V0/V1 (fallback for old clients / http stateless)
|
||||
|
||||
|
||||
@@ -70,6 +70,15 @@ compile-time-rejected invariant, so the config can't drift silently.
|
||||
- `gix-packetline` — pkt-line encode/decode, both blocking and `futures-io`
|
||||
async (`async-io` feature). This is the one crate both the http and ssh
|
||||
interfaces need most; it is small and stable (0.22.x).
|
||||
**API contract notes (POC-1, validated live):** `async_io` codec is built
|
||||
on `futures_io` traits — bridging a tokio stream takes
|
||||
`tokio::io::split` + `tokio_util::compat::{compat, compat_write}` (two
|
||||
one-liners). `StreamingPeekableIter` treats `Flush` as a stop-delimiter:
|
||||
on flush, `read_line()` returns `None` (no line is yielded) and the
|
||||
iterator is inert until `.reset()` — a command loop must reset after
|
||||
every request or the next command is silently swallowed. Return type is
|
||||
`Option<io::Result<Result<PacketLineRef, decode::Error>>>` (three layers;
|
||||
wrap it in a session type).
|
||||
- `gix-transport` — defines `Protocol` (V0/V1/V2), `client::MessageKind`, and
|
||||
the fetch-side abstractions. Only the *types* are reusable server-side;
|
||||
its transport implementations (client) are not what we need.
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
# POC-1 findings: pkt-line over alkcall BiStream
|
||||
|
||||
**Status**: complete — **proceed**
|
||||
**Date**: 2026-09-20
|
||||
**Mode**: standalone (`/workspace/alkgit-poc1`, disposable per `pocs.md`; this
|
||||
document is the deliverable on main)
|
||||
|
||||
## Hypothesis (restated)
|
||||
|
||||
A full V2 fetch handshake (`ls-refs` + one `fetch` round) between real `git`
|
||||
CLI (as client, `git clone`/`git fetch`) and a Rust listener that bridges
|
||||
alkcall `BiStream` to `gix-packetline`'s async codec can be completed.
|
||||
|
||||
## Result
|
||||
|
||||
**Confirmed.** Real `git` 2.43 successfully completed a full protocol V2
|
||||
conversation with a Rust server whose entire wire path is:
|
||||
|
||||
```
|
||||
tokio TcpStream
|
||||
-> alkcall Connection::from_bidi(stream, alpn, addr) [alkcall 0.8.0]
|
||||
-> ProtocolHandler::handle(conn, &auth)
|
||||
-> Connection::accept_bi() -> BiStream [joined, boxed dyn]
|
||||
-> tokio::io::split(stream) -> (ReadHalf, WriteHalf)
|
||||
-> tokio_util::compat::{compat, compat_write} [tokio <-> futures-io]
|
||||
-> gix_packetline::async_io::{StreamingPeekableIter, Writer, encode} [0.22.2]
|
||||
-> V2 server state machine (ours)
|
||||
```
|
||||
|
||||
Validated against real git's parser (the strictest pkt-line validator
|
||||
available), all over the alkcall `Connection`/`BiStream`/`ProtocolHandler`
|
||||
surface — no simulation, no bypass:
|
||||
|
||||
| Operation | Result |
|
||||
|---|---|
|
||||
| `git ls-remote git://…/repo.git` | all 5 refs + peeled tag + `^{}` deref, clean exit |
|
||||
| `git clone` (V2, full handshake) | clone completes, `git fsck --strict` passes |
|
||||
| `git fetch` (incremental, `refs/heads/dev:…`) | completes, ref updated |
|
||||
| `git checkout v1.0` (annotated tag served) | checks out at the tagged commit |
|
||||
| in-process `selftest` over `tokio::io::duplex` | advertisement → ls-refs → fetch → sideband pack, asserts pass |
|
||||
|
||||
The fixture repo (built with the `gix` facade: 3 commits, 2 branches +
|
||||
feature branch, annotated tag, symbolic HEAD) round-trips byte-exact — the
|
||||
clone's object graph is identical to the fixture's.
|
||||
|
||||
## Answered hypotheses
|
||||
|
||||
### 1. The flagged risk — `futures-io` vs tokio — is real and cheap to solve
|
||||
|
||||
`gix-packetline` 0.22.2's `async-io` feature is built on `futures_io`
|
||||
traits; alkcall's `BiStream` is a tokio `AsyncRead + AsyncWrite`. The bridge
|
||||
is two one-line trait calls from `tokio-util`'s `compat` feature:
|
||||
|
||||
```rust
|
||||
let (read_half, write_half) = tokio::io::split(stream); // BiStream: Unpin, so plain split works
|
||||
let reader = StreamingPeekableIter::new(read_half.compat(), &[PacketLineRef::Flush], false);
|
||||
let writer = write_half.compat_write();
|
||||
```
|
||||
|
||||
No hand-written adapter needed. `BiStream` being `Unpin` by construction
|
||||
(boxed `dyn AsyncReadWrite + Unpin`) means `tokio::io::split(stream)` works
|
||||
directly on it, exactly as ADR-009 intends. **The composition boundary
|
||||
("consume (identity, repo id, duplex stream, limits)") survives contact with
|
||||
the packetline codec.**
|
||||
|
||||
### 2. Full V2 fetch handshake shape (byte-level, observed)
|
||||
|
||||
Captured ground truth from real `git upload-pack` via stdio pipes and
|
||||
`GIT_TRACE_PACKET`, then reproduced it:
|
||||
|
||||
- **Advertisement (once per session, not per command)**: `version 2`,
|
||||
`agent=…`, capability lines, flush. git does **not** re-advertise between
|
||||
commands — my first implementation re-advertised in the command loop and
|
||||
the client hung. This is now encoded in the POC's state machine
|
||||
(advertise once → command loop).
|
||||
- **`command=ls-refs`**: request is command line(s) + `0001` delimiter +
|
||||
args (`peel`, `symrefs`, `ref-prefix …`) + flush. Response is ref lines
|
||||
(`"<oid> <ref>[ symref-target:…][ peeled:…]"`) + flush. `ref-prefix`
|
||||
filtering is client-driven; `peel`/`symrefs` are per-request flags.
|
||||
- **`command=fetch` with `done`**: git 2.43's response is just
|
||||
`packfile\n` + sideband chunks (`0001`-framed band-1 data) + flush —
|
||||
**no `acknowledgments` section and no `ready`** when `done` is present.
|
||||
The git-scm protocol docs' grammar suggests an acknowledgments section;
|
||||
the real wire does not send one on the `done` path. (Recorded because
|
||||
`git-protocol.md`'s inventory assumed the docs' shape.)
|
||||
- **`git://` framing** (used by the POC's tcp bridge): one pkt-line
|
||||
`git-upload-pack <repo>\0host=<host>\0\0version=2\0` — the V2 request
|
||||
rides NUL-separated extras after an empty segment. Repo name is in-band
|
||||
here, so a real server resolves it against the registry before any ref
|
||||
line is emitted (ACL-before-advertisement invariant holds naturally).
|
||||
|
||||
### 3. gix-packetline `StreamingPeekableIter` has a non-obvious API contract
|
||||
|
||||
Two findings worth encoding in `alkgit-transport`'s design:
|
||||
|
||||
1. **Flush is a stop-delimiter, not a yielded line.** Constructed with
|
||||
`&[PacketLineRef::Flush]`, `read_line()` returns `None` when the peer's
|
||||
flush arrives; the line is *not* returned as `PacketLineRef::Flush`.
|
||||
After the stop, the iterator is inert until `.reset()`. Consequence: a
|
||||
V2 command loop must `read_command() { …; reader.reset() }` after every
|
||||
request — forgetting `reset()` silently swallows the next command
|
||||
(this was the POC's second hang).
|
||||
2. **`read_line()` returns `Option<io::Result<Result<PacketLineRef,
|
||||
decode::Error>>>`** — three layers deep (`None` | io error | decode
|
||||
error | line). The POC normalized this into a small helper; the real
|
||||
crate should wrap it in a friendlier session type.
|
||||
|
||||
### 4. Honest capability advertisement is easy to get right
|
||||
|
||||
The POC advertises exactly `ls-refs=unborn`, `fetch=wait-for-done`,
|
||||
`object-format=sha1` — and accepts/ignores `thin-pack`/`ofs-delta`/`no-progress`
|
||||
because the behavior (never emit deltas, never send progress) matches what
|
||||
those args would request. Declining `shallow`, `filter`, `packfile-uris`,
|
||||
`object-info`, `server-option` by not advertising them was accepted by real
|
||||
git without complaint. Real git sends `agent=` and `object-format=sha1` on
|
||||
its command requests; parsing `object-format` from command requests (and
|
||||
rejecting mismatches) is a to-do for the real implementation.
|
||||
|
||||
## What the POC deliberately does NOT settle (POC-2's job)
|
||||
|
||||
The `fetch` response pack is a **framing vehicle**: hand-rolled, no deltas,
|
||||
fully materialized in memory (`Vec<u8>`), whole-object zlib only. It is
|
||||
deliberately naive — POC-2 owns pack *composition* quality (delta
|
||||
compression, `gix-pack` bundle-write vs entries-to-bytes, streaming,
|
||||
memory behavior at 10k objects). One lesson already visible from here:
|
||||
|
||||
- The naive pack writer needs the **full reachable closure**
|
||||
(commit → tree → subtrees → blobs, tag → target). The first version
|
||||
walked only commit/tag edges and real git rejected the pack with
|
||||
`did not receive expected object` — object closure computation is
|
||||
non-optional and is exactly the want/have-set walk `git-protocol.md`
|
||||
describes. This strengthens the case for measuring POC-2's option (b)
|
||||
(`gix-pack::data::output::bytes` fed by an odb walk) early.
|
||||
|
||||
## What it changes in the research docs
|
||||
|
||||
- `alk-stack.md` §"What to verify" item 1: **resolved** — BiStream ↔
|
||||
pkt-line fit confirmed; the compat adapter is the only glue.
|
||||
- `git-protocol.md`: add the observed `done`-path shape (no
|
||||
acknowledgments on `done`), the git:// first-request framing detail, and
|
||||
the advertise-once rule to the V2 section.
|
||||
- `gitoxide.md`: `gix-packetline` async usage note (stop-delimiter +
|
||||
`reset()` contract) so the transport crate gets it right the first time.
|
||||
- `pocs.md`: POC-1 outcome recorded.
|
||||
|
||||
## Follow-ups for phase 1 (architecture input)
|
||||
|
||||
1. `alkgit-transport` session type should wrap the three-layer bridge
|
||||
(split → compat → packetline) so handlers never see `futures-io` types;
|
||||
the POC's `serve_v2` is the seed of that type's interface.
|
||||
2. Command-request parsing (`command=…` + delim + args + flush) and the
|
||||
`reset()` discipline belong in one type (`RequestReader`), not spread
|
||||
across handlers.
|
||||
3. Sideband pack streaming (band-1 chunks + flush) is trivial over the
|
||||
same `Writer`; `max chunk size` (65000 < 65515) and a max-pack-size
|
||||
budget should be server-configurable (bounded-resources invariant).
|
||||
4. `read_request_line`'s repo-id extraction is the ACL hook: resolve repo
|
||||
ID → registry → storage root, `AccessControl::check(peer)` **before**
|
||||
the advertisement or any ref line is written (visible-surface =
|
||||
authorized-surface).
|
||||
|
||||
## Verification transcript (2026-09-20)
|
||||
|
||||
```
|
||||
$ cargo run -- bridge 9419 /tmp/opencode/poc1-run # serves fixture
|
||||
$ git -c protocol.version=2 clone git://127.0.0.1:9419/repo.git clone-final
|
||||
-> completes
|
||||
$ cd clone-final && git fsck --strict # FSCK-OK
|
||||
$ git checkout v1.0 # annotated tag works
|
||||
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev
|
||||
-> * [new branch] dev -> newdev
|
||||
$ git -c protocol.version=2 ls-remote … # 5 refs + peeled
|
||||
```
|
||||
|
||||
Traces captured with `GIT_TRACE_PACKET=1` show the client receiving the
|
||||
POC's advertisement (`agent=alkgit-poc1/0.1`), ref advertisement with
|
||||
`symref-target`/`peeled` attributes, and the `packfile` section over
|
||||
sideband with flush termination.
|
||||
@@ -24,6 +24,11 @@ docs.
|
||||
|
||||
## POC-1: pkt-line over alkcall BiStream
|
||||
|
||||
**Status**: complete (2026-09-20) — **proceed**. See `poc-1-findings.md`.
|
||||
Real git 2.43 clones/fetches/fsck-clean over the alkcall
|
||||
`Connection → BiStream → tokio-util compat → gix-packetline` path; the
|
||||
flagged futures-io risk resolved with a two-line compat adapter.
|
||||
|
||||
**Hypothesis**: a full V2 fetch handshake (`ls-refs` + one `fetch` round)
|
||||
between real `git` CLI (as client, `git clone`/`git fetch`) and a Rust
|
||||
listener that bridges alkcall `BiStream` to `gix-packetline`'s async codec
|
||||
|
||||
Reference in new issue
Block a user