alkgit/docs/research/poc2-findings.md

POC-2 findings: server-side pack generation

Status: complete — proceed Date: 2026-09-20 Mode: standalone (/workspace/alkgit-poc2, disposable per pocs.md; this document is the deliverable on main) Depends on: POC-1 (pkt-line over alkcall BiStream, poc-1-findings.md)

Hypothesis (restated)

Given a fixture repo and a want/have set, produce a valid pack stream that git verify-pack/git unpack-objects accepts, using either: a. gix-pack::bundle::write into a temp dir then read the file back (correctness baseline), or b. gix-pack::data::output::bytes fed by an odb object walk (streaming), with a decision on streaming composition that doesn't materialize the whole pack in memory for large repos — memory behavior recorded for a 10k-object repo at minimum.

Result

Confirmed, with one course correction. Real git 2.43 clones, fetches, and tag-checkouts against a bridge that generates packs with gitoxide's own generation pipeline, streamed over sideband without ever holding the whole pack in memory. git fsck --strict passes on every clone; git index-pack --strict and git verify-pack accept the generated bytes standalone.

Check	Result
git index-pack --strict on generated pack	OK (both fixtures)
git verify-pack -v	ok, object list + sizes verified
git unpack-objects --strict + git fsck --strict	OK
git clone (V2, full handshake) from the bridge	OK, 10k commits / 60k objects
git fsck --strict in the clone	OK
git fetch (incremental ref fetch)	OK, ref updated, fsck clean
git checkout <annotated tag>	OK, checks out at the peeled commit
in-process sideband selftest	band-1 framing exact, flush-terminated, no stray bytes
closure cross-check	own BFS walk == gitoxide count, 15k/60k objects
approach (a) Bundle::write_to_directory round-trip	indexes our pack, 0 objects materialized in memory

The composition that works (approach b, as built)

gitoxide already ships a complete pack generation pipeline behind gix-pack's generate feature. The serving path is three stages chained as iterators, ending in plain io::Write:

want tips (peeled to commits)
  -> gix_traverse::commit::Simple (Parents::All)      [commit ancestry]
  -> gix_pack::data::output::count::objects_unthreaded(TreeContents)
     [per-commit tree + subtree + blob expansion -> Vec<Count>]
  -> gix_pack::data::output::entry::iter_from_counts  [chunked deflate,
     delta-copy from existing packs, back-pressured]
  -> gix_features::parallel::InOrderIter              [re-order chunk results]
  -> gix_pack::data::output::bytes::FromEntriesIter   [header, entries,
     trailer sha1 -> W: io::Write]
  -> SidebandSink (ours)                              [65000-byte band-1
                                                       pkt-line chunks]

This is exactly gitoxide-core/src/pack/create.rs's composition (gix pack create) — it works against any gix_pack::Find implementor, i.e. against gix_odb::Cache<Handle<Arc<Store>>> directly. No temp files, no git binary, no full-pack buffer.

The course correction: TreeContents does not follow commit parents

The obvious call — feed wants into count::objects(_unthreaded) with ObjectExpansion::TreeContents and expect the full reachable closure — produces an incomplete pack. TreeContents expands each input object's own tree (commit → its tree → subtrees/blobs; tag → target's tree); it does not traverse commit ancestry. Feeding tip commits alone drops all ancestor commits, their trees, and their blobs. Real git's index-pack --strict catches this (did not receive expected object <ancestor>).

Correct closure = commit-ancestry walk (gix_traverse::commit::Simple with Parents::All) whose output commits, plus any non-commit tips (annotated tags), feed the TreeContents count stage. This mirrors what gix pack create does and what POC-1's hand-rolled walker did. Recorded in git-protocol.md's inventory (below) — the want/have-set walk is two stages, not one.

Two smaller API facts along the same line:

1. Tips must be peeled before the commit walk. commit::Simple requires find_commit_iter; handing it an annotated-tag id errors with Expected object of kind commit but got tag. Peel via TagRef::target() in a loop until a commit (or bare tree/blob — no ancestry, count as-is).
2. The odb handle needs prevent_pack_unload() + ignore_replacements = true before use in generation (location_by_oid asserts the former; the latter matches gix pack create). Forgetting the first panics on the first packed-object lookup; the fixture that works loose-only silently passes until a packed repo is used.

Approach (a) baseline: Bundle::write_to_directory

Used as the correctness check, not as the serving path. It consumes an existing pack stream (it is the index-from-stream machinery: data::input::BytesToEntriesIter → index write → pack file persist), so it is the wrong tool for on-demand generation — the pipeline above is the generation path. It needs the streaming-input feature and Option::<gix_object::find::Never>::None as the thin-pack lookup. Feeding it our generated pack: indexes cleanly, index_hash/data_hash match git's index-pack output (both 997bc209… / 05c910d8…-style trailer hashes on the same pack). Conclusion: bundle::write is for receive-pack (POC on pack ingestion), not for fetch-side generation. git-protocol.md's option list said "bundle::write with an in-memory sink" — that reading was wrong; it writes to a temp file per design (it mmaps the pack to resolve deltas for the index).

Streaming & memory behavior (the decision POC-2 was for)

Streaming composition is real and needs no new machinery. FromEntriesIter is a lazy Iterator<Item = Result<u64>> over any io::Write; iter_from_counts yields entries chunk-by-chunk with back-pressure (one chunk of compressed entries in memory at a time, default chunk 10). The only in-memory artifact is Vec<Count> (24 B/object: 20-byte id + pack-location option) — for 60k objects that's ~1.4 MB, dwarfed by the pack itself.

Measured (release build, /proc/self/status VmRSS, 5 ms sampler thread):

Fixture	Objects	Pack bytes	Time	RSS before	RSS peak	RSS after
2.5k commits, 15k objects	15,000	1,327,523	1.4 s	5.5 MB	9.3 MB	7.6 MB
4k commits, 24k objects	24,000	2,128,268	2.5 s	5.4 MB	9.7 MB	7.6 MB
10k commits, 60k objects	60,000	5,334,304	5.5 s	5.4 MB	16.4 MB	9.0 MB

Peak ≈ base + ~(counts × ~200 B) + object buffers — i.e. O(counts), not O(pack): the 5.3 MB pack for 60k objects went to the socket through a 64 KB-bounded sink (65000-byte sideband chunks) with ~11 MB of extra RSS above baseline. No whole-pack materialization anywhere on the path.

Delta behavior, as-built:

• iter_from_counts copies existing pack deltas (from_pack_entry — ofs-delta entries referencing bases also in the pack, translated through the counts index) and deflates loose objects as whole bases (from_data). It does not synthesize new deltas between loose objects — gitoxide has no delta encoding API for the server path (gix-delta is apply/decode only; there is no encode in the published crates).
• Empirical effect on the 60k-object fixture: repo git gc'd to one pack (5.5 MB, 29,981 chain-1 deltas) → our generated pack 3.96 MB (deltas copied through, compressed bases for loose objects); same repo all-loose → generated pack 5.33 MB. Serving a fully packed repo is smaller than what git stores because our walk writes every entry as-is without bitmap reuse and without delta re-selection — and both clone cleanly.
• For phase 1: fresh clones of loose-ish repos ship uncompressed bases (fine for v1; git on the client re-packs at rest if fetch.unpackLimit says so), and repos kept packed get pack-copy for free. Delta synthesis (window search over loose objects) is a real gap vs git upload-pack — record as an optimization backlog item, not a blocker.

Sideband streaming over the POC-1 wire path (verified shape)

The fetch response composes exactly as POC-1's protocol inventory described, now with real generation feeding it:

• packfile\n text line, then band-1 chunks. SidebandSink (an io::Write adapter) caps chunks at 65000 bytes (git's own cap is 65515; headroom for the 4-byte length prefix under the 65520 pkt-line max) and bridges to gix_packetline::async_io::encode::band_to_write via futures_lite::future::block_on — safe because exactly one task drives the sink. flush() is idempotent (the first flush-pkt closes the section; a second would be a protocol error — caught by the selftest's wire decoder).
• The generator is sync; run it on tokio::task::spawn_blocking. The async side passes the compat writer into the blocking task and awaits the result. This also resolves a type-system fact worth encoding now: gix_odb::Cache is deliberately not Sync (per-thread RefCell pack/object caches by design), so it cannot be held across an .await in a Send future. The handler therefore holds Arc<gix_odb::Store> (thread-safe: parking_lot::Mutex + ArcSwap + atomics) and builds a Cache handle per connection/generation (store.to_handle_arc() + prevent_pack_unload()), moving the owned Cache into the blocking task. This is the exact shape alkgit-transport should encode: store shared, handle per session, generation on blocking threads.
• End-to-end over the alkcall surface (Connection::from_bidi → accept_bi → BiStream → compat → packetline): unchanged from POC-1; the pack path adds only the sink.

gix-pack API surface notes for alkgit-core (phase-1 input)

1. Pipeline pieces and paths (pinned versions from gitoxide.md): gix_pack::data::output::count::objects_unthreaded (or the threaded count::objects for scale), entry::iter_from_counts (needs generate
   • parallel features; yields Result<(SequenceId, Vec<Entry>), Error>), gix_features::parallel::InOrderIter to re-sequence, and bytes::FromEntriesIter::new(input, out, num_entries, Version::V2, Kind::Sha1) (asserts V2 — only V2 can be written). Composition is exactly gitoxide-core/src/pack/create.rs; there is no higher-level "write pack to socket" API upstream.
2. FromEntriesIter writes header + count + trailer and flushes the writer; the trailer is digest() after the final next() returns None. iter_from_counts' finalize() (via gix_features::parallel::reduce::Finalize) yields entry statistics (copied-from-pack / recompressed / missing counts) — free server metrics.
3. Progress plumbing: Box<dyn DynNestedProgress>; Discard works everywhere. Server will want a real sink for session progress/budget later.
4. Error taxonomy: generation errors are thiserror enums per stage; missing_objects do not abort — they become "invalid" entries (null-id sentinel) which FromEntriesIter skips while blocking deltas against them. For us: a missing object mid-generation should abort the fetch with a sideband error band, not silently produce a broken pack — check the entry stats and error out if missing_objects > 0.
5. Determinism: single-threaded count (objects_unthreaded) gives deterministic order; the threaded count::objects + multi-thread iter_from_counts reorder chunks (hence InOrderIter). For a server with per-connection tasks, thread_limit=1 in-process plus the blocking pool is plenty at these speeds (60k objects in 5.5 s debug, 1.4 s for 15k release).
6. Cache is not Sync — see above. Also Bundle::write_to_directory requires the streaming-input feature (default on upstream; we run lean, so list features explicitly: sha1, generate, parallel, streaming-input).

What the POC does NOT settle

• Want/have negotiation: haves are parsed and recorded but never subtracted (we advertise wait-for-done and send full closures on done, the POC-1 wire shape). Multi-round ack logic stays phase-1 work; the generator is negotiation-agnostic (boundary sets in → pack out).
• Delta synthesis for loose objects (above) — optimization backlog.
• sha256: pipeline is hash-generic (object_hash parameter) but untested here; the sha256 feature flag passthrough remains an OQ.
• Concurrency: one pack generation per connection on blocking threads; a shared-process thread budget per repo is an architecture concern (bounded-resources invariant), not a POC question.

What it changes in the research docs

• git-protocol.md: pack generation option list resolved — (b) wins; record the two-stage closure (commit ancestry + TreeContents), the prevent_pack_unload()/replacements requirements, bundle::write's actual role (receive-pack indexing), and the missing-delta-synthesis note.
• gitoxide.md: add generation-pipeline notes (feature set, composition, Cache !Sync, Version::V2-only writer, entry statistics via finalize()).
• pocs.md: POC-2 outcome recorded.

Follow-ups for phase 1 (architecture input)

1. alkgit-core should expose a PackWriter type: (odb handle, wants, haves) -> impl io::Write-streaming pack, encapsulating tip peeling, commit walk, count, entries, bytes, and the missing-objects check. The POC's packgen::generate_streaming is the seed.
2. alkgit-transport fetch handler: generation belongs on spawn_blocking; the handler holds Arc<Store> and builds per-session handles. Budget the blocking pool (bounded-resources invariant).
3. Sideband chunk size (65000) and any max-pack-size cap should be server-configurable, flowing through the same limits struct as POC-1's session limits.
4. Consider pack.cache (LRU) sizing on the odb handle for hot-repo fetches; gitoxide exposes set_pack_cache/set_object_cache with byte budgets (gitoxide-core wires exactly this in pack/create.rs).

Verification transcript (2026-09-20)

$ cargo run -- selftest          # small fixture: gen -> bundle roundtrip ->
    git index-pack --strict -> verify-pack -> unpack-objects --strict ->
    fsck --strict -> closure check -> sideband wire decode
  ... sideband framing OK: 763 band-1 bytes in chunks <= 65001, pack bytes 763
$ cargo run -- stats /tmp/…/fixture-large.git
  closure (own walk): 60000 == gitoxide count::objects: 60000
$ cargo run --release -- bench /tmp/…/fixture-large.git   # 10k commits
  gen: 60000 objects, 5334304 pack bytes, 5517 ms, RSS peak=16364kB
$ cargo run -- bridge 9423 /tmp/…/fixture-large.git       # packed 60k-object repo
$ git -c protocol.version=2 clone git://127.0.0.1:9423/repo.git c
    -> completes (6.1 s wall); git fsck --strict OK
$ cargo run -- bridge 9422 /tmp/…/fixture.git             # small fixture, tags
$ git -c protocol.version=2 clone git://127.0.0.1:9422/repo.git c2
    -> clone; git fsck --strict OK; git checkout v1.0 OK (annotated tag)
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev
    -> * [new branch] dev -> newdev; git fsck --strict OK

Traces: GIT_TRACE_PACKET=1 shows the client receiving the POC-2 advertisement (agent=alkgit-poc2/0.1), ls-refs with peeled:/symref-target: attributes, and the packfile section over band-1 sideband with flush termination — identical shape to POC-1's captured ground truth, now fed by real generation.