# POC-2 findings: server-side pack generation

**Status**: complete — **proceed**
**Date**: 2026-09-20
**Mode**: standalone (`/workspace/alkgit-poc2`, disposable per `pocs.md`; this
document is the deliverable on main)
**Depends on**: POC-1 (pkt-line over alkcall BiStream, `poc-1-findings.md`)

## Hypothesis (restated)

Given a fixture repo and a want/have set, produce a valid pack stream that
`git verify-pack`/`git unpack-objects` accepts, using either:
a. `gix-pack::bundle::write` into a temp dir then read the file back
   (correctness baseline), or
b. `gix-pack::data::output::bytes` fed by an odb object walk (streaming),
with a decision on streaming composition that doesn't materialize the whole
pack in memory for large repos — memory behavior recorded for a 10k-object
repo at minimum.

## Result

**Confirmed, with one course correction.** Real `git` 2.43 clones, fetches,
and tag-checkouts against a bridge that generates packs with gitoxide's own
generation pipeline, streamed over sideband without ever holding the whole
pack in memory. `git fsck --strict` passes on every clone; `git index-pack
--strict` and `git verify-pack` accept the generated bytes standalone.

| Check | Result |
|---|---|
| `git index-pack --strict` on generated pack | OK (both fixtures) |
| `git verify-pack -v` | `ok`, object list + sizes verified |
| `git unpack-objects --strict` + `git fsck --strict` | OK |
| `git clone` (V2, full handshake) from the bridge | OK, 10k commits / 60k objects |
| `git fsck --strict` in the clone | OK |
| `git fetch` (incremental ref fetch) | OK, ref updated, fsck clean |
| `git checkout <annotated tag>` | OK, checks out at the peeled commit |
| in-process sideband selftest | band-1 framing exact, flush-terminated, no stray bytes |
| closure cross-check | own BFS walk == gitoxide count, 15k/60k objects |
| approach (a) `Bundle::write_to_directory` round-trip | indexes our pack, 0 objects materialized in memory |

## The composition that works (approach b, as built)

gitoxide already ships a complete pack *generation* pipeline behind
`gix-pack`'s `generate` feature. The serving path is three stages chained as
iterators, ending in plain `io::Write`:

```
want tips (peeled to commits)
  -> gix_traverse::commit::Simple (Parents::All)      [commit ancestry]
  -> gix_pack::data::output::count::objects_unthreaded(TreeContents)
     [per-commit tree + subtree + blob expansion -> Vec<Count>]
  -> gix_pack::data::output::entry::iter_from_counts  [chunked deflate,
     delta-copy from existing packs, back-pressured]
  -> gix_features::parallel::InOrderIter              [re-order chunk results]
  -> gix_pack::data::output::bytes::FromEntriesIter   [header, entries,
     trailer sha1 -> W: io::Write]
  -> SidebandSink (ours)                              [65000-byte band-1
                                                       pkt-line chunks]
```

This is exactly `gitoxide-core/src/pack/create.rs`'s composition (`gix pack
create`) — it works against any `gix_pack::Find` implementor, i.e. against
`gix_odb::Cache<Handle<Arc<Store>>>` directly. No temp files, no `git`
binary, no full-pack buffer.

### The course correction: `TreeContents` does not follow commit parents

The obvious call — feed wants into `count::objects(_unthreaded)` with
`ObjectExpansion::TreeContents` and expect the full reachable closure —
produces an **incomplete pack**. `TreeContents` expands each *input object's*
own tree (commit → its tree → subtrees/blobs; tag → target's tree); it does
**not** traverse commit ancestry. Feeding tip commits alone drops all ancestor
commits, their trees, and their blobs. Real git's `index-pack --strict`
catches this (`did not receive expected object <ancestor>`).

Correct closure = **commit-ancestry walk** (`gix_traverse::commit::Simple`
with `Parents::All`) whose output commits, plus any non-commit tips
(annotated tags), feed the `TreeContents` count stage. This mirrors what
`gix pack create` does and what POC-1's hand-rolled walker did. Recorded in
`git-protocol.md`'s inventory (below) — the want/have-set walk is two stages,
not one.

Two smaller API facts along the same line:

1. **Tips must be peeled before the commit walk.** `commit::Simple` requires
   `find_commit_iter`; handing it an annotated-tag id errors with
   `Expected object of kind commit but got tag`. Peel via `TagRef::target()`
   in a loop until a commit (or bare tree/blob — no ancestry, count as-is).
2. **The odb handle needs `prevent_pack_unload()` + `ignore_replacements = true`**
   before use in generation (`location_by_oid` asserts the former; the latter
   matches `gix pack create`). Forgetting the first panics on the first
   packed-object lookup; the fixture that works loose-only silently passes
   until a packed repo is used.

## Approach (a) baseline: `Bundle::write_to_directory`

Used as the correctness check, not as the serving path. It consumes an
*existing pack stream* (it is the index-from-stream machinery:
`data::input::BytesToEntriesIter` → index write → pack file persist), so it
is the wrong tool for on-demand generation — the pipeline above is the
generation path. It needs the `streaming-input` feature and
`Option::<gix_object::find::Never>::None` as the thin-pack lookup. Feeding it
our generated pack: indexes cleanly, `index_hash`/`data_hash` match git's
`index-pack` output (both `997bc209…` / `05c910d8…`-style trailer hashes on
the same pack). Conclusion: **bundle::write is for receive-pack (POC on pack
ingestion), not for fetch-side generation.** `git-protocol.md`'s option list
said "bundle::write with an in-memory sink" — that reading was wrong; it
writes to a temp file per design (it mmaps the pack to resolve deltas for the
index).

## Streaming & memory behavior (the decision POC-2 was for)

**Streaming composition is real and needs no new machinery.** `FromEntriesIter`
is a lazy `Iterator<Item = Result<u64>>` over any `io::Write`; `iter_from_counts`
yields entries chunk-by-chunk with back-pressure (one chunk of compressed
entries in memory at a time, default chunk 10). The only in-memory artifact is
`Vec<Count>` (24 B/object: 20-byte id + pack-location option) — for 60k
objects that's ~1.4 MB, dwarfed by the pack itself.

Measured (release build, `/proc/self/status` VmRSS, 5 ms sampler thread):

| Fixture | Objects | Pack bytes | Time | RSS before | RSS peak | RSS after |
|---|---|---|---|---|---|---|
| 2.5k commits, 15k objects | 15,000 | 1,327,523 | 1.4 s | 5.5 MB | 9.3 MB | 7.6 MB |
| 4k commits, 24k objects | 24,000 | 2,128,268 | 2.5 s | 5.4 MB | 9.7 MB | 7.6 MB |
| 10k commits, 60k objects | 60,000 | 5,334,304 | 5.5 s | 5.4 MB | 16.4 MB | 9.0 MB |

Peak ≈ base + ~(counts × ~200 B) + object buffers — i.e. **O(counts), not
O(pack)**: the 5.3 MB pack for 60k objects went to the socket through a
64 KB-bounded sink (65000-byte sideband chunks) with ~11 MB of extra RSS
above baseline. No whole-pack materialization anywhere on the path.

Delta behavior, as-built:

- `iter_from_counts` **copies existing pack deltas** (`from_pack_entry` —
  ofs-delta entries referencing bases also in the pack, translated through
  the counts index) and **deflates loose objects as whole bases**
  (`from_data`). It does **not synthesize new deltas** between loose objects
  — gitoxide has no delta *encoding* API for the server path (`gix-delta`
  is apply/decode only; there is no `encode` in the published crates).
- Empirical effect on the 60k-object fixture: repo `git gc`'d to one pack
  (5.5 MB, 29,981 chain-1 deltas) → our generated pack 3.96 MB (deltas
  copied through, compressed bases for loose objects); same repo all-loose →
  generated pack 5.33 MB. Serving a fully packed repo is *smaller* than what
  git stores because our walk writes every entry as-is without bitmap
  reuse and without delta re-selection — and both clone cleanly.
- For phase 1: fresh clones of loose-ish repos ship uncompressed bases
  (fine for v1; `git` on the client re-packs at rest if `fetch.unpackLimit`
  says so), and repos kept packed get pack-copy for free. Delta *synthesis*
  (window search over loose objects) is a real gap vs `git upload-pack` —
  record as an optimization backlog item, not a blocker.

## Sideband streaming over the POC-1 wire path (verified shape)

The fetch response composes exactly as POC-1's protocol inventory described,
now with real generation feeding it:

- `packfile\n` text line, then band-1 chunks. `SidebandSink` (an `io::Write`
  adapter) caps chunks at 65000 bytes (git's own cap is 65515; headroom for
  the 4-byte length prefix under the 65520 pkt-line max) and bridges to
  `gix_packetline::async_io::encode::band_to_write` via
  `futures_lite::future::block_on` — safe because exactly one task drives the
  sink. `flush()` is idempotent (the first flush-pkt closes the section; a
  second would be a protocol error — caught by the selftest's wire decoder).
- **The generator is sync; run it on `tokio::task::spawn_blocking`.** The
  async side passes the compat writer into the blocking task and awaits the
  result. This also resolves a type-system fact worth encoding now:
  `gix_odb::Cache` is deliberately **not `Sync`** (per-thread `RefCell`
  pack/object caches by design), so it cannot be held across an `.await` in
  a `Send` future. The handler therefore holds `Arc<gix_odb::Store>`
  (thread-safe: `parking_lot::Mutex` + `ArcSwap` + atomics) and builds a
  `Cache` handle per connection/generation (`store.to_handle_arc()` +
  `prevent_pack_unload()`), moving the owned `Cache` into the blocking task.
  This is the exact shape `alkgit-transport` should encode: **store shared,
  handle per session, generation on blocking threads.**
- End-to-end over the alkcall surface (`Connection::from_bidi` → `accept_bi`
  → `BiStream` → compat → packetline): unchanged from POC-1; the pack path
  adds only the sink.

## gix-pack API surface notes for `alkgit-core` (phase-1 input)

1. **Pipeline pieces and paths** (pinned versions from `gitoxide.md`):
   `gix_pack::data::output::count::objects_unthreaded` (or the threaded
   `count::objects` for scale), `entry::iter_from_counts` (needs `generate`
   + `parallel` features; yields `Result<(SequenceId, Vec<Entry>), Error>`),
   `gix_features::parallel::InOrderIter` to re-sequence, and
   `bytes::FromEntriesIter::new(input, out, num_entries, Version::V2,
   Kind::Sha1)` (asserts V2 — only V2 can be written). Composition is exactly
   `gitoxide-core/src/pack/create.rs`; there is no higher-level "write pack
   to socket" API upstream.
2. **`FromEntriesIter` writes header + count + trailer and flushes the
   writer**; the trailer is `digest()` after the final `next()` returns
   `None`. `iter_from_counts`' `finalize()` (via
   `gix_features::parallel::reduce::Finalize`) yields entry statistics
   (copied-from-pack / recompressed / missing counts) — free server metrics.
3. **Progress plumbing**: `Box<dyn DynNestedProgress>`; `Discard` works
   everywhere. Server will want a real sink for session progress/budget
   later.
4. **Error taxonomy**: generation errors are `thiserror` enums per stage;
   `missing_objects` do not abort — they become "invalid" entries (null-id
   sentinel) which `FromEntriesIter` skips while blocking deltas against
   them. For us: a missing object mid-generation should abort the fetch with
   a sideband error band, not silently produce a broken pack — check the
   entry stats and error out if `missing_objects > 0`.
5. **Determinism**: single-threaded count (`objects_unthreaded`) gives
   deterministic order; the threaded `count::objects` + multi-thread
   `iter_from_counts` reorder chunks (hence `InOrderIter`). For a server
   with per-connection tasks, thread_limit=1 in-process plus the blocking
   pool is plenty at these speeds (60k objects in 5.5 s debug, 1.4 s for
   15k release).
6. **`Cache` is not `Sync`** — see above. Also `Bundle::write_to_directory`
   requires the `streaming-input` feature (default on upstream; we run
   lean, so list features explicitly: `sha1`, `generate`, `parallel`,
   `streaming-input`).

## What the POC does NOT settle

- **Want/have negotiation**: `haves` are parsed and recorded but never
  subtracted (we advertise `wait-for-done` and send full closures on `done`,
  the POC-1 wire shape). Multi-round ack logic stays phase-1 work; the
  generator is negotiation-agnostic (boundary sets in → pack out).
- **Delta synthesis** for loose objects (above) — optimization backlog.
- **sha256**: pipeline is hash-generic (`object_hash` parameter) but
  untested here; the `sha256` feature flag passthrough remains an OQ.
- **Concurrency**: one pack generation per connection on blocking threads;
  a shared-process thread budget per repo is an architecture concern
  (bounded-resources invariant), not a POC question.

## What it changes in the research docs

- `git-protocol.md`: pack generation option list resolved — (b) wins;
  record the two-stage closure (commit ancestry + TreeContents), the
  `prevent_pack_unload()`/replacements requirements, `bundle::write`'s
  actual role (receive-pack indexing), and the missing-delta-synthesis note.
- `gitoxide.md`: add generation-pipeline notes (feature set, composition,
  `Cache` !Sync, `Version::V2`-only writer, entry statistics via
  `finalize()`).
- `pocs.md`: POC-2 outcome recorded.

## Follow-ups for phase 1 (architecture input)

1. `alkgit-core` should expose a `PackWriter` type: `(odb handle, wants,
   haves) -> impl io::Write-streaming pack`, encapsulating tip peeling,
   commit walk, count, entries, bytes, and the missing-objects check. The
   POC's `packgen::generate_streaming` is the seed.
2. `alkgit-transport` fetch handler: generation belongs on
   `spawn_blocking`; the handler holds `Arc<Store>` and builds per-session
   handles. Budget the blocking pool (bounded-resources invariant).
3. Sideband chunk size (65000) and any max-pack-size cap should be
   server-configurable, flowing through the same limits struct as POC-1's
   session limits.
4. Consider `pack.cache` (LRU) sizing on the odb handle for hot-repo
   fetches; gitoxide exposes `set_pack_cache`/`set_object_cache` with byte
   budgets (gitoxide-core wires exactly this in `pack/create.rs`).

## Verification transcript (2026-09-20)

```
$ cargo run -- selftest          # small fixture: gen -> bundle roundtrip ->
    git index-pack --strict -> verify-pack -> unpack-objects --strict ->
    fsck --strict -> closure check -> sideband wire decode
  ... sideband framing OK: 763 band-1 bytes in chunks <= 65001, pack bytes 763
$ cargo run -- stats /tmp/…/fixture-large.git
  closure (own walk): 60000 == gitoxide count::objects: 60000
$ cargo run --release -- bench /tmp/…/fixture-large.git   # 10k commits
  gen: 60000 objects, 5334304 pack bytes, 5517 ms, RSS peak=16364kB
$ cargo run -- bridge 9423 /tmp/…/fixture-large.git       # packed 60k-object repo
$ git -c protocol.version=2 clone git://127.0.0.1:9423/repo.git c
    -> completes (6.1 s wall); git fsck --strict OK
$ cargo run -- bridge 9422 /tmp/…/fixture.git             # small fixture, tags
$ git -c protocol.version=2 clone git://127.0.0.1:9422/repo.git c2
    -> clone; git fsck --strict OK; git checkout v1.0 OK (annotated tag)
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev
    -> * [new branch] dev -> newdev; git fsck --strict OK
```

Traces: `GIT_TRACE_PACKET=1` shows the client receiving the POC-2
advertisement (`agent=alkgit-poc2/0.1`), ls-refs with `peeled:`/`symref-target:`
attributes, and the `packfile` section over band-1 sideband with flush
termination — identical shape to POC-1's captured ground truth, now fed by
real generation.