glm-5.3-flash
3e68cb5b68
POC-2 (standalone mode, /workspace/alkgit-poc2) confirms the fetch crux: gitoxide's own generation pipeline streams valid packs to any io::Write, and real git 2.43 clones/fetches over the POC-1 bridge with fsck --strict passing on every clone. - Findings: docs/research/poc2-findings.md (proceed) - git-protocol.md: pack-generation option list resolved ((b) wins); record two-stage closure (commit ancestry -> TreeContents), odb handle prerequisites, bundle::write's real role (receive-pack indexing), missing-delta-synthesis note - gitoxide.md: generation pipeline + gix-odb API contract notes (Cache !Sync, prevent_pack_unload, missing_objects check) - pocs.md: POC-2 outcome recorded; POC-3 is the remaining phase-0 gate Verification: cargo run -- selftest (gen -> bundle roundtrip -> index-pack --strict -> verify-pack -> unpack-objects --strict -> fsck -> closure cross-check -> sideband wire decode); 10k-commit/60k-object fixture: closure 60000==60000, pack 5.33MB in 5.5s at ~11MB extra RSS; real git clones (loose and packed sources) + tag checkout + incremental fetch all fsck-clean over git:// through the alkcall Connection/BiStream path
281 lines
15 KiB
Markdown
281 lines
15 KiB
Markdown
# POC-2 findings: server-side pack generation
|
||
|
||
**Status**: complete — **proceed**
|
||
**Date**: 2026-09-20
|
||
**Mode**: standalone (`/workspace/alkgit-poc2`, disposable per `pocs.md`; this
|
||
document is the deliverable on main)
|
||
**Depends on**: POC-1 (pkt-line over alkcall BiStream, `poc-1-findings.md`)
|
||
|
||
## Hypothesis (restated)
|
||
|
||
Given a fixture repo and a want/have set, produce a valid pack stream that
|
||
`git verify-pack`/`git unpack-objects` accepts, using either:
|
||
a. `gix-pack::bundle::write` into a temp dir then read the file back
|
||
(correctness baseline), or
|
||
b. `gix-pack::data::output::bytes` fed by an odb object walk (streaming),
|
||
with a decision on streaming composition that doesn't materialize the whole
|
||
pack in memory for large repos — memory behavior recorded for a 10k-object
|
||
repo at minimum.
|
||
|
||
## Result
|
||
|
||
**Confirmed, with one course correction.** Real `git` 2.43 clones, fetches,
|
||
and tag-checkouts against a bridge that generates packs with gitoxide's own
|
||
generation pipeline, streamed over sideband without ever holding the whole
|
||
pack in memory. `git fsck --strict` passes on every clone; `git index-pack
|
||
--strict` and `git verify-pack` accept the generated bytes standalone.
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| `git index-pack --strict` on generated pack | OK (both fixtures) |
|
||
| `git verify-pack -v` | `ok`, object list + sizes verified |
|
||
| `git unpack-objects --strict` + `git fsck --strict` | OK |
|
||
| `git clone` (V2, full handshake) from the bridge | OK, 10k commits / 60k objects |
|
||
| `git fsck --strict` in the clone | OK |
|
||
| `git fetch` (incremental ref fetch) | OK, ref updated, fsck clean |
|
||
| `git checkout <annotated tag>` | OK, checks out at the peeled commit |
|
||
| in-process sideband selftest | band-1 framing exact, flush-terminated, no stray bytes |
|
||
| closure cross-check | own BFS walk == gitoxide count, 15k/60k objects |
|
||
| approach (a) `Bundle::write_to_directory` round-trip | indexes our pack, 0 objects materialized in memory |
|
||
|
||
## The composition that works (approach b, as built)
|
||
|
||
gitoxide already ships a complete pack *generation* pipeline behind
|
||
`gix-pack`'s `generate` feature. The serving path is three stages chained as
|
||
iterators, ending in plain `io::Write`:
|
||
|
||
```
|
||
want tips (peeled to commits)
|
||
-> gix_traverse::commit::Simple (Parents::All) [commit ancestry]
|
||
-> gix_pack::data::output::count::objects_unthreaded(TreeContents)
|
||
[per-commit tree + subtree + blob expansion -> Vec<Count>]
|
||
-> gix_pack::data::output::entry::iter_from_counts [chunked deflate,
|
||
delta-copy from existing packs, back-pressured]
|
||
-> gix_features::parallel::InOrderIter [re-order chunk results]
|
||
-> gix_pack::data::output::bytes::FromEntriesIter [header, entries,
|
||
trailer sha1 -> W: io::Write]
|
||
-> SidebandSink (ours) [65000-byte band-1
|
||
pkt-line chunks]
|
||
```
|
||
|
||
This is exactly `gitoxide-core/src/pack/create.rs`'s composition (`gix pack
|
||
create`) — it works against any `gix_pack::Find` implementor, i.e. against
|
||
`gix_odb::Cache<Handle<Arc<Store>>>` directly. No temp files, no `git`
|
||
binary, no full-pack buffer.
|
||
|
||
### The course correction: `TreeContents` does not follow commit parents
|
||
|
||
The obvious call — feed wants into `count::objects(_unthreaded)` with
|
||
`ObjectExpansion::TreeContents` and expect the full reachable closure —
|
||
produces an **incomplete pack**. `TreeContents` expands each *input object's*
|
||
own tree (commit → its tree → subtrees/blobs; tag → target's tree); it does
|
||
**not** traverse commit ancestry. Feeding tip commits alone drops all ancestor
|
||
commits, their trees, and their blobs. Real git's `index-pack --strict`
|
||
catches this (`did not receive expected object <ancestor>`).
|
||
|
||
Correct closure = **commit-ancestry walk** (`gix_traverse::commit::Simple`
|
||
with `Parents::All`) whose output commits, plus any non-commit tips
|
||
(annotated tags), feed the `TreeContents` count stage. This mirrors what
|
||
`gix pack create` does and what POC-1's hand-rolled walker did. Recorded in
|
||
`git-protocol.md`'s inventory (below) — the want/have-set walk is two stages,
|
||
not one.
|
||
|
||
Two smaller API facts along the same line:
|
||
|
||
1. **Tips must be peeled before the commit walk.** `commit::Simple` requires
|
||
`find_commit_iter`; handing it an annotated-tag id errors with
|
||
`Expected object of kind commit but got tag`. Peel via `TagRef::target()`
|
||
in a loop until a commit (or bare tree/blob — no ancestry, count as-is).
|
||
2. **The odb handle needs `prevent_pack_unload()` + `ignore_replacements = true`**
|
||
before use in generation (`location_by_oid` asserts the former; the latter
|
||
matches `gix pack create`). Forgetting the first panics on the first
|
||
packed-object lookup; the fixture that works loose-only silently passes
|
||
until a packed repo is used.
|
||
|
||
## Approach (a) baseline: `Bundle::write_to_directory`
|
||
|
||
Used as the correctness check, not as the serving path. It consumes an
|
||
*existing pack stream* (it is the index-from-stream machinery:
|
||
`data::input::BytesToEntriesIter` → index write → pack file persist), so it
|
||
is the wrong tool for on-demand generation — the pipeline above is the
|
||
generation path. It needs the `streaming-input` feature and
|
||
`Option::<gix_object::find::Never>::None` as the thin-pack lookup. Feeding it
|
||
our generated pack: indexes cleanly, `index_hash`/`data_hash` match git's
|
||
`index-pack` output (both `997bc209…` / `05c910d8…`-style trailer hashes on
|
||
the same pack). Conclusion: **bundle::write is for receive-pack (POC on pack
|
||
ingestion), not for fetch-side generation.** `git-protocol.md`'s option list
|
||
said "bundle::write with an in-memory sink" — that reading was wrong; it
|
||
writes to a temp file per design (it mmaps the pack to resolve deltas for the
|
||
index).
|
||
|
||
## Streaming & memory behavior (the decision POC-2 was for)
|
||
|
||
**Streaming composition is real and needs no new machinery.** `FromEntriesIter`
|
||
is a lazy `Iterator<Item = Result<u64>>` over any `io::Write`; `iter_from_counts`
|
||
yields entries chunk-by-chunk with back-pressure (one chunk of compressed
|
||
entries in memory at a time, default chunk 10). The only in-memory artifact is
|
||
`Vec<Count>` (24 B/object: 20-byte id + pack-location option) — for 60k
|
||
objects that's ~1.4 MB, dwarfed by the pack itself.
|
||
|
||
Measured (release build, `/proc/self/status` VmRSS, 5 ms sampler thread):
|
||
|
||
| Fixture | Objects | Pack bytes | Time | RSS before | RSS peak | RSS after |
|
||
|---|---|---|---|---|---|---|
|
||
| 2.5k commits, 15k objects | 15,000 | 1,327,523 | 1.4 s | 5.5 MB | 9.3 MB | 7.6 MB |
|
||
| 4k commits, 24k objects | 24,000 | 2,128,268 | 2.5 s | 5.4 MB | 9.7 MB | 7.6 MB |
|
||
| 10k commits, 60k objects | 60,000 | 5,334,304 | 5.5 s | 5.4 MB | 16.4 MB | 9.0 MB |
|
||
|
||
Peak ≈ base + ~(counts × ~200 B) + object buffers — i.e. **O(counts), not
|
||
O(pack)**: the 5.3 MB pack for 60k objects went to the socket through a
|
||
64 KB-bounded sink (65000-byte sideband chunks) with ~11 MB of extra RSS
|
||
above baseline. No whole-pack materialization anywhere on the path.
|
||
|
||
Delta behavior, as-built:
|
||
|
||
- `iter_from_counts` **copies existing pack deltas** (`from_pack_entry` —
|
||
ofs-delta entries referencing bases also in the pack, translated through
|
||
the counts index) and **deflates loose objects as whole bases**
|
||
(`from_data`). It does **not synthesize new deltas** between loose objects
|
||
— gitoxide has no delta *encoding* API for the server path (`gix-delta`
|
||
is apply/decode only; there is no `encode` in the published crates).
|
||
- Empirical effect on the 60k-object fixture: repo `git gc`'d to one pack
|
||
(5.5 MB, 29,981 chain-1 deltas) → our generated pack 3.96 MB (deltas
|
||
copied through, compressed bases for loose objects); same repo all-loose →
|
||
generated pack 5.33 MB. Serving a fully packed repo is *smaller* than what
|
||
git stores because our walk writes every entry as-is without bitmap
|
||
reuse and without delta re-selection — and both clone cleanly.
|
||
- For phase 1: fresh clones of loose-ish repos ship uncompressed bases
|
||
(fine for v1; `git` on the client re-packs at rest if `fetch.unpackLimit`
|
||
says so), and repos kept packed get pack-copy for free. Delta *synthesis*
|
||
(window search over loose objects) is a real gap vs `git upload-pack` —
|
||
record as an optimization backlog item, not a blocker.
|
||
|
||
## Sideband streaming over the POC-1 wire path (verified shape)
|
||
|
||
The fetch response composes exactly as POC-1's protocol inventory described,
|
||
now with real generation feeding it:
|
||
|
||
- `packfile\n` text line, then band-1 chunks. `SidebandSink` (an `io::Write`
|
||
adapter) caps chunks at 65000 bytes (git's own cap is 65515; headroom for
|
||
the 4-byte length prefix under the 65520 pkt-line max) and bridges to
|
||
`gix_packetline::async_io::encode::band_to_write` via
|
||
`futures_lite::future::block_on` — safe because exactly one task drives the
|
||
sink. `flush()` is idempotent (the first flush-pkt closes the section; a
|
||
second would be a protocol error — caught by the selftest's wire decoder).
|
||
- **The generator is sync; run it on `tokio::task::spawn_blocking`.** The
|
||
async side passes the compat writer into the blocking task and awaits the
|
||
result. This also resolves a type-system fact worth encoding now:
|
||
`gix_odb::Cache` is deliberately **not `Sync`** (per-thread `RefCell`
|
||
pack/object caches by design), so it cannot be held across an `.await` in
|
||
a `Send` future. The handler therefore holds `Arc<gix_odb::Store>`
|
||
(thread-safe: `parking_lot::Mutex` + `ArcSwap` + atomics) and builds a
|
||
`Cache` handle per connection/generation (`store.to_handle_arc()` +
|
||
`prevent_pack_unload()`), moving the owned `Cache` into the blocking task.
|
||
This is the exact shape `alkgit-transport` should encode: **store shared,
|
||
handle per session, generation on blocking threads.**
|
||
- End-to-end over the alkcall surface (`Connection::from_bidi` → `accept_bi`
|
||
→ `BiStream` → compat → packetline): unchanged from POC-1; the pack path
|
||
adds only the sink.
|
||
|
||
## gix-pack API surface notes for `alkgit-core` (phase-1 input)
|
||
|
||
1. **Pipeline pieces and paths** (pinned versions from `gitoxide.md`):
|
||
`gix_pack::data::output::count::objects_unthreaded` (or the threaded
|
||
`count::objects` for scale), `entry::iter_from_counts` (needs `generate`
|
||
+ `parallel` features; yields `Result<(SequenceId, Vec<Entry>), Error>`),
|
||
`gix_features::parallel::InOrderIter` to re-sequence, and
|
||
`bytes::FromEntriesIter::new(input, out, num_entries, Version::V2,
|
||
Kind::Sha1)` (asserts V2 — only V2 can be written). Composition is exactly
|
||
`gitoxide-core/src/pack/create.rs`; there is no higher-level "write pack
|
||
to socket" API upstream.
|
||
2. **`FromEntriesIter` writes header + count + trailer and flushes the
|
||
writer**; the trailer is `digest()` after the final `next()` returns
|
||
`None`. `iter_from_counts`' `finalize()` (via
|
||
`gix_features::parallel::reduce::Finalize`) yields entry statistics
|
||
(copied-from-pack / recompressed / missing counts) — free server metrics.
|
||
3. **Progress plumbing**: `Box<dyn DynNestedProgress>`; `Discard` works
|
||
everywhere. Server will want a real sink for session progress/budget
|
||
later.
|
||
4. **Error taxonomy**: generation errors are `thiserror` enums per stage;
|
||
`missing_objects` do not abort — they become "invalid" entries (null-id
|
||
sentinel) which `FromEntriesIter` skips while blocking deltas against
|
||
them. For us: a missing object mid-generation should abort the fetch with
|
||
a sideband error band, not silently produce a broken pack — check the
|
||
entry stats and error out if `missing_objects > 0`.
|
||
5. **Determinism**: single-threaded count (`objects_unthreaded`) gives
|
||
deterministic order; the threaded `count::objects` + multi-thread
|
||
`iter_from_counts` reorder chunks (hence `InOrderIter`). For a server
|
||
with per-connection tasks, thread_limit=1 in-process plus the blocking
|
||
pool is plenty at these speeds (60k objects in 5.5 s debug, 1.4 s for
|
||
15k release).
|
||
6. **`Cache` is not `Sync`** — see above. Also `Bundle::write_to_directory`
|
||
requires the `streaming-input` feature (default on upstream; we run
|
||
lean, so list features explicitly: `sha1`, `generate`, `parallel`,
|
||
`streaming-input`).
|
||
|
||
## What the POC does NOT settle
|
||
|
||
- **Want/have negotiation**: `haves` are parsed and recorded but never
|
||
subtracted (we advertise `wait-for-done` and send full closures on `done`,
|
||
the POC-1 wire shape). Multi-round ack logic stays phase-1 work; the
|
||
generator is negotiation-agnostic (boundary sets in → pack out).
|
||
- **Delta synthesis** for loose objects (above) — optimization backlog.
|
||
- **sha256**: pipeline is hash-generic (`object_hash` parameter) but
|
||
untested here; the `sha256` feature flag passthrough remains an OQ.
|
||
- **Concurrency**: one pack generation per connection on blocking threads;
|
||
a shared-process thread budget per repo is an architecture concern
|
||
(bounded-resources invariant), not a POC question.
|
||
|
||
## What it changes in the research docs
|
||
|
||
- `git-protocol.md`: pack generation option list resolved — (b) wins;
|
||
record the two-stage closure (commit ancestry + TreeContents), the
|
||
`prevent_pack_unload()`/replacements requirements, `bundle::write`'s
|
||
actual role (receive-pack indexing), and the missing-delta-synthesis note.
|
||
- `gitoxide.md`: add generation-pipeline notes (feature set, composition,
|
||
`Cache` !Sync, `Version::V2`-only writer, entry statistics via
|
||
`finalize()`).
|
||
- `pocs.md`: POC-2 outcome recorded.
|
||
|
||
## Follow-ups for phase 1 (architecture input)
|
||
|
||
1. `alkgit-core` should expose a `PackWriter` type: `(odb handle, wants,
|
||
haves) -> impl io::Write-streaming pack`, encapsulating tip peeling,
|
||
commit walk, count, entries, bytes, and the missing-objects check. The
|
||
POC's `packgen::generate_streaming` is the seed.
|
||
2. `alkgit-transport` fetch handler: generation belongs on
|
||
`spawn_blocking`; the handler holds `Arc<Store>` and builds per-session
|
||
handles. Budget the blocking pool (bounded-resources invariant).
|
||
3. Sideband chunk size (65000) and any max-pack-size cap should be
|
||
server-configurable, flowing through the same limits struct as POC-1's
|
||
session limits.
|
||
4. Consider `pack.cache` (LRU) sizing on the odb handle for hot-repo
|
||
fetches; gitoxide exposes `set_pack_cache`/`set_object_cache` with byte
|
||
budgets (gitoxide-core wires exactly this in `pack/create.rs`).
|
||
|
||
## Verification transcript (2026-09-20)
|
||
|
||
```
|
||
$ cargo run -- selftest # small fixture: gen -> bundle roundtrip ->
|
||
git index-pack --strict -> verify-pack -> unpack-objects --strict ->
|
||
fsck --strict -> closure check -> sideband wire decode
|
||
... sideband framing OK: 763 band-1 bytes in chunks <= 65001, pack bytes 763
|
||
$ cargo run -- stats /tmp/…/fixture-large.git
|
||
closure (own walk): 60000 == gitoxide count::objects: 60000
|
||
$ cargo run --release -- bench /tmp/…/fixture-large.git # 10k commits
|
||
gen: 60000 objects, 5334304 pack bytes, 5517 ms, RSS peak=16364kB
|
||
$ cargo run -- bridge 9423 /tmp/…/fixture-large.git # packed 60k-object repo
|
||
$ git -c protocol.version=2 clone git://127.0.0.1:9423/repo.git c
|
||
-> completes (6.1 s wall); git fsck --strict OK
|
||
$ cargo run -- bridge 9422 /tmp/…/fixture.git # small fixture, tags
|
||
$ git -c protocol.version=2 clone git://127.0.0.1:9422/repo.git c2
|
||
-> clone; git fsck --strict OK; git checkout v1.0 OK (annotated tag)
|
||
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev
|
||
-> * [new branch] dev -> newdev; git fsck --strict OK
|
||
```
|
||
|
||
Traces: `GIT_TRACE_PACKET=1` shows the client receiving the POC-2
|
||
advertisement (`agent=alkgit-poc2/0.1`), ls-refs with `peeled:`/`symref-target:`
|
||
attributes, and the `packfile` section over band-1 sideband with flush
|
||
termination — identical shape to POC-1's captured ground truth, now fed by
|
||
real generation. |