diff --git a/docs/research/git-protocol.md b/docs/research/git-protocol.md index 040630c..642e258 100644 --- a/docs/research/git-protocol.md +++ b/docs/research/git-protocol.md @@ -91,6 +91,30 @@ the surface is small enough to own outright with gitoxide primitives: b. `gix-pack::data::output::bytes` entries-to-bytes writer fed by an object walk over `gix-odb` (full control over want/have closure; no file intermediates). + **Resolved (POC-2, `poc2-findings.md`)**: (b) wins — it is gitoxide's own + generation pipeline (`count::objects` → `entry::iter_from_counts` → + `bytes::FromEntriesIter`, as composed by `gix pack create`) and streams to + any `io::Write` with O(counts) memory. (a) is not a generation tool: it + indexes an *existing pack stream* (receive-pack's job) and always writes a + temp file (it mmaps the pack to resolve deltas for the index). +- **The closure is two stages**: commit ancestry + (`gix_traverse::commit::Simple`, `Parents::All`, tips peeled to commits) + feeding `ObjectExpansion::TreeContents` (which expands each commit's own + tree but does **not** follow parents). Wants alone through `TreeContents` + produce an incomplete pack that `git index-pack --strict` rejects. +- **odb handle prerequisites** for generation: + `prevent_pack_unload()` + `ignore_replacements = true` (asserts/corruption + otherwise). `gix_odb::Cache` is not `Sync` (per-thread `RefCell` caches): + share the `Arc`, build a handle per session, generate on blocking + threads. +- **Delta behavior**: existing pack deltas are copied through + (`from_pack_entry`); loose objects are written as compressed bases. No + delta *synthesis* exists upstream (`gix-delta` is apply-only) — an + optimization backlog item vs `git upload-pack`, not a correctness gap. +- Missing objects during generation do not abort by default (they become + skipped "invalid" entries); the server must check entry statistics + (`missing_objects`) and abort the fetch with a sideband error band instead + of emitting a broken pack. - Thin packs (delta against client haves) are an optimization — v0 can declare `thin-pack` unsupported initially; V2 fetch arg parsing must still accept/decline it gracefully. @@ -126,7 +150,8 @@ Minimal correct initial policy: - OQ: support V0/V1 at all on http (or V2-only http, V0/V1 ssh)? - OQ: shallow clone support timeline. - OQ: thin-pack on fetch (requires delta against client have set). -- OQ: pack streaming composition (POC-1 outcome decides bundle-write vs - entries-to-bytes). +- Resolved: pack streaming composition (POC-2 decided: entries-to-bytes + pipeline; see `poc2-findings.md`). +- OQ: delta synthesis for loose-object serving (optimization backlog). - OQ: `object-format=sha256` support policy (feature flag exists; no real ecosystem need yet — default sha1, keep flag). \ No newline at end of file diff --git a/docs/research/gitoxide.md b/docs/research/gitoxide.md index 004add1..71943da 100644 --- a/docs/research/gitoxide.md +++ b/docs/research/gitoxide.md @@ -53,10 +53,30 @@ compile-time-rejected invariant, so the config can't drift silently. - `gix-odb` — object database with loose + packed stores, dynamic multi-index loading, `Handle` with caching; the `dynamic` store refreshes on mtime changes, which suits a long-running server. + **API contract notes (POC-2, validated live):** handles used for pack + generation need `prevent_pack_unload()` and `ignore_replacements = true`. + `Cache` is deliberately **not `Sync`** (per-thread `RefCell` pack/object + caches); share `Arc` across tasks and build a handle per session — + generation belongs on blocking threads with the owned handle moved in. - `gix-pack` — pack data reading (index file, multi-index, delta resolution) and pack *writing* (`bundle::write` → `write_to_directory` for index-from-stream). Pack writing produces index+pack into a directory; streaming a pack directly to a socket needs evaluation (POC candidate). + **Generation pipeline (POC-2, validated live):** `data::output` (feature + `generate`) is the on-demand generation path: + `count::objects(_unthreaded)` (closure expansion; `TreeContents` expands + each input commit's own tree but does NOT follow parents — feed the + commit-ancestry walk through it) → `entry::iter_from_counts` (chunked + deflate + pack-delta copy, back-pressured) → `InOrderIter` → + `bytes::FromEntriesIter` (header/entries/trailer to any `io::Write`, V2 + only). Composition reference: `gitoxide-core/src/pack/create.rs`. + `bundle::write_to_directory` is the *indexing* side (consume an existing + pack stream; needs `streaming-input`) — receive-pack's tool, not fetch's. + Deltas are copied from existing packs; no delta synthesis exists upstream + (`gix-delta` is apply-only). Entry stats (`finalize()`) expose + copied/recompressed/missing counts — use `missing_objects` to abort + instead of emitting broken packs. Feature set for our lean build: + `sha1` + `generate` + `parallel` (+ `streaming-input` for bundle write). - `gix-ref` — ref store with `transaction` module (compare-and-swap semantics, reflog), which is what receive-pack needs for atomic ref updates. - `gix-object` — object parsing/encoding (commit, tree, tag, blob). @@ -99,10 +119,14 @@ compile-time-rejected invariant, so the config can't drift silently. and the request→response state machine. 3. **Server-side negotiation** — evaluating client haves against our refs (ack/NAK logic, shallow handling on the serve side). -4. **On-demand pack generation for a fetch** — pack writing exists +4. **On-demand pack generation for a fetch** — ~~pack writing exists (`gix-pack bundle::write`) but "stream a pack computed from a want/have set to a socket" composition needs a POC; worst case we walk objects ourselves - via odb and feed `gix-pack::data::output::bytes` entries-to-bytes writers. + via odb and feed `gix-pack::data::output::bytes` entries-to-bytes writers.~~ + **Resolved (POC-2)**: the `data::output` pipeline streams to any + `io::Write`; we own only the orchestration (peel tips → commit walk → + count → entries → bytes) and the missing-objects check. Delta *synthesis* + for loose objects remains an upstream gap (optimization backlog). 5. **receive-pack** — parsing client pack stream (`gix-pack::data::input` can parse a pack from a reader), fsck, ref CAS updates via `gix-ref` transaction, reporting status (`unpack ok`/`ng` lines). diff --git a/docs/research/poc2-findings.md b/docs/research/poc2-findings.md new file mode 100644 index 0000000..6ab93f8 --- /dev/null +++ b/docs/research/poc2-findings.md @@ -0,0 +1,281 @@ +# POC-2 findings: server-side pack generation + +**Status**: complete — **proceed** +**Date**: 2026-09-20 +**Mode**: standalone (`/workspace/alkgit-poc2`, disposable per `pocs.md`; this +document is the deliverable on main) +**Depends on**: POC-1 (pkt-line over alkcall BiStream, `poc-1-findings.md`) + +## Hypothesis (restated) + +Given a fixture repo and a want/have set, produce a valid pack stream that +`git verify-pack`/`git unpack-objects` accepts, using either: +a. `gix-pack::bundle::write` into a temp dir then read the file back + (correctness baseline), or +b. `gix-pack::data::output::bytes` fed by an odb object walk (streaming), +with a decision on streaming composition that doesn't materialize the whole +pack in memory for large repos — memory behavior recorded for a 10k-object +repo at minimum. + +## Result + +**Confirmed, with one course correction.** Real `git` 2.43 clones, fetches, +and tag-checkouts against a bridge that generates packs with gitoxide's own +generation pipeline, streamed over sideband without ever holding the whole +pack in memory. `git fsck --strict` passes on every clone; `git index-pack +--strict` and `git verify-pack` accept the generated bytes standalone. + +| Check | Result | +|---|---| +| `git index-pack --strict` on generated pack | OK (both fixtures) | +| `git verify-pack -v` | `ok`, object list + sizes verified | +| `git unpack-objects --strict` + `git fsck --strict` | OK | +| `git clone` (V2, full handshake) from the bridge | OK, 10k commits / 60k objects | +| `git fsck --strict` in the clone | OK | +| `git fetch` (incremental ref fetch) | OK, ref updated, fsck clean | +| `git checkout ` | OK, checks out at the peeled commit | +| in-process sideband selftest | band-1 framing exact, flush-terminated, no stray bytes | +| closure cross-check | own BFS walk == gitoxide count, 15k/60k objects | +| approach (a) `Bundle::write_to_directory` round-trip | indexes our pack, 0 objects materialized in memory | + +## The composition that works (approach b, as built) + +gitoxide already ships a complete pack *generation* pipeline behind +`gix-pack`'s `generate` feature. The serving path is three stages chained as +iterators, ending in plain `io::Write`: + +``` +want tips (peeled to commits) + -> gix_traverse::commit::Simple (Parents::All) [commit ancestry] + -> gix_pack::data::output::count::objects_unthreaded(TreeContents) + [per-commit tree + subtree + blob expansion -> Vec] + -> gix_pack::data::output::entry::iter_from_counts [chunked deflate, + delta-copy from existing packs, back-pressured] + -> gix_features::parallel::InOrderIter [re-order chunk results] + -> gix_pack::data::output::bytes::FromEntriesIter [header, entries, + trailer sha1 -> W: io::Write] + -> SidebandSink (ours) [65000-byte band-1 + pkt-line chunks] +``` + +This is exactly `gitoxide-core/src/pack/create.rs`'s composition (`gix pack +create`) — it works against any `gix_pack::Find` implementor, i.e. against +`gix_odb::Cache>>` directly. No temp files, no `git` +binary, no full-pack buffer. + +### The course correction: `TreeContents` does not follow commit parents + +The obvious call — feed wants into `count::objects(_unthreaded)` with +`ObjectExpansion::TreeContents` and expect the full reachable closure — +produces an **incomplete pack**. `TreeContents` expands each *input object's* +own tree (commit → its tree → subtrees/blobs; tag → target's tree); it does +**not** traverse commit ancestry. Feeding tip commits alone drops all ancestor +commits, their trees, and their blobs. Real git's `index-pack --strict` +catches this (`did not receive expected object `). + +Correct closure = **commit-ancestry walk** (`gix_traverse::commit::Simple` +with `Parents::All`) whose output commits, plus any non-commit tips +(annotated tags), feed the `TreeContents` count stage. This mirrors what +`gix pack create` does and what POC-1's hand-rolled walker did. Recorded in +`git-protocol.md`'s inventory (below) — the want/have-set walk is two stages, +not one. + +Two smaller API facts along the same line: + +1. **Tips must be peeled before the commit walk.** `commit::Simple` requires + `find_commit_iter`; handing it an annotated-tag id errors with + `Expected object of kind commit but got tag`. Peel via `TagRef::target()` + in a loop until a commit (or bare tree/blob — no ancestry, count as-is). +2. **The odb handle needs `prevent_pack_unload()` + `ignore_replacements = true`** + before use in generation (`location_by_oid` asserts the former; the latter + matches `gix pack create`). Forgetting the first panics on the first + packed-object lookup; the fixture that works loose-only silently passes + until a packed repo is used. + +## Approach (a) baseline: `Bundle::write_to_directory` + +Used as the correctness check, not as the serving path. It consumes an +*existing pack stream* (it is the index-from-stream machinery: +`data::input::BytesToEntriesIter` → index write → pack file persist), so it +is the wrong tool for on-demand generation — the pipeline above is the +generation path. It needs the `streaming-input` feature and +`Option::::None` as the thin-pack lookup. Feeding it +our generated pack: indexes cleanly, `index_hash`/`data_hash` match git's +`index-pack` output (both `997bc209…` / `05c910d8…`-style trailer hashes on +the same pack). Conclusion: **bundle::write is for receive-pack (POC on pack +ingestion), not for fetch-side generation.** `git-protocol.md`'s option list +said "bundle::write with an in-memory sink" — that reading was wrong; it +writes to a temp file per design (it mmaps the pack to resolve deltas for the +index). + +## Streaming & memory behavior (the decision POC-2 was for) + +**Streaming composition is real and needs no new machinery.** `FromEntriesIter` +is a lazy `Iterator>` over any `io::Write`; `iter_from_counts` +yields entries chunk-by-chunk with back-pressure (one chunk of compressed +entries in memory at a time, default chunk 10). The only in-memory artifact is +`Vec` (24 B/object: 20-byte id + pack-location option) — for 60k +objects that's ~1.4 MB, dwarfed by the pack itself. + +Measured (release build, `/proc/self/status` VmRSS, 5 ms sampler thread): + +| Fixture | Objects | Pack bytes | Time | RSS before | RSS peak | RSS after | +|---|---|---|---|---|---|---| +| 2.5k commits, 15k objects | 15,000 | 1,327,523 | 1.4 s | 5.5 MB | 9.3 MB | 7.6 MB | +| 4k commits, 24k objects | 24,000 | 2,128,268 | 2.5 s | 5.4 MB | 9.7 MB | 7.6 MB | +| 10k commits, 60k objects | 60,000 | 5,334,304 | 5.5 s | 5.4 MB | 16.4 MB | 9.0 MB | + +Peak ≈ base + ~(counts × ~200 B) + object buffers — i.e. **O(counts), not +O(pack)**: the 5.3 MB pack for 60k objects went to the socket through a +64 KB-bounded sink (65000-byte sideband chunks) with ~11 MB of extra RSS +above baseline. No whole-pack materialization anywhere on the path. + +Delta behavior, as-built: + +- `iter_from_counts` **copies existing pack deltas** (`from_pack_entry` — + ofs-delta entries referencing bases also in the pack, translated through + the counts index) and **deflates loose objects as whole bases** + (`from_data`). It does **not synthesize new deltas** between loose objects + — gitoxide has no delta *encoding* API for the server path (`gix-delta` + is apply/decode only; there is no `encode` in the published crates). +- Empirical effect on the 60k-object fixture: repo `git gc`'d to one pack + (5.5 MB, 29,981 chain-1 deltas) → our generated pack 3.96 MB (deltas + copied through, compressed bases for loose objects); same repo all-loose → + generated pack 5.33 MB. Serving a fully packed repo is *smaller* than what + git stores because our walk writes every entry as-is without bitmap + reuse and without delta re-selection — and both clone cleanly. +- For phase 1: fresh clones of loose-ish repos ship uncompressed bases + (fine for v1; `git` on the client re-packs at rest if `fetch.unpackLimit` + says so), and repos kept packed get pack-copy for free. Delta *synthesis* + (window search over loose objects) is a real gap vs `git upload-pack` — + record as an optimization backlog item, not a blocker. + +## Sideband streaming over the POC-1 wire path (verified shape) + +The fetch response composes exactly as POC-1's protocol inventory described, +now with real generation feeding it: + +- `packfile\n` text line, then band-1 chunks. `SidebandSink` (an `io::Write` + adapter) caps chunks at 65000 bytes (git's own cap is 65515; headroom for + the 4-byte length prefix under the 65520 pkt-line max) and bridges to + `gix_packetline::async_io::encode::band_to_write` via + `futures_lite::future::block_on` — safe because exactly one task drives the + sink. `flush()` is idempotent (the first flush-pkt closes the section; a + second would be a protocol error — caught by the selftest's wire decoder). +- **The generator is sync; run it on `tokio::task::spawn_blocking`.** The + async side passes the compat writer into the blocking task and awaits the + result. This also resolves a type-system fact worth encoding now: + `gix_odb::Cache` is deliberately **not `Sync`** (per-thread `RefCell` + pack/object caches by design), so it cannot be held across an `.await` in + a `Send` future. The handler therefore holds `Arc` + (thread-safe: `parking_lot::Mutex` + `ArcSwap` + atomics) and builds a + `Cache` handle per connection/generation (`store.to_handle_arc()` + + `prevent_pack_unload()`), moving the owned `Cache` into the blocking task. + This is the exact shape `alkgit-transport` should encode: **store shared, + handle per session, generation on blocking threads.** +- End-to-end over the alkcall surface (`Connection::from_bidi` → `accept_bi` + → `BiStream` → compat → packetline): unchanged from POC-1; the pack path + adds only the sink. + +## gix-pack API surface notes for `alkgit-core` (phase-1 input) + +1. **Pipeline pieces and paths** (pinned versions from `gitoxide.md`): + `gix_pack::data::output::count::objects_unthreaded` (or the threaded + `count::objects` for scale), `entry::iter_from_counts` (needs `generate` + + `parallel` features; yields `Result<(SequenceId, Vec), Error>`), + `gix_features::parallel::InOrderIter` to re-sequence, and + `bytes::FromEntriesIter::new(input, out, num_entries, Version::V2, + Kind::Sha1)` (asserts V2 — only V2 can be written). Composition is exactly + `gitoxide-core/src/pack/create.rs`; there is no higher-level "write pack + to socket" API upstream. +2. **`FromEntriesIter` writes header + count + trailer and flushes the + writer**; the trailer is `digest()` after the final `next()` returns + `None`. `iter_from_counts`' `finalize()` (via + `gix_features::parallel::reduce::Finalize`) yields entry statistics + (copied-from-pack / recompressed / missing counts) — free server metrics. +3. **Progress plumbing**: `Box`; `Discard` works + everywhere. Server will want a real sink for session progress/budget + later. +4. **Error taxonomy**: generation errors are `thiserror` enums per stage; + `missing_objects` do not abort — they become "invalid" entries (null-id + sentinel) which `FromEntriesIter` skips while blocking deltas against + them. For us: a missing object mid-generation should abort the fetch with + a sideband error band, not silently produce a broken pack — check the + entry stats and error out if `missing_objects > 0`. +5. **Determinism**: single-threaded count (`objects_unthreaded`) gives + deterministic order; the threaded `count::objects` + multi-thread + `iter_from_counts` reorder chunks (hence `InOrderIter`). For a server + with per-connection tasks, thread_limit=1 in-process plus the blocking + pool is plenty at these speeds (60k objects in 5.5 s debug, 1.4 s for + 15k release). +6. **`Cache` is not `Sync`** — see above. Also `Bundle::write_to_directory` + requires the `streaming-input` feature (default on upstream; we run + lean, so list features explicitly: `sha1`, `generate`, `parallel`, + `streaming-input`). + +## What the POC does NOT settle + +- **Want/have negotiation**: `haves` are parsed and recorded but never + subtracted (we advertise `wait-for-done` and send full closures on `done`, + the POC-1 wire shape). Multi-round ack logic stays phase-1 work; the + generator is negotiation-agnostic (boundary sets in → pack out). +- **Delta synthesis** for loose objects (above) — optimization backlog. +- **sha256**: pipeline is hash-generic (`object_hash` parameter) but + untested here; the `sha256` feature flag passthrough remains an OQ. +- **Concurrency**: one pack generation per connection on blocking threads; + a shared-process thread budget per repo is an architecture concern + (bounded-resources invariant), not a POC question. + +## What it changes in the research docs + +- `git-protocol.md`: pack generation option list resolved — (b) wins; + record the two-stage closure (commit ancestry + TreeContents), the + `prevent_pack_unload()`/replacements requirements, `bundle::write`'s + actual role (receive-pack indexing), and the missing-delta-synthesis note. +- `gitoxide.md`: add generation-pipeline notes (feature set, composition, + `Cache` !Sync, `Version::V2`-only writer, entry statistics via + `finalize()`). +- `pocs.md`: POC-2 outcome recorded. + +## Follow-ups for phase 1 (architecture input) + +1. `alkgit-core` should expose a `PackWriter` type: `(odb handle, wants, + haves) -> impl io::Write-streaming pack`, encapsulating tip peeling, + commit walk, count, entries, bytes, and the missing-objects check. The + POC's `packgen::generate_streaming` is the seed. +2. `alkgit-transport` fetch handler: generation belongs on + `spawn_blocking`; the handler holds `Arc` and builds per-session + handles. Budget the blocking pool (bounded-resources invariant). +3. Sideband chunk size (65000) and any max-pack-size cap should be + server-configurable, flowing through the same limits struct as POC-1's + session limits. +4. Consider `pack.cache` (LRU) sizing on the odb handle for hot-repo + fetches; gitoxide exposes `set_pack_cache`/`set_object_cache` with byte + budgets (gitoxide-core wires exactly this in `pack/create.rs`). + +## Verification transcript (2026-09-20) + +``` +$ cargo run -- selftest # small fixture: gen -> bundle roundtrip -> + git index-pack --strict -> verify-pack -> unpack-objects --strict -> + fsck --strict -> closure check -> sideband wire decode + ... sideband framing OK: 763 band-1 bytes in chunks <= 65001, pack bytes 763 +$ cargo run -- stats /tmp/…/fixture-large.git + closure (own walk): 60000 == gitoxide count::objects: 60000 +$ cargo run --release -- bench /tmp/…/fixture-large.git # 10k commits + gen: 60000 objects, 5334304 pack bytes, 5517 ms, RSS peak=16364kB +$ cargo run -- bridge 9423 /tmp/…/fixture-large.git # packed 60k-object repo +$ git -c protocol.version=2 clone git://127.0.0.1:9423/repo.git c + -> completes (6.1 s wall); git fsck --strict OK +$ cargo run -- bridge 9422 /tmp/…/fixture.git # small fixture, tags +$ git -c protocol.version=2 clone git://127.0.0.1:9422/repo.git c2 + -> clone; git fsck --strict OK; git checkout v1.0 OK (annotated tag) +$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev + -> * [new branch] dev -> newdev; git fsck --strict OK +``` + +Traces: `GIT_TRACE_PACKET=1` shows the client receiving the POC-2 +advertisement (`agent=alkgit-poc2/0.1`), ls-refs with `peeled:`/`symref-target:` +attributes, and the `packfile` section over band-1 sideband with flush +termination — identical shape to POC-1's captured ground truth, now fed by +real generation. \ No newline at end of file diff --git a/docs/research/pocs.md b/docs/research/pocs.md index 49a31d2..52e66fe 100644 --- a/docs/research/pocs.md +++ b/docs/research/pocs.md @@ -53,6 +53,14 @@ compatibility (they should, being tokio io objects; may need a thin adapter). ## POC-2: server-side pack generation +**Status**: complete (2026-09-20) — **proceed**. See `poc2-findings.md`. +Real git 2.43 clones/fetches (fsck-strict clean) packs generated by +gitoxide's own pipeline (`count::objects` + `iter_from_counts` + +`FromEntriesIter`) streamed over sideband; memory is O(counts), not +O(pack) — 60k-object pack written through a 64 KB sink at ~11 MB extra RSS. +Course correction recorded: `TreeContents` does not follow commit parents — +the closure needs a commit-ancestry walk feeding the count stage. + **Hypothesis**: given a fixture repo and a want/have set, we can produce a valid pack stream in memory that `git verify-pack`/`git unpack-objects` accepts, using one of: @@ -91,6 +99,9 @@ POC-1 first (the BiStream/packetline fit is the load-bearing assumption). POC-2 next (the crux of fetch). POC-3 last (http shape). Each POC updates the corresponding research doc with results and a proceed/pivot/block note. +**Status**: POC-1 and POC-2 complete and proceed; POC-3 is the remaining +gate for phase 0 completion. + ## Modes applied to these POCs - POC-1, POC-2: **standalone mode** — nothing in the alkgit repo exists to