docs(research): POC-2 complete — server-side pack generation verified
POC-2 (standalone mode, /workspace/alkgit-poc2) confirms the fetch crux: gitoxide's own generation pipeline streams valid packs to any io::Write, and real git 2.43 clones/fetches over the POC-1 bridge with fsck --strict passing on every clone. - Findings: docs/research/poc2-findings.md (proceed) - git-protocol.md: pack-generation option list resolved ((b) wins); record two-stage closure (commit ancestry -> TreeContents), odb handle prerequisites, bundle::write's real role (receive-pack indexing), missing-delta-synthesis note - gitoxide.md: generation pipeline + gix-odb API contract notes (Cache !Sync, prevent_pack_unload, missing_objects check) - pocs.md: POC-2 outcome recorded; POC-3 is the remaining phase-0 gate Verification: cargo run -- selftest (gen -> bundle roundtrip -> index-pack --strict -> verify-pack -> unpack-objects --strict -> fsck -> closure cross-check -> sideband wire decode); 10k-commit/60k-object fixture: closure 60000==60000, pack 5.33MB in 5.5s at ~11MB extra RSS; real git clones (loose and packed sources) + tag checkout + incremental fetch all fsck-clean over git:// through the alkcall Connection/BiStream path
This commit is contained in:
1 parent
665f701133
commit
3e68cb5b68
4 files changed
+345
-4
No files matched your search
@@ -91,6 +91,30 @@ the surface is small enough to own outright with gitoxide primitives:
|
||||
b. `gix-pack::data::output::bytes` entries-to-bytes writer fed by an object
|
||||
walk over `gix-odb` (full control over want/have closure; no file
|
||||
intermediates).
|
||||
**Resolved (POC-2, `poc2-findings.md`)**: (b) wins — it is gitoxide's own
|
||||
generation pipeline (`count::objects` → `entry::iter_from_counts` →
|
||||
`bytes::FromEntriesIter`, as composed by `gix pack create`) and streams to
|
||||
any `io::Write` with O(counts) memory. (a) is not a generation tool: it
|
||||
indexes an *existing pack stream* (receive-pack's job) and always writes a
|
||||
temp file (it mmaps the pack to resolve deltas for the index).
|
||||
- **The closure is two stages**: commit ancestry
|
||||
(`gix_traverse::commit::Simple`, `Parents::All`, tips peeled to commits)
|
||||
feeding `ObjectExpansion::TreeContents` (which expands each commit's own
|
||||
tree but does **not** follow parents). Wants alone through `TreeContents`
|
||||
produce an incomplete pack that `git index-pack --strict` rejects.
|
||||
- **odb handle prerequisites** for generation:
|
||||
`prevent_pack_unload()` + `ignore_replacements = true` (asserts/corruption
|
||||
otherwise). `gix_odb::Cache` is not `Sync` (per-thread `RefCell` caches):
|
||||
share the `Arc<Store>`, build a handle per session, generate on blocking
|
||||
threads.
|
||||
- **Delta behavior**: existing pack deltas are copied through
|
||||
(`from_pack_entry`); loose objects are written as compressed bases. No
|
||||
delta *synthesis* exists upstream (`gix-delta` is apply-only) — an
|
||||
optimization backlog item vs `git upload-pack`, not a correctness gap.
|
||||
- Missing objects during generation do not abort by default (they become
|
||||
skipped "invalid" entries); the server must check entry statistics
|
||||
(`missing_objects`) and abort the fetch with a sideband error band instead
|
||||
of emitting a broken pack.
|
||||
- Thin packs (delta against client haves) are an optimization — v0 can
|
||||
declare `thin-pack` unsupported initially; V2 fetch arg parsing must still
|
||||
accept/decline it gracefully.
|
||||
@@ -126,7 +150,8 @@ Minimal correct initial policy:
|
||||
- OQ: support V0/V1 at all on http (or V2-only http, V0/V1 ssh)?
|
||||
- OQ: shallow clone support timeline.
|
||||
- OQ: thin-pack on fetch (requires delta against client have set).
|
||||
- OQ: pack streaming composition (POC-1 outcome decides bundle-write vs
|
||||
entries-to-bytes).
|
||||
- Resolved: pack streaming composition (POC-2 decided: entries-to-bytes
|
||||
pipeline; see `poc2-findings.md`).
|
||||
- OQ: delta synthesis for loose-object serving (optimization backlog).
|
||||
- OQ: `object-format=sha256` support policy (feature flag exists; no real
|
||||
ecosystem need yet — default sha1, keep flag).
|
||||
@@ -53,10 +53,30 @@ compile-time-rejected invariant, so the config can't drift silently.
|
||||
- `gix-odb` — object database with loose + packed stores, dynamic multi-index
|
||||
loading, `Handle` with caching; the `dynamic` store refreshes on mtime
|
||||
changes, which suits a long-running server.
|
||||
**API contract notes (POC-2, validated live):** handles used for pack
|
||||
generation need `prevent_pack_unload()` and `ignore_replacements = true`.
|
||||
`Cache` is deliberately **not `Sync`** (per-thread `RefCell` pack/object
|
||||
caches); share `Arc<Store>` across tasks and build a handle per session —
|
||||
generation belongs on blocking threads with the owned handle moved in.
|
||||
- `gix-pack` — pack data reading (index file, multi-index, delta resolution)
|
||||
and pack *writing* (`bundle::write` → `write_to_directory` for
|
||||
index-from-stream). Pack writing produces index+pack into a directory;
|
||||
streaming a pack directly to a socket needs evaluation (POC candidate).
|
||||
**Generation pipeline (POC-2, validated live):** `data::output` (feature
|
||||
`generate`) is the on-demand generation path:
|
||||
`count::objects(_unthreaded)` (closure expansion; `TreeContents` expands
|
||||
each input commit's own tree but does NOT follow parents — feed the
|
||||
commit-ancestry walk through it) → `entry::iter_from_counts` (chunked
|
||||
deflate + pack-delta copy, back-pressured) → `InOrderIter` →
|
||||
`bytes::FromEntriesIter` (header/entries/trailer to any `io::Write`, V2
|
||||
only). Composition reference: `gitoxide-core/src/pack/create.rs`.
|
||||
`bundle::write_to_directory` is the *indexing* side (consume an existing
|
||||
pack stream; needs `streaming-input`) — receive-pack's tool, not fetch's.
|
||||
Deltas are copied from existing packs; no delta synthesis exists upstream
|
||||
(`gix-delta` is apply-only). Entry stats (`finalize()`) expose
|
||||
copied/recompressed/missing counts — use `missing_objects` to abort
|
||||
instead of emitting broken packs. Feature set for our lean build:
|
||||
`sha1` + `generate` + `parallel` (+ `streaming-input` for bundle write).
|
||||
- `gix-ref` — ref store with `transaction` module (compare-and-swap semantics,
|
||||
reflog), which is what receive-pack needs for atomic ref updates.
|
||||
- `gix-object` — object parsing/encoding (commit, tree, tag, blob).
|
||||
@@ -99,10 +119,14 @@ compile-time-rejected invariant, so the config can't drift silently.
|
||||
and the request→response state machine.
|
||||
3. **Server-side negotiation** — evaluating client haves against our refs
|
||||
(ack/NAK logic, shallow handling on the serve side).
|
||||
4. **On-demand pack generation for a fetch** — pack writing exists
|
||||
4. **On-demand pack generation for a fetch** — ~~pack writing exists
|
||||
(`gix-pack bundle::write`) but "stream a pack computed from a want/have set
|
||||
to a socket" composition needs a POC; worst case we walk objects ourselves
|
||||
via odb and feed `gix-pack::data::output::bytes` entries-to-bytes writers.
|
||||
via odb and feed `gix-pack::data::output::bytes` entries-to-bytes writers.~~
|
||||
**Resolved (POC-2)**: the `data::output` pipeline streams to any
|
||||
`io::Write`; we own only the orchestration (peel tips → commit walk →
|
||||
count → entries → bytes) and the missing-objects check. Delta *synthesis*
|
||||
for loose objects remains an upstream gap (optimization backlog).
|
||||
5. **receive-pack** — parsing client pack stream (`gix-pack::data::input`
|
||||
can parse a pack from a reader), fsck, ref CAS updates via `gix-ref`
|
||||
transaction, reporting status (`unpack ok`/`ng` lines).
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
# POC-2 findings: server-side pack generation
|
||||
|
||||
**Status**: complete — **proceed**
|
||||
**Date**: 2026-09-20
|
||||
**Mode**: standalone (`/workspace/alkgit-poc2`, disposable per `pocs.md`; this
|
||||
document is the deliverable on main)
|
||||
**Depends on**: POC-1 (pkt-line over alkcall BiStream, `poc-1-findings.md`)
|
||||
|
||||
## Hypothesis (restated)
|
||||
|
||||
Given a fixture repo and a want/have set, produce a valid pack stream that
|
||||
`git verify-pack`/`git unpack-objects` accepts, using either:
|
||||
a. `gix-pack::bundle::write` into a temp dir then read the file back
|
||||
(correctness baseline), or
|
||||
b. `gix-pack::data::output::bytes` fed by an odb object walk (streaming),
|
||||
with a decision on streaming composition that doesn't materialize the whole
|
||||
pack in memory for large repos — memory behavior recorded for a 10k-object
|
||||
repo at minimum.
|
||||
|
||||
## Result
|
||||
|
||||
**Confirmed, with one course correction.** Real `git` 2.43 clones, fetches,
|
||||
and tag-checkouts against a bridge that generates packs with gitoxide's own
|
||||
generation pipeline, streamed over sideband without ever holding the whole
|
||||
pack in memory. `git fsck --strict` passes on every clone; `git index-pack
|
||||
--strict` and `git verify-pack` accept the generated bytes standalone.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `git index-pack --strict` on generated pack | OK (both fixtures) |
|
||||
| `git verify-pack -v` | `ok`, object list + sizes verified |
|
||||
| `git unpack-objects --strict` + `git fsck --strict` | OK |
|
||||
| `git clone` (V2, full handshake) from the bridge | OK, 10k commits / 60k objects |
|
||||
| `git fsck --strict` in the clone | OK |
|
||||
| `git fetch` (incremental ref fetch) | OK, ref updated, fsck clean |
|
||||
| `git checkout <annotated tag>` | OK, checks out at the peeled commit |
|
||||
| in-process sideband selftest | band-1 framing exact, flush-terminated, no stray bytes |
|
||||
| closure cross-check | own BFS walk == gitoxide count, 15k/60k objects |
|
||||
| approach (a) `Bundle::write_to_directory` round-trip | indexes our pack, 0 objects materialized in memory |
|
||||
|
||||
## The composition that works (approach b, as built)
|
||||
|
||||
gitoxide already ships a complete pack *generation* pipeline behind
|
||||
`gix-pack`'s `generate` feature. The serving path is three stages chained as
|
||||
iterators, ending in plain `io::Write`:
|
||||
|
||||
```
|
||||
want tips (peeled to commits)
|
||||
-> gix_traverse::commit::Simple (Parents::All) [commit ancestry]
|
||||
-> gix_pack::data::output::count::objects_unthreaded(TreeContents)
|
||||
[per-commit tree + subtree + blob expansion -> Vec<Count>]
|
||||
-> gix_pack::data::output::entry::iter_from_counts [chunked deflate,
|
||||
delta-copy from existing packs, back-pressured]
|
||||
-> gix_features::parallel::InOrderIter [re-order chunk results]
|
||||
-> gix_pack::data::output::bytes::FromEntriesIter [header, entries,
|
||||
trailer sha1 -> W: io::Write]
|
||||
-> SidebandSink (ours) [65000-byte band-1
|
||||
pkt-line chunks]
|
||||
```
|
||||
|
||||
This is exactly `gitoxide-core/src/pack/create.rs`'s composition (`gix pack
|
||||
create`) — it works against any `gix_pack::Find` implementor, i.e. against
|
||||
`gix_odb::Cache<Handle<Arc<Store>>>` directly. No temp files, no `git`
|
||||
binary, no full-pack buffer.
|
||||
|
||||
### The course correction: `TreeContents` does not follow commit parents
|
||||
|
||||
The obvious call — feed wants into `count::objects(_unthreaded)` with
|
||||
`ObjectExpansion::TreeContents` and expect the full reachable closure —
|
||||
produces an **incomplete pack**. `TreeContents` expands each *input object's*
|
||||
own tree (commit → its tree → subtrees/blobs; tag → target's tree); it does
|
||||
**not** traverse commit ancestry. Feeding tip commits alone drops all ancestor
|
||||
commits, their trees, and their blobs. Real git's `index-pack --strict`
|
||||
catches this (`did not receive expected object <ancestor>`).
|
||||
|
||||
Correct closure = **commit-ancestry walk** (`gix_traverse::commit::Simple`
|
||||
with `Parents::All`) whose output commits, plus any non-commit tips
|
||||
(annotated tags), feed the `TreeContents` count stage. This mirrors what
|
||||
`gix pack create` does and what POC-1's hand-rolled walker did. Recorded in
|
||||
`git-protocol.md`'s inventory (below) — the want/have-set walk is two stages,
|
||||
not one.
|
||||
|
||||
Two smaller API facts along the same line:
|
||||
|
||||
1. **Tips must be peeled before the commit walk.** `commit::Simple` requires
|
||||
`find_commit_iter`; handing it an annotated-tag id errors with
|
||||
`Expected object of kind commit but got tag`. Peel via `TagRef::target()`
|
||||
in a loop until a commit (or bare tree/blob — no ancestry, count as-is).
|
||||
2. **The odb handle needs `prevent_pack_unload()` + `ignore_replacements = true`**
|
||||
before use in generation (`location_by_oid` asserts the former; the latter
|
||||
matches `gix pack create`). Forgetting the first panics on the first
|
||||
packed-object lookup; the fixture that works loose-only silently passes
|
||||
until a packed repo is used.
|
||||
|
||||
## Approach (a) baseline: `Bundle::write_to_directory`
|
||||
|
||||
Used as the correctness check, not as the serving path. It consumes an
|
||||
*existing pack stream* (it is the index-from-stream machinery:
|
||||
`data::input::BytesToEntriesIter` → index write → pack file persist), so it
|
||||
is the wrong tool for on-demand generation — the pipeline above is the
|
||||
generation path. It needs the `streaming-input` feature and
|
||||
`Option::<gix_object::find::Never>::None` as the thin-pack lookup. Feeding it
|
||||
our generated pack: indexes cleanly, `index_hash`/`data_hash` match git's
|
||||
`index-pack` output (both `997bc209…` / `05c910d8…`-style trailer hashes on
|
||||
the same pack). Conclusion: **bundle::write is for receive-pack (POC on pack
|
||||
ingestion), not for fetch-side generation.** `git-protocol.md`'s option list
|
||||
said "bundle::write with an in-memory sink" — that reading was wrong; it
|
||||
writes to a temp file per design (it mmaps the pack to resolve deltas for the
|
||||
index).
|
||||
|
||||
## Streaming & memory behavior (the decision POC-2 was for)
|
||||
|
||||
**Streaming composition is real and needs no new machinery.** `FromEntriesIter`
|
||||
is a lazy `Iterator<Item = Result<u64>>` over any `io::Write`; `iter_from_counts`
|
||||
yields entries chunk-by-chunk with back-pressure (one chunk of compressed
|
||||
entries in memory at a time, default chunk 10). The only in-memory artifact is
|
||||
`Vec<Count>` (24 B/object: 20-byte id + pack-location option) — for 60k
|
||||
objects that's ~1.4 MB, dwarfed by the pack itself.
|
||||
|
||||
Measured (release build, `/proc/self/status` VmRSS, 5 ms sampler thread):
|
||||
|
||||
| Fixture | Objects | Pack bytes | Time | RSS before | RSS peak | RSS after |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2.5k commits, 15k objects | 15,000 | 1,327,523 | 1.4 s | 5.5 MB | 9.3 MB | 7.6 MB |
|
||||
| 4k commits, 24k objects | 24,000 | 2,128,268 | 2.5 s | 5.4 MB | 9.7 MB | 7.6 MB |
|
||||
| 10k commits, 60k objects | 60,000 | 5,334,304 | 5.5 s | 5.4 MB | 16.4 MB | 9.0 MB |
|
||||
|
||||
Peak ≈ base + ~(counts × ~200 B) + object buffers — i.e. **O(counts), not
|
||||
O(pack)**: the 5.3 MB pack for 60k objects went to the socket through a
|
||||
64 KB-bounded sink (65000-byte sideband chunks) with ~11 MB of extra RSS
|
||||
above baseline. No whole-pack materialization anywhere on the path.
|
||||
|
||||
Delta behavior, as-built:
|
||||
|
||||
- `iter_from_counts` **copies existing pack deltas** (`from_pack_entry` —
|
||||
ofs-delta entries referencing bases also in the pack, translated through
|
||||
the counts index) and **deflates loose objects as whole bases**
|
||||
(`from_data`). It does **not synthesize new deltas** between loose objects
|
||||
— gitoxide has no delta *encoding* API for the server path (`gix-delta`
|
||||
is apply/decode only; there is no `encode` in the published crates).
|
||||
- Empirical effect on the 60k-object fixture: repo `git gc`'d to one pack
|
||||
(5.5 MB, 29,981 chain-1 deltas) → our generated pack 3.96 MB (deltas
|
||||
copied through, compressed bases for loose objects); same repo all-loose →
|
||||
generated pack 5.33 MB. Serving a fully packed repo is *smaller* than what
|
||||
git stores because our walk writes every entry as-is without bitmap
|
||||
reuse and without delta re-selection — and both clone cleanly.
|
||||
- For phase 1: fresh clones of loose-ish repos ship uncompressed bases
|
||||
(fine for v1; `git` on the client re-packs at rest if `fetch.unpackLimit`
|
||||
says so), and repos kept packed get pack-copy for free. Delta *synthesis*
|
||||
(window search over loose objects) is a real gap vs `git upload-pack` —
|
||||
record as an optimization backlog item, not a blocker.
|
||||
|
||||
## Sideband streaming over the POC-1 wire path (verified shape)
|
||||
|
||||
The fetch response composes exactly as POC-1's protocol inventory described,
|
||||
now with real generation feeding it:
|
||||
|
||||
- `packfile\n` text line, then band-1 chunks. `SidebandSink` (an `io::Write`
|
||||
adapter) caps chunks at 65000 bytes (git's own cap is 65515; headroom for
|
||||
the 4-byte length prefix under the 65520 pkt-line max) and bridges to
|
||||
`gix_packetline::async_io::encode::band_to_write` via
|
||||
`futures_lite::future::block_on` — safe because exactly one task drives the
|
||||
sink. `flush()` is idempotent (the first flush-pkt closes the section; a
|
||||
second would be a protocol error — caught by the selftest's wire decoder).
|
||||
- **The generator is sync; run it on `tokio::task::spawn_blocking`.** The
|
||||
async side passes the compat writer into the blocking task and awaits the
|
||||
result. This also resolves a type-system fact worth encoding now:
|
||||
`gix_odb::Cache` is deliberately **not `Sync`** (per-thread `RefCell`
|
||||
pack/object caches by design), so it cannot be held across an `.await` in
|
||||
a `Send` future. The handler therefore holds `Arc<gix_odb::Store>`
|
||||
(thread-safe: `parking_lot::Mutex` + `ArcSwap` + atomics) and builds a
|
||||
`Cache` handle per connection/generation (`store.to_handle_arc()` +
|
||||
`prevent_pack_unload()`), moving the owned `Cache` into the blocking task.
|
||||
This is the exact shape `alkgit-transport` should encode: **store shared,
|
||||
handle per session, generation on blocking threads.**
|
||||
- End-to-end over the alkcall surface (`Connection::from_bidi` → `accept_bi`
|
||||
→ `BiStream` → compat → packetline): unchanged from POC-1; the pack path
|
||||
adds only the sink.
|
||||
|
||||
## gix-pack API surface notes for `alkgit-core` (phase-1 input)
|
||||
|
||||
1. **Pipeline pieces and paths** (pinned versions from `gitoxide.md`):
|
||||
`gix_pack::data::output::count::objects_unthreaded` (or the threaded
|
||||
`count::objects` for scale), `entry::iter_from_counts` (needs `generate`
|
||||
+ `parallel` features; yields `Result<(SequenceId, Vec<Entry>), Error>`),
|
||||
`gix_features::parallel::InOrderIter` to re-sequence, and
|
||||
`bytes::FromEntriesIter::new(input, out, num_entries, Version::V2,
|
||||
Kind::Sha1)` (asserts V2 — only V2 can be written). Composition is exactly
|
||||
`gitoxide-core/src/pack/create.rs`; there is no higher-level "write pack
|
||||
to socket" API upstream.
|
||||
2. **`FromEntriesIter` writes header + count + trailer and flushes the
|
||||
writer**; the trailer is `digest()` after the final `next()` returns
|
||||
`None`. `iter_from_counts`' `finalize()` (via
|
||||
`gix_features::parallel::reduce::Finalize`) yields entry statistics
|
||||
(copied-from-pack / recompressed / missing counts) — free server metrics.
|
||||
3. **Progress plumbing**: `Box<dyn DynNestedProgress>`; `Discard` works
|
||||
everywhere. Server will want a real sink for session progress/budget
|
||||
later.
|
||||
4. **Error taxonomy**: generation errors are `thiserror` enums per stage;
|
||||
`missing_objects` do not abort — they become "invalid" entries (null-id
|
||||
sentinel) which `FromEntriesIter` skips while blocking deltas against
|
||||
them. For us: a missing object mid-generation should abort the fetch with
|
||||
a sideband error band, not silently produce a broken pack — check the
|
||||
entry stats and error out if `missing_objects > 0`.
|
||||
5. **Determinism**: single-threaded count (`objects_unthreaded`) gives
|
||||
deterministic order; the threaded `count::objects` + multi-thread
|
||||
`iter_from_counts` reorder chunks (hence `InOrderIter`). For a server
|
||||
with per-connection tasks, thread_limit=1 in-process plus the blocking
|
||||
pool is plenty at these speeds (60k objects in 5.5 s debug, 1.4 s for
|
||||
15k release).
|
||||
6. **`Cache` is not `Sync`** — see above. Also `Bundle::write_to_directory`
|
||||
requires the `streaming-input` feature (default on upstream; we run
|
||||
lean, so list features explicitly: `sha1`, `generate`, `parallel`,
|
||||
`streaming-input`).
|
||||
|
||||
## What the POC does NOT settle
|
||||
|
||||
- **Want/have negotiation**: `haves` are parsed and recorded but never
|
||||
subtracted (we advertise `wait-for-done` and send full closures on `done`,
|
||||
the POC-1 wire shape). Multi-round ack logic stays phase-1 work; the
|
||||
generator is negotiation-agnostic (boundary sets in → pack out).
|
||||
- **Delta synthesis** for loose objects (above) — optimization backlog.
|
||||
- **sha256**: pipeline is hash-generic (`object_hash` parameter) but
|
||||
untested here; the `sha256` feature flag passthrough remains an OQ.
|
||||
- **Concurrency**: one pack generation per connection on blocking threads;
|
||||
a shared-process thread budget per repo is an architecture concern
|
||||
(bounded-resources invariant), not a POC question.
|
||||
|
||||
## What it changes in the research docs
|
||||
|
||||
- `git-protocol.md`: pack generation option list resolved — (b) wins;
|
||||
record the two-stage closure (commit ancestry + TreeContents), the
|
||||
`prevent_pack_unload()`/replacements requirements, `bundle::write`'s
|
||||
actual role (receive-pack indexing), and the missing-delta-synthesis note.
|
||||
- `gitoxide.md`: add generation-pipeline notes (feature set, composition,
|
||||
`Cache` !Sync, `Version::V2`-only writer, entry statistics via
|
||||
`finalize()`).
|
||||
- `pocs.md`: POC-2 outcome recorded.
|
||||
|
||||
## Follow-ups for phase 1 (architecture input)
|
||||
|
||||
1. `alkgit-core` should expose a `PackWriter` type: `(odb handle, wants,
|
||||
haves) -> impl io::Write-streaming pack`, encapsulating tip peeling,
|
||||
commit walk, count, entries, bytes, and the missing-objects check. The
|
||||
POC's `packgen::generate_streaming` is the seed.
|
||||
2. `alkgit-transport` fetch handler: generation belongs on
|
||||
`spawn_blocking`; the handler holds `Arc<Store>` and builds per-session
|
||||
handles. Budget the blocking pool (bounded-resources invariant).
|
||||
3. Sideband chunk size (65000) and any max-pack-size cap should be
|
||||
server-configurable, flowing through the same limits struct as POC-1's
|
||||
session limits.
|
||||
4. Consider `pack.cache` (LRU) sizing on the odb handle for hot-repo
|
||||
fetches; gitoxide exposes `set_pack_cache`/`set_object_cache` with byte
|
||||
budgets (gitoxide-core wires exactly this in `pack/create.rs`).
|
||||
|
||||
## Verification transcript (2026-09-20)
|
||||
|
||||
```
|
||||
$ cargo run -- selftest # small fixture: gen -> bundle roundtrip ->
|
||||
git index-pack --strict -> verify-pack -> unpack-objects --strict ->
|
||||
fsck --strict -> closure check -> sideband wire decode
|
||||
... sideband framing OK: 763 band-1 bytes in chunks <= 65001, pack bytes 763
|
||||
$ cargo run -- stats /tmp/…/fixture-large.git
|
||||
closure (own walk): 60000 == gitoxide count::objects: 60000
|
||||
$ cargo run --release -- bench /tmp/…/fixture-large.git # 10k commits
|
||||
gen: 60000 objects, 5334304 pack bytes, 5517 ms, RSS peak=16364kB
|
||||
$ cargo run -- bridge 9423 /tmp/…/fixture-large.git # packed 60k-object repo
|
||||
$ git -c protocol.version=2 clone git://127.0.0.1:9423/repo.git c
|
||||
-> completes (6.1 s wall); git fsck --strict OK
|
||||
$ cargo run -- bridge 9422 /tmp/…/fixture.git # small fixture, tags
|
||||
$ git -c protocol.version=2 clone git://127.0.0.1:9422/repo.git c2
|
||||
-> clone; git fsck --strict OK; git checkout v1.0 OK (annotated tag)
|
||||
$ git -c protocol.version=2 fetch origin refs/heads/dev:refs/heads/newdev
|
||||
-> * [new branch] dev -> newdev; git fsck --strict OK
|
||||
```
|
||||
|
||||
Traces: `GIT_TRACE_PACKET=1` shows the client receiving the POC-2
|
||||
advertisement (`agent=alkgit-poc2/0.1`), ls-refs with `peeled:`/`symref-target:`
|
||||
attributes, and the `packfile` section over band-1 sideband with flush
|
||||
termination — identical shape to POC-1's captured ground truth, now fed by
|
||||
real generation.
|
||||
@@ -53,6 +53,14 @@ compatibility (they should, being tokio io objects; may need a thin adapter).
|
||||
|
||||
## POC-2: server-side pack generation
|
||||
|
||||
**Status**: complete (2026-09-20) — **proceed**. See `poc2-findings.md`.
|
||||
Real git 2.43 clones/fetches (fsck-strict clean) packs generated by
|
||||
gitoxide's own pipeline (`count::objects` + `iter_from_counts` +
|
||||
`FromEntriesIter`) streamed over sideband; memory is O(counts), not
|
||||
O(pack) — 60k-object pack written through a 64 KB sink at ~11 MB extra RSS.
|
||||
Course correction recorded: `TreeContents` does not follow commit parents —
|
||||
the closure needs a commit-ancestry walk feeding the count stage.
|
||||
|
||||
**Hypothesis**: given a fixture repo and a want/have set, we can produce a
|
||||
valid pack stream in memory that `git verify-pack`/`git unpack-objects`
|
||||
accepts, using one of:
|
||||
@@ -91,6 +99,9 @@ POC-1 first (the BiStream/packetline fit is the load-bearing assumption).
|
||||
POC-2 next (the crux of fetch). POC-3 last (http shape). Each POC updates
|
||||
the corresponding research doc with results and a proceed/pivot/block note.
|
||||
|
||||
**Status**: POC-1 and POC-2 complete and proceed; POC-3 is the remaining
|
||||
gate for phase 0 completion.
|
||||
|
||||
## Modes applied to these POCs
|
||||
|
||||
- POC-1, POC-2: **standalone mode** — nothing in the alkgit repo exists to
|
||||
|
||||
Reference in new issue
Block a user