glm-5.3-flash
e76f91f6d7
- ADR-013: V0-framed push machine grounded in real git 2.43.0 captures
(file://, git://, smart-http mock, raw stdio into real receive-pack):
V0-shaped ref advertisement (caps on first ref line, capabilities^{}
sentinel only for empty repos), served capability set, shallow requests
rejected for v1, thin packs accepted with server-odb bases (no
capability involved; push.thin default), ingestion bound to
Bundle::write_to_directory_eagerly + gix-fsck + one gix-ref transaction
per push (.keep-guarded), unpack-first CAS timing with observed
upstream order, band-1 pkt-line-framed status report, http framing
(probe/Content-Length/chunked), v1 update policy (CAS only; deletes
and force-push allowed)
- docs/research/push-captures.md: the normative push wire record
- transport.md/backend.md/doors.md: receive-pack sections rewritten to
the decided shapes; backend.md ingestion composition bound; stale
OQ-04 references resolved
- ADR-003 amended: V2-only governs fetch; push is V0-framed by upstream
design (fixes the V2-only contradiction found in review)
- ADR-009 amended: haves default reconciled with the client's stateless
ceiling (16384); blocking-pipeline budget covers generation+ingestion
- OQ-04 resolved; tracker task closed; CAS-fail-fast optimization
tracked (tasks/architecture/oq-13-cas-failfast.md)
- research index: poc findings + capture docs listed
Verification: cargo test / clippy -D warnings / fmt --check / doc pass
85 lines
3.8 KiB
Markdown
85 lines
3.8 KiB
Markdown
# ADR-004: Pack pipeline — gitoxide `data::output` generation, `data::input` streaming ingestion
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
Pack generation and ingestion are the two halves of fetch and receive-pack.
|
|
POC-2 resolved the generation question empirically; the ingestion tool
|
|
turned out to be mislabeled in the original research plan (the plan called
|
|
it `bundle::write` — the correct tool is `data::input` streaming) and was
|
|
corrected there.
|
|
|
|
Generation options were:
|
|
|
|
a. `gix-pack::bundle::write` into a temp dir, read back — **wrong tool**:
|
|
it is the index-from-stream machinery (consumes an *existing* pack
|
|
stream, mmaps to resolve deltas, always writes files). Not a generation
|
|
path.
|
|
b. `gix-pack::data::output::bytes` fed by an odb walk — validated live:
|
|
streams to any `io::Write` with O(counts) memory (60k-object pack →
|
|
5.3 MB out, ~11 MB extra RSS), real git 2.43 clones fsck-clean.
|
|
|
|
Ingestion (receive-pack): client sends a possibly-thin pack stream; server
|
|
must index it, fsck it, and apply ref updates via CAS.
|
|
|
|
## Decision
|
|
|
|
**Fetch-side generation** is gitoxide's own pipeline, composed exactly as
|
|
`gitoxide-core/src/pack/create.rs` does (POC-2's validated composition):
|
|
|
|
```
|
|
peel wants to commit tips
|
|
→ commit-ancestry walk (gix_traverse::commit::Simple, Parents::All)
|
|
[+ non-commit tips as-is]
|
|
→ count::objects(_unthreaded, TreeContents) [per-commit tree expansion]
|
|
→ entry::iter_from_counts [chunked deflate + pack-delta copy]
|
|
→ InOrderIter → bytes::FromEntriesIter (V2, sha1)
|
|
→ io::Write sink (sideband on the wire, or http response body)
|
|
```
|
|
|
|
Two-stage closure is mandatory: `TreeContents` does **not** follow commit
|
|
parents; feeding tip commits alone produces incomplete packs (POC-2's
|
|
course correction). The odb handle needs `prevent_pack_unload()` +
|
|
`ignore_replacements = true`. `gix_odb::Cache` is not `Sync`: share
|
|
`Arc<Store>`, build a handle per session, generate on blocking threads.
|
|
|
|
Missing objects mid-generation must **abort** the fetch (sideband error
|
|
band), never emit a broken pack — check entry statistics
|
|
(`missing_objects > 0`).
|
|
|
|
**Receive-side ingestion** parses the client pack stream
|
|
(`gix-pack::data::input` with `streaming-input`), fscks it
|
|
(`gix-fsck` connectivity), and applies ref updates via `gix-ref`
|
|
transaction CAS. The push state machine (capability advertisement
|
|
set, CAS timing, status report, http framing) is decided in ADR-013;
|
|
the ingestion-tool choice above is decided.
|
|
|
|
**Delta synthesis** for loose objects is an optimization backlog item, not
|
|
a v1 commitment: existing pack deltas copy through for free; loose objects
|
|
ship as compressed bases. No upstream delta-encode API exists
|
|
(`gix-delta` is apply-only).
|
|
|
|
## Consequences
|
|
|
|
(2026-09-21 amendment, ADR-010: the crate layout changed — the type
|
|
below is now the `GitPackGen` backend trait in the single crate,
|
|
gix-free by signature `(repo, wants, haves, limits)`; backend.md is
|
|
authoritative for the trait shape. The pipeline itself is unchanged.)
|
|
|
|
- Fresh clones of loose-ish repos ship uncompressed bases (fine for v1;
|
|
clients re-pack at rest); repos kept packed get pack-copy efficiency.
|
|
- Memory stays O(counts); streaming under back pressure is proven on the
|
|
http path (POC-3: flat RSS under a 650 KB/s reader).
|
|
- Determinism: `objects_unthreaded` gives deterministic order; threaded
|
|
count + `InOrderIter` is the scale path later.
|
|
- The generation seam is negotiation-agnostic: boundary sets in → pack
|
|
out (POC-2 follow-up 1).
|
|
|
|
## References
|
|
- `docs/research/poc2-findings.md` (the whole basis), `docs/research/poc3-findings.md` (streaming proof)
|
|
- `docs/research/gitoxide.md` §"Storage" (generation-pipeline notes)
|
|
- `docs/research/git-protocol.md` §"Server-side pack generation"
|
|
- ADR-005 (how the sink reaches the wire), ADR-009 (memory/time budgets)
|
|
- transport.md §fetch, backend.md §"The trait family" (GitPackGen) |