glm-5.3-flash
8f73da5d12
Architecture documentation structure per sdd_process phase 1: - README index (doc table, ADR table, lifecycle), overview with crate map, dependency rules, and security invariants - Component specs: storage, transport, http, ssh, alkgitd (all draft) - ADRs 001-009: crate decomposition, front-door-blind core, V2-first protocol, pack pipeline (data::output generation / data::input ingestion), session substrate types, http adapter composition (proposed, OQ-01), ACL-before-advertisement, registry-resolved repo identity, bounded-resources budgets - open-questions.md: OQ-01..08 with two deferred(scope), one deferred(unclear), door-type definitions, blocker tracker tasks in tasks/architecture/ - v1 ssh-door decision recorded: russh terminates wire SSH in alkgitd; alkcall channels stay the internal substrate (OQ-03 partially resolved) Two review rounds (fresh-context subagent): 4 critical + 17 warnings fixed in round one; zero critical + 4 warnings + 5 suggestions fixed in round two. All ADR/OQ cross-references verified resolving.
80 lines
3.6 KiB
Markdown
80 lines
3.6 KiB
Markdown
# ADR-004: Pack pipeline — gitoxide `data::output` generation, `data::input` streaming ingestion
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
Pack generation and ingestion are the two halves of fetch and receive-pack.
|
|
POC-2 resolved the generation question empirically; the ingestion tool
|
|
turned out to be mislabeled in the original research plan (the plan called
|
|
it `bundle::write` — the correct tool is `data::input` streaming) and was
|
|
corrected there.
|
|
|
|
Generation options were:
|
|
|
|
a. `gix-pack::bundle::write` into a temp dir, read back — **wrong tool**:
|
|
it is the index-from-stream machinery (consumes an *existing* pack
|
|
stream, mmaps to resolve deltas, always writes files). Not a generation
|
|
path.
|
|
b. `gix-pack::data::output::bytes` fed by an odb walk — validated live:
|
|
streams to any `io::Write` with O(counts) memory (60k-object pack →
|
|
5.3 MB out, ~11 MB extra RSS), real git 2.43 clones fsck-clean.
|
|
|
|
Ingestion (receive-pack): client sends a possibly-thin pack stream; server
|
|
must index it, fsck it, and apply ref updates via CAS.
|
|
|
|
## Decision
|
|
|
|
**Fetch-side generation** is gitoxide's own pipeline, composed exactly as
|
|
`gitoxide-core/src/pack/create.rs` does (POC-2's validated composition):
|
|
|
|
```
|
|
peel wants to commit tips
|
|
→ commit-ancestry walk (gix_traverse::commit::Simple, Parents::All)
|
|
[+ non-commit tips as-is]
|
|
→ count::objects(_unthreaded, TreeContents) [per-commit tree expansion]
|
|
→ entry::iter_from_counts [chunked deflate + pack-delta copy]
|
|
→ InOrderIter → bytes::FromEntriesIter (V2, sha1)
|
|
→ io::Write sink (sideband on the wire, or http response body)
|
|
```
|
|
|
|
Two-stage closure is mandatory: `TreeContents` does **not** follow commit
|
|
parents; feeding tip commits alone produces incomplete packs (POC-2's
|
|
course correction). The odb handle needs `prevent_pack_unload()` +
|
|
`ignore_replacements = true`. `gix_odb::Cache` is not `Sync`: share
|
|
`Arc<Store>`, build a handle per session, generate on blocking threads.
|
|
|
|
Missing objects mid-generation must **abort** the fetch (sideband error
|
|
band), never emit a broken pack — check entry statistics
|
|
(`missing_objects > 0`).
|
|
|
|
**Receive-side ingestion** parses the client pack stream
|
|
(`gix-pack::data::input` with `streaming-input`), fscks it
|
|
(`gix-fsck` connectivity), and applies ref updates via `gix-ref`
|
|
transaction CAS. The exact push state machine (capability advertisement
|
|
set, CAS timing, status report, http framing) is OQ-04's investigation
|
|
target; the ingestion-tool choice above is decided.
|
|
|
|
**Delta synthesis** for loose objects is an optimization backlog item, not
|
|
a v1 commitment: existing pack deltas copy through for free; loose objects
|
|
ship as compressed bases. No upstream delta-encode API exists
|
|
(`gix-delta` is apply-only).
|
|
|
|
## Consequences
|
|
|
|
- Fresh clones of loose-ish repos ship uncompressed bases (fine for v1;
|
|
clients re-pack at rest); repos kept packed get pack-copy efficiency.
|
|
- Memory stays O(counts); streaming under back pressure is proven on the
|
|
http path (POC-3: flat RSS under a 650 KB/s reader).
|
|
- Determinism: `objects_unthreaded` gives deterministic order; threaded
|
|
count + `InOrderIter` is the scale path later.
|
|
- `alkgit-core` exposes this as a type taking (odb handle, wants, haves) →
|
|
streaming pack; negotiation-agnostic (POC-2 follow-up 1).
|
|
|
|
## References
|
|
- `docs/research/poc2-findings.md` (the whole basis), `docs/research/poc3-findings.md` (streaming proof)
|
|
- `docs/research/gitoxide.md` §"Storage" (generation-pipeline notes)
|
|
- `docs/research/git-protocol.md` §"Server-side pack generation"
|
|
- ADR-005 (how the sink reaches the wire), ADR-009 (memory/time budgets)
|
|
- transport.md §fetch, storage.md §packs |