docs(architecture): phase 1 bootstrap — specs, 9 ADRs, OQ tracker
Architecture documentation structure per sdd_process phase 1: - README index (doc table, ADR table, lifecycle), overview with crate map, dependency rules, and security invariants - Component specs: storage, transport, http, ssh, alkgitd (all draft) - ADRs 001-009: crate decomposition, front-door-blind core, V2-first protocol, pack pipeline (data::output generation / data::input ingestion), session substrate types, http adapter composition (proposed, OQ-01), ACL-before-advertisement, registry-resolved repo identity, bounded-resources budgets - open-questions.md: OQ-01..08 with two deferred(scope), one deferred(unclear), door-type definitions, blocker tracker tasks in tasks/architecture/ - v1 ssh-door decision recorded: russh terminates wire SSH in alkgitd; alkcall channels stay the internal substrate (OQ-03 partially resolved) Two review rounds (fresh-context subagent): 4 critical + 17 warnings fixed in round one; zero critical + 4 warnings + 5 suggestions fixed in round two. All ADR/OQ cross-references verified resolving.
This commit is contained in:
1 parent
70815e22a0
commit
8f73da5d12
21 files changed
+1669
No files matched your search
@@ -0,0 +1,80 @@
|
||||
# ADR-004: Pack pipeline — gitoxide `data::output` generation, `data::input` streaming ingestion
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Pack generation and ingestion are the two halves of fetch and receive-pack.
|
||||
POC-2 resolved the generation question empirically; the ingestion tool
|
||||
turned out to be mislabeled in the original research plan (the plan called
|
||||
it `bundle::write` — the correct tool is `data::input` streaming) and was
|
||||
corrected there.
|
||||
|
||||
Generation options were:
|
||||
|
||||
a. `gix-pack::bundle::write` into a temp dir, read back — **wrong tool**:
|
||||
it is the index-from-stream machinery (consumes an *existing* pack
|
||||
stream, mmaps to resolve deltas, always writes files). Not a generation
|
||||
path.
|
||||
b. `gix-pack::data::output::bytes` fed by an odb walk — validated live:
|
||||
streams to any `io::Write` with O(counts) memory (60k-object pack →
|
||||
5.3 MB out, ~11 MB extra RSS), real git 2.43 clones fsck-clean.
|
||||
|
||||
Ingestion (receive-pack): client sends a possibly-thin pack stream; server
|
||||
must index it, fsck it, and apply ref updates via CAS.
|
||||
|
||||
## Decision
|
||||
|
||||
**Fetch-side generation** is gitoxide's own pipeline, composed exactly as
|
||||
`gitoxide-core/src/pack/create.rs` does (POC-2's validated composition):
|
||||
|
||||
```
|
||||
peel wants to commit tips
|
||||
→ commit-ancestry walk (gix_traverse::commit::Simple, Parents::All)
|
||||
[+ non-commit tips as-is]
|
||||
→ count::objects(_unthreaded, TreeContents) [per-commit tree expansion]
|
||||
→ entry::iter_from_counts [chunked deflate + pack-delta copy]
|
||||
→ InOrderIter → bytes::FromEntriesIter (V2, sha1)
|
||||
→ io::Write sink (sideband on the wire, or http response body)
|
||||
```
|
||||
|
||||
Two-stage closure is mandatory: `TreeContents` does **not** follow commit
|
||||
parents; feeding tip commits alone produces incomplete packs (POC-2's
|
||||
course correction). The odb handle needs `prevent_pack_unload()` +
|
||||
`ignore_replacements = true`. `gix_odb::Cache` is not `Sync`: share
|
||||
`Arc<Store>`, build a handle per session, generate on blocking threads.
|
||||
|
||||
Missing objects mid-generation must **abort** the fetch (sideband error
|
||||
band), never emit a broken pack — check entry statistics
|
||||
(`missing_objects > 0`).
|
||||
|
||||
**Receive-side ingestion** parses the client pack stream
|
||||
(`gix-pack::data::input` with `streaming-input`), fscks it
|
||||
(`gix-fsck` connectivity), and applies ref updates via `gix-ref`
|
||||
transaction CAS. The exact push state machine (capability advertisement
|
||||
set, CAS timing, status report, http framing) is OQ-04's investigation
|
||||
target; the ingestion-tool choice above is decided.
|
||||
|
||||
**Delta synthesis** for loose objects is an optimization backlog item, not
|
||||
a v1 commitment: existing pack deltas copy through for free; loose objects
|
||||
ship as compressed bases. No upstream delta-encode API exists
|
||||
(`gix-delta` is apply-only).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Fresh clones of loose-ish repos ship uncompressed bases (fine for v1;
|
||||
clients re-pack at rest); repos kept packed get pack-copy efficiency.
|
||||
- Memory stays O(counts); streaming under back pressure is proven on the
|
||||
http path (POC-3: flat RSS under a 650 KB/s reader).
|
||||
- Determinism: `objects_unthreaded` gives deterministic order; threaded
|
||||
count + `InOrderIter` is the scale path later.
|
||||
- `alkgit-core` exposes this as a type taking (odb handle, wants, haves) →
|
||||
streaming pack; negotiation-agnostic (POC-2 follow-up 1).
|
||||
|
||||
## References
|
||||
- `docs/research/poc2-findings.md` (the whole basis), `docs/research/poc3-findings.md` (streaming proof)
|
||||
- `docs/research/gitoxide.md` §"Storage" (generation-pipeline notes)
|
||||
- `docs/research/git-protocol.md` §"Server-side pack generation"
|
||||
- ADR-005 (how the sink reaches the wire), ADR-009 (memory/time budgets)
|
||||
- transport.md §fetch, storage.md §packs
|
||||
Reference in new issue
Block a user