docs(architecture): Phase 1 spec tree — 7 specs, ADRs 001-006, OQ tracker

- overview/hashing/store-api/backends/gc/ops specs + README index,
  all draft status with YAML frontmatter and cross-referenced ADR/OQ
- ADR-001: substrate posture (store layer alkcall-free = family
  conformance, not deviation) + ops placement decided (feature-gated
  module, no consumer-waited deferral)
- ADR-002: canonical git-blob-sha-256 + key encoding — the declared
  wire-format ADR (one-way door)
- ADR-003: backend contract, sqlite/fs/mem tiers, pure-function
  dispatch, no migration, unknown-length buffering semantics pinned
- ADR-004: exactly two production backends; manifest/path-tree layers
  are consumer-side (the corrected two-vs-three-backends framing)
- ADR-005: pooled CAS, three liveness sources, delete windows with
  pin-before-publish + delete-time arbitration (race closed), no
  ambient scheduling
- ADR-006: whole-blob verification; chunk-tree encodings excluded by
  scoping, not deferred on a paused consumer
- open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07
  /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task
- AGENTS.md: status updated to Phase 1, gix-odb path corrected

Verification: cargo test / clippy -D warnings / fmt --check clean
This commit is contained in:
glm-5.3-flash committed 2026-10-01 14:45:32 +00:00
1 parent 7d9ac8a6ec
commit 4b5009d85a
16 files changed
+1953 -8

No files matched your search

+124
View File
@@ -0,0 +1,124 @@
---
status: draft
last_updated: 2026-10-01
---
# Backends and dispatch
## What this is
The physical storage layer: the `Backend` trait contract every backend
implements, the two shipped backends (kv, fs), and the size-threshold
dispatch that routes between them. The trait is the crate's second
stable seam (with the key encoding, ADR-002): backends must survive
digest-layer evolution, so the boundary stays dumb.
## The Backend trait contract (ADR-003)
- **Opaque byte keys, opaque byte values.** Backends never learn what a
digest is; typed `Key` converts at the store boundary (ADR-002). This
is also the persistence boundary — a kv row or an fs filename has no
schema-migration story, so the byte layout must be self-describing
(algorithm-tagged keys).
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name`.
Streaming put/get shapes ride the store layer's seams (store-api.md);
backends provide byte- or handle-level primitives underneath
(`get` on the fs tier yields a file handle, not a loaded buffer).
- **`list()` complete by contract.** A *malformed* list is as deadly as
an incomplete one: list correctness is only observable through GC
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
deleted the wrong blobs, silently). Therefore:
- every backend implementation must prove list correctness through
sweep-outcome tests (the invariant's test gate lives in
store-api.md);
- an implementation that cannot enumerate (the rudolfs S3
anti-lesson) must not ship — GC is a structural requirement, not an
optional extra (ADR-005).
- **Virgin-store reads are no-ops:** read paths treat a missing table
as absent/empty (POC #1 finding 2).
- **Namespace-blind:** backends never see namespaces, tenants, or
reference structure (ADR-005 — the rudolfs inversion; physical
storage is `hash → bytes` flat).
## Shipped backends
Two, exactly — this is the complete set the problem requires; the
"third backend" fear is a category error fixed in ADR-004.
### kv backend (feature `kv`, default-on; sqlite)
Small blobs. Evidence (POC #3 finding A4, first-party measured):
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
— most git objects, workspace files, manifests), with the crossover at
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
- Bounded reads; `has` as an EXISTS probe; prepared statements.
- WAL + `synchronous=NORMAL` as the shipped durability tier (matching
what the benchmark measured and what iroh's store ships).
- Read paths tolerate the no-tables-yet database (see contract).
### fs backend (feature `fs`, default-on)
Large blobs. Flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}`
sharding survives from iroh's conclusion (limits directory size on
huge pools); stage-then-commit-rename for the two-pass unknown-length
path; pread-based range reads (POC #3 finding A2 — local range serving
is sound, e.g. for packfiles).
### mem backend (feature `mem`, default-off)
`BTreeMap`-shaped ephemeral backend for tests and in-process
ephemerality. A testing/utility tier, never a production story.
## Size-threshold dispatch (ADR-003)
- **Routing is a pure function of content length.** Same content ⇒ same
length ⇒ same backend; re-puts are deterministic. No content ever
migrates between backends — the migration question existed only to
patch the unknown-length asymmetry, which ADR-003's pre-threshold
buffering eliminates by construction.
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
constructor parameter, not an API change.
- **Get fall-through:** small-tier miss queries the large tier
(deterministic, cheap — a stat probe).
- Per-namespace or per-tenant backend configuration: **rejected**
(ADR-003 §Consequences — it would re-weld namespacing into the
physical layer, the rudolfs anti-pattern ADR-005 inverts).
## Where a *new* backend could come from
The trait is open to future implementations (network stores, S3-like
tiers), but nothing in the current consumer set requires one, and the
contract is deliberately hostile to half-implementations (complete
`list()`, GC-participating `delete`). Any future backend is a new ADR
carrying its own sweep-safety story. This crate's roadmap is not
blocked on one (see open-questions.md — alkfs intake may name needs
externally; OQ-08).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
## Open Questions
- **OQ-08**: alkfs requirement intake may name storage requirements
(e.g., durability tiers, sync-friendly layouts) that touch this
layer — open, external owner (alkfs Phase 0).
## References
- `docs/research/poc-trait-dispatch-findings.md` findings 1/2/6
- `docs/research/poc-largeblob-findings.md` findings A2/A4 (+ the
re-runnable benchmark harness)
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
(sharding, crash ordering, inline thresholds rejected as weld)
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
not adopted (threshold dispatch chose the simpler policy;
ADR-003 §Context)
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
backends must satisfy)