docs(architecture): Phase 1 spec tree — 7 specs, ADRs 001-006, OQ tracker
- overview/hashing/store-api/backends/gc/ops specs + README index, all draft status with YAML frontmatter and cross-referenced ADR/OQ - ADR-001: substrate posture (store layer alkcall-free = family conformance, not deviation) + ops placement decided (feature-gated module, no consumer-waited deferral) - ADR-002: canonical git-blob-sha-256 + key encoding — the declared wire-format ADR (one-way door) - ADR-003: backend contract, sqlite/fs/mem tiers, pure-function dispatch, no migration, unknown-length buffering semantics pinned - ADR-004: exactly two production backends; manifest/path-tree layers are consumer-side (the corrected two-vs-three-backends framing) - ADR-005: pooled CAS, three liveness sources, delete windows with pin-before-publish + delete-time arbitration (race closed), no ambient scheduling - ADR-006: whole-blob verification; chunk-tree encodings excluded by scoping, not deferred on a paused consumer - open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07 /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task - AGENTS.md: status updated to Phase 1, gix-odb path corrected Verification: cargo test / clippy -D warnings / fmt --check clean
This commit is contained in:
1 parent
7d9ac8a6ec
commit
4b5009d85a
16 files changed
+1953
-8
No files matched your search
@@ -0,0 +1,124 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-01
|
||||
---
|
||||
|
||||
# Backends and dispatch
|
||||
|
||||
## What this is
|
||||
|
||||
The physical storage layer: the `Backend` trait contract every backend
|
||||
implements, the two shipped backends (kv, fs), and the size-threshold
|
||||
dispatch that routes between them. The trait is the crate's second
|
||||
stable seam (with the key encoding, ADR-002): backends must survive
|
||||
digest-layer evolution, so the boundary stays dumb.
|
||||
|
||||
## The Backend trait contract (ADR-003)
|
||||
|
||||
- **Opaque byte keys, opaque byte values.** Backends never learn what a
|
||||
digest is; typed `Key` converts at the store boundary (ADR-002). This
|
||||
is also the persistence boundary — a kv row or an fs filename has no
|
||||
schema-migration story, so the byte layout must be self-describing
|
||||
(algorithm-tagged keys).
|
||||
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name`.
|
||||
Streaming put/get shapes ride the store layer's seams (store-api.md);
|
||||
backends provide byte- or handle-level primitives underneath
|
||||
(`get` on the fs tier yields a file handle, not a loaded buffer).
|
||||
- **`list()` complete by contract.** A *malformed* list is as deadly as
|
||||
an incomplete one: list correctness is only observable through GC
|
||||
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
|
||||
deleted the wrong blobs, silently). Therefore:
|
||||
- every backend implementation must prove list correctness through
|
||||
sweep-outcome tests (the invariant's test gate lives in
|
||||
store-api.md);
|
||||
- an implementation that cannot enumerate (the rudolfs S3
|
||||
anti-lesson) must not ship — GC is a structural requirement, not an
|
||||
optional extra (ADR-005).
|
||||
- **Virgin-store reads are no-ops:** read paths treat a missing table
|
||||
as absent/empty (POC #1 finding 2).
|
||||
- **Namespace-blind:** backends never see namespaces, tenants, or
|
||||
reference structure (ADR-005 — the rudolfs inversion; physical
|
||||
storage is `hash → bytes` flat).
|
||||
|
||||
## Shipped backends
|
||||
|
||||
Two, exactly — this is the complete set the problem requires; the
|
||||
"third backend" fear is a category error fixed in ADR-004.
|
||||
|
||||
### kv backend (feature `kv`, default-on; sqlite)
|
||||
|
||||
Small blobs. Evidence (POC #3 finding A4, first-party measured):
|
||||
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
|
||||
— most git objects, workspace files, manifests), with the crossover at
|
||||
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
|
||||
|
||||
- Bounded reads; `has` as an EXISTS probe; prepared statements.
|
||||
- WAL + `synchronous=NORMAL` as the shipped durability tier (matching
|
||||
what the benchmark measured and what iroh's store ships).
|
||||
- Read paths tolerate the no-tables-yet database (see contract).
|
||||
|
||||
### fs backend (feature `fs`, default-on)
|
||||
|
||||
Large blobs. Flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}`
|
||||
sharding survives from iroh's conclusion (limits directory size on
|
||||
huge pools); stage-then-commit-rename for the two-pass unknown-length
|
||||
path; pread-based range reads (POC #3 finding A2 — local range serving
|
||||
is sound, e.g. for packfiles).
|
||||
|
||||
### mem backend (feature `mem`, default-off)
|
||||
|
||||
`BTreeMap`-shaped ephemeral backend for tests and in-process
|
||||
ephemerality. A testing/utility tier, never a production story.
|
||||
|
||||
## Size-threshold dispatch (ADR-003)
|
||||
|
||||
- **Routing is a pure function of content length.** Same content ⇒ same
|
||||
length ⇒ same backend; re-puts are deterministic. No content ever
|
||||
migrates between backends — the migration question existed only to
|
||||
patch the unknown-length asymmetry, which ADR-003's pre-threshold
|
||||
buffering eliminates by construction.
|
||||
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
|
||||
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
|
||||
constructor parameter, not an API change.
|
||||
- **Get fall-through:** small-tier miss queries the large tier
|
||||
(deterministic, cheap — a stat probe).
|
||||
- Per-namespace or per-tenant backend configuration: **rejected**
|
||||
(ADR-003 §Consequences — it would re-weld namespacing into the
|
||||
physical layer, the rudolfs anti-pattern ADR-005 inverts).
|
||||
|
||||
## Where a *new* backend could come from
|
||||
|
||||
The trait is open to future implementations (network stores, S3-like
|
||||
tiers), but nothing in the current consumer set requires one, and the
|
||||
contract is deliberately hostile to half-implementations (complete
|
||||
`list()`, GC-participating `delete`). Any future backend is a new ADR
|
||||
carrying its own sweep-safety story. This crate's roadmap is not
|
||||
blocked on one (see open-questions.md — alkfs intake may name needs
|
||||
externally; OQ-08).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| ADR | Decision | Summary |
|
||||
|---|---|---|
|
||||
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
|
||||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **OQ-08**: alkfs requirement intake may name storage requirements
|
||||
(e.g., durability tiers, sync-friendly layouts) that touch this
|
||||
layer — open, external owner (alkfs Phase 0).
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/poc-trait-dispatch-findings.md` findings 1/2/6
|
||||
- `docs/research/poc-largeblob-findings.md` findings A2/A4 (+ the
|
||||
re-runnable benchmark harness)
|
||||
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
|
||||
(sharding, crash ordering, inline thresholds rejected as weld)
|
||||
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
|
||||
not adopted (threshold dispatch chose the simpler policy;
|
||||
ADR-003 §Context)
|
||||
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
|
||||
backends must satisfy)
|
||||
Reference in new issue
Block a user