glm-5.3-flash
4b5009d85a
- overview/hashing/store-api/backends/gc/ops specs + README index, all draft status with YAML frontmatter and cross-referenced ADR/OQ - ADR-001: substrate posture (store layer alkcall-free = family conformance, not deviation) + ops placement decided (feature-gated module, no consumer-waited deferral) - ADR-002: canonical git-blob-sha-256 + key encoding — the declared wire-format ADR (one-way door) - ADR-003: backend contract, sqlite/fs/mem tiers, pure-function dispatch, no migration, unknown-length buffering semantics pinned - ADR-004: exactly two production backends; manifest/path-tree layers are consumer-side (the corrected two-vs-three-backends framing) - ADR-005: pooled CAS, three liveness sources, delete windows with pin-before-publish + delete-time arbitration (race closed), no ambient scheduling - ADR-006: whole-blob verification; chunk-tree encodings excluded by scoping, not deferred on a paused consumer - open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07 /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task - AGENTS.md: status updated to Phase 1, gix-odb path corrected Verification: cargo test / clippy -D warnings / fmt --check clean
124 lines
5.6 KiB
Markdown
124 lines
5.6 KiB
Markdown
---
|
||
status: draft
|
||
last_updated: 2026-10-01
|
||
---
|
||
|
||
# Backends and dispatch
|
||
|
||
## What this is
|
||
|
||
The physical storage layer: the `Backend` trait contract every backend
|
||
implements, the two shipped backends (kv, fs), and the size-threshold
|
||
dispatch that routes between them. The trait is the crate's second
|
||
stable seam (with the key encoding, ADR-002): backends must survive
|
||
digest-layer evolution, so the boundary stays dumb.
|
||
|
||
## The Backend trait contract (ADR-003)
|
||
|
||
- **Opaque byte keys, opaque byte values.** Backends never learn what a
|
||
digest is; typed `Key` converts at the store boundary (ADR-002). This
|
||
is also the persistence boundary — a kv row or an fs filename has no
|
||
schema-migration story, so the byte layout must be self-describing
|
||
(algorithm-tagged keys).
|
||
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name`.
|
||
Streaming put/get shapes ride the store layer's seams (store-api.md);
|
||
backends provide byte- or handle-level primitives underneath
|
||
(`get` on the fs tier yields a file handle, not a loaded buffer).
|
||
- **`list()` complete by contract.** A *malformed* list is as deadly as
|
||
an incomplete one: list correctness is only observable through GC
|
||
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
|
||
deleted the wrong blobs, silently). Therefore:
|
||
- every backend implementation must prove list correctness through
|
||
sweep-outcome tests (the invariant's test gate lives in
|
||
store-api.md);
|
||
- an implementation that cannot enumerate (the rudolfs S3
|
||
anti-lesson) must not ship — GC is a structural requirement, not an
|
||
optional extra (ADR-005).
|
||
- **Virgin-store reads are no-ops:** read paths treat a missing table
|
||
as absent/empty (POC #1 finding 2).
|
||
- **Namespace-blind:** backends never see namespaces, tenants, or
|
||
reference structure (ADR-005 — the rudolfs inversion; physical
|
||
storage is `hash → bytes` flat).
|
||
|
||
## Shipped backends
|
||
|
||
Two, exactly — this is the complete set the problem requires; the
|
||
"third backend" fear is a category error fixed in ADR-004.
|
||
|
||
### kv backend (feature `kv`, default-on; sqlite)
|
||
|
||
Small blobs. Evidence (POC #3 finding A4, first-party measured):
|
||
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
|
||
— most git objects, workspace files, manifests), with the crossover at
|
||
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
|
||
|
||
- Bounded reads; `has` as an EXISTS probe; prepared statements.
|
||
- WAL + `synchronous=NORMAL` as the shipped durability tier (matching
|
||
what the benchmark measured and what iroh's store ships).
|
||
- Read paths tolerate the no-tables-yet database (see contract).
|
||
|
||
### fs backend (feature `fs`, default-on)
|
||
|
||
Large blobs. Flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}`
|
||
sharding survives from iroh's conclusion (limits directory size on
|
||
huge pools); stage-then-commit-rename for the two-pass unknown-length
|
||
path; pread-based range reads (POC #3 finding A2 — local range serving
|
||
is sound, e.g. for packfiles).
|
||
|
||
### mem backend (feature `mem`, default-off)
|
||
|
||
`BTreeMap`-shaped ephemeral backend for tests and in-process
|
||
ephemerality. A testing/utility tier, never a production story.
|
||
|
||
## Size-threshold dispatch (ADR-003)
|
||
|
||
- **Routing is a pure function of content length.** Same content ⇒ same
|
||
length ⇒ same backend; re-puts are deterministic. No content ever
|
||
migrates between backends — the migration question existed only to
|
||
patch the unknown-length asymmetry, which ADR-003's pre-threshold
|
||
buffering eliminates by construction.
|
||
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
|
||
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
|
||
constructor parameter, not an API change.
|
||
- **Get fall-through:** small-tier miss queries the large tier
|
||
(deterministic, cheap — a stat probe).
|
||
- Per-namespace or per-tenant backend configuration: **rejected**
|
||
(ADR-003 §Consequences — it would re-weld namespacing into the
|
||
physical layer, the rudolfs anti-pattern ADR-005 inverts).
|
||
|
||
## Where a *new* backend could come from
|
||
|
||
The trait is open to future implementations (network stores, S3-like
|
||
tiers), but nothing in the current consumer set requires one, and the
|
||
contract is deliberately hostile to half-implementations (complete
|
||
`list()`, GC-participating `delete`). Any future backend is a new ADR
|
||
carrying its own sweep-safety story. This crate's roadmap is not
|
||
blocked on one (see open-questions.md — alkfs intake may name needs
|
||
externally; OQ-08).
|
||
|
||
## Design Decisions
|
||
|
||
| ADR | Decision | Summary |
|
||
|---|---|---|
|
||
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
|
||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
|
||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
|
||
|
||
## Open Questions
|
||
|
||
- **OQ-08**: alkfs requirement intake may name storage requirements
|
||
(e.g., durability tiers, sync-friendly layouts) that touch this
|
||
layer — open, external owner (alkfs Phase 0).
|
||
|
||
## References
|
||
|
||
- `docs/research/poc-trait-dispatch-findings.md` findings 1/2/6
|
||
- `docs/research/poc-largeblob-findings.md` findings A2/A4 (+ the
|
||
re-runnable benchmark harness)
|
||
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
|
||
(sharding, crash ordering, inline thresholds rejected as weld)
|
||
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
|
||
not adopted (threshold dispatch chose the simpler policy;
|
||
ADR-003 §Context)
|
||
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
|
||
backends must satisfy) |