alkdev/alkblobs
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round

Browse Source 
ADR-010 (new): the Backend trait's I/O seams pinned before any code —
'get' returns a crate-internal read cursor (kv engines materialize;
'local' pread loop; pg-lo lo_get windows), 'put' has two named forms
(whole-value kv / staged-put large); GC state is store-core-owned via
a crate-internal contract-tested engine-state seam on the SQL-backed
engines (backends never learn liveness — ADR-005 verbatim); the store
core holds the joint entry+pin transaction (corrects ADR-009 §2's
ownership statement); kv-tier fleet-validity rule added (mirror of
ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin;
renewal rides have/re-put; pause-past-TTL falls to delete-then-recover);
feature graph pinned (pg-lo does not imply postgres).

ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs
engine (concrete impl, exactly five); store instance / node / fleet
split (fleet = pool-sharing, not node count — fixes requirements.md's
self-contradiction with REQ-2); mem demoted from 'backend'/'testing
tier' to the kv tier's third engine (contract-reference engine);
fs tier renamed 'large' including the feature name (last free moment
before code exists); constructor modes pinned (dual-tier default,
kv-only, mem-only) plus the composite fleet-validity predicate.

Consistency round across all specs and ADR-003/004/007/008/009: stale
pg-lo passages resolved per ADR-009's supersession; ADR-008 owner
'node id' → store-instance id; threshold wording corrected (conservative
edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport
shape specced; mem classification unified; 008/009 ADR files renamed to
match the tier rename; README deferral-policy recap aligned (third
category = decided-but-sequenced work, not a parking kind); POC crate
list completed.

Verification: two independent architecture review rounds (the first
found 4 criticals — unpinned trait I/O shapes, missing fleet-state
seam, node/fleet self-contradiction, pin-token/TTL conflict — all
resolved; final round: zero criticals); all markdown links resolve;
ADR tables complete (11 ADRs).
This commit is contained in:
glm-5.3-flash committed 2026-10-03 05:55:20 +00:00
1 parent 9bbe6f3ad9
commit abf8c5ae9c
18 files changed
+1137 -336
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/backends-and-dispatch.md
+170 -99
View File
@@ -1,17 +1,23 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — fs tier → large; mem is a kv engine)
---
# Backends and dispatch
# Backends, tiers, and dispatch
## What this is
The physical storage layer: the `Backend` trait contract every backend
implements, the two shipped backends (kv, fs), and the size-threshold
dispatch that routes between them. The trait is the crate's second
stable seam (with the key encoding, ADR-002): backends must survive
digest-layer evolution, so the boundary stays dumb.
The physical storage layer: the `Backend` trait contract every engine
implements, the two shipped tiers (kv, large — the tier formerly named
`fs`; renamed in ADR-011), and the size-threshold dispatch that routes
between them. The trait is the crate's second stable seam (with the
key encoding, ADR-002): engines must survive digest-layer evolution,
so the boundary stays dumb.
Vocabulary is pinned in requirements.md (ADR-011): **tier** = contract
+ routing class (exactly two; ADR-004), **engine** = concrete impl of
one tier's contract (exactly five), and this document's table is the
spec home of the constructor table (ADR-011 §4).
## What the Backend trait is (and is not)
@@ -32,28 +38,46 @@ purpose-built option. The wrong turn to rule out is "my data isn't
blobs, therefore I implement a Backend for it"; nothing a consumer needs
is ever reachable through that door.
## The Backend trait contract (ADR-003)
## The Backend trait contract (ADR-003; I/O seams per ADR-010)
- **Opaque byte keys, opaque byte values.** Backends never learn what a
digest is; typed `Key` converts at the store boundary (ADR-002). This
is also the persistence boundary — a kv row or an fs filename has no
schema-migration story, so the byte layout must be self-describing
(algorithm-tagged keys).
- **Opaque byte keys, opaque byte-valued entries.** Engines never learn
what a digest is; typed `Key` converts at the store boundary
(ADR-002). This is also the persistence boundary — a kv row, a
`local` filename, or an LO companion row has no schema-migration story, so
the byte layout must be self-describing (algorithm-tagged keys).
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` /
`size` (the length probe added by ADR-008). Streaming put/get shapes
ride the store layer's seams (store-api.md);
backends provide byte- or handle-level primitives underneath
(`get` on the fs tier yields a file handle, not a loaded buffer).
`size(key)` returns the entry's length or `None` if absent —
metadata only, never a content fetch: kv engines probe the indexed
`size` column both row shapes carry (POC #5 B5); the `local` fs
engine stats the file. Store-level `stat` and pool accounting ride
this method.
`size` (the length probe added by ADR-008). `size(key)` returns the
entry's length or `None` if absent — metadata only, never a content
fetch: kv engines probe the indexed `size` column both row shapes
carry (POC #5 B5); `local` stats the file; pg-lo reads the companion
row. Store-level `stat` and pool accounting ride this method.
- **The I/O seams are crate-internal abstractions, not engine shapes**
(ADR-010 §1 — the gap that would otherwise force guessing):
- `get` returns a **read cursor** the store core drives. kv-engine
cursors materialize the whole value (bounded by tier policy);
large-engine cursors stream — `local`'s is a pread loop over the
committed file, pg-lo's drives descriptorless `lo_get(oid, off,
len)` windows (ADR-009 §3, the C4 posture). Both serve
`read_range` (slice + slice digest, ADR-006) at one seam. The
cursor is not public API: store consumers see the facade's
`(len, stream)` shapes with the cursor behind them.
- `put` has **two forms**: the **whole-value put** (kv tier — the
settled CAS row write) and the **staged put** (large tier —
ADR-003's stage-then-commit-rename as a named per-engine
contract: the engine hands out a *staging cursor*; commit
publishes; every failure path converges on discard with zero
residue). Both land in the CAS entry shape; both participate in
pin-before-publish (ADR-005/010 §2 — the store core holds the
joint entry+pin transaction via the engine-state seam).
- Per-engine conformance: an engine is admitted only if its
cursor/stage/list bodies pass the uniform contract suite — the
POC #7 gate (10/10 exact-count sweep-outcome tests) is the norm;
suite details in store-api.md's invariants.
- **`list()` complete by contract.** A *malformed* list is as deadly as
an incomplete one: list correctness is only observable through GC
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
deleted the wrong blobs, silently). Therefore:
- every backend implementation must prove list correctness through
- every engine implementation must prove list correctness through
sweep-outcome tests (the invariant's test gate lives in
store-api.md);
- an implementation that cannot enumerate (the rudolfs S3
@@ -61,49 +85,60 @@ is ever reachable through that door.
optional extra (ADR-005).
- **Virgin-store reads are no-ops:** read paths treat a missing table
as absent/empty (POC #1 finding 2).
- **Namespace-blind:** backends never see namespaces, tenants, or
- **Namespace-blind:** engines never see namespaces, tenants, or
reference structure (ADR-005 — the rudolfs inversion; physical
storage is `hash → bytes` flat).
- **GC state never rides this trait** (ADR-010 §2): pins, window rows,
and sweep locks are store-core-owned via a crate-internal
engine-state companion seam on the SQL-backed engines (sqlite,
postgres, pg-lo) — invisible at the public trait; `local` and `mem`
cannot express it and are never fleet engines.
## Shipped backends
## Shipped tiers and engines
Two tiers, exactly — this counts *trait implementations this crate
ships* as CAS tiers, the dispatch routes between them; it is the
ships as CAS tiers* (the contracts dispatch routes between); it is the
complete set the problem requires, and the "third backend" fear is a
category error fixed in ADR-004. It says nothing about other storage
existing in a deployment: a downstream runs whatever else it needs on
its own media, beside the pool, above the store's seams (see "What the
Backend trait is"). **Tier count is not engine count**: the kv tier
carries two engines (sqlite, postgres) behind one trait — ADR-007;
per-node engine selection is a constructor parameter.
category error fixed in ADR-004 (tier count is not engine count). It
says nothing about other storage existing in a deployment: a downstream
runs whatever else it needs on its own media, beside the pool, above
the store's seams (see "What the Backend trait is").
### kv backend (feature `kv`, default-on); engines: sqlite (default) + postgres (feature `postgres`, default-off)
| Tier | Engine | Feature | Default | Fleet-valid |
|---|---|---|---|---|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
| kv | postgres | `postgres` | no | **yes** |
| kv | mem | `mem` | no | no (test/reference engine) |
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
| large | pg-lo | `pg-lo` | no | **yes** |
### kv tier (tier default-on, feature `kv`); engines sqlite (default) + postgres (feature `postgres`, default-off) + mem (feature `mem`, default-off)
Small blobs — **`kv` is the tier name; the engine is a per-node
constructor choice** between the shipped engines (ADR-007; ADR-003
carries sqlite's on-disk format trade: file compat is upstream
sqlite's guarantee). The shipped trait impls write plain
content-addressed rows into the chosen engine, nothing else. A
*third* engine remains the substitution door as amended: a new ADR
with its own sweep-safety proof (the door is no longer hypothetical —
ADR-007 is its first in-repo exercise); and a downstream's *other*
storage — schemas, manifests, whatever it likes including some other
sqlite file — never comes near this tier at all, per "What the
Backend trait is".
content-addressed rows into the chosen engine, nothing else (ADR-010:
the fleet state rides the companion seam, never the public trait). A
*third* non-mem engine remains the substitution door as amended: a new
ADR with its own sweep-safety proof (the door is exercised — ADR-007
in-repo); and a downstream's *other* storage — schemas, manifests,
whatever it likes including some other sqlite file — never comes near
this tier at all, per "What the Backend trait is".
The collapse ADR-007 buys: the classical downstream stack (git
server, vfs node) needed three storage systems — kv + fs + relational.
server, vfs node) needed three storage systems — kv + large content + relational.
Here the kv tier rides the relational engine itself, so a node
provisions **one relational engine + fs**; consumer tables (refs,
manifests, queues) sit beside the pool on the same engine or file by
the deployment's own choice (co-tenancy note: same-file consumer
writes share sqlite's single-writer ceiling — that sharing is the
node's trade; separate files remain available).
provisions **one relational engine + large-tier storage**; consumer
tables (refs, manifests, queues) sit beside the pool on the same
engine or file by the deployment's own choice (co-tenancy note: same-
file consumer writes share sqlite's single-writer ceiling — that
sharing is the node's trade; separate files remain available).
Evidence (POC #3 finding A4, first-party measured):
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
— most git objects, workspace files, manifests), with the crossover at
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
Evidence (POC #3 finding A4, first-party measured): sqlite is ~9-10×
faster than fs at 1-16 KiB (the git small-blob regime — most git
objects, workspace files, manifests), with the crossover at ~128-256
KiB where fs stops paying the B-tree row rewrite and wins.
**sqlite engine** (the default): WAL + `synchronous=NORMAL` as the
shipped durability tier (matching what the benchmark measured and what
@@ -128,22 +163,31 @@ sweep-outcome CI gate, run against dockerized postgres under
near-linearly to ~37k puts/s at 24 conns — the only engine whose
throughput *increases* under concurrency.
### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
**mem engine** (feature `mem`, default-off; ADR-011 §2 — formerly
described as the "mem backend"/"testing tier" before the vocabulary
was pinned): a `BTreeMap`-shaped ephemeral implementation of the kv
tier's contract, including `size` (trivially). For tests and
in-process ephemerality — never a production story, never
fleet-valid, and the **contract-reference engine**: every other
engine's conformance tests mirror its suite (the role it already
played in POC #1's miniature).
Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract,
multiple engines** — the medium is a per-node constructor choice
(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing
for fleets):
### large tier (tier default-on, feature `large`; the tier formerly named `fs` — renamed in ADR-011); engines `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
Large blobs. Like the kv tier (ADR-007), the large tier is **one
contract, multiple engines** — the medium is a per-node constructor
choice (ADR-008; REQ-2 in requirements.md makes the engine concept
load-bearing for fleets):
- **`local` engine** (the default; the ADR-003 backend as originally
specced): flat sharded layout:
`{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's
conclusion (limits directory size on huge pools); stage-then-commit-
rename for the two-pass unknown-length path; pread-based range
reads (POC #3 finding A2 — local range serving is sound, e.g. for
packfiles).
rename for the two-pass unknown-length path (the staged-put form,
ADR-010); pread-based range reads (POC #3 finding A2 — local range
serving is sound, e.g. for packfiles).
- **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009):
postgres Large Objects as the fs tier's storage — the same
postgres Large Objects as the large tier's storage — the same
engine-behind-one-trait move as ADR-007, one tier over, admitted on
POC #7's measured evidence (contract 10/10 exact-count tests;
durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC
@@ -154,66 +198,90 @@ for fleets):
operator's shrink path), autovacuum inherited, cached-get 20–50×
behind page-cache fs single-stream (~700 MB/s aggregate at 16
readers — the fleet serving picture).
- **Fleet locality contract (ADR-008, extended by ADR-009):** over one
shared pool, the fs tier is one of: shared media (every node's
`local` root on the same fleet-shared media — soundness caveats and
the required media guarantees in ADR-008), re-routed (one storage
node serves pool large-blob content via the ops surface; client
nodes are kv-tier-only over pool content), **or `pg-lo` for the
whole fleet** (ADR-009 — one engine, no shared media, no re-routing;
pool content lives in the same pg instance the kv tier rides).
Mixed per-node-local fs tiers over one pool is the partitioning
failure this contract exists to prevent — a documented deployment
invariant the constructor cannot fully prove but must be declared
against (ADR-008 names the enforceable seam and the detection
symptom).
### mem backend (feature `mem`, default-off)
`BTreeMap`-shaped ephemeral backend for tests and in-process
ephemerality. A testing/utility tier, never a production story.
Implements the full trait contract including `size` (trivially) — it
exists partly as the reference impl every other backend's conformance
tests can mirror.
- **Fleet locality contract (ADR-008, extended by ADR-009/010):** over
one shared pool, the large tier is one of: shared media (every
node's `local` root on the same fleet-shared media — soundness
caveats and the required media guarantees in ADR-008), re-routed
(one storage node serves pool large-blob content via the ops
surface; client nodes are kv-only over pool content), **or
`pg-lo` for the whole fleet** (ADR-009 — one engine, no shared
media, no re-routing; pool content lives in the same pg instance
the kv tier rides; the joint entry+pin tx is store-core-held,
ADR-010 §2). Mixed per-node-local large tiers over one pool is the
partitioning failure this contract exists to prevent — a documented
deployment invariant the constructor cannot fully prove but must be
declared against (ADR-008 names the enforceable seam and the
detection symptom). The kv tier has the mirrored rule (ADR-010 §3):
a fleet constructor requires a pool-shared kv engine (postgres);
two sqlite engines are two pools, never one.
- **The composite fleet-validity rule (one sentence, both tiers):** a
constructor configuration is fleet-valid iff **kv = postgres AND
(large = pg-lo OR large = `local`-on-declared-shared-media OR
large = none with the re-routing posture carrying pool large
content)**. Any other combination over one shared pool is invalid —
the constructor requires the declarations that make this predicate
checkable per node; cross-node truth remains the deployment's
verified invariant (ADR-008's seam). This predicate is
backends-and-dispatch.md's and ADR-011 §4's table combined; it is
stated here once so no reader composes it by inference.
## Size-threshold dispatch (ADR-003)
- **Routing is a pure function of content length.** Same content ⇒ same
length ⇒ same backend; re-puts are deterministic. No content ever
migrates between backends — the migration question existed only to
length ⇒ same tier; re-puts are deterministic. No content ever
migrates between tiers — the migration question existed only to
patch the unknown-length asymmetry, which ADR-003's pre-threshold
buffering eliminates by construction.
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
constructor parameter, not an API change.
- **Default threshold: 128 KiB** (constructor-tunable) — the bottom of
the measured crossover zone (~128-256 KiB, POC #3 A4; chosen at the
zone's conservative edge, not its midpoint). Re-tuning per
deployment media is a constructor parameter, not an API change.
- **Get fall-through:** small-tier miss queries the large tier
(deterministic, cheap — a stat probe).
- Per-namespace or per-tenant backend configuration: **rejected**
- Per-namespace or per-tenant engine configuration: **rejected**
(ADR-003 §Consequences — it would re-weld namespacing into the
physical layer, the rudolfs anti-pattern ADR-005 inverts).
## Where a *new* backend or engine could come from
### Constructor modes (ADR-011 §4)
- **dual-tier** (the default): kv + large, size-threshold dispatch
(ADR-003).
- **kv-only** (ADR-008's re-routing client posture, now named): no
large tier; over-threshold puts are a constructor-time error (the
re-routing posture handles them via the ops surface — the
consumer's composition, not a store mode).
- **mem-only:** the mem engine alone, no dispatch threshold in
effect; a testing/embedder-ephemeral posture, never production.
- Single-tier SQL modes (postgres-only, pg-lo-only *tiers*) do not
exist: both tiers are load-bearing (ADR-004). "Postgres-only" as in
one SQL *instance* serving both tiers exists and is the ADR-009
consolidation: dual-tier mode with kv=postgres + large=pg-lo over
one pool.
## Where a *new* engine could come from
The trait is open to future implementations (network stores, S3-like
tiers), but nothing in the current consumer set requires one, and the
contract is deliberately hostile to half-implementations (complete
`list()`, GC-participating `delete`). Any future backend — or a third
kv engine or second fs engine — is a new ADR carrying its own
sweep-safety story (ADR-007 is the precedent for a kv engine
addition; ADR-008 named pg-lo as the candidate second fs engine with
POC #7 as its admission evidence — the door is exercised twice). This
crate's roadmap is not blocked on one (see open-questions.md — alkfs
intake may name needs externally; OQ-08).
`list()`, GC-participating `delete`). Any future engine — per tier —
is a new ADR carrying its own sweep-safety story (ADR-007 is the
precedent for a kv engine addition; ADR-008 named pg-lo and ADR-009
admitted it with POC #7's evidence — the door is exercised twice).
This crate's roadmap is not blocked on one (see open-questions.md —
alkfs intake may name needs externally; OQ-08).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) |
| [004](decisions/004-two-backends-no-third.md) | Two tiers, no third | scope boundary against manifest-layer absorption |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | engines see hashes only |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + large tier per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; large tier engine-selectable (`local` default) |
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo admitted | postgres Large Objects as the large tier's second engine, on POC #7 |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; the kv fleet-validity rule |
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary | tier/engine/instance/node/fleet; mem is a kv engine; fs → large; constructor modes |
## Open Questions
@@ -230,10 +298,13 @@ intake may name needs externally; OQ-08).
curves + engine-posture deltas (ADR-007's evidence base)
- `docs/research/poc-redb-kv-findings.md` — redb ruled out at a
durability-tier mismatch (POC #6; recorded in ADR-007)
- `docs/research/poc-pglo-findings.md` — pg-lo's admission evidence
(POC #7; ADR-009)
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
(sharding, crash ordering, inline thresholds rejected as weld)
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
not adopted (threshold dispatch chose the simpler policy;
ADR-003 §Context)
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
backends must satisfy)
engines must satisfy); [requirements.md](requirements.md) (the pinned
vocabulary)