alkdev/alkblobs
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round

Browse Source 
ADR-010 (new): the Backend trait's I/O seams pinned before any code —
'get' returns a crate-internal read cursor (kv engines materialize;
'local' pread loop; pg-lo lo_get windows), 'put' has two named forms
(whole-value kv / staged-put large); GC state is store-core-owned via
a crate-internal contract-tested engine-state seam on the SQL-backed
engines (backends never learn liveness — ADR-005 verbatim); the store
core holds the joint entry+pin transaction (corrects ADR-009 §2's
ownership statement); kv-tier fleet-validity rule added (mirror of
ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin;
renewal rides have/re-put; pause-past-TTL falls to delete-then-recover);
feature graph pinned (pg-lo does not imply postgres).

ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs
engine (concrete impl, exactly five); store instance / node / fleet
split (fleet = pool-sharing, not node count — fixes requirements.md's
self-contradiction with REQ-2); mem demoted from 'backend'/'testing
tier' to the kv tier's third engine (contract-reference engine);
fs tier renamed 'large' including the feature name (last free moment
before code exists); constructor modes pinned (dual-tier default,
kv-only, mem-only) plus the composite fleet-validity predicate.

Consistency round across all specs and ADR-003/004/007/008/009: stale
pg-lo passages resolved per ADR-009's supersession; ADR-008 owner
'node id' → store-instance id; threshold wording corrected (conservative
edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport
shape specced; mem classification unified; 008/009 ADR files renamed to
match the tier rename; README deferral-policy recap aligned (third
category = decided-but-sequenced work, not a parking kind); POC crate
list completed.

Verification: two independent architecture review rounds (the first
found 4 criticals — unpinned trait I/O shapes, missing fleet-state
seam, node/fleet self-contradiction, pin-token/TTL conflict — all
resolved; final round: zero criticals); all markdown links resolve;
ADR tables complete (11 ADRs).
This commit is contained in:
glm-5.3-flash committed 2026-10-03 05:55:20 +00:00
1 parent 9bbe6f3ad9
commit abf8c5ae9c
18 files changed
+1121 -320
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/README.md
+55 -21
View File
@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7)
last_updated: 2026-10-03 (ADR-010/011 — I/O seams + GC-state home; vocabulary pinned, fs → large, mem joins kv)
---
# alkblobs — Architecture
@@ -10,8 +10,9 @@ last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7)
**Phase 1 (Architecture).** Phase 0 (Exploration) converged — see
`docs/research/phase-0.md` for the vision, the evidence trail, and the
POC register (all POCs passed or were absorbed; POC code lives in
standalone crates (`/workspace/alkblobs-trait-poc`,
`/workspace/alkblobs-largeblob-poc`); findings live in
standalone crates under `/workspace/` — `alkblobs-trait-poc`,
`alkblobs-largeblob-poc`, `alkblobs-postgres-poc`, `alkblobs-redb-poc`,
`alkblobs-pglo-poc`; findings live in
`docs/research/`). No implementation exists yet (`src/lib.rs` is a
placeholder).
@@ -24,14 +25,14 @@ there — not inherited as hedges:
the family pattern, not deviation from it — and the ops surface rides
alkcall, feature-gated.
- **ADR-004** fixes the "two backends" framing: the crate ships exactly
two backends (kv small / fs large — both required by measured scale
two tiers (kv small / large — both required by measured scale
economics); the path→hash mapping layer (the alkfs/alknet "vfs" shape)
is a consumer-layer concern and must never become a third backend.
- **ADR-007** widens the kv tier's engine set rather than hedging it:
sqlite (default) + postgres (feature-gated) behind one trait. The
deployment economics this records: a downstream node (git server, vfs)
classically needed three storage systems — kv + fs + relational — and
this design collapses them to **one relational engine + fs**, with the
classically needed three storage systems — kv + large content + relational — and
this design collapses the three to two: **one relational engine + large-tier storage**, with the
engine a per-node constructor choice. Resolved from evidence in hand
(POCs #3/#5/#6); the would-be "deployment trigger" framing was caught
at review as circular hedging (a deployment running alkblobs cannot
@@ -42,13 +43,13 @@ there — not inherited as hedges:
planning fact predating all POCs): the `Backend` trait gains a
`size` probe (before any backend ships data); fleet GC gets
DB-backed pins, an advisory-locked single sweeper, and SQL-level
delete arbitration (single-node postures unchanged); and the fs tier
delete arbitration (single-node postures unchanged); and the large tier
becomes engine-selectable (`local` default; `pg-lo` named candidate,
admission-gated on POC #7). It also adds
[requirements.md](requirements.md) — the REQ anchor that ends
per-session re-derivation of consumer facts.
- **ADR-009** (2026-10-03) admits `pg-lo` — postgres Large Objects as
the fs tier's second engine (feature `pg-lo`, default-off) — on
the large tier's second engine (feature `pg-lo`, default-off) — on
POC #7's passed gate (`docs/research/poc-pglo-findings.md`):
contract 10/10 exact-count tests, durable put ≈ fs durable put at
≥1 MiB, descriptorless `lo_get` window gets, transactional LO
@@ -59,16 +60,44 @@ there — not inherited as hedges:
readers), catalog space reused-but-never-returned, autovacuum
inherited. The admission door (ADR-003, exercised by ADR-007 and
ADR-009) has now shipped both its tier-extending engines.
- **ADR-010** (2026-10-03) pins the `Backend` trait's I/O seams —
the last unpinned signatures before implementation: `get` returns a
crate-internal **read cursor** (kv engines materialize; large-tier
engines stream — `local`'s pread loop, pg-lo's `lo_get` windows), `put` has
two named forms (whole-value for kv, **staged put** for large —
stage-then-commit-rename promoted from behavior to contract); rules
that **GC state is store-core-owned** (backends never learn
liveness — ADR-005 verbatim) hosted by SQL-backed engines via a
crate-internal contract-tested **engine-state companion seam**
(schema/pin/window/lock clauses), rules **the store core holds the
joint entry+pin transaction** (correcting ADR-009 §2's ownership
statement), adds the **kv tier's fleet-validity rule** (mirror of
ADR-008 §3 — postgres is the fleet engine; a sqlite-fleet posture
is not on offer), resolves the **ops pin-token TTL race** (the
token *is* the pin; renewal rides have/re-put; pause-past-TTL falls
to delete-then-recover), and pins the **feature graph** (`pg-lo`
does not imply `postgres`).
- **ADR-011** (2026-10-03) pins the vocabulary: **tier** (contract +
routing class — exactly two) vs **engine** (concrete impl — exactly
five: kv's sqlite/postgres/mem, large's `local`/pg-lo);
**mem demoted from "backend"/"testing tier" to the kv tier's
third engine — the contract-reference engine**; **store instance** /
**node** / **fleet** split so REQ-2's "one node may run multiple
instances" phrasing is no longer self-contradictory (fleet =
pool-sharing, not node count); and the **large tier is renamed
`large`** (feature `fs` → `large`) — its routing criterion is
length, not medium — with constructor modes (dual-tier default,
kv-only, mem-only) pinned in the same ADR's table.
## Architecture Documents
| Document | Scope | Status |
|---|---|---|
| [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft |
| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, node/pool/fleet vocabulary | draft |
| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, pinned vocabulary (tier/engine/instance/node/fleet) | draft |
| [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft |
| [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft |
| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, shipped backends, size-threshold dispatch | draft |
| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, tiers + engines table, size-threshold dispatch, constructor modes | draft |
| [gc-and-namespaces.md](gc-and-namespaces.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | draft |
| [ops-surface.md](ops-surface.md) | alkcall-backed have/need + fetch/put ops, ACL mapping | draft |
| [open-questions.md](open-questions.md) | Centralized OQ tracker (incl. promoted Phase 0 register) | draft |
@@ -84,8 +113,10 @@ there — not inherited as hedges:
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted |
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted |
| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo — postgres Large Objects as the fs tier's second engine (admitted on POC #7) | Accepted |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); large-tier engines | Accepted |
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo — postgres Large Objects as the large tier's second engine (admitted on POC #7) | Accepted |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Backend I/O seams (read cursor, staged put); GC state store-core-owned via the engine-state seam; kv fleet-validity; pin-token renewal | Accepted |
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary: tier/engine/instance/node/fleet; mem joins kv; large tier renamed `large`; constructor modes | Accepted |
## Open Questions
@@ -94,25 +125,28 @@ Tracked in [open-questions.md](open-questions.md). The Phase 0 register
remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping,
alkfs intake; carried for visibility, gating nothing here). OQ-09
(namespace-visibility default) resolved closed-by-default; OQ-10
(second kv engine) resolved by ADR-007. The pg-lo fs-engine question
(second kv engine) resolved by ADR-007. The pg-lo large-engine question
resolved to *sequenced work*, not a parked question: REQ-2/ADR-008
named it the candidate fs-tier engine and POC #7
named it the candidate large-tier engine and POC #7
(`docs/research/poc-pglo-spec.md`) was its admission evidence — a
task, not a wait; that task completed 2026-10-03
(`docs/research/poc-pglo-findings.md`, passed) and ADR-009 shipped
the engine. **Deferral
policy (the "Schrödinger's code" rule):** a *decision this crate needs
before shipping* may not be deferred on a dependency that is itself
waiting for this crate to exist. Two parking kinds remain legitimate:
waiting for this crate to exist. Legitimate parked kinds:
`deferred(scope)` on a deciding fact that exists independently of this
crate, and externally-owned questions (how a consumer maps onto this
crate) that gate no decision here — full definitions in the header of
[open-questions.md](open-questions.md). A corollary worth restating,
because it recurs: a fact this crate must create (its own first
deployment, its own first consumer) is never a deciding input —
decisions stand on evidence in hand, and future needs reopen via named
requirements, not via waiting. (Corollary applications so far: OQ-09
and OQ-10 both resolved on it.)
[open-questions.md](open-questions.md). A third category is *not* a
parked kind: **decided-but-sequenced work** (a resolved decision whose
admission gate requires evidence — the pg-lo round's POC #7 is the
worked example; a task with an owner, not a parked question). A
corollary worth restating, because it recurs: a fact this crate must
create (its own first deployment, its own first consumer) is never a
deciding input — decisions stand on evidence in hand, and future needs
reopen via named requirements, not via waiting. (Corollary
applications so far: OQ-09 and OQ-10 both resolved on it.)
## Lifecycle
docs/architecture/backends-and-dispatch.md
+170 -99
View File
@@ -1,17 +1,23 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — fs tier → large; mem is a kv engine)
---
# Backends and dispatch
# Backends, tiers, and dispatch
## What this is
The physical storage layer: the `Backend` trait contract every backend
implements, the two shipped backends (kv, fs), and the size-threshold
dispatch that routes between them. The trait is the crate's second
stable seam (with the key encoding, ADR-002): backends must survive
digest-layer evolution, so the boundary stays dumb.
The physical storage layer: the `Backend` trait contract every engine
implements, the two shipped tiers (kv, large — the tier formerly named
`fs`; renamed in ADR-011), and the size-threshold dispatch that routes
between them. The trait is the crate's second stable seam (with the
key encoding, ADR-002): engines must survive digest-layer evolution,
so the boundary stays dumb.
Vocabulary is pinned in requirements.md (ADR-011): **tier** = contract
+ routing class (exactly two; ADR-004), **engine** = concrete impl of
one tier's contract (exactly five), and this document's table is the
spec home of the constructor table (ADR-011 §4).
## What the Backend trait is (and is not)
@@ -32,28 +38,46 @@ purpose-built option. The wrong turn to rule out is "my data isn't
blobs, therefore I implement a Backend for it"; nothing a consumer needs
is ever reachable through that door.
## The Backend trait contract (ADR-003)
## The Backend trait contract (ADR-003; I/O seams per ADR-010)
- **Opaque byte keys, opaque byte values.** Backends never learn what a
digest is; typed `Key` converts at the store boundary (ADR-002). This
is also the persistence boundary — a kv row or an fs filename has no
schema-migration story, so the byte layout must be self-describing
(algorithm-tagged keys).
- **Opaque byte keys, opaque byte-valued entries.** Engines never learn
what a digest is; typed `Key` converts at the store boundary
(ADR-002). This is also the persistence boundary — a kv row, a
`local` filename, or an LO companion row has no schema-migration story, so
the byte layout must be self-describing (algorithm-tagged keys).
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` /
`size` (the length probe added by ADR-008). Streaming put/get shapes
ride the store layer's seams (store-api.md);
backends provide byte- or handle-level primitives underneath
(`get` on the fs tier yields a file handle, not a loaded buffer).
`size(key)` returns the entry's length or `None` if absent —
metadata only, never a content fetch: kv engines probe the indexed
`size` column both row shapes carry (POC #5 B5); the `local` fs
engine stats the file. Store-level `stat` and pool accounting ride
this method.
`size` (the length probe added by ADR-008). `size(key)` returns the
entry's length or `None` if absent — metadata only, never a content
fetch: kv engines probe the indexed `size` column both row shapes
carry (POC #5 B5); `local` stats the file; pg-lo reads the companion
row. Store-level `stat` and pool accounting ride this method.
- **The I/O seams are crate-internal abstractions, not engine shapes**
(ADR-010 §1 — the gap that would otherwise force guessing):
- `get` returns a **read cursor** the store core drives. kv-engine
cursors materialize the whole value (bounded by tier policy);
large-engine cursors stream — `local`'s is a pread loop over the
committed file, pg-lo's drives descriptorless `lo_get(oid, off,
len)` windows (ADR-009 §3, the C4 posture). Both serve
`read_range` (slice + slice digest, ADR-006) at one seam. The
cursor is not public API: store consumers see the facade's
`(len, stream)` shapes with the cursor behind them.
- `put` has **two forms**: the **whole-value put** (kv tier — the
settled CAS row write) and the **staged put** (large tier —
ADR-003's stage-then-commit-rename as a named per-engine
contract: the engine hands out a *staging cursor*; commit
publishes; every failure path converges on discard with zero
residue). Both land in the CAS entry shape; both participate in
pin-before-publish (ADR-005/010 §2 — the store core holds the
joint entry+pin transaction via the engine-state seam).
- Per-engine conformance: an engine is admitted only if its
cursor/stage/list bodies pass the uniform contract suite — the
POC #7 gate (10/10 exact-count sweep-outcome tests) is the norm;
suite details in store-api.md's invariants.
- **`list()` complete by contract.** A *malformed* list is as deadly as
an incomplete one: list correctness is only observable through GC
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
deleted the wrong blobs, silently). Therefore:
- every backend implementation must prove list correctness through
- every engine implementation must prove list correctness through
sweep-outcome tests (the invariant's test gate lives in
store-api.md);
- an implementation that cannot enumerate (the rudolfs S3
@@ -61,49 +85,60 @@ is ever reachable through that door.
optional extra (ADR-005).
- **Virgin-store reads are no-ops:** read paths treat a missing table
as absent/empty (POC #1 finding 2).
- **Namespace-blind:** backends never see namespaces, tenants, or
- **Namespace-blind:** engines never see namespaces, tenants, or
reference structure (ADR-005 — the rudolfs inversion; physical
storage is `hash → bytes` flat).
- **GC state never rides this trait** (ADR-010 §2): pins, window rows,
and sweep locks are store-core-owned via a crate-internal
engine-state companion seam on the SQL-backed engines (sqlite,
postgres, pg-lo) — invisible at the public trait; `local` and `mem`
cannot express it and are never fleet engines.
## Shipped backends
## Shipped tiers and engines
Two tiers, exactly — this counts *trait implementations this crate
ships* as CAS tiers, the dispatch routes between them; it is the
ships as CAS tiers* (the contracts dispatch routes between); it is the
complete set the problem requires, and the "third backend" fear is a
category error fixed in ADR-004. It says nothing about other storage
existing in a deployment: a downstream runs whatever else it needs on
its own media, beside the pool, above the store's seams (see "What the
Backend trait is"). **Tier count is not engine count**: the kv tier
carries two engines (sqlite, postgres) behind one trait — ADR-007;
per-node engine selection is a constructor parameter.
category error fixed in ADR-004 (tier count is not engine count). It
says nothing about other storage existing in a deployment: a downstream
runs whatever else it needs on its own media, beside the pool, above
the store's seams (see "What the Backend trait is").
### kv backend (feature `kv`, default-on); engines: sqlite (default) + postgres (feature `postgres`, default-off)
| Tier | Engine | Feature | Default | Fleet-valid |
|---|---|---|---|---|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
| kv | postgres | `postgres` | no | **yes** |
| kv | mem | `mem` | no | no (test/reference engine) |
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
| large | pg-lo | `pg-lo` | no | **yes** |
### kv tier (tier default-on, feature `kv`); engines sqlite (default) + postgres (feature `postgres`, default-off) + mem (feature `mem`, default-off)
Small blobs — **`kv` is the tier name; the engine is a per-node
constructor choice** between the shipped engines (ADR-007; ADR-003
carries sqlite's on-disk format trade: file compat is upstream
sqlite's guarantee). The shipped trait impls write plain
content-addressed rows into the chosen engine, nothing else. A
*third* engine remains the substitution door as amended: a new ADR
with its own sweep-safety proof (the door is no longer hypothetical —
ADR-007 is its first in-repo exercise); and a downstream's *other*
storage — schemas, manifests, whatever it likes including some other
sqlite file — never comes near this tier at all, per "What the
Backend trait is".
content-addressed rows into the chosen engine, nothing else (ADR-010:
the fleet state rides the companion seam, never the public trait). A
*third* non-mem engine remains the substitution door as amended: a new
ADR with its own sweep-safety proof (the door is exercised — ADR-007
in-repo); and a downstream's *other* storage — schemas, manifests,
whatever it likes including some other sqlite file — never comes near
this tier at all, per "What the Backend trait is".
The collapse ADR-007 buys: the classical downstream stack (git
server, vfs node) needed three storage systems — kv + fs + relational.
server, vfs node) needed three storage systems — kv + large content + relational.
Here the kv tier rides the relational engine itself, so a node
provisions **one relational engine + fs**; consumer tables (refs,
manifests, queues) sit beside the pool on the same engine or file by
the deployment's own choice (co-tenancy note: same-file consumer
writes share sqlite's single-writer ceiling — that sharing is the
node's trade; separate files remain available).
provisions **one relational engine + large-tier storage**; consumer
tables (refs, manifests, queues) sit beside the pool on the same
engine or file by the deployment's own choice (co-tenancy note: same-
file consumer writes share sqlite's single-writer ceiling — that
sharing is the node's trade; separate files remain available).
Evidence (POC #3 finding A4, first-party measured):
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
— most git objects, workspace files, manifests), with the crossover at
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
Evidence (POC #3 finding A4, first-party measured): sqlite is ~9-10×
faster than fs at 1-16 KiB (the git small-blob regime — most git
objects, workspace files, manifests), with the crossover at ~128-256
KiB where fs stops paying the B-tree row rewrite and wins.
**sqlite engine** (the default): WAL + `synchronous=NORMAL` as the
shipped durability tier (matching what the benchmark measured and what
@@ -128,22 +163,31 @@ sweep-outcome CI gate, run against dockerized postgres under
near-linearly to ~37k puts/s at 24 conns — the only engine whose
throughput *increases* under concurrency.
### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
**mem engine** (feature `mem`, default-off; ADR-011 §2 — formerly
described as the "mem backend"/"testing tier" before the vocabulary
was pinned): a `BTreeMap`-shaped ephemeral implementation of the kv
tier's contract, including `size` (trivially). For tests and
in-process ephemerality — never a production story, never
fleet-valid, and the **contract-reference engine**: every other
engine's conformance tests mirror its suite (the role it already
played in POC #1's miniature).
Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract,
multiple engines** — the medium is a per-node constructor choice
(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing
for fleets):
### large tier (tier default-on, feature `large`; the tier formerly named `fs` — renamed in ADR-011); engines `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
Large blobs. Like the kv tier (ADR-007), the large tier is **one
contract, multiple engines** — the medium is a per-node constructor
choice (ADR-008; REQ-2 in requirements.md makes the engine concept
load-bearing for fleets):
- **`local` engine** (the default; the ADR-003 backend as originally
specced): flat sharded layout:
`{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's
conclusion (limits directory size on huge pools); stage-then-commit-
rename for the two-pass unknown-length path; pread-based range
reads (POC #3 finding A2 — local range serving is sound, e.g. for
packfiles).
rename for the two-pass unknown-length path (the staged-put form,
ADR-010); pread-based range reads (POC #3 finding A2 — local range
serving is sound, e.g. for packfiles).
- **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009):
postgres Large Objects as the fs tier's storage — the same
postgres Large Objects as the large tier's storage — the same
engine-behind-one-trait move as ADR-007, one tier over, admitted on
POC #7's measured evidence (contract 10/10 exact-count tests;
durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC
@@ -154,66 +198,90 @@ for fleets):
operator's shrink path), autovacuum inherited, cached-get 20–50×
behind page-cache fs single-stream (~700 MB/s aggregate at 16
readers — the fleet serving picture).
- **Fleet locality contract (ADR-008, extended by ADR-009):** over one
shared pool, the fs tier is one of: shared media (every node's
`local` root on the same fleet-shared media — soundness caveats and
the required media guarantees in ADR-008), re-routed (one storage
node serves pool large-blob content via the ops surface; client
nodes are kv-tier-only over pool content), **or `pg-lo` for the
whole fleet** (ADR-009 — one engine, no shared media, no re-routing;
pool content lives in the same pg instance the kv tier rides).
Mixed per-node-local fs tiers over one pool is the partitioning
failure this contract exists to prevent — a documented deployment
invariant the constructor cannot fully prove but must be declared
against (ADR-008 names the enforceable seam and the detection
symptom).
### mem backend (feature `mem`, default-off)
`BTreeMap`-shaped ephemeral backend for tests and in-process
ephemerality. A testing/utility tier, never a production story.
Implements the full trait contract including `size` (trivially) — it
exists partly as the reference impl every other backend's conformance
tests can mirror.
- **Fleet locality contract (ADR-008, extended by ADR-009/010):** over
one shared pool, the large tier is one of: shared media (every
node's `local` root on the same fleet-shared media — soundness
caveats and the required media guarantees in ADR-008), re-routed
(one storage node serves pool large-blob content via the ops
surface; client nodes are kv-only over pool content), **or
`pg-lo` for the whole fleet** (ADR-009 — one engine, no shared
media, no re-routing; pool content lives in the same pg instance
the kv tier rides; the joint entry+pin tx is store-core-held,
ADR-010 §2). Mixed per-node-local large tiers over one pool is the
partitioning failure this contract exists to prevent — a documented
deployment invariant the constructor cannot fully prove but must be
declared against (ADR-008 names the enforceable seam and the
detection symptom). The kv tier has the mirrored rule (ADR-010 §3):
a fleet constructor requires a pool-shared kv engine (postgres);
two sqlite engines are two pools, never one.
- **The composite fleet-validity rule (one sentence, both tiers):** a
constructor configuration is fleet-valid iff **kv = postgres AND
(large = pg-lo OR large = `local`-on-declared-shared-media OR
large = none with the re-routing posture carrying pool large
content)**. Any other combination over one shared pool is invalid —
the constructor requires the declarations that make this predicate
checkable per node; cross-node truth remains the deployment's
verified invariant (ADR-008's seam). This predicate is
backends-and-dispatch.md's and ADR-011 §4's table combined; it is
stated here once so no reader composes it by inference.
## Size-threshold dispatch (ADR-003)
- **Routing is a pure function of content length.** Same content ⇒ same
length ⇒ same backend; re-puts are deterministic. No content ever
migrates between backends — the migration question existed only to
length ⇒ same tier; re-puts are deterministic. No content ever
migrates between tiers — the migration question existed only to
patch the unknown-length asymmetry, which ADR-003's pre-threshold
buffering eliminates by construction.
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
constructor parameter, not an API change.
- **Default threshold: 128 KiB** (constructor-tunable) — the bottom of
the measured crossover zone (~128-256 KiB, POC #3 A4; chosen at the
zone's conservative edge, not its midpoint). Re-tuning per
deployment media is a constructor parameter, not an API change.
- **Get fall-through:** small-tier miss queries the large tier
(deterministic, cheap — a stat probe).
- Per-namespace or per-tenant backend configuration: **rejected**
- Per-namespace or per-tenant engine configuration: **rejected**
(ADR-003 §Consequences — it would re-weld namespacing into the
physical layer, the rudolfs anti-pattern ADR-005 inverts).
## Where a *new* backend or engine could come from
### Constructor modes (ADR-011 §4)
- **dual-tier** (the default): kv + large, size-threshold dispatch
(ADR-003).
- **kv-only** (ADR-008's re-routing client posture, now named): no
large tier; over-threshold puts are a constructor-time error (the
re-routing posture handles them via the ops surface — the
consumer's composition, not a store mode).
- **mem-only:** the mem engine alone, no dispatch threshold in
effect; a testing/embedder-ephemeral posture, never production.
- Single-tier SQL modes (postgres-only, pg-lo-only *tiers*) do not
exist: both tiers are load-bearing (ADR-004). "Postgres-only" as in
one SQL *instance* serving both tiers exists and is the ADR-009
consolidation: dual-tier mode with kv=postgres + large=pg-lo over
one pool.
## Where a *new* engine could come from
The trait is open to future implementations (network stores, S3-like
tiers), but nothing in the current consumer set requires one, and the
contract is deliberately hostile to half-implementations (complete
`list()`, GC-participating `delete`). Any future backend — or a third
kv engine or second fs engine — is a new ADR carrying its own
sweep-safety story (ADR-007 is the precedent for a kv engine
addition; ADR-008 named pg-lo as the candidate second fs engine with
POC #7 as its admission evidence — the door is exercised twice). This
crate's roadmap is not blocked on one (see open-questions.md — alkfs
intake may name needs externally; OQ-08).
`list()`, GC-participating `delete`). Any future engine — per tier —
is a new ADR carrying its own sweep-safety story (ADR-007 is the
precedent for a kv engine addition; ADR-008 named pg-lo and ADR-009
admitted it with POC #7's evidence — the door is exercised twice).
This crate's roadmap is not blocked on one (see open-questions.md —
alkfs intake may name needs externally; OQ-08).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) |
| [004](decisions/004-two-backends-no-third.md) | Two tiers, no third | scope boundary against manifest-layer absorption |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | engines see hashes only |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + large tier per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; large tier engine-selectable (`local` default) |
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo admitted | postgres Large Objects as the large tier's second engine, on POC #7 |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; the kv fleet-validity rule |
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary | tier/engine/instance/node/fleet; mem is a kv engine; fs → large; constructor modes |
## Open Questions
@@ -230,10 +298,13 @@ intake may name needs externally; OQ-08).
curves + engine-posture deltas (ADR-007's evidence base)
- `docs/research/poc-redb-kv-findings.md` — redb ruled out at a
durability-tier mismatch (POC #6; recorded in ADR-007)
- `docs/research/poc-pglo-findings.md` — pg-lo's admission evidence
(POC #7; ADR-009)
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
(sharding, crash ordering, inline thresholds rejected as weld)
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
not adopted (threshold dispatch chose the simpler policy;
ADR-003 §Context)
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
backends must satisfy)
engines must satisfy); [requirements.md](requirements.md) (the pinned
vocabulary)
docs/architecture/decisions/002-canonical-hash-and-key-encoding.md
+2 -2
View File
@@ -16,7 +16,7 @@ addresses, not algorithms).
The residual was the **key encoding mechanics** — the byte layout of
keys inside backends. This is the crate's declared one-way door
(AGENTS convention 5): key bytes persist in kv rows and fs filenames,
(AGENTS convention 5): key bytes persist in kv rows and `local`-engine filenames,
with no schema-migration story; and this is the wire-format ADR that
must exist before the first consumer (convention 7's gate on publishing
wire-relevant decisions). POC #1 validated the mechanics empirically:
@@ -26,7 +26,7 @@ both algorithms in one flat pool (finding 5).
## Decision
### Hash policy (three tiers)
### Hash policy (three classes)
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
SHA-256. The store's canonical algorithm; git objects are first-class
docs/architecture/decisions/003-backend-contract-and-dispatch.md
+27 -15
View File
@@ -5,10 +5,15 @@
Accepted (engine pin amended by ADR-007 — the kv tier ships two
engines, sqlite and postgres, behind this contract; the dispatch layer
and trait are unchanged. The trait was extended by ADR-008 with a
`size` length probe — before any backend ships data — and the fs
tier became engine-selectable (`local` default; pg-lo candidate), so
this ADR's "engines"/"backends" counts are read through ADR-004's
tier≠engine scope note)
`size` length probe — before any backend ships data — and the second
tier became engine-selectable (`local` default; pg-lo admitted by
ADR-009), so this ADR's "engines"/"backends" counts are read through
ADR-004's tier≠engine scope note. The I/O shapes behind `get`/`put`
are pinned by ADR-010 (read cursor / staged put). The tier's name
changed: `fs` → `large` (ADR-011 — its routing criterion is length,
not medium); this ADR's per-engine counts also gained the `mem` kv
engine as ADR-011's vocabulary resolves — mem is a kv engine, not a
tier)
## Context
@@ -54,28 +59,34 @@ Evidence in hand (no further POC or consumer-waiting needed):
- **Virgin-store reads are no-ops** (missing table = absent/empty).
- **Namespace-blind** (ADR-005): backends see hashes only.
### Shipped backends (two, default-on; plus a testing tier)
### Shipped tiers (two, default-on; the kv tier also carries the mem engine)
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
the regime holding most git objects, workspace files, and manifests.
WAL + `synchronous=NORMAL` as shipped durability; prepared
statements; no-tables-yet tolerance. (The tier's engine set was
widened to include postgres by ADR-007.)
- **`fs` (default-on)** for large blobs — flat
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
widened to include postgres by ADR-007, and mem — the ephemeral
reference engine — by ADR-011's vocabulary resolution.)
- **`fs` (default-on; renamed `large` by ADR-011)** for large blobs —
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
conclusion); stage-then-commit-rename; pread range reads (sound
locally per ADR-006).
- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and
in-process ephemerality. Never a production story.
locally per ADR-006). (The tier's engine set was widened by ADR-008/
009.)
- **`mem` (default-off):** ephemeral `BTreeMap` kv engine for tests
and in-process ephemerality — **an engine of the kv tier, not a
third tier** (ADR-011 §2; the pre-ADR-011 "testing tier" phrasing
is superseded). Never a production story; the contract-reference
engine.
Both production backends are default-on (batteries-included, the
Both production tiers are default-on (batteries-included, the
alkgit feature-model precedent); `mem` is not.
### Dispatch policy
- **Size-threshold routing; default 128 KiB, constructor-tunable**
(midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4).
(chosen at the bottom of the measured ~128-256 KiB crossover zone —
the conservative edge of the flat zone; POC #3 A4).
- **Unknown-length puts buffer in memory up to the threshold; the
buffer's fate has three pinned semantics** (finding A6's option (d)):
1. **Threshold exceeded mid-stream** (non-restartable sources are
@@ -141,9 +152,10 @@ alkgit feature-model precedent); `mem` is not.
**Neutral**
- A network-backed backend (S3-like) remains possible via the same
- A network-backed engine (S3-like) remains possible via the same
trait but ships nothing here; its list/GC obligations are the
acceptance gate.
acceptance gate; per-tier, via a new ADR (ADR-011's vocabulary: an
engine addition, not a tier addition).
## References
docs/architecture/decisions/004-two-backends-no-third.md
+27 -19
View File
@@ -2,7 +2,12 @@
## Status
Accepted
Accepted (the count's vocabulary is pinned by ADR-011: the "two" here
are *tiers* — kv + large, the latter renamed from `fs`; "backend"
names the same thing at a different altitude. The `mem` item below is
the kv tier's third engine, per ADR-011 §2, not an extra tier —
the pre-ADR-011 "testing tier" carve-out is superseded. The I/O
seam-level refinements of the trait contract are ADR-010's)
## Context
@@ -11,9 +16,10 @@ rounds before being edited away, but the correction it was reaching for
arrived only in the Phase 1 review, and needs recording so no future
round reabsorbs the concern incorrectly:
- There are **always** two physical backends in this problem: storage
for small content (fast, kv/sqlite) and the filesystem for large
content. This is not ugly, it is the measured shape of the problem
- There are **always** two physical tiers in this problem: storage
for small content (fast, kv/sqlite) and storage for large content
(the large tier / filesystem). This is not ugly, it is the measured
shape of the problem
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
- The "two backends is ugly" sentiment was actually about a **third
layer** in the old alknet-filesystem research — the *virtual
@@ -30,23 +36,25 @@ is the boundary it will build against.
## Decision
**This crate ships exactly two production backends and nothing more;
**This crate ships exactly two production tiers and nothing more;
every manifest/namespace/path-tree layer is a consumer, by construction
and by contract.** (A non-production ephemeral `mem` tier exists for
tests — ADR-003; it is not a storage story and does not extend this
boundary.)
and by contract.** (A non-production ephemeral `mem` engine exists for
tests — ADR-003, resolved to a kv-tier engine by ADR-011 §2; it is not
a storage story and does not extend this boundary.)
**Scope note (post-ADR-007 reconciliation): the "two" here counts
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
engines behind it to a set (sqlite default + postgres, feature-gated)
so a node's kv rides its relational engine of choice. The count that
matters to this ADR — trait impls *this crate ships* as CAS tiers —
remains two (kv + fs); a second kv engine does not make a third
backend, and nothing in the engine set touches the manifest-layer
boundary below.
**Scope note (post-ADR-007 reconciliation, vocabulary finalized by
ADR-011): the "two" here counts *tiers*, not engines.** The kv tier is
one contract; ADR-007 widened the engines behind it to a set (sqlite
default + postgres, feature-gated; mem the ephemeral reference engine,
ADR-011) so a node's kv rides its relational engine of choice. The
count that matters to this ADR — trait impls *this crate ships* as CAS
tiers — remains two (kv + large, the tier ADR-011 renamed from `fs`);
a second kv engine does not make a third tier, and nothing in the
engine set touches the manifest-layer boundary below.
1. **Two production backends is the complete set.** kv/sqlite (small)
+ fs (large) cover the scale economics with measured evidence
1. **Two production tiers is the complete set.** kv/sqlite (small)
+ large (the tier formerly named `fs`) cover the scale economics
with measured evidence
(ADR-003); no other physical tier is needed by any current or
planned consumer's *blob* needs.
2. **The "third backend" is not this crate's layer.** The vfs/appfile
@@ -86,7 +94,7 @@ boundary below.
**Positive**
- The crate's scope stays testable and small: two backends, one pool,
- The crate's scope stays testable and small: two tiers, one pool,
one hash domain. No schema/versioning surface for manifest formats —
consumers iterate on theirs freely.
- alkgit and alkfs build their (very different) mapping layers on one
docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md
+1 -1
View File
@@ -49,7 +49,7 @@ served by range reads).
(git refs, alkfs path trees) and hand liveness over per sweep via the
live-shared seam. In-memory namespace tables remain available for
ephemeral consumers. This also dissolves the hidden wrinkle of a
store-internal root table under fs-only deployments (where would its
store-internal root table under deployments without a SQL engine (where would its
durable bytes live?). Ephemeral consumers are covered by pins; a
deployment that needs store-durable roots registers them as
consumer-owned durable state and sweeps through the seam.
docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md
+7 -5
View File
@@ -2,7 +2,9 @@
## Status
Accepted
Accepted (the engine story's I/O seams and GC-state home are ruled by
ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as
its third engine — is pinned by ADR-011)
## Context
@@ -57,9 +59,9 @@ in OQ-09.
**The kv tier ships two engines — sqlite (default) and postgres —
behind one `Backend` trait; engine selection is a constructor
parameter on the backend's configuration.** A node provisions one
relational engine plus the fs tier; the kv tier rides whichever
relational engine plus the large tier; the kv tier rides whichever
engine the operator chose. This collapses the classical three-engine
downstream stack (kv + fs + relational) to two, and makes which
downstream stack (kv + large content + relational) to two, and makes which
relational engine a per-node choice rather than a crate fork.
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
@@ -83,7 +85,7 @@ relational engine a per-node choice rather than a crate fork.
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
virgin-store no-op reads, namespace-blindness. The dispatch layer
(size threshold, kv↔fs routing, fall-through) is engine-blind and
(size threshold, kv↔large routing, fall-through) is engine-blind and
unchanged.
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
as requirements** (they are the difference between a port and an
@@ -127,7 +129,7 @@ collapses to its own deployment choice among shipped engines.
**Positive**
- The classical downstream stack lands at two engines per node (one
relational + fs), eliminating the "sqlite *beside* the real
relational engine + large tier), eliminating the "sqlite *beside* the real
database" duplication this crate's consumers would otherwise carry
forever.
- Per-node engine choice (constructor parameter, per ADR-003/004)
docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md → docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md
+56 -37
View File
@@ -1,10 +1,14 @@
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
# ADR-008: Trait length probe, fleet GC, and large-tier engines
## Status
Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
passed gate; every other part of this ADR stands as written)
passed gate — the two §Consequences passages that said pg-lo "remains
unbuilt" are resolved by that same supersession; the tier this ADR's
title and text call "fs" is renamed `large` by ADR-011; the fleet GC
mechanism's state home and the joint-tx ownership are ruled by
ADR-010; every other part of this ADR stands as written)
## Context
@@ -18,7 +22,7 @@ sharing one pool via postgres; requirements.md):
probe"; store-api.md) and pool accounting rides `list` + `stat`
(ADR-005), but the trait's methods are `has/get/put/delete/list/
name` with no value-metadata access. The kv engines' row shapes
both carry a `size` column (POC #5 B5), and the fs tier can stat a
both carry a `size` column (POC #5 B5), and the large tier can stat a
file — but with no trait method, kv `stat` degenerates to a full
value load. The trait is the crate's second stable seam (ADR-003);
discovering this during implementation would be a breaking trait
@@ -32,8 +36,8 @@ sharing one pool via postgres; requirements.md):
node B's in-flight put is invisible. Data loss, structurally: the
topology REQ-2 requires does not satisfy the protocol ADR-005
specifies.
3. **The fs tier is node-local, silently partitioning a fleet.** The
fs backend stores to a sharded layout on the node's own filesystem
3. **The large tier is node-local, silently partitioning a fleet.** The
large tier stores to a sharded layout on the node's own filesystem
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
put via node A exists only on A's disk; node B's get fall-through
queries its own fs and misses, and `list()` unions disagree per
@@ -41,7 +45,7 @@ sharing one pool via postgres; requirements.md):
hypothetical.
These three travel together: the fleet topology forces #2 and #3, and
#3's resolution constrains what the fs tier must support, while #1 is
#3's resolution constrains what the large tier must support, while #1 is
a trait-level gap any topology hits.
## Decision
@@ -55,7 +59,7 @@ tier:
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
never a value load. The `size` column both POC row shapes carry
becomes contract-relevant.
- **fs engine (`local`)**: stat/metadata read, no content fetch.
- **large engine (`local`)**: stat/metadata read, no content fetch.
- The store facade's `stat` (store-api.md) and the accounting path
(ADR-005) ride this method; the `has` method remains a pure
existence probe.
@@ -73,7 +77,9 @@ Under a fleet, the mechanism moves into the shared engine:
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
inserts in one transaction on the fleet engine: the entry row and
its pin row commit together (or neither does). The pin row carries
an owner (node id) and expiry: the pin guard renews its row every
an owner (a store-instance id — ADR-011's vocabulary; "node id" is
this ADR's original wording, superseded because a node may host
several instances) and expiry: the pin guard renews its row every
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
parameters, not API). **Protection = an un-expired pin row**: at
delete-time arbitration, only `expiry > now()` pins count; an
@@ -131,9 +137,9 @@ Under a fleet, the mechanism moves into the shared engine:
is shared machinery. The pin-row expiry/ttl machinery exists only
where fleet engines are active.
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible
The fs tier is a tier, not a medium. It ships engines the way the kv
The large tier is a tier, not a medium. It ships engines the way the kv
tier does (ADR-007), and a deployment's topology determines which are
valid:
@@ -141,7 +147,7 @@ valid:
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
REQ-4) and for fleet nodes using **shared media** (below).
- **Fleet + shared media (deployment requirement):** multi-node pools
may keep the `local` engine by pointing every node's fs tier at
may keep the `local` engine by pointing every node's large tier at
shared/replicated media. The soundness claim is *qualified*, not
assumed: immutable-once-committed files plus stage-then-commit-
rename are sound on media that provides (a) atomic rename within a
@@ -158,43 +164,52 @@ valid:
not share media designates **one storage node**; large-blob writes
and reads over the pool route through it via the ops surface
(verified fetch/put; ADR-001). Client nodes' constructors are
configured with a **kv-tier-only dispatch** (or an fs tier reserved
for node-private, consumer-owned content — which is outside the
pool and outside this crate's GC; it is the consumer's own storage
beside the pool per ADR-004, not pool content). The facade shape on
configured with **kv-only mode** (ADR-011 §4's name for what this
bullet first called "kv-tier-only dispatch") — no local large tier
over pool content at all; a client node's *node-private,
consumer-owned* large-tier content is outside the pool and outside
this crate's GC entirely: it is the consumer's own storage beside
the pool per ADR-004, not a constructor mode over pool content, and
the composite fleet-validity rule (backends-and-dispatch.md)
therefore has no "local on node-private media over one pool" entry.
The facade shape on
a client node is its own store for sub-threshold content plus its
registered ops-put path for large content — a consumer-side
composition, not a store-level mode.
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
as the fs tier's storage — the same "engine behind one contract"
move as ADR-007, one tier over. Named here so the door is explicit,
with its admission gated on measured evidence (POC #7: LO write/read
- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects
as the large tier's storage —
the same "engine behind one contract"
move as ADR-007, one tier over. Originally named as a candidate,
admission gated on measured evidence (POC #7: LO write/read
curves at the packfile regime, tx-scoped handle cost under
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
behavior) and its own engine ADR. **Not shipped speculatively;**
behavior) and its own engine ADR. NOT shipped speculatively;
REQ-2's immediate fleet answers are shared media or re-routing;
pg-lo is the consolidation option if those are unacceptable. *Update
2026-10-03: POC #7 ran and passed its gate —
`docs/research/poc-pglo-findings.md` (companion-table authority,
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
behind page-cache fs by a measured 20-50x aggregate-multiplied to
~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture).
The engine ADR's remaining evidence path is open; admission remains
gated on it, not on speculation.*
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s
aggregate at 16 readers). Superseded the same day: ADR-009 admitted
the engine — this bullet's original "candidate, not shipped"
posture is historical; the engine is shipped, feature `pg-lo`.*
- **The no-mixed-large-tiers rule is a deployment invariant, enforced at
the enforceable seam.** Cross-node configuration cannot be validated
by any one constructor (it sees only its own node). What the
constructor *can* enforce: a fleet-mode constructor (fleet engines
active) requires an explicit fs-tier declaration — `local` (with the
active) requires an explicit large-tier declaration — `local` (with the
operator's assertion that its root is fleet-shared media), routed
(no local fs over pool content), or — when shipped — `pg-lo`. The
(kv-only mode, ADR-011 §4; no local filesystem over pool content), or
`pg-lo`. The
cross-node truth ("is node B's `local` actually the same media as
node A's?") is a deployment invariant this crate documents and ops
verifies, not something a constructor can prove; a partitioned
tier's observable signature (get fall-through misses for entries
another node wrote) is the documented detection symptom.
another node wrote) is the documented detection symptom. The
composite fleet-validity predicate over both tiers is pinned in
backends-and-dispatch.md ("The composite fleet-validity rule").
In sum: a fleet's fs tier over pool content is either shared media (all
In sum: a fleet's large tier over pool content is either shared media (all
nodes' `local` roots on the same fleet-shared media) or routed (one
storage node); mixing per-node-local media over one pool is the
partitioning failure this decision exists to prevent.
@@ -211,9 +226,10 @@ partitioning failure this decision exists to prevent.
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
invariant extends to fleets as a *stated* mechanism, not an
accident of single-process luck.
- The fs tier's fleet problem has named, shipped-code-free answers
(shared media, re-routing) and a named consolidation candidate
(pg-lo) whose admission path is the same POC-first door as ADR-007.
- The large tier's fleet problem has named answers: shipped-code-free
for shared media and re-routing, shipped-code for the consolidation
option (pg-lo — admitted by ADR-009, the same POC-first door as
ADR-007).
**Negative**
@@ -223,9 +239,9 @@ partitioning failure this decision exists to prevent.
surface for orphaned pins at node crash — the expiry is the cleanup
handle), advisory-lock coordination, and SQL-level arbitration
added on top of the already-intricate delete-window state machine.
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
re-route pays for its ADR + POC before shipping large blobs — the
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
- `pg-lo` was built under ADR-009 after POC #7 passed — the
measured, deliberate sequence of the ADR-006 pattern (admission
gate first, engine second), not a hedge; resolved same-day.
**Neutral**
@@ -245,7 +261,7 @@ partitioning failure this decision exists to prevent.
columns; the driver stack pg-lo would ride)
- ADR-002 (the one-way-door discipline this trait change applies),
ADR-003 (the trait contract and admission gate `size` and pg-lo
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
amend/enter), ADR-004 (tier≠engine, extended to the large tier),
ADR-005 (the GC invariant and single-node protocol extended here,
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
follows), ADR-007 (the engine-behind-one-trait precedent)
@@ -254,4 +270,7 @@ partitioning failure this decision exists to prevent.
[store-api.md](../store-api.md) (`stat`'s contract);
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
topology's transport)
topology's transport); ADR-010 (the fleet mechanism's state home
and the joint-tx ownership this ADR's §2 implied but did not rule);
ADR-011 (the vocabulary — instance/fleet — this ADR's topology
language resolves to; the large tier renamed `large`)
docs/architecture/decisions/009-pg-lo-fs-tier-engine.md → docs/architecture/decisions/009-pg-lo-large-tier-engine.md
+15 -12
View File
@@ -1,14 +1,17 @@
# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine
# ADR-009: pg-lo — postgres Large Objects as the large tier's second engine
## Status
Accepted (admission evidence: POC #7, passed 2026-10-03 —
`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
gate)
gate. The tier this ADR calls "fs" is renamed `large` by ADR-011);
§2's "put = one transaction" tx-ownership statement is corrected by
ADR-010 §2 (the store core holds the tx; the engine contributes its
staged-put mechanics into it — engine content unchanged))
## Context
ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's
ADR-008 named `pg-lo` as the candidate large-tier engine for REQ-2's
fleet topology (multiple store instances sharing one pg pool, serving
large/beyond-threshold pool content) — expressly gated on the ADR-003
admission door: measured evidence first, in the ADR-007 shape. POC #7
@@ -38,7 +41,7 @@ The gate's three legs all passed:
## Decision
**The fs tier ships `pg-lo` as its second engine — postgres Large
**The large tier ships `pg-lo` as its second engine — postgres Large
Objects — behind the same `Backend` trait, feature `pg-lo` (default-
off), constructor-selected per node.** REQ-2's deployment choice set
(ADR-008) extends from {shared media, re-routing} to include the
@@ -58,7 +61,7 @@ consolidation option as a shipped engine.
into the LO and commit the row on completion; rollback discards
both.
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
the fs-tier handle closes over the companion row (oid + length) —
the large-tier handle closes over the companion row (oid + length) —
immutable, so the row is authoritative — not over a tx-scoped
descriptor. Ranged reads are single stateless statements,
pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
@@ -82,7 +85,7 @@ consolidation option as a shipped engine.
(committed LOs without companion rows — mixed/legacy stores). The
crash story is structurally clean: LO lifecycle is transactional
in modern postgres; kill mid-tx leaves zero pages, zero
half-commits (cleaner than fs `local`'s stage-file residue,
half-commits (cleaner than `local`'s stage-file residue,
which needs its own recovery sweep).
6. **Performance deltas, as named capacity facts (not hidden, not
solved):** single-stream cached get runs 20–50× behind page-cache
@@ -105,13 +108,13 @@ consolidation option as a shipped engine.
**Positive**
- REQ-2's consolidation option is real and shipped: a fleet node runs
*postgres only* — kv (bytea) tier + fs (LO) tier from one engine,
*postgres only* — kv (bytea) tier + large (LO) tier from one engine,
no shared-media requirement, no re-routing topology, no consumer-ops
dependency for pool content.
- The engine adds ~400 lines of impl over an already-fixed contract —
the additive shape both ADR-007 and ADR-008 projected, now
evidenced.
- Crash hygiene is *better* than fs `local` (transactional LO
- Crash hygiene is *better* than `local` (transactional LO
lifecycle vs. stage-file residue); the engine's only recovery sweep
targets a class (bypassed-LO) that healthy deployments never produce.
@@ -123,9 +126,9 @@ consolidation option as a shipped engine.
— the fleet case — scales to ~700 MB/s on the POC box; per-box
numbers shift with cores.)
- LO catalog space is never returned (monitor + occasional
operator-run `VACUUM FULL`), the inverse of fs `local`'s
operator-run `VACUUM FULL`), the inverse of `local`'s
immediate-return.
- Feature/maintenance surface: the fs tier now carries two engines
- Feature/maintenance surface: the large tier now carries two engines
(like the kv tier), each with posture deltas to keep honest
(ADR-007's negative, inherited).
@@ -147,10 +150,10 @@ consolidation option as a shipped engine.
- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
this engine serves); REQ-1/3/4 unchanged postures
- ADR-003 (the trait contract + admission door exercised twice now),
ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC
ADR-004 (tier ≠ engine, large tier edition), ADR-005/008 (the GC
invariants this engine participates in — table-level, nothing
fs-specific), ADR-007 (the engine-admission precedent and driver
stack)
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the large-tier
engine surface this amends); [overview.md](../overview.md);
[store-api.md](../store-api.md)
docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
+325
View File
@@ -0,0 +1,325 @@
# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam
## Status
Accepted
## Context
The engine story closed at the decision level (ADR-007/008/009), but
three seams that implementation must not guess are unpinned. All three
concentrate where the tier/engine split stretched the original
contract:
1. **The `get`/`put` value shapes were never unified.** The trait POC's
`get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
a file handle, not a loaded buffer"; the pg-lo POC's get is a
`LoGetHandle` driving `lo_get(oid, off, len)` windows — three
structurally different shapes behind one trait, and no document
named the unified contract. Same on the put side: a byte slice
(kv), a stage-then-commit-rename stream (`local`), and a
chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
trait is the crate's second stable seam (ADR-003); its I/O shapes
are decided before code, not discovered per engine.
2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
rows committed with the entry; staged delete-window candidates
(including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
advisory lock for the sweeper. None of that is expressible through
the public trait method set, and ADR-003's "the shipped trait
impls write plain content-addressed rows into the chosen engine,
nothing else" reads as a bar the fleet mechanism violates. Also
unresolved: who owns the joint entry+pin transaction — ADR-009
says the pg-lo put's single tx carries the pin row, but ADR-005
says `Pin` is a store-core RAII guard under the arbitration lock.
The trait has no parameter for either, and "the backend learns
about pins" would contradict ADR-005. The ops pin-token contract
(ops-surface.md) inherits the same conflict: a remote putter holds
a token, but an expiry-guarded pin row goes stale if the holder
pauses — a sweep could delete a remotely-in-flight put whose row
expired.
3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
the large tier's fleet forms (shared media, re-routing) but pins
nothing for the kv tier, which the fleet story centers on: two
nodes' sqlite engines are trivially partitioned (two files = two
pools, not one), and no document said so.
The deciding facts are in hand (Schrödinger's-code rule — none of
these may wait on consumers): the POCs fixed every engine's native
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
advisory lock, staged window); ADR-001 §4 fixed that ops is a
store-facade consumer with no backend side-door. What is missing is
composition: how the store core drives all engines through one
contract while owning GC state.
Alternatives considered for the fleet-state seam, rejected:
- **Put the fleet machinery on the public trait** (pin/window/table
methods on `Backend`): every engine — including `mem` and `local` —
must then implement SQL-shaped machinery it cannot express. Rejected.
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
row, backend-maintained): contradicts ADR-005's "backends never
learn liveness"; and the arbitration decision (expiry compare,
liveness consult) still belongs to the store, so the store ends up
reaching into the engine's tx anyway. Rejected.
- **A raw-SQL side-channel** (connection escape hatch the store core
may use arbitrarily): turns the engine abstraction off for exactly
the machinery (fleet GC) it most needs to be sound on. Rejected.
The seam below is that escape hatch made honest — named clauses,
contract-tested per engine, instead of raw access.
## Decision
### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put
**The trait stays one trait.** Every engine implements the same method
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
What changes: the trait's I/O types are the crate-internal
abstractions below, which each engine constructs its own way — the
store core programs the abstraction, never the engine shape.
- **`get` returns a read cursor.** One abstraction, two body families:
- **kv engines** (sqlite, postgres, mem): the cursor materializes
the whole value (the engine's row read; bounded by the small-blob
tier policy). Its range reads are in-memory slices; `len()` reads
the row's `size` column.
- **large engines** (`local`, pg-lo): the cursor streams; the medium
is hidden in the cursor body — `local`'s is a pread loop over the
committed file (immutable once committed); pg-lo's drives
descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
posture — pool-friendly, stateless per window; the
tx-scoped-descriptor form is the fallback). `len()` reads the
companion row or stats the file — metadata only, never content
(ADR-008's discipline).
- Both body families serve the ADR-006 range contract (`read_range`
→ slice + slice digest) at one seam. The cursor is
**store-internal — not public API**: store consumers see the
facade's `(len, stream)` shapes (store-api.md) with the cursor
staying behind them; the store's stream utility adapts engine
cursor bodies, it does not expose them.
- **`put` exists in two forms, selected by the store core before
content flows; both land in the CAS entry shape and both
participate in pin-before-publish (§2 decides who holds the tx).**
- **Whole-value put** for the kv tier: the settled row write.
POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
`ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
- **Staged put** for the large tier — ADR-003's
stage-then-commit-rename promoted from behavior to a **named
per-engine contract**: the engine hands out a *staging cursor*
(`local`: the stage file under the store's stage-hygiene protocol;
pg-lo: chunked `lowrite` statements over a tx the store core
holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
measured knee). Commit publishes the entry;
every failure path converges on discard (store-api.md stage
hygiene, per engine, exact-count tested — POC #3 A5's shape).
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
walks the engine's companion structure (rows / files / LO
companion); `size` is metadata-only (ADR-008). The exact-count
sweep-outcome gate (ADR-003) runs per engine.
**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
`pg-lo` does **not** imply feature `postgres`. Each is an engine
feature pulling the shared driver cluster (`tokio-postgres` +
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
node is coherent and supported. overview.md's dependency table pins
the edge.
### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested
**Ownership: the store core owns all GC state — in-process pin and
liveness maps *and* the fleet state's rows and locks. The public
`Backend` trait stays exactly what ADR-003 says (opaque
`hash → bytes`, namespace-blind; "plain content-addressed rows,
nothing else" remains true of the public trait). Backends never learn
liveness (ADR-005) — verbatim.**
The store core reaches the fleet machinery through a **crate-internal
(not exported) engine-state companion trait**, implemented by the
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
`mem` excluded by construction (it cannot be expressed on them and
they are never fleet engines). ADR-008's mechanism fixes *what*; the
seam is *where* the mechanism's SQL lives. Clauses:
- `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
`gc_window (key, visible_after)` at store construction (store-core
call, engine-neutral API; dialect-specific SQL bodies). `owner`
is the **store-instance id** (ADR-011's vocabulary; ADR-008's
"node id" wording predates that split). The pin row's identity is
`(key, owner)` — a dedup re-put by a different instance is a
different pin row; protection composes by row (an entry is
protected while any un-expired pin row exists), which is
ADR-005's refcount semantics expressed in rows. The embedder's
liveness table is *not* created by this seam — ADR-008: schema
pinned by the embedder; the sweep reads it and never owns it.
**Per-engine applicability (the test-gate shape):** sqlite —
`ensure_gc_schema` (window table only, since the window staging
is shared machinery per ADR-008 §2) + `window_stage`/
`window_take` + the sweep-lock clause's single-node body; the
pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
bodies but are exercised by unit tests only — they are dead
machinery on sqlite because no sqlite fleet is on offer (§3).
postgres and pg-lo — all clause families, contract-tested,
fleet-active.
- `pin_commit(joint_tx, entries)` — the store core **holds the
transaction**; the engine contributes its put mechanics (row
shape, CAS clause, LO writes) and the pin-row inserts, and both
commit together. This is the joint entry+pin transaction's owner,
ruled: **the store holds the tx; the engine contributes mechanics
to it.** The public `put` remains the plain, pin-free path — the
store's normal put path routes through `pin_commit` wherever the
engine is seam-capable and fleet state is active.
- pg-lo correction: ADR-009 §2's "put = one transaction" is
unchanged in content but its ownership statement is hereby
corrected — the tx is held by the store core; the pg-lo engine
contributes its staged-put mechanics and companion-row insert
into it. The engine sees pins only as rows; never as semantics.
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
update. The store's pin guard owns the TTL/3 renewal schedule
(ADR-008); the engine owns only the SQL.
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
executes ADR-008's delete-time arbitration (un-expired pin rows +
embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
keyed, at delete time); the store core receives the answer and
applies the decision. If fleet clauses ever grow, they grow here —
the seam is the named home, with contract tests per clause.
- `window_stage(candidates)` / `window_take(now)` — the staged
delete-window candidates. Per ADR-008 §2 the window table exists
on **every** engine (sqlite included): staged candidates are
engine rows; what differs by topology is where *liveness* lives
(in-process maps for single-node engines, tables for fleets), not
whether staging exists.
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
pg advisory lock (ADR-008) for the SQL engines it applies to; the
store treats lock contention exactly as ADR-008 says — the second
sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
nothing deleted. (SQLite today has no fleet story — see §3 — so its
clause body exists but is exercised by single-node tests only.)
Each clause is contract-tested per its applicability (the
per-engine split above): the conformance suite of §1 grows the fleet
clauses for postgres and pg-lo; `local` and
`mem` are excluded by construction.
### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)
**A fleet constructor requires a pool-shared kv engine.** Postgres is
the shipped engine that satisfies this. A sqlite-fleet mechanism does
not exist and is not on offer (per ADR-007's redb precedent this is
recorded as a non-option, not an open question); two nodes' sqlite
engines over "one pool" are two pools — the partitioning failure, the
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
enforceable
seam is the same as ADR-008 §3's: the constructor can require an
explicit declaration; it cannot prove cross-node truth. The
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
and is closed under both tiers.
### 4. The ops pin-token contract's TTL race, resolved by the ownership rule
**The token *is* the pin — there is no second pin state.** Under fleet
engines the put's pin is the expiry-carrying row (ADR-008); the token
identifies the putter as the row's owner. The token itself is an
**opaque id minted by the ops layer** over the pin row it just
committed — the ops layer (not the store facade) owns the
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
a lookup by the renewal op; no schema is specced for it beyond that
(it is a handle, not durable state — the durable state is the pin row
the seam wrote). Concretely:
- **When the server-side `Pin` guard drops:** at response time. The
put lands pinned (the joint tx guarantees the entry and its pin row
commit together); the handler's in-process guard — the thing ADR-005
pins *in-process* with — is released when the `blobs/put` response
completes and the token is handed out. From that moment the pin
row's expiry is the only protection, which is exactly why the
renewal duty below exists. (If the handler kept its guard for the
token's whole life, the TTL machinery would be dead code on the ops
path — the row's expiry, not the guard's presence, is the fleet
contract's liveness signal.)
- The putter renews the pin by renewing the row — through ops it
already speaks: a `blobs/have` probe whose handler, observing a
valid token presented alongside the digest, calls the
crate-internal `pin_renew(key, owner)` via the ops module's own
store-core access (the ops module is in-crate — it crosses the
crate-internal seam by construction, the same as the put handler
holds the `Pin` guard; what never happens is a *consumer* reaching
the seam. ADR-008's `has` remains a pure probe: the renewal is the
ops layer's token-authenticated act on the pin row, not a `has`
semantic), or a re-put of the same digest (CAS dedup; the re-put's
joint tx upserts the key+owner pin row, renewing it). The renewal
cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
20 s at the default 60 s TTL.**
- A putter pausing past TTL falls to ADR-008's expiry semantics
verbatim: the row is abandoned and reaped; the backstop is
delete-then-recover (ADR-005) — the digest is re-offered, the put
is idempotent under CAS. The race resolves to a re-put, never to
silent loss.
- For the `local` engine the token is the in-process guard as ADR-005
had it: nothing persists, nothing expires; the process crash is
ADR-005's in-flight-put loss case with the same backstop. (The
cross-wire token under `local` is the ops layer's in-memory handle
to that guard — same mapping discipline, no expiry.)
ops-surface.md's hand-over contract gains the renewal line and the
token-mapping note; nothing else changes.
## Consequences
**Positive**
- The trait's I/O shapes are decided before code; engine tasks can be
written without guessing signatures. The per-engine
conformance-suite shape (POC #7's 10/10 gate as the norm) gives
decomposition a uniform test gate across all five engines.
- "Backends never learn liveness" survives the fleet mechanism
intact; the store core owns GC state per ADR-005 exactly as
written; ADR-008's mechanism lives in a named, tested seam instead
of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
else" stays true of the public trait.
- The ops pin-token TTL race is resolved with no new ops; the
backstop is already-validated semantics (delete-then-recover).
- The kv tier's fleet-validity rule closes the constructor table
under both tiers.
**Negative**
- A crate-internal companion trait is real machinery: six clause
families on postgres/pg-lo (fleet-active, all contract-tested);
sqlite carries only the window + lock clauses as fleet-relevant
machinery (its pin clauses have unit-test bodies but no fleet mode
on offer — §3). The seam bounds where fleet-GC growth lands; it
does not shrink it.
- Remote putters gain a real protocol duty (renew past 20 s;
pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
- The sqlite sweep-lock clause body exists but is exercised only by
single-node tests until a sqlite-fleet posture is ever
decided (it is not on offer — §3).
**Neutral**
- The `Backend` trait's method set is unchanged from ADR-003/008 —
no public API change beyond what store-api.md already specced; the
cursor/staging abstractions are crate-internal.
- The staged-put contract already existed behaviorally (POC #3 A5);
this ADR names it.
- ADR-009's engine content is unchanged; only its tx-ownership
statement is corrected here.
## References
- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
(tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
liveness-ownership rule this ADR preserves), ADR-006 (the range
contract the cursor serves), ADR-008 (the fleet mechanism whose
state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
(vocabulary + constructor table)
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
`docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
`docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
512 KiB knee, windowed gets)
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
extends); [store-api.md](../store-api.md) (stage hygiene, sweep
semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
(REQ-2, the fleet requirement §2/§3 serve)
docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md
+225
View File
@@ -0,0 +1,225 @@
# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large`
## Status
Accepted
## Context
The engine story (ADR-007/008/009) grew the physical layer faster than
its vocabulary grew, and the pre-decomposition review found the drift
is now self-contradictory in the anchor document itself:
1. **Node vs store instance.** requirements.md defines **Node** as
"one running store instance" — then REQ-2 states "a replicator node
may run multiple store instances (a fleet)" and **Fleet** as "two
or more nodes over one shared pool". Under the written definition,
one process hosting two instances is simultaneously one node (by
definition) and two nodes (by REQ-2), and the GC arbitration
domain ("node is the GC arbitration domain") is undefined for the
exact topology ADR-008 exists for. ADR-010 fixes pin ownership;
the pin's *owner id* and the node/instance distinction it needs
are still unpinned vocabulary.
2. **The "fs" tier name is now wrong.** The tier was named when its
only engine was a filesystem (ADR-003's `local` layout). After
ADR-009, half its engines are postgres Large Objects — the name
describes one engine's medium, not the tier. Worse, "fs" collides
with the dispatch language: "large content goes to fs" never said
*why* — the tier's actual routing criterion is content length
(ADR-003's size-threshold dispatch). The docs already speak the
true name informally: store-api.md says "small tier"/"large tier".
3. **mem's classification drifts.** backends-and-dispatch.md calls it
a "backend" in one place and a "testing/utility tier" in another;
ADR-003 lists it under "shipped backends … plus a testing tier";
ADR-004 calls it an ephemeral tier and carves it out of the
two-backend count. In truth it is neither a tier nor a separate
backend: it implements exactly the kv tier's contract (including
`size`, trivially), it is the contract-reference engine every
other engine's conformance tests mirror, and "engine" names what
it is (ADR-007 established that an engine is a concrete
implementation of one tier's contract — requirements.md already
defines the word this way).
4. **Tier and Backend were never formally defined** even though
ADR-004's scope note ("the 'two' here counts tiers, not engines")
leans on both words. Also undecided: whether the feature names
follow the tier rename now or after code exists — the last free
moment is now (zero code; after implementation the rename breaks
crates.io/feature ergonomics for every consumer).
Deciding facts are in hand: REQ-2's fleet fact, the engine set's
final shape (ADR-007/009), the dispatch mechanism (ADR-003), and
ADR-010's GC-state-home table. This is a vocabulary-and-boundary
decision, not an architecture change: ADR-004's tier≠engine note
already established the shape; this ADR gives it final, consistent
names.
Candidates weighed for the tier rename:
- **`lo`/`los`** — names the *postgres* engine (Large Objects), so
"the lo tier's `local` engine" would be backwards. Rejected.
- **`lfs`** — workable, echoes git-lfs specifically; but the tier
also serves packfiles and any oversized workspace artifact, and the
echo binds the crate's naming to one consumer's protocol shape.
Rejected.
- **`large`** — the dispatch-true name (size-threshold routing), the
word the specs already use informally ("large tier"), medium-agnostic.
**Chosen.** For symmetry in prose, the kv tier stays `kv` (its
routing criterion — "small" — lives in the prose as "small blobs"
and needs no rename: `kv` names the mechanism class, and `small`
as a feature/tier name would read as an adjective; the pair's
asymmetry is cosmetic, the *load-bearing* correction is that
neither tier is named after a medium).
Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it
names the engine's mechanism, accurate); `postgres` keeps its name
(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep
their names.
## Decision
### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy)
- **Tier** — one *contract* (a `Backend` trait implementation
boundary) serving one routing class of the pool. The crate ships
exactly two tiers: **kv** (small content) and **large** (over
the size-threshold content). Tier count (2) is ADR-004's count.
- **Engine** — a concrete storage implementation of one tier's
contract (the existing definition, unchanged). Engine count (5) is
this ADR's count.
- **Backend** — *the* contract (the `Backend` trait) plus its shipped
engine set, i.e. the tier as shipped. "Backend" and "tier" name the
same thing at different altitudes; the crate says "tier" for
counting, "the `Backend` trait" for the contract. "Two backends"
reads forever as "two tiers".
- **Store instance** — one running instantiation of the store facade
(one engine set — one or two engines per the constructor mode, §4 —
plus dispatch and GC state). **The GC
arbitration domain** for single-instance topologies; the pin-row
owner id under fleet engines (ADR-010's `owner`; this corrects
ADR-008's "node id" wording, which predates the instance/node
split — a store instance is what owns pins).
- **Node** — one running *process* (an embedder's binary). A node
hosts **one or more store instances**. Node is the process/ops unit:
alkcall connections, ACL grants, sweep cadence, and deployment
identity attach to nodes; GC arbitration and pin ownership attach
to store instances.
- **Fleet** — **two or more store instances over one shared pool**
(via a shared SQL engine). Deliberately not defined by node count:
two instances in one process over one postgres pool are a fleet
under this definition (and REQ-2's phrasing "a replicator node may
run multiple store instances (a fleet)" reads true as written);
two processes on two sqlite files are two pools, never a fleet.
- **Deployment** — unchanged (one operator's whole system).
### 2. mem joins the kv tier as its third engine
`mem` is the **kv tier's ephemeral engine** (feature `mem`,
default-off): a `BTreeMap`-shaped implementation of the kv contract,
including `size` (trivially) and list; never fleet-valid (ADR-010 §3's
rule and this ADR §4's constructor table); the **contract-reference
engine** — every other engine's
conformance tests mirror its suite. It is not a tier, not a third
backend, and the ADR-004 count ("two") survives untouched because
tier and engine were never the same count — this ADR is that
distinction's final form.
### 3. The fs tier is renamed `large`
In every document, status line, feature name, and constructor
parameter: tier `fs` → tier `large`; feature `fs` → feature `large`
(default-on, as `fs` was); dispatch prose says "large tier"/"small
tier" as store-api.md already does; the `local` engine's name is
kept (its meaning — this node's own filesystem — stays the
constructor's meaning even under the shared-media fleet posture,
where it is explicitly *not* only-node-local; the constructor's
explicit shared-media assertion, ADR-008 §3, is what disambiguates).
### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here)
| Tier | Engine | Feature | Default | Fleet-valid |
|---|---|---|---|---|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
| kv | postgres | `postgres` | no | **yes** |
| kv | mem | `mem` | no | no |
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
| large | pg-lo | `pg-lo` | no | **yes** |
Constructor modes (dispatch shapes) — pinned here (implied by
ADR-008 §3's re-routing bullet and the mem tests, but never named):
- **dual-tier** (the default): kv + large, size-threshold dispatch
between them (ADR-003).
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
constructor mode with no large tier; over-threshold puts are a
constructor-time error (the client-node re-routing posture handles
them via the ops surface instead — its composition, not a store
mode).
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
no dispatch threshold in effect (everything is local and small);
a testing posture, explicitly not a production story.
- Single-tier SQL modes (postgres-only, pg-lo-only) are **not**
constructor modes this crate ships: the two-tier shape is
ADR-004's requirement (both tiers are load-bearing by measured
economics); "postgres-only" as in *one SQL instance serving both
tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
consolidation), which is the dual-tier mode over one engine.
### 5. Deferral-policy wording (README alignment)
The deferral policy's categories are: `deferred(scope)`,
externally-owned questions, and decided-but-sequenced work — three
names, defined in open-questions.md's header. README.md's recap says
"two parking kinds" and is corrected to name all three (the third is
not a *parking* kind — it is work, not a parked question — hence
this ADR's wording here rather than a silent edit).
## Consequences
**Positive**
- The anchor document's self-contradiction (node/fleet) becomes a
three-level vocabulary (store instance / node / deployment) whose
levels map one-to-one onto the mechanisms (GC domain / process-ops
unit / operator whole).
- The fs→large rename removes the last medium-named tier; feature
names, tier names, and dispatch prose now agree, before any code
exists to break.
- mem's classification is stated once, as the contract-reference kv
engine; the "plus a testing tier" carve-outs dissolve.
- The constructor table gives task decomposition a closed set: 5
engine implementations, 3 constructor modes, 1 dispatch policy.
**Negative**
- Vocabulary churn in every doc that says "fs" (the rename is
mechanical but touches all specs and ADR-003/004/007/008/009 status
lines and reference sections). Zero code exists — this is the last
free moment; after Phase 1 starts, feature renames break consumers.
- "Backend"/"tier" as interchangeable words risks reader confusion
with ADR-003's "Backend trait" — mitigated by the rule that the
*trait* is always named with "trait" ("the `Backend` trait"), never
bare "the backend".
**Neutral**
- No engine's contract, feature flag semantics, or constructor
parameter changes; `mem` gains the word "engine" and loses the
word "backend".
- REQ-2's meaning is unchanged (the fleet definition is re-derived
from the same fact, not re-fact'd).
## References
- ADR-003 (the trait contract; its "shipped backends" list is now
read through this ADR's vocabulary), ADR-004 (tier≠engine — the
scope note this ADR gives final names), ADR-007 (kv engines;
constructor selection), ADR-008 (the second tier's engine candidate
and the fleet topology; the vocabulary it assumed is now pinned),
ADR-009 (pg-lo admission; the consolidation option whose
constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
(the I/O seams and GC-state home this vocabulary must fit)
- [requirements.md](../requirements.md) (the canonical vocabulary
copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
(the tier/engine table's spec home)
- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the
reference-impl role mem already played in the POCs);
`poc-pglo-findings.md` (the gate the engine suite mirrors)
docs/architecture/gc-and-namespaces.md
+20 -12
View File
@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (ADR-010 state-home rule + ADR-011 instance/fleet vocabulary)
---
# Pooling, namespaces, and GC
@@ -16,17 +16,23 @@ price, and this document specifies the mechanism the POCs validated
## The pooled CAS
- **One pool per deployment; a node is one store instance** (the
engine/vocabulary definitions live in requirements.md). Repos,
workspaces, and appfile stores are all
- **One pool per deployment; a store instance is one GC arbitration
domain** (the engine/vocabulary definitions live in requirements.md
/ ADR-011: a node — one process — may host several instances; the
instance is where pins and arbitration live for single-instance
topologies). Repos, workspaces, and appfile stores are all
*sets of hash references* over the same CAS — never walled-off
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
object pools) is the same idea done awkwardly; the pool generalizes it
to unrelated repos and to p2p replication. **A fleet (REQ-2) — two or
more nodes sharing one pool — is the one topology where the GC
mechanism below extends beyond a node: ADR-008 pins, sweeper
locking, and delete arbitration move into the shared engine;
the invariants are unchanged.**
to unrelated repos and to p2p replication. **A fleet (REQ-2) — two
or more store instances sharing one pool — is the one topology where
the GC mechanism below extends beyond one instance: ADR-008 pins,
sweeper locking, and delete arbitration move into the shared engine;
the invariants are unchanged. Where this crate's GC state (fleet
rows and locks) physically lives, and who holds the joint
entry+pin transaction, is ruled by ADR-010 §2: the store core owns
all GC state; SQL-backed engines host it through a crate-internal,
contract-tested seam; backends never learn liveness.**
- **One address space.** Under the canonical hash (ADR-002), the same
file content has the same address whether it entered via a git oid or
a workspace manifest — cross-consumer dedup is free by construction,
@@ -46,7 +52,8 @@ price, and this document specifies the mechanism the POCs validated
`register_liveness_source`; the POC's copy-semantics failure finding 7
is the pinning reason). Namespace tables may also exist purely
in-memory for ephemeral consumers; nothing in the store persists
namespaces — where roots live when a deployment runs fs-only is a
namespaces — where roots live when a deployment has no SQL engine (no
kv engine with rows) is a
non-question by design: roots live with the consumer, whose references
are as durable as they need to be.
- ACL mapping for network ops: the namespace is the resource
@@ -112,7 +119,7 @@ existence flips atomically per key).
- **Accounting** (per-namespace sizes): computed from reference sets by
consumers; the store reports pool-level totals only (`list` +
`stat`). Physical-layer accounting would re-introduce namespace
physics — rejected with the per-namespace backend config
physics — rejected with the per-namespace engine config
(ADR-003 §Consequences).
## Design Decisions
@@ -122,7 +129,8 @@ existence flips atomically per key).
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | GC-state home | the store core owns all GC state; SQL engines host it via the contract-tested engine-state seam; pin-tx ownership ruled |
## Open Questions
docs/architecture/hashing-and-keys.md
+1 -1
View File
@@ -17,7 +17,7 @@ residual).
## The hash family
Three tiers (ADR-002 §Decision; resolved from OQ-BL-03 after the
Three hash-policy classes (ADR-002 §Decision; resolved from OQ-BL-03 after the
"multi-hash" premise dissolved — nothing forces BLAKE3 once iroh-blobs'
inheritance is rejected):
docs/architecture/open-questions.md
+5 -4
View File
@@ -27,7 +27,8 @@ not one. Two kinds of parked question remain legitimate:
no decision here; it would be Schrödinger's code if it gated one).
Invalid for either kind: a blocker that is a decision this crate must
make before shipping. Phase 0's residual list was swept under this
make before shipping. (A third category — **decided-but-sequenced
work** — is not a parked kind at all; see below.) Phase 0's residual list was swept under this
rule: every residual either resolved into an ADR (the evidence was
already in hand) or re-owned as below.
@@ -38,7 +39,7 @@ OQ-08). Deferring on a consumer's *deployment of this crate* is
invalid — that fact is created by this crate (OQ-10's original
framing, OQ-09's original framing). And a third thing is neither:
**decided-but-sequenced work**. Once a question resolves to a decision
(the pg-lo engine is the named fs-tier candidate under REQ-2/ADR-008),
(the pg-lo engine is the named large-tier candidate under REQ-2/ADR-008),
what remains is the evidence pipeline the decision's own admission
gate requires (POC #7) — a task with an owner, not a parked question,
and not a hedge. A decision is not "unmade" because its verification
@@ -246,12 +247,12 @@ a deployment trigger — the same invalid shape, caught at review: the
trigger was a fact only this crate could create. It resolved to
ADR-007 (two kv engines; see its entry above).
The pg-lo fs-engine question (raised in the ADR-008 review round,
The pg-lo large-engine question (raised in the ADR-008 review round,
2026-10-03) never became a parked question: REQ-2 made the
multi-instance-with-large-blobs topology a standing requirement
(requirements.md — a planning fact predating all POCs), so there was
nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the
candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
candidate large-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
as its admission evidence — sequenced work, per the deferral-policy
header's third category. That work completed the same day: POC #7
passed (`docs/research/poc-pglo-findings.md`, gate verdict passed
docs/architecture/ops-surface.md
+39 -7
View File
@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (pin-token renewal contract pinned per ADR-010 §4)
---
# Ops surface (alkcall-backed network operations)
@@ -96,8 +96,11 @@ the server. The hand-over contract:
1. the put lands pinned (ADR-005) — the entry cannot be swept while
the handler holds the pin;
2. the `blobs/put` response returns a **pin token** (opaque handle);
the putter can later confirm its registration landed by observing
2. the `blobs/put` response returns a **pin token** (opaque handle —
minted by the ops layer over the pin row the put committed; the
token↔(key, owner) mapping is ops-layer state, not durable state —
the durable state is the pin row itself, ADR-010 §4); the putter
can later confirm its registration landed by observing
the digest present via `blobs/have`;
3. the embedder's registered liveness source is the *actual* root of
record: the contract on the embedder (documented on the ops module)
@@ -112,6 +115,32 @@ the server. The hand-over contract:
conversion on the server side (token → registered liveness source)
is the mechanism the embedder plugs its registry into.
**Token renewal (ADR-010 §4 — the fleet-TTL race, resolved): the
token *is* the pin — there is no second pin state.** The server-side
handler's in-process `Pin` guard drops when the `blobs/put` response
completes (if it never dropped, the TTL machinery would be dead on
the ops path — the pin row's expiry, not the guard's presence, is the
fleet liveness signal). Under fleet
engines the put's pin is an expiry-carrying row (ADR-008: TTL 60 s
default, renewed every TTL/3). The remote putter holding a token is
the row's owner and renews by the ops it already speaks: a
`blobs/have` probe **presenting the token alongside the digest** (the
handler verifies the token maps to the digest's pin row and calls the
crate-internal `pin_renew` via the ops module's store-core access —
the ops module is in-crate and crosses the seam by construction, the
same way the put handler holds the `Pin` guard; `has` itself remains
a pure existence probe per ADR-008 — renewal is a token-authenticated
act on the pin row, not a `has` semantic), or a re-put of the same
digest (CAS dedup renews the row). **A putter may pause past the
TTL/3 mark (20 s at the default TTL) only if it renews before expiry;
a putter pausing past TTL lets the row expire** — the sweeper reaps
it (ADR-008) and delete-then-recover (ADR-005) is the backstop: the
putter re-offers the digest; CAS makes the re-put idempotent. The
race resolves to a re-put, never to silent loss. For non-fleet
(`local`-engine) serving nodes the token is the in-process guard —
nothing expires; the process-crash case is ADR-005's in-flight-put
loss with the same backstop.
Placement of the op *registration* (which embedders wire where they
want them exposed) matches the alkgit ops pattern (its
`git/repo/*` call ops): the module exports spec/handler pairs; the
@@ -143,10 +172,11 @@ handlers call `put`; both honor pins and delete windows. There is no
ops-internal path around the store facade (ADR-001 §Consequences —
this is what makes "re-homing the ops surface later loses nothing"
true). One fleet topology additionally rides this module: **re-routing
(ADR-008)** — a fleet whose nodes do not share fs media routes
(ADR-008)** — a fleet whose nodes do not share large-tier media routes
large-blob access through a single storage node via these ops, with
client nodes' fs tiers serving only node-private content. The module
supports this shape without change; no fleet-specific op exists.
client nodes' constructors in kv-only mode (ADR-011 §4) for pool
content. The module supports this shape without change; no
fleet-specific op exists.
## Design Decisions
@@ -154,7 +184,9 @@ supports this shape without change; no fleet-specific op exists.
|---|---|---|
| [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale |
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Pin-token renewal | the token is the pin; renewal rides have/re-put; TTL expiry falls to delete-then-recover |
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + modes | instance/node/fleet split; kv-only mode named (the re-routing posture's client shape) |
## Open Questions
docs/architecture/overview.md
+47 -36
View File
@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (ADR-010/011 — I/O seams pinned; vocabulary fs→large, mem joins kv)
---
# Overview
@@ -9,10 +9,11 @@ last_updated: 2026-10-03
`alkblobs` is content-addressed blob storage in the alk* family: a
put/get/verify store keyed under one canonical content hash (the
git-family derivation; legacy SHA-1-tolerant), with pluggable backends
(small blobs in a kv store, large blobs on a filesystem fallback),
split from any wire/ops-protocol layer. Network ops ride the alkcall
substrate behind a feature-gated `ops` module (ADR-001).
git-family derivation; legacy SHA-1-tolerant), with pluggable tier
engines (small blobs in a kv store per ADR-007/011, over-threshold
blobs on the `large` tier per ADR-008/009 — the tier formerly named
`fs`), split from any wire/ops-protocol layer. Network ops ride the
alkcall substrate behind a feature-gated `ops` module (ADR-001).
Deployment-relevant consumer facts — what each use case forces on this
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
@@ -63,15 +64,22 @@ or three times.
│ store core (always on) │
│ hashing & keys (ADR-002) · put/get/stat/range/pin seams │
│ mark-and-sweep GC + liveness registration (ADR-005) │
│ size-threshold dispatch (ADR-003) │
│ size-threshold dispatch (ADR-003); GC state owned │
│ here, hosted by SQL engines via the seam (ADR-010) │
└───────┬───────────────────┬───────────────────────────────┘
▼ ▼
│ kv backend fs backend (ADR-003: exactly two tiers;
│ │ ADR-007: kv = 2 engines;
│ ▼ ADR-008: fs engines too)
│ engine per node: sqlite (default) | postgres (feature "postgres")
│ fs engine per node: local (default) | pg-lo (feature "pg-lo",
│ default-off; ADR-009)
┌──────────────────────┐ ┌─────────────────────────────────┐
│ kv tier (ADR-003/ │ │ large tier (ADR-003/008/011; │
│ 007/011) │ │ formerly "fs") │
│ sqlite (default) │ │ local (default) │
│ postgres ("postgres") │ pg-lo ("pg-lo"; ADR-009) │
│ mem ("mem"; tests + │ │
│ contract reference) │ │
└──────────────────────┘ └─────────────────────────────────┘
▲ ▲
└── one Backend trait (ADR-003/010: read cursor +
staged put I/O seams); engine = per-node
constructor choice (ADR-011's table)
transport: none, anywhere. alkgit/alkfs or the embedder dials;
an alkcall Connection is handed in at the ops boundary.
@@ -81,36 +89,37 @@ or three times.
| Layer | Dependencies | Notes |
|---|---|---|
| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
| `pg-lo` feature (ADR-009) | Large Objects as the fs tier's second engine, default-off | rides the same `tokio-postgres`/`deadpool-postgres` stack as the `postgres` kv engine (SQL `lo_*` functions, no new dependency); same admission/test-gate shape |
| base crate (store core + tier engines) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `large`/`mem` | no alkcall, no serialization frameworks |
| `postgres` engine feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
| `pg-lo` engine feature (ADR-009) | Large Objects as the large tier's second engine, default-off | pulls the same `tokio-postgres`/`deadpool-postgres` cluster as the `postgres` kv engine (SQL `lo_*` functions, no additional dependency); **does not imply the `postgres` feature** — engine features pull the driver cluster independently (ADR-010 §1), so a `large`=pg-lo + kv=sqlite node is coherent and supported; same admission/test-gate shape |
| `mem` engine feature (ADR-011) | the kv tier's ephemeral engine, default-off | `BTreeMap`-shaped; the contract-reference engine; no dependencies |
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
Backends are feature-gated; both `kv` and `fs` ship default-on
(batteries-included, the alkgit feature-model pattern — default-on
features, opt-out rather than opt-in); the `kv` tier's postgres engine
(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only
kv node (postgres engine without rusqlite) is **not** a supported
configuration this crate optimizes for — feature-graph work for it is
deferred until a named requirement exists (the base `kv` feature
pulling rusqlite is the accepted v1 shape; convention 8's
lean-base principle applies to *default* builds, which stay lean since
`kv` is what the feature model controls, not what default-on means
downstream). Wasm: the backends are not a wasm story; a wasm client is
a consumer of the `ops` surface (which rides alkcall, itself
wasm-clean), never an in-process embedder (ADR-001 §Consequences).
Tiers are feature-gated; both default-on (batteries-included, the
alkgit feature-model pattern — default-on features, opt-out rather
than opt-in); the engine features `postgres` and `pg-lo`, and `mem`,
are default-off, as are `ops`. A pg-only kv node (postgres engine
without rusqlite) is **not** a supported configuration this crate
optimizes for — feature-graph work for it is deferred until a named
requirement exists (the base `kv` feature pulling rusqlite is the
accepted v1 shape; convention 8's lean-base principle applies to
*default* builds, which stay lean since `kv` is what the feature model
controls, not what default-on means downstream). Wasm: the tier
engines are not a wasm story; a wasm client is a consumer of the `ops`
surface (which rides alkcall, itself wasm-clean), never an in-process
embedder (ADR-001 §Consequences).
## Non-goals (each enforced by an ADR)
- No wire framing, op dispatch, ACL, or transport inside the store layer
(ADR-001).
- No adopted iroh-blobs wire surface — no tickets, no postcard, no
provider protocol (AGENTS convention 7; ADR-001 §Context).
- **No adopted iroh-blobs wire surface — no tickets, no postcard, no
provider protocol (AGENTS convention 7; ADR-001 §Context).**
- No namespaces-as-physical-storage; the pool is flat, namespaces are
consumer-level reference sets (ADR-005).
- No manifest/path-tree/vfs tables; consumers never implement a Backend
to model paths (ADR-004).
- No manifest/path-tree/vfs tables; consumers never implement a
Backend to model paths (ADR-004).
- No chunk-tree/CDC storage encoding; whole-file CAS is the store's only
granularity (ADR-006).
- No per-repo or per-namespace walled stores — one pooled CAS
@@ -126,9 +135,11 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences).
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo fs engine | postgres Large Objects as the fs tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + large tier per node |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; fleet pins/sweeps engine-backed; the large tier (now `large`) engine-selectable |
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo large engine | postgres Large Objects as the large tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; kv fleet-validity rule; pin-token renewal |
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + rename | tier/engine/instance/node/fleet pinned; mem joins the kv tier; large tier renamed `large`; constructor modes |
## Open Questions
@@ -144,7 +155,7 @@ crate level:
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
shipped behind one trait; per-node constructor choice)
- **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects
as the fs tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
as the large tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
a shipped engine by ADR-009 (REQ-2's consolidation option)
## References
docs/architecture/requirements.md
+40 -20
View File
@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-10-03 (pg-lo admitted by ADR-009 on POC #7 — REQ-2's consolidation option is shipped)
last_updated: 2026-10-03 (vocabulary pinned by ADR-011 — tiers/engines/instances/fleet; large tier renamed `large`)
---
# Requirements
@@ -12,24 +12,44 @@ conversation or vision prose. **This document is the anchor: when a
consumer fact changes, it changes here first, with a dated note, and
the ADRs re-derive from it.**
## Vocabulary (defined once; used everywhere)
## Vocabulary (defined once; used everywhere — ADR-011 is the decision behind this section)
- **Tier** — one contract (a `Backend` trait implementation boundary)
serving one routing class of the pool. Exactly two tiers: **kv**
(small content) and **large** (over the size-threshold content;
the tier formerly named `fs` — renamed in ADR-011 because the tier's
routing criterion is length, and half its engines are not
filesystems). "Backend" names the same thing at a different
altitude (the tier as shipped including its engine set); "two
backends" (ADR-004) reads forever as "two tiers".
- **Engine** — a concrete storage implementation of one tier's
contract: the kv tier's engines are sqlite and postgres (ADR-007);
the fs tier's engines are `local` (default) and `pg-lo` (ADR-009).
contract. Five ship: the kv tier's `sqlite` (default),
`postgres` (feature `postgres`), and `mem` (ephemeral; feature
`mem`; the contract-reference engine; ADR-003); the large tier's
`local` (default; ADR-003) and `pg-lo` (ADR-009).
- **Pool** — the flat content-addressed namespace of one deployment
(`hash → bytes` everywhere; ADR-005).
- **Node** — one running store instance (one process embedder; its fs
tier is its own media). Node is the GC arbitration domain for
single-node topologies (ADR-005's pins, liveness locks, and delete
windows live inside a node); under a fleet, arbitration moves into
the shared engine (ADR-008).
- **Fleet** — two or more nodes over **one shared pool** (e.g. via one
postgres instance). A fleet is the only topology in which
cross-node GC and fs-locality questions exist (ADR-008).
- **Store instance** — one running instantiation of the store facade
(one engine set — one or two engines per the constructor mode,
ADR-011 §4 — plus dispatch and GC state). The **GC arbitration
domain** for single-instance topologies (ADR-005's pins, liveness
locks, and delete windows live inside one); the pin-row owner id
under fleet engines (ADR-008/010 — ADR-008 wrote "node id" before
this split existed; a store instance is what owns pins).
- **Node** — one running *process* (an embedder's binary). **A node
hosts one or more store instances** — this fixes the prior
definition's collapse (which REQ-2's phrasing contradicted). Node
is the process/ops unit: alkcall connections, ACL grants, sweep
cadence, and deployment identity attach to nodes.
- **Fleet** — **two or more store instances over one shared pool**
(via a shared SQL engine). Defined by pool-sharing, not node count:
two instances in one process over one postgres pool are a fleet;
two processes on two sqlite files are two pools, never a fleet. A
fleet is the only topology in which cross-instance GC and
large-tier locality questions exist (ADR-008).
- **Deployment** — one operator's whole system: typically one fleet
(multi-node) or one node; consumer tables, op registrations, and
sweep cadence are deployment choices.
or one node; consumer tables, op registrations, and sweep cadence
are deployment choices.
## The use cases and their requirements
@@ -37,7 +57,7 @@ the ADRs re-derive from it.**
Git repos for a user/team on one machine. Small-object regime dominated
by loose objects in the kv tier; packs and large artifacts (git-lfs
shape) on the fs tier.
shape) on the large tier.
- **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics:
sqlite is the default kv engine; single-writer WAL ceiling (~1.2k
@@ -59,9 +79,9 @@ replicators; ACL/ownership above (alkcall layer).
discussion; it predates all POCs and architecture docs, recorded
here on the operator's authority as the use case's stated shape).
Consequences are derived in ADR-008: fleet GC (DB-backed pins,
advisory-locked sweeper, SQL delete arbitration), the fs-tier
engine concept with its locality contract, and the pg-lo candidate
engine (POC #7 — **passed** 2026-10-03, admitted as a shipped
advisory-locked sweeper, SQL delete arbitration), the large-tier
engine concept with its locality contract, and the pg-lo engine
(POC #7 — **passed** 2026-10-03, admitted as a shipped
engine by ADR-009: a fleet node can consolidate on postgres-only).
- **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out:
the fleet's hub-class nodes need put throughput that *increases*
@@ -79,7 +99,7 @@ path→hash manifests above.
- **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space
across workspaces; dedup must be structural (not configured). Small
edits mean the small-blob regime dominates: sqlite-class kv
performance is what matters; the fs tier is hit rarely (large
performance is what matters; the large tier is hit rarely (large
workspace artifacts).
## What this document is not
@@ -100,4 +120,4 @@ path→hash manifests above.
| REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 |
| REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 |
| REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 |
| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-005 |
| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-002, ADR-005 |
docs/architecture/store-api.md
+59 -29
View File
@@ -1,19 +1,20 @@
---
status: draft
last_updated: 2026-10-03
last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — large tier → large)
---
# Store API
## What this is
The store core's public surface: the typed facade above the backends
that consumers (alkgit, alkfs, the ops module) program against. It owns
typed keys and hashing (ADR-002), size-threshold dispatch (ADR-003),
the put-pinning seam, and the liveness/sweep seam (ADR-005) — and
nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). The
facade is **engine-blind** (ADR-007/008): which kv/fs engine a node
runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
The store core's public surface: the typed facade above the tier
engines that consumers (alkgit, alkfs, the ops module) program against.
It owns typed keys and hashing (ADR-002), size-threshold dispatch
(ADR-003), the put-pinning seam, and the liveness/sweep seam (ADR-005)
— and nothing else: no paths, no manifests, no wire (ADR-004, ADR-001).
The facade is **engine-blind** (ADR-007/008/010): which engines a node
runs changes nothing here except fleet-GC mechanism wiring (ADR-008's
mechanism lives in the engine-state seam, ADR-010 §2).
## Public surface
@@ -29,13 +30,17 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
threshold; on commit the buffer is the whole stored artifact
(kv tier) — or, on mid-stream threshold overflow, the buffer
flushes into a stage file and the stream continues into it
(fs tier — the streaming pass cannot include the preamble, so the
derivation pass restarts over preamble + staged bytes; forced by
the preamble being inside the hashed input, not a design choice).
Verification runs over the eventual stored artifact as a whole;
a stream error on either path leaves no entry and no stage file.
Eliminates the A6 dispatch asymmetry by construction: routing is
a pure function of content length (ADR-003).
(large tier — the streaming pass cannot include the preamble, so
the derivation pass restarts over preamble + staged bytes; forced
by the preamble being inside the hashed input, not a design
choice). Verification runs over the eventual stored artifact as a
whole; a stream error on either path leaves no entry and no stage
file. Eliminates the A6 dispatch asymmetry by construction:
routing is a pure function of content length (ADR-003). The large
tier's stage mechanics are the **staged-put contract** (ADR-010
§1): the engine hands out a staging cursor (`local`: the stage file;
pg-lo: chunked `lowrite` statements inside its tx), commit
publishes, every failure path discards.
- **Stage-file hygiene invariant** (POC #3 finding A5, codified): every
failure path converges on stage-discard; every success path converges
on commit-rename. No early returns that bypass cleanup; this is an
@@ -51,7 +56,10 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
### Read path
- **`get(key) → Option<(len, stream)>`** — small tier returns a
bounded read; large tier streams a file handle. Dispatch
bounded read; large tier streams (`local`: a pread loop; pg-lo: `lo_get`
windows). The per-engine shapes are hidden behind the crate-internal
read cursor the store core drives (ADR-010 §1 — engines construct
their own cursor bodies; the facade's shape is uniform). Dispatch
fall-through: a small-tier miss queries the large tier (POC #1
finding 6).
- **`stat(key) → Option<EntryMeta>`** — cheap length/type probe
@@ -60,8 +68,9 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
carries `len: u64` and the key's algorithm (the "type" from the
store's perspective is the hash algorithm — there is no other type at
this layer). `stat` rides the `Backend` trait's `size` method
(ADR-008): an indexed-column probe on kv engines, a file stat on the
fs `local` engine — never a value/content load.
(ADR-008): an indexed-column probe on kv engines, a file stat on
`local`, a companion-row probe on pg-lo — never a value/content
load.
- **`read_range(key, range) → (slice, slice_digest)`** — slice plus an
out-of-band slice digest: SHA-256 over the slice bytes, fixed by
convention so callers and ops handlers agree without negotiation
@@ -89,12 +98,26 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
(finding 7) is why the verb/name is pinned here).
- **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own
the cadence (ADR-005, no ambient timers). A sweep with no registered
liveness sources aborts without deleting — the safe default. Under
a fleet (multiple nodes over one pool — requirements.md REQ-2),
sweep/put/delete semantics are mechanism-extended by ADR-008: pins
are DB-backed rows committed atomically with entries, one sweeper
holds the advisory lock, and delete arbitration is SQL-level — the
facade's invariant text is unchanged.
liveness sources aborts without deleting — the safe default.
`SweepReport`'s shape (specced here, resolving its earlier
name-only status): `candidates_staged`, `deleted`,
`cancelled_by_arbitration` (u64 counts), `aborted: Option<GcAbortCause>`
(the typed cause — protection-source failure (ADR-005) or fleet
sweeper-lock contention (ADR-008) — present iff the sweep deleted
nothing for an abort reason; named `GcAbortCause` to keep it a data
enum, distinct from the `GcAborted` error which carries the
direct-delete refusal path), and `started_at`/`finished_at`. The
report is data, not an error carrier: an aborted sweep returns
`Ok(report)` with `aborted: Some(..)` — the typed error
`GcAborted` is reserved for the direct-delete refusal path.
Under a fleet (multiple store instances over one pool —
requirements.md REQ-2), sweep/put/delete semantics are
mechanism-extended by ADR-008: pins are DB-backed rows committed
atomically with entries (the store core holds the joint tx,
ADR-010 §2), one sweeper holds the advisory lock, and delete
arbitration is SQL-level — the facade's invariant text is
unchanged.
## Error model
@@ -129,8 +152,8 @@ lesson generalizes to any kv engine).
1. Whole-blob put/get round-trips byte-identical under the canonical
derivation, both tiers, both put paths, **and every shipped engine
of both tiers** (kv: sqlite + postgres; fs: `local` + pg-lo;
ADR-007/008/009)
of both tiers** (kv: sqlite + postgres + mem; large: `local` +
pg-lo; ADR-007/008/009/011)
(interop with real git remains the source of truth, per POC #1's
lesson about hardcoded vectors).
2. Dedup: putting identical content twice (same or different path) is
@@ -148,8 +171,14 @@ lesson generalizes to any kv engine).
tests assert through sweep outcomes (POC #1 finding 2's lesson
codified).
6. `size`/`stat` on any engine returns the entry length without
fetching content (asserted per-backend: kv probe, fs stat, pg-lo
companion-row probe — ADR-008/009).
fetching content (asserted per engine: kv row probe, `local` stat,
pg-lo companion-row probe — ADR-008/009).
7. **Engine-conformance uniformity (ADR-010 §1):** every engine's
cursor/stage/list bodies pass the same contract suite mem
(the reference engine) passes — exact-count sweep outcomes, stage
hygiene, virgin-store no-ops, CAS re-put, ranged reads (the POC #7
gate is the norm; per-engine suite runs in CI, `--all-features` for
the feature-gated engines).
## Design Decisions
@@ -159,7 +188,8 @@ lesson generalizes to any kv engine).
| [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering |
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics |
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band |
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed |
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed |
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put behind the facade; the store core holds the joint entry+pin tx |
## Open Questions