docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round

ADR-010 (new): the Backend trait's I/O seams pinned before any code —
'get' returns a crate-internal read cursor (kv engines materialize;
'local' pread loop; pg-lo lo_get windows), 'put' has two named forms
(whole-value kv / staged-put large); GC state is store-core-owned via
a crate-internal contract-tested engine-state seam on the SQL-backed
engines (backends never learn liveness — ADR-005 verbatim); the store
core holds the joint entry+pin transaction (corrects ADR-009 §2's
ownership statement); kv-tier fleet-validity rule added (mirror of
ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin;
renewal rides have/re-put; pause-past-TTL falls to delete-then-recover);
feature graph pinned (pg-lo does not imply postgres).

ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs
engine (concrete impl, exactly five); store instance / node / fleet
split (fleet = pool-sharing, not node count — fixes requirements.md's
self-contradiction with REQ-2); mem demoted from 'backend'/'testing
tier' to the kv tier's third engine (contract-reference engine);
fs tier renamed 'large' including the feature name (last free moment
before code exists); constructor modes pinned (dual-tier default,
kv-only, mem-only) plus the composite fleet-validity predicate.

Consistency round across all specs and ADR-003/004/007/008/009: stale
pg-lo passages resolved per ADR-009's supersession; ADR-008 owner
'node id' → store-instance id; threshold wording corrected (conservative
edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport
shape specced; mem classification unified; 008/009 ADR files renamed to
match the tier rename; README deferral-policy recap aligned (third
category = decided-but-sequenced work, not a parking kind); POC crate
list completed.

Verification: two independent architecture review rounds (the first
found 4 criticals — unpinned trait I/O shapes, missing fleet-state
seam, node/fleet self-contradiction, pin-token/TTL conflict — all
resolved; final round: zero criticals); all markdown links resolve;
ADR tables complete (11 ADRs).
This commit is contained in:
glm-5.3-flash committed 2026-10-03 05:55:20 +00:00
1 parent 9bbe6f3ad9
commit abf8c5ae9c
18 files changed
+1137 -336

No files matched your search

@@ -16,7 +16,7 @@ addresses, not algorithms).
The residual was the **key encoding mechanics** — the byte layout of
keys inside backends. This is the crate's declared one-way door
(AGENTS convention 5): key bytes persist in kv rows and fs filenames,
(AGENTS convention 5): key bytes persist in kv rows and `local`-engine filenames,
with no schema-migration story; and this is the wire-format ADR that
must exist before the first consumer (convention 7's gate on publishing
wire-relevant decisions). POC #1 validated the mechanics empirically:
@@ -26,7 +26,7 @@ both algorithms in one flat pool (finding 5).
## Decision
### Hash policy (three tiers)
### Hash policy (three classes)
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
SHA-256. The store's canonical algorithm; git objects are first-class
@@ -5,10 +5,15 @@
Accepted (engine pin amended by ADR-007 — the kv tier ships two
engines, sqlite and postgres, behind this contract; the dispatch layer
and trait are unchanged. The trait was extended by ADR-008 with a
`size` length probe — before any backend ships data — and the fs
tier became engine-selectable (`local` default; pg-lo candidate), so
this ADR's "engines"/"backends" counts are read through ADR-004's
tier≠engine scope note)
`size` length probe — before any backend ships data — and the second
tier became engine-selectable (`local` default; pg-lo admitted by
ADR-009), so this ADR's "engines"/"backends" counts are read through
ADR-004's tier≠engine scope note. The I/O shapes behind `get`/`put`
are pinned by ADR-010 (read cursor / staged put). The tier's name
changed: `fs` → `large` (ADR-011 — its routing criterion is length,
not medium); this ADR's per-engine counts also gained the `mem` kv
engine as ADR-011's vocabulary resolves — mem is a kv engine, not a
tier)
## Context
@@ -54,28 +59,34 @@ Evidence in hand (no further POC or consumer-waiting needed):
- **Virgin-store reads are no-ops** (missing table = absent/empty).
- **Namespace-blind** (ADR-005): backends see hashes only.
### Shipped backends (two, default-on; plus a testing tier)
### Shipped tiers (two, default-on; the kv tier also carries the mem engine)
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
the regime holding most git objects, workspace files, and manifests.
WAL + `synchronous=NORMAL` as shipped durability; prepared
statements; no-tables-yet tolerance. (The tier's engine set was
widened to include postgres by ADR-007.)
- **`fs` (default-on)** for large blobs — flat
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
widened to include postgres by ADR-007, and mem — the ephemeral
reference engine — by ADR-011's vocabulary resolution.)
- **`fs` (default-on; renamed `large` by ADR-011)** for large blobs —
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
conclusion); stage-then-commit-rename; pread range reads (sound
locally per ADR-006).
- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and
in-process ephemerality. Never a production story.
locally per ADR-006). (The tier's engine set was widened by ADR-008/
009.)
- **`mem` (default-off):** ephemeral `BTreeMap` kv engine for tests
and in-process ephemerality — **an engine of the kv tier, not a
third tier** (ADR-011 §2; the pre-ADR-011 "testing tier" phrasing
is superseded). Never a production story; the contract-reference
engine.
Both production backends are default-on (batteries-included, the
Both production tiers are default-on (batteries-included, the
alkgit feature-model precedent); `mem` is not.
### Dispatch policy
- **Size-threshold routing; default 128 KiB, constructor-tunable**
(midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4).
(chosen at the bottom of the measured ~128-256 KiB crossover zone —
the conservative edge of the flat zone; POC #3 A4).
- **Unknown-length puts buffer in memory up to the threshold; the
buffer's fate has three pinned semantics** (finding A6's option (d)):
1. **Threshold exceeded mid-stream** (non-restartable sources are
@@ -141,9 +152,10 @@ alkgit feature-model precedent); `mem` is not.
**Neutral**
- A network-backed backend (S3-like) remains possible via the same
- A network-backed engine (S3-like) remains possible via the same
trait but ships nothing here; its list/GC obligations are the
acceptance gate.
acceptance gate; per-tier, via a new ADR (ADR-011's vocabulary: an
engine addition, not a tier addition).
## References
@@ -2,7 +2,12 @@
## Status
Accepted
Accepted (the count's vocabulary is pinned by ADR-011: the "two" here
are *tiers* — kv + large, the latter renamed from `fs`; "backend"
names the same thing at a different altitude. The `mem` item below is
the kv tier's third engine, per ADR-011 §2, not an extra tier —
the pre-ADR-011 "testing tier" carve-out is superseded. The I/O
seam-level refinements of the trait contract are ADR-010's)
## Context
@@ -11,9 +16,10 @@ rounds before being edited away, but the correction it was reaching for
arrived only in the Phase 1 review, and needs recording so no future
round reabsorbs the concern incorrectly:
- There are **always** two physical backends in this problem: storage
for small content (fast, kv/sqlite) and the filesystem for large
content. This is not ugly, it is the measured shape of the problem
- There are **always** two physical tiers in this problem: storage
for small content (fast, kv/sqlite) and storage for large content
(the large tier / filesystem). This is not ugly, it is the measured
shape of the problem
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
- The "two backends is ugly" sentiment was actually about a **third
layer** in the old alknet-filesystem research — the *virtual
@@ -30,23 +36,25 @@ is the boundary it will build against.
## Decision
**This crate ships exactly two production backends and nothing more;
**This crate ships exactly two production tiers and nothing more;
every manifest/namespace/path-tree layer is a consumer, by construction
and by contract.** (A non-production ephemeral `mem` tier exists for
tests — ADR-003; it is not a storage story and does not extend this
boundary.)
and by contract.** (A non-production ephemeral `mem` engine exists for
tests — ADR-003, resolved to a kv-tier engine by ADR-011 §2; it is not
a storage story and does not extend this boundary.)
**Scope note (post-ADR-007 reconciliation): the "two" here counts
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
engines behind it to a set (sqlite default + postgres, feature-gated)
so a node's kv rides its relational engine of choice. The count that
matters to this ADR — trait impls *this crate ships* as CAS tiers —
remains two (kv + fs); a second kv engine does not make a third
backend, and nothing in the engine set touches the manifest-layer
boundary below.
**Scope note (post-ADR-007 reconciliation, vocabulary finalized by
ADR-011): the "two" here counts *tiers*, not engines.** The kv tier is
one contract; ADR-007 widened the engines behind it to a set (sqlite
default + postgres, feature-gated; mem the ephemeral reference engine,
ADR-011) so a node's kv rides its relational engine of choice. The
count that matters to this ADR — trait impls *this crate ships* as CAS
tiers — remains two (kv + large, the tier ADR-011 renamed from `fs`);
a second kv engine does not make a third tier, and nothing in the
engine set touches the manifest-layer boundary below.
1. **Two production backends is the complete set.** kv/sqlite (small)
+ fs (large) cover the scale economics with measured evidence
1. **Two production tiers is the complete set.** kv/sqlite (small)
+ large (the tier formerly named `fs`) cover the scale economics
with measured evidence
(ADR-003); no other physical tier is needed by any current or
planned consumer's *blob* needs.
2. **The "third backend" is not this crate's layer.** The vfs/appfile
@@ -86,8 +94,8 @@ boundary below.
**Positive**
- The crate's scope stays testable and small: two backends, one pool,
one hash domain. No schema/versioning surface for manifest formats —
- The crate's scope stays testable and small: two tiers, one pool,
one hash domain. No schema/versioning surface for manifest formats —
consumers iterate on theirs freely.
- alkgit and alkfs build their (very different) mapping layers on one
shared pool without either bending the store; their manifests interop
@@ -49,7 +49,7 @@ served by range reads).
(git refs, alkfs path trees) and hand liveness over per sweep via the
live-shared seam. In-memory namespace tables remain available for
ephemeral consumers. This also dissolves the hidden wrinkle of a
store-internal root table under fs-only deployments (where would its
store-internal root table under deployments without a SQL engine (where would its
durable bytes live?). Ephemeral consumers are covered by pins; a
deployment that needs store-durable roots registers them as
consumer-owned durable state and sweeps through the seam.
@@ -2,7 +2,9 @@
## Status
Accepted
Accepted (the engine story's I/O seams and GC-state home are ruled by
ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as
its third engine — is pinned by ADR-011)
## Context
@@ -57,9 +59,9 @@ in OQ-09.
**The kv tier ships two engines — sqlite (default) and postgres —
behind one `Backend` trait; engine selection is a constructor
parameter on the backend's configuration.** A node provisions one
relational engine plus the fs tier; the kv tier rides whichever
relational engine plus the large tier; the kv tier rides whichever
engine the operator chose. This collapses the classical three-engine
downstream stack (kv + fs + relational) to two, and makes which
downstream stack (kv + large content + relational) to two, and makes which
relational engine a per-node choice rather than a crate fork.
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
@@ -83,7 +85,7 @@ relational engine a per-node choice rather than a crate fork.
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
virgin-store no-op reads, namespace-blindness. The dispatch layer
(size threshold, kv↔fs routing, fall-through) is engine-blind and
(size threshold, kv↔large routing, fall-through) is engine-blind and
unchanged.
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
as requirements** (they are the difference between a port and an
@@ -127,7 +129,7 @@ collapses to its own deployment choice among shipped engines.
**Positive**
- The classical downstream stack lands at two engines per node (one
relational + fs), eliminating the "sqlite *beside* the real
relational engine + large tier), eliminating the "sqlite *beside* the real
database" duplication this crate's consumers would otherwise carry
forever.
- Per-node engine choice (constructor parameter, per ADR-003/004)
@@ -1,10 +1,14 @@
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
# ADR-008: Trait length probe, fleet GC, and large-tier engines
## Status
Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
passed gate; every other part of this ADR stands as written)
passed gate — the two §Consequences passages that said pg-lo "remains
unbuilt" are resolved by that same supersession; the tier this ADR's
title and text call "fs" is renamed `large` by ADR-011; the fleet GC
mechanism's state home and the joint-tx ownership are ruled by
ADR-010; every other part of this ADR stands as written)
## Context
@@ -18,7 +22,7 @@ sharing one pool via postgres; requirements.md):
probe"; store-api.md) and pool accounting rides `list` + `stat`
(ADR-005), but the trait's methods are `has/get/put/delete/list/
name` with no value-metadata access. The kv engines' row shapes
both carry a `size` column (POC #5 B5), and the fs tier can stat a
both carry a `size` column (POC #5 B5), and the large tier can stat a
file — but with no trait method, kv `stat` degenerates to a full
value load. The trait is the crate's second stable seam (ADR-003);
discovering this during implementation would be a breaking trait
@@ -32,8 +36,8 @@ sharing one pool via postgres; requirements.md):
node B's in-flight put is invisible. Data loss, structurally: the
topology REQ-2 requires does not satisfy the protocol ADR-005
specifies.
3. **The fs tier is node-local, silently partitioning a fleet.** The
fs backend stores to a sharded layout on the node's own filesystem
3. **The large tier is node-local, silently partitioning a fleet.** The
large tier stores to a sharded layout on the node's own filesystem
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
put via node A exists only on A's disk; node B's get fall-through
queries its own fs and misses, and `list()` unions disagree per
@@ -41,7 +45,7 @@ sharing one pool via postgres; requirements.md):
hypothetical.
These three travel together: the fleet topology forces #2 and #3, and
#3's resolution constrains what the fs tier must support, while #1 is
#3's resolution constrains what the large tier must support, while #1 is
a trait-level gap any topology hits.
## Decision
@@ -55,7 +59,7 @@ tier:
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
never a value load. The `size` column both POC row shapes carry
becomes contract-relevant.
- **fs engine (`local`)**: stat/metadata read, no content fetch.
- **large engine (`local`)**: stat/metadata read, no content fetch.
- The store facade's `stat` (store-api.md) and the accounting path
(ADR-005) ride this method; the `has` method remains a pure
existence probe.
@@ -71,18 +75,20 @@ never deleted by a sweep started before it committed** (ADR-005).
Under a fleet, the mechanism moves into the shared engine:
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
inserts in one transaction on the fleet engine: the entry row and
its pin row commit together (or neither does). The pin row carries
an owner (node id) and expiry: the pin guard renews its row every
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
parameters, not API). **Protection = an un-expired pin row**: at
delete-time arbitration, only `expiry > now()` pins count; an
expired pin is abandoned (its holder failed to renew — the same
liveness-failure signal as an in-process guard's drop) and is reaped
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
remains the correctness backstop — content is re-put byte-identically
under the same key.
inserts in one transaction on the fleet engine: the entry row and
its pin row commit together (or neither does). The pin row carries
an owner (a store-instance id — ADR-011's vocabulary; "node id" is
this ADR's original wording, superseded because a node may host
several instances) and expiry: the pin guard renews its row every
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
parameters, not API). **Protection = an un-expired pin row**: at
delete-time arbitration, only `expiry > now()` pins count; an
expired pin is abandoned (its holder failed to renew — the same
liveness-failure signal as an in-process guard's drop) and is reaped
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
remains the correctness backstop — content is re-put byte-identically
under the same key.
- **Registered liveness sources have one fleet form: a liveness
table.** A fleet embedder registers its liveness as a named table of
`(key, …)` rows in the shared engine (schema pinned by the embedder,
@@ -131,9 +137,9 @@ Under a fleet, the mechanism moves into the shared engine:
is shared machinery. The pin-row expiry/ttl machinery exists only
where fleet engines are active.
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible
The fs tier is a tier, not a medium. It ships engines the way the kv
The large tier is a tier, not a medium. It ships engines the way the kv
tier does (ADR-007), and a deployment's topology determines which are
valid:
@@ -141,7 +147,7 @@ valid:
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
REQ-4) and for fleet nodes using **shared media** (below).
- **Fleet + shared media (deployment requirement):** multi-node pools
may keep the `local` engine by pointing every node's fs tier at
may keep the `local` engine by pointing every node's large tier at
shared/replicated media. The soundness claim is *qualified*, not
assumed: immutable-once-committed files plus stage-then-commit-
rename are sound on media that provides (a) atomic rename within a
@@ -158,43 +164,52 @@ valid:
not share media designates **one storage node**; large-blob writes
and reads over the pool route through it via the ops surface
(verified fetch/put; ADR-001). Client nodes' constructors are
configured with a **kv-tier-only dispatch** (or an fs tier reserved
for node-private, consumer-owned content — which is outside the
pool and outside this crate's GC; it is the consumer's own storage
beside the pool per ADR-004, not pool content). The facade shape on
configured with **kv-only mode** (ADR-011 §4's name for what this
bullet first called "kv-tier-only dispatch") — no local large tier
over pool content at all; a client node's *node-private,
consumer-owned* large-tier content is outside the pool and outside
this crate's GC entirely: it is the consumer's own storage beside
the pool per ADR-004, not a constructor mode over pool content, and
the composite fleet-validity rule (backends-and-dispatch.md)
therefore has no "local on node-private media over one pool" entry.
The facade shape on
a client node is its own store for sub-threshold content plus its
registered ops-put path for large content — a consumer-side
composition, not a store-level mode.
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
as the fs tier's storage — the same "engine behind one contract"
move as ADR-007, one tier over. Named here so the door is explicit,
with its admission gated on measured evidence (POC #7: LO write/read
- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects
as the large tier's storage —
the same "engine behind one contract"
move as ADR-007, one tier over. Originally named as a candidate,
admission gated on measured evidence (POC #7: LO write/read
curves at the packfile regime, tx-scoped handle cost under
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
behavior) and its own engine ADR. **Not shipped speculatively;**
behavior) and its own engine ADR. NOT shipped speculatively;
REQ-2's immediate fleet answers are shared media or re-routing;
pg-lo is the consolidation option if those are unacceptable. *Update
2026-10-03: POC #7 ran and passed its gate —
`docs/research/poc-pglo-findings.md` (companion-table authority,
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
behind page-cache fs by a measured 20-50x aggregate-multiplied to
~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture).
The engine ADR's remaining evidence path is open; admission remains
gated on it, not on speculation.*
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s
aggregate at 16 readers). Superseded the same day: ADR-009 admitted
the engine — this bullet's original "candidate, not shipped"
posture is historical; the engine is shipped, feature `pg-lo`.*
- **The no-mixed-large-tiers rule is a deployment invariant, enforced at
the enforceable seam.** Cross-node configuration cannot be validated
by any one constructor (it sees only its own node). What the
constructor *can* enforce: a fleet-mode constructor (fleet engines
active) requires an explicit fs-tier declaration — `local` (with the
active) requires an explicit large-tier declaration — `local` (with the
operator's assertion that its root is fleet-shared media), routed
(no local fs over pool content), or — when shipped — `pg-lo`. The
(kv-only mode, ADR-011 §4; no local filesystem over pool content), or
`pg-lo`. The
cross-node truth ("is node B's `local` actually the same media as
node A's?") is a deployment invariant this crate documents and ops
verifies, not something a constructor can prove; a partitioned
tier's observable signature (get fall-through misses for entries
another node wrote) is the documented detection symptom.
another node wrote) is the documented detection symptom. The
composite fleet-validity predicate over both tiers is pinned in
backends-and-dispatch.md ("The composite fleet-validity rule").
In sum: a fleet's fs tier over pool content is either shared media (all
In sum: a fleet's large tier over pool content is either shared media (all
nodes' `local` roots on the same fleet-shared media) or routed (one
storage node); mixing per-node-local media over one pool is the
partitioning failure this decision exists to prevent.
@@ -211,9 +226,10 @@ partitioning failure this decision exists to prevent.
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
invariant extends to fleets as a *stated* mechanism, not an
accident of single-process luck.
- The fs tier's fleet problem has named, shipped-code-free answers
(shared media, re-routing) and a named consolidation candidate
(pg-lo) whose admission path is the same POC-first door as ADR-007.
- The large tier's fleet problem has named answers: shipped-code-free
for shared media and re-routing, shipped-code for the consolidation
option (pg-lo — admitted by ADR-009, the same POC-first door as
ADR-007).
**Negative**
@@ -223,9 +239,9 @@ partitioning failure this decision exists to prevent.
surface for orphaned pins at node crash — the expiry is the cleanup
handle), advisory-lock coordination, and SQL-level arbitration
added on top of the already-intricate delete-window state machine.
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
re-route pays for its ADR + POC before shipping large blobs — the
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
- `pg-lo` was built under ADR-009 after POC #7 passed — the
measured, deliberate sequence of the ADR-006 pattern (admission
gate first, engine second), not a hedge; resolved same-day.
**Neutral**
@@ -245,13 +261,16 @@ partitioning failure this decision exists to prevent.
columns; the driver stack pg-lo would ride)
- ADR-002 (the one-way-door discipline this trait change applies),
ADR-003 (the trait contract and admission gate `size` and pg-lo
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
amend/enter), ADR-004 (tier≠engine, extended to the large tier),
ADR-005 (the GC invariant and single-node protocol extended here,
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
follows), ADR-007 (the engine-behind-one-trait precedent)
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
and tier surface this amends);
[store-api.md](../store-api.md) (`stat`'s contract);
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
topology's transport)
and tier surface this amends);
[store-api.md](../store-api.md) (`stat`'s contract);
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
topology's transport); ADR-010 (the fleet mechanism's state home
and the joint-tx ownership this ADR's §2 implied but did not rule);
ADR-011 (the vocabulary — instance/fleet — this ADR's topology
language resolves to; the large tier renamed `large`)
@@ -1,14 +1,17 @@
# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine
# ADR-009: pg-lo — postgres Large Objects as the large tier's second engine
## Status
Accepted (admission evidence: POC #7, passed 2026-10-03 —
`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
gate)
gate. The tier this ADR calls "fs" is renamed `large` by ADR-011);
§2's "put = one transaction" tx-ownership statement is corrected by
ADR-010 §2 (the store core holds the tx; the engine contributes its
staged-put mechanics into it — engine content unchanged))
## Context
ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's
ADR-008 named `pg-lo` as the candidate large-tier engine for REQ-2's
fleet topology (multiple store instances sharing one pg pool, serving
large/beyond-threshold pool content) — expressly gated on the ADR-003
admission door: measured evidence first, in the ADR-007 shape. POC #7
@@ -38,7 +41,7 @@ The gate's three legs all passed:
## Decision
**The fs tier ships `pg-lo` as its second engine — postgres Large
**The large tier ships `pg-lo` as its second engine — postgres Large
Objects — behind the same `Backend` trait, feature `pg-lo` (default-
off), constructor-selected per node.** REQ-2's deployment choice set
(ADR-008) extends from {shared media, re-routing} to include the
@@ -58,7 +61,7 @@ consolidation option as a shipped engine.
into the LO and commit the row on completion; rollback discards
both.
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
the fs-tier handle closes over the companion row (oid + length) —
the large-tier handle closes over the companion row (oid + length) —
immutable, so the row is authoritative — not over a tx-scoped
descriptor. Ranged reads are single stateless statements,
pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
@@ -82,7 +85,7 @@ consolidation option as a shipped engine.
(committed LOs without companion rows — mixed/legacy stores). The
crash story is structurally clean: LO lifecycle is transactional
in modern postgres; kill mid-tx leaves zero pages, zero
half-commits (cleaner than fs `local`'s stage-file residue,
half-commits (cleaner than `local`'s stage-file residue,
which needs its own recovery sweep).
6. **Performance deltas, as named capacity facts (not hidden, not
solved):** single-stream cached get runs 20–50× behind page-cache
@@ -105,13 +108,13 @@ consolidation option as a shipped engine.
**Positive**
- REQ-2's consolidation option is real and shipped: a fleet node runs
*postgres only* — kv (bytea) tier + fs (LO) tier from one engine,
*postgres only* — kv (bytea) tier + large (LO) tier from one engine,
no shared-media requirement, no re-routing topology, no consumer-ops
dependency for pool content.
- The engine adds ~400 lines of impl over an already-fixed contract —
the additive shape both ADR-007 and ADR-008 projected, now
evidenced.
- Crash hygiene is *better* than fs `local` (transactional LO
- Crash hygiene is *better* than `local` (transactional LO
lifecycle vs. stage-file residue); the engine's only recovery sweep
targets a class (bypassed-LO) that healthy deployments never produce.
@@ -123,9 +126,9 @@ consolidation option as a shipped engine.
— the fleet case — scales to ~700 MB/s on the POC box; per-box
numbers shift with cores.)
- LO catalog space is never returned (monitor + occasional
operator-run `VACUUM FULL`), the inverse of fs `local`'s
operator-run `VACUUM FULL`), the inverse of `local`'s
immediate-return.
- Feature/maintenance surface: the fs tier now carries two engines
- Feature/maintenance surface: the large tier now carries two engines
(like the kv tier), each with posture deltas to keep honest
(ADR-007's negative, inherited).
@@ -147,10 +150,10 @@ consolidation option as a shipped engine.
- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
this engine serves); REQ-1/3/4 unchanged postures
- ADR-003 (the trait contract + admission door exercised twice now),
ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC
ADR-004 (tier ≠ engine, large tier edition), ADR-005/008 (the GC
invariants this engine participates in — table-level, nothing
fs-specific), ADR-007 (the engine-admission precedent and driver
stack)
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the large-tier
engine surface this amends); [overview.md](../overview.md);
[store-api.md](../store-api.md)
@@ -0,0 +1,325 @@
# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam
## Status
Accepted
## Context
The engine story closed at the decision level (ADR-007/008/009), but
three seams that implementation must not guess are unpinned. All three
concentrate where the tier/engine split stretched the original
contract:
1. **The `get`/`put` value shapes were never unified.** The trait POC's
`get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
a file handle, not a loaded buffer"; the pg-lo POC's get is a
`LoGetHandle` driving `lo_get(oid, off, len)` windows — three
structurally different shapes behind one trait, and no document
named the unified contract. Same on the put side: a byte slice
(kv), a stage-then-commit-rename stream (`local`), and a
chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
trait is the crate's second stable seam (ADR-003); its I/O shapes
are decided before code, not discovered per engine.
2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
rows committed with the entry; staged delete-window candidates
(including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
advisory lock for the sweeper. None of that is expressible through
the public trait method set, and ADR-003's "the shipped trait
impls write plain content-addressed rows into the chosen engine,
nothing else" reads as a bar the fleet mechanism violates. Also
unresolved: who owns the joint entry+pin transaction — ADR-009
says the pg-lo put's single tx carries the pin row, but ADR-005
says `Pin` is a store-core RAII guard under the arbitration lock.
The trait has no parameter for either, and "the backend learns
about pins" would contradict ADR-005. The ops pin-token contract
(ops-surface.md) inherits the same conflict: a remote putter holds
a token, but an expiry-guarded pin row goes stale if the holder
pauses — a sweep could delete a remotely-in-flight put whose row
expired.
3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
the large tier's fleet forms (shared media, re-routing) but pins
nothing for the kv tier, which the fleet story centers on: two
nodes' sqlite engines are trivially partitioned (two files = two
pools, not one), and no document said so.
The deciding facts are in hand (Schrödinger's-code rule — none of
these may wait on consumers): the POCs fixed every engine's native
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
advisory lock, staged window); ADR-001 §4 fixed that ops is a
store-facade consumer with no backend side-door. What is missing is
composition: how the store core drives all engines through one
contract while owning GC state.
Alternatives considered for the fleet-state seam, rejected:
- **Put the fleet machinery on the public trait** (pin/window/table
methods on `Backend`): every engine — including `mem` and `local` —
must then implement SQL-shaped machinery it cannot express. Rejected.
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
row, backend-maintained): contradicts ADR-005's "backends never
learn liveness"; and the arbitration decision (expiry compare,
liveness consult) still belongs to the store, so the store ends up
reaching into the engine's tx anyway. Rejected.
- **A raw-SQL side-channel** (connection escape hatch the store core
may use arbitrarily): turns the engine abstraction off for exactly
the machinery (fleet GC) it most needs to be sound on. Rejected.
The seam below is that escape hatch made honest — named clauses,
contract-tested per engine, instead of raw access.
## Decision
### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put
**The trait stays one trait.** Every engine implements the same method
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
What changes: the trait's I/O types are the crate-internal
abstractions below, which each engine constructs its own way — the
store core programs the abstraction, never the engine shape.
- **`get` returns a read cursor.** One abstraction, two body families:
- **kv engines** (sqlite, postgres, mem): the cursor materializes
the whole value (the engine's row read; bounded by the small-blob
tier policy). Its range reads are in-memory slices; `len()` reads
the row's `size` column.
- **large engines** (`local`, pg-lo): the cursor streams; the medium
is hidden in the cursor body — `local`'s is a pread loop over the
committed file (immutable once committed); pg-lo's drives
descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
posture — pool-friendly, stateless per window; the
tx-scoped-descriptor form is the fallback). `len()` reads the
companion row or stats the file — metadata only, never content
(ADR-008's discipline).
- Both body families serve the ADR-006 range contract (`read_range`
→ slice + slice digest) at one seam. The cursor is
**store-internal — not public API**: store consumers see the
facade's `(len, stream)` shapes (store-api.md) with the cursor
staying behind them; the store's stream utility adapts engine
cursor bodies, it does not expose them.
- **`put` exists in two forms, selected by the store core before
content flows; both land in the CAS entry shape and both
participate in pin-before-publish (§2 decides who holds the tx).**
- **Whole-value put** for the kv tier: the settled row write.
POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
`ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
- **Staged put** for the large tier — ADR-003's
stage-then-commit-rename promoted from behavior to a **named
per-engine contract**: the engine hands out a *staging cursor*
(`local`: the stage file under the store's stage-hygiene protocol;
pg-lo: chunked `lowrite` statements over a tx the store core
holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
measured knee). Commit publishes the entry;
every failure path converges on discard (store-api.md stage
hygiene, per engine, exact-count tested — POC #3 A5's shape).
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
walks the engine's companion structure (rows / files / LO
companion); `size` is metadata-only (ADR-008). The exact-count
sweep-outcome gate (ADR-003) runs per engine.
**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
`pg-lo` does **not** imply feature `postgres`. Each is an engine
feature pulling the shared driver cluster (`tokio-postgres` +
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
node is coherent and supported. overview.md's dependency table pins
the edge.
### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested
**Ownership: the store core owns all GC state — in-process pin and
liveness maps *and* the fleet state's rows and locks. The public
`Backend` trait stays exactly what ADR-003 says (opaque
`hash → bytes`, namespace-blind; "plain content-addressed rows,
nothing else" remains true of the public trait). Backends never learn
liveness (ADR-005) — verbatim.**
The store core reaches the fleet machinery through a **crate-internal
(not exported) engine-state companion trait**, implemented by the
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
`mem` excluded by construction (it cannot be expressed on them and
they are never fleet engines). ADR-008's mechanism fixes *what*; the
seam is *where* the mechanism's SQL lives. Clauses:
- `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
`gc_window (key, visible_after)` at store construction (store-core
call, engine-neutral API; dialect-specific SQL bodies). `owner`
is the **store-instance id** (ADR-011's vocabulary; ADR-008's
"node id" wording predates that split). The pin row's identity is
`(key, owner)` — a dedup re-put by a different instance is a
different pin row; protection composes by row (an entry is
protected while any un-expired pin row exists), which is
ADR-005's refcount semantics expressed in rows. The embedder's
liveness table is *not* created by this seam — ADR-008: schema
pinned by the embedder; the sweep reads it and never owns it.
**Per-engine applicability (the test-gate shape):** sqlite —
`ensure_gc_schema` (window table only, since the window staging
is shared machinery per ADR-008 §2) + `window_stage`/
`window_take` + the sweep-lock clause's single-node body; the
pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
bodies but are exercised by unit tests only — they are dead
machinery on sqlite because no sqlite fleet is on offer (§3).
postgres and pg-lo — all clause families, contract-tested,
fleet-active.
- `pin_commit(joint_tx, entries)` — the store core **holds the
transaction**; the engine contributes its put mechanics (row
shape, CAS clause, LO writes) and the pin-row inserts, and both
commit together. This is the joint entry+pin transaction's owner,
ruled: **the store holds the tx; the engine contributes mechanics
to it.** The public `put` remains the plain, pin-free path — the
store's normal put path routes through `pin_commit` wherever the
engine is seam-capable and fleet state is active.
- pg-lo correction: ADR-009 §2's "put = one transaction" is
unchanged in content but its ownership statement is hereby
corrected — the tx is held by the store core; the pg-lo engine
contributes its staged-put mechanics and companion-row insert
into it. The engine sees pins only as rows; never as semantics.
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
update. The store's pin guard owns the TTL/3 renewal schedule
(ADR-008); the engine owns only the SQL.
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
executes ADR-008's delete-time arbitration (un-expired pin rows +
embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
keyed, at delete time); the store core receives the answer and
applies the decision. If fleet clauses ever grow, they grow here —
the seam is the named home, with contract tests per clause.
- `window_stage(candidates)` / `window_take(now)` — the staged
delete-window candidates. Per ADR-008 §2 the window table exists
on **every** engine (sqlite included): staged candidates are
engine rows; what differs by topology is where *liveness* lives
(in-process maps for single-node engines, tables for fleets), not
whether staging exists.
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
pg advisory lock (ADR-008) for the SQL engines it applies to; the
store treats lock contention exactly as ADR-008 says — the second
sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
nothing deleted. (SQLite today has no fleet story — see §3 — so its
clause body exists but is exercised by single-node tests only.)
Each clause is contract-tested per its applicability (the
per-engine split above): the conformance suite of §1 grows the fleet
clauses for postgres and pg-lo; `local` and
`mem` are excluded by construction.
### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)
**A fleet constructor requires a pool-shared kv engine.** Postgres is
the shipped engine that satisfies this. A sqlite-fleet mechanism does
not exist and is not on offer (per ADR-007's redb precedent this is
recorded as a non-option, not an open question); two nodes' sqlite
engines over "one pool" are two pools — the partitioning failure, the
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
enforceable
seam is the same as ADR-008 §3's: the constructor can require an
explicit declaration; it cannot prove cross-node truth. The
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
and is closed under both tiers.
### 4. The ops pin-token contract's TTL race, resolved by the ownership rule
**The token *is* the pin — there is no second pin state.** Under fleet
engines the put's pin is the expiry-carrying row (ADR-008); the token
identifies the putter as the row's owner. The token itself is an
**opaque id minted by the ops layer** over the pin row it just
committed — the ops layer (not the store facade) owns the
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
a lookup by the renewal op; no schema is specced for it beyond that
(it is a handle, not durable state — the durable state is the pin row
the seam wrote). Concretely:
- **When the server-side `Pin` guard drops:** at response time. The
put lands pinned (the joint tx guarantees the entry and its pin row
commit together); the handler's in-process guard — the thing ADR-005
pins *in-process* with — is released when the `blobs/put` response
completes and the token is handed out. From that moment the pin
row's expiry is the only protection, which is exactly why the
renewal duty below exists. (If the handler kept its guard for the
token's whole life, the TTL machinery would be dead code on the ops
path — the row's expiry, not the guard's presence, is the fleet
contract's liveness signal.)
- The putter renews the pin by renewing the row — through ops it
already speaks: a `blobs/have` probe whose handler, observing a
valid token presented alongside the digest, calls the
crate-internal `pin_renew(key, owner)` via the ops module's own
store-core access (the ops module is in-crate — it crosses the
crate-internal seam by construction, the same as the put handler
holds the `Pin` guard; what never happens is a *consumer* reaching
the seam. ADR-008's `has` remains a pure probe: the renewal is the
ops layer's token-authenticated act on the pin row, not a `has`
semantic), or a re-put of the same digest (CAS dedup; the re-put's
joint tx upserts the key+owner pin row, renewing it). The renewal
cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
20 s at the default 60 s TTL.**
- A putter pausing past TTL falls to ADR-008's expiry semantics
verbatim: the row is abandoned and reaped; the backstop is
delete-then-recover (ADR-005) — the digest is re-offered, the put
is idempotent under CAS. The race resolves to a re-put, never to
silent loss.
- For the `local` engine the token is the in-process guard as ADR-005
had it: nothing persists, nothing expires; the process crash is
ADR-005's in-flight-put loss case with the same backstop. (The
cross-wire token under `local` is the ops layer's in-memory handle
to that guard — same mapping discipline, no expiry.)
ops-surface.md's hand-over contract gains the renewal line and the
token-mapping note; nothing else changes.
## Consequences
**Positive**
- The trait's I/O shapes are decided before code; engine tasks can be
written without guessing signatures. The per-engine
conformance-suite shape (POC #7's 10/10 gate as the norm) gives
decomposition a uniform test gate across all five engines.
- "Backends never learn liveness" survives the fleet mechanism
intact; the store core owns GC state per ADR-005 exactly as
written; ADR-008's mechanism lives in a named, tested seam instead
of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
else" stays true of the public trait.
- The ops pin-token TTL race is resolved with no new ops; the
backstop is already-validated semantics (delete-then-recover).
- The kv tier's fleet-validity rule closes the constructor table
under both tiers.
**Negative**
- A crate-internal companion trait is real machinery: six clause
families on postgres/pg-lo (fleet-active, all contract-tested);
sqlite carries only the window + lock clauses as fleet-relevant
machinery (its pin clauses have unit-test bodies but no fleet mode
on offer — §3). The seam bounds where fleet-GC growth lands; it
does not shrink it.
- Remote putters gain a real protocol duty (renew past 20 s;
pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
- The sqlite sweep-lock clause body exists but is exercised only by
single-node tests until a sqlite-fleet posture is ever
decided (it is not on offer — §3).
**Neutral**
- The `Backend` trait's method set is unchanged from ADR-003/008 —
no public API change beyond what store-api.md already specced; the
cursor/staging abstractions are crate-internal.
- The staged-put contract already existed behaviorally (POC #3 A5);
this ADR names it.
- ADR-009's engine content is unchanged; only its tx-ownership
statement is corrected here.
## References
- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
(tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
liveness-ownership rule this ADR preserves), ADR-006 (the range
contract the cursor serves), ADR-008 (the fleet mechanism whose
state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
(vocabulary + constructor table)
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
`docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
`docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
512 KiB knee, windowed gets)
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
extends); [store-api.md](../store-api.md) (stage hygiene, sweep
semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
(REQ-2, the fleet requirement §2/§3 serve)
@@ -0,0 +1,225 @@
# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large`
## Status
Accepted
## Context
The engine story (ADR-007/008/009) grew the physical layer faster than
its vocabulary grew, and the pre-decomposition review found the drift
is now self-contradictory in the anchor document itself:
1. **Node vs store instance.** requirements.md defines **Node** as
"one running store instance" — then REQ-2 states "a replicator node
may run multiple store instances (a fleet)" and **Fleet** as "two
or more nodes over one shared pool". Under the written definition,
one process hosting two instances is simultaneously one node (by
definition) and two nodes (by REQ-2), and the GC arbitration
domain ("node is the GC arbitration domain") is undefined for the
exact topology ADR-008 exists for. ADR-010 fixes pin ownership;
the pin's *owner id* and the node/instance distinction it needs
are still unpinned vocabulary.
2. **The "fs" tier name is now wrong.** The tier was named when its
only engine was a filesystem (ADR-003's `local` layout). After
ADR-009, half its engines are postgres Large Objects — the name
describes one engine's medium, not the tier. Worse, "fs" collides
with the dispatch language: "large content goes to fs" never said
*why* — the tier's actual routing criterion is content length
(ADR-003's size-threshold dispatch). The docs already speak the
true name informally: store-api.md says "small tier"/"large tier".
3. **mem's classification drifts.** backends-and-dispatch.md calls it
a "backend" in one place and a "testing/utility tier" in another;
ADR-003 lists it under "shipped backends … plus a testing tier";
ADR-004 calls it an ephemeral tier and carves it out of the
two-backend count. In truth it is neither a tier nor a separate
backend: it implements exactly the kv tier's contract (including
`size`, trivially), it is the contract-reference engine every
other engine's conformance tests mirror, and "engine" names what
it is (ADR-007 established that an engine is a concrete
implementation of one tier's contract — requirements.md already
defines the word this way).
4. **Tier and Backend were never formally defined** even though
ADR-004's scope note ("the 'two' here counts tiers, not engines")
leans on both words. Also undecided: whether the feature names
follow the tier rename now or after code exists — the last free
moment is now (zero code; after implementation the rename breaks
crates.io/feature ergonomics for every consumer).
Deciding facts are in hand: REQ-2's fleet fact, the engine set's
final shape (ADR-007/009), the dispatch mechanism (ADR-003), and
ADR-010's GC-state-home table. This is a vocabulary-and-boundary
decision, not an architecture change: ADR-004's tier≠engine note
already established the shape; this ADR gives it final, consistent
names.
Candidates weighed for the tier rename:
- **`lo`/`los`** — names the *postgres* engine (Large Objects), so
"the lo tier's `local` engine" would be backwards. Rejected.
- **`lfs`** — workable, echoes git-lfs specifically; but the tier
also serves packfiles and any oversized workspace artifact, and the
echo binds the crate's naming to one consumer's protocol shape.
Rejected.
- **`large`** — the dispatch-true name (size-threshold routing), the
word the specs already use informally ("large tier"), medium-agnostic.
**Chosen.** For symmetry in prose, the kv tier stays `kv` (its
routing criterion — "small" — lives in the prose as "small blobs"
and needs no rename: `kv` names the mechanism class, and `small`
as a feature/tier name would read as an adjective; the pair's
asymmetry is cosmetic, the *load-bearing* correction is that
neither tier is named after a medium).
Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it
names the engine's mechanism, accurate); `postgres` keeps its name
(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep
their names.
## Decision
### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy)
- **Tier** — one *contract* (a `Backend` trait implementation
boundary) serving one routing class of the pool. The crate ships
exactly two tiers: **kv** (small content) and **large** (over
the size-threshold content). Tier count (2) is ADR-004's count.
- **Engine** — a concrete storage implementation of one tier's
contract (the existing definition, unchanged). Engine count (5) is
this ADR's count.
- **Backend** — *the* contract (the `Backend` trait) plus its shipped
engine set, i.e. the tier as shipped. "Backend" and "tier" name the
same thing at different altitudes; the crate says "tier" for
counting, "the `Backend` trait" for the contract. "Two backends"
reads forever as "two tiers".
- **Store instance** — one running instantiation of the store facade
(one engine set — one or two engines per the constructor mode, §4 —
plus dispatch and GC state). **The GC
arbitration domain** for single-instance topologies; the pin-row
owner id under fleet engines (ADR-010's `owner`; this corrects
ADR-008's "node id" wording, which predates the instance/node
split — a store instance is what owns pins).
- **Node** — one running *process* (an embedder's binary). A node
hosts **one or more store instances**. Node is the process/ops unit:
alkcall connections, ACL grants, sweep cadence, and deployment
identity attach to nodes; GC arbitration and pin ownership attach
to store instances.
- **Fleet** — **two or more store instances over one shared pool**
(via a shared SQL engine). Deliberately not defined by node count:
two instances in one process over one postgres pool are a fleet
under this definition (and REQ-2's phrasing "a replicator node may
run multiple store instances (a fleet)" reads true as written);
two processes on two sqlite files are two pools, never a fleet.
- **Deployment** — unchanged (one operator's whole system).
### 2. mem joins the kv tier as its third engine
`mem` is the **kv tier's ephemeral engine** (feature `mem`,
default-off): a `BTreeMap`-shaped implementation of the kv contract,
including `size` (trivially) and list; never fleet-valid (ADR-010 §3's
rule and this ADR §4's constructor table); the **contract-reference
engine** — every other engine's
conformance tests mirror its suite. It is not a tier, not a third
backend, and the ADR-004 count ("two") survives untouched because
tier and engine were never the same count — this ADR is that
distinction's final form.
### 3. The fs tier is renamed `large`
In every document, status line, feature name, and constructor
parameter: tier `fs` → tier `large`; feature `fs` → feature `large`
(default-on, as `fs` was); dispatch prose says "large tier"/"small
tier" as store-api.md already does; the `local` engine's name is
kept (its meaning — this node's own filesystem — stays the
constructor's meaning even under the shared-media fleet posture,
where it is explicitly *not* only-node-local; the constructor's
explicit shared-media assertion, ADR-008 §3, is what disambiguates).
### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here)
| Tier | Engine | Feature | Default | Fleet-valid |
|---|---|---|---|---|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
| kv | postgres | `postgres` | no | **yes** |
| kv | mem | `mem` | no | no |
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
| large | pg-lo | `pg-lo` | no | **yes** |
Constructor modes (dispatch shapes) — pinned here (implied by
ADR-008 §3's re-routing bullet and the mem tests, but never named):
- **dual-tier** (the default): kv + large, size-threshold dispatch
between them (ADR-003).
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
constructor mode with no large tier; over-threshold puts are a
constructor-time error (the client-node re-routing posture handles
them via the ops surface instead — its composition, not a store
mode).
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
no dispatch threshold in effect (everything is local and small);
a testing posture, explicitly not a production story.
- Single-tier SQL modes (postgres-only, pg-lo-only) are **not**
constructor modes this crate ships: the two-tier shape is
ADR-004's requirement (both tiers are load-bearing by measured
economics); "postgres-only" as in *one SQL instance serving both
tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
consolidation), which is the dual-tier mode over one engine.
### 5. Deferral-policy wording (README alignment)
The deferral policy's categories are: `deferred(scope)`,
externally-owned questions, and decided-but-sequenced work — three
names, defined in open-questions.md's header. README.md's recap says
"two parking kinds" and is corrected to name all three (the third is
not a *parking* kind — it is work, not a parked question — hence
this ADR's wording here rather than a silent edit).
## Consequences
**Positive**
- The anchor document's self-contradiction (node/fleet) becomes a
three-level vocabulary (store instance / node / deployment) whose
levels map one-to-one onto the mechanisms (GC domain / process-ops
unit / operator whole).
- The fs→large rename removes the last medium-named tier; feature
names, tier names, and dispatch prose now agree, before any code
exists to break.
- mem's classification is stated once, as the contract-reference kv
engine; the "plus a testing tier" carve-outs dissolve.
- The constructor table gives task decomposition a closed set: 5
engine implementations, 3 constructor modes, 1 dispatch policy.
**Negative**
- Vocabulary churn in every doc that says "fs" (the rename is
mechanical but touches all specs and ADR-003/004/007/008/009 status
lines and reference sections). Zero code exists — this is the last
free moment; after Phase 1 starts, feature renames break consumers.
- "Backend"/"tier" as interchangeable words risks reader confusion
with ADR-003's "Backend trait" — mitigated by the rule that the
*trait* is always named with "trait" ("the `Backend` trait"), never
bare "the backend".
**Neutral**
- No engine's contract, feature flag semantics, or constructor
parameter changes; `mem` gains the word "engine" and loses the
word "backend".
- REQ-2's meaning is unchanged (the fleet definition is re-derived
from the same fact, not re-fact'd).
## References
- ADR-003 (the trait contract; its "shipped backends" list is now
read through this ADR's vocabulary), ADR-004 (tier≠engine — the
scope note this ADR gives final names), ADR-007 (kv engines;
constructor selection), ADR-008 (the second tier's engine candidate
and the fleet topology; the vocabulary it assumed is now pinned),
ADR-009 (pg-lo admission; the consolidation option whose
constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
(the I/O seams and GC-state home this vocabulary must fit)
- [requirements.md](../requirements.md) (the canonical vocabulary
copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
(the tier/engine table's spec home)
- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the
reference-impl role mem already played in the POCs);
`poc-pglo-findings.md` (the gate the engine suite mirrors)