docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round
ADR-010 (new): the Backend trait's I/O seams pinned before any code — 'get' returns a crate-internal read cursor (kv engines materialize; 'local' pread loop; pg-lo lo_get windows), 'put' has two named forms (whole-value kv / staged-put large); GC state is store-core-owned via a crate-internal contract-tested engine-state seam on the SQL-backed engines (backends never learn liveness — ADR-005 verbatim); the store core holds the joint entry+pin transaction (corrects ADR-009 §2's ownership statement); kv-tier fleet-validity rule added (mirror of ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin; renewal rides have/re-put; pause-past-TTL falls to delete-then-recover); feature graph pinned (pg-lo does not imply postgres). ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs engine (concrete impl, exactly five); store instance / node / fleet split (fleet = pool-sharing, not node count — fixes requirements.md's self-contradiction with REQ-2); mem demoted from 'backend'/'testing tier' to the kv tier's third engine (contract-reference engine); fs tier renamed 'large' including the feature name (last free moment before code exists); constructor modes pinned (dual-tier default, kv-only, mem-only) plus the composite fleet-validity predicate. Consistency round across all specs and ADR-003/004/007/008/009: stale pg-lo passages resolved per ADR-009's supersession; ADR-008 owner 'node id' → store-instance id; threshold wording corrected (conservative edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport shape specced; mem classification unified; 008/009 ADR files renamed to match the tier rename; README deferral-policy recap aligned (third category = decided-but-sequenced work, not a parking kind); POC crate list completed. Verification: two independent architecture review rounds (the first found 4 criticals — unpinned trait I/O shapes, missing fleet-state seam, node/fleet self-contradiction, pin-token/TTL conflict — all resolved; final round: zero criticals); all markdown links resolve; ADR tables complete (11 ADRs).
This commit is contained in:
1 parent
9bbe6f3ad9
commit
abf8c5ae9c
18 files changed
+1137
-336
No files matched your search
@@ -16,7 +16,7 @@ addresses, not algorithms).
|
||||
|
||||
The residual was the **key encoding mechanics** — the byte layout of
|
||||
keys inside backends. This is the crate's declared one-way door
|
||||
(AGENTS convention 5): key bytes persist in kv rows and fs filenames,
|
||||
(AGENTS convention 5): key bytes persist in kv rows and `local`-engine filenames,
|
||||
with no schema-migration story; and this is the wire-format ADR that
|
||||
must exist before the first consumer (convention 7's gate on publishing
|
||||
wire-relevant decisions). POC #1 validated the mechanics empirically:
|
||||
@@ -26,7 +26,7 @@ both algorithms in one flat pool (finding 5).
|
||||
|
||||
## Decision
|
||||
|
||||
### Hash policy (three tiers)
|
||||
### Hash policy (three classes)
|
||||
|
||||
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
|
||||
SHA-256. The store's canonical algorithm; git objects are first-class
|
||||
|
||||
@@ -5,10 +5,15 @@
|
||||
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
||||
engines, sqlite and postgres, behind this contract; the dispatch layer
|
||||
and trait are unchanged. The trait was extended by ADR-008 with a
|
||||
`size` length probe — before any backend ships data — and the fs
|
||||
tier became engine-selectable (`local` default; pg-lo candidate), so
|
||||
this ADR's "engines"/"backends" counts are read through ADR-004's
|
||||
tier≠engine scope note)
|
||||
`size` length probe — before any backend ships data — and the second
|
||||
tier became engine-selectable (`local` default; pg-lo admitted by
|
||||
ADR-009), so this ADR's "engines"/"backends" counts are read through
|
||||
ADR-004's tier≠engine scope note. The I/O shapes behind `get`/`put`
|
||||
are pinned by ADR-010 (read cursor / staged put). The tier's name
|
||||
changed: `fs` → `large` (ADR-011 — its routing criterion is length,
|
||||
not medium); this ADR's per-engine counts also gained the `mem` kv
|
||||
engine as ADR-011's vocabulary resolves — mem is a kv engine, not a
|
||||
tier)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -54,28 +59,34 @@ Evidence in hand (no further POC or consumer-waiting needed):
|
||||
- **Virgin-store reads are no-ops** (missing table = absent/empty).
|
||||
- **Namespace-blind** (ADR-005): backends see hashes only.
|
||||
|
||||
### Shipped backends (two, default-on; plus a testing tier)
|
||||
### Shipped tiers (two, default-on; the kv tier also carries the mem engine)
|
||||
|
||||
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
|
||||
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
|
||||
the regime holding most git objects, workspace files, and manifests.
|
||||
WAL + `synchronous=NORMAL` as shipped durability; prepared
|
||||
statements; no-tables-yet tolerance. (The tier's engine set was
|
||||
widened to include postgres by ADR-007.)
|
||||
- **`fs` (default-on)** for large blobs — flat
|
||||
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
|
||||
widened to include postgres by ADR-007, and mem — the ephemeral
|
||||
reference engine — by ADR-011's vocabulary resolution.)
|
||||
- **`fs` (default-on; renamed `large` by ADR-011)** for large blobs —
|
||||
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
|
||||
conclusion); stage-then-commit-rename; pread range reads (sound
|
||||
locally per ADR-006).
|
||||
- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and
|
||||
in-process ephemerality. Never a production story.
|
||||
locally per ADR-006). (The tier's engine set was widened by ADR-008/
|
||||
009.)
|
||||
- **`mem` (default-off):** ephemeral `BTreeMap` kv engine for tests
|
||||
and in-process ephemerality — **an engine of the kv tier, not a
|
||||
third tier** (ADR-011 §2; the pre-ADR-011 "testing tier" phrasing
|
||||
is superseded). Never a production story; the contract-reference
|
||||
engine.
|
||||
|
||||
Both production backends are default-on (batteries-included, the
|
||||
Both production tiers are default-on (batteries-included, the
|
||||
alkgit feature-model precedent); `mem` is not.
|
||||
|
||||
### Dispatch policy
|
||||
|
||||
- **Size-threshold routing; default 128 KiB, constructor-tunable**
|
||||
(midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4).
|
||||
(chosen at the bottom of the measured ~128-256 KiB crossover zone —
|
||||
the conservative edge of the flat zone; POC #3 A4).
|
||||
- **Unknown-length puts buffer in memory up to the threshold; the
|
||||
buffer's fate has three pinned semantics** (finding A6's option (d)):
|
||||
1. **Threshold exceeded mid-stream** (non-restartable sources are
|
||||
@@ -141,9 +152,10 @@ alkgit feature-model precedent); `mem` is not.
|
||||
|
||||
**Neutral**
|
||||
|
||||
- A network-backed backend (S3-like) remains possible via the same
|
||||
- A network-backed engine (S3-like) remains possible via the same
|
||||
trait but ships nothing here; its list/GC obligations are the
|
||||
acceptance gate.
|
||||
acceptance gate; per-tier, via a new ADR (ADR-011's vocabulary: an
|
||||
engine addition, not a tier addition).
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -2,7 +2,12 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (the count's vocabulary is pinned by ADR-011: the "two" here
|
||||
are *tiers* — kv + large, the latter renamed from `fs`; "backend"
|
||||
names the same thing at a different altitude. The `mem` item below is
|
||||
the kv tier's third engine, per ADR-011 §2, not an extra tier —
|
||||
the pre-ADR-011 "testing tier" carve-out is superseded. The I/O
|
||||
seam-level refinements of the trait contract are ADR-010's)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -11,9 +16,10 @@ rounds before being edited away, but the correction it was reaching for
|
||||
arrived only in the Phase 1 review, and needs recording so no future
|
||||
round reabsorbs the concern incorrectly:
|
||||
|
||||
- There are **always** two physical backends in this problem: storage
|
||||
for small content (fast, kv/sqlite) and the filesystem for large
|
||||
content. This is not ugly, it is the measured shape of the problem
|
||||
- There are **always** two physical tiers in this problem: storage
|
||||
for small content (fast, kv/sqlite) and storage for large content
|
||||
(the large tier / filesystem). This is not ugly, it is the measured
|
||||
shape of the problem
|
||||
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
|
||||
- The "two backends is ugly" sentiment was actually about a **third
|
||||
layer** in the old alknet-filesystem research — the *virtual
|
||||
@@ -30,23 +36,25 @@ is the boundary it will build against.
|
||||
|
||||
## Decision
|
||||
|
||||
**This crate ships exactly two production backends and nothing more;
|
||||
**This crate ships exactly two production tiers and nothing more;
|
||||
every manifest/namespace/path-tree layer is a consumer, by construction
|
||||
and by contract.** (A non-production ephemeral `mem` tier exists for
|
||||
tests — ADR-003; it is not a storage story and does not extend this
|
||||
boundary.)
|
||||
and by contract.** (A non-production ephemeral `mem` engine exists for
|
||||
tests — ADR-003, resolved to a kv-tier engine by ADR-011 §2; it is not
|
||||
a storage story and does not extend this boundary.)
|
||||
|
||||
**Scope note (post-ADR-007 reconciliation): the "two" here counts
|
||||
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
|
||||
engines behind it to a set (sqlite default + postgres, feature-gated)
|
||||
so a node's kv rides its relational engine of choice. The count that
|
||||
matters to this ADR — trait impls *this crate ships* as CAS tiers —
|
||||
remains two (kv + fs); a second kv engine does not make a third
|
||||
backend, and nothing in the engine set touches the manifest-layer
|
||||
boundary below.
|
||||
**Scope note (post-ADR-007 reconciliation, vocabulary finalized by
|
||||
ADR-011): the "two" here counts *tiers*, not engines.** The kv tier is
|
||||
one contract; ADR-007 widened the engines behind it to a set (sqlite
|
||||
default + postgres, feature-gated; mem the ephemeral reference engine,
|
||||
ADR-011) so a node's kv rides its relational engine of choice. The
|
||||
count that matters to this ADR — trait impls *this crate ships* as CAS
|
||||
tiers — remains two (kv + large, the tier ADR-011 renamed from `fs`);
|
||||
a second kv engine does not make a third tier, and nothing in the
|
||||
engine set touches the manifest-layer boundary below.
|
||||
|
||||
1. **Two production backends is the complete set.** kv/sqlite (small)
|
||||
+ fs (large) cover the scale economics with measured evidence
|
||||
1. **Two production tiers is the complete set.** kv/sqlite (small)
|
||||
+ large (the tier formerly named `fs`) cover the scale economics
|
||||
with measured evidence
|
||||
(ADR-003); no other physical tier is needed by any current or
|
||||
planned consumer's *blob* needs.
|
||||
2. **The "third backend" is not this crate's layer.** The vfs/appfile
|
||||
@@ -86,8 +94,8 @@ boundary below.
|
||||
|
||||
**Positive**
|
||||
|
||||
- The crate's scope stays testable and small: two backends, one pool,
|
||||
one hash domain. No schema/versioning surface for manifest formats —
|
||||
- The crate's scope stays testable and small: two tiers, one pool,
|
||||
one hash domain. No schema/versioning surface for manifest formats —
|
||||
consumers iterate on theirs freely.
|
||||
- alkgit and alkfs build their (very different) mapping layers on one
|
||||
shared pool without either bending the store; their manifests interop
|
||||
|
||||
@@ -49,7 +49,7 @@ served by range reads).
|
||||
(git refs, alkfs path trees) and hand liveness over per sweep via the
|
||||
live-shared seam. In-memory namespace tables remain available for
|
||||
ephemeral consumers. This also dissolves the hidden wrinkle of a
|
||||
store-internal root table under fs-only deployments (where would its
|
||||
store-internal root table under deployments without a SQL engine (where would its
|
||||
durable bytes live?). Ephemeral consumers are covered by pins; a
|
||||
deployment that needs store-durable roots registers them as
|
||||
consumer-owned durable state and sweeps through the seam.
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (the engine story's I/O seams and GC-state home are ruled by
|
||||
ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as
|
||||
its third engine — is pinned by ADR-011)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -57,9 +59,9 @@ in OQ-09.
|
||||
**The kv tier ships two engines — sqlite (default) and postgres —
|
||||
behind one `Backend` trait; engine selection is a constructor
|
||||
parameter on the backend's configuration.** A node provisions one
|
||||
relational engine plus the fs tier; the kv tier rides whichever
|
||||
relational engine plus the large tier; the kv tier rides whichever
|
||||
engine the operator chose. This collapses the classical three-engine
|
||||
downstream stack (kv + fs + relational) to two, and makes which
|
||||
downstream stack (kv + large content + relational) to two, and makes which
|
||||
relational engine a per-node choice rather than a crate fork.
|
||||
|
||||
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
||||
@@ -83,7 +85,7 @@ relational engine a per-node choice rather than a crate fork.
|
||||
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
|
||||
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
|
||||
virgin-store no-op reads, namespace-blindness. The dispatch layer
|
||||
(size threshold, kv↔fs routing, fall-through) is engine-blind and
|
||||
(size threshold, kv↔large routing, fall-through) is engine-blind and
|
||||
unchanged.
|
||||
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
|
||||
as requirements** (they are the difference between a port and an
|
||||
@@ -127,7 +129,7 @@ collapses to its own deployment choice among shipped engines.
|
||||
**Positive**
|
||||
|
||||
- The classical downstream stack lands at two engines per node (one
|
||||
relational + fs), eliminating the "sqlite *beside* the real
|
||||
relational engine + large tier), eliminating the "sqlite *beside* the real
|
||||
database" duplication this crate's consumers would otherwise carry
|
||||
forever.
|
||||
- Per-node engine choice (constructor parameter, per ADR-003/004)
|
||||
|
||||
+71
-52
@@ -1,10 +1,14 @@
|
||||
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
|
||||
# ADR-008: Trait length probe, fleet GC, and large-tier engines
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
|
||||
superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
|
||||
passed gate; every other part of this ADR stands as written)
|
||||
passed gate — the two §Consequences passages that said pg-lo "remains
|
||||
unbuilt" are resolved by that same supersession; the tier this ADR's
|
||||
title and text call "fs" is renamed `large` by ADR-011; the fleet GC
|
||||
mechanism's state home and the joint-tx ownership are ruled by
|
||||
ADR-010; every other part of this ADR stands as written)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -18,7 +22,7 @@ sharing one pool via postgres; requirements.md):
|
||||
probe"; store-api.md) and pool accounting rides `list` + `stat`
|
||||
(ADR-005), but the trait's methods are `has/get/put/delete/list/
|
||||
name` with no value-metadata access. The kv engines' row shapes
|
||||
both carry a `size` column (POC #5 B5), and the fs tier can stat a
|
||||
both carry a `size` column (POC #5 B5), and the large tier can stat a
|
||||
file — but with no trait method, kv `stat` degenerates to a full
|
||||
value load. The trait is the crate's second stable seam (ADR-003);
|
||||
discovering this during implementation would be a breaking trait
|
||||
@@ -32,8 +36,8 @@ sharing one pool via postgres; requirements.md):
|
||||
node B's in-flight put is invisible. Data loss, structurally: the
|
||||
topology REQ-2 requires does not satisfy the protocol ADR-005
|
||||
specifies.
|
||||
3. **The fs tier is node-local, silently partitioning a fleet.** The
|
||||
fs backend stores to a sharded layout on the node's own filesystem
|
||||
3. **The large tier is node-local, silently partitioning a fleet.** The
|
||||
large tier stores to a sharded layout on the node's own filesystem
|
||||
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
|
||||
put via node A exists only on A's disk; node B's get fall-through
|
||||
queries its own fs and misses, and `list()` unions disagree per
|
||||
@@ -41,7 +45,7 @@ sharing one pool via postgres; requirements.md):
|
||||
hypothetical.
|
||||
|
||||
These three travel together: the fleet topology forces #2 and #3, and
|
||||
#3's resolution constrains what the fs tier must support, while #1 is
|
||||
#3's resolution constrains what the large tier must support, while #1 is
|
||||
a trait-level gap any topology hits.
|
||||
|
||||
## Decision
|
||||
@@ -55,7 +59,7 @@ tier:
|
||||
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
|
||||
never a value load. The `size` column both POC row shapes carry
|
||||
becomes contract-relevant.
|
||||
- **fs engine (`local`)**: stat/metadata read, no content fetch.
|
||||
- **large engine (`local`)**: stat/metadata read, no content fetch.
|
||||
- The store facade's `stat` (store-api.md) and the accounting path
|
||||
(ADR-005) ride this method; the `has` method remains a pure
|
||||
existence probe.
|
||||
@@ -71,18 +75,20 @@ never deleted by a sweep started before it committed** (ADR-005).
|
||||
Under a fleet, the mechanism moves into the shared engine:
|
||||
|
||||
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
|
||||
inserts in one transaction on the fleet engine: the entry row and
|
||||
its pin row commit together (or neither does). The pin row carries
|
||||
an owner (node id) and expiry: the pin guard renews its row every
|
||||
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
||||
parameters, not API). **Protection = an un-expired pin row**: at
|
||||
delete-time arbitration, only `expiry > now()` pins count; an
|
||||
expired pin is abandoned (its holder failed to renew — the same
|
||||
liveness-failure signal as an in-process guard's drop) and is reaped
|
||||
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
||||
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
||||
remains the correctness backstop — content is re-put byte-identically
|
||||
under the same key.
|
||||
inserts in one transaction on the fleet engine: the entry row and
|
||||
its pin row commit together (or neither does). The pin row carries
|
||||
an owner (a store-instance id — ADR-011's vocabulary; "node id" is
|
||||
this ADR's original wording, superseded because a node may host
|
||||
several instances) and expiry: the pin guard renews its row every
|
||||
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
||||
parameters, not API). **Protection = an un-expired pin row**: at
|
||||
delete-time arbitration, only `expiry > now()` pins count; an
|
||||
expired pin is abandoned (its holder failed to renew — the same
|
||||
liveness-failure signal as an in-process guard's drop) and is reaped
|
||||
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
||||
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
||||
remains the correctness backstop — content is re-put byte-identically
|
||||
under the same key.
|
||||
- **Registered liveness sources have one fleet form: a liveness
|
||||
table.** A fleet embedder registers its liveness as a named table of
|
||||
`(key, …)` rows in the shared engine (schema pinned by the embedder,
|
||||
@@ -131,9 +137,9 @@ Under a fleet, the mechanism moves into the shared engine:
|
||||
is shared machinery. The pin-row expiry/ttl machinery exists only
|
||||
where fleet engines are active.
|
||||
|
||||
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
|
||||
### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible
|
||||
|
||||
The fs tier is a tier, not a medium. It ships engines the way the kv
|
||||
The large tier is a tier, not a medium. It ships engines the way the kv
|
||||
tier does (ADR-007), and a deployment's topology determines which are
|
||||
valid:
|
||||
|
||||
@@ -141,7 +147,7 @@ valid:
|
||||
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
|
||||
REQ-4) and for fleet nodes using **shared media** (below).
|
||||
- **Fleet + shared media (deployment requirement):** multi-node pools
|
||||
may keep the `local` engine by pointing every node's fs tier at
|
||||
may keep the `local` engine by pointing every node's large tier at
|
||||
shared/replicated media. The soundness claim is *qualified*, not
|
||||
assumed: immutable-once-committed files plus stage-then-commit-
|
||||
rename are sound on media that provides (a) atomic rename within a
|
||||
@@ -158,43 +164,52 @@ valid:
|
||||
not share media designates **one storage node**; large-blob writes
|
||||
and reads over the pool route through it via the ops surface
|
||||
(verified fetch/put; ADR-001). Client nodes' constructors are
|
||||
configured with a **kv-tier-only dispatch** (or an fs tier reserved
|
||||
for node-private, consumer-owned content — which is outside the
|
||||
pool and outside this crate's GC; it is the consumer's own storage
|
||||
beside the pool per ADR-004, not pool content). The facade shape on
|
||||
configured with **kv-only mode** (ADR-011 §4's name for what this
|
||||
bullet first called "kv-tier-only dispatch") — no local large tier
|
||||
over pool content at all; a client node's *node-private,
|
||||
consumer-owned* large-tier content is outside the pool and outside
|
||||
this crate's GC entirely: it is the consumer's own storage beside
|
||||
the pool per ADR-004, not a constructor mode over pool content, and
|
||||
the composite fleet-validity rule (backends-and-dispatch.md)
|
||||
therefore has no "local on node-private media over one pool" entry.
|
||||
The facade shape on
|
||||
a client node is its own store for sub-threshold content plus its
|
||||
registered ops-put path for large content — a consumer-side
|
||||
composition, not a store-level mode.
|
||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
||||
as the fs tier's storage — the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
||||
with its admission gated on measured evidence (POC #7: LO write/read
|
||||
- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects
|
||||
as the large tier's storage —
|
||||
the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Originally named as a candidate,
|
||||
admission gated on measured evidence (POC #7: LO write/read
|
||||
curves at the packfile regime, tx-scoped handle cost under
|
||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
||||
behavior) and its own engine ADR. NOT shipped speculatively;
|
||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||
pg-lo is the consolidation option if those are unacceptable. *Update
|
||||
2026-10-03: POC #7 ran and passed its gate —
|
||||
`docs/research/poc-pglo-findings.md` (companion-table authority,
|
||||
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
|
||||
behind page-cache fs by a measured 20-50x aggregate-multiplied to
|
||||
~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture).
|
||||
The engine ADR's remaining evidence path is open; admission remains
|
||||
gated on it, not on speculation.*
|
||||
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
|
||||
behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s
|
||||
aggregate at 16 readers). Superseded the same day: ADR-009 admitted
|
||||
the engine — this bullet's original "candidate, not shipped"
|
||||
posture is historical; the engine is shipped, feature `pg-lo`.*
|
||||
- **The no-mixed-large-tiers rule is a deployment invariant, enforced at
|
||||
the enforceable seam.** Cross-node configuration cannot be validated
|
||||
by any one constructor (it sees only its own node). What the
|
||||
constructor *can* enforce: a fleet-mode constructor (fleet engines
|
||||
active) requires an explicit fs-tier declaration — `local` (with the
|
||||
active) requires an explicit large-tier declaration — `local` (with the
|
||||
operator's assertion that its root is fleet-shared media), routed
|
||||
(no local fs over pool content), or — when shipped — `pg-lo`. The
|
||||
(kv-only mode, ADR-011 §4; no local filesystem over pool content), or
|
||||
`pg-lo`. The
|
||||
cross-node truth ("is node B's `local` actually the same media as
|
||||
node A's?") is a deployment invariant this crate documents and ops
|
||||
verifies, not something a constructor can prove; a partitioned
|
||||
tier's observable signature (get fall-through misses for entries
|
||||
another node wrote) is the documented detection symptom.
|
||||
another node wrote) is the documented detection symptom. The
|
||||
composite fleet-validity predicate over both tiers is pinned in
|
||||
backends-and-dispatch.md ("The composite fleet-validity rule").
|
||||
|
||||
In sum: a fleet's fs tier over pool content is either shared media (all
|
||||
In sum: a fleet's large tier over pool content is either shared media (all
|
||||
nodes' `local` roots on the same fleet-shared media) or routed (one
|
||||
storage node); mixing per-node-local media over one pool is the
|
||||
partitioning failure this decision exists to prevent.
|
||||
@@ -211,9 +226,10 @@ partitioning failure this decision exists to prevent.
|
||||
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
|
||||
invariant extends to fleets as a *stated* mechanism, not an
|
||||
accident of single-process luck.
|
||||
- The fs tier's fleet problem has named, shipped-code-free answers
|
||||
(shared media, re-routing) and a named consolidation candidate
|
||||
(pg-lo) whose admission path is the same POC-first door as ADR-007.
|
||||
- The large tier's fleet problem has named answers: shipped-code-free
|
||||
for shared media and re-routing, shipped-code for the consolidation
|
||||
option (pg-lo — admitted by ADR-009, the same POC-first door as
|
||||
ADR-007).
|
||||
|
||||
**Negative**
|
||||
|
||||
@@ -223,9 +239,9 @@ partitioning failure this decision exists to prevent.
|
||||
surface for orphaned pins at node crash — the expiry is the cleanup
|
||||
handle), advisory-lock coordination, and SQL-level arbitration
|
||||
added on top of the already-intricate delete-window state machine.
|
||||
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
|
||||
re-route pays for its ADR + POC before shipping large blobs — the
|
||||
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
|
||||
- `pg-lo` was built under ADR-009 after POC #7 passed — the
|
||||
measured, deliberate sequence of the ADR-006 pattern (admission
|
||||
gate first, engine second), not a hedge; resolved same-day.
|
||||
|
||||
**Neutral**
|
||||
|
||||
@@ -245,13 +261,16 @@ partitioning failure this decision exists to prevent.
|
||||
columns; the driver stack pg-lo would ride)
|
||||
- ADR-002 (the one-way-door discipline this trait change applies),
|
||||
ADR-003 (the trait contract and admission gate `size` and pg-lo
|
||||
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
|
||||
amend/enter), ADR-004 (tier≠engine, extended to the large tier),
|
||||
ADR-005 (the GC invariant and single-node protocol extended here,
|
||||
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
|
||||
follows), ADR-007 (the engine-behind-one-trait precedent)
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
|
||||
and tier surface this amends);
|
||||
[store-api.md](../store-api.md) (`stat`'s contract);
|
||||
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
||||
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
||||
topology's transport)
|
||||
and tier surface this amends);
|
||||
[store-api.md](../store-api.md) (`stat`'s contract);
|
||||
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
||||
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
||||
topology's transport); ADR-010 (the fleet mechanism's state home
|
||||
and the joint-tx ownership this ADR's §2 implied but did not rule);
|
||||
ADR-011 (the vocabulary — instance/fleet — this ADR's topology
|
||||
language resolves to; the large tier renamed `large`)
|
||||
+15
-12
@@ -1,14 +1,17 @@
|
||||
# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine
|
||||
# ADR-009: pg-lo — postgres Large Objects as the large tier's second engine
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (admission evidence: POC #7, passed 2026-10-03 —
|
||||
`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
|
||||
gate)
|
||||
gate. The tier this ADR calls "fs" is renamed `large` by ADR-011);
|
||||
§2's "put = one transaction" tx-ownership statement is corrected by
|
||||
ADR-010 §2 (the store core holds the tx; the engine contributes its
|
||||
staged-put mechanics into it — engine content unchanged))
|
||||
|
||||
## Context
|
||||
|
||||
ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's
|
||||
ADR-008 named `pg-lo` as the candidate large-tier engine for REQ-2's
|
||||
fleet topology (multiple store instances sharing one pg pool, serving
|
||||
large/beyond-threshold pool content) — expressly gated on the ADR-003
|
||||
admission door: measured evidence first, in the ADR-007 shape. POC #7
|
||||
@@ -38,7 +41,7 @@ The gate's three legs all passed:
|
||||
|
||||
## Decision
|
||||
|
||||
**The fs tier ships `pg-lo` as its second engine — postgres Large
|
||||
**The large tier ships `pg-lo` as its second engine — postgres Large
|
||||
Objects — behind the same `Backend` trait, feature `pg-lo` (default-
|
||||
off), constructor-selected per node.** REQ-2's deployment choice set
|
||||
(ADR-008) extends from {shared media, re-routing} to include the
|
||||
@@ -58,7 +61,7 @@ consolidation option as a shipped engine.
|
||||
into the LO and commit the row on completion; rollback discards
|
||||
both.
|
||||
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
|
||||
the fs-tier handle closes over the companion row (oid + length) —
|
||||
the large-tier handle closes over the companion row (oid + length) —
|
||||
immutable, so the row is authoritative — not over a tx-scoped
|
||||
descriptor. Ranged reads are single stateless statements,
|
||||
pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
|
||||
@@ -82,7 +85,7 @@ consolidation option as a shipped engine.
|
||||
(committed LOs without companion rows — mixed/legacy stores). The
|
||||
crash story is structurally clean: LO lifecycle is transactional
|
||||
in modern postgres; kill mid-tx leaves zero pages, zero
|
||||
half-commits (cleaner than fs `local`'s stage-file residue,
|
||||
half-commits (cleaner than `local`'s stage-file residue,
|
||||
which needs its own recovery sweep).
|
||||
6. **Performance deltas, as named capacity facts (not hidden, not
|
||||
solved):** single-stream cached get runs 20–50× behind page-cache
|
||||
@@ -105,13 +108,13 @@ consolidation option as a shipped engine.
|
||||
**Positive**
|
||||
|
||||
- REQ-2's consolidation option is real and shipped: a fleet node runs
|
||||
*postgres only* — kv (bytea) tier + fs (LO) tier from one engine,
|
||||
*postgres only* — kv (bytea) tier + large (LO) tier from one engine,
|
||||
no shared-media requirement, no re-routing topology, no consumer-ops
|
||||
dependency for pool content.
|
||||
- The engine adds ~400 lines of impl over an already-fixed contract —
|
||||
the additive shape both ADR-007 and ADR-008 projected, now
|
||||
evidenced.
|
||||
- Crash hygiene is *better* than fs `local` (transactional LO
|
||||
- Crash hygiene is *better* than `local` (transactional LO
|
||||
lifecycle vs. stage-file residue); the engine's only recovery sweep
|
||||
targets a class (bypassed-LO) that healthy deployments never produce.
|
||||
|
||||
@@ -123,9 +126,9 @@ consolidation option as a shipped engine.
|
||||
— the fleet case — scales to ~700 MB/s on the POC box; per-box
|
||||
numbers shift with cores.)
|
||||
- LO catalog space is never returned (monitor + occasional
|
||||
operator-run `VACUUM FULL`), the inverse of fs `local`'s
|
||||
operator-run `VACUUM FULL`), the inverse of `local`'s
|
||||
immediate-return.
|
||||
- Feature/maintenance surface: the fs tier now carries two engines
|
||||
- Feature/maintenance surface: the large tier now carries two engines
|
||||
(like the kv tier), each with posture deltas to keep honest
|
||||
(ADR-007's negative, inherited).
|
||||
|
||||
@@ -147,10 +150,10 @@ consolidation option as a shipped engine.
|
||||
- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
|
||||
this engine serves); REQ-1/3/4 unchanged postures
|
||||
- ADR-003 (the trait contract + admission door exercised twice now),
|
||||
ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC
|
||||
ADR-004 (tier ≠ engine, large tier edition), ADR-005/008 (the GC
|
||||
invariants this engine participates in — table-level, nothing
|
||||
fs-specific), ADR-007 (the engine-admission precedent and driver
|
||||
stack)
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the large-tier
|
||||
engine surface this amends); [overview.md](../overview.md);
|
||||
[store-api.md](../store-api.md)
|
||||
@@ -0,0 +1,325 @@
|
||||
# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The engine story closed at the decision level (ADR-007/008/009), but
|
||||
three seams that implementation must not guess are unpinned. All three
|
||||
concentrate where the tier/engine split stretched the original
|
||||
contract:
|
||||
|
||||
1. **The `get`/`put` value shapes were never unified.** The trait POC's
|
||||
`get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
|
||||
a file handle, not a loaded buffer"; the pg-lo POC's get is a
|
||||
`LoGetHandle` driving `lo_get(oid, off, len)` windows — three
|
||||
structurally different shapes behind one trait, and no document
|
||||
named the unified contract. Same on the put side: a byte slice
|
||||
(kv), a stage-then-commit-rename stream (`local`), and a
|
||||
chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
|
||||
trait is the crate's second stable seam (ADR-003); its I/O shapes
|
||||
are decided before code, not discovered per engine.
|
||||
|
||||
2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
|
||||
rows committed with the entry; staged delete-window candidates
|
||||
(including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
|
||||
advisory lock for the sweeper. None of that is expressible through
|
||||
the public trait method set, and ADR-003's "the shipped trait
|
||||
impls write plain content-addressed rows into the chosen engine,
|
||||
nothing else" reads as a bar the fleet mechanism violates. Also
|
||||
unresolved: who owns the joint entry+pin transaction — ADR-009
|
||||
says the pg-lo put's single tx carries the pin row, but ADR-005
|
||||
says `Pin` is a store-core RAII guard under the arbitration lock.
|
||||
The trait has no parameter for either, and "the backend learns
|
||||
about pins" would contradict ADR-005. The ops pin-token contract
|
||||
(ops-surface.md) inherits the same conflict: a remote putter holds
|
||||
a token, but an expiry-guarded pin row goes stale if the holder
|
||||
pauses — a sweep could delete a remotely-in-flight put whose row
|
||||
expired.
|
||||
|
||||
3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
|
||||
the large tier's fleet forms (shared media, re-routing) but pins
|
||||
nothing for the kv tier, which the fleet story centers on: two
|
||||
nodes' sqlite engines are trivially partitioned (two files = two
|
||||
pools, not one), and no document said so.
|
||||
|
||||
The deciding facts are in hand (Schrödinger's-code rule — none of
|
||||
these may wait on consumers): the POCs fixed every engine's native
|
||||
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
|
||||
advisory lock, staged window); ADR-001 §4 fixed that ops is a
|
||||
store-facade consumer with no backend side-door. What is missing is
|
||||
composition: how the store core drives all engines through one
|
||||
contract while owning GC state.
|
||||
|
||||
Alternatives considered for the fleet-state seam, rejected:
|
||||
|
||||
- **Put the fleet machinery on the public trait** (pin/window/table
|
||||
methods on `Backend`): every engine — including `mem` and `local` —
|
||||
must then implement SQL-shaped machinery it cannot express. Rejected.
|
||||
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
|
||||
row, backend-maintained): contradicts ADR-005's "backends never
|
||||
learn liveness"; and the arbitration decision (expiry compare,
|
||||
liveness consult) still belongs to the store, so the store ends up
|
||||
reaching into the engine's tx anyway. Rejected.
|
||||
- **A raw-SQL side-channel** (connection escape hatch the store core
|
||||
may use arbitrarily): turns the engine abstraction off for exactly
|
||||
the machinery (fleet GC) it most needs to be sound on. Rejected.
|
||||
The seam below is that escape hatch made honest — named clauses,
|
||||
contract-tested per engine, instead of raw access.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put
|
||||
|
||||
**The trait stays one trait.** Every engine implements the same method
|
||||
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
|
||||
What changes: the trait's I/O types are the crate-internal
|
||||
abstractions below, which each engine constructs its own way — the
|
||||
store core programs the abstraction, never the engine shape.
|
||||
|
||||
- **`get` returns a read cursor.** One abstraction, two body families:
|
||||
- **kv engines** (sqlite, postgres, mem): the cursor materializes
|
||||
the whole value (the engine's row read; bounded by the small-blob
|
||||
tier policy). Its range reads are in-memory slices; `len()` reads
|
||||
the row's `size` column.
|
||||
- **large engines** (`local`, pg-lo): the cursor streams; the medium
|
||||
is hidden in the cursor body — `local`'s is a pread loop over the
|
||||
committed file (immutable once committed); pg-lo's drives
|
||||
descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
|
||||
posture — pool-friendly, stateless per window; the
|
||||
tx-scoped-descriptor form is the fallback). `len()` reads the
|
||||
companion row or stats the file — metadata only, never content
|
||||
(ADR-008's discipline).
|
||||
- Both body families serve the ADR-006 range contract (`read_range`
|
||||
→ slice + slice digest) at one seam. The cursor is
|
||||
**store-internal — not public API**: store consumers see the
|
||||
facade's `(len, stream)` shapes (store-api.md) with the cursor
|
||||
staying behind them; the store's stream utility adapts engine
|
||||
cursor bodies, it does not expose them.
|
||||
- **`put` exists in two forms, selected by the store core before
|
||||
content flows; both land in the CAS entry shape and both
|
||||
participate in pin-before-publish (§2 decides who holds the tx).**
|
||||
- **Whole-value put** for the kv tier: the settled row write.
|
||||
POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
|
||||
`ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
|
||||
- **Staged put** for the large tier — ADR-003's
|
||||
stage-then-commit-rename promoted from behavior to a **named
|
||||
per-engine contract**: the engine hands out a *staging cursor*
|
||||
(`local`: the stage file under the store's stage-hygiene protocol;
|
||||
pg-lo: chunked `lowrite` statements over a tx the store core
|
||||
holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
|
||||
measured knee). Commit publishes the entry;
|
||||
every failure path converges on discard (store-api.md stage
|
||||
hygiene, per engine, exact-count tested — POC #3 A5's shape).
|
||||
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
|
||||
walks the engine's companion structure (rows / files / LO
|
||||
companion); `size` is metadata-only (ADR-008). The exact-count
|
||||
sweep-outcome gate (ADR-003) runs per engine.
|
||||
|
||||
**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
|
||||
`pg-lo` does **not** imply feature `postgres`. Each is an engine
|
||||
feature pulling the shared driver cluster (`tokio-postgres` +
|
||||
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
|
||||
node is coherent and supported. overview.md's dependency table pins
|
||||
the edge.
|
||||
|
||||
### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested
|
||||
|
||||
**Ownership: the store core owns all GC state — in-process pin and
|
||||
liveness maps *and* the fleet state's rows and locks. The public
|
||||
`Backend` trait stays exactly what ADR-003 says (opaque
|
||||
`hash → bytes`, namespace-blind; "plain content-addressed rows,
|
||||
nothing else" remains true of the public trait). Backends never learn
|
||||
liveness (ADR-005) — verbatim.**
|
||||
|
||||
The store core reaches the fleet machinery through a **crate-internal
|
||||
(not exported) engine-state companion trait**, implemented by the
|
||||
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
|
||||
`mem` excluded by construction (it cannot be expressed on them and
|
||||
they are never fleet engines). ADR-008's mechanism fixes *what*; the
|
||||
seam is *where* the mechanism's SQL lives. Clauses:
|
||||
|
||||
- `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
|
||||
`gc_window (key, visible_after)` at store construction (store-core
|
||||
call, engine-neutral API; dialect-specific SQL bodies). `owner`
|
||||
is the **store-instance id** (ADR-011's vocabulary; ADR-008's
|
||||
"node id" wording predates that split). The pin row's identity is
|
||||
`(key, owner)` — a dedup re-put by a different instance is a
|
||||
different pin row; protection composes by row (an entry is
|
||||
protected while any un-expired pin row exists), which is
|
||||
ADR-005's refcount semantics expressed in rows. The embedder's
|
||||
liveness table is *not* created by this seam — ADR-008: schema
|
||||
pinned by the embedder; the sweep reads it and never owns it.
|
||||
**Per-engine applicability (the test-gate shape):** sqlite —
|
||||
`ensure_gc_schema` (window table only, since the window staging
|
||||
is shared machinery per ADR-008 §2) + `window_stage`/
|
||||
`window_take` + the sweep-lock clause's single-node body; the
|
||||
pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
|
||||
bodies but are exercised by unit tests only — they are dead
|
||||
machinery on sqlite because no sqlite fleet is on offer (§3).
|
||||
postgres and pg-lo — all clause families, contract-tested,
|
||||
fleet-active.
|
||||
- `pin_commit(joint_tx, entries)` — the store core **holds the
|
||||
transaction**; the engine contributes its put mechanics (row
|
||||
shape, CAS clause, LO writes) and the pin-row inserts, and both
|
||||
commit together. This is the joint entry+pin transaction's owner,
|
||||
ruled: **the store holds the tx; the engine contributes mechanics
|
||||
to it.** The public `put` remains the plain, pin-free path — the
|
||||
store's normal put path routes through `pin_commit` wherever the
|
||||
engine is seam-capable and fleet state is active.
|
||||
- pg-lo correction: ADR-009 §2's "put = one transaction" is
|
||||
unchanged in content but its ownership statement is hereby
|
||||
corrected — the tx is held by the store core; the pg-lo engine
|
||||
contributes its staged-put mechanics and companion-row insert
|
||||
into it. The engine sees pins only as rows; never as semantics.
|
||||
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
|
||||
update. The store's pin guard owns the TTL/3 renewal schedule
|
||||
(ADR-008); the engine owns only the SQL.
|
||||
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
|
||||
executes ADR-008's delete-time arbitration (un-expired pin rows +
|
||||
embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
|
||||
keyed, at delete time); the store core receives the answer and
|
||||
applies the decision. If fleet clauses ever grow, they grow here —
|
||||
the seam is the named home, with contract tests per clause.
|
||||
- `window_stage(candidates)` / `window_take(now)` — the staged
|
||||
delete-window candidates. Per ADR-008 §2 the window table exists
|
||||
on **every** engine (sqlite included): staged candidates are
|
||||
engine rows; what differs by topology is where *liveness* lives
|
||||
(in-process maps for single-node engines, tables for fleets), not
|
||||
whether staging exists.
|
||||
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
|
||||
pg advisory lock (ADR-008) for the SQL engines it applies to; the
|
||||
store treats lock contention exactly as ADR-008 says — the second
|
||||
sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
|
||||
nothing deleted. (SQLite today has no fleet story — see §3 — so its
|
||||
clause body exists but is exercised by single-node tests only.)
|
||||
|
||||
Each clause is contract-tested per its applicability (the
|
||||
per-engine split above): the conformance suite of §1 grows the fleet
|
||||
clauses for postgres and pg-lo; `local` and
|
||||
`mem` are excluded by construction.
|
||||
|
||||
### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)
|
||||
|
||||
**A fleet constructor requires a pool-shared kv engine.** Postgres is
|
||||
the shipped engine that satisfies this. A sqlite-fleet mechanism does
|
||||
not exist and is not on offer (per ADR-007's redb precedent this is
|
||||
recorded as a non-option, not an open question); two nodes' sqlite
|
||||
engines over "one pool" are two pools — the partitioning failure, the
|
||||
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
|
||||
enforceable
|
||||
seam is the same as ADR-008 §3's: the constructor can require an
|
||||
explicit declaration; it cannot prove cross-node truth. The
|
||||
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
|
||||
and is closed under both tiers.
|
||||
|
||||
### 4. The ops pin-token contract's TTL race, resolved by the ownership rule
|
||||
|
||||
**The token *is* the pin — there is no second pin state.** Under fleet
|
||||
engines the put's pin is the expiry-carrying row (ADR-008); the token
|
||||
identifies the putter as the row's owner. The token itself is an
|
||||
**opaque id minted by the ops layer** over the pin row it just
|
||||
committed — the ops layer (not the store facade) owns the
|
||||
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
|
||||
a lookup by the renewal op; no schema is specced for it beyond that
|
||||
(it is a handle, not durable state — the durable state is the pin row
|
||||
the seam wrote). Concretely:
|
||||
|
||||
- **When the server-side `Pin` guard drops:** at response time. The
|
||||
put lands pinned (the joint tx guarantees the entry and its pin row
|
||||
commit together); the handler's in-process guard — the thing ADR-005
|
||||
pins *in-process* with — is released when the `blobs/put` response
|
||||
completes and the token is handed out. From that moment the pin
|
||||
row's expiry is the only protection, which is exactly why the
|
||||
renewal duty below exists. (If the handler kept its guard for the
|
||||
token's whole life, the TTL machinery would be dead code on the ops
|
||||
path — the row's expiry, not the guard's presence, is the fleet
|
||||
contract's liveness signal.)
|
||||
- The putter renews the pin by renewing the row — through ops it
|
||||
already speaks: a `blobs/have` probe whose handler, observing a
|
||||
valid token presented alongside the digest, calls the
|
||||
crate-internal `pin_renew(key, owner)` via the ops module's own
|
||||
store-core access (the ops module is in-crate — it crosses the
|
||||
crate-internal seam by construction, the same as the put handler
|
||||
holds the `Pin` guard; what never happens is a *consumer* reaching
|
||||
the seam. ADR-008's `has` remains a pure probe: the renewal is the
|
||||
ops layer's token-authenticated act on the pin row, not a `has`
|
||||
semantic), or a re-put of the same digest (CAS dedup; the re-put's
|
||||
joint tx upserts the key+owner pin row, renewing it). The renewal
|
||||
cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
|
||||
20 s at the default 60 s TTL.**
|
||||
- A putter pausing past TTL falls to ADR-008's expiry semantics
|
||||
verbatim: the row is abandoned and reaped; the backstop is
|
||||
delete-then-recover (ADR-005) — the digest is re-offered, the put
|
||||
is idempotent under CAS. The race resolves to a re-put, never to
|
||||
silent loss.
|
||||
- For the `local` engine the token is the in-process guard as ADR-005
|
||||
had it: nothing persists, nothing expires; the process crash is
|
||||
ADR-005's in-flight-put loss case with the same backstop. (The
|
||||
cross-wire token under `local` is the ops layer's in-memory handle
|
||||
to that guard — same mapping discipline, no expiry.)
|
||||
|
||||
ops-surface.md's hand-over contract gains the renewal line and the
|
||||
token-mapping note; nothing else changes.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The trait's I/O shapes are decided before code; engine tasks can be
|
||||
written without guessing signatures. The per-engine
|
||||
conformance-suite shape (POC #7's 10/10 gate as the norm) gives
|
||||
decomposition a uniform test gate across all five engines.
|
||||
- "Backends never learn liveness" survives the fleet mechanism
|
||||
intact; the store core owns GC state per ADR-005 exactly as
|
||||
written; ADR-008's mechanism lives in a named, tested seam instead
|
||||
of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
|
||||
else" stays true of the public trait.
|
||||
- The ops pin-token TTL race is resolved with no new ops; the
|
||||
backstop is already-validated semantics (delete-then-recover).
|
||||
- The kv tier's fleet-validity rule closes the constructor table
|
||||
under both tiers.
|
||||
|
||||
**Negative**
|
||||
|
||||
- A crate-internal companion trait is real machinery: six clause
|
||||
families on postgres/pg-lo (fleet-active, all contract-tested);
|
||||
sqlite carries only the window + lock clauses as fleet-relevant
|
||||
machinery (its pin clauses have unit-test bodies but no fleet mode
|
||||
on offer — §3). The seam bounds where fleet-GC growth lands; it
|
||||
does not shrink it.
|
||||
- Remote putters gain a real protocol duty (renew past 20 s;
|
||||
pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
|
||||
- The sqlite sweep-lock clause body exists but is exercised only by
|
||||
single-node tests until a sqlite-fleet posture is ever
|
||||
decided (it is not on offer — §3).
|
||||
|
||||
**Neutral**
|
||||
|
||||
- The `Backend` trait's method set is unchanged from ADR-003/008 —
|
||||
no public API change beyond what store-api.md already specced; the
|
||||
cursor/staging abstractions are crate-internal.
|
||||
- The staged-put contract already existed behaviorally (POC #3 A5);
|
||||
this ADR names it.
|
||||
- ADR-009's engine content is unchanged; only its tx-ownership
|
||||
statement is corrected here.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
|
||||
(tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
|
||||
liveness-ownership rule this ADR preserves), ADR-006 (the range
|
||||
contract the cursor serves), ADR-008 (the fleet mechanism whose
|
||||
state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
|
||||
correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
|
||||
(vocabulary + constructor table)
|
||||
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
|
||||
`docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
|
||||
`docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
|
||||
512 KiB knee, windowed gets)
|
||||
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
|
||||
extends); [store-api.md](../store-api.md) (stage hygiene, sweep
|
||||
semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
|
||||
(REQ-2, the fleet requirement §2/§3 serve)
|
||||
@@ -0,0 +1,225 @@
|
||||
# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large`
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The engine story (ADR-007/008/009) grew the physical layer faster than
|
||||
its vocabulary grew, and the pre-decomposition review found the drift
|
||||
is now self-contradictory in the anchor document itself:
|
||||
|
||||
1. **Node vs store instance.** requirements.md defines **Node** as
|
||||
"one running store instance" — then REQ-2 states "a replicator node
|
||||
may run multiple store instances (a fleet)" and **Fleet** as "two
|
||||
or more nodes over one shared pool". Under the written definition,
|
||||
one process hosting two instances is simultaneously one node (by
|
||||
definition) and two nodes (by REQ-2), and the GC arbitration
|
||||
domain ("node is the GC arbitration domain") is undefined for the
|
||||
exact topology ADR-008 exists for. ADR-010 fixes pin ownership;
|
||||
the pin's *owner id* and the node/instance distinction it needs
|
||||
are still unpinned vocabulary.
|
||||
2. **The "fs" tier name is now wrong.** The tier was named when its
|
||||
only engine was a filesystem (ADR-003's `local` layout). After
|
||||
ADR-009, half its engines are postgres Large Objects — the name
|
||||
describes one engine's medium, not the tier. Worse, "fs" collides
|
||||
with the dispatch language: "large content goes to fs" never said
|
||||
*why* — the tier's actual routing criterion is content length
|
||||
(ADR-003's size-threshold dispatch). The docs already speak the
|
||||
true name informally: store-api.md says "small tier"/"large tier".
|
||||
3. **mem's classification drifts.** backends-and-dispatch.md calls it
|
||||
a "backend" in one place and a "testing/utility tier" in another;
|
||||
ADR-003 lists it under "shipped backends … plus a testing tier";
|
||||
ADR-004 calls it an ephemeral tier and carves it out of the
|
||||
two-backend count. In truth it is neither a tier nor a separate
|
||||
backend: it implements exactly the kv tier's contract (including
|
||||
`size`, trivially), it is the contract-reference engine every
|
||||
other engine's conformance tests mirror, and "engine" names what
|
||||
it is (ADR-007 established that an engine is a concrete
|
||||
implementation of one tier's contract — requirements.md already
|
||||
defines the word this way).
|
||||
4. **Tier and Backend were never formally defined** even though
|
||||
ADR-004's scope note ("the 'two' here counts tiers, not engines")
|
||||
leans on both words. Also undecided: whether the feature names
|
||||
follow the tier rename now or after code exists — the last free
|
||||
moment is now (zero code; after implementation the rename breaks
|
||||
crates.io/feature ergonomics for every consumer).
|
||||
|
||||
Deciding facts are in hand: REQ-2's fleet fact, the engine set's
|
||||
final shape (ADR-007/009), the dispatch mechanism (ADR-003), and
|
||||
ADR-010's GC-state-home table. This is a vocabulary-and-boundary
|
||||
decision, not an architecture change: ADR-004's tier≠engine note
|
||||
already established the shape; this ADR gives it final, consistent
|
||||
names.
|
||||
|
||||
Candidates weighed for the tier rename:
|
||||
|
||||
- **`lo`/`los`** — names the *postgres* engine (Large Objects), so
|
||||
"the lo tier's `local` engine" would be backwards. Rejected.
|
||||
- **`lfs`** — workable, echoes git-lfs specifically; but the tier
|
||||
also serves packfiles and any oversized workspace artifact, and the
|
||||
echo binds the crate's naming to one consumer's protocol shape.
|
||||
Rejected.
|
||||
- **`large`** — the dispatch-true name (size-threshold routing), the
|
||||
word the specs already use informally ("large tier"), medium-agnostic.
|
||||
**Chosen.** For symmetry in prose, the kv tier stays `kv` (its
|
||||
routing criterion — "small" — lives in the prose as "small blobs"
|
||||
and needs no rename: `kv` names the mechanism class, and `small`
|
||||
as a feature/tier name would read as an adjective; the pair's
|
||||
asymmetry is cosmetic, the *load-bearing* correction is that
|
||||
neither tier is named after a medium).
|
||||
|
||||
Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it
|
||||
names the engine's mechanism, accurate); `postgres` keeps its name
|
||||
(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep
|
||||
their names.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy)
|
||||
|
||||
- **Tier** — one *contract* (a `Backend` trait implementation
|
||||
boundary) serving one routing class of the pool. The crate ships
|
||||
exactly two tiers: **kv** (small content) and **large** (over
|
||||
the size-threshold content). Tier count (2) is ADR-004's count.
|
||||
- **Engine** — a concrete storage implementation of one tier's
|
||||
contract (the existing definition, unchanged). Engine count (5) is
|
||||
this ADR's count.
|
||||
- **Backend** — *the* contract (the `Backend` trait) plus its shipped
|
||||
engine set, i.e. the tier as shipped. "Backend" and "tier" name the
|
||||
same thing at different altitudes; the crate says "tier" for
|
||||
counting, "the `Backend` trait" for the contract. "Two backends"
|
||||
reads forever as "two tiers".
|
||||
- **Store instance** — one running instantiation of the store facade
|
||||
(one engine set — one or two engines per the constructor mode, §4 —
|
||||
plus dispatch and GC state). **The GC
|
||||
arbitration domain** for single-instance topologies; the pin-row
|
||||
owner id under fleet engines (ADR-010's `owner`; this corrects
|
||||
ADR-008's "node id" wording, which predates the instance/node
|
||||
split — a store instance is what owns pins).
|
||||
- **Node** — one running *process* (an embedder's binary). A node
|
||||
hosts **one or more store instances**. Node is the process/ops unit:
|
||||
alkcall connections, ACL grants, sweep cadence, and deployment
|
||||
identity attach to nodes; GC arbitration and pin ownership attach
|
||||
to store instances.
|
||||
- **Fleet** — **two or more store instances over one shared pool**
|
||||
(via a shared SQL engine). Deliberately not defined by node count:
|
||||
two instances in one process over one postgres pool are a fleet
|
||||
under this definition (and REQ-2's phrasing "a replicator node may
|
||||
run multiple store instances (a fleet)" reads true as written);
|
||||
two processes on two sqlite files are two pools, never a fleet.
|
||||
- **Deployment** — unchanged (one operator's whole system).
|
||||
|
||||
### 2. mem joins the kv tier as its third engine
|
||||
|
||||
`mem` is the **kv tier's ephemeral engine** (feature `mem`,
|
||||
default-off): a `BTreeMap`-shaped implementation of the kv contract,
|
||||
including `size` (trivially) and list; never fleet-valid (ADR-010 §3's
|
||||
rule and this ADR §4's constructor table); the **contract-reference
|
||||
engine** — every other engine's
|
||||
conformance tests mirror its suite. It is not a tier, not a third
|
||||
backend, and the ADR-004 count ("two") survives untouched because
|
||||
tier and engine were never the same count — this ADR is that
|
||||
distinction's final form.
|
||||
|
||||
### 3. The fs tier is renamed `large`
|
||||
|
||||
In every document, status line, feature name, and constructor
|
||||
parameter: tier `fs` → tier `large`; feature `fs` → feature `large`
|
||||
(default-on, as `fs` was); dispatch prose says "large tier"/"small
|
||||
tier" as store-api.md already does; the `local` engine's name is
|
||||
kept (its meaning — this node's own filesystem — stays the
|
||||
constructor's meaning even under the shared-media fleet posture,
|
||||
where it is explicitly *not* only-node-local; the constructor's
|
||||
explicit shared-media assertion, ADR-008 §3, is what disambiguates).
|
||||
|
||||
### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here)
|
||||
|
||||
| Tier | Engine | Feature | Default | Fleet-valid |
|
||||
|---|---|---|---|---|
|
||||
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
|
||||
| kv | postgres | `postgres` | no | **yes** |
|
||||
| kv | mem | `mem` | no | no |
|
||||
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
|
||||
| large | pg-lo | `pg-lo` | no | **yes** |
|
||||
|
||||
Constructor modes (dispatch shapes) — pinned here (implied by
|
||||
ADR-008 §3's re-routing bullet and the mem tests, but never named):
|
||||
- **dual-tier** (the default): kv + large, size-threshold dispatch
|
||||
between them (ADR-003).
|
||||
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
|
||||
constructor mode with no large tier; over-threshold puts are a
|
||||
constructor-time error (the client-node re-routing posture handles
|
||||
them via the ops surface instead — its composition, not a store
|
||||
mode).
|
||||
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
|
||||
no dispatch threshold in effect (everything is local and small);
|
||||
a testing posture, explicitly not a production story.
|
||||
- Single-tier SQL modes (postgres-only, pg-lo-only) are **not**
|
||||
constructor modes this crate ships: the two-tier shape is
|
||||
ADR-004's requirement (both tiers are load-bearing by measured
|
||||
economics); "postgres-only" as in *one SQL instance serving both
|
||||
tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
|
||||
consolidation), which is the dual-tier mode over one engine.
|
||||
|
||||
### 5. Deferral-policy wording (README alignment)
|
||||
|
||||
The deferral policy's categories are: `deferred(scope)`,
|
||||
externally-owned questions, and decided-but-sequenced work — three
|
||||
names, defined in open-questions.md's header. README.md's recap says
|
||||
"two parking kinds" and is corrected to name all three (the third is
|
||||
not a *parking* kind — it is work, not a parked question — hence
|
||||
this ADR's wording here rather than a silent edit).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The anchor document's self-contradiction (node/fleet) becomes a
|
||||
three-level vocabulary (store instance / node / deployment) whose
|
||||
levels map one-to-one onto the mechanisms (GC domain / process-ops
|
||||
unit / operator whole).
|
||||
- The fs→large rename removes the last medium-named tier; feature
|
||||
names, tier names, and dispatch prose now agree, before any code
|
||||
exists to break.
|
||||
- mem's classification is stated once, as the contract-reference kv
|
||||
engine; the "plus a testing tier" carve-outs dissolve.
|
||||
- The constructor table gives task decomposition a closed set: 5
|
||||
engine implementations, 3 constructor modes, 1 dispatch policy.
|
||||
|
||||
**Negative**
|
||||
|
||||
- Vocabulary churn in every doc that says "fs" (the rename is
|
||||
mechanical but touches all specs and ADR-003/004/007/008/009 status
|
||||
lines and reference sections). Zero code exists — this is the last
|
||||
free moment; after Phase 1 starts, feature renames break consumers.
|
||||
- "Backend"/"tier" as interchangeable words risks reader confusion
|
||||
with ADR-003's "Backend trait" — mitigated by the rule that the
|
||||
*trait* is always named with "trait" ("the `Backend` trait"), never
|
||||
bare "the backend".
|
||||
|
||||
**Neutral**
|
||||
|
||||
- No engine's contract, feature flag semantics, or constructor
|
||||
parameter changes; `mem` gains the word "engine" and loses the
|
||||
word "backend".
|
||||
- REQ-2's meaning is unchanged (the fleet definition is re-derived
|
||||
from the same fact, not re-fact'd).
|
||||
|
||||
## References
|
||||
|
||||
- ADR-003 (the trait contract; its "shipped backends" list is now
|
||||
read through this ADR's vocabulary), ADR-004 (tier≠engine — the
|
||||
scope note this ADR gives final names), ADR-007 (kv engines;
|
||||
constructor selection), ADR-008 (the second tier's engine candidate
|
||||
and the fleet topology; the vocabulary it assumed is now pinned),
|
||||
ADR-009 (pg-lo admission; the consolidation option whose
|
||||
constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
|
||||
(the I/O seams and GC-state home this vocabulary must fit)
|
||||
- [requirements.md](../requirements.md) (the canonical vocabulary
|
||||
copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
|
||||
(the tier/engine table's spec home)
|
||||
- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the
|
||||
reference-impl role mem already played in the POCs);
|
||||
`poc-pglo-findings.md` (the gate the engine suite mirrors)
|
||||
Reference in new issue
Block a user