docs(architecture): ADR-012 — pre-decomposition consistency rulings; tree verified single-valued

Full-tree review before task decomposition found five composition
defects (mechanisms specced correctly in isolation, composition
unruled) and a set of caller-facing gaps. ADR-012 rules each:

- Sweeps are never errors: aborts are GcAbortCause report data in
  Ok(SweepReport) (ProtectFailed / SweeperLock / NoLivenessSources);
  GcAborted retired from the error enum; direct-delete refusal is
  the GcRefuse error covering the full protection set
- Pin token gains its wire shape: blobs/put response {token, digest};
  blobs/have's token-renewal form (one digest + token); token
  validity domain = the minting serving node, process-lifetime
  mapping
- Fleet mode is an explicit constructor declaration (fleet: true),
  never inferred from engine choice
- All fleet GC state hosts on the fleet's kv engine (postgres) — one
  arbitration domain; large=pg-lo fleet nodes required onto the same
  pg instance (composite predicate's new clause); large=local fleet
  puts pin-row-first, publish-second
- Engine-state seam reduced: sqlite pin/sweep-lock bodies dropped
  (dead machinery); non-SQL engines stage delete-window candidates
  in-process; one-window-host rule per instance
- Facade clarifications: fall-through for all key-addressed ops,
  kv-only put-time rejection, mem+local dual-tier valid, error-model
  member/return-shape ruling (trait/facade family split), has ->
  bool, PinState variants, fleet liveness-table registration form,
  window executor = the next sweep

Alignment edits across all specs and ADR-005/008/009/010/011
(bracketed corrections per the established pattern); OQ-11 (pg-only-kv
feature graph) added to the parked index for auditability.

Verification: two independent review passes; all findings resolved;
verdict READY for task decomposition.
This commit is contained in:
glm-5.3-flash committed 2026-10-03 07:14:45 +00:00
1 parent abf8c5ae9c
commit 7b9d904b8a
13 files changed
+771 -184

No files matched your search

@@ -8,7 +8,13 @@ advisory-locked single sweeper, SQL-level delete arbitration under
shared-pool engines (REQ-2) — while keeping this ADR's invariants
and the in-process protocol unchanged everywhere else. The "one pool
per node" phrasing below reads as "per deployment" for fleets; the
vocabulary is pinned in requirements.md)
vocabulary is pinned in requirements.md. ADR-012 rules the abort
*carrier*: sweep aborts — including the protect callback's — are
`GcAbortCause` report data in `Ok(SweepReport)`, never a typed sweep
error (this ADR's "typed `GcAborted`" phrasing reads through that
ruling), and direct `delete()`'s refusal covers the full protection
set (this ADR's "pinned keys" wording reads as ADR-012 §6.4's
"protected keys"))
## Context
@@ -73,9 +79,11 @@ served by range reads).
observes the replacement (the local analog of the ops-layer pin
token contract).
3. **Protect callback** — consulted pre-sweep; may add known-live keys
or **abort the run** (typed `GcAborted`, nothing deleted). iroh's
`ProtectOutcome::Abort` conclusion adopted: a flaky protection
source skips the sweep rather than risk deletion.
or **abort the run** (nothing deleted; the abort is report data —
`GcAbortCause::ProtectFailed` in `Ok(SweepReport)` per ADR-012
§1, not a typed error; this ADR's original "typed `GcAborted`"
phrasing is superseded on the carrier only — the semantics iroh's
`ProtectOutcome::Abort` conclusion adopted stand).
### Sweep: mark → delete window → commit
@@ -133,10 +141,14 @@ served by range reads).
existence flips atomically per key).
- **Delete-then-recover is a re-put** (validated: byte-identical under
the same key); no tombstone layer exists.
- **Direct `delete(key)`:** refuses pinned keys with a typed error
(same arbitration); permitted for embedder correction flows on
unpinned entries. Outside-of-sweep deletion is unusual by posture —
most deletion should flow through sweeps.
- **Direct `delete(key)`:** refuses protected keys (pinned,
registered-source-live, or protect-callback-named — the full
protection set, ADR-012 §6.4; this ADR's original "refuses pinned
keys" wording is superseded upward; the refusal is the typed
`GcRefuse` error, ADR-012 §1) — same delete-time arbitration;
permitted for embedder correction flows on unprotected entries.
Outside-of-sweep deletion is unusual by posture — most deletion
should flow through sweeps.
### Traversal and scheduling
@@ -202,6 +214,6 @@ OQ-07's.
gix-odb (alternates/pool prior art)
- ADR-002 (one address space), ADR-003 (list contract, pins), ADR-004
(manifest layers above), ADR-006 (verification limits inform recover
semantics)
semantics), ADR-012 (abort carrier + direct-delete protection set)
- [gc-and-namespaces.md](../gc-and-namespaces.md);
[store-api.md](../store-api.md)
@@ -8,7 +8,9 @@ passed gate — the two §Consequences passages that said pg-lo "remains
unbuilt" are resolved by that same supersession; the tier this ADR's
title and text call "fs" is renamed `large` by ADR-011; the fleet GC
mechanism's state home and the joint-tx ownership are ruled by
ADR-010; every other part of this ADR stands as written)
ADR-010; fleet-mode activation, the GC-state host, the sweep-abort
return shape, and the window executor are ruled by ADR-012; every
other part of this ADR stands as written)
## Context
@@ -105,13 +107,16 @@ Under a fleet, the mechanism moves into the shared engine:
single-node/topology-local sweeps that cannot express their
protection as rows.
- **One sweeper per pool.** A fleet node acquires a pg advisory lock
before sweeping and holds it for the whole sweep. Two nodes
sweeping concurrently is not a state this crate tolerates — the
lock is the admission gate, and a second-node sweep returns the
typed `GcAborted` (no deletion) rather than racing. `GcAborted`
thus carries two cause classes (protection-source failure —
ADR-005; sweeper lock contention — this ADR); the typed error's
fleet variant identifies which.
before sweeping and holds it for the whole sweep. Two nodes
sweeping concurrently is not a state this crate tolerates — the
lock is the admission gate, and a second-node sweep returns the
typed `GcAborted` (no deletion) rather than racing. `GcAborted`
thus carries two cause classes (protection-source failure —
ADR-005; sweeper lock contention — this ADR); the typed error's
fleet variant identifies which. *(Return carrier corrected by
ADR-012 §1: the abort is `GcAbortCause::SweeperLock` report data —
`Ok(report)` — the typed sweep error is retired; the two cause
classes and the no-deletion semantics stand.)*
- **Delete-time arbitration is SQL-level.** The delete window's
re-check (ADR-005: "arbitrate under the pin/liveness lock at delete
time") executes as `SELECT … FOR UPDATE` against pin rows and
@@ -120,22 +125,30 @@ Under a fleet, the mechanism moves into the shared engine:
as rows carrying `visible_after = mark_end + window` (the delete
window's delay — default 5 min, constructor-tunable per
deployment); the executor (the sweeper, or the next sweep's owner)
re-arbitrates each staged candidate *at execution time* against
pins and liveness tables before deleting. This is the durable
delete window on every engine — the window table is engine rows on
sqlite as well; what differs by topology is only where the liveness
state lives (in-process maps and callbacks for single nodes;
tables and SQL arbitration for fleets), not whether staging exists.
No external queue dependency is required (the pattern is
engine-native transactions; consumer-side queues like honker or
pg-boss-lineage tools remain consumer-layer options, per ADR-004's
boundary).
re-arbitrates each staged candidate *at execution time* against
pins and liveness tables before deleting. This is the durable
delete window on every SQL-backed engine — the window table is
engine rows on
sqlite as well; what differs by topology is only where the liveness
state lives (in-process maps and callbacks for single nodes;
tables and SQL arbitration for fleets), not whether staging exists.
*(ADR-012 §5/§6.7: "every SQL-backed engine" is the final wording
— `local`/`mem` stage in-process, window-inline; the executor of
overdue staged rows is the next sweep.)* No external queue
dependency is required (the pattern is
engine-native transactions; consumer-side queues like honker or
pg-boss-lineage tools remain consumer-layer options, per ADR-004's
boundary).
- **Single-node postures are unchanged.** sqlite and single-instance
postgres keep ADR-005's in-process protocol verbatim: the pin map,
RAII guards, and arbitration lock stay in-process; only the
*persistence* of the delete window (the staged-candidate table)
is shared machinery. The pin-row expiry/ttl machinery exists only
where fleet engines are active.
postgres keep ADR-005's in-process protocol verbatim: the pin map,
RAII guards, and arbitration lock stay in-process; only the
*persistence* of the delete window (the staged-candidate table) is
shared machinery. The pin-row expiry/ttl machinery exists only
where fleet engines are active. *(Activation is explicit: ADR-012
§3's `fleet: true` constructor declaration turns the fleet
machinery on; it is never inferred from engine choice alone — a
single-instance postgres declared `fleet: false` keeps the
in-process protocol.)*
### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible
@@ -166,7 +179,9 @@ valid:
(verified fetch/put; ADR-001). Client nodes' constructors are
configured with **kv-only mode** (ADR-011 §4's name for what this
bullet first called "kv-tier-only dispatch") — no local large tier
over pool content at all; a client node's *node-private,
over pool content at all; over-threshold puts reject at put time
(ADR-012 §6.2), and the re-routing posture handles them via the ops
surface; a client node's *node-private,
consumer-owned* large-tier content is outside the pool and outside
this crate's GC entirely: it is the consumer's own storage beside
the pool per ADR-004, not a constructor mode over pool content, and
@@ -176,23 +191,21 @@ valid:
a client node is its own store for sub-threshold content plus its
registered ops-put path for large content — a consumer-side
composition, not a store-level mode.
- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects
- **`pg-lo` (shipped; admitted by ADR-009):** postgres Large Objects
as the large tier's storage —
the same "engine behind one contract"
move as ADR-007, one tier over. Originally named as a candidate,
admission gated on measured evidence (POC #7: LO write/read
curves at the packfile regime, tx-scoped handle cost under
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
behavior) and its own engine ADR. NOT shipped speculatively;
REQ-2's immediate fleet answers are shared media or re-routing;
pg-lo is the consolidation option if those are unacceptable. *Update
2026-10-03: POC #7 ran and passed its gate —
`docs/research/poc-pglo-findings.md` (companion-table authority,
move as ADR-007, one tier over. Originally named here as a candidate
gated on measured evidence, POC #7 ran and passed that gate
(`docs/research/poc-pglo-findings.md`: companion-table authority,
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s
aggregate at 16 readers). Superseded the same day: ADR-009 admitted
the engine — this bullet's original "candidate, not shipped"
posture is historical; the engine is shipped, feature `pg-lo`.*
aggregate at 16 readers), and ADR-009 admitted the engine the same
day — this bullet's original "candidate, not shipped / REQ-2's
immediate answers are shared media or re-routing; pg-lo the
consolidation option" posture is historical; the engine is shipped,
feature `pg-lo`, and it is the fleet consolidation option as a
first-class engine (ADR-012 §4 adds the same-instance requirement
under fleet mode).
- **The no-mixed-large-tiers rule is a deployment invariant, enforced at
the enforceable seam.** Cross-node configuration cannot be validated
by any one constructor (it sees only its own node). What the
@@ -240,8 +253,12 @@ partitioning failure this decision exists to prevent.
handle), advisory-lock coordination, and SQL-level arbitration
added on top of the already-intricate delete-window state machine.
- `pg-lo` was built under ADR-009 after POC #7 passed — the
measured, deliberate sequence of the ADR-006 pattern (admission
gate first, engine second), not a hedge; resolved same-day.
measured, deliberate sequence of the ADR-006 pattern (admission
gate first, engine second), not a hedge; resolved same-day.
- *(ADR-012 §4: fleet GC state hosts on the fleet's kv engine — the
large tier's `local` engine never joins the pin transaction;
large=local fleet puts commit the pin row first, publish
(commit-rename) second.)*
**Neutral**
@@ -7,7 +7,10 @@ Accepted (admission evidence: POC #7, passed 2026-10-03 —
gate. The tier this ADR calls "fs" is renamed `large` by ADR-011);
§2's "put = one transaction" tx-ownership statement is corrected by
ADR-010 §2 (the store core holds the tx; the engine contributes its
staged-put mechanics into it — engine content unchanged))
staged-put mechanics into it — engine content unchanged); ADR-012 §4
adds the same-pg-instance requirement for fleet-mode pg-lo nodes
(the consolidation posture this ADR described is the fleet-valid
shape).
## Context
@@ -56,8 +59,12 @@ consolidation option as a shipped engine.
2. **Put = one transaction**: BEGIN → `lo_create` → chunked `lowrite`
(512 KiB statements — the measured knee; statement count, not
LOBLKSIZE, is what costs) → companion-row insert → COMMIT.
Pin-before-publish (ADR-005/008) rides this shape: entry row + pin
row commit together with the content. Unknown-length puts stage
*(Ownership corrected by ADR-010 §2: the store core holds this tx;
the engine contributes LO writes + the companion-row insert into
it.)* Pin-before-publish (ADR-005/008) rides this shape: entry row + pin
row commit together with the content — the pin-row insert is the
store's contribution (hosted on the fleet's kv engine under fleet
mode, ADR-012 §4 — same pg instance, one tx). Unknown-length puts stage
into the LO and commit the row on completion; rollback discards
both.
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
@@ -100,7 +107,7 @@ consolidation option as a shipped engine.
many-user aggregate. A third engine (S3-like) remains the
ADR-003 door; nothing here touches it.
7. **redb-class guardrail maintained:** this is the second engine the
admission door ships (ADR-007: kv; ADR-009: fs) — both on the same
admission door ships (ADR-007: kv; ADR-009: large) — both on the same
measured-evidence shape; no third engine is opened by this one.
## Consequences
@@ -2,7 +2,12 @@
## Status
Accepted
Accepted (fleet-state activation, the GC-state host under mixed
engines, the sweep-abort return shape, the pin token's wire shape,
the sqlite pin/sweep-lock clause reductions, and the window staging
for non-SQL engines are ruled by [ADR-012](012-pre-decomposition-consistency-rulings.md);
§2's sqlite rows and the "fleet state is active" predicate read
through that ADR — where noted inline)
## Context
@@ -139,7 +144,10 @@ The store core reaches the fleet machinery through a **crate-internal
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
`mem` excluded by construction (it cannot be expressed on them and
they are never fleet engines). ADR-008's mechanism fixes *what*; the
seam is *where* the mechanism's SQL lives. Clauses:
seam is *where* the mechanism's SQL lives. Clauses (per-engine
applicability is ruled by [ADR-012](012-pre-decomposition-consistency-rulings.md)
§5's table, which reduces the original sqlite rows; fleet state hosts
on the kv-tier engine when fleet mode is declared — ADR-012 §4):
- `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
`gc_window (key, visible_after)` at store construction (store-core
@@ -152,15 +160,19 @@ seam is *where* the mechanism's SQL lives. Clauses:
ADR-005's refcount semantics expressed in rows. The embedder's
liveness table is *not* created by this seam — ADR-008: schema
pinned by the embedder; the sweep reads it and never owns it.
**Per-engine applicability (the test-gate shape):** sqlite —
`ensure_gc_schema` (window table only, since the window staging
is shared machinery per ADR-008 §2) + `window_stage`/
`window_take` + the sweep-lock clause's single-node body; the
pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
bodies but are exercised by unit tests only — they are dead
machinery on sqlite because no sqlite fleet is on offer (§3).
postgres and pg-lo — all clause families, contract-tested,
fleet-active.
**Per-engine applicability (the test-gate shape; final per
ADR-012 §5):** sqlite — schema creates **`gc_window` only** +
`window_stage`/`window_take` (durable single-node staging);
the originally-specced sqlite pin clauses and sweep-lock body
are **dropped** by ADR-012 §5 (the pin bodies inserted into a
table nothing created — dead machinery; sqlite has no advisory
lock, and ADR-008's unchanged single-node protocol keeps the
sweep single-flighter in-process). postgres — all clause
families, contract-tested, fleet-active (the fleet host). pg-lo —
the window family only (durable window staging where the kv
engine is not SQL-backed — the one-window-host rule, ADR-012 §5;
it hosts no fleet pin/lock state,
ADR-012 §4), contract-tested.
- `pin_commit(joint_tx, entries)` — the store core **holds the
transaction**; the engine contributes its put mechanics (row
shape, CAS clause, LO writes) and the pin-row inserts, and both
@@ -168,37 +180,59 @@ seam is *where* the mechanism's SQL lives. Clauses:
ruled: **the store holds the tx; the engine contributes mechanics
to it.** The public `put` remains the plain, pin-free path — the
store's normal put path routes through `pin_commit` wherever the
engine is seam-capable and fleet state is active.
engine is seam-capable and fleet mode is active (**activation
ruled by ADR-012 §3: fleet mode is an explicit `fleet: true`
constructor declaration, never inferred from the engine choice**;
for large=local fleet puts the joint tx is unachievable across
media — ADR-012 §4 pins pin-row-first, publish-second ordering).
- pg-lo correction: ADR-009 §2's "put = one transaction" is
unchanged in content but its ownership statement is hereby
corrected — the tx is held by the store core; the pg-lo engine
contributes its staged-put mechanics and companion-row insert
into it. The engine sees pins only as rows; never as semantics.
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
update. The store's pin guard owns the TTL/3 renewal schedule
(ADR-008); the engine owns only the SQL.
- `pin_renew(key, owner)` — the engine executes the expiry update
(renamed and TTL dropped by ADR-012 §2: the TTL lives with the
store's renewal schedule, not in the clause signature; the original
`pin_renew(key, owner, ttl)` read is superseded). The store's pin
guard owns the TTL/3 renewal schedule
(ADR-008); the engine owns only the SQL. *(postgres host only per
ADR-012 §5 — sqlite bodies dropped.)*
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
executes ADR-008's delete-time arbitration (un-expired pin rows +
embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
keyed, at delete time); the store core receives the answer and
applies the decision. If fleet clauses ever grow, they grow here —
applies the decision. `PinState` is a data enum with two variants:
`Protected` (an un-expired pin row exists) and `Unprotected` — the
engine never decides deletion, it only classifies; *when the seam
supports SQL-level liveness tables the store consults those through
`arbitrate` instead, which folds the full answer into the bool*. If
fleet clauses ever grow, they grow here —
the seam is the named home, with contract tests per clause.
- `window_stage(candidates)` / `window_take(now)` — the staged
delete-window candidates. Per ADR-008 §2 the window table exists
on **every** engine (sqlite included): staged candidates are
delete-window candidates. Per ADR-008 §2 the durable window table
exists on **every SQL-backed engine** (final wording per ADR-012
§5: sqlite included; `local` and `mem` are excluded from the seam
here and stage candidates in-process, executing the ADR-005 window
protocol inline in the same sweep; in dual-SQL instances the
non-host engine's window clauses are dormant — the one-window-host
rule, ADR-012 §5): staged candidates are
engine rows; what differs by topology is where *liveness* lives
(in-process maps for single-node engines, tables for fleets), not
whether staging exists.
whether staging exists among SQL-backed engines.
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
pg advisory lock (ADR-008) for the SQL engines it applies to; the
pg advisory lock (ADR-008), applying to the fleet host (postgres)
only — ADR-012 §5 drops the originally-specced sqlite body (no
advisory-lock equivalent; in-process lock suffices and is
ADR-008-verbatim). The
store treats lock contention exactly as ADR-008 says — the second
sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
nothing deleted. (SQLite today has no fleet story — see §3 — so its
clause body exists but is exercised by single-node tests only.)
sweeper's sweep aborts with **`GcAbortCause::SweeperLock` as report
data (`Ok(report)` — ADR-012 §1 retired the typed `GcAborted` sweep
error)**, nothing deleted.
Each clause is contract-tested per its applicability (the
per-engine split above): the conformance suite of §1 grows the fleet
clauses for postgres and pg-lo; `local` and
clauses for postgres (the host); pg-lo runs the window-family
contract tests; `local` and
`mem` are excluded by construction.
### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)
@@ -222,10 +256,19 @@ engines the put's pin is the expiry-carrying row (ADR-008); the token
identifies the putter as the row's owner. The token itself is an
**opaque id minted by the ops layer** over the pin row it just
committed — the ops layer (not the store facade) owns the
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
a lookup by the renewal op; no schema is specced for it beyond that
token↔(key, owner) mapping, in-process state with **process lifetime**
(ADR-012 §2 corrected the original "handler's lifetime" — the handler's
guard drops at response time, but the mapping must outlive it, or
renewal could never arrive); no schema is specced
for it beyond that
(it is a handle, not durable state — the durable state is the pin row
the seam wrote). Concretely:
the seam wrote). The token's **validity domain is the minting node**
(ADR-012 §2: renewal probes reach the serving node that minted the
token; fleet peers refuse foreign tokens; a dead minting node is the
ADR-005 crash case with the re-put backstop).
**Wire shapes (ADR-012 §2):** the `blobs/put` response is
`{token, digest}`; renewal rides the token form of `blobs/have`
(one digest, token alongside). Concretely:
- **When the server-side `Pin` guard drops:** at response time. The
put lands pinned (the joint tx guarantees the entry and its pin row
@@ -259,7 +302,10 @@ the seam wrote). Concretely:
had it: nothing persists, nothing expires; the process crash is
ADR-005's in-flight-put loss case with the same backstop. (The
cross-wire token under `local` is the ops layer's in-memory handle
to that guard — same mapping discipline, no expiry.)
to that guard — same mapping discipline, no expiry.) ADR-012 §2
generalizes this to every non-fleet mode: the token is the
in-process guard's handle on sqlite, single-instance postgres, and
pg-lo serving nodes too — nothing expires outside fleet mode.
ops-surface.md's hand-over contract gains the renewal line and the
token-mapping note; nothing else changes.
@@ -284,17 +330,15 @@ token-mapping note; nothing else changes.
**Negative**
- A crate-internal companion trait is real machinery: six clause
families on postgres/pg-lo (fleet-active, all contract-tested);
sqlite carries only the window + lock clauses as fleet-relevant
machinery (its pin clauses have unit-test bodies but no fleet mode
on offer — §3). The seam bounds where fleet-GC growth lands; it
does not shrink it.
- A crate-internal companion trait is real machinery: the full clause
family on postgres (the fleet host — pins, window, sweep lock,
contract-tested, fleet-active); sqlite and pg-lo carry the window
family only (ADR-012 §5's reduction — the original draft's sqlite
pin/sweep-lock bodies were dead machinery and are dropped). The
seam bounds where fleet-GC growth lands; it does not shrink it.
- Remote putters gain a real protocol duty (renew past 20 s;
pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
- The sqlite sweep-lock clause body exists but is exercised only by
single-node tests until a sqlite-fleet posture is ever
decided (it is not on offer — §3).
pause-past-TTL ⇒ delete-then-recover; route renewal to the minting
node — ADR-012 §2) — documented, but a duty.
**Neutral**
@@ -314,7 +358,10 @@ token-mapping note; nothing else changes.
contract the cursor serves), ADR-008 (the fleet mechanism whose
state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
(vocabulary + constructor table)
(vocabulary + constructor table), [ADR-012](012-pre-decomposition-consistency-rulings.md)
(fleet activation predicate, GC-state host, sweep-abort shape,
token wire shape, the §5 clause reduction this ADR's seam table
now references)
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
`docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
`docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
@@ -2,7 +2,12 @@
## Status
Accepted
Accepted (fleet-mode activation, the GC-state host, the sweep-abort
shape, and kv-only's put-time rejection point are ruled by
[ADR-012](012-pre-decomposition-consistency-rulings.md) — this ADR's
constructor table is closed under ADR-012 §4's host rule; §4's
"constructor-time error" wording reads as ADR-012 §6.2's put-time
rejection)
## Context
@@ -147,12 +152,17 @@ explicit shared-media assertion, ADR-008 §3, is what disambiguates).
Constructor modes (dispatch shapes) — pinned here (implied by
ADR-008 §3's re-routing bullet and the mem tests, but never named):
- **dual-tier** (the default): kv + large, size-threshold dispatch
between them (ADR-003).
between them (ADR-003). Engine pairs beyond the default
(`kv = mem` + `large = local`) are valid dual-tier constructions —
ADR-012 §6.3.
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
constructor mode with no large tier; over-threshold puts are a
constructor-time error (the client-node re-routing posture handles
them via the ops surface instead — its composition, not a store
mode).
constructor mode with no large tier; over-threshold puts reject at
**put time** — known-length immediately, unknown-length at
threshold overflow (the original "constructor-time error" wording
is corrected by ADR-012 §6.2: lengths are not known at
construction). The client-node re-routing posture handles
over-threshold content via the ops surface instead — its
composition, not a store mode.
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
no dispatch threshold in effect (everything is local and small);
a testing posture, explicitly not a production story.
@@ -161,7 +171,9 @@ ADR-008 §3's re-routing bullet and the mem tests, but never named):
ADR-004's requirement (both tiers are load-bearing by measured
economics); "postgres-only" as in *one SQL instance serving both
tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
consolidation), which is the dual-tier mode over one engine.
consolidation), which is the dual-tier mode over one engine —
under fleet mode that one instance is a requirement, not a
coincidence (ADR-012 §4).
### 5. Deferral-policy wording (README alignment)
@@ -216,7 +228,10 @@ this ADR's wording here rather than a silent edit).
and the fleet topology; the vocabulary it assumed is now pinned),
ADR-009 (pg-lo admission; the consolidation option whose
constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
(the I/O seams and GC-state home this vocabulary must fit)
(the I/O seams and GC-state home this vocabulary must fit),
[ADR-012](012-pre-decomposition-consistency-rulings.md) (fleet-mode
activation + GC-state host + kv-only put-time rejection — the
constructor table's final closure)
- [requirements.md](../requirements.md) (the canonical vocabulary
copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
(the tier/engine table's spec home)
@@ -0,0 +1,352 @@
# ADR-012: Pre-decomposition consistency rulings — sweep-abort shape, pin-token wire shape, fleet-mode activation, and the GC-state host
## Status
Accepted
## Context
The ADR-011 vocabulary round was followed by a full-tree consistency
review — the gate before task decomposition. The vocabulary itself
held everywhere; what the review surfaced is a set of **composition
defects**: mechanisms each specced correctly in isolation, whose
interaction was never ruled, plus two caller-facing shapes stated two
ways. All are pre-implementation (zero code exists), all deciding
facts are in hand, and none waits on a consumer:
1. **The sweep-abort return shape is stated two ways.**
store-api.md's §Lifecycle pins an aborted sweep as
`Ok(SweepReport { aborted: Some(GcAbortCause), .. })` ("the report
is data, not an error carrier"), while §Error model of the *same
file* defines `GcAborted` as the sweep-abort error carrying the
cause classes — and ADR-005, ADR-008, ADR-010, and
gc-and-namespaces.md all follow the error reading. An implementer
writing `sweep()` cannot pick a signature.
2. **The pin token has no wire shape.** The renewal contract (ADR-010
§4) has the putter renew by presenting the token "alongside the
digest" via `blobs/have` — but the specced `have` payload is
`{namespace, digests[]}` with no token field, `blobs/put`'s
token-carrying response is never specced, and an unknown-length
putter does not know its digest until the server computes it. The
token's *validity domain* is also unstated (renewal may arrive
after the minting handler's guard dropped, possibly to a different
fleet node).
3. **"Fleet state is active" has no activation predicate.** ADR-010
routes puts through `pin_commit` "wherever the engine is
seam-capable and fleet state is active", and ADR-008 keeps
single-node postures on the in-process protocol — but nothing
defines what activates fleet state. Under ADR-011's vocabulary
fleetness (two instances sharing one pool) is not observable by
any one constructor.
4. **The GC-state home under mixed fleet engines is unnamed.** In the
kv=postgres + large=local-on-shared-media fleet posture (valid per
the composite rule), pin-before-publish for a large entry is a file
rename, not an insert — the joint entry+pin tx is not achievable on
the large tier's medium — and nothing says which engine hosts
`gc_pins`/`gc_window`/the sweep lock, or whether two SQL engines
mean two pin tables and an arbitration union.
5. **The sqlite seam rows are self-contradictory.** ADR-010 §2 rules
sqlite's `ensure_gc_schema` creates the window table *only*, yet
assigns sqlite `pin_commit`/`pin_renew`/`pin_check` "bodies
exercised by unit tests only" — bodies that insert into a table
nothing creates — and gives sqlite a sweep-lock clause body whose
semantics are undefined (sqlite has no advisory locks) and which
ADR-008's "single-node postures unchanged" says never runs.
The review also surfaced smaller caller-facing gaps (fall-through for
non-get ops, the kv-only error point, a missing abort cause, the
fleet liveness-table registration's API home) that need a ruled home
before signatures are written; §6 records them.
## Decision
### 1. Sweeps are never errors — every abort is data in the report
`sweep()` returns `Result<SweepReport, Error>` where the error family
exists for *operational* failure (I/O, engine failure), never for a
deliberate no-op. All abort causes are report data:
- **`GcAborted` is retired from the error enum.** The only abort
vocabulary is `GcAbortCause`, a data enum carried in
`SweepReport.aborted: Option<GcAbortCause>`, with exactly three
variants: `ProtectFailed` (ADR-005's protection-source failure),
`SweeperLock` (ADR-008's fleet sweeper-lock contention),
`NoLivenessSources` (the safe default — a sweep invoked with zero
registered sources; previously this cause existed in prose,
store-api.md's sweep bullet, but had no variant to carry it).
- Direct `delete()` on a protected key keeps a typed refusal error,
named **`GcRefuse`** — a distinct name so no reader composes the
retired `GcAborted` semantics onto it. (This renames the refusal
path; nothing else about it changes — ADR-005's arbitration still
decides at delete time.)
- `GcAbortCause` is matched non-exhaustively by callers; new causes
are new ADRs.
Rationale: fleet lock contention is a *routine, expected* outcome of
scheduled sweeps — the embedder's next run takes the lock — and a
typed error would push the common case onto the exceptional path.
The report is the data embedders schedule on; "sweep counts are
exact" (the test gate) is asserted against report fields.
This ruling picks store-api.md §Lifecycle's shape over the error
shape used by ADR-005/008/010 and the gc spec; those documents'
wording is aligned to it (mechanical edits; no semantic change — all
four said "nothing deleted" identically, they only disagreed on the
carrier).
### 2. The pin token's wire shape and validity domain
**`blobs/put` response: `{token, digest}`.** Both fields always
present: `digest` is the stored entry's canonical digest (hex) — the
unknown-length putter cannot know it beforehand and needs it for
every later op; `token` is the opaque pin-token handle (ADR-010 §4's
minting unchanged). Known-length putters receive the same shape.
**`blobs/have` gains the token as the renewal form.** Payload
`{namespace, digests[], token?}`:
- without `token` — unchanged: pure existence probe over `digests[]`
(ADR-008's `has` discipline preserved; response `present[]`
corresponds to `digests[]` by index);
- with `token` — the **renewal probe**: exactly one digest in
`digests[]`, and the handler (a) verifies the token maps to a pin
row for that digest, (b) calls the crate-internal `pin_renew(key,
owner)` via the ops module's store-core access (ADR-010 §4's
mechanism, unchanged), (c) responds with `present[]` as usual — the
putter's registration-landed signal. One token binds one pin row,
so renewal is one digest per probe; a putter renewing several pins
issues one probe per token. If the row has expired (putter paused
past TTL), the handler finds no live row: `present: false` and no
renewal occurs — the putter's backstop is re-put (ADR-005's
delete-then-recover), making ADR-010 §4's expiry race end-to-end
explicit on the wire.
**Token validity domain: the minting node.** The token↔(key, owner)
mapping is ops-layer, in-memory state in the process that minted it
(process lifetime, not handler lifetime — the handler's guard drops
at response time per ADR-010 §4, but the mapping outlives the
handler); renewal probes must reach the minting serving node, and
fleet peer nodes cannot serve a token they did not mint (`present:
false`-equivalent refusal, not an error). A putter whose minting node
died before renewal is exactly ADR-005's in-flight-put loss case with
the same backstop: re-put. This is a documented duty of the
re-routing/replication embedder (route renewal to the serving node),
not new machinery.
**Non-fleet serving nodes (any engine):** the token is the ops
layer's in-memory handle to the in-process `Pin` guard — ADR-005
posture, nothing expires. This sentence generalizes ADR-010 §4's
`local`-only wording to every non-fleet mode (kv=sqlite, kv=postgres
single-instance, large=pg-lo non-fleet).
### 3. Fleet mode is a constructor declaration
**A store instance is constructed with fleet mode explicitly on or
off; it is never inferred.** Under `fleet: true`: pins are expiry
rows, puts route through `pin_commit`, sweep acquires the SQL sweep
lock, and the composite fleet-validity predicate (with its per-node
declarations — ADR-008 §3's media assertion, kv-only re-routing
declaration) is enforced at construction. Under `fleet: false`: the
ADR-005 in-process protocol verbatim, regardless of engine.
ADR-011's *deployment* vocabulary is unchanged (fleet = two or more
instances sharing one pool); the constructor flag is the per-instance
admission of that posture. Two `fleet: true` instances over one
postgres pool are a fleet; an undeclared instance sharing the pool is
**misconfigured** — the observable failure is ADR-008 §2's exact
data-loss scenario, and it is the same class of deployment-verified
invariant as the shared-media declaration: the constructor cannot
prove cross-node/cross-instance truth, the deployment asserts it, and
the detection symptom (an unrenewed, un-pinned entry swept while in
flight) is documented.
`fleet: true` is a constructor error for every non-fleet-valid
configuration: any kv engine other than postgres, `mem` anywhere,
`local` large tiers without the shared-media declaration, `pg-lo` on
a different pg instance than the kv engine (§4).
### 4. GC state has exactly one home: the fleet's kv engine
**All fleet GC state — `gc_pins`, `gc_window`, the sweep lock — lives
on the kv-tier engine.** Fleet mode requires kv=postgres, so
postgres hosts the tables and the advisory lock for the whole store
instance, large tier included. The per-engine alternative (each SQL
engine hosting pin rows for its own tier, arbitration unioning two
tables) is rejected: two pin tables mean two arbitration domains and
two locks — the partitioning failure ADR-008 §3 exists to prevent,
re-expressed inside one store instance. One host is the whole point
of the seam.
Consequences per fleet large-tier engine:
- **`large = pg-lo` (fleet): the same pg instance as the kv engine is
required** — this is ADR-009's consolidation posture ("pool content
lives in the same pg instance the kv tier rides") now made a
checkable clause of the composite fleet-validity predicate, because
it is what makes the joint entry+pin tx expressible: the store core
holds one transaction over the shared instance; the pg-lo engine
contributes LO writes + companion-row insert; the pin-row insert is
the store's (§2 of ADR-010, ownership unchanged). Cross-instance
kv=postgres + large=pg-lo is valid *only* with `fleet: false`
(in-process protocol).
- **`large = local` on declared shared media (fleet): pin-first,
publish-second.** The joint tx is not achievable across media, so
the ordering is pinned: the pin row commits on the kv engine
*before* the stage file's commit-rename publishes the entry.
- The invariant survives: an entry becomes visible only after its
pin row is durable — sweep arbitration (which queries pin rows)
can never observe a visible-unpinned entry.
- The crash window moves to the harmless side: crash between
pin-commit and rename leaves an orphan pin row, reaped by the
existing expiry machinery (ADR-008's TTL + sweeper maintenance
step). No new recovery mechanism.
- The reverse ordering (rename-then-pin) is rejected: a crash
there reopens ADR-005's original race (visible, unpinned,
sweepable).
**Non-fleet postures are unchanged and single-hosted:** the GC
arbitration domain is the store instance (ADR-005/011); pins and
liveness arbitration are in-process; the durable window table (§5)
may live on a SQL engine but carries no fleet semantics.
### 5. The engine-state seam's final shape (sqlite rows repaired)
Per-engine clause applicability — the correction of ADR-010 §2's
contradictory rows:
| Engine | `ensure_gc_schema` | Window clauses | Pin clauses | Sweep-lock clause |
|---|---|---|---|---|
| postgres | `gc_pins` + `gc_window` | yes (fleet-active when declared) | yes (fleet-active when declared) | yes (pg advisory lock; fleet-active when declared) |
| sqlite | `gc_window` only | yes (durable single-node staging) | **no** (bodies dropped) | **no** (body dropped — single-flighter is the in-process lock) |
| pg-lo | `gc_window` only | conditional — see the one-host rule below | **no** (fleet pins live on the kv host, §4) | no |
| `local`, `mem` | — excluded from the seam by construction (ADR-010 unchanged); their delete-window staging is **in-process** (ADR-005 protocol inline) | | | |
**The one-window-host rule (per store instance):** the durable staged
window lives on **one** SQL-backed engine — the kv engine when it is
SQL-backed (sqlite or postgres), otherwise the large engine (pg-lo,
e.g. the kv=mem + large=pg-lo shape). A dual-SQL instance (kv=sqlite +
large=pg-lo, non-fleet) stages on the kv engine only; pg-lo's
window clauses are dormant there. Never two window tables in one
instance — the candidate set is one set, and a split staging domain
would be a miniature of the two-arbitration-domain failure §4
rejects.
- The sqlite pin-clause bodies are **dropped**: they were dead
machinery inserting into a table nothing created. A sqlite fleet is
not on offer (ADR-010 §3); if one ever is, it is a new ADR that
re-adds the clauses with their schema.
- The sqlite sweep-lock body is **dropped**: sqlite has no advisory
lock, and ADR-008's "single-node postures keep the arbitration lock
in-process" is the actual mechanism. The seam's sweep-lock clause
is the pg advisory lock and applies to the fleet host only.
- pg-lo remains a seam implementer for the window family (it hosts
the durable window on non-fleet topologies where the kv engine is
not SQL-backed, e.g. kv=mem + large=pg-lo test shapes) and
contributes staged-put mechanics into the store-held tx in the
consolidation posture; it hosts no fleet state.
- The durable staged delete window stays a **SQL-backed-engine
mechanism** (sqlite included) per ADR-008 §2 — what this ruling
adds is the non-SQL half: "`local` and `mem` stage candidates
in-process and execute the ADR-005 window protocol inline within
the same sweep." "On every engine" (ADR-008 §2, ADR-010 §2) is
corrected to "on every SQL-backed engine".
### 6. Facade clarifications (ruled, not deferred)
1. **Fall-through applies to every key-addressed op, not just
get/read_range.** `stat`, `has`, and direct `delete` all consult
the kv tier first, then the large tier (a `size` probe for
`stat`/`has`; arbitration-then-delete where the entry lives for
`delete`). POC #1 finding 6's fall-through is the general rule.
2. **kv-only mode's over-threshold rejection is a put-time error.**
Known-length puts (`len: Some`) exceeding the threshold reject
immediately; unknown-length puts reject at threshold overflow
(mid-buffer). The prior "constructor-time error" wording was
impossible — put lengths are not known at construction.
3. **`dual-tier` with `kv = mem` + `large = local` is a valid
construction** (the dispatch-coverage test shape; `mem` is merely
the kv engine choice in dual-tier mode). `mem-only` remains the
no-large-tier ephemeral mode. Both are non-fleet by definition.
4. **Direct `delete()` refuses any protected key** — pinned,
observed-live via a registered source, or named by the protect
callback — not just pinned keys. This is the ADR-005 invariant
("a visible pool entry is never deleted while liveness protects
it") applied to the direct path; ADR-005's "refuses pinned keys"
wording is aligned upward. An embedder's correction flow
deregisters/re-registers around the delete.
5. **Error-model member vs return shape:** `get`/`stat` return
`Option` (a miss is a value, not an error — `Missing` is not
theirs); `read_range` and `delete` return `Err(Missing)` on absent
keys (informative for the embedder flows that use them).
`Io(String)` covers backend media failure including cursor and
staged-put I/O errors; `Verification` covers hash-check failure on
put and get paths alike; `KeyInvalid` malformed keys. `Backend`
trait methods return this error family (crate-internal variants,
thiserror).
6. **Fleet liveness-table registration has an API home:** the fleet
form of `register_liveness_source` registers a *named table
reference* in the shared engine (the embedder's table per ADR-008
§2; the sweep's SQL arbitration consults it). The facade method is
the same verb with a fleet-shaped argument; the store core reads
the table and never owns it (ADR-008's schema-ownership rule).
7. **The staged window's executor is the next sweep** (ADR-008 §2's
"the sweeper, or the next sweep's owner"): a sweep takes overdue
`gc_window` rows (past `visible_after`), re-arbitrates each, and
commits the deletes before/alongside its own marking; non-SQL
engines execute inline in the same sweep (§5). Exact-count sweep
invariants are asserted relative to this schedule.
## Consequences
**Positive**
- Every signature the implementation needs is now single-valued:
`sweep()`'s return shape, put/have payload shapes, the fleet
activation predicate, the GC-state host, and the per-engine seam
table. Task decomposition has no contradictory text to resolve.
- The fleet mechanism gained the missing pieces (activation flag,
state host, large=local ordering) without new machinery — every
piece reuses an existing mechanism (constructor declarations,
expiry reaper, the store-held tx).
- The sqlite seam shrank (dead bodies dropped); the per-engine
applicability table is the contract-test matrix.
**Negative**
- `GcAborted`'s retirement is a renaming cost across four documents
(mechanical; no semantic change — and no code exists yet to break).
- The pin token gains a documented duty (route renewal to the minting
node) and the have/put payload shapes change from the earlier prose
(a token field and a digest return field) — wire-shape decisions
locked before any consumer, per the ADR-002 door discipline.
- Fleet mode is one more constructor parameter with per-node
declaration duties; the deployment-verified invariant class grows
by one row (undeclared-instance sharing).
**Neutral**
- No tier, engine, feature flag, or constructor *mode* changes; the
composite fleet-validity predicate gains one clause (pg-lo
same-instance) it had already assumed in prose (ADR-009).
- SweepReport, GcAbortCause, GcRefuse, and the fall-through/put-time
rules are the only *new* facade vocabulary; everything else in
store-api.md stands.
## References
- ADR-005 (the GC invariant and single-node protocol this ADR extends
and aligns), ADR-008 (fleet GC mechanism — §2's window wording and
§3's declarations feed §4/§5 here), ADR-009 (pg-lo admission —
§4's same-instance clause formalizes its consolidation posture),
ADR-010 (the I/O seams and GC-state home — §1/§2/§4 here repair its
sqlite rows and rule its activation predicate; §2 here completes
its §4 token contract with a wire shape), ADR-011 (the vocabulary
this ADR's constructor flag and host rules compose with)
- [store-api.md](../store-api.md) (sweep shape, error model,
invariants); [ops-surface.md](../ops-surface.md) (the op payloads
§2 pins); [backends-and-dispatch.md](../backends-and-dispatch.md)
(the composite fleet-validity rule §4 amends; constructor modes);
[gc-and-namespaces.md](../gc-and-namespaces.md) (the delete-window
execution model §6.7 names)
- `docs/research/poc-trait-dispatch-findings.md` finding 6
(fall-through, §6.1's basis); `poc-pglo-findings.md` (the
consolidation posture §4 formalizes)