alkblobs/docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md

ADR-012: Pre-decomposition consistency rulings — sweep-abort shape, pin-token wire shape, fleet-mode activation, and the GC-state host

Status

Accepted

Context

The ADR-011 vocabulary round was followed by a full-tree consistency review — the gate before task decomposition. The vocabulary itself held everywhere; what the review surfaced is a set of composition defects: mechanisms each specced correctly in isolation, whose interaction was never ruled, plus two caller-facing shapes stated two ways. All are pre-implementation (zero code exists), all deciding facts are in hand, and none waits on a consumer:

1. The sweep-abort return shape is stated two ways. store-api.md's §Lifecycle pins an aborted sweep as Ok(SweepReport { aborted: Some(GcAbortCause), .. }) ("the report is data, not an error carrier"), while §Error model of the same file defines GcAborted as the sweep-abort error carrying the cause classes — and ADR-005, ADR-008, ADR-010, and gc-and-namespaces.md all follow the error reading. An implementer writing sweep() cannot pick a signature.
2. The pin token has no wire shape. The renewal contract (ADR-010 §4) has the putter renew by presenting the token "alongside the digest" via blobs/have — but the specced have payload is {namespace, digests[]} with no token field, blobs/put's token-carrying response is never specced, and an unknown-length putter does not know its digest until the server computes it. The token's validity domain is also unstated (renewal may arrive after the minting handler's guard dropped, possibly to a different fleet node).
3. "Fleet state is active" has no activation predicate. ADR-010 routes puts through pin_commit "wherever the engine is seam-capable and fleet state is active", and ADR-008 keeps single-node postures on the in-process protocol — but nothing defines what activates fleet state. Under ADR-011's vocabulary fleetness (two instances sharing one pool) is not observable by any one constructor.
4. The GC-state home under mixed fleet engines is unnamed. In the kv=postgres + large=local-on-shared-media fleet posture (valid per the composite rule), pin-before-publish for a large entry is a file rename, not an insert — the joint entry+pin tx is not achievable on the large tier's medium — and nothing says which engine hosts gc_pins/gc_window/the sweep lock, or whether two SQL engines mean two pin tables and an arbitration union.
5. The sqlite seam rows are self-contradictory. ADR-010 §2 rules sqlite's ensure_gc_schema creates the window table only, yet assigns sqlite pin_commit/pin_renew/pin_check "bodies exercised by unit tests only" — bodies that insert into a table nothing creates — and gives sqlite a sweep-lock clause body whose semantics are undefined (sqlite has no advisory locks) and which ADR-008's "single-node postures unchanged" says never runs.

The review also surfaced smaller caller-facing gaps (fall-through for non-get ops, the kv-only error point, a missing abort cause, the fleet liveness-table registration's API home) that need a ruled home before signatures are written; §6 records them.

Decision

1. Sweeps are never errors — every abort is data in the report

sweep() returns Result<SweepReport, Error> where the error family exists for operational failure (I/O, engine failure), never for a deliberate no-op. All abort causes are report data:

• GcAborted is retired from the error enum. The only abort vocabulary is GcAbortCause, a data enum carried in SweepReport.aborted: Option<GcAbortCause>, with exactly three variants: ProtectFailed (ADR-005's protection-source failure), SweeperLock (ADR-008's fleet sweeper-lock contention), NoLivenessSources (the safe default — a sweep invoked with zero registered sources; previously this cause existed in prose, store-api.md's sweep bullet, but had no variant to carry it).
• Direct delete() on a protected key keeps a typed refusal error, named GcRefuse — a distinct name so no reader composes the retired GcAborted semantics onto it. (This renames the refusal path; nothing else about it changes — ADR-005's arbitration still decides at delete time.)
• GcAbortCause is matched non-exhaustively by callers; new causes are new ADRs.

Rationale: fleet lock contention is a routine, expected outcome of scheduled sweeps — the embedder's next run takes the lock — and a typed error would push the common case onto the exceptional path. The report is the data embedders schedule on; "sweep counts are exact" (the test gate) is asserted against report fields.

This ruling picks store-api.md §Lifecycle's shape over the error shape used by ADR-005/008/010 and the gc spec; those documents' wording is aligned to it (mechanical edits; no semantic change — all four said "nothing deleted" identically, they only disagreed on the carrier).

2. The pin token's wire shape and validity domain

blobs/put response: {token, digest}. Both fields always present: digest is the stored entry's canonical digest (hex) — the unknown-length putter cannot know it beforehand and needs it for every later op; token is the opaque pin-token handle (ADR-010 §4's minting unchanged). Known-length putters receive the same shape.

blobs/have gains the token as the renewal form. Payload {namespace, digests[], token?}:

• without token — unchanged: pure existence probe over digests[] (ADR-008's has discipline preserved; response present[] corresponds to digests[] by index);
• with token — the renewal probe: exactly one digest in digests[], and the handler (a) verifies the token maps to a pin row for that digest, (b) calls the crate-internal pin_renew(key, owner) via the ops module's store-core access (ADR-010 §4's mechanism, unchanged), (c) responds with present[] as usual — the putter's registration-landed signal. One token binds one pin row, so renewal is one digest per probe; a putter renewing several pins issues one probe per token. If the row has expired (putter paused past TTL), the handler finds no live row: present: false and no renewal occurs — the putter's backstop is re-put (ADR-005's delete-then-recover), making ADR-010 §4's expiry race end-to-end explicit on the wire.

Token validity domain: the minting node. The token↔(key, owner) mapping is ops-layer, in-memory state in the process that minted it (process lifetime, not handler lifetime — the handler's guard drops at response time per ADR-010 §4, but the mapping outlives the handler); renewal probes must reach the minting serving node, and fleet peer nodes cannot serve a token they did not mint (present: false-equivalent refusal, not an error). A putter whose minting node died before renewal is exactly ADR-005's in-flight-put loss case with the same backstop: re-put. This is a documented duty of the re-routing/replication embedder (route renewal to the serving node), not new machinery.

Non-fleet serving nodes (any engine): the token is the ops layer's in-memory handle to the in-process Pin guard — ADR-005 posture, nothing expires. This sentence generalizes ADR-010 §4's local-only wording to every non-fleet mode (kv=sqlite, kv=postgres single-instance, large=pg-lo non-fleet).

3. Fleet mode is a constructor declaration

A store instance is constructed with fleet mode explicitly on or off; it is never inferred. Under fleet: true: pins are expiry rows, puts route through pin_commit, sweep acquires the SQL sweep lock, and the composite fleet-validity predicate (with its per-node declarations — ADR-008 §3's media assertion, kv-only re-routing declaration) is enforced at construction. Under fleet: false: the ADR-005 in-process protocol verbatim, regardless of engine.

ADR-011's deployment vocabulary is unchanged (fleet = two or more instances sharing one pool); the constructor flag is the per-instance admission of that posture. Two fleet: true instances over one postgres pool are a fleet; an undeclared instance sharing the pool is misconfigured — the observable failure is ADR-008 §2's exact data-loss scenario, and it is the same class of deployment-verified invariant as the shared-media declaration: the constructor cannot prove cross-node/cross-instance truth, the deployment asserts it, and the detection symptom (an unrenewed, un-pinned entry swept while in flight) is documented.

fleet: true is a constructor error for every non-fleet-valid configuration: any kv engine other than postgres, mem anywhere, local large tiers without the shared-media declaration, pg-lo on a different pg instance than the kv engine (§4).

4. GC state has exactly one home: the fleet's kv engine

All fleet GC state — gc_pins, gc_window, the sweep lock — lives on the kv-tier engine. Fleet mode requires kv=postgres, so postgres hosts the tables and the advisory lock for the whole store instance, large tier included. The per-engine alternative (each SQL engine hosting pin rows for its own tier, arbitration unioning two tables) is rejected: two pin tables mean two arbitration domains and two locks — the partitioning failure ADR-008 §3 exists to prevent, re-expressed inside one store instance. One host is the whole point of the seam.

Consequences per fleet large-tier engine:

• large = pg-lo (fleet): the same pg instance as the kv engine is required — this is ADR-009's consolidation posture ("pool content lives in the same pg instance the kv tier rides") now made a checkable clause of the composite fleet-validity predicate, because it is what makes the joint entry+pin tx expressible: the store core holds one transaction over the shared instance; the pg-lo engine contributes LO writes + companion-row insert; the pin-row insert is the store's (§2 of ADR-010, ownership unchanged). Cross-instance kv=postgres + large=pg-lo is valid only with fleet: false (in-process protocol).
• large = local on declared shared media (fleet): pin-first, publish-second. The joint tx is not achievable across media, so the ordering is pinned: the pin row commits on the kv engine before the stage file's commit-rename publishes the entry.
  • The invariant survives: an entry becomes visible only after its pin row is durable — sweep arbitration (which queries pin rows) can never observe a visible-unpinned entry.
  • The crash window moves to the harmless side: crash between pin-commit and rename leaves an orphan pin row, reaped by the existing expiry machinery (ADR-008's TTL + sweeper maintenance step). No new recovery mechanism.
  • The reverse ordering (rename-then-pin) is rejected: a crash there reopens ADR-005's original race (visible, unpinned, sweepable).

Non-fleet postures are unchanged and single-hosted: the GC arbitration domain is the store instance (ADR-005/011); pins and liveness arbitration are in-process; the durable window table (§5) may live on a SQL engine but carries no fleet semantics.

5. The engine-state seam's final shape (sqlite rows repaired)

Per-engine clause applicability — the correction of ADR-010 §2's contradictory rows:

Engine	ensure_gc_schema	Window clauses	Pin clauses	Sweep-lock clause
postgres	gc_pins + gc_window	yes (fleet-active when declared)	yes (fleet-active when declared)	yes (pg advisory lock; fleet-active when declared)
sqlite	gc_window only	yes (durable single-node staging)	no (bodies dropped)	no (body dropped — single-flighter is the in-process lock)
pg-lo	gc_window only	conditional — see the one-host rule below	no (fleet pins live on the kv host, §4)	no
local, mem	— excluded from the seam by construction (ADR-010 unchanged); their delete-window staging is in-process (ADR-005 protocol inline)

The one-window-host rule (per store instance): the durable staged window lives on one SQL-backed engine — the kv engine when it is SQL-backed (sqlite or postgres), otherwise the large engine (pg-lo, e.g. the kv=mem + large=pg-lo shape). A dual-SQL instance (kv=sqlite + large=pg-lo, non-fleet) stages on the kv engine only; pg-lo's window clauses are dormant there. Never two window tables in one instance — the candidate set is one set, and a split staging domain would be a miniature of the two-arbitration-domain failure §4 rejects.

• The sqlite pin-clause bodies are dropped: they were dead machinery inserting into a table nothing created. A sqlite fleet is not on offer (ADR-010 §3); if one ever is, it is a new ADR that re-adds the clauses with their schema.
• The sqlite sweep-lock body is dropped: sqlite has no advisory lock, and ADR-008's "single-node postures keep the arbitration lock in-process" is the actual mechanism. The seam's sweep-lock clause is the pg advisory lock and applies to the fleet host only.
• pg-lo remains a seam implementer for the window family (it hosts the durable window on non-fleet topologies where the kv engine is not SQL-backed, e.g. kv=mem + large=pg-lo test shapes) and contributes staged-put mechanics into the store-held tx in the consolidation posture; it hosts no fleet state.
• The durable staged delete window stays a SQL-backed-engine mechanism (sqlite included) per ADR-008 §2 — what this ruling adds is the non-SQL half: "local and mem stage candidates in-process and execute the ADR-005 window protocol inline within the same sweep." "On every engine" (ADR-008 §2, ADR-010 §2) is corrected to "on every SQL-backed engine".

6. Facade clarifications (ruled, not deferred)

1. Fall-through applies to every key-addressed op, not just get/read_range. stat, has, and direct delete all consult the kv tier first, then the large tier (a size probe for stat/has; arbitration-then-delete where the entry lives for delete). POC #1 finding 6's fall-through is the general rule.
2. kv-only mode's over-threshold rejection is a put-time error. Known-length puts (len: Some) exceeding the threshold reject immediately; unknown-length puts reject at threshold overflow (mid-buffer). The prior "constructor-time error" wording was impossible — put lengths are not known at construction.
3. dual-tier with kv = mem + large = local is a valid construction (the dispatch-coverage test shape; mem is merely the kv engine choice in dual-tier mode). mem-only remains the no-large-tier ephemeral mode. Both are non-fleet by definition.
4. Direct delete() refuses any protected key — pinned, observed-live via a registered source, or named by the protect callback — not just pinned keys. This is the ADR-005 invariant ("a visible pool entry is never deleted while liveness protects it") applied to the direct path; ADR-005's "refuses pinned keys" wording is aligned upward. An embedder's correction flow deregisters/re-registers around the delete.
5. Error-model member vs return shape: get/stat return Option (a miss is a value, not an error — Missing is not theirs); read_range and delete return Err(Missing) on absent keys (informative for the embedder flows that use them). Io(String) covers backend media failure including cursor and staged-put I/O errors; Verification covers hash-check failure on put and get paths alike; KeyInvalid malformed keys. Backend trait methods return this error family (crate-internal variants, thiserror).
6. Fleet liveness-table registration has an API home: the fleet form of register_liveness_source registers a named table reference in the shared engine (the embedder's table per ADR-008 §2; the sweep's SQL arbitration consults it). The facade method is the same verb with a fleet-shaped argument; the store core reads the table and never owns it (ADR-008's schema-ownership rule).
7. The staged window's executor is the next sweep (ADR-008 §2's "the sweeper, or the next sweep's owner"): a sweep takes overdue gc_window rows (past visible_after), re-arbitrates each, and commits the deletes before/alongside its own marking; non-SQL engines execute inline in the same sweep (§5). Exact-count sweep invariants are asserted relative to this schedule.

Consequences

Positive

• Every signature the implementation needs is now single-valued: sweep()'s return shape, put/have payload shapes, the fleet activation predicate, the GC-state host, and the per-engine seam table. Task decomposition has no contradictory text to resolve.
• The fleet mechanism gained the missing pieces (activation flag, state host, large=local ordering) without new machinery — every piece reuses an existing mechanism (constructor declarations, expiry reaper, the store-held tx).
• The sqlite seam shrank (dead bodies dropped); the per-engine applicability table is the contract-test matrix.

Negative

• GcAborted's retirement is a renaming cost across four documents (mechanical; no semantic change — and no code exists yet to break).
• The pin token gains a documented duty (route renewal to the minting node) and the have/put payload shapes change from the earlier prose (a token field and a digest return field) — wire-shape decisions locked before any consumer, per the ADR-002 door discipline.
• Fleet mode is one more constructor parameter with per-node declaration duties; the deployment-verified invariant class grows by one row (undeclared-instance sharing).

Neutral

• No tier, engine, feature flag, or constructor mode changes; the composite fleet-validity predicate gains one clause (pg-lo same-instance) it had already assumed in prose (ADR-009).
• SweepReport, GcAbortCause, GcRefuse, and the fall-through/put-time rules are the only new facade vocabulary; everything else in store-api.md stands.

References

• ADR-005 (the GC invariant and single-node protocol this ADR extends and aligns), ADR-008 (fleet GC mechanism — §2's window wording and §3's declarations feed §4/§5 here), ADR-009 (pg-lo admission — §4's same-instance clause formalizes its consolidation posture), ADR-010 (the I/O seams and GC-state home — §1/§2/§4 here repair its sqlite rows and rule its activation predicate; §2 here completes its §4 token contract with a wire shape), ADR-011 (the vocabulary this ADR's constructor flag and host rules compose with)
• store-api.md (sweep shape, error model, invariants); ops-surface.md (the op payloads §2 pins); backends-and-dispatch.md (the composite fleet-validity rule §4 amends; constructor modes); gc-and-namespaces.md (the delete-window execution model §6.7 names)
• docs/research/poc-trait-dispatch-findings.md finding 6 (fall-through, §6.1's basis); poc-pglo-findings.md (the consolidation posture §4 formalizes)