alkblobs/docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md

# ADR-012: Pre-decomposition consistency rulings — sweep-abort shape, pin-token wire shape, fleet-mode activation, and the GC-state host

## Status

Accepted

## Context

The ADR-011 vocabulary round was followed by a full-tree consistency
review — the gate before task decomposition. The vocabulary itself
held everywhere; what the review surfaced is a set of **composition
defects**: mechanisms each specced correctly in isolation, whose
interaction was never ruled, plus two caller-facing shapes stated two
ways. All are pre-implementation (zero code exists), all deciding
facts are in hand, and none waits on a consumer:

1. **The sweep-abort return shape is stated two ways.**
   store-api.md's §Lifecycle pins an aborted sweep as
   `Ok(SweepReport { aborted: Some(GcAbortCause), .. })` ("the report
   is data, not an error carrier"), while §Error model of the *same
   file* defines `GcAborted` as the sweep-abort error carrying the
   cause classes — and ADR-005, ADR-008, ADR-010, and
   gc-and-namespaces.md all follow the error reading. An implementer
   writing `sweep()` cannot pick a signature.
2. **The pin token has no wire shape.** The renewal contract (ADR-010
   §4) has the putter renew by presenting the token "alongside the
   digest" via `blobs/have` — but the specced `have` payload is
   `{namespace, digests[]}` with no token field, `blobs/put`'s
   token-carrying response is never specced, and an unknown-length
   putter does not know its digest until the server computes it. The
   token's *validity domain* is also unstated (renewal may arrive
   after the minting handler's guard dropped, possibly to a different
   fleet node).
3. **"Fleet state is active" has no activation predicate.** ADR-010
   routes puts through `pin_commit` "wherever the engine is
   seam-capable and fleet state is active", and ADR-008 keeps
   single-node postures on the in-process protocol — but nothing
   defines what activates fleet state. Under ADR-011's vocabulary
   fleetness (two instances sharing one pool) is not observable by
   any one constructor.
4. **The GC-state home under mixed fleet engines is unnamed.** In the
   kv=postgres + large=local-on-shared-media fleet posture (valid per
   the composite rule), pin-before-publish for a large entry is a file
   rename, not an insert — the joint entry+pin tx is not achievable on
   the large tier's medium — and nothing says which engine hosts
   `gc_pins`/`gc_window`/the sweep lock, or whether two SQL engines
   mean two pin tables and an arbitration union.
5. **The sqlite seam rows are self-contradictory.** ADR-010 §2 rules
   sqlite's `ensure_gc_schema` creates the window table *only*, yet
   assigns sqlite `pin_commit`/`pin_renew`/`pin_check` "bodies
   exercised by unit tests only" — bodies that insert into a table
   nothing creates — and gives sqlite a sweep-lock clause body whose
   semantics are undefined (sqlite has no advisory locks) and which
   ADR-008's "single-node postures unchanged" says never runs.

The review also surfaced smaller caller-facing gaps (fall-through for
non-get ops, the kv-only error point, a missing abort cause, the
fleet liveness-table registration's API home) that need a ruled home
before signatures are written; §6 records them.

## Decision

### 1. Sweeps are never errors — every abort is data in the report

`sweep()` returns `Result<SweepReport, Error>` where the error family
exists for *operational* failure (I/O, engine failure), never for a
deliberate no-op. All abort causes are report data:

- **`GcAborted` is retired from the error enum.** The only abort
  vocabulary is `GcAbortCause`, a data enum carried in
  `SweepReport.aborted: Option<GcAbortCause>`, with exactly three
  variants: `ProtectFailed` (ADR-005's protection-source failure),
  `SweeperLock` (ADR-008's fleet sweeper-lock contention),
  `NoLivenessSources` (the safe default — a sweep invoked with zero
  registered sources; previously this cause existed in prose,
  store-api.md's sweep bullet, but had no variant to carry it).
- Direct `delete()` on a protected key keeps a typed refusal error,
  named **`GcRefuse`** — a distinct name so no reader composes the
  retired `GcAborted` semantics onto it. (This renames the refusal
  path; nothing else about it changes — ADR-005's arbitration still
  decides at delete time.)
- `GcAbortCause` is matched non-exhaustively by callers; new causes
  are new ADRs.

Rationale: fleet lock contention is a *routine, expected* outcome of
scheduled sweeps — the embedder's next run takes the lock — and a
typed error would push the common case onto the exceptional path.
The report is the data embedders schedule on; "sweep counts are
exact" (the test gate) is asserted against report fields.

This ruling picks store-api.md §Lifecycle's shape over the error
shape used by ADR-005/008/010 and the gc spec; those documents'
wording is aligned to it (mechanical edits; no semantic change — all
four said "nothing deleted" identically, they only disagreed on the
carrier).

### 2. The pin token's wire shape and validity domain

**`blobs/put` response: `{token, digest}`.** Both fields always
present: `digest` is the stored entry's canonical digest (hex) — the
unknown-length putter cannot know it beforehand and needs it for
every later op; `token` is the opaque pin-token handle (ADR-010 §4's
minting unchanged). Known-length putters receive the same shape.

**`blobs/have` gains the token as the renewal form.** Payload
`{namespace, digests[], token?}`:
- without `token` — unchanged: pure existence probe over `digests[]`
  (ADR-008's `has` discipline preserved; response `present[]`
  corresponds to `digests[]` by index);
- with `token` — the **renewal probe**: exactly one digest in
  `digests[]`, and the handler (a) verifies the token maps to a pin
  row for that digest, (b) calls the crate-internal `pin_renew(key,
  owner)` via the ops module's store-core access (ADR-010 §4's
  mechanism, unchanged), (c) responds with `present[]` as usual — the
  putter's registration-landed signal. One token binds one pin row,
  so renewal is one digest per probe; a putter renewing several pins
  issues one probe per token. If the row has expired (putter paused
  past TTL), the handler finds no live row: `present: false` and no
  renewal occurs — the putter's backstop is re-put (ADR-005's
  delete-then-recover), making ADR-010 §4's expiry race end-to-end
  explicit on the wire.

**Token validity domain: the minting node.** The token↔(key, owner)
mapping is ops-layer, in-memory state in the process that minted it
(process lifetime, not handler lifetime — the handler's guard drops
at response time per ADR-010 §4, but the mapping outlives the
handler); renewal probes must reach the minting serving node, and
fleet peer nodes cannot serve a token they did not mint (`present:
false`-equivalent refusal, not an error). A putter whose minting node
died before renewal is exactly ADR-005's in-flight-put loss case with
the same backstop: re-put. This is a documented duty of the
re-routing/replication embedder (route renewal to the serving node),
not new machinery.

**Non-fleet serving nodes (any engine):** the token is the ops
layer's in-memory handle to the in-process `Pin` guard — ADR-005
posture, nothing expires. This sentence generalizes ADR-010 §4's
`local`-only wording to every non-fleet mode (kv=sqlite, kv=postgres
single-instance, large=pg-lo non-fleet).

### 3. Fleet mode is a constructor declaration

**A store instance is constructed with fleet mode explicitly on or
off; it is never inferred.** Under `fleet: true`: pins are expiry
rows, puts route through `pin_commit`, sweep acquires the SQL sweep
lock, and the composite fleet-validity predicate (with its per-node
declarations — ADR-008 §3's media assertion, kv-only re-routing
declaration) is enforced at construction. Under `fleet: false`: the
ADR-005 in-process protocol verbatim, regardless of engine.

ADR-011's *deployment* vocabulary is unchanged (fleet = two or more
instances sharing one pool); the constructor flag is the per-instance
admission of that posture. Two `fleet: true` instances over one
postgres pool are a fleet; an undeclared instance sharing the pool is
**misconfigured** — the observable failure is ADR-008 §2's exact
data-loss scenario, and it is the same class of deployment-verified
invariant as the shared-media declaration: the constructor cannot
prove cross-node/cross-instance truth, the deployment asserts it, and
the detection symptom (an unrenewed, un-pinned entry swept while in
flight) is documented.

`fleet: true` is a constructor error for every non-fleet-valid
configuration: any kv engine other than postgres, `mem` anywhere,
`local` large tiers without the shared-media declaration, `pg-lo` on
a different pg instance than the kv engine (§4).

### 4. GC state has exactly one home: the fleet's kv engine

**All fleet GC state — `gc_pins`, `gc_window`, the sweep lock — lives
on the kv-tier engine.** Fleet mode requires kv=postgres, so
postgres hosts the tables and the advisory lock for the whole store
instance, large tier included. The per-engine alternative (each SQL
engine hosting pin rows for its own tier, arbitration unioning two
tables) is rejected: two pin tables mean two arbitration domains and
two locks — the partitioning failure ADR-008 §3 exists to prevent,
re-expressed inside one store instance. One host is the whole point
of the seam.

Consequences per fleet large-tier engine:

- **`large = pg-lo` (fleet): the same pg instance as the kv engine is
  required** — this is ADR-009's consolidation posture ("pool content
  lives in the same pg instance the kv tier rides") now made a
  checkable clause of the composite fleet-validity predicate, because
  it is what makes the joint entry+pin tx expressible: the store core
  holds one transaction over the shared instance; the pg-lo engine
  contributes LO writes + companion-row insert; the pin-row insert is
  the store's (§2 of ADR-010, ownership unchanged). Cross-instance
  kv=postgres + large=pg-lo is valid *only* with `fleet: false`
  (in-process protocol).
- **`large = local` on declared shared media (fleet): pin-first,
  publish-second.** The joint tx is not achievable across media, so
  the ordering is pinned: the pin row commits on the kv engine
  *before* the stage file's commit-rename publishes the entry.
  - The invariant survives: an entry becomes visible only after its
    pin row is durable — sweep arbitration (which queries pin rows)
    can never observe a visible-unpinned entry.
  - The crash window moves to the harmless side: crash between
    pin-commit and rename leaves an orphan pin row, reaped by the
    existing expiry machinery (ADR-008's TTL + sweeper maintenance
    step). No new recovery mechanism.
  - The reverse ordering (rename-then-pin) is rejected: a crash
    there reopens ADR-005's original race (visible, unpinned,
    sweepable).

**Non-fleet postures are unchanged and single-hosted:** the GC
arbitration domain is the store instance (ADR-005/011); pins and
liveness arbitration are in-process; the durable window table (§5)
may live on a SQL engine but carries no fleet semantics.

### 5. The engine-state seam's final shape (sqlite rows repaired)

Per-engine clause applicability — the correction of ADR-010 §2's
contradictory rows:

| Engine | `ensure_gc_schema` | Window clauses | Pin clauses | Sweep-lock clause |
|---|---|---|---|---|
| postgres | `gc_pins` + `gc_window` | yes (fleet-active when declared) | yes (fleet-active when declared) | yes (pg advisory lock; fleet-active when declared) |
| sqlite | `gc_window` only | yes (durable single-node staging) | **no** (bodies dropped) | **no** (body dropped — single-flighter is the in-process lock) |
| pg-lo | `gc_window` only | conditional — see the one-host rule below | **no** (fleet pins live on the kv host, §4) | no |
| `local`, `mem` | — excluded from the seam by construction (ADR-010 unchanged); their delete-window staging is **in-process** (ADR-005 protocol inline) | | | |

**The one-window-host rule (per store instance):** the durable staged
window lives on **one** SQL-backed engine — the kv engine when it is
SQL-backed (sqlite or postgres), otherwise the large engine (pg-lo,
e.g. the kv=mem + large=pg-lo shape). A dual-SQL instance (kv=sqlite +
large=pg-lo, non-fleet) stages on the kv engine only; pg-lo's
window clauses are dormant there. Never two window tables in one
instance — the candidate set is one set, and a split staging domain
would be a miniature of the two-arbitration-domain failure §4
rejects.

- The sqlite pin-clause bodies are **dropped**: they were dead
  machinery inserting into a table nothing created. A sqlite fleet is
  not on offer (ADR-010 §3); if one ever is, it is a new ADR that
  re-adds the clauses with their schema.
- The sqlite sweep-lock body is **dropped**: sqlite has no advisory
  lock, and ADR-008's "single-node postures keep the arbitration lock
  in-process" is the actual mechanism. The seam's sweep-lock clause
  is the pg advisory lock and applies to the fleet host only.
- pg-lo remains a seam implementer for the window family (it hosts
  the durable window on non-fleet topologies where the kv engine is
  not SQL-backed, e.g. kv=mem + large=pg-lo test shapes) and
  contributes staged-put mechanics into the store-held tx in the
  consolidation posture; it hosts no fleet state.
- The durable staged delete window stays a **SQL-backed-engine
  mechanism** (sqlite included) per ADR-008 §2 — what this ruling
  adds is the non-SQL half: "`local` and `mem` stage candidates
  in-process and execute the ADR-005 window protocol inline within
  the same sweep." "On every engine" (ADR-008 §2, ADR-010 §2) is
  corrected to "on every SQL-backed engine".

### 6. Facade clarifications (ruled, not deferred)

1. **Fall-through applies to every key-addressed op, not just
   get/read_range.** `stat`, `has`, and direct `delete` all consult
   the kv tier first, then the large tier (a `size` probe for
   `stat`/`has`; arbitration-then-delete where the entry lives for
   `delete`). POC #1 finding 6's fall-through is the general rule.
2. **kv-only mode's over-threshold rejection is a put-time error.**
   Known-length puts (`len: Some`) exceeding the threshold reject
   immediately; unknown-length puts reject at threshold overflow
   (mid-buffer). The prior "constructor-time error" wording was
   impossible — put lengths are not known at construction.
3. **`dual-tier` with `kv = mem` + `large = local` is a valid
   construction** (the dispatch-coverage test shape; `mem` is merely
   the kv engine choice in dual-tier mode). `mem-only` remains the
   no-large-tier ephemeral mode. Both are non-fleet by definition.
4. **Direct `delete()` refuses any protected key** — pinned,
   observed-live via a registered source, or named by the protect
   callback — not just pinned keys. This is the ADR-005 invariant
   ("a visible pool entry is never deleted while liveness protects
   it") applied to the direct path; ADR-005's "refuses pinned keys"
   wording is aligned upward. An embedder's correction flow
   deregisters/re-registers around the delete.
5. **Error-model member vs return shape:** `get`/`stat` return
   `Option` (a miss is a value, not an error — `Missing` is not
   theirs); `read_range` and `delete` return `Err(Missing)` on absent
   keys (informative for the embedder flows that use them).
   `Io(String)` covers backend media failure including cursor and
   staged-put I/O errors; `Verification` covers hash-check failure on
   put and get paths alike; `KeyInvalid` malformed keys. `Backend`
   trait methods return this error family (crate-internal variants,
   thiserror).
6. **Fleet liveness-table registration has an API home:** the fleet
   form of `register_liveness_source` registers a *named table
   reference* in the shared engine (the embedder's table per ADR-008
   §2; the sweep's SQL arbitration consults it). The facade method is
   the same verb with a fleet-shaped argument; the store core reads
   the table and never owns it (ADR-008's schema-ownership rule).
7. **The staged window's executor is the next sweep** (ADR-008 §2's
   "the sweeper, or the next sweep's owner"): a sweep takes overdue
   `gc_window` rows (past `visible_after`), re-arbitrates each, and
   commits the deletes before/alongside its own marking; non-SQL
   engines execute inline in the same sweep (§5). Exact-count sweep
   invariants are asserted relative to this schedule.

## Consequences

**Positive**

- Every signature the implementation needs is now single-valued:
  `sweep()`'s return shape, put/have payload shapes, the fleet
  activation predicate, the GC-state host, and the per-engine seam
  table. Task decomposition has no contradictory text to resolve.
- The fleet mechanism gained the missing pieces (activation flag,
  state host, large=local ordering) without new machinery — every
  piece reuses an existing mechanism (constructor declarations,
  expiry reaper, the store-held tx).
- The sqlite seam shrank (dead bodies dropped); the per-engine
  applicability table is the contract-test matrix.

**Negative**

- `GcAborted`'s retirement is a renaming cost across four documents
  (mechanical; no semantic change — and no code exists yet to break).
- The pin token gains a documented duty (route renewal to the minting
  node) and the have/put payload shapes change from the earlier prose
  (a token field and a digest return field) — wire-shape decisions
  locked before any consumer, per the ADR-002 door discipline.
- Fleet mode is one more constructor parameter with per-node
  declaration duties; the deployment-verified invariant class grows
  by one row (undeclared-instance sharing).

**Neutral**

- No tier, engine, feature flag, or constructor *mode* changes; the
  composite fleet-validity predicate gains one clause (pg-lo
  same-instance) it had already assumed in prose (ADR-009).
- SweepReport, GcAbortCause, GcRefuse, and the fall-through/put-time
  rules are the only *new* facade vocabulary; everything else in
  store-api.md stands.

## References

- ADR-005 (the GC invariant and single-node protocol this ADR extends
  and aligns), ADR-008 (fleet GC mechanism — §2's window wording and
  §3's declarations feed §4/§5 here), ADR-009 (pg-lo admission —
  §4's same-instance clause formalizes its consolidation posture),
  ADR-010 (the I/O seams and GC-state home — §1/§2/§4 here repair its
  sqlite rows and rule its activation predicate; §2 here completes
  its §4 token contract with a wire shape), ADR-011 (the vocabulary
  this ADR's constructor flag and host rules compose with)
- [store-api.md](../store-api.md) (sweep shape, error model,
  invariants); [ops-surface.md](../ops-surface.md) (the op payloads
  §2 pins); [backends-and-dispatch.md](../backends-and-dispatch.md)
  (the composite fleet-validity rule §4 amends; constructor modes);
  [gc-and-namespaces.md](../gc-and-namespaces.md) (the delete-window
  execution model §6.7 names)
- `docs/research/poc-trait-dispatch-findings.md` finding 6
  (fall-through, §6.1's basis); `poc-pglo-findings.md` (the
  consolidation posture §4 formalizes)