alkblobs/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md

ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam

Status

Accepted (fleet-state activation, the GC-state host under mixed engines, the sweep-abort return shape, the pin token's wire shape, the sqlite pin/sweep-lock clause reductions, and the window staging for non-SQL engines are ruled by ADR-012; §2's sqlite rows and the "fleet state is active" predicate read through that ADR — where noted inline)

Context

The engine story closed at the decision level (ADR-007/008/009), but three seams that implementation must not guess are unpinned. All three concentrate where the tier/engine split stretched the original contract:

1. The get/put value shapes were never unified. The trait POC's get returns Option<Vec<u8>>; the largeblob POC's fs arm "yields a file handle, not a loaded buffer"; the pg-lo POC's get is a LoGetHandle driving lo_get(oid, off, len) windows — three structurally different shapes behind one trait, and no document named the unified contract. Same on the put side: a byte slice (kv), a stage-then-commit-rename stream (local), and a chunked-lowrite transaction (pg-lo) are all "the put". A unified trait is the crate's second stable seam (ADR-003); its I/O shapes are decided before code, not discovered per engine.

2. ADR-008's fleet state has no seam. Pins become expiry-carrying rows committed with the entry; staged delete-window candidates (including on sqlite); SELECT … FOR UPDATE arbitration; a pg advisory lock for the sweeper. None of that is expressible through the public trait method set, and ADR-003's "the shipped trait impls write plain content-addressed rows into the chosen engine, nothing else" reads as a bar the fleet mechanism violates. Also unresolved: who owns the joint entry+pin transaction — ADR-009 says the pg-lo put's single tx carries the pin row, but ADR-005 says Pin is a store-core RAII guard under the arbitration lock. The trait has no parameter for either, and "the backend learns about pins" would contradict ADR-005. The ops pin-token contract (ops-surface.md) inherits the same conflict: a remote putter holds a token, but an expiry-guarded pin row goes stale if the holder pauses — a sweep could delete a remotely-in-flight put whose row expired.

3. The kv tier had no fleet-validity rule. ADR-008 §3 constrains the large tier's fleet forms (shared media, re-routing) but pins nothing for the kv tier, which the fleet story centers on: two nodes' sqlite engines are trivially partitioned (two files = two pools, not one), and no document said so.

The deciding facts are in hand (Schrödinger's-code rule — none of these may wait on consumers): the POCs fixed every engine's native shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry, advisory lock, staged window); ADR-001 §4 fixed that ops is a store-facade consumer with no backend side-door. What is missing is composition: how the store core drives all engines through one contract while owning GC state.

Alternatives considered for the fleet-state seam, rejected:

• Put the fleet machinery on the public trait (pin/window/table methods on Backend): every engine — including mem and local — must then implement SQL-shaped machinery it cannot express. Rejected.
• Backends grow pin awareness (the pg-lo put tx carries the pin row, backend-maintained): contradicts ADR-005's "backends never learn liveness"; and the arbitration decision (expiry compare, liveness consult) still belongs to the store, so the store ends up reaching into the engine's tx anyway. Rejected.
• A raw-SQL side-channel (connection escape hatch the store core may use arbitrarily): turns the engine abstraction off for exactly the machinery (fleet GC) it most needs to be sound on. Rejected. The seam below is that escape hatch made honest — named clauses, contract-tested per engine, instead of raw access.

Decision

1. One Backend trait; two I/O abstractions per tier — read cursor and staged put

The trait stays one trait. Every engine implements the same method set (ADR-003's methods + ADR-008's size) — no per-tier trait split. What changes: the trait's I/O types are the crate-internal abstractions below, which each engine constructs its own way — the store core programs the abstraction, never the engine shape.

• get returns a read cursor. One abstraction, two body families:
  • kv engines (sqlite, postgres, mem): the cursor materializes the whole value (the engine's row read; bounded by the small-blob tier policy). Its range reads are in-memory slices; len() reads the row's size column.
  • large engines (local, pg-lo): the cursor streams; the medium is hidden in the cursor body — local's is a pread loop over the committed file (immutable once committed); pg-lo's drives descriptorless lo_get(oid, off, len) windows (ADR-009 §3's C4 posture — pool-friendly, stateless per window; the tx-scoped-descriptor form is the fallback). len() reads the companion row or stats the file — metadata only, never content (ADR-008's discipline).
  • Both body families serve the ADR-006 range contract (read_range → slice + slice digest) at one seam. The cursor is store-internal — not public API: store consumers see the facade's (len, stream) shapes (store-api.md) with the cursor staying behind them; the store's stream utility adapts engine cursor bodies, it does not expose them.
• put exists in two forms, selected by the store core before content flows; both land in the CAS entry shape and both participate in pin-before-publish (§2 decides who holds the tx).
  • Whole-value put for the kv tier: the settled row write. POC-shaped bodies: sqlite INSERT OR IGNORE, postgres ON CONFLICT DO NOTHING, mem entry().or_insert().
  • Staged put for the large tier — ADR-003's stage-then-commit-rename promoted from behavior to a named per-engine contract: the engine hands out a staging cursor (local: the stage file under the store's stage-hygiene protocol; pg-lo: chunked lowrite statements over a tx the store core holds — §2's pin_commit is that holder; 512 KiB per POC #7's measured knee). Commit publishes the entry; every failure path converges on discard (store-api.md stage hygiene, per engine, exact-count tested — POC #3 A5's shape).
• list/delete/has/size/name are tier-agnostic. list walks the engine's companion structure (rows / files / LO companion); size is metadata-only (ADR-008). The exact-count sweep-outcome gate (ADR-003) runs per engine.

Feature-graph consequence (fills a gap ADR-007/009 left): feature pg-lo does not imply feature postgres. Each is an engine feature pulling the shared driver cluster (tokio-postgres + deadpool-postgres) independently — a large=pg-lo + kv=sqlite node is coherent and supported. overview.md's dependency table pins the edge. (The same independent-driver discipline is extended to every engine by ADR-013 §4 — sqlite becomes its own default-on feature carrying rusqlite, the kv/large tier features retire, and pg-pure builds become supported, non-default.)

2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested

Ownership: the store core owns all GC state — in-process pin and liveness maps and the fleet state's rows and locks. The public Backend trait stays exactly what ADR-003 says (opaque hash → bytes, namespace-blind; "plain content-addressed rows, nothing else" remains true of the public trait). Backends never learn liveness (ADR-005) — verbatim.

The store core reaches the fleet machinery through a crate-internal (not exported) engine-state companion trait, implemented by the SQL-backed engines (sqlite, postgres, pg-lo), with local and mem excluded by construction (it cannot be expressed on them and they are never fleet engines). ADR-008's mechanism fixes what; the seam is where the mechanism's SQL lives. Clauses (per-engine applicability is ruled by ADR-012 §5's table, which reduces the original sqlite rows; fleet state hosts on the kv-tier engine when fleet mode is declared — ADR-012 §4):

• ensure_gc_schema() — create gc_pins (key, owner, expiry) and gc_window (key, visible_after) at store construction (store-core call, engine-neutral API; dialect-specific SQL bodies). owner is the store-instance id (ADR-011's vocabulary; ADR-008's "node id" wording predates that split). The pin row's identity is (key, owner) — a dedup re-put by a different instance is a different pin row; protection composes by row (an entry is protected while any un-expired pin row exists), which is ADR-005's refcount semantics expressed in rows. The embedder's liveness table is not created by this seam — ADR-008: schema pinned by the embedder; the sweep reads it and never owns it. Per-engine applicability (the test-gate shape; final per ADR-012 §5): sqlite — schema creates gc_window only + window_stage/window_take (durable single-node staging); the originally-specced sqlite pin clauses and sweep-lock body are dropped by ADR-012 §5 (the pin bodies inserted into a table nothing created — dead machinery; sqlite has no advisory lock, and ADR-008's unchanged single-node protocol keeps the sweep single-flighter in-process). postgres — all clause families, contract-tested, fleet-active (the fleet host). pg-lo — the window family only (durable window staging where the kv engine is not SQL-backed — the one-window-host rule, ADR-012 §5; it hosts no fleet pin/lock state, ADR-012 §4), contract-tested.
• pin_commit(joint_tx, entries) — the store core holds the transaction; the engine contributes its put mechanics (row shape, CAS clause, LO writes) and the pin-row inserts, and both commit together. This is the joint entry+pin transaction's owner, ruled: the store holds the tx; the engine contributes mechanics to it. The public put remains the plain, pin-free path — the store's normal put path routes through pin_commit wherever the engine is seam-capable and fleet mode is active (activation ruled by ADR-012 §3: fleet mode is an explicit fleet: true constructor declaration, never inferred from the engine choice; for large=local fleet puts the joint tx is unachievable across media — ADR-012 §4 pins pin-row-first, publish-second ordering).
  • pg-lo correction: ADR-009 §2's "put = one transaction" is unchanged in content but its ownership statement is hereby corrected — the tx is held by the store core; the pg-lo engine contributes its staged-put mechanics and companion-row insert into it. The engine sees pins only as rows; never as semantics.
• pin_renew(key, owner) — the engine executes the expiry update (renamed and TTL dropped by ADR-012 §2: the TTL lives with the store's renewal schedule, not in the clause signature; the original pin_renew(key, owner, ttl) read is superseded). The store's pin guard owns the TTL/3 renewal schedule (ADR-008); the engine owns only the SQL. (postgres host only per ADR-012 §5 — sqlite bodies dropped.)
• pin_check(key) → PinState / arbitrate(key) → bool — the engine executes ADR-008's delete-time arbitration (un-expired pin rows + embedded liveness tables, SELECT … FOR UPDATE finger-lock, keyed, at delete time); the store core receives the answer and applies the decision. PinState is a data enum with two variants: Protected (an un-expired pin row exists) and Unprotected — the engine never decides deletion, it only classifies; when the seam supports SQL-level liveness tables the store consults those through arbitrate instead, which folds the full answer into the bool. If fleet clauses ever grow, they grow here — the seam is the named home, with contract tests per clause.
• window_stage(candidates) / window_take(now) — the staged delete-window candidates. Per ADR-008 §2 the durable window table exists on every SQL-backed engine (final wording per ADR-012 §5: sqlite included; local and mem are excluded from the seam here and stage candidates in-process, executing the ADR-005 window protocol inline in the same sweep; in dual-SQL instances the non-host engine's window clauses are dormant — the one-window-host rule, ADR-012 §5): staged candidates are engine rows; what differs by topology is where liveness lives (in-process maps for single-node engines, tables for fleets), not whether staging exists among SQL-backed engines.
• sweep_lock_acquire()/release() — the sweep single-flighter: the pg advisory lock (ADR-008), applying to the fleet host (postgres) only — ADR-012 §5 drops the originally-specced sqlite body (no advisory-lock equivalent; in-process lock suffices and is ADR-008-verbatim). The store treats lock contention exactly as ADR-008 says — the second sweeper's sweep aborts with GcAbortCause::SweeperLock as report data (Ok(report) — ADR-012 §1 retired the typed GcAborted sweep error), nothing deleted.

Each clause is contract-tested per its applicability (the per-engine split above): the conformance suite of §1 grows the fleet clauses for postgres (the host); pg-lo runs the window-family contract tests; local and mem are excluded by construction.

3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)

A fleet constructor requires a pool-shared kv engine. Postgres is the shipped engine that satisfies this. A sqlite-fleet mechanism does not exist and is not on offer (per ADR-007's redb precedent this is recorded as a non-option, not an open question); two nodes' sqlite engines over "one pool" are two pools — the partitioning failure, the kv-tier edition of what ADR-008 §3 prevents for the large tier. The enforceable seam is the same as ADR-008 §3's: the constructor can require an explicit declaration; it cannot prove cross-node truth. The constructor table is pinned in ADR-011 and is closed under both tiers.

4. The ops pin-token contract's TTL race, resolved by the ownership rule

The token is the pin — there is no second pin state. Under fleet engines the put's pin is the expiry-carrying row (ADR-008); the token identifies the putter as the row's owner. The token itself is an opaque id minted by the ops layer over the pin row it just committed — the ops layer (not the store facade) owns the token↔(key, owner) mapping, in-process state with process lifetime (ADR-012 §2 corrected the original "handler's lifetime" — the handler's guard drops at response time, but the mapping must outlive it, or renewal could never arrive); no schema is specced for it beyond that (it is a handle, not durable state — the durable state is the pin row the seam wrote). The token's validity domain is the minting node (ADR-012 §2: renewal probes reach the serving node that minted the token; fleet peers refuse foreign tokens; a dead minting node is the ADR-005 crash case with the re-put backstop). Wire shapes (ADR-012 §2): the blobs/put response is {token, digest}; renewal rides the token form of blobs/have (one digest, token alongside). Concretely:

• When the server-side Pin guard drops: at response time. The put lands pinned (the joint tx guarantees the entry and its pin row commit together); the handler's in-process guard — the thing ADR-005 pins in-process with — is released when the blobs/put response completes and the token is handed out. From that moment the pin row's expiry is the only protection, which is exactly why the renewal duty below exists. (If the handler kept its guard for the token's whole life, the TTL machinery would be dead code on the ops path — the row's expiry, not the guard's presence, is the fleet contract's liveness signal.)
• The putter renews the pin by renewing the row — through ops it already speaks: a blobs/have probe whose handler, observing a valid token presented alongside the digest, calls the crate-internal pin_renew(key, owner) via the ops module's own store-core access (the ops module is in-crate — it crosses the crate-internal seam by construction, the same as the put handler holds the Pin guard; what never happens is a consumer reaching the seam. ADR-008's has remains a pure probe: the renewal is the ops layer's token-authenticated act on the pin row, not a has semantic), or a re-put of the same digest (CAS dedup; the re-put's joint tx upserts the key+owner pin row, renewing it). The renewal cadence is the one ADR-008 fixed on the pin guard: every TTL/3 — 20 s at the default 60 s TTL.
• A putter pausing past TTL falls to ADR-008's expiry semantics verbatim: the row is abandoned and reaped; the backstop is delete-then-recover (ADR-005) — the digest is re-offered, the put is idempotent under CAS. The race resolves to a re-put, never to silent loss.
• For the local engine the token is the in-process guard as ADR-005 had it: nothing persists, nothing expires; the process crash is ADR-005's in-flight-put loss case with the same backstop. (The cross-wire token under local is the ops layer's in-memory handle to that guard — same mapping discipline, no expiry.) ADR-012 §2 generalizes this to every non-fleet mode: the token is the in-process guard's handle on sqlite, single-instance postgres, and pg-lo serving nodes too — nothing expires outside fleet mode.

ops-surface.md's hand-over contract gains the renewal line and the token-mapping note; nothing else changes.

Consequences

Positive

• The trait's I/O shapes are decided before code; engine tasks can be written without guessing signatures. The per-engine conformance-suite shape (POC #7's 10/10 gate as the norm) gives decomposition a uniform test gate across all five engines.
• "Backends never learn liveness" survives the fleet mechanism intact; the store core owns GC state per ADR-005 exactly as written; ADR-008's mechanism lives in a named, tested seam instead of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing else" stays true of the public trait.
• The ops pin-token TTL race is resolved with no new ops; the backstop is already-validated semantics (delete-then-recover).
• The kv tier's fleet-validity rule closes the constructor table under both tiers.

Negative

• A crate-internal companion trait is real machinery: the full clause family on postgres (the fleet host — pins, window, sweep lock, contract-tested, fleet-active); sqlite and pg-lo carry the window family only (ADR-012 §5's reduction — the original draft's sqlite pin/sweep-lock bodies were dead machinery and are dropped). The seam bounds where fleet-GC growth lands; it does not shrink it.
• Remote putters gain a real protocol duty (renew past 20 s; pause-past-TTL ⇒ delete-then-recover; route renewal to the minting node — ADR-012 §2) — documented, but a duty.

Neutral

• The Backend trait's method set is unchanged from ADR-003/008 — no public API change beyond what store-api.md already specced; the cursor/staging abstractions are crate-internal.
• The staged-put contract already existed behaviorally (POC #3 A5); this ADR names it.
• ADR-009's engine content is unchanged; only its tx-ownership statement is corrected here.

References

• ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004 (tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the liveness-ownership rule this ADR preserves), ADR-006 (the range contract the cursor serves), ADR-008 (the fleet mechanism whose state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership correction), ADR-011 (vocabulary + constructor table), ADR-012 (fleet activation predicate, GC-state host, sweep-abort shape, token wire shape, the §5 clause reduction this ADR's seam table now references)
• docs/research/poc-trait-dispatch-findings.md finding 1; docs/research/poc-largeblob-findings.md findings A1/A5/A6; docs/research/poc-pglo-findings.md (C1-C7 — the handle shape, the 512 KiB knee, windowed gets)
• ops-surface.md (the pin-token contract §4 extends); store-api.md (stage hygiene, sweep semantics, cursor-adjacent facade shapes); requirements.md (REQ-2, the fleet requirement §2/§3 serve)
• ADR-013 (family composition — §4's feature-graph extension of this ADR's §1 consequence; §2's sweep-lease semantics for this ADR's sweep-lock clause)