alkblobs/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md

ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam

Status

Accepted

Context

The engine story closed at the decision level (ADR-007/008/009), but three seams that implementation must not guess are unpinned. All three concentrate where the tier/engine split stretched the original contract:

1. The get/put value shapes were never unified. The trait POC's get returns Option<Vec<u8>>; the largeblob POC's fs arm "yields a file handle, not a loaded buffer"; the pg-lo POC's get is a LoGetHandle driving lo_get(oid, off, len) windows — three structurally different shapes behind one trait, and no document named the unified contract. Same on the put side: a byte slice (kv), a stage-then-commit-rename stream (local), and a chunked-lowrite transaction (pg-lo) are all "the put". A unified trait is the crate's second stable seam (ADR-003); its I/O shapes are decided before code, not discovered per engine.

2. ADR-008's fleet state has no seam. Pins become expiry-carrying rows committed with the entry; staged delete-window candidates (including on sqlite); SELECT … FOR UPDATE arbitration; a pg advisory lock for the sweeper. None of that is expressible through the public trait method set, and ADR-003's "the shipped trait impls write plain content-addressed rows into the chosen engine, nothing else" reads as a bar the fleet mechanism violates. Also unresolved: who owns the joint entry+pin transaction — ADR-009 says the pg-lo put's single tx carries the pin row, but ADR-005 says Pin is a store-core RAII guard under the arbitration lock. The trait has no parameter for either, and "the backend learns about pins" would contradict ADR-005. The ops pin-token contract (ops-surface.md) inherits the same conflict: a remote putter holds a token, but an expiry-guarded pin row goes stale if the holder pauses — a sweep could delete a remotely-in-flight put whose row expired.

3. The kv tier had no fleet-validity rule. ADR-008 §3 constrains the large tier's fleet forms (shared media, re-routing) but pins nothing for the kv tier, which the fleet story centers on: two nodes' sqlite engines are trivially partitioned (two files = two pools, not one), and no document said so.

The deciding facts are in hand (Schrödinger's-code rule — none of these may wait on consumers): the POCs fixed every engine's native shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry, advisory lock, staged window); ADR-001 §4 fixed that ops is a store-facade consumer with no backend side-door. What is missing is composition: how the store core drives all engines through one contract while owning GC state.

Alternatives considered for the fleet-state seam, rejected:

• Put the fleet machinery on the public trait (pin/window/table methods on Backend): every engine — including mem and local — must then implement SQL-shaped machinery it cannot express. Rejected.
• Backends grow pin awareness (the pg-lo put tx carries the pin row, backend-maintained): contradicts ADR-005's "backends never learn liveness"; and the arbitration decision (expiry compare, liveness consult) still belongs to the store, so the store ends up reaching into the engine's tx anyway. Rejected.
• A raw-SQL side-channel (connection escape hatch the store core may use arbitrarily): turns the engine abstraction off for exactly the machinery (fleet GC) it most needs to be sound on. Rejected. The seam below is that escape hatch made honest — named clauses, contract-tested per engine, instead of raw access.

Decision

1. One Backend trait; two I/O abstractions per tier — read cursor and staged put

The trait stays one trait. Every engine implements the same method set (ADR-003's methods + ADR-008's size) — no per-tier trait split. What changes: the trait's I/O types are the crate-internal abstractions below, which each engine constructs its own way — the store core programs the abstraction, never the engine shape.

• get returns a read cursor. One abstraction, two body families:
  • kv engines (sqlite, postgres, mem): the cursor materializes the whole value (the engine's row read; bounded by the small-blob tier policy). Its range reads are in-memory slices; len() reads the row's size column.
  • large engines (local, pg-lo): the cursor streams; the medium is hidden in the cursor body — local's is a pread loop over the committed file (immutable once committed); pg-lo's drives descriptorless lo_get(oid, off, len) windows (ADR-009 §3's C4 posture — pool-friendly, stateless per window; the tx-scoped-descriptor form is the fallback). len() reads the companion row or stats the file — metadata only, never content (ADR-008's discipline).
  • Both body families serve the ADR-006 range contract (read_range → slice + slice digest) at one seam. The cursor is store-internal — not public API: store consumers see the facade's (len, stream) shapes (store-api.md) with the cursor staying behind them; the store's stream utility adapts engine cursor bodies, it does not expose them.
• put exists in two forms, selected by the store core before content flows; both land in the CAS entry shape and both participate in pin-before-publish (§2 decides who holds the tx).
  • Whole-value put for the kv tier: the settled row write. POC-shaped bodies: sqlite INSERT OR IGNORE, postgres ON CONFLICT DO NOTHING, mem entry().or_insert().
  • Staged put for the large tier — ADR-003's stage-then-commit-rename promoted from behavior to a named per-engine contract: the engine hands out a staging cursor (local: the stage file under the store's stage-hygiene protocol; pg-lo: chunked lowrite statements over a tx the store core holds — §2's pin_commit is that holder; 512 KiB per POC #7's measured knee). Commit publishes the entry; every failure path converges on discard (store-api.md stage hygiene, per engine, exact-count tested — POC #3 A5's shape).
• list/delete/has/size/name are tier-agnostic. list walks the engine's companion structure (rows / files / LO companion); size is metadata-only (ADR-008). The exact-count sweep-outcome gate (ADR-003) runs per engine.

Feature-graph consequence (fills a gap ADR-007/009 left): feature pg-lo does not imply feature postgres. Each is an engine feature pulling the shared driver cluster (tokio-postgres + deadpool-postgres) independently — a large=pg-lo + kv=sqlite node is coherent and supported. overview.md's dependency table pins the edge.

2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested

Ownership: the store core owns all GC state — in-process pin and liveness maps and the fleet state's rows and locks. The public Backend trait stays exactly what ADR-003 says (opaque hash → bytes, namespace-blind; "plain content-addressed rows, nothing else" remains true of the public trait). Backends never learn liveness (ADR-005) — verbatim.

The store core reaches the fleet machinery through a crate-internal (not exported) engine-state companion trait, implemented by the SQL-backed engines (sqlite, postgres, pg-lo), with local and mem excluded by construction (it cannot be expressed on them and they are never fleet engines). ADR-008's mechanism fixes what; the seam is where the mechanism's SQL lives. Clauses:

• ensure_gc_schema() — create gc_pins (key, owner, expiry) and gc_window (key, visible_after) at store construction (store-core call, engine-neutral API; dialect-specific SQL bodies). owner is the store-instance id (ADR-011's vocabulary; ADR-008's "node id" wording predates that split). The pin row's identity is (key, owner) — a dedup re-put by a different instance is a different pin row; protection composes by row (an entry is protected while any un-expired pin row exists), which is ADR-005's refcount semantics expressed in rows. The embedder's liveness table is not created by this seam — ADR-008: schema pinned by the embedder; the sweep reads it and never owns it. Per-engine applicability (the test-gate shape): sqlite — ensure_gc_schema (window table only, since the window staging is shared machinery per ADR-008 §2) + window_stage/ window_take + the sweep-lock clause's single-node body; the pin clauses (pin_commit/pin_renew/pin_check) have sqlite bodies but are exercised by unit tests only — they are dead machinery on sqlite because no sqlite fleet is on offer (§3). postgres and pg-lo — all clause families, contract-tested, fleet-active.
• pin_commit(joint_tx, entries) — the store core holds the transaction; the engine contributes its put mechanics (row shape, CAS clause, LO writes) and the pin-row inserts, and both commit together. This is the joint entry+pin transaction's owner, ruled: the store holds the tx; the engine contributes mechanics to it. The public put remains the plain, pin-free path — the store's normal put path routes through pin_commit wherever the engine is seam-capable and fleet state is active.
  • pg-lo correction: ADR-009 §2's "put = one transaction" is unchanged in content but its ownership statement is hereby corrected — the tx is held by the store core; the pg-lo engine contributes its staged-put mechanics and companion-row insert into it. The engine sees pins only as rows; never as semantics.
• pin_renew(key, owner, ttl) — the engine executes the expiry update. The store's pin guard owns the TTL/3 renewal schedule (ADR-008); the engine owns only the SQL.
• pin_check(key) → PinState / arbitrate(key) → bool — the engine executes ADR-008's delete-time arbitration (un-expired pin rows + embedded liveness tables, SELECT … FOR UPDATE finger-lock, keyed, at delete time); the store core receives the answer and applies the decision. If fleet clauses ever grow, they grow here — the seam is the named home, with contract tests per clause.
• window_stage(candidates) / window_take(now) — the staged delete-window candidates. Per ADR-008 §2 the window table exists on every engine (sqlite included): staged candidates are engine rows; what differs by topology is where liveness lives (in-process maps for single-node engines, tables for fleets), not whether staging exists.
• sweep_lock_acquire()/release() — the sweep single-flighter: the pg advisory lock (ADR-008) for the SQL engines it applies to; the store treats lock contention exactly as ADR-008 says — the second sweeper's sweep aborts with the typed GcAborted (fleet cause), nothing deleted. (SQLite today has no fleet story — see §3 — so its clause body exists but is exercised by single-node tests only.)

Each clause is contract-tested per its applicability (the per-engine split above): the conformance suite of §1 grows the fleet clauses for postgres and pg-lo; local and mem are excluded by construction.

3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)

A fleet constructor requires a pool-shared kv engine. Postgres is the shipped engine that satisfies this. A sqlite-fleet mechanism does not exist and is not on offer (per ADR-007's redb precedent this is recorded as a non-option, not an open question); two nodes' sqlite engines over "one pool" are two pools — the partitioning failure, the kv-tier edition of what ADR-008 §3 prevents for the large tier. The enforceable seam is the same as ADR-008 §3's: the constructor can require an explicit declaration; it cannot prove cross-node truth. The constructor table is pinned in ADR-011 and is closed under both tiers.

4. The ops pin-token contract's TTL race, resolved by the ownership rule

The token is the pin — there is no second pin state. Under fleet engines the put's pin is the expiry-carrying row (ADR-008); the token identifies the putter as the row's owner. The token itself is an opaque id minted by the ops layer over the pin row it just committed — the ops layer (not the store facade) owns the token↔(key, owner) mapping, in-memory for the handler's lifetime plus a lookup by the renewal op; no schema is specced for it beyond that (it is a handle, not durable state — the durable state is the pin row the seam wrote). Concretely:

• When the server-side Pin guard drops: at response time. The put lands pinned (the joint tx guarantees the entry and its pin row commit together); the handler's in-process guard — the thing ADR-005 pins in-process with — is released when the blobs/put response completes and the token is handed out. From that moment the pin row's expiry is the only protection, which is exactly why the renewal duty below exists. (If the handler kept its guard for the token's whole life, the TTL machinery would be dead code on the ops path — the row's expiry, not the guard's presence, is the fleet contract's liveness signal.)
• The putter renews the pin by renewing the row — through ops it already speaks: a blobs/have probe whose handler, observing a valid token presented alongside the digest, calls the crate-internal pin_renew(key, owner) via the ops module's own store-core access (the ops module is in-crate — it crosses the crate-internal seam by construction, the same as the put handler holds the Pin guard; what never happens is a consumer reaching the seam. ADR-008's has remains a pure probe: the renewal is the ops layer's token-authenticated act on the pin row, not a has semantic), or a re-put of the same digest (CAS dedup; the re-put's joint tx upserts the key+owner pin row, renewing it). The renewal cadence is the one ADR-008 fixed on the pin guard: every TTL/3 — 20 s at the default 60 s TTL.
• A putter pausing past TTL falls to ADR-008's expiry semantics verbatim: the row is abandoned and reaped; the backstop is delete-then-recover (ADR-005) — the digest is re-offered, the put is idempotent under CAS. The race resolves to a re-put, never to silent loss.
• For the local engine the token is the in-process guard as ADR-005 had it: nothing persists, nothing expires; the process crash is ADR-005's in-flight-put loss case with the same backstop. (The cross-wire token under local is the ops layer's in-memory handle to that guard — same mapping discipline, no expiry.)

ops-surface.md's hand-over contract gains the renewal line and the token-mapping note; nothing else changes.

Consequences

Positive

• The trait's I/O shapes are decided before code; engine tasks can be written without guessing signatures. The per-engine conformance-suite shape (POC #7's 10/10 gate as the norm) gives decomposition a uniform test gate across all five engines.
• "Backends never learn liveness" survives the fleet mechanism intact; the store core owns GC state per ADR-005 exactly as written; ADR-008's mechanism lives in a named, tested seam instead of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing else" stays true of the public trait.
• The ops pin-token TTL race is resolved with no new ops; the backstop is already-validated semantics (delete-then-recover).
• The kv tier's fleet-validity rule closes the constructor table under both tiers.

Negative

• A crate-internal companion trait is real machinery: six clause families on postgres/pg-lo (fleet-active, all contract-tested); sqlite carries only the window + lock clauses as fleet-relevant machinery (its pin clauses have unit-test bodies but no fleet mode on offer — §3). The seam bounds where fleet-GC growth lands; it does not shrink it.
• Remote putters gain a real protocol duty (renew past 20 s; pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
• The sqlite sweep-lock clause body exists but is exercised only by single-node tests until a sqlite-fleet posture is ever decided (it is not on offer — §3).

Neutral

• The Backend trait's method set is unchanged from ADR-003/008 — no public API change beyond what store-api.md already specced; the cursor/staging abstractions are crate-internal.
• The staged-put contract already existed behaviorally (POC #3 A5); this ADR names it.
• ADR-009's engine content is unchanged; only its tx-ownership statement is corrected here.

References

• ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004 (tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the liveness-ownership rule this ADR preserves), ADR-006 (the range contract the cursor serves), ADR-008 (the fleet mechanism whose state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership correction), ADR-011 (vocabulary + constructor table)
• docs/research/poc-trait-dispatch-findings.md finding 1; docs/research/poc-largeblob-findings.md findings A1/A5/A6; docs/research/poc-pglo-findings.md (C1-C7 — the handle shape, the 512 KiB knee, windowed gets)
• ops-surface.md (the pin-token contract §4 extends); store-api.md (stage hygiene, sweep semantics, cursor-adjacent facade shapes); requirements.md (REQ-2, the fleet requirement §2/§3 serve)