alkblobs/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md

# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam

## Status

Accepted (fleet-state activation, the GC-state host under mixed
engines, the sweep-abort return shape, the pin token's wire shape,
the sqlite pin/sweep-lock clause reductions, and the window staging
for non-SQL engines are ruled by [ADR-012](012-pre-decomposition-consistency-rulings.md);
§2's sqlite rows and the "fleet state is active" predicate read
through that ADR — where noted inline)

## Context

The engine story closed at the decision level (ADR-007/008/009), but
three seams that implementation must not guess are unpinned. All three
concentrate where the tier/engine split stretched the original
contract:

1. **The `get`/`put` value shapes were never unified.** The trait POC's
   `get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
   a file handle, not a loaded buffer"; the pg-lo POC's get is a
   `LoGetHandle` driving `lo_get(oid, off, len)` windows — three
   structurally different shapes behind one trait, and no document
   named the unified contract. Same on the put side: a byte slice
   (kv), a stage-then-commit-rename stream (`local`), and a
   chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
   trait is the crate's second stable seam (ADR-003); its I/O shapes
   are decided before code, not discovered per engine.

2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
   rows committed with the entry; staged delete-window candidates
   (including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
   advisory lock for the sweeper. None of that is expressible through
   the public trait method set, and ADR-003's "the shipped trait
   impls write plain content-addressed rows into the chosen engine,
   nothing else" reads as a bar the fleet mechanism violates. Also
   unresolved: who owns the joint entry+pin transaction — ADR-009
   says the pg-lo put's single tx carries the pin row, but ADR-005
   says `Pin` is a store-core RAII guard under the arbitration lock.
   The trait has no parameter for either, and "the backend learns
   about pins" would contradict ADR-005. The ops pin-token contract
   (ops-surface.md) inherits the same conflict: a remote putter holds
   a token, but an expiry-guarded pin row goes stale if the holder
   pauses — a sweep could delete a remotely-in-flight put whose row
   expired.

3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
   the large tier's fleet forms (shared media, re-routing) but pins
   nothing for the kv tier, which the fleet story centers on: two
   nodes' sqlite engines are trivially partitioned (two files = two
   pools, not one), and no document said so.

The deciding facts are in hand (Schrödinger's-code rule — none of
these may wait on consumers): the POCs fixed every engine's native
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
advisory lock, staged window); ADR-001 §4 fixed that ops is a
store-facade consumer with no backend side-door. What is missing is
composition: how the store core drives all engines through one
contract while owning GC state.

Alternatives considered for the fleet-state seam, rejected:

- **Put the fleet machinery on the public trait** (pin/window/table
  methods on `Backend`): every engine — including `mem` and `local` —
  must then implement SQL-shaped machinery it cannot express. Rejected.
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
  row, backend-maintained): contradicts ADR-005's "backends never
  learn liveness"; and the arbitration decision (expiry compare,
  liveness consult) still belongs to the store, so the store ends up
  reaching into the engine's tx anyway. Rejected.
- **A raw-SQL side-channel** (connection escape hatch the store core
  may use arbitrarily): turns the engine abstraction off for exactly
  the machinery (fleet GC) it most needs to be sound on. Rejected.
  The seam below is that escape hatch made honest — named clauses,
  contract-tested per engine, instead of raw access.

## Decision

### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put

**The trait stays one trait.** Every engine implements the same method
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
What changes: the trait's I/O types are the crate-internal
abstractions below, which each engine constructs its own way — the
store core programs the abstraction, never the engine shape.

- **`get` returns a read cursor.** One abstraction, two body families:
  - **kv engines** (sqlite, postgres, mem): the cursor materializes
    the whole value (the engine's row read; bounded by the small-blob
    tier policy). Its range reads are in-memory slices; `len()` reads
    the row's `size` column.
  - **large engines** (`local`, pg-lo): the cursor streams; the medium
    is hidden in the cursor body — `local`'s is a pread loop over the
    committed file (immutable once committed); pg-lo's drives
    descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
    posture — pool-friendly, stateless per window; the
    tx-scoped-descriptor form is the fallback). `len()` reads the
    companion row or stats the file — metadata only, never content
    (ADR-008's discipline).
  - Both body families serve the ADR-006 range contract (`read_range`
    → slice + slice digest) at one seam. The cursor is
    **store-internal — not public API**: store consumers see the
    facade's `(len, stream)` shapes (store-api.md) with the cursor
    staying behind them; the store's stream utility adapts engine
    cursor bodies, it does not expose them.
- **`put` exists in two forms, selected by the store core before
  content flows; both land in the CAS entry shape and both
  participate in pin-before-publish (§2 decides who holds the tx).**
  - **Whole-value put** for the kv tier: the settled row write.
    POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
    `ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
  - **Staged put** for the large tier — ADR-003's
    stage-then-commit-rename promoted from behavior to a **named
    per-engine contract**: the engine hands out a *staging cursor*
    (`local`: the stage file under the store's stage-hygiene protocol;
    pg-lo: chunked `lowrite` statements over a tx the store core
    holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
    measured knee). Commit publishes the entry;
    every failure path converges on discard (store-api.md stage
    hygiene, per engine, exact-count tested — POC #3 A5's shape).
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
  walks the engine's companion structure (rows / files / LO
  companion); `size` is metadata-only (ADR-008). The exact-count
  sweep-outcome gate (ADR-003) runs per engine.

**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
`pg-lo` does **not** imply feature `postgres`. Each is an engine
feature pulling the shared driver cluster (`tokio-postgres` +
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
node is coherent and supported. overview.md's dependency table pins
the edge. *(The same independent-driver discipline is extended to
every engine by ADR-013 §4 — `sqlite` becomes its own default-on
feature carrying rusqlite, the `kv`/`large` tier features retire, and
pg-pure builds become supported, non-default.)*

### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested

**Ownership: the store core owns all GC state — in-process pin and
liveness maps *and* the fleet state's rows and locks. The public
`Backend` trait stays exactly what ADR-003 says (opaque
`hash → bytes`, namespace-blind; "plain content-addressed rows,
nothing else" remains true of the public trait). Backends never learn
liveness (ADR-005) — verbatim.**

The store core reaches the fleet machinery through a **crate-internal
(not exported) engine-state companion trait**, implemented by the
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
`mem` excluded by construction (it cannot be expressed on them and
they are never fleet engines). ADR-008's mechanism fixes *what*; the
seam is *where* the mechanism's SQL lives. Clauses (per-engine
applicability is ruled by [ADR-012](012-pre-decomposition-consistency-rulings.md)
§5's table, which reduces the original sqlite rows; fleet state hosts
on the kv-tier engine when fleet mode is declared — ADR-012 §4):

  - `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
    `gc_window (key, visible_after)` at store construction (store-core
    call, engine-neutral API; dialect-specific SQL bodies). `owner`
    is the **store-instance id** (ADR-011's vocabulary; ADR-008's
    "node id" wording predates that split). The pin row's identity is
    `(key, owner)` — a dedup re-put by a different instance is a
    different pin row; protection composes by row (an entry is
    protected while any un-expired pin row exists), which is
    ADR-005's refcount semantics expressed in rows. The embedder's
    liveness table is *not* created by this seam — ADR-008: schema
    pinned by the embedder; the sweep reads it and never owns it.
    **Per-engine applicability (the test-gate shape; final per
    ADR-012 §5):** sqlite — schema creates **`gc_window` only** +
    `window_stage`/`window_take` (durable single-node staging);
    the originally-specced sqlite pin clauses and sweep-lock body
    are **dropped** by ADR-012 §5 (the pin bodies inserted into a
    table nothing created — dead machinery; sqlite has no advisory
    lock, and ADR-008's unchanged single-node protocol keeps the
    sweep single-flighter in-process). postgres — all clause
    families, contract-tested, fleet-active (the fleet host). pg-lo —
    the window family only (durable window staging where the kv
    engine is not SQL-backed — the one-window-host rule, ADR-012 §5;
    it hosts no fleet pin/lock state,
    ADR-012 §4), contract-tested.
- `pin_commit(joint_tx, entries)` — the store core **holds the
  transaction**; the engine contributes its put mechanics (row
  shape, CAS clause, LO writes) and the pin-row inserts, and both
  commit together. This is the joint entry+pin transaction's owner,
  ruled: **the store holds the tx; the engine contributes mechanics
  to it.** The public `put` remains the plain, pin-free path — the
  store's normal put path routes through `pin_commit` wherever the
  engine is seam-capable and fleet mode is active (**activation
  ruled by ADR-012 §3: fleet mode is an explicit `fleet: true`
  constructor declaration, never inferred from the engine choice**;
  for large=local fleet puts the joint tx is unachievable across
  media — ADR-012 §4 pins pin-row-first, publish-second ordering).
  - pg-lo correction: ADR-009 §2's "put = one transaction" is
    unchanged in content but its ownership statement is hereby
    corrected — the tx is held by the store core; the pg-lo engine
    contributes its staged-put mechanics and companion-row insert
    into it. The engine sees pins only as rows; never as semantics.
- `pin_renew(key, owner)` — the engine executes the expiry update
  (renamed and TTL dropped by ADR-012 §2: the TTL lives with the
  store's renewal schedule, not in the clause signature; the original
  `pin_renew(key, owner, ttl)` read is superseded). The store's pin
  guard owns the TTL/3 renewal schedule
  (ADR-008); the engine owns only the SQL. *(postgres host only per
  ADR-012 §5 — sqlite bodies dropped.)*
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
  executes ADR-008's delete-time arbitration (un-expired pin rows +
  embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
  keyed, at delete time); the store core receives the answer and
  applies the decision. `PinState` is a data enum with two variants:
  `Protected` (an un-expired pin row exists) and `Unprotected` — the
  engine never decides deletion, it only classifies; *when the seam
  supports SQL-level liveness tables the store consults those through
  `arbitrate` instead, which folds the full answer into the bool*. If
  fleet clauses ever grow, they grow here —
  the seam is the named home, with contract tests per clause.
- `window_stage(candidates)` / `window_take(now)` — the staged
  delete-window candidates. Per ADR-008 §2 the durable window table
  exists on **every SQL-backed engine** (final wording per ADR-012
  §5: sqlite included; `local` and `mem` are excluded from the seam
  here and stage candidates in-process, executing the ADR-005 window
  protocol inline in the same sweep; in dual-SQL instances the
  non-host engine's window clauses are dormant — the one-window-host
  rule, ADR-012 §5): staged candidates are
  engine rows; what differs by topology is where *liveness* lives
  (in-process maps for single-node engines, tables for fleets), not
  whether staging exists among SQL-backed engines.
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
  pg advisory lock (ADR-008), applying to the fleet host (postgres)
  only — ADR-012 §5 drops the originally-specced sqlite body (no
  advisory-lock equivalent; in-process lock suffices and is
  ADR-008-verbatim). The
  store treats lock contention exactly as ADR-008 says — the second
  sweeper's sweep aborts with **`GcAbortCause::SweeperLock` as report
  data (`Ok(report)` — ADR-012 §1 retired the typed `GcAborted` sweep
  error)**, nothing deleted.

Each clause is contract-tested per its applicability (the
per-engine split above): the conformance suite of §1 grows the fleet
clauses for postgres (the host); pg-lo runs the window-family
contract tests; `local` and
`mem` are excluded by construction.

### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)

**A fleet constructor requires a pool-shared kv engine.** Postgres is
the shipped engine that satisfies this. A sqlite-fleet mechanism does
not exist and is not on offer (per ADR-007's redb precedent this is
recorded as a non-option, not an open question); two nodes' sqlite
engines over "one pool" are two pools — the partitioning failure, the
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
enforceable
seam is the same as ADR-008 §3's: the constructor can require an
explicit declaration; it cannot prove cross-node truth. The
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
and is closed under both tiers.

### 4. The ops pin-token contract's TTL race, resolved by the ownership rule

**The token *is* the pin — there is no second pin state.** Under fleet
engines the put's pin is the expiry-carrying row (ADR-008); the token
identifies the putter as the row's owner. The token itself is an
**opaque id minted by the ops layer** over the pin row it just
committed — the ops layer (not the store facade) owns the
token↔(key, owner) mapping, in-process state with **process lifetime**
(ADR-012 §2 corrected the original "handler's lifetime" — the handler's
guard drops at response time, but the mapping must outlive it, or
renewal could never arrive); no schema is specced
for it beyond that
(it is a handle, not durable state — the durable state is the pin row
the seam wrote). The token's **validity domain is the minting node**
(ADR-012 §2: renewal probes reach the serving node that minted the
token; fleet peers refuse foreign tokens; a dead minting node is the
ADR-005 crash case with the re-put backstop).
**Wire shapes (ADR-012 §2):** the `blobs/put` response is
`{token, digest}`; renewal rides the token form of `blobs/have`
(one digest, token alongside). Concretely:

- **When the server-side `Pin` guard drops:** at response time. The
  put lands pinned (the joint tx guarantees the entry and its pin row
  commit together); the handler's in-process guard — the thing ADR-005
  pins *in-process* with — is released when the `blobs/put` response
  completes and the token is handed out. From that moment the pin
  row's expiry is the only protection, which is exactly why the
  renewal duty below exists. (If the handler kept its guard for the
  token's whole life, the TTL machinery would be dead code on the ops
  path — the row's expiry, not the guard's presence, is the fleet
  contract's liveness signal.)
- The putter renews the pin by renewing the row — through ops it
  already speaks: a `blobs/have` probe whose handler, observing a
  valid token presented alongside the digest, calls the
  crate-internal `pin_renew(key, owner)` via the ops module's own
  store-core access (the ops module is in-crate — it crosses the
  crate-internal seam by construction, the same as the put handler
  holds the `Pin` guard; what never happens is a *consumer* reaching
  the seam. ADR-008's `has` remains a pure probe: the renewal is the
  ops layer's token-authenticated act on the pin row, not a `has`
  semantic), or a re-put of the same digest (CAS dedup; the re-put's
  joint tx upserts the key+owner pin row, renewing it). The renewal
  cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
  20 s at the default 60 s TTL.**
- A putter pausing past TTL falls to ADR-008's expiry semantics
  verbatim: the row is abandoned and reaped; the backstop is
  delete-then-recover (ADR-005) — the digest is re-offered, the put
  is idempotent under CAS. The race resolves to a re-put, never to
  silent loss.
- For the `local` engine the token is the in-process guard as ADR-005
  had it: nothing persists, nothing expires; the process crash is
  ADR-005's in-flight-put loss case with the same backstop. (The
  cross-wire token under `local` is the ops layer's in-memory handle
  to that guard — same mapping discipline, no expiry.) ADR-012 §2
  generalizes this to every non-fleet mode: the token is the
  in-process guard's handle on sqlite, single-instance postgres, and
  pg-lo serving nodes too — nothing expires outside fleet mode.

ops-surface.md's hand-over contract gains the renewal line and the
token-mapping note; nothing else changes.

## Consequences

**Positive**

- The trait's I/O shapes are decided before code; engine tasks can be
  written without guessing signatures. The per-engine
  conformance-suite shape (POC #7's 10/10 gate as the norm) gives
  decomposition a uniform test gate across all five engines.
- "Backends never learn liveness" survives the fleet mechanism
  intact; the store core owns GC state per ADR-005 exactly as
  written; ADR-008's mechanism lives in a named, tested seam instead
  of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
  else" stays true of the public trait.
- The ops pin-token TTL race is resolved with no new ops; the
  backstop is already-validated semantics (delete-then-recover).
- The kv tier's fleet-validity rule closes the constructor table
  under both tiers.

**Negative**

- A crate-internal companion trait is real machinery: the full clause
  family on postgres (the fleet host — pins, window, sweep lock,
  contract-tested, fleet-active); sqlite and pg-lo carry the window
  family only (ADR-012 §5's reduction — the original draft's sqlite
  pin/sweep-lock bodies were dead machinery and are dropped). The
  seam bounds where fleet-GC growth lands; it does not shrink it.
- Remote putters gain a real protocol duty (renew past 20 s;
  pause-past-TTL ⇒ delete-then-recover; route renewal to the minting
  node — ADR-012 §2) — documented, but a duty.

**Neutral**

- The `Backend` trait's method set is unchanged from ADR-003/008 —
  no public API change beyond what store-api.md already specced; the
  cursor/staging abstractions are crate-internal.
- The staged-put contract already existed behaviorally (POC #3 A5);
  this ADR names it.
- ADR-009's engine content is unchanged; only its tx-ownership
  statement is corrected here.

## References

- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
  (tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
  liveness-ownership rule this ADR preserves), ADR-006 (the range
  contract the cursor serves), ADR-008 (the fleet mechanism whose
  state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
  correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
  (vocabulary + constructor table), [ADR-012](012-pre-decomposition-consistency-rulings.md)
  (fleet activation predicate, GC-state host, sweep-abort shape,
  token wire shape, the §5 clause reduction this ADR's seam table
  now references)
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
  `docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
  `docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
  512 KiB knee, windowed gets)
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
  extends); [store-api.md](../store-api.md) (stage hygiene, sweep
  semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
  (REQ-2, the fleet requirement §2/§3 serve)
- [ADR-013](013-family-composition-reactive-boundary-and-feature-graph.md)
  (family composition — §4's feature-graph extension of this ADR's §1
  consequence; §2's sweep-lease semantics for this ADR's sweep-lock
  clause)