alkblobs/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md

# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam

## Status

Accepted

## Context

The engine story closed at the decision level (ADR-007/008/009), but
three seams that implementation must not guess are unpinned. All three
concentrate where the tier/engine split stretched the original
contract:

1. **The `get`/`put` value shapes were never unified.** The trait POC's
   `get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
   a file handle, not a loaded buffer"; the pg-lo POC's get is a
   `LoGetHandle` driving `lo_get(oid, off, len)` windows — three
   structurally different shapes behind one trait, and no document
   named the unified contract. Same on the put side: a byte slice
   (kv), a stage-then-commit-rename stream (`local`), and a
   chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
   trait is the crate's second stable seam (ADR-003); its I/O shapes
   are decided before code, not discovered per engine.

2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
   rows committed with the entry; staged delete-window candidates
   (including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
   advisory lock for the sweeper. None of that is expressible through
   the public trait method set, and ADR-003's "the shipped trait
   impls write plain content-addressed rows into the chosen engine,
   nothing else" reads as a bar the fleet mechanism violates. Also
   unresolved: who owns the joint entry+pin transaction — ADR-009
   says the pg-lo put's single tx carries the pin row, but ADR-005
   says `Pin` is a store-core RAII guard under the arbitration lock.
   The trait has no parameter for either, and "the backend learns
   about pins" would contradict ADR-005. The ops pin-token contract
   (ops-surface.md) inherits the same conflict: a remote putter holds
   a token, but an expiry-guarded pin row goes stale if the holder
   pauses — a sweep could delete a remotely-in-flight put whose row
   expired.

3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
   the large tier's fleet forms (shared media, re-routing) but pins
   nothing for the kv tier, which the fleet story centers on: two
   nodes' sqlite engines are trivially partitioned (two files = two
   pools, not one), and no document said so.

The deciding facts are in hand (Schrödinger's-code rule — none of
these may wait on consumers): the POCs fixed every engine's native
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
advisory lock, staged window); ADR-001 §4 fixed that ops is a
store-facade consumer with no backend side-door. What is missing is
composition: how the store core drives all engines through one
contract while owning GC state.

Alternatives considered for the fleet-state seam, rejected:

- **Put the fleet machinery on the public trait** (pin/window/table
  methods on `Backend`): every engine — including `mem` and `local` —
  must then implement SQL-shaped machinery it cannot express. Rejected.
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
  row, backend-maintained): contradicts ADR-005's "backends never
  learn liveness"; and the arbitration decision (expiry compare,
  liveness consult) still belongs to the store, so the store ends up
  reaching into the engine's tx anyway. Rejected.
- **A raw-SQL side-channel** (connection escape hatch the store core
  may use arbitrarily): turns the engine abstraction off for exactly
  the machinery (fleet GC) it most needs to be sound on. Rejected.
  The seam below is that escape hatch made honest — named clauses,
  contract-tested per engine, instead of raw access.

## Decision

### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put

**The trait stays one trait.** Every engine implements the same method
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
What changes: the trait's I/O types are the crate-internal
abstractions below, which each engine constructs its own way — the
store core programs the abstraction, never the engine shape.

- **`get` returns a read cursor.** One abstraction, two body families:
  - **kv engines** (sqlite, postgres, mem): the cursor materializes
    the whole value (the engine's row read; bounded by the small-blob
    tier policy). Its range reads are in-memory slices; `len()` reads
    the row's `size` column.
  - **large engines** (`local`, pg-lo): the cursor streams; the medium
    is hidden in the cursor body — `local`'s is a pread loop over the
    committed file (immutable once committed); pg-lo's drives
    descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
    posture — pool-friendly, stateless per window; the
    tx-scoped-descriptor form is the fallback). `len()` reads the
    companion row or stats the file — metadata only, never content
    (ADR-008's discipline).
  - Both body families serve the ADR-006 range contract (`read_range`
    → slice + slice digest) at one seam. The cursor is
    **store-internal — not public API**: store consumers see the
    facade's `(len, stream)` shapes (store-api.md) with the cursor
    staying behind them; the store's stream utility adapts engine
    cursor bodies, it does not expose them.
- **`put` exists in two forms, selected by the store core before
  content flows; both land in the CAS entry shape and both
  participate in pin-before-publish (§2 decides who holds the tx).**
  - **Whole-value put** for the kv tier: the settled row write.
    POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
    `ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
  - **Staged put** for the large tier — ADR-003's
    stage-then-commit-rename promoted from behavior to a **named
    per-engine contract**: the engine hands out a *staging cursor*
    (`local`: the stage file under the store's stage-hygiene protocol;
    pg-lo: chunked `lowrite` statements over a tx the store core
    holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
    measured knee). Commit publishes the entry;
    every failure path converges on discard (store-api.md stage
    hygiene, per engine, exact-count tested — POC #3 A5's shape).
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
  walks the engine's companion structure (rows / files / LO
  companion); `size` is metadata-only (ADR-008). The exact-count
  sweep-outcome gate (ADR-003) runs per engine.

**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
`pg-lo` does **not** imply feature `postgres`. Each is an engine
feature pulling the shared driver cluster (`tokio-postgres` +
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
node is coherent and supported. overview.md's dependency table pins
the edge.

### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested

**Ownership: the store core owns all GC state — in-process pin and
liveness maps *and* the fleet state's rows and locks. The public
`Backend` trait stays exactly what ADR-003 says (opaque
`hash → bytes`, namespace-blind; "plain content-addressed rows,
nothing else" remains true of the public trait). Backends never learn
liveness (ADR-005) — verbatim.**

The store core reaches the fleet machinery through a **crate-internal
(not exported) engine-state companion trait**, implemented by the
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
`mem` excluded by construction (it cannot be expressed on them and
they are never fleet engines). ADR-008's mechanism fixes *what*; the
seam is *where* the mechanism's SQL lives. Clauses:

  - `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
    `gc_window (key, visible_after)` at store construction (store-core
    call, engine-neutral API; dialect-specific SQL bodies). `owner`
    is the **store-instance id** (ADR-011's vocabulary; ADR-008's
    "node id" wording predates that split). The pin row's identity is
    `(key, owner)` — a dedup re-put by a different instance is a
    different pin row; protection composes by row (an entry is
    protected while any un-expired pin row exists), which is
    ADR-005's refcount semantics expressed in rows. The embedder's
    liveness table is *not* created by this seam — ADR-008: schema
    pinned by the embedder; the sweep reads it and never owns it.
    **Per-engine applicability (the test-gate shape):** sqlite —
    `ensure_gc_schema` (window table only, since the window staging
    is shared machinery per ADR-008 §2) + `window_stage`/
    `window_take` + the sweep-lock clause's single-node body; the
    pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
    bodies but are exercised by unit tests only — they are dead
    machinery on sqlite because no sqlite fleet is on offer (§3).
    postgres and pg-lo — all clause families, contract-tested,
    fleet-active.
- `pin_commit(joint_tx, entries)` — the store core **holds the
  transaction**; the engine contributes its put mechanics (row
  shape, CAS clause, LO writes) and the pin-row inserts, and both
  commit together. This is the joint entry+pin transaction's owner,
  ruled: **the store holds the tx; the engine contributes mechanics
  to it.** The public `put` remains the plain, pin-free path — the
  store's normal put path routes through `pin_commit` wherever the
  engine is seam-capable and fleet state is active.
  - pg-lo correction: ADR-009 §2's "put = one transaction" is
    unchanged in content but its ownership statement is hereby
    corrected — the tx is held by the store core; the pg-lo engine
    contributes its staged-put mechanics and companion-row insert
    into it. The engine sees pins only as rows; never as semantics.
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
  update. The store's pin guard owns the TTL/3 renewal schedule
  (ADR-008); the engine owns only the SQL.
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
  executes ADR-008's delete-time arbitration (un-expired pin rows +
  embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
  keyed, at delete time); the store core receives the answer and
  applies the decision. If fleet clauses ever grow, they grow here —
  the seam is the named home, with contract tests per clause.
- `window_stage(candidates)` / `window_take(now)` — the staged
  delete-window candidates. Per ADR-008 §2 the window table exists
  on **every** engine (sqlite included): staged candidates are
  engine rows; what differs by topology is where *liveness* lives
  (in-process maps for single-node engines, tables for fleets), not
  whether staging exists.
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
  pg advisory lock (ADR-008) for the SQL engines it applies to; the
  store treats lock contention exactly as ADR-008 says — the second
  sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
  nothing deleted. (SQLite today has no fleet story — see §3 — so its
  clause body exists but is exercised by single-node tests only.)

Each clause is contract-tested per its applicability (the
per-engine split above): the conformance suite of §1 grows the fleet
clauses for postgres and pg-lo; `local` and
`mem` are excluded by construction.

### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)

**A fleet constructor requires a pool-shared kv engine.** Postgres is
the shipped engine that satisfies this. A sqlite-fleet mechanism does
not exist and is not on offer (per ADR-007's redb precedent this is
recorded as a non-option, not an open question); two nodes' sqlite
engines over "one pool" are two pools — the partitioning failure, the
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
enforceable
seam is the same as ADR-008 §3's: the constructor can require an
explicit declaration; it cannot prove cross-node truth. The
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
and is closed under both tiers.

### 4. The ops pin-token contract's TTL race, resolved by the ownership rule

**The token *is* the pin — there is no second pin state.** Under fleet
engines the put's pin is the expiry-carrying row (ADR-008); the token
identifies the putter as the row's owner. The token itself is an
**opaque id minted by the ops layer** over the pin row it just
committed — the ops layer (not the store facade) owns the
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
a lookup by the renewal op; no schema is specced for it beyond that
(it is a handle, not durable state — the durable state is the pin row
the seam wrote). Concretely:

- **When the server-side `Pin` guard drops:** at response time. The
  put lands pinned (the joint tx guarantees the entry and its pin row
  commit together); the handler's in-process guard — the thing ADR-005
  pins *in-process* with — is released when the `blobs/put` response
  completes and the token is handed out. From that moment the pin
  row's expiry is the only protection, which is exactly why the
  renewal duty below exists. (If the handler kept its guard for the
  token's whole life, the TTL machinery would be dead code on the ops
  path — the row's expiry, not the guard's presence, is the fleet
  contract's liveness signal.)
- The putter renews the pin by renewing the row — through ops it
  already speaks: a `blobs/have` probe whose handler, observing a
  valid token presented alongside the digest, calls the
  crate-internal `pin_renew(key, owner)` via the ops module's own
  store-core access (the ops module is in-crate — it crosses the
  crate-internal seam by construction, the same as the put handler
  holds the `Pin` guard; what never happens is a *consumer* reaching
  the seam. ADR-008's `has` remains a pure probe: the renewal is the
  ops layer's token-authenticated act on the pin row, not a `has`
  semantic), or a re-put of the same digest (CAS dedup; the re-put's
  joint tx upserts the key+owner pin row, renewing it). The renewal
  cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
  20 s at the default 60 s TTL.**
- A putter pausing past TTL falls to ADR-008's expiry semantics
  verbatim: the row is abandoned and reaped; the backstop is
  delete-then-recover (ADR-005) — the digest is re-offered, the put
  is idempotent under CAS. The race resolves to a re-put, never to
  silent loss.
- For the `local` engine the token is the in-process guard as ADR-005
  had it: nothing persists, nothing expires; the process crash is
  ADR-005's in-flight-put loss case with the same backstop. (The
  cross-wire token under `local` is the ops layer's in-memory handle
  to that guard — same mapping discipline, no expiry.)

ops-surface.md's hand-over contract gains the renewal line and the
token-mapping note; nothing else changes.

## Consequences

**Positive**

- The trait's I/O shapes are decided before code; engine tasks can be
  written without guessing signatures. The per-engine
  conformance-suite shape (POC #7's 10/10 gate as the norm) gives
  decomposition a uniform test gate across all five engines.
- "Backends never learn liveness" survives the fleet mechanism
  intact; the store core owns GC state per ADR-005 exactly as
  written; ADR-008's mechanism lives in a named, tested seam instead
  of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
  else" stays true of the public trait.
- The ops pin-token TTL race is resolved with no new ops; the
  backstop is already-validated semantics (delete-then-recover).
- The kv tier's fleet-validity rule closes the constructor table
  under both tiers.

**Negative**

- A crate-internal companion trait is real machinery: six clause
  families on postgres/pg-lo (fleet-active, all contract-tested);
  sqlite carries only the window + lock clauses as fleet-relevant
  machinery (its pin clauses have unit-test bodies but no fleet mode
  on offer — §3). The seam bounds where fleet-GC growth lands; it
  does not shrink it.
- Remote putters gain a real protocol duty (renew past 20 s;
  pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
- The sqlite sweep-lock clause body exists but is exercised only by
  single-node tests until a sqlite-fleet posture is ever
  decided (it is not on offer — §3).

**Neutral**

- The `Backend` trait's method set is unchanged from ADR-003/008 —
  no public API change beyond what store-api.md already specced; the
  cursor/staging abstractions are crate-internal.
- The staged-put contract already existed behaviorally (POC #3 A5);
  this ADR names it.
- ADR-009's engine content is unchanged; only its tx-ownership
  statement is corrected here.

## References

- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
  (tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
  liveness-ownership rule this ADR preserves), ADR-006 (the range
  contract the cursor serves), ADR-008 (the fleet mechanism whose
  state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
  correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
  (vocabulary + constructor table)
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
  `docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
  `docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
  512 KiB knee, windowed gets)
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
  extends); [store-api.md](../store-api.md) (stage hygiene, sweep
  semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
  (REQ-2, the fleet requirement §2/§3 serve)