alkblobs/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md

ADR-008: Trait length probe, fleet GC, and fs-tier engines

Status

Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's passed gate; every other part of this ADR stands as written)

Context

Three defects surfaced in the post-ADR-007 architecture review, all concentrated at the seams ADR-007 touched and all load-bearing for REQ-2 (a replicator node may run multiple store instances — a fleet — sharing one pool via postgres; requirements.md):

1. The Backend trait has no length probe. The store surface promises stat(key) → Option<EntryMeta> ("cheap length/type probe"; store-api.md) and pool accounting rides list + stat (ADR-005), but the trait's methods are has/get/put/delete/list/ name with no value-metadata access. The kv engines' row shapes both carry a size column (POC #5 B5), and the fs tier can stat a file — but with no trait method, kv stat degenerates to a full value load. The trait is the crate's second stable seam (ADR-003); discovering this during implementation would be a breaking trait change after backends exist.
2. ADR-005's sweep-safety protocol is per-node by construction. Pins, the pin/liveness arbitration lock, and delete-window arbitration all live inside one process (ADR-005 §Decision). That is sound for sqlite and single-instance postgres (REQ-1, REQ-3 topologies) — but under a fleet (REQ-2), node A's sweep enumerates the shared pool and arbitrates against A's in-process pin map; node B's in-flight put is invisible. Data loss, structurally: the topology REQ-2 requires does not satisfy the protocol ADR-005 specifies.
3. The fs tier is node-local, silently partitioning a fleet. The fs backend stores to a sharded layout on the node's own filesystem (ADR-003). Under a fleet sharing one kv pool, a >threshold blob put via node A exists only on A's disk; node B's get fall-through queries its own fs and misses, and list() unions disagree per node. REQ-2 makes this topology a shipping requirement, not a hypothetical.

These three travel together: the fleet topology forces #2 and #3, and #3's resolution constrains what the fs tier must support, while #1 is a trait-level gap any topology hits.

Decision

1. size(key) → Option<u64> joins the Backend trait

The trait gains a length probe: size(key) returns the entry's byte length or None if absent. Required of all engines; semantics per tier:

• kv engines: an indexed column probe (SELECT size …WHERE key), never a value load. The size column both POC row shapes carry becomes contract-relevant.
• fs engine (local): stat/metadata read, no content fetch.
• The store facade's stat (store-api.md) and the accounting path (ADR-005) ride this method; the has method remains a pure existence probe.

size is a trait addition before any backend ships data — the one-way-door discipline of ADR-002/003 applied to the trait itself.

2. Fleet GC: the delete window is durable and re-arbitrated in the database; pins are DB-backed under fleet engines

The store facade's GC invariant is unchanged — a visible pool entry is never deleted while liveness protects it, and an in-flight put is never deleted by a sweep started before it committed (ADR-005). Under a fleet, the mechanism moves into the shared engine:

• Pins are rows. put's pin-before-publish (ADR-005) becomes two inserts in one transaction on the fleet engine: the entry row and its pin row commit together (or neither does). The pin row carries an owner (node id) and expiry: the pin guard renews its row every TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor parameters, not API). Protection = an un-expired pin row: at delete-time arbitration, only expiry > now() pins count; an expired pin is abandoned (its holder failed to renew — the same liveness-failure signal as an in-process guard's drop) and is reaped by the sweeper's maintenance step (the GC-of-GC cleanup). A node that pauses past TTL loses its pins; delete-then-recover (ADR-005) remains the correctness backstop — content is re-put byte-identically under the same key.
• Registered liveness sources have one fleet form: a liveness table. A fleet embedder registers its liveness as a named table of (key, …) rows in the shared engine (schema pinned by the embedder, shape contractually (key bytea, …) — any columns it needs beyond the key). In-process callback views are not a fleet liveness form: delete-time arbitration queries/finger-locks tables per key, so a source must be a table for the SQL protocol to see it. Single-node embedders keep callback sources unchanged (ADR-005).
• The protect callback is single-node-only. A fleet sweeper cannot consult another node's in-process callback, so under fleet engines the callback's fleet expression is: the embedder materializes its protect set into its liveness table (or pin rows) before invoking sweep — the callback's abort semantics remain for single-node/topology-local sweeps that cannot express their protection as rows.
• One sweeper per pool. A fleet node acquires a pg advisory lock before sweeping and holds it for the whole sweep. Two nodes sweeping concurrently is not a state this crate tolerates — the lock is the admission gate, and a second-node sweep returns the typed GcAborted (no deletion) rather than racing. GcAborted thus carries two cause classes (protection-source failure — ADR-005; sweeper lock contention — this ADR); the typed error's fleet variant identifies which.
• Delete-time arbitration is SQL-level. The delete window's re-check (ADR-005: "arbitrate under the pin/liveness lock at delete time") executes as SELECT … FOR UPDATE against pin rows and liveness tables at delete time, then batch-deletes. The staged re-arbitrated delete: during mark, candidate-dead keys are staged as rows carrying visible_after = mark_end + window (the delete window's delay — default 5 min, constructor-tunable per deployment); the executor (the sweeper, or the next sweep's owner) re-arbitrates each staged candidate at execution time against pins and liveness tables before deleting. This is the durable delete window on every engine — the window table is engine rows on sqlite as well; what differs by topology is only where the liveness state lives (in-process maps and callbacks for single nodes; tables and SQL arbitration for fleets), not whether staging exists. No external queue dependency is required (the pattern is engine-native transactions; consumer-side queues like honker or pg-boss-lineage tools remain consumer-layer options, per ADR-004's boundary).
• Single-node postures are unchanged. sqlite and single-instance postgres keep ADR-005's in-process protocol verbatim: the pin map, RAII guards, and arbitration lock stay in-process; only the persistence of the delete window (the staged-candidate table) is shared machinery. The pin-row expiry/ttl machinery exists only where fleet engines are active.

3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible

The fs tier is a tier, not a medium. It ships engines the way the kv tier does (ADR-007), and a deployment's topology determines which are valid:

• local (default): the sharded-layout filesystem backend, unchanged (ADR-003). Valid for any single-node topology (REQ-1, REQ-4) and for fleet nodes using shared media (below).
• Fleet + shared media (deployment requirement): multi-node pools may keep the local engine by pointing every node's fs tier at shared/replicated media. The soundness claim is qualified, not assumed: immutable-once-committed files plus stage-then-commit- rename are sound on media that provides (a) atomic rename within a directory, (b) close-to-open consistency (a written+closed file is complete to any later reader — NFSv3+ with the close-to-open discipline; NFSv4 and EFS-class stores qualify), and (c) no cross-node POSIX locking reliance (this store takes none — its cross-node coordination is the fleet engine's locks, never file locks; honker's NFS warning is about sqlite's lock protocol, not this shape). Verifying (a)-(c) on the deployment's actual media is the deployment's responsibility; a verification pass is recorded ops work, not crate code.
• Fleet re-routing (topology alternative): a fleet whose nodes do not share media designates one storage node; large-blob writes and reads over the pool route through it via the ops surface (verified fetch/put; ADR-001). Client nodes' constructors are configured with a kv-tier-only dispatch (or an fs tier reserved for node-private, consumer-owned content — which is outside the pool and outside this crate's GC; it is the consumer's own storage beside the pool per ADR-004, not pool content). The facade shape on a client node is its own store for sub-threshold content plus its registered ops-put path for large content — a consumer-side composition, not a store-level mode.
• pg-lo (candidate engine, not shipped): postgres Large Objects as the fs tier's storage — the same "engine behind one contract" move as ADR-007, one tier over. Named here so the door is explicit, with its admission gated on measured evidence (POC #7: LO write/read curves at the packfile regime, tx-scoped handle cost under pooling, pg_largeobject/vacuum posture under churn, crash-orphan behavior) and its own engine ADR. Not shipped speculatively; REQ-2's immediate fleet answers are shared media or re-routing; pg-lo is the consolidation option if those are unacceptable. Update 2026-10-03: POC #7 ran and passed its gate — docs/research/poc-pglo-findings.md (companion-table authority, lo_get window gets, durable put ≈ fs's durable put, cached gets behind page-cache fs by a measured 20-50x aggregate-multiplied to ~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture). The engine ADR's remaining evidence path is open; admission remains gated on it, not on speculation.
• The no-mixed-fs-tiers rule is a deployment invariant, enforced at the enforceable seam. Cross-node configuration cannot be validated by any one constructor (it sees only its own node). What the constructor can enforce: a fleet-mode constructor (fleet engines active) requires an explicit fs-tier declaration — local (with the operator's assertion that its root is fleet-shared media), routed (no local fs over pool content), or — when shipped — pg-lo. The cross-node truth ("is node B's local actually the same media as node A's?") is a deployment invariant this crate documents and ops verifies, not something a constructor can prove; a partitioned tier's observable signature (get fall-through misses for entries another node wrote) is the documented detection symptom.

In sum: a fleet's fs tier over pool content is either shared media (all nodes' local roots on the same fleet-shared media) or routed (one storage node); mixing per-node-local media over one pool is the partitioning failure this decision exists to prevent.

Consequences

Positive

• The trait now expressibly supports stat and pool accounting on every engine; the "cheap probe" claim of store-api.md is true rather than aspirational.
• REQ-2 (fleet over one pool) has a complete, specced GC story: pins move to the shared engine atomically with put-commits, one sweeper holds the lock, deletes re-arbitrate in-transaction. The ADR-005 invariant extends to fleets as a stated mechanism, not an accident of single-process luck.
• The fs tier's fleet problem has named, shipped-code-free answers (shared media, re-routing) and a named consolidation candidate (pg-lo) whose admission path is the same POC-first door as ADR-007.

Negative

• The Backend trait gains a method before any backend ships — the correct moment, but every engine's task list grows by one method.
• Fleet GC is real state machinery: pin rows with expiry (GC-of-GC surface for orphaned pins at node crash — the expiry is the cleanup handle), advisory-lock coordination, and SQL-level arbitration added on top of the already-intricate delete-window state machine.
• pg-lo remains unbuilt: a fleet that cannot share media and cannot re-route pays for its ADR + POC before shipping large blobs — the measured, deliberate deferral of the ADR-006 pattern, not a hedge.

Neutral

• The sqlite and single-node-postgres postures are unchanged; this ADR costs existing paths nothing at runtime.
• pg-lo, if admitted, rides the same driver stack ADR-007 already ships (tokio-postgres + deadpool) via SQL lo_* functions — no new driver dependency (the postgres_large_object crate is a dead 0.15-era io-trait glue; unnecessary; POC #7 confirms: ~400 lines of SQL-statement shapes over the existing stack).

References

• requirements.md — REQ-2 (the fleet fact this ADR binds), REQ-1/3/4 (the topologies left unchanged)
• POC #5 (poc-postgres-kv-findings.md B5 — row shapes with size columns; the driver stack pg-lo would ride)
• ADR-002 (the one-way-door discipline this trait change applies), ADR-003 (the trait contract and admission gate size and pg-lo amend/enter), ADR-004 (tier≠engine, extended to the fs tier), ADR-005 (the GC invariant and single-node protocol extended here, not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission follows), ADR-007 (the engine-behind-one-trait precedent)
• backends-and-dispatch.md (the trait and tier surface this amends); store-api.md (stat's contract); gc-and-namespaces.md (the fleet GC mechanism); ops-surface.md (the re-routing topology's transport)