alkblobs/tasks/store-fleet-sweep.md

---
id: store-fleet-sweep
name: the fleet GC protocol — SQL sweep lease, SQL-level arbitration, pin reaping, the fleet liveness verb
status: completed
depends_on: [store-fleet-construction]
scope: broad
risk: high
impact: phase
level: implementation
tags: [wave-6, store, gc, fleet, sweep]
---

## Description

Grow the sweep and the delete paths into their **fleet forms** under
`fleet: true` (ADR-008 §2; ADR-010 §2; ADR-012 §1/§5/§6.6/§6.7;
ADR-013 §2) — the architecture's most intricate state machine extended
across instances, its invariants unchanged (a visible pool entry is
never deleted while liveness protects it; an in-flight put is never
deleted by a sweep started before it committed). The non-fleet forms
(inline + durable, single-node) are byte-unchanged.

- **The sweep lease** (ADR-013 §2's named exclusive semantics; the
  shipped provider is the wave-4 advisory-lock clause): under fleet
  mode the sweep acquires `sweep_lock_acquire()` on the fleet host
  before sweeping and holds it for the whole sweep; contention is the
  `GcAbortCause::SweeperLock` report abort (`Ok(SweepReport)`, nothing
  deleted — ADR-012 §1); holdership ends at release, owner death, or
  session loss (the clause's session-scoped semantics, wave 4). The
  in-process single-flight flag composes (both held — the instance-
  local guard stays cheap and uniform). **Named design point — the
  lease's hold across the sweep body**: the lease's held checkout
  must not deadlock the sweep's own engine work (the wave-4 lease
  holds one pooled connection for its duration; the sweep's other
  clauses ride further checkouts — verify the pool sizing composes
  and record the posture).
- **SQL-level delete arbitration** (ADR-008 §2): under fleet mode the
  delete-time decision is the seam's `arbitrate(key)` — the
  `SELECT … FOR UPDATE` finger-lock folding un-expired pin rows +
  registered liveness tables into one answer — on **both** the direct
  `delete()` path (protected → `Error::GcRefuse`, the facade owns the
  refusal) and the durable window's execution-time re-arbitration.
  **Named design point — the two arbitration domains' composition**:
  the in-process `GcHome` machinery (markers, in-process pins,
  callbacks, in-process sources) and the SQL `arbitrate` both run —
  the in-process domain guards this instance's in-flight state (the
  marker mechanism closes the local sweep-vs-put race without engine
  I/O under the lock), the SQL domain is the cross-instance
  authority; decide the composition order (in-process first — cheap,
  local; SQL second — fleet-authoritative) and record it. The
  engine-side `pin_check` classification stays available to the store
  core where a pins-only answer suffices (its consumer beyond
  `arbitrate` may be wave 7's ops renewal probe — do not force it
  here).
- **The expired-pin reaping maintenance step** (ADR-008 §2's GC-of-GC
  cleanup: "an expired pin is abandoned … and is reaped by the
  sweeper's maintenance step"): the fleet sweep gains the maintenance
  step deleting expired pin rows. **Named design point — the clause
  home**: `EngineFleetState` has no reap clause; the seam is the
  named home for fleet-clause growth (ADR-010 §2: "If fleet clauses
  ever grow, they grow here") — add the clause body on the postgres
  host (e.g. `pin_reap(now) → count`, one statement, contract-tested
  per the applicability row) rather than any raw-SQL side-channel
  (rejected by ADR-010 §2). The step runs under the lease, before or
  alongside the overdue execution (record the ordering).
- **The fleet liveness registration verb** (ADR-012 §6.6): "the same
  verb with a fleet-shaped argument" — the facade's
  `register_liveness_source` gains its fleet form, registering a
  **named table reference** in the shared engine (the embedder's own
  table per ADR-008 §2; the store core reads it, never owns it — the
  schema-ownership rule). The engine-side registration landed wave 4
  (`PostgresBackend::register_liveness_source` — quoted-identifier
  boundary, idempotent re-registration, malformed refs error). **Named
  design points — the verb's shape**: how the fleet form rides the
  public verb (a fleet-shaped argument vs a sibling method — decide
  consistent with §6.6's "same verb" wording and the existing
  signature's infallibility: the engine-side registration is
  fallible, so the fleet form's error surface needs deciding), and
  whether in-process `LivenessSource` registrations remain legal
  under fleet mode (ADR-008: in-process callback views are not a
  *fleet* liveness form — an instance-local registration's protection
  is instance-local only; decide allow-and-document vs reject and
  record it). Deregistration composes (the correction flow, ADR-012
  §6.4).
- **The protect callback's fleet posture** (ADR-008 §2): single-node-
  only — under fleet mode the callback remains consultable by *this*
  instance's sweeps (its protection is instance-local; the embedder's
  fleet-wide protection materializes into its liveness table or pin
  rows before invoking sweep). Record the posture in the verb docs;
  no new machinery.
- **The fleet sweep body** (per sweep, under the lease): the protect
  consult (instance-local, abort `ProtectFailed` precedes everything)
  → the maintenance step (expired-pin reaping) → **execute overdue**
  (`window_take` on the postgres host — the one-window-host rule
  holds: every fleet-valid configuration's window hosts on the kv
  engine; each overdue candidate re-arbitrated at execution time via
  the SQL arbitration, deleted where it lives — the fall-through
  delete across both tiers) → **mark** (the pool enumerated; the live
  set from the in-process sources — the SQL liveness tables are
  consulted at arbitration time, not enumerated into the mark set) →
  **stage** at `visible_after = mark_end + window` → release the
  lease. Report counts exact relative to the schedule (the durable
  form's reading carries: staged-this-sweep vs executed-this-sweep are
  different sets).
- **The sweep-vs-put race under fleet, closed by the row**: a fleet
  put's pin row commits atomically with its entry (the joint tx,
  `store-fleet-put`), so a sweep's window execution can never observe
  a visible-unpinned entry; a put landing during the window delay has
  its row committed before its entry is visible and the execution-time
  SQL arbitration cancels the candidate. Unit-test the race across
  the delay in its fleet form (the wave-3 durable-form race test's
  fleet edition).
- **Unit tests** (env-driven, skipping cleanly server-less; the
  harness server `pglo-poc`, :15432, db `blobs`, `postgres`/`poc` —
  verified live 2026-10-12): the lease's fleet form (contention
  between two instances of one store shape → the second aborts
  `SweeperLock` with nothing deleted; release/loss reaping); SQL
  arbitration on the direct-delete path (a pin row from a *foreign*
  owner id refuses the delete — the cross-instance protection proof;
  `GcRefuse`); the maintenance step reaps exactly the expired set
  (un-expired rows survive); the fleet liveness verb registers a
  test-owned embedder table and its rows protect (the schema-
  ownership rule — the engine never creates the table); the
  two-instance exact-count sweep shape (instance A pins, instance B
  sweeps — the pinned entry survives; unpinned twins delete); the
  race across the delay; the non-fleet forms unchanged (all existing
  suites green).

## Acceptance Criteria

- [x] Under `fleet: true`, the sweep acquires the SQL lease (the
      ADR-013 §2 semantics: atomic test-and-set, one sweeper per
      pool across instances, contention = `SweeperLock` report abort
      with nothing deleted); the in-process flag composes; under
      `fleet: false` the sweep forms are byte-unchanged
- [x] Delete-time arbitration is SQL-level under fleet mode on both
      the direct path (`GcRefuse`) and the window execution
      (execution-time re-arbitration); a foreign owner's un-expired
      pin row protects (cross-instance proof); the two-domain
      composition decided and recorded
- [x] The expired-pin reaping maintenance step landed as a new
      `EngineFleetState` clause on the postgres host (the seam is the
      named home; no raw-SQL side-channel), contract-tested, run
      under the lease
- [x] The fleet liveness registration verb landed (the named table
      reference form; the verb-shape and in-process-source design
      points decided and recorded; the schema-ownership rule holds)
- [x] The protect callback's fleet posture recorded (instance-local;
      materialization is the embedder's fleet-wide duty)
- [x] The fleet race property unit-tested across the window delay (a
      put landing during the delay survives via the row + the
      execution-time SQL re-arbitration)
- [x] `cargo test` (default) green; `cargo test --features postgres`
      green server-less (skipping) and against the harness server at
      acceptance; `cargo clippy --all-targets -- -D warnings` and
      `cargo fmt --check` green (default + `--features postgres`);
      `cargo doc --no-deps` clean

## References

- src/store.rs (the sweep's two forms; `arbitrated_delete`; the
  fleet flag from `store-fleet-construction`)
- src/gc.rs (`GcHome` — the in-process domain; `EngineFleetState` —
  the clauses; the seam-as-named-home note)
- src/postgres.rs (the lease, `arbitrate`, `pin_check` — the wave-4
  clause bodies this drives; the liveness registration)
- docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md
  §2 (the fleet mechanism: SQL arbitration, one sweeper per pool, the
  maintenance step, the protect callback's single-node-only posture)
- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md
  §2 (the named exclusive sweep lease — the semantics this implements)
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
  §1 (abort-as-data), §5 (the one-window-host rule), §6.4 (the full
  protection set), §6.6 (the fleet liveness verb's API home), §6.7
  (the executor is the next sweep)
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
  §2 (the seam clauses; the raw-SQL rejection)
- tasks/store-sweep-durable-window.md (the durable form's shape — the
  single-node structural twin)
- tasks/store-fleet-put.md (the joint puts whose rows this sweep
  arbitrates against)

## Notes

Landed in `src/gc.rs`, `src/facade.rs`, `src/postgres.rs`, `src/store.rs`, `tests/conformance/rows.rs` (call-site mechanical only). Decisions and design points, as recorded:

- **The lease's hold across the sweep body (pool sizing).** The SQL lease holds **one pooled checkout** for the sweep's duration (the session-scoped advisory lock's same-session acquire/release discipline); the fleet sweep body's other engine clauses — `pin_reap`, `window_take`, the mark's `list` enumeration, `window_stage` — ride **further checkouts** under the held lease. A sequential sweep body holds exactly two concurrent checkouts (the lease's plus the current clause's), so the pool must size ≥ 2 or the sweep acquire-waits against itself (a deadlock). Enforced: `fleet_pool_size_guard` rejects `max_size < 2` at every `_fleet` constructor's entry (errors closed, server-less-testable); the shipped default (8) provides headroom. Recorded in `sweep()`/`sweep_under_leases` docs.
- **The two arbitration domains' composition: in-process first, SQL second.** Implemented once, inside `arbitrated_delete`: a `Proceed` from the in-process domain (pins, in-flight markers, protect callbacks, in-process sources — the marker mechanism closes the local race without engine I/O under the lock) still consults the SQL `arbitrate` (un-expired pin rows of *any* instance + registered liveness tables, under the `SELECT … FOR UPDATE` finger-lock); both must pass. On a SQL refusal the marker releases (`conclude_delete`) and the caller sees `Cancelled` — a blocked pin mint proceeds against the live entry. Because the composition lives at this one seam, the direct path, the durable window's execution-time re-arbitration, and any future consumer share it uniformly; the SQL answer's failure is operational (`Err`). `pin_check` stays available for pins-only answers, unconsumed (its remaining consumer is wave 7's ops renewal probe).
- **The maintenance step's clause home.** `pin_reap(now) → count` is a new `EngineFleetState` clause (ADR-010 §2's "if fleet clauses ever grow, they grow here"); the postgres host body is one statement — `DELETE … WHERE expiry <= $1` — with the boundary matching `pin_renew`'s live predicate (`expiry > now`), never touching un-expired rows; contract-tested (`pin_reap_reaps_exactly_the_expired_set`, including per-row composition: a key with one expired and one live row keeps the live row). Ordering recorded in `sweep_window_fleet`: protect consult → reap → execute overdue → mark → stage → release. Reap *before* the overdue execution is a recorded hygiene choice, not a correctness one — `arbitrate` already requires `expiry > now`, so the protection answer is identical either way.
- **The fleet liveness verb's shape.** Same verb, fleet-shaped argument: `Store::register_liveness_source` takes a new public `LivenessRegistration` enum — `InProcess(Arc<dyn LivenessSource>)` | `FleetTable(String)` (ADR-012 §6.6's "same verb" wording; a sibling method was rejected). The verb now returns `Result<Registration, Error>`: the fleet form validates the reference (malformed references error at registration, never interpolate unquoted — `postgres.rs`'s `quote_table_reference`) and forwards to the shared engine, so its fallibility is real; the in-process form never errors in practice — the pre-fleet signature's infallibility was that form's accident, retired by the unified surface. The store-side registration bookkeeping rides `RegistryCore` beside the in-process source records (`RegistrationKind::{Source, Protect, FleetSource}` behind `Registration`), re-registration of the same reference is idempotent (the original token returns), and deregistration composes the correction flow: the store-side record removal forwards to a new engine-side `PostgresBackend::unregister_liveness_source` (quote-then-remove).
- **In-process sources under `fleet: true`: allow-and-document.** An instance-local registration's protection is instance-local only — it covers this instance's sweeps (mark-time enumeration) and direct deletes (the in-process domain's probes); it is not a *fleet* liveness form (ADR-008 §2), so the embedder's fleet-wide protection materializes into its liveness table (or pin rows) before the fleet's other instances sweep. Recorded in the `LivenessRegistration` type docs. The sweep's safe default counts both forms together (`registered_liveness_count = sources + fleet tables`) — a fleet embedder may register nothing but its table.
- **The protect callback's fleet posture: single-node-only, no new machinery.** A registered callback remains consultable by *this* instance's sweeps (its protection is instance-local, abort `ProtectFailed` precedes everything in the body); the embedder materializes its fleet-wide set into its liveness table or pin rows before invoking sweep. Recorded in the `register_liveness_source` verb docs.
- **The SQL liveness tables are never enumerated into the mark set** — consulted at arbitration time only (the fleet body's mark enumerates the pool + in-process sources). Test-asserted: entries protected only by SQL state stage as candidates and cancel at their execution-time re-arbitration (`two_instance_exact_counts…`, `the_fleet_liveness_verb…`).
- **Superseded test rewritten:** `postgres_fleet_construction_tests::fleet_declaration_leaves_the_sweep_machinery_to_its_task` (its premise — inert machinery arbitrating in-process — was the pre-task state) became `fleet_declaration_runs_the_sweep_machinery_on_the_kv_host`, asserting the now-live fleet semantics: the instance's own un-expired row cancels the candidate at execution; expiry + reap → execution.
- **Verification:** `cargo test` (default) green; `cargo test --features postgres` green server-less (skipping) and against `pglo-poc` :15432; `cargo test --all-features` green; `clippy --all-targets -D warnings` green (default, `postgres`, `all-features`); `fmt --check` green; `cargo doc --no-deps` clean (postgres-feature doc warnings unchanged at the pre-existing 21).

## Summary

The sweep and delete paths grew their fleet forms under `fleet: true` — the architecture's most intricate state machine extended across instances, invariants unchanged (a visible pool entry is never deleted while liveness protects it; an in-flight put is never deleted by a sweep started before it committed), the non-fleet forms byte-unchanged:

- **The sweep lease**: under fleet mode `sweep()` acquires the in-process single-flight flag *and* the SQL lease (`sweep_lock_acquire` on the fleet host — ADR-013 §2's named exclusive semantics), holding both for the whole sweep and releasing the lease on every exit path; contention at either domain aborts as `SweeperLock` report data with nothing deleted. The pool-sizing composition is verified and enforced (`max_size ≥ 2` at the fleet constructors' entry).
- **SQL-level delete arbitration**: `arbitrated_delete` composes the two domains — in-process first (cheap, local), the SQL `arbitrate` second (fleet-authoritative) — on the direct path (`GcRefuse`, facade-owned) and the durable window's execution-time re-arbitration alike; a foreign instance's un-expired pin row or a liveness-table row refuses cross-instance.
- **The maintenance step**: `EngineFleetState::pin_reap` (the seam as the named clause home) reaps exactly the expired pin rows, one statement on the postgres host, contract-tested, driven under the lease before the overdue execution.
- **The fleet liveness verb**: `register_liveness_source` takes the fleet-shaped `LivenessRegistration` argument (ADR-012 §6.6's same-verb wording): the named-table form registers the embedder's own table in the shared engine (schema-ownership rule holds — the engine never creates it; idempotent; deregistration composes); the in-process form remains legal (instance-local protection, documented). `Registration` now carries its kind; the verb is fallible (the fleet form's error surface).
- **The protect callback's fleet posture** recorded: single-node-only, instance-local consult remains, materialization is the embedder's duty — no new machinery.
- **The fleet sweep body**: protect consult → `pin_reap` → execute overdue (`window_take` on the kv host; execution-time two-domain re-arbitration; fall-through delete across both tiers) → mark (pool + in-process sources only; SQL liveness consulted at arbitration time, never enumerated) → stage at `visible_after = mark_end + window` → release. Counts exact relative to the schedule.
- **The fleet race across the delay** unit-tested: a foreign instance's fleet put landing during the window delay survives via its atomically-committed row plus the execution-time SQL re-arbitration.

Tests: lease contention/reaping between two instances of one shape, the foreign-row `GcRefuse`, the exact-set reap, the embedder-table protection + correction flow + malformed-reference and `fleet: false` rejections, the two-instance exact-count shape, and the delay race — env-driven, skipping cleanly server-less, green against the harness server.