alkblobs/tasks/store-fleet-sweep.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
store-fleet-sweep	the fleet GC protocol — SQL sweep lease, SQL-level arbitration, pin reaping, the fleet liveness verb	completed	store-fleet-construction	broad	high	phase	implementation	wave-6 store gc fleet sweep

Description

Grow the sweep and the delete paths into their fleet forms under fleet: true (ADR-008 §2; ADR-010 §2; ADR-012 §1/§5/§6.6/§6.7; ADR-013 §2) — the architecture's most intricate state machine extended across instances, its invariants unchanged (a visible pool entry is never deleted while liveness protects it; an in-flight put is never deleted by a sweep started before it committed). The non-fleet forms (inline + durable, single-node) are byte-unchanged.

• The sweep lease (ADR-013 §2's named exclusive semantics; the shipped provider is the wave-4 advisory-lock clause): under fleet mode the sweep acquires sweep_lock_acquire() on the fleet host before sweeping and holds it for the whole sweep; contention is the GcAbortCause::SweeperLock report abort (Ok(SweepReport), nothing deleted — ADR-012 §1); holdership ends at release, owner death, or session loss (the clause's session-scoped semantics, wave 4). The in-process single-flight flag composes (both held — the instance- local guard stays cheap and uniform). Named design point — the lease's hold across the sweep body: the lease's held checkout must not deadlock the sweep's own engine work (the wave-4 lease holds one pooled connection for its duration; the sweep's other clauses ride further checkouts — verify the pool sizing composes and record the posture).
• SQL-level delete arbitration (ADR-008 §2): under fleet mode the delete-time decision is the seam's arbitrate(key) — the SELECT … FOR UPDATE finger-lock folding un-expired pin rows + registered liveness tables into one answer — on both the direct delete() path (protected → Error::GcRefuse, the facade owns the refusal) and the durable window's execution-time re-arbitration. Named design point — the two arbitration domains' composition: the in-process GcHome machinery (markers, in-process pins, callbacks, in-process sources) and the SQL arbitrate both run — the in-process domain guards this instance's in-flight state (the marker mechanism closes the local sweep-vs-put race without engine I/O under the lock), the SQL domain is the cross-instance authority; decide the composition order (in-process first — cheap, local; SQL second — fleet-authoritative) and record it. The engine-side pin_check classification stays available to the store core where a pins-only answer suffices (its consumer beyond arbitrate may be wave 7's ops renewal probe — do not force it here).
• The expired-pin reaping maintenance step (ADR-008 §2's GC-of-GC cleanup: "an expired pin is abandoned … and is reaped by the sweeper's maintenance step"): the fleet sweep gains the maintenance step deleting expired pin rows. Named design point — the clause home: EngineFleetState has no reap clause; the seam is the named home for fleet-clause growth (ADR-010 §2: "If fleet clauses ever grow, they grow here") — add the clause body on the postgres host (e.g. pin_reap(now) → count, one statement, contract-tested per the applicability row) rather than any raw-SQL side-channel (rejected by ADR-010 §2). The step runs under the lease, before or alongside the overdue execution (record the ordering).
• The fleet liveness registration verb (ADR-012 §6.6): "the same verb with a fleet-shaped argument" — the facade's register_liveness_source gains its fleet form, registering a named table reference in the shared engine (the embedder's own table per ADR-008 §2; the store core reads it, never owns it — the schema-ownership rule). The engine-side registration landed wave 4 (PostgresBackend::register_liveness_source — quoted-identifier boundary, idempotent re-registration, malformed refs error). Named design points — the verb's shape: how the fleet form rides the public verb (a fleet-shaped argument vs a sibling method — decide consistent with §6.6's "same verb" wording and the existing signature's infallibility: the engine-side registration is fallible, so the fleet form's error surface needs deciding), and whether in-process LivenessSource registrations remain legal under fleet mode (ADR-008: in-process callback views are not a fleet liveness form — an instance-local registration's protection is instance-local only; decide allow-and-document vs reject and record it). Deregistration composes (the correction flow, ADR-012 §6.4).
• The protect callback's fleet posture (ADR-008 §2): single-node- only — under fleet mode the callback remains consultable by this instance's sweeps (its protection is instance-local; the embedder's fleet-wide protection materializes into its liveness table or pin rows before invoking sweep). Record the posture in the verb docs; no new machinery.
• The fleet sweep body (per sweep, under the lease): the protect consult (instance-local, abort ProtectFailed precedes everything) → the maintenance step (expired-pin reaping) → execute overdue (window_take on the postgres host — the one-window-host rule holds: every fleet-valid configuration's window hosts on the kv engine; each overdue candidate re-arbitrated at execution time via the SQL arbitration, deleted where it lives — the fall-through delete across both tiers) → mark (the pool enumerated; the live set from the in-process sources — the SQL liveness tables are consulted at arbitration time, not enumerated into the mark set) → stage at visible_after = mark_end + window → release the lease. Report counts exact relative to the schedule (the durable form's reading carries: staged-this-sweep vs executed-this-sweep are different sets).
• The sweep-vs-put race under fleet, closed by the row: a fleet put's pin row commits atomically with its entry (the joint tx, store-fleet-put), so a sweep's window execution can never observe a visible-unpinned entry; a put landing during the window delay has its row committed before its entry is visible and the execution-time SQL arbitration cancels the candidate. Unit-test the race across the delay in its fleet form (the wave-3 durable-form race test's fleet edition).
• Unit tests (env-driven, skipping cleanly server-less; the harness server pglo-poc, :15432, db blobs, postgres/poc — verified live 2026-10-12): the lease's fleet form (contention between two instances of one store shape → the second aborts SweeperLock with nothing deleted; release/loss reaping); SQL arbitration on the direct-delete path (a pin row from a foreign owner id refuses the delete — the cross-instance protection proof; GcRefuse); the maintenance step reaps exactly the expired set (un-expired rows survive); the fleet liveness verb registers a test-owned embedder table and its rows protect (the schema- ownership rule — the engine never creates the table); the two-instance exact-count sweep shape (instance A pins, instance B sweeps — the pinned entry survives; unpinned twins delete); the race across the delay; the non-fleet forms unchanged (all existing suites green).

Acceptance Criteria

• Under fleet: true, the sweep acquires the SQL lease (the ADR-013 §2 semantics: atomic test-and-set, one sweeper per pool across instances, contention = SweeperLock report abort with nothing deleted); the in-process flag composes; under fleet: false the sweep forms are byte-unchanged
• Delete-time arbitration is SQL-level under fleet mode on both the direct path (GcRefuse) and the window execution (execution-time re-arbitration); a foreign owner's un-expired pin row protects (cross-instance proof); the two-domain composition decided and recorded
• The expired-pin reaping maintenance step landed as a new EngineFleetState clause on the postgres host (the seam is the named home; no raw-SQL side-channel), contract-tested, run under the lease
• The fleet liveness registration verb landed (the named table reference form; the verb-shape and in-process-source design points decided and recorded; the schema-ownership rule holds)
• The protect callback's fleet posture recorded (instance-local; materialization is the embedder's fleet-wide duty)
• The fleet race property unit-tested across the window delay (a put landing during the delay survives via the row + the execution-time SQL re-arbitration)
• cargo test (default) green; cargo test --features postgres green server-less (skipping) and against the harness server at acceptance; cargo clippy --all-targets -- -D warnings and cargo fmt --check green (default + --features postgres); cargo doc --no-deps clean

References

• src/store.rs (the sweep's two forms; arbitrated_delete; the fleet flag from store-fleet-construction)
• src/gc.rs (GcHome — the in-process domain; EngineFleetState — the clauses; the seam-as-named-home note)
• src/postgres.rs (the lease, arbitrate, pin_check — the wave-4 clause bodies this drives; the liveness registration)
• docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md §2 (the fleet mechanism: SQL arbitration, one sweeper per pool, the maintenance step, the protect callback's single-node-only posture)
• docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §2 (the named exclusive sweep lease — the semantics this implements)
• docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (the one-window-host rule), §6.4 (the full protection set), §6.6 (the fleet liveness verb's API home), §6.7 (the executor is the next sweep)
• docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §2 (the seam clauses; the raw-SQL rejection)
• tasks/store-sweep-durable-window.md (the durable form's shape — the single-node structural twin)
• tasks/store-fleet-put.md (the joint puts whose rows this sweep arbitrates against)

Notes

Landed in src/gc.rs, src/facade.rs, src/postgres.rs, src/store.rs, tests/conformance/rows.rs (call-site mechanical only). Decisions and design points, as recorded:

• The lease's hold across the sweep body (pool sizing). The SQL lease holds one pooled checkout for the sweep's duration (the session-scoped advisory lock's same-session acquire/release discipline); the fleet sweep body's other engine clauses — pin_reap, window_take, the mark's list enumeration, window_stage — ride further checkouts under the held lease. A sequential sweep body holds exactly two concurrent checkouts (the lease's plus the current clause's), so the pool must size ≥ 2 or the sweep acquire-waits against itself (a deadlock). Enforced: fleet_pool_size_guard rejects max_size < 2 at every _fleet constructor's entry (errors closed, server-less-testable); the shipped default (8) provides headroom. Recorded in sweep()/sweep_under_leases docs.
• The two arbitration domains' composition: in-process first, SQL second. Implemented once, inside arbitrated_delete: a Proceed from the in-process domain (pins, in-flight markers, protect callbacks, in-process sources — the marker mechanism closes the local race without engine I/O under the lock) still consults the SQL arbitrate (un-expired pin rows of any instance + registered liveness tables, under the SELECT … FOR UPDATE finger-lock); both must pass. On a SQL refusal the marker releases (conclude_delete) and the caller sees Cancelled — a blocked pin mint proceeds against the live entry. Because the composition lives at this one seam, the direct path, the durable window's execution-time re-arbitration, and any future consumer share it uniformly; the SQL answer's failure is operational (Err). pin_check stays available for pins-only answers, unconsumed (its remaining consumer is wave 7's ops renewal probe).
• The maintenance step's clause home. pin_reap(now) → count is a new EngineFleetState clause (ADR-010 §2's "if fleet clauses ever grow, they grow here"); the postgres host body is one statement — DELETE … WHERE expiry <= $1 — with the boundary matching pin_renew's live predicate (expiry > now), never touching un-expired rows; contract-tested (pin_reap_reaps_exactly_the_expired_set, including per-row composition: a key with one expired and one live row keeps the live row). Ordering recorded in sweep_window_fleet: protect consult → reap → execute overdue → mark → stage → release. Reap before the overdue execution is a recorded hygiene choice, not a correctness one — arbitrate already requires expiry > now, so the protection answer is identical either way.
• The fleet liveness verb's shape. Same verb, fleet-shaped argument: Store::register_liveness_source takes a new public LivenessRegistration enum — InProcess(Arc<dyn LivenessSource>) | FleetTable(String) (ADR-012 §6.6's "same verb" wording; a sibling method was rejected). The verb now returns Result<Registration, Error>: the fleet form validates the reference (malformed references error at registration, never interpolate unquoted — postgres.rs's quote_table_reference) and forwards to the shared engine, so its fallibility is real; the in-process form never errors in practice — the pre-fleet signature's infallibility was that form's accident, retired by the unified surface. The store-side registration bookkeeping rides RegistryCore beside the in-process source records (RegistrationKind::{Source, Protect, FleetSource} behind Registration), re-registration of the same reference is idempotent (the original token returns), and deregistration composes the correction flow: the store-side record removal forwards to a new engine-side PostgresBackend::unregister_liveness_source (quote-then-remove).
• In-process sources under fleet: true: allow-and-document. An instance-local registration's protection is instance-local only — it covers this instance's sweeps (mark-time enumeration) and direct deletes (the in-process domain's probes); it is not a fleet liveness form (ADR-008 §2), so the embedder's fleet-wide protection materializes into its liveness table (or pin rows) before the fleet's other instances sweep. Recorded in the LivenessRegistration type docs. The sweep's safe default counts both forms together (registered_liveness_count = sources + fleet tables) — a fleet embedder may register nothing but its table.
• The protect callback's fleet posture: single-node-only, no new machinery. A registered callback remains consultable by this instance's sweeps (its protection is instance-local, abort ProtectFailed precedes everything in the body); the embedder materializes its fleet-wide set into its liveness table or pin rows before invoking sweep. Recorded in the register_liveness_source verb docs.
• The SQL liveness tables are never enumerated into the mark set — consulted at arbitration time only (the fleet body's mark enumerates the pool + in-process sources). Test-asserted: entries protected only by SQL state stage as candidates and cancel at their execution-time re-arbitration (two_instance_exact_counts…, the_fleet_liveness_verb…).
• Superseded test rewritten: postgres_fleet_construction_tests::fleet_declaration_leaves_the_sweep_machinery_to_its_task (its premise — inert machinery arbitrating in-process — was the pre-task state) became fleet_declaration_runs_the_sweep_machinery_on_the_kv_host, asserting the now-live fleet semantics: the instance's own un-expired row cancels the candidate at execution; expiry + reap → execution.
• Verification: cargo test (default) green; cargo test --features postgres green server-less (skipping) and against pglo-poc :15432; cargo test --all-features green; clippy --all-targets -D warnings green (default, postgres, all-features); fmt --check green; cargo doc --no-deps clean (postgres-feature doc warnings unchanged at the pre-existing 21).

Summary

The sweep and delete paths grew their fleet forms under fleet: true — the architecture's most intricate state machine extended across instances, invariants unchanged (a visible pool entry is never deleted while liveness protects it; an in-flight put is never deleted by a sweep started before it committed), the non-fleet forms byte-unchanged:

• The sweep lease: under fleet mode sweep() acquires the in-process single-flight flag and the SQL lease (sweep_lock_acquire on the fleet host — ADR-013 §2's named exclusive semantics), holding both for the whole sweep and releasing the lease on every exit path; contention at either domain aborts as SweeperLock report data with nothing deleted. The pool-sizing composition is verified and enforced (max_size ≥ 2 at the fleet constructors' entry).
• SQL-level delete arbitration: arbitrated_delete composes the two domains — in-process first (cheap, local), the SQL arbitrate second (fleet-authoritative) — on the direct path (GcRefuse, facade-owned) and the durable window's execution-time re-arbitration alike; a foreign instance's un-expired pin row or a liveness-table row refuses cross-instance.
• The maintenance step: EngineFleetState::pin_reap (the seam as the named clause home) reaps exactly the expired pin rows, one statement on the postgres host, contract-tested, driven under the lease before the overdue execution.
• The fleet liveness verb: register_liveness_source takes the fleet-shaped LivenessRegistration argument (ADR-012 §6.6's same-verb wording): the named-table form registers the embedder's own table in the shared engine (schema-ownership rule holds — the engine never creates it; idempotent; deregistration composes); the in-process form remains legal (instance-local protection, documented). Registration now carries its kind; the verb is fallible (the fleet form's error surface).
• The protect callback's fleet posture recorded: single-node-only, instance-local consult remains, materialization is the embedder's duty — no new machinery.
• The fleet sweep body: protect consult → pin_reap → execute overdue (window_take on the kv host; execution-time two-domain re-arbitration; fall-through delete across both tiers) → mark (pool + in-process sources only; SQL liveness consulted at arbitration time, never enumerated) → stage at visible_after = mark_end + window → release. Counts exact relative to the schedule.
• The fleet race across the delay unit-tested: a foreign instance's fleet put landing during the window delay survives via its atomically-committed row plus the execution-time SQL re-arbitration.

Tests: lease contention/reaping between two instances of one shape, the foreign-row GcRefuse, the exact-set reap, the embedder-table protection + correction flow + malformed-reference and fleet: false rejections, the two-instance exact-count shape, and the delay race — env-driven, skipping cleanly server-less, green against the harness server.