feat(store): the fleet GC protocol — the sweep lease, SQL arbitration, pin reaping, and the fleet liveness verb (wave 6)
Grow the sweep and delete paths into their fleet forms under fleet: true (ADR-008 §2, ADR-010 §2, ADR-012 §1/§5/§6.6/§6.7, ADR-013 §2); the non-fleet forms are byte-unchanged. - the sweep lease: sweep() composes the in-process single-flight flag with the SQL lease (sweep_lock_acquire on the fleet host), both held for the whole sweep and released on every exit path; contention at either domain aborts as SweeperLock report data, nothing deleted; the pool-sizing composition enforced (max_size >= 2 at the fleet constructors' entry — the lease holds one checkout; the body's clauses ride further checkouts) - SQL-level delete arbitration: arbitrated_delete composes the two domains (in-process first — cheap, local; the engine's arbitrate second — fleet-authoritative) on the direct path (GcRefuse) and the window execution's re-arbitration; a foreign instance's un-expired pin row or liveness-table row refuses cross-instance - the maintenance step: EngineFleetState::pin_reap (the seam as the named clause home) reaps exactly the expired pin rows, one statement on the postgres host, contract-tested, driven under the lease before the overdue execution - the fleet liveness verb: register_liveness_source takes the fleet-shaped LivenessRegistration argument (ADR-012 §6.6's same-verb wording) — the named-table form registers the embedder's own table (schema-ownership rule holds; idempotent; deregistration composes via a new engine-side unregister), the in-process form stays legal (instance-local protection, documented); Registration carries its kind; the verb is fallible (the fleet form's error surface) - the protect callback's fleet posture recorded: single-node-only, instance-local consult remains, materialization is the embedder's duty — no new machinery - the fleet sweep body: protect consult → reap → execute overdue (two-domain re-arbitration, fall-through delete across both tiers) → mark (pool + in-process sources; SQL liveness consulted at arbitration time, never enumerated) → stage → release; counts exact relative to the schedule - tests: lease contention/reaping between two instances, the foreign-row GcRefuse, the exact-set reap, the embedder-table protection + correction flow + malformed and fleet:false rejections, the two-instance exact-count shape, and the delay race in its fleet form; the wave-6 inert-machinery construction test rewritten to the now-live fleet semantics Verification: cargo test (default) green; cargo test --features postgres green server-less (skipping) and against pglo-poc :15432; cargo test --all-features green; cargo clippy --all-targets -D warnings green (default, postgres, all-features); cargo fmt --check green; cargo doc --no-deps clean (postgres-feature warnings unchanged at the pre-existing 21).
This commit is contained in:
1 parent
950af9ab0d
commit
3a51bb43fc
7 files changed
+1550
-129
No files matched your search
+30
-10
@@ -1,7 +1,7 @@
|
||||
---
|
||||
id: store-fleet-sweep
|
||||
name: the fleet GC protocol — SQL sweep lease, SQL-level arbitration, pin reaping, the fleet liveness verb
|
||||
status: pending
|
||||
status: completed
|
||||
depends_on: [store-fleet-construction]
|
||||
scope: broad
|
||||
risk: high
|
||||
@@ -128,29 +128,29 @@ deleted by a sweep started before it committed). The non-fleet forms
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Under `fleet: true`, the sweep acquires the SQL lease (the
|
||||
- [x] Under `fleet: true`, the sweep acquires the SQL lease (the
|
||||
ADR-013 §2 semantics: atomic test-and-set, one sweeper per
|
||||
pool across instances, contention = `SweeperLock` report abort
|
||||
with nothing deleted); the in-process flag composes; under
|
||||
`fleet: false` the sweep forms are byte-unchanged
|
||||
- [ ] Delete-time arbitration is SQL-level under fleet mode on both
|
||||
- [x] Delete-time arbitration is SQL-level under fleet mode on both
|
||||
the direct path (`GcRefuse`) and the window execution
|
||||
(execution-time re-arbitration); a foreign owner's un-expired
|
||||
pin row protects (cross-instance proof); the two-domain
|
||||
composition decided and recorded
|
||||
- [ ] The expired-pin reaping maintenance step landed as a new
|
||||
- [x] The expired-pin reaping maintenance step landed as a new
|
||||
`EngineFleetState` clause on the postgres host (the seam is the
|
||||
named home; no raw-SQL side-channel), contract-tested, run
|
||||
under the lease
|
||||
- [ ] The fleet liveness registration verb landed (the named table
|
||||
- [x] The fleet liveness registration verb landed (the named table
|
||||
reference form; the verb-shape and in-process-source design
|
||||
points decided and recorded; the schema-ownership rule holds)
|
||||
- [ ] The protect callback's fleet posture recorded (instance-local;
|
||||
- [x] The protect callback's fleet posture recorded (instance-local;
|
||||
materialization is the embedder's fleet-wide duty)
|
||||
- [ ] The fleet race property unit-tested across the window delay (a
|
||||
- [x] The fleet race property unit-tested across the window delay (a
|
||||
put landing during the delay survives via the row + the
|
||||
execution-time SQL re-arbitration)
|
||||
- [ ] `cargo test` (default) green; `cargo test --features postgres`
|
||||
- [x] `cargo test` (default) green; `cargo test --features postgres`
|
||||
green server-less (skipping) and against the harness server at
|
||||
acceptance; `cargo clippy --all-targets -- -D warnings` and
|
||||
`cargo fmt --check` green (default + `--features postgres`);
|
||||
@@ -182,8 +182,28 @@ deleted by a sweep started before it committed). The non-fleet forms
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
Landed in `src/gc.rs`, `src/facade.rs`, `src/postgres.rs`, `src/store.rs`, `tests/conformance/rows.rs` (call-site mechanical only). Decisions and design points, as recorded:
|
||||
|
||||
- **The lease's hold across the sweep body (pool sizing).** The SQL lease holds **one pooled checkout** for the sweep's duration (the session-scoped advisory lock's same-session acquire/release discipline); the fleet sweep body's other engine clauses — `pin_reap`, `window_take`, the mark's `list` enumeration, `window_stage` — ride **further checkouts** under the held lease. A sequential sweep body holds exactly two concurrent checkouts (the lease's plus the current clause's), so the pool must size ≥ 2 or the sweep acquire-waits against itself (a deadlock). Enforced: `fleet_pool_size_guard` rejects `max_size < 2` at every `_fleet` constructor's entry (errors closed, server-less-testable); the shipped default (8) provides headroom. Recorded in `sweep()`/`sweep_under_leases` docs.
|
||||
- **The two arbitration domains' composition: in-process first, SQL second.** Implemented once, inside `arbitrated_delete`: a `Proceed` from the in-process domain (pins, in-flight markers, protect callbacks, in-process sources — the marker mechanism closes the local race without engine I/O under the lock) still consults the SQL `arbitrate` (un-expired pin rows of *any* instance + registered liveness tables, under the `SELECT … FOR UPDATE` finger-lock); both must pass. On a SQL refusal the marker releases (`conclude_delete`) and the caller sees `Cancelled` — a blocked pin mint proceeds against the live entry. Because the composition lives at this one seam, the direct path, the durable window's execution-time re-arbitration, and any future consumer share it uniformly; the SQL answer's failure is operational (`Err`). `pin_check` stays available for pins-only answers, unconsumed (its remaining consumer is wave 7's ops renewal probe).
|
||||
- **The maintenance step's clause home.** `pin_reap(now) → count` is a new `EngineFleetState` clause (ADR-010 §2's "if fleet clauses ever grow, they grow here"); the postgres host body is one statement — `DELETE … WHERE expiry <= $1` — with the boundary matching `pin_renew`'s live predicate (`expiry > now`), never touching un-expired rows; contract-tested (`pin_reap_reaps_exactly_the_expired_set`, including per-row composition: a key with one expired and one live row keeps the live row). Ordering recorded in `sweep_window_fleet`: protect consult → reap → execute overdue → mark → stage → release. Reap *before* the overdue execution is a recorded hygiene choice, not a correctness one — `arbitrate` already requires `expiry > now`, so the protection answer is identical either way.
|
||||
- **The fleet liveness verb's shape.** Same verb, fleet-shaped argument: `Store::register_liveness_source` takes a new public `LivenessRegistration` enum — `InProcess(Arc<dyn LivenessSource>)` | `FleetTable(String)` (ADR-012 §6.6's "same verb" wording; a sibling method was rejected). The verb now returns `Result<Registration, Error>`: the fleet form validates the reference (malformed references error at registration, never interpolate unquoted — `postgres.rs`'s `quote_table_reference`) and forwards to the shared engine, so its fallibility is real; the in-process form never errors in practice — the pre-fleet signature's infallibility was that form's accident, retired by the unified surface. The store-side registration bookkeeping rides `RegistryCore` beside the in-process source records (`RegistrationKind::{Source, Protect, FleetSource}` behind `Registration`), re-registration of the same reference is idempotent (the original token returns), and deregistration composes the correction flow: the store-side record removal forwards to a new engine-side `PostgresBackend::unregister_liveness_source` (quote-then-remove).
|
||||
- **In-process sources under `fleet: true`: allow-and-document.** An instance-local registration's protection is instance-local only — it covers this instance's sweeps (mark-time enumeration) and direct deletes (the in-process domain's probes); it is not a *fleet* liveness form (ADR-008 §2), so the embedder's fleet-wide protection materializes into its liveness table (or pin rows) before the fleet's other instances sweep. Recorded in the `LivenessRegistration` type docs. The sweep's safe default counts both forms together (`registered_liveness_count = sources + fleet tables`) — a fleet embedder may register nothing but its table.
|
||||
- **The protect callback's fleet posture: single-node-only, no new machinery.** A registered callback remains consultable by *this* instance's sweeps (its protection is instance-local, abort `ProtectFailed` precedes everything in the body); the embedder materializes its fleet-wide set into its liveness table or pin rows before invoking sweep. Recorded in the `register_liveness_source` verb docs.
|
||||
- **The SQL liveness tables are never enumerated into the mark set** — consulted at arbitration time only (the fleet body's mark enumerates the pool + in-process sources). Test-asserted: entries protected only by SQL state stage as candidates and cancel at their execution-time re-arbitration (`two_instance_exact_counts…`, `the_fleet_liveness_verb…`).
|
||||
- **Superseded test rewritten:** `postgres_fleet_construction_tests::fleet_declaration_leaves_the_sweep_machinery_to_its_task` (its premise — inert machinery arbitrating in-process — was the pre-task state) became `fleet_declaration_runs_the_sweep_machinery_on_the_kv_host`, asserting the now-live fleet semantics: the instance's own un-expired row cancels the candidate at execution; expiry + reap → execution.
|
||||
- **Verification:** `cargo test` (default) green; `cargo test --features postgres` green server-less (skipping) and against `pglo-poc` :15432; `cargo test --all-features` green; `clippy --all-targets -D warnings` green (default, `postgres`, `all-features`); `fmt --check` green; `cargo doc --no-deps` clean (postgres-feature doc warnings unchanged at the pre-existing 21).
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
The sweep and delete paths grew their fleet forms under `fleet: true` — the architecture's most intricate state machine extended across instances, invariants unchanged (a visible pool entry is never deleted while liveness protects it; an in-flight put is never deleted by a sweep started before it committed), the non-fleet forms byte-unchanged:
|
||||
|
||||
- **The sweep lease**: under fleet mode `sweep()` acquires the in-process single-flight flag *and* the SQL lease (`sweep_lock_acquire` on the fleet host — ADR-013 §2's named exclusive semantics), holding both for the whole sweep and releasing the lease on every exit path; contention at either domain aborts as `SweeperLock` report data with nothing deleted. The pool-sizing composition is verified and enforced (`max_size ≥ 2` at the fleet constructors' entry).
|
||||
- **SQL-level delete arbitration**: `arbitrated_delete` composes the two domains — in-process first (cheap, local), the SQL `arbitrate` second (fleet-authoritative) — on the direct path (`GcRefuse`, facade-owned) and the durable window's execution-time re-arbitration alike; a foreign instance's un-expired pin row or a liveness-table row refuses cross-instance.
|
||||
- **The maintenance step**: `EngineFleetState::pin_reap` (the seam as the named clause home) reaps exactly the expired pin rows, one statement on the postgres host, contract-tested, driven under the lease before the overdue execution.
|
||||
- **The fleet liveness verb**: `register_liveness_source` takes the fleet-shaped `LivenessRegistration` argument (ADR-012 §6.6's same-verb wording): the named-table form registers the embedder's own table in the shared engine (schema-ownership rule holds — the engine never creates it; idempotent; deregistration composes); the in-process form remains legal (instance-local protection, documented). `Registration` now carries its kind; the verb is fallible (the fleet form's error surface).
|
||||
- **The protect callback's fleet posture** recorded: single-node-only, instance-local consult remains, materialization is the embedder's duty — no new machinery.
|
||||
- **The fleet sweep body**: protect consult → `pin_reap` → execute overdue (`window_take` on the kv host; execution-time two-domain re-arbitration; fall-through delete across both tiers) → mark (pool + in-process sources only; SQL liveness consulted at arbitration time, never enumerated) → stage at `visible_after = mark_end + window` → release. Counts exact relative to the schedule.
|
||||
- **The fleet race across the delay** unit-tested: a foreign instance's fleet put landing during the window delay survives via its atomically-committed row plus the execution-time SQL re-arbitration.
|
||||
|
||||
Tests: lease contention/reaping between two instances of one shape, the foreign-row `GcRefuse`, the exact-set reap, the embedder-table protection + correction flow + malformed-reference and `fleet: false` rejections, the two-instance exact-count shape, and the delay race — env-driven, skipping cleanly server-less, green against the harness server.
|
||||
Reference in new issue
Block a user