--- id: suite-column-fleet name: Conformance suite — the fleet column (exact-count fleet sweep outcomes under concurrency) status: completed depends_on: [store-fleet-put, store-fleet-sweep] scope: moderate risk: medium impact: phase level: implementation tags: [wave-6, suite, fleet, postgres] --- ## Description Grow the **fleet column** — the suite's proof that the architecture's most intricate state machine holds under concurrency (the wave's gate: "exact-count fleet sweep outcomes against live postgres", docs/plans/implementation.md §Wave 6). The column runs under `--features postgres` (fleet mode requires kv = postgres — the composite predicate), rides the SKIP posture (server-less runs skip cleanly; acceptance requires green **against the server**), and extends the factory contract to the fleet shape: **two store instances over one shared pool**. - **The fleet factory** (`tests/conformance/fleet_column.rs`): the postgres column's schema-per-open pattern adapted — a fleet factory mints **one schema** and vends **two (or more) `Store` instances over it**, each `fleet: true`, each with its own owner id (the instances share the pool; the pin rows compose by owner — the fleet's actual definition, ADR-011/ADR-012 §3). Teardown drops exactly the minted schemas (`DROP SCHEMA … CASCADE`, the postgres column's registry pattern — panicking-row-safe, idempotent, owned). The test tuning: window delay `0` (execute-on-next-sweep) and a **short test TTL** (the fleet constructor parameter — the renewal/expiry rows need observable cadences; the TTL is a constructor parameter, not API, so the column declares its tuning). A kv-only fleet factory (the re-routing posture's client shape) and a consolidation fleet factory (kv = postgres + large = pg-lo, same instance) join as the column's second/third shapes. - **The fleet-validity rows** (the predicate at the constructor): every fleet-valid configuration constructs; every non-fleet-valid configuration is a constructor error (kv ≠ postgres, `mem` anywhere, `local` without the shared-media declaration, pg-lo cross-instance) — the predicate's clause-per-row form. - **The joint entry+pin tx rows** (through the public `put`, not the seam): entry + pin row commit together (an independent connection sees both or neither); the dedup re-put upsert-renews the row; a second instance's re-put composes a second row (protection by row); the pg-lo consolidation round-trip (one tx over the shared instance, byte-exact large-entry form). - **The TTL/renewal rows**: the guard's TTL/3 renewal moves the row's expiry (the short test TTL makes the cadence observable); a guard dropped past expiry never resurrects its row (the never-resurrect semantics through the facade); a paused holder (renewal stopped) loses protection at expiry — the delete-then-recover backstop (a byte-identical re-put recovers). - **The exact-count fleet sweep rows** (the wave's gate — INV-4's engine-gated fleet form, store-api.md §Invariants 4: "fleet postures assert it under DB-backed pins + the advisory-locked sweep, with the exact-count sweep-outcome shape run against dockerized postgres"): - the **two-instance exact-count shape**: instance A pins an entry (its row), instance B sweeps — the pinned entry survives, the unpinned twins delete, counts exact against the staged schedule; - the **lease contention row**: two instances sweep concurrently — exactly one sweeps, the other aborts `SweeperLock` as report data with nothing deleted; - the **cross-instance arbitration row**: a foreign owner's un-expired pin row refuses instance B's direct delete (`GcRefuse`); after the row expires (renewal stopped, short TTL), the delete proceeds; - the **race across the window delay** in its fleet form: a put landing during the delay (its row committed before its entry is visible) survives the next sweep's execution-time SQL re-arbitration; - the **maintenance-step row**: expired rows reaped exactly (the GC-of-GC cleanup), un-expired rows survive; - the **fleet liveness row**: a test-owned embedder table's rows protect through the fleet registration verb (the schema-ownership rule asserted — the engine never created the table). - **The kv-only re-routing row**: the fleet kv-only client shape rejects over-threshold puts at put time (ADR-012 §6.2's machinery, fleet edition) — the re-routing posture's store-side half. - **README amendments** (the convention's normative test-side home): the fleet rows table in §Columns beyond the reference set; the column plan's SKIP-posture note naming the fleet column; current state. - **Connection settings ride the environment** — the established harness convention: `ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`, the reachability probe gating the rows. **The harness server**: dockerized `pglo-poc` (`postgres:16-alpine`) on :15432, db `blobs`, user/password `postgres`/`poc` — verified live 2026-10-12; the container is long-running in the dev environment. Concurrency rows are deterministic by construction (the lease/arbitration outcomes are decided by the server, not by timing) — where a row needs ordering, the window delay and the TTL tunings provide it; no sleep-based assertions. ## Acceptance Criteria - [x] The fleet column green **against the harness server** (`pglo-poc`, :15432): the fleet-validity rows, the joint-tx rows, the TTL/renewal rows, the exact-count fleet sweep rows (two-instance survival, lease contention, cross-instance arbitration, the race across the delay, the maintenance step, the fleet liveness row), and the kv-only re-routing row — every row ADR-stamped and traceable to the backlog - [x] The fleet factory contract holds: one schema, two+ instances over it (distinct owner ids), teardown drops exactly the minted schemas, panicking-row-safe — structurally exercised - [x] Server-less runs skip cleanly (the reachability probe; the default-build gates stay green with the column skipped) - [x] README amendments recorded (the fleet rows table; current state) - [x] `cargo test` (default) green — the column compiles out; `cargo test --features postgres` green server-less (skipping) and green against the server at acceptance; `cargo test --all-features` green (the consolidation factory's home); `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` green (default + the postgres/pg-lo variants + `--all-features`) ## References - tests/conformance/README.md (the conventions this grows against; §The factory contract, §The SKIP posture, §The schedule-relative reading, §Columns beyond the reference set) - tests/conformance/postgres_column.rs (the SQL-backed column pattern — the schema-per-open factory this extends to the fleet shape) - tests/conformance/factory.rs (the factory contract) - docs/architecture/store-api.md (§Invariants 4 — the engine-gated fleet form this column asserts) - docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md §2 (the fleet mechanism the rows assert) - docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §3 (the declaration), §4 (the host), §6.6 (the fleet liveness verb) - docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §2 (the lease semantics the contention row asserts) - docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md §5 (the exact-count sweep-outcome gate against dockerized postgres) - tasks/store-fleet-put.md, tasks/store-fleet-sweep.md, tasks/store-fleet-construction.md (the machinery the rows exercise) ## Notes Landed as `tests/conformance/fleet_column.rs` (cfg `feature = "postgres"`; its consolidation pieces cfg the `postgres`+`pg-lo` pair), wired in `tests/conformance/main.rs`, README amended (§Columns beyond the reference set's fleet rows table, §Column plan, §Current state — six columns). Three factories: `FleetDualTierFactory` (one minted schema, two instances over it — each its own owner id and private local root riding `FleetSharedMedia::asserted`), `FleetKvOnlyFactory` (the re-routing client shape; implements `StoreFactory` so the reference kv-only rejection row mirrors verbatim), `FleetConsolidationFactory` (a dedicated database, kv + pg-lo schemas, two fleet instances; the pg-lo column's LO-isolation corollary — LO reaping, `WITH (FORCE)` drop, drop-guard janitors; a `TrackedDb` constructor was added to pg_lo_column.rs since the janitor field is private). The oracle reuses the postgres column's skip probe (`harness_ready`) and observes the durable state over independent connections (pool/window/pin-row counts, schema/table existence, the embedder-table instruments, and the manual-expiry instrument for the abandonment shape). Column tuning: window delay `0`, TEST_TTL = 900 ms (TTL/3 ≈ 300 ms), waits are the expiry cadence itself or the manual instrument — no sleep-based race assertions. Recorded deviations from the task text, none blocking: - The composite predicate's *structural* non-fleet-valid shapes (kv ≠ postgres, mem anywhere, `local` undeclared) have no fleet constructor to reach through (the recorded enforcement map) — the column asserts the constructor-set shape in prose + the runtime rejects that do exist (zero TTL, single-connection pool, the same-instance probe's failure posture). A `TrackedDb::new` constructor was added rather than widening the field's visibility. - The pg-lo **cross-instance mismatch** leg is unexercisable against the one-instance harness server (the probe reads `pg_control_system()` off one data directory); the row tests the probe's errors-closed failure posture (an unreachable large instance rejects at the constructor) and records the mismatch leg — the same posture the engine suite's tests carry. - The renewal row's "renewed == at_drop" compare reads at_drop *after* the drop and asserts `>=` — one TTL/3 tick may legitimately land between the pre-drop read and the drop; the frozen-after-drop assertion (EXPIRY_HORIZON, `==`) is the never-leaks proof. ## Summary The fleet column is green: the wave's gate rows run against the harness server under `--features postgres` (17 rows + 2 structural factory tests) and under `--all-features` (the consolidation rows join) — the fleet-validity rows, the joint-tx rows (both-or-neither, the upsert-renew, second-row composition, the byte-exact consolidation round-trip), the TTL/renewal rows (cadence, never-resurrect, paused-holder + recover), the exact-count fleet sweep rows (two-instance survival with exact counts, lease contention as report data, cross-instance arbitration until expiry, the race closed by the row, the maintenance step's exact reap, the fleet liveness verb with the schema-ownership rule), and the kv-only re-routing row — every row ADR-stamped and traced. The fleet factory contract holds structurally (one schema, two instances, distinct owner ids, owned idempotent teardown, panicking-row-safe); server-less runs skip cleanly; the default build compiles the column out. Verification: `cargo test` (default) green; `cargo test --features postgres` green server-less (skipping) and green against `pglo-poc` (:15432); `cargo test --all-features` green against the server (125 conformance tests + the full library suite, 226 tests); `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` green on the default, postgres, and all-features variants.