--- id: engine-postgres-fleet name: postgres engine-state clause families — the fleet host's full seam status: completed depends_on: [engine-postgres] scope: moderate risk: medium impact: component level: implementation tags: [wave-4, engines, postgres, gc] --- ## Description Implement the engine-state companion seam's **full clause families** on the postgres backend (`src/postgres.rs` or its seam module): the `EngineWindowState` impl **and** the `EngineFleetState` impl per ADR-012 §5's postgres row — every family **yes** (postgres is the fleet host, ADR-012 §4). This is the seam implementation pattern pg-lo's window family reuses (wave 5) and the fleet wave's mechanics ride (wave 6). - **`EngineWindowState`** (the window family — active durably single-node, fleet-active when declared): - **`ensure_gc_schema`** — creates **`gc_pins` + `gc_window`** (the fleet host's both-tables row; unconditionally — the applicability row does not condition on the fleet declaration): `gc_pins (key bytea, owner bytea, expiry bigint)` and `gc_window (key bytea, visible_after bigint)` per ADR-010 §2's schema. `owner` is the store-instance id (ADR-011's vocabulary); the pin row's identity is `(key, owner)` — protection composes by row. The embedder's liveness table is never created here (ADR-008's schema-ownership rule — the sweep reads it and never owns it). Called at store construction by the store core (`store-postgres-construction` wires the call); the clause body is engine-side. - **`window_stage`** — stage candidates into the durable window, **idempotent per `(key, deadline)`** (the upsert discipline — a same-deadline re-stage is idempotent, a new-deadline re-stage updates the row; the sqlite engine's recorded semantics). - **`window_take(now)`** — take overdue candidates (`visible_after <= now`, the `<=` boundary the seam doc pins): the rows **leave the window table and return as keys** — one atomic step (selected rows removed and returned together; a row is never handed out twice). Empty when nothing is overdue; virgin store: empty. - **Timestamp encoding** (recorded design point): the sqlite engine's signed i64 nanosecond count from the Unix epoch, carried as `BIGINT` — the seam's crate-wide encoding (sub-second precision for the test tunings, pre-epoch negatives, full i64 range); the sqlite engine's `time_to_nanos`/`nanos_to_time` helpers are reference, not a dependency (per-engine bodies, one encoding). - **`EngineFleetState`** (the fleet family — the postgres host's alone; **fleet activation is wave 6**: under `fleet: false` the store core never calls these — the clause bodies exist and are contract-tested per applicability): - **`JointTx`** — the pg transaction the pooled client carries (recorded design point: the concrete pooled-transaction type and how it keeps its connection checked out for the tx's life; `Send` per the trait bound). **Ownership rule, pinned in the signature**: the store core holds the tx from `begin_joint_tx` until `pin_commit` consumes it; the engine contributes mechanics and never owns it (ADR-010 §2's ruling). - **`begin_joint_tx`** — open the tx on a pooled connection, uncommitted until `pin_commit`. - **`joint_put`** — the entry half: the engine's put mechanics (the `ON CONFLICT DO NOTHING` CAS clause) executed **inside the held tx**, same first-writer-wins semantics as the public `put`. - **`pin_commit(tx, entries)`** — the joint terminal step: the pin-row inserts **and** the commit — publish and protection land in one step (ADR-005's pin-before-publish, fleet form). On any failure the entire tx aborts: no pin rows, no published entry, zero residue — the stage-hygiene invariant's tx form. Consumes the tx. - **`pin_renew(key, owner)`** — `true` when a **live** (un-expired) `(key, owner)` row existed and its expiry moved to this call's successor time; an expired or absent row is never resurrected: `false`, nothing updated (ADR-012 §2's expiry semantics verbatim). **Named design point — the successor-time source**: the clause signature carries no expiry (ADR-010 §2: the TTL lives with the store's renewal schedule, the engine owns only the SQL) while the row update needs the new expiry — decide the wiring consistent with the seam doc's reading (the natural shape: the engine computes `expiry = now + TTL` from the shipped pin-TTL constant — ADR-008's default 60 s, constructor-tunable when wave 6's fleet constructor lands — and the store owns the TTL/3 cadence, i.e. *when* renewal is called) and record it. - **`pin_check(key)`** — classify against un-expired pin rows: any un-expired `(key, *)` row exists → `PinState::Protected`; else `Unprotected`. The engine never decides deletion — it only classifies. - **`arbitrate(key)`** — the full delete-time arbitration (ADR-008): the un-expired pin rows **plus the embedder's registered liveness tables**, under the `SELECT … FOR UPDATE` finger-lock keyed at delete time, folded into one answer — `true` = liveness protects the key at this instant. The engine-side **registration of named liveness-table references** is part of this body (ADR-012 §6.6: the fleet form of `register_liveness_source` registers a named table reference *in the shared engine*) — a crate-internal engine-side mechanism the store core's fleet registration verb forwards to (the facade verb itself is wave 6). The engine holds table *references*, never liveness semantics; the referenced tables are never created by the engine (the schema-ownership rule). Contract tests use a test-owned embedder table. - **`sweep_lock_acquire()` / `sweep_lock_release()`** — the sweep lease's shipped provider: the **pg advisory lock** with ADR-013 §2's named exclusive semantics — atomic test-and-set (`pg_try_advisory_lock`), one sweeper per pool, `Ok(true)` = held, `Ok(false)` = contention (the second sweeper's sweep aborts with `GcAbortCause::SweeperLock` as report data — the store core's reading, wave 6). **Named design points**: session-scoped vs transaction-scoped (session-scoped — ADR-013 §2's holdership rule requires holdership to end at owner death/session loss, which only the session scope provides); the lock key (a fixed key, schema-qualified or namespaced so co-tenant engines on one instance never collide); and **acquire/release on the same pooled connection** (advisory locks are per-session — the lease must hold its client for its duration; the release-on-another-connection hazard is the recorded pitfall). - **Contract tests** per the applicability row (ADR-012 §5's table is the contract-test matrix): the schema creates both tables (and the embedder's liveness table never); stage idempotence per `(key, deadline)` incl. the re-stage-at-new-deadline form; take returns exactly the overdue set, removes it, leaves future rows; the `<=` boundary; the joint tx atomicity (a forced failure → no pin rows, no entry, zero residue); `pin_renew`'s live/expired/absent semantics; `pin_check`'s two-variant classification; `arbitrate` folding pin rows + a registered test-owned liveness table under the finger-lock; the advisory lock's acquire/contention/release shape and the session-loss reaping semantics (a dropped client's lock is gone — testable by dropping the lease's connection). - **Tests need a live server** — the alkstore harness convention: env-driven (`ALKBLOBS_PG_*`), skip cleanly server-less, acceptance requires green against the harness server (`pglo-poc`, :15432, db `blobs`, `postgres`/`poc`). The store core drives these clauses in wave 6 (fleet mode); the non-fleet store construction consumes `ensure_gc_schema` only (`store-postgres-construction`). This task ships the clauses and their engine-level contract tests. ## Acceptance Criteria - [ ] The postgres backend implements `EngineWindowState` and `EngineFleetState` completely — every clause family of ADR-012 §5's postgres row - [ ] `ensure_gc_schema` creates `gc_pins` + `gc_window`; the embedder's liveness table is never created by the engine (contract-tested) - [ ] `window_stage` idempotent per `(key, deadline)`; `window_take` returns exactly the overdue set (`<= now` boundary), removes it atomically, leaves future rows; empty on none/virgin - [ ] The joint tx atomicity holds: `pin_commit` publishes entry + pin rows in one step; any failure aborts with zero residue (contract-tested) - [ ] `pin_renew` never resurrects an expired/absent row; `pin_check` classifies per the two-variant `PinState`; `arbitrate` folds pin rows + registered liveness-table references under the `FOR UPDATE` finger-lock (contract-tested with a test-owned embedder table) - [ ] The advisory lock: atomic test-and-set, contention → `Ok(false)`, release no-op-safe, session-loss reaping; the same-connection acquire/release discipline recorded - [ ] Named design points recorded: the `JointTx` concrete type, the `pin_renew` successor-time source, the timestamp encoding, the lock key + scope, the lease's connection holding - [ ] Contract tests env-driven, skipping cleanly server-less; green against the harness server at acceptance - [ ] `cargo test --features postgres` green server-less (skipping); `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` green (default + `--features postgres`) ## References - src/gc.rs (`EngineWindowState` + `EngineFleetState` — the seam contracts and their clause docs) - docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §2 (the clause bodies; the tx-ownership ruling; the advisory lock) - docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §3 (fleet activation is a declaration), §4 (the GC-state host), §5 (the applicability table — postgres all-yes), §6.6 (the liveness registration's API home), §6.7 (the executor) - docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md §2 (the fleet mechanism: pin rows, expiry, TTL 60 s, the advisory lock, the staged window) - docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §2 (the sweep lease's named exclusive semantics) - src/sqlite.rs (the window family's reference implementation — same semantics, different dialect) - tasks/engine-postgres.md (the pool/schema/driver posture this rides) ## Notes > Implemented in `src/postgres.rs` (wave 4, this task's only code > change; both impls sit behind the pre-existing `#[cfg(feature = > "postgres")]` module gate). Named design points, as resolved: > > - **`JointTx`** — a crate-internal `PgJointTx`: the deadpool checkout > (`deadpool_postgres::Object`) held for the tx's life, with the > transaction opened as an explicit session-level `BEGIN` and closed > by `COMMIT`/`ROLLBACK` in the clause bodies. tokio-postgres's > `Transaction` borrows its client (`&mut`), so it cannot be > co-owned with the checkout behind one `Send` field; the exclusive > checkout *is* what "the pooled client carries the tx" means > (nothing else can interleave on the session), and ownership stays > with the store core from `begin_joint_tx` to `pin_commit` per the > trait's pinned ruling. Every `pin_commit` failure path rolls the > tx back before surfacing `Err` (zero residue); dropping a tx > mid-flight is outside the seam contract (the store core holds it). > - **`pin_renew` successor time** — the engine computes > `expiry = now + DEFAULT_PIN_TTL` from the shipped 60 s constant > (`DEFAULT_PIN_TTL_SECS`, a backend field, constructor-tunable when > wave 6's fleet constructor lands); the store owns the TTL and the > TTL/3 cadence (`pin_commit`'s rows carry the store-resolved > expiry). The update's predicate requires un-expired rows only, so > expired/absent rows are never resurrected. > - **Timestamp encoding** — signed i64 nanos from the Unix epoch, > `time_to_nanos`/`nanos_to_time` mirrored as per-engine bodies (the > sqlite helpers are reference, one encoding). > - **Sweep lease** — session-scoped pg advisory lock > (`pg_try_advisory_lock`/`pg_advisory_unlock`): the only scope whose > holdership ends at owner death/session loss (ADR-013 §2). Lock key > = FNV-1a over `alkblobs:sweep:{schema}` folded to positive i64 — > the fleet's shared schema is one lock domain (one sweeper per > pool), co-tenant schemas derive distinct keys. **Connection > holding**: the acquired checkout is stored under a > `parking_lot::Mutex>` until release, so > acquire/release ride the same session (the recorded pitfall); > a second acquire answers `Ok(false)` from the mutex before any > pool checkout; release is no-op-safe and never errors (an unlock > failure is the session-loss reaping case itself). > - **Pin rows** — `gc_pins` carries `PRIMARY KEY (key, owner)` (the > row identity) and `pin_commit` inserts with `ON CONFLICT (key, > owner) DO UPDATE SET expiry`: a re-put by the same instance > upsert-renews its row (ADR-010 §4), protection composes by row. > - **`arbitrate`** — one `BEGIN … COMMIT` fold: un-expired pin rows > first, then each registered liveness-table reference, all > `SELECT … FOR UPDATE LIMIT 1` keyed at delete time, one bool > answer. The registration list (a `parking_lot::Mutex>` > behind an `Arc`, shared with clones) is snapshotted before the > server round trip. > - **Liveness-table registration** — > `PostgresBackend::register_liveness_source(ref)` is the engine-side > home of ADR-012 §6.6's fleet registration: every dot-component of > the reference rides `quote_identifier` at registration (hostile > names safe, malformed refs error), re-registration is idempotent, > the engine holds references only and never creates the tables. > - **Virgin tolerance asymmetry** — `window_take` maps SQLSTATE 42P01 > to empty (the sqlite twin's read parity); the fleet clauses carry > no tolerance — they run in fleet mode where store construction has > ensured the GC schema. > - **Error mapping** — new `pg_err` for the clause bodies surfaces > the `DbError` message + SQLSTATE (plain `Display` is just "db > error"); the landed engine's sites keep the original mapping. > - **Contract tests (13 new; 28 postgres tests total)** — the > both-tables schema + the never-created embedder column probe, > stage idempotence/re-stage, take's exact set / atomic re-take / > future rows / `<=` boundary / pre-epoch encoding / virgin shapes, > the joint publish-together path with CAS dedup + upsert renewal, > the injected joint-failure zero-residue abort, `pin_renew`'s > live/expired/absent trio, `pin_check`'s two variants, arbitrate > folding pins + a test-owned embedder table plus the live > finger-lock-blocks-concurrent-renewal proof, and the advisory > lock's acquire/contention/release + foreign-session-loss reaping. > Harness server verified live 2026-10-12. ## Summary > Completed 2026-10-12 (wave 4). `src/postgres.rs` implements > `EngineWindowState` (the fleet host's both-tables `ensure_gc_schema`, > idempotent `window_stage`, atomic `DELETE … RETURNING` > `window_take` with the `<=` boundary) and `EngineFleetState` > (PgJointTx joint entry+pin tx, `pin_commit` with rollback-on-failure > and upsert-renewal pin-row inserts, `pin_renew`'s no-resurrection > expiry semantics, two-variant `pin_check`, the `FOR UPDATE` > finger-lock `arbitrate` folding pin rows + registered liveness-table > references, and the session-scoped advisory sweep lease with its > same-connection holding discipline) — every clause family of > ADR-012 §5's postgres row, contract-tested in-module with the > established env-driven harness convention (skip cleanly server-less, > green against the live pglo-poc server). 13 new tests; fleet > activation remains wave 6 (the store core never calls the fleet > clauses under `fleet: false`), and the `ensure_gc_schema` > constructor wiring remains `store-postgres-construction`. Verified: > `cargo test` (69/19/3 ok), `cargo test --features postgres` > server-less (97 ok, 28 skips; with server 97 ok, all live), > `cargo test --all-features` (152/54/4 ok), `cargo clippy > --all-targets -- -D warnings` and `--all-features --all-targets` > green, `cargo fmt --check` green, `cargo doc --no-deps` clean.