pg-fix-scheduler-tick-seam — the pg scheduler's transient-fault tick seam lands, retiring review 003 Finding 1 (MEDIUM: the tick retry loop was dead code under test — the last uncovered hardening arm of the wave-4 fix batch, the exact class review 002's live bugs came from). The seam follows the sqlite-commit-error-arm pattern of record: a cfg(test) per-store counting fault arm (crate::seam::tick_fault — Flag = Arc<AtomicU64>, born disarmed at 0, arm(n) adds, take() consumes one via CAS try_update; a counting arm because a boolean cannot reach the exhaustion arm — arm(TICK_RETRIES+1) faults the loop's whole budget while arm(1) leaves the later ticks real) feeding a fabricated pool/database-shaped opaque Database error into the REAL retry loop — both tests drive the full run_schedules → tick_with_retries → tick_once path end-to-end, no loop replication. Plumbing per the house pattern: cfg-gated tick_fault field on EngineCtx and PgStore (engine_ctx() carries the clone; production builds never materialize it), cfg-gated fault param on tick_once (the seam check sits before the real attempt — one attempt faults, never ticks) and tick_with_retries, cfg-branched attempt calls; PgStore::arm_tick_fault(faults) arming surface + tick_fault_flag() observation surface (a Flag clone so the consumption pin survives the store moving into a runner task). Two arms pinned: (a) one_faulted_tick_is_retried_and_the_job_fires — fault consumed on attempt 1, the 250 ms backoff sleeps, the retried attempt fires the backdated due boundary into the queue, exactly-one consumption, the runner survives the hiccup to a clean stop Ok(()); (b) tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse — the exhaustion exits typed (opaque Database, never LeadershipLost; the tick-phase context rides the source chain with the fabricated fault at its bottom; wall-clock ≥ 1.7 s proving the real backoff window), the leadership lock row is left (not released) to lapse at its TTL (present, expiry bounded by the run's TTL horizon), exact consumption, and the store remains fully usable after — the "runner survives" pin as the module docs' contract states it. Replay-proofed live: with the seam's consumption disabled the exhaustion test cannot complete (the runner loops forever — no fault ever enters the budget; the mutated run's timeout is itself the proof); the test is then bounded (20 s) so a future regression of this shape fails, not hangs. LEADER_TTL_S widened private → pub(crate) for the lock-left-to-TTL expiry pin. Verified: full pg column live vs the harness (124 lib + 25 suite + 9 schema; new tests green solo ×3); cargo test --workspace green server-less, skips clean (core 25 + harness 3, sqlite 191 + 25); clippy --all-targets -D warnings clean; fmt clean; cargo-llvm-cov confirms the Finding-1 sites (the retry/backoff/exhaustion-arm region + the tick_once seam arm) carry zero missed lines; taskgraph validate green (61 tasks)

tasks/pg-fix-scheduler-tick-seam.md

@@ -1,7 +1,7 @@
 ---
 id: pg-fix-scheduler-tick-seam
 name: "pg scheduler — transient-fault tick seam: retry/exhaustion arms tested (review 003 Finding 1)"
-status: pending
+status: completed
 depends_on: []
 scope: narrow
 risk: low
@@ -42,13 +42,13 @@ disarmed, consumed-on-use.
 
 ## Acceptance Criteria
 
-- [ ] The seam lands cfg(test)-gated; the real `tick_with_retries`
+- [x] The seam lands cfg(test)-gated; the real `tick_with_retries`
       executes under test (no loop replication)
-- [ ] Arms (a) and (b) pinned as above; the runner survives
+- [x] Arms (a) and (b) pinned as above; the runner survives
       exhaustion
-- [ ] `cargo test -p alkstore-postgres` green against the harness
+- [x] `cargo test -p alkstore-postgres` green against the harness
       server; skips clean server-less
-- [ ] Clippy `-D warnings`, fmt clean; workspace gates green
+- [x] Clippy `-D warnings`, fmt clean; workspace gates green
 
 ## References
 
@@ -61,8 +61,104 @@ disarmed, consumed-on-use.
 
 ## Notes
 
-> To be filled by implementation agent
+> Decisions of record the implementation made that the description
+> didn't pin:
+
+- **The seam is a per-store *counting* fault arm, not a boolean.** A
+  boolean arm (the sqlite commit-fault shape) reaches only the
+  retry-once arm; exhaustion needs the fault to survive the loop's
+  whole budget. `crate::seam::tick_fault` holds
+  `Flag = Arc<AtomicU64>` — born disarmed at 0, `arm(n)` adds n,
+  `take` consumes one (CAS `try_update`, no negative underflow) — so
+  arm(1) gives the retry-then-success arm and arm(`TICK_RETRIES + 1`)
+  the exhaustion arm with one mechanism. Consumption-exactness is
+  pinned by loading the flag after the run (0 = exactly the armed
+  count consumed).
+- **Plumbing follows the `sqlite-commit-error-arm` precedent
+  exactly** (the `pg-fix-forwarder-reconnect` house pattern):
+  `#[cfg(test)]` field on `EngineCtx` and `PgStore`;
+  `PgStore::engine_ctx()` carries the store's flag clone into every
+  scheduler ctx; `#[cfg(test)]` param on `tick_once` / `tick_with_retries`
+  (the seam check sits at `tick_once`'s top — one *attempt* faults,
+  never ticks) with cfg-branched attempt statements — production
+  builds compile the plain path and never materialize the field.
+  Arming/observation surfaces: `PgStore::arm_tick_fault(faults)` and
+  `PgStore::tick_fault_flag()` (a `Flag` clone so the consumption pin
+  survives the store moving into a runner task — a load is enough,
+  no extra accessor). Per-store `Arc` scoping keeps parallel test
+  stores isolated.
+- **The fault enters the loop via the entry points it already calls**
+  — both tests drive the full real path (`run_schedules` → the loop's
+  `tick_with_retries` → `tick_once`), no loop or error-shape
+  replication anywhere. The fabricated error is pool/database-shaped
+  (`Error::database(io::Error)` — the taxonomy's opaque fallback; the
+  loop retries on any `Err`, so the shape is an honesty choice, not a
+  correctness need).
+- **"The runner survives exhaustion" pinned as written.** The
+  exhausting loop's exit *is* the typed error (the module docs'
+  contract — exhaustion exits `Err`); "survives" is pinned as: no
+  panic, opaque `Database` (never `LeadershipLost`), the tick-phase
+  context riding the source chain with the fabricated fault at its
+  bottom, the leadership lock row left (not released) to lapse at its
+  TTL (present, expiry bounded by the TTL of the run — never renewed
+  past the exit), and the store fully usable after (subsequent entry
+  points work). Arm (a) carries the *hiccups-don't-kill* survival
+  property (fault → backoff → real tick fires the due boundary →
+  clean stop `Ok(())`).
+- **Replay-proofed live** (the sqlite task's discipline): with the
+  seam's fault consumption disabled (`false && take(…)` probe), the
+  exhaustion test cannot complete — the runner loops forever on its
+  nothing-due schedule because no fault ever enters the budget (the
+  mutation run was killed on timeout, which is itself the proof the
+  seam is load-bearing); reverted, green. The exhaustion test was
+  then wrapped in a 20 s bounded wait so a future regression of this
+  shape fails instead of hanging.
+- **The backoff-sleep arms are pinned by wall-clock**, not value: the
+  exhaustion run must take ≥ 1.7 s (the 250 ms → 500 ms → 1 s window;
+  a fast-fail exit would not). The retry/exhaustion arms' `eprintln`
+  diagnostics execute (lcov-verified) but are not value-pinned —
+  stderr capture isn't in the harness convention; the exhaustion
+  *context* message (chain-carried) is the pinned diagnostic shape.
+- `LEADER_TTL_S` widened `private` → `pub(crate)` (the lock-left-to-
+  TTL expiry pin reads it; no production-path change).
+- lcov after: the Finding-1 sites (the retry loop — retry arm, backoff
+  sleeps, exhaustion `context_error`, diagnostics — and the `tick_once`
+  seam arm) carry **zero missed lines**.
 
 ## Summary
 
-> To be filled on completion
+> What landed, verified how:
+
+- **`alkstore-postgres/src/seam.rs`**: the `#[cfg(test)]`
+  `tick_fault` module — the per-store counting `Flag`
+  (`Arc<AtomicU64>`), `disarmed`/`arm`/`take` (consume-one CAS), and
+  the fabricated pool/database-shaped transient error, with the
+  pattern provenance (review 003 Finding 1 /
+  `sqlite-commit-error-arm`) in its doc comment.
+- **`alkstore-postgres/src/scheduler.rs`**: cfg-gated `tick_fault`
+  field on `EngineCtx` (wildcard-safe destructures elsewhere); the
+  cfg-gated fault param on `tick_once` (seam check precedes the real
+  attempt) and `tick_with_retries`, attempt calls cfg-branched;
+  `run_schedules` threads the store's flag into the loop. The
+  production path never materializes the seam.
+- **`alkstore-postgres/src/store.rs`**: cfg-gated `tick_fault` field
+  on `PgStore` (born `disarmed()`), `engine_ctx()` clones it into the
+  scheduler ctx, and the `arm_tick_fault` / `tick_fault_flag`
+  test-observation surfaces.
+- **`alkstore-postgres/src/store/scheduler_tests.rs`**: two pins —
+  `one_faulted_tick_is_retried_and_the_job_fires` (arm a: faulted
+  attempt 1 → 250 ms backoff → the retried attempt fires the
+  backdated due boundary; exactly-one consumption; clean stop `Ok(())`)
+  and `tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse`
+  (arm b: `TICK_RETRIES + 1` armed faults exhaust the budget; ≥ 1.7 s
+  real backoff window; opaque `Database` with the tick-phase context
+  and the fabricated fault in the source chain; exact consumption; the
+  leadership lock row left to its TTL lapse; the store usable after).
+- **Gates**: `cargo build`; the full pg column live vs the harness
+  (124 lib + 25 suite + 9 schema; the two new tests green solo ×3 —
+  4.2–4.4 s); `cargo test --workspace` server-less green (skips
+  clean: core 25 + harness 3; sqlite 191 + 25); `cargo clippy
+  --all-targets -- -D warnings`; `cargo fmt --check`. cargo-llvm-cov
+  confirms scheduler.rs's retry-loop region zero missed lines (the
+  crate stands at 90.73% scheduler line coverage; the residue is the
+  accepted defensive-arm posture).