Wave-4 fix-batch decomposition: seven pg-fix tasks + review-wave-4-fixes gate from the general review (002) — forwarder reconnect retry (Finding 1, with the failed-connect test seam), tx pg_notify wakes retiring F-1's engine arm (Finding 2), max_size/DSN-options open path (Finding 3), stale-UNLISTEN generation drop (Finding 4), scheduler quarantine/retry (Finding 5), decode dedupe + cleanups, doc alignment; wave 5 gates on the two HIGH fixes landing

tasks/pg-fix-scheduler-resilience.md

@@ -0,0 +1,112 @@
+---
+id: pg-fix-scheduler-resilience
+name: Scheduler runner resilience — quarantine bad rows, retry transient errors (review 002 Finding 5)
+status: pending
+depends_on: []
+scope: narrow
+risk: medium
+impact: component
+level: implementation
+tags: [wave-4-fixes, postgres-engine, scheduler]
+---
+
+## Description
+
+Fix the LOW/MEDIUM finding from the wave-4 general review
+(`docs/reviews/002-wave-4-general-review.md` §Finding 5): one bad row
+or one transient error permanently kills `run_schedules`. Sites:
+`alkstore-postgres/src/scheduler.rs:403`
+(`parse_every_interval(&row.spec)?`), `:429` (fire
+`enqueue_row(...)?`), `:519-526` (the `in_tx` tick `?`) — all
+propagating straight out of the leader loop. The `@every` grammar is
+re-parsed from stored text per tick (unvalidated-on-read), so a
+tampered or future-foreign spec row makes the whole runner exit with
+`InvalidSpec`; one transient pool/database error does the same. The
+consumer's respawn recipe makes this survivable but brittle —
+schedules never fire without a runner, and the runner dies on any
+hiccup rather than skipping/retrying. The exiting error also carries
+no schedule-name context.
+
+**Fix shape** (the review's, made concrete):
+
+- **Bad-row quarantine, not fatality**: a stored spec that fails
+  re-parse is *quarantined* — log (`eprintln!`, the engine's
+  diagnostic posture) with the schedule name, advance the row's
+  `next_fire_at` past now (the tick's skip-forward helper — the row
+  must not spin), and continue with the remaining due rows. The tick
+  tx still commits (the quarantine advance is a normal write).
+  Re-validate-on-read at *registration* is unnecessary (specs are
+  pre-validated at `schedule()`); quarantine is only the tampered/
+  foreign-row arm.
+- **Transient-error retry**: a tick that fails with a pool/database
+  error retries N times (pin N — 3 is the natural floor) with a short
+  backoff before giving up and returning `Err` from the loop (the
+  consumer's respawn recipe remains the last resort, but a single
+  hiccup no longer kills the runner). Leadership renewal continues to
+  own the loss decision — a retry must not mask a lost lease (the
+  renew check at the loop top already runs before each tick).
+- **Error context**: the exiting error's source chain carries the
+  schedule name / tick phase (via the `database_error` message shape
+  — `Database` Display is opaque; the chain is the carrier).
+- **Exit-path leadership honesty**: on the `Err` exit path the
+  leadership lock is not released — the TTL lapse covers it. Say so
+  in the module docs (the review's fine print); opportunistically
+  releasing on the error exit is acceptable if it stays simple, but
+  the documented TTL-lapse posture is the bar.
+- **Doc-comment fix (Finding 6's scheduler bullet, same file)**:
+  `scheduler.rs:46-53` claims "a newly-registered schedule is noticed
+  by the re-read no later than the next slice (1 s)" — false: the 1 s
+  slices only renew the lease and check the stop token; the soonest
+  is never re-read mid-sleep. An idle runner notices a new schedule
+  up to `IDLE_SLEEP_S` (60 s) late; an active runner waits for the
+  current soonest deadline. Correct the comment to the honest posture
+  (the 60 s idle-tick floor is already acknowledged two paragraphs
+  down as acceptable — the doc comment above it contradicts it).
+  Implementing the per-slice soonest re-check is *optional* (a
+  behavior improvement, not required by this task) — if done, the
+  corrected comment describes it.
+
+Tests: a tampered spec row is quarantined (the runner survives, other
+schedules still fire, the bad row's boundary advanced — behavioral,
+against the harness server, tampering via a direct SQL UPDATE); the
+retry policy pinned (unit-level pin of the retry count/backoff shape
+is the bar; a behavioral transient-error injection is welcome if a
+deterministic arrangement exists); the doc-comment claim matches the
+code.
+
+## Acceptance Criteria
+
+- [ ] A tampered/foreign spec row is quarantined (logged with name,
+      boundary advanced past now, other schedules still fire, runner
+      survives) — pinned by behavioral test
+- [ ] A transient tick error is retried N times (pinned N) with
+      backoff before the loop returns `Err` — policy pinned by test
+- [ ] The exiting error's chain carries schedule-name/tick-phase
+      context
+- [ ] The error-exit leadership posture (TTL lapse covers the
+      unreleased lock) stated in the module docs
+- [ ] The `scheduler.rs:46-53` doc claim corrected to the honest
+      slice/idle posture
+- [ ] Existing scheduler tests stay green (clean stop,
+      `LeadershipLost`, catch-up cap, boundary math)
+- [ ] `cargo test -p alkstore-postgres` (harness server), clippy
+      `-D warnings`, fmt clean; gates green server-less
+
+## References
+
+- docs/reviews/002-wave-4-general-review.md §Finding 5 + §Finding 6
+  (the scheduler doc bullet)
+- docs/architecture/decisions/009-scheduler-boundaries.md (the
+  boundary guarantee the quarantine advance must preserve)
+- docs/architecture/decisions/010-queue-stamps-and-backoff.md §3
+- tasks/pg-engine-scheduler-outbox.md (the scheduler's task of record)
+- alkstore-sqlite/src/scheduler.rs (the structural twin — check its
+  error posture for parity notes)
+
+## Notes
+
+> To be filled by implementation agent
+
+## Summary
+
+> To be filled on completion