alkstore/tasks/pg-fix-scheduler-resilience.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
pg-fix-scheduler-resilience	Scheduler runner resilience — quarantine bad rows, retry transient errors (review 002 Finding 5)	pending		narrow	medium	component	implementation	wave-4-fixes postgres-engine scheduler

Description

Fix the LOW/MEDIUM finding from the wave-4 general review (docs/reviews/002-wave-4-general-review.md §Finding 5): one bad row or one transient error permanently kills run_schedules. Sites: alkstore-postgres/src/scheduler.rs:403 (parse_every_interval(&row.spec)?), :429 (fire enqueue_row(...)?), :519-526 (the in_tx tick ?) — all propagating straight out of the leader loop. The @every grammar is re-parsed from stored text per tick (unvalidated-on-read), so a tampered or future-foreign spec row makes the whole runner exit with InvalidSpec; one transient pool/database error does the same. The consumer's respawn recipe makes this survivable but brittle — schedules never fire without a runner, and the runner dies on any hiccup rather than skipping/retrying. The exiting error also carries no schedule-name context.

Fix shape (the review's, made concrete):

• Bad-row quarantine, not fatality: a stored spec that fails re-parse is quarantined — log (eprintln!, the engine's diagnostic posture) with the schedule name, advance the row's next_fire_at past now (the tick's skip-forward helper — the row must not spin), and continue with the remaining due rows. The tick tx still commits (the quarantine advance is a normal write). Re-validate-on-read at registration is unnecessary (specs are pre-validated at schedule()); quarantine is only the tampered/ foreign-row arm.
• Transient-error retry: a tick that fails with a pool/database error retries N times (pin N — 3 is the natural floor) with a short backoff before giving up and returning Err from the loop (the consumer's respawn recipe remains the last resort, but a single hiccup no longer kills the runner). Leadership renewal continues to own the loss decision — a retry must not mask a lost lease (the renew check at the loop top already runs before each tick).
• Error context: the exiting error's source chain carries the schedule name / tick phase (via the database_error message shape — Database Display is opaque; the chain is the carrier).
• Exit-path leadership honesty: on the Err exit path the leadership lock is not released — the TTL lapse covers it. Say so in the module docs (the review's fine print); opportunistically releasing on the error exit is acceptable if it stays simple, but the documented TTL-lapse posture is the bar.
• Doc-comment fix (Finding 6's scheduler bullet, same file): scheduler.rs:46-53 claims "a newly-registered schedule is noticed by the re-read no later than the next slice (1 s)" — false: the 1 s slices only renew the lease and check the stop token; the soonest is never re-read mid-sleep. An idle runner notices a new schedule up to IDLE_SLEEP_S (60 s) late; an active runner waits for the current soonest deadline. Correct the comment to the honest posture (the 60 s idle-tick floor is already acknowledged two paragraphs down as acceptable — the doc comment above it contradicts it). Implementing the per-slice soonest re-check is optional (a behavior improvement, not required by this task) — if done, the corrected comment describes it.

Tests: a tampered spec row is quarantined (the runner survives, other schedules still fire, the bad row's boundary advanced — behavioral, against the harness server, tampering via a direct SQL UPDATE); the retry policy pinned (unit-level pin of the retry count/backoff shape is the bar; a behavioral transient-error injection is welcome if a deterministic arrangement exists); the doc-comment claim matches the code.

Acceptance Criteria

• A tampered/foreign spec row is quarantined (logged with name, boundary advanced past now, other schedules still fire, runner survives) — pinned by behavioral test
• A transient tick error is retried N times (pinned N) with backoff before the loop returns Err — policy pinned by test
• The exiting error's chain carries schedule-name/tick-phase context
• The error-exit leadership posture (TTL lapse covers the unreleased lock) stated in the module docs
• The scheduler.rs:46-53 doc claim corrected to the honest slice/idle posture
• Existing scheduler tests stay green (clean stop, LeadershipLost, catch-up cap, boundary math)
• cargo test -p alkstore-postgres (harness server), clippy -D warnings, fmt clean; gates green server-less

References

• docs/reviews/002-wave-4-general-review.md §Finding 5 + §Finding 6 (the scheduler doc bullet)
• docs/architecture/decisions/009-scheduler-boundaries.md (the boundary guarantee the quarantine advance must preserve)
• docs/architecture/decisions/010-queue-stamps-and-backoff.md §3
• tasks/pg-engine-scheduler-outbox.md (the scheduler's task of record)
• alkstore-sqlite/src/scheduler.rs (the structural twin — check its error posture for parity notes)

Notes

To be filled by implementation agent

Summary

To be filled on completion