205 lines
11 KiB
Markdown
205 lines
11 KiB
Markdown
---
|
|
id: pg-fix-scheduler-fire-wake
|
|
name: "pg scheduler — fire wakes: the tick's in-tx enqueue fires issue their pg_notify wake (the recorded latency-parity gap, closed)"
|
|
status: completed
|
|
depends_on: []
|
|
scope: narrow
|
|
risk: low
|
|
impact: component
|
|
level: implementation
|
|
tags: [wave-5-aftermath, postgres-engine, scheduler, wake]
|
|
---
|
|
|
|
## Description
|
|
|
|
Close the recorded cross-engine scheduler fire-wake latency-parity gap —
|
|
the one item the wave-5 review gate left standing "for a later task"
|
|
(`tasks/review-wave-5.md` §Notes 4, audit verdict: "the recorded gap
|
|
stands for a later task; accepted as-is"). Provenance chain:
|
|
`tasks/review-wave-4-fixes.md` §"Recorded for later" (the original note:
|
|
"Scheduler *fire* wakes (the tick's in-tx enqueue) issue no `pg_notify`
|
|
— SQLite's watcher wakes on those commits… if wave 5's cross-engine
|
|
equivalence rows demand the parity, it is a one-call `wake_tx` inside
|
|
the tick's tx"), dispositioned not-demanded by
|
|
`tasks/suite-scheduler-rows.md` §Notes (the suite rows observe fires via
|
|
claim polling; wake latency out of scope), and audited as-is by the
|
|
wave-5 gate. Correctness was never at stake — the queues row's pinned
|
|
re-poll safety net covers it on both engines — but the asymmetry stands:
|
|
a schedule fire on pg is invisible to a registered listener until the
|
|
listener's re-poll, while on SQLite the watcher wakes that queue's
|
|
listeners at the tick's commit.
|
|
|
|
**Site**: `alkstore-postgres/src/scheduler.rs:518-548` — the tick's
|
|
one-firer-per-boundary loop calls `enqueue_row(client, …)` inside the
|
|
[`crate::queue::in_tx`] frame (ADR-009 §4's layer (b)) and issues no
|
|
wake. The fix is the recorded one-call shape, riding the
|
|
`pg-fix-tx-wake` pattern of record (`tasks/pg-fix-tx-wake.md`):
|
|
`tx::wake_tx` (currently private at `tx.rs:287` — widen to
|
|
`pub(crate)`) issued **inside the tick's `in_tx` frame**, so NOTIFY's
|
|
native transactional delivery makes the wake commit-atomic — a rolled-
|
|
back tick (crash mid-tick ⇒ the boundary refires) discards the wake
|
|
with its fire rows, exactly the tx/no-ghosts discipline the tx producer
|
|
paths already pin.
|
|
|
|
**Fix constraints** (pinned):
|
|
|
|
- Channel = the fired queue's name (the mechanism-name-is-the-channel
|
|
realization; `schedule()` rejects reserved names per ADR-021 §3, so a
|
|
fire's target can never be a derived outbox channel — the plain queue
|
|
name is always the right channel).
|
|
- Empty payload — `Wake { channel }` only (ADR-008 §3); best-effort
|
|
log-and-swallow posture (`wake_tx`'s existing stance; a wake must
|
|
never fail a tick — ADR-006).
|
|
- **Coalescing default: one wake per tick per schedule that fired ≥ 1**
|
|
(after the fire loop, still inside the `in_tx` body), not per-fire —
|
|
wakes are hint-only (ADR-006; no per-wake semantics exist to
|
|
preserve), and this matches the SQLite twin's shape (the substrate
|
|
watcher wakes per committed tick transaction, not per enqueued row).
|
|
Deviate to per-fire only with the reason recorded in Notes.
|
|
- No double-wake risk: the wake is applied at the tick call site, NOT
|
|
inside the shared `enqueue_row` (the `pg-fix-tx-wake` one-owner
|
|
posture — `enqueue_row` stays wake-free; the auto-commit
|
|
`Queue::enqueue` and the tx paths keep their own single wakes,
|
|
untouched).
|
|
|
|
**Test shape** (engine-side, pg): a harness test in the scheduler tests
|
|
— a schedule registered against a queue, a listener registered on that
|
|
queue's channel (`forwarder().register`; `listener_hears`), the runner
|
|
ticks with the schedule due: the fire wake arrives at/after the tick's
|
|
commit (pre-commit silence window to pin commit-atomicity, the
|
|
`tx_producer_wakes_are_commit_atomic_and_rollback_discards_them`
|
|
pattern); a tick with nothing due issues no wake. The rollback arm is
|
|
native (NOTIFY's transactional delivery — already proven by `notify_tx`
|
|
and the tx-wake test); do not build a tick-fault seam for it here
|
|
(`pg-fix-scheduler-tick-seam`'s seam, if landed first, is the retry
|
|
loop's, not this site's — same file, disjoint site, either landing
|
|
order is fine).
|
|
|
|
**Record updates** (the gap is recorded in three places — retire all
|
|
three): `tasks/review-wave-4-fixes.md`'s "Recorded for later" bullet
|
|
(marked retired by this fix), `tasks/suite-scheduler-rows.md` §Notes'
|
|
fire-wake-parity disposition (gap closed, pointer), and
|
|
`tasks/review-wave-5.md` §Notes 4's audit-table row (pointer to the
|
|
closing task). Add the review-rounds line to
|
|
`docs/plans/implementation.md`.
|
|
|
|
**Ordering note**: no hard dependency on wave 6, but landing before
|
|
`mem-engine-scheduler-outbox` keeps the mem engine's fire posture
|
|
written against a closed gap (mem's in-memory wake routing is trivially
|
|
immediate; the suite's three-way story then starts symmetric).
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] The tick's fire path issues its wake commit-atomically inside the
|
|
`in_tx` frame (channel = the fired queue's name, empty payload,
|
|
best-effort `wake_tx` posture); the coalescing decision recorded
|
|
in Notes (default: one wake per firing schedule per tick)
|
|
- [x] Pinned by test: a listener on the target queue's channel receives
|
|
the wake at/after the tick's commit (pre-commit silence window);
|
|
a tick with nothing due issues no wake
|
|
- [x] No wake regression: auto-commit `Queue::enqueue`, the tx producer
|
|
paths, and `enqueue_row` itself unchanged (no double-wake)
|
|
- [x] The three gap records retired with pointers
|
|
(`review-wave-4-fixes`, `suite-scheduler-rows` Notes,
|
|
`review-wave-5` §Notes 4) + the `implementation.md` review-rounds
|
|
line
|
|
- [x] `cargo test -p alkstore-postgres` green against the harness
|
|
server; skips clean server-less; clippy `-D warnings`; fmt clean;
|
|
workspace gates green
|
|
|
|
## References
|
|
|
|
- tasks/review-wave-4-fixes.md §"Recorded for later" (the original
|
|
recorded note, incl. the one-call shape)
|
|
- tasks/suite-scheduler-rows.md §Notes (the not-demanded disposition)
|
|
- tasks/review-wave-5.md §Notes 4 (the gate's audit row)
|
|
- tasks/pg-fix-tx-wake.md (the wake_tx pattern + commit-atomicity
|
|
precedent; `tx.rs:287`)
|
|
- docs/architecture/decisions/006-wake-and-delivery-contract.md (wakes
|
|
best-effort, hint-only)
|
|
- docs/architecture/decisions/009-scheduler-collapse.md §4 (the tick's
|
|
tx frame)
|
|
- docs/architecture/engine-postgres.md (pg_notify as the wake trigger)
|
|
|
|
## Notes
|
|
|
|
Fix-shape decisions of record (what the description left open):
|
|
|
|
- **Coalescing: the default** — one wake per tick per schedule that
|
|
fired ≥ 1, issued after the fire loop (before the boundary-advance
|
|
UPDATE), inside the caller's `in_tx` frame. Not per-fire: wakes are
|
|
hint-only (ADR-006, no per-wake semantics exist to preserve), and
|
|
the SQLite substrate's watcher wakes per committed tick transaction,
|
|
not per enqueued row — the parity shape is the committed tick, not
|
|
the individual fire. The `for row in &due` loop naturally yields one
|
|
wake per firing schedule (multi-schedule ticks wake each schedule's
|
|
own channel once). A quarantined row (`continue`s before the fire
|
|
loop) fired nothing and wakes nothing. No per-fire deviation.
|
|
- **Wake placement & ownership**: the wake is issued at the tick call
|
|
site in `scheduler.rs::tick` via the shared `tx::wake_tx` (widened
|
|
private → `pub(crate)` exactly as the task pinned); `enqueue_row`
|
|
stays wake-free (no double-wake), the auto-commit `Queue::enqueue`
|
|
and the tx producer paths untouched. Channel = the fired queue's
|
|
name; empty payload (`Wake { channel }`, ADR-008 §3); best-effort —
|
|
a swallowed wake failure aborts the tx server-side so the truth
|
|
surfaces typed at COMMIT (and the tick rolls back with its fires).
|
|
- **No wake issued for a due row whose fire count is zero-guarded** —
|
|
no guard added: a non-quarantined due row fires ≥ 1 by construction
|
|
(the `while next_fire_at <= now` loop runs at least once), so the
|
|
unconditional post-loop wake is exactly "per schedule that fired
|
|
≥ 1".
|
|
- **Test shape**: one harness test in `scheduler_tests.rs`
|
|
(`scheduler_fire_wakes_are_commit_atomic_coalesced_and_nothing_due_is_silent`),
|
|
four arms: (a) pre-commit silence — the tick driven directly through
|
|
the runner's own `in_tx` frame (the `rogue_concurrent_ticks` shape;
|
|
the silence window needs the tick body observable before its COMMIT,
|
|
which the runner's loop holds internally), listener silent
|
|
statement-side, hears at/after commit; (b) coalescing — backdated
|
|
one interval ⇒ ≥ 2 boundaries fire in the one tick, exactly one wake
|
|
(not per fire); (c) nothing-due silence — a same-tick not-due
|
|
schedule and a later all-not-due tick wake nobody (in-frame and
|
|
post-commit); (d) end-to-end runner leg — a live `run_schedules`
|
|
leader's fire wake reaches a registered listener. The direct-tick
|
|
drive is the runner's exact frame (`tick_once` = `in_tx` + `tick`);
|
|
the runner loop itself is separately exercised by the existing fire
|
|
tests and by arm (d). A local `listener_hears` helper copy rides the
|
|
per-file helper convention (`tx_tests.rs` keeps its own). The
|
|
rollback arm was not test-built here (native — NOTIFY's transactional
|
|
delivery, proven by `notify_tx` and the tx-wake test; the task
|
|
pinned no tick-fault seam for it).
|
|
- **Doc updates rode along**: `tx.rs`'s "The tx wakes" module section
|
|
and `wake_tx`'s doc now name the scheduler fire path as a shared
|
|
caller; `scheduler.rs`'s module docs and `tick`'s doc pin the
|
|
coalesced commit-atomic fire-wake semantics (the contract-relevant
|
|
text).
|
|
- **Flake-ledger occurrence (unrelated to this task's mechanism)**:
|
|
during the first full pg suite run this session,
|
|
`row_lock_ttl_expiry_and_reacquisition` (contract suite) failed
|
|
once — the wave-5 ledger's second recorded occurrence. Green on the
|
|
next two full runs plus six focused runs. Recorded in
|
|
`tasks/review-wave-5.md` §6 with the ledger's own conclusion: the
|
|
"fails again" trigger is met, a dedicated investigative session is
|
|
owed by the wave-7 watch carriage before any postulate-and-fix. Not
|
|
load-bearing here (locks, not scheduler/wakes; scheduler code
|
|
untouched by that row).
|
|
|
|
## Summary
|
|
|
|
**Landed**: the scheduler tick's commit-atomic fire wake —
|
|
`scheduler.rs::tick` now issues one coalesced `pg_notify` per firing
|
|
schedule per tick (channel = the fired queue's name, empty payload)
|
|
inside the tick's `in_tx` frame via the shared `tx::wake_tx` (widened
|
|
to `pub(crate)`); the new four-arm pinning test; the driver docs
|
|
(`tx.rs` wake section, `scheduler.rs` module + `tick` docs); and the
|
|
three gap records retired with pointers (`review-wave-4-fixes`'s
|
|
recorded-for-later note, `suite-scheduler-rows`'s gate disposition
|
|
note + Notes bullet, `review-wave-5` §Notes 4's audit row) plus the
|
|
`implementation.md` review-rounds line — the three-way (SQLite / pg /
|
|
mem-to-be) fire-wake story is symmetric ahead of wave 6.
|
|
|
|
**Verified**: full pg lib suite 122/122 green against the harness
|
|
server (the new test deterministically green across three solo runs);
|
|
pg contract suite 25/25 and schema 9/9 green (one unrelated single-run
|
|
flake noted above, ledger-carried); workspace `cargo test` green
|
|
server-less (pg skips clean, incl. the new test); `cargo clippy
|
|
--all-targets -- -D warnings` clean; `cargo fmt --check` clean. |