pg-fix-scheduler-fire-wake — the scheduler tick's fire wake lands, closing the recorded cross-engine fire-wake latency-parity gap (the wave-5 gate's one standing item, review-wave-4-fixes' recorded-for-later note): scheduler.rs::tick now issues one coalesced pg_notify per firing schedule per tick after the fire loop, still inside the tick's in_tx frame — channel = the fired queue's name (schedule() rejects reserved names per ADR-021 §3, so the plain queue name is always the right channel), empty payload (Wake { channel } only, ADR-008 §3), best-effort via the shared tx::wake_tx (widened private → pub(crate) per the recorded one-call shape; enqueue_row stays wake-free — no double-wake; auto-commit Queue::enqueue and the tx producer paths untouched). NOTIFY's native transactional delivery makes the wake commit-atomic: a rolled-back tick (crash mid-tick ⇒ boundary refires) discards the wake with its fire rows — the tx/no-ghosts discipline the tx producer paths already pin; coalescing default one-per-firing-schedule-per-tick (not per-fire) matching the SQLite watcher's per-committed-tick cadence, decision recorded in the task's Notes. New four-arm pinning test (scheduler_tests.rs, tx_producer_wakes_are_commit_atomic pattern): pre-commit silence inside the runner's own in_tx frame driving the engine's own tick (rogue-ticks shape), deterministic delivery at/after commit, coalescing (≥2 boundaries → one wake), nothing-due silence (same-tick not-due schedule + a later all-not-due tick, in-frame and post-commit), and the end-to-end runner leg (a live run_schedules leader's wake reaches a registered listener); local listener_hears helper per the per-file helper convention; rollback arm native (NOTIFY transactional delivery — no tick-fault seam built, per the task pin). Docs: tx.rs 'The tx wakes' section + wake_tx doc name the scheduler fire path as a shared caller; scheduler.rs module docs + tick doc pin the coalesced commit-atomic semantics. Record updates: review-wave-4-fixes' recorded-for-later bullet, suite-scheduler-rows' gate disposition note + Notes bullet, review-wave-5 §Notes 4's audit row — all retired with pointers; implementation.md gains the review-rounds line (fires visible to registered listeners ahead of the mem engine's posture being written). Also carried: review-wave-5 §6 flake-ledger update — row_lock_ttl_expiry_and_reacquisition failed once more in a full pg run this session (green on the next two full runs + six focused; unrelated mechanism to this change), meeting the ledger's own 'fails again' trigger with the dedicated investigative session owed by the wave-7 watch carriage. Verified: full pg lib suite 122/122 vs the harness (new test green ×3 solo); pg contract suite 25/25 + schema 9/9; workspace cargo test green server-less (pg skips clean incl. the new test); clippy --all-targets -D warnings clean; fmt clean
This commit is contained in:
1 parent
5abd351c86
commit
57451cef46
8 files changed
+538
-14
No files matched your search
@@ -0,0 +1,205 @@
|
||||
---
|
||||
id: pg-fix-scheduler-fire-wake
|
||||
name: "pg scheduler — fire wakes: the tick's in-tx enqueue fires issue their pg_notify wake (the recorded latency-parity gap, closed)"
|
||||
status: completed
|
||||
depends_on: []
|
||||
scope: narrow
|
||||
risk: low
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-5-aftermath, postgres-engine, scheduler, wake]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Close the recorded cross-engine scheduler fire-wake latency-parity gap —
|
||||
the one item the wave-5 review gate left standing "for a later task"
|
||||
(`tasks/review-wave-5.md` §Notes 4, audit verdict: "the recorded gap
|
||||
stands for a later task; accepted as-is"). Provenance chain:
|
||||
`tasks/review-wave-4-fixes.md` §"Recorded for later" (the original note:
|
||||
"Scheduler *fire* wakes (the tick's in-tx enqueue) issue no `pg_notify`
|
||||
— SQLite's watcher wakes on those commits… if wave 5's cross-engine
|
||||
equivalence rows demand the parity, it is a one-call `wake_tx` inside
|
||||
the tick's tx"), dispositioned not-demanded by
|
||||
`tasks/suite-scheduler-rows.md` §Notes (the suite rows observe fires via
|
||||
claim polling; wake latency out of scope), and audited as-is by the
|
||||
wave-5 gate. Correctness was never at stake — the queues row's pinned
|
||||
re-poll safety net covers it on both engines — but the asymmetry stands:
|
||||
a schedule fire on pg is invisible to a registered listener until the
|
||||
listener's re-poll, while on SQLite the watcher wakes that queue's
|
||||
listeners at the tick's commit.
|
||||
|
||||
**Site**: `alkstore-postgres/src/scheduler.rs:518-548` — the tick's
|
||||
one-firer-per-boundary loop calls `enqueue_row(client, …)` inside the
|
||||
[`crate::queue::in_tx`] frame (ADR-009 §4's layer (b)) and issues no
|
||||
wake. The fix is the recorded one-call shape, riding the
|
||||
`pg-fix-tx-wake` pattern of record (`tasks/pg-fix-tx-wake.md`):
|
||||
`tx::wake_tx` (currently private at `tx.rs:287` — widen to
|
||||
`pub(crate)`) issued **inside the tick's `in_tx` frame**, so NOTIFY's
|
||||
native transactional delivery makes the wake commit-atomic — a rolled-
|
||||
back tick (crash mid-tick ⇒ the boundary refires) discards the wake
|
||||
with its fire rows, exactly the tx/no-ghosts discipline the tx producer
|
||||
paths already pin.
|
||||
|
||||
**Fix constraints** (pinned):
|
||||
|
||||
- Channel = the fired queue's name (the mechanism-name-is-the-channel
|
||||
realization; `schedule()` rejects reserved names per ADR-021 §3, so a
|
||||
fire's target can never be a derived outbox channel — the plain queue
|
||||
name is always the right channel).
|
||||
- Empty payload — `Wake { channel }` only (ADR-008 §3); best-effort
|
||||
log-and-swallow posture (`wake_tx`'s existing stance; a wake must
|
||||
never fail a tick — ADR-006).
|
||||
- **Coalescing default: one wake per tick per schedule that fired ≥ 1**
|
||||
(after the fire loop, still inside the `in_tx` body), not per-fire —
|
||||
wakes are hint-only (ADR-006; no per-wake semantics exist to
|
||||
preserve), and this matches the SQLite twin's shape (the substrate
|
||||
watcher wakes per committed tick transaction, not per enqueued row).
|
||||
Deviate to per-fire only with the reason recorded in Notes.
|
||||
- No double-wake risk: the wake is applied at the tick call site, NOT
|
||||
inside the shared `enqueue_row` (the `pg-fix-tx-wake` one-owner
|
||||
posture — `enqueue_row` stays wake-free; the auto-commit
|
||||
`Queue::enqueue` and the tx paths keep their own single wakes,
|
||||
untouched).
|
||||
|
||||
**Test shape** (engine-side, pg): a harness test in the scheduler tests
|
||||
— a schedule registered against a queue, a listener registered on that
|
||||
queue's channel (`forwarder().register`; `listener_hears`), the runner
|
||||
ticks with the schedule due: the fire wake arrives at/after the tick's
|
||||
commit (pre-commit silence window to pin commit-atomicity, the
|
||||
`tx_producer_wakes_are_commit_atomic_and_rollback_discards_them`
|
||||
pattern); a tick with nothing due issues no wake. The rollback arm is
|
||||
native (NOTIFY's transactional delivery — already proven by `notify_tx`
|
||||
and the tx-wake test); do not build a tick-fault seam for it here
|
||||
(`pg-fix-scheduler-tick-seam`'s seam, if landed first, is the retry
|
||||
loop's, not this site's — same file, disjoint site, either landing
|
||||
order is fine).
|
||||
|
||||
**Record updates** (the gap is recorded in three places — retire all
|
||||
three): `tasks/review-wave-4-fixes.md`'s "Recorded for later" bullet
|
||||
(marked retired by this fix), `tasks/suite-scheduler-rows.md` §Notes'
|
||||
fire-wake-parity disposition (gap closed, pointer), and
|
||||
`tasks/review-wave-5.md` §Notes 4's audit-table row (pointer to the
|
||||
closing task). Add the review-rounds line to
|
||||
`docs/plans/implementation.md`.
|
||||
|
||||
**Ordering note**: no hard dependency on wave 6, but landing before
|
||||
`mem-engine-scheduler-outbox` keeps the mem engine's fire posture
|
||||
written against a closed gap (mem's in-memory wake routing is trivially
|
||||
immediate; the suite's three-way story then starts symmetric).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] The tick's fire path issues its wake commit-atomically inside the
|
||||
`in_tx` frame (channel = the fired queue's name, empty payload,
|
||||
best-effort `wake_tx` posture); the coalescing decision recorded
|
||||
in Notes (default: one wake per firing schedule per tick)
|
||||
- [x] Pinned by test: a listener on the target queue's channel receives
|
||||
the wake at/after the tick's commit (pre-commit silence window);
|
||||
a tick with nothing due issues no wake
|
||||
- [x] No wake regression: auto-commit `Queue::enqueue`, the tx producer
|
||||
paths, and `enqueue_row` itself unchanged (no double-wake)
|
||||
- [x] The three gap records retired with pointers
|
||||
(`review-wave-4-fixes`, `suite-scheduler-rows` Notes,
|
||||
`review-wave-5` §Notes 4) + the `implementation.md` review-rounds
|
||||
line
|
||||
- [x] `cargo test -p alkstore-postgres` green against the harness
|
||||
server; skips clean server-less; clippy `-D warnings`; fmt clean;
|
||||
workspace gates green
|
||||
|
||||
## References
|
||||
|
||||
- tasks/review-wave-4-fixes.md §"Recorded for later" (the original
|
||||
recorded note, incl. the one-call shape)
|
||||
- tasks/suite-scheduler-rows.md §Notes (the not-demanded disposition)
|
||||
- tasks/review-wave-5.md §Notes 4 (the gate's audit row)
|
||||
- tasks/pg-fix-tx-wake.md (the wake_tx pattern + commit-atomicity
|
||||
precedent; `tx.rs:287`)
|
||||
- docs/architecture/decisions/006-wake-and-delivery-contract.md (wakes
|
||||
best-effort, hint-only)
|
||||
- docs/architecture/decisions/009-scheduler-collapse.md §4 (the tick's
|
||||
tx frame)
|
||||
- docs/architecture/engine-postgres.md (pg_notify as the wake trigger)
|
||||
|
||||
## Notes
|
||||
|
||||
Fix-shape decisions of record (what the description left open):
|
||||
|
||||
- **Coalescing: the default** — one wake per tick per schedule that
|
||||
fired ≥ 1, issued after the fire loop (before the boundary-advance
|
||||
UPDATE), inside the caller's `in_tx` frame. Not per-fire: wakes are
|
||||
hint-only (ADR-006, no per-wake semantics exist to preserve), and
|
||||
the SQLite substrate's watcher wakes per committed tick transaction,
|
||||
not per enqueued row — the parity shape is the committed tick, not
|
||||
the individual fire. The `for row in &due` loop naturally yields one
|
||||
wake per firing schedule (multi-schedule ticks wake each schedule's
|
||||
own channel once). A quarantined row (`continue`s before the fire
|
||||
loop) fired nothing and wakes nothing. No per-fire deviation.
|
||||
- **Wake placement & ownership**: the wake is issued at the tick call
|
||||
site in `scheduler.rs::tick` via the shared `tx::wake_tx` (widened
|
||||
private → `pub(crate)` exactly as the task pinned); `enqueue_row`
|
||||
stays wake-free (no double-wake), the auto-commit `Queue::enqueue`
|
||||
and the tx producer paths untouched. Channel = the fired queue's
|
||||
name; empty payload (`Wake { channel }`, ADR-008 §3); best-effort —
|
||||
a swallowed wake failure aborts the tx server-side so the truth
|
||||
surfaces typed at COMMIT (and the tick rolls back with its fires).
|
||||
- **No wake issued for a due row whose fire count is zero-guarded** —
|
||||
no guard added: a non-quarantined due row fires ≥ 1 by construction
|
||||
(the `while next_fire_at <= now` loop runs at least once), so the
|
||||
unconditional post-loop wake is exactly "per schedule that fired
|
||||
≥ 1".
|
||||
- **Test shape**: one harness test in `scheduler_tests.rs`
|
||||
(`scheduler_fire_wakes_are_commit_atomic_coalesced_and_nothing_due_is_silent`),
|
||||
four arms: (a) pre-commit silence — the tick driven directly through
|
||||
the runner's own `in_tx` frame (the `rogue_concurrent_ticks` shape;
|
||||
the silence window needs the tick body observable before its COMMIT,
|
||||
which the runner's loop holds internally), listener silent
|
||||
statement-side, hears at/after commit; (b) coalescing — backdated
|
||||
one interval ⇒ ≥ 2 boundaries fire in the one tick, exactly one wake
|
||||
(not per fire); (c) nothing-due silence — a same-tick not-due
|
||||
schedule and a later all-not-due tick wake nobody (in-frame and
|
||||
post-commit); (d) end-to-end runner leg — a live `run_schedules`
|
||||
leader's fire wake reaches a registered listener. The direct-tick
|
||||
drive is the runner's exact frame (`tick_once` = `in_tx` + `tick`);
|
||||
the runner loop itself is separately exercised by the existing fire
|
||||
tests and by arm (d). A local `listener_hears` helper copy rides the
|
||||
per-file helper convention (`tx_tests.rs` keeps its own). The
|
||||
rollback arm was not test-built here (native — NOTIFY's transactional
|
||||
delivery, proven by `notify_tx` and the tx-wake test; the task
|
||||
pinned no tick-fault seam for it).
|
||||
- **Doc updates rode along**: `tx.rs`'s "The tx wakes" module section
|
||||
and `wake_tx`'s doc now name the scheduler fire path as a shared
|
||||
caller; `scheduler.rs`'s module docs and `tick`'s doc pin the
|
||||
coalesced commit-atomic fire-wake semantics (the contract-relevant
|
||||
text).
|
||||
- **Flake-ledger occurrence (unrelated to this task's mechanism)**:
|
||||
during the first full pg suite run this session,
|
||||
`row_lock_ttl_expiry_and_reacquisition` (contract suite) failed
|
||||
once — the wave-5 ledger's second recorded occurrence. Green on the
|
||||
next two full runs plus six focused runs. Recorded in
|
||||
`tasks/review-wave-5.md` §6 with the ledger's own conclusion: the
|
||||
"fails again" trigger is met, a dedicated investigative session is
|
||||
owed by the wave-7 watch carriage before any postulate-and-fix. Not
|
||||
load-bearing here (locks, not scheduler/wakes; scheduler code
|
||||
untouched by that row).
|
||||
|
||||
## Summary
|
||||
|
||||
**Landed**: the scheduler tick's commit-atomic fire wake —
|
||||
`scheduler.rs::tick` now issues one coalesced `pg_notify` per firing
|
||||
schedule per tick (channel = the fired queue's name, empty payload)
|
||||
inside the tick's `in_tx` frame via the shared `tx::wake_tx` (widened
|
||||
to `pub(crate)`); the new four-arm pinning test; the driver docs
|
||||
(`tx.rs` wake section, `scheduler.rs` module + `tick` docs); and the
|
||||
three gap records retired with pointers (`review-wave-4-fixes`'s
|
||||
recorded-for-later note, `suite-scheduler-rows`'s gate disposition
|
||||
note + Notes bullet, `review-wave-5` §Notes 4's audit row) plus the
|
||||
`implementation.md` review-rounds line — the three-way (SQLite / pg /
|
||||
mem-to-be) fire-wake story is symmetric ahead of wave 6.
|
||||
|
||||
**Verified**: full pg lib suite 122/122 green against the harness
|
||||
server (the new test deterministically green across three solo runs);
|
||||
pg contract suite 25/25 and schema 9/9 green (one unrelated single-run
|
||||
flake noted above, ledger-carried); workspace `cargo test` green
|
||||
server-less (pg skips clean, incl. the new test); `cargo clippy
|
||||
--all-targets -- -D warnings` clean; `cargo fmt --check` clean.
|
||||
@@ -216,7 +216,12 @@ the truthful "no other path targets it" claim.
|
||||
twins; correctness is covered by the queues row's pinned re-poll
|
||||
safety net. Not in review 002's fix shape; if wave 5's
|
||||
cross-engine equivalence rows demand the parity, it is a one-call
|
||||
`wake_tx` inside the tick's tx.
|
||||
`wake_tx` inside the tick's tx. **[Retired 2026-10-10]** Closed by
|
||||
`tasks/pg-fix-scheduler-fire-wake.md` — the tick now issues the
|
||||
one-call `wake_tx` inside its own `in_tx` frame (one coalesced wake
|
||||
per firing schedule per tick, commit-atomic; the disposition note
|
||||
in `tasks/suite-scheduler-rows.md` and the wave-5 gate's audit row
|
||||
carry the same retirement pointer).
|
||||
|
||||
## Summary
|
||||
|
||||
|
||||
+14
-1
@@ -174,7 +174,7 @@ Focused stress: 6 consecutive `--test-threads=6` runs of
|
||||
| M-1 retention-failure pin's location | `suite-queue-depth-rows` Notes | Engine-side (SQLite trigger seam; pg by frame structure) — sound; the pg leg rests on code-read, accepted with the row's doc text recording it |
|
||||
| Beyond-64 skip-forward leg | `suite-scheduler-rows` Notes | Engine-side twins exist on both engines (`catch_up_replays_up_to_the_cap_then_skips_forward`); suite pins the ≤64 in-band leg; row doc states it |
|
||||
| Backoff cap leg | `suite-opts-backoff-rows` Notes | Engine-side pins verified present (both engines' unit tests); row doc records the disposition |
|
||||
| Scheduler fire-wake parity | `suite-scheduler-rows` Notes | Not demanded by the row shapes (claim-polling observation only); the recorded gap stands for a later task; accepted as-is |
|
||||
| Scheduler fire-wake parity | `suite-scheduler-rows` Notes | Not demanded by the row shapes (claim-polling observation only); the recorded gap stands for a later task; accepted as-is — **closed post-gate** (2026-10-10) by `pg-fix-scheduler-fire-wake` (the tick's commit-atomic fire wake; the row's Notes and `review-wave-4-fixes` carry the retirement pointers) |
|
||||
| Lock-row divergence call | `suite-lock-rows` Notes | No divergence found — the row pins the convergence; nothing to fix |
|
||||
| SQLite reconnect-success test | `sqlite-commit-error-arm` Notes | Taken — `reconnect_success_resumes_wake_delivery` exists and pins the success arm's re-baseline + restored delivery |
|
||||
|
||||
@@ -219,6 +219,19 @@ clean pg runs across both rows since the failures, all conditions
|
||||
covered. **Flagged for watch in the next wave window** (release
|
||||
readiness at gate time — since renumbered to wave 7, ADR-024).
|
||||
|
||||
*Ledger update (2026-10-10, from the `pg-fix-scheduler-fire-wake`
|
||||
session):* `row_lock_ttl_expiry_and_reacquisition` failed once more —
|
||||
a single occurrence in one full pg suite run (`cargo test
|
||||
-p alkstore-postgres`, sequential, against the harness; the row's
|
||||
failure message was lost to the run's output filtering), then green on
|
||||
two subsequent full runs and six consecutive focused `row_lock_ttl`
|
||||
runs. Unrelated mechanism to that task's change (scheduler fire wake;
|
||||
locks untouched). Per this ledger's own prescription the row now
|
||||
meets the "fails again" trigger — a dedicated investigative session
|
||||
(instrumentation + focused repro loop) is owed by the wave-7 watch
|
||||
carriage before any postulate-and-fix. Recorded here for the ledger's
|
||||
honesty; not blocking the fire-wake task.
|
||||
|
||||
**7. Findings.** No code, suite-row, or stamp defects requiring change
|
||||
were found at gate time; all changes this gate made are doc-side
|
||||
(engine specs `draft` → `stable` with dated gate annotations;
|
||||
|
||||
@@ -59,7 +59,11 @@ defect): the pg tick's in-tx enqueue fires issue no `pg_notify` wake
|
||||
wake-latency-parity gap the re-poll safety net covers for correctness.
|
||||
If this row's shape demands the parity, the one-call `wake_tx` inside
|
||||
the tick's tx rides **this** task; otherwise record the row's
|
||||
latency-posture in Notes and leave the engines as they are.
|
||||
latency-posture in Notes and leave the engines as they are. **[Retired
|
||||
2026-10-10]** The parity was not demanded here, but the gap itself is
|
||||
now closed: `tasks/pg-fix-scheduler-fire-wake.md` landed the tick's
|
||||
commit-atomic fire wake (one coalesced wake per firing schedule per
|
||||
tick) — see the updated Notes bullet below.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
@@ -102,14 +106,20 @@ latency-posture in Notes and leave the engines as they are.
|
||||
- **Fire-wake parity: not demanded by this row set; engines left
|
||||
as-is.** The suite rows observe fire outcomes through ordinary
|
||||
claim polling, never through wake latency, so the row shapes demand
|
||||
no wake. The recorded cross-engine latency-parity gap stands (the pg
|
||||
no wake. The recorded cross-engine latency-parity gap stood (the pg
|
||||
tick's in-tx enqueue fires no `pg_notify`; SQLite's watcher wakes on
|
||||
those commits — `tasks/review-wave-4-fixes.md`'s recorded-for-later
|
||||
note): correctness is covered by the queues row's pinned re-poll
|
||||
safety net on both engines; the row set here pins state outcomes
|
||||
with tolerances bounded well above the boundary resolution, so wake
|
||||
latency is out of scope. If a later task demands the parity, the
|
||||
one-call `wake_tx` inside the pg tick's tx is the shape to ride.
|
||||
latency is out of scope. **[Retired 2026-10-10]** The gap itself is
|
||||
closed by `tasks/pg-fix-scheduler-fire-wake.md`: the pg tick now
|
||||
issues its commit-atomic `pg_notify` fire wake inside the tick's tx
|
||||
(one coalesced wake per firing schedule per tick, channel = the
|
||||
fired queue's name, best-effort per ADR-006), so both engines wake
|
||||
on a firing tick's commit and the three-way suite story starts
|
||||
symmetric. The row shapes here stay as landed — claim-polling
|
||||
observation, wake latency out of scope — with no re-shape owed.
|
||||
- **Runner-driving posture details** (the extension's riders, beyond
|
||||
what the description pinned): the spawned runners share the row's
|
||||
single store handle via `Arc<dyn Store>` clones — leadership is
|
||||
|
||||
Reference in new issue
Block a user