pg-fix-scheduler-fire-wake — the scheduler tick's fire wake lands, closing the recorded cross-engine fire-wake latency-parity gap (the wave-5 gate's one standing item, review-wave-4-fixes' recorded-for-later note): scheduler.rs::tick now issues one coalesced pg_notify per firing schedule per tick after the fire loop, still inside the tick's in_tx frame — channel = the fired queue's name (schedule() rejects reserved names per ADR-021 §3, so the plain queue name is always the right channel), empty payload (Wake { channel } only, ADR-008 §3), best-effort via the shared tx::wake_tx (widened private → pub(crate) per the recorded one-call shape; enqueue_row stays wake-free — no double-wake; auto-commit Queue::enqueue and the tx producer paths untouched). NOTIFY's native transactional delivery makes the wake commit-atomic: a rolled-back tick (crash mid-tick ⇒ boundary refires) discards the wake with its fire rows — the tx/no-ghosts discipline the tx producer paths already pin; coalescing default one-per-firing-schedule-per-tick (not per-fire) matching the SQLite watcher's per-committed-tick cadence, decision recorded in the task's Notes. New four-arm pinning test (scheduler_tests.rs, tx_producer_wakes_are_commit_atomic pattern): pre-commit silence inside the runner's own in_tx frame driving the engine's own tick (rogue-ticks shape), deterministic delivery at/after commit, coalescing (≥2 boundaries → one wake), nothing-due silence (same-tick not-due schedule + a later all-not-due tick, in-frame and post-commit), and the end-to-end runner leg (a live run_schedules leader's wake reaches a registered listener); local listener_hears helper per the per-file helper convention; rollback arm native (NOTIFY transactional delivery — no tick-fault seam built, per the task pin). Docs: tx.rs 'The tx wakes' section + wake_tx doc name the scheduler fire path as a shared caller; scheduler.rs module docs + tick doc pin the coalesced commit-atomic semantics. Record updates: review-wave-4-fixes' recorded-for-later bullet, suite-scheduler-rows' gate disposition note + Notes bullet, review-wave-5 §Notes 4's audit row — all retired with pointers; implementation.md gains the review-rounds line (fires visible to registered listeners ahead of the mem engine's posture being written). Also carried: review-wave-5 §6 flake-ledger update — row_lock_ttl_expiry_and_reacquisition failed once more in a full pg run this session (green on the next two full runs + six focused; unrelated mechanism to this change), meeting the ledger's own 'fails again' trigger with the dedicated investigative session owed by the wave-7 watch carriage. Verified: full pg lib suite 122/122 vs the harness (new test green ×3 solo); pg contract suite 25/25 + schema 9/9; workspace cargo test green server-less (pg skips clean incl. the new test); clippy --all-targets -D warnings clean; fmt clean

This commit is contained in:
glm-5.3-flash committed 2026-10-10 10:27:35 +00:00
1 parent 5abd351c86
commit 57451cef46
8 files changed
+538 -14

No files matched your search

+30 -3
View File
@@ -54,7 +54,14 @@
//! quarantine advance is an ordinary write). The quarantined row re-
//! enters the due set on every subsequent tick (advanced strictly
//! past now each time), so it never fires and never fails the
//! runner while its stored spec is broken. Wake-driven tick
//! runner while its stored spec is broken. Each firing schedule's
//! fire wake rides the tick tx ([`crate::tx::wake_tx`]): one
//! coalesced `pg_notify` per tick per schedule that fired ≥ 1, on
//! the fired queue's name — commit-atomic (a rolled-back tick
//! discards the wake with its fire rows) and best-effort (ADR-006;
//! the queues row's pinned re-poll safety net covers any gap) —
//! closing the recorded cross-engine fire-wake latency-parity gap
//! (SQLite's watcher wakes on those commits). Wake-driven tick
//! advance: the runner parks on the sleep deadline but the loop's
//! stop-select shortens every sleep slice to ≤ 1 s; the slices renew
//! the lease and check the stop token only — the soonest is *not*
@@ -114,7 +121,7 @@ use crate::resolution::{
};
use crate::schema::{QualifiedTable, tables};
use crate::seam::{database_error, pg_error, pool_error};
use crate::tx::{encode_payload_bytes, enqueue_row};
use crate::tx::{encode_payload_bytes, enqueue_row, wake_tx};
/// The contract-constant per-task per-tick catch-up cap (ADR-009 §4 —
/// honker parity, not a knob): a downtime longer than 64 interval
@@ -424,7 +431,12 @@ pub(crate) fn unschedule(name: &str, ctx: EngineCtx) -> BoxedFuture<'static, Res
/// row lock that serializes rogue second tickers (layer (b)) — a
/// second ticker re-reading under the lock sees the advanced
/// `next_fire_at` and skips. pg's default isolation is exactly the
/// committed-snapshot+row-lock semantics the guarantee names.
/// committed-snapshot+row-lock semantics the guarantee names. Each
/// firing schedule also issues its wake — one coalesced `pg_notify`
/// per tick per schedule that fired ≥ 1, on the fired queue's name,
/// inside the same tx ([`crate::tx::wake_tx`]: commit-atomic — a
/// rolled-back tick discards the wake with its fire rows; best-effort,
/// ADR-006).
/// Returns the soonest next fire across all schedules for the sleep
/// deadline (0 = none registered).
pub(crate) async fn tick(client: &tokio_postgres::Client, schema: &str, now: i64) -> Result<i64> {
@@ -547,6 +559,21 @@ pub(crate) async fn tick(client: &tokio_postgres::Client, schema: &str, now: i64
next_fire_at = next_fire_at.saturating_add(interval);
}
// The fire wake (the recorded cross-engine latency-parity gap
// closed): one coalesced wake per firing schedule per tick —
// after the fire loop, still inside the caller's `in_tx` frame,
// so NOTIFY's native transactional delivery makes it commit-
// atomic (a rolled-back tick discards the wake with its fire
// rows). Channel = the fired queue's name; empty payload; one
// per tick per schedule that fired ≥ 1 (not per fire) — the
// wakes are hint-only (ADR-006), and the SQLite watcher's
// cadence is the committed tick transaction, not the enqueued
// row; a quarantined row (the `continue` above) fired nothing
// and wakes nothing. Best-effort — a failed wake never fails
// the tick (ADR-006); the queues row's pinned re-poll safety
// net covers any gap.
wake_tx(client, &row.queue).await;
client
.execute(
&format!(
@@ -136,6 +136,32 @@ fn qopts() -> QueueOpts {
}
}
/// The forwarder's fanout subscription surface — the listener hears
/// what the server delivers; the listener-side probe for the fire-wake
/// assertions (commit-atomic delivery, coalescing, nothing-due
/// silence). Same shape as `tx_tests.rs`'s helper (the per-file
/// helper convention).
async fn listener_hears(
rx: &mut tokio::sync::broadcast::Receiver<crate::forwarder::RawNotification>,
channel: &str,
within: Duration,
) -> bool {
let deadline = tokio::time::Instant::now() + within;
loop {
let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
if remaining.is_zero() {
return false;
}
match tokio::time::timeout(remaining, rx.recv()).await {
Ok(Ok(n)) if n.channel == channel => return true,
Ok(Ok(_)) => continue,
Ok(Err(tokio::sync::broadcast::error::RecvError::Lagged(_))) => continue,
Ok(Err(tokio::sync::broadcast::error::RecvError::Closed)) => return false,
Err(_) => return false,
}
}
}
/// The scheduler table row probe (raw admin SQL — the schedule
/// registry is engine-internal). Returns (next_fire_at, spec, queue).
async fn sched_row(
@@ -1167,6 +1193,216 @@ async fn tampered_spec_row_is_quarantined_and_the_runner_survives() {
drop_schema(&fx.admin, &fx.schema).await;
}
/// Acceptance: the tick's fire path issues its `pg_notify` wake
/// commit-atomically (the recorded cross-engine fire-wake parity gap,
/// closed) — pinned by four arms, the
/// `tx_producer_wakes_are_commit_atomic_and_rollback_discards_them`
/// pattern at the tick's `in_tx` frame:
/// - pre-commit silence: the wake is statement-issued inside the
/// tick's open tx and delivers only at/after its COMMIT (commit-
/// atomic, not statement-atomic);
/// - coalesced to one wake per firing schedule per tick, not per fire
/// (two elapsed boundaries in one tick ⇒ one wake);
/// - a schedule with nothing due in the same tick (and a later tick
/// with nothing due at all) issues no wake;
/// - the end-to-end runner shape: a live `run_schedules` leader's fire
/// wake reaches a registered listener at the tick's commit.
///
/// The direct-tick legs drive the engine's own tick through the
/// runner's own `in_tx` frame (the `rogue_concurrent_ticks` shape —
/// the pre-commit silence window needs the tick body observable before
/// its COMMIT, which the runner's loop holds internally). The rollback
/// arm is native (NOTIFY's transactional delivery — proven by
/// `notify_tx` and the tx-wake test); no tick-fault seam is built
/// here.
#[tokio::test(flavor = "multi_thread")]
async fn scheduler_fire_wakes_are_commit_atomic_coalesced_and_nothing_due_is_silent() {
let Some(fx) = Fixture::open(&instance_namer("so_wake")).await else {
eprintln!("skip: no harness server");
return;
};
// `beat` fires two elapsed boundaries in the first tick (backdated
// exactly one interval into the past — no live boundary races the
// window); `later` sits an hour out (nothing due anywhere).
fx.store
.schedule("beat", "@every 1s", "fireq", json!(1), Default::default())
.await
.unwrap();
fx.store
.schedule(
"later",
"@every 3600s",
"quietq",
json!(1),
Default::default(),
)
.await
.unwrap();
let now = now_unix();
fx.admin
.execute(
&format!(
"UPDATE {} SET next_fire_at = $1 WHERE name = 'beat'",
crate::schema::QualifiedTable {
schema: &fx.schema,
name: crate::schema::tables::SCHEDULE,
}
),
&[&(now - 1)],
)
.await
.unwrap();
// Listeners registered BEFORE any write rides (the tx-wake test's
// posture): the fired queue's channel and the nothing-due queue's.
fx.store.forwarder().register("fireq").await.unwrap();
fx.store.forwarder().register("quietq").await.unwrap();
let mut fire_rx = fx.store.forwarder().subscribe().unwrap();
let mut quiet_rx = fx.store.forwarder().subscribe().unwrap();
// The tick through the runner's own `in_tx` frame: fire the two
// boundaries, then observe the pre-commit silence window inside
// the still-open tx (statement-issued, not delivered).
let schema = fx.schema.clone();
let (silent_pre_commit, mut fire_rx) = {
let client = fx.store.pool().get().await.unwrap();
crate::queue::in_tx(client, |client| {
let schema = schema.clone();
Box::pin(async move {
crate::scheduler::tick(client, &schema, now_unix()).await?;
tokio::time::sleep(Duration::from_millis(300)).await;
let silent =
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(100)).await;
Ok((silent, fire_rx))
})
})
.await
.unwrap()
};
assert!(
silent_pre_commit,
"no wake may deliver before the tick's COMMIT"
);
// At/after the COMMIT the coalesced wake delivers: one wake for
// the firing schedule, not one per fire.
assert!(
listener_hears(&mut fire_rx, "fireq", Duration::from_secs(5)).await,
"a committed tick wakes the fired queue's channel"
);
assert!(
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(400)).await,
"the wake is coalesced per firing schedule per tick, not per fire"
);
assert!(
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(300)).await,
"a schedule with nothing due in the same tick issues no wake"
);
// The coalescing arm's load-bearing premise: both elapsed
// boundaries really fired in that one tick.
let fired: i64 = fx
.admin
.query_one(
&format!(
"SELECT count(*) FROM {} WHERE queue = 'fireq'",
crate::schema::QualifiedTable {
schema: &fx.schema,
name: crate::schema::tables::JOB,
}
),
&[],
)
.await
.unwrap()
.get(0);
assert!(fired >= 2, "two elapsed boundaries fired, got {fired}");
// Pin `beat` an hour out (deterministic — the engine-internal
// boundary, backdated via the same admin-SQL posture as the
// catch-up tests), then the nothing-due tick: no wake inside the
// frame nor after its commit, on either channel.
fx.admin
.execute(
&format!(
"UPDATE {} SET next_fire_at = $1 WHERE name = 'beat'",
crate::schema::QualifiedTable {
schema: &fx.schema,
name: crate::schema::tables::SCHEDULE,
}
),
&[&(now_unix() + 3600)],
)
.await
.unwrap();
let (both_silent, mut fire_rx, mut quiet_rx) = {
let client = fx.store.pool().get().await.unwrap();
let schema = fx.schema.clone();
crate::queue::in_tx(client, |client| {
let schema = schema.clone();
Box::pin(async move {
crate::scheduler::tick(client, &schema, now_unix()).await?;
tokio::time::sleep(Duration::from_millis(300)).await;
let fire_silent =
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(100)).await;
let quiet_silent =
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(100)).await;
Ok((fire_silent && quiet_silent, fire_rx, quiet_rx))
})
})
.await
.unwrap()
};
assert!(
both_silent,
"the nothing-due tick must wake nobody inside its frame"
);
assert!(
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(300)).await,
"a tick with nothing due issues no wake (fireq)"
);
assert!(
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(300)).await,
"a tick with nothing due issues no wake (quietq)"
);
// The end-to-end runner leg: a live `run_schedules` leader's fire
// wake reaches a registered listener at the tick's commit — the
// original gap's headline (fires on pg visible to listeners).
fx.store.forwarder().register("runnerq").await.unwrap();
let mut runner_rx = fx.store.forwarder().subscribe().unwrap();
fx.store
.schedule(
"runner",
"@every 1s",
"runnerq",
json!(1),
Default::default(),
)
.await
.unwrap();
let stop = StopToken::new();
let runner_store = open_store(&harness_dsn().unwrap(), test_opts(&fx.schema))
.await
.unwrap();
let runner_stop = stop.clone();
let runner = tokio::spawn(async move { runner_store.run_schedules(runner_stop).await });
assert!(
listener_hears(&mut runner_rx, "runnerq", Duration::from_secs(15)).await,
"a live runner's fire wake reaches a registered listener"
);
stop.cancel();
let outcome = tokio::time::timeout(Duration::from_secs(10), runner)
.await
.expect("clean stop must terminate promptly")
.unwrap();
assert!(matches!(outcome, Ok(())), "{outcome:?}");
drop_schema(&fx.admin, &fx.schema).await;
}
/// The pg engine's own `@every` parser's unit pins (server-less —
/// ADR-012 §2's one-owner rule: this engine owns its grammar; the
/// contract suite pins the twins' outcomes identical, wave 5).
+8 -3
View File
@@ -63,7 +63,10 @@
//! channel realization) **inside the caller's transaction** ([`wake_tx`]).
//! NOTIFY's native transactional delivery makes the wake commit-atomic:
//! it delivers at the caller's COMMIT, and a rollback discards the row
//! *with* its wake (the tx/no-ghosts rows' pin). Wakes are best-effort
//! *with* its wake (the tx/no-ghosts rows' pin). The scheduler's tick
//! fire path rides the same [`wake_tx`] (one coalesced wake per firing
//! schedule per tick, on the fired queue's name — the scheduler's
//! docs). Wakes are best-effort
//! (ADR-006; the engine-wide posture — a wake must never fail a write):
//! a failed statement is logged and swallowed, and since a failed
//! statement aborts the tx server-side the truth still surfaces typed
@@ -283,8 +286,10 @@ fn enqueue_row_sql(schema: &str) -> String {
/// effort at the op (the engine-wide posture — a wake must never fail
/// a write): a failed statement is logged and swallowed; because a
/// failed statement aborts the tx server-side, the truth still
/// surfaces typed at COMMIT.
async fn wake_tx(client: &tokio_postgres::Client, channel: &str) {
/// surfaces typed at COMMIT. Shared by the tx producer paths and the
/// scheduler's tick fire path (the tick's in-tx fire wake — the same
/// commit-atomic semantics at the tick's own `in_tx` frame).
pub(crate) async fn wake_tx(client: &tokio_postgres::Client, channel: &str) {
if let Err(wake_err) = client
.query_one("SELECT pg_notify($1, '')", &[&channel])
.await