pg-fix-scheduler-fire-wake — the scheduler tick's fire wake lands, closing the recorded cross-engine fire-wake latency-parity gap (the wave-5 gate's one standing item, review-wave-4-fixes' recorded-for-later note): scheduler.rs::tick now issues one coalesced pg_notify per firing schedule per tick after the fire loop, still inside the tick's in_tx frame — channel = the fired queue's name (schedule() rejects reserved names per ADR-021 §3, so the plain queue name is always the right channel), empty payload (Wake { channel } only, ADR-008 §3), best-effort via the shared tx::wake_tx (widened private → pub(crate) per the recorded one-call shape; enqueue_row stays wake-free — no double-wake; auto-commit Queue::enqueue and the tx producer paths untouched). NOTIFY's native transactional delivery makes the wake commit-atomic: a rolled-back tick (crash mid-tick ⇒ boundary refires) discards the wake with its fire rows — the tx/no-ghosts discipline the tx producer paths already pin; coalescing default one-per-firing-schedule-per-tick (not per-fire) matching the SQLite watcher's per-committed-tick cadence, decision recorded in the task's Notes. New four-arm pinning test (scheduler_tests.rs, tx_producer_wakes_are_commit_atomic pattern): pre-commit silence inside the runner's own in_tx frame driving the engine's own tick (rogue-ticks shape), deterministic delivery at/after commit, coalescing (≥2 boundaries → one wake), nothing-due silence (same-tick not-due schedule + a later all-not-due tick, in-frame and post-commit), and the end-to-end runner leg (a live run_schedules leader's wake reaches a registered listener); local listener_hears helper per the per-file helper convention; rollback arm native (NOTIFY transactional delivery — no tick-fault seam built, per the task pin). Docs: tx.rs 'The tx wakes' section + wake_tx doc name the scheduler fire path as a shared caller; scheduler.rs module docs + tick doc pin the coalesced commit-atomic semantics. Record updates: review-wave-4-fixes' recorded-for-later bullet, suite-scheduler-rows' gate disposition note + Notes bullet, review-wave-5 §Notes 4's audit row — all retired with pointers; implementation.md gains the review-rounds line (fires visible to registered listeners ahead of the mem engine's posture being written). Also carried: review-wave-5 §6 flake-ledger update — row_lock_ttl_expiry_and_reacquisition failed once more in a full pg run this session (green on the next two full runs + six focused; unrelated mechanism to this change), meeting the ledger's own 'fails again' trigger with the dedicated investigative session owed by the wave-7 watch carriage. Verified: full pg lib suite 122/122 vs the harness (new test green ×3 solo); pg contract suite 25/25 + schema 9/9; workspace cargo test green server-less (pg skips clean incl. the new test); clippy --all-targets -D warnings clean; fmt clean
This commit is contained in:
1 parent
5abd351c86
commit
57451cef46
8 files changed
+538
-14
No files matched your search
@@ -54,7 +54,14 @@
|
||||
//! quarantine advance is an ordinary write). The quarantined row re-
|
||||
//! enters the due set on every subsequent tick (advanced strictly
|
||||
//! past now each time), so it never fires and never fails the
|
||||
//! runner while its stored spec is broken. Wake-driven tick
|
||||
//! runner while its stored spec is broken. Each firing schedule's
|
||||
//! fire wake rides the tick tx ([`crate::tx::wake_tx`]): one
|
||||
//! coalesced `pg_notify` per tick per schedule that fired ≥ 1, on
|
||||
//! the fired queue's name — commit-atomic (a rolled-back tick
|
||||
//! discards the wake with its fire rows) and best-effort (ADR-006;
|
||||
//! the queues row's pinned re-poll safety net covers any gap) —
|
||||
//! closing the recorded cross-engine fire-wake latency-parity gap
|
||||
//! (SQLite's watcher wakes on those commits). Wake-driven tick
|
||||
//! advance: the runner parks on the sleep deadline but the loop's
|
||||
//! stop-select shortens every sleep slice to ≤ 1 s; the slices renew
|
||||
//! the lease and check the stop token only — the soonest is *not*
|
||||
@@ -114,7 +121,7 @@ use crate::resolution::{
|
||||
};
|
||||
use crate::schema::{QualifiedTable, tables};
|
||||
use crate::seam::{database_error, pg_error, pool_error};
|
||||
use crate::tx::{encode_payload_bytes, enqueue_row};
|
||||
use crate::tx::{encode_payload_bytes, enqueue_row, wake_tx};
|
||||
|
||||
/// The contract-constant per-task per-tick catch-up cap (ADR-009 §4 —
|
||||
/// honker parity, not a knob): a downtime longer than 64 interval
|
||||
@@ -424,7 +431,12 @@ pub(crate) fn unschedule(name: &str, ctx: EngineCtx) -> BoxedFuture<'static, Res
|
||||
/// row lock that serializes rogue second tickers (layer (b)) — a
|
||||
/// second ticker re-reading under the lock sees the advanced
|
||||
/// `next_fire_at` and skips. pg's default isolation is exactly the
|
||||
/// committed-snapshot+row-lock semantics the guarantee names.
|
||||
/// committed-snapshot+row-lock semantics the guarantee names. Each
|
||||
/// firing schedule also issues its wake — one coalesced `pg_notify`
|
||||
/// per tick per schedule that fired ≥ 1, on the fired queue's name,
|
||||
/// inside the same tx ([`crate::tx::wake_tx`]: commit-atomic — a
|
||||
/// rolled-back tick discards the wake with its fire rows; best-effort,
|
||||
/// ADR-006).
|
||||
/// Returns the soonest next fire across all schedules for the sleep
|
||||
/// deadline (0 = none registered).
|
||||
pub(crate) async fn tick(client: &tokio_postgres::Client, schema: &str, now: i64) -> Result<i64> {
|
||||
@@ -547,6 +559,21 @@ pub(crate) async fn tick(client: &tokio_postgres::Client, schema: &str, now: i64
|
||||
next_fire_at = next_fire_at.saturating_add(interval);
|
||||
}
|
||||
|
||||
// The fire wake (the recorded cross-engine latency-parity gap
|
||||
// closed): one coalesced wake per firing schedule per tick —
|
||||
// after the fire loop, still inside the caller's `in_tx` frame,
|
||||
// so NOTIFY's native transactional delivery makes it commit-
|
||||
// atomic (a rolled-back tick discards the wake with its fire
|
||||
// rows). Channel = the fired queue's name; empty payload; one
|
||||
// per tick per schedule that fired ≥ 1 (not per fire) — the
|
||||
// wakes are hint-only (ADR-006), and the SQLite watcher's
|
||||
// cadence is the committed tick transaction, not the enqueued
|
||||
// row; a quarantined row (the `continue` above) fired nothing
|
||||
// and wakes nothing. Best-effort — a failed wake never fails
|
||||
// the tick (ADR-006); the queues row's pinned re-poll safety
|
||||
// net covers any gap.
|
||||
wake_tx(client, &row.queue).await;
|
||||
|
||||
client
|
||||
.execute(
|
||||
&format!(
|
||||
|
||||
@@ -136,6 +136,32 @@ fn qopts() -> QueueOpts {
|
||||
}
|
||||
}
|
||||
|
||||
/// The forwarder's fanout subscription surface — the listener hears
|
||||
/// what the server delivers; the listener-side probe for the fire-wake
|
||||
/// assertions (commit-atomic delivery, coalescing, nothing-due
|
||||
/// silence). Same shape as `tx_tests.rs`'s helper (the per-file
|
||||
/// helper convention).
|
||||
async fn listener_hears(
|
||||
rx: &mut tokio::sync::broadcast::Receiver<crate::forwarder::RawNotification>,
|
||||
channel: &str,
|
||||
within: Duration,
|
||||
) -> bool {
|
||||
let deadline = tokio::time::Instant::now() + within;
|
||||
loop {
|
||||
let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
|
||||
if remaining.is_zero() {
|
||||
return false;
|
||||
}
|
||||
match tokio::time::timeout(remaining, rx.recv()).await {
|
||||
Ok(Ok(n)) if n.channel == channel => return true,
|
||||
Ok(Ok(_)) => continue,
|
||||
Ok(Err(tokio::sync::broadcast::error::RecvError::Lagged(_))) => continue,
|
||||
Ok(Err(tokio::sync::broadcast::error::RecvError::Closed)) => return false,
|
||||
Err(_) => return false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The scheduler table row probe (raw admin SQL — the schedule
|
||||
/// registry is engine-internal). Returns (next_fire_at, spec, queue).
|
||||
async fn sched_row(
|
||||
@@ -1167,6 +1193,216 @@ async fn tampered_spec_row_is_quarantined_and_the_runner_survives() {
|
||||
drop_schema(&fx.admin, &fx.schema).await;
|
||||
}
|
||||
|
||||
/// Acceptance: the tick's fire path issues its `pg_notify` wake
|
||||
/// commit-atomically (the recorded cross-engine fire-wake parity gap,
|
||||
/// closed) — pinned by four arms, the
|
||||
/// `tx_producer_wakes_are_commit_atomic_and_rollback_discards_them`
|
||||
/// pattern at the tick's `in_tx` frame:
|
||||
/// - pre-commit silence: the wake is statement-issued inside the
|
||||
/// tick's open tx and delivers only at/after its COMMIT (commit-
|
||||
/// atomic, not statement-atomic);
|
||||
/// - coalesced to one wake per firing schedule per tick, not per fire
|
||||
/// (two elapsed boundaries in one tick ⇒ one wake);
|
||||
/// - a schedule with nothing due in the same tick (and a later tick
|
||||
/// with nothing due at all) issues no wake;
|
||||
/// - the end-to-end runner shape: a live `run_schedules` leader's fire
|
||||
/// wake reaches a registered listener at the tick's commit.
|
||||
///
|
||||
/// The direct-tick legs drive the engine's own tick through the
|
||||
/// runner's own `in_tx` frame (the `rogue_concurrent_ticks` shape —
|
||||
/// the pre-commit silence window needs the tick body observable before
|
||||
/// its COMMIT, which the runner's loop holds internally). The rollback
|
||||
/// arm is native (NOTIFY's transactional delivery — proven by
|
||||
/// `notify_tx` and the tx-wake test); no tick-fault seam is built
|
||||
/// here.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn scheduler_fire_wakes_are_commit_atomic_coalesced_and_nothing_due_is_silent() {
|
||||
let Some(fx) = Fixture::open(&instance_namer("so_wake")).await else {
|
||||
eprintln!("skip: no harness server");
|
||||
return;
|
||||
};
|
||||
|
||||
// `beat` fires two elapsed boundaries in the first tick (backdated
|
||||
// exactly one interval into the past — no live boundary races the
|
||||
// window); `later` sits an hour out (nothing due anywhere).
|
||||
fx.store
|
||||
.schedule("beat", "@every 1s", "fireq", json!(1), Default::default())
|
||||
.await
|
||||
.unwrap();
|
||||
fx.store
|
||||
.schedule(
|
||||
"later",
|
||||
"@every 3600s",
|
||||
"quietq",
|
||||
json!(1),
|
||||
Default::default(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let now = now_unix();
|
||||
fx.admin
|
||||
.execute(
|
||||
&format!(
|
||||
"UPDATE {} SET next_fire_at = $1 WHERE name = 'beat'",
|
||||
crate::schema::QualifiedTable {
|
||||
schema: &fx.schema,
|
||||
name: crate::schema::tables::SCHEDULE,
|
||||
}
|
||||
),
|
||||
&[&(now - 1)],
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Listeners registered BEFORE any write rides (the tx-wake test's
|
||||
// posture): the fired queue's channel and the nothing-due queue's.
|
||||
fx.store.forwarder().register("fireq").await.unwrap();
|
||||
fx.store.forwarder().register("quietq").await.unwrap();
|
||||
let mut fire_rx = fx.store.forwarder().subscribe().unwrap();
|
||||
let mut quiet_rx = fx.store.forwarder().subscribe().unwrap();
|
||||
|
||||
// The tick through the runner's own `in_tx` frame: fire the two
|
||||
// boundaries, then observe the pre-commit silence window inside
|
||||
// the still-open tx (statement-issued, not delivered).
|
||||
let schema = fx.schema.clone();
|
||||
let (silent_pre_commit, mut fire_rx) = {
|
||||
let client = fx.store.pool().get().await.unwrap();
|
||||
crate::queue::in_tx(client, |client| {
|
||||
let schema = schema.clone();
|
||||
Box::pin(async move {
|
||||
crate::scheduler::tick(client, &schema, now_unix()).await?;
|
||||
tokio::time::sleep(Duration::from_millis(300)).await;
|
||||
let silent =
|
||||
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(100)).await;
|
||||
Ok((silent, fire_rx))
|
||||
})
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
};
|
||||
assert!(
|
||||
silent_pre_commit,
|
||||
"no wake may deliver before the tick's COMMIT"
|
||||
);
|
||||
|
||||
// At/after the COMMIT the coalesced wake delivers: one wake for
|
||||
// the firing schedule, not one per fire.
|
||||
assert!(
|
||||
listener_hears(&mut fire_rx, "fireq", Duration::from_secs(5)).await,
|
||||
"a committed tick wakes the fired queue's channel"
|
||||
);
|
||||
assert!(
|
||||
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(400)).await,
|
||||
"the wake is coalesced per firing schedule per tick, not per fire"
|
||||
);
|
||||
assert!(
|
||||
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(300)).await,
|
||||
"a schedule with nothing due in the same tick issues no wake"
|
||||
);
|
||||
|
||||
// The coalescing arm's load-bearing premise: both elapsed
|
||||
// boundaries really fired in that one tick.
|
||||
let fired: i64 = fx
|
||||
.admin
|
||||
.query_one(
|
||||
&format!(
|
||||
"SELECT count(*) FROM {} WHERE queue = 'fireq'",
|
||||
crate::schema::QualifiedTable {
|
||||
schema: &fx.schema,
|
||||
name: crate::schema::tables::JOB,
|
||||
}
|
||||
),
|
||||
&[],
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.get(0);
|
||||
assert!(fired >= 2, "two elapsed boundaries fired, got {fired}");
|
||||
|
||||
// Pin `beat` an hour out (deterministic — the engine-internal
|
||||
// boundary, backdated via the same admin-SQL posture as the
|
||||
// catch-up tests), then the nothing-due tick: no wake inside the
|
||||
// frame nor after its commit, on either channel.
|
||||
fx.admin
|
||||
.execute(
|
||||
&format!(
|
||||
"UPDATE {} SET next_fire_at = $1 WHERE name = 'beat'",
|
||||
crate::schema::QualifiedTable {
|
||||
schema: &fx.schema,
|
||||
name: crate::schema::tables::SCHEDULE,
|
||||
}
|
||||
),
|
||||
&[&(now_unix() + 3600)],
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let (both_silent, mut fire_rx, mut quiet_rx) = {
|
||||
let client = fx.store.pool().get().await.unwrap();
|
||||
let schema = fx.schema.clone();
|
||||
crate::queue::in_tx(client, |client| {
|
||||
let schema = schema.clone();
|
||||
Box::pin(async move {
|
||||
crate::scheduler::tick(client, &schema, now_unix()).await?;
|
||||
tokio::time::sleep(Duration::from_millis(300)).await;
|
||||
let fire_silent =
|
||||
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(100)).await;
|
||||
let quiet_silent =
|
||||
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(100)).await;
|
||||
Ok((fire_silent && quiet_silent, fire_rx, quiet_rx))
|
||||
})
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
};
|
||||
assert!(
|
||||
both_silent,
|
||||
"the nothing-due tick must wake nobody inside its frame"
|
||||
);
|
||||
assert!(
|
||||
!listener_hears(&mut fire_rx, "fireq", Duration::from_millis(300)).await,
|
||||
"a tick with nothing due issues no wake (fireq)"
|
||||
);
|
||||
assert!(
|
||||
!listener_hears(&mut quiet_rx, "quietq", Duration::from_millis(300)).await,
|
||||
"a tick with nothing due issues no wake (quietq)"
|
||||
);
|
||||
|
||||
// The end-to-end runner leg: a live `run_schedules` leader's fire
|
||||
// wake reaches a registered listener at the tick's commit — the
|
||||
// original gap's headline (fires on pg visible to listeners).
|
||||
fx.store.forwarder().register("runnerq").await.unwrap();
|
||||
let mut runner_rx = fx.store.forwarder().subscribe().unwrap();
|
||||
fx.store
|
||||
.schedule(
|
||||
"runner",
|
||||
"@every 1s",
|
||||
"runnerq",
|
||||
json!(1),
|
||||
Default::default(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let stop = StopToken::new();
|
||||
let runner_store = open_store(&harness_dsn().unwrap(), test_opts(&fx.schema))
|
||||
.await
|
||||
.unwrap();
|
||||
let runner_stop = stop.clone();
|
||||
let runner = tokio::spawn(async move { runner_store.run_schedules(runner_stop).await });
|
||||
assert!(
|
||||
listener_hears(&mut runner_rx, "runnerq", Duration::from_secs(15)).await,
|
||||
"a live runner's fire wake reaches a registered listener"
|
||||
);
|
||||
|
||||
stop.cancel();
|
||||
let outcome = tokio::time::timeout(Duration::from_secs(10), runner)
|
||||
.await
|
||||
.expect("clean stop must terminate promptly")
|
||||
.unwrap();
|
||||
assert!(matches!(outcome, Ok(())), "{outcome:?}");
|
||||
|
||||
drop_schema(&fx.admin, &fx.schema).await;
|
||||
}
|
||||
|
||||
/// The pg engine's own `@every` parser's unit pins (server-less —
|
||||
/// ADR-012 §2's one-owner rule: this engine owns its grammar; the
|
||||
/// contract suite pins the twins' outcomes identical, wave 5).
|
||||
|
||||
@@ -63,7 +63,10 @@
|
||||
//! channel realization) **inside the caller's transaction** ([`wake_tx`]).
|
||||
//! NOTIFY's native transactional delivery makes the wake commit-atomic:
|
||||
//! it delivers at the caller's COMMIT, and a rollback discards the row
|
||||
//! *with* its wake (the tx/no-ghosts rows' pin). Wakes are best-effort
|
||||
//! *with* its wake (the tx/no-ghosts rows' pin). The scheduler's tick
|
||||
//! fire path rides the same [`wake_tx`] (one coalesced wake per firing
|
||||
//! schedule per tick, on the fired queue's name — the scheduler's
|
||||
//! docs). Wakes are best-effort
|
||||
//! (ADR-006; the engine-wide posture — a wake must never fail a write):
|
||||
//! a failed statement is logged and swallowed, and since a failed
|
||||
//! statement aborts the tx server-side the truth still surfaces typed
|
||||
@@ -283,8 +286,10 @@ fn enqueue_row_sql(schema: &str) -> String {
|
||||
/// effort at the op (the engine-wide posture — a wake must never fail
|
||||
/// a write): a failed statement is logged and swallowed; because a
|
||||
/// failed statement aborts the tx server-side, the truth still
|
||||
/// surfaces typed at COMMIT.
|
||||
async fn wake_tx(client: &tokio_postgres::Client, channel: &str) {
|
||||
/// surfaces typed at COMMIT. Shared by the tx producer paths and the
|
||||
/// scheduler's tick fire path (the tick's in-tx fire wake — the same
|
||||
/// commit-atomic semantics at the tick's own `in_tx` frame).
|
||||
pub(crate) async fn wake_tx(client: &tokio_postgres::Client, channel: &str) {
|
||||
if let Err(wake_err) = client
|
||||
.query_one("SELECT pg_notify($1, '')", &[&channel])
|
||||
.await
|
||||
|
||||
Reference in new issue
Block a user