pg-fix-scheduler-tick-seam — the pg scheduler's transient-fault tick seam lands, retiring review 003 Finding 1 (MEDIUM: the tick retry loop was dead code under test — the last uncovered hardening arm of the wave-4 fix batch, the exact class review 002's live bugs came from). The seam follows the sqlite-commit-error-arm pattern of record: a cfg(test) per-store counting fault arm (crate::seam::tick_fault — Flag = Arc<AtomicU64>, born disarmed at 0, arm(n) adds, take() consumes one via CAS try_update; a counting arm because a boolean cannot reach the exhaustion arm — arm(TICK_RETRIES+1) faults the loop's whole budget while arm(1) leaves the later ticks real) feeding a fabricated pool/database-shaped opaque Database error into the REAL retry loop — both tests drive the full run_schedules → tick_with_retries → tick_once path end-to-end, no loop replication. Plumbing per the house pattern: cfg-gated tick_fault field on EngineCtx and PgStore (engine_ctx() carries the clone; production builds never materialize it), cfg-gated fault param on tick_once (the seam check sits before the real attempt — one attempt faults, never ticks) and tick_with_retries, cfg-branched attempt calls; PgStore::arm_tick_fault(faults) arming surface + tick_fault_flag() observation surface (a Flag clone so the consumption pin survives the store moving into a runner task). Two arms pinned: (a) one_faulted_tick_is_retried_and_the_job_fires — fault consumed on attempt 1, the 250 ms backoff sleeps, the retried attempt fires the backdated due boundary into the queue, exactly-one consumption, the runner survives the hiccup to a clean stop Ok(()); (b) tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse — the exhaustion exits typed (opaque Database, never LeadershipLost; the tick-phase context rides the source chain with the fabricated fault at its bottom; wall-clock ≥ 1.7 s proving the real backoff window), the leadership lock row is left (not released) to lapse at its TTL (present, expiry bounded by the run's TTL horizon), exact consumption, and the store remains fully usable after — the "runner survives" pin as the module docs' contract states it. Replay-proofed live: with the seam's consumption disabled the exhaustion test cannot complete (the runner loops forever — no fault ever enters the budget; the mutated run's timeout is itself the proof); the test is then bounded (20 s) so a future regression of this shape fails, not hangs. LEADER_TTL_S widened private → pub(crate) for the lock-left-to-TTL expiry pin. Verified: full pg column live vs the harness (124 lib + 25 suite + 9 schema; new tests green solo ×3); cargo test --workspace green server-less, skips clean (core 25 + harness 3, sqlite 191 + 25); clippy --all-targets -D warnings clean; fmt clean; cargo-llvm-cov confirms the Finding-1 sites (the retry/backoff/exhaustion-arm region + the tick_once seam arm) carry zero missed lines; taskgraph validate green (61 tasks)
This commit is contained in:
1 parent
57451cef46
commit
a48c12745a
5 files changed
+461
-13
No files matched your search
@@ -120,6 +120,8 @@ use crate::resolution::{
|
|||||||
Stamps, now_unix, outbox_backing_queue_name, outbox_default_stamps, plain_queue_default_stamps,
|
Stamps, now_unix, outbox_backing_queue_name, outbox_default_stamps, plain_queue_default_stamps,
|
||||||
};
|
};
|
||||||
use crate::schema::{QualifiedTable, tables};
|
use crate::schema::{QualifiedTable, tables};
|
||||||
|
#[cfg(test)]
|
||||||
|
use crate::seam::tick_fault;
|
||||||
use crate::seam::{database_error, pg_error, pool_error};
|
use crate::seam::{database_error, pg_error, pool_error};
|
||||||
use crate::tx::{encode_payload_bytes, enqueue_row, wake_tx};
|
use crate::tx::{encode_payload_bytes, enqueue_row, wake_tx};
|
||||||
|
|
||||||
@@ -132,7 +134,7 @@ const SCHEDULER_MAX_CATCHUP_FIRES: i64 = 64;
|
|||||||
/// renewed every loop iteration *and* across the sleeps (a runner
|
/// renewed every loop iteration *and* across the sleeps (a runner
|
||||||
/// that cannot renew has lost the lock, ADR-009 §4 — the lock
|
/// that cannot renew has lost the lock, ADR-009 §4 — the lock
|
||||||
/// machinery is the engine's own).
|
/// machinery is the engine's own).
|
||||||
const LEADER_TTL_S: i64 = 10;
|
pub(crate) const LEADER_TTL_S: i64 = 10;
|
||||||
|
|
||||||
/// Seconds between lease renewals during the sleep.
|
/// Seconds between lease renewals during the sleep.
|
||||||
const SLEEP_RENEW_EVERY_S: u64 = 2;
|
const SLEEP_RENEW_EVERY_S: u64 = 2;
|
||||||
@@ -300,6 +302,11 @@ pub(crate) struct EngineCtx {
|
|||||||
pub pool: Pool,
|
pub pool: Pool,
|
||||||
pub schema: String,
|
pub schema: String,
|
||||||
pub closed: Arc<AtomicBool>,
|
pub closed: Arc<AtomicBool>,
|
||||||
|
/// The tick-fault seam flag (the seam module's per-store scoping —
|
||||||
|
/// `born disarmed`, never armed in production builds, which do not
|
||||||
|
/// compile the field at all).
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) tick_fault: crate::seam::tick_fault::Flag,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl EngineCtx {
|
impl EngineCtx {
|
||||||
@@ -612,10 +619,13 @@ pub(crate) async fn tick(client: &tokio_postgres::Client, schema: &str, now: i64
|
|||||||
|
|
||||||
/// The leader loop (see the module docs).
|
/// The leader loop (see the module docs).
|
||||||
pub(crate) fn run_schedules(stop: StopToken, ctx: EngineCtx) -> BoxedFuture<'static, Result<()>> {
|
pub(crate) fn run_schedules(stop: StopToken, ctx: EngineCtx) -> BoxedFuture<'static, Result<()>> {
|
||||||
|
#[cfg(test)]
|
||||||
|
let tick_fault = ctx.tick_fault.clone();
|
||||||
let EngineCtx {
|
let EngineCtx {
|
||||||
pool,
|
pool,
|
||||||
schema,
|
schema,
|
||||||
closed,
|
closed,
|
||||||
|
..
|
||||||
} = ctx;
|
} = ctx;
|
||||||
Box::pin(async move {
|
Box::pin(async move {
|
||||||
if closed.load(Ordering::Acquire) {
|
if closed.load(Ordering::Acquire) {
|
||||||
@@ -667,6 +677,9 @@ pub(crate) fn run_schedules(stop: StopToken, ctx: EngineCtx) -> BoxedFuture<'sta
|
|||||||
// before ticking: never fire on a stolen lease.
|
// before ticking: never fire on a stolen lease.
|
||||||
return Err(Error::LeadershipLost);
|
return Err(Error::LeadershipLost);
|
||||||
}
|
}
|
||||||
|
#[cfg(test)]
|
||||||
|
let soonest = tick_with_retries(&pool, &schema, Some(&tick_fault)).await?;
|
||||||
|
#[cfg(not(test))]
|
||||||
let soonest = tick_with_retries(&pool, &schema).await?;
|
let soonest = tick_with_retries(&pool, &schema).await?;
|
||||||
if stop.is_cancelled() {
|
if stop.is_cancelled() {
|
||||||
let _ = lease.release().await;
|
let _ = lease.release().await;
|
||||||
@@ -684,8 +697,22 @@ pub(crate) fn run_schedules(stop: StopToken, ctx: EngineCtx) -> BoxedFuture<'sta
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// One tick attempt over a fresh pool checkout (the loop body's
|
/// One tick attempt over a fresh pool checkout (the loop body's
|
||||||
/// original shape, hoisted for the retry wrapper).
|
/// original shape, hoisted for the retry wrapper). Under `cfg(test)`
|
||||||
async fn tick_once(pool: &Pool, schema: &str) -> Result<i64> {
|
/// the tick-fault seam (see `crate::seam::tick_fault`) precedes the
|
||||||
|
/// real attempt: an armed store faults this attempt with the
|
||||||
|
/// fabricated transient error into the loop's retry arms; the flag is
|
||||||
|
/// consumed-on-use, the next attempt is real.
|
||||||
|
async fn tick_once(
|
||||||
|
pool: &Pool,
|
||||||
|
schema: &str,
|
||||||
|
#[cfg(test)] fault: Option<&tick_fault::Flag>,
|
||||||
|
) -> Result<i64> {
|
||||||
|
#[cfg(test)]
|
||||||
|
if let Some(flag) = fault
|
||||||
|
&& tick_fault::take(flag)
|
||||||
|
{
|
||||||
|
return Err(tick_fault::fabricated_transient_error());
|
||||||
|
}
|
||||||
let conn = pool.get().await.map_err(pool_error)?;
|
let conn = pool.get().await.map_err(pool_error)?;
|
||||||
in_tx(conn, |client| {
|
in_tx(conn, |client| {
|
||||||
let schema = schema.to_string();
|
let schema = schema.to_string();
|
||||||
@@ -703,11 +730,20 @@ async fn tick_once(pool: &Pool, schema: &str) -> Result<i64> {
|
|||||||
/// the loss decision regardless). Each failed attempt logs (the
|
/// the loss decision regardless). Each failed attempt logs (the
|
||||||
/// engine's diagnostic posture); the exhausting exit wraps the last
|
/// engine's diagnostic posture); the exhausting exit wraps the last
|
||||||
/// error with the tick-phase context in the source chain
|
/// error with the tick-phase context in the source chain
|
||||||
/// ([`context_error`]).
|
/// ([`context_error`]). The attempt call below is cfg-branched (the
|
||||||
async fn tick_with_retries(pool: &Pool, schema: &str) -> Result<i64> {
|
/// house pattern — production builds compile the plain path).
|
||||||
|
async fn tick_with_retries(
|
||||||
|
pool: &Pool,
|
||||||
|
schema: &str,
|
||||||
|
#[cfg(test)] fault: Option<&tick_fault::Flag>,
|
||||||
|
) -> Result<i64> {
|
||||||
let mut retries = 0u64;
|
let mut retries = 0u64;
|
||||||
loop {
|
loop {
|
||||||
match tick_once(pool, schema).await {
|
#[cfg(test)]
|
||||||
|
let attempt = tick_once(pool, schema, fault).await;
|
||||||
|
#[cfg(not(test))]
|
||||||
|
let attempt = tick_once(pool, schema).await;
|
||||||
|
match attempt {
|
||||||
Ok(soonest) => return Ok(soonest),
|
Ok(soonest) => return Ok(soonest),
|
||||||
Err(e) if retries < TICK_RETRIES => {
|
Err(e) if retries < TICK_RETRIES => {
|
||||||
retries += 1;
|
retries += 1;
|
||||||
@@ -792,6 +828,7 @@ pub(crate) fn new_outbox_handle(name: &str, ctx: EngineCtx) -> Result<Box<dyn Ou
|
|||||||
pool,
|
pool,
|
||||||
schema,
|
schema,
|
||||||
closed,
|
closed,
|
||||||
|
..
|
||||||
} = ctx;
|
} = ctx;
|
||||||
Ok(Box::new(PgOutboxHandle {
|
Ok(Box::new(PgOutboxHandle {
|
||||||
name: name.to_string(),
|
name: name.to_string(),
|
||||||
|
|||||||
@@ -41,3 +41,50 @@ pub(crate) fn build_error(e: deadpool_postgres::BuildError) -> Error {
|
|||||||
pub(crate) fn database_error(message: impl Into<String>) -> Error {
|
pub(crate) fn database_error(message: impl Into<String>) -> Error {
|
||||||
Error::database(std::io::Error::other(message.into()))
|
Error::database(std::io::Error::other(message.into()))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The `cfg(test)` tick-fault seam (`pg-fix-scheduler-tick-seam`,
|
||||||
|
/// review 003 Finding 1): a per-store arm that, when set, substitutes
|
||||||
|
/// a fabricated pool/database-shaped error for one `tick_once`
|
||||||
|
/// attempt, feeding the *real* retry loop (`tick_with_retries`) — the
|
||||||
|
/// retry, its backoff sleeps, the exhaustion error, and the diagnostic
|
||||||
|
/// execute under test, never a replica (the
|
||||||
|
/// `sqlite-commit-error-arm` pattern).
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) mod tick_fault {
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
|
||||||
|
use alkstore::Error;
|
||||||
|
|
||||||
|
/// The remaining-fault count (born disarmed at 0; consumed-on-use;
|
||||||
|
/// parallel test stores never interfere — the commit-fault seam's
|
||||||
|
/// per-store scoping).
|
||||||
|
pub(crate) type Flag = Arc<AtomicU64>;
|
||||||
|
|
||||||
|
pub(crate) fn disarmed() -> Flag {
|
||||||
|
Arc::new(AtomicU64::new(0))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn arm(flag: &Flag, faults: u64) {
|
||||||
|
flag.fetch_add(faults, Ordering::Release);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Consume one fault (the count semantics: arming `N` faults the
|
||||||
|
/// next `N` attempts, so the exhaustion arm reaches the typed
|
||||||
|
/// error while a 1-count arm leaves the later ticks real).
|
||||||
|
pub(crate) fn take(flag: &Flag) -> bool {
|
||||||
|
flag.try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| {
|
||||||
|
(remaining > 0).then(|| remaining - 1)
|
||||||
|
})
|
||||||
|
.is_ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A pool/database-shaped transient error (the taxonomy's opaque
|
||||||
|
/// fallback with the source chain — the shape a hiccup surfaces).
|
||||||
|
pub(crate) fn fabricated_transient_error() -> Error {
|
||||||
|
Error::database(std::io::Error::other(
|
||||||
|
"fabricated transient fault (cfg(test) seam): pool/database \
|
||||||
|
unavailable for one tick attempt",
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -54,6 +54,11 @@ pub struct PgStore {
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
listener_application_name: String,
|
listener_application_name: String,
|
||||||
closed: Arc<AtomicBool>,
|
closed: Arc<AtomicBool>,
|
||||||
|
/// The tick-fault seam's per-store flag (the `sqlite-commit-error-
|
||||||
|
/// arm` pattern — born disarmed, consumed-on-use; production
|
||||||
|
/// builds do not compile the field).
|
||||||
|
#[cfg(test)]
|
||||||
|
tick_fault: crate::seam::tick_fault::Flag,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for PgStore {
|
impl std::fmt::Debug for PgStore {
|
||||||
@@ -101,6 +106,24 @@ impl PgStore {
|
|||||||
&self.schema
|
&self.schema
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Arm the store's tick-fault seam — the next `faults` `tick_once`
|
||||||
|
/// attempts fault with the fabricated transient error into the
|
||||||
|
/// leader loop's real retry arms. Per-store scoping (`Arc` counter,
|
||||||
|
/// born disarmed, consumed-on-use) keeps parallel test stores
|
||||||
|
/// isolated. `#[cfg(test)]`-gated.
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) fn arm_tick_fault(&self, faults: u64) {
|
||||||
|
crate::seam::tick_fault::arm(&self.tick_fault, faults);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The seam's flag (a `Flag` clone — the remaining-fault count is
|
||||||
|
/// loadable from it for the consumption-exactness pins even while
|
||||||
|
/// the store has moved into a runner). `#[cfg(test)]`-gated.
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) fn tick_fault_flag(&self) -> crate::seam::tick_fault::Flag {
|
||||||
|
self.tick_fault.clone()
|
||||||
|
}
|
||||||
|
|
||||||
/// The scheduler/outbox paths' wiring bundle (the pool, the
|
/// The scheduler/outbox paths' wiring bundle (the pool, the
|
||||||
/// schema, the closed flag).
|
/// schema, the closed flag).
|
||||||
fn engine_ctx(&self) -> crate::scheduler::EngineCtx {
|
fn engine_ctx(&self) -> crate::scheduler::EngineCtx {
|
||||||
@@ -108,6 +131,8 @@ impl PgStore {
|
|||||||
pool: self.pool.clone(),
|
pool: self.pool.clone(),
|
||||||
schema: self.schema.clone(),
|
schema: self.schema.clone(),
|
||||||
closed: self.closed.clone(),
|
closed: self.closed.clone(),
|
||||||
|
#[cfg(test)]
|
||||||
|
tick_fault: self.tick_fault.clone(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -224,6 +249,8 @@ pub(crate) async fn open_store(config: &str, opts: PgOpts) -> alkstore::Result<P
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
listener_application_name,
|
listener_application_name,
|
||||||
closed: Arc::new(AtomicBool::new(false)),
|
closed: Arc::new(AtomicBool::new(false)),
|
||||||
|
#[cfg(test)]
|
||||||
|
tick_fault: crate::seam::tick_fault::disarmed(),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1472,3 +1472,244 @@ fn the_tick_retry_policy_is_pinned() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Acceptance: the transient-fault seam drives the *real* retry loop
|
||||||
|
/// (`tick_with_retries` — no loop replication; the
|
||||||
|
/// `sqlite-commit-error-arm` pattern). Arm (a), retry-then-success:
|
||||||
|
/// the seam arms one fault, the faulted first attempt is retried after
|
||||||
|
/// its backoff sleep, the retried attempt ticks for real and the due
|
||||||
|
/// boundary fires the job into the queue — the runner survives the
|
||||||
|
/// hiccup (the resilience property the fix promised) to a clean stop.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn one_faulted_tick_is_retried_and_the_job_fires() {
|
||||||
|
use std::sync::atomic::Ordering;
|
||||||
|
|
||||||
|
let Some(fx) = Fixture::open(&instance_namer("so_flt")).await else {
|
||||||
|
eprintln!("skip: no harness server");
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
|
||||||
|
fx.store
|
||||||
|
.schedule("sweep", "@every 1s", "retryq", json!(1), Default::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
// One elapsed boundary due before the runner ever starts (the
|
||||||
|
// catch-up tests' backdate posture — no live boundary races the
|
||||||
|
// window): the retried attempt is the tick that fires it.
|
||||||
|
let now = now_unix();
|
||||||
|
fx.admin
|
||||||
|
.execute(
|
||||||
|
&format!(
|
||||||
|
"UPDATE {} SET next_fire_at = $1 WHERE name = 'sweep'",
|
||||||
|
crate::schema::QualifiedTable {
|
||||||
|
schema: &fx.schema,
|
||||||
|
name: crate::schema::tables::SCHEDULE,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
&[&(now - 1)],
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let runner_store = open_store(&harness_dsn().unwrap(), test_opts(&fx.schema))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
runner_store.arm_tick_fault(1);
|
||||||
|
let flag = runner_store.tick_fault_flag();
|
||||||
|
|
||||||
|
let stop = StopToken::new();
|
||||||
|
let runner_stop = stop.clone();
|
||||||
|
let runner = tokio::spawn(async move { runner_store.run_schedules(runner_stop).await });
|
||||||
|
|
||||||
|
// Attempt 1 faults (consumed, no round trip), the loop sleeps its
|
||||||
|
// backoff, and the retried attempt ticks for real — the due
|
||||||
|
// boundary fires into the queue.
|
||||||
|
let deadline = tokio::time::Instant::now() + Duration::from_secs(20);
|
||||||
|
let fired: i64 = loop {
|
||||||
|
let c: i64 = fx
|
||||||
|
.admin
|
||||||
|
.query_one(
|
||||||
|
&format!(
|
||||||
|
"SELECT count(*) FROM {} WHERE queue = 'retryq'",
|
||||||
|
crate::schema::QualifiedTable {
|
||||||
|
schema: &fx.schema,
|
||||||
|
name: crate::schema::tables::JOB,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
&[],
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.get(0);
|
||||||
|
if c >= 1 {
|
||||||
|
break c;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
tokio::time::Instant::now() < deadline,
|
||||||
|
"the retried tick never fired — the retry arm broke the runner"
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||||
|
};
|
||||||
|
assert!(fired >= 1, "the retried attempt fired the due boundary");
|
||||||
|
|
||||||
|
// The seam consumed exactly its one fault (the runner's later
|
||||||
|
// ticks stayed real), and the runner survived the hiccup: the
|
||||||
|
// clean stop is `Ok(())`.
|
||||||
|
assert_eq!(
|
||||||
|
flag.load(Ordering::Acquire),
|
||||||
|
0,
|
||||||
|
"exactly one fault armed must be exactly one consumed"
|
||||||
|
);
|
||||||
|
stop.cancel();
|
||||||
|
let outcome = tokio::time::timeout(Duration::from_secs(10), runner)
|
||||||
|
.await
|
||||||
|
.expect("clean stop must terminate promptly")
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
matches!(outcome, Ok(())),
|
||||||
|
"the runner must survive the faulted tick: {outcome:?}"
|
||||||
|
);
|
||||||
|
|
||||||
|
drop_schema(&fx.admin, &fx.schema).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Acceptance: arm (b), exhaustion — the seam faults the loop's whole
|
||||||
|
/// budget (`TICK_RETRIES + 1`: the initial attempt plus every retry),
|
||||||
|
/// the loop exhausts to the typed error shape (opaque `Database`; the
|
||||||
|
/// tick-phase context rides the source chain with the fabricated fault
|
||||||
|
/// at its bottom), the backoff-sleep arms execute for real, and the
|
||||||
|
/// exit strands nothing: the leadership lock row is left (not
|
||||||
|
/// released) to lapse at its TTL and the store remains fully usable —
|
||||||
|
/// the resilience posture the exhaustion exit owes.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse() {
|
||||||
|
use std::sync::atomic::Ordering;
|
||||||
|
|
||||||
|
let Some(fx) = Fixture::open(&instance_namer("so_exh")).await else {
|
||||||
|
eprintln!("skip: no harness server");
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Nothing is due ever (`@every 3600s`, one boundary out) — the
|
||||||
|
// exhausting exit is fault-driven, not boundary-driven.
|
||||||
|
fx.store
|
||||||
|
.schedule(
|
||||||
|
"slow",
|
||||||
|
"@every 3600s",
|
||||||
|
"exhaustq",
|
||||||
|
json!(1),
|
||||||
|
Default::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let runner_store = open_store(&harness_dsn().unwrap(), test_opts(&fx.schema))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
runner_store.arm_tick_fault(crate::scheduler::TICK_RETRIES + 1);
|
||||||
|
let flag = runner_store.tick_fault_flag();
|
||||||
|
|
||||||
|
// The runner is driven in-line (bounded — a regression that never
|
||||||
|
// exhausts fails here, not a hang); its exit shape *is* the pin.
|
||||||
|
let started = std::time::Instant::now();
|
||||||
|
let err = tokio::time::timeout(
|
||||||
|
Duration::from_secs(20),
|
||||||
|
runner_store.run_schedules(StopToken::new()),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("the exhausting loop must exit within its backoff budget")
|
||||||
|
.unwrap_err();
|
||||||
|
|
||||||
|
// The backoff-sleep arms executed for real: the exhaustion took the
|
||||||
|
// full 250 ms → 500 ms → 1 s window (a fast-fail exit would not).
|
||||||
|
let elapsed = started.elapsed();
|
||||||
|
assert!(
|
||||||
|
elapsed >= Duration::from_millis(1_700),
|
||||||
|
"the exhaustion must have slept the full backoff window (1.75 s), took {elapsed:?}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
elapsed < Duration::from_secs(20),
|
||||||
|
"the exhaustion must exit promptly after the budget, took {elapsed:?}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The typed exhaustion shape: opaque `Database` (never
|
||||||
|
// LeadershipLost, never a panic), the tick-phase context riding the
|
||||||
|
// source chain with the fabricated fault preserved at its bottom.
|
||||||
|
assert!(
|
||||||
|
matches!(err, Error::Database(..)),
|
||||||
|
"the exhausting loop must exit opaque Database: {err:?}"
|
||||||
|
);
|
||||||
|
let mut link: &(dyn std::error::Error + 'static) = &err;
|
||||||
|
let mut saw_context = false;
|
||||||
|
let mut saw_fault = false;
|
||||||
|
loop {
|
||||||
|
let text = link.to_string();
|
||||||
|
if text.contains("scheduler tick failed after 3 retries")
|
||||||
|
&& text.contains("the leadership lock row is left to lapse at its TTL")
|
||||||
|
{
|
||||||
|
saw_context = true;
|
||||||
|
}
|
||||||
|
if text.contains("fabricated transient fault") {
|
||||||
|
saw_fault = true;
|
||||||
|
}
|
||||||
|
match link.source() {
|
||||||
|
Some(next) => link = next,
|
||||||
|
None => break,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
saw_context,
|
||||||
|
"the exhaustion context (tick phase, lock posture) must ride the source chain"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
saw_fault,
|
||||||
|
"the fabricated transiency must stay at the chain's bottom (source preserved)"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Exactly the four attempts consumed the four armed faults.
|
||||||
|
assert_eq!(
|
||||||
|
flag.load(Ordering::Acquire),
|
||||||
|
0,
|
||||||
|
"the budget's fault count must be consumed exactly"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The leadership lock row is left (not released) on the `Err` exit —
|
||||||
|
// its `expires_at` is the TTL's own horizon (renewed once at the
|
||||||
|
// loop's top, never renewed again past the exhaustion).
|
||||||
|
let row = fx
|
||||||
|
.admin
|
||||||
|
.query_one(
|
||||||
|
&format!(
|
||||||
|
"SELECT count(*), max(expires_at) FROM {} WHERE name = $1",
|
||||||
|
crate::schema::QualifiedTable {
|
||||||
|
schema: &fx.schema,
|
||||||
|
name: crate::schema::tables::LOCKS,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
&[&crate::scheduler::SCHEDULER_LEADER_LOCK],
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let count: i64 = row.get(0);
|
||||||
|
let expires_at: i64 = row.get(1);
|
||||||
|
assert_eq!(count, 1, "the lock row is left to lapse, not released");
|
||||||
|
assert!(
|
||||||
|
expires_at > now_unix(),
|
||||||
|
"the left-behind expiry is still live (inside its TTL)"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
expires_at <= now_unix() + crate::scheduler::LEADER_TTL_S,
|
||||||
|
"the lock row was left to the TTL, not renewed past the exit"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The store remains fully usable after exhaustion (the runner's
|
||||||
|
// exit strands nothing): a subsequent entry point works.
|
||||||
|
let after = runner_store
|
||||||
|
.schedule("after", "@every 1s", "postq", json!(1), Default::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(after.name, "after");
|
||||||
|
|
||||||
|
drop_schema(&fx.admin, &fx.schema).await;
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
id: pg-fix-scheduler-tick-seam
|
id: pg-fix-scheduler-tick-seam
|
||||||
name: "pg scheduler — transient-fault tick seam: retry/exhaustion arms tested (review 003 Finding 1)"
|
name: "pg scheduler — transient-fault tick seam: retry/exhaustion arms tested (review 003 Finding 1)"
|
||||||
status: pending
|
status: completed
|
||||||
depends_on: []
|
depends_on: []
|
||||||
scope: narrow
|
scope: narrow
|
||||||
risk: low
|
risk: low
|
||||||
@@ -42,13 +42,13 @@ disarmed, consumed-on-use.
|
|||||||
|
|
||||||
## Acceptance Criteria
|
## Acceptance Criteria
|
||||||
|
|
||||||
- [ ] The seam lands cfg(test)-gated; the real `tick_with_retries`
|
- [x] The seam lands cfg(test)-gated; the real `tick_with_retries`
|
||||||
executes under test (no loop replication)
|
executes under test (no loop replication)
|
||||||
- [ ] Arms (a) and (b) pinned as above; the runner survives
|
- [x] Arms (a) and (b) pinned as above; the runner survives
|
||||||
exhaustion
|
exhaustion
|
||||||
- [ ] `cargo test -p alkstore-postgres` green against the harness
|
- [x] `cargo test -p alkstore-postgres` green against the harness
|
||||||
server; skips clean server-less
|
server; skips clean server-less
|
||||||
- [ ] Clippy `-D warnings`, fmt clean; workspace gates green
|
- [x] Clippy `-D warnings`, fmt clean; workspace gates green
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
@@ -61,8 +61,104 @@ disarmed, consumed-on-use.
|
|||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
> To be filled by implementation agent
|
> Decisions of record the implementation made that the description
|
||||||
|
> didn't pin:
|
||||||
|
|
||||||
|
- **The seam is a per-store *counting* fault arm, not a boolean.** A
|
||||||
|
boolean arm (the sqlite commit-fault shape) reaches only the
|
||||||
|
retry-once arm; exhaustion needs the fault to survive the loop's
|
||||||
|
whole budget. `crate::seam::tick_fault` holds
|
||||||
|
`Flag = Arc<AtomicU64>` — born disarmed at 0, `arm(n)` adds n,
|
||||||
|
`take` consumes one (CAS `try_update`, no negative underflow) — so
|
||||||
|
arm(1) gives the retry-then-success arm and arm(`TICK_RETRIES + 1`)
|
||||||
|
the exhaustion arm with one mechanism. Consumption-exactness is
|
||||||
|
pinned by loading the flag after the run (0 = exactly the armed
|
||||||
|
count consumed).
|
||||||
|
- **Plumbing follows the `sqlite-commit-error-arm` precedent
|
||||||
|
exactly** (the `pg-fix-forwarder-reconnect` house pattern):
|
||||||
|
`#[cfg(test)]` field on `EngineCtx` and `PgStore`;
|
||||||
|
`PgStore::engine_ctx()` carries the store's flag clone into every
|
||||||
|
scheduler ctx; `#[cfg(test)]` param on `tick_once` / `tick_with_retries`
|
||||||
|
(the seam check sits at `tick_once`'s top — one *attempt* faults,
|
||||||
|
never ticks) with cfg-branched attempt statements — production
|
||||||
|
builds compile the plain path and never materialize the field.
|
||||||
|
Arming/observation surfaces: `PgStore::arm_tick_fault(faults)` and
|
||||||
|
`PgStore::tick_fault_flag()` (a `Flag` clone so the consumption pin
|
||||||
|
survives the store moving into a runner task — a load is enough,
|
||||||
|
no extra accessor). Per-store `Arc` scoping keeps parallel test
|
||||||
|
stores isolated.
|
||||||
|
- **The fault enters the loop via the entry points it already calls**
|
||||||
|
— both tests drive the full real path (`run_schedules` → the loop's
|
||||||
|
`tick_with_retries` → `tick_once`), no loop or error-shape
|
||||||
|
replication anywhere. The fabricated error is pool/database-shaped
|
||||||
|
(`Error::database(io::Error)` — the taxonomy's opaque fallback; the
|
||||||
|
loop retries on any `Err`, so the shape is an honesty choice, not a
|
||||||
|
correctness need).
|
||||||
|
- **"The runner survives exhaustion" pinned as written.** The
|
||||||
|
exhausting loop's exit *is* the typed error (the module docs'
|
||||||
|
contract — exhaustion exits `Err`); "survives" is pinned as: no
|
||||||
|
panic, opaque `Database` (never `LeadershipLost`), the tick-phase
|
||||||
|
context riding the source chain with the fabricated fault at its
|
||||||
|
bottom, the leadership lock row left (not released) to lapse at its
|
||||||
|
TTL (present, expiry bounded by the TTL of the run — never renewed
|
||||||
|
past the exit), and the store fully usable after (subsequent entry
|
||||||
|
points work). Arm (a) carries the *hiccups-don't-kill* survival
|
||||||
|
property (fault → backoff → real tick fires the due boundary →
|
||||||
|
clean stop `Ok(())`).
|
||||||
|
- **Replay-proofed live** (the sqlite task's discipline): with the
|
||||||
|
seam's fault consumption disabled (`false && take(…)` probe), the
|
||||||
|
exhaustion test cannot complete — the runner loops forever on its
|
||||||
|
nothing-due schedule because no fault ever enters the budget (the
|
||||||
|
mutation run was killed on timeout, which is itself the proof the
|
||||||
|
seam is load-bearing); reverted, green. The exhaustion test was
|
||||||
|
then wrapped in a 20 s bounded wait so a future regression of this
|
||||||
|
shape fails instead of hanging.
|
||||||
|
- **The backoff-sleep arms are pinned by wall-clock**, not value: the
|
||||||
|
exhaustion run must take ≥ 1.7 s (the 250 ms → 500 ms → 1 s window;
|
||||||
|
a fast-fail exit would not). The retry/exhaustion arms' `eprintln`
|
||||||
|
diagnostics execute (lcov-verified) but are not value-pinned —
|
||||||
|
stderr capture isn't in the harness convention; the exhaustion
|
||||||
|
*context* message (chain-carried) is the pinned diagnostic shape.
|
||||||
|
- `LEADER_TTL_S` widened `private` → `pub(crate)` (the lock-left-to-
|
||||||
|
TTL expiry pin reads it; no production-path change).
|
||||||
|
- lcov after: the Finding-1 sites (the retry loop — retry arm, backoff
|
||||||
|
sleeps, exhaustion `context_error`, diagnostics — and the `tick_once`
|
||||||
|
seam arm) carry **zero missed lines**.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
> To be filled on completion
|
> What landed, verified how:
|
||||||
|
|
||||||
|
- **`alkstore-postgres/src/seam.rs`**: the `#[cfg(test)]`
|
||||||
|
`tick_fault` module — the per-store counting `Flag`
|
||||||
|
(`Arc<AtomicU64>`), `disarmed`/`arm`/`take` (consume-one CAS), and
|
||||||
|
the fabricated pool/database-shaped transient error, with the
|
||||||
|
pattern provenance (review 003 Finding 1 /
|
||||||
|
`sqlite-commit-error-arm`) in its doc comment.
|
||||||
|
- **`alkstore-postgres/src/scheduler.rs`**: cfg-gated `tick_fault`
|
||||||
|
field on `EngineCtx` (wildcard-safe destructures elsewhere); the
|
||||||
|
cfg-gated fault param on `tick_once` (seam check precedes the real
|
||||||
|
attempt) and `tick_with_retries`, attempt calls cfg-branched;
|
||||||
|
`run_schedules` threads the store's flag into the loop. The
|
||||||
|
production path never materializes the seam.
|
||||||
|
- **`alkstore-postgres/src/store.rs`**: cfg-gated `tick_fault` field
|
||||||
|
on `PgStore` (born `disarmed()`), `engine_ctx()` clones it into the
|
||||||
|
scheduler ctx, and the `arm_tick_fault` / `tick_fault_flag`
|
||||||
|
test-observation surfaces.
|
||||||
|
- **`alkstore-postgres/src/store/scheduler_tests.rs`**: two pins —
|
||||||
|
`one_faulted_tick_is_retried_and_the_job_fires` (arm a: faulted
|
||||||
|
attempt 1 → 250 ms backoff → the retried attempt fires the
|
||||||
|
backdated due boundary; exactly-one consumption; clean stop `Ok(())`)
|
||||||
|
and `tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse`
|
||||||
|
(arm b: `TICK_RETRIES + 1` armed faults exhaust the budget; ≥ 1.7 s
|
||||||
|
real backoff window; opaque `Database` with the tick-phase context
|
||||||
|
and the fabricated fault in the source chain; exact consumption; the
|
||||||
|
leadership lock row left to its TTL lapse; the store usable after).
|
||||||
|
- **Gates**: `cargo build`; the full pg column live vs the harness
|
||||||
|
(124 lib + 25 suite + 9 schema; the two new tests green solo ×3 —
|
||||||
|
4.2–4.4 s); `cargo test --workspace` server-less green (skips
|
||||||
|
clean: core 25 + harness 3; sqlite 191 + 25); `cargo clippy
|
||||||
|
--all-targets -- -D warnings`; `cargo fmt --check`. cargo-llvm-cov
|
||||||
|
confirms scheduler.rs's retry-loop region zero missed lines (the
|
||||||
|
crate stands at 90.73% scheduler line coverage; the residue is the
|
||||||
|
accepted defensive-arm posture).
|
||||||
Reference in new issue
Block a user