glm-5.3-flash a48c12745a pg-fix-scheduler-tick-seam — the pg scheduler's transient-fault tick seam lands, retiring review 003 Finding 1 (MEDIUM: the tick retry loop was dead code under test — the last uncovered hardening arm of the wave-4 fix batch, the exact class review 002's live bugs came from). The seam follows the sqlite-commit-error-arm pattern of record: a cfg(test) per-store counting fault arm (crate::seam::tick_fault — Flag = Arc<AtomicU64>, born disarmed at 0, arm(n) adds, take() consumes one via CAS try_update; a counting arm because a boolean cannot reach the exhaustion arm — arm(TICK_RETRIES+1) faults the loop's whole budget while arm(1) leaves the later ticks real) feeding a fabricated pool/database-shaped opaque Database error into the REAL retry loop — both tests drive the full run_schedules → tick_with_retries → tick_once path end-to-end, no loop replication. Plumbing per the house pattern: cfg-gated tick_fault field on EngineCtx and PgStore (engine_ctx() carries the clone; production builds never materialize it), cfg-gated fault param on tick_once (the seam check sits before the real attempt — one attempt faults, never ticks) and tick_with_retries, cfg-branched attempt calls; PgStore::arm_tick_fault(faults) arming surface + tick_fault_flag() observation surface (a Flag clone so the consumption pin survives the store moving into a runner task). Two arms pinned: (a) one_faulted_tick_is_retried_and_the_job_fires — fault consumed on attempt 1, the 250 ms backoff sleeps, the retried attempt fires the backdated due boundary into the queue, exactly-one consumption, the runner survives the hiccup to a clean stop Ok(()); (b) tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse — the exhaustion exits typed (opaque Database, never LeadershipLost; the tick-phase context rides the source chain with the fabricated fault at its bottom; wall-clock ≥ 1.7 s proving the real backoff window), the leadership lock row is left (not released) to lapse at its TTL (present, expiry bounded by the run's TTL horizon), exact consumption, and the store remains fully usable after — the "runner survives" pin as the module docs' contract states it. Replay-proofed live: with the seam's consumption disabled the exhaustion test cannot complete (the runner loops forever — no fault ever enters the budget; the mutated run's timeout is itself the proof); the test is then bounded (20 s) so a future regression of this shape fails, not hangs. LEADER_TTL_S widened private → pub(crate) for the lock-left-to-TTL expiry pin. Verified: full pg column live vs the harness (124 lib + 25 suite + 9 schema; new tests green solo ×3); cargo test --workspace green server-less, skips clean (core 25 + harness 3, sqlite 191 + 25); clippy --all-targets -D warnings clean; fmt clean; cargo-llvm-cov confirms the Finding-1 sites (the retry/backoff/exhaustion-arm region + the tick_once seam arm) carry zero missed lines; taskgraph validate green (61 tasks)

2026-10-10 11:00:26 +00:00

..

src

pg-fix-scheduler-tick-seam — the pg scheduler's transient-fault tick seam lands, retiring review 003 Finding 1 (MEDIUM: the tick retry loop was dead code under test — the last uncovered hardening arm of the wave-4 fix batch, the exact class review 002's live bugs came from). The seam follows the sqlite-commit-error-arm pattern of record: a cfg(test) per-store counting fault arm (crate::seam::tick_fault — Flag = Arc<AtomicU64>, born disarmed at 0, arm(n) adds, take() consumes one via CAS try_update; a counting arm because a boolean cannot reach the exhaustion arm — arm(TICK_RETRIES+1) faults the loop's whole budget while arm(1) leaves the later ticks real) feeding a fabricated pool/database-shaped opaque Database error into the REAL retry loop — both tests drive the full run_schedules → tick_with_retries → tick_once path end-to-end, no loop replication. Plumbing per the house pattern: cfg-gated tick_fault field on EngineCtx and PgStore (engine_ctx() carries the clone; production builds never materialize it), cfg-gated fault param on tick_once (the seam check sits before the real attempt — one attempt faults, never ticks) and tick_with_retries, cfg-branched attempt calls; PgStore::arm_tick_fault(faults) arming surface + tick_fault_flag() observation surface (a Flag clone so the consumption pin survives the store moving into a runner task). Two arms pinned: (a) one_faulted_tick_is_retried_and_the_job_fires — fault consumed on attempt 1, the 250 ms backoff sleeps, the retried attempt fires the backdated due boundary into the queue, exactly-one consumption, the runner survives the hiccup to a clean stop Ok(()); (b) tick_retry_exhaustion_exits_typed_and_leaves_the_lock_to_lapse — the exhaustion exits typed (opaque Database, never LeadershipLost; the tick-phase context rides the source chain with the fabricated fault at its bottom; wall-clock ≥ 1.7 s proving the real backoff window), the leadership lock row is left (not released) to lapse at its TTL (present, expiry bounded by the run's TTL horizon), exact consumption, and the store remains fully usable after — the "runner survives" pin as the module docs' contract states it. Replay-proofed live: with the seam's consumption disabled the exhaustion test cannot complete (the runner loops forever — no fault ever enters the budget; the mutated run's timeout is itself the proof); the test is then bounded (20 s) so a future regression of this shape fails, not hangs. LEADER_TTL_S widened private → pub(crate) for the lock-left-to-TTL expiry pin. Verified: full pg column live vs the harness (124 lib + 25 suite + 9 schema; new tests green solo ×3); cargo test --workspace green server-less, skips clean (core 25 + harness 3, sqlite 191 + 25); clippy --all-targets -D warnings clean; fmt clean; cargo-llvm-cov confirms the Finding-1 sites (the retry/backoff/exhaustion-arm region + the tick_once seam arm) carry zero missed lines; taskgraph validate green (61 tasks)

2026-10-10 11:00:26 +00:00

tests

Contract-suite rows (task suite-opts-backoff-rows): the backoff-curve equivalence row and the deferred enqueue-opts clock legs. backoff_curve_equivalence — the equal-jitter exponential pinned as the range, not a jitter label (ADR-010 §3): claim → retry(err, None) cycles on a backoff_base_s = 2 queue land the ranges [1,2], [2,4], [4,8] across attempts 1–3, each computed delay read back through get_job's resolved run_at minus a clock read taken before the retry — the lower bound race-free (the retry's internal clock read cannot precede the suite's, so the observed value over-reads the delay, never under-reads) and the upper bound carrying a one-second straddle tolerance for the integer-second stamp crossing a wall second; identical bounds on both engines is the equivalence pin (ADR-012 §2, the row body shared). The explicit-delay override legs ride the same row: retry(err, Some(5)) honored verbatim ([5, 6] under the same straddle tolerance) and retry(err, Some(0)) resolving ready-now per the boundary-total rule (ADR-023 §2), claimed within a bounded wait. enqueue_opts_resolution extended in place with the wave-5 legs its deferral note named — the run_at-alone literal leg (a future run_at is the row's ready time verbatim, a deterministic equality with no clock read involved; a past run_at stores the literal too and is claimable now through the run_at <= now predicate within a bounded wait, the run_at-ASC ordering making the observation unambiguous among the row's other legs) and the neither-field leg (ready at the enqueue instant, abs_diff(now) <= 5 tolerance-bounded proximity, never a tight timing assert); the deferral note replaced by the completion statement, ADR-023 §2 stamp accumulated per the convention (ADR-020 §1 governs the resolutions). Cap-leg disposition recorded in Notes for the review gate: the 1-hour cap is not suite-pinnable (capped attempts need minute-scale waits) and stays pinned engine-side on both engines' unit tests per the wave-3/4 reviews. Ready-now waits are bounded claim_one poll loops (deadline asserts only, sequential single-store drive), with the one draft defect the finding surfaced (a re-claim after the bounded wait consumed nothing; the helper returns the claimed handle) noted. Wired into both engines' suite targets (SQLite tokio test, pg harness_row!), 25 rows per column up from 24. Verified: SQLite column green server-less (25/25 suite, 189 lib), pg column green vs harness (postgres/poc@:15432 — 25/25 suite, 121 lib + 9 schema), 3 solo re-runs of each new/extended row per engine (determinism), server-less pg skips cleanly via the reachability gate, cargo test -p alkstore-sqlite -p alkstore-postgres green, clippy -D warnings, fmt clean

2026-10-10 07:26:55 +00:00

Cargo.toml

pg forwarder: the reconnect arm retries failed connects until success or shutdown — one failed connect no longer kills the loop permanently; every loop exit path releases the fanout sender via the shared-slot drop guard (receivers-terminal-iff-loop-gone); reconnect-config test seam + failed-connect pins: six-cycle exhausted-backoff server-less unit, pre-flip abort, guard drop, and the harness end-to-end unreachable-outage → full-recovery test (bug replay-proven against the old shape) (task pg-fix-forwarder-reconnect, review 002 Finding 1)

2026-10-09 22:31:05 +00:00