release-flake-investigation: the fired lock-row trigger investigated and resolved — harness-model, fixed test-side
Instrumentation first: the tee-before-filter capture procedure of record (demonstrated to preserve failing messages, immediately vindicating itself — two natural repros captured verbatim), then the arithmetic read against lock.rs/resolution.rs (delay-only argument confirmed for the lapse legs, corrected for the contention leg: second-precision now_unix stamps let a ttl=1 holder's exclusion window collapse below a second), the conditions matrix (48+ rounds: threads, concurrency with the SQLite column, cold/warm, CPU-load stress), mechanism probes (1.2 s delay fails deterministically on the exact assert; 200 ms lapses 2/10, frac-dependent), and the natural repro (2/30 loaded solo runs, exact message at properties.rs:563). Disposition: reproduced, root cause harness-model — the engine acted per ADR-008 §7 throughout; the pre-fix row's contention leg asserted a live-TTL state across an unbounded wall-clock gap. Trivially test-side, so the fix landed in-session under the AC's exception: lock_ttl_expiry_and_reacquisition is now two-legged (300 s exclusion leg released; the 1 s lapse leg's post-sleep asserts stay delay-only-safe); every pinned semantic survives. Verified: 30/30 loaded runs of the fixed row green under the exact repro condition; sqlite + mem columns green; workspace gates green (build, live-pg workspace tests, clippy -D warnings, fmt); taskgraph validate green. Watch ledger: the lock row's flag retires; the trim row stays watch-flagged (no repro, 10 focused + 7 loaded rounds); a new single occurrence of row_stream_ordering_equivalence under synthetic load (message captured — the row's drain window raced the pg bridge's incomplete channel fill) is watch-flagged for review-wave-8's flake outcome; no ride-along fix (instrumentation-first discipline). Tasks: release-flake-investigation -> completed with Notes + Summary; review-wave-5 ledger dated update.
This commit is contained in:
1 parent
c3a0c54df0
commit
0344d101d1
3 files changed
+238
-20
No files matched your search
@@ -532,9 +532,21 @@ pub async fn duration_refusal_on_non_positive_ttl(factory: &dyn StoreFactory) {
|
||||
/// acquire succeeds; the original holder's post-expiry `renew` is
|
||||
/// refused (`false` — the deadline-lapse refusal of the handle's
|
||||
/// uniform predicate shape); and the second owner's `release` frees
|
||||
/// the name for a third acquirer. Short-TTL (1 s) stamps with the
|
||||
/// tolerance-sleep posture (state outcomes asserted, never durations;
|
||||
/// sequential single-store drive).
|
||||
/// the name for a third acquirer.
|
||||
///
|
||||
/// Two-leg shape, each leg owning the posture its assertions can
|
||||
/// bear. The **exclusion leg** holds a long TTL (300 s — a span the
|
||||
/// row's own duration cannot cross): exclusion-while-held is a
|
||||
/// *live-TTL* state, and a short-TTL holder would hand that assertion
|
||||
/// a wall-clock-collapsible window — second-precision engine clocks
|
||||
/// stamp a 1 s expiry that lapses after as little as `1 - frac` of a
|
||||
/// second, so any scheduling/server stall between the two acquisitions
|
||||
/// (>~1 s suffices) spuriously fails the exclusion assert itself (the
|
||||
/// 2026-10-11 flake investigation's natural repro: two of thirty
|
||||
/// loaded solo runs failed exactly so, probe-confirmed on the leg). The **lapse leg** carries the Short-TTL (1 s) stamp with the
|
||||
/// tolerance-sleep posture: its assertions are *post-sleep states* a
|
||||
/// slow clock can only delay, never un-lapse (state outcomes
|
||||
/// asserted, never durations; sequential single-store drive).
|
||||
///
|
||||
/// Cross-reference: the duration-guard legs (`ttl <= 0` rejecting on
|
||||
/// `try_lock`/`renew`) pin in `duration_refusal_on_non_positive_ttl`
|
||||
@@ -551,8 +563,11 @@ pub async fn duration_refusal_on_non_positive_ttl(factory: &dyn StoreFactory) {
|
||||
pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
|
||||
let store = factory.open().await.expect("factory opens a store");
|
||||
|
||||
// The exclusion leg — long-TTL holder deliberately (see the row
|
||||
// doc's two-leg posture): the held-lock exclusion assert must not
|
||||
// race the holder's own expiry.
|
||||
let holder = store
|
||||
.try_lock("lapse", "a", 1)
|
||||
.try_lock("lapse", "a", 300)
|
||||
.await
|
||||
.expect("try_lock on a free name must not error")
|
||||
.expect("the first acquire holds the free name");
|
||||
@@ -564,11 +579,24 @@ pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
|
||||
contender.is_none(),
|
||||
"a held lock excludes a second acquirer"
|
||||
);
|
||||
assert!(
|
||||
holder
|
||||
.release()
|
||||
.await
|
||||
.expect("the exclusion leg's holder release errs only on storage failure"),
|
||||
"the exclusion leg's holder releases its own row"
|
||||
);
|
||||
|
||||
// TTL expiry (a 1 s stamp; the sleep is bounded well past the
|
||||
// second-precision stamp's resolution): exclusion lapses silently
|
||||
// — the second owner's acquire is a clean Ok(Some) with no error
|
||||
// preceding it and no revocation event to observe.
|
||||
// The lapse leg — a fresh short-TTL holder: TTL expiry (a 1 s
|
||||
// stamp; the sleep is bounded well past the second-precision
|
||||
// stamp's resolution) lapses exclusion silently — the second
|
||||
// owner's acquire is a clean Ok(Some) with no error preceding it
|
||||
// and no revocation event to observe.
|
||||
let lapsing_holder = store
|
||||
.try_lock("lapse", "a", 1)
|
||||
.await
|
||||
.expect("try_lock on the freed name must not error")
|
||||
.expect("the lapse leg's acquire holds the freed name");
|
||||
past_stamp_sleep(2_500).await;
|
||||
let second = store
|
||||
.try_lock("lapse", "b", 300)
|
||||
@@ -577,7 +605,7 @@ pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
|
||||
.expect("after TTL expiry exclusion has silently lapsed: a second owner acquires");
|
||||
|
||||
assert!(
|
||||
!holder
|
||||
!lapsing_holder
|
||||
.renew(300)
|
||||
.await
|
||||
.expect("renew errs only on storage failure, never on lapse"),
|
||||
|
||||
Reference in new issue
Block a user