release-flake-investigation: the fired lock-row trigger investigated and resolved — harness-model, fixed test-side

Instrumentation first: the tee-before-filter capture procedure of
record (demonstrated to preserve failing messages, immediately
vindicating itself — two natural repros captured verbatim), then the
arithmetic read against lock.rs/resolution.rs (delay-only argument
confirmed for the lapse legs, corrected for the contention leg:
second-precision now_unix stamps let a ttl=1 holder's exclusion
window collapse below a second), the conditions matrix (48+ rounds:
threads, concurrency with the SQLite column, cold/warm, CPU-load
stress), mechanism probes (1.2 s delay fails deterministically on the
exact assert; 200 ms lapses 2/10, frac-dependent), and the natural
repro (2/30 loaded solo runs, exact message at properties.rs:563).

Disposition: reproduced, root cause harness-model — the engine acted
per ADR-008 §7 throughout; the pre-fix row's contention leg asserted
a live-TTL state across an unbounded wall-clock gap. Trivially
test-side, so the fix landed in-session under the AC's exception:
lock_ttl_expiry_and_reacquisition is now two-legged (300 s exclusion
leg released; the 1 s lapse leg's post-sleep asserts stay
delay-only-safe); every pinned semantic survives.

Verified: 30/30 loaded runs of the fixed row green under the exact
repro condition; sqlite + mem columns green; workspace gates green
(build, live-pg workspace tests, clippy -D warnings, fmt); taskgraph
validate green.

Watch ledger: the lock row's flag retires; the trim row stays
watch-flagged (no repro, 10 focused + 7 loaded rounds); a new single
occurrence of row_stream_ordering_equivalence under synthetic load
(message captured — the row's drain window raced the pg bridge's
incomplete channel fill) is watch-flagged for review-wave-8's flake
outcome; no ride-along fix (instrumentation-first discipline).
Tasks: release-flake-investigation -> completed with Notes + Summary;
review-wave-5 ledger dated update.
This commit is contained in:
glm-5.3-flash committed 2026-10-11 01:13:05 +00:00
1 parent c3a0c54df0
commit 0344d101d1
3 files changed
+238 -20

No files matched your search

+37 -9
View File
@@ -532,9 +532,21 @@ pub async fn duration_refusal_on_non_positive_ttl(factory: &dyn StoreFactory) {
/// acquire succeeds; the original holder's post-expiry `renew` is
/// refused (`false` — the deadline-lapse refusal of the handle's
/// uniform predicate shape); and the second owner's `release` frees
/// the name for a third acquirer. Short-TTL (1 s) stamps with the
/// tolerance-sleep posture (state outcomes asserted, never durations;
/// sequential single-store drive).
/// the name for a third acquirer.
///
/// Two-leg shape, each leg owning the posture its assertions can
/// bear. The **exclusion leg** holds a long TTL (300 s — a span the
/// row's own duration cannot cross): exclusion-while-held is a
/// *live-TTL* state, and a short-TTL holder would hand that assertion
/// a wall-clock-collapsible window — second-precision engine clocks
/// stamp a 1 s expiry that lapses after as little as `1 - frac` of a
/// second, so any scheduling/server stall between the two acquisitions
/// (>~1 s suffices) spuriously fails the exclusion assert itself (the
/// 2026-10-11 flake investigation's natural repro: two of thirty
/// loaded solo runs failed exactly so, probe-confirmed on the leg). The **lapse leg** carries the Short-TTL (1 s) stamp with the
/// tolerance-sleep posture: its assertions are *post-sleep states* a
/// slow clock can only delay, never un-lapse (state outcomes
/// asserted, never durations; sequential single-store drive).
///
/// Cross-reference: the duration-guard legs (`ttl <= 0` rejecting on
/// `try_lock`/`renew`) pin in `duration_refusal_on_non_positive_ttl`
@@ -551,8 +563,11 @@ pub async fn duration_refusal_on_non_positive_ttl(factory: &dyn StoreFactory) {
pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
let store = factory.open().await.expect("factory opens a store");
// The exclusion leg — long-TTL holder deliberately (see the row
// doc's two-leg posture): the held-lock exclusion assert must not
// race the holder's own expiry.
let holder = store
.try_lock("lapse", "a", 1)
.try_lock("lapse", "a", 300)
.await
.expect("try_lock on a free name must not error")
.expect("the first acquire holds the free name");
@@ -564,11 +579,24 @@ pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
contender.is_none(),
"a held lock excludes a second acquirer"
);
assert!(
holder
.release()
.await
.expect("the exclusion leg's holder release errs only on storage failure"),
"the exclusion leg's holder releases its own row"
);
// TTL expiry (a 1 s stamp; the sleep is bounded well past the
// second-precision stamp's resolution): exclusion lapses silently
// — the second owner's acquire is a clean Ok(Some) with no error
// preceding it and no revocation event to observe.
// The lapse leg — a fresh short-TTL holder: TTL expiry (a 1 s
// stamp; the sleep is bounded well past the second-precision
// stamp's resolution) lapses exclusion silently — the second
// owner's acquire is a clean Ok(Some) with no error preceding it
// and no revocation event to observe.
let lapsing_holder = store
.try_lock("lapse", "a", 1)
.await
.expect("try_lock on the freed name must not error")
.expect("the lapse leg's acquire holds the freed name");
past_stamp_sleep(2_500).await;
let second = store
.try_lock("lapse", "b", 300)
@@ -577,7 +605,7 @@ pub async fn lock_ttl_expiry_and_reacquisition(factory: &dyn StoreFactory) {
.expect("after TTL expiry exclusion has silently lapsed: a second owner acquires");
assert!(
!holder
!lapsing_holder
.renew(300)
.await
.expect("renew errs only on storage failure, never on lapse"),