91 lines
3.9 KiB
Markdown
91 lines
3.9 KiB
Markdown
---
|
|
id: release-flake-investigation
|
|
name: Flake investigation — the pg lock-row trigger fired (dedicated session)
|
|
status: pending
|
|
depends_on: []
|
|
scope: moderate
|
|
risk: medium
|
|
impact: component
|
|
level: implementation
|
|
tags: [wave-8, postgres, flake-watch]
|
|
---
|
|
|
|
## Description
|
|
|
|
The wave-5 gate's flake-watch flag, carried to this pass
|
|
(`docs/plans/implementation.md` §Wave 8; `tasks/review-wave-5.md`
|
|
§Notes 6). The watch's trigger has **fired**:
|
|
`row_lock_ttl_expiry_and_reacquisition` failed a second time (the
|
|
ledger update, 2026-10-10, from the `pg-fix-scheduler-fire-wake`
|
|
session — one occurrence in one full sequential pg suite run, failure
|
|
message lost to output filtering, then green on two subsequent full
|
|
runs and six focused runs). Per the ledger's own prescription, a
|
|
**dedicated investigative session (instrumentation + focused repro
|
|
loop) is owed before any postulate-and-fix**. The trim row
|
|
(`row_trim_to_semantics`, one development-time failure) stays
|
|
watch-flagged; if it reproduces in this session's loops, it joins the
|
|
investigation.
|
|
|
|
The investigation's shape (the ledger's terms, not a
|
|
postulate-and-fix license):
|
|
|
|
1. **Instrument first**: capture the failing row's full assertion
|
|
output (the prior failures' messages were lost to truncation/filtering
|
|
— fix that blind spot before anything else: run with
|
|
`--nocapture`, or a file-tee'd harness, or a retry-wrapper that
|
|
preserves output on failure).
|
|
2. **Focused repro loop**: the row under stress — many consecutive
|
|
runs, varied `--test-threads`, solo vs concurrent-with-SQLite-column,
|
|
cold vs warm server — the conditions matrix the two prior
|
|
occurrences happened under (cold first run; full sequential run).
|
|
A deliberate slow-clock/lapse-boundary probe (the row's lapse
|
|
assertions are post-sleep states a slower clock can only delay,
|
|
never un-lapse — verify that argument by reading the row's actual
|
|
sleep/tolerance arithmetic against the engine's lock expiry
|
|
implementation, not by trusting it).
|
|
3. **Disposition**: either (a) reproduced → root-cause analysis → the
|
|
fix is its own follow-up task (engine bug vs harness-model, called
|
|
explicitly — the review-002 precedent is that a "flake" was a real
|
|
engine bug); or (b) not reproduced → the bounded-investigation
|
|
record (conditions matrix, instrumented output, the read-back of
|
|
the delay-only argument) with the watch posture updated (keep
|
|
watching vs instrument permanently vs accept). Both outcomes are
|
|
acceptable task completions; an uninvestigated shrug is not.
|
|
|
|
Requires the harness Postgres server (the POC container posture —
|
|
`docs/reviews/003-wave-5-general-review.md` §Op note has the env).
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] The output-capture blind spot fixed (a failing run's assertion
|
|
message is preserved by the recorded procedure)
|
|
- [ ] The conditions matrix actually run (counts + configurations
|
|
recorded in Notes)
|
|
- [ ] The row's sleep/tolerance arithmetic read against the pg lock
|
|
expiry implementation; the delay-only argument confirmed or
|
|
corrected
|
|
- [ ] Disposition recorded: reproduced (→ root cause + follow-up task
|
|
created) or not-reproduced (→ bounded-investigation record +
|
|
watch posture updated)
|
|
- [ ] If the trim row reproduces: it joins the investigation and its
|
|
outcome is recorded too
|
|
- [ ] Workspace gates green at session end (any fix is a follow-up
|
|
task, not a ride-along — unless the root cause is trivially
|
|
test-side, in which case fix + full gates + Notes)
|
|
|
|
## References
|
|
|
|
- tasks/review-wave-5.md §Notes 6 (the ledger + the trigger update)
|
|
- docs/plans/implementation.md §Wave 8 (the watch carriage)
|
|
- docs/reviews/003-wave-5-general-review.md (the concurrence read +
|
|
the harness-credentials op note)
|
|
- docs/reviews/002-wave-4-general-review.md (the flake-was-a-real-bug
|
|
precedent, F-1)
|
|
|
|
## Notes
|
|
|
|
> To be filled by implementation agent
|
|
|
|
## Summary
|
|
|
|
> To be filled on completion |