pg suite-infra hardening (task pg-suite-infra-hardening): the wave-4 review's F-1 defense-in-depth candidates — must_recv_event re-shaped from the 20 ms try_recv polling loop to a parked recv().await under the 15 s timeout wrapper (stream_tests.rs, the sole pg copy — SQLite twin untouched; the poll loop's wake-subscription interaction surface the F-1 flake suspected is out of the hot path, and the deadline assert still bounds the property). The parked form's terminal arms are explicit: Some(Err(e)) keeps the errored-instead-of-idling panic, None gets a receiver-closed panic the poll form never saw. The deadline miss self-diagnoses (the small-honest realization of candidate (b)'s discriminator): read_since(offset, 1000) over rows beyond the receiver's position names the residual class — rows durable undelivered (wake/re-drain class) vs no rows (publish-visibility class); with the parked recv a wake-arrived-but-re-drain-missed arm is structurally implausible (only a read error could miss, and it surfaces Err). Candidate (b)'s literal bridge-side counter skipped, reason recorded in the task (surfacing the bridge wake count across the dyn EventReceiver boundary needs downcast/keyed-global machinery beyond the small-honest bar); the small honest piece did land: wait_wake's Lagged(n) arm now logs (eprintln, house posture matching notify.rs's bridge_loop — and forwarder.rs's 'the receiver bridge's Lagged arm logs / recovers' doc claim now true at both bridges), stream name threaded into wait_wake's signature so the log attributes the lag. Verified: pg stream module 21/21 vs harness server (full module run), tx_publishes_compose_with_the_handle 5 solo re-runs green (determinism re-check under the parked shape), two consecutive full pg harness runs green (121 lib + 25 suite + 9 schema, ~72 s each), cargo test -p alkstore-postgres green server-less (skips clean), workspace cargo test green (12 binaries), clippy -D warnings, fmt clean

This commit is contained in:
glm-5.3-flash committed 2026-10-10 08:00:29 +00:00
1 parent 0c7744977c
commit 5c03fb2130
3 files changed
+100 -23

No files matched your search

+31 -14
View File
@@ -86,22 +86,39 @@ fn test_opts(schema: &str) -> PgOpts {
} }
} }
/// Wait for the next event with a bounded deadline — polling loosely /// Wait for the next event with a bounded deadline — parked on the
/// so no tight-race assertions ride (the suite's stability posture). /// receiver's `recv()` under the timeout wrapper (no polling loop:
/// the deadline still bounds the property, and the wave-4 F-1 flake's
/// poll-loop wake-subscription surface is out of the hot path). A
/// deadline miss self-diagnoses: the durable rows beyond the
/// receiver's position discriminate a wake/re-drain class (rows exist
/// undelivered) from a publish that never landed (none do).
async fn must_recv_event(receiver: &mut dyn EventReceiver, tag: &str) -> alkstore::StreamEvent { async fn must_recv_event(receiver: &mut dyn EventReceiver, tag: &str) -> alkstore::StreamEvent {
let deadline = tokio::time::Instant::now() + Duration::from_secs(15); match tokio::time::timeout(Duration::from_secs(15), receiver.recv()).await {
loop { Ok(Some(Ok(event))) => return event,
match receiver.try_recv() { Ok(Some(Err(e))) => panic!("{tag}: receiver errored instead of idling: {e}"),
Ok(Some(event)) => return event, Ok(None) => panic!("{tag}: receiver closed before the event arrived"),
Ok(None) => {} Err(_) => {}
Err(e) => panic!("{tag}: receiver errored instead of idling: {e}"),
}
assert!(
tokio::time::Instant::now() < deadline,
"{tag}: event never arrived"
);
tokio::time::sleep(Duration::from_millis(20)).await;
} }
let offset = receiver.offset();
let durable_beyond = match receiver.read_since(offset, 1000).await {
Ok(rows) => rows.len(),
Err(e) => panic!(
"{tag}: event never arrived after 15 s (receiver at offset {offset}; \
the diagnosis read also failed: {e})"
),
};
if durable_beyond > 0 {
panic!(
"{tag}: event never arrived after 15 s (receiver at offset {offset}; \
{durable_beyond} durable rows exist beyond it undelivered — \
wake/re-drain class)"
);
}
panic!(
"{tag}: event never arrived after 15 s (receiver at offset {offset}; \
no durable rows beyond it — publish-visibility class)"
);
} }
/// Kill this store's listener backend (the POC's /// Kill this store's listener backend (the POC's
+10 -3
View File
@@ -635,11 +635,18 @@ enum WakeTick {
Closed, Closed,
} }
async fn wait_wake(rx: &mut tokio::sync::broadcast::Receiver<RawNotification>) -> WakeTick { async fn wait_wake(
stream: &str,
rx: &mut tokio::sync::broadcast::Receiver<RawNotification>,
) -> WakeTick {
match rx.recv().await { match rx.recv().await {
Ok(_raw) => WakeTick::Ok, Ok(_raw) => WakeTick::Ok,
Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => { Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => {
// Still surfaced — treat like a wake: re-drain. // Still surfaced — treat like a wake: re-drain.
eprintln!(
"alkstore-postgres: stream bridge on {stream:?} lagged the wake feed, \
{n} notifications skipped (recovery is the re-drain)"
);
WakeTick::Ok WakeTick::Ok
} }
Err(tokio::sync::broadcast::error::RecvError::Closed) => WakeTick::Closed, Err(tokio::sync::broadcast::error::RecvError::Closed) => WakeTick::Closed,
@@ -727,7 +734,7 @@ async fn run_subscribe_loop(
// 3. Park on the wake feed (this stream's channel — and the // 3. Park on the wake feed (this stream's channel — and the
// reserved reconnect-wake: the same re-drain heals a // reserved reconnect-wake: the same re-drain heals a
// gap's missed wakes by reading the durable rows). // gap's missed wakes by reading the durable rows).
match wait_wake(&mut wake_rx).await { match wait_wake(&stream, &mut wake_rx).await {
WakeTick::Closed => return, WakeTick::Closed => return,
WakeTick::Ok => {} WakeTick::Ok => {}
} }
+59 -6
View File
@@ -1,7 +1,7 @@
--- ---
id: pg-suite-infra-hardening id: pg-suite-infra-hardening
name: Postgres test-infra hardening — F-1's defense-in-depth candidates name: Postgres test-infra hardening — F-1's defense-in-depth candidates
status: pending status: completed
depends_on: [] depends_on: []
scope: single scope: single
risk: low risk: low
@@ -39,12 +39,12 @@ adopted by default for anything not landed.
## Acceptance Criteria ## Acceptance Criteria
- [ ] `must_recv_event` parks on `recv().await` under a timeout - [x] `must_recv_event` parks on `recv().await` under a timeout
wrapper (no polling loop); the pg stream tests stay green wrapper (no polling loop); the pg stream tests stay green
against the harness server (a full stream-module run) against the harness server (a full stream-module run)
- [ ] Candidate (b) either landed small or recorded as skipped with - [x] Candidate (b) either landed small or recorded as skipped with
the reason the reason
- [ ] `cargo test -p alkstore-postgres` green (server-less skips - [x] `cargo test -p alkstore-postgres` green (server-less skips
clean); clippy `-D warnings`; fmt clean clean); clippy `-D warnings`; fmt clean
## References ## References
@@ -56,8 +56,61 @@ adopted by default for anything not landed.
## Notes ## Notes
> To be filled by implementation agent Decisions of record:
- **(a)** landed as specified: `must_recv_event` is now
`timeout(15 s, receiver.recv())` — parked, no polling loop. The
old helper's `try_recv`-loop `Err` arm split into the parked
shape's two failure arms: `Some(Err(e))` → the same
"receiver errored instead of idling" panic (an error during the
wait surfaces immediately, as before), and `None` → a new
"receiver closed before the event arrived" panic (the parked form
makes the terminal arm visible, the poll form never saw it).
`try_recv` is otherwise untouched as a test tool (other pg test
modules still drain with it).
- **(b)**: the `wait_wake`'s `Lagged(n)` arm now logs (eprintln,
house posture — matching notify.rs's `bridge_loop`; it also makes
forwarder.rs's "the receiver bridge's `Lagged` arm logs / recovers"
doc claim true at both bridges — that claim was half-wrong on the
stream side). The arm still treats lag as a wake (re-drain).
The stream name is threaded into `wait_wake`'s signature so the
log attributes lag to the stream, not anonymous.
- **(b)'s full bridge-side counter — skipped, recorded**: surfacing
the bridge's wake-observation count into the deadline panic would
require plumbing a counter Arc across the `dyn EventReceiver`
trait boundary (a test downcast or a keyed global registry) —
more machinery than the small-honest bar. The discriminator's job
is covered test-side instead: with the parked recv, a "wake arrived,
re-drain missed" arm is structurally implausible (the re-drain
reads exactly what a re-read reads; only a read error could miss,
and it surfaces `Err`), so the deadline panic self-diagnoses: it
reads the durable rows beyond the receiver's position via
`read_since(offset, 1000)` and names the residual class — rows
undelivered (wake/re-drain class; wait_wake's Lagged log speaks to
feed lag there) vs no rows (publish-visibility class). Test
harness only; no engine-state change.
- The deadline stays 15 s (the review's bound of record; unchanged
— only the wait shape moved from poll to park).
## Summary ## Summary
> To be filled on completion **Landed**: `must_recv_event` re-shaped from the 20 ms `try_recv`
polling loop to a parked `recv().await` under a 15 s timeout wrapper
(stream_tests.rs — the sole pg copy; the SQLite twin untouched, this
task is pg-specific), with a self-diagnosing deadline panic
(`read_since` presence check naming the failure class); the stream
bridge's `Lagged(n)` arm now logs the lag (`stream.rs` `wait_wake`,
stream name threaded in — the surfaced-not-silent posture now true
at all wake-feed bridges); the literal bridge-side diagnostic counter
skipped with reason above (the deadline diagnosis covers the
discrimination test-side).
**Verified**: the pg stream module 21/21 green against the harness
server (one full module run, ~19 s at `--test-threads=4`);
`tx_publishes_compose_with_the_handle` 5 solo re-runs green
(determinism re-check under the new parked shape); two consecutive
full pg harness runs green (121 lib + 25 contract-suite + 9 schema,
~72 s each); `cargo test -p alkstore-postgres` green server-less
(harness rows skip cleanly); workspace `cargo test` green (12 test
binaries); `cargo clippy --all-targets -- -D warnings` clean;
`cargo fmt --check` clean.