alkblobs/tasks/review-wave-9.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
review-wave-9	Wave 9 review gate — the failure-path gate	completed	f1-arbitrate-conclude f2-cancellation-guards f3-local-durability f4-reap-declaration smells-n5-n7 window-family-smells test-isolation-harness d2-failure-path-tests	narrow	low	phase	review	wave-9 review review-001

Description

The wave-9 review gate (docs/plans/implementation.md §Review gates — the failure-path gate), before wave 10's security round decomposes. The wave's claim: the four correctness majors fixed with the failure/cancellation/contension tests that drive their previously-uncovered neighborhoods (D-2's correlation is the wave's working method), the neighborhood smells dispositioned, and the coverage residue absorbed. The gate audits exactly that claim.

Gate checks:

• F-1..F-4 green with their regression tests — each fix verified against its diff (not the task's claim): the arbitration conclude-on-every-path inspection, the arbitration wait/protected neighborhood driven, the cancellation-mid-tx checkout check, the local durability posture matching ADR-015's terms line by line (fsync scope held to the publish paths; no constructor knob), the reap declaration matching ADR-016's (refusal named, riding fixes landed, the hostile-schema framing shifted).
• ADR fidelity — OQ-12/OQ-13's resolutions (ADR-015/016) read against the landed behavior; any divergence flagged or the ADR amended through the OQ/ADR path (not silently).
• The smells' dispositions — N-5/N-7 landed as specced; the window batch's three (N-2/N-3/N-4) landed across all three hosts with the executor loop's semantics preserved; anything deferred named with its reason.
• The isolation hardening's acceptance — the full live suite green with default parallelism across repeated runs (the test-isolation-harness task's criterion); the wave's flake inventory (f2's Summary) resolved or consciously dispositioned.
• The coverage diff — a live-harness cargo llvm-cov --all-features run (ALKBLOBS_PG_* set — the honest configuration) diffed against review-001's D-2 uncovered list: the residue absorbed, no uncovered regression introduced; any consciously-recorded residue listed with its reason.
• review-001 stamped — F-1..F-4 and the landed N-items marked fixed on the review's findings (its file:line references predate the split; the stamp discipline from the wave-8 gate applies); the review's overall status disposition judged (remaining S-items keep it open for wave 10).
• Statuses updated — the plan's wave table row and the task files' statuses/summaries reflect what landed.
• Gates green (the AGENTS verification matrix): cargo test (default — the mem reference + local columns), cargo test --all-features server-less (clean skips), cargo test --all-features against the live harness (the pg acceptance), clippy --all-targets -- -D warnings across the feature variants (default, ops, all-features, the pg-pure shapes), cargo fmt --check, cargo doc --no-deps zero warnings on the default and ops builds.

Live harness

export ALKBLOBS_PG_HOST=localhost
export ALKBLOBS_PG_PORT=15432
export ALKBLOBS_PG_USER=postgres
export ALKBLOBS_PG_PASSWORD=poc
export ALKBLOBS_PG_DB=blobs

The local dockerized postgres container (pglo-poc, postgres:16-alpine on :15432) — running for dev/testing; the coverage run's honest configuration requires it.

Findings are fixed inline where cheap, or recorded as wave-10 amendments (the security round's decomposition reads them). The gate's completion unblocks wave 10's decomposition — which is gated additionally on OQ-14's resolution (already satisfied: ADR-017, 2026-10-15).

Acceptance Criteria

• Every gate check above verified against the diffs and the coverage artifacts (not from memory of the claims)
• ADR-015/016 fidelity verified line by line against the landed behavior
• The live-harness coverage diff recorded: residue absorbed, no uncovered regression, recorded residue justified
• review-001 stamped; the plan table updated
• All gates re-run green after any fixes
• Wave 10 decomposition unblocked (recorded in this task's Summary)

References

• docs/plans/implementation.md (§Wave 9; §Review gates — the failure-path gate)
• docs/reviews/001-storage-surface-general-review.md (F-1..F-4, N-2..N-7, D-2 — the audited items; S-1..S-7 are wave 10's)
• docs/architecture/decisions/015-local-engine-durability-posture.md, docs/architecture/decisions/016-one-large-object-consumer-per-database.md (the ADRs whose fidelity this gate audits)
• docs/architecture/open-questions.md (OQ-12/13 resolved; OQ-14/ADR-017 gating wave 10)
• tasks/f1-arbitrate-conclude.md, tasks/f2-cancellation-guards.md, tasks/f3-local-durability.md, tasks/f4-reap-declaration.md, tasks/smells-n5-n7.md, tasks/window-family-smells.md, tasks/d2-failure-path-tests.md (the wave's tasks)
• tasks/review-wave-8.md (the gate method this follows)

Notes

Completed 2026-10-16. Every check verified against the diffs and the coverage artifacts, not the task claims.

F-1..F-4 against their diffs — all four verified in the landed code: F-1's arbitrated_delete consult Err arm runs conclude_delete before propagating (src/store/sweep.rs:779-782), all four arms read (Cancelled / consult-refusal Ok(true) / consult-failure Err / engine-delete) — the doc states the posture; the five contended-pin/consult-failure tests exist in postgres_fleet_sweep_tests (commit 69dbca0's red-pre-fix/green-post-fix record). F-2: PgJointTx carries the Drop guard over the Option<Object> client (src/postgres/mod.rs:386-399, trigger conditions documented on the type), LeaseUnlockGuard defends both lease windows (src/postgres/pins.rs:429-436), the in-process begin_sweep_guarded/FleetSweepLeaseGuard realize N-6 — all aliased onto the shared pg_common::detached_batch_execute body (one body, three call sites, as claimed). F-3: fsync_publish (src/local.rs:390-418) is file-sync_all → hard_link(stage, committed) → stage unlink → dir fsync, exactly ADR-015's order; both publish sites use it (577, 654) with the exists() pre-checks gone; only two publish-site call sites exist (scope held); the unix pread_all errors on truncation (UnexpectedEof); no constructor knob (the probe/injection knobs are cfg(test)); the module doc carries the operator-facing contract verbatim. F-4: PgOpts.orphan_reap_permitted default false (src/pg_common.rs:241/262), the self-naming Error::Io refusal (src/pg_lo/reap.rs:48-53, no new variant — the family is ADR-pinned), the savepoint-guarded 42704 missing-LO pass with candidates re-validated at delete time (no REPEATABLE READ, per the ADR's chosen form), the lo_entries_loid_idx in bootstrap_ddl (idempotent), and the hostile-schema test/doc carrying ADR-016 §3's framing.

ADR fidelity (the gate's line-by-line read) — ADR-015: all four Decision bullets match the landed behavior; the fsync scope held to the publish paths (staging/delete/discard unfsynced); the ADR's hardlink-as-primitive, no-knob, and platform-note positions are literal in the code. ADR-016: the declaration + refusal + the three riding fixes + the §3 framing shift are all literal. No divergence found; no amendment needed.

Smells' dispositions — N-5: the debug-assert chosen (the review's close; the rationale and the no-new-variant point recorded on the task), test drives the foreign-Batch case. N-7: the wrapped-Error::Io homes report, both report arms driven (the gate added the kv-absent arm variant, below). N-2/N-3/N-4: landed across all three hosts with the executor loop's semantics preserved (WINDOW_TAKE_BATCH const in gc.rs, oldest-deadline-first, the drain loop; sqlite's one-statement select+remove form; forward-only GREATEST/max conflict updates) — test names verified in all three window modules plus the two backlog-drain store tests; the gc.rs seam doc restates the landed contracts. Nothing deferred.

Isolation hardening's acceptance — the full live suite green with default parallelism at the gate (319 + 125 + 4, pglo-poc), on top of the task's own 5x repeated-runs record (412 passed each, pre-D2); AGENTS.md carries the capacity posture (max_connections=200 with the saturation math) and the run-discipline note. The wave's flake inventory (f2's Summary: the paused-token ops test, pre-existing and load-sensitive, reproduced on the pre-change tree) is resolved — the isolation task's cadence-tolerance work plus the record of 5x no-flake full runs; not observed in any post-hardening run (including the gate's).

The coverage diff — live-harness cargo llvm-cov --all-features --summary-only (ALKBLOBS_PG_* set, --test-threads=2 per the coverage-run convention): 95.22% lines / 94.91% functions / 95.52% regions (25,456 lines; the review's honest baseline predates the split and wave-9's growth). Diffed against review-001's D-2 uncovered list via the lcov per-line report: every named finding region is now covered — gc.rs's Plan::Wait loop and protected break (98.75% lines; only three off-list minor arms missed), the arbitrate consult incl. its ROLLBACK arm, PgLoStaging's Drop body, ops/wire.rs and ops/specs.rs 100% (the malformed-payload sweep drives schema_validation_error), store/put.rs 89.8% and store/fleet.rs 92.0% lines (the failing-reader-through-fleet-put driven), local.rs 97.8% regions (the walk_committed fatal-malformed doctrine driven). The remaining missed lines match the tasks' consciously-recorded residue (defensive consumed-arm postgres/pins.rs:126; the real-media ROLLBACK arms postgres/pins.rs:142-164 per f2's notes; the structurally-dead and server-down shapes per d2's notes). No uncovered regression.

Consciously-recorded residue at the gate (production arms left uncovered, with reasons):

• src/store/sweep.rs:615 — the inline form's _ => cancelled_by_arbitration counter arm (sweep_window_inline, the mem/in-process host form): the durable executor's same counter is covered (the sqlite inv4 window-survival tests); the inline arm is drivable only by a mid-window pin race on a mem store — a race-shaped test the repo's determinism discipline avoids. The arbitration cancellation itself is driven on the fleet and durable paths by F-1's tests.
• src/store/sweep.rs:817 — delete_everywhere's kv-delete-failure short-circuit arm (propagates unwrapped — nothing moved): no kv delete-failure knob exists on the mem engine to drive it; the arm's propagating form is the trivially-correct ? passthrough.
• The cargo doc --no-deps warning the gate found (below) was fixed inline rather than recorded.

Inline findings fixed at the gate (2):

1. cargo doc --no-deps carried one warning on both the default and ops builds — Store::sweep's public doc linked the crate-internal WINDOW_TAKE_BATCH (introduced by the window batch, c839fcd); the link de-linked to code font. Zero warnings on both builds now (re-run verified).
2. The gate's coverage review found delete_everywhere's kv-absent homes-report arm (src/store/sweep.rs:827) undrivable by any existing test — a second N-7 test variant lands (delete_everywhere_partial_failure_reports_the_kv_absent_arm, a large-only entry + injected large-home delete failure; runs in the default build). The kv-failure short-circuit arm (817) stays recorded residue (no injection knob, trivial passthrough).

review-001 stamped — the frontmatter gains the dated wave-9 stamp; F-1..F-4 and D-2 carry fixed-stamps, N-2..N-7 carry fixed-parentheticals (each naming its commit). The review's overall status stays open — S-1..S-7 remain for wave 10.

Statuses updated — the plan's wave-9 row reads completed; the eight wave-9 task files are completed with Notes/Summaries (this gate's own file included).

Summary

Completed 2026-10-16 — the failure-path gate is green; wave 10's security round decomposition is unblocked.

The wave-9 claim audited and confirmed against the diffs and coverage artifacts: the four correctness majors are fixed with the tests that drive their previously-uncovered neighborhoods (all five tasks' regression rounds verified present and named in the code), the smells are dispositioned as specced (nothing deferred), the isolation hardening's criterion is met (default-parallelism full live runs green, repeatedly), and the coverage diff shows the D-2 residue absorbed with no uncovered regression (95.22% lines / 95.52% regions live-harness, the honest configuration; the finding regions all covered; residue consciously recorded). Two gate findings fixed inline: the Store::sweep doc's private-item link (rustdoc zero warnings restored) and the delete_everywhere kv-absent report arm's missing test (a second N-7 variant). ADR-015/016 fidelity holds line by line — no amendment needed; the ADRs' postures are readable on the landed code.

Gates re-run green at the gate (final numbers) — cargo test (default) 146 + 54 + 4 (the mem reference + local columns; +1 from the gate's own test); cargo test --all-features server-less 319 + 125 + 4 (pg skips clean); cargo test --all-features + live harness (pglo-poc, postgres:16-alpine :15432, default parallelism) 319 + 125 + 4, 0 failed; cargo clippy --all-targets -- -D warnings across the feature variants (default, ops, all-features; the pg-pure shapes postgres, postgres,pg-lo, mem,pg-lo); cargo fmt --check; the llvm-cov live-harness run per the coverage convention. Wave 9 → completed.

Wave 10 decomposition unblocked — its additional gate (OQ-14 resolving to an ADR) is satisfied since 2026-10-15 (ADR-017, the CAS global pool authority model). The decomposition reads review-001's remaining S-1..S-7 (still open on the review, stamped per-finding), this gate's recorded residue (the two inline-form/counted arms above — S-2's pump re-shape and the wave-10 security round do not own them, but the next residual-inventory read should), and the wave-8 gate's remaining recorded residuals (the no-kv feature shape; the all-features doc-link warnings — wave 12's docs pass owns those).