alkblobs/tasks/d2-failure-path-tests.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
d2-failure-path-tests	D-2's test round — the op handlers' malformed-payload sweep, the failing-reader-through-fleet-put, the local error family	completed	f3-local-durability test-isolation-harness	moderate	low	component	implementation	wave-9 review-001 correctness tests coverage

Description

The D-2(a) remainder the plan names for wave 9 (docs/reviews/ 001-storage-surface-general-review.md D-2 — the uncovered regions are exactly the finding regions; the fixes' own regression tests ride F-1/F-2/F-3/F-4, this task carries the rest of the named list). Three regions:

Sequencing note: this task depends on tasks/test-isolation-harness.md — the wave's largest new-pg-tests round must not be built on the flaky ground the f2 session diagnosed (the cross-test session observables, the harness capacity, the cadence windows); landing the isolation hardening first keeps this round's red/green signal honest.

1. The op handlers' error arms — the malformed-payload sweep over the five ops (blobs/stat, blobs/have, blobs/delete, blobs/fetch, blobs/put): no test has ever sent a malformed payload to a single op — schema_validation_error is 0%-covered. Drive each op with malformed/invalid payloads (wrong types, missing fields, over-bound values where the schema bounds exist) through the in-process call harness (src/ops/harness.rs) and assert the wire-error shape. This region is server-less-drivable (validation precedes store contact) — the sweep should run in the default/ops builds, not only under the live harness.
2. The failing-reader-through-fleet-put: the existing Unreadable/Failing test doubles (used in the non-fleet tests only) driven through the fleet put path — the put paths' mid-read error arms (stream failure, under-read derivation mismatch) and the fleet put/commit error arms. Live-harness shape (kv=postgres + large=pg-lo).
3. The local engine's error family: walk_committed's fatal-malformed doctrine actually driven (a malformed committed tree — bad names, wrong shapes — fails loudly; the doctrine is implemented per spec and compile-checked only), plus the commit-consumed and discard arms and the blocking joint-failure arm as drivable.

Read first: the wave-8 review's inline finding 1 (tasks/review-wave-8.md — the fleet-renewal e2e flake): the renewal outcome folds closed on any error by design (RenewalOutcome::Expired on Err(_) — ops/tokens.rs's decided composition, reviewed-correct backdrop for this error-arm work), and its test-robustness fix (the re-ask-within-cadence pattern) is the reference for flake-sensitive error-arm asserts under the shared harness server.

Residue discipline: the wave gate's coverage diff (review-wave-9) runs after this task; if driving these regions surfaces adjacent uncovered arms beyond the named three, absorb where cheap and record the rest consciously — the gate judges "the residue absorbed, no uncovered regression introduced".

Live harness

Region 2 (and any fleet-fixture region 1 cases) run against the local dockerized postgres container (pglo-poc, postgres:16-alpine on :15432, running for dev/testing):

export ALKBLOBS_PG_HOST=localhost
export ALKBLOBS_PG_PORT=15432
export ALKBLOBS_PG_USER=postgres
export ALKBLOBS_PG_PASSWORD=poc
export ALKBLOBS_PG_DB=blobs

Server-less, the pg tests skip cleanly; acceptance requires green against the live harness. Regions 1 (non-fleet cases) and 3 are server-less.

Acceptance Criteria

• The malformed-payload sweep over the five ops lands; schema_validation_error and the ingress rejection arms are driven (server-less where the shape allows)
• The failing-reader-through-fleet-put test lands; the fleet put/commit mid-read error arms are driven
• The local error family lands: walk_committed's fatal-malformed doctrine driven; the commit-consumed/discard and blocking joint-failure arms driven as drivable
• No uncovered regression introduced; adjacent residue absorbed or consciously recorded
• Gates green: cargo test (default), cargo test --all-features server-less, cargo test --all-features + live harness, clippy --all-targets -- -D warnings across the feature variants, cargo fmt --check

References

• docs/reviews/001-storage-surface-general-review.md (D-2 — the uncovered-set enumeration this round absorbs; the ops error-arm list in the review's pre-split coordinates maps to src/ops/*)
• docs/plans/implementation.md (§Wave 9 — "D-2's test round")
• tasks/review-wave-8.md (inline finding 1 — the renewal fold's closed-on-error posture and the re-ask test pattern)
• src/ops/harness.rs, src/ops/specs.rs, src/ops/wire.rs (the op entry, validation, and wire-error synthesis)
• src/store/fleet.rs, src/store/put.rs (the fleet put path and its error arms; the Unreadable/Failing doubles' current homes)
• src/local.rs (walk_committed and the error family)

Notes

The round's tests (by region): region 1 in src/ops/tests.rs (the wire schema-gate sweep over all five ops incl. the composed internal delete, the post-schema handler-ingress sweep via bare_context invocation, the establisher ingest guards driven directly, the fetch pump's three silent-return arms plus the per-path mid-service write-failure forms via a one-shot PumpSource stand-in, the put's over-fed VERIFICATION_REFUSED settle through put_failed_response, the fanout begin-None/dead-arm-prune unit form, the ranged-absent typed miss, and the store-error family through a chmod 000-broken local shard: stat/delete INTERNAL + fetch establishment refusals — mem-gated, unix-gated); region 2 in src/store/fleet.rs (the failing-reader five forms through the fleet pg-lo put with zero- residue probes, the kv pin_commit failure through the pg-lo joint publish, and the shared-media pin-only-commit failure incl. the unknown-length put's own fleet staged-commit dispatch); region 3 in src/local.rs (the walk_committed fatal-malformed doctrine over every malformed shape, the blocking joint failure, the short-key rejection, the append-failure discard, the hard-link other-failure + whole-put temp cleanup, and the get/delete/size non-NotFound arms — permission and directory-craft forms).

Two small test-support additions rode along: OpsHarness's CALL_STREAMS 8 → 64 (the malformed sweep makes dozens of calls over one assembled connection) and PostgresBackend::clear_fail_joint_commit (the failure tests' disarm for the follow-up-put pool-health probes, mirroring the other engines' knob family).

Wave-8's inline finding 1 shaped the live-harness asserts: the re-ask-within-cadence pattern was reused only where an error fold could swallow a transient pool error — nothing new folds here; the round's zero-residue probes run once per failure form (each form is its own tx; no cadence races involved).

Summary

All three regions landed and green under every gate: cargo test (default), cargo test --all-features server-less, cargo test --all-features + live harness (319+125+4 passing), clippy --all-targets -- -D warnings across both feature variants, cargo fmt --check. Live-harness coverage (the honest number) 94.70% → 95.52%; the finding-region files: ops/wire.rs and ops/specs.rs 100%, ops/fetch.rs 99.2%, ops/put.rs 90.6% (from 81.9), store/put.rs 89.8% (from 86.5), store/fleet.rs 92.0% (from 91.4), local.rs 97.8% (from 95.0). No uncovered regression introduced (the fleet.rs delta is this round's own test-side fallback arms).

Conscious residue (production arms left uncovered, with their shapes): structurally-dead defensive arms — the no-host/no-threshold arms in the fleet put drivers (fleet_host's sqlite/mem arms, the kv-large pairing is predicate-structural), fleet_pin_row's clock overflow, put.rs's no-large-tier arms (Tier::Large implies DualTier), the feed-after-eof dead control flow, the broadcast-to- ended arm (dead by the publisher lifecycle), the ops mint-refusal arms (each structurally excluded by the fleet_owner() pairing); server-down/transient shapes — the joint-tx begin Err arms, the pin-renewal Err(_) => {} fold (wave-8's reviewed-correct closed degradation), the dispatch accept/WireFailed torn-transport arms (both duplex and mpsc teardown resolve to clean EOF); engine knobs' scope — joint_put's own failure arm, the pg-lo drivers' mid-staged append failure and the staged-artifact under-read arms (the pg-lo engine has no staging-append injection; the local tier's knob drove the store-put.rs equivalents), local.rs's real staging-write and pread-Interrupted arms, put.rs's flush-phase put_staged failure, and the put's loop-top PutEnded ordering (the driven ordering is the feed-side arm; which detection wins is timing-dependent).