alkblobs/tasks/test-isolation-harness.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
test-isolation-harness	PG test-isolation hardening: pool-scoped session observables, harness-server capacity, and the wall-clock cadence tests' load tolerance	completed		moderate	medium	component	implementation	wave-9 test-infra flakiness postgres

Description

Wave 9's first full-suite runs (against the live pglo-poc harness) flaked intermittently across different pre-existing ttl/cadence tests (ops::tests::the_paused_fleet_token_expires_reaps_and_the_reput_ backstop_never_loses_silently, fleet_pin_row_renews_until_the_guard_drops_or_converts, fleet_batch_release_ends_the_members_renewals) and one zero-assert (fleet_kv_joint_commit_failure_leaves_neither_entry_nor_pin's "idle-in-transaction == 0"). The f2-cancellation-guards session diagnosed the vectors (and reproduced the ops flake on the pre-change tree — pre-existing, load-sensitive, not a code regression):

1. Cross-test session observables (the real isolation leak). Two idle_in_transaction_sessions probes (src/store/fleet.rs:831, :1648) count pg_stat_activity sessions scoped only by datname = $1 — the whole harness database. Every other concurrently-running test's transient idle-in-transaction session (the finger-lock test's ~300 ms held BEGIN; the ops token tests' waits) lands in another test's count. pg_locks-derived probes are schema-keyed and safe (advisory keys derive per schema), but any probe reading server-global catalog views by database name is cross-test by construction.
2. Harness-server capacity. One shared postgres:16-alpine on :15432, max_connections 100 (default), ~100 pg-heavy tests each constructing pools (max_size 8) — parallel full-suite runs pay connection-creation latency and occasionally saturate; slow-pool checkouts are the feedstock for vector 3.
3. Wall-clock cadence timing tests. The renewal-cadence tests assert behavior at tight horizons (SHORT_TTL 1200 ms, TTL/3 cadence ≈ 400 ms; ops' EXPIRY_HORIZON) scheduled on real wall time — under parallel/instrumentation load the renewal tasks starve and the assertions miss their windows. Not an isolation leak per se: a load-tolerance gap.

Vectors 2 and 3 compound: saturation stretches the cadence tests past their horizons.

Proposed shape (the design point, resolved at decomposition)

• Pool-scoped observables: attach a unique per-backend-instance tag to every pooled session — application_name wired through PgOpts — merged in build_pool's options like the existing synchronous_commit knob. Decomposition's choice: the prod-readable per-instance opt (a new PgOpts field defaulting to a stable crate-identifying name; tests pass a unique tag per instance), not a test-only suffix — the same connect-option pattern as synchronous_commit (ADR-007 §4's recorded posture), no prod/test divergence, and operators get the per-instance observability for free. No ADR: this is observability plumbing, not a contract decision (unlike the durability/reap declarations). The re-scoped probes filter application_name = $1 (the test's own backends' tag), making the zero-asserts exact instead of best-effort. The hazard to design against: a mis-scoped tag makes the zero-assert silently vacuous — worse than flaky — so the tag uniqueness across concurrently-running tests (each test instance's tag distinct) needs an explicit test of its own, and the re-scoped probe must be shown to still see real residue (a red/green check: with a deliberately stranded session in the test's own pool, the assert fires).
• Capacity: raise the harness server's ceiling (e.g. max_connections: 200 — a container-env change on pglo-poc, recorded where the harness is documented) or lower per-test pool sizes in the tests-only constructors; pick one, record it.
• Cadence tolerance: widen the timing tests' horizons proportional to their sensitivity (or mark the tightest ones' acceptance as "isolation-gated": green when the suite runs at modest parallelism, documented alongside the coverage run's already-recorded --test-threads=2 workaround). The goal is not zero timing assertions — it is that their windows survive a loaded, but unsaturated, server.

Sequencing

Before d2-failure-path-tests (the wave's largest new-pg-tests round) — that task multiplies both vectors otherwise. Independent of the code tasks; no ADR pressure; the application_name opt (resolved above: the prod-readable per-instance field) is the only surface touch.

Acceptance Criteria

• The pg_stat_activity probes are scoped to the test's own sessions (application_name or equivalent), and a deliberately concurrent noisy test cannot poison the zero-assert
• The re-scoped probe red/green-verified: a deliberately stranded session in the test's own pool still fires the assert (no vacuous scoping); concurrent test instances' tags cannot collide
• The harness capacity posture recorded (container config or test pool sizing), with the saturation math checked once
• The full --all-features live suite green with default parallelism across repeated runs (e.g. 5x), cadence tests stable under the loaded-but-unsaturated server
• The run-discipline note (when --test-threads reduction is the honest acceptance posture, vs the fixed suite) updated in AGENTS.md's verification block if needed

References

• Observed during f2-cancellation-guards (2026-10-15): the flake inventory and the baseline reproduction are recorded in tasks/f2-cancellation-guards.md's Summary
• src/store/fleet.rs:831/:1648 (the datname-scoped probes)
• src/pg_common.rs build_pool (the options-merge point the application_name wiring rides)
• AGENTS.md §Verification Commands (the pg columns' acceptance shape)

Notes

Implementation record (2026-10-15):

• The tag surface (src/pg_common.rs): PgOpts gained application_name: String, defaulting to "alkblobs" — the prod-readable per-instance opt. build_pool now takes &PgOpts (the three loose arguments folded into it) and wires the tag through tokio_postgres::Config::application_name — the startup parameter, so it survives the pool's Fast recycle verbatim (session-state resets never touch it), stronger than an options= -c merge for this particular knob (the synchronous_commit options-merge stays as-is). postgres/pool.rs, pg_lo/mod.rs, and both engines' new_schemaless fixtures ride the new signature. Two wiring-proof tests (postgres/pool.rs): opens_with_the_default_crate_tag, application_name_tag_wires_per_session (via the new shown_application_name fixture probe).
• The probes (src/store/fleet.rs, both test modules): filter datname = $1 AND application_name = $2 — $2 is the test's own tag. In the kv module the tag is the test's fresh schema name (TestSite::session_tag); in the pg-lo module the dedicated database's name — each unique per test instance by construction (pid + atomic sequence). Every store construction in those modules passes the tag, so the whole test family's pools are identifiable. Ops stores and other modules keep the default alkblobs tag — harmless: the probes filter on the test's unique tag, so any other test's sessions (default-tagged or otherwise) are invisible.
• Red/green + uniqueness (the_session_tag_scopes_the_dirty_ session_probe_exactly): two fresh sites asserts the tags distinct (uniqueness act); a foreign-tagged BEGIN session (the noise vector materialized) leaves the probe at zero — noise cannot poison it; a same-tag stranded session fires (count 1 — never vacuous); ROLLBACK restores zero. The stranded sessions are raw connections carrying the test's own tag — observationally identical for a pg_stat_activity reader to a dirty pooled session (the tag is what scopes, and the probe reads only datname/application_name /state).
• Harness capacity: the container recreated with postgres -c max_connections=200 (same volume, port, image; the data — roles, blobs db — unaffected). Saturation math checked: 8 test threads x up to 2 pools/test x max_size 8 worst case ~128 sessions + per-test raw connections, over the stock 100, under 200 with headroom. Recorded in AGENTS.md's verification block (the posture paragraph).
• Cadence load tolerance:
  • kv fleet tests (store/fleet.rs): TEST_TTL 2 s → 4 s (cadence ≈1.3 s), ONE_CADENCE/BEYOND_CADENCE replaced by RENEW_WINDOW (3 s poll deadline for a renewal to land, 3x the nominal cadence and inside the widened TTL's life), POLL_STEP (100 ms), QUIET_WINDOW (1.8 s, > one cadence, for the frozen-row/never-resurrect arms). The positive cadence arms poll until the expiry moves instead of sampling once at a fixed wall time (fleet_pin_row_renews…, fleet_batch_release…, and the pg-lo consolidation's renewal arm) — a starved-but-landed tick is still seen; a missed tick is a real failure. The negative arms keep fixed sleeps (a stop can only be proven by absence).
  • ops tests (ops/tests.rs): PAIR_TTL (new const, 900 ms → 2 s, riding open_fleet_pair and the rerouting test) and EXPIRY_HORIZON 1400 ms → 3 s. The paused-token test's expiry arm polls the row instrument until the expiry is past (the have probe itself is the renewal form — every present: true answer feeds the row; polling it would never expire) with a 10 s deadline, then asserts the expired answer once; the re-put backstop's renewal loop is deadline-based (1.5 s budget < the widened TTL) instead of 5 x 100 ms. The registered-source and rerouting expiry arms ride EXPIRY_HORIZON.
• AGENTS.md: the harness capacity posture paragraph and the run-discipline paragraph (default parallelism is the acceptance; --test-threads=2 remains the coverage-run workaround).

Summary

The isolation leak is closed: both pg_stat_activity dirty-session probes scope by the test's own application_name tag (wired as a prod-readable PgOpts opt), their zero-asserts are exact, and the red/green test proves the scoping both ways (foreign noise ignored, own residue detected) with tag uniqueness asserted explicitly. The harness runs at max_connections=200 (posture + saturation math recorded in AGENTS.md). The cadence tests are load-tolerant: positive arms poll within bounded windows (3 s / 1.5 s deadlines sized against the widened TTLs 4 s / PAIR_TTL 2 s), negative arms keep fixed quiet-widow sleeps, and the ops paused-token arm polls the non-renewing row instrument (not the renewing have probe).

Verification (2026-10-15): cargo test (default) 186 passed; cargo test --all-features live-harness 5x full runs, all green with default parallelism (412 passed each: 283 + 125 + 4 + doctests 0) — no flake observed in any run, cadence tests stable; a sixth full run (pre-capacity-change, harness at 100) also green. cargo clippy --all-targets -- -D warnings and cargo clippy --all-features --all-targets -- -D warnings clean; cargo fmt --check clean; cargo doc --no-deps zero warnings.