Wave-4 fix-batch decomposition: seven pg-fix tasks + review-wave-4-fixes gate from the general review (002) — forwarder reconnect retry (Finding 1, with the failed-connect test seam), tx pg_notify wakes retiring F-1's engine arm (Finding 2), max_size/DSN-options open path (Finding 3), stale-UNLISTEN generation drop (Finding 4), scheduler quarantine/retry (Finding 5), decode dedupe + cleanups, doc alignment; wave 5 gates on the two HIGH fixes landing

tasks/pg-fix-forwarder-reconnect.md

@@ -0,0 +1,110 @@
+---
+id: pg-fix-forwarder-reconnect
+name: Fix forwarder permanent death after one failed reconnect (review 002 Finding 1)
+status: pending
+depends_on: []
+scope: moderate
+risk: medium
+impact: component
+level: implementation
+tags: [wave-4-fixes, postgres-engine, forwarder]
+---
+
+## Description
+
+Fix the HIGH finding from the wave-4 general review
+(`docs/reviews/002-wave-4-general-review.md` §Finding 1, live-proven):
+the forwarder loop **dies permanently after one failed reconnect
+connect**. Site: `alkstore-postgres/src/forwarder.rs:498-509` →
+`:369`. On a failed `reconnect_config.connect(NoTls)` the loop's `Err`
+arm `continue`s — but the loop top is
+`let Some(poll_connection) = connection.take() else { return; }` and
+`connection` is still `None` (the old connection was consumed by
+`take()` when its generation started). One failed connect ⇒ the loop
+takes the `else` arm and the forwarder task exits forever. Consequences
+(all live-proven): every later `listen()` fails with a spurious
+"mid-reconnect" `Database` for the store's remaining lifetime; wake
+delivery stops permanently (no re-LISTEN, no reconnect-wake); stream
+subscribers parked on `wait_wake` stay parked forever (the fanout's
+senders stay alive — handle-held — so the broadcast never closes and
+the consumer never even sees a terminal arm), contradicting the
+streams row's pinned posture that *events never require polling to
+become visible* (core-contract.md streams section).
+
+**Fix shape** (the review's): retry the connect with the live backoff
+*inside* the reconnect arm — loop `sleep(backoff)` → `connect` until
+success or shutdown — never falling back into the `take()`'s
+`else return`. The loop top's `None` arm must become unreachable by
+construction (or restructured away entirely: e.g. carry the connection
+slot through the reconnect arm so a failed connect retries in place
+with the slot still empty but the loop alive).
+
+**Hardening (the review's second point)**: make every `forwarder_loop`
+exit path release the `Forwarder` struct's fanout sender (the
+shutdown-take machinery already exists — `Forwarder::shutdown` takes
+it) so *any* loop exit — not just the shutdown flip — surfaces the
+terminal `Closed`/`None` arm to subscribers instead of a
+forever-silence. Invariant to establish and pin: **receivers go
+terminal iff the forwarder loop is gone**. With the retry fix the loop
+only exits at shutdown, so this is defense-in-depth against the next
+abnormal-exit bug, not a behavior change.
+
+**Testability — the gap that hid the bug**: the gate's reconnect test
+(`open_tests.rs:244`) kills the backend via `pg_terminate_backend`
+while the server stays up — the 50 ms reconnect attempt connects
+successfully every time. The *unreachable-server window* (connect
+fails) is the one case the machinery exists for and the one case
+untested. The fix must come with a test that exercises it. Recommended
+shape: a `cfg(test)` seam on the reconnect config (the loop ctx's
+`reconnect_config` behind a swappable slot, or the connect step behind
+an injectable closure) so a test can point reconnects at an
+unreachable endpoint after `open`, force a reconnect (backend kill),
+verify the loop survives *repeated* failed connects across several
+backoff cycles, then swap the real config back and verify full
+recovery. Alternative: extract the connect step behind a connector
+trait/closure and unit-test the retry loop with a failing connector —
+implementer's choice, but the failed-connect arm must be
+test-exercised, not reasoned-about.
+
+## Acceptance Criteria
+
+- [ ] A failed reconnect connect no longer exits the loop: the
+      reconnect arm retries with the live backoff (50 ms → 2 s cap)
+      until success or shutdown — pinned by a test that forces
+      repeated failed connects (unreachable-endpoint seam or connector
+      injection) across multiple backoff cycles
+- [ ] After the server becomes reachable again: the synthetic
+      reconnect-wake broadcasts, post-reconnect `NOTIFY` delivers on
+      registered channels, and `listen()` succeeds (no spurious
+      "mid-reconnect" `Database`) — recovery pinned by test
+- [ ] Every `forwarder_loop` exit path releases the fanout sender —
+      an abnormal loop exit surfaces the terminal arm to subscribers
+      (the receivers-terminal-iff-loop-gone invariant; the shutdown
+      case's existing pin stays green)
+- [ ] The existing backend-kill reconnect tests
+      (`forwarder_reconnects_after_backend_kill_and_delivers`,
+      `receiver_stays_open_across_reconnects_closes_at_shutdown`, the
+      no-replay pin) stay green unchanged
+- [ ] `cargo test -p alkstore-postgres` (harness server), clippy
+      `-D warnings`, fmt clean; gates green server-less
+
+## References
+
+- docs/reviews/002-wave-4-general-review.md §Finding 1 (the finding,
+  the probe, the fix shape)
+- docs/architecture/decisions/004-postgres-driver.md (forwarder
+  decision, the hand-rolled posture's owned failure modes)
+- docs/architecture/decisions/006-wake-and-delivery-contract.md
+  (reconnect-recovery semantics)
+- docs/architecture/core-contract.md (streams: events never require
+  polling to become visible)
+- tasks/pg-engine-notify-listen.md (the forwarder's task of record;
+  Notes carry the shutdown sender-take decision this hardening builds on)
+
+## Notes
+
+> To be filled by implementation agent
+
+## Summary
+
+> To be filled on completion