pg forwarder: the reconnect arm retries failed connects until success or shutdown — one failed connect no longer kills the loop permanently; every loop exit path releases the fanout sender via the shared-slot drop guard (receivers-terminal-iff-loop-gone); reconnect-config test seam + failed-connect pins: six-cycle exhausted-backoff server-less unit, pre-flip abort, guard drop, and the harness end-to-end unreachable-outage → full-recovery test (bug replay-proven against the old shape) (task pg-fix-forwarder-reconnect, review 002 Finding 1)

tasks/pg-fix-forwarder-reconnect.md

@@ -1,7 +1,7 @@
 ---
 id: pg-fix-forwarder-reconnect
 name: Fix forwarder permanent death after one failed reconnect (review 002 Finding 1)
-status: pending
+status: completed
 depends_on: []
 scope: moderate
 risk: medium
@@ -103,8 +103,92 @@ test-exercised, not reasoned-about.
 
 ## Notes
 
-> To be filled by implementation agent
+> Decisions of record the implementation made that the description
+> didn't pin:
+
+- **The retry core is an extracted helper (`reconnect_with_retry`)**
+  rather than an inline restructure: sleep → shutdown-select → attempt
+  → repeat, generic over an `FnMut() -> Future<Output =
+  Option<GenerationPair>>` attempt closure. It is only ever exited
+  with a fresh generation pair (`Some`) or at the shutdown flip
+  (`None`) — the loop top's empty-slot arm is restructured away
+  entirely: `LoopCtx.connection` is no longer `Option`; the loop
+  re-binds a plain loop-carried connection from the successful connect
+  at the arm's bottom, so there is no fall-through path left to die
+  on.
+- **The attempt closure is the connector injection** (the
+  implementer's-choice seam, one mechanism for both shapes): the real
+  closure clones `tokio_postgres::Config` out of a shared
+  `ReconnectConfigSlot` (`Arc<Mutex<Config>>`) and calls
+  `Config::connect(NoTls)`, mapping errors to `None`; the server-less
+  unit tests substitute their own always-failing closures. The slot is
+  also the config-swap seam: `Forwarder::swap_reconnect_config` /
+  `reconnect_config_snapshot`(`#[cfg(test)]`) let a test point
+  reconnects at an unreachable endpoint post-open and restore the
+  real one. The field is `#[cfg(test)]` on `Forwarder` (dead in prod
+  otherwise — the loop reads its own slot clone); the shutdown
+  watch's `changed()` is selected *with* the backoff sleep so a close
+  surfaces the terminal arm mid-outage instead of waiting out a cycle.
+- **The fanout hardening rides a shared slot + drop guard**: the
+  handle's sender became the shared `FanoutSlot` (`Arc<Mutex<Option<
+  Sender>>>`) the loop also receives, and the loop holds a
+  `FanoutRelease` guard whose `Drop` empties the slot — Drop runs on
+  return, unwind, and task abort alike, so *any* loop exit (plus
+  `Forwarder::shutdown`'s take, unchanged) releases the fanout.
+  `subscribe` reads the shared slot — a subscribe arriving past any
+  loop exit fails closed (`None`) rather than parking on a
+  forever-silent broadcast. receivers-terminal-iff-loop-gone is now
+  enforced by construction, not just the shutdown path.
+- **Backoff reset semantics unchanged**: the cap/growth live in the
+  helper (doubled per attempt, `saturating_mul(2)` → `min(2000)`);
+  the reset to the 50 ms base still fires on the *verified-alive*
+  generation (successful LISTEN/probe), so a connect that succeeds
+  but dies before its LISTEN reconnects at the carried backoff.
+- **The success path of the retry helper has no server-less unit pin**
+  (a real `(Client, Connection)` pair cannot be fabricated); it is
+  pinned end-to-end by the harness recovery test. The unit pins carry:
+  repeated failed connects across six backoff cycles with the cap,
+  pre-flipped shutdown aborts with zero attempts, and the guard's
+  drop-releases-slot mechanism.
+- `tokio` gained a `test-util` feature on the crate's
+  `[dev-dependencies]` (feature-unified into test builds only) for the
+  paused-time retry unit test — the gate battery runs deterministic
+  and instant.
+- The bug was replay-proofed: with the old one-shot `Err → return`
+  shape temporarily reintroduced, the new harness test
+  (`forwarder_survives_failed_reconnects_and_recovers_fully`) fails as
+  expected; with the fix it passes (verified both ways live).
 
 ## Summary
 
-> To be filled on completion
+> What landed, verified how:
+
+- **`alkstore-postgres/src/forwarder.rs`**: the forwarder loop
+  restructured — the reconnect arm now carries
+  `reconnect_with_retry` (sleep with live backoff → shutdown-arms →
+  connect attempt, repeating until success or shutdown; failed
+  connects never leave the arm), the loop-carried connection is a
+  plain non-`Option` rebinding (the empty-slot `take() else return`
+  that killed the loop permanently after one failed connect is gone by
+  construction), and the shared `FanoutSlot` + `FanoutRelease` guard
+  make every loop exit path release the fanout sender
+  (receivers-terminal-iff-loop-gone). `LoopCtx`/`Forwarder::spawn`
+  adjusted; the `cfg(test)` reconnect-config seam (swap/snapshot)
+  added; subscribe/shutdown doc comments updated for the invariant.
+- **`alkstore-postgres/src/store/open_tests.rs`** (4 new tests):
+  `forwarder_survives_failed_reconnects_and_recovers_fully` (harness:
+  config swapped to unreachable → backend kill → repeated failed
+  connects across ~4 backoff cycles mid-outage with the honest
+  transient `register` error pinned → config restored → reconnect-wake
+  + post-recovery NOTIFY delivery + working `listen()`), two
+  server-less paused-time unit pins of the retry core (six failed
+  attempts across six backoff cycles capped at 2 s, shutdown-abort
+  before any attempt), and the fanout release guard's drop mechanism.
+- **`alkstore-postgres/Cargo.toml`**: `tokio` `test-util` added to
+  `[dev-dependencies]` only.
+- **Gates**: `cargo test -p alkstore-postgres` with the harness server
+  green (115 lib + 10 contract-suite + 9 schema, run twice full);
+  every pre-existing forwarder/reconnect/no-replay/shutdown pin green
+  unchanged; workspace `cargo build`, `cargo test` (server-less — new
+  tests skip or run server-less cleanly), `cargo clippy --all-targets
+  -- -D warnings`, `cargo fmt --check` all green.