alkstore/tasks/pg-fix-forwarder-hardening.md

---
id: pg-fix-forwarder-hardening
name: Forwarder TOCTOU restructure + hang-shaped arms (review 004 items 4+6)
status: completed
depends_on: []
scope: broad
risk: high
impact: component
level: implementation
tags: [review-004-fixes, postgres-engine, forwarder]
---

## Description

Forwarder surgery that should ride its own session, per
`docs/reviews/004-pre-release-general-review.md` §"Findings recorded,
not fixed" items 4 and 6 (the review bundles them as the
forwarder-hardening session). Line refs are as of the review; the
inline fixes may have shifted them slightly — re-locate on pickup.

**(a) Register/unregister TOCTOU (item 4).** The registry refcount and
the LISTEN/UNLISTEN command emission are not one critical section
(`forwarder.rs:131-176` vs `438-482`): interleaved register/unregister
across tasks can leave a live refcount with the server-side LISTEN
cleared (both queued commands share a generation, so no stale check
saves it) until the next reconnect re-issue. Fix shape: a
decide-and-queue restructure inside the forwarder — the
register/unregister decision and the derived command enqueue must be
atomic with respect to each other (decide under the registry guard;
queue the command *with* the refcount state it was derived from).

**(b) Generation-verify/LISTEN failure parks forever (item 6, arm 1).**
A failed generation-verify/LISTEN batch is treated as connection death
and parks on `poll_task.await` forever even though the session may be
alive (`forwarder.rs:698-702/803-808`).

**(c) Shutdown can leak the listener session (item 6, arm 2).**
`close()`/`Drop` can leak the listener session past shutdown when the
serve loop is awaiting a command on a black-holed connection (the
`run_command` await sits outside the shutdown `select`, `:781`; no
timeout on this connection).

Treatment family for (b)+(c): timeouts/bounded waits on the listener
path.

**Preserve while restructuring** (the gate will check these): the two
POC-pinned deadlock pitfalls stay structurally excluded; the reconnect
invariants from `pg-fix-forwarder-reconnect` (the retry core, the
fanout release guard, receivers-terminal-iff-loop-gone) stay green;
the register-failure unregister fix from review 004 (inline fix 3)
stays; the poison-recovery lock idiom from review 004 (inline fix 4)
stays.

Use the existing test-seam vocabulary (the reconnect-config slot, the
connector injection, the generation tests) for the new pins.

## Acceptance Criteria

- [x] The TOCTOU is closed: register/unregister interleavings across
      tasks cannot leave a live refcount with the server-side LISTEN
      cleared — pinned by a test that interleaves the two and asserts
      the server-side LISTEN state matches the refcount
- [x] A failed generation-verify/LISTEN batch no longer parks forever:
      bounded wait with an honest error or reconnect — pinned by test
- [x] `close()`/`Drop` reclaims the listener session even when the
      serve loop is stuck awaiting on a dead connection — pinned by
      test (black-holed connection seam)
- [x] The preserved postures listed above re-verified green (existing
      forwarder pins: reconnect survival, no-replay, shutdown close,
      stale-UNLISTEN generation tests, the register-failure
      unregister pin)
- [x] `cargo test -p alkstore-postgres` (harness server), clippy
      `-D warnings`, fmt clean; workspace gates green server-less

## References

- docs/reviews/004-pre-release-general-review.md §"Findings recorded,
  not fixed" items 4 and 6 (the findings, the fix shapes, the
  own-session scoping)
- docs/architecture/decisions/004-postgres-driver.md (the forwarder's
  owned failure modes, the two POC-pinned deadlock pitfalls)
- tasks/pg-fix-forwarder-reconnect.md (the retry core, the fanout
  guard, the test-seam vocabulary)
- tasks/pg-fix-stale-unlisten.md (the generation/staleness machinery
  the TOCTOU restructure must compose with)

## Notes

> Decisions of record the implementation made that the description
> didn't pin:

- **The real interleaving window was thread preemption, not a yield.**
  Locating the TOCTOU on pickup: the old `register` (add → connected
  check → send) and `unregister` (remove → send) each had no `await`
  between their decide and their send, so the split is not an
  interleave point under tokio's cooperative scheduling *within one
  thread* — but the multi-thread runtime's worker threads preempt
  mid-unit, so thread-A's decide can land, thread-B's whole unit runs
  (decide + send), and thread-A resumes and sends its stale-contrary
  command. That is the review item 4's shape, and it quiesces
  persistently: registry count 1 with the queue `[L, U]` (UNLISTEN
  replayed after the live registration's LISTEN). The fix makes
  decide+send one critical section under the registry guard, so the
  queue order is the registry transitions' linearization.
- **Every registration keeps its own (redundant) LISTEN + ack.** A
  shape that skipped the redundant LISTEN for shared (non-new)
  refcounts was considered and rejected: it would let a later
  `listen()` return `Ok` while the *first* registration's LISTEN still
  sat queued — breaking the "a `listen()` that returned sees every
  later commit / starts from now" contract for that caller. The
  redundant per-registration LISTEN is the ack's vehicle; semantics
  preserved exactly (documented in `enqueue_register`).
- **The decide-and-queue lives on `ChannelSet`** (`enqueue_register` /
  `enqueue_unregister`), not on `Forwarder`: the registry guard hosts
  the critical section; the `connected` read and the generation-tag
  load ride the same section; the ack receiver is returned so the
  caller awaits *outside* the guard (never a std lock across an
  await). The old standalone `add_subscriber`/`remove_subscriber`
  were folded into the under-guard decisions and deleted — the
  refcount unit pin (`channel_set_refcounts_last_subscriber_rule`) was
  reworked to drive the decide-and-queue API (now it also pins that
  each registration derives its own LISTEN, and the mid-reconnect
  Err-with-entry-stays posture).
- **One bounded-wait primitive for the whole treatment family**
  (`bounded_wait`: output / bound expiry / shutdown flip, `biased`
  toward shutdown): the verify batch, the replay commands, the serve
  commands, and the failed-verify settle all ride it. Constants:
  `LISTENER_QUERY_TIMEOUT` (5 s) and `LISTEN_SETTLE_TIMEOUT` (2 s) —
  generous against a merely-slow server (an overrun re-issues from
  the snapshot on reconnect; nothing is lost either way; the settle
  sits far above real death-detection latency, which surfaces in
  `poll_message` at once).
- **Death postures, two shapes.** (1) A verify overrun/failure runs
  the bounded settle: the poll task (the reaper) gets
  `LISTEN_SETTLE_TIMEOUT`; an unsettled reaper is *aborted* — the
  connection closes with it (an aborted settle's socket close absorbs
  the reaper's job). (2) A serve/replay command overrun is treated as
  honest connection death *directly* (abort + fall to the reconnect),
  not via a second settle wait — the unresponsive connection is
  already proven; the reaper is aborted immediately and the
  reconnect arm proceeds. Every bounded await selects the shutdown
  watch alongside (`biased`), so `close()` surfaces through any stuck
  listener-path await — the review item 6 arm 2's hostage shape is
  closed at *all* of its await sites (the review cited only the serve
  arm's `run_command`; the verify batch and the generation-top replay
  had the same shape and got the same treatment family).
- **`first_listen` flip moved to the verify's success** (before the
  drain/replay): a generation that dies mid-replay keeps the wake for
  the *next* generation's top (a replay-death generation never
  broadcasts its own wake). The existing exactly-one-wake pin
  (LISTEN-window death) stays green — re-derived against the new
  order.
- **(c)'s live harness shape is a TCP black-hole proxy seam** (the
  task's "black-holed connection seam"): a local pass-through TCP
  proxy in front of the harness server (the store's DSN points at
  it); engagement eats server→client bytes (reads hang forever, both
  TCP ends up — the one failure shape backend kills cannot reach:
  alive, attached, unresponsive). An eaten-bytes counter pins that
  the serve loop is demonstrably parked inside the UNLISTEN await
  before `close()`, and the reclaim is measured server-side by the
  store's exact `application_name` vanishing from `pg_stat_activity`
  (the proxy propagates the client's close to the backend). No
  fabricatable alternatives exist for the connection types (the
  Client/Connection pair cannot be faked), and OS-level freezes
  (SIGSTOP) cannot reach dockerized harness backends — the network
  layer is the testable seam. `LISTENER_QUERY_TIMEOUT`,
  `LISTEN_SETTLE_TIMEOUT`, `bounded_wait`, and `BoundedOutcome` are
  `pub(crate)`.
- **(b)'s harness-level live arrangement is not separately added** —
  the stale-unlisten precedent's bar applies ("the unit pin is the
  acceptance bar"): the failed-verify settle is pinned server-less in
  paused time (a wedged reaper bounded + aborted + cancelled, a
  promptly-dying reaper reaped, a close surfacing through the settle
  promptly), and the genuinely-dead-verify path through the real loop
  stays covered by the existing LISTEN-window-death harness test
  (`forwarder_recovers_from_connection_death_during_the_listen_window`),
  which reaps under the bound and recovers unchanged.
- The stale-generation machinery (tags, counter bump, the
  generation-top drain, the recv-arm staleness guard), the retry
  core, the fanout release guard, the register-failure unregister
  discipline, and the poison-recovery lock idiom are all preserved —
  re-verified green (existing pins' full battery in Gates).
- **Replay-proofs (both ways, verified live):**
  (1) with the old decide/send split temporarily reintroduced in
  `enqueue_unregister`/`enqueue_register` (a widened post-decision
  window), the interleave pin fails on exactly the review's assert
  (registry/model divergence — a live refcount at a cleared
  server-side LISTEN); with the fix restored it passes instantly and
  deterministically (the finale anchors on the observable decision —
  the entry vanishing — so a commit-time run never flirts with the
  timing).
  (2) with the serve arm's bounded await temporarily reverted to the
  old unbounded `run_command` await, the black-hole harness test
  fails on "close must reclaim the listener session…" (the session
  leaked past close, as reviewed); with the fix restored it reclaims
  (2.4 s wall, well inside the 5 s command bound + the 15 s
  observation window).

## Summary

> What landed, verified how:

- **`alkstore-postgres/src/forwarder.rs`**:
  (a) the TOCTOU — `ChannelSet` gained the decide-and-queue critical
  sections (`enqueue_register`: refcount + connected-check + acked
  LISTEN enqueue under one registry guard; `enqueue_unregister`:
  last-subscriber decision + UNLISTEN enqueue under one guard);
  `Forwarder::register`/`unregister` ride them (the ack await is
  outside the guard); the standalone refcount helpers folded away;
  the module/method docs state the invariant (the queue's order is
  the registry transitions' linearization; a live refcount can never
  sit against a cleared server-side LISTEN).
  (b) the failed-verify settle — the old unbounded
  `poll_task.await` is replaced by the bounded settle
  (`LISTEN_SETTLE_TIMEOUT`=2 s, shutdown-selected): reaped fast when
  the session is genuinely dead, aborted at the bound (the connection
  closes with the reaper) when it is alive-but-unusable, then the
  reconnect either way.
  (c) the close-time leak — a new `bounded_wait` primitive bounds
  every listener-client query (5 s) with the shutdown watch selected
  alongside: the verify batch, the generation-top replay commands,
  and the serve loop's LISTEN/UNLISTEN commands; an overrun is the
  honest death posture (abort + reconnect; the snapshot re-issue
  recovers, the stalled LISTEN's caller sees the transient error
  family), so `close()`/`Drop` can no longer leak the listener
  session through a stuck await.
- **Pins added** (`open_tests.rs`, 4 new; `notify_tests.rs`, 1
  reworked):
  `interleaved_register_unregister_keep_server_listen_set_matching_registry`
  (server-less: a server-model task re-plays the command queue in
  FIFO order onto a LISTEN model, two interleaving tasks hammer
  register/unregister on two channels, a per-channel deterministic
  finale races a last-subscriber drop against one re-register with
  the decision anchored on the registry snapshot — the final
  registry/model equality is the pin);
  `bounded_wait_done_overran_and_shutdown_arms` and
  `failed_verify_settle_bounds_aborts_and_surfaces_shutdown`
  (paused-time, the settle/query bounds' arms incl. prompt close);
  `close_reclaims_the_listener_session_through_a_black_holed_connection`
  (harness, the TCP black-hole proxy seam: eaten-response proof the
  serve arm is parked, close reclaims inside a bounded window, the
  session stays reclaimed, post-close ops fail closed, subscribe past
  the loop exit fails closed);
  `channel_set_refcounts_last_subscriber_rule` reworked onto the
  decide-and-queue API (last-subscriber rule, redundant-LISTEN ack
  vehicle, past-zero no-op, mid-reconnect Err + entry-stays + nothing
  enqueued).
- **`Gates`**: `cargo test -p alkstore_postgres` with the harness
  server green (131 lib + 25 contract-suite + 9 schema; lib battery
  run twice full), every pre-existing forwarder/reconnect/no-replay/
  shutdown/stale-UNLISTEN/LISTEN-window-death pin green unchanged;
  workspace `cargo build`, server-less `cargo test --workspace`,
  `cargo clippy --all-targets -- -D warnings` (workspace),
  `cargo fmt --check`
  — all green; the doc gate (`RUSTDOCFLAGS="-D warnings" cargo doc
  --no-deps`) green.