alkstore/docs/research/poc-pg-posture-spec.md

---
status: findings-filed
title: "POC #2 — Postgres engine posture: LISTEN/NOTIFY wiring, the pool tx-seam, and reactive parity on tokio-postgres"
last_updated: 2026-10-04
---

# POC: Postgres engine posture — spec

> **POC register #2** (phase-0.md). Feeds OQ-ST-03's Postgres half (the
> driver question's remaining open part) and OQ-ST-04's Postgres wake
> side (the reactive contract). Code: standalone crate in the global
> workspace (`/workspace/alkstore-pg-posture-poc`; harness conventions
> cloned from `/workspace/alkstore-sqlite-posture-poc`). Findings land
> in `docs/research/poc-pg-posture-findings.md` here regardless.
> Server: dockerized postgres (POC-harness convention from the alkblobs
> POCs — `postgres:16-alpine` on :15432, config knobs stated per-probe).
> Pre-spec research input (2026-10-04): the dedicated-LISTEN-connection
> requirement (verified against deadpool-rs/deadpool#360, open upstream
> issue — pooled connections discard async notifications) and the
> `postgres-notify` 0.3.8 candidate (verified: reconnection + multi-
> channel subscribe) — both folded in below; neither's *behavioral*
> claims are trusted until probed.
>
> **Ran 2026-10-04: PASS** — findings in
> [poc-pg-posture-findings.md](poc-pg-posture-findings.md). Both
> pre-spec research claims verified in-probe and pinned as tests
> (#360: pooled LISTEN registers but never delivers; postgres-notify:
> reconnect + re-LISTEN verified *with* recorded quirks — lazy
> reconnect, no script at initial connect, unquoted identifiers).
> The spec's decision gate passed on all three conditions; OQ-ST-03
> closed; the sqlx PgListener fallback retired unfired.

## What this POC must decide

OQ-ST-03's SQLite half is resolved (POC #1: honker-core on our
rusqlite). The Postgres half asks the structural twin of the same
question, with three differences that make it *not* a rerun:

1. **The queue machinery is greenfield either way.** pgboss-rs (the
   schema-family port) has no LISTEN/NOTIFY (verified 2026-10-03) and
   is sqlx-coupled; under OQ-ST-02's per-engine-crate split it brings
   no driver synergy. So the engine crate writes queue tables + claim/
   ack + retry/dead-letter itself on *some* driver — this POC grounds
   that build on tokio-postgres (the alkblobs POC #5/#7-validated
   stack: deadpool-postgres pool, per-connection prepared-statement
   discipline, `synchronous_commit` as the durability knob).
2. **The wake mechanism is native server push** (LISTEN/NOTIFY), structurally
   different from SQLite's `data_version` polling: connection-bound,
   delivered on a dedicated connection, no retry/visibility semantics.
   The reactive contract's honest-shape question (OQ-ST-04) needs
   measured ground: what LISTEN plumbing costs, how it fails, and what
   a transaction-scoped interplay (NOTIFY fires only on commit of the
   sending transaction — the pg analogue of honker's notify-in-tx
   atomicity) looks like over a pool.
3. **The tx-seam shape differs from SQLite's resolved one.** SQLite
   (POC #1): an owned writer-slot lease, ops via `spawn_blocking`.
   Postgres: interactive transactions handed out by the pool
   (`deadpool` gives `Object` → `transaction()`), LISTEN connections
   are *separate long-lived connections* — the `*_tx` seam and the
   listener lifecycle have to compose on one driver without one
   emulating the other's weaknesses.

The question, concretely: **does tokio-postgres carry the unified
surface (enqueue_tx / claim / ack / notify_tx / listen / stream
offset ops / lock ops) with the transactional property intact, at
seam/pool/wake costs comparable to what POC #1 pinned for SQLite — and
what are the honest posture deltas (pool sizing vs LISTEN connection
dedication, `synchronous_commit` posture, notification payload limits)?**
If yes, OQ-ST-03 closes with "per-engine drivers: rusqlite+honker-core
(SQLite) / tokio-postgres+deadpool (Postgres)" and OQ-ST-04's
contract-pinning has measured ground on both sides. If no — if LISTEN
over deadpool proves fragile or the tx-seam shape fights the pool — the
findings name the fight (sqlx's `PgListener` as the alternative is a
*fallback posture*, evaluated only on that failure).

## The one arm, three sub-modules

Unlike POC #1's A/B split, this POC has one driver posture with three
sub-modules to validate — the comparison axis is *within* the engine
(listen-vs-poll, tx-seam shape), not between drivers:

### Sub-module L (listen plumbing)

- **Upstream-verified constraint (2026-10-04, from the deadpool issue
  tracker — deadpool-rs/deadpool#360, open):** pooled connections
  cannot carry LISTEN. deadpool's connect task awaits the underlying
  `tokio_postgres::Connection` to completion, which discards the async
  notification messages only `Connection::poll_message` exposes — and
  any LISTEN on a pooled connection is lost at recycle. So the shape is
  confirmed *required*, not just preferred: **a dedicated, non-pooled
  `tokio-postgres` connection per listener process**, with a forwarding
  task polling `poll_message` and fanning notifications out to
  per-subscriber tokio mpsc channels. `pg_notify` as a *sender* rides
  pooled connections freely (it is ordinary SQL). The POC asserts the
  discarded-notifications failure empirically (LISTEN issued on a
  pooled connection, verify no delivery) so the constraint is in our
  evidence base with our own test, not only the issue's word.
- **Two implementation postures for the listener substrate, compared:**
  - **(a) hand-rolled forwarder** — the thin forwarding task above,
    ours: reconnect policy, channel re-LISTEN after reconnect, and
    backpressure are all ours (~the size of a small module; the POC #1
    Arm-B watcher taught what "ours" costs — every failure mode is
    maintained by us).
  - **(b) `postgres-notify` 0.3.8** (MIT, tokio-postgres-based,
    published, verified 2026-10-04: auto-reconnect with exponential
    backoff + jitter, multi-channel `subscribe_notify`, a
    `connect_script` hook executed on (re)connect — which is the
    LISTEN-restoration mechanism — query timeout + cancellation, and
    documented callback-panic safety properties). A turnkey shape for
    exactly this sub-module. *Caveat to verify in-probe:* docs do not
    explicitly promise subscription restoration across reconnect —
    the `connect_script` is the mechanism but restoration behavior
    (does it re-issue our LISTENs) must be verified by test, plus the
    single-maintainer/small-crate posture recorded per the AGENTS.md
    ownership rules. It also *owns* its connection for queries too
    (PGRobustClient wraps the whole client) — the engine crate would
    use it as the dedicated-listener-side client only, not the query
    path.
  Both are implemented in-probe; the comparison axes are exactly the
  ones sub-module W probes (reconnect honesty, fan-out, latency) plus
  dependency-posture cost. Preference found here is OQ-ST-04/OQ-ST-06
  input (it is a per-subsystem adopt-or-derive vote, the POC-level
  answer to the same calculus as honker-core).
- Multi-channel: one LISTEN connection serving N channels (`LISTEN`
  accepts multiple registrations per connection) — measure whether the
  dedicated-connection-per-channel shape is ever warranted (connection
  budget vs fan-out complexity).
- Payload: `pg_notify` carries ≤ 8000 bytes. Probe the boundary
  behavior (payload size limits, the "payload too large" error path)
  and record the honest contract: notify payloads are hints; large
  payloads ride a table row + the notify carries the row id (the
  honker-outbox shape, pg-side).

### Sub-module T (tx-seam over the pool)

- `deadpool-postgres` pool; `enqueue_tx`/`publish_tx`/`save_offset_tx`
  receive a caller-held transaction. The property the SQLite side
  pinned (business write + enqueue + notify in one caller tx, rollback
  drops both) must hold with `NOTIFY` issued inside the caller's tx
  (Postgres delivers only on commit — verify, don't assume).
- The seam question: transaction *ownership* vs transaction *handle*.
  SQLite's lease shape cannot port (no slot model under a pool). Two
  candidate shapes, both implemented and compared:
  - **(a) caller-owned tx handle** — the trait hands the caller a
    `TxHandle` (deadpool `Transaction<'_>` wrapped) and `*_tx` methods
    take it by reference; the caller drives begin/commit.
  - **(b) closure-scoped** — the trait exposes
    `with_tx(|tx| async { ... })` (pool checkout + begin + commit/
    rollback inside the closure), and the business write also happens
    through the same handle — the caller never holds the raw
    transaction.
  Both are viable-looking; the POC measures and records the ergonomic
  and correctness trade (deadlock risk, spawn_blocking needs, what
  the caching-subscriber pattern wants). This is direct OQ-ST-04
  contract input.
- Prepared-statement discipline per B2/B5 (POC #5): per-connection
  caching across pool recycling (deadpool's statement cache posture) —
  verify the "prepared statement s1 does not exist" failure mode does
  not recur with the queue SQL shape.

### Sub-module W (wake-vs-poll parity)

- The queue consumption path two ways: **(1) LISTEN-driven** — a
  queue-table notify on commit wakes claimants (the push channel
  pgboss-rs lacks; our machinery), with claim on wake + re-poll safety
  net; **(2) poll-only** — interval polling (the pgboss-rs posture).
  Measure claim latency (enqueue→claim) both ways, p50/p99.
- `pg_notify` visibility timing: the notifier's NOTIFY is delivered to
  listeners at tx commit, but the *notifying* connection's subsequent
  reads may or may not see their own effects timing-wise — probe
  reader-visibility (the "read-your-writes over the pool" boundary
  under `read committed`, the honest default).
- LISTEN connection failure modes: drop the listen connection
  mid-subscription (network kill), verify reconnect + the replay hole
  it implies (LISTEN has no replay — anything committed while the
  listener was down is *not* re-delivered; the honest contract is
  opaque-wake + re-read, so recovery = on reconnect, wake all
  subscribers once — and, under listener posture (b),
  `connect_script`-driven re-LISTEN must be verified to run). Re-attach
  storm: N listeners × M channels on one
  re-connecting connection.

## Instruments

1. **Seam/cost probe** (`pgdiag-st-1`): the POC #1 seam workload's pg
   twin — `begin + enqueue_tx + commit` per iteration through the pool,
   n=3000, `synchronous_commit=on` (ship config) and `=off`
   (the B5 knob, both reported), p50/p90/p99. Compare shape (not
   absolute number) against POC #1's SQLite table — the cross-engine
   relative claim OQ-ST-04's contract must absorb. Plus the raw floor
   re-measure (prepared round-trip ~150–500 µs per POC #5 B2) as
   methodology cross-check.
2. **Wake probe** (`pgdiag-st-2`): N=300 notify-commits at 25 ms
   spacing, listener attach → wake latency (p50/p99/max); burst-stress
   (30 rapid commits — LISTEN does not coalesce per-tick the way
   data_version polling does; measure whether bursts each deliver or
   the notifications() stream backs up); payload-size boundary; 
   channel fan-out (4 subscribers, same channel, every one wakes).
3. **Property tests** (`tests/contract.rs` — the POC #1 suite's pg
   twin, all shared assertions reused): commit-atomicity
   (enqueue+notify+business write, rollback drops all — ghost-claim
   check against the queue table), exactly-once claim under 4
   concurrent claimants, offset save/read through the caller tx,
   lock acquire/release/renew with TTL, listener-reconnect
   recovery (kill listen conn, commit during the gap, verify
   reconnect wakes + state re-read correct), and the
   **pooled-LISTEN-discard assertion** (LISTEN on a pool connection,
   verify nothing is delivered — pinning deadpool#360's behavior in
   our own evidence base; if upstream ever fixes it, the test flips
   and the fix becomes usable — but the engine design must not count
   on it).
4. **Poll-vs-listen claim-latency probe** (`pgdiag-st-3`): enqueue→
   claim latency under both consumption postures at 1/8/32 claimant
   workers — the honest comparison that justifies (or retires) the
   LISTEN-driven claim path as the engine's default.
5. **Pool/posture probe** (`pgdiag-st-4`): deadpool pool sizing vs
   the LISTEN connection budget (the dedicated listen connection is
   *outside* the pool by the #360 constraint — the probe validates the
   budget accounting: pool max_size + one listener connection per
   process); fresh-session cost re-verify
   (~19–25 ms per B2) against the pooling discipline; `synchronous_commit`
   per-session posture (SET on checkout vs system config) mechanics.

## Decision gate

Sub-module findings constitute OQ-ST-03's Postgres half resolution iff:

- **Contract:** every shared property holds (the commit-atomicity
  property via in-tx NOTIFY is the load-bearing one; exactly-once claim
  under concurrency the second);
- **Seam:** the caller-tx seam shape survives with measured costs
  (either candidate shape; the findings name the preferred one with
  evidence — that preference is OQ-ST-04 input, not a Phase 0 ADR);
- **Wake:** LISTEN plumbing is robust enough to be the engine's wake
  story (reconnect recovery honest, fan-out correct, latency ≤ the
  sqlite watcher's order — LISTEN should beat poll; if it doesn't,
  that is a finding that reshapes OQ-ST-04's contract);

Failure on any: the findings name the failure precisely, the sqlx
`PgListener` fallback posture gets its own (smaller) evaluation round,
and OQ-ST-03's pg half stays open with a narrowed question.

## Out of scope

- The queue *semantics* depth (retry/backoff/dead-letter/sweep tuning —
  OQ-ST-05's design work; this POC uses a minimal queue table: enqueue,
  claim, ack — enough for the property tests).
- The full pg-boss schema family (job states beyond the minimal set,
  `pgboss-rs`'s DDL compatibility) — schema *design reference* status
  is OQ-ST-05; this POC does not adopt or validate pgboss-rs' code.
- The sqlite engine (POC #1's ground; its seam/cost numbers are used
  only as the cross-engine reference points).
- Multi-host stress (failover, pooling across machines) — OQ-ST-08's
  deployment-matrix work; this POC's dockerized server rides the
  established harness convention.
- Streams' full contract (offset *save/read* verified; per-consumer
  replay windows/filtering is OQ-ST-04 design work).

## Register note

This POC is **#2** in alkstore's register (phase-0.md), specified
2026-10-04. Sequencing rationale: it completes OQ-ST-03's driver
resolution (the SQLite half is POC #1's) and gives OQ-ST-04's
contract-pinning measured ground on *both* engines — after which the
reactive-contract and ownership questions (OQ-ST-04/05/06) are
paper-decisions over a filled evidence base, which is what Phase 1
should inherit.