Wave 4 decomposed: Postgres engine — 11 tasks (schema, open/opts, seam, forwarder, mechanisms, scheduler/outbox, integration, review gate); plan synced

tasks/pg-engine-queues.md

@@ -0,0 +1,141 @@
+---
+id: pg-engine-queues
+name: Postgres engine — queues (`Queue`, `FOR UPDATE SKIP LOCKED` claims, job handles, backoff curve)
+status: pending
+depends_on: [pg-engine-seam-tx, pg-engine-notify-listen]
+scope: moderate
+risk: medium
+impact: component
+level: implementation
+tags: [wave-4, postgres-engine]
+---
+
+## Description
+
+Implement the queue mechanism's pg arm — the re-derived queue
+machinery (ADR-004/005's greenfield posture, pg-boss schema family as
+design reference; ADR-010's depth) over the schema task's job/dead
+tables and the forwarder's wake substrate:
+
+- **`Store::queue(name, opts)`** — validated constructor (shared-
+  namespace kinds; construction stays free, queues are names not
+  registered objects). The handle carries its `QueueOpts` as the
+  stamp source.
+- **`enqueue`** (auto-commit): resolves `EnqueueOpts` over the
+  handle's stamps via the seam task's `resolve_enqueue_opts` +
+  `stamps_with_override` (the pg engine's own `resolution.rs` —
+  one clock read per enqueue, delay-over-`run_at`, relative-
+  `expires`, `max_attempts` the one per-job override), `encode_payload`
+  at the seam, `INSERT` on a pool connection. Then **wake**:
+  `pg_notify` on the queue's wake channel (the mechanism-name-is-
+  the-channel realization) — the LISTEN-driven claim path's push
+  half.
+- **`claim_one` / `claim_batch`** — exactly-once handout under
+  concurrency: the `FOR UPDATE SKIP LOCKED` claim (POC-pinned under
+  4×4 concurrency) on a pool connection, in one statement or a
+  CTE-scoped transaction — implementer's choice, but the
+  exactly-once property is what the tests pin (two workers, one row;
+  N×M concurrent drain, zero double-claims). Semantics per ADR-010:
+  ordering `priority DESC, run_at ASC, enqueue order`; `attempts +=
+  1` per claim (a reclaim consumes an attempt); the claim sets the
+  row's deadline from the job's own stamped `visibility_timeout_s`;
+  the pre-claim dead-letter sweep for exhausted reclaimables (the
+  substrate's D-12 shape — a worker dying right after its last
+  allowed claim dead-letters, not strands); `worker_id` stamps the
+  claimant column (non-empty validated, consumer-local — ADR-019 §2).
+  The **extent guard** (`n <= 0` → empty `Vec` at trait-impl entry,
+  ADR-023 §2). Claims return boxed `JobHandle`s (ops + row value).
+- **`JobHandle` impl** (ADR-019 §3, ADR-010 §2's uniform validity
+  predicate — `processing` + unexpired claim deadline; refusal =
+  `Ok(false)`, never error):
+  - one-shot `ack`/`retry`/`fail` as `'static` boxed futures over the
+    consumed handle (mirroring `TxHandle::commit`'s shape);
+    repeatable `heartbeat(&self, extend)` — absolute reset of the
+    claim deadline (not additive), late heartbeat refused.
+  - `ack` deletes the row; `retry(err, None)` computes the queue's
+    **equal-jitter exponential curve** — the pg engine's own
+    implementation of ADR-010 §3's pinned form
+    (`base * 2^(attempt-1)`, jitter over `[half, cap]` integerized
+    inclusive, capped 3600 s; attempt index = the row's post-claim
+    count). **The jitter's randomness source is this task's flagged
+    choice**: the SQLite engine used a std-only `RandomState`-hash
+    draw (no `rand` dep, ADR-005's posture) — reuse that approach
+    unless a pg-native source is materially better, and **flag the
+    choice in Notes**. `retry(err, Some(d))` overrides; `fail(None)`
+    pins `"failed"`; `retry`'s exhaust string pins
+    `"max attempts exceeded"` (the engine-default strings land
+    identically on both engines — the backlog row's wording).
+  - Dead-letter moves (retry-at-budget, fail) write the dead table
+    with `last_error`/`died_at`; dead rows are `get_job`-visible.
+- **`ack_batch(ids)`** — the batch ack applies the uniform predicate
+  per id (lapsed or non-claimed ids silently not counted; count
+  returned; no worker filter — the wave-3 D-31 delta's shape is the
+  contract's, not a substrate artifact).
+- **`cancel(job_id)`** — unconditional delete (pending or processing,
+  not an interrupt); `Ok(false)` when row-less.
+- **`get_job(job_id)`** — dead-visible pure read (pool connection;
+  `Option<Job>` with `last_error`/`died_at` on dead rows; the
+  resolved values — not raw opts — are what the row shows, ADR-020).
+- **`sweep_expired()`** — both-states no-stranded-rows move (any
+  past-expiry row → dead with `last_error='expired'`) + retention
+  enforcement (`dead_letter_retention_s`, `None` = forever) —
+  ADR-010 §5; the queue-scoped handle form.
+- **LISTEN-driven claim with re-poll safety net** (the engine's
+  default consumption posture, ADR-004): the *contract* claim ops are
+  pull-shaped (callers claim in their own loops); the wake substrate
+  makes those loops reactive — a consumer that `listen(queue_name)`s
+  and claims on wake gets the measured 3–6 ms path; the re-poll
+  safety net (interval-tunable, `PgOpts`-carried if wired —
+  implementer's choice, documented either way) covers LISTEN loss
+  (the no-replay hole). The engine's own tests exercise the
+  wake-driven claim loop (commit → notify → wake → claim, the POC's
+  property) to pin the posture; the contract surface itself stays
+  pull-shaped.
+- **Closed-store posture** on all queue ops (the wave-3 shape).
+
+## Acceptance Criteria
+
+- [ ] `enqueue` resolves opts over the handle's stamps (the shared
+      resolution arithmetic; `max_attempts` the one override);
+      payload bytes exact; wake notify fires on the queue's channel
+- [ ] `claim_one`/`claim_batch`: exactly-once under concurrency
+      (two-workers-one-row; 6×40 concurrent drain, zero
+      double-claims); ordering row pinned; reclaim consumes an
+      attempt; deadline from the row's stamp; extent guard
+      (`n <= 0` → empty `Vec`); pre-claim exhausted-reclaimable sweep
+- [ ] `JobHandle`: validity predicate uniform (lapse refuses all ops,
+      `Ok(false)`); heartbeat absolute-reset + late-refused; ack
+      deletes; retry curve matches ADR-010 §3's range/cap
+      (statistical + cap tests, the SQLite task's shape); dead-letter
+      defaults (`"max attempts exceeded"`, `"failed"`, caller strings
+      verbatim); dead rows `get_job`-visible
+- [ ] `ack_batch` per-id predicate (mixed lapsed/valid ids); `cancel`
+      unconditional both states; `sweep_expired` both-states +
+      retention (incl. forever-none)
+- [ ] Wake-driven claim loop pinned end-to-end (enqueue → wake →
+      claim, cross-connection)
+- [ ] Validation (empty/reserved queue names, empty `worker_id`) +
+      closed-store fail-closed on all queue ops
+- [ ] `cargo test -p alkstore-postgres` (harness server), clippy
+      `-D warnings`, fmt clean; gates green server-less
+
+## References
+
+- docs/architecture/core-contract.md (queues section)
+- docs/architecture/decisions/010-queue-semantics-depth.md (§1–§5, the depth)
+- docs/architecture/decisions/019-mechanism-handle-surfaces.md §1–§3
+- docs/architecture/decisions/020-enqueue-opt-semantics-and-bridges.md
+- docs/architecture/decisions/023-fourth-review-round.md §2 (extent guard)
+- docs/architecture/decisions/004-postgres-driver.md (re-derive posture, LISTEN-driven default)
+- docs/architecture/queues.md (lifecycle, consumption posture)
+- docs/research/reference-pgboss-rs-semantics.md (design reference)
+- alkstore-sqlite/src/queue.rs (the structural twin — same semantics, own SQL)
+- alkstore/src/queue.rs, job_handle.rs (the traits)
+
+## Notes
+
+> To be filled by implementation agent
+
+## Summary
+
+> To be filled on completion