--- id: pg-engine-queues name: Postgres engine — queues (`Queue`, `FOR UPDATE SKIP LOCKED` claims, job handles, backoff curve) status: pending depends_on: [pg-engine-seam-tx, pg-engine-notify-listen] scope: moderate risk: medium impact: component level: implementation tags: [wave-4, postgres-engine] --- ## Description Implement the queue mechanism's pg arm — the re-derived queue machinery (ADR-004/005's greenfield posture, pg-boss schema family as design reference; ADR-010's depth) over the schema task's job/dead tables and the forwarder's wake substrate: - **`Store::queue(name, opts)`** — validated constructor (shared- namespace kinds; construction stays free, queues are names not registered objects). The handle carries its `QueueOpts` as the stamp source. - **`enqueue`** (auto-commit): resolves `EnqueueOpts` over the handle's stamps via the seam task's `resolve_enqueue_opts` + `stamps_with_override` (the pg engine's own `resolution.rs` — one clock read per enqueue, delay-over-`run_at`, relative- `expires`, `max_attempts` the one per-job override), `encode_payload` at the seam, `INSERT` on a pool connection. Then **wake**: `pg_notify` on the queue's wake channel (the mechanism-name-is- the-channel realization) — the LISTEN-driven claim path's push half. - **`claim_one` / `claim_batch`** — exactly-once handout under concurrency: the `FOR UPDATE SKIP LOCKED` claim (POC-pinned under 4×4 concurrency) on a pool connection, in one statement or a CTE-scoped transaction — implementer's choice, but the exactly-once property is what the tests pin (two workers, one row; N×M concurrent drain, zero double-claims). Semantics per ADR-010: ordering `priority DESC, run_at ASC, enqueue order`; `attempts += 1` per claim (a reclaim consumes an attempt); the claim sets the row's deadline from the job's own stamped `visibility_timeout_s`; the pre-claim dead-letter sweep for exhausted reclaimables (the substrate's D-12 shape — a worker dying right after its last allowed claim dead-letters, not strands); `worker_id` stamps the claimant column (non-empty validated, consumer-local — ADR-019 §2). The **extent guard** (`n <= 0` → empty `Vec` at trait-impl entry, ADR-023 §2). Claims return boxed `JobHandle`s (ops + row value). - **`JobHandle` impl** (ADR-019 §3, ADR-010 §2's uniform validity predicate — `processing` + unexpired claim deadline; refusal = `Ok(false)`, never error): - one-shot `ack`/`retry`/`fail` as `'static` boxed futures over the consumed handle (mirroring `TxHandle::commit`'s shape); repeatable `heartbeat(&self, extend)` — absolute reset of the claim deadline (not additive), late heartbeat refused. - `ack` deletes the row; `retry(err, None)` computes the queue's **equal-jitter exponential curve** — the pg engine's own implementation of ADR-010 §3's pinned form (`base * 2^(attempt-1)`, jitter over `[half, cap]` integerized inclusive, capped 3600 s; attempt index = the row's post-claim count). **The jitter's randomness source is this task's flagged choice**: the SQLite engine used a std-only `RandomState`-hash draw (no `rand` dep, ADR-005's posture) — reuse that approach unless a pg-native source is materially better, and **flag the choice in Notes**. `retry(err, Some(d))` overrides; `fail(None)` pins `"failed"`; `retry`'s exhaust string pins `"max attempts exceeded"` (the engine-default strings land identically on both engines — the backlog row's wording). - Dead-letter moves (retry-at-budget, fail) write the dead table with `last_error`/`died_at`; dead rows are `get_job`-visible. - **`ack_batch(ids)`** — the batch ack applies the uniform predicate per id (lapsed or non-claimed ids silently not counted; count returned; no worker filter — the wave-3 D-31 delta's shape is the contract's, not a substrate artifact). - **`cancel(job_id)`** — unconditional delete (pending or processing, not an interrupt); `Ok(false)` when row-less. - **`get_job(job_id)`** — dead-visible pure read (pool connection; `Option` with `last_error`/`died_at` on dead rows; the resolved values — not raw opts — are what the row shows, ADR-020). - **`sweep_expired()`** — both-states no-stranded-rows move (any past-expiry row → dead with `last_error='expired'`) + retention enforcement (`dead_letter_retention_s`, `None` = forever) — ADR-010 §5; the queue-scoped handle form. - **LISTEN-driven claim with re-poll safety net** (the engine's default consumption posture, ADR-004): the *contract* claim ops are pull-shaped (callers claim in their own loops); the wake substrate makes those loops reactive — a consumer that `listen(queue_name)`s and claims on wake gets the measured 3–6 ms path; the re-poll safety net (interval-tunable, `PgOpts`-carried if wired — implementer's choice, documented either way) covers LISTEN loss (the no-replay hole). The engine's own tests exercise the wake-driven claim loop (commit → notify → wake → claim, the POC's property) to pin the posture; the contract surface itself stays pull-shaped. - **Closed-store posture** on all queue ops (the wave-3 shape). ## Acceptance Criteria - [ ] `enqueue` resolves opts over the handle's stamps (the shared resolution arithmetic; `max_attempts` the one override); payload bytes exact; wake notify fires on the queue's channel - [ ] `claim_one`/`claim_batch`: exactly-once under concurrency (two-workers-one-row; 6×40 concurrent drain, zero double-claims); ordering row pinned; reclaim consumes an attempt; deadline from the row's stamp; extent guard (`n <= 0` → empty `Vec`); pre-claim exhausted-reclaimable sweep - [ ] `JobHandle`: validity predicate uniform (lapse refuses all ops, `Ok(false)`); heartbeat absolute-reset + late-refused; ack deletes; retry curve matches ADR-010 §3's range/cap (statistical + cap tests, the SQLite task's shape); dead-letter defaults (`"max attempts exceeded"`, `"failed"`, caller strings verbatim); dead rows `get_job`-visible - [ ] `ack_batch` per-id predicate (mixed lapsed/valid ids); `cancel` unconditional both states; `sweep_expired` both-states + retention (incl. forever-none) - [ ] Wake-driven claim loop pinned end-to-end (enqueue → wake → claim, cross-connection) - [ ] Validation (empty/reserved queue names, empty `worker_id`) + closed-store fail-closed on all queue ops - [ ] `cargo test -p alkstore-postgres` (harness server), clippy `-D warnings`, fmt clean; gates green server-less ## References - docs/architecture/core-contract.md (queues section) - docs/architecture/decisions/010-queue-semantics-depth.md (§1–§5, the depth) - docs/architecture/decisions/019-mechanism-handle-surfaces.md §1–§3 - docs/architecture/decisions/020-enqueue-opt-semantics-and-bridges.md - docs/architecture/decisions/023-fourth-review-round.md §2 (extent guard) - docs/architecture/decisions/004-postgres-driver.md (re-derive posture, LISTEN-driven default) - docs/architecture/queues.md (lifecycle, consumption posture) - docs/research/reference-pgboss-rs-semantics.md (design reference) - alkstore-sqlite/src/queue.rs (the structural twin — same semantics, own SQL) - alkstore/src/queue.rs, job_handle.rs (the traits) ## Notes > To be filled by implementation agent ## Summary > To be filled on completion