141 lines
7.3 KiB
Markdown
141 lines
7.3 KiB
Markdown
---
|
||
id: pg-engine-queues
|
||
name: Postgres engine — queues (`Queue`, `FOR UPDATE SKIP LOCKED` claims, job handles, backoff curve)
|
||
status: pending
|
||
depends_on: [pg-engine-seam-tx, pg-engine-notify-listen]
|
||
scope: moderate
|
||
risk: medium
|
||
impact: component
|
||
level: implementation
|
||
tags: [wave-4, postgres-engine]
|
||
---
|
||
|
||
## Description
|
||
|
||
Implement the queue mechanism's pg arm — the re-derived queue
|
||
machinery (ADR-004/005's greenfield posture, pg-boss schema family as
|
||
design reference; ADR-010's depth) over the schema task's job/dead
|
||
tables and the forwarder's wake substrate:
|
||
|
||
- **`Store::queue(name, opts)`** — validated constructor (shared-
|
||
namespace kinds; construction stays free, queues are names not
|
||
registered objects). The handle carries its `QueueOpts` as the
|
||
stamp source.
|
||
- **`enqueue`** (auto-commit): resolves `EnqueueOpts` over the
|
||
handle's stamps via the seam task's `resolve_enqueue_opts` +
|
||
`stamps_with_override` (the pg engine's own `resolution.rs` —
|
||
one clock read per enqueue, delay-over-`run_at`, relative-
|
||
`expires`, `max_attempts` the one per-job override), `encode_payload`
|
||
at the seam, `INSERT` on a pool connection. Then **wake**:
|
||
`pg_notify` on the queue's wake channel (the mechanism-name-is-
|
||
the-channel realization) — the LISTEN-driven claim path's push
|
||
half.
|
||
- **`claim_one` / `claim_batch`** — exactly-once handout under
|
||
concurrency: the `FOR UPDATE SKIP LOCKED` claim (POC-pinned under
|
||
4×4 concurrency) on a pool connection, in one statement or a
|
||
CTE-scoped transaction — implementer's choice, but the
|
||
exactly-once property is what the tests pin (two workers, one row;
|
||
N×M concurrent drain, zero double-claims). Semantics per ADR-010:
|
||
ordering `priority DESC, run_at ASC, enqueue order`; `attempts +=
|
||
1` per claim (a reclaim consumes an attempt); the claim sets the
|
||
row's deadline from the job's own stamped `visibility_timeout_s`;
|
||
the pre-claim dead-letter sweep for exhausted reclaimables (the
|
||
substrate's D-12 shape — a worker dying right after its last
|
||
allowed claim dead-letters, not strands); `worker_id` stamps the
|
||
claimant column (non-empty validated, consumer-local — ADR-019 §2).
|
||
The **extent guard** (`n <= 0` → empty `Vec` at trait-impl entry,
|
||
ADR-023 §2). Claims return boxed `JobHandle`s (ops + row value).
|
||
- **`JobHandle` impl** (ADR-019 §3, ADR-010 §2's uniform validity
|
||
predicate — `processing` + unexpired claim deadline; refusal =
|
||
`Ok(false)`, never error):
|
||
- one-shot `ack`/`retry`/`fail` as `'static` boxed futures over the
|
||
consumed handle (mirroring `TxHandle::commit`'s shape);
|
||
repeatable `heartbeat(&self, extend)` — absolute reset of the
|
||
claim deadline (not additive), late heartbeat refused.
|
||
- `ack` deletes the row; `retry(err, None)` computes the queue's
|
||
**equal-jitter exponential curve** — the pg engine's own
|
||
implementation of ADR-010 §3's pinned form
|
||
(`base * 2^(attempt-1)`, jitter over `[half, cap]` integerized
|
||
inclusive, capped 3600 s; attempt index = the row's post-claim
|
||
count). **The jitter's randomness source is this task's flagged
|
||
choice**: the SQLite engine used a std-only `RandomState`-hash
|
||
draw (no `rand` dep, ADR-005's posture) — reuse that approach
|
||
unless a pg-native source is materially better, and **flag the
|
||
choice in Notes**. `retry(err, Some(d))` overrides; `fail(None)`
|
||
pins `"failed"`; `retry`'s exhaust string pins
|
||
`"max attempts exceeded"` (the engine-default strings land
|
||
identically on both engines — the backlog row's wording).
|
||
- Dead-letter moves (retry-at-budget, fail) write the dead table
|
||
with `last_error`/`died_at`; dead rows are `get_job`-visible.
|
||
- **`ack_batch(ids)`** — the batch ack applies the uniform predicate
|
||
per id (lapsed or non-claimed ids silently not counted; count
|
||
returned; no worker filter — the wave-3 D-31 delta's shape is the
|
||
contract's, not a substrate artifact).
|
||
- **`cancel(job_id)`** — unconditional delete (pending or processing,
|
||
not an interrupt); `Ok(false)` when row-less.
|
||
- **`get_job(job_id)`** — dead-visible pure read (pool connection;
|
||
`Option<Job>` with `last_error`/`died_at` on dead rows; the
|
||
resolved values — not raw opts — are what the row shows, ADR-020).
|
||
- **`sweep_expired()`** — both-states no-stranded-rows move (any
|
||
past-expiry row → dead with `last_error='expired'`) + retention
|
||
enforcement (`dead_letter_retention_s`, `None` = forever) —
|
||
ADR-010 §5; the queue-scoped handle form.
|
||
- **LISTEN-driven claim with re-poll safety net** (the engine's
|
||
default consumption posture, ADR-004): the *contract* claim ops are
|
||
pull-shaped (callers claim in their own loops); the wake substrate
|
||
makes those loops reactive — a consumer that `listen(queue_name)`s
|
||
and claims on wake gets the measured 3–6 ms path; the re-poll
|
||
safety net (interval-tunable, `PgOpts`-carried if wired —
|
||
implementer's choice, documented either way) covers LISTEN loss
|
||
(the no-replay hole). The engine's own tests exercise the
|
||
wake-driven claim loop (commit → notify → wake → claim, the POC's
|
||
property) to pin the posture; the contract surface itself stays
|
||
pull-shaped.
|
||
- **Closed-store posture** on all queue ops (the wave-3 shape).
|
||
|
||
## Acceptance Criteria
|
||
|
||
- [ ] `enqueue` resolves opts over the handle's stamps (the shared
|
||
resolution arithmetic; `max_attempts` the one override);
|
||
payload bytes exact; wake notify fires on the queue's channel
|
||
- [ ] `claim_one`/`claim_batch`: exactly-once under concurrency
|
||
(two-workers-one-row; 6×40 concurrent drain, zero
|
||
double-claims); ordering row pinned; reclaim consumes an
|
||
attempt; deadline from the row's stamp; extent guard
|
||
(`n <= 0` → empty `Vec`); pre-claim exhausted-reclaimable sweep
|
||
- [ ] `JobHandle`: validity predicate uniform (lapse refuses all ops,
|
||
`Ok(false)`); heartbeat absolute-reset + late-refused; ack
|
||
deletes; retry curve matches ADR-010 §3's range/cap
|
||
(statistical + cap tests, the SQLite task's shape); dead-letter
|
||
defaults (`"max attempts exceeded"`, `"failed"`, caller strings
|
||
verbatim); dead rows `get_job`-visible
|
||
- [ ] `ack_batch` per-id predicate (mixed lapsed/valid ids); `cancel`
|
||
unconditional both states; `sweep_expired` both-states +
|
||
retention (incl. forever-none)
|
||
- [ ] Wake-driven claim loop pinned end-to-end (enqueue → wake →
|
||
claim, cross-connection)
|
||
- [ ] Validation (empty/reserved queue names, empty `worker_id`) +
|
||
closed-store fail-closed on all queue ops
|
||
- [ ] `cargo test -p alkstore-postgres` (harness server), clippy
|
||
`-D warnings`, fmt clean; gates green server-less
|
||
|
||
## References
|
||
|
||
- docs/architecture/core-contract.md (queues section)
|
||
- docs/architecture/decisions/010-queue-semantics-depth.md (§1–§5, the depth)
|
||
- docs/architecture/decisions/019-mechanism-handle-surfaces.md §1–§3
|
||
- docs/architecture/decisions/020-enqueue-opt-semantics-and-bridges.md
|
||
- docs/architecture/decisions/023-fourth-review-round.md §2 (extent guard)
|
||
- docs/architecture/decisions/004-postgres-driver.md (re-derive posture, LISTEN-driven default)
|
||
- docs/architecture/queues.md (lifecycle, consumption posture)
|
||
- docs/research/reference-pgboss-rs-semantics.md (design reference)
|
||
- alkstore-sqlite/src/queue.rs (the structural twin — same semantics, own SQL)
|
||
- alkstore/src/queue.rs, job_handle.rs (the traits)
|
||
|
||
## Notes
|
||
|
||
> To be filled by implementation agent
|
||
|
||
## Summary
|
||
|
||
> To be filled on completion |