# ADR-010: Queue semantics depth — visibility, retry/backoff, dead-letter, sweep, layout ## Status Accepted (2026-10-05, Phase 1 — OQ-05's resolution; composes with [ADR-009](009-scheduler-collapse.md)) ## Context Contract v1 pinned the queue *skeleton* ([ADR-008](008-contract-v1-pinning.md) §1) and explicitly left the semantics depth to OQ-05: retry policy shape, visibility/renewal mechanics, dead-letter move-vs-flag and retention, sweep/maintenance design, the result-storage cut-flag's disposition, and queue/stream/lock **table** layout (the co-tenancy collision surface, [queues.md](../queues.md)). The evidence base: honker's queue machinery (the SQLite-side incumbent — `/workspace/honker` @ f4e53c6, whose functions the engine rides directly) and the pg-boss family (the pg-side design reference — `/workspace/pgboss-rs` @ 98f7d9e standing in for node pg-boss v10; design-reference only per [ADR-005](005-dependency-ownership.md)). Both models were read in full for this resolution (reference notes: `docs/research/reference-honker-machinery.md`, `docs/research/reference-pgboss-rs-semantics.md`). The hard driver-coupled properties (transactional enqueue, exactly-once claim) are already POC-pinned on both engines. The two references disagree on several semantics points, and honker has real gaps (a dead-letter table `get_job` can't see; expired processing rows no path can reach — the "zombie" hole; no dead-row retention mechanics; no backoff in the core at all). OQ-05 is the place to fix what re-derivation should fix and inheriting should inherit. ## Decision ### 1. The job state machine: three states, delete-on-ack Contract job states: **`pending` → `processing` → `dead`** (+ absence: `ack` and `cancel` delete the row). - Honker's model, adopted whole — one live table, move-to-dead (physical row move, savepoint-guarded), not the pg-boss seven-state enum. `max_attempts` is frozen at enqueue; `attempts` counts every claim. - **`ack` deletes** (honker parity): completed work leaves no row. The completed-job-with-output model (pg-boss's `completed` state + `output` column) is **not** adopted — it is result storage under another name, and result storage is a cut-flag row ([ADR-002](002-feature-scope.md)); see §7. - **`get_job` sees dead jobs.** The returned job carries state, payload, attempts, priority, and the timestamps `created_at`, `run_at`, `claimed_at`, `expires_at` — plus, dead-only, `last_error` and `died_at` (no job = `None`). This deliberately *narrows* honker's surface (honker's `get_job` reads only live rows — post-mortem diagnosis was SQL-only); diagnosis-by-API is the honest fix, and it is what makes move-to-dead (vs flag-in-place) inspectable without raw SQL. - **`cancel` is unconditional** (honker parity): it deletes the row in either `pending` or `processing`, regardless of which worker holds it. Not an interrupt — the holder's next ack/heartbeat returns false, the same shape as expiry. `ack_batch` is the batch form of ack (count returned; non-claimed ids silently not counted). - Claim ordering (same queue): `priority DESC`, then ready-time (`run_at`) ascending, then enqueue order — FIFO under equal priority, both references agree; pinned. ### 2. Visibility timeout and heartbeat: explicit renewal, late-heartbeat refusal One answer, contract-pinned (honker's design; pg-boss's no-renewal/expire-sweep model is not inherited): - `visibility_timeout_s` is a per-job stamped value (see §3a's resolution rule — default **300 s**, honker parity); each claim sets the row's deadline from the job's stamp. - **`heartbeat(extend)` is renewal** — an absolute reset of the claim deadline from now. There is no progress-signal meaning; a consumer signaling progress uses its own channels. Renewal cadence is the consumer's obligation: handlers that might outlive the visibility timeout must heartbeat inside it. - **A late heartbeat is refused** (deadline already passed ⇒ returns false): it can never steal the job back from a reclaimer. The consequence is the honest at-least-once window — between deadline lapse and another worker's reclaim, the original worker may still complete: dual-execution is possible and idempotence is the consumer's job (matches the locks row's silent-expiry honesty, [ADR-008](008-contract-v1-pinning.md) §7). - **A reclaim consumes an attempt** (honker's counting): a claim is an attempt, whether fresh or a visibility reclaim. Documented explicitly as the contract's counting rule — the footgun is stated, not discovered: a handler that forgets to heartbeat looks like a repeatedly-failing job and dead-letters on budget exhaustion. - Deadline lapse itself is lazy: an expired-claim row becomes claimable by ordinary claim (no transition fires on lapse); there is no separate reaper for expired claims, only the budget rules below and the no-stranded-rows sweep (§5). - **Job-handle op validity predicate** (the D-12 disposition, stated uniformly): every handle op (`ack`/`heartbeat`/`retry`/`fail`) succeeds only when the row is in `processing` **and** the caller's claim deadline has not lapsed — ADR-008 §5's false-case list ("expired, acked elsewhere, cancelled") is the same rule one predicate-wide: deadline lapse refuses *all* ops, not just heartbeat. The dual-execution window stays honest under this: the original worker between lapse and reclaim may still *complete its work* (the side effects happen), but its ack will not land and the row is reprocessed at reclaim — at-least-once, as documented. The reclaim wins atomically when it races the ack (one statement, same predicate). This state check is engine-pinned in both engines' implementations (upstream's missing-check defect class D-12 is *not* inherited). ### 3. Retry and backoff: explicit delay or the queue's curve - Job handle: `ack()`, `retry(err, delay)`, `fail(err)`, and `heartbeat(extend)` — the v1 skeleton ([ADR-008](008-contract-v1-pinning.md) §1), with `heartbeat`'s meaning pinned by §2 and `renew` remaining the *lock*-side term ([ADR-008](008-contract-v1-pinning.md) §8's split). - **`retry(err, None)`** — the deferred delay — is computed by the engine from the queue's curve; **`retry(err, Some(d))`** overrides it. Honker's core takes a caller-supplied delay with no curve; pg-boss computes in-engine from a base. The contract pins the curve so both engines compute identically. - **The curve: equal-jitter exponential**, `base * 2^(attempt-1)` with uniform jitter over the *lower half* of the doubling range — `delay ∈ [base·2^(a−1)/2, base·2^(a−1)]` — capped at **1 hour** (a pinned contract constant; no `backoff_max` knob — no consumer names one; the explicit-delay override is the escape hatch for bespoke policies). The range, not a jitter-label, is the definition: this is *not* AWS-canonical full jitter (uniform over `[0, cap]`), and the formula above is what both engines compute identically. The attempt index `a` is the value of `attempts` on the row at the moment the retry is scheduled (post-claim count, including reclaims). Jitter prevents sync-failure herding across a fleet; honker's wrappers have no jitter (a spread-out fleet of failing workers retries in lockstep) and the pg-boss family's jittered exponential is the design adopted. - **`QueueOpts { visibility_timeout_s, max_attempts, backoff_base_s, dead_letter_retention_s }`** — queue-level defaults (respectively 300 s / 3 / 5 s / none, honker parities except retention which is ours); resolved and stamped per §3a. `backoff_base_s` feeds the curve. This is the `QueueOpts` depth contract v1 deliberately withheld ([ADR-008](008-contract-v1-pinning.md) §1) — pinned here as the extension path. **Outbox**: `outbox(name)` carries no opts of its own; its backing queue uses a *derived* QueueOpts set as engine defaults (honker's outbox parities: visibility 60 s, max_attempts 5, backoff base 5 s) — engine-documented, not separately configurable in v1. - Explicit `fail(err)` = immediate dead-letter (honker parity; it is the "stop retrying this" operator). `retry` at exhausted budget = dead-letter. Exhaustion via reclaim = dead-letter. *(The pre-claim sweep of already-exhausted reclaimable rows — a laziness optimization in both upstream lineages — is engine-side and optional; SQLite rides the fork's re-derivation ([ADR-011](011-sqlite-substrate-fork.md)), pg re-derives it ([engine-postgres.md](../engine-postgres.md)).)* ### 3a. QueueOpts resolution: stamped at enqueue, per job Queues are names, not registered objects — so queue-level `QueueOpts` need a defined attachment point, and the contract pins one: - **Every job row carries the resolved opts stamped at enqueue**: enqueue resolves `EnqueueOpts.max_attempts` over the queue handle's `max_attempts` (the one per-job override the v1 skeleton names), and stamps the result — `visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`, `max_attempts` — onto the job row. Claims, heartbeats, retries, and sweeps read the **job's own stamps**, never a live registry. - **Why stamped**: uniform across engines (no per-queue config state to invent on either side — SQLite needs no new table beyond honker's columns, Postgres is job-table columns), and it makes a job's behavior immutable and inspectable (`get_job` returns the stamps with the rest of the row) no matter which process's handle enqueued it. Two processes opening handles with different `QueueOpts` for the same queue name do not fight — each enqueue stamps its own opts; the queue name is a routing key, not a configuration owner. - The consequence, stated: opts changes apply to *future enqueues only*; in-flight jobs keep their stamps. This is the honest, durable-row semantics — the same reason `max_attempts` was already frozen at enqueue in the references. - SQLite realization note: honker's `claim_batch` takes one uniform `timeout_s` per call, while per-job stamps make deadlines row-local — bridging that gap (post-claim re-stamp, per-row claiming, or another shape) was implementation work over honker's function surface and rode the same OQ-06 assessment as §5's zombie fix. *(Resolved 2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md)); the claim statement is re-derived with per-row visibility from the stamps, in owned contract-blind substrate code ([ADR-012](012-forked-substrate-design.md) §2).) The *contract* (deadline from the job's stamp) is engine-independent either way. ### 4. Dead-letter: move, retention = never-expire by default, no redrive API Move-to-table (both references converge; flag-in-place is not adopted — a dead row must not occupy the claim path's indexes): dead rows physically move to engine-owned dead storage with `last_error` and `died_at`. Triggers: explicit `fail`, `retry` at budget, exhaustion-by-reclaim, and `expires` lapses (§5). - **Retention**: dead rows live until explicitly removed by default (honker's posture). `QueueOpts.dead_letter_retention_s` (default `None` = forever) makes the sweep enforce per-queue dead TTL (§5) — the retention *mechanism* is ours (honker has none); the default stays non-magical. `last_error` source strings are contract-pinned for the engine-triggered cases: `"max attempts exceeded"` (budget exhaustion, any path) and `"expired"` (sweep-by-`expires`); explicit `fail(err)`/`retry(err, …)` carry the *caller's* error string. - **No redrive/requeue API in v1.** The recipe is pinned instead: `get_job` (now sees dead rows, §1) + fresh `enqueue` — both primitives already on the surface. No consumer names an API shape for redrive; when one does, it gets a contract extension with its own OQ. - The engine-owned dead table/rows are storage-internal — not a consumer namespace (no reserved-prefix question; they're not name-addressable). ### 5. `sweep_expired`: the no-stranded-rows property Contract semantics for `sweep_expired(queue)` (the v1 skeleton's maintenance entry point): - It moves to dead (`last_error = 'expired'`) **every row of the queue past its `expires_at` in any state** — pending rows (honker parity) *and* processing rows whose job-level expiry passed. This is the **no-stranded-rows property**: an enqueued job with an `expires` deadline is eventually in exactly one of pending, processing, or dead — never stuck unreachable. Honker fails this property: an expired *processing* row whose worker died is unreachable by claim (predicate requires future expiry), by pre-claim dead-lettering (same), and by `sweep_expired` (pending only) — the zombie hole, a real defect found in the reference read (D-6 in the quality read's register). This ADR fixes it at the contract level; the Postgres engine enforces it directly; the SQLite engine's realization was implementation work **contingent on OQ-06** — *(resolved 2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md)) and the both-states sweep lands in owned code.)* - When `dead_letter_retention_s` is set, the same sweep also deletes dead rows past their retention (the only sweeper dead rows ever have — nothing runs without a caller). - Sweep is single-statement atomic per queue (SQLite: writer serialization; Postgres: row-locked `UPDATE … RETURNING`); **no leader lock is required** for a bare `sweep_expired` call — the multi-process recipe gets its coordination from §6's machinery, not from the sweep itself. ### 6. Sweep/maintenance cadence: no ambient sweeper; the collapse recipe - **The engine ships no background sweeper, no maintenance thread, no default cadence** — the family's no-ambient-timers posture (alkblobs ADR-005) and honker's own verified posture (nothing in honker-core auto-runs; the only implicit maintenance is laziness in claim paths). The engine's *responsibility* is confined to correctness-of-transition (§1–§5); **timeliness is the consumer's**. - The pinned recipe for the family-wide "who sweeps" problem is the collapse machinery ([ADR-009](009-scheduler-collapse.md)): `store.schedule("maintenance", "@every 300s", maintenance_queue, …)` + `run_schedules` (leader-elected) + a worker whose handler calls `sweep_expired` per queue. One schedule row, one worker — the pattern every consumer was going to hand-roll, answered twice in the contract instead of N times above it. - Dead-letter and notifications-table hygiene (SQLite engine): `prune_notifications`/`prune_notifications_keep_latest` remain **out of the contract** (ADR-008 §8's disposition, resolved here by disposition): the notifications table is the SQLite wake mechanism's *transport* detail — consumers interact with wakes, not rows. Its hygiene is engine-internal: **an at-attach pruning cap (engine opts)** — the mechanism the fork scope realizes ([ADR-011](011-sqlite-substrate-fork.md)); no cadence, ambient or engine-side, contradicts this section's no-ambient-sweeper bullet. The fix for upstream's unbounded notifications growth must not become a consumer's chore. *(Annotated 2026-10-05: the earlier "engine-side maintenance cadence" wording was wrong — nothing in the fork scope realizes a cadence; the at-attach cap is the pin.)* ### 7. Result storage: cut-flag stands Reconsidered per OQ-05's mandate and **left cut**: no consumer row names outcome-query-by-id; the §1 delete-on-ack decision keeps completed jobs out of storage entirely (the pg-boss `output`/completed -row model is that feature under another name — adopting it would silent-include the cut flag); the documented workaround (a job writes its own result record atomically with ack) rides the tx seam ([ADR-007](007-transactional-seam.md)) cleanly: business row + result write + ack-equivalent in one transaction. Re-entry stays gated on a consumer-inventory row. ### 8. Table layout: engine-owned schemas/tables; queues are rows, not tables - **Postgres**: all engine-owned tables (job, dead, stream log, offsets, schedule rows, internal state) live in one **engine-owned PostgreSQL schema** — default `alkstore`, overridable per-engine option — co-tenanted safely with consumer tables (alkblobs ADR-008's co-tenancy precedent; schema scoping is the collision answer, and the reserved *name* namespace ([ADR-008](008-contract-v1-pinning.md) §4) governs the name column values inside it). Queue names become row values in one job table — **no per-queue tables, no per-queue schemas** (pg-boss's partition-per-queue opt-in is not inherited; at this scale one table + partial indexes matches honker's proven shape and keeps `queue(name)` a name, not a DDL operation — queue creation is not registry-gated, per [ADR-009](009-scheduler-collapse.md) §5). Per-queue config storage is likewise stamped per §3a. - **SQLite**: honker's `_honker_*` table family in the caller's database file — storage-internal per [ADR-008](008-contract-v1-pinning.md) §4. *(Annotated 2026-10-05: the expectation that job-stamped opts would ride honker's existing columns with no schema change was falsified by the quality read — no stamp columns exist in the upstream schema (D-7). The family's fate is resolved: the fork ([ADR-011](011-sqlite-substrate-fork.md)) re-owns the whole family as `__alkstore_*`, and the stamp columns + `claimed_at` are added by the re-derivation on contract v1 ([ADR-012](012-forked-substrate-design.md) §5). Nothing consumer-visible changes.)* ### 9. Error taxonomy: no delta The depth adds **no new error variants** — checked against [ADR-008](008-contract-v1-pinning.md) §5's act-differently rule: no work claimed is a value; deadline operations are booleans; dead-letter is observable state (`get_job`), not an error thrown; queues are unregistered names, so a payload enqueued to a queue nobody claims yet is not an error (it's durable work awaiting a worker). `InvalidSpec { spec }` was the scheduler's one variant ([ADR-009](009-scheduler-collapse.md) §6) and is the only taxonomy delta this track produces. ## Consequences **Positive** - The queue mechanism is fully specified end-to-end: both references' semantics decisions are made once, pinned identically on both engines, with the references' gaps (zombie rows, dead-job invisibility, no retention mechanics, no jitter) fixed at the contract level rather than inherited. - The counting/heartbeat rules and the opts-stamping resolution are stated as contract text — the honest at-least-once story (dual-execution window, reclaim-eats-budget) is documented consumer obligation, not per-engine surprise, and per-queue configuration never fights across processes. - One engine-owned pg schema keeps co-tenancy mechanical. **Negative** - Three-state + delete-on-ack means no completion history and no built-in outcome storage — by design (the cut-flag stands, §7); consumers needing audit trails build them on the tx seam. - Honker's zombie fix, dead-row `get_job` visibility, and per-job visibility stamps over the uniform-claim-timeout function surface required either an over-machinery complement or a fork on the SQLite side — resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md), designed in [ADR-012](012-forked-substrate-design.md)). - The backoff curve's 1-hour cap and jitter formula are pinned constants — a consumer needing a different policy uses `retry(err, Some(d))` per attempt (correct, but manual). ## References - OQ-05 (`docs/architecture/open-questions.md`) — this ADR's resolution (with OQ-09, resolved by [ADR-009](009-scheduler-collapse.md)). - `docs/research/reference-honker-machinery.md` and `docs/research/reference-pgboss-rs-semantics.md` — the full reference reads (schemas, state machines, defect list with file/line cites) this resolution was made over. - [ADR-002](002-feature-scope.md) — scope (cut-flags §7 stands), [ADR-005](005-dependency-ownership.md) — the design-reference postures the two reads serve, [ADR-006](006-wake-and-delivery-contract.md) — the queues guarantee row this ADR's §2/§3 make precise, [ADR-007](007-transactional-seam.md) — the tx seam §7's workaround rides, [ADR-008](008-contract-v1-pinning.md) — the v1 skeleton this ADR extends, §5's rule §9 applies, §8's dispositions §6 resolves. - [ADR-009](009-scheduler-collapse.md) — the collapse machinery §6's recipe composes with. - OQ-06 — the SQLite-side fork candidates (§5, §3a realization note) — resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md), designed in [ADR-012](012-forked-substrate-design.md)). - [queues.md](../queues.md), [core-contract.md](../core-contract.md), engine specs — the specs carrying this depth.