alkstore/docs/architecture/queues.md

status, last_updated

status	last_updated
draft	2026-10-05

Queues, scheduler, outbox — semantics depth

The queue family's posture is decided (re-derived on both engines; ADR-004 and ADR-005) and the hard driver-coupled properties are POC-pinned (transactional enqueue, exactly-once claim under concurrency). What is not yet designed is the semantics depth — this document's subject, and [OQ-05]'s home. It states the design space, the decided constraints any answer must fit, and the reference material; actual semantics decisions become ADRs when made.

Decided constraints (inherited, not re-opened)

• Mechanisms: durable at-least-once queues; the outbox is a helper over queues; scheduler is cron/@every enqueueing into named queues (ADR-002) — pending OQ-09's collapse decision.
• Transactional enqueue: commit-atomic with the caller's business write on both engines (ADR-007); rollback drops the job row with no ghosts.
• Exactly-once claim: FOR UPDATE SKIP LOCKED (pg, POC-pinned under 4×4 concurrency) / honker's machinery (SQLite). At-least-once work; exactly-once processing needs the ack + visibility model.
• Wake-driven consumption: the pg engine's default is LISTEN-driven claim with a re-poll safety net; SQLite's is watcher wake with per-subscription fanout (ADR-006).
• Job options (the contract v1 skeleton, ADR-008 §1, from the honker-rs surface): delay, priority, max_attempts, expires at enqueue; ack / retry / fail / heartbeat on the job handle. Deeper QueueOpts/retry surface is this document's design space.
• Cut-flag context: result storage is ADR-002's cut-flag row; if OQ-05's design shows queue consumers provably need result-query-by-id, that is the channel to revisit it — with a consumer-inventory row, not silent inclusion.

Design space (what OQ-05 must pin)

Retry / backoff

• Trigger model: attempts counted from claim-completion; a job re-appears after visibility timeout OR explicit retry. Does max_attempts exhaustion move the job to dead-letter, flag it, or drop it?
• Backoff curve between attempts: honker and the pg-boss family have their own shapes (fixed, exponential); which does the contract pin, what is configurable, what's the default?
• heartbeat semantics: renewal vs progress signal (or both) — and what a missed heartbeat means (visibility re-expiry? nothing?).

Visibility timeouts

• Where the renewal lives: explicit (heartbeat) only, or implicit renewal per op on the job handle? Honker's design is the SQLite-side answer; pg-boss's is the reference. One answer, contract-pinned.
• Interaction with long business transactions (claims inside caller txs, ADR-007).

Dead-letter

• Move-to-table (honker's _honker_dead, pg-boss's dead-letter queue) vs flag-in-place. Inspection/requeue surface shape.
• Retention: does a dead-lettered job expire (an expires-driven sweep — an OQ-05 sub-question) or live until explicitly cleared?

Sweep / maintenance

• sweep_expired is on the queue's surface; cadence ownership is the open question. Honker's scheduler machinery (leader-elected, missed-boundary catch-up) can run it; a consumer's own scheduler row can too; the engine could ship a default. Decision shaped by OQ-09's scheduler collapse — same machinery either way.
• Multi-process sweep safety on SQLite vs Postgres: named-lock coordination (the alkblobs fleet-sweeper pattern, ADR-002) vs native advisory locks.

Scheduler surface

• OQ-09's question, stated crisply: is scheduler a first-class mechanism (own handle, methods, inspection) or queues + a schedule(cron_expression, queue_name, payload) call? The inventory's evidence is thin (documented-thin); a v1 that is queues
  • a schedule call with leader-election behavior documented is the smaller contract; if a consumer needs inspectable/pausable schedule objects (add/pause/resume/update/list/remove), that's the full honker shape. Decide against the inventory rows, not against the whole honker menu.

Namespaces / schema layout (pg side)

• Schema-scoped DDL (the pg-boss design) vs shared-schema table naming. Rides the reserved-namespace contract (ADR-006, ADR-008 §4): the reserved channel namespace is pinned (__alkstore_ prefix); queue/stream/lock table layout remains this document's OQ-05 work and must be collision-proof against consumer tables in the same database (alkblobs' ADR-008 — co-tenant-tables precedent — at /workspace/@alkdev/alkblobs/docs/architecture/decisions/).

Reference material

• honker's queue design — the SQLite-side incumbent (/workspace/honker, its honker-core machinery; the engine rides it directly, so the SQLite side's depth is largely "inherit + pin").
• pg-boss family — /workspace/pgboss-rs @ 98f7d9e (queue states, maintenance/dead-letter behavior) and the node original (pg-boss, the upstream of record — compare semantics the port may have dropped).
• POC ground — poc-pg-posture-findings.md (the minimal queue table + SKIP LOCKED claim measured; the property suite green); poc-sqlite-posture-findings.md (the SQLite twin).
• Consumers — alkfs OQ-FS-14 (sync outbox), alkblobs ops-surface/gc docs (maintenance cadences), the family-wide "who sweeps" punt (inventory §scheduler).

Design Decisions

ADR	Decision	Summary
002	Feature scope	queues + outbox in; result storage cut-flag
004	pg driver	queue machinery re-derived, pg-boss as reference
005	Ownership	design-reference postures for the queue family
006	Wake contract	wake-driven consumption, guarantee table
007	Tx seam	enqueue_tx commit-atomicity
008	Contract v1	queue skeleton + EnqueueOpts pinned; depth is OQ-05's extension path

Open Questions

Open questions are tracked in open-questions.md. Key questions affecting this document:

• OQ-05: the semantics-depth pinning this document frames (open, high priority)
• OQ-09: scheduler as first-class mechanism vs queues + schedule (open)