alkdev/alkstore
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: resolve OQ-05 + OQ-09 — queue semantics depth (ADR-010) and scheduler collapse (ADR-009)

Browse Source 
ADR-009: scheduler collapses into queues — schedule()/unschedule()/
run_schedules (opt-in, no ambient timers), @every-only v1 grammar
(dissolves honker's local-TZ cron brittleness), boundary guarantee
row (at-least-once per boundary, fixed 64-cap catch-up with
skip-forward, row-locked fire tx as engine-generic no-double-fire
floor), __alkstore_scheduler leadership lock, InvalidSpec +
LeadershipLost taxonomy additions.

ADR-010: queue depth pinned engine-uniformly — three-state machine
(pending/processing/dead) with delete-on-ack, get_job sees dead rows,
heartbeat = renewal with late-heartbeat refusal, reclaim-consumes-
an-attempt stated as contract text, equal-jitter exponential backoff
(range definitionally pinned, 1 h cap), QueueOpts stamped onto job
rows at enqueue (no per-queue registry), move-to-dead dead-letter
with retention-sweep support and no redrive API, sweep_expired
carries the no-stranded-rows property (fixes honker's expired-
processing zombie hole — SQLite-side realization rides OQ-06 as a
concrete fork candidate), one engine-owned pg schema, queues are
rows not tables, result-storage cut-flag stands.

Also: full honker-machinery and pgboss-rs reference reads persisted
(docs/research/reference-*.md — the honker defect list pre-stages the
OQ-06 quality read), queues.md rewritten from design-space frame to
resolved-depth spec, core-contract/engines/README/overview/deployment/
ADR-002 propagated.
This commit is contained in:
glm-5.3-flash committed 2026-10-05 03:10:52 +00:00
1 parent 7ad8ac56bc
commit 79a135c934
13 files changed
+1863 -198
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/queues.md
+190 -79
View File
@@ -7,20 +7,21 @@ last_updated: 2026-10-05
The queue family's *posture* is decided (re-derived on both engines;
[ADR-004](decisions/004-postgres-driver.md) and
[ADR-005](decisions/005-dependency-ownership.md)) and the hard
[ADR-005](decisions/005-dependency-ownership.md)), the hard
driver-coupled properties are POC-pinned (transactional enqueue,
exactly-once claim under concurrency). What is *not* yet designed is
the semantics depth — this document's subject, and [OQ-05]'s home. It
states the design space, the decided constraints any answer must fit,
and the reference material; actual semantics decisions become ADRs
when made.
exactly-once claim under concurrency), and the semantics depth is
designed — OQ-05 and OQ-09 resolved by
[ADR-010](decisions/010-queue-semantics-depth.md) and
[ADR-009](decisions/009-scheduler-collapse.md) (2026-10-05). This
document states the resulting semantics and the constraints they were
made under; the ADRs carry the WHY.
## Decided constraints (inherited, not re-opened)
- **Mechanisms**: durable at-least-once queues; the outbox is a helper
over queues; scheduler is cron/`@every` enqueueing into named
queues ([ADR-002](decisions/002-feature-scope.md)) — pending
OQ-09's collapse decision.
over queues; the scheduler collapses into queues + `schedule()` +
`run_schedules` ([ADR-002](decisions/002-feature-scope.md),
[ADR-009](decisions/009-scheduler-collapse.md)).
- **Transactional enqueue**: commit-atomic with the caller's business
write on both engines ([ADR-007](decisions/007-transactional-seam.md));
rollback drops the job row with no ghosts.
@@ -32,104 +33,211 @@ when made.
per-subscription fanout ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
- **Job options** (the contract v1 skeleton,
[ADR-008](decisions/008-contract-v1-pinning.md) §1, from the
honker-rs surface): `delay, priority, max_attempts, expires` at
enqueue; `ack / retry / fail / heartbeat` on the job handle. Deeper
`QueueOpts`/retry surface is this document's design space.
honker-rs surface): `delay, priority, max_attempts, expires, run_at`
at enqueue; `ack / retry / fail / heartbeat` on the job handle.
- **Cut-flag context**: result storage is
[ADR-002](decisions/002-feature-scope.md)'s cut-flag row; if
OQ-05's design shows queue consumers provably need
result-query-by-id, that is the channel to revisit it — with a
consumer-inventory row, not silent inclusion.
[ADR-002](decisions/002-feature-scope.md)'s cut-flag row;
[ADR-010](decisions/010-queue-semantics-depth.md) §7 reconsidered it
as part of this design and left it cut (delete-on-ack keeps
completed jobs out of storage; the pg-boss completed-row/output
model is that feature under another name). Re-entry stays gated on
a consumer-inventory row.
## Design space (what OQ-05 must pin)
## Job lifecycle (ADR-010 §1–§3a)
### Retry / backoff
```text
enqueue
│
▼
┌─────────┐ claim (attempts += 1) ┌───────────────────┐
│ pending │ ───────────────────────────► │ processing │
└─────────┘ ◄─────────────────────────── └───────────────────┘
│ retry(err, d) │ deadline = job-stamped
│ │ visibility timeout;
cancel deletes the row in EITHER │ heartbeat(extend) resets it
state (pending or processing) — ├─────────────────────────
not an interrupt; the holder's │ visibility lapse:
next ack/heartbeat returns false │ no transition fires — the
│ │ row is just reclaimable
▼ │ (a reclaim eats an attempt)
┌────────┐ fail(err) / budget │
│ dead │ ◄────── exhausted ───────────┘
└────────┘
│ ack deletes the row
│ (from processing, claim still valid)
│
▼
(gone)
- Trigger model: attempts counted from claim-completion; a job
re-appears after visibility timeout OR explicit `retry`. Does
`max_attempts` exhaustion move the job to dead-letter, flag it, or
drop it?
- Backoff curve between attempts: honker and the pg-boss family have
their own shapes (fixed, exponential); which does the contract
pin, what is configurable, what's the default?
- `heartbeat` semantics: renewal vs progress signal (or both) — and
what a missed heartbeat means (visibility re-expiry? nothing?).
expires lapse (any state) — sweep_expired(q) moves the row to
dead with last_error='expired'; also enforces dead-letter
retention. ack/cancel leave no row; get_job sees dead rows.
```
### Visibility timeouts
- **States**: `pending` → `processing` → `dead` (+ absence:
`ack`/`cancel` delete the row). `max_attempts` frozen at enqueue;
`attempts` counts every claim — fresh or reclaim.
- **Visibility**: each claim sets the deadline from the job's stamped
`visibility_timeout_s` (default 300 s, stamped per
[ADR-010](decisions/010-queue-semantics-depth.md) §3a).
`heartbeat(extend)` is renewal — an absolute reset from now (the new
full deadline, not additive); a **late heartbeat is refused** (never
steals the job back from a reclaimer). Between deadline lapse and
another worker's reclaim, the original worker may still complete —
the dual-execution window is the contract's honest at-least-once
posture; downstream idempotence is the consumer's job.
- **Reclaim consumes an attempt**: a handler that forgets to
heartbeat looks like a repeatedly-failing job and dead-letters on
budget exhaustion. Stated in the contract text
([ADR-010](decisions/010-queue-semantics-depth.md) §2), not
discovered in production.
- **Retry**: `retry(err, None)` — the engine computes the queue's
curve; `retry(err, Some(d))` overrides. `fail(err)` = immediate
dead-letter. Ordering within a queue: `priority DESC`, ready-time
ASC, enqueue order (FIFO under equal priority).
- **get_job sees dead jobs** — state, payload, attempts, timestamps,
and (dead only) `last_error`, `died_at`; post-mortem diagnosis by
API, not by SQL.
- Where the renewal lives: explicit (`heartbeat`) only, or implicit
renewal per op on the job handle? Honker's design is the SQLite-side
answer; pg-boss's is the reference. One answer, contract-pinned.
- Interaction with long business transactions (claims inside
caller txs, [ADR-007](decisions/007-transactional-seam.md)).
## Retry / backoff (ADR-010 §3, §3a)
### Dead-letter
- **The curve: equal-jitter exponential** from the job's stamped
`backoff_base_s` (default 5 s): `delay ∈ [base·2^(a−1)/2,
base·2^(a−1)]` for attempt `a` — the range is the definition (not
AWS-canonical full jitter); capped at **1 hour** (pinned contract
constant, no knob). The formula is what both engines compute
identically; the range's herding rationale lives in the ADR.
- **Where configured**: queue-level `QueueOpts
{ visibility_timeout_s, max_attempts, backoff_base_s,
dead_letter_retention_s }` (honker-parity defaults: 300 s / 3 / 5 s
/ none) — stamped onto each job row at enqueue (§3a), never read
from a live registry.
- Move-to-table (honker's `_honker_dead`, pg-boss's dead-letter
queue) vs flag-in-place. Inspection/requeue surface shape.
- Retention: does a dead-lettered job expire (an `expires`-driven
sweep — an OQ-05 sub-question) or live until explicitly cleared?
## Dead-letter (ADR-010 §4)
### Sweep / maintenance
- **Move, not flag**: dead rows physically move to engine-owned dead
storage (`last_error`, `died_at`) — never scanned by the claim path.
Triggers: explicit `fail`, `retry` at budget, exhaustion-by-reclaim,
`expires` lapses (swept).
- **Retention**: forever by default; `dead_letter_retention_s` (queue
opt) makes `sweep_expired` enforce a per-queue dead TTL. Honker's
no-retention posture kept as default; the mechanism is ours.
- **No redrive API in v1**: the recipe is `get_job` + fresh `enqueue`.
No consumer names an API shape; re-entry needs an OQ.
- `sweep_expired` is on the queue's surface; cadence ownership is
the open question. Honker's scheduler machinery (leader-elected,
missed-boundary catch-up) can run it; a consumer's own scheduler row
can too; the engine could ship a default. Decision shaped by OQ-09's
scheduler collapse — same machinery either way.
- Multi-process sweep safety on SQLite vs Postgres: named-lock
coordination (the alkblobs fleet-sweeper pattern,
[ADR-002](decisions/002-feature-scope.md)) vs native advisory locks.
## Sweep / maintenance (ADR-010 §5–§6)
### Scheduler surface
- `sweep_expired(queue)` moves *every* past-`expires_at` row (pending
**and** processing) to dead, and enforces dead-letter retention —
the **no-stranded-rows property**
([ADR-010](decisions/010-queue-semantics-depth.md) §5): a job with
an `expires` deadline is eventually in exactly one of
pending/processing/dead, never stuck unreachable. (SQLite-side
realization over honker's machinery rides OQ-06 — the zombie hole
is that read's concrete fork candidate.)
- **No ambient sweeper**: the engine ships no background maintenance,
no default cadence (the no-ambient-timers posture). Correctness of
transitions is the engine's; **timeliness is the consumer's**.
- **The "who sweeps" recipe** is the scheduler collapse's payoff
([ADR-009](decisions/009-scheduler-collapse.md)):
`store.schedule("maintenance", "@every 300s", mq, …)` +
`run_schedules(stop)` (leader-elected) + a worker handler calling
`sweep_expired` per queue. One schedule row, one worker — the
pattern every consumer would have hand-rolled.
- Multi-process sweeps need no leader lock on a bare `sweep_expired`
call (single-statement atomic per queue); the cadence coordination
comes from the scheduler machinery above. The SQLite notifications
table's hygiene is engine-internal (not a consumer chore;
`prune_notifications*` stay out of the contract, resolving ADR-008
§8's disposition).
- OQ-09's question, stated crisply: is scheduler a first-class
mechanism (own handle, methods, inspection) or queues + a
`schedule(cron_expression, queue_name, payload)` call? The
inventory's evidence is thin (documented-thin); a v1 that is queues
+ a `schedule` call with leader-election behavior documented is the
smaller contract; if a consumer needs inspectable/pausable schedule
*objects* (`add/pause/resume/update/list/remove`), that's the
full honker shape. Decide against the inventory rows, not against
the whole honker menu.
## Scheduler (ADR-009)
### Namespaces / schema layout (pg side)
Collapsed into queues: no `Scheduler` handle, no schedule objects.
- Schema-scoped DDL (the pg-boss design) vs shared-schema table
naming. Rides the reserved-namespace contract
([ADR-006](decisions/006-wake-and-delivery-contract.md),
[ADR-008](decisions/008-contract-v1-pinning.md) §4):
the reserved *channel* namespace is pinned (`__alkstore_` prefix);
queue/stream/lock **table** layout remains this document's
OQ-05 work and must be collision-proof against consumer tables in
the same database (alkblobs' ADR-008 — co-tenant-tables precedent —
at `/workspace/@alkdev/alkblobs/docs/architecture/decisions/`).
- Surface: `schedule(name, spec, queue, payload, opts)` (upsert by
name), `unschedule(name) -> bool`, `run_schedules(stop) -> Result<()>`.
Update = re-register; pause = unregister + re-register.
- **Spec grammar v1: `@every <n><unit>` only** (`s|m|h|d`). Cron
strings rejected with `InvalidSpec { spec }` (grammar rationale in
the ADR; extension path: a consumer-inventory row first).
- `run_schedules` is opt-in: schedules never fire without a runner
(the no-ambient-timers posture). The runner: leadership lock →
{renew (on loss, return before ticking — `Err(LeadershipLost)`,
distinct from clean `Ok(())` stop) → fire due boundaries → sleep}.
Leadership lock: reserved name `__alkstore_scheduler`
(engine-derived name under the reserved prefix,
[ADR-008](decisions/008-contract-v1-pinning.md) §4).
- **Boundary guarantee** ([ADR-009](decisions/009-scheduler-collapse.md)
§4): at-least-once per elapsed boundary while a leader runs; fire +
boundary-advance commit atomically under a row lock on the schedule
row (crash mid-tick refires the boundary; a rogue ticker is defeated
by the row-locked advance, not just by the leadership lock); bounded
catch-up — missed boundaries replay one-by-one up to a **fixed
contract constant cap of 64** per task per tick, beyond which
remaining boundaries skip forward. Honest loss mode, documented.
- Each boundary fire enqueues ordinary work: `payload` into the named
queue with `ScheduleOpts { priority, max_attempts, expires }` —
stamped onto the enqueued job per
[ADR-010](decisions/010-queue-semantics-depth.md) §3a; the queue
mechanism's guarantees apply, no separate delivery machinery.
- Schedule storage is engine-internal (SQLite: honker's
`_honker_scheduler_tasks`; Postgres: the engine-owned schema) — not
a consumer namespace; queues are names, not registered objects, so a
schedule may target a queue nothing has claimed yet. Schedule names
validate like queue names (non-empty, reserved-prefix rejected).
## Namespaces / schema layout (ADR-010 §8)
- **Postgres**: all engine-owned tables (job, dead, stream, offsets,
schedules, internal) in one **engine-owned schema** (default
`alkstore`, per-engine option) — co-tenancy-safe with consumer
tables (alkblobs ADR-008 precedent; the reserved *name* namespace
governs name-column values inside). **Queues are rows in one job
table** — no per-queue tables/schemas; pg-boss's partition-per-queue
opt-in is not inherited (one table + partial indexes matches honker's
proven shape and keeps `queue(name)` a name, not a DDL operation).
- **SQLite**: honker's `_honker_*` family — storage-internal
([ADR-008](decisions/008-contract-v1-pinning.md) §4); no new tables
minted by this design; the family's fate rides OQ-06 (a fork
re-owns the names; nothing consumer-visible changes).
- Partial indexes pinned identically on both engines: partial indexes
matching the claim hot path; dead rows outside it; single clock
source (second-precision timestamps); savepoint-guarded
multi-statement moves.
## Reference material
- **honker's queue design** — the SQLite-side incumbent
(`/workspace/honker`, its honker-core machinery; the engine rides it
directly, so the SQLite side's depth is largely "inherit + pin").
- **pg-boss family** — `/workspace/pgboss-rs` @ 98f7d9e (queue states,
maintenance/dead-letter behavior) and the node original (pg-boss, the
upstream of record — compare semantics the port may have dropped).
Full read: `docs/research/reference-honker-machinery.md` (schema,
claim/visibility/retry/dead-letter mechanics with file/line cites;
the defect list §8 feeds OQ-06).
- **pg-boss family** — `/workspace/pgboss-rs` @ 98f7d9e (design
reference only, [ADR-005](decisions/005-dependency-ownership.md)).
Full read: `docs/research/reference-pgboss-rs-semantics.md` (states,
backoff-with-jitter formula, dead-letter-as-queue, maintenance
postures; the node-upstream deltas §8).
- **POC ground** — `poc-pg-posture-findings.md` (the minimal queue
table + SKIP LOCKED claim measured; the property suite green);
`poc-sqlite-posture-findings.md` (the SQLite twin).
- **Consumers** — alkfs OQ-FS-14 (sync outbox), alkblobs
ops-surface/gc docs (maintenance cadences), the family-wide
"who sweeps" punt (inventory §scheduler).
- **Consumers** — alkfs OQ-FS-14 (sync outbox), alkblobs ops-surface/gc
docs (maintenance cadences), the family-wide "who sweeps" punt
(inventory §scheduler — answered by the collapse recipe above).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [002](decisions/002-feature-scope.md) | Feature scope | queues + outbox in; result storage cut-flag |
| [002](decisions/002-feature-scope.md) | Feature scope | queues + outbox in; result storage cut-flag (stands, re-checked) |
| [004](decisions/004-postgres-driver.md) | pg driver | queue machinery re-derived, pg-boss as reference |
| [005](decisions/005-dependency-ownership.md) | Ownership | design-reference postures for the queue family |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | wake-driven consumption, guarantee table |
| [007](decisions/007-transactional-seam.md) | Tx seam | enqueue_tx commit-atomicity |
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | queue skeleton + EnqueueOpts pinned; depth is OQ-05's extension path |
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | queue skeleton + EnqueueOpts pinned; depth was this doc's design space |
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | queues + `schedule()` + runner; `@every`-only; boundary guarantee row |
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | visibility/heartbeat rules, backoff curve, dead-letter, no-stranded-rows sweep, schema layout |
## Open Questions
@@ -137,7 +245,10 @@ Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-05**: the semantics-depth pinning this document frames (open,
high priority)
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
(open)
- **OQ-06**: honker-core quality read — the SQLite-side zombie fix and
dead-row visibility are concrete fork candidates for it to weigh
(open).
Resolved on this document's surface: **OQ-05** and **OQ-09**
(2026-10-05, [ADR-010](decisions/010-queue-semantics-depth.md) /
[ADR-009](decisions/009-scheduler-collapse.md)).