docs: open Phase 1 — architecture spec set over the Phase 0 evidence
docs/architecture/ now exists: README index, overview, five component specs (core-contract, engine-sqlite, engine-postgres, queues, deployment), ADR-001..007 carrying the Phase 0 resolved decisions (crate split, feature scope, per-engine drivers, dependency ownership, wake contract, tx seam), and the centralized open-questions tracker promotion: OQ-ST-01..08 mirror to OQ-01..08 one-to-one with statuses/resolutions carried; new Phase 1 questions append (OQ-09 scheduler collapse, OQ-10 contract versioning). Open Phase 1 work: OQ-04 contract pinning (high), OQ-05 queue semantics depth (high), OQ-06 honker-core quality read (high; fork-trigger gate), OQ-08 capability surface, OQ-09, OQ-10. Erratum fixed in phase-0 OQ-ST-04 (thread-affinity friction is SQLite-side, previously garbled as pg-side) and a stale scheduler- boundary pointer corrected in consumer-inventory.md. Two review passes run (findings: OQ-promotion numbering faithfulness, ADR back-reference sync) — all critical/warning findings resolved.
This commit is contained in:
1 parent
db73678090
commit
4391f6e879
17 files changed
+1812
-14
No files matched your search
@@ -0,0 +1,138 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-04
|
||||
---
|
||||
|
||||
# Queues, scheduler, outbox — semantics depth
|
||||
|
||||
The queue family's *posture* is decided (re-derived on both engines;
|
||||
[ADR-004](decisions/004-postgres-driver.md) and
|
||||
[ADR-005](decisions/005-dependency-ownership.md)) and the hard
|
||||
driver-coupled properties are POC-pinned (transactional enqueue,
|
||||
exactly-once claim under concurrency). What is *not* yet designed is
|
||||
the semantics depth — this document's subject, and [OQ-05]'s home. It
|
||||
states the design space, the decided constraints any answer must fit,
|
||||
and the reference material; actual semantics decisions become ADRs
|
||||
when made.
|
||||
|
||||
## Decided constraints (inherited, not re-opened)
|
||||
|
||||
- **Mechanisms**: durable at-least-once queues; the outbox is a helper
|
||||
over queues; scheduler is cron/`@every` enqueueing into named
|
||||
queues ([ADR-002](decisions/002-feature-scope.md)) — pending
|
||||
OQ-09's collapse decision.
|
||||
- **Transactional enqueue**: commit-atomic with the caller's business
|
||||
write on both engines ([ADR-007](decisions/007-transactional-seam.md));
|
||||
rollback drops the job row with no ghosts.
|
||||
- **Exactly-once claim**: `FOR UPDATE SKIP LOCKED` (pg, POC-pinned
|
||||
under 4×4 concurrency) / honker's machinery (SQLite). At-least-once
|
||||
*work*; exactly-once *processing* needs the ack + visibility model.
|
||||
- **Wake-driven consumption**: the pg engine's default is LISTEN-driven
|
||||
claim with a re-poll safety net; SQLite's is watcher wake with
|
||||
per-subscription fanout ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||
- **Job options** (the honker-rs surface, the contract starting
|
||||
artifact): `delay, priority, max_attempts, expires` at enqueue;
|
||||
`ack / retry / fail / heartbeat` on the job handle.
|
||||
- **Cut-flag context**: result storage is
|
||||
[ADR-002](decisions/002-feature-scope.md)'s cut-flag row; if
|
||||
OQ-05's design shows queue consumers provably need
|
||||
result-query-by-id, that is the channel to revisit it — with a
|
||||
consumer-inventory row, not silent inclusion.
|
||||
|
||||
## Design space (what OQ-05 must pin)
|
||||
|
||||
### Retry / backoff
|
||||
|
||||
- Trigger model: attempts counted from claim-completion; a job
|
||||
re-appears after visibility timeout OR explicit `retry`. Does
|
||||
`max_attempts` exhaustion move the job to dead-letter, flag it, or
|
||||
drop it?
|
||||
- Backoff curve between attempts: honker and the pg-boss family have
|
||||
their own shapes (fixed, exponential); which does the contract
|
||||
pin, what is configurable, what's the default?
|
||||
- `heartbeat` semantics: renewal vs progress signal (or both) — and
|
||||
what a missed heartbeat means (visibility re-expiry? nothing?).
|
||||
|
||||
### Visibility timeouts
|
||||
|
||||
- Where the renewal lives: explicit (`heartbeat`) only, or implicit
|
||||
renewal per op on the job handle? Honker's design is the SQLite-side
|
||||
answer; pg-boss's is the reference. One answer, contract-pinned.
|
||||
- Interaction with long business transactions (claims inside
|
||||
caller txs, [ADR-007](decisions/007-transactional-seam.md)).
|
||||
|
||||
### Dead-letter
|
||||
|
||||
- Move-to-table (honker's `_honker_dead`, pg-boss's dead-letter
|
||||
queue) vs flag-in-place. Inspection/requeue surface shape.
|
||||
- Retention: does a dead-lettered job expire (an `expires`-driven
|
||||
sweep — an OQ-05 sub-question) or live until explicitly cleared?
|
||||
|
||||
### Sweep / maintenance
|
||||
|
||||
- `sweep_expired` is on the queue's surface; cadence ownership is
|
||||
the open question. Honker's scheduler machinery (leader-elected,
|
||||
missed-boundary catch-up) can run it; a consumer's own scheduler row
|
||||
can too; the engine could ship a default. Decision shaped by OQ-09's
|
||||
scheduler collapse — same machinery either way.
|
||||
- Multi-process sweep safety on SQLite vs Postgres: named-lock
|
||||
coordination (the alkblobs fleet-sweeper pattern,
|
||||
[ADR-002](decisions/002-feature-scope.md)) vs native advisory locks.
|
||||
|
||||
### Scheduler surface
|
||||
|
||||
- OQ-09's question, stated crisply: is scheduler a first-class
|
||||
mechanism (own handle, methods, inspection) or queues + a
|
||||
`schedule(cron_expression, queue_name, payload)` call? The
|
||||
inventory's evidence is thin (documented-thin); a v1 that is queues
|
||||
+ a `schedule` call with leader-election behavior documented is the
|
||||
smaller contract; if a consumer needs inspectable/pausable schedule
|
||||
*objects* (`add/pause/resume/update/list/remove`), that's the
|
||||
full honker shape. Decide against the inventory rows, not against
|
||||
the whole honker menu.
|
||||
|
||||
### Namespaces / schema layout (pg side)
|
||||
|
||||
- Schema-scoped DDL (the pg-boss design) vs shared-schema table
|
||||
naming. Rides the reserved-namespace contract
|
||||
([ADR-006](decisions/006-wake-and-delivery-contract.md), OQ-04):
|
||||
queue/stream/lock tables and the reserved channel prefix
|
||||
must be collision-proof against consumer tables in the same
|
||||
database (the alkblobs co-tenant-tables precedent — ADR-008 in
|
||||
`/workspace/@alkdev/alkblobs/docs/architecture/decisions/`).
|
||||
|
||||
## Reference material
|
||||
|
||||
- **honker's queue design** — the SQLite-side incumbent
|
||||
(`/workspace/honker`, its honker-core machinery; the engine rides it
|
||||
directly, so the SQLite side's depth is largely "inherit + pin").
|
||||
- **pg-boss family** — `/workspace/pgboss-rs` @ 98f7d9e (queue states,
|
||||
maintenance/dead-letter behavior) and the node original (pg-boss, the
|
||||
upstream of record — compare semantics the port may have dropped).
|
||||
- **POC ground** — `poc-pg-posture-findings.md` (the minimal queue
|
||||
table + SKIP LOCKED claim measured; the property suite green);
|
||||
`poc-sqlite-posture-findings.md` (the SQLite twin).
|
||||
- **Consumers** — alkfs OQ-FS-14 (sync outbox), alkblobs
|
||||
ops-surface/gc docs (maintenance cadences), the family-wide
|
||||
"who sweeps" punt (inventory §scheduler).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| ADR | Decision | Summary |
|
||||
|---|---|---|
|
||||
| [002](decisions/002-feature-scope.md) | Feature scope | queues + outbox in; result storage cut-flag |
|
||||
| [004](decisions/004-postgres-driver.md) | pg driver | queue machinery re-derived, pg-boss as reference |
|
||||
| [005](decisions/005-dependency-ownership.md) | Ownership | design-reference postures for the queue family |
|
||||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | wake-driven consumption, guarantee table |
|
||||
| [007](decisions/007-transactional-seam.md) | Tx seam | enqueue_tx commit-atomicity |
|
||||
|
||||
## Open Questions
|
||||
|
||||
Open questions are tracked in
|
||||
[open-questions.md](open-questions.md). Key
|
||||
questions affecting this document:
|
||||
|
||||
- **OQ-05**: the semantics-depth pinning this document frames (open,
|
||||
high priority)
|
||||
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
|
||||
(open)
|
||||
Reference in new issue
Block a user