docs: open Phase 1 — architecture spec set over the Phase 0 evidence

docs/architecture/ now exists: README index, overview, five component
specs (core-contract, engine-sqlite, engine-postgres, queues,
deployment), ADR-001..007 carrying the Phase 0 resolved decisions
(crate split, feature scope, per-engine drivers, dependency
ownership, wake contract, tx seam), and the centralized
open-questions tracker promotion: OQ-ST-01..08 mirror to OQ-01..08
one-to-one with statuses/resolutions carried; new Phase 1 questions
append (OQ-09 scheduler collapse, OQ-10 contract versioning).
Open Phase 1 work: OQ-04 contract pinning (high), OQ-05 queue
semantics depth (high), OQ-06 honker-core quality read (high;
fork-trigger gate), OQ-08 capability surface, OQ-09, OQ-10.

Erratum fixed in phase-0 OQ-ST-04 (thread-affinity friction is
SQLite-side, previously garbled as pg-side) and a stale scheduler-
boundary pointer corrected in consumer-inventory.md. Two review
passes run (findings: OQ-promotion numbering faithfulness, ADR
back-reference sync) — all critical/warning findings resolved.
This commit is contained in:
glm-5.3-flash committed 2026-10-04 18:13:10 +00:00
1 parent db73678090
commit 4391f6e879
17 files changed
+1812 -14

No files matched your search

+92
View File
@@ -0,0 +1,92 @@
---
status: draft
last_updated: 2026-10-04
---
# alkstore — Architecture
Architecture documentation for the alkstore project: one reactive
store interface (notify, streams, queues, locks, scheduler, outbox)
over SQLite and Postgres, with each engine native underneath
(see [overview.md](overview.md)).
## Current State
**Phase 1 (Architecture) — in progress.** Phase 0 is complete
(`docs/research/phase-0.md`): both POCs ran and passed, the scope
inventory is confirmed, and the crate split, drivers, and ownership
postures are decided. This directory carries the architecture spec
build-out over that evidence base; all spec documents are `draft`
pending architecture review and OQ resolution.
## Architecture Documents
| Doc | Status | Purpose | Key OQs |
|---|---|---|---|
| [overview.md](overview.md) | draft | Crate family, feature surface, non-goals, evidence base | — |
| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-04, OQ-08, OQ-09, OQ-10 |
| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: honker-core/rusqlite mapping | OQ-05, OQ-06, OQ-09 |
| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-05, OQ-08, OQ-09 |
| [queues.md](queues.md) | draft | Queue/scheduler/outbox semantics depth frame | OQ-05, OQ-09 |
| [deployment.md](deployment.md) | draft | Host semantics, connection budgets, knobs, matrix | OQ-04, OQ-08 |
| [open-questions.md](open-questions.md) | draft | OQ tracker (promoted from OQ-ST register) | — |
## Architecture Decision Records
| ADR | Title | Status |
|---|---|---|
| [001](decisions/001-crate-split.md) | Reactive-core crate + per-engine crates | Accepted |
| [002](decisions/002-feature-scope.md) | Feature scope — inventory-confirmed surface | Accepted |
| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core, bridged seam | Accepted |
| [004](decisions/004-postgres-driver.md) | Postgres engine — tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract — opaque wake + re-read; notify-vs-streams split | Accepted |
| [007](decisions/007-transactional-seam.md) | Transactional seam — caller-held `TxHandle`, `*_tx` methods | Accepted |
## Open Questions
Tracked in [open-questions.md](open-questions.md) (OQ-01..NN; the
Phase 0 register's OQ-ST-01..08 promote one-to-one — OQ-NN mirrors
OQ-ST-NN — with new Phase 1 questions appended after). Highlights,
in suggested resolution order (OQ-04 first):
- **OQ-04** (high): contract pinning — exact trait shape, handle
representation, error taxonomy, reserved strings, guarantee rows
for locks/scheduler.
- **OQ-05** (high): queue semantics depth — retry/backoff/dead-letter/
sweep design.
- **OQ-06** (high): honker-core quality read — fork-trigger gate.
- **OQ-09** (medium): scheduler as first-class mechanism vs queues +
`schedule()`.
- **OQ-08** (medium): capability-surface shape.
- **OQ-10** (medium): contract versioning across engine crates.
Resolved (Phase 0, kept with resolutions): OQ-01 (feature scope),
OQ-02 (crate split), OQ-03 (drivers), OQ-07 (extension surface cut).
No deferred OQs: all open questions are actionable Phase 1 work with
complete evidence bases.
## Document Lifecycle
| Status | Meaning | Transitions |
|---|---|---|
| `draft` | Under active development; may change significantly | → `reviewed` when the doc's OQs are resolved |
| `reviewed` | Architecture final; implementation may begin | → `stable` when implementation verified |
| `stable` | Locked; changes need review, may warrant an ADR | → `deprecated` when superseded |
| `deprecated` | Superseded; kept for reference | Removed when unreferenced |
All spec documents carry YAML frontmatter (`status`, `last_updated`);
ADRs carry a `## Status` section (Accepted/Proposed/Superseded).
## Provenance of decisions
Phase 1 inherits its decisions from Phase 0's evidence base — every
Accepted ADR above cites its POC findings and register record. The
research documents remain the deep background:
- `docs/research/phase-0.md` — vision, prior art, OQ-ST register,
convergence.
- `docs/research/consumer-inventory.md` — per-feature scope evidence.
- `docs/research/poc-sqlite-posture-findings.md` /
`docs/research/poc-pg-posture-findings.md` — measured ground.
+204
View File
@@ -0,0 +1,204 @@
---
status: draft
last_updated: 2026-10-04
---
# Core contract
The unified, engine-agnostic surface a `Store` exposes. This document
specifies WHAT the contract is; the per-engine specs map it onto their
machinery; ADRs carry the WHY. The starting artifact is the honker-rs
surface (the Phase 0 §Interface finding) scoped to the
inventory-confirmed features — [ADR-002](decisions/002-feature-scope.md)
— and pinned against it. Exact Rust shapes are OQ-04's pinning work;
the *obligations* are this document.
## Concepts
- **Store** — what a consumer opens from a connection string or file
path ([ADR-001](decisions/001-crate-split.md)): one handle, the engine
behind it chosen at open time. Consumer code never branches on
engine type (guiding principle 4).
- **Mechanism** — one of the surface's coordination families: notify,
streams, queues (+outbox), locks, scheduler. Delivery guarantees are
pinned per mechanism in [ADR-006](decisions/006-wake-and-delivery-contract.md)'s
table (notify/streams/queues); lock and scheduler guarantee rows are
OQ-04/OQ-09 resolution items, not assumed.
*(Guiding principles are defined in `docs/research/phase-0.md`
§Vision and cited by number throughout this directory.)*
- **Wake** — the opaque "something changed, re-read" signal the wake
contract delivers ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
The contract's central abstraction.
- **TxHandle** — the caller-held transaction object whose `*_tx`
methods make side effects commit-atomic with a business write
([ADR-007](decisions/007-transactional-seam.md)).
## The seam
```text
Store::open(config) -> Store
Store::begin_tx() -> TxHandle
```
Mechanism handles come off the store (or, for transactional variants,
off the handle):
```text
store.notify(channel, payload) handle.notify_tx(channel, payload)
store.stream(name) -> Stream handle.publish_tx / save_offset_tx
store.queue(name) -> Queue handle.enqueue_tx
store.try_lock(name, ttl) -> Lock
```
The exact shape of these handles (traits vs concrete types, `dyn` vs
generic `TxHandle`) is OQ-04's pinning work.
## Mechanism contracts
### notify / listen
Fire-and-forget signals, commit-atomic when sent in a transaction
([ADR-007](decisions/007-transactional-seam.md)). No durability, no
replay, no per-listener retry; a listener attached after a commit
never sees it.
- `notify(channel, payload)` — payload ≤ 8000 bytes on Postgres
(client-side checked, typed error, verified by POC #2); no limit on
SQLite. The asymmetry is a documented engine capability, see
OQ-04/OQ-08.
- `listen(channel)` — starts from "now"; delivers
opaque wakes ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
Wakes are at-least-once, possibly coalesced (SQLite) or per-notify
(Postgres), possibly repeated after reconnect. **Consumers must be
idempotent on wake.** (The sketched return type — a wake receiver vs
a subscription handle — is an OQ-04 pinning item, like all exact
shapes here.)
- Failure surfaces as channel events, never silence: watcher death
closes the receiver (SQLite); the synthetic reconnect-wake (a
reserved channel, [ADR-004](decisions/004-postgres-driver.md)) covers
Postgres connection gaps.
- Channel name is the one piece of semantic content a wake carries —
the invalidation key for caching subscribers.
### streams
Durable pub/sub with per-consumer offsets
([ADR-006](decisions/006-wake-and-delivery-contract.md)). The durable
cousin of notify: publish is commit-atomic; every committed event is
readable by every consumer whose offset hasn't passed it;
replay-on-attach is the default; offsets are explicit and
transaction-aware (`save_offset_tx` gives exactly-once-within-a-
business-tx shape).
- `publish` / `publish_tx` / `publish_with_key` — append to the
stream's durable log.
- `read_since` / `read_from_consumer(offset)` — cursor-based reads.
- `save_offset` / `save_offset_tx` — consumer checkpoint, explicit;
**the contract's save is always explicit** (no auto-checkpoint
cadence — honker's per-binding ambiguity is not inherited).
- `subscribe(consumer)` — durable consumption: attach, read to
current tail, resume after restart from the stored offset.
### queues
Durable at-least-once work
([ADR-002](decisions/002-feature-scope.md)). The semantics-depth design
is [queues.md](queues.md)'s (OQ-05); the contract-level obligations
here:
- `enqueue` / `enqueue_tx` — commit-atomic; `EnqueueOpts { delay,
priority, max_attempts, expires, ... }`.
- `claim` / `claim_batch` — exactly-once handout under concurrency
(POC-pinned on both engines).
- Job handle: `ack / retry / fail / heartbeat`; visibility timeouts
make at-least-once *safe* (work re-appears if a claimant dies).
- `sweep_expired` — maintenance entry point (design in
[queues.md](queues.md)).
### named locks
TTL-bounded coordination locks, transactional-friendly:
- `try_lock(name, ttl)` — acquire or fail; `renew`; release on
explicit unlock or TTL expiry. Re-acquirable after expiry
(POC-pinned on Postgres; on SQLite it rests on honker's machinery —
the SQLite-side pin is in OQ-04's verification backlog).
- Lock names are shared-namespace (contract text pins collision
rules with the reserved namespace, OQ-04).
### outbox
A helper over queues, not a separate mechanism: enqueue inside the
business transaction + the delivery/consumption worker entry points.
Same evidence base and guarantee as queues
([ADR-002](decisions/002-feature-scope.md)).
### scheduler
Cron/`@every` enqueueing into named queues, leader-elected where the
engine has peers (SQLite: single-host, no election needed beyond
documented posture; Postgres: advisory-lock election). Whether this is
a first-class mechanism or queues + `schedule()` is OQ-09.
## Cross-cutting contracts
### Delivery-guarantee table
The per-mechanism table in
[ADR-006](decisions/006-wake-and-delivery-contract.md) is the contract
of record for notify/streams/queues; this spec inherits it and adds
the consumer obligations:
- wake idempotence (notify's at-least-once, coalescing behavior);
- explicit offset saves (streams);
- visibility-timeout budgeting (queues);
- `*_tx` operations are *only* durable after the caller's commit —
rollback drops job rows, event rows, notifications, and offset saves
together (the no-ghosts property, POC-pinned on both engines).
### Errors
`thiserror`-typed, per the family standard. The taxonomy (what
callers match on per mechanism; engine-capability errors like
`PayloadTooLarge` on pg vs no-limit on SQLite) is OQ-04's pinning
work.
### Capability surface
Whether the `Store` exposes engine capabilities at all — and if so,
which (payload limits, host semantics, wake-cadence knobs) — is
OQ-08's decision ([deployment.md](deployment.md)).
### Naming / reserved namespace
Consumer-visible names (channels, streams, queues, locks) share
engine-visible namespaces on Postgres (LISTEN channel names are
server-global per database). Exact rules: OQ-04. Decided already: a
**reserved/meta namespace exists** for engine-internal names — the
Postgres reconnect-wake channel, any internal bookkeeping — and
consumer names must not collide with it
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [001](decisions/001-crate-split.md) | Crate split | core + per-engine crates; single-driver binaries |
| [002](decisions/002-feature-scope.md) | Feature scope | inventory-confirmed features only; cut-flags explicit |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | opaque wake + re-read; notify-vs-streams guarantee split |
| [007](decisions/007-transactional-seam.md) | Tx seam | caller-held handle, `*_tx` methods, native commit-atomicity |
## Open Questions
Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-04**: exact trait-surface pinning — handle shape, error
taxonomy, reserved strings, rename/grouping decisions (open,
high priority)
- **OQ-10**: contract versioning discipline across engine crates (open)
- **OQ-08**: capability-surface shape (open)
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
(open)
@@ -0,0 +1,88 @@
# ADR-001: Reactive-core crate + per-engine crates
## Status
Accepted
## Context
The store must present one reactive interface (notify, streams, queues,
locks, scheduler, outbox) over two backing engines, SQLite and Postgres.
The engines differ sharply in how much of that surface is natively
theirs:
- The **SQLite** engine rides honker's existing, battle-tested machinery
(published `honker-core` — a port-and-adapt of a sync library to the
family's async-facing-trait posture).
- The **Postgres** engine is the build-heavy side: LISTEN/NOTIFY wiring
and the pg-boss-family queue schema are built by this crate.
A single crate with feature-gated engines would force both engines'
dependency graphs into every downstream binary that enables both
features. That is not hypothetical: rusqlite 0.40's `libsqlite3-sys`
link collision with sqlx's sqlite driver is a real constraint
(see [ADR-003](003-sqlite-driver.md)); single-driver binaries are the
only clean escape.
Scope evidence lives in `docs/research/consumer-inventory.md`; the
operator decision was recorded 2026-10-04 against
OQ-ST-02 (`docs/research/phase-0.md`).
## Decision
The project ships as a **family of crates**:
1. **`alkstore` (core)** — the crate carrying the unified trait
surface, types, error model, capability flags, and the contract
documentation. No driver dependencies. Compile-lean by construction:
the base crate has no engine machinery to keep out.
2. **`alkstore-sqlite`** — the SQLite engine implementing the core
surface. Single driver: rusqlite + honker-core ([ADR-003]).
3. **`alkstore-postgres`** — the Postgres engine implementing the core
surface. Single driver: tokio-postgres + deadpool-postgres
([ADR-004]).
4. **A mem-shaped engine** (in-tree or separate test crate) may exist as
a third implementation for tests and doctests, if the test story
turns out to want it. Treated as an implementation convenience, not
a contract artifact — decided at implementation time, not now.
Downstream consumers depend on `alkstore` plus exactly one engine
crate. A consumer binary links at most one driver.
## Consequences
**Positive**
- The engines' asymmetry of work is structural: the SQLite engine has a
small, mostly-wiring job; the Postgres engine carries the build-heavy
LISTEN/queue work. Neither is burdened by the other's dependencies.
- Any future engine (alkfs's in-tree needs, an ops-surface engine) is
additive — a new crate implementing the core traits — rather than a
feature-graph edit to one crate.
- Core-lean is structural, not a feature-discipline to be enforced by
review.
- The libsqlite3-sys link-collision class of problems is eliminated by
construction (single-driver binaries).
- Per-engine compilation/test gates are independent.
**Negative**
- A version-coordination duty: the core's trait surface is a contract
the engine crates must track. Version bumps in core must be adopted
by engines in lockstep when the contract changes (minor/major
discipline; no trait-default drift).
- Slightly more crate plumbing; naming/publishing overhead.
- Consumers who want *both* engines in one binary (rare, unsupported-by
design) cannot get a both-features build today.
## References
- OQ-ST-02 (`docs/research/phase-0.md`) — the decision record with
options considered.
- `docs/research/consumer-inventory.md` — the inventory whose lean
(single crate) was superseded, with the correction recorded there.
- [ADR-003](003-sqlite-driver.md) — the SQLite driver choice whose
link-collision constraint motivates this split.
- [ADR-004](004-postgres-driver.md) — the Postgres driver choice.
- OQ-10 (`docs/architecture/open-questions.md`) — trait-versioning
duties created here.
@@ -0,0 +1,99 @@
# ADR-002: Feature scope — what the store surface includes
## Status
Accepted
## Context
Honker's full surface (the interface prior art) covers nine feature
families: notify/listen, streams, queues, scheduler, outbox, named
locks, rate limits, result storage, and loadable-extension serving.
The crate must decide which of these it ships, and the evidence-first
answer (`docs/research/consumer-inventory.md`, 2026-10-04) grades each
row by whether any consumer document actually names a need. This ADR
fixes that scope so contract work isn't done against features no
consumer wants.
## Decision
**In scope, first-class** (a named consumer is pinned or the operator
has recorded the need):
- **notify / listen** — the transactional fire-and-forget signal layer.
Pinned (alkfs path-tree invalidation; the `watch_fires_on_commit`
POC evidence) and documented (alkblobs fleet visibility).
- **streams** — durable pub/sub with per-consumer offsets and
replay-on-attach. Operator-authority record (2026-10-04,
`consumer-inventory.md`): type-filtered event watching / repo-change
subscriptions that notify cannot honestly serve (no replay, no
durability).
**In scope** (documented need, thinner):
- **named locks** — TTL coordination locks. Pinned (alkblobs fleet
sweeper) and documented (alkfs writer coordination); alkgit's CAS has
an alternative design a lock is allowed to replace but needn't.
- **queues** — durable at-least-once work with the transactional
enqueue shape. Documented (alkfs sync/fetch-on-miss outbox; alkblobs
maintenance cadence).
- **outbox helper** — a thin helper *over* queues (enqueue inside the
business transaction + a delivery/consumption worker shape), not a
separately-needed feature. Same evidence as queues.
- **scheduler** — cron/`@every` enqueueing into named queues,
leader-elected. Documented-thin; the family-wide
"who sweeps/renews/reaps" need. May collapse into queues if design
shows scheduler = queues + tick (tracked in OQ-09).
**Cut-flag** (no named consumer; not silently included):
- **rate limits** — alkgit enforces budgets in its own wire layer; a
store-level rate limit is not the proven mechanism. The row stays
listed with its flag visible; cut at implementation time rather than
carried on momentum. If a distributed (cross-process) rate-limit need
materializes, it gets a consumer-inventory row first.
- **result storage** (`save_result`/`get_result`/`sweep_results`) —
adjacent-tooling shape, useful but named by no consumer. Same
cut-flag treatment.
**Out:**
- The **loadable-extension surface** — no consumer needs it; every
identified consumer is in-process Rust attaching to its own
connection (OQ-ST-07, resolved cut-only). Cut unless a consumer
appears.
- Honker's exclusion lines, inherited as out: workflow DAGs, task
chains/chords, multi-writer replication, distributed (cross-machine)
locking. They stay out unless a consumer document grows a row.
New consumers add a row to `docs/research/consumer-inventory.md`
*before* being assumed into scope.
## Consequences
**Positive**
- The contract surface is small and every part of it has a nameable
consumer — contract-pinning work isn't spent on speculative surface.
- Scope honesty is mechanical: the inventory is the ledger, and the
cut-flag rows keep their flags visible instead of being silently
dropped or silently shipped.
**Negative**
- Rate limits and result storage, if ever needed, require the consumer
inventory + contract process first — no ad-hoc additions.
- Cutting the extension surface narrows honker compatibility; any
future non-Rust consumer of these SQLite files cannot use honker's
extension (acceptable: none exists).
## References
- `docs/research/consumer-inventory.md` — per-feature evidence grades.
- OQ-ST-01, OQ-ST-07 (`docs/research/phase-0.md`) — resolved by this
evidence; promoted as OQ-01 and OQ-07 in
`docs/architecture/open-questions.md`.
- [core-contract.md](../core-contract.md) — the trait surface this
scope defines.
- [ADR-007](007-transactional-seam.md) — the transactional property all
in-scope enqueue/publish/notify features lean on.
@@ -0,0 +1,104 @@
# ADR-003: SQLite engine — rusqlite + published honker-core, bridged at the trait seam
## Status
Accepted
## Context
The SQLite engine's options were never a single "rusqlite vs sqlx"
axis (OQ-ST-03). Three distinct postures existed:
1. **honker-core on our own rusqlite connection** — we own
connection/schema/watcher wiring; honker supplies the SQL-function
machinery (`attach_notify`, `attach_honker_functions`,
`bootstrap_honker_schema`).
2. **honker-rs as the SQLite substrate** (`Database::open` — honker
holds its own connections; sync-only; transactions pin the
connection mutex).
3. **raw SQL over sqlx-sqlite with the honker loadable extension**
(`.so` built from published source, loaded per pool connection).
POC #1 (`docs/research/poc-sqlite-posture-findings.md`, ran 2026-10-04;
POC #2 is its pg twin) measured postures 1
and 3 end-to-end and settled the choice. Constraint facts that outlive
the comparison:
- rusqlite 0.40.x (what honker-core 0.5.0 pins) needs rustc ≥ 1.99
(`cfg_select!`) — a deployment note for any binary linking it.
- sqlx's `libsqlite3-sys` range collides with rusqlite 0.40's — mixed
rusqlite+sqlx binaries need a vendored one-line patch today. The
per-engine-crate split ([ADR-001]) keeps engine binaries
single-driver, dissolving this.
- The async-facing-trait + sync-bridge posture is family-standard
(alktty REQ-TTY-01; alkblobs store-api.md): bridge-at-the-seam is a
supported posture, not a workaround — which removes native-async's
main differentiator before any measurement.
## Decision
The SQLite engine uses **posture 1**: published `honker-core = 0.5`
as a library dependency over the crate's own rusqlite connections
(`bundled-sqlite` for hermetic builds).
- **Driver**: rusqlite, riding honker-core's pinned version.
- **Async seam**: bridged. The writer slot (`Writer`) +
reader pool (`Readers`) + every engine call in `spawn_blocking`.
A dedicated std-thread bridge (one thread owning the writer conn,
ops over mpsc) is the recorded optimization path if seam throughput
ever demands it — not the default.
- **Wake**: honker-core's `SharedUpdateWatcher` inherited (p50 ≈ 1.4 ms
at the default 1 ms `PRAGMA data_version` cadence, with
battle-tested failure handling — subscriber senders close on watcher
death, never silent-hang). A self-built thin watcher was benchmarked
only to confirm honker-core's is tighter (p50 1.40 vs 2.15 ms; max
29 vs 172 ms); no hybrid is warranted.
- **Transactions**: the engine opens `BEGIN IMMEDIATE` on the writer
slot and hands back a caller-held tx handle ([ADR-007]). The slot
lease models WAL's single-writer honestly: a long transaction parks
the writer — that is SQLite's shape, not an emulation artifact.
- **No `.so` runtime artifact, no vendored patches** in the engine
crate.
Rejected alternatives, briefly, for the record: posture 3 is *viable*
but pays ~2× p50 seam cost, a runtime dlopen artifact, wider
dependency tree, and sqlx 0.9 ergonomic warts (unsafe `extension()`,
`SqlSafeStr`) for no compensating advantage. Posture 2 (honker-rs)
was never measured because posture 1 dominates it on control with
comparable reuse.
## Consequences
**Positive**
- ~0.35 ms p50 transactional seam (measured), ~1.4 ms wake latency,
and honker-core's watcher failure-handling inherited for free.
- Static linkage: "open a path, get a store" needs no runtime
artifacts.
- The engine's job shrinks to wiring + the SQL surfaces honker doesn't
cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's
machinery directly) + the async bridge.
**Negative**
- honker-core pins rusqlite ^0.40.1; version movement in honker-core
moves our rusqlite. A honker-core quality read is the recorded fork
trigger ([ADR-005], OQ-06).
- rustc ≥ 1.99 required by rusqlite 0.40.x — binaries linking this
engine carry that toolchain floor (deployment-matrix row,
[deployment.md](../deployment.md)).
- The sync bridge means engine ops pay a `spawn_blocking` hop — mostly
invisible next to SQLite write costs, but it shapes the tx-handle
design ([ADR-007]).
## References
- `docs/research/poc-sqlite-posture-findings.md` — the measurements.
- OQ-ST-03 (`docs/research/phase-0.md`) — resolution record.
- [ADR-001](001-crate-split.md) — why the engine crate is
single-driver.
- [ADR-004](004-postgres-driver.md) — the Postgres counterpart.
- [ADR-007](007-transactional-seam.md) — the tx-handle shape both
engines implement.
- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this
decision defines.
@@ -0,0 +1,104 @@
# ADR-004: Postgres engine — tokio-postgres + deadpool-postgres, hand-rolled LISTEN forwarder
## Status
Accepted
## Context
The Postgres engine's driver question (OQ-ST-03, Postgres half) had
three candidate families: sqlx (single API across engines, and the
driver pgboss-rs carries), tokio-postgres + deadpool-postgres (the
alkblobs POC evidence base, natively async), and adopt/fork pgboss-rs
(bringing sqlx where the queue lives). POC #2
(`docs/research/poc-pg-posture-findings.md`, ran 2026-10-04) validated
the tokio-postgres posture end-to-end and measured the pieces the
contract depends on. The sqlite-arm comparison facts from
[ADR-003] apply on this side too: pgboss-rs brings sqlx (a second
driver per binary), has no LISTEN/NOTIFY at all (verified against the
checkout — consumption is `fetch_job` polling), so the push-reactivity
half is this crate's work regardless of fork-or-adopt.
## Decision
The Postgres engine uses:
- **tokio-postgres 0.7.x + deadpool-postgres 0.14.x** — pooled
connections for queries/claims (exactly-once via
`FOR UPDATE SKIP LOCKED`), natively async (client is `Send + Sync` —
no bridge, no spawn_blocking, the tx handle holds the pooled object
directly).
- **A hand-rolled LISTEN forwarder** (the POC's ~90-line shape) — a
dedicated non-pooled listener connection per process, a poll-message
loop fanning out to a bounded broadcast channel, immediate reconnect
with exponential backoff (50 ms → 2 s cap), re-LISTEN from the
channel list after every reconnect, and a **synthetic reconnect-wake
on a reserved channel** closing the no-replay hole
([ADR-006]). `postgres-notify` 0.3.8 was evaluated in-probe and
passed over (derive-not-adopt: lazy reconnect, connect_script
skipped at initial connect, unquoted-identifier LISTENs,
single-maintainer posture; it stays a recorded fallback if upstream
improves).
- **Queue machinery re-derived on this driver** with the pg-boss
schema family as design reference — not adopted/forked as a
dependency ([ADR-005]; semantics depth is [queues.md]'s work,
OQ-ST-05). The minimal-queue ground is measured: enqueue/claim/ack
with `SKIP LOCKED` is ~40 lines of SQL and passed the full property
suite.
- **Default consumption posture: LISTEN-driven claim with a re-poll
safety net**; poll-only remains the fallback (interval-tunable) when
a LISTEN connection is unavailable or unwanted. Measured: claim
latency p50 3–6 ms LISTEN-driven vs 32–50 ms at a 50 ms poll.
Structural constraints the engine owns (all test-pinned in POC #2):
- Pooled connections **cannot carry LISTEN** (deadpool#360 —
registers server-side, never delivers). The listener connection is a
per-process budget line outside the pool (`max_size + 1`).
- `pg_notify` payloads are ≤ 8000 bytes; the engine checks the limit
client-side and returns a typed error before the round-trip. Large
payloads ride a table row with the id in the notification (the
outbox shape).
- A query-vs-poll starvation deadlock exists if the listener's poll
loop isn't running before the first client query on the connection;
and dropping the `Client` closes the server session even if the
`Connection` task survives. Both are hand-rolled-forwarder failure
modes, owned and test-pinned by the engine.
## Consequences
**Positive**
- Single-driver engine, natively async, zero version conflicts, no
vendoring.
- The push channel pgboss-rs lacks, this engine gets — measured at
5–16× claim-latency improvement over even a 50 ms poll.
- Multi-host is native (no single-host assumption anywhere; property
tests ran all-through-network).
- The forwarder's two deadlock pitfalls are *learned territory* with
pinned tests, not unknowns.
**Negative**
- The listener forwarder is ours to maintain (~90 lines + the two
pinned failure modes) — the cost of not adopting `postgres-notify`.
- Per-process connection budget grows by one listener connection per
process that listens (a deployment-matrix row; consumers sizing
pools must account for it).
- The pg queue schema is re-derived work; the pg-boss family is
reference, not free.
## References
- `docs/research/poc-pg-posture-findings.md` — measurements and the
pinned property suite.
- OQ-ST-03, OQ-ST-05's posture half (`docs/research/phase-0.md`).
- [ADR-001](001-crate-split.md) — single-driver binaries.
- [ADR-005](005-dependency-ownership.md) — the ownership calculus this
decision applies per-subsystem.
- [ADR-006](006-wake-and-delivery-contract.md) — the wake contract the
forwarder implements on this engine.
- [engine-postgres.md](../engine-postgres.md) — the engine spec.
@@ -0,0 +1,70 @@
# ADR-005: Dependency ownership — published libraries by default, named fork triggers
## Status
Accepted (posture); fork triggers are live assessments tracked in OQ-06
## Context
The crate's reactive machinery sits on third-party work: honker
(SQLite side) and, conceptually, pgboss-rs (Postgres queue family).
Neither is adopted-by-adjacency — per guiding principle 3, ownership of
each subsystem is a deliberate per-question decision. The workspace
precedent is the targeted fork (alksocks' fast-socks5 extraction), but
a fork is expensive (especially when the forked code is sync and the
family standard is tokio — a fork of honker-rs is already a serious
port).
Two POCs (2026-10-04) measured the relevant postures per subsystem;
each carries its own record. This ADR fixes the *default calculus* and
the per-subsystem resolutions so Phase 1 design work starts from
stable ground.
## Decision
**Default posture: consume published libraries.** Fork or vendor only
when a named trigger fires. Per subsystem:
| Subsystem | Posture | Notes |
|---|---|---|
| honker-core (SQLite engine machinery) | published library, `honker-core = 0.5` | Fork trigger: the Phase 1 quality read of the watcher/transactional core finds a defect, OR a needed change upstream won't take. OQ-06 tracks the read. |
| rusqlite | published, riding honker-core's pin | Pin movement is honker-core's; we ride it ([ADR-003]). |
| tokio-postgres + deadpool-postgres | published library, as-is | Clean, zero conflicts, actively maintained (POC #2). |
| postgres-notify | **not adopted** (derive-not-adopt) | Lazy reconnect, connect_script skipped at initial connect, unquoted-identifier LISTENs, single maintainer. Hand-rolled forwarder instead ([ADR-004]). Fallback if upstream improves materially. |
| pg-boss queue machinery (pgboss-rs / node pg-boss) | **re-derived** on our driver; schema family as *design reference* only | pgboss-rs brings sqlx (second driver per binary) and has no LISTEN/NOTIFY (verified); the push half is ours either way, so queue-machinery reuse value is the honest comparison point and it loses on that arithmetic. Semantics depth: [queues.md], OQ-05. |
| honker-rs interface shape | **design reference**, not a dependency | Sync-only (no tokio); the surface is prior art for contract pinning ([ADR-006]), not code to ship. |
If any fork fires, forking is normal work we own (license/provenance
recorded per AGENTS.md §3 at adoption time) — it is not an exception
to this posture.
## Consequences
**Positive**
- Zero vendored code in the tree by default; upgrades are
`cargo update` work, not patch-management work.
- The fork triggers are concrete and named *before* the quality read,
so the read produces a decision, not a debate.
- Per-subsystem votes are recorded with evidence, so no future
discussion re-litigates from scratch.
**Negative**
- honker-core 0.5 remains alpha-quality software per its own README in
the link graph; the quality read (OQ-06) is the mitigation gate.
- The pg queue machinery is written by us — the pg-boss family's
battle-tested edge cases must be re-earned by design + tests
([queues.md]).
## References
- `docs/research/poc-sqlite-posture-findings.md` §"What feeds where"
(posture votes), `docs/research/poc-pg-posture-findings.md`
§"Dependency postures".
- OQ-ST-05, OQ-ST-06 (`docs/research/phase-0.md`) — the resolution
records.
- [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) —
the per-engine driver decisions applying this posture.
- OQ-06 (`docs/architecture/open-questions.md`) — the quality-read
tracker.
@@ -0,0 +1,128 @@
# ADR-006: Wake contract — opaque wake + re-read, and the notify-vs-streams delivery split
## Status
Accepted (the contract skeleton the evidence supports); remaining
surface pinning is OQ-04's contract work against this ADR
## Context
The two engines' wake mechanisms are structurally different — SQLite
wakes by a watcher polling `PRAGMA data_version` (deliver-on-commit,
no server push), Postgres by LISTEN/NOTIFY (server push,
connection-bound, no replay). A naive unification either collapses to
polling behavior on the strong side or promises semantics only one
engine has. Both POCs (2026-10-04) measured the pieces and — the
load-bearing finding — verified that **the two engines can share the
same wake contract unchanged**: wake is opaque, delivery is not
guaranteed to be precise, and consumers re-read state.
Honker's own processing-guarantees table is the cautionary prior art:
per-binding auto-checkpoint-vs-manual-save ambiguity produces
*different* guarantees under one function name. A single-crate version
must pick one answer per mechanism, not inherit the table.
## Decision
The contract fixes three things:
### 1. The unified wake contract: opaque wake + re-read state
`listen()` (and stream/queue subscription wake) delivers an **opaque
wake signal: "something changed; re-read the state you care about."**
The contract never carries semantic content, ordering promises, or
change descriptions. Consumers that need the actual data re-read it
(the hot-path pattern the ecosystem already uses: a subscriber holding
a cache invalidates on wake and re-reads, instead of re-polling).
- SQLite: `data_version` watcher fires on commit; wakes **coalesce**
under bursts (overtriggering on purpose — "one indexed SELECT is
cheap; a missed wake is a correctness bug").
- Postgres: LISTEN delivers per-notification (no coalescing), and the
forwarder's synthetic reconnect-wake covers connection gaps.
- Both: consumer code must be correct if wakes repeat, coalesce, or
arrive in any order, with at-least-once wake delivery. Exactly-once
*processing* semantics belong to queues/streams (below), never to
the wake layer.
Measured ground: wake latency p50 ≈ 1.1–2.2 ms on both engines at
default cadence; per-payload burst delivery verified (30/30 on pg,
coalesced-but-complete re-read on SQLite).
### 2. Delivery guarantees, pinned per mechanism (no per-binding table)
| Mechanism | Durability | Replay | Atomicity | Guarantee |
|---|---|---|---|---|
| `notify` / listen | none | never | commit-atomic (delivers at commit; rollback drops) | fire-and-forget, at-most-once per listener session |
| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset |
| queues | durable row | claim/ack model | enqueue is commit-atomic | at-least-once *work* with visibility timeouts |
- The trait **does not promise replay under `listen()`** — on either
engine. Durability+replay needs are what streams are for; this split
is the contract's load-bearing line.
- Listener sessions start "from now" (SQLite: the watcher's current
state; Postgres: `MAX(id)`-equivalent at attach). History belongs to
streams.
- Wake *delivery* failures surface, not silence: watcher death closes
subscriber channels (SQLite, honker-core's
`WatcherDeathGuard` behavior); Postgres connection gaps surface as
the synthetic reconnect-wake on a reserved channel
([ADR-004], [engine-postgres.md]).
### 3. Reserved names are a contract surface
The Postgres forwarder's synthetic reconnect-wake needs a reserved
channel namespace no consumer channel may collide with. The naming
convention for reserved/meta channels (and any internal queue/stream
names) is pinned in the core contract — exact strings are OQ-04
contract work; the *existence of a reserved namespace* is decided
here.
### The caching-subscriber pattern rides the same contract
A subscriber that also holds a cache gets invalidation from the
opaque wake + re-read: the contract deliberately does not carry
key-level payloads, so cache clients key their invalidation on their
own read-set, informed by channel names (the channel name is the one
piece of semantic content `listen()` carries). Key-payload design,
if any consumer needs richer invalidation, is a
consumer-inventory-row-gated extension — not assumed now.
## Consequences
**Positive**
- One wake contract, verified identically on both engines — the
"how does a change become visible" question alkstore exists to
answer, answered once.
- The delivery-guarantee table is pinned per mechanism, killing the
per-binding ambiguity class honker's docs exhibit.
- Consumer code branches on *mechanism choice* (notify vs streams vs
queues), never on engine type (guiding principle 4; the honest
single/multi-host line is [deployment.md]'s and OQ-08's, not this
contract's).
**Negative**
- The opaque wake is deliberately less informative than
key-payload notify systems; consumers needing fine-grained
invalidation pay a re-read or need streams.
- Wake overtriggering on SQLite means consumers must be idempotent on
wake (a documented consumer obligation, part of the contract text).
## References
- `docs/research/poc-sqlite-posture-findings.md` §Probe 2,
§"What feeds where"; `docs/research/poc-pg-posture-findings.md`
§Sub-module W.
- OQ-ST-04 (`docs/research/phase-0.md`) — de-risked by these
measurements; the contract remainder is OQ-04 +
`core-contract.md`.
- [ADR-002](002-feature-scope.md) — notify/listen and streams are
first-class; the guarantee split is why both exist.
- [ADR-004](004-postgres-driver.md) — the forwarder owning the
Postgres side of this contract.
- [ADR-007](007-transactional-seam.md) — commit-atomicity's mechanism
(the `*_tx` seam).
- [core-contract.md](../core-contract.md) — the spec carrying this
contract's full surface.
@@ -0,0 +1,102 @@
# ADR-007: Transactional seam — caller-held tx handle with `*_tx` methods
## Status
Accepted (the seam shape both POCs verified); handle-type ergonomics
(generic vs downcast) is OQ-04 contract work
## Context
The load-bearing property of the whole store (guiding principle 2) is
**transactional local-adjacency**: enqueue/publish/notify inside the
same transaction as the caller's business write; rollback drops both.
Serving it through an async trait over two transaction models —
rusqlite's sync, thread-bound connection vs tokio-postgres's
`Send + Sync` client — is the one genuinely driver-coupled design
point (OQ-ST-03's framing). Both POCs (2026-10-04) built the candidates
and measured them.
## Decision
The core contract's transactional seam is a **caller-held transaction
handle**:
```text
store.begin_tx() -> TxHandle
handle.enqueue_tx(name, opts, payload) -> job id
handle.publish_tx(name, payload) -> event id
handle.notify_tx(channel, payload)
handle.save_offset_tx(stream, consumer, offset)
handle.commit() / handle.rollback()
```
- The handle is owned by the caller across `await` points; every
`*_tx` operation lands in *the caller's transaction*; commit/rollback
are explicit and owned by the caller. Non-`_tx` operations are the
auto-commit convenience counterparts, each atomic alone.
- **Postgres** ([ADR-004]): the handle holds the pooled connection
object directly (tokio-postgres `Client` is `Send + Sync` — verified
by a compile-time probe). Straight `.await`s, no bridge, no hop
cost. Commit/rollback return the object to the pool.
- **SQLite** ([ADR-003]): the handle is a **writer-slot lease** —
`begin_tx` acquires the `Writer` slot and opens `BEGIN IMMEDIATE`;
every handle op round-trips `spawn_blocking` to that connection
(rusqlite is not `Send`-across-await / not `Sync`). Holding the slot
across `await` points *is* how the lease works — a long transaction
parks the single WAL writer, which is SQLite's own shape.
- **Closure-scoped transactions** (`with_tx`) are *available as a
wrapper over* the handle shape, not instead of it (the POC verified
the reverse composition doesn't work: a closure cannot outlive
itself, so caching-subscriber state can't escape it).
- Both engines deliver the property **natively** — no emulation:
in-tx notify/Notify delivers only at commit; rollback drops job rows,
business rows, and notifications together (the POC property tests on
both engines green: no ghosts).
The POC's trait sketch carried two frictions, both now Phase 1
contract-pinning input rather than open risk:
1. The `as_any_mut` downcast per engine for `*_tx` methods on a
`dyn TxHandle` — small, but a generic or enum-favored handle may be
cleaner. OQ-04 decides with the full contract.
2. Thread-affinity of rusqlite tx ops (SQLite handle ops must
round-trip the same blocking thread) — inherent to [ADR-003]'s
bridge, documented as a contract note ("SQLite tx ops are
serialized by the writer slot"), not a defect.
## Consequences
**Positive**
- The load-bearing property is one seam, both engines, POC-verified
end-to-end — the dual-write problem is structurally prevented, not
consumer-disciplined.
- Long transactions compose naturally (checkout-then-work shape);
nothing about the seam forbids multi-statement business
transactions with interleaved reads.
- Per-engine bridging differences are invisible to consumer code: the
same `*_tx` calls, the same commit/rollback ownership.
**Negative**
- The SQLite handle parks the writer for its duration — a slow
consumer transaction throttles all writers on the file. This is
honest (WAL single-writer), but contract docs must say it so
consumers budget transactions accordingly.
- Per-op `spawn_blocking` hop on SQLite tx ops (measured fine at
~0.35 ms p50; the dedicated-thread bridge is the recorded
optimization).
- `with_tx` wrapping is a convenience surface we must ship and test
so it doesn't accrete ad-hoc in consumer code.
## References
- `docs/research/poc-sqlite-posture-findings.md` §Arm A, §Probe 1/3;
`docs/research/poc-pg-posture-findings.md` §Sub-module T.
- OQ-ST-04 (`docs/research/phase-0.md`), the `*_tx` seam bullet.
- [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) —
the per-engine handle mechanics.
- [ADR-006](006-wake-and-delivery-contract.md) — commit-atomicity is
the `*_tx` property this seam carries into the notify contract.
- [core-contract.md](../core-contract.md) — the spec whose transaction
section this ADR defines.
+118
View File
@@ -0,0 +1,118 @@
---
status: draft
last_updated: 2026-10-04
---
# Deployment
What a deployer must know to size, run, and reason about alkstore
engines: host semantics, connection budgets, durability knobs, and
where engine differences may honestly surface in the contract. The
capability-surface *decision* (how much of this the trait exposes) is
OQ-08's; this document holds the facts and the decision's frame.
## Host semantics
| Engine | Host posture | Notes |
|---|---|---|
| SQLite | **single-machine**, file-backed | NFS two-writers unsupported (honker's honesty posture, inherited, [ADR-003](decisions/003-sqlite-driver.md)). Cross-process *on one host* is verified POC ground (`data_version` is cross-process by nature). |
| Postgres | **multi-host native** | Nothing assumes a shared host; POC #2 ran all-through-network (docker bridge) with the same properties ([ADR-004](decisions/004-postgres-driver.md)). |
The unified trait must not pretend SQLite is multi-host — but whether
that honesty lives as runtime capability flags, compile-time engine
knowledge, or a documented matrix only is OQ-08
([ADR-006](decisions/006-wake-and-delivery-contract.md) note: the
trait's shape constrains where capability differences can surface).
Options for OQ-08, with their shape:
1. **Compile-time only** — a consumer chooses an engine crate at
dependency time; the engine's docs carry its deployment facts.
Smallest contract; nothing runtime to match on.
2. **`Store::capabilities()`** — a runtime description
(payload limits, wake cadence knobs, host semantics). Lets a
consumer adapt (e.g., chunk large notify payloads) but adds a
contract surface all engines must keep honest.
3. **Deployment matrix only** (this document) — no API surface. The
honest-middle choice; matches the ecosystem's doc-first posture
but provides no programmatic guard.
## Connection budgets
### SQLite engine
- Connections are in-process (writer + reader pool + watcher thread
owning a connection). No external budget lines; the file lock is
the OS-level resource.
### Postgres engine
| Connection class | Count | Notes |
|---|---|---|
| Pool | `max_size` per process | claims/queries via deadpool |
| Listener | **+1** per LISTEN-ing process | non-pooled, dedicated; pooled connections cannot carry LISTEN (deadpool#360, test-pinned — [ADR-004](decisions/004-postgres-driver.md)) |
| — | — | Sizing rule: `max_size + 1` per process; verify end-to-end accounting (POC #2's pgdiag-st-4: pool 6 + 1 listener + probe conns = exact server-side count) |
Shared-server co-tenancy (the alkblobs ADR-008 precedent —
`/workspace/@alkdev/alkblobs/docs/architecture/decisions/` — consumer
tables co-tenant the pg instance) is supported and expected — the
[naming / reserved namespace contract](core-contract.md#naming--reserved-namespace)
protects reserved names; queue/stream/lock tables are schema-named to
avoid collisions (layout decision in [queues.md](queues.md), OQ-05's
namespace bullet).
## Durability knobs
| Engine | Knob | Shape |
|---|---|---|
| SQLite | `synchronous` | WAL + `NORMAL` shipped ([ADR-003](decisions/003-sqlite-driver.md)); FULL is available consumer-side for stricter durability; commit fsyncs land at WAL checkpoints (the ~1000-commit spike cadence, POC #1) |
| Postgres | `synchronous_commit` | per-session knob; `on` is ship config (p50 2.40 ms seam); `off` trades max-tail (40.9 ms) for slightly better p50 — measured, honest trade ([ADR-004](decisions/004-postgres-driver.md)); session-level SET mechanics POC-verified |
These are engine-configuration concerns, *not* trait surface (a
consumer may set them via their own engine config — what part of
engine config is contract-level `Store::open` shape vs engine-crate
docs is OQ-04's config-shape item).
## Toolchain / platform notes
| Note | Engine | Affects |
|---|---|---|
| rusqlite 0.40.x needs rustc ≥ 1.99 | SQLite | any binary linking `alkstore-sqlite` ([ADR-003](decisions/003-sqlite-driver.md)) |
| `bundled-sqlite` adds a C build (~10 s dev, cacheable) | SQLite | build/CI time |
| `libsqlite3-sys` collision with sqlx today | any mixed-driver binary | structurally avoided ([ADR-001](decisions/001-crate-split.md) single-driver rule) |
| Listener `application_name` set for diagnosability (kill-targetable) | Postgres | ops runbooks ([ADR-004](decisions/004-postgres-driver.md)) |
## Consumer-facing latency profile (indicative, POC-measured)
From both POCs (single-box, relative shapes are the deliverable —
[ADR-003](decisions/003-sqlite-driver.md) and
[ADR-004](decisions/004-postgres-driver.md) carry the full tables):
- Tx seam: SQLite ~0.35 ms p50; Postgres ~2.4 ms p50 (ship config).
- Wake: ~1.1–2.2 ms p50 both engines at default cadence.
- Queue claim: LISTEN-driven 3–6 ms p50 (pg); poll-only interval-bound
(32–50 ms at a 50 ms poll).
- Absolute numbers will differ per hardware/network; they set
*expectations of order*, not SLAs — contract docs must not bake
them in ([ADR-007](decisions/007-transactional-seam.md)'s pg-POC
note).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [001](decisions/001-crate-split.md) | Crate split | single-driver binaries shape the matrix |
| [003](decisions/003-sqlite-driver.md) | SQLite driver | bundling, toolchain floor |
| [004](decisions/004-postgres-driver.md) | Postgres driver | listener budget line, forwarder posture |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | where capability differences may surface |
## Open Questions
Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-08**: capability-surface shape — compile-time vs runtime flags
vs matrix-only (open)
- **OQ-04**: engine-config surface in the contract's `Store::open`
shape (shared) (open)
+113
View File
@@ -0,0 +1,113 @@
---
status: draft
last_updated: 2026-10-04
---
# Postgres engine
The `alkstore-postgres` engine implements
[core-contract.md](core-contract.md) on tokio-postgres +
deadpool-postgres. This spec records WHAT the engine is internally —
connection architecture, the LISTEN forwarder, queue machinery
re-derivation — not code-level HOW. Decisions live in ADRs; contract
obligations live in the core spec.
## Identity and posture
- Single driver, natively async: tokio-postgres 0.7.x +
deadpool-postgres 0.14.x ([ADR-004](decisions/004-postgres-driver.md)).
No bridge, no `spawn_blocking`; the tx handle holds the pooled
object directly (client is `Send + Sync` — POC #2 compile-probe
verified).
- Multi-host by nature: connections are per-process state; nothing
assumes a shared host ([ADR-004](decisions/004-postgres-driver.md),
verified all-through-network in POC #2). See
[deployment.md](deployment.md).
- Queue machinery re-derived on this driver with the pg-boss schema
family as design reference
([ADR-005](decisions/005-dependency-ownership.md)); semantics depth
is [queues.md](queues.md)'s work (OQ-05).
## Connection architecture
- **Pool** (deadpool) — queries, claims, and all non-transactional
work. Per-connection statement cache, `RecyclingMethod::Fast`
(no `DISCARD ALL` recycling; claim SQL re-prepared implicitly with
zero errors at POC scale).
- **Listener connection** — one dedicated, **non-pooled** connection
per process that listens. Pooled connections cannot carry LISTEN
(deadpool#360 — registration succeeds, delivery is impossible; the
client-wrapper exposes no notification surface, source-verified and
test-pinned as `pooled_listen_registers_but_cannot_deliver`). The
listener is therefore a per-process budget line *outside* the pool:
`max_size + 1` per LISTEN-ing process
([deployment.md](deployment.md)).
- **Forwarder** — the listener's loop: `poll_message`
fanning out into a bounded broadcast channel (lag surfaced, not
silent), re-LISTEN from the channel list after every reconnect
(exponential backoff 50 ms → 2 s cap), and the synthetic
reconnect-wake on the reserved channel
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
- One listener serves N channels and N subscribers; re-attach is a
broadcast re-subscribe (no server round-trips); per-channel
connections are never warranted at this scale (POC-verified).
## Mapping the contract
| Contract piece | Engine realization |
|---|---|
| notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers |
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) |
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback) |
| named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; depth in OQ-05's design work) |
| scheduler / outbox | pg-boss-family design reference, per [queues.md](queues.md) |
| begin_tx | pool checkout + `BEGIN`, returning the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
| handle ops | straight `.await`s through the held object; commit/rollback returns the object to the pool |
## Owned failure modes (all test-pinned in POC #2)
Hand-rolling the forwarder means owning its pitfalls — they are
*learned territory*, pinned as passing tests:
1. **Query-vs-poll starvation deadlock** — the poll loop must be
running before the first client query on the listener connection.
2. **Client-drop closes the server session** — a long-lived listener
keeps its `Client` alive for the listener's lifetime.
3. **The no-replay hole** — commit during a connection gap is never
re-delivered; recovery = reconnect + synthetic wake + consumer
re-read ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
The `!saw_replay` test pins the honesty.
4. **Payload boundary** — `pg_notify` ≤ 8000 bytes; client-side
checked, typed error before the round-trip. Large payloads ride a
table row with the id in the notification (outbox shape).
5. **Read-your-writes** — `read committed` default verified in both
directions (in-tx and post-commit).
Constraint also carried: mixed rusqlite+sqlx binaries would need a
vendored patch today — excluded by construction in this engine
(single driver, [ADR-001](decisions/001-crate-split.md)), noted for
the record in [ADR-003](decisions/003-sqlite-driver.md).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
| [004](decisions/004-postgres-driver.md) | Driver | tokio-postgres + deadpool; hand-rolled forwarder; re-derived queues |
| [005](decisions/005-dependency-ownership.md) | Ownership | published libs as-is; `postgres-notify` derive-not-adopt |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | LISTEN push, no replay, synthetic reconnect-wake |
| [007](decisions/007-transactional-seam.md) | Tx seam | direct pooled-object handle, no bridging |
## Open Questions
Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-09**: scheduler collapse into queues (shared with
[queues.md](queues.md)) (open)
- **OQ-05**: queue semantics depth — the pg-boss-family design-input
work (open)
- **OQ-08**: capability surface (shared with
[deployment.md](deployment.md)) (open)
+114
View File
@@ -0,0 +1,114 @@
---
status: draft
last_updated: 2026-10-04
---
# SQLite engine
The `alkstore-sqlite` engine implements
[core-contract.md](core-contract.md) on rusqlite + published
honker-core. This spec records WHAT the engine is internally (its
connection architecture, seam, and wake plumbing) — not code-level
HOW. Decisions live in ADRs; per-contract obligations live in the core
spec and are not restated here.
## Identity and posture
- Single driver: rusqlite (riding
[honker-core's pin](decisions/003-sqlite-driver.md)),
`bundled-sqlite` for hermetic builds. No `.so` runtime artifacts, no
vendored patches
([ADR-003](decisions/003-sqlite-driver.md),
[ADR-001](decisions/001-crate-split.md)).
- Sync machinery, async-facing trait: honker-core is sync (std
threads, blocking iterators); the engine bridges at the trait seam
per the family-standard posture (alktty REQ-TTY-01 precedent).
Every engine call runs in `spawn_blocking`.
- Single-host by nature: file-backed, one machine, NFS-two-writers
unsupported (honker's honesty posture, inherited). See
[deployment.md](deployment.md).
## Connection architecture
- **Writer** — one dedicated connection; the only one permitted to
write. Serializes all mutations (WAL single-writer, modeled
honestly, not fought).
- **Readers** — a small pool of read connections for lookups and
claim/ack work.
- **Watcher** — honker-core's `SharedUpdateWatcher`: a dedicated
thread polling `PRAGMA data_version` at the default 1 ms cadence,
fanning out to listeners, overtriggering on purpose (waking all
subscribers per poll tick, even when several commits coalesced
inside one tick — wake is a hint; consumers re-read indexed state,
[ADR-006](decisions/006-wake-and-delivery-contract.md)).
- Each connection runs honker's bootstrap at open: pragmas
(WAL, `synchronous=NORMAL`, busy timeout), `attach_notify`,
`attach_honker_functions`, `bootstrap_honker_schema` (the
alknet-filesystem POC's wiring shape).
## Mapping the contract
| Contract piece | Engine realization |
|---|---|
| notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) |
| streams | honker's stream machinery; explicit offset saves through the tx seam |
| queues | honker's queue functions ([ADR-002](decisions/002-feature-scope.md)); semantics depth design in [queues.md](queues.md) |
| named locks | honker's lock machinery |
| scheduler / outbox | honker's counterparts, per [queues.md](queues.md) |
| begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
| handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) |
| commit/rollback | releases the writer slot |
## Obligations and constraints
- **A long transaction parks the writer** (the slot lease is the
honest model). Contract docs must surface this so consumers budget
transactions ([ADR-007](decisions/007-transactional-seam.md) negative
consequence).
- Watcher failure handling is inherited: on watcher death, every
subscriber's receiver closes (`WatcherDeathGuard` behavior) —
consumers see the close, never a silent hang
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
- Wake coalescing: bursts inside one poll tick produce one wake;
correctness is preserved by the re-read contract
(POC-pinned: missed-wake stress with correct post-burst re-reads).
- **Deployment note**: honker-core 0.5 pins rusqlite ^0.40.1, whose
rustc floor is ≥ 1.99; binaries linking this engine carry that
requirement ([deployment.md](deployment.md) matrix).
- The optimization path, if seam throughput ever demands it: a
dedicated std-thread bridge (one thread owning the writer conn, ops
over mpsc — measured ~2× the spawn_blocking shape at p50 in POC #1),
or a raised watcher cadence for idle CPU. Neither is the default.
## What rides on the fork question
Nearly all of honker-core's surface is consumed (Writer/Readers/
SharedUpdateWatcher/attach_*). The [quality read
(OQ-06)](decisions/005-dependency-ownership.md) is this engine's only
open dependency gate: if it names a defect or an upstream-unwon't
change, the fork posture
([ADR-005](decisions/005-dependency-ownership.md)) fires and this
engine's substrate becomes owned code. Until then, published-library
consumption stands.
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + honker-core, bridged seam, inherited watcher |
| [005](decisions/005-dependency-ownership.md) | Ownership | published honker-core; named fork triggers |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | data_version watcher, coalescing, death-closes-receivers |
| [007](decisions/007-transactional-seam.md) | Tx seam | writer-slot lease, `BEGIN IMMEDIATE`, `spawn_blocking` round-trips |
## Open Questions
Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-06**: honker-core quality read — fork-trigger assessment (open)
- **OQ-09**: scheduler collapse into queues (shared with
[queues.md](queues.md)) (open)
- **OQ-05**: queue semantics depth on honker's machinery (open)
+217
View File
@@ -0,0 +1,217 @@
---
status: draft
last_updated: 2026-10-04
---
# alkstore — Open Questions
Centralized tracker. IDs `OQ-NN` are stable — never renumber; append.
The Phase 0 register's questions (`OQ-ST-01..08` in
`docs/research/phase-0.md`) are promoted here faithfully: **OQ-01..08
mirror OQ-ST-01..08 one-to-one**, keeping their Phase 0 statuses
(a resolved register question stays listed here as resolved, with the
ADR that carries its decision). New Phase 1 questions append from
OQ-09. Suggested resolution order: **OQ-04 first** (the contract
surface everything else hangs off), then OQ-09/OQ-10 (scoped to
OQ-04's outcome), then OQ-05/OQ-08 (OQ-05 has an independent design
track; OQ-08 depends only on the trait's shape, OQ-04).
Resolved questions stay listed with their resolution; they are not
deleted.
## Theme: Scope
### OQ-01: Scope boundary — which honker features are in-scope? *(== OQ-ST-01)*
- **Origin**: [consumer-inventory.md](../research/consumer-inventory.md),
[ADR-002](decisions/002-feature-scope.md)
- **Status**: resolved (2026-10-04, Phase 0)
- **Priority**: medium
- **Resolution**: Answered by the consumer inventory — scope votes
shrink to named rows, not the whole honker feature list. First-class:
notify/listen (pinned), streams (operator-authority record). In
scope: named locks, queues + outbox (documented), scheduler
(documented-thin). Cut-flags: rate limits, result storage. Out:
honker's exclusion lines. Decision recorded in
[ADR-002](decisions/002-feature-scope.md).
- **Cross-references**: OQ-07, OQ-04.
### OQ-02: Crate scope — one store crate, or reactive-core + engines? *(== OQ-ST-02)*
- **Origin**: [overview.md](overview.md)
- **Status**: resolved (2026-10-04, operator decision; Phase 0)
- **Priority**: medium
- **Resolution**: Reactive-core crate + per-engine engine crates; the
split isolates the engines' asymmetry of work and keeps engine
binaries single-driver (which the libsqlite3-sys collision
effectively requires). Decision recorded in
[ADR-001](decisions/001-crate-split.md) — including the reasoning
that superseded the inventory's single-crate lean.
- **Cross-references**: OQ-03, OQ-10.
### OQ-03: Driver story — sqlx, tokio-postgres, or per-engine drivers? *(== OQ-ST-03)*
- **Origin**: [overview.md](overview.md)
- **Status**: resolved (2026-10-04, POC-backed both engines; Phase 0)
- **Priority**: medium
- **Resolution**: Per-engine drivers — rusqlite + honker-core 0.5.0
(SQLite; bridged seam), tokio-postgres 0.7.18 + deadpool-postgres
0.14.2 (Postgres), under OQ-02's crate split. Decisions recorded in
[ADR-003](decisions/003-sqlite-driver.md) and
[ADR-004](decisions/004-postgres-driver.md); measurements in
`docs/research/poc-sqlite-posture-findings.md` and
`docs/research/poc-pg-posture-findings.md`.
- **Cross-references**: OQ-02, OQ-05, OQ-06.
### OQ-07: SQLite-side scope — loadable extension? *(== OQ-ST-07)*
- **Origin**: [ADR-002](decisions/002-feature-scope.md)
- **Status**: resolved (cut-only, 2026-10-04; Phase 0)
- **Priority**: low
- **Resolution**: The loadable-extension surface is out: every
identified consumer is in-process Rust attaching to its own
connection (the honker-core `attach_honker_functions` shape). Folded
into OQ-01's scope resolution and [ADR-002](decisions/002-feature-scope.md)
(explicitly, so the slot's disposition is visible — the Phase 0
register carried it as its own entry).
- **Cross-references**: OQ-01.
## Theme: Core contract
### OQ-04: Contract pinning — the exact trait surface and its semantics *(== OQ-ST-04)*
*(Retitled in promotion: OQ-ST-04's framing — "the reactive
abstraction — what does the unified notify surface look like?" —
narrowed to the pinning work its own record already scoped.)*
- **Origin**: [core-contract.md](core-contract.md),
[ADR-006](decisions/006-wake-and-delivery-contract.md),
[ADR-007](decisions/007-transactional-seam.md)
- **Status**: open (de-risked; both engine sides POC-verified — this is
paper work over a complete evidence base)
- **Priority**: high
- **Resolution**: open. The decisions that bound it are made
([ADR-006](decisions/006-wake-and-delivery-contract.md): opaque-wake + re-read, the delivery-guarantee table, the
reserved namespace's existence; [ADR-007](decisions/007-transactional-seam.md): caller-held `TxHandle`
with `*_tx` methods). What remains to pin, per the honker-rs surface
as starting artifact against the inventory rows:
- Which surface parts are contract v1 and which are engine-extension
(queue depth, scheduler surface, lock semantics details).
- `TxHandle` representation: `dyn` + per-engine downcast (POC
sketch, works, two recorded frictions) vs generic/enum handle.
- The reserved/meta namespace: exact strings (Postgres
reconnect-wake channel like `__listener_reconnected__`), and
whether SQLite-side internal names carry an equivalent prefix.
- Error taxonomy: what callers can match on per mechanism (e.g.
`PayloadTooLarge` is POC-verified pg-side; SQLite's notify has no
8000-byte limit — is the error universal?).
- Naming/grouping of the honker-rs surface where renames clarify
(e.g. what `listen()` returns — a wake receiver? a subscription
handle?).
- The `Store::open` config shape — what engine-configuration
(durability knobs, pool sizing, listener posture) is consumer
surface vs engine crate docs.
- Verification backlog for properties the POCs pinned on one engine
only — e.g. lock TTL/expiry re-acquisition is POC-pinned on
Postgres but rests on honker's machinery on SQLite; the contract
test suite must pin the SQLite side too.
- Delivery-guarantee rows for locks and scheduler (the guarantee
table in [ADR-006](decisions/006-wake-and-delivery-contract.md) covers notify/streams/queues; lock-vs-TTL-race
and scheduler-tick guarantees need either rows there or an
explicit "scheduler is queues" absorption via OQ-09).
- **Cross-references**: OQ-01, OQ-10, OQ-09, OQ-08.
## Theme: Queues and scheduling
### OQ-05: Queue semantics depth — retry/backoff/dead-letter/sweep design *(== OQ-ST-05)*
- **Origin**: [queues.md](queues.md)
- **Status**: open (posture resolved: re-derived on both engines with
the pg-boss schema family as design reference and honker's queue
design as the SQLite-side one — ADR-004/ADR-005)
- **Priority**: high
- **Resolution**: open. The transactional and claim properties (the
hard driver-coupled part) are POC-pinned on both engines. Remaining:
the semantics-depth design — retry policy shape (attempts, backoff
curve), visibility-timeout/renewal mechanics, dead-letter
move-vs-flag and retention, sweep/maintenance design
(`sweep_expired` cadence and owner), and whether result-storage's
cut-flag gets reconsidered as part of this surface (it is
queue-adjacent tooling).
- **Cross-references**: OQ-09, OQ-06.
### OQ-09: Is the scheduler a first-class mechanism, or queues + `schedule()`?
- **Origin**: [queues.md](queues.md) (spin-out of OQ-05's Phase 0
framing, where the scheduler's boundary question lived)
- **Status**: open
- **Priority**: medium
- **Resolution**: open. The inventory found the scheduler need real but
thin (the family-wide "who sweeps/renews/reaps" problem), and
mechanically scheduler = cron/`@every` enqueueing into named queues
+ leader election + a tick. If design confirms the collapse, the
contract surface is queues + `store.schedule(cron, queue)` with the
scheduler mechanism absorbed; if consumers need
inspectable/pausable schedule objects (`add/pause/resume/update/
list/remove`), it stays a first-class mechanism (the full honker
shape). Decided against the inventory rows, not the whole honker
menu.
- **Cross-references**: OQ-05, OQ-04.
## Theme: Engines and dependencies
### OQ-06: honker-core quality read — does the default posture hold? *(== OQ-ST-06)*
- **Origin**: [ADR-005](decisions/005-dependency-ownership.md),
[engine-sqlite.md](engine-sqlite.md)
- **Status**: open
- **Priority**: high (fork trigger is a gate on the SQLite engine's
dependency posture)
- **Resolution**: open. ADR-005 fixed the calculus and the trigger: the
Phase 1 quality read of honker-core 0.5's watcher/transactional core
(Writer/Readers/SharedUpdateWatcher/attach_*) — looking for defects
the POCs wouldn't surface, unsafe assumptions in the watcher
failure-handling, and schema-migration brittleness. Outcomes: posture
holds (no ADR change), or a fork/patch need is named (fork is normal
work per ADR-005).
- **Cross-references**: ADR-003, ADR-005, OQ-05 (if the read forces a
fork, queue-on-honker-machinery work changes shape).
## Theme: Deployment and capabilities
### OQ-08: Where does the honest single-host/multi-host boundary live in the trait surface? *(== OQ-ST-08)*
- **Origin**: [deployment.md](deployment.md)
- **Status**: open
- **Priority**: medium
- **Resolution**: open. The pg engine is natively multi-host (POC #2
verified — no single-host assumption to remove); SQLite is
single-machine by nature (file-backed, NFS-two-writers unsupported —
honker's honesty posture, inherited). The unified surface must not
pretend SQLite is multi-host. Options: per-engine capability flags
(`Store::capabilities()`), a documented deployment matrix only
([deployment.md] carries the facts), or compile-time knowledge only
(a consumer choosing the SQLite engine knows). Rides OQ-04: the
trait's shape constrains where capability differences can surface.
- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md).
### OQ-10: How do engine crates track core-contract version changes?
- **Origin**: [overview.md](overview.md),
[ADR-001](decisions/001-crate-split.md)
- **Status**: open
- **Priority**: medium
- **Resolution**: open. The core crate's trait surface is a contract
the engine crates must track ([ADR-001](decisions/001-crate-split.md) negative consequence). What
is the versioning/sync discipline — semver-bump-only-when-
contract-changes, engines pin core ranges, a contract-compatibility
test suite the engines run against the core's trait definitions?
What happens to a released engine crate when core makes a contract
breaking change?
- **Cross-references**: OQ-04 (the contract being versioned), OQ-02.
## Deferred / Blocked
None currently. Every open OQ above is actionable Phase 1 architecture
work (contract pinning, design, quality read) with its evidence base
complete — no external arrivals are being waited on.
+94
View File
@@ -0,0 +1,94 @@
---
status: draft
last_updated: 2026-10-04
---
# alkstore — Overview
One reactive store interface over SQLite and Postgres: durable
notify/subscribe signals, streams with per-consumer offsets, durable
queues with the transactional enqueue property, named locks, a
scheduler, and an outbox helper — with each engine using its own
native machinery underneath (SQLite: honker's watcher design;
Postgres: `pg_notify`/LISTEN and the pg-boss schema family).
## Why this crate exists
The alk* ecosystem's current shape is a repository pattern with a
default in-memory adapter, cache-invalidation wiring in hot paths, and
per-project approximations where a durable reactive substrate was
needed. alkblobs paused partly on that fuzziness. alkstore is the
substrate, made once: the answer to "how does a change in the database
become visible to other processes/connections?" that downstream stores
don't re-derive.
## Crate family
Per [ADR-001](decisions/001-crate-split.md):
| Crate | Contents | Driver dependencies |
|---|---|---|
| `alkstore` (core) | trait surface, types, error model, capability flags | none |
| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, honker-core |
| `alkstore-postgres` | Postgres engine ([ADR-004](decisions/004-postgres-driver.md)) | tokio-postgres, deadpool-postgres |
| (mem engine, optional) | test convenience, decided at implementation ([ADR-001](decisions/001-crate-split.md)) | none |
Downstream consumers depend on core + exactly one engine. Family
standards apply throughout: tokio async runtime, `thiserror` errors, no
panics in library code, no `unwrap()`/`expect()` outside tests,
lean base crate.
## Feature surface
Per [ADR-002](decisions/002-feature-scope.md):
- **First-class**: notify/listen; streams.
- **In scope**: named locks; queues + outbox helper; scheduler.
- **Cut-flag**: rate limits; result storage.
- **Out**: loadable-extension surface; DAGs/task
chains/chords/multi-writer replication/cross-machine locking.
## Document map
| Doc | Purpose |
|---|---|
| [core-contract.md](core-contract.md) | The unified trait surface: mechanisms, delivery guarantees, tx seam, naming |
| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto honker-core/rusqlite |
| [engine-postgres.md](engine-postgres.md) | Postgres engine: mapping the contract onto tokio-postgres/LISTEN |
| [queues.md](queues.md) | Queue/scheduler/outbox semantics depth (OQ-05/OQ-09) |
| [deployment.md](deployment.md) | Host capabilities, connection budgets, deployment matrix (OQ-08) |
| [open-questions.md](open-questions.md) | OQ-01..NN tracker |
| [decisions/](decisions/) | ADRs |
## What is decided (ADR index)
| ADR | Decision | Status |
|---|---|---|
| [001](decisions/001-crate-split.md) | Reactive-core + per-engine crates | Accepted |
| [002](decisions/002-feature-scope.md) | Feature scope (inventory-confirmed) | Accepted |
| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core, bridged seam | Accepted |
| [004](decisions/004-postgres-driver.md) | Postgres: tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
| [006](decisions/006-wake-and-delivery-contract.md) | Opaque wake + re-read; notify-vs-streams guarantee split | Accepted |
| [007](decisions/007-transactional-seam.md) | Caller-held `TxHandle` with `*_tx` methods | Accepted |
## Non-goals
- Not a database abstraction/ORM: the store covers the reactive
coordination surface, not general row storage (consumers keep their
own engines/schema for business data, sharing the connection when
appropriate — the outbox pattern).
- Not a network service: no transport; any networked ops surface rides
alkcall and is a separate future decision (store layer stays
substrate-free, the alkblobs store-layer precedent).
- Not multi-machine on SQLite: single-host honesty is inherited
([deployment.md](deployment.md)).
## Evidence base
Phase 0 (`docs/research/phase-0.md`) is complete: two POCs
(`poc-sqlite-posture-findings.md`, `poc-pg-posture-findings.md`) passed
all gate conditions; the consumer inventory
(`consumer-inventory.md`) graded every feature row. The Phase 1
architecture work runs over that complete evidence base; open Phase 1
work is tracked in [open-questions.md](open-questions.md).
+138
View File
@@ -0,0 +1,138 @@
---
status: draft
last_updated: 2026-10-04
---
# Queues, scheduler, outbox — semantics depth
The queue family's *posture* is decided (re-derived on both engines;
[ADR-004](decisions/004-postgres-driver.md) and
[ADR-005](decisions/005-dependency-ownership.md)) and the hard
driver-coupled properties are POC-pinned (transactional enqueue,
exactly-once claim under concurrency). What is *not* yet designed is
the semantics depth — this document's subject, and [OQ-05]'s home. It
states the design space, the decided constraints any answer must fit,
and the reference material; actual semantics decisions become ADRs
when made.
## Decided constraints (inherited, not re-opened)
- **Mechanisms**: durable at-least-once queues; the outbox is a helper
over queues; scheduler is cron/`@every` enqueueing into named
queues ([ADR-002](decisions/002-feature-scope.md)) — pending
OQ-09's collapse decision.
- **Transactional enqueue**: commit-atomic with the caller's business
write on both engines ([ADR-007](decisions/007-transactional-seam.md));
rollback drops the job row with no ghosts.
- **Exactly-once claim**: `FOR UPDATE SKIP LOCKED` (pg, POC-pinned
under 4×4 concurrency) / honker's machinery (SQLite). At-least-once
*work*; exactly-once *processing* needs the ack + visibility model.
- **Wake-driven consumption**: the pg engine's default is LISTEN-driven
claim with a re-poll safety net; SQLite's is watcher wake with
per-subscription fanout ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
- **Job options** (the honker-rs surface, the contract starting
artifact): `delay, priority, max_attempts, expires` at enqueue;
`ack / retry / fail / heartbeat` on the job handle.
- **Cut-flag context**: result storage is
[ADR-002](decisions/002-feature-scope.md)'s cut-flag row; if
OQ-05's design shows queue consumers provably need
result-query-by-id, that is the channel to revisit it — with a
consumer-inventory row, not silent inclusion.
## Design space (what OQ-05 must pin)
### Retry / backoff
- Trigger model: attempts counted from claim-completion; a job
re-appears after visibility timeout OR explicit `retry`. Does
`max_attempts` exhaustion move the job to dead-letter, flag it, or
drop it?
- Backoff curve between attempts: honker and the pg-boss family have
their own shapes (fixed, exponential); which does the contract
pin, what is configurable, what's the default?
- `heartbeat` semantics: renewal vs progress signal (or both) — and
what a missed heartbeat means (visibility re-expiry? nothing?).
### Visibility timeouts
- Where the renewal lives: explicit (`heartbeat`) only, or implicit
renewal per op on the job handle? Honker's design is the SQLite-side
answer; pg-boss's is the reference. One answer, contract-pinned.
- Interaction with long business transactions (claims inside
caller txs, [ADR-007](decisions/007-transactional-seam.md)).
### Dead-letter
- Move-to-table (honker's `_honker_dead`, pg-boss's dead-letter
queue) vs flag-in-place. Inspection/requeue surface shape.
- Retention: does a dead-lettered job expire (an `expires`-driven
sweep — an OQ-05 sub-question) or live until explicitly cleared?
### Sweep / maintenance
- `sweep_expired` is on the queue's surface; cadence ownership is
the open question. Honker's scheduler machinery (leader-elected,
missed-boundary catch-up) can run it; a consumer's own scheduler row
can too; the engine could ship a default. Decision shaped by OQ-09's
scheduler collapse — same machinery either way.
- Multi-process sweep safety on SQLite vs Postgres: named-lock
coordination (the alkblobs fleet-sweeper pattern,
[ADR-002](decisions/002-feature-scope.md)) vs native advisory locks.
### Scheduler surface
- OQ-09's question, stated crisply: is scheduler a first-class
mechanism (own handle, methods, inspection) or queues + a
`schedule(cron_expression, queue_name, payload)` call? The
inventory's evidence is thin (documented-thin); a v1 that is queues
+ a `schedule` call with leader-election behavior documented is the
smaller contract; if a consumer needs inspectable/pausable schedule
*objects* (`add/pause/resume/update/list/remove`), that's the
full honker shape. Decide against the inventory rows, not against
the whole honker menu.
### Namespaces / schema layout (pg side)
- Schema-scoped DDL (the pg-boss design) vs shared-schema table
naming. Rides the reserved-namespace contract
([ADR-006](decisions/006-wake-and-delivery-contract.md), OQ-04):
queue/stream/lock tables and the reserved channel prefix
must be collision-proof against consumer tables in the same
database (the alkblobs co-tenant-tables precedent — ADR-008 in
`/workspace/@alkdev/alkblobs/docs/architecture/decisions/`).
## Reference material
- **honker's queue design** — the SQLite-side incumbent
(`/workspace/honker`, its honker-core machinery; the engine rides it
directly, so the SQLite side's depth is largely "inherit + pin").
- **pg-boss family** — `/workspace/pgboss-rs` @ 98f7d9e (queue states,
maintenance/dead-letter behavior) and the node original (pg-boss, the
upstream of record — compare semantics the port may have dropped).
- **POC ground** — `poc-pg-posture-findings.md` (the minimal queue
table + SKIP LOCKED claim measured; the property suite green);
`poc-sqlite-posture-findings.md` (the SQLite twin).
- **Consumers** — alkfs OQ-FS-14 (sync outbox), alkblobs
ops-surface/gc docs (maintenance cadences), the family-wide
"who sweeps" punt (inventory §scheduler).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [002](decisions/002-feature-scope.md) | Feature scope | queues + outbox in; result storage cut-flag |
| [004](decisions/004-postgres-driver.md) | pg driver | queue machinery re-derived, pg-boss as reference |
| [005](decisions/005-dependency-ownership.md) | Ownership | design-reference postures for the queue family |
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | wake-driven consumption, guarantee table |
| [007](decisions/007-transactional-seam.md) | Tx seam | enqueue_tx commit-atomicity |
## Open Questions
Open questions are tracked in
[open-questions.md](open-questions.md). Key
questions affecting this document:
- **OQ-05**: the semantics-depth pinning this document frames (open,
high priority)
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
(open)