docs: open Phase 1 — architecture spec set over the Phase 0 evidence
docs/architecture/ now exists: README index, overview, five component specs (core-contract, engine-sqlite, engine-postgres, queues, deployment), ADR-001..007 carrying the Phase 0 resolved decisions (crate split, feature scope, per-engine drivers, dependency ownership, wake contract, tx seam), and the centralized open-questions tracker promotion: OQ-ST-01..08 mirror to OQ-01..08 one-to-one with statuses/resolutions carried; new Phase 1 questions append (OQ-09 scheduler collapse, OQ-10 contract versioning). Open Phase 1 work: OQ-04 contract pinning (high), OQ-05 queue semantics depth (high), OQ-06 honker-core quality read (high; fork-trigger gate), OQ-08 capability surface, OQ-09, OQ-10. Erratum fixed in phase-0 OQ-ST-04 (thread-affinity friction is SQLite-side, previously garbled as pg-side) and a stale scheduler- boundary pointer corrected in consumer-inventory.md. Two review passes run (findings: OQ-promotion numbering faithfulness, ADR back-reference sync) — all critical/warning findings resolved.
This commit is contained in:
1 parent
db73678090
commit
4391f6e879
17 files changed
+1812
-14
No files matched your search
@@ -0,0 +1,92 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# alkstore — Architecture
|
||||||
|
|
||||||
|
Architecture documentation for the alkstore project: one reactive
|
||||||
|
store interface (notify, streams, queues, locks, scheduler, outbox)
|
||||||
|
over SQLite and Postgres, with each engine native underneath
|
||||||
|
(see [overview.md](overview.md)).
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
**Phase 1 (Architecture) — in progress.** Phase 0 is complete
|
||||||
|
(`docs/research/phase-0.md`): both POCs ran and passed, the scope
|
||||||
|
inventory is confirmed, and the crate split, drivers, and ownership
|
||||||
|
postures are decided. This directory carries the architecture spec
|
||||||
|
build-out over that evidence base; all spec documents are `draft`
|
||||||
|
pending architecture review and OQ resolution.
|
||||||
|
|
||||||
|
## Architecture Documents
|
||||||
|
|
||||||
|
| Doc | Status | Purpose | Key OQs |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [overview.md](overview.md) | draft | Crate family, feature surface, non-goals, evidence base | — |
|
||||||
|
| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-04, OQ-08, OQ-09, OQ-10 |
|
||||||
|
| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: honker-core/rusqlite mapping | OQ-05, OQ-06, OQ-09 |
|
||||||
|
| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-05, OQ-08, OQ-09 |
|
||||||
|
| [queues.md](queues.md) | draft | Queue/scheduler/outbox semantics depth frame | OQ-05, OQ-09 |
|
||||||
|
| [deployment.md](deployment.md) | draft | Host semantics, connection budgets, knobs, matrix | OQ-04, OQ-08 |
|
||||||
|
| [open-questions.md](open-questions.md) | draft | OQ tracker (promoted from OQ-ST register) | — |
|
||||||
|
|
||||||
|
## Architecture Decision Records
|
||||||
|
|
||||||
|
| ADR | Title | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Reactive-core crate + per-engine crates | Accepted |
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope — inventory-confirmed surface | Accepted |
|
||||||
|
| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core, bridged seam | Accepted |
|
||||||
|
| [004](decisions/004-postgres-driver.md) | Postgres engine — tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
|
||||||
|
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract — opaque wake + re-read; notify-vs-streams split | Accepted |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Transactional seam — caller-held `TxHandle`, `*_tx` methods | Accepted |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Tracked in [open-questions.md](open-questions.md) (OQ-01..NN; the
|
||||||
|
Phase 0 register's OQ-ST-01..08 promote one-to-one — OQ-NN mirrors
|
||||||
|
OQ-ST-NN — with new Phase 1 questions appended after). Highlights,
|
||||||
|
in suggested resolution order (OQ-04 first):
|
||||||
|
|
||||||
|
- **OQ-04** (high): contract pinning — exact trait shape, handle
|
||||||
|
representation, error taxonomy, reserved strings, guarantee rows
|
||||||
|
for locks/scheduler.
|
||||||
|
- **OQ-05** (high): queue semantics depth — retry/backoff/dead-letter/
|
||||||
|
sweep design.
|
||||||
|
- **OQ-06** (high): honker-core quality read — fork-trigger gate.
|
||||||
|
- **OQ-09** (medium): scheduler as first-class mechanism vs queues +
|
||||||
|
`schedule()`.
|
||||||
|
- **OQ-08** (medium): capability-surface shape.
|
||||||
|
- **OQ-10** (medium): contract versioning across engine crates.
|
||||||
|
|
||||||
|
Resolved (Phase 0, kept with resolutions): OQ-01 (feature scope),
|
||||||
|
OQ-02 (crate split), OQ-03 (drivers), OQ-07 (extension surface cut).
|
||||||
|
|
||||||
|
No deferred OQs: all open questions are actionable Phase 1 work with
|
||||||
|
complete evidence bases.
|
||||||
|
|
||||||
|
## Document Lifecycle
|
||||||
|
|
||||||
|
| Status | Meaning | Transitions |
|
||||||
|
|---|---|---|
|
||||||
|
| `draft` | Under active development; may change significantly | → `reviewed` when the doc's OQs are resolved |
|
||||||
|
| `reviewed` | Architecture final; implementation may begin | → `stable` when implementation verified |
|
||||||
|
| `stable` | Locked; changes need review, may warrant an ADR | → `deprecated` when superseded |
|
||||||
|
| `deprecated` | Superseded; kept for reference | Removed when unreferenced |
|
||||||
|
|
||||||
|
All spec documents carry YAML frontmatter (`status`, `last_updated`);
|
||||||
|
ADRs carry a `## Status` section (Accepted/Proposed/Superseded).
|
||||||
|
|
||||||
|
## Provenance of decisions
|
||||||
|
|
||||||
|
Phase 1 inherits its decisions from Phase 0's evidence base — every
|
||||||
|
Accepted ADR above cites its POC findings and register record. The
|
||||||
|
research documents remain the deep background:
|
||||||
|
|
||||||
|
- `docs/research/phase-0.md` — vision, prior art, OQ-ST register,
|
||||||
|
convergence.
|
||||||
|
- `docs/research/consumer-inventory.md` — per-feature scope evidence.
|
||||||
|
- `docs/research/poc-sqlite-posture-findings.md` /
|
||||||
|
`docs/research/poc-pg-posture-findings.md` — measured ground.
|
||||||
@@ -0,0 +1,204 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# Core contract
|
||||||
|
|
||||||
|
The unified, engine-agnostic surface a `Store` exposes. This document
|
||||||
|
specifies WHAT the contract is; the per-engine specs map it onto their
|
||||||
|
machinery; ADRs carry the WHY. The starting artifact is the honker-rs
|
||||||
|
surface (the Phase 0 §Interface finding) scoped to the
|
||||||
|
inventory-confirmed features — [ADR-002](decisions/002-feature-scope.md)
|
||||||
|
— and pinned against it. Exact Rust shapes are OQ-04's pinning work;
|
||||||
|
the *obligations* are this document.
|
||||||
|
|
||||||
|
## Concepts
|
||||||
|
|
||||||
|
- **Store** — what a consumer opens from a connection string or file
|
||||||
|
path ([ADR-001](decisions/001-crate-split.md)): one handle, the engine
|
||||||
|
behind it chosen at open time. Consumer code never branches on
|
||||||
|
engine type (guiding principle 4).
|
||||||
|
- **Mechanism** — one of the surface's coordination families: notify,
|
||||||
|
streams, queues (+outbox), locks, scheduler. Delivery guarantees are
|
||||||
|
pinned per mechanism in [ADR-006](decisions/006-wake-and-delivery-contract.md)'s
|
||||||
|
table (notify/streams/queues); lock and scheduler guarantee rows are
|
||||||
|
OQ-04/OQ-09 resolution items, not assumed.
|
||||||
|
|
||||||
|
*(Guiding principles are defined in `docs/research/phase-0.md`
|
||||||
|
§Vision and cited by number throughout this directory.)*
|
||||||
|
- **Wake** — the opaque "something changed, re-read" signal the wake
|
||||||
|
contract delivers ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
The contract's central abstraction.
|
||||||
|
- **TxHandle** — the caller-held transaction object whose `*_tx`
|
||||||
|
methods make side effects commit-atomic with a business write
|
||||||
|
([ADR-007](decisions/007-transactional-seam.md)).
|
||||||
|
|
||||||
|
## The seam
|
||||||
|
|
||||||
|
```text
|
||||||
|
Store::open(config) -> Store
|
||||||
|
Store::begin_tx() -> TxHandle
|
||||||
|
```
|
||||||
|
|
||||||
|
Mechanism handles come off the store (or, for transactional variants,
|
||||||
|
off the handle):
|
||||||
|
|
||||||
|
```text
|
||||||
|
store.notify(channel, payload) handle.notify_tx(channel, payload)
|
||||||
|
store.stream(name) -> Stream handle.publish_tx / save_offset_tx
|
||||||
|
store.queue(name) -> Queue handle.enqueue_tx
|
||||||
|
store.try_lock(name, ttl) -> Lock
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact shape of these handles (traits vs concrete types, `dyn` vs
|
||||||
|
generic `TxHandle`) is OQ-04's pinning work.
|
||||||
|
|
||||||
|
## Mechanism contracts
|
||||||
|
|
||||||
|
### notify / listen
|
||||||
|
|
||||||
|
Fire-and-forget signals, commit-atomic when sent in a transaction
|
||||||
|
([ADR-007](decisions/007-transactional-seam.md)). No durability, no
|
||||||
|
replay, no per-listener retry; a listener attached after a commit
|
||||||
|
never sees it.
|
||||||
|
|
||||||
|
- `notify(channel, payload)` — payload ≤ 8000 bytes on Postgres
|
||||||
|
(client-side checked, typed error, verified by POC #2); no limit on
|
||||||
|
SQLite. The asymmetry is a documented engine capability, see
|
||||||
|
OQ-04/OQ-08.
|
||||||
|
- `listen(channel)` — starts from "now"; delivers
|
||||||
|
opaque wakes ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
Wakes are at-least-once, possibly coalesced (SQLite) or per-notify
|
||||||
|
(Postgres), possibly repeated after reconnect. **Consumers must be
|
||||||
|
idempotent on wake.** (The sketched return type — a wake receiver vs
|
||||||
|
a subscription handle — is an OQ-04 pinning item, like all exact
|
||||||
|
shapes here.)
|
||||||
|
- Failure surfaces as channel events, never silence: watcher death
|
||||||
|
closes the receiver (SQLite); the synthetic reconnect-wake (a
|
||||||
|
reserved channel, [ADR-004](decisions/004-postgres-driver.md)) covers
|
||||||
|
Postgres connection gaps.
|
||||||
|
- Channel name is the one piece of semantic content a wake carries —
|
||||||
|
the invalidation key for caching subscribers.
|
||||||
|
|
||||||
|
### streams
|
||||||
|
|
||||||
|
Durable pub/sub with per-consumer offsets
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md)). The durable
|
||||||
|
cousin of notify: publish is commit-atomic; every committed event is
|
||||||
|
readable by every consumer whose offset hasn't passed it;
|
||||||
|
replay-on-attach is the default; offsets are explicit and
|
||||||
|
transaction-aware (`save_offset_tx` gives exactly-once-within-a-
|
||||||
|
business-tx shape).
|
||||||
|
|
||||||
|
- `publish` / `publish_tx` / `publish_with_key` — append to the
|
||||||
|
stream's durable log.
|
||||||
|
- `read_since` / `read_from_consumer(offset)` — cursor-based reads.
|
||||||
|
- `save_offset` / `save_offset_tx` — consumer checkpoint, explicit;
|
||||||
|
**the contract's save is always explicit** (no auto-checkpoint
|
||||||
|
cadence — honker's per-binding ambiguity is not inherited).
|
||||||
|
- `subscribe(consumer)` — durable consumption: attach, read to
|
||||||
|
current tail, resume after restart from the stored offset.
|
||||||
|
|
||||||
|
### queues
|
||||||
|
|
||||||
|
Durable at-least-once work
|
||||||
|
([ADR-002](decisions/002-feature-scope.md)). The semantics-depth design
|
||||||
|
is [queues.md](queues.md)'s (OQ-05); the contract-level obligations
|
||||||
|
here:
|
||||||
|
|
||||||
|
- `enqueue` / `enqueue_tx` — commit-atomic; `EnqueueOpts { delay,
|
||||||
|
priority, max_attempts, expires, ... }`.
|
||||||
|
- `claim` / `claim_batch` — exactly-once handout under concurrency
|
||||||
|
(POC-pinned on both engines).
|
||||||
|
- Job handle: `ack / retry / fail / heartbeat`; visibility timeouts
|
||||||
|
make at-least-once *safe* (work re-appears if a claimant dies).
|
||||||
|
- `sweep_expired` — maintenance entry point (design in
|
||||||
|
[queues.md](queues.md)).
|
||||||
|
|
||||||
|
### named locks
|
||||||
|
|
||||||
|
TTL-bounded coordination locks, transactional-friendly:
|
||||||
|
|
||||||
|
- `try_lock(name, ttl)` — acquire or fail; `renew`; release on
|
||||||
|
explicit unlock or TTL expiry. Re-acquirable after expiry
|
||||||
|
(POC-pinned on Postgres; on SQLite it rests on honker's machinery —
|
||||||
|
the SQLite-side pin is in OQ-04's verification backlog).
|
||||||
|
- Lock names are shared-namespace (contract text pins collision
|
||||||
|
rules with the reserved namespace, OQ-04).
|
||||||
|
|
||||||
|
### outbox
|
||||||
|
|
||||||
|
A helper over queues, not a separate mechanism: enqueue inside the
|
||||||
|
business transaction + the delivery/consumption worker entry points.
|
||||||
|
Same evidence base and guarantee as queues
|
||||||
|
([ADR-002](decisions/002-feature-scope.md)).
|
||||||
|
|
||||||
|
### scheduler
|
||||||
|
|
||||||
|
Cron/`@every` enqueueing into named queues, leader-elected where the
|
||||||
|
engine has peers (SQLite: single-host, no election needed beyond
|
||||||
|
documented posture; Postgres: advisory-lock election). Whether this is
|
||||||
|
a first-class mechanism or queues + `schedule()` is OQ-09.
|
||||||
|
|
||||||
|
## Cross-cutting contracts
|
||||||
|
|
||||||
|
### Delivery-guarantee table
|
||||||
|
|
||||||
|
The per-mechanism table in
|
||||||
|
[ADR-006](decisions/006-wake-and-delivery-contract.md) is the contract
|
||||||
|
of record for notify/streams/queues; this spec inherits it and adds
|
||||||
|
the consumer obligations:
|
||||||
|
|
||||||
|
- wake idempotence (notify's at-least-once, coalescing behavior);
|
||||||
|
- explicit offset saves (streams);
|
||||||
|
- visibility-timeout budgeting (queues);
|
||||||
|
- `*_tx` operations are *only* durable after the caller's commit —
|
||||||
|
rollback drops job rows, event rows, notifications, and offset saves
|
||||||
|
together (the no-ghosts property, POC-pinned on both engines).
|
||||||
|
|
||||||
|
### Errors
|
||||||
|
|
||||||
|
`thiserror`-typed, per the family standard. The taxonomy (what
|
||||||
|
callers match on per mechanism; engine-capability errors like
|
||||||
|
`PayloadTooLarge` on pg vs no-limit on SQLite) is OQ-04's pinning
|
||||||
|
work.
|
||||||
|
|
||||||
|
### Capability surface
|
||||||
|
|
||||||
|
Whether the `Store` exposes engine capabilities at all — and if so,
|
||||||
|
which (payload limits, host semantics, wake-cadence knobs) — is
|
||||||
|
OQ-08's decision ([deployment.md](deployment.md)).
|
||||||
|
|
||||||
|
### Naming / reserved namespace
|
||||||
|
|
||||||
|
Consumer-visible names (channels, streams, queues, locks) share
|
||||||
|
engine-visible namespaces on Postgres (LISTEN channel names are
|
||||||
|
server-global per database). Exact rules: OQ-04. Decided already: a
|
||||||
|
**reserved/meta namespace exists** for engine-internal names — the
|
||||||
|
Postgres reconnect-wake channel, any internal bookkeeping — and
|
||||||
|
consumer names must not collide with it
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
| ADR | Decision | Summary |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Crate split | core + per-engine crates; single-driver binaries |
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope | inventory-confirmed features only; cut-flags explicit |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | opaque wake + re-read; notify-vs-streams guarantee split |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Tx seam | caller-held handle, `*_tx` methods, native commit-atomicity |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Open questions are tracked in
|
||||||
|
[open-questions.md](open-questions.md). Key
|
||||||
|
questions affecting this document:
|
||||||
|
|
||||||
|
- **OQ-04**: exact trait-surface pinning — handle shape, error
|
||||||
|
taxonomy, reserved strings, rename/grouping decisions (open,
|
||||||
|
high priority)
|
||||||
|
- **OQ-10**: contract versioning discipline across engine crates (open)
|
||||||
|
- **OQ-08**: capability-surface shape (open)
|
||||||
|
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
|
||||||
|
(open)
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# ADR-001: Reactive-core crate + per-engine crates
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The store must present one reactive interface (notify, streams, queues,
|
||||||
|
locks, scheduler, outbox) over two backing engines, SQLite and Postgres.
|
||||||
|
The engines differ sharply in how much of that surface is natively
|
||||||
|
theirs:
|
||||||
|
|
||||||
|
- The **SQLite** engine rides honker's existing, battle-tested machinery
|
||||||
|
(published `honker-core` — a port-and-adapt of a sync library to the
|
||||||
|
family's async-facing-trait posture).
|
||||||
|
- The **Postgres** engine is the build-heavy side: LISTEN/NOTIFY wiring
|
||||||
|
and the pg-boss-family queue schema are built by this crate.
|
||||||
|
|
||||||
|
A single crate with feature-gated engines would force both engines'
|
||||||
|
dependency graphs into every downstream binary that enables both
|
||||||
|
features. That is not hypothetical: rusqlite 0.40's `libsqlite3-sys`
|
||||||
|
link collision with sqlx's sqlite driver is a real constraint
|
||||||
|
(see [ADR-003](003-sqlite-driver.md)); single-driver binaries are the
|
||||||
|
only clean escape.
|
||||||
|
|
||||||
|
Scope evidence lives in `docs/research/consumer-inventory.md`; the
|
||||||
|
operator decision was recorded 2026-10-04 against
|
||||||
|
OQ-ST-02 (`docs/research/phase-0.md`).
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The project ships as a **family of crates**:
|
||||||
|
|
||||||
|
1. **`alkstore` (core)** — the crate carrying the unified trait
|
||||||
|
surface, types, error model, capability flags, and the contract
|
||||||
|
documentation. No driver dependencies. Compile-lean by construction:
|
||||||
|
the base crate has no engine machinery to keep out.
|
||||||
|
2. **`alkstore-sqlite`** — the SQLite engine implementing the core
|
||||||
|
surface. Single driver: rusqlite + honker-core ([ADR-003]).
|
||||||
|
3. **`alkstore-postgres`** — the Postgres engine implementing the core
|
||||||
|
surface. Single driver: tokio-postgres + deadpool-postgres
|
||||||
|
([ADR-004]).
|
||||||
|
4. **A mem-shaped engine** (in-tree or separate test crate) may exist as
|
||||||
|
a third implementation for tests and doctests, if the test story
|
||||||
|
turns out to want it. Treated as an implementation convenience, not
|
||||||
|
a contract artifact — decided at implementation time, not now.
|
||||||
|
|
||||||
|
Downstream consumers depend on `alkstore` plus exactly one engine
|
||||||
|
crate. A consumer binary links at most one driver.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- The engines' asymmetry of work is structural: the SQLite engine has a
|
||||||
|
small, mostly-wiring job; the Postgres engine carries the build-heavy
|
||||||
|
LISTEN/queue work. Neither is burdened by the other's dependencies.
|
||||||
|
- Any future engine (alkfs's in-tree needs, an ops-surface engine) is
|
||||||
|
additive — a new crate implementing the core traits — rather than a
|
||||||
|
feature-graph edit to one crate.
|
||||||
|
- Core-lean is structural, not a feature-discipline to be enforced by
|
||||||
|
review.
|
||||||
|
- The libsqlite3-sys link-collision class of problems is eliminated by
|
||||||
|
construction (single-driver binaries).
|
||||||
|
- Per-engine compilation/test gates are independent.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- A version-coordination duty: the core's trait surface is a contract
|
||||||
|
the engine crates must track. Version bumps in core must be adopted
|
||||||
|
by engines in lockstep when the contract changes (minor/major
|
||||||
|
discipline; no trait-default drift).
|
||||||
|
- Slightly more crate plumbing; naming/publishing overhead.
|
||||||
|
- Consumers who want *both* engines in one binary (rare, unsupported-by
|
||||||
|
design) cannot get a both-features build today.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- OQ-ST-02 (`docs/research/phase-0.md`) — the decision record with
|
||||||
|
options considered.
|
||||||
|
- `docs/research/consumer-inventory.md` — the inventory whose lean
|
||||||
|
(single crate) was superseded, with the correction recorded there.
|
||||||
|
- [ADR-003](003-sqlite-driver.md) — the SQLite driver choice whose
|
||||||
|
link-collision constraint motivates this split.
|
||||||
|
- [ADR-004](004-postgres-driver.md) — the Postgres driver choice.
|
||||||
|
- OQ-10 (`docs/architecture/open-questions.md`) — trait-versioning
|
||||||
|
duties created here.
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
# ADR-002: Feature scope — what the store surface includes
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Honker's full surface (the interface prior art) covers nine feature
|
||||||
|
families: notify/listen, streams, queues, scheduler, outbox, named
|
||||||
|
locks, rate limits, result storage, and loadable-extension serving.
|
||||||
|
The crate must decide which of these it ships, and the evidence-first
|
||||||
|
answer (`docs/research/consumer-inventory.md`, 2026-10-04) grades each
|
||||||
|
row by whether any consumer document actually names a need. This ADR
|
||||||
|
fixes that scope so contract work isn't done against features no
|
||||||
|
consumer wants.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
**In scope, first-class** (a named consumer is pinned or the operator
|
||||||
|
has recorded the need):
|
||||||
|
|
||||||
|
- **notify / listen** — the transactional fire-and-forget signal layer.
|
||||||
|
Pinned (alkfs path-tree invalidation; the `watch_fires_on_commit`
|
||||||
|
POC evidence) and documented (alkblobs fleet visibility).
|
||||||
|
- **streams** — durable pub/sub with per-consumer offsets and
|
||||||
|
replay-on-attach. Operator-authority record (2026-10-04,
|
||||||
|
`consumer-inventory.md`): type-filtered event watching / repo-change
|
||||||
|
subscriptions that notify cannot honestly serve (no replay, no
|
||||||
|
durability).
|
||||||
|
|
||||||
|
**In scope** (documented need, thinner):
|
||||||
|
|
||||||
|
- **named locks** — TTL coordination locks. Pinned (alkblobs fleet
|
||||||
|
sweeper) and documented (alkfs writer coordination); alkgit's CAS has
|
||||||
|
an alternative design a lock is allowed to replace but needn't.
|
||||||
|
- **queues** — durable at-least-once work with the transactional
|
||||||
|
enqueue shape. Documented (alkfs sync/fetch-on-miss outbox; alkblobs
|
||||||
|
maintenance cadence).
|
||||||
|
- **outbox helper** — a thin helper *over* queues (enqueue inside the
|
||||||
|
business transaction + a delivery/consumption worker shape), not a
|
||||||
|
separately-needed feature. Same evidence as queues.
|
||||||
|
- **scheduler** — cron/`@every` enqueueing into named queues,
|
||||||
|
leader-elected. Documented-thin; the family-wide
|
||||||
|
"who sweeps/renews/reaps" need. May collapse into queues if design
|
||||||
|
shows scheduler = queues + tick (tracked in OQ-09).
|
||||||
|
|
||||||
|
**Cut-flag** (no named consumer; not silently included):
|
||||||
|
|
||||||
|
- **rate limits** — alkgit enforces budgets in its own wire layer; a
|
||||||
|
store-level rate limit is not the proven mechanism. The row stays
|
||||||
|
listed with its flag visible; cut at implementation time rather than
|
||||||
|
carried on momentum. If a distributed (cross-process) rate-limit need
|
||||||
|
materializes, it gets a consumer-inventory row first.
|
||||||
|
- **result storage** (`save_result`/`get_result`/`sweep_results`) —
|
||||||
|
adjacent-tooling shape, useful but named by no consumer. Same
|
||||||
|
cut-flag treatment.
|
||||||
|
|
||||||
|
**Out:**
|
||||||
|
|
||||||
|
- The **loadable-extension surface** — no consumer needs it; every
|
||||||
|
identified consumer is in-process Rust attaching to its own
|
||||||
|
connection (OQ-ST-07, resolved cut-only). Cut unless a consumer
|
||||||
|
appears.
|
||||||
|
- Honker's exclusion lines, inherited as out: workflow DAGs, task
|
||||||
|
chains/chords, multi-writer replication, distributed (cross-machine)
|
||||||
|
locking. They stay out unless a consumer document grows a row.
|
||||||
|
|
||||||
|
New consumers add a row to `docs/research/consumer-inventory.md`
|
||||||
|
*before* being assumed into scope.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- The contract surface is small and every part of it has a nameable
|
||||||
|
consumer — contract-pinning work isn't spent on speculative surface.
|
||||||
|
- Scope honesty is mechanical: the inventory is the ledger, and the
|
||||||
|
cut-flag rows keep their flags visible instead of being silently
|
||||||
|
dropped or silently shipped.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- Rate limits and result storage, if ever needed, require the consumer
|
||||||
|
inventory + contract process first — no ad-hoc additions.
|
||||||
|
- Cutting the extension surface narrows honker compatibility; any
|
||||||
|
future non-Rust consumer of these SQLite files cannot use honker's
|
||||||
|
extension (acceptable: none exists).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/consumer-inventory.md` — per-feature evidence grades.
|
||||||
|
- OQ-ST-01, OQ-ST-07 (`docs/research/phase-0.md`) — resolved by this
|
||||||
|
evidence; promoted as OQ-01 and OQ-07 in
|
||||||
|
`docs/architecture/open-questions.md`.
|
||||||
|
- [core-contract.md](../core-contract.md) — the trait surface this
|
||||||
|
scope defines.
|
||||||
|
- [ADR-007](007-transactional-seam.md) — the transactional property all
|
||||||
|
in-scope enqueue/publish/notify features lean on.
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# ADR-003: SQLite engine — rusqlite + published honker-core, bridged at the trait seam
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The SQLite engine's options were never a single "rusqlite vs sqlx"
|
||||||
|
axis (OQ-ST-03). Three distinct postures existed:
|
||||||
|
|
||||||
|
1. **honker-core on our own rusqlite connection** — we own
|
||||||
|
connection/schema/watcher wiring; honker supplies the SQL-function
|
||||||
|
machinery (`attach_notify`, `attach_honker_functions`,
|
||||||
|
`bootstrap_honker_schema`).
|
||||||
|
2. **honker-rs as the SQLite substrate** (`Database::open` — honker
|
||||||
|
holds its own connections; sync-only; transactions pin the
|
||||||
|
connection mutex).
|
||||||
|
3. **raw SQL over sqlx-sqlite with the honker loadable extension**
|
||||||
|
(`.so` built from published source, loaded per pool connection).
|
||||||
|
|
||||||
|
POC #1 (`docs/research/poc-sqlite-posture-findings.md`, ran 2026-10-04;
|
||||||
|
POC #2 is its pg twin) measured postures 1
|
||||||
|
and 3 end-to-end and settled the choice. Constraint facts that outlive
|
||||||
|
the comparison:
|
||||||
|
|
||||||
|
- rusqlite 0.40.x (what honker-core 0.5.0 pins) needs rustc ≥ 1.99
|
||||||
|
(`cfg_select!`) — a deployment note for any binary linking it.
|
||||||
|
- sqlx's `libsqlite3-sys` range collides with rusqlite 0.40's — mixed
|
||||||
|
rusqlite+sqlx binaries need a vendored one-line patch today. The
|
||||||
|
per-engine-crate split ([ADR-001]) keeps engine binaries
|
||||||
|
single-driver, dissolving this.
|
||||||
|
- The async-facing-trait + sync-bridge posture is family-standard
|
||||||
|
(alktty REQ-TTY-01; alkblobs store-api.md): bridge-at-the-seam is a
|
||||||
|
supported posture, not a workaround — which removes native-async's
|
||||||
|
main differentiator before any measurement.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The SQLite engine uses **posture 1**: published `honker-core = 0.5`
|
||||||
|
as a library dependency over the crate's own rusqlite connections
|
||||||
|
(`bundled-sqlite` for hermetic builds).
|
||||||
|
|
||||||
|
- **Driver**: rusqlite, riding honker-core's pinned version.
|
||||||
|
- **Async seam**: bridged. The writer slot (`Writer`) +
|
||||||
|
reader pool (`Readers`) + every engine call in `spawn_blocking`.
|
||||||
|
A dedicated std-thread bridge (one thread owning the writer conn,
|
||||||
|
ops over mpsc) is the recorded optimization path if seam throughput
|
||||||
|
ever demands it — not the default.
|
||||||
|
- **Wake**: honker-core's `SharedUpdateWatcher` inherited (p50 ≈ 1.4 ms
|
||||||
|
at the default 1 ms `PRAGMA data_version` cadence, with
|
||||||
|
battle-tested failure handling — subscriber senders close on watcher
|
||||||
|
death, never silent-hang). A self-built thin watcher was benchmarked
|
||||||
|
only to confirm honker-core's is tighter (p50 1.40 vs 2.15 ms; max
|
||||||
|
29 vs 172 ms); no hybrid is warranted.
|
||||||
|
- **Transactions**: the engine opens `BEGIN IMMEDIATE` on the writer
|
||||||
|
slot and hands back a caller-held tx handle ([ADR-007]). The slot
|
||||||
|
lease models WAL's single-writer honestly: a long transaction parks
|
||||||
|
the writer — that is SQLite's shape, not an emulation artifact.
|
||||||
|
- **No `.so` runtime artifact, no vendored patches** in the engine
|
||||||
|
crate.
|
||||||
|
|
||||||
|
Rejected alternatives, briefly, for the record: posture 3 is *viable*
|
||||||
|
but pays ~2× p50 seam cost, a runtime dlopen artifact, wider
|
||||||
|
dependency tree, and sqlx 0.9 ergonomic warts (unsafe `extension()`,
|
||||||
|
`SqlSafeStr`) for no compensating advantage. Posture 2 (honker-rs)
|
||||||
|
was never measured because posture 1 dominates it on control with
|
||||||
|
comparable reuse.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- ~0.35 ms p50 transactional seam (measured), ~1.4 ms wake latency,
|
||||||
|
and honker-core's watcher failure-handling inherited for free.
|
||||||
|
- Static linkage: "open a path, get a store" needs no runtime
|
||||||
|
artifacts.
|
||||||
|
- The engine's job shrinks to wiring + the SQL surfaces honker doesn't
|
||||||
|
cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's
|
||||||
|
machinery directly) + the async bridge.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- honker-core pins rusqlite ^0.40.1; version movement in honker-core
|
||||||
|
moves our rusqlite. A honker-core quality read is the recorded fork
|
||||||
|
trigger ([ADR-005], OQ-06).
|
||||||
|
- rustc ≥ 1.99 required by rusqlite 0.40.x — binaries linking this
|
||||||
|
engine carry that toolchain floor (deployment-matrix row,
|
||||||
|
[deployment.md](../deployment.md)).
|
||||||
|
- The sync bridge means engine ops pay a `spawn_blocking` hop — mostly
|
||||||
|
invisible next to SQLite write costs, but it shapes the tx-handle
|
||||||
|
design ([ADR-007]).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/poc-sqlite-posture-findings.md` — the measurements.
|
||||||
|
- OQ-ST-03 (`docs/research/phase-0.md`) — resolution record.
|
||||||
|
- [ADR-001](001-crate-split.md) — why the engine crate is
|
||||||
|
single-driver.
|
||||||
|
- [ADR-004](004-postgres-driver.md) — the Postgres counterpart.
|
||||||
|
- [ADR-007](007-transactional-seam.md) — the tx-handle shape both
|
||||||
|
engines implement.
|
||||||
|
- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this
|
||||||
|
decision defines.
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# ADR-004: Postgres engine — tokio-postgres + deadpool-postgres, hand-rolled LISTEN forwarder
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The Postgres engine's driver question (OQ-ST-03, Postgres half) had
|
||||||
|
three candidate families: sqlx (single API across engines, and the
|
||||||
|
driver pgboss-rs carries), tokio-postgres + deadpool-postgres (the
|
||||||
|
alkblobs POC evidence base, natively async), and adopt/fork pgboss-rs
|
||||||
|
(bringing sqlx where the queue lives). POC #2
|
||||||
|
(`docs/research/poc-pg-posture-findings.md`, ran 2026-10-04) validated
|
||||||
|
the tokio-postgres posture end-to-end and measured the pieces the
|
||||||
|
contract depends on. The sqlite-arm comparison facts from
|
||||||
|
[ADR-003] apply on this side too: pgboss-rs brings sqlx (a second
|
||||||
|
driver per binary), has no LISTEN/NOTIFY at all (verified against the
|
||||||
|
checkout — consumption is `fetch_job` polling), so the push-reactivity
|
||||||
|
half is this crate's work regardless of fork-or-adopt.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The Postgres engine uses:
|
||||||
|
|
||||||
|
- **tokio-postgres 0.7.x + deadpool-postgres 0.14.x** — pooled
|
||||||
|
connections for queries/claims (exactly-once via
|
||||||
|
`FOR UPDATE SKIP LOCKED`), natively async (client is `Send + Sync` —
|
||||||
|
no bridge, no spawn_blocking, the tx handle holds the pooled object
|
||||||
|
directly).
|
||||||
|
|
||||||
|
- **A hand-rolled LISTEN forwarder** (the POC's ~90-line shape) — a
|
||||||
|
dedicated non-pooled listener connection per process, a poll-message
|
||||||
|
loop fanning out to a bounded broadcast channel, immediate reconnect
|
||||||
|
with exponential backoff (50 ms → 2 s cap), re-LISTEN from the
|
||||||
|
channel list after every reconnect, and a **synthetic reconnect-wake
|
||||||
|
on a reserved channel** closing the no-replay hole
|
||||||
|
([ADR-006]). `postgres-notify` 0.3.8 was evaluated in-probe and
|
||||||
|
passed over (derive-not-adopt: lazy reconnect, connect_script
|
||||||
|
skipped at initial connect, unquoted-identifier LISTENs,
|
||||||
|
single-maintainer posture; it stays a recorded fallback if upstream
|
||||||
|
improves).
|
||||||
|
|
||||||
|
- **Queue machinery re-derived on this driver** with the pg-boss
|
||||||
|
schema family as design reference — not adopted/forked as a
|
||||||
|
dependency ([ADR-005]; semantics depth is [queues.md]'s work,
|
||||||
|
OQ-ST-05). The minimal-queue ground is measured: enqueue/claim/ack
|
||||||
|
with `SKIP LOCKED` is ~40 lines of SQL and passed the full property
|
||||||
|
suite.
|
||||||
|
|
||||||
|
- **Default consumption posture: LISTEN-driven claim with a re-poll
|
||||||
|
safety net**; poll-only remains the fallback (interval-tunable) when
|
||||||
|
a LISTEN connection is unavailable or unwanted. Measured: claim
|
||||||
|
latency p50 3–6 ms LISTEN-driven vs 32–50 ms at a 50 ms poll.
|
||||||
|
|
||||||
|
Structural constraints the engine owns (all test-pinned in POC #2):
|
||||||
|
|
||||||
|
- Pooled connections **cannot carry LISTEN** (deadpool#360 —
|
||||||
|
registers server-side, never delivers). The listener connection is a
|
||||||
|
per-process budget line outside the pool (`max_size + 1`).
|
||||||
|
- `pg_notify` payloads are ≤ 8000 bytes; the engine checks the limit
|
||||||
|
client-side and returns a typed error before the round-trip. Large
|
||||||
|
payloads ride a table row with the id in the notification (the
|
||||||
|
outbox shape).
|
||||||
|
- A query-vs-poll starvation deadlock exists if the listener's poll
|
||||||
|
loop isn't running before the first client query on the connection;
|
||||||
|
and dropping the `Client` closes the server session even if the
|
||||||
|
`Connection` task survives. Both are hand-rolled-forwarder failure
|
||||||
|
modes, owned and test-pinned by the engine.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- Single-driver engine, natively async, zero version conflicts, no
|
||||||
|
vendoring.
|
||||||
|
- The push channel pgboss-rs lacks, this engine gets — measured at
|
||||||
|
5–16× claim-latency improvement over even a 50 ms poll.
|
||||||
|
- Multi-host is native (no single-host assumption anywhere; property
|
||||||
|
tests ran all-through-network).
|
||||||
|
- The forwarder's two deadlock pitfalls are *learned territory* with
|
||||||
|
pinned tests, not unknowns.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- The listener forwarder is ours to maintain (~90 lines + the two
|
||||||
|
pinned failure modes) — the cost of not adopting `postgres-notify`.
|
||||||
|
- Per-process connection budget grows by one listener connection per
|
||||||
|
process that listens (a deployment-matrix row; consumers sizing
|
||||||
|
pools must account for it).
|
||||||
|
- The pg queue schema is re-derived work; the pg-boss family is
|
||||||
|
reference, not free.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/poc-pg-posture-findings.md` — measurements and the
|
||||||
|
pinned property suite.
|
||||||
|
- OQ-ST-03, OQ-ST-05's posture half (`docs/research/phase-0.md`).
|
||||||
|
- [ADR-001](001-crate-split.md) — single-driver binaries.
|
||||||
|
- [ADR-005](005-dependency-ownership.md) — the ownership calculus this
|
||||||
|
decision applies per-subsystem.
|
||||||
|
- [ADR-006](006-wake-and-delivery-contract.md) — the wake contract the
|
||||||
|
forwarder implements on this engine.
|
||||||
|
- [engine-postgres.md](../engine-postgres.md) — the engine spec.
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# ADR-005: Dependency ownership — published libraries by default, named fork triggers
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (posture); fork triggers are live assessments tracked in OQ-06
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The crate's reactive machinery sits on third-party work: honker
|
||||||
|
(SQLite side) and, conceptually, pgboss-rs (Postgres queue family).
|
||||||
|
Neither is adopted-by-adjacency — per guiding principle 3, ownership of
|
||||||
|
each subsystem is a deliberate per-question decision. The workspace
|
||||||
|
precedent is the targeted fork (alksocks' fast-socks5 extraction), but
|
||||||
|
a fork is expensive (especially when the forked code is sync and the
|
||||||
|
family standard is tokio — a fork of honker-rs is already a serious
|
||||||
|
port).
|
||||||
|
|
||||||
|
Two POCs (2026-10-04) measured the relevant postures per subsystem;
|
||||||
|
each carries its own record. This ADR fixes the *default calculus* and
|
||||||
|
the per-subsystem resolutions so Phase 1 design work starts from
|
||||||
|
stable ground.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
**Default posture: consume published libraries.** Fork or vendor only
|
||||||
|
when a named trigger fires. Per subsystem:
|
||||||
|
|
||||||
|
| Subsystem | Posture | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| honker-core (SQLite engine machinery) | published library, `honker-core = 0.5` | Fork trigger: the Phase 1 quality read of the watcher/transactional core finds a defect, OR a needed change upstream won't take. OQ-06 tracks the read. |
|
||||||
|
| rusqlite | published, riding honker-core's pin | Pin movement is honker-core's; we ride it ([ADR-003]). |
|
||||||
|
| tokio-postgres + deadpool-postgres | published library, as-is | Clean, zero conflicts, actively maintained (POC #2). |
|
||||||
|
| postgres-notify | **not adopted** (derive-not-adopt) | Lazy reconnect, connect_script skipped at initial connect, unquoted-identifier LISTENs, single maintainer. Hand-rolled forwarder instead ([ADR-004]). Fallback if upstream improves materially. |
|
||||||
|
| pg-boss queue machinery (pgboss-rs / node pg-boss) | **re-derived** on our driver; schema family as *design reference* only | pgboss-rs brings sqlx (second driver per binary) and has no LISTEN/NOTIFY (verified); the push half is ours either way, so queue-machinery reuse value is the honest comparison point and it loses on that arithmetic. Semantics depth: [queues.md], OQ-05. |
|
||||||
|
| honker-rs interface shape | **design reference**, not a dependency | Sync-only (no tokio); the surface is prior art for contract pinning ([ADR-006]), not code to ship. |
|
||||||
|
|
||||||
|
If any fork fires, forking is normal work we own (license/provenance
|
||||||
|
recorded per AGENTS.md §3 at adoption time) — it is not an exception
|
||||||
|
to this posture.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- Zero vendored code in the tree by default; upgrades are
|
||||||
|
`cargo update` work, not patch-management work.
|
||||||
|
- The fork triggers are concrete and named *before* the quality read,
|
||||||
|
so the read produces a decision, not a debate.
|
||||||
|
- Per-subsystem votes are recorded with evidence, so no future
|
||||||
|
discussion re-litigates from scratch.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- honker-core 0.5 remains alpha-quality software per its own README in
|
||||||
|
the link graph; the quality read (OQ-06) is the mitigation gate.
|
||||||
|
- The pg queue machinery is written by us — the pg-boss family's
|
||||||
|
battle-tested edge cases must be re-earned by design + tests
|
||||||
|
([queues.md]).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/poc-sqlite-posture-findings.md` §"What feeds where"
|
||||||
|
(posture votes), `docs/research/poc-pg-posture-findings.md`
|
||||||
|
§"Dependency postures".
|
||||||
|
- OQ-ST-05, OQ-ST-06 (`docs/research/phase-0.md`) — the resolution
|
||||||
|
records.
|
||||||
|
- [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) —
|
||||||
|
the per-engine driver decisions applying this posture.
|
||||||
|
- OQ-06 (`docs/architecture/open-questions.md`) — the quality-read
|
||||||
|
tracker.
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
# ADR-006: Wake contract — opaque wake + re-read, and the notify-vs-streams delivery split
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (the contract skeleton the evidence supports); remaining
|
||||||
|
surface pinning is OQ-04's contract work against this ADR
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The two engines' wake mechanisms are structurally different — SQLite
|
||||||
|
wakes by a watcher polling `PRAGMA data_version` (deliver-on-commit,
|
||||||
|
no server push), Postgres by LISTEN/NOTIFY (server push,
|
||||||
|
connection-bound, no replay). A naive unification either collapses to
|
||||||
|
polling behavior on the strong side or promises semantics only one
|
||||||
|
engine has. Both POCs (2026-10-04) measured the pieces and — the
|
||||||
|
load-bearing finding — verified that **the two engines can share the
|
||||||
|
same wake contract unchanged**: wake is opaque, delivery is not
|
||||||
|
guaranteed to be precise, and consumers re-read state.
|
||||||
|
|
||||||
|
Honker's own processing-guarantees table is the cautionary prior art:
|
||||||
|
per-binding auto-checkpoint-vs-manual-save ambiguity produces
|
||||||
|
*different* guarantees under one function name. A single-crate version
|
||||||
|
must pick one answer per mechanism, not inherit the table.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The contract fixes three things:
|
||||||
|
|
||||||
|
### 1. The unified wake contract: opaque wake + re-read state
|
||||||
|
|
||||||
|
`listen()` (and stream/queue subscription wake) delivers an **opaque
|
||||||
|
wake signal: "something changed; re-read the state you care about."**
|
||||||
|
The contract never carries semantic content, ordering promises, or
|
||||||
|
change descriptions. Consumers that need the actual data re-read it
|
||||||
|
(the hot-path pattern the ecosystem already uses: a subscriber holding
|
||||||
|
a cache invalidates on wake and re-reads, instead of re-polling).
|
||||||
|
|
||||||
|
- SQLite: `data_version` watcher fires on commit; wakes **coalesce**
|
||||||
|
under bursts (overtriggering on purpose — "one indexed SELECT is
|
||||||
|
cheap; a missed wake is a correctness bug").
|
||||||
|
- Postgres: LISTEN delivers per-notification (no coalescing), and the
|
||||||
|
forwarder's synthetic reconnect-wake covers connection gaps.
|
||||||
|
- Both: consumer code must be correct if wakes repeat, coalesce, or
|
||||||
|
arrive in any order, with at-least-once wake delivery. Exactly-once
|
||||||
|
*processing* semantics belong to queues/streams (below), never to
|
||||||
|
the wake layer.
|
||||||
|
|
||||||
|
Measured ground: wake latency p50 ≈ 1.1–2.2 ms on both engines at
|
||||||
|
default cadence; per-payload burst delivery verified (30/30 on pg,
|
||||||
|
coalesced-but-complete re-read on SQLite).
|
||||||
|
|
||||||
|
### 2. Delivery guarantees, pinned per mechanism (no per-binding table)
|
||||||
|
|
||||||
|
| Mechanism | Durability | Replay | Atomicity | Guarantee |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `notify` / listen | none | never | commit-atomic (delivers at commit; rollback drops) | fire-and-forget, at-most-once per listener session |
|
||||||
|
| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset |
|
||||||
|
| queues | durable row | claim/ack model | enqueue is commit-atomic | at-least-once *work* with visibility timeouts |
|
||||||
|
|
||||||
|
- The trait **does not promise replay under `listen()`** — on either
|
||||||
|
engine. Durability+replay needs are what streams are for; this split
|
||||||
|
is the contract's load-bearing line.
|
||||||
|
- Listener sessions start "from now" (SQLite: the watcher's current
|
||||||
|
state; Postgres: `MAX(id)`-equivalent at attach). History belongs to
|
||||||
|
streams.
|
||||||
|
- Wake *delivery* failures surface, not silence: watcher death closes
|
||||||
|
subscriber channels (SQLite, honker-core's
|
||||||
|
`WatcherDeathGuard` behavior); Postgres connection gaps surface as
|
||||||
|
the synthetic reconnect-wake on a reserved channel
|
||||||
|
([ADR-004], [engine-postgres.md]).
|
||||||
|
|
||||||
|
### 3. Reserved names are a contract surface
|
||||||
|
|
||||||
|
The Postgres forwarder's synthetic reconnect-wake needs a reserved
|
||||||
|
channel namespace no consumer channel may collide with. The naming
|
||||||
|
convention for reserved/meta channels (and any internal queue/stream
|
||||||
|
names) is pinned in the core contract — exact strings are OQ-04
|
||||||
|
contract work; the *existence of a reserved namespace* is decided
|
||||||
|
here.
|
||||||
|
|
||||||
|
### The caching-subscriber pattern rides the same contract
|
||||||
|
|
||||||
|
A subscriber that also holds a cache gets invalidation from the
|
||||||
|
opaque wake + re-read: the contract deliberately does not carry
|
||||||
|
key-level payloads, so cache clients key their invalidation on their
|
||||||
|
own read-set, informed by channel names (the channel name is the one
|
||||||
|
piece of semantic content `listen()` carries). Key-payload design,
|
||||||
|
if any consumer needs richer invalidation, is a
|
||||||
|
consumer-inventory-row-gated extension — not assumed now.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- One wake contract, verified identically on both engines — the
|
||||||
|
"how does a change become visible" question alkstore exists to
|
||||||
|
answer, answered once.
|
||||||
|
- The delivery-guarantee table is pinned per mechanism, killing the
|
||||||
|
per-binding ambiguity class honker's docs exhibit.
|
||||||
|
- Consumer code branches on *mechanism choice* (notify vs streams vs
|
||||||
|
queues), never on engine type (guiding principle 4; the honest
|
||||||
|
single/multi-host line is [deployment.md]'s and OQ-08's, not this
|
||||||
|
contract's).
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- The opaque wake is deliberately less informative than
|
||||||
|
key-payload notify systems; consumers needing fine-grained
|
||||||
|
invalidation pay a re-read or need streams.
|
||||||
|
- Wake overtriggering on SQLite means consumers must be idempotent on
|
||||||
|
wake (a documented consumer obligation, part of the contract text).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/poc-sqlite-posture-findings.md` §Probe 2,
|
||||||
|
§"What feeds where"; `docs/research/poc-pg-posture-findings.md`
|
||||||
|
§Sub-module W.
|
||||||
|
- OQ-ST-04 (`docs/research/phase-0.md`) — de-risked by these
|
||||||
|
measurements; the contract remainder is OQ-04 +
|
||||||
|
`core-contract.md`.
|
||||||
|
- [ADR-002](002-feature-scope.md) — notify/listen and streams are
|
||||||
|
first-class; the guarantee split is why both exist.
|
||||||
|
- [ADR-004](004-postgres-driver.md) — the forwarder owning the
|
||||||
|
Postgres side of this contract.
|
||||||
|
- [ADR-007](007-transactional-seam.md) — commit-atomicity's mechanism
|
||||||
|
(the `*_tx` seam).
|
||||||
|
- [core-contract.md](../core-contract.md) — the spec carrying this
|
||||||
|
contract's full surface.
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
# ADR-007: Transactional seam — caller-held tx handle with `*_tx` methods
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (the seam shape both POCs verified); handle-type ergonomics
|
||||||
|
(generic vs downcast) is OQ-04 contract work
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The load-bearing property of the whole store (guiding principle 2) is
|
||||||
|
**transactional local-adjacency**: enqueue/publish/notify inside the
|
||||||
|
same transaction as the caller's business write; rollback drops both.
|
||||||
|
Serving it through an async trait over two transaction models —
|
||||||
|
rusqlite's sync, thread-bound connection vs tokio-postgres's
|
||||||
|
`Send + Sync` client — is the one genuinely driver-coupled design
|
||||||
|
point (OQ-ST-03's framing). Both POCs (2026-10-04) built the candidates
|
||||||
|
and measured them.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The core contract's transactional seam is a **caller-held transaction
|
||||||
|
handle**:
|
||||||
|
|
||||||
|
```text
|
||||||
|
store.begin_tx() -> TxHandle
|
||||||
|
handle.enqueue_tx(name, opts, payload) -> job id
|
||||||
|
handle.publish_tx(name, payload) -> event id
|
||||||
|
handle.notify_tx(channel, payload)
|
||||||
|
handle.save_offset_tx(stream, consumer, offset)
|
||||||
|
handle.commit() / handle.rollback()
|
||||||
|
```
|
||||||
|
|
||||||
|
- The handle is owned by the caller across `await` points; every
|
||||||
|
`*_tx` operation lands in *the caller's transaction*; commit/rollback
|
||||||
|
are explicit and owned by the caller. Non-`_tx` operations are the
|
||||||
|
auto-commit convenience counterparts, each atomic alone.
|
||||||
|
- **Postgres** ([ADR-004]): the handle holds the pooled connection
|
||||||
|
object directly (tokio-postgres `Client` is `Send + Sync` — verified
|
||||||
|
by a compile-time probe). Straight `.await`s, no bridge, no hop
|
||||||
|
cost. Commit/rollback return the object to the pool.
|
||||||
|
- **SQLite** ([ADR-003]): the handle is a **writer-slot lease** —
|
||||||
|
`begin_tx` acquires the `Writer` slot and opens `BEGIN IMMEDIATE`;
|
||||||
|
every handle op round-trips `spawn_blocking` to that connection
|
||||||
|
(rusqlite is not `Send`-across-await / not `Sync`). Holding the slot
|
||||||
|
across `await` points *is* how the lease works — a long transaction
|
||||||
|
parks the single WAL writer, which is SQLite's own shape.
|
||||||
|
- **Closure-scoped transactions** (`with_tx`) are *available as a
|
||||||
|
wrapper over* the handle shape, not instead of it (the POC verified
|
||||||
|
the reverse composition doesn't work: a closure cannot outlive
|
||||||
|
itself, so caching-subscriber state can't escape it).
|
||||||
|
- Both engines deliver the property **natively** — no emulation:
|
||||||
|
in-tx notify/Notify delivers only at commit; rollback drops job rows,
|
||||||
|
business rows, and notifications together (the POC property tests on
|
||||||
|
both engines green: no ghosts).
|
||||||
|
|
||||||
|
The POC's trait sketch carried two frictions, both now Phase 1
|
||||||
|
contract-pinning input rather than open risk:
|
||||||
|
|
||||||
|
1. The `as_any_mut` downcast per engine for `*_tx` methods on a
|
||||||
|
`dyn TxHandle` — small, but a generic or enum-favored handle may be
|
||||||
|
cleaner. OQ-04 decides with the full contract.
|
||||||
|
2. Thread-affinity of rusqlite tx ops (SQLite handle ops must
|
||||||
|
round-trip the same blocking thread) — inherent to [ADR-003]'s
|
||||||
|
bridge, documented as a contract note ("SQLite tx ops are
|
||||||
|
serialized by the writer slot"), not a defect.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- The load-bearing property is one seam, both engines, POC-verified
|
||||||
|
end-to-end — the dual-write problem is structurally prevented, not
|
||||||
|
consumer-disciplined.
|
||||||
|
- Long transactions compose naturally (checkout-then-work shape);
|
||||||
|
nothing about the seam forbids multi-statement business
|
||||||
|
transactions with interleaved reads.
|
||||||
|
- Per-engine bridging differences are invisible to consumer code: the
|
||||||
|
same `*_tx` calls, the same commit/rollback ownership.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- The SQLite handle parks the writer for its duration — a slow
|
||||||
|
consumer transaction throttles all writers on the file. This is
|
||||||
|
honest (WAL single-writer), but contract docs must say it so
|
||||||
|
consumers budget transactions accordingly.
|
||||||
|
- Per-op `spawn_blocking` hop on SQLite tx ops (measured fine at
|
||||||
|
~0.35 ms p50; the dedicated-thread bridge is the recorded
|
||||||
|
optimization).
|
||||||
|
- `with_tx` wrapping is a convenience surface we must ship and test
|
||||||
|
so it doesn't accrete ad-hoc in consumer code.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/research/poc-sqlite-posture-findings.md` §Arm A, §Probe 1/3;
|
||||||
|
`docs/research/poc-pg-posture-findings.md` §Sub-module T.
|
||||||
|
- OQ-ST-04 (`docs/research/phase-0.md`), the `*_tx` seam bullet.
|
||||||
|
- [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) —
|
||||||
|
the per-engine handle mechanics.
|
||||||
|
- [ADR-006](006-wake-and-delivery-contract.md) — commit-atomicity is
|
||||||
|
the `*_tx` property this seam carries into the notify contract.
|
||||||
|
- [core-contract.md](../core-contract.md) — the spec whose transaction
|
||||||
|
section this ADR defines.
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# Deployment
|
||||||
|
|
||||||
|
What a deployer must know to size, run, and reason about alkstore
|
||||||
|
engines: host semantics, connection budgets, durability knobs, and
|
||||||
|
where engine differences may honestly surface in the contract. The
|
||||||
|
capability-surface *decision* (how much of this the trait exposes) is
|
||||||
|
OQ-08's; this document holds the facts and the decision's frame.
|
||||||
|
|
||||||
|
## Host semantics
|
||||||
|
|
||||||
|
| Engine | Host posture | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| SQLite | **single-machine**, file-backed | NFS two-writers unsupported (honker's honesty posture, inherited, [ADR-003](decisions/003-sqlite-driver.md)). Cross-process *on one host* is verified POC ground (`data_version` is cross-process by nature). |
|
||||||
|
| Postgres | **multi-host native** | Nothing assumes a shared host; POC #2 ran all-through-network (docker bridge) with the same properties ([ADR-004](decisions/004-postgres-driver.md)). |
|
||||||
|
|
||||||
|
The unified trait must not pretend SQLite is multi-host — but whether
|
||||||
|
that honesty lives as runtime capability flags, compile-time engine
|
||||||
|
knowledge, or a documented matrix only is OQ-08
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md) note: the
|
||||||
|
trait's shape constrains where capability differences can surface).
|
||||||
|
|
||||||
|
Options for OQ-08, with their shape:
|
||||||
|
|
||||||
|
1. **Compile-time only** — a consumer chooses an engine crate at
|
||||||
|
dependency time; the engine's docs carry its deployment facts.
|
||||||
|
Smallest contract; nothing runtime to match on.
|
||||||
|
2. **`Store::capabilities()`** — a runtime description
|
||||||
|
(payload limits, wake cadence knobs, host semantics). Lets a
|
||||||
|
consumer adapt (e.g., chunk large notify payloads) but adds a
|
||||||
|
contract surface all engines must keep honest.
|
||||||
|
3. **Deployment matrix only** (this document) — no API surface. The
|
||||||
|
honest-middle choice; matches the ecosystem's doc-first posture
|
||||||
|
but provides no programmatic guard.
|
||||||
|
|
||||||
|
## Connection budgets
|
||||||
|
|
||||||
|
### SQLite engine
|
||||||
|
|
||||||
|
- Connections are in-process (writer + reader pool + watcher thread
|
||||||
|
owning a connection). No external budget lines; the file lock is
|
||||||
|
the OS-level resource.
|
||||||
|
|
||||||
|
### Postgres engine
|
||||||
|
|
||||||
|
| Connection class | Count | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| Pool | `max_size` per process | claims/queries via deadpool |
|
||||||
|
| Listener | **+1** per LISTEN-ing process | non-pooled, dedicated; pooled connections cannot carry LISTEN (deadpool#360, test-pinned — [ADR-004](decisions/004-postgres-driver.md)) |
|
||||||
|
| — | — | Sizing rule: `max_size + 1` per process; verify end-to-end accounting (POC #2's pgdiag-st-4: pool 6 + 1 listener + probe conns = exact server-side count) |
|
||||||
|
|
||||||
|
Shared-server co-tenancy (the alkblobs ADR-008 precedent —
|
||||||
|
`/workspace/@alkdev/alkblobs/docs/architecture/decisions/` — consumer
|
||||||
|
tables co-tenant the pg instance) is supported and expected — the
|
||||||
|
[naming / reserved namespace contract](core-contract.md#naming--reserved-namespace)
|
||||||
|
protects reserved names; queue/stream/lock tables are schema-named to
|
||||||
|
avoid collisions (layout decision in [queues.md](queues.md), OQ-05's
|
||||||
|
namespace bullet).
|
||||||
|
|
||||||
|
## Durability knobs
|
||||||
|
|
||||||
|
| Engine | Knob | Shape |
|
||||||
|
|---|---|---|
|
||||||
|
| SQLite | `synchronous` | WAL + `NORMAL` shipped ([ADR-003](decisions/003-sqlite-driver.md)); FULL is available consumer-side for stricter durability; commit fsyncs land at WAL checkpoints (the ~1000-commit spike cadence, POC #1) |
|
||||||
|
| Postgres | `synchronous_commit` | per-session knob; `on` is ship config (p50 2.40 ms seam); `off` trades max-tail (40.9 ms) for slightly better p50 — measured, honest trade ([ADR-004](decisions/004-postgres-driver.md)); session-level SET mechanics POC-verified |
|
||||||
|
|
||||||
|
These are engine-configuration concerns, *not* trait surface (a
|
||||||
|
consumer may set them via their own engine config — what part of
|
||||||
|
engine config is contract-level `Store::open` shape vs engine-crate
|
||||||
|
docs is OQ-04's config-shape item).
|
||||||
|
|
||||||
|
## Toolchain / platform notes
|
||||||
|
|
||||||
|
| Note | Engine | Affects |
|
||||||
|
|---|---|---|
|
||||||
|
| rusqlite 0.40.x needs rustc ≥ 1.99 | SQLite | any binary linking `alkstore-sqlite` ([ADR-003](decisions/003-sqlite-driver.md)) |
|
||||||
|
| `bundled-sqlite` adds a C build (~10 s dev, cacheable) | SQLite | build/CI time |
|
||||||
|
| `libsqlite3-sys` collision with sqlx today | any mixed-driver binary | structurally avoided ([ADR-001](decisions/001-crate-split.md) single-driver rule) |
|
||||||
|
| Listener `application_name` set for diagnosability (kill-targetable) | Postgres | ops runbooks ([ADR-004](decisions/004-postgres-driver.md)) |
|
||||||
|
|
||||||
|
## Consumer-facing latency profile (indicative, POC-measured)
|
||||||
|
|
||||||
|
From both POCs (single-box, relative shapes are the deliverable —
|
||||||
|
[ADR-003](decisions/003-sqlite-driver.md) and
|
||||||
|
[ADR-004](decisions/004-postgres-driver.md) carry the full tables):
|
||||||
|
|
||||||
|
- Tx seam: SQLite ~0.35 ms p50; Postgres ~2.4 ms p50 (ship config).
|
||||||
|
- Wake: ~1.1–2.2 ms p50 both engines at default cadence.
|
||||||
|
- Queue claim: LISTEN-driven 3–6 ms p50 (pg); poll-only interval-bound
|
||||||
|
(32–50 ms at a 50 ms poll).
|
||||||
|
- Absolute numbers will differ per hardware/network; they set
|
||||||
|
*expectations of order*, not SLAs — contract docs must not bake
|
||||||
|
them in ([ADR-007](decisions/007-transactional-seam.md)'s pg-POC
|
||||||
|
note).
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
| ADR | Decision | Summary |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Crate split | single-driver binaries shape the matrix |
|
||||||
|
| [003](decisions/003-sqlite-driver.md) | SQLite driver | bundling, toolchain floor |
|
||||||
|
| [004](decisions/004-postgres-driver.md) | Postgres driver | listener budget line, forwarder posture |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | where capability differences may surface |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Open questions are tracked in
|
||||||
|
[open-questions.md](open-questions.md). Key
|
||||||
|
questions affecting this document:
|
||||||
|
|
||||||
|
- **OQ-08**: capability-surface shape — compile-time vs runtime flags
|
||||||
|
vs matrix-only (open)
|
||||||
|
- **OQ-04**: engine-config surface in the contract's `Store::open`
|
||||||
|
shape (shared) (open)
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# Postgres engine
|
||||||
|
|
||||||
|
The `alkstore-postgres` engine implements
|
||||||
|
[core-contract.md](core-contract.md) on tokio-postgres +
|
||||||
|
deadpool-postgres. This spec records WHAT the engine is internally —
|
||||||
|
connection architecture, the LISTEN forwarder, queue machinery
|
||||||
|
re-derivation — not code-level HOW. Decisions live in ADRs; contract
|
||||||
|
obligations live in the core spec.
|
||||||
|
|
||||||
|
## Identity and posture
|
||||||
|
|
||||||
|
- Single driver, natively async: tokio-postgres 0.7.x +
|
||||||
|
deadpool-postgres 0.14.x ([ADR-004](decisions/004-postgres-driver.md)).
|
||||||
|
No bridge, no `spawn_blocking`; the tx handle holds the pooled
|
||||||
|
object directly (client is `Send + Sync` — POC #2 compile-probe
|
||||||
|
verified).
|
||||||
|
- Multi-host by nature: connections are per-process state; nothing
|
||||||
|
assumes a shared host ([ADR-004](decisions/004-postgres-driver.md),
|
||||||
|
verified all-through-network in POC #2). See
|
||||||
|
[deployment.md](deployment.md).
|
||||||
|
- Queue machinery re-derived on this driver with the pg-boss schema
|
||||||
|
family as design reference
|
||||||
|
([ADR-005](decisions/005-dependency-ownership.md)); semantics depth
|
||||||
|
is [queues.md](queues.md)'s work (OQ-05).
|
||||||
|
|
||||||
|
## Connection architecture
|
||||||
|
|
||||||
|
- **Pool** (deadpool) — queries, claims, and all non-transactional
|
||||||
|
work. Per-connection statement cache, `RecyclingMethod::Fast`
|
||||||
|
(no `DISCARD ALL` recycling; claim SQL re-prepared implicitly with
|
||||||
|
zero errors at POC scale).
|
||||||
|
- **Listener connection** — one dedicated, **non-pooled** connection
|
||||||
|
per process that listens. Pooled connections cannot carry LISTEN
|
||||||
|
(deadpool#360 — registration succeeds, delivery is impossible; the
|
||||||
|
client-wrapper exposes no notification surface, source-verified and
|
||||||
|
test-pinned as `pooled_listen_registers_but_cannot_deliver`). The
|
||||||
|
listener is therefore a per-process budget line *outside* the pool:
|
||||||
|
`max_size + 1` per LISTEN-ing process
|
||||||
|
([deployment.md](deployment.md)).
|
||||||
|
- **Forwarder** — the listener's loop: `poll_message`
|
||||||
|
fanning out into a bounded broadcast channel (lag surfaced, not
|
||||||
|
silent), re-LISTEN from the channel list after every reconnect
|
||||||
|
(exponential backoff 50 ms → 2 s cap), and the synthetic
|
||||||
|
reconnect-wake on the reserved channel
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
- One listener serves N channels and N subscribers; re-attach is a
|
||||||
|
broadcast re-subscribe (no server round-trips); per-channel
|
||||||
|
connections are never warranted at this scale (POC-verified).
|
||||||
|
|
||||||
|
## Mapping the contract
|
||||||
|
|
||||||
|
| Contract piece | Engine realization |
|
||||||
|
|---|---|
|
||||||
|
| notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers |
|
||||||
|
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) |
|
||||||
|
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback) |
|
||||||
|
| named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; depth in OQ-05's design work) |
|
||||||
|
| scheduler / outbox | pg-boss-family design reference, per [queues.md](queues.md) |
|
||||||
|
| begin_tx | pool checkout + `BEGIN`, returning the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||||
|
| handle ops | straight `.await`s through the held object; commit/rollback returns the object to the pool |
|
||||||
|
|
||||||
|
## Owned failure modes (all test-pinned in POC #2)
|
||||||
|
|
||||||
|
Hand-rolling the forwarder means owning its pitfalls — they are
|
||||||
|
*learned territory*, pinned as passing tests:
|
||||||
|
|
||||||
|
1. **Query-vs-poll starvation deadlock** — the poll loop must be
|
||||||
|
running before the first client query on the listener connection.
|
||||||
|
2. **Client-drop closes the server session** — a long-lived listener
|
||||||
|
keeps its `Client` alive for the listener's lifetime.
|
||||||
|
3. **The no-replay hole** — commit during a connection gap is never
|
||||||
|
re-delivered; recovery = reconnect + synthetic wake + consumer
|
||||||
|
re-read ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
The `!saw_replay` test pins the honesty.
|
||||||
|
4. **Payload boundary** — `pg_notify` ≤ 8000 bytes; client-side
|
||||||
|
checked, typed error before the round-trip. Large payloads ride a
|
||||||
|
table row with the id in the notification (outbox shape).
|
||||||
|
5. **Read-your-writes** — `read committed` default verified in both
|
||||||
|
directions (in-tx and post-commit).
|
||||||
|
|
||||||
|
Constraint also carried: mixed rusqlite+sqlx binaries would need a
|
||||||
|
vendored patch today — excluded by construction in this engine
|
||||||
|
(single driver, [ADR-001](decisions/001-crate-split.md)), noted for
|
||||||
|
the record in [ADR-003](decisions/003-sqlite-driver.md).
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
| ADR | Decision | Summary |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
|
||||||
|
| [004](decisions/004-postgres-driver.md) | Driver | tokio-postgres + deadpool; hand-rolled forwarder; re-derived queues |
|
||||||
|
| [005](decisions/005-dependency-ownership.md) | Ownership | published libs as-is; `postgres-notify` derive-not-adopt |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | LISTEN push, no replay, synthetic reconnect-wake |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Tx seam | direct pooled-object handle, no bridging |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Open questions are tracked in
|
||||||
|
[open-questions.md](open-questions.md). Key
|
||||||
|
questions affecting this document:
|
||||||
|
|
||||||
|
- **OQ-09**: scheduler collapse into queues (shared with
|
||||||
|
[queues.md](queues.md)) (open)
|
||||||
|
- **OQ-05**: queue semantics depth — the pg-boss-family design-input
|
||||||
|
work (open)
|
||||||
|
- **OQ-08**: capability surface (shared with
|
||||||
|
[deployment.md](deployment.md)) (open)
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# SQLite engine
|
||||||
|
|
||||||
|
The `alkstore-sqlite` engine implements
|
||||||
|
[core-contract.md](core-contract.md) on rusqlite + published
|
||||||
|
honker-core. This spec records WHAT the engine is internally (its
|
||||||
|
connection architecture, seam, and wake plumbing) — not code-level
|
||||||
|
HOW. Decisions live in ADRs; per-contract obligations live in the core
|
||||||
|
spec and are not restated here.
|
||||||
|
|
||||||
|
## Identity and posture
|
||||||
|
|
||||||
|
- Single driver: rusqlite (riding
|
||||||
|
[honker-core's pin](decisions/003-sqlite-driver.md)),
|
||||||
|
`bundled-sqlite` for hermetic builds. No `.so` runtime artifacts, no
|
||||||
|
vendored patches
|
||||||
|
([ADR-003](decisions/003-sqlite-driver.md),
|
||||||
|
[ADR-001](decisions/001-crate-split.md)).
|
||||||
|
- Sync machinery, async-facing trait: honker-core is sync (std
|
||||||
|
threads, blocking iterators); the engine bridges at the trait seam
|
||||||
|
per the family-standard posture (alktty REQ-TTY-01 precedent).
|
||||||
|
Every engine call runs in `spawn_blocking`.
|
||||||
|
- Single-host by nature: file-backed, one machine, NFS-two-writers
|
||||||
|
unsupported (honker's honesty posture, inherited). See
|
||||||
|
[deployment.md](deployment.md).
|
||||||
|
|
||||||
|
## Connection architecture
|
||||||
|
|
||||||
|
- **Writer** — one dedicated connection; the only one permitted to
|
||||||
|
write. Serializes all mutations (WAL single-writer, modeled
|
||||||
|
honestly, not fought).
|
||||||
|
- **Readers** — a small pool of read connections for lookups and
|
||||||
|
claim/ack work.
|
||||||
|
- **Watcher** — honker-core's `SharedUpdateWatcher`: a dedicated
|
||||||
|
thread polling `PRAGMA data_version` at the default 1 ms cadence,
|
||||||
|
fanning out to listeners, overtriggering on purpose (waking all
|
||||||
|
subscribers per poll tick, even when several commits coalesced
|
||||||
|
inside one tick — wake is a hint; consumers re-read indexed state,
|
||||||
|
[ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
- Each connection runs honker's bootstrap at open: pragmas
|
||||||
|
(WAL, `synchronous=NORMAL`, busy timeout), `attach_notify`,
|
||||||
|
`attach_honker_functions`, `bootstrap_honker_schema` (the
|
||||||
|
alknet-filesystem POC's wiring shape).
|
||||||
|
|
||||||
|
## Mapping the contract
|
||||||
|
|
||||||
|
| Contract piece | Engine realization |
|
||||||
|
|---|---|
|
||||||
|
| notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) |
|
||||||
|
| streams | honker's stream machinery; explicit offset saves through the tx seam |
|
||||||
|
| queues | honker's queue functions ([ADR-002](decisions/002-feature-scope.md)); semantics depth design in [queues.md](queues.md) |
|
||||||
|
| named locks | honker's lock machinery |
|
||||||
|
| scheduler / outbox | honker's counterparts, per [queues.md](queues.md) |
|
||||||
|
| begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||||
|
| handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) |
|
||||||
|
| commit/rollback | releases the writer slot |
|
||||||
|
|
||||||
|
## Obligations and constraints
|
||||||
|
|
||||||
|
- **A long transaction parks the writer** (the slot lease is the
|
||||||
|
honest model). Contract docs must surface this so consumers budget
|
||||||
|
transactions ([ADR-007](decisions/007-transactional-seam.md) negative
|
||||||
|
consequence).
|
||||||
|
- Watcher failure handling is inherited: on watcher death, every
|
||||||
|
subscriber's receiver closes (`WatcherDeathGuard` behavior) —
|
||||||
|
consumers see the close, never a silent hang
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
- Wake coalescing: bursts inside one poll tick produce one wake;
|
||||||
|
correctness is preserved by the re-read contract
|
||||||
|
(POC-pinned: missed-wake stress with correct post-burst re-reads).
|
||||||
|
- **Deployment note**: honker-core 0.5 pins rusqlite ^0.40.1, whose
|
||||||
|
rustc floor is ≥ 1.99; binaries linking this engine carry that
|
||||||
|
requirement ([deployment.md](deployment.md) matrix).
|
||||||
|
- The optimization path, if seam throughput ever demands it: a
|
||||||
|
dedicated std-thread bridge (one thread owning the writer conn, ops
|
||||||
|
over mpsc — measured ~2× the spawn_blocking shape at p50 in POC #1),
|
||||||
|
or a raised watcher cadence for idle CPU. Neither is the default.
|
||||||
|
|
||||||
|
## What rides on the fork question
|
||||||
|
|
||||||
|
Nearly all of honker-core's surface is consumed (Writer/Readers/
|
||||||
|
SharedUpdateWatcher/attach_*). The [quality read
|
||||||
|
(OQ-06)](decisions/005-dependency-ownership.md) is this engine's only
|
||||||
|
open dependency gate: if it names a defect or an upstream-unwon't
|
||||||
|
change, the fork posture
|
||||||
|
([ADR-005](decisions/005-dependency-ownership.md)) fires and this
|
||||||
|
engine's substrate becomes owned code. Until then, published-library
|
||||||
|
consumption stands.
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
| ADR | Decision | Summary |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
|
||||||
|
| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + honker-core, bridged seam, inherited watcher |
|
||||||
|
| [005](decisions/005-dependency-ownership.md) | Ownership | published honker-core; named fork triggers |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | data_version watcher, coalescing, death-closes-receivers |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Tx seam | writer-slot lease, `BEGIN IMMEDIATE`, `spawn_blocking` round-trips |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Open questions are tracked in
|
||||||
|
[open-questions.md](open-questions.md). Key
|
||||||
|
questions affecting this document:
|
||||||
|
|
||||||
|
- **OQ-06**: honker-core quality read — fork-trigger assessment (open)
|
||||||
|
- **OQ-09**: scheduler collapse into queues (shared with
|
||||||
|
[queues.md](queues.md)) (open)
|
||||||
|
- **OQ-05**: queue semantics depth on honker's machinery (open)
|
||||||
@@ -0,0 +1,217 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# alkstore — Open Questions
|
||||||
|
|
||||||
|
Centralized tracker. IDs `OQ-NN` are stable — never renumber; append.
|
||||||
|
The Phase 0 register's questions (`OQ-ST-01..08` in
|
||||||
|
`docs/research/phase-0.md`) are promoted here faithfully: **OQ-01..08
|
||||||
|
mirror OQ-ST-01..08 one-to-one**, keeping their Phase 0 statuses
|
||||||
|
(a resolved register question stays listed here as resolved, with the
|
||||||
|
ADR that carries its decision). New Phase 1 questions append from
|
||||||
|
OQ-09. Suggested resolution order: **OQ-04 first** (the contract
|
||||||
|
surface everything else hangs off), then OQ-09/OQ-10 (scoped to
|
||||||
|
OQ-04's outcome), then OQ-05/OQ-08 (OQ-05 has an independent design
|
||||||
|
track; OQ-08 depends only on the trait's shape, OQ-04).
|
||||||
|
|
||||||
|
Resolved questions stay listed with their resolution; they are not
|
||||||
|
deleted.
|
||||||
|
|
||||||
|
## Theme: Scope
|
||||||
|
|
||||||
|
### OQ-01: Scope boundary — which honker features are in-scope? *(== OQ-ST-01)*
|
||||||
|
|
||||||
|
- **Origin**: [consumer-inventory.md](../research/consumer-inventory.md),
|
||||||
|
[ADR-002](decisions/002-feature-scope.md)
|
||||||
|
- **Status**: resolved (2026-10-04, Phase 0)
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: Answered by the consumer inventory — scope votes
|
||||||
|
shrink to named rows, not the whole honker feature list. First-class:
|
||||||
|
notify/listen (pinned), streams (operator-authority record). In
|
||||||
|
scope: named locks, queues + outbox (documented), scheduler
|
||||||
|
(documented-thin). Cut-flags: rate limits, result storage. Out:
|
||||||
|
honker's exclusion lines. Decision recorded in
|
||||||
|
[ADR-002](decisions/002-feature-scope.md).
|
||||||
|
- **Cross-references**: OQ-07, OQ-04.
|
||||||
|
|
||||||
|
### OQ-02: Crate scope — one store crate, or reactive-core + engines? *(== OQ-ST-02)*
|
||||||
|
|
||||||
|
- **Origin**: [overview.md](overview.md)
|
||||||
|
- **Status**: resolved (2026-10-04, operator decision; Phase 0)
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: Reactive-core crate + per-engine engine crates; the
|
||||||
|
split isolates the engines' asymmetry of work and keeps engine
|
||||||
|
binaries single-driver (which the libsqlite3-sys collision
|
||||||
|
effectively requires). Decision recorded in
|
||||||
|
[ADR-001](decisions/001-crate-split.md) — including the reasoning
|
||||||
|
that superseded the inventory's single-crate lean.
|
||||||
|
- **Cross-references**: OQ-03, OQ-10.
|
||||||
|
|
||||||
|
### OQ-03: Driver story — sqlx, tokio-postgres, or per-engine drivers? *(== OQ-ST-03)*
|
||||||
|
|
||||||
|
- **Origin**: [overview.md](overview.md)
|
||||||
|
- **Status**: resolved (2026-10-04, POC-backed both engines; Phase 0)
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: Per-engine drivers — rusqlite + honker-core 0.5.0
|
||||||
|
(SQLite; bridged seam), tokio-postgres 0.7.18 + deadpool-postgres
|
||||||
|
0.14.2 (Postgres), under OQ-02's crate split. Decisions recorded in
|
||||||
|
[ADR-003](decisions/003-sqlite-driver.md) and
|
||||||
|
[ADR-004](decisions/004-postgres-driver.md); measurements in
|
||||||
|
`docs/research/poc-sqlite-posture-findings.md` and
|
||||||
|
`docs/research/poc-pg-posture-findings.md`.
|
||||||
|
- **Cross-references**: OQ-02, OQ-05, OQ-06.
|
||||||
|
|
||||||
|
### OQ-07: SQLite-side scope — loadable extension? *(== OQ-ST-07)*
|
||||||
|
|
||||||
|
- **Origin**: [ADR-002](decisions/002-feature-scope.md)
|
||||||
|
- **Status**: resolved (cut-only, 2026-10-04; Phase 0)
|
||||||
|
- **Priority**: low
|
||||||
|
- **Resolution**: The loadable-extension surface is out: every
|
||||||
|
identified consumer is in-process Rust attaching to its own
|
||||||
|
connection (the honker-core `attach_honker_functions` shape). Folded
|
||||||
|
into OQ-01's scope resolution and [ADR-002](decisions/002-feature-scope.md)
|
||||||
|
(explicitly, so the slot's disposition is visible — the Phase 0
|
||||||
|
register carried it as its own entry).
|
||||||
|
- **Cross-references**: OQ-01.
|
||||||
|
|
||||||
|
## Theme: Core contract
|
||||||
|
|
||||||
|
### OQ-04: Contract pinning — the exact trait surface and its semantics *(== OQ-ST-04)*
|
||||||
|
|
||||||
|
*(Retitled in promotion: OQ-ST-04's framing — "the reactive
|
||||||
|
abstraction — what does the unified notify surface look like?" —
|
||||||
|
narrowed to the pinning work its own record already scoped.)*
|
||||||
|
|
||||||
|
- **Origin**: [core-contract.md](core-contract.md),
|
||||||
|
[ADR-006](decisions/006-wake-and-delivery-contract.md),
|
||||||
|
[ADR-007](decisions/007-transactional-seam.md)
|
||||||
|
- **Status**: open (de-risked; both engine sides POC-verified — this is
|
||||||
|
paper work over a complete evidence base)
|
||||||
|
- **Priority**: high
|
||||||
|
- **Resolution**: open. The decisions that bound it are made
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md): opaque-wake + re-read, the delivery-guarantee table, the
|
||||||
|
reserved namespace's existence; [ADR-007](decisions/007-transactional-seam.md): caller-held `TxHandle`
|
||||||
|
with `*_tx` methods). What remains to pin, per the honker-rs surface
|
||||||
|
as starting artifact against the inventory rows:
|
||||||
|
- Which surface parts are contract v1 and which are engine-extension
|
||||||
|
(queue depth, scheduler surface, lock semantics details).
|
||||||
|
- `TxHandle` representation: `dyn` + per-engine downcast (POC
|
||||||
|
sketch, works, two recorded frictions) vs generic/enum handle.
|
||||||
|
- The reserved/meta namespace: exact strings (Postgres
|
||||||
|
reconnect-wake channel like `__listener_reconnected__`), and
|
||||||
|
whether SQLite-side internal names carry an equivalent prefix.
|
||||||
|
- Error taxonomy: what callers can match on per mechanism (e.g.
|
||||||
|
`PayloadTooLarge` is POC-verified pg-side; SQLite's notify has no
|
||||||
|
8000-byte limit — is the error universal?).
|
||||||
|
- Naming/grouping of the honker-rs surface where renames clarify
|
||||||
|
(e.g. what `listen()` returns — a wake receiver? a subscription
|
||||||
|
handle?).
|
||||||
|
- The `Store::open` config shape — what engine-configuration
|
||||||
|
(durability knobs, pool sizing, listener posture) is consumer
|
||||||
|
surface vs engine crate docs.
|
||||||
|
- Verification backlog for properties the POCs pinned on one engine
|
||||||
|
only — e.g. lock TTL/expiry re-acquisition is POC-pinned on
|
||||||
|
Postgres but rests on honker's machinery on SQLite; the contract
|
||||||
|
test suite must pin the SQLite side too.
|
||||||
|
- Delivery-guarantee rows for locks and scheduler (the guarantee
|
||||||
|
table in [ADR-006](decisions/006-wake-and-delivery-contract.md) covers notify/streams/queues; lock-vs-TTL-race
|
||||||
|
and scheduler-tick guarantees need either rows there or an
|
||||||
|
explicit "scheduler is queues" absorption via OQ-09).
|
||||||
|
- **Cross-references**: OQ-01, OQ-10, OQ-09, OQ-08.
|
||||||
|
|
||||||
|
## Theme: Queues and scheduling
|
||||||
|
|
||||||
|
### OQ-05: Queue semantics depth — retry/backoff/dead-letter/sweep design *(== OQ-ST-05)*
|
||||||
|
|
||||||
|
- **Origin**: [queues.md](queues.md)
|
||||||
|
- **Status**: open (posture resolved: re-derived on both engines with
|
||||||
|
the pg-boss schema family as design reference and honker's queue
|
||||||
|
design as the SQLite-side one — ADR-004/ADR-005)
|
||||||
|
- **Priority**: high
|
||||||
|
- **Resolution**: open. The transactional and claim properties (the
|
||||||
|
hard driver-coupled part) are POC-pinned on both engines. Remaining:
|
||||||
|
the semantics-depth design — retry policy shape (attempts, backoff
|
||||||
|
curve), visibility-timeout/renewal mechanics, dead-letter
|
||||||
|
move-vs-flag and retention, sweep/maintenance design
|
||||||
|
(`sweep_expired` cadence and owner), and whether result-storage's
|
||||||
|
cut-flag gets reconsidered as part of this surface (it is
|
||||||
|
queue-adjacent tooling).
|
||||||
|
- **Cross-references**: OQ-09, OQ-06.
|
||||||
|
|
||||||
|
### OQ-09: Is the scheduler a first-class mechanism, or queues + `schedule()`?
|
||||||
|
|
||||||
|
- **Origin**: [queues.md](queues.md) (spin-out of OQ-05's Phase 0
|
||||||
|
framing, where the scheduler's boundary question lived)
|
||||||
|
- **Status**: open
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: open. The inventory found the scheduler need real but
|
||||||
|
thin (the family-wide "who sweeps/renews/reaps" problem), and
|
||||||
|
mechanically scheduler = cron/`@every` enqueueing into named queues
|
||||||
|
+ leader election + a tick. If design confirms the collapse, the
|
||||||
|
contract surface is queues + `store.schedule(cron, queue)` with the
|
||||||
|
scheduler mechanism absorbed; if consumers need
|
||||||
|
inspectable/pausable schedule objects (`add/pause/resume/update/
|
||||||
|
list/remove`), it stays a first-class mechanism (the full honker
|
||||||
|
shape). Decided against the inventory rows, not the whole honker
|
||||||
|
menu.
|
||||||
|
- **Cross-references**: OQ-05, OQ-04.
|
||||||
|
|
||||||
|
## Theme: Engines and dependencies
|
||||||
|
|
||||||
|
### OQ-06: honker-core quality read — does the default posture hold? *(== OQ-ST-06)*
|
||||||
|
|
||||||
|
- **Origin**: [ADR-005](decisions/005-dependency-ownership.md),
|
||||||
|
[engine-sqlite.md](engine-sqlite.md)
|
||||||
|
- **Status**: open
|
||||||
|
- **Priority**: high (fork trigger is a gate on the SQLite engine's
|
||||||
|
dependency posture)
|
||||||
|
- **Resolution**: open. ADR-005 fixed the calculus and the trigger: the
|
||||||
|
Phase 1 quality read of honker-core 0.5's watcher/transactional core
|
||||||
|
(Writer/Readers/SharedUpdateWatcher/attach_*) — looking for defects
|
||||||
|
the POCs wouldn't surface, unsafe assumptions in the watcher
|
||||||
|
failure-handling, and schema-migration brittleness. Outcomes: posture
|
||||||
|
holds (no ADR change), or a fork/patch need is named (fork is normal
|
||||||
|
work per ADR-005).
|
||||||
|
- **Cross-references**: ADR-003, ADR-005, OQ-05 (if the read forces a
|
||||||
|
fork, queue-on-honker-machinery work changes shape).
|
||||||
|
|
||||||
|
## Theme: Deployment and capabilities
|
||||||
|
|
||||||
|
### OQ-08: Where does the honest single-host/multi-host boundary live in the trait surface? *(== OQ-ST-08)*
|
||||||
|
|
||||||
|
- **Origin**: [deployment.md](deployment.md)
|
||||||
|
- **Status**: open
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: open. The pg engine is natively multi-host (POC #2
|
||||||
|
verified — no single-host assumption to remove); SQLite is
|
||||||
|
single-machine by nature (file-backed, NFS-two-writers unsupported —
|
||||||
|
honker's honesty posture, inherited). The unified surface must not
|
||||||
|
pretend SQLite is multi-host. Options: per-engine capability flags
|
||||||
|
(`Store::capabilities()`), a documented deployment matrix only
|
||||||
|
([deployment.md] carries the facts), or compile-time knowledge only
|
||||||
|
(a consumer choosing the SQLite engine knows). Rides OQ-04: the
|
||||||
|
trait's shape constrains where capability differences can surface.
|
||||||
|
- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md).
|
||||||
|
|
||||||
|
### OQ-10: How do engine crates track core-contract version changes?
|
||||||
|
|
||||||
|
- **Origin**: [overview.md](overview.md),
|
||||||
|
[ADR-001](decisions/001-crate-split.md)
|
||||||
|
- **Status**: open
|
||||||
|
- **Priority**: medium
|
||||||
|
- **Resolution**: open. The core crate's trait surface is a contract
|
||||||
|
the engine crates must track ([ADR-001](decisions/001-crate-split.md) negative consequence). What
|
||||||
|
is the versioning/sync discipline — semver-bump-only-when-
|
||||||
|
contract-changes, engines pin core ranges, a contract-compatibility
|
||||||
|
test suite the engines run against the core's trait definitions?
|
||||||
|
What happens to a released engine crate when core makes a contract
|
||||||
|
breaking change?
|
||||||
|
- **Cross-references**: OQ-04 (the contract being versioned), OQ-02.
|
||||||
|
|
||||||
|
## Deferred / Blocked
|
||||||
|
|
||||||
|
None currently. Every open OQ above is actionable Phase 1 architecture
|
||||||
|
work (contract pinning, design, quality read) with its evidence base
|
||||||
|
complete — no external arrivals are being waited on.
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# alkstore — Overview
|
||||||
|
|
||||||
|
One reactive store interface over SQLite and Postgres: durable
|
||||||
|
notify/subscribe signals, streams with per-consumer offsets, durable
|
||||||
|
queues with the transactional enqueue property, named locks, a
|
||||||
|
scheduler, and an outbox helper — with each engine using its own
|
||||||
|
native machinery underneath (SQLite: honker's watcher design;
|
||||||
|
Postgres: `pg_notify`/LISTEN and the pg-boss schema family).
|
||||||
|
|
||||||
|
## Why this crate exists
|
||||||
|
|
||||||
|
The alk* ecosystem's current shape is a repository pattern with a
|
||||||
|
default in-memory adapter, cache-invalidation wiring in hot paths, and
|
||||||
|
per-project approximations where a durable reactive substrate was
|
||||||
|
needed. alkblobs paused partly on that fuzziness. alkstore is the
|
||||||
|
substrate, made once: the answer to "how does a change in the database
|
||||||
|
become visible to other processes/connections?" that downstream stores
|
||||||
|
don't re-derive.
|
||||||
|
|
||||||
|
## Crate family
|
||||||
|
|
||||||
|
Per [ADR-001](decisions/001-crate-split.md):
|
||||||
|
|
||||||
|
| Crate | Contents | Driver dependencies |
|
||||||
|
|---|---|---|
|
||||||
|
| `alkstore` (core) | trait surface, types, error model, capability flags | none |
|
||||||
|
| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, honker-core |
|
||||||
|
| `alkstore-postgres` | Postgres engine ([ADR-004](decisions/004-postgres-driver.md)) | tokio-postgres, deadpool-postgres |
|
||||||
|
| (mem engine, optional) | test convenience, decided at implementation ([ADR-001](decisions/001-crate-split.md)) | none |
|
||||||
|
|
||||||
|
Downstream consumers depend on core + exactly one engine. Family
|
||||||
|
standards apply throughout: tokio async runtime, `thiserror` errors, no
|
||||||
|
panics in library code, no `unwrap()`/`expect()` outside tests,
|
||||||
|
lean base crate.
|
||||||
|
|
||||||
|
## Feature surface
|
||||||
|
|
||||||
|
Per [ADR-002](decisions/002-feature-scope.md):
|
||||||
|
|
||||||
|
- **First-class**: notify/listen; streams.
|
||||||
|
- **In scope**: named locks; queues + outbox helper; scheduler.
|
||||||
|
- **Cut-flag**: rate limits; result storage.
|
||||||
|
- **Out**: loadable-extension surface; DAGs/task
|
||||||
|
chains/chords/multi-writer replication/cross-machine locking.
|
||||||
|
|
||||||
|
## Document map
|
||||||
|
|
||||||
|
| Doc | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| [core-contract.md](core-contract.md) | The unified trait surface: mechanisms, delivery guarantees, tx seam, naming |
|
||||||
|
| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto honker-core/rusqlite |
|
||||||
|
| [engine-postgres.md](engine-postgres.md) | Postgres engine: mapping the contract onto tokio-postgres/LISTEN |
|
||||||
|
| [queues.md](queues.md) | Queue/scheduler/outbox semantics depth (OQ-05/OQ-09) |
|
||||||
|
| [deployment.md](deployment.md) | Host capabilities, connection budgets, deployment matrix (OQ-08) |
|
||||||
|
| [open-questions.md](open-questions.md) | OQ-01..NN tracker |
|
||||||
|
| [decisions/](decisions/) | ADRs |
|
||||||
|
|
||||||
|
## What is decided (ADR index)
|
||||||
|
|
||||||
|
| ADR | Decision | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| [001](decisions/001-crate-split.md) | Reactive-core + per-engine crates | Accepted |
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope (inventory-confirmed) | Accepted |
|
||||||
|
| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core, bridged seam | Accepted |
|
||||||
|
| [004](decisions/004-postgres-driver.md) | Postgres: tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
|
||||||
|
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Opaque wake + re-read; notify-vs-streams guarantee split | Accepted |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Caller-held `TxHandle` with `*_tx` methods | Accepted |
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- Not a database abstraction/ORM: the store covers the reactive
|
||||||
|
coordination surface, not general row storage (consumers keep their
|
||||||
|
own engines/schema for business data, sharing the connection when
|
||||||
|
appropriate — the outbox pattern).
|
||||||
|
- Not a network service: no transport; any networked ops surface rides
|
||||||
|
alkcall and is a separate future decision (store layer stays
|
||||||
|
substrate-free, the alkblobs store-layer precedent).
|
||||||
|
- Not multi-machine on SQLite: single-host honesty is inherited
|
||||||
|
([deployment.md](deployment.md)).
|
||||||
|
|
||||||
|
## Evidence base
|
||||||
|
|
||||||
|
Phase 0 (`docs/research/phase-0.md`) is complete: two POCs
|
||||||
|
(`poc-sqlite-posture-findings.md`, `poc-pg-posture-findings.md`) passed
|
||||||
|
all gate conditions; the consumer inventory
|
||||||
|
(`consumer-inventory.md`) graded every feature row. The Phase 1
|
||||||
|
architecture work runs over that complete evidence base; open Phase 1
|
||||||
|
work is tracked in [open-questions.md](open-questions.md).
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
---
|
||||||
|
status: draft
|
||||||
|
last_updated: 2026-10-04
|
||||||
|
---
|
||||||
|
|
||||||
|
# Queues, scheduler, outbox — semantics depth
|
||||||
|
|
||||||
|
The queue family's *posture* is decided (re-derived on both engines;
|
||||||
|
[ADR-004](decisions/004-postgres-driver.md) and
|
||||||
|
[ADR-005](decisions/005-dependency-ownership.md)) and the hard
|
||||||
|
driver-coupled properties are POC-pinned (transactional enqueue,
|
||||||
|
exactly-once claim under concurrency). What is *not* yet designed is
|
||||||
|
the semantics depth — this document's subject, and [OQ-05]'s home. It
|
||||||
|
states the design space, the decided constraints any answer must fit,
|
||||||
|
and the reference material; actual semantics decisions become ADRs
|
||||||
|
when made.
|
||||||
|
|
||||||
|
## Decided constraints (inherited, not re-opened)
|
||||||
|
|
||||||
|
- **Mechanisms**: durable at-least-once queues; the outbox is a helper
|
||||||
|
over queues; scheduler is cron/`@every` enqueueing into named
|
||||||
|
queues ([ADR-002](decisions/002-feature-scope.md)) — pending
|
||||||
|
OQ-09's collapse decision.
|
||||||
|
- **Transactional enqueue**: commit-atomic with the caller's business
|
||||||
|
write on both engines ([ADR-007](decisions/007-transactional-seam.md));
|
||||||
|
rollback drops the job row with no ghosts.
|
||||||
|
- **Exactly-once claim**: `FOR UPDATE SKIP LOCKED` (pg, POC-pinned
|
||||||
|
under 4×4 concurrency) / honker's machinery (SQLite). At-least-once
|
||||||
|
*work*; exactly-once *processing* needs the ack + visibility model.
|
||||||
|
- **Wake-driven consumption**: the pg engine's default is LISTEN-driven
|
||||||
|
claim with a re-poll safety net; SQLite's is watcher wake with
|
||||||
|
per-subscription fanout ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||||
|
- **Job options** (the honker-rs surface, the contract starting
|
||||||
|
artifact): `delay, priority, max_attempts, expires` at enqueue;
|
||||||
|
`ack / retry / fail / heartbeat` on the job handle.
|
||||||
|
- **Cut-flag context**: result storage is
|
||||||
|
[ADR-002](decisions/002-feature-scope.md)'s cut-flag row; if
|
||||||
|
OQ-05's design shows queue consumers provably need
|
||||||
|
result-query-by-id, that is the channel to revisit it — with a
|
||||||
|
consumer-inventory row, not silent inclusion.
|
||||||
|
|
||||||
|
## Design space (what OQ-05 must pin)
|
||||||
|
|
||||||
|
### Retry / backoff
|
||||||
|
|
||||||
|
- Trigger model: attempts counted from claim-completion; a job
|
||||||
|
re-appears after visibility timeout OR explicit `retry`. Does
|
||||||
|
`max_attempts` exhaustion move the job to dead-letter, flag it, or
|
||||||
|
drop it?
|
||||||
|
- Backoff curve between attempts: honker and the pg-boss family have
|
||||||
|
their own shapes (fixed, exponential); which does the contract
|
||||||
|
pin, what is configurable, what's the default?
|
||||||
|
- `heartbeat` semantics: renewal vs progress signal (or both) — and
|
||||||
|
what a missed heartbeat means (visibility re-expiry? nothing?).
|
||||||
|
|
||||||
|
### Visibility timeouts
|
||||||
|
|
||||||
|
- Where the renewal lives: explicit (`heartbeat`) only, or implicit
|
||||||
|
renewal per op on the job handle? Honker's design is the SQLite-side
|
||||||
|
answer; pg-boss's is the reference. One answer, contract-pinned.
|
||||||
|
- Interaction with long business transactions (claims inside
|
||||||
|
caller txs, [ADR-007](decisions/007-transactional-seam.md)).
|
||||||
|
|
||||||
|
### Dead-letter
|
||||||
|
|
||||||
|
- Move-to-table (honker's `_honker_dead`, pg-boss's dead-letter
|
||||||
|
queue) vs flag-in-place. Inspection/requeue surface shape.
|
||||||
|
- Retention: does a dead-lettered job expire (an `expires`-driven
|
||||||
|
sweep — an OQ-05 sub-question) or live until explicitly cleared?
|
||||||
|
|
||||||
|
### Sweep / maintenance
|
||||||
|
|
||||||
|
- `sweep_expired` is on the queue's surface; cadence ownership is
|
||||||
|
the open question. Honker's scheduler machinery (leader-elected,
|
||||||
|
missed-boundary catch-up) can run it; a consumer's own scheduler row
|
||||||
|
can too; the engine could ship a default. Decision shaped by OQ-09's
|
||||||
|
scheduler collapse — same machinery either way.
|
||||||
|
- Multi-process sweep safety on SQLite vs Postgres: named-lock
|
||||||
|
coordination (the alkblobs fleet-sweeper pattern,
|
||||||
|
[ADR-002](decisions/002-feature-scope.md)) vs native advisory locks.
|
||||||
|
|
||||||
|
### Scheduler surface
|
||||||
|
|
||||||
|
- OQ-09's question, stated crisply: is scheduler a first-class
|
||||||
|
mechanism (own handle, methods, inspection) or queues + a
|
||||||
|
`schedule(cron_expression, queue_name, payload)` call? The
|
||||||
|
inventory's evidence is thin (documented-thin); a v1 that is queues
|
||||||
|
+ a `schedule` call with leader-election behavior documented is the
|
||||||
|
smaller contract; if a consumer needs inspectable/pausable schedule
|
||||||
|
*objects* (`add/pause/resume/update/list/remove`), that's the
|
||||||
|
full honker shape. Decide against the inventory rows, not against
|
||||||
|
the whole honker menu.
|
||||||
|
|
||||||
|
### Namespaces / schema layout (pg side)
|
||||||
|
|
||||||
|
- Schema-scoped DDL (the pg-boss design) vs shared-schema table
|
||||||
|
naming. Rides the reserved-namespace contract
|
||||||
|
([ADR-006](decisions/006-wake-and-delivery-contract.md), OQ-04):
|
||||||
|
queue/stream/lock tables and the reserved channel prefix
|
||||||
|
must be collision-proof against consumer tables in the same
|
||||||
|
database (the alkblobs co-tenant-tables precedent — ADR-008 in
|
||||||
|
`/workspace/@alkdev/alkblobs/docs/architecture/decisions/`).
|
||||||
|
|
||||||
|
## Reference material
|
||||||
|
|
||||||
|
- **honker's queue design** — the SQLite-side incumbent
|
||||||
|
(`/workspace/honker`, its honker-core machinery; the engine rides it
|
||||||
|
directly, so the SQLite side's depth is largely "inherit + pin").
|
||||||
|
- **pg-boss family** — `/workspace/pgboss-rs` @ 98f7d9e (queue states,
|
||||||
|
maintenance/dead-letter behavior) and the node original (pg-boss, the
|
||||||
|
upstream of record — compare semantics the port may have dropped).
|
||||||
|
- **POC ground** — `poc-pg-posture-findings.md` (the minimal queue
|
||||||
|
table + SKIP LOCKED claim measured; the property suite green);
|
||||||
|
`poc-sqlite-posture-findings.md` (the SQLite twin).
|
||||||
|
- **Consumers** — alkfs OQ-FS-14 (sync outbox), alkblobs
|
||||||
|
ops-surface/gc docs (maintenance cadences), the family-wide
|
||||||
|
"who sweeps" punt (inventory §scheduler).
|
||||||
|
|
||||||
|
## Design Decisions
|
||||||
|
|
||||||
|
| ADR | Decision | Summary |
|
||||||
|
|---|---|---|
|
||||||
|
| [002](decisions/002-feature-scope.md) | Feature scope | queues + outbox in; result storage cut-flag |
|
||||||
|
| [004](decisions/004-postgres-driver.md) | pg driver | queue machinery re-derived, pg-boss as reference |
|
||||||
|
| [005](decisions/005-dependency-ownership.md) | Ownership | design-reference postures for the queue family |
|
||||||
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | wake-driven consumption, guarantee table |
|
||||||
|
| [007](decisions/007-transactional-seam.md) | Tx seam | enqueue_tx commit-atomicity |
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
Open questions are tracked in
|
||||||
|
[open-questions.md](open-questions.md). Key
|
||||||
|
questions affecting this document:
|
||||||
|
|
||||||
|
- **OQ-05**: the semantics-depth pinning this document frames (open,
|
||||||
|
high priority)
|
||||||
|
- **OQ-09**: scheduler as first-class mechanism vs queues + schedule
|
||||||
|
(open)
|
||||||
@@ -210,7 +210,11 @@ none of them yet.
|
|||||||
features as contract candidates; the ambiguity found in the
|
features as contract candidates; the ambiguity found in the
|
||||||
inventory (queues vs scheduler boundary; streams' absent consumer;
|
inventory (queues vs scheduler boundary; streams' absent consumer;
|
||||||
rate limits' alternative mechanism) is input to which parts of the
|
rate limits' alternative mechanism) is input to which parts of the
|
||||||
honker-rs surface become contract versus cut.
|
honker-rs surface become contract versus cut. *(Correction
|
||||||
|
2026-10-04, promotion pass: the scheduler-collapse question was
|
||||||
|
OQ-ST-05/06 territory in this line's original framing and is now
|
||||||
|
tracked as OQ-09 in `docs/architecture/open-questions.md` — not
|
||||||
|
OQ-ST-04/02 as this line previously garbled.)*
|
||||||
|
|
||||||
## The consumer list, honestly bounded
|
## The consumer list, honestly bounded
|
||||||
|
|
||||||
|
|||||||
+22
-13
@@ -1,10 +1,11 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-04 (Phase 0 research complete. Both POCs ran, passed,
|
last_updated: 2026-10-04 (Phase 0 complete and the register promoted:
|
||||||
and are folded into the register; the register itself was restructured
|
docs/architecture/ opened with ADR-001..007 carrying the resolved
|
||||||
after the findings review for consistency. OQ-ST-01/02/03/07 resolved;
|
decisions and the OQ tracker mirroring OQ-ST-01..08 1:1. Phase 0
|
||||||
OQ-ST-04/05/06 de-risked with only contract/design work remaining;
|
research itself is closed; remaining work is in docs/architecture/
|
||||||
OQ-ST-08 open. Convergence recorded in §Convergence. Next: Phase 1.)
|
open-questions.md. One erratum fixed in OQ-ST-04's tx-friction line —
|
||||||
|
thread-affinity is SQLite-side, previously garbled as "pg-side".)
|
||||||
---
|
---
|
||||||
|
|
||||||
# alkstore — Phase 0 (Exploration)
|
# alkstore — Phase 0 (Exploration)
|
||||||
@@ -370,7 +371,10 @@ convention in the contract.
|
|||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
Register in `docs/research/phase-0.md`; IDs OQ-ST-NN (stable, append
|
Register in `docs/research/phase-0.md`; IDs OQ-ST-NN (stable, append
|
||||||
only). Promotion target: Phase 1 `docs/architecture/open-questions.md`.
|
only). Promotion target: Phase 1 `docs/architecture/open-questions.md`
|
||||||
|
— **promoted 2026-10-04**: OQ-ST-01..08 mirror 1:1 to OQ-01..08 there
|
||||||
|
(their statuses and resolutions carried; resolved decisions carried
|
||||||
|
into ADRs 001–007), and new Phase 1 questions append from OQ-09.
|
||||||
Status conventions: `resolved` (evidence recorded here); `open —
|
Status conventions: `resolved` (evidence recorded here); `open —
|
||||||
<work-type>` where the work-type names the remaining work and the
|
<work-type>` where the work-type names the remaining work and the
|
||||||
entry is de-risked (the remainder is Phase 1 architecture work, not
|
entry is de-risked (the remainder is Phase 1 architecture work, not
|
||||||
@@ -591,11 +595,14 @@ Evidence now in hand (both POCs, 2026-10-04):
|
|||||||
caller-held tx handle (`*_tx` methods on the handle); pg's instance
|
caller-held tx handle (`*_tx` methods on the handle); pg's instance
|
||||||
is async-native (tokio-postgres Client is Send+Sync — the handle
|
is async-native (tokio-postgres Client is Send+Sync — the handle
|
||||||
holds the pooled connection directly, no spawn_blocking), SQLite's
|
holds the pooled connection directly, no spawn_blocking), SQLite's
|
||||||
is a bridged writer-slot lease. The core-crate `TxHandle` trait
|
is a bridged writer-slot lease. The core-crate `TxHandle` trait
|
||||||
from POC #1's sketch stands unchanged; the per-engine difference is
|
from POC #1's sketch stands unchanged; the per-engine difference is
|
||||||
bridging mechanism, not trait shape. Phase 1 starts from that shape
|
bridging mechanism, not trait shape. Phase 1 starts from that shape
|
||||||
plus its two recorded frictions (the `as_any_mut` downcast and the
|
plus its two recorded frictions (the `as_any_mut` downcast and the
|
||||||
thread-affinity of rusqlite tx ops — the latter pg-side only).
|
thread-affinity of rusqlite tx ops — the latter SQLite-side only;
|
||||||
|
erratum 2026-10-04, this line previously said "pg-side only," which
|
||||||
|
garbled POC #2's finding that the affinity friction does not carry
|
||||||
|
over to pg).
|
||||||
|
|
||||||
### OQ-ST-05: Queue semantics — adopt, fork, or re-derive?
|
### OQ-ST-05: Queue semantics — adopt, fork, or re-derive?
|
||||||
|
|
||||||
@@ -743,8 +750,10 @@ The expected sequence, with what actually happened:
|
|||||||
with the POCs; per-subsystem votes recorded at the OQs. Remaining:
|
with the POCs; per-subsystem votes recorded at the OQs. Remaining:
|
||||||
the semantics-depth design inputs (OQ-ST-05) and the Phase-1
|
the semantics-depth design inputs (OQ-ST-05) and the Phase-1
|
||||||
quality read (OQ-ST-06's fork trigger).
|
quality read (OQ-ST-06's fork trigger).
|
||||||
5. **Converge** — done (§Convergence). Phase 1 opens with the ADR
|
5. **Converge** — done (§Convergence). Phase 1 opened 2026-10-04:
|
||||||
backlog this register becomes.
|
`docs/architecture/` now exists (README index, seven ADRs
|
||||||
|
001–007 carrying this register's resolved decisions, spec docs,
|
||||||
|
and the promoted open-questions tracker).
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user