ADR-024 — the mem engine adopted as the family's third engine (wave 6; release readiness renumbered to wave 7)
Row-first gate: consumer-inventory.md gains an engine-tier section — the mem-engine need recorded operator-authority, dated 2026-10-10 (wasm32 sandboxing across the alk protocol family + downstream ffi/napi/python-adapter economics; rusqlite/tokio-postgres structurally out on wasm, so mem is the only triad member the sandbox can carry). ADR-024 (decisions/024-mem-engine.md) pins the decision set: (1) separate `alkstore-mem` crate — discharges and amends ADR-001 §4 (superseded in part: full contract-v1 engine, not an "implementation convenience"; annotation on ADR-001 §4 + Status, class-1 window still open); (2) honest-ephemeral posture — no durability, single-process, ephemeral by design, never fleet-valid (ADR-010 §3's shared-pool predicate fails structurally), riding ADR-016's carriers unchanged (identity + docs + matrix); (3) full contract v1 tier with the asymmetry classification — `PayloadTooLarge` never produced (SQLite arm), no reconnection concept (reconnect-wake/watcher rows absent from mem's per-engine column), receiver close at engine drop, wake coalescing N/A; (4) `MemStore::new()` engine-native constructor — the one documented exception to the open-prose, core-contract.md Store concept amended; shared-instance (two opens → one engine) rejected with the ADR-016 §4-style re-entry gate (consumer-inventory row), register stays closed; (5) wasm32 — compile-clean for wasm32-unknown-unknown plus the suite's mem column green on host under a current-thread runtime flavor are wave-6 acceptance items; on-target suite execution scoped out with a named collapse condition (wasm-bindgen-test adapter crate) — a scope decision, not a pending-client hedge; (6) mem is not a core dev-dependency — core's doc examples stay mock-based, ADR-001 §4's doctest posture declines. engine-mem.md (draft): the in-process mapping — guarded per-mechanism state (no pool, no schema bootstrap, no forwarder, no spawn_blocking seam), the tx overlay (staged *_tx effects merged atomically at commit; read-your-own-writes easy, drop-=-rollback trivial; wake-at-commit structural — the seam pg needed pg-fix-tx-wake for, mem pins from birth via the existing tx-commit-atomicity rows), ADR-012 §2 third owner of the curve/opts/@every arithmetic (three-way equivalence, ADR-022's StoreFactory reused verbatim — born pinned). deployment.md: host-semantics row (single-process, ephemeral, never fleet-valid, wasm32-clean), mem connection-budget subsection (none — nothing to budget), durability-knobs row (the honest row is the absence), toolchain row (wasm32 subset + current-thread-clean, acceptance-cited). README/overview: engine-mem.md row, ADR-024 row, family table entry (alkstore-mem, no driver deps), current-state updated to the implementation phase with engine specs' stable statuses reflected. implementation.md: wave table gains wave 6 (mem engine, depends waves 1 + 5) and renumbers release readiness to wave 7 (fuzzing decision — the one remaining decided-at-implementation deferral — rides wave 7); wave-6 section records the born-pinned posture, acceptance items, the one suite-harness pre-task (past_stamp_sleep blocking sleep → async sleep, review 003's properties.rs:1683 note), and the window riders (review 003 Finding 1: the pg transient-fault seam, MEDIUM, rides wave 6; Findings 2–3 follow their park into wave 7's pre-release); wave-6 review gate defined; Decided points updated (mem discharged, fuzzing renumbered); stale wave-6 references swept across review docs and the landed tasks' notes (release-readiness items renumbered or annotated). Docs-only change; verified by cross-reference audit (all ADR/OQ/anchor references resolve, residual wave-6 mentions are correct under the new numbering or carry dated annotations).
This commit is contained in:
glm-5.3-flash committed
1 parent
dbb17068cc
commit
060f262e62
17 files changed
+608
-53
No files matched your search
+19
-11
@@ -1,23 +1,29 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-08 (ADR-023: fourth review round resolved — waves-1–2 general-review findings)
|
||||
last_updated: 2026-10-10 (ADR-024: the mem engine adopted as a third
|
||||
engine — engine-mem.md added, ADR table row, index updated for the
|
||||
implementation phase)
|
||||
---
|
||||
|
||||
# alkstore — Architecture
|
||||
|
||||
Architecture documentation for the alkstore project: one reactive
|
||||
store interface (notify, streams, queues, locks, scheduler, outbox)
|
||||
over SQLite and Postgres, with each engine native underneath
|
||||
(see [overview.md](overview.md)).
|
||||
over SQLite, Postgres, and the in-process mem engine, with each
|
||||
engine native underneath (see [overview.md](overview.md)).
|
||||
|
||||
## Current State
|
||||
|
||||
**Phase 1 (Architecture) — in progress.** Phase 0 is complete
|
||||
(`docs/research/phase-0.md`): both POCs ran and passed, the scope
|
||||
inventory is confirmed, and the crate split, drivers, and ownership
|
||||
postures are decided. This directory carries the architecture spec
|
||||
build-out over that evidence base; all spec documents are `draft`
|
||||
pending architecture review and OQ resolution.
|
||||
**Phase 1 (Implementation) — waves 1–5 implemented and reviewed.**
|
||||
Phase 0 is complete (`docs/research/phase-0.md`): both POCs ran and
|
||||
passed, the scope inventory is confirmed, and the crate split,
|
||||
drivers, and ownership postures are decided. All open questions are
|
||||
resolved (through ADR-023), and the engine specs flipped `stable` at
|
||||
the wave-5 review gate. Waves 1–5 (core, substrate fork, SQLite
|
||||
engine, Postgres engine, contract suite) are implemented and reviewed
|
||||
per `docs/plans/implementation.md`. The mem engine was adopted as a
|
||||
third engine by [ADR-024](decisions/024-mem-engine.md) (2026-10-10,
|
||||
wave 6 — release readiness renumbered to wave 7).
|
||||
|
||||
## Architecture Documents
|
||||
|
||||
@@ -25,8 +31,9 @@ pending architecture review and OQ resolution.
|
||||
|---|---|---|---|
|
||||
| [overview.md](overview.md) | draft | Crate family, feature surface, non-goals, evidence base | — |
|
||||
| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-10 (resolved) |
|
||||
| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 (resolved), OQ-13 (resolved) |
|
||||
| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-08 (resolved), OQ-12 (resolved), OQ-13 (resolved) |
|
||||
| [engine-sqlite.md](engine-sqlite.md) | stable | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 (resolved), OQ-13 (resolved) |
|
||||
| [engine-postgres.md](engine-postgres.md) | stable | Postgres engine: tokio-postgres/LISTEN mapping | OQ-08 (resolved), OQ-12 (resolved), OQ-13 (resolved) |
|
||||
| [engine-mem.md](engine-mem.md) | draft | Mem engine: in-process ephemeral mapping of the contract ([ADR-024](decisions/024-mem-engine.md)) | — |
|
||||
| [queues.md](queues.md) | draft | Queue/scheduler/outbox semantics depth (resolved: ADR-009/ADR-010) | OQ-05 (resolved), OQ-09 (resolved), OQ-06 (resolved) |
|
||||
| [deployment.md](deployment.md) | draft | Host semantics, connection budgets, knobs, matrix | OQ-08 (resolved) |
|
||||
| [open-questions.md](open-questions.md) | draft | OQ tracker (promoted from OQ-ST register) | — |
|
||||
@@ -58,6 +65,7 @@ pending architecture review and OQ resolution.
|
||||
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round — tx-read methods on `TxHandle`, `Job.claimed_at`, `schedule()` queue-argument validation, drop = rollback, receiver close/error arms | Accepted |
|
||||
| [022](decisions/022-contract-suite-layout.md) | Contract-suite layout — shared internal suite crate, properties parameterized over a store factory (discharges ADR-017 §4.2's deferral) | Accepted |
|
||||
| [023](decisions/023-fourth-review-round.md) | Fourth review round — `encode_payload` typed (`Codec`), numeric-argument domains by kind (extents clamp empty, durations reject, boundaries total), plain-path SQLite open (URI flag dropped), watcher cadence 1 ms default + `SqliteOpts` knob | Accepted |
|
||||
| [024](decisions/024-mem-engine.md) | The mem engine — a third engine (`alkstore-mem`), full contract v1, honest-ephemeral posture (collapses + amends ADR-001 §4's deferral) | Accepted |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-08 (ADR-023 — fourth review round: numeric-argument domains pinned by kind, encode_payload typed, plain-path SQLite open, watcher cadence posture)
|
||||
last_updated: 2026-10-10 (ADR-024 — the mem engine joins the family;
|
||||
the Store concept prose admits the engine-native constructor
|
||||
exception)
|
||||
---
|
||||
|
||||
# Core contract
|
||||
@@ -37,7 +39,11 @@ descriptor; the
|
||||
- **Store** — what a consumer opens from a connection string or file
|
||||
path ([ADR-001](decisions/001-crate-split.md)): one handle, the engine
|
||||
behind it chosen at open time. Consumer code never branches on
|
||||
engine type (guiding principle 4).
|
||||
engine type (guiding principle 4). (The mem engine is the one
|
||||
documented exception to the open-prose: its constructor is
|
||||
engine-native — `MemStore::new()`, no connection string — per
|
||||
[ADR-024](decisions/024-mem-engine.md) §4; the engine choice
|
||||
remains a dependency-graph fact made before any `Store` exists.)
|
||||
- **Mechanism** — one of the surface's coordination families: notify,
|
||||
streams, queues (+outbox), locks, scheduler. Delivery guarantees are
|
||||
pinned per mechanism in [ADR-006](decisions/006-wake-and-delivery-contract.md)'s
|
||||
|
||||
@@ -3,7 +3,10 @@
|
||||
## Status
|
||||
|
||||
Accepted *(capability-flags content annotated 2026-10-06 by
|
||||
[ADR-016](016-deployment-honesty.md))*
|
||||
[ADR-016](016-deployment-honesty.md); §4 superseded in part
|
||||
2026-10-10 by [ADR-024](024-mem-engine.md) — the mem engine exists as
|
||||
the separate `alkstore-mem` crate, a full contract-v1 implementation,
|
||||
not an implementation convenience)*
|
||||
|
||||
## Context
|
||||
|
||||
@@ -56,6 +59,12 @@ The project ships as a **family of crates**:
|
||||
a third implementation for tests and doctests, if the test story
|
||||
turns out to want it. Treated as an implementation convenience, not
|
||||
a contract artifact — decided at implementation time, not now.
|
||||
*(Superseded in part 2026-10-10 by
|
||||
[ADR-024](024-mem-engine.md): the mem engine exists as the
|
||||
separate `alkstore-mem` crate — full contract v1, no durability,
|
||||
single-process, ephemeral by design, never fleet-valid, wasm32
|
||||
compile-clean by acceptance. Not a test convenience, and not a
|
||||
core dev-dependency: core's doc examples stay mock-based.)*
|
||||
|
||||
Downstream consumers depend on `alkstore` plus exactly one engine
|
||||
crate. A consumer binary links at most one driver.
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
# ADR-024: The mem engine — a third engine (`alkstore-mem`), full contract v1, honest-ephemeral
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (2026-10-10, Phase 1 — [ADR-001](001-crate-split.md) §4's
|
||||
implementation-time deferral collapses here, and this ADR *amends* that
|
||||
§4's framing rather than merely discharging it: the mem engine is a
|
||||
full contract-v1 implementation, not the "implementation convenience,
|
||||
not a contract artifact" §4 originally imagined. The class-1
|
||||
amend-in-place window ([ADR-017](017-contract-versioning.md) §2) is
|
||||
still open — no versioned artifact exists — so that amendment is
|
||||
pre-implementation. The row-first gate is satisfied: the
|
||||
consumer-inventory engine-tier row (operator-authority, 2026-10-10)
|
||||
precedes this decision.)
|
||||
|
||||
## Context
|
||||
|
||||
ADR-001 §4 allowed "a mem-shaped engine (in-tree or separate test
|
||||
crate) … if the test story turns out to want it," deferred to
|
||||
implementation time. The plan recorded the deferral as a wave-6
|
||||
decision point ("or earlier if the test story demands the mem engine
|
||||
sooner"). The wave-5 review gate closed with the contract suite green
|
||||
on both engines, and two things now make the mem engine concrete:
|
||||
|
||||
- **The consumer pause + the wasm motivation.** The paused consumer
|
||||
work that surfaced the need is not a test story but a consumer
|
||||
story: the alk protocol crates (alkcall, alktty, alktunnels,
|
||||
alksocks) are wasm-compatible at the protocol level by design, and
|
||||
sandboxing — browser or otherwise — plus downstream
|
||||
ffi/napi/python-adapter economics are the operator's recorded need
|
||||
(the consumer-inventory engine-tier row, 2026-10-10). On those
|
||||
paths the mem engine is not a hedge: rusqlite and tokio-postgres
|
||||
are structurally out on wasm, so it is the *only* engine the
|
||||
sandbox can carry.
|
||||
- **The contract suite changed the economics.** A third engine used
|
||||
to mean re-deriving a verification story; ADR-022's shared
|
||||
`StoreFactory` suite means a third engine is born pinned — wiring a
|
||||
third factory column onto already-authored, version-stamped rows.
|
||||
And the class-1 amendment window (ADR-017 §2) is still open: any
|
||||
contract wobble a third full implementation exposes is
|
||||
amend-in-place now, a migration after first release.
|
||||
|
||||
The scale is honest to state up front: this is a *third full engine*,
|
||||
not alkblobs' mem tier (a `BTreeMap` behind a handful of CAS
|
||||
methods). It implements the queue state machine (visibility/reclaim/
|
||||
dead-letter/sweep), the scheduler (boundaries, catch-up, leadership),
|
||||
the tx seam, streams with per-consumer offsets and trim, TTL locks,
|
||||
and wakes. What it does *not* carry is the machinery that made the
|
||||
Postgres wave build-heavy: no pooling, no schema bootstrap, no
|
||||
forwarder, no `spawn_blocking` seam, no reconnection concept.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Crate placement — a separate `alkstore-mem` crate
|
||||
|
||||
The mem engine is a **separate workspace member**, `alkstore-mem`,
|
||||
implementing the core traits. Dependency surface: `alkstore` plus
|
||||
tokio (the wasm-supported subset — sync and time), no driver
|
||||
dependencies; payload encoding rides core's `encode_payload`
|
||||
([ADR-020](020-enqueue-opt-semantics-and-bridges.md) §4). This
|
||||
discharges ADR-001 §4's "in-tree or separate crate" fork as
|
||||
**separate**, for the reason that ADR stated in reverse for the other
|
||||
engines: engine crates are additive by construction (ADR-001 §
|
||||
Consequences), and a mem engine in core would contradict ADR-001 §1's
|
||||
contents list (core is the contract artifact and nothing else).
|
||||
[ADR-013](013-fold-substrate-into-sqlite.md)'s "no fourth crate"
|
||||
rationale does not apply — that ruled the *substrate's* placement
|
||||
(fork lineage, versioning surface, link collisions); none of those
|
||||
facts attach to a greenfield in-memory implementation.
|
||||
|
||||
### 2. Maturity posture — honest-ephemeral, stated by the family's carriers
|
||||
|
||||
The mem engine's posture is the honest middle between "test
|
||||
convenience" (ADR-001 §4's original framing — *superseded*) and
|
||||
"durable production engine": **no durability, single-process,
|
||||
ephemeral by design.** Concretely:
|
||||
|
||||
- A **legitimate production choice for workloads that accept
|
||||
ephemerality** — the consumer's dependency on `alkstore-mem` *is*
|
||||
the ephemerality declaration (the ADR-016 pattern: the dependency
|
||||
edge states the truth and cannot drift from it).
|
||||
- **Never fleet-valid** — the fleet predicate ([ADR-010](010-queue-semantics-depth.md)
|
||||
§3's shared-pool rule) fails structurally: there is no pool, no
|
||||
process boundary, nothing shared. Two `MemStore` instances share
|
||||
nothing by design (§4).
|
||||
- Honesty rides the [ADR-016](016-deployment-honesty.md) carriers
|
||||
unchanged — no new surface: compile-time engine identity (the
|
||||
crate name), the engine crate's docs as the posture statement, and
|
||||
the [deployment matrix](../deployment.md) gaining the ephemeral
|
||||
row. The compile-time identity carries the honesty; nothing runtime
|
||||
is fabricated to detect what the dependency already declares.
|
||||
- alkblobs' mem line ("never a production story, never fleet-valid")
|
||||
is deliberately *not* copied verbatim — "never production" would
|
||||
contradict the sandbox use case that motivates this engine.
|
||||
|
||||
### 3. Full contract v1 — tier and asymmetry classification
|
||||
|
||||
The tier is **full contract v1**: every mechanism, every guarantee
|
||||
row, no reduced profile. A reduced test-only engine never gets
|
||||
born-pinned (the suite's equivalence rows would be absent from its
|
||||
column, exactly the drift the suite exists to prevent), and the
|
||||
motivating need is consumer-shaped, not test-shaped.
|
||||
|
||||
The documented engine asymmetries classify as:
|
||||
|
||||
- **`PayloadTooLarge`**: never produced, at any size — the SQLite
|
||||
occurrence posture. The variant stays contract-uniform and
|
||||
matchable ([ADR-016](016-deployment-honesty.md) §5); mem's suite
|
||||
column wires the non-occurrence arm.
|
||||
- **Reconnection**: no such concept — no connection can drop. The
|
||||
reconnect-wake and watcher-death rows are pg/SQLite-shaped and are
|
||||
*per-engine wired* in the suite, so mem's column simply does not
|
||||
carry them; nothing weakens across engines.
|
||||
- **Receiver close**: the one close path is engine drop (process
|
||||
exit/shutdown) closing open receivers — the contract's close-arm
|
||||
shapes apply, engine-realized as drop.
|
||||
- **Wake coalescing**: not applicable — wakes deliver at commit
|
||||
through in-process channels directly to each subscriber; there is
|
||||
no batching boundary to coalesce across. The engine spec pins the
|
||||
delivery shape.
|
||||
|
||||
### 4. Constructor shape — engine-native, isolated per open; shared instance rejected
|
||||
|
||||
Constructors live in engine crates ([ADR-008](008-contract-v1-pinning.md)
|
||||
§6), so the mem engine's constructor is engine-native: **`MemStore::new()`**
|
||||
— no connection string, no file path. This is the one documented
|
||||
exception to the core contract's open-prose ("connection string or
|
||||
file path"), and the prose is amended to name the exception
|
||||
(pre-implementation, class-1). **Isolation per open**: each
|
||||
`MemStore::new()` yields a distinct engine instance sharing nothing —
|
||||
exactly the suite factory's isolation contract; teardown is drop.
|
||||
|
||||
**A shared-instance constructor (two opens → one engine) is rejected
|
||||
for v1, with a re-entry gate**: it introduces real concurrency
|
||||
questions (claims and locks contended across two handles) that no
|
||||
consumer row names. The re-entry condition is a consumer-inventory
|
||||
row naming the two-opens-one-engine need — the ADR-016 §4 pattern
|
||||
(decision against, gate named); the open-questions register stays
|
||||
closed.
|
||||
|
||||
### 5. Wasm32 — compile-clean is an acceptance item; on-target suite execution is scoped out with a named collapse condition
|
||||
|
||||
The wasm motivation makes wasm compatibility load-bearing, so it
|
||||
moves from aspiration to **wave-6 acceptance item**:
|
||||
|
||||
1. `alkstore-mem` **compiles clean for `wasm32-unknown-unknown`**;
|
||||
2. the workspace host gates stay green (`cargo build` / `test` /
|
||||
clippy `-D warnings` / fmt);
|
||||
3. the contract-suite mem column passes on the host;
|
||||
4. the mem column runs under a **current-thread runtime flavor** —
|
||||
the test that keeps internals current-thread-clean, which is what
|
||||
wasm needs (no `spawn_blocking` anywhere, tokio's sync + time
|
||||
subset only). One suite-harness pre-task follows from this: the
|
||||
suite's `past_stamp_sleep` blocking sleep (the review-003 posture
|
||||
note, `properties.rs:1683`) becomes an async sleep — the only
|
||||
suite-side change the wave carries.
|
||||
|
||||
Running the *suite itself* on the wasm target (wasm-bindgen-test
|
||||
harnessing) is a different investment and is **scoped out of wave 6**
|
||||
with a named collapse condition: **a wasm-bindgen-test adapter crate
|
||||
for the suite existing**. This is a scope decision with a definite
|
||||
trigger, not a pending-client hedge — the acceptance the sandbox and
|
||||
adapter motivation actually needs (compile-clean, host-verified
|
||||
contract conformance) is delivered by items 1–4 and independently
|
||||
verifiable.
|
||||
|
||||
### 6. What the mem engine is not
|
||||
|
||||
- **Not a core dev-dependency.** ADR-001 §4's "doctests" posture
|
||||
declines: core takes no engine dev-dependency (the core crate has
|
||||
no driver dependencies, ever, dev-side included); core contract
|
||||
doc examples remain mock-store-based; mem's doctests live in the
|
||||
mem crate itself.
|
||||
- **Not a re-derivation of the suite** — ADR-022's factory is reused
|
||||
verbatim (its consequences already anticipated this).
|
||||
- **Not new contract surface.** The contract text changes here are
|
||||
the core-prose exception note (§4) and documentation rows (§2) —
|
||||
amend-in-place pre-release; the trait surface is untouched.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The wasm/sandbox and downstream-adapter paths gain their engine;
|
||||
every future family consumer that must run in a sandbox can depend
|
||||
on `alkstore-mem` without inventing one.
|
||||
- A third full implementation stresses contract v1 while the class-1
|
||||
window is open — born pinned against the finished suite rather
|
||||
than pinned retroactively; any wobble is amend-in-place now.
|
||||
- The suite's arithmetic-equivalence pinning goes three-way (backoff
|
||||
curve, opts resolution — [ADR-012](012-forked-substrate-design.md)
|
||||
§2's one-owner rule applied per engine, third owner).
|
||||
- Zero new contract surface; the ADR-016 carriers absorb the posture
|
||||
(identity + docs + matrix row), no runtime descriptor.
|
||||
|
||||
**Negative**
|
||||
|
||||
- The ADR-017 lockstep duty now covers three engines: a class-2
|
||||
(additive) contract change needs three adoption releases; a third
|
||||
engine's tests/opt machinery ride every wave forever.
|
||||
- Engine-owned arithmetic gains a third owner (backoff curve, opts
|
||||
resolution, the `@every` parser) — three copies to keep honest, and
|
||||
the honesty mechanism is the suite's equivalence rows, not shared
|
||||
code.
|
||||
- The mem engine's maintenance is *real* engine maintenance (the
|
||||
queue/scheduler/stream/lock machinery), not a weekend adapter —
|
||||
the honest-scale flag this ADR's Context records.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-001](001-crate-split.md) §4 — the deferral this ADR collapses;
|
||||
§4's "implementation convenience, not a contract artifact" framing
|
||||
superseded (amend-in-place, class-1 window open).
|
||||
- [ADR-022](022-contract-suite-layout.md) — the shared `StoreFactory`
|
||||
suite reused verbatim; the born-pinned mechanism.
|
||||
- [ADR-017](017-contract-versioning.md) — §2's class-1 window (still
|
||||
open, what makes this amendment cheap); §4's pairing carriers the
|
||||
mem engine adopts (manifest pin, suite column, engine docs).
|
||||
- [ADR-016](016-deployment-honesty.md) — the carriers the
|
||||
honest-ephemeral posture rides (§2); the compile-time-identity
|
||||
honesty pattern (§3); §4's rejection-with-gate pattern §4 uses; §5
|
||||
the `PayloadTooLarge` pinning mem's column wires.
|
||||
- [ADR-010](010-queue-semantics-depth.md) §3 — the fleet predicate
|
||||
that decides "never fleet-valid" structurally.
|
||||
- [ADR-008](008-contract-v1-pinning.md) §6 — constructors live in
|
||||
engine crates (the `MemStore::new()` home).
|
||||
- [ADR-012](012-forked-substrate-design.md) §2 — the one-owner rule
|
||||
the engine-owned arithmetic applies per engine.
|
||||
- `docs/research/consumer-inventory.md` §Engine-tier inventory — the
|
||||
row-first record (operator-authority, 2026-10-10) that precedes
|
||||
this decision.
|
||||
- Review 003 (`docs/reviews/003-wave-5-general-review.md`) — the
|
||||
`properties.rs:1683` current-thread posture note the suite-harness
|
||||
pre-task disposes.
|
||||
- `docs/architecture/engine-mem.md` — the engine spec this ADR
|
||||
requires as a wave-6 deliverable.
|
||||
- `docs/plans/implementation.md` — the wave-6 insertion (release
|
||||
readiness renumbered to wave 7).
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-10 (docs alignment — v1 TLS posture stated; QueueOpts numeric consumer-obligation notes)
|
||||
last_updated: 2026-10-10 (ADR-024 — the mem engine joins the matrix:
|
||||
host row, connection budget, durability posture, wasm32 toolchain note)
|
||||
---
|
||||
|
||||
# Deployment
|
||||
@@ -20,6 +21,7 @@ document holds the facts.
|
||||
|---|---|---|
|
||||
| SQLite | **single-machine**, file-backed | NFS two-writers unsupported (honker's honesty posture, inherited, [ADR-003](decisions/003-sqlite-driver.md)). Cross-process *on one host* is verified POC ground (`data_version` is cross-process by nature). |
|
||||
| Postgres | **multi-host native** | Nothing assumes a shared host; POC #2 ran all-through-network (docker bridge) with the same properties ([ADR-004](decisions/004-postgres-driver.md)). |
|
||||
| Mem | **single-process, ephemeral** ([ADR-024](decisions/024-mem-engine.md) §2) | In-memory, per-instance; no cross-process story at all — two engine instances share nothing by design. No durability, by design: loss on process exit is the documented posture, and the dependency on `alkstore-mem` *is* the ephemerality declaration (honest for workloads that accept it). Never fleet-valid — the shared-pool predicate ([ADR-010](decisions/010-queue-semantics-depth.md) §3) fails structurally; there is no pool. Compile-clean on `wasm32-unknown-unknown` ([ADR-024](decisions/024-mem-engine.md) §5) — the family's member that runs in the sandbox. |
|
||||
|
||||
The unified trait must not pretend SQLite is multi-host — and it does
|
||||
not: [ADR-016](decisions/016-deployment-honesty.md) resolves that
|
||||
@@ -82,6 +84,14 @@ engine-owned PostgreSQL schema (default `alkstore`, per-engine option)
|
||||
— layout per [ADR-010](decisions/010-queue-semantics-depth.md) §8
|
||||
(resolved from [queues.md](queues.md)'s namespace bullet).
|
||||
|
||||
### Mem engine
|
||||
|
||||
- No connections — there is nothing to budget. The engine instance
|
||||
owns its state in-process; sharing nothing is the isolation design
|
||||
([ADR-024](decisions/024-mem-engine.md) §4's instance rule; the
|
||||
two-opens-one-engine constructor is rejected, re-entry via a
|
||||
consumer-inventory row).
|
||||
|
||||
### TLS posture (v1)
|
||||
|
||||
The pg engine hardwires `NoTls` on **every connection path** in v1 —
|
||||
@@ -143,6 +153,7 @@ runtime is fabricated.
|
||||
| SQLite | `synchronous` | WAL + `NORMAL` shipped ([ADR-003](decisions/003-sqlite-driver.md)); FULL is available consumer-side for stricter durability; commit fsyncs land at WAL checkpoints (the ~1000-commit spike cadence, POC #1) |
|
||||
| SQLite | `poll_interval` | the watcher's `data_version` poll cadence — 1 ms shipping default ([ADR-023](decisions/023-fourth-review-round.md) §4, the measured-wake-latency posture), carried on `SqliteOpts`; the idle cost is ~1000 poll reads/sec/instance; raising the interval trades wake latency (interval-bound) for idle CPU — the tuning recipe for latency-tolerant deployments |
|
||||
| Postgres | `synchronous_commit` | per-session knob; `on` is ship config (p50 2.40 ms seam); `off` trades max-tail (40.9 ms) for slightly better p50 — measured, honest trade ([ADR-004](decisions/004-postgres-driver.md)); session-level SET mechanics POC-verified |
|
||||
| Mem | — (none) | There is no durability knob and no durability: the honest row is the absence itself ([ADR-024](decisions/024-mem-engine.md) §2). Wake latency is sub-millisecond in-process; no tuning surface exists |
|
||||
|
||||
These are engine-configuration concerns, *not* trait surface. What
|
||||
part of engine config is contract-level shape vs engine-crate docs is
|
||||
@@ -158,6 +169,7 @@ the contract is the trait the constructor returns
|
||||
| `bundled-sqlite` adds a C build (~10 s dev, cacheable) | SQLite | build/CI time |
|
||||
| `libsqlite3-sys` collision with sqlx today | any mixed-driver binary | structurally avoided ([ADR-001](decisions/001-crate-split.md) single-driver rule) |
|
||||
| Listener `application_name` set for diagnosability (kill-targetable) | Postgres | ops runbooks ([ADR-004](decisions/004-postgres-driver.md)) |
|
||||
| `alkstore-mem` compiles clean on `wasm32-unknown-unknown` (wave-6 acceptance, [ADR-024](decisions/024-mem-engine.md) §5); tokio features limited to the wasm-supported subset (sync + time); current-thread-runtime-clean internals (the suite's mem column runs under that flavor) | Mem | wasm targets, sandboxes, downstream ffi/napi/python adapters — the only engine that can compile to wasm (rusqlite/tokio-postgres are structurally out) |
|
||||
|
||||
## Consumer-facing latency profile (indicative, POC-measured)
|
||||
|
||||
@@ -185,6 +197,7 @@ From both POCs (single-box, relative shapes are the deliverable —
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | constructor/options in engine crates; no capability surface in v1 (OQ-08; resolved by [ADR-016](decisions/016-deployment-honesty.md)) |
|
||||
| [016](decisions/016-deployment-honesty.md) | Deployment honesty | no runtime capability surface — compile-time identity + this matrix; `PayloadTooLarge` is the one runtime asymmetry carriage |
|
||||
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round | drop = rollback keeps the connection budgets exact under error paths (SQLite lease releases; pg client re-pools) |
|
||||
| [024](decisions/024-mem-engine.md) | Mem engine | the matrix's third engine — honest-ephemeral: single-process, no durability, never fleet-valid, wasm32 compile-clean; no connection budget, no knobs |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-10 (initial spec — ADR-024 adopted the mem
|
||||
engine as wave 6; implementation pending, spec grows with the wave's
|
||||
decomposition)
|
||||
---
|
||||
|
||||
# Mem engine
|
||||
|
||||
The `alkstore-mem` engine implements
|
||||
[core-contract.md](core-contract.md) on guarded in-memory structures.
|
||||
This spec records WHAT the engine is — its architecture, the tx
|
||||
overlay, the asymmetry classification — not code-level HOW. Decisions
|
||||
live in ADRs ([ADR-024](decisions/024-mem-engine.md) is this engine's
|
||||
charter); contract obligations live in the core spec.
|
||||
|
||||
## Identity and posture
|
||||
|
||||
- Separate workspace crate (ADR-024 §1): `alkstore`, tokio (sync +
|
||||
time — the wasm-supported subset), no driver dependencies. Natively
|
||||
async; no `spawn_blocking` seam anywhere; internals are
|
||||
current-thread-runtime-clean by construction, and the suite's mem
|
||||
column runs *under a current-thread flavor* to keep that honest.
|
||||
- **Honest-ephemeral** (ADR-024 §2): no durability, single-process,
|
||||
ephemeral by design; legitimate production for workloads that
|
||||
accept ephemerality; never fleet-valid (the
|
||||
[ADR-010](decisions/010-queue-semantics-depth.md) §3 shared-pool
|
||||
predicate fails structurally — there is no pool). The dependency
|
||||
edge on `alkstore-mem` is the ephemerality declaration; see
|
||||
[deployment.md](deployment.md).
|
||||
- **Full contract v1** (ADR-024 §3): every mechanism, every guarantee
|
||||
row. Born pinned — the engine lands against the finished
|
||||
version-stamped suite ([ADR-022](decisions/022-contract-suite-layout.md)'s
|
||||
`StoreFactory`, reused verbatim as a third factory column), which
|
||||
is also what pins its engine-owned arithmetic three-way
|
||||
([ADR-012](decisions/012-forked-substrate-design.md) §2's one-owner
|
||||
rule: the backoff curve, opts resolution, and the `@every` parser
|
||||
get mem's own implementations; equivalence with the other two
|
||||
engines is suite-pinned).
|
||||
- wasm32: `alkstore-mem` compiling clean for
|
||||
`wasm32-unknown-unknown` is an acceptance item of its wave
|
||||
(ADR-024 §5) and an identity line of this spec.
|
||||
|
||||
## Architecture
|
||||
|
||||
Everything the other engines realize with connections, schemas, and
|
||||
daemons, the mem engine realizes with process-owned state:
|
||||
|
||||
- **No connections at all** — no pool (pg's), no writer/reader slots
|
||||
(SQLite's), no listener. The store instance *is* the state: guarded
|
||||
per-mechanism tables (queues + jobs, dead letters, streams +
|
||||
offsets, locks, schedules) owned by the engine instance.
|
||||
- **No background machinery to bootstrap** — no schema bootstrap, no
|
||||
watcher thread (nothing cross-connection to poll), no forwarder.
|
||||
The only async machinery is tokio-time-driven: lock TTLs, the
|
||||
scheduler's boundary waits, and wake delivery.
|
||||
- **Wake substrate**: in-process channels. Wakes deliver at the
|
||||
commit boundary directly to each open receiver — no batching
|
||||
boundary exists, so coalescing is not applicable; per-subscriber
|
||||
delivery order is FIFO by commit order.
|
||||
|
||||
### The tx overlay
|
||||
|
||||
The tx seam's mem realization: a `begin_tx()` allocates an **overlay**
|
||||
private to the caller's handle.
|
||||
|
||||
- `*_tx` methods stage effects in the overlay (enqueues, publishes,
|
||||
notifies); reads *inside* the tx merge the overlay over committed
|
||||
state — read-your-own-writes with no round-trips.
|
||||
- **Commit** merges the overlay atomically (a single guarded merge)
|
||||
and fires the overlay's pending wakes — *wake-at-commit* is the
|
||||
discipline-critical property: a notify staged in the tx delivers
|
||||
nothing until the overlay merges, exactly the seam the pg engine
|
||||
had to fix retroactively (`pg-fix-tx-wake`) and the suite's
|
||||
tx-commit-atomicity rows pin.
|
||||
- **Drop** discards the overlay — drop = rollback with no ghosts,
|
||||
trivially, because uncommitted state never touched committed state.
|
||||
- No isolation questions can arise across handles: an overlay is
|
||||
invisible to every other store/view until its commit.
|
||||
|
||||
## Mapping the contract
|
||||
|
||||
| Contract piece | Engine realization |
|
||||
|---|---|
|
||||
| notify / listen | staged notify in the tx overlay; delivered at commit to open receivers. Commit-atomicity is structural ([ADR-007](decisions/007-transactional-seam.md)) — the merge and the wake fire are one operation |
|
||||
| streams | in-memory event log + per-consumer offsets, global-FIFO by monotone offset; keyed publish rides the overlay ([ADR-015](decisions/015-streams-depth.md)); `trim_to` mutates the log and wakes nothing |
|
||||
| queues | the ADR-010 state machine over an in-memory job table (visibility, reclaim, the equal-jitter curve engine-owned, dead-letter move, the no-stranded-rows sweep) — all driver-free, swept on caller-driven `sweep_expired` |
|
||||
| named locks | in-memory TTL registry ([ADR-008](decisions/008-contract-v1-pinning.md) §7's row); expiry via tokio time; single-instance contention semantics |
|
||||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule table, tick (boundary advance + 64-boundary catch-up cap), leadership on the `__alkstore_scheduler` lock — single-instance leadership is trivially owned but the lock machinery still runs, per ADR-009 §4's engine-uniform rule; outbox = helper over queues ([ADR-014](decisions/014-outbox-tx-enqueue.md)) |
|
||||
| begin_tx | overlay allocation ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||
| handle ops | direct `.await`s on guarded state; `encode_payload` via core (ADR-020 §4) |
|
||||
|
||||
## Asymmetry classification (ADR-024 §3)
|
||||
|
||||
| Asymmetry | Mem posture |
|
||||
|---|---|
|
||||
| `PayloadTooLarge` | never produced, any size (SQLite occurrence arm); the non-occurrence arm wires in mem's suite column ([ADR-016](decisions/016-deployment-honesty.md) §5) |
|
||||
| Reconnection | does not exist — no connection can drop; the reconnect-wake and watcher-death rows are pg/SQLite-shaped and absent from mem's column (rows are per-engine wired; nothing cross-wires or weakens) |
|
||||
| Receiver close | one close path: engine drop closes open receivers (process exit/shutdown); contract close-arm shapes apply unchanged ([ADR-021](decisions/021-tx-reads-and-value-shape-fixes.md)) |
|
||||
| Wake coalescing | not applicable — direct at-commit delivery, per-subscriber FIFO; the engine spec line above pins it |
|
||||
|
||||
## Isolation and the suite factory
|
||||
|
||||
Each `MemStore::new()` yields a distinct engine instance sharing
|
||||
nothing (ADR-024 §4's isolation rule — the contract's *Store* concept
|
||||
amended to admit the engine-native constructor). The suite factory:
|
||||
a fresh store per run, teardown = drop; the mem column carries the
|
||||
mechanism rows that express on it, omits the engine-shaped rows that
|
||||
cannot (the table above), and runs under the current-thread flavor
|
||||
(ADR-024 §5 item 4 — the suite harness's `past_stamp_sleep` async
|
||||
sleep is the one pre-task the wave carries). Running the suite
|
||||
*on* the wasm target is scoped out behind a named collapse condition
|
||||
(a wasm-bindgen-test adapter crate), per ADR-024 §5.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| ADR | Decision | Summary |
|
||||
|---|---|---|
|
||||
| [001](decisions/001-crate-split.md) | Crate split | §4's mem clause collapsed + amended by [ADR-024](decisions/024-mem-engine.md) — separate crate, full contract v1 |
|
||||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | opaque wake + re-read; in-process channels as the wake substrate; no reconnection concept (no connection can drop) |
|
||||
| [007](decisions/007-transactional-seam.md) | Tx seam | the tx overlay — staging, commit-merge, drop-discard; wake-at-commit structural |
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | full-surface implementation; §6 config split (constructor in the engine crate); §7 locks row as a TTL registry |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule table + tick + catch-up + leadership lock (trivially owned, still run — engine-uniform) |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | the state machine driver-free; curve engine-owned; §3's fleet predicate decides never-fleet-valid |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Fork design | §2 one-owner rule, third owner: backoff curve, opts resolution, `@every` parser — three-way equivalence suite-pinned |
|
||||
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` staged in the overlay |
|
||||
| [015](decisions/015-streams-depth.md) | Streams depth | in-memory log + per-consumer offsets; keyed publish; `trim_to` (wakes nothing) |
|
||||
| [016](decisions/016-deployment-honesty.md) | Deployment honesty | no runtime surface — honest-ephemeral by compile-time identity + docs + matrix row; `PayloadTooLarge` never produced |
|
||||
| [017](decisions/017-contract-versioning.md) | Contract versioning | manifest pins core per §4; suite column is the pairing instrument |
|
||||
| [019](decisions/019-mechanism-handle-surfaces.md) | Handle surfaces | boxed handle traits implemented over guarded state |
|
||||
| [020](decisions/020-enqueue-opt-semantics-and-bridges.md) | Enqueue options | opts resolution engine-owned (third copy); payload encoding via core |
|
||||
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round | drop = rollback trivial by overlay discard; receiver close at engine drop |
|
||||
| [022](decisions/022-contract-suite-layout.md) | Suite layout | factory reused verbatim — third column, born pinned |
|
||||
| [024](decisions/024-mem-engine.md) | Mem engine charter | crate placement, honest-ephemeral posture, full tier, constructor + shared-instance rejection, wasm acceptance items |
|
||||
|
||||
## Open Questions
|
||||
|
||||
Open questions are tracked in
|
||||
[open-questions.md](open-questions.md). This spec opens none — the
|
||||
register is closed, and every decision this engine needed is in
|
||||
[ADR-024](decisions/024-mem-engine.md). The two deliberately-not-OQ
|
||||
dispositions live there: the shared-instance constructor rejection
|
||||
(§4, re-entry = a consumer-inventory row) and the on-target wasm
|
||||
suite execution scope-out (§5, collapse condition = an adapter
|
||||
crate).
|
||||
@@ -1,16 +1,20 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-07 (ADR-022 — contract-suite layout: shared internal suite crate)
|
||||
last_updated: 2026-10-10 (ADR-024 — the mem engine joins the crate
|
||||
family as `alkstore-mem`; family/ADR tables updated)
|
||||
---
|
||||
|
||||
# alkstore — Overview
|
||||
|
||||
One reactive store interface over SQLite and Postgres: durable
|
||||
One reactive store interface over SQLite, Postgres, and the
|
||||
in-process mem engine: durable
|
||||
notify/subscribe signals, streams with per-consumer offsets, durable
|
||||
queues with the transactional enqueue property, named locks, a
|
||||
scheduler, and an outbox helper — with each engine using its own
|
||||
native machinery underneath (SQLite: honker's watcher design;
|
||||
Postgres: `pg_notify`/LISTEN and the pg-boss schema family).
|
||||
Postgres: `pg_notify`/LISTEN and the pg-boss schema family; mem:
|
||||
guarded in-process state, ephemeral by design —
|
||||
[ADR-024](decisions/024-mem-engine.md)).
|
||||
|
||||
## Why this crate exists
|
||||
|
||||
@@ -31,7 +35,7 @@ Per [ADR-001](decisions/001-crate-split.md):
|
||||
| `alkstore` (core) | trait surface, types, error model | none (no capability surface — [ADR-016](decisions/016-deployment-honesty.md): engine differences are compile-time identity + the deployment matrix, never a runtime descriptor) |
|
||||
| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite; the forked honker-core lineage rides in-tree as the engine crate's substrate module subtree ([ADR-011](decisions/011-sqlite-substrate-fork.md), designed in [ADR-012](decisions/012-forked-substrate-design.md), folded per [ADR-013](decisions/013-fold-substrate-into-sqlite.md)) |
|
||||
| `alkstore-postgres` | Postgres engine ([ADR-004](decisions/004-postgres-driver.md)) | tokio-postgres, deadpool-postgres |
|
||||
| (mem engine, optional) | test convenience, decided at implementation ([ADR-001](decisions/001-crate-split.md)) | none |
|
||||
| `alkstore-mem` | mem engine — full contract v1, honest-ephemeral (no durability, single-process, never fleet-valid; [ADR-024](decisions/024-mem-engine.md)) | none (tokio sync + time only; the wasm32-clean member of the family) |
|
||||
|
||||
Downstream consumers depend on core + exactly one engine. Family
|
||||
standards apply throughout: tokio async runtime, `thiserror` errors, no
|
||||
@@ -55,6 +59,7 @@ Per [ADR-002](decisions/002-feature-scope.md):
|
||||
| [core-contract.md](core-contract.md) | The unified trait surface: mechanisms, delivery guarantees, tx seam, naming |
|
||||
| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto the forked substrate module/rusqlite |
|
||||
| [engine-postgres.md](engine-postgres.md) | Postgres engine: mapping the contract onto tokio-postgres/LISTEN |
|
||||
| [engine-mem.md](engine-mem.md) | Mem engine: mapping the contract onto guarded in-process state (ADR-024) |
|
||||
| [queues.md](queues.md) | Queue/scheduler/outbox semantics depth (ADR-009/ADR-010 resolved) |
|
||||
| [deployment.md](deployment.md) | Host capabilities, connection budgets, deployment matrix (OQ-08 resolved) |
|
||||
| [open-questions.md](open-questions.md) | OQ-01..NN tracker |
|
||||
@@ -86,6 +91,7 @@ Per [ADR-002](decisions/002-feature-scope.md):
|
||||
| [020](decisions/020-enqueue-opt-semantics-and-bridges.md) | Enqueue-option semantics (delay/run_at, expires, scheduler stamp source, payload encoding) | Accepted |
|
||||
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round (tx-read methods on `TxHandle`, `Job.claimed_at`, `schedule()` queue-argument validation, drop = rollback, receiver close/error arms) | Accepted |
|
||||
| [022](decisions/022-contract-suite-layout.md) | Contract-suite layout (shared internal suite crate, store-factory parameterization) | Accepted |
|
||||
| [024](decisions/024-mem-engine.md) | The mem engine (`alkstore-mem` — full contract v1, honest-ephemeral, wasm32 compile-clean acceptance) | Accepted |
|
||||
|
||||
## Non-goals
|
||||
|
||||
|
||||
Reference in new issue
Block a user