146 lines
9.8 KiB
Markdown
146 lines
9.8 KiB
Markdown
---
|
|
status: implementation-complete
|
|
last_updated: 2026-10-10 (the wave-6 wave landed: the full surface
|
|
wired over guarded state, the suite's third `StoreFactory` column
|
|
green on the host under the current-thread flavor, the wasm32
|
|
compile gate clean — the review gate holds the stable flip)
|
|
---
|
|
|
|
# Mem engine
|
|
|
|
The `alkstore-mem` engine implements
|
|
[core-contract.md](core-contract.md) on guarded in-memory structures.
|
|
This spec records WHAT the engine is — its architecture, the tx
|
|
overlay, the asymmetry classification — not code-level HOW. Decisions
|
|
live in ADRs ([ADR-024](decisions/024-mem-engine.md) is this engine's
|
|
charter); contract obligations live in the core spec.
|
|
|
|
## Identity and posture
|
|
|
|
- Separate workspace crate (ADR-024 §1): `alkstore`, tokio (sync +
|
|
time — the wasm-supported subset), no driver dependencies. Natively
|
|
async; no `spawn_blocking` seam anywhere; internals are
|
|
current-thread-runtime-clean by construction, and the suite's mem
|
|
column runs *under a current-thread flavor* to keep that honest.
|
|
- **Honest-ephemeral** (ADR-024 §2): no durability, single-process,
|
|
ephemeral by design; legitimate production for workloads that
|
|
accept ephemerality; never fleet-valid (the
|
|
[ADR-010](decisions/010-queue-semantics-depth.md) §3 shared-pool
|
|
predicate fails structurally — there is no pool). The dependency
|
|
edge on `alkstore-mem` is the ephemerality declaration; see
|
|
[deployment.md](deployment.md).
|
|
- **Full contract v1** (ADR-024 §3): every mechanism, every guarantee
|
|
row. Born pinned — the engine lands against the finished
|
|
version-stamped suite ([ADR-022](decisions/022-contract-suite-layout.md)'s
|
|
`StoreFactory`, reused verbatim as a third factory column), which
|
|
is also what pins its engine-owned arithmetic three-way
|
|
([ADR-012](decisions/012-forked-substrate-design.md) §2's one-owner
|
|
rule: the backoff curve, opts resolution, and the `@every` parser
|
|
get mem's own implementations; equivalence with the other two
|
|
engines is suite-pinned).
|
|
- wasm32: `alkstore-mem` compiling clean for
|
|
`wasm32-unknown-unknown` is an acceptance item of its wave
|
|
(ADR-024 §5) and an identity line of this spec.
|
|
|
|
## Architecture
|
|
|
|
Everything the other engines realize with connections, schemas, and
|
|
daemons, the mem engine realizes with process-owned state:
|
|
|
|
- **No connections at all** — no pool (pg's), no writer/reader slots
|
|
(SQLite's), no listener. The store instance *is* the state: guarded
|
|
per-mechanism tables (queues + jobs, dead letters, streams +
|
|
offsets, locks, schedules) owned by the engine instance.
|
|
- **No background machinery to bootstrap** — no schema bootstrap, no
|
|
watcher thread (nothing cross-connection to poll), no forwarder.
|
|
The only async machinery is tokio-time-driven: lock TTLs, the
|
|
scheduler's boundary waits, and wake delivery.
|
|
- **Wake substrate**: in-process channels. Wakes deliver at the
|
|
commit boundary directly to each open receiver — no batching
|
|
boundary exists, so coalescing is not applicable; per-subscriber
|
|
delivery order is FIFO by commit order.
|
|
|
|
### The tx overlay
|
|
|
|
The tx seam's mem realization: a `begin_tx()` allocates an **overlay**
|
|
private to the caller's handle.
|
|
|
|
- `*_tx` methods stage effects in the overlay (enqueues, publishes,
|
|
notifies); reads *inside* the tx merge the overlay over committed
|
|
state — read-your-own-writes with no round-trips.
|
|
- **Commit** merges the overlay atomically (a single guarded merge)
|
|
and fires the overlay's pending wakes — *wake-at-commit* is the
|
|
discipline-critical property: a notify staged in the tx delivers
|
|
nothing until the overlay merges, exactly the seam the pg engine
|
|
had to fix retroactively (`pg-fix-tx-wake`) and the suite's
|
|
tx-commit-atomicity rows pin.
|
|
- **Drop** discards the overlay — drop = rollback with no ghosts,
|
|
trivially, because uncommitted state never touched committed state.
|
|
- No isolation questions can arise across handles: an overlay is
|
|
invisible to every other store/view until its commit.
|
|
|
|
## Mapping the contract
|
|
|
|
| Contract piece | Engine realization |
|
|
|---|---|
|
|
| notify / listen | staged notify in the tx overlay; delivered at commit to open receivers. Commit-atomicity is structural ([ADR-007](decisions/007-transactional-seam.md)) — the merge and the wake fire are one operation |
|
|
| streams | in-memory event log + per-consumer offsets, global-FIFO by monotone offset; keyed publish rides the overlay ([ADR-015](decisions/015-streams-depth.md)); `trim_to` mutates the log and wakes nothing |
|
|
| queues | the ADR-010 state machine over an in-memory job table (visibility, reclaim, the equal-jitter curve engine-owned, dead-letter move, the no-stranded-rows sweep) — all driver-free, swept on caller-driven `sweep_expired` |
|
|
| named locks | in-memory TTL registry ([ADR-008](decisions/008-contract-v1-pinning.md) §7's row); expiry via tokio time; single-instance contention semantics |
|
|
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule table, tick (boundary advance + 64-boundary catch-up cap), leadership on the `__alkstore_scheduler` lock — single-instance leadership is trivially owned but the lock machinery still runs, per ADR-009 §4's engine-uniform rule; outbox = helper over queues ([ADR-014](decisions/014-outbox-tx-enqueue.md)) |
|
|
| begin_tx | overlay allocation ([ADR-007](decisions/007-transactional-seam.md)) |
|
|
| handle ops | direct `.await`s on guarded state; `encode_payload` via core (ADR-020 §4) |
|
|
|
|
## Asymmetry classification (ADR-024 §3)
|
|
|
|
| Asymmetry | Mem posture |
|
|
|---|---|
|
|
| `PayloadTooLarge` | never produced, any size (SQLite occurrence arm); the non-occurrence arm wires in mem's suite column ([ADR-016](decisions/016-deployment-honesty.md) §5) |
|
|
| Reconnection | does not exist — no connection can drop; the reconnect-wake and watcher-death rows are pg/SQLite-shaped and absent from mem's column (rows are per-engine wired; nothing cross-wires or weakens) |
|
|
| Receiver close | one close path: engine drop closes open receivers (process exit/shutdown); contract close-arm shapes apply unchanged ([ADR-021](decisions/021-tx-reads-and-value-shape-fixes.md)) |
|
|
| Wake coalescing | not applicable — direct at-commit delivery, per-subscriber FIFO; the engine spec line above pins it |
|
|
|
|
## Isolation and the suite factory
|
|
|
|
Each `MemStore::new()` yields a distinct engine instance sharing
|
|
nothing (ADR-024 §4's isolation rule — the contract's *Store* concept
|
|
amended to admit the engine-native constructor). The suite factory:
|
|
a fresh store per run, teardown = drop; the mem column carries the
|
|
mechanism rows that express on it, omits the engine-shaped rows that
|
|
cannot (the table above), and runs under the current-thread flavor
|
|
(ADR-024 §5 item 4 — the suite harness's `past_stamp_sleep` async
|
|
sleep is the one pre-task the wave carries). Running the suite
|
|
*on* the wasm target is scoped out behind a named collapse condition
|
|
(a wasm-bindgen-test adapter crate), per ADR-024 §5.
|
|
|
|
## Design Decisions
|
|
|
|
| ADR | Decision | Summary |
|
|
|---|---|---|
|
|
| [001](decisions/001-crate-split.md) | Crate split | §4's mem clause collapsed + amended by [ADR-024](decisions/024-mem-engine.md) — separate crate, full contract v1 |
|
|
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | opaque wake + re-read; in-process channels as the wake substrate; no reconnection concept (no connection can drop) |
|
|
| [007](decisions/007-transactional-seam.md) | Tx seam | the tx overlay — staging, commit-merge, drop-discard; wake-at-commit structural |
|
|
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | full-surface implementation; §6 config split (constructor in the engine crate); §7 locks row as a TTL registry |
|
|
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule table + tick + catch-up + leadership lock (trivially owned, still run — engine-uniform) |
|
|
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | the state machine driver-free; curve engine-owned; §3's fleet predicate decides never-fleet-valid |
|
|
| [012](decisions/012-forked-substrate-design.md) | Fork design | §2 one-owner rule, third owner: backoff curve, opts resolution, `@every` parser — three-way equivalence suite-pinned |
|
|
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` staged in the overlay |
|
|
| [015](decisions/015-streams-depth.md) | Streams depth | in-memory log + per-consumer offsets; keyed publish; `trim_to` (wakes nothing) |
|
|
| [016](decisions/016-deployment-honesty.md) | Deployment honesty | no runtime surface — honest-ephemeral by compile-time identity + docs + matrix row; `PayloadTooLarge` never produced |
|
|
| [017](decisions/017-contract-versioning.md) | Contract versioning | manifest pins core per §4; suite column is the pairing instrument |
|
|
| [019](decisions/019-mechanism-handle-surfaces.md) | Handle surfaces | boxed handle traits implemented over guarded state |
|
|
| [020](decisions/020-enqueue-opt-semantics-and-bridges.md) | Enqueue options | opts resolution engine-owned (third copy); payload encoding via core |
|
|
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round | drop = rollback trivial by overlay discard; receiver close at engine drop |
|
|
| [022](decisions/022-contract-suite-layout.md) | Suite layout | factory reused verbatim — third column, born pinned |
|
|
| [024](decisions/024-mem-engine.md) | Mem engine charter | crate placement, honest-ephemeral posture, full tier, constructor + shared-instance rejection, wasm acceptance items |
|
|
|
|
## Open Questions
|
|
|
|
Open questions are tracked in
|
|
[open-questions.md](open-questions.md). This spec opens none — the
|
|
register is closed, and every decision this engine needed is in
|
|
[ADR-024](decisions/024-mem-engine.md). The two deliberately-not-OQ
|
|
dispositions live there: the shared-instance constructor rejection
|
|
(§4, re-entry = a consumer-inventory row) and the on-target wasm
|
|
suite execution scope-out (§5, collapse condition = an adapter
|
|
crate). |