alkstore/docs/architecture/engine-mem.md

status, last_updated

status

last_updated

stable

2026-10-11 (review-004 items 10+11 pinned: the growth postures section — subscribe backlog drain, O(live-rows) claim scan — and the outbox's delivered-or-reclaimable semantic; the wave-6 review gate had passed earlier — conformance read clean, three edge-domain fixes landed inline (the stream cursor read's `i64::MAX` overflow, `trim_to`'s horizon endpoint, the scheduler register-time interval overflow now rejecting with `InvalidSpec`), the third column's 25-row set green under the current-thread flavor, the wasm32 gate + feature subset verified, the rider's arms verified)

Mem engine

The alkstore-mem engine implements core-contract.md on guarded in-memory structures. This spec records WHAT the engine is — its architecture, the tx overlay, the asymmetry classification — not code-level HOW. Decisions live in ADRs (ADR-024 is this engine's charter); contract obligations live in the core spec.

Identity and posture

• Separate workspace crate (ADR-024 §1): alkstore, tokio (sync + time — the wasm-supported subset), no driver dependencies. Natively async; no spawn_blocking seam anywhere; internals are current-thread-runtime-clean by construction, and the suite's mem column runs under a current-thread flavor to keep that honest.
• Honest-ephemeral (ADR-024 §2): no durability, single-process, ephemeral by design; legitimate production for workloads that accept ephemerality; never fleet-valid (the ADR-010 §3 shared-pool predicate fails structurally — there is no pool). The dependency edge on alkstore-mem is the ephemerality declaration; see deployment.md.
• Full contract v1 (ADR-024 §3): every mechanism, every guarantee row. Born pinned — the engine lands against the finished version-stamped suite (ADR-022's StoreFactory, reused verbatim as a third factory column), which is also what pins its engine-owned arithmetic three-way (ADR-012 §2's one-owner rule: the backoff curve, opts resolution, and the @every parser get mem's own implementations; equivalence with the other two engines is suite-pinned).
• wasm32: alkstore-mem compiling clean for wasm32-unknown-unknown is an acceptance item of its wave (ADR-024 §5) and an identity line of this spec.

Architecture

Everything the other engines realize with connections, schemas, and daemons, the mem engine realizes with process-owned state:

• No connections at all — no pool (pg's), no writer/reader slots (SQLite's), no listener. The store instance is the state: guarded per-mechanism tables (queues + jobs, dead letters, streams + offsets, locks, schedules) owned by the engine instance.
• No background machinery to bootstrap — no schema bootstrap, no watcher thread (nothing cross-connection to poll), no forwarder. The only async machinery is tokio-time-driven: lock TTLs, the scheduler's boundary waits, and wake delivery. All timestamps read the process clock through one internal accessor — no server clock to defer to, no per-mechanism clock reads (the wasm-posture note, ADR-024 §5).
• Wake substrate: in-process channels. Wakes deliver at the commit boundary directly to each open receiver — no batching boundary exists, so coalescing is not applicable; per-subscriber delivery order is FIFO by commit order.

The tx overlay

The tx seam's mem realization: a begin_tx() allocates an overlay private to the caller's handle.

• *_tx methods stage effects in the overlay (enqueues, publishes, notifies); reads inside the tx merge the overlay over committed state — read-your-own-writes with no round-trips.
• Commit merges the overlay atomically (a single guarded merge) and fires the overlay's pending wakes — wake-at-commit is the discipline-critical property: a notify staged in the tx delivers nothing until the overlay merges, exactly the seam the pg engine had to fix retroactively (pg-fix-tx-wake) and the suite's tx-commit-atomicity rows pin.
• Drop discards the overlay — drop = rollback with no ghosts, trivially, because uncommitted state never touched committed state.
• No isolation questions can arise across handles: an overlay is invisible to every other store/view until its commit.

Mapping the contract

Contract piece	Engine realization
notify / listen	staged notify in the tx overlay; delivered at commit to open receivers. Commit-atomicity is structural (ADR-007) — the merge and the wake fire are one operation
streams	in-memory event log + per-consumer offsets, global-FIFO by monotone offset; keyed publish rides the overlay (ADR-015); trim_to mutates the log and wakes nothing
queues	the ADR-010 state machine over an in-memory job table (visibility, reclaim, the equal-jitter curve engine-owned, dead-letter move, the no-stranded-rows sweep) — all driver-free, swept on caller-driven sweep_expired
named locks	in-memory TTL registry (ADR-008 §7's row); expiry via tokio time; single-instance contention semantics
scheduler / outbox	collapse shape (ADR-009): schedule table, tick (boundary advance + 64-boundary catch-up cap), leadership on the __alkstore_scheduler lock — single-instance leadership is trivially owned but the lock machinery still runs, per ADR-009 §4's engine-uniform rule; outbox = helper over queues (ADR-014)
begin_tx	overlay allocation (ADR-007)
handle ops	direct .awaits on guarded state; encode_payload via core (ADR-020 §4)

Outbox semantics

The outbox is a helper over queues (ADR-014): run_once claims one job through the ordinary claim machinery and applies the outcome when delivery returns — Ok ⇒ ack, Err ⇒ retry on the queue's own curve. The ack/retry application's row-ref booleans are dropped: a delivery that outlived its visibility window (or a consumer side-effect that handled the row first, e.g. an in-closure ack/retry on the boxed handle) leaves the outer outcome unapplied, and run_once still reports true. Inherent to the no-heartbeat outbox design — at-least-once reclaim is the recovery, not a second report. Processed means delivered-or-reclaimable: true attests the claim was processed, not that the ack/exhaust landed; renewal inside the window is the consumer's obligation (ADR-019 §5) and reclaim closes a lapsed one.

Asymmetry classification (ADR-024 §3)

Asymmetry	Mem posture
PayloadTooLarge	never produced, any size (SQLite occurrence arm); the non-occurrence arm wires in mem's suite column (ADR-016 §5)
Reconnection	does not exist — no connection can drop; the reconnect-wake and watcher-death rows are pg/SQLite-shaped and absent from mem's column (rows are per-engine wired; nothing cross-wires or weakens)
Receiver close	one close path: engine drop closes open receivers (process exit/shutdown); contract close-arm shapes apply unchanged (ADR-021)
Wake coalescing	not applicable — direct at-commit delivery, per-subscriber FIFO; the engine spec line above pins it

Growth postures (review 004 items 10–11)

Two perf/posture notes recorded at the pre-release general review (the mem side of deployment.md's accepted growth families), pinned here as the code-level economics:

• Subscribe backlog drain — the subscribe path drains the full event backlog into the receiver's pending deque before the first yield, and pending keeps growing at publish rate while a consumer is stalled (alkstore-mem/src/stream.rs:314-331 — the documented "drains to the tail then parks" shape, unbounded on the consumer side). The feed ends at engine drop; trim_to bounds the stored log, not a stalled consumer's pending deque.
• Claim scan — the claim path is an O(total-live-rows) scan with full-row clones per claim (alkstore-mem/src/queue.rs:122-181) — blessed economics for the ephemeral tier (ADR-024 §2).

Isolation and the suite factory

Each MemStore::new() yields a distinct engine instance sharing nothing (ADR-024 §4's isolation rule — the contract's Store concept amended to admit the engine-native constructor). The suite factory: a fresh store per run, teardown = drop; the mem column carries the mechanism rows that express on it, omits the engine-shaped rows that cannot (the table above), and runs under the current-thread flavor (ADR-024 §5 item 4 — the suite harness's past_stamp_sleep async sleep is the one pre-task the wave carries). Running the suite on the wasm target is scoped out behind a named collapse condition (a wasm-bindgen-test adapter crate), per ADR-024 §5.

Design Decisions

ADR	Decision	Summary
001	Crate split	§4's mem clause collapsed + amended by ADR-024 — separate crate, full contract v1
006	Wake contract	opaque wake + re-read; in-process channels as the wake substrate; no reconnection concept (no connection can drop)
007	Tx seam	the tx overlay — staging, commit-merge, drop-discard; wake-at-commit structural
008	Contract v1	full-surface implementation; §6 config split (constructor in the engine crate); §7 locks row as a TTL registry
009	Scheduler collapse	schedule table + tick + catch-up + leadership lock (trivially owned, still run — engine-uniform)
010	Queue depth	the state machine driver-free; curve engine-owned; §3's fleet predicate decides never-fleet-valid
012	Fork design	§2 one-owner rule, third owner: backoff curve, opts resolution, @every parser — three-way equivalence suite-pinned
014	Outbox tx enqueue	outbox_enqueue_tx staged in the overlay
015	Streams depth	in-memory log + per-consumer offsets; keyed publish; trim_to (wakes nothing)
016	Deployment honesty	no runtime surface — honest-ephemeral by compile-time identity + docs + matrix row; PayloadTooLarge never produced
017	Contract versioning	manifest pins core per §4; suite column is the pairing instrument
019	Handle surfaces	boxed handle traits implemented over guarded state
020	Enqueue options	opts resolution engine-owned (third copy); payload encoding via core
021	Third review round	drop = rollback trivial by overlay discard; receiver close at engine drop
022	Suite layout	factory reused verbatim — third column, born pinned
024	Mem engine charter	crate placement, honest-ephemeral posture, full tier, constructor + shared-instance rejection, wasm acceptance items

Open Questions

Open questions are tracked in open-questions.md. This spec opens none — the register is closed, and every decision this engine needed is in ADR-024. The two deliberately-not-OQ dispositions live there: the shared-instance constructor rejection (§4, re-entry = a consumer-inventory row) and the on-target wasm suite execution scope-out (§5, collapse condition = an adapter crate).