glm-5.3-flash
f2f5e03d7c
Review 004 items 10+11: the two recorded mem growth postures (subscribe backlog drain unbounded on the consumer side; O(live-rows) claim scan with full-row clones) and the outbox's 'processed means delivered-or-reclaimable' semantic, pinned in engine-mem.md. The debug_assert arm was declined: the dropped ack/retry booleans also refuse benignly on legitimate in-closure handle ops, so it cannot separate a visibility lapse from a deliberate first-party outcome.
189 lines
12 KiB
Markdown
189 lines
12 KiB
Markdown
---
|
||
status: stable
|
||
last_updated: "2026-10-11 (review-004 items 10+11 pinned: the growth
|
||
postures section — subscribe backlog drain, O(live-rows) claim scan —
|
||
and the outbox's delivered-or-reclaimable semantic; the wave-6
|
||
review gate had passed earlier — conformance read clean, three
|
||
edge-domain fixes landed inline (the stream cursor read's `i64::MAX`
|
||
overflow, `trim_to`'s horizon endpoint, the scheduler register-time
|
||
interval overflow now rejecting with `InvalidSpec`), the third
|
||
column's 25-row set green under the current-thread flavor, the
|
||
wasm32 gate + feature subset verified, the rider's arms verified)"
|
||
---
|
||
|
||
# Mem engine
|
||
|
||
The `alkstore-mem` engine implements
|
||
[core-contract.md](core-contract.md) on guarded in-memory structures.
|
||
This spec records WHAT the engine is — its architecture, the tx
|
||
overlay, the asymmetry classification — not code-level HOW. Decisions
|
||
live in ADRs ([ADR-024](decisions/024-mem-engine.md) is this engine's
|
||
charter); contract obligations live in the core spec.
|
||
|
||
## Identity and posture
|
||
|
||
- Separate workspace crate (ADR-024 §1): `alkstore`, tokio (sync +
|
||
time — the wasm-supported subset), no driver dependencies. Natively
|
||
async; no `spawn_blocking` seam anywhere; internals are
|
||
current-thread-runtime-clean by construction, and the suite's mem
|
||
column runs *under a current-thread flavor* to keep that honest.
|
||
- **Honest-ephemeral** (ADR-024 §2): no durability, single-process,
|
||
ephemeral by design; legitimate production for workloads that
|
||
accept ephemerality; never fleet-valid (the
|
||
[ADR-010](decisions/010-queue-semantics-depth.md) §3 shared-pool
|
||
predicate fails structurally — there is no pool). The dependency
|
||
edge on `alkstore-mem` is the ephemerality declaration; see
|
||
[deployment.md](deployment.md).
|
||
- **Full contract v1** (ADR-024 §3): every mechanism, every guarantee
|
||
row. Born pinned — the engine lands against the finished
|
||
version-stamped suite ([ADR-022](decisions/022-contract-suite-layout.md)'s
|
||
`StoreFactory`, reused verbatim as a third factory column), which
|
||
is also what pins its engine-owned arithmetic three-way
|
||
([ADR-012](decisions/012-forked-substrate-design.md) §2's one-owner
|
||
rule: the backoff curve, opts resolution, and the `@every` parser
|
||
get mem's own implementations; equivalence with the other two
|
||
engines is suite-pinned).
|
||
- wasm32: `alkstore-mem` compiling clean for
|
||
`wasm32-unknown-unknown` is an acceptance item of its wave
|
||
(ADR-024 §5) and an identity line of this spec.
|
||
|
||
## Architecture
|
||
|
||
Everything the other engines realize with connections, schemas, and
|
||
daemons, the mem engine realizes with process-owned state:
|
||
|
||
- **No connections at all** — no pool (pg's), no writer/reader slots
|
||
(SQLite's), no listener. The store instance *is* the state: guarded
|
||
per-mechanism tables (queues + jobs, dead letters, streams +
|
||
offsets, locks, schedules) owned by the engine instance.
|
||
- **No background machinery to bootstrap** — no schema bootstrap, no
|
||
watcher thread (nothing cross-connection to poll), no forwarder.
|
||
The only async machinery is tokio-time-driven: lock TTLs, the
|
||
scheduler's boundary waits, and wake delivery. All timestamps read
|
||
the process clock through one internal accessor — no server clock
|
||
to defer to, no per-mechanism clock reads (the wasm-posture note,
|
||
[ADR-024](decisions/024-mem-engine.md) §5).
|
||
- **Wake substrate**: in-process channels. Wakes deliver at the
|
||
commit boundary directly to each open receiver — no batching
|
||
boundary exists, so coalescing is not applicable; per-subscriber
|
||
delivery order is FIFO by commit order.
|
||
|
||
### The tx overlay
|
||
|
||
The tx seam's mem realization: a `begin_tx()` allocates an **overlay**
|
||
private to the caller's handle.
|
||
|
||
- `*_tx` methods stage effects in the overlay (enqueues, publishes,
|
||
notifies); reads *inside* the tx merge the overlay over committed
|
||
state — read-your-own-writes with no round-trips.
|
||
- **Commit** merges the overlay atomically (a single guarded merge)
|
||
and fires the overlay's pending wakes — *wake-at-commit* is the
|
||
discipline-critical property: a notify staged in the tx delivers
|
||
nothing until the overlay merges, exactly the seam the pg engine
|
||
had to fix retroactively (`pg-fix-tx-wake`) and the suite's
|
||
tx-commit-atomicity rows pin.
|
||
- **Drop** discards the overlay — drop = rollback with no ghosts,
|
||
trivially, because uncommitted state never touched committed state.
|
||
- No isolation questions can arise across handles: an overlay is
|
||
invisible to every other store/view until its commit.
|
||
|
||
## Mapping the contract
|
||
|
||
| Contract piece | Engine realization |
|
||
|---|---|
|
||
| notify / listen | staged notify in the tx overlay; delivered at commit to open receivers. Commit-atomicity is structural ([ADR-007](decisions/007-transactional-seam.md)) — the merge and the wake fire are one operation |
|
||
| streams | in-memory event log + per-consumer offsets, global-FIFO by monotone offset; keyed publish rides the overlay ([ADR-015](decisions/015-streams-depth.md)); `trim_to` mutates the log and wakes nothing |
|
||
| queues | the ADR-010 state machine over an in-memory job table (visibility, reclaim, the equal-jitter curve engine-owned, dead-letter move, the no-stranded-rows sweep) — all driver-free, swept on caller-driven `sweep_expired` |
|
||
| named locks | in-memory TTL registry ([ADR-008](decisions/008-contract-v1-pinning.md) §7's row); expiry via tokio time; single-instance contention semantics |
|
||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule table, tick (boundary advance + 64-boundary catch-up cap), leadership on the `__alkstore_scheduler` lock — single-instance leadership is trivially owned but the lock machinery still runs, per ADR-009 §4's engine-uniform rule; outbox = helper over queues ([ADR-014](decisions/014-outbox-tx-enqueue.md)) |
|
||
| begin_tx | overlay allocation ([ADR-007](decisions/007-transactional-seam.md)) |
|
||
| handle ops | direct `.await`s on guarded state; `encode_payload` via core (ADR-020 §4) |
|
||
|
||
### Outbox semantics
|
||
|
||
The outbox is a helper over queues ([ADR-014](decisions/014-outbox-tx-enqueue.md)):
|
||
`run_once` claims one job through the ordinary claim machinery and
|
||
applies the outcome when delivery returns — `Ok` ⇒ ack, `Err` ⇒
|
||
retry on the queue's own curve. The ack/retry application's
|
||
row-ref booleans are dropped: a delivery that outlived its
|
||
visibility window (or a consumer side-effect that handled the row
|
||
first, e.g. an in-closure `ack`/`retry` on the boxed handle) leaves
|
||
the outer outcome unapplied, and `run_once` still reports `true`.
|
||
Inherent to the no-heartbeat outbox design — at-least-once reclaim
|
||
is the recovery, not a second report. **Processed means
|
||
delivered-or-reclaimable**: `true` attests the claim was processed,
|
||
not that the ack/exhaust landed; renewal inside the window is the
|
||
consumer's obligation ([ADR-019](decisions/019-mechanism-handle-surfaces.md)
|
||
§5) and reclaim closes a lapsed one.
|
||
|
||
## Asymmetry classification (ADR-024 §3)
|
||
|
||
| Asymmetry | Mem posture |
|
||
|---|---|
|
||
| `PayloadTooLarge` | never produced, any size (SQLite occurrence arm); the non-occurrence arm wires in mem's suite column ([ADR-016](decisions/016-deployment-honesty.md) §5) |
|
||
| Reconnection | does not exist — no connection can drop; the reconnect-wake and watcher-death rows are pg/SQLite-shaped and absent from mem's column (rows are per-engine wired; nothing cross-wires or weakens) |
|
||
| Receiver close | one close path: engine drop closes open receivers (process exit/shutdown); contract close-arm shapes apply unchanged ([ADR-021](decisions/021-tx-reads-and-value-shape-fixes.md)) |
|
||
| Wake coalescing | not applicable — direct at-commit delivery, per-subscriber FIFO; the engine spec line above pins it |
|
||
|
||
## Growth postures (review 004 items 10–11)
|
||
|
||
Two perf/posture notes recorded at the pre-release general review
|
||
(the mem side of [deployment.md](deployment.md)'s accepted growth
|
||
families), pinned here as the code-level economics:
|
||
|
||
- **Subscribe backlog drain** — the subscribe path drains the full
|
||
event backlog into the receiver's pending deque *before the first
|
||
yield*, and pending keeps growing at publish rate while a consumer
|
||
is stalled (`alkstore-mem/src/stream.rs:314-331` — the documented
|
||
"drains to the tail then parks" shape, unbounded on the consumer
|
||
side). The feed ends at engine drop; `trim_to` bounds the stored
|
||
log, not a stalled consumer's pending deque.
|
||
- **Claim scan** — the claim path is an O(total-live-rows) scan with
|
||
full-row clones per claim (`alkstore-mem/src/queue.rs:122-181`) —
|
||
blessed economics for the ephemeral tier ([ADR-024](decisions/024-mem-engine.md)
|
||
§2).
|
||
|
||
## Isolation and the suite factory
|
||
|
||
Each `MemStore::new()` yields a distinct engine instance sharing
|
||
nothing (ADR-024 §4's isolation rule — the contract's *Store* concept
|
||
amended to admit the engine-native constructor). The suite factory:
|
||
a fresh store per run, teardown = drop; the mem column carries the
|
||
mechanism rows that express on it, omits the engine-shaped rows that
|
||
cannot (the table above), and runs under the current-thread flavor
|
||
(ADR-024 §5 item 4 — the suite harness's `past_stamp_sleep` async
|
||
sleep is the one pre-task the wave carries). Running the suite
|
||
*on* the wasm target is scoped out behind a named collapse condition
|
||
(a wasm-bindgen-test adapter crate), per ADR-024 §5.
|
||
|
||
## Design Decisions
|
||
|
||
| ADR | Decision | Summary |
|
||
|---|---|---|
|
||
| [001](decisions/001-crate-split.md) | Crate split | §4's mem clause collapsed + amended by [ADR-024](decisions/024-mem-engine.md) — separate crate, full contract v1 |
|
||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | opaque wake + re-read; in-process channels as the wake substrate; no reconnection concept (no connection can drop) |
|
||
| [007](decisions/007-transactional-seam.md) | Tx seam | the tx overlay — staging, commit-merge, drop-discard; wake-at-commit structural |
|
||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | full-surface implementation; §6 config split (constructor in the engine crate); §7 locks row as a TTL registry |
|
||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule table + tick + catch-up + leadership lock (trivially owned, still run — engine-uniform) |
|
||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | the state machine driver-free; curve engine-owned; §3's fleet predicate decides never-fleet-valid |
|
||
| [012](decisions/012-forked-substrate-design.md) | Fork design | §2 one-owner rule, third owner: backoff curve, opts resolution, `@every` parser — three-way equivalence suite-pinned |
|
||
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` staged in the overlay |
|
||
| [015](decisions/015-streams-depth.md) | Streams depth | in-memory log + per-consumer offsets; keyed publish; `trim_to` (wakes nothing) |
|
||
| [016](decisions/016-deployment-honesty.md) | Deployment honesty | no runtime surface — honest-ephemeral by compile-time identity + docs + matrix row; `PayloadTooLarge` never produced |
|
||
| [017](decisions/017-contract-versioning.md) | Contract versioning | manifest pins core per §4; suite column is the pairing instrument |
|
||
| [019](decisions/019-mechanism-handle-surfaces.md) | Handle surfaces | boxed handle traits implemented over guarded state |
|
||
| [020](decisions/020-enqueue-opt-semantics-and-bridges.md) | Enqueue options | opts resolution engine-owned (third copy); payload encoding via core |
|
||
| [021](decisions/021-tx-reads-and-value-shape-fixes.md) | Third review round | drop = rollback trivial by overlay discard; receiver close at engine drop |
|
||
| [022](decisions/022-contract-suite-layout.md) | Suite layout | factory reused verbatim — third column, born pinned |
|
||
| [024](decisions/024-mem-engine.md) | Mem engine charter | crate placement, honest-ephemeral posture, full tier, constructor + shared-instance rejection, wasm acceptance items |
|
||
|
||
## Open Questions
|
||
|
||
Open questions are tracked in
|
||
[open-questions.md](open-questions.md). This spec opens none — the
|
||
register is closed, and every decision this engine needed is in
|
||
[ADR-024](decisions/024-mem-engine.md). The two deliberately-not-OQ
|
||
dispositions live there: the shared-instance constructor rejection
|
||
(§4, re-entry = a consumer-inventory row) and the on-target wasm
|
||
suite execution scope-out (§5, collapse condition = an adapter
|
||
crate). |