glm-5.3-flash
060f262e62
Row-first gate: consumer-inventory.md gains an engine-tier section — the mem-engine need recorded operator-authority, dated 2026-10-10 (wasm32 sandboxing across the alk protocol family + downstream ffi/napi/python-adapter economics; rusqlite/tokio-postgres structurally out on wasm, so mem is the only triad member the sandbox can carry). ADR-024 (decisions/024-mem-engine.md) pins the decision set: (1) separate `alkstore-mem` crate — discharges and amends ADR-001 §4 (superseded in part: full contract-v1 engine, not an "implementation convenience"; annotation on ADR-001 §4 + Status, class-1 window still open); (2) honest-ephemeral posture — no durability, single-process, ephemeral by design, never fleet-valid (ADR-010 §3's shared-pool predicate fails structurally), riding ADR-016's carriers unchanged (identity + docs + matrix); (3) full contract v1 tier with the asymmetry classification — `PayloadTooLarge` never produced (SQLite arm), no reconnection concept (reconnect-wake/watcher rows absent from mem's per-engine column), receiver close at engine drop, wake coalescing N/A; (4) `MemStore::new()` engine-native constructor — the one documented exception to the open-prose, core-contract.md Store concept amended; shared-instance (two opens → one engine) rejected with the ADR-016 §4-style re-entry gate (consumer-inventory row), register stays closed; (5) wasm32 — compile-clean for wasm32-unknown-unknown plus the suite's mem column green on host under a current-thread runtime flavor are wave-6 acceptance items; on-target suite execution scoped out with a named collapse condition (wasm-bindgen-test adapter crate) — a scope decision, not a pending-client hedge; (6) mem is not a core dev-dependency — core's doc examples stay mock-based, ADR-001 §4's doctest posture declines. engine-mem.md (draft): the in-process mapping — guarded per-mechanism state (no pool, no schema bootstrap, no forwarder, no spawn_blocking seam), the tx overlay (staged *_tx effects merged atomically at commit; read-your-own-writes easy, drop-=-rollback trivial; wake-at-commit structural — the seam pg needed pg-fix-tx-wake for, mem pins from birth via the existing tx-commit-atomicity rows), ADR-012 §2 third owner of the curve/opts/@every arithmetic (three-way equivalence, ADR-022's StoreFactory reused verbatim — born pinned). deployment.md: host-semantics row (single-process, ephemeral, never fleet-valid, wasm32-clean), mem connection-budget subsection (none — nothing to budget), durability-knobs row (the honest row is the absence), toolchain row (wasm32 subset + current-thread-clean, acceptance-cited). README/overview: engine-mem.md row, ADR-024 row, family table entry (alkstore-mem, no driver deps), current-state updated to the implementation phase with engine specs' stable statuses reflected. implementation.md: wave table gains wave 6 (mem engine, depends waves 1 + 5) and renumbers release readiness to wave 7 (fuzzing decision — the one remaining decided-at-implementation deferral — rides wave 7); wave-6 section records the born-pinned posture, acceptance items, the one suite-harness pre-task (past_stamp_sleep blocking sleep → async sleep, review 003's properties.rs:1683 note), and the window riders (review 003 Finding 1: the pg transient-fault seam, MEDIUM, rides wave 6; Findings 2–3 follow their park into wave 7's pre-release); wave-6 review gate defined; Decided points updated (mem discharged, fuzzing renumbered); stale wave-6 references swept across review docs and the landed tasks' notes (release-readiness items renumbered or annotated). Docs-only change; verified by cross-reference audit (all ADR/OQ/anchor references resolve, residual wave-6 mentions are correct under the new numbering or carry dated annotations).
238 lines
12 KiB
Markdown
238 lines
12 KiB
Markdown
# ADR-024: The mem engine — a third engine (`alkstore-mem`), full contract v1, honest-ephemeral
|
||
|
||
## Status
|
||
|
||
Accepted (2026-10-10, Phase 1 — [ADR-001](001-crate-split.md) §4's
|
||
implementation-time deferral collapses here, and this ADR *amends* that
|
||
§4's framing rather than merely discharging it: the mem engine is a
|
||
full contract-v1 implementation, not the "implementation convenience,
|
||
not a contract artifact" §4 originally imagined. The class-1
|
||
amend-in-place window ([ADR-017](017-contract-versioning.md) §2) is
|
||
still open — no versioned artifact exists — so that amendment is
|
||
pre-implementation. The row-first gate is satisfied: the
|
||
consumer-inventory engine-tier row (operator-authority, 2026-10-10)
|
||
precedes this decision.)
|
||
|
||
## Context
|
||
|
||
ADR-001 §4 allowed "a mem-shaped engine (in-tree or separate test
|
||
crate) … if the test story turns out to want it," deferred to
|
||
implementation time. The plan recorded the deferral as a wave-6
|
||
decision point ("or earlier if the test story demands the mem engine
|
||
sooner"). The wave-5 review gate closed with the contract suite green
|
||
on both engines, and two things now make the mem engine concrete:
|
||
|
||
- **The consumer pause + the wasm motivation.** The paused consumer
|
||
work that surfaced the need is not a test story but a consumer
|
||
story: the alk protocol crates (alkcall, alktty, alktunnels,
|
||
alksocks) are wasm-compatible at the protocol level by design, and
|
||
sandboxing — browser or otherwise — plus downstream
|
||
ffi/napi/python-adapter economics are the operator's recorded need
|
||
(the consumer-inventory engine-tier row, 2026-10-10). On those
|
||
paths the mem engine is not a hedge: rusqlite and tokio-postgres
|
||
are structurally out on wasm, so it is the *only* engine the
|
||
sandbox can carry.
|
||
- **The contract suite changed the economics.** A third engine used
|
||
to mean re-deriving a verification story; ADR-022's shared
|
||
`StoreFactory` suite means a third engine is born pinned — wiring a
|
||
third factory column onto already-authored, version-stamped rows.
|
||
And the class-1 amendment window (ADR-017 §2) is still open: any
|
||
contract wobble a third full implementation exposes is
|
||
amend-in-place now, a migration after first release.
|
||
|
||
The scale is honest to state up front: this is a *third full engine*,
|
||
not alkblobs' mem tier (a `BTreeMap` behind a handful of CAS
|
||
methods). It implements the queue state machine (visibility/reclaim/
|
||
dead-letter/sweep), the scheduler (boundaries, catch-up, leadership),
|
||
the tx seam, streams with per-consumer offsets and trim, TTL locks,
|
||
and wakes. What it does *not* carry is the machinery that made the
|
||
Postgres wave build-heavy: no pooling, no schema bootstrap, no
|
||
forwarder, no `spawn_blocking` seam, no reconnection concept.
|
||
|
||
## Decision
|
||
|
||
### 1. Crate placement — a separate `alkstore-mem` crate
|
||
|
||
The mem engine is a **separate workspace member**, `alkstore-mem`,
|
||
implementing the core traits. Dependency surface: `alkstore` plus
|
||
tokio (the wasm-supported subset — sync and time), no driver
|
||
dependencies; payload encoding rides core's `encode_payload`
|
||
([ADR-020](020-enqueue-opt-semantics-and-bridges.md) §4). This
|
||
discharges ADR-001 §4's "in-tree or separate crate" fork as
|
||
**separate**, for the reason that ADR stated in reverse for the other
|
||
engines: engine crates are additive by construction (ADR-001 §
|
||
Consequences), and a mem engine in core would contradict ADR-001 §1's
|
||
contents list (core is the contract artifact and nothing else).
|
||
[ADR-013](013-fold-substrate-into-sqlite.md)'s "no fourth crate"
|
||
rationale does not apply — that ruled the *substrate's* placement
|
||
(fork lineage, versioning surface, link collisions); none of those
|
||
facts attach to a greenfield in-memory implementation.
|
||
|
||
### 2. Maturity posture — honest-ephemeral, stated by the family's carriers
|
||
|
||
The mem engine's posture is the honest middle between "test
|
||
convenience" (ADR-001 §4's original framing — *superseded*) and
|
||
"durable production engine": **no durability, single-process,
|
||
ephemeral by design.** Concretely:
|
||
|
||
- A **legitimate production choice for workloads that accept
|
||
ephemerality** — the consumer's dependency on `alkstore-mem` *is*
|
||
the ephemerality declaration (the ADR-016 pattern: the dependency
|
||
edge states the truth and cannot drift from it).
|
||
- **Never fleet-valid** — the fleet predicate ([ADR-010](010-queue-semantics-depth.md)
|
||
§3's shared-pool rule) fails structurally: there is no pool, no
|
||
process boundary, nothing shared. Two `MemStore` instances share
|
||
nothing by design (§4).
|
||
- Honesty rides the [ADR-016](016-deployment-honesty.md) carriers
|
||
unchanged — no new surface: compile-time engine identity (the
|
||
crate name), the engine crate's docs as the posture statement, and
|
||
the [deployment matrix](../deployment.md) gaining the ephemeral
|
||
row. The compile-time identity carries the honesty; nothing runtime
|
||
is fabricated to detect what the dependency already declares.
|
||
- alkblobs' mem line ("never a production story, never fleet-valid")
|
||
is deliberately *not* copied verbatim — "never production" would
|
||
contradict the sandbox use case that motivates this engine.
|
||
|
||
### 3. Full contract v1 — tier and asymmetry classification
|
||
|
||
The tier is **full contract v1**: every mechanism, every guarantee
|
||
row, no reduced profile. A reduced test-only engine never gets
|
||
born-pinned (the suite's equivalence rows would be absent from its
|
||
column, exactly the drift the suite exists to prevent), and the
|
||
motivating need is consumer-shaped, not test-shaped.
|
||
|
||
The documented engine asymmetries classify as:
|
||
|
||
- **`PayloadTooLarge`**: never produced, at any size — the SQLite
|
||
occurrence posture. The variant stays contract-uniform and
|
||
matchable ([ADR-016](016-deployment-honesty.md) §5); mem's suite
|
||
column wires the non-occurrence arm.
|
||
- **Reconnection**: no such concept — no connection can drop. The
|
||
reconnect-wake and watcher-death rows are pg/SQLite-shaped and are
|
||
*per-engine wired* in the suite, so mem's column simply does not
|
||
carry them; nothing weakens across engines.
|
||
- **Receiver close**: the one close path is engine drop (process
|
||
exit/shutdown) closing open receivers — the contract's close-arm
|
||
shapes apply, engine-realized as drop.
|
||
- **Wake coalescing**: not applicable — wakes deliver at commit
|
||
through in-process channels directly to each subscriber; there is
|
||
no batching boundary to coalesce across. The engine spec pins the
|
||
delivery shape.
|
||
|
||
### 4. Constructor shape — engine-native, isolated per open; shared instance rejected
|
||
|
||
Constructors live in engine crates ([ADR-008](008-contract-v1-pinning.md)
|
||
§6), so the mem engine's constructor is engine-native: **`MemStore::new()`**
|
||
— no connection string, no file path. This is the one documented
|
||
exception to the core contract's open-prose ("connection string or
|
||
file path"), and the prose is amended to name the exception
|
||
(pre-implementation, class-1). **Isolation per open**: each
|
||
`MemStore::new()` yields a distinct engine instance sharing nothing —
|
||
exactly the suite factory's isolation contract; teardown is drop.
|
||
|
||
**A shared-instance constructor (two opens → one engine) is rejected
|
||
for v1, with a re-entry gate**: it introduces real concurrency
|
||
questions (claims and locks contended across two handles) that no
|
||
consumer row names. The re-entry condition is a consumer-inventory
|
||
row naming the two-opens-one-engine need — the ADR-016 §4 pattern
|
||
(decision against, gate named); the open-questions register stays
|
||
closed.
|
||
|
||
### 5. Wasm32 — compile-clean is an acceptance item; on-target suite execution is scoped out with a named collapse condition
|
||
|
||
The wasm motivation makes wasm compatibility load-bearing, so it
|
||
moves from aspiration to **wave-6 acceptance item**:
|
||
|
||
1. `alkstore-mem` **compiles clean for `wasm32-unknown-unknown`**;
|
||
2. the workspace host gates stay green (`cargo build` / `test` /
|
||
clippy `-D warnings` / fmt);
|
||
3. the contract-suite mem column passes on the host;
|
||
4. the mem column runs under a **current-thread runtime flavor** —
|
||
the test that keeps internals current-thread-clean, which is what
|
||
wasm needs (no `spawn_blocking` anywhere, tokio's sync + time
|
||
subset only). One suite-harness pre-task follows from this: the
|
||
suite's `past_stamp_sleep` blocking sleep (the review-003 posture
|
||
note, `properties.rs:1683`) becomes an async sleep — the only
|
||
suite-side change the wave carries.
|
||
|
||
Running the *suite itself* on the wasm target (wasm-bindgen-test
|
||
harnessing) is a different investment and is **scoped out of wave 6**
|
||
with a named collapse condition: **a wasm-bindgen-test adapter crate
|
||
for the suite existing**. This is a scope decision with a definite
|
||
trigger, not a pending-client hedge — the acceptance the sandbox and
|
||
adapter motivation actually needs (compile-clean, host-verified
|
||
contract conformance) is delivered by items 1–4 and independently
|
||
verifiable.
|
||
|
||
### 6. What the mem engine is not
|
||
|
||
- **Not a core dev-dependency.** ADR-001 §4's "doctests" posture
|
||
declines: core takes no engine dev-dependency (the core crate has
|
||
no driver dependencies, ever, dev-side included); core contract
|
||
doc examples remain mock-store-based; mem's doctests live in the
|
||
mem crate itself.
|
||
- **Not a re-derivation of the suite** — ADR-022's factory is reused
|
||
verbatim (its consequences already anticipated this).
|
||
- **Not new contract surface.** The contract text changes here are
|
||
the core-prose exception note (§4) and documentation rows (§2) —
|
||
amend-in-place pre-release; the trait surface is untouched.
|
||
|
||
## Consequences
|
||
|
||
**Positive**
|
||
|
||
- The wasm/sandbox and downstream-adapter paths gain their engine;
|
||
every future family consumer that must run in a sandbox can depend
|
||
on `alkstore-mem` without inventing one.
|
||
- A third full implementation stresses contract v1 while the class-1
|
||
window is open — born pinned against the finished suite rather
|
||
than pinned retroactively; any wobble is amend-in-place now.
|
||
- The suite's arithmetic-equivalence pinning goes three-way (backoff
|
||
curve, opts resolution — [ADR-012](012-forked-substrate-design.md)
|
||
§2's one-owner rule applied per engine, third owner).
|
||
- Zero new contract surface; the ADR-016 carriers absorb the posture
|
||
(identity + docs + matrix row), no runtime descriptor.
|
||
|
||
**Negative**
|
||
|
||
- The ADR-017 lockstep duty now covers three engines: a class-2
|
||
(additive) contract change needs three adoption releases; a third
|
||
engine's tests/opt machinery ride every wave forever.
|
||
- Engine-owned arithmetic gains a third owner (backoff curve, opts
|
||
resolution, the `@every` parser) — three copies to keep honest, and
|
||
the honesty mechanism is the suite's equivalence rows, not shared
|
||
code.
|
||
- The mem engine's maintenance is *real* engine maintenance (the
|
||
queue/scheduler/stream/lock machinery), not a weekend adapter —
|
||
the honest-scale flag this ADR's Context records.
|
||
|
||
## References
|
||
|
||
- [ADR-001](001-crate-split.md) §4 — the deferral this ADR collapses;
|
||
§4's "implementation convenience, not a contract artifact" framing
|
||
superseded (amend-in-place, class-1 window open).
|
||
- [ADR-022](022-contract-suite-layout.md) — the shared `StoreFactory`
|
||
suite reused verbatim; the born-pinned mechanism.
|
||
- [ADR-017](017-contract-versioning.md) — §2's class-1 window (still
|
||
open, what makes this amendment cheap); §4's pairing carriers the
|
||
mem engine adopts (manifest pin, suite column, engine docs).
|
||
- [ADR-016](016-deployment-honesty.md) — the carriers the
|
||
honest-ephemeral posture rides (§2); the compile-time-identity
|
||
honesty pattern (§3); §4's rejection-with-gate pattern §4 uses; §5
|
||
the `PayloadTooLarge` pinning mem's column wires.
|
||
- [ADR-010](010-queue-semantics-depth.md) §3 — the fleet predicate
|
||
that decides "never fleet-valid" structurally.
|
||
- [ADR-008](008-contract-v1-pinning.md) §6 — constructors live in
|
||
engine crates (the `MemStore::new()` home).
|
||
- [ADR-012](012-forked-substrate-design.md) §2 — the one-owner rule
|
||
the engine-owned arithmetic applies per engine.
|
||
- `docs/research/consumer-inventory.md` §Engine-tier inventory — the
|
||
row-first record (operator-authority, 2026-10-10) that precedes
|
||
this decision.
|
||
- Review 003 (`docs/reviews/003-wave-5-general-review.md`) — the
|
||
`properties.rs:1683` current-thread posture note the suite-harness
|
||
pre-task disposes.
|
||
- `docs/architecture/engine-mem.md` — the engine spec this ADR
|
||
requires as a wave-6 deliverable.
|
||
- `docs/plans/implementation.md` — the wave-6 insertion (release
|
||
readiness renumbered to wave 7). |