--- id: mem-engine-scheduler-outbox name: Mem engine — scheduler (`schedule`/`unschedule`/`run_schedules`) and outbox helper status: completed depends_on: [mem-engine-queues, mem-engine-locks] scope: moderate risk: medium impact: component level: implementation tags: [wave-6, mem-engine, scheduler, outbox] --- ## Description The collapsed scheduler and the outbox helper ([ADR-009](../architecture/decisions/009-scheduler-collapse.md), [ADR-014](../architecture/decisions/014-outbox-tx-enqueue.md)): - **Scheduler surface**: `schedule(name, spec, queue, payload, opts)` (upsert by name; *both* name-bearing arguments validated — schedule name and queue argument identically, `InvalidName`/`ReservedName`), `unschedule(name) -> bool`, and the opt-in `run_schedules(stop: StopToken)` runner over a mem-owned tick task. - **Tick machinery — engine-uniform** ([ADR-009](../architecture/decisions/009-scheduler-collapse.md) §4's rule; single-instance leadership is trivially owned but the machinery still runs): each boundary fire enqueues ordinary work stamped with the schedule's `ScheduleOpts` resolved over the target queue's derived engine defaults; fire+advance is atomic under the guarded state (the no-double-fire floor); **bounded catch-up with the fixed 64-boundary cap and skip-forward past the cap** (mem-owned tick implementation — third owner per [ADR-012](../architecture/decisions/012-forked-substrate-design.md) §2); leadership rides the reserved `__alkstore_scheduler` lock through the lock mechanism (so two runners on one instance contend through the same machinery the other engines use; loss → `Err(LeadershipLost)`, clean stop → `Ok(())`). - **Spec grammar v1**: `@every ` (`s|m|h|d`) only — mem owns its parser (third copy, the grammar is small and pinned); cron strings rejected (`InvalidSpec`). Boundary waits ride tokio time. - **Outbox helper**: `Outbox { enqueue, run_once }` per [ADR-019](../architecture/decisions/019-mechanism-handle-surfaces.md) §5 — `run_once(worker_id, delivery) -> bool` is pull: claim one, ack on `Ok`, `retry(err, None)` on `Err`; **no engine-issued heartbeat inside delivery** (inherited posture); `outbox_enqueue_tx` staged in the overlay by `mem-engine-seam-tx` (the 60/5/5 stamp set); the derived backing queue `__alkstore_outbox:{outbox}` unreachable by `enqueue_tx` (validation already pins the reserved prefix), name format pinned identical to the other engines (the suite's derived-queue-naming row composes it). Unit tests pin: boundary fires (deterministic, tokio-time driven), bounded catch-up + skip-forward, leadership lock contention/loss through two `run_schedules` calls, outbox run_once ack/retry shapes, derived-name format. ## Acceptance Criteria - [x] Scheduler surface + tick: boundary fire, catch-up (≤64), skip-forward — unit-pinned with tokio-time determinism - [x] Leadership contention through the `__alkstore_scheduler` lock; `LeadershipLost` on loss, `Ok(())` on clean stop — pinned - [x] Schedule name + queue argument validation; `@every`-only parser (cron → `InvalidSpec`) — pinned - [x] Outbox: `enqueue`/`run_once` semantics (no engine heartbeat); derived backing-queue name format identical to both engines — pinned - [x] `cargo check --target wasm32-unknown-unknown -p alkstore-mem` clean; workspace gates green ## References - docs/architecture/queues.md; core-contract.md §scheduler, §outbox - docs/architecture/decisions/009-scheduler-collapse.md §1/§3/§4/§6; 014-outbox-tx-enqueue.md; 019 §4/§5; 020 §3 - docs/architecture/engine-mem.md §Mapping the contract (scheduler/outbox row) - tasks/pg-engine-scheduler-outbox.md (the structural twin's task of record — tick/catch-up/leadership shapes) ## Notes > Filled by implementation agent, 2026-10-10. 1. **The implementation already landed in the seam task** — the same all-or-nothing trait-impl disposition the previous wave-6 mechanism tasks recorded: the scheduler (`scheduler.rs`'s register table, `run_tick`, the runner) and the outbox helper (`outbox.rs`) were implemented with the seam's whole-engine scope; this task found its mechanisms *implemented*. 2. **Coverage reconciliation**: already pinned by the seam suite — the boundary-fire row (deterministic `run_tick` fires, ready-at- fire-instant rows stamped with the registered values + plain defaults, no-double-fire floor, fire+advance as one guarded op), the bounded-catch-up row (600 s behind → exactly 64 fires, never 300; skip-forward lands the next boundary strictly future with the grid honest; never-due schedules untouched), the leadership rows (two contenders through the `__alkstore_scheduler` lock: one wins, the loser `Err(LeadershipLost)`; a blocked lock loses before any fire; clean stop → `Ok(())` and the lease released for the next runner), the register rows (upsert-by-name + stamp resolution + read-back; both name-bearing arguments validated + cron rejected pre-storage with nothing written; the `@every` grammar matrix), `unschedule` (removal + absent-false + reserved-name rejection), `fired_rows_are_ordinary_queue_work` (the fire claims through the ordinary queue), the outbox rows (Ok ⇒ ack / Err ⇒ retry with the queue's equal-jitter curve and the delivery error riding the exhaust path onto the dead row; the derived 60/5/5 stamps with `max_attempts` the one override; no-work `Ok(false)`; empty `worker_id` rejected), and the derived backing-queue name format + `enqueue_tx` unreachability. 3. **What this task added** — the one unpinned posture from the description: `run_once_issues_no_engine_heartbeat_in_delivery` — the handover carries the raw claim deadline (no engine-side renewal before or inside delivery), and the boxed [`JobHandle`]'s own `heartbeat` remains available to the closure (renewal is the consumer's obligation, ADR-019 §5). The delivery fixture grew `handover_deadline` + a manual-heartbeat arm for it. 4. **No production code changed** — verified correct as-found; tests only (+1 in `outbox.rs`, fixture widened, 100 mem tests total). The scheduler's boundary waits ride realtime tokio time with stamps on the engine clock; the tick/catch-up/leadership pins drive the crate-internal `run_tick`/`run_schedules` directly, so determinism holds without realtime waits. ## Summary > Filled on completion, 2026-10-10. Completed the mem engine's scheduler/outbox task against the seam task's foundation: reconciled the description's pin list against `alkstore-mem/src/scheduler.rs` + `outbox.rs`'s existing suite (14 pre-existing tests), found every tick/catch-up/leadership/validation/ outbox shape already pinned, and added the one gap — the no-engine-issued-heartbeat posture pinned end-to-end (raw claim deadline at handover; the closure's own heartbeat path available and honored). Acceptance criteria: scheduler surface + tick determinism ✓ (seam rows), leadership ✓ (seam rows), validation + grammar ✓ (seam rows), outbox semantics + derived name format ✓ (seam rows + the new heartbeat row), gates ✓. Verified: `cargo test -p alkstore-mem` 100/100 (99 + 1 new, new test green solo), workspace `cargo test` green, `cargo clippy --all-targets -- -D warnings` clean, `cargo fmt --check` clean, `cargo check --target wasm32-unknown-unknown -p alkstore-mem` clean.