144 lines
7.3 KiB
Markdown
144 lines
7.3 KiB
Markdown
---
|
|
id: mem-engine-scheduler-outbox
|
|
name: Mem engine — scheduler (`schedule`/`unschedule`/`run_schedules`) and outbox helper
|
|
status: completed
|
|
depends_on: [mem-engine-queues, mem-engine-locks]
|
|
scope: moderate
|
|
risk: medium
|
|
impact: component
|
|
level: implementation
|
|
tags: [wave-6, mem-engine, scheduler, outbox]
|
|
---
|
|
|
|
## Description
|
|
|
|
The collapsed scheduler and the outbox helper
|
|
([ADR-009](../architecture/decisions/009-scheduler-collapse.md),
|
|
[ADR-014](../architecture/decisions/014-outbox-tx-enqueue.md)):
|
|
|
|
- **Scheduler surface**: `schedule(name, spec, queue, payload, opts)`
|
|
(upsert by name; *both* name-bearing arguments validated — schedule
|
|
name and queue argument identically, `InvalidName`/`ReservedName`),
|
|
`unschedule(name) -> bool`, and the opt-in
|
|
`run_schedules(stop: StopToken)` runner over a mem-owned tick task.
|
|
- **Tick machinery — engine-uniform** ([ADR-009](../architecture/decisions/009-scheduler-collapse.md)
|
|
§4's rule; single-instance leadership is trivially owned but the
|
|
machinery still runs): each boundary fire enqueues ordinary work
|
|
stamped with the schedule's `ScheduleOpts` resolved over the target
|
|
queue's derived engine defaults; fire+advance is atomic under the
|
|
guarded state (the no-double-fire floor); **bounded catch-up with
|
|
the fixed 64-boundary cap and skip-forward past the cap**
|
|
(mem-owned tick implementation — third owner per
|
|
[ADR-012](../architecture/decisions/012-forked-substrate-design.md)
|
|
§2); leadership rides the reserved `__alkstore_scheduler` lock
|
|
through the lock mechanism (so two runners on one instance contend
|
|
through the same machinery the other engines use; loss →
|
|
`Err(LeadershipLost)`, clean stop → `Ok(())`).
|
|
- **Spec grammar v1**: `@every <n><unit>` (`s|m|h|d`) only — mem owns
|
|
its parser (third copy, the grammar is small and pinned); cron
|
|
strings rejected (`InvalidSpec`). Boundary waits ride tokio time.
|
|
- **Outbox helper**: `Outbox { enqueue, run_once }` per
|
|
[ADR-019](../architecture/decisions/019-mechanism-handle-surfaces.md)
|
|
§5 — `run_once(worker_id, delivery) -> bool` is pull: claim one,
|
|
ack on `Ok`, `retry(err, None)` on `Err`; **no engine-issued
|
|
heartbeat inside delivery** (inherited posture);
|
|
`outbox_enqueue_tx` staged in the overlay by `mem-engine-seam-tx`
|
|
(the 60/5/5 stamp set); the derived backing queue
|
|
`__alkstore_outbox:{outbox}` unreachable by `enqueue_tx`
|
|
(validation already pins the reserved prefix), name format pinned
|
|
identical to the other engines (the suite's derived-queue-naming
|
|
row composes it).
|
|
|
|
Unit tests pin: boundary fires (deterministic, tokio-time driven),
|
|
bounded catch-up + skip-forward, leadership lock contention/loss
|
|
through two `run_schedules` calls, outbox run_once ack/retry shapes,
|
|
derived-name format.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [x] Scheduler surface + tick: boundary fire, catch-up (≤64),
|
|
skip-forward — unit-pinned with tokio-time determinism
|
|
- [x] Leadership contention through the `__alkstore_scheduler` lock;
|
|
`LeadershipLost` on loss, `Ok(())` on clean stop — pinned
|
|
- [x] Schedule name + queue argument validation; `@every`-only parser
|
|
(cron → `InvalidSpec`) — pinned
|
|
- [x] Outbox: `enqueue`/`run_once` semantics (no engine heartbeat);
|
|
derived backing-queue name format identical to both engines —
|
|
pinned
|
|
- [x] `cargo check --target wasm32-unknown-unknown -p alkstore-mem`
|
|
clean; workspace gates green
|
|
|
|
## References
|
|
|
|
- docs/architecture/queues.md; core-contract.md §scheduler, §outbox
|
|
- docs/architecture/decisions/009-scheduler-collapse.md §1/§3/§4/§6;
|
|
014-outbox-tx-enqueue.md; 019 §4/§5; 020 §3
|
|
- docs/architecture/engine-mem.md §Mapping the contract
|
|
(scheduler/outbox row)
|
|
- tasks/pg-engine-scheduler-outbox.md (the structural twin's task of
|
|
record — tick/catch-up/leadership shapes)
|
|
|
|
## Notes
|
|
|
|
> Filled by implementation agent, 2026-10-10.
|
|
|
|
1. **The implementation already landed in the seam task** — the same
|
|
all-or-nothing trait-impl disposition the previous wave-6 mechanism
|
|
tasks recorded: the scheduler (`scheduler.rs`'s register table,
|
|
`run_tick`, the runner) and the outbox helper (`outbox.rs`) were
|
|
implemented with the seam's whole-engine scope; this task found its
|
|
mechanisms *implemented*.
|
|
2. **Coverage reconciliation**: already pinned by the seam suite —
|
|
the boundary-fire row (deterministic `run_tick` fires, ready-at-
|
|
fire-instant rows stamped with the registered values + plain
|
|
defaults, no-double-fire floor, fire+advance as one guarded op),
|
|
the bounded-catch-up row (600 s behind → exactly 64 fires, never
|
|
300; skip-forward lands the next boundary strictly future with the
|
|
grid honest; never-due schedules untouched), the leadership rows
|
|
(two contenders through the `__alkstore_scheduler` lock: one wins,
|
|
the loser `Err(LeadershipLost)`; a blocked lock loses before any
|
|
fire; clean stop → `Ok(())` and the lease released for the next
|
|
runner), the register rows (upsert-by-name + stamp resolution +
|
|
read-back; both name-bearing arguments validated + cron rejected
|
|
pre-storage with nothing written; the `@every` grammar matrix),
|
|
`unschedule` (removal + absent-false + reserved-name rejection),
|
|
`fired_rows_are_ordinary_queue_work` (the fire claims through the
|
|
ordinary queue), the outbox rows (Ok ⇒ ack / Err ⇒ retry with the
|
|
queue's equal-jitter curve and the delivery error riding the
|
|
exhaust path onto the dead row; the derived 60/5/5 stamps with
|
|
`max_attempts` the one override; no-work `Ok(false)`; empty
|
|
`worker_id` rejected), and the derived backing-queue name format +
|
|
`enqueue_tx` unreachability.
|
|
3. **What this task added** — the one unpinned posture from the
|
|
description: `run_once_issues_no_engine_heartbeat_in_delivery` —
|
|
the handover carries the raw claim deadline (no engine-side
|
|
renewal before or inside delivery), and the boxed
|
|
[`JobHandle`]'s own `heartbeat` remains available to the closure
|
|
(renewal is the consumer's obligation, ADR-019 §5). The delivery
|
|
fixture grew `handover_deadline` + a manual-heartbeat arm for it.
|
|
4. **No production code changed** — verified correct as-found; tests
|
|
only (+1 in `outbox.rs`, fixture widened, 100 mem tests total).
|
|
The scheduler's boundary waits ride realtime tokio time with
|
|
stamps on the engine clock; the tick/catch-up/leadership pins
|
|
drive the crate-internal `run_tick`/`run_schedules` directly, so
|
|
determinism holds without realtime waits.
|
|
|
|
## Summary
|
|
|
|
> Filled on completion, 2026-10-10.
|
|
|
|
Completed the mem engine's scheduler/outbox task against the seam
|
|
task's foundation: reconciled the description's pin list against
|
|
`alkstore-mem/src/scheduler.rs` + `outbox.rs`'s existing suite (14
|
|
pre-existing tests), found every tick/catch-up/leadership/validation/
|
|
outbox shape already pinned, and added the one gap — the
|
|
no-engine-issued-heartbeat posture pinned end-to-end (raw claim
|
|
deadline at handover; the closure's own heartbeat path available and
|
|
honored). Acceptance criteria: scheduler surface + tick determinism ✓
|
|
(seam rows), leadership ✓ (seam rows), validation + grammar ✓ (seam
|
|
rows), outbox semantics + derived name format ✓ (seam rows + the new
|
|
heartbeat row), gates ✓. Verified: `cargo test -p alkstore-mem`
|
|
100/100 (99 + 1 new, new test green solo), workspace `cargo test`
|
|
green, `cargo clippy --all-targets -- -D warnings` clean,
|
|
`cargo fmt --check` clean, `cargo check --target
|
|
wasm32-unknown-unknown -p alkstore-mem` clean. |