mem-engine-seam-tx — the wave's load-bearing seam lands: the complete Store/TxHandle trait surfaces over the guarded state core, with the tx overlay as the discipline centerpiece. Wider than the sqlite/pg seam twins by design (those rode pre-ported substrate ops and left mechanism methods stubbed; mem has no substrate, and the task's acceptance pins the full trait surfaces implemented over guarded state — a Rust trait impl is all-or-nothing per method): the seam task therefore realizes the mechanisms' committed-state effect layer and every mechanism handle itself, and the wave-6 mechanism tasks now own their acceptance-criteria unit pins against this foundation. Store surface: notify (codec-typed encode as the entry check only — the wake delivers the channel alone; PayloadTooLarge never produced, the ADR-016 §5 non-occurrence arm pinned at 2 MiB), listen (WakeReceiver bridge with the pinned recv forms; attach starts from now, one close path at engine drop), stream/queue/outbox/try_lock (validated constructors → boxed core-owned handle traits: the ADR-010 queue state machine with claims/reclaim-eats-attempt/uniform validity predicate/dead-letter/sweep + mem's third-owner equal-jitter curve — std-only RandomState hash jitter, no rand; ADR-015 log+offsets+trim+wake-driven subscribe; ADR-008 §7 TTL registry with renew-past-expiry refused; ADR-014 outbox helper with run_once ack/retry), schedule/unschedule/run_schedules (ADR-009 collapse: @every-only third-owner parser + register-table with pre-parsed interval and resolved stamps; the 64-cap bounded catch-up + grid-aligned skip-forward tick fired-and-advanced under one guarded op (schedules→queues lock order); leadership through the reserved __alkstore_scheduler lock with runner-unique owner, renew-before-tick, keep-awake sleep renewing on cadence, LeadershipLost / clean-stop-Ok returns). Tx overlay: begin_tx allocates the handle-private overlay; all eleven *_tx stage with entry-point validation before any effect, Codec-typed encode, one clock read, ADR-020 §3 stamp resolution (plain 300/3/5/none, outbox 60/5/5, the per-job max_attempts override via resolution.rs's stamps_with_override); tx reads merge overlay over committed (read-your-own-writes: queue-scoped + dead-visible get_job_tx, monotone staged saves, offset-ASC merged reads, extent guards clamping empty); commit consumes the handle, merges under one guarded op (queues→streams, the canonical lock order documented in state.rs), and fires the overlay's pending wakes AFTER the merge — wake-at-commit structural from birth (the pg-fix-tx-wake failure shape cannot exist here); drop (explicit, or through an unwinding panic — the overlay is plain local data, nothing runs on discard) = rollback with no ghosts, trivially. Stage-time id/offset allocation (the pg-shaped story — rolled-back allocations gap out; claims order id ASC, offsets immutable/never renumbered). Stream subscribers ride a reserved-prefix engine-internal wake channel (__alkstore_stream_wake:{stream}) unreachable by consumer entry points, attached before the stored-offset read; notify's transport stores nothing. 89 mem unit tests: the overlay discipline rows the task pins, the numeric-domain rows (extents/durations/boundaries/grammar/validation), and the mechanism smoke pins (the defect-class catch: ack/ack_batch originally deleted before checking the predicate — fixed; a refusal must leave the row exactly as it was). Manifest: serde_json = 1 joins (the trait signatures cross Value; both engine twins carry it; not a driver dep — wasm32 gate stays green). Task file: status completed, Notes (the whole-engine scope disposition + 11 decision-of-record entries incl. lock-renewal delta vs the sqlite substrate, ScheduleRow shape, runner lease mechanics, substrate predicate parity) + Summary filled. Verified: cargo build; workspace cargo test green (core 25, suite harness 3, mem 89, SQLite 191 lib + 25 suite, pg 124 lib + 25 suite + 9 schema); clippy --all-targets -D warnings clean; cargo fmt --check clean; cargo check --target wasm32-unknown-unknown -p alkstore-mem clean
This commit is contained in:
1 parent
6632dcce1f
commit
1c3e96802f
14 files changed
+5375
-70
No files matched your search
Generated
+1
@@ -26,6 +26,7 @@ name = "alkstore-mem"
|
|||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"alkstore",
|
"alkstore",
|
||||||
|
"serde_json",
|
||||||
"tokio",
|
"tokio",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ repository.workspace = true
|
|||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
alkstore = { version = "0.1", path = "../alkstore" }
|
alkstore = { version = "0.1", path = "../alkstore" }
|
||||||
|
serde_json = "1"
|
||||||
tokio = { version = "1", features = ["sync", "time"] }
|
tokio = { version = "1", features = ["sync", "time"] }
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
|
|||||||
+15
-13
@@ -10,7 +10,7 @@
|
|||||||
//! The dependency on `alkstore-mem` *is* the ephemerality declaration;
|
//! The dependency on `alkstore-mem` *is* the ephemerality declaration;
|
||||||
//! compile-time engine identity (this crate) carries the honesty —
|
//! compile-time engine identity (this crate) carries the honesty —
|
||||||
//! no runtime surface is fabricated to detect it. Each
|
//! no runtime surface is fabricated to detect it. Each
|
||||||
//! `MemStore::new` yields a fully isolated instance sharing nothing;
|
//! [`MemStore::new`] yields a fully isolated instance sharing nothing;
|
||||||
//! teardown is drop, and engine drop closes open receivers.
|
//! teardown is drop, and engine drop closes open receivers.
|
||||||
//!
|
//!
|
||||||
//! **Full contract v1** (ADR-024 §3): every mechanism, every guarantee
|
//! **Full contract v1** (ADR-024 §3): every mechanism, every guarantee
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
//! pinned: the contract suite's third `StoreFactory` column (ADR-022)
|
//! pinned: the contract suite's third `StoreFactory` column (ADR-022)
|
||||||
//! runs this engine under a current-thread runtime flavor.
|
//! runs this engine under a current-thread runtime flavor.
|
||||||
//!
|
//!
|
||||||
//! **wasm32** ([ADR-024] §5): this crate compiles clean for
|
//! **wasm32** (ADR-024 §5): this crate compiles clean for
|
||||||
//! `wasm32-unknown-unknown` — tokio restricted to the wasm-supported
|
//! `wasm32-unknown-unknown` — tokio restricted to the wasm-supported
|
||||||
//! subset (sync + time), no driver dependencies, no `spawn_blocking`
|
//! subset (sync + time), no driver dependencies, no `spawn_blocking`
|
||||||
//! anywhere; internals are current-thread-runtime-clean by
|
//! anywhere; internals are current-thread-runtime-clean by
|
||||||
@@ -31,21 +31,23 @@
|
|||||||
//! The store instance *is* the state: guarded per-mechanism sections
|
//! The store instance *is* the state: guarded per-mechanism sections
|
||||||
//! (`crate::state`) behind short-held synchronous internal locks,
|
//! (`crate::state`) behind short-held synchronous internal locks,
|
||||||
//! tokio primitives for cross-await coordination. The tx seam is an
|
//! tokio primitives for cross-await coordination. The tx seam is an
|
||||||
//! overlay (stage → commit-merge → wake-at-commit; drop = discard);
|
//! overlay ([`crate::tx`] — stage in the handle-private overlay, commit
|
||||||
//! wakes route through in-process channels (`crate::wakes`) —
|
//! merges atomically + wakes at commit, drop discards); wakes route
|
||||||
//! delivered at the commit boundary directly to each open receiver,
|
//! through in-process channels (`crate::wakes`) — delivered at the
|
||||||
//! per-subscriber FIFO by commit order, never coalesced (no batching
|
//! commit boundary directly to each open receiver, per-subscriber FIFO
|
||||||
//! boundary exists).
|
//! by commit order, never coalesced (no batching boundary exists).
|
||||||
//!
|
|
||||||
//! Wave 6 is under construction (ADR-024): the foundations —
|
|
||||||
//! `MemStore::new`, the guarded state core, the wake-routing
|
|
||||||
//! substrate, the clock accessor, entry-point validation — are laid;
|
|
||||||
//! the `Store`/`TxHandle` trait surfaces and the mechanism handles
|
|
||||||
//! land with the wave's seam and mechanism tasks.
|
|
||||||
|
|
||||||
mod clock;
|
mod clock;
|
||||||
|
mod lock;
|
||||||
|
mod notify;
|
||||||
|
mod outbox;
|
||||||
|
mod queue;
|
||||||
|
mod resolution;
|
||||||
|
mod scheduler;
|
||||||
mod state;
|
mod state;
|
||||||
mod store;
|
mod store;
|
||||||
|
mod stream;
|
||||||
|
mod tx;
|
||||||
mod validation;
|
mod validation;
|
||||||
mod wakes;
|
mod wakes;
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,327 @@
|
|||||||
|
//! The named-locks mechanism's mem arm — the in-memory TTL registry
|
||||||
|
//! (engine-mem.md §Mapping the contract, locks row; ADR-008 §7's
|
||||||
|
//! guarantee row).
|
||||||
|
//!
|
||||||
|
//! - Registry entries carry owner + deadline; expiry is a clock check
|
||||||
|
//! at the try/renew boundaries (no ambient revoker exists). After
|
||||||
|
//! TTL expiry exclusion lapses *silently* — no revocation event —
|
||||||
|
//! and the name is re-acquirable; the holder may still release.
|
||||||
|
//! - Contention on one store instance is serialized by the guarded
|
||||||
|
//! section: first wins, the loser reads `None` cleanly (a value,
|
||||||
|
//! never an error). A re-`try_lock` by the *same* owner while held
|
||||||
|
//! grants (no second lease row exists to fight) without refreshing
|
||||||
|
//! the deadline — the SQL engines' `INSERT OR IGNORE` + owner-match
|
||||||
|
//! shape, mirrored.
|
||||||
|
//! - **The duration guard at entry** (ADR-023 §2, durations kind):
|
||||||
|
//! `ttl <= 0` rejects on `try_lock` *and* `renew` with the opaque
|
||||||
|
//! `Database` error — never a silent immediately-expired lease.
|
||||||
|
//! - Locks are not part of the tx seam (no `lock_tx` — contract
|
||||||
|
//! verbatim): the overlay neither sees nor stages locks. The
|
||||||
|
//! scheduler's leadership lock rides this registry through the
|
||||||
|
//! engine-internal acquire/renew path (a reserved name consumer
|
||||||
|
//! entry points can never collide with).
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use alkstore::{BoxedFuture, Error, Lock, Result, validate_local_name, validate_shared_name};
|
||||||
|
|
||||||
|
use crate::clock::Clock;
|
||||||
|
use crate::state::Guard;
|
||||||
|
use crate::state::{LockState, State};
|
||||||
|
|
||||||
|
/// The scheduler's leadership-lock name (ADR-009 §4 — the reserved-
|
||||||
|
/// prefix derived name in the lock namespace; engine-internal: every
|
||||||
|
/// consumer entry point rejects the same prefix, so a consumer can
|
||||||
|
/// never collide with it).
|
||||||
|
pub(crate) const SCHEDULER_LEADER_LOCK: &str = "__alkstore_scheduler";
|
||||||
|
|
||||||
|
fn duration_error(ttl: i64) -> Error {
|
||||||
|
Error::database(std::io::Error::other(format!(
|
||||||
|
"ttl must be a positive duration (seconds), got {ttl}"
|
||||||
|
)))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The registry acquire (the one owner of the row-level mechanical):
|
||||||
|
/// purge the name's expired entry, grant iff the name is absent or
|
||||||
|
/// already owned by this owner. `now` is the caller's clock read.
|
||||||
|
pub(crate) fn lock_acquire(l: &mut LockState, name: &str, owner: &str, ttl: i64, now: i64) -> bool {
|
||||||
|
if let Some(entry) = l.entries.get(name)
|
||||||
|
&& entry.expires_at <= now
|
||||||
|
{
|
||||||
|
l.entries.remove(name);
|
||||||
|
}
|
||||||
|
match l.entries.get(name) {
|
||||||
|
Some(entry) => entry.owner == owner,
|
||||||
|
None => {
|
||||||
|
l.entries.insert(
|
||||||
|
name.to_string(),
|
||||||
|
crate::state::LockEntry {
|
||||||
|
owner: owner.to_string(),
|
||||||
|
expires_at: now.saturating_add(ttl),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The registry renew: a new full TTL window from now. Refused
|
||||||
|
/// (`false`) when the name is held by another owner, or past expiry —
|
||||||
|
/// a renew can never revive a lapsed exclusion (the mem pin: renew
|
||||||
|
/// past expiry is refused, the holder's window is the TTL's).
|
||||||
|
pub(crate) fn lock_renew(l: &mut LockState, name: &str, owner: &str, ttl: i64, now: i64) -> bool {
|
||||||
|
let Some(entry) = l.entries.get_mut(name) else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
if entry.owner != owner || entry.expires_at <= now {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
entry.expires_at = now.saturating_add(ttl);
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The registry release: delete the owner's row only. `false` =
|
||||||
|
/// re-acquired elsewhere (a no-op, not an error). The holder may
|
||||||
|
/// release past expiry — exclusion lapsed silently; releasing the
|
||||||
|
/// lapsed row is still the holder's.
|
||||||
|
pub(crate) fn lock_release(l: &mut LockState, name: &str, owner: &str) -> bool {
|
||||||
|
match l.entries.get(name) {
|
||||||
|
Some(entry) if entry.owner == owner => {
|
||||||
|
l.entries.remove(name);
|
||||||
|
true
|
||||||
|
}
|
||||||
|
_ => false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The consumer `try_lock` path (validation + duration guard + the
|
||||||
|
/// guarded acquire; `Ok(None)` is the held-by-someone-else arm).
|
||||||
|
pub(crate) fn try_lock(
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
name: &str,
|
||||||
|
owner: &str,
|
||||||
|
ttl: i64,
|
||||||
|
) -> Result<Option<Box<dyn Lock>>> {
|
||||||
|
validate_shared_name(name)?;
|
||||||
|
validate_local_name(owner)?;
|
||||||
|
if ttl <= 0 {
|
||||||
|
return Err(duration_error(ttl));
|
||||||
|
}
|
||||||
|
let now = clock.unix_now();
|
||||||
|
let granted = {
|
||||||
|
let mut l = state.locks.guard();
|
||||||
|
lock_acquire(&mut l, name, owner, ttl, now)
|
||||||
|
};
|
||||||
|
if granted {
|
||||||
|
Ok(Some(Box::new(MemLock {
|
||||||
|
name: name.to_string(),
|
||||||
|
owner: owner.to_string(),
|
||||||
|
state,
|
||||||
|
clock,
|
||||||
|
})))
|
||||||
|
} else {
|
||||||
|
Ok(None)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The held named-lock handle (the repeatable `renew` case and the
|
||||||
|
/// consuming `release`).
|
||||||
|
pub struct MemLock {
|
||||||
|
name: String,
|
||||||
|
owner: String,
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemLock {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemLock")
|
||||||
|
.field("name", &self.name)
|
||||||
|
.finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemLock {
|
||||||
|
/// The engine-internal leadership lease (the scheduler's): the
|
||||||
|
/// same registry machinery without consumer validation (the name
|
||||||
|
/// is engine-derived and reserved).
|
||||||
|
pub(crate) fn new_internal(
|
||||||
|
name: &str,
|
||||||
|
owner: &str,
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
name: name.to_string(),
|
||||||
|
owner: owner.to_string(),
|
||||||
|
state,
|
||||||
|
clock,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn acquire_now(&self, ttl: i64, now: i64) -> bool {
|
||||||
|
let mut l = self.state.locks.guard();
|
||||||
|
lock_acquire(&mut l, &self.name, &self.owner, ttl, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn renew_at(&self, ttl: i64, now: i64) -> bool {
|
||||||
|
let mut l = self.state.locks.guard();
|
||||||
|
lock_renew(&mut l, &self.name, &self.owner, ttl, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn release_now(&self) -> bool {
|
||||||
|
let mut l = self.state.locks.guard();
|
||||||
|
lock_release(&mut l, &self.name, &self.owner)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Lock for MemLock {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
|
||||||
|
fn renew<'a>(&'a self, ttl: i64) -> BoxedFuture<'a, Result<bool>> {
|
||||||
|
if ttl <= 0 {
|
||||||
|
return Box::pin(async move { Err(duration_error(ttl)) });
|
||||||
|
}
|
||||||
|
let now = self.clock.unix_now();
|
||||||
|
let granted = self.renew_at(ttl, now);
|
||||||
|
Box::pin(async move { Ok(granted) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn release(self: Box<Self>) -> BoxedFuture<'static, Result<bool>> {
|
||||||
|
Box::pin(async move { Ok(self.release_now()) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::Store;
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn contention_grants_once_and_the_loser_reads_none() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
let first = store.try_lock("db", "w1", 60).await.unwrap();
|
||||||
|
assert!(first.is_some(), "the first acquire wins");
|
||||||
|
let loser = store.try_lock("db", "w2", 60).await.unwrap();
|
||||||
|
assert!(
|
||||||
|
loser.is_none(),
|
||||||
|
"the loser is a clean None (a value, never an error)"
|
||||||
|
);
|
||||||
|
let same_owner = store.try_lock("db", "w1", 60).await.unwrap();
|
||||||
|
assert!(
|
||||||
|
same_owner.is_some(),
|
||||||
|
"the owner re-grants (no second lease row)"
|
||||||
|
);
|
||||||
|
drop(first);
|
||||||
|
drop(loser);
|
||||||
|
if let Some(re) = same_owner {
|
||||||
|
assert!(re.renew(60).await.unwrap(), "renew is an absolute reset");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn duration_guards_reject_non_positive_ttls() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
for ttl in [0i64, -1] {
|
||||||
|
match store.try_lock("db", "w", ttl).await {
|
||||||
|
Err(alkstore::Error::Database(_)) => {}
|
||||||
|
_other => panic!("try_lock guards a non-positive ttl"),
|
||||||
|
}
|
||||||
|
let granted = store.try_lock("db", "w", 60).await.unwrap().unwrap();
|
||||||
|
match granted.renew(ttl).await {
|
||||||
|
Err(alkstore::Error::Database(_)) => {}
|
||||||
|
_other => panic!("renew guards a non-positive ttl"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn expiry_lapses_silently_and_reacquires() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
let holder = store.try_lock("db", "w1", 60).await.unwrap().unwrap();
|
||||||
|
assert!(store.try_lock("db", "w2", 60).await.unwrap().is_none());
|
||||||
|
store.clock.advance(59);
|
||||||
|
assert!(
|
||||||
|
store.try_lock("db", "w2", 60).await.unwrap().is_none(),
|
||||||
|
"still within TTL"
|
||||||
|
);
|
||||||
|
store.clock.advance(2);
|
||||||
|
assert!(
|
||||||
|
store.try_lock("db", "w2", 60).await.unwrap().is_some(),
|
||||||
|
"exclusion lapsed silently — re-acquirable, no revocation event"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!holder.renew(60).await.unwrap(),
|
||||||
|
"the re-acquired lock is not the old holder's to renew"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn renew_past_expiry_is_refused() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
let holder = store.try_lock("db", "w1", 60).await.unwrap().unwrap();
|
||||||
|
store.clock.advance(61);
|
||||||
|
assert!(
|
||||||
|
!holder.renew(60).await.unwrap(),
|
||||||
|
"a renew cannot revive a lapsed exclusion"
|
||||||
|
);
|
||||||
|
let re = store.try_lock("db", "w2", 60).await.unwrap().unwrap();
|
||||||
|
assert!(re.release().await.unwrap(), "the new holder releases");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn release_shapes() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
let holder = store.try_lock("db", "w1", 60).await.unwrap().unwrap();
|
||||||
|
assert!(holder.release().await.unwrap(), "a held lock releases true");
|
||||||
|
let w1 = store.try_lock("db", "w1", 60).await.unwrap().unwrap();
|
||||||
|
store.clock.advance(120);
|
||||||
|
assert!(
|
||||||
|
w1.release().await.unwrap(),
|
||||||
|
"release past expiry stays the holder's — exclusion lapsed silently"
|
||||||
|
);
|
||||||
|
let a = store.try_lock("db", "w1", 60).await.unwrap().unwrap();
|
||||||
|
store.clock.advance(120);
|
||||||
|
store.try_lock("db", "w2", 60).await.unwrap().unwrap();
|
||||||
|
assert!(
|
||||||
|
!a.release().await.unwrap(),
|
||||||
|
"re-acquired elsewhere: the old holder's release is a no-op"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn entry_validation() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
for bad in ["", " ", "__alkstore_scheduler"] {
|
||||||
|
match store.try_lock(bad, "w", 60).await {
|
||||||
|
Err(alkstore::Error::InvalidName { .. } | alkstore::Error::ReservedName { .. }) => {
|
||||||
|
}
|
||||||
|
_other => panic!("try_lock({bad:?}) rejects"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
match store.try_lock("db", "", 60).await {
|
||||||
|
Err(alkstore::Error::InvalidName { .. }) => {}
|
||||||
|
_other => panic!("an empty owner rejects"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn locks_are_not_tx_scoped() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
assert!(store.try_lock("db", "w", 60).await.unwrap().is_some());
|
||||||
|
let tx = store.begin_tx().await.unwrap();
|
||||||
|
drop(tx);
|
||||||
|
assert!(
|
||||||
|
store.try_lock("db", "w2", 60).await.unwrap().is_none(),
|
||||||
|
"rollback or not — the overlay never sees or stages locks"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
//! The notify/listen mechanism's mem arm over the wake-routing
|
||||||
|
//! substrate (`sqlite-engine-notify-listen`'s shape, realized
|
||||||
|
//! in-process; `notify_tx` stages in the tx overlay).
|
||||||
|
//!
|
||||||
|
//! - `notify` — the auto-commit path: validate, encode the payload
|
||||||
|
//! (`Codec` typed at the helper, ADR-023 §1), and fire the wake at
|
||||||
|
//! the op's own commit boundary — one guarded operation, nothing to
|
||||||
|
//! roll back. **No payload limit at any size** — `PayloadTooLarge`
|
||||||
|
//! is never produced on this engine (the ADR-016 §5 occurrence
|
||||||
|
//! asymmetry; mem's column of the classification). The encoded bytes
|
||||||
|
//! are nowhere stored: the wake contract's `Wake { channel }` is the
|
||||||
|
//! entire delivery (ADR-008 §3) — the payload encoding run is the
|
||||||
|
//! codec-typed entry check, not a transport.
|
||||||
|
//! - `listen` — attach a per-channel feed (starts from "now";
|
||||||
|
//! attach-after-commit no-replay is structural) and bridge it into
|
||||||
|
//! the pinned [`alkstore::WakeReceiver`] forms. Delivery is
|
||||||
|
//! channel-scoped, per-notify (never coalesced — no batching
|
||||||
|
//! boundary exists), per-subscriber FIFO by commit order. **One
|
||||||
|
//! close path — engine drop**: the bus close disconnects every open
|
||||||
|
//! receiver (`recv() -> None`, terminal); `try_recv`/`recv_timeout`
|
||||||
|
//! carry [`alkstore::Error::Closed`] distinguishable from idle, and
|
||||||
|
//! timeout expiry is the documented `Ok(None)` arm, never a
|
||||||
|
//! close-shaped signal. The reserved reconnect-wake channel is never
|
||||||
|
//! emitted (no reconnection concept exists) and stays rejected as a
|
||||||
|
//! consumer name.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use alkstore::{BoxedFuture, Error, Result, Wake, WakeReceiver, validate_shared_name};
|
||||||
|
|
||||||
|
use crate::wakes::{WakeBus, WakeFeed};
|
||||||
|
|
||||||
|
/// The auto-commit notify path (see the module docs). Name validation
|
||||||
|
/// fires at the entry point, before the payload is touched (ADR-008
|
||||||
|
/// §4); the wake fires at the op's own commit boundary.
|
||||||
|
pub(crate) fn notify(
|
||||||
|
wakes: Arc<WakeBus>,
|
||||||
|
channel: &str,
|
||||||
|
payload: Value,
|
||||||
|
) -> BoxedFuture<'static, Result<()>> {
|
||||||
|
if let Err(e) = validate_shared_name(channel) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let owned = channel.to_string();
|
||||||
|
let encoded = alkstore::encode_payload(&payload);
|
||||||
|
Box::pin(async move {
|
||||||
|
encoded?;
|
||||||
|
wakes.fire_commit_wakes([owned]);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The listen path (see the module docs): validate, attach, bridge.
|
||||||
|
pub(crate) fn listen(wakes: Arc<WakeBus>, channel: &str) -> Result<Box<dyn WakeReceiver>> {
|
||||||
|
validate_shared_name(channel)?;
|
||||||
|
let feed = wakes.attach(channel);
|
||||||
|
Ok(Box::new(MemWakeReceiver { feed }))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The consumer-visible wake receiver: the bridge over the feed, with
|
||||||
|
/// the pinned recv/try_recv/recv_timeout forms and the terminal None
|
||||||
|
/// close arm (its own drop detaches the feed, the engine's drop
|
||||||
|
/// disconnects it).
|
||||||
|
struct MemWakeReceiver {
|
||||||
|
feed: WakeFeed,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemWakeReceiver {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemWakeReceiver").finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl WakeReceiver for MemWakeReceiver {
|
||||||
|
fn recv<'a>(&'a mut self) -> BoxedFuture<'a, Option<Wake>> {
|
||||||
|
Box::pin(async move { self.feed.rx().recv().await })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn try_recv(&mut self) -> Result<Option<Wake>> {
|
||||||
|
match self.feed.rx().try_recv() {
|
||||||
|
Ok(wake) => Ok(Some(wake)),
|
||||||
|
Err(tokio::sync::mpsc::error::TryRecvError::Empty) => Ok(None),
|
||||||
|
Err(tokio::sync::mpsc::error::TryRecvError::Disconnected) => Err(Error::Closed),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn recv_timeout<'a>(&'a mut self, timeout: Duration) -> BoxedFuture<'a, Result<Option<Wake>>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
match tokio::time::timeout(timeout, self.feed.rx().recv()).await {
|
||||||
|
Ok(Some(wake)) => Ok(Some(wake)),
|
||||||
|
Ok(None) => Err(Error::Closed),
|
||||||
|
Err(_expired) => Ok(None),
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::Store;
|
||||||
|
|
||||||
|
async fn listener(store: &MemStore, channel: &str) -> Box<dyn WakeReceiver> {
|
||||||
|
store.listen(channel).await.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn notify_wakes_open_listeners_at_the_commit_boundary() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let mut a = listener(&store, "orders").await;
|
||||||
|
let mut foreign = listener(&store, "payments").await;
|
||||||
|
store
|
||||||
|
.notify("orders", serde_json::json!({"n": 1}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let wake = tokio::time::timeout(std::time::Duration::from_millis(200), a.recv()).await;
|
||||||
|
let wake = wake.expect("the committed notify delivers").unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
wake.channel, "orders",
|
||||||
|
"the channel is the wake's one piece of content"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
foreign.try_recv().unwrap().is_none(),
|
||||||
|
"delivery is channel-scoped"
|
||||||
|
);
|
||||||
|
// Per-notify delivery — one wake per commit-notify, never
|
||||||
|
// coalesced.
|
||||||
|
store.notify("orders", serde_json::json!(2)).await.unwrap();
|
||||||
|
store.notify("orders", serde_json::json!(3)).await.unwrap();
|
||||||
|
let first = a.try_recv().unwrap();
|
||||||
|
let second = a.try_recv().unwrap();
|
||||||
|
let drained = a.try_recv().unwrap();
|
||||||
|
assert!(
|
||||||
|
first.is_some() && second.is_some() && drained.is_none(),
|
||||||
|
"two notifies, two wakes"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn listener_attached_after_a_commit_never_sees_it() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.notify("orders", serde_json::json!(1)).await.unwrap();
|
||||||
|
let mut late = listener(&store, "orders").await;
|
||||||
|
assert!(late.try_recv().unwrap().is_none(), "no replay");
|
||||||
|
store.notify("orders", serde_json::json!(2)).await.unwrap();
|
||||||
|
assert!(late.recv().await.is_some(), "post-attach commits deliver");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn entry_validation_on_both_paths() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
for bad in [
|
||||||
|
"",
|
||||||
|
" ",
|
||||||
|
"__alkstore_orders",
|
||||||
|
alkstore::RESERVED_LISTENER_RECONNECTED,
|
||||||
|
] {
|
||||||
|
let err = store.notify(bad, serde_json::json!(1)).await.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
err,
|
||||||
|
alkstore::Error::InvalidName { .. } | alkstore::Error::ReservedName { .. }
|
||||||
|
),
|
||||||
|
"notify({bad:?}) rejects: {err:?}"
|
||||||
|
);
|
||||||
|
match store.listen(bad).await {
|
||||||
|
Err(alkstore::Error::InvalidName { .. } | alkstore::Error::ReservedName { .. }) => {
|
||||||
|
}
|
||||||
|
_other => panic!("listen({bad:?}) rejects"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn payload_too_large_is_never_produced_at_any_size() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
// 2 MiB of value content — the pg-side rejection arm does not
|
||||||
|
// exist on this engine (the ADR-016 §5 non-occurrence asymmetry).
|
||||||
|
let big = serde_json::Value::String("x".repeat(2 * 1024 * 1024));
|
||||||
|
store.notify("orders", big).await.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn recv_forms_carry_the_pinned_arms() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let mut rx = listener(&store, "orders").await;
|
||||||
|
assert!(rx.try_recv().unwrap().is_none(), "idle is Ok(None)");
|
||||||
|
let expired = tokio::time::timeout(
|
||||||
|
std::time::Duration::from_millis(50),
|
||||||
|
rx.recv_timeout(std::time::Duration::from_millis(30)),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("recv_timeout resolves")
|
||||||
|
.expect("timeout expiry is Ok(None) — never a close-shaped signal");
|
||||||
|
assert!(expired.is_none());
|
||||||
|
store.notify("orders", serde_json::json!(1)).await.unwrap();
|
||||||
|
assert!(
|
||||||
|
rx.recv_timeout(std::time::Duration::from_millis(200))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.is_some()
|
||||||
|
);
|
||||||
|
drop(store);
|
||||||
|
assert!(
|
||||||
|
matches!(rx.try_recv(), Err(alkstore::Error::Closed)),
|
||||||
|
"closed is distinguishable"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
rx.recv().await.is_none(),
|
||||||
|
"engine drop is the terminal None"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn receiver_drop_detaches_its_feed() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
{
|
||||||
|
let rx = listener(&store, "orders").await;
|
||||||
|
drop(rx);
|
||||||
|
}
|
||||||
|
store.notify("orders", serde_json::json!(1)).await.unwrap();
|
||||||
|
let mut kept = listener(&store, "orders").await;
|
||||||
|
assert!(
|
||||||
|
kept.try_recv().unwrap().is_none(),
|
||||||
|
"a dropped receiver detached: its feed got nothing from later commits"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,319 @@
|
|||||||
|
//! The outbox mechanism's mem arm — the helper over queues
|
||||||
|
//! (ADR-014's auto-commit half; the tx half — `outbox_enqueue_tx` —
|
||||||
|
//! stages in the tx overlay):
|
||||||
|
//!
|
||||||
|
//! - `outbox(name)` — the validated constructor (shared-namespace
|
||||||
|
//! rules); the backing queue `__alkstore_outbox:{name}` derives
|
||||||
|
//! engine-side under the reserved prefix — unreachable by
|
||||||
|
//! `enqueue_tx` by design (ADR-014 §1: outbox enqueue is the only
|
||||||
|
//! path into it).
|
||||||
|
//! - `enqueue` — auto-commit into the backing queue, stamping the
|
||||||
|
//! outbox's derived 60/5/5 set (ADR-010 §3 / ADR-014 §1);
|
||||||
|
//! `EnqueueOpts::max_attempts` is the one per-job override,
|
||||||
|
//! `delay`/`run_at`/`expires` resolve per ADR-020 at one clock read.
|
||||||
|
//! - `run_once(worker_id, delivery)` — claim one job from the backing
|
||||||
|
//! queue via the ordinary claim machinery, run `delivery`:
|
||||||
|
//! `Ok` ⇒ ack; `Err(e)` ⇒ `retry(Some(e), None)` — the queue's own
|
||||||
|
//! equal-jitter curve (mem's third-owner computation), the delivery
|
||||||
|
//! error string riding the exhaust path (surfaced on a dead row).
|
||||||
|
//! `true` = claimed and processed; `false` = no work (a value, not
|
||||||
|
//! an error). **No engine-issued heartbeat inside delivery** — the
|
||||||
|
//! boxed [`alkstore::JobHandle`]'s own `heartbeat` remains available
|
||||||
|
//! to the closure; renewal inside the deadline is the consumer's
|
||||||
|
//! obligation (ADR-019 §5).
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use alkstore::{
|
||||||
|
BoxedFuture, Delivery, EnqueueOpts, JobHandle, JobState, Outbox, Result, validate_local_name,
|
||||||
|
};
|
||||||
|
|
||||||
|
use crate::clock::Clock;
|
||||||
|
use crate::queue::{MemJobHandle, ack_row, claim_via_state, enqueue_row, job_from_row};
|
||||||
|
use crate::resolution::{
|
||||||
|
backoff_delay_s, outbox_backing_queue_name, outbox_default_stamps, resolve_enqueue_opts,
|
||||||
|
stamps_with_override,
|
||||||
|
};
|
||||||
|
use crate::state::Guard;
|
||||||
|
use crate::state::State;
|
||||||
|
|
||||||
|
/// The outbox-scoped handle (see the module docs).
|
||||||
|
pub struct MemOutbox {
|
||||||
|
name: String,
|
||||||
|
backing: String,
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemOutbox {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemOutbox")
|
||||||
|
.field("name", &self.name)
|
||||||
|
.finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemOutbox {
|
||||||
|
pub(crate) fn new(name: String, state: Arc<State>, clock: Arc<Clock>) -> Self {
|
||||||
|
Self {
|
||||||
|
backing: outbox_backing_queue_name(&name),
|
||||||
|
name,
|
||||||
|
state,
|
||||||
|
clock,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Outbox for MemOutbox {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
|
||||||
|
fn enqueue<'a>(&'a self, payload: Value, opts: EnqueueOpts) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
match self.stage_enqueue(payload, &opts) {
|
||||||
|
Ok(id) => Box::pin(async move { Ok(id) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_once<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
worker_id: &str,
|
||||||
|
delivery: &'a mut dyn Delivery,
|
||||||
|
) -> BoxedFuture<'a, Result<bool>> {
|
||||||
|
if let Err(e) = validate_local_name(worker_id) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let backing = self.backing.clone();
|
||||||
|
let state = self.state.clone();
|
||||||
|
let clock = self.clock.clone();
|
||||||
|
let worker_id = worker_id.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
let claim_now = clock.unix_now();
|
||||||
|
let claimed = claim_via_state(&state, &backing, &worker_id, 1, claim_now).pop();
|
||||||
|
let Some(job_row) = claimed else {
|
||||||
|
return Ok(false);
|
||||||
|
};
|
||||||
|
let job = job_from_row(job_row.clone(), JobState::Processing, None, None);
|
||||||
|
let handle: Box<dyn JobHandle> =
|
||||||
|
Box::new(MemJobHandle::new(job, state.clone(), clock.clone()));
|
||||||
|
match delivery.deliver(handle).await {
|
||||||
|
Ok(()) => {
|
||||||
|
let ack_now = clock.unix_now();
|
||||||
|
let mut q = state.queues.guard();
|
||||||
|
ack_row(&mut q, job_row.id, &worker_id, ack_now);
|
||||||
|
Ok(true)
|
||||||
|
}
|
||||||
|
Err(err) => {
|
||||||
|
// `retry(Some(e), None)` exactly — the delivery error
|
||||||
|
// string carries the exhaust path (surfaced on a dead
|
||||||
|
// row), and the delay is the queue's curve (the
|
||||||
|
// engine's one-owner equal-jitter computation).
|
||||||
|
let exhaust = err.to_string();
|
||||||
|
let delay =
|
||||||
|
backoff_delay_s(job_row.backoff_base_s, job_row.attempts, job_row.id);
|
||||||
|
let retry_now = clock.unix_now();
|
||||||
|
let mut q = state.queues.guard();
|
||||||
|
crate::queue::retry_row(
|
||||||
|
&mut q, job_row.id, &worker_id, delay, &exhaust, retry_now,
|
||||||
|
);
|
||||||
|
Ok(true)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemOutbox {
|
||||||
|
fn stage_enqueue(&self, payload: Value, opts: &EnqueueOpts) -> Result<i64> {
|
||||||
|
let bytes = alkstore::encode_payload(&payload)?;
|
||||||
|
let now = self.clock.unix_now();
|
||||||
|
let resolved = resolve_enqueue_opts(now, opts);
|
||||||
|
let stamps = stamps_with_override(outbox_default_stamps(), opts);
|
||||||
|
Ok(enqueue_row(
|
||||||
|
&self.state,
|
||||||
|
now,
|
||||||
|
&self.backing,
|
||||||
|
bytes,
|
||||||
|
opts,
|
||||||
|
stamps,
|
||||||
|
&resolved,
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::{Error, Store};
|
||||||
|
|
||||||
|
/// The delivery callable the tests drive: records the claimed
|
||||||
|
/// payload (and stamps), resolves with a take()-able outcome.
|
||||||
|
struct RecordingDelivery {
|
||||||
|
saw_payload: Option<Vec<u8>>,
|
||||||
|
stamps: Option<(i64, i64, i64, Option<i64>)>,
|
||||||
|
outcome: Option<Result<()>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RecordingDelivery {
|
||||||
|
fn with(outcome: Result<()>) -> Self {
|
||||||
|
Self {
|
||||||
|
saw_payload: None,
|
||||||
|
stamps: None,
|
||||||
|
outcome: Some(outcome),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn db_error(msg: &str) -> Error {
|
||||||
|
Error::database(std::io::Error::other(msg))
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Delivery for RecordingDelivery {
|
||||||
|
fn deliver<'a>(&'a mut self, job: Box<dyn JobHandle>) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
self.saw_payload = Some(job.job().payload.clone());
|
||||||
|
let j = job.job();
|
||||||
|
self.stamps = Some((
|
||||||
|
j.visibility_timeout_s,
|
||||||
|
j.max_attempts,
|
||||||
|
j.backoff_base_s,
|
||||||
|
j.dead_letter_retention_s,
|
||||||
|
));
|
||||||
|
let outcome = self.outcome.take().unwrap_or(Ok(()));
|
||||||
|
Box::pin(async move { outcome })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn outbox(store: &MemStore, name: &str) -> MemOutbox {
|
||||||
|
MemOutbox::new(name.to_string(), store.state.clone(), store.clock.clone())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn run_once_acks_on_delivery_ok_and_stamps_the_derived_set() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000);
|
||||||
|
let mut ob = outbox(&store, "mail");
|
||||||
|
let id = ob
|
||||||
|
.enqueue(serde_json::json!({"send": true}), EnqueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let mut delivery = RecordingDelivery::with(Ok(()));
|
||||||
|
assert!(ob.run_once("w1", &mut delivery).await.unwrap());
|
||||||
|
assert_eq!(
|
||||||
|
delivery.saw_payload,
|
||||||
|
Some(serde_json::to_vec(&serde_json::json!({"send": true})).unwrap()),
|
||||||
|
"the delivery claims ride the claimed row's exact payload"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!ob.run_once("w1", &mut delivery).await.unwrap(),
|
||||||
|
"a second run_once finds no work (a value, not an error)"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!store.state.queues.guard().live.contains_key(&id),
|
||||||
|
"Ok ⇒ ack deletes"
|
||||||
|
);
|
||||||
|
|
||||||
|
let id2 = ob
|
||||||
|
.enqueue(
|
||||||
|
serde_json::json!(2),
|
||||||
|
EnqueueOpts {
|
||||||
|
max_attempts: Some(8),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let mut recorder = RecordingDelivery::with(Ok(()));
|
||||||
|
assert!(ob.run_once("w2", &mut recorder).await.unwrap());
|
||||||
|
assert_eq!(
|
||||||
|
recorder.stamps,
|
||||||
|
Some((60, 8, 5, None)),
|
||||||
|
"the derived 60/5/5 set stamps the row, max_attempts the one override"
|
||||||
|
);
|
||||||
|
assert!(!store.state.queues.guard().live.contains_key(&id2), "acked");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn run_once_retries_with_the_queue_curve_and_delivery_error_on_exhaust() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(2_000);
|
||||||
|
let mut ob = outbox(&store, "mail");
|
||||||
|
let id = ob
|
||||||
|
.enqueue(serde_json::json!(1), EnqueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let mut delivery = RecordingDelivery::with(Err(db_error("smtp down")));
|
||||||
|
assert!(ob.run_once("w1", &mut delivery).await.unwrap());
|
||||||
|
{
|
||||||
|
let q = store.state.queues.guard();
|
||||||
|
let row = q.live.get(&id).unwrap();
|
||||||
|
assert!(row.pending, "the retry rescheduled pending");
|
||||||
|
assert_eq!(row.worker_id, None);
|
||||||
|
assert_eq!(row.attempts, 1);
|
||||||
|
let cap = row
|
||||||
|
.backoff_base_s
|
||||||
|
.saturating_mul(1i64 << (row.attempts - 1).min(30))
|
||||||
|
.min(3600);
|
||||||
|
let half = (cap + 1) / 2;
|
||||||
|
let delay = row.run_at - 2_000;
|
||||||
|
assert!(
|
||||||
|
(half..=cap).contains(&delay),
|
||||||
|
"the delay is the equal-jitter curve: {delay}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Exhaustion surfaces the delivery error string on the dead row.
|
||||||
|
let mut claims = 0;
|
||||||
|
for _ in 0..5 {
|
||||||
|
store.clock.advance(60);
|
||||||
|
claims += ob
|
||||||
|
.run_once(
|
||||||
|
"w1",
|
||||||
|
&mut RecordingDelivery::with(Err(db_error("smtp down"))),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap() as i64;
|
||||||
|
}
|
||||||
|
assert!(claims >= 1, "the retried row is claimable again");
|
||||||
|
let q = store.state.queues.guard();
|
||||||
|
let dead = q.dead.get(&id).cloned().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
dead.last_error,
|
||||||
|
db_error("smtp down").to_string(),
|
||||||
|
"the delivery error rides the exhaust path as its Display"
|
||||||
|
);
|
||||||
|
assert!(dead.died_at == 2_000 || dead.died_at > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn empty_worker_id_is_invalid() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let mut ob = outbox(&store, "mail");
|
||||||
|
let err = ob
|
||||||
|
.run_once("", &mut RecordingDelivery::with(Ok(())))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(err, Error::InvalidName { .. }));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn derived_backing_queue_name_and_unreachability() {
|
||||||
|
assert_eq!(
|
||||||
|
outbox_backing_queue_name("mail"),
|
||||||
|
format!("{}outbox:mail", alkstore::RESERVED_PREFIX)
|
||||||
|
);
|
||||||
|
let store = MemStore::new();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let err = tx
|
||||||
|
.enqueue_tx(
|
||||||
|
&outbox_backing_queue_name("mail"),
|
||||||
|
EnqueueOpts::default(),
|
||||||
|
serde_json::json!(1),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(err, Error::ReservedName { .. }));
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,248 @@
|
|||||||
|
//! The engine-side opts-resolution arithmetic (ADR-020 §1–§3) and the
|
||||||
|
//! derived stamp sets — mem's own implementation (ADR-012 §2's
|
||||||
|
//! one-owner rule, third owner; the other engines' copies are
|
||||||
|
//! references, not dependencies, and the suite pins three-way
|
||||||
|
//! equivalence).
|
||||||
|
//!
|
||||||
|
//! Every function takes the enqueue instant as a parameter — the one
|
||||||
|
//! clock read per enqueue is the caller's (`crate::clock`), and these
|
||||||
|
//! functions never read a clock of their own except the jitter draw's
|
||||||
|
//! subsecond entropy.
|
||||||
|
|
||||||
|
use std::collections::hash_map::RandomState;
|
||||||
|
use std::hash::BuildHasher;
|
||||||
|
use std::time::{SystemTime, UNIX_EPOCH};
|
||||||
|
|
||||||
|
use alkstore::{EnqueueOpts, QueueOpts};
|
||||||
|
|
||||||
|
/// The four §3a stamps a queue row carries (resolved at enqueue).
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
pub(crate) struct Stamps {
|
||||||
|
pub(crate) max_attempts: i64,
|
||||||
|
pub(crate) visibility_timeout_s: i64,
|
||||||
|
pub(crate) backoff_base_s: i64,
|
||||||
|
pub(crate) dead_letter_retention_s: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The pinned dead-letter default strings (row content, not error
|
||||||
|
/// variants — ADR-010 §4 / ADR-019 §3's annotation: `"max attempts
|
||||||
|
/// exceeded"` at exhaustion, `"failed"` for `fail(None)`, `"expired"`
|
||||||
|
/// on sweeps; a caller-supplied string carries verbatim).
|
||||||
|
pub(crate) const EXHAUSTED_ERROR: &str = "max attempts exceeded";
|
||||||
|
pub(crate) const FAILED_ERROR: &str = "failed";
|
||||||
|
pub(crate) const EXPIRED_ERROR: &str = "expired";
|
||||||
|
|
||||||
|
/// The backoff cap (ADR-010 §3 — a pinned contract constant, no knob).
|
||||||
|
const BACKOFF_CAP_S: i64 = 3600;
|
||||||
|
|
||||||
|
/// The plain queue's derived `QueueOpts` defaults (300/3/5/none —
|
||||||
|
/// ADR-010 §3): the stamp source for every no-handle-open shape
|
||||||
|
/// (`enqueue_tx`, scheduler boundary fires, the scheduler-outbox task's
|
||||||
|
/// auto-commit uses). Named so the derived-defaults formula has one
|
||||||
|
/// owner engine-wide.
|
||||||
|
pub(crate) fn plain_queue_default_stamps() -> Stamps {
|
||||||
|
let QueueOpts {
|
||||||
|
visibility_timeout_s,
|
||||||
|
max_attempts,
|
||||||
|
backoff_base_s,
|
||||||
|
dead_letter_retention_s,
|
||||||
|
} = QueueOpts::default();
|
||||||
|
Stamps {
|
||||||
|
max_attempts,
|
||||||
|
visibility_timeout_s,
|
||||||
|
backoff_base_s,
|
||||||
|
dead_letter_retention_s,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The outbox backing queue's derived `QueueOpts` set (60/5/5 —
|
||||||
|
/// ADR-010 §3 / ADR-014 §1). The stamp source for `outbox_enqueue_tx`
|
||||||
|
/// and `Outbox::enqueue`.
|
||||||
|
pub(crate) fn outbox_default_stamps() -> Stamps {
|
||||||
|
Stamps {
|
||||||
|
visibility_timeout_s: 60,
|
||||||
|
max_attempts: 5,
|
||||||
|
backoff_base_s: 5,
|
||||||
|
dead_letter_retention_s: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The derived backing-queue name for an outbox (the reserved
|
||||||
|
/// namespace, ADR-008 §4): unreachable by `enqueue_tx` by design
|
||||||
|
/// (ADR-014 §1) — outbox enqueue is the only path into it.
|
||||||
|
pub(crate) fn outbox_backing_queue_name(outbox: &str) -> String {
|
||||||
|
format!("{}outbox:{outbox}", alkstore::RESERVED_PREFIX)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The one per-enqueue stamp override (ADR-010 §3a):
|
||||||
|
/// `EnqueueOpts::max_attempts` is the only per-job override — the
|
||||||
|
/// other stamps always take the base's. One helper for every enqueue
|
||||||
|
/// shape (tx, handle, outbox, scheduler fire).
|
||||||
|
pub(crate) fn stamps_with_override(base: Stamps, opts: &EnqueueOpts) -> Stamps {
|
||||||
|
Stamps {
|
||||||
|
max_attempts: opts.max_attempts.unwrap_or(base.max_attempts),
|
||||||
|
..base
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The resolved enqueue fields (ADR-020 §1/§2): `delay` set → ready
|
||||||
|
/// time `now + delay` (wins over `run_at`); else `run_at` literally;
|
||||||
|
/// else `now`. `expires` is relative seconds from the enqueue instant,
|
||||||
|
/// resolved to the absolute row `expires_at`; `None` = never.
|
||||||
|
pub(crate) struct ResolvedEnqueue {
|
||||||
|
pub(crate) run_at: i64,
|
||||||
|
pub(crate) expires_at: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn resolve_enqueue_opts(now: i64, opts: &EnqueueOpts) -> ResolvedEnqueue {
|
||||||
|
let run_at = match opts.delay {
|
||||||
|
Some(delay) => now.saturating_add(delay),
|
||||||
|
None => opts.run_at.unwrap_or(now),
|
||||||
|
};
|
||||||
|
let expires_at = opts.expires.map(|s| now.saturating_add(s));
|
||||||
|
ResolvedEnqueue { run_at, expires_at }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The equal-jitter exponential backoff (ADR-010 §3 — mem's own
|
||||||
|
/// implementation): `delay ∈ [base·2^(a−1)/2, base·2^(a−1)]` from the
|
||||||
|
/// job's stamped `backoff_base_s`, capped at 1 hour. The attempt index
|
||||||
|
/// `a` is the row's `attempts` at the retry (post-claim count,
|
||||||
|
/// reclaims included); `Some(d)` overrides pass through without this
|
||||||
|
/// function.
|
||||||
|
///
|
||||||
|
/// Integerized second-precision: the draw is uniform over the integers
|
||||||
|
/// in `[ceil(half), cap]` (inclusive both ends) — never below the
|
||||||
|
/// pinned lower bound, never above the cap. The jitter source is
|
||||||
|
/// std-only (the ADR-005 dependency posture — no `rand` dep): a fresh
|
||||||
|
/// [`RandomState`] hasher over the row id, the attempt count, and the
|
||||||
|
/// subsecond clock spreads the draw; the pinned property is the range.
|
||||||
|
pub(crate) fn backoff_delay_s(base_s: i64, attempts: i64, id: i64) -> i64 {
|
||||||
|
let exponent: u32 = (attempts.max(1) as u32).saturating_sub(1).min(30);
|
||||||
|
let doubling: i64 = base_s.saturating_mul(1i64 << exponent).max(1);
|
||||||
|
let cap = doubling.min(BACKOFF_CAP_S);
|
||||||
|
let half = ((cap + 1) / 2).max(1);
|
||||||
|
let span = (cap - half + 1).max(1);
|
||||||
|
let now_nanos = SystemTime::now()
|
||||||
|
.duration_since(UNIX_EPOCH)
|
||||||
|
.map(|d| d.subsec_nanos())
|
||||||
|
.unwrap_or(0);
|
||||||
|
let draw = RandomState::new().hash_one((id, attempts, now_nanos));
|
||||||
|
half + (draw % span as u64) as i64
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn stamp_sets_match_the_pinned_defaults() {
|
||||||
|
let plain = plain_queue_default_stamps();
|
||||||
|
assert_eq!(
|
||||||
|
(
|
||||||
|
plain.visibility_timeout_s,
|
||||||
|
plain.max_attempts,
|
||||||
|
plain.backoff_base_s,
|
||||||
|
plain.dead_letter_retention_s
|
||||||
|
),
|
||||||
|
(300, 3, 5, None)
|
||||||
|
);
|
||||||
|
let outbox = outbox_default_stamps();
|
||||||
|
assert_eq!(
|
||||||
|
(
|
||||||
|
outbox.visibility_timeout_s,
|
||||||
|
outbox.max_attempts,
|
||||||
|
outbox.backoff_base_s,
|
||||||
|
outbox.dead_letter_retention_s
|
||||||
|
),
|
||||||
|
(60, 5, 5, None)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn override_touches_only_max_attempts() {
|
||||||
|
let base = Stamps {
|
||||||
|
max_attempts: 3,
|
||||||
|
visibility_timeout_s: 300,
|
||||||
|
backoff_base_s: 5,
|
||||||
|
dead_letter_retention_s: None,
|
||||||
|
};
|
||||||
|
let opts = EnqueueOpts {
|
||||||
|
max_attempts: Some(9),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let stamped = stamps_with_override(base, &opts);
|
||||||
|
assert_eq!(stamped.max_attempts, 9);
|
||||||
|
assert_eq!(stamped.visibility_timeout_s, 300);
|
||||||
|
assert_eq!(stamped.backoff_base_s, 5);
|
||||||
|
assert_eq!(stamped.dead_letter_retention_s, None);
|
||||||
|
let plain = EnqueueOpts::default();
|
||||||
|
assert_eq!(stamps_with_override(base, &plain).max_attempts, 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn delay_wins_over_run_at_and_neither_resolves_to_now() {
|
||||||
|
let now = 1_000_000;
|
||||||
|
let both = EnqueueOpts {
|
||||||
|
delay: Some(10),
|
||||||
|
run_at: Some(now + 500),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
assert_eq!(resolve_enqueue_opts(now, &both).run_at, now + 10);
|
||||||
|
let absolute = EnqueueOpts {
|
||||||
|
run_at: Some(now + 500),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
assert_eq!(resolve_enqueue_opts(now, &absolute).run_at, now + 500);
|
||||||
|
let neither = EnqueueOpts::default();
|
||||||
|
assert_eq!(resolve_enqueue_opts(now, &neither).run_at, now);
|
||||||
|
let negative = EnqueueOpts {
|
||||||
|
delay: Some(-5),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
resolve_enqueue_opts(now, &negative).run_at,
|
||||||
|
now - 5,
|
||||||
|
"a negative delay is a ready time in the past — total, ready now"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn expires_resolves_relative_to_enqueue() {
|
||||||
|
let now = 1_000_000;
|
||||||
|
let opts = EnqueueOpts {
|
||||||
|
expires: Some(60),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
assert_eq!(resolve_enqueue_opts(now, &opts).expires_at, Some(now + 60));
|
||||||
|
let none = EnqueueOpts::default();
|
||||||
|
assert_eq!(resolve_enqueue_opts(now, &none).expires_at, None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn backoff_lands_in_the_pinned_equal_jitter_range() {
|
||||||
|
for attempts in [1i64, 2, 3, 5, 30] {
|
||||||
|
let expected_cap = 5i64
|
||||||
|
.saturating_mul(1i64 << (attempts - 1).min(30))
|
||||||
|
.min(3600);
|
||||||
|
let half = (expected_cap + 1) / 2;
|
||||||
|
for id in [1i64, 42, 777, 123_456] {
|
||||||
|
let delay = backoff_delay_s(5, attempts, id);
|
||||||
|
assert!(
|
||||||
|
(half..=expected_cap).contains(&delay),
|
||||||
|
"attempt {attempts} id {id}: delay {delay} outside [{half}, {expected_cap}]"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn backoff_caps_at_one_hour_and_spreads() {
|
||||||
|
let draws: Vec<i64> = (0..400).map(|i| backoff_delay_s(5, 40, i)).collect();
|
||||||
|
assert!(
|
||||||
|
draws.iter().all(|d| (1800..=3600).contains(d)),
|
||||||
|
"capped range"
|
||||||
|
);
|
||||||
|
assert!(draws.windows(2).any(|w| w[0] != w[1]), "the jitter spreads");
|
||||||
|
assert!(draws.iter().any(|d| *d < 2700) && draws.iter().any(|d| *d > 2700));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,765 @@
|
|||||||
|
//! The scheduler mechanism's mem arm — the collapsed scheduler
|
||||||
|
//! ([ADR-009](../../docs/architecture/decisions/009-scheduler-collapse.md)):
|
||||||
|
//! two registrations + one opt-in runner, the leadership lock through
|
||||||
|
//! the locks registry (engine-uniform — single-instance leadership is
|
||||||
|
//! trivially owned but the machinery still runs), and the tick's
|
||||||
|
//! guarded fire-and-advance.
|
||||||
|
//!
|
||||||
|
//! - `schedule(name, spec, queue, payload, opts)` — upsert by name;
|
||||||
|
//! *both* name-bearing arguments validate at the entry point
|
||||||
|
//! (schedule name and queue identically, `InvalidName`/`ReservedName`
|
||||||
|
//! — ADR-021 §3: a schedule cannot target a reserved name); the spec
|
||||||
|
//! parser rejects everything v1 does not accept (`InvalidSpec`,
|
||||||
|
//! before storage, no round trip). The payload encodes once here;
|
||||||
|
//! the stamps resolve over the plain-queue derived defaults with
|
||||||
|
//! `ScheduleOpts::max_attempts` as the one override (ADR-020 §3) and
|
||||||
|
//! the spec's interval carries pre-parsed — the fire path reads the
|
||||||
|
//! registered values, never the raw spec.
|
||||||
|
//! - `unschedule(name)` — remove; `true` = a row was removed.
|
||||||
|
//! - `run_schedules(stop)` — acquire the leadership lock
|
||||||
|
//! (`SCHEDULER_LEADER_LOCK`, an engine-internal TTL, a runner-unique
|
||||||
|
//! owner), then loop {renew — on loss return `Err(LeadershipLost)`
|
||||||
|
//! *before ticking*; fire every due boundary; sleep until the next
|
||||||
|
//! due boundary or `stop`}. A cancelled token ends the sleep with
|
||||||
|
//! the clean `Ok(())` and releases the lease.
|
||||||
|
//! - The tick's fire+advance is one guarded operation (the
|
||||||
|
//! no-double-fire floor, ADR-009 §4: fire and boundary advance
|
||||||
|
//! together under the schedules→queues guard order — a boundary
|
||||||
|
//! observed mid-tick is either fully fired or fully unfired):
|
||||||
|
//! boundary-by-boundary replay up to the fixed **64-per-task cap**,
|
||||||
|
//! beyond which the schedule skips forward past the gap
|
||||||
|
//! (never a 10,000-job catch-up stacking from a week's downtime).
|
||||||
|
//! Fired jobs are ordinary queue rows — ready at the fire instant,
|
||||||
|
//! the registered stamps, `ScheduleOpts::{priority, expires}` riding
|
||||||
|
//! themselves — claimable via the ordinary queue surface.
|
||||||
|
//! - Boundary waits ride tokio time (the wasm-supported subset); the
|
||||||
|
//! sleep renews the lease across itself so an idle store never
|
||||||
|
//! loses leadership by drifting past the TTL.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use alkstore::{
|
||||||
|
BoxedFuture, Error, Result, Schedule, ScheduleOpts, StopToken, validate_shared_name,
|
||||||
|
};
|
||||||
|
|
||||||
|
use crate::clock::Clock;
|
||||||
|
use crate::lock::{MemLock, SCHEDULER_LEADER_LOCK};
|
||||||
|
use crate::queue::build_job_row;
|
||||||
|
use crate::resolution::{Stamps, plain_queue_default_stamps, stamps_with_override};
|
||||||
|
use crate::state::Guard;
|
||||||
|
use crate::state::{QueueState, ScheduleRow, ScheduleState, State};
|
||||||
|
|
||||||
|
/// The leadership lock's TTL (seconds): engine-internal detail —
|
||||||
|
/// renewed every loop iteration *and* across the sleeps (a runner that
|
||||||
|
/// cannot renew has lost the lock).
|
||||||
|
const LEADER_TTL_S: i64 = 10;
|
||||||
|
|
||||||
|
/// Seconds between lease renewals during the sleep.
|
||||||
|
const SLEEP_RENEW_EVERY_S: u64 = 2;
|
||||||
|
|
||||||
|
/// The idle sleep when nothing is scheduled (soonest = 0).
|
||||||
|
const IDLE_SLEEP_S: u64 = 60;
|
||||||
|
|
||||||
|
/// The bounded catch-up cap (ADR-009 §4 — a fixed contract constant,
|
||||||
|
/// honker parity, not a knob).
|
||||||
|
const MAX_CATCHUP_FIRES: i64 = 64;
|
||||||
|
|
||||||
|
/// Runner-unique leadership owner counter: the owner string is the
|
||||||
|
/// claim's identity token (a shared constant across runners would make
|
||||||
|
/// the owner-match mistake a foreign leader for self).
|
||||||
|
static LEADER_INSTANCE: AtomicU64 = AtomicU64::new(0);
|
||||||
|
|
||||||
|
fn leader_owner() -> String {
|
||||||
|
let instance = LEADER_INSTANCE.fetch_add(1, Ordering::Relaxed);
|
||||||
|
let nanos = std::time::SystemTime::now()
|
||||||
|
.duration_since(std::time::UNIX_EPOCH)
|
||||||
|
.map(|d| d.subsec_nanos())
|
||||||
|
.unwrap_or(0);
|
||||||
|
format!("scheduler-leader-{instance}-{nanos}")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The v1 spec grammar (`ADR-009` §2): exactly `@every <n><unit>` with
|
||||||
|
/// `<n>` a positive integer and `<unit>` in `s|m|h|d` — mem's own
|
||||||
|
/// parser (third owner per ADR-012 §2; the grammar is small and
|
||||||
|
/// pinned). Everything else — 5-field cron, seconds-field cron —
|
||||||
|
/// rejects as `InvalidSpec`.
|
||||||
|
pub(crate) fn parse_every_interval(spec: &str) -> Result<i64> {
|
||||||
|
let Some(body) = spec.strip_prefix("@every ") else {
|
||||||
|
return Err(invalid_spec(spec));
|
||||||
|
};
|
||||||
|
if body.is_empty() {
|
||||||
|
return Err(invalid_spec(spec));
|
||||||
|
}
|
||||||
|
let digits = body.chars().take_while(|c| c.is_ascii_digit()).count();
|
||||||
|
if digits == 0 || digits == body.len() {
|
||||||
|
return Err(invalid_spec(spec));
|
||||||
|
}
|
||||||
|
let n: i64 = body[..digits].parse().map_err(|_| invalid_spec(spec))?;
|
||||||
|
if n <= 0 {
|
||||||
|
return Err(invalid_spec(spec));
|
||||||
|
}
|
||||||
|
let mult = match &body[digits..] {
|
||||||
|
"s" => 1,
|
||||||
|
"m" => 60,
|
||||||
|
"h" => 3600,
|
||||||
|
"d" => 86400,
|
||||||
|
_ => return Err(invalid_spec(spec)),
|
||||||
|
};
|
||||||
|
n.checked_mul(mult).ok_or_else(|| invalid_spec(spec))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn invalid_spec(spec: &str) -> Error {
|
||||||
|
Error::InvalidSpec {
|
||||||
|
spec: spec.to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The auto-commit register path: pre-storage validation, the payload
|
||||||
|
/// encode (one owner of those bytes for every later fire), the stamps
|
||||||
|
/// resolution, the upsert, the [`Schedule`] read-back.
|
||||||
|
pub(crate) fn schedule(
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
name: &str,
|
||||||
|
spec: &str,
|
||||||
|
queue: &str,
|
||||||
|
payload: Value,
|
||||||
|
opts: ScheduleOpts,
|
||||||
|
) -> BoxedFuture<'static, Result<Schedule>> {
|
||||||
|
if let Err(e) = validate_shared_name(name) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if let Err(e) = validate_shared_name(queue) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let interval = match parse_every_interval(spec) {
|
||||||
|
Ok(interval) => interval,
|
||||||
|
Err(e) => return Box::pin(async move { Err(e) }),
|
||||||
|
};
|
||||||
|
let encoded = alkstore::encode_payload(&payload);
|
||||||
|
let name = name.to_string();
|
||||||
|
let spec = spec.to_string();
|
||||||
|
let queue = queue.to_string();
|
||||||
|
let read_back = Schedule::new(name.clone(), spec.clone(), queue.clone(), opts.clone());
|
||||||
|
Box::pin(async move {
|
||||||
|
let bytes = encoded?;
|
||||||
|
let now = clock.unix_now();
|
||||||
|
let stamps = stamps_with_override(
|
||||||
|
plain_queue_default_stamps(),
|
||||||
|
&alkstore::EnqueueOpts {
|
||||||
|
max_attempts: opts.max_attempts,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
);
|
||||||
|
let row = ScheduleRow {
|
||||||
|
queue,
|
||||||
|
interval_s: interval,
|
||||||
|
payload: bytes,
|
||||||
|
priority: opts.priority,
|
||||||
|
expires_s: opts.expires,
|
||||||
|
max_attempts: stamps.max_attempts,
|
||||||
|
visibility_timeout_s: stamps.visibility_timeout_s,
|
||||||
|
backoff_base_s: stamps.backoff_base_s,
|
||||||
|
dead_letter_retention_s: stamps.dead_letter_retention_s,
|
||||||
|
next_fire_at: now.saturating_add(interval),
|
||||||
|
};
|
||||||
|
{
|
||||||
|
let mut sched = state.schedules.guard();
|
||||||
|
sched.rows.insert(name, row);
|
||||||
|
}
|
||||||
|
Ok(read_back)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The auto-commit unregister path (validation mirrors `schedule`'s —
|
||||||
|
/// a reserved/empty schedule name never reaches the table).
|
||||||
|
pub(crate) fn unschedule(state: Arc<State>, name: &str) -> BoxedFuture<'static, Result<bool>> {
|
||||||
|
if let Err(e) = validate_shared_name(name) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let removed = {
|
||||||
|
let mut sched = state.schedules.guard();
|
||||||
|
sched.rows.remove(name).is_some()
|
||||||
|
};
|
||||||
|
Box::pin(async move { Ok(removed) })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One tick under the canonical schedules→queues guard order: fire
|
||||||
|
/// every due boundary (bounded catch-up with the 64 cap, skip-forward
|
||||||
|
/// past the cap) and advance the schedule rows atomically with the
|
||||||
|
/// fires. Returns the soonest next fire across the table (0 = none).
|
||||||
|
pub(crate) fn run_tick(state: &State, now: i64) -> i64 {
|
||||||
|
let mut sched = state.schedules.guard();
|
||||||
|
let mut queues = state.queues.guard();
|
||||||
|
fire_due(&mut sched, &mut queues, state, now);
|
||||||
|
soonest(&sched)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fire_due(sched: &mut ScheduleState, queues: &mut QueueState, state: &State, now: i64) {
|
||||||
|
for row in sched.rows.values_mut() {
|
||||||
|
if row.next_fire_at > now {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let interval = row.interval_s;
|
||||||
|
let expires_at = row.expires_s.map(|s| now.saturating_add(s));
|
||||||
|
let stamps = Stamps {
|
||||||
|
max_attempts: row.max_attempts,
|
||||||
|
visibility_timeout_s: row.visibility_timeout_s,
|
||||||
|
backoff_base_s: row.backoff_base_s,
|
||||||
|
dead_letter_retention_s: row.dead_letter_retention_s,
|
||||||
|
};
|
||||||
|
let mut next_fire_at = row.next_fire_at;
|
||||||
|
let mut fires = 0;
|
||||||
|
while next_fire_at <= now {
|
||||||
|
if fires >= MAX_CATCHUP_FIRES {
|
||||||
|
next_fire_at = skip_forward(next_fire_at, now, interval);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let id = state.allocate_job_id();
|
||||||
|
let resolved = crate::resolution::ResolvedEnqueue {
|
||||||
|
run_at: now,
|
||||||
|
expires_at,
|
||||||
|
};
|
||||||
|
let built = build_job_row(
|
||||||
|
id,
|
||||||
|
row.queue.clone(),
|
||||||
|
row.payload.clone(),
|
||||||
|
now,
|
||||||
|
&resolved,
|
||||||
|
row.priority,
|
||||||
|
&stamps,
|
||||||
|
);
|
||||||
|
queues.live.insert(id, built);
|
||||||
|
fires += 1;
|
||||||
|
next_fire_at = next_fire_at.checked_add(interval).unwrap_or(i64::MAX);
|
||||||
|
}
|
||||||
|
row.next_fire_at = next_fire_at;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The bounded catch-up's skip-forward: land the next boundary strictly
|
||||||
|
/// in the future, grid-aligned (the schedule row's advance makes the
|
||||||
|
/// gap inspectable).
|
||||||
|
fn skip_forward(next_fire_at: i64, now: i64, interval: i64) -> i64 {
|
||||||
|
let gap = (now - next_fire_at) as i128;
|
||||||
|
let interval = interval as i128;
|
||||||
|
let next = (next_fire_at as i128) + (gap / interval + 1) * interval;
|
||||||
|
if next > i64::MAX as i128 {
|
||||||
|
i64::MAX
|
||||||
|
} else {
|
||||||
|
next as i64
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The soonest boundary across the table (0 = none scheduled).
|
||||||
|
pub(crate) fn soonest(sched: &ScheduleState) -> i64 {
|
||||||
|
sched
|
||||||
|
.rows
|
||||||
|
.values()
|
||||||
|
.map(|r| r.next_fire_at)
|
||||||
|
.min()
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The leader loop (see the module docs).
|
||||||
|
pub(crate) fn run_schedules(
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
stop: StopToken,
|
||||||
|
) -> BoxedFuture<'static, Result<()>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
let lease = MemLock::new_internal(
|
||||||
|
SCHEDULER_LEADER_LOCK,
|
||||||
|
&leader_owner(),
|
||||||
|
state.clone(),
|
||||||
|
clock.clone(),
|
||||||
|
);
|
||||||
|
// Step 1: acquire the leadership lock. A refused acquire —
|
||||||
|
// someone else leads — is `Err(LeadershipLost)` at start: no
|
||||||
|
// leader loop exists without the lock, and the caller's recipe
|
||||||
|
// (respawn) is identical to a mid-run loss.
|
||||||
|
let now = clock.unix_now();
|
||||||
|
if !lease.acquire_now(LEADER_TTL_S, now) {
|
||||||
|
return Err(Error::LeadershipLost);
|
||||||
|
}
|
||||||
|
loop {
|
||||||
|
if stop.is_cancelled() {
|
||||||
|
let _ = lease.release_now();
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let now = clock.unix_now();
|
||||||
|
if !lease.renew_at(LEADER_TTL_S, now) {
|
||||||
|
return Err(Error::LeadershipLost);
|
||||||
|
}
|
||||||
|
let soonest_at = run_tick(&state, now);
|
||||||
|
if stop.is_cancelled() {
|
||||||
|
let _ = lease.release_now();
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
keep_awake_renewing(&lease, clock.as_ref(), &stop, soonest_at).await;
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The sliced sleep until the next due boundary (or the idle horizon)
|
||||||
|
/// or the stop token — the loop's wake path between ticks. The lease
|
||||||
|
/// renews on cadence across the sleep: a boundary further out than the
|
||||||
|
/// TTL must not lose the lock by drifting past it (the leader-loop
|
||||||
|
/// pattern: renew while holding).
|
||||||
|
async fn keep_awake_renewing(lease: &MemLock, clock: &Clock, stop: &StopToken, soonest_at: i64) {
|
||||||
|
let renew_every = Duration::from_secs(SLEEP_RENEW_EVERY_S);
|
||||||
|
let mut next_renew = tokio::time::Instant::now() + renew_every;
|
||||||
|
let started = tokio::time::Instant::now();
|
||||||
|
let now = clock.unix_now();
|
||||||
|
let wait_s: u64 = if soonest_at <= 0 {
|
||||||
|
IDLE_SLEEP_S
|
||||||
|
} else {
|
||||||
|
soonest_at.saturating_sub(now).max(1) as u64
|
||||||
|
};
|
||||||
|
let deadline = started + Duration::from_secs(wait_s);
|
||||||
|
loop {
|
||||||
|
let now = tokio::time::Instant::now();
|
||||||
|
if stop.is_cancelled() || now >= deadline {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if now >= next_renew {
|
||||||
|
// A failed mid-sleep renewal is not acted on here — the
|
||||||
|
// loop's own top-of-iteration renew decides (one owner of
|
||||||
|
// the loss decision, before any tick).
|
||||||
|
let _ = lease.renew_at(LEADER_TTL_S, clock.unix_now());
|
||||||
|
next_renew = tokio::time::Instant::now() + renew_every;
|
||||||
|
}
|
||||||
|
let slice = deadline.min(next_renew);
|
||||||
|
tokio::time::sleep_until(slice).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::{Queue, Store};
|
||||||
|
|
||||||
|
fn store() -> MemStore {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000_000);
|
||||||
|
store
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parser_accepts_only_the_v1_grammar() {
|
||||||
|
for (spec, expected) in [
|
||||||
|
("@every 1s", 1),
|
||||||
|
("@every 30s", 30),
|
||||||
|
("@every 5m", 300),
|
||||||
|
("@every 2h", 7200),
|
||||||
|
("@every 1d", 86_400),
|
||||||
|
("@every 1000000s", 1_000_000),
|
||||||
|
] {
|
||||||
|
assert_eq!(parse_every_interval(spec).unwrap(), expected, "{spec}");
|
||||||
|
}
|
||||||
|
for bad in [
|
||||||
|
"0 0 * * *",
|
||||||
|
"0 0 0 * * *",
|
||||||
|
"@every",
|
||||||
|
"@every ",
|
||||||
|
"@every 0s",
|
||||||
|
"@every -5s",
|
||||||
|
"@every 5",
|
||||||
|
"@every s",
|
||||||
|
"@every 5x",
|
||||||
|
"@every 5sm",
|
||||||
|
"@every 1.5s",
|
||||||
|
"@every 5s",
|
||||||
|
"@every5s",
|
||||||
|
"every 5s",
|
||||||
|
"@every 99999999999999999999s",
|
||||||
|
] {
|
||||||
|
let err = parse_every_interval(bad).expect_err(bad);
|
||||||
|
assert!(matches!(err, Error::InvalidSpec { .. }), "{bad} → {err:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn register_upserts_resolves_stamps_and_reads_back() {
|
||||||
|
let store = store();
|
||||||
|
let registered = store
|
||||||
|
.schedule(
|
||||||
|
"sweep",
|
||||||
|
"@every 300s",
|
||||||
|
"maintenance",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(registered.name, "sweep");
|
||||||
|
assert_eq!(registered.spec, "@every 300s");
|
||||||
|
assert_eq!(registered.queue, "maintenance");
|
||||||
|
let replaced = store
|
||||||
|
.schedule(
|
||||||
|
"sweep",
|
||||||
|
"@every 60s",
|
||||||
|
"maintenance2",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(replaced.spec, "@every 60s", "upsert replaces the whole row");
|
||||||
|
{
|
||||||
|
let sched = store.state.schedules.guard();
|
||||||
|
assert_eq!(sched.rows.len(), 1);
|
||||||
|
let row = sched.rows.get("sweep").unwrap();
|
||||||
|
assert_eq!(row.interval_s, 60);
|
||||||
|
assert_eq!(
|
||||||
|
row.next_fire_at, 1_000_060,
|
||||||
|
"the first boundary is one interval out"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
row.payload,
|
||||||
|
serde_json::to_vec(&serde_json::json!(1)).unwrap()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let with_opts = store
|
||||||
|
.schedule(
|
||||||
|
"opts",
|
||||||
|
"@every 5s",
|
||||||
|
"maintenance",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts {
|
||||||
|
priority: 7,
|
||||||
|
max_attempts: Some(9),
|
||||||
|
expires: Some(120),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(with_opts.opts.priority, 7);
|
||||||
|
let sched = store.state.schedules.guard();
|
||||||
|
let row = sched.rows.get("opts").unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
row.max_attempts, 9,
|
||||||
|
"the schedule's max_attempts resolves over the defaults"
|
||||||
|
);
|
||||||
|
assert_eq!(row.visibility_timeout_s, 300);
|
||||||
|
assert_eq!(row.backoff_base_s, 5);
|
||||||
|
assert_eq!(row.dead_letter_retention_s, None);
|
||||||
|
assert_eq!(row.priority, 7);
|
||||||
|
assert_eq!(row.expires_s, Some(120));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn register_rejects_bad_names_and_specs_pre_storage() {
|
||||||
|
let store = store();
|
||||||
|
for (bad_name, bad_queue, bad_spec) in [
|
||||||
|
("", "q", "@every 5s"),
|
||||||
|
("__alkstore_s", "q", "@every 5s"),
|
||||||
|
("ok", "", "@every 5s"),
|
||||||
|
("ok", "__alkstore_q", "@every 5s"),
|
||||||
|
("ok", "q", "0 0 * * *"),
|
||||||
|
] {
|
||||||
|
let err = store
|
||||||
|
.schedule(
|
||||||
|
bad_name,
|
||||||
|
bad_spec,
|
||||||
|
bad_queue,
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect_err("register rejects");
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
err,
|
||||||
|
Error::InvalidName { .. }
|
||||||
|
| Error::ReservedName { .. }
|
||||||
|
| Error::InvalidSpec { .. }
|
||||||
|
),
|
||||||
|
"{bad_name}/{bad_queue}/{bad_spec} → {err:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
store.state.schedules.guard().rows.is_empty(),
|
||||||
|
"nothing stored"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn unschedule_removes_and_reports_absence() {
|
||||||
|
let store = store();
|
||||||
|
store
|
||||||
|
.schedule(
|
||||||
|
"sweep",
|
||||||
|
"@every 60s",
|
||||||
|
"maintenance",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(store.unschedule("sweep").await.unwrap());
|
||||||
|
assert!(!store.unschedule("sweep").await.unwrap(), "absent = false");
|
||||||
|
let err = store.unschedule("__alkstore_x").await.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
matches!(err, Error::ReservedName { .. }),
|
||||||
|
"the reserved name never round trips"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn tick_fires_read_registered_values_and_never_double_fire() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000_000);
|
||||||
|
store
|
||||||
|
.schedule(
|
||||||
|
"job",
|
||||||
|
"@every 10s",
|
||||||
|
"work",
|
||||||
|
serde_json::json!({"task": "sweep"}),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(run_tick(&store.state, 1_000_005), 1_000_010);
|
||||||
|
assert!(store.state.queues.guard().live.is_empty());
|
||||||
|
let soonest = run_tick(&store.state, 1_000_013);
|
||||||
|
let rows: Vec<crate::state::JobRow> =
|
||||||
|
store.state.queues.guard().live.values().cloned().collect();
|
||||||
|
assert_eq!(rows.len(), 1);
|
||||||
|
let fire = &rows[0];
|
||||||
|
assert_eq!(fire.queue, "work");
|
||||||
|
assert_eq!(
|
||||||
|
fire.run_at, 1_000_013,
|
||||||
|
"a boundary fire is ready at the fire instant"
|
||||||
|
);
|
||||||
|
assert_eq!(fire.priority, 0);
|
||||||
|
assert_eq!(
|
||||||
|
fire.visibility_timeout_s, 300,
|
||||||
|
"the plain defaults stamp fired rows"
|
||||||
|
);
|
||||||
|
assert_eq!(fire.max_attempts, 3);
|
||||||
|
assert_eq!(fire.created_at, 1_000_013);
|
||||||
|
assert_eq!(
|
||||||
|
fire.payload,
|
||||||
|
serde_json::to_vec(&serde_json::json!({"task": "sweep"})).unwrap()
|
||||||
|
);
|
||||||
|
assert_eq!(soonest, 1_000_020, "the boundary advanced with the fire");
|
||||||
|
assert_eq!(run_tick(&store.state, 1_000_015), 1_000_020);
|
||||||
|
assert_eq!(
|
||||||
|
store.state.queues.guard().live.len(),
|
||||||
|
1,
|
||||||
|
"the no-double-fire floor"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn catchup_replays_boundary_by_boundary_up_to_the_cap_then_skips_forward() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000_000);
|
||||||
|
{
|
||||||
|
let mut sched = store.state.schedules.guard();
|
||||||
|
let mut seed = |name: &str, interval: i64| {
|
||||||
|
sched.rows.insert(
|
||||||
|
name.to_string(),
|
||||||
|
crate::state::ScheduleRow {
|
||||||
|
queue: "work".to_string(),
|
||||||
|
interval_s: interval,
|
||||||
|
payload: serde_json::to_vec(&serde_json::json!(1)).unwrap(),
|
||||||
|
priority: 0,
|
||||||
|
expires_s: None,
|
||||||
|
max_attempts: 3,
|
||||||
|
visibility_timeout_s: 300,
|
||||||
|
backoff_base_s: 5,
|
||||||
|
dead_letter_retention_s: None,
|
||||||
|
next_fire_at: 1_000_000 + interval,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
};
|
||||||
|
seed("job", 10);
|
||||||
|
seed("neverdue", 3600);
|
||||||
|
}
|
||||||
|
assert_eq!(run_tick(&store.state, 1_003_000), 1_003_010);
|
||||||
|
assert_eq!(
|
||||||
|
store.state.queues.guard().live.len(),
|
||||||
|
64,
|
||||||
|
"the bounded catch-up: 64 fires, never 300"
|
||||||
|
);
|
||||||
|
let row = store
|
||||||
|
.state
|
||||||
|
.schedules
|
||||||
|
.guard()
|
||||||
|
.rows
|
||||||
|
.get("job")
|
||||||
|
.unwrap()
|
||||||
|
.clone();
|
||||||
|
assert!(
|
||||||
|
row.next_fire_at > 1_003_000,
|
||||||
|
"skip-forward lands the next boundary strictly in the future ({})",
|
||||||
|
row.next_fire_at
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
(row.next_fire_at - 1_000_010) % 10,
|
||||||
|
0,
|
||||||
|
"the grid stays honest"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
store
|
||||||
|
.state
|
||||||
|
.schedules
|
||||||
|
.guard()
|
||||||
|
.rows
|
||||||
|
.get("neverdue")
|
||||||
|
.unwrap()
|
||||||
|
.next_fire_at,
|
||||||
|
1_003_600,
|
||||||
|
"the never-due schedule is untouched"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn fire_and_advance_commit_together_under_one_guarded_op() {
|
||||||
|
let store = store();
|
||||||
|
store
|
||||||
|
.schedule(
|
||||||
|
"job",
|
||||||
|
"@every 10s",
|
||||||
|
"work",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
// The tick itself is the guarded operation: assert both effects
|
||||||
|
// landed together and the soonest is consistent with the rows.
|
||||||
|
let soonest = run_tick(&store.state, 1_000_000 + 10);
|
||||||
|
{
|
||||||
|
let sched = store.state.schedules.guard();
|
||||||
|
let queues = store.state.queues.guard();
|
||||||
|
assert_eq!(sched.rows.get("job").unwrap().next_fire_at, 1_000_020);
|
||||||
|
assert_eq!(queues.live.len(), 1);
|
||||||
|
}
|
||||||
|
assert_eq!(soonest, 1_000_020);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn leadership_is_won_once_and_the_loser_is_leadership_lost() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000_000);
|
||||||
|
store
|
||||||
|
.schedule(
|
||||||
|
"job",
|
||||||
|
"@every 1s",
|
||||||
|
"work",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let stop = StopToken::new();
|
||||||
|
let leader_stop = stop.clone();
|
||||||
|
let leader = tokio::spawn(super::super::scheduler::run_schedules(
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
leader_stop,
|
||||||
|
));
|
||||||
|
// Let the first runner acquire before the second contends.
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
let loser = tokio::spawn(super::super::scheduler::run_schedules(
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
StopToken::new(),
|
||||||
|
));
|
||||||
|
let loser_out = tokio::time::timeout(Duration::from_secs(2), loser)
|
||||||
|
.await
|
||||||
|
.expect("the contender resolves quickly")
|
||||||
|
.expect("the loser runner exits");
|
||||||
|
assert!(
|
||||||
|
matches!(loser_out, Err(Error::LeadershipLost)),
|
||||||
|
"the contender loses cleanly: {loser_out:?}"
|
||||||
|
);
|
||||||
|
stop.cancel();
|
||||||
|
let leader_out = tokio::time::timeout(Duration::from_secs(3), leader)
|
||||||
|
.await
|
||||||
|
.expect("the leader resolves after stop")
|
||||||
|
.expect("the leader runner exits");
|
||||||
|
assert!(
|
||||||
|
matches!(leader_out, Ok(())),
|
||||||
|
"a cancelled token is the clean-stop return"
|
||||||
|
);
|
||||||
|
// The lease released on the clean stop: a new runner acquires
|
||||||
|
// the leadership lock cleanly again.
|
||||||
|
let re_leader = MemLock::new_internal(
|
||||||
|
SCHEDULER_LEADER_LOCK,
|
||||||
|
"fresh-runner",
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
re_leader.acquire_now(10, store.clock.unix_now()),
|
||||||
|
"the clean stop released the lease"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn run_schedules_returns_leadership_lost_without_a_free_lock() {
|
||||||
|
let store = store();
|
||||||
|
let lease = MemLock::new_internal(
|
||||||
|
SCHEDULER_LEADER_LOCK,
|
||||||
|
"usurper",
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
);
|
||||||
|
assert!(lease.acquire_now(10, store.clock.unix_now()));
|
||||||
|
let out = super::super::scheduler::run_schedules(
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
StopToken::new(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
assert!(matches!(out, Err(Error::LeadershipLost)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn fired_rows_are_ordinary_queue_work() {
|
||||||
|
let store = store();
|
||||||
|
store
|
||||||
|
.schedule(
|
||||||
|
"job",
|
||||||
|
"@every 5s",
|
||||||
|
"work",
|
||||||
|
serde_json::json!(1),
|
||||||
|
ScheduleOpts::default(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
run_tick(&store.state, 1_000_005);
|
||||||
|
store.clock.freeze(1_000_005);
|
||||||
|
let id = {
|
||||||
|
let q = store.state.queues.guard();
|
||||||
|
*q.live.keys().next().unwrap()
|
||||||
|
};
|
||||||
|
let queue = crate::queue::MemQueueHandle::new(
|
||||||
|
"work".to_string(),
|
||||||
|
alkstore::QueueOpts::default(),
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
);
|
||||||
|
let claimed = queue
|
||||||
|
.claim_one("w")
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.expect("the fired job claims");
|
||||||
|
assert_eq!(claimed.job().id, id);
|
||||||
|
assert_eq!(claimed.job().queue, "work");
|
||||||
|
assert_eq!(
|
||||||
|
claimed.job().visibility_timeout_s,
|
||||||
|
300,
|
||||||
|
"the plain defaults stamp fired rows"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
+152
-23
@@ -14,15 +14,19 @@
|
|||||||
//! [`Guard::guard`] fold into the inner value keeps the type honest
|
//! [`Guard::guard`] fold into the inner value keeps the type honest
|
||||||
//! anyway.
|
//! anyway.
|
||||||
//!
|
//!
|
||||||
//! This module owns the shared scaffolding only: the container, the
|
//! Section shapes are the commit-merge targets ([`crate::tx`]'s
|
||||||
//! section shapes' names, and the guard idiom. Each mechanism's table
|
//! overlay stages effects of exactly these rows). Row-level operations
|
||||||
//! shape is owned by its own wave-6 task (`mem-engine-queues`,
|
//! live with their mechanisms (`crate::queue`, `crate::stream`,
|
||||||
//! `mem-engine-streams`, `mem-engine-locks`,
|
//! `crate::lock`, `crate::scheduler`) — modules must take section
|
||||||
//! `mem-engine-scheduler-outbox`) — the sections here start empty and
|
//! guards in the canonical order **schedules → queues → streams**
|
||||||
//! grow there.
|
//! (locks and wakes depend on no other section; the commit-merge takes
|
||||||
|
//! queues → streams, the scheduler tick schedules → queues) so no path
|
||||||
|
//! can deadlock.
|
||||||
|
|
||||||
|
use std::collections::{BTreeMap, HashMap};
|
||||||
use std::sync::Mutex;
|
use std::sync::Mutex;
|
||||||
use std::sync::MutexGuard;
|
use std::sync::MutexGuard;
|
||||||
|
use std::sync::atomic::{AtomicI64, Ordering};
|
||||||
|
|
||||||
/// A short-held synchronous internal lock: `guard()` is the one access
|
/// A short-held synchronous internal lock: `guard()` is the one access
|
||||||
/// idiom (see the module docs for the discipline).
|
/// idiom (see the module docs for the discipline).
|
||||||
@@ -36,37 +40,140 @@ impl<T> Guard<T> for Mutex<T> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A live job row (ADR-010's state machine, minus the dead state —
|
||||||
|
/// dead rows move to [`QueueState::dead`] with their diagnosis). The
|
||||||
|
/// row carries the resolved enqueue fields and the §3a stamps; the
|
||||||
|
/// `pending` flag is the `pending`/`processing` split (absence = acked
|
||||||
|
/// or cancelled).
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub(crate) struct JobRow {
|
||||||
|
pub(crate) id: i64,
|
||||||
|
pub(crate) queue: String,
|
||||||
|
pub(crate) payload: Vec<u8>,
|
||||||
|
pub(crate) pending: bool,
|
||||||
|
pub(crate) priority: i64,
|
||||||
|
pub(crate) run_at: i64,
|
||||||
|
pub(crate) worker_id: Option<String>,
|
||||||
|
pub(crate) claim_expires_at: Option<i64>,
|
||||||
|
pub(crate) claimed_at: Option<i64>,
|
||||||
|
pub(crate) attempts: i64,
|
||||||
|
pub(crate) max_attempts: i64,
|
||||||
|
pub(crate) created_at: i64,
|
||||||
|
pub(crate) expires_at: Option<i64>,
|
||||||
|
pub(crate) visibility_timeout_s: i64,
|
||||||
|
pub(crate) backoff_base_s: i64,
|
||||||
|
pub(crate) dead_letter_retention_s: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A dead-lettered job row (ADR-010 §4): the live row's durable fields
|
||||||
|
/// plus the diagnosis (`last_error`, `died_at`) — `get_job` sees dead
|
||||||
|
/// rows with the stamps still attached. The claim identity
|
||||||
|
/// (`worker_id`/`claim_expires_at`) never rides a dead row.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub(crate) struct DeadRow {
|
||||||
|
pub(crate) id: i64,
|
||||||
|
pub(crate) queue: String,
|
||||||
|
pub(crate) payload: Vec<u8>,
|
||||||
|
pub(crate) priority: i64,
|
||||||
|
pub(crate) run_at: i64,
|
||||||
|
pub(crate) attempts: i64,
|
||||||
|
pub(crate) max_attempts: i64,
|
||||||
|
pub(crate) created_at: i64,
|
||||||
|
pub(crate) expires_at: Option<i64>,
|
||||||
|
pub(crate) claimed_at: Option<i64>,
|
||||||
|
pub(crate) visibility_timeout_s: i64,
|
||||||
|
pub(crate) backoff_base_s: i64,
|
||||||
|
pub(crate) dead_letter_retention_s: Option<i64>,
|
||||||
|
pub(crate) last_error: String,
|
||||||
|
pub(crate) died_at: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A committed stream event (ADR-015 §3's shape, engine-internal) —
|
||||||
|
/// `offset` is the per-stream log key, immutable, never renumbered.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub(crate) struct EventRow {
|
||||||
|
pub(crate) offset: i64,
|
||||||
|
pub(crate) key: Option<String>,
|
||||||
|
pub(crate) payload: Vec<u8>,
|
||||||
|
pub(crate) created_at: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A named TTL lock's registry entry (ADR-008 §7): owner identity +
|
||||||
|
/// deadline. Expiry is a clock check at the try/renew boundaries — no
|
||||||
|
/// ambient revoker exists.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub(crate) struct LockEntry {
|
||||||
|
pub(crate) owner: String,
|
||||||
|
pub(crate) expires_at: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A registered schedule row (ADR-009): the collapsed scheduler's
|
||||||
|
/// table. The interval is parsed and the stamps are resolved at
|
||||||
|
/// register time (the fire path stamps rows from the registered
|
||||||
|
/// values, never the raw spec) — `next_fire_at` is the collapse
|
||||||
|
/// machinery's boundary state.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub(crate) struct ScheduleRow {
|
||||||
|
pub(crate) queue: String,
|
||||||
|
pub(crate) interval_s: i64,
|
||||||
|
pub(crate) payload: Vec<u8>,
|
||||||
|
pub(crate) priority: i64,
|
||||||
|
pub(crate) expires_s: Option<i64>,
|
||||||
|
pub(crate) max_attempts: i64,
|
||||||
|
pub(crate) visibility_timeout_s: i64,
|
||||||
|
pub(crate) backoff_base_s: i64,
|
||||||
|
pub(crate) dead_letter_retention_s: Option<i64>,
|
||||||
|
pub(crate) next_fire_at: i64,
|
||||||
|
}
|
||||||
|
|
||||||
/// The queue mechanism's committed section: the jobs table and the
|
/// The queue mechanism's committed section: the jobs table and the
|
||||||
/// dead-letter storage (`mem-engine-queues` owns the shapes).
|
/// dead-letter storage. `live` keys by job id (the enqueue-order
|
||||||
|
/// claim tiebreak reads the id directly); `dead` keys the same id
|
||||||
|
/// space.
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub(crate) struct QueueState {}
|
pub(crate) struct QueueState {
|
||||||
|
pub(crate) live: BTreeMap<i64, JobRow>,
|
||||||
|
pub(crate) dead: BTreeMap<i64, DeadRow>,
|
||||||
|
}
|
||||||
|
|
||||||
/// The stream mechanism's committed section: event logs and
|
/// The stream mechanism's committed section: per-stream event logs
|
||||||
/// per-consumer offsets (`mem-engine-streams` owns the shapes).
|
/// (keyed by offset, ordered) and the per-(stream, consumer) offset
|
||||||
|
/// checkpoints.
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub(crate) struct StreamState {}
|
pub(crate) struct StreamState {
|
||||||
|
pub(crate) logs: BTreeMap<String, BTreeMap<i64, EventRow>>,
|
||||||
|
pub(crate) offsets: HashMap<(String, String), i64>,
|
||||||
|
}
|
||||||
|
|
||||||
/// The named-locks mechanism's committed section: the TTL registry
|
/// The named-locks mechanism's committed section: the TTL registry.
|
||||||
/// (`mem-engine-locks` owns the shapes).
|
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub(crate) struct LockState {}
|
pub(crate) struct LockState {
|
||||||
|
pub(crate) entries: HashMap<String, LockEntry>,
|
||||||
|
}
|
||||||
|
|
||||||
/// The scheduler mechanism's committed section: the schedule table
|
/// The scheduler mechanism's committed section: the schedule table.
|
||||||
/// (`mem-engine-scheduler-outbox` owns the shapes; the outbox rides
|
|
||||||
/// queues).
|
|
||||||
#[derive(Debug, Default)]
|
#[derive(Debug, Default)]
|
||||||
pub(crate) struct ScheduleState {}
|
pub(crate) struct ScheduleState {
|
||||||
|
pub(crate) rows: BTreeMap<String, ScheduleRow>,
|
||||||
|
}
|
||||||
|
|
||||||
/// The store's committed state: one guarded section per mechanism.
|
/// The store's committed state: one guarded section per mechanism, the
|
||||||
/// Sections are independent — a mechanism task locks only its own.
|
/// engine-wide id/offset allocators, and the clock read that every
|
||||||
// The sections are read by their owning wave-6 mechanism tasks; the
|
/// timestamp in every section defers to.
|
||||||
// foundations build alone sees them as dead.
|
///
|
||||||
#[allow(dead_code)]
|
/// `job_ids`/`event_offsets` are the allocators the tx overlay and the
|
||||||
|
/// auto-commit paths share: ids and offsets are assigned monotone at
|
||||||
|
/// allocation time (stage time for a tx, op time for auto-commit),
|
||||||
|
/// which is the pg-shaped story — an uncommitted allocation gaps out
|
||||||
|
/// when the tx rolls back (legal: ids are claim-order tiebreaks and
|
||||||
|
/// offsets are immutable, never renumbered; gaps are ordinary).
|
||||||
pub(crate) struct State {
|
pub(crate) struct State {
|
||||||
pub(crate) queues: Mutex<QueueState>,
|
pub(crate) queues: Mutex<QueueState>,
|
||||||
pub(crate) streams: Mutex<StreamState>,
|
pub(crate) streams: Mutex<StreamState>,
|
||||||
pub(crate) locks: Mutex<LockState>,
|
pub(crate) locks: Mutex<LockState>,
|
||||||
pub(crate) schedules: Mutex<ScheduleState>,
|
pub(crate) schedules: Mutex<ScheduleState>,
|
||||||
|
pub(crate) job_ids: AtomicI64,
|
||||||
|
pub(crate) event_offsets: AtomicI64,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl State {
|
impl State {
|
||||||
@@ -76,8 +183,21 @@ impl State {
|
|||||||
streams: Mutex::new(StreamState::default()),
|
streams: Mutex::new(StreamState::default()),
|
||||||
locks: Mutex::new(LockState::default()),
|
locks: Mutex::new(LockState::default()),
|
||||||
schedules: Mutex::new(ScheduleState::default()),
|
schedules: Mutex::new(ScheduleState::default()),
|
||||||
|
job_ids: AtomicI64::new(0),
|
||||||
|
event_offsets: AtomicI64::new(0),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The next job id — monotone, never reused (a rolled-back tx's
|
||||||
|
/// staged allocation gaps).
|
||||||
|
pub(crate) fn allocate_job_id(&self) -> i64 {
|
||||||
|
self.job_ids.fetch_add(1, Ordering::Relaxed) + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The next event offset — monotone, immutable, never renumbered.
|
||||||
|
pub(crate) fn allocate_event_offset(&self) -> i64 {
|
||||||
|
self.event_offsets.fetch_add(1, Ordering::Relaxed) + 1
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl std::fmt::Debug for State {
|
impl std::fmt::Debug for State {
|
||||||
@@ -130,4 +250,13 @@ mod tests {
|
|||||||
held.join().expect("the lock churn task never panics");
|
held.join().expect("the lock churn task never panics");
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn allocators_are_monotone_and_distinct() {
|
||||||
|
let state = State::new();
|
||||||
|
let ids: Vec<i64> = (0..8).map(|_| state.allocate_job_id()).collect();
|
||||||
|
let offsets: Vec<i64> = (0..8).map(|_| state.allocate_event_offset()).collect();
|
||||||
|
assert_eq!(ids, (1..=8).collect::<Vec<_>>(), "job ids count from 1");
|
||||||
|
assert_eq!(offsets, (1..=8).collect::<Vec<_>>(), "offsets count from 1");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+168
-25
@@ -1,40 +1,48 @@
|
|||||||
//! The mem engine's store: `MemStore::new`, the engine-native
|
//! The mem engine's store: `MemStore::new`, the engine-native
|
||||||
//! constructor (ADR-024 §4 — the one documented exception to the
|
//! constructor (ADR-024 §4 — the one documented exception to the
|
||||||
//! open prose), and the guarded state it owns.
|
//! open prose), and the guarded state it owns — plus the `Store` trait
|
||||||
|
//! surface this crate's whole point rides on.
|
||||||
//!
|
//!
|
||||||
//! Isolation per open: each `new()` yields a fully isolated instance
|
//! Isolation per open: each `new()` yields a fully isolated instance
|
||||||
//! sharing nothing — a fresh `State` (the guarded per-mechanism
|
//! sharing nothing — a fresh `State` (the guarded per-mechanism core),
|
||||||
//! core), a fresh `WakeBus` (the wake-routing substrate), a fresh
|
//! a fresh `WakeBus` (the wake-routing substrate), a fresh `Clock` —
|
||||||
//! `Clock` — exactly the suite factory's isolation contract;
|
//! exactly the suite factory's isolation contract; two instances never
|
||||||
//! two instances never contend and never observe each other. There is
|
//! contend and never observe each other. There is no connection, no
|
||||||
//! no connection, no file, no pool; teardown is drop — the
|
//! file, no pool; teardown is drop — the engine-wide `Drop` runs the
|
||||||
//! engine-wide `Drop` runs the close path (the wake bus empties, every
|
//! close path (the wake bus empties, every open feed disconnects), and
|
||||||
//! open feed disconnects), and the state is gone with the process's
|
//! the state is gone with the process's references, the honest-
|
||||||
//! references, honestly ephemeral ADR-016-shaped posture:
|
//! ephemeral posture (ADR-016 carriers, ADR-024 §2: the dependency on
|
||||||
//! the dependency on `alkstore-mem` *is* the ephemerality
|
//! `alkstore-mem` *is* the ephemerality declaration).
|
||||||
//! declaration).
|
|
||||||
//!
|
//!
|
||||||
//! The `Store` trait implementation is the seam task's artifact
|
//! The `Store` surface wires the mechanism handles (core-owned trait
|
||||||
//! (`mem-engine-seam-tx` — the tx overlay is the load-bearing half);
|
//! objects — engine types never appear in consumer signatures): `queue`
|
||||||
//! this type is the trait boundary's substrate.
|
//! carries its `QueueOpts` stamps (queues are names, not config
|
||||||
|
//! owners), `stream`/`outbox` carry validated names, `try_lock` rides
|
||||||
|
//! the TTL registry, `schedule`/`unschedule`/`run_schedules` the
|
||||||
|
//! collapsed scheduler, `notify`/`listen` the wake substrate, and
|
||||||
|
//! `begin_tx` allocates the tx overlay (`crate::tx` — the load-bearing
|
||||||
|
//! seam; `with_tx` is the trait's provided composition).
|
||||||
|
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use alkstore::{BoxedFuture, Result, StopToken, Store, TxHandle};
|
||||||
|
use alkstore::{QueueOpts, ScheduleOpts};
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
use crate::clock::Clock;
|
use crate::clock::Clock;
|
||||||
|
use crate::queue::MemQueueHandle;
|
||||||
|
use crate::scheduler;
|
||||||
use crate::state::State;
|
use crate::state::State;
|
||||||
|
use crate::stream::MemStreamHandle;
|
||||||
|
use crate::tx::MemTxHandle;
|
||||||
use crate::wakes::WakeBus;
|
use crate::wakes::WakeBus;
|
||||||
|
|
||||||
/// The mem engine's store handle. Consumers take it from
|
/// The mem engine's store handle. Consumers hold
|
||||||
/// `MemStore::new`; once the seam task lands, they hold
|
/// [`alkstore::Store`] — the trait is the contract surface, never the
|
||||||
/// `alkstore::Store` — the trait is the contract surface, never the
|
/// engine type; `MemStore::new` is the engine-native constructor.
|
||||||
/// engine type.
|
|
||||||
pub struct MemStore {
|
pub struct MemStore {
|
||||||
// Read by the wave-6 mechanism task code (the `Store` seam and the
|
|
||||||
// handles); the foundations build alone sees `state`/`clock` as dead.
|
|
||||||
#[allow(dead_code)]
|
|
||||||
pub(crate) state: Arc<State>,
|
pub(crate) state: Arc<State>,
|
||||||
pub(crate) wakes: Arc<WakeBus>,
|
pub(crate) wakes: Arc<WakeBus>,
|
||||||
#[allow(dead_code)]
|
|
||||||
pub(crate) clock: Arc<Clock>,
|
pub(crate) clock: Arc<Clock>,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -52,9 +60,9 @@ impl std::fmt::Debug for MemStore {
|
|||||||
|
|
||||||
impl MemStore {
|
impl MemStore {
|
||||||
/// Construct a fully isolated mem engine instance. Engine-native:
|
/// Construct a fully isolated mem engine instance. Engine-native:
|
||||||
/// no connection string, no file path (ADR-024 §4's
|
/// no connection string, no file path (ADR-024 §4's documented
|
||||||
/// documented exception to the open prose). Teardown is drop;
|
/// exception to the open prose). Teardown is drop; each instance
|
||||||
/// each instance shares nothing with any other.
|
/// shares nothing with any other.
|
||||||
pub fn new() -> Self {
|
pub fn new() -> Self {
|
||||||
Self {
|
Self {
|
||||||
state: Arc::new(State::new()),
|
state: Arc::new(State::new()),
|
||||||
@@ -80,9 +88,110 @@ impl Drop for MemStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl Store for MemStore {
|
||||||
|
fn begin_tx(&self) -> BoxedFuture<'_, Result<Box<dyn TxHandle + Send>>> {
|
||||||
|
let handle = MemTxHandle::begin(self.state.clone(), self.clock.clone(), self.wakes.clone());
|
||||||
|
Box::pin(async move { Ok(Box::new(handle) as Box<dyn TxHandle + Send>) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn notify<'a>(&'a self, channel: &str, payload: Value) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
crate::notify::notify(self.wakes.clone(), channel, payload)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn listen<'a>(
|
||||||
|
&'a self,
|
||||||
|
channel: &str,
|
||||||
|
) -> BoxedFuture<'a, Result<Box<dyn alkstore::WakeReceiver>>> {
|
||||||
|
let wakes = self.wakes.clone();
|
||||||
|
let owned = channel.to_string();
|
||||||
|
Box::pin(async move { crate::notify::listen(wakes, &owned) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn stream<'a>(
|
||||||
|
&'a self,
|
||||||
|
name: &str,
|
||||||
|
) -> BoxedFuture<'a, Result<Box<dyn alkstore::StreamHandle>>> {
|
||||||
|
let handle = alkstore::validate_shared_name(name).map(|()| {
|
||||||
|
MemStreamHandle::new(
|
||||||
|
name.to_string(),
|
||||||
|
self.state.clone(),
|
||||||
|
self.clock.clone(),
|
||||||
|
self.wakes.clone(),
|
||||||
|
)
|
||||||
|
});
|
||||||
|
Box::pin(async move { handle.map(|h| Box::new(h) as Box<dyn alkstore::StreamHandle>) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn queue<'a>(
|
||||||
|
&'a self,
|
||||||
|
name: &str,
|
||||||
|
opts: QueueOpts,
|
||||||
|
) -> BoxedFuture<'a, Result<Box<dyn alkstore::Queue>>> {
|
||||||
|
let handle = alkstore::validate_shared_name(name).map(|()| {
|
||||||
|
Box::new(MemQueueHandle::new(
|
||||||
|
name.to_string(),
|
||||||
|
opts,
|
||||||
|
self.state.clone(),
|
||||||
|
self.clock.clone(),
|
||||||
|
)) as Box<dyn alkstore::Queue>
|
||||||
|
});
|
||||||
|
Box::pin(async move { handle })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn outbox<'a>(&'a self, name: &str) -> BoxedFuture<'a, Result<Box<dyn alkstore::Outbox>>> {
|
||||||
|
let handle = alkstore::validate_shared_name(name).map(|()| {
|
||||||
|
Box::new(crate::outbox::MemOutbox::new(
|
||||||
|
name.to_string(),
|
||||||
|
self.state.clone(),
|
||||||
|
self.clock.clone(),
|
||||||
|
)) as Box<dyn alkstore::Outbox>
|
||||||
|
});
|
||||||
|
Box::pin(async move { handle })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn try_lock<'a>(
|
||||||
|
&'a self,
|
||||||
|
name: &str,
|
||||||
|
owner: &str,
|
||||||
|
ttl: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Option<Box<dyn alkstore::Lock>>>> {
|
||||||
|
let outcome =
|
||||||
|
crate::lock::try_lock(self.state.clone(), self.clock.clone(), name, owner, ttl);
|
||||||
|
Box::pin(async move { outcome })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn schedule<'a>(
|
||||||
|
&'a self,
|
||||||
|
name: &str,
|
||||||
|
spec: &str,
|
||||||
|
queue: &str,
|
||||||
|
payload: Value,
|
||||||
|
opts: ScheduleOpts,
|
||||||
|
) -> BoxedFuture<'a, Result<alkstore::Schedule>> {
|
||||||
|
scheduler::schedule(
|
||||||
|
self.state.clone(),
|
||||||
|
self.clock.clone(),
|
||||||
|
name,
|
||||||
|
spec,
|
||||||
|
queue,
|
||||||
|
payload,
|
||||||
|
opts,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn unschedule<'a>(&'a self, name: &str) -> BoxedFuture<'a, Result<bool>> {
|
||||||
|
scheduler::unschedule(self.state.clone(), name)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_schedules<'a>(&'a self, stop: StopToken) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
scheduler::run_schedules(self.state.clone(), self.clock.clone(), stop)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use alkstore::EnqueueOpts;
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn two_new_instances_share_nothing() {
|
fn two_new_instances_share_nothing() {
|
||||||
@@ -145,4 +254,38 @@ mod tests {
|
|||||||
"teardown is drop: the engine's close path disconnects every feed"
|
"teardown is drop: the engine's close path disconnects every feed"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn constructor_entry_validation() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
for bad in ["", " ", "__alkstore_q"] {
|
||||||
|
match store.queue(bad, QueueOpts::default()).await {
|
||||||
|
Err(alkstore::Error::InvalidName { .. } | alkstore::Error::ReservedName { .. }) => {
|
||||||
|
}
|
||||||
|
_other => panic!("queue({bad:?}) rejects"),
|
||||||
|
}
|
||||||
|
assert!(store.stream(bad).await.is_err());
|
||||||
|
assert!(store.outbox(bad).await.is_err());
|
||||||
|
}
|
||||||
|
// Nothing was created on a rejected name — the next queue with
|
||||||
|
// a fresh name starts clean (ids allocate nowhere).
|
||||||
|
let good = store.queue("good", QueueOpts::default()).await.unwrap();
|
||||||
|
let id = good
|
||||||
|
.enqueue(serde_json::json!(1), EnqueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(id, 1, "ids count from 1");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn begin_tx_grants_a_boxed_handle_that_commits() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
let q = store.queue("q", QueueOpts::default()).await.unwrap();
|
||||||
|
assert!(q.get_job(1).await.unwrap().is_some());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,704 @@
|
|||||||
|
//! The stream mechanism's mem arm — the in-memory event log +
|
||||||
|
//! per-consumer offsets (engine-mem.md §Mapping the contract: global
|
||||||
|
//! FIFO by monotone offset; keyed publish rides the overlay).
|
||||||
|
//!
|
||||||
|
//! - Publishes append to the per-stream log under its section guard:
|
||||||
|
//! engine-assigned offsets (the engine-wide allocator — pg-bigserial
|
||||||
|
//! shaped, monotone, immutable, never renumbered; a fresh stream's
|
||||||
|
//! first event does not carry offset 1, the ADR-015 §4 note), keys
|
||||||
|
//! are carried metadata (a present key must be non-empty —
|
||||||
|
//! `InvalidName`, ADR-015 §1), and the commit boundary fires one
|
||||||
|
//! wake per event on the stream's engine-internal wake channel.
|
||||||
|
//! - Cursor reads are `offset ASC` (global FIFO, ADR-015 §4) with the
|
||||||
|
//! extent guard at the trait-impl entry (`limit <= 0` → empty
|
||||||
|
//! `Vec`, ADR-023 §2).
|
||||||
|
//! - Checkpoints are explicit, monotone saves (a save below the
|
||||||
|
//! stored checkpoint is a silent no-op — ADR-019 §6; an absent
|
||||||
|
//! consumer reads 0, ADR-019 §1).
|
||||||
|
//! - `trim_to` deletes `offset <= horizon` exactly (a negative horizon
|
||||||
|
//! deletes nothing, idempotently — the boundary argument is total,
|
||||||
|
//! ADR-023 §2), **wakes nothing**, and surviving events keep their
|
||||||
|
//! offsets (gaps legal, ADR-015 §5). Reads from a trimmed region
|
||||||
|
//! resume at the horizon's first remaining row.
|
||||||
|
//! - `subscribe` attaches the receiver's wake feed **before** the
|
||||||
|
//! stored-offset read (no commit can fall between — a buffered wake
|
||||||
|
//! for a commit already re-read costs an empty re-poll, never a
|
||||||
|
//! missed event), drains the log to the tail, then parks on the
|
||||||
|
//! feed: wake-driven re-read with no polling requirement; explicit
|
||||||
|
//! saves only. The one close path is engine drop — the bus close
|
||||||
|
//! disconnects every feed and `recv()` returns the terminal `None`
|
||||||
|
//! (ADR-021 §5).
|
||||||
|
|
||||||
|
use std::collections::VecDeque;
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use alkstore::{
|
||||||
|
BoxedFuture, Error, EventReceiver, Result, StreamEvent, StreamHandle, validate_local_name,
|
||||||
|
};
|
||||||
|
|
||||||
|
use crate::clock::Clock;
|
||||||
|
use crate::state::Guard;
|
||||||
|
use crate::state::{EventRow, State, StreamState};
|
||||||
|
use crate::wakes::WakeBus;
|
||||||
|
|
||||||
|
/// How many events a wake-driven re-read fetches per page while
|
||||||
|
/// draining to the tail.
|
||||||
|
const SUBSCRIBE_PAGE: i64 = 256;
|
||||||
|
|
||||||
|
/// The engine-internal wake channel a stream's subscribers ride:
|
||||||
|
/// engine-derived, under the reserved prefix — consumer entry points
|
||||||
|
/// reject the prefix, so a consumer `listen` can never attach to it
|
||||||
|
/// (and `notify` can never fire into it). The publish-side wake feed
|
||||||
|
/// is plumbing, not contract surface.
|
||||||
|
pub(crate) fn stream_wake_channel(stream: &str) -> String {
|
||||||
|
format!("{}stream_wake:{}", alkstore::RESERVED_PREFIX, stream)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Insert one committed event under the streams guard (the auto-commit
|
||||||
|
/// publish and the commit-merge share this effect). Returns nothing —
|
||||||
|
/// the caller owns the wake firing at its commit boundary.
|
||||||
|
pub(crate) fn insert_event(s: &mut StreamState, stream: &str, row: EventRow) {
|
||||||
|
s.logs
|
||||||
|
.entry(stream.to_string())
|
||||||
|
.or_default()
|
||||||
|
.insert(row.offset, row);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read `offset ASC`, `offset > cursor`, up to `limit` (the extent
|
||||||
|
/// guard is the caller's — this helper is contract-blind to it).
|
||||||
|
pub(crate) fn read_events(s: &StreamState, stream: &str, from: i64, limit: i64) -> Vec<EventRow> {
|
||||||
|
s.logs
|
||||||
|
.get(stream)
|
||||||
|
.map(|log| {
|
||||||
|
log.range((from + 1)..)
|
||||||
|
.take(limit.max(0) as usize)
|
||||||
|
.map(|(_, row)| row.clone())
|
||||||
|
.collect()
|
||||||
|
})
|
||||||
|
.unwrap_or_default()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The monotone checkpoint save (ADR-019 §6): a save at-or-below the
|
||||||
|
/// stored checkpoint is a silent no-op; the direct form, the tx form,
|
||||||
|
/// and the receiver's no-arg form all land here.
|
||||||
|
pub(crate) fn save_offset_row(s: &mut StreamState, stream: &str, consumer: &str, offset: i64) {
|
||||||
|
let key = (stream.to_string(), consumer.to_string());
|
||||||
|
match s.offsets.get(&key) {
|
||||||
|
Some(current) if offset <= *current => {}
|
||||||
|
_ => {
|
||||||
|
s.offsets.insert(key, offset);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The stored checkpoint; absent consumer = 0.
|
||||||
|
pub(crate) fn get_offset_row(s: &StreamState, stream: &str, consumer: &str) -> i64 {
|
||||||
|
s.offsets
|
||||||
|
.get(&(stream.to_string(), consumer.to_string()))
|
||||||
|
.copied()
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `trim_to`: delete `offset <= horizon` (exact boundary); surviving
|
||||||
|
/// events keep their offsets. Returns the deleted count. Wakes nothing.
|
||||||
|
pub(crate) fn trim_events(s: &mut StreamState, stream: &str, horizon: i64) -> i64 {
|
||||||
|
let Some(log) = s.logs.get_mut(stream) else {
|
||||||
|
return 0;
|
||||||
|
};
|
||||||
|
let horizon_next = horizon.saturating_add(1);
|
||||||
|
let removed: Vec<i64> = log.range(..horizon_next).map(|(k, _)| *k).collect();
|
||||||
|
let count = removed.len();
|
||||||
|
for offset in removed {
|
||||||
|
log.remove(&offset);
|
||||||
|
}
|
||||||
|
count as i64
|
||||||
|
}
|
||||||
|
|
||||||
|
fn event_from_row(row: EventRow, stream: &str) -> StreamEvent {
|
||||||
|
StreamEvent::from_row(
|
||||||
|
row.offset,
|
||||||
|
stream.to_string(),
|
||||||
|
row.key,
|
||||||
|
row.payload,
|
||||||
|
row.created_at,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The stream-scoped handle (see the module docs) for one validated
|
||||||
|
/// stream name.
|
||||||
|
pub struct MemStreamHandle {
|
||||||
|
pub(crate) name: String,
|
||||||
|
pub(crate) state: Arc<State>,
|
||||||
|
pub(crate) clock: Arc<Clock>,
|
||||||
|
pub(crate) wakes: Arc<WakeBus>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemStreamHandle {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemStreamHandle")
|
||||||
|
.field("name", &self.name)
|
||||||
|
.finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemStreamHandle {
|
||||||
|
pub(crate) fn new(
|
||||||
|
name: String,
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
wakes: Arc<WakeBus>,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
name,
|
||||||
|
state,
|
||||||
|
clock,
|
||||||
|
wakes,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The keyed publish is the one publish shape (the plain form is
|
||||||
|
/// `key = None`).
|
||||||
|
fn stage_publish(&self, key: Option<String>, payload: Value) -> Result<i64> {
|
||||||
|
if let Some(k) = &key
|
||||||
|
&& (k.is_empty() || k.trim().is_empty())
|
||||||
|
{
|
||||||
|
return Err(Error::InvalidName { name: k.clone() });
|
||||||
|
}
|
||||||
|
let bytes = alkstore::encode_payload(&payload)?;
|
||||||
|
let now = self.clock.unix_now();
|
||||||
|
let offset = self.state.allocate_event_offset();
|
||||||
|
let row = EventRow {
|
||||||
|
offset,
|
||||||
|
key,
|
||||||
|
payload: bytes,
|
||||||
|
created_at: now,
|
||||||
|
};
|
||||||
|
{
|
||||||
|
let mut s = self.state.streams.guard();
|
||||||
|
insert_event(&mut s, &self.name, row);
|
||||||
|
}
|
||||||
|
// The auto-commit publish's commit boundary is its own
|
||||||
|
// completion: one wake per event, per-subscriber FIFO.
|
||||||
|
let channel = stream_wake_channel(&self.name);
|
||||||
|
self.wakes.fire_commit_wakes([channel]);
|
||||||
|
Ok(offset)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl StreamHandle for MemStreamHandle {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
&self.name
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish<'a>(&'a self, payload: Value) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
match self.stage_publish(None, payload) {
|
||||||
|
Ok(offset) => Box::pin(async move { Ok(offset) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish_with_key<'a>(
|
||||||
|
&'a self,
|
||||||
|
key: Option<String>,
|
||||||
|
payload: Value,
|
||||||
|
) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
match self.stage_publish(key, payload) {
|
||||||
|
Ok(offset) => Box::pin(async move { Ok(offset) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_since<'a>(
|
||||||
|
&'a self,
|
||||||
|
offset: i64,
|
||||||
|
limit: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Vec<StreamEvent>>> {
|
||||||
|
if limit <= 0 {
|
||||||
|
return Box::pin(async { Ok(Vec::new()) });
|
||||||
|
}
|
||||||
|
let events = {
|
||||||
|
let s = self.state.streams.guard();
|
||||||
|
read_events(&s, &self.name, offset, limit)
|
||||||
|
};
|
||||||
|
let name = self.name.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
Ok(events
|
||||||
|
.into_iter()
|
||||||
|
.map(|r| event_from_row(r, &name))
|
||||||
|
.collect())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_from_consumer<'a>(
|
||||||
|
&'a self,
|
||||||
|
consumer: &str,
|
||||||
|
limit: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Vec<StreamEvent>>> {
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if limit <= 0 {
|
||||||
|
return Box::pin(async { Ok(Vec::new()) });
|
||||||
|
}
|
||||||
|
let name = self.name.clone();
|
||||||
|
let state = self.state.clone();
|
||||||
|
let consumer = consumer.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
let events = {
|
||||||
|
let s = state.streams.guard();
|
||||||
|
let from = get_offset_row(&s, &name, &consumer);
|
||||||
|
read_events(&s, &name, from, limit)
|
||||||
|
};
|
||||||
|
Ok(events
|
||||||
|
.into_iter()
|
||||||
|
.map(|r| event_from_row(r, &name))
|
||||||
|
.collect())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn save_offset<'a>(&'a self, consumer: &str, offset: i64) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let name = self.name.clone();
|
||||||
|
let state = self.state.clone();
|
||||||
|
let consumer = consumer.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
let mut s = state.streams.guard();
|
||||||
|
save_offset_row(&mut s, &name, &consumer, offset);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_offset<'a>(&'a self, consumer: &str) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let name = self.name.clone();
|
||||||
|
let state = self.state.clone();
|
||||||
|
let consumer = consumer.to_string();
|
||||||
|
Box::pin(async move {
|
||||||
|
let s = state.streams.guard();
|
||||||
|
Ok(get_offset_row(&s, &name, &consumer))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn trim_to<'a>(&'a self, horizon: i64) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
let name = self.name.clone();
|
||||||
|
let state = self.state.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
let mut s = state.streams.guard();
|
||||||
|
Ok(trim_events(&mut s, &name, horizon))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn subscribe<'a>(&'a self, consumer: &str) -> BoxedFuture<'a, Result<Box<dyn EventReceiver>>> {
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let name = self.name.clone();
|
||||||
|
let consumer = consumer.to_string();
|
||||||
|
let state = self.state.clone();
|
||||||
|
let wakes = self.wakes.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
// Attach first (starts from "now"; attach-after-commit
|
||||||
|
// no-replay is structural), then read the stored offset and
|
||||||
|
// drain to the tail.
|
||||||
|
let feed = wakes.attach(&stream_wake_channel(&name));
|
||||||
|
let mut cursor;
|
||||||
|
let mut pending = VecDeque::new();
|
||||||
|
{
|
||||||
|
let s = state.streams.guard();
|
||||||
|
cursor = get_offset_row(&s, &name, &consumer);
|
||||||
|
loop {
|
||||||
|
let rows = read_events(&s, &name, cursor, SUBSCRIBE_PAGE);
|
||||||
|
let got = rows.len();
|
||||||
|
for row in rows {
|
||||||
|
cursor = row.offset;
|
||||||
|
pending.push_back(event_from_row(row, &name));
|
||||||
|
}
|
||||||
|
if got < SUBSCRIBE_PAGE as usize {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Box::new(MemEventReceiver {
|
||||||
|
stream: name,
|
||||||
|
consumer,
|
||||||
|
state,
|
||||||
|
feed,
|
||||||
|
pending,
|
||||||
|
cursor,
|
||||||
|
position: 0,
|
||||||
|
}) as Box<dyn EventReceiver>)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The durable subscription receiver (see the module docs). Its
|
||||||
|
/// position is the last-*yielded* event's offset, so `save_offset()`
|
||||||
|
/// never checkpoints an event the consumer did not observe.
|
||||||
|
pub struct MemEventReceiver {
|
||||||
|
stream: String,
|
||||||
|
consumer: String,
|
||||||
|
state: Arc<State>,
|
||||||
|
feed: crate::wakes::WakeFeed,
|
||||||
|
pending: VecDeque<StreamEvent>,
|
||||||
|
cursor: i64,
|
||||||
|
position: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemEventReceiver {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemEventReceiver")
|
||||||
|
.field("stream", &self.stream)
|
||||||
|
.finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemEventReceiver {
|
||||||
|
/// One forward read from the cursor into the pending buffer.
|
||||||
|
fn drain_a_page(&mut self) -> Result<()> {
|
||||||
|
let events = {
|
||||||
|
let s = self.state.streams.guard();
|
||||||
|
read_events(&s, &self.stream, self.cursor, SUBSCRIBE_PAGE)
|
||||||
|
};
|
||||||
|
for row in events {
|
||||||
|
self.cursor = row.offset;
|
||||||
|
self.pending.push_back(event_from_row(row, &self.stream));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pop_positional(&mut self) -> Option<StreamEvent> {
|
||||||
|
let event = self.pending.pop_front()?;
|
||||||
|
self.position = event.offset;
|
||||||
|
Some(event)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EventReceiver for MemEventReceiver {
|
||||||
|
fn recv<'a>(&'a mut self) -> BoxedFuture<'a, Option<Result<StreamEvent>>> {
|
||||||
|
Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
if let Some(event) = self.pop_positional() {
|
||||||
|
return Some(Ok(event));
|
||||||
|
}
|
||||||
|
// Park on the wake feed: a commit that lands after this
|
||||||
|
// park fires a wake that releases it; a commit that
|
||||||
|
// landed before it already buffered a wake (attach-first).
|
||||||
|
let _wake = match self.feed.rx().recv().await {
|
||||||
|
Some(wake) => wake,
|
||||||
|
None => return None,
|
||||||
|
};
|
||||||
|
if let Err(e) = self.drain_a_page() {
|
||||||
|
return Some(Err(database_only(e)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn try_recv(&mut self) -> Result<Option<StreamEvent>> {
|
||||||
|
if let Some(event) = self.pop_positional() {
|
||||||
|
return Ok(Some(event));
|
||||||
|
}
|
||||||
|
// A buffered wake is the hint that a commit landed which the
|
||||||
|
// drain has not seen yet; the poll drains it and re-reads once.
|
||||||
|
let closed = matches!(
|
||||||
|
self.feed.rx().try_recv(),
|
||||||
|
Err(tokio::sync::mpsc::error::TryRecvError::Disconnected)
|
||||||
|
);
|
||||||
|
self.drain_a_page()?;
|
||||||
|
if let Some(event) = self.pop_positional() {
|
||||||
|
return Ok(Some(event));
|
||||||
|
}
|
||||||
|
if closed {
|
||||||
|
// The bus close (engine drop) is the one close path: once
|
||||||
|
// the feed is disconnected and nothing is buffered or
|
||||||
|
// readable, Closed is the terminal poll arm (never reopens).
|
||||||
|
return Err(Error::Closed);
|
||||||
|
}
|
||||||
|
Ok(None)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_since<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
offset: i64,
|
||||||
|
limit: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Vec<StreamEvent>>> {
|
||||||
|
if limit <= 0 {
|
||||||
|
return Box::pin(async { Ok(Vec::new()) });
|
||||||
|
}
|
||||||
|
let events = {
|
||||||
|
let s = self.state.streams.guard();
|
||||||
|
read_events(&s, &self.stream, offset, limit)
|
||||||
|
};
|
||||||
|
let stream = self.stream.clone();
|
||||||
|
Box::pin(async move {
|
||||||
|
Ok(events
|
||||||
|
.into_iter()
|
||||||
|
.map(|r| event_from_row(r, &stream))
|
||||||
|
.collect())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn save_offset(&mut self) -> Result<()> {
|
||||||
|
let position = self.position;
|
||||||
|
if position == 0 {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let mut s = self.state.streams.guard();
|
||||||
|
save_offset_row(&mut s, &self.stream, &self.consumer, position);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn offset(&self) -> i64 {
|
||||||
|
self.position
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `recv()`'s `Err` arm carries `Database` only (ADR-021 §5): a read
|
||||||
|
/// error mapped elsewhere is remapped into the opaque fallback with
|
||||||
|
/// the original preserved as the source (the decode-side posture
|
||||||
|
/// guard — mem reads cannot fail in practice).
|
||||||
|
fn database_only(e: Error) -> Error {
|
||||||
|
match e {
|
||||||
|
already @ Error::Database(_) => already,
|
||||||
|
other => Error::database(other),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::{EventReceiver, Store, StreamHandle};
|
||||||
|
|
||||||
|
fn handle(store: &MemStore, name: &str) -> MemStreamHandle {
|
||||||
|
MemStreamHandle::new(
|
||||||
|
name.to_string(),
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
store.wakes.clone(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn publish_read_round_trip_is_global_fifo_by_offset() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let a = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = handle.publish(serde_json::json!(2)).await.unwrap();
|
||||||
|
let keyed = handle
|
||||||
|
.publish_with_key(Some("cust-7".into()), serde_json::json!(3))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(a < b && b < keyed, "offsets are monotone per stream");
|
||||||
|
let events = handle.read_since(0, 10).await.unwrap();
|
||||||
|
assert_eq!(events.len(), 3);
|
||||||
|
assert_eq!(
|
||||||
|
events.iter().map(|e| e.offset).collect::<Vec<_>>(),
|
||||||
|
vec![a, b, keyed],
|
||||||
|
"offset ASC — global FIFO regardless of keys"
|
||||||
|
);
|
||||||
|
assert_eq!(events[0].stream, "orders");
|
||||||
|
assert_eq!(events[0].key, None);
|
||||||
|
assert_eq!(events[2].key.as_deref(), Some("cust-7"));
|
||||||
|
assert_eq!(events[2].payload_as::<i64>().unwrap(), 3);
|
||||||
|
assert!(
|
||||||
|
events
|
||||||
|
.iter()
|
||||||
|
.all(|e| e.created_at == store.clock.unix_now())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn empty_some_key_is_invalid_at_every_entry() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let err = handle
|
||||||
|
.publish_with_key(Some(String::new()), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(err, alkstore::Error::InvalidName { .. }));
|
||||||
|
let err = handle
|
||||||
|
.publish_with_key(Some(" ".into()), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(err, alkstore::Error::InvalidName { .. }));
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let err = tx
|
||||||
|
.publish_with_key_tx("orders", Some(String::new()), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(err, alkstore::Error::InvalidName { .. }));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn read_extents_clamp_empty() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
assert!(handle.read_since(0, 0).await.unwrap().is_empty());
|
||||||
|
assert!(
|
||||||
|
handle.read_since(0, -3).await.unwrap().is_empty(),
|
||||||
|
"the LIMIT -1 dialect artifact is dead"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn saves_are_monotone_and_absent_consumers_read_zero() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
0,
|
||||||
|
"an absent consumer reads 0"
|
||||||
|
);
|
||||||
|
let tail = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
handle.save_offset("c1", tail).await.unwrap();
|
||||||
|
assert_eq!(handle.get_offset("c1").await.unwrap(), tail);
|
||||||
|
handle.save_offset("c1", tail - 1).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
tail,
|
||||||
|
"a below-checkpoint save is a silent no-op (monotone)"
|
||||||
|
);
|
||||||
|
handle.save_offset("c1", tail).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
tail,
|
||||||
|
"an equal save is also a no-op"
|
||||||
|
);
|
||||||
|
handle.save_offset("c1", tail + 5).await.unwrap();
|
||||||
|
assert_eq!(handle.get_offset("c1").await.unwrap(), tail + 5);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn trim_is_exact_boundary_and_wakes_nothing() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let a = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = handle.publish(serde_json::json!(2)).await.unwrap();
|
||||||
|
let c = handle.publish(serde_json::json!(3)).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
handle.trim_to(b - 1).await.unwrap(),
|
||||||
|
1,
|
||||||
|
"exact boundary: offset <= horizon"
|
||||||
|
);
|
||||||
|
let events = handle.read_since(0, 10).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
events.iter().map(|e| e.offset).collect::<Vec<_>>(),
|
||||||
|
vec![b, c],
|
||||||
|
"surviving rows keep their offsets, never renumbered"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
handle.trim_to(-100).await.unwrap(),
|
||||||
|
0,
|
||||||
|
"a negative horizon deletes nothing"
|
||||||
|
);
|
||||||
|
assert_eq!(handle.trim_to(c).await.unwrap(), 2);
|
||||||
|
assert_eq!(handle.trim_to(c).await.unwrap(), 0, "idempotent");
|
||||||
|
// Reads from a trimmed region resume at the horizon's first
|
||||||
|
// remaining row.
|
||||||
|
assert!(handle.read_since(0, 10).await.unwrap().is_empty());
|
||||||
|
let _ = a;
|
||||||
|
assert_eq!(handle.get_offset("late").await.unwrap(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn subscriber_replays_from_the_saved_offset_and_ride_wakes() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let _a = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = handle.publish(serde_json::json!(2)).await.unwrap();
|
||||||
|
handle.save_offset("c1", b).await.unwrap();
|
||||||
|
let mut receiver = handle.subscribe("c1").await.unwrap();
|
||||||
|
// Everything after the checkpoint replays on attach.
|
||||||
|
let c = handle.publish(serde_json::json!(3)).await.unwrap();
|
||||||
|
let first = receiver.recv().await.unwrap().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
first.offset, c,
|
||||||
|
"replay starts at the stored checkpoint's forward tail"
|
||||||
|
);
|
||||||
|
assert_eq!(first.payload_as::<i64>().unwrap(), 3);
|
||||||
|
// Post-attach publishes deliver without polling.
|
||||||
|
let d = handle.publish(serde_json::json!(4)).await.unwrap();
|
||||||
|
let second = receiver.recv().await.unwrap().unwrap();
|
||||||
|
assert_eq!(second.offset, d);
|
||||||
|
// The explicit-save receiver form and the direct form compose.
|
||||||
|
receiver.save_offset().unwrap();
|
||||||
|
assert_eq!(receiver.offset(), d);
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
d,
|
||||||
|
"the receiver saves the last-yielded offset"
|
||||||
|
);
|
||||||
|
// A save before any yield is a no-op on the stored checkpoint.
|
||||||
|
let mut fresh = handle.subscribe("c2").await.unwrap();
|
||||||
|
assert_eq!(fresh.offset(), 0);
|
||||||
|
fresh.save_offset().unwrap();
|
||||||
|
assert_eq!(handle.get_offset("c2").await.unwrap(), 0);
|
||||||
|
let _ = _a;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn receiver_try_recv_reads_without_await() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let mut receiver = handle.subscribe("c1").await.unwrap();
|
||||||
|
assert!(try_none(receiver.as_mut()), "idle reads as Ok(None)");
|
||||||
|
let a = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
let got = receiver.try_recv().unwrap().unwrap();
|
||||||
|
assert_eq!(got.offset, a);
|
||||||
|
assert!(try_none(receiver.as_mut()));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn try_none(receiver: &mut dyn EventReceiver) -> bool {
|
||||||
|
matches!(receiver.try_recv(), Ok(None))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn engine_drop_closes_the_receiver_on_both_arms() {
|
||||||
|
let (mut receiver, offset) = {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let offset = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
(handle.subscribe("c1").await.unwrap(), offset)
|
||||||
|
};
|
||||||
|
// The engine is gone; the replayed pre-drop event still
|
||||||
|
// delivers (it was committed and visible), then the receiver
|
||||||
|
// is closed — terminal on both arms (never reopens).
|
||||||
|
let delivered = receiver
|
||||||
|
.try_recv()
|
||||||
|
.unwrap()
|
||||||
|
.expect("the pre-drop event replays");
|
||||||
|
assert_eq!(delivered.offset, offset);
|
||||||
|
assert!(matches!(receiver.try_recv(), Err(alkstore::Error::Closed)));
|
||||||
|
let closed = tokio::time::timeout(std::time::Duration::from_millis(200), receiver.recv())
|
||||||
|
.await
|
||||||
|
.expect("the closed feed parks no receiver");
|
||||||
|
assert!(
|
||||||
|
closed.is_none(),
|
||||||
|
"the terminal None arm (engine drop is the close cause)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn trim_wakes_nothing_for_an_open_subscriber() {
|
||||||
|
let store = MemStore::new();
|
||||||
|
let handle = handle(&store, "orders");
|
||||||
|
let a = handle.publish(serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = handle.publish(serde_json::json!(2)).await.unwrap();
|
||||||
|
let mut receiver = handle.subscribe("c1").await.unwrap();
|
||||||
|
assert_eq!(receiver.recv().await.unwrap().unwrap().offset, a);
|
||||||
|
assert_eq!(receiver.recv().await.unwrap().unwrap().offset, b);
|
||||||
|
handle.trim_to(b).await.unwrap();
|
||||||
|
let quiet =
|
||||||
|
tokio::time::timeout(std::time::Duration::from_millis(50), receiver.recv()).await;
|
||||||
|
assert!(quiet.is_err(), "trim emits no wake and delivers no event");
|
||||||
|
let _ = receiver.read_since(0, 10).await.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,982 @@
|
|||||||
|
//! The transactional seam's mem arm (ADR-007): `begin_tx` allocates an
|
||||||
|
//! **overlay** private to the caller's handle; the eleven `*_tx`
|
||||||
|
//! methods stage effects in it; commit merges the overlay atomically
|
||||||
|
//! and fires the overlay's pending wakes; drop discards it.
|
||||||
|
//!
|
||||||
|
//! - **Staging** (engine-mem.md §The tx overlay): every `*_tx` call
|
||||||
|
//! validates its names at the entry point (before any effect),
|
||||||
|
//! encodes the payload (`Codec` typed, ADR-023 §1), reads one clock
|
||||||
|
//! instant, resolves/stamps identically to the no-handle paths
|
||||||
|
//! (ADR-020 §3 — `enqueue_tx` stamps the plain derived defaults
|
||||||
|
//! 300/3/5/none, `outbox_enqueue_tx` the outbox's 60/5/5 set), and
|
||||||
|
//! appends the effect to the handle-private overlay. Ids and offsets
|
||||||
|
//! allocate at stage time from the engine-wide allocators (the
|
||||||
|
//! pg-shaped story — a rolled-back tx's allocation gaps out; ids and
|
||||||
|
//! offsets are monotone and never reused, gaps ordinary).
|
||||||
|
//! - **Reads inside the tx** merge the overlay over the committed
|
||||||
|
//! state — read-your-own-writes with no round-trips:
|
||||||
|
//! `get_job_tx`/`get_offset_tx`/`read_since_tx`/
|
||||||
|
//! `read_from_consumer_tx` see the transaction's own staged
|
||||||
|
//! enqueues/publishes/saves and the committed rows underneath; every
|
||||||
|
//! other store view sees none of it pre-commit (an overlay is
|
||||||
|
//! invisible to every other view until its commit — no isolation
|
||||||
|
//! questions can arise across handles).
|
||||||
|
//! - **Commit-merge + wake-at-commit**: `commit` consumes the handle;
|
||||||
|
//! the merge is one guarded operation (queues → streams, the
|
||||||
|
//! canonical lock order) and the overlay's pending wakes fire in the
|
||||||
|
//! same operation, after the merged state is visible — a tx-staged
|
||||||
|
//! notify delivers nothing until the merge, one wake per staged
|
||||||
|
//! notify (per-notify, never coalesced). That discipline is what the
|
||||||
|
//! pg engine had to fix retroactively (`pg-fix-tx-wake`); here it is
|
||||||
|
//! structural, pinned from birth.
|
||||||
|
//! - **Drop = discard** (ADR-021 §4): dropping the handle without
|
||||||
|
//! commit discards the overlay — no-ghosts trivially, because
|
||||||
|
//! uncommitted state never touched committed state; the discard is
|
||||||
|
//! panic-safe through an unwinding drop (the overlay is plain local
|
||||||
|
//! data; dropping runs nothing).
|
||||||
|
//! - **Numeric domains** (ADR-023 §2) at the trait-impl entry: the
|
||||||
|
//! stream reads' extents clamp empty (`limit <= 0`), the enqueue
|
||||||
|
//! boundaries are total (a negative delay resolves to a ready time
|
||||||
|
//! in the past — ready now), and durations never ride the tx seam.
|
||||||
|
|
||||||
|
use std::collections::{HashMap, VecDeque};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
|
use alkstore::{
|
||||||
|
BoxedFuture, EnqueueOpts, Error, Job, JobState, Result, StreamEvent, TxHandle,
|
||||||
|
validate_local_name, validate_shared_name,
|
||||||
|
};
|
||||||
|
|
||||||
|
use crate::clock::Clock;
|
||||||
|
use crate::queue::{build_job_row, get_job_row, job_from_row};
|
||||||
|
use crate::resolution::{
|
||||||
|
Stamps, outbox_backing_queue_name, outbox_default_stamps, plain_queue_default_stamps,
|
||||||
|
resolve_enqueue_opts, stamps_with_override,
|
||||||
|
};
|
||||||
|
use crate::state::Guard;
|
||||||
|
use crate::state::{EventRow, JobRow, State};
|
||||||
|
use crate::stream::{
|
||||||
|
get_offset_row, insert_event, read_events, save_offset_row, stream_wake_channel,
|
||||||
|
};
|
||||||
|
use crate::wakes::WakeBus;
|
||||||
|
|
||||||
|
/// The handle-private overlay: every staged effect, in stage order.
|
||||||
|
#[derive(Debug, Default)]
|
||||||
|
struct Overlay {
|
||||||
|
/// Staged enqueue rows (ids allocated at stage time).
|
||||||
|
jobs: HashMap<i64, JobRow>,
|
||||||
|
/// Staged event rows per stream (offsets allocated at stage time;
|
||||||
|
/// stage order = offset order — the allocators are globally
|
||||||
|
/// monotone).
|
||||||
|
events: HashMap<String, Vec<EventRow>>,
|
||||||
|
/// Staged notifies — channels in stage order, duplicates allowed
|
||||||
|
/// (per-notify delivery, never coalesced).
|
||||||
|
notifies: Vec<String>,
|
||||||
|
/// Staged checkpoint saves (stream, consumer) → the staged max.
|
||||||
|
saves: HashMap<(String, String), i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One staged enqueue effect — the shared body of `enqueue_tx` and
|
||||||
|
/// `outbox_enqueue_tx` (the only difference: the queue name and the
|
||||||
|
/// stamp source).
|
||||||
|
fn stage_enqueued(
|
||||||
|
overlay: &mut Overlay,
|
||||||
|
state: &State,
|
||||||
|
clock: &Clock,
|
||||||
|
queue: &str,
|
||||||
|
opts: &EnqueueOpts,
|
||||||
|
payload: Value,
|
||||||
|
stamps: Stamps,
|
||||||
|
) -> Result<i64> {
|
||||||
|
let bytes = alkstore::encode_payload(&payload)?;
|
||||||
|
let now = clock.unix_now();
|
||||||
|
let resolved = resolve_enqueue_opts(now, opts);
|
||||||
|
let id = state.allocate_job_id();
|
||||||
|
let row = build_job_row(
|
||||||
|
id,
|
||||||
|
queue.to_string(),
|
||||||
|
bytes,
|
||||||
|
now,
|
||||||
|
&resolved,
|
||||||
|
opts.priority,
|
||||||
|
&stamps,
|
||||||
|
);
|
||||||
|
overlay.jobs.insert(id, row);
|
||||||
|
Ok(id)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The staged-job read: overlay first, then the committed live/dead
|
||||||
|
/// tables (dead rows visible — the auto-commit twin's shape, ADR-010
|
||||||
|
/// §1) — both queue-scoped against the argument.
|
||||||
|
fn merged_get_job(overlay: &Overlay, state: &State, queue: &str, job_id: i64) -> Option<Job> {
|
||||||
|
if let Some(row) = overlay.jobs.get(&job_id) {
|
||||||
|
return if row.queue == queue {
|
||||||
|
Some(job_from_row(row.clone(), JobState::Pending, None, None))
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let q = state.queues.guard();
|
||||||
|
let job = get_job_row(&q, job_id)?;
|
||||||
|
(job.queue == queue).then_some(job)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The merged event read: the committed log and the staged rows under
|
||||||
|
/// one offset-ASC ordering (the global-FIFO shape holds across the
|
||||||
|
/// merge — stage-time offsets are globally monotone). The extent guard
|
||||||
|
/// (ADR-023 §2) lives here: `limit <= 0` yields the empty `Vec`.
|
||||||
|
fn merged_read_since(
|
||||||
|
overlay: &Overlay,
|
||||||
|
state: &State,
|
||||||
|
stream: &str,
|
||||||
|
offset: i64,
|
||||||
|
limit: i64,
|
||||||
|
) -> Vec<EventRow> {
|
||||||
|
if limit <= 0 {
|
||||||
|
return Vec::new();
|
||||||
|
}
|
||||||
|
let committed = {
|
||||||
|
let s = state.streams.guard();
|
||||||
|
read_events(&s, stream, offset, limit)
|
||||||
|
};
|
||||||
|
let mut merged = committed;
|
||||||
|
if let Some(staged) = overlay.events.get(stream) {
|
||||||
|
merged.extend(staged.iter().filter(|row| row.offset > offset).cloned());
|
||||||
|
}
|
||||||
|
merged.sort_by_key(|row| row.offset);
|
||||||
|
merged.truncate(limit as usize);
|
||||||
|
merged
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The merged checkpoint read (absent everywhere = 0).
|
||||||
|
fn merged_get_offset(overlay: &Overlay, state: &State, stream: &str, consumer: &str) -> i64 {
|
||||||
|
let committed = {
|
||||||
|
let s = state.streams.guard();
|
||||||
|
get_offset_row(&s, stream, consumer)
|
||||||
|
};
|
||||||
|
let staged = overlay
|
||||||
|
.saves
|
||||||
|
.get(&(stream.to_string(), consumer.to_string()));
|
||||||
|
staged.map_or(committed, |s| (*s).max(committed))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn event_of(row: EventRow, stream: &str) -> StreamEvent {
|
||||||
|
StreamEvent::from_row(
|
||||||
|
row.offset,
|
||||||
|
stream.to_string(),
|
||||||
|
row.key,
|
||||||
|
row.payload,
|
||||||
|
row.created_at,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The caller-held mem transaction handle (constructed only through
|
||||||
|
/// [`crate::MemStore`]'s `begin_tx`; consumers see
|
||||||
|
/// `Box<dyn TxHandle + Send>`). [`TxHandle::commit`] consumes it;
|
||||||
|
/// dropping it discards the overlay.
|
||||||
|
pub struct MemTxHandle {
|
||||||
|
overlay: Overlay,
|
||||||
|
state: Arc<State>,
|
||||||
|
clock: Arc<Clock>,
|
||||||
|
wakes: Arc<WakeBus>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for MemTxHandle {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.debug_struct("MemTxHandle").finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemTxHandle {
|
||||||
|
pub(crate) fn begin(state: Arc<State>, clock: Arc<Clock>, wakes: Arc<WakeBus>) -> Self {
|
||||||
|
Self {
|
||||||
|
overlay: Overlay::default(),
|
||||||
|
state,
|
||||||
|
clock,
|
||||||
|
wakes,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TxHandle for MemTxHandle {
|
||||||
|
fn enqueue_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
queue: &str,
|
||||||
|
opts: EnqueueOpts,
|
||||||
|
payload: Value,
|
||||||
|
) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
if let Err(e) = validate_shared_name(queue) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
match stage_enqueued(
|
||||||
|
&mut self.overlay,
|
||||||
|
&self.state,
|
||||||
|
&self.clock,
|
||||||
|
queue,
|
||||||
|
&opts,
|
||||||
|
payload,
|
||||||
|
stamps_with_override(plain_queue_default_stamps(), &opts),
|
||||||
|
) {
|
||||||
|
Ok(id) => Box::pin(async move { Ok(id) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish_tx<'a>(&'a mut self, stream: &str, payload: Value) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
let stream = stream.to_string();
|
||||||
|
Box::pin(async move { self.publish_with_key_tx(&stream, None, payload).await })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish_with_key_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
stream: &str,
|
||||||
|
key: Option<String>,
|
||||||
|
payload: Value,
|
||||||
|
) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
if let Err(e) = validate_shared_name(stream) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if let Some(k) = &key
|
||||||
|
&& (k.is_empty() || k.trim().is_empty())
|
||||||
|
{
|
||||||
|
let bad = k.clone();
|
||||||
|
return Box::pin(async move { Err(Error::InvalidName { name: bad }) });
|
||||||
|
}
|
||||||
|
match self.stage_publish(stream, key, payload) {
|
||||||
|
Ok(offset) => Box::pin(async move { Ok(offset) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn notify_tx<'a>(&'a mut self, channel: &str, payload: Value) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
if let Err(e) = validate_shared_name(channel) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
match self.stage_notify(channel, payload) {
|
||||||
|
Ok(()) => Box::pin(async { Ok(()) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn save_offset_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
stream: &str,
|
||||||
|
consumer: &str,
|
||||||
|
offset: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
if let Err(e) = validate_shared_name(stream) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
self.stage_save_offset(stream, consumer, offset);
|
||||||
|
Box::pin(async { Ok(()) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_job_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
queue: &str,
|
||||||
|
job_id: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Option<Job>>> {
|
||||||
|
if let Err(e) = validate_shared_name(queue) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let job = merged_get_job(&self.overlay, &self.state, queue, job_id);
|
||||||
|
Box::pin(async move { Ok(job) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_offset_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
stream: &str,
|
||||||
|
consumer: &str,
|
||||||
|
) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
if let Err(e) = validate_shared_name(stream) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let offset = merged_get_offset(&self.overlay, &self.state, stream, consumer);
|
||||||
|
Box::pin(async move { Ok(offset) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_since_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
stream: &str,
|
||||||
|
offset: i64,
|
||||||
|
limit: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Vec<StreamEvent>>> {
|
||||||
|
if let Err(e) = validate_shared_name(stream) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let events = merged_read_since(&self.overlay, &self.state, stream, offset, limit);
|
||||||
|
let stream = stream.to_string();
|
||||||
|
Box::pin(async move { Ok(events.into_iter().map(|r| event_of(r, &stream)).collect()) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_from_consumer_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
stream: &str,
|
||||||
|
consumer: &str,
|
||||||
|
limit: i64,
|
||||||
|
) -> BoxedFuture<'a, Result<Vec<StreamEvent>>> {
|
||||||
|
if let Err(e) = validate_shared_name(stream) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
if let Err(e) = validate_local_name(consumer) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let events = {
|
||||||
|
let from = merged_get_offset(&self.overlay, &self.state, stream, consumer);
|
||||||
|
merged_read_since(&self.overlay, &self.state, stream, from, limit)
|
||||||
|
};
|
||||||
|
let stream = stream.to_string();
|
||||||
|
Box::pin(async move { Ok(events.into_iter().map(|r| event_of(r, &stream)).collect()) })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn outbox_enqueue_tx<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
outbox: &str,
|
||||||
|
opts: EnqueueOpts,
|
||||||
|
payload: Value,
|
||||||
|
) -> BoxedFuture<'a, Result<i64>> {
|
||||||
|
if let Err(e) = validate_shared_name(outbox) {
|
||||||
|
return Box::pin(async move { Err(e) });
|
||||||
|
}
|
||||||
|
let backing = outbox_backing_queue_name(outbox);
|
||||||
|
match stage_enqueued(
|
||||||
|
&mut self.overlay,
|
||||||
|
&self.state,
|
||||||
|
&self.clock,
|
||||||
|
&backing,
|
||||||
|
&opts,
|
||||||
|
payload,
|
||||||
|
stamps_with_override(outbox_default_stamps(), &opts),
|
||||||
|
) {
|
||||||
|
Ok(id) => Box::pin(async move { Ok(id) }),
|
||||||
|
Err(e) => Box::pin(async move { Err(e) }),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn commit(self: Box<Self>) -> BoxedFuture<'static, Result<()>> {
|
||||||
|
// Consume the handle: the overlay moves out (the one-shot
|
||||||
|
// discipline); a dropped-before-commit handle discards instead
|
||||||
|
// — nothing runs on that path, panic-safe through the unwind.
|
||||||
|
let MemTxHandle {
|
||||||
|
overlay,
|
||||||
|
state,
|
||||||
|
clock: _,
|
||||||
|
wakes,
|
||||||
|
} = *self;
|
||||||
|
Box::pin(async move {
|
||||||
|
merge_and_wake(state, wakes, overlay);
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MemTxHandle {
|
||||||
|
fn stage_publish(&mut self, stream: &str, key: Option<String>, payload: Value) -> Result<i64> {
|
||||||
|
let bytes = alkstore::encode_payload(&payload)?;
|
||||||
|
let now = self.clock.unix_now();
|
||||||
|
let offset = self.state.allocate_event_offset();
|
||||||
|
let row = EventRow {
|
||||||
|
offset,
|
||||||
|
key,
|
||||||
|
payload: bytes,
|
||||||
|
created_at: now,
|
||||||
|
};
|
||||||
|
self.overlay
|
||||||
|
.events
|
||||||
|
.entry(stream.to_string())
|
||||||
|
.or_default()
|
||||||
|
.push(row);
|
||||||
|
Ok(offset)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn stage_notify(&mut self, channel: &str, payload: Value) -> Result<()> {
|
||||||
|
// The encoding run is the codec-typed entry check (the
|
||||||
|
// measured serialization, ADR-020 §4); the wake contract's
|
||||||
|
// delivery is the channel alone — the bytes are nowhere stored.
|
||||||
|
let encoded = alkstore::encode_payload(&payload)?;
|
||||||
|
let _ = encoded;
|
||||||
|
self.overlay.notifies.push(channel.to_string());
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn stage_save_offset(&mut self, stream: &str, consumer: &str, offset: i64) {
|
||||||
|
let entry = self
|
||||||
|
.overlay
|
||||||
|
.saves
|
||||||
|
.entry((stream.to_string(), consumer.to_string()))
|
||||||
|
.or_insert(i64::MIN);
|
||||||
|
if offset > *entry {
|
||||||
|
*entry = offset;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The commit-merge: move every staged effect into the committed state
|
||||||
|
/// under one guarded operation (the canonical queues → streams lock
|
||||||
|
/// order; each section's committed view is self-consistent at its
|
||||||
|
/// boundary), then fire the overlay's pending wakes — the
|
||||||
|
/// wake-at-commit discipline: nothing delivers until the merge lands.
|
||||||
|
fn merge_and_wake(state: Arc<State>, wakes: Arc<WakeBus>, mut overlay: Overlay) {
|
||||||
|
{
|
||||||
|
let mut q = state.queues.guard();
|
||||||
|
for (_, row) in std::mem::take(&mut overlay.jobs) {
|
||||||
|
q.live.insert(row.id, row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
{
|
||||||
|
let mut s = state.streams.guard();
|
||||||
|
for (stream, rows) in overlay.events.iter() {
|
||||||
|
for row in rows {
|
||||||
|
insert_event(&mut s, stream, row.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for ((stream, consumer), offset) in overlay.saves.iter() {
|
||||||
|
save_offset_row(&mut s, stream, consumer, *offset);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Wake-at-commit: staged notifies in stage order (duplicates ride —
|
||||||
|
// per-notify), plus one wake per staged event on its stream's
|
||||||
|
// engine-internal feed (per event — never coalesced).
|
||||||
|
let mut fired: VecDeque<String> = VecDeque::with_capacity(overlay.notifies.len());
|
||||||
|
fired.extend(overlay.notifies.iter().cloned());
|
||||||
|
for stream in overlay.events.keys() {
|
||||||
|
fired.push_back(stream_wake_channel(stream));
|
||||||
|
}
|
||||||
|
wakes.fire_commit_wakes(fired);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::store::MemStore;
|
||||||
|
use alkstore::{Delivery, Outbox, Store, TxHandle};
|
||||||
|
|
||||||
|
fn store() -> MemStore {
|
||||||
|
let store = MemStore::new();
|
||||||
|
store.clock.freeze(1_000_000);
|
||||||
|
store
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn listener(store: &MemStore, channel: &str) -> Box<dyn alkstore::WakeReceiver> {
|
||||||
|
store.listen(channel).await.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn listener_quiet(rx: &mut dyn alkstore::WakeReceiver) {
|
||||||
|
let outcome = tokio::time::timeout(std::time::Duration::from_millis(60), rx.recv()).await;
|
||||||
|
assert!(
|
||||||
|
matches!(outcome, Err(_) | Ok(None)),
|
||||||
|
"no wake may arrive in the quiet window"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn tx_staged_notify_delivers_nothing_pre_commit_one_wake_post() {
|
||||||
|
let store = store();
|
||||||
|
let mut rx = listener(&store, "orders").await;
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.notify_tx("orders", serde_json::json!({"n": 1}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
listener_quiet(rx.as_mut()).await;
|
||||||
|
assert!(tx.commit().await.is_ok());
|
||||||
|
let wake = tokio::time::timeout(std::time::Duration::from_millis(200), rx.recv())
|
||||||
|
.await
|
||||||
|
.expect("the wake delivers")
|
||||||
|
.expect("a wake");
|
||||||
|
assert_eq!(wake.channel, "orders");
|
||||||
|
// Per-notify: a second staged notify delivers its own wake.
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.notify_tx("orders", serde_json::json!(2)).await.unwrap();
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
assert!(rx.recv().await.is_some());
|
||||||
|
assert!(
|
||||||
|
rx.try_recv().unwrap().is_none(),
|
||||||
|
"never coalesced — exactly one wake per staged notify"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn enqueue_tx_stamps_the_plain_defaults_and_commits_atomically() {
|
||||||
|
let store = store();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let id = tx
|
||||||
|
.enqueue_tx(
|
||||||
|
"emails",
|
||||||
|
EnqueueOpts {
|
||||||
|
delay: Some(30),
|
||||||
|
max_attempts: Some(7),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
serde_json::json!({"to": "a@b.c"}),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let staged = tx.get_job_tx("emails", id).await.unwrap().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
staged.visibility_timeout_s, 300,
|
||||||
|
"the plain defaults stamp (no handle carries opts)"
|
||||||
|
);
|
||||||
|
assert_eq!(staged.max_attempts, 7);
|
||||||
|
assert_eq!(staged.backoff_base_s, 5);
|
||||||
|
assert_eq!(staged.dead_letter_retention_s, None);
|
||||||
|
assert_eq!(staged.run_at, 1_000_030);
|
||||||
|
assert_eq!(staged.state, JobState::Pending);
|
||||||
|
let committed_view = store
|
||||||
|
.queue("emails", alkstore::QueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
committed_view.get_job(id).await.unwrap().is_none(),
|
||||||
|
"other views see none pre-commit"
|
||||||
|
);
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
let job = committed_view.get_job(id).await.unwrap().unwrap();
|
||||||
|
assert_eq!(job.queue, "emails");
|
||||||
|
assert_eq!(
|
||||||
|
job.payload,
|
||||||
|
serde_json::to_vec(&serde_json::json!({"to": "a@b.c"})).unwrap()
|
||||||
|
);
|
||||||
|
assert_eq!(job.created_at, 1_000_000);
|
||||||
|
assert_eq!(
|
||||||
|
job.run_at, 1_000_030,
|
||||||
|
"the resolved values, not the raw fields"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn publish_tx_reads_and_merge_ordering_plus_wake_at_commit() {
|
||||||
|
let store = store();
|
||||||
|
let mut feed = store.wakes().attach(&stream_wake_channel("orders"));
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let a = tx.publish_tx("orders", serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = tx
|
||||||
|
.publish_with_key_tx("orders", Some("cust-7".into()), serde_json::json!(2))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(a < b);
|
||||||
|
let seen = tx.read_since_tx("orders", 0, 10).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
seen.len(),
|
||||||
|
2,
|
||||||
|
"the tx's own staged publishes are visible inside the tx"
|
||||||
|
);
|
||||||
|
assert_eq!(seen[0].key, None);
|
||||||
|
assert_eq!(seen[1].key.as_deref(), Some("cust-7"));
|
||||||
|
assert_eq!(
|
||||||
|
seen.iter().map(|e| e.offset).collect::<Vec<_>>(),
|
||||||
|
vec![a, b]
|
||||||
|
);
|
||||||
|
let handle = store.stream("orders").await.unwrap();
|
||||||
|
assert!(
|
||||||
|
handle.read_since(0, 10).await.unwrap().is_empty(),
|
||||||
|
"other views see none pre-commit"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
feed.rx().try_recv().is_err(),
|
||||||
|
"no wake fires before the merge"
|
||||||
|
);
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
let committed = handle.read_since(0, 10).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
committed.iter().map(|e| e.offset).collect::<Vec<_>>(),
|
||||||
|
vec![a, b]
|
||||||
|
);
|
||||||
|
assert_eq!(committed[1].payload_as::<i64>().unwrap(), 2);
|
||||||
|
assert!(
|
||||||
|
feed.rx().try_recv().is_ok(),
|
||||||
|
"one wake fired at the commit (after the merge)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn drop_without_commit_discards_the_whole_overlay_no_ghosts() {
|
||||||
|
let store = store();
|
||||||
|
let mut rx = listener(&store, "ch").await;
|
||||||
|
let handle = store.stream("s").await.unwrap();
|
||||||
|
let committed_anchor = handle.publish(serde_json::json!("anchor")).await.unwrap();
|
||||||
|
{
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
tx.publish_tx("s", serde_json::json!(2)).await.unwrap();
|
||||||
|
tx.notify_tx("ch", serde_json::json!(3)).await.unwrap();
|
||||||
|
tx.save_offset_tx("s", "c1", 999).await.unwrap();
|
||||||
|
drop(tx);
|
||||||
|
}
|
||||||
|
let q = store
|
||||||
|
.queue("q", alkstore::QueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(q.get_job(1).await.unwrap().is_none(), "no ghost job");
|
||||||
|
let events = handle.read_since(0, 10).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
events.len(),
|
||||||
|
1,
|
||||||
|
"no ghost event — only the committed anchor"
|
||||||
|
);
|
||||||
|
assert_eq!(events[0].offset, committed_anchor);
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
0,
|
||||||
|
"no ghost offset save"
|
||||||
|
);
|
||||||
|
listener_quiet(rx.as_mut()).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn other_views_see_none_pre_commit_and_rollback_leaves_no_read_subjects() {
|
||||||
|
let store = store();
|
||||||
|
let handle = store.stream("s").await.unwrap();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let offset = tx.publish_tx("s", serde_json::json!(1)).await.unwrap();
|
||||||
|
tx.save_offset_tx("s", "c1", offset).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
tx.get_offset_tx("s", "c1").await.unwrap(),
|
||||||
|
offset,
|
||||||
|
"the tx's own save is visible inside"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
0,
|
||||||
|
"other views see none pre-commit"
|
||||||
|
);
|
||||||
|
drop(tx);
|
||||||
|
assert!(
|
||||||
|
handle.read_since(0, 10).await.unwrap().is_empty(),
|
||||||
|
"post-rollback the read's subject is gone"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
handle.get_offset("c1").await.unwrap(),
|
||||||
|
0,
|
||||||
|
"no-ghosts for reads"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn save_offset_tx_is_monotone_and_only_the_staged_max_merges() {
|
||||||
|
let store = store();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.save_offset_tx("s", "c", 5).await.unwrap();
|
||||||
|
tx.save_offset_tx("s", "c", 3).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
tx.get_offset_tx("s", "c").await.unwrap(),
|
||||||
|
5,
|
||||||
|
"monotone across staged saves"
|
||||||
|
);
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
let handle = store.stream("s").await.unwrap();
|
||||||
|
assert_eq!(handle.get_offset("c").await.unwrap(), 5);
|
||||||
|
// A staged save below the committed checkpoint silently no-ops.
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
tx.save_offset_tx("s", "c", 2).await.unwrap();
|
||||||
|
assert_eq!(tx.get_offset_tx("s", "c").await.unwrap(), 5);
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
assert_eq!(handle.get_offset("c").await.unwrap(), 5);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cold]
|
||||||
|
#[inline(never)]
|
||||||
|
fn mid_flight_panic() -> alkstore::Error {
|
||||||
|
panic!("mid-flight, after staging all four write kinds")
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn with_tx_ok_commits_err_rolls_back_and_panic_rolls_back_through_the_unwind() {
|
||||||
|
let store = std::sync::Arc::new(store());
|
||||||
|
// Ok ⇒ commit.
|
||||||
|
let committed_id = {
|
||||||
|
let s = store.clone();
|
||||||
|
let captured = std::sync::Arc::new(std::sync::Mutex::new(0i64));
|
||||||
|
let sink = captured.clone();
|
||||||
|
s.with_tx(Box::new(move |tx| {
|
||||||
|
Box::pin(async move {
|
||||||
|
let id = tx
|
||||||
|
.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await?;
|
||||||
|
*sink.lock().unwrap_or_else(|p| p.into_inner()) = id;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
*std::sync::Arc::try_unwrap(captured)
|
||||||
|
.unwrap_or_else(|_| panic!("the sink outlives the run"))
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(|p| p.into_inner())
|
||||||
|
};
|
||||||
|
assert!(committed_id > 0);
|
||||||
|
// Err ⇒ rollback.
|
||||||
|
let rolled = {
|
||||||
|
let s = store.clone();
|
||||||
|
s.with_tx(Box::new(|tx| {
|
||||||
|
Box::pin(async move {
|
||||||
|
tx.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(2))
|
||||||
|
.await?;
|
||||||
|
Err::<(), alkstore::Error>(alkstore::Error::database(std::io::Error::other(
|
||||||
|
"business write failed",
|
||||||
|
)))
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
.await
|
||||||
|
};
|
||||||
|
assert!(rolled.is_err());
|
||||||
|
// Panic across the closure ⇒ rollback through the unwind's drop
|
||||||
|
// (the overlay discard runs nothing — panic-safe by construction).
|
||||||
|
let probe_store = store.clone();
|
||||||
|
let spawned = tokio::spawn(async move {
|
||||||
|
probe_store
|
||||||
|
.with_tx(Box::new(|tx| {
|
||||||
|
Box::pin(async move {
|
||||||
|
tx.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(3))
|
||||||
|
.await?;
|
||||||
|
tx.publish_tx("s", serde_json::json!(4)).await?;
|
||||||
|
tx.notify_tx("ch", serde_json::json!(5)).await?;
|
||||||
|
tx.save_offset_tx("s", "c", 42).await?;
|
||||||
|
Err::<(), alkstore::Error>(mid_flight_panic())
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
.await
|
||||||
|
});
|
||||||
|
let joined = spawned.await;
|
||||||
|
assert!(joined.is_err(), "the panic surfaced");
|
||||||
|
assert!(
|
||||||
|
joined.unwrap_err().is_panic(),
|
||||||
|
"the JoinError carries the panic"
|
||||||
|
);
|
||||||
|
// No residue of any kind (the uniform no-ghosts list).
|
||||||
|
let q = store
|
||||||
|
.queue("q", alkstore::QueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
q.get_job(1).await.unwrap().is_some(),
|
||||||
|
"the Ok leg committed"
|
||||||
|
);
|
||||||
|
for id in [2, 3] {
|
||||||
|
assert!(
|
||||||
|
q.get_job(id).await.unwrap().is_none(),
|
||||||
|
"no ghost job (err/panic)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let stream = store.stream("s").await.unwrap();
|
||||||
|
assert!(
|
||||||
|
stream.read_since(0, 10).await.unwrap().is_empty(),
|
||||||
|
"no ghost event"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
stream.get_offset("c").await.unwrap(),
|
||||||
|
0,
|
||||||
|
"no ghost offset save"
|
||||||
|
);
|
||||||
|
let mut rx = listener(&store, "ch").await;
|
||||||
|
assert!(rx.try_recv().unwrap().is_none(), "no ghost notify wake");
|
||||||
|
// The store remains usable: a fresh tx commits through the same
|
||||||
|
// seam (the next id gaps past the discarded allocations').
|
||||||
|
let usable = {
|
||||||
|
let s = store.clone();
|
||||||
|
let captured = std::sync::Arc::new(std::sync::Mutex::new(0i64));
|
||||||
|
let sink = captured.clone();
|
||||||
|
s.with_tx(Box::new(move |tx| {
|
||||||
|
Box::pin(async move {
|
||||||
|
let id = tx
|
||||||
|
.enqueue_tx("q", EnqueueOpts::default(), serde_json::json!(6))
|
||||||
|
.await?;
|
||||||
|
*sink.lock().unwrap_or_else(|p| p.into_inner()) = id;
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
}))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
*std::sync::Arc::try_unwrap(captured)
|
||||||
|
.unwrap_or_else(|_| panic!("the sink outlives the run"))
|
||||||
|
.lock()
|
||||||
|
.unwrap_or_else(|p| p.into_inner())
|
||||||
|
};
|
||||||
|
let next = store
|
||||||
|
.queue("q", alkstore::QueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert!(next.get_job(usable).await.unwrap().is_some());
|
||||||
|
assert!(usable > 3, "ids allocate forward — gaps are legal");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn get_job_tx_is_queue_scoped_and_sees_dead_rows() {
|
||||||
|
let store = store();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let id = tx
|
||||||
|
.enqueue_tx("emails", EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
let q = store
|
||||||
|
.queue("emails", alkstore::QueueOpts::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let claimed = q.claim_batch("w", 1).await.unwrap().pop().unwrap();
|
||||||
|
claimed.fail(Some("boom".into())).await.unwrap();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
let dead = tx.get_job_tx("emails", id).await.unwrap().unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
dead.state,
|
||||||
|
JobState::Dead,
|
||||||
|
"dead rows are get_job_tx-visible"
|
||||||
|
);
|
||||||
|
assert_eq!(dead.last_error.as_deref(), Some("boom"));
|
||||||
|
assert!(
|
||||||
|
tx.get_job_tx("other", id).await.unwrap().is_none(),
|
||||||
|
"queue-scoped: another queue reads None"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
struct NoDelivery;
|
||||||
|
impl Delivery for NoDelivery {
|
||||||
|
fn deliver<'a>(
|
||||||
|
&'a mut self,
|
||||||
|
_job: Box<dyn alkstore::JobHandle>,
|
||||||
|
) -> BoxedFuture<'a, Result<()>> {
|
||||||
|
Box::pin(async { Ok(()) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn outbox_enqueue_tx_stamps_the_60_5_5_set() {
|
||||||
|
let store = store();
|
||||||
|
// The concrete handle (crate-internal): the staging inspection
|
||||||
|
// reads the overlay before the commit-merge.
|
||||||
|
let mut tx = MemTxHandle::begin(
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
store.wakes().clone(),
|
||||||
|
);
|
||||||
|
let id = tx
|
||||||
|
.outbox_enqueue_tx(
|
||||||
|
"mail",
|
||||||
|
EnqueueOpts {
|
||||||
|
delay: Some(5),
|
||||||
|
max_attempts: Some(2),
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
serde_json::json!(1),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let staged = tx.overlay.jobs.get(&id).unwrap().clone();
|
||||||
|
assert_eq!(staged.queue, outbox_backing_queue_name("mail"));
|
||||||
|
assert_eq!(staged.visibility_timeout_s, 60);
|
||||||
|
assert_eq!(staged.max_attempts, 2, "the per-job override rides");
|
||||||
|
assert_eq!(staged.backoff_base_s, 5);
|
||||||
|
assert_eq!(staged.dead_letter_retention_s, None);
|
||||||
|
assert_eq!(staged.run_at, 1_000_005);
|
||||||
|
let err = tx
|
||||||
|
.get_job_tx(&outbox_backing_queue_name("mail"), id)
|
||||||
|
.await
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
matches!(err, Error::ReservedName { .. }),
|
||||||
|
"the derived name is unreachable by get_job_tx"
|
||||||
|
);
|
||||||
|
Box::new(tx).commit().await.unwrap();
|
||||||
|
// Commit makes it claimable by run_once exactly when the
|
||||||
|
// business write commits.
|
||||||
|
store.clock.advance(10);
|
||||||
|
let mut ob = crate::outbox::MemOutbox::new(
|
||||||
|
"mail".to_string(),
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
ob.run_once("w", &mut NoDelivery).await.unwrap(),
|
||||||
|
"the committed outbox job is claimable"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn entry_validation_on_every_tx_surface() {
|
||||||
|
let store = store();
|
||||||
|
let mut tx = MemTxHandle::begin(
|
||||||
|
store.state.clone(),
|
||||||
|
store.clock.clone(),
|
||||||
|
store.wakes().clone(),
|
||||||
|
);
|
||||||
|
for bad in ["", " ", "__alkstore_q"] {
|
||||||
|
assert!(
|
||||||
|
tx.enqueue_tx(bad, EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
assert!(tx.publish_tx(bad, serde_json::json!(1)).await.is_err());
|
||||||
|
assert!(
|
||||||
|
tx.publish_with_key_tx(bad, None, serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
assert!(tx.notify_tx(bad, serde_json::json!(1)).await.is_err());
|
||||||
|
assert!(tx.save_offset_tx(bad, "c", 1).await.is_err());
|
||||||
|
assert!(tx.get_job_tx(bad, 1).await.is_err());
|
||||||
|
assert!(tx.get_offset_tx(bad, "c").await.is_err());
|
||||||
|
assert!(tx.read_since_tx(bad, 0, 1).await.is_err());
|
||||||
|
assert!(tx.read_from_consumer_tx(bad, "c", 1).await.is_err());
|
||||||
|
assert!(
|
||||||
|
tx.outbox_enqueue_tx(bad, EnqueueOpts::default(), serde_json::json!(1))
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Consumer-local identifiers take the non-empty rule only.
|
||||||
|
let err = tx.save_offset_tx("s", "", 1).await.unwrap_err();
|
||||||
|
assert!(matches!(err, Error::InvalidName { .. }));
|
||||||
|
let err = tx.read_from_consumer_tx("s", "", 1).await.unwrap_err();
|
||||||
|
assert!(matches!(err, Error::InvalidName { .. }));
|
||||||
|
assert!(
|
||||||
|
tx.overlay.jobs.is_empty(),
|
||||||
|
"validation errors stage nothing"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn numeric_domain_guards_on_the_tx_reads() {
|
||||||
|
let store = store();
|
||||||
|
let mut tx = store.begin_tx().await.unwrap();
|
||||||
|
assert!(tx.read_since_tx("s", 0, 0).await.unwrap().is_empty());
|
||||||
|
assert!(tx.read_since_tx("s", 0, -5).await.unwrap().is_empty());
|
||||||
|
assert!(
|
||||||
|
tx.read_from_consumer_tx("s", "c", -1)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
tx.commit().await.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn concurrent_txs_merge_serially_without_isolation_questions() {
|
||||||
|
let store = store();
|
||||||
|
let mut t1 = store.begin_tx().await.unwrap();
|
||||||
|
let mut t2 = store.begin_tx().await.unwrap();
|
||||||
|
let a = t1.publish_tx("s", serde_json::json!(1)).await.unwrap();
|
||||||
|
let b = t2.publish_tx("s", serde_json::json!(2)).await.unwrap();
|
||||||
|
t1.commit().await.unwrap();
|
||||||
|
t2.commit().await.unwrap();
|
||||||
|
let handle = store.stream("s").await.unwrap();
|
||||||
|
let events = handle.read_since(0, 10).await.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
events.iter().map(|e| e.offset).collect::<Vec<_>>(),
|
||||||
|
vec![a, b],
|
||||||
|
"both commits merged; reads yield offset ASC — no isolation questions across handles"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
store.wakes().commit_watermark() >= 2,
|
||||||
|
"each commit fired one boundary"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
+169
-9
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
id: mem-engine-seam-tx
|
id: mem-engine-seam-tx
|
||||||
name: Mem engine — tx overlay, `Store` surface, `TxHandle` (staged `*_tx`, commit-merge + wake-at-commit, drop = discard)
|
name: Mem engine — tx overlay, `Store` surface, `TxHandle` (staged `*_tx`, commit-merge + wake-at-commit, drop = discard)
|
||||||
status: pending
|
status: completed
|
||||||
depends_on: [mem-engine-foundations]
|
depends_on: [mem-engine-foundations]
|
||||||
scope: moderate
|
scope: moderate
|
||||||
risk: high
|
risk: high
|
||||||
@@ -60,18 +60,18 @@ cross-engine rows land in `mem-engine-integration`.
|
|||||||
|
|
||||||
## Acceptance Criteria
|
## Acceptance Criteria
|
||||||
|
|
||||||
- [ ] The full `Store`/`TxHandle` trait surfaces implemented over
|
- [x] The full `Store`/`TxHandle` trait surfaces implemented over
|
||||||
guarded state; no engine types in consumer signatures
|
guarded state; no engine types in consumer signatures
|
||||||
- [ ] Overlay staging → commit-merge → wake-at-commit pinned by unit
|
- [x] Overlay staging → commit-merge → wake-at-commit pinned by unit
|
||||||
tests (a tx-staged notify delivers nothing pre-commit, one wake
|
tests (a tx-staged notify delivers nothing pre-commit, one wake
|
||||||
post-commit; enqueue/publish commit-atomicity shapes)
|
post-commit; enqueue/publish commit-atomicity shapes)
|
||||||
- [ ] Drop-without-commit and explicit rollback leave zero residue;
|
- [x] Drop-without-commit and explicit rollback leave zero residue;
|
||||||
`with_tx` panic rolls back through the unwind's drop
|
`with_tx` panic rolls back through the unwind's drop
|
||||||
- [ ] Tx reads see staged effects; other views see none pre-commit;
|
- [x] Tx reads see staged effects; other views see none pre-commit;
|
||||||
post-rollback reads' subjects gone (no-ghosts for reads)
|
post-rollback reads' subjects gone (no-ghosts for reads)
|
||||||
- [ ] Numeric-domain guards at entry (extents/durations/boundaries)
|
- [x] Numeric-domain guards at entry (extents/durations/boundaries)
|
||||||
pinned per [ADR-023](../architecture/decisions/023-fourth-review-round.md) §2
|
pinned per [ADR-023](../architecture/decisions/023-fourth-review-round.md) §2
|
||||||
- [ ] `cargo check --target wasm32-unknown-unknown -p alkstore-mem`
|
- [x] `cargo check --target wasm32-unknown-unknown -p alkstore-mem`
|
||||||
clean; workspace gates green
|
clean; workspace gates green
|
||||||
|
|
||||||
## References
|
## References
|
||||||
@@ -88,8 +88,168 @@ cross-engine rows land in `mem-engine-integration`.
|
|||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
> To be filled by implementation agent
|
> Filled by implementation agent, 2026-10-10.
|
||||||
|
|
||||||
|
1. **The whole engine surface landed here, deliberately** — this task
|
||||||
|
is wider than the SQLite/pg seam twins. Those seams rode
|
||||||
|
pre-ported/forked substrate ops and left the store's mechanism
|
||||||
|
methods as wiring stubs; the mem engine has no substrate layer, and
|
||||||
|
this task's description + acceptance criterion 1 pin "the full
|
||||||
|
`Store`/`TxHandle` trait surfaces implemented over guarded state"
|
||||||
|
— a Rust trait impl is all-or-nothing per method. So the seam task
|
||||||
|
realized the mechanisms' committed-state effect layer itself (queue
|
||||||
|
machine, stream log/offsets, TTL registry, scheduler table+runner,
|
||||||
|
notify bridge) and every mechanism handle, with the tx overlay as
|
||||||
|
the discipline centerpiece. **Consequence for the wave's mechanism
|
||||||
|
tasks**: `mem-engine-{notify-listen, streams, queues, locks,
|
||||||
|
scheduler-outbox}` now find their mechanisms *implemented* — their
|
||||||
|
work is the engine-side unit pins their acceptance criteria
|
||||||
|
enumerate (the ADR-010/015/009/008-§7 properties, the mem-owned
|
||||||
|
curve/parser/stats pins, the receiver close arms) plus any depth
|
||||||
|
refinement, against this foundation — not first implementations.
|
||||||
|
Recorded here so those agents don't re-derive the split.
|
||||||
|
2. **`serde_json = "1"` joined the manifest** (both engine twins carry
|
||||||
|
it): the trait signatures cross `serde_json::Value`, so
|
||||||
|
implementing them requires naming the type. Not a driver dep; the
|
||||||
|
ADR-024 §1 surface (core + tokio sync+time) is otherwise intact,
|
||||||
|
and the wasm32-compile gate stays green with it.
|
||||||
|
3. **Stage-time id/offset allocation — the pg-shaped story**: `enqueue_tx`
|
||||||
|
/`publish_tx` return the id/offset immediately, so ids and offsets
|
||||||
|
allocate at *stage* time from engine-wide atomic allocators
|
||||||
|
(`State::allocate_job_id`/`allocate_event_offset`). A rolled-back
|
||||||
|
(or panicking) tx's allocation gaps out — legal: ids are the
|
||||||
|
claim-order tiebreak (claims order by id ASC like the SQL engines'
|
||||||
|
enqueue order) and offsets are immutable/never renumbered (gaps
|
||||||
|
ordinary). Cross-handle concurrency: merges serialize; reads yield
|
||||||
|
`offset ASC`, so an event's committed position is always its stage
|
||||||
|
order (no isolation questions arise — an overlay is invisible to
|
||||||
|
every other view until its commit).
|
||||||
|
4. **Canonical guard order `schedules → queues → streams`** (locks and
|
||||||
|
wakes depend on no other section; documented in `state.rs`). The
|
||||||
|
commit-merge takes queues+streams in one scope; the scheduler tick
|
||||||
|
takes schedules+queues (the fire-and-advance one guarded op — the
|
||||||
|
ADR-009 §4 no-double-fire floor); the wake fire happens *after* the
|
||||||
|
merges land (wake-at-commit is structural: the fire code cannot run
|
||||||
|
first).
|
||||||
|
5. **Merge mechanics**: `merge_and_wake` drains the overlay into the
|
||||||
|
committed sections and then fires staged notifies (stage order,
|
||||||
|
duplicates ride — per-notify, never coalesced) plus one wake per
|
||||||
|
staged event on the stream's engine-internal feed. The stream
|
||||||
|
subscriber's wake channel is `__alkstore_stream_wake:{stream}` —
|
||||||
|
reserved-prefixed, so consumer `listen`/`notify` can never attach
|
||||||
|
to or fire into it (entry validation rejects the prefix), and the
|
||||||
|
EventReceiver attaches *before* its stored-offset read (no commit
|
||||||
|
can fall between; a buffered wake costs an empty re-poll, never a
|
||||||
|
missed event — the SQLite bridge's trick, now thread-free).
|
||||||
|
6. **notify's payload encoding run is the codec-typed entry check
|
||||||
|
only**: the wake contract delivers `Wake { channel }` alone
|
||||||
|
(ADR-008 §3), so mem stores no notifications transport — the
|
||||||
|
encoded bytes are dropped after the `Codec`-typed validation
|
||||||
|
(sqLite's transport row has no mem analog). `PayloadTooLarge` is
|
||||||
|
never produced at any size (the ADR-016 §5 non-occurrence arm,
|
||||||
|
pinned by a 2 MiB test).
|
||||||
|
7. **Lock semantics vs the SQLite substrate**: mem pins renew-past-
|
||||||
|
expiry *refused* (`false`) — the SQLite substrate's `lock_renew`
|
||||||
|
would revive a not-yet-purged lapsed row; the mem task text pins
|
||||||
|
the refusal and the suite's rows read the same way. Same-owner
|
||||||
|
re-`try_lock` grants without refreshing the deadline (the
|
||||||
|
`INSERT OR IGNORE` + owner-match parity); release past expiry stays
|
||||||
|
the holder's (`true`); a lapsed-then-re-acquired row's old-holder
|
||||||
|
release refuses (`false`). Locks are not tx-staged (no `lock_tx` —
|
||||||
|
contract verbatim).
|
||||||
|
8. **`ScheduleRow` carries the pre-parsed interval + resolved stamps**
|
||||||
|
— not the raw spec/name pair. The name is the map key (upsert by
|
||||||
|
name) and `parse_every_interval` runs once at register; the fire
|
||||||
|
path reads registered values, never the spec (the substrate
|
||||||
|
re-parses per tick — mem's third-owner parser makes that
|
||||||
|
unnecessary). `schedule()` encodes the payload once at register;
|
||||||
|
each fire enqueues those bytes with the registered stamps and the
|
||||||
|
`ScheduleOpts::{priority, expires}` riding themselves, resolved
|
||||||
|
against the fire instant (one clock read per tick).
|
||||||
|
9. **Runner lease**: owner is runner-unique (`scheduler-leader-{
|
||||||
|
instance}-{subsecond-nanos}` — an instance counter instead of
|
||||||
|
`process::id`, wasm-clean); renew-before-tick with the loss return
|
||||||
|
*before any fire*; the keep-awake sleep renews on a 2 s cadence
|
||||||
|
(an idle store must not drift past the 10 s TTL); idle sleep 60 s;
|
||||||
|
refused acquire at start = `Err(LeadershipLost)` (no runner exists
|
||||||
|
without the lock); a clean stop releases the lease. Boundary waits
|
||||||
|
ride realtime tokio time while stamps read the engine clock — the
|
||||||
|
tick/catch-up/skip-forward pins run through the crate-internal
|
||||||
|
`run_tick(now)` with the frozen clock, deterministic without
|
||||||
|
realtime waits.
|
||||||
|
10. **Row predicates mirror the substrate's SQL comparison-for-
|
||||||
|
comparison** (claim: pending-ready or lapsed-processing + budget
|
||||||
|
unspent + expiry unpassed; ack `claim_expires_at >= now`, reclaim
|
||||||
|
`<`, lock purge/renew boundary `<=`, sweep `expires_at <= now`).
|
||||||
|
The test-shaped pins caught one real defect class mid-flight:
|
||||||
|
ack/ack_batch originally *deleted before checking* the predicate —
|
||||||
|
a refused ack must leave the row exactly as it was (the substrate's
|
||||||
|
`DELETE … WHERE` checks per row); fixed, and the uniform-lapse
|
||||||
|
refusal tests now pin it.
|
||||||
|
11. **Sweep scoping note for later tasks**: `sweep_expired` counts both
|
||||||
|
the dead-letter moves *and* the retention deletions in its return
|
||||||
|
(substrate parity: `live_moved + retained`). A row whose
|
||||||
|
`expires_at` has already passed at enqueue is not claimable and
|
||||||
|
not pre-claim-dead-letterable — it sits for `sweep_expired`
|
||||||
|
(the pre-claim sweep only takes exhausted *claimable-shaped* rows).
|
||||||
|
12. **`enqueue_tx`/`outbox_enqueue_tx` stamp resolution** uses
|
||||||
|
`stamps_with_override` over the plain (300/3/5/none) and outbox
|
||||||
|
(60/5/5) derived sets exactly as ADR-020 §3 pins; the stamp sets
|
||||||
|
are named functions (one owner), resolution happens after
|
||||||
|
entry-point validation and before any effect lands.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
> To be filled on completion
|
> Filled on completion, 2026-10-10.
|
||||||
|
|
||||||
|
Landed the mem engine's seam: the complete `Store` trait surface wired
|
||||||
|
over the guarded state core — `notify`/`listen` (wake substrate bridge
|
||||||
|
with the pinned recv forms and close-at-engine-drop), `stream`/`queue`/
|
||||||
|
`outbox`/`try_lock` (validated constructors → boxed core-owned handle
|
||||||
|
traits carrying the mem mechanisms' full surfaces: ADR-010 queue state
|
||||||
|
machine with claims/reclaim/dead-letter/sweep and the equal-jitter
|
||||||
|
curve, ADR-015 log+offsets+trim+subscribe, ADR-008 §7 TTL registry,
|
||||||
|
ADR-014 outbox helper with `run_once`), `schedule`/`unschedule`/
|
||||||
|
`run_schedules` (ADR-009 collapse: `@every`-only third-owner parser, upsert
|
||||||
|
table, the 64-cap bounded catch-up + grid-aligned skip-forward tick
|
||||||
|
under the schedules→queues guard, leadership through the reserved
|
||||||
|
`__alkstore_scheduler` lock with renew-before-tick and the clean-stop/
|
||||||
|
`LeadershipLost` returns) — plus the load-bearing tx overlay: `begin_tx`
|
||||||
|
allocates the handle-private overlay; all eleven `*_tx` methods stage
|
||||||
|
with entry-point validation, `Codec`-typed encoding, one clock read,
|
||||||
|
ADR-020 §3 stamp resolution (plain 300/3/5/none, outbox 60/5/5, the
|
||||||
|
per-job `max_attempts` override); tx reads merge overlay-over-committed
|
||||||
|
(read-your-own-writes, queue-scoped + dead-visible `get_job_tx`,
|
||||||
|
monotone saves, offset-ASC reads, extent guards); `commit` is the one
|
||||||
|
guarded merge (queues → streams) followed by the wake-at-commit fire
|
||||||
|
(staged notifies + per-event stream wakes — nothing delivers pre-merge,
|
||||||
|
one wake per staged effect); drop (explicit or through an unwinding
|
||||||
|
panic) discards the overlay — no-ghosts by construction. Engine-side
|
||||||
|
arithmetic (`resolution.rs`) is mem's third-owner copy
|
||||||
|
(ADR-012 §2): stamp sets, `delay`-wins-over-`run_at`, relative-
|
||||||
|
`expires`, the reserved-prefix outbox derivation, `stamps_with_override`,
|
||||||
|
and the equal-jitter curve (std-only `RandomState` hash jitter, no
|
||||||
|
`rand`, pinned range).
|
||||||
|
|
||||||
|
`resolution.rs` — mem's third-owner arithmetic — landed with
|
||||||
|
the tx stamping and is what the mechanism tasks should consume for the
|
||||||
|
cross-engine equivalence rows (suite-pinned).
|
||||||
|
|
||||||
|
89 unit tests on `alkstore-mem` (17 foundations + 72 new): the overlay
|
||||||
|
discipline (staged-notify silence pre-commit → one wake post-commit,
|
||||||
|
stamps + commit-atomicity shapes, drop/rollback/panic-discard
|
||||||
|
no-ghosts across job/event/notify/offset, tx reads merged,
|
||||||
|
concurrent-tx merge order, with_tx Ok/Err/panic postures), numeric
|
||||||
|
domains (claim/read extents clamp, duration guards on lock paths,
|
||||||
|
negative-horizon trim, the `@every` grammar matrix, entry validation
|
||||||
|
across every surface), and the mechanism smoke pins (lifecycle,
|
||||||
|
ordering, validity predicate + late-heartbeat refusal, reclaim-eats-
|
||||||
|
attempt, dead-letter strings, sweep both-states + retention, stream
|
||||||
|
FIFO + trim + subscriber replay/wake + close arms, lock
|
||||||
|
contention/expiry, tick/catch-up/skip-forward, leadership contention +
|
||||||
|
clean stop, outbox ack/retry curves). Verified: workspace `cargo test`
|
||||||
|
green (core 25, suite-harness 3, **mem 89**, SQLite 124 lib + 191
|
||||||
|
suite, pg 124 lib + 25 suite), `cargo clippy --all-targets -- -D
|
||||||
|
warnings` clean, `cargo fmt --check` clean, and
|
||||||
|
`cargo check --target wasm32-unknown-unknown -p alkstore-mem` clean —
|
||||||
|
the wave-6 wasm gate stays green.
|
||||||
Reference in new issue
Block a user