general review 004 (pre-release, whole workspace): one HIGH slot-poisoning fix, two contract-observable divergence fixes, robustness + doc corrections — all gates re-run green
- sqlite scheduler tick: a failed COMMIT (or ROLLBACK) left the connection released with a stuck-open transaction, wedging the writer slot forever — the one commit site missing the drop-+-reopen-replenish arm; tick now takes the conn by value with a poisoned disposition (drop + replenish), the commit-fault seam reaches the tick, and failed_tick_commit_replenishes_the_writer_slot pins the arm - pg save_offset_tx stored GREATEST($3,0) where the wave-8 gate's negative-seed fix (raw seed, ADR-023 §2 2026-10-11 annotation) covered only the auto-commit form; mem's merged in-tx read clamped a staged negative seed against a phantom zero; both fixed and the tx-form totality pinned by a new stamped suite leg (in_tx_reads_see_own_writes) green on all three columns vs the live harness server - pg: a failed forwarder register leaked the channel refcount forever (no receiver to drop it) — listen/spawn_bridge now unregister on the register error, mirroring the subscribe-failure arm; register doc corrected to put the failure disposition at the call site - pg forwarder: nine poisoned-mutex expect() panics converted to the mem crate's into_inner recovery idiom (no-expect-outside-tests rule) - core encode_payload: a deeply-nested Value aborted the process in serde_json's unguarded serializer recursion and the fuzz pin blessed storing bytes no decoder could read back; encode now rejects nesting past 127 containers (the empirically-verified decode limit) with typed Codec — decode-aligned boundary pins added to the fuzz invariant and value_type_tests, applied pre-publish within the open class-1 window - scheduler sleep deadlines: Instant + Duration overflow caps (MAX_SLEEP_S, tick-and-resleep) on all three engines for far-future boundaries; sqlite tick claim stamp now saturating_add - docs read true again: Error::Closed scope (stream try_recv arm), validation whitespace wording, sqlite lib.rs phantom durability knobs, pg lock.rs honest autocommit wording, mem tx/stream merge + wake- granularity wording; wakes.rs stale allow(dead_code) removed - tests: value_type_tests exercised its own map_err instead of the function under test (fixed); unwrap_or_default reference encodes panic with cause - docs/reviews/004-pre-release-general-review.md records nine fixed + sixteen parked findings (sqlite seam panic-safety armor, watcher post-death-subscribe hang + initial-stat self-kill, pg forwarder TOCTOU/hang/diagnosability, pg BIGINT server-side stamp overflow, mem growth/outbox notes, pg identifier-truncation doc note, smaller) and two false positives caught on verification; implementation.md review-rounds entry + core-contract backlog rows updated - gates: build, workspace tests (sqlite 194, core 27, suite legs new), full live pg column 127+25+9, clippy -D warnings, fmt, fuzz replay 6/6, wasm32 check, doc gate -D warnings — all green
This commit is contained in:
1 parent
5385f6bee3
commit
029f885aa1
24 files changed
+738
-103
No files matched your search
@@ -11,9 +11,11 @@
|
||||
//! unsupported is the inherited lineage honesty, stated
|
||||
//! here as the crate's deployment boundary (compile-time engine
|
||||
//! identity is the only engine-discovery mechanism — pick this crate,
|
||||
//! get this posture). Durability knobs, pool sizing, and the watcher
|
||||
//! cadence are engine-configuration concerns on [`SqliteOpts`]
|
||||
//! (ADR-008 §6) — never contract surface.
|
||||
//! get this posture). The engine's configuration surface is
|
||||
//! [`SqliteOpts`] (the watcher cadence and the reader-pool size) —
|
||||
//! the durability settings (`synchronous = NORMAL`, the busy
|
||||
//! timeout) are bootstrapped constants, not knobs, per
|
||||
//! deployment.md's posture (ADR-008 §6) — never contract surface.
|
||||
//!
|
||||
//! **Writer parking** (ADR-007's negative consequence, stated
|
||||
//! honestly): the writer slot is a lease, and long transactions park
|
||||
|
||||
@@ -42,6 +42,15 @@ use crate::substrate::{
|
||||
scheduler_soonest, scheduler_tick, scheduler_unregister,
|
||||
};
|
||||
|
||||
/// The upper bound on one sleep slice, in seconds: a schedule whose
|
||||
/// next boundary is further out sleeps this long, ticks (a no-op), and
|
||||
/// sleeps again — the tick recomputes the soonest every iteration.
|
||||
/// The cap keeps `Instant + Duration` arithmetic on the sleep deadline
|
||||
/// inside every platform's `Instant` range (a far-future boundary's
|
||||
/// raw wait can exceed a platform's representable span; a bounded
|
||||
/// duration never does).
|
||||
const MAX_SLEEP_S: u64 = 60 * 60 * 24 * 30;
|
||||
|
||||
/// The leadership lock's TTL (seconds): engine-internal detail —
|
||||
/// renewed every loop iteration *and* across the sleeps (a runner that
|
||||
/// cannot renew has lost the lock, ADR-009 §4 — the lock machinery is
|
||||
@@ -164,6 +173,8 @@ pub(crate) fn run_schedules(
|
||||
stop: StopToken,
|
||||
writer: Arc<Writer>,
|
||||
closed: Arc<AtomicBool>,
|
||||
db_path: String,
|
||||
#[cfg(test)] commit_fault: crate::seam::commit_fault::Flag,
|
||||
) -> BoxedFuture<'static, Result<()>> {
|
||||
Box::pin(async move {
|
||||
if closed.load(Ordering::Acquire) {
|
||||
@@ -211,7 +222,32 @@ pub(crate) fn run_schedules(
|
||||
}
|
||||
let soonest = {
|
||||
let writer = writer.clone();
|
||||
with_writer(writer, scheduler_tick_transactional).await?
|
||||
let db_path = db_path.clone();
|
||||
#[cfg(test)]
|
||||
let commit_fault = commit_fault.clone();
|
||||
crate::seam::blocking(move || {
|
||||
let conn = writer.acquire().ok_or_else(closed_store_error)?;
|
||||
#[cfg(test)]
|
||||
let (out, conn, poisoned) =
|
||||
scheduler_tick_transactional(conn, commit_fault.clone());
|
||||
#[cfg(not(test))]
|
||||
let (out, conn, poisoned) = scheduler_tick_transactional(conn);
|
||||
if poisoned {
|
||||
// A failed COMMIT (or ROLLBACK) leaves the
|
||||
// transaction's state unknown: the connection
|
||||
// is dropped, not released — the tx-handle
|
||||
// commit arm's posture, with the slot
|
||||
// replenished via a fresh bootstrapped open.
|
||||
drop(conn);
|
||||
if let Ok(fresh) = crate::seam::open_writer_connection(&db_path) {
|
||||
writer.release(fresh);
|
||||
}
|
||||
} else {
|
||||
writer.release(conn);
|
||||
}
|
||||
out
|
||||
})
|
||||
.await?
|
||||
};
|
||||
if stop.is_cancelled() {
|
||||
let _ = lease.release().await;
|
||||
@@ -229,23 +265,47 @@ pub(crate) fn run_schedules(
|
||||
/// One tick under `BEGIN IMMEDIATE`: the fire enqueues + the boundary
|
||||
/// advance commit together, or the whole tick rolls back (crash
|
||||
/// mid-tick ⇒ the boundary refires; two tickers serialize on the
|
||||
/// writer). Returns the soonest next fire for the sleep (0 = none).
|
||||
fn scheduler_tick_transactional(conn: &rusqlite::Connection) -> Result<i64> {
|
||||
conn.execute_batch("BEGIN IMMEDIATE")
|
||||
.map_err(sqlite_error)?;
|
||||
/// writer). Takes the connection by value and returns it with its
|
||||
/// disposition: `poisoned` marks the commit-failure/rollback-failure
|
||||
/// arms — the transaction's state is unknown there (SQLite documented
|
||||
/// behavior: a failed COMMIT leaves the transaction active), and the
|
||||
/// caller must drop the connection and replenish the slot rather than
|
||||
/// release it (a released connection would keep the open transaction,
|
||||
/// silently swallowing every later write until close). Returns
|
||||
/// `(result, connection, poisoned)`; the soonest next fire is for the
|
||||
/// sleep (0 = none).
|
||||
fn scheduler_tick_transactional(
|
||||
conn: rusqlite::Connection,
|
||||
#[cfg(test)] commit_fault: crate::seam::commit_fault::Flag,
|
||||
) -> (Result<i64>, rusqlite::Connection, bool) {
|
||||
if let Err(begin) = conn.execute_batch("BEGIN IMMEDIATE") {
|
||||
// A failed BEGIN opened no transaction: the connection is
|
||||
// clean and releasable.
|
||||
return (Err(sqlite_error(begin)), conn, false);
|
||||
}
|
||||
let body = (|| {
|
||||
let now = crate::resolution::now_unix(conn)?;
|
||||
scheduler_tick(conn, now).map_err(sqlite_error)?;
|
||||
scheduler_soonest(conn).map_err(sqlite_error)
|
||||
let now = crate::resolution::now_unix(&conn)?;
|
||||
scheduler_tick(&conn, now).map_err(sqlite_error)?;
|
||||
scheduler_soonest(&conn).map_err(sqlite_error)
|
||||
})();
|
||||
match body {
|
||||
Ok(soonest) => {
|
||||
conn.execute_batch("COMMIT").map_err(sqlite_error)?;
|
||||
Ok(soonest)
|
||||
#[cfg(test)]
|
||||
let committed = if crate::seam::commit_fault::take(&commit_fault) {
|
||||
Err(crate::seam::commit_fault::failure())
|
||||
} else {
|
||||
conn.execute_batch("COMMIT")
|
||||
};
|
||||
#[cfg(not(test))]
|
||||
let committed = conn.execute_batch("COMMIT");
|
||||
match committed {
|
||||
Ok(()) => (Ok(soonest), conn, false),
|
||||
Err(commit) => (Err(sqlite_error(commit)), conn, true),
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
let _ = conn.execute_batch("ROLLBACK");
|
||||
Err(e)
|
||||
let rolled = conn.execute_batch("ROLLBACK");
|
||||
(Err(e), conn, rolled.is_err())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -265,7 +325,7 @@ async fn keep_awake_renewing(lease: &SqliteLockHandle, stop: &StopToken, soonest
|
||||
.unwrap_or(0);
|
||||
(soonest - now).max(1) as u64
|
||||
};
|
||||
let deadline = started + Duration::from_secs(wait_s);
|
||||
let deadline = started + Duration::from_secs(wait_s.min(MAX_SLEEP_S));
|
||||
loop {
|
||||
let now = tokio::time::Instant::now();
|
||||
if stop.is_cancelled() || now >= deadline {
|
||||
|
||||
@@ -293,7 +293,18 @@ impl Store for SqliteStore {
|
||||
) -> alkstore::BoxedFuture<'a, alkstore::Result<()>> {
|
||||
let writer = self.writer.clone();
|
||||
let closed = self.closed.clone();
|
||||
crate::scheduler::run_schedules(stop, writer, closed)
|
||||
let db_path = self.db_path.to_string_lossy().into_owned();
|
||||
#[cfg(test)]
|
||||
let fut = crate::scheduler::run_schedules(
|
||||
stop,
|
||||
writer,
|
||||
closed,
|
||||
db_path,
|
||||
self.commit_fault.clone(),
|
||||
);
|
||||
#[cfg(not(test))]
|
||||
let fut = crate::scheduler::run_schedules(stop, writer, closed, db_path);
|
||||
fut
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -543,3 +543,59 @@ async fn catch_up_replays_up_to_the_cap_then_skips_forward() {
|
||||
drop(store);
|
||||
cleanup(&dir);
|
||||
}
|
||||
|
||||
/// Acceptance: a failing `COMMIT` inside the scheduler tick's
|
||||
/// transaction (the same `SQLITE_FULL`-shaped seam injection as the
|
||||
/// tx-handle arm) surfaces as the opaque `Database`, and the writer
|
||||
/// slot is replenished through the tick's poison arm — the connection
|
||||
/// whose transaction state is unknowable (SQLite documented behavior:
|
||||
/// a failed COMMIT leaves the transaction active) is dropped, the slot
|
||||
/// refilled via a fresh bootstrapped open, the tx-handle commit arm's
|
||||
/// posture applied to the loop's tick (the one commit site that was
|
||||
/// missing it — the 2026-10-11 general review's finding; a released
|
||||
/// connection would have kept the open transaction and every later
|
||||
/// `BEGIN IMMEDIATE` on the slot would have errored until close). The
|
||||
/// store stays fully usable: schedule, notify, and a follow-up runner
|
||||
/// all proceed.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn failed_tick_commit_replenishes_the_writer_slot() {
|
||||
let dir = temp_dir("tick-commit-err");
|
||||
let concrete = open_store(dir.join("store.db").to_str().unwrap(), Default::default()).unwrap();
|
||||
|
||||
concrete.arm_commit_fault();
|
||||
|
||||
// The run errors at the first tick's commit (the leadership
|
||||
// acquire and the tick's BEGIN both preceded it).
|
||||
let outcome = concrete.run_schedules(StopToken::new()).await;
|
||||
match outcome {
|
||||
Err(Error::Database(source)) => {
|
||||
assert!(
|
||||
source.to_string().contains("full"),
|
||||
"the tick commit error carries the SQLITE_FULL-shaped source chain: {source}"
|
||||
);
|
||||
}
|
||||
other => panic!("a failed tick commit errs opaque Database, got {other:?}"),
|
||||
}
|
||||
|
||||
let registered = concrete
|
||||
.schedule(
|
||||
"after",
|
||||
"@every 10s",
|
||||
"work",
|
||||
json!({"n": 1}),
|
||||
ScheduleOpts::default(),
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
registered.is_ok(),
|
||||
"the slot must be grantable after the failed tick commit (replenished via reopen): \
|
||||
{registered:?}"
|
||||
);
|
||||
concrete
|
||||
.notify("probe", json!({"after": "tick-commit-error"}))
|
||||
.await
|
||||
.expect("auto-commit ops work after the failed tick commit");
|
||||
|
||||
drop(concrete);
|
||||
cleanup(&dir);
|
||||
}
|
||||
@@ -343,7 +343,7 @@ pub(crate) fn scheduler_tick(conn: &Connection, now: i64) -> rusqlite::Result<St
|
||||
backoff_base_s,
|
||||
dead_letter_retention_s,
|
||||
};
|
||||
let expires_at = expires_s.map(|s| now + s);
|
||||
let expires_at = expires_s.map(|s| now.saturating_add(s));
|
||||
let mut fires: i64 = 0;
|
||||
while next_fire_at <= now {
|
||||
if fires >= SCHEDULER_MAX_CATCHUP_FIRES {
|
||||
|
||||
Reference in new issue
Block a user