sqlite: run_schedules' failed-renewal arm classifies as Database, never LeadershipLost (review 004 item 9) — a lost lease (Ok(false)) keeps the respawn arm unchanged; a failed renewal (closed store / storage failure) propagates its opaque classification, so respawning misdirects at a dead store no more; dated (2026-10-11) ADR-017 §2 class-1 annotation of the loss-vs-failure arm set in ADR-009 §1, core-contract §Errors, queues.md, engine-sqlite.md, core+sqlite doc comments; two tests pin both arms; pg twin recorded for follow-up; task -> completed
This commit is contained in:
1 parent
acdef94058
commit
9a9ccf26e0
8 files changed
+281
-21
No files matched your search
@@ -13,8 +13,14 @@
|
|||||||
//! - `run_schedules(stop)` — the leader loop (ADR-009 §1): acquire the
|
//! - `run_schedules(stop)` — the leader loop (ADR-009 §1): acquire the
|
||||||
//! leadership lock `__alkstore_scheduler` (the engine's own lock
|
//! leadership lock `__alkstore_scheduler` (the engine's own lock
|
||||||
//! machinery with a derived engine-internal TTL), then loop {renew —
|
//! machinery with a derived engine-internal TTL), then loop {renew —
|
||||||
//! on loss return `Err(LeadershipLost)` *before ticking*; tick; sleep
|
//! on a refused renewal return `Err(LeadershipLost)` *before
|
||||||
//! until the next due boundary or stop}. The tick's fire enqueues and
|
//! ticking* (the lease expired and re-acquired elsewhere; the
|
||||||
|
//! respawn arm); a *failed* renewal is not a loss and propagates its
|
||||||
|
//! opaque `Database` classification unchanged (the closed-store
|
||||||
|
//! posture — respawning a leader against a dead store is misdirected,
|
||||||
|
//! review 004 item 9 — and any other storage failure; 2026-10-11
|
||||||
|
//! classification); tick; sleep until the next due boundary or stop}.
|
||||||
|
//! The tick's fire enqueues and
|
||||||
//! the boundary advance commit in one `BEGIN IMMEDIATE` transaction —
|
//! the boundary advance commit in one `BEGIN IMMEDIATE` transaction —
|
||||||
//! writer serialization as the row lock on the schedule row (ADR-009
|
//! writer serialization as the row lock on the schedule row (ADR-009
|
||||||
//! §4's layer (b): even a rogue second ticker observing the same
|
//! §4's layer (b): even a rogue second ticker observing the same
|
||||||
@@ -213,12 +219,26 @@ pub(crate) fn run_schedules(
|
|||||||
let _ = lease.release().await;
|
let _ = lease.release().await;
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
let renewed = lease.renew(LEADER_TTL_S).await.unwrap_or(false);
|
let renewed = lease.renew(LEADER_TTL_S).await;
|
||||||
if !renewed {
|
match renewed {
|
||||||
// The lock was lost (expired and re-acquired elsewhere
|
Ok(true) => {}
|
||||||
// — exclusion lapses silently at TTL expiry). Return
|
Ok(false) => {
|
||||||
// before ticking: never fire on a stolen lease.
|
// The lock was lost (expired and re-acquired
|
||||||
return Err(Error::LeadershipLost);
|
// elsewhere — exclusion lapses silently at TTL
|
||||||
|
// expiry). Return before ticking: never fire on a
|
||||||
|
// stolen lease.
|
||||||
|
return Err(Error::LeadershipLost);
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
// A failed renewal is not a leadership loss: the
|
||||||
|
// lock semantics were never reached. The closed-
|
||||||
|
// store arm (the engine-wide `Database` posture with
|
||||||
|
// the "the store is closed" source — respawning a
|
||||||
|
// leader against a dead store is misdirected, review
|
||||||
|
// 004 item 9) and every other renewal failure
|
||||||
|
// propagate their opaque classification unchanged.
|
||||||
|
return Err(e);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
let soonest = {
|
let soonest = {
|
||||||
let writer = writer.clone();
|
let writer = writer.clone();
|
||||||
|
|||||||
@@ -599,3 +599,145 @@ async fn failed_tick_commit_replenishes_the_writer_slot() {
|
|||||||
drop(concrete);
|
drop(concrete);
|
||||||
cleanup(&dir);
|
cleanup(&dir);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Acceptance: the closed-store renewal arm reports the engine-wide
|
||||||
|
/// closed-store `Database` classification — never `LeadershipLost`
|
||||||
|
/// (review 004 item 9: flattening every renew `Err` into the loss arm
|
||||||
|
/// misdirected the documented respawn recipe at a dead store — an
|
||||||
|
/// operator respawns a leader against a store that is closed; the
|
||||||
|
/// lock semantics were never reached when the machinery itself
|
||||||
|
/// errors, 2026-10-11 classification). The runner closes mid-run; the
|
||||||
|
/// next loop-top renewal fails and the classification propagates
|
||||||
|
/// unchanged.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn closed_store_run_reports_the_closed_classification_not_leadership_lost() {
|
||||||
|
let dir = temp_dir("sched-closed");
|
||||||
|
let concrete = open_store(dir.join("store.db").to_str().unwrap(), Default::default()).unwrap();
|
||||||
|
|
||||||
|
concrete
|
||||||
|
.schedule("beat", "@every 1s", "closedq", json!(1), Default::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let runner = tokio::spawn(crate::scheduler::run_schedules(
|
||||||
|
StopToken::new(),
|
||||||
|
concrete.writer.clone(),
|
||||||
|
concrete.closed.clone(),
|
||||||
|
concrete.db_path.to_string_lossy().into_owned(),
|
||||||
|
concrete.commit_fault.clone(),
|
||||||
|
));
|
||||||
|
|
||||||
|
// Cross the renew arm at least once: the loop is demonstrably
|
||||||
|
// mid-run (one boundary fired) before the store closes.
|
||||||
|
let deadline = tokio::time::Instant::now() + Duration::from_secs(20);
|
||||||
|
loop {
|
||||||
|
let fired: i64 = {
|
||||||
|
let conn = rusqlite::Connection::open(dir.join("store.db")).unwrap();
|
||||||
|
conn.query_row(
|
||||||
|
"SELECT count(*) FROM __alkstore_live WHERE queue = 'closedq'",
|
||||||
|
[],
|
||||||
|
|r| r.get(0),
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
};
|
||||||
|
if fired >= 1 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
tokio::time::Instant::now() < deadline,
|
||||||
|
"the runner never fired the first boundary"
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
concrete.close();
|
||||||
|
|
||||||
|
// The exit is prompt (the sleep is at most one 1 s boundary wide;
|
||||||
|
// the next loop-top renewal sees the closed store).
|
||||||
|
let outcome = tokio::time::timeout(Duration::from_secs(10), runner)
|
||||||
|
.await
|
||||||
|
.expect("the closed-store exit must terminate promptly")
|
||||||
|
.unwrap();
|
||||||
|
match outcome {
|
||||||
|
Err(Error::Database(source)) => {
|
||||||
|
assert!(
|
||||||
|
source.to_string().contains("closed"),
|
||||||
|
"the closed-store arm carries the closed source chain: {source}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Err(Error::LeadershipLost) => {
|
||||||
|
panic!("a closed store must never report LeadershipLost (review 004 item 9)")
|
||||||
|
}
|
||||||
|
other => {
|
||||||
|
panic!("the closed-store arm reports the closed Database classification, got {other:?}")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
drop(concrete);
|
||||||
|
cleanup(&dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Acceptance: the genuine mid-run leadership-loss arm keeps its
|
||||||
|
/// disposition — `Err(LeadershipLost)`. Deleting the lease row while
|
||||||
|
/// the runner runs (the expired-and-re-acquired-elsewhere shape —
|
||||||
|
/// exclusion lapses silently at TTL expiry, ADR-008 §7) makes the next
|
||||||
|
/// loop-top renewal report the refused-lease outcome and the loop
|
||||||
|
/// returns the loss before any tick; the respawn recipe's arm is
|
||||||
|
/// unchanged by the 2026-10-11 classification.
|
||||||
|
#[tokio::test(flavor = "multi_thread")]
|
||||||
|
async fn mid_run_lease_loss_still_returns_leadership_lost() {
|
||||||
|
let (dir, store) = open_test_store("sched-midrun-loss");
|
||||||
|
|
||||||
|
store
|
||||||
|
.schedule("beat", "@every 1s", "lossq", json!(1), Default::default())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let runner_store =
|
||||||
|
open_store(dir.join("store.db").to_str().unwrap(), Default::default()).unwrap();
|
||||||
|
let runner = tokio::spawn(async move { runner_store.run_schedules(StopToken::new()).await });
|
||||||
|
|
||||||
|
// Demonstrably mid-run: leadership held, one boundary fired.
|
||||||
|
let deadline = tokio::time::Instant::now() + Duration::from_secs(20);
|
||||||
|
loop {
|
||||||
|
let fired: i64 = {
|
||||||
|
let conn = rusqlite::Connection::open(dir.join("store.db")).unwrap();
|
||||||
|
conn.query_row(
|
||||||
|
"SELECT count(*) FROM __alkstore_live WHERE queue = 'lossq'",
|
||||||
|
[],
|
||||||
|
|r| r.get(0),
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
};
|
||||||
|
if fired >= 1 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
tokio::time::Instant::now() < deadline,
|
||||||
|
"the runner never fired the first boundary"
|
||||||
|
);
|
||||||
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The rogue deletion: the runner's lease row vanishes.
|
||||||
|
{
|
||||||
|
let conn = rusqlite::Connection::open(dir.join("store.db")).unwrap();
|
||||||
|
conn.execute(
|
||||||
|
"DELETE FROM __alkstore_locks WHERE name = ?1",
|
||||||
|
[crate::lock::SCHEDULER_LEADER_LOCK],
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let outcome = tokio::time::timeout(Duration::from_secs(10), runner)
|
||||||
|
.await
|
||||||
|
.expect("the lease-loss exit must terminate promptly")
|
||||||
|
.unwrap();
|
||||||
|
assert!(
|
||||||
|
matches!(outcome, Err(Error::LeadershipLost)),
|
||||||
|
"a deleted lease mid-run must return LeadershipLost: {outcome:?}"
|
||||||
|
);
|
||||||
|
|
||||||
|
drop(store);
|
||||||
|
cleanup(&dir);
|
||||||
|
}
|
||||||
@@ -127,7 +127,13 @@ pub trait Store: Send + Sync {
|
|||||||
/// boundaries; sleep until the next boundary or `stop`}. A
|
/// boundaries; sleep until the next boundary or `stop`}. A
|
||||||
/// cancelled [`StopToken`] ends the sleep and returns the clean
|
/// cancelled [`StopToken`] ends the sleep and returns the clean
|
||||||
/// `Ok(())`; leadership loss returns `Err(LeadershipLost)` — the
|
/// `Ok(())`; leadership loss returns `Err(LeadershipLost)` — the
|
||||||
/// caller's recipe is to respawn (ADR-009 §1).
|
/// caller's recipe is to respawn (ADR-009 §1). A *failed* renewal
|
||||||
|
/// is not a loss and never returns `LeadershipLost`: a closed
|
||||||
|
/// store (respawning a leader against a dead store is misdirected
|
||||||
|
/// — the engine-wide closed-store posture) and storage failure
|
||||||
|
/// both surface the opaque `Err(Database)` fallback with the
|
||||||
|
/// engine detail preserved via the source chain (annotated
|
||||||
|
/// 2026-10-11 — review 004 item 9; ADR-017 §2 class 1).
|
||||||
fn run_schedules<'a>(&'a self, stop: StopToken) -> BoxedFuture<'a, Result<()>>;
|
fn run_schedules<'a>(&'a self, stop: StopToken) -> BoxedFuture<'a, Result<()>>;
|
||||||
|
|
||||||
/// Closure-scoped transaction — a wrapper over [`Store::begin_tx`]
|
/// Closure-scoped transaction — a wrapper over [`Store::begin_tx`]
|
||||||
|
|||||||
@@ -605,7 +605,15 @@ and `LeadershipLost` (`run_schedules` return on leadership loss) —
|
|||||||
both from [ADR-009](decisions/009-scheduler-collapse.md) §6 — are
|
both from [ADR-009](decisions/009-scheduler-collapse.md) §6 — are
|
||||||
part of contract v1's initial text (pre-release additions per
|
part of contract v1's initial text (pre-release additions per
|
||||||
[ADR-017](decisions/017-contract-versioning.md) class 1; the
|
[ADR-017](decisions/017-contract-versioning.md) class 1; the
|
||||||
"post-v1 additions" framing an earlier draft used is retired). Pinning rule: a variant exists only when
|
"post-v1 additions" framing an earlier draft used is retired).
|
||||||
|
`LeadershipLost` is the *loss* arm only — the lease expired and was
|
||||||
|
re-acquired elsewhere; a *failed* renewal is not a loss and never
|
||||||
|
returns `LeadershipLost`: a closed store (respawning a leader
|
||||||
|
against a dead store is misdirected — the engine-wide closed-store
|
||||||
|
posture) and storage failure both surface the opaque `Database`
|
||||||
|
fallback with the engine detail preserved via the source chain
|
||||||
|
(annotated 2026-10-11 — review 004 item 9; ADR-009 §1's dated
|
||||||
|
classification note). Pinning rule: a variant exists only when
|
||||||
callers can act differently on it. Queue claim "no work" and job/lock
|
callers can act differently on it. Queue claim "no work" and job/lock
|
||||||
boolean results are values, not errors; dead-letter is observable
|
boolean results are values, not errors; dead-letter is observable
|
||||||
state (`get_job`), not an error.
|
state (`get_job`), not an error.
|
||||||
|
|||||||
@@ -97,6 +97,19 @@ store.run_schedules(stop) -> Result<()> // runs until `stop`
|
|||||||
on it. (One taxonomy note: `LeadershipLost` is a *return-shape*
|
on it. (One taxonomy note: `LeadershipLost` is a *return-shape*
|
||||||
variant of the runner, pinned by the same act-differently rule as
|
variant of the runner, pinned by the same act-differently rule as
|
||||||
ADR-008 §5 — the caller acts on it by respawning.)
|
ADR-008 §5 — the caller acts on it by respawning.)
|
||||||
|
*(Annotated 2026-10-11 — review 004 item 9; amend-in-place per
|
||||||
|
ADR-017 §2 class 1, no publish tag exists.)* The loss arm is the
|
||||||
|
refused renewal (`renew` → `Ok(false)`): the lease expired and was
|
||||||
|
re-acquired elsewhere. A *failed* renewal is not a loss and never
|
||||||
|
returns `LeadershipLost` — the lock semantics were never reached:
|
||||||
|
a closed store (respawning a leader against a dead store is
|
||||||
|
misdirected; the engine-wide closed-store posture applies) and
|
||||||
|
storage failure both surface the opaque `Database` fallback with
|
||||||
|
the engine detail preserved via the source chain. Engines whose
|
||||||
|
renew is infallible (`Ok(bool)` only) have no failure arm to
|
||||||
|
classify; engines whose renew can fail must propagate the `Err`
|
||||||
|
unchanged (the SQLite engine's pre-annotation flattening —
|
||||||
|
`unwrap_or(false)` — was the finding).
|
||||||
- Schedules do not fire until a consumer runs `run_schedules` — the
|
- Schedules do not fire until a consumer runs `run_schedules` — the
|
||||||
opt-in is explicit, the no-ambient-timers posture holds,
|
opt-in is explicit, the no-ambient-timers posture holds,
|
||||||
[ADR-002](002-feature-scope.md)'s scheduler evidence is served, and
|
[ADR-002](002-feature-scope.md)'s scheduler evidence is served, and
|
||||||
|
|||||||
@@ -83,7 +83,7 @@ per-contract obligations live in the core spec and are not restated here.
|
|||||||
| streams | the forked substrate's stream machinery (inherited near-verbatim — the `__alkstore_stream` table's field set is already the contract's shape (its `topic` column carries the contract's `stream` field — the one name delta, ADR-012 §3's fidelity posture), the offset-ASC read path, keyed publish's nullable key column, monotone offset saves ([ADR-015](decisions/015-streams-depth.md)); explicit offset saves through the tx seam; `trim_to` is a `DELETE FROM … WHERE offset <= ?` inside the writer-slot lease) |
|
| streams | the forked substrate's stream machinery (inherited near-verbatim — the `__alkstore_stream` table's field set is already the contract's shape (its `topic` column carries the contract's `stream` field — the one name delta, ADR-012 §3's fidelity posture), the offset-ASC read path, keyed publish's nullable key column, monotone offset saves ([ADR-015](decisions/015-streams-depth.md)); explicit offset saves through the tx seam; `trim_to` is a `DELETE FROM … WHERE offset <= ?` inside the writer-slot lease) |
|
||||||
| queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) |
|
| queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) |
|
||||||
| named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) |
|
| named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) |
|
||||||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) |
|
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4). The loss decision is one owner — the loop's top-of-iteration renew: a refused renewal (`Ok(false)`, lease expired and re-acquired elsewhere) exits with `LeadershipLost`; a *failed* renewal is not a loss and propagates its opaque `Database` classification unchanged — the closed-store posture (a dead store must not misdirect the respawn recipe, review 004 item 9, annotated 2026-10-11 per [ADR-009](decisions/009-scheduler-collapse.md) §1) and every other renewal failure |
|
||||||
| begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
| begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||||
| handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) |
|
| handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) |
|
||||||
| commit/rollback | releases the writer slot |
|
| commit/rollback | releases the writer slot |
|
||||||
|
|||||||
@@ -207,8 +207,13 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects.
|
|||||||
the ADR; extension path: a consumer-inventory row first).
|
the ADR; extension path: a consumer-inventory row first).
|
||||||
- `run_schedules` is opt-in: schedules never fire without a runner
|
- `run_schedules` is opt-in: schedules never fire without a runner
|
||||||
(the no-ambient-timers posture). The runner: leadership lock →
|
(the no-ambient-timers posture). The runner: leadership lock →
|
||||||
{renew (on loss, return before ticking — `Err(LeadershipLost)`,
|
{renew (an actual lease loss — the refused renewal — returns before
|
||||||
distinct from clean `Ok(())` stop) → fire due boundaries → sleep}.
|
ticking with `Err(LeadershipLost)`, distinct from clean `Ok(())`
|
||||||
|
stop; a *failed* renewal is not a loss — closed store and storage
|
||||||
|
failure surface the opaque `Err(Database)` classification, a dead
|
||||||
|
store must not misdirect the respawn recipe, annotated 2026-10-11
|
||||||
|
— [ADR-009](decisions/009-scheduler-collapse.md) §1, review 004
|
||||||
|
item 9) → fire due boundaries → sleep}.
|
||||||
The `stop` parameter is a core-crate cloneable `StopToken`
|
The `stop` parameter is a core-crate cloneable `StopToken`
|
||||||
([ADR-019](decisions/019-mechanism-handle-surfaces.md) §4).
|
([ADR-019](decisions/019-mechanism-handle-surfaces.md) §4).
|
||||||
Leadership lock: reserved name `__alkstore_scheduler`
|
Leadership lock: reserved name `__alkstore_scheduler`
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
id: sqlite-fix-scheduler-error-classification
|
id: sqlite-fix-scheduler-error-classification
|
||||||
name: Classify run_schedules' renew-failure arms (review 004 item 9)
|
name: Classify run_schedules' renew-failure arms (review 004 item 9)
|
||||||
status: pending
|
status: completed
|
||||||
depends_on: []
|
depends_on: []
|
||||||
scope: narrow
|
scope: narrow
|
||||||
risk: low
|
risk: low
|
||||||
@@ -35,17 +35,17 @@ the observable error surface, per AGENTS.md's ADR-wins discipline.
|
|||||||
|
|
||||||
## Acceptance Criteria
|
## Acceptance Criteria
|
||||||
|
|
||||||
- [ ] The arm set is decided and recorded in Notes (which error maps
|
- [x] The arm set is decided and recorded in Notes (which error maps
|
||||||
where, citing the contract text that pins `run_schedules`' error
|
where, citing the contract text that pins `run_schedules`' error
|
||||||
surface); the closed-store arm no longer reports
|
surface); the closed-store arm no longer reports
|
||||||
`LeadershipLost`
|
`LeadershipLost`
|
||||||
- [ ] The window state was verified and recorded; a dated annotation
|
- [x] The window state was verified and recorded; a dated annotation
|
||||||
landed if class-1, or user sign-off if post-publish
|
landed if class-1, or user sign-off if post-publish
|
||||||
- [ ] A test pins the closed-store arm (and the genuine
|
- [x] A test pins the closed-store arm (and the genuine
|
||||||
leadership-lost arm's unchanged disposition)
|
leadership-lost arm's unchanged disposition)
|
||||||
- [ ] The documented respawn recipe's text (wherever it lives) matches
|
- [x] The documented respawn recipe's text (wherever it lives) matches
|
||||||
the new classification
|
the new classification
|
||||||
- [ ] `cargo test -p alkstore-sqlite`, clippy `-D warnings`, fmt
|
- [x] `cargo test -p alkstore-sqlite`, clippy `-D warnings`, fmt
|
||||||
clean; workspace gates green server-less
|
clean; workspace gates green server-less
|
||||||
|
|
||||||
## References
|
## References
|
||||||
@@ -58,8 +58,74 @@ the observable error surface, per AGENTS.md's ADR-wins discipline.
|
|||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
> To be filled by implementation agent
|
- **Window state (verified 2026-10-11)**: `git tag -l` is empty — no
|
||||||
|
publish tag exists, the class-1 window is open. The classification
|
||||||
|
landed as a dated ADR-023 §2-style annotation (2026-10-11,
|
||||||
|
amend-in-place per ADR-017 §2 class 1); post-publish sign-off was
|
||||||
|
not needed.
|
||||||
|
- **The arm set (decisions of record)** — citing the contract text
|
||||||
|
that pins `run_schedules`' error surface (core-contract.md §Errors:
|
||||||
|
`LeadershipLost` is "`run_schedules` return on leadership loss";
|
||||||
|
`Database` is the opaque fallback with engine detail preserved via
|
||||||
|
the source chain; ADR-009 §1 pins the respawn recipe on the loss):
|
||||||
|
- `renew` → `Ok(false)` — the genuine loss arm (the lease expired
|
||||||
|
and was re-acquired elsewhere; exclusion lapses silently at TTL
|
||||||
|
expiry, ADR-008 §7) — returns `Err(LeadershipLost)`, unchanged,
|
||||||
|
return-before-tick. The respawn recipe matches on this arm only.
|
||||||
|
- `renew` → `Err(e)` — the machinery never reached the lock
|
||||||
|
semantics; not a loss, never `LeadershipLost`. The `Err`
|
||||||
|
propagates unchanged: the closed-store arm arrives pre-classified
|
||||||
|
(SqliteLockHandle::renew checks the closed sign first and fails
|
||||||
|
closed with the engine-wide `Database` posture, source "the
|
||||||
|
store is closed" — the same classification every other entry
|
||||||
|
point and the tick's writer-acquire arm already report), and any
|
||||||
|
other storage failure carries the driver's source chain.
|
||||||
|
- No new variant minted — ADR-008 §5's act-differently rule holds
|
||||||
|
(`Closed` stays the wake-receiver variant; closed store is
|
||||||
|
`Database` engine-wide).
|
||||||
|
- **Engine notes**: the closed-store entry checks (`run_schedules`
|
||||||
|
top, lock `renew`/`release`/`try_lock`, tick's `writer.acquire`
|
||||||
|
via `ok_or_else(closed_store_error)`) already reported `Database`,
|
||||||
|
so the pre-annotation defect was solely the mid-run
|
||||||
|
`unwrap_or(false)` flattening. Mem's renew is infallible
|
||||||
|
(`Ok(bool)`) — no failure arm to classify; the ADR-009 annotation
|
||||||
|
covers that shape explicitly.
|
||||||
|
- **Unresolved twin (for a pg follow-up task)**:
|
||||||
|
`alkstore-postgres/src/scheduler.rs:682` carries the identical
|
||||||
|
`.unwrap_or(false)` flattening — post-close renew there fails
|
||||||
|
closed with `Database` (pinned by pg lock_tests) but a mid-run
|
||||||
|
renewal failure still misreports `LeadershipLost`. Review item 9
|
||||||
|
recorded the sqlite finding only; the contract annotation now
|
||||||
|
applies to both engines, so pg is divergent until a twin task
|
||||||
|
discharges it. Not fixed here (task scope is narrow/sqlite; pg
|
||||||
|
tests need the harness server).
|
||||||
|
- **Docs aligned**: core-contract.md §Errors, ADR-009 §1 (the dated
|
||||||
|
annotation of record), queues.md's runner bullet,
|
||||||
|
engine-sqlite.md's scheduler/outbox map row, core
|
||||||
|
`Store::run_schedules` doc comment, and the sqlite scheduler's
|
||||||
|
module doc — all state the loss-vs-failure arm set; the respawn
|
||||||
|
recipe text elsewhere ("the caller's recipe is to respawn",
|
||||||
|
matching on `LeadershipLost`) stays accurate since the loss arm is
|
||||||
|
unchanged.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
> To be filled on completion
|
The sqlite leader loop's top-of-iteration renew no longer flattens a
|
||||||
|
failed renewal into `LeadershipLost`: `Ok(false)` keeps the
|
||||||
|
`Err(LeadershipLost)` respawn arm unchanged, and a failed renewal
|
||||||
|
propagates its opaque `Database` classification unchanged (closed
|
||||||
|
store carries the engine-wide "the store is closed" source) —
|
||||||
|
respawning a leader against a dead store is no longer the documented
|
||||||
|
outcome. Landed with the dated (2026-10-11) classification
|
||||||
|
annotation in ADR-009 §1, core-contract.md §Errors, queues.md,
|
||||||
|
engine-sqlite.md, and the core/sqlite doc comments. Two new
|
||||||
|
acceptance tests pin the arms: `closed_store_run_reports_the_closed_
|
||||||
|
classification_not_leadership_lost` (a mid-run close exits with the
|
||||||
|
closed `Database` classification, never `LeadershipLost`) and
|
||||||
|
`mid_run_lease_loss_still_returns_leadership_lost` (a deleted lease
|
||||||
|
mid-run still returns the loss, the respawn arm unchanged). Verified:
|
||||||
|
`cargo test -p alkstore-sqlite` (203+25 green, both new pins pass),
|
||||||
|
workspace `cargo test` green server-less, clippy `-D warnings` clean,
|
||||||
|
`cargo fmt --check` clean, doc gate (`RUSTDOCFLAGS="-D warnings"`
|
||||||
|
cargo doc) clean, fuzz corpus replay green. The pg twin
|
||||||
|
(`scheduler.rs:682`) is recorded in Notes as a follow-up.
|
||||||
Reference in new issue
Block a user