Contract-suite queue-depth rows (task suite-queue-depth-rows): the job-handle validity predicate row (in-window heartbeat/ack landing, late-heartbeat/post-lapse-ack/retry/fail refusals past a 1 s stamp with the row untouched, ack_batch per-id predicate live-1/lapsed-0/nonexistent-0, fail(None)→"failed" and retry-at-budget→"max attempts exceeded"), the ADR-010 depth row (reclaim consumes an attempt with claimed_at/deadline refreshed and the original holder refusing, reclaim-exhaustion dead-lettering with the pre-claim sweep — get_job-visible "max attempts exceeded"+died_at, cancel unconditional delete with the not-an-interrupt refusal shape plus pending/dead/missing arms), and the no-stranded-rows sweep row (both states move with "expired", unexpired/never-expiring untouched, retention TTL enforcing with the moved+deleted sum, None=forever) — each version-stamped per the suite convention (ADR-010 §1–§5, ADR-019 §3, ADR-008 §5), wired into both engines' suite targets (SQLite tokio tests, pg harness_row!s). M-1's retention-failure pin dispositioned engine-side per the task's honest call: the storage-level DELETE failure is not injectable through the contract surface, so it lands in the SQLite substrate's trigger seam (sweep_rolls_back_the_retention_half_with_the_move — the move half rolls back with the retention half), with the pg twin verified structurally (both halves inside in_tx's frame) and the disposition recorded in the task Notes. Verified: sqlite suite 13/13, pg suite 13/13 vs harness twice (postgres/poc@:15432), substrate sweep-rollback tests 4/4, workspace build/test green, clippy -D warnings, fmt clean

alkstore-contract-suite/src/lib.rs

@@ -39,7 +39,8 @@ pub mod version_stamp;
 pub use factory::StoreFactory;
 pub use properties::{
     duration_refusal_on_non_positive_ttl, enqueue_opts_resolution, extent_clamp_semantics,
-    in_tx_reads_see_own_writes, name_validation_rejects_empty_and_reserved,
-    payload_round_trip_stores_exact_encoding, payload_too_large_never_produced_on_sqlite,
-    payload_too_large_produced_on_pg, receiver_close_and_save_arms,
+    in_tx_reads_see_own_writes, job_handle_validity_predicate,
+    name_validation_rejects_empty_and_reserved, payload_round_trip_stores_exact_encoding,
+    payload_too_large_never_produced_on_sqlite, payload_too_large_produced_on_pg,
+    queue_depth_reclaim_and_dead_letter, receiver_close_and_save_arms, sweep_no_stranded_rows,
 };

alkstore-contract-suite/src/properties.rs

@@ -996,6 +996,548 @@ pub async fn payload_too_large_produced_on_pg(factory: &dyn StoreFactory) {
     factory.teardown().await.expect("factory teardown");
 }
 
+/// The suite's lapse-tolerance sleep: bounded well past a 1-second
+/// stamp's resolution (a stamp set on second `T` has lapsed for any
+/// clock read `> T + 1`; 2.5 s guarantees it, 3.5 s the `expires: 2`
+/// stamps). Timing posture: state outcomes are asserted, never
+/// durations.
+fn past_stamp_sleep(millis: u64) {
+    std::thread::sleep(std::time::Duration::from_millis(millis));
+}
+
+/// **Job-handle validity predicate** — the uniform rule (processing
+/// state + unexpired claim deadline) on every handle op: an in-window
+/// heartbeat lands (renewal) and an in-window ack deletes
+/// (delete-on-ack); the late heartbeat is refused exactly at the
+/// lapsed claim — the short-stamp row claims with a 1 s visibility
+/// stamp and tolerance-sleeps past it (the sleep-then-assert-state
+/// shape is sequential single-store and fits the suite's determinism
+/// posture); the post-lapse ack does not land (at-least-once: the
+/// worker's ack does not land after lapse), and `retry`'s refusal is
+/// `false` with the row untouched (state, attempts, claimant, and
+/// deadline all as the claim left them). `fail(None)` after lapse is
+/// refused identically — deadline lapse refuses *all* ops, not just
+/// heartbeat. `ack_batch` applies the predicate per id: a lapsed id
+/// and a never-claimed id are silently not counted — the returned
+/// count reflects only the live acks.
+///
+/// The engine-default strings land identically: `fail(None)` records
+/// `"failed"`, and retry-at-exhausted-budget dead-letters with
+/// `"max attempts exceeded"` (the reclaim-exhaustion arm of that
+/// string pins in the queue-depth row's exhaustion leg).
+///
+/// Contract stamp: ADR-010 §2 (the uniform validity predicate — the
+/// late-heartbeat refusal, post-lapse ack refusal); ADR-010 §1
+/// (delete-on-ack; `ack_batch`'s per-id predicate, partial success
+/// ordinary); ADR-010 §4 (the engine-default strings); ADR-019 §3
+/// (the claim handle's op surface — one-shot consumables, repeatable
+/// heartbeat); ADR-008 §5 (refusals are `false`, never errors).
+pub async fn job_handle_validity_predicate(factory: &dyn StoreFactory) {
+    let store = factory.open().await.expect("factory opens a store");
+
+    let slow = store
+        .queue("handles", alkstore::QueueOpts::default())
+        .await
+        .unwrap();
+    let fast = store
+        .queue(
+            "lapse",
+            alkstore::QueueOpts {
+                visibility_timeout_s: 1,
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+
+    // In-window ops land: renewal + delete-on-ack under the live
+    // claim.
+    let live_id = fast
+        .enqueue(json!({"leg": "in-window"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let held = fast
+        .claim_one("w1")
+        .await
+        .unwrap()
+        .expect("a ready row claims");
+    assert_eq!(held.job().id, live_id);
+    assert_eq!(held.job().state, JobState::Processing);
+    assert!(
+        held.heartbeat(600).await.unwrap(),
+        "an in-window heartbeat lands (renewal)"
+    );
+    let mid = fast
+        .get_job(live_id)
+        .await
+        .unwrap()
+        .expect("the row still lives mid-claim");
+    assert_eq!(mid.state, JobState::Processing);
+    assert!(held.ack().await.unwrap(), "an in-window ack lands");
+    assert!(
+        fast.get_job(live_id).await.unwrap().is_none(),
+        "delete-on-ack completed work leaves no row"
+    );
+
+    // Lapse legs: one claim per refusal op, all stamped with the 1 s
+    // visibility, one tolerance sleep past every deadline at once.
+    let ack_refused = fast
+        .enqueue(json!({"leg": "ack-refused"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let retry_refused = fast
+        .enqueue(json!({"leg": "retry-refused"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let fail_refused = fast
+        .enqueue(json!({"leg": "fail-refused"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let lapsed_ack = fast.claim_one("w2").await.unwrap().unwrap();
+    let lapsed_retry = fast.claim_one("w2").await.unwrap().unwrap();
+    let lapsed_fail = fast.claim_one("w2").await.unwrap().unwrap();
+    assert_eq!(
+        (
+            lapsed_ack.job().id,
+            lapsed_retry.job().id,
+            lapsed_fail.job().id
+        ),
+        (ack_refused, retry_refused, fail_refused),
+        "claims hand out in enqueue order (equal priority)"
+    );
+    let retry_before = lapsed_retry.job().clone();
+    past_stamp_sleep(2_500);
+    assert!(
+        !lapsed_ack.heartbeat(600).await.unwrap(),
+        "late heartbeat must be refused: the deadline has lapsed"
+    );
+    assert!(
+        !lapsed_retry
+            .retry(Some("boom".to_string()), Some(5))
+            .await
+            .unwrap(),
+        "retry's refusal under the lapsed claim is false"
+    );
+    assert!(
+        !lapsed_fail.fail(None).await.unwrap(),
+        "deadline lapse refuses fail identically — the predicate is uniform"
+    );
+    let retry_after = fast
+        .get_job(retry_refused)
+        .await
+        .unwrap()
+        .expect("the retried-at row survived");
+    assert_eq!(
+        retry_after, retry_before,
+        "retry's refusal leaves the row untouched: the claim keeps pre-call state"
+    );
+
+    // ack_batch applies the predicate per id: the live id counts, the
+    // lapsed id and a never-claimed id silently do not. (Own queue —
+    // the lapsed legs above left reclaimable rows in `lapse`, and any
+    // later claim there would reclaim FIFO-first. The live id survives
+    // the 1 s stamp the same way the lapsed one lapsed: a renewal.)
+    let batch = store
+        .queue(
+            "batch",
+            alkstore::QueueOpts {
+                visibility_timeout_s: 1,
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let batch_live_id = batch
+        .enqueue(json!({"leg": "batch-live"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let batch_lapse_id = batch
+        .enqueue(json!({"leg": "batch-lapsed"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let batch_live = batch.claim_one("w3").await.unwrap().unwrap();
+    let batch_lapsed = batch.claim_one("w3").await.unwrap().unwrap();
+    assert_eq!(
+        (batch_live.job().id, batch_lapsed.job().id),
+        (batch_live_id, batch_lapse_id),
+        "claims hand out in enqueue order (equal priority)"
+    );
+    assert!(
+        batch_live.heartbeat(600).await.unwrap(),
+        "the renewal keeps this id's claim live through the batch"
+    );
+    past_stamp_sleep(2_500);
+    let counted = batch
+        .ack_batch(&[batch_live_id, batch_lapse_id, 987_654_321])
+        .await
+        .unwrap();
+    assert_eq!(
+        counted, 1,
+        "the returned count reflects only the live acks (lapsed and nonexistent ids not counted)"
+    );
+    assert!(
+        batch.get_job(batch_live_id).await.unwrap().is_none(),
+        "the counted id acked (deleted)"
+    );
+    let lapsed_alive = batch
+        .get_job(batch_lapse_id)
+        .await
+        .unwrap()
+        .expect("the lapsed id's row survives the batch");
+    assert_eq!(
+        lapsed_alive.state,
+        JobState::Processing,
+        "the lapsed id was silently not counted, not deleted"
+    );
+
+    // The engine-default strings (fail(None); retry at exhausted
+    // budget) on a long-visibility queue — no lapse interference.
+    let failed = slow
+        .enqueue(
+            json!({"leg": "fail-default"}),
+            EnqueueOpts {
+                max_attempts: Some(1),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let held_fail = slow.claim_one("w4").await.unwrap().unwrap();
+    assert_eq!(held_fail.job().id, failed);
+    assert!(held_fail.fail(None).await.unwrap());
+    let dead_failed = slow
+        .get_job(failed)
+        .await
+        .unwrap()
+        .expect("fail(None) dead-letters");
+    assert_eq!(dead_failed.state, JobState::Dead);
+    assert_eq!(dead_failed.last_error.as_deref(), Some("failed"));
+    assert!(dead_failed.died_at.is_some());
+
+    let exhausting = slow
+        .enqueue(
+            json!({"leg": "exhaustion"}),
+            EnqueueOpts {
+                max_attempts: Some(1),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let held_retry = slow.claim_one("w5").await.unwrap().unwrap();
+    assert_eq!(held_retry.job().id, exhausting);
+    assert!(held_retry.retry(None, Some(5)).await.unwrap());
+    let dead_exhausted = slow
+        .get_job(exhausting)
+        .await
+        .unwrap()
+        .expect("retry at exhausted budget dead-letters");
+    assert_eq!(dead_exhausted.state, JobState::Dead);
+    assert_eq!(
+        dead_exhausted.last_error.as_deref(),
+        Some("max attempts exceeded"),
+        "the engine-default exhaustion string lands identically"
+    );
+    assert!(dead_exhausted.died_at.is_some());
+
+    factory.teardown().await.expect("factory teardown");
+}
+
+/// **Queue-depth reclaim and dead-letter** — the ADR-010 depth
+/// properties over a 1 s-visibility queue: a visibility reclaim
+/// consumes an attempt (the expired claim reclaims to a second
+/// claimant with `attempts` incremented and `claimed_at` refreshed —
+/// the original holder's ops refuse after the reclaim), exhaustion
+/// dead-letters into get_job-visible dead storage (`"max attempts
+/// exceeded"` + `died_at` — the pre-claim sweep shape: a row whose
+/// reclaim consumed its budget dead-letters at the next claim), and
+/// `cancel` is an unconditional delete of either state — not an
+/// interrupt: the row is gone (live and dead alike refuse `cancel`
+/// with `false`), and the holder's next op refuses with the same
+/// shape as a lapse.
+///
+/// Contract stamp: ADR-010 §2 (a reclaim consumes an attempt; the
+/// original holder refuses post-reclaim); ADR-010 §3 (exhaustion via
+/// reclaim dead-letters); ADR-010 §1 (cancel — unconditional, not an
+/// interrupt); ADR-010 §4 (dead-visible `get_job` with
+/// `last_error`/`died_at`; the exhaustion string).
+pub async fn queue_depth_reclaim_and_dead_letter(factory: &dyn StoreFactory) {
+    let store = factory.open().await.expect("factory opens a store");
+    let depth = store
+        .queue(
+            "depth",
+            alkstore::QueueOpts {
+                visibility_timeout_s: 1,
+                max_attempts: 2,
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+
+    // Visibility reclaim consumes an attempt.
+    let reclaimed = depth
+        .enqueue(json!({"leg": "reclaim"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let first = depth.claim_one("w1").await.unwrap().unwrap();
+    assert_eq!(first.job().id, reclaimed);
+    assert_eq!(first.job().attempts, 1);
+    let t1 = first.job().claimed_at.expect("claim stamps claimed_at");
+    let e1 = first
+        .job()
+        .claim_expires_at
+        .expect("claim stamps the deadline");
+    past_stamp_sleep(2_500);
+
+    let second = depth
+        .claim_one("w2")
+        .await
+        .unwrap()
+        .expect("the lapsed claim reclaims");
+    assert_eq!(second.job().id, reclaimed, "the same row reclaims");
+    assert_eq!(second.job().attempts, 2, "a reclaim consumes an attempt");
+    assert_eq!(second.job().state, JobState::Processing);
+    assert_eq!(second.job().worker_id.as_deref(), Some("w2"));
+    let t2 = second
+        .job()
+        .claimed_at
+        .expect("the reclaim re-stamps claimed_at");
+    let e2 = second
+        .job()
+        .claim_expires_at
+        .expect("the reclaim re-stamps the deadline");
+    assert!(t2 > t1, "claimed_at is refreshed by the reclaim");
+    assert!(e2 > e1, "the deadline is refreshed, not inherited");
+    assert!(
+        !first.heartbeat(600).await.unwrap(),
+        "the original holder refuses after the reclaim"
+    );
+    assert!(second.ack().await.unwrap(), "the reclaiming holder acks");
+    assert!(depth.get_job(reclaimed).await.unwrap().is_none());
+
+    // Exhaustion: the reclaim that spends the budget, then the
+    // next claim's pre-claim sweep dead-letters the row.
+    let exhausted = depth
+        .enqueue(json!({"leg": "exhausted"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    depth.claim_one("w1").await.unwrap().unwrap();
+    past_stamp_sleep(2_500);
+    let at_budget = depth.claim_one("w2").await.unwrap().unwrap();
+    assert_eq!(at_budget.job().id, exhausted);
+    assert_eq!(
+        at_budget.job().attempts,
+        at_budget.job().max_attempts,
+        "the reclaim spent the budget"
+    );
+    past_stamp_sleep(2_500);
+    assert!(
+        depth.claim_one("w3").await.unwrap().is_none(),
+        "the budget-spent row dead-letters instead of reclaiming"
+    );
+    let dead = depth
+        .get_job(exhausted)
+        .await
+        .unwrap()
+        .expect("the dead row is get_job-visible");
+    assert_eq!(
+        dead.state,
+        JobState::Dead,
+        "exhaustion moves to dead storage"
+    );
+    assert_eq!(dead.last_error.as_deref(), Some("max attempts exceeded"));
+    assert!(dead.died_at.is_some());
+
+    // Cancel is an unconditional delete — either state, any holder —
+    // and not an interrupt: the holder's next op refuses the same
+    // shape as a lapse, and nothing re-enters the claim path.
+    let cancelled_live = depth
+        .enqueue(json!({"leg": "cancel-live"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let holder = depth.claim_one("w1").await.unwrap().unwrap();
+    assert_eq!(holder.job().id, cancelled_live);
+    assert!(depth.cancel(cancelled_live).await.unwrap());
+    assert!(
+        depth.get_job(cancelled_live).await.unwrap().is_none(),
+        "cancel deletes the row whole: no re-claim, no dead row"
+    );
+    assert!(
+        !holder.heartbeat(600).await.unwrap(),
+        "not an interrupt — the holder's next op refuses identically"
+    );
+
+    let cancelled_pending = depth
+        .enqueue(json!({"leg": "cancel-pending"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    assert!(depth.cancel(cancelled_pending).await.unwrap());
+    assert!(
+        depth.claim_one("w1").await.unwrap().is_none(),
+        "a cancelled pending row never becomes claimable"
+    );
+
+    assert!(
+        !depth.cancel(987_654_321).await.unwrap(),
+        "no missing live row cancels"
+    );
+    assert!(
+        !depth.cancel(exhausted).await.unwrap(),
+        "a dead row is outside cancel's live-row reach"
+    );
+    assert!(
+        depth.get_job(exhausted).await.unwrap().is_some(),
+        "cancel on a dead id deletes nothing"
+    );
+
+    factory.teardown().await.expect("factory teardown");
+}
+
+/// **Sweep: no stranded rows** — `sweep_expired()` moves *every*
+/// past-expiry row (any state: the pending-expired and the
+/// processing-expired alike — an enqueued job with an `expires`
+/// deadline is eventually in exactly one of pending, processing, or
+/// dead, never stuck unreachable), stamps the `"expired"` string, and
+/// enforces dead-letter retention (rows past a set
+/// `dead_letter_retention_s` delete; retention `None` = forever) —
+/// the count being the moved + retention-deleted sum. Unexpired and
+/// never-expiring rows stay untouched.
+///
+/// The M-1 follow-up (the sweep's retention half rolling back with
+/// the sweep on a retention-DELETE failure) is pinned engine-side —
+/// the failure is storage-level, not injectable through the contract
+/// surface; see the SQLite substrate's
+/// `sweep_rolls_back_the_retention_half_with_the_move` and this
+/// task's Notes for the disposition.
+///
+/// Contract stamp: ADR-010 §5 (the no-stranded-rows property, both
+/// states, and the retention enforcement); ADR-010 §4 (the
+/// `"expired"` string; retention default = forever).
+pub async fn sweep_no_stranded_rows(factory: &dyn StoreFactory) {
+    let store = factory.open().await.expect("factory opens a store");
+
+    let sweeping = store
+        .queue("sweepme", alkstore::QueueOpts::default())
+        .await
+        .unwrap();
+    // The processing-expired row is claimed first (it is the only row
+    // at claim time) with a 2 s expiry stamp; the pending-expired row
+    // enqueues after the claim so FIFO never hands it out.
+    let processing = sweeping
+        .enqueue(
+            json!({"leg": "expired-processing"}),
+            EnqueueOpts {
+                expires: Some(2),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let held = sweeping.claim_one("w1").await.unwrap().unwrap();
+    assert_eq!(held.job().id, processing, "the expiry-stamped row claims");
+    let pending = sweeping
+        .enqueue(
+            json!({"leg": "expired-pending"}),
+            EnqueueOpts {
+                expires: Some(2),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let unexpired = sweeping
+        .enqueue(
+            json!({"leg": "unexpired"}),
+            EnqueueOpts {
+                expires: Some(3600),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let never = sweeping
+        .enqueue(json!({"leg": "never-expiring"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    past_stamp_sleep(3_500);
+
+    let moved = sweeping.sweep_expired().await.unwrap();
+    assert_eq!(
+        moved, 2,
+        "every past-expiry row moves — both states, exactly the expired ones"
+    );
+    let dead_processing = sweeping
+        .get_job(processing)
+        .await
+        .unwrap()
+        .expect("the processing-expired row moved to dead");
+    assert_eq!(dead_processing.state, JobState::Dead);
+    assert_eq!(dead_processing.last_error.as_deref(), Some("expired"));
+    assert!(dead_processing.died_at.is_some());
+    let dead_pending = sweeping
+        .get_job(pending)
+        .await
+        .unwrap()
+        .expect("the pending-expired row moved to dead");
+    assert_eq!(dead_pending.state, JobState::Dead);
+    assert_eq!(dead_pending.last_error.as_deref(), Some("expired"));
+    for (id, label) in [(unexpired, "unexpired"), (never, "never-expiring")] {
+        let live = sweeping.get_job(id).await.unwrap().unwrap();
+        assert_eq!(live.state, JobState::Pending, "{label} row untouched");
+    }
+    drop(held);
+
+    // Retention enforcement: a dead row past its stamp deletes; the
+    // same-shaped row on a retention-None queue lives forever.
+    let retained = store
+        .queue(
+            "retained-1s",
+            alkstore::QueueOpts {
+                dead_letter_retention_s: Some(1),
+                ..Default::default()
+            },
+        )
+        .await
+        .unwrap();
+    let keeping = store
+        .queue("keeper", alkstore::QueueOpts::default())
+        .await
+        .unwrap();
+    let expires_1s = retained
+        .enqueue(json!({"leg": "aged-out"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let held_dies = retained.claim_one("w1").await.unwrap().unwrap();
+    assert!(held_dies.fail(None).await.unwrap());
+    let expires_never = keeping
+        .enqueue(json!({"leg": "kept"}), EnqueueOpts::default())
+        .await
+        .unwrap();
+    let kept_dies = keeping.claim_one("w1").await.unwrap().unwrap();
+    assert!(kept_dies.fail(None).await.unwrap());
+    past_stamp_sleep(2_500);
+
+    let deleted = retained.sweep_expired().await.unwrap();
+    assert_eq!(
+        deleted, 1,
+        "the retention deletion reports through the sweep count"
+    );
+    assert!(
+        retained.get_job(expires_1s).await.unwrap().is_none(),
+        "the dead row past its retention stamp is gone"
+    );
+    assert_eq!(keeping.sweep_expired().await.unwrap(), 0);
+    let kept = keeping.get_job(expires_never).await.unwrap().unwrap();
+    assert_eq!(
+        kept.state,
+        JobState::Dead,
+        "retention None = dead rows live forever"
+    );
+
+    factory.teardown().await.expect("factory teardown");
+}
+
 /// Shared clock helper for the suite's clock-adjacent rows: unix
 /// seconds now (rows never assert tight timings — the tolerance is
 /// the stamp's resolution).