---
id: fork-rederive-queue-ops
name: Fork re-derivation — queue ops on contract v1 (stamps, claim, sweep, get_job)
status: completed
depends_on: [fork-port-connection-watcher]
scope: broad
risk: high
impact: component
level: implementation
tags: [wave-2, substrate]
---

## Description

Re-derive the queue machinery in the substrate on contract v1 — the
new-code half of the fork (ADR-011's re-derivation list; quality-read
§6). This is the highest-risk task of wave 2: it is where ADR-010's
semantics depth becomes SQL, and where the upstream defect class (the
unreleased fix train's savepoint hardening, the `.ok()` error
swallows) must be *not* inherited.

**Re-derive (contract-derived names; the substrate API stays
contract-blind — primitives in, primitives out, ADR-012 §2):**

- **Enqueue + stamping** (ADR-010 §3a): job rows carry the
  `QueueOpts` stamps (`visibility_timeout_s`, `max_attempts`,
  `backoff_base_s`, `dead_letter_retention_s`) plus the `EnqueueOpts`
  resolution inputs. The *resolution rules* (delay-over-run_at,
  relative-expires, which stamps apply per call shape) are engine-layer
  work (wave 3) — the substrate's enqueue takes already-resolved
  primitive stamp values.
- **Single-statement claim** with per-row visibility from the job's
  stamps: exactly-once handout under concurrency (POC-pinned shape),
  `attempts += 1` per claim, claim ordering `priority DESC, ready-time
  ASC, enqueue order`, claimant column stamped, `claimed_at` +
  `claim_expires_at` set (ADR-019 §3's `Job` fields; ADR-021 §2).
- **Savepoint-guarded retry/fail/dead-letter** (ADR-010 §1–§4): the
  DELETE→INSERT dead-letter moves are savepoint-hardened (the #133
  defect class — a mid-flight error must strand the row in *neither*
  table); retry computes nothing itself (the engine passes the
  resolved delay in — the curve is engine-layer arithmetic per
  ADR-012 §2); budget exhaustion and `fail` move to dead with
  `last_error`/`died_at`; the pinned default strings (`"failed"`,
  `"max attempts exceeded"`) are engine-layer (contract-derived
  strings do not belong in the contract-blind substrate — pass them
  in).
- **Both-states no-stranded-rows `sweep_expired`** (ADR-010 §5): moves
  every past-`expires_at` row (pending and processing) to dead with
  `last_error='expired'`, and enforces `dead_letter_retention_s`
  deletion. Single-statement atomic per queue.
- **Dead-visible `get_job`** (ADR-010 §1): reads dead rows with
  stamps, `claimed_at`, `last_error`, `died_at` — the zombie hole and
  dead-visibility gap are the fork's motivating fixes; both land here.
- **Handle-op validity predicate support** (ADR-010 §2): the
  ack/heartbeat/retry/fail transitions carry the uniform predicate
  (row `processing` + caller's claim deadline unexpired) — false, not
  error, on refusal; the ack-vs-reclaim race resolves atomically.
- **Scheduler tick over the new enqueue** with `@every` next-boundary
  math (numeric, a few lines — cron machinery not ported): the
  `__alkstore_scheduler_tasks` storage, boundary advance + fire
  enqueue + row-locked advance, the 64-cap catch-up (ADR-009 §4). The
  leadership-lock loop pattern is ported with the lock machinery; the
  boundary math is the re-derivation.

**Schema:** the `__alkstore_*` job/dead/scheduler table family with the
stamp columns and `claimed_at` (ADR-012 §5 — the re-derivation adds
them); partial indexes matching the claim hot path, dead rows outside
it, single clock source (second-precision timestamps), per queues.md
§Namespaces.

**Tests:** the contract-property floor — no-stranded-rows, dead-visible
`get_job`, stamps immutability, claim exclusivity under concurrency,
the validity predicate's refusal boundaries, savepoint hardening
(mid-flight-error stranding test), scheduler boundary/catch-up math.
Plus the inherited queue-machinery tests adapted where applicable.

## Acceptance Criteria

- [x] All re-derived ops above present with contract-blind primitive
      APIs (no contract types, no error-taxonomy types in the
      substrate)
- [x] Savepoint hardening proven by test: a forced mid-flight error in
      the dead-letter move strands no row in either table
- [x] No `.ok()`-style error swallows (the D-class defects); all
      substrate errors propagate typed
- [x] Claim exclusivity under concurrent claims (multi-connection
      test); reclaim consumes an attempt; validity predicate refuses
      lapsed-deadline ops as values (false), not errors
- [x] `sweep_expired` moves both states; retention deletion enforced;
      no-stranded-rows property test green
- [x] Scheduler: boundary advance + fire atomic under row lock;
      64-cap catch-up; `@every` math unit-tested (s|m|h|d)
- [x] `get_job` sees dead rows with full stamp/error fields
- [x] `cargo test -p alkstore-sqlite`, clippy `-D warnings`, fmt clean

## References

- docs/architecture/decisions/010-queue-semantics-depth.md (§1–§6, §8)
- docs/architecture/decisions/009-scheduler-collapse.md §3–§4
- docs/architecture/decisions/012-forked-substrate-design.md §2, §5
- docs/research/quality-read-honker-core.md §3 (defect register), §6
- docs/research/reference-honker-machinery.md (the lineage mechanics, file/line cites)
- docs/architecture/queues.md (the semantics this realizes)

## Notes

- **Module shape**: the re-derivation is one new substrate module,
  `queue_ops.rs` (beside the ported `schema.rs` / `watcher.rs` /
  `ops.rs`), with contract-derived names throughout per ADR-012 §3 —
  the re-derived half is new code, so no lineage-name fidelity
  obligation applies. It reuses the ported `in_savepoint` machinery
  (made `pub(crate)` for the new module — the only touch to the kept
  half besides schema).
- **API shape**: stamp values bundle into a `Stamps { max_attempts,
  visibility_timeout_s, backoff_base_s, dead_letter_retention_s }`
  struct (clippy arg-count floor of 7 forced the bundling; the bundle
  is also the honest shape — ADR-010 §3a's four stamps land together).
  Scheduler registration takes `FireOpts { priority, expires_s }` +
  `Stamps`. All resolution inputs (ready time as an absolute `run_at`,
  retry delay as a literal, the default error strings
  `"max attempts exceeded"` / `"expired"`) are engine-layer — the
  substrate takes already-resolved primitives. Cron machinery absent:
  `parse_every_interval` is the whole spec grammar (ADR-009 §2), and a
  non-`@every` spec found in storage is rejected (pinned by test).
- **Claim mechanics**: single `UPDATE … WHERE id IN (WITH picked AS
  select-ordered-limit)` statement — per-row visibility via
  `claim_expires_at = unixepoch() + visibility_timeout_s` (the §3a
  bridge: the deadline computes from the row's own stamp, not a
  uniform call value). Ordering `priority DESC, run_at ASC, id ASC`
  in the picked subquery. The pre-claim dead-letter sweep for
  exhausted reclaimables is retained from the lineage (savepoint-
  guarded) with the engine-supplied exhaust string.
- **Schema deltas**: stamp columns on `__alkstore_live`
  (`visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`)
  alongside the ported `claimed_at`; the dead table gains the same
  stamps plus `claimed_at`/`expires_at` (ADR-019 §3's dead-row field
  list); the scheduler table carries the stamps too (schedule fires
  enqueue through the same stamping path); a `__alkstore_dead(queue,
  died_at)` index serves the retention sweep. All append-column
  migrations, so an existing `__alkstore_*` file migrates in place.
- **`retry` shape**: the retry branch is deliberately *not*
  savepoint-wrapped (it is a single UPDATE; there is no move), while
  the exhaustion branch is (DELETE→INSERT) — mirroring the lineage's
  post-#133 discipline, with the caller's resolved delay and the
  engine's exhaust string passed in.
- **Boundary-advance transactionality**: `scheduler_tick` keeps the
  lineage's shape — enqueue + advance in the caller's (writer)
  transaction, so crash mid-tick rolls both back; per-tick row-lock
  re-check rides SQLite's writer serialization exactly as ADR-009 §4
  pins for this engine. The 64-cap (`SCHEDULER_MAX_CATCHUP_FIRES`) and
  skip-forward (next boundary strictly after `now`) are pinned by
  tests, including the never-doubles-fire re-tick check.
- **Test floor**: the inherited queue-op suites adapted (savepoint
  rollback paths through the Rust API; the scalar-function-path
  variants are not re-created — the substrate's SQL-function surface
  for queues gets attached at wave-3 wiring, where those adapters
  reproduce mechanically if needed), plus the contract-property suite
  (41 new tests): claim exclusivity under 6 concurrent connections ×
  120 jobs (zero double-handout, integrity intact), stamps
  immutability across claim/heartbeat, lapsed-deadline refusal as
  values for all four handle ops, reclaim-eats-attempt, ack-vs-reclaim
  race resolution, both-states sweep + retention TTL, the
  no-stranded-rows property including the zombie shape, savepoint
  hardening at every dead-letter site via a forced mid-flight error
  and a blocked dead INSERT, boundary/catch-up/skip-forward
  scheduler math, `@every` s/m/h/d unit math, `get_job` dead
  visibility with the full stamp list.
- **Register**: D-12 updated to the landed state (the only pre-declared
  entry this task owned); no new port-surfaced register entries arose —
  the deltas above are all within D-12's declared scope.

## Summary

The queue half of the fork's re-derivation landed as
`alkstore-sqlite/src/substrate/queue_ops.rs` (~560 lines of owned
contract-v1 code + a 41-test property suite; 84 sqlite-crate tests
green total): contract-blind `enqueue` with per-job stamping
(`Stamps`), single-statement `claim_batch` with per-row visibility
from the job's own stamps + the retained pre-claim dead-letter sweep,
`ack`/`ack_batch`/`heartbeat`/`retry`/`fail`/`cancel` under the
uniform validity predicate (refusals as values), savepoint-guarded
dead-letter moves at every DELETE→INSERT site (forced-error and
blocked-INSERT tests pin no-strand-in-either-table), the both-states
no-stranded-rows `sweep_expired` with `dead_letter_retention_s`
enforcement, dead-visible `get_job` carrying the full ADR-019 §3 field
list (stamps, `claimed_at`, `last_error`, `died_at`), and the
scheduler surface (`register` upsert, `tick` with `@every` boundary
math, 64-cap catch-up + skip-forward, `soonest`, `unregister`) with
fire+advance atomic under the writer transaction. Schema: stamp
columns + `claimed_at`/`expires_at` on live and dead, stamps on the
schedule table, retention index on dead — all append-column
migrations. No contract or taxonomy types in the substrate; the
default error strings are call-site parameters. PROVENANCE.md D-12
recorded as landed. Verified: `cargo build`, workspace `cargo test`
(84 sqlite + 26 elsewhere), `cargo clippy --all-targets -- -D
warnings`, `cargo fmt --check` all clean; zero tokio, no
panics/unwrap/expect outside tests.