197 lines
11 KiB
Markdown
197 lines
11 KiB
Markdown
---
|
||
id: fork-rederive-queue-ops
|
||
name: Fork re-derivation — queue ops on contract v1 (stamps, claim, sweep, get_job)
|
||
status: completed
|
||
depends_on: [fork-port-connection-watcher]
|
||
scope: broad
|
||
risk: high
|
||
impact: component
|
||
level: implementation
|
||
tags: [wave-2, substrate]
|
||
---
|
||
|
||
## Description
|
||
|
||
Re-derive the queue machinery in the substrate on contract v1 — the
|
||
new-code half of the fork (ADR-011's re-derivation list; quality-read
|
||
§6). This is the highest-risk task of wave 2: it is where ADR-010's
|
||
semantics depth becomes SQL, and where the upstream defect class (the
|
||
unreleased fix train's savepoint hardening, the `.ok()` error
|
||
swallows) must be *not* inherited.
|
||
|
||
**Re-derive (contract-derived names; the substrate API stays
|
||
contract-blind — primitives in, primitives out, ADR-012 §2):**
|
||
|
||
- **Enqueue + stamping** (ADR-010 §3a): job rows carry the
|
||
`QueueOpts` stamps (`visibility_timeout_s`, `max_attempts`,
|
||
`backoff_base_s`, `dead_letter_retention_s`) plus the `EnqueueOpts`
|
||
resolution inputs. The *resolution rules* (delay-over-run_at,
|
||
relative-expires, which stamps apply per call shape) are engine-layer
|
||
work (wave 3) — the substrate's enqueue takes already-resolved
|
||
primitive stamp values.
|
||
- **Single-statement claim** with per-row visibility from the job's
|
||
stamps: exactly-once handout under concurrency (POC-pinned shape),
|
||
`attempts += 1` per claim, claim ordering `priority DESC, ready-time
|
||
ASC, enqueue order`, claimant column stamped, `claimed_at` +
|
||
`claim_expires_at` set (ADR-019 §3's `Job` fields; ADR-021 §2).
|
||
- **Savepoint-guarded retry/fail/dead-letter** (ADR-010 §1–§4): the
|
||
DELETE→INSERT dead-letter moves are savepoint-hardened (the #133
|
||
defect class — a mid-flight error must strand the row in *neither*
|
||
table); retry computes nothing itself (the engine passes the
|
||
resolved delay in — the curve is engine-layer arithmetic per
|
||
ADR-012 §2); budget exhaustion and `fail` move to dead with
|
||
`last_error`/`died_at`; the pinned default strings (`"failed"`,
|
||
`"max attempts exceeded"`) are engine-layer (contract-derived
|
||
strings do not belong in the contract-blind substrate — pass them
|
||
in).
|
||
- **Both-states no-stranded-rows `sweep_expired`** (ADR-010 §5): moves
|
||
every past-`expires_at` row (pending and processing) to dead with
|
||
`last_error='expired'`, and enforces `dead_letter_retention_s`
|
||
deletion. Single-statement atomic per queue.
|
||
- **Dead-visible `get_job`** (ADR-010 §1): reads dead rows with
|
||
stamps, `claimed_at`, `last_error`, `died_at` — the zombie hole and
|
||
dead-visibility gap are the fork's motivating fixes; both land here.
|
||
- **Handle-op validity predicate support** (ADR-010 §2): the
|
||
ack/heartbeat/retry/fail transitions carry the uniform predicate
|
||
(row `processing` + caller's claim deadline unexpired) — false, not
|
||
error, on refusal; the ack-vs-reclaim race resolves atomically.
|
||
- **Scheduler tick over the new enqueue** with `@every` next-boundary
|
||
math (numeric, a few lines — cron machinery not ported): the
|
||
`__alkstore_scheduler_tasks` storage, boundary advance + fire
|
||
enqueue + row-locked advance, the 64-cap catch-up (ADR-009 §4). The
|
||
leadership-lock loop pattern is ported with the lock machinery; the
|
||
boundary math is the re-derivation.
|
||
|
||
**Schema:** the `__alkstore_*` job/dead/scheduler table family with the
|
||
stamp columns and `claimed_at` (ADR-012 §5 — the re-derivation adds
|
||
them); partial indexes matching the claim hot path, dead rows outside
|
||
it, single clock source (second-precision timestamps), per queues.md
|
||
§Namespaces.
|
||
|
||
**Tests:** the contract-property floor — no-stranded-rows, dead-visible
|
||
`get_job`, stamps immutability, claim exclusivity under concurrency,
|
||
the validity predicate's refusal boundaries, savepoint hardening
|
||
(mid-flight-error stranding test), scheduler boundary/catch-up math.
|
||
Plus the inherited queue-machinery tests adapted where applicable.
|
||
|
||
## Acceptance Criteria
|
||
|
||
- [x] All re-derived ops above present with contract-blind primitive
|
||
APIs (no contract types, no error-taxonomy types in the
|
||
substrate)
|
||
- [x] Savepoint hardening proven by test: a forced mid-flight error in
|
||
the dead-letter move strands no row in either table
|
||
- [x] No `.ok()`-style error swallows (the D-class defects); all
|
||
substrate errors propagate typed
|
||
- [x] Claim exclusivity under concurrent claims (multi-connection
|
||
test); reclaim consumes an attempt; validity predicate refuses
|
||
lapsed-deadline ops as values (false), not errors
|
||
- [x] `sweep_expired` moves both states; retention deletion enforced;
|
||
no-stranded-rows property test green
|
||
- [x] Scheduler: boundary advance + fire atomic under row lock;
|
||
64-cap catch-up; `@every` math unit-tested (s|m|h|d)
|
||
- [x] `get_job` sees dead rows with full stamp/error fields
|
||
- [x] `cargo test -p alkstore-sqlite`, clippy `-D warnings`, fmt clean
|
||
|
||
## References
|
||
|
||
- docs/architecture/decisions/010-queue-semantics-depth.md (§1–§6, §8)
|
||
- docs/architecture/decisions/009-scheduler-collapse.md §3–§4
|
||
- docs/architecture/decisions/012-forked-substrate-design.md §2, §5
|
||
- docs/research/quality-read-honker-core.md §3 (defect register), §6
|
||
- docs/research/reference-honker-machinery.md (the lineage mechanics, file/line cites)
|
||
- docs/architecture/queues.md (the semantics this realizes)
|
||
|
||
## Notes
|
||
|
||
- **Module shape**: the re-derivation is one new substrate module,
|
||
`queue_ops.rs` (beside the ported `schema.rs` / `watcher.rs` /
|
||
`ops.rs`), with contract-derived names throughout per ADR-012 §3 —
|
||
the re-derived half is new code, so no lineage-name fidelity
|
||
obligation applies. It reuses the ported `in_savepoint` machinery
|
||
(made `pub(crate)` for the new module — the only touch to the kept
|
||
half besides schema).
|
||
- **API shape**: stamp values bundle into a `Stamps { max_attempts,
|
||
visibility_timeout_s, backoff_base_s, dead_letter_retention_s }`
|
||
struct (clippy arg-count floor of 7 forced the bundling; the bundle
|
||
is also the honest shape — ADR-010 §3a's four stamps land together).
|
||
Scheduler registration takes `FireOpts { priority, expires_s }` +
|
||
`Stamps`. All resolution inputs (ready time as an absolute `run_at`,
|
||
retry delay as a literal, the default error strings
|
||
`"max attempts exceeded"` / `"expired"`) are engine-layer — the
|
||
substrate takes already-resolved primitives. Cron machinery absent:
|
||
`parse_every_interval` is the whole spec grammar (ADR-009 §2), and a
|
||
non-`@every` spec found in storage is rejected (pinned by test).
|
||
- **Claim mechanics**: single `UPDATE … WHERE id IN (WITH picked AS
|
||
select-ordered-limit)` statement — per-row visibility via
|
||
`claim_expires_at = unixepoch() + visibility_timeout_s` (the §3a
|
||
bridge: the deadline computes from the row's own stamp, not a
|
||
uniform call value). Ordering `priority DESC, run_at ASC, id ASC`
|
||
in the picked subquery. The pre-claim dead-letter sweep for
|
||
exhausted reclaimables is retained from the lineage (savepoint-
|
||
guarded) with the engine-supplied exhaust string.
|
||
- **Schema deltas**: stamp columns on `__alkstore_live`
|
||
(`visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`)
|
||
alongside the ported `claimed_at`; the dead table gains the same
|
||
stamps plus `claimed_at`/`expires_at` (ADR-019 §3's dead-row field
|
||
list); the scheduler table carries the stamps too (schedule fires
|
||
enqueue through the same stamping path); a `__alkstore_dead(queue,
|
||
died_at)` index serves the retention sweep. All append-column
|
||
migrations, so an existing `__alkstore_*` file migrates in place.
|
||
- **`retry` shape**: the retry branch is deliberately *not*
|
||
savepoint-wrapped (it is a single UPDATE; there is no move), while
|
||
the exhaustion branch is (DELETE→INSERT) — mirroring the lineage's
|
||
post-#133 discipline, with the caller's resolved delay and the
|
||
engine's exhaust string passed in.
|
||
- **Boundary-advance transactionality**: `scheduler_tick` keeps the
|
||
lineage's shape — enqueue + advance in the caller's (writer)
|
||
transaction, so crash mid-tick rolls both back; per-tick row-lock
|
||
re-check rides SQLite's writer serialization exactly as ADR-009 §4
|
||
pins for this engine. The 64-cap (`SCHEDULER_MAX_CATCHUP_FIRES`) and
|
||
skip-forward (next boundary strictly after `now`) are pinned by
|
||
tests, including the never-doubles-fire re-tick check.
|
||
- **Test floor**: the inherited queue-op suites adapted (savepoint
|
||
rollback paths through the Rust API; the scalar-function-path
|
||
variants are not re-created — the substrate's SQL-function surface
|
||
for queues gets attached at wave-3 wiring, where those adapters
|
||
reproduce mechanically if needed), plus the contract-property suite
|
||
(41 new tests): claim exclusivity under 6 concurrent connections ×
|
||
120 jobs (zero double-handout, integrity intact), stamps
|
||
immutability across claim/heartbeat, lapsed-deadline refusal as
|
||
values for all four handle ops, reclaim-eats-attempt, ack-vs-reclaim
|
||
race resolution, both-states sweep + retention TTL, the
|
||
no-stranded-rows property including the zombie shape, savepoint
|
||
hardening at every dead-letter site via a forced mid-flight error
|
||
and a blocked dead INSERT, boundary/catch-up/skip-forward
|
||
scheduler math, `@every` s/m/h/d unit math, `get_job` dead
|
||
visibility with the full stamp list.
|
||
- **Register**: D-12 updated to the landed state (the only pre-declared
|
||
entry this task owned); no new port-surfaced register entries arose —
|
||
the deltas above are all within D-12's declared scope.
|
||
|
||
## Summary
|
||
|
||
The queue half of the fork's re-derivation landed as
|
||
`alkstore-sqlite/src/substrate/queue_ops.rs` (~560 lines of owned
|
||
contract-v1 code + a 41-test property suite; 84 sqlite-crate tests
|
||
green total): contract-blind `enqueue` with per-job stamping
|
||
(`Stamps`), single-statement `claim_batch` with per-row visibility
|
||
from the job's own stamps + the retained pre-claim dead-letter sweep,
|
||
`ack`/`ack_batch`/`heartbeat`/`retry`/`fail`/`cancel` under the
|
||
uniform validity predicate (refusals as values), savepoint-guarded
|
||
dead-letter moves at every DELETE→INSERT site (forced-error and
|
||
blocked-INSERT tests pin no-strand-in-either-table), the both-states
|
||
no-stranded-rows `sweep_expired` with `dead_letter_retention_s`
|
||
enforcement, dead-visible `get_job` carrying the full ADR-019 §3 field
|
||
list (stamps, `claimed_at`, `last_error`, `died_at`), and the
|
||
scheduler surface (`register` upsert, `tick` with `@every` boundary
|
||
math, 64-cap catch-up + skip-forward, `soonest`, `unregister`) with
|
||
fire+advance atomic under the writer transaction. Schema: stamp
|
||
columns + `claimed_at`/`expires_at` on live and dead, stamps on the
|
||
schedule table, retention index on dead — all append-column
|
||
migrations. No contract or taxonomy types in the substrate; the
|
||
default error strings are call-site parameters. PROVENANCE.md D-12
|
||
recorded as landed. Verified: `cargo build`, workspace `cargo test`
|
||
(84 sqlite + 26 elsewhere), `cargo clippy --all-targets -- -D
|
||
warnings`, `cargo fmt --check` all clean; zero tokio, no
|
||
panics/unwrap/expect outside tests. |