diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 3439fe2..56fbd86 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -37,7 +37,7 @@ pending architecture review and OQ resolution. |---|---|---| | [001](decisions/001-crate-split.md) | Reactive-core crate + per-engine crates | Accepted | | [002](decisions/002-feature-scope.md) | Feature scope — inventory-confirmed surface | Accepted | -| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core, bridged seam | Accepted | +| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core lineage, bridged seam (ownership: ADR-011) | Accepted | | [004](decisions/004-postgres-driver.md) | Postgres engine — tokio-postgres + deadpool, hand-rolled LISTEN | Accepted | | [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted | | [006](decisions/006-wake-and-delivery-contract.md) | Wake contract — opaque wake + re-read; notify-vs-streams split | Accepted | @@ -45,6 +45,8 @@ pending architecture review and OQ resolution. | [008](decisions/008-contract-v1-pinning.md) | Contract v1 surface pinning — surface partition, `TxHandle` shape, wake type, reserved strings, error taxonomy | Accepted | | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse — queues + `schedule()`/`run_schedules`, `@every`-only v1, boundary guarantee row | Accepted | | [010](decisions/010-queue-semantics-depth.md) | Queue semantics depth — visibility/renewal, backoff curve, dead-letter, no-stranded-rows sweep, schema layout | Accepted | +| [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted | +| [012](decisions/012-forked-substrate-design.md) | Forked substrate design — contract-blind boundary, fidelity posture, port deltas | Accepted | ## Open Questions @@ -59,6 +61,9 @@ in suggested resolution order (OQ-04, OQ-09, OQ-05, OQ-06 resolved): honker-core's lineage, queue ops re-derived on contract v1. - **OQ-08** (medium): capability-surface shape. - **OQ-10** (medium): contract versioning across engine crates. +- **OQ-11** (medium): forked-substrate follow-through (scaffold-time + decisions — [ADR-012](decisions/012-forked-substrate-design.md)'s + residue). Resolved (kept with resolutions): OQ-01 (feature scope), OQ-02 (crate split), OQ-03 (drivers), OQ-07 (extension surface cut), diff --git a/docs/architecture/core-contract.md b/docs/architecture/core-contract.md index be71120..42f2c64 100644 --- a/docs/architecture/core-contract.md +++ b/docs/architecture/core-contract.md @@ -187,8 +187,8 @@ TTL-bounded coordination locks, transactional-friendly: no-work is a value, not an error); `renew` and `release` on the lock handle. Release on explicit unlock or TTL expiry. Re-acquirable after expiry (POC-pinned on Postgres; on SQLite it - rests on honker's machinery — the SQLite-side pin is in the - verification backlog below). + rests on the forked substrate's lock machinery — the SQLite-side pin + is in the verification backlog below). - **Guarantee row** ([ADR-008](decisions/008-contract-v1-pinning.md) §7): mutual exclusion bounded by TTL + renewal — after TTL expiry exclusion lapses *silently* (no revocation event); holders must @@ -285,8 +285,11 @@ added by [ADR-009](decisions/009-scheduler-collapse.md) §1): - Engine-derived names in consumer namespaces carry the reserved prefix (the outbox's backing queue is derived under the prefix — honker's `_outbox:{name}` scheme is not inherited verbatim). -- SQLite: honker's `_honker_*` internal table family is storage- - internal (not consumer namespace); Postgres: engine tables are +- SQLite: the substrate's `__alkstore_*` internal table family is + storage-internal (not consumer namespace) — re-owned from upstream's + `_honker_*` by the fork ([ADR-011](decisions/011-sqlite-substrate-fork.md), + designed per [ADR-012](decisions/012-forked-substrate-design.md)). + Postgres: engine tables are schema-scoped — one engine-owned schema ([ADR-010](decisions/010-queue-semantics-depth.md) §8) — the *channel* namespace is this section's. @@ -297,9 +300,11 @@ Contract properties POC-pinned on one engine only (or sketched rather than surface-verified) — the contract test suite must pin both engines before the engine specs are called `stable`: -- **Lock TTL/expiry re-acquisition on SQLite** — pinned on Postgres - (pg POC's lock probe); on SQLite it rests on honker's machinery - unverified. +- **Named-lock TTL/expiry re-acquisition on SQLite** — pinned on + Postgres (pg POC's lock probe); on SQLite it rests on the forked + substrate's lock machinery (its `lock_renew` verified in the quality + read; the re-acquire-does-not-refresh-TTL behavior is upstream's, + inherited deliberately) — pin it in the contract suite. - **Wake semantics under `WakeReceiver` shapes** — POCs verified wake delivery/coalescing through their own probe types; the pinned `Wake { channel }` / recv forms (§3 of @@ -307,21 +312,29 @@ before the engine specs are called `stable`: suite's own property tests on both engines. - **`save_offset_tx` exactly-once-within-a-business-tx shape** — both POCs verified it through their *sketch* implementations (SQLite's - offset-save was a plain SQL upsert, not the full honker surface; + offset-save was a plain SQL upsert, not the forked substrate's full + surface; the pg side likewise through its probe), so the property must be re-pinned against the real engines' `save_offset_tx` in the contract suite. - **Concurrent `try_lock` loser/error behavior** on SQLite (the pg - side returns cleanly; honker's busy-path under lock contention is - the thing to pin). + side returns cleanly; the substrate's busy-path under lock + contention is the thing to pin). - **Scheduler + depth properties on Postgres** — the SQLite side's - tick/leader/catch-up machinery rides honker's test-pinned - implementation; the pg engine's re-derived tick (boundary advance, + tick/leader/catch-up machinery inherits the forked substrate's + test-pinned implementation; the pg engine's re-derived tick (boundary + advance, 64-cap skip-forward, leadership-loss discipline) and the [ADR-010](decisions/010-queue-semantics-depth.md) depth properties (visibility reclaim consuming attempts, dead-letter moves, the no-stranded-rows sweep) pin in the contract suite at implementation. +- **Backoff-curve and stamp-resolution equivalence across engines** — + the equal-jitter curve ([ADR-010](decisions/010-queue-semantics-depth.md) + §3) and the opts-stamping resolution (§3a) are computed engine-side + per [ADR-012](decisions/012-forked-substrate-design.md) §2's + contract-blind boundary; the contract suite must pin both engines' + arithmetic to identical outputs. ## Design Decisions @@ -334,6 +347,8 @@ before the engine specs are called `stable`: | [008](decisions/008-contract-v1-pinning.md) | Contract v1 pinning | surface partition, `TxHandle` trait shape, `Wake` type, reserved strings, error taxonomy, config split, locks guarantee row | | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse (post-v1 extension) | queues + `schedule()`/`run_schedules`, `@every`-only v1, boundary guarantee row | | [010](decisions/010-queue-semantics-depth.md) | Queue depth (post-v1 extension) | visibility/renewal, opts stamping, backoff curve, dead-letter, sweep, layout | +| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite substrate owned (`__alkstore_*` naming); queue ops re-derived on contract v1 | +| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; engine-side formula arithmetic pinned equivalent by the contract suite | ## Open Questions diff --git a/docs/architecture/decisions/001-crate-split.md b/docs/architecture/decisions/001-crate-split.md index 161167e..904307b 100644 --- a/docs/architecture/decisions/001-crate-split.md +++ b/docs/architecture/decisions/001-crate-split.md @@ -13,7 +13,8 @@ theirs: - The **SQLite** engine rides honker's existing, battle-tested machinery (published `honker-core` — a port-and-adapt of a sync library to the - family's async-facing-trait posture). + family's async-facing-trait posture). *(Ownership since resolved: + the substrate is forked into owned code — [ADR-011](011-sqlite-substrate-fork.md).)* - The **Postgres** engine is the build-heavy side: LISTEN/NOTIFY wiring and the pg-boss-family queue schema are built by this crate. @@ -37,7 +38,9 @@ The project ships as a **family of crates**: documentation. No driver dependencies. Compile-lean by construction: the base crate has no engine machinery to keep out. 2. **`alkstore-sqlite`** — the SQLite engine implementing the core - surface. Single driver: rusqlite + honker-core ([ADR-003]). + surface. Single driver: rusqlite + the forked `alkstore-substrate` + lineage ([ADR-003]; ownership per + [ADR-011](011-sqlite-substrate-fork.md)). 3. **`alkstore-postgres`** — the Postgres engine implementing the core surface. Single driver: tokio-postgres + deadpool-postgres ([ADR-004]). @@ -85,4 +88,6 @@ crate. A consumer binary links at most one driver. link-collision constraint motivates this split. - [ADR-004](004-postgres-driver.md) — the Postgres driver choice. - OQ-10 (`docs/architecture/open-questions.md`) — trait-versioning - duties created here. \ No newline at end of file + duties created here. +[ADR-003]: 003-sqlite-driver.md +[ADR-004]: 004-postgres-driver.md diff --git a/docs/architecture/decisions/003-sqlite-driver.md b/docs/architecture/decisions/003-sqlite-driver.md index 6350d06..7ad7a01 100644 --- a/docs/architecture/decisions/003-sqlite-driver.md +++ b/docs/architecture/decisions/003-sqlite-driver.md @@ -39,9 +39,15 @@ the comparison: The SQLite engine uses **posture 1**: published `honker-core = 0.5` as a library dependency over the crate's own rusqlite connections -(`bundled-sqlite` for hermetic builds). +(`bundled-sqlite` for hermetic builds). *(Ownership superseded +2026-10-05 by [ADR-011](011-sqlite-substrate-fork.md): the substrate +is the forked `alkstore-substrate` lineage crate, not the published +dependency — the driver, seam, and watcher architecture below are +unchanged.)* -- **Driver**: rusqlite, riding honker-core's pinned version. +- **Driver**: rusqlite, riding honker-core's pinned version. *(The + pin is ours to move deliberately since the fork — [ADR-011], per + [ADR-005]'s annotated rusqlite row.)* - **Async seam**: bridged. The writer slot (`Writer`) + reader pool (`Readers`) + every engine call in `spawn_blocking`. A dedicated std-thread bridge (one thread owning the writer conn, @@ -76,8 +82,10 @@ comparable reuse. - Static linkage: "open a path, get a store" needs no runtime artifacts. - The engine's job shrinks to wiring + the SQL surfaces honker doesn't - cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's - machinery directly) + the async bridge. + cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's + machinery directly) + the async bridge. *(Ownership since resolved: + the substrate is forked — [ADR-011](011-sqlite-substrate-fork.md) — + and the queue ops re-derived on contract v1 within it.)* **Negative** @@ -103,4 +111,7 @@ comparable reuse. - [ADR-007](007-transactional-seam.md) — the tx-handle shape both engines implement. - [engine-sqlite.md](../engine-sqlite.md) — the engine spec this - decision defines. \ No newline at end of file + decision defines. +[ADR-001]: 001-crate-split.md +[ADR-005]: 005-dependency-ownership.md +[ADR-007]: 007-transactional-seam.md diff --git a/docs/architecture/decisions/004-postgres-driver.md b/docs/architecture/decisions/004-postgres-driver.md index e57f9ff..3e4ad91 100644 --- a/docs/architecture/decisions/004-postgres-driver.md +++ b/docs/architecture/decisions/004-postgres-driver.md @@ -101,4 +101,8 @@ Structural constraints the engine owns (all test-pinned in POC #2): decision applies per-subsystem. - [ADR-006](006-wake-and-delivery-contract.md) — the wake contract the forwarder implements on this engine. -- [engine-postgres.md](../engine-postgres.md) — the engine spec. \ No newline at end of file +- [engine-postgres.md](../engine-postgres.md) — the engine spec. +[ADR-003]: 003-sqlite-driver.md +[ADR-005]: 005-dependency-ownership.md +[ADR-006]: 006-wake-and-delivery-contract.md +[queues.md]: ../queues.md diff --git a/docs/architecture/decisions/005-dependency-ownership.md b/docs/architecture/decisions/005-dependency-ownership.md index eb47fe7..ddc5d01 100644 --- a/docs/architecture/decisions/005-dependency-ownership.md +++ b/docs/architecture/decisions/005-dependency-ownership.md @@ -28,7 +28,7 @@ when a named trigger fires. Per subsystem: | Subsystem | Posture | Notes | |---|---|---| | honker-core (SQLite engine machinery) | published library, `honker-core = 0.5` — **trigger fired 2026-10-05**: forked per [ADR-011] | Fork trigger: the Phase 1 quality read of the watcher/transactional core finds a defect, OR a needed change upstream won't take. OQ-06 tracked the read; the read fired the trigger (published 0.5.0 carries the unreleased-fix-train defect class + ADR-010's queue depth requires engine-owned queue SQL in any posture). | -| rusqlite | published, riding honker-core's pin | Pin movement is honker-core's; we ride it ([ADR-003]). | +| rusqlite | published, ~~riding honker-core's pin~~ **pin ours after the fork** | Post-fork ([ADR-011]), the rusqlite generation is moved deliberately in our own substrate crate, not by upstream releases ([ADR-003]'s annotated negative consequence; [ADR-012] ports the substrate onto the family standard). | | tokio-postgres + deadpool-postgres | published library, as-is | Clean, zero conflicts, actively maintained (POC #2). | | postgres-notify | **not adopted** (derive-not-adopt) | Lazy reconnect, connect_script skipped at initial connect, unquoted-identifier LISTENs, single maintainer. Hand-rolled forwarder instead ([ADR-004]). Fallback if upstream improves materially. | | pg-boss queue machinery (pgboss-rs / node pg-boss) | **re-derived** on our driver; schema family as *design reference* only | pgboss-rs brings sqlx (second driver per binary) and has no LISTEN/NOTIFY (verified); the push half is ours either way, so queue-machinery reuse value is the honest comparison point and it loses on that arithmetic. Semantics depth: [queues.md], OQ-05. | @@ -73,4 +73,10 @@ to this posture. - [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) — the per-engine driver decisions applying this posture. - OQ-06 (`docs/architecture/open-questions.md`) — the quality-read - tracker. \ No newline at end of file + tracker (resolved: [ADR-011], designed per [ADR-012]). +[ADR-003]: 003-sqlite-driver.md +[ADR-004]: 004-postgres-driver.md +[ADR-006]: 006-wake-and-delivery-contract.md +[ADR-011]: 011-sqlite-substrate-fork.md +[ADR-012]: 012-forked-substrate-design.md +[queues.md]: ../queues.md diff --git a/docs/architecture/decisions/006-wake-and-delivery-contract.md b/docs/architecture/decisions/006-wake-and-delivery-contract.md index 0f7dba9..0189b2b 100644 --- a/docs/architecture/decisions/006-wake-and-delivery-contract.md +++ b/docs/architecture/decisions/006-wake-and-delivery-contract.md @@ -126,4 +126,7 @@ consumer-inventory-row-gated extension — not assumed now. - [ADR-007](007-transactional-seam.md) — commit-atomicity's mechanism (the `*_tx` seam). - [core-contract.md](../core-contract.md) — the spec carrying this - contract's full surface. \ No newline at end of file + contract's full surface. +[ADR-004]: 004-postgres-driver.md +[deployment.md]: ../deployment.md +[engine-postgres.md]: ../engine-postgres.md diff --git a/docs/architecture/decisions/007-transactional-seam.md b/docs/architecture/decisions/007-transactional-seam.md index d4832fc..0206b45 100644 --- a/docs/architecture/decisions/007-transactional-seam.md +++ b/docs/architecture/decisions/007-transactional-seam.md @@ -101,4 +101,6 @@ contract-pinning input rather than open risk: - [ADR-006](006-wake-and-delivery-contract.md) — commit-atomicity is the `*_tx` property this seam carries into the notify contract. - [core-contract.md](../core-contract.md) — the spec whose transaction - section this ADR defines. \ No newline at end of file + section this ADR defines. +[ADR-003]: 003-sqlite-driver.md +[ADR-004]: 004-postgres-driver.md diff --git a/docs/architecture/decisions/009-scheduler-collapse.md b/docs/architecture/decisions/009-scheduler-collapse.md index 0f023f9..fcc68a4 100644 --- a/docs/architecture/decisions/009-scheduler-collapse.md +++ b/docs/architecture/decisions/009-scheduler-collapse.md @@ -24,7 +24,8 @@ The evidence for the decision: consumer-inventory.md](../../research/consumer-inventory.md) — needs "run the sweep every N ticks with leader election across a fleet." An interval, not a schedule object. -- **alkfs (orphan reaping)** — [recalled: OQ-FS-07] — a periodic +- **alkfs (orphan reaping)** — [recalled: alkfs OQ-FS-07, + `/workspace/@alkdev/alkfs/docs/research/phase-0.md`] — a periodic orphan-cleanup pass. An interval. - No consumer document (pinned, documented, or operator-authority) names any of: wall-clock cron expressions, per-schedule timezones, @@ -166,10 +167,13 @@ resolving the transfer from [ADR-008](008-contract-v1-pinning.md) §7: ### 5. Schedule storage is engine-internal -Schedule rows live in engine-owned storage (SQLite: honker's -`_honker_scheduler_tasks` — reused as-is, its `cron_expr` column -carrying `@every` specs; Postgres: the engine-owned schema per -[ADR-010](010-queue-semantics-depth.md) §8). Not consumer namespace; +Schedule rows live in engine-owned storage (SQLite: the forked +substrate's `__alkstore_scheduler_tasks` table per +[ADR-011](011-sqlite-substrate-fork.md) — honker's scheduler-task +table re-owned and renamed, carrying `@every` specs in its spec +column, the cron machinery not ported; Postgres: the engine-owned +schema per [ADR-010](010-queue-semantics-depth.md) §8). Not consumer +namespace; schedule *names* take the same validation as queue names (§1 — non-empty, reserved-prefix rejected). No registry validation against queue names — enqueuing into a queue nobody has @@ -240,5 +244,8 @@ respawning; distinct from clean stop, §1). surface (`packages/honker-rs/src/lib.rs`, `honker-core/src/honker_ops.rs`), `@every` grammar and local-TZ brittleness (`honker-core/src/cron.rs`), tick/leader mechanics. +- [ADR-011](011-sqlite-substrate-fork.md) — the SQLite-side schedule + storage this §5 describes is the forked substrate's; cron machinery + not ported. - [core-contract.md](../core-contract.md) — the spec carrying this surface. \ No newline at end of file diff --git a/docs/architecture/decisions/010-queue-semantics-depth.md b/docs/architecture/decisions/010-queue-semantics-depth.md index 8556d4a..cdf10fc 100644 --- a/docs/architecture/decisions/010-queue-semantics-depth.md +++ b/docs/architecture/decisions/010-queue-semantics-depth.md @@ -135,9 +135,12 @@ no-renewal/expire-sweep model is not inherited): in v1. - Explicit `fail(err)` = immediate dead-letter (honker parity; it is the "stop retrying this" operator). `retry` at exhausted budget = - dead-letter. Exhaustion via reclaim = dead-letter (the pre-claim - sweep of already-exhausted reclaimable rows is an engine-side - laziness optimization — SQLite rides honker's; pg re-derives). + dead-letter. Exhaustion via reclaim = dead-letter. *(The pre-claim + sweep of already-exhausted reclaimable rows — a laziness optimization + in both upstream lineages — is engine-side and optional; SQLite rides + the fork's re-derivation + ([ADR-011](011-sqlite-substrate-fork.md)), pg re-derives it + ([engine-postgres.md](../engine-postgres.md)).)* ### 3a. QueueOpts resolution: stamped at enqueue, per job @@ -166,9 +169,14 @@ need a defined attachment point, and the contract pins one: - SQLite realization note: honker's `claim_batch` takes one uniform `timeout_s` per call, while per-job stamps make deadlines row-local — bridging that gap (post-claim re-stamp, per-row claiming, or another - shape) is implementation work over honker's function surface and - rides the same OQ-06 assessment as §5's zombie fix. The *contract* - (deadline from the job's stamp) is engine-independent either way. + shape) was implementation work over honker's function surface and + rode the same OQ-06 assessment as §5's zombie fix. *(Resolved + 2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md)); + the claim statement is re-derived with per-row visibility from the + stamps, in owned + contract-blind substrate code ([ADR-012](012-forked-substrate-design.md) + §2).) The *contract* (deadline from the job's stamp) is + engine-independent either way. ### 4. Dead-letter: move, retention = never-expire by default, no redrive API @@ -209,13 +217,13 @@ maintenance entry point): property: an expired *processing* row whose worker died is unreachable by claim (predicate requires future expiry), by pre-claim dead-lettering (same), and by `sweep_expired` (pending - only) — the zombie hole, a real defect found in the reference read. - This ADR fixes it at the contract level; the Postgres engine - enforces it directly; the SQLite engine's realization over honker's - machinery is implementation work **contingent on OQ-06** (a - complement over honker's function surface, or the fork fires and - the fix lands in owned code — the concrete fork candidate the - quality read should weigh). + only) — the zombie hole, a real defect found in the reference read + (D-6 in the quality read's register). This ADR fixes it at the + contract level; the Postgres engine + enforces it directly; the SQLite engine's realization was + implementation work **contingent on OQ-06** — *(resolved + 2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md)) + and the both-states sweep lands in owned code.)* - When `dead_letter_retention_s` is set, the same sweep also deletes dead rows past their retention (the only sweeper dead rows ever have — nothing runs without a caller). @@ -246,9 +254,14 @@ maintenance entry point): **out of the contract** (ADR-008 §8's disposition, resolved here by disposition): the notifications table is the SQLite wake mechanism's *transport* detail — consumers interact with wakes, not - rows. Its hygiene is engine-internal (capped at attach, engine-side - maintenance cadence, engine opts) — the fix for honker's unbounded - `_honker_notifications` growth must not become a consumer's chore. + rows. Its hygiene is engine-internal: **an at-attach pruning cap + (engine opts)** — the mechanism the fork scope realizes + ([ADR-011](011-sqlite-substrate-fork.md)); no cadence, ambient or + engine-side, contradicts this section's no-ambient-sweeper bullet. + The fix for upstream's unbounded notifications growth must not become + a consumer's chore. *(Annotated 2026-10-05: the earlier "engine-side + maintenance cadence" wording was wrong — nothing in the fork scope + realizes a cadence; the at-attach cap is the pin.)* ### 7. Result storage: cut-flag stands @@ -277,15 +290,18 @@ consumer-inventory row. shape and keeps `queue(name)` a name, not a DDL operation — queue creation is not registry-gated, per [ADR-009](009-scheduler-collapse.md) §5). Per-queue config storage - is likewise unnecessary (opts stamp onto job rows, §3a). + is likewise stamped per §3a. - **SQLite**: honker's `_honker_*` table family in the caller's database file — storage-internal per - [ADR-008](008-contract-v1-pinning.md) §4; no new tables minted by - this ADR (job-stamped opts ride honker's existing columns and the - contract's stated resolution rule, no schema change needed — this - is exactly the seam the quality read (OQ-06) assesses). The - family's exact fate rides that read: a fork re-owns the names, - nothing consumer-visible changes. + [ADR-008](008-contract-v1-pinning.md) §4. *(Annotated + 2026-10-05: the expectation that job-stamped opts would ride honker's + existing columns with no schema change was falsified by the quality + read — no stamp columns exist in the upstream schema (D-7). The + family's fate is resolved: the fork ([ADR-011](011-sqlite-substrate-fork.md)) + re-owns the whole family as `__alkstore_*`, and the stamp columns + + `claimed_at` are added by the re-derivation on contract v1 + ([ADR-012](012-forked-substrate-design.md) §5). Nothing + consumer-visible changes.)* ### 9. Error taxonomy: no delta @@ -322,9 +338,9 @@ delta this track produces. consumers needing audit trails build them on the tx seam. - Honker's zombie fix, dead-row `get_job` visibility, and per-job visibility stamps over the uniform-claim-timeout function surface - require either an over-machinery complement or a fork on the SQLite - side — the fork calculus gains concrete candidates to weigh - (OQ-06). + required either an over-machinery complement or a fork on the SQLite + side — resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md), + designed in [ADR-012](012-forked-substrate-design.md)). - The backoff curve's 1-hour cap and jitter formula are pinned constants — a consumer needing a different policy uses `retry(err, Some(d))` per attempt (correct, but manual). @@ -348,6 +364,8 @@ delta this track produces. extends, §5's rule §9 applies, §8's dispositions §6 resolves. - [ADR-009](009-scheduler-collapse.md) — the collapse machinery §6's recipe composes with. -- OQ-06 — the SQLite-side fork candidates (§5, §3a realization note). +- OQ-06 — the SQLite-side fork candidates (§5, §3a realization note) — + resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md), + designed in [ADR-012](012-forked-substrate-design.md)). - [queues.md](../queues.md), [core-contract.md](../core-contract.md), engine specs — the specs carrying this depth. \ No newline at end of file diff --git a/docs/architecture/decisions/011-sqlite-substrate-fork.md b/docs/architecture/decisions/011-sqlite-substrate-fork.md index f73ec33..0ab2d4d 100644 --- a/docs/architecture/decisions/011-sqlite-substrate-fork.md +++ b/docs/architecture/decisions/011-sqlite-substrate-fork.md @@ -54,17 +54,22 @@ Three facts drive this decision: code: fork honker-core at the reference revision (`/workspace/honker` @ `f4e53c6`, MIT OR Apache-2.0, provenance recorded per AGENTS.md §3) into the alkstore family, ported to the -family standard (tokio-facing consumers, no comments discipline, panics -out of library code), re-derived on contract v1 where ADR-010 pinned -semantics honker's functions don't provide. +family standard (the discipline deltas — no comments discipline, +panics out of library code — with tokio-facing consumers at the engine +seam above; per [ADR-012](012-forked-substrate-design.md) §4 the +substrate itself stays sync), re-derived on contract v1 where ADR-010 +pinned semantics honker's functions don't provide. Per-scope (the full register with keeps/re-derivations/drops is `docs/research/quality-read-honker-core.md` §6): - **Inherited near-verbatim:** the PRAGMA/WAL open posture, `Writer`, `Readers`, the polling watcher + `SharedUpdateWatcher` + - `WatcherDeathGuard` + dead-man's switch (with two port fixes the read - named: reconnect backoff, fallible watcher spawn), the + `WatcherDeathGuard` + dead-man's switch (with three port deltas the + read named / ADR-012 decided: reconnect backoff, fallible watcher + spawn, and the dead-man's-switch panic replaced by a deliberate + watcher-fatal death — [ADR-012](012-forked-substrate-design.md) §4), + the `in_savepoint` mutation-discipline machinery, arg-coercion helpers, the notify scalar + notifications table (renamed, with engine- internal hygiene per ADR-010 §6), streams, locks. @@ -126,4 +131,7 @@ crate (provenance + license recorded), not re-published. - [ADR-001](001-crate-split.md) — packaging (engine crate's single-driver substrate). - [engine-sqlite.md](../engine-sqlite.md) — the living spec, updated - to the forked substrate. \ No newline at end of file + to the forked substrate. +- [ADR-012](012-forked-substrate-design.md) — the fork's design + decisions (crate identity, contract-blind boundary, fidelity + posture, port deltas, bootstrap machinery, upstream tracking). \ No newline at end of file diff --git a/docs/architecture/decisions/012-forked-substrate-design.md b/docs/architecture/decisions/012-forked-substrate-design.md new file mode 100644 index 0000000..43dcb47 --- /dev/null +++ b/docs/architecture/decisions/012-forked-substrate-design.md @@ -0,0 +1,230 @@ +# ADR-012: Forked substrate design — contract-blind boundary, fidelity posture, port deltas + +## Status + +Accepted (2026-10-05, Phase 1 — the design decisions ADR-011's fork +requires; the follow-through of OQ-06's resolution) + +## Context + +[ADR-011] decided *that* the fork happens and its scope (keep / +re-derive / drop). Deciding it surfaced structural questions the fork +record deliberately did not settle — where the forked crate lives and +what it is called, which layer carries contract-pinned semantics, how +upstream diffs stay cheap, which of the read's port notes fire, and how +a fork with no upstream dependency to update tracks its lineage. None +of these were blockers for the fork trigger, but all of them shape the +engine's dependency graph and the fork's maintenance cost — they are +architecture, and they are decidable now, from decided material: + +1. **The dependency direction is fixed** ([ADR-001]): engines depend on + core, never the reverse. The substrate sits *below* the engine. +2. **The substrate's Rust API is engine-internal.** The contract is the + trait the constructor returns ([ADR-008] §6); no substrate type or + name reaches a consumer signature. +3. **Contract semantics are pinned engine-generically** — the + equal-jitter curve ([ADR-010] §3), the opts-stamping resolution rule + (§3a), schedule boundary math (ADR-009 §4) — and must compute + identically on both engines. One engine carrying its copy inside a + vendored lineage crate splits each pinned formula across codebases + with different ownership characters. +4. **The family bans panics in library code**; upstream's dead-man's + switch panics the watcher thread on db-file replacement + (`lib.rs:919` at the reference revision). Verified in source: the + panic is *observed* only through `WatcherDeathGuard` — the guard's + `Drop` fires on thread exit or panic and closes every subscriber + (the contract-pinned failure surface). The observable semantics do + not depend on the panic. +5. **No alkstore databases exist.** The crate is unreleased; the + `_honker_*` → `__alkstore_*` rename therefore has no migration + surface and creates one only if we invent it. + +## Decision + +### 1. Crate identity and location + +The forked substrate is **`alkstore-substrate`** — a vendored +workspace-family crate in this repository, a path dependency of +`alkstore-sqlite`, not published ([ADR-011]'s packaging, now named). +At fork-scaffold time the crate carries: the dual license files +upstream ships (MIT OR Apache-2.0), and a **provenance register** +(`provenance.md` at the crate root) recording upstream identity — +checkout path, commit `f4e53c6`, license — and the delta list below +(AGENTS.md §3's recording duty). + +### 2. The substrate is contract-blind storage machinery + +**The substrate depends on nothing of ours** — not on `alkstore` (the +core crate), on the contract types, or on the error taxonomy. Its API +takes and returns primitives: connections, names, stamp values +(`visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`, +`max_attempts`, `priority`, timestamps), payload strings. The **engine +crate layer** resolves contract semantics into primitive arguments: +`EnqueueOpts`-over-`QueueOpts` resolution, the equal-jitter curve's +arithmetic, the schedule fire's boundary math, and the mapping of +substrate outcomes into the contract taxonomy ([ADR-008] §5 / ADR-010 +§9's rules). + +Why the boundary sits there, not inside the substrate: + +- **Dependency direction**: a vendored third-party-lineage crate must + not acquire a dependency on our contract crate (and the mirror — + core depending on the substrate — inverts [ADR-001] outright). +- **One normative owner per pinned formula — the contract text, not a + substrate copy**: the curve, the stamps-resolution rule, and boundary + math are contract-pinned to compute identically on both engines + ([ADR-010] §3). The contract is the formula's single normative + owner; each engine owns one *implementation* of it, pinned to + identical outputs by the contract suite (the verification backlog in + [core-contract.md](../core-contract.md)). A substrate copy would add + a second normative-adjacent definition in a codebase whose upstream + does not know the contract exists. +- **Diff reviewability**: the substrate stays as close to its lineage + as the port allows; contract semantics are exactly the re-derivation + surface ([ADR-011]), and they are new code with contract-derived + names either way. + +Dependency graph (downward only): + +```text + consumers + │ + alkstore (core: traits, types, errors — no drivers) + ────────▲──────── + alkstore-sqlite alkstore-postgres + │ (no substrate) + alkstore-substrate (vendored lineage; path dep; unpublished) +``` + +### 3. API fidelity posture: keep the kept half's names + +Where the fork scope *keeps* upstream machinery ([ADR-011]'s inherited +list), the fork keeps upstream's module structure and internal function +names — including names carrying lineage tokens (`Writer`, `Readers`, +`run_poll_loop`, the `in_savepoint` machinery, the lock/stream/notify +functions). Renames are confined to: crate identity, the table family +(`__alkstore_*`, [ADR-011]), and the hygiene deltas [ADR-011] names. +The **re-derived** half ([ADR-011]'s list) is new code; its functions +take contract-derived names naturally. + +Rationale: the fork's ongoing value includes cheap cherry-picks of +future upstream fixes — the calculus ADR-011 inherited from ADR-005 +assumes the inherited half still *tracks* its lineage. Mechanical token +renames turn every future cherry-pick noisy; internal oddness survives +a rename only to be paid for forever. The names are engine-internal and +bounded by the fork's ownership. + +### 4. Port deltas — the quality read's notes, decided + +- **W-1 (reconnect backoff) — applied.** The watcher reconnect loop + gains bounded backoff; a vanished db file no longer produces ~1000 + open attempts per second. +- **W-2 (watcher spawn panics) — applied.** Watcher spawn becomes + fallible; the engine surfaces the failure at open time (a store whose + watcher could not start is not a store — open fails with the + taxonomy's `Database`, detail in the source chain). +- **Dead-man's switch (no-panics tension) — panic replaced by a + deliberate watcher-fatal death.** On db-file identity change the + watcher logs the precise diagnostic and exits through the ordinary + death path. `WatcherDeathGuard` closes every subscriber exactly as it + does today for thread panic or exit — the contract-pinned surface + ("consumers see the close, never a silent hang") is unchanged; the + unwind is what goes away. +- **W-4 (watcher connection opens RW) — keep RW (deliberate + inheritance).** The read found no proof a RO watcher is safe on + busy non-WAL paths; our posture is WAL-always anyway. Revisit only if + a read-only/mode knob ever appears as an engine option. +- **W-3 (`data_version` u32 wrap) — not actionable**, recorded in the + fork's notes. +- **The substrate stays sync.** ADR-011's "ported to the family + standard" means the discipline deltas (no comments, panics out of + library code per the deltas above, no `unwrap()`/`expect()` outside + tests) — **not** an asyncification port. The bridged seam + ([ADR-003]) remains the async story; a substrate that grew tokio + would forfeit the diff-reviewability the fork's economics rest on. + +### 5. Bootstrap and schema machinery + +- The fork owns the bootstrap; **append-column migrations stay** (the + read's §4 scale verdict). The concurrent-bootstrap duplicate-column + race swallow is re-keyed: on `ALTER TABLE` failure, verify the column + via `pragma_table_info` — present ⇒ benign race swallowed, absent ⇒ + propagate. Upstream's error-*string* matching + (`.contains("duplicate column")`) is not inherited. +- The **stamp columns and `claimed_at`** are added by the re-derivation + on contract v1 ([ADR-010] §3a's realization; superseding the "no + schema change needed" expectation recorded in ADR-010 §8 — annotated + there). +- **No online rename migration.** The substrate bootstraps + `__alkstore_*` fresh. A file carrying leftover upstream tables gets + inert, ignored `_honker_*` orphans (documented; never touched, never + read) — no detection, no export path, no migration machinery, because + no alkstore database under the old names exists to migrate. + +### 6. Upstream tracking without a dependency + +No ambient upstream tracking — the family's no-ambient posture applies +to maintenance too. When upstream ships something relevant (the fix +train already being the motivating example), adoption is a deliberate +cherry-pick recorded in the fork's delta register, evaluated by the +same port-cost logic as any dependency change. Re-adoption of upstream +as a *dependency* again keeps quality-read §7's bar: it must beat the +owned, tested, contract-ahead fork on maintenance, not on novelty. + +## Consequences + +**Positive** + +- One normative owner for every contract-pinned formula — the contract + text — with each engine owning one pinned-equivalent implementation; + both engines keep symmetric layering. +- Cherry-picks from upstream stay cheap where they matter (the kept + half); the diff between the substrate and its lineage remains + reviewable. +- The shipped library is panic-free without giving up the dead-man's + switch's observable honesty. +- The rename creates no migration machinery, and foreign-database + leftovers have a defined, inert fate. + +**Negative** + +- Lineage tokens survive in internal names (`Writer`, upstream + function names) — bounded by §3's ownership argument, invisible to + consumers, but a fossil a future reader must understand as + deliberate. +- The contract formulas' arithmetic sits in engine code — small, but + now it must be *tested into equivalence* by the contract suite (the + verification backlog gains the re-pin rows this implies). +- Upstream relevance is ours to notice; no release cadence, changelog, + or `cargo update` flow will tell us. + +## References + +- [ADR-011](011-sqlite-substrate-fork.md) — the fork decision this ADR + designs; its scope register and packaging are inputs. +- [ADR-005](005-dependency-ownership.md) — the posture calculus the + fork's maintenance assumptions inherit. +- [ADR-001](001-crate-split.md) — the dependency direction §2 + preserves. +- [ADR-008](008-contract-v1-pinning.md) — §6 (contract = returned + trait; substrate API never consumer-visible), §5 (taxonomy mapping + site). +- [ADR-010](010-queue-semantics-depth.md) — §3a (stamps the schema + machinery must now carry), §3 (the curve one-owner rule). +- [ADR-003](003-sqlite-driver.md) — the bridged seam §4 keeps; + watcher spawn fallibility lands there. +- `docs/research/quality-read-honker-core.md` — W-1..W-4, the dead-man's + switch verification, the bootstrap-brittleness finding, §7's + re-adoption bar. +- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this ADR's + boundary realizes. +- OQ-06 (`docs/architecture/open-questions.md`) — the resolved + assessment this ADR follow-throughs; OQ-11 — the scaffold-time + residue this ADR leaves (workspace wiring, register format, + cherry-pick procedure). + +[ADR-001]: 001-crate-split.md +[ADR-003]: 003-sqlite-driver.md +[ADR-008]: 008-contract-v1-pinning.md +[ADR-010]: 010-queue-semantics-depth.md +[ADR-011]: 011-sqlite-substrate-fork.md diff --git a/docs/architecture/engine-postgres.md b/docs/architecture/engine-postgres.md index f25f935..33afdf3 100644 --- a/docs/architecture/engine-postgres.md +++ b/docs/architecture/engine-postgres.md @@ -65,7 +65,7 @@ obligations live in the core spec. |---|---| | notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers | | streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) | -| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema | +| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema; the curve/stamps arithmetic is computed engine-side per [ADR-012](decisions/012-forked-substrate-design.md) §2 (equivalence with the SQLite engine pinned by the contract suite) | | named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; guarantee row pinned by [ADR-008](decisions/008-contract-v1-pinning.md) §7) | | scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule rows in the engine-owned schema, tick re-derived (boundary advance + 64-boundary catch-up cap, honker parity), leadership via the engine's lock machinery on `__alkstore_scheduler`; outbox = helper over queues | | begin_tx | pool checkout + `BEGIN`, returning the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) | @@ -108,6 +108,7 @@ the record in [ADR-003](decisions/003-sqlite-driver.md). | [008](decisions/008-contract-v1-pinning.md) | Contract v1 | pinned surface; reserved reconnect-wake channel string; `PayloadTooLarge` taxonomy variant | | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule rows in the engine schema, re-derived tick, row-locked fire tx, `__alkstore_scheduler` leadership | | [010](decisions/010-queue-semantics-depth.md) | Queue depth | job-stamped opts, equal-jitter backoff, dead-letter move, no-stranded-rows sweep, one engine-owned schema | +| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate (SQLite side); pg engine owns its own curve/stamps arithmetic, equivalence pinned by the contract suite | ## Open Questions diff --git a/docs/architecture/engine-sqlite.md b/docs/architecture/engine-sqlite.md index 328af1a..38a2604 100644 --- a/docs/architecture/engine-sqlite.md +++ b/docs/architecture/engine-sqlite.md @@ -17,8 +17,8 @@ spec and are not restated here. - Single driver: rusqlite, riding the [forked honker-core substrate](decisions/011-sqlite-substrate-fork.md) - (fork of published honker-core 0.5.0's lineage at the reference - revision), + (`alkstore-substrate` — the vendored lineage crate, + [ADR-012](decisions/012-forked-substrate-design.md)) with `bundled-sqlite` for hermetic builds. No `.so` runtime artifacts ([ADR-003](decisions/003-sqlite-driver.md) for the driver decision; ownership resolved by @@ -45,10 +45,12 @@ spec and are not restated here. subscribers per poll tick, even when several commits coalesced inside one tick — wake is a hint; consumers re-read indexed state, [ADR-006](decisions/006-wake-and-delivery-contract.md)). -- Each connection runs honker's bootstrap at open: pragmas +- Each connection runs the substrate's bootstrap at open: pragmas (WAL, `synchronous=NORMAL`, busy timeout), `attach_notify`, - `attach_honker_functions`, `bootstrap_honker_schema` (the - alknet-filesystem POC's wiring shape). + the substrate's function attachments and schema bootstrap (the + alknet-filesystem POC's wiring shape). Watcher spawn is fallible in + the substrate ([ADR-012](decisions/012-forked-substrate-design.md) + §4); open fails if the watcher cannot start. ## Mapping the contract @@ -57,8 +59,8 @@ spec and are not restated here. | notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) | | streams | honker's stream machinery; explicit offset saves through the tx seam | | queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) | -| named locks | honker's lock machinery | -| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) | | +| named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) | +| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) | | begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) | | handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) | | commit/rollback | releases the writer slot | @@ -86,18 +88,21 @@ spec and are not restated here. ## What the fork question resolved -The [quality read (OQ-06)](decisions/005-dependency-ownership.md) — +The [quality read (OQ-06)](../research/quality-read-honker-core.md) — this engine's dependency gate — resolved 2026-10-05: ADR-005's fork -trigger **fired** ([ADR-011](decisions/011-sqlite-substrate-fork.md)). -The substrate is owned code forked from honker-core's lineage; the +trigger **fired** ([ADR-011](decisions/011-sqlite-substrate-fork.md)), +and the fork's design is pinned by +[ADR-012](decisions/012-forked-substrate-design.md): the substrate is +`alkstore-substrate`, a vendored contract-blind lineage crate; the watcher/connection architecture this spec describes is inherited verbatim where kept ([ADR-003](decisions/003-sqlite-driver.md) -unchanged in architecture). The +unchanged in architecture), with the port deltas ADR-012 §4 decides +(fallible watcher spawn, bounded reconnect backoff, panic-free +death-closes-subscribers). The [evidence](../research/quality-read-honker-core.md): published 0.5.0's unreleased-fix-train defect class, plus ADR-010's queue depth -requiring engine-owned queue SQL in any posture. The forked -substrate's table family is `__alkstore_*` (ADR-010 §8's naming -authorization). +requiring engine-owned queue SQL in any posture. The substrate's table +family is `__alkstore_*` (ADR-010 §8's naming authorization). ## Design Decisions @@ -105,7 +110,7 @@ authorization). |---|---|---| | [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate | | [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves | -| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + honker-core lineage, bridged seam, inherited watcher — ownership resolved by 011 | +| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + the forked substrate lineage, bridged seam, inherited watcher — ownership resolved by 011 | | [005](decisions/005-dependency-ownership.md) | Ownership | published honker-core; fork trigger fired by OQ-06 (ADR-011) | | [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | data_version watcher, coalescing, death-closes-receivers | | [007](decisions/007-transactional-seam.md) | Tx seam | writer-slot lease, `BEGIN IMMEDIATE`, `spawn_blocking` round-trips | @@ -113,6 +118,7 @@ authorization). | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | `@every` specs; `__alkstore_scheduler` leadership; cron rejected | | [010](decisions/010-queue-semantics-depth.md) | Queue depth | semantics pinned; §5/§3a realization resolved by ADR-011 (owned code) | | [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | OQ-06's trigger fired; substrate owned, queue ops re-derived on contract v1 | +| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate, fidelity posture, port deltas, panic-free watcher death | ## Open Questions diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index 90d262c..40fb7fa 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -14,12 +14,16 @@ everything else hangs off); **OQ-09 + OQ-05 resolved** (2026-10-05, [ADR-010](decisions/010-queue-semantics-depth.md) — the queue semantics track and the scheduler collapse/guarantee row); **OQ-06 resolved** (2026-10-05, [ADR-011](decisions/011-sqlite-substrate-fork.md) — the -fork trigger fired on the published-artifact facts). Next: OQ-08 -(rides the now-pinned trait shape and the ADR-011 substrate fork), -OQ-10 (versioning discipline for contract extensions; note ADR-011 -changes its substrate-side facts for SQLite — the forked crate is -versioned in the family workspace, so the engine/core contract pairing -is what the discipline must track). +fork trigger fired on the published-artifact facts); **the fork +design follow-through** (2026-10-05, +[ADR-012](decisions/012-forked-substrate-design.md) — substrate +boundaries, fidelity posture, port deltas; its substrate-side residue +is OQ-11). Next: OQ-08 (rides the now-pinned trait shape and the +ADR-011 substrate fork), OQ-10 (versioning discipline for contract +extensions; note ADR-011 changes its substrate-side facts for SQLite — +the forked crate is versioned in this repository's Cargo workspace, so +the engine/core contract pairing is what the discipline must track), and +OQ-11 (fork follow-through items — substrate-side, non-consumer-facing). Resolved questions stay listed with their resolution; they are not deleted. @@ -152,7 +156,11 @@ narrowed to the pinning work its own record already scoped.)* **stands** (delete-on-ack; pg-boss's completed-row model is result storage under another name). (8) Table layout — pg engine-owned schema (`alkstore` default), queues are rows not tables; SQLite - rides `_honker_*`. No new error variants (ADR-008 §5's rule). + rides `_honker_*`. *(Item 8's SQLite half superseded 2026-10-05: + the `_honker_*` family is re-owned as `__alkstore_*` by the fork — + [ADR-011](decisions/011-sqlite-substrate-fork.md), designed per + [ADR-012](decisions/012-forked-substrate-design.md).)* No new error + variants (ADR-008 §5's rule). - **Cross-references**: OQ-09, OQ-06. ### OQ-09: Is the scheduler a first-class mechanism, or queues + `schedule()`? — **RESOLVED** @@ -225,15 +233,15 @@ narrowed to the pinning work its own record already scoped.)* - **Resolution**: open. The pg engine is natively multi-host (POC #2 verified — no single-host assumption to remove); SQLite is single-machine by nature (file-backed, NFS-two-writers unsupported — - honker's honesty posture, inherited). The unified surface must not - pretend SQLite is multi-host. Options: per-engine capability flags - (`Store::capabilities()`), a documented deployment matrix only - ([deployment.md](deployment.md) carries the facts), or compile-time - knowledge only - (a consumer choosing the SQLite engine knows). Rides the now-pinned - contract shape ([ADR-008](decisions/008-contract-v1-pinning.md)): - the trait constrains where capability differences can surface. -- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md). + honker's honesty posture, inherited by the forked substrate). The + unified surface must not pretend SQLite is multi-host. Options: + per-engine capability flags (`Store::capabilities()`), a documented + deployment matrix only ([deployment.md](deployment.md) carries the + facts), or compile-time knowledge only (a consumer choosing the + SQLite engine knows). Rides the now-pinned contract shape + ([ADR-008](decisions/008-contract-v1-pinning.md)): the trait + constrains where capability differences can surface. +- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md), [ADR-012](decisions/012-forked-substrate-design.md) (the substrate inherits the honesty posture). ### OQ-10: How do engine crates track core-contract version changes? @@ -251,8 +259,38 @@ narrowed to the pinning work its own record already scoped.)* - **Cross-references**: OQ-04 (the contract being versioned — now including its first post-v1 extensions, ADR-009/ADR-010), OQ-02. +### OQ-11: Forked-substrate follow-through — scaffold, provenance register, and cherry-pick discipline + +- **Origin**: [ADR-012](decisions/012-forked-substrate-design.md) + (the fork design; substrate-side residue), [ADR-011](decisions/011-sqlite-substrate-fork.md) +- **Status**: open +- **Priority**: medium (nothing consumer-facing rides on it; it + resolves within the fork-scaffold task, which it does not gate) +- **Resolution**: open. ADR-012 fixed the *design* (contract-blind + boundary, fidelity posture, port deltas, bootstrap machinery, + upstream-tracking stance) and also pinned the provenance register's + location, timing, and initial contents (§1). The residue is: (1) the + exact Cargo-workspace layout and build wiring of + `alkstore-substrate` in this repository (crate location, feature + gating of the `bundled-sqlite` interplay with the engine crate's own + rusqlite dependency); (2) the provenance register's *format and + delta-list granularity* (per-commit entries vs per-delta-class; + whether cherry-pick records append at adoption time — location, + timing, and initial contents are ADR-012 §1's, not re-opened here); + (3) the cherry-pick *procedure* in practice — how a candidate + upstream fix is evaluated, applied, and recorded so ADR-012 §6's + deliberate-work posture stays auditable. These resolve *within* the + fork-scaffold task (its opening section), not before it and not as a + gate on writing it; the scaffold's build wiring depends on (1)'s + outcome. +- **Cross-references**: OQ-08 (the contract surface the engine maps + the substrate under), OQ-10 (the substrate is a crate in this + repository's Cargo workspace — its versioning interacts with the + discipline), ADR-011, ADR-012. + ## Deferred / Blocked -None currently. Every open OQ above is actionable Phase 1 architecture -work (capability-surface shape, versioning discipline) with its -evidence base complete — no external arrivals are being waited on. \ No newline at end of file +None currently. Every open OQ above is actionable Phase 1 work +(capability-surface shape, versioning discipline, fork-scaffold +follow-through) with its evidence base complete — no external arrivals +are being waited on. \ No newline at end of file diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 490b714..28d57e7 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -29,7 +29,7 @@ Per [ADR-001](decisions/001-crate-split.md): | Crate | Contents | Driver dependencies | |---|---|---| | `alkstore` (core) | trait surface, types, error model | none (capability flags, if ever, are OQ-08's to add) | -| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, honker-core | +| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, `alkstore-substrate` (the forked honker-core lineage — [ADR-011](decisions/011-sqlite-substrate-fork.md), designed in [ADR-012](decisions/012-forked-substrate-design.md)) | | `alkstore-postgres` | Postgres engine ([ADR-004](decisions/004-postgres-driver.md)) | tokio-postgres, deadpool-postgres | | (mem engine, optional) | test convenience, decided at implementation ([ADR-001](decisions/001-crate-split.md)) | none | @@ -53,7 +53,7 @@ Per [ADR-002](decisions/002-feature-scope.md): | Doc | Purpose | |---|---| | [core-contract.md](core-contract.md) | The unified trait surface: mechanisms, delivery guarantees, tx seam, naming | -| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto honker-core/rusqlite | +| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto the forked substrate (`alkstore-substrate`)/rusqlite | | [engine-postgres.md](engine-postgres.md) | Postgres engine: mapping the contract onto tokio-postgres/LISTEN | | [queues.md](queues.md) | Queue/scheduler/outbox semantics depth (ADR-009/ADR-010 resolved) | | [deployment.md](deployment.md) | Host capabilities, connection budgets, deployment matrix (OQ-08) | @@ -66,7 +66,7 @@ Per [ADR-002](decisions/002-feature-scope.md): |---|---|---| | [001](decisions/001-crate-split.md) | Reactive-core + per-engine crates | Accepted | | [002](decisions/002-feature-scope.md) | Feature scope (inventory-confirmed) | Accepted | -| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core, bridged seam | Accepted | +| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core lineage, bridged seam (ownership: ADR-011) | Accepted | | [004](decisions/004-postgres-driver.md) | Postgres: tokio-postgres + deadpool, hand-rolled LISTEN | Accepted | | [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted | | [006](decisions/006-wake-and-delivery-contract.md) | Opaque wake + re-read; notify-vs-streams guarantee split | Accepted | @@ -74,6 +74,8 @@ Per [ADR-002](decisions/002-feature-scope.md): | [008](decisions/008-contract-v1-pinning.md) | Contract v1 surface pinning (partition, TxHandle shape, wake type, reserved strings, error taxonomy) | Accepted | | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse (queues + `schedule()`, `@every`-only) | Accepted | | [010](decisions/010-queue-semantics-depth.md) | Queue semantics depth (visibility, backoff, dead-letter, sweep, layout) | Accepted | +| [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted | +| [012](decisions/012-forked-substrate-design.md) | Forked substrate design (contract-blind boundary, fidelity, port deltas) | Accepted | ## Non-goals diff --git a/docs/architecture/queues.md b/docs/architecture/queues.md index 5b5315b..9e98f04 100644 --- a/docs/architecture/queues.md +++ b/docs/architecture/queues.md @@ -26,7 +26,9 @@ made under; the ADRs carry the WHY. write on both engines ([ADR-007](decisions/007-transactional-seam.md)); rollback drops the job row with no ghosts. - **Exactly-once claim**: `FOR UPDATE SKIP LOCKED` (pg, POC-pinned - under 4×4 concurrency) / honker's machinery (SQLite). At-least-once + under 4×4 concurrency) / the forked substrate's single-statement + claim (SQLite, re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) + §3a — [ADR-011](decisions/011-sqlite-substrate-fork.md)). At-least-once *work*; exactly-once *processing* needs the ack + visibility model. - **Wake-driven consumption**: the pg engine's default is LISTEN-driven claim with a re-poll safety net; SQLite's is watcher wake with @@ -181,11 +183,12 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects. stamped onto the enqueued job per [ADR-010](decisions/010-queue-semantics-depth.md) §3a; the queue mechanism's guarantees apply, no separate delivery machinery. -- Schedule storage is engine-internal (SQLite: honker's - `_honker_scheduler_tasks`; Postgres: the engine-owned schema) — not - a consumer namespace; queues are names, not registered objects, so a - schedule may target a queue nothing has claimed yet. Schedule names - validate like queue names (non-empty, reserved-prefix rejected). +- Schedule storage is engine-internal (SQLite: the forked substrate's + `__alkstore_scheduler_tasks` table; Postgres: the engine-owned + schema) — not a consumer namespace; queues are names, not registered + objects, so a schedule may target a queue nothing has claimed yet. + Schedule names validate like queue names (non-empty, + reserved-prefix rejected). ## Namespaces / schema layout (ADR-010 §8) @@ -197,11 +200,11 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects. table** — no per-queue tables/schemas; pg-boss's partition-per-queue opt-in is not inherited (one table + partial indexes matches honker's proven shape and keeps `queue(name)` a name, not a DDL operation). -- **SQLite**: honker's `_honker_*` family — storage-internal - ([ADR-008](decisions/008-contract-v1-pinning.md) §4); no new tables - minted by this design; the family's fate resolved with OQ-06 — the - fork ([ADR-011](decisions/011-sqlite-substrate-fork.md)) re-owns the - names (`__alkstore_*`); nothing consumer-visible changes. +- **SQLite**: the forked substrate's `__alkstore_*` table family — + storage-internal ([ADR-008](decisions/008-contract-v1-pinning.md) + §4); the fork ([ADR-011](decisions/011-sqlite-substrate-fork.md)) + re-owns the names from upstream (`_honker_*`); nothing + consumer-visible changes. - Partial indexes pinned identically on both engines: partial indexes matching the claim hot path; dead rows outside it; single clock source (second-precision timestamps); savepoint-guarded @@ -209,9 +212,10 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects. ## Reference material -- **honker's queue design** — the SQLite-side incumbent design - reference (`/workspace/honker`; ownership resolved by - [ADR-011](decisions/011-sqlite-substrate-fork.md) — forked, queue ops +- **honker's queue design** — the SQLite-side lineage design reference + (`/workspace/honker`; ownership resolved by + [ADR-011](decisions/011-sqlite-substrate-fork.md) — forked per + [ADR-012](decisions/012-forked-substrate-design.md), queue ops re-derived on this design). Full read: `docs/research/reference-honker-machinery.md` (schema, claim/visibility/retry/dead-letter mechanics with file/line cites; @@ -243,6 +247,8 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects. | [008](decisions/008-contract-v1-pinning.md) | Contract v1 | queue skeleton + EnqueueOpts pinned; depth was this doc's design space | | [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | queues + `schedule()` + runner; `@every`-only; boundary guarantee row | | [010](decisions/010-queue-semantics-depth.md) | Queue depth | visibility/heartbeat rules, backoff curve, dead-letter, no-stranded-rows sweep, schema layout | +| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite-side queue ops re-derived in owned code; `__alkstore_*` naming | +| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; fidelity posture; port deltas | ## Open Questions diff --git a/docs/research/quality-read-honker-core.md b/docs/research/quality-read-honker-core.md index a66beea..19adb83 100644 --- a/docs/research/quality-read-honker-core.md +++ b/docs/research/quality-read-honker-core.md @@ -285,7 +285,10 @@ pre-paid. - **Keep as-ported:** the PRAGMA block + `set_journal_mode_wal` retry logic; `Writer`; `Readers`; the polling watcher + `SharedUpdateWatcher` + `WatcherDeathGuard` + `stat_identity` dead-man's switch (W-1 - backoff, W-2 fallible spawn applied in port); the `in_savepoint` / + backoff, W-2 fallible spawn applied in port; the dead-man's-switch + panic replaced by a deliberate watcher-fatal death — decided after + this register by [ADR-012](../architecture/decisions/012-forked-substrate-design.md) + §4, three watcher deltas total); the `in_savepoint` / `UnwindUndo` machinery as the mutation-transition discipline; the REAL-coercion arg helpers; the `notify()` scalar + notifications table (renamed, plus an at-attach pruning cap implementing ADR-010 @@ -317,6 +320,10 @@ pre-paid. - [ADR-011](../architecture/decisions/011-sqlite-substrate-fork.md) — the fork decision and packaging. +- [ADR-012](../architecture/decisions/012-forked-substrate-design.md) — + the fork's design follow-through: the contract-blind substrate + boundary, the API fidelity posture, and this document's port notes + (W-1..W-4 and the dead-man's-switch panic) decided per item. - [ADR-005](../architecture/decisions/005-dependency-ownership.md) — posture row for honker-core annotated (trigger fired); the calculus is applied, not renegotiated.