ADR-012: forked-substrate design — contract-blind boundary, fidelity posture, port deltas
Follow-through on OQ-06/ADR-011: pin the fork's structural decisions (alkstore-substrate as a vendored path-dep crate, contract-blind API boundary with contract formulas computed engine-side and pinned equivalent by the contract suite, keep-the-kept-half API fidelity for cheap cherry-picks, the W-1/W-2/dead-man's-switch/W-4 port deltas decided per item, bootstrap re-keying off error-string matching, no rename migration, deliberate upstream tracking). Consistency sweep across the doc set for the fork: annotate ADR-003/ 005/009/010 and core-contract for superseded ownership facts, fix schedule-storage table naming (ADR-009 §5, queues.md), re-key ADR-010 §6's notifications hygiene to the at-attach cap the fork scope realizes, add OQ-11 (scaffold-time residue), and complete both ADR indexes. Independent review: 0 critical, warnings addressed.
This commit is contained in:
1 parent
befbe2e714
commit
2949612e2c
18 files changed
+491
-117
No files matched your search
@@ -37,7 +37,7 @@ pending architecture review and OQ resolution.
|
||||
|---|---|---|
|
||||
| [001](decisions/001-crate-split.md) | Reactive-core crate + per-engine crates | Accepted |
|
||||
| [002](decisions/002-feature-scope.md) | Feature scope — inventory-confirmed surface | Accepted |
|
||||
| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core, bridged seam | Accepted |
|
||||
| [003](decisions/003-sqlite-driver.md) | SQLite engine — rusqlite + honker-core lineage, bridged seam (ownership: ADR-011) | Accepted |
|
||||
| [004](decisions/004-postgres-driver.md) | Postgres engine — tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
|
||||
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
|
||||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract — opaque wake + re-read; notify-vs-streams split | Accepted |
|
||||
@@ -45,6 +45,8 @@ pending architecture review and OQ resolution.
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 surface pinning — surface partition, `TxHandle` shape, wake type, reserved strings, error taxonomy | Accepted |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse — queues + `schedule()`/`run_schedules`, `@every`-only v1, boundary guarantee row | Accepted |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue semantics depth — visibility/renewal, backoff curve, dead-letter, no-stranded-rows sweep, schema layout | Accepted |
|
||||
| [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Forked substrate design — contract-blind boundary, fidelity posture, port deltas | Accepted |
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -59,6 +61,9 @@ in suggested resolution order (OQ-04, OQ-09, OQ-05, OQ-06 resolved):
|
||||
honker-core's lineage, queue ops re-derived on contract v1.
|
||||
- **OQ-08** (medium): capability-surface shape.
|
||||
- **OQ-10** (medium): contract versioning across engine crates.
|
||||
- **OQ-11** (medium): forked-substrate follow-through (scaffold-time
|
||||
decisions — [ADR-012](decisions/012-forked-substrate-design.md)'s
|
||||
residue).
|
||||
|
||||
Resolved (kept with resolutions): OQ-01 (feature scope), OQ-02
|
||||
(crate split), OQ-03 (drivers), OQ-07 (extension surface cut),
|
||||
|
||||
@@ -187,8 +187,8 @@ TTL-bounded coordination locks, transactional-friendly:
|
||||
no-work is a value, not an error); `renew` and `release` on the
|
||||
lock handle. Release on explicit unlock or TTL expiry.
|
||||
Re-acquirable after expiry (POC-pinned on Postgres; on SQLite it
|
||||
rests on honker's machinery — the SQLite-side pin is in the
|
||||
verification backlog below).
|
||||
rests on the forked substrate's lock machinery — the SQLite-side pin
|
||||
is in the verification backlog below).
|
||||
- **Guarantee row** ([ADR-008](decisions/008-contract-v1-pinning.md)
|
||||
§7): mutual exclusion bounded by TTL + renewal — after TTL expiry
|
||||
exclusion lapses *silently* (no revocation event); holders must
|
||||
@@ -285,8 +285,11 @@ added by [ADR-009](decisions/009-scheduler-collapse.md) §1):
|
||||
- Engine-derived names in consumer namespaces carry the reserved
|
||||
prefix (the outbox's backing queue is derived under the prefix —
|
||||
honker's `_outbox:{name}` scheme is not inherited verbatim).
|
||||
- SQLite: honker's `_honker_*` internal table family is storage-
|
||||
internal (not consumer namespace); Postgres: engine tables are
|
||||
- SQLite: the substrate's `__alkstore_*` internal table family is
|
||||
storage-internal (not consumer namespace) — re-owned from upstream's
|
||||
`_honker_*` by the fork ([ADR-011](decisions/011-sqlite-substrate-fork.md),
|
||||
designed per [ADR-012](decisions/012-forked-substrate-design.md)).
|
||||
Postgres: engine tables are
|
||||
schema-scoped — one engine-owned schema
|
||||
([ADR-010](decisions/010-queue-semantics-depth.md) §8) — the
|
||||
*channel* namespace is this section's.
|
||||
@@ -297,9 +300,11 @@ Contract properties POC-pinned on one engine only (or sketched rather
|
||||
than surface-verified) — the contract test suite must pin both engines
|
||||
before the engine specs are called `stable`:
|
||||
|
||||
- **Lock TTL/expiry re-acquisition on SQLite** — pinned on Postgres
|
||||
(pg POC's lock probe); on SQLite it rests on honker's machinery
|
||||
unverified.
|
||||
- **Named-lock TTL/expiry re-acquisition on SQLite** — pinned on
|
||||
Postgres (pg POC's lock probe); on SQLite it rests on the forked
|
||||
substrate's lock machinery (its `lock_renew` verified in the quality
|
||||
read; the re-acquire-does-not-refresh-TTL behavior is upstream's,
|
||||
inherited deliberately) — pin it in the contract suite.
|
||||
- **Wake semantics under `WakeReceiver` shapes** — POCs verified
|
||||
wake delivery/coalescing through their own probe types; the pinned
|
||||
`Wake { channel }` / recv forms (§3 of
|
||||
@@ -307,21 +312,29 @@ before the engine specs are called `stable`:
|
||||
suite's own property tests on both engines.
|
||||
- **`save_offset_tx` exactly-once-within-a-business-tx shape** — both
|
||||
POCs verified it through their *sketch* implementations (SQLite's
|
||||
offset-save was a plain SQL upsert, not the full honker surface;
|
||||
offset-save was a plain SQL upsert, not the forked substrate's full
|
||||
surface;
|
||||
the pg side likewise through its probe), so the property must be
|
||||
re-pinned against the real engines' `save_offset_tx` in the
|
||||
contract suite.
|
||||
- **Concurrent `try_lock` loser/error behavior** on SQLite (the pg
|
||||
side returns cleanly; honker's busy-path under lock contention is
|
||||
the thing to pin).
|
||||
side returns cleanly; the substrate's busy-path under lock
|
||||
contention is the thing to pin).
|
||||
- **Scheduler + depth properties on Postgres** — the SQLite side's
|
||||
tick/leader/catch-up machinery rides honker's test-pinned
|
||||
implementation; the pg engine's re-derived tick (boundary advance,
|
||||
tick/leader/catch-up machinery inherits the forked substrate's
|
||||
test-pinned implementation; the pg engine's re-derived tick (boundary
|
||||
advance,
|
||||
64-cap skip-forward, leadership-loss discipline) and the
|
||||
[ADR-010](decisions/010-queue-semantics-depth.md) depth properties
|
||||
(visibility reclaim consuming attempts, dead-letter moves, the
|
||||
no-stranded-rows sweep) pin in the contract suite at
|
||||
implementation.
|
||||
- **Backoff-curve and stamp-resolution equivalence across engines** —
|
||||
the equal-jitter curve ([ADR-010](decisions/010-queue-semantics-depth.md)
|
||||
§3) and the opts-stamping resolution (§3a) are computed engine-side
|
||||
per [ADR-012](decisions/012-forked-substrate-design.md) §2's
|
||||
contract-blind boundary; the contract suite must pin both engines'
|
||||
arithmetic to identical outputs.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
@@ -334,6 +347,8 @@ before the engine specs are called `stable`:
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 pinning | surface partition, `TxHandle` trait shape, `Wake` type, reserved strings, error taxonomy, config split, locks guarantee row |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse (post-v1 extension) | queues + `schedule()`/`run_schedules`, `@every`-only v1, boundary guarantee row |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth (post-v1 extension) | visibility/renewal, opts stamping, backoff curve, dead-letter, sweep, layout |
|
||||
| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite substrate owned (`__alkstore_*` naming); queue ops re-derived on contract v1 |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; engine-side formula arithmetic pinned equivalent by the contract suite |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -13,7 +13,8 @@ theirs:
|
||||
|
||||
- The **SQLite** engine rides honker's existing, battle-tested machinery
|
||||
(published `honker-core` — a port-and-adapt of a sync library to the
|
||||
family's async-facing-trait posture).
|
||||
family's async-facing-trait posture). *(Ownership since resolved:
|
||||
the substrate is forked into owned code — [ADR-011](011-sqlite-substrate-fork.md).)*
|
||||
- The **Postgres** engine is the build-heavy side: LISTEN/NOTIFY wiring
|
||||
and the pg-boss-family queue schema are built by this crate.
|
||||
|
||||
@@ -37,7 +38,9 @@ The project ships as a **family of crates**:
|
||||
documentation. No driver dependencies. Compile-lean by construction:
|
||||
the base crate has no engine machinery to keep out.
|
||||
2. **`alkstore-sqlite`** — the SQLite engine implementing the core
|
||||
surface. Single driver: rusqlite + honker-core ([ADR-003]).
|
||||
surface. Single driver: rusqlite + the forked `alkstore-substrate`
|
||||
lineage ([ADR-003]; ownership per
|
||||
[ADR-011](011-sqlite-substrate-fork.md)).
|
||||
3. **`alkstore-postgres`** — the Postgres engine implementing the core
|
||||
surface. Single driver: tokio-postgres + deadpool-postgres
|
||||
([ADR-004]).
|
||||
@@ -85,4 +88,6 @@ crate. A consumer binary links at most one driver.
|
||||
link-collision constraint motivates this split.
|
||||
- [ADR-004](004-postgres-driver.md) — the Postgres driver choice.
|
||||
- OQ-10 (`docs/architecture/open-questions.md`) — trait-versioning
|
||||
duties created here.
|
||||
duties created here.
|
||||
[ADR-003]: 003-sqlite-driver.md
|
||||
[ADR-004]: 004-postgres-driver.md
|
||||
@@ -39,9 +39,15 @@ the comparison:
|
||||
|
||||
The SQLite engine uses **posture 1**: published `honker-core = 0.5`
|
||||
as a library dependency over the crate's own rusqlite connections
|
||||
(`bundled-sqlite` for hermetic builds).
|
||||
(`bundled-sqlite` for hermetic builds). *(Ownership superseded
|
||||
2026-10-05 by [ADR-011](011-sqlite-substrate-fork.md): the substrate
|
||||
is the forked `alkstore-substrate` lineage crate, not the published
|
||||
dependency — the driver, seam, and watcher architecture below are
|
||||
unchanged.)*
|
||||
|
||||
- **Driver**: rusqlite, riding honker-core's pinned version.
|
||||
- **Driver**: rusqlite, riding honker-core's pinned version. *(The
|
||||
pin is ours to move deliberately since the fork — [ADR-011], per
|
||||
[ADR-005]'s annotated rusqlite row.)*
|
||||
- **Async seam**: bridged. The writer slot (`Writer`) +
|
||||
reader pool (`Readers`) + every engine call in `spawn_blocking`.
|
||||
A dedicated std-thread bridge (one thread owning the writer conn,
|
||||
@@ -76,8 +82,10 @@ comparable reuse.
|
||||
- Static linkage: "open a path, get a store" needs no runtime
|
||||
artifacts.
|
||||
- The engine's job shrinks to wiring + the SQL surfaces honker doesn't
|
||||
cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's
|
||||
machinery directly) + the async bridge.
|
||||
cover (queue semantics depth, OQ-ST-05's SQLite side rides honker's
|
||||
machinery directly) + the async bridge. *(Ownership since resolved:
|
||||
the substrate is forked — [ADR-011](011-sqlite-substrate-fork.md) —
|
||||
and the queue ops re-derived on contract v1 within it.)*
|
||||
|
||||
**Negative**
|
||||
|
||||
@@ -103,4 +111,7 @@ comparable reuse.
|
||||
- [ADR-007](007-transactional-seam.md) — the tx-handle shape both
|
||||
engines implement.
|
||||
- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this
|
||||
decision defines.
|
||||
decision defines.
|
||||
[ADR-001]: 001-crate-split.md
|
||||
[ADR-005]: 005-dependency-ownership.md
|
||||
[ADR-007]: 007-transactional-seam.md
|
||||
@@ -101,4 +101,8 @@ Structural constraints the engine owns (all test-pinned in POC #2):
|
||||
decision applies per-subsystem.
|
||||
- [ADR-006](006-wake-and-delivery-contract.md) — the wake contract the
|
||||
forwarder implements on this engine.
|
||||
- [engine-postgres.md](../engine-postgres.md) — the engine spec.
|
||||
- [engine-postgres.md](../engine-postgres.md) — the engine spec.
|
||||
[ADR-003]: 003-sqlite-driver.md
|
||||
[ADR-005]: 005-dependency-ownership.md
|
||||
[ADR-006]: 006-wake-and-delivery-contract.md
|
||||
[queues.md]: ../queues.md
|
||||
@@ -28,7 +28,7 @@ when a named trigger fires. Per subsystem:
|
||||
| Subsystem | Posture | Notes |
|
||||
|---|---|---|
|
||||
| honker-core (SQLite engine machinery) | published library, `honker-core = 0.5` — **trigger fired 2026-10-05**: forked per [ADR-011] | Fork trigger: the Phase 1 quality read of the watcher/transactional core finds a defect, OR a needed change upstream won't take. OQ-06 tracked the read; the read fired the trigger (published 0.5.0 carries the unreleased-fix-train defect class + ADR-010's queue depth requires engine-owned queue SQL in any posture). |
|
||||
| rusqlite | published, riding honker-core's pin | Pin movement is honker-core's; we ride it ([ADR-003]). |
|
||||
| rusqlite | published, ~~riding honker-core's pin~~ **pin ours after the fork** | Post-fork ([ADR-011]), the rusqlite generation is moved deliberately in our own substrate crate, not by upstream releases ([ADR-003]'s annotated negative consequence; [ADR-012] ports the substrate onto the family standard). |
|
||||
| tokio-postgres + deadpool-postgres | published library, as-is | Clean, zero conflicts, actively maintained (POC #2). |
|
||||
| postgres-notify | **not adopted** (derive-not-adopt) | Lazy reconnect, connect_script skipped at initial connect, unquoted-identifier LISTENs, single maintainer. Hand-rolled forwarder instead ([ADR-004]). Fallback if upstream improves materially. |
|
||||
| pg-boss queue machinery (pgboss-rs / node pg-boss) | **re-derived** on our driver; schema family as *design reference* only | pgboss-rs brings sqlx (second driver per binary) and has no LISTEN/NOTIFY (verified); the push half is ours either way, so queue-machinery reuse value is the honest comparison point and it loses on that arithmetic. Semantics depth: [queues.md], OQ-05. |
|
||||
@@ -73,4 +73,10 @@ to this posture.
|
||||
- [ADR-003](003-sqlite-driver.md), [ADR-004](004-postgres-driver.md) —
|
||||
the per-engine driver decisions applying this posture.
|
||||
- OQ-06 (`docs/architecture/open-questions.md`) — the quality-read
|
||||
tracker.
|
||||
tracker (resolved: [ADR-011], designed per [ADR-012]).
|
||||
[ADR-003]: 003-sqlite-driver.md
|
||||
[ADR-004]: 004-postgres-driver.md
|
||||
[ADR-006]: 006-wake-and-delivery-contract.md
|
||||
[ADR-011]: 011-sqlite-substrate-fork.md
|
||||
[ADR-012]: 012-forked-substrate-design.md
|
||||
[queues.md]: ../queues.md
|
||||
@@ -126,4 +126,7 @@ consumer-inventory-row-gated extension — not assumed now.
|
||||
- [ADR-007](007-transactional-seam.md) — commit-atomicity's mechanism
|
||||
(the `*_tx` seam).
|
||||
- [core-contract.md](../core-contract.md) — the spec carrying this
|
||||
contract's full surface.
|
||||
contract's full surface.
|
||||
[ADR-004]: 004-postgres-driver.md
|
||||
[deployment.md]: ../deployment.md
|
||||
[engine-postgres.md]: ../engine-postgres.md
|
||||
@@ -101,4 +101,6 @@ contract-pinning input rather than open risk:
|
||||
- [ADR-006](006-wake-and-delivery-contract.md) — commit-atomicity is
|
||||
the `*_tx` property this seam carries into the notify contract.
|
||||
- [core-contract.md](../core-contract.md) — the spec whose transaction
|
||||
section this ADR defines.
|
||||
section this ADR defines.
|
||||
[ADR-003]: 003-sqlite-driver.md
|
||||
[ADR-004]: 004-postgres-driver.md
|
||||
@@ -24,7 +24,8 @@ The evidence for the decision:
|
||||
consumer-inventory.md](../../research/consumer-inventory.md) — needs
|
||||
"run the sweep every N ticks with leader election across a fleet."
|
||||
An interval, not a schedule object.
|
||||
- **alkfs (orphan reaping)** — [recalled: OQ-FS-07] — a periodic
|
||||
- **alkfs (orphan reaping)** — [recalled: alkfs OQ-FS-07,
|
||||
`/workspace/@alkdev/alkfs/docs/research/phase-0.md`] — a periodic
|
||||
orphan-cleanup pass. An interval.
|
||||
- No consumer document (pinned, documented, or operator-authority)
|
||||
names any of: wall-clock cron expressions, per-schedule timezones,
|
||||
@@ -166,10 +167,13 @@ resolving the transfer from [ADR-008](008-contract-v1-pinning.md) §7:
|
||||
|
||||
### 5. Schedule storage is engine-internal
|
||||
|
||||
Schedule rows live in engine-owned storage (SQLite: honker's
|
||||
`_honker_scheduler_tasks` — reused as-is, its `cron_expr` column
|
||||
carrying `@every` specs; Postgres: the engine-owned schema per
|
||||
[ADR-010](010-queue-semantics-depth.md) §8). Not consumer namespace;
|
||||
Schedule rows live in engine-owned storage (SQLite: the forked
|
||||
substrate's `__alkstore_scheduler_tasks` table per
|
||||
[ADR-011](011-sqlite-substrate-fork.md) — honker's scheduler-task
|
||||
table re-owned and renamed, carrying `@every` specs in its spec
|
||||
column, the cron machinery not ported; Postgres: the engine-owned
|
||||
schema per [ADR-010](010-queue-semantics-depth.md) §8). Not consumer
|
||||
namespace;
|
||||
schedule *names* take the same validation as queue names
|
||||
(§1 — non-empty, reserved-prefix rejected). No registry
|
||||
validation against queue names — enqueuing into a queue nobody has
|
||||
@@ -240,5 +244,8 @@ respawning; distinct from clean stop, §1).
|
||||
surface (`packages/honker-rs/src/lib.rs`, `honker-core/src/honker_ops.rs`),
|
||||
`@every` grammar and local-TZ brittleness (`honker-core/src/cron.rs`),
|
||||
tick/leader mechanics.
|
||||
- [ADR-011](011-sqlite-substrate-fork.md) — the SQLite-side schedule
|
||||
storage this §5 describes is the forked substrate's; cron machinery
|
||||
not ported.
|
||||
- [core-contract.md](../core-contract.md) — the spec carrying this
|
||||
surface.
|
||||
@@ -135,9 +135,12 @@ no-renewal/expire-sweep model is not inherited):
|
||||
in v1.
|
||||
- Explicit `fail(err)` = immediate dead-letter (honker parity; it is
|
||||
the "stop retrying this" operator). `retry` at exhausted budget =
|
||||
dead-letter. Exhaustion via reclaim = dead-letter (the pre-claim
|
||||
sweep of already-exhausted reclaimable rows is an engine-side
|
||||
laziness optimization — SQLite rides honker's; pg re-derives).
|
||||
dead-letter. Exhaustion via reclaim = dead-letter. *(The pre-claim
|
||||
sweep of already-exhausted reclaimable rows — a laziness optimization
|
||||
in both upstream lineages — is engine-side and optional; SQLite rides
|
||||
the fork's re-derivation
|
||||
([ADR-011](011-sqlite-substrate-fork.md)), pg re-derives it
|
||||
([engine-postgres.md](../engine-postgres.md)).)*
|
||||
|
||||
### 3a. QueueOpts resolution: stamped at enqueue, per job
|
||||
|
||||
@@ -166,9 +169,14 @@ need a defined attachment point, and the contract pins one:
|
||||
- SQLite realization note: honker's `claim_batch` takes one uniform
|
||||
`timeout_s` per call, while per-job stamps make deadlines row-local —
|
||||
bridging that gap (post-claim re-stamp, per-row claiming, or another
|
||||
shape) is implementation work over honker's function surface and
|
||||
rides the same OQ-06 assessment as §5's zombie fix. The *contract*
|
||||
(deadline from the job's stamp) is engine-independent either way.
|
||||
shape) was implementation work over honker's function surface and
|
||||
rode the same OQ-06 assessment as §5's zombie fix. *(Resolved
|
||||
2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md));
|
||||
the claim statement is re-derived with per-row visibility from the
|
||||
stamps, in owned
|
||||
contract-blind substrate code ([ADR-012](012-forked-substrate-design.md)
|
||||
§2).) The *contract* (deadline from the job's stamp) is
|
||||
engine-independent either way.
|
||||
|
||||
### 4. Dead-letter: move, retention = never-expire by default, no redrive API
|
||||
|
||||
@@ -209,13 +217,13 @@ maintenance entry point):
|
||||
property: an expired *processing* row whose worker died is
|
||||
unreachable by claim (predicate requires future expiry), by
|
||||
pre-claim dead-lettering (same), and by `sweep_expired` (pending
|
||||
only) — the zombie hole, a real defect found in the reference read.
|
||||
This ADR fixes it at the contract level; the Postgres engine
|
||||
enforces it directly; the SQLite engine's realization over honker's
|
||||
machinery is implementation work **contingent on OQ-06** (a
|
||||
complement over honker's function surface, or the fork fires and
|
||||
the fix lands in owned code — the concrete fork candidate the
|
||||
quality read should weigh).
|
||||
only) — the zombie hole, a real defect found in the reference read
|
||||
(D-6 in the quality read's register). This ADR fixes it at the
|
||||
contract level; the Postgres engine
|
||||
enforces it directly; the SQLite engine's realization was
|
||||
implementation work **contingent on OQ-06** — *(resolved
|
||||
2026-10-05: the fork fired ([ADR-011](011-sqlite-substrate-fork.md))
|
||||
and the both-states sweep lands in owned code.)*
|
||||
- When `dead_letter_retention_s` is set, the same sweep also deletes
|
||||
dead rows past their retention (the only sweeper dead rows ever
|
||||
have — nothing runs without a caller).
|
||||
@@ -246,9 +254,14 @@ maintenance entry point):
|
||||
**out of the contract** (ADR-008 §8's disposition, resolved here by
|
||||
disposition): the notifications table is the SQLite wake
|
||||
mechanism's *transport* detail — consumers interact with wakes, not
|
||||
rows. Its hygiene is engine-internal (capped at attach, engine-side
|
||||
maintenance cadence, engine opts) — the fix for honker's unbounded
|
||||
`_honker_notifications` growth must not become a consumer's chore.
|
||||
rows. Its hygiene is engine-internal: **an at-attach pruning cap
|
||||
(engine opts)** — the mechanism the fork scope realizes
|
||||
([ADR-011](011-sqlite-substrate-fork.md)); no cadence, ambient or
|
||||
engine-side, contradicts this section's no-ambient-sweeper bullet.
|
||||
The fix for upstream's unbounded notifications growth must not become
|
||||
a consumer's chore. *(Annotated 2026-10-05: the earlier "engine-side
|
||||
maintenance cadence" wording was wrong — nothing in the fork scope
|
||||
realizes a cadence; the at-attach cap is the pin.)*
|
||||
|
||||
### 7. Result storage: cut-flag stands
|
||||
|
||||
@@ -277,15 +290,18 @@ consumer-inventory row.
|
||||
shape and keeps `queue(name)` a name, not a DDL operation — queue
|
||||
creation is not registry-gated, per
|
||||
[ADR-009](009-scheduler-collapse.md) §5). Per-queue config storage
|
||||
is likewise unnecessary (opts stamp onto job rows, §3a).
|
||||
is likewise stamped per §3a.
|
||||
- **SQLite**: honker's `_honker_*` table family in the caller's
|
||||
database file — storage-internal per
|
||||
[ADR-008](008-contract-v1-pinning.md) §4; no new tables minted by
|
||||
this ADR (job-stamped opts ride honker's existing columns and the
|
||||
contract's stated resolution rule, no schema change needed — this
|
||||
is exactly the seam the quality read (OQ-06) assesses). The
|
||||
family's exact fate rides that read: a fork re-owns the names,
|
||||
nothing consumer-visible changes.
|
||||
[ADR-008](008-contract-v1-pinning.md) §4. *(Annotated
|
||||
2026-10-05: the expectation that job-stamped opts would ride honker's
|
||||
existing columns with no schema change was falsified by the quality
|
||||
read — no stamp columns exist in the upstream schema (D-7). The
|
||||
family's fate is resolved: the fork ([ADR-011](011-sqlite-substrate-fork.md))
|
||||
re-owns the whole family as `__alkstore_*`, and the stamp columns +
|
||||
`claimed_at` are added by the re-derivation on contract v1
|
||||
([ADR-012](012-forked-substrate-design.md) §5). Nothing
|
||||
consumer-visible changes.)*
|
||||
|
||||
### 9. Error taxonomy: no delta
|
||||
|
||||
@@ -322,9 +338,9 @@ delta this track produces.
|
||||
consumers needing audit trails build them on the tx seam.
|
||||
- Honker's zombie fix, dead-row `get_job` visibility, and per-job
|
||||
visibility stamps over the uniform-claim-timeout function surface
|
||||
require either an over-machinery complement or a fork on the SQLite
|
||||
side — the fork calculus gains concrete candidates to weigh
|
||||
(OQ-06).
|
||||
required either an over-machinery complement or a fork on the SQLite
|
||||
side — resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md),
|
||||
designed in [ADR-012](012-forked-substrate-design.md)).
|
||||
- The backoff curve's 1-hour cap and jitter formula are pinned
|
||||
constants — a consumer needing a different policy uses
|
||||
`retry(err, Some(d))` per attempt (correct, but manual).
|
||||
@@ -348,6 +364,8 @@ delta this track produces.
|
||||
extends, §5's rule §9 applies, §8's dispositions §6 resolves.
|
||||
- [ADR-009](009-scheduler-collapse.md) — the collapse machinery §6's
|
||||
recipe composes with.
|
||||
- OQ-06 — the SQLite-side fork candidates (§5, §3a realization note).
|
||||
- OQ-06 — the SQLite-side fork candidates (§5, §3a realization note) —
|
||||
resolved by the fork ([ADR-011](011-sqlite-substrate-fork.md),
|
||||
designed in [ADR-012](012-forked-substrate-design.md)).
|
||||
- [queues.md](../queues.md), [core-contract.md](../core-contract.md),
|
||||
engine specs — the specs carrying this depth.
|
||||
@@ -54,17 +54,22 @@ Three facts drive this decision:
|
||||
code: fork honker-core at the reference revision
|
||||
(`/workspace/honker` @ `f4e53c6`, MIT OR Apache-2.0, provenance
|
||||
recorded per AGENTS.md §3) into the alkstore family, ported to the
|
||||
family standard (tokio-facing consumers, no comments discipline, panics
|
||||
out of library code), re-derived on contract v1 where ADR-010 pinned
|
||||
semantics honker's functions don't provide.
|
||||
family standard (the discipline deltas — no comments discipline,
|
||||
panics out of library code — with tokio-facing consumers at the engine
|
||||
seam above; per [ADR-012](012-forked-substrate-design.md) §4 the
|
||||
substrate itself stays sync), re-derived on contract v1 where ADR-010
|
||||
pinned semantics honker's functions don't provide.
|
||||
|
||||
Per-scope (the full register with keeps/re-derivations/drops is
|
||||
`docs/research/quality-read-honker-core.md` §6):
|
||||
|
||||
- **Inherited near-verbatim:** the PRAGMA/WAL open posture,
|
||||
`Writer`, `Readers`, the polling watcher + `SharedUpdateWatcher` +
|
||||
`WatcherDeathGuard` + dead-man's switch (with two port fixes the read
|
||||
named: reconnect backoff, fallible watcher spawn), the
|
||||
`WatcherDeathGuard` + dead-man's switch (with three port deltas the
|
||||
read named / ADR-012 decided: reconnect backoff, fallible watcher
|
||||
spawn, and the dead-man's-switch panic replaced by a deliberate
|
||||
watcher-fatal death — [ADR-012](012-forked-substrate-design.md) §4),
|
||||
the
|
||||
`in_savepoint` mutation-discipline machinery, arg-coercion helpers,
|
||||
the notify scalar + notifications table (renamed, with engine-
|
||||
internal hygiene per ADR-010 §6), streams, locks.
|
||||
@@ -126,4 +131,7 @@ crate (provenance + license recorded), not re-published.
|
||||
- [ADR-001](001-crate-split.md) — packaging (engine crate's
|
||||
single-driver substrate).
|
||||
- [engine-sqlite.md](../engine-sqlite.md) — the living spec, updated
|
||||
to the forked substrate.
|
||||
to the forked substrate.
|
||||
- [ADR-012](012-forked-substrate-design.md) — the fork's design
|
||||
decisions (crate identity, contract-blind boundary, fidelity
|
||||
posture, port deltas, bootstrap machinery, upstream tracking).
|
||||
@@ -0,0 +1,230 @@
|
||||
# ADR-012: Forked substrate design — contract-blind boundary, fidelity posture, port deltas
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (2026-10-05, Phase 1 — the design decisions ADR-011's fork
|
||||
requires; the follow-through of OQ-06's resolution)
|
||||
|
||||
## Context
|
||||
|
||||
[ADR-011] decided *that* the fork happens and its scope (keep /
|
||||
re-derive / drop). Deciding it surfaced structural questions the fork
|
||||
record deliberately did not settle — where the forked crate lives and
|
||||
what it is called, which layer carries contract-pinned semantics, how
|
||||
upstream diffs stay cheap, which of the read's port notes fire, and how
|
||||
a fork with no upstream dependency to update tracks its lineage. None
|
||||
of these were blockers for the fork trigger, but all of them shape the
|
||||
engine's dependency graph and the fork's maintenance cost — they are
|
||||
architecture, and they are decidable now, from decided material:
|
||||
|
||||
1. **The dependency direction is fixed** ([ADR-001]): engines depend on
|
||||
core, never the reverse. The substrate sits *below* the engine.
|
||||
2. **The substrate's Rust API is engine-internal.** The contract is the
|
||||
trait the constructor returns ([ADR-008] §6); no substrate type or
|
||||
name reaches a consumer signature.
|
||||
3. **Contract semantics are pinned engine-generically** — the
|
||||
equal-jitter curve ([ADR-010] §3), the opts-stamping resolution rule
|
||||
(§3a), schedule boundary math (ADR-009 §4) — and must compute
|
||||
identically on both engines. One engine carrying its copy inside a
|
||||
vendored lineage crate splits each pinned formula across codebases
|
||||
with different ownership characters.
|
||||
4. **The family bans panics in library code**; upstream's dead-man's
|
||||
switch panics the watcher thread on db-file replacement
|
||||
(`lib.rs:919` at the reference revision). Verified in source: the
|
||||
panic is *observed* only through `WatcherDeathGuard` — the guard's
|
||||
`Drop` fires on thread exit or panic and closes every subscriber
|
||||
(the contract-pinned failure surface). The observable semantics do
|
||||
not depend on the panic.
|
||||
5. **No alkstore databases exist.** The crate is unreleased; the
|
||||
`_honker_*` → `__alkstore_*` rename therefore has no migration
|
||||
surface and creates one only if we invent it.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Crate identity and location
|
||||
|
||||
The forked substrate is **`alkstore-substrate`** — a vendored
|
||||
workspace-family crate in this repository, a path dependency of
|
||||
`alkstore-sqlite`, not published ([ADR-011]'s packaging, now named).
|
||||
At fork-scaffold time the crate carries: the dual license files
|
||||
upstream ships (MIT OR Apache-2.0), and a **provenance register**
|
||||
(`provenance.md` at the crate root) recording upstream identity —
|
||||
checkout path, commit `f4e53c6`, license — and the delta list below
|
||||
(AGENTS.md §3's recording duty).
|
||||
|
||||
### 2. The substrate is contract-blind storage machinery
|
||||
|
||||
**The substrate depends on nothing of ours** — not on `alkstore` (the
|
||||
core crate), on the contract types, or on the error taxonomy. Its API
|
||||
takes and returns primitives: connections, names, stamp values
|
||||
(`visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`,
|
||||
`max_attempts`, `priority`, timestamps), payload strings. The **engine
|
||||
crate layer** resolves contract semantics into primitive arguments:
|
||||
`EnqueueOpts`-over-`QueueOpts` resolution, the equal-jitter curve's
|
||||
arithmetic, the schedule fire's boundary math, and the mapping of
|
||||
substrate outcomes into the contract taxonomy ([ADR-008] §5 / ADR-010
|
||||
§9's rules).
|
||||
|
||||
Why the boundary sits there, not inside the substrate:
|
||||
|
||||
- **Dependency direction**: a vendored third-party-lineage crate must
|
||||
not acquire a dependency on our contract crate (and the mirror —
|
||||
core depending on the substrate — inverts [ADR-001] outright).
|
||||
- **One normative owner per pinned formula — the contract text, not a
|
||||
substrate copy**: the curve, the stamps-resolution rule, and boundary
|
||||
math are contract-pinned to compute identically on both engines
|
||||
([ADR-010] §3). The contract is the formula's single normative
|
||||
owner; each engine owns one *implementation* of it, pinned to
|
||||
identical outputs by the contract suite (the verification backlog in
|
||||
[core-contract.md](../core-contract.md)). A substrate copy would add
|
||||
a second normative-adjacent definition in a codebase whose upstream
|
||||
does not know the contract exists.
|
||||
- **Diff reviewability**: the substrate stays as close to its lineage
|
||||
as the port allows; contract semantics are exactly the re-derivation
|
||||
surface ([ADR-011]), and they are new code with contract-derived
|
||||
names either way.
|
||||
|
||||
Dependency graph (downward only):
|
||||
|
||||
```text
|
||||
consumers
|
||||
│
|
||||
alkstore (core: traits, types, errors — no drivers)
|
||||
────────▲────────
|
||||
alkstore-sqlite alkstore-postgres
|
||||
│ (no substrate)
|
||||
alkstore-substrate (vendored lineage; path dep; unpublished)
|
||||
```
|
||||
|
||||
### 3. API fidelity posture: keep the kept half's names
|
||||
|
||||
Where the fork scope *keeps* upstream machinery ([ADR-011]'s inherited
|
||||
list), the fork keeps upstream's module structure and internal function
|
||||
names — including names carrying lineage tokens (`Writer`, `Readers`,
|
||||
`run_poll_loop`, the `in_savepoint` machinery, the lock/stream/notify
|
||||
functions). Renames are confined to: crate identity, the table family
|
||||
(`__alkstore_*`, [ADR-011]), and the hygiene deltas [ADR-011] names.
|
||||
The **re-derived** half ([ADR-011]'s list) is new code; its functions
|
||||
take contract-derived names naturally.
|
||||
|
||||
Rationale: the fork's ongoing value includes cheap cherry-picks of
|
||||
future upstream fixes — the calculus ADR-011 inherited from ADR-005
|
||||
assumes the inherited half still *tracks* its lineage. Mechanical token
|
||||
renames turn every future cherry-pick noisy; internal oddness survives
|
||||
a rename only to be paid for forever. The names are engine-internal and
|
||||
bounded by the fork's ownership.
|
||||
|
||||
### 4. Port deltas — the quality read's notes, decided
|
||||
|
||||
- **W-1 (reconnect backoff) — applied.** The watcher reconnect loop
|
||||
gains bounded backoff; a vanished db file no longer produces ~1000
|
||||
open attempts per second.
|
||||
- **W-2 (watcher spawn panics) — applied.** Watcher spawn becomes
|
||||
fallible; the engine surfaces the failure at open time (a store whose
|
||||
watcher could not start is not a store — open fails with the
|
||||
taxonomy's `Database`, detail in the source chain).
|
||||
- **Dead-man's switch (no-panics tension) — panic replaced by a
|
||||
deliberate watcher-fatal death.** On db-file identity change the
|
||||
watcher logs the precise diagnostic and exits through the ordinary
|
||||
death path. `WatcherDeathGuard` closes every subscriber exactly as it
|
||||
does today for thread panic or exit — the contract-pinned surface
|
||||
("consumers see the close, never a silent hang") is unchanged; the
|
||||
unwind is what goes away.
|
||||
- **W-4 (watcher connection opens RW) — keep RW (deliberate
|
||||
inheritance).** The read found no proof a RO watcher is safe on
|
||||
busy non-WAL paths; our posture is WAL-always anyway. Revisit only if
|
||||
a read-only/mode knob ever appears as an engine option.
|
||||
- **W-3 (`data_version` u32 wrap) — not actionable**, recorded in the
|
||||
fork's notes.
|
||||
- **The substrate stays sync.** ADR-011's "ported to the family
|
||||
standard" means the discipline deltas (no comments, panics out of
|
||||
library code per the deltas above, no `unwrap()`/`expect()` outside
|
||||
tests) — **not** an asyncification port. The bridged seam
|
||||
([ADR-003]) remains the async story; a substrate that grew tokio
|
||||
would forfeit the diff-reviewability the fork's economics rest on.
|
||||
|
||||
### 5. Bootstrap and schema machinery
|
||||
|
||||
- The fork owns the bootstrap; **append-column migrations stay** (the
|
||||
read's §4 scale verdict). The concurrent-bootstrap duplicate-column
|
||||
race swallow is re-keyed: on `ALTER TABLE` failure, verify the column
|
||||
via `pragma_table_info` — present ⇒ benign race swallowed, absent ⇒
|
||||
propagate. Upstream's error-*string* matching
|
||||
(`.contains("duplicate column")`) is not inherited.
|
||||
- The **stamp columns and `claimed_at`** are added by the re-derivation
|
||||
on contract v1 ([ADR-010] §3a's realization; superseding the "no
|
||||
schema change needed" expectation recorded in ADR-010 §8 — annotated
|
||||
there).
|
||||
- **No online rename migration.** The substrate bootstraps
|
||||
`__alkstore_*` fresh. A file carrying leftover upstream tables gets
|
||||
inert, ignored `_honker_*` orphans (documented; never touched, never
|
||||
read) — no detection, no export path, no migration machinery, because
|
||||
no alkstore database under the old names exists to migrate.
|
||||
|
||||
### 6. Upstream tracking without a dependency
|
||||
|
||||
No ambient upstream tracking — the family's no-ambient posture applies
|
||||
to maintenance too. When upstream ships something relevant (the fix
|
||||
train already being the motivating example), adoption is a deliberate
|
||||
cherry-pick recorded in the fork's delta register, evaluated by the
|
||||
same port-cost logic as any dependency change. Re-adoption of upstream
|
||||
as a *dependency* again keeps quality-read §7's bar: it must beat the
|
||||
owned, tested, contract-ahead fork on maintenance, not on novelty.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- One normative owner for every contract-pinned formula — the contract
|
||||
text — with each engine owning one pinned-equivalent implementation;
|
||||
both engines keep symmetric layering.
|
||||
- Cherry-picks from upstream stay cheap where they matter (the kept
|
||||
half); the diff between the substrate and its lineage remains
|
||||
reviewable.
|
||||
- The shipped library is panic-free without giving up the dead-man's
|
||||
switch's observable honesty.
|
||||
- The rename creates no migration machinery, and foreign-database
|
||||
leftovers have a defined, inert fate.
|
||||
|
||||
**Negative**
|
||||
|
||||
- Lineage tokens survive in internal names (`Writer`, upstream
|
||||
function names) — bounded by §3's ownership argument, invisible to
|
||||
consumers, but a fossil a future reader must understand as
|
||||
deliberate.
|
||||
- The contract formulas' arithmetic sits in engine code — small, but
|
||||
now it must be *tested into equivalence* by the contract suite (the
|
||||
verification backlog gains the re-pin rows this implies).
|
||||
- Upstream relevance is ours to notice; no release cadence, changelog,
|
||||
or `cargo update` flow will tell us.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-011](011-sqlite-substrate-fork.md) — the fork decision this ADR
|
||||
designs; its scope register and packaging are inputs.
|
||||
- [ADR-005](005-dependency-ownership.md) — the posture calculus the
|
||||
fork's maintenance assumptions inherit.
|
||||
- [ADR-001](001-crate-split.md) — the dependency direction §2
|
||||
preserves.
|
||||
- [ADR-008](008-contract-v1-pinning.md) — §6 (contract = returned
|
||||
trait; substrate API never consumer-visible), §5 (taxonomy mapping
|
||||
site).
|
||||
- [ADR-010](010-queue-semantics-depth.md) — §3a (stamps the schema
|
||||
machinery must now carry), §3 (the curve one-owner rule).
|
||||
- [ADR-003](003-sqlite-driver.md) — the bridged seam §4 keeps;
|
||||
watcher spawn fallibility lands there.
|
||||
- `docs/research/quality-read-honker-core.md` — W-1..W-4, the dead-man's
|
||||
switch verification, the bootstrap-brittleness finding, §7's
|
||||
re-adoption bar.
|
||||
- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this ADR's
|
||||
boundary realizes.
|
||||
- OQ-06 (`docs/architecture/open-questions.md`) — the resolved
|
||||
assessment this ADR follow-throughs; OQ-11 — the scaffold-time
|
||||
residue this ADR leaves (workspace wiring, register format,
|
||||
cherry-pick procedure).
|
||||
|
||||
[ADR-001]: 001-crate-split.md
|
||||
[ADR-003]: 003-sqlite-driver.md
|
||||
[ADR-008]: 008-contract-v1-pinning.md
|
||||
[ADR-010]: 010-queue-semantics-depth.md
|
||||
[ADR-011]: 011-sqlite-substrate-fork.md
|
||||
@@ -65,7 +65,7 @@ obligations live in the core spec.
|
||||
|---|---|
|
||||
| notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers |
|
||||
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) |
|
||||
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema |
|
||||
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema; the curve/stamps arithmetic is computed engine-side per [ADR-012](decisions/012-forked-substrate-design.md) §2 (equivalence with the SQLite engine pinned by the contract suite) |
|
||||
| named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; guarantee row pinned by [ADR-008](decisions/008-contract-v1-pinning.md) §7) |
|
||||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule rows in the engine-owned schema, tick re-derived (boundary advance + 64-boundary catch-up cap, honker parity), leadership via the engine's lock machinery on `__alkstore_scheduler`; outbox = helper over queues |
|
||||
| begin_tx | pool checkout + `BEGIN`, returning the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||
@@ -108,6 +108,7 @@ the record in [ADR-003](decisions/003-sqlite-driver.md).
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | pinned surface; reserved reconnect-wake channel string; `PayloadTooLarge` taxonomy variant |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule rows in the engine schema, re-derived tick, row-locked fire tx, `__alkstore_scheduler` leadership |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | job-stamped opts, equal-jitter backoff, dead-letter move, no-stranded-rows sweep, one engine-owned schema |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate (SQLite side); pg engine owns its own curve/stamps arithmetic, equivalence pinned by the contract suite |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -17,8 +17,8 @@ spec and are not restated here.
|
||||
|
||||
- Single driver: rusqlite, riding the
|
||||
[forked honker-core substrate](decisions/011-sqlite-substrate-fork.md)
|
||||
(fork of published honker-core 0.5.0's lineage at the reference
|
||||
revision),
|
||||
(`alkstore-substrate` — the vendored lineage crate,
|
||||
[ADR-012](decisions/012-forked-substrate-design.md)) with
|
||||
`bundled-sqlite` for hermetic builds. No `.so` runtime artifacts
|
||||
([ADR-003](decisions/003-sqlite-driver.md) for the driver decision;
|
||||
ownership resolved by
|
||||
@@ -45,10 +45,12 @@ spec and are not restated here.
|
||||
subscribers per poll tick, even when several commits coalesced
|
||||
inside one tick — wake is a hint; consumers re-read indexed state,
|
||||
[ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||||
- Each connection runs honker's bootstrap at open: pragmas
|
||||
- Each connection runs the substrate's bootstrap at open: pragmas
|
||||
(WAL, `synchronous=NORMAL`, busy timeout), `attach_notify`,
|
||||
`attach_honker_functions`, `bootstrap_honker_schema` (the
|
||||
alknet-filesystem POC's wiring shape).
|
||||
the substrate's function attachments and schema bootstrap (the
|
||||
alknet-filesystem POC's wiring shape). Watcher spawn is fallible in
|
||||
the substrate ([ADR-012](decisions/012-forked-substrate-design.md)
|
||||
§4); open fails if the watcher cannot start.
|
||||
|
||||
## Mapping the contract
|
||||
|
||||
@@ -57,8 +59,8 @@ spec and are not restated here.
|
||||
| notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) |
|
||||
| streams | honker's stream machinery; explicit offset saves through the tx seam |
|
||||
| queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) |
|
||||
| named locks | honker's lock machinery |
|
||||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) | |
|
||||
| named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) |
|
||||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) |
|
||||
| begin_tx | acquires the writer slot, opens `BEGIN IMMEDIATE`, returns the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||||
| handle ops | each `*_tx` op round-trips `spawn_blocking` to the same connection (thread-affinity note in [ADR-007](decisions/007-transactional-seam.md)) |
|
||||
| commit/rollback | releases the writer slot |
|
||||
@@ -86,18 +88,21 @@ spec and are not restated here.
|
||||
|
||||
## What the fork question resolved
|
||||
|
||||
The [quality read (OQ-06)](decisions/005-dependency-ownership.md) —
|
||||
The [quality read (OQ-06)](../research/quality-read-honker-core.md) —
|
||||
this engine's dependency gate — resolved 2026-10-05: ADR-005's fork
|
||||
trigger **fired** ([ADR-011](decisions/011-sqlite-substrate-fork.md)).
|
||||
The substrate is owned code forked from honker-core's lineage; the
|
||||
trigger **fired** ([ADR-011](decisions/011-sqlite-substrate-fork.md)),
|
||||
and the fork's design is pinned by
|
||||
[ADR-012](decisions/012-forked-substrate-design.md): the substrate is
|
||||
`alkstore-substrate`, a vendored contract-blind lineage crate; the
|
||||
watcher/connection architecture this spec describes is inherited
|
||||
verbatim where kept ([ADR-003](decisions/003-sqlite-driver.md)
|
||||
unchanged in architecture). The
|
||||
unchanged in architecture), with the port deltas ADR-012 §4 decides
|
||||
(fallible watcher spawn, bounded reconnect backoff, panic-free
|
||||
death-closes-subscribers). The
|
||||
[evidence](../research/quality-read-honker-core.md): published
|
||||
0.5.0's unreleased-fix-train defect class, plus ADR-010's queue depth
|
||||
requiring engine-owned queue SQL in any posture. The forked
|
||||
substrate's table family is `__alkstore_*` (ADR-010 §8's naming
|
||||
authorization).
|
||||
requiring engine-owned queue SQL in any posture. The substrate's table
|
||||
family is `__alkstore_*` (ADR-010 §8's naming authorization).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
@@ -105,7 +110,7 @@ authorization).
|
||||
|---|---|---|
|
||||
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
|
||||
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
|
||||
| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + honker-core lineage, bridged seam, inherited watcher — ownership resolved by 011 |
|
||||
| [003](decisions/003-sqlite-driver.md) | Driver | rusqlite + the forked substrate lineage, bridged seam, inherited watcher — ownership resolved by 011 |
|
||||
| [005](decisions/005-dependency-ownership.md) | Ownership | published honker-core; fork trigger fired by OQ-06 (ADR-011) |
|
||||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | data_version watcher, coalescing, death-closes-receivers |
|
||||
| [007](decisions/007-transactional-seam.md) | Tx seam | writer-slot lease, `BEGIN IMMEDIATE`, `spawn_blocking` round-trips |
|
||||
@@ -113,6 +118,7 @@ authorization).
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | `@every` specs; `__alkstore_scheduler` leadership; cron rejected |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | semantics pinned; §5/§3a realization resolved by ADR-011 (owned code) |
|
||||
| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | OQ-06's trigger fired; substrate owned, queue ops re-derived on contract v1 |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate, fidelity posture, port deltas, panic-free watcher death |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -14,12 +14,16 @@ everything else hangs off); **OQ-09 + OQ-05 resolved** (2026-10-05,
|
||||
[ADR-010](decisions/010-queue-semantics-depth.md) — the queue semantics
|
||||
track and the scheduler collapse/guarantee row); **OQ-06 resolved**
|
||||
(2026-10-05, [ADR-011](decisions/011-sqlite-substrate-fork.md) — the
|
||||
fork trigger fired on the published-artifact facts). Next: OQ-08
|
||||
(rides the now-pinned trait shape and the ADR-011 substrate fork),
|
||||
OQ-10 (versioning discipline for contract extensions; note ADR-011
|
||||
changes its substrate-side facts for SQLite — the forked crate is
|
||||
versioned in the family workspace, so the engine/core contract pairing
|
||||
is what the discipline must track).
|
||||
fork trigger fired on the published-artifact facts); **the fork
|
||||
design follow-through** (2026-10-05,
|
||||
[ADR-012](decisions/012-forked-substrate-design.md) — substrate
|
||||
boundaries, fidelity posture, port deltas; its substrate-side residue
|
||||
is OQ-11). Next: OQ-08 (rides the now-pinned trait shape and the
|
||||
ADR-011 substrate fork), OQ-10 (versioning discipline for contract
|
||||
extensions; note ADR-011 changes its substrate-side facts for SQLite —
|
||||
the forked crate is versioned in this repository's Cargo workspace, so
|
||||
the engine/core contract pairing is what the discipline must track), and
|
||||
OQ-11 (fork follow-through items — substrate-side, non-consumer-facing).
|
||||
|
||||
Resolved questions stay listed with their resolution; they are not
|
||||
deleted.
|
||||
@@ -152,7 +156,11 @@ narrowed to the pinning work its own record already scoped.)*
|
||||
**stands** (delete-on-ack; pg-boss's completed-row model is result
|
||||
storage under another name). (8) Table layout — pg engine-owned
|
||||
schema (`alkstore` default), queues are rows not tables; SQLite
|
||||
rides `_honker_*`. No new error variants (ADR-008 §5's rule).
|
||||
rides `_honker_*`. *(Item 8's SQLite half superseded 2026-10-05:
|
||||
the `_honker_*` family is re-owned as `__alkstore_*` by the fork —
|
||||
[ADR-011](decisions/011-sqlite-substrate-fork.md), designed per
|
||||
[ADR-012](decisions/012-forked-substrate-design.md).)* No new error
|
||||
variants (ADR-008 §5's rule).
|
||||
- **Cross-references**: OQ-09, OQ-06.
|
||||
|
||||
### OQ-09: Is the scheduler a first-class mechanism, or queues + `schedule()`? — **RESOLVED**
|
||||
@@ -225,15 +233,15 @@ narrowed to the pinning work its own record already scoped.)*
|
||||
- **Resolution**: open. The pg engine is natively multi-host (POC #2
|
||||
verified — no single-host assumption to remove); SQLite is
|
||||
single-machine by nature (file-backed, NFS-two-writers unsupported —
|
||||
honker's honesty posture, inherited). The unified surface must not
|
||||
pretend SQLite is multi-host. Options: per-engine capability flags
|
||||
(`Store::capabilities()`), a documented deployment matrix only
|
||||
([deployment.md](deployment.md) carries the facts), or compile-time
|
||||
knowledge only
|
||||
(a consumer choosing the SQLite engine knows). Rides the now-pinned
|
||||
contract shape ([ADR-008](decisions/008-contract-v1-pinning.md)):
|
||||
the trait constrains where capability differences can surface.
|
||||
- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md).
|
||||
honker's honesty posture, inherited by the forked substrate). The
|
||||
unified surface must not pretend SQLite is multi-host. Options:
|
||||
per-engine capability flags (`Store::capabilities()`), a documented
|
||||
deployment matrix only ([deployment.md](deployment.md) carries the
|
||||
facts), or compile-time knowledge only (a consumer choosing the
|
||||
SQLite engine knows). Rides the now-pinned contract shape
|
||||
([ADR-008](decisions/008-contract-v1-pinning.md)): the trait
|
||||
constrains where capability differences can surface.
|
||||
- **Cross-references**: OQ-04, [ADR-006](decisions/006-wake-and-delivery-contract.md), [ADR-012](decisions/012-forked-substrate-design.md) (the substrate inherits the honesty posture).
|
||||
|
||||
### OQ-10: How do engine crates track core-contract version changes?
|
||||
|
||||
@@ -251,8 +259,38 @@ narrowed to the pinning work its own record already scoped.)*
|
||||
- **Cross-references**: OQ-04 (the contract being versioned — now
|
||||
including its first post-v1 extensions, ADR-009/ADR-010), OQ-02.
|
||||
|
||||
### OQ-11: Forked-substrate follow-through — scaffold, provenance register, and cherry-pick discipline
|
||||
|
||||
- **Origin**: [ADR-012](decisions/012-forked-substrate-design.md)
|
||||
(the fork design; substrate-side residue), [ADR-011](decisions/011-sqlite-substrate-fork.md)
|
||||
- **Status**: open
|
||||
- **Priority**: medium (nothing consumer-facing rides on it; it
|
||||
resolves within the fork-scaffold task, which it does not gate)
|
||||
- **Resolution**: open. ADR-012 fixed the *design* (contract-blind
|
||||
boundary, fidelity posture, port deltas, bootstrap machinery,
|
||||
upstream-tracking stance) and also pinned the provenance register's
|
||||
location, timing, and initial contents (§1). The residue is: (1) the
|
||||
exact Cargo-workspace layout and build wiring of
|
||||
`alkstore-substrate` in this repository (crate location, feature
|
||||
gating of the `bundled-sqlite` interplay with the engine crate's own
|
||||
rusqlite dependency); (2) the provenance register's *format and
|
||||
delta-list granularity* (per-commit entries vs per-delta-class;
|
||||
whether cherry-pick records append at adoption time — location,
|
||||
timing, and initial contents are ADR-012 §1's, not re-opened here);
|
||||
(3) the cherry-pick *procedure* in practice — how a candidate
|
||||
upstream fix is evaluated, applied, and recorded so ADR-012 §6's
|
||||
deliberate-work posture stays auditable. These resolve *within* the
|
||||
fork-scaffold task (its opening section), not before it and not as a
|
||||
gate on writing it; the scaffold's build wiring depends on (1)'s
|
||||
outcome.
|
||||
- **Cross-references**: OQ-08 (the contract surface the engine maps
|
||||
the substrate under), OQ-10 (the substrate is a crate in this
|
||||
repository's Cargo workspace — its versioning interacts with the
|
||||
discipline), ADR-011, ADR-012.
|
||||
|
||||
## Deferred / Blocked
|
||||
|
||||
None currently. Every open OQ above is actionable Phase 1 architecture
|
||||
work (capability-surface shape, versioning discipline) with its
|
||||
evidence base complete — no external arrivals are being waited on.
|
||||
None currently. Every open OQ above is actionable Phase 1 work
|
||||
(capability-surface shape, versioning discipline, fork-scaffold
|
||||
follow-through) with its evidence base complete — no external arrivals
|
||||
are being waited on.
|
||||
@@ -29,7 +29,7 @@ Per [ADR-001](decisions/001-crate-split.md):
|
||||
| Crate | Contents | Driver dependencies |
|
||||
|---|---|---|
|
||||
| `alkstore` (core) | trait surface, types, error model | none (capability flags, if ever, are OQ-08's to add) |
|
||||
| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, honker-core |
|
||||
| `alkstore-sqlite` | SQLite engine ([ADR-003](decisions/003-sqlite-driver.md)) | rusqlite, `alkstore-substrate` (the forked honker-core lineage — [ADR-011](decisions/011-sqlite-substrate-fork.md), designed in [ADR-012](decisions/012-forked-substrate-design.md)) |
|
||||
| `alkstore-postgres` | Postgres engine ([ADR-004](decisions/004-postgres-driver.md)) | tokio-postgres, deadpool-postgres |
|
||||
| (mem engine, optional) | test convenience, decided at implementation ([ADR-001](decisions/001-crate-split.md)) | none |
|
||||
|
||||
@@ -53,7 +53,7 @@ Per [ADR-002](decisions/002-feature-scope.md):
|
||||
| Doc | Purpose |
|
||||
|---|---|
|
||||
| [core-contract.md](core-contract.md) | The unified trait surface: mechanisms, delivery guarantees, tx seam, naming |
|
||||
| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto honker-core/rusqlite |
|
||||
| [engine-sqlite.md](engine-sqlite.md) | SQLite engine: mapping the contract onto the forked substrate (`alkstore-substrate`)/rusqlite |
|
||||
| [engine-postgres.md](engine-postgres.md) | Postgres engine: mapping the contract onto tokio-postgres/LISTEN |
|
||||
| [queues.md](queues.md) | Queue/scheduler/outbox semantics depth (ADR-009/ADR-010 resolved) |
|
||||
| [deployment.md](deployment.md) | Host capabilities, connection budgets, deployment matrix (OQ-08) |
|
||||
@@ -66,7 +66,7 @@ Per [ADR-002](decisions/002-feature-scope.md):
|
||||
|---|---|---|
|
||||
| [001](decisions/001-crate-split.md) | Reactive-core + per-engine crates | Accepted |
|
||||
| [002](decisions/002-feature-scope.md) | Feature scope (inventory-confirmed) | Accepted |
|
||||
| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core, bridged seam | Accepted |
|
||||
| [003](decisions/003-sqlite-driver.md) | SQLite: rusqlite + honker-core lineage, bridged seam (ownership: ADR-011) | Accepted |
|
||||
| [004](decisions/004-postgres-driver.md) | Postgres: tokio-postgres + deadpool, hand-rolled LISTEN | Accepted |
|
||||
| [005](decisions/005-dependency-ownership.md) | Published libraries by default, named fork triggers | Accepted |
|
||||
| [006](decisions/006-wake-and-delivery-contract.md) | Opaque wake + re-read; notify-vs-streams guarantee split | Accepted |
|
||||
@@ -74,6 +74,8 @@ Per [ADR-002](decisions/002-feature-scope.md):
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 surface pinning (partition, TxHandle shape, wake type, reserved strings, error taxonomy) | Accepted |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse (queues + `schedule()`, `@every`-only) | Accepted |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue semantics depth (visibility, backoff, dead-letter, sweep, layout) | Accepted |
|
||||
| [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Forked substrate design (contract-blind boundary, fidelity, port deltas) | Accepted |
|
||||
|
||||
## Non-goals
|
||||
|
||||
|
||||
+20
-14
@@ -26,7 +26,9 @@ made under; the ADRs carry the WHY.
|
||||
write on both engines ([ADR-007](decisions/007-transactional-seam.md));
|
||||
rollback drops the job row with no ghosts.
|
||||
- **Exactly-once claim**: `FOR UPDATE SKIP LOCKED` (pg, POC-pinned
|
||||
under 4×4 concurrency) / honker's machinery (SQLite). At-least-once
|
||||
under 4×4 concurrency) / the forked substrate's single-statement
|
||||
claim (SQLite, re-derived on [ADR-010](decisions/010-queue-semantics-depth.md)
|
||||
§3a — [ADR-011](decisions/011-sqlite-substrate-fork.md)). At-least-once
|
||||
*work*; exactly-once *processing* needs the ack + visibility model.
|
||||
- **Wake-driven consumption**: the pg engine's default is LISTEN-driven
|
||||
claim with a re-poll safety net; SQLite's is watcher wake with
|
||||
@@ -181,11 +183,12 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects.
|
||||
stamped onto the enqueued job per
|
||||
[ADR-010](decisions/010-queue-semantics-depth.md) §3a; the queue
|
||||
mechanism's guarantees apply, no separate delivery machinery.
|
||||
- Schedule storage is engine-internal (SQLite: honker's
|
||||
`_honker_scheduler_tasks`; Postgres: the engine-owned schema) — not
|
||||
a consumer namespace; queues are names, not registered objects, so a
|
||||
schedule may target a queue nothing has claimed yet. Schedule names
|
||||
validate like queue names (non-empty, reserved-prefix rejected).
|
||||
- Schedule storage is engine-internal (SQLite: the forked substrate's
|
||||
`__alkstore_scheduler_tasks` table; Postgres: the engine-owned
|
||||
schema) — not a consumer namespace; queues are names, not registered
|
||||
objects, so a schedule may target a queue nothing has claimed yet.
|
||||
Schedule names validate like queue names (non-empty,
|
||||
reserved-prefix rejected).
|
||||
|
||||
## Namespaces / schema layout (ADR-010 §8)
|
||||
|
||||
@@ -197,11 +200,11 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects.
|
||||
table** — no per-queue tables/schemas; pg-boss's partition-per-queue
|
||||
opt-in is not inherited (one table + partial indexes matches honker's
|
||||
proven shape and keeps `queue(name)` a name, not a DDL operation).
|
||||
- **SQLite**: honker's `_honker_*` family — storage-internal
|
||||
([ADR-008](decisions/008-contract-v1-pinning.md) §4); no new tables
|
||||
minted by this design; the family's fate resolved with OQ-06 — the
|
||||
fork ([ADR-011](decisions/011-sqlite-substrate-fork.md)) re-owns the
|
||||
names (`__alkstore_*`); nothing consumer-visible changes.
|
||||
- **SQLite**: the forked substrate's `__alkstore_*` table family —
|
||||
storage-internal ([ADR-008](decisions/008-contract-v1-pinning.md)
|
||||
§4); the fork ([ADR-011](decisions/011-sqlite-substrate-fork.md))
|
||||
re-owns the names from upstream (`_honker_*`); nothing
|
||||
consumer-visible changes.
|
||||
- Partial indexes pinned identically on both engines: partial indexes
|
||||
matching the claim hot path; dead rows outside it; single clock
|
||||
source (second-precision timestamps); savepoint-guarded
|
||||
@@ -209,9 +212,10 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects.
|
||||
|
||||
## Reference material
|
||||
|
||||
- **honker's queue design** — the SQLite-side incumbent design
|
||||
reference (`/workspace/honker`; ownership resolved by
|
||||
[ADR-011](decisions/011-sqlite-substrate-fork.md) — forked, queue ops
|
||||
- **honker's queue design** — the SQLite-side lineage design reference
|
||||
(`/workspace/honker`; ownership resolved by
|
||||
[ADR-011](decisions/011-sqlite-substrate-fork.md) — forked per
|
||||
[ADR-012](decisions/012-forked-substrate-design.md), queue ops
|
||||
re-derived on this design).
|
||||
Full read: `docs/research/reference-honker-machinery.md` (schema,
|
||||
claim/visibility/retry/dead-letter mechanics with file/line cites;
|
||||
@@ -243,6 +247,8 @@ Collapsed into queues: no `Scheduler` handle, no schedule objects.
|
||||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | queue skeleton + EnqueueOpts pinned; depth was this doc's design space |
|
||||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | queues + `schedule()` + runner; `@every`-only; boundary guarantee row |
|
||||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | visibility/heartbeat rules, backoff curve, dead-letter, no-stranded-rows sweep, schema layout |
|
||||
| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite-side queue ops re-derived in owned code; `__alkstore_*` naming |
|
||||
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; fidelity posture; port deltas |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -285,7 +285,10 @@ pre-paid.
|
||||
- **Keep as-ported:** the PRAGMA block + `set_journal_mode_wal` retry
|
||||
logic; `Writer`; `Readers`; the polling watcher + `SharedUpdateWatcher`
|
||||
+ `WatcherDeathGuard` + `stat_identity` dead-man's switch (W-1
|
||||
backoff, W-2 fallible spawn applied in port); the `in_savepoint` /
|
||||
backoff, W-2 fallible spawn applied in port; the dead-man's-switch
|
||||
panic replaced by a deliberate watcher-fatal death — decided after
|
||||
this register by [ADR-012](../architecture/decisions/012-forked-substrate-design.md)
|
||||
§4, three watcher deltas total); the `in_savepoint` /
|
||||
`UnwindUndo` machinery as the mutation-transition discipline; the
|
||||
REAL-coercion arg helpers; the `notify()` scalar + notifications
|
||||
table (renamed, plus an at-attach pruning cap implementing ADR-010
|
||||
@@ -317,6 +320,10 @@ pre-paid.
|
||||
|
||||
- [ADR-011](../architecture/decisions/011-sqlite-substrate-fork.md) —
|
||||
the fork decision and packaging.
|
||||
- [ADR-012](../architecture/decisions/012-forked-substrate-design.md) —
|
||||
the fork's design follow-through: the contract-blind substrate
|
||||
boundary, the API fidelity posture, and this document's port notes
|
||||
(W-1..W-4 and the dead-man's-switch panic) decided per item.
|
||||
- [ADR-005](../architecture/decisions/005-dependency-ownership.md) —
|
||||
posture row for honker-core annotated (trigger fired); the calculus
|
||||
is applied, not renegotiated.
|
||||
|
||||
Reference in new issue
Block a user