alkstore/docs/architecture/decisions/012-forked-substrate-design.md

# ADR-012: Forked substrate design — contract-blind boundary, fidelity posture, port deltas

## Status

Accepted (2026-10-05, Phase 1 — the design decisions ADR-011's fork
requires; the follow-through of OQ-06's resolution). **Amended by
[ADR-013](013-fold-substrate-into-sqlite.md) (2026-10-05)**: §1's
`alkstore-substrate` crate identity is superseded — the fork folds
into `alkstore-sqlite` as a bounded module subtree; §2–§6 are retained
verbatim under that packaging.

## Context

[ADR-011] decided *that* the fork happens and its scope (keep /
re-derive / drop). Deciding it surfaced structural questions the fork
record deliberately did not settle — where the forked crate lives and
what it is called, which layer carries contract-pinned semantics, how
upstream diffs stay cheap, which of the read's port notes fire, and how
a fork with no upstream dependency to update tracks its lineage. None
of these were blockers for the fork trigger, but all of them shape the
engine's dependency graph and the fork's maintenance cost — they are
architecture, and they are decidable now, from decided material:

1. **The dependency direction is fixed** ([ADR-001]): engines depend on
   core, never the reverse. The substrate sits *below* the engine.
2. **The substrate's Rust API is engine-internal.** The contract is the
   trait the constructor returns ([ADR-008] §6); no substrate type or
   name reaches a consumer signature.
3. **Contract semantics are pinned engine-generically** — the
   equal-jitter curve ([ADR-010] §3), the opts-stamping resolution rule
   (§3a), schedule boundary math (ADR-009 §4) — and must compute
   identically on both engines. One engine carrying its copy inside a
   vendored lineage crate splits each pinned formula across codebases
   with different ownership characters.
4. **The family bans panics in library code**; upstream's dead-man's
   switch panics the watcher thread on db-file replacement
   (`lib.rs:919` at the reference revision). Verified in source: the
   panic is *observed* only through `WatcherDeathGuard` — the guard's
   `Drop` fires on thread exit or panic and closes every subscriber
   (the contract-pinned failure surface). The observable semantics do
   not depend on the panic.
5. **No alkstore databases exist.** The crate is unreleased; the
   `_honker_*` → `__alkstore_*` rename therefore has no migration
   surface and creates one only if we invent it.

## Decision

### 1. Crate identity and location

*(Superseded by [ADR-013](013-fold-substrate-into-sqlite.md): there is
no `alkstore-substrate` crate. The forked substrate is a bounded
module subtree of `alkstore-sqlite` — `alkstore-sqlite/src/substrate/`
— with the provenance register and bundled dual-license notice carried
in-tree. Retained below as the original record.)*

The forked substrate is **`alkstore-substrate`** — a vendored
workspace-family crate in this repository, a path dependency of
`alkstore-sqlite`, not published ([ADR-011]'s packaging, now named).
At fork-scaffold time the crate carries: the dual license files
upstream ships (MIT OR Apache-2.0), and a **provenance register**
(`provenance.md` at the crate root) recording upstream identity —
checkout path, commit `f4e53c6`, license — and the delta list below
(AGENTS.md §3's recording duty).

### 2. The substrate is contract-blind storage machinery

**The substrate depends on nothing of ours** — not on `alkstore` (the
core crate), on the contract types, or on the error taxonomy. Its API
takes and returns primitives: connections, names, stamp values
(`visibility_timeout_s`, `backoff_base_s`, `dead_letter_retention_s`,
`max_attempts`, `priority`, timestamps), payload strings. The **engine
crate layer** resolves contract semantics into primitive arguments:
`EnqueueOpts`-over-`QueueOpts` resolution, the equal-jitter curve's
arithmetic, the schedule fire's boundary math, and the mapping of
substrate outcomes into the contract taxonomy ([ADR-008] §5 / ADR-010
§9's rules).

Why the boundary sits there, not inside the substrate:

- **Dependency direction**: a vendored third-party-lineage crate must
  not acquire a dependency on our contract crate (and the mirror —
  core depending on the substrate — inverts [ADR-001] outright).
- **One normative owner per pinned formula — the contract text, not a
  substrate copy**: the curve, the stamps-resolution rule, and boundary
  math are contract-pinned to compute identically on both engines
  ([ADR-010] §3). The contract is the formula's single normative
  owner; each engine owns one *implementation* of it, pinned to
  identical outputs by the contract suite (the verification backlog in
  [core-contract.md](../core-contract.md)). A substrate copy would add
  a second normative-adjacent definition in a codebase whose upstream
  does not know the contract exists.
- **Diff reviewability**: the substrate stays as close to its lineage
  as the port allows; contract semantics are exactly the re-derivation
  surface ([ADR-011]), and they are new code with contract-derived
  names either way.

Dependency graph (downward only; no substrate crate — the forked
machinery is in-tree inside `alkstore-sqlite`, per
[ADR-013](013-fold-substrate-into-sqlite.md)):

```text
        consumers
            │
        alkstore  (core: traits, types, errors — no drivers)
        ────────▲────────
   alkstore-sqlite        alkstore-postgres
   (src/substrate/ —      (no substrate)
   the forked lineage
   machinery, in-tree)
```

### 3. API fidelity posture: keep the kept half's names

Where the fork scope *keeps* upstream machinery ([ADR-011]'s inherited
list), the fork keeps upstream's module structure and internal function
names — including names carrying lineage tokens (`Writer`, `Readers`,
`run_poll_loop`, the `in_savepoint` machinery, the lock/stream/notify
functions). Renames are confined to: crate identity *(a slot the
  fold — [ADR-013](013-fold-substrate-into-sqlite.md) — dissolves;
  there is no crate to rename)*, the table family
  (`__alkstore_*`, [ADR-011]), and the hygiene deltas [ADR-011] names.
The **re-derived** half ([ADR-011]'s list) is new code; its functions
take contract-derived names naturally.

Rationale: the fork's ongoing value includes cheap cherry-picks of
future upstream fixes — the calculus ADR-011 inherited from ADR-005
assumes the inherited half still *tracks* its lineage. Mechanical token
renames turn every future cherry-pick noisy; internal oddness survives
a rename only to be paid for forever. The names are engine-internal and
bounded by the fork's ownership.

### 4. Port deltas — the quality read's notes, decided

- **W-1 (reconnect backoff) — applied.** The watcher reconnect loop
  gains bounded backoff; a vanished db file no longer produces ~1000
  open attempts per second.
- **W-2 (watcher spawn panics) — applied.** Watcher spawn becomes
  fallible; the engine surfaces the failure at open time (a store whose
  watcher could not start is not a store — open fails with the
  taxonomy's `Database`, detail in the source chain).
- **Dead-man's switch (no-panics tension) — panic replaced by a
  deliberate watcher-fatal death.** On db-file identity change the
  watcher logs the precise diagnostic and exits through the ordinary
  death path. `WatcherDeathGuard` closes every subscriber exactly as it
  does today for thread panic or exit — the contract-pinned surface
  ("consumers see the close, never a silent hang") is unchanged; the
  unwind is what goes away.
- **W-4 (watcher connection opens RW) — keep RW (deliberate
  inheritance).** The read found no proof a RO watcher is safe on
  busy non-WAL paths; our posture is WAL-always anyway. Revisit only if
  a read-only/mode knob ever appears as an engine option.
- **W-3 (`data_version` u32 wrap) — not actionable**, recorded in the
  fork's notes.
- **The substrate stays sync.** ADR-011's "ported to the family
  standard" means the discipline deltas (no comments, panics out of
  library code per the deltas above, no `unwrap()`/`expect()` outside
  tests) — **not** an asyncification port. The bridged seam
  ([ADR-003]) remains the async story; a substrate that grew tokio
  would forfeit the diff-reviewability the fork's economics rest on.

### 5. Bootstrap and schema machinery

- The fork owns the bootstrap; **append-column migrations stay** (the
  read's §4 scale verdict). The concurrent-bootstrap duplicate-column
  race swallow is re-keyed: on `ALTER TABLE` failure, verify the column
  via `pragma_table_info` — present ⇒ benign race swallowed, absent ⇒
  propagate. Upstream's error-*string* matching
  (`.contains("duplicate column")`) is not inherited.
- The **stamp columns and `claimed_at`** are added by the re-derivation
  on contract v1 ([ADR-010] §3a's realization; superseding the "no
  schema change needed" expectation recorded in ADR-010 §8 — annotated
  there).
- **No online rename migration.** The substrate bootstraps
  `__alkstore_*` fresh. A file carrying leftover upstream tables gets
  inert, ignored `_honker_*` orphans (documented; never touched, never
  read) — no detection, no export path, no migration machinery, because
  no alkstore database under the old names exists to migrate.

### 6. Upstream tracking without a dependency

No ambient upstream tracking — the family's no-ambient posture applies
to maintenance too. When upstream ships something relevant (the fix
train already being the motivating example), adoption is a deliberate
cherry-pick recorded in the fork's delta register, evaluated by the
same port-cost logic as any dependency change. Re-adoption of upstream
as a *dependency* again keeps quality-read §7's bar: it must beat the
owned, tested, contract-ahead fork on maintenance, not on novelty.

## Consequences

**Positive**

- One normative owner for every contract-pinned formula — the contract
  text — with each engine owning one pinned-equivalent implementation;
  both engines keep symmetric layering.
- Cherry-picks from upstream stay cheap where they matter (the kept
  half); the diff between the substrate and its lineage remains
  reviewable.
- The shipped library is panic-free without giving up the dead-man's
  switch's observable honesty.
- The rename creates no migration machinery, and foreign-database
  leftovers have a defined, inert fate.

**Negative**

- Lineage tokens survive in internal names (`Writer`, upstream
  function names) — bounded by §3's ownership argument, invisible to
  consumers, but a fossil a future reader must understand as
  deliberate.
- The contract formulas' arithmetic sits in engine code — small, but
  now it must be *tested into equivalence* by the contract suite (the
  verification backlog gains the re-pin rows this implies).
- Upstream relevance is ours to notice; no release cadence, changelog,
  or `cargo update` flow will tell us.

## References

- [ADR-011](011-sqlite-substrate-fork.md) — the fork decision this ADR
  designs; its scope register and packaging are inputs (packaging since
  amended by [ADR-013](013-fold-substrate-into-sqlite.md)).
- [ADR-013](013-fold-substrate-into-sqlite.md) — the packaging
  amendment (substrate folded into `alkstore-sqlite`; this ADR's
  §3–§6 retained verbatim, §2 with its enforcement re-sited).
- [ADR-005](005-dependency-ownership.md) — the posture calculus the
  fork's maintenance assumptions inherit.
- [ADR-001](001-crate-split.md) — the dependency direction §2
  preserves.
- [ADR-008](008-contract-v1-pinning.md) — §6 (contract = returned
  trait; substrate API never consumer-visible), §5 (taxonomy mapping
  site).
- [ADR-010](010-queue-semantics-depth.md) — §3a (stamps the schema
  machinery must now carry), §3 (the curve one-owner rule).
- [ADR-003](003-sqlite-driver.md) — the bridged seam §4 keeps;
  watcher spawn fallibility lands there.
- `docs/research/quality-read-honker-core.md` — W-1..W-4, the dead-man's
  switch verification, the bootstrap-brittleness finding, §7's
  re-adoption bar.
- [engine-sqlite.md](../engine-sqlite.md) — the engine spec this ADR's
  boundary realizes.
- OQ-06 (`docs/architecture/open-questions.md`) — the resolved
  assessment this ADR follow-throughs; OQ-11 — the scaffold-time
  residue this ADR leaves (workspace wiring, register format,
  cherry-pick procedure) — since resolved:
  [ADR-013](013-fold-substrate-into-sqlite.md) (item 1, dissolved)
  and [ADR-018](018-provenance-register-and-cherry-picks.md) (items
  2–3, the register and the procedure).

[ADR-001]: 001-crate-split.md
[ADR-003]: 003-sqlite-driver.md
[ADR-008]: 008-contract-v1-pinning.md
[ADR-010]: 010-queue-semantics-depth.md
[ADR-011]: 011-sqlite-substrate-fork.md