alkstore/docs/architecture/decisions/012-forked-substrate-design.md

ADR-012: Forked substrate design — contract-blind boundary, fidelity posture, port deltas

Status

Accepted (2026-10-05, Phase 1 — the design decisions ADR-011's fork requires; the follow-through of OQ-06's resolution)

Context

ADR-011 decided that the fork happens and its scope (keep / re-derive / drop). Deciding it surfaced structural questions the fork record deliberately did not settle — where the forked crate lives and what it is called, which layer carries contract-pinned semantics, how upstream diffs stay cheap, which of the read's port notes fire, and how a fork with no upstream dependency to update tracks its lineage. None of these were blockers for the fork trigger, but all of them shape the engine's dependency graph and the fork's maintenance cost — they are architecture, and they are decidable now, from decided material:

1. The dependency direction is fixed (ADR-001): engines depend on core, never the reverse. The substrate sits below the engine.
2. The substrate's Rust API is engine-internal. The contract is the trait the constructor returns (ADR-008 §6); no substrate type or name reaches a consumer signature.
3. Contract semantics are pinned engine-generically — the equal-jitter curve (ADR-010 §3), the opts-stamping resolution rule (§3a), schedule boundary math (ADR-009 §4) — and must compute identically on both engines. One engine carrying its copy inside a vendored lineage crate splits each pinned formula across codebases with different ownership characters.
4. The family bans panics in library code; upstream's dead-man's switch panics the watcher thread on db-file replacement (lib.rs:919 at the reference revision). Verified in source: the panic is observed only through WatcherDeathGuard — the guard's Drop fires on thread exit or panic and closes every subscriber (the contract-pinned failure surface). The observable semantics do not depend on the panic.
5. No alkstore databases exist. The crate is unreleased; the _honker_* → __alkstore_* rename therefore has no migration surface and creates one only if we invent it.

Decision

1. Crate identity and location

The forked substrate is alkstore-substrate — a vendored workspace-family crate in this repository, a path dependency of alkstore-sqlite, not published (ADR-011's packaging, now named). At fork-scaffold time the crate carries: the dual license files upstream ships (MIT OR Apache-2.0), and a provenance register (provenance.md at the crate root) recording upstream identity — checkout path, commit f4e53c6, license — and the delta list below (AGENTS.md §3's recording duty).

2. The substrate is contract-blind storage machinery

The substrate depends on nothing of ours — not on alkstore (the core crate), on the contract types, or on the error taxonomy. Its API takes and returns primitives: connections, names, stamp values (visibility_timeout_s, backoff_base_s, dead_letter_retention_s, max_attempts, priority, timestamps), payload strings. The engine crate layer resolves contract semantics into primitive arguments: EnqueueOpts-over-QueueOpts resolution, the equal-jitter curve's arithmetic, the schedule fire's boundary math, and the mapping of substrate outcomes into the contract taxonomy (ADR-008 §5 / ADR-010 §9's rules).

Why the boundary sits there, not inside the substrate:

• Dependency direction: a vendored third-party-lineage crate must not acquire a dependency on our contract crate (and the mirror — core depending on the substrate — inverts ADR-001 outright).
• One normative owner per pinned formula — the contract text, not a substrate copy: the curve, the stamps-resolution rule, and boundary math are contract-pinned to compute identically on both engines (ADR-010 §3). The contract is the formula's single normative owner; each engine owns one implementation of it, pinned to identical outputs by the contract suite (the verification backlog in core-contract.md). A substrate copy would add a second normative-adjacent definition in a codebase whose upstream does not know the contract exists.
• Diff reviewability: the substrate stays as close to its lineage as the port allows; contract semantics are exactly the re-derivation surface (ADR-011), and they are new code with contract-derived names either way.

Dependency graph (downward only):

        consumers
            │
        alkstore  (core: traits, types, errors — no drivers)
        ────────▲────────
      alkstore-sqlite       alkstore-postgres
            │              (no substrate)
   alkstore-substrate  (vendored lineage; path dep; unpublished)

3. API fidelity posture: keep the kept half's names

Where the fork scope keeps upstream machinery (ADR-011's inherited list), the fork keeps upstream's module structure and internal function names — including names carrying lineage tokens (Writer, Readers, run_poll_loop, the in_savepoint machinery, the lock/stream/notify functions). Renames are confined to: crate identity, the table family (__alkstore_*, ADR-011), and the hygiene deltas ADR-011 names. The re-derived half (ADR-011's list) is new code; its functions take contract-derived names naturally.

Rationale: the fork's ongoing value includes cheap cherry-picks of future upstream fixes — the calculus ADR-011 inherited from ADR-005 assumes the inherited half still tracks its lineage. Mechanical token renames turn every future cherry-pick noisy; internal oddness survives a rename only to be paid for forever. The names are engine-internal and bounded by the fork's ownership.

4. Port deltas — the quality read's notes, decided

• W-1 (reconnect backoff) — applied. The watcher reconnect loop gains bounded backoff; a vanished db file no longer produces ~1000 open attempts per second.
• W-2 (watcher spawn panics) — applied. Watcher spawn becomes fallible; the engine surfaces the failure at open time (a store whose watcher could not start is not a store — open fails with the taxonomy's Database, detail in the source chain).
• Dead-man's switch (no-panics tension) — panic replaced by a deliberate watcher-fatal death. On db-file identity change the watcher logs the precise diagnostic and exits through the ordinary death path. WatcherDeathGuard closes every subscriber exactly as it does today for thread panic or exit — the contract-pinned surface ("consumers see the close, never a silent hang") is unchanged; the unwind is what goes away.
• W-4 (watcher connection opens RW) — keep RW (deliberate inheritance). The read found no proof a RO watcher is safe on busy non-WAL paths; our posture is WAL-always anyway. Revisit only if a read-only/mode knob ever appears as an engine option.
• W-3 (data_version u32 wrap) — not actionable, recorded in the fork's notes.
• The substrate stays sync. ADR-011's "ported to the family standard" means the discipline deltas (no comments, panics out of library code per the deltas above, no unwrap()/expect() outside tests) — not an asyncification port. The bridged seam (ADR-003) remains the async story; a substrate that grew tokio would forfeit the diff-reviewability the fork's economics rest on.

5. Bootstrap and schema machinery

• The fork owns the bootstrap; append-column migrations stay (the read's §4 scale verdict). The concurrent-bootstrap duplicate-column race swallow is re-keyed: on ALTER TABLE failure, verify the column via pragma_table_info — present ⇒ benign race swallowed, absent ⇒ propagate. Upstream's error-string matching (.contains("duplicate column")) is not inherited.
• The stamp columns and claimed_at are added by the re-derivation on contract v1 (ADR-010 §3a's realization; superseding the "no schema change needed" expectation recorded in ADR-010 §8 — annotated there).
• No online rename migration. The substrate bootstraps __alkstore_* fresh. A file carrying leftover upstream tables gets inert, ignored _honker_* orphans (documented; never touched, never read) — no detection, no export path, no migration machinery, because no alkstore database under the old names exists to migrate.

6. Upstream tracking without a dependency

No ambient upstream tracking — the family's no-ambient posture applies to maintenance too. When upstream ships something relevant (the fix train already being the motivating example), adoption is a deliberate cherry-pick recorded in the fork's delta register, evaluated by the same port-cost logic as any dependency change. Re-adoption of upstream as a dependency again keeps quality-read §7's bar: it must beat the owned, tested, contract-ahead fork on maintenance, not on novelty.

Consequences

Positive

• One normative owner for every contract-pinned formula — the contract text — with each engine owning one pinned-equivalent implementation; both engines keep symmetric layering.
• Cherry-picks from upstream stay cheap where they matter (the kept half); the diff between the substrate and its lineage remains reviewable.
• The shipped library is panic-free without giving up the dead-man's switch's observable honesty.
• The rename creates no migration machinery, and foreign-database leftovers have a defined, inert fate.

Negative

• Lineage tokens survive in internal names (Writer, upstream function names) — bounded by §3's ownership argument, invisible to consumers, but a fossil a future reader must understand as deliberate.
• The contract formulas' arithmetic sits in engine code — small, but now it must be tested into equivalence by the contract suite (the verification backlog gains the re-pin rows this implies).
• Upstream relevance is ours to notice; no release cadence, changelog, or cargo update flow will tell us.

References

• ADR-011 — the fork decision this ADR designs; its scope register and packaging are inputs.
• ADR-005 — the posture calculus the fork's maintenance assumptions inherit.
• ADR-001 — the dependency direction §2 preserves.
• ADR-008 — §6 (contract = returned trait; substrate API never consumer-visible), §5 (taxonomy mapping site).
• ADR-010 — §3a (stamps the schema machinery must now carry), §3 (the curve one-owner rule).
• ADR-003 — the bridged seam §4 keeps; watcher spawn fallibility lands there.
• docs/research/quality-read-honker-core.md — W-1..W-4, the dead-man's switch verification, the bootstrap-brittleness finding, §7's re-adoption bar.
• engine-sqlite.md — the engine spec this ADR's boundary realizes.
• OQ-06 (docs/architecture/open-questions.md) — the resolved assessment this ADR follow-throughs; OQ-11 — the scaffold-time residue this ADR leaves (workspace wiring, register format, cherry-pick procedure).