alkstore/docs/research/phase-0.md

---

status: draft last_updated: 2026-10-03 (initial draft + interface finding: honker-rs surface read as the unified-API candidate; pgboss-rs LISTEN/NOTIFY-absence folded into the driver/queue/ownership questions)

alkstore — Phase 0 (Exploration)

This document captures Phase 0 (Exploration) for the alkstore crate: vision, guiding principles, prior art, and the open-question register (OQ-ST-01..NN). Phase 0's objective per docs/sdd_process.md: capture vision and guiding principles; research options; validate approaches; converge on a recommended approach. This is the initial draft — none of the questions below are settled, and there is no POC register yet.

Context for why this crate starts now: alkblobs (/workspace/@alkdev/alkblobs — spec + POCs only, paused mid-planning) hit repeated circular hedging in its Phase 0, and a root cause was that the storage substrate it deploys onto was itself disjoint and fuzzy — a repo pattern with a default in-memory adapter across the alk* ecosystem, cache-invalidation patches in hot paths, non-invalidated caches where delay was tolerable, no single definition of "how does a change in the database become visible to other processes/connections?" alkblobs paused partly to let this crate answer that first. alkstore is the attempt to make that substrate real once, so downstream stores don't re-derive it.

Vision and guiding principles

One sentence (draft): one reactive store interface over SQLite and Postgres — durable pub/sub notify, queues, streams, and the transactional integration (write + enqueue in one transaction) that honker delivers on SQLite — with the Postgres side building on the natively-available machinery (pg_notify/LISTEN, and the pgboss job-queue schema family) rather than emulating it.

The honker relationship. /workspace/honker (read-only reference; alpha-quality per its own README, MIT/Apache-2.0 dual) adds Postgres-style NOTIFY/LISTEN semantics to SQLite without a broker: durable at-least-once queues with retries/delay/priority/visibility timeouts/dead-letter, durable streams with per-consumer offsets, cron/@every scheduling, named locks, rate limits, transactional outbox — all as INSERTs inside the caller's transaction, with the cross-process wake delivered by a shared watcher that polls PRAGMA data_version (default 1 ms → single-digit-ms delivery) and re-reads indexed state after every wake. Its own Prior Art section names the

The interface finding (2026-10-03, from the four honker.dev guides + packages/honker-rs/src/lib.rs, the Rust binding): honker's surface is already the unified-interface candidate. Its Rust binding exposes exactly the surface this crate wants, engine-clean:

• db.queue(name, QueueOpts) → enqueue / enqueue_tx / claim_one / claim_batch / ack_batch / cancel / get_job / sweep_expired / claim_waker, with job.ack / retry / fail / heartbeat and EnqueueOpts {delay, priority, max_attempts, expires, ...} — semantically the pg-boss model (visibility timeouts, retries, dead-letter via move-to-_honker_dead), not a LISTEN-emulation.
• db.stream(name) → publish / publish_tx / publish_with_key / read_since / read_from_consumer / save_offset(_tx) / get_offset / subscribe(consumer) — offsets are explicit, transaction-aware (save_offset_tx) for the exactly-once-within-a-business-tx shape, and replay-on-reconnect is the default.
• db.notify(channel, payload) / notify_tx / db.listen(channel) — the pg_notify-analogue fire-and-forget signal layer ("fire-and-forget, no replay, no guarantees" — streams are the durable cousin), listener starts from MAX(id) at attach, no historical replay.
• db.scheduler() → add/pause/resume/update/list/remove/tick/run — cron + @every enqueueing into named queues, leader-elected via advisory lock with TTL heartbeat, missed-boundary catch-up.
• db.outbox(name) — the transactional outbox helper (enqueue + run_once delivery worker).
• db.try_lock / try_rate_limit / save_result / get_result / sweep_results — the coordination/adjacent-tools surface.

The implication flips the framing of the unified-API work: it is not "invent a shape both engines fit" — it is "this shape both engines can fit" (pg-boss's queue model is already the native Postgres tooling model; streams/offsets and notify have direct Postgres counterparts) and the work is pinning which parts of the shape are the crate's contract — the delivery-guarantee differences honker's own guide documents per-binding (auto-checkpoint cadence vs manual offset save; the processing-guarantees table) are exactly the seams a single-crate version must clean up. Honker-rs is the concrete prior art for that pinning exercise. Postgres side: pg_notify gives fast triggers with no retry or visibility semantics; pg-boss/Oban are the durable-layer gold standards — "If you already run Postgres, use the Postgres tools."

The goal is not "port honker to Postgres." The goal is the interface: one store API whose consumer code (queues, streams, notify) looks the same whether the backing engine is SQLite or Postgres, while each engine uses its own native wake/delivery story under the hood. The honker docs recommend pgboss + pg_notify for the Postgres equivalent — that recommendation is the design brief for this crate's Postgres engine — and honker's Rust binding (honker-rs, v0.5.0) is the concrete candidate for what that unified interface literally looks like (§Interface finding).

Consumer shape (from the paused alkblobs planning): any crate that today uses the ecosystem's repo-pattern + in-memory adapter should be able to swap in an alkstore-backed engine and get durability + true cross-process/cross-instance reactivity. That means the reactive surface must compose with client-side caching: a subscriber that also holds a cache can invalidate on notification instead of re-querying or re-polling — the hot-path pattern the ecosystem already uses, given a real invalidation source.

Guiding principles:

1. One interface, two engines, native underneath. The abstraction layer unifies the consumer-visible features; the engines stay dialects, not two emulations of one dialect. SQLite follows honker's design (queue in the same file, same transaction, watcher-based wake); Postgres follows pg-boss' design (schema-based job tables + pg_notify-driven wake). A "lowest common denominator" unification (both sides polling, both sides emulating LISTEN) is explicitly the failure mode to avoid — it would re-create the fuzziness this crate exists to remove.
2. Transactional local-adjacency is the load-bearing property. Honker's core claim: enqueue/publish/notify in the same transaction as the business write; rollback drops both. The unified surface must preserve this on any engine, because the ecosystem's repo pattern assumes it (a business write that loses its side-effect notification is the dual-write problem honker names).
3. Ownership of the whole stack. honker is third-party; pgboss-rs is third-party. Whether any of them are adopted, forked, or used as schema/design reference only is a deliberate per-question decision — not inherited by adjacency. The workspace precedent is the targeted fork (alksocks' fast-socks5 extraction: adopt the design, own the code, port to our conventions).
4. Substrate-agnostic consumer API, engine-specific setup. A consumer opens a Store from a connection string / file path and gets the same trait surface. Which engine is behind what can vary (per-deployment config), but consumer code must not branch on engine type.
5. No panics, tokio, thiserror, lean base crate, feature-gated optional engines — family-standard, pre-committed (AGENTS.md).

The driver conflict (Phase 0's central tension)

The immediate design fork, flagged by the user:

• The alkblobs POCs used tokio-postgres + deadpool-postgres (validated in poc-postgres-kv-findings.md, poc-pglo-findings.md — including Large Objects work).
• pgboss-rs (/workspace/pgboss-rs) uses sqlx (sqlx Postgres runtime-tokio).
• honker-core uses rusqlite.

A single crate with both engines means a driver decision, and the reactivity story is entangled with it:

• pgboss-rs currently has no LISTEN/NOTIFY at all (verified 2026-10-03 against the checkout — src/ contains no LISTEN/NOTIFY usage; consumption is fetch_job polling). The node original relies on pg-boss's own maintenance/polling; the port did not pick up a push channel. So even "use pgboss for the queue" does not deliver reactivity — LISTEN/NOTIFY wiring would be new work either way, and the driver choice determines whose LISTEN plumbing (sqlx's PgListener is built-in; tokio-postgres uses its Connection notifications).
• Honker's reactivity on SQLite is a watcher polling PRAGMA data_version — a fundamentally different mechanism from LISTEN/NOTIFY. The unified reactive trait must abstract over both without collapsing to the polling behavior of the weaker side.

Two corrections to the original framing (2026-10-03, after reading honker-rs and the honker.dev guides):

1. The honker-rs interface is largely driver-independent. The Queue/Stream/notify/scheduler surface (§Interface finding) speaks in domain terms (channels, offsets, job ids, visibility timeouts), not driver terms. What is driver-coupled is the transactional seam: enqueue_tx/publish_tx/save_offset_tx and the extension's notify-in-transaction all assume the caller can hand the engine a live transaction handle from its driver. The unified trait's transactional seam is therefore the driver-sensitive design point — and it interacts with the engines' different transaction models (SQLite: single writer, synchronous; Postgres: interactive transactions over a pool, transaction-scoped LISTEN).
2. **honker-rs is sync (std threads + blocking iterators) — parking_lot
   • rusqlite, no tokio.** The family standard is tokio-async. So even the SQLite side is a port-and-adapt (sync → async), not an adopt; and the driver question for SQLite is also entangled with whether tokio-native sqlite drivers (sqlx sqlite) change the queue/wake machinery's shape versus rusqlite-in-a-pool.

This tension is OQ-ST-04 below. It is not resolved by "pgboss is well written so start there" — that is exactly the inherited-assumption shape the SDD process flags. What pgboss-rs genuinely offers (schema DDL, job states, retry semantics, the node-compatible API) is design reference regardless of driver. The reactivity gap is the concrete, verified difference between pgboss-rs and what this crate needs — whether forked or re-derived, the push channel is work this crate builds itself (OQ-ST-05).

Prior art

Notes below are from reading the checkouts on 2026-10-03; both external projects are read-only references. Provenance/licensing gets recorded per AGENTS.md §3 when code is adopted, not while only reading.

honker — the SQLite-side template

/workspace/honker (SQLite extension + bindings). What matters for this crate:

• The full feature set to match on Postgres (its §What It Does): notify/listen across processes, durable at-least-once queues (retries, delayed jobs, priority, visibility timeouts, dead-letter rows, result storage), durable streams with per-consumer offsets, cron/@every scheduling, named locks, rate limits, transactional outbox helpers. Deliberately excluded there: workflow DAGs, task chains/chords, multi-writer replication, cross-machine locking — scope line likely inherited, to be confirmed.
• The wake mechanism — PRAGMA data_version polling watcher (default 1 ms; raise for idle CPU), re-read indexed state after wake, overtriggering on purpose ("one indexed SELECT is cheap; a missed wake is a correctness bug."). Optional kernel-events and WAL shared-memory backends exist in source builds.
• Single-machine honesty — file-backed, one host; NFS-two-writers explicitly not supported. This posture needs an explicit Postgres counterpart (multi-host is Postgres' normal case, so the interface must not bake SQLite's single-host assumption into the shared surface).
• The transactional enqueue shape — every feature is an INSERT inside the caller's transaction. This is the pattern the unified API must keep visible and cheap.
• The honker-rs binding is the concrete interface prior art (v0.5.0, packages/honker-rs, read 2026-10-03): the full surface per §Interface finding. Notable honest limitations documented by its own guides — the per-binding processing-guarantees table (auto-checkpoint cadence vs manual offset save; several bindings "may persist an offset on a cadence... without knowing whether downstream application work committed"), the Node reverse-order consumer-checkpoint bug, per- binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++ missing typed pruning) — are exactly the seams a single-crate version designed-for-the-contract from day one can clean up. Its sync-only shape (std threads, blocking iterators, no tokio) is a port-and-adapt constraint, not an adopt candidate as-is.

pgboss-rs — the Postgres queue family reference

/workspace/pgboss-rs (v0.1.0-rc6, MIT/Apache-2.0 dual). Ported from node pg-boss: builder-based queue/job API, retry/delay/priority/ singleton/dead-letter concepts, sqlx 0.8, schema-scoped DDL.

• Verified gap (2026-10-03): no LISTEN/NOTIFY anywhere in src/ — consumption is polling fetch_job. Any push-reactivity is new work, not an adoption freebie. This is the substantive difference between pgboss-rs and what alkstore needs: regardless of fork vs re-derive, reactivity is ours to build on the Postgres side either way.
• Its value as reference: the pg-boss schema family (job states, maintenance/dead-letter behavior) is battle-tested against real Postgres semantics — worth borrowing as design, independent of the driver decision.
• The node original (pg-boss) is the upstream of record for semantics the port may have dropped; compare against it when adopting queue semantics.

Honker's Postgres-side recommendation

The honker README's own posture: if you run Postgres, use the Postgres tools. pg_notify + pgboss is the recommended assembly. The design brief: the queue machinery from the pg-boss family, the push semantics from LISTEN/NOTIFY, the unified API shape from honker's Rust binding.

The alk* repo pattern — what this crate replaces

The ecosystem's current shape: a repository trait with a default in-memory adapter; cache-invalidation wiring in hot paths; uninvalidated (non-reactive) caches where delay was acceptable; each project composing these slightly differently. No persistence-backed reactive substrate exists in the family — alkblobs was the first project to try to plan against one, found it missing, and paused. This crate's reason to exist is precisely that that substrate should exist once, well, instead of per-project approximations.

alkcall — the substrate (not a dependency of the store layer)

/workspace/@alkdev/alkcall (pure protocol crate, no transport). The alk* crates (alktty, alktunnels, alksocks) are its consumers; a future alkstore ops/protocol surface (if this crate ever exposes store access over alkcall channels) rides the same substrate. Like the alkblobs split (store layer stays substrate-free), the store layer here stays alkcall-free; any networked surface is an ops module/sibling concern and a separate decision.

Open Questions

Register in docs/research/phase-0.md; IDs OQ-ST-NN (stable, append only). Promotion target: Phase 1 docs/architecture/open-questions.md.

OQ-ST-01: Scope boundary — which honker features are in-scope?

Honker's feature list (queues, streams, notify, scheduling, locks, rate limits, outbox helpers) is large; per-feature scope decisions don't exist yet. Also undetermined: the exclusion lines (honker deliberately excludes DAGs, task chains/chords, multi-writer replication, distributed locking).

Not yet decidable without working through the concrete consumers (the paused alkblobs crates and alkfs planning) — per-feature need hasn't been articulated. Blocked on a consumer-driven inventory pass.

OQ-ST-02: Crate scope — one store crate, or reactive-core + engines?

Options include: single crate with feature-gated engines (the alk* feature-gate pattern); a core trait crate + per-engine crates; engine crates consuming a thin core. The answer constrains the driver decision (OQ-ST-04) and the base-crate-lean invariant.

The use case isn't concrete yet (no engine code written, no consumer wired). Deferred(scope) — concrete use case: the first engine implementation would force this shape.

OQ-ST-03: Driver story — sqlx, tokio-postgres, or per-engine drivers?

The named tension (§The driver conflict), restated as the decision:

• One driver across engines: sqlx (both sqlite + postgres native support, one API — but the alkblobs POC evidence is tokio-postgres) vs tokio-postgres per-engine (sqlite story unclear — tokio-postgres is pg-only; rusqlite is the sqlite native).
• Per-engine drivers under a unified trait: tokio-postgres + deadpool-postgres (POC-validated in alkblobs findings) + rusqlite (honker's choice, so honker's SQLite machinery ports cleanly).
• Adopt/fork pgboss-rs: brings sqlx along where the queue lives.

Honest unknowns worth surfacing: does sqlx support SQLite data_version/extension-style machinery equally well? Does a unified trait over (tokio-postgres, rusqlite) pay more trait-fitting cost than sqlx's single-API convenience costs elsewhere? What does SQLITE_ENABLE/extension loading look like under sqlx vs rusqlite? (And the async question honker-rs forces, §driver-conflict correction 2: is rusqlite-in-a-pool the right tokio shape, or does sqlx-sqlite's native async change the watcher machinery's design?)

Genuinely open; needs research rounds (library capabilities vs the unified-trait shape) and possibly a POC. Not deferred — this is the central Phase 0 research question.

OQ-ST-04: The reactive abstraction — what does the unified notify surface look like?

The two engines' mechanisms are structurally different: SQLite = watcher polling PRAGMA data_version (deliver on commit; no server-side push exists), Postgres = LISTEN/NOTIFY (server push, connection-bound, no retry/visibility semantics). The reactive trait must have a shape both implement without one emulating the other's weaknesses.

The honker-rs surface (§Interface finding) is the concrete starting point — the question decomposes into contract-pinning rather than shape-invention:

• Which parts of the honker-rs surface become the crate's contract: the notify/listen pair, the stream/offset/consumer model, the queue claim/ack/visibility model, scheduler, locks/rate-limits, outbox — all, a subset, or renamed/regrouped?
• What is the delivery-guarantee contract, per mechanism (honker's own guide table shows how easily per-binding auto-checkpoint vs manual-save ambiguity produces different guarantees under one function name — the single-crate version must pick one answer, not inherit the table)?
• Listener semantics: honker starts from MAX(id) and replays nothing; Postgres LISTEN has no replay either but delivers via a dedicated connection with its own lifecycle. Does listen() abstract over both honestly (opaque wake + re-read contract) or promise durability it only has on one engine (that's what streams are for)?
• The transactional seam (enqueue_tx/publish_tx/save_offset_tx) across two transaction models — the driver-coupled point (§driver- conflict correction 1).

How does a caching subscriber (the ecosystem's hot-path pattern) receive sufficient invalidation information (keys? table/channel names? opaque wake + re-read contract?) — rides the same contract decision.

Open; this is the second central research question, coupled to OQ-ST-03 (the driver determines what LISTEN plumbing exists).

OQ-ST-05: Queue semantics — adopt, fork, or re-derive?

If queues land in scope (OQ-ST-01), the pg-boss schema family is the Postgres-side incumbent and honker's queue design is the SQLite-side one. Options: adopt pgboss-rs as a dependency (new feature-gated option); targeted-fork the relevant subsystem (alksocks precedent, ported to our conventions); schema/design-reference only (re-derive on our driver). Fork-vs-derive depends on how much of pgboss-rs is queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03), on our tolerance for the alpha-state rc port, and on the verified gap (§pgboss-rs): the push-reactivity half has to be built on top of any choice, so the queue-machinery reuse value is the honest comparison point, not the whole.

Open; inputs: the OQ-ST-01 inventory + OQ-ST-03 resolution.

OQ-ST-06: Honker relationship — reference, fork, or vendor?

Design-reference only (read, don't copy), targeted fork of honker-core's engine machinery, or vendor the extension? Honker is alpha-quality per its own README, MIT/Apache-2.0 dual-licensed, and covers only the SQLite side — but it embodies exactly the watcher/ transactional design this crate wants on SQLite, and honker-rs demonstrates the interface shape is sound. Three refinements from the 2026-10-03 reading:

• honker-rs is sync-only (std threads, blocking iterators) — the tokio port is required work under any fork posture, which changes the fork-vs-reference calculus (a fork is already a serious port).
• The crate likely needs only the core engine machinery (honker-core minus the extension C surface — see OQ-ST-07), a smaller extraction than the whole project.
• Honker's own documented per-binding inconsistencies (the processing-guarantees table, OQ-ST-04) suggest extracting design+ semantics with our contract pinned, rather than preserving its behavior verbatim — closer to the alkblobs "borrow conclusions, not wire surface" principle than to alksocks' verbatim extraction.

Open; needs the license/provenance check (AGENTS.md §3) and a quality assessment of honker-core's watcher/transactional core.

OQ-ST-07: SQLite-side scope — loadable extension, embedded rusqlite, or both?

Honker ships as a loadable extension usable by any SQLite client, plus per-language bindings. This crate (a Rust library) may not need the loadable-extension surface at all — embedding the engine machinery in-process may be the whole story (the honker-core shape minus the extension/binding packaging). Determines how much of honker is even candidate material.

Open; rides the first consumer-driven scope pass (OQ-ST-01).

OQ-ST-08: Multi-host / deployment posture

Honker is explicitly single-machine (file-backed SQLite). Postgres is natively multi-host. The unified surface must not pretend SQLite is multi-host, but where does the honest boundary live — per-engine capability flags? A documented deployment matrix? Does the trait need to expose engine capabilities at all?

Open; partially rides OQ-ST-04 (the trait's shape constrains where capability differences can surface).

Phase 0 plan

Iteration expected; this register grows as the consumer inventory and research rounds land. Expected sequence (deliberately rough):

1. Consumer-driven scope inventory (OQ-ST-01) — what the paused alkblobs consumers and alkfs planning actually need from the reactive store; walks back the per-feature scope decisions.
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library capability matrices, then a POC if the unified-trait shape needs validation (likely, given the structural mismatch noted in OQ-ST-04). The honker-rs surface (§Interface finding) is the concrete starting artifact for the OQ-ST-04 work: pinning its contract costs less and is more honest than inventing a parallel shape.
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem, after the driver and shape questions narrow the option space.
4. Converge; Phase 1 opens with the ADR backlog this register becomes.

References

• honker — /workspace/honker (git checkout; README + honker-core/src/ read 2026-10-03): the SQLite-side feature/wake template. The four guides (queues/streams/pubsub/scheduler on honker.dev) + packages/honker-rs/src/lib.rs (v0.5.0) are the interface prior art (§Interface finding).
• pgboss-rs — /workspace/pgboss-rs (git checkout of github.com/rustworthy/pgboss-rs, v0.1.0-rc6; read 2026-10-03, LISTEN/NOTIFY-absence verified): the Postgres queue-family reference.
• honker's own prior-art section: pg_notify, pg-boss, Oban, Huey — the external lineage this crate inherits from both sides.
• alkblobs — /workspace/@alkdev/alkblobs (spec+POCs, paused): the paused planning this crate unblocks; its POC findings (poc-postgres-kv-findings.md, poc-pglo-findings.md) are the tokio-postgres evidence base.
• alkcall — /workspace/@alkdev/alkcall: the family substrate; referenced for the store-layer-isolation principle only.