diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md index d37cd2d..7d6a3fd 100644 --- a/docs/research/phase-0.md +++ b/docs/research/phase-0.md @@ -1,6 +1,8 @@ --- status: draft -last_updated: 2026-10-03 (initial draft from the setup discussion) +last_updated: 2026-10-03 (initial draft + interface finding: honker-rs surface +read as the unified-API candidate; pgboss-rs LISTEN/NOTIFY-absence folded into +the driver/queue/ownership questions) --- # alkstore — Phase 0 (Exploration) @@ -42,6 +44,45 @@ outbox — all as INSERTs inside the caller's transaction, with the cross-process wake delivered by a shared watcher that polls `PRAGMA data_version` (default 1 ms → single-digit-ms delivery) and re-reads indexed state after every wake. Its own Prior Art section names the + +**The interface finding (2026-10-03, from the four honker.dev guides + +`packages/honker-rs/src/lib.rs`, the Rust binding): honker's *surface* +is already the unified-interface candidate.** Its Rust binding exposes +exactly the surface this crate wants, engine-clean: + +- `db.queue(name, QueueOpts)` → `enqueue / enqueue_tx / claim_one / + claim_batch / ack_batch / cancel / get_job / sweep_expired / + claim_waker`, with `job.ack / retry / fail / heartbeat` and + `EnqueueOpts {delay, priority, max_attempts, expires, ...}` — + semantically the pg-boss model (visibility timeouts, retries, + dead-letter via move-to-`_honker_dead`), not a LISTEN-emulation. +- `db.stream(name)` → `publish / publish_tx / publish_with_key / + read_since / read_from_consumer / save_offset(_tx) / get_offset / + subscribe(consumer)` — offsets are explicit, transaction-aware + (`save_offset_tx`) for the exactly-once-within-a-business-tx shape, + and replay-on-reconnect is the default. +- `db.notify(channel, payload)` / `notify_tx` / `db.listen(channel)` — + the `pg_notify`-analogue fire-and-forget signal layer ("fire-and-forget, + no replay, no guarantees" — streams are the durable cousin), listener + starts from `MAX(id)` at attach, no historical replay. +- `db.scheduler()` → `add/pause/resume/update/list/remove/tick/run` — + cron + `@every` enqueueing into named queues, leader-elected via + advisory lock with TTL heartbeat, missed-boundary catch-up. +- `db.outbox(name)` — the transactional outbox helper (enqueue + + `run_once` delivery worker). +- `db.try_lock / try_rate_limit / save_result / get_result / + sweep_results` — the coordination/adjacent-tools surface. + +The implication flips the framing of the unified-API work: it is not +"invent a shape both engines fit" — it is **"this shape both engines +can fit"** (pg-boss's queue model is *already* the native Postgres +tooling model; streams/offsets and notify have direct Postgres +counterparts) **and the work is pinning which parts of the shape are +the crate's contract** — the delivery-guarantee differences honker's +own guide documents per-binding (auto-checkpoint cadence vs manual +offset save; the processing-guarantees table) are exactly the seams a +single-crate version must clean up. Honker-rs is the concrete prior +art for that pinning exercise. Postgres side: `pg_notify` gives fast triggers with no retry or visibility semantics; pg-boss/Oban are the durable-layer gold standards — "If you already run Postgres, use the Postgres tools." @@ -51,7 +92,10 @@ one store API whose consumer code (queues, streams, notify) looks the same whether the backing engine is SQLite or Postgres, while each engine uses its own native wake/delivery story under the hood. The honker docs recommend pgboss + `pg_notify` for the Postgres equivalent — that -recommendation is the design brief for this crate's Postgres engine. +recommendation is the design brief for this crate's Postgres engine — +and honker's Rust binding (`honker-rs`, v0.5.0) is the concrete +candidate for what that unified interface literally looks like (§Interface +finding). **Consumer shape (from the paused alkblobs planning):** any crate that today uses the ecosystem's repo-pattern + in-memory adapter should be @@ -121,11 +165,35 @@ reactivity story is entangled with it: LISTEN/NOTIFY. The unified reactive trait must abstract over both without collapsing to the polling behavior of the weaker side. +Two corrections to the original framing (2026-10-03, after reading +honker-rs and the honker.dev guides): + +1. **The honker-rs *interface* is largely driver-independent.** The + Queue/Stream/notify/scheduler surface (§Interface finding) speaks in + domain terms (channels, offsets, job ids, visibility timeouts), not + driver terms. What *is* driver-coupled is the transactional seam: + `enqueue_tx`/`publish_tx`/`save_offset_tx` and the extension's + notify-in-transaction all assume the caller can hand the engine a + live transaction handle from *its* driver. The unified trait's + transactional seam is therefore the driver-sensitive design point — + and it interacts with the engines' different transaction models + (SQLite: single writer, synchronous; Postgres: interactive + transactions over a pool, transaction-scoped LISTEN). +2. **honker-rs is sync (`std` threads + blocking iterators) — parking_lot + + rusqlite, no tokio.** The family standard is tokio-async. So even + the SQLite side is a *port-and-adapt* (sync → async), not an adopt; + and the driver question for SQLite is also entangled with whether + tokio-native sqlite drivers (sqlx sqlite) change the queue/wake + machinery's shape versus rusqlite-in-a-pool. + This tension is OQ-ST-04 below. It is *not* resolved by "pgboss is well written so start there" — that is exactly the inherited-assumption shape the SDD process flags. What pgboss-rs genuinely offers (schema DDL, job states, retry semantics, the node-compatible API) is design -reference regardless of driver. +reference regardless of driver. The reactivity gap is the concrete, +verified difference between pgboss-rs and what this crate needs — +whether forked or re-derived, the push channel is work this crate +builds itself (OQ-ST-05). ## Prior art @@ -159,6 +227,18 @@ this crate: - **The transactional enqueue shape** — every feature is an INSERT inside the caller's transaction. This is the pattern the unified API must keep visible and cheap. +- **The honker-rs binding is the concrete interface prior art** (v0.5.0, + `packages/honker-rs`, read 2026-10-03): the full surface per §Interface + finding. Notable honest limitations documented by its own guides — + the per-binding processing-guarantees table (auto-checkpoint cadence + vs manual offset save; several bindings "may persist an offset on a + cadence... without knowing whether downstream application work + committed"), the Node reverse-order consumer-checkpoint bug, per- + binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++ missing + typed pruning) — are exactly the seams a single-crate version + designed-for-the-contract from day one can clean up. Its sync-only + shape (std threads, blocking iterators, no tokio) is a + port-and-adapt constraint, not an adopt candidate as-is. ### pgboss-rs — the Postgres queue family reference @@ -168,7 +248,10 @@ singleton/dead-letter concepts, `sqlx` 0.8, schema-scoped DDL. - Verified gap (2026-10-03): **no LISTEN/NOTIFY** anywhere in `src/` — consumption is polling `fetch_job`. Any push-reactivity is new - work, not an adoption freebie. + work, not an adoption freebie. This is the substantive difference + between pgboss-rs and what alkstore needs: regardless of fork vs + re-derive, reactivity is ours to build on the Postgres side either + way. - Its value as reference: the pg-boss schema family (job states, maintenance/dead-letter behavior) is battle-tested against real Postgres semantics — worth borrowing *as design*, independent of the @@ -251,6 +334,9 @@ Honest unknowns worth surfacing: does sqlx support SQLite trait over `(tokio-postgres, rusqlite)` pay more trait-fitting cost than sqlx's single-API convenience costs elsewhere? What does SQLITE_ENABLE/extension loading look like under sqlx vs rusqlite? +(And the async question honker-rs forces, §driver-conflict correction +2: is rusqlite-in-a-pool the right tokio shape, or does sqlx-sqlite's +native async change the watcher machinery's design?) Genuinely open; needs research rounds (library capabilities vs the unified-trait shape) and possibly a POC. Not deferred — this is the @@ -263,18 +349,35 @@ watcher polling `PRAGMA data_version` (deliver on commit; no server-side push exists), Postgres = LISTEN/NOTIFY (server push, connection-bound, no retry/visibility semantics). The reactive trait must have a shape both implement without one emulating the other's -weaknesses: +weaknesses. -- What is the subscription type (`channel`? stream of envelopes?) -- What is the delivery guarantee contract on each engine (honker's - wake-on-commit + re-read is *not* exactly-once — what does the trait - promise?) -- Does the trait absorb the enqueue+notify-in-one-transaction shape - (honker's core) — and how does that compose with Postgres - transaction-scoped LISTEN semantics? -- How does a caching subscriber (the ecosystem's hot-path pattern) - receive sufficient invalidation information (keys? table/channel - names? opaque wake + re-read contract?) +The honker-rs surface (§Interface finding) is the concrete starting +point — the question decomposes into contract-pinning rather than +shape-invention: + +- Which parts of the honker-rs surface become the crate's *contract*: + the `notify`/`listen` pair, the `stream`/offset/consumer model, the + queue claim/ack/visibility model, scheduler, locks/rate-limits, + outbox — all, a subset, or renamed/regrouped? +- What is the delivery-guarantee contract, per mechanism (honker's + own guide table shows how easily per-binding auto-checkpoint vs + manual-save ambiguity produces *different* guarantees under one + function name — the single-crate version must pick one answer, not + inherit the table)? +- Listener semantics: honker starts from `MAX(id)` and replays + nothing; Postgres LISTEN has no replay either but delivers via a + dedicated connection with its own lifecycle. Does `listen()` + abstract over both honestly (opaque wake + re-read contract) or + promise durability it only has on one engine (that's what streams + are for)? +- The transactional seam (`enqueue_tx`/`publish_tx`/`save_offset_tx`) + across two transaction models — the driver-coupled point (§driver- + conflict correction 1). + +How does a caching subscriber (the ecosystem's hot-path pattern) +receive sufficient invalidation information (keys? table/channel +names? opaque wake + re-read contract?) — rides the same contract +decision. Open; this is the second central research question, coupled to OQ-ST-03 (the driver determines what LISTEN plumbing exists). @@ -287,8 +390,11 @@ one. Options: adopt pgboss-rs as a dependency (new feature-gated option); targeted-fork the relevant subsystem (alksocks precedent, ported to our conventions); schema/design-reference only (re-derive on our driver). Fork-vs-derive depends on how much of pgboss-rs is -queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03) and on -our tolerance for the alpha-state rc port. +queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03), on +our tolerance for the alpha-state rc port, and on the verified gap +(§pgboss-rs): the push-reactivity half has to be built on top of any +choice, so the queue-machinery reuse value is the honest comparison +point, not the whole. Open; inputs: the OQ-ST-01 inventory + OQ-ST-03 resolution. @@ -298,12 +404,24 @@ Design-reference only (read, don't copy), targeted fork of honker-core's engine machinery, or vendor the extension? Honker is alpha-quality per its own README, MIT/Apache-2.0 dual-licensed, and covers only the SQLite side — but it embodies exactly the watcher/ -transactional design this crate wants on SQLite. Fork-postures in this -workspace have precedent (alksocks' extraction) but have been for -*owning* a needed subset, not for adopting an alpha wholesale. +transactional design this crate wants on SQLite, and honker-rs +demonstrates the interface shape is sound. Three refinements from the +2026-10-03 reading: + +- honker-rs is **sync-only** (std threads, blocking iterators) — the + tokio port is required work under any fork posture, which changes + the fork-vs-reference calculus (a fork is already a serious port). +- The crate likely needs only the core engine machinery (honker-core + minus the extension C surface — see OQ-ST-07), a smaller extraction + than the whole project. +- Honker's own documented per-binding inconsistencies (the + processing-guarantees table, OQ-ST-04) suggest extracting *design+ + semantics* with our contract pinned, rather than preserving its + behavior verbatim — closer to the alkblobs "borrow conclusions, not + wire surface" principle than to alksocks' verbatim extraction. Open; needs the license/provenance check (AGENTS.md §3) and a quality -assessment honker's watcher/transactional core. +assessment of honker-core's watcher/transactional core. ### OQ-ST-07: SQLite-side scope — loadable extension, embedded rusqlite, or both? @@ -338,6 +456,9 @@ research rounds land. Expected sequence (deliberately rough): 2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library capability matrices, then a POC if the unified-trait shape needs validation (likely, given the structural mismatch noted in OQ-ST-04). + The honker-rs surface (§Interface finding) is the concrete starting + artifact for the OQ-ST-04 work: pinning its contract costs less and + is more honest than inventing a parallel shape. 3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem, after the driver and shape questions narrow the option space. 4. Converge; Phase 1 opens with the ADR backlog this register becomes. @@ -345,7 +466,10 @@ research rounds land. Expected sequence (deliberately rough): ## References - honker — `/workspace/honker` (git checkout; README + `honker-core/src/` - read 2026-10-03): the SQLite-side feature/wake template. + read 2026-10-03): the SQLite-side feature/wake template. The four + guides (queues/streams/pubsub/scheduler on honker.dev) + + `packages/honker-rs/src/lib.rs` (v0.5.0) are the interface prior art + (§Interface finding). - pgboss-rs — `/workspace/pgboss-rs` (git checkout of github.com/rustworthy/pgboss-rs, v0.1.0-rc6; read 2026-10-03, LISTEN/NOTIFY-absence verified): the Postgres queue-family reference.