alkdev/alkstore
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase-0: interface finding — honker-rs surface as the unified-API candidate

Browse Source 
Read the four honker.dev guides (queues/streams/pubsub/scheduler) +
packages/honker-rs/src/lib.rs (v0.5.0, 1706 lines):

- honker's Rust binding exposes the exact surface shape alkstore wants
  (queue claim/ack/visibility, streams with tx-aware offsets, notify/
  listen, leader-elected scheduler, outbox, locks/rate-limits/results)
  — the unified-API question shifts from shape-invention to contract-
  pinning on that surface (new 'Interface finding' section)
- honker's own processing-guarantees table (per-binding auto-checkpoint
  vs manual offset save) is the named seam a single-crate contract
  cleans up
- honker-rs is sync-only (std threads, no tokio) — SQLite side is a
  port-and-adapt under any posture, folded into driver-conflict
  corrections + OQ-ST-03/04/06 refinements
- pgboss-rs LISTEN/NOTIFY absence (verified earlier) now stated as the
  substantive fork-or-derive comparison point (OQ-ST-05)
This commit is contained in:
glm-5.3-flash committed 2026-10-03 16:50:54 +00:00
1 parent f6531b5532
commit 8e6da2f6c9
1 file changed
+146 -22
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/phase-0.md
+146 -22
View File
@@ -1,6 +1,8 @@
---
status: draft
last_updated: 2026-10-03 (initial draft from the setup discussion)
last_updated: 2026-10-03 (initial draft + interface finding: honker-rs surface
read as the unified-API candidate; pgboss-rs LISTEN/NOTIFY-absence folded into
the driver/queue/ownership questions)
---
# alkstore — Phase 0 (Exploration)
@@ -42,6 +44,45 @@ outbox — all as INSERTs inside the caller's transaction, with the
cross-process wake delivered by a shared watcher that polls `PRAGMA
data_version` (default 1 ms → single-digit-ms delivery) and re-reads
indexed state after every wake. Its own Prior Art section names the
**The interface finding (2026-10-03, from the four honker.dev guides +
`packages/honker-rs/src/lib.rs`, the Rust binding): honker's *surface*
is already the unified-interface candidate.** Its Rust binding exposes
exactly the surface this crate wants, engine-clean:
- `db.queue(name, QueueOpts)` → `enqueue / enqueue_tx / claim_one /
claim_batch / ack_batch / cancel / get_job / sweep_expired /
claim_waker`, with `job.ack / retry / fail / heartbeat` and
`EnqueueOpts {delay, priority, max_attempts, expires, ...}` —
semantically the pg-boss model (visibility timeouts, retries,
dead-letter via move-to-`_honker_dead`), not a LISTEN-emulation.
- `db.stream(name)` → `publish / publish_tx / publish_with_key /
read_since / read_from_consumer / save_offset(_tx) / get_offset /
subscribe(consumer)` — offsets are explicit, transaction-aware
(`save_offset_tx`) for the exactly-once-within-a-business-tx shape,
and replay-on-reconnect is the default.
- `db.notify(channel, payload)` / `notify_tx` / `db.listen(channel)` —
the `pg_notify`-analogue fire-and-forget signal layer ("fire-and-forget,
no replay, no guarantees" — streams are the durable cousin), listener
starts from `MAX(id)` at attach, no historical replay.
- `db.scheduler()` → `add/pause/resume/update/list/remove/tick/run` —
cron + `@every` enqueueing into named queues, leader-elected via
advisory lock with TTL heartbeat, missed-boundary catch-up.
- `db.outbox(name)` — the transactional outbox helper (enqueue +
`run_once` delivery worker).
- `db.try_lock / try_rate_limit / save_result / get_result /
sweep_results` — the coordination/adjacent-tools surface.
The implication flips the framing of the unified-API work: it is not
"invent a shape both engines fit" — it is **"this shape both engines
can fit"** (pg-boss's queue model is *already* the native Postgres
tooling model; streams/offsets and notify have direct Postgres
counterparts) **and the work is pinning which parts of the shape are
the crate's contract** — the delivery-guarantee differences honker's
own guide documents per-binding (auto-checkpoint cadence vs manual
offset save; the processing-guarantees table) are exactly the seams a
single-crate version must clean up. Honker-rs is the concrete prior
art for that pinning exercise.
Postgres side: `pg_notify` gives fast triggers with no retry or
visibility semantics; pg-boss/Oban are the durable-layer gold standards
— "If you already run Postgres, use the Postgres tools."
@@ -51,7 +92,10 @@ one store API whose consumer code (queues, streams, notify) looks the
same whether the backing engine is SQLite or Postgres, while each engine
uses its own native wake/delivery story under the hood. The honker docs
recommend pgboss + `pg_notify` for the Postgres equivalent — that
recommendation is the design brief for this crate's Postgres engine.
recommendation is the design brief for this crate's Postgres engine —
and honker's Rust binding (`honker-rs`, v0.5.0) is the concrete
candidate for what that unified interface literally looks like (§Interface
finding).
**Consumer shape (from the paused alkblobs planning):** any crate that
today uses the ecosystem's repo-pattern + in-memory adapter should be
@@ -121,11 +165,35 @@ reactivity story is entangled with it:
LISTEN/NOTIFY. The unified reactive trait must abstract over both
without collapsing to the polling behavior of the weaker side.
Two corrections to the original framing (2026-10-03, after reading
honker-rs and the honker.dev guides):
1. **The honker-rs *interface* is largely driver-independent.** The
Queue/Stream/notify/scheduler surface (§Interface finding) speaks in
domain terms (channels, offsets, job ids, visibility timeouts), not
driver terms. What *is* driver-coupled is the transactional seam:
`enqueue_tx`/`publish_tx`/`save_offset_tx` and the extension's
notify-in-transaction all assume the caller can hand the engine a
live transaction handle from *its* driver. The unified trait's
transactional seam is therefore the driver-sensitive design point —
and it interacts with the engines' different transaction models
(SQLite: single writer, synchronous; Postgres: interactive
transactions over a pool, transaction-scoped LISTEN).
2. **honker-rs is sync (`std` threads + blocking iterators) — parking_lot
+ rusqlite, no tokio.** The family standard is tokio-async. So even
the SQLite side is a *port-and-adapt* (sync → async), not an adopt;
and the driver question for SQLite is also entangled with whether
tokio-native sqlite drivers (sqlx sqlite) change the queue/wake
machinery's shape versus rusqlite-in-a-pool.
This tension is OQ-ST-04 below. It is *not* resolved by "pgboss is well
written so start there" — that is exactly the inherited-assumption
shape the SDD process flags. What pgboss-rs genuinely offers (schema
DDL, job states, retry semantics, the node-compatible API) is design
reference regardless of driver.
reference regardless of driver. The reactivity gap is the concrete,
verified difference between pgboss-rs and what this crate needs —
whether forked or re-derived, the push channel is work this crate
builds itself (OQ-ST-05).
## Prior art
@@ -159,6 +227,18 @@ this crate:
- **The transactional enqueue shape** — every feature is an INSERT
inside the caller's transaction. This is the pattern the unified API
must keep visible and cheap.
- **The honker-rs binding is the concrete interface prior art** (v0.5.0,
`packages/honker-rs`, read 2026-10-03): the full surface per §Interface
finding. Notable honest limitations documented by its own guides —
the per-binding processing-guarantees table (auto-checkpoint cadence
vs manual offset save; several bindings "may persist an offset on a
cadence... without knowing whether downstream application work
committed"), the Node reverse-order consumer-checkpoint bug, per-
binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++ missing
typed pruning) — are exactly the seams a single-crate version
designed-for-the-contract from day one can clean up. Its sync-only
shape (std threads, blocking iterators, no tokio) is a
port-and-adapt constraint, not an adopt candidate as-is.
### pgboss-rs — the Postgres queue family reference
@@ -168,7 +248,10 @@ singleton/dead-letter concepts, `sqlx` 0.8, schema-scoped DDL.
- Verified gap (2026-10-03): **no LISTEN/NOTIFY** anywhere in `src/`
— consumption is polling `fetch_job`. Any push-reactivity is new
work, not an adoption freebie.
work, not an adoption freebie. This is the substantive difference
between pgboss-rs and what alkstore needs: regardless of fork vs
re-derive, reactivity is ours to build on the Postgres side either
way.
- Its value as reference: the pg-boss schema family (job states,
maintenance/dead-letter behavior) is battle-tested against real
Postgres semantics — worth borrowing *as design*, independent of the
@@ -251,6 +334,9 @@ Honest unknowns worth surfacing: does sqlx support SQLite
trait over `(tokio-postgres, rusqlite)` pay more trait-fitting cost
than sqlx's single-API convenience costs elsewhere? What does
SQLITE_ENABLE/extension loading look like under sqlx vs rusqlite?
(And the async question honker-rs forces, §driver-conflict correction
2: is rusqlite-in-a-pool the right tokio shape, or does sqlx-sqlite's
native async change the watcher machinery's design?)
Genuinely open; needs research rounds (library capabilities vs the
unified-trait shape) and possibly a POC. Not deferred — this is the
@@ -263,18 +349,35 @@ watcher polling `PRAGMA data_version` (deliver on commit; no
server-side push exists), Postgres = LISTEN/NOTIFY (server push,
connection-bound, no retry/visibility semantics). The reactive trait
must have a shape both implement without one emulating the other's
weaknesses:
weaknesses.
- What is the subscription type (`channel`? stream of envelopes?)
- What is the delivery guarantee contract on each engine (honker's
wake-on-commit + re-read is *not* exactly-once — what does the trait
promise?)
- Does the trait absorb the enqueue+notify-in-one-transaction shape
(honker's core) — and how does that compose with Postgres
transaction-scoped LISTEN semantics?
- How does a caching subscriber (the ecosystem's hot-path pattern)
receive sufficient invalidation information (keys? table/channel
names? opaque wake + re-read contract?)
The honker-rs surface (§Interface finding) is the concrete starting
point — the question decomposes into contract-pinning rather than
shape-invention:
- Which parts of the honker-rs surface become the crate's *contract*:
the `notify`/`listen` pair, the `stream`/offset/consumer model, the
queue claim/ack/visibility model, scheduler, locks/rate-limits,
outbox — all, a subset, or renamed/regrouped?
- What is the delivery-guarantee contract, per mechanism (honker's
own guide table shows how easily per-binding auto-checkpoint vs
manual-save ambiguity produces *different* guarantees under one
function name — the single-crate version must pick one answer, not
inherit the table)?
- Listener semantics: honker starts from `MAX(id)` and replays
nothing; Postgres LISTEN has no replay either but delivers via a
dedicated connection with its own lifecycle. Does `listen()`
abstract over both honestly (opaque wake + re-read contract) or
promise durability it only has on one engine (that's what streams
are for)?
- The transactional seam (`enqueue_tx`/`publish_tx`/`save_offset_tx`)
across two transaction models — the driver-coupled point (§driver-
conflict correction 1).
How does a caching subscriber (the ecosystem's hot-path pattern)
receive sufficient invalidation information (keys? table/channel
names? opaque wake + re-read contract?) — rides the same contract
decision.
Open; this is the second central research question, coupled to OQ-ST-03
(the driver determines what LISTEN plumbing exists).
@@ -287,8 +390,11 @@ one. Options: adopt pgboss-rs as a dependency (new feature-gated
option); targeted-fork the relevant subsystem (alksocks precedent,
ported to our conventions); schema/design-reference only (re-derive on
our driver). Fork-vs-derive depends on how much of pgboss-rs is
queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03) and on
our tolerance for the alpha-state rc port.
queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03), on
our tolerance for the alpha-state rc port, and on the verified gap
(§pgboss-rs): the push-reactivity half has to be built on top of any
choice, so the queue-machinery reuse value is the honest comparison
point, not the whole.
Open; inputs: the OQ-ST-01 inventory + OQ-ST-03 resolution.
@@ -298,12 +404,24 @@ Design-reference only (read, don't copy), targeted fork of
honker-core's engine machinery, or vendor the extension? Honker is
alpha-quality per its own README, MIT/Apache-2.0 dual-licensed, and
covers only the SQLite side — but it embodies exactly the watcher/
transactional design this crate wants on SQLite. Fork-postures in this
workspace have precedent (alksocks' extraction) but have been for
*owning* a needed subset, not for adopting an alpha wholesale.
transactional design this crate wants on SQLite, and honker-rs
demonstrates the interface shape is sound. Three refinements from the
2026-10-03 reading:
- honker-rs is **sync-only** (std threads, blocking iterators) — the
tokio port is required work under any fork posture, which changes
the fork-vs-reference calculus (a fork is already a serious port).
- The crate likely needs only the core engine machinery (honker-core
minus the extension C surface — see OQ-ST-07), a smaller extraction
than the whole project.
- Honker's own documented per-binding inconsistencies (the
processing-guarantees table, OQ-ST-04) suggest extracting *design+
semantics* with our contract pinned, rather than preserving its
behavior verbatim — closer to the alkblobs "borrow conclusions, not
wire surface" principle than to alksocks' verbatim extraction.
Open; needs the license/provenance check (AGENTS.md §3) and a quality
assessment honker's watcher/transactional core.
assessment of honker-core's watcher/transactional core.
### OQ-ST-07: SQLite-side scope — loadable extension, embedded rusqlite, or both?
@@ -338,6 +456,9 @@ research rounds land. Expected sequence (deliberately rough):
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library
capability matrices, then a POC if the unified-trait shape needs
validation (likely, given the structural mismatch noted in OQ-ST-04).
The honker-rs surface (§Interface finding) is the concrete starting
artifact for the OQ-ST-04 work: pinning its contract costs less and
is more honest than inventing a parallel shape.
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem,
after the driver and shape questions narrow the option space.
4. Converge; Phase 1 opens with the ADR backlog this register becomes.
@@ -345,7 +466,10 @@ research rounds land. Expected sequence (deliberately rough):
## References
- honker — `/workspace/honker` (git checkout; README + `honker-core/src/`
read 2026-10-03): the SQLite-side feature/wake template.
read 2026-10-03): the SQLite-side feature/wake template. The four
guides (queues/streams/pubsub/scheduler on honker.dev) +
`packages/honker-rs/src/lib.rs` (v0.5.0) are the interface prior art
(§Interface finding).
- pgboss-rs — `/workspace/pgboss-rs` (git checkout of
github.com/rustworthy/pgboss-rs, v0.1.0-rc6; read 2026-10-03,
LISTEN/NOTIFY-absence verified): the Postgres queue-family reference.