The async-facing-trait + sync-bridge posture (blocking impl on dedicated threads/spawn_blocking feeding tokio channels, documented as a supported strategy not a workaround) is already family-standard twice over: alktty REQ-TTY-01 and alkblobs' spawn_blocking-in-engine-impls execution posture. Recorded verbatim-sourced in OQ-ST-03; reframes the honker-rs sync→async port as bridge-at-the-trait-seam vs native-async-rewrite and weakens sqlx's main differentiator on the sqlite side. alktty added to references.
595 lines
32 KiB
Markdown
595 lines
32 KiB
Markdown
---
|
|
status: draft
|
|
last_updated: 2026-10-04 (consumer inventory landed — OQ-ST-01 answered
|
|
per-feature from the paused consumers' documents; phase-0 plan step 1
|
|
done; streams upgraded to in-scope by operator-authority record; OQ-ST-02
|
|
resolved: reactive-core + engine crates, operator decision; alktty
|
|
REQ-TTY-01's async-facing-trait + sync-bridge posture recorded as family
|
|
precedent bearing on OQ-ST-03's async sub-question. OQ-ST-03/04/05
|
|
remain the open research. Interface finding and driver tension from
|
|
2026-10-03 remain trusted-but-unverified working input.)
|
|
---
|
|
|
|
# alkstore — Phase 0 (Exploration)
|
|
|
|
This document captures Phase 0 (Exploration) for the `alkstore` crate:
|
|
vision, guiding principles, prior art, and the open-question register
|
|
(OQ-ST-01..NN). Phase 0's objective per `docs/sdd_process.md`: *capture
|
|
vision and guiding principles; research options; validate approaches;
|
|
converge on a recommended approach.* The scope question (OQ-ST-01) has
|
|
been answered per-feature from the consumers' documents
|
|
(`consumer-inventory.md`); the driver and reactive-shape questions
|
|
(OQ-ST-03/04) are the open research; there is no POC register yet
|
|
(see `consumer-inventory.md` for the first cut candidates that keep its
|
|
register small).
|
|
|
|
Context for why this crate starts now: **alkblobs**
|
|
(`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning)
|
|
hit repeated circular hedging in its Phase 0, and a root cause was that
|
|
the storage substrate it deploys onto was itself disjoint and fuzzy — a
|
|
repo pattern with a default in-memory adapter across the alk* ecosystem,
|
|
cache-invalidation patches in hot paths, non-invalidated caches where
|
|
delay was tolerable, no single definition of "how does a change in the
|
|
database become visible to other processes/connections?" alkblobs paused
|
|
partly to let this crate answer that first. alkstore is the attempt to
|
|
make that substrate real once, so downstream stores don't re-derive it.
|
|
|
|
## Vision and guiding principles
|
|
|
|
**One sentence (draft):** one reactive store interface over SQLite and
|
|
Postgres — durable pub/sub notify, queues, streams, and the transactional
|
|
integration (write + enqueue in one transaction) that honker delivers on
|
|
SQLite — with the Postgres side building on the natively-available
|
|
machinery (`pg_notify`/`LISTEN`, and the pgboss job-queue schema family)
|
|
rather than emulating it.
|
|
|
|
**The honker relationship.** `/workspace/honker` (reference checkout,
|
|
not for direct use as a dependency; alpha-quality per its own README,
|
|
MIT/Apache-2.0 dual) adds
|
|
Postgres-style `NOTIFY`/`LISTEN` semantics to SQLite without a broker:
|
|
durable at-least-once queues with retries/delay/priority/visibility
|
|
timeouts/dead-letter, durable streams with per-consumer offsets,
|
|
cron/`@every` scheduling, named locks, rate limits, transactional
|
|
outbox — all as INSERTs inside the caller's transaction, with the
|
|
cross-process wake delivered by a shared watcher that polls `PRAGMA
|
|
data_version` (default 1 ms → single-digit-ms delivery) and re-reads
|
|
indexed state after every wake. Its own Prior Art section names the
|
|
lineage: `pg_notify`, pg-boss, Oban, Huey.
|
|
|
|
**The interface finding (2026-10-03, from the four honker.dev guides +
|
|
`packages/honker-rs/src/lib.rs`, the Rust binding): honker's *surface*
|
|
is already the unified-interface candidate.** Its Rust binding exposes
|
|
exactly the surface this crate wants, engine-clean:
|
|
|
|
- `db.queue(name, QueueOpts)` → `enqueue / enqueue_tx / claim_one /
|
|
claim_batch / ack_batch / cancel / get_job / sweep_expired /
|
|
claim_waker`, with `job.ack / retry / fail / heartbeat` and
|
|
`EnqueueOpts {delay, priority, max_attempts, expires, ...}` —
|
|
semantically the pg-boss model (visibility timeouts, retries,
|
|
dead-letter via move-to-`_honker_dead`), not a LISTEN-emulation.
|
|
- `db.stream(name)` → `publish / publish_tx / publish_with_key /
|
|
read_since / read_from_consumer / save_offset(_tx) / get_offset /
|
|
subscribe(consumer)` — offsets are explicit, transaction-aware
|
|
(`save_offset_tx`) for the exactly-once-within-a-business-tx shape,
|
|
and replay-on-reconnect is the default.
|
|
- `db.notify(channel, payload)` / `notify_tx` / `db.listen(channel)` —
|
|
the `pg_notify`-analogue fire-and-forget signal layer ("fire-and-forget,
|
|
no replay, no guarantees" — streams are the durable cousin), listener
|
|
starts from `MAX(id)` at attach, no historical replay.
|
|
- `db.scheduler()` → `add/pause/resume/update/list/remove/tick/run` —
|
|
cron + `@every` enqueueing into named queues, leader-elected via
|
|
advisory lock with TTL heartbeat, missed-boundary catch-up.
|
|
- `db.outbox(name)` — the transactional outbox helper (enqueue +
|
|
`run_once` delivery worker).
|
|
- `db.try_lock / try_rate_limit / save_result / get_result /
|
|
sweep_results` — the coordination/adjacent-tools surface.
|
|
|
|
The implication flips the framing of the unified-API work: it is not
|
|
"invent a shape both engines fit" — it is **"this shape both engines
|
|
can fit"** (pg-boss's queue model is *already* the native Postgres
|
|
tooling model; streams/offsets and notify have direct Postgres
|
|
counterparts) **and the work is pinning which parts of the shape are
|
|
the crate's contract** — the delivery-guarantee differences honker's
|
|
own guide documents per-binding (auto-checkpoint cadence vs manual
|
|
offset save; the processing-guarantees table) are exactly the seams a
|
|
single-crate version must clean up. Honker-rs is the concrete prior
|
|
art for that pinning exercise.
|
|
|
|
Postgres side: `pg_notify` gives fast triggers with no retry or
|
|
visibility semantics; pg-boss/Oban are the durable-layer gold standards
|
|
— "If you already run Postgres, use the Postgres tools."
|
|
|
|
The goal is **not** "port honker to Postgres." The goal is the interface:
|
|
one store API whose consumer code (queues, streams, notify) looks the
|
|
same whether the backing engine is SQLite or Postgres, while each engine
|
|
uses its own native wake/delivery story under the hood. The honker docs
|
|
recommend pgboss + `pg_notify` for the Postgres equivalent — that
|
|
recommendation is the design brief for this crate's Postgres engine —
|
|
and honker's Rust binding (`honker-rs`, v0.5.0) is the concrete
|
|
candidate for what that unified interface literally looks like (§Interface
|
|
finding).
|
|
|
|
**Consumer shape (from the paused alkblobs planning):** any crate that
|
|
today uses the ecosystem's repo-pattern + in-memory adapter should be
|
|
able to swap in an alkstore-backed engine and get durability + true
|
|
cross-process/cross-instance reactivity. That means the reactive surface
|
|
must compose with *client-side* caching: a subscriber that also holds a
|
|
cache can invalidate on notification instead of re-querying or re-polling
|
|
— the hot-path pattern the ecosystem already uses, given a real
|
|
invalidation source.
|
|
|
|
Guiding principles:
|
|
|
|
1. **One interface, two engines, native underneath.** The abstraction
|
|
layer unifies the consumer-visible features; the engines stay
|
|
dialects, not two emulations of one dialect. SQLite follows honker's
|
|
design (queue in the same file, same transaction, watcher-based
|
|
wake); Postgres follows pg-boss' design (schema-based job tables +
|
|
`pg_notify`-driven wake). A "lowest common denominator" unification
|
|
(both sides polling, both sides emulating LISTEN) is explicitly the
|
|
failure mode to avoid — it would re-create the fuzziness this crate
|
|
exists to remove.
|
|
2. **Transactional local-adjacency is the load-bearing property.**
|
|
Honker's core claim: enqueue/publish/notify in the same transaction
|
|
as the business write; rollback drops both. The unified surface must
|
|
preserve this on any engine, because the ecosystem's repo pattern
|
|
assumes it (a business write that loses its side-effect notification
|
|
is the dual-write problem honker names).
|
|
3. **Ownership of the whole stack.** honker is third-party; pgboss-rs
|
|
is third-party. Whether any of them are adopted, forked, or used as
|
|
schema/design reference only is a deliberate per-question decision —
|
|
not inherited by adjacency. The workspace precedent is the targeted
|
|
fork (alksocks' fast-socks5 extraction: adopt the design, own the
|
|
code, port to our conventions).
|
|
4. **Substrate-agnostic consumer API, engine-specific setup.** A
|
|
consumer opens a `Store` from a connection string / file path and
|
|
gets the same trait surface. Which engine is behind what can vary
|
|
(per-deployment config), but consumer code must not branch on
|
|
engine type.
|
|
5. **No panics, `tokio`, `thiserror`, lean base crate, feature-gated
|
|
optional engines** — family-standard, pre-committed (AGENTS.md).
|
|
|
|
## The driver conflict (Phase 0's central tension)
|
|
|
|
The immediate design fork, flagged by the user:
|
|
|
|
- **The alkblobs POCs used `tokio-postgres` + `deadpool-postgres`**
|
|
(validated in `poc-postgres-kv-findings.md`, `poc-pglo-findings.md` —
|
|
including Large Objects work).
|
|
- **pgboss-rs (`/workspace/pgboss-rs` @ 98f7d9e) uses `sqlx`** (`sqlx Postgres
|
|
runtime-tokio`).
|
|
- honker-core uses `rusqlite`.
|
|
|
|
A single crate with both engines means a driver decision, and the
|
|
reactivity story is entangled with it:
|
|
|
|
- **pgboss-rs currently has no `LISTEN`/`NOTIFY` at all** (verified
|
|
2026-10-03 against the checkout — `src/` contains no LISTEN/NOTIFY
|
|
usage; consumption is `fetch_job` polling). The node original relies
|
|
on `pg-boss`'s own maintenance/polling; the port did not pick up a
|
|
push channel. So *even* "use pgboss for the queue" does not deliver
|
|
reactivity — LISTEN/NOTIFY wiring would be new work either way, and
|
|
the driver choice determines *whose* LISTEN plumbing (sqlx's
|
|
`PgListener` is built-in; tokio-postgres uses its `Connection`
|
|
notifications).
|
|
- **Honker's reactivity on SQLite is a watcher polling `PRAGMA
|
|
data_version`** — a fundamentally different mechanism from
|
|
LISTEN/NOTIFY. The unified reactive trait must abstract over both
|
|
without collapsing to the polling behavior of the weaker side.
|
|
|
|
Two corrections to the original framing (2026-10-03, after reading
|
|
honker-rs and the honker.dev guides):
|
|
|
|
1. **The honker-rs *interface* is largely driver-independent.** The
|
|
Queue/Stream/notify/scheduler surface (§Interface finding) speaks in
|
|
domain terms (channels, offsets, job ids, visibility timeouts), not
|
|
driver terms. What *is* driver-coupled is the transactional seam:
|
|
`enqueue_tx`/`publish_tx`/`save_offset_tx` and the extension's
|
|
notify-in-transaction all assume the caller can hand the engine a
|
|
live transaction handle from *its* driver. The unified trait's
|
|
transactional seam is therefore the driver-sensitive design point —
|
|
and it interacts with the engines' different transaction models
|
|
(SQLite: single writer, synchronous; Postgres: interactive
|
|
transactions over a pool, transaction-scoped LISTEN).
|
|
2. **honker-rs is sync (`std` threads + blocking iterators) — parking_lot
|
|
+ rusqlite, no tokio.** The family standard is tokio-async. So even
|
|
the SQLite side is a *port-and-adapt* (sync → async), not an adopt;
|
|
and the driver question for SQLite is also entangled with whether
|
|
tokio-native sqlite drivers (sqlx sqlite) change the queue/wake
|
|
machinery's shape versus rusqlite-in-a-pool.
|
|
|
|
This tension is OQ-ST-04 below. It is *not* resolved by "pgboss is well
|
|
written so start there" — that is exactly the inherited-assumption
|
|
shape the SDD process flags. What pgboss-rs genuinely offers (schema
|
|
DDL, job states, retry semantics, the node-compatible API) is design
|
|
reference regardless of driver. The reactivity gap is the concrete,
|
|
verified difference between pgboss-rs and what this crate needs —
|
|
whether forked or re-derived, the push channel is work this crate
|
|
builds itself (OQ-ST-05).
|
|
|
|
## Prior art
|
|
|
|
Notes below are from reading the checkouts on 2026-10-03; both external
|
|
projects are reference checkouts — read freely, but not for direct use
|
|
as a dependency. We use the published version of anything that lives in
|
|
the global workspace unless we vendor or fork it (the alksocks
|
|
fast-socks5 precedent); if adoption ever requires a fork, forking is
|
|
normal work we own, not an exception. Provenance/licensing gets
|
|
recorded per AGENTS.md §3 when code is adopted, not while only reading.
|
|
|
|
### honker — the SQLite-side template
|
|
|
|
`/workspace/honker` (checkout @ f4e53c6; SQLite extension +
|
|
bindings). What matters for
|
|
this crate:
|
|
|
|
- **The full feature set to match on Postgres** (its §What It Does):
|
|
notify/listen across processes, durable at-least-once queues
|
|
(retries, delayed jobs, priority, visibility timeouts, dead-letter
|
|
rows, result storage), durable streams with per-consumer offsets,
|
|
cron/`@every` scheduling, named locks, rate limits, transactional
|
|
outbox helpers. Deliberately excluded there: workflow DAGs, task
|
|
chains/chords, multi-writer replication, cross-machine locking —
|
|
scope line likely inherited, to be confirmed.
|
|
- **The wake mechanism** — `PRAGMA data_version` polling watcher
|
|
(default 1 ms; raise for idle CPU), re-read indexed state after
|
|
wake, overtriggering on purpose ("one indexed SELECT is cheap; a
|
|
missed wake is a correctness bug."). Optional kernel-events and WAL
|
|
shared-memory backends exist in source builds.
|
|
- **Single-machine honesty** — file-backed, one host; NFS-two-writers
|
|
explicitly not supported. This posture needs an explicit Postgres
|
|
counterpart (multi-host is Postgres' normal case, so the interface
|
|
must not bake SQLite's single-host assumption into the shared
|
|
surface).
|
|
- **The transactional enqueue shape** — every feature is an INSERT
|
|
inside the caller's transaction. This is the pattern the unified API
|
|
must keep visible and cheap.
|
|
- **The honker-rs binding is the concrete interface prior art** (v0.5.0,
|
|
`packages/honker-rs`, read 2026-10-03): the full surface per §Interface
|
|
finding. Notable honest limitations documented by its own guides —
|
|
the per-binding processing-guarantees table (auto-checkpoint cadence
|
|
vs manual offset save; several bindings "may persist an offset on a
|
|
cadence... without knowing whether downstream application work
|
|
committed"), the Node reverse-order consumer-checkpoint bug, per-
|
|
binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++
|
|
missing typed pruning) — are exactly the seams a single-crate version
|
|
designed-for-the-contract from day one can clean up. Its sync-only
|
|
shape (std threads, blocking iterators, no tokio) is a
|
|
port-and-adapt constraint, not an adopt candidate as-is.
|
|
|
|
### pgboss-rs — the Postgres queue family reference
|
|
|
|
`/workspace/pgboss-rs` (checkout @ 98f7d9e; v0.1.0-rc6, MIT/Apache-2.0 dual). Ported from
|
|
node pg-boss: builder-based queue/job API, retry/delay/priority/
|
|
singleton/dead-letter concepts, `sqlx` 0.8, schema-scoped DDL.
|
|
|
|
- Verified gap (2026-10-03): **no LISTEN/NOTIFY** anywhere in `src/`
|
|
— consumption is polling `fetch_job`. Any push-reactivity is new
|
|
work, not an adoption freebie. This is the substantive difference
|
|
between pgboss-rs and what alkstore needs: regardless of fork vs
|
|
re-derive, reactivity is ours to build on the Postgres side either
|
|
way.
|
|
- Its value as reference: the pg-boss schema family (job states,
|
|
maintenance/dead-letter behavior) is battle-tested against real
|
|
Postgres semantics — worth borrowing *as design*, independent of the
|
|
driver decision.
|
|
- The node original (pg-boss) is the upstream of record for semantics
|
|
the port may have dropped; compare against it when adopting queue
|
|
semantics.
|
|
|
|
### Honker's Postgres-side recommendation
|
|
|
|
The honker README's own posture: if you run Postgres, use the Postgres
|
|
tools. `pg_notify` + pgboss is the recommended assembly. The design
|
|
brief: the queue machinery from the pg-boss family, the push semantics
|
|
from LISTEN/NOTIFY, the unified API shape from honker's Rust binding.
|
|
|
|
### The alk* repo pattern — what this crate replaces
|
|
|
|
The ecosystem's current shape: a repository trait with a default
|
|
in-memory adapter; cache-invalidation wiring in hot paths; uninvalidated
|
|
(non-reactive) caches where delay was acceptable; each project
|
|
composing these slightly differently. No persistence-backed reactive
|
|
substrate exists in the family — alkblobs was the first project to try
|
|
to plan against one, found it missing, and paused. This crate's reason
|
|
to exist is precisely that that substrate should exist once, well,
|
|
instead of per-project approximations.
|
|
|
|
### alkcall — the substrate (not a dependency of the store layer)
|
|
|
|
`/workspace/@alkdev/alkcall` (pure protocol crate, no transport). The
|
|
alk* crates (alktty, alktunnels, alksocks) are its consumers; a future
|
|
alkstore ops/protocol surface (if this crate ever exposes store access
|
|
over alkcall channels) rides the same substrate. Like the alkblobs
|
|
split (store layer stays substrate-free), the store layer here stays
|
|
alkcall-free; any networked surface is an ops module/sibling concern
|
|
and a separate decision.
|
|
|
|
## Open Questions
|
|
|
|
Register in `docs/research/phase-0.md`; IDs OQ-ST-NN (stable, append
|
|
only). Promotion target: Phase 1 `docs/architecture/open-questions.md`.
|
|
|
|
### OQ-ST-01: Scope boundary — which honker features are in-scope?
|
|
|
|
**Answered by the consumer inventory (2026-10-04) —
|
|
`docs/research/consumer-inventory.md`; scope votes now shrink to named
|
|
rows, not the whole feature list.** The original framing ("blocked on a
|
|
consumer-driven inventory pass") was circular hedging: the consumers
|
|
are paused, so the input would never arrive — but their *documents*
|
|
are stable evidence, and the inventory walks them per feature.
|
|
In-short: notify/listen and named locks are pinned or ADR-shaped
|
|
(alkfs path-tree invalidation; alkblobs fleet sweeper lock; alkfs
|
|
OQ-FS-05 writer coordination); queues and the outbox helper are
|
|
documented needs (alkfs sync/fetch-on-miss outbox; alkblobs
|
|
embedder-owned maintenance cadence); the scheduler is documented-thin
|
|
(the family-wide "who sweeps/renews/reaps" problem, possibly collapsing
|
|
into queues); streams was upgraded by an operator-authority record
|
|
(the 2026-10-04 correction to the inventory's initial read:
|
|
type-filtered event watching from several places — repo-change
|
|
subscriptions in a git app, cross-app event watching — a reactivity
|
|
requirement notify cannot serve honestly, being fire-and-forget);
|
|
rate limits and result storage have **no
|
|
named consumer** — carried per the keep-until-implementation posture
|
|
with cut-flags visible, cut later rather than silently included.
|
|
|
|
Per-feature exclusion lines (honker's own: DAGs, task chains/chords,
|
|
multi-writer replication, distributed locking) have no consumer either;
|
|
they stay out unless a consumer document grows one. New consumers
|
|
(alksftp, the alknet rewrite) add a row to the inventory *before* being
|
|
assumed into scope.
|
|
|
|
### OQ-ST-02: Crate scope — one store crate, or reactive-core + engines?
|
|
|
|
Options include: single crate with feature-gated engines (the alk*
|
|
feature-gate pattern); a core trait crate + per-engine crates; engine
|
|
crates consuming a thin core. The answer constrains the driver decision
|
|
(OQ-ST-03) and the base-crate-lean invariant.
|
|
|
|
**Resolved (2026-10-04, operator decision): reactive-core + engine
|
|
crates — a core crate carrying the trait surface/types, per-engine
|
|
crates implementing it (sqlite, postgres; mem-shaped test engine as a
|
|
third impl if useful).** The reasoning, recorded because it overrode
|
|
the inventory's lean: the split isolates the engines' real asymmetry of
|
|
work — the SQLite engine rides honker's existing machinery as the
|
|
baseline (port-and-adapt sync→async), the Postgres engine is the
|
|
build-heavy side (LISTEN/NOTIFY wiring + pg-boss-family schema work,
|
|
OQ-ST-05) — and it makes any future engine (alkfs's in-tree needs, an
|
|
ops-surface engine) additive rather than a feature-graph edit to one
|
|
crate. More future-proof by construction; the base-crate-lean
|
|
invariant becomes structural rather than a feature-discipline.
|
|
|
|
The inventory's uniform-feature-family fact (2026-10-04) still stands
|
|
and is not contradicted: it now reads as "the core contract can stay
|
|
small — one feature family, both engines" instead of "the crates should
|
|
merge." Its single-crate lean was inductive from that fact; the
|
|
structural reasoning above supersedes it (correction recorded in
|
|
consumer-inventory.md too).
|
|
|
|
### OQ-ST-03: Driver story — sqlx, tokio-postgres, or per-engine drivers?
|
|
|
|
The named tension (§The driver conflict), restated as the decision:
|
|
|
|
- **One driver across engines**: sqlx (both sqlite + postgres native
|
|
support, one API — but the alkblobs POC evidence is tokio-postgres)
|
|
vs tokio-postgres per-engine (sqlite story unclear — tokio-postgres
|
|
is pg-only; rusqlite is the sqlite native).
|
|
- **Per-engine drivers under a unified trait**: tokio-postgres +
|
|
deadpool-postgres (POC-validated in alkblobs findings) + rusqlite
|
|
(honker's choice, so honker's SQLite machinery ports cleanly).
|
|
- **Adopt/fork pgboss-rs**: brings sqlx along where the queue lives.
|
|
|
|
Honest unknowns worth surfacing: does sqlx support SQLite
|
|
`data_version`/extension-style machinery equally well? Does a unified
|
|
trait over `(tokio-postgres, rusqlite)` pay more trait-fitting cost
|
|
than sqlx's single-API convenience costs elsewhere? What does
|
|
SQLITE_ENABLE/extension loading look like under sqlx vs rusqlite?
|
|
(And the async question honker-rs forces, §driver-conflict correction
|
|
2: is rusqlite-in-a-pool the right tokio shape, or does sqlx-sqlite's
|
|
native async change the watcher machinery's design?)
|
|
|
|
**Family precedent bearing on the async sub-question (2026-10-04):
|
|
the async-facing-trait + sync-bridge posture is already family-standard,
|
|
twice over.** alktty REQ-TTY-01 (`/workspace/@alkdev/alktty/docs/
|
|
architecture/tty-backend.md` §REQ-TTY-01): "backends are not required
|
|
to be natively async" — the trait's adapter-facing types are the async
|
|
contract; a backend may expose blocking handles internally and bridge
|
|
them (blocking std::io on dedicated std threads or `spawn_blocking`,
|
|
feeding tokio mpsc/oneshot channels) as a **documented, supported
|
|
implementation strategy, not a workaround** (the wezterm/portable_pty
|
|
pattern; the local-PTY reference impl runs three bridge threads).
|
|
alkblobs store-api.md pins the same execution posture from the storage
|
|
side: blocking file work lives in `spawn_blocking` inside engine impls
|
|
(the alkgit trait-execution pattern) — the store never blocks the
|
|
executor. Together these reframe the honker-rs correction above: the
|
|
sync→async port's *shape* is less "rewrite onto a native-async driver"
|
|
and more "keep the sync machinery (rusqlite + watcher thread, blocking
|
|
iterators) and bridge at the trait seam" — which weakens sqlx's main
|
|
differentiator (native async) for the SQLite side specifically, and
|
|
makes the honest OQ-ST-03 comparison: bridge-posture rusqlite
|
|
(honker-machinery ports verbatim, bridge cost measured in threads not
|
|
rewrites) vs sqlx-sqlite (native async, but the watcher/extension
|
|
machinery must be re-validated against a driver that owns its
|
|
connection/pool internals differently).
|
|
|
|
Genuinely open; needs research rounds (library capabilities vs the
|
|
unified-trait shape) and possibly a POC. Not deferred — this is the
|
|
central Phase 0 research question.
|
|
|
|
### OQ-ST-04: The reactive abstraction — what does the unified notify surface look like?
|
|
|
|
The two engines' mechanisms are structurally different: SQLite =
|
|
watcher polling `PRAGMA data_version` (deliver on commit; no
|
|
server-side push exists), Postgres = LISTEN/NOTIFY (server push,
|
|
connection-bound, no retry/visibility semantics). The reactive trait
|
|
must have a shape both implement without one emulating the other's
|
|
weaknesses.
|
|
|
|
The honker-rs surface (§Interface finding) is the concrete starting
|
|
point — the question decomposes into contract-pinning rather than
|
|
shape-invention:
|
|
|
|
- Which parts of the honker-rs surface become the crate's *contract*:
|
|
the `notify`/`listen` pair, the stream/offset/consumer model (in
|
|
scope per the inventory — subscriptions are the durable reactivity
|
|
half notify can't serve), the queue claim/ack/visibility model,
|
|
locks, outbox, scheduler — all have named consumers now; rate-limits
|
|
have an in-crate alternative mechanism (alkgit's wire layer) —
|
|
subset, renamed/regrouped, decided against the inventory
|
|
rows rather than against the whole honker menu.
|
|
- What is the delivery-guarantee contract, per mechanism (honker's
|
|
own guide table shows how easily per-binding auto-checkpoint vs
|
|
manual-save ambiguity produces *different* guarantees under one
|
|
function name — the single-crate version must pick one answer, not
|
|
inherit the table)?
|
|
- Listener semantics: honker starts from `MAX(id)` and replays
|
|
nothing; Postgres LISTEN has no replay either but delivers via a
|
|
dedicated connection with its own lifecycle. Does `listen()`
|
|
abstract over both honestly (opaque wake + re-read contract) or
|
|
promise durability it only has on one engine (that's what streams
|
|
are for)?
|
|
- The transactional seam (`enqueue_tx`/`publish_tx`/`save_offset_tx`)
|
|
across two transaction models — the driver-coupled point (§driver-
|
|
conflict correction 1).
|
|
|
|
How does a caching subscriber (the ecosystem's hot-path pattern)
|
|
receive sufficient invalidation information (keys? table/channel
|
|
names? opaque wake + re-read contract?) — rides the same contract
|
|
decision.
|
|
|
|
Open; this is the second central research question, coupled to OQ-ST-03
|
|
(the driver determines what LISTEN plumbing exists).
|
|
|
|
### OQ-ST-05: Queue semantics — adopt, fork, or re-derive?
|
|
|
|
If queues land in scope (OQ-ST-01), the pg-boss schema family is the
|
|
Postgres-side incumbent and honker's queue design is the SQLite-side
|
|
one. Options: adopt pgboss-rs as a dependency (new feature-gated
|
|
option); targeted-fork the relevant subsystem (alksocks precedent,
|
|
ported to our conventions); schema/design-reference only (re-derive on
|
|
our driver). Fork-vs-derive depends on how much of pgboss-rs is
|
|
queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03), on
|
|
our tolerance for the alpha-state rc port, and on the verified gap
|
|
(§pgboss-rs): the push-reactivity half has to be built on top of any
|
|
choice, so the queue-machinery reuse value is the honest comparison
|
|
point, not the whole.
|
|
|
|
Open; inputs: OQ-ST-01's scope vote on queues (inventory: documented
|
|
need) + OQ-ST-03 resolution.
|
|
|
|
### OQ-ST-06: Honker relationship — reference, fork, or vendor?
|
|
|
|
Design-reference only (read, don't copy), targeted fork of
|
|
honker-core's engine machinery, or vendor the extension? Honker is
|
|
alpha-quality per its own README, MIT/Apache-2.0 dual-licensed, and
|
|
covers only the SQLite side — but it embodies exactly the watcher/
|
|
transactional design this crate wants on SQLite, and honker-rs
|
|
demonstrates the interface shape is sound. Three refinements from the
|
|
2026-10-03 reading:
|
|
|
|
- honker-rs is **sync-only** (std threads, blocking iterators) — the
|
|
tokio port is required work under any fork posture, which changes
|
|
the fork-vs-reference calculus (a fork is already a serious port).
|
|
- The crate likely needs only the core engine machinery (honker-core
|
|
minus the extension C surface — see OQ-ST-07), a smaller extraction
|
|
than the whole project.
|
|
- Honker's own documented per-binding inconsistencies (the
|
|
processing-guarantees table, OQ-ST-04) suggest extracting *design+
|
|
semantics* with our contract pinned, rather than preserving its
|
|
behavior verbatim — closer to the alkblobs "borrow conclusions, not
|
|
wire surface" principle than to alksocks' verbatim extraction.
|
|
|
|
Open; needs the license/provenance check (AGENTS.md §3) and a quality
|
|
assessment of honker-core's watcher/transactional core.
|
|
|
|
### OQ-ST-07: SQLite-side scope — loadable extension, embedded rusqlite, or both?
|
|
|
|
Honker ships as a loadable extension usable by *any* SQLite client,
|
|
plus per-language bindings. This crate (a Rust library) may not need
|
|
the loadable-extension surface at all — embedding the engine machinery
|
|
in-process may be the whole story (the honker-core shape minus the
|
|
extension/binding packaging). Determines how much of honker is even
|
|
candidate material.
|
|
|
|
**Sharpened by the inventory (2026-10-04):** every identified consumer
|
|
is in-process Rust attaching to its own connection (the
|
|
`honker-core`/`attach_honker_functions` shape — the alknet-filesystem
|
|
POC's actual usage). No consumer needs the loadable-extension surface.
|
|
The question is now cut-only: loadable extension is out unless a
|
|
consumer appears; the open residue is just how much of honker-core's
|
|
machinery survives the extraction.
|
|
|
|
### OQ-ST-08: Multi-host / deployment posture
|
|
|
|
Honker is explicitly single-machine (file-backed SQLite). Postgres is
|
|
natively multi-host. The unified surface must not pretend SQLite is
|
|
multi-host, but where does the honest boundary live — per-engine
|
|
capability flags? A documented deployment matrix? Does the trait need
|
|
to expose engine capabilities at all?
|
|
|
|
Open; partially rides OQ-ST-04 (the trait's shape constrains where
|
|
capability differences can surface).
|
|
|
|
## Phase 0 plan
|
|
|
|
Iteration expected; this register grows as research rounds land.
|
|
Expected sequence (deliberately rough):
|
|
|
|
1. ~~Consumer-driven scope inventory (OQ-ST-01)~~ — **done
|
|
(2026-10-04)**: `consumer-inventory.md`, run against the paused
|
|
consumers' documents (alkfs phase-0, alkgit architecture, alkblobs
|
|
architecture + the alknet-filesystem POC). OQ-ST-01 answered down
|
|
to named per-feature rows; OQ-ST-02/07 sharpened by it.
|
|
1. ~~Crate scope (OQ-ST-02)~~ — **resolved (2026-10-04, operator
|
|
decision)**: reactive-core + engine crates. Reasoning and the
|
|
superseded inventory lean recorded at the OQ and in the inventory.
|
|
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library
|
|
capability matrices, then a POC if the unified-trait shape needs
|
|
validation (likely, given the structural mismatch noted in OQ-ST-04).
|
|
The honker-rs surface (§Interface finding) is the concrete starting
|
|
artifact for the OQ-ST-04 work: pinning its contract costs less and
|
|
is more honest than inventing a parallel shape — now scoped against
|
|
the inventory's confirmed features rather than the full honker menu,
|
|
and shaped as the core-crate trait surface per OQ-ST-02's split.
|
|
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem,
|
|
after the driver and shape questions narrow the option space.
|
|
4. Converge; Phase 1 opens with the ADR backlog this register becomes.
|
|
|
|
## References
|
|
|
|
- honker — `/workspace/honker` (git checkout @ f4e53c6 of
|
|
github.com/russellromney/honker; README +
|
|
`honker-core/src/` read 2026-10-03): the SQLite-side
|
|
feature/wake template. The four
|
|
guides (queues/streams/pubsub/scheduler on honker.dev) +
|
|
`packages/honker-rs/src/lib.rs` (v0.5.0) are the interface prior art
|
|
(§Interface finding).
|
|
- pgboss-rs — `/workspace/pgboss-rs` (git checkout @ 98f7d9e of
|
|
github.com/rustworthy/pgboss-rs, v0.1.0-rc6; read 2026-10-03,
|
|
LISTEN/NOTIFY-absence verified): the Postgres queue-family reference.
|
|
- honker's own prior-art section: pg_notify, pg-boss, Oban, Huey — the
|
|
external lineage this crate inherits from both sides.
|
|
- alkblobs — `/workspace/@alkdev/alkblobs` (spec+POCs, paused): the
|
|
paused planning this crate unblocks; its POC findings
|
|
(`poc-postgres-kv-findings.md`, `poc-pglo-findings.md`) are the
|
|
tokio-postgres evidence base.
|
|
- alkgit — `/workspace/@alkdev/alkgit` (paused mid-Phase-1,
|
|
architecture reviewed): its backend.md trait seam and ADR set are
|
|
consumer evidence for the inventory (queues/locks rows).
|
|
- alkfs — `/workspace/@alkdev/alkfs` (Phase 0 drafted, 2026-09-23): its
|
|
phase-0 OQs (OQ-FS-05/07/14/16/17) are consumer evidence for the
|
|
inventory (notify/locks/queues rows).
|
|
- alknet-filesystem POC —
|
|
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/
|
|
poc-summary.md`: the ran-once evidence that the honker-coordination
|
|
layer works (notify-on-commit test; named-locks and outbox usage
|
|
identified).
|
|
- consumer-inventory.md (`docs/research/consumer-inventory.md`) — the
|
|
per-feature synthesis (2026-10-04) answering OQ-ST-01 from the above.
|
|
- alkcall — `/workspace/@alkdev/alkcall`: the family substrate;
|
|
referenced for the store-layer-isolation principle only.
|
|
- alktty — `/workspace/@alkdev/alktty` (architecture reviewed): REQ-TTY-01
|
|
(`docs/architecture/tty-backend.md`) — the async-facing-trait +
|
|
sync-bridge posture ("backends are not required to be natively
|
|
async"), the family precedent bearing on OQ-ST-03's async
|
|
sub-question. |