Files
alkstore/docs/research/phase-0.md
T
glm-5.3-flash 299603b164 docs: POC #1 findings land — SQLite posture resolved (Arm A: honker-core on our rusqlite)
Findings (poc-sqlite-posture-findings.md, run in a parallel session;
tests re-verified in this session — 4 passing): all three of Arm A's
gate conditions fired in its favor — bridged rusqlite ~2x sqlx
native-async at p50 (B's premise measured false), honker-core's
inherited watcher tighter than a re-derived one (p50 1.40 vs 2.15 ms,
max 29 vs 172 ms, battle-tested failure handling), and the .so runtime
dependency is packaging cost with no compensating advantage.
Transactional property holds identically on both (SQLite's property,
not the posture's). Constraints recorded: honker-core 0.5.0 pins
rusqlite ^0.40.1 (rustc >=1.99); mixed rusqlite+sqlx binaries need a
vendored libsqlite3-sys patch (OQ-ST-02's per-engine-crate split keeps
the engine binary single-driver). Fixed the findings' test-count
discrepancy (4 tests, verified running). Phase-0: OQ-ST-03 SQLite half
resolved (pg half remains), OQ-ST-04/05/06 carry POC input, register
row gains findings link + status, plan step 2 split into done/next,
frontmatter updated, POC crate added to references.
2026-10-04 15:13:25 +00:00

723 lines
39 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
status: draft
last_updated: 2026-10-04 (consumer inventory landed — OQ-ST-01 answered
per-feature from the paused consumers' documents; phase-0 plan step 1
done; streams upgraded to in-scope by operator-authority record; OQ-ST-02
resolved: reactive-core + engine crates, operator decision; OQ-ST-03's
SQLite half resolved by POC #1 (honker-core on our rusqlite — findings
in poc-sqlite-posture-findings.md); OQ-ST-04/05/06 carry POC #1 input.
Remaining: OQ-ST-03's Postgres half, OQ-ST-04 contract pinning,
OQ-ST-05/06 pg-side ownership, OQ-ST-07/08.) Interface finding and
driver tension from 2026-10-03 remain trusted-but-unverified working
input except where POC #1 verified them.)
---
# alkstore — Phase 0 (Exploration)
This document captures Phase 0 (Exploration) for the `alkstore` crate:
vision, guiding principles, prior art, the open-question register
(OQ-ST-01..NN), and the POC register. Phase 0's objective per
`docs/sdd_process.md`: *capture
vision and guiding principles; research options; validate approaches;
converge on a recommended approach.* The scope question (OQ-ST-01) has
been answered per-feature from the consumers' documents
(`consumer-inventory.md`); the driver question (OQ-ST-03) has a named
SQLite option space with POC #1 specified against it; the reactive
contract (OQ-ST-04) and ownership questions (OQ-ST-05/06) are the open
work the research rounds feed.
Context for why this crate starts now: **alkblobs**
(`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning)
hit repeated circular hedging in its Phase 0, and a root cause was that
the storage substrate it deploys onto was itself disjoint and fuzzy — a
repo pattern with a default in-memory adapter across the alk* ecosystem,
cache-invalidation patches in hot paths, non-invalidated caches where
delay was tolerable, no single definition of "how does a change in the
database become visible to other processes/connections?" alkblobs paused
partly to let this crate answer that first. alkstore is the attempt to
make that substrate real once, so downstream stores don't re-derive it.
## Vision and guiding principles
**One sentence (draft):** one reactive store interface over SQLite and
Postgres — durable pub/sub notify, queues, streams, and the transactional
integration (write + enqueue in one transaction) that honker delivers on
SQLite — with the Postgres side building on the natively-available
machinery (`pg_notify`/`LISTEN`, and the pgboss job-queue schema family)
rather than emulating it.
**The honker relationship.** `/workspace/honker` (reference checkout,
not for direct use as a dependency; alpha-quality per its own README,
MIT/Apache-2.0 dual) adds
Postgres-style `NOTIFY`/`LISTEN` semantics to SQLite without a broker:
durable at-least-once queues with retries/delay/priority/visibility
timeouts/dead-letter, durable streams with per-consumer offsets,
cron/`@every` scheduling, named locks, rate limits, transactional
outbox — all as INSERTs inside the caller's transaction, with the
cross-process wake delivered by a shared watcher that polls `PRAGMA
data_version` (default 1 ms → single-digit-ms delivery) and re-reads
indexed state after every wake. Its own Prior Art section names the
lineage: `pg_notify`, pg-boss, Oban, Huey.
**The interface finding (2026-10-03, from the four honker.dev guides +
`packages/honker-rs/src/lib.rs`, the Rust binding): honker's *surface*
is already the unified-interface candidate.** Its Rust binding exposes
exactly the surface this crate wants, engine-clean:
- `db.queue(name, QueueOpts)` → `enqueue / enqueue_tx / claim_one /
claim_batch / ack_batch / cancel / get_job / sweep_expired /
claim_waker`, with `job.ack / retry / fail / heartbeat` and
`EnqueueOpts {delay, priority, max_attempts, expires, ...}` —
semantically the pg-boss model (visibility timeouts, retries,
dead-letter via move-to-`_honker_dead`), not a LISTEN-emulation.
- `db.stream(name)` → `publish / publish_tx / publish_with_key /
read_since / read_from_consumer / save_offset(_tx) / get_offset /
subscribe(consumer)` — offsets are explicit, transaction-aware
(`save_offset_tx`) for the exactly-once-within-a-business-tx shape,
and replay-on-reconnect is the default.
- `db.notify(channel, payload)` / `notify_tx` / `db.listen(channel)` —
the `pg_notify`-analogue fire-and-forget signal layer ("fire-and-forget,
no replay, no guarantees" — streams are the durable cousin), listener
starts from `MAX(id)` at attach, no historical replay.
- `db.scheduler()` → `add/pause/resume/update/list/remove/tick/run` —
cron + `@every` enqueueing into named queues, leader-elected via
advisory lock with TTL heartbeat, missed-boundary catch-up.
- `db.outbox(name)` — the transactional outbox helper (enqueue +
`run_once` delivery worker).
- `db.try_lock / try_rate_limit / save_result / get_result /
sweep_results` — the coordination/adjacent-tools surface.
The implication flips the framing of the unified-API work: it is not
"invent a shape both engines fit" — it is **"this shape both engines
can fit"** (pg-boss's queue model is *already* the native Postgres
tooling model; streams/offsets and notify have direct Postgres
counterparts) **and the work is pinning which parts of the shape are
the crate's contract** — the delivery-guarantee differences honker's
own guide documents per-binding (auto-checkpoint cadence vs manual
offset save; the processing-guarantees table) are exactly the seams a
single-crate version must clean up. Honker-rs is the concrete prior
art for that pinning exercise.
Postgres side: `pg_notify` gives fast triggers with no retry or
visibility semantics; pg-boss/Oban are the durable-layer gold standards
— "If you already run Postgres, use the Postgres tools."
The goal is **not** "port honker to Postgres." The goal is the interface:
one store API whose consumer code (queues, streams, notify) looks the
same whether the backing engine is SQLite or Postgres, while each engine
uses its own native wake/delivery story under the hood. The honker docs
recommend pgboss + `pg_notify` for the Postgres equivalent — that
recommendation is the design brief for this crate's Postgres engine —
and honker's Rust binding (`honker-rs`, v0.5.0) is the concrete
candidate for what that unified interface literally looks like (§Interface
finding).
**Consumer shape (from the paused alkblobs planning):** any crate that
today uses the ecosystem's repo-pattern + in-memory adapter should be
able to swap in an alkstore-backed engine and get durability + true
cross-process/cross-instance reactivity. That means the reactive surface
must compose with *client-side* caching: a subscriber that also holds a
cache can invalidate on notification instead of re-querying or re-polling
— the hot-path pattern the ecosystem already uses, given a real
invalidation source.
Guiding principles:
1. **One interface, two engines, native underneath.** The abstraction
layer unifies the consumer-visible features; the engines stay
dialects, not two emulations of one dialect. SQLite follows honker's
design (queue in the same file, same transaction, watcher-based
wake); Postgres follows pg-boss' design (schema-based job tables +
`pg_notify`-driven wake). A "lowest common denominator" unification
(both sides polling, both sides emulating LISTEN) is explicitly the
failure mode to avoid — it would re-create the fuzziness this crate
exists to remove.
2. **Transactional local-adjacency is the load-bearing property.**
Honker's core claim: enqueue/publish/notify in the same transaction
as the business write; rollback drops both. The unified surface must
preserve this on any engine, because the ecosystem's repo pattern
assumes it (a business write that loses its side-effect notification
is the dual-write problem honker names).
3. **Ownership of the whole stack.** honker is third-party; pgboss-rs
is third-party. Whether any of them are adopted, forked, or used as
schema/design reference only is a deliberate per-question decision —
not inherited by adjacency. The workspace precedent is the targeted
fork (alksocks' fast-socks5 extraction: adopt the design, own the
code, port to our conventions).
4. **Substrate-agnostic consumer API, engine-specific setup.** A
consumer opens a `Store` from a connection string / file path and
gets the same trait surface. Which engine is behind what can vary
(per-deployment config), but consumer code must not branch on
engine type.
5. **No panics, `tokio`, `thiserror`, lean base crate, feature-gated
optional engines** — family-standard, pre-committed (AGENTS.md).
## The driver conflict (Phase 0's central tension)
The immediate design fork, flagged by the user:
- **The alkblobs POCs used `tokio-postgres` + `deadpool-postgres`**
(validated in `poc-postgres-kv-findings.md`, `poc-pglo-findings.md` —
including Large Objects work).
- **pgboss-rs (`/workspace/pgboss-rs` @ 98f7d9e) uses `sqlx`** (`sqlx Postgres
runtime-tokio`).
- honker-core uses `rusqlite`.
A single crate with both engines means a driver decision, and the
reactivity story is entangled with it:
- **pgboss-rs currently has no `LISTEN`/`NOTIFY` at all** (verified
2026-10-03 against the checkout — `src/` contains no LISTEN/NOTIFY
usage; consumption is `fetch_job` polling). The node original relies
on `pg-boss`'s own maintenance/polling; the port did not pick up a
push channel. So *even* "use pgboss for the queue" does not deliver
reactivity — LISTEN/NOTIFY wiring would be new work either way, and
the driver choice determines *whose* LISTEN plumbing (sqlx's
`PgListener` is built-in; tokio-postgres uses its `Connection`
notifications).
- **Honker's reactivity on SQLite is a watcher polling `PRAGMA
data_version`** — a fundamentally different mechanism from
LISTEN/NOTIFY. The unified reactive trait must abstract over both
without collapsing to the polling behavior of the weaker side.
Two corrections to the original framing (2026-10-03, after reading
honker-rs and the honker.dev guides):
1. **The honker-rs *interface* is largely driver-independent.** The
Queue/Stream/notify/scheduler surface (§Interface finding) speaks in
domain terms (channels, offsets, job ids, visibility timeouts), not
driver terms. What *is* driver-coupled is the transactional seam:
`enqueue_tx`/`publish_tx`/`save_offset_tx` and the extension's
notify-in-transaction all assume the caller can hand the engine a
live transaction handle from *its* driver. The unified trait's
transactional seam is therefore the driver-sensitive design point —
and it interacts with the engines' different transaction models
(SQLite: single writer, synchronous; Postgres: interactive
transactions over a pool, transaction-scoped LISTEN).
2. **honker-rs is sync (`std` threads + blocking iterators) — parking_lot
+ rusqlite, no tokio.** The family standard is tokio-async. So even
the SQLite side is a *port-and-adapt* (sync → async), not an adopt;
and the driver question for SQLite is also entangled with whether
tokio-native sqlite drivers (sqlx sqlite) change the queue/wake
machinery's shape versus rusqlite-in-a-pool.
This tension is OQ-ST-04 below. It is *not* resolved by "pgboss is well
written so start there" — that is exactly the inherited-assumption
shape the SDD process flags. What pgboss-rs genuinely offers (schema
DDL, job states, retry semantics, the node-compatible API) is design
reference regardless of driver. The reactivity gap is the concrete,
verified difference between pgboss-rs and what this crate needs —
whether forked or re-derived, the push channel is work this crate
builds itself (OQ-ST-05).
## Prior art
Notes below are from reading the checkouts on 2026-10-03; both external
projects are reference checkouts — read freely, but not for direct use
as a dependency. We use the published version of anything that lives in
the global workspace unless we vendor or fork it (the alksocks
fast-socks5 precedent); if adoption ever requires a fork, forking is
normal work we own, not an exception. Provenance/licensing gets
recorded per AGENTS.md §3 when code is adopted, not while only reading.
### honker — the SQLite-side template
`/workspace/honker` (checkout @ f4e53c6; SQLite extension +
bindings). What matters for
this crate:
- **The full feature set to match on Postgres** (its §What It Does):
notify/listen across processes, durable at-least-once queues
(retries, delayed jobs, priority, visibility timeouts, dead-letter
rows, result storage), durable streams with per-consumer offsets,
cron/`@every` scheduling, named locks, rate limits, transactional
outbox helpers. Deliberately excluded there: workflow DAGs, task
chains/chords, multi-writer replication, cross-machine locking —
scope line likely inherited, to be confirmed.
- **The wake mechanism** — `PRAGMA data_version` polling watcher
(default 1 ms; raise for idle CPU), re-read indexed state after
wake, overtriggering on purpose ("one indexed SELECT is cheap; a
missed wake is a correctness bug."). Optional kernel-events and WAL
shared-memory backends exist in source builds.
- **Single-machine honesty** — file-backed, one host; NFS-two-writers
explicitly not supported. This posture needs an explicit Postgres
counterpart (multi-host is Postgres' normal case, so the interface
must not bake SQLite's single-host assumption into the shared
surface).
- **The transactional enqueue shape** — every feature is an INSERT
inside the caller's transaction. This is the pattern the unified API
must keep visible and cheap.
- **The honker-rs binding is the concrete interface prior art** (v0.5.0,
`packages/honker-rs`, read 2026-10-03): the full surface per §Interface
finding. Notable honest limitations documented by its own guides —
the per-binding processing-guarantees table (auto-checkpoint cadence
vs manual offset save; several bindings "may persist an offset on a
cadence... without knowing whether downstream application work
committed"), the Node reverse-order consumer-checkpoint bug, per-
binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++
missing typed pruning) — are exactly the seams a single-crate version
designed-for-the-contract from day one can clean up. Its sync-only
shape (std threads, blocking iterators, no tokio) is a
port-and-adapt constraint, not an adopt candidate as-is.
### pgboss-rs — the Postgres queue family reference
`/workspace/pgboss-rs` (checkout @ 98f7d9e; v0.1.0-rc6, MIT/Apache-2.0 dual). Ported from
node pg-boss: builder-based queue/job API, retry/delay/priority/
singleton/dead-letter concepts, `sqlx` 0.8, schema-scoped DDL.
- Verified gap (2026-10-03): **no LISTEN/NOTIFY** anywhere in `src/`
— consumption is polling `fetch_job`. Any push-reactivity is new
work, not an adoption freebie. This is the substantive difference
between pgboss-rs and what alkstore needs: regardless of fork vs
re-derive, reactivity is ours to build on the Postgres side either
way.
- Its value as reference: the pg-boss schema family (job states,
maintenance/dead-letter behavior) is battle-tested against real
Postgres semantics — worth borrowing *as design*, independent of the
driver decision.
- The node original (pg-boss) is the upstream of record for semantics
the port may have dropped; compare against it when adopting queue
semantics.
### Honker's Postgres-side recommendation
The honker README's own posture: if you run Postgres, use the Postgres
tools. `pg_notify` + pgboss is the recommended assembly. The design
brief: the queue machinery from the pg-boss family, the push semantics
from LISTEN/NOTIFY, the unified API shape from honker's Rust binding.
### The alk* repo pattern — what this crate replaces
The ecosystem's current shape: a repository trait with a default
in-memory adapter; cache-invalidation wiring in hot paths; uninvalidated
(non-reactive) caches where delay was acceptable; each project
composing these slightly differently. No persistence-backed reactive
substrate exists in the family — alkblobs was the first project to try
to plan against one, found it missing, and paused. This crate's reason
to exist is precisely that that substrate should exist once, well,
instead of per-project approximations.
### alkcall — the substrate (not a dependency of the store layer)
`/workspace/@alkdev/alkcall` (pure protocol crate, no transport). The
alk* crates (alktty, alktunnels, alksocks) are its consumers; a future
alkstore ops/protocol surface (if this crate ever exposes store access
over alkcall channels) rides the same substrate. Like the alkblobs
split (store layer stays substrate-free), the store layer here stays
alkcall-free; any networked surface is an ops module/sibling concern
and a separate decision.
## Open Questions
Register in `docs/research/phase-0.md`; IDs OQ-ST-NN (stable, append
only). Promotion target: Phase 1 `docs/architecture/open-questions.md`.
### OQ-ST-01: Scope boundary — which honker features are in-scope?
**Answered by the consumer inventory (2026-10-04) —
`docs/research/consumer-inventory.md`; scope votes now shrink to named
rows, not the whole feature list.** The original framing ("blocked on a
consumer-driven inventory pass") was circular hedging: the consumers
are paused, so the input would never arrive — but their *documents*
are stable evidence, and the inventory walks them per feature.
In-short: notify/listen and named locks are pinned or ADR-shaped
(alkfs path-tree invalidation; alkblobs fleet sweeper lock; alkfs
OQ-FS-05 writer coordination); queues and the outbox helper are
documented needs (alkfs sync/fetch-on-miss outbox; alkblobs
embedder-owned maintenance cadence); the scheduler is documented-thin
(the family-wide "who sweeps/renews/reaps" problem, possibly collapsing
into queues); streams was upgraded by an operator-authority record
(the 2026-10-04 correction to the inventory's initial read:
type-filtered event watching from several places — repo-change
subscriptions in a git app, cross-app event watching — a reactivity
requirement notify cannot serve honestly, being fire-and-forget);
rate limits and result storage have **no
named consumer** — carried per the keep-until-implementation posture
with cut-flags visible, cut later rather than silently included.
Per-feature exclusion lines (honker's own: DAGs, task chains/chords,
multi-writer replication, distributed locking) have no consumer either;
they stay out unless a consumer document grows one. New consumers
(alksftp, the alknet rewrite) add a row to the inventory *before* being
assumed into scope.
### OQ-ST-02: Crate scope — one store crate, or reactive-core + engines?
Options include: single crate with feature-gated engines (the alk*
feature-gate pattern); a core trait crate + per-engine crates; engine
crates consuming a thin core. The answer constrains the driver decision
(OQ-ST-03) and the base-crate-lean invariant.
**Resolved (2026-10-04, operator decision): reactive-core + engine
crates — a core crate carrying the trait surface/types, per-engine
crates implementing it (sqlite, postgres; mem-shaped test engine as a
third impl if useful).** The reasoning, recorded because it overrode
the inventory's lean: the split isolates the engines' real asymmetry of
work — the SQLite engine rides honker's existing machinery as the
baseline (port-and-adapt sync→async), the Postgres engine is the
build-heavy side (LISTEN/NOTIFY wiring + pg-boss-family schema work,
OQ-ST-05) — and it makes any future engine (alkfs's in-tree needs, an
ops-surface engine) additive rather than a feature-graph edit to one
crate. More future-proof by construction; the base-crate-lean
invariant becomes structural rather than a feature-discipline.
The inventory's uniform-feature-family fact (2026-10-04) still stands
and is not contradicted: it now reads as "the core contract can stay
small — one feature family, both engines" instead of "the crates should
merge." Its single-crate lean was inductive from that fact; the
structural reasoning above supersedes it (correction recorded in
consumer-inventory.md too).
### OQ-ST-03: Driver story — sqlx, tokio-postgres, or per-engine drivers?
The named tension (§The driver conflict), restated as the decision:
- **One driver across engines**: sqlx (both sqlite + postgres native
support, one API — but the alkblobs POC evidence is tokio-postgres)
vs tokio-postgres per-engine (sqlite story unclear — tokio-postgres
is pg-only; rusqlite is the sqlite native).
- **Per-engine drivers under a unified trait**: tokio-postgres +
deadpool-postgres (POC-validated in alkblobs findings) + rusqlite
(honker's choice, so honker's SQLite machinery ports cleanly).
- **Adopt/fork pgboss-rs**: brings sqlx along where the queue lives.
Honest unknowns worth surfacing: does sqlx support SQLite
`data_version`/extension-style machinery equally well? Does a unified
trait over `(tokio-postgres, rusqlite)` pay more trait-fitting cost
than sqlx's single-API convenience costs elsewhere? What does
SQLITE_ENABLE/extension loading look like under sqlx vs rusqlite?
(And the async question honker-rs forces, §driver-conflict correction
2: is rusqlite-in-a-pool the right tokio shape, or does sqlx-sqlite's
native async change the watcher machinery's design?)
**Family precedent bearing on the async sub-question (2026-10-04):
the async-facing-trait + sync-bridge posture is already family-standard,
twice over.** alktty REQ-TTY-01 (`/workspace/@alkdev/alktty/docs/
architecture/tty-backend.md` §REQ-TTY-01): "backends are not required
to be natively async" — the trait's adapter-facing types are the async
contract; a backend may expose blocking handles internally and bridge
them (blocking std::io on dedicated std threads or `spawn_blocking`,
feeding tokio mpsc/oneshot channels) as a **documented, supported
implementation strategy, not a workaround** (the wezterm/portable_pty
pattern; the local-PTY reference impl runs three bridge threads).
alkblobs store-api.md pins the same execution posture from the storage
side: blocking file work lives in `spawn_blocking` inside engine impls
(the alkgit trait-execution pattern) — the store never blocks the
executor. Together these reframe the honker-rs correction above: the
sync→async port's *shape* is less "rewrite onto a native-async driver"
and more "keep the sync machinery (rusqlite + watcher thread, blocking
iterators) and bridge at the trait seam" — which weakens sqlx's main
differentiator (native async) for the SQLite side specifically, and
makes the honest OQ-ST-03 comparison: bridge-posture rusqlite
(honker-machinery ports verbatim, bridge cost measured in threads not
rewrites) vs sqlx-sqlite (native async, but the watcher/extension
machinery must be re-validated against a driver that owns its
connection/pool internals differently).
Genuinely open; needs research rounds (library capabilities vs the
unified-trait shape) and possibly a POC. Not deferred — this is the
central Phase 0 research question.
**The SQLite option space, named explicitly (2026-10-04, operator +
verified against the checkout @ f4e53c6):** three distinct postures, not
one "rusqlite vs sqlx" axis —
1. **honker-core on our own rusqlite connection** (the
`attach_honker_functions` shape — the alknet-filesystem POC's actual
usage): we own the connection, the schema bootstrap, and the
watcher; honker supplies the SQL-function machinery.
2. **honker-rs as the crate's SQLite substrate** (`Database::open`,
typed Queue/Stream/Transaction primitives; the guide's "it *is* the
integration" posture): the trade is ownership — honker-rs opens and
holds its own connections, its `Database` wraps a connection mutex
(transactions pin the mutex; same-thread `*_tx` methods only,
deadlock-by-mutex on cross-thread reuse), and the whole engine is
sync under OUR async core (bridge at every seam). Maximum reuse,
least control; the transactional seam inherits honker-rs's
mutex-pinned transaction model rather than ours.
3. **raw SQL over sqlx-sqlite with the honker loadable extension**
(guides/orm/rust §sqlx: `SqliteConnectOptions::extension(ext)` +
`SELECT honker_bootstrap()`, then every feature is plain SQL —
`honker_enqueue`, `notify`, ... — callable through
`SqliteExecutor<'e>`, satisfied by pool, connection, AND
Transaction alike). **Verified in-harness**: the pattern is CI-proven
in the honker checkout itself (scripts/proof/orm/rust — async
business-write + `honker_enqueue` inside a `conn.begin()` tx,
commit-visibility + rollback-drops-job asserted), so the
transactional property holds *natively async* here — no sync bridge
on the call path at all.
Option 3 is the genuinely interesting one: it dissolves most of the
async tension for the SQLite engine (native-async calls, sqlx-owned
pooling, transactional enqueue proven in the checkout's own proof
suite), keeps honker's machinery as a library-free extension artifact
we don't own code-wise, and moves ALL our custom logic (watcher
ownership, stream cursors, extra SQL) into our own crate. The cost
side: a runtime-loaded .so (build-or-download per the guide) is a
packaging/deployment dependency the other options don't have; sqlx's
load-then-disable extension discipline is worth pinning at the source
(the guide documents it — `SqliteConnectOptions::extension` loads
during connect only, then disables the C load-extension API); and the
watcher (whose machinery otherwise rides option 1/2's honker-core
library linkage) becomes OUR component watching a sqlx-managed pool —
`PRAGMA data_version` re-read design (OQ-ST-04's SQLite wake side)
lands in our lap either way, but under option 3 we can't lean on
honker-core's `SharedUpdateWatcher` thread shape as-is. Options are not
mutually exclusive across engines: option 3 (or 1) for SQLite is
compatible with tokio-postgres for the Postgres engine under the
OQ-ST-02 split.
This expands OQ-ST-03's option list; the comparison matrix ("honker's
machinery as a linked library on our connection" vs "honker's machinery
as a loaded extension under sqlx") is now the concrete research round,
and a POC comparing options 1 and 3 directly (same queue/stream/notify
surface both ways, incl. watcher wiring) is the natural first POC for
the register.
**SQLite half resolved by POC #1 (2026-10-04; findings:
`poc-sqlite-posture-findings.md`): posture 1 — honker-core on our
rusqlite.** All three gate conditions fired in A's favor, mildly: the
bridged path is ~2× sqlx's native-async at p50 (0.354 vs 0.707 ms on
the tx-enqueue workload; B's premise measured false), honker-core's
inherited watcher is tighter than a re-derived one (p50 1.40 vs 2.15
ms, max 29 vs 172 ms, with battle-tested failure handling), and the
`.so` runtime dependency is packaging cost A doesn't pay for no
compensating advantage. The transactional contract holds identically
on both (it is SQLite's property, not the posture's). The Postgres
half remains the open part of OQ-ST-03 (tokio-postgres per the POC
evidence base is the working lean; the OQ is not closed until the pg
side's LISTEN/pool/tx-seam story is validated the same way). New
constraints recorded for the engine crate regardless:
honker-core=0.5.0 pins rusqlite ^0.40.1 whose rustc requirement
(≥1.99) is a deployment note, and mixed rusqlite+sqlx binaries
currently need a vendored one-line libsqlite3-sys patch — OQ-ST-02's
per-engine-crate split is what keeps the engine binary single-driver.
### OQ-ST-04: The reactive abstraction — what does the unified notify surface look like?
The two engines' mechanisms are structurally different: SQLite =
watcher polling `PRAGMA data_version` (deliver on commit; no
server-side push exists), Postgres = LISTEN/NOTIFY (server push,
connection-bound, no retry/visibility semantics). The reactive trait
must have a shape both implement without one emulating the other's
weaknesses.
The honker-rs surface (§Interface finding) is the concrete starting
point — the question decomposes into contract-pinning rather than
shape-invention:
- Which parts of the honker-rs surface become the crate's *contract*:
the `notify`/`listen` pair, the stream/offset/consumer model (in
scope per the inventory — subscriptions are the durable reactivity
half notify can't serve), the queue claim/ack/visibility model,
locks, outbox, scheduler — all have named consumers now; rate-limits
have an in-crate alternative mechanism (alkgit's wire layer) —
subset, renamed/regrouped, decided against the inventory
rows rather than against the whole honker menu.
- What is the delivery-guarantee contract, per mechanism (honker's
own guide table shows how easily per-binding auto-checkpoint vs
manual-save ambiguity produces *different* guarantees under one
function name — the single-crate version must pick one answer, not
inherit the table)?
- Listener semantics: honker starts from `MAX(id)` and replays
nothing; Postgres LISTEN has no replay either but delivers via a
dedicated connection with its own lifecycle. Does `listen()`
abstract over both honestly (opaque wake + re-read contract) or
promise durability it only has on one engine (that's what streams
are for)?
- The transactional seam (`enqueue_tx`/`publish_tx`/`save_offset_tx`)
across two transaction models — the driver-coupled point (§driver-
conflict correction 1).
How does a caching subscriber (the ecosystem's hot-path pattern)
receive sufficient invalidation information (keys? table/channel
names? opaque wake + re-read contract?) — rides the same contract
decision.
**POC #1 input (2026-10-04):** the opaque-wake + re-read contract
survives both SQLite postures unchanged (same `data_version` mechanism
underneath — wake coalescing verified correct by design, p50
1.4–2.2 ms at the default 1 ms cadence, missed-wake stress passes with
correct re-reads on both); listener semantics (start at MAX(id), no
replay) unchanged. The `*_tx` seam question crystallized into
something concrete: the SQLite-side tx handle is an owned writer-slot
lease whose ops each ride `spawn_blocking` (the connection is not
`Sync`; holding it across await points is wrong) — Phase 1's contract
work starts from that shape plus the POC's `TxHandle` sketch (with its
two recorded frictions: the `as_any_mut` downcast and the
thread-affinity of rusqlite tx ops). What remains open is everything
Postgres (transaction-scoped LISTEN over a pool, and the delivery-
guarantee/unification bullets above).
Open; this is the second central research question, coupled to OQ-ST-03
(the driver determines what LISTEN plumbing exists) — SQLite side now
de-risked, Postgres side is the remaining shape work.
### OQ-ST-05: Queue semantics — adopt, fork, or re-derive?
If queues land in scope (OQ-ST-01), the pg-boss schema family is the
Postgres-side incumbent and honker's queue design is the SQLite-side
one. Options: adopt pgboss-rs as a dependency (new feature-gated
option); targeted-fork the relevant subsystem (alksocks precedent,
ported to our conventions); schema/design-reference only (re-derive on
our driver). Fork-vs-derive depends on how much of pgboss-rs is
queue-machinery vs driver-wiring (the sqlx coupling — OQ-ST-03), on
our tolerance for the alpha-state rc port, and on the verified gap
(§pgboss-rs): the push-reactivity half has to be built on top of any
choice, so the queue-machinery reuse value is the honest comparison
point, not the whole.
Open; inputs: OQ-ST-01's scope vote on queues (inventory: documented
need) + OQ-ST-03 resolution. **POC #1 input (2026-10-04):** on the
SQLite side the adopt question dissolved — honker-core is consumed as a
published-library dependency (the inventory-confirmed feature rows ride
its machinery; OQ-ST-06 holds the fork-vs-reference question). The
Postgres side keeps this OQ's full option space (pgboss-rs vs fork vs
re-derive on tokio-postgres) — with the POC-confirmed constraint that
whichever choice is made, reactivity is built by this crate (the
verified pgboss-rs LISTEN/NOTIFY gap stands).
### OQ-ST-06: Honker relationship — reference, fork, or vendor?
Design-reference only (read, don't copy), targeted fork of
honker-core's engine machinery, or vendor the extension? Honker is
alpha-quality per its own README, MIT/Apache-2.0 dual-licensed, and
covers only the SQLite side — but it embodies exactly the watcher/
transactional design this crate wants on SQLite, and honker-rs
demonstrates the interface shape is sound. Three refinements from the
2026-10-03 reading:
- honker-rs is **sync-only** (std threads, blocking iterators) — the
tokio port is required work under any fork posture, which changes
the fork-vs-reference calculus (a fork is already a serious port).
- The crate likely needs only the core engine machinery (honker-core
minus the extension C surface — see OQ-ST-07), a smaller extraction
than the whole project.
- Honker's own documented per-binding inconsistencies (the
processing-guarantees table, OQ-ST-04) suggest extracting *design+
semantics* with our contract pinned, rather than preserving its
behavior verbatim — closer to the alkblobs "borrow conclusions, not
wire surface" principle than to alksocks' verbatim extraction.
Open; needs the license/provenance check (AGENTS.md §3) and a quality
assessment of honker-core's watcher/transactional core. **POC #1 input
(2026-10-04):** under the resolved SQLite posture (posture 1), honker-core
is consumed as a *published library* (Writer/Readers/SharedUpdateWatcher/
attach_* — nearly all its surface minus the experimental backends), not
vendored or forked to ship; 0.5.0 is published with clean deps and the
reference-usage posture works as-is. The remaining fork trigger would be
the Phase-1 quality read (the watcher/transactional core assessment) or a
needed change upstream won't take — the calculus is unchanged in kind,
but the *default* posture is now evidenced: depend on the published crate.
### OQ-ST-07: SQLite-side scope — loadable extension, embedded rusqlite, or both?
Honker ships as a loadable extension usable by *any* SQLite client,
plus per-language bindings. This crate (a Rust library) may not need
the loadable-extension surface at all — embedding the engine machinery
in-process may be the whole story (the honker-core shape minus the
extension/binding packaging). Determines how much of honker is even
candidate material.
**Sharpened by the inventory (2026-10-04):** every identified consumer
is in-process Rust attaching to its own connection (the
`honker-core`/`attach_honker_functions` shape — the alknet-filesystem
POC's actual usage). No consumer needs the loadable-extension surface.
The question is now cut-only: loadable extension is out unless a
consumer appears; the open residue is just how much of honker-core's
machinery survives the extraction.
### OQ-ST-08: Multi-host / deployment posture
Honker is explicitly single-machine (file-backed SQLite). Postgres is
natively multi-host. The unified surface must not pretend SQLite is
multi-host, but where does the honest boundary live — per-engine
capability flags? A documented deployment matrix? Does the trait need
to expose engine capabilities at all?
Open; partially rides OQ-ST-04 (the trait's shape constrains where
capability differences can surface).
## POC register
Proposals run as standalone crates in the global workspace; findings
land in `docs/research/` here. Named per the OQ each feeds:
| # | POC | Spec | Findings |
|---|---|---|---|
| 1 | SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx (async seam, watcher, transactional contract, packaging, interop) | [poc-sqlite-posture-spec.md](poc-sqlite-posture-spec.md) | [poc-sqlite-posture-findings.md](poc-sqlite-posture-findings.md) — **passed** (verdict: Arm A; ran 2026-10-04) |
## Phase 0 plan
Iteration expected; this register grows as research rounds land.
Expected sequence (deliberately rough):
1. ~~Consumer-driven scope inventory (OQ-ST-01)~~ — **done
(2026-10-04)**: `consumer-inventory.md`, run against the paused
consumers' documents (alkfs phase-0, alkgit architecture, alkblobs
architecture + the alknet-filesystem POC). OQ-ST-01 answered down
to named per-feature rows; OQ-ST-02/07 sharpened by it.
1. ~~Crate scope (OQ-ST-02)~~ — **resolved (2026-10-04, operator
decision)**: reactive-core + engine crates. Reasoning and the
superseded inventory lean recorded at the OQ and in the inventory.
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape):
- ~~SQLite driver posture~~ — **POC #1 passed (2026-10-04)**:
posture 1 (honker-core on our rusqlite); findings + constraint
notes in the register row and OQ-ST-03.
- Postgres side of OQ-ST-03 (tokio-postgres LISTEN/pool/tx-seam
validation) — the next research round, likely POC #2.
- OQ-ST-04's contract pinning — the honker-rs surface (§Interface
finding) is the concrete starting artifact: pinning its contract
costs less and is more honest than inventing a parallel shape —
now scoped against the inventory's confirmed features rather than
the full honker menu, shaped as the core-crate trait surface per
OQ-ST-02's split, with the POC's `TxHandle`/writer-slot-lease
shape as the SQLite-seam starting point.
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem,
after the driver and shape questions narrow the option space
(OQ-ST-06's default is now evidenced as published-library use;
OQ-ST-05 waits on the pg side of OQ-ST-03).
4. Converge; Phase 1 opens with the ADR backlog this register becomes.
## References
- honker — `/workspace/honker` (git checkout @ f4e53c6 of
github.com/russellromney/honker; README +
`honker-core/src/` read 2026-10-03): the SQLite-side
feature/wake template. The four
guides (queues/streams/pubsub/scheduler on honker.dev) +
`packages/honker-rs/src/lib.rs` (v0.5.0) are the interface prior art
(§Interface finding).
- pgboss-rs — `/workspace/pgboss-rs` (git checkout @ 98f7d9e of
github.com/rustworthy/pgboss-rs, v0.1.0-rc6; read 2026-10-03,
LISTEN/NOTIFY-absence verified): the Postgres queue-family reference.
- honker's own prior-art section: pg_notify, pg-boss, Oban, Huey — the
external lineage this crate inherits from both sides.
- alkblobs — `/workspace/@alkdev/alkblobs` (spec+POCs, paused): the
paused planning this crate unblocks; its POC findings
(`poc-postgres-kv-findings.md`, `poc-pglo-findings.md`) are the
tokio-postgres evidence base.
- alkgit — `/workspace/@alkdev/alkgit` (paused mid-Phase-1,
architecture reviewed): its backend.md trait seam and ADR set are
consumer evidence for the inventory (queues/locks rows).
- alkfs — `/workspace/@alkdev/alkfs` (Phase 0 drafted, 2026-09-23): its
phase-0 OQs (OQ-FS-05/07/14/16/17) are consumer evidence for the
inventory (notify/locks/queues rows).
- alknet-filesystem POC —
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/
poc-summary.md`: the ran-once evidence that the honker-coordination
layer works (notify-on-commit test; named-locks and outbox usage
identified).
- consumer-inventory.md (`docs/research/consumer-inventory.md`) — the
per-feature synthesis (2026-10-04) answering OQ-ST-01 from the above.
- alkcall — `/workspace/@alkdev/alkcall`: the family substrate;
referenced for the store-layer-isolation principle only.
- alkstore-sqlite-posture-poc — `/workspace/alkstore-sqlite-posture-poc`
(standalone POC crate, published-deps-only): POC #1's code —
both arms end-to-end, property tests, seam/watcher probes.
- alktty — `/workspace/@alkdev/alktty` (architecture reviewed): REQ-TTY-01
(`docs/architecture/tty-backend.md`) — the async-facing-trait +
sync-bridge posture ("backends are not required to be natively
async"), the family precedent bearing on OQ-ST-03's async
sub-question.