alkstore/docs/research/poc-sqlite-posture-spec.md

status, title, last_updated

status	title	last_updated
spec	POC #1 — SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx	2026-10-04

POC: SQLite engine posture — spec

POC register #1 (phase-0.md). Feeds OQ-ST-03 (the driver question's SQLite half) and OQ-ST-04 (the reactive contract's SQLite wake side). Code: standalone crate in the global workspace (/workspace/alkstore-sqlite-posture-poc — self-contained, per the Phase 0 convention; POC code stays out of this repo). Findings land in docs/research/poc-sqlite-posture-findings.md here regardless.

What this POC must decide

OQ-ST-03 named three SQLite postures (§OQ-ST-03, option list 1–3). Option 2 (honker-rs as substrate) is dropped from POC scope: its blocking Database API and mutex-pinned transaction model would have to sit entirely under our async core, and both other postures give us the machinery with more ownership — the POC's comparison between 1 and 3 subsumes the honest parts of 2 (if 3 works and 1 works, 2 is an intermediate we can revisit; its unique value — the typed binding — is what the core crate's trait surface supersedes anyway). The POC decides:

Posture A (library): honker-core linked on our rusqlite connection — apply_default_pragmas + attach_notify + attach_honker_functions

• bootstrap_honker_schema (the alknet-filesystem POC's usage), sync work bridged to the async core per the family precedent (alktty REQ-TTY-01: blocking impl on dedicated threads / spawn_blocking, feeding tokio channels).

Posture B (extension): honker's loadable .so under sqlx-sqlite — SqliteConnectOptions::extension(ext) per pool connection + SELECT honker_bootstrap(), all feature calls as plain SQL through SqliteExecutor<'e> (pool / connection / transaction alike). CI-proven shape (honker's own scripts/proof/orm/rust, checkout @ f4e53c6, PR #106) — this POC does not re-prove basic wiring; it stress-tests what the proof script does not cover.

The question is not "which is faster at raw SQL" (they call the same machinery — any SQL-path delta is small); it is which posture the sqlite engine crate should be built on, decided at four load-bearing points: the async seam, the watcher, the packaging/deployment story, and the transactional contract.

The two arms

Both arms implement the same narrow slice of the alkstore core-crate trait sketch (the trait surface is OQ-ST-04's contract question — the POC sketches just enough to compare postures, not to pin the contract):

• queue_enqueue_tx / queue_claim_batch / queue_ack_batch — including the transactional-enqueue property (business write + enqueue in one tx, rollback drops both — principle 2).
• notify_tx(channel, payload) / listen(channel) — the reactive pair.
• stream_publish_tx / read_since(consumer) (offset save/read only — not the full stream contract).
• lock_acquire / lock_release with TTL, sufficient for alkfs's writer-coordination row.

Arm A — honker-core on our rusqlite (library posture)

• honker-core = "0.5" (published crate, per the AGENTS.md posture — read the checkout for reference, build against the published version).
• Connection model: one writer connection + reader pool (rusqlite, bundled-sqlite feature for the POC's hermetic build), bridged to async via a small spawn_blocking wrapper (measure dedicated-thread + mpsc as the alternative, the REQ-TTY-01 double).
• Watcher: honker-core's SharedUpdateWatcher::spawn_with_config (default 1 ms polling) driving listen() wakes; listener re-reads after wake (the opaque-wake + re-read contract).

Arm B — honker extension over sqlx (extension posture)

• Build the extension from the published honker source tree once (cargo build --release -p honker-extension → .so); the POC loads it via HONKER_EXTENSION_PATH (the guide's wiring; SeaORM/Toasty sections are out of scope — sqlx is the only ORM-family shape that matters here).
• sqlx 0.8 runtime-tokio, per-pool-connection extension(ext) + honker_bootstrap() on connect (the proof script's pattern, applied to a SqlitePool rather than a single connection — the POC checks the pool story the script didn't run).
• All honker SQL through SqliteExecutor<'e>; async calls natively — no sync bridge on the call path (this is the posture's load- bearing advantage; the POC verifies it survives the full surface, not just the proof script's enqueue path).
• Watcher: ours — a spawn_blocking/std-thread loop polling PRAGMA data_version on a dedicated sqlx connection (the honker-core-watcher logic re-implemented thin over sqlx; no honker-core dependency in this arm — that is the point of the posture).

Instruments

1. Async-seam probe — the decision's first axis. Arm A: call-path latency and executor-neutrality of the spawn_blocking bridge at realistic claim/enqueue cadence (single-digit-millisecond queue ops under WAL-NORMAL); thread cost per connection vs a shared bridge pool; Send/Sync trait-shape friction actually encountered when handing rusqlite types across the seam. Arm B: baseline async call cost through sqlx (prepared statements, pool checkout) — the honest floor any posture pays. Deliverable: the seam-cost table.
2. Watcher probe — both arms, the reactive pair end-to-end: commit→wake→receiver-notified latency distribution (p50/p99) at the default 1 ms cadence; wake correctness under a pool of writers (data_version observed from a different connection than the writer's — Arm B's dedicated-watcher-connection shape vs Arm A's honker-core watcher); missed-wake stress (N rapid commits, assert wake count ≥ 1 and state re-read correct — the overtriggering contract, not per-commit wakes); behavior when the watcher connection is lost (Arm B) or the watcher thread panics (both). Deliverable: the wake-latency table + failure-mode notes.
3. Transactional/contract probe — both arms, the property tests: business-write + enqueue + notify inside one caller-owned transaction; forced rollback drops both (assert no ghost rows in _honker_jobs/notification tables); concurrent writers (two async tasks, each in a transaction, same queue) — Arm A's rusqlite single-writer contention behavior vs Arm B's sqlx pool; SQLITE_BUSY behavior surfaced through each driver (the async error path honker's busy_timeout story must survive). Deliverable: property-test matrix per arm.
4. Packaging/deployment probe — the honest costs, not benchmarks: Arm A's build graph (honker-core pulls rusqlite + chrono + feature surface; bundled-sqlite compile time; version-pinning coupling between our rusqlite and honker-core's); Arm B's .so lifecycle (build-or-download per release, load path resolution, sqlx's load-then-disable extension discipline verified on the pool, macOS and system-sqlite caveats recorded but not blockers). Deliverable: the packaging-notes table.
5. Interop check (both arms) — two processes against one db file: writer in one process, listener in another (the cross-process wake being the whole point of the substrate). Each arm with its own watcher/extension setup on both sides.

Decision gate

Arm A is chosen iff any of:

• the async seam costs are materially worse under B than A (B's native async is the posture's whole premise); or
• the extension/pool wiring (per-connection extension + bootstrap on a real pool, lost-connection recovery) fails or proves fragile; or
• the .so packaging burden is judged unacceptable for the engine crate's zero-ops posture (REQ-1's shape: no daemon, but also no binary-artifact management at runtime).

Arm B is chosen iff:

• the seam costs are materially worse under A (bridge overhead at claim/ack cadence, thread-per-connection, trait friction), AND B's wiring + watcher story holds under the probes above; or
• sqlx's pool/prepared-statement machinery is measurably the better fit for the engine's concurrency posture, and the .so cost is accepted.

The gate may also land "A for the engine core, B's watcher pattern for wake plumbing" or similar hybrid — the probes are designed to separate the call-path question from the watcher question, and a hybrid verdict is a legitimate finding if the evidence slices that way. Either way the findings feed: OQ-ST-03's resolution (the SQLite driver posture, the tokio-postgres pg side rides separately), OQ-ST-04's SQLite wake contract, OQ-ST-06's fork/reference calculus (how much honker-core machinery each posture actually consumes), and the core-crate trait sketch's transactional seam (what a transaction handle looks like in each posture).

Out of scope

• The Postgres engine (tokio-postgres LISTEN/NOTIFY wiring — after the OQ-ST-04 contract shape is pinned; OQ-ST-03 remains coupled to it).
• The full honker surface (scheduler, rate limits, result storage — the scheduler may collapse into queues per the inventory; the POC covers enough to validate wake + queue + offset machinery).
• The loadable-extension surface as a consumer-visible feature (OQ-ST-07's cut stands regardless of this POC's outcome — option B uses the extension mechanism internally but the crate stays an in-process Rust library; the .so is never a user-facing artifact).
• Performance optimization beyond what the probes need to produce comparable numbers — both arms call the same SQL machinery, so this POC is not a bench-fest.

Register note

This POC is #1 in alkstore's register (phase-0.md), specified 2026-10-04. It is deliberately the first POC: it de-risks the SQLite side of OQ-ST-03 (the driver question the whole crate's shape leans on) with a direct A-vs-B comparison, and its trait-sketch exercise doubles as the first concrete pass at OQ-ST-04's contract pinning.