alkstore/docs/research/poc-sqlite-posture-findings.md

status, title, last_updated

status	title	last_updated
findings	POC #1 findings — SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx	2026-10-04

POC #1 findings — SQLite engine posture

Verdict: Arm A (honker-core on our rusqlite connection) — with Arm B's watcher pattern rejected in favor of honker-core's, and one caveat about sqlx 0.9. Both arms work end-to-end; every load-bearing property holds on both; the decision hinges on packaging and friction, where A is strictly lighter for this crate's posture. Gate conditions from the spec: none of Arm A's failure conditions fired (seam costs are A's to defend and A won them), while B carries a runtime .so dependency with no compensating advantage A lacks.

POC code: /workspace/alkstore-sqlite-posture-poc (standalone crate, published-deps-only). This doc records what was measured and why.

Context recorded at POC start (versions/toolchains)

• rusqlite 0.40.1 requires a very fresh rustc (it uses the unstable cfg_select!; rustc 1.94 fails, 1.99 works). This is a real deployment constraint for any crate pinned to rusqlite 0.40.x, independent of posture.
• The arm-vs-arm driver conflict is real but resolvable: sqlx 0.8 pins libsqlite3-sys exactly, which collides with rusqlite's links = "sqlite3". sqlx 0.9 relaxed to a range (>=0.30.1, <0.38) — but that range still excludes rusqlite 0.40's libsqlite3-sys 0.38.x. The POC carries a one-line vendored patch of sqlx-sqlite ([patch.crates-io], libsqlite3-sys req → 0.38.1). A single binary mixing rusqlite 0.40 + sqlx 0.9 sqlite needs either that patch, a honker-core pin below rusqlite 0.40, or sqlx's next range bump. This is permanent friction between the two arms' dependencies in one link unit — and it is itself an argument for choosing one driver per engine binary (OQ-ST-02's split keeps that possible).

Arm A — honker-core on our rusqlite (library posture)

• Published honker-core = "0.5.0" from crates.io; bundled-sqlite for hermetic builds. Wiring exactly per the alknet-filesystem POC's usage: apply_default_pragmas + attach_notify + attach_honker_functions + bootstrap_honker_schema.
• Async seam: Writer slot (single writer connection) + Readers pool
  • every call in spawn_blocking. A dedicated-std-thread bridge (one thread owning the writer conn, ops over mpsc — the REQ-TTY-01 "double") was also implemented and benchmarked.
• Watcher: honker-core's SharedUpdateWatcher (default 1 ms), each listen() subscription bridged to a tokio mpsc via one spawn_blocking thread doing blocking_send.
• Transactional seam: begin_tx acquires the writer slot, opens BEGIN IMMEDIATE manually, and hands back a handle that owns the slot until commit/rollback. This is the one place honker-core's Writer slot doesn't help: the slot model assumes acquire → work → release atomically, so a long transaction parks the single writer (as it must — WAL single-writer). The trait sketch's tx handle maps naturally: hold the slot across await points is wrong (the connection is not Sync; all tx ops must round-trip the same blocking thread). The POC's shape: the tx handle keeps the Connection in an Arc<Mutex<Option<Connection>>> and every tx op is its own spawn_blocking — correct but per-op hop cost.

Arm B — honker extension over sqlx (extension posture)

• Built .so from the published crates.io source of honker-extension 0.5.0 (one command, 12 s release build, 1.2 MB artifact); byte-identical logic to the checkout's honker-extension/src/lib.rs. Loads into the system sqlite3 CLI and into every sqlx connection.
• Pool wiring: SqliteConnectOptions::extension(ext) + after_connect running SELECT honker_bootstrap() + business DDL — per-pool-connection, matches the proof script's single-connection pattern applied to a pool. Works; extension load happens during connect only (sqlx's load-then-disable discipline verified by source read; the C load-extension API stays disabled to caller SQL).
• All feature calls are plain SQL through pool / connection / transaction (Executor'<'e> satisfied by all three, including inside pool.begin() transactions — the transactional property rides entirely on SQLite).
• Watcher: ours thin — a dedicated sqlx connection polling PRAGMA data_version on a 1 ms tokio interval, fanning out to listeners, last-error exposed for tests. Works, including cross-process; ~60 lines.
• sqlx 0.9 made SqliteConnectOptions::extension unsafe (it dlopens a runtime artifact; the unsafe contract is "path trusted"), and 0.9's new SqlSafeStr lint rejects dynamic SQL strings (AssertSqlSafe() wrapping needed for the DDL constant). Neither is a blocker; both are real ergonomic friction the proof script (written on 0.8) does not show.

Probe 1 — the async seam (decision axis 1)

Workload: begin_tx + queue_enqueue_tx + commit per iteration, sequential, n=3000, WAL + synchronous=NORMAL, release build, 8-core linux box (shared dev machine, load average ~3).

Arm	p50	p90	p99	max (unbounded tail)
A: spawn_blocking per op	0.354 ms	0.685	0.838	~1.4 s spikes
A: dedicated bridge thread	0.183 ms	0.463	0.705	~0.6 s spikes
B: sqlx native async	0.707 ms	0.828	1.235	~1.2 s spikes

Findings:

• Arm A wins the seam — 2× at p50. The spawn_blocking hop (µs-scale) plus rusqlite's lighter call machinery beats sqlx's per-statement prepare/cache/worker-thread dispatch. The dedicated bridge is better still (one fewer thread park per op) at the cost of hand-rolled plumbing; for the engine crate the spawn_blocking shape is fine and the dedicated thread is an optimization, not a necessity.
• The max column is the same story on both arms and is not seam cost: WAL autocheckpoint fsyncs (synchronous=NORMAL fsyncs at checkpoint, every N pages) plus kernel writeback. A pure-python sqlite3 control at synchronous=NORMAL showed the same spike pattern (~every 1000 commits, 200–400 ms spikes) on this box. No posture escapes durable-commit boundaries; both arms pay identical fsync costs at the same cadence.
• Trait-shape friction (A): Send/Sync on rusqlite types across the seam required the tx-handle round-trip trick noted above and one as_any_mut() downcast for the *_tx methods (the trait hands callers a dyn TxHandle; each arm asserts its concrete type). Real but small; a Phase 1 contract could make tx handles generic instead.
• sqlx's honest floor (pool checkout + prepared-statement cache) is visible in B's p50 — not pathological, just consistently ~2× A's.

Probe 2 — the watcher (decision axis 2)

Workload: 300 notify-commits, each commit followed by 25 ms sleep, listener attached from MAX(id) before first commit; latency = first wake after commit (pairing by monotonic timestamp, honker's own test-suite scheme).

Arm	wakes / commits	missed	p50	p99	max
A (honker-core SharedUpdateWatcher)	299/300	0	1.40 ms	2.07 ms	29 ms
B (our thin data_version poller)	299/300	0	2.15 ms	4.51 ms	172 ms

• The 299/300 is wake coalescing (the pair-by-first-wake-after- commit analysis counts one wake per commit window; a burst of two commits inside one 1 ms poll tick produces one wake for both — correct by the overtriggering contract), not missed wakes. Both arms delivered a wake per isolated commit.
• Both arms sub-2.5 ms at p50 at the default 1 ms cadence. A is slightly tighter (its watcher is a dedicated std thread doing one PRAGMA poll + channel fanout; B's adds a tokio select! + channel hop per tick).
• B's max is worse (172 ms vs 29 ms): a watcher-connection poll colliding with a busy writer (busy_timeout retry inside sqlx's worker) occasionally stalls a tick. A's std-thread poller skips the tick on SQLITE_BUSY and catches on the next poll (honker-core's transient-lock handling).
• Missed-wake stress (30 rapid commits, no sleep): both arms delivered wakes ≥ 1 with correct coalescing (A: 15–28 wakes, B: 24 wakes for 30 commits) and post-burst re-read of stream state was correct. The overtriggering contract ("wake is a hint; re-read indexed state") holds identically on both because it's the same data_version mechanism underneath.
• Failure modes:
  • Watcher thread panic (A): honker-core's SharedUpdateWatcher drops every subscriber's sender on death (WatcherDeathGuard) — subscribers see the channel close, not a silent hang. Good behavior, inherited for free.
  • Watcher connection loss (B): our thin watcher retries with 200 ms backoff and surfaces last_error; recovery verified in test (watcher_death_recovery_arm_b — stop watcher, commit, restart watcher, next commit delivers). Acceptable, but every line of it is ours to maintain.
• Cross-process interop (probe 5): writer process commits a notify; listener process attached in-process watcher wakes. Verified for both arms (1 wake, 0 missed, sub-10 ms). The substrate property (one db file, one watcher either side) holds identically — again the same mechanism underneath.

Probe 3 — transactional contract (decision axis 4)

Property tests (Rust, tests/contract.rs, all green, 4 tests — tx_enqueue_rollback_drops_both_ghost_free, concurrent_writers_exactly_once, missed_wake_stress_burst_coalescing, watcher_death_recovery_arm_b — covering the rows below; the first test also asserts the committed-tx happy path):

Property	A	B
enqueue + business write + notify in one caller tx	✅	✅
rollback drops job + row + notification (no ghosts)	✅	✅
4 concurrent producers, 40 tx-enqueues, exactly-once claim	✅	✅
claim/ack round-trip counts	✅	✅
watcher restart recovery (B-specific)	n/a	✅

• The transactional property is SQLite's, not the posture's — proof script already showed it on sqlx; it holds identically through the trait seam on both.
• SQLITE_BUSY under concurrency: surfaced as an error through both drivers (honker-core's 5 s busy_timeout first, sqlx's busy_timeout(5s) config on connect). At 4 concurrent producers neither arm hit BUSY in the tests (WAL single-writer serialized by A's Writer slot / B's pool + busy handler).
• The spec's "what does a transaction handle look like" question crystallized: Arm A's handle is an owned writer-slot lease; Arm B's is sqlx's Transaction. Both are caller-visible for the *_tx seam; the core-crate contract can genuinely abstract over both with the TxHandle trait shape this POC sketched (downcast per arm).

Probe 4 — packaging/deployment (decision axis 3)

	A	B
Runtime artifacts	none (static link)	libhonker_ext.so (~1.2 MB) must ship beside the binary
Clean dev build (deps only, this box)	~19 s (bundled sqlite C build ~10 s of it)	~34 s (sqlx-core tree wider)
Clean release build	slower C optimization, no dlopen	+ one separate release build per platform
Cargo dependency tree (direct)	rusqlite + honker-core (chrono, parking_lot) + tokio	sqlx tree: wider (idna/url/zerovez/serde stack, tracing, flume…)
Version coupling	honker-core pins rusqlite ^0.40.1; we must ride its rusqlite	sqlx 0.9 ↔ libsqlite3-sys range still conflicts with rusqlite 0.40 (vendored one-line patch in POC)
.so lifecycle	—	build-per-release from published source (12 s, cargo build -p honker-extension --release), load path resolution, macOS dylib + system-sqlite caveats (honker's own docs)
sqlx 0.9 ergonomics	—	extension() now unsafe; SqlSafeStr lint needs AssertSqlSafe() for dynamic DDL

• Arm A's bundled-sqlite compile is the biggest single build cost but is cacheable and CI-normal.
• Arm B's .so is buildable from published source in one step (no checkout required) — a genuinely clean artifact pipeline — but it is a load-time runtime dependency, a failure mode A doesn't have (a missing/mismatched .so fails pool connect; the engine crate's zero-ops posture — REQ-1's shape — would eat an artifact-management problem for no measured benefit).

Verdict against the decision gate

Arm A is chosen iff any of the spec's gate conditions fire; here they all do, mildly and all in A's favor:

1. Async seam: B's premise (native async wins) is false on measurements — A's bridged path is ~2× faster at p50 and ~2.5× at p99 in this workload, and the family precedent (REQ-TTY-01) already blesses bridge-at-the-seam as a supported posture. A's dedicated bridge variant doubles the margin when needed.
2. Extension/pool wiring: works (verified: per-connection extension + bootstrap on a real pool, cross-process, watcher restart). Not fragile — just ours to own, with two sqlx-0.9 ergonomic warts (unsafe extension(), SqlSafeStr).
3. .so packaging: unacceptable for a zero-ops library whose consumers want "open a path, get a store" — a dlopen dependency with platform-specific artifact management for a mechanism A links statically at zero runtime cost.

Hybrid question answered: the spec allowed "A for the engine core, B's watcher pattern for wake plumbing." The watcher comparison went the same direction (A's inherited watcher is tighter and comes with battle-tested failure-handling), so no hybrid is warranted on this evidence; the thin-watcher pattern is worth keeping as a design note for a future non-rusqlite engine (e.g. any native-async SQLite driver if one ever displaces rusqlite).

What feeds where

• OQ-ST-03 (SQLite half): resolved in favor of posture 1 (honker-core on our rusqlite) contingent on honker-core's version-pinning posture being acceptable; the sqlx+extension posture is viable but pays more (seam perf, runtime artifact, dependency-graph coupling) for nothing A lacks.
• OQ-ST-04 (SQLite wake side): the opaque-wake + re-read contract survives both postures unchanged (same data_version mechanism); wake contract numbers: p50 ~1.4–2.1 ms at 1 ms cadence; wake bursts coalesce; listener semantics (start at MAX(id), no replay) unchanged. The reactive contract pinning proceeds with honker-rs's surface as the starting point (per phase-0 §Interface finding).
• OQ-ST-06 (fork/reference calculus): honker-core is consumed as library (Writer/Readers/SharedUpdateWatcher/attach_*), ~all its lib.rs surface minus experimental backends. It is not a vendoring candidate yet (0.5.0 published, clean deps); fork question stays open for the Phase-1 quality read, but the reference-usage posture works as-is.
• Core-crate trait sketch: the POC's StoreEngine/TxHandle sketch (queue/stream/notify/lock surface, *_tx seam, downcast to per-driver impls) is a workable first pass; its two frictions (as_any_mut downcast, tx-handle thread-affinity on rusqlite) are Phase-1 contract-pinning input.

Honest caveats

• All numbers are single-machine (8-core shared dev box, ext4, load ~3). Absolute values are indicative; relative deltas between arms are the deliverable, and they were consistent across repeat runs.
• The POC's trait surface is a sketch, not the contract: it omits heartbeat/sweep/scheduler/results, and the stream offset-save in this POC is a plain SQL upsert, not honker's full save_offset_tx surface.
• The sqlx arm ran against a vendored (one-line) patch of sqlx-sqlite for libsqlite3-sys version resolution; upstream sqlx will presumably widen its range eventually (0.8 pinned exactly, 0.9 ranged to <0.38), but until then a mixed-driver binary must patch or pin — recorded above as a constraint, not as an Arm B defect (Arm A-only binaries never mix the drivers).
• rusqlite 0.40.x's rustc requirement (≥ current-nightly-ish for cfg_select! as of the POC date... rustc 1.99 stable passes) is a deployment constraint to record in the engine crate's docs regardless of posture.

Artifacts

• POC crate: /workspace/alkstore-sqlite-posture-poc (self-contained; extbuild/libhonker_ext.so built from crates.io-published source; [patch.crates-io] one-liner for sqlx-sqlite/libsqlite3-sys).
• Tests: cargo test → tests/contract.rs (4 tests; transactional properties, concurrency, wake coalescing, watcher recovery).
• Probes: cargo run --release -- seam|watch <arm> <db> per the binary's help; watchloop for the dedicated data_version watcher.