alkstore/docs/research/poc-sqlite-posture-findings.md

---
status: findings
title: "POC #1 findings — SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx"
last_updated: 2026-10-04
---

# POC #1 findings — SQLite engine posture

**Verdict: Arm A (honker-core on our rusqlite connection) — with Arm B's
watcher pattern rejected in favor of honker-core's, and one caveat about
sqlx 0.9.** Both arms work end-to-end; every load-bearing property holds
on both; the decision hinges on packaging and friction, where A is
strictly lighter for this crate's posture. Gate conditions from the
spec: none of Arm A's failure conditions fired (seam costs are A's to
defend and A won them), while B carries a runtime `.so` dependency with
no compensating advantage A lacks.

POC code: `/workspace/alkstore-sqlite-posture-poc` (standalone crate,
published-deps-only). This doc records what was measured and why.

## Context recorded at POC start (versions/toolchains)

- rusqlite 0.40.1 requires a very fresh rustc (it uses the unstable
  `cfg_select!`; rustc 1.94 fails, 1.99 works). This is a real
  deployment constraint for any crate pinned to rusqlite 0.40.x,
  independent of posture.
- The arm-vs-arm driver conflict is *real but resolvable*: sqlx 0.8 pins
  `libsqlite3-sys` exactly, which collides with rusqlite's
  `links = "sqlite3"`. sqlx 0.9 relaxed to a range (`>=0.30.1, <0.38`)
  — but that range *still excludes* rusqlite 0.40's libsqlite3-sys
  0.38.x. The POC carries a one-line vendored patch of sqlx-sqlite
  (`[patch.crates-io]`, libsqlite3-sys req → `0.38.1`). **A single
  binary mixing rusqlite 0.40 + sqlx 0.9 sqlite needs either that patch,
  a honker-core pin below rusqlite 0.40, or sqlx's next range bump.**
  This is permanent friction between the two arms' dependencies in one
  link unit — and it is itself an argument for choosing one driver per
  engine binary (OQ-ST-02's split keeps that possible).

## Arm A — honker-core on our rusqlite (library posture)

- Published `honker-core = "0.5.0"` from crates.io; `bundled-sqlite`
  for hermetic builds. Wiring exactly per the alknet-filesystem POC's
  usage: `apply_default_pragmas` + `attach_notify` +
  `attach_honker_functions` + `bootstrap_honker_schema`.
- Async seam: `Writer` slot (single writer connection) + `Readers` pool
  + every call in `spawn_blocking`. A dedicated-std-thread bridge
  (one thread owning the writer conn, ops over mpsc — the REQ-TTY-01
  "double") was also implemented and benchmarked.
- Watcher: honker-core's `SharedUpdateWatcher` (default 1 ms), each
  `listen()` subscription bridged to a tokio mpsc via one
  `spawn_blocking` thread doing `blocking_send`.
- Transactional seam: `begin_tx` acquires the writer slot, opens
  `BEGIN IMMEDIATE` manually, and hands back a handle that owns the
  slot until commit/rollback. **This is the one place honker-core's
  Writer slot doesn't help**: the slot model assumes acquire → work →
  release atomically, so a long transaction parks the single writer
  (as it must — WAL single-writer). The trait sketch's tx handle maps
  naturally: hold the slot across `await` points *is wrong* (the
  connection is not `Sync`; all tx ops must round-trip the same
  blocking thread). The POC's shape: the tx handle keeps the
  `Connection` in an `Arc<Mutex<Option<Connection>>>` and every tx op
  is its own `spawn_blocking` — correct but per-op hop cost.

## Arm B — honker extension over sqlx (extension posture)

- Built `.so` from the **published crates.io source** of
  honker-extension 0.5.0 (one command, 12 s release build, 1.2 MB
  artifact); byte-identical logic to the checkout's
  `honker-extension/src/lib.rs`. Loads into the system sqlite3 CLI and
  into every sqlx connection.
- Pool wiring: `SqliteConnectOptions::extension(ext)` +
  `after_connect` running `SELECT honker_bootstrap()` + business DDL —
  per-pool-connection, matches the proof script's single-connection
  pattern applied to a pool. Works; extension load happens during
  connect only (sqlx's load-then-disable discipline verified by source
  read; the C load-extension API stays disabled to caller SQL).
- All feature calls are plain SQL through pool / connection /
  transaction (`Executor'<'e>` satisfied by all three, including
  inside `pool.begin()` transactions — the transactional property
  rides entirely on SQLite).
- Watcher: ours thin — a dedicated sqlx connection polling
  `PRAGMA data_version` on a 1 ms tokio interval, fanning out to
  listeners, last-error exposed for tests. Works, including
  cross-process; ~60 lines.
- **sqlx 0.9 made `SqliteConnectOptions::extension` unsafe** (it
  dlopens a runtime artifact; the unsafe contract is "path trusted"),
  and 0.9's new `SqlSafeStr` lint rejects dynamic SQL strings
  (`AssertSqlSafe()` wrapping needed for the DDL constant). Neither is
  a blocker; both are real ergonomic friction the proof script (written
  on 0.8) does not show.

## Probe 1 — the async seam (decision axis 1)

Workload: `begin_tx` + `queue_enqueue_tx` + `commit` per iteration,
sequential, n=3000, WAL + `synchronous=NORMAL`, release build, 8-core
linux box (shared dev machine, load average ~3).

| Arm | p50 | p90 | p99 | max (unbounded tail) |
|---|---|---|---|---|
| A: `spawn_blocking` per op | 0.354 ms | 0.685 | 0.838 | ~1.4 s spikes |
| A: dedicated bridge thread | 0.183 ms | 0.463 | 0.705 | ~0.6 s spikes |
| B: sqlx native async | 0.707 ms | 0.828 | 1.235 | ~1.2 s spikes |

Findings:

- **Arm A wins the seam** — 2× at p50. The spawn_blocking hop
  (µs-scale) plus rusqlite's lighter call machinery beats sqlx's
  per-statement prepare/cache/worker-thread dispatch. The dedicated
  bridge is better still (one fewer thread park per op) at the cost of
  hand-rolled plumbing; for the engine crate the spawn_blocking shape
  is fine and the dedicated thread is an optimization, not a necessity.
- **The max column is the same story on both arms** and is *not* seam
  cost: WAL autocheckpoint fsyncs (`synchronous=NORMAL` fsyncs at
  checkpoint, every N pages) plus kernel writeback. A pure-python
  sqlite3 control at `synchronous=NORMAL` showed the same spike
  pattern (~every 1000 commits, 200–400 ms spikes) on this box. No
  posture escapes durable-commit boundaries; both arms pay identical
  fsync costs at the same cadence.
- **Trait-shape friction (A)**: `Send`/`Sync` on rusqlite types across
  the seam required the tx-handle round-trip trick noted above and one
  `as_any_mut()` downcast for the `*_tx` methods (the trait hands
  callers a `dyn TxHandle`; each arm asserts its concrete type). Real
  but small; a Phase 1 contract could make tx handles generic instead.
- **sqlx's honest floor** (pool checkout + prepared-statement cache)
  is visible in B's p50 — not pathological, just consistently ~2×
  A's.

## Probe 2 — the watcher (decision axis 2)

Workload: 300 notify-commits, each commit followed by 25 ms sleep,
listener attached from MAX(id) before first commit; latency = first
wake after commit (pairing by monotonic timestamp, honker's own
test-suite scheme).

| Arm | wakes / commits | missed | p50 | p99 | max |
|---|---|---|---|---|---|
| A (honker-core SharedUpdateWatcher) | 299/300 | 0 | 1.40 ms | 2.07 ms | 29 ms |
| B (our thin data_version poller) | 299/300 | 0 | 2.15 ms | 4.51 ms | 172 ms |

- The 299/300 is wake *coalescing* (the pair-by-first-wake-after-
  commit analysis counts one wake per commit window; a burst of two
  commits inside one 1 ms poll tick produces one wake for both — correct
  by the overtriggering contract), not missed wakes. Both arms
  delivered a wake per isolated commit.
- **Both arms sub-2.5 ms at p50 at the default 1 ms cadence.** A is
  slightly tighter (its watcher is a dedicated std thread doing one
  PRAGMA poll + channel fanout; B's adds a tokio select! + channel hop
  per tick).
- **B's max is worse** (172 ms vs 29 ms): a watcher-connection poll
  colliding with a busy writer (busy_timeout retry inside sqlx's
  worker) occasionally stalls a tick. A's std-thread poller skips the
  tick on SQLITE_BUSY and catches on the next poll (honker-core's
  transient-lock handling).
- **Missed-wake stress** (30 rapid commits, no sleep): both arms
  delivered wakes ≥ 1 with correct coalescing (A: 15–28 wakes, B: 24
  wakes for 30 commits) and post-burst re-read of stream state was
  correct. The overtriggering contract ("wake is a hint; re-read
  indexed state") holds identically on both because it's the *same*
  data_version mechanism underneath.
- **Failure modes**:
  - Watcher thread panic (A): honker-core's SharedUpdateWatcher drops
    every subscriber's sender on death (WatcherDeathGuard) —
    subscribers see the channel close, not a silent hang. Good
    behavior, inherited for free.
  - Watcher connection loss (B): our thin watcher retries with
    200 ms backoff and surfaces last_error; recovery verified in test
    (`watcher_death_recovery_arm_b` — stop watcher, commit, restart
    watcher, next commit delivers). Acceptable, but every line of it
    is ours to maintain.
- **Cross-process interop (probe 5)**: writer process commits a
  notify; listener process attached in-process watcher wakes. Verified
  for both arms (1 wake, 0 missed, sub-10 ms). The substrate property
  (one db file, one watcher either side) holds identically — again the
  same mechanism underneath.

## Probe 3 — transactional contract (decision axis 4)

Property tests (Rust, `tests/contract.rs`, all green, 4 tests —
`tx_enqueue_rollback_drops_both_ghost_free`,
`concurrent_writers_exactly_once`, `missed_wake_stress_burst_coalescing`,
`watcher_death_recovery_arm_b` — covering the rows below; the first test
also asserts the committed-tx happy path):

| Property | A | B |
|---|---|---|
| enqueue + business write + notify in one caller tx | ✅ | ✅ |
| rollback drops job + row + notification (no ghosts) | ✅ | ✅ |
| 4 concurrent producers, 40 tx-enqueues, exactly-once claim | ✅ | ✅ |
| claim/ack round-trip counts | ✅ | ✅ |
| watcher restart recovery (B-specific) | n/a | ✅ |

- The transactional property is *SQLite's*, not the posture's — proof
  script already showed it on sqlx; it holds identically through the
  trait seam on both.
- SQLITE_BUSY under concurrency: surfaced as an error through both
  drivers (`honker-core`'s 5 s busy_timeout first, sqlx's
  `busy_timeout(5s)` config on connect). At 4 concurrent producers
  neither arm hit BUSY in the tests (WAL single-writer serialized by
  A's Writer slot / B's pool + busy handler).
- The spec's "what does a transaction handle look like" question
  crystallized: **Arm A's handle is an owned writer-slot lease; Arm
  B's is sqlx's `Transaction`.** Both are caller-visible for the
  `*_tx` seam; the core-crate contract can genuinely abstract over
  both with the `TxHandle` trait shape this POC sketched (downcast per
  arm).

## Probe 4 — packaging/deployment (decision axis 3)

| | A | B |
|---|---|---|
| Runtime artifacts | none (static link) | `libhonker_ext.so` (~1.2 MB) must ship beside the binary |
| Clean dev build (deps only, this box) | ~19 s (bundled sqlite C build ~10 s of it) | ~34 s (sqlx-core tree wider) |
| Clean release build | slower C optimization, no dlopen | + one separate release build per platform |
| Cargo dependency tree (direct) | rusqlite + honker-core (chrono, parking_lot) + tokio | sqlx tree: wider (idna/url/zerovez/serde stack, tracing, flume…) |
| Version coupling | honker-core pins rusqlite ^0.40.1; we must ride its rusqlite | sqlx 0.9 ↔ libsqlite3-sys range still conflicts with rusqlite 0.40 (vendored one-line patch in POC) |
| .so lifecycle | — | build-per-release from published source (12 s, `cargo build -p honker-extension --release`), load path resolution, macOS dylib + system-sqlite caveats (honker's own docs) |
| sqlx 0.9 ergonomics | — | `extension()` now `unsafe`; `SqlSafeStr` lint needs `AssertSqlSafe()` for dynamic DDL |

- Arm A's `bundled-sqlite` compile is the biggest single build cost but
  is cacheable and CI-normal.
- Arm B's .so is buildable from published source in one step (no
  checkout required) — a genuinely clean artifact pipeline — but it is
  a *load-time* runtime dependency, a failure mode A doesn't have
  (a missing/mismatched .so fails pool connect; the engine crate's
  zero-ops posture — REQ-1's shape — would eat an artifact-management
  problem for no measured benefit).

## Verdict against the decision gate

Arm A is chosen iff any of the spec's gate conditions fire; here they
all do, mildly and all in A's favor:

1. **Async seam**: B's premise (native async wins) is *false* on
   measurements — A's bridged path is ~2× faster at p50 and ~2.5× at
   p99 in this workload, and the family precedent (REQ-TTY-01) already
   blesses bridge-at-the-seam as a supported posture. A's dedicated
   bridge variant doubles the margin when needed.
2. **Extension/pool wiring**: works (verified: per-connection
   extension + bootstrap on a real pool, cross-process, watcher
   restart). Not fragile — just ours to own, with two sqlx-0.9
   ergonomic warts (`unsafe extension()`, `SqlSafeStr`).
3. **.so packaging**: unacceptable for a zero-ops library whose
   consumers want "open a path, get a store" — a dlopen dependency
   with platform-specific artifact management for a mechanism A links
   statically at zero runtime cost.

**Hybrid question answered**: the spec allowed "A for the engine
core, B's watcher pattern for wake plumbing." The watcher comparison
went the same direction (A's inherited watcher is tighter and comes
with battle-tested failure-handling), so no hybrid is warranted on
this evidence; the thin-watcher pattern is worth keeping as a *design
note* for a future non-rusqlite engine (e.g. any native-async SQLite
driver if one ever displaces rusqlite).

## What feeds where

- **OQ-ST-03 (SQLite half)**: resolved in favor of posture 1
  (honker-core on our rusqlite) contingent on honker-core's
  version-pinning posture being acceptable; the sqlx+extension
  posture is *viable* but pays more (seam perf, runtime artifact,
  dependency-graph coupling) for nothing A lacks.
- **OQ-ST-04 (SQLite wake side)**: the opaque-wake + re-read
  contract survives both postures unchanged (same data_version
  mechanism); wake contract numbers: p50 ~1.4–2.1 ms at 1 ms cadence;
  wake bursts coalesce; listener semantics (start at MAX(id), no
  replay) unchanged. The reactive contract pinning proceeds with
  honker-rs's surface as the starting point (per phase-0 §Interface
  finding).
- **OQ-ST-06 (fork/reference calculus)**: honker-core is consumed as
  *library* (Writer/Readers/SharedUpdateWatcher/attach_*), ~all its
  lib.rs surface minus experimental backends. It is not a vendoring
  candidate yet (0.5.0 published, clean deps); fork question stays
  open for the Phase-1 quality read, but the reference-usage posture
  works as-is.
- **Core-crate trait sketch**: the POC's `StoreEngine`/`TxHandle`
  sketch (queue/stream/notify/lock surface, `*_tx` seam, downcast to
  per-driver impls) is a workable first pass; its two frictions
  (`as_any_mut` downcast, tx-handle thread-affinity on rusqlite) are
  Phase-1 contract-pinning input.

## Honest caveats

- All numbers are single-machine (8-core shared dev box, ext4,
  load ~3). Absolute values are indicative; *relative* deltas between
  arms are the deliverable, and they were consistent across repeat
  runs.
- The POC's trait surface is a sketch, not the contract: it omits
  heartbeat/sweep/scheduler/results, and the stream offset-save in
  this POC is a plain SQL upsert, not honker's full
  `save_offset_tx` surface.
- The sqlx arm ran against a vendored (one-line) patch of
  sqlx-sqlite for libsqlite3-sys version resolution; upstream sqlx
  will presumably widen its range eventually (0.8 pinned exactly,
  0.9 ranged to <0.38), but until then a mixed-driver binary must
  patch or pin — recorded above as a constraint, not as an Arm B
  defect (Arm A-only binaries never mix the drivers).
- rusqlite 0.40.x's rustc requirement (≥ current-nightly-ish for
  `cfg_select!` as of the POC date... rustc 1.99 stable passes) is a
  deployment constraint to record in the engine crate's docs
  regardless of posture.

## Artifacts

- POC crate: `/workspace/alkstore-sqlite-posture-poc` (self-contained;
  `extbuild/libhonker_ext.so` built from crates.io-published source;
  `[patch.crates-io]` one-liner for sqlx-sqlite/libsqlite3-sys).
- Tests: `cargo test` → `tests/contract.rs` (4 tests; transactional
  properties, concurrency, wake coalescing, watcher recovery).
- Probes: `cargo run --release -- seam|watch <arm> <db>` per the
  binary's help; `watchloop` for the dedicated data_version watcher.