From 4165c94ab05da7c6ed7f350e9e998293b34043b2 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Sun, 4 Oct 2026 09:31:46 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20specify=20POC=20#1=20=E2=80=94=20SQLite?= =?UTF-8?q?=20engine=20posture=20comparison=20(poc-sqlite-posture-spec.md)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Arm A: honker-core linked on our rusqlite (bridge per REQ-TTY-01, honker's watcher). Arm B: honker extension .so over sqlx-sqlite (natively async call path, own watcher, per-pool-connection extension + bootstrap — stress-testing what the CI proof script doesn't cover: pool wiring, lost connections, full surface). Option 2 (honker-rs-as-substrate) dropped from scope with reasoning: its mutex-pinned sync transaction model is subsumed by both other postures' trade space. Five probes (async seam, watcher, transactional contract, packaging, cross-process interop), a decision gate including a legitimate hybrid verdict, and out-of-scope boundaries (postgres side, full surface, extension-as-consumer-feature regardless of outcome). Phase-0: POC register added, plan/frontmatter updated; AGENTS.md: POC-register convention codified. --- AGENTS.md | 4 +- docs/research/phase-0.md | 37 +++-- docs/research/poc-sqlite-posture-spec.md | 190 +++++++++++++++++++++++ 3 files changed, 217 insertions(+), 14 deletions(-) create mode 100644 docs/research/poc-sqlite-posture-spec.md diff --git a/AGENTS.md b/AGENTS.md index c245792..68058d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,9 @@ not change `git config`, skip hooks, or use `git commit -i`. 2. **Open questions get OQ-ST-NN IDs** and live in the register in `docs/research/phase-0.md` until Phase 1 promotes them to `docs/architecture/open-questions.md`. Numbering is stable — never - renumber; append. + renumber; append. POCs live in the register there too (numbered, + spec'd under `docs/research/poc--spec.md`, findings in + `poc--findings.md`). 3. **Cite reference checkouts by path and revision.** The external references for this crate (`/workspace/honker` @ f4e53c6, `/workspace/pgboss-rs` @ 98f7d9e) are reference checkouts of diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md index fe7ed61..5dda959 100644 --- a/docs/research/phase-0.md +++ b/docs/research/phase-0.md @@ -8,23 +8,25 @@ REQ-TTY-01's async-facing-trait + sync-bridge posture recorded as family precedent bearing on OQ-ST-03; OQ-ST-03's SQLite option space expanded to three named postures (honker-core on our rusqlite / honker-rs as substrate / honker extension over sqlx — the last CI-proven in the -honker checkout's own ORM proof suite). OQ-ST-03/04/05 remain open -research. Interface finding and driver tension from 2026-10-03 remain -trusted-but-unverified working input.) +honker checkout's own ORM proof suite) and POC #1 (the A-vs-B posture +comparison) specified. OQ-ST-03/04/05 remain open +research. Interface finding and driver tension from +2026-10-03 remain trusted-but-unverified working input.) --- # alkstore — Phase 0 (Exploration) This document captures Phase 0 (Exploration) for the `alkstore` crate: -vision, guiding principles, prior art, and the open-question register -(OQ-ST-01..NN). Phase 0's objective per `docs/sdd_process.md`: *capture +vision, guiding principles, prior art, the open-question register +(OQ-ST-01..NN), and the POC register. Phase 0's objective per +`docs/sdd_process.md`: *capture vision and guiding principles; research options; validate approaches; converge on a recommended approach.* The scope question (OQ-ST-01) has been answered per-feature from the consumers' documents -(`consumer-inventory.md`); the driver and reactive-shape questions -(OQ-ST-03/04) are the open research; there is no POC register yet -(see `consumer-inventory.md` for the first cut candidates that keep its -register small). +(`consumer-inventory.md`); the driver question (OQ-ST-03) has a named +SQLite option space with POC #1 specified against it; the reactive +contract (OQ-ST-04) and ownership questions (OQ-ST-05/06) are the open +work the research rounds feed. Context for why this crate starts now: **alkblobs** (`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning) @@ -589,6 +591,15 @@ to expose engine capabilities at all? Open; partially rides OQ-ST-04 (the trait's shape constrains where capability differences can surface). +## POC register + +Proposals run as standalone crates in the global workspace; findings +land in `docs/research/` here. Named per the OQ each feeds: + +| # | POC | Spec | Feeds | +|---|---|---|---| +| 1 | SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx (async seam, watcher, transactional contract, packaging, interop) | [poc-sqlite-posture-spec.md](poc-sqlite-posture-spec.md) | OQ-ST-03, OQ-ST-04, OQ-ST-06 | + ## Phase 0 plan Iteration expected; this register grows as research rounds land. @@ -603,13 +614,13 @@ Expected sequence (deliberately rough): decision)**: reactive-core + engine crates. Reasoning and the superseded inventory lean recorded at the OQ and in the inventory. 2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library - capability matrices, then a POC if the unified-trait shape needs - validation (likely, given the structural mismatch noted in OQ-ST-04). - The honker-rs surface (§Interface finding) is the concrete starting + capability matrices (**POC #1 specified**: the SQLite posture + comparison, [poc-sqlite-posture-spec.md](poc-sqlite-posture-spec.md); + the honker-rs surface (§Interface finding) is the concrete starting artifact for the OQ-ST-04 work: pinning its contract costs less and is more honest than inventing a parallel shape — now scoped against the inventory's confirmed features rather than the full honker menu, - and shaped as the core-crate trait surface per OQ-ST-02's split. + and shaped as the core-crate trait surface per OQ-ST-02's split). 3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem, after the driver and shape questions narrow the option space. 4. Converge; Phase 1 opens with the ADR backlog this register becomes. diff --git a/docs/research/poc-sqlite-posture-spec.md b/docs/research/poc-sqlite-posture-spec.md new file mode 100644 index 0000000..702d831 --- /dev/null +++ b/docs/research/poc-sqlite-posture-spec.md @@ -0,0 +1,190 @@ +--- +status: spec +title: "POC #1 — SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx" +last_updated: 2026-10-04 +--- + +# POC: SQLite engine posture — spec + +> **POC register #1** (phase-0.md). Feeds OQ-ST-03 (the driver question's +> SQLite half) and OQ-ST-04 (the reactive contract's SQLite wake side). +> Code: standalone crate in the global workspace +> (`/workspace/alkstore-sqlite-posture-poc` — self-contained, per the +> Phase 0 convention; POC code stays out of this repo). Findings land in +> `docs/research/poc-sqlite-posture-findings.md` here regardless. + +## What this POC must decide + +OQ-ST-03 named three SQLite postures (§OQ-ST-03, option list 1–3). Option +2 (honker-rs as substrate) is **dropped from POC scope**: its blocking +`Database` API and mutex-pinned transaction model would have to sit +entirely under our async core, and both other postures give us the +machinery with more ownership — the POC's comparison between 1 and 3 +subsumes the honest parts of 2 (if 3 works and 1 works, 2 is an +intermediate we can revisit; its unique value — the typed binding — is +what the core crate's trait surface supersedes anyway). The POC decides: + +**Posture A (library):** honker-core linked on *our* rusqlite connection +— `apply_default_pragmas` + `attach_notify` + `attach_honker_functions` ++ `bootstrap_honker_schema` (the alknet-filesystem POC's usage), sync +work bridged to the async core per the family precedent (alktty +REQ-TTY-01: blocking impl on dedicated threads / `spawn_blocking`, +feeding tokio channels). + +**Posture B (extension):** honker's loadable `.so` under **sqlx-sqlite** +— `SqliteConnectOptions::extension(ext)` per pool connection + +`SELECT honker_bootstrap()`, all feature calls as plain SQL through +`SqliteExecutor<'e>` (pool / connection / transaction alike). CI-proven +shape (honker's own `scripts/proof/orm/rust`, checkout @ f4e53c6, PR +#106) — this POC does not re-prove basic wiring; it stress-tests what +the proof script does not cover. + +The question is *not* "which is faster at raw SQL" (they call the same +machinery — any SQL-path delta is small); it is **which posture the +sqlite engine crate should be built on**, decided at four load-bearing +points: the async seam, the watcher, the packaging/deployment story, and +the transactional contract. + +## The two arms + +Both arms implement the *same narrow slice of the alkstore core-crate +trait sketch* (the trait surface is OQ-ST-04's contract question — the +POC sketches just enough to compare postures, not to pin the contract): + +- `queue_enqueue_tx` / `queue_claim_batch` / `queue_ack_batch` — + including the transactional-enqueue property (business write + enqueue + in one tx, rollback drops both — principle 2). +- `notify_tx(channel, payload)` / `listen(channel)` — the reactive pair. +- `stream_publish_tx` / `read_since(consumer)` (offset save/read only — + not the full stream contract). +- `lock_acquire` / `lock_release` with TTL, sufficient for alkfs's + writer-coordination row. + +### Arm A — honker-core on our rusqlite (library posture) + +- `honker-core = "0.5"` (published crate, per the AGENTS.md posture — + read the checkout for reference, build against the published version). +- Connection model: one writer connection + reader pool (rusqlite, + `bundled-sqlite` feature for the POC's hermetic build), bridged to + async via a small `spawn_blocking` wrapper (measure dedicated-thread + + mpsc as the alternative, the REQ-TTY-01 double). +- Watcher: honker-core's `SharedUpdateWatcher::spawn_with_config` + (default 1 ms polling) driving `listen()` wakes; listener re-reads + after wake (the opaque-wake + re-read contract). + +### Arm B — honker extension over sqlx (extension posture) + +- Build the extension from the published honker source tree once + (`cargo build --release -p honker-extension` → `.so`); the POC loads it + via `HONKER_EXTENSION_PATH` (the guide's wiring; SeaORM/Toasty + sections are out of scope — sqlx is the only ORM-family shape that + matters here). +- sqlx 0.8 `runtime-tokio`, per-pool-connection `extension(ext)` + + `honker_bootstrap()` on connect (the proof script's pattern, applied + to a `SqlitePool` rather than a single connection — the POC checks the + *pool* story the script didn't run). +- All honker SQL through `SqliteExecutor<'e>`; async calls natively — + **no sync bridge on the call path** (this is the posture's load- + bearing advantage; the POC verifies it survives the full surface, not + just the proof script's enqueue path). +- Watcher: **ours** — a `spawn_blocking`/std-thread loop polling + `PRAGMA data_version` on a dedicated sqlx connection (the + honker-core-watcher logic re-implemented thin over sqlx; no + honker-core dependency in this arm — that is the point of the + posture). + +## Instruments + +1. **Async-seam probe** — the decision's first axis. Arm A: call-path + latency and executor-neutrality of the `spawn_blocking` bridge at + realistic claim/enqueue cadence (single-digit-millisecond queue ops + under WAL-NORMAL); thread cost per connection vs a shared bridge + pool; `Send`/`Sync` trait-shape friction actually encountered when + handing rusqlite types across the seam. Arm B: baseline async call + cost through sqlx (prepared statements, pool checkout) — the honest + floor any posture pays. Deliverable: the seam-cost table. +2. **Watcher probe** — both arms, the reactive pair end-to-end: + commit→wake→receiver-notified latency distribution (p50/p99) at the + default 1 ms cadence; wake correctness under a pool of writers + (`data_version` observed from a *different* connection than the + writer's — Arm B's dedicated-watcher-connection shape vs Arm A's + honker-core watcher); missed-wake stress (N rapid commits, assert + wake count ≥ 1 and state re-read correct — the overtriggering + contract, not per-commit wakes); behavior when the watcher + connection is lost (Arm B) or the watcher thread panics (both). + Deliverable: the wake-latency table + failure-mode notes. +3. **Transactional/contract probe** — both arms, the property tests: + business-write + enqueue + notify inside one caller-owned + transaction; forced rollback drops both (assert no ghost rows in + `_honker_jobs`/notification tables); concurrent writers (two async + tasks, each in a transaction, same queue) — Arm A's rusqlite + single-writer contention behavior vs Arm B's sqlx pool; `SQLITE_BUSY` + behavior surfaced through each driver (the async error path honker's + busy_timeout story must survive). Deliverable: property-test matrix + per arm. +4. **Packaging/deployment probe** — the honest costs, not benchmarks: + Arm A's build graph (honker-core pulls rusqlite + chrono + feature + surface; `bundled-sqlite` compile time; version-pinning coupling + between our rusqlite and honker-core's); Arm B's .so lifecycle + (build-or-download per release, load path resolution, sqlx's + load-then-disable extension discipline verified on the pool, macOS + and system-sqlite caveats recorded but not blockers). Deliverable: + the packaging-notes table. +5. **Interop check (both arms)** — two processes against one db file: + writer in one process, listener in another (the cross-process wake + being the whole point of the substrate). Each arm with its own + watcher/extension setup on both sides. + +## Decision gate + +Arm A is chosen iff any of: + +- the async seam costs are materially worse under B than A (B's native + async is the posture's whole premise); or +- the extension/pool wiring (per-connection extension + bootstrap on a + real pool, lost-connection recovery) fails or proves fragile; or +- the .so packaging burden is judged unacceptable for the engine crate's + zero-ops posture (REQ-1's shape: no daemon, but also no + binary-artifact management at runtime). + +Arm B is chosen iff: + +- the seam costs are materially worse under A (bridge overhead at + claim/ack cadence, thread-per-connection, trait friction), AND B's + wiring + watcher story holds under the probes above; or +- sqlx's pool/prepared-statement machinery is measurably the better fit + for the engine's concurrency posture, and the .so cost is accepted. + +The gate may also land **"A for the engine core, B's watcher pattern for +wake plumbing"** or similar hybrid — the probes are designed to +separate the *call-path* question from the *watcher* question, and a +hybrid verdict is a legitimate finding if the evidence slices that way. +Either way the findings feed: OQ-ST-03's resolution (the SQLite driver +posture, the tokio-postgres pg side rides separately), OQ-ST-04's SQLite +wake contract, OQ-ST-06's fork/reference calculus (how much honker-core +machinery each posture actually consumes), and the core-crate trait +sketch's transactional seam (what a transaction handle looks like in +each posture). + +## Out of scope + +- The Postgres engine (tokio-postgres LISTEN/NOTIFY wiring — after the + OQ-ST-04 contract shape is pinned; OQ-ST-03 remains coupled to it). +- The full honker surface (scheduler, rate limits, result storage — the + scheduler may collapse into queues per the inventory; the POC covers + enough to validate wake + queue + offset machinery). +- The loadable-extension surface as a *consumer-visible* feature + (OQ-ST-07's cut stands regardless of this POC's outcome — option B + uses the extension *mechanism* internally but the crate stays an + in-process Rust library; the .so is never a user-facing artifact). +- Performance optimization beyond what the probes need to produce + comparable numbers — both arms call the same SQL machinery, so this + POC is not a bench-fest. + +## Register note + +This POC is **#1** in alkstore's register (phase-0.md), specified +2026-10-04. It is deliberately the first POC: it de-risks the SQLite +side of OQ-ST-03 (the driver question the whole crate's shape leans on) +with a direct A-vs-B comparison, and its trait-sketch exercise doubles +as the first concrete pass at OQ-ST-04's contract pinning. \ No newline at end of file