docs: specify POC #1 — SQLite engine posture comparison (poc-sqlite-posture-spec.md)
Arm A: honker-core linked on our rusqlite (bridge per REQ-TTY-01, honker's watcher). Arm B: honker extension .so over sqlx-sqlite (natively async call path, own watcher, per-pool-connection extension + bootstrap — stress-testing what the CI proof script doesn't cover: pool wiring, lost connections, full surface). Option 2 (honker-rs-as-substrate) dropped from scope with reasoning: its mutex-pinned sync transaction model is subsumed by both other postures' trade space. Five probes (async seam, watcher, transactional contract, packaging, cross-process interop), a decision gate including a legitimate hybrid verdict, and out-of-scope boundaries (postgres side, full surface, extension-as-consumer-feature regardless of outcome). Phase-0: POC register added, plan/frontmatter updated; AGENTS.md: POC-register convention codified.
This commit is contained in:
1 parent
8331a96817
commit
4165c94ab0
3 files changed
+217
-14
No files matched your search
@@ -57,7 +57,9 @@ not change `git config`, skip hooks, or use `git commit -i`.
|
||||
2. **Open questions get OQ-ST-NN IDs** and live in the register in
|
||||
`docs/research/phase-0.md` until Phase 1 promotes them to
|
||||
`docs/architecture/open-questions.md`. Numbering is stable — never
|
||||
renumber; append.
|
||||
renumber; append. POCs live in the register there too (numbered,
|
||||
spec'd under `docs/research/poc-<name>-spec.md`, findings in
|
||||
`poc-<name>-findings.md`).
|
||||
3. **Cite reference checkouts by path and revision.** The external
|
||||
references for this crate (`/workspace/honker` @ f4e53c6,
|
||||
`/workspace/pgboss-rs` @ 98f7d9e) are reference checkouts of
|
||||
|
||||
+24
-13
@@ -8,23 +8,25 @@ REQ-TTY-01's async-facing-trait + sync-bridge posture recorded as family
|
||||
precedent bearing on OQ-ST-03; OQ-ST-03's SQLite option space expanded
|
||||
to three named postures (honker-core on our rusqlite / honker-rs as
|
||||
substrate / honker extension over sqlx — the last CI-proven in the
|
||||
honker checkout's own ORM proof suite). OQ-ST-03/04/05 remain open
|
||||
research. Interface finding and driver tension from 2026-10-03 remain
|
||||
trusted-but-unverified working input.)
|
||||
honker checkout's own ORM proof suite) and POC #1 (the A-vs-B posture
|
||||
comparison) specified. OQ-ST-03/04/05 remain open
|
||||
research. Interface finding and driver tension from
|
||||
2026-10-03 remain trusted-but-unverified working input.)
|
||||
---
|
||||
|
||||
# alkstore — Phase 0 (Exploration)
|
||||
|
||||
This document captures Phase 0 (Exploration) for the `alkstore` crate:
|
||||
vision, guiding principles, prior art, and the open-question register
|
||||
(OQ-ST-01..NN). Phase 0's objective per `docs/sdd_process.md`: *capture
|
||||
vision, guiding principles, prior art, the open-question register
|
||||
(OQ-ST-01..NN), and the POC register. Phase 0's objective per
|
||||
`docs/sdd_process.md`: *capture
|
||||
vision and guiding principles; research options; validate approaches;
|
||||
converge on a recommended approach.* The scope question (OQ-ST-01) has
|
||||
been answered per-feature from the consumers' documents
|
||||
(`consumer-inventory.md`); the driver and reactive-shape questions
|
||||
(OQ-ST-03/04) are the open research; there is no POC register yet
|
||||
(see `consumer-inventory.md` for the first cut candidates that keep its
|
||||
register small).
|
||||
(`consumer-inventory.md`); the driver question (OQ-ST-03) has a named
|
||||
SQLite option space with POC #1 specified against it; the reactive
|
||||
contract (OQ-ST-04) and ownership questions (OQ-ST-05/06) are the open
|
||||
work the research rounds feed.
|
||||
|
||||
Context for why this crate starts now: **alkblobs**
|
||||
(`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning)
|
||||
@@ -589,6 +591,15 @@ to expose engine capabilities at all?
|
||||
Open; partially rides OQ-ST-04 (the trait's shape constrains where
|
||||
capability differences can surface).
|
||||
|
||||
## POC register
|
||||
|
||||
Proposals run as standalone crates in the global workspace; findings
|
||||
land in `docs/research/` here. Named per the OQ each feeds:
|
||||
|
||||
| # | POC | Spec | Feeds |
|
||||
|---|---|---|---|
|
||||
| 1 | SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx (async seam, watcher, transactional contract, packaging, interop) | [poc-sqlite-posture-spec.md](poc-sqlite-posture-spec.md) | OQ-ST-03, OQ-ST-04, OQ-ST-06 |
|
||||
|
||||
## Phase 0 plan
|
||||
|
||||
Iteration expected; this register grows as research rounds land.
|
||||
@@ -603,13 +614,13 @@ Expected sequence (deliberately rough):
|
||||
decision)**: reactive-core + engine crates. Reasoning and the
|
||||
superseded inventory lean recorded at the OQ and in the inventory.
|
||||
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library
|
||||
capability matrices, then a POC if the unified-trait shape needs
|
||||
validation (likely, given the structural mismatch noted in OQ-ST-04).
|
||||
The honker-rs surface (§Interface finding) is the concrete starting
|
||||
capability matrices (**POC #1 specified**: the SQLite posture
|
||||
comparison, [poc-sqlite-posture-spec.md](poc-sqlite-posture-spec.md);
|
||||
the honker-rs surface (§Interface finding) is the concrete starting
|
||||
artifact for the OQ-ST-04 work: pinning its contract costs less and
|
||||
is more honest than inventing a parallel shape — now scoped against
|
||||
the inventory's confirmed features rather than the full honker menu,
|
||||
and shaped as the core-crate trait surface per OQ-ST-02's split.
|
||||
and shaped as the core-crate trait surface per OQ-ST-02's split).
|
||||
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem,
|
||||
after the driver and shape questions narrow the option space.
|
||||
4. Converge; Phase 1 opens with the ADR backlog this register becomes.
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
status: spec
|
||||
title: "POC #1 — SQLite engine posture: honker-core-on-rusqlite vs honker-extension-over-sqlx"
|
||||
last_updated: 2026-10-04
|
||||
---
|
||||
|
||||
# POC: SQLite engine posture — spec
|
||||
|
||||
> **POC register #1** (phase-0.md). Feeds OQ-ST-03 (the driver question's
|
||||
> SQLite half) and OQ-ST-04 (the reactive contract's SQLite wake side).
|
||||
> Code: standalone crate in the global workspace
|
||||
> (`/workspace/alkstore-sqlite-posture-poc` — self-contained, per the
|
||||
> Phase 0 convention; POC code stays out of this repo). Findings land in
|
||||
> `docs/research/poc-sqlite-posture-findings.md` here regardless.
|
||||
|
||||
## What this POC must decide
|
||||
|
||||
OQ-ST-03 named three SQLite postures (§OQ-ST-03, option list 1–3). Option
|
||||
2 (honker-rs as substrate) is **dropped from POC scope**: its blocking
|
||||
`Database` API and mutex-pinned transaction model would have to sit
|
||||
entirely under our async core, and both other postures give us the
|
||||
machinery with more ownership — the POC's comparison between 1 and 3
|
||||
subsumes the honest parts of 2 (if 3 works and 1 works, 2 is an
|
||||
intermediate we can revisit; its unique value — the typed binding — is
|
||||
what the core crate's trait surface supersedes anyway). The POC decides:
|
||||
|
||||
**Posture A (library):** honker-core linked on *our* rusqlite connection
|
||||
— `apply_default_pragmas` + `attach_notify` + `attach_honker_functions`
|
||||
+ `bootstrap_honker_schema` (the alknet-filesystem POC's usage), sync
|
||||
work bridged to the async core per the family precedent (alktty
|
||||
REQ-TTY-01: blocking impl on dedicated threads / `spawn_blocking`,
|
||||
feeding tokio channels).
|
||||
|
||||
**Posture B (extension):** honker's loadable `.so` under **sqlx-sqlite**
|
||||
— `SqliteConnectOptions::extension(ext)` per pool connection +
|
||||
`SELECT honker_bootstrap()`, all feature calls as plain SQL through
|
||||
`SqliteExecutor<'e>` (pool / connection / transaction alike). CI-proven
|
||||
shape (honker's own `scripts/proof/orm/rust`, checkout @ f4e53c6, PR
|
||||
#106) — this POC does not re-prove basic wiring; it stress-tests what
|
||||
the proof script does not cover.
|
||||
|
||||
The question is *not* "which is faster at raw SQL" (they call the same
|
||||
machinery — any SQL-path delta is small); it is **which posture the
|
||||
sqlite engine crate should be built on**, decided at four load-bearing
|
||||
points: the async seam, the watcher, the packaging/deployment story, and
|
||||
the transactional contract.
|
||||
|
||||
## The two arms
|
||||
|
||||
Both arms implement the *same narrow slice of the alkstore core-crate
|
||||
trait sketch* (the trait surface is OQ-ST-04's contract question — the
|
||||
POC sketches just enough to compare postures, not to pin the contract):
|
||||
|
||||
- `queue_enqueue_tx` / `queue_claim_batch` / `queue_ack_batch` —
|
||||
including the transactional-enqueue property (business write + enqueue
|
||||
in one tx, rollback drops both — principle 2).
|
||||
- `notify_tx(channel, payload)` / `listen(channel)` — the reactive pair.
|
||||
- `stream_publish_tx` / `read_since(consumer)` (offset save/read only —
|
||||
not the full stream contract).
|
||||
- `lock_acquire` / `lock_release` with TTL, sufficient for alkfs's
|
||||
writer-coordination row.
|
||||
|
||||
### Arm A — honker-core on our rusqlite (library posture)
|
||||
|
||||
- `honker-core = "0.5"` (published crate, per the AGENTS.md posture —
|
||||
read the checkout for reference, build against the published version).
|
||||
- Connection model: one writer connection + reader pool (rusqlite,
|
||||
`bundled-sqlite` feature for the POC's hermetic build), bridged to
|
||||
async via a small `spawn_blocking` wrapper (measure dedicated-thread +
|
||||
mpsc as the alternative, the REQ-TTY-01 double).
|
||||
- Watcher: honker-core's `SharedUpdateWatcher::spawn_with_config`
|
||||
(default 1 ms polling) driving `listen()` wakes; listener re-reads
|
||||
after wake (the opaque-wake + re-read contract).
|
||||
|
||||
### Arm B — honker extension over sqlx (extension posture)
|
||||
|
||||
- Build the extension from the published honker source tree once
|
||||
(`cargo build --release -p honker-extension` → `.so`); the POC loads it
|
||||
via `HONKER_EXTENSION_PATH` (the guide's wiring; SeaORM/Toasty
|
||||
sections are out of scope — sqlx is the only ORM-family shape that
|
||||
matters here).
|
||||
- sqlx 0.8 `runtime-tokio`, per-pool-connection `extension(ext)` +
|
||||
`honker_bootstrap()` on connect (the proof script's pattern, applied
|
||||
to a `SqlitePool` rather than a single connection — the POC checks the
|
||||
*pool* story the script didn't run).
|
||||
- All honker SQL through `SqliteExecutor<'e>`; async calls natively —
|
||||
**no sync bridge on the call path** (this is the posture's load-
|
||||
bearing advantage; the POC verifies it survives the full surface, not
|
||||
just the proof script's enqueue path).
|
||||
- Watcher: **ours** — a `spawn_blocking`/std-thread loop polling
|
||||
`PRAGMA data_version` on a dedicated sqlx connection (the
|
||||
honker-core-watcher logic re-implemented thin over sqlx; no
|
||||
honker-core dependency in this arm — that is the point of the
|
||||
posture).
|
||||
|
||||
## Instruments
|
||||
|
||||
1. **Async-seam probe** — the decision's first axis. Arm A: call-path
|
||||
latency and executor-neutrality of the `spawn_blocking` bridge at
|
||||
realistic claim/enqueue cadence (single-digit-millisecond queue ops
|
||||
under WAL-NORMAL); thread cost per connection vs a shared bridge
|
||||
pool; `Send`/`Sync` trait-shape friction actually encountered when
|
||||
handing rusqlite types across the seam. Arm B: baseline async call
|
||||
cost through sqlx (prepared statements, pool checkout) — the honest
|
||||
floor any posture pays. Deliverable: the seam-cost table.
|
||||
2. **Watcher probe** — both arms, the reactive pair end-to-end:
|
||||
commit→wake→receiver-notified latency distribution (p50/p99) at the
|
||||
default 1 ms cadence; wake correctness under a pool of writers
|
||||
(`data_version` observed from a *different* connection than the
|
||||
writer's — Arm B's dedicated-watcher-connection shape vs Arm A's
|
||||
honker-core watcher); missed-wake stress (N rapid commits, assert
|
||||
wake count ≥ 1 and state re-read correct — the overtriggering
|
||||
contract, not per-commit wakes); behavior when the watcher
|
||||
connection is lost (Arm B) or the watcher thread panics (both).
|
||||
Deliverable: the wake-latency table + failure-mode notes.
|
||||
3. **Transactional/contract probe** — both arms, the property tests:
|
||||
business-write + enqueue + notify inside one caller-owned
|
||||
transaction; forced rollback drops both (assert no ghost rows in
|
||||
`_honker_jobs`/notification tables); concurrent writers (two async
|
||||
tasks, each in a transaction, same queue) — Arm A's rusqlite
|
||||
single-writer contention behavior vs Arm B's sqlx pool; `SQLITE_BUSY`
|
||||
behavior surfaced through each driver (the async error path honker's
|
||||
busy_timeout story must survive). Deliverable: property-test matrix
|
||||
per arm.
|
||||
4. **Packaging/deployment probe** — the honest costs, not benchmarks:
|
||||
Arm A's build graph (honker-core pulls rusqlite + chrono + feature
|
||||
surface; `bundled-sqlite` compile time; version-pinning coupling
|
||||
between our rusqlite and honker-core's); Arm B's .so lifecycle
|
||||
(build-or-download per release, load path resolution, sqlx's
|
||||
load-then-disable extension discipline verified on the pool, macOS
|
||||
and system-sqlite caveats recorded but not blockers). Deliverable:
|
||||
the packaging-notes table.
|
||||
5. **Interop check (both arms)** — two processes against one db file:
|
||||
writer in one process, listener in another (the cross-process wake
|
||||
being the whole point of the substrate). Each arm with its own
|
||||
watcher/extension setup on both sides.
|
||||
|
||||
## Decision gate
|
||||
|
||||
Arm A is chosen iff any of:
|
||||
|
||||
- the async seam costs are materially worse under B than A (B's native
|
||||
async is the posture's whole premise); or
|
||||
- the extension/pool wiring (per-connection extension + bootstrap on a
|
||||
real pool, lost-connection recovery) fails or proves fragile; or
|
||||
- the .so packaging burden is judged unacceptable for the engine crate's
|
||||
zero-ops posture (REQ-1's shape: no daemon, but also no
|
||||
binary-artifact management at runtime).
|
||||
|
||||
Arm B is chosen iff:
|
||||
|
||||
- the seam costs are materially worse under A (bridge overhead at
|
||||
claim/ack cadence, thread-per-connection, trait friction), AND B's
|
||||
wiring + watcher story holds under the probes above; or
|
||||
- sqlx's pool/prepared-statement machinery is measurably the better fit
|
||||
for the engine's concurrency posture, and the .so cost is accepted.
|
||||
|
||||
The gate may also land **"A for the engine core, B's watcher pattern for
|
||||
wake plumbing"** or similar hybrid — the probes are designed to
|
||||
separate the *call-path* question from the *watcher* question, and a
|
||||
hybrid verdict is a legitimate finding if the evidence slices that way.
|
||||
Either way the findings feed: OQ-ST-03's resolution (the SQLite driver
|
||||
posture, the tokio-postgres pg side rides separately), OQ-ST-04's SQLite
|
||||
wake contract, OQ-ST-06's fork/reference calculus (how much honker-core
|
||||
machinery each posture actually consumes), and the core-crate trait
|
||||
sketch's transactional seam (what a transaction handle looks like in
|
||||
each posture).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- The Postgres engine (tokio-postgres LISTEN/NOTIFY wiring — after the
|
||||
OQ-ST-04 contract shape is pinned; OQ-ST-03 remains coupled to it).
|
||||
- The full honker surface (scheduler, rate limits, result storage — the
|
||||
scheduler may collapse into queues per the inventory; the POC covers
|
||||
enough to validate wake + queue + offset machinery).
|
||||
- The loadable-extension surface as a *consumer-visible* feature
|
||||
(OQ-ST-07's cut stands regardless of this POC's outcome — option B
|
||||
uses the extension *mechanism* internally but the crate stays an
|
||||
in-process Rust library; the .so is never a user-facing artifact).
|
||||
- Performance optimization beyond what the probes need to produce
|
||||
comparable numbers — both arms call the same SQL machinery, so this
|
||||
POC is not a bench-fest.
|
||||
|
||||
## Register note
|
||||
|
||||
This POC is **#1** in alkstore's register (phase-0.md), specified
|
||||
2026-10-04. It is deliberately the first POC: it de-risks the SQLite
|
||||
side of OQ-ST-03 (the driver question the whole crate's shape leans on)
|
||||
with a direct A-vs-B comparison, and its trait-sketch exercise doubles
|
||||
as the first concrete pass at OQ-ST-04's contract pinning.
|
||||
Reference in new issue
Block a user