Pivot: fold drizzlebox as utils, HonkerEventTarget, OperationSpecs as repo surface
- Update architecture docs to reflect pivot from @libsql/client to Honker - Fold @alkdev/drizzlebox Phase 0 into src/sqlite/utils/ (ADR-046) - Add HonkerEventTarget adapter for pubsub TypedEventTarget (ADR-047) - Replace hand-written CRUD with OperationSpec generation (ADR-048) - Resolved OQ-26: Honker replaces Redis for single-node pub/sub (POC validated) - Updated OQ-17, OQ-18, OQ-19 for OperationSpec repository surface - Added OQ-30 (composite event target), OQ-31 (consumer naming), OQ-32 (Drizzle Kit) - POC results: adapter buildable, same-process pub/sub works, transactional outbox semantics confirmed, concurrent listeners/streams work - Research doc at docs/research/pivot-honker-sqlite-adapter.md
This commit is contained in:
@@ -1,12 +1,14 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-05-31
|
||||
last_updated: 2026-06-01
|
||||
---
|
||||
|
||||
# Honker Integration
|
||||
|
||||
How @alkdev/storage integrates with Honker for SQLite database operations,
|
||||
transactional pub/sub, durable event streams, and task queues.
|
||||
transactional pub/sub, durable event streams, and task queues. Includes the
|
||||
HonkerEventTarget adapter that bridges `@alkdev/pubsub`'s `TypedEventTarget`
|
||||
to Honker primitives.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -118,10 +120,134 @@ db.transaction((tx) => {
|
||||
|
||||
| Limitation | Impact | Mitigation |
|
||||
|------------|--------|------------|
|
||||
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | Small Rust PR to honker-node |
|
||||
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | POC 1 confirmed `tx.query("SELECT last_insert_rowid() as id")` works |
|
||||
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
|
||||
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
|
||||
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | Stable across minor versions |
|
||||
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | POC 1 confirmed all classes accessible and extendable |
|
||||
| `:memory:` databases don't work | Reader pool gets separate in-memory DB | Always use file-based paths. Tests use temp files. |
|
||||
|
||||
### POC-Validated Adapter Architecture
|
||||
|
||||
POC 1 (2026-06-01) confirmed the adapter is buildable. Key findings:
|
||||
|
||||
- `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` are all
|
||||
accessible from `drizzle-orm/sqlite-core/session` and extendable.
|
||||
- `BaseSQLiteDatabase` is accessible from `drizzle-orm/sqlite-core/db`.
|
||||
- `LibSQLSession` in `drizzle-orm/libsql/session` is the reference
|
||||
implementation to follow.
|
||||
- Honker `query()` returns `{ columnName: value }` object rows, compatible
|
||||
with Drizzle's `mapResultRow()`.
|
||||
- `last_insert_rowid()` accessible via `tx.query("SELECT last_insert_rowid() as id")`.
|
||||
- `tx.execute()` returns a number (affected rows count).
|
||||
- JSON-mode columns need manual `JSON.parse()` in the adapter.
|
||||
|
||||
## HonkerEventTarget
|
||||
|
||||
The `HonkerEventTarget` adapts `@alkdev/pubsub`'s `TypedEventTarget` interface
|
||||
to Honker's `notify`/`listen` (ephemeral) and `stream`/`subscribe` (durable)
|
||||
primitives. It lives in `src/sqlite/event-target.ts`.
|
||||
|
||||
### Interface
|
||||
|
||||
```ts
|
||||
import type { TypedEventTarget } from "@alkdev/pubsub";
|
||||
|
||||
interface HonkerEventTargetOptions {
|
||||
db: Database;
|
||||
mode: "ephemeral" | "durable";
|
||||
streamName?: string;
|
||||
consumerName?: string;
|
||||
prefix?: string;
|
||||
}
|
||||
|
||||
function createHonkerEventTarget<TEvent>(
|
||||
options: HonkerEventTargetOptions
|
||||
): TypedEventTarget<TEvent> & { close(): void };
|
||||
```
|
||||
|
||||
### Ephemeral Mode
|
||||
|
||||
Maps to Honker's `notify()`/`listen()`:
|
||||
|
||||
| pubsub operation | Honker operation |
|
||||
|-------------------|------------------|
|
||||
| `addEventListener("topic:id", callback)` | `db.listen("topic:id")` → start async consumer loop |
|
||||
| `dispatchEvent(event)` | `db.notify(event.type, event.detail)` |
|
||||
| `removeEventListener("topic:id", callback)` | Close the listener when no callbacks remain |
|
||||
| `close()` | Close all active listeners |
|
||||
|
||||
POC 2 confirmed: same-process `notify`→`listen` works, multiple concurrent
|
||||
listeners on different channels work, cross-channel isolation is correct.
|
||||
|
||||
POC 4 confirmed: `tx.notify()` within a Drizzle transaction only fires the
|
||||
notification on `tx.commit()`. On `tx.rollback()`, the notification is
|
||||
suppressed. This enables transactional outbox semantics for ephemeral events.
|
||||
|
||||
### Durable Mode
|
||||
|
||||
Maps to Honker's `Stream.publish()`/`Stream.subscribe()`:
|
||||
|
||||
| pubsub operation | Honker operation |
|
||||
|-------------------|------------------|
|
||||
| `addEventListener("topic:id", callback)` | `db.stream(name).subscribe(consumer)` → start async consumer loop |
|
||||
| `dispatchEvent(event)` | `db.stream(name).publish(event.detail)` |
|
||||
| `removeEventListener("topic:id", callback)` | Close the subscription when no callbacks remain |
|
||||
| `close()` | Close all active subscriptions, save offsets |
|
||||
|
||||
POC 3 confirmed: `Stream.publish()`/`Stream.subscribe()` work, consumer
|
||||
offsets are tracked and persisted, `publishTx(tx, payload)` works within
|
||||
transactions, and concurrent stream + listener operation works.
|
||||
|
||||
Durable mode provides crash recovery — consumers resume from their last saved
|
||||
offset after restart. Consumer names must be stable across restarts (not
|
||||
PID-based).
|
||||
|
||||
### Topic Routing
|
||||
|
||||
pubsub uses `topic:id` composite topics. Honker channels and streams are flat
|
||||
strings. The mapping:
|
||||
|
||||
- **Ephemeral**: Each unique `topic:id` gets its own `db.listen()` call.
|
||||
For high-cardinality topics (many request IDs), consider topic prefix
|
||||
matching with client-side filtering instead.
|
||||
- **Durable**: Streams are name-keyed, not topic-keyed. A single stream
|
||||
carries all events for a domain. Client-side filtering dispatches only
|
||||
to matching listeners.
|
||||
|
||||
Suggested split:
|
||||
|
||||
| Event category | Mode | Reason |
|
||||
|---------------|------|--------|
|
||||
| Call protocol events (`call.requested`, `call.responded`, etc.) | Durable stream | Crash recovery, audit trail, flowgraph replay |
|
||||
| Cache invalidation signals | Ephemeral | Fire-and-forget, loss acceptable |
|
||||
| UI/dashboard push | Ephemeral | Low latency, loss acceptable |
|
||||
| Schema migration jobs | Queue (not pubsub) | At-least-once processing |
|
||||
|
||||
### Latency Consideration
|
||||
|
||||
POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
|
||||
single process. For hot-path call protocol request/response where sub-ms
|
||||
latency matters, pair the Honker event target with an in-process `EventTarget`
|
||||
(pubsub's default). A composite pattern (dispatch to both) provides both
|
||||
in-process speed and Honker durability/cross-process coordination.
|
||||
|
||||
### Hub-Spoke Event Routing
|
||||
|
||||
```
|
||||
Hub (system.db + tenant-{orgId}.db)
|
||||
├── HonkerEventTarget (durable: call-protocol stream)
|
||||
├── WebSocketServerEventTarget (spoke fan-out)
|
||||
└── In-process EventTarget (local subscribers)
|
||||
|
||||
Spoke (tenant-{orgId}.db)
|
||||
├── HonkerEventTarget (ephemeral: local channels)
|
||||
├── WebSocketClientEventTarget (hub connection)
|
||||
└── In-process EventTarget (local subscribers)
|
||||
```
|
||||
|
||||
Both hub and spoke use the same `createPubSub({ eventTarget })` call.
|
||||
Different event target instances determine the routing. No code changes
|
||||
between hub and spoke — only configuration.
|
||||
|
||||
## Event-Driven Patterns
|
||||
|
||||
@@ -338,7 +464,7 @@ would require a coordinator — the hub fills this role.
|
||||
- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
|
||||
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
|
||||
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
|
||||
- TypeBox schemas from drizzlebox
|
||||
- TypeBox schemas from local utils (folded from drizzlebox)
|
||||
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
|
||||
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)
|
||||
|
||||
@@ -359,13 +485,18 @@ would require a coordinator — the hub fills this role.
|
||||
|-----|----------|--------|
|
||||
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
|
||||
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
|
||||
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils/ | Accepted |
|
||||
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter for pubsub | Accepted |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **OQ-26**: Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?
|
||||
- **OQ-26**: ~~Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?~~ Resolved: Yes. HonkerEventTarget (ADR-047) provides the adapter. Redis still needed for multi-node.
|
||||
- **OQ-27**: How are schema migrations applied across all tenant DBs?
|
||||
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
|
||||
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?
|
||||
- **OQ-30**: Composite event target pattern — how should an in-process EventTarget and HonkerEventTarget be combined for single-node hub deployments? POC 2 showed ~17ms Honker latency vs sub-ms in-process. Design needed.
|
||||
- **OQ-31**: Consumer naming convention for durable subscriptions — must be stable across hub restarts (not PID-based).
|
||||
- **OQ-32**: Drizzle Kit migration compatibility — does `drizzle-kit push`/`drizzle-kit generate` work with the custom Honker adapter, or do we need a custom migration runner?
|
||||
|
||||
## References
|
||||
|
||||
|
||||
Reference in New Issue
Block a user