storage/docs/architecture/decisions/039-honker-as-sqlite-extension.md

ADR-039: Honker as SQLite Extension and Transport

Status

Accepted

Context

The hub architecture was designed around three separate infrastructure components:

1. PostgreSQL — persistence (tables, queries, transactions)
2. Redis — pub/sub transport for event-driven communication
3. Application-level task queues — background job processing

This creates operational complexity (three services to deploy, monitor, and secure) and a dual-write problem: writing data to PostgreSQL and publishing events to Redis cannot happen in a single transaction. If the process crashes between the DB commit and the Redis publish, data and events become inconsistent.

Honker (@russellthehippo/honker-node) is a SQLite extension that adds Postgres-style NOTIFY/LISTEN semantics, durable event streams with per-consumer offsets, at-least-once work queues with retries and dead-letter handling, cron scheduling, advisory locks, and rate limiting — all within the same SQLite .db file.

Decision

@alkdev/storage uses Honker as its SQLite extension and transport layer. Honker provides:

1. Database operations — SQLite with WAL mode, a bounded reader pool, and a single writer slot
2. Ephemeral pub/sub — notify()/listen() for fire-and-forget notifications within the DB transaction
3. Durable event streams — stream() with per-consumer offset tracking for replay-safe delivery
4. Task queues — queue() with at-least-once claims, retries, priority, delayed jobs, and dead-letter
5. Advisory locks — tryLock() for leader election and exclusive access
6. Cron scheduling — scheduler() for time-triggered operations

Drizzle ORM integrates with Honker via a thin session adapter (~100 lines) that wraps Honker's query()/execute() API inside Drizzle's SQLiteSession<'sync'> contract. No Drizzle fork required. The adapter exposes the Honker Database as $client on the Drizzle instance for direct access to pubsub/queue features.

Consequences

Positive:

• Transactional consistency: INSERT INTO nodes and queue.enqueue() commit atomically. No dual-write problem.
• No Redis dependency: Honker's stream() replaces Redis as the durable pub/sub transport
• No PostgreSQL dependency: SQLite with Honker covers persistence + events + queues
• Operational simplicity: One .db file contains everything — data, events, queues, schedules
• Drizzle integration: Full Drizzle type safety for queries + Honker for pubsub/queue on the same connection
• Ecosystem fit: The @alkdev platform is event-driven. Honker's durable streams with per-consumer offsets map directly to @alkdev/operations' call protocol events and @alkdev/flowgraph's event-sourced model

Negative:

• Single-machine: Honker is a single-process SQLite extension. No cross-server events. For multi-node deployment, a separate transport (Redis, NATS) would still be needed for internode communication.
• lastInsertRowid overhead: Honker's execute() returns only affected row count. Getting lastInsertRowid requires an extra SELECT last_insert_rowid() call via napi, or a small Rust addition to honker-node's Transaction class.
• No prepared statement handles at JS level: Every Drizzle query goes through query(sql, params). Mitigated by honker-core's prepare_cached on the Rust side.
• Honker is alpha software: Not yet beta-quality. API may change. Risk mitigated by the thin adapter — if Honker's query API changes, only the adapter needs updating.

References

• Honker source: /workspace/honker/
• Honker Node binding: /workspace/honker/packages/honker-node/
• ADR-038: SQLite-first, Postgres removed
• ADR-004: Injectable clients, no side effects (Honker client is injectable)
• @alkdev/operations call protocol architecture
• @alkdev/pubsub — Honker may replace or supplement the Redis transport