storage/docs/architecture/honker-integration.md

status, last_updated

status	last_updated
draft	2026-06-01

Honker Integration

How @alkdev/storage integrates with Honker for SQLite database operations, transactional pub/sub, durable event streams, and task queues. Includes the HonkerEventTarget adapter that bridges @alkdev/pubsub's TypedEventTarget to Honker primitives.

Purpose

Honker provides SQLite with built-in pub/sub, event streams, work queues, advisory locks, and cron scheduling — all within the same .db file. This eliminates the need for separate PostgreSQL and Redis deployments and solves the dual-write problem between data writes and event publishing.

This document specifies:

• The Drizzle-Honker session adapter architecture (ADR-044)
• The event-driven patterns enabled by Honker
• Transaction coordination between Drizzle and Honker
• The system/tenant DB model for Honker-managed databases
• Migration from the previous PostgreSQL + Redis architecture

The Adapter

Architecture

Consumer code
  ├── db.select().from(graphs)...       → Drizzle query builder
  │     └── HonkerPreparedQuery.run/all/get/values
  │           └── honkerDb.query(sql, params) / tx.execute(sql, params)
  ├── db.$client.notify('channel', p)   → Honker ephemeral pub/sub
  ├── db.$client.queue('jobs')          → Honker work queue
  ├── db.$client.stream('events')       → Honker durable stream
  └── db.$client.listen('channel')      → Honker listener

The adapter wraps Honker's Database inside Drizzle's SQLiteSession<'sync'> contract. Drizzle handles typed queries; Honker handles pubsub/queue/stream primitives. They share the same SQLite connection and transaction context.

Adapter Components

Component	Extends	Purpose
HonkerSQLiteSession	SQLiteSession<'sync'>	Routes Drizzle queries to honkerDb.query() / tx.execute()
HonkerPreparedQuery	SQLitePreparedQuery	Implements run(), all(), get(), values() via Honker
HonkerSQLiteTransaction	SQLiteTransaction<'sync'>	Drizzle transaction with $honkerTx for Honker access
drizzle(client, config)	—	Factory: creates BaseSQLiteDatabase from Honker client

Key Integration Points

Reading — all() and get() delegate to honkerDb.query(sql, params), which uses the reader pool. Honker returns Array<Record<string, any>> (row objects). The adapter converts to Drizzle's column format via mapResultRow().

Writing — run() delegates to tx.execute(sql, params) which acquires the writer slot. For standalone writes (no explicit transaction), the adapter creates a temporary transaction, executes, and commits.

Transactions — Drizzle's callback-based db.transaction((tx) => ...) wraps honker's explicit begin/commit/rollback:

transaction(callback) {
  const honkerTx = this.client.transaction();
  const txSession = new HonkerTxSession(honkerTx, ...);
  const drizzleTx = new HonkerSQLiteTransaction('sync', ..., txSession, ...);
  try {
    const result = callback(drizzleTx);
    honkerTx.commit();
    return result;
  } catch (e) {
    honkerTx.rollback();
    throw e;
  }
}

Nested transactions — Savepoints via raw SQL:

// Inside a Drizzle transaction callback
drizzleTx.transaction((nestedTx) => {
  // Adapter fires: tx.execute('SAVEPOINT sp0')
  // ... queries ...
  // Adapter fires: tx.execute('RELEASE sp0')
});

Accessing Honker from Drizzle

The adapter exposes two access points:

1. db.$client — The Honker Database instance, for use outside transactions:

const db = drizzle(honkerDb, { schema });
db.$client.notify('graph:created', { graphId });

2. tx.$honkerTx — The Honker Transaction instance, for use within Drizzle transaction callbacks:

db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
  tx.$honkerTx.notify('nodes:created', { graphId, key: 'call-1' });
  // Both commit atomically — data + notification in one transaction
});

Known Limitations

Limitation	Impact	Mitigation
No lastInsertRowid from execute()	run() needs extra SELECT last_insert_rowid()	POC 1 confirmed tx.query("SELECT last_insert_rowid() as id") works
No prepared statement handle at JS level	Every query re-enters napi	Rust-side prepare_cached amortizes
Object rows only, no raw arrays	values() must convert Object.values()	Column order preserved from SQLite
Drizzle internal API dependency	Adapter imports from drizzle-orm/sqlite-core/session	POC 1 confirmed all classes accessible and extendable
:memory: databases don't work	Reader pool gets separate in-memory DB	Always use file-based paths. Tests use temp files.

POC-Validated Adapter Architecture

POC 1 (2026-06-01) confirmed the adapter is buildable. Key findings:

• SQLiteSession, SQLitePreparedQuery, SQLiteTransaction are all accessible from drizzle-orm/sqlite-core/session and extendable.
• BaseSQLiteDatabase is accessible from drizzle-orm/sqlite-core/db.
• LibSQLSession in drizzle-orm/libsql/session is the reference implementation to follow.
• Honker query() returns { columnName: value } object rows, compatible with Drizzle's mapResultRow().
• last_insert_rowid() accessible via tx.query("SELECT last_insert_rowid() as id").
• tx.execute() returns a number (affected rows count).
• JSON-mode columns need manual JSON.parse() in the adapter.

HonkerEventTarget

The HonkerEventTarget adapts @alkdev/pubsub's TypedEventTarget interface to Honker's notify/listen (ephemeral) and stream/subscribe (durable) primitives. It lives in src/sqlite/event-target.ts.

Interface

import type { TypedEventTarget } from "@alkdev/pubsub";

interface HonkerEventTargetOptions {
  db: Database;
  mode: "ephemeral" | "durable";
  streamName?: string;
  consumerName?: string;
  prefix?: string;
}

function createHonkerEventTarget<TEvent>(
  options: HonkerEventTargetOptions
): TypedEventTarget<TEvent> & { close(): void };

Ephemeral Mode

Maps to Honker's notify()/listen():

pubsub operation	Honker operation
addEventListener("topic:id", callback)	db.listen("topic:id") → start async consumer loop
dispatchEvent(event)	db.notify(event.type, event.detail)
removeEventListener("topic:id", callback)	Close the listener when no callbacks remain
close()	Close all active listeners

POC 2 confirmed: same-process notify→listen works, multiple concurrent listeners on different channels work, cross-channel isolation is correct.

POC 4 confirmed: tx.notify() within a Drizzle transaction only fires the notification on tx.commit(). On tx.rollback(), the notification is suppressed. This enables transactional outbox semantics for ephemeral events.

Durable Mode

Maps to Honker's Stream.publish()/Stream.subscribe():

pubsub operation	Honker operation
addEventListener("topic:id", callback)	db.stream(name).subscribe(consumer) → start async consumer loop
dispatchEvent(event)	db.stream(name).publish(event.detail)
removeEventListener("topic:id", callback)	Close the subscription when no callbacks remain
close()	Close all active subscriptions, save offsets

POC 3 confirmed: Stream.publish()/Stream.subscribe() work, consumer offsets are tracked and persisted, publishTx(tx, payload) works within transactions, and concurrent stream + listener operation works.

Durable mode provides crash recovery — consumers resume from their last saved offset after restart. Consumer names must be stable across restarts (not PID-based).

Topic Routing

pubsub uses topic:id composite topics. Honker channels and streams are flat strings. The mapping:

• Ephemeral: Each unique topic:id gets its own db.listen() call. For high-cardinality topics (many request IDs), consider topic prefix matching with client-side filtering instead.
• Durable: Streams are name-keyed, not topic-keyed. A single stream carries all events for a domain. Client-side filtering dispatches only to matching listeners.

Suggested split:

Event category	Mode	Reason
Call protocol events (call.requested, call.responded, etc.)	Durable stream	Crash recovery, audit trail, flowgraph replay
Cache invalidation signals	Ephemeral	Fire-and-forget, loss acceptable
UI/dashboard push	Ephemeral	Low latency, loss acceptable
Schema migration jobs	Queue (not pubsub)	At-least-once processing

Latency Consideration

POC 2 measured ~17ms median latency for Honker notify→listen within a single process. For hot-path call protocol request/response where sub-ms latency matters, pair the Honker event target with an in-process EventTarget (pubsub's default). A composite pattern (dispatch to both) provides both in-process speed and Honker durability/cross-process coordination.

Hub-Spoke Event Routing

Hub (system.db + tenant-{orgId}.db)
  ├── HonkerEventTarget (durable: call-protocol stream)
  ├── WebSocketServerEventTarget (spoke fan-out)
  └── In-process EventTarget (local subscribers)

Spoke (tenant-{orgId}.db)
  ├── HonkerEventTarget (ephemeral: local channels)
  ├── WebSocketClientEventTarget (hub connection)
  └── In-process EventTarget (local subscribers)

Both hub and spoke use the same createPubSub({ eventTarget }) call. Different event target instances determine the routing. No code changes between hub and spoke — only configuration.

Event-Driven Patterns

Ephemeral Notifications

Fire-and-forget events for signaling. Like PostgreSQL's NOTIFY/LISTEN.

// After creating a graph, notify listeners
db.transaction((tx) => {
  tx.insert(schema.graphs).values({ name: 'session-1', status: 'draft' }).run();
  tx.$honkerTx.notify('graphs:created', { name: 'session-1' });
});

// Listener (separate process or same process)
const listener = db.$client.listen('graphs:created');
for await (const event of listener) {
  console.log('New graph:', event.payload);
}

Use cases: Cache invalidation signals, UI update push, spoke reconnection triggers.

Durable Event Streams

Per-consumer replay-safe delivery. Each consumer tracks its own offset. Crash recovery replays from the last saved offset.

// Publish call protocol events to a durable stream
const callStream = db.$client.stream('call-protocol');
callStream.publish({ type: 'call.requested', requestId: 'req-1', operationId: 'op-call' });

// Consumer: dashboard replays from its last offset
const subscription = callStream.subscribe('dashboard');
for await (const event of subscription) {
  // event has: id, payload, timestamp
  updateDashboard(event.payload);
  // offset auto-saved every 1000 events or 1 second
}

Use cases: Call protocol event persistence, audit trail replay, session event replay, cross-process coordination.

Transactional Outbox

Combine data writes with side-effect delivery. A queue job is enqueued in the same transaction as the data write, guaranteeing the side effect will eventually be processed.

// Insert a node AND schedule a downstream processing job
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'task-1', attributes: taskData }).run();
  // Outbox: enqueue a job that will be processed after commit
  const q = db.$client.queue('task-processing');
  q.enqueueTx(tx.$honkerTx, { taskKey: 'task-1', priority: 5 });
});

Use cases: ACL evaluation after permission change, encrypted data key rotation, schema migration jobs, retention cleanup.

Work Queues

At-least-once background job processing with retries, priority, delayed execution, claim expiration, and dead-letter handling.

// Enqueue a background job
const q = db.$client.queue('key-rotation');
q.enqueue({ keyVersion: 2, batchSize: 100 }, { priority: 3 });

// Worker: claim and process
const waker = q.claimWaker('rotator-1');
for await (const job of waker) {
  await rotateKeys(job.payload);
  job.ack();
}

Use cases: Key rotation, rate limit sweep, retention cleanup, schema migration across tenant DBs, notification pruning.

Advisory Locks

Named locks with TTL for leader election and exclusive access.

// Only one hub instance runs the scheduler
const lock = db.$client.tryLock('scheduler-leader', 'hub-instance-1', 30);
if (lock) {
  startScheduler(db.$client.scheduler());
  // Renew periodically
  setInterval(() => lock.heartbeat(), 10000);
}

Cron Scheduling

Time-triggered operations using 5-field cron or @every intervals.

const scheduler = db.$client.scheduler();
scheduler.add('retention-cleanup', '0 3 * * *', {
  queue: 'maintenance',
  payload: { task: 'cleanup-expired-graphs', olderThanDays: 90 },
});

System DB + Tenant DB with Honker

Each database — system and tenant — is a separate Honker-managed SQLite file with its own pubsub channels, streams, and queues.

System DB Event Channels

Channel	Direction	Payload
account:created	Ephemeral	{ accountId, email, accessLevel }
account:updated	Ephemeral	{ accountId, changes }
org:created	Ephemeral	{ orgId, slug, ownerId }
org:member_added	Ephemeral	{ orgId, accountId, membershipLevel }
auth:key_verified	Ephemeral	{ keyId, ownerId }

System DB Streams

Stream	Consumers	Purpose
audit-events	Compliance, monitoring	Append-only audit trail

System DB Queues

Queue	Worker	Purpose
key-management	Hub key service	API key rotation, expired key cleanup
account-maintenance	Hub account service	Deactivation processing, org transfer

Tenant DB Event Channels

Channel	Direction	Payload
graph:created	Ephemeral	{ graphId, graphTypeId, ownerId }
graph:updated	Ephemeral	{ graphId, changes }
nodes:created	Ephemeral	{ graphId, keys[] }
acl:delegation_changed	Ephemeral	{ principalId, agentId }

Tenant DB Streams

Stream	Consumers	Purpose
call-protocol	Flowgraph projector, observability	Call protocol event replay
session-events	Session manager, audit	Session lifecycle events

Tenant DB Queues

Queue	Worker	Purpose
acl-evaluation	ACL evaluator	Scope recalculation after delegation change
secret-rotation	Key service	Re-encryption with new key version
graph-maintenance	Maintenance service	Graph archival, retention cleanup
schema-migration	Migration service	Data migration for schema version bumps

Cross-DB Coordination

The system DB and tenant DBs are separate files. The hub mediates between them at the application layer.

Authentication Flow

1. Request → API key hash → system.db: SELECT * FROM api_keys WHERE keyHash = ?
2. system.db: Verify key → resolve ownerId → accounts row → org memberships
3. Open tenant-{orgId}.db → check ACL graph for operation access
4. Execute operation on tenant DB

Cross-Tenant Operations

If a user in org A delegates to a user in org B, both tenant DBs are involved:

1. hub opens tenant-a.db and tenant-b.db
2. tenant-a.db: Read PrincipalNode for delegator
3. tenant-b.db: Create DelegatesEdge in ACL graph
4. hub ensures both writes succeed (application-level two-phase commit or
   best-effort with reconciliation)

Cross-tenant operations are expected to be rare. For v1, best-effort with reconciliation is acceptable. A formal two-phase commit across SQLite files would require a coordinator — the hub fills this role.

Migration from PostgreSQL + Redis

What Changes

Component	Before	After
Hub database	PostgreSQL (drizzle-orm/node-postgres)	SQLite via Honker (drizzle-orm/sqlite-core + adapter)
Hub pub/sub	@alkdev/pubsub Redis transport	Honker notify()/stream() within SQLite
Hub task queue	Custom or none	Honker queue()
Hub leader election	Redis SET NX or none	Honker tryLock()
Hub scheduling	Cron daemon or none	Honker scheduler()
Hub connection	Pool → drizzle(pool, { schema })	Honker open() → drizzle(honkerDb, { schema })
Spoke database	SQLite via @libsql/client	SQLite via Honker (same engine, richer features)
Schema	pgTable (hub) + sqliteTable (spoke)	sqliteTable only
Testing	PostgreSQL + Redis containers	In-memory SQLite (:memory:) via Honker

What Doesn't Change

• The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
• Bridge functions (moduleToDbSchema, validateNode, validateEdge)
• Crypto utility (encrypt, decrypt, EncryptedDataSchema)
• TypeBox schemas from local utils (folded from drizzlebox)
• The Drizzle query builder API (same .select(), .insert(), .update(), .delete() calls)
• The @alkdev/operations call protocol (events are now published via Honker streams instead of Redis)

Migration Path

1. Add identity tables to src/sqlite/tables/ (from hub)
2. Add scoping columns to graphs table
3. Add scope column to graph_types table
4. Remove actors table and pg/ directory
5. Implement adapter in src/sqlite/adapter.ts
6. Split client factory into createSystemDatabase() / createTenantDatabase()
7. Update hub to consume new storage API instead of its own table definitions
8. Migrate hub data from PostgreSQL to SQLite (export → import script)

Design Decisions

ADR	Decision	Status
039	Honker as SQLite extension and transport	Accepted
044	Drizzle-Honker session adapter	Accepted
046	Fold @alkdev/drizzlebox as src/sqlite/utils/	Accepted
047	HonkerEventTarget adapter for pubsub	Accepted

Open Questions

• OQ-26: Can Honker fully replace @alkdev/pubsub's Redis transport for single-node deployments? Resolved: Yes. HonkerEventTarget (ADR-047) provides the adapter. Redis still needed for multi-node.
• OQ-27: How are schema migrations applied across all tenant DBs?
• OQ-28: How does cross-tenant delegation work with separate DBs?
• OQ-29: Should the adapter be published as a standalone drizzle-honker npm package for community use?
• OQ-30: Composite event target pattern — how should an in-process EventTarget and HonkerEventTarget be combined for single-node hub deployments? POC 2 showed ~17ms Honker latency vs sub-ms in-process. Design needed.
• OQ-31: Consumer naming convention for durable subscriptions — must be stable across hub restarts (not PID-based).
• OQ-32: Drizzle Kit migration compatibility — does drizzle-kit push/drizzle-kit generate work with the custom Honker adapter, or do we need a custom migration runner?

References

• Honker README: /workspace/honker/README.md
• Honker Node binding: /workspace/honker/packages/honker-node/
• Drizzle SQLite session: /workspace/drizzle-orm/src/sqlite-core/session.ts
• ADR-039: Honker as SQLite extension
• ADR-044: Drizzle-Honker session adapter
• ADR-040: System DB + tenant DB