storage/docs/architecture/honker-integration.md

status, last_updated

status	last_updated
draft	2026-05-31

Honker Integration

How @alkdev/storage integrates with Honker for SQLite database operations, transactional pub/sub, durable event streams, and task queues.

Purpose

Honker provides SQLite with built-in pub/sub, event streams, work queues, advisory locks, and cron scheduling — all within the same .db file. This eliminates the need for separate PostgreSQL and Redis deployments and solves the dual-write problem between data writes and event publishing.

This document specifies:

• The Drizzle-Honker session adapter architecture (ADR-044)
• The event-driven patterns enabled by Honker
• Transaction coordination between Drizzle and Honker
• The system/tenant DB model for Honker-managed databases
• Migration from the previous PostgreSQL + Redis architecture

The Adapter

Architecture

Consumer code
  ├── db.select().from(graphs)...       → Drizzle query builder
  │     └── HonkerPreparedQuery.run/all/get/values
  │           └── honkerDb.query(sql, params) / tx.execute(sql, params)
  ├── db.$client.notify('channel', p)   → Honker ephemeral pub/sub
  ├── db.$client.queue('jobs')          → Honker work queue
  ├── db.$client.stream('events')       → Honker durable stream
  └── db.$client.listen('channel')      → Honker listener

The adapter wraps Honker's Database inside Drizzle's SQLiteSession<'sync'> contract. Drizzle handles typed queries; Honker handles pubsub/queue/stream primitives. They share the same SQLite connection and transaction context.

Adapter Components

Component	Extends	Purpose
HonkerSQLiteSession	SQLiteSession<'sync'>	Routes Drizzle queries to honkerDb.query() / tx.execute()
HonkerPreparedQuery	SQLitePreparedQuery	Implements run(), all(), get(), values() via Honker
HonkerSQLiteTransaction	SQLiteTransaction<'sync'>	Drizzle transaction with $honkerTx for Honker access
drizzle(client, config)	—	Factory: creates BaseSQLiteDatabase from Honker client

Key Integration Points

Reading — all() and get() delegate to honkerDb.query(sql, params), which uses the reader pool. Honker returns Array<Record<string, any>> (row objects). The adapter converts to Drizzle's column format via mapResultRow().

Writing — run() delegates to tx.execute(sql, params) which acquires the writer slot. For standalone writes (no explicit transaction), the adapter creates a temporary transaction, executes, and commits.

Transactions — Drizzle's callback-based db.transaction((tx) => ...) wraps honker's explicit begin/commit/rollback:

transaction(callback) {
  const honkerTx = this.client.transaction();
  const txSession = new HonkerTxSession(honkerTx, ...);
  const drizzleTx = new HonkerSQLiteTransaction('sync', ..., txSession, ...);
  try {
    const result = callback(drizzleTx);
    honkerTx.commit();
    return result;
  } catch (e) {
    honkerTx.rollback();
    throw e;
  }
}

Nested transactions — Savepoints via raw SQL:

// Inside a Drizzle transaction callback
drizzleTx.transaction((nestedTx) => {
  // Adapter fires: tx.execute('SAVEPOINT sp0')
  // ... queries ...
  // Adapter fires: tx.execute('RELEASE sp0')
});

Accessing Honker from Drizzle

The adapter exposes two access points:

1. db.$client — The Honker Database instance, for use outside transactions:

const db = drizzle(honkerDb, { schema });
db.$client.notify('graph:created', { graphId });

2. tx.$honkerTx — The Honker Transaction instance, for use within Drizzle transaction callbacks:

db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
  tx.$honkerTx.notify('nodes:created', { graphId, key: 'call-1' });
  // Both commit atomically — data + notification in one transaction
});

Known Limitations

Limitation	Impact	Mitigation
No lastInsertRowid from execute()	run() needs extra SELECT last_insert_rowid()	Small Rust PR to honker-node
No prepared statement handle at JS level	Every query re-enters napi	Rust-side prepare_cached amortizes
Object rows only, no raw arrays	values() must convert Object.values()	Column order preserved from SQLite
Drizzle internal API dependency	Adapter imports from drizzle-orm/sqlite-core/session	Stable across minor versions

Event-Driven Patterns

Ephemeral Notifications

Fire-and-forget events for signaling. Like PostgreSQL's NOTIFY/LISTEN.

// After creating a graph, notify listeners
db.transaction((tx) => {
  tx.insert(schema.graphs).values({ name: 'session-1', status: 'draft' }).run();
  tx.$honkerTx.notify('graphs:created', { name: 'session-1' });
});

// Listener (separate process or same process)
const listener = db.$client.listen('graphs:created');
for await (const event of listener) {
  console.log('New graph:', event.payload);
}

Use cases: Cache invalidation signals, UI update push, spoke reconnection triggers.

Durable Event Streams

Per-consumer replay-safe delivery. Each consumer tracks its own offset. Crash recovery replays from the last saved offset.

// Publish call protocol events to a durable stream
const callStream = db.$client.stream('call-protocol');
callStream.publish({ type: 'call.requested', requestId: 'req-1', operationId: 'op-call' });

// Consumer: dashboard replays from its last offset
const subscription = callStream.subscribe('dashboard');
for await (const event of subscription) {
  // event has: id, payload, timestamp
  updateDashboard(event.payload);
  // offset auto-saved every 1000 events or 1 second
}

Use cases: Call protocol event persistence, audit trail replay, session event replay, cross-process coordination.

Transactional Outbox

Combine data writes with side-effect delivery. A queue job is enqueued in the same transaction as the data write, guaranteeing the side effect will eventually be processed.

// Insert a node AND schedule a downstream processing job
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'task-1', attributes: taskData }).run();
  // Outbox: enqueue a job that will be processed after commit
  const q = db.$client.queue('task-processing');
  q.enqueueTx(tx.$honkerTx, { taskKey: 'task-1', priority: 5 });
});

Use cases: ACL evaluation after permission change, encrypted data key rotation, schema migration jobs, retention cleanup.

Work Queues

At-least-once background job processing with retries, priority, delayed execution, claim expiration, and dead-letter handling.

// Enqueue a background job
const q = db.$client.queue('key-rotation');
q.enqueue({ keyVersion: 2, batchSize: 100 }, { priority: 3 });

// Worker: claim and process
const waker = q.claimWaker('rotator-1');
for await (const job of waker) {
  await rotateKeys(job.payload);
  job.ack();
}

Use cases: Key rotation, rate limit sweep, retention cleanup, schema migration across tenant DBs, notification pruning.

Advisory Locks

Named locks with TTL for leader election and exclusive access.

// Only one hub instance runs the scheduler
const lock = db.$client.tryLock('scheduler-leader', 'hub-instance-1', 30);
if (lock) {
  startScheduler(db.$client.scheduler());
  // Renew periodically
  setInterval(() => lock.heartbeat(), 10000);
}

Cron Scheduling

Time-triggered operations using 5-field cron or @every intervals.

const scheduler = db.$client.scheduler();
scheduler.add('retention-cleanup', '0 3 * * *', {
  queue: 'maintenance',
  payload: { task: 'cleanup-expired-graphs', olderThanDays: 90 },
});

System DB + Tenant DB with Honker

Each database — system and tenant — is a separate Honker-managed SQLite file with its own pubsub channels, streams, and queues.

System DB Event Channels

Channel	Direction	Payload
account:created	Ephemeral	{ accountId, email, accessLevel }
account:updated	Ephemeral	{ accountId, changes }
org:created	Ephemeral	{ orgId, slug, ownerId }
org:member_added	Ephemeral	{ orgId, accountId, membershipLevel }
auth:key_verified	Ephemeral	{ keyId, ownerId }

System DB Streams

Stream	Consumers	Purpose
audit-events	Compliance, monitoring	Append-only audit trail

System DB Queues

Queue	Worker	Purpose
key-management	Hub key service	API key rotation, expired key cleanup
account-maintenance	Hub account service	Deactivation processing, org transfer

Tenant DB Event Channels

Channel	Direction	Payload
graph:created	Ephemeral	{ graphId, graphTypeId, ownerId }
graph:updated	Ephemeral	{ graphId, changes }
nodes:created	Ephemeral	{ graphId, keys[] }
acl:delegation_changed	Ephemeral	{ principalId, agentId }

Tenant DB Streams

Stream	Consumers	Purpose
call-protocol	Flowgraph projector, observability	Call protocol event replay
session-events	Session manager, audit	Session lifecycle events

Tenant DB Queues

Queue	Worker	Purpose
acl-evaluation	ACL evaluator	Scope recalculation after delegation change
secret-rotation	Key service	Re-encryption with new key version
graph-maintenance	Maintenance service	Graph archival, retention cleanup
schema-migration	Migration service	Data migration for schema version bumps

Cross-DB Coordination

The system DB and tenant DBs are separate files. The hub mediates between them at the application layer.

Authentication Flow

1. Request → API key hash → system.db: SELECT * FROM api_keys WHERE keyHash = ?
2. system.db: Verify key → resolve ownerId → accounts row → org memberships
3. Open tenant-{orgId}.db → check ACL graph for operation access
4. Execute operation on tenant DB

Cross-Tenant Operations

If a user in org A delegates to a user in org B, both tenant DBs are involved:

1. hub opens tenant-a.db and tenant-b.db
2. tenant-a.db: Read PrincipalNode for delegator
3. tenant-b.db: Create DelegatesEdge in ACL graph
4. hub ensures both writes succeed (application-level two-phase commit or
   best-effort with reconciliation)

Cross-tenant operations are expected to be rare. For v1, best-effort with reconciliation is acceptable. A formal two-phase commit across SQLite files would require a coordinator — the hub fills this role.

Migration from PostgreSQL + Redis

What Changes

Component	Before	After
Hub database	PostgreSQL (drizzle-orm/node-postgres)	SQLite via Honker (drizzle-orm/sqlite-core + adapter)
Hub pub/sub	@alkdev/pubsub Redis transport	Honker notify()/stream() within SQLite
Hub task queue	Custom or none	Honker queue()
Hub leader election	Redis SET NX or none	Honker tryLock()
Hub scheduling	Cron daemon or none	Honker scheduler()
Hub connection	Pool → drizzle(pool, { schema })	Honker open() → drizzle(honkerDb, { schema })
Spoke database	SQLite via @libsql/client	SQLite via Honker (same engine, richer features)
Schema	pgTable (hub) + sqliteTable (spoke)	sqliteTable only
Testing	PostgreSQL + Redis containers	In-memory SQLite (:memory:) via Honker

What Doesn't Change

• The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
• Bridge functions (moduleToDbSchema, validateNode, validateEdge)
• Crypto utility (encrypt, decrypt, EncryptedDataSchema)
• TypeBox schemas from drizzlebox
• The Drizzle query builder API (same .select(), .insert(), .update(), .delete() calls)
• The @alkdev/operations call protocol (events are now published via Honker streams instead of Redis)

Migration Path

1. Add identity tables to src/sqlite/tables/ (from hub)
2. Add scoping columns to graphs table
3. Add scope column to graph_types table
4. Remove actors table and pg/ directory
5. Implement adapter in src/sqlite/adapter.ts
6. Split client factory into createSystemDatabase() / createTenantDatabase()
7. Update hub to consume new storage API instead of its own table definitions
8. Migrate hub data from PostgreSQL to SQLite (export → import script)

Design Decisions

ADR	Decision	Status
039	Honker as SQLite extension and transport	Accepted
044	Drizzle-Honker session adapter	Accepted

Open Questions

• OQ-26: Can Honker fully replace @alkdev/pubsub's Redis transport for single-node deployments?
• OQ-27: How are schema migrations applied across all tenant DBs?
• OQ-28: How does cross-tenant delegation work with separate DBs?
• OQ-29: Should the adapter be published as a standalone drizzle-honker npm package for community use?

References

• Honker README: /workspace/honker/README.md
• Honker Node binding: /workspace/honker/packages/honker-node/
• Drizzle SQLite session: /workspace/drizzle-orm/src/sqlite-core/session.ts
• ADR-039: Honker as SQLite extension
• ADR-044: Drizzle-Honker session adapter
• ADR-040: System DB + tenant DB