storage/docs/architecture/honker-integration.md

---
status: draft
last_updated: 2026-06-01
---

# Honker Integration

How @alkdev/storage integrates with Honker for SQLite database operations,
transactional pub/sub, durable event streams, and task queues. Includes the
HonkerEventTarget adapter that bridges `@alkdev/pubsub`'s `TypedEventTarget`
to Honker primitives.

## Purpose

Honker provides SQLite with built-in pub/sub, event streams, work queues,
advisory locks, and cron scheduling — all within the same `.db` file. This
eliminates the need for separate PostgreSQL and Redis deployments and solves
the dual-write problem between data writes and event publishing.

This document specifies:

- The Drizzle-Honker session adapter architecture (ADR-044)
- The event-driven patterns enabled by Honker
- Transaction coordination between Drizzle and Honker
- The system/tenant DB model for Honker-managed databases
- Migration from the previous PostgreSQL + Redis architecture

## The Adapter

### Architecture

```
Consumer code
  ├── db.select().from(graphs)...       → Drizzle query builder
  │     └── HonkerPreparedQuery.run/all/get/values
  │           └── honkerDb.query(sql, params) / tx.execute(sql, params)
  ├── db.$client.notify('channel', p)   → Honker ephemeral pub/sub
  ├── db.$client.queue('jobs')          → Honker work queue
  ├── db.$client.stream('events')       → Honker durable stream
  └── db.$client.listen('channel')      → Honker listener
```

The adapter wraps Honker's `Database` inside Drizzle's `SQLiteSession<'sync'>`
contract. Drizzle handles typed queries; Honker handles pubsub/queue/stream
primitives. They share the same SQLite connection and transaction context.

### Adapter Components

| Component | Extends | Purpose |
|-----------|---------|---------|
| `HonkerSQLiteSession` | `SQLiteSession<'sync'>` | Routes Drizzle queries to `honkerDb.query()` / `tx.execute()` |
| `HonkerPreparedQuery` | `SQLitePreparedQuery` | Implements `run()`, `all()`, `get()`, `values()` via Honker |
| `HonkerSQLiteTransaction` | `SQLiteTransaction<'sync'>` | Drizzle transaction with `$honkerTx` for Honker access |
| `drizzle(client, config)` | — | Factory: creates `BaseSQLiteDatabase` from Honker client |

### Key Integration Points

**Reading** — `all()` and `get()` delegate to `honkerDb.query(sql, params)`,
which uses the reader pool. Honker returns `Array<Record<string, any>>`
(row objects). The adapter converts to Drizzle's column format via
`mapResultRow()`.

**Writing** — `run()` delegates to `tx.execute(sql, params)` which acquires the
writer slot. For standalone writes (no explicit transaction), the adapter
creates a temporary transaction, executes, and commits.

**Transactions** — Drizzle's callback-based `db.transaction((tx) => ...)` wraps
honker's explicit `begin/commit/rollback`:

```ts
transaction(callback) {
  const honkerTx = this.client.transaction();
  const txSession = new HonkerTxSession(honkerTx, ...);
  const drizzleTx = new HonkerSQLiteTransaction('sync', ..., txSession, ...);
  try {
    const result = callback(drizzleTx);
    honkerTx.commit();
    return result;
  } catch (e) {
    honkerTx.rollback();
    throw e;
  }
}
```

**Nested transactions** — Savepoints via raw SQL:

```ts
// Inside a Drizzle transaction callback
drizzleTx.transaction((nestedTx) => {
  // Adapter fires: tx.execute('SAVEPOINT sp0')
  // ... queries ...
  // Adapter fires: tx.execute('RELEASE sp0')
});
```

### Accessing Honker from Drizzle

The adapter exposes two access points:

1. **`db.$client`** — The Honker `Database` instance, for use outside transactions:

```ts
const db = drizzle(honkerDb, { schema });
db.$client.notify('graph:created', { graphId });
```

2. **`tx.$honkerTx`** — The Honker `Transaction` instance, for use within
Drizzle transaction callbacks:

```ts
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
  tx.$honkerTx.notify('nodes:created', { graphId, key: 'call-1' });
  // Both commit atomically — data + notification in one transaction
});
```

### Known Limitations

| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | POC 1 confirmed `tx.query("SELECT last_insert_rowid() as id")` works |
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | POC 1 confirmed all classes accessible and extendable |
| `:memory:` databases don't work | Reader pool gets separate in-memory DB | Always use file-based paths. Tests use temp files. |

### POC-Validated Adapter Architecture

POC 1 (2026-06-01) confirmed the adapter is buildable. Key findings:

- `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` are all
  accessible from `drizzle-orm/sqlite-core/session` and extendable.
- `BaseSQLiteDatabase` is accessible from `drizzle-orm/sqlite-core/db`.
- `LibSQLSession` in `drizzle-orm/libsql/session` is the reference
  implementation to follow.
- Honker `query()` returns `{ columnName: value }` object rows, compatible
  with Drizzle's `mapResultRow()`.
- `last_insert_rowid()` accessible via `tx.query("SELECT last_insert_rowid() as id")`.
- `tx.execute()` returns a number (affected rows count).
- JSON-mode columns need manual `JSON.parse()` in the adapter.

## HonkerEventTarget

The `HonkerEventTarget` adapts `@alkdev/pubsub`'s `TypedEventTarget` interface
to Honker's `notify`/`listen` (ephemeral) and `stream`/`subscribe` (durable)
primitives. It lives in `src/sqlite/event-target.ts`.

### Interface

```ts
import type { TypedEventTarget } from "@alkdev/pubsub";

interface HonkerEventTargetOptions {
  db: Database;
  mode: "ephemeral" | "durable";
  streamName?: string;
  consumerName?: string;
  prefix?: string;
}

function createHonkerEventTarget<TEvent>(
  options: HonkerEventTargetOptions
): TypedEventTarget<TEvent> & { close(): void };
```

### Ephemeral Mode

Maps to Honker's `notify()`/`listen()`:

| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.listen("topic:id")` → start async consumer loop |
| `dispatchEvent(event)` | `db.notify(event.type, event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the listener when no callbacks remain |
| `close()` | Close all active listeners |

POC 2 confirmed: same-process `notify`→`listen` works, multiple concurrent
listeners on different channels work, cross-channel isolation is correct.

POC 4 confirmed: `tx.notify()` within a Drizzle transaction only fires the
notification on `tx.commit()`. On `tx.rollback()`, the notification is
suppressed. This enables transactional outbox semantics for ephemeral events.

### Durable Mode

Maps to Honker's `Stream.publish()`/`Stream.subscribe()`:

| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.stream(name).subscribe(consumer)` → start async consumer loop |
| `dispatchEvent(event)` | `db.stream(name).publish(event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the subscription when no callbacks remain |
| `close()` | Close all active subscriptions, save offsets |

POC 3 confirmed: `Stream.publish()`/`Stream.subscribe()` work, consumer
offsets are tracked and persisted, `publishTx(tx, payload)` works within
transactions, and concurrent stream + listener operation works.

Durable mode provides crash recovery — consumers resume from their last saved
offset after restart. Consumer names must be stable across restarts (not
PID-based).

### Topic Routing

pubsub uses `topic:id` composite topics. Honker channels and streams are flat
strings. The mapping:

- **Ephemeral**: Each unique `topic:id` gets its own `db.listen()` call.
  For high-cardinality topics (many request IDs), consider topic prefix
  matching with client-side filtering instead.
- **Durable**: Streams are name-keyed, not topic-keyed. A single stream
  carries all events for a domain. Client-side filtering dispatches only
  to matching listeners.

Suggested split:

| Event category | Mode | Reason |
|---------------|------|--------|
| Call protocol events (`call.requested`, `call.responded`, etc.) | Durable stream | Crash recovery, audit trail, flowgraph replay |
| Cache invalidation signals | Ephemeral | Fire-and-forget, loss acceptable |
| UI/dashboard push | Ephemeral | Low latency, loss acceptable |
| Schema migration jobs | Queue (not pubsub) | At-least-once processing |

### Latency Consideration

POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.

### Hub-Spoke Event Routing

```
Hub (system.db + tenant-{orgId}.db)
  ├── HonkerEventTarget (durable: call-protocol stream)
  ├── WebSocketServerEventTarget (spoke fan-out)
  └── In-process EventTarget (local subscribers)

Spoke (tenant-{orgId}.db)
  ├── HonkerEventTarget (ephemeral: local channels)
  ├── WebSocketClientEventTarget (hub connection)
  └── In-process EventTarget (local subscribers)
```

Both hub and spoke use the same `createPubSub({ eventTarget })` call.
Different event target instances determine the routing. No code changes
between hub and spoke — only configuration.

## Event-Driven Patterns

### Ephemeral Notifications

Fire-and-forget events for signaling. Like PostgreSQL's `NOTIFY/LISTEN`.

```ts
// After creating a graph, notify listeners
db.transaction((tx) => {
  tx.insert(schema.graphs).values({ name: 'session-1', status: 'draft' }).run();
  tx.$honkerTx.notify('graphs:created', { name: 'session-1' });
});

// Listener (separate process or same process)
const listener = db.$client.listen('graphs:created');
for await (const event of listener) {
  console.log('New graph:', event.payload);
}
```

**Use cases**: Cache invalidation signals, UI update push, spoke reconnection
triggers.

### Durable Event Streams

Per-consumer replay-safe delivery. Each consumer tracks its own offset.
Crash recovery replays from the last saved offset.

```ts
// Publish call protocol events to a durable stream
const callStream = db.$client.stream('call-protocol');
callStream.publish({ type: 'call.requested', requestId: 'req-1', operationId: 'op-call' });

// Consumer: dashboard replays from its last offset
const subscription = callStream.subscribe('dashboard');
for await (const event of subscription) {
  // event has: id, payload, timestamp
  updateDashboard(event.payload);
  // offset auto-saved every 1000 events or 1 second
}
```

**Use cases**: Call protocol event persistence, audit trail replay, session
event replay, cross-process coordination.

### Transactional Outbox

Combine data writes with side-effect delivery. A queue job is enqueued in the
same transaction as the data write, guaranteeing the side effect will eventually
be processed.

```ts
// Insert a node AND schedule a downstream processing job
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'task-1', attributes: taskData }).run();
  // Outbox: enqueue a job that will be processed after commit
  const q = db.$client.queue('task-processing');
  q.enqueueTx(tx.$honkerTx, { taskKey: 'task-1', priority: 5 });
});
```

**Use cases**: ACL evaluation after permission change, encrypted data key
rotation, schema migration jobs, retention cleanup.

### Work Queues

At-least-once background job processing with retries, priority, delayed
execution, claim expiration, and dead-letter handling.

```ts
// Enqueue a background job
const q = db.$client.queue('key-rotation');
q.enqueue({ keyVersion: 2, batchSize: 100 }, { priority: 3 });

// Worker: claim and process
const waker = q.claimWaker('rotator-1');
for await (const job of waker) {
  await rotateKeys(job.payload);
  job.ack();
}
```

**Use cases**: Key rotation, rate limit sweep, retention cleanup, schema
migration across tenant DBs, notification pruning.

### Advisory Locks

Named locks with TTL for leader election and exclusive access.

```ts
// Only one hub instance runs the scheduler
const lock = db.$client.tryLock('scheduler-leader', 'hub-instance-1', 30);
if (lock) {
  startScheduler(db.$client.scheduler());
  // Renew periodically
  setInterval(() => lock.heartbeat(), 10000);
}
```

### Cron Scheduling

Time-triggered operations using 5-field cron or `@every` intervals.

```ts
const scheduler = db.$client.scheduler();
scheduler.add('retention-cleanup', '0 3 * * *', {
  queue: 'maintenance',
  payload: { task: 'cleanup-expired-graphs', olderThanDays: 90 },
});
```

## System DB + Tenant DB with Honker

Each database — system and tenant — is a separate Honker-managed SQLite file
with its own pubsub channels, streams, and queues.

### System DB Event Channels

| Channel | Direction | Payload |
|---------|-----------|---------|
| `account:created` | Ephemeral | `{ accountId, email, accessLevel }` |
| `account:updated` | Ephemeral | `{ accountId, changes }` |
| `org:created` | Ephemeral | `{ orgId, slug, ownerId }` |
| `org:member_added` | Ephemeral | `{ orgId, accountId, membershipLevel }` |
| `auth:key_verified` | Ephemeral | `{ keyId, ownerId }` |

### System DB Streams

| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `audit-events` | Compliance, monitoring | Append-only audit trail |

### System DB Queues

| Queue | Worker | Purpose |
|-------|--------|---------|
| `key-management` | Hub key service | API key rotation, expired key cleanup |
| `account-maintenance` | Hub account service | Deactivation processing, org transfer |

### Tenant DB Event Channels

| Channel | Direction | Payload |
|---------|-----------|---------|
| `graph:created` | Ephemeral | `{ graphId, graphTypeId, ownerId }` |
| `graph:updated` | Ephemeral | `{ graphId, changes }` |
| `nodes:created` | Ephemeral | `{ graphId, keys[] }` |
| `acl:delegation_changed` | Ephemeral | `{ principalId, agentId }` |

### Tenant DB Streams

| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `call-protocol` | Flowgraph projector, observability | Call protocol event replay |
| `session-events` | Session manager, audit | Session lifecycle events |

### Tenant DB Queues

| Queue | Worker | Purpose |
|-------|--------|---------|
| `acl-evaluation` | ACL evaluator | Scope recalculation after delegation change |
| `secret-rotation` | Key service | Re-encryption with new key version |
| `graph-maintenance` | Maintenance service | Graph archival, retention cleanup |
| `schema-migration` | Migration service | Data migration for schema version bumps |

## Cross-DB Coordination

The system DB and tenant DBs are separate files. The hub mediates between them
at the application layer.

### Authentication Flow

```
1. Request → API key hash → system.db: SELECT * FROM api_keys WHERE keyHash = ?
2. system.db: Verify key → resolve ownerId → accounts row → org memberships
3. Open tenant-{orgId}.db → check ACL graph for operation access
4. Execute operation on tenant DB
```

### Cross-Tenant Operations

If a user in org A delegates to a user in org B, both tenant DBs are involved:

```
1. hub opens tenant-a.db and tenant-b.db
2. tenant-a.db: Read PrincipalNode for delegator
3. tenant-b.db: Create DelegatesEdge in ACL graph
4. hub ensures both writes succeed (application-level two-phase commit or
   best-effort with reconciliation)
```

Cross-tenant operations are expected to be rare. For v1, best-effort with
reconciliation is acceptable. A formal two-phase commit across SQLite files
would require a coordinator — the hub fills this role.

## Migration from PostgreSQL + Redis

### What Changes

| Component | Before | After |
|----------|--------|-------|
| Hub database | PostgreSQL (`drizzle-orm/node-postgres`) | SQLite via Honker (`drizzle-orm/sqlite-core` + adapter) |
| Hub pub/sub | `@alkdev/pubsub` Redis transport | Honker `notify()`/`stream()` within SQLite |
| Hub task queue | Custom or none | Honker `queue()` |
| Hub leader election | Redis `SET NX` or none | Honker `tryLock()` |
| Hub scheduling | Cron daemon or none | Honker `scheduler()` |
| Hub connection | `Pool` → `drizzle(pool, { schema })` | Honker `open()` → `drizzle(honkerDb, { schema })` |
| Spoke database | SQLite via `@libsql/client` | SQLite via Honker (same engine, richer features) |
| Schema | `pgTable` (hub) + `sqliteTable` (spoke) | `sqliteTable` only |
| Testing | PostgreSQL + Redis containers | In-memory SQLite (`:memory:`) via Honker |

### What Doesn't Change

- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
- TypeBox schemas from local utils (folded from drizzlebox)
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)

### Migration Path

1. **Add identity tables** to `src/sqlite/tables/` (from hub)
2. **Add scoping columns** to `graphs` table
3. **Add scope column** to `graph_types` table
4. **Remove `actors` table** and `pg/` directory
5. **Implement adapter** in `src/sqlite/adapter.ts`
6. **Split client factory** into `createSystemDatabase()` / `createTenantDatabase()`
7. **Update hub** to consume new storage API instead of its own table definitions
8. **Migrate hub data** from PostgreSQL to SQLite (export → import script)

## Design Decisions

| ADR | Decision | Status |
|-----|----------|--------|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils/ | Accepted |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter for pubsub | Accepted |

## Open Questions

- **OQ-26**: ~~Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?~~ Resolved: Yes. HonkerEventTarget (ADR-047) provides the adapter. Redis still needed for multi-node.
- **OQ-27**: How are schema migrations applied across all tenant DBs?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?
- **OQ-30**: Composite event target pattern — how should an in-process EventTarget and HonkerEventTarget be combined for single-node hub deployments? POC 2 showed ~17ms Honker latency vs sub-ms in-process. Design needed.
- **OQ-31**: Consumer naming convention for durable subscriptions — must be stable across hub restarts (not PID-based).
- **OQ-32**: Drizzle Kit migration compatibility — does `drizzle-kit push`/`drizzle-kit generate` work with the custom Honker adapter, or do we need a custom migration runner?

## References

- Honker README: `/workspace/honker/README.md`
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- Drizzle SQLite session: `/workspace/drizzle-orm/src/sqlite-core/session.ts`
- ADR-039: Honker as SQLite extension
- ADR-044: Drizzle-Honker session adapter
- ADR-040: System DB + tenant DB