storage/docs/architecture/honker-integration.md

---
status: draft
last_updated: 2026-05-31
---

# Honker Integration

How @alkdev/storage integrates with Honker for SQLite database operations,
transactional pub/sub, durable event streams, and task queues.

## Purpose

Honker provides SQLite with built-in pub/sub, event streams, work queues,
advisory locks, and cron scheduling — all within the same `.db` file. This
eliminates the need for separate PostgreSQL and Redis deployments and solves
the dual-write problem between data writes and event publishing.

This document specifies:

- The Drizzle-Honker session adapter architecture (ADR-044)
- The event-driven patterns enabled by Honker
- Transaction coordination between Drizzle and Honker
- The system/tenant DB model for Honker-managed databases
- Migration from the previous PostgreSQL + Redis architecture

## The Adapter

### Architecture

```
Consumer code
  ├── db.select().from(graphs)...       → Drizzle query builder
  │     └── HonkerPreparedQuery.run/all/get/values
  │           └── honkerDb.query(sql, params) / tx.execute(sql, params)
  ├── db.$client.notify('channel', p)   → Honker ephemeral pub/sub
  ├── db.$client.queue('jobs')          → Honker work queue
  ├── db.$client.stream('events')       → Honker durable stream
  └── db.$client.listen('channel')      → Honker listener
```

The adapter wraps Honker's `Database` inside Drizzle's `SQLiteSession<'sync'>`
contract. Drizzle handles typed queries; Honker handles pubsub/queue/stream
primitives. They share the same SQLite connection and transaction context.

### Adapter Components

| Component | Extends | Purpose |
|-----------|---------|---------|
| `HonkerSQLiteSession` | `SQLiteSession<'sync'>` | Routes Drizzle queries to `honkerDb.query()` / `tx.execute()` |
| `HonkerPreparedQuery` | `SQLitePreparedQuery` | Implements `run()`, `all()`, `get()`, `values()` via Honker |
| `HonkerSQLiteTransaction` | `SQLiteTransaction<'sync'>` | Drizzle transaction with `$honkerTx` for Honker access |
| `drizzle(client, config)` | — | Factory: creates `BaseSQLiteDatabase` from Honker client |

### Key Integration Points

**Reading** — `all()` and `get()` delegate to `honkerDb.query(sql, params)`,
which uses the reader pool. Honker returns `Array<Record<string, any>>`
(row objects). The adapter converts to Drizzle's column format via
`mapResultRow()`.

**Writing** — `run()` delegates to `tx.execute(sql, params)` which acquires the
writer slot. For standalone writes (no explicit transaction), the adapter
creates a temporary transaction, executes, and commits.

**Transactions** — Drizzle's callback-based `db.transaction((tx) => ...)` wraps
honker's explicit `begin/commit/rollback`:

```ts
transaction(callback) {
  const honkerTx = this.client.transaction();
  const txSession = new HonkerTxSession(honkerTx, ...);
  const drizzleTx = new HonkerSQLiteTransaction('sync', ..., txSession, ...);
  try {
    const result = callback(drizzleTx);
    honkerTx.commit();
    return result;
  } catch (e) {
    honkerTx.rollback();
    throw e;
  }
}
```

**Nested transactions** — Savepoints via raw SQL:

```ts
// Inside a Drizzle transaction callback
drizzleTx.transaction((nestedTx) => {
  // Adapter fires: tx.execute('SAVEPOINT sp0')
  // ... queries ...
  // Adapter fires: tx.execute('RELEASE sp0')
});
```

### Accessing Honker from Drizzle

The adapter exposes two access points:

1. **`db.$client`** — The Honker `Database` instance, for use outside transactions:

```ts
const db = drizzle(honkerDb, { schema });
db.$client.notify('graph:created', { graphId });
```

2. **`tx.$honkerTx`** — The Honker `Transaction` instance, for use within
Drizzle transaction callbacks:

```ts
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
  tx.$honkerTx.notify('nodes:created', { graphId, key: 'call-1' });
  // Both commit atomically — data + notification in one transaction
});
```

### Known Limitations

| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | Small Rust PR to honker-node |
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | Stable across minor versions |

## Event-Driven Patterns

### Ephemeral Notifications

Fire-and-forget events for signaling. Like PostgreSQL's `NOTIFY/LISTEN`.

```ts
// After creating a graph, notify listeners
db.transaction((tx) => {
  tx.insert(schema.graphs).values({ name: 'session-1', status: 'draft' }).run();
  tx.$honkerTx.notify('graphs:created', { name: 'session-1' });
});

// Listener (separate process or same process)
const listener = db.$client.listen('graphs:created');
for await (const event of listener) {
  console.log('New graph:', event.payload);
}
```

**Use cases**: Cache invalidation signals, UI update push, spoke reconnection
triggers.

### Durable Event Streams

Per-consumer replay-safe delivery. Each consumer tracks its own offset.
Crash recovery replays from the last saved offset.

```ts
// Publish call protocol events to a durable stream
const callStream = db.$client.stream('call-protocol');
callStream.publish({ type: 'call.requested', requestId: 'req-1', operationId: 'op-call' });

// Consumer: dashboard replays from its last offset
const subscription = callStream.subscribe('dashboard');
for await (const event of subscription) {
  // event has: id, payload, timestamp
  updateDashboard(event.payload);
  // offset auto-saved every 1000 events or 1 second
}
```

**Use cases**: Call protocol event persistence, audit trail replay, session
event replay, cross-process coordination.

### Transactional Outbox

Combine data writes with side-effect delivery. A queue job is enqueued in the
same transaction as the data write, guaranteeing the side effect will eventually
be processed.

```ts
// Insert a node AND schedule a downstream processing job
db.transaction((tx) => {
  tx.insert(schema.nodes).values({ graphId, key: 'task-1', attributes: taskData }).run();
  // Outbox: enqueue a job that will be processed after commit
  const q = db.$client.queue('task-processing');
  q.enqueueTx(tx.$honkerTx, { taskKey: 'task-1', priority: 5 });
});
```

**Use cases**: ACL evaluation after permission change, encrypted data key
rotation, schema migration jobs, retention cleanup.

### Work Queues

At-least-once background job processing with retries, priority, delayed
execution, claim expiration, and dead-letter handling.

```ts
// Enqueue a background job
const q = db.$client.queue('key-rotation');
q.enqueue({ keyVersion: 2, batchSize: 100 }, { priority: 3 });

// Worker: claim and process
const waker = q.claimWaker('rotator-1');
for await (const job of waker) {
  await rotateKeys(job.payload);
  job.ack();
}
```

**Use cases**: Key rotation, rate limit sweep, retention cleanup, schema
migration across tenant DBs, notification pruning.

### Advisory Locks

Named locks with TTL for leader election and exclusive access.

```ts
// Only one hub instance runs the scheduler
const lock = db.$client.tryLock('scheduler-leader', 'hub-instance-1', 30);
if (lock) {
  startScheduler(db.$client.scheduler());
  // Renew periodically
  setInterval(() => lock.heartbeat(), 10000);
}
```

### Cron Scheduling

Time-triggered operations using 5-field cron or `@every` intervals.

```ts
const scheduler = db.$client.scheduler();
scheduler.add('retention-cleanup', '0 3 * * *', {
  queue: 'maintenance',
  payload: { task: 'cleanup-expired-graphs', olderThanDays: 90 },
});
```

## System DB + Tenant DB with Honker

Each database — system and tenant — is a separate Honker-managed SQLite file
with its own pubsub channels, streams, and queues.

### System DB Event Channels

| Channel | Direction | Payload |
|---------|-----------|---------|
| `account:created` | Ephemeral | `{ accountId, email, accessLevel }` |
| `account:updated` | Ephemeral | `{ accountId, changes }` |
| `org:created` | Ephemeral | `{ orgId, slug, ownerId }` |
| `org:member_added` | Ephemeral | `{ orgId, accountId, membershipLevel }` |
| `auth:key_verified` | Ephemeral | `{ keyId, ownerId }` |

### System DB Streams

| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `audit-events` | Compliance, monitoring | Append-only audit trail |

### System DB Queues

| Queue | Worker | Purpose |
|-------|--------|---------|
| `key-management` | Hub key service | API key rotation, expired key cleanup |
| `account-maintenance` | Hub account service | Deactivation processing, org transfer |

### Tenant DB Event Channels

| Channel | Direction | Payload |
|---------|-----------|---------|
| `graph:created` | Ephemeral | `{ graphId, graphTypeId, ownerId }` |
| `graph:updated` | Ephemeral | `{ graphId, changes }` |
| `nodes:created` | Ephemeral | `{ graphId, keys[] }` |
| `acl:delegation_changed` | Ephemeral | `{ principalId, agentId }` |

### Tenant DB Streams

| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `call-protocol` | Flowgraph projector, observability | Call protocol event replay |
| `session-events` | Session manager, audit | Session lifecycle events |

### Tenant DB Queues

| Queue | Worker | Purpose |
|-------|--------|---------|
| `acl-evaluation` | ACL evaluator | Scope recalculation after delegation change |
| `secret-rotation` | Key service | Re-encryption with new key version |
| `graph-maintenance` | Maintenance service | Graph archival, retention cleanup |
| `schema-migration` | Migration service | Data migration for schema version bumps |

## Cross-DB Coordination

The system DB and tenant DBs are separate files. The hub mediates between them
at the application layer.

### Authentication Flow

```
1. Request → API key hash → system.db: SELECT * FROM api_keys WHERE keyHash = ?
2. system.db: Verify key → resolve ownerId → accounts row → org memberships
3. Open tenant-{orgId}.db → check ACL graph for operation access
4. Execute operation on tenant DB
```

### Cross-Tenant Operations

If a user in org A delegates to a user in org B, both tenant DBs are involved:

```
1. hub opens tenant-a.db and tenant-b.db
2. tenant-a.db: Read PrincipalNode for delegator
3. tenant-b.db: Create DelegatesEdge in ACL graph
4. hub ensures both writes succeed (application-level two-phase commit or
   best-effort with reconciliation)
```

Cross-tenant operations are expected to be rare. For v1, best-effort with
reconciliation is acceptable. A formal two-phase commit across SQLite files
would require a coordinator — the hub fills this role.

## Migration from PostgreSQL + Redis

### What Changes

| Component | Before | After |
|----------|--------|-------|
| Hub database | PostgreSQL (`drizzle-orm/node-postgres`) | SQLite via Honker (`drizzle-orm/sqlite-core` + adapter) |
| Hub pub/sub | `@alkdev/pubsub` Redis transport | Honker `notify()`/`stream()` within SQLite |
| Hub task queue | Custom or none | Honker `queue()` |
| Hub leader election | Redis `SET NX` or none | Honker `tryLock()` |
| Hub scheduling | Cron daemon or none | Honker `scheduler()` |
| Hub connection | `Pool` → `drizzle(pool, { schema })` | Honker `open()` → `drizzle(honkerDb, { schema })` |
| Spoke database | SQLite via `@libsql/client` | SQLite via Honker (same engine, richer features) |
| Schema | `pgTable` (hub) + `sqliteTable` (spoke) | `sqliteTable` only |
| Testing | PostgreSQL + Redis containers | In-memory SQLite (`:memory:`) via Honker |

### What Doesn't Change

- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
- TypeBox schemas from drizzlebox
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)

### Migration Path

1. **Add identity tables** to `src/sqlite/tables/` (from hub)
2. **Add scoping columns** to `graphs` table
3. **Add scope column** to `graph_types` table
4. **Remove `actors` table** and `pg/` directory
5. **Implement adapter** in `src/sqlite/adapter.ts`
6. **Split client factory** into `createSystemDatabase()` / `createTenantDatabase()`
7. **Update hub** to consume new storage API instead of its own table definitions
8. **Migrate hub data** from PostgreSQL to SQLite (export → import script)

## Design Decisions

| ADR | Decision | Status |
|-----|----------|--------|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |

## Open Questions

- **OQ-26**: Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?
- **OQ-27**: How are schema migrations applied across all tenant DBs?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?

## References

- Honker README: `/workspace/honker/README.md`
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- Drizzle SQLite session: `/workspace/drizzle-orm/src/sqlite-core/session.ts`
- ADR-039: Honker as SQLite extension
- ADR-044: Drizzle-Honker session adapter
- ADR-040: System DB + tenant DB