alkdev/storage
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pivot: fold drizzlebox as utils, HonkerEventTarget, OperationSpecs as repo surface

Browse Source 
- Update architecture docs to reflect pivot from @libsql/client to Honker
- Fold @alkdev/drizzlebox Phase 0 into src/sqlite/utils/ (ADR-046)
- Add HonkerEventTarget adapter for pubsub TypedEventTarget (ADR-047)
- Replace hand-written CRUD with OperationSpec generation (ADR-048)
- Resolved OQ-26: Honker replaces Redis for single-node pub/sub (POC validated)
- Updated OQ-17, OQ-18, OQ-19 for OperationSpec repository surface
- Added OQ-30 (composite event target), OQ-31 (consumer naming), OQ-32 (Drizzle Kit)
- POC results: adapter buildable, same-process pub/sub works, transactional
  outbox semantics confirmed, concurrent listeners/streams work
- Research doc at docs/research/pivot-honker-sqlite-adapter.md
This commit is contained in:
glm-5.1
2026-06-01 16:31:40 +00:00
parent 6aa2fcc6ff
commit 412ad98f11
10 changed files with 1342 additions and 230 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
AGENTS.md
View File

@@ -28,10 +28,13 @@ infrastructure from org-scoped graph data.
│ │ └── mod.ts # Re-exports all graphs exports
│ └── sqlite/ # SQLite host (drizzle-orm + honker-node)
│ ├── tables/
│ │ ├── common.ts # shared column definitions
│ │ ├── identity/ # accounts, organizations, org_members, api_keys, audit_logs
│ │ └── metagraph/ # graph_types, node_types, edge_types, graphs, nodes, edges
│ ├── utils/ # createSelectSchema, createInsertSchema, column mappings (ADR-046)
│ ├── relations.ts # Drizzle relations
│ ├── adapter.ts # Drizzle-Honker session adapter
│ ├── event-target.ts # HonkerEventTarget (pubsub TypedEventTarget on Honker)
│ ├── schema.ts # Re-exports
│ └── client.ts # createSystemDatabase(), createTenantDatabase()
└── test/
@@ -42,7 +45,7 @@ infrastructure from org-scoped graph data.
- `@alkdev/storage` → Metagraph Module, graph type definitions (zero deps)
- `@alkdev/storage/sqlite` → SQLite tables (metagraph + identity), relations,
client, Honker adapter (drizzle-orm + honker-node)
client, adapter, event-target, utils (drizzle-orm + honker-node, peer: @alkdev/pubsub)
PostgreSQL has been removed (ADR-038). SQLite via Honker is the sole database host.
@@ -60,12 +63,20 @@ PostgreSQL has been removed (ADR-038). SQLite via Honker is the sole database ho
4. **Injectable clients**: `createSystemDatabase(client)` and
`createTenantDatabase(client)` take pre-created Honker clients, not
env vars. Module-level side effects are forbidden.
5. **Dependencies**: `@alkdev/typebox` and `@alkdev/drizzlebox` are npm deps
(not yet on JSR). This works fine — JSR handles npm dependencies natively.
5. **Dependencies**: `@alkdev/typebox` is an npm dep (not yet on JSR).
`@alkdev/pubsub` is a peer dep (for `TypedEventTarget` type). The
`@alkdev/drizzlebox` external dep has been folded into `src/sqlite/utils/`
(ADR-046).
6. **SQLite-only via Honker**: No PostgreSQL. Honker provides DB + pub/sub +
queues in a single SQLite file (ADR-038, ADR-039).
7. **System/tenant DB split**: Identity tables in `system.db`, graph data in
`tenant-{orgId}.db` (ADR-040).
8. **OperationSpecs as repository surface**: Storage outputs `OperationSpec[]`
from table definitions. No hand-written CRUD. The consumer (hub/spoke)
registers handlers (ADR-048).
9. **HonkerEventTarget**: Bridges pubsub `TypedEventTarget` to Honker
`notify`/`listen` and `stream`/`subscribe`. Enables single-node pub/sub
without Redis. Transactional outbox semantics confirmed via POC (ADR-047).
## Commands
@@ -91,7 +102,7 @@ The codebase has diverged significantly from the originals:
Modules to DB row values
- Crypto utility ported from `@alkdev/hub/src/crypto/mod.ts` with `EncryptedDataSchema`
as a TypeBox schema
- `@sinclair/typebox` → `@alkdev/typebox`, `drizzle-typebox` → `@alkdev/drizzlebox`
- `@sinclair/typebox` → `@alkdev/typebox`, `drizzle-typebox` → `@alkdev/drizzlebox` → folded into `src/sqlite/utils/` (ADR-046)
- TypeScript enums replaced with `as const` objects (`GRAPH_STATUS`, `ACTOR_TYPE`)
- `Type.Unknown()` used for unvalidated fields (not `Type.Any()`)
- Injectable client pattern (`createSqliteDatabase(client)` takes a pre-created client)
@@ -104,7 +115,7 @@ The codebase has diverged significantly from the originals:
- Entry points are `mod.ts` files that re-export from subdirectories
- TypeBox schemas are named with PascalCase (`NodeType`, `GraphConfig`)
- Drizzle table objects are named with camelCase (`graphTypes`, `nodeTypes`)
- Schema objects from drizzlebox are named with PascalCase (`InsertGraph`,
- Schema objects from utils are named with PascalCase (`InsertGraph`,
`SelectGraph`)
- Enum constants use `SCREAMING_SNAKE_CASE` objects (`GRAPH_STATUS`,
`ACTOR_TYPE`)
@@ -129,11 +140,15 @@ are not yet implemented.
## What's Not Done Yet
- Drizzle-Honker session adapter (`src/sqlite/adapter.ts`)
- Fold dbtype Phase 0 → `src/sqlite/utils/` (ADR-046, import path changes)
- Drizzle-Honker session adapter (`src/sqlite/adapter.ts`, POC validated)
- HonkerEventTarget (`src/sqlite/event-target.ts`, POC validated, ADR-047)
- Identity tables in `src/sqlite/tables/identity/` (accounts, organizations, etc.)
- Scoping columns on `graphs` table (`ownerId`, `projectId`)
- Graph type `scope` column on `graph_types` table
- Remove `actors` table and `src/pg/` directory
- Split client factory into `createSystemDatabase()` / `createTenantDatabase()`
- Repository/CRUD layer (typed query functions beyond table definitions)
- Update client factory to `createSystemDatabase()` / `createTenantDatabase()` (accept Honker client)
- Table restructure into subdirectories (metagraph/, identity/)
- OperationSpec generation from tables (ADR-048)
- ACL graph type (ADR-034)
- JSR publication setup (need to create scope/package on jsr.io first)

53
docs/architecture/decisions/046-fold-drizzlebox-as-utils.md Normal file
View File

@@ -0,0 +1,53 @@
---
status: accepted
date: 2026-06-01
supersedes: ADR-005, ADR-018
---
# ADR-046: Fold @alkdev/drizzlebox as src/sqlite/utils/
## Context
`@alkdev/drizzlebox` (a fork of `drizzle-typebox` adapted for
`@alkdev/typebox`) provides `createSelectSchema` and `createInsertSchema`
functions that derive TypeBox validation schemas from Drizzle table
definitions. It is consumed as an external npm dependency by all SQLite
table files in `@alkdev/storage`.
The parent project `@alkdev/dbtype` also planned a Phase 1 (UJSX→HostConfig
→Drizzle pipeline) that was never implemented. With SQLite as the sole
database target (ADR-038), the multi-dialect column mappings in dbtype
(PG, MySQL, SingleStore) are dead weight for storage.
ADR-018 deferred dbtype integration to post-v1. But the fold of just the
Phase 0 subset (column→TypeBox mappings + schema generation) is a
straightforward import path change with no behavioral difference.
## Decision
Fold the SQLite-only subset of `@alkdev/dbtype` Phase 0 into
`src/sqlite/utils/`:
| Source (dbtype) | Target (storage) | Changes |
|-----------------|-------------------|---------|
| `schema.ts` | `utils/schema.ts` | Remove PgEnum handling. Keep createSelectSchema, createInsertSchema, createUpdateSchema. |
| `column.ts` | `utils/column.ts` | Strip PG, MySQL, SingleStore branches. Keep SQLiteInteger, SQLiteReal, SQLiteText + generic dataType dispatch. |
| `schema.types.ts` + `schema.types.internal.ts` + `column.types.ts` | `utils/types.ts` | Merged. Remove PgEnum overloads. |
| `constants.ts` | `utils/constants.ts` | Keep as-is. |
| `utils.ts` | `utils/utils.ts` | Remove PgEnum alias. Keep isColumnType, isWithEnum, JsonSchema, BufferSchema. |
All table files change their import from `@alkdev/drizzlebox` to `../utils/schema.ts`.
## Consequences
- **No external drizzlebox dependency** — one fewer npm package in the
dependency graph.
- **SQLite-only column mappings** — dead multi-dialect code removed. If a
new database host is added later, mappings would need to be re-added.
- **Same API surface** — `createSelectSchema`, `createInsertSchema`,
`createUpdateSchema` produce the same TypeBox schemas.
- **Co-located with tables** — utility code lives next to the tables it
derives schemas from. Easier to maintain.
- **dbtype Phase 1 (UJSX→HostConfig) not affected** — that remains a
separate architectural concern, could live in storage or as its own
package when built.

58
docs/architecture/decisions/047-honker-event-target.md Normal file
View File

@@ -0,0 +1,58 @@
---
status: accepted
date: 2026-06-01
resolves: OQ-26
---
# ADR-047: HonkerEventTarget Adapter for pubsub
## Context
`@alkdev/pubsub` defines a `TypedEventTarget` interface that all transport
adapters implement: in-process `EventTarget`, Redis, WebSocket client/server,
and Worker. This provides transport-agnostic pub/sub — consumers call
`createPubSub({ eventTarget })` without knowing the underlying transport.
Honker provides SQLite with built-in pub/sub primitives:
- `db.notify(channel, payload)` / `db.listen(channel)` — ephemeral, fire-and-forget
- `db.stream(name).publish(payload)` / `db.stream(name).subscribe(consumer)` — durable, offset-tracked
POC 2-4 (2026-06-01) validated:
- Same-process notify→listen works. ~17ms median latency.
- Multiple concurrent listeners on different channels work.
- `tx.notify()` only fires on `tx.commit()`. Rollback suppresses notification.
- `queue.enqueueTx(tx, payload)` only visible after commit. Rollback suppresses.
- Stream publish/subscribe works with consumer offset tracking.
These results confirm Honker can back the pubsub `TypedEventTarget`
interface for single-node deployments.
## Decision
Implement `HonkerEventTarget` in `src/sqlite/event-target.ts`. It adapts
`@alkdev/pubsub`'s `TypedEventTarget` to Honker primitives with two modes:
1. **Ephemeral mode**: `addEventListener` → `db.listen()`, `dispatchEvent` →
`db.notify()`. Fire-and-forget semantics, no delivery guarantee.
2. **Durable mode**: `addEventListener` → `db.stream().subscribe()`,
`dispatchEvent` → `db.stream().publish()`. Per-consumer offset tracking,
crash recovery replays from last saved offset.
`@alkdev/pubsub` is a peer dependency (needed only when using
HonkerEventTarget). The `graphs/` module remains zero-dep.
## Consequences
- **Single-node hub can use Honker instead of Redis** — no separate Redis
deployment needed for pub/sub.
- **Transactional outbox semantics** — `dispatchEvent` inside a Drizzle
transaction (via `tx.notify()` or `stream.publishTx()`) commits atomically
with data writes. No dual-write problem.
- **Hub-spoke symmetry** — both hub and spoke use the same `createPubSub()`
call. Different event target instances determine routing.
- **Multi-node still needs Redis or WebSocket** — Honker events don't cross
process boundaries. For multi-node, the `WebSocketServerEventTarget` (hub)
and `WebSocketClientEventTarget` (spoke) handle cross-process routing.
- **Latency trade-off** — ~17ms Honker round-trip vs sub-ms in-process. For
hot-path call protocol, pair with in-process `EventTarget`. Design of a
composite event target is an open question (OQ-30).

74
docs/architecture/decisions/048-operation-specs-as-repo-surface.md Normal file
View File

@@ -0,0 +1,74 @@
---
status: accepted
date: 2026-06-01
supersedes: ADR-033 (partially — OQ-17, OQ-18, OQ-19 updated)
---
# ADR-048: OperationSpecs as Repository Surface
## Context
ADR-033 established JSON path queries with hand-written CRUD for v1. The
question was whether the repository layer would be hand-written query
functions, auto-generated from table definitions, or something else.
The `@alkdev/operations` package provides `OperationSpec` — a serializable
descriptor for a query, mutation, or subscription operation. Hubs and spokes
register specs in an `OperationRegistry` along with handlers. The operations
runtime handles execution, call protocol routing, subscriptions, access
control, and validation.
This presents a natural alternative to hand-written CRUD: storage outputs
`OperationSpec[]` describing CRUD operations per table, and the consumer
wires them into the operations registry.
## Decision
Storage does not ship a "repository layer" of hand-written query functions.
Instead, it outputs `OperationSpec[]` per table — flat arrays describing
CRUD operations. The consumer (hub/spoke) imports these specs, registers
handlers, and the operations runtime handles execution.
```ts
// Storage defines table + operation contracts
export const graphTypes = sqliteTable("graph_types", { ... });
export const graphTypeSpecs: OperationSpec[] = [
{ name: "create", namespace: "graph_types", type: "mutation", ... },
{ name: "find", namespace: "graph_types", type: "query", ... },
{ name: "list", namespace: "graph_types", type: "query", ... },
{ name: "update", namespace: "graph_types", type: "mutation", ... },
{ name: "delete", namespace: "graph_types", type: "mutation", ... },
];
// Hub registers specs + handlers
for (const spec of graphTypeSpecs) {
registry.registerSpec(spec);
registry.registerHandler(`${spec.namespace}.${spec.name}`, handler);
}
```
The handler is consumer-provided. Storage defines the contract (input/output
schemas, operation type); the hub provides the execution (Drizzle queries,
Honker transactions, notifications).
`@alkdev/operations` is a type-only peer dependency of storage. No circular
dependency.
## Consequences
- **No hand-written repository functions** — storage avoids the maintenance
burden of typed CRUD code. The contract is the OperationSpec.
- **Consumer owns execution** — the hub decides how to handle each operation
(raw Drizzle query, with Honker notification, with access control, etc.).
Storage doesn't execute queries.
- **Subscriptions for free** — OperationSpec supports `type: "subscription"`.
Table-level change streams become operation subscriptions rather than
custom event handling.
- **Clean dependency graph** — storage depends on operations only for the
`OperationSpec` type (peer dep). No runtime dependency.
- **Attribute queries remain JSON path** — for metagraph tables where
attributes are dynamic JSON, `json_extract()` is still the query mechanism.
Domain-specific tables with native columns produce simpler specs.
- **Supersedes ADR-033 partially** — the "hand-written CRUD" part is replaced.
The "JSON path queries for attributes" part still applies to metagraph
tables.

283
docs/architecture/forward-look.md
View File

@@ -1,16 +1,15 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Forward Look: Pointers, dbtype, and Universal IR
How the Module-based metagraph connects to the broader @alkdev ecosystem —
typed graph pointers, dbtype table rendering, and the ujsx universal IR
pipeline. These are forward-looking designs that justify why certain structural
decisions were made now
(pointer abstraction deferred per [ADR-017](./decisions/017-pointer-abstraction-is-forward-looking.md),
dbtype integration deferred per [ADR-018](./decisions/018-dbtype-integration-is-post-v1.md)).
typed graph pointers, local utils (folded from dbtype), and the ujsx universal IR
pipeline. The dbtype integration is no longer deferred (ADR-046) — the SQLite-only
Phase 0 subset folds into `src/sqlite/utils/`. The repository surface is now
OperationSpecs (ADR-048), not hand-written CRUD.
## Overview
@@ -109,68 +108,69 @@ feasible because it provides the schema the pointer validates against.
## Relationship to @alkdev/dbtype
`@alkdev/dbtype` defines database schemas as ujsx element trees and renders them
to Drizzle dialects via HostConfig. Storage's SQLite/PG table definitions are a
natural consumer of this pipeline.
`@alkdev/dbtype` defined database schemas as ujsx element trees and planned to
render them to Drizzle dialects via HostConfig. Its Phase 0 (Drizzle→TypeBox
schema generation) was consumed as `@alkdev/drizzlebox`. Phase 1 (UJSX→Drizzle)
was never implemented.
### Current vs. Future Table Definition
### Fold: Phase 0 → `src/sqlite/utils/` (ADR-046)
**Current** (manual Drizzle table defs):
With SQLite as the sole target (ADR-038), the multi-dialect column mappings in
dbtype are dead weight. The SQLite-only subset has been folded into storage as
`src/sqlite/utils/`:
| What folds in | Source (dbtype) | Target (storage) |
|---------------|-----------------|-------------------|
| Schema generation | `schema.ts` | `utils/schema.ts` |
| Column→TypeBox mappings | `column.ts` (SQLite branches only) | `utils/column.ts` |
| Type interfaces | `schema.types.ts` + `schema.types.internal.ts` + `column.types.ts` | `utils/types.ts` |
| Integer constants | `constants.ts` | `utils/constants.ts` |
| Type guards | `utils.ts` (minus PgEnum) | `utils/utils.ts` |
What does NOT fold in: PG, MySQL, SingleStore column handlers; `isPgEnum` /
`handleEnum`; `createSchemaFactory`; the Phase 1 UJSX→HostConfig pipeline.
Import changes in table files:
```ts
export const graphTypes = sqliteTable("graph_types", {
id: text("id").primaryKey(),
name: text("name").notNull(),
config: text("config", { mode: "json" }).notNull(),
// ...
});
// Before
import { createInsertSchema, createSelectSchema } from "@alkdev/drizzlebox";
// After
import { createInsertSchema, createSelectSchema } from "../utils/schema.ts";
```
**Future** (dbtype element tree → HostConfig rendering):
The API surface is identical — same functions, same TypeBox schemas produced.
```tsx
const GraphTypesEl = h("table", { name: "graph_types" },
h(IdColumn, {}),
h("column", { name: "name", type: "string", notNull: true }),
h("column", { name: "config", type: "json", mode: "json", notNull: true }),
h(AuditColumns, {}),
);
### Phase 1 (UJSX→Drizzle): future path
const root = createRoot(sqliteHost, {});
root.render(GraphTypesEl);
const drizzleTable = root.ctx.tables.graph_types;
```
The broader UJSX→HostConfig→Drizzle pipeline from dbtype's architecture remains
architecturally sound but is not part of this pivot. When and if it's built, it
could live in storage as a `HostConfig` sub-module rather than a separate
package, since storage is the primary consumer. The TypeBox Module format used
by the metagraph is already compatible with what a ujsx HostConfig would produce.
### Why this matters for storage
1. **Single source of truth**: Today's `sqlite/tables/` and future `pg/tables/`
define the same shapes in two different Drizzle dialects. dbtype renders the
same element tree to both — no manual duplication.
2. **Schema extraction**: `extractTable()` produces both TypeBox schemas (for
validation) and column metadata (for Drizzle rendering) from the same tree.
Storage gets `SelectGraphType` and `InsertGraphType` schemas for free.
3. **Module alignment**: dbtype assembles extracted schemas into a
`Type.Module` for cross-table references. Storage's metagraph Module and
dbtype's table Module could share a namespace — the `graph_types.config`
column stores the JSON Schema from `Metagraph.Config`.
1. **Single source of truth**: The `utils/` code derives TypeBox schemas from
Drizzle tables. Table definitions are the source of truth for both the DB
schema and the validation schema.
2. **Schema extraction**: `createSelectSchema` / `createInsertSchema` produce
TypeBox schemas that validate data at the application layer.
3. **Module alignment**: The metagraph Module and the table-derived schemas
share the same TypeBox namespace. `graph_types.config` stores the JSON
Schema from `Metagraph.Config`.
### v1 approach
For v1, storage continues with manual Drizzle table definitions. The dbtype
integration is deferred because:
For v1, storage uses the folded utils for TypeBox schema derivation from Drizzle
tables (what was `@alkdev/drizzlebox`). The metagraph Module independently
validates graph type definitions. These two schema sources serve different
purposes: table schemas validate DB row shapes, Module schemas validate graph
type semantics.
- dbtype is Phase 0 (architecture complete, no implementation)
- The manual defs work and are well-understood
- The Module pattern for graph types can be adopted independently (no dbtype
dependency)
- With PostgreSQL removed (ADR-038), the original pressure for dbtype —
eliminating dual SQLite/PG table maintenance — is significantly reduced.
There is now only one set of table definitions to maintain.
When dbtype reaches Phase 1 (implementation), storage can migrate from
Drizzle table definitions to dbtype elements one table at a time. The Module-based
graph type definitions are already compatible — they're both TypeBox `Type.Module`
objects.
When dbtype's Phase 1 (UJSX→HostConfig) is implemented, it would unify both
directions — a TypeBox Module could produce both the Drizzle table definition
and the validation schemas from the same element tree.
## ujsx as Universal IR
@@ -219,10 +219,17 @@ Rendered to different hosts:
| Reference graph type Modules (CallGraph, SecretGraph) | ✅ Implemented |
| Crypto utility (`encrypt`, `decrypt`, `generateEncryptionKey`, `EncryptedDataSchema`) | ✅ Implemented |
| Codegen from TypeScript interfaces → Module entries | ✅ TsToModule exists |
| dbtype element trees → Drizzle tables | ⚠️ dbtype Phase 0, no implementation |
| SQLite column→TypeBox mappings (folded from dbtype) | ✅ Folded into `src/sqlite/utils/` (ADR-046) |
| `createSelectSchema` / `createInsertSchema` (folded from drizzlebox) | ✅ Folded into `src/sqlite/utils/` (ADR-046) |
| Drizzle-Honker session adapter | ✅ POC validated, implementation pending |
| HonkerEventTarget for pubsub | ✅ POC validated, implementation pending |
| Transactional notify + outbox (Honker) | ✅ POC validated — atomic commit for data + events + queue |
| OperationSpec generation from tables | ⚠️ Design complete (ADR-048), implementation pending |
| Domain-specific native-column tables | ⚠️ Conceptual — for known graph types (CallGraph, etc.) |
| `<graphSchema>` ujsx elements | ⚠️ Conceptual — needs HostConfig design |
| Typed graph pointers via JPATH | ⚠️ Conceptual — needs JPATH Module design |
| Reactive graph observation via ValuePointer | ⚠️ Conceptual — needs signal integration |
| dbtype Phase 1 (UJSX→Drizzle HostConfig) | ⚠️ Architecture exists, not implemented. Could live in storage if built. |
The Module-based graph type definitions (this spec) are the **first concrete
step** in this pipeline. Everything else builds on having a `Type.Module` as
@@ -230,126 +237,66 @@ the schema source of truth.
## Repository Layer Strategy
The repository layer (typed CRUD for the 6 metagraph tables + queries for graph data)
is the next major feature to implement. The question of *how* it queries attributes
connects to broader ecosystem decisions about dbtype and operations.
The repository layer (typed CRUD for the 6 metagraph tables + identity tables +
queries for graph data) is now defined as **OperationSpec output** rather than
hand-written query functions (ADR-048).
### Three Approaches
### OperationSpecs as Repository Surface
#### A. JSON Path Queries (Near-Term)
Storage outputs `OperationSpec[]` per table — flat arrays describing CRUD
operations. The consumer (hub/spoke) imports these, registers handlers, and
the operations runtime handles execution, call protocol, and subscriptions.
The repository layer maps filter criteria to JSON path extraction:
```ts
// Storage defines the table + operation contracts
export const callNodes = sqliteTable("call_nodes", { ... });
export const callNodeSpecs: OperationSpec[] = [
{ name: "create", namespace: "call_nodes", type: "mutation", inputSchema: ..., outputSchema: ... },
{ name: "find", namespace: "call_nodes", type: "query", ... },
{ name: "list", namespace: "call_nodes", type: "query", ... },
{ name: "update", namespace: "call_nodes", type: "mutation", ... },
{ name: "delete", namespace: "call_nodes", type: "mutation", ... },
];
// Hub registers specs + handlers
for (const spec of callNodeSpecs) {
registry.registerSpec(spec);
registry.registerHandler(`${spec.namespace}.${spec.name}`, handler);
}
```
The handler is consumer-provided — not in storage. Storage doesn't execute
queries. Storage defines the contract; the hub provides the execution layer.
### Attribute Queries
The metagraph's `attributes` column remains JSON — node types are dynamic
schemas defined at runtime, not static columns. Attribute queries use
`json_extract()` for v1:
```ts
findNodes({ graphId, attributes: { status: "active" } })
// SQLite: json_extract(attributes, '$.status') = 'active'
// PG: attributes ->> 'status' = 'active'
```
- Works with current table definitions (no schema changes)
- SQLite `json_extract()` and PG `->>` / `#>>` operators handle JSON path
- No native index support on individual JSON attributes
- PG can add GIN indexes on `jsonb` columns for containment queries, but not for
arbitrary key-value lookups
- Simple, immediate, no new infrastructure
For known graph types (CallGraph, SecretGraph), domain-specific tables with
native columns can complement the generic metagraph tables. These domain
tables also produce OperationSpecs with native-column queries.
This is the pragmatic v1 approach. The metagraph pattern *requires* JSON attributes
because node types are dynamic schemas (defined at runtime, stored in
`node_types.schema`), not static columns known at database definition time.
### Connection to @alkdev/operations
#### B. Native Columns via dbtype (Long-Term, Speculative)
If storage migrates to dbtype element trees for table definitions, the 6 static
metagraph tables (graph_types, node_types, edge_types, graphs, nodes, edges) could
be rendered via the dbtype pipeline: element tree → HostConfig → Drizzle tables.
This would eliminate the manual duplication between `sqlite/` and future `pg/`.
However, dbtype does NOT solve the attribute indexing problem:
- The metagraph's `attributes` column MUST remain JSON because the shape is defined
by runtime schemas (node type definitions), not by static column definitions
- dbtype generates static table schemas; it does not handle dynamic schema-as-data
patterns like the metagraph
- A "call" node's attributes (`requestId`, `status`, `duration`) are not columns
on the `nodes` table — they're values in the `attributes` JSON column, validated
by the corresponding node type's TypeBox schema
#### C. Hybrid: Static Tables via dbtype, Dynamic Attributes Remain JSON
The hybrid approach preserves the metagraph's dynamic schema model while leveraging
dbtype for the static table scaffolding:
1. **Static tables**: dbtype renders the 6 metagraph tables to Drizzle dialects.
This eliminates the SQLite/PG manual duplication for table *structure*.
The `attributes` column is still `text/jsonb` across both dialects.
2. **Dynamic attributes**: Remain JSON. The Module-based node type schemas validate
data at the application layer, not the database layer. This is by design
(ADR-003, ADR-014).
3. **Virtual columns / computed columns**: A post-v1 optimization, not a v1 concern.
Frequently queried attributes could be extracted to indexed columns as a
performance optimization. For example, if `nodes.attributes.status` is a common
filter, a computed column or trigger could copy it to `nodes.status_column` with
an index. This would be a denormalization trade-off (triggers, migration
complexity, dual-write responsibility) and is not designed or planned for v1.
4. **Repository CRUD**: The static table CRUD operations (insert graph type, find
node by key) could be auto-generated like drizzle-graphql or the dbtype
`from-dbtype` adapter. Graph-specific attribute queries remain JSON path.
### Implications for Each Approach
| Concern | Path A (JSON) | Path B (Native) | Path C (Hybrid) |
|---------|---------------|-----------------|------------------|
| Works today | ✅ | ❌ (requires dbtype) | ❌ (requires dbtype) |
| Preserves metagraph pattern | ✅ | ❌ (conflicts with dynamic schemas) | ✅ |
| Eliminates SQLite/PG duplication | ❌ | ✅ | ✅ |
| Indexes on attributes | GIN on PG only | ✅ full native | GIN + virtual columns |
| Repository generation | Hand-write CRUD | Auto-gen from dbtype | Auto-gen for static, JSON path for dynamic |
| Dependency on dbtype | None | Full | Partial (static tables only) |
### Connection to drizzle-graphql
The overview references drizzle-graphql as a pattern for auto-generating a CRUD/query
surface. The dbtype `from-dbtype` adapter is the @alkdev equivalent: it consumes
element trees + Type.Module bundles and produces `OperationSpec[]` for the
operations registry.
The parallel:
| Concern | drizzle-graphql | dbtype from-dbtype |
|---------|----------------|-------------------|
| Input | Drizzle schema (tables + relations) | UJSX element tree + Type.Module |
| Output | GraphQL schema (queries + mutations) | `OperationSpec[]` (CRUD operations) |
| Dialects | SQLite, PG, MySQL | SQLite, PG, MySQL (via HostConfig) |
| Table model | Static columns only | Static columns only |
| Dynamic data (JSON attrs) | Not handled | Not handled |
Neither drizzle-graphql nor dbtype's `from-dbtype` handles dynamic schema-as-data
patterns. The metagraph's JSON attributes require their own query layer, regardless
of whether the static tables are auto-generated. This means the repository layer
for `@alkdev/storage` will always have two parts:
1. **Static table CRUD** — could be auto-generated (by dbtype or hand-written)
2. **Graph data queries** — JSON path queries against the `attributes` column,
validated by the Module schema at the application layer
`@alkdev/operations` is a type-only peer dependency of storage. The
`OperationSpec` type is straightforward. Storage builds the specs; the
consumer wires them into the registry. No circular dependency.
### v1 Decision
For v1, the practical path is **A (JSON path queries) with hand-written CRUD**. This
decision is recorded as [ADR-033](./decisions/033-json-path-queries-for-v1.md). The
hybrid approach (C) remains viable for a future iteration when dbtype reaches
implementation, and it doesn't require any changes to the metagraph data model —
only to how the static table definitions are generated. See OQ-17, OQ-18, OQ-19
in [open-questions.md](./open-questions.md) for the specific long-term questions
that remain open beyond v1.
### Decisions Required
- **OQ-17**: JSON path vs native columns vs hybrid for attribute queries (resolved for v1 — see ADR-033)
- **OQ-18**: Auto-generated vs hand-written CRUD for static tables (resolved for v1 — see ADR-033)
- **OQ-19**: Where the storage-operations bridge package should live (open)
For v1, the practical path is **OperationSpecs with JSON path attribute
queries** (ADR-048, supersedes ADR-033). Spec generation from tables is
straightforward once domain tables exist. The metagraph's generic CRUD
(graphs, nodes, edges) uses JSON attributes; domain-specific CRUD uses
native columns. Both produce OperationSpecs that the hub registers in the
same operations registry.
## Constraints on Current Design
@@ -382,18 +329,28 @@ design in [metagraph-module.md](./metagraph-module.md):
opaque builder objects or Drizzle column definitions. See
[schema-evolution.md](./schema-evolution.md).
6. **OperationSpec output is consumer-agnostic** — storage defines
`OperationSpec[]` from table definitions. The consumer (hub/spoke) decides
how to register handlers. Storage does not execute queries or depend on
the operations runtime.
7. **The folded utils are SQLite-only** — `src/sqlite/utils/` contains only
SQLite column→TypeBox mappings. If a new database host is added later, the
utils would need the corresponding dialect mappings. dbtype's Phase 1
(UJSX→HostConfig) would be the mechanism for multi-dialect support.
## References
- ujsx pointer system: `/workspace/@alkdev/ujsx/src/core/pointer.ts`
- ujsx HostConfig adapter: `/workspace/@alkdev/ujsx/src/host/config.ts`
- dbtype architecture: `/workspace/@alkdev/dbtype/docs/architecture/README.md`
- dbtype architecture: `/workspace/@alkdev/dbtype/docs/architecture/README.md` (Phase 0 source folded into storage)
- dbtype elements: `/workspace/@alkdev/dbtype/docs/architecture/elements.md`
- dbtype module: `/workspace/@alkdev/dbtype/docs/architecture/module.md`
- dbtype repo adapter: `/workspace/@alkdev/dbtype/docs/architecture/repo-adapter.md`
- drizzle-graphql (reference for CRUD generation pattern): `/workspace/drizzle-graphql/`
- Operations registry: `/workspace/@alkdev/operations/docs/architecture/README.md`
- JPATH Module (JSONPath as TypeBox Module): `/workspace/research/typebox_research/ujsx/jpath.gen.ts`
- jsonpathly source: `/workspace/jsonpathly/`
- Module evolution spec: [metagraph-module.md](./metagraph-module.md)
- Schema evolution spec: [schema-evolution.md](./schema-evolution.md)
- ADR-033: JSON path queries and hand-written CRUD for v1
- ADR-046: Fold drizzlebox as utils (supersedes ADR-033)
- ADR-048: OperationSpecs as repository surface

143
docs/architecture/honker-integration.md
View File

@@ -1,12 +1,14 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Honker Integration
How @alkdev/storage integrates with Honker for SQLite database operations,
transactional pub/sub, durable event streams, and task queues.
transactional pub/sub, durable event streams, and task queues. Includes the
HonkerEventTarget adapter that bridges `@alkdev/pubsub`'s `TypedEventTarget`
to Honker primitives.
## Purpose
@@ -118,10 +120,134 @@ db.transaction((tx) => {
| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | Small Rust PR to honker-node |
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | POC 1 confirmed `tx.query("SELECT last_insert_rowid() as id")` works |
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | Stable across minor versions |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | POC 1 confirmed all classes accessible and extendable |
| `:memory:` databases don't work | Reader pool gets separate in-memory DB | Always use file-based paths. Tests use temp files. |
### POC-Validated Adapter Architecture
POC 1 (2026-06-01) confirmed the adapter is buildable. Key findings:
- `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` are all
accessible from `drizzle-orm/sqlite-core/session` and extendable.
- `BaseSQLiteDatabase` is accessible from `drizzle-orm/sqlite-core/db`.
- `LibSQLSession` in `drizzle-orm/libsql/session` is the reference
implementation to follow.
- Honker `query()` returns `{ columnName: value }` object rows, compatible
with Drizzle's `mapResultRow()`.
- `last_insert_rowid()` accessible via `tx.query("SELECT last_insert_rowid() as id")`.
- `tx.execute()` returns a number (affected rows count).
- JSON-mode columns need manual `JSON.parse()` in the adapter.
## HonkerEventTarget
The `HonkerEventTarget` adapts `@alkdev/pubsub`'s `TypedEventTarget` interface
to Honker's `notify`/`listen` (ephemeral) and `stream`/`subscribe` (durable)
primitives. It lives in `src/sqlite/event-target.ts`.
### Interface
```ts
import type { TypedEventTarget } from "@alkdev/pubsub";
interface HonkerEventTargetOptions {
db: Database;
mode: "ephemeral" | "durable";
streamName?: string;
consumerName?: string;
prefix?: string;
}
function createHonkerEventTarget<TEvent>(
options: HonkerEventTargetOptions
): TypedEventTarget<TEvent> & { close(): void };
```
### Ephemeral Mode
Maps to Honker's `notify()`/`listen()`:
| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.listen("topic:id")` → start async consumer loop |
| `dispatchEvent(event)` | `db.notify(event.type, event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the listener when no callbacks remain |
| `close()` | Close all active listeners |
POC 2 confirmed: same-process `notify`→`listen` works, multiple concurrent
listeners on different channels work, cross-channel isolation is correct.
POC 4 confirmed: `tx.notify()` within a Drizzle transaction only fires the
notification on `tx.commit()`. On `tx.rollback()`, the notification is
suppressed. This enables transactional outbox semantics for ephemeral events.
### Durable Mode
Maps to Honker's `Stream.publish()`/`Stream.subscribe()`:
| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.stream(name).subscribe(consumer)` → start async consumer loop |
| `dispatchEvent(event)` | `db.stream(name).publish(event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the subscription when no callbacks remain |
| `close()` | Close all active subscriptions, save offsets |
POC 3 confirmed: `Stream.publish()`/`Stream.subscribe()` work, consumer
offsets are tracked and persisted, `publishTx(tx, payload)` works within
transactions, and concurrent stream + listener operation works.
Durable mode provides crash recovery — consumers resume from their last saved
offset after restart. Consumer names must be stable across restarts (not
PID-based).
### Topic Routing
pubsub uses `topic:id` composite topics. Honker channels and streams are flat
strings. The mapping:
- **Ephemeral**: Each unique `topic:id` gets its own `db.listen()` call.
For high-cardinality topics (many request IDs), consider topic prefix
matching with client-side filtering instead.
- **Durable**: Streams are name-keyed, not topic-keyed. A single stream
carries all events for a domain. Client-side filtering dispatches only
to matching listeners.
Suggested split:
| Event category | Mode | Reason |
|---------------|------|--------|
| Call protocol events (`call.requested`, `call.responded`, etc.) | Durable stream | Crash recovery, audit trail, flowgraph replay |
| Cache invalidation signals | Ephemeral | Fire-and-forget, loss acceptable |
| UI/dashboard push | Ephemeral | Low latency, loss acceptable |
| Schema migration jobs | Queue (not pubsub) | At-least-once processing |
### Latency Consideration
POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.
### Hub-Spoke Event Routing
```
Hub (system.db + tenant-{orgId}.db)
├── HonkerEventTarget (durable: call-protocol stream)
├── WebSocketServerEventTarget (spoke fan-out)
└── In-process EventTarget (local subscribers)
Spoke (tenant-{orgId}.db)
├── HonkerEventTarget (ephemeral: local channels)
├── WebSocketClientEventTarget (hub connection)
└── In-process EventTarget (local subscribers)
```
Both hub and spoke use the same `createPubSub({ eventTarget })` call.
Different event target instances determine the routing. No code changes
between hub and spoke — only configuration.
## Event-Driven Patterns
@@ -338,7 +464,7 @@ would require a coordinator — the hub fills this role.
- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
- TypeBox schemas from drizzlebox
- TypeBox schemas from local utils (folded from drizzlebox)
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)
@@ -359,13 +485,18 @@ would require a coordinator — the hub fills this role.
|-----|----------|--------|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils/ | Accepted |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter for pubsub | Accepted |
## Open Questions
- **OQ-26**: Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?
- **OQ-26**: ~~Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?~~ Resolved: Yes. HonkerEventTarget (ADR-047) provides the adapter. Redis still needed for multi-node.
- **OQ-27**: How are schema migrations applied across all tenant DBs?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?
- **OQ-30**: Composite event target pattern — how should an in-process EventTarget and HonkerEventTarget be combined for single-node hub deployments? POC 2 showed ~17ms Honker latency vs sub-ms in-process. Design needed.
- **OQ-31**: Consumer naming convention for durable subscriptions — must be stable across hub restarts (not PID-based).
- **OQ-32**: Drizzle Kit migration compatibility — does `drizzle-kit push`/`drizzle-kit generate` work with the custom Honker adapter, or do we need a custom migration runner?
## References

58
docs/architecture/open-questions.md
View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Open Questions Tracker
@@ -12,8 +12,8 @@ architecture documents, organized by theme.
| Status | Count |
|--------|-------|
| Open | 9 |
| Resolved (this revision) | 15 |
| Open | 10 |
| Resolved (this revision) | 18 |
| Previously resolved | 11 |
**Open questions requiring decisions:**
@@ -23,10 +23,13 @@ architecture documents, organized by theme.
- **OQ-11** (auto-migrate vs explicit) — conditional on OQ-10
- **OQ-12** (schema evolution vs event-sourced replay) — post-v1 concern
- **OQ-13** (schema evolution events in event stream) — post-v1
- **OQ-26** (Honker replaces @alkdev/pubsub Redis transport) — integration design
- **OQ-25** (scope string semantics) — evaluator concern
- **OQ-27** (tenant DB schema migration strategy) — multi-tenant operations
- **OQ-28** (cross-tenant delegation with separate DBs) — cross-DB coordination
- **OQ-29** (standalone drizzle-honker npm package) — community value
- **OQ-30** (composite event target for single-node hub) — latency optimization
- **OQ-31** (consumer naming for durable subscriptions) — restart stability
- **OQ-32** (Drizzle Kit migration compatibility) — custom adapter
## Theme 1: Package Boundaries and Dependencies
@@ -152,21 +155,23 @@ architecture documents, organized by theme.
### OQ-17: Attribute queries — JSON path, native columns, or dbtype-generated?
- **Status**: resolved (v1)
- **Status**: resolved (updated)
- **Priority**: high
- **Resolution**: JSON path for v1. ADR-033. Long-term hybrid still open but less pressing without PG dual maintenance.
- **Resolution**: JSON path for metagraph attributes (dynamic schemas). Native columns for domain-specific tables (CallGraph, etc.). OperationSpecs provide the CRUD contract for both patterns. ADR-048 supersedes ADR-033.
- **Cross-references**: ADR-048
### OQ-18: Auto-generated vs hand-written CRUD?
- **Status**: resolved (v1)
- **Status**: resolved (updated)
- **Priority**: medium
- **Resolution**: Hand-write for v1. ADR-033.
- **Resolution**: Not hand-written CRUD — OperationSpecs. Storage outputs `OperationSpec[]` from table definitions. The consumer (hub/spoke) registers handlers. ADR-048.
- **Cross-references**: ADR-048
### OQ-19: Storage-operations bridge package location?
- **Status**: resolved
- **Status**: resolved (updated)
- **Priority**: medium
- **Resolution**: Less pressing now that Honker replaces the Redis transport. Can live in the hub for v1. Revisit if an adapter package becomes valuable.
- **Resolution**: No separate bridge package needed. Storage outputs `OperationSpec[]` as part of its table definitions (type-only peer dep on `@alkdev/operations`). The consumer wires specs into the registry. ADR-048.
## Theme 7: Access Control
@@ -220,9 +225,10 @@ architecture documents, organized by theme.
### OQ-26: Can Honker fully replace @alkdev/pubsub's Redis transport for single-node deployments?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Status**: resolved
- **Priority**: high
- **Notes**: Honker's `notify()`/`listen()` and `stream()`/`subscribe()` provide the pub/sub primitives. The question is whether `@alkdev/pubsub`'s `TypedEventTarget` interface can be backed by Honker instead of Redis, and whether multi-node deployments still need Redis for internode communication.
- **Resolution**: Yes for single-node. The `HonkerEventTarget` adapter (ADR-047) implements pubsub's `TypedEventTarget` on Honker's `notify`/`listen` and `stream`/`subscribe`. POC 2-4 validated: same-process pub/sub works, transactional semantics hold, concurrent listeners work. Redis still needed for multi-node deployments. In-process EventTarget provides sub-ms latency for hot paths (vs ~17ms for Honker round-trip).
- **Cross-references**: ADR-047
### OQ-27: How are schema migrations applied across all tenant DBs?
@@ -245,6 +251,27 @@ architecture documents, organized by theme.
- **Priority**: low
- **Notes**: The adapter is ~100 lines and useful to anyone combining Drizzle with Honker. Publishing as `drizzle-honker` would benefit the community. Decision: start inside `@alkdev/storage`, extract later if there's demand.
### OQ-30: Composite event target for single-node hub deployments?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: POC 2 showed ~17ms median latency for Honker notify→listen vs sub-ms for in-process EventTarget. For single-node hubs, a composite that dispatches to both (in-process for speed, Honker for durability/cross-process) would be the ideal default. Design needed.
### OQ-31: Consumer naming convention for durable stream subscriptions?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: Honker's `stream.subscribe(consumer)` requires a consumer name for offset tracking. The name must be stable across hub restarts (PID-based names don't survive restart). Need a convention: `{service}:{host}` or a configurable consumer group ID.
### OQ-32: Drizzle Kit migration compatibility with Honker adapter?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: Drizzle Kit supports SQLite migrations but expects `better-sqlite3` or `libsql`. Need to verify `drizzle-kit push`/`drizzle-kit generate` works with the custom Honker adapter, or whether we need a custom migration runner.
## ADR Impact
| ADR | Resolves | Informs |
@@ -255,7 +282,7 @@ architecture documents, organized by theme.
| ADR-020 | OQ-24 | |
| ADR-023 | OQ-14 | |
| ADR-026 | OQ-15 | |
| ADR-033 | OQ-04, OQ-16, OQ-17, OQ-18 | |
| ADR-033 | OQ-04, OQ-16, OQ-17, OQ-18 | OQ-17 (superseded by ADR-048) |
| ADR-034 | OQ-03, OQ-21 | OQ-25 |
| ADR-035 | OQ-03 | |
| ADR-038 | OQ-04 (moot) | OQ-17 (less pressure) |
@@ -263,5 +290,8 @@ architecture documents, organized by theme.
| ADR-041 | OQ-24 | |
| ADR-042 | | OQ-24 |
| ADR-043 | | |
| ADR-044 | OQ-19 (less pressure) | |
| ADR-044 | OQ-19 (less pressure) | OQ-29, OQ-32 |
| ADR-045 | OQ-23 | OQ-20 |
| ADR-046 | | OQ-17 |
| ADR-047 | OQ-26 | OQ-30 |
| ADR-048 | OQ-17 (updated), OQ-18 (updated), OQ-19 (updated) | |

90
docs/architecture/overview.md
View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# @alkdev/storage — Overview
@@ -24,6 +24,12 @@ organizations, api_keys, audit_logs) for multi-tenant authentication and
authorization, and integrates with **Honker** for transactional pub/sub, event
streams, and task queues within the same SQLite database.
Storage also provides a **HonkerEventTarget** adapter that bridges
`@alkdev/pubsub`'s `TypedEventTarget` interface to Honker's `notify`/`listen`
and `stream`/`subscribe` primitives, and **OperationSpec** generation from
table definitions so downstream hubs/spokes can register CRUD operations
directly into the `@alkdev/operations` registry.
## Architecture
```
@@ -45,9 +51,11 @@ streams, and task queues within the same SQLite database.
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
│ │ └── index.ts → barrel re-export
│ ├── utils/ → createSelectSchema, createInsertSchema, column mappings (folded from @alkdev/dbtype)
│ ├── relations.ts → drizzle relational mappings
│ ├── schema.ts → barrel re-export
│ ├── adapter.ts → Drizzle-Honker session adapter
│ ├── event-target.ts → HonkerEventTarget (pubsub TypedEventTarget on Honker)
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
└── test/
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
@@ -59,7 +67,7 @@ streams, and task queues within the same SQLite database.
| ------------------------ | --------------------------------------- | --------------------------------------- |
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter | `@alkdev/drizzlebox`, `drizzle-orm`, `@russellthehipp/honker-node` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter, event-target, utils | `drizzle-orm`, `@russellthehippo/honker-node`, `@alkdev/pubsub` (peer) |
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
database host.
@@ -117,8 +125,10 @@ fixed — changes require a version bump and migration (ADR-029).
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
| **HonkerEventTarget** | Adapter that implements `@alkdev/pubsub`'s `TypedEventTarget` interface on Honker's `notify`/`listen` and `stream`/`subscribe` primitives. Bridges pubsub to Honker for single-node and cross-process scenarios. |
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. |
| **Repository surface** | Not hand-written CRUD functions. Storage outputs `OperationSpec[]` — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers handlers, and the operations runtime handles execution. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the bridge functions (at mutation time), NOT in the database. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
## Design Decisions
@@ -131,7 +141,10 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via drizzlebox | Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via local utils | Drizzle tables are single source of truth; `src/sqlite/utils/` generates TypeBox schemas (folded from @alkdev/drizzlebox) |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils | SQLite-only column mappings and schema generation co-located with tables |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter | pubsub TypedEventTarget on Honker notify/listen and stream/subscribe |
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repository surface | Storage outputs OperationSpec[] not hand-written CRUD |
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
@@ -150,12 +163,14 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| Package | Purpose | Layer |
| -------------------- | ------------------------------------ | ------------------------ |
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
| `@alkdev/drizzlebox` | Generate TypeBox from Drizzle tables | sqlite/ |
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
| `@alkdev/pubsub` | `TypedEventTarget` interface (peer) | sqlite/event-target.ts |
`@alkdev/typebox` and `@alkdev/drizzlebox` are npm packages (not yet on JSR).
JSR handles npm dependencies natively.
`@alkdev/typebox` is an npm package (not yet on JSR). JSR handles npm
dependencies natively. `@alkdev/pubsub` is a peer dependency — only needed
when using `HonkerEventTarget`. The `graphs/` module has zero dependencies
beyond `@alkdev/typebox`.
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
ecosystem references describe consumer-side data shapes and integration
@@ -169,22 +184,26 @@ patterns, not import dependencies.
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Reference graph type Modules (CallGraph, SecretGraph)
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
- SQLite host: 6 metagraph tables + `actors` placeholder + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
- SQLite host: 6 metagraph tables (flat structure, pending reorganize) + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (current: `@alkdev/drizzlebox`; pending fold to `src/sqlite/utils/`, ADR-046)
- Reference module tests (bridge functions, validation, Module composition)
### Not Yet Implemented
| Gap | Priority | Notes |
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| Drizzle-Honker session adapter | High | ~100-line adapter wrapping `HonkerSQLiteSession` + `HonkerPreparedQuery`. ADR-044. |
| Drizzle-Honker session adapter | High | `HonkerSQLiteSession` + `HonkerPreparedQuery` + `HonkerSQLiteTransaction`. POC validated. ADR-044. |
| Fold dbtype Phase 0 → `src/sqlite/utils/` | High | SQLite-only column mappings + createSelectSchema/createInsertSchema. ADR-046. |
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
| Graph type `scope` column | High | ADR-043. |
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. |
| Repository/CRUD layer | Medium | ⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges. |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. Accept Honker client. |
| Table restructure into subdirectories | High | `tables/metagraph/`, `tables/identity/`, update relative imports. |
| HonkerEventTarget adapter | High | pubsub `TypedEventTarget` on Honker primitives. POC validated. ADR-047. |
| OperationSpec generation from tables | Medium | `OperationSpec[]` per table for CRUD operations. ADR-048. |
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
| Domain-specific native-column tables | Low | CallGraph, SecretGraph, etc. with native columns alongside JSON attributes. |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
@@ -207,7 +226,8 @@ other way.
(depends on: @alkdev/typebox)
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
(depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)
(depends on: @alkdev/typebox, drizzle-orm, honker-node)
(peer dep: @alkdev/pubsub for TypedEventTarget type)
↑ ↑
| |
@@ -218,7 +238,9 @@ Hub / Spoke Any consumer that needs
### Event-Driven Architecture with Honker
The @alkdev platform is event-driven. Honker provides the transport mechanism
within each SQLite database:
within each SQLite database. Storage provides the `HonkerEventTarget` adapter
so consumers can use `@alkdev/pubsub`'s `TypedEventTarget` interface regardless
of whether events route in-process, through Honker, or over WebSocket.
| Concern | Honker Primitive | Example |
|---------|------------------|---------|
@@ -230,7 +252,17 @@ within each SQLite database:
A single Drizzle transaction via the Honker adapter can insert graph data AND
publish a notification AND enqueue a side-effect job — all committing atomically.
No dual-write problem between data and events.
No dual-write problem between data and events. POC 4 confirmed: `tx.notify()`
only fires on commit; on rollback, both the data write and the notification are
suppressed.
### Latency Consideration
POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.
### What Comes from Where
@@ -241,26 +273,32 @@ No dual-write problem between data and events.
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
| Event transport | Honker (within storage) | Replaces `@alkdev/pubsub`'s Redis transport for single-node deployments |
| Event transport | Honker (within storage) | `HonkerEventTarget` bridges pubsub `TypedEventTarget` to Honker primitives |
| Pubsub interface | `@alkdev/pubsub` | Storage provides `HonkerEventTarget` adapter; does not replace pubsub |
### Repository Layer Bridging Pattern
### Repository Surface as OperationSpecs
The repository layer in `@alkdev/storage` provides typed CRUD — no
`@alkdev/operations` dependency. A consumer-side bridging module wires CRUD
functions into the operations registry when needed.
Storage does not ship a "repository layer" of hand-written CRUD functions.
Instead, it outputs `OperationSpec[]` — flat arrays describing CRUD and query
operations for each table. The consumer (hub/spoke) imports these specs,
registers them in the `@alkdev/operations` registry along with handlers, and
the operations runtime handles execution, call protocol, and subscriptions.
Storage depends on `@alkdev/operations` only as a type-level peer dependency
(for the `OperationSpec` type). No circular dependency.
```
@alkdev/storage → defines types + tables + CRUD (no operations dependency)
@alkdev/storage → defines types + tables + OperationSpec[] (type-only operations dep)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas
Consumer (hub / spoke) → imports both, registers specs + handlers
```
### Avoiding Circular Dependencies
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
other directly. Storage defines the schema types and database tables; operations
defines the call protocol and execution model. The consumer imports both and
bridges them.
other at runtime. Storage defines the schema types, database tables, and
operation contract definitions; operations defines the call protocol and
execution model. The consumer imports both and bridges them.
## Open Questions
@@ -270,7 +308,7 @@ questions affecting this package:
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (open)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (resolved: yes — HonkerEventTarget adapter, POC validated. Redis still needed for multi-node. See ADR-047.)
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)

27
docs/architecture/sqlite-host.md
View File

@@ -1,13 +1,14 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# SQLite Host
The SQLite database host for `@alkdev/storage`. Uses Drizzle ORM with Honker
for database operations, pub/sub, event streams, and task queues. TypeBox
schemas are auto-generated from Drizzle table definitions via `@alkdev/drizzlebox`.
schemas are generated from Drizzle table definitions via `src/sqlite/utils/`
(folded from `@alkdev/dbtype`/`@alkdev/drizzlebox`, ADR-046).
## Overview
@@ -18,11 +19,13 @@ The SQLite host provides:
2. **Identity tables** — accounts, organizations, organization_members, api_keys,
audit_logs for multi-tenant authentication and authorization (ADR-041)
3. **Drizzle relations** for the relational query API
4. **TypeBox schemas** auto-generated from Drizzle tables (select/insert
validation) via `@alkdev/drizzlebox`
4. **TypeBox schemas** generated from Drizzle tables (select/insert
validation) via `src/sqlite/utils/` (folded from @alkdev/drizzlebox, ADR-046)
5. **Drizzle-Honker adapter** — thin session adapter for Honker integration
(ADR-044)
6. **Client factories** — `createSystemDatabase(client)` and
(ADR-044, POC validated)
6. **HonkerEventTarget** — pubsub `TypedEventTarget` on Honker primitives
(ADR-047, POC validated)
7. **Client factories** — `createSystemDatabase(client)` and
`createTenantDatabase(client)` for the system/tenant DB model (ADR-040)
## Package Structure
@@ -47,8 +50,15 @@ src/sqlite/
│ │ ├── edges.ts # edges table + select/insert schemas
│ │ └── index.ts # barrel re-export
│ └── index.ts # barrel re-export
├── utils/ # folded from @alkdev/dbtype Phase 0 (ADR-046)
│ ├── schema.ts # createSelectSchema, createInsertSchema, createUpdateSchema
│ ├── column.ts # Column→TypeBox mappings (SQLite-only dispatch)
│ ├── types.ts # Public + internal TypeScript interfaces
│ ├── constants.ts # Integer range constants
│ └── utils.ts # isColumnType, isWithEnum, type helpers
├── relations.ts # Drizzle relational mappings
├── adapter.ts # Drizzle-Honker session adapter
├── event-target.ts # HonkerEventTarget (pubsub TypedEventTarget on Honker)
├── schema.ts # re-exports all tables + relations
└── client.ts # createSystemDatabase(), createTenantDatabase()
```
@@ -339,8 +349,11 @@ db.transaction((tx) => {
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage | accounts, organizations, api_keys, audit_logs |
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graphs | `ownerId`, `projectId` on `graphs` table |
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope on `graph_types` |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker adapter | ~100-line session adapter |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker adapter | ~100-line session adapter, POC validated |
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | org_members authoritative | SQL table is source of truth; BelongsToEdge is derived |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold drizzlebox as utils | SQLite-only column mappings in src/sqlite/utils/ |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget | pubsub TypedEventTarget on Honker |
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repo surface | Table-defined operation contracts |
| [019](decisions/019-json-text-for-schema-columns.md) | JSON text for schema columns | SQLite uses `text` with JSON mode |
| [020](decisions/020-no-nodetypeid-on-nodes.md) | No nodeTypeId on nodes | Node type enforced at application layer |
| [022](decisions/022-composite-fks-for-node-references.md) | Composite FKs for node refs | Edges reference `(graphId, sourceNodeKey)` |

743
docs/research/pivot-honker-sqlite-adapter.md Normal file
View File

@@ -0,0 +1,743 @@
---
status: draft
created: 2026-06-01
last_updated: 2026-06-01
---
# Pivot: Honker SQLite Adapter, dbtype Fold, and Event Target Integration
Research document for the architecture pivot from `@libsql/client` to Honker,
folding `@alkdev/dbtype` Phase 0 into `src/utils/`, and adding a pubsub
`HonkerEventTarget`.
## Motivation
1. Honker provides SQLite + pub/sub + streams + queues + locks + scheduler in
a single `.db` file via a Rust NAPI binding. This replaces both
`@libsql/client` (SQLite only) and the conceptual need for Redis.
2. `@alkdev/dbtype` Phase 0 is already consumed as `@alkdev/drizzlebox`. With
SQLite as the sole target, the multi-dialect column mappings are dead
weight. Folding the SQLite-relevant subset into `src/utils/` eliminates an
external dependency and co-locates utility code with the tables it
derives schemas from.
3. `@alkdev/pubsub` defines a `TypedEventTarget` interface that Honker's
primitives (`notify`/`listen`, `stream`/`stream.subscribe`) map to
naturally. A `HonkerEventTarget` adapter lets storage's downstream
consumers use the same pubsub API regardless of whether events route
in-process, through Honker, through Redis, or over WebSocket.
4. The hub-spoke model needs events for call protocol routing, cache
invalidation, flowgraph reactive updates, and cross-process coordination.
Honker handles all of these within SQLite.
## Architecture Changes
### Current Structure
```
src/
├── graphs/ # Metagraph Module + bridge (no db deps)
│ ├── modules/ # CallGraph, SecretGraph, Metagraph
│ ├── bridge.ts # moduleToDbSchema, validateNode, validateEdge
│ ├── crypto.ts # encrypt, decrypt, generateEncryptionKey
│ └── mod.ts
├── pg/ # Placeholder (to be removed)
└── sqlite/ # SQLite host
├── client.ts # createSqliteDatabase(@libsql/client)
├── mod.ts
├── relations.ts
├── schema.ts
└── tables/ # Flat: graphTypes, nodeTypes, edgeTypes, graphs, nodes, edges, actors
```
Dependencies: `@alkdev/drizzlebox` (external npm), `@libsql/client`, `drizzle-orm`
### Target Structure
```
src/
├── graphs/ # Unchanged
│ ├── modules/
│ ├── bridge.ts
│ ├── crypto.ts
│ └── mod.ts
└── sqlite/
├── adapter.ts # NEW: Drizzle-Honker session adapter
├── event-target.ts # NEW: HonkerEventTarget implements pubsub TypedEventTarget
├── client.ts # UPDATED: createSystemDatabase(client), createTenantDatabase(client)
├── mod.ts # UPDATED: re-exports
├── relations.ts # UPDATED: remove actors relations
├── schema.ts # UPDATED: remove actors, remove pg refs
├── utils/ # NEW: folded from @alkdev/dbtype Phase 0 (SQLite only)
│ ├── schema.ts # createSelectSchema, createInsertSchema, createUpdateSchema
│ ├── column.ts # Column→TypeBox mappings (SQLite-only dispatch)
│ ├── types.ts # Public TypeScript interfaces (merged from schema.types + schema.types.internal)
│ ├── constants.ts # Integer range constants
│ └── utils.ts # isColumnType, isWithEnum, type helpers
└── tables/ # Restructured into subdirectories
├── common.ts # shared column definitions, ACTOR_TYPE removed
├── metagraph/ # graph_types, node_types, edge_types, graphs, nodes, edges
│ └── index.ts
└── identity/ # NEW: accounts, organizations, org_members, api_keys, audit_logs
└── index.ts
```
Removed: `src/pg/`, `actors` table, `@alkdev/drizzlebox` external dep,
`@libsql/client` dep.
New deps: `@alkdev/pubsub` (peer dep for `TypedEventTarget` type only),
`honker` (replaces `@libsql/client`).
## 1. Fold dbtype Phase 0 → `src/sqlite/utils/`
### What to fold
Only SQLite-relevant code from `@alkdev/dbtype/src/`:
| Source (dbtype) | Target (storage) | Changes |
|-----------------|-------------------|---------|
| `schema.ts` | `utils/schema.ts` | Remove `PgEnum` import/handling, remove `isPgEnum` calls. Remove `handleEnum()`. Keep `createSelectSchema`, `createInsertSchema`, `createUpdateSchema`. |
| `column.ts` | `utils/column.ts` | Strip all non-SQLite dispatch branches. Remove PG, MySQL, SingleStore type imports and `isColumnType` branches. Keep: `SQLiteInteger`, `SQLiteReal`, `SQLiteText` handling. Keep generic dispatch (`dataType`-based) as fallback. Keep `literalSchema`, `jsonSchema`, `bufferSchema`, `mapEnumValues`. |
| `column.types.ts` | `utils/types.ts` (merged) | Strip non-SQLite type branches. Keep `SQLiteInteger` → `t.TInteger`. Keep generic fallback types. |
| `schema.types.ts` | `utils/types.ts` (merged) | Remove `PgEnum` overloads. |
| `schema.types.internal.ts` | `utils/types.ts` (merged) | Keep `Conditions`, `BuildRefine`, `BuildSchema`, `NoUnknownKeys`. |
| `constants.ts` | `utils/constants.ts` | Keep as-is (ranges used by SQLite integer/real/bigiint mappings). |
| `utils.ts` | `utils/utils.ts` | Remove `PgEnum` type alias. Keep `isColumnType`, `isWithEnum`. Keep `JsonSchema`, `BufferSchema` types. Remove unused type helpers. |
### What NOT to fold
- PG/MySQL/SingleStore column type handlers (dead code for SQLite-only)
- `isPgEnum` and `handleEnum` (PostgreSQL-specific)
- `createSchemaFactory` (no known consumer in storage; can add later if needed)
- The `scripts/probe-e2e.ts` (dbtype Phase 1 POC, not relevant)
### Import changes in existing table files
Every table file currently imports from `@alkdev/drizzlebox`:
```ts
// Before
import { createInsertSchema, createSelectSchema } from "@alkdev/drizzlebox";
// After
import { createInsertSchema, createSelectSchema } from "../utils/schema.ts";
```
The `createSelectSchema` and `createInsertSchema` functions produce the same
TypeBox schemas — the API surface is identical, the implementation just
excludes non-SQLite branches.
### Deno.json import map changes
```jsonc
// Before
"imports": {
"@alkdev/drizzlebox": "npm:@alkdev/drizzlebox",
"@libsql/client": "npm:@libsql/client",
// ...
}
// After (remove drizzlebox and libsql, add honker and pubsub peer)
"imports": {
"honker": "npm:honker",
"@alkdev/pubsub": "npm:@alkdev/pubsub",
// ...
}
```
Note: `@alkdev/pubsub` is a peer dependency for the `TypedEventTarget` type.
It is only needed by consumers who use the `HonkerEventTarget`. The
`src/graphs/` module remains zero-dep.
## 2. Drizzle-Honker Session Adapter (`src/sqlite/adapter.ts`)
### Problem
Drizzle ORM expects a session that implements `SQLiteSession<'sync'>`. Honker
provides `Database.query(sql, params)` for reads and `Transaction.execute(sql,
params)` for writes. These don't implement Drizzle's session interface.
### Adapter Architecture
```
Consumer code
├── db.select().from(graphs)... → Drizzle query builder
│ └── HonkerPreparedQuery.run/all/get/values
│ └── honkerDb.query(sql, params) / tx.execute(sql, params)
├── db.$client.notify('channel', p) → Honker ephemeral pub/sub
├── db.$client.stream('events') → Honker durable stream
├── db.$client.queue('jobs') → Honker work queue
└── db.$client.listen('channel') → Honker listener
```
Three adapter classes:
| Class | Extends | Purpose |
|-------|---------|---------|
| `HonkerSQLiteSession` | `SQLiteSession<'sync'>` | Routes Drizzle queries to `honkerDb.query()` / `tx.execute()` |
| `HonkerPreparedQuery` | `SQLitePreparedQuery` | Implements `run()`, `all()`, `get()`, `values()` via Honker |
| `HonkerSQLiteTransaction` | `SQLiteTransaction<'sync'>` | Drizzle transaction with `$honkerTx` for Honker primitives |
### Accessing Honker from Drizzle
- `db.$client` — the Honker `Database` instance (outside transactions)
- `tx.$honkerTx` — the Honker `Transaction` instance (inside Drizzle tx callbacks)
```ts
db.transaction((tx) => {
tx.insert(schema.nodes).values({ graphId, key: "call-1", attributes: {} }).run();
tx.$honkerTx.notify("nodes:created", { graphId, key: "call-1" });
});
```
Both data write and notification commit atomically.
### Known challenges (need POC validation)
1. **No `lastInsertRowid` from Honker `execute()`** — Drizzle's `run()`
needs `lastInsertRowid` for auto-increment inserts. Honker's JS API
doesn't expose it. POC 1 confirmed workaround: `tx.query("SELECT last_insert_rowid() as id")` returns the rowid.
2. **Object rows only** — Honker `query()` returns
`Array<Record<string, any>>`. Drizzle's `values()` mode expects raw
arrays. The adapter must convert via `Object.values()` with column
order from the query metadata.
3. **Drizzle internal API dependency** — The adapter imports from
`drizzle-orm/sqlite-core/session`, which is not a guaranteed-stable API.
POC 1 confirmed all needed classes are accessible and extendable.
4. **Session constructor shape** — POC 1 confirmed: `SQLiteSession` takes
`(mode, executeMethod, query, cache?, queryMetadata?)`. `BaseSQLiteDatabase`
is accessible from `drizzle-orm/sqlite-core/db`. The `LibSQLSession` in
`drizzle-orm/libsql/session` is a working reference implementation.
5. **`:memory:` databases don't work** — Honker's reader pool creates
separate in-memory databases. File-based paths must be used. POC 1
confirmed this. Not a real limitation — production uses file paths,
tests use temp files.
## 3. HonkerEventTarget (`src/sqlite/event-target.ts`)
### Interface
Implements `@alkdev/pubsub`'s `TypedEventTarget<TEvent>`:
```ts
interface HonkerEventTargetOptions {
db: Database;
mode: "ephemeral" | "durable";
streamName?: string;
prefix?: string;
}
function createHonkerEventTarget<TEvent>(options: HonkerEventTargetOptions): HonkerEventTarget<TEvent>
```
### Ephemeral mode mapping
Maps to Honker's `notify()` / `listen()`:
```ts
addEventListener(type, callback):
if (!listeners.has(type)):
honkerListener = db.listen(type)
// start async loop: for await (notification of honkerListener)
// dispatch CustomEvent with notification as detail
listeners.set(type, callback)
dispatchEvent(event):
db.notify(event.type, event.detail) // or tx.notify() inside transaction
removeEventListener(type, callback):
listeners.delete(type, callback)
if (no more listeners for type):
honkerListener.close()
```
### Durable mode mapping
Maps to Honker's `Stream.publish()` / `Stream.subscribe()`:
```ts
addEventListener(type, callback):
if (!subscriptions.has(type)):
stream = db.stream(streamName ?? type)
subscription = stream.subscribe(consumerName)
// start async loop: for await (event of subscription)
// dispatch CustomEvent with event.payload as detail
subscriptions.set(type, { stream, subscription, callback })
dispatchEvent(event):
db.stream(streamName).publish(event.detail)
// or stream.publishTx(tx) inside transaction
removeEventListener(type, callback):
subscriptions.delete(type, callback)
if (no more subscriptions for type):
subscription.close()
```
### Topic routing considerations
pubsub uses `type:id` composite topics (e.g., `"call.responded:req-123"`).
Honker channels are flat strings. The mapping is 1:1 — no transformation
needed. But:
- **Ephemeral**: Each unique `type:id` gets its own `db.listen()` call. For
high-cardinality topics (many request IDs), this could mean many concurrent
listeners. POC 3 confirmed multiple concurrent listeners work.
- **Durable**: Streams are name-keyed, not topic-keyed. A single stream
(e.g., `"call-protocol"`) carries all call events. The adapter would need
to handle topic filtering client-side (dispatch only to matching listeners)
rather than using separate streams per topic.
- **Suggested split**: Call protocol events → durable stream. Cache
invalidation, UI push → ephemeral channels. The configuration for which
topics go where is a consumer decision, not built into the adapter.
- **Latency consideration**: POC 2 measured ~17ms median for
`notify`→`listen` round-trip. For hot-path call protocol request/response
in a single process, pair the Honker event target with an in-process
`EventTarget` (pubsub's default) for sub-ms latency. The Honker event
target provides durability and cross-process coordination; the default
EventTarget provides speed. A composite pattern (dispatch to both) could
serve both needs.
### Consumer name strategy
For durable mode, each consumer needs a unique name for offset tracking.
Suggested pattern: `{processId}:{topic}` or a configurable consumer group ID.
## 4. Client Factory Update (`src/sqlite/client.ts`)
### Before
```ts
import { drizzle, type LibSQLDatabase } from "drizzle-orm/libsql";
import type { Client } from "@libsql/client";
export function createSqliteDatabase(client: Client): LibSQLDatabase<SqliteSchema> {
return drizzle(client, { schema });
}
```
### After
The factory accepts a pre-created Honker `Database` instance (injectable
client pattern, per ADR-004). It constructs the Drizzle-Honker adapter
internally.
```ts
import type { Database as HonkerDatabase } from "honker";
export type SqliteDatabase = BaseSQLiteDatabase<'sync', SqliteSchema>;
export function createSystemDatabase(client: HonkerDatabase): SqliteDatabase {
return createDrizzleWithAdapter(client, { schema });
}
export function createTenantDatabase(client: HonkerDatabase): SqliteDatabase {
return createDrizzleWithAdapter(client, { schema });
}
```
`createDrizzleWithAdapter` is the internal factory that constructs the
`HonkerSQLiteSession` and wraps it in a `BaseSQLiteDatabase`. This may
require using `drizzle-orm/sqlite-core` internals rather than the
`drizzle-orm/libsql` high-level wrapper.
## 5. Table Restructure
### Remove actors table
Per existing ADRs (035, 041), actors are being replaced by ACL nodes in
graph instances. Remove `src/sqlite/tables/actors.ts` and all exports.
### Reorganize into subdirectories
```
src/sqlite/tables/
├── common.ts # commonCols (UPDATED: remove ACTOR_TYPE)
├── metagraph/
│ ├── index.ts # barrel
│ ├── graph-types.ts # renamed from graphTypes.ts
│ ├── node-types.ts # renamed from nodeTypes.ts
│ ├── edge-types.ts # renamed from edgeTypes.ts
│ ├── graphs.ts
│ ├── nodes.ts
│ └── edges.ts
└── identity/
├── index.ts # barrel
├── accounts.ts
├── organizations.ts
├── org-members.ts
├── api-keys.ts
└── audit-logs.ts
```
### Import updates in table files
All table files currently importing `@alkdev/drizzlebox` update to
`../utils/schema.ts` (or `../../utils/schema.ts` depending on depth).
The `commonCols` import path changes from `./common.ts` to `../common.ts`
for metagraph/identity subdirectories.
## 6. Domain-Specific Tables (Future, documented for scope)
The current metagraph tables store node/edge attributes as JSON. For known
graph types (CallGraph, SecretGraph, AclGraph), native columns give better
type safety and query performance.
This is **not part of the initial pivot**. But the pivot should not prevent
it. The folded `utils/` code already provides `createInsertSchema` /
`createSelectSchema` which derives TypeBox schemas from Drizzle tables with
native columns. When domain tables are added later, they follow the same
pattern as the metagraph tables.
### Where OperationSpecs fit
Domain tables would come with auto-generated `OperationSpec[]` — flat arrays
describing CRUD operations for each table. The hub/spoke imports these specs,
registers them in the operations registry, and the call protocol handles
execution.
This means storage's "repository layer" is not hand-written query functions.
It's **table definitions + operation contract definitions**. The consumer
(hub/spoke) provides the execution layer.
```ts
// In storage: table definition + operation contracts
export const callNodes = sqliteTable("call_nodes", { ... });
export const callNodeOperations: OperationSpec[] = [
{ name: "create", namespace: "call_nodes", type: "mutation", ... },
{ name: "find", namespace: "call_nodes", type: "query", ... },
// ...
];
// In hub: register operations
for (const spec of callNodeOperations) {
registry.registerSpec(spec);
registry.registerHandler(specId, (input) => db.insert(callNodes).values(input).returning());
}
```
The OperationSpec type is straightforward (`@alkdev/operations` peer dep for
type only). Storage defines the contract; the hub provides the handler.
## 7. Identity Tables (New, documented for scope)
Per `sqlite-host.md` and ADR-040/041, these live in the system database:
| Table | Key Columns | Purpose |
|-------|-------------|---------|
| `accounts` | id, email, name, accessLevel | User accounts |
| `organizations` | id, slug, name, ownerId | Tenant organizations |
| `org_members` | id, orgId, accountId, level | Membership junction |
| `api_keys` | id, keyHash, ownerId, name | API key auth |
| `audit_logs` | id, actorType, actorId, action, resource | Append-only audit |
These need actual implementation (currently only spec in architecture docs).
They're part of the pivot scope but can be implemented after the adapter and
utils fold.
## 8. Event Pipeline: Operations → Flowgraph → Storage
### How the pieces connect after the pivot
```
[operations: OperationRegistry]
│
│ call.requested / call.responded / call.completed / call.aborted / call.error
▼
[pubsub: createPubSub({ eventTarget: honkerEventTarget })]
│
├──→ [flowgraph: WorkflowReactiveRoot.append(event)]
│ └── signal propagation → UI updates
│
├──→ [storage: db.stream("call-protocol").publish(event)]
│ └── durable replay for crash recovery
│
└──→ [spoke sync: via WebSocket or Honker fan-out]
└── cross-process coordination
```
The `HonkerEventTarget` is the single integration point that makes this work.
Operations doesn't need to know about Honker. Flowgraph doesn't need to know
about Honker. They both use `@alkdev/pubsub`'s `TypedEventTarget`. The
`HonkerEventTarget` is just another adapter in pubsub's existing portfolio
(alongside Redis, WebSocket, Worker).
### Hub-spoke event routing
```
Hub (system.db + tenant-{orgId}.db)
├── HonkerEventTarget (durable: call-protocol stream)
├── WebSocketServerEventTarget (spoke fan-out)
└── In-process EventTarget (local subscribers)
Spoke (tenant-{orgId}.db)
├── HonkerEventTarget (ephemeral: local channels)
├── WebSocketClientEventTarget (hub connection)
└── In-process EventTarget (local subscribers)
```
Both hub and spoke use the same code. Same `createPubSub({ eventTarget })`.
Different event target instance determines the routing.
## POC Plan
Before committing to the full pivot, validate the critical path with targeted
POCs. Each POC answers one specific hard question.
### POC 1: Drizzle queries through Honker
**Question**: Can we construct a `BaseSQLiteDatabase` that routes queries
through Honker's `Database.query()` and `Transaction.execute()`?
**Approach**:
1. Import `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` from
`drizzle-orm/sqlite-core/session`.
2. Implement `HonkerSQLiteSession` that delegates to `Database.query()`.
3. Implement `HonkerPreparedQuery` with `run()`, `all()`, `get()`.
4. Wire it into a `BaseSQLiteDatabase` instance.
5. Run a simple `db.select().from(graphs)` against a Honker-opened `.db` file.
**Hard fail conditions**:
- Drizzle's `SQLiteSession` requires constructor arguments we can't provide
from Honker's primitives.
- The `BaseSQLiteDatabase` constructor requires internals not exported from
`drizzle-orm/sqlite-core`.
- `lastInsertRowid` is required and no workaround exists.
### POC 2: Honker pub/sub from same process
**Question**: Does `db.notify()` → `db.listen()` work within the same Node.js
process? What's the latency?
**Approach**:
1. Open a Honker database.
2. Start `db.listen("test-channel")`.
3. Call `db.notify("test-channel", { hello: "world" })`.
4. Verify the listener receives the notification.
5. Measure round-trip latency (10,000 iterations).
**Hard fail conditions**:
- Same-process `notify`/`listen` doesn't work (would require two DB instances
or fallback to in-process EventTarget).
- Latency exceeds 10ms median (would be too slow for call protocol
request/response).
### POC 3: Concurrent listeners and stream subscriptions
**Question**: Can multiple `Listener` and `StreamSubscription` instances
coexist on the same `Database`? Do they interleave correctly?
**Approach**:
1. Open a Honker database.
2. Start two `db.listen()` calls on different channels.
3. Start a `db.stream("test-stream").subscribe("consumer-1")`.
4. Publish to both channels and the stream concurrently.
5. Verify all three consumers receive their events without blocking each
other.
**Hard fail conditions**:
- Only one `Listener` can be active at a time (would require serialization).
- `Listener.next()` blocks the JS event loop (would prevent concurrent
subscriptions).
### POC 4: Transactional notify + data write
**Question**: Does `tx.notify()` + `tx.execute()` in the same transaction
atomically commit both the data write and the notification?
**Approach**:
1. Open a Honker database.
2. Start `db.listen("test-channel")`.
3. Begin a transaction: `const tx = db.transaction()`.
4. Execute a data write: `tx.execute("INSERT INTO test ...")`.
5. Call `tx.notify("test-channel", { inserted: true })`.
6. Rollback the transaction.
7. Verify the listener does NOT receive the notification.
8. Repeat with commit instead of rollback.
9. Verify the listener DOES receive the notification after commit.
**Hard fail conditions**:
- Notifications fire immediately on `tx.notify()` regardless of commit/rollback
(breaks transactional outbox semantics).
### POC file structure
```
poc/
├── README.md # POC objectives, how to run
├── 01-drizzle-honker-adapter.ts # POC 1
├── 02-same-process-pubsub.ts # POC 2
├── 03-concurrent-listeners.ts # POC 3
├── 04-transactional-notify.ts # POC 4
└── shared/
└── honker-helpers.ts # Shared setup/teardown utilities
```
Each POC is a standalone script that can be run with `deno run` or `node`.
The POCs use the actual `honker` npm package from the workspace.
## ADR Updates Required
| ADR | Action | Reason |
|-----|--------|--------|
| ADR-018 (dbtype integration post-v1) | **Revise** | No longer deferred — Phase 0 folds in now as `src/sqlite/utils/` |
| ADR-038 (SQLite-first, PG removed) | **Confirm** | Pivot confirms this. Remove `src/pg/` and PG imports from deno.json. |
| ADR-039 (Honker as SQLite extension) | **Confirm** | Pivot is the implementation of this decision. |
| ADR-044 (Drizzle-Honker adapter) | **Update** | Add specifics about session/PreparedQuery/Transaction classes. |
| ADR-005 (drizzle + typebox via drizzlebox) | **Revise** | Drizzlebox is no longer an external dep. It's `src/sqlite/utils/`. |
| ADR-019 (JSON text for schema columns) | **Confirm** | Still relevant for metagraph attributes. Domain tables add native columns alongside JSON. |
| ADR-033 (JSON path queries for v1) | **Revise** | Repository layer is now "operation specs," not hand-written CRUD. The query approach (JSON path vs native) still applies, but the output format changes. |
| New ADR | Fold dbtype Phase 0 as utils | Records the decision to fold SQLite mappings into `src/sqlite/utils/`. |
| New ADR | HonkerEventTarget adapter | Records the decision to implement pubsub TypedEventTarget on Honker primitives. |
| New ADR | Operation specs as repository surface | Records that storage outputs OperationSpec[], not hand-written query functions. |
## Architecture Doc Updates Required
| Document | Changes |
|----------|---------|
| `overview.md` | Subpath exports (`./pg` removed), dependency list, module structure, "what's not done" section, system/tenant DB model |
| `forward-look.md` | dbtype section (folded, not deferred), repository layer strategy (operations, not CRUD), pointer abstraction (still deferred) |
| `honker-integration.md` | Add HonkerEventTarget section, update adapter section with verified API, add event pipeline diagram, resolve OQ-26 (Honker replaces Redis for single-node) |
| `sqlite-host.md` | Table structure changes (subdirectories, actors removed, identity tables), client factory signature, Honker import instead of libsql |
| `metagraph-module.md` | No changes (Module format is independent of storage host) |
| `schema-evolution.md` | Minor: `createSelectSchema`/`createInsertSchema` now come from local utils, not external dep |
| `open-questions.md` | Resolve OQ-17 (operations, not CRUD), OQ-18 (auto-generated specs from tables), OQ-26 (Honker replaces Redis). Update OQ-19 (bridge package location → it's storage itself). |
| `encrypted-data.md` | No changes |
## Implementation Order (post-POC)
If POCs pass, the implementation sequence should be:
1. **Fold dbtype → `src/sqlite/utils/`** — import path changes only, no
behavioral changes. Run existing tests to verify.
2. **Remove `src/pg/` and actors table** — clean up dead code. Update deno.json
imports, remove `postgres` and PG-related entries.
3. **Restructure tables into subdirectories** — move files, update relative
imports, update `schema.ts` and `relations.ts`.
4. **Implement `adapter.ts`** — Drizzle-Honker session adapter based on POC
1 findings.
5. **Update `client.ts`** — `createSystemDatabase()` / `createTenantDatabase()`
taking Honker `Database` instances.
6. **Implement `event-target.ts`** — `HonkerEventTarget` based on POC 2-4
findings.
7. **Implement identity tables** — `src/sqlite/tables/identity/`.
8. **Update architecture docs** — bring all documents in line with the new
structure.
9. **Add domain tables** (future, not in initial pivot) — native-column
tables for CallGraph, SecretGraph, etc. with OperationSpec[] output.
Steps 1-3 are mechanical (no new behavior). Steps 4-5 depend on POC 1.
Step 6 depends on POCs 2-4. Step 7 can proceed independently. Steps 8-9
are follow-ups.
## POC Results (2026-06-01)
### POC 1: Drizzle queries through Honker — PASS
- `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` are all
accessible from `drizzle-orm/sqlite-core/session` — can be extended.
- `BaseSQLiteDatabase` is accessible from `drizzle-orm/sqlite-core/db`.
- Honker `query()` returns `{ columnName: value }` object rows, compatible
with Drizzle's `mapResultRow()`.
- `last_insert_rowid()` accessible via `tx.query("SELECT last_insert_rowid() as id")`.
- `tx.execute()` returns a number (affected rows count for UPDATE/DELETE).
- JSON-mode columns need manual `JSON.parse()` in the adapter.
- `:memory:` databases DON'T work — reader pool gets separate in-memory DBs.
Always use file-based paths (or `:memory:` with a single reader, which
defeats Honker's concurrent reader design).
- `LibSQLSession` in `drizzle-orm/libsql/session` is a good reference
implementation for the adapter pattern.
Session method signatures discovered:
```
SQLiteSession methods:
prepareOneTimeQuery — key method to implement
all, run, get, values — delegated to PreparedQuery
extractRaw*ValueFromBatchResult — batch result extraction
count — cursor count
SQLitePreparedQuery methods:
execute, getQuery, mapAllResult, mapGetResult, mapResult, mapRunResult
queryWithCache
SQLiteTransaction methods:
rollback — only explicit method (commit handled by Session.transaction)
```
### POC 2: Same-process pub/sub — PASS (with latency caveat)
- `db.notify()` → `db.listen()` works within the same process.
- Multiple concurrent listeners on different channels work.
- Cross-channel isolation works — listeners only receive notifications on
their subscribed channel.
- **Latency**: ~17ms median, ~29ms P99 (100 samples). This is higher than
ideal for call protocol request/response. The `Listener` uses a polling
fallback (`updateEvents().next()`) internally. For hot-path events where
sub-ms latency matters, the in-process `EventTarget` should be preferred.
Honker listeners are better suited for cross-process and durable scenarios.
### POC 3: Concurrent listeners and streams — PASS
- Multiple `Listener` instances coexist on the same `Database`.
- `Stream.publish()` / `Stream.subscribe()` works.
- `readSince(offset, limit)` returns historical events.
- `StreamSubscription` implements `AsyncIterableIterator` — works with
`for await...of`.
- Consumer offsets are tracked and persisted.
- Concurrent stream + listener operation works.
- `publishTx(tx, payload)` works within a transaction.
- **Caveat**: `db.close()` doesn't cleanly exit when listeners are still
polling. Need to close all listeners before closing the DB.
### POC 4: Transactional notify + data write — PASS (atomicity confirmed)
- `tx.notify()` + `tx.execute()` — notification fires ONLY after `tx.commit()`.
Before commit, no notification is received. **Transactional semantics hold.**
- On `tx.rollback()` — notification is NOT sent. Data is NOT persisted.
**Atomicity holds for both data and events.**
- `db.notifyTx(tx, channel, payload)` — convenience wrapper, same semantics.
- `queue.enqueueTx(tx, payload)` — job is only visible after commit. On
rollback, job does not appear in the queue. **Transactional outbox works.**
- Read-your-writes: after receiving a notification, the data IS queryable
(no stale read issue with single DB).
**This is the most important result.** Honker provides true transactional
outbox semantics — data writes and event/queue publishes commit atomically.
This eliminates the dual-write problem that necessitated Redis for PostgreSQL.
## Open Research Questions
1. ~~**Drizzle session construction**~~ — RESOLVED by POC 1. All needed
classes are accessible and extendable. `LibSQLSession` is the reference.
2. ~~**Honker listener concurrency model**~~ — RESOLVED by POC 3. Multiple
listeners coexist. The `updateEvents` mechanism works for concurrent
polling. `db.close()` requires all listeners to be closed first.
3. **Consumer name management for streams** — When a hub restarts, durable
stream consumers resume from their last saved offset. But the consumer
name must be stable across restarts. Need a naming convention that's
deterministic (e.g., `{service}:{pid}` is not restart-stable).
4. **Schema migrations** — Honker opens a single `.db` file. Drizzle Kit
supports SQLite migrations but expects `better-sqlite3` or `libsql`. Need
to verify that `drizzle-kit push` or `drizzle-kit generate` works with the
Honker adapter, or whether we need a custom migration runner.
5. **OperationSpec generation from tables** — What's the concrete mapping?
For each table: one `create` mutation, one `find` query, one `list` query,
one `update` mutation, one `delete` mutation? Or something more
domain-specific? This is a design question, not a POC question.
6. **In-process + Honker composite event target** — POC 2 showed ~17ms
latency for Honker notify→listen. For single-process hot paths, an
in-process EventTarget is sub-ms. A composite that dispatches to both
(in-process for speed, Honker for durability/cross-process) would be the
ideal default for a hub. Needs design work.