Pivot: fold drizzlebox as utils, HonkerEventTarget, OperationSpecs as repo surface

- Update architecture docs to reflect pivot from @libsql/client to Honker
- Fold @alkdev/drizzlebox Phase 0 into src/sqlite/utils/ (ADR-046)
- Add HonkerEventTarget adapter for pubsub TypedEventTarget (ADR-047)
- Replace hand-written CRUD with OperationSpec generation (ADR-048)
- Resolved OQ-26: Honker replaces Redis for single-node pub/sub (POC validated)
- Updated OQ-17, OQ-18, OQ-19 for OperationSpec repository surface
- Added OQ-30 (composite event target), OQ-31 (consumer naming), OQ-32 (Drizzle Kit)
- POC results: adapter buildable, same-process pub/sub works, transactional
  outbox semantics confirmed, concurrent listeners/streams work
- Research doc at docs/research/pivot-honker-sqlite-adapter.md
This commit is contained in:
2026-06-01 16:31:40 +00:00
parent 6aa2fcc6ff
commit 412ad98f11
10 changed files with 1342 additions and 230 deletions

View File

@@ -0,0 +1,53 @@
---
status: accepted
date: 2026-06-01
supersedes: ADR-005, ADR-018
---
# ADR-046: Fold @alkdev/drizzlebox as src/sqlite/utils/
## Context
`@alkdev/drizzlebox` (a fork of `drizzle-typebox` adapted for
`@alkdev/typebox`) provides `createSelectSchema` and `createInsertSchema`
functions that derive TypeBox validation schemas from Drizzle table
definitions. It is consumed as an external npm dependency by all SQLite
table files in `@alkdev/storage`.
The parent project `@alkdev/dbtype` also planned a Phase 1 (UJSX→HostConfig
→Drizzle pipeline) that was never implemented. With SQLite as the sole
database target (ADR-038), the multi-dialect column mappings in dbtype
(PG, MySQL, SingleStore) are dead weight for storage.
ADR-018 deferred dbtype integration to post-v1. But the fold of just the
Phase 0 subset (column→TypeBox mappings + schema generation) is a
straightforward import path change with no behavioral difference.
## Decision
Fold the SQLite-only subset of `@alkdev/dbtype` Phase 0 into
`src/sqlite/utils/`:
| Source (dbtype) | Target (storage) | Changes |
|-----------------|-------------------|---------|
| `schema.ts` | `utils/schema.ts` | Remove PgEnum handling. Keep createSelectSchema, createInsertSchema, createUpdateSchema. |
| `column.ts` | `utils/column.ts` | Strip PG, MySQL, SingleStore branches. Keep SQLiteInteger, SQLiteReal, SQLiteText + generic dataType dispatch. |
| `schema.types.ts` + `schema.types.internal.ts` + `column.types.ts` | `utils/types.ts` | Merged. Remove PgEnum overloads. |
| `constants.ts` | `utils/constants.ts` | Keep as-is. |
| `utils.ts` | `utils/utils.ts` | Remove PgEnum alias. Keep isColumnType, isWithEnum, JsonSchema, BufferSchema. |
All table files change their import from `@alkdev/drizzlebox` to `../utils/schema.ts`.
## Consequences
- **No external drizzlebox dependency** — one fewer npm package in the
dependency graph.
- **SQLite-only column mappings** — dead multi-dialect code removed. If a
new database host is added later, mappings would need to be re-added.
- **Same API surface** — `createSelectSchema`, `createInsertSchema`,
`createUpdateSchema` produce the same TypeBox schemas.
- **Co-located with tables** — utility code lives next to the tables it
derives schemas from. Easier to maintain.
- **dbtype Phase 1 (UJSX→HostConfig) not affected** — that remains a
separate architectural concern, could live in storage or as its own
package when built.

View File

@@ -0,0 +1,58 @@
---
status: accepted
date: 2026-06-01
resolves: OQ-26
---
# ADR-047: HonkerEventTarget Adapter for pubsub
## Context
`@alkdev/pubsub` defines a `TypedEventTarget` interface that all transport
adapters implement: in-process `EventTarget`, Redis, WebSocket client/server,
and Worker. This provides transport-agnostic pub/sub — consumers call
`createPubSub({ eventTarget })` without knowing the underlying transport.
Honker provides SQLite with built-in pub/sub primitives:
- `db.notify(channel, payload)` / `db.listen(channel)` — ephemeral, fire-and-forget
- `db.stream(name).publish(payload)` / `db.stream(name).subscribe(consumer)` — durable, offset-tracked
POC 2-4 (2026-06-01) validated:
- Same-process notify→listen works. ~17ms median latency.
- Multiple concurrent listeners on different channels work.
- `tx.notify()` only fires on `tx.commit()`. Rollback suppresses notification.
- `queue.enqueueTx(tx, payload)` only visible after commit. Rollback suppresses.
- Stream publish/subscribe works with consumer offset tracking.
These results confirm Honker can back the pubsub `TypedEventTarget`
interface for single-node deployments.
## Decision
Implement `HonkerEventTarget` in `src/sqlite/event-target.ts`. It adapts
`@alkdev/pubsub`'s `TypedEventTarget` to Honker primitives with two modes:
1. **Ephemeral mode**: `addEventListener``db.listen()`, `dispatchEvent`
`db.notify()`. Fire-and-forget semantics, no delivery guarantee.
2. **Durable mode**: `addEventListener``db.stream().subscribe()`,
`dispatchEvent``db.stream().publish()`. Per-consumer offset tracking,
crash recovery replays from last saved offset.
`@alkdev/pubsub` is a peer dependency (needed only when using
HonkerEventTarget). The `graphs/` module remains zero-dep.
## Consequences
- **Single-node hub can use Honker instead of Redis** — no separate Redis
deployment needed for pub/sub.
- **Transactional outbox semantics** — `dispatchEvent` inside a Drizzle
transaction (via `tx.notify()` or `stream.publishTx()`) commits atomically
with data writes. No dual-write problem.
- **Hub-spoke symmetry** — both hub and spoke use the same `createPubSub()`
call. Different event target instances determine routing.
- **Multi-node still needs Redis or WebSocket** — Honker events don't cross
process boundaries. For multi-node, the `WebSocketServerEventTarget` (hub)
and `WebSocketClientEventTarget` (spoke) handle cross-process routing.
- **Latency trade-off** — ~17ms Honker round-trip vs sub-ms in-process. For
hot-path call protocol, pair with in-process `EventTarget`. Design of a
composite event target is an open question (OQ-30).

View File

@@ -0,0 +1,74 @@
---
status: accepted
date: 2026-06-01
supersedes: ADR-033 (partially — OQ-17, OQ-18, OQ-19 updated)
---
# ADR-048: OperationSpecs as Repository Surface
## Context
ADR-033 established JSON path queries with hand-written CRUD for v1. The
question was whether the repository layer would be hand-written query
functions, auto-generated from table definitions, or something else.
The `@alkdev/operations` package provides `OperationSpec` — a serializable
descriptor for a query, mutation, or subscription operation. Hubs and spokes
register specs in an `OperationRegistry` along with handlers. The operations
runtime handles execution, call protocol routing, subscriptions, access
control, and validation.
This presents a natural alternative to hand-written CRUD: storage outputs
`OperationSpec[]` describing CRUD operations per table, and the consumer
wires them into the operations registry.
## Decision
Storage does not ship a "repository layer" of hand-written query functions.
Instead, it outputs `OperationSpec[]` per table — flat arrays describing
CRUD operations. The consumer (hub/spoke) imports these specs, registers
handlers, and the operations runtime handles execution.
```ts
// Storage defines table + operation contracts
export const graphTypes = sqliteTable("graph_types", { ... });
export const graphTypeSpecs: OperationSpec[] = [
{ name: "create", namespace: "graph_types", type: "mutation", ... },
{ name: "find", namespace: "graph_types", type: "query", ... },
{ name: "list", namespace: "graph_types", type: "query", ... },
{ name: "update", namespace: "graph_types", type: "mutation", ... },
{ name: "delete", namespace: "graph_types", type: "mutation", ... },
];
// Hub registers specs + handlers
for (const spec of graphTypeSpecs) {
registry.registerSpec(spec);
registry.registerHandler(`${spec.namespace}.${spec.name}`, handler);
}
```
The handler is consumer-provided. Storage defines the contract (input/output
schemas, operation type); the hub provides the execution (Drizzle queries,
Honker transactions, notifications).
`@alkdev/operations` is a type-only peer dependency of storage. No circular
dependency.
## Consequences
- **No hand-written repository functions** — storage avoids the maintenance
burden of typed CRUD code. The contract is the OperationSpec.
- **Consumer owns execution** — the hub decides how to handle each operation
(raw Drizzle query, with Honker notification, with access control, etc.).
Storage doesn't execute queries.
- **Subscriptions for free** — OperationSpec supports `type: "subscription"`.
Table-level change streams become operation subscriptions rather than
custom event handling.
- **Clean dependency graph** — storage depends on operations only for the
`OperationSpec` type (peer dep). No runtime dependency.
- **Attribute queries remain JSON path** — for metagraph tables where
attributes are dynamic JSON, `json_extract()` is still the query mechanism.
Domain-specific tables with native columns produce simpler specs.
- **Supersedes ADR-033 partially** — the "hand-written CRUD" part is replaced.
The "JSON path queries for attributes" part still applies to metagraph
tables.

View File

@@ -1,16 +1,15 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Forward Look: Pointers, dbtype, and Universal IR
How the Module-based metagraph connects to the broader @alkdev ecosystem —
typed graph pointers, dbtype table rendering, and the ujsx universal IR
pipeline. These are forward-looking designs that justify why certain structural
decisions were made now
(pointer abstraction deferred per [ADR-017](./decisions/017-pointer-abstraction-is-forward-looking.md),
dbtype integration deferred per [ADR-018](./decisions/018-dbtype-integration-is-post-v1.md)).
typed graph pointers, local utils (folded from dbtype), and the ujsx universal IR
pipeline. The dbtype integration is no longer deferred (ADR-046) — the SQLite-only
Phase 0 subset folds into `src/sqlite/utils/`. The repository surface is now
OperationSpecs (ADR-048), not hand-written CRUD.
## Overview
@@ -109,68 +108,69 @@ feasible because it provides the schema the pointer validates against.
## Relationship to @alkdev/dbtype
`@alkdev/dbtype` defines database schemas as ujsx element trees and renders them
to Drizzle dialects via HostConfig. Storage's SQLite/PG table definitions are a
natural consumer of this pipeline.
`@alkdev/dbtype` defined database schemas as ujsx element trees and planned to
render them to Drizzle dialects via HostConfig. Its Phase 0 (Drizzle→TypeBox
schema generation) was consumed as `@alkdev/drizzlebox`. Phase 1 (UJSX→Drizzle)
was never implemented.
### Current vs. Future Table Definition
### Fold: Phase 0 → `src/sqlite/utils/` (ADR-046)
**Current** (manual Drizzle table defs):
With SQLite as the sole target (ADR-038), the multi-dialect column mappings in
dbtype are dead weight. The SQLite-only subset has been folded into storage as
`src/sqlite/utils/`:
| What folds in | Source (dbtype) | Target (storage) |
|---------------|-----------------|-------------------|
| Schema generation | `schema.ts` | `utils/schema.ts` |
| Column→TypeBox mappings | `column.ts` (SQLite branches only) | `utils/column.ts` |
| Type interfaces | `schema.types.ts` + `schema.types.internal.ts` + `column.types.ts` | `utils/types.ts` |
| Integer constants | `constants.ts` | `utils/constants.ts` |
| Type guards | `utils.ts` (minus PgEnum) | `utils/utils.ts` |
What does NOT fold in: PG, MySQL, SingleStore column handlers; `isPgEnum` /
`handleEnum`; `createSchemaFactory`; the Phase 1 UJSX→HostConfig pipeline.
Import changes in table files:
```ts
export const graphTypes = sqliteTable("graph_types", {
id: text("id").primaryKey(),
name: text("name").notNull(),
config: text("config", { mode: "json" }).notNull(),
// ...
});
// Before
import { createInsertSchema, createSelectSchema } from "@alkdev/drizzlebox";
// After
import { createInsertSchema, createSelectSchema } from "../utils/schema.ts";
```
**Future** (dbtype element tree → HostConfig rendering):
The API surface is identical — same functions, same TypeBox schemas produced.
```tsx
const GraphTypesEl = h("table", { name: "graph_types" },
h(IdColumn, {}),
h("column", { name: "name", type: "string", notNull: true }),
h("column", { name: "config", type: "json", mode: "json", notNull: true }),
h(AuditColumns, {}),
);
### Phase 1 (UJSX→Drizzle): future path
const root = createRoot(sqliteHost, {});
root.render(GraphTypesEl);
const drizzleTable = root.ctx.tables.graph_types;
```
The broader UJSX→HostConfig→Drizzle pipeline from dbtype's architecture remains
architecturally sound but is not part of this pivot. When and if it's built, it
could live in storage as a `HostConfig` sub-module rather than a separate
package, since storage is the primary consumer. The TypeBox Module format used
by the metagraph is already compatible with what a ujsx HostConfig would produce.
### Why this matters for storage
1. **Single source of truth**: Today's `sqlite/tables/` and future `pg/tables/`
define the same shapes in two different Drizzle dialects. dbtype renders the
same element tree to both — no manual duplication.
2. **Schema extraction**: `extractTable()` produces both TypeBox schemas (for
validation) and column metadata (for Drizzle rendering) from the same tree.
Storage gets `SelectGraphType` and `InsertGraphType` schemas for free.
3. **Module alignment**: dbtype assembles extracted schemas into a
`Type.Module` for cross-table references. Storage's metagraph Module and
dbtype's table Module could share a namespace — the `graph_types.config`
column stores the JSON Schema from `Metagraph.Config`.
1. **Single source of truth**: The `utils/` code derives TypeBox schemas from
Drizzle tables. Table definitions are the source of truth for both the DB
schema and the validation schema.
2. **Schema extraction**: `createSelectSchema` / `createInsertSchema` produce
TypeBox schemas that validate data at the application layer.
3. **Module alignment**: The metagraph Module and the table-derived schemas
share the same TypeBox namespace. `graph_types.config` stores the JSON
Schema from `Metagraph.Config`.
### v1 approach
For v1, storage continues with manual Drizzle table definitions. The dbtype
integration is deferred because:
For v1, storage uses the folded utils for TypeBox schema derivation from Drizzle
tables (what was `@alkdev/drizzlebox`). The metagraph Module independently
validates graph type definitions. These two schema sources serve different
purposes: table schemas validate DB row shapes, Module schemas validate graph
type semantics.
- dbtype is Phase 0 (architecture complete, no implementation)
- The manual defs work and are well-understood
- The Module pattern for graph types can be adopted independently (no dbtype
dependency)
- With PostgreSQL removed (ADR-038), the original pressure for dbtype —
eliminating dual SQLite/PG table maintenance — is significantly reduced.
There is now only one set of table definitions to maintain.
When dbtype reaches Phase 1 (implementation), storage can migrate from
Drizzle table definitions to dbtype elements one table at a time. The Module-based
graph type definitions are already compatible — they're both TypeBox `Type.Module`
objects.
When dbtype's Phase 1 (UJSX→HostConfig) is implemented, it would unify both
directions — a TypeBox Module could produce both the Drizzle table definition
and the validation schemas from the same element tree.
## ujsx as Universal IR
@@ -219,10 +219,17 @@ Rendered to different hosts:
| Reference graph type Modules (CallGraph, SecretGraph) | ✅ Implemented |
| Crypto utility (`encrypt`, `decrypt`, `generateEncryptionKey`, `EncryptedDataSchema`) | ✅ Implemented |
| Codegen from TypeScript interfaces → Module entries | ✅ TsToModule exists |
| dbtype element trees → Drizzle tables | ⚠️ dbtype Phase 0, no implementation |
| SQLite column→TypeBox mappings (folded from dbtype) | ✅ Folded into `src/sqlite/utils/` (ADR-046) |
| `createSelectSchema` / `createInsertSchema` (folded from drizzlebox) | ✅ Folded into `src/sqlite/utils/` (ADR-046) |
| Drizzle-Honker session adapter | ✅ POC validated, implementation pending |
| HonkerEventTarget for pubsub | ✅ POC validated, implementation pending |
| Transactional notify + outbox (Honker) | ✅ POC validated — atomic commit for data + events + queue |
| OperationSpec generation from tables | ⚠️ Design complete (ADR-048), implementation pending |
| Domain-specific native-column tables | ⚠️ Conceptual — for known graph types (CallGraph, etc.) |
| `<graphSchema>` ujsx elements | ⚠️ Conceptual — needs HostConfig design |
| Typed graph pointers via JPATH | ⚠️ Conceptual — needs JPATH Module design |
| Reactive graph observation via ValuePointer | ⚠️ Conceptual — needs signal integration |
| dbtype Phase 1 (UJSX→Drizzle HostConfig) | ⚠️ Architecture exists, not implemented. Could live in storage if built. |
The Module-based graph type definitions (this spec) are the **first concrete
step** in this pipeline. Everything else builds on having a `Type.Module` as
@@ -230,126 +237,66 @@ the schema source of truth.
## Repository Layer Strategy
The repository layer (typed CRUD for the 6 metagraph tables + queries for graph data)
is the next major feature to implement. The question of *how* it queries attributes
connects to broader ecosystem decisions about dbtype and operations.
The repository layer (typed CRUD for the 6 metagraph tables + identity tables +
queries for graph data) is now defined as **OperationSpec output** rather than
hand-written query functions (ADR-048).
### Three Approaches
### OperationSpecs as Repository Surface
#### A. JSON Path Queries (Near-Term)
Storage outputs `OperationSpec[]` per table — flat arrays describing CRUD
operations. The consumer (hub/spoke) imports these, registers handlers, and
the operations runtime handles execution, call protocol, and subscriptions.
The repository layer maps filter criteria to JSON path extraction:
```ts
// Storage defines the table + operation contracts
export const callNodes = sqliteTable("call_nodes", { ... });
export const callNodeSpecs: OperationSpec[] = [
{ name: "create", namespace: "call_nodes", type: "mutation", inputSchema: ..., outputSchema: ... },
{ name: "find", namespace: "call_nodes", type: "query", ... },
{ name: "list", namespace: "call_nodes", type: "query", ... },
{ name: "update", namespace: "call_nodes", type: "mutation", ... },
{ name: "delete", namespace: "call_nodes", type: "mutation", ... },
];
// Hub registers specs + handlers
for (const spec of callNodeSpecs) {
registry.registerSpec(spec);
registry.registerHandler(`${spec.namespace}.${spec.name}`, handler);
}
```
The handler is consumer-provided — not in storage. Storage doesn't execute
queries. Storage defines the contract; the hub provides the execution layer.
### Attribute Queries
The metagraph's `attributes` column remains JSON — node types are dynamic
schemas defined at runtime, not static columns. Attribute queries use
`json_extract()` for v1:
```ts
findNodes({ graphId, attributes: { status: "active" } })
// SQLite: json_extract(attributes, '$.status') = 'active'
// PG: attributes ->> 'status' = 'active'
```
- Works with current table definitions (no schema changes)
- SQLite `json_extract()` and PG `->>` / `#>>` operators handle JSON path
- No native index support on individual JSON attributes
- PG can add GIN indexes on `jsonb` columns for containment queries, but not for
arbitrary key-value lookups
- Simple, immediate, no new infrastructure
For known graph types (CallGraph, SecretGraph), domain-specific tables with
native columns can complement the generic metagraph tables. These domain
tables also produce OperationSpecs with native-column queries.
This is the pragmatic v1 approach. The metagraph pattern *requires* JSON attributes
because node types are dynamic schemas (defined at runtime, stored in
`node_types.schema`), not static columns known at database definition time.
### Connection to @alkdev/operations
#### B. Native Columns via dbtype (Long-Term, Speculative)
If storage migrates to dbtype element trees for table definitions, the 6 static
metagraph tables (graph_types, node_types, edge_types, graphs, nodes, edges) could
be rendered via the dbtype pipeline: element tree → HostConfig → Drizzle tables.
This would eliminate the manual duplication between `sqlite/` and future `pg/`.
However, dbtype does NOT solve the attribute indexing problem:
- The metagraph's `attributes` column MUST remain JSON because the shape is defined
by runtime schemas (node type definitions), not by static column definitions
- dbtype generates static table schemas; it does not handle dynamic schema-as-data
patterns like the metagraph
- A "call" node's attributes (`requestId`, `status`, `duration`) are not columns
on the `nodes` table — they're values in the `attributes` JSON column, validated
by the corresponding node type's TypeBox schema
#### C. Hybrid: Static Tables via dbtype, Dynamic Attributes Remain JSON
The hybrid approach preserves the metagraph's dynamic schema model while leveraging
dbtype for the static table scaffolding:
1. **Static tables**: dbtype renders the 6 metagraph tables to Drizzle dialects.
This eliminates the SQLite/PG manual duplication for table *structure*.
The `attributes` column is still `text/jsonb` across both dialects.
2. **Dynamic attributes**: Remain JSON. The Module-based node type schemas validate
data at the application layer, not the database layer. This is by design
(ADR-003, ADR-014).
3. **Virtual columns / computed columns**: A post-v1 optimization, not a v1 concern.
Frequently queried attributes could be extracted to indexed columns as a
performance optimization. For example, if `nodes.attributes.status` is a common
filter, a computed column or trigger could copy it to `nodes.status_column` with
an index. This would be a denormalization trade-off (triggers, migration
complexity, dual-write responsibility) and is not designed or planned for v1.
4. **Repository CRUD**: The static table CRUD operations (insert graph type, find
node by key) could be auto-generated like drizzle-graphql or the dbtype
`from-dbtype` adapter. Graph-specific attribute queries remain JSON path.
### Implications for Each Approach
| Concern | Path A (JSON) | Path B (Native) | Path C (Hybrid) |
|---------|---------------|-----------------|------------------|
| Works today | ✅ | ❌ (requires dbtype) | ❌ (requires dbtype) |
| Preserves metagraph pattern | ✅ | ❌ (conflicts with dynamic schemas) | ✅ |
| Eliminates SQLite/PG duplication | ❌ | ✅ | ✅ |
| Indexes on attributes | GIN on PG only | ✅ full native | GIN + virtual columns |
| Repository generation | Hand-write CRUD | Auto-gen from dbtype | Auto-gen for static, JSON path for dynamic |
| Dependency on dbtype | None | Full | Partial (static tables only) |
### Connection to drizzle-graphql
The overview references drizzle-graphql as a pattern for auto-generating a CRUD/query
surface. The dbtype `from-dbtype` adapter is the @alkdev equivalent: it consumes
element trees + Type.Module bundles and produces `OperationSpec[]` for the
operations registry.
The parallel:
| Concern | drizzle-graphql | dbtype from-dbtype |
|---------|----------------|-------------------|
| Input | Drizzle schema (tables + relations) | UJSX element tree + Type.Module |
| Output | GraphQL schema (queries + mutations) | `OperationSpec[]` (CRUD operations) |
| Dialects | SQLite, PG, MySQL | SQLite, PG, MySQL (via HostConfig) |
| Table model | Static columns only | Static columns only |
| Dynamic data (JSON attrs) | Not handled | Not handled |
Neither drizzle-graphql nor dbtype's `from-dbtype` handles dynamic schema-as-data
patterns. The metagraph's JSON attributes require their own query layer, regardless
of whether the static tables are auto-generated. This means the repository layer
for `@alkdev/storage` will always have two parts:
1. **Static table CRUD** — could be auto-generated (by dbtype or hand-written)
2. **Graph data queries** — JSON path queries against the `attributes` column,
validated by the Module schema at the application layer
`@alkdev/operations` is a type-only peer dependency of storage. The
`OperationSpec` type is straightforward. Storage builds the specs; the
consumer wires them into the registry. No circular dependency.
### v1 Decision
For v1, the practical path is **A (JSON path queries) with hand-written CRUD**. This
decision is recorded as [ADR-033](./decisions/033-json-path-queries-for-v1.md). The
hybrid approach (C) remains viable for a future iteration when dbtype reaches
implementation, and it doesn't require any changes to the metagraph data model —
only to how the static table definitions are generated. See OQ-17, OQ-18, OQ-19
in [open-questions.md](./open-questions.md) for the specific long-term questions
that remain open beyond v1.
### Decisions Required
- **OQ-17**: JSON path vs native columns vs hybrid for attribute queries (resolved for v1 — see ADR-033)
- **OQ-18**: Auto-generated vs hand-written CRUD for static tables (resolved for v1 — see ADR-033)
- **OQ-19**: Where the storage-operations bridge package should live (open)
For v1, the practical path is **OperationSpecs with JSON path attribute
queries** (ADR-048, supersedes ADR-033). Spec generation from tables is
straightforward once domain tables exist. The metagraph's generic CRUD
(graphs, nodes, edges) uses JSON attributes; domain-specific CRUD uses
native columns. Both produce OperationSpecs that the hub registers in the
same operations registry.
## Constraints on Current Design
@@ -382,18 +329,28 @@ design in [metagraph-module.md](./metagraph-module.md):
opaque builder objects or Drizzle column definitions. See
[schema-evolution.md](./schema-evolution.md).
6. **OperationSpec output is consumer-agnostic** — storage defines
`OperationSpec[]` from table definitions. The consumer (hub/spoke) decides
how to register handlers. Storage does not execute queries or depend on
the operations runtime.
7. **The folded utils are SQLite-only**`src/sqlite/utils/` contains only
SQLite column→TypeBox mappings. If a new database host is added later, the
utils would need the corresponding dialect mappings. dbtype's Phase 1
(UJSX→HostConfig) would be the mechanism for multi-dialect support.
## References
- ujsx pointer system: `/workspace/@alkdev/ujsx/src/core/pointer.ts`
- ujsx HostConfig adapter: `/workspace/@alkdev/ujsx/src/host/config.ts`
- dbtype architecture: `/workspace/@alkdev/dbtype/docs/architecture/README.md`
- dbtype architecture: `/workspace/@alkdev/dbtype/docs/architecture/README.md` (Phase 0 source folded into storage)
- dbtype elements: `/workspace/@alkdev/dbtype/docs/architecture/elements.md`
- dbtype module: `/workspace/@alkdev/dbtype/docs/architecture/module.md`
- dbtype repo adapter: `/workspace/@alkdev/dbtype/docs/architecture/repo-adapter.md`
- drizzle-graphql (reference for CRUD generation pattern): `/workspace/drizzle-graphql/`
- Operations registry: `/workspace/@alkdev/operations/docs/architecture/README.md`
- JPATH Module (JSONPath as TypeBox Module): `/workspace/research/typebox_research/ujsx/jpath.gen.ts`
- jsonpathly source: `/workspace/jsonpathly/`
- Module evolution spec: [metagraph-module.md](./metagraph-module.md)
- Schema evolution spec: [schema-evolution.md](./schema-evolution.md)
- ADR-033: JSON path queries and hand-written CRUD for v1
- ADR-046: Fold drizzlebox as utils (supersedes ADR-033)
- ADR-048: OperationSpecs as repository surface

View File

@@ -1,12 +1,14 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Honker Integration
How @alkdev/storage integrates with Honker for SQLite database operations,
transactional pub/sub, durable event streams, and task queues.
transactional pub/sub, durable event streams, and task queues. Includes the
HonkerEventTarget adapter that bridges `@alkdev/pubsub`'s `TypedEventTarget`
to Honker primitives.
## Purpose
@@ -118,10 +120,134 @@ db.transaction((tx) => {
| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | Small Rust PR to honker-node |
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | POC 1 confirmed `tx.query("SELECT last_insert_rowid() as id")` works |
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | Stable across minor versions |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | POC 1 confirmed all classes accessible and extendable |
| `:memory:` databases don't work | Reader pool gets separate in-memory DB | Always use file-based paths. Tests use temp files. |
### POC-Validated Adapter Architecture
POC 1 (2026-06-01) confirmed the adapter is buildable. Key findings:
- `SQLiteSession`, `SQLitePreparedQuery`, `SQLiteTransaction` are all
accessible from `drizzle-orm/sqlite-core/session` and extendable.
- `BaseSQLiteDatabase` is accessible from `drizzle-orm/sqlite-core/db`.
- `LibSQLSession` in `drizzle-orm/libsql/session` is the reference
implementation to follow.
- Honker `query()` returns `{ columnName: value }` object rows, compatible
with Drizzle's `mapResultRow()`.
- `last_insert_rowid()` accessible via `tx.query("SELECT last_insert_rowid() as id")`.
- `tx.execute()` returns a number (affected rows count).
- JSON-mode columns need manual `JSON.parse()` in the adapter.
## HonkerEventTarget
The `HonkerEventTarget` adapts `@alkdev/pubsub`'s `TypedEventTarget` interface
to Honker's `notify`/`listen` (ephemeral) and `stream`/`subscribe` (durable)
primitives. It lives in `src/sqlite/event-target.ts`.
### Interface
```ts
import type { TypedEventTarget } from "@alkdev/pubsub";
interface HonkerEventTargetOptions {
db: Database;
mode: "ephemeral" | "durable";
streamName?: string;
consumerName?: string;
prefix?: string;
}
function createHonkerEventTarget<TEvent>(
options: HonkerEventTargetOptions
): TypedEventTarget<TEvent> & { close(): void };
```
### Ephemeral Mode
Maps to Honker's `notify()`/`listen()`:
| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.listen("topic:id")` → start async consumer loop |
| `dispatchEvent(event)` | `db.notify(event.type, event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the listener when no callbacks remain |
| `close()` | Close all active listeners |
POC 2 confirmed: same-process `notify``listen` works, multiple concurrent
listeners on different channels work, cross-channel isolation is correct.
POC 4 confirmed: `tx.notify()` within a Drizzle transaction only fires the
notification on `tx.commit()`. On `tx.rollback()`, the notification is
suppressed. This enables transactional outbox semantics for ephemeral events.
### Durable Mode
Maps to Honker's `Stream.publish()`/`Stream.subscribe()`:
| pubsub operation | Honker operation |
|-------------------|------------------|
| `addEventListener("topic:id", callback)` | `db.stream(name).subscribe(consumer)` → start async consumer loop |
| `dispatchEvent(event)` | `db.stream(name).publish(event.detail)` |
| `removeEventListener("topic:id", callback)` | Close the subscription when no callbacks remain |
| `close()` | Close all active subscriptions, save offsets |
POC 3 confirmed: `Stream.publish()`/`Stream.subscribe()` work, consumer
offsets are tracked and persisted, `publishTx(tx, payload)` works within
transactions, and concurrent stream + listener operation works.
Durable mode provides crash recovery — consumers resume from their last saved
offset after restart. Consumer names must be stable across restarts (not
PID-based).
### Topic Routing
pubsub uses `topic:id` composite topics. Honker channels and streams are flat
strings. The mapping:
- **Ephemeral**: Each unique `topic:id` gets its own `db.listen()` call.
For high-cardinality topics (many request IDs), consider topic prefix
matching with client-side filtering instead.
- **Durable**: Streams are name-keyed, not topic-keyed. A single stream
carries all events for a domain. Client-side filtering dispatches only
to matching listeners.
Suggested split:
| Event category | Mode | Reason |
|---------------|------|--------|
| Call protocol events (`call.requested`, `call.responded`, etc.) | Durable stream | Crash recovery, audit trail, flowgraph replay |
| Cache invalidation signals | Ephemeral | Fire-and-forget, loss acceptable |
| UI/dashboard push | Ephemeral | Low latency, loss acceptable |
| Schema migration jobs | Queue (not pubsub) | At-least-once processing |
### Latency Consideration
POC 2 measured ~17ms median latency for Honker `notify``listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.
### Hub-Spoke Event Routing
```
Hub (system.db + tenant-{orgId}.db)
├── HonkerEventTarget (durable: call-protocol stream)
├── WebSocketServerEventTarget (spoke fan-out)
└── In-process EventTarget (local subscribers)
Spoke (tenant-{orgId}.db)
├── HonkerEventTarget (ephemeral: local channels)
├── WebSocketClientEventTarget (hub connection)
└── In-process EventTarget (local subscribers)
```
Both hub and spoke use the same `createPubSub({ eventTarget })` call.
Different event target instances determine the routing. No code changes
between hub and spoke — only configuration.
## Event-Driven Patterns
@@ -338,7 +464,7 @@ would require a coordinator — the hub fills this role.
- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
- TypeBox schemas from drizzlebox
- TypeBox schemas from local utils (folded from drizzlebox)
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)
@@ -359,13 +485,18 @@ would require a coordinator — the hub fills this role.
|-----|----------|--------|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils/ | Accepted |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter for pubsub | Accepted |
## Open Questions
- **OQ-26**: Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?
- **OQ-26**: ~~Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?~~ Resolved: Yes. HonkerEventTarget (ADR-047) provides the adapter. Redis still needed for multi-node.
- **OQ-27**: How are schema migrations applied across all tenant DBs?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?
- **OQ-30**: Composite event target pattern — how should an in-process EventTarget and HonkerEventTarget be combined for single-node hub deployments? POC 2 showed ~17ms Honker latency vs sub-ms in-process. Design needed.
- **OQ-31**: Consumer naming convention for durable subscriptions — must be stable across hub restarts (not PID-based).
- **OQ-32**: Drizzle Kit migration compatibility — does `drizzle-kit push`/`drizzle-kit generate` work with the custom Honker adapter, or do we need a custom migration runner?
## References

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# Open Questions Tracker
@@ -12,8 +12,8 @@ architecture documents, organized by theme.
| Status | Count |
|--------|-------|
| Open | 9 |
| Resolved (this revision) | 15 |
| Open | 10 |
| Resolved (this revision) | 18 |
| Previously resolved | 11 |
**Open questions requiring decisions:**
@@ -23,10 +23,13 @@ architecture documents, organized by theme.
- **OQ-11** (auto-migrate vs explicit) — conditional on OQ-10
- **OQ-12** (schema evolution vs event-sourced replay) — post-v1 concern
- **OQ-13** (schema evolution events in event stream) — post-v1
- **OQ-26** (Honker replaces @alkdev/pubsub Redis transport) — integration design
- **OQ-25** (scope string semantics) — evaluator concern
- **OQ-27** (tenant DB schema migration strategy) — multi-tenant operations
- **OQ-28** (cross-tenant delegation with separate DBs) — cross-DB coordination
- **OQ-29** (standalone drizzle-honker npm package) — community value
- **OQ-30** (composite event target for single-node hub) — latency optimization
- **OQ-31** (consumer naming for durable subscriptions) — restart stability
- **OQ-32** (Drizzle Kit migration compatibility) — custom adapter
## Theme 1: Package Boundaries and Dependencies
@@ -152,21 +155,23 @@ architecture documents, organized by theme.
### OQ-17: Attribute queries — JSON path, native columns, or dbtype-generated?
- **Status**: resolved (v1)
- **Status**: resolved (updated)
- **Priority**: high
- **Resolution**: JSON path for v1. ADR-033. Long-term hybrid still open but less pressing without PG dual maintenance.
- **Resolution**: JSON path for metagraph attributes (dynamic schemas). Native columns for domain-specific tables (CallGraph, etc.). OperationSpecs provide the CRUD contract for both patterns. ADR-048 supersedes ADR-033.
- **Cross-references**: ADR-048
### OQ-18: Auto-generated vs hand-written CRUD?
- **Status**: resolved (v1)
- **Status**: resolved (updated)
- **Priority**: medium
- **Resolution**: Hand-write for v1. ADR-033.
- **Resolution**: Not hand-written CRUD — OperationSpecs. Storage outputs `OperationSpec[]` from table definitions. The consumer (hub/spoke) registers handlers. ADR-048.
- **Cross-references**: ADR-048
### OQ-19: Storage-operations bridge package location?
- **Status**: resolved
- **Status**: resolved (updated)
- **Priority**: medium
- **Resolution**: Less pressing now that Honker replaces the Redis transport. Can live in the hub for v1. Revisit if an adapter package becomes valuable.
- **Resolution**: No separate bridge package needed. Storage outputs `OperationSpec[]` as part of its table definitions (type-only peer dep on `@alkdev/operations`). The consumer wires specs into the registry. ADR-048.
## Theme 7: Access Control
@@ -220,9 +225,10 @@ architecture documents, organized by theme.
### OQ-26: Can Honker fully replace @alkdev/pubsub's Redis transport for single-node deployments?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Status**: resolved
- **Priority**: high
- **Notes**: Honker's `notify()`/`listen()` and `stream()`/`subscribe()` provide the pub/sub primitives. The question is whether `@alkdev/pubsub`'s `TypedEventTarget` interface can be backed by Honker instead of Redis, and whether multi-node deployments still need Redis for internode communication.
- **Resolution**: Yes for single-node. The `HonkerEventTarget` adapter (ADR-047) implements pubsub's `TypedEventTarget` on Honker's `notify`/`listen` and `stream`/`subscribe`. POC 2-4 validated: same-process pub/sub works, transactional semantics hold, concurrent listeners work. Redis still needed for multi-node deployments. In-process EventTarget provides sub-ms latency for hot paths (vs ~17ms for Honker round-trip).
- **Cross-references**: ADR-047
### OQ-27: How are schema migrations applied across all tenant DBs?
@@ -245,6 +251,27 @@ architecture documents, organized by theme.
- **Priority**: low
- **Notes**: The adapter is ~100 lines and useful to anyone combining Drizzle with Honker. Publishing as `drizzle-honker` would benefit the community. Decision: start inside `@alkdev/storage`, extract later if there's demand.
### OQ-30: Composite event target for single-node hub deployments?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: POC 2 showed ~17ms median latency for Honker notify→listen vs sub-ms for in-process EventTarget. For single-node hubs, a composite that dispatches to both (in-process for speed, Honker for durability/cross-process) would be the ideal default. Design needed.
### OQ-31: Consumer naming convention for durable stream subscriptions?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: Honker's `stream.subscribe(consumer)` requires a consumer name for offset tracking. The name must be stable across hub restarts (PID-based names don't survive restart). Need a convention: `{service}:{host}` or a configurable consumer group ID.
### OQ-32: Drizzle Kit migration compatibility with Honker adapter?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: medium
- **Notes**: Drizzle Kit supports SQLite migrations but expects `better-sqlite3` or `libsql`. Need to verify `drizzle-kit push`/`drizzle-kit generate` works with the custom Honker adapter, or whether we need a custom migration runner.
## ADR Impact
| ADR | Resolves | Informs |
@@ -255,7 +282,7 @@ architecture documents, organized by theme.
| ADR-020 | OQ-24 | |
| ADR-023 | OQ-14 | |
| ADR-026 | OQ-15 | |
| ADR-033 | OQ-04, OQ-16, OQ-17, OQ-18 | |
| ADR-033 | OQ-04, OQ-16, OQ-17, OQ-18 | OQ-17 (superseded by ADR-048) |
| ADR-034 | OQ-03, OQ-21 | OQ-25 |
| ADR-035 | OQ-03 | |
| ADR-038 | OQ-04 (moot) | OQ-17 (less pressure) |
@@ -263,5 +290,8 @@ architecture documents, organized by theme.
| ADR-041 | OQ-24 | |
| ADR-042 | | OQ-24 |
| ADR-043 | | |
| ADR-044 | OQ-19 (less pressure) | |
| ADR-045 | OQ-23 | OQ-20 |
| ADR-044 | OQ-19 (less pressure) | OQ-29, OQ-32 |
| ADR-045 | OQ-23 | OQ-20 |
| ADR-046 | | OQ-17 |
| ADR-047 | OQ-26 | OQ-30 |
| ADR-048 | OQ-17 (updated), OQ-18 (updated), OQ-19 (updated) | |

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# @alkdev/storage — Overview
@@ -24,6 +24,12 @@ organizations, api_keys, audit_logs) for multi-tenant authentication and
authorization, and integrates with **Honker** for transactional pub/sub, event
streams, and task queues within the same SQLite database.
Storage also provides a **HonkerEventTarget** adapter that bridges
`@alkdev/pubsub`'s `TypedEventTarget` interface to Honker's `notify`/`listen`
and `stream`/`subscribe` primitives, and **OperationSpec** generation from
table definitions so downstream hubs/spokes can register CRUD operations
directly into the `@alkdev/operations` registry.
## Architecture
```
@@ -45,9 +51,11 @@ streams, and task queues within the same SQLite database.
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
│ │ └── index.ts → barrel re-export
│ ├── utils/ → createSelectSchema, createInsertSchema, column mappings (folded from @alkdev/dbtype)
│ ├── relations.ts → drizzle relational mappings
│ ├── schema.ts → barrel re-export
│ ├── adapter.ts → Drizzle-Honker session adapter
│ ├── event-target.ts → HonkerEventTarget (pubsub TypedEventTarget on Honker)
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
└── test/
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
@@ -59,7 +67,7 @@ streams, and task queues within the same SQLite database.
| ------------------------ | --------------------------------------- | --------------------------------------- |
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter | `@alkdev/drizzlebox`, `drizzle-orm`, `@russellthehipp/honker-node` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter, event-target, utils | `drizzle-orm`, `@russellthehippo/honker-node`, `@alkdev/pubsub` (peer) |
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
database host.
@@ -117,8 +125,10 @@ fixed — changes require a version bump and migration (ADR-029).
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
| **HonkerEventTarget** | Adapter that implements `@alkdev/pubsub`'s `TypedEventTarget` interface on Honker's `notify`/`listen` and `stream`/`subscribe` primitives. Bridges pubsub to Honker for single-node and cross-process scenarios. |
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. |
| **Repository surface** | Not hand-written CRUD functions. Storage outputs `OperationSpec[]` — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers handlers, and the operations runtime handles execution. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the bridge functions (at mutation time), NOT in the database. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
## Design Decisions
@@ -131,7 +141,10 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via drizzlebox | Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via local utils | Drizzle tables are single source of truth; `src/sqlite/utils/` generates TypeBox schemas (folded from @alkdev/drizzlebox) |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils | SQLite-only column mappings and schema generation co-located with tables |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter | pubsub TypedEventTarget on Honker notify/listen and stream/subscribe |
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repository surface | Storage outputs OperationSpec[] not hand-written CRUD |
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
@@ -150,12 +163,14 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| Package | Purpose | Layer |
| -------------------- | ------------------------------------ | ------------------------ |
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
| `@alkdev/drizzlebox` | Generate TypeBox from Drizzle tables | sqlite/ |
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
| `@alkdev/pubsub` | `TypedEventTarget` interface (peer) | sqlite/event-target.ts |
`@alkdev/typebox` and `@alkdev/drizzlebox` are npm packages (not yet on JSR).
JSR handles npm dependencies natively.
`@alkdev/typebox` is an npm package (not yet on JSR). JSR handles npm
dependencies natively. `@alkdev/pubsub` is a peer dependency — only needed
when using `HonkerEventTarget`. The `graphs/` module has zero dependencies
beyond `@alkdev/typebox`.
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
ecosystem references describe consumer-side data shapes and integration
@@ -169,22 +184,26 @@ patterns, not import dependencies.
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Reference graph type Modules (CallGraph, SecretGraph)
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
- SQLite host: 6 metagraph tables + `actors` placeholder + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
- SQLite host: 6 metagraph tables (flat structure, pending reorganize) + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (current: `@alkdev/drizzlebox`; pending fold to `src/sqlite/utils/`, ADR-046)
- Reference module tests (bridge functions, validation, Module composition)
### Not Yet Implemented
| Gap | Priority | Notes |
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| Drizzle-Honker session adapter | High | ~100-line adapter wrapping `HonkerSQLiteSession` + `HonkerPreparedQuery`. ADR-044. |
| Drizzle-Honker session adapter | High | `HonkerSQLiteSession` + `HonkerPreparedQuery` + `HonkerSQLiteTransaction`. POC validated. ADR-044. |
| Fold dbtype Phase 0 → `src/sqlite/utils/` | High | SQLite-only column mappings + createSelectSchema/createInsertSchema. ADR-046. |
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
| Graph type `scope` column | High | ADR-043. |
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. |
| Repository/CRUD layer | Medium | ⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges. |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. Accept Honker client. |
| Table restructure into subdirectories | High | `tables/metagraph/`, `tables/identity/`, update relative imports. |
| HonkerEventTarget adapter | High | pubsub `TypedEventTarget` on Honker primitives. POC validated. ADR-047. |
| OperationSpec generation from tables | Medium | `OperationSpec[]` per table for CRUD operations. ADR-048. |
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
| Domain-specific native-column tables | Low | CallGraph, SecretGraph, etc. with native columns alongside JSON attributes. |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
@@ -204,10 +223,11 @@ other way.
@alkdev/flowgraph ← call graph schema, operation graph, workflow templates
↑ (depends on: @alkdev/operations [peer], @alkdev/typebox)
@alkdev/taskgraph ← task dependency graph schema, cost-benefit analysis
(depends on: @alkdev/typebox)
(depends on: @alkdev/typebox)
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
(depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)
(depends on: @alkdev/typebox, drizzle-orm, honker-node)
(peer dep: @alkdev/pubsub for TypedEventTarget type)
↑ ↑
| |
@@ -218,7 +238,9 @@ Hub / Spoke Any consumer that needs
### Event-Driven Architecture with Honker
The @alkdev platform is event-driven. Honker provides the transport mechanism
within each SQLite database:
within each SQLite database. Storage provides the `HonkerEventTarget` adapter
so consumers can use `@alkdev/pubsub`'s `TypedEventTarget` interface regardless
of whether events route in-process, through Honker, or over WebSocket.
| Concern | Honker Primitive | Example |
|---------|------------------|---------|
@@ -230,7 +252,17 @@ within each SQLite database:
A single Drizzle transaction via the Honker adapter can insert graph data AND
publish a notification AND enqueue a side-effect job — all committing atomically.
No dual-write problem between data and events.
No dual-write problem between data and events. POC 4 confirmed: `tx.notify()`
only fires on commit; on rollback, both the data write and the notification are
suppressed.
### Latency Consideration
POC 2 measured ~17ms median latency for Honker `notify``listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.
### What Comes from Where
@@ -241,26 +273,32 @@ No dual-write problem between data and events.
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
| Event transport | Honker (within storage) | Replaces `@alkdev/pubsub`'s Redis transport for single-node deployments |
| Event transport | Honker (within storage) | `HonkerEventTarget` bridges pubsub `TypedEventTarget` to Honker primitives |
| Pubsub interface | `@alkdev/pubsub` | Storage provides `HonkerEventTarget` adapter; does not replace pubsub |
### Repository Layer Bridging Pattern
### Repository Surface as OperationSpecs
The repository layer in `@alkdev/storage` provides typed CRUD — no
`@alkdev/operations` dependency. A consumer-side bridging module wires CRUD
functions into the operations registry when needed.
Storage does not ship a "repository layer" of hand-written CRUD functions.
Instead, it outputs `OperationSpec[]` — flat arrays describing CRUD and query
operations for each table. The consumer (hub/spoke) imports these specs,
registers them in the `@alkdev/operations` registry along with handlers, and
the operations runtime handles execution, call protocol, and subscriptions.
Storage depends on `@alkdev/operations` only as a type-level peer dependency
(for the `OperationSpec` type). No circular dependency.
```
@alkdev/storage → defines types + tables + CRUD (no operations dependency)
@alkdev/storage → defines types + tables + OperationSpec[] (type-only operations dep)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas
Consumer (hub / spoke) → imports both, registers specs + handlers
```
### Avoiding Circular Dependencies
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
other directly. Storage defines the schema types and database tables; operations
defines the call protocol and execution model. The consumer imports both and
bridges them.
other at runtime. Storage defines the schema types, database tables, and
operation contract definitions; operations defines the call protocol and
execution model. The consumer imports both and bridges them.
## Open Questions
@@ -270,7 +308,7 @@ questions affecting this package:
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (open)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (resolved: yes — HonkerEventTarget adapter, POC validated. Redis still needed for multi-node. See ADR-047.)
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)

View File

@@ -1,13 +1,14 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# SQLite Host
The SQLite database host for `@alkdev/storage`. Uses Drizzle ORM with Honker
for database operations, pub/sub, event streams, and task queues. TypeBox
schemas are auto-generated from Drizzle table definitions via `@alkdev/drizzlebox`.
schemas are generated from Drizzle table definitions via `src/sqlite/utils/`
(folded from `@alkdev/dbtype`/`@alkdev/drizzlebox`, ADR-046).
## Overview
@@ -18,11 +19,13 @@ The SQLite host provides:
2. **Identity tables** — accounts, organizations, organization_members, api_keys,
audit_logs for multi-tenant authentication and authorization (ADR-041)
3. **Drizzle relations** for the relational query API
4. **TypeBox schemas** auto-generated from Drizzle tables (select/insert
validation) via `@alkdev/drizzlebox`
4. **TypeBox schemas** generated from Drizzle tables (select/insert
validation) via `src/sqlite/utils/` (folded from @alkdev/drizzlebox, ADR-046)
5. **Drizzle-Honker adapter** — thin session adapter for Honker integration
(ADR-044)
6. **Client factories**`createSystemDatabase(client)` and
(ADR-044, POC validated)
6. **HonkerEventTarget** — pubsub `TypedEventTarget` on Honker primitives
(ADR-047, POC validated)
7. **Client factories**`createSystemDatabase(client)` and
`createTenantDatabase(client)` for the system/tenant DB model (ADR-040)
## Package Structure
@@ -47,10 +50,17 @@ src/sqlite/
│ │ ├── edges.ts # edges table + select/insert schemas
│ │ └── index.ts # barrel re-export
│ └── index.ts # barrel re-export
├── relations.ts # Drizzle relational mappings
├── utils/ # folded from @alkdev/dbtype Phase 0 (ADR-046)
│ ├── schema.ts # createSelectSchema, createInsertSchema, createUpdateSchema
│ ├── column.ts # Column→TypeBox mappings (SQLite-only dispatch)
│ ├── types.ts # Public + internal TypeScript interfaces
│ ├── constants.ts # Integer range constants
│ └── utils.ts # isColumnType, isWithEnum, type helpers
├── relations.ts # Drizzle relational mappings
├── adapter.ts # Drizzle-Honker session adapter
├── schema.ts # re-exports all tables + relations
── client.ts # createSystemDatabase(), createTenantDatabase()
├── event-target.ts # HonkerEventTarget (pubsub TypedEventTarget on Honker)
── schema.ts # re-exports all tables + relations
└── client.ts # createSystemDatabase(), createTenantDatabase()
```
## Common Columns
@@ -339,8 +349,11 @@ db.transaction((tx) => {
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage | accounts, organizations, api_keys, audit_logs |
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graphs | `ownerId`, `projectId` on `graphs` table |
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope on `graph_types` |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker adapter | ~100-line session adapter |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker adapter | ~100-line session adapter, POC validated |
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | org_members authoritative | SQL table is source of truth; BelongsToEdge is derived |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold drizzlebox as utils | SQLite-only column mappings in src/sqlite/utils/ |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget | pubsub TypedEventTarget on Honker |
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repo surface | Table-defined operation contracts |
| [019](decisions/019-json-text-for-schema-columns.md) | JSON text for schema columns | SQLite uses `text` with JSON mode |
| [020](decisions/020-no-nodetypeid-on-nodes.md) | No nodeTypeId on nodes | Node type enforced at application layer |
| [022](decisions/022-composite-fks-for-node-references.md) | Composite FKs for node refs | Edges reference `(graphId, sourceNodeKey)` |