alkdev/storage
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pivot: fold drizzlebox as utils, HonkerEventTarget, OperationSpecs as repo surface

Browse Source 
- Update architecture docs to reflect pivot from @libsql/client to Honker
- Fold @alkdev/drizzlebox Phase 0 into src/sqlite/utils/ (ADR-046)
- Add HonkerEventTarget adapter for pubsub TypedEventTarget (ADR-047)
- Replace hand-written CRUD with OperationSpec generation (ADR-048)
- Resolved OQ-26: Honker replaces Redis for single-node pub/sub (POC validated)
- Updated OQ-17, OQ-18, OQ-19 for OperationSpec repository surface
- Added OQ-30 (composite event target), OQ-31 (consumer naming), OQ-32 (Drizzle Kit)
- POC results: adapter buildable, same-process pub/sub works, transactional
  outbox semantics confirmed, concurrent listeners/streams work
- Research doc at docs/research/pivot-honker-sqlite-adapter.md
This commit is contained in:
glm-5.1
2026-06-01 16:31:40 +00:00
parent 6aa2fcc6ff
commit 412ad98f11
10 changed files with 1342 additions and 230 deletions
Download Patch File Download Diff File Expand all files Collapse all files

92
docs/architecture/overview.md
View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-31
last_updated: 2026-06-01
---
# @alkdev/storage — Overview
@@ -24,6 +24,12 @@ organizations, api_keys, audit_logs) for multi-tenant authentication and
authorization, and integrates with **Honker** for transactional pub/sub, event
streams, and task queues within the same SQLite database.
Storage also provides a **HonkerEventTarget** adapter that bridges
`@alkdev/pubsub`'s `TypedEventTarget` interface to Honker's `notify`/`listen`
and `stream`/`subscribe` primitives, and **OperationSpec** generation from
table definitions so downstream hubs/spokes can register CRUD operations
directly into the `@alkdev/operations` registry.
## Architecture
```
@@ -45,9 +51,11 @@ streams, and task queues within the same SQLite database.
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
│ │ └── index.ts → barrel re-export
│ ├── utils/ → createSelectSchema, createInsertSchema, column mappings (folded from @alkdev/dbtype)
│ ├── relations.ts → drizzle relational mappings
│ ├── schema.ts → barrel re-export
│ ├── adapter.ts → Drizzle-Honker session adapter
│ ├── event-target.ts → HonkerEventTarget (pubsub TypedEventTarget on Honker)
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
└── test/
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
@@ -59,7 +67,7 @@ streams, and task queues within the same SQLite database.
| ------------------------ | --------------------------------------- | --------------------------------------- |
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter | `@alkdev/drizzlebox`, `drizzle-orm`, `@russellthehipp/honker-node` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter, event-target, utils | `drizzle-orm`, `@russellthehippo/honker-node`, `@alkdev/pubsub` (peer) |
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
database host.
@@ -117,8 +125,10 @@ fixed — changes require a version bump and migration (ADR-029).
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
| **HonkerEventTarget** | Adapter that implements `@alkdev/pubsub`'s `TypedEventTarget` interface on Honker's `notify`/`listen` and `stream`/`subscribe` primitives. Bridges pubsub to Honker for single-node and cross-process scenarios. |
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. |
| **Repository surface** | Not hand-written CRUD functions. Storage outputs `OperationSpec[]` — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers handlers, and the operations runtime handles execution. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the bridge functions (at mutation time), NOT in the database. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
## Design Decisions
@@ -131,7 +141,10 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via drizzlebox | Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via local utils | Drizzle tables are single source of truth; `src/sqlite/utils/` generates TypeBox schemas (folded from @alkdev/drizzlebox) |
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils | SQLite-only column mappings and schema generation co-located with tables |
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter | pubsub TypedEventTarget on Honker notify/listen and stream/subscribe |
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repository surface | Storage outputs OperationSpec[] not hand-written CRUD |
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
@@ -150,12 +163,14 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| Package | Purpose | Layer |
| -------------------- | ------------------------------------ | ------------------------ |
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
| `@alkdev/drizzlebox` | Generate TypeBox from Drizzle tables | sqlite/ |
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
| `@alkdev/pubsub` | `TypedEventTarget` interface (peer) | sqlite/event-target.ts |
`@alkdev/typebox` and `@alkdev/drizzlebox` are npm packages (not yet on JSR).
JSR handles npm dependencies natively.
`@alkdev/typebox` is an npm package (not yet on JSR). JSR handles npm
dependencies natively. `@alkdev/pubsub` is a peer dependency — only needed
when using `HonkerEventTarget`. The `graphs/` module has zero dependencies
beyond `@alkdev/typebox`.
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
ecosystem references describe consumer-side data shapes and integration
@@ -169,22 +184,26 @@ patterns, not import dependencies.
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Reference graph type Modules (CallGraph, SecretGraph)
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
- SQLite host: 6 metagraph tables + `actors` placeholder + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
- SQLite host: 6 metagraph tables (flat structure, pending reorganize) + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (current: `@alkdev/drizzlebox`; pending fold to `src/sqlite/utils/`, ADR-046)
- Reference module tests (bridge functions, validation, Module composition)
### Not Yet Implemented
| Gap | Priority | Notes |
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| Drizzle-Honker session adapter | High | ~100-line adapter wrapping `HonkerSQLiteSession` + `HonkerPreparedQuery`. ADR-044. |
| Drizzle-Honker session adapter | High | `HonkerSQLiteSession` + `HonkerPreparedQuery` + `HonkerSQLiteTransaction`. POC validated. ADR-044. |
| Fold dbtype Phase 0 → `src/sqlite/utils/` | High | SQLite-only column mappings + createSelectSchema/createInsertSchema. ADR-046. |
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
| Graph type `scope` column | High | ADR-043. |
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. |
| Repository/CRUD layer | Medium | ⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges. |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. Accept Honker client. |
| Table restructure into subdirectories | High | `tables/metagraph/`, `tables/identity/`, update relative imports. |
| HonkerEventTarget adapter | High | pubsub `TypedEventTarget` on Honker primitives. POC validated. ADR-047. |
| OperationSpec generation from tables | Medium | `OperationSpec[]` per table for CRUD operations. ADR-048. |
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
| Domain-specific native-column tables | Low | CallGraph, SecretGraph, etc. with native columns alongside JSON attributes. |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
@@ -204,10 +223,11 @@ other way.
@alkdev/flowgraph ← call graph schema, operation graph, workflow templates
↑ (depends on: @alkdev/operations [peer], @alkdev/typebox)
@alkdev/taskgraph ← task dependency graph schema, cost-benefit analysis
(depends on: @alkdev/typebox)
(depends on: @alkdev/typebox)
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
(depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)
(depends on: @alkdev/typebox, drizzle-orm, honker-node)
(peer dep: @alkdev/pubsub for TypedEventTarget type)
↑ ↑
| |
@@ -218,7 +238,9 @@ Hub / Spoke Any consumer that needs
### Event-Driven Architecture with Honker
The @alkdev platform is event-driven. Honker provides the transport mechanism
within each SQLite database:
within each SQLite database. Storage provides the `HonkerEventTarget` adapter
so consumers can use `@alkdev/pubsub`'s `TypedEventTarget` interface regardless
of whether events route in-process, through Honker, or over WebSocket.
| Concern | Honker Primitive | Example |
|---------|------------------|---------|
@@ -230,7 +252,17 @@ within each SQLite database:
A single Drizzle transaction via the Honker adapter can insert graph data AND
publish a notification AND enqueue a side-effect job — all committing atomically.
No dual-write problem between data and events.
No dual-write problem between data and events. POC 4 confirmed: `tx.notify()`
only fires on commit; on rollback, both the data write and the notification are
suppressed.
### Latency Consideration
POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
single process. For hot-path call protocol request/response where sub-ms
latency matters, pair the Honker event target with an in-process `EventTarget`
(pubsub's default). A composite pattern (dispatch to both) provides both
in-process speed and Honker durability/cross-process coordination.
### What Comes from Where
@@ -241,26 +273,32 @@ No dual-write problem between data and events.
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
| Event transport | Honker (within storage) | Replaces `@alkdev/pubsub`'s Redis transport for single-node deployments |
| Event transport | Honker (within storage) | `HonkerEventTarget` bridges pubsub `TypedEventTarget` to Honker primitives |
| Pubsub interface | `@alkdev/pubsub` | Storage provides `HonkerEventTarget` adapter; does not replace pubsub |
### Repository Layer Bridging Pattern
### Repository Surface as OperationSpecs
The repository layer in `@alkdev/storage` provides typed CRUD — no
`@alkdev/operations` dependency. A consumer-side bridging module wires CRUD
functions into the operations registry when needed.
Storage does not ship a "repository layer" of hand-written CRUD functions.
Instead, it outputs `OperationSpec[]` — flat arrays describing CRUD and query
operations for each table. The consumer (hub/spoke) imports these specs,
registers them in the `@alkdev/operations` registry along with handlers, and
the operations runtime handles execution, call protocol, and subscriptions.
Storage depends on `@alkdev/operations` only as a type-level peer dependency
(for the `OperationSpec` type). No circular dependency.
```
@alkdev/storage → defines types + tables + CRUD (no operations dependency)
@alkdev/storage → defines types + tables + OperationSpec[] (type-only operations dep)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas
Consumer (hub / spoke) → imports both, registers specs + handlers
```
### Avoiding Circular Dependencies
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
other directly. Storage defines the schema types and database tables; operations
defines the call protocol and execution model. The consumer imports both and
bridges them.
other at runtime. Storage defines the schema types, database tables, and
operation contract definitions; operations defines the call protocol and
execution model. The consumer imports both and bridges them.
## Open Questions
@@ -270,7 +308,7 @@ questions affecting this package:
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (open)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (resolved: yes — HonkerEventTarget adapter, POC validated. Redis still needed for multi-node. See ADR-047.)
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)