glm-5.1
412ad98f11
- Update architecture docs to reflect pivot from @libsql/client to Honker - Fold @alkdev/drizzlebox Phase 0 into src/sqlite/utils/ (ADR-046) - Add HonkerEventTarget adapter for pubsub TypedEventTarget (ADR-047) - Replace hand-written CRUD with OperationSpec generation (ADR-048) - Resolved OQ-26: Honker replaces Redis for single-node pub/sub (POC validated) - Updated OQ-17, OQ-18, OQ-19 for OperationSpec repository surface - Added OQ-30 (composite event target), OQ-31 (consumer naming), OQ-32 (Drizzle Kit) - POC results: adapter buildable, same-process pub/sub works, transactional outbox semantics confirmed, concurrent listeners/streams work - Research doc at docs/research/pivot-honker-sqlite-adapter.md
327 lines
22 KiB
Markdown
327 lines
22 KiB
Markdown
---
|
|
status: draft
|
|
last_updated: 2026-06-01
|
|
---
|
|
|
|
# @alkdev/storage — Overview
|
|
|
|
Typed graph storage with SQLite via Honker. Deno-first, published via JSR.
|
|
|
|
## Purpose
|
|
|
|
`@alkdev/storage` provides a **metagraph** storage model: graph types define
|
|
schemas, node types define data shapes within those graphs, and edge types
|
|
define typed relationships. Instances of these type definitions become actual
|
|
graphs populated with nodes and edges.
|
|
|
|
This pattern replaces domain-specific table proliferation with a small number of
|
|
general-purpose tables that can model anything — call graphs, ACL rules, task
|
|
dependencies, encrypted secrets, session trees, operation registries — while
|
|
enforcing schema integrity through TypeBox validation.
|
|
|
|
The package also provides **identity infrastructure** tables (accounts,
|
|
organizations, api_keys, audit_logs) for multi-tenant authentication and
|
|
authorization, and integrates with **Honker** for transactional pub/sub, event
|
|
streams, and task queues within the same SQLite database.
|
|
|
|
Storage also provides a **HonkerEventTarget** adapter that bridges
|
|
`@alkdev/pubsub`'s `TypedEventTarget` interface to Honker's `notify`/`listen`
|
|
and `stream`/`subscribe` primitives, and **OperationSpec** generation from
|
|
table definitions so downstream hubs/spokes can register CRUD operations
|
|
directly into the `@alkdev/operations` registry.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
@alkdev/storage/
|
|
├── mod.ts → re-exports graphs/ (zero db deps)
|
|
├── src/
|
|
│ ├── graphs/ → Metagraph Module, bridge functions, crypto (no db deps)
|
|
│ │ ├── modules/ → TypeBox Module definitions
|
|
│ │ │ ├── metagraph.ts → Config, BaseNode, BaseEdge
|
|
│ │ │ ├── call-graph.ts → CallGraph reference Module
|
|
│ │ │ ├── secret-graph.ts → SecretGraph reference Module
|
|
│ │ │ └── index.ts → barrel re-export
|
|
│ │ ├── bridge.ts → moduleToDbSchema, validateNode, validateEdge
|
|
│ │ ├── crypto.ts → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
|
|
│ │ └── mod.ts → re-exports all graphs exports
|
|
│ └── sqlite/ → SQLite host (Drizzle + Honker)
|
|
│ ├── tables/ → drizzle table definitions
|
|
│ │ ├── common.ts → commonCols
|
|
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
|
|
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
|
|
│ │ └── index.ts → barrel re-export
|
|
│ ├── utils/ → createSelectSchema, createInsertSchema, column mappings (folded from @alkdev/dbtype)
|
|
│ ├── relations.ts → drizzle relational mappings
|
|
│ ├── schema.ts → barrel re-export
|
|
│ ├── adapter.ts → Drizzle-Honker session adapter
|
|
│ ├── event-target.ts → HonkerEventTarget (pubsub TypedEventTarget on Honker)
|
|
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
|
|
└── test/
|
|
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
|
|
```
|
|
|
|
### Subpath Exports (JSR/npm)
|
|
|
|
| Export | Contents | Dependencies |
|
|
| ------------------------ | --------------------------------------- | --------------------------------------- |
|
|
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
|
|
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
|
|
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter, event-target, utils | `drizzle-orm`, `@russellthehippo/honker-node`, `@alkdev/pubsub` (peer) |
|
|
|
|
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
|
|
database host.
|
|
|
|
## Database Model
|
|
|
|
### System DB + Tenant DB
|
|
|
|
The package uses a two-database model for multi-tenant isolation (ADR-040):
|
|
|
|
| Database | Contents | Purpose |
|
|
|----------|----------|---------|
|
|
| `system.db` | accounts, organizations, organization_members, api_keys, audit_logs, system graph_types | Identity infrastructure, authentication, authorization anchors |
|
|
| `tenant-{orgId}.db` | graphs, nodes, edges, graph_types, node_types, edge_types | All graph data for one org — call graphs, ACL instances, session trees, secrets, tasks |
|
|
|
|
The system DB is opened by `createSystemDatabase(client)`. Each tenant DB is
|
|
opened by `createTenantDatabase(client)`. Both return typed Drizzle instances
|
|
with their respective schemas attached.
|
|
|
|
**Why separate files**: File-level isolation means one tenant's data cannot leak
|
|
to another, even via application bugs. Each tenant DB is independently
|
|
backupable, migratable, and compactable. No `orgId` column is needed on tenant
|
|
tables because the entire file IS the org scope.
|
|
|
|
**Cross-DB references**: The tenant DB's `graphs.ownerId` and `graphs.projectId`
|
|
logically reference (not FK) the system DB's identity tables. The consumer
|
|
enforces referential integrity at the application layer, consistent with
|
|
ADR-020.
|
|
|
|
### Graph Type Scope
|
|
|
|
Graph types have a `scope` column (ADR-043) controlling who can create and
|
|
modify them:
|
|
|
|
| Scope | Examples | Who can modify |
|
|
|-------|----------|----------------|
|
|
| `system` | acl, call-graph, secret, operation-registry, message-session | Setup/seeding only |
|
|
| `tenant` | Custom org graph types (sprint-board, etc.) | Org admins |
|
|
| `user` | Personal graph types (my-notes, etc.) | Creating user |
|
|
|
|
System-scoped graph types are seeded during initialization. Their schemas are
|
|
fixed — changes require a version bump and migration (ADR-029).
|
|
|
|
## Terminology
|
|
|
|
| Term | Definition |
|
|
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| **Metagraph** | A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions. |
|
|
| **Hub** | The central service in the hub-spoke architecture. A consumer of `@alkdev/storage` — opens both system and tenant databases. The hub also depends on `@alkdev/operations`, `@alkdev/pubsub`, `@alkdev/flowgraph`. |
|
|
| **Spoke** | A local/embedded instance that runs per-project or per-session. A consumer of `@alkdev/storage` — opens a tenant database (or its own standalone DB for single-user mode). |
|
|
| **System DB** | The SQLite database holding identity/auth tables and system graph type definitions. One per deployment. |
|
|
| **Tenant DB** | The SQLite database holding all graph data for one organization. One per org. |
|
|
| **Graph type** | A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the `graph_types` table. |
|
|
| **Node type** | A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the `node_types` table. |
|
|
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
|
|
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
|
|
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
|
|
| **HonkerEventTarget** | Adapter that implements `@alkdev/pubsub`'s `TypedEventTarget` interface on Honker's `notify`/`listen` and `stream`/`subscribe` primitives. Bridges pubsub to Honker for single-node and cross-process scenarios. |
|
|
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
|
|
| **Repository surface** | Not hand-written CRUD functions. Storage outputs `OperationSpec[]` — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers handlers, and the operations runtime handles execution. |
|
|
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the bridge functions (at mutation time), NOT in the database. |
|
|
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
|
|
|
|
## Design Decisions
|
|
|
|
All design decisions are documented as ADRs in [decisions/](decisions/).
|
|
|
|
| ADR | Decision | Summary |
|
|
|-----|----------|---------|
|
|
| [001](decisions/001-deno-first-jsr-publishes.md) | Deno-first, JSR publishes | Published to JSR; npm comes free via `@jsr/alkdev__storage` |
|
|
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
|
|
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
|
|
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
|
|
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via local utils | Drizzle tables are single source of truth; `src/sqlite/utils/` generates TypeBox schemas (folded from @alkdev/drizzlebox) |
|
|
| [046](decisions/046-fold-drizzlebox-as-utils.md) | Fold @alkdev/drizzlebox as src/sqlite/utils | SQLite-only column mappings and schema generation co-located with tables |
|
|
| [047](decisions/047-honker-event-target.md) | HonkerEventTarget adapter | pubsub TypedEventTarget on Honker notify/listen and stream/subscribe |
|
|
| [048](decisions/048-operation-specs-as-repo-surface.md) | OperationSpecs as repository surface | Storage outputs OperationSpec[] not hand-written CRUD |
|
|
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
|
|
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
|
|
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
|
|
| [033](decisions/033-json-path-queries-for-v1.md) | JSON path queries and hand-written CRUD for v1 | Attribute queries use JSON path; CRUD is hand-written |
|
|
| [038](decisions/038-sqlite-first-pg-removed.md) | SQLite-first, Postgres removed | Single database host; no dual maintenance |
|
|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | DB + pub/sub + queues + events in one SQLite file |
|
|
| [040](decisions/040-system-db-tenant-db.md) | System DB + tenant DB separation | Identity in system.db, graph data in tenant-{orgId}.db |
|
|
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage package | accounts, organizations, api_keys, audit_logs defined in storage |
|
|
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graph instances | `ownerId`, `projectId` on `graphs` table |
|
|
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope controls who can modify graph types |
|
|
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | ~100-line adapter; no Drizzle fork; `$client` and `$honkerTx` for honker access |
|
|
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | organization_members authoritative, BelongsToEdge derived | SQL table for fast lookups; ACL edge for traversal evaluation |
|
|
|
|
## Dependencies
|
|
|
|
| Package | Purpose | Layer |
|
|
| -------------------- | ------------------------------------ | ------------------------ |
|
|
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
|
|
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
|
|
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
|
|
| `@alkdev/pubsub` | `TypedEventTarget` interface (peer) | sqlite/event-target.ts |
|
|
|
|
`@alkdev/typebox` is an npm package (not yet on JSR). JSR handles npm
|
|
dependencies natively. `@alkdev/pubsub` is a peer dependency — only needed
|
|
when using `HonkerEventTarget`. The `graphs/` module has zero dependencies
|
|
beyond `@alkdev/typebox`.
|
|
|
|
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
|
|
ecosystem references describe consumer-side data shapes and integration
|
|
patterns, not import dependencies.
|
|
|
|
## What Exists vs. What's Needed
|
|
|
|
### Implemented
|
|
|
|
- Metagraph Module (`Type.Module` with Config, BaseNode, BaseEdge entries)
|
|
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
|
|
- Reference graph type Modules (CallGraph, SecretGraph)
|
|
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
|
|
- SQLite host: 6 metagraph tables (flat structure, pending reorganize) + Drizzle relations + client factory
|
|
- TypeBox select/insert schemas generated from Drizzle tables (current: `@alkdev/drizzlebox`; pending fold to `src/sqlite/utils/`, ADR-046)
|
|
- Reference module tests (bridge functions, validation, Module composition)
|
|
|
|
### Not Yet Implemented
|
|
|
|
| Gap | Priority | Notes |
|
|
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
|
|
| Drizzle-Honker session adapter | High | `HonkerSQLiteSession` + `HonkerPreparedQuery` + `HonkerSQLiteTransaction`. POC validated. ADR-044. |
|
|
| Fold dbtype Phase 0 → `src/sqlite/utils/` | High | SQLite-only column mappings + createSelectSchema/createInsertSchema. ADR-046. |
|
|
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
|
|
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
|
|
| Graph type `scope` column | High | ADR-043. |
|
|
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
|
|
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. Accept Honker client. |
|
|
| Table restructure into subdirectories | High | `tables/metagraph/`, `tables/identity/`, update relative imports. |
|
|
| HonkerEventTarget adapter | High | pubsub `TypedEventTarget` on Honker primitives. POC validated. ADR-047. |
|
|
| OperationSpec generation from tables | Medium | `OperationSpec[]` per table for CRUD operations. ADR-048. |
|
|
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
|
|
| Domain-specific native-column tables | Low | CallGraph, SecretGraph, etc. with native columns alongside JSON attributes. |
|
|
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
|
|
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
|
|
|
|
## Ecosystem Integration
|
|
|
|
`@alkdev/storage` is a **data layer package** consumed by other packages in the
|
|
@alkdev ecosystem. It does not depend on the hub — the dependency flows the
|
|
other way.
|
|
|
|
### Dependency Direction
|
|
|
|
```
|
|
@alkdev/pubsub ← transport only (no storage dependency)
|
|
↑
|
|
@alkdev/operations ← call protocol, registry, identity, access control
|
|
↑ (depends on: @alkdev/pubsub, @alkdev/typebox)
|
|
@alkdev/flowgraph ← call graph schema, operation graph, workflow templates
|
|
↑ (depends on: @alkdev/operations [peer], @alkdev/typebox)
|
|
@alkdev/taskgraph ← task dependency graph schema, cost-benefit analysis
|
|
(depends on: @alkdev/typebox)
|
|
|
|
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
|
|
(depends on: @alkdev/typebox, drizzle-orm, honker-node)
|
|
(peer dep: @alkdev/pubsub for TypedEventTarget type)
|
|
|
|
↑ ↑
|
|
| |
|
|
Hub / Spoke Any consumer that needs
|
|
(consumes all) persistent graph storage
|
|
```
|
|
|
|
### Event-Driven Architecture with Honker
|
|
|
|
The @alkdev platform is event-driven. Honker provides the transport mechanism
|
|
within each SQLite database. Storage provides the `HonkerEventTarget` adapter
|
|
so consumers can use `@alkdev/pubsub`'s `TypedEventTarget` interface regardless
|
|
of whether events route in-process, through Honker, or over WebSocket.
|
|
|
|
| Concern | Honker Primitive | Example |
|
|
|---------|------------------|---------|
|
|
| Fire-and-forget notifications | `notify(channel, payload)` | "graph:updated", "node:created" |
|
|
| Durable per-consumer delivery | `stream(name).subscribe(consumer)` | Call protocol events, audit trail |
|
|
| At-least-once background jobs | `queue(name).enqueue(payload)` | Key rotation, schema migration, retention cleanup |
|
|
| Leader election | `tryLock(name, owner, ttl)` | Only one hub instance runs the scheduler |
|
|
| Scheduled operations | `scheduler().add(cron, handler)` | Retention cleanup, key sweep, rate limit sweep |
|
|
|
|
A single Drizzle transaction via the Honker adapter can insert graph data AND
|
|
publish a notification AND enqueue a side-effect job — all committing atomically.
|
|
No dual-write problem between data and events. POC 4 confirmed: `tx.notify()`
|
|
only fires on commit; on rollback, both the data write and the notification are
|
|
suppressed.
|
|
|
|
### Latency Consideration
|
|
|
|
POC 2 measured ~17ms median latency for Honker `notify`→`listen` within a
|
|
single process. For hot-path call protocol request/response where sub-ms
|
|
latency matters, pair the Honker event target with an in-process `EventTarget`
|
|
(pubsub's default). A composite pattern (dispatch to both) provides both
|
|
in-process speed and Honker durability/cross-process coordination.
|
|
|
|
### What Comes from Where
|
|
|
|
| Concept | Source package | Storage's role |
|
|
|---------|---------------|----------------|
|
|
| Call protocol events | `@alkdev/operations` | Storage persists the outcomes as graph nodes + publishes via Honker stream |
|
|
| Identity | `@alkdev/operations` | Storage stores identity in SQL accounts table + as PrincipalNode in ACL graphs |
|
|
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
|
|
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
|
|
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
|
|
| Event transport | Honker (within storage) | `HonkerEventTarget` bridges pubsub `TypedEventTarget` to Honker primitives |
|
|
| Pubsub interface | `@alkdev/pubsub` | Storage provides `HonkerEventTarget` adapter; does not replace pubsub |
|
|
|
|
### Repository Surface as OperationSpecs
|
|
|
|
Storage does not ship a "repository layer" of hand-written CRUD functions.
|
|
Instead, it outputs `OperationSpec[]` — flat arrays describing CRUD and query
|
|
operations for each table. The consumer (hub/spoke) imports these specs,
|
|
registers them in the `@alkdev/operations` registry along with handlers, and
|
|
the operations runtime handles execution, call protocol, and subscriptions.
|
|
|
|
Storage depends on `@alkdev/operations` only as a type-level peer dependency
|
|
(for the `OperationSpec` type). No circular dependency.
|
|
|
|
```
|
|
@alkdev/storage → defines types + tables + OperationSpec[] (type-only operations dep)
|
|
@alkdev/operations → defines call protocol + registry (no storage dependency)
|
|
Consumer (hub / spoke) → imports both, registers specs + handlers
|
|
```
|
|
|
|
### Avoiding Circular Dependencies
|
|
|
|
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
|
|
other at runtime. Storage defines the schema types, database tables, and
|
|
operation contract definitions; operations defines the call protocol and
|
|
execution model. The consumer imports both and bridges them.
|
|
|
|
## Open Questions
|
|
|
|
Open questions are tracked in [open-questions.md](open-questions.md). Key
|
|
questions affecting this package:
|
|
|
|
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
|
|
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
|
|
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
|
|
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (resolved: yes — HonkerEventTarget adapter, POC validated. Redis still needed for multi-node. See ADR-047.)
|
|
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
|
|
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)
|
|
|
|
## References
|
|
|
|
- Metagraph Module: [metagraph-module.md](./metagraph-module.md)
|
|
- Honker integration: [honker-integration.md](./honker-integration.md)
|
|
- SQLite host tables: [sqlite-host.md](./sqlite-host.md)
|
|
- Schema evolution: [schema-evolution.md](./schema-evolution.md)
|
|
- Encrypted data: [encrypted-data.md](./encrypted-data.md)
|
|
- ACL graph: [acl.md](./acl.md)
|
|
- Forward-looking connections: [forward-look.md](./forward-look.md)
|
|
- Open questions: [open-questions.md](./open-questions.md)
|
|
- Honker source: `/workspace/honker/`
|
|
- Operations architecture: `/workspace/@alkdev/operations/docs/architecture/`
|
|
- Flowgraph architecture: `/workspace/@alkdev/flowgraph/docs/architecture/` |