glm-5.1
6aa2fcc6ff
Reorient @alkdev/storage around a single SQLite database host with Honker
for pub/sub, event streams, and task queues. PostgreSQL is removed as a
target (ADR-038), eliminating dual schema maintenance and infrastructure
complexity. Honker provides DB + pubsub + queues in one .db file (ADR-039).
Add system/tenant DB model (ADR-040): identity tables in system.db, all
graph data in tenant-{orgId}.db files. Identity tables move from the hub
into storage (ADR-041). Scoping columns (ownerId, projectId) added to
graphs table (ADR-042). Graph types get scope (system/tenant/user) to
protect infrastructure schemas (ADR-043).
Define Drizzle-Honker session adapter (ADR-044): ~100-line adapter enabling
Drizzle typed queries and Honker pubsub/queue on a single connection with
transactional consistency.
Resolve OQ-03, OQ-04, OQ-19, OQ-21, OQ-22, OQ-23, OQ-24. Add new
open questions OQ-26 through OQ-29 for Honker integration specifics.
New docs: honker-integration.md (adapter, event patterns, migration).
Scrub all PG/jsonb/libsql references from existing spec docs.
289 lines
19 KiB
Markdown
289 lines
19 KiB
Markdown
---
|
|
status: draft
|
|
last_updated: 2026-05-31
|
|
---
|
|
|
|
# @alkdev/storage — Overview
|
|
|
|
Typed graph storage with SQLite via Honker. Deno-first, published via JSR.
|
|
|
|
## Purpose
|
|
|
|
`@alkdev/storage` provides a **metagraph** storage model: graph types define
|
|
schemas, node types define data shapes within those graphs, and edge types
|
|
define typed relationships. Instances of these type definitions become actual
|
|
graphs populated with nodes and edges.
|
|
|
|
This pattern replaces domain-specific table proliferation with a small number of
|
|
general-purpose tables that can model anything — call graphs, ACL rules, task
|
|
dependencies, encrypted secrets, session trees, operation registries — while
|
|
enforcing schema integrity through TypeBox validation.
|
|
|
|
The package also provides **identity infrastructure** tables (accounts,
|
|
organizations, api_keys, audit_logs) for multi-tenant authentication and
|
|
authorization, and integrates with **Honker** for transactional pub/sub, event
|
|
streams, and task queues within the same SQLite database.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
@alkdev/storage/
|
|
├── mod.ts → re-exports graphs/ (zero db deps)
|
|
├── src/
|
|
│ ├── graphs/ → Metagraph Module, bridge functions, crypto (no db deps)
|
|
│ │ ├── modules/ → TypeBox Module definitions
|
|
│ │ │ ├── metagraph.ts → Config, BaseNode, BaseEdge
|
|
│ │ │ ├── call-graph.ts → CallGraph reference Module
|
|
│ │ │ ├── secret-graph.ts → SecretGraph reference Module
|
|
│ │ │ └── index.ts → barrel re-export
|
|
│ │ ├── bridge.ts → moduleToDbSchema, validateNode, validateEdge
|
|
│ │ ├── crypto.ts → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
|
|
│ │ └── mod.ts → re-exports all graphs exports
|
|
│ └── sqlite/ → SQLite host (Drizzle + Honker)
|
|
│ ├── tables/ → drizzle table definitions
|
|
│ │ ├── common.ts → commonCols
|
|
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
|
|
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
|
|
│ │ └── index.ts → barrel re-export
|
|
│ ├── relations.ts → drizzle relational mappings
|
|
│ ├── schema.ts → barrel re-export
|
|
│ ├── adapter.ts → Drizzle-Honker session adapter
|
|
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
|
|
└── test/
|
|
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
|
|
```
|
|
|
|
### Subpath Exports (JSR/npm)
|
|
|
|
| Export | Contents | Dependencies |
|
|
| ------------------------ | --------------------------------------- | --------------------------------------- |
|
|
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
|
|
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
|
|
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter | `@alkdev/drizzlebox`, `drizzle-orm`, `@russellthehipp/honker-node` |
|
|
|
|
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
|
|
database host.
|
|
|
|
## Database Model
|
|
|
|
### System DB + Tenant DB
|
|
|
|
The package uses a two-database model for multi-tenant isolation (ADR-040):
|
|
|
|
| Database | Contents | Purpose |
|
|
|----------|----------|---------|
|
|
| `system.db` | accounts, organizations, organization_members, api_keys, audit_logs, system graph_types | Identity infrastructure, authentication, authorization anchors |
|
|
| `tenant-{orgId}.db` | graphs, nodes, edges, graph_types, node_types, edge_types | All graph data for one org — call graphs, ACL instances, session trees, secrets, tasks |
|
|
|
|
The system DB is opened by `createSystemDatabase(client)`. Each tenant DB is
|
|
opened by `createTenantDatabase(client)`. Both return typed Drizzle instances
|
|
with their respective schemas attached.
|
|
|
|
**Why separate files**: File-level isolation means one tenant's data cannot leak
|
|
to another, even via application bugs. Each tenant DB is independently
|
|
backupable, migratable, and compactable. No `orgId` column is needed on tenant
|
|
tables because the entire file IS the org scope.
|
|
|
|
**Cross-DB references**: The tenant DB's `graphs.ownerId` and `graphs.projectId`
|
|
logically reference (not FK) the system DB's identity tables. The consumer
|
|
enforces referential integrity at the application layer, consistent with
|
|
ADR-020.
|
|
|
|
### Graph Type Scope
|
|
|
|
Graph types have a `scope` column (ADR-043) controlling who can create and
|
|
modify them:
|
|
|
|
| Scope | Examples | Who can modify |
|
|
|-------|----------|----------------|
|
|
| `system` | acl, call-graph, secret, operation-registry, message-session | Setup/seeding only |
|
|
| `tenant` | Custom org graph types (sprint-board, etc.) | Org admins |
|
|
| `user` | Personal graph types (my-notes, etc.) | Creating user |
|
|
|
|
System-scoped graph types are seeded during initialization. Their schemas are
|
|
fixed — changes require a version bump and migration (ADR-029).
|
|
|
|
## Terminology
|
|
|
|
| Term | Definition |
|
|
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| **Metagraph** | A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions. |
|
|
| **Hub** | The central service in the hub-spoke architecture. A consumer of `@alkdev/storage` — opens both system and tenant databases. The hub also depends on `@alkdev/operations`, `@alkdev/pubsub`, `@alkdev/flowgraph`. |
|
|
| **Spoke** | A local/embedded instance that runs per-project or per-session. A consumer of `@alkdev/storage` — opens a tenant database (or its own standalone DB for single-user mode). |
|
|
| **System DB** | The SQLite database holding identity/auth tables and system graph type definitions. One per deployment. |
|
|
| **Tenant DB** | The SQLite database holding all graph data for one organization. One per org. |
|
|
| **Graph type** | A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the `graph_types` table. |
|
|
| **Node type** | A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the `node_types` table. |
|
|
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
|
|
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
|
|
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
|
|
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
|
|
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. |
|
|
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
|
|
|
|
## Design Decisions
|
|
|
|
All design decisions are documented as ADRs in [decisions/](decisions/).
|
|
|
|
| ADR | Decision | Summary |
|
|
|-----|----------|---------|
|
|
| [001](decisions/001-deno-first-jsr-publishes.md) | Deno-first, JSR publishes | Published to JSR; npm comes free via `@jsr/alkdev__storage` |
|
|
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
|
|
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
|
|
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
|
|
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via drizzlebox | Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas |
|
|
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
|
|
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
|
|
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
|
|
| [033](decisions/033-json-path-queries-for-v1.md) | JSON path queries and hand-written CRUD for v1 | Attribute queries use JSON path; CRUD is hand-written |
|
|
| [038](decisions/038-sqlite-first-pg-removed.md) | SQLite-first, Postgres removed | Single database host; no dual maintenance |
|
|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | DB + pub/sub + queues + events in one SQLite file |
|
|
| [040](decisions/040-system-db-tenant-db.md) | System DB + tenant DB separation | Identity in system.db, graph data in tenant-{orgId}.db |
|
|
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage package | accounts, organizations, api_keys, audit_logs defined in storage |
|
|
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graph instances | `ownerId`, `projectId` on `graphs` table |
|
|
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope controls who can modify graph types |
|
|
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | ~100-line adapter; no Drizzle fork; `$client` and `$honkerTx` for honker access |
|
|
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | organization_members authoritative, BelongsToEdge derived | SQL table for fast lookups; ACL edge for traversal evaluation |
|
|
|
|
## Dependencies
|
|
|
|
| Package | Purpose | Layer |
|
|
| -------------------- | ------------------------------------ | ------------------------ |
|
|
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
|
|
| `@alkdev/drizzlebox` | Generate TypeBox from Drizzle tables | sqlite/ |
|
|
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
|
|
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
|
|
|
|
`@alkdev/typebox` and `@alkdev/drizzlebox` are npm packages (not yet on JSR).
|
|
JSR handles npm dependencies natively.
|
|
|
|
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
|
|
ecosystem references describe consumer-side data shapes and integration
|
|
patterns, not import dependencies.
|
|
|
|
## What Exists vs. What's Needed
|
|
|
|
### Implemented
|
|
|
|
- Metagraph Module (`Type.Module` with Config, BaseNode, BaseEdge entries)
|
|
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
|
|
- Reference graph type Modules (CallGraph, SecretGraph)
|
|
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
|
|
- SQLite host: 6 metagraph tables + `actors` placeholder + Drizzle relations + client factory
|
|
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
|
|
- Reference module tests (bridge functions, validation, Module composition)
|
|
|
|
### Not Yet Implemented
|
|
|
|
| Gap | Priority | Notes |
|
|
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
|
|
| Drizzle-Honker session adapter | High | ~100-line adapter wrapping `HonkerSQLiteSession` + `HonkerPreparedQuery`. ADR-044. |
|
|
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
|
|
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
|
|
| Graph type `scope` column | High | ADR-043. |
|
|
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
|
|
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. |
|
|
| Repository/CRUD layer | Medium | ⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges. |
|
|
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
|
|
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
|
|
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
|
|
|
|
## Ecosystem Integration
|
|
|
|
`@alkdev/storage` is a **data layer package** consumed by other packages in the
|
|
@alkdev ecosystem. It does not depend on the hub — the dependency flows the
|
|
other way.
|
|
|
|
### Dependency Direction
|
|
|
|
```
|
|
@alkdev/pubsub ← transport only (no storage dependency)
|
|
↑
|
|
@alkdev/operations ← call protocol, registry, identity, access control
|
|
↑ (depends on: @alkdev/pubsub, @alkdev/typebox)
|
|
@alkdev/flowgraph ← call graph schema, operation graph, workflow templates
|
|
↑ (depends on: @alkdev/operations [peer], @alkdev/typebox)
|
|
@alkdev/taskgraph ← task dependency graph schema, cost-benefit analysis
|
|
(depends on: @alkdev/typebox)
|
|
|
|
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
|
|
(depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)
|
|
|
|
↑ ↑
|
|
| |
|
|
Hub / Spoke Any consumer that needs
|
|
(consumes all) persistent graph storage
|
|
```
|
|
|
|
### Event-Driven Architecture with Honker
|
|
|
|
The @alkdev platform is event-driven. Honker provides the transport mechanism
|
|
within each SQLite database:
|
|
|
|
| Concern | Honker Primitive | Example |
|
|
|---------|------------------|---------|
|
|
| Fire-and-forget notifications | `notify(channel, payload)` | "graph:updated", "node:created" |
|
|
| Durable per-consumer delivery | `stream(name).subscribe(consumer)` | Call protocol events, audit trail |
|
|
| At-least-once background jobs | `queue(name).enqueue(payload)` | Key rotation, schema migration, retention cleanup |
|
|
| Leader election | `tryLock(name, owner, ttl)` | Only one hub instance runs the scheduler |
|
|
| Scheduled operations | `scheduler().add(cron, handler)` | Retention cleanup, key sweep, rate limit sweep |
|
|
|
|
A single Drizzle transaction via the Honker adapter can insert graph data AND
|
|
publish a notification AND enqueue a side-effect job — all committing atomically.
|
|
No dual-write problem between data and events.
|
|
|
|
### What Comes from Where
|
|
|
|
| Concept | Source package | Storage's role |
|
|
|---------|---------------|----------------|
|
|
| Call protocol events | `@alkdev/operations` | Storage persists the outcomes as graph nodes + publishes via Honker stream |
|
|
| Identity | `@alkdev/operations` | Storage stores identity in SQL accounts table + as PrincipalNode in ACL graphs |
|
|
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
|
|
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
|
|
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
|
|
| Event transport | Honker (within storage) | Replaces `@alkdev/pubsub`'s Redis transport for single-node deployments |
|
|
|
|
### Repository Layer Bridging Pattern
|
|
|
|
The repository layer in `@alkdev/storage` provides typed CRUD — no
|
|
`@alkdev/operations` dependency. A consumer-side bridging module wires CRUD
|
|
functions into the operations registry when needed.
|
|
|
|
```
|
|
@alkdev/storage → defines types + tables + CRUD (no operations dependency)
|
|
@alkdev/operations → defines call protocol + registry (no storage dependency)
|
|
Consumer (hub / adapter) → imports both, generates operations from schemas
|
|
```
|
|
|
|
### Avoiding Circular Dependencies
|
|
|
|
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
|
|
other directly. Storage defines the schema types and database tables; operations
|
|
defines the call protocol and execution model. The consumer imports both and
|
|
bridges them.
|
|
|
|
## Open Questions
|
|
|
|
Open questions are tracked in [open-questions.md](open-questions.md). Key
|
|
questions affecting this package:
|
|
|
|
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
|
|
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
|
|
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
|
|
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (open)
|
|
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
|
|
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)
|
|
|
|
## References
|
|
|
|
- Metagraph Module: [metagraph-module.md](./metagraph-module.md)
|
|
- Honker integration: [honker-integration.md](./honker-integration.md)
|
|
- SQLite host tables: [sqlite-host.md](./sqlite-host.md)
|
|
- Schema evolution: [schema-evolution.md](./schema-evolution.md)
|
|
- Encrypted data: [encrypted-data.md](./encrypted-data.md)
|
|
- ACL graph: [acl.md](./acl.md)
|
|
- Forward-looking connections: [forward-look.md](./forward-look.md)
|
|
- Open questions: [open-questions.md](./open-questions.md)
|
|
- Honker source: `/workspace/honker/`
|
|
- Operations architecture: `/workspace/@alkdev/operations/docs/architecture/`
|
|
- Flowgraph architecture: `/workspace/@alkdev/flowgraph/docs/architecture/` |