storage/docs/architecture/overview.md

status, last_updated

status	last_updated
draft	2026-06-01

@alkdev/storage — Overview

Typed graph storage with SQLite via Honker. Deno-first, published via JSR.

Purpose

@alkdev/storage provides a metagraph storage model: graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Instances of these type definitions become actual graphs populated with nodes and edges.

This pattern replaces domain-specific table proliferation with a small number of general-purpose tables that can model anything — call graphs, ACL rules, task dependencies, encrypted secrets, session trees, operation registries — while enforcing schema integrity through TypeBox validation.

The package also provides identity infrastructure tables (accounts, organizations, api_keys, audit_logs) for multi-tenant authentication and authorization, and integrates with Honker for transactional pub/sub, event streams, and task queues within the same SQLite database.

Storage also provides a HonkerEventTarget adapter that bridges @alkdev/pubsub's TypedEventTarget interface to Honker's notify/listen and stream/subscribe primitives, and OperationSpec generation from table definitions so downstream hubs/spokes can register CRUD operations directly into the @alkdev/operations registry.

Architecture

@alkdev/storage/
├── mod.ts                   → re-exports graphs/ (zero db deps)
├── src/
│   ├── graphs/              → Metagraph Module, bridge functions, crypto (no db deps)
│   │   ├── modules/         → TypeBox Module definitions
│   │   │   ├── metagraph.ts → Config, BaseNode, BaseEdge
│   │   │   ├── call-graph.ts → CallGraph reference Module
│   │   │   ├── secret-graph.ts → SecretGraph reference Module
│   │   │   └── index.ts     → barrel re-export
│   │   ├── bridge.ts        → moduleToDbSchema, validateNode, validateEdge
│   │   ├── crypto.ts        → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│   │   └── mod.ts           → re-exports all graphs exports
│   └── sqlite/              → SQLite host (Drizzle + Honker)
│       ├── tables/          → drizzle table definitions
│       │   ├── common.ts    → commonCols
│       │   ├── identity/    → accounts, organizations, org_members, api_keys, audit_logs
│       │   ├── metagraph/   → graph_types, node_types, edge_types, graphs, nodes, edges
│       │   └── index.ts     → barrel re-export
│       ├── utils/           → createSelectSchema, createInsertSchema, column mappings (folded from @alkdev/dbtype)
│       ├── relations.ts     → drizzle relational mappings
│       ├── schema.ts        → barrel re-export
│       ├── adapter.ts       → Drizzle-Honker session adapter
│       ├── event-target.ts  → HonkerEventTarget (pubsub TypedEventTarget on Honker)
│       └── client.ts        → createSystemDatabase(), createTenantDatabase()
└── test/
    └── reference-modules.test.ts → Metagraph, bridge, crypto tests

Subpath Exports (JSR/npm)

Export	Contents	Dependencies
@alkdev/storage	Graph schema types, Metagraph Module	@alkdev/typebox
@alkdev/storage/graphs	Same as . — alias for the main export	Same as .
@alkdev/storage/sqlite	SQLite tables, relations, client, adapter, event-target, utils	drizzle-orm, @russellthehippo/honker-node, @alkdev/pubsub (peer)

The pg/ subpath has been removed (ADR-038). SQLite via Honker is the sole database host.

Database Model

System DB + Tenant DB

The package uses a two-database model for multi-tenant isolation (ADR-040):

Database	Contents	Purpose
system.db	accounts, organizations, organization_members, api_keys, audit_logs, system graph_types	Identity infrastructure, authentication, authorization anchors
tenant-{orgId}.db	graphs, nodes, edges, graph_types, node_types, edge_types	All graph data for one org — call graphs, ACL instances, session trees, secrets, tasks

The system DB is opened by createSystemDatabase(client). Each tenant DB is opened by createTenantDatabase(client). Both return typed Drizzle instances with their respective schemas attached.

Why separate files: File-level isolation means one tenant's data cannot leak to another, even via application bugs. Each tenant DB is independently backupable, migratable, and compactable. No orgId column is needed on tenant tables because the entire file IS the org scope.

Cross-DB references: The tenant DB's graphs.ownerId and graphs.projectId logically reference (not FK) the system DB's identity tables. The consumer enforces referential integrity at the application layer, consistent with ADR-020.

Graph Type Scope

Graph types have a scope column (ADR-043) controlling who can create and modify them:

Scope	Examples	Who can modify
system	acl, call-graph, secret, operation-registry, message-session	Setup/seeding only
tenant	Custom org graph types (sprint-board, etc.)	Org admins
user	Personal graph types (my-notes, etc.)	Creating user

System-scoped graph types are seeded during initialization. Their schemas are fixed — changes require a version bump and migration (ADR-029).

Terminology

Term	Definition
Metagraph	A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions.
Hub	The central service in the hub-spoke architecture. A consumer of @alkdev/storage — opens both system and tenant databases. The hub also depends on @alkdev/operations, @alkdev/pubsub, @alkdev/flowgraph.
Spoke	A local/embedded instance that runs per-project or per-session. A consumer of @alkdev/storage — opens a tenant database (or its own standalone DB for single-user mode).
System DB	The SQLite database holding identity/auth tables and system graph type definitions. One per deployment.
Tenant DB	The SQLite database holding all graph data for one organization. One per org.
Graph type	A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the graph_types table.
Node type	A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the node_types table.
Edge type	A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the edge_types table.
Graph instance	A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the graphs table.
Honker	SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same .db file.
HonkerEventTarget	Adapter that implements @alkdev/pubsub's TypedEventTarget interface on Honker's notify/listen and stream/subscribe primitives. Bridges pubsub to Honker for single-node and cross-process scenarios.
Consumer	Code that imports @alkdev/storage (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers.
Repository surface	Not hand-written CRUD functions. Storage outputs OperationSpec[] — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers handlers, and the operations runtime handles execution.
Validation boundary	The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the bridge functions (at mutation time), NOT in the database.
Validation boundary	The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database.

Design Decisions

All design decisions are documented as ADRs in decisions/.

ADR	Decision	Summary
001	Deno-first, JSR publishes	Published to JSR; npm comes free via @jsr/alkdev__storage
002	Metagraph over domain-specific tables	6 general-purpose tables serve all graph-shaped domains
003	TypeBox Module as API surface	Type.Module() replaces SchemaBuilder; Metagraph.Import() + Type.Composite()
004	Injectable clients, no side effects	createSystemDatabase(client) / createTenantDatabase(client) take pre-created clients
005	Drizzle + TypeBox via local utils	Drizzle tables are single source of truth; src/sqlite/utils/ generates TypeBox schemas (folded from @alkdev/drizzlebox)
046	Fold @alkdev/drizzlebox as src/sqlite/utils	SQLite-only column mappings and schema generation co-located with tables
047	HonkerEventTarget adapter	pubsub TypedEventTarget on Honker notify/listen and stream/subscribe
048	OperationSpecs as repository surface	Storage outputs OperationSpec[] not hand-written CRUD
006	as const objects, not TypeScript enums	Avoids JSR slow-types; consistent pattern across codebase
007	No comments in code	Documentation lives in architecture docs and TypeBox descriptions
008	Common columns pattern	id, metadata, createdAt, updatedAt on every table
033	JSON path queries and hand-written CRUD for v1	Attribute queries use JSON path; CRUD is hand-written
038	SQLite-first, Postgres removed	Single database host; no dual maintenance
039	Honker as SQLite extension and transport	DB + pub/sub + queues + events in one SQLite file
040	System DB + tenant DB separation	Identity in system.db, graph data in tenant-{orgId}.db
041	Identity tables in storage package	accounts, organizations, api_keys, audit_logs defined in storage
042	Scoping columns on graph instances	ownerId, projectId on graphs table
043	Graph type scope	system / tenant / user scope controls who can modify graph types
044	Drizzle-Honker session adapter	~100-line adapter; no Drizzle fork; $client and $honkerTx for honker access
045	organization_members authoritative, BelongsToEdge derived	SQL table for fast lookups; ACL edge for traversal evaluation

Dependencies

Package	Purpose	Layer
@alkdev/typebox	Runtime schema validation	graphs/
drizzle-orm	ORM, table definitions, queries	sqlite/
@russellthehippo/honker-node	SQLite + pub/sub + queues + events	sqlite/
@alkdev/pubsub	TypedEventTarget interface (peer)	sqlite/event-target.ts

@alkdev/typebox is an npm package (not yet on JSR). JSR handles npm dependencies natively. @alkdev/pubsub is a peer dependency — only needed when using HonkerEventTarget. The graphs/ module has zero dependencies beyond @alkdev/typebox.

Ecosystem packages are not runtime dependencies of @alkdev/storage. All ecosystem references describe consumer-side data shapes and integration patterns, not import dependencies.

What Exists vs. What's Needed

Implemented

• Metagraph Module (Type.Module with Config, BaseNode, BaseEdge entries)
• Bridge functions (moduleToDbSchema, validateNode, validateEdge)
• Reference graph type Modules (CallGraph, SecretGraph)
• Crypto utility (AES-256-GCM + PBKDF2, EncryptedDataSchema)
• SQLite host: 6 metagraph tables (flat structure, pending reorganize) + Drizzle relations + client factory
• TypeBox select/insert schemas generated from Drizzle tables (current: @alkdev/drizzlebox; pending fold to src/sqlite/utils/, ADR-046)
• Reference module tests (bridge functions, validation, Module composition)

Not Yet Implemented

Gap	Priority	Notes
Drizzle-Honker session adapter	High	HonkerSQLiteSession + HonkerPreparedQuery + HonkerSQLiteTransaction. POC validated. ADR-044.
Fold dbtype Phase 0 → src/sqlite/utils/	High	SQLite-only column mappings + createSelectSchema/createInsertSchema. ADR-046.
Identity tables (accounts, organizations, api_keys, audit_logs, organization_members)	High	Moving from hub into storage. ADR-041.
Scoping columns on graphs table (ownerId, projectId)	High	ADR-042.
Graph type scope column	High	ADR-043.
Remove actors table and pg/ directory	High	ADR-035 (actors), ADR-038 (pg).
createSystemDatabase() / createTenantDatabase() factories	High	Split from current createSqliteDatabase(). Accept Honker client.
Table restructure into subdirectories	High	tables/metagraph/, tables/identity/, update relative imports.
HonkerEventTarget adapter	High	pubsub TypedEventTarget on Honker primitives. POC validated. ADR-047.
OperationSpec generation from tables	Medium	OperationSpec[] per table for CRUD operations. ADR-048.
ACL graph type	Medium	Access control as a metagraph. ADR-034.
Domain-specific native-column tables	Low	CallGraph, SecretGraph, etc. with native columns alongside JSON attributes.
Task graph type	Low	Informed by @alkdev/taskgraph's schemas.
Graphology bridge	Low	moduleToGraphology() and fromGraphologyExport() — Phase 4.

Ecosystem Integration

@alkdev/storage is a data layer package consumed by other packages in the @alkdev ecosystem. It does not depend on the hub — the dependency flows the other way.

Dependency Direction

@alkdev/pubsub          ← transport only (no storage dependency)
    ↑
@alkdev/operations      ← call protocol, registry, identity, access control
    ↑                   (depends on: @alkdev/pubsub, @alkdev/typebox)
@alkdev/flowgraph       ← call graph schema, operation graph, workflow templates
    ↑                   (depends on: @alkdev/operations [peer], @alkdev/typebox)
@alkdev/taskgraph       ← task dependency graph schema, cost-benefit analysis
                         (depends on: @alkdev/typebox)

@alkdev/storage         ← YOU ARE HERE — typed graph persistence + identity
                         (depends on: @alkdev/typebox, drizzle-orm, honker-node)
                         (peer dep: @alkdev/pubsub for TypedEventTarget type)

    ↑                       ↑
    |                       |
Hub / Spoke          Any consumer that needs
(consumes all)       persistent graph storage

Event-Driven Architecture with Honker

The @alkdev platform is event-driven. Honker provides the transport mechanism within each SQLite database. Storage provides the HonkerEventTarget adapter so consumers can use @alkdev/pubsub's TypedEventTarget interface regardless of whether events route in-process, through Honker, or over WebSocket.

Concern	Honker Primitive	Example
Fire-and-forget notifications	notify(channel, payload)	"graph:updated", "node:created"
Durable per-consumer delivery	stream(name).subscribe(consumer)	Call protocol events, audit trail
At-least-once background jobs	queue(name).enqueue(payload)	Key rotation, schema migration, retention cleanup
Leader election	tryLock(name, owner, ttl)	Only one hub instance runs the scheduler
Scheduled operations	scheduler().add(cron, handler)	Retention cleanup, key sweep, rate limit sweep

A single Drizzle transaction via the Honker adapter can insert graph data AND publish a notification AND enqueue a side-effect job — all committing atomically. No dual-write problem between data and events. POC 4 confirmed: tx.notify() only fires on commit; on rollback, both the data write and the notification are suppressed.

Latency Consideration

POC 2 measured ~17ms median latency for Honker notify→listen within a single process. For hot-path call protocol request/response where sub-ms latency matters, pair the Honker event target with an in-process EventTarget (pubsub's default). A composite pattern (dispatch to both) provides both in-process speed and Honker durability/cross-process coordination.

What Comes from Where

Concept	Source package	Storage's role
Call protocol events	@alkdev/operations	Storage persists the outcomes as graph nodes + publishes via Honker stream
Identity	@alkdev/operations	Storage stores identity in SQL accounts table + as PrincipalNode in ACL graphs
Access control	@alkdev/operations	Storage's AclGraph mirrors AccessControl schema as graph structure
Call graph schema	@alkdev/flowgraph	Storage persists in-memory shapes to the database
Task graph schema	@alkdev/taskgraph	Storage persists task dependency shapes
Event transport	Honker (within storage)	HonkerEventTarget bridges pubsub TypedEventTarget to Honker primitives
Pubsub interface	@alkdev/pubsub	Storage provides HonkerEventTarget adapter; does not replace pubsub

Repository Surface as OperationSpecs

Storage does not ship a "repository layer" of hand-written CRUD functions. Instead, it outputs OperationSpec[] — flat arrays describing CRUD and query operations for each table. The consumer (hub/spoke) imports these specs, registers them in the @alkdev/operations registry along with handlers, and the operations runtime handles execution, call protocol, and subscriptions.

Storage depends on @alkdev/operations only as a type-level peer dependency (for the OperationSpec type). No circular dependency.

@alkdev/storage → defines types + tables + OperationSpec[] (type-only operations dep)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / spoke) → imports both, registers specs + handlers

Avoiding Circular Dependencies

Neither @alkdev/storage nor @alkdev/operations should depend on each other at runtime. Storage defines the schema types, database tables, and operation contract definitions; operations defines the call protocol and execution model. The consumer imports both and bridges them.

Open Questions

Open questions are tracked in open-questions.md. Key questions affecting this package:

• OQ-04: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
• OQ-22: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
• OQ-23: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
• OQ-26: Can Honker replace @alkdev/pubsub's Redis transport for single-node deployments? (resolved: yes — HonkerEventTarget adapter, POC validated. Redis still needed for multi-node. See ADR-047.)
• OQ-27: How are schema migrations applied across all tenant DBs? (open)
• OQ-28: How does cross-tenant delegation work with separate DBs? (open)

References

• Metagraph Module: metagraph-module.md
• Honker integration: honker-integration.md
• SQLite host tables: sqlite-host.md
• Schema evolution: schema-evolution.md
• Encrypted data: encrypted-data.md
• ACL graph: acl.md
• Forward-looking connections: forward-look.md
• Open questions: open-questions.md
• Honker source: /workspace/honker/
• Operations architecture: /workspace/@alkdev/operations/docs/architecture/
• Flowgraph architecture: /workspace/@alkdev/flowgraph/docs/architecture/