storage/docs/architecture/overview.md

status, last_updated

status	last_updated
draft	2026-05-31

@alkdev/storage — Overview

Typed graph storage with SQLite via Honker. Deno-first, published via JSR.

Purpose

@alkdev/storage provides a metagraph storage model: graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Instances of these type definitions become actual graphs populated with nodes and edges.

This pattern replaces domain-specific table proliferation with a small number of general-purpose tables that can model anything — call graphs, ACL rules, task dependencies, encrypted secrets, session trees, operation registries — while enforcing schema integrity through TypeBox validation.

The package also provides identity infrastructure tables (accounts, organizations, api_keys, audit_logs) for multi-tenant authentication and authorization, and integrates with Honker for transactional pub/sub, event streams, and task queues within the same SQLite database.

Architecture

@alkdev/storage/
├── mod.ts                   → re-exports graphs/ (zero db deps)
├── src/
│   ├── graphs/              → Metagraph Module, bridge functions, crypto (no db deps)
│   │   ├── modules/         → TypeBox Module definitions
│   │   │   ├── metagraph.ts → Config, BaseNode, BaseEdge
│   │   │   ├── call-graph.ts → CallGraph reference Module
│   │   │   ├── secret-graph.ts → SecretGraph reference Module
│   │   │   └── index.ts     → barrel re-export
│   │   ├── bridge.ts        → moduleToDbSchema, validateNode, validateEdge
│   │   ├── crypto.ts        → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│   │   └── mod.ts           → re-exports all graphs exports
│   └── sqlite/              → SQLite host (Drizzle + Honker)
│       ├── tables/          → drizzle table definitions
│       │   ├── common.ts    → commonCols
│       │   ├── identity/    → accounts, organizations, org_members, api_keys, audit_logs
│       │   ├── metagraph/   → graph_types, node_types, edge_types, graphs, nodes, edges
│       │   └── index.ts     → barrel re-export
│       ├── relations.ts     → drizzle relational mappings
│       ├── schema.ts        → barrel re-export
│       ├── adapter.ts       → Drizzle-Honker session adapter
│       └── client.ts        → createSystemDatabase(), createTenantDatabase()
└── test/
    └── reference-modules.test.ts → Metagraph, bridge, crypto tests

Subpath Exports (JSR/npm)

Export	Contents	Dependencies
@alkdev/storage	Graph schema types, Metagraph Module	@alkdev/typebox
@alkdev/storage/graphs	Same as . — alias for the main export	Same as .
@alkdev/storage/sqlite	SQLite tables, relations, client, adapter	@alkdev/drizzlebox, drizzle-orm, @russellthehipp/honker-node

The pg/ subpath has been removed (ADR-038). SQLite via Honker is the sole database host.

Database Model

System DB + Tenant DB

The package uses a two-database model for multi-tenant isolation (ADR-040):

Database	Contents	Purpose
system.db	accounts, organizations, organization_members, api_keys, audit_logs, system graph_types	Identity infrastructure, authentication, authorization anchors
tenant-{orgId}.db	graphs, nodes, edges, graph_types, node_types, edge_types	All graph data for one org — call graphs, ACL instances, session trees, secrets, tasks

The system DB is opened by createSystemDatabase(client). Each tenant DB is opened by createTenantDatabase(client). Both return typed Drizzle instances with their respective schemas attached.

Why separate files: File-level isolation means one tenant's data cannot leak to another, even via application bugs. Each tenant DB is independently backupable, migratable, and compactable. No orgId column is needed on tenant tables because the entire file IS the org scope.

Cross-DB references: The tenant DB's graphs.ownerId and graphs.projectId logically reference (not FK) the system DB's identity tables. The consumer enforces referential integrity at the application layer, consistent with ADR-020.

Graph Type Scope

Graph types have a scope column (ADR-043) controlling who can create and modify them:

Scope	Examples	Who can modify
system	acl, call-graph, secret, operation-registry, message-session	Setup/seeding only
tenant	Custom org graph types (sprint-board, etc.)	Org admins
user	Personal graph types (my-notes, etc.)	Creating user

System-scoped graph types are seeded during initialization. Their schemas are fixed — changes require a version bump and migration (ADR-029).

Terminology

Term	Definition
Metagraph	A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions.
Hub	The central service in the hub-spoke architecture. A consumer of @alkdev/storage — opens both system and tenant databases. The hub also depends on @alkdev/operations, @alkdev/pubsub, @alkdev/flowgraph.
Spoke	A local/embedded instance that runs per-project or per-session. A consumer of @alkdev/storage — opens a tenant database (or its own standalone DB for single-user mode).
System DB	The SQLite database holding identity/auth tables and system graph type definitions. One per deployment.
Tenant DB	The SQLite database holding all graph data for one organization. One per org.
Graph type	A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the graph_types table.
Node type	A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the node_types table.
Edge type	A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the edge_types table.
Graph instance	A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the graphs table.
Honker	SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same .db file.
Consumer	Code that imports @alkdev/storage (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers.
Repository layer	⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes.
Validation boundary	The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database.

Design Decisions

All design decisions are documented as ADRs in decisions/.

ADR	Decision	Summary
001	Deno-first, JSR publishes	Published to JSR; npm comes free via @jsr/alkdev__storage
002	Metagraph over domain-specific tables	6 general-purpose tables serve all graph-shaped domains
003	TypeBox Module as API surface	Type.Module() replaces SchemaBuilder; Metagraph.Import() + Type.Composite()
004	Injectable clients, no side effects	createSystemDatabase(client) / createTenantDatabase(client) take pre-created clients
005	Drizzle + TypeBox via drizzlebox	Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas
006	as const objects, not TypeScript enums	Avoids JSR slow-types; consistent pattern across codebase
007	No comments in code	Documentation lives in architecture docs and TypeBox descriptions
008	Common columns pattern	id, metadata, createdAt, updatedAt on every table
033	JSON path queries and hand-written CRUD for v1	Attribute queries use JSON path; CRUD is hand-written
038	SQLite-first, Postgres removed	Single database host; no dual maintenance
039	Honker as SQLite extension and transport	DB + pub/sub + queues + events in one SQLite file
040	System DB + tenant DB separation	Identity in system.db, graph data in tenant-{orgId}.db
041	Identity tables in storage package	accounts, organizations, api_keys, audit_logs defined in storage
042	Scoping columns on graph instances	ownerId, projectId on graphs table
043	Graph type scope	system / tenant / user scope controls who can modify graph types
044	Drizzle-Honker session adapter	~100-line adapter; no Drizzle fork; $client and $honkerTx for honker access
045	organization_members authoritative, BelongsToEdge derived	SQL table for fast lookups; ACL edge for traversal evaluation

Dependencies

Package	Purpose	Layer
@alkdev/typebox	Runtime schema validation	graphs/
@alkdev/drizzlebox	Generate TypeBox from Drizzle tables	sqlite/
drizzle-orm	ORM, table definitions, queries	sqlite/
@russellthehippo/honker-node	SQLite + pub/sub + queues + events	sqlite/

@alkdev/typebox and @alkdev/drizzlebox are npm packages (not yet on JSR). JSR handles npm dependencies natively.

Ecosystem packages are not runtime dependencies of @alkdev/storage. All ecosystem references describe consumer-side data shapes and integration patterns, not import dependencies.

What Exists vs. What's Needed

Implemented

• Metagraph Module (Type.Module with Config, BaseNode, BaseEdge entries)
• Bridge functions (moduleToDbSchema, validateNode, validateEdge)
• Reference graph type Modules (CallGraph, SecretGraph)
• Crypto utility (AES-256-GCM + PBKDF2, EncryptedDataSchema)
• SQLite host: 6 metagraph tables + actors placeholder + Drizzle relations + client factory
• TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
• Reference module tests (bridge functions, validation, Module composition)

Not Yet Implemented

Gap	Priority	Notes
Drizzle-Honker session adapter	High	~100-line adapter wrapping HonkerSQLiteSession + HonkerPreparedQuery. ADR-044.
Identity tables (accounts, organizations, api_keys, audit_logs, organization_members)	High	Moving from hub into storage. ADR-041.
Scoping columns on graphs table (ownerId, projectId)	High	ADR-042.
Graph type scope column	High	ADR-043.
Remove actors table and pg/ directory	High	ADR-035 (actors), ADR-038 (pg).
createSystemDatabase() / createTenantDatabase() factories	High	Split from current createSqliteDatabase().
Repository/CRUD layer	Medium	⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges.
ACL graph type	Medium	Access control as a metagraph. ADR-034.
Task graph type	Low	Informed by @alkdev/taskgraph's schemas.
Graphology bridge	Low	moduleToGraphology() and fromGraphologyExport() — Phase 4.

Ecosystem Integration

@alkdev/storage is a data layer package consumed by other packages in the @alkdev ecosystem. It does not depend on the hub — the dependency flows the other way.

Dependency Direction

@alkdev/pubsub          ← transport only (no storage dependency)
    ↑
@alkdev/operations      ← call protocol, registry, identity, access control
    ↑                   (depends on: @alkdev/pubsub, @alkdev/typebox)
@alkdev/flowgraph       ← call graph schema, operation graph, workflow templates
    ↑                   (depends on: @alkdev/operations [peer], @alkdev/typebox)
@alkdev/taskgraph       ← task dependency graph schema, cost-benefit analysis
                        (depends on: @alkdev/typebox)

@alkdev/storage         ← YOU ARE HERE — typed graph persistence + identity
                        (depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)

    ↑                       ↑
    |                       |
Hub / Spoke          Any consumer that needs
(consumes all)       persistent graph storage

Event-Driven Architecture with Honker

The @alkdev platform is event-driven. Honker provides the transport mechanism within each SQLite database:

Concern	Honker Primitive	Example
Fire-and-forget notifications	notify(channel, payload)	"graph:updated", "node:created"
Durable per-consumer delivery	stream(name).subscribe(consumer)	Call protocol events, audit trail
At-least-once background jobs	queue(name).enqueue(payload)	Key rotation, schema migration, retention cleanup
Leader election	tryLock(name, owner, ttl)	Only one hub instance runs the scheduler
Scheduled operations	scheduler().add(cron, handler)	Retention cleanup, key sweep, rate limit sweep

A single Drizzle transaction via the Honker adapter can insert graph data AND publish a notification AND enqueue a side-effect job — all committing atomically. No dual-write problem between data and events.

What Comes from Where

Concept	Source package	Storage's role
Call protocol events	@alkdev/operations	Storage persists the outcomes as graph nodes + publishes via Honker stream
Identity	@alkdev/operations	Storage stores identity in SQL accounts table + as PrincipalNode in ACL graphs
Access control	@alkdev/operations	Storage's AclGraph mirrors AccessControl schema as graph structure
Call graph schema	@alkdev/flowgraph	Storage persists in-memory shapes to the database
Task graph schema	@alkdev/taskgraph	Storage persists task dependency shapes
Event transport	Honker (within storage)	Replaces @alkdev/pubsub's Redis transport for single-node deployments

Repository Layer Bridging Pattern

The repository layer in @alkdev/storage provides typed CRUD — no @alkdev/operations dependency. A consumer-side bridging module wires CRUD functions into the operations registry when needed.

@alkdev/storage → defines types + tables + CRUD (no operations dependency)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas

Avoiding Circular Dependencies

Neither @alkdev/storage nor @alkdev/operations should depend on each other directly. Storage defines the schema types and database tables; operations defines the call protocol and execution model. The consumer imports both and bridges them.

Open Questions

Open questions are tracked in open-questions.md. Key questions affecting this package:

• OQ-04: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
• OQ-22: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
• OQ-23: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
• OQ-26: Can Honker replace @alkdev/pubsub's Redis transport for single-node deployments? (open)
• OQ-27: How are schema migrations applied across all tenant DBs? (open)
• OQ-28: How does cross-tenant delegation work with separate DBs? (open)

References

• Metagraph Module: metagraph-module.md
• Honker integration: honker-integration.md
• SQLite host tables: sqlite-host.md
• Schema evolution: schema-evolution.md
• Encrypted data: encrypted-data.md
• ACL graph: acl.md
• Forward-looking connections: forward-look.md
• Open questions: open-questions.md
• Honker source: /workspace/honker/
• Operations architecture: /workspace/@alkdev/operations/docs/architecture/
• Flowgraph architecture: /workspace/@alkdev/flowgraph/docs/architecture/