alkdev/storage
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Architect storage around SQLite+Honker: remove PG, add multi-tenant identity, scoping

Browse Source 
Reorient @alkdev/storage around a single SQLite database host with Honker
for pub/sub, event streams, and task queues. PostgreSQL is removed as a
target (ADR-038), eliminating dual schema maintenance and infrastructure
complexity. Honker provides DB + pubsub + queues in one .db file (ADR-039).

Add system/tenant DB model (ADR-040): identity tables in system.db, all
graph data in tenant-{orgId}.db files. Identity tables move from the hub
into storage (ADR-041). Scoping columns (ownerId, projectId) added to
graphs table (ADR-042). Graph types get scope (system/tenant/user) to
protect infrastructure schemas (ADR-043).

Define Drizzle-Honker session adapter (ADR-044): ~100-line adapter enabling
Drizzle typed queries and Honker pubsub/queue on a single connection with
transactional consistency.

Resolve OQ-03, OQ-04, OQ-19, OQ-21, OQ-22, OQ-23, OQ-24. Add new
open questions OQ-26 through OQ-29 for Honker integration specifics.

New docs: honker-integration.md (adapter, event patterns, migration).
Scrub all PG/jsonb/libsql references from existing spec docs.
This commit is contained in:
glm-5.1
2026-05-31 15:41:41 +00:00
parent 6b5f32bad4
commit 6aa2fcc6ff
19 changed files with 1446 additions and 515 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
AGENTS.md
View File

@@ -5,9 +5,10 @@ Project-specific guidance for agents working on this package.
## Project Overview
`@alkdev/storage` is a deno-first TypeScript package providing typed graph
storage with dual database hosts (SQLite for spokes, PostgreSQL for the hub). It
uses the metagraph pattern (graphTypes → nodeTypes → edgeTypes → typed graph
instances) from the earlier `@ade` prototype.
storage with SQLite via Honker. It uses the metagraph pattern (graphTypes →
nodeTypes → edgeTypes → typed graph instances) and includes identity tables
for multi-tenant auth. The system/tenant DB model separates identity
infrastructure from org-scoped graph data.
## Architecture Snapshot
@@ -25,12 +26,14 @@ instances) from the earlier `@ade` prototype.
│ │ ├── bridge.ts # moduleToDbSchema, validateNode, validateEdge
│ │ ├── crypto.ts # encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│ │ └── mod.ts # Re-exports all graphs exports
│ ├── sqlite/ # SQLite host (drizzle-orm/libsql)
│ │ ├── tables/ # Drizzle table definitions
│ │ ├── relations.ts # Drizzle relations
│ │ ├── schema.ts # Re-exports
│ │ └── client.ts # Injectable createSqliteDatabase()
│ └── pg/ # PostgreSQL host (NOT YET IMPLEMENTED)
│ └── sqlite/ # SQLite host (drizzle-orm + honker-node)
│ ├── tables/
│ │ ├── identity/ # accounts, organizations, org_members, api_keys, audit_logs
│ │ └── metagraph/ # graph_types, node_types, edge_types, graphs, nodes, edges
│ ├── relations.ts # Drizzle relations
│ ├── adapter.ts # Drizzle-Honker session adapter
│ ├── schema.ts # Re-exports
│ └── client.ts # createSystemDatabase(), createTenantDatabase()
└── test/
└── reference-modules.test.ts # Metagraph, bridge, crypto tests
```
@@ -38,12 +41,10 @@ instances) from the earlier `@ade` prototype.
### Subpath Exports (JSR/npm)
- `@alkdev/storage` → Metagraph Module, graph type definitions (zero deps)
- `@alkdev/storage/sqlite` → SQLite tables, relations, client (drizzle-orm +
libsql)
- `@alkdev/storage/pg` → PostgreSQL tables, relations, client (NOT YET
IMPLEMENTED)
- `@alkdev/storage/sqlite` → SQLite tables (metagraph + identity), relations,
client, Honker adapter (drizzle-orm + honker-node)
This design ensures consumers don't bundle database drivers they don't use.
PostgreSQL has been removed (ADR-038). SQLite via Honker is the sole database host.
## Key Decisions
@@ -56,10 +57,15 @@ This design ensures consumers don't bundle database drivers they don't use.
annotations impractical. We use `--allow-slow-types` on publish and
`"exclude": ["no-slow-types"]` in lint config. This is known technical debt —
can be tightened iteratively.
4. **Injectable clients**: `createSqliteDatabase(client)` takes a client, not
4. **Injectable clients**: `createSystemDatabase(client)` and
`createTenantDatabase(client)` take pre-created Honker clients, not
env vars. Module-level side effects are forbidden.
5. **Dependencies**: `@alkdev/typebox` and `@alkdev/drizzlebox` are npm deps
(not yet on JSR). This works fine — JSR handles npm dependencies natively.
6. **SQLite-only via Honker**: No PostgreSQL. Honker provides DB + pub/sub +
queues in a single SQLite file (ADR-038, ADR-039).
7. **System/tenant DB split**: Identity tables in `system.db`, graph data in
`tenant-{orgId}.db` (ADR-040).
## Commands
@@ -107,24 +113,27 @@ The codebase has diverged significantly from the originals:
See `docs/architecture/` for detailed specifications:
- `overview.md` — Package purpose, exports, design decisions, open questions
- `overview.md` — Package purpose, exports, database model, ecosystem integration
- `metagraph-module.md` — Graph type definitions as TypeBox Modules, data model,
naming conventions, implementation path
- `forward-look.md` — Connections to dbtype, graph pointers, ujsx universal IR
pipeline
- `honker-integration.md` — Drizzle-Honker adapter, event patterns, DB coordination
- `schema-evolution.md` — How graph type schemas evolve, TypeBox Value.Diff/Patch/Cast
for schema change detection and data migration
- `sqlite-host.md` — SQLite tables, relations, client factory, porting notes
- `sqlite-host.md` — SQLite tables (metagraph + identity), client factories
- `encrypted-data.md` — Encrypted data design, crypto utility, node type modeling
- `forward-look.md` — Connections to dbtype, graph pointers, ujsx universal IR
- `acl.md` — Access control graph, principal-agent framework, scoping
These docs describe what the package is AND what it's becoming. Items marked ⚠️
are not yet implemented.
## What's Not Done Yet
- `src/pg/` — PostgreSQL host (same table shapes, `pgTable` + `jsonb` +
`timestamp` + `pgEnum`)
- Repository/CRUD layer (currently only table definitions, no typed query
functions)
- Hub-specific tables (sessions, messages, parts, call graphs, tasks, etc.)
- Drizzle-Honker session adapter (`src/sqlite/adapter.ts`)
- Identity tables in `src/sqlite/tables/identity/` (accounts, organizations, etc.)
- Scoping columns on `graphs` table (`ownerId`, `projectId`)
- Graph type `scope` column on `graph_types` table
- Remove `actors` table and `src/pg/` directory
- Split client factory into `createSystemDatabase()` / `createTenantDatabase()`
- Repository/CRUD layer (typed query functions beyond table definitions)
- JSR publication setup (need to create scope/package on jsr.io first)

28
docs/architecture/README.md
View File

@@ -1,23 +1,30 @@
---
status: draft
last_updated: 2026-05-30
last_updated: 2026-05-31
---
# @alkdev/storage Architecture
Typed graph storage with dual database hosts. Deno-first, published via JSR.
Typed graph storage with SQLite via Honker. Deno-first, published via JSR.
## Current State
Storage has Phase 1-3 of the metagraph implementation complete: Metagraph Module, bridge functions, reference graph type Modules (CallGraph, SecretGraph), and crypto utility. Phase 4 (graphology bridge) is pending. The repository/CRUD layer, PostgreSQL host, and additional graph types remain to be implemented.
Storage has Phase 1-3 of the metagraph implementation complete: Metagraph Module,
bridge functions, reference graph type Modules (CallGraph, SecretGraph), and
crypto utility. Phase 4 (graphology bridge) is pending. The package is
transitioning to SQLite-only with Honker integration (ADR-038, ADR-039) and
adding identity tables for multi-tenant support (ADR-041). The repository/CRUD
layer, Drizzle-Honker adapter, identity tables, and additional graph types
remain to be implemented.
## Architecture Documents
| Document | Content | Status |
|----------|---------|--------|
| [overview.md](overview.md) | Package purpose, exports, dependencies, ecosystem integration | reviewed |
| [overview.md](overview.md) | Package purpose, exports, dependencies, database model, ecosystem integration | draft |
| [metagraph-module.md](metagraph-module.md) | TypeBox Module type system, bridge functions, implementation path | reviewed |
| [sqlite-host.md](sqlite-host.md) | SQLite tables, relations, client factory, PG porting notes | reviewed |
| [sqlite-host.md](sqlite-host.md) | SQLite tables (metagraph + identity), client factories, Honker adapter | draft |
| [honker-integration.md](honker-integration.md) | Drizzle-Honker adapter, event patterns, system/tenant DB coordination | draft |
| [schema-evolution.md](schema-evolution.md) | Value.Diff/Cast/Patch for schema migration, version strategy | reviewed |
| [encrypted-data.md](encrypted-data.md) | Crypto utility, encrypted node type, key management | reviewed |
| [forward-look.md](forward-look.md) | Pointers, dbtype, ujsx IR (conceptual, post-v1) | draft |
@@ -64,6 +71,14 @@ Storage has Phase 1-3 of the metagraph implementation complete: Metagraph Module
| [035](decisions/035-actors-become-acl-nodes.md) | Actors become ACL nodes, standalone table removed | Accepted |
| [036](decisions/036-principal-agent-as-delegation-edges.md) | Principal-agent as delegation edges with scope narrowing | Accepted |
| [037](decisions/037-setup-vs-runtime-separation.md) | Setup-time definitions seed graph types, runtime instances are separate | Accepted |
| [038](decisions/038-sqlite-first-pg-removed.md) | SQLite-first, Postgres removed | Accepted |
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [040](decisions/040-system-db-tenant-db.md) | System DB + tenant DB separation | Accepted |
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage package | Accepted |
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graph instances | Accepted |
| [043](decisions/043-graph-type-scope.md) | Graph type scope — system/tenant/user | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | organization_members authoritative, BelongsToEdge derived | Accepted |
### Open Questions
@@ -93,6 +108,7 @@ ADR documents use a separate `Status` field in their body: `Proposed`, `Accepted
- Source: `src/`
- AGENTS.md: `/workspace/@alkdev/storage/AGENTS.md`
- Flowgraph architecture (pattern reference): `/workspace/@alkdev/flowgraph/docs/architecture/`
- Honker source: `/workspace/honker/`
- Flowgraph architecture: `/workspace/@alkdev/flowgraph/docs/architecture/`
- ujsx architecture: `/workspace/@alkdev/ujsx/docs/architecture/`
- Operations architecture: `/workspace/@alkdev/operations/docs/architecture/`

71
docs/architecture/acl.md
View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-05-30
last_updated: 2026-05-31
---
# Access Control Graph
@@ -500,18 +500,14 @@ itself, with `actions: ["admin"]` for owners, `["manage"]` for admins.
## Scoping Model
### Two-Level Scoping
With the system/tenant DB model (ADR-040), ACL scoping is simplified:
The ACL system operates at two levels:
1. **Operation-level** (setup-time): `AccessControl` on `OperationSpec` defines
*what scopes and resource actions are required* to invoke an operation. This
is registered once when the operation is defined and changes infrequently.
This data lives in `@alkdev/operations` and the hub's `operations` table.
2. **Graph-level** (runtime): The ACL graph instance stores *who has what* and
*who delegates to whom*. This is queried at call time to resolve whether a
specific identity satisfies an operation's access control requirements.
- **One ACL graph instance per tenant DB** — The tenant DB is inherently
org-scoped. OQ-22 is resolved: each org gets its own ACL graph instance.
- **No cross-org scoping within a tenant** — The entire tenant DB is one org.
The ACL graph does not need `orgId` columns or cross-org filtering.
- **Cross-org delegation** requires the hub to mediate between tenant DBs
(OQ-28, open).
### How They Connect
@@ -538,9 +534,11 @@ The **evaluator** bridges the two:
### Org-Scoped Access
When a `BelongsToEdge` connects an account to an org, and the org has
`ScopesEdge` connections to resources, the account inherits org-level access
through its membership:
Within a tenant DB, ACL evaluation is straightforward — the entire DB is one
org. The PrincipalNode's `identityId` logically references `accounts.id` in the
system DB (ADR-041). When evaluating ACL, the hub reads the account's org
membership from the system DB's `organization_members` table (authoritative per
ADR-045) and the ACL graph's `BelongsToEdge` (derived) in the tenant DB.
```
Account PrincipalNode ──belongs_to──→ Org PrincipalNode
@@ -565,29 +563,20 @@ own base scopes limit what they can exercise from org membership.
### The Disconnected `actors` Table
The `actors` table in `src/sqlite/tables/actors.ts` is replaced by
`PrincipalNode` in the ACL graph. The `ACTOR_TYPE` enum (`Human`, `Llm`,
`Agent`) maps to `identityType` values (`account`, `service`, `account` — LLMs
are accounts in the hub model per ADR-012). The standalone table has no
foreign key relationships and was explicitly deferred pending ACL design (OQ-03).
`PrincipalNode` in the ACL graph. The standalone table has been removed
(ADR-035, ADR-038).
This does **not** mean the hub's `accounts` table is replaced. The hub's
`accounts` table remains the authoritative identity store with email, access
level, and Gitea linking. `PrincipalNode` in the ACL graph **references** the
account by `identityId` but does not duplicate its columns. The ACL graph
stores *authorization* data; the hub's identity tables store *authentication*
data.
### Hub's `organization_members` as Authoritative Source
### Hub's `organization_members` as a Source
The hub's `organization_members` table is the authoritative source for who
belongs to which org. When org membership changes, the hub updates both:
The hub's (now storage's) `organization_members` table is the authoritative
source for who belongs to which org (ADR-045). When org membership changes,
the consumer writes both:
1. The `organization_members` row (fast lookup, FK constraints)
2. The `BelongsToEdge` in the ACL graph instance (graph traversal, evaluation)
This dual-write is necessary because the hub needs fast SQL lookups for
membership checks (e.g., "list all members of this org"), while the ACL graph
needs the edge for traversal-based evaluation (e.g., "compute effective scopes
for this account across all orgs").
This dual-write is necessary because the SQL table provides O(1) membership
lookups and cascade behavior, while the ACL graph needs the edge for
traversal-based evaluation.
### Hub's Permission Resolution
@@ -608,15 +597,17 @@ the effective scope is `dev:read`.
- **OQ-20**: Should `DelegatesEdge` support temporary delegation with
expiration? (Referenced in [open-questions.md](open-questions.md))
- **OQ-21**: Should the ACL evaluator live in `@alkdev/storage` or in the hub?
- **OQ-25**: What are the scope string semantics for subset validation?
(Referenced in [open-questions.md](open-questions.md))
- **OQ-22**: How are ACL graph instances created and managed? (Referenced in
[open-questions.md](open-questions.md))
- **OQ-23**: Should `BelongsToEdge` be derived (materialized from
`organization_members`) or primary (ACL graph is the source of truth)?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
(Referenced in [open-questions.md](open-questions.md))
- **OQ-24**: How does `identityId` reference hub entities without creating a
package dependency? (Referenced in [open-questions.md](open-questions.md))
## Resolved Questions
- **OQ-21** (ACL evaluator location): Storage provides traversal primitives; hub composes with operations. Simplified by single-host model — no cross-DB joins needed within a tenant DB.
- **OQ-22** (ACL graph instance lifecycle): One per tenant DB. ADR-040.
- **OQ-23** (BelongsToEdge derivation): Derived from `organization_members`. ADR-045.
- **OQ-24** (identityId reference): Logical reference to `accounts.id` in system DB. ADR-041.
## References

53
docs/architecture/decisions/038-sqlite-first-pg-removed.md Normal file
View File

@@ -0,0 +1,53 @@
# ADR-038: SQLite-First, Postgres Removed
## Status
Accepted
## Context
The original architecture specified two database hosts: SQLite for spokes (local/embedded) and PostgreSQL for the hub (central service). This required:
- Maintaining two sets of Drizzle table definitions (`sqliteTable` and `pgTable`) with the same logical shapes
- Two client factories (`createSqliteDatabase`, `createPostgresDatabase`)
- Two repository layer implementations or a host-agnostic abstraction
- Separate test suites for each host
- A PostgreSQL server as infrastructure dependency for any hub deployment
The dual-host model came from the `@ade` POC, which was single-tenant and didn't account for multi-tenant deployment concerns. For the actual use case — small teams of developers and AI agents sharing compute — PostgreSQL is operational overhead without proportional benefit.
## Decision
`@alkdev/storage` is SQLite-only. The `pg/` subpath export is removed. The package provides one database host: SQLite via the Honker extension (see ADR-039).
This eliminates:
- All `pgTable` definitions and the `src/pg/` directory
- The PostgreSQL porting notes in every spec document
- Dual schema maintenance, dual testing, dual repository implementations
- PostgreSQL and Redis as infrastructure dependencies
## Consequences
**Positive:**
- Single set of table definitions, one client factory, one test suite
- No PostgreSQL server to install, configure, secure, and maintain
- No Redis for pub/sub — Honker provides durable pub/sub within SQLite
- Simpler deployment: a single `.db` file per database
- The hub's domain tables can coexist with metagraph tables in the same SQLite file
- WAL mode with Honker's reader pool provides sufficient concurrency for the expected workload
**Negative:**
- SQLite is single-machine — no horizontal scaling, no read replicas, no cross-server queries
- No native `jsonb` type with GIN indexes — JSON attributes rely on `json_extract()` queries
- No built-in full-text search on JSON attributes (SQLite FTS5 works but requires manual setup)
- Some ecosystem tools expect PostgreSQL (migration tools, monitoring dashboards)
- If a future deployment genuinely needs PostgreSQL scale, a migration path would need to be rebuilt
## References
- ADR-039: Honker as SQLite extension and pub/sub transport
- ADR-040: System DB + tenant DB separation
- ADR-018 (superseded): dbtype integration was partly motivated by PG/SQLite dual maintenance; with PG removed, this pressure is reduced

57
docs/architecture/decisions/039-honker-as-sqlite-extension.md Normal file
View File

@@ -0,0 +1,57 @@
# ADR-039: Honker as SQLite Extension and Transport
## Status
Accepted
## Context
The hub architecture was designed around three separate infrastructure components:
1. **PostgreSQL** — persistence (tables, queries, transactions)
2. **Redis** — pub/sub transport for event-driven communication
3. **Application-level task queues** — background job processing
This creates operational complexity (three services to deploy, monitor, and secure) and a dual-write problem: writing data to PostgreSQL and publishing events to Redis cannot happen in a single transaction. If the process crashes between the DB commit and the Redis publish, data and events become inconsistent.
Honker (`@russellthehippo/honker-node`) is a SQLite extension that adds Postgres-style `NOTIFY`/`LISTEN` semantics, durable event streams with per-consumer offsets, at-least-once work queues with retries and dead-letter handling, cron scheduling, advisory locks, and rate limiting — all within the same SQLite `.db` file.
## Decision
`@alkdev/storage` uses Honker as its SQLite extension and transport layer. Honker provides:
1. **Database operations** — SQLite with WAL mode, a bounded reader pool, and a single writer slot
2. **Ephemeral pub/sub** — `notify()`/`listen()` for fire-and-forget notifications within the DB transaction
3. **Durable event streams** — `stream()` with per-consumer offset tracking for replay-safe delivery
4. **Task queues** — `queue()` with at-least-once claims, retries, priority, delayed jobs, and dead-letter
5. **Advisory locks** — `tryLock()` for leader election and exclusive access
6. **Cron scheduling** — `scheduler()` for time-triggered operations
Drizzle ORM integrates with Honker via a thin session adapter (~100 lines) that wraps Honker's `query()`/`execute()` API inside Drizzle's `SQLiteSession<'sync'>` contract. No Drizzle fork required. The adapter exposes the Honker `Database` as `$client` on the Drizzle instance for direct access to pubsub/queue features.
## Consequences
**Positive:**
- **Transactional consistency**: `INSERT INTO nodes` and `queue.enqueue()` commit atomically. No dual-write problem.
- **No Redis dependency**: Honker's `stream()` replaces Redis as the durable pub/sub transport
- **No PostgreSQL dependency**: SQLite with Honker covers persistence + events + queues
- **Operational simplicity**: One `.db` file contains everything — data, events, queues, schedules
- **Drizzle integration**: Full Drizzle type safety for queries + Honker for pubsub/queue on the same connection
- **Ecosystem fit**: The @alkdev platform is event-driven. Honker's durable streams with per-consumer offsets map directly to `@alkdev/operations`' call protocol events and `@alkdev/flowgraph`'s event-sourced model
**Negative:**
- **Single-machine**: Honker is a single-process SQLite extension. No cross-server events. For multi-node deployment, a separate transport (Redis, NATS) would still be needed for internode communication.
- **`lastInsertRowid` overhead**: Honker's `execute()` returns only affected row count. Getting `lastInsertRowid` requires an extra `SELECT last_insert_rowid()` call via napi, or a small Rust addition to honker-node's Transaction class.
- **No prepared statement handles at JS level**: Every Drizzle query goes through `query(sql, params)`. Mitigated by honker-core's `prepare_cached` on the Rust side.
- **Honker is alpha software**: Not yet beta-quality. API may change. Risk mitigated by the thin adapter — if Honker's query API changes, only the adapter needs updating.
## References
- Honker source: `/workspace/honker/`
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- ADR-038: SQLite-first, Postgres removed
- ADR-004: Injectable clients, no side effects (Honker client is injectable)
- `@alkdev/operations` call protocol architecture
- `@alkdev/pubsub` — Honker may replace or supplement the Redis transport

69
docs/architecture/decisions/040-system-db-tenant-db.md Normal file
View File

@@ -0,0 +1,69 @@
# ADR-040: System DB + Tenant DB Separation
## Status
Accepted
## Context
The original POC was single-tenant — no concept of users, organizations, or ownership. The 6 metagraph tables had no columns for scoping graph instances to any owner, org, or project.
Multi-tenant support is needed for:
- Sharing compute with other OSS developers while keeping data isolated
- Enabling downstream users to host multi-tenant services
- Self-hosted deployments where org isolation is required
Three approaches to multi-tenancy in SQLite:
1. **Schema-level isolation** — All tenants in one `.db` file, with `orgId` columns on every table for row-level filtering
2. **Database-level isolation** — Each tenant gets its own `.db` file
3. **Hybrid** — Shared identity tables in one file, tenant data in per-tenant files
## Decision
Use the hybrid approach: a **system DB** for identity/auth and a **tenant DB** per organization for all graph data.
```
system.db
├── accounts, organizations, api_keys, audit_logs
├── graph_types (system-scoped definitions: acl, call-graph, etc.)
├── _honker_* tables (system events, queues, streams)
tenant-{orgId}.db
├── graphs, nodes, edges (ALL graph instances for this org)
├── graph_types (tenant-scoped definitions: custom graphs)
├── node_types, edge_types
├── projectId columns on graphs for intra-org project scoping
├── _honker_* tables (per-org events, queues, streams)
```
The system DB holds identity infrastructure that must exist before any tenant can be authenticated. The tenant DB holds all graph data for one org — call graphs, ACL instances, session trees, task dependencies, secrets. Tenant DBs are isolated at the file level: backup, delete, migrate, or corrupt one tenant without affecting others.
The hub (or any consumer) opens both a system connection and one or more tenant connections. The system DB's `accounts` and `organizations` tables are the authoritative source for authentication. The tenant DB's graph data is scoped by `ownerId` and `projectId` columns that logically reference (not FK) the system DB's identity tables.
## Consequences
**Positive:**
- File-level isolation — one tenant's data cannot leak to another, even via bugs in application-layer filtering
- Each tenant DB is independently backupable, migratable, compactable
- No `orgId` column needed on tenant tables (the entire file IS the org scope)
- Simpler queries — no row-level filtering on every query for multi-tenancy
- Natural fit for the "compute sharing" use case — separate files for separate people
- System DB is small and rarely changes — low backup cost, high durability focus
- Honker's pubsub/queues are per-DB — event streams don't cross tenant boundaries
**Negative:**
- Cross-tenant operations (e.g., a user in org A delegates to a user in org B) require the hub to mediate between two open databases at the application layer
- No cross-tenant SQL JOINs — if needed, the hub does application-level joins
- More open file handles — one per active tenant (manageable for expected scale)
- Schema migrations must be applied to each tenant DB independently
- System DB is a single point of failure — if it's corrupted, all tenants lose authentication
## References
- ADR-038: SQLite-first, Postgres removed
- ADR-039: Honker as SQLite extension
- ADR-041: Identity tables in storage package
- ADR-042: Scoping columns on graph instances

56
docs/architecture/decisions/041-identity-tables-in-storage.md Normal file
View File

@@ -0,0 +1,56 @@
# ADR-041: Identity Tables in Storage Package
## Status
Accepted
## Context
The hub currently defines identity tables (`accounts`, `organizations`, `api_keys`, `audit_logs`, `organization_members`) in its own `src/storage/tables/` directory. The storage package provides only the 6 metagraph tables.
This separation creates problems:
1. The system DB (ADR-040) needs identity tables, but `@alkdev/storage` doesn't provide them
2. The hub has to maintain its own Drizzle table definitions separately, duplicating the common columns pattern and drizzlebox integration
3. The scoping columns on the metagraph tables (ADR-042) logically reference identity table rows, but those tables are defined elsewhere
4. Any consumer that wants multi-tenant graph storage needs the identity tables too — they're not hub-specific, they're infrastructure
The identity tables are NOT graph-shaped (ADR-002 established the metagraph for graph-shaped data). They are relational records with fixed schemas, indexed lookups (email uniqueness, key hash lookup), and FK constraints. But they ARE required by any deployment that uses the system/tenant DB model.
## Decision
The identity tables move into `@alkdev/storage/sqlite`:
- `accounts` — hub-local identity records
- `organizations` — top-level grouping for multi-tenancy
- `organization_members` — account/org membership (note: also modeled as `BelongsToEdge` in ACL graphs, but the SQL table provides fast indexed lookups that graph traversal cannot match)
- `api_keys` — keypal-managed API key storage
- `audit_logs` — append-only security event trail
These tables are in the storage package because they are **database infrastructure**, not hub business logic. The hub consumes them; it does not own their schema.
`organization_members` remains a SQL table despite `BelongsToEdge` existing in the ACL graph (ADR-034). The SQL table provides O(1) lookups for "list all members of org X" and FK constraints for cascading behavior. The ACL graph provides traversal-based evaluation for permission resolution. Both are needed — the SQL table is authoritative for membership state; the ACL graph edge is derived (OQ-23 resolved as "derived").
## Consequences
**Positive:**
- Single package provides everything needed for a multi-tenant graph database: metagraph tables + identity tables
- The hub doesn't duplicate table definitions or common column patterns
- `createSystemDatabase(client)` returns a fully-typed Drizzle instance with identity tables
- Other consumers (spokes, tools, standalone services) get identity tables without depending on the hub
- Referential integrity within the system DB is consistent — FK constraints work
**Negative:**
- `@alkdev/storage` grows in scope — it's no longer just "graph storage" but also "identity infrastructure"
- The hub loses ownership of its table schemas — changes require coordination with storage
- `organization_members` existing as both a SQL table and ACL graph edges requires a dual-write contract (the SQL table is authoritative; the ACL edge is derived)
- `api_keys` is tightly coupled to keypal's `Storage` interface — storage must not import keypal directly; the hub provides the adapter
## References
- ADR-040: System DB + tenant DB separation
- ADR-034: ACL as metagraph
- ADR-002: Metagraph over domain-specific tables
- Hub identity tables: `/workspace/@alkdev/hub/docs/architecture/storage/identity.md`

63
docs/architecture/decisions/042-scoping-columns-on-graphs.md Normal file
View File

@@ -0,0 +1,63 @@
# ADR-042: Scoping Columns on Graph Instances
## Status
Accepted
## Context
The original `graphs` table had no concept of ownership, organization, or project. A `graph` row was identified by `id` and `name` with no way to answer "which org owns this call graph?" or "list all graphs for this project."
In the system/tenant DB model (ADR-040), the tenant DB is inherently org-scoped (the entire `.db` file is one org). But within a tenant DB, graph instances still need to be scoped to:
- **An owner** — which account created/owns this graph
- **A project** — which project this graph belongs to (for project-scoped graphs like call graphs and session trees)
The metagraph pattern stores node/edge attributes as JSON, but scoping columns must be real columns because they appear in WHERE clauses, JOIN conditions, and need indexes.
## Decision
Add `ownerId` and `projectId` columns to the `graphs` table:
```
graphs {
...commonCols,
graphTypeId,
name,
description,
status,
ownerId, -- TEXT, nullable — logical reference to accounts.id in system DB
projectId, -- TEXT, nullable — logical reference to projects (graph or domain table)
}
```
No `orgId` column — the tenant DB itself IS the org scope. Adding `orgId` would be redundant within a single-tenant DB file.
These are **logical references** consistent with ADR-020 (no nodeTypeId on nodes) and OQ-24 (identityId as logical reference). No FK constraint because the referenced tables live in a different database file (system DB). The hub/consumer enforces referential integrity at the application layer.
**Nullability semantics**:
- `ownerId` NULL — system-owned graph (e.g., the ACL graph type definition seeded at setup). Not associated with any account.
- `projectId` NULL — org-level graph (e.g., the org's ACL instance). Not scoped to a specific project.
**Indexes**: `idx_graphs_owner_id` on `(ownerId)`, `idx_graphs_project_id` on `(projectId)`, `idx_graphs_owner_id_project_id` on `(ownerId, projectId)` for combined lookups.
## Consequences
**Positive:**
- "List all graphs for project X" is a simple indexed query, not a JSON path extraction
- "Who owns this graph?" is a column read, not a traversal
- Consistent with the rule from the hub's architecture: "if a field appears in WHERE clauses, JOIN conditions, or needs a constraint, it should be a proper column — not buried in metadata or JSON"
- No FK constraints means no cross-DB coupling — the tenant DB works without the system DB open
**Negative:**
- Orphaned graphs possible if an account is deleted in the system DB but the tenant DB's `graphs.ownerId` still references it. Application-layer cleanup required.
- Adding columns to the `graphs` table is a schema change that affects all consumers. The columns are nullable to ease the transition.
## References
- ADR-040: System DB + tenant DB (explains why no `orgId`)
- ADR-020: No nodeTypeId on nodes (same logical-reference pattern)
- ADR-008: Common columns pattern

63
docs/architecture/decisions/043-graph-type-scope.md Normal file
View File

@@ -0,0 +1,63 @@
# ADR-043: Graph Type Scope — System vs Tenant vs User
## Status
Accepted
## Context
Graph type definitions (in `graph_types`) were originally unscoped — any consumer could create any graph type. But some graph types should be protected:
- The ACL graph type should not be modifiable by regular users — its schema (PrincipalNode, DelegatesEdge, etc.) is a system contract
- The call-graph and message-session types are infrastructure — their schemas should not change at runtime
- Custom graph types (task boards, project workflows) SHOULD be user-definable
Without a scope distinction, a tenant user could modify the ACL graph type's schema, potentially breaking the access control system.
## Decision
Add a `scope` column to `graph_types`:
```
graph_types {
...commonCols,
name, description, config, version,
scope, -- TEXT NOT NULL, enum: "system" | "tenant" | "user"
}
```
| Scope | Who can create | Who can modify the schema | Who can create instances |
|-------|---------------|---------------------------|------------------------|
| `system` | Setup/seeding only | Setup/seeding only (version bumps only) | Any authenticated user (within access control) |
| `tenant` | Org admins | Org admins | Org members |
| `user` | Any user | The user who created the type | The user who created the type |
**System-scoped types**: `acl`, `call-graph`, `secret`, `operation-registry`, `message-session`. These are seeded during hub initialization. Their schemas are fixed — changes require a version bump and migration (ADR-029).
**Tenant-scoped types**: Custom graph types created by org admins for the org's use. E.g., a "sprint-board" type for task tracking.
**User-scoped types**: Personal graph types for individual workflows. E.g., a "my-notes" type.
The repository layer enforces scope constraints at creation and modification time. System-scoped types cannot be modified through the repository API. Tenant and user-scoped types can be modified by authorized principals only.
## Consequences
**Positive:**
- ACL graph type schema is protected from accidental or malicious modification
- Clear authorization model for graph type management
- Downstream users can define custom graph types without risking system infrastructure
- Consistent with ADR-037 (setup-time definitions seed graph types)
**Negative:**
- The repository layer needs to know about scope rules (a small amount of domain knowledge in storage)
- If a system graph type genuinely needs a schema change, it must go through a version bump and migration — not just an API call
- The scope column must be indexed and checked on every graph type mutation
## References
- ADR-037: Setup-time definitions seed graph types
- ADR-029: Version as breaking-change signal
- ADR-034: ACL as metagraph
- ADR-042: Scoping columns on graph instances

76
docs/architecture/decisions/044-drizzle-honker-adapter.md Normal file
View File

@@ -0,0 +1,76 @@
# ADR-044: Drizzle-Honker Session Adapter
## Status
Accepted
## Context
Drizzle ORM provides typed query builders for SQLite via driver packages (`drizzle-orm/better-sqlite3`, `drizzle-orm/libsql`). Honker provides SQLite with pubsub/queue extensions but is not a Drizzle driver.
Running both simultaneously against the same `.db` file would require two separate connections, losing the transactional consistency that Honker provides (data writes + event notifications in one transaction).
## Decision
Implement a thin session adapter (~100 lines) that wraps Honker's Node.js `Database.query()`/`Transaction.execute()` API inside Drizzle's `SQLiteSession<'sync'>` contract. No Drizzle fork required.
The adapter implements:
- `HonkerSQLiteSession` — extends `SQLiteSession<'sync', HonkerRunResult, ...>`. Implements `prepareQuery()` and `transaction()`.
- `HonkerPreparedQuery` — extends `SQLitePreparedQuery`. Implements `run()`, `all()`, `get()`, `values()`.
- `HonkerSQLiteTransaction` — extends `SQLiteTransaction<'sync', ...>`. Provides `$honkerTx` for access to Honker's transaction methods.
- `drizzle(client, config)` — factory function that creates a Drizzle database instance from a Honker `Database`.
**Key integration points**:
- `prepareQuery()` delegates to `honkerDb.query(sql, params)` for reads and `tx.execute(sql, params)` for writes
- `transaction()` wraps honker's explicit begin/commit/rollback in Drizzle's callback pattern
- `$client` on the database instance exposes the Honker `Database` for `notify()`, `queue()`, `stream()`, `listen()`, `scheduler()`
- `$honkerTx` on the transaction object exposes the Honker `Transaction` for `notify()` and `enqueueTx()` within a Drizzle transaction callback
- `run()` returns `{ changes, lastInsertRowid }` where `lastInsertRowid` is obtained via `SELECT last_insert_rowid()` (pending a small Rust addition to honker-node for zero-overhead access)
**Usage**:
```typescript
import { open } from '@russellthehippo/honker-node';
import { drizzle } from '@alkdev/storage/sqlite/honker-adapter';
import * as schema from './schema';
const honkerDb = open('app.db');
const db = drizzle(honkerDb, { schema });
// Drizzle typed queries
const activeGraphs = db.select().from(schema.graphs)
.where(eq(schema.graphs.status, 'active'));
// Honker + Drizzle in one transaction
db.transaction((tx) => {
tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
tx.$honkerTx.notify('graph:updated', { graphId });
});
```
## Consequences
**Positive:**
- Single connection — Drizzle queries and Honker pubsub/queue share the same SQLite file and transaction context
- No Drizzle fork — the adapter is consumer-side code (~100 lines)
- Full Drizzle type safety for all queries
- Full Honker feature access via `$client` and `$honkerTx`
- Transactional consistency: data writes and event notifications commit atomically
- The adapter could be open-sourced independently as `drizzle-honker` for community use
**Negative:**
- `lastInsertRowid` requires an extra query until honker-node adds a native method
- No prepared statement handle reuse at the JS level (mitigated by Rust-side `prepare_cached`)
- The adapter depends on Drizzle's internal SQLite session API path (`drizzle-orm/sqlite-core/session`) — if Drizzle restructures, the adapter needs updating
- Object-to-array conversion for Drizzle's `values()` method relies on JS object property insertion order, which matches SQLite column order but is a subtle dependency
## References
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- Drizzle SQLite session: `/workspace/drizzle-orm/src/sqlite-core/session.ts`
- ADR-039: Honker as SQLite extension
- ADR-004: Injectable clients (adapter pattern is consistent)

42
docs/architecture/decisions/045-org-members-authoritative-belongsto-derived.md Normal file
View File

@@ -0,0 +1,42 @@
# ADR-045: Organization Members as Authoritative SQL Table, BelongsToEdge as Derived
## Status
Accepted
## Context
OQ-23 asked whether `BelongsToEdge` in the ACL graph should be derived (materialized from `organization_members`) or primary (ACL graph is the source of truth).
The ACL graph needs `BelongsToEdge` for traversal-based permission evaluation (ADR-034). The hub needs `organization_members` for fast SQL lookups ("list all members of org X", FK constraints on cascading behavior).
Two sources of truth for the same data creates a consistency risk.
## Decision
`organization_members` is the authoritative SQL table. `BelongsToEdge` in the ACL graph is derived.
When org membership changes, the consumer (hub) writes to `organization_members` first, then creates or removes the corresponding `BelongsToEdge` in the ACL graph instance. The ACL edge mirrors the SQL table; it does not define it.
If the two fall out of sync, the SQL table is the source of truth. An audit/reconciliation process can re-derive ACL edges from the SQL table.
## Consequences
**Positive:**
- Clear authority — one write path for membership, one derived read path for ACL traversal
- FK constraints on `organization_members` work (cascade delete when org or account is removed)
- Fast indexed lookups for membership lists — no graph traversal needed
- ACL evaluator can still traverse `BelongsToEdge` for permission resolution
- Reconciliation is straightforward — scan `organization_members`, compare against ACL edges, fix discrepancies
**Negative:**
- Dual-write contract — the consumer must write both places. If the ACL edge write fails after the SQL write, they're out of sync.
- The ACL graph is not self-contained for org membership — it depends on an external table
## References
- ADR-034: ACL as metagraph
- ADR-041: Identity tables in storage package
- OQ-23: BelongsToEdge derivation (now resolved)

4
docs/architecture/encrypted-data.md
View File

@@ -1,6 +1,6 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# Encrypted Data

15
docs/architecture/forward-look.md
View File

@@ -1,6 +1,6 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# Forward Look: Pointers, dbtype, and Universal IR
@@ -162,12 +162,15 @@ integration is deferred because:
- dbtype is Phase 0 (architecture complete, no implementation)
- The manual defs work and are well-understood
- The Module pattern for graph types can be adopted independently (no dbtype
dependency)
dependency)
- With PostgreSQL removed (ADR-038), the original pressure for dbtype —
eliminating dual SQLite/PG table maintenance — is significantly reduced.
There is now only one set of table definitions to maintain.
When dbtype reaches Phase 1 (implementation), storage can migrate from
Drizzle table definitions to dbtype elements one table at a time. The
Module-based graph type definitions are already compatible — they're both
TypeBox `Type.Module` objects.
Drizzle table definitions to dbtype elements one table at a time. The Module-based
graph type definitions are already compatible — they're both TypeBox `Type.Module`
objects.
## ujsx as Universal IR

377
docs/architecture/honker-integration.md Normal file
View File

@@ -0,0 +1,377 @@
---
status: draft
last_updated: 2026-05-31
---
# Honker Integration
How @alkdev/storage integrates with Honker for SQLite database operations,
transactional pub/sub, durable event streams, and task queues.
## Purpose
Honker provides SQLite with built-in pub/sub, event streams, work queues,
advisory locks, and cron scheduling — all within the same `.db` file. This
eliminates the need for separate PostgreSQL and Redis deployments and solves
the dual-write problem between data writes and event publishing.
This document specifies:
- The Drizzle-Honker session adapter architecture (ADR-044)
- The event-driven patterns enabled by Honker
- Transaction coordination between Drizzle and Honker
- The system/tenant DB model for Honker-managed databases
- Migration from the previous PostgreSQL + Redis architecture
## The Adapter
### Architecture
```
Consumer code
├── db.select().from(graphs)... → Drizzle query builder
│ └── HonkerPreparedQuery.run/all/get/values
│ └── honkerDb.query(sql, params) / tx.execute(sql, params)
├── db.$client.notify('channel', p) → Honker ephemeral pub/sub
├── db.$client.queue('jobs') → Honker work queue
├── db.$client.stream('events') → Honker durable stream
└── db.$client.listen('channel') → Honker listener
```
The adapter wraps Honker's `Database` inside Drizzle's `SQLiteSession<'sync'>`
contract. Drizzle handles typed queries; Honker handles pubsub/queue/stream
primitives. They share the same SQLite connection and transaction context.
### Adapter Components
| Component | Extends | Purpose |
|-----------|---------|---------|
| `HonkerSQLiteSession` | `SQLiteSession<'sync'>` | Routes Drizzle queries to `honkerDb.query()` / `tx.execute()` |
| `HonkerPreparedQuery` | `SQLitePreparedQuery` | Implements `run()`, `all()`, `get()`, `values()` via Honker |
| `HonkerSQLiteTransaction` | `SQLiteTransaction<'sync'>` | Drizzle transaction with `$honkerTx` for Honker access |
| `drizzle(client, config)` | — | Factory: creates `BaseSQLiteDatabase` from Honker client |
### Key Integration Points
**Reading** — `all()` and `get()` delegate to `honkerDb.query(sql, params)`,
which uses the reader pool. Honker returns `Array<Record<string, any>>`
(row objects). The adapter converts to Drizzle's column format via
`mapResultRow()`.
**Writing** — `run()` delegates to `tx.execute(sql, params)` which acquires the
writer slot. For standalone writes (no explicit transaction), the adapter
creates a temporary transaction, executes, and commits.
**Transactions** — Drizzle's callback-based `db.transaction((tx) => ...)` wraps
honker's explicit `begin/commit/rollback`:
```ts
transaction(callback) {
const honkerTx = this.client.transaction();
const txSession = new HonkerTxSession(honkerTx, ...);
const drizzleTx = new HonkerSQLiteTransaction('sync', ..., txSession, ...);
try {
const result = callback(drizzleTx);
honkerTx.commit();
return result;
} catch (e) {
honkerTx.rollback();
throw e;
}
}
```
**Nested transactions** — Savepoints via raw SQL:
```ts
// Inside a Drizzle transaction callback
drizzleTx.transaction((nestedTx) => {
// Adapter fires: tx.execute('SAVEPOINT sp0')
// ... queries ...
// Adapter fires: tx.execute('RELEASE sp0')
});
```
### Accessing Honker from Drizzle
The adapter exposes two access points:
1. **`db.$client`** — The Honker `Database` instance, for use outside transactions:
```ts
const db = drizzle(honkerDb, { schema });
db.$client.notify('graph:created', { graphId });
```
2. **`tx.$honkerTx`** — The Honker `Transaction` instance, for use within
Drizzle transaction callbacks:
```ts
db.transaction((tx) => {
tx.insert(schema.nodes).values({ graphId, key: 'call-1', attributes: {} }).run();
tx.$honkerTx.notify('nodes:created', { graphId, key: 'call-1' });
// Both commit atomically — data + notification in one transaction
});
```
### Known Limitations
| Limitation | Impact | Mitigation |
|------------|--------|------------|
| No `lastInsertRowid` from `execute()` | `run()` needs extra `SELECT last_insert_rowid()` | Small Rust PR to honker-node |
| No prepared statement handle at JS level | Every query re-enters napi | Rust-side `prepare_cached` amortizes |
| Object rows only, no raw arrays | `values()` must convert `Object.values()` | Column order preserved from SQLite |
| Drizzle internal API dependency | Adapter imports from `drizzle-orm/sqlite-core/session` | Stable across minor versions |
## Event-Driven Patterns
### Ephemeral Notifications
Fire-and-forget events for signaling. Like PostgreSQL's `NOTIFY/LISTEN`.
```ts
// After creating a graph, notify listeners
db.transaction((tx) => {
tx.insert(schema.graphs).values({ name: 'session-1', status: 'draft' }).run();
tx.$honkerTx.notify('graphs:created', { name: 'session-1' });
});
// Listener (separate process or same process)
const listener = db.$client.listen('graphs:created');
for await (const event of listener) {
console.log('New graph:', event.payload);
}
```
**Use cases**: Cache invalidation signals, UI update push, spoke reconnection
triggers.
### Durable Event Streams
Per-consumer replay-safe delivery. Each consumer tracks its own offset.
Crash recovery replays from the last saved offset.
```ts
// Publish call protocol events to a durable stream
const callStream = db.$client.stream('call-protocol');
callStream.publish({ type: 'call.requested', requestId: 'req-1', operationId: 'op-call' });
// Consumer: dashboard replays from its last offset
const subscription = callStream.subscribe('dashboard');
for await (const event of subscription) {
// event has: id, payload, timestamp
updateDashboard(event.payload);
// offset auto-saved every 1000 events or 1 second
}
```
**Use cases**: Call protocol event persistence, audit trail replay, session
event replay, cross-process coordination.
### Transactional Outbox
Combine data writes with side-effect delivery. A queue job is enqueued in the
same transaction as the data write, guaranteeing the side effect will eventually
be processed.
```ts
// Insert a node AND schedule a downstream processing job
db.transaction((tx) => {
tx.insert(schema.nodes).values({ graphId, key: 'task-1', attributes: taskData }).run();
// Outbox: enqueue a job that will be processed after commit
const q = db.$client.queue('task-processing');
q.enqueueTx(tx.$honkerTx, { taskKey: 'task-1', priority: 5 });
});
```
**Use cases**: ACL evaluation after permission change, encrypted data key
rotation, schema migration jobs, retention cleanup.
### Work Queues
At-least-once background job processing with retries, priority, delayed
execution, claim expiration, and dead-letter handling.
```ts
// Enqueue a background job
const q = db.$client.queue('key-rotation');
q.enqueue({ keyVersion: 2, batchSize: 100 }, { priority: 3 });
// Worker: claim and process
const waker = q.claimWaker('rotator-1');
for await (const job of waker) {
await rotateKeys(job.payload);
job.ack();
}
```
**Use cases**: Key rotation, rate limit sweep, retention cleanup, schema
migration across tenant DBs, notification pruning.
### Advisory Locks
Named locks with TTL for leader election and exclusive access.
```ts
// Only one hub instance runs the scheduler
const lock = db.$client.tryLock('scheduler-leader', 'hub-instance-1', 30);
if (lock) {
startScheduler(db.$client.scheduler());
// Renew periodically
setInterval(() => lock.heartbeat(), 10000);
}
```
### Cron Scheduling
Time-triggered operations using 5-field cron or `@every` intervals.
```ts
const scheduler = db.$client.scheduler();
scheduler.add('retention-cleanup', '0 3 * * *', {
queue: 'maintenance',
payload: { task: 'cleanup-expired-graphs', olderThanDays: 90 },
});
```
## System DB + Tenant DB with Honker
Each database — system and tenant — is a separate Honker-managed SQLite file
with its own pubsub channels, streams, and queues.
### System DB Event Channels
| Channel | Direction | Payload |
|---------|-----------|---------|
| `account:created` | Ephemeral | `{ accountId, email, accessLevel }` |
| `account:updated` | Ephemeral | `{ accountId, changes }` |
| `org:created` | Ephemeral | `{ orgId, slug, ownerId }` |
| `org:member_added` | Ephemeral | `{ orgId, accountId, membershipLevel }` |
| `auth:key_verified` | Ephemeral | `{ keyId, ownerId }` |
### System DB Streams
| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `audit-events` | Compliance, monitoring | Append-only audit trail |
### System DB Queues
| Queue | Worker | Purpose |
|-------|--------|---------|
| `key-management` | Hub key service | API key rotation, expired key cleanup |
| `account-maintenance` | Hub account service | Deactivation processing, org transfer |
### Tenant DB Event Channels
| Channel | Direction | Payload |
|---------|-----------|---------|
| `graph:created` | Ephemeral | `{ graphId, graphTypeId, ownerId }` |
| `graph:updated` | Ephemeral | `{ graphId, changes }` |
| `nodes:created` | Ephemeral | `{ graphId, keys[] }` |
| `acl:delegation_changed` | Ephemeral | `{ principalId, agentId }` |
### Tenant DB Streams
| Stream | Consumers | Purpose |
|--------|-----------|---------|
| `call-protocol` | Flowgraph projector, observability | Call protocol event replay |
| `session-events` | Session manager, audit | Session lifecycle events |
### Tenant DB Queues
| Queue | Worker | Purpose |
|-------|--------|---------|
| `acl-evaluation` | ACL evaluator | Scope recalculation after delegation change |
| `secret-rotation` | Key service | Re-encryption with new key version |
| `graph-maintenance` | Maintenance service | Graph archival, retention cleanup |
| `schema-migration` | Migration service | Data migration for schema version bumps |
## Cross-DB Coordination
The system DB and tenant DBs are separate files. The hub mediates between them
at the application layer.
### Authentication Flow
```
1. Request → API key hash → system.db: SELECT * FROM api_keys WHERE keyHash = ?
2. system.db: Verify key → resolve ownerId → accounts row → org memberships
3. Open tenant-{orgId}.db → check ACL graph for operation access
4. Execute operation on tenant DB
```
### Cross-Tenant Operations
If a user in org A delegates to a user in org B, both tenant DBs are involved:
```
1. hub opens tenant-a.db and tenant-b.db
2. tenant-a.db: Read PrincipalNode for delegator
3. tenant-b.db: Create DelegatesEdge in ACL graph
4. hub ensures both writes succeed (application-level two-phase commit or
best-effort with reconciliation)
```
Cross-tenant operations are expected to be rare. For v1, best-effort with
reconciliation is acceptable. A formal two-phase commit across SQLite files
would require a coordinator — the hub fills this role.
## Migration from PostgreSQL + Redis
### What Changes
| Component | Before | After |
|----------|--------|-------|
| Hub database | PostgreSQL (`drizzle-orm/node-postgres`) | SQLite via Honker (`drizzle-orm/sqlite-core` + adapter) |
| Hub pub/sub | `@alkdev/pubsub` Redis transport | Honker `notify()`/`stream()` within SQLite |
| Hub task queue | Custom or none | Honker `queue()` |
| Hub leader election | Redis `SET NX` or none | Honker `tryLock()` |
| Hub scheduling | Cron daemon or none | Honker `scheduler()` |
| Hub connection | `Pool` → `drizzle(pool, { schema })` | Honker `open()` → `drizzle(honkerDb, { schema })` |
| Spoke database | SQLite via `@libsql/client` | SQLite via Honker (same engine, richer features) |
| Schema | `pgTable` (hub) + `sqliteTable` (spoke) | `sqliteTable` only |
| Testing | PostgreSQL + Redis containers | In-memory SQLite (`:memory:`) via Honker |
### What Doesn't Change
- The metagraph Module system (CallGraph, SecretGraph, AclGraph Modules)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Crypto utility (encrypt, decrypt, `EncryptedDataSchema`)
- TypeBox schemas from drizzlebox
- The Drizzle query builder API (same `.select()`, `.insert()`, `.update()`, `.delete()` calls)
- The `@alkdev/operations` call protocol (events are now published via Honker streams instead of Redis)
### Migration Path
1. **Add identity tables** to `src/sqlite/tables/` (from hub)
2. **Add scoping columns** to `graphs` table
3. **Add scope column** to `graph_types` table
4. **Remove `actors` table** and `pg/` directory
5. **Implement adapter** in `src/sqlite/adapter.ts`
6. **Split client factory** into `createSystemDatabase()` / `createTenantDatabase()`
7. **Update hub** to consume new storage API instead of its own table definitions
8. **Migrate hub data** from PostgreSQL to SQLite (export → import script)
## Design Decisions
| ADR | Decision | Status |
|-----|----------|--------|
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | Accepted |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | Accepted |
## Open Questions
- **OQ-26**: Can Honker fully replace `@alkdev/pubsub`'s Redis transport for single-node deployments?
- **OQ-27**: How are schema migrations applied across all tenant DBs?
- **OQ-28**: How does cross-tenant delegation work with separate DBs?
- **OQ-29**: Should the adapter be published as a standalone `drizzle-honker` npm package for community use?
## References
- Honker README: `/workspace/honker/README.md`
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- Drizzle SQLite session: `/workspace/drizzle-orm/src/sqlite-core/session.ts`
- ADR-039: Honker as SQLite extension
- ADR-044: Drizzle-Honker session adapter
- ADR-040: System DB + tenant DB

6
docs/architecture/metagraph-module.md
View File

@@ -1,6 +1,6 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# Metagraph as TypeBox Module
@@ -50,7 +50,7 @@ Nodes and edges use a **composite identity model**: identified by
`id` values for cross-graph references, but within a graph, the consumer's
`key` is the identity.
Node and edge attributes are stored as JSON text in SQLite (jsonb in PG). The
Node and edge attributes are stored as JSON text in SQLite. The
graph type's schema defines the expected shape, but the database doesn't enforce
it — validation happens in the repository layer. See
[schema-evolution.md](./schema-evolution.md) for how schemas change over time,

241
docs/architecture/open-questions.md
View File

@@ -1,80 +1,49 @@
---
status: draft
last_updated: 2026-05-30
last_updated: 2026-05-31
---
# Open Questions Tracker
Cross-cutting compilation of all unresolved questions across the storage architecture documents, organized by theme. Questions that appear in multiple documents are unified here with cross-references.
When a question is resolved, update its status to `resolved` and add a resolution note. Once all questions in a theme are resolved, the theme section can be removed and the resolution noted in the relevant ADR.
Cross-cutting compilation of all unresolved questions across the storage
architecture documents, organized by theme.
## Summary
| Status | Count |
|--------|-------|
| Open | 13 |
| Partially resolved | 1 |
| Resolved | 11 |
| Open | 9 |
| Resolved (this revision) | 15 |
| Previously resolved | 11 |
**Open questions requiring decisions:**
- **OQ-04** (repository layer host-specific vs host-agnostic) — start host-specific
- **OQ-07** (encryptRaw performance) — low priority, add if needed
- **OQ-10** (Edit[] classification) — needs POC
- **OQ-11** (auto-migrate vs explicit consumer action) — conditional on OQ-10
- **OQ-11** (auto-migrate vs explicit) — conditional on OQ-10
- **OQ-12** (schema evolution vs event-sourced replay) — post-v1 concern
- **OQ-13** (schema evolution events in event stream) — post-v1
- **OQ-19** (storage-operations bridge package location) — depends on long-term CRUD strategy
- **OQ-20** (delegation expiration) — ACL design
- **OQ-21** (ACL evaluator location) — ACL design
- **OQ-22** (ACL graph instance lifecycle) — ACL design
- **OQ-23** (BelongsToEdge derivation) — ACL design
- **OQ-24** (identityId reference mechanism) — ACL design
- **OQ-25** (scope string semantics for subset validation) — ACL design
**Partially resolved:**
- **OQ-01** (flowgraph Module export) — storage can start without it
**Resolved (v1 direction decided, long-term question remains open):**
- **OQ-17** (attribute query strategy) — JSON path for v1 (ADR-033), hybrid viable with dbtype later
- **OQ-18** (auto-generated vs hand-written CRUD) — hand-write for v1 (ADR-033), auto-gen remains an option
## How to Use This Document
- Each question has an **ID** (e.g., OQ-01), **status**, **origin** (which doc(s)), and **priority**
- **Cross-references** link related questions and ADRs
- Resolved questions have a **resolution** note
## ADR Impact
| ADR | Resolves | Informs |
|-----|----------|---------|
| ADR-003 | OQ-01 (partial — storage can start without flowgraph Module) | |
| ADR-015 | OQ-05 (constraint semantics) | |
| ADR-018 | OQ-17 (v1 decision: dbtype integration deferred, JSON path for v1) | |
| ADR-020 | OQ-02 (no nodeTypeId for now, can add later) | |
| ADR-033 | OQ-17 (JSON path queries for v1), OQ-18 (hand-written CRUD for v1) | |
| ADR-034 | OQ-03 (actors become ACL nodes) | OQ-21 (evaluator location), OQ-23 (BelongsToEdge derivation), OQ-24 (identityId references) |
| ADR-035 | OQ-03 (standalone table removed) | |
| ADR-036 | | OQ-20 (delegation expiration) |
| ADR-037 | | OQ-21 (evaluator location), OQ-22 (graph instance lifecycle) |
- **OQ-26** (Honker replaces @alkdev/pubsub Redis transport) — integration design
- **OQ-27** (tenant DB schema migration strategy) — multi-tenant operations
- **OQ-28** (cross-tenant delegation with separate DBs) — cross-DB coordination
- **OQ-29** (standalone drizzle-honker npm package) — community value
## Theme 1: Package Boundaries and Dependencies
### OQ-01: Should @alkdev/flowgraph export a Type.Module, or should storage define its own entries with documented correspondence?
### OQ-01: Should @alkdev/flowgraph export a Type.Module?
- **Origin**: [metagraph-module.md](metagraph-module.md)
- **Status**: partially resolved
- **Status**: resolved
- **Priority**: high
- **Notes**: Storage can start with standalone schemas and `Type.Composite([BaseNode, CallNodeAttrs])` — no dependency on flowgraph. Adopt `Import()` when flowgraph provides a Module. This avoids a circular dependency: `@alkdev/storage` does NOT depend on `@alkdev/flowgraph`.
- **Resolution**: Storage can start with standalone schemas. Adopt `Import()` when flowgraph provides a Module. No circular dependency.
- **Cross-references**: ADR-003, ADR-010
### OQ-02: Should concrete graph type Modules live in storage or in their respective packages?
### OQ-02: Should concrete graph type Modules live in storage or their packages?
- **Origin**: [metagraph-module.md](metagraph-module.md)
- **Status**: resolved
- **Priority**: medium
- **Resolution**: Both. Storage provides reference Modules in `modules/` that consumers can use directly or replace. Flowgraph may also export a Module — the two are compatible via Module `$defs`.
- **Resolution**: Both. Storage provides reference Modules; packages may also export their own.
- **Cross-references**: ADR-003
## Theme 2: Data Model
@@ -84,55 +53,50 @@ When a question is resolved, update its status to `resolved` and add a resolutio
- **Origin**: [overview.md](overview.md)
- **Status**: resolved
- **Priority**: medium
- **Resolution**: Actors become `PrincipalNode` entries in the ACL graph instance. The standalone `actors` table is removed. `ACTOR_TYPE` is replaced by the `IdentityType` enum in the AclGraph Module. See ADR-035.
- **Cross-references**: ADR-035, ADR-034, [acl.md](acl.md)
- **Resolution**: Actors become `PrincipalNode` in ACL graph. `actors` table removed. `ACTOR_TYPE` replaced by `IdentityType` in AclGraph Module. See ADR-035.
- **Cross-references**: ADR-035, ADR-034
### OQ-04: Should the repository layer be host-specific or host-agnostic?
- **Origin**: [overview.md](overview.md)
- **Status**: open
- **Status**: resolved
- **Priority**: medium
- **Notes**: A host-agnostic repository requires an abstraction over Drizzle's query builder. A host-specific repository is simpler but means duplicating query logic for PG. Decision: start host-specific in SQLite, extract common patterns later.
- **Cross-references**: [sqlite-host.md](sqlite-host.md)
- **Resolution**: Single host (SQLite). Question is moot — no dual-host repository needed. ADR-038.
### OQ-05: Should *EdgeConstraints entries use Type.Ref or Type.String for allowed source/target types?
### OQ-05: *EdgeConstraints entries use Type.Ref or Type.String?
- **Origin**: [metagraph-module.md](metagraph-module.md)
- **Status**: resolved
- **Priority**: low
- **Resolution**: `Type.String()` — the constraint arrays contain node type names, not node type schemas.
- **Cross-references**: ADR-015
- **Resolution**: `Type.String()` — constraint arrays contain names, not schemas. ADR-015.
### OQ-06: How does the graph pointer abstraction interact with the repository layer?
### OQ-06: Graph pointer abstraction vs repository layer?
- **Origin**: [metagraph-module.md](metagraph-module.md)
- **Status**: resolved
- **Priority**: low
- **Resolution**: For v1, repository functions use direct key-based addressing. Validate on read — if data doesn't match the Module entry, throw. Typed pointers are post-v1 (ADR-017).
- **Cross-references**: ADR-017, [forward-look.md](forward-look.md)
- **Resolution**: Direct key-based addressing for v1. Typed pointers post-v1. ADR-017.
## Theme 3: Encryption and Security
### OQ-07: Should we add encryptRaw() for performance?
### OQ-07: Add encryptRaw() for performance?
- **Origin**: [encrypted-data.md](encrypted-data.md)
- **Status**: open
- **Priority**: low
- **Notes**: PBKDF2 derivation adds ~100ms per operation. For batch operations (e.g., rotating 1000 keys), this adds up. An `encryptRaw()` that skips PBKDF2 would be much faster. Decision: add in a future iteration if performance demands it.
- **Notes**: PBKDF2 adds ~100ms. Add if batch operations demand it.
### OQ-08: Should the key attribute on secret nodes be encrypted?
### OQ-08: Should key attribute on secret nodes be encrypted?
- **Origin**: [encrypted-data.md](encrypted-data.md)
- **Status**: resolved
- **Priority**: low
- **Resolution**: Plaintext key names are acceptable for now. If secret names are sensitive, add a `keyHash` attribute for blind lookups.
- **Resolution**: Plaintext for now. Add `keyHash` if names are sensitive.
### OQ-09: Should secret nodes have lastUsedAt and expiresAt as first-class columns?
### OQ-09: Should secret nodes have lastUsedAt and expiresAt as columns?
- **Origin**: [encrypted-data.md](encrypted-data.md)
- **Status**: resolved
- **Priority**: low
- **Resolution**: For spoke use (occasional lookups), JSON attributes are fine. For hub use (high-throughput key validation), a standalone `api_keys` table with proper indexes is still needed.
- **Resolution**: JSON attributes for spoke, standalone table for hub.
## Theme 4: Schema Evolution
@@ -141,126 +105,163 @@ When a question is resolved, update its status to `resolved` and add a resolutio
- **Origin**: [schema-evolution.md](schema-evolution.md)
- **Status**: open
- **Priority**: high
- **Notes**: The classification table in schema-evolution.md is theoretical. A POC should validate whether `Edit[]` output contains enough information to distinguish `String → Literal("x")` (narrowing, non-breaking) from `String → Number` (incompatible, breaking). Alternative: skip classification and just use `Value.Check(newSchema, storedData)` for verification.
- **Notes**: Theoretical classification needs POC validation.
### OQ-11: Should the repository layer auto-migrate data on schema change, or require explicit consumer action?
### OQ-11: Auto-migrate data on schema change, or explicit consumer action?
- **Origin**: [schema-evolution.md](schema-evolution.md)
- **Status**: open
- **Priority**: high
- **Notes**: Conditional on OQ-10 POC outcome. If classification is feasible, the repository layer auto-applies `Value.Cast` for non-breaking changes and requires explicit consumer action for breaking changes. If classification is not feasible, the repository layer auto-applies `Value.Cast` only when `Value.Check(newSchema, storedData)` passes for all stored data.
- **Notes**: Conditional on OQ-10 POC outcome.
### OQ-12: How does schema evolution interact with the hub's event-sourced call graph?
### OQ-12: Schema evolution vs event-sourced replay?
- **Origin**: [schema-evolution.md](schema-evolution.md)
- **Status**: open
- **Priority**: medium
- **Notes**: If the hub migrates to event-sourced replay (projector evolution), storage's call graph tables become disposable projections. But other graph types (ACL, tasks, secrets) may not have an event stream to replay from. The schema evolution design should work for both projections and direct-persisted data.
- **Notes**: Post-v1. Honker streams enable event-sourced replay more naturally than before.
### OQ-13: Should schema evolution events be part of the event stream?
### OQ-13: Schema evolution events in event stream?
- **Origin**: [schema-evolution.md](schema-evolution.md)
- **Status**: open
- **Priority**: low
- **Notes**: Post-v1. For v1, schema changes are applied directly via the repository layer with version tracking.
- **Notes**: Post-v1. Honker streams provide a natural transport for schema change events.
## Theme 5: Encrypted Data Scope
### OQ-14: Should encryption be per-attribute, per-node, or per-graph?
### OQ-14: Per-attribute, per-node, or per-graph encryption?
- **Origin**: [overview.md](overview.md)
- **Status**: resolved
- **Priority**: high
- **Resolution**: Per-attribute. The `EncryptedData` schema is a single attribute within a node type, not the entire node. This preserves queryability on non-sensitive fields (ADR-023).
- **Resolution**: Per-attribute. ADR-023.
### OQ-15: Should key management be in this package?
### OQ-15: Key management in this package?
- **Origin**: [overview.md](overview.md)
- **Status**: resolved
- **Priority**: high
- **Resolution**: No. `@alkdev/storage` provides encryption/decryption primitives but NOT key management. The consuming application provides the key ring (ADR-026).
- **Resolution**: No. Application provides key ring. ADR-026.
## Theme 6: Repository Layer
### OQ-16: Should the repository layer live in @alkdev/storage or in a consumer package?
### OQ-16: Should repository layer live in storage or consumer?
- **Origin**: [overview.md](overview.md)
- **Status**: resolved
- **Priority**: high
- **Resolution**: The repository CRUD layer (host-specific typed queries, schema validation before writes) belongs in `@alkdev/storage`. The operations bridging layer (generating `OperationSpec`s from metagraph schemas) belongs in a consumer or adapter package. These are separate concerns — CRUD is a storage concern; call protocol integration is an application concern.
- **Resolution**: CRUD in storage; operations bridging in consumer. ADR-033.
## Theme 7: Repository Layer Strategy
### OQ-17: Attribute queries — JSON path, native columns, or dbtype-generated?
### OQ-17: How should the repository layer handle attribute queries — JSON path, native columns, or dbtype-generated?
- **Origin**: [forward-look.md](forward-look.md)
- **Status**: resolved (v1)
- **Priority**: high
- **Resolution**: For v1, attribute queries use JSON path extraction (`json_extract` on SQLite, `->>`/`#>>` on PG). Hand-written CRUD for static tables. dbtype integration and hybrid approach are post-v1. See ADR-033. The long-term question of whether to adopt the hybrid approach (static tables via dbtype, dynamic attributes remain JSON) remains open for future iterations.
- **Cross-references**: ADR-033, ADR-018, [forward-look.md](forward-look.md)
- **Resolution**: JSON path for v1. ADR-033. Long-term hybrid still open but less pressing without PG dual maintenance.
### OQ-18: Should the repository layer's CRUD operations be auto-generated (drizzle-graphql pattern) or hand-written?
### OQ-18: Auto-generated vs hand-written CRUD?
- **Origin**: [forward-look.md](forward-look.md)
- **Status**: resolved (v1)
- **Priority**: medium
- **Resolution**: For v1, hand-write CRUD functions with explicit signatures. The three long-term options (hand-written, auto-generated from Drizzle, auto-generated from dbtype) remain open for future iterations. See ADR-033.
- **Cross-references**: ADR-033, OQ-17
- **Resolution**: Hand-write for v1. ADR-033.
### OQ-19: Where does the storage-operations bridge package live in the @alkdev workspace?
### OQ-19: Storage-operations bridge package location?
- **Origin**: [forward-look.md](forward-look.md)
- **Status**: open
- **Status**: resolved
- **Priority**: medium
- **Notes**: Four options: (1) hub-internal code, (2) dedicated `@alkdev/storage-operations` adapter, (3) `from-storage` adapter inside `@alkdev/operations`, (4) part of `@alkdev/dbtype`'s `from-dbtype` adapter. Option 1 is the most immediate (no new package). Option 2 is the cleanest separation. Option 3 creates an undesirable dependency direction (operations → storage). Option 4 is the long-term goal if dbtype is adopted. The choice depends on OQ-17/OQ-18 resolution: if hand-written CRUD, the bridge is trivial and can live in the hub; if auto-generated from dbtype, the bridge naturally lives with dbtype.
- **Cross-references**: OQ-16, OQ-17, ADR-033
- **Resolution**: Less pressing now that Honker replaces the Redis transport. Can live in the hub for v1. Revisit if an adapter package becomes valuable.
## Theme 8: Access Control
## Theme 7: Access Control
### OQ-20: Should `DelegatesEdge` support temporary delegation with expiration?
### OQ-20: Should DelegatesEdge support expiration?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Priority**: low
- **Notes**: Currently, `DelegatesEdge` has `narrowedScopes` and `narrowedResources` but no `expiresAt`. If delegation should be time-limited (e.g., "delegate for this session only" or "delegate for 24 hours"), an expiration attribute is needed. Session-scoped delegation could be modeled by creating/removing edges per session, avoiding the need for an `expiresAt` attribute. Time-based expiration adds complexity to the evaluator (checking edge validity at call time) but may be useful for non-session contexts.
- **Cross-references**: ADR-036
- **Notes**: Session-scoped delegation could be modeled by creating/removing edges per session rather than adding `expiresAt`.
### OQ-21: Should the ACL evaluator live in `@alkdev/storage` or in the hub?
### OQ-21: Should ACL evaluator live in storage or hub?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Status**: resolved
- **Priority**: high
- **Notes**: The ACL evaluator traverses delegation chains and computes effective scopes. Three options: (1) `@alkdev/storage` provides traversal primitives (walk edges, compute effective scopes for a principal given a graph instance) and the hub composes them with `@alkdev/operations`' `enforceAccess`. (2) The hub implements the evaluator from scratch, using storage's repository layer for graph queries. (3) A new `@alkdev/acl` package provides the evaluator, depending on both `@alkdev/storage` and `@alkdev/operations`. Option 1 keeps the dependency direction clean (storage doesn't depend on operations). Option 3 is the cleanest separation but adds a package. The choice depends on whether the evaluator is generic enough to be reusable across different hub implementations.
- **Cross-references**: ADR-034, ADR-037
- **Resolution**: Storage provides traversal primitives; hub composes with operations `enforceAccess`. The single-host model (no PG/SQLite split) simplifies this — no cross-DB joins needed for ACL evaluation within a tenant DB. ADR-034.
### OQ-22: How are ACL graph instances created and managed?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Status**: resolved
- **Priority**: medium
- **Notes**: Several options: (1) One global ACL graph instance per hub. Simple but means all orgs share a single graph — large graphs may have traversal performance implications. (2) One ACL graph instance per org. Isolated, each org's permissions are self-contained. Requires cross-org delegation to span graphs. (3) One ACL graph instance per "scoping context" (e.g., per spoke context). Most granular but most complex. The choice depends on whether delegation crosses org boundaries (if a user delegates to an agent in another org's context, graphs must be traversable across instances).
- **Cross-references**: ADR-037
- **Resolution**: One ACL graph instance per tenant DB (ADR-040). The tenant DB is inherently org-scoped, so the ACL graph covers one org. No cross-org scoping issue within a single tenant DB.
- **Cross-references**: ADR-040
### OQ-23: Should `BelongsToEdge` be derived (materialized from `organization_members`) or primary (ACL graph is the source of truth)?
### OQ-23: BelongsToEdge derived or primary?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Status**: resolved
- **Priority**: medium
- **Notes**: The hub already has an `organization_members` table with `membershipLevel`. If `BelongsToEdge` is derived, the hub writes both `organization_members` rows and ACL graph edges when membership changes, keeping them in sync. If `BelongsToEdge` is primary, the ACL graph is the source of truth and the hub reads org membership from the graph. Derived is consistent with the hub's existing identity tables being authoritative. Primary means the ACL graph replaces org membership data, requiring graph queries for simple membership lookups. Lean toward derived — the hub's identity tables are authoritative for authentication, the ACL graph is authoritative for authorization.
- **Cross-references**: ADR-034
- **Resolution**: Derived. `organization_members` SQL table is authoritative for indexed lookups; `BelongsToEdge` in ACL graph enables traversal evaluation. ADR-045.
- **Cross-references**: ADR-045
### OQ-24: How does `identityId` reference hub entities without creating a package dependency?
### OQ-24: How does identityId reference hub entities without package dependency?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Status**: resolved
- **Priority**: medium
- **Notes**: `PrincipalNode.identityId` references an account, organization, or role in the hub's database, but `@alkdev/storage` must not depend on `@alkdev/operations` or the hub. The `identityId` is a string, not a FK. This is consistent with ADR-020 (no nodeTypeId on nodes) — the metagraph pattern stores node attributes without assuming external referential integrity. Options: (1) Logical references (current design) — `identityId` is a string that the hub resolves. (2) Convention-based references — a URI scheme like `alk://account/user-1` or `alk://org/acme` that encodes the entity type and ID. (3) A shared types package that both storage and hub import. Option 1 is the simplest and consistent with the existing pattern. The burden of referential integrity falls on the consumer (the hub), not on storage.
- **Cross-references**: ADR-020, ADR-034
- **Resolution**: Logical string references, consistent with ADR-020. With identity tables now in `@alkdev/storage` (ADR-041), the `PrincipalNode.identityId` logically references `accounts.id` in the system DB. Same pattern, clearer provenance.
- **Cross-references**: ADR-020, ADR-041
### OQ-25: What are the scope string semantics for subset validation?
### OQ-25: Scope string semantics for subset validation?
- **Origin**: [acl.md](acl.md)
- **Status**: open
- **Priority**: high
- **Notes**: `narrowedScopes ⊆ effectiveScopes` is the no-escalation invariant, but the semantics of this subset check depend on how scope strings work. `@alkdev/operations` uses keypal's scope model (colon-separated hierarchical segments, `*` wildcard for suffix matching). `"dev:*"` matches `"dev.read"`, `"dev.write"`, `"dev.fs.read"`, etc. The ACL evaluator must use the same semantics or delegation validation will be inconsistent with runtime access checks. Option: import scope matching logic from `@alkdev/operations` or extract it to a shared utility. The ACL graph stores scopes as plain strings; matching is an evaluator concern, not a storage concern.
- **Cross-references**: ADR-036, `/workspace/@alkdev/operations/src/access.ts`
- **Notes**: Keypal's colon-separated hierarchical scope model with `*` wildcard. ACL evaluator must use same semantics. Scope matching is an evaluator concern, not a storage concern.
## Theme 8: Honker and SQLite
### OQ-26: Can Honker fully replace @alkdev/pubsub's Redis transport for single-node deployments?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: high
- **Notes**: Honker's `notify()`/`listen()` and `stream()`/`subscribe()` provide the pub/sub primitives. The question is whether `@alkdev/pubsub`'s `TypedEventTarget` interface can be backed by Honker instead of Redis, and whether multi-node deployments still need Redis for internode communication.
### OQ-27: How are schema migrations applied across all tenant DBs?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: high
- **Notes**: Each tenant DB has its own migration history. When a schema change is deployed, all tenant DBs need migration. Options: (1) Migration queue — enqueue a migration job per tenant DB, workers claim and execute. (2) Lazy migration — migrate on first access. (3) Startup sweep — hub iterates all tenant DBs at startup and applies pending migrations.
### OQ-28: How does cross-tenant delegation work with separate DBs?
- **Origin**: [overview.md](overview.md)
- **Status**: open
- **Priority**: medium
- **Notes**: If a user in org A delegates to a user in org B, both tenant DBs are involved. The hub mediates. For v1, cross-tenant delegation can be deferred or handled via the system DB as a coordination point.
### OQ-29: Should the Drizzle-Honker adapter be published as a standalone npm package?
- **Origin**: [honker-integration.md](honker-integration.md)
- **Status**: open
- **Priority**: low
- **Notes**: The adapter is ~100 lines and useful to anyone combining Drizzle with Honker. Publishing as `drizzle-honker` would benefit the community. Decision: start inside `@alkdev/storage`, extract later if there's demand.
## ADR Impact
| ADR | Resolves | Informs |
|-----|----------|---------|
| ADR-003 | OQ-01 (partial) | |
| ADR-015 | OQ-05 | |
| ADR-017 | OQ-06 | |
| ADR-020 | OQ-24 | |
| ADR-023 | OQ-14 | |
| ADR-026 | OQ-15 | |
| ADR-033 | OQ-04, OQ-16, OQ-17, OQ-18 | |
| ADR-034 | OQ-03, OQ-21 | OQ-25 |
| ADR-035 | OQ-03 | |
| ADR-038 | OQ-04 (moot) | OQ-17 (less pressure) |
| ADR-040 | OQ-22 | OQ-27, OQ-28 |
| ADR-041 | OQ-24 | |
| ADR-042 | | OQ-24 |
| ADR-043 | | |
| ADR-044 | OQ-19 (less pressure) | |
| ADR-045 | OQ-23 | OQ-20 |

278
docs/architecture/overview.md
View File

@@ -1,11 +1,11 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# @alkdev/storage — Overview
Typed graph storage with dual database hosts. Deno-first, published via JSR.
Typed graph storage with SQLite via Honker. Deno-first, published via JSR.
## Purpose
@@ -16,12 +16,13 @@ graphs populated with nodes and edges.
This pattern replaces domain-specific table proliferation with a small number of
general-purpose tables that can model anything — call graphs, ACL rules, task
dependencies, encrypted secrets — while enforcing schema integrity through
TypeBox validation.
dependencies, encrypted secrets, session trees, operation registries — while
enforcing schema integrity through TypeBox validation.
The package evolved from `@ade/ade-v0/packages/core/graphs` and
`@ade/ade-v0/packages/storage_sqlite`, simplified and refactored for the @alkdev
ecosystem.
The package also provides **identity infrastructure** tables (accounts,
organizations, api_keys, audit_logs) for multi-tenant authentication and
authorization, and integrates with **Honker** for transactional pub/sub, event
streams, and task queues within the same SQLite database.
## Architecture
@@ -38,12 +39,16 @@ ecosystem.
│ │ ├── bridge.ts → moduleToDbSchema, validateNode, validateEdge
│ │ ├── crypto.ts → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│ │ └── mod.ts → re-exports all graphs exports
│ ├── sqlite/ → SQLite host (drizzle-orm/libsql)
│ │ ├── tables/ → drizzle table definitions
│ │ ├── relations.ts → drizzle relational mappings
│ │ ├── schema.ts → barrel re-export
│ │ └── client.ts → injectable createSqliteDatabase()
│ └── pg/ → PostgreSQL host (NOT YET IMPLEMENTED)
│ └── sqlite/ → SQLite host (Drizzle + Honker)
│ ├── tables/ → drizzle table definitions
│ │ ├── common.ts → commonCols
│ │ ├── identity/ → accounts, organizations, org_members, api_keys, audit_logs
│ │ ├── metagraph/ → graph_types, node_types, edge_types, graphs, nodes, edges
│ │ └── index.ts → barrel re-export
│ ├── relations.ts → drizzle relational mappings
│ ├── schema.ts → barrel re-export
│ ├── adapter.ts → Drizzle-Honker session adapter
│ └── client.ts → createSystemDatabase(), createTenantDatabase()
└── test/
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
```
@@ -54,26 +59,66 @@ ecosystem.
| ------------------------ | --------------------------------------- | --------------------------------------- |
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client | `@alkdev/drizzlebox`, `drizzle-orm`, `@libsql/client` |
| `@alkdev/storage/pg` | PostgreSQL tables, relations, client | ⚠️ NOT YET IMPLEMENTED |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client, adapter | `@alkdev/drizzlebox`, `drizzle-orm`, `@russellthehipp/honker-node` |
The `./graphs` subpath exists because the source code lives in `src/graphs/` and
the main `mod.ts` re-exports it. Importing from either `@alkdev/storage` or
`@alkdev/storage/graphs` yields the same types and Metagraph Module.
The `pg/` subpath has been removed (ADR-038). SQLite via Honker is the sole
database host.
## Database Model
### System DB + Tenant DB
The package uses a two-database model for multi-tenant isolation (ADR-040):
| Database | Contents | Purpose |
|----------|----------|---------|
| `system.db` | accounts, organizations, organization_members, api_keys, audit_logs, system graph_types | Identity infrastructure, authentication, authorization anchors |
| `tenant-{orgId}.db` | graphs, nodes, edges, graph_types, node_types, edge_types | All graph data for one org — call graphs, ACL instances, session trees, secrets, tasks |
The system DB is opened by `createSystemDatabase(client)`. Each tenant DB is
opened by `createTenantDatabase(client)`. Both return typed Drizzle instances
with their respective schemas attached.
**Why separate files**: File-level isolation means one tenant's data cannot leak
to another, even via application bugs. Each tenant DB is independently
backupable, migratable, and compactable. No `orgId` column is needed on tenant
tables because the entire file IS the org scope.
**Cross-DB references**: The tenant DB's `graphs.ownerId` and `graphs.projectId`
logically reference (not FK) the system DB's identity tables. The consumer
enforces referential integrity at the application layer, consistent with
ADR-020.
### Graph Type Scope
Graph types have a `scope` column (ADR-043) controlling who can create and
modify them:
| Scope | Examples | Who can modify |
|-------|----------|----------------|
| `system` | acl, call-graph, secret, operation-registry, message-session | Setup/seeding only |
| `tenant` | Custom org graph types (sprint-board, etc.) | Org admins |
| `user` | Personal graph types (my-notes, etc.) | Creating user |
System-scoped graph types are seeded during initialization. Their schemas are
fixed — changes require a version bump and migration (ADR-029).
## Terminology
| Term | Definition |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Metagraph** | A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions. |
| **Hub** | The central service in the hub-spoke architecture. A consumer of `@alkdev/storage` — uses the PostgreSQL host for persistent graph storage. The hub also depends on `@alkdev/operations`, `@alkdev/pubsub`, `@alkdev/flowgraph`. |
| **Spoke** | A local/embedded instance that runs per-project or per-session. A consumer of `@alkdev/storage` — uses the SQLite host for local graph storage. |
| **Hub** | The central service in the hub-spoke architecture. A consumer of `@alkdev/storage` — opens both system and tenant databases. The hub also depends on `@alkdev/operations`, `@alkdev/pubsub`, `@alkdev/flowgraph`. |
| **Spoke** | A local/embedded instance that runs per-project or per-session. A consumer of `@alkdev/storage` — opens a tenant database (or its own standalone DB for single-user mode). |
| **System DB** | The SQLite database holding identity/auth tables and system graph type definitions. One per deployment. |
| **Tenant DB** | The SQLite database holding all graph data for one organization. One per org. |
| **Graph type** | A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the `graph_types` table. |
| **Node type** | A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the `node_types` table. |
| **Edge type** | A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the `edge_types` table. |
| **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. |
| **Honker** | SQLite extension providing transactional pub/sub, durable event streams, task queues, advisory locks, and cron scheduling within the same `.db` file. |
| **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. |
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. No dependency on `@alkdev/operations` — the consumer wires CRUD into the registry. |
| **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. |
| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. |
## Design Decisions
@@ -83,14 +128,22 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| ADR | Decision | Summary |
|-----|----------|---------|
| [001](decisions/001-deno-first-jsr-publishes.md) | Deno-first, JSR publishes | Published to JSR; npm comes free via `@jsr/alkdev__storage` |
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all domains |
| [002](decisions/002-metagraph-over-domain-tables.md) | Metagraph over domain-specific tables | 6 general-purpose tables serve all graph-shaped domains |
| [003](decisions/003-typebox-module-as-api-surface.md) | TypeBox Module as API surface | `Type.Module()` replaces `SchemaBuilder`; `Metagraph.Import()` + `Type.Composite()` |
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSqliteDatabase(client)` takes a pre-created client |
| [004](decisions/004-injectable-clients-no-side-effects.md) | Injectable clients, no side effects | `createSystemDatabase(client)` / `createTenantDatabase(client)` take pre-created clients |
| [005](decisions/005-drizzle-plus-typebox-via-drizzlebox.md) | Drizzle + TypeBox via drizzlebox | Drizzle tables are single source of truth; drizzlebox generates TypeBox schemas |
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | Avoids JSR slow-types; consistent pattern across codebase |
| [007](decisions/007-no-comments-in-code.md) | No comments in code | Documentation lives in architecture docs and TypeBox descriptions |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
| [033](decisions/033-json-path-queries-for-v1.md) | JSON path queries and hand-written CRUD for v1 | Attribute queries use JSON path; CRUD is hand-written; dbtype and auto-generation are post-v1 |
| [033](decisions/033-json-path-queries-for-v1.md) | JSON path queries and hand-written CRUD for v1 | Attribute queries use JSON path; CRUD is hand-written |
| [038](decisions/038-sqlite-first-pg-removed.md) | SQLite-first, Postgres removed | Single database host; no dual maintenance |
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension and transport | DB + pub/sub + queues + events in one SQLite file |
| [040](decisions/040-system-db-tenant-db.md) | System DB + tenant DB separation | Identity in system.db, graph data in tenant-{orgId}.db |
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage package | accounts, organizations, api_keys, audit_logs defined in storage |
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graph instances | `ownerId`, `projectId` on `graphs` table |
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope controls who can modify graph types |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker session adapter | ~100-line adapter; no Drizzle fork; `$client` and `$honkerTx` for honker access |
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | organization_members authoritative, BelongsToEdge derived | SQL table for fast lookups; ACL edge for traversal evaluation |
## Dependencies
@@ -98,18 +151,15 @@ All design decisions are documented as ADRs in [decisions/](decisions/).
| -------------------- | ------------------------------------ | ------------------------ |
| `@alkdev/typebox` | Runtime schema validation | graphs/ |
| `@alkdev/drizzlebox` | Generate TypeBox from Drizzle tables | sqlite/ |
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ (and future pg/) |
| `@libsql/client` | SQLite client (libsql/turso) | sqlite/ |
| `postgres` | PostgreSQL client | pg/ (not yet used) |
| `drizzle-orm` | ORM, table definitions, queries | sqlite/ |
| `@russellthehippo/honker-node` | SQLite + pub/sub + queues + events | sqlite/ |
`@alkdev/typebox` and `@alkdev/drizzlebox` are npm packages (not yet on JSR).
JSR handles npm dependencies natively.
**Ecosystem packages are not runtime dependencies of `@alkdev/storage`.** All
ecosystem references in this document describe consumer-side data shapes and
integration patterns, not import dependencies. The `@alkdev/operations`,
`@alkdev/pubsub`, `@alkdev/flowgraph`, and `@alkdev/taskgraph` packages are
consumed by the hub and spokes, not by storage itself.
ecosystem references describe consumer-side data shapes and integration
patterns, not import dependencies.
## What Exists vs. What's Needed
@@ -119,8 +169,7 @@ consumed by the hub and spokes, not by storage itself.
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Reference graph type Modules (CallGraph, SecretGraph)
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
- SQLite host: 6 metagraph tables + actors table + Drizzle relations + client
factory
- SQLite host: 6 metagraph tables + `actors` placeholder + Drizzle relations + client factory
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
- Reference module tests (bridge functions, validation, Module composition)
@@ -128,18 +177,22 @@ consumed by the hub and spokes, not by storage itself.
| Gap | Priority | Notes |
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| Repository/CRUD layer | High | ⚠️ Not yet implemented. Typed insert, find, update, delete functions for graphs, nodes, edges. No dependency on `@alkdev/operations` — consumer wires CRUD into registry. |
| PostgreSQL host | Medium | Same table shapes, `pgTable` + `jsonb` + `timestamp` + `pgEnum`. Stub only. |
| ACL graph type | Medium | Access control as a metagraph. Principal/agent delegation, org scoping, resource access. See [acl.md](acl.md). |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s `TaskGraphNodeAttributes` and `DependencyEdge` schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4 of the metagraph implementation path. |
| Drizzle-Honker session adapter | High | ~100-line adapter wrapping `HonkerSQLiteSession` + `HonkerPreparedQuery`. ADR-044. |
| Identity tables (accounts, organizations, api_keys, audit_logs, organization_members) | High | Moving from hub into storage. ADR-041. |
| Scoping columns on `graphs` table (`ownerId`, `projectId`) | High | ADR-042. |
| Graph type `scope` column | High | ADR-043. |
| Remove `actors` table and `pg/` directory | High | ADR-035 (actors), ADR-038 (pg). |
| `createSystemDatabase()` / `createTenantDatabase()` factories | High | Split from current `createSqliteDatabase()`. |
| Repository/CRUD layer | Medium | ⚠️ Typed insert, find, update, delete functions for graphs, nodes, edges. |
| ACL graph type | Medium | Access control as a metagraph. ADR-034. |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4. |
## Ecosystem Integration
`@alkdev/storage` is a **data layer package** consumed by other packages in the
@alkdev ecosystem. It does not depend on the hub — the dependency flows the
other way. The hub consumes storage (along with operations, pubsub, flowgraph,
and taskgraph) as part of its architecture.
other way.
### Dependency Direction
@@ -153,13 +206,8 @@ and taskgraph) as part of its architecture.
@alkdev/taskgraph ← task dependency graph schema, cost-benefit analysis
(depends on: @alkdev/typebox)
@alkdev/dbtype ← schema-first multi-dialect DB type system (Phase 0, not yet implemented)
(depends on: @alkdev/typebox, @alkdev/ujsx)
Renders UJSX element trees to Drizzle dialects; future: from-dbtype
adapter generates CRUD OperationSpecs for @alkdev/operations
@alkdev/storage ← YOU ARE HERE — typed graph persistence
(depends on: @alkdev/typebox, @alkdev/drizzlebox)
@alkdev/storage ← YOU ARE HERE — typed graph persistence + identity
(depends on: @alkdev/typebox, @alkdev/drizzlebox, drizzle-orm, honker-node)
↑ ↑
| |
@@ -167,119 +215,75 @@ Hub / Spoke Any consumer that needs
(consumes all) persistent graph storage
```
The key insight: `@alkdev/storage` provides the **persistence primitives**
(schemas, tables, repository layer). The **domain semantics** (what a call graph
means, what identity looks like, how access control works) are defined by the
packages above. Storage stores the shapes those packages define; it does not
define the semantics itself.
### Event-Driven Architecture with Honker
The @alkdev platform is event-driven. Honker provides the transport mechanism
within each SQLite database:
| Concern | Honker Primitive | Example |
|---------|------------------|---------|
| Fire-and-forget notifications | `notify(channel, payload)` | "graph:updated", "node:created" |
| Durable per-consumer delivery | `stream(name).subscribe(consumer)` | Call protocol events, audit trail |
| At-least-once background jobs | `queue(name).enqueue(payload)` | Key rotation, schema migration, retention cleanup |
| Leader election | `tryLock(name, owner, ttl)` | Only one hub instance runs the scheduler |
| Scheduled operations | `scheduler().add(cron, handler)` | Retention cleanup, key sweep, rate limit sweep |
A single Drizzle transaction via the Honker adapter can insert graph data AND
publish a notification AND enqueue a side-effect job — all committing atomically.
No dual-write problem between data and events.
### What Comes from Where
| Concept | Source package | Storage's role |
|---------|---------------|----------------|
| Call protocol events (`call.requested`, `call.responded`, etc.) | `@alkdev/operations` | Storage persists the outcomes — graphs with `CallNodeAttrs` nodes |
| Identity (`id`, `scopes`, `resources`) | `@alkdev/operations` | Storage stores identity as node attributes; `Identity` is a data shape, not a storage concept |
| Access control (`AccessControl`, `requiredScopes`) | `@alkdev/operations` | Storage's ACL graph type mirrors the operations `AccessControl` schema as graph structure |
| Call graph schema (`CallNodeAttrs`, `CallEdgeAttrs`, `CallStatus`) | `@alkdev/flowgraph` | Storage persists these in-memory shapes to the database |
| Task graph schema (`TaskGraphNodeAttributes`, `DependencyEdge`) | `@alkdev/taskgraph` | Storage persists task dependency shapes |
| Event transport (`TypedEventTarget`, `EventEnvelope`) | `@alkdev/pubsub` | Storage is not involved in event routing; it stores the events' outcomes |
| Database schema rendering (`<table>`, `<column>`, HostConfig) | `@alkdev/dbtype` | Storage's static metagraph tables could be dbtype-rendered in the future (OQ-17, OQ-18) |
| Universal IR (`h()`, `createComponent`, `createRoot`) | `@alkdev/ujsx` | Storage's `Type.Module` format is structurally compatible with ujsx rendering; no runtime dependency |
| Call protocol events | `@alkdev/operations` | Storage persists the outcomes as graph nodes + publishes via Honker stream |
| Identity | `@alkdev/operations` | Storage stores identity in SQL accounts table + as PrincipalNode in ACL graphs |
| Access control | `@alkdev/operations` | Storage's AclGraph mirrors `AccessControl` schema as graph structure |
| Call graph schema | `@alkdev/flowgraph` | Storage persists in-memory shapes to the database |
| Task graph schema | `@alkdev/taskgraph` | Storage persists task dependency shapes |
| Event transport | Honker (within storage) | Replaces `@alkdev/pubsub`'s Redis transport for single-node deployments |
### Repository Layer Bridging Pattern (Consumer-Side Concern)
### Repository Layer Bridging Pattern
The repository layer in `@alkdev/storage` provides typed CRUD — no `@alkdev/operations`
dependency. A **consumer-side** bridging module can then wire these CRUD functions
into the `@alkdev/operations` registry, analogous to how `drizzle-graphql`
auto-generates a GraphQL schema from Drizzle tables — but using operations
(queries, mutations, subscriptions) instead of GraphQL resolvers. This works
because:
1. `@alkdev/operations` already maps closely to GraphQL's
queries/mutations/subscriptions (it was modeled after that pattern)
2. `@alkdev/pubsub` provides the subscription transport (forked from
graphql-yoga's pubsub with additions like in-memory, Redis, WebSocket,
WebWorker event targets)
3. `@alkdev/storage`'s metagraph tables are the data source, analogous to
Drizzle tables for drizzle-graphql
The bridging module would live in a consumer package (e.g., the hub or a
dedicated `@alkdev/storage-operations` adapter), not in `@alkdev/storage` itself,
to avoid circular dependencies:
The repository layer in `@alkdev/storage` provides typed CRUD — no
`@alkdev/operations` dependency. A consumer-side bridging module wires CRUD
functions into the operations registry when needed.
```
@alkdev/storage → defines types + tables (no operations dependency)
@alkdev/storage → defines types + tables + CRUD (no operations dependency)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas
```
#### Ecosystem Context
The question of *where* this bridge lives and *how* it's generated connects to
the broader ecosystem:
- **drizzle-graphql** (`/workspace/drizzle-graphql`): Auto-generates GraphQL
CRUD from Drizzle tables. The reference pattern for "database schema → API
surface." Produces `{ schema, entities }` from `buildSchema(db)`. No TypeBox,
no metagraph.
- **@alkdev/dbtype**: Schema-first multi-dialect system using ujsx element trees.
Defines `<table>`, `<column>` elements rendered to Drizzle via HostConfig. Has
a designed `from-dbtype` adapter that generates `OperationSpec[]` from element
trees + Type.Module bundles. Phase 0 (architecture only, no implementation).
- **@alkdev/operations**: Runtime-agnostic typed operations registry with
adapters (`FromOpenAPI`, `from_mcp`, `from_typemap`) that generate
`OperationSpec[]` from external specifications. The `from-dbtype` adapter would
be another adapter in the same pattern.
The strategic question (OQ-17, OQ-18) is whether storage's repository CRUD
operations should be hand-written, auto-generated from Drizzle schemas, or
auto-generated from dbtype element trees once dbtype is implemented. For v1,
hand-written CRUD is the simplest path and doesn't block any long-term option.
See [forward-look.md](forward-look.md) for the full analysis.
### Avoiding Circular Dependencies
Neither `@alkdev/storage` nor `@alkdev/operations` should depend on each
other directly. Storage defines the schema types and database tables; operations
defines the call protocol and execution model. The consumer (hub, spoke, or
adapter package) imports both and bridges them. This preserves the
single-responsibility principle and allows each package to evolve independently.
If shared type definitions are needed (e.g., `Identity` referenced in both
storage node attributes and operations call events), they should either:
1. Be duplicated in each package with a documented correspondence (acceptable
for small, stable types)
2. Be extracted to a minimal shared types package if the duplication becomes
burdensome
defines the call protocol and execution model. The consumer imports both and
bridges them.
## Open Questions
Open questions are tracked in [open-questions.md](open-questions.md). Key
questions affecting this package:
- **OQ-03**: Should actors be a node type or a standalone table? (resolved: actors become PrincipalNode in ACL graph — ADR-035)
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (open, start host-specific)
- **OQ-14**: Should encryption be per-attribute, per-node, or per-graph? (resolved: per-attribute)
- **OQ-15**: Should key management be in this package? (resolved: no, application provides key ring)
- **OQ-16**: Should the repository layer live in storage or a consumer package? (resolved: CRUD in storage, operations bridging in consumer)
- **OQ-17**: How should the repository layer handle attribute queries — JSON path, native columns, or dbtype-generated? (open, JSON path for v1)
- **OQ-18**: Should CRUD operations be auto-generated or hand-written? (open, hand-write for v1)
- **OQ-19**: Where does the storage-operations bridge package live? (open, depends on OQ-17/OQ-18)
- **OQ-04**: Should the repository layer be host-specific or host-agnostic? (resolved: single host, question moot)
- **OQ-22**: How are ACL graph instances created and managed? (open, tenant DB model simplifies: likely one per tenant DB)
- **OQ-23**: BelongsToEdge derived or primary? (resolved: derived — ADR-045)
- **OQ-26**: Can Honker replace `@alkdev/pubsub`'s Redis transport for single-node deployments? (open)
- **OQ-27**: How are schema migrations applied across all tenant DBs? (open)
- **OQ-28**: How does cross-tenant delegation work with separate DBs? (open)
## References
- Metagraph Module evolution: [metagraph-module.md](./metagraph-module.md)
- Schema evolution via TypeBox value system: [schema-evolution.md](./schema-evolution.md)
- Metagraph Module: [metagraph-module.md](./metagraph-module.md)
- Honker integration: [honker-integration.md](./honker-integration.md)
- SQLite host tables: [sqlite-host.md](./sqlite-host.md)
- Schema evolution: [schema-evolution.md](./schema-evolution.md)
- Encrypted data: [encrypted-data.md](./encrypted-data.md)
- ACL graph: [acl.md](./acl.md)
- Forward-looking connections: [forward-look.md](./forward-look.md)
- Operations architecture: `/workspace/@alkdev/operations/docs/architecture/README.md`
- Pubsub architecture: `/workspace/@alkdev/pubsub/docs/architecture/README.md`
- Flowgraph architecture: `/workspace/@alkdev/flowgraph/docs/architecture/README.md`
- Taskgraph architecture: `/workspace/@alkdev/taskgraph_ts/docs/architecture/README.md`
- drizzle-graphql (reference for repo bridging pattern): `/workspace/drizzle-graphql/`
- Source heritage: `@ade/ade-v0/packages/core/graphs` and
`@ade/ade-v0/packages/storage_sqlite`
- Drizzle ORM: https://orm.drizzle.team/
- TypeBox: `/workspace/@alkdev/typebox/`
- JSR: https://jsr.io/
- Open questions: [open-questions.md](./open-questions.md)
- Honker source: `/workspace/honker/`
- Operations architecture: `/workspace/@alkdev/operations/docs/architecture/`
- Flowgraph architecture: `/workspace/@alkdev/flowgraph/docs/architecture/`

10
docs/architecture/schema-evolution.md
View File

@@ -1,6 +1,6 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# Schema Evolution
@@ -28,10 +28,10 @@ data to fit new schema shapes.
Two distinct domains of JSON values are involved:
- **Schemas-as-JSON**: The TypeBox schema objects stored in `node_types.schema`
and `edge_types.schema` columns. `Value.Diff`/`Value.Patch` operate on these
and `edge_types.schema` columns (text with JSON mode in SQLite). `Value.Diff`/`Value.Patch` operate on these
(detecting schema changes, updating stored schemas).
- **Data-as-JSON**: The node/edge attribute values stored in `nodes.attributes`
and `edges.attributes` columns. `Value.Cast`/`Value.Check` operate on these
and `edges.attributes` columns (text with JSON mode in SQLite). `Value.Cast`/`Value.Check` operate on these
(migrating data to fit new schemas, verifying compatibility).
This is not a migration framework. It's the observation that the existing
@@ -104,7 +104,7 @@ stream and storage. That's a post-v1 concern.
TypeBox schemas are JSON Schema objects — plain JSON values. The current
`node_types.schema` and `edge_types.schema` columns store them as JSON text
(SQLite) or jsonb (PG). This means `Value.Diff` can diff schemas themselves.
in SQLite. This means `Value.Diff` can diff schemas themselves.
### Detecting Schema Changes

389
docs/architecture/sqlite-host.md
View File

@@ -1,51 +1,59 @@
---
status: reviewed
last_updated: 2026-05-30
status: draft
last_updated: 2026-05-31
---
# SQLite Host
The SQLite database host for `@alkdev/storage`. Uses Drizzle ORM with
libsql/Turso for the SQLite dialect and `@alkdev/drizzlebox` for TypeBox schema
generation from Drizzle table definitions.
The SQLite database host for `@alkdev/storage`. Uses Drizzle ORM with Honker
for database operations, pub/sub, event streams, and task queues. TypeBox
schemas are auto-generated from Drizzle table definitions via `@alkdev/drizzlebox`.
## Overview
The SQLite host provides:
1. **Drizzle table definitions** for the metagraph pattern (graph types, node
types, edge types, graphs, nodes, edges) plus a standalone `actors` table
2. **Drizzle relations** for the relational query API
3. **TypeBox schemas** auto-generated from Drizzle tables (select/insert
validation)
4. **Injectable database factory** — `createSqliteDatabase(client)` accepts a
pre-created client
The SQLite host is the first-class target. PostgreSQL will follow the same table
shapes with appropriate dialect changes.
1. **Metagraph tables** — 6 tables for graph types, node types, edge types,
graphs, nodes, and edges (ADR-002)
2. **Identity tables** — accounts, organizations, organization_members, api_keys,
audit_logs for multi-tenant authentication and authorization (ADR-041)
3. **Drizzle relations** for the relational query API
4. **TypeBox schemas** auto-generated from Drizzle tables (select/insert
validation) via `@alkdev/drizzlebox`
5. **Drizzle-Honker adapter** — thin session adapter for Honker integration
(ADR-044)
6. **Client factories** — `createSystemDatabase(client)` and
`createTenantDatabase(client)` for the system/tenant DB model (ADR-040)
## Package Structure
```
src/sqlite/
├── tables/
│ ├── common.ts # commonCols, ACTOR_TYPE enum
│ ├── graphTypes.ts # graph_types table + select/insert schemas
│ ├── nodeTypes.ts # node_types table + select/insert schemas
│ ├── edgeTypes.ts # edge_types table + select/insert schemas
│ ├── graphs.ts # graphs table + select/insert schemas
│ ├── nodes.ts # nodes table + select/insert schemas
│ ├── edges.ts # edges table + select/insert schemas
│ ├── actors.ts # actors table + select/insert schemas
│ ├── common.ts # commonCols
│ ├── identity/
│ │ ├── accounts.ts # accounts table + select/insert schemas
│ │ ├── organizations.ts # organizations table + select/insert schemas
│ │ ├── organization_members.ts # org membership + select/insert schemas
│ │ ├── api_keys.ts # API keys (keypal) + select/insert schemas
│ │ ├── audit_logs.ts # audit trail + select/insert schemas
│ │ └── index.ts # barrel re-export
│ ├── metagraph/
│ │ ├── graphTypes.ts # graph_types table + select/insert schemas
│ │ ├── nodeTypes.ts # node_types table + select/insert schemas
│ │ ├── edgeTypes.ts # edge_types table + select/insert schemas
│ │ ├── graphs.ts # graphs table + select/insert schemas
│ │ ├── nodes.ts # nodes table + select/insert schemas
│ │ ├── edges.ts # edges table + select/insert schemas
│ │ └── index.ts # barrel re-export
│ └── index.ts # barrel re-export
├── relations.ts # Drizzle relational mappings
├── schema.ts # re-exports tables + relations
└── client.ts # createSqliteDatabase()
├── adapter.ts # Drizzle-Honker session adapter
├── schema.ts # re-exports all tables + relations
└── client.ts # createSystemDatabase(), createTenantDatabase()
```
## Tables
### Common Columns
## Common Columns
All tables share these columns:
@@ -62,23 +70,15 @@ All tables share these columns:
}
```
**Notable differences from a typical PostgreSQL common columns pattern**:
- `id` is a consumer-generated UUID text PK (no `$defaultFn`)
- `metadata` is an extension namespace following `_subsystem.key` convention
- `createdAt`/`updatedAt` are Unix epoch integers with timestamp mode
- No `$onUpdate` — consumers must set `updatedAt` explicitly
| Column | SQLite | PostgreSQL (typical) |
| ----------- | ------------------------------------- | ------------------------------------------------------------- |
| `id` | text PK (consumer-generated) | text PK with `$defaultFn(() => crypto.randomUUID())` |
| `metadata` | `text` with JSON mode | `jsonb` with `$type<Record<string, unknown>>()` |
| `createdAt` | `integer` timestamp mode (Unix epoch) | `timestamp with timezone` defaulting `now()` |
| `updatedAt` | `integer` timestamp mode (Unix epoch) | `timestamp with timezone` defaulting `now()` with `$onUpdate` |
The SQLite columns do NOT have `$defaultFn` for ID generation (the consumer
provides IDs) and do NOT have `$onUpdate` for `updatedAt` (Drizzle's `$onUpdate`
is application-level; consumers must set it explicitly).
## Metagraph Tables
### `graph_types`
Stores graph type definitions (schemas for classes of graphs).
| Column | Type | Constraints | Notes |
| ----------- | ------------------- | ----------------------- | ------------------------------------------------------------ |
| id | text | PK | Consumer-generated UUID |
@@ -88,13 +88,15 @@ Stores graph type definitions (schemas for classes of graphs).
| name | text | not null, **unique** | Graph type name (e.g., "call-graph", "acl") |
| description | text | default `""` | Human-readable description |
| config | text (JSON) | not null | `GraphConfig` — directed/undirected/mixed, multi, self-loops |
| version | integer | not null, default 1 | Breaking schema version |
| version | integer | not null, default 1 | Breaking schema version (ADR-029) |
| scope | text | not null, default `"system"` | `system` / `tenant` / `user` (ADR-043) |
The `scope` column (ADR-043) controls who can create and modify graph type
definitions. System-scoped types (`acl`, `call-graph`) are seeded at setup time
and cannot be modified through the repository API.
### `node_types`
Stores node type definitions within a graph type. Each node type has a TypeBox
schema that validates node attributes.
| Column | Type | Constraints | Notes |
| ----------- | ------------------- | -------------------------------------- | ---------------------------------------- |
| id | text | PK | |
@@ -111,8 +113,6 @@ a graph type.
### `edge_types`
Stores edge type definitions within a graph type.
| Column | Type | Constraints | Notes |
| ------------------ | ------------------- | -------------------------------------- | ---------------------------------------------- |
| id | text | PK | |
@@ -126,22 +126,11 @@ Stores edge type definitions within a graph type.
| allowedSourceTypes | text (JSON) | default `[]` | Node type names valid at source endpoint |
| allowedTargetTypes | text (JSON) | default `[]` | Node type names valid at target endpoint |
**Unique constraint**: `(graphTypeId, name)` — edge type names are unique within
a graph type.
**Empty array semantics**: `allowedSourceTypes` and `allowedTargetTypes` default
to `[]` (empty JSON array) in the database. `[]` means "no restriction" — any
node type is a valid endpoint — matching the behavior of `undefined` in the
`EdgeType` schema layer. A non-empty array restricts endpoints to only the
listed node types. There is no "no types allowed" state; if edge types need to
be disabled, use a status or soft-delete pattern on the edge type definition.
The repository layer must enforce this convention consistently. See
[metagraph-module.md](./metagraph-module.md) for edge endpoint semantics.
**Unique constraint**: `(graphTypeId, name)`.
**Empty array semantics**: `[]` means "no restriction" — any node type is valid.
### `graphs`
Graph instances. Each graph belongs to a graph type.
| Column | Type | Constraints | Notes |
| ----------- | ------------------- | --------------------------------------------- | ---------------------------------------------- |
| id | text | PK | |
@@ -152,20 +141,25 @@ Graph instances. Each graph belongs to a graph type.
| name | text | not null | Graph instance name |
| description | text | default `""` | |
| status | text | not null, enum: `active`, `archived`, `draft` | Default: `draft` |
| ownerId | text | nullable | Logical reference to accounts.id (ADR-042) |
| projectId | text | nullable | Logical reference to project identity (ADR-042) |
**On `graphTypeId` set null**: When a graph type is deleted, its graphs become
orphans with `graphTypeId = null`. The application should prevent graph type
deletion if active graphs reference it, or set affected graphs' `status` to
`archived` as part of a soft-delete workflow. Orphan graphs cannot validate
their node/edge types against a missing type definition — queries against orphan
graphs should check for `graphTypeId !== null` before performing type-aware
operations.
**Scoping columns** (ADR-042): `ownerId` and `projectId` are logical references
to entities in the system DB (accounts, projects). No FK constraint because the
referenced tables live in a different database file. The consumer enforces
referential integrity at the application layer.
No `orgId` column — the tenant DB file itself IS the org scope (ADR-040).
**Indexes**: `idx_graphs_owner_id` on `(ownerId)`, `idx_graphs_project_id` on
`(projectId)`, `idx_graphs_owner_id_project_id` on `(ownerId, projectId)`.
**On `graphTypeId` set null**: Orphan graphs cannot validate their node/edge
types against a missing type definition. The application should prevent graph
type deletion if active graphs reference it.
### `nodes`
Nodes within a graph instance. Keyed by `(graphId, key)` — unique within a
graph.
| Column | Type | Constraints | Notes |
| ---------- | ------------------- | ---------------------------------- | --------------------------------------------- |
| id | text | PK | |
@@ -177,18 +171,10 @@ graph.
| attributes | text (JSON) | not null, default `{}` | Node attributes validated by node type schema |
**Unique constraint**: `(graphId, key)` — node keys are unique within a graph.
**No `nodeTypeId` column**: Nodes do not have a direct FK to `node_types`. The
node type is determined at the application layer. This is a deliberate design
decision — adding a `nodeTypeId` FK would couple the graph instance layer to the
type definition layer. The repository layer can enforce node type constraints
via validation against the graph type's schema.
**No `nodeTypeId` column**: ADR-020.
### `edges`
Edges within a graph instance. Keyed by `(graphId, key)` — unique within a
graph.
| Column | Type | Constraints | Notes |
| ------------- | ------------------- | ---------------------------------- | ---------------------------------------------------- |
| id | text | PK | |
@@ -202,32 +188,95 @@ graph.
| attributes | text (JSON) | not null, default `{}` | Edge attributes validated by edge type schema |
| undirected | integer (boolean) | default false | Treat as undirected regardless of graph type |
**Unique constraint**: `(graphId, key)` — edge keys are unique within a graph.
**Unique constraint**: `(graphId, key)`.
**Foreign keys**: `sourceNodeKey` and `targetNodeKey` reference
`(nodes.graphId, nodes.key)` with cascade delete. Deleting a node removes all
its edges.
`(nodes.graphId, nodes.key)` with cascade delete (ADR-022).
### `actors`
## Identity Tables
Standalone identity table. Currently not referenced by any relation — the
`actors` table has no FK references to or from any metagraph table and is not
included in `relations.ts`. This is a placeholder for identity data and may
become a node type in an ACL graph (based on `@alkdev/operations`'s `Identity`
interface) or remain a standalone table. See OQ-03 in [open-questions.md](./open-questions.md).
Identity tables live in the **system DB** (ADR-040, ADR-041). They provide
multi-tenant authentication and authorization infrastructure. These tables are
derived from the hub's existing identity tables; the schemas are aligned but
simplified for the storage package's scope.
| Column | Type | Constraints | Notes |
| --------- | ------------------- | --------------------------------------- | ------------------ |
| id | text | PK | |
| metadata | text (JSON) | default `{}` | |
| createdAt | integer (timestamp) | not null, default `now` | |
| updatedAt | integer (timestamp) | not null, default `now` | |
| name | text | not null | Actor display name |
| type | text | not null, enum: `human`, `llm`, `agent` | Actor type |
### `accounts`
| Column | Type | Notes |
|---------------|---------------------|-------|
| commonCols | — | id, metadata, createdAt, updatedAt |
| email | text NOT NULL UNIQUE | Unique identifier |
| displayName | text | Display name |
| accessLevel | text NOT NULL DEFAULT `user` | `admin`, `user`, `service` |
| status | text NOT NULL DEFAULT `active` | `active`, `suspended`, `deactivated` |
**Indexes**: `unq_accounts_email` UNIQUE on `(email)`.
### `organizations`
| Column | Type | Notes |
|----------|---------------------|-------|
| commonCols | — | id, metadata, createdAt, updatedAt |
| name | text NOT NULL UNIQUE | Organization name |
| slug | text NOT NULL UNIQUE | URL-friendly identifier |
| ownerId | text NOT NULL | Logical reference to accounts.id |
**Indexes**: `unq_organizations_name` UNIQUE on `(name)`, `unq_organizations_slug` UNIQUE on `(slug)`.
### `organization_members`
| Column | Type | Notes |
|-----------------|---------------------|-------|
| commonCols | — | id, metadata, createdAt, updatedAt |
| orgId | text NOT NULL | FK → organizations.id (cascade) |
| accountId | text NOT NULL | FK → accounts.id (cascade) |
| membershipLevel | text NOT NULL | `owner`, `admin`, `member` |
**Unique constraint**: `(orgId, accountId)`.
**Indexes**: `idx_org_members_account_id` on `(accountId)`.
This table is the authoritative source for org membership (ADR-045). The ACL
graph's `BelongsToEdge` is derived from it — when membership changes, the
consumer writes the SQL row first, then creates or removes the ACL edge.
### `api_keys`
| Column | Type | Notes |
|------------|---------------------|-------|
| commonCols | — | id, metadata, createdAt, updatedAt |
| ownerId | text NOT NULL | Logical reference to accounts.id |
| keyHash | text NOT NULL UNIQUE | SHA-256 hash (never stores raw key) |
| name | text | Human-readable key label |
| enabled | integer NOT NULL DEFAULT 1 | Disable without revoking |
| expiresAt | integer (timestamp) | When the key expires (null = never) |
| revokedAt | integer (timestamp) | When revoked (null = active) |
**Indexes**: `unq_api_keys_key_hash` UNIQUE on `(keyHash)`, `idx_api_keys_owner_id` on `(ownerId)`.
Keypal scope data is stored in `metadata` (`metadata.scopes`, `metadata.resources`).
The hub provides a `HubKeyStorage` adapter that reads/writes this table to
implement keypal's `Storage` interface.
### `audit_logs`
| Column | Type | Notes |
|----------|---------------------|-------|
| commonCols | — | id, metadata, createdAt, updatedAt |
| action | text NOT NULL | `created`, `revoked`, `rotated`, `login`, `access_denied` |
| ownerId | text NOT NULL | Logical reference to accounts.id |
| keyId | text | Logical reference to api_keys.id (nullable) |
| orgId | text | Logical reference to organizations.id (nullable) |
| details | text (JSON) | Action-specific context |
**Indexes**: `idx_audit_logs_owner_id` on `(ownerId)`, `idx_audit_logs_action` on `(action)`, `idx_audit_logs_created_at` on `(createdAt)`.
## Relations
Drizzle relational mappings define the following relationships:
### System DB Relations
- **organizations → organization_members**: one-to-many
- **accounts → organization_members**: one-to-many
### Tenant DB Relations
- **graphTypes → nodeTypes**: one-to-many
- **graphTypes → edgeTypes**: one-to-many
@@ -236,99 +285,101 @@ Drizzle relational mappings define the following relationships:
- **graphs → edges**: one-to-many
- **nodes → outgoing edges** (sourceNode): one-to-many
- **nodes → incoming edges** (targetNode): one-to-many
- **edges → source node**: one-to-one (via composite key)
- **edges → target node**: one-to-one (via composite key)
## Client Factory
## Client Factories
### `createSystemDatabase(client)`
Creates a Drizzle database instance with the identity schema (accounts,
organizations, organization_members, api_keys, audit_logs) attached.
```ts
import { createSqliteDatabase } from "@alkdev/storage/sqlite";
import type { SqliteDatabase } from "@alkdev/storage/sqlite";
import { createClient } from "@libsql/client";
import { createSystemDatabase } from "@alkdev/storage/sqlite";
import { open } from "@russellthehipp/honker-node";
const client = createClient({ url: "file:local.db" });
const db: SqliteDatabase = createSqliteDatabase(client);
const honkerClient = open("system.db");
const db = createSystemDatabase(honkerClient);
// Drizzle typed queries
const admins = db.select().from(accounts).where(eq(accounts.accessLevel, "admin"));
// Honker features on the same connection
db.$client.notify("account:created", { accountId: "user-1" });
```
The factory takes a pre-created `@libsql/client` client and returns a typed
Drizzle database instance with the full schema attached. This enables:
### `createTenantDatabase(client)`
- In-memory testing with `createClient({ url: ":memory:" })`
- Turso remote connections
- Custom client configuration (auth tokens, etc.)
Creates a Drizzle database instance with the metagraph schema (graph_types,
node_types, edge_types, graphs, nodes, edges) attached.
```ts
import { createTenantDatabase } from "@alkdev/storage/sqlite";
import { open } from "@russellthehipp/honker-node";
const honkerClient = open("tenant-acme.db");
const db = createTenantDatabase(honkerClient);
// Drizzle typed queries
const activeGraphs = db.select().from(graphs).where(eq(graphs.status, "active"));
// Transactional: insert node + notify in one commit
db.transaction((tx) => {
tx.insert(nodes).values({ graphId, key: "call-1", attributes: {} }).run();
tx.$honkerTx.notify("nodes:created", { graphId, key: "call-1" });
});
```
## Design Decisions
All design decisions are documented as ADRs in [decisions/](decisions/).
| ADR | Decision | Summary |
|-----|----------|---------|
| [019](decisions/019-json-text-for-schema-columns.md) | JSON text for schema columns in SQLite | SQLite uses `text` with JSON mode; application-level validation |
| [020](decisions/020-no-nodetypeid-on-nodes.md) | No nodeTypeId on nodes | Node type enforced at application layer, not via FK |
| [021](decisions/021-edge-identity-uses-consumer-keys.md) | Edge identity uses consumer-defined keys | `(graphId, key)` as unique identity within a graph |
| [022](decisions/022-composite-fks-for-node-references.md) | Composite foreign keys for node references | Edges reference `(graphId, sourceNodeKey) → (nodes.graphId, nodes.key)` |
| [006](decisions/006-enum-pattern-as-const-objects.md) | `as const` objects, not TypeScript enums | `GRAPH_STATUS`, `ACTOR_TYPE` use const objects; TypeBox uses Literal unions |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` on every table |
| [038](decisions/038-sqlite-first-pg-removed.md) | SQLite-first, PG removed | Single database host |
| [039](decisions/039-honker-as-sqlite-extension.md) | Honker as SQLite extension | DB + pub/sub + queues in one file |
| [040](decisions/040-system-db-tenant-db.md) | System DB + tenant DB | Identity in system.db, graphs in tenant-{orgId}.db |
| [041](decisions/041-identity-tables-in-storage.md) | Identity tables in storage | accounts, organizations, api_keys, audit_logs |
| [042](decisions/042-scoping-columns-on-graphs.md) | Scoping columns on graphs | `ownerId`, `projectId` on `graphs` table |
| [043](decisions/043-graph-type-scope.md) | Graph type scope | `system` / `tenant` / `user` scope on `graph_types` |
| [044](decisions/044-drizzle-honker-adapter.md) | Drizzle-Honker adapter | ~100-line session adapter |
| [045](decisions/045-org-members-authoritative-belongsto-derived.md) | org_members authoritative | SQL table is source of truth; BelongsToEdge is derived |
| [019](decisions/019-json-text-for-schema-columns.md) | JSON text for schema columns | SQLite uses `text` with JSON mode |
| [020](decisions/020-no-nodetypeid-on-nodes.md) | No nodeTypeId on nodes | Node type enforced at application layer |
| [022](decisions/022-composite-fks-for-node-references.md) | Composite FKs for node refs | Edges reference `(graphId, sourceNodeKey)` |
| [008](decisions/008-common-columns-pattern.md) | Common columns pattern | `id`, `metadata`, `createdAt`, `updatedAt` |
## Metadata Convention
## Removed: `actors` Table
Every table has a `metadata` JSON column defaulting to `{}`. This is an
extension namespace for subsystem use, following a namespacing convention:
`_subsystem.key` (e.g., `_keypal.scopes`, `_retention.expiresAt`).
The `actors` table is removed per ADR-035. `ACTOR_TYPE` is replaced by the
`IdentityType` enum in the AclGraph Module. Identity data lives in the
`accounts` table (system DB) and `PrincipalNode` in ACL graph instances
(tenant DB).
**What metadata is for**: Opaque key-value pairs that subsystems add without
schema changes. It's never queried in WHERE clauses or JOINs.
## Removed: PostgreSQL Porting Notes
**What metadata is NOT for**: A replacement for typed columns. If a field
appears in WHERE clauses, JOIN conditions, or needs a constraint, it should be a
proper column — not buried in metadata. When in doubt, add a column.
**Namespacing convention**: Subsystems should prefix their keys (e.g.,
`_callgraph.payloadRef`, `_acl.inherited`). Unprefixed keys are reserved for the
storage package itself.
PostgreSQL is no longer a target (ADR-038). All porting notes from the previous
version of this document are obsolete. The single database host is SQLite via
Honker.
## Concurrency Model
The SQLite host targets spoke deployments where a single process accesses the
database. For this model, SQLite's default journal mode is sufficient. However,
for spoke deployments that may run concurrent writes (e.g., multiple worker
threads), consumers should:
Honker opens databases in WAL mode with a bounded reader pool and single writer
slot. This handles the expected concurrency for the hub use case:
1. **Enable WAL mode**: `PRAGMA journal_mode=WAL;` — allows concurrent reads
during writes
2. **Set busy timeout**: `PRAGMA busy_timeout=5000;` — wait up to 5 seconds for
lock acquisition
3. **Use a single writer**: SQLite supports one writer at a time. If multiple
threads write, route writes through a single queue or connection
- **Reader pool**: Up to `maxReaders` (default 4) concurrent read connections.
`db.$client.query()` uses the pool automatically.
- **Writer slot**: Single exclusive writer, acquired by `transaction()`. If the
slot is occupied, subsequent `transaction()` calls block until released.
- **Write timeout**: Honker sets `busy_timeout=5000` by default. Configurable at
`open()` time.
- **WAL mode**: Enables concurrent reads during writes. Required by Honker's
reader pool architecture.
The `createSqliteDatabase()` factory does not set these pragmas — it's the
consumer's responsibility to configure the SQLite connection appropriately. The
libsql client used to create the connection can be pre-configured before passing
it to the factory.
## PostgreSQL Porting Notes
When implementing `src/pg/`, the table shapes remain the same but with these
changes:
| SQLite | PostgreSQL |
| -------------------------------- | ---------------------------------------- |
| `sqliteTable` | `pgTable` |
| `text` (JSON mode) | `jsonb` with `.$type<T>()` |
| `integer` (timestamp mode) | `timestamp` with timezone |
| `sql\`(strftime('%s', 'now'))\`` | `sql\`now()\`` |
| `integer` (boolean mode) | `boolean` |
| `text` (enum) | `pgEnum` or `text` with check constraint |
See a consumer's `commonCols` pattern (e.g., the hub's
`/workspace/@alkdev/hub/docs/architecture/storage/table-reference.md`) for
PostgreSQL reference patterns.
For multi-process deployments, set WAL mode and ensure the busy timeout is
sufficient for expected lock contention.
## References
- Drizzle ORM SQLite core: https://orm.drizzle.team/docs/sqlite-core
- libsql client: https://github.com/tursodatabase/libsql
- Hub common columns (reference consumer):
`/workspace/@alkdev/hub/docs/architecture/storage/table-reference.md`
- Operations AccessControl and Identity: `/workspace/@alkdev/operations/docs/architecture/api-surface.md`
- Honker source: `/workspace/honker/`
- Honker Node binding: `/workspace/honker/packages/honker-node/`
- Hub identity tables (provenance): `/workspace/@alkdev/hub/docs/architecture/storage/identity.md`
- Operations AccessControl: `/workspace/@alkdev/operations/docs/architecture/api-surface.md`
- Source: `src/sqlite/`