alkdev/storage
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
95e02f939d96f054d92dba51280bf859523fb1be
storage/docs/architecture/overview.md

19 KiB
Raw Blame History

status, last_updated
status last_updated
draft 2026-05-28

@alkdev/storage — Overview

Typed graph storage with dual database hosts. Deno-first, published via JSR.

Purpose

@alkdev/storage provides a metagraph storage model: graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Instances of these type definitions become actual graphs populated with nodes and edges.

This pattern replaces domain-specific table proliferation with a small number of general-purpose tables that can model anything — call graphs, ACL rules, task dependencies, encrypted secrets — while enforcing schema integrity through TypeBox validation.

The package evolved from @ade/ade-v0/packages/core/graphs and @ade/ade-v0/packages/storage_sqlite, simplified and refactored for the @alkdev ecosystem.

Architecture

@alkdev/storage/
├── mod.ts                   → re-exports graphs/ (zero db deps)
├── src/
│   ├── graphs/              → Metagraph Module, bridge functions (no db deps)
│   ├── sqlite/              → SQLite host (drizzle-orm/libsql)
│   │   ├── tables/          → drizzle table definitions
│   │   ├── relations.ts     → drizzle relational mappings
│   │   ├── schema.ts        → barrel re-export
│   │   └── client.ts        → injectable createSqliteDatabase()
│   └── pg/                  → PostgreSQL host (NOT YET IMPLEMENTED)
└── test/                    → empty — tests not yet written

Subpath Exports (JSR/npm)

Export Contents Dependencies
@alkdev/storage Graph schema types, Metagraph Module @alkdev/typebox, @alkdev/drizzlebox
@alkdev/storage/graphs Same as . — alias for the main export Same as .
@alkdev/storage/sqlite SQLite tables, relations, client + drizzle-orm, @libsql/client
@alkdev/storage/pg PostgreSQL tables, relations, client ⚠️ NOT YET IMPLEMENTED

The ./graphs subpath exists because the source code lives in src/graphs/ and the main mod.ts re-exports it. Importing from either @alkdev/storage or @alkdev/storage/graphs yields the same types and Metagraph Module.

Terminology

Term Definition
Metagraph A type system where graph types define schemas, node types define data shapes within those graphs, and edge types define typed relationships. Graph instances are concrete data conforming to these type definitions.
Hub The central service in the hub-spoke architecture. A consumer of @alkdev/storage — uses the PostgreSQL host for persistent graph storage. The hub also depends on @alkdev/operations, @alkdev/pubsub, @alkdev/flowgraph.
Spoke A local/embedded instance that runs per-project or per-session. A consumer of @alkdev/storage — uses the SQLite host for local graph storage.
Graph type A class of graphs (e.g., "call-graph", "acl"). Defines structural constraints (directed/undirected/mixed, multi-edges, self-loops) and the valid node/edge type vocabularies. Stored in the graph_types table.
Node type A category of node within a graph type. Defines the attribute schema for nodes of that type. Stored in the node_types table.
Edge type A category of edge within a graph type. Defines the attribute schema and optionally restricts which node types can be source/target. Stored in the edge_types table.
Graph instance A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the graphs table.
Consumer Code that imports @alkdev/storage (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers.
Repository layer ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. No dependency on @alkdev/operations — the consumer wires CRUD into the registry.
Validation boundary The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database.

Design Decisions

D1: Deno-first, JSR publishes, npm comes free

The package is published to JSR (deno publish). npm compatibility is automatic via JSR's npm layer (@jsr/alkdev__storage). No separate dnt build step.

D2: Metagraph over domain-specific tables

Instead of a table per domain concept (call graphs, ACL rules, task trees), we define graph types with typed node and edge schemas. A "call graph" is a graph type with specific node types (operation call, subcall) and edge types (triggered, depends_on). An "ACL graph" is a graph type with node types (account, resource) and edge types (can_read, can_write).

This trades some query convenience for generality. Domain-specific queries are built on top of the graph query layer, not baked into table schemas.

D3: Type.Module as the primary API surface

The Type.Module() construction API is the intended way to define graph type definitions. The Metagraph Module provides base entries (BaseNode, BaseEdge, Config); concrete graph types compose them via Metagraph.Import() and Type.Composite(). The SchemaBuilder is removed.

This replaces the earlier fluent builder pattern. The Module format provides native Type.Ref() for internal references, Module.Import() for cross-package references, and JSON Schema $defs that map directly to DB storage.

D4: Injectable clients, no module-level side effects

createSqliteDatabase(client) receives a pre-created client. Module-level side effects (auto-connections, env-based configuration) are forbidden. This enables testing with in-memory databases and containerized deployment patterns.

D5: Drizzle + TypeBox (via drizzlebox) as the table definition pattern

Drizzle table definitions are the single source of truth for database schema. @alkdev/drizzlebox generates TypeBox Select* and Insert* schemas from Drizzle tables, enabling runtime validation without manual schema duplication.

D6: Enumeration pattern — as const objects, not TypeScript enums

All enumerations use the as const object pattern (e.g., GRAPH_STATUS = { Active: "active", ... } as const) rather than TypeScript enum. This avoids JSR slow-type issues and provides a consistent pattern across the codebase. The TypeBox schemas use Type.Union of Type.Literal values derived from the const object.

D7: No comments in code

Per project convention across @alkdev packages, source files contain no inline comments. Documentation lives in architecture docs and TypeBox schema descriptions.

D8: Common columns pattern

All tables share id (text PK), metadata (JSON text defaulting to {}), createdAt, and updatedAt (integer timestamps in SQLite, will be timestamptz in PG). This ensures every row has auditability and extensibility.

Dependencies

Package Purpose Layer
@alkdev/typebox Runtime schema validation graphs/
@alkdev/drizzlebox Generate TypeBox from Drizzle tables sqlite/
drizzle-orm ORM, table definitions, queries sqlite/ (and future pg/)
@libsql/client SQLite client (libsql/turso) sqlite/
postgres PostgreSQL client pg/ (not yet used)

@alkdev/typebox and @alkdev/drizzlebox are npm packages (not yet on JSR). JSR handles npm dependencies natively.

Ecosystem packages are not runtime dependencies of @alkdev/storage. All ecosystem references in this document describe consumer-side data shapes and integration patterns, not import dependencies. The @alkdev/operations, @alkdev/pubsub, @alkdev/flowgraph, and @alkdev/taskgraph packages are consumed by the hub and spokes, not by storage itself.

What Exists vs. What's Needed

Implemented

  • Graph schema types and Metagraph Module (replaces SchemaBuilder)
  • SQLite host: 6 metagraph tables + actors table + Drizzle relations + client factory
  • TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)

Not Yet Implemented

Gap Priority Notes
Encrypted data node type + crypto utility Critical ⚠️ Not yet implemented. API keys and secrets at rest. See encrypted-data.md.
Repository/CRUD layer High ⚠️ Not yet implemented. Typed insert, find, update, delete functions for graphs, nodes, edges. No dependency on @alkdev/operations — consumer wires CRUD into registry.
Tests High Zero tests exist. Needed before any real use.
PostgreSQL host Medium Same table shapes, pgTable + jsonb + timestamp + pgEnum. Stub only.
Call graph type Medium Informed by @alkdev/flowgraph's CallNodeAttrs/CallEdgeAttrs schemas and @alkdev/operations' call protocol events. Not hub-specific — any consumer that tracks call invocations needs this.
ACL graph type Medium Access control as a graph. Informed by @alkdev/operations' Identity and AccessControl. Depends on encrypted data and CRUD layer.
Task graph type Low Informed by @alkdev/taskgraph's TaskGraphNodeAttributes and DependencyEdge schemas.

Ecosystem Integration

@alkdev/storage is a data layer package consumed by other packages in the @alkdev ecosystem. It does not depend on the hub — the dependency flows the other way. The hub consumes storage (along with operations, pubsub, flowgraph, and taskgraph) as part of its architecture.

Dependency Direction

@alkdev/pubsub          ← transport only (no storage dependency)
    ↑
@alkdev/operations      ← call protocol, registry, identity, access control
    ↑                   (depends on: @alkdev/pubsub, @alkdev/typebox)
@alkdev/flowgraph       ← call graph schema, operation graph, workflow templates
    ↑                   (depends on: @alkdev/operations [peer], @alkdev/typebox)
@alkdev/taskgraph       ← task dependency graph schema, cost-benefit analysis
                        (depends on: @alkdev/typebox)

@alkdev/storage         ← YOU ARE HERE — typed graph persistence
                        (depends on: @alkdev/typebox, @alkdev/drizzlebox)

    ↑                       ↑
    |                       |
Hub / Spoke          Any consumer that needs
(consumes all)       persistent graph storage

The key insight: @alkdev/storage provides the persistence primitives (schemas, tables, repository layer). The domain semantics (what a call graph means, what identity looks like, how access control works) are defined by the packages above. Storage stores the shapes those packages define; it does not define the semantics itself.

What Comes from Where

Concept Source package Storage's role
Call protocol events (call.requested, call.responded, etc.) @alkdev/operations Storage persists the outcomes — graphs with CallNodeAttrs nodes
Identity (id, scopes, resources) @alkdev/operations Storage stores identity as node attributes; Identity is a data shape, not a storage concept
Access control (AccessControl, requiredScopes) @alkdev/operations Storage's ACL graph type mirrors the operations AccessControl schema as graph structure
Call graph schema (CallNodeAttrs, CallEdgeAttrs, CallStatus) @alkdev/flowgraph Storage persists these in-memory shapes to the database
Task graph schema (TaskGraphNodeAttributes, DependencyEdge) @alkdev/taskgraph Storage persists task dependency shapes
Event transport (TypedEventTarget, EventEnvelope) @alkdev/pubsub Storage is not involved in event routing; it stores the events' outcomes

Repository Layer Bridging Pattern (Consumer-Side Concern)

The repository layer in @alkdev/storage provides typed CRUD — no @alkdev/operations dependency. A consumer-side bridging module can then wire these CRUD functions into the @alkdev/operations registry, analogous to how drizzle-graphql auto-generates a GraphQL schema from Drizzle tables — but using operations (queries, mutations, subscriptions) instead of GraphQL resolvers. This works because:

  1. @alkdev/operations already maps closely to GraphQL's queries/mutations/subscriptions (it was modeled after that pattern)
  2. @alkdev/pubsub provides the subscription transport (forked from graphql-yoga's pubsub with additions)
  3. @alkdev/storage's metagraph tables are the data source, analogous to Drizzle tables for drizzle-graphql

The bridging module would live in a consumer package (e.g., the hub or a dedicated @alkdev/storage-operations adapter), not in @alkdev/storage itself, to avoid circular dependencies:

@alkdev/storage → defines types + tables (no operations dependency)
@alkdev/operations → defines call protocol + registry (no storage dependency)
Consumer (hub / adapter) → imports both, generates operations from schemas

Avoiding Circular Dependencies

Neither @alkdev/storage nor @alkdev/operations should depend on each other directly. Storage defines the schema types and database tables; operations defines the call protocol and execution model. The consumer (hub, spoke, or adapter package) imports both and bridges them. This preserves the single-responsibility principle and allows each package to evolve independently.

If shared type definitions are needed (e.g., Identity referenced in both storage node attributes and operations call events), they should either:

  1. Be duplicated in each package with a documented correspondence (acceptable for small, stable types)
  2. Be extracted to a minimal shared types package if the duplication becomes burdensome

Open Questions

  1. Should actors be a node type or a standalone table? Currently actors is a standalone table in the SQLite host that isn't referenced by any relation. If identity/authentication is a graph (ACL nodes based on @alkdev/operations's Identity interface), actors become node types. If identity is a domain concept that needs special query patterns (auth lookups, session joins), standalone tables may be better. Decision: defer until ACL design, informed by @alkdev/operations's AccessControl model.

  2. Should the repository layer be host-specific or host-agnostic? A host-agnostic repository (insert graph, find nodes by type) requires an abstraction over Drizzle's query builder. A host-specific repository is simpler but means duplicating query logic for PG. Decision: start host-specific in SQLite, extract common patterns later.

  3. Encrypted data scope: Should encryption be per-attribute, per-node, or per-graph? Per-attribute (like hub's client_secrets.value) allows selective encryption. Per-node encrypts the entire attributes blob. Per-graph is overkill. Decision: per-attribute, modeled as an encrypted node type with a dedicated attribute for the ciphertext.

  4. Key management scope: @alkdev/storage should provide the encryption/decryption primitives but NOT key management. The consuming application provides the key ring. This keeps the storage package agnostic to deployment-specific secret management.

  5. Schema evolution strategy: When graph type schemas evolve (new node types, changed attribute schemas), how are changes detected and data migrated? TypeBox's Value.Diff can diff schemas-as-JSON to detect changes, Value.Cast can migrate data shapes, and Value.Check can verify compatibility. The version field on graph_types tracks breaking changes. See schema-evolution.md for the full design.

  6. Should the repository layer live in @alkdev/storage or in a consumer package? Decision: the repository CRUD layer (host-specific typed queries, schema validation before writes) belongs in @alkdev/storage. The operations bridging layer (generating OperationSpecs from metagraph schemas) belongs in a consumer or adapter package. These are separate concerns — CRUD is a storage concern; call protocol integration is an application concern. The repository layer in @alkdev/storage has no dependency on @alkdev/operations. It performs typed inserts, finds, updates, and deletes with schema validation. The consumer then wires these CRUD functions into the operations registry if desired.

References

  • Metagraph Module evolution: metagraph-module.md
  • Schema evolution via TypeBox value system: schema-evolution.md
  • Forward-looking connections: forward-look.md
  • Operations architecture: /workspace/@alkdev/operations/docs/architecture/README.md
  • Pubsub architecture: /workspace/@alkdev/pubsub/docs/architecture/README.md
  • Flowgraph architecture: /workspace/@alkdev/flowgraph/docs/architecture/README.md
  • Taskgraph architecture: /workspace/@alkdev/taskgraph_ts/docs/architecture/README.md
  • drizzle-graphql (reference for repo bridging pattern): /workspace/drizzle-graphql/
  • Source heritage: @ade/ade-v0/packages/core/graphs and @ade/ade-v0/packages/storage_sqlite
  • Drizzle ORM: https://orm.drizzle.team/
  • TypeBox: https://github.com/sinclairzx/typebox
  • JSR: https://jsr.io/
Reference in New Issue View Git Blame Copy Permalink