alkdev/storage
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up architecture specs: remove stale references, align docs with code, improve readability

Browse Source 
- Replace stale DD references (DD3, DD6, DD9, DD10) with proper ADR links
- Fix 'Open Question 1' → OQ-01/OQ-03 cross-references
- Rewrite metagraph-module.md 'Why TypeBox Modules' to describe capabilities
  directly instead of framing as SchemaBuilder replacement
- Remove 'Transition from SchemaBuilder' section, replace with Source Structure
- Clean up implementation path: strikethrough phases → status table
- Fix data model diagram: remove non-existent nodeTypeId, fix EdgeType label
- Align EdgeConstraints examples with actual code (add default values)
- Clarify validateNode/validateEdge error behavior in docs
- Align EncryptedDataSchema code example with actual implementation
- Fix overview.md: correct dependency table, update current state, fix TypeBox URL
- Fix forward-look.md garbled text about dbtype element migration
- Fix open-questions.md: correct OQ count (4→7 open), add summary table
- Update doc statuses: schema-evolution, encrypted-data, open-questions → reviewed
- Update AGENTS.md to reflect current implementation state
This commit is contained in:
glm-5.1
2026-05-30 09:12:24 +00:00
parent 33e66bc414
commit ed8710a7f5
9 changed files with 184 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
AGENTS.md
View File

@@ -17,6 +17,14 @@ instances) from the earlier `@ade` prototype.
├── deno.json # JSR config, imports, tasks, lint rules ├── deno.json # JSR config, imports, tasks, lint rules
├── src/ ├── src/
│ ├── graphs/ # Metagraph Module + bridge functions (no db deps) │ ├── graphs/ # Metagraph Module + bridge functions (no db deps)
│ │ ├── modules/ # TypeBox Module definitions
│ │ │ ├── metagraph.ts # Base Metagraph Module (Config, BaseNode, BaseEdge)
│ │ │ ├── call-graph.ts # CallGraph reference Module
│ │ │ ├── secret-graph.ts # SecretGraph reference Module
│ │ │ └── index.ts # Barrel re-export
│ │ ├── bridge.ts # moduleToDbSchema, validateNode, validateEdge
│ │ ├── crypto.ts # encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│ │ └── mod.ts # Re-exports all graphs exports
│ ├── sqlite/ # SQLite host (drizzle-orm/libsql) │ ├── sqlite/ # SQLite host (drizzle-orm/libsql)
│ │ ├── tables/ # Drizzle table definitions │ │ ├── tables/ # Drizzle table definitions
│ │ ├── relations.ts # Drizzle relations │ │ ├── relations.ts # Drizzle relations
@@ -24,6 +32,7 @@ instances) from the earlier `@ade` prototype.
│ │ └── client.ts # Injectable createSqliteDatabase() │ │ └── client.ts # Injectable createSqliteDatabase()
│ └── pg/ # PostgreSQL host (NOT YET IMPLEMENTED) │ └── pg/ # PostgreSQL host (NOT YET IMPLEMENTED)
└── test/ └── test/
└── reference-modules.test.ts # Metagraph, bridge, crypto tests
``` ```
### Subpath Exports (JSR/npm) ### Subpath Exports (JSR/npm)
@@ -67,19 +76,20 @@ deno publish --allow-slow-types --dry-run # Dry-run publish
The `graphs/` and `sqlite/` modules were adapted from The `graphs/` and `sqlite/` modules were adapted from
`@ade/ade-v0/packages/core/graphs` and `@ade/ade-v0/packages/storage_sqlite`. `@ade/ade-v0/packages/core/graphs` and `@ade/ade-v0/packages/storage_sqlite`.
Key changes from the originals: The codebase has diverged significantly from the originals:
- `@sinclair/typebox` → `@alkdev/typebox` - All schemas use `Type.Module()` construction (not `SchemaBuilder`)
- `drizzle-typebox` → `@alkdev/drizzlebox` - `Metagraph`, `CallGraph`, `SecretGraph` are TypeBox Modules composing via
- `@ade/core` imports → relative imports within `src/graphs/` `Import()` and `Type.Composite()`
- `import type { GraphConfig }` → `import { GraphConfig }` (TypeBox schemas are - Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`) project
both values and types) Modules to DB row values
- `Relation` type alias removed (JSR slow type) - Crypto utility ported from `@alkdev/hub/src/crypto/mod.ts` with `EncryptedDataSchema`
- TypeScript enums replaced with `as const` objects (`EnumGraphStatus` → as a TypeBox schema
`GRAPH_STATUS`) - `@sinclair/typebox` → `@alkdev/typebox`, `drizzle-typebox` → `@alkdev/drizzlebox`
- `client.ts` refactored to be injectable - TypeScript enums replaced with `as const` objects (`GRAPH_STATUS`, `ACTOR_TYPE`)
- Module-level `db` and `client` exports removed - `Type.Unknown()` used for unvalidated fields (not `Type.Any()`)
- `SchemaBuilder` removed — replaced by `Type.Module()` construction - Injectable client pattern (`createSqliteDatabase(client)` takes a pre-created client)
- No module-level side effects or state
## File Conventions ## File Conventions
@@ -105,8 +115,7 @@ See `docs/architecture/` for detailed specifications:
- `schema-evolution.md` — How graph type schemas evolve, TypeBox Value.Diff/Patch/Cast - `schema-evolution.md` — How graph type schemas evolve, TypeBox Value.Diff/Patch/Cast
for schema change detection and data migration for schema change detection and data migration
- `sqlite-host.md` — SQLite tables, relations, client factory, porting notes - `sqlite-host.md` — SQLite tables, relations, client factory, porting notes
- `encrypted-data.md` — Encrypted data design (planned), crypto utility, node - `encrypted-data.md` — Encrypted data design, crypto utility, node type modeling
type modeling
These docs describe what the package is AND what it's becoming. Items marked ⚠️ These docs describe what the package is AND what it's becoming. Items marked ⚠️
are not yet implemented. are not yet implemented.
@@ -115,9 +124,6 @@ are not yet implemented.
- `src/pg/` — PostgreSQL host (same table shapes, `pgTable` + `jsonb` + - `src/pg/` — PostgreSQL host (same table shapes, `pgTable` + `jsonb` +
`timestamp` + `pgEnum`) `timestamp` + `pgEnum`)
- `src/graphs/crypto.ts` — Crypto utility (`encrypt`, `decrypt`,
`generateEncryptionKey`, `EncryptedDataSchema`)
- Tests
- Repository/CRUD layer (currently only table definitions, no typed query - Repository/CRUD layer (currently only table definitions, no typed query
functions) functions)
- Hub-specific tables (sessions, messages, parts, call graphs, tasks, etc.) - Hub-specific tables (sessions, messages, parts, call graphs, tasks, etc.)

8
docs/architecture/README.md
View File

@@ -1,6 +1,6 @@
--- ---
status: reviewed status: reviewed
last_updated: 2026-05-29 last_updated: 2026-05-30
--- ---
# @alkdev/storage Architecture # @alkdev/storage Architecture
@@ -9,7 +9,7 @@ Typed graph storage with dual database hosts. Deno-first, published via JSR.
## Current State ## Current State
Storage is in Phase 1/2 (SQLite tables and type system implemented; repository layer, tests, PostgreSQL host, and encrypted data not yet implemented). Storage has Phase 1-3 of the metagraph implementation complete: Metagraph Module, bridge functions, reference graph type Modules (CallGraph, SecretGraph), and crypto utility. Phase 4 (graphology bridge) is pending. The repository/CRUD layer, PostgreSQL host, and additional graph types remain to be implemented.
## Architecture Documents ## Architecture Documents
@@ -18,8 +18,8 @@ Storage is in Phase 1/2 (SQLite tables and type system implemented; repository l
| [overview.md](overview.md) | Package purpose, exports, dependencies, ecosystem integration | reviewed | | [overview.md](overview.md) | Package purpose, exports, dependencies, ecosystem integration | reviewed |
| [metagraph-module.md](metagraph-module.md) | TypeBox Module type system, bridge functions, implementation path | reviewed | | [metagraph-module.md](metagraph-module.md) | TypeBox Module type system, bridge functions, implementation path | reviewed |
| [sqlite-host.md](sqlite-host.md) | SQLite tables, relations, client factory, PG porting notes | reviewed | | [sqlite-host.md](sqlite-host.md) | SQLite tables, relations, client factory, PG porting notes | reviewed |
| [schema-evolution.md](schema-evolution.md) | Value.Diff/Cast/Patch for schema migration, version strategy | draft | | [schema-evolution.md](schema-evolution.md) | Value.Diff/Cast/Patch for schema migration, version strategy | reviewed |
| [encrypted-data.md](encrypted-data.md) | Crypto utility, encrypted node type, key management | draft | | [encrypted-data.md](encrypted-data.md) | Crypto utility, encrypted node type, key management | reviewed |
| [forward-look.md](forward-look.md) | Pointers, dbtype, ujsx IR (conceptual, post-v1) | draft | | [forward-look.md](forward-look.md) | Pointers, dbtype, ujsx IR (conceptual, post-v1) | draft |
### Design Decisions ### Design Decisions

48
docs/architecture/encrypted-data.md
View File

@@ -1,6 +1,6 @@
--- ---
status: draft status: reviewed
last_updated: 2026-05-28 last_updated: 2026-05-30
--- ---
# Encrypted Data # Encrypted Data
@@ -117,9 +117,9 @@ graph edge rather than a foreign key.
| Layer | Responsibility | Package | | Layer | Responsibility | Package |
| ------------------------ | --------------------------------------------------------- | ------------------------ | | ------------------------ | --------------------------------------------------------- | ------------------------ |
| `@alkdev/storage` graphs | `EncryptedDataSchema` (TypeBox shape) | `@alkdev/storage` | | `@alkdev/storage` graphs | `EncryptedDataSchema` (TypeBox shape) | `@alkdev/storage` |
| `@alkdev/storage` crypto | `encrypt()`, `decrypt()`, `generateEncryptionKey()` | `@alkdev/storage` | | `@alkdev/storage` crypto | `encrypt()`, `decrypt()`, `generateEncryptionKey()` | `@alkdev/storage` |
| `@alkdev/storage` sqlite | Node storage (attributes contain encrypted JSON) | `@alkdev/storage/sqlite` | | `@alkdev/storage` sqlite | Node storage (attributes contain encrypted JSON) | `@alkdev/storage/sqlite` |
| `@alkdev/storage` repo | Validate schema, encrypt before insert (⚠️ not yet impl) | `@alkdev/storage` | | `@alkdev/storage` repo | Validate schema, encrypt on insert (⚠️ CRUD layer not yet built) | `@alkdev/storage` |
| Application | Key management (key ring, key rotation) | Consumer | | Application | Key management (key ring, key rotation) | Consumer |
## EncryptedData Schema ## EncryptedData Schema
@@ -131,20 +131,18 @@ schema in `@alkdev/storage`:
import { Type } from "@alkdev/typebox"; import { Type } from "@alkdev/typebox";
export const EncryptedDataSchema = Type.Object({ export const EncryptedDataSchema = Type.Object({
keyVersion: Type.Integer({ keyVersion: Type.Integer({ minimum: 1 }),
minimum: 1, salt: Type.String(), // Base64-encoded 16-byte PBKDF2 salt
description: "Encryption key version for rotation", iv: Type.String(), // Base64-encoded 12-byte AES-GCM initialization vector
}), data: Type.String(), // Base64-encoded AES-256-GCM ciphertext
salt: Type.String({ description: "Base64-encoded 16-byte PBKDF2 salt" }),
iv: Type.String({
description: "Base64-encoded 12-byte AES-GCM initialization vector",
}),
data: Type.String({ description: "Base64-encoded AES-256-GCM ciphertext" }),
}); });
``` ```
This is the same structure as the hub's `EncryptedData` interface but as a The fields contain: `keyVersion` — which encryption key version was used (enables key
TypeBox schema, enabling runtime validation when inserting encrypted nodes. rotation), `salt` — base64-encoded 16-byte PBKDF2 salt, `iv` — base64-encoded
12-byte AES-GCM initialization vector, `data` — base64-encoded AES-256-GCM
ciphertext. This is the same structure as the hub's `EncryptedData` interface but
as a TypeBox schema, enabling runtime validation when inserting encrypted nodes.
## Crypto Utility ## Crypto Utility
@@ -255,18 +253,24 @@ const attributes = {
## Export Plan ## Export Plan
The crypto module will be exported from the main `@alkdev/storage` package (no The crypto module is exported from the main `@alkdev/storage` package (no
db deps): db deps):
``` ```
src/graphs/ src/graphs/
├── modules/metagraph.ts # Metagraph Module (BaseNode, BaseEdge, Config) ├── modules/
├── crypto.ts # new: encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema │ ├── metagraph.ts # Metagraph Module (Config, BaseNode, BaseEdge)
└── mod.ts # re-exports all of the above │ ├── call-graph.ts # CallGraph reference Module
│ ├── secret-graph.ts # SecretGraph reference Module (uses EncryptedDataSchema)
│ └── index.ts # Barrel re-export
├── bridge.ts # moduleToDbSchema, validateNode, validateEdge
├── crypto.ts # encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema
└── mod.ts # Re-exports all of the above
``` ```
This keeps the encryption utility in the zero-dep export path (it only uses Web The encryption utility is in the zero-dep export path (it only uses Web Crypto
Crypto API and `@alkdev/typebox` for the schema). API and `@alkdev/typebox` for the schema). `SecretGraph` in `secret-graph.ts`
composes `EncryptedDataSchema` into a node type via `Type.Composite`.
## Open Questions ## Open Questions

19
docs/architecture/forward-look.md
View File

@@ -7,9 +7,10 @@ last_updated: 2026-05-30
How the Module-based metagraph connects to the broader @alkdev ecosystem — How the Module-based metagraph connects to the broader @alkdev ecosystem —
typed graph pointers, dbtype table rendering, and the ujsx universal IR typed graph pointers, dbtype table rendering, and the ujsx universal IR
pipeline. These are forward-looking designs that justify why certain pipeline. These are forward-looking designs that justify why certain structural
structural decisions are made now (DD9, DD10 in decisions were made now
[metagraph-module.md](./metagraph-module.md)). (pointer abstraction deferred per [ADR-017](./decisions/017-pointer-abstraction-is-forward-looking.md),
dbtype integration deferred per [ADR-018](./decisions/018-dbtype-integration-is-post-v1.md)).
## Overview ## Overview
@@ -163,9 +164,10 @@ integration is deferred because:
- The Module pattern for graph types can be adopted independently (no dbtype - The Module pattern for graph types can be adopted independently (no dbtype
dependency) dependency)
When dbtype reaches Phase 1 (implementation), storage can adopt dbtype element When dbtype reaches Phase 1 (implementation), storage can migrate from
to dbtype elements one table at a time. The Module-based graph type definitions Drizzle table definitions to dbtype elements one table at a time. The
are already compatible — they're both TypeBox `Type.Module` objects. Module-based graph type definitions are already compatible — they're both
TypeBox `Type.Module` objects.
## ujsx as Universal IR ## ujsx as Universal IR
@@ -209,7 +211,10 @@ Rendered to different hosts:
| Capability | Status | | Capability | Status |
|-----------|--------| |-----------|--------|
| `Type.Module` for graph type definitions | ✅ Ready to implement now | | `Type.Module` for graph type definitions | ✅ Implemented — Metagraph, CallGraph, SecretGraph Modules |
| Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`) | ✅ Implemented |
| Reference graph type Modules (CallGraph, SecretGraph) | ✅ Implemented |
| Crypto utility (`encrypt`, `decrypt`, `generateEncryptionKey`, `EncryptedDataSchema`) | ✅ Implemented |
| Codegen from TypeScript interfaces → Module entries | ✅ TsToModule exists | | Codegen from TypeScript interfaces → Module entries | ✅ TsToModule exists |
| dbtype element trees → Drizzle tables | ⚠️ dbtype Phase 0, no implementation | | dbtype element trees → Drizzle tables | ⚠️ dbtype Phase 0, no implementation |
| `<graphSchema>` ujsx elements | ⚠️ Conceptual — needs HostConfig design | | `<graphSchema>` ujsx elements | ⚠️ Conceptual — needs HostConfig design |

148
docs/architecture/metagraph-module.md
View File

@@ -1,6 +1,6 @@
--- ---
status: reviewed status: reviewed
last_updated: 2026-05-29 last_updated: 2026-05-30
--- ---
# Metagraph as TypeBox Module # Metagraph as TypeBox Module
@@ -35,17 +35,14 @@ GraphType "call-graph" (directed, multi, self-loops allowed)
└── EdgeType "depends_on" → allowedSourceTypes: ["Call", "Subcall"], allowedTargetTypes: ["Call", "Subcall"] └── EdgeType "depends_on" → allowedSourceTypes: ["Call", "Subcall"], allowedTargetTypes: ["Call", "Subcall"]
Graph "session-abc-call-graph" (instance) Graph "session-abc-call-graph" (instance)
│ graphTypeId → GraphType "call-graph" │ graphTypeId → GraphType "call-graph"
│ status: "active" │ status: "active"
│ │
├── Node "call-001" → nodeTypeId → NodeType "call" ├── Node "call-001" → type: "call", attributes: { requestId, operationId, status, ... }
│ └── attributes: { requestId, operationId, status, ... } ├── Node "call-002" → type: "subcall", attributes: { requestId, parentRequestId, ... }
├── Node "call-002" → nodeTypeId → NodeType "subcall" └── Edge "edge-001" → type: "triggered"
│ └── attributes: { requestId, parentRequestId, ... } sourceNodeKey: "call-001", targetNodeKey: "call-002"
└── Edge "edge-001" → edgeTypeId → NodeType "triggered" attributes: { type: "triggered" }
└── attributes: { type: "triggered" }
sourceNodeKey: "call-001"
targetNodeKey: "call-002"
``` ```
Nodes and edges use a **composite identity model**: identified by Nodes and edges use a **composite identity model**: identified by
@@ -71,18 +68,19 @@ reference each other. `Type.Module` is the natural fit:
- **`Value.Check(Module.Import("CallNode"), data)`** — runtime validation - **`Value.Check(Module.Import("CallNode"), data)`** — runtime validation
- **`Static<typeof Module>`** — TypeScript types from the Module - **`Static<typeof Module>`** — TypeScript types from the Module
This replaces the removed `SchemaBuilder`, which produced a flat The Module pattern enables three capabilities that flat schema records cannot
`Record<string, NodeType>` + `Record<string, EdgeType>`. That approach had provide:
three limitations that Modules solve natively:
1. **No cross-graph-type references** — a call graph node type couldn't 1. **Cross-graph-type references** — A call graph node type can reference
reference `CallStatus` from `@alkdev/flowgraph` without manual `CallStatus` from `@alkdev/flowgraph` via `Module.Import()` without manual
`Type.Intersect`. Each package duplicated types independently. `Type.Intersect`. Each package composes from shared Modules rather than
2. **No graphology compatibility** — the flat JSON output didn't map to duplicating types.
graphology's `import()`/`export()`. Consumers manually mapped node/edge 2. **Graphology compatibility** — Module entries map directly to graphology's
attributes. `SerializedGraph` format. Each `*Node` entry validates `nodes[].attributes`,
3. **No codegen leverage** — `TsToModule` generates TypeBox Modules from each `*Edge` entry validates `edges[].attributes`.
TypeScript interfaces, but the builder couldn't consume Module output. 3. **Codegen leverage** — `TsToModule.Generate()` produces TypeBox Modules from
TypeScript interfaces, and storage's `Type.Composite()` can consume Module
output directly.
This aligns with the pattern proven in `@alkdev/ujsx`, where `UJSX` is a Module This aligns with the pattern proven in `@alkdev/ujsx`, where `UJSX` is a Module
with `UPrimitive`, `UElement`, `URoot`, `UNode` recursively referencing each with `UPrimitive`, `UElement`, `URoot`, `UNode` recursively referencing each
@@ -254,7 +252,7 @@ into the importing Module's JSON Schema. When `CallGraph` imports from
in `$defs`. The repository layer stores **dereferenced entry schemas** — each in `$defs`. The repository layer stores **dereferenced entry schemas** — each
`node_types` row gets its entry's resolved JSON Schema (with inline `$defs` for `node_types` row gets its entry's resolved JSON Schema (with inline `$defs` for
just its transitive references), not the entire importing Module. This avoids just its transitive references), not the entire importing Module. This avoids
storage bloat and version coupling (DD6). storage bloat and version coupling (ADR-014).
### BaseNode/BaseEdge: Import vs Local Re-declaration ### BaseNode/BaseEdge: Import vs Local Re-declaration
@@ -306,8 +304,13 @@ export const CallGraph = Type.Module({
]), ]),
TriggeredEdgeConstraints: Type.Object({ TriggeredEdgeConstraints: Type.Object({
edgeType: Type.Literal("triggered"), edgeType: Type.Literal("triggered"),
allowedSourceTypes: Type.Array(Type.String()), // ["Call"] allowedSourceTypes: Type.Array(Type.String(), { default: ["Call"] }),
allowedTargetTypes: Type.Array(Type.String()), // ["Call", "Subcall"] allowedTargetTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }),
}),
DependsOnEdgeConstraints: Type.Object({
edgeType: Type.Literal("depends_on"),
allowedSourceTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }),
allowedTargetTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }),
}), }),
DependsOnEdge: Type.Composite([ DependsOnEdge: Type.Composite([
Metagraph.Import("BaseEdge"), Metagraph.Import("BaseEdge"),
@@ -332,7 +335,9 @@ at Module-to-DB projection time.
**Empty array semantics**: In the DB, `[]` means "no restriction" (any node **Empty array semantics**: In the DB, `[]` means "no restriction" (any node
type valid). In the Module, omitting the `*EdgeConstraints` entry means the same type valid). In the Module, omitting the `*EdgeConstraints` entry means the same
thing. An explicit entry with empty arrays is not valid — it would mean "no node thing. An explicit entry with empty arrays is not valid — it would mean "no node
types are valid at this endpoint," which is nonsensical. types are valid at this endpoint," which is nonsensical. The `default` values on
`allowedSourceTypes`/`allowedTargetTypes` arrays provide `Value.Create()` defaults
for the constraint object; they do not affect the "no restriction" semantics.
## Entry Naming Convention ## Entry Naming Convention
@@ -421,9 +426,9 @@ the Module entry.
### Bridge Functions ### Bridge Functions
#### `moduleToDbSchema(module)` #### `moduleToDbSchema(module, name)`
Maps a graph type Module to DB row values for the metagraph tables. Maps a graph type Module to DB row values for the metagraph tables. The `name` parameter specifies the graph type name (e.g., `"call-graph"`, `"secret"`).
```ts ```ts
interface DbGraphTypeRow { interface DbGraphTypeRow {
@@ -449,7 +454,7 @@ interface DbSchema {
edgeTypes: DbEdgeTypeRow[]; edgeTypes: DbEdgeTypeRow[];
} }
function moduleToDbSchema(module: TModule): DbSchema function moduleToDbSchema(module: TModule<TProperties>, name: string): DbSchema
``` ```
**Error behavior**: Throws on: **Error behavior**: Throws on:
@@ -473,8 +478,10 @@ function validateEdge(module: TModule, entryName: string, data: unknown): boolea
``` ```
Returns `true` if data passes `Value.Check` against the resolved Module entry. Returns `true` if data passes `Value.Check` against the resolved Module entry.
Throws if `entryName` doesn't match an `*Node`/`*Edge` entry in the Module. Throws if `entryName` doesn't match an `*Node`/`*Edge` entry in the Module
Does NOT throw on invalid data — returns `false`. (rejects `BaseNode`/`BaseEdge` as well — these are base schemas, not concrete
types). Returns `false` (does NOT throw) when data fails validation against
the entry schema.
### Performance ### Performance
@@ -506,65 +513,50 @@ TypeScript interface → TsToModule.Generate() → Module entry
Since flowgraph already defines `CallNodeAttrs` as a standalone TypeBox schema, Since flowgraph already defines `CallNodeAttrs` as a standalone TypeBox schema,
the codegen can produce a Module entry from it. Storage's `CallGraph` Module then the codegen can produce a Module entry from it. Storage's `CallGraph` Module then
composes `BaseNode` with `CallNodeAttrs` via `Type.Composite`, or imports from composes `BaseNode` with `CallNodeAttrs` via `Type.Composite`, or imports from
the flowgraph Module if flowgraph exports one (see Open Question 1). the flowgraph Module if flowgraph exports one (see OQ-01).
## Transition from SchemaBuilder ## Source Structure
The existing `schemaBuilder.ts` and `types.ts` use a different approach that is Graph type definitions are `Type.Module` objects composed from a base `Metagraph` Module. The current source structure:
being replaced:
| Before (unreleased) | After | ```
|---------|-----| src/graphs/
| `types.ts` — standalone schemas | `modules/metagraph.ts` — `Metagraph` Module | ├── modules/
| `schemaBuilder.ts` — fluent builder | Removed — replaced by `Type.Module()` construction | │ ├── metagraph.ts # Base Module: Config, BaseNode, BaseEdge
| `types.ts` — `BaseNodeAttributes`, `BaseEdgeAttributes` | `Metagraph` Module entries | │ ├── call-graph.ts # CallGraph Module (reference implementation)
| `types.ts` — `GraphConfig`, `GraphStatus`, `GraphBaseType` | `Metagraph` Module entries + const objects | │ ├── secret-graph.ts # SecretGraph Module (reference implementation)
| `allowedSourceTypes`/`allowedTargetTypes` as DB columns only | Named `*EdgeConstraints` Module entries (projected to DB columns) | │ └── index.ts # Barrel re-export
| No concrete graph type Modules | `modules/call-graph.ts`, `modules/acl-graph.ts`, etc. | ├── bridge.ts # moduleToDbSchema, validateNode, validateEdge
| No bridge between Module ↔ DB ↔ graphology | `bridge.ts` — validation, DB mapping, graphology format | ├── crypto.ts # encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
└── mod.ts # Re-exports all graphs exports
```
Note: `Type.Any()` used in the old `types.ts` for `metadata` and `schema` fields `Type.Unknown()` is the canonical choice for "no validation applied" fields (e.g., `metadata`, `input`, `output`). It communicates intent explicitly, even though it produces the same JSON Schema (`{}`) as `Type.Any()`.
is replaced by `Type.Unknown()` in the Module approach. Both produce `{}` in
JSON Schema, but `Type.Unknown()` is the canonical choice — it explicitly
communicates "no validation applied."
**What doesn't change**: The 6 metagraph database tables, their columns, and The 6 metagraph database tables, their columns, and relations are defined in [sqlite-host.md](./sqlite-host.md). `allowedSourceTypes` and `allowedTargetTypes` are DB columns with Module entries as the source of truth, projected to columns by `moduleToDbSchema()`.
relations remain the same. SQLite host table definitions, client factory, and
drizzlebox-generated schemas are unchanged. The `@alkdev/typebox` dependency is `GRAPH_STATUS` and `GraphStatus` are exported from `modules/metagraph.ts` as standalone constants and TypeBox schemas (not Module entries) because they're used by the SQLite host for the `graphs.status` column enum.
unchanged. The encryption utility (planned) is unchanged. `allowedSourceTypes`
and `allowedTargetTypes` remain DB columns with the same semantics — Module
entries are the source of truth, projected to columns by `moduleToDbSchema()`.
## Implementation Path ## Implementation Path
1. **Phase 1**: Add `Metagraph` Module, replace `types.ts` and remove | Phase | Description | Status |
`schemaBuilder.ts`. Export Module construction API. |-------|-------------|--------|
2. **Phase 2**: Add `bridge.ts` with `moduleToDbSchema()`, `validateNode()`, | 1 | Add `Metagraph` Module, export Module construction API | ✅ Complete |
`validateEdge()`. | 2 | Add `bridge.ts` with `moduleToDbSchema()`, `validateNode()`, `validateEdge()` | ✅ Complete |
3. **Phase 3**: Add `modules/` directory with reference graph type Modules | 3 | Add `modules/` directory with reference graph type Modules (CallGraph, SecretGraph) | ✅ Complete |
(call-graph, acl-graph, task-graph, secret-graph). These use | 4 | Add `moduleToGraphology()` and `fromGraphologyExport()` for the graphology bridge | Pending |
`Metagraph.Import()` for `BaseNode`/`BaseEdge` and `Type.Composite()`
for node/edge type composition.
4. **Phase 4**: Add `moduleToGraphology()` and `fromGraphologyExport()` for the
graphology bridge. Storage produces the format, flowgraph consumes it.
Acceptance criteria:
- **Phase 2 complete**: `moduleToDbSchema()` produces values compatible with
all 6 metagraph tables
- **Phase 3 complete**: Reference Modules validate against their
flowgraph/taskgraph counterparts
### Relationship to Other Packages ### Relationship to Other Packages
| Package | What changes | What stays | | Package | Status | Notes |
|---------|-------------|------------| |---------|--------|-------|
| `@alkdev/storage` | `types.ts` → Module, `schemaBuilder.ts` → removed, new `modules/` and `bridge.ts` | Tables, relations, crypto, client factory | | `@alkdev/storage` | Module structure complete | `modules/`, `bridge.ts`, `crypto.ts` all implemented |
| `@alkdev/flowgraph` | `CallNodeAttrs`, `CallEdgeAttrs`, `CallStatus` become Module entries (optional, exported from `/schema`) | FlowGraph class, analysis, all runtime logic | | `@alkdev/flowgraph` | No change | `CallNodeAttrs`, `CallEdgeAttrs`, `CallStatus` may become Module entries in future |
| `@alkdev/taskgraph` | `TaskGraphNodeAttributes`, `DependencyEdge` become Module entries (optional) | TaskGraph class, analysis, all runtime logic | | `@alkdev/taskgraph` | No change | `TaskGraphNodeAttributes`, `DependencyEdge` may become Module entries in future |
| `@alkdev/operations` | `Identity`, `AccessControl` become Module entries (optional) | Registry, call protocol, adapters | | `@alkdev/operations` | No change | `Identity`, `AccessControl` may become Module entries in future |
| `@alkdev/pubsub` | No change | Transport layer | | `@alkdev/pubsub` | No change | Transport layer |
| `@alkdev/ujsx` | No change (already a Module) | The pattern we're following | | `@alkdev/ujsx` | No change (already a Module) | The pattern we're following |
| `@alkdev/dbtype` | No change (Phase 0) | Future: storage table defs could be dbtype element trees | | `@alkdev/dbtype` | Not yet implemented | Future: storage table defs could be dbtype element trees |
## Design Decisions ## Design Decisions

24
docs/architecture/open-questions.md
View File

@@ -1,6 +1,6 @@
--- ---
status: draft status: reviewed
last_updated: 2026-05-29 last_updated: 2026-05-30
--- ---
# Open Questions Tracker # Open Questions Tracker
@@ -9,6 +9,26 @@ Cross-cutting compilation of all unresolved questions across the storage archite
When a question is resolved, update its status to `resolved` and add a resolution note. Once all questions in a theme are resolved, the theme section can be removed and the resolution noted in the relevant ADR. When a question is resolved, update its status to `resolved` and add a resolution note. Once all questions in a theme are resolved, the theme section can be removed and the resolution noted in the relevant ADR.
## Summary
| Status | Count |
|--------|-------|
| Open | 7 |
| Partially resolved | 1 |
| Resolved | 8 |
**Open questions requiring decisions:**
- **OQ-03** (actors table design) — deferred to ACL design
- **OQ-04** (repository layer host-specific vs host-agnostic) — start host-specific
- **OQ-07** (encryptRaw performance) — low priority, add if needed
- **OQ-10** (Edit[] classification) — needs POC
- **OQ-11** (auto-migrate vs explicit consumer action) — conditional on OQ-10
- **OQ-12** (schema evolution vs event-sourced replay) — post-v1 concern
- **OQ-13** (schema evolution events in event stream) — post-v1
**Partially resolved:**
- **OQ-01** (flowgraph Module export) — storage can start without it
## How to Use This Document ## How to Use This Document
- Each question has an **ID** (e.g., OQ-01), **status**, **origin** (which doc(s)), and **priority** - Each question has an **ID** (e.g., OQ-01), **status**, **origin** (which doc(s)), and **priority**

33
docs/architecture/overview.md
View File

@@ -1,6 +1,6 @@
--- ---
status: reviewed status: reviewed
last_updated: 2026-05-29 last_updated: 2026-05-30
--- ---
# @alkdev/storage — Overview # @alkdev/storage — Overview
@@ -29,23 +29,32 @@ ecosystem.
@alkdev/storage/ @alkdev/storage/
├── mod.ts → re-exports graphs/ (zero db deps) ├── mod.ts → re-exports graphs/ (zero db deps)
├── src/ ├── src/
│ ├── graphs/ → Metagraph Module, bridge functions (no db deps) │ ├── graphs/ → Metagraph Module, bridge functions, crypto (no db deps)
│ │ ├── modules/ → TypeBox Module definitions
│ │ │ ├── metagraph.ts → Config, BaseNode, BaseEdge
│ │ │ ├── call-graph.ts → CallGraph reference Module
│ │ │ ├── secret-graph.ts → SecretGraph reference Module
│ │ │ └── index.ts → barrel re-export
│ │ ├── bridge.ts → moduleToDbSchema, validateNode, validateEdge
│ │ ├── crypto.ts → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema
│ │ └── mod.ts → re-exports all graphs exports
│ ├── sqlite/ → SQLite host (drizzle-orm/libsql) │ ├── sqlite/ → SQLite host (drizzle-orm/libsql)
│ │ ├── tables/ → drizzle table definitions │ │ ├── tables/ → drizzle table definitions
│ │ ├── relations.ts → drizzle relational mappings │ │ ├── relations.ts → drizzle relational mappings
│ │ ├── schema.ts → barrel re-export │ │ ├── schema.ts → barrel re-export
│ │ └── client.ts → injectable createSqliteDatabase() │ │ └── client.ts → injectable createSqliteDatabase()
│ └── pg/ → PostgreSQL host (NOT YET IMPLEMENTED) │ └── pg/ → PostgreSQL host (NOT YET IMPLEMENTED)
└── test/ → empty — tests not yet written └── test/
└── reference-modules.test.ts → Metagraph, bridge, crypto tests
``` ```
### Subpath Exports (JSR/npm) ### Subpath Exports (JSR/npm)
| Export | Contents | Dependencies | | Export | Contents | Dependencies |
| ------------------------ | --------------------------------------- | --------------------------------------- | | ------------------------ | --------------------------------------- | --------------------------------------- |
| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox`, `@alkdev/drizzlebox` | | `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` |
| `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` | | `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` |
| `@alkdev/storage/sqlite` | SQLite tables, relations, client | + `drizzle-orm`, `@libsql/client` | | `@alkdev/storage/sqlite` | SQLite tables, relations, client | `@alkdev/drizzlebox`, `drizzle-orm`, `@libsql/client` |
| `@alkdev/storage/pg` | PostgreSQL tables, relations, client | ⚠️ NOT YET IMPLEMENTED | | `@alkdev/storage/pg` | PostgreSQL tables, relations, client | ⚠️ NOT YET IMPLEMENTED |
The `./graphs` subpath exists because the source code lives in `src/graphs/` and The `./graphs` subpath exists because the source code lives in `src/graphs/` and
@@ -105,22 +114,24 @@ consumed by the hub and spokes, not by storage itself.
### Implemented ### Implemented
- Graph schema types and Metagraph Module (replaces SchemaBuilder) - Metagraph Module (`Type.Module` with Config, BaseNode, BaseEdge entries)
- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`)
- Reference graph type Modules (CallGraph, SecretGraph)
- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`)
- SQLite host: 6 metagraph tables + actors table + Drizzle relations + client - SQLite host: 6 metagraph tables + actors table + Drizzle relations + client
factory factory
- TypeBox select/insert schemas generated from Drizzle tables (drizzlebox) - TypeBox select/insert schemas generated from Drizzle tables (drizzlebox)
- Reference module tests (bridge functions, validation, Module composition)
### Not Yet Implemented ### Not Yet Implemented
| Gap | Priority | Notes | | Gap | Priority | Notes |
| ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- | | ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- |
| Encrypted data node type + crypto utility | **Critical** | ⚠️ Not yet implemented. API keys and secrets at rest. See [encrypted-data.md](./encrypted-data.md). |
| Repository/CRUD layer | High | ⚠️ Not yet implemented. Typed insert, find, update, delete functions for graphs, nodes, edges. No dependency on `@alkdev/operations` — consumer wires CRUD into registry. | | Repository/CRUD layer | High | ⚠️ Not yet implemented. Typed insert, find, update, delete functions for graphs, nodes, edges. No dependency on `@alkdev/operations` — consumer wires CRUD into registry. |
| Tests | High | Zero tests exist. Needed before any real use. |
| PostgreSQL host | Medium | Same table shapes, `pgTable` + `jsonb` + `timestamp` + `pgEnum`. Stub only. | | PostgreSQL host | Medium | Same table shapes, `pgTable` + `jsonb` + `timestamp` + `pgEnum`. Stub only. |
| Call graph type | Medium | Informed by `@alkdev/flowgraph`'s `CallNodeAttrs`/`CallEdgeAttrs` schemas and `@alkdev/operations`' call protocol events. Not hub-specific — any consumer that tracks call invocations needs this. | | ACL graph type | Medium | Access control as a graph. Informed by `@alkdev/operations`' `Identity` and `AccessControl`. Depends on CRUD layer. |
| ACL graph type | Medium | Access control as a graph. Informed by `@alkdev/operations`' `Identity` and `AccessControl`. Depends on encrypted data and CRUD layer. |
| Task graph type | Low | Informed by `@alkdev/taskgraph`'s `TaskGraphNodeAttributes` and `DependencyEdge` schemas. | | Task graph type | Low | Informed by `@alkdev/taskgraph`'s `TaskGraphNodeAttributes` and `DependencyEdge` schemas. |
| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4 of the metagraph implementation path. |
## Ecosystem Integration ## Ecosystem Integration
@@ -232,5 +243,5 @@ questions affecting this package:
- Source heritage: `@ade/ade-v0/packages/core/graphs` and - Source heritage: `@ade/ade-v0/packages/core/graphs` and
`@ade/ade-v0/packages/storage_sqlite` `@ade/ade-v0/packages/storage_sqlite`
- Drizzle ORM: https://orm.drizzle.team/ - Drizzle ORM: https://orm.drizzle.team/
- TypeBox: https://github.com/sinclairzx/typebox - TypeBox: `/workspace/@alkdev/typebox/`
- JSR: https://jsr.io/ - JSR: https://jsr.io/

6
docs/architecture/schema-evolution.md
View File

@@ -1,6 +1,6 @@
--- ---
status: draft status: reviewed
last_updated: 2026-05-28 last_updated: 2026-05-30
--- ---
# Schema Evolution # Schema Evolution
@@ -504,4 +504,4 @@ questions affecting schema evolution:
- Event Log as Source of Truth (ADR-005): `/workspace/@alkdev/flowgraph/docs/architecture/decisions/005-event-log-as-source-of-truth.md` - Event Log as Source of Truth (ADR-005): `/workspace/@alkdev/flowgraph/docs/architecture/decisions/005-event-log-as-source-of-truth.md`
- Call protocol: `/workspace/@alkdev/operations/docs/architecture/call-protocol.md` - Call protocol: `/workspace/@alkdev/operations/docs/architecture/call-protocol.md`
- Metagraph Module: [metagraph-module.md](./metagraph-module.md) - Metagraph Module: [metagraph-module.md](./metagraph-module.md)
- Schema versioning in the data model: [metagraph-module.md](./metagraph-module.md) (Versioning section and DD3) - Schema versioning in the data model: ADR-029, [metagraph-module.md](./metagraph-module.md)

5
docs/architecture/sqlite-host.md
View File

@@ -1,6 +1,6 @@
--- ---
status: reviewed status: reviewed
last_updated: 2026-05-29 last_updated: 2026-05-30
--- ---
# SQLite Host # SQLite Host
@@ -214,8 +214,7 @@ Standalone identity table. Currently not referenced by any relation — the
`actors` table has no FK references to or from any metagraph table and is not `actors` table has no FK references to or from any metagraph table and is not
included in `relations.ts`. This is a placeholder for identity data and may included in `relations.ts`. This is a placeholder for identity data and may
become a node type in an ACL graph (based on `@alkdev/operations`'s `Identity` become a node type in an ACL graph (based on `@alkdev/operations`'s `Identity`
interface) or remain a standalone table. See [overview.md](./overview.md) Open interface) or remain a standalone table. See OQ-03 in [open-questions.md](./open-questions.md).
Question 1.
| Column | Type | Constraints | Notes | | Column | Type | Constraints | Notes |
| --------- | ------------------- | --------------------------------------- | ------------------ | | --------- | ------------------- | --------------------------------------- | ------------------ |