diff --git a/AGENTS.md b/AGENTS.md index 614a1f7..dfbdf88 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,14 @@ instances) from the earlier `@ade` prototype. ├── deno.json # JSR config, imports, tasks, lint rules ├── src/ │ ├── graphs/ # Metagraph Module + bridge functions (no db deps) +│ │ ├── modules/ # TypeBox Module definitions +│ │ │ ├── metagraph.ts # Base Metagraph Module (Config, BaseNode, BaseEdge) +│ │ │ ├── call-graph.ts # CallGraph reference Module +│ │ │ ├── secret-graph.ts # SecretGraph reference Module +│ │ │ └── index.ts # Barrel re-export +│ │ ├── bridge.ts # moduleToDbSchema, validateNode, validateEdge +│ │ ├── crypto.ts # encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema +│ │ └── mod.ts # Re-exports all graphs exports │ ├── sqlite/ # SQLite host (drizzle-orm/libsql) │ │ ├── tables/ # Drizzle table definitions │ │ ├── relations.ts # Drizzle relations @@ -24,6 +32,7 @@ instances) from the earlier `@ade` prototype. │ │ └── client.ts # Injectable createSqliteDatabase() │ └── pg/ # PostgreSQL host (NOT YET IMPLEMENTED) └── test/ + └── reference-modules.test.ts # Metagraph, bridge, crypto tests ``` ### Subpath Exports (JSR/npm) @@ -67,19 +76,20 @@ deno publish --allow-slow-types --dry-run # Dry-run publish The `graphs/` and `sqlite/` modules were adapted from `@ade/ade-v0/packages/core/graphs` and `@ade/ade-v0/packages/storage_sqlite`. -Key changes from the originals: +The codebase has diverged significantly from the originals: -- `@sinclair/typebox` → `@alkdev/typebox` -- `drizzle-typebox` → `@alkdev/drizzlebox` -- `@ade/core` imports → relative imports within `src/graphs/` -- `import type { GraphConfig }` → `import { GraphConfig }` (TypeBox schemas are - both values and types) -- `Relation` type alias removed (JSR slow type) -- TypeScript enums replaced with `as const` objects (`EnumGraphStatus` → - `GRAPH_STATUS`) -- `client.ts` refactored to be injectable -- Module-level `db` and `client` exports removed -- `SchemaBuilder` removed — replaced by `Type.Module()` construction +- All schemas use `Type.Module()` construction (not `SchemaBuilder`) +- `Metagraph`, `CallGraph`, `SecretGraph` are TypeBox Modules composing via + `Import()` and `Type.Composite()` +- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`) project + Modules to DB row values +- Crypto utility ported from `@alkdev/hub/src/crypto/mod.ts` with `EncryptedDataSchema` + as a TypeBox schema +- `@sinclair/typebox` → `@alkdev/typebox`, `drizzle-typebox` → `@alkdev/drizzlebox` +- TypeScript enums replaced with `as const` objects (`GRAPH_STATUS`, `ACTOR_TYPE`) +- `Type.Unknown()` used for unvalidated fields (not `Type.Any()`) +- Injectable client pattern (`createSqliteDatabase(client)` takes a pre-created client) +- No module-level side effects or state ## File Conventions @@ -105,8 +115,7 @@ See `docs/architecture/` for detailed specifications: - `schema-evolution.md` — How graph type schemas evolve, TypeBox Value.Diff/Patch/Cast for schema change detection and data migration - `sqlite-host.md` — SQLite tables, relations, client factory, porting notes -- `encrypted-data.md` — Encrypted data design (planned), crypto utility, node - type modeling +- `encrypted-data.md` — Encrypted data design, crypto utility, node type modeling These docs describe what the package is AND what it's becoming. Items marked ⚠️ are not yet implemented. @@ -115,9 +124,6 @@ are not yet implemented. - `src/pg/` — PostgreSQL host (same table shapes, `pgTable` + `jsonb` + `timestamp` + `pgEnum`) -- `src/graphs/crypto.ts` — Crypto utility (`encrypt`, `decrypt`, - `generateEncryptionKey`, `EncryptedDataSchema`) -- Tests - Repository/CRUD layer (currently only table definitions, no typed query functions) - Hub-specific tables (sessions, messages, parts, call graphs, tasks, etc.) diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 7f89c20..f2f821c 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,6 +1,6 @@ --- status: reviewed -last_updated: 2026-05-29 +last_updated: 2026-05-30 --- # @alkdev/storage Architecture @@ -9,7 +9,7 @@ Typed graph storage with dual database hosts. Deno-first, published via JSR. ## Current State -Storage is in Phase 1/2 (SQLite tables and type system implemented; repository layer, tests, PostgreSQL host, and encrypted data not yet implemented). +Storage has Phase 1-3 of the metagraph implementation complete: Metagraph Module, bridge functions, reference graph type Modules (CallGraph, SecretGraph), and crypto utility. Phase 4 (graphology bridge) is pending. The repository/CRUD layer, PostgreSQL host, and additional graph types remain to be implemented. ## Architecture Documents @@ -18,8 +18,8 @@ Storage is in Phase 1/2 (SQLite tables and type system implemented; repository l | [overview.md](overview.md) | Package purpose, exports, dependencies, ecosystem integration | reviewed | | [metagraph-module.md](metagraph-module.md) | TypeBox Module type system, bridge functions, implementation path | reviewed | | [sqlite-host.md](sqlite-host.md) | SQLite tables, relations, client factory, PG porting notes | reviewed | -| [schema-evolution.md](schema-evolution.md) | Value.Diff/Cast/Patch for schema migration, version strategy | draft | -| [encrypted-data.md](encrypted-data.md) | Crypto utility, encrypted node type, key management | draft | +| [schema-evolution.md](schema-evolution.md) | Value.Diff/Cast/Patch for schema migration, version strategy | reviewed | +| [encrypted-data.md](encrypted-data.md) | Crypto utility, encrypted node type, key management | reviewed | | [forward-look.md](forward-look.md) | Pointers, dbtype, ujsx IR (conceptual, post-v1) | draft | ### Design Decisions diff --git a/docs/architecture/encrypted-data.md b/docs/architecture/encrypted-data.md index 11adb3c..6f829e6 100644 --- a/docs/architecture/encrypted-data.md +++ b/docs/architecture/encrypted-data.md @@ -1,6 +1,6 @@ --- -status: draft -last_updated: 2026-05-28 +status: reviewed +last_updated: 2026-05-30 --- # Encrypted Data @@ -117,9 +117,9 @@ graph edge rather than a foreign key. | Layer | Responsibility | Package | | ------------------------ | --------------------------------------------------------- | ------------------------ | | `@alkdev/storage` graphs | `EncryptedDataSchema` (TypeBox shape) | `@alkdev/storage` | -| `@alkdev/storage` crypto | `encrypt()`, `decrypt()`, `generateEncryptionKey()` | `@alkdev/storage` | -| `@alkdev/storage` sqlite | Node storage (attributes contain encrypted JSON) | `@alkdev/storage/sqlite` | -| `@alkdev/storage` repo | Validate schema, encrypt before insert (⚠️ not yet impl) | `@alkdev/storage` | +| `@alkdev/storage` crypto | `encrypt()`, `decrypt()`, `generateEncryptionKey()` | `@alkdev/storage` | +| `@alkdev/storage` sqlite | Node storage (attributes contain encrypted JSON) | `@alkdev/storage/sqlite` | +| `@alkdev/storage` repo | Validate schema, encrypt on insert (⚠️ CRUD layer not yet built) | `@alkdev/storage` | | Application | Key management (key ring, key rotation) | Consumer | ## EncryptedData Schema @@ -131,20 +131,18 @@ schema in `@alkdev/storage`: import { Type } from "@alkdev/typebox"; export const EncryptedDataSchema = Type.Object({ - keyVersion: Type.Integer({ - minimum: 1, - description: "Encryption key version for rotation", - }), - salt: Type.String({ description: "Base64-encoded 16-byte PBKDF2 salt" }), - iv: Type.String({ - description: "Base64-encoded 12-byte AES-GCM initialization vector", - }), - data: Type.String({ description: "Base64-encoded AES-256-GCM ciphertext" }), + keyVersion: Type.Integer({ minimum: 1 }), + salt: Type.String(), // Base64-encoded 16-byte PBKDF2 salt + iv: Type.String(), // Base64-encoded 12-byte AES-GCM initialization vector + data: Type.String(), // Base64-encoded AES-256-GCM ciphertext }); ``` -This is the same structure as the hub's `EncryptedData` interface but as a -TypeBox schema, enabling runtime validation when inserting encrypted nodes. +The fields contain: `keyVersion` — which encryption key version was used (enables key +rotation), `salt` — base64-encoded 16-byte PBKDF2 salt, `iv` — base64-encoded +12-byte AES-GCM initialization vector, `data` — base64-encoded AES-256-GCM +ciphertext. This is the same structure as the hub's `EncryptedData` interface but +as a TypeBox schema, enabling runtime validation when inserting encrypted nodes. ## Crypto Utility @@ -255,18 +253,24 @@ const attributes = { ## Export Plan -The crypto module will be exported from the main `@alkdev/storage` package (no +The crypto module is exported from the main `@alkdev/storage` package (no db deps): ``` src/graphs/ -├── modules/metagraph.ts # Metagraph Module (BaseNode, BaseEdge, Config) -├── crypto.ts # new: encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema -└── mod.ts # re-exports all of the above +├── modules/ +│ ├── metagraph.ts # Metagraph Module (Config, BaseNode, BaseEdge) +│ ├── call-graph.ts # CallGraph reference Module +│ ├── secret-graph.ts # SecretGraph reference Module (uses EncryptedDataSchema) +│ └── index.ts # Barrel re-export +├── bridge.ts # moduleToDbSchema, validateNode, validateEdge +├── crypto.ts # encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema +└── mod.ts # Re-exports all of the above ``` -This keeps the encryption utility in the zero-dep export path (it only uses Web -Crypto API and `@alkdev/typebox` for the schema). +The encryption utility is in the zero-dep export path (it only uses Web Crypto +API and `@alkdev/typebox` for the schema). `SecretGraph` in `secret-graph.ts` +composes `EncryptedDataSchema` into a node type via `Type.Composite`. ## Open Questions diff --git a/docs/architecture/forward-look.md b/docs/architecture/forward-look.md index 437ff0b..40ce209 100644 --- a/docs/architecture/forward-look.md +++ b/docs/architecture/forward-look.md @@ -7,9 +7,10 @@ last_updated: 2026-05-30 How the Module-based metagraph connects to the broader @alkdev ecosystem — typed graph pointers, dbtype table rendering, and the ujsx universal IR -pipeline. These are forward-looking designs that justify why certain -structural decisions are made now (DD9, DD10 in -[metagraph-module.md](./metagraph-module.md)). +pipeline. These are forward-looking designs that justify why certain structural +decisions were made now +(pointer abstraction deferred per [ADR-017](./decisions/017-pointer-abstraction-is-forward-looking.md), +dbtype integration deferred per [ADR-018](./decisions/018-dbtype-integration-is-post-v1.md)). ## Overview @@ -163,9 +164,10 @@ integration is deferred because: - The Module pattern for graph types can be adopted independently (no dbtype dependency) -When dbtype reaches Phase 1 (implementation), storage can adopt dbtype element -to dbtype elements one table at a time. The Module-based graph type definitions -are already compatible — they're both TypeBox `Type.Module` objects. +When dbtype reaches Phase 1 (implementation), storage can migrate from +Drizzle table definitions to dbtype elements one table at a time. The +Module-based graph type definitions are already compatible — they're both +TypeBox `Type.Module` objects. ## ujsx as Universal IR @@ -209,7 +211,10 @@ Rendered to different hosts: | Capability | Status | |-----------|--------| -| `Type.Module` for graph type definitions | ✅ Ready to implement now | +| `Type.Module` for graph type definitions | ✅ Implemented — Metagraph, CallGraph, SecretGraph Modules | +| Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`) | ✅ Implemented | +| Reference graph type Modules (CallGraph, SecretGraph) | ✅ Implemented | +| Crypto utility (`encrypt`, `decrypt`, `generateEncryptionKey`, `EncryptedDataSchema`) | ✅ Implemented | | Codegen from TypeScript interfaces → Module entries | ✅ TsToModule exists | | dbtype element trees → Drizzle tables | ⚠️ dbtype Phase 0, no implementation | | `` ujsx elements | ⚠️ Conceptual — needs HostConfig design | diff --git a/docs/architecture/metagraph-module.md b/docs/architecture/metagraph-module.md index fb4c000..f69b60c 100644 --- a/docs/architecture/metagraph-module.md +++ b/docs/architecture/metagraph-module.md @@ -1,6 +1,6 @@ --- status: reviewed -last_updated: 2026-05-29 +last_updated: 2026-05-30 --- # Metagraph as TypeBox Module @@ -35,17 +35,14 @@ GraphType "call-graph" (directed, multi, self-loops allowed) └── EdgeType "depends_on" → allowedSourceTypes: ["Call", "Subcall"], allowedTargetTypes: ["Call", "Subcall"] Graph "session-abc-call-graph" (instance) - │ graphTypeId → GraphType "call-graph" - │ status: "active" - │ - ├── Node "call-001" → nodeTypeId → NodeType "call" - │ └── attributes: { requestId, operationId, status, ... } - ├── Node "call-002" → nodeTypeId → NodeType "subcall" - │ └── attributes: { requestId, parentRequestId, ... } - └── Edge "edge-001" → edgeTypeId → NodeType "triggered" - └── attributes: { type: "triggered" } - sourceNodeKey: "call-001" - targetNodeKey: "call-002" + │ graphTypeId → GraphType "call-graph" + │ status: "active" + │ + ├── Node "call-001" → type: "call", attributes: { requestId, operationId, status, ... } + ├── Node "call-002" → type: "subcall", attributes: { requestId, parentRequestId, ... } + └── Edge "edge-001" → type: "triggered" + sourceNodeKey: "call-001", targetNodeKey: "call-002" + attributes: { type: "triggered" } ``` Nodes and edges use a **composite identity model**: identified by @@ -71,18 +68,19 @@ reference each other. `Type.Module` is the natural fit: - **`Value.Check(Module.Import("CallNode"), data)`** — runtime validation - **`Static`** — TypeScript types from the Module -This replaces the removed `SchemaBuilder`, which produced a flat -`Record` + `Record`. That approach had -three limitations that Modules solve natively: +The Module pattern enables three capabilities that flat schema records cannot +provide: -1. **No cross-graph-type references** — a call graph node type couldn't - reference `CallStatus` from `@alkdev/flowgraph` without manual - `Type.Intersect`. Each package duplicated types independently. -2. **No graphology compatibility** — the flat JSON output didn't map to - graphology's `import()`/`export()`. Consumers manually mapped node/edge - attributes. -3. **No codegen leverage** — `TsToModule` generates TypeBox Modules from - TypeScript interfaces, but the builder couldn't consume Module output. +1. **Cross-graph-type references** — A call graph node type can reference + `CallStatus` from `@alkdev/flowgraph` via `Module.Import()` without manual + `Type.Intersect`. Each package composes from shared Modules rather than + duplicating types. +2. **Graphology compatibility** — Module entries map directly to graphology's + `SerializedGraph` format. Each `*Node` entry validates `nodes[].attributes`, + each `*Edge` entry validates `edges[].attributes`. +3. **Codegen leverage** — `TsToModule.Generate()` produces TypeBox Modules from + TypeScript interfaces, and storage's `Type.Composite()` can consume Module + output directly. This aligns with the pattern proven in `@alkdev/ujsx`, where `UJSX` is a Module with `UPrimitive`, `UElement`, `URoot`, `UNode` recursively referencing each @@ -254,7 +252,7 @@ into the importing Module's JSON Schema. When `CallGraph` imports from in `$defs`. The repository layer stores **dereferenced entry schemas** — each `node_types` row gets its entry's resolved JSON Schema (with inline `$defs` for just its transitive references), not the entire importing Module. This avoids -storage bloat and version coupling (DD6). +storage bloat and version coupling (ADR-014). ### BaseNode/BaseEdge: Import vs Local Re-declaration @@ -306,8 +304,13 @@ export const CallGraph = Type.Module({ ]), TriggeredEdgeConstraints: Type.Object({ edgeType: Type.Literal("triggered"), - allowedSourceTypes: Type.Array(Type.String()), // ["Call"] - allowedTargetTypes: Type.Array(Type.String()), // ["Call", "Subcall"] + allowedSourceTypes: Type.Array(Type.String(), { default: ["Call"] }), + allowedTargetTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }), + }), + DependsOnEdgeConstraints: Type.Object({ + edgeType: Type.Literal("depends_on"), + allowedSourceTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }), + allowedTargetTypes: Type.Array(Type.String(), { default: ["Call", "Subcall"] }), }), DependsOnEdge: Type.Composite([ Metagraph.Import("BaseEdge"), @@ -332,7 +335,9 @@ at Module-to-DB projection time. **Empty array semantics**: In the DB, `[]` means "no restriction" (any node type valid). In the Module, omitting the `*EdgeConstraints` entry means the same thing. An explicit entry with empty arrays is not valid — it would mean "no node -types are valid at this endpoint," which is nonsensical. +types are valid at this endpoint," which is nonsensical. The `default` values on +`allowedSourceTypes`/`allowedTargetTypes` arrays provide `Value.Create()` defaults +for the constraint object; they do not affect the "no restriction" semantics. ## Entry Naming Convention @@ -421,9 +426,9 @@ the Module entry. ### Bridge Functions -#### `moduleToDbSchema(module)` +#### `moduleToDbSchema(module, name)` -Maps a graph type Module to DB row values for the metagraph tables. +Maps a graph type Module to DB row values for the metagraph tables. The `name` parameter specifies the graph type name (e.g., `"call-graph"`, `"secret"`). ```ts interface DbGraphTypeRow { @@ -449,7 +454,7 @@ interface DbSchema { edgeTypes: DbEdgeTypeRow[]; } -function moduleToDbSchema(module: TModule): DbSchema +function moduleToDbSchema(module: TModule, name: string): DbSchema ``` **Error behavior**: Throws on: @@ -473,8 +478,10 @@ function validateEdge(module: TModule, entryName: string, data: unknown): boolea ``` Returns `true` if data passes `Value.Check` against the resolved Module entry. -Throws if `entryName` doesn't match an `*Node`/`*Edge` entry in the Module. -Does NOT throw on invalid data — returns `false`. +Throws if `entryName` doesn't match an `*Node`/`*Edge` entry in the Module +(rejects `BaseNode`/`BaseEdge` as well — these are base schemas, not concrete +types). Returns `false` (does NOT throw) when data fails validation against +the entry schema. ### Performance @@ -506,65 +513,50 @@ TypeScript interface → TsToModule.Generate() → Module entry Since flowgraph already defines `CallNodeAttrs` as a standalone TypeBox schema, the codegen can produce a Module entry from it. Storage's `CallGraph` Module then composes `BaseNode` with `CallNodeAttrs` via `Type.Composite`, or imports from -the flowgraph Module if flowgraph exports one (see Open Question 1). +the flowgraph Module if flowgraph exports one (see OQ-01). -## Transition from SchemaBuilder +## Source Structure -The existing `schemaBuilder.ts` and `types.ts` use a different approach that is -being replaced: +Graph type definitions are `Type.Module` objects composed from a base `Metagraph` Module. The current source structure: -| Before (unreleased) | After | -|---------|-----| -| `types.ts` — standalone schemas | `modules/metagraph.ts` — `Metagraph` Module | -| `schemaBuilder.ts` — fluent builder | Removed — replaced by `Type.Module()` construction | -| `types.ts` — `BaseNodeAttributes`, `BaseEdgeAttributes` | `Metagraph` Module entries | -| `types.ts` — `GraphConfig`, `GraphStatus`, `GraphBaseType` | `Metagraph` Module entries + const objects | -| `allowedSourceTypes`/`allowedTargetTypes` as DB columns only | Named `*EdgeConstraints` Module entries (projected to DB columns) | -| No concrete graph type Modules | `modules/call-graph.ts`, `modules/acl-graph.ts`, etc. | -| No bridge between Module ↔ DB ↔ graphology | `bridge.ts` — validation, DB mapping, graphology format | +``` +src/graphs/ +├── modules/ +│ ├── metagraph.ts # Base Module: Config, BaseNode, BaseEdge +│ ├── call-graph.ts # CallGraph Module (reference implementation) +│ ├── secret-graph.ts # SecretGraph Module (reference implementation) +│ └── index.ts # Barrel re-export +├── bridge.ts # moduleToDbSchema, validateNode, validateEdge +├── crypto.ts # encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema +└── mod.ts # Re-exports all graphs exports +``` -Note: `Type.Any()` used in the old `types.ts` for `metadata` and `schema` fields -is replaced by `Type.Unknown()` in the Module approach. Both produce `{}` in -JSON Schema, but `Type.Unknown()` is the canonical choice — it explicitly -communicates "no validation applied." +`Type.Unknown()` is the canonical choice for "no validation applied" fields (e.g., `metadata`, `input`, `output`). It communicates intent explicitly, even though it produces the same JSON Schema (`{}`) as `Type.Any()`. -**What doesn't change**: The 6 metagraph database tables, their columns, and -relations remain the same. SQLite host table definitions, client factory, and -drizzlebox-generated schemas are unchanged. The `@alkdev/typebox` dependency is -unchanged. The encryption utility (planned) is unchanged. `allowedSourceTypes` -and `allowedTargetTypes` remain DB columns with the same semantics — Module -entries are the source of truth, projected to columns by `moduleToDbSchema()`. +The 6 metagraph database tables, their columns, and relations are defined in [sqlite-host.md](./sqlite-host.md). `allowedSourceTypes` and `allowedTargetTypes` are DB columns with Module entries as the source of truth, projected to columns by `moduleToDbSchema()`. + +`GRAPH_STATUS` and `GraphStatus` are exported from `modules/metagraph.ts` as standalone constants and TypeBox schemas (not Module entries) because they're used by the SQLite host for the `graphs.status` column enum. ## Implementation Path -1. **Phase 1**: Add `Metagraph` Module, replace `types.ts` and remove - `schemaBuilder.ts`. Export Module construction API. -2. **Phase 2**: Add `bridge.ts` with `moduleToDbSchema()`, `validateNode()`, - `validateEdge()`. -3. **Phase 3**: Add `modules/` directory with reference graph type Modules - (call-graph, acl-graph, task-graph, secret-graph). These use - `Metagraph.Import()` for `BaseNode`/`BaseEdge` and `Type.Composite()` - for node/edge type composition. -4. **Phase 4**: Add `moduleToGraphology()` and `fromGraphologyExport()` for the - graphology bridge. Storage produces the format, flowgraph consumes it. - -Acceptance criteria: -- **Phase 2 complete**: `moduleToDbSchema()` produces values compatible with - all 6 metagraph tables -- **Phase 3 complete**: Reference Modules validate against their - flowgraph/taskgraph counterparts +| Phase | Description | Status | +|-------|-------------|--------| +| 1 | Add `Metagraph` Module, export Module construction API | ✅ Complete | +| 2 | Add `bridge.ts` with `moduleToDbSchema()`, `validateNode()`, `validateEdge()` | ✅ Complete | +| 3 | Add `modules/` directory with reference graph type Modules (CallGraph, SecretGraph) | ✅ Complete | +| 4 | Add `moduleToGraphology()` and `fromGraphologyExport()` for the graphology bridge | Pending | ### Relationship to Other Packages -| Package | What changes | What stays | -|---------|-------------|------------| -| `@alkdev/storage` | `types.ts` → Module, `schemaBuilder.ts` → removed, new `modules/` and `bridge.ts` | Tables, relations, crypto, client factory | -| `@alkdev/flowgraph` | `CallNodeAttrs`, `CallEdgeAttrs`, `CallStatus` become Module entries (optional, exported from `/schema`) | FlowGraph class, analysis, all runtime logic | -| `@alkdev/taskgraph` | `TaskGraphNodeAttributes`, `DependencyEdge` become Module entries (optional) | TaskGraph class, analysis, all runtime logic | -| `@alkdev/operations` | `Identity`, `AccessControl` become Module entries (optional) | Registry, call protocol, adapters | +| Package | Status | Notes | +|---------|--------|-------| +| `@alkdev/storage` | Module structure complete | `modules/`, `bridge.ts`, `crypto.ts` all implemented | +| `@alkdev/flowgraph` | No change | `CallNodeAttrs`, `CallEdgeAttrs`, `CallStatus` may become Module entries in future | +| `@alkdev/taskgraph` | No change | `TaskGraphNodeAttributes`, `DependencyEdge` may become Module entries in future | +| `@alkdev/operations` | No change | `Identity`, `AccessControl` may become Module entries in future | | `@alkdev/pubsub` | No change | Transport layer | | `@alkdev/ujsx` | No change (already a Module) | The pattern we're following | -| `@alkdev/dbtype` | No change (Phase 0) | Future: storage table defs could be dbtype element trees | +| `@alkdev/dbtype` | Not yet implemented | Future: storage table defs could be dbtype element trees | ## Design Decisions diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index 2e01f7c..ea17fed 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -1,6 +1,6 @@ --- -status: draft -last_updated: 2026-05-29 +status: reviewed +last_updated: 2026-05-30 --- # Open Questions Tracker @@ -9,6 +9,26 @@ Cross-cutting compilation of all unresolved questions across the storage archite When a question is resolved, update its status to `resolved` and add a resolution note. Once all questions in a theme are resolved, the theme section can be removed and the resolution noted in the relevant ADR. +## Summary + +| Status | Count | +|--------|-------| +| Open | 7 | +| Partially resolved | 1 | +| Resolved | 8 | + +**Open questions requiring decisions:** +- **OQ-03** (actors table design) — deferred to ACL design +- **OQ-04** (repository layer host-specific vs host-agnostic) — start host-specific +- **OQ-07** (encryptRaw performance) — low priority, add if needed +- **OQ-10** (Edit[] classification) — needs POC +- **OQ-11** (auto-migrate vs explicit consumer action) — conditional on OQ-10 +- **OQ-12** (schema evolution vs event-sourced replay) — post-v1 concern +- **OQ-13** (schema evolution events in event stream) — post-v1 + +**Partially resolved:** +- **OQ-01** (flowgraph Module export) — storage can start without it + ## How to Use This Document - Each question has an **ID** (e.g., OQ-01), **status**, **origin** (which doc(s)), and **priority** diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 7f0d8ec..bc0c528 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,6 +1,6 @@ --- status: reviewed -last_updated: 2026-05-29 +last_updated: 2026-05-30 --- # @alkdev/storage — Overview @@ -29,23 +29,32 @@ ecosystem. @alkdev/storage/ ├── mod.ts → re-exports graphs/ (zero db deps) ├── src/ -│ ├── graphs/ → Metagraph Module, bridge functions (no db deps) +│ ├── graphs/ → Metagraph Module, bridge functions, crypto (no db deps) +│ │ ├── modules/ → TypeBox Module definitions +│ │ │ ├── metagraph.ts → Config, BaseNode, BaseEdge +│ │ │ ├── call-graph.ts → CallGraph reference Module +│ │ │ ├── secret-graph.ts → SecretGraph reference Module +│ │ │ └── index.ts → barrel re-export +│ │ ├── bridge.ts → moduleToDbSchema, validateNode, validateEdge +│ │ ├── crypto.ts → encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema +│ │ └── mod.ts → re-exports all graphs exports │ ├── sqlite/ → SQLite host (drizzle-orm/libsql) │ │ ├── tables/ → drizzle table definitions │ │ ├── relations.ts → drizzle relational mappings │ │ ├── schema.ts → barrel re-export │ │ └── client.ts → injectable createSqliteDatabase() │ └── pg/ → PostgreSQL host (NOT YET IMPLEMENTED) -└── test/ → empty — tests not yet written +└── test/ + └── reference-modules.test.ts → Metagraph, bridge, crypto tests ``` ### Subpath Exports (JSR/npm) | Export | Contents | Dependencies | | ------------------------ | --------------------------------------- | --------------------------------------- | -| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox`, `@alkdev/drizzlebox` | +| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox` | | `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` | -| `@alkdev/storage/sqlite` | SQLite tables, relations, client | + `drizzle-orm`, `@libsql/client` | +| `@alkdev/storage/sqlite` | SQLite tables, relations, client | `@alkdev/drizzlebox`, `drizzle-orm`, `@libsql/client` | | `@alkdev/storage/pg` | PostgreSQL tables, relations, client | ⚠️ NOT YET IMPLEMENTED | The `./graphs` subpath exists because the source code lives in `src/graphs/` and @@ -105,22 +114,24 @@ consumed by the hub and spokes, not by storage itself. ### Implemented -- Graph schema types and Metagraph Module (replaces SchemaBuilder) +- Metagraph Module (`Type.Module` with Config, BaseNode, BaseEdge entries) +- Bridge functions (`moduleToDbSchema`, `validateNode`, `validateEdge`) +- Reference graph type Modules (CallGraph, SecretGraph) +- Crypto utility (AES-256-GCM + PBKDF2, `EncryptedDataSchema`) - SQLite host: 6 metagraph tables + actors table + Drizzle relations + client factory - TypeBox select/insert schemas generated from Drizzle tables (drizzlebox) +- Reference module tests (bridge functions, validation, Module composition) ### Not Yet Implemented | Gap | Priority | Notes | | ----------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------- | -| Encrypted data node type + crypto utility | **Critical** | ⚠️ Not yet implemented. API keys and secrets at rest. See [encrypted-data.md](./encrypted-data.md). | | Repository/CRUD layer | High | ⚠️ Not yet implemented. Typed insert, find, update, delete functions for graphs, nodes, edges. No dependency on `@alkdev/operations` — consumer wires CRUD into registry. | -| Tests | High | Zero tests exist. Needed before any real use. | | PostgreSQL host | Medium | Same table shapes, `pgTable` + `jsonb` + `timestamp` + `pgEnum`. Stub only. | -| Call graph type | Medium | Informed by `@alkdev/flowgraph`'s `CallNodeAttrs`/`CallEdgeAttrs` schemas and `@alkdev/operations`' call protocol events. Not hub-specific — any consumer that tracks call invocations needs this. | -| ACL graph type | Medium | Access control as a graph. Informed by `@alkdev/operations`' `Identity` and `AccessControl`. Depends on encrypted data and CRUD layer. | +| ACL graph type | Medium | Access control as a graph. Informed by `@alkdev/operations`' `Identity` and `AccessControl`. Depends on CRUD layer. | | Task graph type | Low | Informed by `@alkdev/taskgraph`'s `TaskGraphNodeAttributes` and `DependencyEdge` schemas. | +| Graphology bridge | Low | `moduleToGraphology()` and `fromGraphologyExport()` — Phase 4 of the metagraph implementation path. | ## Ecosystem Integration @@ -232,5 +243,5 @@ questions affecting this package: - Source heritage: `@ade/ade-v0/packages/core/graphs` and `@ade/ade-v0/packages/storage_sqlite` - Drizzle ORM: https://orm.drizzle.team/ -- TypeBox: https://github.com/sinclairzx/typebox +- TypeBox: `/workspace/@alkdev/typebox/` - JSR: https://jsr.io/ diff --git a/docs/architecture/schema-evolution.md b/docs/architecture/schema-evolution.md index 8ef3c40..109422a 100644 --- a/docs/architecture/schema-evolution.md +++ b/docs/architecture/schema-evolution.md @@ -1,6 +1,6 @@ --- -status: draft -last_updated: 2026-05-28 +status: reviewed +last_updated: 2026-05-30 --- # Schema Evolution @@ -504,4 +504,4 @@ questions affecting schema evolution: - Event Log as Source of Truth (ADR-005): `/workspace/@alkdev/flowgraph/docs/architecture/decisions/005-event-log-as-source-of-truth.md` - Call protocol: `/workspace/@alkdev/operations/docs/architecture/call-protocol.md` - Metagraph Module: [metagraph-module.md](./metagraph-module.md) -- Schema versioning in the data model: [metagraph-module.md](./metagraph-module.md) (Versioning section and DD3) \ No newline at end of file +- Schema versioning in the data model: ADR-029, [metagraph-module.md](./metagraph-module.md) \ No newline at end of file diff --git a/docs/architecture/sqlite-host.md b/docs/architecture/sqlite-host.md index 50745be..b02d476 100644 --- a/docs/architecture/sqlite-host.md +++ b/docs/architecture/sqlite-host.md @@ -1,6 +1,6 @@ --- status: reviewed -last_updated: 2026-05-29 +last_updated: 2026-05-30 --- # SQLite Host @@ -214,8 +214,7 @@ Standalone identity table. Currently not referenced by any relation — the `actors` table has no FK references to or from any metagraph table and is not included in `relations.ts`. This is a placeholder for identity data and may become a node type in an ACL graph (based on `@alkdev/operations`'s `Identity` -interface) or remain a standalone table. See [overview.md](./overview.md) Open -Question 1. +interface) or remain a standalone table. See OQ-03 in [open-questions.md](./open-questions.md). | Column | Type | Constraints | Notes | | --------- | ------------------- | --------------------------------------- | ------------------ |