diff --git a/docs/architecture/encrypted-data.md b/docs/architecture/encrypted-data.md index ad7a6d6..b74edb9 100644 --- a/docs/architecture/encrypted-data.md +++ b/docs/architecture/encrypted-data.md @@ -49,48 +49,50 @@ Instead of a dedicated `client_secrets` table, encrypted data becomes a **node type** in a graph: ```ts -import { BaseNodeAttributes, SchemaBuilder } from "@alkdev/storage"; +import { Metagraph } from "@alkdev/storage"; import { Type } from "@alkdev/typebox"; import { EncryptedDataSchema } from "@alkdev/storage"; -const SecretNodeType = Type.Intersect([ - BaseNodeAttributes, - Type.Object({ - key: Type.String({ minLength: 1, maxLength: 255 }), - encryptedData: EncryptedDataSchema, - expiresAt: Type.Optional(Type.String({ format: "date-time" })), +const SecretGraph = Type.Module({ + Config: Type.Object({ + type: Type.Literal("undirected"), + multi: Type.Literal(false), + allowSelfLoops: Type.Literal(false), }), -]); -const schema = new SchemaBuilder() - .config({ type: "undirected", multi: false, allowSelfLoops: false }) - .nodeType("secret", SecretNodeType) - .nodeType( - "client", - Type.Intersect([ - BaseNodeAttributes, - Type.Object({ - name: Type.String(), - type: Type.String(), - config: Type.Record(Type.String(), Type.Any()), - enabled: Type.Boolean({ default: true }), - }), - ]), - ) - .edgeType( - "has_secret", - Type.Intersect([ - BaseEdgeAttributes, - Type.Object({ - secretKey: Type.String(), - }), - ]), - { - allowedSourceTypes: ["client"], - allowedTargetTypes: ["secret"], - }, - ) - .build(); + SecretNode: Type.Composite([ + Metagraph.Import("BaseNode"), + Type.Object({ + key: Type.String({ minLength: 1, maxLength: 255 }), + encryptedData: EncryptedDataSchema, + expiresAt: Type.Optional(Type.String({ format: "date-time" })), + }), + ]), + + ClientNode: Type.Composite([ + Metagraph.Import("BaseNode"), + Type.Object({ + name: Type.String(), + type: Type.String(), + config: Type.Record(Type.String(), Type.Unknown()), + enabled: Type.Boolean({ default: true }), + }), + ]), + + HasSecretEdge: Type.Composite([ + Metagraph.Import("BaseEdge"), + Type.Object({ + type: Type.Literal("has_secret"), + secretKey: Type.String(), + }), + ]), + + HasSecretEdgeConstraints: Type.Object({ + edgeType: Type.Literal("has_secret"), + allowedSourceTypes: Type.Array(Type.String()), // ["Client"] + allowedTargetTypes: Type.Array(Type.String()), // ["Secret"] + }), +}); ``` This represents the same relationship as `client_secrets.clientId` — but as a @@ -336,10 +338,9 @@ db deps): ``` src/graphs/ -├── types.ts # existing: GraphConfig, NodeType, EdgeType, etc. -├── schemaBuilder.ts # existing: SchemaBuilder -├── crypto.ts # new: encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema -└── mod.ts # re-exports all of the above +├── modules/metagraph.ts # Metagraph Module (BaseNode, BaseEdge, Config) +├── crypto.ts # new: encrypt(), decrypt(), generateEncryptionKey(), EncryptedDataSchema +└── mod.ts # re-exports all of the above ``` This keeps the encryption utility in the zero-dep export path (it only uses Web diff --git a/docs/architecture/forward-look.md b/docs/architecture/forward-look.md index 64003c8..bea2621 100644 --- a/docs/architecture/forward-look.md +++ b/docs/architecture/forward-look.md @@ -156,14 +156,14 @@ const drizzleTable = root.ctx.tables.graph_types; ### v1 approach For v1, storage continues with manual Drizzle table definitions. The dbtype -integration is a post-v1 migration path because: +integration is deferred because: - dbtype is Phase 0 (architecture complete, no implementation) - The manual defs work and are well-understood - The Module pattern for graph types can be adopted independently (no dbtype dependency) -When dbtype reaches Phase 1 (implementation), storage can migrate table defs +When dbtype reaches Phase 1 (implementation), storage can adopt dbtype element to dbtype elements one table at a time. The Module-based graph type definitions are already compatible — they're both TypeBox `Type.Module` objects. diff --git a/docs/architecture/metagraph-module.md b/docs/architecture/metagraph-module.md index 547dfda..76a73a7 100644 --- a/docs/architecture/metagraph-module.md +++ b/docs/architecture/metagraph-module.md @@ -17,7 +17,7 @@ compose with `Type.Composite()`, and can cross-reference other Modules with a Module with `UPrimitive`, `UElement`, `URoot`, `UNode` recursively referencing each other). -The current `SchemaBuilder` produces a flat `GraphSchema` object — an ad-hoc +The removed `SchemaBuilder` produced a flat `GraphSchema` object — an ad-hoc `Record` + `Record`. This works but creates friction: @@ -28,7 +28,7 @@ creates friction: not a format that maps to graphology's `import()`/`export()`. Consumers manually map node/edge attributes. 3. **No codegen leverage** — `TsToModule` generates TypeBox Modules from - TypeScript interfaces. The SchemaBuilder can't consume Module output, so + TypeScript interfaces. The SchemaBuilder couldn't consume Module output, so codegen-produced types must be manually translated. The Module approach treats each graph type as a `Type.Module`, aligning storage @@ -305,10 +305,10 @@ construction-time validation: "any valid config"). Specific graph types use // Specific (frozen): Type.Literal("directed") ``` -The transition: consumer provides a general config → validated against +The construction flow: consumer provides a general config → validated against `Metagraph.Config` → the specific graph type Module uses `Type.Literal` to -freeze the value. The `SchemaBuilder` (during transition) performs this -narrowing automatically. +freeze the value. Narrowing from `Type.Union` to `Type.Literal` is explicit +in the Module — no builder step needed. ### Edge Type Constraints: named constraint entries @@ -364,7 +364,7 @@ correspond to `*Node` entry names in the same Module. This is enforced by `moduleToDbSchema()` at Module-to-DB projection time, not by the schema itself. See Open Question 4 for the `Type.Ref` vs `Type.String` trade-off. -**Transition note**: The current DB schema stores `allowedSourceTypes` and +**DB mapping note**: The current DB schema stores `allowedSourceTypes` and `allowedTargetTypes` as JSON text columns (arrays of strings, default `[]`). In the Module, these become `Type.Array(Type.String())` entries — the DB column values are the same string arrays. `moduleToDbSchema()` extracts them @@ -532,8 +532,8 @@ Does NOT throw on invalid data — returns `false`. ### Type.Any vs Type.Unknown -The existing `types.ts` uses `Type.Any()` for `metadata` and `schema` fields. -The Module examples use `Type.Unknown()`. These have different JSON Schema +The pre-Module `types.ts` used `Type.Any()` for `metadata` and `schema` fields. +The Module approach uses `Type.Unknown()`. These have different JSON Schema outputs: - `Type.Any()` → `{}` (accepts anything, no validation) @@ -542,8 +542,7 @@ outputs: For the Module approach, **`Type.Unknown()` is canonical**. It's the more explicit choice — it communicates "this field stores arbitrary data, no validation applied." `Type.Any()` is a legacy from the original TypeBox API. -The existing `types.ts` schemas should be aligned to `Type.Unknown()` during -the Module migration (Phase 1). +The `Metagraph` Module uses `Type.Unknown()` throughout. ### Performance Expectations @@ -579,13 +578,15 @@ the codegen can produce a Module entry from it. Storage's `CallGraph` Module the composes `BaseNode` with `CallNodeAttrs` via `Type.Composite`, or imports from the flowgraph Module if flowgraph exports one (see Open Question 1). -## SchemaBuilder.build() → Module Equivalence +## SchemaBuilder Equivalence -The current `SchemaBuilder.build()` returns a `GraphSchema` — a flat object with -`config`, `nodeTypes: Record`, and `edgeTypes: Record`. -A `Type.Module` with the same entries is essentially the same thing. +The removed `SchemaBuilder.build()` used to return a `GraphSchema` — a flat +object with `config`, `nodeTypes: Record`, and `edgeTypes: +Record`. A `Type.Module` with the same entries is +structurally equivalent. This section documents what the builder was doing +internally to show the correspondence. -### What the builder does internally +### What the builder was doing internally ``` SchemaBuilder @@ -605,13 +606,12 @@ defs = { return Type.Module(defs) ``` -The `.build()` return type changes from `GraphSchema` (flat object) to -`TModule` (TypeBox Module). The `SchemaBuilder` is removed — consumers use -Module construction directly. +The `.build()` return type was `TModule` (TypeBox Module). The `SchemaBuilder` is +removed — consumers use Module construction directly. -### Why this works +### Why this is equivalent -The `SchemaBuilder` was always building a module — it just didn't have a +The `SchemaBuilder` was building a module under the hood — it just didn't have a module system to target. Named entries referencing each other via strings is exactly what `Type.Ref()` does natively. The Module format: @@ -725,14 +725,14 @@ complete, no implementation). For v1, storage uses manual Drizzle table definitions. The Module-based graph type definitions are compatible with dbtype because both produce `Type.Module` objects — the integration path is clear. -**Alternative considered**: Implement dbtype integration in v1 alongside Module -migration. Rejected because it adds a dependency on an unimplemented package +**Alternative considered**: Implement dbtype integration alongside the initial Module +construction. Rejected because it adds a dependency on an unimplemented package and the manual table definitions work well. The cost of deferring is continued dual SQLite/PG maintenance, which is manageable for 6 metagraph tables. ## What Changes -| Current | New | +| Before (unreleased) | After | |---------|-----| | `types.ts` — standalone schemas | `modules/metagraph.ts` — `Metagraph` Module | | `schemaBuilder.ts` — fluent builder | Removed — replaced by Module construction | @@ -753,7 +753,7 @@ dual SQLite/PG maintenance, which is manageable for 6 metagraph tables. (Module entries are the source of truth, projected to DB columns by `moduleToDbSchema()`) -## Migration Path +## Implementation Path 1. **Phase 1**: Add `Metagraph` Module, replace `types.ts` and remove `schemaBuilder.ts`. Export Module construction API. @@ -768,7 +768,7 @@ dual SQLite/PG maintenance, which is manageable for 6 metagraph tables. Acceptance criteria per phase: - **Phase 2 complete**: `moduleToDbSchema()` produces values compatible with all - 6 existing metagraph tables + 6 metagraph tables - **Phase 3 complete**: Reference Modules validate against their flowgraph/taskgraph counterparts @@ -791,7 +791,7 @@ Acceptance criteria per phase: `CallNodeAttrs` as a standalone `Type.Object`. To use `Import()`, flowgraph needs to export a Module. But storage can start with standalone schemas and `Type.Composite([BaseNode, CallNodeAttrs])` — no dependency on flowgraph. - Migrate to `Import()` when flowgraph provides a Module. **This avoids a + Adopt `Import()` when flowgraph provides a Module. **This avoids a circular dependency: `@alkdev/storage` does NOT depend on `@alkdev/flowgraph`.** 2. **Should concrete graph type Modules live in storage or in their respective @@ -804,12 +804,7 @@ Acceptance criteria per phase: `modules/` that consumers can use directly or replace. Flowgraph may also export a Module — the two are compatible via Module `$defs`. -3. ~~**How does `SchemaBuilder.build()` return a Module while maintaining backward - compat?**~~ **Resolved**: No backward compat needed (no existing consumers). - `SchemaBuilder` is removed. Consumers use `Type.Module()` construction - directly. See "SchemaBuilder.build() → Module Equivalence". - -4. **Should `*EdgeConstraints` entries use `Type.Ref("CallNode")` or +3. **Should `*EdgeConstraints` entries use `Type.Ref("CallNode")` or `Type.String()` for allowed source/target types?** Using `Type.Ref` would mean "each element in the array must validate against the CallNode schema," which is semantically wrong — the constraint is about which named @@ -820,7 +815,7 @@ Acceptance criteria per phase: contract that string values should correspond to `*Node` entry names, enforced by `moduleToDbSchema()` at projection time. -5. **How does the graph pointer abstraction interact with the repository layer?** +4. **How does the graph pointer abstraction interact with the repository layer?** For v1, repository functions use direct key-based addressing. Typed pointers (JPATH Module, reactive ValuePointer) could layer on top of the repository later. The key question: does the repository return raw data (untyped JSON), diff --git a/docs/architecture/metagraph.md b/docs/architecture/metagraph.md index 7c41230..6b89418 100644 --- a/docs/architecture/metagraph.md +++ b/docs/architecture/metagraph.md @@ -5,10 +5,12 @@ last_updated: 2026-05-28 # Metagraph Model -> **Superseded by [metagraph-module.md](./metagraph-module.md)** — graph type -> definitions are now TypeBox Modules, not standalone schemas + SchemaBuilder. -> This document describes the current (pre-Module) data model. The Module -> migration is specified in metagraph-module.md. +> **Historical reference only.** Graph type definitions are now TypeBox Modules, +> not standalone schemas + SchemaBuilder. The current data model and construction +> patterns are specified in [metagraph-module.md](./metagraph-module.md). This +> document is retained for understanding the data model concepts (graph types, +> node types, edge types, graph instances) and the versioning/attributes storage +> design, which carry forward unchanged into the Module approach. The core data model: graph types define schemas, node types define shapes, edge types define relationships, and typed graph instances hold actual data. @@ -55,7 +57,13 @@ Graph "session-abc-call-graph" (instance) ## Schema Types -Defined in `src/graphs/types.ts`. Zero database dependencies — these are pure +> These are the pre-Module representations. The `Metagraph` Module +> ([metagraph-module.md](./metagraph-module.md)) replaces these standalone +> schemas with Module entries (`Metagraph.BaseNode`, `Metagraph.BaseEdge`, +> `Metagraph.Config`). The data shapes are the same; the Module format adds +> `Type.Ref()`, `Type.Composite()`, and `$defs` support. + +Were defined in `src/graphs/types.ts`. Zero database dependencies — pure TypeBox schemas used for validation and type inference. ### BaseNodeAttributes @@ -108,7 +116,8 @@ features. A node type definition. The `schema` validates the `attributes` of nodes that belong to this type. Consumer must extend `BaseNodeAttributes` in their schema — the metagraph model does not enforce this at the database level (SQLite can't -enforce JSON schema), but the SchemaBuilder validates it at definition time. +enforce JSON schema), but the SchemaBuilder validated it at definition time +(now handled by `Type.Module()` construction). ### EdgeType @@ -137,7 +146,7 @@ valid endpoints. ``` The complete definition of a graph type. This is what `SchemaBuilder.build()` -produces. +produced (now `Type.Module()` produces the same structure). ### GraphStatus & GraphBaseType @@ -150,9 +159,12 @@ These are provided as `as const` object constants and TypeBox `Type.Union` of `Type.Literal` schemas, following the project convention (see overview.md D6). The TypeBox schemas derive their literal values from the same const object. -## SchemaBuilder +## SchemaBuilder (Historical — Replaced by Module Construction) -Defined in `src/graphs/schemaBuilder.ts`. Fluent builder API: +Was defined in `src/graphs/schemaBuilder.ts`. Fluent builder API. **This builder +is removed in the Module approach** — see [metagraph-module.md](./metagraph-module.md) +DD1/DD2. The builder's internal structure documents the structural equivalence +with `Type.Module()` (see "SchemaBuilder Equivalence" in that document). ```ts const schema = new SchemaBuilder() @@ -169,23 +181,23 @@ const schema = new SchemaBuilder() ### Validation -The builder validates at each step: +The builder validated at each step: -1. **`config()`** — Validates against `GraphConfig` schema. Applies defaults for +1. **`config()`** — Validated against `GraphConfig` schema. Applied defaults for missing fields. -2. **`nodeType()`** — Validates the schema is a valid TypeBox schema - (`KindGuard.IsSchema`). Validates the resulting object against `NodeType` +2. **`nodeType()`** — Validated the schema was a valid TypeBox schema + (`KindGuard.IsSchema`). Validated the resulting object against `NodeType` schema. -3. **`edgeType()`** — Same as nodeType, plus validates - allowedSourceTypes/allowedTargetTypes are strings. -4. **`build()`** — Validates the complete schema against `GraphSchema`. Throws +3. **`edgeType()`** — Same as nodeType, plus validated + allowedSourceTypes/allowedTargetTypes were strings. +4. **`build()`** — Validated the complete schema against `GraphSchema`. Threw on any invalid structure. -**Error behavior**: The builder throws `Error` with a JSON-stringified list of -validation errors (path + message). Validation failures do not roll back partial -state — a builder that fails on the second `nodeType()` call still has the first -node type in its schema. Callers should not reuse a builder after a failure. -Create a new `SchemaBuilder` instead. +**Error behavior**: The builder threw `Error` with a JSON-stringified list of +validation errors (path + message). Validation failures did not roll back partial +state — a builder that failed on the second `nodeType()` call still had the first +node type in its schema. Callers were advised not to reuse a builder after a +failure. **Edge type enforcement**: When `allowedSourceTypes` or `allowedTargetTypes` is `undefined` in the schema layer, any node type is a valid endpoint. When a @@ -196,7 +208,9 @@ There is no "no types allowed" state; if edge types need to be disabled, use a status or soft-delete pattern on the edge type definition. The repository layer must enforce this convention consistently. -The SchemaBuilder enforces structural integrity at definition time. The database +The SchemaBuilder enforced structural integrity at definition time. In the +Module approach, `Type.Module()` construction and `Value.Check()` provide the +same guarantee. The database stores graph/node/edge type schemas as JSON blobs (`text` mode in SQLite, will be `jsonb` in PG). Database-level constraints (unique composite keys, cascade deletes) protect referential integrity, but the database does NOT validate JSON @@ -233,7 +247,7 @@ JSON is provided. This design means: - **Schema evolution**: Add optional fields to a node type schema without - migration. Old nodes are still valid. + data migration. Old nodes are still valid. - **Schema versioning**: The `version` field on graph types tracks breaking schema changes. Consumer code can check the version before processing. - **Validation boundary**: All validation happens in the repository layer @@ -242,7 +256,7 @@ This design means: ## Versioning Graph types have a `version` integer (default 1). This tracks **breaking** -schema changes — field removals, type changes that break backward compatibility. +schema changes — field removals, type changes that invalidate existing data. Non-breaking changes (adding optional fields) do not require a version bump. The `version` field is stored as a column on the `graph_types` table (see @@ -254,7 +268,12 @@ The repository layer should check `version` before processing to ensure compatibility. A version mismatch indicates the data format has changed incompatibly and the consumer should handle it explicitly. -## Usage Patterns +## Usage Patterns (Historical — SchemaBuilder API) + +> **⚠️ These examples use the removed SchemaBuilder API.** They are retained here +> as structural reference for the data model concepts. For the current Module +> construction API, see [metagraph-module.md](./metagraph-module.md). For current +> encrypted data examples, see [encrypted-data.md](./encrypted-data.md). ### Defining a Call Graph Type @@ -399,5 +418,5 @@ the call protocol and registry, and the consumer bridges them. - Task graph schema: `/workspace/@alkdev/taskgraph_ts/docs/architecture/schemas.md` - Pubsub architecture: `/workspace/@alkdev/pubsub/docs/architecture/README.md` - TypeBox: https://github.com/sinclairzx/typebox -- SchemaBuilder source: `src/graphs/schemaBuilder.ts` -- Schema types source: `src/graphs/types.ts` +- SchemaBuilder source (pre-Module, removed): `src/graphs/schemaBuilder.ts` +- Schema types source (pre-Module, being replaced): `src/graphs/types.ts` diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 7beac51..aaa8899 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -43,14 +43,14 @@ ecosystem. | Export | Contents | Dependencies | | ------------------------ | --------------------------------------- | --------------------------------------- | -| `@alkdev/storage` | Graph schema types, SchemaBuilder | `@alkdev/typebox`, `@alkdev/drizzlebox` | +| `@alkdev/storage` | Graph schema types, Metagraph Module | `@alkdev/typebox`, `@alkdev/drizzlebox` | | `@alkdev/storage/graphs` | Same as `.` — alias for the main export | Same as `.` | | `@alkdev/storage/sqlite` | SQLite tables, relations, client | + `drizzle-orm`, `@libsql/client` | | `@alkdev/storage/pg` | PostgreSQL tables, relations, client | ⚠️ NOT YET IMPLEMENTED | The `./graphs` subpath exists because the source code lives in `src/graphs/` and the main `mod.ts` re-exports it. Importing from either `@alkdev/storage` or -`@alkdev/storage/graphs` yields the same types and SchemaBuilder. +`@alkdev/storage/graphs` yields the same types and Metagraph Module. ## Terminology @@ -65,7 +65,7 @@ the main `mod.ts` re-exports it. Importing from either `@alkdev/storage` or | **Graph instance** | A concrete graph belonging to a graph type. Contains nodes and edges conforming to its type definitions. Stored in the `graphs` table. | | **Consumer** | Code that imports `@alkdev/storage` (or a subpath) to define graph types and persist graph data. The hub, spokes, and other @alkdev packages are consumers. | | **Repository layer** | ⚠️ Not yet implemented. The typed CRUD functions (insert, find, update, delete) that sit between consumer code and raw Drizzle queries. Performs schema validation before writes. No dependency on `@alkdev/operations` — the consumer wires CRUD into the registry. | -| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the SchemaBuilder (at type definition time) and the repository layer (at mutation time), NOT in the database. | +| **Validation boundary** | The line where schema validation is enforced. In this package, validation happens in the Metagraph Module (at type definition time) and the repository layer (at mutation time), NOT in the database. | ## Design Decisions @@ -281,11 +281,11 @@ storage node attributes and operations call events), they should either: application provides the key ring. This keeps the storage package agnostic to deployment-specific secret management. -5. **Migration strategy**: When graph type schemas evolve (new node types, - changed attribute schemas), who handles migration? The repository layer - should support schema version checking, but actual migration scripts are - application-level. See [metagraph.md](./metagraph.md) for the versioning - approach. +5. **Schema evolution strategy**: When graph type schemas evolve (new node types, + changed attribute schemas), who handles migration? The repository layer + should support schema version checking, but actual data migration scripts are + application-level. See [metagraph.md](./metagraph.md) for the versioning + approach. 6. **~~Should the repository layer live in `@alkdev/storage` or in a consumer package?~~** Decision: the repository CRUD layer (host-specific typed