storage/docs/architecture/forward-look.md

status, last_updated

status	last_updated
draft	2026-05-31

Forward Look: Pointers, dbtype, and Universal IR

How the Module-based metagraph connects to the broader @alkdev ecosystem — typed graph pointers, dbtype table rendering, and the ujsx universal IR pipeline. These are forward-looking designs that justify why certain structural decisions were made now (pointer abstraction deferred per ADR-017, dbtype integration deferred per ADR-018).

Overview

Three packages in the @alkdev ecosystem share the same pipeline shape:

Schema (TypeBox Module)  →  Element Tree (ujsx)  →  Host (HostConfig)

Package	Schema	Element tree	Host
@alkdev/ujsx	UJSX Module	<element>, <root>	DOM, custom
@alkdev/dbtype	Table/Column schemas	<table>, <column>	SQLite, PG, MySQL drizzle dialects
@alkdev/storage	Metagraph Module	⚠️ Future: <graphSchema>, <nodeType>	⚠️ Future: graph DB hosts

When storage's graph type definitions align with the Module pattern, they join this same pipeline. The immediate benefit is recursive/cross-referencing schemas (today). The forward benefit is that graph type definitions, table definitions, and pointer expressions can all be authored as ujsx element trees rendered to different hosts.

Pointer Abstraction

Addressing nodes and edges within a graph instance follows the same pattern as ujsx's ValuePointer and selectNode/setNode — and the same pattern as jsonpathly's JPATH Module for path expressions.

ujsx's pointer system (proven)

ujsx already implements a reactive pointer system:

class ValuePointer<T> {
  private _signal: Signal<T>;
  private _path: string[];
  get value(): T
  set value(v: T)
  get reactive(): ReadonlySignal<T>
  get path(): string[]
}

function selectNode(root: UNode, path: string[]): UNode | undefined
function setNode(root: UNode, path: string[], value: UNode): UNode

This addresses elements within a ujsx tree by path segments (child indices, prop names). A graph instance has analogous structure: nodes identified by key, edges identified by key, attributes addressed by JSON path.

Graph pointer analogy

// ujsx pointer: element tree → path → value
selectNode(root, ["children", 0, "props", "name"])

// Graph pointer: graph instance → path → value
selectNode(graph, ["nodes", "call-001", "attributes", "requestId"])

The structural analogy:

ujsx concept	Graph concept
Element tree root	Graph instance
UNode	Node or Edge
path: string[]	Key path: ["nodes", key] or ["edges", key]
selectNode(root, path)	selectGraphNode(graph, path)
setNode(root, path, value)	setGraphNode(graph, path, value) (via repository)

JPATH Module (jsonpathly)

The research shows that JSONPath expressions can themselves be a TypeBox Module (JPATH = Type.Module({...}) with recursive Type.Ref("Subscript")). This means pointer paths are not just runtime strings — they're typed schemas that can be validated and composed.

For graph storage, this opens the possibility of typed graph queries — a pointer expression like nodes.call-001.attributes.requestId has a schema that validates against the graph type's Module. If CallNode doesn't have a requestId field, the pointer expression is invalid at compile time.

Scope for v1

The pointer abstraction is a forward-looking design. For v1:

• Repository functions use direct key-based addressing: findNode(graphId, nodeKey), findEdge(graphId, edgeKey)
• Attribute access is untyped JSON retrieval: node.attributes.requestId
• The Module validates attribute shapes, but query paths are strings

The jump to typed pointers requires either the JPATH Module (for path validation) or ujsx-style ValuePointer with signals (for reactive graph observation). Both are post-v1 concerns, but the graph type Module makes them feasible because it provides the schema the pointer validates against.

Relationship to @alkdev/dbtype

@alkdev/dbtype defines database schemas as ujsx element trees and renders them to Drizzle dialects via HostConfig. Storage's SQLite/PG table definitions are a natural consumer of this pipeline.

Current vs. Future Table Definition

Current (manual Drizzle table defs):

export const graphTypes = sqliteTable("graph_types", {
  id: text("id").primaryKey(),
  name: text("name").notNull(),
  config: text("config", { mode: "json" }).notNull(),
  // ...
});

Future (dbtype element tree → HostConfig rendering):

const GraphTypesEl = h("table", { name: "graph_types" },
  h(IdColumn, {}),
  h("column", { name: "name", type: "string", notNull: true }),
  h("column", { name: "config", type: "json", mode: "json", notNull: true }),
  h(AuditColumns, {}),
);

const root = createRoot(sqliteHost, {});
root.render(GraphTypesEl);
const drizzleTable = root.ctx.tables.graph_types;

Why this matters for storage

1. Single source of truth: Today's sqlite/tables/ and future pg/tables/ define the same shapes in two different Drizzle dialects. dbtype renders the same element tree to both — no manual duplication.
2. Schema extraction: extractTable() produces both TypeBox schemas (for validation) and column metadata (for Drizzle rendering) from the same tree. Storage gets SelectGraphType and InsertGraphType schemas for free.
3. Module alignment: dbtype assembles extracted schemas into a Type.Module for cross-table references. Storage's metagraph Module and dbtype's table Module could share a namespace — the graph_types.config column stores the JSON Schema from Metagraph.Config.

v1 approach

For v1, storage continues with manual Drizzle table definitions. The dbtype integration is deferred because:

• dbtype is Phase 0 (architecture complete, no implementation)
• The manual defs work and are well-understood
• The Module pattern for graph types can be adopted independently (no dbtype dependency)
• With PostgreSQL removed (ADR-038), the original pressure for dbtype — eliminating dual SQLite/PG table maintenance — is significantly reduced. There is now only one set of table definitions to maintain.

When dbtype reaches Phase 1 (implementation), storage can migrate from Drizzle table definitions to dbtype elements one table at a time. The Module-based graph type definitions are already compatible — they're both TypeBox Type.Module objects.

ujsx as Universal IR

The three packages (ujsx, dbtype, storage) share the same pipeline shape: Schema → Element Tree → Host. This is not coincidental — ujsx is a universal declarative IR, and different "render targets" are just different HostConfigs.

What this could look like

// Graph type definitions as ujsx elements (future)
const CallGraphSchema = h("graphSchema", { name: "call-graph" },
  h("config", { type: "directed", multi: false, allowSelfLoops: false }),
  h("nodeType", { name: "call" },
    h(BaseNode, {}),
    h("attr", { name: "requestId", type: "string", required: true }),
    h("attr", { name: "status", ref: "CallStatus" }),
  ),
  h("edgeType", { name: "triggered" },
    h(BaseEdge, {}),
    h("attr", { name: "type", literal: "triggered" }),
  ),
  h("edgeConstraints", { edgeType: "triggered",
    allowedSourceTypes: ["Call"],
    allowedTargetTypes: ["Call", "Subcall"] }),
);

Rendered to different hosts:

Host	Output
TypeBox Host	Type.Module({ CallNode: ..., TriggeredEdge: ... })
SQLite Host	sqliteTable("node_types", { ... }) + sqliteTable("edge_types", { ... })
PG Host	pgTable("node_types", { ... }) + pgTable("edge_types", { ... })
graphology Host	SerializedGraph format
Documentation Host	Mermaid diagram, typed API docs

What's real today vs. aspirational

Capability	Status
Type.Module for graph type definitions	✅ Implemented — Metagraph, CallGraph, SecretGraph Modules
Bridge functions (moduleToDbSchema, validateNode, validateEdge)	✅ Implemented
Reference graph type Modules (CallGraph, SecretGraph)	✅ Implemented
Crypto utility (encrypt, decrypt, generateEncryptionKey, EncryptedDataSchema)	✅ Implemented
Codegen from TypeScript interfaces → Module entries	✅ TsToModule exists
dbtype element trees → Drizzle tables	⚠️ dbtype Phase 0, no implementation
<graphSchema> ujsx elements	⚠️ Conceptual — needs HostConfig design
Typed graph pointers via JPATH	⚠️ Conceptual — needs JPATH Module design
Reactive graph observation via ValuePointer	⚠️ Conceptual — needs signal integration

The Module-based graph type definitions (this spec) are the first concrete step in this pipeline. Everything else builds on having a Type.Module as the schema source of truth.

Repository Layer Strategy

The repository layer (typed CRUD for the 6 metagraph tables + queries for graph data) is the next major feature to implement. The question of how it queries attributes connects to broader ecosystem decisions about dbtype and operations.

Three Approaches

A. JSON Path Queries (Near-Term)

The repository layer maps filter criteria to JSON path extraction:

findNodes({ graphId, attributes: { status: "active" } })
// SQLite: json_extract(attributes, '$.status') = 'active'
// PG:     attributes ->> 'status' = 'active'

• Works with current table definitions (no schema changes)
• SQLite json_extract() and PG ->> / #>> operators handle JSON path
• No native index support on individual JSON attributes
• PG can add GIN indexes on jsonb columns for containment queries, but not for arbitrary key-value lookups
• Simple, immediate, no new infrastructure

This is the pragmatic v1 approach. The metagraph pattern requires JSON attributes because node types are dynamic schemas (defined at runtime, stored in node_types.schema), not static columns known at database definition time.

B. Native Columns via dbtype (Long-Term, Speculative)

If storage migrates to dbtype element trees for table definitions, the 6 static metagraph tables (graph_types, node_types, edge_types, graphs, nodes, edges) could be rendered via the dbtype pipeline: element tree → HostConfig → Drizzle tables. This would eliminate the manual duplication between sqlite/ and future pg/.

However, dbtype does NOT solve the attribute indexing problem:

• The metagraph's attributes column MUST remain JSON because the shape is defined by runtime schemas (node type definitions), not by static column definitions
• dbtype generates static table schemas; it does not handle dynamic schema-as-data patterns like the metagraph
• A "call" node's attributes (requestId, status, duration) are not columns on the nodes table — they're values in the attributes JSON column, validated by the corresponding node type's TypeBox schema

C. Hybrid: Static Tables via dbtype, Dynamic Attributes Remain JSON

The hybrid approach preserves the metagraph's dynamic schema model while leveraging dbtype for the static table scaffolding:

1. Static tables: dbtype renders the 6 metagraph tables to Drizzle dialects. This eliminates the SQLite/PG manual duplication for table structure. The attributes column is still text/jsonb across both dialects.

2. Dynamic attributes: Remain JSON. The Module-based node type schemas validate data at the application layer, not the database layer. This is by design (ADR-003, ADR-014).

3. Virtual columns / computed columns: A post-v1 optimization, not a v1 concern. Frequently queried attributes could be extracted to indexed columns as a performance optimization. For example, if nodes.attributes.status is a common filter, a computed column or trigger could copy it to nodes.status_column with an index. This would be a denormalization trade-off (triggers, migration complexity, dual-write responsibility) and is not designed or planned for v1.

4. Repository CRUD: The static table CRUD operations (insert graph type, find node by key) could be auto-generated like drizzle-graphql or the dbtype from-dbtype adapter. Graph-specific attribute queries remain JSON path.

Implications for Each Approach

Concern	Path A (JSON)	Path B (Native)	Path C (Hybrid)
Works today	✅	❌ (requires dbtype)	❌ (requires dbtype)
Preserves metagraph pattern	✅	❌ (conflicts with dynamic schemas)	✅
Eliminates SQLite/PG duplication	❌	✅	✅
Indexes on attributes	GIN on PG only	✅ full native	GIN + virtual columns
Repository generation	Hand-write CRUD	Auto-gen from dbtype	Auto-gen for static, JSON path for dynamic
Dependency on dbtype	None	Full	Partial (static tables only)

Connection to drizzle-graphql

The overview references drizzle-graphql as a pattern for auto-generating a CRUD/query surface. The dbtype from-dbtype adapter is the @alkdev equivalent: it consumes element trees + Type.Module bundles and produces OperationSpec[] for the operations registry.

The parallel:

Concern	drizzle-graphql	dbtype from-dbtype
Input	Drizzle schema (tables + relations)	UJSX element tree + Type.Module
Output	GraphQL schema (queries + mutations)	OperationSpec[] (CRUD operations)
Dialects	SQLite, PG, MySQL	SQLite, PG, MySQL (via HostConfig)
Table model	Static columns only	Static columns only
Dynamic data (JSON attrs)	Not handled	Not handled

Neither drizzle-graphql nor dbtype's from-dbtype handles dynamic schema-as-data patterns. The metagraph's JSON attributes require their own query layer, regardless of whether the static tables are auto-generated. This means the repository layer for @alkdev/storage will always have two parts:

1. Static table CRUD — could be auto-generated (by dbtype or hand-written)
2. Graph data queries — JSON path queries against the attributes column, validated by the Module schema at the application layer

v1 Decision

For v1, the practical path is A (JSON path queries) with hand-written CRUD. This decision is recorded as ADR-033. The hybrid approach (C) remains viable for a future iteration when dbtype reaches implementation, and it doesn't require any changes to the metagraph data model — only to how the static table definitions are generated. See OQ-17, OQ-18, OQ-19 in open-questions.md for the specific long-term questions that remain open beyond v1.

Decisions Required

• OQ-17: JSON path vs native columns vs hybrid for attribute queries (resolved for v1 — see ADR-033)
• OQ-18: Auto-generated vs hand-written CRUD for static tables (resolved for v1 — see ADR-033)
• OQ-19: Where the storage-operations bridge package should live (open)

Constraints on Current Design

The forward-looking patterns documented here constrain the Module evolution design in metagraph-module.md:

1. The Module format must be self-contained — Type.Module({...}) entries with Type.Ref and Type.Composite are the same structures that a ujsx TypeBox Host would produce. If the Module format were an ad-hoc builder output, it couldn't be rendered by a different host later.

2. Edge constraints must be schema entries, not just DB columns — the constraint data needs to survive serialization/deserialization and be validatable independently. DB-only columns can't do this.

3. The base attribute schemas (BaseNode, BaseEdge) must be TypeBox schemas — not Drizzle column definitions, not builder-internal objects. Only TypeBox schemas can be composed via Type.Composite, referenced via Type.Ref, and serialized to JSON Schema.

4. No ujsx dependency — storage's Module-based graph types join the pipeline conceptually, not as a runtime dependency. The Type.Module output is the same shape that a ujsx HostConfig would produce, but storage doesn't need ujsx to create it. The alignment is structural, not dependent.

5. Schemas-as-JSON enables Value.Diff/Value.Patch/Value.Cast — because TypeBox Modules serialize to JSON Schema, the TypeBox value system can operate on schemas themselves (diff to detect changes, patch to update stored schemas, cast to migrate data). This is not possible if schemas are opaque builder objects or Drizzle column definitions. See schema-evolution.md.

References

• ujsx pointer system: /workspace/@alkdev/ujsx/src/core/pointer.ts
• ujsx HostConfig adapter: /workspace/@alkdev/ujsx/src/host/config.ts
• dbtype architecture: /workspace/@alkdev/dbtype/docs/architecture/README.md
• dbtype elements: /workspace/@alkdev/dbtype/docs/architecture/elements.md
• dbtype module: /workspace/@alkdev/dbtype/docs/architecture/module.md
• dbtype repo adapter: /workspace/@alkdev/dbtype/docs/architecture/repo-adapter.md
• drizzle-graphql (reference for CRUD generation pattern): /workspace/drizzle-graphql/
• Operations registry: /workspace/@alkdev/operations/docs/architecture/README.md
• JPATH Module (JSONPath as TypeBox Module): /workspace/research/typebox_research/ujsx/jpath.gen.ts
• jsonpathly source: /workspace/jsonpathly/
• Module evolution spec: metagraph-module.md
• Schema evolution spec: schema-evolution.md
• ADR-033: JSON path queries and hand-written CRUD for v1