diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 012e113..77464ad 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # @alkdev/flowgraph Architecture @@ -54,12 +54,14 @@ Flowgraph is in Phase 0/1 (exploration → architecture). No code exists yet. Th | [schema.md](schema.md) | TypeBox Module, TypeScript types, enums (CallStatus, EdgeType, NodeStatus), node/edge attribute schemas, SerializedGraph factory | | [operation-graph.md](operation-graph.md) | Static graph from OperationSpecs, type-compatibility edges, construction paths, validation | | [call-graph.md](call-graph.md) | Dynamic graph from call events, node lifecycle, abort cascading, fromCallEvents construction | -| [workflow-templates.md](workflow-templates.md) | ujsx components (``, ``, ``, ``), template→DAG hydration, serialization | -| [host-configs.md](host-configs.md) | Graphology HostConfig (template→DAG analysis), Reactive HostConfig (template→execution engine), Instance types | -| [reactive-execution.md](reactive-execution.md) | Signal-driven status propagation, computed preconditions, abort cascade via signals, ReactiveRoot integration | -| [analysis.md](analysis.md) | Type-compatibility checking (input/output schema matching), precondition validation, execution ordering | +| [workflow-templates.md](workflow-templates.md) | ujsx components (``, ``, ``, ``, ``), composition rules, template→DAG hydration, serialization | +| [host-configs.md](host-configs.md) | Graphology HostConfig (template→DAG analysis), Reactive HostConfig (template→execution engine), Instance types, removeChild | +| [reactive-execution.md](reactive-execution.md) | Signal-driven status propagation, computed preconditions, abort cascade via signals, ReactiveRoot integration, lifecycle and ownership, error boundaries | +| [analysis.md](analysis.md) | Type-compatibility checking (input/output schema matching), compatibility depth, precondition validation, execution ordering, performance characteristics | | [error-handling.md](error-handling.md) | FlowgraphError hierarchy, CycleError, TypeIncompatError, ValidationError, error collection strategy | | [build-distribution.md](build-distribution.md) | Package structure, exports map, dependencies, platform targets | +| [flowgraph-api.md](flowgraph-api.md) | FlowGraph class public API: constructor, type parameters, methods, delegation model, immutability guarantees | +| [consumer-integration.md](consumer-integration.md) | End-to-end walkthrough from operation specs to running workflow, common patterns, module dependency map | ### Design Decisions @@ -68,6 +70,7 @@ Flowgraph is in Phase 0/1 (exploration → architecture). No code exists yet. Th | [001](decisions/001-ujsx-as-template-ir.md) | ujsx tree as workflow template intermediate representation | | [002](decisions/002-dag-only-graph.md) | Enforce DAG invariants — no cycles in flowgraph | | [003](decisions/003-storage-decoupled.md) | Storage is not flowgraph's concern — in-memory graph with export/import boundary | +| [004](decisions/004-no-schema-version.md) | No schema version field in serialized format — consumers wrap in their own versioned envelope | ## Consumer Context @@ -100,6 +103,7 @@ src/ sequential.ts # ... parallel.ts # ... conditional.ts # ... + map.ts # ... index.ts host/ graphology.ts # HostConfig: ujsx tree → graphology DAG diff --git a/docs/architecture/analysis.md b/docs/architecture/analysis.md index c432096..73780d2 100644 --- a/docs/architecture/analysis.md +++ b/docs/architecture/analysis.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Analysis Functions @@ -23,6 +23,46 @@ All analysis functions are pure: they don't mutate the graph, they don't depend ## Type Compatibility +### Compatibility Contract + +The `typeCompat` function defines a clear contract for what each result means: + +| Result | Meaning | What the consumer should do | +|--------|---------|-----------------------------| +| `{ compatible: true }` | Output schema is a subtype of input schema | Allow the edge; data can flow from source to target without transformation | +| `{ compatible: true, detail }` | Compatible with notes | Allow the edge; the `detail` string describes why (e.g., "output has extra fields beyond input requirements") | +| `{ compatible: false, mismatches }` | Structural incompatibility | Reject the edge or add a transformation step; `mismatches` lists specific field-level problems | +| No edge at all | Unknown compatibility (one or both schemas are `Type.Unknown()`) | Neither compatible nor incompatible; no edge is created | + +### Depth of Compatibility Checking + +`typeCompat` performs **deep recursive structural comparison**: + +1. **Top-level fields** — all required fields in `inputSchema` must be present in `outputSchema` +2. **Nested objects** — recursively compared. If `inputSchema` requires `{ address: { city: string } }`, `outputSchema` providing `{ address: { city: string, zip: string } }` is compatible (output is a superset) +3. **Arrays** — element types are compared. If `inputSchema` requires `string[]`, `outputSchema` providing `(string | number)[]` is **not** compatible (output could produce non-string elements) +4. **Optional fields** — if `inputSchema` marks a field as optional (`Type.Optional()`), it's not required in `outputSchema`. If `outputSchema` omits it, compatibility is still `true`. +5. **Union types** — if `inputSchema` accepts `string | number`, `outputSchema` providing just `string` is compatible (string is a subtype of string | number). The reverse (input requires `string`, output provides `string | number`) is **not** compatible. + +The `mismatches` array provides field-level diagnostics for incompatible results: + +```typescript +interface TypeMismatch { + path: string; // JSON path to the mismatched field (e.g., "/address/city") + expected: string; // What input requires (e.g., "string") + actual: string; // What output provides (e.g., "number") +} +``` + +### Compatibility Rules Summary + +| Output \ Input | Exact match | Superset of input | Subset of input | Unknown | +|---------------|-------------|-------------------|-----------------|---------| +| Exact match | ✅ compatible | ✅ compatible | ❌ incompatible | No edge | +| Superset | ✅ compatible | ✅ compatible | ❌ incompatible | No edge | +| Subset | ❌ incompatible | ❌ incompatible | Depends on which fields | No edge | +| Unknown | No edge | No edge | No edge | No edge | + ### `typeCompat(outputSchema, inputSchema)` ```typescript @@ -99,7 +139,7 @@ function topologicalOrder(graph: FlowGraph): string[] Returns node keys in topological order (prerequisites before dependents). Uses `graphology-dag`'s `topologicalSort` algorithm. -Throws `CircularDependencyError` if the graph contains cycles, with `cycles` populated by `findCycles()`. +Throws `CycleError` if the graph contains cycles, with `cycles` populated by `findCycles()`. ### `parallelGroups(graph)` @@ -239,13 +279,54 @@ This pattern enables: - **Testing** — standalone functions are easier to test in isolation - **Composition** — consumers can chain analysis functions without creating intermediate `FlowGraph` instances +## Performance Characteristics + +Analysis functions are pure and operate on the graph in memory. Their complexity is: + +| Function | Complexity | Notes | +|----------|-----------|-------| +| `topologicalOrder()` | O(V + E) | Linear in nodes + edges. Single traversal. | +| `parallelGroups()` | O(V + E) | Same as topological sort. One pass. | +| `criticalPath()` | O(V + E) | Longest path in DAG. Single traversal with path tracking. | +| `reachableFrom()` | O(V + E) | BFS/DFS from starting nodes. | +| `ancestors()` | O(V + E) | Backward traversal from target. | +| `descendants()` | O(V + E) | Forward traversal from target. | +| `hasCycles()` | O(V + E) | DFS-based cycle detection. Always `false` after validated construction. | +| `findCycles()` | O(V + E) | Johnson's algorithm for finding all elementary cycles. | +| `typeCompat()` | O(depth) | Depends on schema depth. Schemas are typically shallow (5-10 fields). Fast for realistic schemas. | +| `buildTypeEdges()` | O(V²) | Pairwise comparison of all operations. For 50 operations: 2,500 comparisons. For 200: 40,000. Each comparison is `O(depth)`. | +| `validateTemplate()` | O(V + E) | Template traversal plus DAG validation. | +| `validatePreconditions()` | O(V × E) | For each node, check all predecessors. | +| `validateGraph()` | O(V + E) | Cycle detection + edge validation + orphan detection. | + +### Practical Performance + +For expected graph sizes (10-200 nodes): + +- `buildTypeEdges()`: 0.5-5ms for 50 operations, 5-50ms for 200 operations +- `topologicalOrder()`: <1ms for any realistic graph +- `typeCompat()`: <0.01ms per comparison +- All query functions: <1ms for any realistic graph + +These are in-memory operations with no I/O. The dominant cost is `buildTypeEdges()` which scales quadratically with the number of operations. For very large registries (>500 operations), consider lazy edge construction or caching. + +### Optimization Opportunities + +1. **Lazy edge construction** — `buildTypeEdges()` currently compares all pairs. For large registries, edges could be computed on demand: when `typeCompat(A, B)` is queried, compute and cache the result. This trades startup time for query-time cost. + +2. **Type compatibility caching** — `typeCompat()` results could be cached by schema hash. Identical schemas always produce the same result. This helps when the same operation appears in multiple templates. + +3. **Incremental graph updates** — when a single operation is added to the registry, only compute edges for the new node (O(V) instead of O(V²)). + +4. **Parallel group scheduling** — `parallelGroups()` is useful for the hub coordinator to determine max parallelism. An optional `maxConcurrency` parameter could be added to limit group sizes for realistic scheduling. + ## Constraints - **Analysis functions are pure** — they don't mutate the graph, don't depend on external state, and don't throw on validation failures (they return error arrays) - **Type compatibility is structural, not semantic** — `typeCompat()` checks schema shapes, not whether the data makes sense. "Age as number" is compatible with "count as number" even though they're semantically different. - **Template validation is advisory** — warnings are not errors. A template with an unknown operation is a warning, not a validation failure (the operation might be added to the registry later). - **Analysis functions work on the underlying `DirectedGraph`** — they're thin wrappers around graphology and graphology-dag functions, following the same pattern as taskgraph -- **`topologicalOrder()` throws on cycles** — unlike `validateGraph()` which returns errors, `topologicalOrder()` throws `CircularDependencyError` because it cannot produce a valid ordering from a cyclic graph +- **`topologicalOrder()` throws on cycles** — unlike `validateGraph()` which returns errors, `topologicalOrder()` throws `CycleError` because it cannot produce a valid ordering from a cyclic graph ## Open Questions @@ -260,6 +341,6 @@ This pattern enables: ## References - Schema: [schema.md](schema.md) — `TypeCompatResult`, `TypeMismatch`, `ValidationError` -- Error handling: [error-handling.md](error-handling.md) — `CircularDependencyError`, `TypeIncompatError` +- Error handling: [error-handling.md](error-handling.md) — `CycleError`, `TypeIncompatError` - Taskgraph analysis pattern: `@alkdev/taskgraph_ts/src/analysis/` - TypeBox Value utilities: `@alkdev/typebox/value` \ No newline at end of file diff --git a/docs/architecture/build-distribution.md b/docs/architecture/build-distribution.md index ee5c680..034ee0a 100644 --- a/docs/architecture/build-distribution.md +++ b/docs/architecture/build-distribution.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Build & Distribution @@ -17,6 +17,7 @@ Package structure, exports map, dependencies, and platform targets. │ │ ├── sequential.ts # component │ │ ├── parallel.ts # component │ │ ├── conditional.ts # component +│ │ ├── map.ts # component │ │ └── index.ts │ ├── host/ │ │ ├── graphology.ts # GraphologyHostConfig @@ -133,7 +134,7 @@ Following the taskgraph pattern, each module has a sub-path export: | Sub-path | Content | Use case | |----------|---------|----------| | `@alkdev/flowgraph` | Barrel export (everything) | Full import | -| `@alkdev/flowgraph/component` | ``, ``, ``, `` | Template authoring | +| `@alkdev/flowgraph/component` | ``, ``, ``, ``, `` | Template authoring | | `@alkdev/flowgraph/host` | `GraphologyHostConfig`, `ReactiveHostConfig` | ujsx HostConfig implementations | | `@alkdev/flowgraph/schema` | TypeBox schemas, enums, types | Schema-only import (no graph dependency) | | `@alkdev/flowgraph/graph` | `FlowGraph` class, construction, mutation, queries | Core graph operations | @@ -272,6 +273,81 @@ The sub-path export structure enables effective tree-shaking: The barrel export (`@alkdev/flowgraph`) re-exports everything for convenience, but consumers concerned about bundle size should use sub-path imports. +## Testing Strategy + +### Unit Test Categories + +| Category | What to test | How | +|----------|-------------|-----| +| Schema validation | TypeBox schemas validate/correct shapes | `Value.Check()` / `Value.Errors()` | +| Graph construction | `fromSpecs`, `fromCallEvents`, `fromJSON` | Build graphs, assert node/edge counts | +| Graph mutations | `addNode`, `addEdge`, `updateStatus` | Assert success, assert throws on violations | +| Graph queries | `topologicalOrder`, `ancestors`, `descendants` | Known graphs, expected results | +| Type compatibility | `typeCompat` for known schema pairs | Compatible/incompatible/unknown | +| Template validation | `validateTemplate` against known graphs | Known valid/invalid templates | +| Error hierarchy | `CycleError`, `InvalidTransitionError`, etc. | Assert throw types, assert message format | +| Reactive execution | Signal propagation, preconditions, abort cascade | Set up mini reactive graph, assert state transitions | + +### Testing Reactive Graphs + +Testing signal-based state propagation requires specific patterns: + +1. **Setup**: Create a `WorkflowReactiveRoot` with a known DAG. Assert initial state (all nodes `idle`). + +2. **Transition**: Set a node's status signal to a known value. Assert that dependents' `preconditions` and `blockedByFailure` computeds update correctly. + +3. **Assertion**: Check `node.status.value`, `node.preconditions.value`, `node.blockedByFailure.value` at each step. + +```typescript +// Example test pattern +const root = new WorkflowReactiveRoot(dag); +const nodeA = root.statusMap.get("A")!; +const nodeB = root.statusMap.get("B")!; + +// Initially: both idle +expect(nodeA.value).toBe("idle"); +expect(nodeB.preconditions.value).toBe(false); // A not completed yet + +// Complete A → B's preconditions met +nodeA.value = "completed"; +expect(nodeB.preconditions.value).toBe(true); +``` + +4. **Cleanup**: Call `root.dispose()` after each test to prevent signal leaks. + +### Testing Template Rendering + +Template rendering tests follow the same pattern for both HostConfigs: + +1. Define a template +2. Render to the target (graphology or reactive) +3. Assert the output (graph structure or signal state) + +```typescript +// GraphologyHostConfig test +const host = new GraphologyHostConfig(); +const root = createRoot(host, new DirectedGraph()); +root.render(template); +const graph = root.ctx.graph; +expect(graph.nodes()).toEqual(["A", "B", "C"]); +expect(graph.edges()).toEqual(["A->B", "B->C"]); + +// ReactiveHostConfig test +const reactiveHost = new ReactiveHostConfig(registry, workflowRoot); +const reactiveRoot = createRoot(reactiveHost, {}); +reactiveRoot.render(template); +expect(workflowRoot.statusMap.size).toBe(3); +``` + +### Testing Error Paths + +All error paths should be tested: + +- Cycle detection: adding a cycle-creating edge throws `CycleError` +- Duplicate node/edge: adding duplicates throws `ConstructionError` +- Invalid status transition: `updateStatus(completed → running)` throws `InvalidTransitionError` +- Validation errors: `validateGraph()` returns arrays, never throws + ## Constraints - **No filesystem access** — flowgraph is a pure computation library. Persistence is the hub's concern. diff --git a/docs/architecture/consumer-integration.md b/docs/architecture/consumer-integration.md new file mode 100644 index 0000000..4cffff2 --- /dev/null +++ b/docs/architecture/consumer-integration.md @@ -0,0 +1,417 @@ +--- +status: draft +last_updated: 2026-05-19 +--- + +# Consumer Integration Guide + +End-to-end walkthrough: from operation specs to a running workflow. This document shows how a consumer (alkhub, OpenCode, cograph) uses flowgraph's components together. + +## Overview + +The integration path follows five phases: + +``` +1. Register operations → Build operation graph +2. Define workflow template → Validate against operation graph +3. Render template to DAG → Validate DAG structure +4. Create reactive execution → Drive workflow via signals +5. Subscribe to status changes → Respond to completion/failure +``` + +Each phase uses a different flowgraph module. The complete integration uses all modules; partial integrations are possible. + +## Phase 1: Register Operations → Build Operation Graph + +```typescript +import { OperationRegistry } from "@alkdev/operations"; +import { FlowGraph } from "@alkdev/flowgraph/graph"; +import { buildTypeEdges } from "@alkdev/flowgraph/analysis"; + +// 1. Create the registry with operation specs +const registry = new OperationRegistry([ + { namespace: "task", name: "classify", type: "query", inputSchema: {...}, outputSchema: {...} }, + { namespace: "task", name: "enrich", type: "query", inputSchema: {...}, outputSchema: {...} }, + { namespace: "task", name: "summarize", type: "mutation", inputSchema: {...}, outputSchema: {...} }, + // ... more operations +]); + +// 2. Build the operation graph +const operationGraph = FlowGraph.fromSpecs(registry.getAll()); + +// 3. The graph now has type-compatibility edges +operationGraph.hasEdge("task.classify", "task.enrich"); // → true (if compatible) +operationGraph.getEdgeAttributes("task.classify", "task.enrich"); +// → { edgeType: "typed", compatible: true, detail: "classify.output → enrich.input" } +``` + +**What happens internally**: +- `fromSpecs()` creates a node for each operation (key: `namespace.name`) +- `buildTypeEdges()` compares each pair's `outputSchema` → `inputSchema` and adds edges +- Cycles are rejected at construction time (DAG invariant) + +**Partial integration**: If you only need the operation graph (no workflows), stop here. The operation graph is useful for type-compatibility queries and topological ordering without defining any templates. + +## Phase 2: Define Workflow Template → Validate + +```typescript +import { h } from "@alkdev/ujsx"; +import { Operation, Sequential, Parallel, Conditional, Map } from "@alkdev/flowgraph/component"; +import { validateTemplate } from "@alkdev/flowgraph/analysis"; + +// Define a template +const template = h(Sequential, {}, + h(Operation, { name: "task.classify" }), + h(Conditional, { + test: (results) => results["task.classify"].output.confidence > 0.8, + }, + // High-confidence path + h(Parallel, {}, + h(Operation, { name: "task.enrich" }), + h(Operation, { name: "task.summarize" }), + ), + // Low-confidence fallback + h(Operation, { name: "task.classify" }), // re-classify with different params + ), +); + +// Validate against the operation graph +const errors = validateTemplate(template, operationGraph); +if (errors.length > 0) { + for (const error of errors) { + console.error(`Validation error: ${error.type}`, error); + } + // Handle errors... +} +``` + +**Validation checks**: +1. All `Operation` names exist in the registry +2. No cycles in the rendered DAG +3. Type compatibility between sequential operations +4. All operations are reachable from the start + +**Template serialization** (for storage/transmission): + +```typescript +// Serialize to JSON +const json = JSON.stringify(template); + +// Deserialize and validate +const parsed = JSON.parse(json); +const templateErrors = validateTemplate(parsed, operationGraph); +``` + +Note: function-valued props (like `Conditional.test`) don't survive JSON serialization. Use string references for stored templates and resolve them at render time. + +## Phase 3: Render Template to DAG → Validate Structure + +```typescript +import { createRoot } from "@alkdev/ujsx"; +import { GraphologyHostConfig } from "@alkdev/flowgraph/host/graphology"; +import { DirectedGraph } from "graphology"; + +// Create the GraphologyHostConfig +const hostConfig = new GraphologyHostConfig(); +const root = createRoot(hostConfig, new DirectedGraph()); + +// Render the template to a DAG +root.render(template); + +// The DAG is now available in the root context +const dag = root.ctx.graph; + +// Validate the DAG +dag.hasCycles(); // → false (always, if template is valid) +dag.nodes(); // → ["task.classify", "task.enrich", "task.summarize"] +dag.edges(); // → ["task.classify->task.enrich", "task.classify->task.summarize"] + +// Query the DAG +dag.inNeighbors("task.enrich"); // → ["task.classify"] +dag.outNeighbors("task.classify"); // → ["task.enrich", "task.summarize"] +``` + +**What happens internally**: +- The `GraphologyHostConfig` renders each `Operation` as a node and each structural relationship (`Sequential`, `Parallel`, `Conditional`) as edges +- Structural containers (`Sequential`, `Parallel`, `Conditional`) are transparent — they produce edges, not nodes +- The result is a pure DAG that can be analyzed, serialized, or used for validation + +## Phase 4: Create Reactive Execution → Drive Workflow + +```typescript +import { WorkflowReactiveRoot } from "@alkdev/flowgraph/reactive"; +import { ReactiveHostConfig } from "@alkdev/flowgraph/host/reactive"; + +// 1. Create the ReactiveRoot from the DAG +const workflowRoot = new WorkflowReactiveRoot(dag); + +// 2. Create the ReactiveHostConfig +const reactiveHost = new ReactiveHostConfig(registry, workflowRoot); + +// 3. Render the template to create reactive state +const reactiveRoot = createRoot(reactiveHost, {}); +reactiveRoot.render(template); + +// 4. Subscribe to status changes and effect-driven execution +for (const [nodeId, node] of workflowRoot.nodes) { + // Start the call when preconditions are met + effect(() => { + if (node.preconditions.value && node.status.value === "idle" || node.status.value === "waiting") { + node.status.value = "running"; + // getInput resolves the node's input from predecessor outputs and static config + // For Operation nodes, input comes from the template props or aggregated predecessor results + const input = resolveInput(nodeId, workflowRoot); + registry.execute(node.operationId, input, { parentRequestId: parentCallId }) + .then(result => { node.status.value = "completed"; node.output.value = result; }) + .catch(error => { node.status.value = "failed"; }); + } + }); + + // Track failures + effect(() => { + if (node.status.value === "failed") { + console.error(`Node ${nodeId} failed`); + } + }); +} + +// 5. Kick off the workflow — root nodes start as "ready" +// (The effect-driven execution above handles the rest automatically) +// Root nodes' preconditions are true by default (no predecessors) +// so they transition to "ready" immediately +``` + +**What happens automatically**: +- Node status changes propagate reactively through `computed` preconditions +- When a predecessor completes, dependents automatically transition to `ready` +- When a predecessor fails, dependents' `blockedByFailure` triggers and they transition to `aborted` +- The entire workflow progresses without manual orchestration + +## Phase 5: Handle Completion → Cleanup + +```typescript +// Track overall workflow status +const allNodes = Array.from(workflowRoot.statusMap.values()); +const allCompleted = () => allNodes.every(s => + s.value === "completed" || s.value === "failed" || s.value === "aborted" || s.value === "skipped" +); + +// Check for success +effect(() => { + if (allCompleted()) { + const failed = allNodes.filter(s => s.value === "failed"); + const aborted = allNodes.filter(s => s.value === "aborted"); + const completed = allNodes.filter(s => s.value === "completed"); + const skipped = allNodes.filter(s => s.value === "skipped"); + + console.log(`Workflow complete: ${completed.length} completed, ${failed.length} failed, ${aborted.length} aborted, ${skipped.length} skipped`); + + // Cleanup + workflowRoot.dispose(); + } +}); + +// Handle system-level abort (e.g., provider outage, auth failure) +function handleSystemFailure(error: Error) { + workflowRoot.abortAll(); + prm.abortAll(pendingRequestIds); + workflowRoot.dispose(); + console.error(`Workflow aborted: ${error.message}`); +} +``` + +## Export/Import for Persistence + +```typescript +import { FlowGraph } from "@alkdev/flowgraph/graph"; + +// Export the call graph for persistence +const serialized = callGraph.export(); +// → FlowGraphSerialized format (graphology native JSON) + +// Store in Postgres (hub's responsibility) +await db.query('INSERT INTO call_graphs (id, data) VALUES ($1, $2)', [workflowId, JSON.stringify(serialized)]); + +// Restore from persistence +const restored = FlowGraph.fromJSON(serialized); +// → FlowGraph with all nodes and edges +``` + +## Call Graph Population (Real-Time) + +The call graph can be populated incrementally from call protocol events: + +```typescript +import { FlowGraph } from "@alkdev/flowgraph/graph"; + +// Create empty call graph +const callGraph = new FlowGraph(); + +// Subscribe to call protocol events +pubsub.subscribe("call.requested", (event) => callGraph.updateFromEvent(event)); +pubsub.subscribe("call.responded", (event) => callGraph.updateFromEvent(event)); +pubsub.subscribe("call.error", (event) => callGraph.updateFromEvent(event)); +pubsub.subscribe("call.aborted", (event) => callGraph.updateFromEvent(event)); +pubsub.subscribe("call.completed", (event) => callGraph.updateFromEvent(event)); + +// Query the call graph for observability +callGraph.filterByStatus("running"); // What's currently running +callGraph.children("req_abc123"); // Children of a call +callGraph.lineage("req_xyz789"); // Ancestor chain +callGraph.duration("req_abc123"); // How long a call took +``` + +## Minimal Integration Example + +For consumers that only need the operation graph and template validation (no reactive execution): + +```typescript +import { FlowGraph } from "@alkdev/flowgraph/graph"; +import { h } from "@alkdev/ujsx"; +import { Operation, Sequential } from "@alkdev/flowgraph/component"; +import { validateTemplate, typeCompat } from "@alkdev/flowgraph/analysis"; + +// 1. Build operation graph +const operationGraph = FlowGraph.fromSpecs(registry.getAll()); + +// 2. Define and validate template +const template = h(Sequential, {}, + h(Operation, { name: "task.classify" }), + h(Operation, { name: "task.enrich" }), +); +const errors = validateTemplate(template, operationGraph); + +// 3. Query type compatibility +const result = typeCompat( + registry.get("task.classify").outputSchema, + registry.get("task.enrich").inputSchema, +); +console.log(result.compatible); // → true or false +console.log(result.mismatches); // → TypeMismatch[] if incompatible +``` + +This integration only requires `@alkdev/flowgraph/graph`, `@alkdev/flowgraph/component`, and `@alkdev/flowgraph/analysis`. No reactive execution, no ujsx HostConfig, no signals. + +## Module Dependency Map + +``` +┌─────────────────────────────────────────────────┐ +│ Consumer (hub coordinator, OpenCode plugin) │ +└────────┬────────────────┬────────────────┬───────┘ + │ │ │ + ┌────▼────┐ ┌──────▼──────┐ ┌──────▼──────┐ + │ graph │ │ component │ │ analysis │ + │ │ │ │ │ │ + │FlowGraph│ │Operation │ │typeCompat │ + │fromSpecs│ │Sequential │ │validate │ + │queries │ │Parallel │ │topological │ + │mutations│ │Conditional │ │parallelGroups│ + └────┬────┘ │Map │ └──────┬──────┘ + │ └──────┬───────┘ │ + │ │ │ + ┌────▼────────────────▼─────────────────▼─────┐ + │ schema │ + │ OperationNodeAttrs CallNodeAttrs │ + │ OperationEdgeAttrs CallEdgeAttrs │ + │ TemplateEdgeAttrs NodeStatus EdgeType │ + └──────────────────┬──────────────────────────┘ + │ + ┌──────────────────▼──────────────────────────┐ + │ host │ + │ GraphologyHostConfig ReactiveHostConfig │ + └──────────────────┬──────────────────────────┘ + │ + ┌──────────────────▼──────────────────────────┐ + │ reactive │ + │ WorkflowReactiveRoot WorkflowNode │ + │ NodeStatus signals computed preconditions │ + └──────────────────┬──────────────────────────┘ + │ + ┌──────────────────▼──────────────────────────┐ + │ error │ + │ FlowgraphError hierarchy │ + └──────────────────────────────────────────────┘ + + External dependencies: + ┌────────────┐ ┌────────────┐ ┌──────────────┐ + │ graphology │ │ ujsx │ │@preact/sign │ + │ graphology │ │ h, create │ │ als-core │ + │ -dag │ │ Root │ │ signal,comp, │ + └─────────────┘ └────────────┘ │ effect │ + └──────────────┘ +``` + +## Common Patterns + +### Pattern: SDD Pipeline + +```typescript +// The archetypal SDD (Spec-Driven Development) pipeline +const sddPipeline = h(Sequential, {}, + h(Operation, { name: "task.architect" }), + h(Conditional, { + test: (results) => results["task.architect"].output.approved, + }, + h(Sequential, {}, + h(Operation, { name: "task.decomposer" }), + h(Operation, { name: "task.coordinator" }), + ), + // else-branch: architect disapproved, loop back or stop + h(Operation, { name: "task.notify-stakeholder" }), + ), +); +``` + +### Pattern: Fan-Out/Fan-In + +```typescript +// Process items in parallel, then aggregate +const fanOut = h(Sequential, {}, + h(Operation, { name: "task.fetch-items" }), + h(Map, { + over: (results) => results["task.fetch-items"].output.items, + as: "item", + }, + h(Operation, { name: "task.process-item" }), + ), + h(Operation, { name: "task.aggregate-results" }), +); +``` + +### Pattern: Error Boundary with Conditional + +```typescript +// Critical operation with graceful degradation +const withFallback = h(Sequential, {}, + h(Conditional, { + test: (results) => results["task.fetch-data"].status !== "failed", + }, + // Happy path + h(Operation, { name: "task.transform" }), + // Fallback + h(Operation, { name: "task.use-cache" }), + ), + // This operation runs regardless — the Conditional resolves + // whether the then or else branch was taken + h(Operation, { name: "task.notify" }), +); +``` + +## Constraints on Consumers + +- **The hub coordinator drives execution** — flowgraph provides reactive state (signals, computed), not call execution. The coordinator reads `preconditions` and `blockedByFailure` and calls `registry.execute()` when appropriate. +- **Dispose is mandatory** — `WorkflowReactiveRoot.dispose()` must be called when the workflow completes or is cancelled. Without disposal, signal subscriptions leak. +- **Template rendering is currently one-shot** — until the ujsx reconciler is implemented, `createRoot(host, container).render(template)` can only be called once per root. To re-render, create a new root. +- **Function props don't survive serialization** — `Conditional.test` and `Map.over` with function values require runtime resolution. Use string references for stored templates. +- **Call graph is independent of reactive execution** — you can build a call graph from events without using the reactive layer. The reactive layer is optional for consumers that only need observability. + +## References + +- Architecture overview: [README.md](README.md) +- FlowGraph API: [flowgraph-api.md](flowgraph-api.md) +- Schema: [schema.md](schema.md) +- Workflow templates: [workflow-templates.md](workflow-templates.md) +- Host configs: [host-configs.md](host-configs.md) +- Reactive execution: [reactive-execution.md](reactive-execution.md) +- Call graph: [call-graph.md](call-graph.md) +- Analysis: [analysis.md](analysis.md) \ No newline at end of file diff --git a/docs/architecture/decisions/004-no-schema-version.md b/docs/architecture/decisions/004-no-schema-version.md new file mode 100644 index 0000000..006ec82 --- /dev/null +++ b/docs/architecture/decisions/004-no-schema-version.md @@ -0,0 +1,64 @@ +# ADR-004: No Schema Version Field in Serialized Format + +## Status + +Accepted + +## Context + +Flowgraph's `FlowGraphSerialized` type follows graphology's native JSON format. The format does not include a `schemaVersion` field. This decision affects: + +1. **Backward compatibility** — how consumers handle format changes across versions +2. **Persistence** — whether stored graphs can be reliably migrated +3. **Interoperability** — whether different versions of flowgraph can exchange data + +The review (001-architecture-gap-analysis.md, W-07) flagged this as a potential gap: "Consumers that need persistence wrap it in their own versioned envelope." + +## Decision + +Flowgraph's serialized format will NOT include a `schemaVersion` field. This follows the same pattern as taskgraph. Consumers that need versioned persistence must wrap the serialized format in their own envelope that includes version metadata. + +## Rationale + +1. **Graphology format is upstream** — the serialized format is graphology's native JSON format. Adding a `schemaVersion` field modifies a format we don't own. This creates a fork between what graphology produces and what flowgraph expects. + +2. **Flowgraph is not a persistence layer** — the library handles in-memory graph construction, validation, and analysis. Persistence is explicitly the consumer's responsibility (see ADR-003). Adding versioning to the serialized format would blur this boundary. + +3. **Versioning belongs at the envelope level** — a consumer storing graphs in Postgres should wrap the serialized data in a versioned envelope: `{ version: 1, data: FlowGraphSerialized, metadata: {...} }`. This gives the consumer full control over migration logic, which they need anyway for their own schema migrations. + +4. **Breaking changes are breaking changes** — if the serialized format changes incompatibly, no version field will help. The consumer must handle the migration. A version field would give false confidence that "old versions can be read" when in practice the consumer must write migration code. + +5. **Taskgraph precedent** — taskgraph uses the same approach and it has worked well. The pattern is established in the ecosystem. + +## Consequences + +- **Positive**: Simpler serialized format, no dependency on version parsing, no false confidence in backward compatibility. +- **Positive**: Clean separation of concerns — flowgraph handles graph operations, consumers handle persistence and versioning. +- **Negative**: Consumers must implement their own versioned envelope if they persist graphs. This is a small burden documented in the consumer integration guide. +- **Negative**: There's no standard way for two different flowgraph versions to detect incompatibility. The consumer must track this themselves. + +## Mitigation + +The consumer integration guide documents the recommended pattern: + +```typescript +// Consumer-side versioned envelope +interface PersistedGraph { + version: number; // Increment on breaking changes + data: FlowGraphSerialized; // Raw graphology format + metadata: { + createdAt: string; + graphType: "operation" | "call"; + flowgraphVersion: string; // The npm package version for reference + }; +} +``` + +When restoring a graph, the consumer checks `version` and `flowgraphVersion` to decide whether migration is needed. This is outside flowgraph's responsibility. + +## References + +- Schema: [schema.md](../schema.md) — `FlowGraphSerialized`, `SerializedGraph` factory +- Storage decoupled ADR: [003-storage-decoupled.md](003-storage-decoupled.md) +- Consumer integration: [consumer-integration.md](../consumer-integration.md) +- Review: [001-architecture-gap-analysis.md](../../reviews/001-architecture-gap-analysis.md) — W-07 \ No newline at end of file diff --git a/docs/architecture/error-handling.md b/docs/architecture/error-handling.md index 809f2b1..1561c4f 100644 --- a/docs/architecture/error-handling.md +++ b/docs/architecture/error-handling.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Error Handling @@ -115,7 +115,7 @@ class CycleError extends ConstructionError { Thrown when adding an edge would create a cycle. The `cycles` field contains the cycle paths that would be created. -Note: Unlike `CircularDependencyError` in taskgraph (which is thrown by `topologicalOrder()`), `CycleError` is thrown by `addEdge()` during construction. Taskgraph allows cycles and detects them later; flowgraph prevents them at construction time. +Note: `CycleError` is flowgraph's cycle error, thrown by `addEdge()` during construction. Taskgraph uses a different error name (`CircularDependencyError`, thrown by `topologicalOrder()`). The two are distinct errors for distinct contexts — flowgraph prevents cycles at construction time, taskgraph allows cycles and detects them later. ### ValidationError @@ -236,7 +236,7 @@ The distinction between thrown errors and returned errors: | `validateGraph()` | Returns `GraphValidationError[]` | Graph issues are validations, not crashes | | `validateTemplate()` | Returns `AnyValidationError[]` | Template issues are validations, not crashes | | `analyzeTypeCompat()` | Returns `TypeCompatResult` (includes mismatches) | Type incompatibility is advisory, not blocking | -| `topologicalOrder()` | Throws `CircularDependencyError` on cycles | No valid ordering exists from a cyclic graph | +| `topologicalOrder()` | Throws `CycleError` on cycles | No valid ordering exists from a cyclic graph | This matches taskgraph's pattern: construction enforces invariants (throwing on violations), validation reports issues (returning error arrays). diff --git a/docs/architecture/flowgraph-api.md b/docs/architecture/flowgraph-api.md new file mode 100644 index 0000000..d73e2fe --- /dev/null +++ b/docs/architecture/flowgraph-api.md @@ -0,0 +1,309 @@ +--- +status: draft +last_updated: 2026-05-19 +--- + +# FlowGraph Public API + +Complete public API surface for the `FlowGraph` class — constructor, type parameters, methods, and the delegation model with graphology. + +## Overview + +`FlowGraph` is the central class that wraps a graphology `DirectedGraph` and enforces DAG invariants. It is generic over node and edge attribute types, supporting three distinct graph modes: operation graph, call graph, and template DAG. + +The class delegates graph operations to graphology while providing flowgraph-specific methods for construction, mutation, queries, and analysis. It is NOT a subclass of `DirectedGraph` — it wraps one, exposing a curated API surface. + +## Type Parameters + +```typescript +class FlowGraph< + NodeAttrs extends TSchema = OperationNodeAttrs | CallNodeAttrs, + EdgeAttrs extends TSchema = OperationEdgeAttrs | CallEdgeAttrs | TemplateEdgeAttrs, +> { + private _graph: DirectedGraph; + // ... +} +``` + +| Parameter | Default | Purpose | +|-----------|---------|---------| +| `NodeAttrs` | `OperationNodeAttrs` or `CallNodeAttrs` | TypeBox schema for node attributes | +| `EdgeAttrs` | `OperationEdgeAttrs`, `CallEdgeAttrs`, or `TemplateEdgeAttrs` | TypeBox schema for edge attributes | + +Common instantiations: + +```typescript +// Operation graph (static type compatibility) +type OperationGraph = FlowGraph; + +// Call graph (dynamic call events) +type CallGraph = FlowGraph; + +// Template DAG (workflow structure) +type TemplateDAG = FlowGraph; +``` + +## Constructor and Factories + +### `new FlowGraph()` + +```typescript +constructor(options?: FlowGraphOptions) +``` + +Creates an empty graph. Options: + +```typescript +interface FlowGraphOptions { + type?: "directed"; // Always "directed" (default) + multi?: false; // Always false (default) — no parallel edges + allowSelfLoops?: false; // Always false (default) — no self-loops +} +``` + +The options object is passed through to `new DirectedGraph()`. The DAG constraints (`multi: false`, `allowSelfLoops: false`) are enforced at the graphology level. + +### `FlowGraph.fromSpecs(specs)` + +```typescript +static fromSpecs(specs: OperationSpec[]): OperationGraph +``` + +Constructs an operation graph from an array of `OperationSpec` objects. Creates nodes for each operation and type-compatibility edges via `buildTypeEdges()`. Throws `CycleError` if the resulting graph has cycles (shouldn't happen with valid operation specs, but validated defensively). + +### `FlowGraph.fromCallEvents(events)` + +```typescript +static fromCallEvents(events: CallEventMapValue[]): CallGraph +``` + +Constructs a call graph from an array of call protocol events. Processes events in order, adding nodes and edges. Idempotent — duplicate events have no effect. + +### `FlowGraph.fromJSON(data)` + +```typescript +static fromJSON(data: FlowGraphSerialized): FlowGraph +``` + +Deserializes from graphology native JSON format. Validates against the appropriate schema (`OperationGraphSerialized` or `CallGraphSerialized`). Throws `InvalidInputError` on validation failure. + +Round-trip guarantee: `fromSpecs()` → `export()` → `fromJSON()` is lossless. + +## Mutation Methods + +### Node Mutations + +| Method | Signature | Behavior | +|--------|-----------|----------| +| `addNode` | `(key: string, attrs: NodeAttrs): void` | Adds a node. Throws `DuplicateNodeError` if key exists. | +| `removeNode` | `(key: string): void` | Removes a node and all attached edges. Throws `NodeNotFoundError` if key doesn't exist. | +| `updateNode` | `(key: string, attrs: Partial): void` | Merges attributes into an existing node. Throws `NodeNotFoundError` if key doesn't exist. | +| `hasNode` | `(key: string): boolean` | Checks if a node exists. | +| `getNodeAttributes` | `(key: string): NodeAttrs` | Returns node attributes. Throws `NodeNotFoundError` if key doesn't exist. | + +### Edge Mutations + +| Method | Signature | Behavior | +|--------|-----------|----------| +| `addEdge` | `(source: string, target: string, attrs?: EdgeAttrs): void` | Adds a directed edge. Throws `NodeNotFoundError` if either endpoint doesn't exist. Throws `CycleError` if the edge would create a cycle. Throws `DuplicateEdgeError` if an edge already exists between the same (source, target). | +| `removeEdge` | `(source: string, target: string): void` | Removes an edge. No-op if the edge doesn't exist. | +| `hasEdge` | `(source: string, target: string): boolean` | Checks if an edge exists between source and target. | +| `getEdgeAttributes` | `(source: string, target: string): EdgeAttrs` | Returns edge attributes. Throws if edge doesn't exist. | + +### Call Graph Mutations + +These are convenience methods specific to `FlowGraph`: + +| Method | Signature | Behavior | +|--------|-----------|----------| +| `addCall` | `(attrs: CallNodeAttrs): void` | Adds a call node. If `attrs.parentRequestId` is set, also creates a `triggered` edge from parent to child. | +| `addDependency` | `(source: string, target: string): void` | Creates a `depends_on` edge. Validates both endpoints exist and the edge wouldn't create a cycle. | +| `updateStatus` | `(requestId: string, status: CallStatus, extra?: Partial): void` | Updates call status. Throws `InvalidTransitionError` on invalid transitions. | +| `updateCall` | `(requestId: string, attrs: Partial): void` | Partial merge of call attributes. | +| `removeCall` | `(requestId: string): void` | Removes a call node and all attached edges. | + +### Operation Graph Mutations + +| Method | Signature | Behavior | +|--------|-----------|----------| +| `addOperation` | `(spec: OperationSpec): void` | Adds an operation node. Key is `${spec.namespace}.${spec.name}`. | +| `addTypedEdge` | `(source: string, target: string, attrs: { compatible: boolean; detail?: string }): void` | Adds a type-compatibility edge with `edgeType: "typed"`. | + +## Query Methods + +### Graph Traversal + +These methods delegate directly to graphology and graphology-dag: + +| Method | Returns | Delegated To | +|--------|---------|-------------| +| `topologicalOrder()` | `string[]` | `graphology-dag.topologicalSort` | +| `hasCycles()` | `boolean` | `graphology-dag.hasCycle` (always `false` after validated construction) | +| `findCycles()` | `string[][]` | `graphology-dag.findCycle` (debugging) | +| `ancestors(nodeId)` | `string[]` | `graphology-dag.ancestors` | +| `descendants(nodeId)` | `string[]` | `graphology-dag.descendants` | +| `predecessors(nodeId)` | `string[]` | `graph.inNeighbors` | +| `successors(nodeId)` | `string[]` | `graph.outNeighbors` | +| `reachableFrom(nodeIds)` | `Set` | Custom BFS/DFS traversal | + +### Call Graph Queries + +| Method | Returns | Description | +|--------|---------|-------------| +| `filterByStatus(status)` | `string[]` | Node keys with the given status | +| `getRoots()` | `string[]` | Top-level call nodes (no `parentRequestId`) | +| `children(requestId)` | `string[]` | Direct children via `triggered` edges | +| `duration(requestId)` | `number` | `completedAt - startedAt` in ms | +| `lineage(requestId)` | `string[]` | Ancestor chain from root to this call | + +## Serialization + +| Method | Signature | Description | +|--------|-----------|-------------| +| `export()` | `FlowGraphSerialized` | Returns graphology native JSON format | +| `toJSON()` | `FlowGraphSerialized` | Alias for `export()` | +| `toString()` | `string` | JSON.stringify of `export()` | + +## Analysis Convenience Methods + +The `FlowGraph` class exposes convenience methods that delegate to standalone analysis functions: + +```typescript +class FlowGraph { + // Delegates to analysis/topologicalOrder + topologicalOrder(): string[] { return _topologicalOrder(this._graph); } + + // Delegates to analysis/hasCycles + hasCycles(): boolean { return _hasCycles(this._graph); } + + // Delegates to analysis/validate + validate(): AnyValidationError[] { return _validate(this._graph); } + + // Delegates to analysis/typeCompat + typeCompat(sourceKey: string, targetKey: string): TypeCompatResult { + const source = this.getNodeAttributes(sourceKey); + const target = this.getNodeAttributes(targetKey); + return _typeCompat(source.outputSchema, target.inputSchema); + } +} +``` + +Standalone functions are also available from `@alkdev/flowgraph/analysis`: + +```typescript +import { topologicalOrder, hasCycles, validateGraph, typeCompat } from "@alkdev/flowgraph/analysis"; +``` + +## Delegation Model + +`FlowGraph` wraps a graphology `DirectedGraph` instance. It does NOT extend `DirectedGraph`: + +```typescript +class FlowGraph { + private _graph: DirectedGraph; + + // Construction + constructor(options?: FlowGraphOptions) { + this._graph = new DirectedGraph({ + type: "directed", + multi: false, + allowSelfLoops: false, + }); + } + + // Internal access for delegation + get graph(): DirectedGraph { return this._graph; } +} +``` + +### What FlowGraph delegates + +The `FlowGraph` class exposes only a subset of graphology's API — the methods that are meaningful for an enforced-DAG graph: + +| Category | What FlowGraph does | What raw graphology provides | +|----------|---------------------|------------------------------| +| Node ops | `addNode`, `removeNode`, `hasNode`, `getNodeAttributes`, `updateNode` | Full node CRUD + attributes | +| Edge ops | `addEdge`, `removeEdge`, `hasEdge`, `getEdgeAttributes` | Full edge CRUD + attributes | +| Traversal | `topologicalOrder`, `ancestors`, `descendants`, `predecessors`, `successors` | All graphology traversal methods | +| Queries | `filterByStatus`, `getRoots`, `children`, `duration`, `lineage` | N/A (flowgraph-specific) | +| Analysis | `typeCompat`, `validate`, `hasCycles` | N/A (flowgraph-specific) | + +### What FlowGraph does NOT expose + +Methods that would violate DAG invariants or are unnecessary for flowgraph's use cases: + +- `addUndirectedEdge` — not applicable, all edges are directed +- `addEdgeWithKey` — edge keys are deterministic (`${source}->${target}`), not user-specified +- `merge` / `mergeEdge` — graph merging is not a supported operation (rebuild instead) +- `import` — use `FlowGraph.fromJSON()` which validates schema +- Any `multi: true` or `allowSelfLoops: true` options + +### Direct graphology access + +Consumers who need graphology's full API can access the underlying graph via `flowGraph.graph`: + +```typescript +const graph = flowGraph.graph; +graph.forEachNode((node, attrs) => { + console.log(node, attrs); +}); +``` + +This is an escape hatch. Direct graph mutation bypasses flowgraph's validation (cycle detection, duplicate checks). Use with caution. + +## Immutability Guarantees + +| Method | Mutates? | Returns | +|--------|----------|---------| +| `fromSpecs()` | Creates new graph | `OperationGraph` | +| `fromCallEvents()` | Creates new graph | `CallGraph` | +| `fromJSON()` | Creates new graph | `FlowGraph` | +| `addNode`, `addEdge`, etc. | **Yes — mutates** | `void` | +| `removeNode`, `removeEdge` | **Yes — mutates** | `void` | +| `updateNode`, `updateStatus` | **Yes — mutates** | `void` | +| `export()`, `toJSON()` | No — reads | Serialized data | +| `topologicalOrder()`, `ancestors()`, etc. | No — reads | Query results | +| `validate()`, `hasCycles()` | No — reads | Validation results | +| `typeCompat()` | No — reads | `TypeCompatResult` | + +**Key invariant**: The operation graph produced by `fromSpecs()` is immutable after construction — no mutation methods are exposed. If the registry changes, rebuild the graph. The call graph produced by `fromCallEvents()` supports incremental mutation via `addCall`, `updateStatus`, and `addDependency`. The initial events populate the graph, and subsequent events update it. + +## Exports Map + +| Sub-path | Key exports | +|-----------|-------------| +| `@alkdev/flowgraph` | `FlowGraph`, all public types | +| `@alkdev/flowgraph/graph` | `FlowGraph`, `FlowGraphOptions` | +| `@alkdev/flowgraph/analysis` | `typeCompat`, `buildTypeEdges`, `validateGraph`, `validateTemplate`, `topologicalOrder`, `parallelGroups`, `criticalPath`, `reachableFrom` | +| `@alkdev/flowgraph/schema` | `OperationNodeAttrs`, `CallNodeAttrs`, `OperationEdgeAttrs`, `CallEdgeAttrs`, `TemplateEdgeAttrs`, `CallStatus`, `NodeStatus`, `EdgeType` | +| `@alkdev/flowgraph/component` | `Operation`, `Sequential`, `Parallel`, `Conditional`, `Map` | +| `@alkdev/flowgraph/host` | `GraphologyHostConfig`, `ReactiveHostConfig` | +| `@alkdev/flowgraph/reactive` | `WorkflowReactiveRoot`, `WorkflowNode`, `ReactiveContext` | +| `@alkdev/flowgraph/error` | `FlowgraphError`, `ConstructionError`, `CycleError`, `ValidationError`, `TypeIncompatError`, `InvalidTransitionError` | + +## Constraints + +- **FlowGraph wraps, not extends, graphology** — direct `DirectedGraph` access is available via `.graph` but bypasses validation. +- **DAG invariants enforced at construction time** — `addEdge` throws `CycleError` if the edge would create a cycle. `hasCycles()` should always return `false` after validated construction. +- **No parallel edges** — `addEdge` throws `DuplicateEdgeError` if an edge already exists between the same (source, target) pair. +- **No self-loops** — enforced at the graphology level (`allowSelfLoops: false`). +- **Edge keys are deterministic** — `${source}->${target}` format. No user-specified edge keys. +- **Operation graph is immutable after construction** — no mutation methods are exposed after `fromSpecs()`. If the registry changes, rebuild the graph. +- **Call graph supports incremental mutation** — `addCall`, `updateStatus`, `addDependency` are the primary mutation paths. + +## Open Questions + +1. **Should `FlowGraph` expose graphology's traversal methods directly or only via convenience methods?** Currently the plan is convenience methods that delegate. Direct graphology access via `.graph` is the escape hatch. But some consumers may find it inconvenient to go through `.graph.forEachNode()` instead of `flowGraph.forEachNode()`. + +2. **Should the operation graph's `addTypedEdge` be auto-populated or manual?** Currently `fromSpecs()` calls `buildTypeEdges()` which adds all type-compatibility edges. `addTypedEdge` is for manual or incremental construction. Should `addOperation` also attempt auto-type-compat edge creation? + +3. **Should `FlowGraph` support multiple graph instances sharing analysis functions?** Currently each `FlowGraph` instance owns its own `DirectedGraph`. A future optimization could pool analysis functions across instances. + +## References + +- Schema: [schema.md](schema.md) — TypeBox schemas for all node/edge attribute types +- Operation graph: [operation-graph.md](operation-graph.md) — Static graph construction and queries +- Call graph: [call-graph.md](call-graph.md) — Dynamic graph from call events +- Analysis: [analysis.md](analysis.md) — Type compatibility, validation, ordering +- Error handling: [error-handling.md](error-handling.md) — Error hierarchy +- Build & distribution: [build-distribution.md](build-distribution.md) — Exports map and package structure \ No newline at end of file diff --git a/docs/architecture/host-configs.md b/docs/architecture/host-configs.md index f56dc77..b509391 100644 --- a/docs/architecture/host-configs.md +++ b/docs/architecture/host-configs.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Host Configs @@ -16,7 +16,7 @@ Flowgraph uses ujsx's `HostConfig` pattern to render the same workflow template | GraphologyHostConfig | `DirectedGraph` | Validate templates, check cycles, compute topological order | | ReactiveHostConfig | `Map` | Runtime execution with signal-driven status propagation | -Both HostConfigs share the same template components (`Operation`, `Sequential`, `Parallel`, `Conditional`) and the same tag type. The difference is what `createInstance` and `appendChild` do: +Both HostConfigs share the same template components (`Operation`, `Sequential`, `Parallel`, `Conditional`, `Map`) and the same tag type. The difference is what `createInstance` and `appendChild` do: - **GraphologyHostConfig**: Creates graph nodes and edges. `appendChild` adds an edge. - **ReactiveHostConfig**: Creates a `WorkflowNode` (with a `signal`) and registers preconditions. `appendChild` registers the parent-child dependency. @@ -29,6 +29,20 @@ type WorkflowTag = "operation" | "sequential" | "parallel" | "conditional" | "ma This constrains `HostConfig` to only accept workflow-specific element types. Attempting to render an unsupported tag (e.g., `"div"`) is a type error at compile time. +### Component-to-Tag Mapping + +Each `UComponent` function produces a `UElement` with a specific `type` string (the `WorkflowTag`). The mapping is: + +| Component function | UElement.type (WorkflowTag) | +|-------------------|---------------------------| +| `Operation` | `"operation"` | +| `Sequential` | `"sequential"` | +| `Parallel` | `"parallel"` | +| `Conditional` | `"conditional"` | +| `Map` | `"map"` | + +When ujsx's reconciler calls `HostConfig.createInstance(tag, props, ...)`, the `tag` parameter is the `WorkflowTag` string. For example, `h(Operation, { name: "classify" })` produces `{ type: "operation", props: { name: "classify" }, children: [] }`, and `createInstance("operation", { name: "classify" }, ctx)` is called. + ## GraphologyHostConfig ### Type Parameters @@ -52,6 +66,8 @@ interface GraphNode { } ``` +Where `TemplateNodeAttrs` is a type alias for `OperationNodeAttrs` (see [schema.md](schema.md#TemplateNodeAttrs)) — template nodes carry the same attributes as operation nodes. Structural containers (`Sequential`, `Parallel`, `Conditional`, `Map`) return a `GraphNode` with an empty `attributes` object and a synthetic key. + The `RootCtx` is: ```typescript @@ -147,9 +163,36 @@ finalizeInstance?(instance: GraphNode, ctx: GraphContext): void { } ``` +### removeChild + +```typescript +removeChild(parent: GraphNode, child: GraphNode, ctx: GraphContext): void { + // Remove the edge between parent and child + // Structural containers are transparent, so parent/child are real operation nodes + if (!isStructuralContainer(parent) && !isStructuralContainer(child)) { + ctx.graph.dropEdge(`${parent.key}->${child.key}`); + } +} +``` + +`removeChild` is called by the ujsx reconciler when a child is removed from a parent. In the `GraphologyHostConfig`, this removes the corresponding DAG edge. The child node itself is NOT removed from the graph — node removal is handled by `removeFromGraph` (see below). + +**Note**: The ujsx reconciler is not yet implemented. Currently, `removeChild` is defined but only called in tests. The `GraphologyHostConfig` is mount-only until the reconciler is available. + +### removeChildFromHost (node removal) + +```typescript +removeChildFromHost?(parent: GraphNode, child: GraphNode, ctx: GraphContext): void { + // Remove the child node from the graph + ctx.graph.dropNode(child.key); +} +``` + +When the reconciler removes a child entirely (not just moving it to a different parent), it calls `removeChildFromHost`. This removes the node and ALL attached edges (graphology cascading removal). This is important for cleanup when a template is re-rendered and a node no longer exists. + ### Cycle Detection -After rendering, the HostConfig checks for cycles using `graphology-dag.hasCycle()`. If a cycle is detected, the rendering throws `CircularDependencyError` with the cycle paths. +After rendering, the HostConfig checks for cycles using `graphology-dag.hasCycle()`. If a cycle is detected, the rendering throws `CycleError` with the cycle paths. This is the primary validation step: a valid workflow template must produce a valid DAG. Cycles in a template mean infinite loops in execution, which are always design errors. @@ -190,12 +233,21 @@ Each `WorkflowNode` holds: ```typescript interface ReactiveContext { - operationRegistry: OperationRegistry; - nodes: Map; // All nodes by key - statusSignals: Map>; // Status signals by key + operationRegistry: OperationRegistry; // Resolves operation names to specs + nodes: Map; // All nodes by key + statusSignals: Map>; // Status signals by key (owned by WorkflowReactiveRoot) + preconditions: Map>; // Precondition computeds by key (owned by WorkflowReactiveRoot) + blockedByFailure: Map>; // blockedByFailure computeds by key (owned by WorkflowReactiveRoot) + parentMap: Map; // Child → parent key mapping (for precondition computation) + siblingMap: Map; // Parent → children keys (for structural context) + results: Map>; // Operation output signals by key } ``` +The `ReactiveContext` is constructed during `ReactiveHostConfig` initialization. It receives the `operationRegistry` and empty maps. During `createInstance`, nodes and signals are registered in the context maps. After rendering completes, the context holds a complete index of the reactive workflow tree. + +**Important**: `statusSignals`, `preconditions`, and `blockedByFailure` are references to the `WorkflowReactiveRoot`'s maps. The `ReactiveHostConfig` does not own these signals — it looks them up during `createInstance` to wire `WorkflowNode` references. Disposal is the `WorkflowReactiveRoot`'s responsibility. + ### createInstance ```typescript @@ -271,6 +323,38 @@ The reactive engine then starts the call associated with the node (when `ready`) **Note**: Failure propagation follows dependency edges, not structural scope. A failed node only causes its downstream dependents (via DAG edges) to abort. Sibling branches in a `Parallel` group are independent and continue running. See [reactive-execution.md](reactive-execution.md) for the full failure propagation model. +### removeChild (ReactiveHostConfig) + +```typescript +removeChild(parent: WorkflowNode, child: WorkflowNode, ctx: ReactiveContext): void { + // Remove the dependency between parent and child + // The child's preconditions are recomputed automatically (reactive) + parent.children = parent.children.filter(c => c.key !== child.key); + // The child's preconditions and blockedByFailure computeds will re-evaluate + // because the predecessor list changes +} +``` + +`removeChild` in the reactive host removes the parent-child dependency. Because preconditions and `blockedByFailure` are `computed` values, they automatically re-evaluate when predecessor nodes are removed. + +```typescript +removeChildFromHost?(parent: WorkflowNode, child: WorkflowNode, ctx: ReactiveContext): void { + // Dispose the child's reactive state + ctx.nodes.delete(child.key); + ctx.statusSignals.delete(child.key); + ctx.preconditions.delete(child.key); + ctx.blockedByFailure.delete(child.key); + if (child.output) { + // Signal disposal is handled by WorkflowReactiveRoot.dispose() + // Here we just remove the reference from the context maps + } +} +``` + +For complete reactive teardown (`removeChildFromHost`), the node's signal references are removed from the context maps. The signals themselves (owned by `WorkflowReactiveRoot`) are disposed via `root.dispose()` which is the authoritative cleanup path. + +**Important**: Individual node disposal (removing a node mid-execution) is not fully supported until the ujsx reconciler is implemented. Currently, the reactive tree is built once and torn down as a whole via `WorkflowReactiveRoot.dispose()`. + ### Abort Cascading System-level abort (e.g., provider outage) aborts the entire workflow: @@ -291,6 +375,28 @@ This is reactive — when a parent node's status changes to `aborted`, the `effe The key insight: the same ujsx template renders to both targets: +``` + ujsx Template (UNode tree) + │ + ┌─────────┴─────────┐ + │ │ + GraphologyHostConfig ReactiveHostConfig + │ │ + ▼ ▼ + DirectedGraph (DAG) Reactive Signal Graph + ┌──────────────────┐ ┌──────────────────┐ + │ Nodes: operations │ │ Nodes: WorkflowNode│ + │ Edges: sequential │ │ signal│ + │ conditional │ │ computed │ + │ typed │ │ computed │ + └──────────────────┘ └──────────────────┘ + │ │ + ▼ ▼ + Structural Runtime Execution + Analysis & Status Tracking & + Validation Abort Propagation +``` + ```typescript const template = h(Sequential, {}, h(Operation, { name: "architect" }), diff --git a/docs/architecture/operation-graph.md b/docs/architecture/operation-graph.md index afccb52..d99f828 100644 --- a/docs/architecture/operation-graph.md +++ b/docs/architecture/operation-graph.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Operation Graph (Static) @@ -113,7 +113,7 @@ The `typeCompat` analysis function determines compatibility. Edges where compati The operation graph validates: -1. **Cycle detection** — throws `CircularDependencyError` if any cycle exists. Unlike taskgraph (which allows cycles and detects them via `hasCycles()`), flowgraph enforces acyclicity at construction time. A cycle in the operation graph means an operation's output feeds back into its own input, which is a design error. +1. **Cycle detection** — throws `CycleError` if any cycle exists. Unlike taskgraph (which allows cycles and detects them via `hasCycles()`), flowgraph enforces acyclicity at construction time. A cycle in the operation graph means an operation's output feeds back into its own input, which is a design error. 2. **Dangling references** — edges that reference operations not in the graph are structural errors. `addTypedEdge` throws `OperationNotFoundError` if either endpoint doesn't exist. diff --git a/docs/architecture/reactive-execution.md b/docs/architecture/reactive-execution.md index 8e20f6e..53ca5f7 100644 --- a/docs/architecture/reactive-execution.md +++ b/docs/architecture/reactive-execution.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Reactive Execution @@ -87,15 +87,49 @@ For each operation node in the DAG: The signal-based status lifecycle mirrors `CallStatus` with workflow-specific additions: ``` -idle → waiting → ready → running → completed - ↓ ↑ - failed │ - ↓ │ - (uncaught) → aborted ←──┘ - ↑ - (cascade from failed predecessor) - ↑ - skipped (conditional) + ┌──────┐ + ┌────────│ idle │────────────┐ + │ └──┬───┘ │ + │ │ predecessor │ (no predecessors — + │ │ starts running │ root node) + │ ▼ │ + │ ┌───────┐ │ + │ │waiting│ │ + │ └───┬───┘ │ + │ │ all preds │ + │ │ completed/ │ + │ ┌────┤ skipped │ + │ │ │ ▼ + │ │ │ ┌──────┐ + │ │ └──────────►│ready │ + │ │ └──┬───┘ + │ │ │ hub starts call + │ │ ▼ + │ │ ┌────────┐ + │ │ │running │──── ──── ──── ────► + │ │ └──┬──┬──┘ │ + │ │ │ │ │ + │ │ call │ │ call │ call + │ │ completed │ │ failed │ aborted + │ │ │ │ │ + │ │ ▼ ▼ ▼ + │ │ ┌───────────┐ ┌──────┐ ┌────────┐ + │ │ │ completed │ │failed│ │aborted │ + │ │ └───────────┘ └──────┘ └────────┘ + │ │ │ │ │ + │ │ │ │ (uncaught) │ + │ │ │ ▼ │ + │ │ │ cascades to all │ + │ │ │ downstream dependents │ + │ │ │ via blockedByFailure │ + │ │ │ │ + └──────┼──────────────┼────────────────────────────┘ + │ │ + │ ┌─────────┐│ + └───►│skipped ││ (Conditional branch + └─────────┘│ not taken) + │ + └─── all are terminal states ``` Full transition rules: @@ -442,6 +476,84 @@ abortAll(): void { This transitions all non-terminal, non-failed nodes to `aborted`. It's for cases where the entire workflow should stop, regardless of which branches are independent. +## Reactive Error Boundaries + +The reactive execution layer has three levels of error handling, each with distinct scope and semantics: + +### Level 1: Signal-level errors (per-node) + +When a call fails, the hub coordinator sets the node's status to `"failed"`: + +```typescript +status.value = "failed"; // Individual node failure +``` + +This triggers `blockedByFailure` in all downstream dependents, causing them to transition to `"aborted"`. The failure propagates through the signal graph reactively — no manual error handling is needed. + +### Level 2: Conditional error boundaries (branch-level) + +A `Conditional` node catches failures and redirects to an alternative branch: + +```typescript +h(Conditional, { + test: (results) => results["fetch-data"].status !== "failed", +}, + // then-branch (happy path) + h(Operation, { name: "process" }), + // else-branch (fallback) + h(Operation, { name: "handle-error" }), +) +``` + +When the `Conditional`'s `test` function evaluates to `false` (because a predecessor failed), the then-branch transitions to `skipped` and the else-branch becomes `ready`. Downstream nodes after the `Conditional` see it as `completed` — the failure is contained. + +This is the reactive equivalent of a `try/catch` block. Without a `Conditional`, failures cascade uncaught through dependency edges. + +### Level 3: Workflow abort (system-level) + +For failures that should cancel everything, the hub calls `workflowRoot.abortAll()`: + +```typescript +workflowRoot.abortAll(); // All non-terminal nodes → "aborted" +``` + +This is for system-level failures: provider outage, authentication failure, or any condition where the workflow cannot meaningfully continue regardless of branch independence. + +### WorkflowErrorBoundary (coordinator-level) + +The hub coordinator wraps the entire reactive execution in a `WorkflowErrorBoundary` — a conceptual boundary, not a signal: + +```typescript +try { + // Drive the workflow + for (const [nodeId, preconditions, blockedByFailure] of workflowRoot.nodes) { + effect(() => { /* start calls when ready */ }); + effect(() => { /* abort when blocked */ }); + } +} catch (error) { + // Unhandled reactive error — signal graph inconsistency + // This shouldn't happen in normal operation + workflowRoot.abortAll(); + prm.abortAll(pendingRequestIds); +} +``` + +The `WorkflowErrorBoundary` catches errors that escape the signal graph (e.g., a `computed` that throws, an `effect` that errors). These are catastrophic — the reactive state is inconsistent. The boundary's job is to: +1. Abort all calls via `prm.abortAll()` +2. Set all non-terminal nodes to `"aborted"` via `workflowRoot.abortAll()` +3. Dispose the reactive root +4. Log the error for diagnostics + +**Error propagation summary**: + +| Error type | Scope | Mechanism | Recovery | +|------------|-------|-----------|----------| +| Call failure | Single node | `status.value = "failed"` | Cascades to dependents via `blockedByFailure` | +| Caught by Conditional | Branch | `Conditional.test` evaluates against failed status | Redirect to else-branch, downstream sees `completed` | +| Uncaught cascade | Downstream chain | `blockedByFailure` effects | Downstream nodes transition to `aborted` | +| System failure | Entire workflow | `abortAll()` | All non-terminal nodes to `aborted` | +| Reactive error | Signal graph | `WorkflowErrorBoundary` catch | Abort everything, dispose, log | + ## Constraints - **Signals are in-memory** — `WorkflowReactiveRoot` state is not persisted. If the hub restarts, the reactive state is lost and must be reconstructed from call protocol events + template re-render. @@ -453,6 +565,94 @@ This transitions all non-terminal, non-failed nodes to `aborted`. It's for cases - **`failed` and `aborted` block preconditions** — a `failed` or `aborted` predecessor means the dependent's preconditions can never be met. The `blockedByFailure` effect transitions the dependent to `aborted`. - **`NodeStatus` and `CallStatus` share terminal states** — `running`, `completed`, `failed`, `aborted` map directly. `idle`, `waiting`, `ready`, `skipped` are workflow-specific additions. +## Lifecycle and Ownership + +The reactive execution pipeline has a clear creation order and ownership model: + +### Creation Order + +``` +1. Template (UNode tree) + ↓ GraphologyHostConfig +2. DAG (DirectedGraph) + ↓ WorkflowReactiveRoot constructor +3. Signal graph (statusMap, preconditions, blockedByFailure) + ↓ ReactiveHostConfig.render() +4. WorkflowNode tree (with effects registered) +``` + +1. **Template → DAG**: The consumer provides a template and renders it through `GraphologyHostConfig`. This produces a `DirectedGraph` stored in the `GraphContext`. + +2. **DAG → Signal graph**: The consumer creates a `WorkflowReactiveRoot` from the DAG. The constructor iterates over all operation nodes in the DAG and creates `signal`, `computed` (preconditions), and `computed` (blockedByFailure) for each. + +3. **Signal graph → WorkflowNode tree**: The consumer renders the template through `ReactiveHostConfig`. The `createInstance` call for each `Operation` node looks up the corresponding signal in the `ReactiveRoot` and wires the node's effects. + +### Ownership + +| Object | Owned by | Disposed by | +|--------|----------|-------------| +| Template (`UNode` tree) | Consumer | Consumer (not a reactive resource) | +| DAG (`DirectedGraph`) | GraphologyHostConfig's `GraphContext` | Consumer (static, no disposal needed) | +| `WorkflowReactiveRoot` | Consumer (typically the hub coordinator) | Consumer calls `root.dispose()` | +| Signal graph (statusMap, preconditions, etc.) | `WorkflowReactiveRoot` | `root.dispose()` clears all maps | +| `WorkflowNode` tree | `ReactiveContext` (created by ReactiveHostConfig) | Cleared when `ReactiveContext` is garbage collected | +| Effects | `WorkflowReactiveRoot.effectDisposers` | `root.dispose()` calls all disposers | + +**Key ownership rules**: +- `WorkflowReactiveRoot` owns the signal graph. It creates every `signal` and `computed`, tracks every `effect` disposer, and is responsible for cleaning them all up. +- `ReactiveHostConfig` is stateless after rendering. It creates `WorkflowNode` instances and registers effects, but the effects are tracked by `WorkflowReactiveRoot`, not by the HostConfig. +- The consumer owns the `WorkflowReactiveRoot` lifecycle. It creates it, drives execution by setting status values, and disposes it when done. + +### Disposal + +```typescript +// When workflow completes or is cancelled: +workflowRoot.dispose(); +``` + +`dispose()` performs the following in order: +1. Calls every `effect()` disposer, unsubscribing all reactive effects. +2. Clears `statusMap`, `preconditions`, and `blockedByFailure` maps, releasing signal references. +3. The `WorkflowNode` tree becomes inert — status signals no longer exist, so no updates propagate. + +**When to dispose**: +- Workflow completes successfully (all nodes `completed`) +- Workflow is aborted (consumer calls `abortAll()`, then `dispose()`) +- Template is being re-rendered (dispose the old root before creating a new one — until ujsx reconciler supports re-rendering) + +**What NOT to dispose**: +- The DAG (`DirectedGraph`) is not a reactive resource. It doesn't need disposal. +- The template (`UNode` tree) is plain data. It doesn't need disposal. + +### Interaction with ReactiveHostConfig + +The `ReactiveHostConfig` does NOT own the reactive state. It creates `WorkflowNode` instances during rendering, but these nodes reference signals that belong to `WorkflowReactiveRoot`. The rendering flow is: + +```typescript +// 1. Create ReactiveRoot from DAG +const workflowRoot = new WorkflowReactiveRoot(dag); + +// 2. Create ReactiveHostConfig with reference to ReactiveRoot's signals +const hostConfig = new ReactiveHostConfig(operationRegistry, workflowRoot); + +// 3. Render template +const root = createRoot(hostConfig, {}); +root.render(template); + +// 4. Drive execution (hub coordinator sets status values) +workflowRoot.statusMap.get("architect")!.value = "ready"; +// ... external code starts the call, eventually: +workflowRoot.statusMap.get("architect")!.value = "completed"; +// ... which triggers downstream preconditions + +// 5. Cleanup +workflowRoot.dispose(); +``` + +The `ReactiveContext` passed to `ReactiveHostConfig` includes a reference to `workflowRoot.statusSignals` so that `createInstance` can look up and wire signals for each node. The context does not own these signals — it's a lookup table. + +**Important**: `WorkflowNode.status` and `WorkflowReactiveRoot.statusMap.get(nodeId)` reference the **same** `Signal` instance. There is one signal per node, owned by `WorkflowReactiveRoot`, and both the `WorkflowNode` and the `statusMap` hold references to it. Setting `workflowRoot.statusMap.get("architect").value = "running"` and setting `workflowNode.status.value = "running"` (where `workflowNode.key === "architect"`) are equivalent operations on the same signal. Similarly, `WorkflowNode.preconditions` and `WorkflowReactiveRoot.preconditions.get(nodeId)` reference the **same** `Computed` instance. + ## Open Questions 1. **Should preconditions support OR logic?** Currently all predecessors must complete (AND logic). An `anyOf` predicate would allow "start this node as soon as any predecessor completes." This would require an edge attribute or node-level configuration. diff --git a/docs/architecture/schema.md b/docs/architecture/schema.md index 3891f31..b598915 100644 --- a/docs/architecture/schema.md +++ b/docs/architecture/schema.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Schema @@ -99,6 +99,53 @@ type NodeStatus = Static; **Precondition semantics**: A predecessor in `completed` or `skipped` status satisfies a dependent's preconditions. A predecessor in `failed` or `aborted` status does NOT satisfy preconditions — it blocks the dependent and triggers failure propagation (the dependent transitions to `aborted`). This enables partial success: independent parallel branches continue running even when one branch fails. +### CallResult + +The result of a completed call, used by `Conditional.test` and `Map.over` to access predecessor outputs: + +```typescript +interface CallResult { + status: NodeStatus; // Status of the call (completed, failed, aborted, skipped) + output: unknown; // Call output (if completed) + error?: { // Call error (if failed) + code: string; + message: string; + details?: unknown; + }; +} +``` + +`CallResult` is the value in the `results` map passed to `Conditional.test` and `Map.over` functions. It's derived from `CallNodeAttrs` but simplified for template use — it omits `requestId`, `operationId`, `identity`, and timestamps, preserving only what template logic needs. + +### OperationTypeEnum + +The type of an operation, determining its call semantics: + +```typescript +const OperationTypeEnum = Type.Union([ + Type.Literal("query"), // Read-only, idempotent + Type.Literal("mutation"), // Side effects, not idempotent + Type.Literal("subscription"), // Streaming, produces multiple results +]); +type OperationType = Static; +``` + +This enum is used in `OperationNodeAttrs.type` to classify operations by their call behavior. + +### CallEventMapValue + +`CallEventMapValue` is imported from `@alkdev/operations` (peer dependency). It represents a single call protocol event — the union type of all event types (`CallRequestedEvent | CallRespondedEvent | CallErrorEvent | CallAbortedEvent | CallCompletedEvent`). The full definition lives in `@alkdev/operations/src/call.ts`. + +Flowgraph's `fromCallEvents()` and `updateFromEvent()` accept this type directly. The mapping from `CallEventMapValue` to `CallNodeAttrs` is: + +| Event type | Action | +|------------|--------| +| `call.requested` | Add node with `status: "pending"`, add `triggered` edge if `parentRequestId` present | +| `call.responded` | Update node status to `completed`, set `output` and `completedAt` | +| `call.error` | Update node status to `failed`, set `error` and `completedAt` | +| `call.aborted` | Update node status to `aborted`, set `completedAt` | +| `call.completed` | Update node status to `completed`, set `completedAt` (if not already set) | + ### EdgeType The type of edge in a flowgraph. Matches the call graph storage schema's `edgeType` column: @@ -196,6 +243,17 @@ type OperationEdgeAttrs = Static; Type-compatibility edges carry a boolean `compatible` flag and optional detail. This allows the operation graph to include both compatible edges (green paths) and incompatible edges (red paths) for diagnostics. +**Edge type storage**: Operation graph edges always have `edgeType: "typed"` stored on the edge as a separate attribute alongside `OperationEdgeAttrs`. Graphology edges carry both the `OperationEdgeAttrs` (compatible, compatibilityDetail) and the `edgeType` field. The `edgeType` is not inside `OperationEdgeAttrs` because it's a universal edge classification that applies to all edge types across all graph modes (operation, call, template). The `OperationEdgeAttrs` schema only defines the mode-specific attributes. + +```typescript +// How operation graph edges are stored in graphology: +{ + edgeType: "typed", // Universal classification (stored alongside attrs) + compatible: true, // OperationEdgeAttrs field + compatibilityDetail: "..." // OperationEdgeAttrs field +} +``` + **Naming note**: Previously named `TypedEdgeAttrs`. Renamed to follow the `{GraphType}EdgeAttrs` pattern used by `CallEdgeAttrs` and `TemplateEdgeAttrs`. ### TriggeredEdgeAttrs (Call Graph) @@ -236,6 +294,18 @@ type TemplateEdgeAttrs = Static; Template edges carry an `edgeType` to distinguish sequential flow from conditional branching. Conditional edges optionally store a `condition` that determines whether the target node executes. +### TemplateNodeAttrs (Workflow Templates) + +Template DAGs use `OperationNodeAttrs` for their operation nodes — the template doesn't need a separate node type because every node in a template DAG corresponds to an operation invocation. The template's structural information (`Sequential`, `Parallel`, `Conditional`, `Map`) is expressed through edges, not through special node types. + +```typescript +// Template DAGs use OperationNodeAttrs for operation nodes +type TemplateNodeAttrs = OperationNodeAttrs; +// This alias makes the intent explicit: a template node represents an operation invocation +``` + +The separation between `OperationNodeAttrs` and `TemplateNodeAttrs` is a type alias for clarity. In the template context, each node carries the same attributes as an operation node (name, namespace, type, input/output schemas), but with template-specific edges (sequential, conditional) rather than type-compatibility edges (typed). + ## SerializedGraph Factory Following the taskgraph pattern, a generic factory for graphology native JSON format: diff --git a/docs/architecture/workflow-templates.md b/docs/architecture/workflow-templates.md index 1ec4def..aa2f2e1 100644 --- a/docs/architecture/workflow-templates.md +++ b/docs/architecture/workflow-templates.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-05-19 +last_updated: 2026-05-20 --- # Workflow Templates @@ -13,7 +13,7 @@ Workflow templates are ujsx trees that define reusable call patterns. Instead of ```typescript import { h, createRoot } from "@alkdev/ujsx"; -import { Operation, Sequential, Parallel, Conditional } from "@alkdev/flowgraph/component"; +import { Operation, Sequential, Parallel, Conditional, Map } from "@alkdev/flowgraph/component"; import { GraphologyHostConfig } from "@alkdev/flowgraph/host/graphology"; const sddPipeline = h(Sequential, {}, @@ -38,7 +38,7 @@ The template is a `UNode` tree — a plain data structure that can be: - **Rendered to a graphology DAG** via the `GraphologyHostConfig` for structural analysis - **Rendered to a reactive execution engine** via the `ReactiveHostConfig` for runtime status tracking -This is the same `UNode` tree that ujsx defines, with flowgraph-specific component functions (`Operation`, `Sequential`, `Parallel`, `Conditional`) that produce `UElement` nodes with workflow-specific props and meaning. +This is the same `UNode` tree that ujsx defines, with flowgraph-specific component functions (`Operation`, `Sequential`, `Parallel`, `Conditional`, `Map`) that produce `UElement` nodes with workflow-specific props and meaning. ## Why ujsx as Template IR @@ -128,7 +128,82 @@ const Conditional: UComponent<{ When rendered to a graphology DAG, `Conditional` creates an edge with `edgeType: "conditional"` and `condition` attribute. When rendered to the reactive engine, the condition is evaluated as a `computed` that depends on the referenced step's status and output. -If the test evaluates to `false`, the branch is marked `skipped` in `NodeStatus`. +If the test evaluates to `false` and no `else` branch is provided, the branch nodes transition to `skipped` in `NodeStatus`. + +#### Else-branch behavior + +When the `else` prop is provided, the `Conditional` renders two subgraphs: + +**DAG rendering (GraphologyHostConfig)**: +- The `then` branch (child) renders with an edge from the conditional's predecessor to the first child, with `edgeType: "conditional"` and `condition: `. +- The `else` branch renders as a separate subgraph with `edgeType: "conditional"` and `condition: `. The negated condition is derived automatically. +- Both branches share the same predecessor — the `Conditional` node's structural position in the template determines the common starting point. + +**Reactive rendering (ReactiveHostConfig)**: +- When `test` evaluates to `true`: `then`-branch nodes become `ready` (preconditions met). `else`-branch nodes transition to `skipped`. Their `preconditions` are satisfied by the `skipped` state — downstream nodes see the `Conditional` as completed regardless of which branch was taken. +- When `test` evaluates to `false`: `else`-branch nodes become `ready`. `then`-branch nodes transition to `skipped`. Downstream nodes after the `Conditional` see all branches as resolved. +- When no `else` prop is provided: the `false` branch simply doesn't exist. Nodes after the `Conditional` that depend on it still see it as `completed` because the `Conditional` itself resolves regardless of which path is taken. + +This means a `Conditional` with an `else` branch acts as a **complete error boundary** — downstream nodes are insulated from the branch choice. The `Conditional` is `completed` whether the `then` or `else` branch executed. + +### `` + +Represents mapping over an array — creates one child instance per array item: + +```typescript +const Map: UComponent<{ + over: Signal | unknown[] | ((results: Record) => unknown[]); + // Static array, signal, or function that resolves against predecessor results + as: string; // Variable name for each item + children: UNode; // Template rendered per item +}>; +``` + +The `` component dynamically replicates its child template for each element in the `over` array. Each replica gets the current element bound to the variable named by `as`. + +**DAG rendering (GraphologyHostConfig)**: +- For each item in `over`, renders a copy of the child template as a node. +- Each mapped node has a `sequential` edge from the `Map`'s predecessor (all mapped nodes start at the same point, like `Parallel`). +- Mapped nodes are named with a composite key: `${parentKey}.${as}[${index}]`. For example, `` with 3 items creates nodes `item[0]`, `item[1]`, `item[2]`. +- The `Map` container itself is transparent in the graph (no node for the container). + +**Reactive rendering (ReactiveHostConfig)**: +- For each item in `over`, creates a `WorkflowNode` with its own `signal` and `computed` preconditions. +- All mapped nodes' preconditions are identical: the `Map`'s predecessor must be `completed` (same as `Parallel`). +- Each mapped node's `output` signal holds the result of its corresponding call. +- The `Map` result is available as an aggregated signal containing all mapped nodes' outputs. + +**Example**: + +```typescript +h(Sequential, {}, + h(Operation, { name: "fetch-items" }), + h(Map, { + over: (results) => results["fetch-items"].output.items, + as: "item" + }, + h(Operation, { name: "process-item", input: (results, { item }) => item }), + ), +) +``` + +This creates a `Sequential` where `process-item` is called once per item returned by `fetch-items`. Each call gets its corresponding item as input. + +**Edge type**: Mapped children use the same `TemplateEdgeAttrs` as `Parallel` children (no `sequential` edges between siblings). The `Map` component is structurally equivalent to a `Parallel` group where the children are dynamically generated from an array. + +**Aggregate completion semantics**: A `Map` node's status follows "worst-case" semantics: + +- If **all** mapped nodes reach a satisfying terminal state (`completed` or `skipped`), the `Map` is considered `completed`. +- If **any** mapped node reaches `failed`, the `Map` is considered `failed` (unless caught by a `Conditional`). +- If **any** mapped node reaches `aborted`, the `Map` is considered `aborted`. +- Downstream nodes whose preconditions include the `Map` will see `blockedByFailure = true` if the `Map` has any `failed` or `aborted` children. + +This means a `Map` with partial failure (some nodes succeeded, one failed) propagates as a failure to downstream dependents. If partial success is needed, the template author should use a `Conditional` to handle the failure case, or process the `Map` results individually rather than treating the `Map` as a single dependency. + +**Reactive behavior for mapped nodes**: +- All mapped nodes become `ready` simultaneously when the `Map`'s predecessor completes (parallel start). +- If any mapped node fails, only that node transitions to `failed`. Other mapped nodes continue independently (failure follows dependency edges, not structural scope, consistent with `Parallel` behavior). +- Mapped nodes participate in failure propagation like any other node: downstream dependents see `blockedByFailure` if a mapped node fails and they depend on it. ## Template → DAG Conversion @@ -259,6 +334,36 @@ Validation checks: Validation returns an array of `ValidationError` objects (never throws). See [analysis.md](analysis.md) for the full validation algorithm. +## Composition Rules + +Not all component combinations are valid. The following rules govern which components can appear as children of which: + +| Parent | Valid children | Notes | +|--------|---------------|-------| +| `Sequential` | `Operation`, `Sequential`, `Parallel`, `Conditional`, `Map` | Children execute in order | +| `Parallel` | `Operation`, `Sequential`, `Parallel`, `Conditional`, `Map` | Children execute concurrently | +| `Conditional` (then) | `Operation`, `Sequential`, `Parallel`, `Map` | Single child or wrapped in structural container | +| `Conditional` (else) | `Operation`, `Sequential`, `Parallel`, `Map` | Single child or wrapped in structural container | +| `Map` | `Operation`, `Sequential`, `Parallel`, `Conditional` | Template rendered per item | + +### Rules + +1. **`Operation` has no children** — an `Operation` is a leaf node. Nesting inside `Operation` is a template validation error. + +2. **`Conditional` takes a single then-child via children, and optional else via `else` prop** — the `children` of `Conditional` are the then-branch. The `else` prop is the alternative branch. Both branches can be single `Operation` nodes or structural containers (`Sequential`, `Parallel`, `Map`). + +3. **`Conditional.test` cannot reference an `Operation` inside the Conditional** — the test evaluates results from predecessor operations, not from the conditional branch itself. This would create a circular dependency. + +4. **`Map.over` must be a serializable expression or signal** — the array can be a static value, a signal, or a function that receives results from predecessor operations. Function-valued `over` props don't survive JSON round-trips (same limitation as `Conditional.test`). + +5. **`Sequential` with one child is valid but degenerate** — it produces no edges (no sequential ordering needed). A single-child `Sequential` is equivalent to the child alone. + +6. **`Parallel` with one child is valid but degenerate** — it produces no edges (no concurrency needed). A single-child `Parallel` is equivalent to the child alone. + +7. **Nesting is allowed to any depth** — `Sequential` inside `Parallel` inside `Sequential` is valid. The DAG flattens nesting into edges between leaf `Operation` nodes. + +8. **Template root must be a structural container** — the root element must be `Sequential`, `Parallel`, or `Map`. A bare `Operation` as root is technically valid but produces a single-node DAG with no edges. + ## Constraints - **Templates are ujsx trees** — no custom format, no parser, no compiler. Components are `UComponent` functions that produce `UElement` nodes. @@ -272,7 +377,7 @@ Validation returns an array of `ValidationError` objects (never throws). See [an 1. **Should `Sequential` and `Parallel` be transparent in the graph?** Currently they produce edges, not nodes. An alternative is to create "virtual" grouping nodes (like a "parallel gateway" in BPMN). This would make the graph structure richer but adds complexity. -2. **Should templates support loops?** A `` component that iterates over an array and produces a child for each element. This would enable dynamic workflows where the number of parallel calls isn't known at template definition time. +2. ~~**Should templates support loops?**~~ **Resolved**: The `` component provides array iteration — one child per array element. It does NOT support general loops (while, do-while). For repeated execution with conditional exit, use `Conditional` inside a `Sequential` group. General-purpose loops with arbitrary termination conditions are not supported because they would require cycle-supporting templates, which conflicts with the DAG-only invariant. 3. **Should templates support `depends_on` edges explicitly?** Currently dependencies are inferred from structure (sequential implies dependency). An explicit `` component would make data dependencies visible in the template without relying on sequential ordering. diff --git a/docs/reviews/001-architecture-gap-analysis.md b/docs/reviews/001-architecture-gap-analysis.md index 2729ea5..ba46673 100644 --- a/docs/reviews/001-architecture-gap-analysis.md +++ b/docs/reviews/001-architecture-gap-analysis.md @@ -274,29 +274,29 @@ When addressing these issues, use this checklist to track progress: - [x] C-02: Add `CallEdgeAttrs` type alias to schema.md - [x] C-03: Resolve `OperationEdgeAttrs` vs `TypedEdgeAttrs` naming (renamed `TypedEdgeAttrs` → `OperationEdgeAttrs`) - [x] C-04: Specify failure propagation semantics in reactive-execution.md (failure follows dependency edges, not structural scope; Conditionals as error boundaries; blockedByFailure computed; partial success for parallel branches) -- [ ] C-05: Create FlowGraph public API document -- [ ] C-06: Document `` component in workflow-templates.md -- [ ] C-07: Specify `Conditional` else-branch behavior -- [ ] C-08: Specify `WorkflowReactiveRoot` ↔ `ReactiveHostConfig` ownership -- [ ] C-09: Create consumer integration guide +- [x] C-05: Create FlowGraph public API document +- [x] C-06: Document `` component in workflow-templates.md +- [x] C-07: Specify `Conditional` else-branch behavior +- [x] C-08: Specify `WorkflowReactiveRoot` ↔ `ReactiveHostConfig` ownership +- [x] C-09: Create consumer integration guide - [x] W-01: Standardize `prerequisites` vs `preconditions` terminology (prerequisites=structural/graph, preconditions=reactive/computed) -- [ ] W-02: Add reactive error boundary semantics -- [ ] W-03: Complete `ReactiveContext` interface definition -- [ ] W-04: Add template composition rules -- [ ] W-05: Document `removeChild` for both HostConfigs -- [ ] W-06: Document signal/effect disposal lifecycle -- [ ] W-07: Consider ADR-004 for "no schema version" -- [ ] W-08: Specify type compatibility depth +- [x] W-02: Add reactive error boundary semantics +- [x] W-03: Complete `ReactiveContext` interface definition +- [x] W-04: Add template composition rules +- [x] W-05: Document `removeChild` for both HostConfigs +- [x] W-06: Document signal/effect disposal lifecycle +- [x] W-07: ADR-004 for "no schema version" decision +- [x] W-08: Specify type compatibility depth/contract (added compatibility contract, depth rules, and result semantics) - [x] W-09: Update ADR statuses to Accepted - [x] W-10: Clarify call graph mutation API (clarified `addCall` creates `triggered` edges automatically, `addDependency` creates `depends_on` edges) -- [ ] W-11: Add performance characteristics section +- [x] W-11: Add performance characteristics section - [x] W-12: Standardize edge attribute naming pattern (now `{GraphType}EdgeAttrs`: `OperationEdgeAttrs`, `CallEdgeAttrs`, `TemplateEdgeAttrs`) -- [ ] S-01: Getting Started walkthrough document -- [ ] S-02: Flow diagrams for template rendering pipeline -- [ ] S-03: Node status state machine diagram -- [ ] S-04: Testing strategy documentation -- [ ] S-05: Additional ADRs for inline decisions -- [ ] S-06: Validate source structure cross-references +- [x] S-01: Getting Started walkthrough document (merged into consumer-integration.md) +- [x] S-02: Flow diagrams for template rendering pipeline (added to host-configs.md) +- [x] S-03: Node status state machine diagram (added to reactive-execution.md) +- [x] S-04: Testing strategy documentation (added to build-distribution.md) +- [ ] S-05: Additional ADRs for inline decisions (deferred — decisions documented inline in existing docs) +- [x] S-06: Validate source structure cross-references (added map.ts to source structure, updated exports map, verified cross-references) ---