Set up project structure, source files, and architecture docs

- Copy core source from alkhub_ts/packages/core/pubsub/ with import path fixups
  (typed_event_target.ts → types.ts, .ts → .js extensions)
- Make PubSubPublishArgsByKey exported (was private type, needed by barrel)
- Add package.json with sub-path exports and optional peer deps (ioredis)
- Add tsup.config.ts with multi-entry + splitting for tree-shaking
- Add tsconfig.json, vitest.config.ts, .gitignore
- Add AGENTS.md with project conventions and adapter checklist
- Add architecture docs following taskgraph/alkhub pattern:
  docs/architecture/README.md, api-surface.md, event-targets.md,
  iroh-transport.md, build-distribution.md
- Add ADRs: 001-graphql-yoga-fork, 002-tree-shake-pattern
- Copy migration research doc to docs/research/migration.md
- Dual-license MIT OR Apache-2.0 (matching taskgraph)
This commit is contained in:
2026-04-30 10:20:41 +00:00
parent a5d128629e
commit 8c025c3433
23 changed files with 4747 additions and 0 deletions

18
docs/architecture.md Normal file
View File

@@ -0,0 +1,18 @@
# Architecture
> **This document has been decomposed into modular documents.** See [docs/architecture/](architecture/) for the current architecture specification.
| Document | Content |
|----------|---------|
| [architecture/README.md](architecture/README.md) | Overview, why this exists, interface contract, consumer context |
| [architecture/api-surface.md](architecture/api-surface.md) | createPubSub, PubSub types, operators |
| [architecture/event-targets.md](architecture/event-targets.md) | In-process, Redis, WebSocket adapters |
| [architecture/iroh-transport.md](architecture/iroh-transport.md) | Iroh P2P QUIC transport, framing, identity, hub/spoke |
| [architecture/build-distribution.md](architecture/build-distribution.md) | Dependencies, project structure, tree-shaking, sub-path exports |
### Design Decisions
| ADR | Decision |
|-----|----------|
| [001](architecture/decisions/001-graphql-yoga-fork.md) | Fork graphql-yoga pubsub rather than depend on it |
| [002](architecture/decisions/002-tree-shake-pattern.md) | Sub-path exports + peer deps for adapter isolation |

View File

@@ -0,0 +1,94 @@
---
status: draft
last_updated: 2026-04-30
---
# @alkdev/pubsub Architecture
Type-safe publish/subscribe with pluggable event target adapters. The core (`createPubSub` + `TypedEventTarget` + operators) has no transport dependency. Each adapter (Redis, WebSocket, Iroh) is an isolated module that only imports its own peer dependency.
## Why This Exists
Extracted from `@alkdev/alkhub_ts/packages/core/pubsub/`, which itself was adapted from `@graphql-yoga/subscription` and `@graphql-yoga/typed-event-target`. The pubsub module was already self-contained within alkhub — zero cross-module imports from operations, config, logger, or MCP. Extracting it into a standalone package:
1. **Reduces coupling** — alkhub depends on pubsub, not the other way around
2. **Enables reuse** — multiple alkhub packages can share the same pubsub instance
3. **Isolates peer deps** — Redis and Iroh are heavy native dependencies; consumers that don't need them shouldn't carry them
4. **Matches established pattern**`@alkdev/taskgraph` and `@alkdev/typemap` already use the standalone-package pattern
## Core Principle
**The TypedEventTarget interface is the contract.** All transports implement the same `addEventListener` / `dispatchEvent` / `removeEventListener` surface. `createPubSub` doesn't know or care which transport is in use — it just dispatches events to whatever `TypedEventTarget` it was given.
This means swapping from in-process to Redis to WebSocket to Iroh is a one-line config change:
```ts
const pubsub = createPubSub<MyEventMap>({
eventTarget: createRedisEventTarget({ publishClient, subscribeClient }),
});
```
## What This Package Provides
- **Core** — `createPubSub`, `TypedEventTarget`, `TypedEvent`, topic scoping, `filter`/`map`/`pipe` operators
- **Adapters** (each is a peer-dep island, importable via sub-path export):
- In-process (default `EventTarget`, no adapter needed)
- Redis (`@alkdev/pubsub/event-target-redis`, peer dep: `ioredis`)
- WebSocket (future: `@alkdev/pubsub/event-target-websocket`)
- Iroh (future: `@alkdev/pubsub/event-target-iroh`, peer dep: `@rayhanadev/iroh`)
## Consumer Context
### alkhub (hub-spoke coordinator)
The hub uses pubsub for event routing between operations, runners, and the SSE interface. The event map is the call protocol — typed JSON events (`call.requested`, `call.responded`, `session.status`, etc.). Transport choice depends on deployment:
| Deployment | Transport |
|------------|-----------|
| Single-process hub | In-process (default) |
| Hub + worker processes | Redis |
| Hub + remote spokes | WebSocket or Iroh |
### Future: standalone spoke SDK
Spokes will import `@alkdev/pubsub` directly to create their event target (WebSocket or Iroh) and wire it into `createPubSub`. The call protocol types live in a separate `@alkdev/call-protocol` package (not yet extracted).
## Threat Model
- **Fork provenance** — core pubsub and typed event target are adapted from graphql-yoga (MIT). All original copyright notices are preserved in file headers. See [ADR-001](decisions/001-graphql-yoga-fork.md).
- **Peer dep isolation** — Redis and Iroh are optional peer dependencies. A consumer that only needs in-process transport installs zero extra packages. A consumer using Redis but not Iroh installs `ioredis` only.
- **Type-only imports** — `event-target-redis.ts` imports `ioredis` types only at compile time. At runtime, the consumer must provide the actual `Redis`/`Cluster` instances.
## Architecture Documents
| Document | Content |
|----------|---------|
| [api-surface.md](api-surface.md) | createPubSub factory, PubSub types, operators, TypedEventTarget types |
| [event-targets.md](event-targets.md) | In-process, Redis, WebSocket adapters — interface, configuration, limitations |
| [iroh-transport.md](iroh-transport.md) | Iroh P2P QUIC transport — protocol, framing, identity, hub/spoke sides, reconnection |
| [build-distribution.md](build-distribution.md) | Dependencies, project structure, tree-shaking, sub-path exports, targets |
## Document Lifecycle
Architecture documents use YAML frontmatter with `status` and `last_updated` fields:
```yaml
---
status: draft | stable | deprecated
last_updated: YYYY-MM-DD
---
```
| Status | Meaning | Transitions |
|--------|---------|-------------|
| `draft` | Under active development. Content may change. | → `stable` when implementation is complete and tests verify API contract. |
| `stable` | API contracts are locked. Changes require review cycle. | → `deprecated` when superseded. |
| `deprecated` | Superseded. Kept for reference. | Removed when no longer referenced. |
## References
- Source: `@alkdev/alkhub_ts/packages/core/pubsub/`
- Upstream: `@graphql-yoga/subscription` and `@graphql-yoga/typed-event-target` (MIT)
- alkhub pubsub-redis doc: `@alkdev/alkhub_ts/docs/architecture/pubsub-redis.md`
- alkhub spoke-runner doc: `@alkdev/alkhub_ts/docs/architecture/spoke-runner.md`
- Migration research: `docs/research/migration.md`

View File

@@ -0,0 +1,101 @@
---
status: draft
last_updated: 2026-04-30
---
# API Surface
Core pubsub creation, types, and operators. No transport dependencies.
## `createPubSub`
```ts
function createPubSub<TPubSubPublishArgsByKey extends PubSubPublishArgsByKey>(
config?: PubSubConfig<TPubSubPublishArgsByKey>,
): PubSub<TPubSubPublishArgsByKey>;
```
Factory function. Accepts an optional `eventTarget` config. If none is provided, uses `new EventTarget()` (in-process).
### Topic Scoping
Topics can be scoped with an id:
- `pubsub.publish("session.status", projectId, payload)` → dispatches to topic `session.status:{projectId}`
- `pubsub.subscribe("session.status", projectId)` → subscribes to topic `session.status:{projectId}` only
- `pubsub.publish("session.status", payload)` → dispatches to topic `session.status` (unscoped)
- `pubsub.subscribe("session.status")` → subscribes to topic `session.status` (unscoped)
The topic string is either the routing key directly (unscoped) or `{routingKey}:{id}` (scoped). This maps naturally to Redis channel naming and WebSocket message routing.
### `PubSubPublishArgsByKey`
The type parameter that defines the event map:
```ts
type PubSubPublishArgsByKey = {
[key: string]: [] | [unknown] | [number | string, unknown];
};
```
- `[]` — event with no payload (trigger only)
- `[payload]` — unscoped event with payload
- `[id, payload]` — scoped event with id and payload
### `PubSub.subscribe()`
Returns a `Repeater<unknown>` (async iterable). Consumers iterate with `for await`:
```ts
for await (const payload of pubsub.subscribe("session.status")) {
// handle payload
}
```
The `Repeater` automatically cleans up its `addEventListener` when the consumer breaks out of the loop (the `stop` promise resolves).
## Types
| Export | Source | Description |
|--------|--------|-------------|
| `TypedEvent<TType, TDetail>` | `types.ts` | Event with typed `type` and `detail`. Omits `CustomEvent`'s untyped fields. |
| `TypedEventTarget<TEvent>` | `types.ts` | Extends `EventTarget` with typed `addEventListener`, `dispatchEvent`, `removeEventListener`. |
| `TypedEventListener<TEvent>` | `types.ts` | `(evt: TEvent) => void` |
| `TypedEventListenerObject<TEvent>` | `types.ts` | `{ handleEvent(object: TEvent): void }` |
| `TypedEventListenerOrEventListenerObject<TEvent>` | `types.ts` | Union of the above |
| `PubSub<TPubSubPublishArgsByKey>` | `create_pubsub.ts` | `{ publish, subscribe }` |
| `PubSubConfig<TPubSubPublishArgsByKey>` | `create_pubsub.ts` | `{ eventTarget?: PubSubEventTarget }` |
| `PubSubEvent<TPubSubPublishArgsByKey, TKey>` | `create_pubsub.ts` | Derived `TypedEvent` for a specific event key |
| `PubSubEventTarget<TPubSubPublishArgsByKey>` | `create_pubsub.ts` | `TypedEventTarget<PubSubEvent<...>>` |
## Operators
All operators return `Repeater` instances and work with any async iterable.
### `filter`
```ts
function filter<T>(filterFn: (value: T) => Promise<boolean> | boolean): (source: AsyncIterable<T>) => Repeater<T>;
```
Type-narrowing overload available: `filter<T, U extends T>(fn: (input: T) => input is U)`.
### `map`
```ts
function map<T, O>(mapper: (input: T) => Promise<O> | O): (source: AsyncIterable<T>) => Repeater<O>;
```
### `pipe`
```ts
function pipe<A, B>(a: A, ab: (a: A) => B): B;
function pipe<A, B, C>(a: A, ab: (a: A) => B, bc: (b: B) => C): C;
// up to 5 arguments
```
Compose operators: `pipe(pubsub.subscribe("myEvent"), filter(isRelevant), map(transform))`
## Attribution
`createPubSub` and operators are adapted from `@graphql-yoga/subscription` (MIT). `TypedEventTarget` types are adapted from `@graphql-yoga/typed-event-target` (MIT). See file headers for full license text.

View File

@@ -0,0 +1,113 @@
---
status: draft
last_updated: 2026-04-30
---
# Build & Distribution
Dependencies, project structure, tree-shaking, sub-path exports, and build targets.
## Dependencies
| Package | Type | Purpose |
|---------|------|---------|
| `@repeaterjs/repeater` | direct | Small (~3KB). Core async iterable primitive for `subscribe()`. |
| `ioredis` | peer (optional) | Redis client. Only imported by `event-target-redis.ts`. Type-only import at compile time. |
| `@rayhanadev/iroh` | peer (optional, future) | Iroh NAPI-RS binding. Only imported by `event-target-iroh.ts`. |
No other external dependencies. No logger dependency.
## Project Structure
```
@alkdev/pubsub/
src/
index.ts # Barrel: re-exports core API
types.ts # TypedEvent, TypedEventTarget, etc.
create_pubsub.ts # createPubSub factory
operators.ts # filter, map, pipe
event-target-redis.ts # createRedisEventTarget (peer dep: ioredis)
# Future adapters (each is its own entry point + peer dep island):
# event-target-websocket.ts # peer dep: none (web standard)
# event-target-iroh.ts # peer dep: @rayhanadev/iroh
test/
create_pubsub.test.ts
operators.test.ts
event-target-redis.test.ts
# event-target-websocket.test.ts
# event-target-iroh.test.ts
docs/
architecture.md
architecture/
research/
package.json
tsconfig.json
tsup.config.ts
vitest.config.ts
```
## Sub-Path Exports
We use explicit sub-path exports rather than barrel-only + tree-shaking. Each adapter is importable by its own path:
```json
{
"exports": {
".": { ... },
"./event-target-redis": { ... },
"./event-target-websocket": { ... },
"./event-target-iroh": { ... }
}
}
```
### Why Sub-Path Exports
- **Explicit** — doesn't rely on bundler tree-shaking behavior
- **Peer dep isolation** — `import from '@alkdev/pubsub/event-target-redis'` makes the dependency on ioredis explicit at the import site
- **Consistent with typemap pattern** — typemap's peer deps (zod, valibot, typebox) are each their own module; sub-path exports make this explicit at the package boundary
Sub-path entries are added as adapters are implemented. The barrel `index.ts` also re-exports everything for convenience — consumers who want tree-shaking can import from the barrel and rely on their bundler.
## Peer Dependencies
| Peer Dep | Required By | Optional |
|----------|-------------|----------|
| `ioredis@^5.0.0` | `event-target-redis` | Yes |
| `@rayhanadev/iroh` | `event-target-iroh` (future) | Yes |
Optional peer deps means `npm install @alkdev/pubsub` does NOT install ioredis or iroh. Consumers opt in by installing the peer dep and importing from the sub-path.
## Build
- **Tool**: `tsup` — produces dual ESM + CJS with declarations automatically
- **Entry points**: `src/index.ts`, `src/event-target-redis.ts`, plus future adapters
- **Format**: ESM + CJS
- **Target**: `es2022`
- **Splitting**: enabled (tsup code splitting for shared chunks)
```ts
// tsup.config.ts
import { defineConfig } from 'tsup';
export default defineConfig({
entry: ['src/index.ts', 'src/event-target-redis.ts'],
format: ['esm', 'cjs'],
dts: true,
sourcemap: true,
clean: true,
splitting: true,
target: 'es2022',
});
```
## Testing
- **Runner**: `vitest` — matches taskgraph, natural fit with tsup/Node build pipeline
- **Config**: `vitest.config.ts` with `globals: true`
## Targets
- **Publish**: npm (`@alkdev/pubsub`)
- **Runtime**: Node 18+, Deno, Bun — pure JS (except iroh adapter which requires NAPI-RS)
- **Deno compatibility**: Source is standard TypeScript with no Deno-specific APIs. Deno can import from npm or JSR.

View File

@@ -0,0 +1,28 @@
# ADR-001: Fork graphql-yoga pubsub
**Status**: Accepted
**Date**: 2026-04-30
## Context
`createPubSub`, `TypedEventTarget`, and operators are adapted from `@graphql-yoga/subscription` and `@graphql-yoga/typed-event-target` (MIT). We carried these into alkhub with modifications (native CustomEvent, our TypedEventTarget types, removed tslib). Now we're extracting to a standalone package.
## Decision
Fork (continue carrying adapted code) rather than depend on graphql-yoga packages directly.
## Rationale
1. **Different evolution path** — graphql-yoga's pubsub is tailored for GraphQL subscriptions. Our use case is general-purpose event routing with multiple transports. The APIs will diverge further as we add WebSocket and Iroh adapters.
2. **Dependency reduction** — graphql-yoga's subscription package pulls in `@whatwg-node/events` and `tslib`. We don't need either — we use native `CustomEvent` and no tslib runtime.
3. **Control over types** — graphql-yoga's `TypedEventTarget` uses their own event type hierarchy. We use a simpler one that maps directly to `CustomEvent`. Maintaining our own types avoids version coupling.
4. **Already forked** — the code in alkhub already diverged from the original. The license headers are in place. A clean extraction doesn't change the provenance story.
## Consequences
- Must preserve MIT license headers in all forked files
- Must update attribution if we make significant changes beyond the original fork scope
- No automatic updates from graphql-yoga — we carry our own maintenance burden

View File

@@ -0,0 +1,36 @@
# ADR-002: Sub-Path Exports + Peer Deps for Adapter Isolation
**Status**: Accepted
**Date**: 2026-04-30
## Context
Each event target adapter has different peer dependencies (ioredis for Redis, @rayhanadev/iroh for Iroh). Consumers that only use in-process or one adapter should not be forced to install peer deps for adapters they don't use. Two approaches:
1. **Barrel + tree-shaking** — single entry point, rely on bundler to drop unused adapters
2. **Sub-path exports** — explicit per-adapter entry points in `package.json` exports map
Typemap uses barrel-only. Taskgraph uses plain barrel with no sub-path exports.
## Decision
Use sub-path exports with optional peer dependencies.
## Rationale
1. **Explicit dependency declaration**`import { createRedisEventTarget } from '@alkdev/pubsub/event-target-redis'` makes it clear at the import site that this module needs ioredis. A barrel import doesn't.
2. **No bundler reliance** — sub-path exports don't depend on the consumer's bundler correctly tree-shaking. Not all consumers use bundlers (Deno, Node with `--experimental-strip-types`).
3. **peerDependenciesMeta** — npm treats optional peer deps as install warnings, not errors. `npm install @alkdev/pubsub` installs only the core. `npm install @alkdev/pubsub ioredis` gets Redis support.
4. **Consistent with typemap's philosophy** — typemap's peer deps (zod, valibot, typebox) are each their own module island. Sub-path exports make this explicit at the package boundary. We're adding the package.json entry points that typemap doesn't have.
5. **Incremental** — adapters can be added one at a time. Each new adapter adds one entry to the exports map and one entry point to tsup.
## Consequences
- More entries in `package.json` exports — maintenance burden scales with adapter count
- Both barrel and sub-path work — barrel re-exports everything for convenience, sub-path for explicitness
- tsup must list each adapter as a separate entry point
- Consumer docs should recommend sub-path imports for adapter-specific code

View File

@@ -0,0 +1,111 @@
---
status: draft
last_updated: 2026-04-30
---
# Event Target Adapters
In-process, Redis, and WebSocket event targets. All implement `TypedEventTarget<TEvent>`.
## Interface Contract
Every adapter must implement:
| Method | Behavior |
|--------|----------|
| `addEventListener(type, callback)` | Register listener for event type. Callback receives `CustomEvent` with typed `detail`. |
| `dispatchEvent(event)` | Send/dispatch event. Returns `boolean` (always `true` for non-cancelable events). |
| `removeEventListener(type, callback)` | Unregister listener. Clean up underlying subscription when no listeners remain for a topic. |
## In-Process (Default)
No adapter needed. `createPubSub` uses `new EventTarget()` by default. This works for single-process deployments where all pubsub participants share the same memory.
No explicit `InProcessEventTarget` class — the web standard `EventTarget` already implements the interface. Could be formalized later if a name makes the API clearer, but `new EventTarget()` is already the standard.
## Redis
**Import**: `@alkdev/pubsub/event-target-redis`
**Peer dep**: `ioredis@^5.0.0` (optional)
### `createRedisEventTarget`
```ts
function createRedisEventTarget<TEvent extends TypedEvent>(
args: CreateRedisEventTargetArgs,
): TypedEventTarget<TEvent>;
```
### `CreateRedisEventTargetArgs`
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `publishClient` | `Redis \| Cluster` | Yes | ioredis client for publishing. Can share a connection. |
| `subscribeClient` | `Redis \| Cluster` | Yes | ioredis client for subscribing. Must be dedicated — Redis requires subscriber connections to only receive messages. |
| `serializer` | `{ stringify, parse }` | No | Custom serializer. Defaults to `JSON`. |
### How It Works
- `dispatchEvent``publishClient.publish(event.type, serializer.stringify(event.detail))`
- `addEventListener``subscribeClient.subscribe(topic)`, track callbacks per topic
- `removeEventListener` → remove callback; if no callbacks remain for topic, `subscribeClient.unsubscribe(topic)`
### Channel Naming
Currently uses raw event type as Redis channel name (e.g., `session.status:proj_123`). Architecture recommends `alk:events:{eventType}` prefix but this is not yet implemented. Should be configurable: `createRedisEventTarget({ ..., prefix: "alk:events:" })`.
### Limitations (Current)
- **No error handling** — connection failures, reconnection, and message parse errors are not handled
- **No channel prefix** — raw event types as channel names risk collision in shared Redis instances
- **No unsubscribe cleanup on client disconnect** — if the subscribe client disconnects, registered callbacks remain in the map but will never fire
### Test Coverage
5 tests in alkhub (publish path only, mocked ioredis). No tests for subscription-receive path, unsubscribe cleanup, or error handling.
## WebSocket
**Import**: `@alkdev/pubsub/event-target-websocket` (not yet implemented)
**Peer dep**: none (WebSocket is a web standard)
### Design (Spec from `spoke-runner.md`)
```ts
class WebSocketEventTarget implements TypedEventTarget<any> {
private listeners = new Map<string, Set<(event: CustomEvent) => void>>()
constructor(private ws: WebSocket) {
ws.onmessage = (msg) => {
const { type, payload } = JSON.parse(msg.data as string)
const event = new CustomEvent(type, { detail: payload })
for (const listener of this.listeners.get(type) ?? []) {
listener(event)
}
}
}
dispatchEvent(event: CustomEvent): boolean {
this.ws.send(JSON.stringify({ type: event.type, payload: event.detail }))
return true
}
addEventListener(type: string, listener: (event: CustomEvent) => void): void { ... }
removeEventListener(type: string, listener: (event: CustomEvent) => void): void { ... }
}
```
### Key Properties
- **Bidirectional** — `dispatchEvent` sends over WS, `addEventListener` receives from WS
- **Per-connection** — hub creates one per spoke connection
- **JSON framing** — WebSocket provides native message boundaries (no length-prefix needed)
- **No native deps** — works in browsers and Node
### Gap: Reconnection
WebSocket connections drop. On reconnect, the spoke must re-register with the hub (same `hub.register` flow). The `WebSocketEventTarget` itself is per-connection — a new connection means a new event target instance. Reconnection logic belongs to the spoke lifecycle, not the event target.
### Gap: Hub-Side Architecture
The hub needs per-connection event target + `PendingRequestMap` creation on accept, cleanup on disconnect. This is a hub architectural concern, not a pubsub concern. See `@alkdev/alkhub_ts/docs/architecture/spoke-runner.md`.

View File

@@ -0,0 +1,140 @@
---
status: draft
last_updated: 2026-04-30
---
# Iroh Transport
P2P QUIC event target using iroh. More complex than the other transports due to NAT traversal, crypto identity, and byte-stream framing.
**Import**: `@alkdev/pubsub/event-target-iroh` (not yet implemented)
**Peer dep**: `@rayhanadev/iroh` (optional, NAPI-RS native addon)
## Why Iroh
WebSocket requires the hub to have a publicly reachable address. Iroh solves:
1. **Hub behind NAT** — spokes dial by `NodeId` through relay servers, no public IP needed
2. **Spoke push** — hub can initiate connections to spokes by `NodeId` (impossible with WS without polling)
3. **P2P spoke-to-spoke** — direct spoke-to-spoke communication without routing through hub
4. **Cryptographic identity** — Ed25519 `NodeId` doubles as spoke authentication
## Iroh Binding
Using `@rayhanadev/iroh` (v0.1.1) as the NAPI-RS binding. Community binding, one author, no tests. It has everything needed for hub-spoke 1:1 bidirectional streams:
| Method | Purpose |
|--------|---------|
| `Endpoint.create()` / `createWithOptions({ alpns })` | Create QUIC endpoint |
| `Endpoint.connect(nodeId, alpn)` | Connect to a peer by public key |
| `Endpoint.accept()` | Accept incoming connection |
| `Endpoint.nodeId()` | Get our public key identity |
| `Connection.openBi()` | Open bidirectional stream (spoke side) |
| `Connection.acceptBi()` | Accept bidirectional stream (hub side) |
| `SendStream.writeAll(data)` | Send data on stream |
| `RecvStream.readExact(len)` | Read exact bytes from stream |
| `Connection.remoteNodeId()` | Get peer's public key |
| `Connection.sendDatagram()` / `readDatagram()` | Unreliable datagrams |
Not exposed (not critical): `Endpoint.watch_addr()`, `Connection.close_reason()`, `Connection.stats()`.
## Protocol
Single bidirectional QUIC stream per connection. Length-prefixed JSON messages.
### Framing
QUIC streams are byte streams (no message boundaries). We use 4-byte big-endian length prefix:
```
[4 bytes: length N][N bytes: JSON payload]
```
`RecvStream.readExact(4)` reads the length, then `readExact(N)` reads the payload. This is trivial with iroh's `readExact()` API.
### Message Format
Same `type` + `detail` shape as all other transports:
```json
{ "type": "call.requested", "detail": { ... } }
```
Maps directly to `new CustomEvent(type, { detail })`.
## Two-Sided Design
Unlike Redis and WebSocket, Iroh has distinct hub and spoke connection patterns:
### Spoke Side
```ts
const conn = await endpoint.connect(hubNodeId, "alkhub/1");
const eventTarget = await createSpokeIrohEventTarget(conn);
```
Spoke opens the bidirectional stream with `openBi()`. The event target wraps the `SendStream` and `RecvStream`.
### Hub Side
```ts
const conn = await endpoint.accept();
const eventTarget = await createHubIrohEventTarget(conn);
```
Hub accepts the connection, then accepts the stream with `acceptBi()`. Same `TypedEventTarget` interface on both sides.
### Why Two Factories?
The connection initiator (spoke) calls `openBi()`. The listener (hub) calls `acceptBi()`. Both get `SendStream` + `RecvStream` — the framing and event handling are identical. The split is about connection establishment, not event handling. Could be unified as `createIrohEventTarget(sendStream, recvStream)` with separate helpers for connection, but the two-factory pattern makes the hub/spoke asymmetry explicit.
## Identity
`Connection.remoteNodeId()` returns the peer's Ed25519 public key. This is cryptographic identity — no separate API key exchange needed for authentication. The hub can verify that a connection comes from an expected spoke by checking its `NodeId`.
This is strictly better than WebSocket's token-in-URL or first-message approach. It's also harder to revoke — disabling a spoke requires a denylist of `NodeId`s rather than rotating a token.
## Connection Startup
On connection, both sides exchange the operations they expose (same `hub.register` pattern as WebSocket). The `NodeId` serves as identity — no separate API key exchange.
## Reconnection
Same pattern as WebSocket — detect connection failure, reconnect, re-register. QUIC handles multipath better than TCP but the application still needs reconnection logic.
Detection: `RecvStream.readExact()` throws on connection close. The event target should propagate this as an error event or let the caller handle it.
## Browser Limitations
Iroh in browsers is relay-only (no UDP hole punching from browser sandbox). This means:
- Browser spokes always route through relay servers
- WebSocketEventTarget is the right browser transport today (native, no extra deps)
- IrohEventTarget for browsers would use the WASM build over relay — future option
## Multi-Node (Future)
For 1:N fan-out, `iroh-gossip` is the right tool. No TS binding exists yet. Options:
1. Write a minimal Rust NAPI crate wrapping `iroh-gossip::Gossip.subscribe() + broadcast()`
2. Contribute gossip to `@rayhanadev/iroh`
3. Use hub as a relay point (hub receives once, fans out to each spoke's `IrohEventTarget` individually)
For now, 1:1 connections are sufficient. The hub fans out to multiple spokes by dispatching to each spoke's `IrohEventTarget` individually — same pattern as WebSocketEventTarget on the hub side.
## Comparison with WebSocketEventTarget
| Aspect | WebSocket | Iroh |
|--------|-----------|------|
| Connection | `new WebSocket(url)` | `endpoint.connect(nodeId, alpn)` |
| Accept | Hono WS upgrade | `endpoint.accept()` |
| Identity | API key/token | Ed25519 NodeId (cryptographic, mutual) |
| NAT traversal | Requires reverse proxy / tunnel | Built-in (relay + hole punching) |
| Framing | WS frames (built-in) | QUIC stream (length-prefix needed) |
| Hub behind NAT | Not possible without tunneling | Yes |
| Browser | Yes (native) | Limited (WASM build, relay-only) |
| Native addon | No | Yes (NAPI-RS) |
## Open Questions
1. **Binding stability**`@rayhanadev/iroh` has one author and no tests. If it breaks, we may need to fork or write our own NAPI wrapper. Mitigation: the API surface we use is small (10 methods) and the binding is thin.
2. **NAPI under Deno** — NAPI-RS `.node` binaries need testing under Deno 2.x. Since we're building with tsup for npm, the runtime is Node.js.
3. **Datagram support**`sendDatagram`/`readDatagram` could be used for fire-and-forget events (no response expected). Not needed for hub-spoke but could be useful for broadcast. Deferred.

280
docs/research/migration.md Normal file
View File

@@ -0,0 +1,280 @@
# Research: `@alkdev/pubsub` Package Extraction
## Goal
Extract `packages/core/pubsub/` into a standalone `@alkdev/pubsub` package, following the same peer-dependency tree-shaking pattern as `@alkdev/typemap`. Each event target adapter (Redis, WebSocket, Iroh) is an isolated module that only imports its own peer dependency. The core `createPubSub + TypedEventTarget + operators` has no peer deps beyond `@repeaterjs/repeater`.
## Current State
### Source: `packages/core/pubsub/`
| File | Lines | Key Exports | Dependencies |
|------|-------|-------------|--------------|
| `typed_event_target.ts` | 59 | `TypedEvent`, `TypedEventTarget`, `TypedEventListener` etc. | None (pure types) |
| `create_pubsub.ts` | 108 | `createPubSub`, `PubSub`, `PubSubConfig`, `PubSubPublishArgsByKey` | `@repeaterjs/repeater` |
| `redis_event_target.ts` | 117 | `createRedisEventTarget`, `CreateRedisEventTargetArgs` | `ioredis` (types only), `typed_event_target.ts` |
| `operators.ts` | 67 | `filter`, `map`, `pipe` | `@repeaterjs/repeater` |
| `mod.ts` | 5 | Re-exports all + `Repeater` | All above |
**Zero cross-module dependencies.** The pubsub module imports nothing from `operations/`, `mcp/`, `config/`, or `logger/`. It is already self-contained.
### Test Coverage
| Test File | Tests | Coverage |
|-----------|-------|----------|
| `tests/pubsub/redis_event_target.test.ts` | 5 tests | Redis publish path only (mocked ioredis). No subscription-receive path, no real Redis. |
| `create_pubsub.ts` | 0 tests | **No tests.** Core pubsub creation, topic scoping, event delivery, Repeater iteration all untested. |
| `operators.ts` | 0 tests | **No tests.** `filter`, `map`, `pipe` all untested. |
| `typed_event_target.ts` | N/A | Pure type definitions — no runtime to test. |
### What's Missing (Not Yet Implemented)
1. **WebSocketEventTarget** — Spec in `spoke-runner.md` (lines 158-204). Implements `TypedEventTarget` over a WebSocket connection. Bidirectional: `dispatchEvent` sends over WS, `addEventListener` receives from WS. Per-connection instance on hub side.
2. **IrohEventTarget** — P2P QUIC transport using iroh. Same role as WebSocketEventTarget but with crypto identity (Ed25519 NodeId) and automatic NAT traversal. The `@rayhanadev/iroh` NAPI-RS binding has everything needed — `Endpoint.connect()`/`accept()`, `Connection.openBi()`/`acceptBi()`, `SendStream`/`RecvStream`. No gossip required for hub↔spoke (1:1 bidirectional). See "Iroh Research" below.
3. **In-process EventTarget** — Currently `createPubSub` defaults to `new EventTarget()`, which works single-process. No explicit adapter class for this (it's just the default). Could be formalized as `InProcessEventTarget` for clarity, or left as-is since `EventTarget` is a web standard.
4. **Redis channel prefixing** — Architecture doc recommends `alk:events:{eventType}` namespacing. Not implemented.
5. **Redis reconnection/error handling** — No error handling for connection failures, reconnection, or message parse errors.
## Proposed Package Structure
```
@alkdev/pubsub/
src/
index.ts # Barrel: re-exports all public API
types.ts # TypedEvent, TypedEventTarget, etc. (from typed_event_target.ts)
create_pubsub.ts # createPubSub factory (no changes)
operators.ts # filter, map, pipe (no changes)
# Adapter modules (tree-shakeable, each is its own peer dep island)
event-target-in-process.ts # Explicit InProcessEventTarget (or just re-export web EventTarget)
event-target-redis.ts # createRedisEventTarget (peer dep: ioredis)
event-target-websocket.ts # createWebSocketEventTarget (peer dep: none — WS is a web standard)
event-target-iroh.ts # createIrohEventTarget (peer dep: @rayhanadev/iroh)
tests/
create_pubsub.test.ts # Core pubsub: publish, subscribe, topic scoping, Repeater
operators.test.ts # filter, map, pipe
event-target-in-process.test.ts
event-target-redis.test.ts # Mocked + integration
event-target-websocket.test.ts
event-target-iroh.test.ts # Mocked or integration
package.json
tsconfig.json
```
The barrel `index.ts` re-exports everything (like typemap). Tree-shaking works because ESM re-exports are statically analyzable. Users who want minimal bundles import specific adapter files directly (e.g., `import { createRedisEventTarget } from '@alkdev/pubsub/event-target-redis'`).
Alternatively, if we want sub-path exports (which typemap doesn't use but many packages do), we could add them to `package.json` exports:
```json
{
"exports": {
".": { "import": "./dist/index.mjs", "types": "./dist/index.d.mts" },
"./event-target-redis": { "import": "./dist/event-target-redis.mjs", "types": "./dist/event-target-redis.d.mts" },
"./event-target-websocket": { "import": "./dist/event-target-websocket.mjs", "types": "./dist/event-target-websocket.d.mts" },
"./event-target-iroh": { "import": "./dist/event-target-iroh.mjs", "types": "./dist/event-target-iroh.d.mts" }
}
}
```
Sub-path exports are more explicit and don't rely on bundler tree-shaking, but add maintenance burden. We should pick one approach and use it consistently across `@alkdev` packages.
## Dependencies
| Dependency | Type | Notes |
|-----------|------|-------|
| `@repeaterjs/repeater` | direct | Small (~3KB), stable. Core async iterable primitive for `subscribe()`. |
| `ioredis` | peer | Only imported by `event-target-redis.ts`. Type-only import at compile time. Consumers who don't need Redis skip it. |
| `@rayhanadev/iroh` | peer | Only imported by `event-target-iroh.ts`. NAPI-RS native addon (~15-20MB). Consumers who don't need P2P QUIC skip it. |
No other external dependencies. No logger dependency.
## Build & Publish
Following `@alkdev/taskgraph` precedent:
- **Build tool**: `tsup` — produces dual ESM + CJS with types automatically
- **Target**: `es2022`
- **Publish target**: npm (`@alkdev/pubsub`)
- **Deno compatibility**: Source is standard TypeScript with no Deno-specific APIs (all web standard). Deno can import from npm or JSR.
- **Testing**: `vitest` (matching taskgraph) or `deno test` (matching current alkhub_ts). Decision needed.
### Build Config Sketch
```ts
// tsup.config.ts
import { defineConfig } from 'tsup';
export default defineConfig({
entry: [
'src/index.ts',
'src/event-target-redis.ts',
'src/event-target-websocket.ts',
'src/event-target-iroh.ts',
],
format: ['esm', 'cjs'],
dts: true,
splitting: true,
clean: true,
target: 'es2022',
});
```
## Iroh Research Summary
### What Is Iroh?
Iroh is a Rust P2P QUIC library by n0.computer. Peers connect by **public key** (Ed25519), not IP address. Key capabilities:
- **NAT traversal**: Automatic UDP hole punching (~90% success rate), QUIC Address Discovery
- **Relay fallback**: If direct connection fails, routes through stateless relay servers (end-to-end encrypted)
- **Public key addressing**: Peers identified by `NodeId`, no DNS or IP needed
- **QUIC transport**: Multiplexed streams, built-in encryption, 0-RTT
- **Gossip protocol** (`iroh-gossip`): Epidemic broadcast trees for topic-based pub/sub (not needed for hub↔spoke — that's 1:1, not 1:N)
### Why It Matters for alkhub
WebSocket transport requires the hub to have a publicly reachable address. Spokes behind NAT can't be reached by the hub for push operations. Iroh solves:
1. **Hub behind NAT** — No public IP needed. Spokes dial the hub by its `NodeId` through relay servers.
2. **Spoke push** — Hub can initiate connections to spokes by `NodeId` (impossible with WS without polling).
3. **P2P spoke↔spoke** — Direct spoke-to-spoke communication without routing through hub.
4. **Cryptographic identity** — Ed25519 `NodeId` doubles as spoke authentication — strictly better than API keys for identification.
### Current TS Binding — `@rayhanadev/iroh`
NAPI-RS binding (v0.1.1) at `/workspace/iroh-ts`. **The binding has everything needed to build IrohEventTarget.** No gossip required — hub↔spoke is 1:1 bidirectional JSON event channels over QUIC streams.
**Core API that we use:**
| Method | Purpose |
|--------|---------|
| `Endpoint.create()` / `createWithOptions({ alpns })` | Create QUIC endpoint |
| `Endpoint.connect(nodeId, alpn)` | Connect to a peer by public key |
| `Endpoint.accept()` | Accept incoming connection |
| `Endpoint.nodeId()` | Get our public key identity |
| `Connection.openBi()` | Open bidirectional stream (spoke side) |
| `Connection.acceptBi()` | Accept bidirectional stream (hub side) |
| `SendStream.writeAll(data)` | Send data on stream |
| `RecvStream.readExact(len)` | Read exact bytes from stream |
| `Connection.remoteNodeId()` | Get peer's public key |
| `Connection.sendDatagram(data)` / `readDatagram()` | Unreliable datagrams (fire-and-forget events) |
**Not exposed (but not critical):**
- `Endpoint.watch_addr()` — detect network changes (workaround: detect connection failure)
- `Connection.close_reason()` — synchronous close check (workaround: await `closed()`)
- `Connection.stats()` — observability (nice to have, not required)
### IrohEventTarget Design
Same `TypedEventTarget` interface as `WebSocketEventTarget` and `RedisEventTarget`. Hub and spoke each create one per connection.
**Protocol**: Single bidirectional QUIC stream per connection, length-prefixed JSON messages. Spoke opens the stream with `openBi()`, hub accepts with `acceptBi()`. Same `type` + `detail` event shape as all other transports.
```ts
// Spoke side
const conn = await endpoint.connect(hubNodeId, "alkhub/1");
const eventTarget = await createSpokeIrohEventTarget(conn);
// Hub side
const conn = await endpoint.accept();
const eventTarget = await createHubIrohEventTarget(conn);
// Both sides — same TypedEventTarget interface
eventTarget.addEventListener("call.responded", (event) => { ... });
eventTarget.dispatchEvent(new CustomEvent("call.requested", { detail: { ... } }));
```
**Framing**: 4-byte big-endian length prefix + JSON payload. Necessary because QUIC streams are byte streams, not message streams. `readExact()` makes this trivial.
**Connection startup**: On connection, both sides exchange the operations they expose (same hub.register pattern as WebSocket). The `NodeId` serves as cryptographic identity — no separate API key exchange needed for authentication.
**Reconnection**: Same pattern as WebSocket — detect connection failure, reconnect, re-register. QUIC handles multipath better than TCP but the application still needs reconnection logic.
**Comparison with WebSocketEventTarget:**
| Aspect | WebSocketEventTarget | IrohEventTarget |
|--------|---------------------|-----------------|
| Connection | `new WebSocket(url)` | `endpoint.connect(nodeId, alpn)` |
| Accept | Hono WS upgrade | `endpoint.accept()` |
| Identity | API key/token in URL or first message | Ed25519 NodeId (cryptographic, mutual) |
| NAT traversal | Requires reverse proxy / CDN / tunnel | Built-in (relay + hole punching) |
| Framing | WS frames (built-in message boundary) | QUIC stream (needs length-prefix framing) |
| Hub behind NAT | Not possible without tunneling | Yes — spoke dials by NodeId |
| Browser | Yes (native WS) | Limited (WASM build, relay-only — use WS for browsers) |
### Multi-Node Scenarios (Future)
For 1:N fan-out (e.g., one event to 50 spokes), `iroh-gossip` is the right tool. No TS binding exposes it yet. Options when we need it:
1. Write a minimal Rust NAPI crate wrapping `iroh-gossip::Gossip.subscribe() + broadcast()` (~500 lines Rust)
2. Contribute gossip to `@rayhanadev/iroh` or `@salvatoret/iroh`
3. Use hub as a relay point (hub receives once, fans out to each spoke's `IrohEventTarget` individually)
For now, 1:1 connections are sufficient. The hub can fan out to multiple spokes by dispatching to each spoke's `IrohEventTarget` individually — same pattern as WebSocketEventTarget on the hub side.
### Browser Considerations
Iroh in browsers is relay-only (no UDP hole punching from browser sandbox). This means:
- Browser spokes always route through relay servers
- WebSocketEventTarget is the right browser transport today (native, no extra deps)
- IrohEventTarget for browsers would use the WASM build over relay — future option
## Migration Steps
### Phase 1: Extract to standalone package
1. **Create `@alkdev/pubsub` repo** (or directory in a monorepo)
2. **Copy source files** from `packages/core/pubsub/` with no modifications to core logic:
- `typed_event_target.ts``types.ts`
- `create_pubsub.ts``create_pubsub.ts`
- `redis_event_target.ts``event-target-redis.ts`
- `operators.ts``operators.ts`
3. **Set up build pipeline** (tsup, package.json, tsconfig)
4. **Move Redis to peer dependency** in `package.json`
5. **Write missing tests**: `create_pubsub.test.ts`, `operators.test.ts`
6. **Add Redis subscription-receive and unsubscribe cleanup tests**
7. **Publish v0.1.0 to npm**
### Phase 2: Add adapters and improve coverage
8. **Implement `WebSocketEventTarget`** per `spoke-runner.md` spec
9. **Implement `IrohEventTarget`**`createHubIrohEventTarget` / `createSpokeIrohEventTarget` with length-prefixed JSON framing over QUIC streams
10. **Add Redis channel prefixing** (`alk:events:*` or configurable prefix)
11. **Add Redis error handling** (connection errors, reconnection, parse errors)
12. **Formalize `InProcessEventTarget`** (explicit or just document that `EventTarget` is the default)
13. **Write adapter tests** (mock WS bidirectional flow, mock iroh connect/accept/stream)
### Phase 3: Production hardening
14. **Redis integration tests** with real Redis instance
15. **WebSocket integration tests** with real WS server/client
16. **Iroh integration tests** — requires relay server or direct P2P between two endpoints
17. **Reconnection logic** for both WebSocket and Iroh adapters
18. **Error propagation** — connection failures should propagate to listeners gracefully
### Phase 4: Integration back into alkhub_ts
19. **Replace** `packages/core/pubsub/` with `@alkdev/pubsub` npm/JSR dependency
20. **Update** `packages/core/deno.json` and `packages/core/mod.ts` to import from `@alkdev/pubsub`
21. **Remove** `ioredis` from `packages/core/deno.json` (it moves to `@alkdev/pubsub`'s peer deps, and hub uses it directly)
22. **Update call protocol, hub, and spoke** to use `@alkdev/pubsub` directly
## Open Questions
1. **Sub-path exports vs. barrel + tree-shaking?** Typemap uses barrel-only + tree-shaking. Taskgraph uses barrel-only. Do we want sub-path exports for explicit adapter imports, or rely on tree-shaking?
2. **Test runner**: `vitest` (matches taskgraph) or `deno test` (matches current alkhub_ts)? If the package publishes to npm via tsup, `vitest` is the natural choice. If we also want to test in Deno, we could support both.
3. **Deno-first or Node-first development?** Current code has no Deno-specific APIs (it's all web standard). We could develop in either. Deno can import from npm. Node can't import from JSR without the JSR npm mirror. If we're using tsup for build, we're effectively Node-first for publishing, Deno-compatible for source.
4. **When to implement `WebSocketEventTarget` and `IrohEventTarget`?** Before or after extracting the package? The specs and interfaces are clear. Could implement both as part of the initial adapter set, since both follow the same `TypedEventTarget` pattern.
5. **Iroh binding**: Should we use `@rayhanadev/iroh` directly (v0.1.1, community binding, 9 commits, no tests) or write/publish our own `@alkdev/iroh` NAPI wrapper? The current binding works but has no tests and one author. Forking/forking-and-maintaining gives us control of the build pipeline.
6. **Iroh + Deno**: NAPI-RS `.node` binaries may need testing under Deno 2.x. If we're building with tsup for npm publish, the runtime is Node.js. For Deno-first development, we need to verify NAPI addons work.
7. **Redis channel prefixing**: Should the prefix be configurable per `createRedisEventTarget({ prefix })?` or hardcoded to `alk:events:`? Configurable is more flexible for multi-tenant scenarios.
### Architecture Decision: WebSocket vs Iroh as Primary Transport
WebSocket is the right default for most deployments — it's native in browsers and Deno, well-supported, and requires no native addons. Iroh is the right choice when:
- The hub is behind NAT (dev laptops, home servers, no CDN)
- Spokes need to be reachable by the hub (push notifications to client spokes)
- Cryptographic identity is preferred over token-based auth
- P2P spoke-to-spoke communication is needed
A deployment can use both: `WebSocketEventTarget` for browser clients, `IrohEventTarget` for native spokes. Same `TypedEventTarget` interface, same call protocol, same `PendingRequestMap`.