Set up project structure, source files, and architecture docs
- Copy core source from alkhub_ts/packages/core/pubsub/ with import path fixups (typed_event_target.ts → types.ts, .ts → .js extensions) - Make PubSubPublishArgsByKey exported (was private type, needed by barrel) - Add package.json with sub-path exports and optional peer deps (ioredis) - Add tsup.config.ts with multi-entry + splitting for tree-shaking - Add tsconfig.json, vitest.config.ts, .gitignore - Add AGENTS.md with project conventions and adapter checklist - Add architecture docs following taskgraph/alkhub pattern: docs/architecture/README.md, api-surface.md, event-targets.md, iroh-transport.md, build-distribution.md - Add ADRs: 001-graphql-yoga-fork, 002-tree-shake-pattern - Copy migration research doc to docs/research/migration.md - Dual-license MIT OR Apache-2.0 (matching taskgraph)
This commit is contained in:
280
docs/research/migration.md
Normal file
280
docs/research/migration.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Research: `@alkdev/pubsub` Package Extraction
|
||||
|
||||
## Goal
|
||||
|
||||
Extract `packages/core/pubsub/` into a standalone `@alkdev/pubsub` package, following the same peer-dependency tree-shaking pattern as `@alkdev/typemap`. Each event target adapter (Redis, WebSocket, Iroh) is an isolated module that only imports its own peer dependency. The core `createPubSub + TypedEventTarget + operators` has no peer deps beyond `@repeaterjs/repeater`.
|
||||
|
||||
## Current State
|
||||
|
||||
### Source: `packages/core/pubsub/`
|
||||
|
||||
| File | Lines | Key Exports | Dependencies |
|
||||
|------|-------|-------------|--------------|
|
||||
| `typed_event_target.ts` | 59 | `TypedEvent`, `TypedEventTarget`, `TypedEventListener` etc. | None (pure types) |
|
||||
| `create_pubsub.ts` | 108 | `createPubSub`, `PubSub`, `PubSubConfig`, `PubSubPublishArgsByKey` | `@repeaterjs/repeater` |
|
||||
| `redis_event_target.ts` | 117 | `createRedisEventTarget`, `CreateRedisEventTargetArgs` | `ioredis` (types only), `typed_event_target.ts` |
|
||||
| `operators.ts` | 67 | `filter`, `map`, `pipe` | `@repeaterjs/repeater` |
|
||||
| `mod.ts` | 5 | Re-exports all + `Repeater` | All above |
|
||||
|
||||
**Zero cross-module dependencies.** The pubsub module imports nothing from `operations/`, `mcp/`, `config/`, or `logger/`. It is already self-contained.
|
||||
|
||||
### Test Coverage
|
||||
|
||||
| Test File | Tests | Coverage |
|
||||
|-----------|-------|----------|
|
||||
| `tests/pubsub/redis_event_target.test.ts` | 5 tests | Redis publish path only (mocked ioredis). No subscription-receive path, no real Redis. |
|
||||
| `create_pubsub.ts` | 0 tests | **No tests.** Core pubsub creation, topic scoping, event delivery, Repeater iteration all untested. |
|
||||
| `operators.ts` | 0 tests | **No tests.** `filter`, `map`, `pipe` all untested. |
|
||||
| `typed_event_target.ts` | N/A | Pure type definitions — no runtime to test. |
|
||||
|
||||
### What's Missing (Not Yet Implemented)
|
||||
|
||||
1. **WebSocketEventTarget** — Spec in `spoke-runner.md` (lines 158-204). Implements `TypedEventTarget` over a WebSocket connection. Bidirectional: `dispatchEvent` sends over WS, `addEventListener` receives from WS. Per-connection instance on hub side.
|
||||
2. **IrohEventTarget** — P2P QUIC transport using iroh. Same role as WebSocketEventTarget but with crypto identity (Ed25519 NodeId) and automatic NAT traversal. The `@rayhanadev/iroh` NAPI-RS binding has everything needed — `Endpoint.connect()`/`accept()`, `Connection.openBi()`/`acceptBi()`, `SendStream`/`RecvStream`. No gossip required for hub↔spoke (1:1 bidirectional). See "Iroh Research" below.
|
||||
3. **In-process EventTarget** — Currently `createPubSub` defaults to `new EventTarget()`, which works single-process. No explicit adapter class for this (it's just the default). Could be formalized as `InProcessEventTarget` for clarity, or left as-is since `EventTarget` is a web standard.
|
||||
4. **Redis channel prefixing** — Architecture doc recommends `alk:events:{eventType}` namespacing. Not implemented.
|
||||
5. **Redis reconnection/error handling** — No error handling for connection failures, reconnection, or message parse errors.
|
||||
|
||||
## Proposed Package Structure
|
||||
|
||||
```
|
||||
@alkdev/pubsub/
|
||||
src/
|
||||
index.ts # Barrel: re-exports all public API
|
||||
types.ts # TypedEvent, TypedEventTarget, etc. (from typed_event_target.ts)
|
||||
create_pubsub.ts # createPubSub factory (no changes)
|
||||
operators.ts # filter, map, pipe (no changes)
|
||||
|
||||
# Adapter modules (tree-shakeable, each is its own peer dep island)
|
||||
event-target-in-process.ts # Explicit InProcessEventTarget (or just re-export web EventTarget)
|
||||
event-target-redis.ts # createRedisEventTarget (peer dep: ioredis)
|
||||
event-target-websocket.ts # createWebSocketEventTarget (peer dep: none — WS is a web standard)
|
||||
event-target-iroh.ts # createIrohEventTarget (peer dep: @rayhanadev/iroh)
|
||||
tests/
|
||||
create_pubsub.test.ts # Core pubsub: publish, subscribe, topic scoping, Repeater
|
||||
operators.test.ts # filter, map, pipe
|
||||
event-target-in-process.test.ts
|
||||
event-target-redis.test.ts # Mocked + integration
|
||||
event-target-websocket.test.ts
|
||||
event-target-iroh.test.ts # Mocked or integration
|
||||
package.json
|
||||
tsconfig.json
|
||||
```
|
||||
|
||||
The barrel `index.ts` re-exports everything (like typemap). Tree-shaking works because ESM re-exports are statically analyzable. Users who want minimal bundles import specific adapter files directly (e.g., `import { createRedisEventTarget } from '@alkdev/pubsub/event-target-redis'`).
|
||||
|
||||
Alternatively, if we want sub-path exports (which typemap doesn't use but many packages do), we could add them to `package.json` exports:
|
||||
|
||||
```json
|
||||
{
|
||||
"exports": {
|
||||
".": { "import": "./dist/index.mjs", "types": "./dist/index.d.mts" },
|
||||
"./event-target-redis": { "import": "./dist/event-target-redis.mjs", "types": "./dist/event-target-redis.d.mts" },
|
||||
"./event-target-websocket": { "import": "./dist/event-target-websocket.mjs", "types": "./dist/event-target-websocket.d.mts" },
|
||||
"./event-target-iroh": { "import": "./dist/event-target-iroh.mjs", "types": "./dist/event-target-iroh.d.mts" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Sub-path exports are more explicit and don't rely on bundler tree-shaking, but add maintenance burden. We should pick one approach and use it consistently across `@alkdev` packages.
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Type | Notes |
|
||||
|-----------|------|-------|
|
||||
| `@repeaterjs/repeater` | direct | Small (~3KB), stable. Core async iterable primitive for `subscribe()`. |
|
||||
| `ioredis` | peer | Only imported by `event-target-redis.ts`. Type-only import at compile time. Consumers who don't need Redis skip it. |
|
||||
| `@rayhanadev/iroh` | peer | Only imported by `event-target-iroh.ts`. NAPI-RS native addon (~15-20MB). Consumers who don't need P2P QUIC skip it. |
|
||||
|
||||
No other external dependencies. No logger dependency.
|
||||
|
||||
## Build & Publish
|
||||
|
||||
Following `@alkdev/taskgraph` precedent:
|
||||
|
||||
- **Build tool**: `tsup` — produces dual ESM + CJS with types automatically
|
||||
- **Target**: `es2022`
|
||||
- **Publish target**: npm (`@alkdev/pubsub`)
|
||||
- **Deno compatibility**: Source is standard TypeScript with no Deno-specific APIs (all web standard). Deno can import from npm or JSR.
|
||||
- **Testing**: `vitest` (matching taskgraph) or `deno test` (matching current alkhub_ts). Decision needed.
|
||||
|
||||
### Build Config Sketch
|
||||
|
||||
```ts
|
||||
// tsup.config.ts
|
||||
import { defineConfig } from 'tsup';
|
||||
|
||||
export default defineConfig({
|
||||
entry: [
|
||||
'src/index.ts',
|
||||
'src/event-target-redis.ts',
|
||||
'src/event-target-websocket.ts',
|
||||
'src/event-target-iroh.ts',
|
||||
],
|
||||
format: ['esm', 'cjs'],
|
||||
dts: true,
|
||||
splitting: true,
|
||||
clean: true,
|
||||
target: 'es2022',
|
||||
});
|
||||
```
|
||||
|
||||
## Iroh Research Summary
|
||||
|
||||
### What Is Iroh?
|
||||
|
||||
Iroh is a Rust P2P QUIC library by n0.computer. Peers connect by **public key** (Ed25519), not IP address. Key capabilities:
|
||||
|
||||
- **NAT traversal**: Automatic UDP hole punching (~90% success rate), QUIC Address Discovery
|
||||
- **Relay fallback**: If direct connection fails, routes through stateless relay servers (end-to-end encrypted)
|
||||
- **Public key addressing**: Peers identified by `NodeId`, no DNS or IP needed
|
||||
- **QUIC transport**: Multiplexed streams, built-in encryption, 0-RTT
|
||||
- **Gossip protocol** (`iroh-gossip`): Epidemic broadcast trees for topic-based pub/sub (not needed for hub↔spoke — that's 1:1, not 1:N)
|
||||
|
||||
### Why It Matters for alkhub
|
||||
|
||||
WebSocket transport requires the hub to have a publicly reachable address. Spokes behind NAT can't be reached by the hub for push operations. Iroh solves:
|
||||
|
||||
1. **Hub behind NAT** — No public IP needed. Spokes dial the hub by its `NodeId` through relay servers.
|
||||
2. **Spoke push** — Hub can initiate connections to spokes by `NodeId` (impossible with WS without polling).
|
||||
3. **P2P spoke↔spoke** — Direct spoke-to-spoke communication without routing through hub.
|
||||
4. **Cryptographic identity** — Ed25519 `NodeId` doubles as spoke authentication — strictly better than API keys for identification.
|
||||
|
||||
### Current TS Binding — `@rayhanadev/iroh`
|
||||
|
||||
NAPI-RS binding (v0.1.1) at `/workspace/iroh-ts`. **The binding has everything needed to build IrohEventTarget.** No gossip required — hub↔spoke is 1:1 bidirectional JSON event channels over QUIC streams.
|
||||
|
||||
**Core API that we use:**
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `Endpoint.create()` / `createWithOptions({ alpns })` | Create QUIC endpoint |
|
||||
| `Endpoint.connect(nodeId, alpn)` | Connect to a peer by public key |
|
||||
| `Endpoint.accept()` | Accept incoming connection |
|
||||
| `Endpoint.nodeId()` | Get our public key identity |
|
||||
| `Connection.openBi()` | Open bidirectional stream (spoke side) |
|
||||
| `Connection.acceptBi()` | Accept bidirectional stream (hub side) |
|
||||
| `SendStream.writeAll(data)` | Send data on stream |
|
||||
| `RecvStream.readExact(len)` | Read exact bytes from stream |
|
||||
| `Connection.remoteNodeId()` | Get peer's public key |
|
||||
| `Connection.sendDatagram(data)` / `readDatagram()` | Unreliable datagrams (fire-and-forget events) |
|
||||
|
||||
**Not exposed (but not critical):**
|
||||
- `Endpoint.watch_addr()` — detect network changes (workaround: detect connection failure)
|
||||
- `Connection.close_reason()` — synchronous close check (workaround: await `closed()`)
|
||||
- `Connection.stats()` — observability (nice to have, not required)
|
||||
|
||||
### IrohEventTarget Design
|
||||
|
||||
Same `TypedEventTarget` interface as `WebSocketEventTarget` and `RedisEventTarget`. Hub and spoke each create one per connection.
|
||||
|
||||
**Protocol**: Single bidirectional QUIC stream per connection, length-prefixed JSON messages. Spoke opens the stream with `openBi()`, hub accepts with `acceptBi()`. Same `type` + `detail` event shape as all other transports.
|
||||
|
||||
```ts
|
||||
// Spoke side
|
||||
const conn = await endpoint.connect(hubNodeId, "alkhub/1");
|
||||
const eventTarget = await createSpokeIrohEventTarget(conn);
|
||||
|
||||
// Hub side
|
||||
const conn = await endpoint.accept();
|
||||
const eventTarget = await createHubIrohEventTarget(conn);
|
||||
|
||||
// Both sides — same TypedEventTarget interface
|
||||
eventTarget.addEventListener("call.responded", (event) => { ... });
|
||||
eventTarget.dispatchEvent(new CustomEvent("call.requested", { detail: { ... } }));
|
||||
```
|
||||
|
||||
**Framing**: 4-byte big-endian length prefix + JSON payload. Necessary because QUIC streams are byte streams, not message streams. `readExact()` makes this trivial.
|
||||
|
||||
**Connection startup**: On connection, both sides exchange the operations they expose (same hub.register pattern as WebSocket). The `NodeId` serves as cryptographic identity — no separate API key exchange needed for authentication.
|
||||
|
||||
**Reconnection**: Same pattern as WebSocket — detect connection failure, reconnect, re-register. QUIC handles multipath better than TCP but the application still needs reconnection logic.
|
||||
|
||||
**Comparison with WebSocketEventTarget:**
|
||||
|
||||
| Aspect | WebSocketEventTarget | IrohEventTarget |
|
||||
|--------|---------------------|-----------------|
|
||||
| Connection | `new WebSocket(url)` | `endpoint.connect(nodeId, alpn)` |
|
||||
| Accept | Hono WS upgrade | `endpoint.accept()` |
|
||||
| Identity | API key/token in URL or first message | Ed25519 NodeId (cryptographic, mutual) |
|
||||
| NAT traversal | Requires reverse proxy / CDN / tunnel | Built-in (relay + hole punching) |
|
||||
| Framing | WS frames (built-in message boundary) | QUIC stream (needs length-prefix framing) |
|
||||
| Hub behind NAT | Not possible without tunneling | Yes — spoke dials by NodeId |
|
||||
| Browser | Yes (native WS) | Limited (WASM build, relay-only — use WS for browsers) |
|
||||
|
||||
### Multi-Node Scenarios (Future)
|
||||
|
||||
For 1:N fan-out (e.g., one event to 50 spokes), `iroh-gossip` is the right tool. No TS binding exposes it yet. Options when we need it:
|
||||
1. Write a minimal Rust NAPI crate wrapping `iroh-gossip::Gossip.subscribe() + broadcast()` (~500 lines Rust)
|
||||
2. Contribute gossip to `@rayhanadev/iroh` or `@salvatoret/iroh`
|
||||
3. Use hub as a relay point (hub receives once, fans out to each spoke's `IrohEventTarget` individually)
|
||||
|
||||
For now, 1:1 connections are sufficient. The hub can fan out to multiple spokes by dispatching to each spoke's `IrohEventTarget` individually — same pattern as WebSocketEventTarget on the hub side.
|
||||
|
||||
### Browser Considerations
|
||||
|
||||
Iroh in browsers is relay-only (no UDP hole punching from browser sandbox). This means:
|
||||
- Browser spokes always route through relay servers
|
||||
- WebSocketEventTarget is the right browser transport today (native, no extra deps)
|
||||
- IrohEventTarget for browsers would use the WASM build over relay — future option
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Phase 1: Extract to standalone package
|
||||
|
||||
1. **Create `@alkdev/pubsub` repo** (or directory in a monorepo)
|
||||
2. **Copy source files** from `packages/core/pubsub/` with no modifications to core logic:
|
||||
- `typed_event_target.ts` → `types.ts`
|
||||
- `create_pubsub.ts` → `create_pubsub.ts`
|
||||
- `redis_event_target.ts` → `event-target-redis.ts`
|
||||
- `operators.ts` → `operators.ts`
|
||||
3. **Set up build pipeline** (tsup, package.json, tsconfig)
|
||||
4. **Move Redis to peer dependency** in `package.json`
|
||||
5. **Write missing tests**: `create_pubsub.test.ts`, `operators.test.ts`
|
||||
6. **Add Redis subscription-receive and unsubscribe cleanup tests**
|
||||
7. **Publish v0.1.0 to npm**
|
||||
|
||||
### Phase 2: Add adapters and improve coverage
|
||||
|
||||
8. **Implement `WebSocketEventTarget`** per `spoke-runner.md` spec
|
||||
9. **Implement `IrohEventTarget`** — `createHubIrohEventTarget` / `createSpokeIrohEventTarget` with length-prefixed JSON framing over QUIC streams
|
||||
10. **Add Redis channel prefixing** (`alk:events:*` or configurable prefix)
|
||||
11. **Add Redis error handling** (connection errors, reconnection, parse errors)
|
||||
12. **Formalize `InProcessEventTarget`** (explicit or just document that `EventTarget` is the default)
|
||||
13. **Write adapter tests** (mock WS bidirectional flow, mock iroh connect/accept/stream)
|
||||
|
||||
### Phase 3: Production hardening
|
||||
|
||||
14. **Redis integration tests** with real Redis instance
|
||||
15. **WebSocket integration tests** with real WS server/client
|
||||
16. **Iroh integration tests** — requires relay server or direct P2P between two endpoints
|
||||
17. **Reconnection logic** for both WebSocket and Iroh adapters
|
||||
18. **Error propagation** — connection failures should propagate to listeners gracefully
|
||||
|
||||
### Phase 4: Integration back into alkhub_ts
|
||||
|
||||
19. **Replace** `packages/core/pubsub/` with `@alkdev/pubsub` npm/JSR dependency
|
||||
20. **Update** `packages/core/deno.json` and `packages/core/mod.ts` to import from `@alkdev/pubsub`
|
||||
21. **Remove** `ioredis` from `packages/core/deno.json` (it moves to `@alkdev/pubsub`'s peer deps, and hub uses it directly)
|
||||
22. **Update call protocol, hub, and spoke** to use `@alkdev/pubsub` directly
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Sub-path exports vs. barrel + tree-shaking?** Typemap uses barrel-only + tree-shaking. Taskgraph uses barrel-only. Do we want sub-path exports for explicit adapter imports, or rely on tree-shaking?
|
||||
2. **Test runner**: `vitest` (matches taskgraph) or `deno test` (matches current alkhub_ts)? If the package publishes to npm via tsup, `vitest` is the natural choice. If we also want to test in Deno, we could support both.
|
||||
3. **Deno-first or Node-first development?** Current code has no Deno-specific APIs (it's all web standard). We could develop in either. Deno can import from npm. Node can't import from JSR without the JSR npm mirror. If we're using tsup for build, we're effectively Node-first for publishing, Deno-compatible for source.
|
||||
4. **When to implement `WebSocketEventTarget` and `IrohEventTarget`?** Before or after extracting the package? The specs and interfaces are clear. Could implement both as part of the initial adapter set, since both follow the same `TypedEventTarget` pattern.
|
||||
5. **Iroh binding**: Should we use `@rayhanadev/iroh` directly (v0.1.1, community binding, 9 commits, no tests) or write/publish our own `@alkdev/iroh` NAPI wrapper? The current binding works but has no tests and one author. Forking/forking-and-maintaining gives us control of the build pipeline.
|
||||
6. **Iroh + Deno**: NAPI-RS `.node` binaries may need testing under Deno 2.x. If we're building with tsup for npm publish, the runtime is Node.js. For Deno-first development, we need to verify NAPI addons work.
|
||||
7. **Redis channel prefixing**: Should the prefix be configurable per `createRedisEventTarget({ prefix })?` or hardcoded to `alk:events:`? Configurable is more flexible for multi-tenant scenarios.
|
||||
|
||||
### Architecture Decision: WebSocket vs Iroh as Primary Transport
|
||||
|
||||
WebSocket is the right default for most deployments — it's native in browsers and Deno, well-supported, and requires no native addons. Iroh is the right choice when:
|
||||
|
||||
- The hub is behind NAT (dev laptops, home servers, no CDN)
|
||||
- Spokes need to be reachable by the hub (push notifications to client spokes)
|
||||
- Cryptographic identity is preferred over token-based auth
|
||||
- P2P spoke-to-spoke communication is needed
|
||||
|
||||
A deployment can use both: `WebSocketEventTarget` for browser clients, `IrohEventTarget` for native spokes. Same `TypedEventTarget` interface, same call protocol, same `PendingRequestMap`.
|
||||
Reference in New Issue
Block a user