alkdev/alkcall
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Pub operation type, HandlerKind::Sink, call.published wire event (ADR-046)

Browse Source 
The call protocol had Subscription (server→client streaming) but lacked
the directional complement: client→server streaming, where the initiator
produces a stream and the responder's handler consumes it. This gap was
inherited from the @alkdev/pubsub EventEnvelope prior art, which has
subscribe but no wire-level publish.

ADR-046 adds the Pub primitive:
- OperationType::Pub (client→server streaming)
- OperationType::Subscription renamed to Sub (wire: "subscription" → "sub")
- SinkHandler type + HandlerKind::Sink variant
- PublishStream type alias (Stream<Item = Result<Value, CallError>>)
- OperationRegistry::invoke_sink() dispatch path
- call.published wire event (sixth event type, additive)
- OperationSpec.publish_schema (Option<Value>, validates per-chunk input)
- DispatchResult::Sink + SinkDispatch (handler future + chunk channel)
- Dispatcher::pump_sink (feeds call.published chunks from wire to handler)
- CallConnection::publish() / publish_with_payload() client methods
- from_call sink forwarding handler (make_sink_forwarding_handler)
- make_sink_handler() helper

Fan-out/broker (one producer, N consumers, topic matching) is deferred to
the channels session — the call protocol is point-to-point; the broker is
a routing concern that sits above it. The Pub primitive is the
load-bearing piece the broker will compose on.

- 23 new tests (366 total, up from 343)
- clippy clean, fmt clean

Verification:
  cargo test                                    — 366 passed
  cargo clippy --all-targets -- -D warnings      — clean
  cargo fmt --check                              — clean
This commit is contained in:
glm-5.2
2026-08-12 08:06:46 +00:00
parent cc470a363a
commit ea66398c88
12 changed files with 1821 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

484
docs/architecture/decisions/046-publish-operation-type-and-handler-kind-sink.md Normal file
View File

@@ -0,0 +1,484 @@
# ADR-046: Publish Operation Type and `HandlerKind::Sink`
## Status
Accepted
## Context
The call protocol has `Subscription` (server→client streaming: a
`StreamingHandler` *produces* a stream of `call.responded` events,
ADR-021). It lacks the directional complement: **client→server
streaming**, where the initiator produces a stream and the responder's
handler *consumes* it. This gap was inherited from the
`@alkdev/pubsub` `EventEnvelope` prior art, which has a `subscribe`
primitive but no `publish` primitive in its `PubSub` type
(`create_pubsub.ts:48-58` — `publish` there dispatches to an in-process
`EventTarget`, not over the wire).
The gap surfaced while speccing the channels crate's
`channel/resources/subscribe` operation (ADR-037): the hub aggregates
worker resources and must push live updates to N browser subscribers
when any worker's resource set changes. Each browser's `subscribe`
arrives on the hub's channel 0, but the worker resource changes arrive
on separate worker connections. The hub needs to fan-out.
The research findings
(`/workspace/@alkdev/alknet/docs/research/call-channels-unification/
findings.md`) identified two separable concerns:
- **(A) Client→server streaming primitive.** A new `OperationType::Pub`
where the initiator streams data to the responder. This is a
call-protocol primitive — it extends the existing
`Query`/`Mutation`/`Subscription` set with the missing direction.
- **(B) Fan-out / broker.** A producer pushes events to subscribers
it didn't directly receive a `call.requested` from. This is a broker
concern that sits *above* the call protocol — the hub-as-broker
pattern (ADR-042 hub relay) composes on top of the call protocol's
primitives.
This ADR implements **(A) only** — the `Pub` primitive in the call
protocol. **(B) is deferred to the channels session** (OQ-22 §fan-out
scope). The rationale: the call protocol's job is point-to-point (one
initiator, one responder, a stream between them); fan-out is a routing
concern with a different lifecycle (a topic registry that outlives
individual calls) and its first real consumer (the channels
`channel/resources/subscribe` / hub-as-broker) isn't implemented yet.
Designing the broker blind would produce a half-design. The `Pub`
primitive is the load-bearing call-protocol piece; the broker composes
on top of it later, in channels, the same way the hub relay composes on
top of existing call ops.
### Gap A from the research findings: `Pub` handler shape
The research findings' Gap A identified that the current
`StreamingHandler` is inherently **server→client** — the handler
*produces* a `Stream<Item = ResponseEnvelope>`:
```rust
pub type StreamingHandler = Arc<
dyn Fn(Value, OperationContext) -> Pin<Box<dyn Stream<Item = ResponseEnvelope> + Send>>
+ Send + Sync,
>;
```
For `Pub`, the initiator is the producer streaming *to* the responder.
The handler on the responder side needs to *consume* a stream from the
initiator, not produce one. Three candidate shapes were considered
(findings.md §Gap A):
- **(a) New `HandlerKind::Sink`** — `Fn(Value, OperationContext,
RecvStream) -> Future<Output = ResponseEnvelope>`. The `RecvStream`
is the initiator's data stream. Clean type-level separation; required
if direct `Pub` (client→server streaming without a hub) is a use
case.
- **(b) `BiStream` on `OperationContext`** — muddies the handler
signature; every handler gets a `BiStream` it may not need.
- **(c) `Pub` is purely a broker concept** — no new handler; the broker
matches `Pub` to `Sub` and proxies. Only works if `Pub` always goes
through a broker.
**Decision: (a).** Direct `Pub` (client→server streaming without a hub)
is a use case — a worker uploading a file stream to a hub's
`fs/writeStream` op, a client pushing telemetry to a `metrics/ingest`
op, a browser streaming a drag-drop file to a `fs/upload` op. These
don't need a broker; they need a handler that consumes a stream and
returns a result. `HandlerKind::Sink` is the type-level expression of
that.
## Decision
### 1. Rename `OperationType::Subscription` → `Sub`
The existing `Subscription` variant is renamed to `Sub` for symmetry
with `Pub`. The wire-format `op_type` string changes from `"subscription"`
to `"sub"`. No deployments exist, so this is a cosmetic rename with no
backward-compat concern. The new wire enum:
```rust
pub enum OperationType {
Query, // request → single JSON response (unchanged)
Mutation, // request → single JSON response (unchanged)
Sub, // client subscribes, server streams (was Subscription)
Pub, // client publishes a stream to the server (new)
}
```
Wire `op_type` strings: `"query"`, `"mutation"`, `"sub"`, `"pub"`.
The old `"subscription"` string is **not** accepted on input — no
deployments exist, no backward-compat shim. Clients that sent
`"subscription"` were never written.
### 2. `OperationType::Pub` — client→server streaming
`Pub` is the directional complement to `Sub`. Both are streaming
operations, but the stream flows the opposite direction:
| `OperationType` | Stream direction | Initiator role | Responder role | Handler kind |
|-----------------|-----------------|---------------|----------------|-------------|
| `Sub` (was `Subscription`) | server → client | consumer (subscribes) | producer (streams) | `Stream` (`StreamingHandler`) |
| `Pub` (new) | client → server | producer (publishes) | consumer (receives) | `Sink` (`SinkHandler`) |
`Sub` and `Pub` are independent operations with independent specs and
ACLs. A peer that may publish a stream is not the same grant as a peer
that may subscribe to one. The op-type split enables the hub-as-proxy
pattern: a worker publishes to the hub, the hub owns the resource, and
consumers subscribe from the hub (research findings §hub-as-proxy). But
that pattern is the broker layer (deferred); the primitive is just the
two directions.
### 3. `SinkHandler` type and `HandlerKind::Sink`
A new handler kind consumes the initiator's stream and returns a single
`ResponseEnvelope`:
```rust
/// Sink handler — `Pub` operations. Receives the initiator's data
/// stream and returns a single `ResponseEnvelope` (the result of
/// consuming the stream). Each `Ok(value)` published by the initiator
/// arrives as an item in the `RecvStream`; the handler processes them
/// and returns one result (success or error).
pub type SinkHandler = Arc<
dyn Fn(
Value,
OperationContext,
Pin<Box<dyn Stream<Item = Value> + Send>>,
) -> Pin<Box<dyn Future<Output = ResponseEnvelope> + Send>>
+ Send
+ Sync,
>;
pub enum HandlerKind {
Once(Handler),
Stream(StreamingHandler),
Sink(SinkHandler),
}
```
The `Pin<Box<dyn Stream<Item = Value> + Send>>` is the initiator's
published stream — each `call.published` event's `payload.input` yields
one `Value` item. The handler consumes the stream to completion, then
returns a single `ResponseEnvelope`. An `Err` in the stream (a
`call.error` from the initiator) terminates the stream early; the
handler may produce its `ResponseEnvelope` from the partial input or
return the initiator's error.
Registration validates: `Pub` → `HandlerKind::Sink`. Mismatch is a
startup error (same pattern as ADR-021's `Subscription` → `Stream`
validation).
A `make_sink_handler()` helper (analogue of `make_handler()` /
`make_streaming_handler()`) wraps a sink-consuming closure into a
`SinkHandler`.
### 4. `OperationSpec` gains `publish_schema`
`Pub` operations have an input stream, not a single input. The
`input_schema` validates the initial `call.requested` payload's `input`
(the "open" parameters — e.g., the file path for `fs/upload`). A new
optional `publish_schema` validates each `call.published` chunk's
`input` (the stream items — e.g., each file chunk):
```rust
pub struct OperationSpec {
// ... existing fields ...
pub input_schema: Value, // validates call.requested input (the open params)
pub output_schema: Value, // validates call.responded output (the result)
/// Schema for each published chunk's `input` (Pub ops only).
/// `None` for Query/Mutation/Sub ops. When set, the dispatch
/// path validates each `call.published` event's payload.input
/// against this schema before yielding it to the SinkHandler.
/// When `None` (Pub op with no per-chunk validation), chunks
/// are yielded as-is.
pub publish_schema: Option<Value>,
}
```
`OperationSpec::new()` defaults `publish_schema` to `None`
(Query/Mutation/Sub). A builder method or direct field construction sets
it for `Pub` ops. This mirrors how `resource_id_path` is `Option` and
defaults to `None` — additive, non-breaking for existing specs.
### 5. New wire event: `call.published`
A sixth event type carries the initiator's stream chunks:
| Event | Direction | Purpose |
|-------|-----------|---------|
| `call.requested` | initiator → responder | Open the Pub (operationId + initial input) |
| `call.published` | initiator → responder | One chunk of the published stream |
| `call.responded` | responder → initiator | The result (single, after stream ends) |
| `call.completed` | responder → initiator | (Not used for Pub — the `call.responded` is the terminal event) |
| `call.aborted` | either side | Cancel the Pub |
| `call.error` | either side | Error (initiator publish error or responder handler error) |
`call.published` payload:
```json
{
"input": <Value>
}
```
The `input` is one published chunk, validated against `publish_schema`
(if set). The `id` field is the request ID (same correlation key as
`call.requested`).
AGENTS.md §7: "New event types may be added; existing ones must not
change shape." `call.published` is additive — existing event types are
unchanged.
**Stream lifecycle for Pub:**
1. Initiator sends `call.requested` with `operationId`, `input` (open
params).
2. Initiator sends zero or more `call.published` events with the same
`id`, each carrying one chunk.
3. Initiator signals stream end by sending a `call.completed` event
(reusing the existing event type in the initiator→responder
direction — it signals "I'm done publishing"). Alternatively, the
initiator closes the write half of the stream (transport-level EOF).
The responder detects end-of-stream and delivers `None` to the
`SinkHandler`.
4. Responder's `SinkHandler` consumes the stream, returns one
`ResponseEnvelope`.
5. Responder sends `call.responded` (the result) or `call.error`.
**Why `call.completed` initiator→responder instead of write-half close?**
Both work. `call.completed` is explicit and works over transports where
write-half close is ambiguous (WebSocket, Worker `postMessage`). The
dispatch path treats either as stream end. The client API
(`CallConnection::publish`) sends `call.completed` by default; the
transport-agnostic `Connection::open_bi` write-half close is the
fallback.
### 6. `OperationRegistry::invoke_sink()` dispatch path
```rust
impl OperationRegistry {
/// Dispatch a Pub operation. The `publish_stream` is the
/// initiator's data stream (each `call.published` chunk's input).
/// Pre-handler errors (not-found, forbidden, invalid operation
/// type) yield a single error `ResponseEnvelope`.
pub async fn invoke_sink(
&self,
name: &str,
input: Value,
publish_stream: Pin<Box<dyn Stream<Item = Result<Value, CallError>> + Send>>,
context: OperationContext,
) -> ResponseEnvelope;
}
```
`invoke_sink()` performs the same visibility + ACL checks as `invoke()`
and `invoke_streaming()`, then dispatches to the `SinkHandler` with the
`publish_stream`. Pre-handler errors (not-found, forbidden,
`INVALID_OPERATION_TYPE` for a non-Pub op) return a single error
`ResponseEnvelope`.
The `publish_stream` carries `Result<Value, CallError>` — `Ok(value)`
for a published chunk, `Err(call_error)` for an initiator-side error
(a `call.error` event from the initiator). An `Err` terminates the
stream; the handler may produce its result from the partial input or
propagate the error.
`invoke()` on a `Pub` op and `invoke_streaming()` on a `Pub` op both
return `INVALID_OPERATION_TYPE` (same pattern as ADR-021's
`invoke()` on `Subscription`). `OperationEnv::invoke()` (composition)
is request/response-only and errors with `INVALID_OPERATION_TYPE` on
`Pub` ops — sink composition (consume a child's stream) is a
handler-level concern, not a protocol composition concern (same
boundary as ADR-021's stream composition).
### 7. `Dispatcher::handle_stream` sink branch
`DispatchResult` gains a `Sink` variant:
```rust
pub enum DispatchResult {
Once(ResponseEnvelope),
Stream(ResponseStream),
Sink(Pin<Box<dyn Stream<Item = Result<Value, CallError>> + Send>>),
}
```
`dispatch()` branches on `op_type`:
- `Query` / `Mutation` → `invoke()` → `DispatchResult::Once`
- `Sub` → `invoke_streaming()` → `DispatchResult::Stream`
- `Pub` → `invoke_sink()` → `DispatchResult::Sink`
`handle_stream` gains a sink branch: after dispatching a
`call.requested` for a `Pub` op, the loop continues reading
`call.published` events (same `id`), feeding each chunk's `input` into
the `publish_stream`. On `call.completed` (initiator→responder) or
stream read end, the `publish_stream` ends. The `SinkHandler` runs
concurrently, consuming the `publish_stream` and producing one
`ResponseEnvelope`, which is written to the wire as `call.responded` or
`call.error`.
The sink branch sets `deadline: None` for `Pub` ops (unbounded — the
stream may be long-lived, same as `Sub`).
### 8. `CallConnection::publish()` client method
```rust
impl CallConnection {
/// Publish a stream to a remote `Pub` operation. Sends
/// `call.requested` (open), then one `call.published` per item in
/// `stream`, then `call.completed` (stream end). Returns the
/// responder's single `ResponseEnvelope`.
pub async fn publish(
&self,
operation_id: &str,
input: Value,
stream: Pin<Box<dyn Stream<Item = Value> + Send>>,
) -> ResponseEnvelope;
}
```
The client opens a bidirectional stream, sends `call.requested`, pumps
the `stream` into `call.published` events, sends `call.completed`, then
reads the single `call.responded` / `call.error` response. The
`PendingRequestMap` correlates by request ID (same as `call()`).
### 9. `from_call` sink forwarding
The `from_call` forwarding handler construction branches on `op_type`
during discovery (ADR-021 §8 added the streaming branch; this adds the
sink branch):
- `Query` / `Mutation` → `make_forwarding_handler()` (single response),
`HandlerKind::Once`.
- `Sub` → `make_streaming_forwarding_handler()` (remote stream → local
stream), `HandlerKind::Stream`.
- `Pub` → `make_sink_forwarding_handler()` (local stream → remote
`call.published` events), `HandlerKind::Sink`. The forwarding handler
receives the local `publish_stream`, opens a `call.requested` +
`call.published` pump to the remote, and returns the remote's single
`ResponseEnvelope`.
A `from_call`-imported `Pub` op forwards the stream end-to-end: the
local `SinkHandler`'s stream is pumped to the remote as
`call.published` events; the remote's `call.responded` is the local
handler's result.
## Consequences
**Positive:**
- `Pub` operations work end-to-end: client-side `publish()` →
server-side `invoke_sink()` → `SinkHandler` → `call.responded`. The
directional complement to `Sub` is in place.
- The `Handler` / `StreamingHandler` / `SinkHandler` triple mirrors the
three operation shapes: request/response, server→client stream,
client→server stream. The `HandlerKind` enum (`Once` | `Stream` |
`Sink`) makes the "one handler kind per op type" invariant
type-level.
- `Sub` rename aligns the wire enum with the conceptual symmetry
(`sub`/`pub`). No deployments exist; the rename is free.
- `call.published` is additive — existing event types are unchanged,
per the stable wire-format constraint (AGENTS.md §7).
- `publish_schema` on `OperationSpec` is additive (`Option`, defaults
`None`) — existing specs are unchanged.
- The fan-out/broker is cleanly deferred. The `Pub` primitive is the
building block; the channels session composes the broker on top.
**Negative:**
- `OperationType::Subscription` → `Sub` is a rename across the codebase
(spec, registry, dispatch, discovery, tests, docs). Mechanical but
wide.
- `HandlerKind` gains a third variant. Existing code matching
`HandlerKind` must handle `Sink`. The builder gains
`with_local_sink` / `with_leaf_sink` methods.
- `OperationSpec::new()` gains a `publish_schema` parameter (or a
builder method). Every spec-constructing site adds the field,
defaulting to `None`. This is a mechanical change.
- `call.published` is a new wire event type. Existing clients that
don't know about it ignore it (the dispatch loop's existing "ignore
unknown event types" behavior). Clients that want to `Pub` must be
updated; clients that only `call`/`subscribe` are unaffected.
- The `Dispatcher::handle_stream` sink branch adds a
publish-stream-pump alongside the existing response-stream-pump. New
code in the hot dispatch path, but it's a straightforward
`while let Some(chunk) = publish_stream.next().await` loop feeding
the `SinkHandler`.
## Door type
**One-way.** The `Handler` / `StreamingHandler` / `SinkHandler` /
`HandlerKind` API surface is what handlers are written against across
crates. Changing it after handlers exist is a rewrite. The
`call.published` wire event is one-way — once emitted, clients may
handle it, and removing it would break those handlers. The
`OperationType::Sub` rename is one-way (the wire string `"sub"` is now
the canonical form; `"subscription"` is gone).
The `HandlerKind` enum shape (`Once(Handler) | Stream(StreamingHandler)
| Sink(SinkHandler)`) is the one-way commitment: three handler variants,
validated against `op_type`. The concrete `Pin<Box<dyn Stream<Item =
Value> + Send>>` choice for the sink's input stream is a two-way-door
implementation detail within the one-way decision.
The `publish_schema: Option<Value>` field on `OperationSpec` is a
one-way addition (the field is part of the spec's public API); its
`Option`/`None`-default keeps it non-breaking for existing specs.
## Fan-out (deferred — OQ-22 §fan-out scope)
The fan-out / broker mechanism (one producer, N consumers, topic
matching by `(op_name, params_hash)`) is **not** part of this ADR. It
is deferred to the channels session for these reasons:
1. **The call protocol is point-to-point.** One initiator, one
responder, a stream between them. A topic registry that outlives
individual calls is a different lifecycle and a different concern.
2. **The broker's first consumer is channels.** The
`channel/resources/subscribe` operation (ADR-037) and the
hub-as-broker pattern (ADR-042) are channels-layer concerns. The
broker naturally lives there.
3. **Designing the broker blind produces a half-design.** The research
findings' Gap B explicitly calls out that the broker is "a new
component with no spec." Speccing it without its first real consumer
(channels isn't implemented yet) risks a design that channels has to
work around rather than compose with.
The `Pub` primitive this ADR adds is the load-bearing piece the broker
composes on. The channels session will add the broker (topic registry,
`Pub`↔`Sub` matching, fan-out) on top — the same way the hub relay
(ADR-042) composes on top of existing call ops.
OQ-22 is resolved for the primitive (the `Pub` op type + handler + wire
event). The fan-out scope remains open as a channels-session concern;
the OQ entry is updated to reflect this split.
## References
- ADR-021: Streaming Handler for Subscriptions (the `StreamingHandler`
/ `HandlerKind::Stream` / `invoke_streaming()` pattern this ADR
mirrors for `SinkHandler` / `HandlerKind::Sink` / `invoke_sink()`)
- ADR-015: Call Protocol Stream Model (the five event types; this ADR
adds the sixth, `call.published`)
- ADR-014: Hand-Rolled EventEnvelope Framing (the wire format this ADR
extends)
- ADR-018: Handler Registration, Provenance, and Composition Authority
(`HandlerRegistration` gains the `Sink` variant)
- ADR-017: Privilege Model and Authority Context (visibility/ACL checks
run identically in `invoke_sink()` as in `invoke()` /
`invoke_streaming()`)
- ADR-020: Abort Cascade for Nested Calls (stream drop on abort; cascade
through sink handlers)
- ADR-032: One-Way Door Decision Framework (the `SinkHandler` split is a
one-way door — handler API surface)
- `@alkdev/pubsub/src/create_pubsub.ts` — the `PubSub` type with
`publish` / `subscribe` (the in-process `EventTarget` dispatch the
wire-level `Pub` primitive complements)
- `/workspace/@alkdev/alknet/docs/research/call-channels-unification/
findings.md` — the research that identified the pub/sub gap, Gap A
(handler shape), and the fan-out deferral
- OQ-22: Call protocol pub/sub primitive (resolved for the primitive by
this ADR; fan-out scope deferred to channels)
- Spec documents amended: `call-protocol.md`, `operation-registry.md`,
`open-questions.md`, `README.md`