diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 4a011b8..3a13898 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -7,6 +7,65 @@ last_updated: 2026-07-12 ## Current State +**alknet-channels specs drafted.** The alknet-channels crate (multiplexing +proxy — `ProtocolHandler` on `alknet/channels`, 9-byte chunk format, N +channels over one transport stream, channel 0 pre-negotiated as +`alknet/call`) now has architecture specs: +[crates/channels/](crates/channels/) (overview, channels-wire, +channels-connection, channels-adapter, channel-operations, channel-client) +and ten ADRs — [ADR-071](decisions/071-channels-wire-format.md) (9-byte +chunk header), [ADR-072](decisions/072-channel-0-pre-negotiated-call.md) +(channel 0 = `alknet/call` pre-negotiated, no special control plane), +[ADR-073](decisions/073-channel-lifecycle-operations.md) (channel +lifecycle operations on the call protocol — `channel/open`/`close`/ +`control`/`resources/subscribe`; `channel/resources/subscribe` is a +`Subscription` operation using the already-implemented `StreamingHandler` +machinery, not a polled `Query`; the `direction` field pins who is the +ALPN-server; the control-message division is call-ops for orchestration, +`stream_type 3` for data-ordered control), +[ADR-074](decisions/074-channelconnection-bidistreamsource.md) +(`ChannelBidiStreamSource` implements `BidiStreamSource` — ADR-070's +extension point; `into_sub_streams()` typed accessor for handlers that need +stderr/control; `accept_bi()` generic path for tunnel/SSH), +[ADR-075](decisions/075-channelsadapter-and-channelmanager.md) +(`ChannelsAdapter` read/demux + `ChannelManager` reassemble/allocate split; +REQ-CH-01..04 wire-level invariants pinned: shutdown emits zero-length +sentinel, transport close drops all senders, mux dynamic registration, +lenient unknown-`channel_id`), +[ADR-076](decisions/076-backpressure-channel-limits-id-reuse.md) +(bounded-buffer backpressure 1 MiB default, 256-channel cap, monotonic IDs +with wrap-around), +[ADR-077](decisions/077-tty-inside-channels.md) (TTY inside channels uses +sub-streams, not its own 5-byte wire format; ADR-052's scope amended to +direct-connect TTY only; `channels` feature on alknet-tty), +[ADR-078](decisions/078-two-pump-shutdown-on-completion.md) (two-pump +handlers MUST shut down the opposite sink on pump completion — the +deadlock contract the POC surfaced; handler-level, not channels-layer; +core helper extraction deferred per OQ-57), +[ADR-079](decisions/079-hub-relay-translate-not-forward.md) (hub relay +translates `channel/open` on channel 0 with `forwarded_for` — ADR-032; +data channels byte-forwarded with `channel_id` rewrite; the hub never runs +protocol-specific handlers), +[ADR-080](decisions/080-channelclient.md) (`ChannelClient` in +alknet-channels, QUIC-only initially, bidirectionality preserved; +`AlknetClient` core extraction stays deferred per OQ-55 — blocked on a +second *transport's* client, not a second client). The specs are grounded +in the completed de-risk POC +(`docs/research/alknet-channels/poc-summary.md`, 28 tests passing, three +validated targets: chunk format + demux/mux, per-channel `Connection` +presentation, tunnel handler). The core prerequisite — ADR-070 +(`BidiStreamSource` trait + `Connection::from_source`) — is landed and +implemented. The spec work converted three research hedges into decisions: +`channel/resources` is subscribe from day one (not poll-for-v1), channel +ID allocation is server-assigned (not "if zero-RTT needed"), and +backpressure is bounded-buffer (not "if HOL blocking becomes a problem"). +Two genuine deferrals: OQ-56 (full windowing — blocked on a real HOL- +blocking observation) and OQ-57 (two-pump helper extraction — blocked on a +second two-pump handler). The TTY integration (ADR-077) amends ADR-052's +scope — the 5-byte format is unchanged for direct `alknet/tty` connections; +inside channels, TTY uses `into_sub_streams()` and the channels layer's +de-chunking. + **Pre-implementation of the storage/repo pattern.** The project has completed a pivot from a three-layer model to an ALPN-as-service model. The greenfield workspace contains `alknet-vault` (stable — implementation complete and verified, local-only by construction per ADR-025, HD-derivation key model per ADR-026) and research/reference material. Foundational ADRs (001–035) are in place, with the call crate implemented and reviewed. The storage and auth strategy research (`docs/research/alknet-storage-strategy/findings.md`) surfaced the repo/adapter pattern as the answer to cross-node state (peer identity, credentials). This has now landed as four ADRs: @@ -98,6 +157,13 @@ adapter location map is now consistent: all HTTP-backed adapters | [crates/vault/service.md](crates/vault/service.md) | stable | VaultServiceHandle lifecycle, direct dispatch, cache, error model | | [crates/vault/protocol.md](crates/vault/protocol.md) | stable | DerivedKey redaction, KeyType, serialization behavior | | [crates/hub/README.md](crates/hub/README.md) | draft | alknet-hub crate — aggregated peer env, connection lifecycle, worker supervision, service discovery | +| [crates/channels/README.md](crates/channels/README.md) | draft | alknet-channels crate — multiplexing proxy, 9-byte chunk format, N channels over one transport stream | +| [crates/channels/overview.md](crates/channels/overview.md) | draft | Crate purpose, the multiplexing collapse, dependencies, transport agnosticism, WASM, relationship to existing crates | +| [crates/channels/channels-wire.md](crates/channels/channels-wire.md) | draft | 9-byte chunk format, stream types, sentinels, framing disambiguation, wire-level invariants (REQ-CH-01..05) | +| [crates/channels/channels-connection.md](crates/channels/channels-connection.md) | draft | `ChannelBidiStreamSource` (implements `BidiStreamSource`), `into_sub_streams()` typed accessor, recursive composition | +| [crates/channels/channels-adapter.md](crates/channels/channels-adapter.md) | draft | `ChannelsAdapter`, `ChannelManager`, demux/mux contracts (REQ-CH-01..04), two-pump pattern (ADR-078) | +| [crates/channels/channel-operations.md](crates/channels/channel-operations.md) | draft | `channel/open`/`close`/`control`/`resources/subscribe`, ACL flow, `direction` semantics, hub relay contract (ADR-079) | +| [crates/channels/channel-client.md](crates/channels/channel-client.md) | draft | `ChannelClient` — client side of a channels connection, QUIC-only initially, bidirectionality preserved | ## ADR Table @@ -173,6 +239,16 @@ adapter location map is now consistent: all HTTP-backed adapters | [068](decisions/068-peer-composite-env-peer-operations.md) | PeerCompositeEnv::peer_operations Override | Proposed | | [069](decisions/069-from-call-manual-free-function.md) | from_call Is a Manual Free Function, Not Auto-Wired | Proposed | | [070](decisions/070-bidistreamsource-trait.md) | BidiStreamSource Trait — Open Connection for Extension | Accepted | +| [071](decisions/071-channels-wire-format.md) | alknet-channels Wire Format — 9-Byte Chunk Header | Accepted | +| [072](decisions/072-channel-0-pre-negotiated-call.md) | Channel 0 Is Pre-Negotiated `alknet/call` | Accepted | +| [073](decisions/073-channel-lifecycle-operations.md) | Channel Lifecycle Operations on the Call Protocol | Accepted | +| [074](decisions/074-channelconnection-bidistreamsource.md) | ChannelConnection — BidiStreamSource over Chunk Reassembly | Accepted | +| [075](decisions/075-channelsadapter-and-channelmanager.md) | ChannelsAdapter and ChannelManager | Accepted | +| [076](decisions/076-backpressure-channel-limits-id-reuse.md) | Backpressure, Channel Limits, and ID Reuse | Accepted | +| [077](decisions/077-tty-inside-channels.md) | TTY Inside Channels — Sub-Streams, Not Wire Format | Accepted (amends ADR-052 scope — 5-byte format scoped to direct TTY) | +| [078](decisions/078-two-pump-shutdown-on-completion.md) | Two-Pump Shutdown-on-Completion Pattern | Accepted | +| [079](decisions/079-hub-relay-translate-not-forward.md) | Hub Relay — Translate, Not Transparently Forward | Accepted | +| [080](decisions/080-channelclient.md) | ChannelClient — the Client Side of a Channels Connection | Accepted | ## Open Questions diff --git a/docs/architecture/crates/channels/README.md b/docs/architecture/crates/channels/README.md new file mode 100644 index 0000000..0af2232 --- /dev/null +++ b/docs/architecture/crates/channels/README.md @@ -0,0 +1,123 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# alknet-channels + +A multiplexing proxy: a `ProtocolHandler` on `alknet/channels` that +decomposes a single bidirectional transport stream into N logical channels, +each carrying a different ALPN. Channel 0 is pre-negotiated as `alknet/call` +(ADR-072); every other channel is opened dynamically via call operations on +channel 0 and routed through the same `HandlerRegistry` as top-level +connections. The channels layer is a re-framing proxy — it converts between +"one transport stream carrying N channels" (the wire) and "N independent +`AsyncRead + AsyncWrite` handles" (what handlers see) — and it does no +protocol work itself. + +## Documents + +| Document | Status | Description | +|----------|--------|-------------| +| [overview.md](overview.md) | draft | Crate purpose, the multiplexing collapse, dependencies, ALPN, transport agnosticism, WASM, relationship to existing crates | +| [channels-wire.md](channels-wire.md) | draft | The 9-byte chunk format (`[channel_id:u32 be][stream_type:u8][length:u32 be][payload]`), stream types, sentinels, framing disambiguation, wire-level invariants (REQ-CH-01..05) | +| [channels-connection.md](channels-connection.md) | draft | `ChannelBidiStreamSource` (implements `BidiStreamSource` — ADR-070/074), `into_sub_streams()` typed destructure, recursive composition | +| [channels-adapter.md](channels-adapter.md) | draft | `ChannelsAdapter` (`ProtocolHandler` on `alknet/channels`), `ChannelManager`, demux/mux contracts (REQ-CH-01..04), the two-pump pattern (ADR-078) | +| [channel-operations.md](channel-operations.md) | draft | `channel/open`, `channel/close`, `channel/control`, `channel/resources/subscribe` — call-protocol operations on channel 0, ACL flow, `direction` semantics, the hub relay contract (ADR-079) | +| [channel-client.md](channel-client.md) | draft | `ChannelClient` — the client side of a channels connection, QUIC-only initially, bidirectionality preserved | + +## Applicable ADRs + +| ADR | Title | Relevance | +|-----|-------|-----------| +| [071](../../decisions/071-channels-wire-format.md) | channels Wire Format — 9-Byte Chunk Header | The chunk format; one-way door | +| [072](../../decisions/072-channel-0-pre-negotiated-call.md) | Channel 0 Is Pre-Negotiated `alknet/call` | Channel 0 = call protocol, no special control plane | +| [073](../../decisions/073-channel-lifecycle-operations.md) | Channel Lifecycle Operations on the Call Protocol | `channel/open`/`close`/`control`/`resources/subscribe`; `direction` semantics; subscribe not poll | +| [074](../../decisions/074-channelconnection-bidistreamsource.md) | ChannelConnection — BidiStreamSource over Chunk Reassembly | Per-channel `BidiStreamSource` impl; `into_sub_streams()` accessor | +| [075](../../decisions/075-channelsadapter-and-channelmanager.md) | ChannelsAdapter and ChannelManager | The read/demux + reassemble/allocate split; REQ-CH-01..04 contracts | +| [076](../../decisions/076-backpressure-channel-limits-id-reuse.md) | Backpressure, Channel Limits, and ID Reuse | Bounded-buffer (1 MiB default), 256-channel cap, monotonic IDs with wrap | +| [077](../../decisions/077-tty-inside-channels.md) | TTY Inside Channels — Sub-Streams, Not Wire Format | TTY's two modes (direct vs channels); amends ADR-052 scope | +| [078](../../decisions/078-two-pump-shutdown-on-completion.md) | Two-Pump Shutdown-on-Completion Pattern | The two-pump deadlock contract; handler-level, not channels-layer | +| [079](../../decisions/079-hub-relay-translate-not-forward.md) | Hub Relay — Translate, Not Transparently Forward | The hub translates channel 0, byte-forwards data channels with ID rewrite | +| [080](../../decisions/080-channelclient.md) | ChannelClient — the Client Side of a Channels Connection | `ChannelClient` in alknet-channels, QUIC-only; `AlknetClient` deferred (OQ-55) | +| [070](../../decisions/070-bidistreamsource-trait.md) | BidiStreamSource Trait | The `Connection` extension point `ChannelBidiStreamSource` implements | +| [065](../../decisions/065-connection-from-stream-generic-single-stream.md) | `Connection::from_stream` | The transport-agnostic `Connection` the channels layer rides on | +| [052](../../decisions/052-alknet-tty-wire-format-and-two-carriage.md) | alknet-tty Wire Format | The 5-byte format the 9-byte format generalizes (amended by ADR-077 — scoped to direct TTY) | +| [049](../../decisions/049-streaming-handler-for-subscriptions.md) | StreamingHandler for Subscriptions | The machinery `channel/resources/subscribe` uses | +| [032](../../decisions/032-forwarded-for-identity.md) | Forwarded-For Identity | The auth chain for hub-relayed channel opens | +| [003](../../decisions/003-crate-decomposition.md) | Crate Decomposition | alknet-channels depends on alknet-core only; no handler-depends-on-handler | + +## Relevant Open Questions + +| OQ | Title | Status | Relevance | +|----|-------|--------|-----------| +| OQ-55 | AlknetClient / Client Establishment Extraction | deferred(scope) | `ChannelClient` is decided (ADR-080); `AlknetClient` core extraction stays deferred — blocked on a second *transport's* client, not a second client | +| OQ-56 | Full channel-level flow-control windowing | deferred(scope) | Bounded-buffer is decided (ADR-076); full windowing is an extension blocked on "a real deployment observes HOL blocking on a saturated channel where bounded buffer is insufficient" | +| OQ-57 | Two-pump helper extraction to alknet-core | deferred(scope) | The shutdown-on-completion *contract* is decided (ADR-078); the *helper* extraction is blocked on a second two-pump handler existing (shape convergence) | + +## Key Design Principles + +1. **Streams are streams.** A TTY session, an SSH channel, a forwarded TCP + connection, a QUIC bidi stream — they're all `AsyncRead + AsyncWrite` + handles. The differences are only in how they're *opened* (negotiation + via `channel/open` on channel 0) and what *multiplexing layer* carries + them (the 9-byte chunk format). Once normalized, every channel is an + ALPN routed through the same `HandlerRegistry`. See + [overview.md](overview.md) and ADR-071. + +2. **Channel 0 is `alknet/call` pre-negotiated, not a special control + plane.** The call protocol runs on channel 0 exactly as on a top-level + `alknet/call` connection. Channel lifecycle operations + (`channel/open`, `channel/close`, `channel/control`, + `channel/resources/subscribe`) are call operations on channel 0's + `OperationRegistry`, gated by the existing `AccessControl::check`. No + new auth machinery, no new framing. See ADR-072, ADR-073. + +3. **The channels layer is a re-framing proxy, not a protocol engine.** It + converts between "one transport stream carrying N channels" (the wire) + and "N independent `AsyncRead + AsyncWrite` handles" (what handlers + see). It does no ALPN-specific parsing, no auth, no transport coupling. + This makes it WASM-compatible and transport-agnostic by construction. + See [channels-adapter.md](channels-adapter.md) and ADR-075. + +4. **`channel/resources/subscribe` is a `Subscription`, not a polled + `Query`.** The call protocol has `StreamingHandler` / `invoke_streaming` + (ADR-049, implemented and tested). The first consumer (the hub + aggregating worker resources) needs live updates. Polling would be built + and immediately reworked. See ADR-073. + +5. **Bidirectional open.** Either side can open a channel to the other, + just like the call protocol's operation overlay. The `direction` field + on `channel/open` pins who is the ALPN-server vs ALPN-client. See + ADR-073 §Direction semantics. + +6. **Wire-level invariants are contracts, not implementation details.** + The POC surfaced five invariants (REQ-CH-01..04, plus REQ-CH-06 for + close ordering) that hang channels silently if underspecified: shutdown + emits a zero-length sentinel; transport close drops all senders; the mux + supports dynamic registration; unknown `channel_id` is lenient-dropped; + bounded-buffer backpressure doesn't deadlock; data chunks flush before + `channel/close`. See [channels-wire.md](channels-wire.md) and + [channels-adapter.md](channels-adapter.md). + +7. **The hub translates, not transparently forwards.** The hub terminates + channel 0 on both legs, runs `AccessControl::check`, and re-issues + `channel/open` on the spoke leg with `forwarded_for` (ADR-032). Data + channels are byte-forwarded with `channel_id` rewrite. This preserves + the auth model. See ADR-079. + +## References + +- `docs/research/alknet-channels/phase-0-findings.md` — Phase 0 research + (vision, hub motivation, wire format, negotiation, internals, DPs, OQs) +- `docs/research/alknet-channels/poc-summary.md` — the de-risk POC (28 + tests, three validated targets, REQ-CH-01..06 wire-level invariants + surfaced; REQ-CH-07 is a cosmetic clippy item, not a wire invariant) +- `docs/research/alknet-channels/poc-plan.md` — the POC plan +- `/workspace/alknet-channels-poc/` — the POC codebase +- `docs/research/alknet-tty/phase-0-findings.md` — the TTY crate's chunk + format (the seed of the channels generalization) +- `docs/research/alknet-ssh/phase-0-findings.md` — SSH's channel + multiplexer (the prior art for N-channel multiplexing) +- `docs/architecture/crates/hub/README.md` — the hub crate (the primary + consumer; the relay implementation's home) \ No newline at end of file diff --git a/docs/architecture/crates/channels/channel-client.md b/docs/architecture/crates/channels/channel-client.md new file mode 100644 index 0000000..9506a26 --- /dev/null +++ b/docs/architecture/crates/channels/channel-client.md @@ -0,0 +1,150 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# channel-client.md — ChannelClient + +The client side of a channels connection. ADR-080 is the decision; this doc +specifies the API. + +## What + +`ChannelClient` is the symmetric counterpart to `ChannelsAdapter` (ADR-075). +The server side is a `ProtocolHandler` (`ChannelsAdapter::handle`); the +client side dials a transport, establishes the channels connection, runs the +demux/mux, and exposes `open_channel(alpn, params) -> Channel` to the +application. + +This is the channels analogue of `CallClient` (server: `CallAdapter`; +client: `CallClient`) in the call protocol. + +## API + +```rust +pub struct ChannelClient { + manager: ChannelManager, + // The transport-side demux/mux, running in a background task. + ... +} + +impl ChannelClient { + /// Open a channels connection to a peer. Dials the transport (QUIC + /// initially), establishes the channels connection, preinstalls + /// channel 0 (alknet/call), and returns the client. + pub async fn connect(addr: SocketAddr, credentials: CallCredentials) + -> Result; + + /// Open a data channel with the given ALPN and params. Sends + /// `channel/open` on channel 0, waits for the response, and returns + /// the channel. + pub async fn open_channel( + &self, + alpn: &str, + stream_types: &[u8], + params: Value, + direction: ChannelDirection, + ) -> Result; + + /// Subscribe to the peer's resource updates. Returns a stream of + /// resource-set events (ADR-073 channel/resources/subscribe). Each + /// event carries the JSON `output.resources` array from ADR-073's + /// `channel/resources/subscribe` response shape. + pub async fn subscribe_resources(&self) + -> Result, ChannelError>; + + /// The call-protocol connection on channel 0, for invoking channel + /// lifecycle operations and any other call ops the peer exposes. + pub fn call(&self) -> &CallConnection; +} + +pub enum ChannelDirection { + InitiatorToResponder, + ResponderToInitiator, +} + +pub struct Channel { + pub channel_id: u32, + pub stream_types: Vec, + /// The sub-streams, accessible via accept_bi() (ADR-074 generic path) + /// or into_sub_streams() (ADR-074 typed path). + pub source: ChannelBidiStreamSource, +} + +/// One event from `channel/resources/subscribe`. Wraps the JSON `output` +/// object from ADR-073's subscribe response — the `resources` array +/// describing what ALPNs the peer exposes and with what `access` preview. +/// The channels crate maps the JSON to this typed struct; the fields mirror +/// ADR-073's response shape. +pub struct ResourceEvent { + pub resources: Vec, +} + +pub struct ResourceEntry { + pub alpn: String, + pub backends_or_targets: Vec, // ALPN-specific enumeration + pub access: Value, // preview of AccessControl (advisory) +} +``` + +## QUIC-only initially + +`ChannelClient::connect` dials a QUIC connection (via the same `quinn` +endpoint `CallClient` uses) and wraps it as a channels connection. This is +the same transport shape as `CallClient`. When a second transport's client +exists (HTTP, TCP+TLS, WebTransport — per OQ-55), the dial can be +generalized. Until then, `ChannelClient` is QUIC-only — the same posture as +`CallClient`. + +## Bidirectionality preserved + +The channels protocol is bidirectional — either side can open a channel +(ADR-073 §direction semantics). `ChannelClient::open_channel` supports both +`ChannelDirection::InitiatorToResponder` and +`ChannelDirection::ResponderToInitiator`. The client is not "the client +side" in the request/response sense — it can also receive `channel/open` +requests from the peer (the peer initiates, the client's `ChannelManager` +responds). This mirrors the call protocol's operation overlay (each side +populates what operations they expose). + +`ChannelClient` is one endpoint of a bidirectional channels connection. The +name follows the `CallClient` convention (the side that dialed), not a +request/response role. + +## Relationship to `AlknetClient` (OQ-55 — deferred) + +`ChannelClient` is a standalone client, not a specialization of a core +`AlknetClient`. The `AlknetClient` extraction (OQ-55) is genuinely deferred: +blocked on a second *transport's* client existing, not on a second client +existing. `ChannelClient` over QUIC is a second client but the same +transport shape as `CallClient` — it doesn't give enough information to +extract the transport-polymorphic dial seam. + +When `AlknetClient` is eventually extracted (after a second transport's +client exists), `ChannelClient` and `CallClient` both refactor onto it. +Until then, they are independent clients with duplicated boilerplate (each +rebuilds verifier selection — ~20 lines). The friction is duplicated +boilerplate, not a missing capability. + +## Design Decisions + +All design decisions are documented as ADRs in [decisions/](../../decisions/). + +| ADR | Decision | Summary | +|-----|----------|---------| +| [080](../../decisions/080-channelclient.md) | ChannelClient | Client side; QUIC-only; `AlknetClient` deferred (OQ-55) | + +## Open Questions + +- **OQ-55** (deferred(scope)): `AlknetClient` core extraction — blocked on + a second *transport's* client. `ChannelClient` does not unblock it. + +## References + +- ADR-080: ChannelClient (the decision) +- ADR-073: channel lifecycle operations (`open_channel` sends `channel/open`) +- ADR-074: ChannelBidiStreamSource (what `Channel.source` wraps) +- ADR-075: ChannelManager (the shared state `ChannelClient` holds) +- OQ-55: AlknetClient / client establishment extraction +- `docs/architecture/crates/call/client-and-adapters.md` — `CallClient` (the + shape `ChannelClient` mirrors) \ No newline at end of file diff --git a/docs/architecture/crates/channels/channel-operations.md b/docs/architecture/crates/channels/channel-operations.md new file mode 100644 index 0000000..9ba23b6 --- /dev/null +++ b/docs/architecture/crates/channels/channel-operations.md @@ -0,0 +1,271 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# channel-operations.md — Channel Lifecycle on the Call Protocol + +Channel lifecycle is orchestrated by the call protocol on channel 0 +(ADR-072). Four operations on channel 0's `OperationRegistry` (ADR-073) +handle open, close, control, and resource discovery. All four go through +the existing `OperationContext` / `AccessControl::check` path — no new auth +machinery, no new framing. + +## The four operations + +### `channel/open` — open a data channel + +Request (on channel 0): + +```json +{ + "operation": "channel/open", + "input": { + "alpn": "alknet/tty", + "stream_types": [0, 1, 2, 3], + "params": { "backend": "docker", "cmd": ["bash"], "container": "abc123" }, + "direction": "initiator-to-responder" + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `alpn` | string | The ALPN the channel will carry. Responder looks this up in its `HandlerRegistry`. | +| `stream_types` | `[u8]` | Which sub-stream types this channel will use. E.g. `[0,1,2,3]` for TTY, `[0,1]` for a tunnel. | +| `params` | object | ALPN-specific parameters. For `alknet/tty` this is `NegotiateRequest`. For `alknet/tunnel` this is the target resource. The channels layer does not interpret `params`. | +| `direction` | string | `initiator-to-responder` or `responder-to-initiator`. See "Direction semantics" below. | + +Response: + +```json +{ + "output": { + "channel_id": 7, + "stream_types": [0, 1, 2, 3] + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `channel_id` | u32 | Server-assigned (DP-1). The responder allocates via monotonic `AtomicU32`. | +| `stream_types` | `[u8]` | The negotiated set — the responder may narrow the initiator's requested set. | + +**Channel ID allocation: server-assigned (DP-1).** One round-trip before +data flows — the same round-trip the call protocol makes for every +operation. All current channel types (TTY, tunnel, SSH) already require a +negotiation round-trip, so the open round-trip is not additive latency. + +**Error codes** (new `CallError.code` strings, not new framing): + +| code | meaning | retryable | +|------|---------|-----------| +| `channel:unknown_alpn` | ALPN not in responder's `HandlerRegistry` | false | +| `channel:forbidden` | `AccessControl::check` denied the open | false | +| `channel:allocation_failed` | Handler allocate failed | true (often transient) | +| `channel:invalid_params` | `params` JSON didn't satisfy the ALPN's expectations | false | +| `channel:too_many_channels` | Per-connection channel limit hit (ADR-076) | false | +| `channel:stream_type_unavailable` | Responder can't provide a requested `stream_type` | false | + +### `channel/close` — tear down a channel + +```json +{ + "operation": "channel/close", + "input": { "channel_id": 7, "reason": "exit" } +} +``` + +The responder (the side that didn't send the close) drains its reassembled +streams for `channel_id`, signals EOF to the handler, and returns +`{ "closed": true }`. The `channel_id` is eligible for reuse after the drain +completes (ADR-076 — monotonic IDs with wrap-around, not a free-list). +`reason` is free-form for observability — not semantically required. + +**REQ-CH-06: exit-chunk-before-close ordering.** The channel's data chunks +MUST be written and flushed before the `channel/close` operation is sent on +channel 0. The side closing must observe the data-channel pump complete +before issuing the call operation. For TTY this is the exit-chunk-is-last +invariant (ADR-055) carried forward; for tunnels it is the last data byte +before close. This invariant crosses two channels (the data channel and +channel 0), so the channels layer owns the ordering guarantee. + +### `channel/control` — out-of-band control on channel 0 + +For control that doesn't need ordering relative to data (resize, signal, +keepalive): + +```json +{ + "operation": "channel/control", + "input": { + "channel_id": 7, + "stream_type": 3, + "message": { "type": "resize", "cols": 80, "rows": 24 } + } +} +``` + +The channels layer routes `message` to the handler's control handle for +`channel_id`. The `message` JSON is ALPN-specific; the channels layer does +not interpret it. + +### `channel/resources/subscribe` — live resource discovery + +**This is a `Subscription` operation (ADR-049), not a polled `Query`.** The +call protocol has `StreamingHandler` / `invoke_streaming` (implemented and +tested). The first consumer (the hub aggregating worker resources) needs +live updates when workers connect/disconnect or containers start/stop. + +```json +{ + "operation": "channel/resources/subscribe", + "input": {} +} +``` + +The responder registers a `StreamingHandler` that emits a `ResponseEnvelope` +whenever the resource set changes. Each event: + +```json +{ + "output": { + "resources": [ + { + "alpn": "alknet/tty", + "backends": ["docker", "local"], + "access": { "required_scopes": ["tty:open"] } + }, + { + "alpn": "alknet/tunnel", + "targets": ["container:*", "service:postgres"], + "access": { "required_scopes_any": ["tunnel:open", "admin"] } + } + ] + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `alpn` | string | The ALPN this side accepts `channel/open` for. | +| `backends` / `targets` | `[string]` | ALPN-specific enumeration of what's available. The channels layer doesn't interpret these. | +| `access` | object | A preview of the `AccessControl` that `channel/open` will check. Advisory — lets the initiator fail fast. The real check happens on `channel/open`. | + +The stream emits an initial snapshot immediately, then subsequent events on +any change. The stream is long-lived; the subscriber cancels by dropping the +subscription (ADR-016 abort cascade applies). + +A `channel/resources` (non-subscribe, `Query`) operation is NOT provided. +The subscription's initial snapshot serves the poll use case (subscribe, +read the first event, cancel). Providing both would be redundant and would +pressure consumers toward the stale-poll path. + +## Direction semantics (OQ-CH-09 — pinned) + +Channel open is **bidirectional** — either side can initiate. The +`direction` field determines who is the ALPN-server (allocates the handler, +writes the negotiation response) vs the ALPN-client (writes the first +request). + +| `direction` | Initiator role | Responder role | Who writes first | +|-------------|----------------|----------------|-------------------| +| `initiator-to-responder` | ALPN-client | ALPN-server | Initiator writes first (the request data); responder's handler is the server side. The common case: "open me a TTY on your docker container." | +| `responder-to-initiator` | ALPN-server | ALPN-client | Responder writes first (the negotiation response); initiator's handler is the client side. The "worker exposes, hub consumes" case: the worker initiates the open to make itself available; the hub is the client. | + +**The channels layer does not enforce write order.** Write order is +ALPN-specific, determined by which side is the ALPN-server. The channels +layer routes chunks; the handlers negotiate who writes first via their +ALPN's `params` contract. + +**`channel_id` allocation is always by the responder** (DP-1), regardless of +`direction`. The responder is the side that receives the `channel/open` call +operation; it allocates the ID and returns it. In the `responder-to- +initiator` case, the initiator (worker) sends the `channel/open`, so the +responder (hub) allocates the ID — even though the worker is the ALPN-server +for the channel's data. This keeps ID allocation in one place and avoids the +collision-prone client-assigned alternative. + +## Control-message division (DP-4 — pinned) + +| Control path | When | Examples | +|--------------|------|----------| +| Call operations on channel 0 (`channel/control`, `channel/close`) | Control that doesn't need ordering relative to data, or lifecycle events | resize, signal, keepalive, close | +| `stream_type 3` chunks on the data channel | Control that MUST be ordered relative to data | EOF before exit, flush before close | + +The TTY crate's exit-chunk-is-last invariant (ADR-055) is the canonical +example of data-ordered control — it rides on `stream_type 3` because it +must arrive after the last stdin chunk, guaranteed by chunk ordering within +`(channel_id, stream_type)`, not by a call-protocol round-trip. The +`channel/close` operation that follows is on channel 0 and is ordered after +the data pump completes (REQ-CH-06). + +## ACL flow (end-to-end) + +A browser opening a TTY channel to a spoke through a hub (ADR-079): + +1. Browser's channel 0 → hub's channel 0: `channel/open` + `{ alpn: "alknet/tty", params: { backend: "docker", cmd: ["bash"], container: "abc123" } }`. + The browser's identity is a bearer token (ADR-034). +2. Hub's `CallAdapter` runs `AccessControl::check` on `channel/open` with + the browser's identity. If denied → `channel:forbidden`. +3. Hub forwards to spoke via `from_call`: the hub's `forwarded_for` handler + constructs a `call.requested` with the hub as caller and the browser as + `forwarded_for` (ADR-032 §3). The spoke receives `channel/open` with + `caller = hub`, `forwarded_for = browser`. +4. Spoke's `CallAdapter` runs `AccessControl::check` with the hub as caller + (the spoke authorizes the hub — ADR-050). The spoke's ownership store + verifies the hub (or the `forwarded_for` browser, per policy) owns + `container:abc123`. +5. Spoke allocates the channel via `TtyAdapter` / `DockerTtyBackend`, + returns `channel_id`. +6. Hub opens a matching channel on the browser's side and bridges them + (byte-forward with `channel_id` rewrite — ADR-079). + +The hub ran **zero** protocol-specific auth. It ran `channel/open`'s +`AccessControl::check` (call-protocol machinery) and forwarded. The channels +layer inherited the auth model by being a call-protocol operation. + +## Hub relay contract (ADR-079 — summary) + +The hub **translates**, not transparently forwards: + +1. **Call-protocol layer (channel 0): translate.** The hub terminates + channel 0 on both legs. `channel/open` from the browser → hub's + `AccessControl::check` → hub re-issues `channel/open` on the spoke leg + with `forwarded_for` → spoke returns its `channel_id` → hub maps + browser-id ↔ spoke-id. +2. **Data-channel layer: byte-forward with `channel_id` rewrite.** The + relay reads chunks for `browser_id`, rewrites the `channel_id` field to + `spoke_id`, writes onto the spoke's channels connection — and vice versa. + The relay does not parse the payload. + +`channel/control` operations on channel 0 carry `channel_id` in their JSON +payload; the hub's `CallAdapter` translates these too (rewrites +`channel_id` in the payload). The relay does not touch `channel/control` — +it's a call operation, translated, not byte-forwarded. + +The hub never runs a handler for `alknet/tty`, `alknet/ssh`, or +`alknet/tunnel`. It runs `alknet/channels` (the relay) and `alknet/call` +(for its own hub-level operations + translation). + +## Design Decisions + +All design decisions are documented as ADRs in [decisions/](../../decisions/). + +| ADR | Decision | Summary | +|-----|----------|---------| +| [073](../../decisions/073-channel-lifecycle-operations.md) | Channel Lifecycle Operations | The four ops; `direction` pinned; subscribe not poll | +| [072](../../decisions/072-channel-0-pre-negotiated-call.md) | Channel 0 Pre-Negotiated | Channel 0 = `alknet/call` | +| [079](../../decisions/079-hub-relay-translate-not-forward.md) | Hub Relay | Translate channel 0, byte-forward data channels | +| [049](../../decisions/049-streaming-handler-for-subscriptions.md) | StreamingHandler | The machinery `channel/resources/subscribe` uses | +| [032](../../decisions/032-forwarded-for-identity.md) | Forwarded-For Identity | The auth chain for hub-relayed opens | +| [050](../../decisions/050-dynamic-resource-ownership-for-runtime-spawned-resources.md) | Dynamic Resource Ownership | The ownership store the spoke queries | + +## References + +- ADR-073: channel lifecycle operations (the decision) +- ADR-079: hub relay (the translate contract) +- `docs/research/alknet-channels/phase-0-findings.md` §Channel Open + Negotiation, §ACL and Security Model \ No newline at end of file diff --git a/docs/architecture/crates/channels/channels-adapter.md b/docs/architecture/crates/channels/channels-adapter.md new file mode 100644 index 0000000..10f12c0 --- /dev/null +++ b/docs/architecture/crates/channels/channels-adapter.md @@ -0,0 +1,260 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# channels-adapter.md — ChannelsAdapter and ChannelManager + +The two internal components of the channels crate: the read/demux half +(`ChannelsAdapter`) and the reassemble/allocate half (`ChannelManager`). +ADR-075 is the decision; this doc specifies the contracts and the demux/mux +invariants. + +## The split + +| Component | Role | What it knows | +|-----------|------|---------------| +| `ChannelsAdapter` | `ProtocolHandler` on `alknet/channels`; reads 9-byte chunk headers off the transport and routes to `ChannelManager` | The transport stream; the `ChannelManager` handle. ALPN-blind. | +| `ChannelManager` | Shared state; holds `channel_id → ChannelState`, `HandlerRegistry`, `OperationRegistry`. Constructs `ChannelBidiStreamSource` per channel. What `channel/open` closes over. | The channel map; the handler registry for ALPN lookup. ALPN-blind (looks up ALPNs, doesn't parse their protocols). | + +The split mirrors the TTY crate's `ChunkReader`/`ChunkWriter` + adapter +pattern, generalized to N channels: the adapter drives N channels, and +channel 0 is special only in that it's pre-allocated. + +## `ChannelsAdapter::handle` + +```rust +#[async_trait] +impl ProtocolHandler for ChannelsAdapter { + fn alpn(&self) -> &'static [u8] { b"alknet/channels" } + + async fn handle(&self, connection: Connection, auth: &AuthContext) + -> Result<(), HandlerError> + { + // 1. One bidi stream carries all channels. + let (send, recv) = connection.accept_bi().await?; + + // 2. Channel 0 is pre-negotiated as alknet/call (ADR-072). + // Construct reassembly buffers, wrap as a Connection via + // from_source(ChannelBidiStreamSource), hand to the CallAdapter. + self.manager.preinstall_channel_0(send, recv, auth).await?; + + // 3. Run the demux loop. + self.manager.run_demux_loop(recv).await + } +} +``` + +### `preinstall_channel_0` + +The only special case: constructs the reassembly buffers for `channel_id = +0`, wraps them as a `Connection` (via `Connection::from_source` with a +`ChannelBidiStreamSource` — ADR-070/074), and hands that `Connection` to +the `CallAdapter` — exactly as if `alknet/call` had been the top-level +ALPN. The `CallAdapter` is looked up in the same `HandlerRegistry` as every +other ALPN. The `CallAdapter` is none the wiser: it calls `accept_bi()`, +gets one bidi stream (the channel-0 reassembled stream), and runs its +dispatch loop. `EventEnvelope` frames ride on `stream_type = 0` of channel 0. + +### `run_demux_loop` + +Reads 9-byte headers off the transport, looks up `channel_id` in the +`ChannelManager`'s `channels` map, and pushes the payload into the right +`ReassemblyBuffer` for `(channel_id, stream_type)`. If the buffer is full +(bounded-buffer backpressure, ADR-076), the loop stops reading that +channel's chunks until the consumer drains — other channels keep flowing. + +## `ChannelManager` + +```rust +pub struct ChannelManager { + channels: Mutex>, + handlers: Arc, + call_ops: Arc, + next_id: AtomicU32, // monotonic; wraps at u32::MAX + buffer_cap: usize, // default 1 MiB (ADR-076) + max_channels: usize, // default 256 (ADR-076) +} + +struct ChannelState { + alpn: String, + streams: HashMap, + handler_task: JoinHandle<()>, + stream_types: Vec, +} +``` + +`ChannelManager` is `Clone` (cheap — `Arc` internally) so the +`ChannelsAdapter`, the `channel/open` operation handler, and relay logic can +all hold a handle. + +> **Type-name convention:** `ChannelManager`, `ChannelsAdapter`, +> `ChannelBidiStreamSource`, `ChannelSubStreams`, and `ChannelClient` are +> the public API surface (contract). `ReassemblyBuffer`, `Demux`, +> `MuxHandle`/`MuxRunner`, `MpscSendStream`/`MpscRecvStream`, and +> `ChannelOperations` are illustrative internal type names — the channels +> crate's implementation may name them differently. The contracts are the +> invariants (REQ-CH-01..04, 06) and the public API; the internal names are +> not contractual. + +### `ChannelManager` is ALPN-blind and auth-blind + +The `ChannelManager` deliberately does **not** hold: + +- **No `ProtocolHandler` implementations.** It holds a `HandlerRegistry` + reference for ALPN lookup, but it doesn't *be* a handler. Handlers live in + their crates and register on the same registry. +- **No ALPN-specific parsing.** It does not parse `NegotiateRequest` JSON, + SSH frames, or tunnel target strings. It hands `params` JSON to the + handler and gets back a handler task; it hands `stream_type 3` JSON to the + handler's control handle. +- **No auth state.** Auth lives in the `OperationContext` that the call + protocol passes to `channel/open`. The `ChannelManager` doesn't check + scopes or ownership — that's `AccessControl::check` in + `OperationRegistry::invoke`, run before the `channel/open` handler. +- **No transport coupling.** It talks to the transport only through the + `ChannelsAdapter`'s read loop and the per-channel write pumps, both of + which use `AsyncRead + AsyncWrite`. + +This is what makes the channels layer WASM-compatible and transport-agnostic +— the `ChannelManager` is pure byte routing with no platform or protocol +dependencies. + +## The `channel/open` handler + +The `channel/open` (and `channel/close`, `channel/control`, +`channel/resources/subscribe`) operations are registered on the call +protocol's `OperationRegistry` at assembly time: + +```rust +let channel_ops = ChannelOperations::new(manager.clone()); +channel_ops.register_on(&mut call_registry)?; +``` + +The `channel/open` handler (ADR-073): +1. ACL is already checked by `OperationRegistry::invoke` before this handler + runs. +2. Looks up the ALPN in `HandlerRegistry` → `channel:unknown_alpn` if + missing. +3. Allocates the `channel_id` via `next_id.fetch_add(1, Relaxed)` (DP-1: + server-assigned). +4. Constructs the `ChannelBidiStreamSource` (ADR-074) for the negotiated + `stream_types`. +5. Spawns the handler task — `tokio::spawn(handler.handle(conn, &auth))`. + Identical to what `TtyAdapter::handle` does today, but on a + channels-backed `Connection`. +6. Records the `ChannelState`. +7. Returns the `channel_id`. + +## Demux invariants (REQ-CH-02, 04) + +### REQ-CH-02: transport close → all channel senders drop → all handlers see EOF + +On transport EOF, `run_demux_loop` clears the `channels` map, dropping all +`ReassemblyBuffer` senders. Every handler's reassembled `RecvStream` sees +EOF even without an explicit zero-length sentinel on the wire. Without this, +`read_to_end` / `tokio::io::copy` in handlers hangs forever waiting for a +sender that never drops. This is a teardown invariant of the +`ChannelsAdapter::handle` contract. + +### REQ-CH-04: lenient unknown-`channel_id` handling + +A chunk with an unallocated `channel_id` (or `stream_type`) is dropped with +a debug log and an error counter (exposed via `Demux::stats()`), and the +demux continues. This matches SSH's behavior and survives transient +mis-ordering during teardown. Validated by the POC +(`demux_unknown_channel_drops_lenient`). + +## Mux invariants (REQ-CH-03) + +### REQ-CH-03: dynamic registration (handle/runner split) + +The mux frames per-channel bytes back onto the transport. The POC surfaced +that `Mux::run(self, transport)` (consume, run pre-registered pumps) does +not compose with the dynamic `channel/open` model — channels are opened +after the run loop starts. + +The mux is split into: + +- **`MuxHandle`** — clone-able, `register(channel_id, stream_type) -> + Sender` callable at any time after the runner starts. +- **`MuxRunner`** — owns the transport, `select!`s on new-pump registrations + and per-channel write pumps. + +The runner's `select!` loop exits when all `MuxHandle` clones drop (the +`new_pumps` sender closes) — the natural shutdown signal. This matches the +dynamic `channel/open` model. + +## The two-pump pattern (ADR-078 — documented here for handler authors) + +Handlers with a two-pump shape (two `tokio::io::copy` pumps, one per +direction — tunnel, SSH `direct-tcpip`) MUST shut down the opposite sink +when one pump completes. `tokio::try_join!` alone deadlocks: each pump +waits for the other's EOF, which only comes after the opposite pump shuts +down its sink. + +```rust +let c2t = async { + tokio::io::copy(&mut recv, &mut tcp_write).await?; + tcp_write.shutdown().await.ok(); // shut down the peer's sink + Ok::<_, std::io::Error>(()) +}; +let t2c = async { + tokio::io::copy(&mut tcp_read, &mut send).await?; + send.shutdown().await.ok(); // shut down the peer's sink (emits sentinel — REQ-CH-01) + Ok::<_, std::io::Error>(()) +}; +tokio::try_join!(c2t, t2c)?; +``` + +The three-pump pattern (TTY's `pump_session`, coordinating via the +`exit_code` future) does not have this deadlock — the `exit_code` future is +the third signal. The two-pump pattern is documented in ADR-078; the +shutdown-on-completion contract is a handler-level concern, not a +channels-layer one. + +## The hub relay interface + +The hub relay (ADR-079) uses the `ChannelManager`'s interface to bridge two +channels connections: + +```rust +// For channel_id=7 on browser side, channel_id=12 on spoke side: +tokio::spawn(async move { + let (b_send, b_recv) = browser_mgr.open_channel_stream(7, stream_type).await; + let (s_send, s_recv) = spoke_mgr.open_channel_stream(12, stream_type).await; + tokio::join!( + pump(b_recv, s_send), // browser → spoke (with channel_id rewrite) + pump(s_recv, b_send), // spoke → browser (with channel_id rewrite) + ); +}); +``` + +The relay reads opaque bytes off one `ChannelManager`'s reassembled stream +and writes them onto the other's write-half, which re-chunks them with the +other leg's `channel_id`. The relay does not parse the bytes — it doesn't +know if they're TTY chunks, SSH frames, or tunnel data. The hub translates +`channel/open` on channel 0 (re-issues on the spoke leg with +`forwarded_for`); data channels are byte-forwarded with `channel_id` +rewrite. See ADR-079 for the full relay contract. + +## Design Decisions + +All design decisions are documented as ADRs in [decisions/](../../decisions/). + +| ADR | Decision | Summary | +|-----|----------|---------| +| [075](../../decisions/075-channelsadapter-and-channelmanager.md) | ChannelsAdapter and ChannelManager | The split; the contracts | +| [076](../../decisions/076-backpressure-channel-limits-id-reuse.md) | Backpressure, Limits, ID Reuse | Bounded-buffer, 256-channel cap, monotonic IDs | +| [078](../../decisions/078-two-pump-shutdown-on-completion.md) | Two-Pump Pattern | Shutdown-on-completion contract | +| [079](../../decisions/079-hub-relay-translate-not-forward.md) | Hub Relay | Translate channel 0, byte-forward data channels | + +## References + +- ADR-075: ChannelsAdapter and ChannelManager (the decision) +- ADR-072: channel 0 pre-negotiated (the `preinstall_channel_0` step) +- ADR-073: channel lifecycle operations (the ops registered on `call_ops`) +- ADR-074: ChannelBidiStreamSource (what the manager constructs per channel) +- ADR-076: backpressure and limits (`buffer_cap`, `max_channels`) +- `docs/research/alknet-channels/poc-summary.md` §Issues Surfaced #4-#7 + (REQ-CH-01..04, the two-pump deadlock) \ No newline at end of file diff --git a/docs/architecture/crates/channels/channels-connection.md b/docs/architecture/crates/channels/channels-connection.md new file mode 100644 index 0000000..0780188 --- /dev/null +++ b/docs/architecture/crates/channels/channels-connection.md @@ -0,0 +1,210 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# channels-connection.md — ChannelBidiStreamSource and Sub-Stream Access + +How a reassembled channel is presented to its handler as a `Connection`. +ADR-074 is the decision; this doc specifies the API shape and the two +access paths. + +## What + +Each channel is reassembled into a set of `AsyncRead + AsyncWrite` handles +—one per active `stream_type` (declared at `channel/open` time, ADR-073). +These handles are wrapped as a `ChannelBidiStreamSource` that implements +`alknet-core`'s `BidiStreamSource` trait (ADR-070), and a `Connection` is +constructed from it via `Connection::from_source(source, alpn)`. + +The handler receives a `Connection` and can either: +1. Call `accept_bi()` once to get the main data pair (`stream_type` 0/1) — + the generic handler path (tunnel, SSH). +2. Call `into_sub_streams()` on the `ChannelBidiStreamSource` to get all + active sub-streams as typed `(stream_type, SendStream, RecvStream)` + tuples — the typed handler path (TTY, which needs stdin/stdout/stderr/ + control). + +Both paths operate on the same reassembly buffers; the difference is how the +handler accesses them. + +## `ChannelBidiStreamSource` + +```rust +// In alknet-channels: + +pub struct ChannelBidiStreamSource { + // The reassembly buffers for this channel's active stream_types, + // plus the mux handle for writing back onto the transport. + // Constructed by ChannelManager::build_channel_connection (ADR-075). + ... +} + +#[async_trait] +impl BidiStreamSource for ChannelBidiStreamSource { + async fn accept_bi(&self) + -> Result<(SendStream, RecvStream), StreamError> + { + // Yields the (stream_type 0, stream_type 1) pair on first call, + // ConnectionClosed on subsequent calls. Yield-once per channel, + // matching the POC's validated shape. + } + + async fn open_bi(&self) + -> Result<(SendStream, RecvStream), StreamError> + { + // StreamClosed — a single channel cannot open new application + // streams (same as ADR-065's Stream backend). Additional sub-streams + // (stream_type 2, 3) are accessed via into_sub_streams(), not + // open_bi(). + } + + fn remote_addr(&self) -> Option { ... } + + fn close(&self, _code: u32, _reason: &str) { ... } +} +``` + +One `ChannelBidiStreamSource` instance represents **one channel** (not the +whole channels connection). The `ChannelManager` (ADR-075) constructs one +per channel at `channel/open` time and wraps it in a `Connection` via +`from_source`. + +## The generic path: `accept_bi()` + +For handlers that only need the main data pair (`stream_type` 0 = data-in, +`stream_type` 1 = data-out): + +```rust +// Tunnel handler — ~15 lines, zero channels-layer awareness +async fn handle(&self, connection: Connection, _auth: &AuthContext) + -> Result<(), HandlerError> +{ + let (mut send, mut recv) = connection.accept_bi().await?; + let mut tcp = TcpStream::connect(target).await?; + let (mut tcp_read, mut tcp_write) = tcp.into_split(); + + // Two-pump with shutdown-on-completion (ADR-078) + let c2t = async { + tokio::io::copy(&mut recv, &mut tcp_write).await?; + tcp_write.shutdown().await.ok(); + Ok::<_, std::io::Error>(()) + }; + let t2c = async { + tokio::io::copy(&mut tcp_read, &mut send).await?; + send.shutdown().await.ok(); // emits zero-length sentinel (REQ-CH-01) + Ok::<_, std::io::Error>(()) + }; + tokio::try_join!(c2t, t2c)?; + Ok(()) +} +``` + +The handler calls `accept_bi()` once, gets the `(SendStream, RecvStream)` +pair, and pumps. It does not know it's inside a channels connection — the +`Connection` looks like any other. This is the path the POC's `EchoHandler` +and `TunnelHandler` validated. + +`accept_bi()` is yield-once: the first call returns the 0/1 pair; subsequent +calls return `ConnectionClosed`. This matches the POC's validated shape and +the `StreamBidiStreamSource` yield-once contract (ADR-070). + +## The typed path: `into_sub_streams()` + +For handlers that need `stream_type` 2 (stderr) or 3 (control) in addition +to 0/1: + +```rust +// In alknet-channels: +pub struct ChannelSubStreams { + /// (stream_type, send_half, recv_half) for each active stream_type. + pub streams: Vec<(u8, SendStream, RecvStream)>, +} + +impl ChannelSubStreams { + pub fn get(&self, stream_type: u8) -> Option<(&SendStream, &RecvStream)> { ... } +} + +impl ChannelBidiStreamSource { + /// Returns all active sub-streams, keyed by stream_type. Consumes the + /// source — call this instead of accept_bi() if the handler needs + /// direct access to stream_types 2/3. + pub fn into_sub_streams(self) -> ChannelSubStreams { ... } +} +``` + +The handler crate destructures `ChannelSubStreams` into its typed names: + +```rust +// In alknet-tty (inside-channels mode, ADR-077): +let sub = channel_source.into_sub_streams(); +let stdin = sub.get(0).unwrap(); // SendStream +let stdout = sub.get(1).unwrap(); // RecvStream +let stderr = sub.get(2); // Option<&RecvStream> +let control = sub.get(3).unwrap(); // RecvStream (JSON control) +``` + +**The channels crate does not know about TTY's `stream_type` semantics.** +It exposes `(stream_type, SendStream, RecvStream)` tuples. The handler crate +maps stream_types to its typed names. This preserves ADR-003's +no-handler-depends-on-another-handler rule and keeps the channels crate +ALPN-blind. + +`into_sub_streams()` consumes the source — a handler can't call both +`accept_bi()` and `into_sub_streams()`. This is by design: the sub-streams +include the 0/1 pair, so `into_sub_streams()` is the superset. + +## Choosing the path + +| Handler shape | Path | Examples | +|---------------|------|---------| +| Main data pair only (0/1) | `accept_bi()` | tunnel, SSH (SSH multiplexes internally) | +| Needs stderr/control (2/3) | `into_sub_streams()` | TTY (stdin/stdout/stderr/control) | + +The handler chooses based on its ALPN's `stream_type` set (declared at +`channel/open` time). The `ChannelsAdapter` (ADR-075) passes the handler a +`Connection` (via `from_source`); handlers that need sub-streams access the +`ChannelBidiStreamSource` via a channels-crate extension trait or downcast +(exact ergonomics are an implementation detail for the channels crate; the +contract is that both paths are available and the handler crate chooses). + +## Recursive composition + +A `ChannelBidiStreamSource` is a `BidiStreamSource`, and `Connection:: +from_source` wraps it. A handler that is itself `alknet/channels` can open a +sub-channels connection on a data channel — `alknet/channels` inside +`alknet/channels`. This is allowed (the `Connection` abstraction permits it) +but not a feature designed for. The primary use case is one level of +multiplexing. Recursive composition is a natural consequence of the +abstraction, not a goal. + +## What does NOT change + +- **`ProtocolHandler` trait** (ADR-002) — handlers still receive a + `Connection` and call `accept_bi()`. The `ChannelBidiStreamSource` is + internal to the channels crate; handlers see a `Connection`. +- **`SendStream` / `RecvStream`** (ADR-007) — unchanged. They continue to + wrap their internal sources. `ChannelBidiStreamSource` constructs them via + the existing `from_stream` constructors, backed by mpsc reassembly + buffers. +- **`HandlerRegistry`** — unchanged. The channels layer looks up ALPNs in + the same registry as top-level connections. + +## Design Decisions + +All design decisions are documented as ADRs in [decisions/](../../decisions/). + +| ADR | Decision | Summary | +|-----|----------|---------| +| [074](../../decisions/074-channelconnection-bidistreamsource.md) | ChannelConnection | Per-channel `BidiStreamSource`; yield-once `accept_bi`; `into_sub_streams()` accessor | +| [070](../../decisions/070-bidistreamsource-trait.md) | BidiStreamSource Trait | The extension point `ChannelBidiStreamSource` implements | +| [065](../../decisions/065-connection-from-stream-generic-single-stream.md) | `Connection::from_stream` | The yield-once path generalized for channels | + +## References + +- ADR-074: ChannelConnection (the decision) +- ADR-070: BidiStreamSource trait +- ADR-065: `Connection::from_stream` +- ADR-077: TTY inside channels (the primary consumer of `into_sub_streams`) +- `docs/research/alknet-channels/poc-summary.md` §POC Target 2 (the + yield-once `Connection::from_stream` validation) \ No newline at end of file diff --git a/docs/architecture/crates/channels/channels-wire.md b/docs/architecture/crates/channels/channels-wire.md new file mode 100644 index 0000000..5a283ad --- /dev/null +++ b/docs/architecture/crates/channels/channels-wire.md @@ -0,0 +1,216 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# channels-wire.md — The 9-Byte Chunk Format + +The wire format for `alknet/channels`: a 9-byte chunk header that +multiplexes N logical channels, each with up to 256 sub-stream types, over +a single ordered, reliable bidirectional transport stream. ADR-071 is the +decision; this doc specifies the format and the wire-level invariants. + +## Chunk header + +``` +[channel_id: u32 be][stream_type: u8][length: u32 be][payload bytes] +``` + +9 bytes of header, followed by `length` bytes of payload. + +| field | offset | width | meaning | +|-------|--------|-------|---------| +| `channel_id` | 0 | 4 (BE) | The logical channel this chunk belongs to. Channel 0 is pre-negotiated as `alknet/call` (ADR-072). Channels 1..N are opened dynamically via `channel/open` (ADR-073). | +| `stream_type` | 4 | 1 | The sub-stream within the channel. See "Stream types" below. | +| `length` | 5 | 4 (BE) | The payload length in bytes. 0 = EOF sentinel. Max `MAX_CHUNK_LEN`. | + +This is a 4-byte extension of alknet-tty's 5-byte format (ADR-052): the +`channel_id` prefix is added; `stream_type` and `length` are identical. The +`ChunkReader` / `ChunkWriter` pattern, the framing-disambiguation trick, +and the zero-length sentinel convention all carry forward from TTY. + +## `MAX_CHUNK_LEN` + +`16 * 1024 * 1024` (16 MiB), matching TTY's cap (ADR-052 §5). A chunk with +`length > MAX_CHUNK_LEN` returns `ChunkTooLarge` and does not corrupt the +stream — the demux drops the chunk and continues. The header is always +exactly 9 bytes, so the demux can always resync by reading the next 9-byte +header. + +## Stream types + +| stream_type | direction | purpose | +|-------------|-----------|---------| +| 0 | write half | data flowing in (stdin equivalent) | +| 1 | read half | data flowing out (stdout equivalent) | +| 2 | read half (optional) | error/diagnostic output (stderr equivalent) | +| 3 | bidirectional | control messages (ALPN-specific JSON) | +| 4-255 | reserved | future sub-stream types | + +Not all channels use all sub-streams. The active set is declared at +`channel/open` time (ADR-073 `stream_types` field) and fixed for the +channel's lifetime. + +| Channel ALPN | Active stream_types | +|--------------|---------------------| +| `alknet/call` (channel 0) | 0 (EventEnvelope frames) | +| `alknet/tty` | 0, 1, 2, 3 (stdin, stdout, stderr, control) | +| `alknet/tunnel` | 0, 1 (data-in, data-out) | +| `alknet/ssh` | 0, 1 (data-in, data-out — SSH multiplexes internally) | + +## Channel 0 — pre-negotiated `alknet/call` + +Channel 0 is not a special "control plane" with its own framing. It is +`alknet/call` pre-negotiated (ADR-072): both sides know `channel_id = 0` is +routed to the `CallAdapter` without an explicit `channel/open` exchange. + +Channel 0 uses only `stream_type` 0 for `EventEnvelope` frames (JSON, +length-prefixed — the call protocol wire format, ADR-064). `stream_type` +1-255 on channel 0 are reserved for future call-protocol sub-streams. + +Channel 0's chunks have `channel_id = 0` in the header — same format as +every other channel. Disambiguation between channel 0 and data channels is +by `channel_id`, not by a special first-byte trick. + +## Framing disambiguation (from ADR-052 §5) + +The 9-byte header is always exactly 9 bytes. `length` is bounded by +`MAX_CHUNK_LEN`. The demux reads 9 bytes, parses the header, reads +`length` bytes of payload, and routes. If a chunk is dropped (e.g., +`ChunkTooLarge`), the demux resyncs by reading the next 9-byte header — +the format is self-synchronizing. + +Within a channel, `stream_type` 0 (stdin) from the server is invalid, so +`0x00` as the first byte of a chunk payload from the server is unambiguous +(carried from ADR-052 §5). + +## Zero-length sentinel = EOF + +A zero-length chunk (`length = 0`) is delivered as an empty `Bytes`, which +the reassembled stream interprets as EOF. This is the clean-shutdown signal +for a `(channel_id, stream_type)` pair — same convention as TTY (ADR-052 +§Sentinels). + +The sentinel is emitted by the write side's `AsyncWrite::shutdown` (see +REQ-CH-01 below) and consumed by the read side's `AsyncRead::poll_read` as +EOF. + +## Wire-level invariants (REQ-CH-01, 02, 04, 05) + +The de-risk POC (`docs/research/alknet-channels/poc-summary.md` §Issues +Surfaced) surfaced invariants that hang channels silently if underspecified. +These are **contracts**, not implementation details — both sides must agree. + +### REQ-CH-01: `AsyncWrite::shutdown` emits a zero-length sentinel + +The reassembled stream's write half (`MpscSendStream` or equivalent) MUST +send an empty `Bytes` (the EOF sentinel) before dropping the sender on +`AsyncWrite::shutdown`. Without this, the demux never sees EOF on the +channel's `stream_type`, and `tokio::io::copy` in the handler never +completes — the session hangs. + +The TTY crate's `pump_session` emits the zero-length stdout sentinel +explicitly via `Chunk::stdout(Bytes::new())`; the channels layer's +per-channel write pump does NOT forward a sentinel on sender-drop, so the +send adapter must. Both sides must agree on this convention, or channels +hang on clean shutdown. + +### REQ-CH-02: transport close → all channel senders drop → all handlers see EOF + +The demux loop MUST clear its `channels` map on transport EOF, dropping all +`ReassemblyBuffer` senders. Every handler's reassembled `RecvStream` sees +EOF even without an explicit zero-length sentinel arriving on the wire. + +Without this, `read_to_end` / `tokio::io::copy` in handlers hangs forever +waiting for a sender that never drops because the demux task is holding the +map. This is a teardown invariant of the `ChannelsAdapter::handle` contract. + +### REQ-CH-04: lenient unknown-`channel_id` handling with error counter + +A chunk with an unallocated `channel_id` (or `stream_type` on an allocated +channel) is dropped with a debug log and an error counter (exposed via +`Demux::stats()`), and the demux continues. This matches SSH's behavior and +survives transient mis-ordering during teardown (a chunk for a channel that +was just closed may arrive after the close is processed). + +The alternative (strict — close the transport on unknown `channel_id`) is +fragile during teardown and catches bugs at the cost of reliability. The +lenient approach with an error counter provides observability without +fragility. + +### REQ-CH-05: bounded-buffer backpressure does not deadlock + +Each `(channel_id, stream_type)` has an independent bounded `mpsc` buffer +(default 1 MiB — ADR-076). A slow reader on one channel does not block +another channel's reads — the demux's per-chunk route awaits the matching +sender without holding a global lock. + +The 1 MiB `tunnel_large_payload` POC test exercised this end-to-end: a +channel writer faster than the TCP echo server consumer, with no deadlock +and no cross-channel blocking. This invariant must hold for all transport +shapes — the bounded-buffer approach is the decision (ADR-076). + +## Sync core / async shell split + +The wire format's core is pure byte manipulation: + +```rust +// wire.rs — sync core, no async, no platform deps, WASM-clean + +const CHUNK_HEADER_LEN: usize = 9; +const MAX_CHUNK_LEN: u32 = 16 * 1024 * 1024; + +pub struct ChunkHeader { + pub channel_id: u32, + pub stream_type: u8, + pub length: u32, +} + +pub fn parse_header(buf: &[u8; 9]) -> Result { ... } +pub fn write_header(channel_id: u32, stream_type: u8, length: u32, out: &mut [u8; 9]) { ... } +``` + +The async shell (demux/mux — see [channels-adapter.md](channels-adapter.md)) +wraps this core with `read_exact` / `write_all` on the transport and `mpsc` +routing. The split keeps the WASM-compatible core separate from the +tokio-dependent shell. The POC validated the sync core compiles under +`wasm32-unknown-unknown`. + +## Channel lifecycle (summary) + +| Phase | Mechanism | Reference | +|-------|-----------|-----------| +| Open | `channel/open` call operation on channel 0; responder allocates `channel_id`, returns it | ADR-073 | +| Data | chunks with `channel_id` routed to reassembly buffers; handler sees `AsyncRead + AsyncWrite` | this doc, [channels-connection.md](channels-connection.md) | +| Control (data-ordered) | `stream_type 3` chunks on the data channel (JSON, in-order with data) | ADR-073 §DP-4 | +| Control (out-of-band) | `channel/control` call operation on channel 0 | ADR-073 | +| Close | `channel/close` call operation on channel 0; data chunks flushed before close | ADR-073, REQ-CH-06 | + +### REQ-CH-06: exit-chunk-before-close ordering (generalizes ADR-055) + +The channel's data chunks MUST be written and flushed before the +`channel/close` operation is sent on channel 0. This is a wire-level +invariant: the side closing must observe the data-channel pump complete +before issuing the call operation. + +For TTY this is the exit-chunk-is-last invariant (ADR-055) carried forward: +the exit control message on `stream_type 3` is the last data before +`channel/close`. For tunnels it is the last data byte before close. The +channels layer's close handler observes the pump completion; the call +operation is issued after. + +This invariant crosses two channels (the data channel and channel 0), so +the channels layer owns the ordering guarantee — it is not a handler +concern. + +## References + +- ADR-071: channels wire format (the decision) +- ADR-052: alknet-tty wire format (the 5-byte format this generalizes; + amended by ADR-077 — scoped to direct TTY) +- ADR-072: channel 0 pre-negotiated +- ADR-073: channel lifecycle operations +- ADR-076: backpressure, channel limits, ID reuse +- `docs/research/alknet-channels/poc-summary.md` §POC Target 1, §Issues + Surfaced #4-#6 (REQ-CH-01, 02, 04) +- `crates/alknet-tty/src/wire.rs` — the 5-byte format implementation \ No newline at end of file diff --git a/docs/architecture/crates/channels/overview.md b/docs/architecture/crates/channels/overview.md new file mode 100644 index 0000000..63f4559 --- /dev/null +++ b/docs/architecture/crates/channels/overview.md @@ -0,0 +1,247 @@ +--- +status: draft +last_updated: 2026-07-12 +--- + +# alknet-channels — Overview + +## What + +`alknet-channels` is a multiplexing proxy crate. It implements +`ProtocolHandler` for the `alknet/channels` ALPN: it receives one +bidirectional transport stream, reads 9-byte chunk headers, and routes each +chunk's payload to the right logical channel. Each channel is reassembled +into an `AsyncRead + AsyncWrite` pair and presented to its handler as a +`Connection` — the handler doesn't know it's inside a channels connection. + +Channel 0 is pre-negotiated as `alknet/call` (ADR-072). Every other channel +is opened dynamically via `channel/open` on channel 0 (ADR-073) and routed +through the same `HandlerRegistry` as top-level connections. The channels +layer does no protocol work itself — it is a re-framing proxy that converts +between "one transport stream carrying N channels" (the wire) and "N +independent stream handles" (what handlers see). + +## Why + +### The problem: three multiplexing models that don't compose + +Before channels, alknet had three multiplexing models: + +| Model | Where | Mechanism | +|-------|-------|-----------| +| Connection-level | ALPN router | One ALPN per QUIC connection | +| Stream-level | QUIC native | Many bidi streams per connection | +| Sub-stream-level | TTY chunk format | 4 logical channels within one bidi stream | + +A docker client needing both JSON call operations and raw TTY sessions +required **two separate QUIC connections** with different ALPNs. The call +protocol can't say "for this operation, open a TTY stream." The hub, +bridging browsers and spokes over multiple transports, faced an +O(protocols × transports × spokes) matrix of per-protocol framing parsers +and per-ALPN connection management. + +### The collapse: one multiplexing model, one connection per leg + +With `alknet/channels`, one connection carries everything: + +``` +Browser ──WebTransport──► Hub ──QUIC──► Spoke + alknet/channels alknet/channels + ┌─────────────┐ ┌─────────────┐ + │ ch0: call │ │ ch0: call │ + │ ch1: tty │ relay │ ch1: tty │ + │ ch2: ssh │ ◄─────► │ ch2: ssh │ + │ ch3: tunnel │ │ ch3: tunnel │ + └─────────────┘ └─────────────┘ +``` + +The hub's relay is channel-by-channel byte forwarding (with `channel_id` +rewrite — ADR-079), not per-protocol framing parsers. The hub's complexity +collapses from O(protocols × transports × spokes) to O(channels). + +The collapse is at three levels: + +1. **One connection per leg, not one per protocol.** All needs (call, TTY, + SSH, tunnel) ride as channels on one connection per leg. +2. **One multiplexing model, not three.** Connection-level, stream-level, + and sub-stream-level all become channels chunks. +3. **The call protocol orchestrates from inside.** Channel 0 is + `alknet/call` on both legs. The call protocol's `OperationRegistry`, + `AccessControl`, and `forwarded_for` machinery govern channel lifecycle + with no new auth. + +## Architecture + +The crate has two internal components (ADR-075): + +- **`ChannelsAdapter`** — implements `ProtocolHandler` for + `alknet/channels`. Its `handle()` receives one `Connection`, reads 9-byte + chunk headers, and routes chunks to the `ChannelManager`. The read/demux + half. +- **`ChannelManager`** — the shared state. Holds `channel_id → + ChannelState`, the `HandlerRegistry` reference, and the + `OperationRegistry` reference. The reassemble/allocate half. What the + `channel/open` operation handler closes over. + +Each channel is presented to its handler as a `Connection` constructed via +`Connection::from_source(ChannelBidiStreamSource, alpn)` (ADR-070/074). The +handler calls `accept_bi()` once (yield-once per channel) and drives its +session — identical to how it works on a top-level QUIC connection. + +See [channels-adapter.md](channels-adapter.md) for the full adapter/manager +design. + +## Crate dependencies + +``` +alknet-channels +├── alknet-core (ProtocolHandler, Connection, HandlerRegistry, +│ BidiStreamSource, SendStream, RecvStream, AuthContext) +├── alknet-call (OperationRegistry, HandlerKind, make_handler, +│ make_streaming_handler, CallError, ResponseEnvelope) +├── tokio (spawn, mpsc, io) +├── bytes (Bytes for chunk payloads) +├── async-trait +├── thiserror +└── tracing +``` + +`alknet-channels` depends on `alknet-call` because channel lifecycle +operations (`channel/open`, `channel/close`, `channel/control`, +`channel/resources/subscribe`) are registered on the call protocol's +`OperationRegistry` (ADR-073). This is the one handler-crate → +`alknet-call` dependency; it is sound because channels *is* a call-protocol +extension (channel lifecycle is call operations), not a peer handler. + +The dependency direction is: handlers depend on `alknet-core`; +`alknet-channels` depends on `alknet-core` and `alknet-call`; nothing +depends on `alknet-channels` except the assembly layer and +`alknet-tty`/`alknet-docker` behind their `channels`/`tty` feature gates. + +## ALPN + +`alknet/channels` — the ALPN the `ChannelsAdapter` registers on. One ALPN +per channels connection; the connection carries N logical channels, each +with its own ALPN (negotiated via `channel/open`). + +## Transport agnosticism + +The channels wire format works over any ordered, reliable bidirectional byte +stream: + +| Transport | How | +|-----------|-----| +| QUIC bidi stream | `alknet/channels` ALPN on a QUIC connection; one bidi stream carries all channels | +| TCP+TLS | `alknet/channels` ALPN on a TLS connection; the TCP stream carries all channels | +| WebTransport | `alknet/channels` session (deferred per ADR-044; the browser path uses WebSocket carrying `alknet/channels`) | +| SSH channel | channels connection riding inside an SSH `direct-tcpip` channel (channels-over-SSH) | +| Another channels connection | recursive composition (channel type `alknet/channels` inside `alknet/channels`) | + +The same wire format, the same chunk reassembly, the same `Connection` +abstraction. The transport is a parameter, not a design constraint. +`Connection::from_stream` / `from_source` (ADR-065/070) handles the +transport-agnostic `Connection` construction. + +## WASM compatibility + +The wire format's core is pure byte manipulation — `parse_header` / +`write_header` are pure functions with no platform dependencies. The de-risk +POC validated the sync core compiles under `wasm32-unknown-unknown`. The +async shell (demux/mux) wraps this core with `read_exact`/`write_all` and +`mpsc` routing. + +The `ChannelManager` is ALPN-blind, auth-blind, and transport-blind (ADR- +075) — pure byte routing with no platform or protocol dependencies. A WASM +build can read chunks from a WebTransport `BiStream`, reassemble them, and +present `AsyncRead + AsyncWrite` handles to WASM-compatible handlers. The +handlers themselves may or may not be WASM-compatible (russh's client is; +`portable_pty` is not), but the channels layer is WASM-compatible by +construction. + +The async shell and `alknet-core` dep graph are not fully WASM-clean yet +(transitive `getrandom`/`rand` deps) — this is an implementation concern, +not an architecture concern. The sync core's WASM compatibility is validated. + +## Relationship to existing crates + +### alknet-call + +Unchanged. The call protocol remains JSON-only, `EventEnvelope`-based. It +runs on channel 0 exactly as on a top-level `alknet/call` connection. The +`CallAdapter` receives a `Connection` backed by channel-0 chunk reassembly +and dispatches operations — it doesn't know it's inside channels. + +What changes: the call protocol gains a new class of operations — channel +lifecycle (ADR-073). These are registered on the `OperationRegistry` at +assembly time and dispatched through the existing `OperationContext` / +`AccessControl::check` path. + +### alknet-tty + +The TTY crate gains a `channels` feature (ADR-077) that enables +inside-channels mode. In direct mode (`alknet/tty` ALPN on a top-level +connection), the TTY adapter uses its own 5-byte wire format (ADR-052, +unchanged). In channels mode (`channel/open` with ALPN `alknet/tty`), the +adapter receives `ChannelSubStreams` (ADR-074) — four named +`SendStream`/`RecvStream` pairs for stream_types 0-3 — and pumps without +chunk parsing. The `TtyBackend` trait and `TtyHandle` are unchanged; +backends don't know which mode the adapter is in. + +### alknet-ssh (future) + +SSH as a channel type: an `alknet/ssh` channel carries the SSH binary +protocol over stream_types 0 and 1. The channels layer hands the +reassembled stream to `SshAdapter`, which feeds it to russh. SSH as a +channels transport: an SSH `direct-tcpip` channel could carry a channels +connection (channels-over-SSH). The SSH crate doesn't need to know about +channels — it implements `ProtocolHandler` for `alknet/ssh` and accepts a +`Connection`. + +### alknet-docker + +Docker lifecycle operations are call operations on channel 0 (unchanged +from ADR-058). Interactive exec/attach opens a TTY channel via +`channel/open` with ALPN `alknet/tty` and backend `docker`. No separate +`alknet/tty` connection needed — one `alknet/channels` connection handles +both JSON operations and raw TTY sessions. + +### alknet-hub + +The hub is the primary consumer. With channels, the hub holds one channels +connection per leg (browser↔hub, hub↔spoke) and relays channels between +them. The hub translates `channel/open` on channel 0 (re-issues on the +spoke leg with `forwarded_for` — ADR-079) and byte-forwards data channels +with `channel_id` rewrite. The hub's complexity collapses from +O(protocols × transports × spokes) to O(channels). + +## Design Decisions + +All design decisions are documented as ADRs in [decisions/](../../decisions/). + +| ADR | Decision | Summary | +|-----|----------|---------| +| [071](../../decisions/071-channels-wire-format.md) | channels Wire Format | 9-byte chunk header; one-way door | +| [072](../../decisions/072-channel-0-pre-negotiated-call.md) | Channel 0 Pre-Negotiated | Channel 0 = `alknet/call`, no special control plane | +| [073](../../decisions/073-channel-lifecycle-operations.md) | Channel Lifecycle Operations | `channel/open`/`close`/`control`/`resources/subscribe`; subscribe not poll; `direction` pinned | +| [074](../../decisions/074-channelconnection-bidistreamsource.md) | ChannelConnection | Per-channel `BidiStreamSource`; `into_sub_streams()` accessor | +| [075](../../decisions/075-channelsadapter-and-channelmanager.md) | ChannelsAdapter and ChannelManager | The read/demux + reassemble/allocate split; REQ-CH-01..04 | +| [076](../../decisions/076-backpressure-channel-limits-id-reuse.md) | Backpressure, Limits, ID Reuse | Bounded-buffer (1 MiB), 256-channel cap, monotonic IDs | +| [077](../../decisions/077-tty-inside-channels.md) | TTY Inside Channels | Two modes (direct vs channels); ADR-052 scoped to direct | +| [078](../../decisions/078-two-pump-shutdown-on-completion.md) | Two-Pump Pattern | Shutdown-on-completion contract; handler-level | +| [079](../../decisions/079-hub-relay-translate-not-forward.md) | Hub Relay | Translate channel 0, byte-forward data channels with ID rewrite | +| [080](../../decisions/080-channelclient.md) | ChannelClient | Client side; QUIC-only; `AlknetClient` deferred (OQ-55) | + +## Open Questions + +Open questions are tracked in [open-questions.md](../../open-questions.md). +Key questions affecting this crate: + +- **OQ-55** (deferred(scope)): `AlknetClient` core extraction — blocked on + a second *transport's* client, not a second client. `ChannelClient` is + decided (ADR-080). +- **OQ-56** (deferred(scope)): Full channel-level flow-control windowing — + bounded-buffer is decided (ADR-076); full windowing is an extension + blocked on a real HOL-blocking deployment observation. +- **OQ-57** (deferred(scope)): Two-pump helper extraction to alknet-core — + the *contract* is decided (ADR-078); the *helper* is blocked on a second + two-pump handler existing. \ No newline at end of file diff --git a/docs/architecture/decisions/071-channels-wire-format.md b/docs/architecture/decisions/071-channels-wire-format.md new file mode 100644 index 0000000..00bd9c5 --- /dev/null +++ b/docs/architecture/decisions/071-channels-wire-format.md @@ -0,0 +1,147 @@ +# ADR-071: alknet-channels Wire Format — 9-Byte Chunk Header + +## Status + +Accepted + +## Context + +`alknet-channels` is a multiplexing proxy: a `ProtocolHandler` on +`alknet/channels` that decomposes a single bidirectional transport stream into +N logical channels, each carrying a different ALPN. The wire format is the +substrate that makes one transport stream carry many channels. + +Two prior formats inform this design: + +1. **SSH's channel multiplexer (RFC 4254)** — `ChannelId(u32)` with + string-named types negotiated per channel, all traffic interleaved on one + encrypted transport stream. +2. **alknet-tty's chunk format (ADR-052)** — `[stream_type: u8][length: u32 be] + [payload]`, a fixed set of four sub-streams (stdin/stdout/stderr/control) + within one bidi stream. Validated by two POCs (alknet-docker-poc, + alknet-tty-poc) and in production code (`crates/alknet-tty/src/wire.rs`). + +The channels format is the generalization: add a `channel_id: u32` prefix to +TTY's 5-byte header, turning a fixed 4-channel multiplexer into an arbitrary +N-channel multiplexer. The de-risk POC (`docs/research/alknet-channels/poc- +summary.md`, 28 tests) validated this is a clean generalization: the sync core +(`parse_header`/`write_header`) is pure and WASM-compatible by construction; +the mpsc-bridged async shell scales to N concurrent channels with per-channel +order preservation and cross-channel isolation. + +## Decision + +### Chunk header + +``` +[channel_id: u32 be][stream_type: u8][length: u32 be][payload bytes] +``` + +9 bytes of header. The `channel_id` is the addition over TTY's 5-byte format; +`stream_type` and `length` are identical to TTY's fields (ADR-052), preserving +the framing-disambiguation soundness property (§5 below). + +| field | width | meaning | +|-------|-------|---------| +| `channel_id` | u32 BE | The logical channel this chunk belongs to. Channel 0 is pre-negotiated as `alknet/call` (ADR-072). Channels 1..N are opened dynamically via `channel/open`. | +| `stream_type` | u8 | The sub-stream within the channel. 0=stdin/data-in, 1=stdout/data-out, 2=stderr/diagnostic (optional), 3=control (JSON), 4-255 reserved. | +| `length` | u32 BE | The payload length in bytes. 0 = EOF sentinel (same convention as TTY — ADR-052 §Sentinels). | + +### `MAX_CHUNK_LEN` + +`16 * 1024 * 1024` (16 MiB), matching TTY's cap (ADR-052 §5). A chunk with +`length > MAX_CHUNK_LEN` returns `ChunkTooLarge` and does not corrupt the +stream — the demux drops the chunk and continues. This preserves the framing- +disambiguation soundness property: the header is always exactly 9 bytes, so +the demux can always resync after a dropped chunk by reading the next 9-byte +header. + +### Stream types (per channel) + +| stream_type | direction | purpose | +|-------------|-----------|---------| +| 0 | write half | data flowing in (stdin equivalent) | +| 1 | read half | data flowing out (stdout equivalent) | +| 2 | read half (optional) | error/diagnostic output (stderr equivalent) | +| 3 | bidirectional | control messages (ALPN-specific JSON) | +| 4-255 | reserved | future sub-stream types | + +Not all channels use all sub-streams. A TTY session uses 0-3. A raw tunnel +uses 0 and 1. An SSH connection uses 0 and 1 (SSH multiplexes internally). +The active `stream_type` set is declared at `channel/open` time +(ADR-073) and fixed for the channel's lifetime. + +### Framing disambiguation (carried from ADR-052 §5) + +Channel 0 is just another channel — its chunks have `channel_id=0` in the +header. Disambiguation between channel 0 (call protocol) and data channels is +by `channel_id`, not by a special first-byte trick. Within a channel, +`stream_type` 0 (stdin) from the server is invalid, so `0x00` as the first +byte of a chunk payload from the server is unambiguous. + +### Zero-length sentinel = EOF + +A zero-length chunk is delivered as an empty `Bytes`, which the reassembled +stream interprets as EOF (same convention as TTY — ADR-052 §Sentinels). This +is the clean-shutdown signal for a `(channel_id, stream_type)` pair. + +### Sync core / async shell split + +The wire format's core is pure byte manipulation — `parse_header(&[u8; 9]) -> +ChunkHeader` and `write_header(channel_id, stream_type, length, &mut [u8; 9])`. +No async, no platform dependencies. Compiles under `wasm32-unknown-unknown` +(validated by the POC). The async shell (demux/mux) wraps this core with +`read_exact`/`write_all` on the transport and `mpsc` routing. This split is +the same pattern as TTY's (REQ-TTY-01 generalization) and keeps the +WASM-compatible core separate from the tokio-dependent shell. + +## Consequences + +**Positive:** +- One multiplexing model replaces three (connection-level ALPN, stream-level + QUIC native, sub-stream-level TTY chunks). The hub's relay logic becomes + channel-by-channel byte forwarding, not per-protocol framing parsers. +- The 9-byte overhead is negligible for the intended use cases (TTY sessions, + SSH, tunnels, call operations). The format is a 4-byte extension to a + proven 5-byte format. +- WASM-compatible by construction — the pure core has no platform deps. +- The framing-disambiguation property from ADR-052 carries forward unchanged. + +**Negative:** +- All channels on one `alknet/channels` connection share one transport + stream's flow-control window. A slow consumer on one channel can + backpressure others. This is mitigated by bounded-buffer backpressure + (ADR-076) but not eliminated. For high-throughput bulk transfer, the + application uses N independent channels connections — same trade-off as + HTTP/2-over-TLS vs HTTP/3-over-QUIC. This is a transport property, not a + channels-format property. +- 9 bytes per chunk is 4 bytes more than TTY's 5-byte format. For + high-frequency small-chunk workloads (e.g., typing in a terminal), this is + a 80% header overhead increase. In practice the chunk size is driven by + the write pattern (a terminal sends a few bytes per keystroke regardless), + and the 4-byte delta is noise next to the TLS/QUIC overhead. + +## Door type + +**One-way.** The chunk header layout (`channel_id:u32 + stream_type:u8 + +length:u32`) is a wire-format commitment. Changing field widths, order, or +semantics after deployments exist requires a version migration. The +`stream_type` assignments (0=stdin, 1=stdout, etc.) are one-way for the same +reason — they are inherited from ADR-052 and preserved. + +The `MAX_CHUNK_LEN` value (16 MiB) is a two-way-door implementation detail +within the one-way format — it can be changed without a wire-format version +bump as long as both ends agree (it's a validation threshold, not a field +width). + +## References + +- ADR-052: alknet-tty wire format (the 5-byte format this generalizes) +- ADR-065: `Connection::from_stream` (the transport-agnostic Connection this + format rides on) +- ADR-070: `BidiStreamSource` trait (the extension point the channels + connection implements) +- `docs/research/alknet-channels/poc-summary.md` — the POC that validated the + format (28 tests, WASM compile check) +- `docs/research/alknet-channels/phase-0-findings.md` §The Wire Format +- `crates/alknet-tty/src/wire.rs` — the 5-byte format implementation \ No newline at end of file diff --git a/docs/architecture/decisions/072-channel-0-pre-negotiated-call.md b/docs/architecture/decisions/072-channel-0-pre-negotiated-call.md new file mode 100644 index 0000000..5d9aa10 --- /dev/null +++ b/docs/architecture/decisions/072-channel-0-pre-negotiated-call.md @@ -0,0 +1,125 @@ +# ADR-072: Channel 0 Is Pre-Negotiated `alknet/call` + +## Status + +Accepted + +## Context + +A channels connection carries N logical channels. One of them must carry the +call protocol — the JSON-RPC layer that orchestrates channel lifecycle +(`channel/open`, `channel/close`, `channel/control`, `channel/resources`). +The question is how channel 0 relates to the call protocol: is it a special +"control plane" with its own framing, or is it just `alknet/call` pre- +negotiated? + +The phase-0 research (`docs/research/alknet-channels/phase-0-findings.md` +§DP-2) recommends channel 0 is `alknet/call` pre-negotiated — no special +framing, no separate control-plane wire format. The call protocol runs on +channel 0 exactly as it runs on a top-level `alknet/call` QUIC connection. + +This matters because the alternative (a special control plane) would mean +the channels layer has its own JSON protocol for channel lifecycle, parallel +to and duplicating the call protocol's `OperationRegistry`, `AccessControl`, +`OperationContext`, and `forwarded_for` machinery. That duplication is the +"re-implement every protocol's framing per transport" problem the hub +motivation (§Hub Motivation) identifies as the thing channels exists to +collapse. + +## Decision + +**Channel 0 is `alknet/call`, pre-negotiated.** Both sides of a channels +connection know that `channel_id = 0` is routed to the `CallAdapter` without +an explicit `channel/open` exchange. The `CallAdapter` receives a +`Connection` backed by channel-0 chunk reassembly and dispatches operations +exactly as it does on a top-level `alknet/call` connection. + +### What this means concretely + +1. **Channel 0 uses the same 9-byte chunk format as every other channel** + (ADR-071). Its chunks have `channel_id = 0` in the header. No special + first-byte trick, no separate framing. + +2. **The `CallAdapter` is unchanged.** It receives a `Connection`, calls + `accept_bi()`, gets one bidi stream (the channel-0 reassembled stream), + and runs its dispatch loop. `EventEnvelope` frames ride on `stream_type = + 0` of channel 0. The `CallAdapter` does not know it is inside a channels + connection. + +3. **Channel lifecycle operations are call operations.** `channel/open`, + `channel/close`, `channel/control`, `channel/resources` are registered on + the call protocol's `OperationRegistry` at assembly time (ADR-073). They + are dispatched through the existing `OperationContext` (identity, scopes, + capabilities, ownership, `forwarded_for`), gated by the existing + `AccessControl::check`. No new auth machinery, no new framing, no + protocol version bump. + +4. **Channel 0 is allocated at `ChannelsAdapter::handle` entry.** The + `ChannelsAdapter` constructs channel 0's reassembly buffers, wraps them + as a `Connection` (via `Connection::from_source` with a + `ChannelBidiStreamSource` — ADR-070/074), and hands that `Connection` to + the `CallAdapter` — exactly as if `alknet/call` had been the top-level + ALPN. The `CallAdapter` is looked up in the same `HandlerRegistry` as + every other ALPN. + +### Channel 0's stream_type usage + +| stream_type | purpose | +|-------------|---------| +| 0 | `EventEnvelope` frames (JSON, length-prefixed — the call protocol wire format) | +| 1-255 | reserved for future call-protocol sub-streams | + +Channel 0 uses only `stream_type` 0. The reservation of `stream_type` +1-255 is a future-proofing detail, not a current commitment — the call +protocol is JSON-only and single-stream by design (ADR-064). + +## Consequences + +**Positive:** +- No control-plane duplication. The channels layer reuses the call protocol's + `OperationRegistry`, `AccessControl`, `OperationContext`, `forwarded_for`, + and `StreamingHandler` (ADR-049) machinery verbatim. Channel lifecycle is + just another class of call operations. +- The `CallAdapter` is transport-agnostic by construction — it works + identically whether the `Connection` is a top-level QUIC stream or a + channels-reassembled channel-0 stream. This is the "streams are streams" + insight made concrete. +- `channel/resources/subscribe` (ADR-073) is a `Subscription` operation on + channel 0, using the already-implemented `StreamingHandler` / + `invoke_streaming` path (ADR-049). The resource registry is a live view, + not a polled snapshot. +- Auth is inherited: `channel/open` goes through `AccessControl::check` + exactly like any other call operation. The channels layer does not re- + implement auth. + +**Negative:** +- Channel 0 is a single point of orchestration. If channel 0's `CallAdapter` + hangs, no new channels can be opened. This is the same property as the call + protocol today (one dispatch loop per connection) and is not a new + vulnerability. +- The call protocol's JSON-only nature means channel lifecycle operations + are JSON. For high-frequency control (e.g., per-keystroke resize), this is + more overhead than a binary control frame. The division (ADR-073 §DP-4) + handles this: `stream_type 3` on the data channel for data-ordered control, + call operations for lifecycle and infrequent control. + +## Door type + +**One-way.** Channel 0's role as `alknet/call` pre-negotiated is a wire- +format and protocol-structure commitment. Changing it after deployments +exist (e.g., to a special control plane) requires a version migration and +re-architecting the channel lifecycle operations. The reservation of +`stream_type` 1-255 on channel 0 is a two-way-door detail (they're currently +unused; assigning them is additive). + +## References + +- ADR-071: channels wire format (the 9-byte chunk header channel 0 uses) +- ADR-073: channel lifecycle operations (registered on channel 0's + `OperationRegistry`) +- ADR-064: irpc never integrated — hand-rolled EventEnvelope framing (the + call protocol channel 0 carries) +- ADR-049: StreamingHandler for subscriptions (the machinery + `channel/resources/subscribe` uses) +- ADR-070: BidiStreamSource trait (the `Connection` extension point) +- `docs/research/alknet-channels/phase-0-findings.md` §DP-2, §Channel 0 \ No newline at end of file diff --git a/docs/architecture/decisions/073-channel-lifecycle-operations.md b/docs/architecture/decisions/073-channel-lifecycle-operations.md new file mode 100644 index 0000000..6c96751 --- /dev/null +++ b/docs/architecture/decisions/073-channel-lifecycle-operations.md @@ -0,0 +1,297 @@ +# ADR-073: Channel Lifecycle Operations on the Call Protocol + +## Status + +Accepted + +## Context + +Channel lifecycle — open, close, control, resource discovery — must be +orchestrated somehow. The phase-0 research (`docs/research/alknet-channels/ +phase-0-findings.md` §Channel Open Negotiation, §DP-4) established that +channel lifecycle is orchestrated by the call protocol on channel 0 +(ADR-072). This ADR pins the exact operation shapes, the `direction` field +semantics, the control-message division, and the resource-discovery model. + +Three things from the research needed real decisions, not hedges: + +1. **Resource discovery: poll vs subscribe (OQ-CH-08).** The research + recommended "poll for v1, add subscription if staleness bites." This is a + hedge: the call protocol already has `StreamingHandler` / + `invoke_streaming` (ADR-049, implemented and tested), and the first + consumer (the hub aggregating worker resources) needs live updates. Polling + would be built, immediately found insufficient, and reworked. This ADR + commits to subscribe from day one. + +2. **The `direction` field and who writes first (OQ-CH-09).** The research + said "ALPN-specific and probably doesn't need a channels-layer rule… + needs to be pinned down." That IS the rule: the channels layer declares + write-order is ALPN-specific (determined by who is the ALPN-server), not + channels-enforced. This ADR pins which side is the ALPN-server for each + `direction` value. + +3. **Control messages: call ops vs stream_type 3 (DP-4).** The research + recommended "both, with clear division." This ADR pins the division. + +## Decision + +### Four operations on channel 0's `OperationRegistry` + +Registered at assembly time by the channels crate (via `ChannelOperations:: +register_on(&mut call_registry)`). All four go through the existing +`OperationContext` / `AccessControl::check` path — no new auth machinery. + +#### `channel/open` — open a data channel + +Request (`call.requested` on channel 0): + +```json +{ + "operation": "channel/open", + "input": { + "alpn": "alknet/tty", + "stream_types": [0, 1, 2, 3], + "params": { "backend": "docker", "cmd": ["bash"], "container": "abc123" }, + "direction": "initiator-to-responder" + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `alpn` | string | The ALPN the channel will carry. The responder looks this up in its `HandlerRegistry`. | +| `stream_types` | `[u8]` | Which sub-stream types this channel will use. Declared at open time so both sides size reassembly. E.g. `[0,1,2,3]` for TTY, `[0,1]` for a tunnel. | +| `params` | object | ALPN-specific parameters. For `alknet/tty` this is the `NegotiateRequest`. For `alknet/tunnel` this is the target resource. The channels layer does not interpret `params` — it hands the JSON to the handler. | +| `direction` | string | `initiator-to-responder` or `responder-to-initiator`. See "Direction semantics" below. | + +Response (`call.responded`): + +```json +{ + "output": { + "channel_id": 7, + "stream_types": [0, 1, 2, 3] + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `channel_id` | u32 | The server-assigned channel ID (DP-1: server-assigned). Both sides route chunks with this ID to the new channel. | +| `stream_types` | `[u8]` | The *negotiated* set — the responder may narrow the initiator's requested set (e.g., refuse stderr). The intersection of requested and supported. | + +**Channel ID allocation (DP-1): server-assigned.** The responder allocates +the `channel_id` via a monotonic `AtomicU32` (`next_id.fetch_add(1, Relaxed)`) +and returns it in the response. One round-trip before data flows — the same +round-trip the call protocol makes for every operation. All current channel +types (TTY, tunnel, SSH) already require a negotiation round-trip, so the +open round-trip is not additive latency. + +**Error codes** (new `CallError.code` strings, not new framing): + +| code | meaning | retryable | +|------|---------|-----------| +| `channel:unknown_alpn` | ALPN not in responder's `HandlerRegistry` | false | +| `channel:forbidden` | `AccessControl::check` denied the open | false | +| `channel:allocation_failed` | Handler allocate failed (e.g., backend couldn't start) | true (often transient) | +| `channel:invalid_params` | `params` JSON didn't satisfy the ALPN's expectations | false | +| `channel:too_many_channels` | Per-connection channel limit hit (ADR-076) | false | +| `channel:stream_type_unavailable` | Responder can't provide a requested `stream_type` | false | + +#### `channel/close` — tear down a channel + +```json +{ + "operation": "channel/close", + "input": { "channel_id": 7, "reason": "exit" } +} +``` + +The responder (the side that didn't send the close) drains its reassembled +streams for `channel_id`, signals EOF to the handler, and returns +`{ "closed": true }`. The `channel_id` is now eligible for reuse after the +drain completes (ADR-076 §channel-id-reuse). `reason` is free-form for +observability — not semantically required. + +**Exit-chunk-before-close ordering (generalizes ADR-055):** the channel's +data chunks must be written and flushed before the `channel/close` operation +is sent on channel 0. This is a wire-level invariant: the side closing must +observe the data-channel pump complete before issuing the call operation. +For TTY this is the exit-chunk-is-last invariant (ADR-055) carried forward; +for tunnels it is the last data byte before close. The channels layer's +close handler observes the pump completion; the call operation is issued +after. This is REQ-CH-06. + +#### `channel/control` — out-of-band control on channel 0 + +For control that doesn't need ordering relative to data (resize, signal, +keepalive): + +```json +{ + "operation": "channel/control", + "input": { + "channel_id": 7, + "stream_type": 3, + "message": { "type": "resize", "cols": 80, "rows": 24 } + } +} +``` + +The channels layer routes `message` to the handler's control handle for +`channel_id`. The `message` JSON is ALPN-specific; the channels layer does +not interpret it. + +#### `channel/resources/subscribe` — live resource discovery + +**This is a `Subscription` operation (ADR-049), not a polled Query.** The +research's "poll for v1, add subscription if staleness bites" is a hedge that +would cause rework — the `StreamingHandler` / `invoke_streaming` machinery +exists and is tested, and the hub consumer needs live updates when workers +connect/disconnect or containers start/stop. + +```json +{ + "operation": "channel/resources/subscribe", + "input": {} +} +``` + +The responder registers a `StreamingHandler` that emits a +`ResponseEnvelope` whenever the resource set changes. Each event: + +```json +{ + "output": { + "resources": [ + { + "alpn": "alknet/tty", + "backends": ["docker", "local"], + "access": { "required_scopes": ["tty:open"] } + }, + { + "alpn": "alknet/tunnel", + "targets": ["container:*", "service:postgres"], + "access": { "required_scopes_any": ["tunnel:open", "admin"] } + } + ] + } +} +``` + +| field | type | meaning | +|-------|------|---------| +| `alpn` | string | The ALPN this side accepts `channel/open` for. | +| `backends` / `targets` | `[string]` | ALPN-specific enumeration of what's available. The channels layer doesn't interpret these; they're for the initiator to know what `params` to send. | +| `access` | object | A preview of the `AccessControl` that `channel/open` will check. Advisory — lets the initiator fail fast. The real check happens on `channel/open`. | + +The stream emits an initial snapshot immediately, then subsequent events on +any change (worker connects/disconnects, container starts/stops, resource +exposed/withdrawn). The stream is long-lived; the subscriber cancels by +dropping the subscription (ADR-016 abort cascade applies). This is the +resource-discovery analogue of `services/list`, but live — matching the +bidirectional symmetry of the operation overlay. + +A `channel/resources` (non-subscribe, Query) operation is NOT provided. The +subscription's initial snapshot serves the poll use case (subscribe, read +the first event, cancel). Providing both would be redundant and would +pressure consumers toward the stale-poll path. + +### Direction semantics (OQ-CH-09 — pinned) + +Channel open is **bidirectional** — either side can initiate. The +`direction` field determines who is the ALPN-server (allocates the handler, +writes the negotiation response) vs the ALPN-client (writes the first +request). + +| `direction` | Initiator role | Responder role | Who writes first | +|-------------|----------------|----------------|-------------------| +| `initiator-to-responder` | ALPN-client | ALPN-server | Initiator writes first (the request data); responder's handler is the server side of the ALPN. The common case: "open me a TTY on your docker container." | +| `responder-to-initiator` | ALPN-server | ALPN-client | Responder writes first (the negotiation response / server greeting); initiator's handler is the client side. The "worker exposes, hub consumes" case: the worker initiates the open to make itself available; the hub is the client that connects to the exposed resource. | + +**The channels layer does not enforce write order.** Write order is +ALPN-specific, determined by which side is the ALPN-server (per the table +above). The channels layer's job is to route chunks; the handlers negotiate +who writes first via their ALPN's `params` contract. This is the rule the +research asked for: "the channels layer declares write-order is ALPN- +specific, not channels-enforced." + +**`channel_id` allocation is always by the responder** (DP-1), regardless of +`direction`. The responder is the side that receives the `channel/open` call +operation; it allocates the ID and returns it. In the `responder-to- +initiator` case, the initiator (worker) sends the `channel/open`, so the +responder (hub) allocates the ID — even though the worker is the ALPN- +server for the channel's data. This keeps ID allocation in one place (the +`channel/open` responder) and avoids the collision-prone client-assigned +alternative. + +### Control-message division (DP-4 — pinned) + +| Control path | When | Examples | +|--------------|------|----------| +| Call operations on channel 0 (`channel/control`, `channel/close`) | Control that doesn't need ordering relative to data, or lifecycle events | resize, signal, keepalive, close | +| `stream_type 3` chunks on the data channel | Control that MUST be ordered relative to data | EOF before exit, flush before close | + +The TTY crate's exit-chunk-is-last invariant (ADR-055) is the canonical +example of data-ordered control — it rides on `stream_type 3` because it +must arrive after the last stdin chunk, guaranteed by chunk ordering within +`(channel_id, stream_type)`, not by a call-protocol round-trip. The +`channel/close` operation that follows is on channel 0 and is ordered after +the data pump completes (REQ-CH-06). + +## Consequences + +**Positive:** +- Channel lifecycle reuses the call protocol's `OperationRegistry`, + `AccessControl`, `OperationContext`, `forwarded_for`, and + `StreamingHandler` verbatim. Zero new auth, zero new framing. +- `channel/resources/subscribe` gives the hub a live view of worker + resources — no polling, no staleness, no rework when the first consumer + needs subscriptions. +- The `direction` field makes bidirectional open explicit and pins who is + the ALPN-server, resolving the "who writes first" ambiguity without + channels-layer write-order enforcement. +- The control-message division (call ops vs stream_type 3) handles both + lifecycle control (infrequent, benefits from auth/observability) and + data-ordered control (frequent, needs ordering) without duplicating + machinery. + +**Negative:** +- Four new operation names in the `OperationRegistry`. The registry already + handles namespaced operations (`docker/container/list`, etc.); these are + in the `channel/` namespace. No registry changes needed. +- `channel/resources/subscribe` is a long-lived `Subscription` stream per + interested peer. This is the same cost as any other subscription (ADR-049); + the hub holds one per connected peer. Acceptable. +- The `direction` field adds one field to the `channel/open` input. It is + required (no default) — the initiator must state its intent. This is a + one-way-door wire-format field (removing it would break the bidirectional + open contract). + +## Door type + +**One-way.** The four operation names (`channel/open`, `channel/close`, +`channel/control`, `channel/resources/subscribe`), their input/output +schemas, and the `direction` field's semantics are wire-format commitments. +Changing them after deployments exist requires a protocol version migration. +The `reason` field on `channel/close` (free-form, observability-only) is a +two-way-door detail. + +The decision to use `Subscription` for resource discovery (not `Query`) is +one-way: consumers will depend on the live stream, and the Decision section +committed to Subscribe-only (a `Query` variant is NOT provided — the +subscription's initial snapshot serves the poll use case). The +`Handler` / `StreamingHandler` / `HandlerKind` API surface (ADR-049) is +the underlying one-way commitment. + +## References + +- ADR-071: channels wire format +- ADR-072: channel 0 is pre-negotiated `alknet/call` +- ADR-049: StreamingHandler for subscriptions (the machinery + `channel/resources/subscribe` uses — implemented and tested) +- ADR-016: abort cascade (subscription cancellation) +- ADR-032: forwarded-for identity (the auth chain for hub-relayed opens) +- ADR-055: exit-chunk-is-last (the TTY invariant generalized by REQ-CH-06) +- `docs/research/alknet-channels/phase-0-findings.md` §Channel Open + Negotiation, §DP-4, §OQ-CH-08, §OQ-CH-09 \ No newline at end of file diff --git a/docs/architecture/decisions/074-channelconnection-bidistreamsource.md b/docs/architecture/decisions/074-channelconnection-bidistreamsource.md new file mode 100644 index 0000000..27eeca4 --- /dev/null +++ b/docs/architecture/decisions/074-channelconnection-bidistreamsource.md @@ -0,0 +1,194 @@ +# ADR-074: ChannelConnection — BidiStreamSource over Chunk Reassembly + +## Status + +Accepted + +## Context + +ADR-070 landed the `BidiStreamSource` trait and `Connection::from_source` +extension point so downstream crates can implement their own connection +shapes without a core edit. The channels crate is the first downstream +consumer: a channels connection carries N logical channels, each a +bidirectional byte stream presented to a `ProtocolHandler` as a `Connection`. + +The phase-0 research (`docs/research/alknet-channels/phase-0-findings.md` +§The Channel Connection Abstraction, §OQ-CH-10) proposed that +`ChannelConnection` *implements* the `Connection` interface (for recursion +and generic handlers) **and** can be destructured into typed sub-stream +handles (`TtyChannel { stdin, stdout, stderr, control }`). The research +recommended "the TTY crate destructures; channels exposes `(channel_id, +stream_type) → (SendStream, RecvStream)` accessors" but did not pin the +exact API shape. This ADR pins it. + +The de-risk POC (`docs/research/alknet-channels/poc-summary.md` §POC Target +2) validated that `Connection::from_stream` (the yield-once path) is +sufficient — an echo `ProtocolHandler` runs through the full +demux→Connection→handler→mux path with zero channels-layer awareness. But +the POC deliberately used the yield-once path (one `Connection` per channel) +rather than the N-stream `ChannelBidiStreamSource` shape. This ADR commits +to the N-stream shape that ADR-070 unblocked. + +## Decision + +### `ChannelBidiStreamSource` implements `BidiStreamSource` + +The channels crate defines a `ChannelBidiStreamSource` that implements +`alknet-core`'s `BidiStreamSource` trait (ADR-070). One +`ChannelBidiStreamSource` instance represents **one channel** (not the +whole channels connection). Its `accept_bi()` yields one bidi stream — the +`(stream_type 0, stream_type 1)` pair for that channel — then returns +`ConnectionClosed` on subsequent calls (yield-once per channel, matching +the POC's validated shape). + +```rust +// In alknet-channels: +pub struct ChannelBidiStreamSource { + // The reassembly buffers for this channel's active stream_types, + // plus the mux handle for writing back onto the transport. + // Constructed by ChannelManager::build_channel_connection (ADR-075). + ... +} + +#[async_trait] +impl BidiStreamSource for ChannelBidiStreamSource { + async fn accept_bi(&self) -> Result<(SendStream, RecvStream), StreamError> { + // Yields the (stream_type 0, stream_type 1) pair on first call, + // ConnectionClosed on subsequent calls. This is the yield-once + // contract per channel, matching the POC's validated shape. + } + async fn open_bi(&self) -> Result<(SendStream, RecvStream), StreamError> { + // StreamClosed — a single channel cannot open new application + // streams (same as ADR-065's Stream backend). Additional sub-streams + // (stream_type 2, 3) are accessed via sub_streams(), not open_bi(). + } + fn remote_addr(&self) -> Option { ... } + fn close(&self, _code: u32, _reason: &str) { ... } +} +``` + +Each channel is presented to its handler as a `Connection` constructed via +`Connection::from_source(ChannelBidiStreamSource::new(...), alpn)`. The +handler calls `accept_bi()` once, gets the main data pair, and drives its +session — exactly as the POC's `EchoHandler` and `TtyAdapter` do today. + +### Sub-stream accessor for typed destructure (OQ-CH-10) + +Some handlers need access to `stream_type` 2 (stderr) and 3 (control) in +addition to the main 0/1 pair. The `Connection` interface alone (accept_bi) +only exposes the 0/1 pair. The channels crate provides a typed-accessor +extension: + +```rust +// In alknet-channels: +pub struct ChannelSubStreams { + pub streams: Vec<(u8, SendStream, RecvStream)>, +} + +impl ChannelBidiStreamSource { + /// Returns the typed sub-streams for this channel, keyed by stream_type. + /// Consumes the source — call this instead of accept_bi() if the handler + /// needs direct access to stream_types 2/3. For handlers that only need + /// the main 0/1 pair, accept_bi() is the path (and sub_streams() is not + /// called). + pub fn into_sub_streams(self) -> ChannelSubStreams { ... } +} +``` + +The handler crate (e.g., `alknet-tty`) destructures `ChannelSubStreams` into +its typed names: + +```rust +// In alknet-tty (inside-channels mode, ADR-077): +let sub = channel_source.into_sub_streams(); +let mut stdin = sub.get(0); // SendStream +let mut stdout = sub.get(1); // RecvStream +let mut stderr = sub.get(2); // RecvStream (optional) +let mut control = sub.get(3); // RecvStream (JSON control) +``` + +**The channels crate does not know about TTY's `stream_type` semantics.** It +exposes `(stream_type, SendStream, RecvStream)` tuples. The handler crate +maps stream_types to its typed names. This preserves ADR-003's +no-handler-depends-on-another-handler rule and keeps the channels crate +ALPN-blind. + +### When to use `accept_bi` vs `into_sub_streams` + +| Handler shape | Path | Example | +|---------------|------|---------| +| Main data pair only (0/1) | `accept_bi()` | tunnel handler, SSH handler (SSH multiplexes internally) | +| Needs stderr/control (2/3) | `into_sub_streams()` | TTY handler (stdin/stdout/stderr/control) | + +The handler chooses at construction time based on its ALPN's `stream_type` +set (declared at `channel/open` time, ADR-073). The `ChannelsAdapter` passes +the handler a `Connection` (via `from_source`); handlers that need sub- +streams downcast or receive the `ChannelBidiStreamSource` directly via a +channels-crate extension trait. The exact ergonomics (downcast vs. a +channels-crate constructor that hands the source directly to handlers that +opt in) are an implementation detail for the channels crate; the contract is +that both paths are available and the handler crate chooses. + +### Recursive composition + +A `ChannelBidiStreamSource` is a `BidiStreamSource`, and `Connection:: +from_source` wraps it. A handler that is itself `alknet/channels` can open a +sub-channels connection on a data channel. This is recursive composition: +`alknet/channels` inside `alknet/channels`. It is allowed (the `Connection` +abstraction permits it) but not a feature designed for — the primary use +case is one level of multiplexing. Recursive composition is a natural +consequence of the abstraction, not a goal. + +## Consequences + +**Positive:** +- `ChannelConnection` is a first-class peer of QUIC: one + `BidiStreamSource` impl per channel, constructed via `from_source` — no + core edit (the ADR-070 extension point). +- Handlers that only need the main data pair use `accept_bi()` — identical + to how they work on top-level QUIC connections. Zero handler changes for + the tunnel/SSH shape. +- Handlers that need typed sub-streams (TTY) use `into_sub_streams()` — the + channels crate provides the accessor, the handler crate maps to typed + names. No channels-crate knowledge of TTY semantics. +- The POC's validated yield-once shape is preserved per-channel; the N-stream + generalization is at the connection level (one channels connection = N + channels = N `ChannelBidiStreamSource` instances), not per-channel. + +**Negative:** +- Two paths to access channel data (`accept_bi` vs `into_sub_streams`). This + is a necessary divergence: the `Connection` interface alone can't express + "give me four named sub-streams" without four `accept_bi` calls (which + would violate the yield-once contract). The two-path design is the + minimum-complexity solution; the alternative (a new `Connection` variant + with multi-stream semantics) would touch `alknet-core` and break the + ADR-070 extension-point model. +- `into_sub_streams()` consumes the source, so a handler can't call both + `accept_bi()` and `into_sub_streams()`. This is by design — the sub- + streams include the 0/1 pair, so `into_sub_streams()` is the superset. + +## Door type + +**One-way.** The `ChannelBidiStreamSource` shape (one source per channel, +yield-once `accept_bi`, `into_sub_streams` accessor) is the handler-facing +API surface. Changing it after handlers exist (TTY, tunnel, SSH) is a +rewrite of those handlers' integration code. The trait impl is in the +channels crate (not core), so the one-way door is the channels crate's API, +not a core type. + +The choice of `into_sub_streams()` returning `Vec<(u8, SendStream, +RecvStream)>` (vs a typed struct, vs a map) is a two-way-door implementation +detail — the return type can change without breaking the contract as long +as the handler crate's destructure code updates. + +## References + +- ADR-070: BidiStreamSource trait (the extension point this implements) +- ADR-065: Connection::from_stream (the yield-once path this generalizes for + channels) +- ADR-071: channels wire format (the chunks this reassembles) +- ADR-075: ChannelsAdapter and ChannelManager (the components that construct + `ChannelBidiStreamSource` instances) +- ADR-077: TTY inside channels (the primary consumer of `into_sub_streams`) +- `docs/research/alknet-channels/poc-summary.md` §POC Target 2, §Issues + Surfaced #1 \ No newline at end of file diff --git a/docs/architecture/decisions/075-channelsadapter-and-channelmanager.md b/docs/architecture/decisions/075-channelsadapter-and-channelmanager.md new file mode 100644 index 0000000..dea953c --- /dev/null +++ b/docs/architecture/decisions/075-channelsadapter-and-channelmanager.md @@ -0,0 +1,225 @@ +# ADR-075: ChannelsAdapter and ChannelManager + +## Status + +Accepted + +## Context + +The channels crate has two internal components, split by responsibility +(`docs/research/alknet-channels/phase-0-findings.md` §Channel Manager and +Connection Internals): + +1. **`ChannelsAdapter`** — implements `ProtocolHandler` for + `alknet/channels`. Its `handle()` receives one `Connection` (the + transport), reads 9-byte chunk headers, and routes each chunk. It is the + read/demux half. + +2. **`ChannelManager`** — the shared state both halves touch. It holds the + map of `channel_id → ChannelState`, the `HandlerRegistry` reference, and + the `OperationRegistry` reference. It is the reassemble/allocate half. + It is what the `channel/open` operation handler closes over. + +The de-risk POC (`docs/research/alknet-channels/poc-summary.md` §Issues +Surfaced) surfaced three invariants the spec must pin: the mux needs dynamic +registration (handle/runner split — REQ-CH-03), the demux must drop all +channel senders on transport EOF (REQ-CH-02), and the `AsyncWrite::shutdown` +must emit a zero-length sentinel (REQ-CH-01). This ADR pins these as +contracts. + +## Decision + +### `ChannelsAdapter` — the read/demux half + +```rust +#[async_trait] +impl ProtocolHandler for ChannelsAdapter { + fn alpn(&self) -> &'static [u8] { b"alknet/channels" } + + async fn handle(&self, connection: Connection, auth: &AuthContext) + -> Result<(), HandlerError> + { + // 1. One bidi stream carries all channels. + let (send, recv) = connection.accept_bi().await?; + + // 2. Channel 0 is pre-negotiated as alknet/call (ADR-072). + // Construct its reassembly buffers, wrap as a Connection via + // from_source(ChannelBidiStreamSource), and hand to the + // CallAdapter (looked up in the registry, same as every ALPN). + self.manager.preinstall_channel_0(send, recv, auth).await?; + + // 3. Run the demux loop: read 9-byte headers, route payloads to + // per-(channel_id, stream_type) reassembly buffers. + self.manager.run_demux_loop(recv).await + } +} +``` + +The `preinstall_channel_0` step is the only special case: it constructs the +reassembly buffers for `channel_id = 0`, wraps them as a `Connection` (via +`Connection::from_source` with a `ChannelBidiStreamSource` — ADR-074), and +hands that `Connection` to the `CallAdapter` — exactly as if `alknet/call` +had been the top-level ALPN. The `CallAdapter` is none the wiser. + +### `ChannelManager` — the shared state + +```rust +pub struct ChannelManager { + /// channel_id → per-channel state. Channel 0 is pre-inserted at + /// construction by preinstall_channel_0. + channels: Mutex>, + /// The handler registry for looking up ALPNs on channel/open. + handlers: Arc, + /// The call protocol's operation registry, so channel/open etc. can be + /// registered at assembly time. + call_ops: Arc, + /// Next server-assigned channel_id. Monotonic; wraps at u32::MAX. + next_id: AtomicU32, + /// Per-channel reassembly buffer cap (ADR-076). Default 1 MiB. + buffer_cap: usize, + /// Per-connection channel limit (ADR-076). Default 256. + max_channels: usize, +} + +struct ChannelState { + /// The ALPN this channel carries, for routing and observability. + alpn: String, + /// Reassembly buffers per active stream_type. + streams: HashMap, + /// The handler task driving this channel. Dropping this aborts it. + handler_task: JoinHandle<()>, + /// Which stream_types are active (from the open negotiation). + stream_types: Vec, +} +``` + +`ChannelManager` is `Clone` (cheap — `Arc` internally) so the +`ChannelsAdapter`, the `channel/open` operation handler, and relay logic +can all hold a handle. + +### The demux loop — REQ-CH-02 and REQ-CH-04 + +`run_demux_loop` reads 9-byte headers, looks up `channel_id` in `channels`, +and pushes the payload into the right `ReassemblyBuffer` for `(channel_id, +stream_type)`. + +**REQ-CH-04 (lenient unknown-channel_id):** a chunk with an unallocated +`channel_id` (or `stream_type`) is dropped with a debug log and an error +counter (exposed via `Demux::stats()`), and the demux continues. This +matches SSH's behavior and survives transient mis-ordering during teardown. +Validated by the POC (`demux_unknown_channel_drops_lenient`). + +**REQ-CH-02 (transport close → all handlers see EOF):** on transport EOF, +the demux loop clears its `channels` map, dropping all `ReassemblyBuffer` +senders. Every handler's reassembled `RecvStream` sees EOF even without an +explicit zero-length sentinel on the wire. Without this, `read_to_end` / +`tokio::io::copy` in handlers hangs forever waiting for a sender that never +drops. This is a teardown invariant of the `ChannelsAdapter::handle` +contract. Validated by the POC. + +### The mux — REQ-CH-03 (handle/runner split) + +The mux frames per-channel bytes back onto the transport. The POC surfaced +that the plan's `Mux::run(self, transport)` shape (consume the mux, run +pumps for pre-registered channels) does not compose with the dynamic +`channel/open` model — channels are opened after the run loop starts. + +**REQ-CH-03 (dynamic registration):** the mux is split into: + +- **`MuxHandle`** — clone-able, `register(channel_id, stream_type) -> + Sender` callable at any time (after the runner has started). +- **`MuxRunner`** — owns the transport, `select!`s on new-pump registrations + and per-channel write pumps. + +The runner's `select!` loop exits when all `MuxHandle` clones drop (the +`new_pumps` sender closes), which is the natural shutdown signal. This +matches the dynamic `channel/open` model. The split adds one +`mpsc::UnboundedSender` + `Arc>` per mux — cheap. Validated +by the POC. + +### `ChannelManager` is ALPN-blind and auth-blind + +The `ChannelManager` deliberately does **not** hold: + +- **No `ProtocolHandler` implementations.** It holds a `HandlerRegistry` + reference for ALPN lookup, but it doesn't *be* a handler. Handlers live in + their crates and register on the same registry. +- **No ALPN-specific parsing.** It does not parse `NegotiateRequest` JSON, + SSH frames, or tunnel target strings. It hands `params` JSON to the + handler and gets back a handler task; it hands `stream_type 3` JSON to the + handler's control handle. +- **No auth state.** Auth lives in the `OperationContext` that the call + protocol passes to `channel/open`. The `ChannelManager` doesn't check + scopes or ownership — that's `AccessControl::check` in + `OperationRegistry::invoke`, run before the `channel/open` handler. +- **No transport coupling.** It talks to the transport only through the + `ChannelsAdapter`'s read loop and the per-channel write pumps, both of + which use `AsyncRead + AsyncWrite`. + +This is what makes the channels layer WASM-compatible and transport-agnostic +— the `ChannelManager` is pure byte routing with no platform or protocol +dependencies. + +### The `channel/open` handler — threading into `OperationRegistry` + +The `channel/open` (and `channel/close`, `channel/control`, +`channel/resources/subscribe`) operations are registered on the call +protocol's `OperationRegistry` at assembly time. The handler closures close +over a `ChannelManager` clone: + +```rust +let channel_ops = ChannelOperations::new(manager.clone()); +channel_ops.register_on(&mut call_registry)?; +``` + +The `channel/open` handler (ADR-073) looks up the ALPN in `HandlerRegistry`, +allocates the `channel_id` via `next_id.fetch_add(1, Relaxed)`, constructs +the `ChannelBidiStreamSource` (ADR-074), spawns the handler task, and +records the `ChannelState`. The key insight: spawning the handler task is +identical to what `TtyAdapter::handle` does today — `tokio::spawn` a +session-driving task. The only difference is the `Connection` passed in is +backed by chunk reassembly rather than a quinn connection. + +## Consequences + +**Positive:** +- The ChannelsAdapter/ChannelManager split mirrors the TTY crate's + ChunkReader/ChunkWriter + adapter pattern, generalized to N channels. +- The demux/mux contracts (REQ-CH-01..04) are pinned as wire-level + invariants, not implementation details. Both sides must agree, or channels + hang on clean shutdown. +- The `ChannelManager` is ALPN-blind, auth-blind, and transport-blind — the + channels layer is a re-framing proxy, not a protocol engine. This is what + makes it reusable across TTY, SSH, tunnel, and future ALPNs. + +**Negative:** +- The mux handle/runner split (REQ-CH-03) adds one `mpsc::UnboundedSender` + + `Arc>` per mux. Cheap, but more moving parts than the + pre-register-all-then-run alternative. The alternative doesn't match the + dynamic `channel/open` model, so the split is necessary, not optional. +- The demux loop is one task per transport. If the demux task panics, all + channels on that transport lose their read side. The teardown invariant + (REQ-CH-02) ensures handlers see EOF, not a hang — but a panic in the + demux is still a transport-wide failure. This is the same property as any + single-task read loop (including the call protocol's dispatch loop). + +## Door type + +**One-way (contracts) + two-way (internals).** The wire-level invariants +(REQ-CH-01..04) are one-way — both sides must agree, and changing them +after deployments exist is a protocol migration. The `ChannelManager`'s +internal structure (fields, `Arc>` vs a concurrent map, etc.) +is two-way — implementation details that can change without breaking the +contract. + +## References + +- ADR-071: channels wire format (the chunks the demux reads) +- ADR-072: channel 0 pre-negotiated (the `preinstall_channel_0` step) +- ADR-073: channel lifecycle operations (the ops registered on `call_ops`) +- ADR-074: ChannelBidiStreamSource (the per-channel source the manager + constructs) +- ADR-076: backpressure, channel limits, ID reuse (the `buffer_cap` / + `max_channels` / reuse invariants) +- `docs/research/alknet-channels/poc-summary.md` §Issues Surfaced #4-#6 + (REQ-CH-01, 02, 03) \ No newline at end of file diff --git a/docs/architecture/decisions/076-backpressure-channel-limits-id-reuse.md b/docs/architecture/decisions/076-backpressure-channel-limits-id-reuse.md new file mode 100644 index 0000000..96d51a2 --- /dev/null +++ b/docs/architecture/decisions/076-backpressure-channel-limits-id-reuse.md @@ -0,0 +1,143 @@ +# ADR-076: Backpressure, Channel Limits, and ID Reuse + +## Status + +Accepted + +## Context + +The phase-0 research (`docs/research/alknet-channels/phase-0-findings.md` +§DP-5, §OQ-CH-03/04/05/06) raised four operational questions about the +channels layer: + +1. **Flow control (DP-5, OQ-CH-03):** if one data channel's consumer is + slow, could it block all other channels on the same transport + (head-of-line blocking)? The research recommended "bounded-buffer + backpressure (option c)… if head-of-line blocking becomes a real problem, + full windowing can be added." The "if it becomes a problem" is a hedge — + the POC validated bounded-buffer with a 1 MiB test and no deadlock. The + decision is bounded-buffer. +2. **Channel ID reuse (OQ-CH-04):** after a channel is closed, can its ID be + reused? +3. **Maximum channels per connection (OQ-CH-05):** is there a limit? +4. **Channel open DoS (OQ-CH-06):** an authenticated peer could open many + channels and never read from them, exhausting memory. + +The de-risk POC (`docs/research/alknet-channels/poc-summary.md` §POC Target +1, §POC Target 3) validated the bounded-buffer backpressure path: the 1 MiB +`tunnel_large_payload` test exercises a channel writer faster than the TCP +echo server consumer, with no deadlock and no cross-channel blocking. + +## Decision + +### Backpressure: bounded-buffer, 1 MiB default (DP-5) + +Each `(channel_id, stream_type)` pair has an independent bounded `mpsc` +buffer. When a channel's buffer is full, the demux stops reading chunks for +that `channel_id` until the consumer drains it. Other channels keep flowing +— the demux's per-chunk route awaits the matching sender without holding a +global lock. + +**Default buffer cap: 1 MiB per `(channel_id, stream_type)`.** Configurable +per `ChannelManager` (`buffer_cap` field). This prevents memory exhaustion +without the complexity of SSH's sliding-window protocol. + +Full channel-level windowing (SSH-style sliding-window per channel) is a +deferred extension, tracked as [OQ-56](../questions/056-full-channel-level-flow-control-windowing.md) +(deferred(scope)). It is blocked on a real deployment observing head-of- +line blocking where the bounded-buffer mitigation is insufficient. The +bounded-buffer decision is made; the extension is not. + +### Channel ID reuse: yes, after drain (OQ-CH-04) + +After a channel is closed (`channel/close` acknowledged), its `channel_id` +is eligible for reuse. The reassembly buffers must be fully drained before +reuse to prevent data from the old channel leaking into the new one. + +**Drain-before-reuse invariant:** the `ChannelManager` marks a closed +channel's ID as "draining" (not in the `channels` map, but not yet returned +to the free pool). The ID returns to the free pool only after: +1. The `channel/close` response is sent (the close is acknowledged). +2. All reassembly buffers for that `channel_id` are empty (the handler has + consumed all data). + +The `next_id: AtomicU32` is monotonic (not a free-list) — IDs are not +immediately reused; the monotonic counter wraps at `u32::MAX`. This is +simpler than a free-list and avoids the drain-tracking complexity. With a +default `max_channels` of 256, the `u32` space is effectively unlimited +(~16.7 million channels before wrap). Reuse happens naturally on wrap, by +which time old channels are long drained. **The "reuse" in OQ-CH-04 is +satisfied by the wrap-around, not by a free-list.** + +### Maximum channels per connection: 256 default (OQ-CH-05/06) + +The `channel_id` is `u32` — the wire format supports ~4 billion channels. +The practical limit is memory (reassembly buffers per channel) and the +transport's flow control. + +**Default per-connection channel limit: 256** (`max_channels` field on +`ChannelManager`, configurable). This is the DoS defense (OQ-CH-06): an +authenticated peer that opens many channels and never reads from them is +bounded by `max_channels × buffer_cap` = 256 × 1 MiB = 256 MiB worst case. +Bounded buffers (DP-5) limit the damage per channel; the connection cap +limits the number of channels. Defense in depth. + +Exceeding the limit returns `channel:too_many_channels` (ADR-073 error +codes). The limit is per-connection, not per-peer — a peer can open more +channels on a second connection. + +### DoS defense summary (OQ-CH-06) + +| Layer | Mechanism | Default | +|-------|-----------|---------| +| Per-channel | Bounded reassembly buffer (stop reading when full) | 1 MiB per `(channel_id, stream_type)` | +| Per-connection | Channel count cap | 256 channels | +| Per-peer | Auth (`AccessControl::check` on `channel/open`) | Assembly-layer policy | + +An authenticated peer that opens 256 channels and never reads from them +consumes at most 256 MiB of reassembly buffers — bounded, not unbounded. +The assembly layer's `AccessControl` policy can further restrict +`channel/open` (e.g., `required_scopes: ["channel:open:alknet/tty"]`) to +limit who can open channels at all. + +## Consequences + +**Positive:** +- Bounded-buffer backpressure is validated by the POC (1 MiB test, no + deadlock, no cross-channel blocking). The decision is made, not hedged. +- The 256-channel default cap with 1 MiB buffers gives a bounded 256 MiB + worst-case memory per connection — a clear DoS ceiling, not an open-ended + one. +- Monotonic `next_id` with wrap-around avoids free-list drain-tracking + complexity while still satisfying ID reuse (on wrap, after ~16.7M + channels). + +**Negative:** +- The 256-channel default may be too low for a hub with many concurrent + browser sessions each opening multiple channels. The cap is configurable + per `ChannelManager`; the hub assembly layer may set it higher for + deployments with many concurrent sessions. This is a deployment-time + decision, not an architecture decision. +- Bounded-buffer backpressure does not eliminate head-of-line blocking — it + bounds the memory cost. A slow consumer still stalls its own channel's + demux reads. For the intended use cases (TTY, SSH, tunnels) this is + acceptable; full windowing is tracked as OQ-56 (deferred(scope)). + +## Door type + +**Two-way.** The buffer cap (1 MiB), the channel limit (256), and the +monotonic-ID-with-wrap strategy are all configurable / changeable without a +wire-format change. The bounded-buffer *approach* (vs full windowing) is +one-way in the sense that the demux/mux code is written around it — but +full windowing is an additive extension (per-channel window tracking) that +doesn't change the wire format, so even that reversal is feasible. + +## References + +- ADR-071: channels wire format (the chunks the buffers hold) +- ADR-073: channel lifecycle operations (`channel:too_many_channels` error) +- ADR-075: ChannelManager (`buffer_cap`, `max_channels`, `next_id` fields) +- `docs/research/alknet-channels/poc-summary.md` §POC Target 1 (backpressure + validation), §POC Target 3 (1 MiB tunnel test) +- `docs/research/alknet-channels/phase-0-findings.md` §DP-5, §OQ-CH-03/04/ + 05/06 \ No newline at end of file diff --git a/docs/architecture/decisions/077-tty-inside-channels.md b/docs/architecture/decisions/077-tty-inside-channels.md new file mode 100644 index 0000000..f04cd90 --- /dev/null +++ b/docs/architecture/decisions/077-tty-inside-channels.md @@ -0,0 +1,162 @@ +# ADR-077: TTY Inside Channels — Sub-Streams, Not Wire Format + +## Status + +Accepted + +## Context + +ADR-052 defines the alknet-tty wire format: `[stream_type: u8][length: u32 +be][payload]`, a 5-byte chunk header for four sub-streams (stdin/stdout/ +stderr/control) within one bidi stream. This format is stable, implemented +(`crates/alknet-tty/src/wire.rs`), and used for direct `alknet/tty` +connections. + +The phase-0 research (`docs/research/alknet-channels/phase-0-findings.md` +§DP-3, §OQ-CH-02) recommended that the TTY chunk format be "absorbed into +channels" and that `alknet/tty` remain as a "direct-connect shortcut." But +the research did not pin what changes in the TTY crate when a TTY session +runs *inside* a channels connection. This is gap #5 from the architecture +assessment — a real integration question the research hand-waved. + +The problem: the `TtyAdapter`'s current `handle()` loops `accept_bi()`, +spawning a `drive_session` task per bidi stream that parses 5-byte TTY +chunks off the stream. Inside a channels connection, the stream is *already +de-chunked* by the channels layer's 9-byte format — the handler sees an +`AsyncRead + AsyncWrite` pair, not a chunk-encoded stream. If the TTY +adapter tries to parse 5-byte chunks off an already-de-chunked stream, it +breaks. + +## Decision + +### Two modes for TTY, one adapter + +The `TtyAdapter` operates in two modes, determined by how it receives its +`Connection`: + +| Mode | When | Wire format | How the adapter gets sub-streams | +|------|------|-------------|---------------------------------| +| **Direct (`alknet/tty` ALPN)** | Top-level QUIC/TCP connection with ALPN `alknet/tty` | TTY's 5-byte format (ADR-052) | `accept_bi()` → parse 5-byte chunks → split into stream_types 0-3 | +| **Inside channels** | `channel/open` with ALPN `alknet/tty` on a channels connection | Channels' 9-byte format (ADR-071) — the channels layer de-chunks | `into_sub_streams()` (ADR-074) → four named `SendStream`/`RecvStream` pairs for stream_types 0-3 | + +In both modes, the `TtyBackend` trait and `TtyHandle` are unchanged +(ADR-053). The backend allocates a PTY and returns a `TtyHandle`; the +adapter pumps data between the handle and the sub-streams. The difference is +only in how the adapter gets the sub-streams — 5-byte chunk parsing (direct) +vs. `into_sub_streams()` (channels). + +### What changes in alknet-tty + +1. **The adapter's session-driving code splits into two entry points:** + - `drive_session_direct(send, recv, backends, ...)` — the existing path: + parse 5-byte chunks, split into stream_types, pump. Used for direct + `alknet/tty` connections. + - `drive_session_channels(sub_streams, backends, ...)` — the new path: + receive `ChannelSubStreams` (four named `SendStream`/`RecvStream` + pairs), pump directly without chunk parsing. Used when the channel's + `Connection` is backed by `ChannelBidiStreamSource`. + +2. **The `TtyAdapter::handle()` branches on the `Connection`'s source type.** + The channels crate's `ChannelBidiStreamSource` is a `BidiStreamSource` + (ADR-070); the `Connection` wraps it. The adapter detects whether the + `Connection` is channels-backed (via a downcast or a channels-crate + extension trait — exact ergonomics per ADR-074's implementation detail) + and calls `drive_session_channels` instead of `drive_session_direct`. + + **This is the one place alknet-tty knows about channels.** It is a + branch on the connection source, not a dependency on channels' wire + format. The branch can be feature-gated (`channels` feature on + alknet-tty) so the direct-only path has no channels dependency. + +3. **The 5-byte wire format (ADR-052) is unchanged for direct connections.** + ADR-052's scope is now "the wire format for direct `alknet/tty` + connections." The channels path does not use it. This amends ADR-052's + scope — the format is not replaced, it's scoped. + +4. **The control channel (stream_type 3) works the same in both modes.** In + direct mode, control JSON rides in 5-byte chunks with `stream_type=3`. In + channels mode, control JSON rides in 9-byte chunks with `stream_type=3` + — but the channels layer de-chunks it, so the adapter reads raw JSON + bytes from its `control` `RecvStream` in both cases. The + `ControlMessage` enum (resize, signal, eof, exit) is unchanged. + +5. **The exit-chunk-is-last invariant (ADR-055) generalizes.** In direct + mode, the exit chunk is the last 5-byte chunk before stream close + (ADR-055). In channels mode, the exit control message is the last data on + `stream_type 3` before `channel/close` is sent on channel 0 + (ADR-073 §channel/close). The ordering invariant is the same — exit + before close — but the mechanism differs: 5-byte chunk ordering (direct) + vs. `stream_type 3` ordering + `channel/close` after pump completion + (channels, REQ-CH-06). + +### What does NOT change + +- **`TtyBackend` trait, `TtyHandle`, `TtyControl`** (ADR-053) — unchanged. + Backends don't know about channels or direct mode. +- **`DockerTtyBackend`, `LocalTtyBackend`** — unchanged. They implement + `TtyBackend::allocate()` and return a `TtyHandle`. +- **`ControlMessage` enum** — unchanged. The JSON shape is the same in both + modes. +- **The `alknet/tty` ALPN string** — unchanged. Direct connections use it; + channels `channel/open` requests it. + +### Crate dependency + +`alknet-tty` does **not** depend on `alknet-channels` unconditionally. The +channels-integration code is behind a `channels` feature on `alknet-tty`. +When the feature is off, `TtyAdapter` only supports direct mode (the +existing behavior). When the feature is on, the adapter branches into +channels mode for channels-backed connections. This preserves ADR-003's +no-handler-depends-on-another-handler rule for the default build; the +feature-gated dependency is opt-in, same as `alknet-docker`'s `tty` feature +(ADR-061). + +## Consequences + +**Positive:** +- The TTY crate's direct mode is unchanged — existing `alknet/tty` + deployments (browser terminals over WebSocket, direct QUIC TTY) keep + working with the 5-byte format. +- The channels path uses the channels layer's de-chunking — no double- + chunking (5-byte inside 9-byte). The TTY adapter sees clean sub-streams. +- The `TtyBackend` trait is insulated — backends don't know which mode the + adapter is in. Docker, SSH, and local backends work in both modes without + changes. +- The control channel and exit-chunk invariant carry forward cleanly — the + `ControlMessage` enum and ordering semantics are mode-independent. + +**Negative:** +- `alknet-tty` has two session-driving entry points (`drive_session_direct` + vs `drive_session_channels`). This is the necessary cost of supporting + both direct and channels modes without double-chunking. The alternative + (always use channels format, even for direct) would break existing direct + deployments and add 4 bytes of overhead per chunk for no benefit. +- The `channels` feature on `alknet-tty` adds a dependency edge + (`alknet-tty` → `alknet-channels`, feature-gated). This is the same + pattern as `alknet-docker`'s `tty` feature (ADR-061) and is opt-in. +- ADR-052's scope is amended (from "the TTY wire format" to "the TTY wire + format for direct connections"). This is a scope clarification, not a + format change — the 5-byte format itself is unchanged. + +## Door type + +**One-way (scope amendment) + two-way (feature gate).** ADR-052's scope +amendment (direct-only) is one-way — once the channels path exists, +re-merging the formats would require unifying 5-byte and 9-byte chunk +handling, which is a rewrite. The `channels` feature gate is two-way — it +can be removed if channels integration is no longer needed. + +## References + +- ADR-052: alknet-tty wire format (amended — scoped to direct connections) +- ADR-053: TtyBackend trait and TtyHandle (unchanged by this ADR) +- ADR-055: exit-chunk-is-last (generalized by this ADR + ADR-073) +- ADR-057: alknet-tty does not depend on alknet-call (preserved — the + channels feature is on alknet-channels, not alknet-call) +- ADR-071: channels wire format (the 9-byte format the channels path uses) +- ADR-074: ChannelBidiStreamSource / `into_sub_streams` (the accessor the + channels path uses) +- ADR-061: DockerTtyBackend in alknet-docker (the feature-gated dependency + pattern this ADR mirrors) +- `docs/research/alknet-channels/phase-0-findings.md` §DP-3, §OQ-CH-02, + §Relationship to Existing Crates / alknet-tty \ No newline at end of file diff --git a/docs/architecture/decisions/078-two-pump-shutdown-on-completion.md b/docs/architecture/decisions/078-two-pump-shutdown-on-completion.md new file mode 100644 index 0000000..488a982 --- /dev/null +++ b/docs/architecture/decisions/078-two-pump-shutdown-on-completion.md @@ -0,0 +1,138 @@ +# ADR-078: Two-Pump Shutdown-on-Completion Pattern + +## Status + +Accepted + +## Context + +The de-risk POC (`docs/research/alknet-channels/poc-summary.md` §Issues +Surfaced #7) surfaced a deadlock in the tunnel handler's two-pump shape. +The naive `tokio::try_join!(c2t, t2c)` deadlocks: each pump waits for the +other's EOF, which only comes once the *opposite* pump completes and shuts +down its sink. The TTY adapter avoids this because its three pumps +coordinate via the `exit_code` future — a third signal. A two-pump handler +(tunnel, SSH `direct-tcpip`) has no such third signal. + +The fix the POC found: shut down the peer's sink when one pump completes. +`c2t` (client→target) shuts down `tcp_write` on EOF; `t2c` (target→client) +shuts down `send` on EOF. This is the `pump_session` shape with an explicit +shutdown-on-completion step that the TTY adapter doesn't need (because +TTY's three pumps coordinate via the exit future). + +This pattern will recur — any handler with a pump-driven two-direction +shape (tunnel, SSH `direct-tcpip`, future port-forward) needs it. Getting it +wrong hangs channels silently. The POC hung; the spec must pin the pattern. + +## Decision + +### The two-pump pattern is a documented contract + +A two-pump handler (two `tokio::io::copy` pumps, one per direction) MUST +shut down the opposite sink when one pump completes. `tokio::try_join!` +alone deadlocks because each pump waits for the other's EOF, which only +comes after the opposite pump shuts down its sink. + +```rust +// Correct two-pump shape: +let (mut send, mut recv) = connection.accept_bi().await?; +let mut tcp = TcpStream::connect(target).await?; +let (mut tcp_read, mut tcp_write) = tcp.into_split(); + +let c2t = async { + tokio::io::copy(&mut recv, &mut tcp_write).await?; + tcp_write.shutdown().await.ok(); // shut down the peer's sink + Result::<_, std::io::Error>::Ok(()) +}; +let t2c = async { + tokio::io::copy(&mut tcp_read, &mut send).await?; + send.shutdown().await.ok(); // shut down the peer's sink + Result::<_, std::io::Error>::Ok(()) +}; +tokio::try_join!(c2t, t2c)?; +``` + +When `c2t` completes (recv EOF), it shuts down `tcp_write`, which causes +`t2c`'s `tcp_read` to eventually EOF, completing `t2c`. When `t2c` +completes (tcp_read EOF), it shuts down `send`, which causes `c2t`'s `recv` +to eventually EOF. Either pump completing unblocks the other. + +### Where the pattern lives + +The pattern is a **handler-level contract**, not a channels-layer concern. +The channels layer routes chunks; the handler owns its pump logic. This ADR +documents the pattern so handlers don't reimplement it incorrectly. + +The channels spec (`channels-adapter.md`) documents the pattern in the +handler-integration section. The tunnel handler (the first two-pump +consumer) implements it. Future two-pump handlers (SSH `direct-tcpip`) +follow the same shape. + +### Consideration: a helper in alknet-core + +The POC summary suggested "a helper in `alknet-core` that encapsulates the +'two-pump with shutdown-on-completion' shape so handlers don't reimplement +it." This is an implementation convenience, not an architecture decision. +The contract is the shutdown-on-completion pattern; whether it's a helper +function or inline code in each handler is a two-way-door implementation +detail. + +**Decision: do not add a core helper yet.** The pattern is ~10 lines of +inline code. A helper would be called from handler crates (`alknet-tty`, +the future tunnel crate, the future SSH crate), which means the helper's +signature (`fn pump_bidi(recv: R, send: W, ...) -> impl Future`) is a +cross-crate API surface. Extracting it prematurely (with one consumer — the +POC's tunnel) would bake in a shape that the second consumer (SSH +`direct-tcpip`) might not fit. The pattern is documented; the helper is +extracted when two real consumers exist and their shapes converge. This is +a genuine deferral (blocked on: a second two-pump handler existing), not a +hedge — the contract is decided (shutdown-on-completion), only the +extraction is deferred. + +### The three-pump pattern (TTY) is unaffected + +The TTY adapter's `pump_session` (three pumps: stdout, stderr, +client→backend, coordinating via the `exit_code` future) does not have this +deadlock because the `exit_code` future is the third signal that unblocks +the pumps. This ADR applies only to two-pump handlers. The TTY adapter is +unchanged. + +## Consequences + +**Positive:** +- The two-pump deadlock is documented as a contract, not left as a POC + finding. Handlers that follow the pattern don't hang. +- The pattern is handler-level — the channels layer stays a re-framing + proxy, not a pump-logic owner. +- The three-pump pattern (TTY) is unaffected — the ADR scopes itself to + two-pump handlers. + +**Negative:** +- Each two-pump handler implements the shutdown-on-completion inline (~10 + lines). Until a core helper is extracted (deferred, blocked on a second + consumer), the pattern is copy-paste with documentation. This is the + correct trade-off: the contract is decided, the extraction is deferred on + a real blocker (shape convergence across consumers), not hedged. + +## Door type + +**One-way.** The shutdown-on-completion contract is a correctness invariant +— two-pump handlers MUST shut down the opposite sink on pump completion, or +they deadlock. This is not a preference; it is a correctness requirement. + +The core helper extraction is a **deferred decision** (OQ-57, +deferred(scope)), not a door-type attribute. Its door type is two-way (a +helper function is additive), but the extraction is not decided in this +ADR — see OQ-57 for the blocking condition (a second two-pump handler +existing, so shape convergence is observable). + +## References + +- ADR-074: ChannelBidiStreamSource (the `accept_bi` that yields the stream + pair the pumps operate on) +- ADR-055: exit-chunk-is-last (the three-pump TTY invariant — the pattern + this ADR does NOT touch) +- `docs/research/alknet-channels/poc-summary.md` §Issues Surfaced #7 (the + deadlock the POC found and fixed) +- `crates/alknet-tty/src/adapter.rs` — `pump_session` (the three-pump + reference shape) \ No newline at end of file diff --git a/docs/architecture/decisions/079-hub-relay-translate-not-forward.md b/docs/architecture/decisions/079-hub-relay-translate-not-forward.md new file mode 100644 index 0000000..f940003 --- /dev/null +++ b/docs/architecture/decisions/079-hub-relay-translate-not-forward.md @@ -0,0 +1,176 @@ +# ADR-079: Hub Relay — Translate, Not Transparently Forward + +## Status + +Accepted + +## Context + +The hub is the architectural role (ADR-029, ADR-034) that bridges peers and +browsers. With channels, the hub holds one channels connection per leg +(browser↔hub, hub↔spoke) and relays channels between them. The phase-0 +research (`docs/research/alknet-channels/phase-0-findings.md` §OQ-CH-11, +§The hub relay) identified the key question: does the hub *translate* +`channel/open` (terminate channel 0 on both legs, re-issue the open on the +spoke leg) or *transparently forward* (pass the call operation through +unchanged)? + +This is the most under-specified part of the research for something that is +the *primary motivation* for the channels crate (§Hub Motivation: the +multi-transport collapse). The research said "Phase 1 must specify whether +the hub translates or transparently forwards, and how the `channel_id` +mapping is maintained." + +The answer is derivable from the existing machinery: +- The hub terminates channel 0 on both legs (it runs its own `CallAdapter` + per leg — ADR-072). +- The hub's `CallAdapter` receives the browser's `channel/open` as a call + operation, runs `AccessControl::check` with the browser's identity, then + forwards via `from_call` to the spoke (the hub as caller, the browser as + `forwarded_for` — ADR-032 §3). +- The spoke allocates its `channel_id` and returns it; the hub maps + browser-id ↔ spoke-id. + +Transparent forwarding (passing the `channel/open` call operation through +without the hub's `CallAdapter` terminating it) would bypass the hub's +`AccessControl::check` and the `forwarded_for` auth chain — the hub would +not authenticate the open, and the spoke would see the browser as the direct +caller (not the hub), breaking the ADR-032/ADR-050 auth model. Translation +is the only option that preserves the auth model. + +## Decision + +### The hub translates, not transparently forwards + +The hub's relay has two layers: + +1. **Call-protocol layer (channel 0): translate.** The hub terminates + channel 0 on both legs. A `channel/open` from the browser is received by + the hub's `CallAdapter`, which: + 1. Runs `AccessControl::check` on `channel/open` with the browser's + identity (bearer token resolved per ADR-034). If denied → + `channel:forbidden` to the browser. + 2. Issues a *new* `channel/open` on the spoke's channel 0 via `from_call`, + with the hub as caller and the browser as `forwarded_for` (ADR-032 + §3). The spoke's `AccessControl::check` sees the hub as the direct + peer (authorized per ADR-050) and the browser as `forwarded_for`. + 3. The spoke allocates its `channel_id` and returns it. + 4. The hub opens a matching channel on the browser's side (the hub is now + the *responder* for the browser leg, *initiator* for the spoke leg) + and records the `channel_id` mapping: `browser_id ↔ spoke_id`. + +2. **Data-channel layer: byte-forward with `channel_id` rewrite.** Once the + mapping is established, the relay reads chunks for `browser_id` off the + browser's channels connection, rewrites the `channel_id` field to + `spoke_id`, and writes them onto the spoke's channels connection — and + vice versa. The relay does not parse the payload; it does not know if the + bytes are TTY chunks, SSH frames, or tunnel data. The channels layer on + each end does the chunk↔stream conversion; the relay just moves bytes + between two `AsyncRead + AsyncWrite` pairs with a 4-byte header rewrite. + +### `channel_id` mapping + +The hub maintains a `HashMap` per (browser, spoke) +pair — the relay map. On `channel/open` (translated), the mapping is +inserted. On `channel/close` (translated the same way), the mapping is +removed. The relay task per channel reads the map to determine the rewrite +target. + +`channel/control` operations on channel 0 carry `channel_id` in their JSON +payload (not in the chunk header). The hub's `CallAdapter` translates these +too: the browser's `channel/control` for `browser_id` is re-issued on the +spoke leg with `spoke_id` in the payload. The relay does not touch +`channel/control` — it's a call operation, translated by the hub's +`CallAdapter`, not byte-forwarded. + +### What the hub runs + +| Leg | What the hub runs | +|-----|-------------------| +| Browser leg | `ChannelsAdapter` (the relay's read/demux) + `CallAdapter` (channel 0, for the hub's own ops + translating the browser's ops) | +| Spoke leg | `ChannelsAdapter` + `CallAdapter` (same) | +| Relay | Per-channel byte-forward tasks with `channel_id` rewrite | + +The hub never runs a handler for `alknet/tty`, `alknet/ssh`, or +`alknet/tunnel`. It runs `alknet/channels` (the relay) and `alknet/call` +(for its own hub-level operations + translation). The endpoints at each end +do the protocol work. + +### What the hub still owns (unchanged from phase-0 §What the hub does still own) + +- **Routing:** which spoke serves `container:abc123`? The hub's resource + registry / ownership store (ADR-050), queried via call operations on + channel 0. Channels doesn't touch this. +- **ACL at the hub:** does this browser's identity have `channel:open` scope + for `alknet/ssh` to `spoke-X`? `AccessControl::check` on `channel/open`, + run by the hub's `CallAdapter` before it forwards. Channels doesn't touch + this. +- **Relay lifecycle:** when a browser disconnects, the hub tears down the + spoke-side channels (and vice versa). `channel/close` on each channel, or + a transport-level close the channels layer observes (REQ-CH-02). + +### Scope note: this is a hub-crate concern, not a channels-crate concern + +This ADR defines the relay *contract* (translate channel 0, byte-forward +data channels with ID rewrite) so the channels crate's `ChannelManager` +exposes the interface the relay needs (`open_channel_stream(channel_id, +stream_type) -> (SendStream, RecvStream)` for the byte-forward pumps). The +relay *implementation* lives in `alknet-hub` (or a downstream hub like +alkapi), not in `alknet-channels`. The channels crate is ALPN-blind and +does not know it is being relayed. + +## Consequences + +**Positive:** +- The auth model reuses cleanly: the hub's `AccessControl::check` + + `forwarded_for` (ADR-032) is the existing machinery, not a new one. The + spoke sees the hub as caller, the browser as `forwarded_for` — the + kernel/user-land + forwarded-for model from ADR-050. +- The relay is one pump function per channel, not per (protocol × transport) + cell. The hub's complexity is O(channels), not O(protocols × transports × + spokes). +- The hub never runs protocol-specific handlers — it doesn't parse TTY + chunks, SSH frames, or tunnel data. It moves bytes and translates call + operations. +- `channel/resources/subscribe` (ADR-073) gives the hub a live view of each + spoke's resources, which the hub aggregates and exposes to the browser. + +**Negative:** +- The hub maintains a `channel_id` mapping per (browser, spoke) pair. This + is per-channel state, not per-connection — a hub with many concurrent + browser sessions each with multiple channels has a non-trivial map. The + map is `HashMap` per pair — cheap per entry, but the entry count + is (browsers × channels-per-browser). Bounded by `max_channels` (ADR-076) + per connection. +- The translate path adds one `channel/open` round-trip per relayed channel + (browser→hub, hub→spoke). This is the same cost as any hub-relayed call + operation and is not avoidable without transparent forwarding, which + breaks the auth model. +- `channel/control` translation requires the hub's `CallAdapter` to rewrite + `channel_id` in the JSON payload. This is a small but real translation + step — the hub is not a pure byte relay for channel 0. + +## Door type + +**One-way.** The translate-vs-forward decision is structural: transparent +forwarding would bypass the hub's `AccessControl::check` and the +`forwarded_for` chain, breaking the auth model. Reversing to transparent +forwarding after deployments exist would require re-architecting the hub's +auth path. The `channel_id` mapping strategy (`HashMap` per pair) is two-way +— an implementation detail that can change without breaking the contract. + +## References + +- ADR-029: peer-graph routing model (the hub's role) +- ADR-032: forwarded-for identity (the auth chain the translate path uses) +- ADR-034: outgoing-only X.509 and the three peer roles (browser identity + resolution) +- ADR-050: dynamic resource ownership (the ownership store the hub queries) +- ADR-072: channel 0 is pre-negotiated `alknet/call` (what the hub + terminates on each leg) +- ADR-073: channel lifecycle operations (what the hub translates) +- ADR-075: ChannelsAdapter and ChannelManager (the interface the relay uses) +- `docs/research/alknet-channels/phase-0-findings.md` §Hub Motivation, + §The hub relay, §OQ-CH-11 +- `docs/architecture/crates/hub/README.md` — the hub crate (the relay + implementation's home) \ No newline at end of file diff --git a/docs/architecture/decisions/080-channelclient.md b/docs/architecture/decisions/080-channelclient.md new file mode 100644 index 0000000..93206e1 --- /dev/null +++ b/docs/architecture/decisions/080-channelclient.md @@ -0,0 +1,156 @@ +# ADR-080: ChannelClient — the Client Side of a Channels Connection + +## Status + +Accepted + +## Context + +Both sides of a channels connection do the demux/mux work. The server side +is a `ProtocolHandler` (`ChannelsAdapter::handle`, ADR-075). The client side +needs a symmetric type — `ChannelClient` — that opens a transport, runs the +demux/mux, and exposes `open_channel(alpn, params) -> Channel` to the +application. This is the channels analogue of `CallClient` (server: +`CallAdapter`; client: `CallClient`) in the call protocol. + +The phase-0 research (`docs/research/alknet-channels/phase-0-findings.md` +§OQ-CH-14) clarified that there are two concerns here: + +1. **`ChannelClient` (channels-specific):** the client type for channels + connections. Decision-ready — build it in `alknet-channels`, same shape + as `CallClient`. +2. **`AlknetClient` (core, transport-polymorphic):** a general downstream- + facing client that crates use to connect to an alknet endpoint. Genuinely + deferred — blocked on a second *transport's* client existing (OQ-55 + tracks this correctly). `ChannelClient` over QUIC does not unblock + `AlknetClient` because it's the same transport shape as `CallClient`. + +This ADR decides #1. #2 stays deferred per OQ-55. + +## Decision + +### `ChannelClient` in `alknet-channels` + +```rust +pub struct ChannelClient { + manager: ChannelManager, + // The transport-side demux/mux, running in a background task. + ... +} + +impl ChannelClient { + /// Open a channels connection to a peer. Dials the transport (QUIC + /// initially), establishes the channels connection, preinstalls channel + /// 0 (alknet/call), and returns the client. + pub async fn connect(addr: SocketAddr, credentials: CallCredentials) + -> Result; + + /// Open a data channel with the given ALPN and params. Sends + /// `channel/open` on channel 0, waits for the response, and returns + /// the channel's sub-streams. + pub async fn open_channel( + &self, + alpn: &str, + stream_types: &[u8], + params: Value, + direction: ChannelDirection, + ) -> Result; + + /// The call-protocol connection on channel 0, for invoking channel + /// lifecycle operations and any other call ops the peer exposes. + pub fn call(&self) -> &CallConnection; +} + +pub struct Channel { + pub channel_id: u32, + pub stream_types: Vec, + /// The sub-streams, accessible via the BidiStreamSource (accept_bi) or + /// into_sub_streams() — ADR-074. + pub source: ChannelBidiStreamSource, +} +``` + +### QUIC-only initially + +`ChannelClient::connect` dials a QUIC connection (via the same `quinn` +endpoint `CallClient` uses) and wraps it as a channels connection. This is +the same transport shape as `CallClient`. When a second transport's client +exists (HTTP, TCP+TLS, WebTransport — per OQ-55), the dial can be +generalized. Until then, `ChannelClient` is QUIC-only — the same posture as +`CallClient`. + +### Bidirectionality preserved + +The channels protocol is bidirectional — either side can open a channel +(ADR-073 §direction semantics). `ChannelClient::open_channel` supports both +`ChannelDirection::InitiatorToResponder` and +`ChannelDirection::ResponderToInitiator`. The client is not "the client +side" in the sense of only initiating — it can also receive `channel/open` +requests from the peer (the peer initiates, the client's `ChannelManager` +responds). This mirrors the call protocol's operation overlay (each side +populates what operations they expose). + +This means `ChannelClient` is not purely a "client" in the request/response +sense — it's one endpoint of a bidirectional channels connection. The name +`ChannelClient` follows the `CallClient` convention (the side that dialed), +not a request/response role. + +### Relationship to `AlknetClient` (OQ-55 — deferred) + +`ChannelClient` is a standalone client, not a specialization of a core +`AlknetClient`. The `AlknetClient` extraction (OQ-55) is genuinely deferred: +blocked on a second *transport's* client existing, not on a second client +existing. `ChannelClient` over QUIC is a second client but the same +transport shape as `CallClient` — it doesn't give enough information to +extract the transport-polymorphic dial seam. Extracting a QUIC-shaped +connector to core and naming it `AlknetClient` would bake QUIC in as *the* +establishment shape — the same welding ADR-065 unwound on the server side. + +When `AlknetClient` is eventually extracted (after a second transport's +client exists), `ChannelClient` and `CallClient` both refactor onto it. +Until then, they are independent clients with duplicated boilerplate (each +rebuilds verifier selection — ~20 lines). The friction is duplicated +boilerplate, not a missing capability. + +## Consequences + +**Positive:** +- `ChannelClient` gives the channels crate a symmetric client/server pair, + matching the call protocol's `CallAdapter`/`CallClient` shape. +- Bidirectionality is preserved — the client can both initiate and receive + `channel/open`. +- The `AlknetClient` deferral (OQ-55) is not blocked by `ChannelClient` — + they are independent concerns. `ChannelClient` builds standalone; the + core extraction happens later when the blocker clears. + +**Negative:** +- `ChannelClient` duplicates ~20 lines of verifier-selection boilerplate + from `CallClient`. This is the known cost of not extracting `AlknetClient` + yet (OQ-55). Acceptable until the second transport's client exists. +- `ChannelClient` is QUIC-only. A non-QUIC channels client (e.g., a browser + over WebTransport) builds separately until `AlknetClient` is extracted. + This is the same posture as `CallClient` and is not a channels-specific + limitation. + +## Door type + +**One-way.** The `ChannelClient::connect` / `open_channel` / `call` / +`subscribe_resources` API is the handler-facing surface; changing it after +consumers exist is a rewrite. + +The `AlknetClient` extraction is a **deferred decision** (OQ-55, +deferred(scope)), not a door-type attribute. Its door type is two-way (the +extraction is a refactor, not a wire-format change), but it is not decided +in this ADR — see OQ-55 for the blocking condition. + +## References + +- ADR-073: channel lifecycle operations (`open_channel` sends `channel/open`) +- ADR-074: ChannelBidiStreamSource (what `Channel.source` wraps) +- ADR-075: ChannelManager (the shared state `ChannelClient` holds) +- OQ-55: AlknetClient / client establishment extraction (the deferred core + concern this ADR does NOT block on) +- `docs/research/alknet-channels/phase-0-findings.md` §OQ-CH-14 (the + research-scope question this ADR carries forward) +- `docs/architecture/crates/call/client-and-adapters.md` — `CallClient` (the + shape `ChannelClient` mirrors) \ No newline at end of file diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index a48ed88..7007a1f 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -167,6 +167,13 @@ Door type is separate from whether a decision is made. A two-way door is a decis | [OQ-53](questions/053-backoff-config-defaults.md) | BackoffConfig default policy | open | two | low | | [OQ-54](questions/054-inbound-worker-hook-placement.md) | Inbound worker on_worker_connected hook placement | resolved | two | low | +### alknet-channels + +| OQ | Title | Status | Door | Pri | +|----|-------|--------|------|-----| +| [OQ-56](questions/056-full-channel-level-flow-control-windowing.md) | Full Channel-Level Flow-Control Windowing | deferred(scope) | two | low | +| [OQ-57](questions/057-two-pump-helper-extraction.md) | Two-Pump Helper Extraction to alknet-core | deferred(scope) | two | low | + ## Deferred / Blocked The safe-exit visibility surface. These questions are parked because the @@ -237,3 +244,25 @@ filtering the tables above. - **Priority**: medium - **Full file**: [OQ-55](questions/055-alknetclient-establishment-extraction.md) +### OQ-56: Full Channel-Level Flow-Control Windowing + +- **Blocked on**: a real deployment observes head-of-line blocking on a + saturated channel where the bounded-buffer's stop-reading mitigation is + insufficient (e.g., a high-throughput file transfer over a tunnel that + saturates a channel and causes frequent demux stalls affecting other + channels). The intended use cases (TTY, SSH, tunnels) are not + high-throughput in the HOL-blocking sense; the trigger requires a + high-throughput use case. +- **Priority**: low +- **Full file**: [OQ-56](questions/056-full-channel-level-flow-control-windowing.md) + +### OQ-57: Two-Pump Helper Extraction to alknet-core + +- **Blocked on**: a second two-pump handler existing (the tunnel handler is + the first; SSH `direct-tcpip` will be the second), so the shape + convergence is observable. Extracting the helper from one consumer would + bake in a shape that the second might not fit. The shutdown-on-completion + *contract* is decided (ADR-078); only the *helper extraction* is deferred. +- **Priority**: low +- **Full file**: [OQ-57](questions/057-two-pump-helper-extraction.md) + diff --git a/docs/architecture/questions/056-full-channel-level-flow-control-windowing.md b/docs/architecture/questions/056-full-channel-level-flow-control-windowing.md new file mode 100644 index 0000000..7892664 --- /dev/null +++ b/docs/architecture/questions/056-full-channel-level-flow-control-windowing.md @@ -0,0 +1,34 @@ +# OQ-56: Full Channel-Level Flow-Control Windowing + +- **Origin**: `docs/research/alknet-channels/phase-0-findings.md` §DP-5, + §OQ-CH-03; `docs/architecture/decisions/076-backpressure-channel-limits-id-reuse.md` +- **Status**: deferred(scope) +- **Door type**: two-way (additive — per-channel window tracking does not + change the wire format) +- **Priority**: low +- **Blocked on**: a real deployment observes head-of-line blocking on a + saturated channel where the bounded-buffer's stop-reading mitigation is + insufficient. The trigger is specific: a channel whose consumer is + persistently slower than its producer, causing the demux to stall that + channel's reads frequently enough that other channels' throughput is + measurably affected. The intended use cases (TTY, SSH, tunnels) are not + high-throughput in the HOL-blocking sense; the trigger requires a + high-throughput use case (e.g., file transfer over a tunnel) that + saturates a channel. +- **Resolution**: Not yet decidable. The bounded-buffer backpressure + (ADR-076, default 1 MiB per `(channel_id, stream_type)`) is the decided + v1 mechanism — validated by the POC's 1 MiB `tunnel_large_payload` test + with no deadlock and no cross-channel blocking. Full channel-level + windowing (SSH-style sliding-window per channel) is an additive extension + that does not change the wire format; it adds per-channel window tracking + to the demux/mux. The decision to add it depends on whether the + bounded-buffer mitigation is sufficient in practice, which can only be + determined by a deployment that hits the limitation. +- **What does NOT block on this**: the bounded-buffer mechanism is decided + and is the v1 implementation. Full windowing is an extension, not a + prerequisite. The channels crate ships with bounded-buffer backpressure; + full windowing is added if and only if the trigger condition is observed. +- **Cross-references**: ADR-076 (bounded-buffer decision), ADR-071 (wire + format — unchanged by windowing extension), + `docs/research/alknet-channels/poc-summary.md` §POC Target 1 + (backpressure validation) \ No newline at end of file diff --git a/docs/architecture/questions/057-two-pump-helper-extraction.md b/docs/architecture/questions/057-two-pump-helper-extraction.md new file mode 100644 index 0000000..fe062c3 --- /dev/null +++ b/docs/architecture/questions/057-two-pump-helper-extraction.md @@ -0,0 +1,32 @@ +# OQ-57: Two-Pump Helper Extraction to alknet-core + +- **Origin**: `docs/research/alknet-channels/poc-summary.md` §Issues + Surfaced #7; `docs/architecture/decisions/078-two-pump-shutdown-on-completion.md` +- **Status**: deferred(scope) +- **Door type**: two-way (additive — a helper function does not change any + API surface; handlers that inline the pattern continue to work) +- **Priority**: low +- **Blocked on**: a second two-pump handler existing, so the shape + convergence is observable. The tunnel handler is the first two-pump + consumer; the SSH `direct-tcpip` channel will be the second. Extracting + the helper from one consumer (the tunnel) would bake in a shape that the + second consumer (SSH) might not fit — the `fn pump_bidi(recv: R, + send: W, ...) -> impl Future` signature is a cross-crate API surface if + it lives in `alknet-core`. The trigger is: two real two-pump handlers + exist and their inline implementations have converged on the same shape. +- **Resolution**: Not yet decidable. The shutdown-on-completion *contract* + is decided (ADR-078) — a two-pump handler MUST shut down the opposite + sink when one pump completes, or it deadlocks. The *helper* extraction is + an implementation convenience: ~10 lines of inline code per handler vs. a + shared function in `alknet-core`. The contract is pinned; only the + extraction is deferred. The helper is extracted when two real consumers + exist and their shapes converge, so the extraction is grounded in two + implementations rather than guessed from one. +- **What does NOT block on this**: the two-pump pattern is documented + (ADR-078) and the tunnel handler implements it inline. The SSH crate's + `direct-tcpip` handler will implement it inline too. Both work without a + shared helper. The friction is copy-paste with documentation (~10 lines), + not a missing capability. +- **Cross-references**: ADR-078 (the shutdown-on-completion contract), + ADR-074 (the `accept_bi` that yields the stream pair the pumps operate + on), `docs/research/alknet-channels/poc-summary.md` §Issues Surfaced #7 \ No newline at end of file