feat(websocket): data-channel + op/register wiring (review 006 Unit 2+3)

The WS path wires the alkcall 0.3 per-session mechanisms — the OQ-05
deferred half (review-003 WS-20/21/22/25/26, the decisions WS-24/WS-25
resolved upstream):

- install_channel_zero reworked per the ADR-047 §4 amendment #2 shape:
  fork the deployment's base registry, register the generic channel
  ops (channel/close, channel/control, channel/resources/subscribe —
  WS-21), the deployment's openable ALPNs (ChannelCore::register_openable,
  WS-22), the bootstrap discovery set closed over the fork
  (install_bootstrap_discovery, F-06), and op/register (WS-25; the
  collision set is the fork per review-005 G-03), then dispatch over
  the fork. The session's ChannelsPolicy rides the hook (one policy
  instance across open wrappers and the demux teardown path).
- OpenableAlpn { spec, open_handler } + HttpAdapter::with_ws_openable_alpns,
  threaded RouterState -> SessionState -> hook, with the OpenableAlpns
  request-extension fallback (mirroring ChannelsPolicy/WsTimeouts).
- WsSessions retains the channel-0 Arc<CallConnection> (WS-26) with a
  self-removing guard (ConnectionGuard); live_connections() is the
  deployment-visible surface.
- UP-01: ALREADY_EXISTS maps to 409 Conflict in the gateway error map.
- from_wss import excludes the protocol-session ops (bootstrap set +
  channel lifecycle ops): the fork serves them per session, and proxying
  session-scoped machinery (e.g. channel/close across sessions) would be
  nonsense. Discovery runs first, the filter is the listing minus those
  names.
- adapter_install_channel_zero cfg matches its caller (WS-27); it
  inherits the reworked hook (session ops now served in the from_wss
  test-server producer too).

Gates (Unit 3, tests/ws_upgrade_session.rs; the WS-23 e2e set):
open -> channel_id -> discoverable in services/list -> chunks both
ways -> handler sees bytes; channel/close resolves + ledger decrement;
cap denial (channel:-prefixed); mid-open disconnect teardown; TooLarge
demux resync through the WS path (16 MiB + 1 skip consumed);
op/register announce + overlay-collision + serving-registry-collision
ALREADY_EXISTS through the WS path.

call_and_await now filters by request id and tolerates data-channel
chunks (a prior Sub's trailing call.completed may interleave).

Verification: cargo test 454 (default) / 582 (all-features), clippy
both sides -D warnings clean, fmt clean, doc clean.
This commit is contained in:
2026-09-04 15:47:15 +00:00
parent 90790374fa
commit 030c5efa51
8 changed files with 891 additions and 28 deletions
+539 -4
View File
@@ -4,7 +4,7 @@
//! Grows from the ws-byte-adapter POC's validated suite.
use alkcall::core::auth::{Identity, IdentityProvider};
use alkcall::protocol::wire::{EventEnvelope, ResponseEnvelope, EVENT_RESPONDED};
use alkcall::protocol::wire::{EventEnvelope, ResponseEnvelope, EVENT_ERROR, EVENT_RESPONDED};
use alkcall::registry::discovery::{services_list_handler, services_list_spec};
use alkcall::registry::registration::{
make_handler, Handler, HandlerKind, HandlerRegistration, OperationProvenance, OperationRegistry,
@@ -250,16 +250,23 @@ async fn call_and_await(
tokio::time::Instant::now() < deadline,
"timed out waiting for response to {request_id}"
);
// A prior request's trailing `call.completed` (a Sub pump's
// natural end) may still be in flight — skip frames for other
// request ids. Data-channel chunks (id != 0) are skipped too:
// this helper asserts only channel-0 responses.
if let Some(env) = frames.next_frame() {
return env;
if env.id == request_id {
return env;
}
}
let bin = ws.next_binary(std::time::Duration::from_millis(500)).await;
match bin {
Some(bytes) => {
chunks.push(&bytes);
while let Some((channel_id, payload)) = chunks.next_chunk() {
assert_eq!(channel_id, 0, "response must ride channel 0");
frames.push(&payload);
if channel_id == 0 {
frames.push(&payload);
}
}
}
None => panic!("ws closed unexpectedly while awaiting {request_id}"),
@@ -969,3 +976,531 @@ async fn ws_timeouts_extension_sets_the_idle_window_on_a_bare_registry_route() {
);
ws.close().await;
}
// --- Unit 2/3 gates (review 006): the data-channel + op/register wiring ---
use alkcall::channels::operations::OpenHandler;
/// The data-plane echo the `OpenHandler` runs on the allocated
/// channel's `Connection`: echo every inbound byte block back, then
/// drain until EOF. The WS-test client writes raw (channel_id,
/// payload) chunks; the handler reads the channel's BiStream and
/// writes the echo — the exact data-plane shape ADR-067 promises.
fn echo_open_handler() -> OpenHandler {
Arc::new(move |_input, channel_conn, _auth| {
tokio::spawn(async move {
let mut stream = match channel_conn.accept_bi().await {
Ok(s) => s,
Err(_) => return,
};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let mut buf = [0u8; 4096];
loop {
match stream.read(&mut buf).await {
Ok(0) | Err(_) => break,
Ok(n) => {
if stream.write_all(&buf[..n]).await.is_err() {
break;
}
}
}
}
let _ = stream.shutdown().await;
})
})
}
/// The open-op spec for the test's openable ALPN (`channels/echo/sub`,
/// opening the `alk/echo` data plane).
fn echo_open_spec() -> OperationSpec {
OperationSpec::new(
"channels/echo/sub",
OperationType::Sub,
Visibility::External,
serde_json::json!({
"type": "object",
"properties": {},
"additionalProperties": false
}),
serde_json::json!({
"type": "object",
"properties": { "channel_id": { "type": "integer" } }
}),
vec![],
AccessControl::default(),
None,
)
.with_channel_open(alkcall::registry::spec::ChannelOpenSpec::new("alk/echo"))
}
/// Spawn a WS server whose adapter declares the echo openable ALPN and
/// a small per-identity channel cap; returns the WS URL and the shared
/// policy (for the ledger assertions).
async fn spawn_openable_ws_server(
registry: Arc<OperationRegistry>,
cap: usize,
) -> (
String,
Arc<alkcall::channels::policy::PerIdentityChannelPolicy>,
) {
use alkhttp::websocket::OpenableAlpns;
let policy = Arc::new(alkcall::channels::policy::PerIdentityChannelPolicy::new(
cap,
));
let policy_dyn: Arc<dyn alkcall::channels::policy::ChannelLifecyclePolicy> =
Arc::clone(&policy) as Arc<dyn alkcall::channels::policy::ChannelLifecyclePolicy>;
let openables = OpenableAlpns(Arc::from([alkhttp::websocket::OpenableAlpn::new(
echo_open_spec(),
echo_open_handler(),
)]));
let app = axum::Router::new()
.route(
"/alk/channels",
axum::routing::get(alkhttp::websocket::ws_upgrade_handler),
)
.layer(axum::middleware::from_fn_with_state(
provider_with(vec![("tok-1", identity("alice", &[]))]),
alkhttp::websocket::ws_bearer_auth,
))
.layer(axum::Extension(openables))
.layer(axum::Extension(alkhttp::websocket::ChannelsPolicy(
policy_dyn,
)))
.with_state(registry);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = format!("ws://{}", listener.local_addr().unwrap());
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
(format!("{addr}/alk/channels"), policy)
}
/// Drain channel-0 chunks until a complete envelope shows up (shared
/// with `call_and_await`'s loop but tolerant of interleaved
/// non-zero-channel chunks).
async fn await_envelope(ws: &mut WsClient, request_id: &str) -> EventEnvelope {
let mut chunks = ChunkAssembler::new();
let mut frames = FrameAssembler::new();
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
assert!(
tokio::time::Instant::now() < deadline,
"timed out waiting for {request_id}"
);
if let Some(env) = frames.next_frame() {
if env.id == request_id {
return env;
}
}
let bin = ws.next_binary(std::time::Duration::from_millis(500)).await;
match bin {
Some(bytes) => {
chunks.push(&bytes);
while let Some((channel_id, payload)) = chunks.next_chunk() {
assert_eq!(channel_id, 0, "response must ride channel 0");
frames.push(&payload);
}
}
None => panic!("ws closed unexpectedly while awaiting {request_id}"),
}
}
}
/// Unit 3 scenario 1+2 (review 006): the open op resolves with a
/// `channel_id`, the per-session openable is discoverable through
/// `services/list`, and data chunks flow both directions on the
/// channel — the `OpenHandler`'s `Connection` sees the bytes.
#[tokio::test]
async fn data_channel_open_discoverable_and_bytes_round_trip() {
let (url, _policy) = spawn_openable_ws_server(
echo_registry(),
alkcall::channels::policy::DEFAULT_CHANNEL_CAP,
)
.await;
let mut ws = WsClient::connect_authorized(&url, "tok-1").await.unwrap();
// Discovery first: the fork's bootstrap discovery lists the
// session's own openable (the F-06 shape).
let env = call_and_await(&mut ws, "req-list", "services/list", serde_json::json!({})).await;
assert_eq!(env.r#type, EVENT_RESPONDED, "got {}", env.r#type);
let names: Vec<String> = env.payload["output"]["operations"]
.as_array()
.expect("operations array")
.iter()
.filter_map(|o| o["name"].as_str().map(String::from))
.collect();
assert!(
names.contains(&"channels/echo/sub".to_string()),
"per-session openable discoverable: {names:?}"
);
assert!(
names.contains(&"channel/close".to_string()) && names.contains(&"op/register".to_string()),
"generic channel ops + op/register served: {names:?}"
);
// The open op resolves with a channel_id.
let env = call_and_await(
&mut ws,
"req-open",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
assert_eq!(env.r#type, EVENT_RESPONDED, "got {}", env.r#type);
let channel_id = env.payload["output"]["channel_id"]
.as_u64()
.expect("channel_id in open response");
assert!(channel_id > 0, "even id from the accept side: {channel_id}");
// Data-plane echo: send one chunk on the data channel and read the
// handler's echo back. Data-channel payloads are opaque (ADR-035:
// the handler owns its framing) — the echo handler echoes
// byte-for-byte per read.
let payload = b"hello-data-plane";
let framed = frame_data_channel(channel_id as u32, payload);
ws.send_binary(framed).await;
let echoed = read_data_channel_block(&mut ws, channel_id as u32).await;
assert_eq!(echoed, payload, "handler echoed the bytes");
ws.close().await;
}
/// Frame a raw data-channel chunk as a WS binary message: the 8-byte
/// chunk header (channel_id BE + length BE) + payload — the same wire
/// format channel 0 uses, the opaque data-plane unit (ADR-035).
fn frame_data_channel(channel_id: u32, block: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(8 + block.len());
out.extend_from_slice(&channel_id.to_be_bytes());
out.extend_from_slice(&(block.len() as u32).to_be_bytes());
out.extend_from_slice(block);
out
}
/// Read one echoed chunk's payload off a data channel, skipping
/// channel-0 traffic.
async fn read_data_channel_block(ws: &mut WsClient, channel_id: u32) -> Vec<u8> {
let mut chunks = ChunkAssembler::new();
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
assert!(
tokio::time::Instant::now() < deadline,
"timed out waiting for data-channel {channel_id}"
);
let bin = ws.next_binary(std::time::Duration::from_millis(500)).await;
match bin {
Some(bytes) => {
chunks.push(&bytes);
while let Some((cid, payload)) = chunks.next_chunk() {
if cid == channel_id {
return payload;
}
}
}
None => panic!("ws closed unexpectedly while awaiting data channel {channel_id}"),
}
}
}
/// Unit 3 scenario 2 (review 006): `channel/close` tears the handler
/// stream down and the opener ledger decrements (the policy count
/// drops) — plus the close op itself resolves `{"closed": true}`.
#[tokio::test]
async fn channel_close_resolves_and_decrements_the_opener_ledger() {
let (url, policy) = spawn_openable_ws_server(
echo_registry(),
alkcall::channels::policy::DEFAULT_CHANNEL_CAP,
)
.await;
let alice = identity("alice", &[]);
let mut ws = WsClient::connect_authorized(&url, "tok-1").await.unwrap();
let env = call_and_await(
&mut ws,
"req-open",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
let channel_id = env.payload["output"]["channel_id"].as_u64().unwrap() as u32;
assert_eq!(policy.count_for(&alice), 1, "open incremented the ledger");
let env = call_and_await(
&mut ws,
"req-close",
"channel/close",
serde_json::json!({ "channel_id": channel_id }),
)
.await;
assert_eq!(env.r#type, EVENT_RESPONDED, "got {}", env.r#type);
assert_eq!(
env.payload["output"]["closed"], true,
"close op resolves: {}",
env.payload["output"]
);
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
assert!(
tokio::time::Instant::now() < deadline,
"ledger decrement never landed"
);
if policy.count_for(&alice) == 0 {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
}
ws.close().await;
}
/// Unit 3 scenario 3: the per-identity cap denies the open over the
/// limit with the `channel:` error prefix (the policy is the
/// `ChannelsPolicy` extension; the open wrapper consults it before
/// allocation).
#[tokio::test]
async fn channel_cap_denies_open_over_the_limit() {
let (url, policy) = spawn_openable_ws_server(echo_registry(), 1).await;
let mut ws = WsClient::connect_authorized(&url, "tok-1").await.unwrap();
let env = call_and_await(
&mut ws,
"req-open-1",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
assert!(
env.payload["output"]["channel_id"].is_u64(),
"first open admitted"
);
let env = call_and_await(
&mut ws,
"req-open-2",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
assert_eq!(
env.r#type, EVENT_ERROR,
"cap denial: got {} payload {}",
env.r#type, env.payload
);
let error = &env.payload;
assert!(
error["code"]
.as_str()
.is_some_and(|c| c.starts_with("channel:")),
"channel-prefixed denial code: {error}"
);
assert_eq!(policy.count_for(&identity("alice", &[])), 1);
ws.close().await;
}
/// Unit 3 gate (the WS-24/WS-25 shape): `op/register` is served per
/// session and its collision policy binds the session fork. Announce
/// lands in the connection overlay (re-announce without `replace`
/// rejects with the overlay gate's `ALREADY_EXISTS`); announcing a name
/// the serving side itself registers is rejected with
/// `ALREADY_EXISTS` regardless of `replace` (review 005 G-03's gate,
/// through the WS path).
#[tokio::test]
async fn op_register_served_per_session_and_collision_is_already_exists() {
use alkcall::registry::op_register::OpRegisterRequest;
let addr = spawn_ws_server(
echo_registry(),
provider_with(vec![("tok-1", identity("alice", &[]))]),
)
.await;
let mut ws = WsClient::connect_authorized(&format!("{addr}/alk/channels"), "tok-1")
.await
.unwrap();
// A distinct name announces cleanly into the connection overlay.
let spec = OperationSpec::new(
"consumer/exec",
OperationType::Query,
Visibility::External,
serde_json::json!({}),
serde_json::json!({}),
vec![],
AccessControl::default(),
None,
);
let request = OpRegisterRequest {
spec,
replace: false,
};
let frame = EventEnvelope::requested(
"req-announce",
serde_json::json!({
"operationId": "op/register",
"input": request.to_json(),
}),
);
ws.send_binary(frame_channel0_chunk(&frame)).await;
let env = await_envelope(&mut ws, &frame.id).await;
assert_eq!(env.r#type, EVENT_RESPONDED, "got {}", env.r#type);
assert_eq!(env.payload["output"]["registered"], true);
// The announced op landed in the overlay: a second announce of the
// same name without `replace` hits the overlay collision gate.
let spec = OperationSpec::new(
"consumer/exec",
OperationType::Query,
Visibility::External,
serde_json::json!({}),
serde_json::json!({}),
vec![],
AccessControl::default(),
None,
);
let request = OpRegisterRequest {
spec,
replace: false,
};
let frame = EventEnvelope::requested(
"req-reannounce",
serde_json::json!({
"operationId": "op/register",
"input": request.to_json(),
}),
);
ws.send_binary(frame_channel0_chunk(&frame)).await;
let env = await_envelope(&mut ws, &frame.id).await;
assert_eq!(env.r#type, EVENT_ERROR, "got {}", env.r#type);
assert_eq!(
env.payload["code"], "ALREADY_EXISTS",
"overlay collision gate: {}",
env.payload
);
// Collision: announcing the serving side's own op name rejects with
// ALREADY_EXISTS regardless of replace (review 005 G-03's gate,
// through the WS path).
let spec = OperationSpec::new(
"echo/run",
OperationType::Query,
Visibility::External,
serde_json::json!({}),
serde_json::json!({}),
vec![],
AccessControl::default(),
None,
);
let request = OpRegisterRequest {
spec,
replace: true,
};
let frame = EventEnvelope::requested(
"req-collide",
serde_json::json!({
"operationId": "op/register",
"input": request.to_json(),
}),
);
ws.send_binary(frame_channel0_chunk(&frame)).await;
let env = await_envelope(&mut ws, &frame.id).await;
assert_eq!(env.r#type, EVENT_ERROR, "got {}", env.r#type);
assert_eq!(
env.payload["code"], "ALREADY_EXISTS",
"serving-registry collision rejected: {}",
env.payload
);
ws.close().await;
}
/// Unit 3 scenario 4: a mid-open disconnect tears the session down
/// without leaking — the ledger count for the opener drops to zero
/// (the channels adapter's teardown decrements), the WS close is
/// observed, and no further open succeeds (the socket is gone).
#[tokio::test]
async fn disconnect_mid_open_tears_down_and_decrements() {
let (url, policy) = spawn_openable_ws_server(
echo_registry(),
alkcall::channels::policy::DEFAULT_CHANNEL_CAP,
)
.await;
let alice = identity("alice", &[]);
let mut ws = WsClient::connect_authorized(&url, "tok-1").await.unwrap();
let env = call_and_await(
&mut ws,
"req-open",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
assert!(env.payload["output"]["channel_id"].is_u64());
assert_eq!(policy.count_for(&alice), 1);
// Hard disconnect (drop without close frame — the pump sees EOF).
drop(ws);
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5);
loop {
assert!(
tokio::time::Instant::now() < deadline,
"teardown never decremented the ledger"
);
if policy.count_for(&alice) == 0 {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
}
}
/// Unit 3 scenario 5: a `TooLarge` chunk on a data channel survives —
/// the demux skips it and resyncs (alkcall's hardening, asserted
/// through the WS path). The oversized chunk is skipped by the demux
/// (TooLarge), the following chunks on other channels still route.
#[tokio::test]
async fn too_large_data_channel_chunk_survives_demux_resync() {
use alkhttp::websocket::MAX_CHUNK_LEN;
let (url, _policy) = spawn_openable_ws_server(
echo_registry(),
alkcall::channels::policy::DEFAULT_CHANNEL_CAP,
)
.await;
let mut ws = WsClient::connect_authorized(&url, "tok-1").await.unwrap();
let env = call_and_await(
&mut ws,
"req-open",
"channels/echo/sub",
serde_json::json!({}),
)
.await;
let channel_id = env.payload["output"]["channel_id"].as_u64().unwrap() as u32;
// An oversized declared chunk on the data channel: the demux skips
// it (bounded by its 256 MiB budget) and stays in sync. The skip
// consumes exactly the declared payload length off the transport,
// so we must actually send that many filler bytes (16 MiB + 1 is
// over MAX_CHUNK_LEN but under the WS message caps, sent as
// separate WS messages).
let oversized_len = MAX_CHUNK_LEN + 1;
let mut header = Vec::with_capacity(8);
header.extend_from_slice(&channel_id.to_be_bytes());
header.extend_from_slice(&oversized_len.to_be_bytes());
ws.send_binary(header).await;
let filler_len = oversized_len as usize;
let filler = vec![0u8; filler_len];
let mut sent = 0usize;
while sent < filler_len {
let take = (filler_len - sent).min(256 * 1024);
ws.send_binary(filler[sent..sent + take].to_vec()).await;
sent += take;
}
// The next small chunk on the data channel still routes and the
// handler echoes it — resync proven.
let payload = b"after-skip";
ws.send_binary(frame_data_channel(channel_id, payload))
.await;
let echoed = read_data_channel_block(&mut ws, channel_id).await;
assert_eq!(echoed, payload, "demux resynced after the TooLarge skip");
ws.close().await;
}