alkdev/alkcall
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(review 005 Unit 1): concurrent serving loops + stub-exercising gates (G-01, G-02)

Browse Source 
- Split dispatch() into dispatch_start() (sync prefix) + spawned
  invocation: both single-stream loops (serve_single_stream and the
  accept-side run_loop_single_stream) spawn Once invocations, Sub
  pumps, and sink response writers; only the Pub sink start stays
  inline (chunk_tx must register before the next call.published).
  Inline dispatch deadlocked same-connection nested composition: the
  read loop awaited the parent handler, which awaited a nested call
  whose response only the same read loop could resolve (resolved only
  via the 30s sweeper). Spawned handles tracked + aborted at loop exit;
  in_flight_sinks behind an Arc<parking_lot::Mutex> with guards dropped
  before awaits.
- run_loop_single_stream gains the pending-resolution arms
  (RESPONDED/COMPLETED/ERROR): the accept side previously served only
  and had no loop resolving its own outbound pendings in single-stream
  mode — the latent accept-side imported-op composition hazard is
  mechanized shut.
- Write-failure in the spawned Once path warns instead of closing the
  loop (matches the Sink arm; dying transport still surfaces via
  ConnectionClosed on the next read).
- G-02 gate: hub_handler_composes_peer_announced_op_via_nested_composition
  — announce -> consumer calls hub/compose -> hub's serving loop
  wire-dispatches it -> handler composes via ctx.env -> forwarding
  stub's nested call crosses back to the consumer. The F-05 gate
  bypassed this path entirely.
- Interleaved-directions gate: outbound_call_resolves_while_inbound_
  subscription_is_being_served — consumer serves a live Sub while a
  wire-dispatched hub handler issues an outbound call on the same
  connection.
- Both gates verified load-bearing: run against the pre-fix loop each
  reproduces the G-01 hang (no progress, bounded-timeout failure);
  post-fix both resolve in <0.2s, no sweeper evictions.

Verification: cargo test 583 / --all-features 600, clippy
(all-targets, all-features, wasm32) clean, fmt clean, doc clean.

Refs docs/reviews/005-...md (G-01, G-02; Units 2-3 open).
This commit is contained in:
glm-5.3-flash committed 2026-09-04 09:36:08 +00:00
1 parent 435ae9da2f
commit 1cbb7c6536
3 files changed
+846 -157
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/protocol/dispatch.rs
+265 -154
View File
@@ -114,6 +114,18 @@ impl std::fmt::Debug for DispatchResult {
}
}
/// The started half of a dispatched `call.requested` event, from
/// [`Dispatcher::dispatch_start`]. `Once`/`Stream` are returned ready
/// for the caller to spawn (review 005 G-01); `Sink` was started
/// inline (its `chunk_tx` must be registered before the next
/// `call.published` frame is read) and carries the started handler
/// future.
pub enum StartedDispatch {
Once(Pin<Box<dyn Future<Output = ResponseEnvelope> + Send>>),
Stream(ResponseStream),
Sink(SinkDispatch),
}
/// Shared dispatcher for an established `CallConnection`. Constructed by
/// both `CallAdapter` (accept path) and `CallClient` (connect path) and used
/// to run the dispatch loop. Holds no per-connection state; the
@@ -299,6 +311,39 @@ impl Dispatcher {
request_id: String,
payload: Value,
) -> DispatchResult {
match self.dispatch_start(connection, request_id, payload) {
StartedDispatch::Once(invoke) => DispatchResult::Once(invoke.await),
StartedDispatch::Stream(stream) => DispatchResult::Stream(stream),
StartedDispatch::Sink(sink) => DispatchResult::Sink(sink),
}
}
/// Start dispatching a `call.requested` event without awaiting the
/// invocation (review 005 G-01). The synchronous prefix of
/// [`dispatch`](Self::dispatch) — identity resolution, root-context
/// construction, op-type branch — is split from the invocation:
///
/// - `Query`/`Mutation` return the invocation as a boxed future
/// (`StartedDispatch::Once`) for the caller to spawn. Awaiting it
/// inline is what deadlocked the serving loops: a handler that
/// nested-composes over the same connection blocks the one read
/// loop that must resolve the nested call's response frame.
/// - `Sub` returns the [`ResponseStream`] synchronously
/// (`invoke_streaming` is a sync call) for the caller to pump in
/// a spawned task — an inline pump starves the same loop for the
/// stream's whole lifetime.
/// - `Pub` runs the full sink start inline (`StartedDispatch::Sink`):
/// ACL failure resolves to a ready `Once` future (the response
/// envelope is already built), success carries the `chunk_tx` the
/// loop must register in `in_flight_sinks` *before* the next
/// `call.published` frame can be read — the insert cannot race
/// the feed, so the sink start stays synchronous by design.
pub(crate) fn dispatch_start(
&self,
connection: &Arc<CallConnection>,
request_id: String,
payload: Value,
) -> StartedDispatch {
let operation_id = payload
.get("operationId")
.and_then(|v| v.as_str())
@@ -320,7 +365,7 @@ impl Dispatcher {
.unwrap_or(OperationType::Query);
let mut context = self.build_root_context(
request_id.clone(),
request_id,
&operation_name,
identity,
forwarded_for,
@@ -329,40 +374,125 @@ impl Dispatcher {
match op_type {
OperationType::Query | OperationType::Mutation => {
let envelope = self.registry.invoke(&operation_name, input, context).await;
DispatchResult::Once(envelope)
let registry = Arc::clone(&self.registry);
let name = operation_name;
StartedDispatch::Once(Box::pin(async move {
registry.invoke(&name, input, context).await
}))
}
OperationType::Sub => {
context.deadline = None;
let stream = self
.registry
.invoke_streaming(&operation_name, input, context);
DispatchResult::Stream(stream)
StartedDispatch::Stream(stream)
}
OperationType::Pub => {
context.deadline = None;
let sink_handler =
match self
.registry
.resolve_sink_handler(&operation_name, &input, &context)
{
Ok(h) => h,
Err(envelope) => return DispatchResult::Once(envelope),
};
let publish_validator = self.registry.publish_validator(&operation_name);
let (chunk_tx, chunk_rx) =
mpsc::channel::<Result<Value, CallError>>(PUBLISH_CHANNEL_BUFFER);
let publish_stream: PublishStream = Box::pin(chunk_rx);
let handler = (sink_handler)(input, context, publish_stream);
DispatchResult::Sink(SinkDispatch {
handler: Box::pin(handler),
chunk_tx,
publish_validator,
})
match self
.registry
.resolve_sink_handler(&operation_name, &input, &context)
{
Ok(sink_handler) => {
let publish_validator = self.registry.publish_validator(&operation_name);
let (chunk_tx, chunk_rx) =
mpsc::channel::<Result<Value, CallError>>(PUBLISH_CHANNEL_BUFFER);
let publish_stream: PublishStream = Box::pin(chunk_rx);
let handler = (sink_handler)(input, context, publish_stream);
StartedDispatch::Sink(SinkDispatch {
handler: Box::pin(handler),
chunk_tx,
publish_validator,
})
}
Err(envelope) => StartedDispatch::Once(Box::pin(std::future::ready(envelope))),
}
}
}
}
/// Spawn the Once invocation to write its single response frame on
/// completion. The write failure is a warning, not a loop exit: the
/// loop keeps reading (a dying transport surfaces on the next read
/// as `ConnectionClosed`), matching the Sink arm's behavior.
fn spawn_once_dispatch(
writer: &Arc<super::connection::SharedFrameWriter>,
request_id: String,
invoke: Pin<Box<dyn Future<Output = ResponseEnvelope> + Send>>,
) -> JoinHandle<()> {
let writer = Arc::clone(writer);
tokio::spawn(async move {
let response = invoke.await;
let event: EventEnvelope = response.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(
error = %err,
request_id = %request_id,
"serving loop: failed to write Once response frame"
);
}
})
}
/// Spawn a subscription's [`ResponseStream`] pump: each
/// [`ResponseEnvelope`] becomes a `call.responded` / `call.error`
/// frame; on natural end, a `call.completed` frame. The shared
/// writer serializes frames so the pump's frames do not interleave
/// with concurrent calls' frames.
fn spawn_stream_pump(
writer: &Arc<super::connection::SharedFrameWriter>,
request_id: String,
mut stream: ResponseStream,
) -> JoinHandle<()> {
let writer = Arc::clone(writer);
tokio::spawn(async move {
let mut last_was_error = false;
while let Some(envelope) = stream.next().await {
last_was_error = envelope.result.is_err();
let event: EventEnvelope = envelope.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(
error = %err,
request_id = %request_id,
"serving loop: failed to write streaming frame"
);
return;
}
}
if !last_was_error {
let completed = EventEnvelope::completed(&request_id);
if let Err(err) = writer.write_frame(&completed).await {
warn!(
error = %err,
request_id = %request_id,
"serving loop: failed to write call.completed"
);
}
}
})
}
/// Spawn a Pub's handler task: awaits the handler future and writes
/// the single `call.responded` / `call.error` frame.
fn spawn_sink_response_writer(
writer: &Arc<super::connection::SharedFrameWriter>,
request_id: String,
handler: Pin<Box<dyn Future<Output = ResponseEnvelope> + Send>>,
) -> JoinHandle<()> {
let writer = Arc::clone(writer);
tokio::spawn(async move {
let response = handler.await;
let event: EventEnvelope = response.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(
error = %err,
request_id = %request_id,
"serving loop: failed to write sink response frame"
);
}
})
}
pub async fn handle_abort(&self, connection: &Arc<CallConnection>, request_id: &str) {
if let Some(tx) = self.in_flight_sink_aborts.lock().remove(request_id) {
let _ = tx.send(());
@@ -716,14 +846,24 @@ impl Dispatcher {
/// `call.published` / `call.completed` / `call.aborted` frames
/// arrive *after* the `call.requested` and are routed to the
/// matching sink's `chunk_tx` while new `call.requested` frames for
/// other requests continue to be dispatched. Query/Mutation and Sub
/// responses are written through `writer` immediately after
/// dispatch.
/// other requests continue to be dispatched. Once invocations and
/// Sub pumps are spawned (review 005 G-01 — inline dispatch
/// deadlocked same-connection nested composition); only the sink
/// start runs inline.
///
/// This is the accept side's serving loop, so the loop also carries
/// the pending-resolution arms the connect side's
/// `serve_single_stream` composes: a nested-composing handler
/// (e.g. a `from_call` imported-op stub riding this connection)
/// blocks its own task, not the read loop, and the loop resolves
/// the stub's response frames here. Without these arms the accept
/// side could dispatch inbound requests but never resolve its own
/// outbound pendings in single-stream mode.
///
/// Returns when the read half closes (transport EOF). Outstanding
/// pending requests are failed with `connection closed`, and
/// in-flight sinks' `chunk_tx` are dropped (the handler's
/// `PublishStream` sees EOF).
/// pending requests are failed with `connection closed`, in-flight
/// sinks' `chunk_tx` are dropped (the handler's `PublishStream`
/// sees EOF), and spawned Once/Stream/Sink tasks are aborted.
pub async fn run_loop_single_stream(
self,
connection: Arc<CallConnection>,
@@ -749,7 +889,10 @@ impl Dispatcher {
});
let mut reader = FrameFramedReader::new(reader);
let mut in_flight_sinks: HashMap<String, InFlightSink> = HashMap::new();
let in_flight_sinks: Arc<ParkingLotMutex<HashMap<String, InFlightSink>>> =
Arc::new(ParkingLotMutex::new(HashMap::new()));
let spawned: Arc<ParkingLotMutex<Vec<JoinHandle<()>>>> =
Arc::new(ParkingLotMutex::new(Vec::new()));
loop {
let envelope = match reader.read_frame().await {
@@ -764,46 +907,29 @@ impl Dispatcher {
match envelope.r#type.as_str() {
EVENT_REQUESTED => {
let request_id = envelope.id.clone();
let payload = envelope.payload.clone();
let dispatch_result = self
.dispatch(&connection, request_id.clone(), payload)
.await;
match dispatch_result {
DispatchResult::Once(response) => {
let event: EventEnvelope = response.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(
error = %err,
"single-stream: failed to write Once response; closing loop"
);
break;
}
match self.dispatch_start(&connection, request_id.clone(), envelope.payload) {
StartedDispatch::Once(invoke) => {
let handle =
Self::spawn_once_dispatch(&writer, request_id.clone(), invoke);
spawned.lock().push(handle);
}
DispatchResult::Stream(stream) => {
self.pump_stream_single_stream(&writer, &request_id, stream)
.await;
StartedDispatch::Stream(stream) => {
let handle =
Self::spawn_stream_pump(&writer, request_id.clone(), stream);
spawned.lock().push(handle);
}
DispatchResult::Sink(sink) => {
StartedDispatch::Sink(sink) => {
let SinkDispatch {
handler,
chunk_tx,
publish_validator,
} = sink;
let writer_clone = Arc::clone(&writer);
let request_id_for_handler = request_id.clone();
let handle = tokio::spawn(async move {
let response = handler.await;
let event: EventEnvelope = response.into();
if let Err(err) = writer_clone.write_frame(&event).await {
warn!(
error = %err,
request_id = %request_id_for_handler,
"single-stream: failed to write sink response frame"
);
}
});
in_flight_sinks.insert(
let handle = Self::spawn_sink_response_writer(
&writer,
request_id.clone(),
handler,
);
in_flight_sinks.lock().insert(
request_id.clone(),
InFlightSink {
chunk_tx,
@@ -814,9 +940,25 @@ impl Dispatcher {
}
}
}
EVENT_RESPONDED => {
let request_id = envelope.id.clone();
let output = envelope
.payload
.get("output")
.cloned()
.unwrap_or(Value::Null);
pending.lock().handle_responded(&request_id, output);
}
EVENT_COMPLETED => {
let request_id = envelope.id.clone();
if in_flight_sinks.lock().remove(&request_id).is_none() {
pending.lock().handle_completed(&request_id);
}
}
EVENT_ABORTED => {
let request_id = envelope.id.clone();
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
entry.handler_handle.abort();
let _ = entry
.chunk_tx
@@ -833,7 +975,8 @@ impl Dispatcher {
.get("input")
.cloned()
.unwrap_or(Value::Null);
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
let validated = match &entry.publish_validator {
Some(validator) => {
if validator.is_valid(&chunk) {
@@ -851,7 +994,7 @@ impl Dispatcher {
let keep = validated.is_ok();
let _ = entry.chunk_tx.send(validated).await;
if keep {
in_flight_sinks.insert(request_id, entry);
in_flight_sinks.lock().insert(request_id, entry);
}
} else {
debug!(
@@ -860,36 +1003,33 @@ impl Dispatcher {
);
}
}
EVENT_COMPLETED => {
let request_id = envelope.id.clone();
in_flight_sinks.remove(&request_id);
}
EVENT_ERROR => {
let request_id = envelope.id.clone();
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let call_error: CallError = serde_json::from_value(envelope.payload)
.unwrap_or_else(|_| {
CallError::internal("publish error from initiator (malformed)")
});
let call_error: CallError = serde_json::from_value(envelope.payload)
.unwrap_or_else(|_| {
CallError::internal("publish error from initiator (malformed)")
});
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
let _ = entry.chunk_tx.send(Err(call_error)).await;
} else {
debug!(
request_id = %request_id,
"single-stream: call.error for unknown in-flight sink; dropping"
);
pending.lock().handle_error(&request_id, call_error);
}
}
other => {
debug!(
event_type = %other,
id = %envelope.id,
"single-stream: ignoring non-requested/non-published/non-aborted/non-completed event"
"single-stream: ignoring unknown event type"
);
}
}
}
in_flight_sinks.clear();
in_flight_sinks.lock().clear();
for handle in spawned.lock().drain(..) {
handle.abort();
}
let failed = pending
.lock()
@@ -904,35 +1044,6 @@ impl Dispatcher {
sweeper_handle.abort();
}
/// Pump a subscription's `ResponseStream` to the wire through the
/// shared single-stream writer (ADR-036 amendment). Each
/// `ResponseEnvelope` becomes a `call.responded` / `call.error`
/// frame; on natural end, a `call.completed` frame. The shared
/// writer serializes frames so this pump's frames do not interleave
/// with concurrent calls' frames.
async fn pump_stream_single_stream(
&self,
writer: &Arc<super::connection::SharedFrameWriter>,
request_id: &str,
mut stream: ResponseStream,
) {
let mut last_was_error = false;
while let Some(envelope) = stream.next().await {
last_was_error = envelope.result.is_err();
let event: EventEnvelope = envelope.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(error = %err, "single-stream: failed to write streaming frame");
return;
}
}
if !last_was_error {
let completed = EventEnvelope::completed(request_id);
if let Err(err) = writer.write_frame(&completed).await {
warn!(error = %err, "single-stream: failed to write call.completed");
}
}
}
/// The full-duplex single-stream serving loop (review 004 F-04):
/// composes the dispatch arms (`run_loop_single_stream`) and the
/// pending-resolution arms (`read_single_stream_until_closed`) in
@@ -941,9 +1052,16 @@ impl Dispatcher {
/// the two directions' frames are multiplexed on one byte stream,
/// so the loop branches per frame:
///
/// - `call.requested` → dispatch against this loop's registry and
/// write the response frame(s) (the serving half — what the
/// accept side's `run_loop_single_stream` does).
/// - `call.requested` → start the dispatch and spawn the work (the
/// serving half — what the accept side's `run_loop_single_stream`
/// does). Once invocations and Sub pumps run as spawned tasks;
/// only the Pub sink start runs inline (its `chunk_tx` must be
/// registered before the next `call.published` frame is read).
/// Inline dispatch was the review 005 G-01 defect: a handler that
/// nested-composes over the same connection blocked the one read
/// loop that must resolve the nested call's response frame. The
/// `SharedFrameWriter` serializes frames, so spawned arms' frames
/// do not interleave mid-frame.
/// - `call.responded` / `call.completed` / `call.error` → resolve
/// the matching outbound pending entry (the calling half — what
/// the client read pump does). `call.responded` frames for
@@ -966,8 +1084,9 @@ impl Dispatcher {
/// cross-correlation between the two directions is not a hazard.
///
/// On read-half close, outbound pendings are failed (`connection
/// closed`) and in-flight inbound sinks are dropped — the same
/// teardown both arms perform separately today.
/// closed`), in-flight inbound sinks are dropped, and the spawned
/// Once/Stream/Sink tasks are aborted (their response frames cannot
/// reach a closed transport anyway).
pub async fn serve_single_stream(
self,
connection: Arc<CallConnection>,
@@ -993,7 +1112,10 @@ impl Dispatcher {
});
let mut reader = FrameFramedReader::new(reader);
let mut in_flight_sinks: HashMap<String, InFlightSink> = HashMap::new();
let in_flight_sinks: Arc<ParkingLotMutex<HashMap<String, InFlightSink>>> =
Arc::new(ParkingLotMutex::new(HashMap::new()));
let spawned: Arc<ParkingLotMutex<Vec<JoinHandle<()>>>> =
Arc::new(ParkingLotMutex::new(Vec::new()));
loop {
let envelope = match reader.read_frame().await {
@@ -1008,46 +1130,29 @@ impl Dispatcher {
match envelope.r#type.as_str() {
EVENT_REQUESTED => {
let request_id = envelope.id.clone();
let payload = envelope.payload.clone();
let dispatch_result = self
.dispatch(&connection, request_id.clone(), payload)
.await;
match dispatch_result {
DispatchResult::Once(response) => {
let event: EventEnvelope = response.into();
if let Err(err) = writer.write_frame(&event).await {
warn!(
error = %err,
"serve loop: failed to write Once response; closing loop"
);
break;
}
match self.dispatch_start(&connection, request_id.clone(), envelope.payload) {
StartedDispatch::Once(invoke) => {
let handle =
Self::spawn_once_dispatch(&writer, request_id.clone(), invoke);
spawned.lock().push(handle);
}
DispatchResult::Stream(stream) => {
self.pump_stream_single_stream(&writer, &request_id, stream)
.await;
StartedDispatch::Stream(stream) => {
let handle =
Self::spawn_stream_pump(&writer, request_id.clone(), stream);
spawned.lock().push(handle);
}
DispatchResult::Sink(sink) => {
StartedDispatch::Sink(sink) => {
let SinkDispatch {
handler,
chunk_tx,
publish_validator,
} = sink;
let writer_clone = Arc::clone(&writer);
let request_id_for_handler = request_id.clone();
let handle = tokio::spawn(async move {
let response = handler.await;
let event: EventEnvelope = response.into();
if let Err(err) = writer_clone.write_frame(&event).await {
warn!(
error = %err,
request_id = %request_id_for_handler,
"serve loop: failed to write sink response frame"
);
}
});
in_flight_sinks.insert(
let handle = Self::spawn_sink_response_writer(
&writer,
request_id.clone(),
handler,
);
in_flight_sinks.lock().insert(
request_id.clone(),
InFlightSink {
chunk_tx,
@@ -1069,13 +1174,14 @@ impl Dispatcher {
}
EVENT_COMPLETED => {
let request_id = envelope.id.clone();
if in_flight_sinks.remove(&request_id).is_none() {
if in_flight_sinks.lock().remove(&request_id).is_none() {
pending.lock().handle_completed(&request_id);
}
}
EVENT_ABORTED => {
let request_id = envelope.id.clone();
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
entry.handler_handle.abort();
let _ = entry
.chunk_tx
@@ -1092,7 +1198,8 @@ impl Dispatcher {
.get("input")
.cloned()
.unwrap_or(Value::Null);
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
let validated = match &entry.publish_validator {
Some(validator) => {
if validator.is_valid(&chunk) {
@@ -1110,7 +1217,7 @@ impl Dispatcher {
let keep = validated.is_ok();
let _ = entry.chunk_tx.send(validated).await;
if keep {
in_flight_sinks.insert(request_id, entry);
in_flight_sinks.lock().insert(request_id, entry);
}
} else {
debug!(
@@ -1125,7 +1232,8 @@ impl Dispatcher {
.unwrap_or_else(|_| {
CallError::internal("publish error from initiator (malformed)")
});
if let Some(mut entry) = in_flight_sinks.remove(&request_id) {
let entry = in_flight_sinks.lock().remove(&request_id);
if let Some(mut entry) = entry {
let _ = entry.chunk_tx.send(Err(call_error)).await;
} else {
pending.lock().handle_error(&request_id, call_error);
@@ -1141,7 +1249,10 @@ impl Dispatcher {
}
}
in_flight_sinks.clear();
in_flight_sinks.lock().clear();
for handle in spawned.lock().drain(..) {
handle.abort();
}
let failed = pending
.lock()