docs(arch): hub-wiring cluster — aggregated env, peer_operations, from_call cleanup, alknet-hub crate spec

Three ADRs addressing gaps surfaced by the first hub consumer (alkapi): - ADR-067: Aggregated peer-env wiring — Dispatcher::with_aggregated_env hook so compose_root_env reads a shared PeerCompositeEnv across all calls, not a fresh per-call one. The hub-defining gap (alkapi OQ-08 / G.1). - ADR-068: PeerCompositeEnv::peer_operations override — adds list_operation_names() to OperationEnv, overrides on OverlayOperationEnv and PeerCompositeEnv. Fixes services/list-peers returning empty operation lists for non-local peers (alkapi G.6). - ADR-069: from_call is a manual free function, not auto-wired — reverses the aspirational OQ-27 resolution to match the implementation. Cleans the 'v1 default' hedging language in ADR-017 and client-and-adapters.md (alkapi G.4). New crate spec: alknet-hub — reusable hub pattern (aggregated env, connection lifecycle, worker supervision with backoff, service discovery). Spec fixes: builder API drift (with_local/with_local_streaming separation), ScopedOperationEnv → ScopedPeerEnv type name, OQ count 51→54. Three new OQs: OQ-52 (wait_for_close), OQ-53 (backoff defaults), OQ-54 (inbound hook placement).
2026-07-09 16:01:30 +00:00
1 parent 2282647f8b
commit 87b2c2a5ef
@@ -390,15 +390,19 @@ where `AdapterError` is a crate-level enum. The *presence* of the error type
 is recorded in [client-and-adapters.md](../crates/call/client-and-adapters.md);
 the exact variants are the two-way-door remainder, tracked as OQ-26.

-### DC-2 — from_call re-import on reconnection: default set
+### DC-2 — from_call re-import on reconnection: manual free function

 Assumption 4 noted re-import "happens on reconnection or is triggered
-explicitly." The v1 default is **auto-re-import on connection establishment**.
-The overlay is per-connection (Layer 2, ADR-024), so re-import is naturally
-scoped; a stale overlay dies with the connection. Explicit re-import via a
-future `CallConnection::refresh()` is additive. Two-way door; recorded in
+explicitly." The decision is **manual**: `from_call` is a free function; the
+assembly layer calls it after `connect()`. The overlay is per-connection
+(Layer 2, ADR-024), so re-import on reconnect is naturally scoped; a stale
+overlay dies with the connection. A `CallConnection::refresh()` method for
+mid-connection re-discovery is a genuine feature addition — non-breaking,
+additive — if a deployment needs manual re-discovery without
+drop-and-reconnect. Two-way door; recorded in
 [client-and-adapters.md](../crates/call/client-and-adapters.md); tracked as
-OQ-27.
+OQ-27. See [ADR-069](069-from-call-manual-free-function.md) for the full
+rationale.

 ### DC-3 — from_call namespace collision: default set

@@ -0,0 +1,208 @@
+# ADR-067: Aggregated Peer-Environment Wiring for Hub Deployments
+
+## Status
+
+Proposed
+
+## Context
+
+`Dispatcher::compose_root_env` (`protocol/dispatch.rs:134-154`) constructs a
+**fresh** `PeerCompositeEnv` per call and attaches **only the current call's
+own connection** as a peer overlay. It does not aggregate the hub's other live
+worker connections into that call's environment.
+
+Consequence: on a hub with N connected workers, a handler composing
+`env.invoke_peer(&PeerRef::Specific("dev1"), "docker", "container/exec",
+input, &ctx, policy)` from a call that arrived on an HTTP connection (or on a
+*different* worker's connection) will **not** find dev1's overlay — the routing
+falls through to the curated base and returns `NOT_FOUND`.
+
+The `PeerCompositeEnv` *type* and the `invoke_peer` routing logic are built for
+multi-peer aggregation (`attach_peer`/`detach_peer` with insertion-order
+preservation, `PeerRef::Specific`/`Any` routing — `registry/env.rs:155-301`).
+The per-call `compose_root_env` does not use that capability. The ADR-029
+*model* is committed; the implementation is incomplete for the head→N-workers
+case.
+
+This is the single highest-impact gap a first hub consumer (alkapi) surfaces.
+A hub is *defined* by composing ops across its connected workers. Without an
+aggregated env shared across all calls, the hub pattern does not work: a call
+arriving on one transport cannot reach a worker connected on another.
+
+The alkapi project identified this as OQ-08 and committed to the aggregation
+decision in their ADR-011. The decision to aggregate is made; the question is
+where the wiring lives — alknet (reusable by any hub) or a hub-side wrapper
+(alkapi-only). This ADR resolves that question: the wiring lives in alknet.
+
+## Decision
+
+### 1. `Dispatcher` gains a `with_aggregated_env` builder method
+
+A new optional field on `Dispatcher` holds a shared aggregated
+`PeerCompositeEnv`:
+
+```rust
+pub struct Dispatcher {
+    pub registry: Arc<OperationRegistry>,
+    pub identity_provider: Arc<dyn IdentityProvider>,
+    pub session_source: Option<Arc<dyn SessionOverlaySource + Send + Sync>>,
+    pub ownership_provider: Option<Arc<dyn OwnershipProvider>>,
+    pub aggregated_env: Option<Arc<std::sync::RwLock<PeerCompositeEnv>>>,
+    pub default_timeout: Duration,
+}
+
+impl Dispatcher {
+    pub fn with_aggregated_env(
+        mut self,
+        env: Arc<std::sync::RwLock<PeerCompositeEnv>>,
+    ) -> Self {
+        self.aggregated_env = Some(env);
+        self
+    }
+}
+```
+
+The builder method mirrors `with_session_source` and `with_ownership_provider`
+— an optional hook the assembly layer wires at construction time. A deployment
+that does not set an aggregated env gets today's `compose_root_env` behavior
+unchanged.
+
+### 2. `CallAdapter` gains a matching `with_aggregated_env` builder method
+
+`CallAdapter` delegates to `Dispatcher`:
+
+```rust
+impl CallAdapter {
+    pub fn with_aggregated_env(
+        mut self,
+        env: Arc<std::sync::RwLock<PeerCompositeEnv>>,
+    ) -> Self {
+        self.dispatcher = self.dispatcher.with_aggregated_env(env);
+        self
+    }
+}
+```
+
+### 3. `compose_root_env` reads the aggregated env when set
+
+When `aggregated_env` is `Some`, `compose_root_env` reads the shared env,
+attaches the current connection's overlay as an override for the current call
+only, and returns the result. When `None`, the existing per-call behavior is
+preserved:
+
+```rust
+pub fn compose_root_env(
+    &self,
+    connection: &CallConnection,
+    context: &OperationContext,
+) -> Arc<dyn OperationEnv + Send + Sync> {
+    let base: Arc<dyn OperationEnv + Send + Sync> =
+        Arc::new(LocalOperationEnv::new(Arc::clone(&self.registry)));
+    let session = self
+        .session_source
+        .as_ref()
+        .and_then(|s| s.overlay_for(context));
+
+    if let Some(aggregated) = &self.aggregated_env {
+        // Clone the shared aggregated env (cheap — all fields are Arc).
+        let mut env = aggregated.read().unwrap().clone();
+        // Attach the current connection's overlay as an override for this
+        // call only. The current connection's overlay is the authoritative
+        // view of *that* peer; the aggregated env is the authoritative view
+        // of *all other* peers. This avoids a race where the aggregated env
+        // has not yet picked up a new op the current peer just registered.
+        if let Some(peer_id) = connection.identity().map(|identity| identity.id.clone()) {
+            env.attach_peer(peer_id, connection.overlay_env());
+        }
+        Arc::new(env)
+    } else {
+        let mut env = PeerCompositeEnv::new(base);
+        if let Some(session) = session {
+            env = env.with_session(session);
+        }
+        if let Some(peer_id) = connection.identity().map(|identity| identity.id.clone()) {
+            env.attach_peer(peer_id, connection.overlay_env());
+        }
+        Arc::new(env)
+    }
+}
+```
+
+The clone of the aggregated env is cheap: `PeerCompositeEnv`'s fields are all
+`Arc` (the `HashMap` values are `Arc<dyn OperationEnv>`, the `Vec` is
+`Vec<PeerId>` which is a `String` clone). The `RwLock::read()` is held only
+for the clone, not for the duration of the call.
+
+### 4. The hub owns the aggregated env lifecycle
+
+The hub (assembly layer) constructs the aggregated env once at startup:
+
+```rust
+let base: Arc<dyn OperationEnv + Send + Sync> =
+    Arc::new(LocalOperationEnv::new(Arc::clone(&registry)));
+let aggregated = Arc::new(RwLock::new(PeerCompositeEnv::new(base)));
+
+let adapter = CallAdapter::new(registry, identity_provider)
+    .with_aggregated_env(Arc::clone(&aggregated));
+```
+
+The hub calls `aggregated.write().unwrap().attach_peer(peer_id, overlay)` on
+every worker connection-establish (after `from_call` populates the overlay)
+and `detach_peer(&peer_id)` on every disconnect. The `RwLock` write is held
+only for the `HashMap` insert/remove — connection-rate, not call-rate.
+
+### 5. `PeerCompositeEnv` gains `Clone`
+
+`PeerCompositeEnv` is made `Clone` (all fields are `Arc` or `Clone` already).
+This is a one-line derive addition.
+
+## Consequences
+
+**Positive:**
+- The hub pattern works. A call arriving on any transport can reach any
+  connected worker's ops via `PeerRef::Specific` or `PeerRef::Any`.
+- The existing single-connection behavior is preserved. A deployment that does
+  not set an aggregated env gets today's `compose_root_env` unchanged.
+- The hook is additive — a new optional field, a new builder method, a branch
+  in `compose_root_env`. No existing code path changes.
+- The capability is reusable by any future hub, not just alkapi. The alkapi
+  project's ADR-011 fallback (hub-side wrapper) is no longer needed.
+
+**Negative:**
+- A `RwLock<PeerCompositeEnv>` on the read hot path of every dispatch. The
+  lock is held only for a clone (all `Arc` fields — cheap). An `ArcSwap`
+  copy-on-write variant could avoid the lock on reads entirely, at the cost
+  of a clone on `attach_peer`/`detach_peer` (infrequent). The `RwLock` is the
+  simpler starting point; `ArcSwap` is an additive optimization.
+- `PeerCompositeEnv` gains `Clone`. The derive is mechanical; all fields are
+  already `Clone`.
+- The hub must manage the aggregated env lifecycle (`attach_peer`/`detach_peer`
+  on connection events). This is assembly-layer code, not alknet-call code.
+  The hooks exist; the hub wires them.
+
+## Assumptions
+
+1. **`PeerCompositeEnv` clone is cheap.** All fields are `Arc` or `Clone` of
+   small types (`String`, `Vec<String>`). The clone does not copy the
+   operation registries or the connection overlays — it copies `Arc` pointers.
+2. **The current connection's overlay is authoritative for that peer.** A call
+   arriving on worker-a uses worker-a's live overlay as the view of worker-a
+   (not the aggregated env's possibly-stale snapshot), and the aggregated env
+   for all other peers. This avoids a race where the aggregated env has not
+   yet picked up a new op worker-a just registered.
+3. **The `RwLock` is not a contention point.** Reads (clones) are call-rate
+   but the lock is held only for the clone duration (microseconds). Writes
+   (`attach_peer`/`detach_peer`) are connection-rate (seconds to minutes). If
+   profiling shows contention, `ArcSwap` is the additive optimization.
+
+## References
+
+- ADR-029: Peer-Graph Routing Model (the model this wiring completes)
+- ADR-030: PeerEntry and Identity.id Decoupling (the `PeerId` source)
+- ADR-068: PeerCompositeEnv::peer_operations Override (sibling hub-wiring decision)
+- ADR-069: from_call Is a Manual Free Function (sibling hub-wiring decision)
+- alkapi ADR-011: Aggregated Peer Environment (the downstream commitment)
+- alkapi OQ-08: alknet aggregated peer-env wiring (the blocking question)
+- `crates/alknet-call/src/protocol/dispatch.rs:134-154` — current
+  `compose_root_env`
+- `crates/alknet-call/src/registry/env.rs:155-301` — `PeerCompositeEnv`
@@ -0,0 +1,154 @@
+# ADR-068: PeerCompositeEnv::peer_operations Override
+
+## Status
+
+Proposed
+
+## Context
+
+`OperationEnv::peer_operations` (defined in `registry/env.rs:63-65`) has a
+default implementation returning `Vec::new()`. `PeerCompositeEnv` overrides
+`invoke_with_policy`, `contains`, `invoke_peer`, `peer_contains`, and
+`peer_ids` — but does **not** override `peer_operations`. This means
+`peer_operations` on a `PeerCompositeEnv` always returns an empty `Vec`.
+
+The `services/list-peers` handler (`registry/discovery.rs:245-296`) calls
+`ctx.env.peer_operations(&peer_id)` to discover what operations each peer
+serves. Since `PeerCompositeEnv` does not override this, non-local peers
+always show empty operation lists in the `list-peers` response. The
+`peer_ids()` method correctly returns the peer IDs, but the operations for
+each peer are always empty.
+
+This is a pure gap — the `services/list-peers` handler is specced to enumerate
+each peer's operations (ADR-029 §6), and the `PeerCompositeEnv` type has all
+the data needed to implement it (each peer's `OverlayOperationEnv` holds a
+`HashMap<String, HandlerRegistration>`). The override is one method collecting
+each peer overlay's registered op names.
+
+The alkapi project identified this as gap G.6: a hub consumer calling
+`services/list-peers` gets `peers: [{peer_id: "dev1", operations: []}]` until
+this is fixed.
+
+## Decision
+
+`PeerCompositeEnv` overrides `peer_operations` to collect the operation names
+from each peer's connection overlay:
+
+```rust
+fn peer_operations(&self, peer: &PeerId) -> Vec<String> {
+    match self.connections.get(peer) {
+        Some(overlay) => {
+            // The overlay is an OverlayOperationEnv wrapping a
+            // HashMap<String, HandlerRegistration>. We need the op names.
+            // Rather than adding a method to OperationEnv (which would
+            // require every impl to add it), we use the existing `contains`
+            // method — but that requires knowing the name to check.
+            //
+            // The correct approach: iterate the overlay's known names.
+            // OverlayOperationEnv already has the data (the HashMap keys).
+            // We add a `list_operation_names(&self) -> Vec<String>` method
+            // to OperationEnv with a default returning Vec::new(), and
+            // OverlayOperationEnv overrides it to return the keys.
+            overlay.list_operation_names()
+        }
+        None => Vec::new(),
+    }
+}
+```
+
+### 1. `OperationEnv` gains `list_operation_names` with a default impl
+
+```rust
+fn list_operation_names(&self) -> Vec<String> {
+    Vec::new()
+}
+```
+
+The default returns empty — existing impls (`LocalOperationEnv`, test-only
+envs) don't need to change. Only `OverlayOperationEnv` overrides it.
+
+### 2. `OverlayOperationEnv` overrides `list_operation_names`
+
+```rust
+impl OperationEnv for OverlayOperationEnv {
+    fn list_operation_names(&self) -> Vec<String> {
+        self.overlay.read().keys().cloned().collect()
+    }
+    // ... existing impl unchanged
+}
+```
+
+### 3. `PeerCompositeEnv::peer_operations` uses `list_operation_names`
+
+The override delegates to each peer's overlay:
+
+```rust
+fn peer_operations(&self, peer: &PeerId) -> Vec<String> {
+    self.connections
+        .get(peer)
+        .map(|overlay| overlay.list_operation_names())
+        .unwrap_or_default()
+}
+```
+
+### Why a new trait method instead of a different approach
+
+Alternatives considered:
+
+- **Add `fn operations(&self) -> Vec<String>` to `OperationEnv`**: Same
+  concept, different name. `list_operation_names` is chosen to match the
+  existing `list_operations` naming on `OperationRegistry`.
+- **Make `peer_operations` on `PeerCompositeEnv` reach into
+  `OverlayOperationEnv`'s internals**: Requires `OverlayOperationEnv` to
+  expose its `HashMap` or a method. The trait method is cleaner — it keeps
+  the abstraction boundary intact.
+- **Have `services/list-peers` iterate `ctx.env.peer_ids()` and call
+  `contains` for every known op name**: Requires knowing all possible op
+  names (from the registry), which is a cross-layer coupling. The trait
+  method keeps the data where it lives.
+
+The trait method is the smallest surface change: one new method with a
+default impl, one override on `OverlayOperationEnv`, one override on
+`PeerCompositeEnv`. No existing code changes.
+
+## Consequences
+
+**Positive:**
+- `services/list-peers` returns actual operation lists for each peer. A hub
+  consumer calling `services/list-peers` gets `peers: [{peer_id: "dev1",
+  operations: [{name: "docker/container/exec", ...}, ...]}]` — the specced
+  behavior.
+- The fix is small: one trait method, two overrides. No existing code paths
+  change.
+- The `list_operation_names` method is generally useful — any future code
+  that needs to enumerate an env's operations can use it.
+
+**Negative:**
+- `OperationEnv` gains a method. The default impl preserves back-compat for
+  all existing implementors. Only `OverlayOperationEnv` and
+  `PeerCompositeEnv` override it.
+- The `OverlayOperationEnv` override holds the `RwLock` read for the
+  duration of the `keys().cloned().collect()`. This is a `Vec<String>`
+  allocation — cheap for typical peer operation counts (tens, not thousands).
+
+## Assumptions
+
+1. **`OverlayOperationEnv`'s `RwLock<HashMap<String, HandlerRegistration>>`
+   read is cheap.** The lock is held only for the `keys()` iteration and
+   `collect()`. Typical peer operation counts are small (tens of ops).
+2. **`list_operation_names` is the right name.** It matches the existing
+   `list_operations` naming on `OperationRegistry` and avoids confusion with
+   `peer_operations` (which takes a `PeerId` parameter).
+
+## References
+
+- ADR-029 §6: `services/list-peers` opt-in peer-attributed re-export listing
+- ADR-067: Aggregated Peer-Environment Wiring (sibling hub-wiring decision)
+- ADR-069: from_call Is a Manual Free Function (sibling hub-wiring decision)
+- `crates/alknet-call/src/registry/env.rs:63-65` — default `peer_operations`
+- `crates/alknet-call/src/registry/env.rs:155-301` — `PeerCompositeEnv`
+- `crates/alknet-call/src/protocol/connection.rs:305-397` —
+  `OverlayOperationEnv`
+- `crates/alknet-call/src/registry/discovery.rs:245-296` —
+  `services_list_peers_handler`
+- alkapi gap G.6: `PeerCompositeEnv::peer_operations` unimplemented
@@ -0,0 +1,122 @@
+# ADR-069: from_call Is a Manual Free Function, Not Auto-Wired
+
+## Status
+
+Proposed
+
+## Context
+
+OQ-27 resolved (2026-06-27): "The decision is **auto-re-import on connection
+establishment**. The overlay is per-connection (Layer 2, ADR-024), so a stale
+overlay dies with the connection; re-import on reconnect is naturally scoped to
+the new connection."
+
+The spec in `client-and-adapters.md` §"from_call" (line 358) states: "This is
+the v1 default; explicit re-import via a future `CallConnection::refresh()` is
+additive."
+
+The implementation does not match. `from_call` is a standalone free function
+(`client/from_call.rs:80`). `CallClient::connect()` does not call it. The
+assembly layer must call `from_call()` + `register_imported_all()` explicitly
+after every `connect()`. There is no `CallConnection::refresh()`.
+
+The "v1 default" language is hedging — it makes a committed-but-not-implemented
+feature sound like a deliberate phase. The spec says "auto-re-import on
+connection establishment" but the code says "the assembly layer calls
+`from_call` immediately after `connect()`" (the doc comment on `from_call`,
+line 76). These are different things: auto-wiring means `connect()` calls
+`from_call()` internally; manual means the caller does it.
+
+The alkapi project identified this as gap G.4: the hedging language in the
+spec, and the question of whether `from_call` should be auto-wired into
+`connect()`.
+
+## Decision
+
+**`from_call` is a manual free function. The assembly layer calls it after
+`connect()`. It is not auto-wired into `CallClient::connect()`.**
+
+### Why manual is correct
+
+1. **The hub controls discovery timing.** A hub may want to verify the
+   connection, resolve the peer's identity, check authorization, and *then*
+   discover operations. Auto-wiring `from_call` into `connect()` would run
+   discovery before the assembly layer has a chance to inspect the connection.
+
+2. **Discovery is not always wanted.** A pure-client connection to a public
+   X.509 endpoint (ADR-034) has no `PeerEntry` and no `PeerId` — the remote
+   is not in the peer graph. Auto-discovering ops on such a connection would
+   register them in a connection overlay that has no peer key, making them
+   unreachable via `PeerRef`. The assembly layer decides whether to run
+   `from_call` based on whether the remote is a known peer.
+
+3. **The `from_call` function is already the right API.** It takes a
+   `&CallConnection` and a `FromCallConfig`, returns
+   `Result<Vec<HandlerRegistration>, AdapterError>`, and the caller registers
+   the bundles. This is a clean separation: connect, discover, register. Each
+   step is independently testable and independently controllable.
+
+4. **Auto-wiring would require `from_call` to know about the registry.**
+   `CallClient` holds an `Arc<OperationRegistry>`, but `from_call` produces
+   `HandlerRegistration` bundles that the caller registers — the caller
+   decides *where* to register them (the connection's overlay, a session
+   overlay, or not at all). Auto-wiring would hardcode the registration
+   target.
+
+### What changes in the spec
+
+The "v1 default" language in `client-and-adapters.md` and ADR-017 is replaced
+with an honest statement: `from_call` is a free function; the assembly layer
+calls it after `connect()`; there is no `CallConnection::refresh()` for
+mid-connection re-discovery. A `CallConnection::refresh()` method is a
+genuine feature addition — non-breaking, additive — if a deployment needs
+manual re-discovery without drop-and-reconnect.
+
+OQ-27 is updated: the resolution changes from "auto-re-import on connection
+establishment" to "manual — the assembly layer calls `from_call` after
+`connect()`." The door type remains two-way (auto-wiring is additive).
+
+### What does NOT change
+
+- The `from_call` function signature, behavior, and tests are unchanged.
+- `CallClient::connect()` is unchanged.
+- The re-import-on-reconnect pattern is unchanged: the assembly layer's
+  supervision loop calls `from_call` after each `connect()`. The overlay is
+  per-connection, so a stale overlay dies with the connection; re-import on
+  reconnect is naturally scoped. This is the correct behavior — it just
+  isn't automatic.
+
+## Consequences
+
+**Positive:**
+- The spec matches the implementation. No hedging language.
+- The assembly layer has full control over discovery timing and registration
+  target.
+- The separation of concerns (connect / discover / register) is clean and
+  testable.
+- No code changes needed — this is a spec correction, not an implementation
+  change.
+
+**Negative:**
+- The assembly layer must remember to call `from_call` after `connect()`.
+  This is a documentation concern, not a correctness concern — forgetting to
+  call `from_call` means the peer's ops are not imported, which is immediately
+  visible (calls to those ops return `NOT_FOUND`).
+- The "auto-re-import on connection establishment" resolution of OQ-27 was
+  aspirational and is now corrected. The resolution was written before the
+  implementation existed; the implementation made the right call (manual),
+  and the spec is catching up.
+
+## References
+
+- ADR-017 §3: `from_call` adapter specification
+- ADR-017 Amendments (DC-2): the amended `from_call` re-import resolution
+  (manual free function)
+- ADR-067: Aggregated Peer-Environment Wiring (sibling hub-wiring decision)
+- ADR-068: PeerCompositeEnv::peer_operations Override (sibling hub-wiring decision)
+- OQ-27: from_call re-import trigger (amended 2026-07-09)
+- `client-and-adapters.md` §"from_call" (updated)
+- `crates/alknet-call/src/client/from_call.rs:80` — `from_call` free function
+- `crates/alknet-call/src/client/call_client.rs:142-168` — `connect()` does
+  not call `from_call`
+- alkapi gap G.4: `from_call` wiring + "v1 default" hedging cleanup