alkdev/alknet
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(arch): rewrite hub README for multi-transport + channels substrate; add OQ-58 (worker registration)

Browse Source 
The hub README predates channels (2026-07-09) and was built on the
call-protocol-directly-over-QUIC model: 'One QUIC connection per peer
carrying the call protocol,' CallClient::connect as the dial path,
QUIC as the only transport. The channels crate replaced that substrate
(ADR-071/079/080), and the hub's primary use case (worker provisioning
on docker/vast.ai/runpod) requires TCP+TLS for registration and QUIC
or TCP+TLS for the ongoing session — coexisting on one hub, not as
alternatives.

Rewrite:
- Multi-transport stated as decided: 'A hub MUST support TCP+TLS and
  QUIC endpoints simultaneously.' TCP+TLS accept loop (ADR-010 Am. 1)
  lives in the hub, shares the HandlerRegistry with the quinn endpoint.
- Channels substrate: the hub uses ChannelsAdapter/ChannelClient
  (from_connection primary, connect_quic convenience — ADR-080),
  not CallClient/CallAdapter directly. One channels connection per
  peer; CallAdapter runs on channel 0 (ADR-072).
- Transport-agnostic dial/accept: dial_worker_connection takes a
  Connection (one-way door); connect_quic_worker is a two-way-door
  convenience. supervise_worker takes a dial closure, not a SocketAddr.
- Identity over transports: fingerprint path (QUIC+raw-key,
  X.509 client cert) via resolve_from_fingerprint; bearer-token path
  (TCP+TLS no client cert, WebTransport, WebSocket) via
  resolve_from_token on the call first frame. Both resolve to the
  same PeerEntry (ADR-030/034).
- Worker registration flow (6 steps): provision → token → download →
  key gen → HTTP POST over TCP+TLS → channels connect. Step 4 is HTTP
  on HttpAdapter; step 6 is channels over QUIC or TCP+TLS. The hub
  creates a mixed-fingerprint PeerEntry (ADR-034 §3) at registration.
- OQ-58: worker registration flow — enrollment-token model,
  endpoint shape, register_worker API. Open (decision-ready, not
  blocked), one-way door, high priority.
- ADR-081: fix stale references to channels-hub/channels-worker
  sub-crates (the ADR's Decision says they don't exist; the References
  section said they did).

Review (architecture-reviewer): 3 critical (ADR-081 stale refs;
bearer-token extraction flow; assembly example QUIC-only +
unspecced into_connection), 7 warnings (registration 'or'→'both',
channel/control translation, OQ-52 interim, HubError variant naming,
RegistrationError definition, X.509-client-cert prose, adapter
ownership phrasing), 5 suggestions. All criticals and warnings #4/#7/#8
addressed; #5/#6/#9/#10 addressed; #11-15 noted as optional.
This commit is contained in:
glm-5.2 committed 2026-07-13 09:22:32 +00:00
1 parent 73c621bf21
commit a26401aadd
5 files changed
+568 -222
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/README.md
+1 -1
View File
@@ -171,7 +171,7 @@ adapter location map is now consistent: all HTTP-backed adapters
| [crates/vault/encryption.md](crates/vault/encryption.md) | stable | AES-256-GCM, EncryptedData, key versioning, salt (Phase B reserved) |
| [crates/vault/service.md](crates/vault/service.md) | stable | VaultServiceHandle lifecycle, direct dispatch, cache, error model |
| [crates/vault/protocol.md](crates/vault/protocol.md) | stable | DerivedKey redaction, KeyType, serialization behavior |
| [crates/hub/README.md](crates/hub/README.md) | draft | alknet-hub crate — aggregated peer env, connection lifecycle, worker supervision, service discovery |
| [crates/hub/README.md](crates/hub/README.md) | draft | alknet-hub crate — multi-transport endpoint (TCP+TLS + QUIC), channels substrate (ADR-079 relay), worker registration flow (OQ-58), identity over transports, aggregated peer env, connection lifecycle, service discovery |
| [crates/channels/README.md](crates/channels/README.md) | draft | alknet-channels crate — multiplexing proxy, 9-byte chunk format, N channels over one transport stream |
| [crates/channels/overview.md](crates/channels/overview.md) | draft | Crate purpose, the multiplexing collapse, dependencies, transport agnosticism, WASM, relationship to existing crates |
| [crates/channels/channels-wire.md](crates/channels/channels-wire.md) | draft | 9-byte chunk format, stream types, sentinels, framing disambiguation, wire-level invariants (REQ-CH-01..05) |
docs/architecture/crates/hub/README.md
+496 -219
View File
@@ -1,61 +1,134 @@
---
status: draft
last_updated: 2026-07-09
last_updated: 2026-07-13
---
# alknet-hub
The hub pattern as a reusable crate: peer lifecycle management, connection
supervision, operation aggregation, and service discovery across connected
peers. One QUIC connection per peer carrying the call protocol.
The hub pattern as a reusable crate: multi-transport endpoint that
accepts workers and browsers over TCP+TLS and QUIC, relays channels
between legs (ADR-079), manages peer lifecycle, aggregates operations,
and exposes service discovery. One channels connection per peer;
multiple endpoint types coexisting.
## What
`alknet-hub` is a thin crate that provides the head/worker (hub-spoke) pattern
as a reusable library. It depends on `alknet-call` and `alknet-core`. It does
not introduce new types — it wires the existing `PeerCompositeEnv`,
`CallClient`, `CallAdapter`, `from_call`, and `Dispatcher` types into a
coherent hub runtime.
`alknet-hub` is the crate that wires the hub role (ADR-029, ADR-034,
ADR-079) into a reusable library. A hub is the central node in a
hub-and-spoke (head/worker) topology: it accepts inbound connections from
workers and browsers, relays channels between legs, aggregates the
workers' operations into a shared environment, and serves the discovery
API. It depends on `alknet-channels` (the substrate), `alknet-call`
(the protocol), and `alknet-core`. It does not introduce new protocol
types — it wires the existing `ChannelsAdapter`, `ChannelClient`,
`CallAdapter`, `PeerCompositeEnv`, `from_call`, and `Dispatcher` types
into a coherent hub runtime.
### The hub is multi-transport
A hub **must** support TCP+TLS and QUIC endpoints simultaneously. This is
not a convenience — it is the structure of the primary use case. A hub
that provisions a worker (in a container, on vast.ai, on runpod) uses
TCP+TLS for registration and QUIC (or another transport) for the ongoing
channels session. The hub's HTTP endpoint (registration, browser access)
runs over TCP+TLS; the hub's channels endpoint runs over QUIC or
TCP+TLS depending on the worker's transport. These coexist on one hub.
The channels protocol is transport-agnostic (ADR-071; ADR-065
`Connection::from_stream`/`from_bidi`). The hub's accept and dial paths
inherit that property — they take a `Connection`, not a `SocketAddr`
welded to a dial. See "Transport" below.
### What the hub provides
1. **Peer lifecycle** — dial, accept, disconnect, reconnect with backoff.
Identity resolution via `IdentityProvider`. One connection per peer.
1. **Multi-transport endpoint** — accepts channels connections over
QUIC (via the quinn `AlknetEndpoint`) and over TCP+TLS (via an
accept loop that wraps each `TlsStream<TcpStream>` as a `Connection`
with `from_bidi`, per ADR-065/010). Both feed the same
`HandlerRegistry`. Also serves HTTP (`h2`/`http/1.1`) over TCP+TLS
for registration and browser access. All three coexist.
2. **Aggregated operation env** — a shared `PeerCompositeEnv` across all
calls (ADR-067). Operations discovered via `from_call` are registered in
each peer's connection overlay and aggregated into the shared env.
`invoke_peer` routes operation calls to the right peer.
2. **Peer lifecycle** — accept, dial, disconnect, reconnect with
backoff. Identity resolution via `IdentityProvider` (fingerprint or
bearer token, depending on transport — see "Identity over
transports"). One channels connection per peer.
3. **Service discovery** — `services/list-peers` returns each connected
worker's operation list via `PeerCompositeEnv::peer_operations` (ADR-068).
3. **Aggregated operation env** — a shared `PeerCompositeEnv` across
all connected workers (ADR-067). Operations discovered via
`from_call` are registered in each peer's connection overlay and
aggregated into the shared env. `invoke_peer` routes operation calls
to the right peer.
4. **Service discovery** — `services/list-peers` returns each
connected worker's operation list via
`PeerCompositeEnv::peer_operations` (ADR-068).
5. **Channel relay** — translates `channel/open` on channel 0 and
byte-forwards data channels with `channel_id` rewrite (ADR-079).
Lets a browser reach a spoke's channels through the hub without the
hub parsing any protocol-specific framing.
6. **Worker registration** (in scope of the hub) — the HTTP endpoint
that lets a freshly-provisioned worker enroll its key with a
one-time registration token. The registration flow is what makes
worker provisioning over TCP+TLS a hard requirement, not an
option. See "Worker registration" below and OQ-58.
## Why
The alkapi project identified that the hub pattern requires wiring that
alknet-call does not provide out of the box:
The hub pattern requires wiring that `alknet-channels` and
`alknet-call` do not provide out of the box:
- `Dispatcher::compose_root_env` builds a fresh `PeerCompositeEnv` per call
with only the current connection — multi-worker aggregation is not wired
(ADR-067).
- `PeerCompositeEnv::peer_operations` is not overridden — `services/list-peers`
returns empty operation lists for non-local peers (ADR-068).
- `from_call` is a free function, not wired into `CallClient::connect` — the
assembly layer must call it manually after every connect (ADR-069).
- The hub accepts channels connections over **multiple transports**
simultaneously. The channels crate is transport-agnostic (ADR-071,
ADR-065); the hub is where the multi-transport accept loop lives.
- The hub relays channels between legs (ADR-079) — terminating
channel 0 on each leg, translating `channel/open`, byte-forwarding
data channels. The channels crate is ALPN-blind and does not know it
is being relayed; the relay is a hub-crate concern.
- `Dispatcher::compose_root_env` builds a fresh `PeerCompositeEnv` per
call with only the current connection — multi-worker aggregation is
not wired (ADR-067).
- `PeerCompositeEnv::peer_operations` is not overridden —
`services/list-peers` returns empty operation lists for non-local
peers (ADR-068).
- `from_call` is a free function, not wired into `ChannelClient` —
the assembly layer must call it manually after every connect
(ADR-069).
- There is no worker supervision loop — reconnection, backoff, and
re-discovery are assembly-layer concerns.
- There is no registration endpoint — a freshly-provisioned worker
has no way to enroll its key with the hub before establishing a
channels connection.
These are not design flaws; they are the correct separation of concerns.
`alknet-call` provides the types and the routing logic; the hub wiring is a
consumer concern. But it is a concern *every* hub consumer shares. Rather than
each downstream project (alkapi, future hubs) building the same wiring
independently, `alknet-hub` provides it once, as a reusable crate.
These are not design flaws; they are the correct separation of
concerns. The channels and call crates provide the types and the
routing logic; the hub wiring is a consumer concern. But it is a
concern *every* hub consumer shares. Rather than each downstream
project (alkapi, future hubs) building the same wiring independently,
`alknet-hub` provides it once, as a reusable crate.
## Architecture
### The hub is built on channels, not call-directly
The hub's substrate is `alknet/channels`, not `alknet/call` directly.
Each peer (worker or browser) holds one channels connection to the
hub. Channel 0 on each connection is `alknet/call` (ADR-072); the
hub's `CallAdapter` runs on channel 0 for the hub's own operations,
for `channel/open` translation (ADR-079), and for `from_call`
discovery. Data channels carry the actual protocol work (TTY, SSH,
tunnels) and are relayed byte-for-byte across legs.
This is the post-channels hub model. The pre-channels model (one QUIC
connection per peer carrying `alknet/call` directly) is replaced: the
connection is `alknet/channels`, and `alknet/call` rides channel 0.
The hub's `CallClient`-direct dial path is replaced by
`ChannelClient::from_connection` (ADR-080).
### Hub struct
The `Hub` is the central type. It owns the aggregated `PeerCompositeEnv`, the
The `Hub` owns the aggregated `PeerCompositeEnv`, the
`OperationRegistry`, and the `Dispatcher`:
```rust
@@ -98,28 +171,98 @@ impl Hub {
}
```
The `Hub` exposes builder methods for optional hooks (`with_session_source`,
`with_ownership_provider`, `with_timeout`) that delegate to the `Dispatcher`.
The `Hub` exposes builder methods for optional hooks
(`with_session_source`, `with_ownership_provider`,
`with_timeout`) that delegate to the `Dispatcher`.
### Peer lifecycle
### Transport
Every hub-worker connection carries the call protocol. The hub establishes the
connection, runs `from_call` to discover the peer's operations, and attaches
the peer to the aggregated env.
The hub accepts and dials channels connections over any transport
the channels protocol supports (ADR-071). In practice, a hub runs
multiple accept loops simultaneously:
#### Inbound workers (workers dial the hub)
| Endpoint | Transport | What it carries |
|----------|-----------|-----------------|
| Quinn `AlknetEndpoint` | QUIC (quinn/iroh) | Channels connections from workers with QUIC reachability |
| TCP+TLS accept loop | TCP + TLS (`h2`/`http/1.1`/`alknet/channels`) | HTTP registration endpoint, browser access, channels-over-TCP from workers |
| (Future) WebTransport | WebTransport | Browser bidirectional path (deferred per ADR-044; WebSocket is the v1 browser path) |
The hub's `CallAdapter` accepts inbound connections. The hub provides a
`WorkerConnectedCallback` that fires inside the accept path, between
connection-establish and dispatch-start. The callback runs `from_call`,
registers the bundles, and attaches the peer to the aggregated env:
All accept loops feed into the same `HandlerRegistry`. The hub's
`HttpAdapter` serves `h2`/`http/1.1` over the TCP+TLS path for
registration and browser access; the `ChannelsAdapter` (registered on
the hub's `HandlerRegistry`) serves
`alknet/channels` over the QUIC path (and over TCP+TLS when a worker
dials channels-over-TCP). Both ALPNs are registered on the same
registry; the TLS handshake on each connection negotiates the ALPN
and dispatches to the right adapter.
The TCP+TLS accept loop constructs a `Connection` per accepted
`TlsStream<TcpStream>` via `Connection::from_bidi` (ADR-065) and
hands it to the same `HandlerRegistry::dispatch` the quinn endpoint
uses. This is the accept-loop-outside-the-endpoint pattern from
ADR-010's Amendment 1: the `AlknetEndpoint` struct stays QUIC-only
(no `tcp` field), and the TCP+TLS loop is a sibling accept source
that shares the registry. The hub is where that sibling loop lives.
#### Dial (outbound workers) — transport-agnostic
The hub dials outbound workers via `ChannelClient`, not `CallClient`.
The dial path mirrors the `from_connection` / `connect_quic` split
(ADR-080):
```rust
/// Callback invoked by CallAdapter::handle when a worker connects inbound.
/// Fires after identity resolution and before the dispatch loop starts.
/// Carries both on_connected (runs from_call, registers discovered ops,
/// attaches peer to aggregated env) and on_disconnected (detaches peer
/// on run_loop exit).
impl Hub {
/// Take over a pre-established channels `Connection` as a worker
/// connection. Transport-agnostic — the caller (or a transport
/// helper) produces the `Connection`. This is the primary path;
/// `connect_quic_worker` and future `connect_tcp_tls_worker` are
/// conveniences over it.
pub async fn dial_worker_connection(
&self,
connection: Connection,
config: FromCallConfig,
) -> Result<(PeerId, ChannelClient), HubError>;
/// QUIC convenience: dial a worker over QUIC, then
/// `dial_worker_connection`. Two-way door — additive over
/// `dial_worker_connection`.
pub async fn connect_quic_worker(
&self,
addr: SocketAddr,
credentials: CallCredentials,
config: FromCallConfig,
) -> Result<(PeerId, ChannelClient), HubError>;
}
```
`dial_worker_connection` is the transport-agnostic primary. After
taking over the `Connection`, it runs `from_call` on channel 0 to
discover the worker's operations, registers the discovered bundles in
the connection's Layer 2 overlay, and attaches the peer to the
aggregated env. `connect_quic_worker` is the "I just want QUIC"
convenience — it calls `ChannelClient::connect_quic`, then
`dial_worker_connection`. A future `connect_tcp_tls_worker` dials
TCP+TLS and calls `dial_worker_connection` the same way. The
one-way-door surface is `dial_worker_connection`; the dial helpers
are two-way-door conveniences.
#### Accept (inbound workers and browsers) — transport-agnostic
Inbound connections arrive over whatever transport the accept loop
yielded them from. The `ChannelsAdapter` (registered on
`alknet/channels` in the `HandlerRegistry`) receives a `Connection` and runs the demux loop
(ADR-075); the `CallAdapter` (running on channel 0) handles
hub-level operations and `channel/open` translation (ADR-079).
The `WorkerConnectedCallback` fires inside `ChannelsAdapter::handle`
between channel-0 establishment and dispatch start:
```rust
/// Callback invoked by ChannelsAdapter::handle when a peer connects
/// inbound. Fires after identity resolution and before the dispatch
/// loop starts. Carries both on_connected (runs from_call, registers
/// discovered ops, attaches peer to aggregated env) and on_disconnected
/// (detaches peer on run_loop exit).
pub struct WorkerConnectedCallback {
hub: Arc<Hub>,
config: FromCallConfig,
@@ -155,142 +298,164 @@ impl WorkerConnectedCallback {
}
```
The callback is wired into `CallAdapter` via a new builder method:
The callback is wired into `ChannelsAdapter` (the channels substrate)
via a builder method. The `ChannelsAdapter::handle` flow becomes:
```rust
impl CallAdapter {
pub fn with_worker_connected_callback(
mut self,
callback: WorkerConnectedCallback,
) -> Self {
self.worker_connected_callback = Some(callback);
self
}
}
```
1. Channel 0 (`alknet/call`) is preinstalled (ADR-072). The channel-0
`Connection` is handed to the `CallAdapter`.
2. Identity resolution — two paths depending on transport (see
"Identity over transports"):
- **Fingerprint path** (QUIC + raw key, or a transport with a
client cert): the `AuthContext` carried into
`ChannelsAdapter::handle` carries the peer's TLS fingerprint. The
`CallAdapter` resolves it via `resolve_from_fingerprint`.
- **Bearer-token path** (TCP+TLS with no client cert, WebTransport,
WebSocket): the `CallAdapter` on channel 0 extracts `auth_token`
from the first call frame (auth.md) and resolves it via
`resolve_from_token`. The transport carries no identity of its
own — the call protocol's first frame does.
3. The `CallConnection` is constructed with the resolved identity.
4. Invoke `on_connected` — runs `from_call`, registers bundles,
attaches peer to aggregated env.
5. Run the call-protocol dispatch loop on channel 0; the demux loop
continues on the other channels.
6. On disconnect, invoke `on_disconnected` — detaches peer from
aggregated env.
The `CallAdapter::handle` flow becomes:
1. Resolve identity from the connection.
2. Create `CallConnection`, set identity.
3. Invoke `on_connected` callback (if set) — runs `from_call`, registers
bundles, attaches peer to aggregated env.
4. Run the dispatch loop (`run_loop`).
5. On `run_loop` exit, invoke `on_disconnected` on the callback — detaches
peer from aggregated env.
The assembly layer constructs the callback and passes it to `CallAdapter`:
The assembly layer constructs the callback and passes it to
`ChannelsAdapter`:
```rust
let callback = WorkerConnectedCallback::new(Arc::clone(&hub), FromCallConfig::new());
let adapter = CallAdapter::new(registry, identity_provider)
.with_aggregated_env(hub.aggregated_env().clone())
let channels_adapter = ChannelsAdapter::new(Arc::clone(&registry), /* ... */)
.with_worker_connected_callback(callback);
// Register channels_adapter on alknet/channels in the HandlerRegistry.
// Both the quinn endpoint and the TCP+TLS accept loop dispatch
// alknet/channels connections to it.
```
> **Note on OQ-54**: The callback fires inside `handle()` because `handle()`
> blocks until disconnect — there is no "after handle accepts" point for the
> assembly layer to hook into. The callback is the committed design; OQ-54 is
> resolved.
### Identity over transports
#### Outbound workers (hub dials workers)
A peer's identity is resolved differently depending on the transport
the connection arrived over. This is not a hub invention — it is the
existing identity model (ADR-030, ADR-034, auth.md) applied to the
channels substrate.
The hub provides a `dial_worker` method that combines `CallClient::connect`,
`from_call`, and `attach_peer` into a single operation. It also provides a
`supervise_worker` method that wraps `dial_worker` in a reconnect loop with
configurable backoff:
| Transport | Identity source | Resolution path |
|-----------|----------------|-----------------|
| QUIC + RFC 7250 raw key | TLS fingerprint (automatic from handshake) | `resolve_from_fingerprint` — both sides present raw keys |
| QUIC + X.509 (client cert) | TLS fingerprint (from client cert) | `resolve_from_fingerprint` — worker presents a client cert |
| TCP+TLS (client cert) | TLS fingerprint (from client cert) | `resolve_from_fingerprint` — worker presents a client cert |
| TCP+TLS (no client cert) | Bearer token (call-protocol `auth_token` in first frame on channel 0) | `resolve_from_token` |
| WebTransport / WebSocket | Bearer token | `resolve_from_token` — browsers have no fingerprint (ADR-034 §4) |
The fingerprint path is the QUIC+raw-key optimization — identity is
"free" because the TLS handshake carries it. The X.509-client-cert
rows (QUIC and TCP+TLS) are the same path with a different cert
format — the hub matches the client cert's fingerprint via
`resolve_from_fingerprint` against the `SHA256:<hex>` entry in the
peer's `PeerEntry` (the mixed-fingerprint case from ADR-034 §3). A
worker may present an X.509 client cert over QUIC or TCP+TLS when the
hub's deployment uses X.509 rather than raw keys; the identity model
handles both identically.
The token path is the transport-agnostic fallback — it works over any
transport that can carry a call-protocol first frame, which is all of
them (channel 0 is `alknet/call`). `resolve_from_token` matches the
token against `PeerEntry.auth_token_hash` and returns the same
`PeerId` the fingerprint path would (ADR-030).
This is why channels-over-TCP is not a special case. The channels
protocol runs `alknet/call` on channel 0 (ADR-072); the call
protocol's `auth_token` path resolves identity from the first frame;
the transport carries no identity burden of its own. A worker that
registered its key over HTTP (TCP+TLS, no client cert) and then
connects via channels-over-TCP authenticates with the bearer token it
received at registration. A worker that connects via channels-over-QUIC
authenticates with its TLS fingerprint. Both resolve to the same
`PeerEntry` and the same `PeerId`.
### Worker registration
The registration flow is what makes TCP+TLS a hard requirement for
the hub. A freshly-provisioned worker (container, vast.ai, runpod)
does not yet have an established peer relationship with the hub — it
has a one-time registration token supplied via the provisioning
config. The flow:
1. The hub provisions a worker instance (docker, vast.ai, runpod —
platform-specific) and supplies a registration token via
`onStartCMD` (or equivalent).
2. The instance downloads the worker binary.
3. The instance generates an Ed25519 key pair (its future identity).
4. The instance POSTs to the hub's HTTP registration endpoint over
TCP+TLS, sending its public key and the registration token.
5. The hub validates the token, creates a `PeerEntry` for the worker
recording *both* the fingerprint (from the public key) and an
`auth_token_hash` (from a session token the hub issues), and returns
the session credential. The `PeerEntry` is the mixed-fingerprint
shape from ADR-034 §3 — fingerprint for the QUIC path,
`auth_token_hash` for the TCP+TLS path.
6. The instance connects to the hub via channels — over QUIC
(fingerprint identity) or over TCP+TLS (bearer-token identity) —
and both resolve to the same `PeerEntry` and the same `PeerId`.
The ongoing session begins.
Step 4 is HTTP over TCP+TLS. Step 6 is channels over QUIC or TCP+TLS.
Both happen; they are not alternatives. The registration endpoint is
an HTTP route on the `HttpAdapter` (registered on the hub's
`HandlerRegistry`, served on `h2`/`http/1.1`
over TCP+TLS), not a call-protocol operation — the worker has no
`CallConnection` yet at step 4.
The registration endpoint and the enrollment-token model are a
one-way-door API (the endpoint shape, the token semantics). That
decision is tracked as OQ-58 — it is decision-ready in shape (HTTP
POST, token in, `PeerEntry` out, session credential returned) but
the exact token model (one-time vs. refresh, single-use vs.
multi-use, rotation) and the endpoint path need a dedicated ADR
before the hub crate stabilizes.
### Supervision
The hub provides a `supervise_worker` method that wraps
`dial_worker_connection` (or a transport-specific helper) in a
reconnect loop with configurable backoff:
```rust
impl Hub {
/// Dial a worker, establish the call-protocol connection, discover its
/// operations, and attach it to the aggregated env. Returns the
/// worker's PeerId and the live CallConnection.
pub async fn dial_worker(
&self,
addr: SocketAddr,
credentials: CallCredentials,
config: FromCallConfig,
) -> Result<(PeerId, CallConnection), HubError> {
let client = CallClient::new(
Arc::clone(&self.registry),
Arc::clone(&self.identity_provider),
);
let connection = client.connect(addr, credentials).await?;
let peer_id = connection.identity()
.map(|id| id.id.clone())
.ok_or(HubError::NoPeerIdentity)?;
let bundles = from_call(&connection, config).await?;
connection.register_imported_all(bundles);
self.aggregated_env
.write()
.expect("aggregated env lock poisoned")
.attach_peer(peer_id.clone(), connection.overlay_env());
Ok((peer_id, connection))
}
/// Detach a peer from the aggregated env. Called on disconnect
/// (both inbound and outbound).
fn on_worker_disconnected(&self, peer_id: &PeerId) {
self.aggregated_env
.write()
.expect("aggregated env lock poisoned")
.detach_peer(peer_id);
}
/// Supervise an outbound worker: dial, discover, attach. On disconnect,
/// detach and retry with backoff. Runs until the Hub is dropped (the
/// returned JoinHandle can be aborted).
pub fn supervise_worker(
/// Supervise an outbound worker: dial, discover, attach. On
/// disconnect, detach and retry with backoff. Runs until the Hub
/// is dropped (the returned JoinHandle can be aborted).
///
/// `dial` is a closure that produces a channels `Connection` —
/// the hub does not bake a transport into the supervision loop.
/// The caller provides e.g. `|| async { Ok(Connection::from_quinn(quinn_endpoint.connect(addr, "alknet/channels")?.await?)) }`
/// for QUIC, or `|| async { Ok(Connection::from_bidi(tls_connector.connect(host, TcpStream::connect(addr).await?).await?, b"alknet/channels".to_vec(), Some(addr))) }`
/// for TCP+TLS. The supervision loop is transport-agnostic.
pub fn supervise_worker<F, Fut>(
self: &Arc<Self>,
addr: SocketAddr,
credentials: CallCredentials,
dial: F,
config: FromCallConfig,
backoff: BackoffConfig,
) -> tokio::task::JoinHandle<()> {
let hub = Arc::clone(self);
tokio::spawn(async move {
let mut retries = 0;
loop {
match hub.dial_worker(addr, credentials.clone(), config.clone()).await {
Ok((peer_id, connection)) => {
retries = 0;
// Wait for the connection to drop (run_loop exits).
// Committed interim (OQ-52): poll the underlying
// Connection's accept_bi() until it returns
// ConnectionClosed. A CallConnection::closed()
// method is the target resolution.
loop {
match connection.connection() {
Some(conn) => {
match conn.accept_bi().await {
Err(StreamError::ConnectionClosed)
| Err(StreamError::StreamClosed)
| Err(StreamError::Timeout) => break,
_ => continue,
}
}
None => break,
}
}
hub.on_worker_disconnected(&peer_id);
}
Err(e) => {
tracing::warn!(?e, retries, "worker dial failed; retrying");
}
}
let delay = backoff.delay_for(retries);
tokio::time::sleep(delay).await;
retries += 1;
}
})
}
) -> tokio::task::JoinHandle<()>
where
F: Fn() -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<Connection, HubError>> + Send + 'static;
}
```
The supervision loop takes a `dial` closure rather than a `SocketAddr`
+ `CallCredentials` pair. This keeps the loop transport-agnostic —
the caller decides the transport by what the closure does. The
backoff and re-discovery logic is the same regardless of transport.
Disconnect is detected via the OQ-52 interim — the loop polls
`connection().accept_bi()` until `ConnectionClosed` — pending a
`CallConnection::closed()` method (OQ-52 target resolution). On
disconnect, the peer is detached from the aggregated env and the
loop retries with backoff.
### Backoff configuration
```rust
@@ -320,12 +485,27 @@ impl Default for BackoffConfig {
}
```
### Channel relay (ADR-079)
When a browser (or any non-peer client) connects to the hub and opens
a channel to a spoke, the hub translates `channel/open` on channel 0
(terminate on both legs, re-issue with `forwarded_for` — ADR-032) and
byte-forwards data channels with `channel_id` rewrite. `channel/control`
operations on channel 0 carry `channel_id` in their JSON payload; the
hub's `CallAdapter` translates these too, rewriting `channel_id` to
the other leg's id (ADR-079). The hub does not run protocol-specific
handlers (`alknet/tty`, `alknet/ssh`, `alknet/tunnel`) — it runs
`alknet/channels` (the relay) and `alknet/call` (for its own ops +
translation). The full relay contract is in ADR-079; the relay
implementation lives in `alknet-hub`.
### Service discovery
The hub registers the built-in service discovery operations (`services/list`,
`services/schema`, `services/list-peers`) automatically. The
`services/list-peers` handler returns each connected worker's operation list
via `PeerCompositeEnv::peer_operations` (ADR-068).
The hub registers the built-in service discovery operations
(`services/list`, `services/schema`, `services/list-peers`)
automatically. The `services/list-peers` handler returns each
connected worker's operation list via
`PeerCompositeEnv::peer_operations` (ADR-068).
### HubError
@@ -339,41 +519,63 @@ pub enum HubError {
Discovery(#[from] AdapterError),
#[error("call client error: {0}")]
Client(#[from] ClientError),
#[error("channel error (client or adapter path): {0}")]
Channel(#[from] ChannelError),
#[error("registration failed: {0}")]
Registration(#[from] RegistrationError),
}
```
`RegistrationError` lives in `alknet-hub` (the registration endpoint is
a hub-crate surface). Its variants are defined alongside the OQ-58
resolution — the likely shape is `InvalidToken`, `ExpiredToken`,
`AlreadyEnrolled`, `Store(StoreError)`, but the exact set is not
fixed until the enrollment-token model is decided.
## What the hub does NOT do
- **Worker authentication policy.** The hub resolves the worker's identity
via `IdentityProvider` (the existing mechanism). Whether a given identity is
*allowed* to connect as a worker is an `AccessControl` decision on the
assembly layer's curated ops — the hub does not add a separate worker-auth
layer.
- **Worker-specific routing policy.** `PeerRef::Any` uses insertion-order
first-match (ADR-029 §2). A richer `RoutingPolicy` (round-robin,
least-loaded) is a future extension behind the same `PeerRef` enum.
- **Multi-hop federation.** The hub is one-hop: workers connect to the hub,
the hub composes their ops. Worker A does not transitively see worker B's
ops through the hub unless the hub explicitly re-exports them (ADR-029
Assumption 5).
- **Worker provisioning.** The hub does not spawn workers, configure them, or
manage their lifecycle beyond connection supervision. Worker provisioning
is an assembly-layer concern.
- **Worker authentication policy.** The hub resolves the worker's
identity via `IdentityProvider` (the existing mechanism). Whether a
given identity is *allowed* to connect as a worker is an
`AccessControl` decision on the assembly layer's curated ops — the
hub does not add a separate worker-auth layer.
- **Worker-specific routing policy.** `PeerRef::Any` uses
insertion-order first-match (ADR-029 §2). A richer
`RoutingPolicy` (round-robin, least-loaded) is a future extension
behind the same `PeerRef` enum.
- **Multi-hop federation.** The hub is one-hop: workers connect to
the hub, the hub composes their ops. Worker A does not transitively
see worker B's ops through the hub unless the hub explicitly
re-exports them (ADR-029 Assumption 5).
- **Worker provisioning.** The hub does not spawn workers, configure
them, or manage their lifecycle beyond connection supervision. The
registration endpoint enrolls a key; it does not provision the
instance. Worker provisioning (docker, vast.ai, runpod) is an
assembly-layer concern that calls the hub's registration endpoint
after provisioning.
## Crate dependencies
```
alknet-hub
├── alknet-call (CallClient, CallAdapter, Dispatcher, PeerCompositeEnv,
├── alknet-channels-call (ChannelClient, ChannelsAdapter, ChannelManager,
│ ChannelBidiStreamSource)
├── alknet-call (CallAdapter, Dispatcher, PeerCompositeEnv,
│ from_call, FromCallConfig, AdapterError, ClientError)
├── alknet-core (IdentityProvider, Connection, OperationRegistry)
├── alknet-http (HttpAdapter — for the registration endpoint and browser access)
├── alknet-core (IdentityProvider, Connection, OperationRegistry,
│ HandlerRegistry, AuthContext)
├── tokio (spawn, time::sleep)
└── tracing (logging)
```
`alknet-hub` depends on `alknet-call`, which depends on `alknet-core`. No new
dependency edges. The crate is a consumer of the call protocol types, not a
new protocol handler.
`alknet-hub` depends on `alknet-channels-call`, `alknet-call`, and
`alknet-http`, all of which depend on `alknet-core`. The hub is a
consumer of the channels substrate and the call protocol, not a new
protocol handler. The `alknet-http` dependency is for the
registration endpoint and browser HTTP access — the hub wires
`HttpAdapter` into the same `HandlerRegistry` as
`ChannelsAdapter`.
## Assembly layer integration
@@ -394,68 +596,143 @@ let hub = Arc::new(Hub::new(
Arc::clone(&identity_provider),
).with_ownership_provider(ownership_provider));
// 3. Start the inbound call-protocol listener (workers dial the hub)
// 3. Register the ChannelsAdapter on alknet/channels. Both the quinn
// endpoint and the TCP+TLS accept loop dispatch alknet/channels
// connections to this adapter.
let callback = WorkerConnectedCallback::new(Arc::clone(&hub), FromCallConfig::new());
let adapter = CallAdapter::new(Arc::clone(&registry), Arc::clone(&identity_provider))
.with_aggregated_env(hub.aggregated_env().clone())
let channels_adapter = ChannelsAdapter::new(/* ... */)
.with_worker_connected_callback(callback);
// ... register adapter on the QUIC endpoint
registry.register(b"alknet/channels", Arc::new(channels_adapter));
// 4. Dial outbound workers (hub dials workers)
// 4. Register the HttpAdapter on h2/http1.1. The TCP+TLS accept loop
// dispatches h2/http1.1 connections to this adapter (registration
// endpoint, browser access, stealth decoy).
let http_adapter = HttpAdapter::new(/* ... */);
registry.register(b"h2", Arc::new(http_adapter.clone()));
registry.register(b"http/1.1", Arc::new(http_adapter));
// 5. Start the QUIC accept loop (workers with QUIC reachability)
let quinn_endpoint = AlknetEndpoint::new(/* ... */, Arc::clone(&registry));
quinn_endpoint.run().await;
// 6. Start the TCP+TLS accept loop (registration, browser access,
// channels-over-TCP from workers)
let tcp_listener = TcpListener::bind(registration_addr).await?;
tokio::spawn(async move {
loop {
let (stream, _) = tcp_listener.accept().await?;
let tls_stream = tls_acceptor.accept(stream).await?;
let alpn = tls_stream.alpn()?;
let conn = Connection::from_bidi(tls_stream, alpn, /* remote_addr */);
registry.dispatch(conn).await; // routes by ALPN to HttpAdapter or ChannelsAdapter
}
});
// 7. Dial outbound workers (hub dials workers). The closure produces a
// channels Connection — the hub's supervise_worker calls
// dial_worker_connection internally. Both transports are shown; a
// real deployment picks one per worker.
// QUIC dial:
hub.supervise_worker(
dev1_addr,
dev1_credentials,
|| async move {
// Dial QUIC, wrap the quinn connection as a channels Connection.
let quinn_conn = quinn_endpoint
.connect(worker_addr, "alknet/channels")?
.await?;
Ok(Connection::from_quinn(quinn_conn))
},
FromCallConfig::new(),
BackoffConfig::default(),
);
// 5. Start the HTTP listener (clients dial the hub)
// ... HttpAdapter with the same registry and identity_provider
// TCP+TLS dial (e.g., for a worker that can't reach the hub over QUIC):
hub.supervise_worker(
|| async move {
let tcp = TcpStream::connect(worker_addr).await?;
let tls = tls_connector.connect("worker.example.com", tcp).await?;
// from_bidi wraps the TlsStream as a channels Connection
// (ADR-065). The ALPN must be alknet/channels.
Ok(Connection::from_bidi(tls, b"alknet/channels".to_vec(), Some(worker_addr)))
},
FromCallConfig::new(),
BackoffConfig::default(),
);
```
The hub's `aggregated_env()` accessor returns the shared `Arc<RwLock<PeerCompositeEnv>>`
so the assembly layer can wire it into `CallAdapter::with_aggregated_env`.
The hub's `aggregated_env()` accessor returns the shared
`Arc<RwLock<PeerCompositeEnv>>` so the assembly layer can wire it
into `CallAdapter::with_aggregated_env`.
## Design Decisions
| Decision | ADR | Summary |
|----------|-----|---------|
| Hub relay — translate, not transparently forward | [ADR-079](../../decisions/079-hub-relay-translate-not-forward.md) | Translate `channel/open` on channel 0 with `forwarded_for`; byte-forward data channels with `channel_id` rewrite |
| Aggregated peer-env wiring | [ADR-067](../../decisions/067-aggregated-peer-env-wiring.md) | `Dispatcher::with_aggregated_env` hook; `compose_root_env` reads shared env |
| PeerCompositeEnv::peer_operations | [ADR-068](../../decisions/068-peer-composite-env-peer-operations.md) | `list_operation_names` trait method; `PeerCompositeEnv` override |
| from_call is manual | [ADR-069](../../decisions/069-from-call-manual-free-function.md) | `from_call` is a free function; the hub calls it after connect |
| Peer-graph routing model | [ADR-029](../../decisions/029-peer-graph-routing-model.md) | Peer-keyed overlays, `PeerRef` routing, `AccessControl`-based peer auth |
| PeerEntry and Identity.id | [ADR-030](../../decisions/030-peerentry-and-identity-id-decoupling.md) | `PeerId` = `Identity.id` = `PeerEntry.peer_id` (stable) |
| Three peer roles | [ADR-034](../../decisions/034-outgoing-only-x509-and-three-peer-roles.md) | Hub = role-3 `PeerEntry` (mixed fingerprints); browsers not peers; bearer-token identity over TCP/WebTransport |
| ChannelClient — transport-agnostic | [ADR-080](../../decisions/080-channelclient.md) | `from_connection` primary, `connect_quic` convenience; the dial path the hub uses |
| Channels transport-agnostic | [ADR-071](../../decisions/071-channels-wire-format.md) | Substrate modes; `Connection::from_stream`/`from_bidi` (ADR-065) — the substrate the hub relays |
| TCP+TLS dispatch via from_stream | [ADR-010](../../decisions/010-alpn-router-and-endpoint.md) Am. 1 | `AlknetEndpoint` stays QUIC-only; TCP+TLS accept loop shares the registry |
| Channel 0 pre-negotiated | [ADR-072](../../decisions/072-channel-0-pre-negotiated-call.md) | Channel 0 = `alknet/call`; the `CallAdapter` runs here |
| Channel lifecycle operations | [ADR-073](../../decisions/073-channel-lifecycle-operations.md) | `channel/open`/`close`/`control`/`resources/subscribe` — what the hub translates |
## Open Questions
See [open-questions.md](../../open-questions.md) for full details.
- **OQ-52** (open): `CallConnection::wait_for_close()` — the supervision loop
needs a way to await connection close. The committed interim is polling
`connection().accept_bi()` until `ConnectionClosed`. A `closed()` method on
`CallConnection` is the target resolution.
- **OQ-53** (open): `BackoffConfig` defaults — the committed policy is 1s
initial, 60s max, 2x multiplier. OQ-53 tracks whether operational
experience warrants a change before the first release.
- **OQ-54** (resolved): Inbound worker hook placement — the callback fires
inside `CallAdapter::handle()` between identity resolution and dispatch
start. `handle()` blocks until disconnect, so there is no "after handle
accepts" point for the assembly layer to hook into. The
`WorkerConnectedCallback` is the committed design.
- **OQ-58** (open): Worker registration flow — the enrollment-token
model, the HTTP registration endpoint shape, and the
`register_worker` API. Decision-ready in shape (HTTP POST, token in,
`PeerEntry` created, session credential returned); the exact token
model (one-time vs. refresh, rotation) and endpoint path need a
dedicated ADR before the hub crate stabilizes.
- **OQ-52** (open): `CallConnection::wait_for_close()` — the
supervision loop needs a way to await connection close. The
committed interim is polling `connection().accept_bi()` until
`ConnectionClosed`. A `closed()` method on `CallConnection` is the
target resolution.
- **OQ-53** (open): `BackoffConfig` defaults — the committed policy
is 1s initial, 60s max, 2x multiplier. OQ-53 tracks whether
operational experience warrants a change before the first release.
- **OQ-54** (resolved): Inbound worker hook placement — the callback
fires inside `ChannelsAdapter::handle()` between identity resolution
and dispatch start. `handle()` blocks until disconnect, so there is
no "after handle accepts" point for the assembly layer to hook into.
The `WorkerConnectedCallback` is the committed design.
## References
- [channel-client.md](../channels/channel-client.md) — `ChannelClient`
(`from_connection` / `connect_quic` — the dial path)
- [channels-adapter.md](../channels/channels-adapter.md) —
`ChannelsAdapter`, `ChannelManager`, the accept path
- [channel-operations.md](../channels/channel-operations.md) —
`channel/open`/`close`/`control`, the hub relay contract
- [client-and-adapters.md](../call/client-and-adapters.md) — `CallClient`,
`from_call`, `OperationAdapter`
- [call-protocol.md](../call/call-protocol.md) — `CallAdapter`, `Dispatcher`,
`CallConnection`
- [operation-registry.md](../call/operation-registry.md) — `OperationRegistry`,
`OperationRegistryBuilder`, `OperationEnv`
- [call-protocol.md](../call/call-protocol.md) — `CallAdapter`,
`Dispatcher`, `CallConnection`
- [operation-registry.md](../call/operation-registry.md) —
`OperationRegistry`, `OperationRegistryBuilder`, `OperationEnv`
- [http-server.md](../http/http-server.md) — `HttpAdapter` (the
registration endpoint and browser HTTP access)
- [auth.md](../core/auth.md) — `resolve_from_token`,
`resolve_from_fingerprint` (the identity paths over transports)
- ADR-029: Peer-Graph Routing Model
- ADR-034: Three Peer Roles (hub = role-3, bearer-token identity)
- ADR-065: `Connection::from_stream`/`from_bidi` (TCP+TLS path)
- ADR-067: Aggregated Peer-Environment Wiring
- ADR-068: PeerCompositeEnv::peer_operations Override
- ADR-069: from_call Is a Manual Free Function
- alkapi [hub.md](/workspace/@alkdev/alkapi/docs/architecture/hub.md) — the
first hub consumer, the concrete use case that informed this crate
- ADR-079: Hub Relay — Translate, Not Transparently Forward
- ADR-080: ChannelClient (transport-agnostic `from_connection`)
- alkapi [hub.md](/workspace/@alkdev/alkapi/docs/architecture/hub.md) —
the first hub consumer, the concrete use case that informed this
crate
- alkapi [ADR-011](/workspace/@alkdev/alkapi/docs/architecture/decisions/011-aggregated-peer-env.md) —
the downstream aggregation decision
the downstream aggregation decision
docs/architecture/decisions/081-channels-subcrate-decomposition.md
+6 -2
View File
@@ -218,7 +218,11 @@ the dependency direction (hub/worker → channels, not channels → hub/worker).
- ADR-074: ChannelBidiStreamSource (in `channels-core`)
- ADR-075: ChannelsAdapter and ChannelManager (split: core demux in
`channels-core`, call coupling in `channels-call`)
- ADR-079: hub relay (in `channels-hub` or `alknet-hub`)
- ADR-080: ChannelClient (in `channels-worker`)
- ADR-079: hub relay (in `alknet-hub` — the hub crate consumes
`channels-call`; there is no `channels-hub` sub-crate, per this ADR's
Decision §"No hub/worker sub-crates")
- ADR-080: ChannelClient (in `channels-call`; there is no
`channels-worker` sub-crate — a worker is any crate that uses
`ChannelClient` to dial)
- `docs/architecture/crates/hub/README.md` — the existing hub crate (the
relay's consumer)
docs/architecture/open-questions.md
+1
View File
@@ -166,6 +166,7 @@ Door type is separate from whether a decision is made. A two-way door is a decis
| [OQ-52](questions/052-callconnection-wait-for-close.md) | CallConnection::wait_for_close() for supervision loop | open | two | med |
| [OQ-53](questions/053-backoff-config-defaults.md) | BackoffConfig default policy | open | two | low |
| [OQ-54](questions/054-inbound-worker-hook-placement.md) | Inbound worker on_worker_connected hook placement | resolved | two | low |
| [OQ-58](questions/058-worker-registration-flow.md) | Worker Registration Flow | open | one | high |
### alknet-channels
docs/architecture/questions/058-worker-registration-flow.md
+64
View File
@@ -0,0 +1,64 @@
# OQ-58: Worker Registration Flow
- **Origin**: `docs/architecture/crates/hub/README.md` §"Worker
registration" — the provisioning → enrollment → connection sequence
that makes TCP+TLS a hard requirement for the hub.
- **Status**: open
- **Door type**: one-way (the registration endpoint shape and the
enrollment-token model are an API surface; changing them after
workers are provisioned against them is a breaking change for every
deployment)
- **Priority**: high
- **Blocked on**: nothing structural — the identity machinery
(`resolve_from_token`, `PeerEntry.auth_token_hash`, ADR-030/034)
and the HTTP substrate (`HttpAdapter` on `h2`/`http/1.1` over
TCP+TLS, ADR-010 Am. 1 / ADR-065) both exist. What remains is
deciding the token model and the endpoint shape, not building new
primitives.
- **Resolution**: Not yet decided. The flow is decision-ready in
shape (see "Decision-ready shape" below). The open sub-questions are:
1. **Enrollment-token model.** Is the registration token one-time
single-use (burned on first POST), one-time-per-worker (a worker
can retry the POST until it succeeds, then the token is
invalidated), or refresh-capable (the token can enroll multiple
workers, e.g. a pool token)? One-time-per-worker is the likely
default (matches the provisioning flow: one token per instance);
pool tokens are a feature extension.
2. **Session credential returned.** After registration, does the
hub return a bearer token for the ongoing channels connection
(the worker authenticates over TCP+TLS via `auth_token`), or
does it record the worker's fingerprint and expect fingerprint
auth on the channels connection (the worker authenticates over
QUIC via its raw key)? Both are valid; the choice depends on
whether the worker's transport is known at registration time. A
hub that accepts both should probably support both return shapes
(return a token *and* record the fingerprint).
3. **Endpoint path.** `POST /register`? `POST /v1/workers/register`?
A versioned path is safer for a one-way door.
4. **Token source.** Does the hub generate the enrollment token, or
does the assembly layer (the provisioning system) generate it
and the hub just validates it? The latter keeps the hub out of
the token-generation business but requires a shared secret or
signature scheme.
- **Decision-ready shape**: the registration endpoint is an HTTP POST
on the hub's `HttpAdapter` (served on `h2`/`http/1.1` over TCP+TLS).
The request carries the worker's public key and the enrollment
token. The hub validates the token, creates a `PeerEntry` for the
worker (fingerprint from the key, `auth_token_hash` from a
session token the hub issues), and returns the session credential.
The worker then connects via channels (QUIC or TCP+TLS) and
authenticates with the fingerprint or the bearer token. The
`PeerEntry` created at registration is what `resolve_from_token`
/ `resolve_from_fingerprint` matches at connection time.
- **What does NOT block on this**: the hub's multi-transport accept
loop, the channels-over-TCP path, the identity-over-TCP path
(bearer token via `resolve_from_token`), and the supervision loop.
All of these are decided and use existing machinery. OQ-58 is
about the registration endpoint specifically — the one new surface
the hub introduces.
- **Cross-references**: ADR-030 (PeerEntry, auth_token_hash),
ADR-034 (bearer-token identity over non-fingerprint transports),
ADR-010 Am. 1 (TCP+TLS dispatch via from_stream), ADR-065
(Connection::from_bidi), ADR-080 (ChannelClient::from_connection),
OQ-52 (CallConnection::wait_for_close — the supervision loop the
registration flow feeds into).