docs(research): add iroh suite deep-dive references for iroh, irpc, iroh-blobs, iroh-gossip, iroh-live, and iroh-docs
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# iroh-live: Overview and Architecture
|
||||
|
||||
## What It Is
|
||||
|
||||
iroh-live is a real-time audio/video streaming system built on top of [iroh](https://github.com/n0-computer/iroh) (QUIC-based P2P networking) and [Media over QUIC (MoQ)](https://moq.dev/). It handles the full pipeline: camera/mic capture → encoding → transport → decoding → rendering. Connections are peer-to-peer by default, with an optional relay server for browser access via WebTransport.
|
||||
|
||||
**Status:** Early tech preview. APIs are unstable. Windows support is missing. Audio-video sync is basic.
|
||||
|
||||
## Workspace Crates
|
||||
|
||||
| Crate | Description |
|
||||
|-------|-------------|
|
||||
| `iroh-live` | High-level API: `Live`, `Call`, `Room`, tickets, subscriptions |
|
||||
| `iroh-moq` | MoQ transport layer over iroh/QUIC via `web-transport-iroh` |
|
||||
| `iroh-live-relay` | Relay server bridging iroh P2P to browser WebTransport |
|
||||
| `moq-media` | Media pipelines: capture, encode, decode, publish, subscribe, adaptive bitrate. No iroh dependency |
|
||||
| `rusty-codecs` | Codec implementations (H264/openh264, AV1/rav1e+ rav1d, Opus), hardware accel (VAAPI, V4L2, VideoToolbox) |
|
||||
| `rusty-capture` | Cross-platform capture: PipeWire, V4L2, X11, ScreenCaptureKit, AVFoundation |
|
||||
| `moq-media-egui` | egui integration for video rendering |
|
||||
| `moq-media-dioxus` | dioxus-native integration for video rendering |
|
||||
| `moq-media-android` | Android camera, EGL rendering, JNI bridge |
|
||||
| `iroh-live-cli` | CLI tool (`irl`) for publishing, playing, calls, rooms, relay |
|
||||
|
||||
## Layer Architecture
|
||||
|
||||
Three distinct layers, each usable independently:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ iroh-live │
|
||||
│ Session management, tickets, rooms, calls │
|
||||
│ Re-exports: moq-media, iroh-moq │
|
||||
├──────────────────────────────────────────────────────────┤
|
||||
│ moq-media │
|
||||
│ Media pipelines: LocalBroadcast, RemoteBroadcast, │
|
||||
│ codecs, adaptive bitrate, playout │
|
||||
│ NO iroh dependency (transport-agnostic) │
|
||||
├──────────────────────────────────────────────────────────┤
|
||||
│ iroh-moq │
|
||||
│ MoQ session management, publish/subscribe over QUIC │
|
||||
│ Uses web-transport-iroh + moq-lite │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
|
||||
Below moq-media:
|
||||
rusty-codecs ─ codec implementations, hardware accel, wgpu rendering
|
||||
rusty-capture ─ platform-specific screen/camera capture
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **`&self` everywhere** — All public types use interior mutability. Safe to share across async tasks/threads without wrappers.
|
||||
2. **Drop-based cleanup** — Dropping a `Call` closes it. Dropping `LocalBroadcast` tears down encoders. Dropping `VideoTrack` stops its decoder thread.
|
||||
3. **Watcher for continuous state, Stream for discrete events** — Connection quality and catalog contents use `n0_watcher::Direct<T>`. Participant joins use `impl Stream`.
|
||||
4. **Declarative intent, not mechanism** — `VideoTarget::default().max_pixels(1280*720)` describes what quality you need. The catalog selects the best rendition.
|
||||
5. **moq-media is standalone** — A recording pipeline can use `LocalBroadcast`/`RemoteBroadcast` without iroh-live. The transport boundary is the `PacketSink`/`PacketSource` trait pair.
|
||||
|
||||
## Data Flow (End-to-End)
|
||||
|
||||
```
|
||||
Publisher Side:
|
||||
capture source (rusty-capture, VideoSource trait)
|
||||
│
|
||||
▼
|
||||
encoder pipeline (moq-media, dedicated OS thread)
|
||||
│
|
||||
▼ EncodedFrame
|
||||
PacketSink (MoqPacketSink — starts new MoQ group on keyframe)
|
||||
│
|
||||
▼ MoQ transport (iroh-moq, QUIC streams)
|
||||
|
||||
Subscriber Side:
|
||||
PacketSource (MoqPacketSource — reads ordered frames from MoQ)
|
||||
│
|
||||
▼ MediaPacket
|
||||
decoder pipeline (moq-media, dedicated OS thread)
|
||||
│
|
||||
▼ VideoFrame
|
||||
FramePacer (PTS-based sleep) or Sync (shared playout clock)
|
||||
│
|
||||
▼
|
||||
renderer (wgpu texture upload or egui widget)
|
||||
```
|
||||
|
||||
Encoder and decoder pipelines run on **dedicated OS threads**, not tokio tasks, so slow codec operations never block the async runtime. The `forward_packets` async task bridges the network-side `PacketSource` into an mpsc channel that the decoder thread reads synchronously.
|
||||
|
||||
## Key Dependencies
|
||||
|
||||
| Dependency | Purpose |
|
||||
|------------|---------|
|
||||
| `iroh` | QUIC endpoint, connection management, P2P connectivity |
|
||||
| `iroh-gossip` | Gossip protocol for room participant discovery |
|
||||
| `iroh-tickets` | Ticket serialization for `RoomTicket` |
|
||||
| `iroh-smol-kv` | Distributed KV store for room state (gossip-backed) |
|
||||
| `moq-lite` | Core MoQ protocol: BroadcastProducer, BroadcastConsumer, Track, Group |
|
||||
| `hang` | Catalog management for broadcast metadata |
|
||||
| `moq-mux` | MoQ multiplexing |
|
||||
| `moq-relay` | Relay server implementation (used by iroh-live-relay) |
|
||||
| `web-transport-iroh` | WebTransport over iroh QUIC connections |
|
||||
| `n0-future` | Async utilities (FuturesUnordered, AbortOnDropHandle) |
|
||||
| `n0-watcher` | Watchable/Direct reactive state |
|
||||
|
||||
## License
|
||||
|
||||
Dual-licensed: MIT OR Apache-2.0. Copyright 2025 N0, INC.
|
||||
167
docs/research/references/iroh/iroh-live/02-core-api.md
Normal file
167
docs/research/references/iroh/iroh-live/02-core-api.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# iroh-live: Core API — Live, Call, Subscription, Ticket
|
||||
|
||||
## `Live` — Entry Point
|
||||
|
||||
The primary entry point for all iroh-live operations. Manages an iroh `Endpoint`, the MoQ transport (`Moq`), and optionally a `Gossip` instance for rooms.
|
||||
|
||||
### Construction
|
||||
|
||||
```rust
|
||||
// Simple: from environment, accept incoming connections
|
||||
let live = Live::from_env().await?.with_router().spawn();
|
||||
|
||||
// With gossip for rooms
|
||||
let live = Live::from_env().await?.with_router().with_gossip().spawn();
|
||||
|
||||
// From an existing endpoint
|
||||
let live = Live::builder(endpoint).with_router().with_gossip().spawn();
|
||||
|
||||
// Manual router mounting (when you have other protocols)
|
||||
let router = live.register_protocols(Router::builder(endpoint));
|
||||
let router = router.accept(other_protocol, other_handler);
|
||||
let router = router.spawn();
|
||||
```
|
||||
|
||||
### Key Methods
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `publish(name, &LocalBroadcast)` | Register a broadcast for all connected peers |
|
||||
| `subscribe(remote, name)` | Connect to a peer and subscribe to a broadcast → `Subscription` |
|
||||
| `subscribe_media(remote, name, audio, config)` | Connect, subscribe, decode → `(MoqSession, MediaTracks)` |
|
||||
| `join_room(ticket)` | Join a gossip-based multi-party room → `Room` |
|
||||
| `endpoint()` | Access the underlying iroh `Endpoint` |
|
||||
| `transport()` | Access the `Moq` transport for advanced operations |
|
||||
| `gossip()` | Access the `Gossip` instance (if enabled) |
|
||||
| `shutdown()` | Close all sessions, stop router, close endpoint |
|
||||
|
||||
### Builder Options
|
||||
|
||||
- **`with_router()`** — Spawns an internal `Router` so the endpoint accepts incoming MoQ sessions. Without this, only outbound connections work.
|
||||
- **`with_gossip()`** — Creates a `Gossip` instance (required for rooms). Internally mounts on the Router if `with_router` is also set.
|
||||
- **`gossip(gossip)`** — Use an externally-managed `Gossip` instance.
|
||||
|
||||
### Internal Architecture
|
||||
|
||||
`Live` holds:
|
||||
- `endpoint: Endpoint` — iroh QUIC endpoint
|
||||
- `moq: Moq` — Internal actor for session/broadcast management
|
||||
- `gossip: Option<Gossip>` — For room coordination
|
||||
- `router: Option<Router>` — For accepting incoming connections
|
||||
|
||||
The `from_env()` method reads `IROH_SECRET` for the secret key and generates one if not set. It uses the `N0` preset for relay and DNS discovery.
|
||||
|
||||
## `LiveTicket` — Connection Sharing
|
||||
|
||||
A serializable ticket that contains everything needed to connect to a publisher.
|
||||
|
||||
```rust
|
||||
// Create a ticket
|
||||
let ticket = LiveTicket::new(endpoint.addr(), "my-stream");
|
||||
|
||||
// Serialize to URI string (fits in QR codes)
|
||||
let s = ticket.to_string();
|
||||
// → "iroh-live:<base64url(postcard(EndpointAddr))>/my-stream"
|
||||
|
||||
// Deserialize
|
||||
let parsed: LiveTicket = s.parse()?;
|
||||
|
||||
// With relay URLs for indirect connectivity
|
||||
let ticket = LiveTicket::new(addr, "stream").with_relay_urls(vec![
|
||||
"https://relay.example.com".to_string(),
|
||||
]);
|
||||
```
|
||||
|
||||
**Format:** `iroh-live:<base64url(postcard(EndpointAddr))>/<name>`
|
||||
|
||||
Also supports legacy `name@base32` format for backward compatibility.
|
||||
|
||||
The ticket string is kept short enough for QR codes (< 2000 bytes). It uses postcard (binary) serialization with base64url encoding.
|
||||
|
||||
## `Call` — 1:1 Video Call
|
||||
|
||||
A convenience wrapper over MoQ primitives for bidirectional calls.
|
||||
|
||||
### Flow
|
||||
|
||||
1. One side creates a `LocalBroadcast` with video/audio configured
|
||||
2. **Dialer:** `Call::dial(live, remote_addr, local_broadcast)` — connects, publishes "call" broadcast, subscribes to remote's "call" broadcast
|
||||
3. **Acceptor:** `Call::accept(session, local_broadcast)` — accepts an incoming session, publishes and subscribes
|
||||
|
||||
The broadcast name is always `"call"` — this is hardcoded (`CALL_BROADCAST_NAME`).
|
||||
|
||||
```rust
|
||||
// Dialer side
|
||||
let local = LocalBroadcast::new();
|
||||
local.video().set_source(camera, VideoCodec::H264, [VideoPreset::P720])?;
|
||||
let call = Call::dial(&live, remote_addr, local).await?;
|
||||
|
||||
// Access remote media
|
||||
let remote_broadcast = call.remote();
|
||||
let video = remote_broadcast.video()?;
|
||||
|
||||
// Wait for call to end
|
||||
let reason = call.closed().await;
|
||||
```
|
||||
|
||||
### Key Properties
|
||||
|
||||
- `call.local()` → `&LocalBroadcast` (your media)
|
||||
- `call.remote()` → `&RemoteBroadcast` (peer's media)
|
||||
- `call.signals()` → `watch::Receiver<NetworkSignals>` (for adaptive bitrate)
|
||||
- `call.close()` — closes with error code 0 and reason "call ended"
|
||||
- `call.closed()` → waits for close, returns `DisconnectReason` (LocalClose, RemoteClose, TransportError)
|
||||
|
||||
Auto-wires stats recording and network signal production on the connection.
|
||||
|
||||
## `Subscription` — Subscribe Handle
|
||||
|
||||
Created by `Live::subscribe()`. Wraps the MoQ session, remote broadcast, and network signals into a single handle. The constructor auto-wires stats recording and signal production.
|
||||
|
||||
```rust
|
||||
let sub = live.subscribe(remote_addr, "stream").await?;
|
||||
|
||||
// Access components
|
||||
sub.session() // &MoqSession
|
||||
sub.broadcast() // &RemoteBroadcast
|
||||
sub.signals() // &watch::Receiver<NetworkSignals>
|
||||
|
||||
// Convenience methods
|
||||
let tracks = sub.media(&audio_backend, Default::default()).await?;
|
||||
let tracks = sub.media_with_decoders::<DefaultDecoders>(&audio_backend, config).await?;
|
||||
|
||||
// Decompose
|
||||
let (session, broadcast, signals) = sub.into_parts();
|
||||
```
|
||||
|
||||
## `DisconnectReason`
|
||||
|
||||
```rust
|
||||
pub enum DisconnectReason {
|
||||
LocalClose,
|
||||
RemoteClose,
|
||||
TransportError,
|
||||
}
|
||||
```
|
||||
|
||||
Derived from the QUIC connection's close reason. Used by `Call::closed()`.
|
||||
|
||||
## `util` Module
|
||||
|
||||
### `secret_key_from_env()`
|
||||
|
||||
Loads the iroh secret key from the `IROH_SECRET` environment variable. Generates a new key if not set, printing the hex-encoded key for reuse.
|
||||
|
||||
### `spawn_signal_producer(conn, shutdown)`
|
||||
|
||||
Spawns a background task that polls QUIC connection path stats every 200ms and produces `NetworkSignals` for adaptive rendition selection. Returns a `watch::Receiver<NetworkSignals>`.
|
||||
|
||||
Computes:
|
||||
- **RTT** — from `selected_path.rtt()`
|
||||
- **Loss rate** — delta-based (lost packets / (sent + lost) over the interval)
|
||||
- **Available bandwidth** — estimated from congestion window: `cwnd * 8 / rtt`
|
||||
- **Congestion events** — monotonically increasing counter
|
||||
|
||||
### `spawn_stats_recorder(conn, net_stats, shutdown)`
|
||||
|
||||
Records connection stats (RTT, loss rate, bandwidth, path type) into `NetStats` for debug overlay display. Runs every 200ms.
|
||||
164
docs/research/references/iroh/iroh-live/03-iroh-moq-transport.md
Normal file
164
docs/research/references/iroh/iroh-live/03-iroh-moq-transport.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# iroh-moq: MoQ Transport Layer
|
||||
|
||||
## Overview
|
||||
|
||||
`iroh-moq` is the transport bridge between iroh's QUIC endpoint and the moq-lite broadcast protocol. It manages connections, session lifecycle, broadcast routing, and subscription handling. This is the only crate in the workspace that directly interacts with QUIC transport — everything above uses `Moq`/`MoqSession` as the interface.
|
||||
|
||||
**ALPN:** `moq-lite-03`
|
||||
|
||||
## Core Types
|
||||
|
||||
### `Moq` — Transport Manager
|
||||
|
||||
The top-level transport entry point. Wraps an iroh `Endpoint` and runs an internal actor (`Actor`) that handles all connection and broadcast management.
|
||||
|
||||
```rust
|
||||
let moq = Moq::new(endpoint);
|
||||
```
|
||||
|
||||
**Internal architecture:**
|
||||
|
||||
`Moq` holds an `mpsc::Sender<ActorMessage>` to communicate with a spawned actor task. The actor manages:
|
||||
- A `HashMap<EndpointId, MoqSession>` of active sessions
|
||||
- A `HashMap<BroadcastName, BroadcastProducer>` of locally published broadcasts
|
||||
- A `JoinSet` of session tasks (each tracks session lifetime)
|
||||
- A `FuturesUnordered` of pending connect tasks
|
||||
- A `broadcast::Sender<MoqSession>` for incoming session notifications
|
||||
|
||||
**Key methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `new(endpoint)` | Creates transport and spawns the actor |
|
||||
| `protocol_handler()` | Returns `MoqProtocolHandler` for Router registration |
|
||||
| `publish(name, producer)` | Register a broadcast for all current and future sessions |
|
||||
| `connect(remote)` | Connect to remote peer, deduplicating existing connections |
|
||||
| `incoming_sessions()` | Get stream of incoming sessions |
|
||||
| `published_broadcasts()` | List currently published broadcast names |
|
||||
| `shutdown()` | Cancel the shutdown token, closing all sessions |
|
||||
|
||||
### `MoqProtocolHandler`
|
||||
|
||||
Implements iroh's `ProtocolHandler` trait. When the Router receives an incoming connection with the `moq-lite-03` ALPN:
|
||||
|
||||
1. Accepts the raw QUIC `Connection`
|
||||
2. Wraps it in a `web_transport_iroh::Session::raw(connection)`
|
||||
3. Completes the moq-lite server handshake: `MoqSession::session_accept(wt_session)`
|
||||
4. Sends the session to the actor via `ActorMessage::HandleSession`
|
||||
|
||||
### `MoqSession` — Single Peer Connection
|
||||
|
||||
Represents a MoQ connection with one remote peer. Created via:
|
||||
- `Moq::connect()` (outbound, client role)
|
||||
- `IncomingSession::accept()` (inbound, server role)
|
||||
|
||||
```rust
|
||||
// Outbound
|
||||
let session = moq.connect(remote_addr).await?;
|
||||
|
||||
// Inbound
|
||||
let incoming = incoming_session.next().await?;
|
||||
let session = incoming.accept(); // or incoming.reject()
|
||||
```
|
||||
|
||||
**Internal structure:**
|
||||
|
||||
```rust
|
||||
pub struct MoqSession {
|
||||
wt_session: web_transport_iroh::Session,
|
||||
_moq_session: Arc<moq_lite::Session>,
|
||||
publish: OriginProducer, // For announcing local broadcasts
|
||||
subscribe: OriginConsumer, // For consuming remote broadcasts
|
||||
}
|
||||
```
|
||||
|
||||
The `OriginProducer`/`OriginConsumer` pair comes from moq-lite. The session creates them before the handshake:
|
||||
|
||||
- **Client (connect):** Creates `OriginProducer` for publish and `OriginConsumer` for subscribe, then `Client::new().with_publish(...).with_consume(...).connect(session)`
|
||||
- **Server (accept):** Same pattern with `Server::new().with_publish(...).with_consume(...).accept(session)`
|
||||
|
||||
**Key methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `subscribe(name)` | Wait for remote to announce broadcast, return `BroadcastConsumer` |
|
||||
| `publish(name, consumer)` | Make a broadcast available to remote peer |
|
||||
| `conn()` | Reference to underlying QUIC `Connection` (for stats) |
|
||||
| `remote_id()` | Remote peer's `EndpointId` |
|
||||
| `close(code, reason)` | Close the session |
|
||||
| `closed()` | Wait for session to close, returns `SessionError` |
|
||||
| `origin_producer()` | Direct access to moq-lite publish origin |
|
||||
| `origin_consumer()` | Direct access to moq-lite subscribe origin |
|
||||
|
||||
### `IncomingSession` / `IncomingSessionStream`
|
||||
|
||||
`IncomingSession` wraps a `MoqSession` that has completed the handshake. Provides:
|
||||
- `remote_id()` — the connecting peer's identity
|
||||
- `accept()` — returns the `MoqSession`
|
||||
- `reject()` — closes with error code 1
|
||||
|
||||
`IncomingSessionStream` is an async stream that yields `IncomingSession` values. Uses a `broadcast::Receiver<MoqSession>` internally, handling lag by skipping missed sessions.
|
||||
|
||||
## Actor Internals
|
||||
|
||||
The `Actor` is the core event loop for the `Moq` transport:
|
||||
|
||||
```
|
||||
loop {
|
||||
select! {
|
||||
msg = inbox.recv() → handle_message(msg)
|
||||
session_closed → remove session, log
|
||||
broadcast_closed → remove from publishing map
|
||||
connect_completed → handle_session or reply to caller
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Message Types
|
||||
|
||||
```rust
|
||||
enum ActorMessage {
|
||||
HandleSession { session: Box<MoqSession> },
|
||||
LocalBroadcast { broadcast_name: String, producer: BroadcastProducer },
|
||||
Connect { remote: EndpointAddr, reply: oneshot::Sender<...> },
|
||||
GetPublished { reply: oneshot::Sender<Vec<String>> },
|
||||
}
|
||||
```
|
||||
|
||||
### Connection Deduplication
|
||||
|
||||
When `Connect` is received for a peer that already has an active session, the existing session is returned immediately. If a connection attempt is already in progress, the oneshot reply is queued and notified when the connection completes.
|
||||
|
||||
### Broadcast Fan-out
|
||||
|
||||
When a `LocalBroadcast` is published via `Moq::publish()`:
|
||||
1. The actor stores the `BroadcastProducer` in its `publishing` map
|
||||
2. It immediately announces the broadcast to all existing sessions by calling `session.publish(name, producer.consume())` on each
|
||||
3. For future sessions, the actor iterates `publishing` entries and announces each one
|
||||
4. A `FuturesUnordered` tracks when each broadcast closes, removing it from the map
|
||||
|
||||
### Session Lifecycle
|
||||
|
||||
When a session is established (either incoming or outgoing):
|
||||
1. All currently published broadcasts are announced to it
|
||||
2. It's stored in `sessions` by `EndpointId`
|
||||
3. A session task is spawned that waits for the session to close
|
||||
4. If there were pending connect requests for this peer, they're fulfilled
|
||||
|
||||
## Error Types
|
||||
|
||||
```rust
|
||||
enum Error {
|
||||
Connect(ConnectError), // iroh connection failure
|
||||
Moq(moq_lite::Error), // MoQ protocol error
|
||||
Server(web_transport_iroh::ServerError), // WebTransport server error
|
||||
InternalConsistencyError(LiveActorDiedError), // Actor died
|
||||
Request(WriteError), // QUIC write error
|
||||
}
|
||||
|
||||
enum SubscribeError {
|
||||
NotAnnounced, // Track was not announced
|
||||
Closed, // Track was closed
|
||||
SessionClosed(SessionError), // Session closed
|
||||
}
|
||||
```
|
||||
185
docs/research/references/iroh/iroh-live/04-rooms.md
Normal file
185
docs/research/references/iroh/iroh-live/04-rooms.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# iroh-live: Rooms — Multi-Party Coordination
|
||||
|
||||
## Overview
|
||||
|
||||
The `rooms` module provides multi-party room coordination over iroh-gossip. Participants discover each other via a gossip topic, automatically connect and subscribe to each other's broadcasts, and receive `RoomEvent` notifications as peers join, publish, and leave.
|
||||
|
||||
## Core Types
|
||||
|
||||
### `Room`
|
||||
|
||||
The main room handle. Created via `Room::new(live, ticket)`. Spawns an internal actor that manages all peer coordination.
|
||||
|
||||
```rust
|
||||
// Create a room (generates a random topic)
|
||||
let ticket = RoomTicket::generate();
|
||||
let room = Room::new(&live, ticket.clone()).await?;
|
||||
|
||||
// Or join an existing room
|
||||
let room = Room::new(&live, existing_ticket).await?;
|
||||
```
|
||||
|
||||
**Methods:**
|
||||
- `recv()` — Wait for next `RoomEvent`
|
||||
- `try_recv()` — Non-blocking event check
|
||||
- `ticket()` — Get a ticket that includes this peer as a bootstrap node
|
||||
- `split()` — Decompose into `(RoomEvents, RoomHandle)` for use in separate tasks
|
||||
- `publish(name, &LocalBroadcast)` — Publish a broadcast to the room
|
||||
- `set_chat_publisher(publisher)` — Register a chat publisher
|
||||
- `send_chat(text)` — Send a chat message
|
||||
|
||||
### `RoomHandle`
|
||||
|
||||
Cloneable handle for publishing into a room. Obtained from `Room::split()`. Can be shared across tasks.
|
||||
|
||||
```rust
|
||||
let (events, handle) = room.split();
|
||||
|
||||
// In one task: receive events
|
||||
while let Some(event) = events.recv().await {
|
||||
match event { ... }
|
||||
}
|
||||
|
||||
// In another task: publish
|
||||
handle.publish("camera", &broadcast).await?;
|
||||
handle.send_chat("Hello!").await?;
|
||||
handle.set_display_name("Alice").await?;
|
||||
```
|
||||
|
||||
### `RoomTicket`
|
||||
|
||||
```rust
|
||||
pub struct RoomTicket {
|
||||
pub bootstrap: Vec<EndpointId>, // Bootstrap peer IDs for gossip
|
||||
pub topic_id: TopicId, // Gossip topic identifier
|
||||
}
|
||||
```
|
||||
|
||||
Serialized via `iroh_tickets` (binary format). Can be created from:
|
||||
- `RoomTicket::generate()` — Random topic, no bootstrap
|
||||
- `RoomTicket::new(topic_id, bootstrap)` — Specific topic and peers
|
||||
- `RoomTicket::new_from_env()` — From `IROH_LIVE_ROOM` or `IROH_LIVE_TOPIC` env vars
|
||||
|
||||
### `RoomEvent`
|
||||
|
||||
```rust
|
||||
pub enum RoomEvent {
|
||||
RemoteAnnounced {
|
||||
remote: EndpointId,
|
||||
broadcasts: Vec<String>,
|
||||
},
|
||||
BroadcastSubscribed {
|
||||
session: Box<MoqSession>,
|
||||
broadcast: Box<RemoteBroadcast>,
|
||||
},
|
||||
PeerJoined {
|
||||
remote: EndpointId,
|
||||
display_name: Option<String>,
|
||||
},
|
||||
PeerLeft {
|
||||
remote: EndpointId,
|
||||
},
|
||||
ChatReceived {
|
||||
remote: EndpointId,
|
||||
message: ChatMessage,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Room Actor — Internal Architecture
|
||||
|
||||
The room actor is a spawned task that manages the gossip KV subscription and coordinates all peer connections.
|
||||
|
||||
### State
|
||||
|
||||
```rust
|
||||
struct Actor {
|
||||
me: EndpointId,
|
||||
_gossip: Gossip,
|
||||
live: Live,
|
||||
active_subscribe: HashSet<BroadcastId>, // (EndpointId, name) pairs
|
||||
active_publish: HashSet<String>, // Locally published broadcast names
|
||||
known_peers: HashMap<EndpointId, Option<String>>, // display names
|
||||
connecting: ConnectingFutures, // In-flight subscribe attempts
|
||||
subscribe_closed: FuturesUnordered, // Track subscription lifetimes
|
||||
publish_closed: FuturesUnordered, // Track publish lifetimes
|
||||
chat_messages: FuturesUnordered, // Active chat subscribers
|
||||
chat_publisher: Option<ChatPublisher>,
|
||||
display_name: Option<String>,
|
||||
event_tx: mpsc::Sender<RoomEvent>,
|
||||
kv: iroh_smol_kv::Client, // Distributed KV for peer state
|
||||
kv_writer: WriteScope, // KV write access
|
||||
}
|
||||
```
|
||||
|
||||
### Gossip KV for Peer Discovery
|
||||
|
||||
The room uses `iroh-smol-kv` over gossip for peer state coordination. Each peer writes their `PeerState` to key `b"s"`:
|
||||
|
||||
```rust
|
||||
struct PeerState {
|
||||
broadcasts: Vec<String>,
|
||||
display_name: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
Serialized with postcard (binary format — **no `skip_serializing_if`** allowed since postcard is positional).
|
||||
|
||||
### Event Loop
|
||||
|
||||
```
|
||||
loop {
|
||||
select! {
|
||||
update = gossip_kv_stream.next() → handle_gossip_update
|
||||
msg = inbox.recv() → handle_api_message
|
||||
result = connecting.next() → subscribe succeeded/failed
|
||||
broadcast_closed → remove from active, maybe emit PeerLeft
|
||||
publish_closed → remove from active_publish, update KV
|
||||
chat_message → emit ChatReceived
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Peer Discovery Flow
|
||||
|
||||
1. Peer A publishes a broadcast via `handle.publish("camera", &broadcast)`
|
||||
2. Actor publishes to MoQ AND updates gossip KV with `PeerState { broadcasts: ["camera"], display_name: ... }`
|
||||
3. Peer B's gossip KV stream receives the update
|
||||
4. Peer B's actor checks `known_peers` — if new, emits `PeerJoined`
|
||||
5. Peer B's actor checks `active_subscribe` — if new broadcast, initiates `live.subscribe(remote, name)`
|
||||
6. When subscription succeeds, Peer B emits `BroadcastSubscribed`
|
||||
7. If the broadcast has a chat track, a chat subscriber is spawned
|
||||
|
||||
### Chat
|
||||
|
||||
Chat uses a dedicated MoQ track within each broadcast. Each message is a single MoQ group containing one frame of UTF-8 text. The sender identity comes from the broadcast context (peer ID), not the message payload.
|
||||
|
||||
### Connection Lifecycle
|
||||
|
||||
- When a broadcast closes (`subscribe_closed`), it's removed from `active_subscribe`
|
||||
- If this was the last broadcast from that peer, `PeerLeft` is emitted
|
||||
- When a publish closes (`publish_closed`), the KV is updated to remove that broadcast
|
||||
|
||||
### `RoomPublisherSync`
|
||||
|
||||
A convenience wrapper for the common pattern of publishing camera+audio and optionally screen share into a room:
|
||||
|
||||
```rust
|
||||
let publisher = RoomPublisherSync::new(room_handle, audio_backend);
|
||||
publisher.set_state(&PublishOpts::default())?;
|
||||
```
|
||||
|
||||
Automatically publishes a "camera" broadcast and manages a "screen" broadcast when screen sharing is toggled on.
|
||||
|
||||
## API Messages
|
||||
|
||||
```rust
|
||||
enum ApiMessage {
|
||||
Publish { name: String, producer: BroadcastProducer },
|
||||
SendChat { text: String },
|
||||
SetChatPublisher { publisher: ChatPublisher },
|
||||
SetDisplayName { name: String },
|
||||
}
|
||||
```
|
||||
|
||||
These are sent from `RoomHandle` to the actor via an mpsc channel.
|
||||
105
docs/research/references/iroh/iroh-live/05-relay.md
Normal file
105
docs/research/references/iroh/iroh-live/05-relay.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# iroh-live-relay: Browser Bridging
|
||||
|
||||
## Overview
|
||||
|
||||
The relay server bridges iroh P2P streams to browser clients via WebTransport. Browsers cannot speak iroh's QUIC protocol directly, so the relay accepts WebTransport connections and either serves locally-published broadcasts or pulls them from remote iroh publishers on demand.
|
||||
|
||||
**Architecture:**
|
||||
|
||||
```
|
||||
iroh-live publish --(iroh P2P)--> iroh-live-relay <--(WebTransport)-- browser
|
||||
browser --(WebTransport)--> iroh-live-relay --(iroh P2P)--> iroh-live subscribe
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### `RelayConfig` (CLI Configuration)
|
||||
|
||||
```rust
|
||||
pub struct RelayConfig {
|
||||
pub bind: SocketAddr, // QUIC/WebTransport bind (default: [::]:4443)
|
||||
pub http_bind: SocketAddr, // HTTP static files bind (default: same as bind)
|
||||
}
|
||||
```
|
||||
|
||||
Flattenable into a clap CLI via `#[command(flatten)]`.
|
||||
|
||||
### `run(config)` — Main Server Loop
|
||||
|
||||
The main entry point. Sets up:
|
||||
|
||||
1. **QUIC/WebTransport server** — Uses `moq-native::ServerConfig` with:
|
||||
- QUIC backend: `noq` (a custom QUIC implementation)
|
||||
- iroh endpoint integration
|
||||
- Self-signed TLS certificates (dev mode) for `localhost`
|
||||
- Max streams: `moq_relay::DEFAULT_MAX_STREAMS`
|
||||
|
||||
2. **iroh endpoint** — Binds an iroh endpoint for P2P connectivity, prints its ID
|
||||
|
||||
3. **moq-relay Cluster** — The broadcast routing engine. Manages broadcast lifecycle: when all subscribers disconnect, the broadcast is removed.
|
||||
|
||||
4. **HTTP server** — Axum router serving:
|
||||
- `GET /certificate.sha256` — TLS fingerprint for dev mode
|
||||
- `GET /` — Web viewer landing page
|
||||
- `GET /{path}` — Static file serving with CORS
|
||||
- Embedded via `include_dir!` from `web/dist/`
|
||||
|
||||
5. **Pull mode** — If iroh endpoint is available, creates a `PullState` for on-demand remote broadcast fetching
|
||||
|
||||
6. **Connection loop** — Accepts incoming connections, parses the URL path as a `LiveTicket`, and if valid, triggers a pull before running the connection
|
||||
|
||||
### `PullState` — On-Demand Remote Fetching
|
||||
|
||||
When a browser connects with a broadcast name that is a valid `LiveTicket`, the relay:
|
||||
|
||||
1. Checks if the broadcast already exists in the cluster (fast path)
|
||||
2. If not, connects to the remote publisher via iroh-live's `Moq::connect()`
|
||||
3. Subscribes to the remote broadcast
|
||||
4. Publishes the consumer into the local cluster under the ticket string as the name
|
||||
5. Spawns a keepalive task that holds the session until it closes
|
||||
|
||||
**Concurrency:** Duplicate concurrent pulls for the same ticket are deduplicated using a `HashMap<String, Arc<Notify>>`. Waiters block on the `Notify` until the first connector finishes.
|
||||
|
||||
```rust
|
||||
pub(crate) struct PullState {
|
||||
live: iroh_live::Live,
|
||||
cluster: Cluster,
|
||||
connecting: Arc<Mutex<HashMap<String, Arc<Notify>>>>>,
|
||||
}
|
||||
```
|
||||
|
||||
### Web Viewer
|
||||
|
||||
The relay embeds a SolidJS + TypeScript web application compiled by Vite. It uses:
|
||||
- `@moq/watch` — Web component for watching streams via WebCodecs
|
||||
- `@moq/publish` — Web component for publishing from browser camera/mic
|
||||
- WebTransport — For QUIC connectivity from the browser
|
||||
|
||||
Watch URLs: `https://relay:4443/?name=<BROADCAST_OR_TICKET>`
|
||||
|
||||
### Data Directory
|
||||
|
||||
The relay persists data to `$IROH_LIVE_RELAY_DATA` (or the platform default). This includes:
|
||||
- iroh secret key (`iroh_secret_key`) — ensures endpoint ID stability across restarts
|
||||
- TLS certificates
|
||||
|
||||
### TLS and Certificates
|
||||
|
||||
Currently **self-signed only**. ACME/Let's Encrypt is planned but not implemented. In dev mode, browsers need `--ignore-certificate-errors` or the relay's fingerprint (served at `/certificate.sha256`) for WebTransport to work.
|
||||
|
||||
## Error Handling
|
||||
|
||||
No authentication is implemented yet. The relay accepts all connections. MoQ supports token-based authentication which could be added.
|
||||
|
||||
## CLI Binary
|
||||
|
||||
```rust
|
||||
// iroh-live-relay/src/main.rs
|
||||
#[derive(Parser)]
|
||||
struct Cli {
|
||||
#[command(flatten)]
|
||||
relay: RelayConfig,
|
||||
}
|
||||
```
|
||||
|
||||
Must call `rustls::crypto::aws_lc_rs::default_provider().install_default()` before `run()`.
|
||||
@@ -0,0 +1,304 @@
|
||||
# moq-media: Media Pipelines
|
||||
|
||||
## Overview
|
||||
|
||||
`moq-media` owns the media pipeline: broadcast management, codec orchestration, playout timing, adaptive bitrate, and audio backend. **It has no dependency on iroh** — it works with any transport that implements `PacketSource` and `PacketSink`. This makes it usable for recording pipelines, studio links, and camera dashboards without RTC.
|
||||
|
||||
## Module Structure
|
||||
|
||||
```
|
||||
moq-media/
|
||||
├── lib.rs — Re-exports and feature-gated modules
|
||||
├── publish.rs — LocalBroadcast, VideoPublisher, AudioPublisher
|
||||
├── subscribe.rs — RemoteBroadcast, VideoTrack, AudioTrack, MediaTracks
|
||||
├── transport.rs — PacketSource/PacketSink traits, MoqPacketSource, MoqPacketSink
|
||||
├── net.rs — NetworkSignals (RTT, loss rate, available bandwidth)
|
||||
├── adaptive.rs — Adaptive rendition switching algorithm
|
||||
├── playout.rs — PlaybackPolicy, SyncMode
|
||||
├── chat.rs — ChatPublisher, ChatSubscriber (MoQ track-based)
|
||||
├── frame_channel.rs — Single-frame channel (last-writer-wins for video)
|
||||
├── sync.rs — Shared playout clock (Sync) for A/V sync
|
||||
├── stats.rs — Metric, Label, NetStats, EncodeStats, RenderStats, etc.
|
||||
├── pipeline.rs — Pipeline orchestration
|
||||
├── pipeline/ — VideoEncoderPipeline, AudioEncoderPipeline, VideoDecoderPipeline, etc.
|
||||
├── audio_backend.rs — AudioBackend trait and device enumeration
|
||||
├── audio_backend/ — Platform-specific audio backends (cpal, etc.)
|
||||
├── capture.rs — Camera/screen capture integration
|
||||
├── source_spec.rs — VideoInput, PreEncodedTrack
|
||||
├── test_util.rs — Test utilities (feature-gated)
|
||||
└── processing/ — Scale, color conversion, etc.
|
||||
```
|
||||
|
||||
## Publish Pipeline — `LocalBroadcast`
|
||||
|
||||
`LocalBroadcast` manages encoder pipelines and publishes a catalog that subscribers use to discover available renditions. It owns a `BroadcastProducer` (from moq-lite) and coordinates video and audio track lifecycles.
|
||||
|
||||
### Construction
|
||||
|
||||
```rust
|
||||
let broadcast = LocalBroadcast::new();
|
||||
broadcast.video().set_source(camera, VideoCodec::H264, [VideoPreset::P720])?;
|
||||
broadcast.audio().set(mic, AudioCodec::Opus, [AudioPreset::Hq])?;
|
||||
|
||||
// Or pre-encoded sources
|
||||
broadcast.video().set(VideoInput::pre_encoded("video/h264-pi", config, factory))?;
|
||||
```
|
||||
|
||||
### Slot Handles
|
||||
|
||||
- `broadcast.video()` → `VideoPublisher` (borrows `&self`)
|
||||
- `broadcast.audio()` → `AudioPublisher` (borrows `&self`)
|
||||
|
||||
Both use interior mutability. Calling `set()` tears down any existing pipeline and installs the new one.
|
||||
|
||||
### Video Input Modes
|
||||
|
||||
```rust
|
||||
pub enum VideoInput {
|
||||
Renditions(VideoRenditions), // Raw source → multiple encoded renditions (simulcast)
|
||||
PreEncoded(Vec<PreEncodedTrack>), // Already-encoded tracks pass through
|
||||
}
|
||||
```
|
||||
|
||||
**`VideoRenditions`** holds a `SharedVideoSource` and a map of rendition names to encoder factories. Multiple renditions share the same source via `watch::Receiver<Option<VideoFrame>>`. Slow encoders never cause backpressure on the source — intermediate frames are silently skipped.
|
||||
|
||||
**`PreEncodedTrack`** is for hardware encoders that produce compressed output directly (e.g., rpicam-vid on Raspberry Pi). Each track carries a name, `VideoConfig`, and a factory closure that creates a fresh source per subscriber.
|
||||
|
||||
### SharedVideoSource
|
||||
|
||||
Runs the capture source on a dedicated OS thread. Parks when no subscribers are connected (releasing camera/screen resources) and unparks when the first subscriber arrives. Uses `AtomicU32` subscriber counting with proper memory ordering (`AcqRel`/`Acquire`).
|
||||
|
||||
Frames are distributed via `watch::Sender<Option<VideoFrame>>` — always contains the latest frame, so slow encoders never block the source.
|
||||
|
||||
### Demand-Driven Track Startup
|
||||
|
||||
The broadcast's run loop (`LocalBroadcast::run_dynamic`) calls `producer.requested_track().await` to wait for subscriber demand. When a subscriber requests a specific rendition:
|
||||
|
||||
1. The loop looks up the rendition in the current `VideoInput` or `AudioRenditions`
|
||||
2. It starts the corresponding encoder pipeline on a dedicated OS thread
|
||||
3. When all subscribers disconnect (tracked via `track.unused().await`), the pipeline is stopped
|
||||
|
||||
This means encoder threads only run when someone is actually consuming.
|
||||
|
||||
### Catalog
|
||||
|
||||
`LocalBroadcast` maintains a catalog track (hang's built-in catalog mechanism) listing all available video and audio renditions with codec configuration, dimensions, and bitrate. Updated whenever video or audio is set/cleared.
|
||||
|
||||
Catalog format follows the `hang::catalog::Catalog` structure with `Video` and `Audio` entries, each containing a `BTreeMap<String, Config>` of rendition names to configurations.
|
||||
|
||||
### Encoder Pipeline Architecture
|
||||
|
||||
All encoder pipelines run on **dedicated OS threads** (`spawn_thread`), not tokio tasks. Codec operations are CPU-intensive and sometimes block on hardware (VAAPI, V4L2), so running on tokio tasks would starve other async work.
|
||||
|
||||
Communication with the async runtime:
|
||||
- **VideoEncoderPipeline**: reads `SharedVideoSource` via `watch::Receiver`, writes encoded frames to `MoqPacketSink`
|
||||
- **AudioEncoderPipeline**: reads from `AudioSource`, writes to `MoqPacketSink`
|
||||
- **PreEncodedVideoPipeline**: reads from `PreEncodedVideoSource`, writes to `MoqPacketSink`
|
||||
|
||||
### Chat
|
||||
|
||||
```rust
|
||||
let chat_publisher = broadcast.enable_chat()?;
|
||||
chat_publisher.send("Hello!")?;
|
||||
|
||||
// Subscriber side
|
||||
if let Some(chat_sub) = remote_broadcast.chat() {
|
||||
let msg = chat_sub.recv().await;
|
||||
}
|
||||
```
|
||||
|
||||
Each chat message is a single MoQ group with one frame of UTF-8 text. The track name is `"chat"` with priority 10.
|
||||
|
||||
## Subscribe Pipeline — `RemoteBroadcast`
|
||||
|
||||
`RemoteBroadcast` wraps a `BroadcastConsumer` and watches its catalog for available video and audio renditions. Created with a `BroadcastConsumer` and a `PlaybackPolicy`.
|
||||
|
||||
### Construction
|
||||
|
||||
```rust
|
||||
let broadcast = RemoteBroadcast::new("stream-name", consumer).await?;
|
||||
// Or with explicit policy
|
||||
let broadcast = RemoteBroadcast::with_playback_policy("stream", consumer, policy).await?;
|
||||
```
|
||||
|
||||
On construction, spawns a catalog-watching task that publishes snapshots via `Watchable<CatalogSnapshot>`.
|
||||
|
||||
### `CatalogSnapshot`
|
||||
|
||||
Point-in-time view of the broadcast's catalog. Derefs to `hang::Catalog`. Carries a sequence number for change detection.
|
||||
|
||||
```rust
|
||||
let catalog = broadcast.catalog();
|
||||
catalog.video_renditions() // Iterator of rendition names sorted by width
|
||||
catalog.audio_renditions() // Iterator of audio rendition names
|
||||
catalog.select_video_rendition(Quality::High)? // Best match for quality
|
||||
catalog.has_video()
|
||||
catalog.has_audio()
|
||||
catalog.has_chat()
|
||||
catalog.user() // User metadata from publisher
|
||||
```
|
||||
|
||||
### Rendition Selection
|
||||
|
||||
```rust
|
||||
pub enum Quality { Highest, High, Mid, Low }
|
||||
|
||||
pub struct VideoTarget {
|
||||
pub max_pixels: Option<u32>,
|
||||
pub max_bitrate_kbps: Option<u32>,
|
||||
pub rendition: Option<String>, // Pin to specific rendition
|
||||
}
|
||||
```
|
||||
|
||||
`Quality::High` → `max_pixels(1280*720)`, etc. If `rendition` is set, it takes priority.
|
||||
|
||||
### VideoTrack
|
||||
|
||||
Represents a decoded video stream from a remote broadcast. The decoder runs on a dedicated OS thread.
|
||||
|
||||
**Creation flow:**
|
||||
|
||||
1. Pick a rendition (via `VideoTarget` or explicit name)
|
||||
2. Create `TrackConsumer` from `BroadcastConsumer`, wrap in `OrderedConsumer` with `PlaybackPolicy::max_latency`
|
||||
3. Wrap in `MoqPacketSource`
|
||||
4. A `forward_packets` async task reads from `MoqPacketSource` → `mpsc` channel
|
||||
5. Decoder thread reads `mpsc` → decoder → output via `Sync` playout clock (or `FramePacer`)
|
||||
6. Output channel: `FrameReceiver<VideoFrame>` (latest-frame wins, suitable for rendering)
|
||||
|
||||
**Frame access:**
|
||||
- `track.try_recv()` — Returns latest frame, draining older buffered frames (for game loops)
|
||||
- `track.next_frame().await` — Async wait for next frame
|
||||
- `track.has_frame()` — Check without consuming
|
||||
|
||||
**Adaptive rendition switching:**
|
||||
```rust
|
||||
track.enable_adaptation(broadcast, signals, config, decode_config)?;
|
||||
track.disable_adaptation();
|
||||
track.is_adaptive();
|
||||
track.selected_rendition();
|
||||
track.set_rendition_mode(RenditionMode::Fixed("video/h264-360p".into()));
|
||||
track.set_rendition_mode(RenditionMode::Auto);
|
||||
track.rendition_watcher(); // Direct<String> watcher for rendition changes
|
||||
```
|
||||
|
||||
### AudioTrack
|
||||
|
||||
Same pattern as `VideoTrack` but sends decoded samples to an `AudioSink` (typically cpal + sonora). The audio decoder thread runs a 10ms tick loop.
|
||||
|
||||
### MediaTracks
|
||||
|
||||
Convenience struct combining `RemoteBroadcast` with optional `VideoTrack` and `AudioTrack`:
|
||||
|
||||
```rust
|
||||
pub struct MediaTracks {
|
||||
pub broadcast: RemoteBroadcast,
|
||||
pub video: Option<VideoTrack>,
|
||||
pub audio: Option<AudioTrack>,
|
||||
}
|
||||
```
|
||||
|
||||
### Lifecycle
|
||||
|
||||
Both `VideoTrack` and `AudioTrack` use drop-based cleanup. Dropping cancels the decoder thread (via `CancellationToken`) and the `forward_packets` task (via `AbortOnDropHandle`). The `OrderedConsumer` is dropped, signaling the transport that the track is no longer needed.
|
||||
|
||||
## Transport Abstraction — `PacketSource` / `PacketSink`
|
||||
|
||||
The transport boundary between moq-media and the network:
|
||||
|
||||
```rust
|
||||
pub trait PacketSource: Send + 'static {
|
||||
fn read(&mut self) -> impl Future<Output = Result<Option<MediaPacket>>> + Send;
|
||||
}
|
||||
|
||||
pub trait PacketSink: Send + 'static {
|
||||
fn write(&mut self, packet: EncodedFrame) -> Result<()>;
|
||||
fn finish(&mut self) -> Result<()>;
|
||||
}
|
||||
```
|
||||
|
||||
**`MoqPacketSink`** wraps an `OrderedProducer`. When it receives an `EncodedFrame` with `is_keyframe = true`, it calls `keyframe()` on the producer to start a new MoQ group. This keyframe-to-group mapping is how subscribers can join at any group boundary.
|
||||
|
||||
**`MoqPacketSource`** wraps an `OrderedConsumer` and reads frames, converting them to `MediaPacket`.
|
||||
|
||||
**`PipeSink` / `PipeSource`** — In-memory pipe for local encode→decode without network (testing, local preview).
|
||||
|
||||
## Adaptive Rendition Switching
|
||||
|
||||
The adaptation algorithm runs in a background task that monitors `NetworkSignals` and decides whether to switch to a different video rendition.
|
||||
|
||||
### Algorithm
|
||||
|
||||
Renditions are ranked by pixel count (highest first). The algorithm maintains state across ticks:
|
||||
|
||||
```rust
|
||||
pub enum Decision {
|
||||
Hold, // Stay on current rendition
|
||||
Downgrade(usize), // Switch to lower at index
|
||||
Emergency, // Drop to lowest immediately
|
||||
StartProbe(usize), // Try upgrading to index
|
||||
}
|
||||
```
|
||||
|
||||
**Emergency** (immediate): Loss rate ≥ 20% → drop to lowest rendition
|
||||
|
||||
**Downgrade** (sustained 500ms): Loss rate ≥ 10% OR available bandwidth < 85% of current rendition's bitrate
|
||||
|
||||
**Upgrade probe** (sustained 4s good conditions): Loss ≤ 2%, bandwidth ≥ 120% of next-higher rendition's bitrate → start 3-second probe on the higher rendition
|
||||
|
||||
**Probe abort**: Loss ≥ 5% or new congestion events during probe → abort, 8s cooldown
|
||||
|
||||
**Post-downgrade cooldown**: 4s after any downgrade before probes are allowed
|
||||
|
||||
### Implementation
|
||||
|
||||
The adaptation task (`adaptation_task_v2`) creates new `VideoDecoderPipeline`s that write to the same `FrameSender` via `with_sender()`. The frame channel stays the same while the underlying decoder pipeline gets swapped. When switching:
|
||||
|
||||
1. Create a new decoder pipeline for the target rendition
|
||||
2. Drop the old pipeline handle
|
||||
3. Update `selected_rendition` Watchable
|
||||
|
||||
## Playback and Sync
|
||||
|
||||
### PlaybackPolicy
|
||||
|
||||
```rust
|
||||
pub struct PlaybackPolicy {
|
||||
pub sync: SyncMode, // Synced (shared clock) or Unmanaged (PTS pacing)
|
||||
pub max_latency: Duration, // Default: 150ms — how much buffering before skipping forward
|
||||
}
|
||||
```
|
||||
|
||||
### SyncMode
|
||||
|
||||
- **`Synced`** (default): Shared playout clock (`Sync`). Video frames are gated by `Sync::wait(pts)`, which blocks until `reference + pts + latency` arrives. Audio paces itself through its ring buffer (~80ms).
|
||||
- **`Unmanaged`**: No synchronization. `FramePacer` sleeps between frames based on PTS deltas, clamped to 2× frame period.
|
||||
|
||||
### Sync
|
||||
|
||||
The `Sync` type records arrival offsets via `received(pts)` and blocks on `wait(pts)` until `reference + pts + latency`. This keeps audio and video aligned without cross-path gating or signaling. Ported from the moq/js implementation.
|
||||
|
||||
## Stats
|
||||
|
||||
moq-media has a structured stats system for debug overlays:
|
||||
|
||||
- **`NetStats`** — RTT, loss%, bandwidth, path type (written by iroh-live transport bridge)
|
||||
- **`EncodeStats`** — FPS, encode time, bitrate, codec, encoder, resolution, capture path
|
||||
- **`RenderStats`** — FPS, decode time, decoder, renderer, rendition
|
||||
- **`TimingStats`** — Audio buffer level, video/audio lag, A/V delta, video buffer depth
|
||||
- **`Timeline`** — Ring buffer of `FrameMeta` entries for timeline visualization
|
||||
|
||||
Each `Metric` has EMA smoothing, a history ring buffer, and optional color thresholds. `Label` provides atomic string values.
|
||||
|
||||
## Codec Support
|
||||
|
||||
Feature-gated codec support:
|
||||
|
||||
| Feature | Codec | Backend |
|
||||
|---------|-------|---------|
|
||||
| `h264` | H.264 | openh264 (software) |
|
||||
| `av1` | AV1 | rav1e encoder, rav1d decoder |
|
||||
| `opus` | Opus | opus crate |
|
||||
| `vaapi` | VAAPI | Linux hardware encode/decode |
|
||||
| `videotoolbox` | VideoToolbox | macOS hardware |
|
||||
| `v4l2` | V4L2 | Raspberry Pi hardware |
|
||||
| `pcm` | Raw PCM | No encoding |
|
||||
@@ -0,0 +1,95 @@
|
||||
# iroh-live: Network Signals and Adaptive Bitrate
|
||||
|
||||
## NetworkSignals
|
||||
|
||||
Produced by polling iroh QUIC connection stats. Consumed by `VideoTrack::enable_adaptation()` to decide when to switch video renditions.
|
||||
|
||||
```rust
|
||||
pub struct NetworkSignals {
|
||||
pub rtt: Duration, // Round-trip time to remote peer
|
||||
pub loss_rate: f64, // Recent packet loss rate (0.0..=1.0), 200ms delta window
|
||||
pub available_bps: u64, // Estimated available bandwidth (cwnd * 8 / rtt)
|
||||
pub congestion_events: u64, // Monotonically increasing congestion counter
|
||||
}
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
`spawn_signal_producer()` in `iroh-live/src/util.rs` polls every 200ms:
|
||||
|
||||
1. Gets connection paths via `conn.paths().get()`
|
||||
2. Finds the selected path (`is_selected()`)
|
||||
3. Reads path stats (`lost_packets`, `udp_tx.datagrams`, `cwnd`) and RTT
|
||||
4. Computes delta-based loss rate: `delta_lost / (delta_sent + delta_lost)`
|
||||
5. Estimates bandwidth: `cwnd * 8 * 1e9 / rtt_ns`
|
||||
6. Writes to `watch::Sender<NetworkSignals>`
|
||||
|
||||
Also: `spawn_stats_recorder()` records into `NetStats` for the debug overlay (RTT, loss%, bandwidth in/out, path type).
|
||||
|
||||
## Adaptive Rendition Algorithm
|
||||
|
||||
Located in `moq-media/src/adaptive.rs`. The algorithm evaluates `NetworkSignals` against configured thresholds and produces `Decision` values.
|
||||
|
||||
### Configuration (`AdaptiveConfig`)
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `upgrade_hold` | 4s | Sustained good conditions before upgrade probe |
|
||||
| `downgrade_hold` | 500ms | Sustained bad conditions before downgrade |
|
||||
| `probe_duration` | 3s | How long a probe runs before committing |
|
||||
| `probe_cooldown` | 8s | Cooldown after a failed probe |
|
||||
| `post_downgrade_cooldown` | 4s | Cooldown after any downgrade |
|
||||
| `loss_downgrade` | 10% | Loss rate threshold for downgrade |
|
||||
| `loss_emergency` | 20% | Loss rate for immediate drop to lowest |
|
||||
| `loss_good` | 2% | Loss rate considered "good" |
|
||||
| `loss_probe_abort` | 5% | Loss rate that aborts an active probe |
|
||||
| `bw_downgrade_ratio` | 85% | Bandwidth utilization ceiling for downgrade |
|
||||
| `bw_probe_headroom` | 120% | Required excess bandwidth for probe |
|
||||
| `check_interval` | 200ms | How often adaptation task checks signals |
|
||||
|
||||
### Decision Logic
|
||||
|
||||
```
|
||||
1. Emergency: loss >= 20% AND not already lowest → Drop to lowest immediately
|
||||
|
||||
2. Downgrade check:
|
||||
- bandwidth_stressed (available < current_bitrate * 85%) OR loss >= 10%
|
||||
- sustained for downgrade_hold (500ms) → Downgrade(next_lower)
|
||||
|
||||
3. Upgrade check:
|
||||
- Already at highest → Hold
|
||||
- Within post_downgrade_cooldown (4s) → Hold
|
||||
- Within probe_cooldown (8s) → Hold
|
||||
- bandwidth_headroom (available >= next_higher_bitrate * 120%) AND loss <= 2%
|
||||
- sustained for upgrade_hold (4s) → StartProbe(next_higher)
|
||||
|
||||
4. Otherwise: Hold
|
||||
```
|
||||
|
||||
### Probe Lifecycle
|
||||
|
||||
When `StartProbe(idx)` is decided:
|
||||
1. Create a new decoder pipeline for the higher rendition
|
||||
2. Write frames to the same `FrameSender` (seamless switch for the consumer)
|
||||
3. Monitor signals during the probe period
|
||||
4. If `should_abort_probe()` (loss ≥ 5% or new congestion events) → abort, drop probe pipeline, cooldown 8s
|
||||
5. If probe duration (3s) passes without abort → commit, replace current pipeline
|
||||
|
||||
### Rendition Ranking
|
||||
|
||||
```rust
|
||||
pub fn rank_renditions(renditions: &BTreeMap<String, VideoConfig>) -> Vec<RankedRendition>
|
||||
```
|
||||
|
||||
Sorts by pixel count descending (highest quality = index 0). Each `RankedRendition` carries name, pixels, bitrate_bps, width, height.
|
||||
|
||||
### RenditionMode
|
||||
|
||||
```rust
|
||||
pub enum RenditionMode {
|
||||
Auto, // Algorithm-driven switching
|
||||
Fixed(String), // Pin to a specific rendition
|
||||
}
|
||||
```
|
||||
|
||||
Controlled via `VideoTrack::set_rendition_mode()`. In Fixed mode, the algorithm switches directly to the named rendition without probing.
|
||||
85
docs/research/references/iroh/iroh-live/08-p2p-and-relay.md
Normal file
85
docs/research/references/iroh/iroh-live/08-p2p-and-relay.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# iroh-live: P2P Connectivity and Relay Architecture
|
||||
|
||||
## Direct Connectivity
|
||||
|
||||
iroh connects peers directly when possible:
|
||||
|
||||
- **Same LAN:** Communicates over the local network without traffic leaving the subnet
|
||||
- **Public IP / simple NAT:** iroh's hole-punching establishes a direct UDP path
|
||||
- **Symmetric NAT / corporate firewalls / CGNAT:** Falls back to iroh relay network
|
||||
|
||||
The iroh endpoint exposes path statistics via `conn.paths()`, which returns a `Watcher<PathInfoList>`. Each `PathInfo` reports RTT, whether the path is selected, and the remote address. The selected path is the one actively carrying traffic; iroh may maintain multiple candidate paths and switch between them.
|
||||
|
||||
The transition between direct and relayed paths is transparent to the application. The media pipeline sees only changes in RTT and bandwidth, which adaptive rendition switching handles automatically.
|
||||
|
||||
## iroh-live-relay: Architecture
|
||||
|
||||
The relay serves two transport protocols simultaneously:
|
||||
|
||||
```
|
||||
iroh P2P publisher ──(QUIC, moq-lite-03)──> iroh-live-relay <──(WebTransport/H3, noq)── browser
|
||||
```
|
||||
|
||||
Both protocols feed into `moq-relay`'s shared `Origin`, which manages broadcast routing. A broadcast published via iroh is automatically available to WebTransport subscribers, and vice versa.
|
||||
|
||||
### Pull Model
|
||||
|
||||
The relay operates in **pull mode**: it connects to iroh publishers on demand when a browser client requests a broadcast. The broadcast name in the URL can be a `LiveTicket` URI. Multiple browser clients watching the same broadcast share a single upstream iroh connection.
|
||||
|
||||
Pull flow:
|
||||
1. Browser connects via WebTransport, requests broadcast by name (or ticket)
|
||||
2. Relay checks if broadcast already exists in local cluster → fast path
|
||||
3. If not, relay uses iroh-live `Moq::connect()` to connect to the remote publisher
|
||||
4. Subscribes to the broadcast via `session.subscribe(broadcast_name)`
|
||||
5. Publishes the consumer into the local cluster under the ticket string as the name
|
||||
6. Spawns a keepalive task holding the session until it closes
|
||||
7. Browser receives the stream through the relay's WebTransport frontend
|
||||
|
||||
### Connection Deduplication
|
||||
|
||||
`PullState` uses a `HashMap<String, Arc<Notify>>` to prevent duplicate concurrent connections to the same remote. If a pull is already in progress for a given ticket, subsequent requests wait on the `Notify` and then check if the broadcast appeared in the cluster.
|
||||
|
||||
### QUIC Backend: noq
|
||||
|
||||
The relay uses `noq` as its QUIC backend (not quinn). This is configured via:
|
||||
|
||||
```rust
|
||||
server_config.backend = Some(moq_native::QuicBackend::Noq);
|
||||
```
|
||||
|
||||
### iroh Endpoint Integration
|
||||
|
||||
The relay also binds an iroh endpoint:
|
||||
|
||||
```rust
|
||||
let mut iroh_config = moq_native::IrohEndpointConfig::default();
|
||||
iroh_config.enabled = Some(true);
|
||||
iroh_config.secret = Some(relay.iroh_secret_path_str());
|
||||
let iroh = iroh_config.bind().await?;
|
||||
```
|
||||
|
||||
This enables the relay to participate in the iroh P2P network directly.
|
||||
|
||||
## Ticket Format
|
||||
|
||||
`LiveTicket` serves as the connection mechanism for both P2P and relay scenarios:
|
||||
|
||||
- **P2P:** Subscriber uses the `EndpointAddr` (node ID + relay URLs) to connect directly
|
||||
- **Relay:** The full ticket string becomes the broadcast name in the URL: `https://relay:4443/?name=iroh-live:...`
|
||||
|
||||
The ticket format: `iroh-live:<base64url(postcard(EndpointAddr))>/<broadcast_name>`
|
||||
|
||||
It also supports a legacy format: `<name>@<base32(postcard(EndpointAddr))>`
|
||||
|
||||
## Connection Access in iroh-moq
|
||||
|
||||
`MoqSession::conn()` returns a reference to the underlying iroh `Connection`. This is used by:
|
||||
|
||||
1. **Signal producer** — Polls path stats for `NetworkSignals`
|
||||
2. **Stats recorder** — Records into `NetStats` for debug overlays
|
||||
3. **Call::closed()** — Inspects QUIC close reason to determine `DisconnectReason`
|
||||
|
||||
The connection provides:
|
||||
- `paths().get()` — List of active network paths with RTT, stats, relay status
|
||||
- `close_reason()` — Why the connection closed (LocallyClosed, ApplicationClosed, ConnectionClosed, Reset)
|
||||
- `remote_id()` — Remote peer's endpoint ID
|
||||
42
docs/research/references/iroh/iroh-live/README.md
Normal file
42
docs/research/references/iroh/iroh-live/README.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# iroh-live Reference Documentation
|
||||
|
||||
> **Status:** Early tech preview. APIs are unstable. Based on source code analysis of the iroh-live workspace.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Topic |
|
||||
|------|-------|
|
||||
| [01-overview-and-architecture](01-overview-and-architecture.md) | Workspace structure, crate layers, design principles, data flow, dependencies |
|
||||
| [02-core-api](02-core-api.md) | `Live`, `LiveTicket`, `Call`, `Subscription`, `DisconnectReason`, `util` module |
|
||||
| [03-iroh-moq-transport](03-iroh-moq-transport.md) | `Moq`, `MoqSession`, `MoqProtocolHandler`, actor internals, session lifecycle, error types |
|
||||
| [04-rooms](04-rooms.md) | `Room`, `RoomHandle`, `RoomTicket`, `RoomEvent`, gossip KV coordination, actor architecture |
|
||||
| [05-relay](05-relay.md) | `iroh-live-relay`: browser bridging, pull model, `RelayConfig`, `PullState`, web viewer |
|
||||
| [06-moq-media-pipelines](06-moq-media-pipelines.md) | `LocalBroadcast`, `RemoteBroadcast`, `VideoTrack`, `AudioTrack`, transport abstraction, codec support |
|
||||
| [07-network-signals-and-adaptive-bitrate](07-network-signals-and-adaptive-bitrate.md) | `NetworkSignals`, adaptation algorithm, `AdaptiveConfig`, `Decision`, probe lifecycle |
|
||||
| [08-p2p-and-relay](08-p2p-and-relay.md) | iroh P2P connectivity, relay architecture, pull model, ticket format, connection access |
|
||||
|
||||
## Quick Navigation
|
||||
|
||||
### "How do I..."
|
||||
|
||||
- **Publish a stream?** → [02-core-api](02-core-api.md) (`Live::publish`) + [06-moq-media-pipelines](06-moq-media-pipelines.md) (`LocalBroadcast`)
|
||||
- **Subscribe to a stream?** → [02-core-api](02-core-api.md) (`Live::subscribe`) + [06-moq-media-pipelines](06-moq-media-pipelines.md) (`RemoteBroadcast`)
|
||||
- **Make a 1:1 call?** → [02-core-api](02-core-api.md) (`Call::dial` / `Call::accept`)
|
||||
- **Create a multi-party room?** → [04-rooms](04-rooms.md) (`Room::new`, `RoomTicket`)
|
||||
- **Bridge to browsers?** → [05-relay](05-relay.md) (`iroh-live-relay`)
|
||||
- **Adapt quality to network conditions?** → [07-network-signals-and-adaptive-bitrate](07-network-signals-and-adaptive-bitrate.md)
|
||||
- **Understand the MoQ transport?** → [03-iroh-moq-transport](03-iroh-moq-transport.md)
|
||||
- **Understand the media pipeline?** → [06-moq-media-pipelines](06-moq-media-pipelines.md)
|
||||
|
||||
### Key Source Files
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| iroh-live crate | `iroh-live/src/{lib, live, call, subscription, ticket, types, util, rooms}.rs` |
|
||||
| iroh-moq crate | `iroh-moq/src/lib.rs` |
|
||||
| iroh-live-relay | `iroh-live-relay/src/{lib, main, pull}.rs` |
|
||||
| moq-media publish | `moq-media/src/publish.rs` |
|
||||
| moq-media subscribe | `moq-media/src/subscribe.rs` |
|
||||
| moq-media adaptive | `moq-media/src/adaptive.rs` |
|
||||
| moq-media transport | `moq-media/src/transport.rs` |
|
||||
| moq-media network signals | `moq-media/src/net.rs` |
|
||||
Reference in New Issue
Block a user