docs(research): add iroh suite deep-dive references for iroh, irpc, iroh-blobs, iroh-gossip, iroh-live, and iroh-docs
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# iroh-gossip: Overview & Architecture
|
||||
|
||||
## What Is iroh-gossip?
|
||||
|
||||
`iroh-gossip` is a Rust crate that implements an **epidemic broadcast tree** protocol for disseminating messages among a swarm of peers interested in a common **topic**. It is based on two academic papers:
|
||||
|
||||
- **HyParView** — A hybrid partial view membership protocol for reliable swarm management ([paper](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf))
|
||||
- **PlumTree** — An epidemic broadcast tree protocol for efficient message dissemination ([paper](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf))
|
||||
|
||||
The crate is designed as a protocol layer for the [iroh](https://docs.rs/iroh) networking library, but the core protocol logic is **IO-free** and can be used independently.
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
The crate is organized into two primary modules:
|
||||
|
||||
| Module | Purpose | IO-aware? |
|
||||
|--------|---------|-----------|
|
||||
| `proto` | Pure state-machine implementation of the gossip protocol | No — completely IO-free |
|
||||
| `net` | Networking layer that runs the protocol over iroh connections | Yes — depends on `iroh` and tokio |
|
||||
|
||||
The `net` module is behind the `net` feature flag (enabled by default). An optional `rpc` feature adds remote procedure call support via the `irpc`/`noq` crates.
|
||||
|
||||
### Module Dependency Graph
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ api │ ← Public API (Gossip, GossipTopic, GossipSender, GossipReceiver)
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ net │ ← Networking actor, connection loops, dialer
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ proto │ ← Pure protocol state machines
|
||||
│ ┌─────────┐ │
|
||||
│ │hyparview│ │ ← Membership layer
|
||||
│ ├─────────┤ │
|
||||
│ │ plumtree│ │ ← Broadcast layer
|
||||
│ ├─────────┤ │
|
||||
│ │ topic │ │ ← Per-topic coordinator
|
||||
│ ├─────────┤ │
|
||||
│ │ state │ │ ← Multi-topic state manager
|
||||
│ ├─────────┤ │
|
||||
│ │ util │ │ ← Shared data structures (IndexSet, TimeBoundCache, TimerMap)
|
||||
│ └─────────┘ │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
### Key Design Principles
|
||||
|
||||
1. **IO-free protocol core**: The `proto` module is a pure state machine. It takes `InEvent`s, produces `OutEvent`s, and has no knowledge of sockets, async runtimes, or network IO.
|
||||
|
||||
2. **Topic-based isolation**: Each topic (`TopicId` = 32-byte identifier) has completely independent state. Topics are separate swarms and broadcast scopes. Joining multiple topics increases connections and routing table size proportionally.
|
||||
|
||||
3. **Actor model for networking**: The `net` module runs a single async `Actor` that manages all topics, connections, and timers. It bridges between the protocol state machine and real network IO.
|
||||
|
||||
4. **Wire protocol**: Messages are serialized with `postcard` (a `no_std`-friendly serde format) and sent over QUIC streams via iroh connections. Each stream is prefixed with a `StreamHeader` containing the topic ID.
|
||||
|
||||
## Crate Features
|
||||
|
||||
| Feature | Default? | Description |
|
||||
|---------|----------|-------------|
|
||||
| `net` | Yes | Networking layer (requires `iroh`, `tokio`, etc.) |
|
||||
| `rpc` | No | RPC support via `irpc`/`noq` for remote control |
|
||||
| `metrics` | Yes | Prometheus-style metrics via `iroh-metrics` |
|
||||
| `test-utils` | No | Test utilities (seeded RNG, etc.) |
|
||||
| `simulator` | No | CLI simulator for testing |
|
||||
| `examples` | No | Example binaries (chat, setup) |
|
||||
|
||||
## Cargo Dependencies (Key Ones)
|
||||
|
||||
- `iroh` / `iroh-base` — Networking primitives (Endpoint, EndpointId, PublicKey, etc.)
|
||||
- `postcard` — Wire serialization (serde-based, `no_std` compatible)
|
||||
- `blake3` — Message ID hashing
|
||||
- `ed25519-dalek` — Cryptographic signatures
|
||||
- `n0-future` / `n0-error` — Async utilities and error handling
|
||||
- `irpc` / `noq` — RPC infrastructure (optional)
|
||||
- `indexmap` — Order-preserving hash collections used in `IndexSet`
|
||||
@@ -0,0 +1,169 @@
|
||||
# iroh-gossip: HyParView Membership Protocol
|
||||
|
||||
## Overview
|
||||
|
||||
The HyParView protocol provides **swarm membership management** — it maintains which peers are currently part of the swarm for a given topic and ensures the overlay network remains connected even as nodes join, leave, or fail.
|
||||
|
||||
It is implemented in `src/proto/hyparview.rs`.
|
||||
|
||||
## Core Concept: Two Views
|
||||
|
||||
Each peer maintains two sets of peers:
|
||||
|
||||
| View | Description | Default Size | Connection? |
|
||||
|------|-------------|--------------|-------------|
|
||||
| **Active View** | Peers we maintain active bidirectional connections to | 5 | Yes — TCP/QUIC connection is kept open |
|
||||
| **Passive View** | An address book of peers we know about but are not connected to | 30 | No — just contact information |
|
||||
|
||||
Key invariants:
|
||||
- **Active connections are always bidirectional**: If peer A has peer B in its active view, peer B also has peer A in its active view.
|
||||
- The passive view serves as a **failover pool**: When an active peer disconnects, a random peer from the passive view is promoted to fill the slot.
|
||||
|
||||
## Configuration (`hyparview::Config`)
|
||||
|
||||
```rust
|
||||
pub struct Config {
|
||||
pub active_view_capacity: usize, // Default: 5
|
||||
pub passive_view_capacity: usize, // Default: 30
|
||||
pub active_random_walk_length: Ttl, // Default: Ttl(6)
|
||||
pub passive_random_walk_length: Ttl, // Default: Ttl(3)
|
||||
pub shuffle_random_walk_length: Ttl, // Default: Ttl(6)
|
||||
pub shuffle_active_view_count: usize, // Default: 3
|
||||
pub shuffle_passive_view_count: usize, // Default: 4
|
||||
pub shuffle_interval: Duration, // Default: 60s
|
||||
pub neighbor_request_timeout: Duration, // Default: 500ms
|
||||
}
|
||||
```
|
||||
|
||||
These defaults come directly from the HyParView paper (p9), except for `shuffle_interval` and `neighbor_request_timeout` which are "wild guesses" in the code.
|
||||
|
||||
## State Structure
|
||||
|
||||
```rust
|
||||
pub struct State<PI, RG = ThreadRng> {
|
||||
me: PI, // Our peer identity
|
||||
me_data: Option<PeerData>, // Opaque data we share with peers
|
||||
pub active_view: IndexSet<PI>, // Connected peers
|
||||
pub passive_view: IndexSet<PI>, // Known but disconnected peers
|
||||
config: Config,
|
||||
shuffle_scheduled: bool, // Whether shuffle timer is active
|
||||
rng: RG, // Random number generator
|
||||
stats: Stats,
|
||||
pending_neighbor_requests: HashSet<PI>, // Peers we've sent Neighbor to but no reply yet
|
||||
peer_data: HashMap<PI, PeerData>, // Opaque data received from other peers
|
||||
alive_disconnect_peers: HashSet<PI>, // Peers disconnecting but to keep in passive view
|
||||
}
|
||||
```
|
||||
|
||||
## Messages (`hyparview::Message`)
|
||||
|
||||
| Message | Direction | Purpose |
|
||||
|---------|-----------|---------|
|
||||
| `Join(Option<PeerData>)` | New node → Contact | Sent to a known peer to join the swarm |
|
||||
| `ForwardJoin(ForwardJoin)` | Propagated | Forwarded to active view to introduce a new member |
|
||||
| `Neighbor(Neighbor)` | Bidirectional | Request to add sender to active view (with priority) |
|
||||
| `Disconnect(Disconnect)` | Bidirectional | Notification that a peer is leaving or being demoted |
|
||||
| `Shuffle(Shuffle)` | Initiated periodically | Sent to random peer to exchange passive view contacts |
|
||||
| `ShuffleReply(ShuffleReply)` | Reply to Shuffle | Returns a random subset of our views to the origin |
|
||||
|
||||
### Message Details
|
||||
|
||||
```rust
|
||||
pub struct ForwardJoin<PI> {
|
||||
peer: PeerInfo<PI>, // The new peer's identity + optional data
|
||||
ttl: Ttl, // Time-to-live, decremented per hop
|
||||
}
|
||||
|
||||
pub struct Shuffle<PI> {
|
||||
origin: PI, // Who initiated the shuffle
|
||||
nodes: Vec<PeerInfo<PI>>, // Random subset of our views
|
||||
ttl: Ttl, // Time-to-live for the random walk
|
||||
}
|
||||
|
||||
pub struct Neighbor {
|
||||
priority: Priority, // High (cannot be denied) or Low (can be denied)
|
||||
data: Option<PeerData>,
|
||||
}
|
||||
|
||||
pub struct Disconnect {
|
||||
alive: bool, // If true, peer is still alive (just demoting)
|
||||
_respond: bool, // Obsolete, kept for wire compat
|
||||
}
|
||||
```
|
||||
|
||||
## Join Procedure (Step by Step)
|
||||
|
||||
1. A new node sends `Join(me_data)` to a known contact peer.
|
||||
2. The contact peer adds the new node to its active view (even evicting a random peer if necessary).
|
||||
3. The contact peer forwards `ForwardJoin` to all other peers in its active view with `TTL = active_random_walk_length`.
|
||||
4. Each peer receiving `ForwardJoin`:
|
||||
- If `TTL == 0` or active view has ≤1 peer: sends `Neighbor(High)` to the new node (which adds it to active view).
|
||||
- If `TTL == passive_random_walk_length`: adds the new node to passive view.
|
||||
- Decrements TTL and forwards to a random active peer (different from sender).
|
||||
|
||||
5. The `Neighbor` message establishes the bidirectional active connection. A `Priority::High` neighbor request **must** be accepted (potentially evicting a random active peer). A `Priority::Low` request is only accepted if there is room.
|
||||
|
||||
## Shuffle Mechanism
|
||||
|
||||
Periodically (every `shuffle_interval`), each node:
|
||||
1. Picks a random active peer.
|
||||
2. Sends `Shuffle` containing a random subset of active + passive views plus the origin's info, with a TTL.
|
||||
3. The shuffle message does a random walk (each hop decrements TTL).
|
||||
4. When TTL reaches 0 or the active view is ≤1, the peer accepts the shuffle and replies with `ShuffleReply` containing its own random peers.
|
||||
5. The origin receives `ShuffleReply` and adds new peers to its passive view.
|
||||
|
||||
This ensures the passive view remains fresh and provides good connectivity even in dynamic networks.
|
||||
|
||||
## Failure Recovery
|
||||
|
||||
When a peer in the active view disconnects (detected via `PeerDisconnected`):
|
||||
1. The peer is removed from the active view.
|
||||
2. A `NeighborDown` event is emitted.
|
||||
3. A random peer from the passive view is selected and sent a `Neighbor(Low)` request.
|
||||
4. If that peer doesn't respond within `neighbor_request_timeout`, it's removed from the passive view and another peer is tried.
|
||||
5. This continues until a connection is established or the passive view is exhausted.
|
||||
|
||||
If a `Disconnect(alive=true)` message is received:
|
||||
- The peer is moved to the passive view (not just dropped), because it's still alive.
|
||||
- The `alive_disconnect_peers` set tracks which disconnected peers should be retained in passive view when their connection eventually closes.
|
||||
|
||||
## PeerData
|
||||
|
||||
`PeerData` is an opaque `Bytes` type that peers exchange when joining. In the `net` module, it is used to serialize and transmit addressing information (`AddrInfo`):
|
||||
|
||||
```rust
|
||||
struct AddrInfo {
|
||||
relay_url: Option<RelayUrl>,
|
||||
direct_addresses: BTreeSet<SocketAddr>,
|
||||
}
|
||||
```
|
||||
|
||||
This allows the gossip protocol itself to help propagate connectivity information, enabling the `GossipAddressLookup` service to feed addresses back into iroh's endpoint discovery system.
|
||||
|
||||
## Events (`hyparview::Event`)
|
||||
|
||||
| Event | Meaning |
|
||||
|-------|---------|
|
||||
| `NeighborUp(PI)` | A peer was added to our active view |
|
||||
| `NeighborDown(PI)` | A peer was removed from our active view |
|
||||
|
||||
These events are forwarded up to the PlumTree layer and to the application.
|
||||
|
||||
## Timers
|
||||
|
||||
| Timer | Purpose |
|
||||
|-------|---------|
|
||||
| `DoShuffle` | Periodically trigger a shuffle operation |
|
||||
| `PendingNeighborRequest(PI)` | Timeout for a pending neighbor request |
|
||||
|
||||
## IO Trait Pattern
|
||||
|
||||
The HyParView state machine is generic over an `IO` trait:
|
||||
|
||||
```rust
|
||||
pub trait IO<PI: Clone> {
|
||||
fn push(&mut self, event: impl Into<OutEvent<PI>>);
|
||||
}
|
||||
```
|
||||
|
||||
This allows the protocol to emit output events without knowing about the networking layer. The upper layers supply a `VecDeque<OutEvent>` or similar container.
|
||||
@@ -0,0 +1,256 @@
|
||||
# iroh-gossip: PlumTree Broadcast Protocol
|
||||
|
||||
## Overview
|
||||
|
||||
The PlumTree (Epidemic Broadcast Trees) protocol provides **efficient message broadcasting** across all peers in a topic's swarm. It builds on top of HyParView's membership layer, using the active view as its peer set.
|
||||
|
||||
It is implemented in `src/proto/plumtree.rs`.
|
||||
|
||||
## Core Concept: Eager vs Lazy Push
|
||||
|
||||
Each peer maintains two subsets of its HyParView active view:
|
||||
|
||||
| Set | Description | Behavior |
|
||||
|-----|-------------|----------|
|
||||
| **Eager push peers** | Peers to whom full messages are sent immediately | Messages are pushed eagerly (full content) |
|
||||
| **Lazy push peers** | Peers to whom only message IDs (hashes) are sent | `IHave` announcements are sent, requesting content only if needed |
|
||||
|
||||
When a peer broadcasts a message:
|
||||
1. The **full message** is pushed to all **eager** peers.
|
||||
2. The **message ID** (a blake3 hash) is pushed to all **lazy** peers (after a short delay for batching).
|
||||
|
||||
This creates an **optimized broadcast tree**: eager peers form a spanning tree for low-latency delivery, while lazy peers provide redundancy through timeout-based recovery.
|
||||
|
||||
## Configuration (`plumtree::Config`)
|
||||
|
||||
```rust
|
||||
pub struct Config {
|
||||
pub graft_timeout_1: Duration, // Default: 80ms
|
||||
pub graft_timeout_2: Duration, // Default: 40ms
|
||||
pub dispatch_timeout: Duration, // Default: 5ms
|
||||
pub optimization_threshold: Round, // Default: Round(7)
|
||||
pub message_cache_retention: Duration, // Default: 30s
|
||||
pub message_id_retention: Duration, // Default: 90s
|
||||
pub cache_evict_interval: Duration, // Default: 1s
|
||||
}
|
||||
```
|
||||
|
||||
### Timeout Semantics
|
||||
|
||||
- **`graft_timeout_1`**: After receiving an `IHave`, wait this long for the full message from an eager peer. If it doesn't arrive, send a `Graft` to the `IHave` sender.
|
||||
- **`graft_timeout_2`**: After sending a `Graft`, wait this shorter timeout for the reply. If no reply, try the next `IHave` sender.
|
||||
- **`dispatch_timeout`**: Delay before batching and sending `IHave` messages. This allows multiple announcements to be aggregated into a single message.
|
||||
- **`optimization_threshold`**: Number of hops difference required to trigger tree optimization (see below).
|
||||
|
||||
### Cache Settings
|
||||
|
||||
- **`message_cache_retention`**: How long to keep full message payloads in cache. This enables replying to `Graft` requests from peers who missed the eager push.
|
||||
- **`message_id_retention`**: How long to remember that we've already seen a message ID. This prevents re-delivering duplicate messages.
|
||||
- **`cache_evict_interval`**: How often to check and evict expired entries.
|
||||
|
||||
## State Structure
|
||||
|
||||
```rust
|
||||
pub struct State<PI> {
|
||||
me: PI, // Our peer identity
|
||||
config: Config, // Protocol configuration
|
||||
|
||||
pub eager_push_peers: BTreeSet<PI>, // Full message delivery peers
|
||||
pub lazy_push_peers: BTreeSet<PI>, // Message-ID-only delivery peers
|
||||
|
||||
lazy_push_queue: BTreeMap<PI, Vec<IHave>>, // Pending IHave announcements (batched)
|
||||
|
||||
missing_messages: HashMap<MessageId, VecDeque<(PI, Round)>>, // IHave senders awaiting delivery
|
||||
received_messages: TimeBoundCache<MessageId, ()>, // Seen message IDs
|
||||
cache: TimeBoundCache<MessageId, Gossip>, // Full message payloads
|
||||
|
||||
graft_timer_scheduled: HashSet<MessageId>, // Active graft timers
|
||||
dispatch_timer_scheduled: bool, // Whether IHave dispatch is pending
|
||||
|
||||
init: bool, // Whether first event was processed
|
||||
stats: Stats, // Message counters
|
||||
max_message_size: usize, // Maximum allowed message size
|
||||
}
|
||||
```
|
||||
|
||||
## Message Types (`plumtree::Message`)
|
||||
|
||||
| Message | Direction | Purpose |
|
||||
|---------|-----------|---------|
|
||||
| `Gossip(Gossip)` | Eager push | Full message content, broadcast to eager peers |
|
||||
| `Prune` | Bidirectional | Sent when moving a peer from eager to lazy set |
|
||||
| `Graft(Graft)` | Lazy → Eager upgrade | Request to become an eager peer; may include a message ID to request re-delivery |
|
||||
| `IHave(Vec<IHave>)` | Lazy push | Announcement: "I have these messages" (batched, sent after `dispatch_timeout`) |
|
||||
|
||||
### Gossip Message Structure
|
||||
|
||||
```rust
|
||||
pub struct Gossip {
|
||||
id: MessageId, // blake3 hash of content
|
||||
content: Bytes, // The actual message payload
|
||||
scope: DeliveryScope, // Swarm(round) or Neighbors
|
||||
}
|
||||
```
|
||||
|
||||
The `DeliveryScope` tracks how many hops the message has traveled:
|
||||
|
||||
```rust
|
||||
pub enum DeliveryScope {
|
||||
Swarm(Round), // Delivered via the swarm; Round = hop count from origin
|
||||
Neighbors, // Delivered only to direct neighbors (not forwarded further)
|
||||
}
|
||||
```
|
||||
|
||||
Each time a `Gossip` message is forwarded, its `Round` is incremented via `next_round()`. `Neighbors`-scope messages are not forwarded at all.
|
||||
|
||||
### IHave Structure
|
||||
|
||||
```rust
|
||||
pub struct IHave {
|
||||
id: MessageId, // The blake3 hash of the message content
|
||||
round: Round, // The hop count at which the sender received this message
|
||||
}
|
||||
```
|
||||
|
||||
### Graft Structure
|
||||
|
||||
```rust
|
||||
pub struct Graft {
|
||||
id: Option<MessageId>, // If set, also reply with full message content
|
||||
round: Round, // The round from the IHave that triggered this graft
|
||||
}
|
||||
```
|
||||
|
||||
### Message ID
|
||||
|
||||
```rust
|
||||
pub struct MessageId([u8; 32]); // blake3 hash of message content
|
||||
|
||||
impl MessageId {
|
||||
pub fn from_content(message: &[u8]) -> Self {
|
||||
Self::from(blake3::hash(message))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Messages are validated: when receiving a `Gossip`, the receiver checks that `MessageId::from_content(&content) == id`. Spoofed messages (where the hash doesn't match the content) are silently discarded.
|
||||
|
||||
## Broadcast Flow
|
||||
|
||||
### Sending a Message
|
||||
|
||||
```
|
||||
1. Compute MessageId = blake3(content)
|
||||
2. Create Gossip { id, content, scope: Swarm(Round(0)) or Neighbors }
|
||||
3. If Swarm scope:
|
||||
a. Add to received_messages and cache
|
||||
b. Queue IHave for lazy peers (dispatched after dispatch_timeout)
|
||||
4. Eager-push Gossip to all eager peers (except self and sender)
|
||||
```
|
||||
|
||||
### Receiving a Gossip Message
|
||||
|
||||
```
|
||||
1. Validate: message.id == blake3(message.content) → discard if invalid
|
||||
2. If already received (in received_messages):
|
||||
→ Send Prune to sender (move sender to lazy set)
|
||||
→ Return (don't re-broadcast)
|
||||
3. If Swarm scope:
|
||||
a. Add to received_messages
|
||||
b. Increment round (next_round)
|
||||
c. Add to cache (for Graft replies)
|
||||
d. Eager-push to all eager peers (except sender)
|
||||
e. Lazy-push IHave to all lazy peers (except sender)
|
||||
f. Check if any prior IHave senders had a shorter path → optimize tree
|
||||
4. Emit Received event to application
|
||||
```
|
||||
|
||||
### Receiving an IHave
|
||||
|
||||
```
|
||||
For each IHave entry:
|
||||
If message ID not in received_messages:
|
||||
Add (sender, round) to missing_messages[message_id]
|
||||
If no graft timer scheduled for this message:
|
||||
Schedule SendGraft timer (graft_timeout_1)
|
||||
```
|
||||
|
||||
### Graft Timer Expiry (Two-Phase)
|
||||
|
||||
**Phase 1 (`graft_timeout_1`):**
|
||||
```
|
||||
If message already received → no-op (cancel)
|
||||
Otherwise:
|
||||
Pop first (peer, round) from missing_messages[message_id]
|
||||
Move peer to eager set
|
||||
Send Graft { id: Some(message_id), round } to that peer
|
||||
Schedule another SendGraft timer (graft_timeout_2) for fallback
|
||||
```
|
||||
|
||||
**Phase 2 (`graft_timeout_2`):**
|
||||
```
|
||||
If message already received → no-op
|
||||
Otherwise:
|
||||
Pop next (peer, round) from missing_messages[message_id]
|
||||
Move that peer to eager set
|
||||
Send Graft { id: Some(message_id), round }
|
||||
Schedule another SendGraft timer (graft_timeout_2)
|
||||
(continues until the message is received or senders are exhausted)
|
||||
```
|
||||
|
||||
### Receiving a Graft
|
||||
|
||||
```
|
||||
1. Move sender to eager set
|
||||
2. If Graft contains a message ID:
|
||||
Look up message in cache
|
||||
If found: send Gossip(message) to the requesting peer
|
||||
```
|
||||
|
||||
### Receiving a Prune
|
||||
|
||||
```
|
||||
Move sender from eager set to lazy set
|
||||
```
|
||||
|
||||
## Tree Optimization
|
||||
|
||||
The PlumTree self-optimizes based on latency. When a `Gossip` message is received, if we previously received an `IHave` for the same message from a different peer, we check whether the IHave path was significantly shorter:
|
||||
|
||||
```
|
||||
if (ihave_round < gossip_round) && (gossip_round - ihave_round) >= optimization_threshold:
|
||||
Graft the IHave sender (move to eager)
|
||||
Prune the Gossip sender (move to lazy)
|
||||
```
|
||||
|
||||
This means if a peer consistently has a shorter path to the message origin, they are promoted to eager, and the longer-path peer is demoted. The `optimization_threshold` (default: 7 hops) prevents thrashing from minor latency differences.
|
||||
|
||||
## Neighbor Events
|
||||
|
||||
PlumTree receives neighbor events from HyParView:
|
||||
|
||||
- **`NeighborUp(peer)`**: Add peer to eager set (all new neighbors start as eager)
|
||||
- **`NeighborDown(peer)`**: Remove from both eager and lazy sets; clean up any `IHave` entries from this peer in `missing_messages`
|
||||
|
||||
## Neighbor-Only Broadcast
|
||||
|
||||
The `Scope::Neighbors` broadcast scope sends a message only to directly connected peers (the active view), without any forwarding:
|
||||
|
||||
```rust
|
||||
pub enum Scope {
|
||||
Swarm, // Broadcast to all peers in the swarm
|
||||
Neighbors, // Broadcast only to immediate neighbors
|
||||
}
|
||||
```
|
||||
|
||||
Neighbor-scoped messages are useful for localized communication and are not cached or re-broadcast.
|
||||
|
||||
## Cache Management
|
||||
|
||||
The PlumTree maintains two time-bounded caches:
|
||||
|
||||
1. **`cache`** (`TimeBoundCache<MessageId, Gossip>`): Stores full message payloads for `message_cache_retention` (default 30s). This enables replying to `Graft` requests for recently-broadcast messages.
|
||||
|
||||
2. **`received_messages`** (`TimeBoundCache<MessageId, ()>`): Tracks which messages have been seen for `message_id_retention` (default 90s). This prevents duplicate delivery.
|
||||
|
||||
Both caches are periodically evicted (every `cache_evict_interval`, default 1s) via the `EvictCache` timer.
|
||||
187
docs/research/references/iroh/iroh-gossip/04-state-and-topic.md
Normal file
187
docs/research/references/iroh/iroh-gossip/04-state-and-topic.md
Normal file
@@ -0,0 +1,187 @@
|
||||
# iroh-gossip: Protocol State & Topic Coordination
|
||||
|
||||
## Overview
|
||||
|
||||
The `state` module (`src/proto/state.rs`) provides the **top-level protocol state machine** that manages multiple topics. The `topic` module (`src/proto/topic.rs`) coordinates the HyParView and PlumTree state machines for a single topic.
|
||||
|
||||
## Multi-Topic State (`state::State`)
|
||||
|
||||
```rust
|
||||
pub struct State<PI, R> {
|
||||
me: PI, // Our peer identity
|
||||
me_data: PeerData, // Our opaque peer data
|
||||
config: Config, // Protocol configuration
|
||||
rng: R, // Random number generator
|
||||
states: HashMap<TopicId, topic::State<PI, R>>, // Per-topic state
|
||||
outbox: Outbox<PI>, // Buffered output events
|
||||
peer_topics: ConnsMap<PI>, // Maps peer → set of shared topics
|
||||
}
|
||||
```
|
||||
|
||||
The `State` acts as a **multiplexer** — it routes events to the correct topic's state and collects output events. It also tracks which topics are shared with each peer (in `peer_topics`), which is used to determine when a peer connection can safely be closed (only when no topic still needs it).
|
||||
|
||||
### TopicId
|
||||
|
||||
```rust
|
||||
#[derive(Clone, Copy, Eq, PartialEq, Hash, Serialize, Ord, PartialOrd, Deserialize)]
|
||||
pub struct TopicId([u8; 32]);
|
||||
```
|
||||
|
||||
A 32-byte identifier for a topic. Typically created as `blake3::hash(topic_name)` or from raw bytes. Each topic is an independent swarm and broadcast scope.
|
||||
|
||||
### Wire Message Format
|
||||
|
||||
```rust
|
||||
pub struct Message<PI> {
|
||||
pub topic: TopicId,
|
||||
pub message: topic::Message<PI>,
|
||||
}
|
||||
```
|
||||
|
||||
Every wire message carries the `TopicId` prefix, allowing multiplexing of multiple topics over a single connection.
|
||||
|
||||
### Event Routing
|
||||
|
||||
`InEvent` is mapped to either a topic-specific event or a global event:
|
||||
|
||||
| InEvent | Routing |
|
||||
|---------|---------|
|
||||
| `RecvMessage(from, Message{topic, message})` | → Topic-specific: `topic::InEvent::RecvMessage` |
|
||||
| `Command(topic, command)` | → Topic-specific: `topic::InEvent::Command` |
|
||||
| `TimerExpired(Timer{topic, timer})` | → Topic-specific: `topic::InEvent::TimerExpired` |
|
||||
| `PeerDisconnected(peer)` | → Broadcast to ALL topics |
|
||||
| `UpdatePeerData(data)` | → Broadcast to ALL topics |
|
||||
|
||||
### Topic Lifecycle
|
||||
|
||||
When a `Command::Join(peers)` is received for a topic that doesn't yet have state, a new `topic::State` is automatically created. When `Command::Quit` is received, the topic's state is removed after processing the quit event.
|
||||
|
||||
### Connection Management
|
||||
|
||||
When a `topic::OutEvent::DisconnectPeer(peer)` is emitted, the state module checks `peer_topics` to see if any other topic still needs a connection to that peer. Only when no topic needs the peer anymore is `OutEvent::DisconnectPeer(peer)` emitted at the top level.
|
||||
|
||||
## Topic State (`topic::State`)
|
||||
|
||||
```rust
|
||||
pub struct State<PI, R> {
|
||||
me: PI,
|
||||
pub swarm: hyparview::State<PI, R>, // HyParView membership
|
||||
pub gossip: plumtree::State<PI>, // PlumTree broadcast
|
||||
outbox: VecDeque<OutEvent<PI>>,
|
||||
stats: Stats,
|
||||
}
|
||||
```
|
||||
|
||||
The topic state **composes** HyParView and PlumTree, bridging them together:
|
||||
|
||||
### Event Forwarding
|
||||
|
||||
When `topic::State::handle()` is called:
|
||||
|
||||
1. **HyParView events** are processed first (membership layer).
|
||||
2. **NeighborUp/NeighborDown events** emitted by HyParView are forwarded to PlumTree:
|
||||
- `NeighborUp(peer)` → `plumtree::InEvent::NeighborUp(peer)` — adds peer to eager set
|
||||
- `NeighborDown(peer)` → `plumtree::InEvent::NeighborDown(peer)` — removes peer from both sets
|
||||
3. All output events from both layers are collected and returned.
|
||||
|
||||
### Command Handling
|
||||
|
||||
| Command | Action |
|
||||
|---------|--------|
|
||||
| `Join(peers)` | Sends `RequestJoin(peer)` to HyParView for each peer in the list |
|
||||
| `Broadcast(data, scope)` | Sends `Broadcast(data, scope)` to PlumTree |
|
||||
| `Quit` | Sends `Quit` to HyParView (which sends `Disconnect` to all active peers) |
|
||||
|
||||
### Message Routing
|
||||
|
||||
When a topic message is received:
|
||||
|
||||
```rust
|
||||
match message {
|
||||
Message::Swarm(message) => hyparview.handle(RecvMessage(from, message)),
|
||||
Message::Gossip(message) => plumtree.handle(RecvMessage(from, message)),
|
||||
}
|
||||
```
|
||||
|
||||
### Timer Routing
|
||||
|
||||
```rust
|
||||
match timer {
|
||||
Timer::Swarm(timer) => hyparview.handle(TimerExpired(timer)),
|
||||
Timer::Gossip(timer) => plumtree.handle(TimerExpired(timer)),
|
||||
}
|
||||
```
|
||||
|
||||
## Topic Messages (`topic::Message`)
|
||||
|
||||
```rust
|
||||
pub enum Message<PI> {
|
||||
Swarm(hyparview::Message<PI>), // Membership messages
|
||||
Gossip(plumtree::Message), // Broadcast messages
|
||||
}
|
||||
```
|
||||
|
||||
The message kind is used for metrics tracking:
|
||||
|
||||
```rust
|
||||
pub fn kind(&self) -> MessageKind {
|
||||
match self {
|
||||
Message::Swarm(_) => MessageKind::Control,
|
||||
Message::Gossip(message) => match message {
|
||||
plumtree::Message::Gossip(_) => MessageKind::Data,
|
||||
_ => MessageKind::Control,
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Topic Events (`topic::Event`)
|
||||
|
||||
```rust
|
||||
pub enum Event<PI> {
|
||||
NeighborUp(PI), // From HyParView: new active neighbor
|
||||
NeighborDown(PI), // From HyParView: lost active neighbor
|
||||
Received(GossipEvent<PI>), // From PlumTree: received a gossip message
|
||||
}
|
||||
```
|
||||
|
||||
The `Received` event contains:
|
||||
|
||||
```rust
|
||||
pub struct GossipEvent<PI> {
|
||||
pub content: Bytes, // Message payload
|
||||
pub delivered_from: PI, // Peer that delivered the message to us
|
||||
pub scope: DeliveryScope, // Swarm(round) or Neighbors
|
||||
}
|
||||
```
|
||||
|
||||
## Topic Configuration
|
||||
|
||||
```rust
|
||||
pub struct Config {
|
||||
pub membership: hyparview::Config, // HyParView configuration
|
||||
pub broadcast: plumtree::Config, // PlumTree configuration
|
||||
pub max_message_size: usize, // Maximum wire message size (default: 4096)
|
||||
}
|
||||
```
|
||||
|
||||
The `max_message_size` is the total wire-level message size including headers. The actual payload capacity is computed as `max_message_size - postcard_header_size`, where the header size accounts for the topic ID and message envelope overhead.
|
||||
|
||||
## Statistics
|
||||
|
||||
Each topic tracks:
|
||||
```rust
|
||||
pub struct Stats {
|
||||
pub messages_sent: usize,
|
||||
pub messages_received: usize,
|
||||
}
|
||||
```
|
||||
|
||||
The PlumTree layer also tracks:
|
||||
```rust
|
||||
pub struct Stats {
|
||||
pub payload_messages_received: u64,
|
||||
pub control_messages_received: u64,
|
||||
pub max_last_delivery_hop: u16,
|
||||
}
|
||||
```
|
||||
244
docs/research/references/iroh/iroh-gossip/05-net-actor.md
Normal file
244
docs/research/references/iroh/iroh-gossip/05-net-actor.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# iroh-gossip: Networking Layer & Actor Model
|
||||
|
||||
## Overview
|
||||
|
||||
The `net` module (`src/net.rs` and submodules) provides the async runtime layer that connects the IO-free protocol state machine to real network IO via iroh QUIC connections. It is built around a **single Actor** that manages all topics and connections.
|
||||
|
||||
## ALPN Protocol
|
||||
|
||||
```rust
|
||||
pub const GOSSIP_ALPN: &[u8] = b"/iroh-gossip/1";
|
||||
```
|
||||
|
||||
This ALPN identifier is used when establishing QUIC connections through iroh.
|
||||
|
||||
## Gossip Handle (`net::Gossip`)
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Gossip {
|
||||
pub(crate) inner: Arc<Inner>,
|
||||
}
|
||||
```
|
||||
|
||||
`Gossip` is the primary public handle. It derefs to `GossipApi`, providing the user-facing interface:
|
||||
|
||||
```rust
|
||||
// Subscribe to a topic
|
||||
let (sender, receiver) = gossip.subscribe(topic_id, bootstrap_peers).await?.split();
|
||||
|
||||
// Subscribe and wait for at least one connection
|
||||
let topic = gossip.subscribe_and_join(topic_id, bootstrap_peers).await?;
|
||||
|
||||
// Broadcast a message
|
||||
sender.broadcast(b"hello world".to_vec().into()).await?;
|
||||
|
||||
// Broadcast to neighbors only
|
||||
sender.broadcast_neighbors(b"local announcement".to_vec().into()).await?;
|
||||
|
||||
// Join additional peers
|
||||
sender.join_peers(vec![peer_id]).await?;
|
||||
```
|
||||
|
||||
### Builder Pattern
|
||||
|
||||
```rust
|
||||
let gossip = Gossip::builder()
|
||||
.max_message_size(8192) // Default: 4096
|
||||
.membership_config(hyparview_config) // HyParView settings
|
||||
.broadcast_config(plumtree_config) // PlumTree settings
|
||||
.alpn(b"/custom-alpn") // Custom ALPN (must match across network)
|
||||
.spawn(endpoint);
|
||||
```
|
||||
|
||||
## Architecture: The Actor
|
||||
|
||||
The core of the networking layer is the `Actor` struct, which runs as a single async task:
|
||||
|
||||
```rust
|
||||
struct Actor {
|
||||
alpn: Bytes,
|
||||
state: proto::State<PublicKey, StdRng>, // Protocol state machine
|
||||
endpoint: Endpoint, // iroh endpoint for connections
|
||||
dialer: Dialer, // Manages outgoing connections
|
||||
rpc_rx: mpsc::Receiver<RpcMessage>, // API commands
|
||||
local_rx: mpsc::Receiver<LocalActorMessage>, // Local commands (connections, shutdown)
|
||||
in_event_tx: mpsc::Sender<InEvent>, // Protocol input channel
|
||||
in_event_rx: mpsc::Receiver<InEvent>, // Protocol input channel (receiver)
|
||||
timers: Timers<Timer>, // Scheduled timers
|
||||
topics: HashMap<TopicId, TopicState>, // Per-topic subscription state
|
||||
peers: HashMap<EndpointId, PeerState>, // Per-peer connection state
|
||||
command_rx: stream_group::Keyed<TopicCommandStream>, // Per-topic command streams
|
||||
quit_queue: VecDeque<TopicId>, // Topics pending unsubscription
|
||||
connection_tasks: JoinSet<...>, // Running connection loop tasks
|
||||
metrics: Arc<Metrics>,
|
||||
topic_event_forwarders: JoinSet<TopicId>, // Tasks forwarding events to subscribers
|
||||
address_lookup: GossipAddressLookup, // Address discovery integration
|
||||
}
|
||||
```
|
||||
|
||||
### Event Loop
|
||||
|
||||
The actor's `run()` method calls `event_loop()` in a loop. Each iteration uses `tokio::select!` to handle:
|
||||
|
||||
| Source | Action |
|
||||
|--------|--------|
|
||||
| `local_rx` (local messages) | Handle incoming connections or shutdown |
|
||||
| `rpc_rx` (RPC messages) | Process `Join` requests from the API |
|
||||
| `command_rx` (per-topic commands) | Process `Broadcast`, `BroadcastNeighbors`, `JoinPeers`, or stream closure |
|
||||
| `addr_updates` (endpoint addr changes) | Update our `PeerData` in the protocol state |
|
||||
| `dialer` (connection establishment) | Handle successful/failed outgoing connections |
|
||||
| `in_event_rx` (protocol events from connections) | Feed events to the protocol state machine |
|
||||
| `timers` (scheduled timers) | Feed timer expirations to the protocol state machine |
|
||||
| `connection_tasks` (connection task completions) | Handle peer disconnections |
|
||||
| `topic_event_forwarders` (subscription tasks) | Handle topic cleanup when all subscribers drop |
|
||||
|
||||
### Processing InEvents
|
||||
|
||||
When an `InEvent` is processed, the actor calls `self.state.handle(event, now, metrics)`, which returns `Vec<OutEvent>`. For each `OutEvent`:
|
||||
|
||||
| OutEvent | Action |
|
||||
|----------|--------|
|
||||
| `SendMessage(peer, message)` | Send via peer's active connection or queue for pending connection |
|
||||
| `EmitEvent(topic, event)` | Forward to topic's `broadcast::Sender` → subscribers |
|
||||
| `ScheduleTimer(delay, timer)` | Schedule timer via `Timers` data structure |
|
||||
| `DisconnectPeer(peer)` | Drop the peer's send channel, removing from `peers` map |
|
||||
| `PeerData(endpoint_id, data)` | Decode `AddrInfo` from `PeerData`, add to `GossipAddressLookup` |
|
||||
|
||||
## Connection Management
|
||||
|
||||
### Peer States
|
||||
|
||||
```rust
|
||||
enum PeerState {
|
||||
Pending {
|
||||
queue: Vec<ProtoMessage>, // Messages queued while connecting
|
||||
},
|
||||
Active {
|
||||
active_send_tx: mpsc::Sender<ProtoMessage>, // Current active send channel
|
||||
active_conn_id: ConnId, // Stable ID of active connection
|
||||
other_conns: Vec<ConnId>, // Older connections still closing
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
When a message needs to be sent to a peer:
|
||||
- **Active**: Send immediately via `active_send_tx`
|
||||
- **Pending**: Queue the message and initiate a dial
|
||||
|
||||
### Dialer
|
||||
|
||||
```rust
|
||||
struct Dialer {
|
||||
endpoint: Endpoint,
|
||||
pending: JoinSet<(EndpointId, Option<Result<Connection, ConnectError>>)>,
|
||||
pending_dials: HashMap<EndpointId, CancellationToken>,
|
||||
}
|
||||
```
|
||||
|
||||
The `Dialer` manages outgoing connections. It:
|
||||
1. Checks if a dial is already pending for a peer
|
||||
2. Spawns an async connection task with cancellation support
|
||||
3. Returns completed connections via `next_conn()`
|
||||
|
||||
### Connection Loop
|
||||
|
||||
Each peer connection runs a `connection_loop` task:
|
||||
|
||||
```rust
|
||||
async fn connection_loop(
|
||||
from: PublicKey, // Remote peer's public key
|
||||
conn: Connection, // QUIC connection
|
||||
origin: ConnOrigin, // Accept (incoming) or Dial (outgoing)
|
||||
send_rx: mpsc::Receiver<ProtoMessage>, // Messages to send
|
||||
in_event_tx: mpsc::Sender<InEvent>, // Channel to protocol
|
||||
max_message_size: usize, // Maximum message size
|
||||
queue: Vec<ProtoMessage>, // Queued messages to send first
|
||||
) -> Result<(), ConnectionLoopError>
|
||||
```
|
||||
|
||||
The connection loop:
|
||||
1. First sends any queued messages
|
||||
2. Runs a send loop and receive loop concurrently (`tokio::join!`)
|
||||
3. Uses iroh QUIC bidirectional streams for communication
|
||||
|
||||
### Wire Protocol
|
||||
|
||||
Messages are serialized with `postcard` and sent as **length-prefixed frames** over QUIC unidirectional streams:
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ Stream Header │ ── Contains TopicId (sent once per stream)
|
||||
├──────────────┤
|
||||
│ Frame (len) │ ── u32 length prefix
|
||||
│ Frame (data) │ ── postcard-encoded topic::Message<PublicKey>
|
||||
├──────────────┤
|
||||
│ Frame (len) │ ── next message...
|
||||
│ Frame (data) │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
Each topic gets its own unidirectional stream. The stream header is sent once when the stream is opened. Disconnect messages close the stream after being sent.
|
||||
|
||||
The `SendLoop` manages per-topic streams within a connection:
|
||||
|
||||
```rust
|
||||
struct SendLoop {
|
||||
conn: Connection,
|
||||
streams: HashMap<TopicId, SendStream>, // One stream per topic
|
||||
buffer: Vec<u8>,
|
||||
max_message_size: usize,
|
||||
send_rx: mpsc::Receiver<ProtoMessage>,
|
||||
}
|
||||
```
|
||||
|
||||
When a disconnect message is sent for a topic, the stream for that topic is closed (via `finish()`).
|
||||
|
||||
## Topic State (Net Layer)
|
||||
|
||||
```rust
|
||||
struct TopicState {
|
||||
neighbors: BTreeSet<EndpointId>, // Current active neighbors (from protocol)
|
||||
event_sender: broadcast::Sender<ProtoEvent>, // Broadcast channel to subscribers
|
||||
command_rx_keys: HashSet<stream_group::Key>, // Active command stream keys
|
||||
}
|
||||
```
|
||||
|
||||
A topic is considered "still needed" if it has either:
|
||||
- Active command receivers (publishers), or
|
||||
- Active event subscribers (subscribers)
|
||||
|
||||
When neither exists, the topic is queued for quit/unsubscription.
|
||||
|
||||
## Address Lookup Integration
|
||||
|
||||
The `GossipAddressLookup` integrates with iroh's address discovery system:
|
||||
|
||||
```rust
|
||||
pub(crate) struct GossipAddressLookup {
|
||||
endpoints: NodeMap, // BTreeMap<EndpointId, StoredEndpointInfo>
|
||||
_task_handle: Arc<AbortOnDropHandle<()>>, // Background eviction task
|
||||
}
|
||||
```
|
||||
|
||||
It implements iroh's `AddressLookup` trait, allowing gossip-discovered peer addresses to feed back into iroh's connection establishment. This means that when a peer shares its address information in `Join` or `ForwardJoin` messages, that information is used to help iroh connect to that peer.
|
||||
|
||||
Entries expire after 5 minutes (configurable via `RetentionOpts`), with eviction checks every 30 seconds.
|
||||
|
||||
## Metrics
|
||||
|
||||
The `Metrics` struct tracks various counters:
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `msgs_ctrl_sent` | Control messages sent |
|
||||
| `msgs_ctrl_recv` | Control messages received |
|
||||
| `msgs_data_sent` | Data messages sent |
|
||||
| `msgs_data_recv` | Data messages received |
|
||||
| `msgs_data_sent_size` | Total size of data messages sent |
|
||||
| `msgs_data_recv_size` | Total size of data messages received |
|
||||
| `msgs_ctrl_sent_size` | Total size of control messages sent |
|
||||
| `msgs_ctrl_recv_size` | Total size of control messages received |
|
||||
| `neighbor_up` | Neighbor connections established |
|
||||
| `neighbor_down` | Neighbor connections lost |
|
||||
| `actor_tick_*` | Various event loop tick counters |
|
||||
290
docs/research/references/iroh/iroh-gossip/06-api-data-flow.md
Normal file
290
docs/research/references/iroh/iroh-gossip/06-api-data-flow.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# iroh-gossip: Public API & Data Flow
|
||||
|
||||
## Public API Types
|
||||
|
||||
### Gossip (Main Handle)
|
||||
|
||||
The `Gossip` struct is the main entry point, created via a `Builder`:
|
||||
|
||||
```rust
|
||||
let gossip = Gossip::builder()
|
||||
.max_message_size(8192)
|
||||
.membership_config(HyparviewConfig { ... })
|
||||
.broadcast_config(PlumtreeConfig { ... })
|
||||
.alpn(b"/custom-alpn")
|
||||
.spawn(endpoint);
|
||||
```
|
||||
|
||||
It derefs to `GossipApi`, which provides:
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `subscribe(topic_id, bootstrap)` | Join a topic with default options |
|
||||
| `subscribe_and_join(topic_id, bootstrap)` | Join and wait for at least one connection |
|
||||
| `subscribe_with_opts(topic_id, opts)` | Join with custom `JoinOptions` |
|
||||
| `handle_connection(conn)` | Handle an incoming QUIC connection |
|
||||
| `shutdown()` | Gracefully leave all topics and stop |
|
||||
| `max_message_size()` | Get configured max message size |
|
||||
| `metrics()` | Get metrics handle |
|
||||
|
||||
### GossipTopic (Subscription Handle)
|
||||
|
||||
Returned by `subscribe()`, it is a `Stream<Item = Result<Event, ApiError>>`:
|
||||
|
||||
```rust
|
||||
let topic: GossipTopic = gossip.subscribe(topic_id, peers).await?;
|
||||
topic.broadcast(b"hello".to_vec().into()).await?;
|
||||
topic.broadcast_neighbors(b"local".to_vec().into()).await?;
|
||||
topic.joined().await?; // Wait for first connection
|
||||
```
|
||||
|
||||
Can be split into sender and receiver:
|
||||
|
||||
```rust
|
||||
let (sender, receiver) = topic.split();
|
||||
// sender: GossipSender - can broadcast and join peers
|
||||
// receiver: GossipReceiver - can receive events and check neighbors
|
||||
```
|
||||
|
||||
### GossipSender
|
||||
|
||||
```rust
|
||||
pub struct GossipSender(mpsc::Sender<Command>);
|
||||
|
||||
impl GossipSender {
|
||||
pub async fn broadcast(&self, message: Bytes) -> Result<(), ApiError>;
|
||||
pub async fn broadcast_neighbors(&self, message: Bytes) -> Result<(), ApiError>;
|
||||
pub async fn join_peers(&self, peers: Vec<EndpointId>) -> Result<(), ApiError>;
|
||||
}
|
||||
```
|
||||
|
||||
### GossipReceiver
|
||||
|
||||
```rust
|
||||
pub struct GossipReceiver {
|
||||
stream: Pin<Box<dyn Stream<Item = Result<Event, ApiError>> + Send + Sync + 'static>>,
|
||||
neighbors: HashSet<EndpointId>,
|
||||
}
|
||||
|
||||
impl GossipReceiver {
|
||||
pub fn neighbors(&self) -> impl Iterator<Item = EndpointId> + '_;
|
||||
pub async fn joined(&mut self) -> Result<(), ApiError>;
|
||||
pub fn is_joined(&self) -> bool;
|
||||
}
|
||||
```
|
||||
|
||||
The `GossipReceiver` tracks the neighbor set internally by processing `NeighborUp` and `NeighborDown` events.
|
||||
|
||||
### Event Types
|
||||
|
||||
```rust
|
||||
pub enum Event {
|
||||
NeighborUp(EndpointId), // New direct neighbor connected
|
||||
NeighborDown(EndpointId), // Direct neighbor disconnected
|
||||
Received(Message), // Gossip message received
|
||||
Lagged, // Internal channel lagged (messages dropped)
|
||||
}
|
||||
|
||||
pub struct Message {
|
||||
pub content: Bytes, // Message content
|
||||
pub scope: DeliveryScope, // Swarm(round) or Neighbors
|
||||
pub delivered_from: EndpointId, // Peer that delivered the message to us
|
||||
}
|
||||
```
|
||||
|
||||
### Command Types
|
||||
|
||||
```rust
|
||||
pub enum Command {
|
||||
Broadcast(Bytes), // Broadcast to all in swarm
|
||||
BroadcastNeighbors(Bytes), // Broadcast to direct neighbors only
|
||||
JoinPeers(Vec<EndpointId>), // Join additional peers
|
||||
}
|
||||
```
|
||||
|
||||
### JoinOptions
|
||||
|
||||
```rust
|
||||
pub struct JoinOptions {
|
||||
pub bootstrap: BTreeSet<EndpointId>, // Initial peers to connect to
|
||||
pub subscription_capacity: usize, // Event channel capacity (default: 2048)
|
||||
}
|
||||
```
|
||||
|
||||
### DeliveryScope
|
||||
|
||||
```rust
|
||||
pub enum DeliveryScope {
|
||||
Swarm(Round), // Message traveled `Round` hops from origin
|
||||
Neighbors, // Direct neighbor message (not forwarded)
|
||||
}
|
||||
```
|
||||
|
||||
`DeliveryScope::Swarm(Round(0))` means the message was sent by a direct neighbor. `Round(n)` means the message traveled n hops.
|
||||
|
||||
## Data Flow Diagrams
|
||||
|
||||
### Joining a Topic
|
||||
|
||||
```
|
||||
User Code GossipApi Actor Proto State
|
||||
| | | |
|
||||
|-- subscribe(topic, peers)->| | |
|
||||
| |-- JoinRequest ------->| |
|
||||
| | |-- Command::Join ------>|
|
||||
| | | |-- RequestJoin(peers)
|
||||
| | | |-- SendMessage(peer, Join)
|
||||
| | | |-- ...
|
||||
| |<-- NeighborUp events--|<-- EmitEvent(NeighborUp)|
|
||||
|<-- Event::NeighborUp ------| | |
|
||||
```
|
||||
|
||||
### Broadcasting a Message
|
||||
|
||||
```
|
||||
User Code GossipSender Actor Proto State Network
|
||||
| | | | |
|
||||
|-- broadcast(msg) ->| | | |
|
||||
| |-- Command:: --> | | |
|
||||
| | Broadcast | | |
|
||||
| | |-- Broadcast ---->| |
|
||||
| | | |-- eager_push --->|
|
||||
| | | | (Gossip msgs) |
|
||||
| | | |-- lazy_push ----->|
|
||||
| | | | (IHave msgs) |
|
||||
| | | | |
|
||||
| (other peer receives Gossip) | | |
|
||||
| | | |<-- RecvMessage --|
|
||||
| | |<-- InEvent -------| |
|
||||
| | | | (validates ID) |
|
||||
| | | | (forwards) |
|
||||
|<-- Received(msg) -|<-- EmitEvent -| | |
|
||||
```
|
||||
|
||||
### Receiving and Processing IHave/Graft
|
||||
|
||||
```
|
||||
Time →
|
||||
|
||||
Peer A Our Node Peer B
|
||||
| | |
|
||||
|-- IHave(id, round) --->| |
|
||||
| | Schedule graft_timeout_1 |
|
||||
| | (wait for eager push) |
|
||||
| | |
|
||||
| [timeout expires] | |
|
||||
| |-- Graft(id, round) ----->| (Peer B sent IHave)
|
||||
| | |
|
||||
| |<-- Gossip(content) -------| (Peer B replies)
|
||||
| | |
|
||||
| |-- Prune ----------------->| (maybe, if optimization)
|
||||
```
|
||||
|
||||
### HyParView Join Flow
|
||||
|
||||
```
|
||||
New Node Contact Node Active Peers of Contact
|
||||
| | |
|
||||
|-- Join(me_data) -->| |
|
||||
| |-- add_active(new) |
|
||||
| |-- Neighbor(High) ----->| (to new node)
|
||||
| |-- ForwardJoin ------->| (to each active peer)
|
||||
| | |-- add_active or add_passive
|
||||
| | |-- Neighbor(Low/High) -> (to new node)
|
||||
| | |-- ForwardJoin -> (random peer)
|
||||
| | |
|
||||
|<-- Neighbor(High) -| |
|
||||
|<-- Neighbor(Low/High) ----------------------|
|
||||
| | |
|
||||
```
|
||||
|
||||
### Shuffle Periodic Operation
|
||||
|
||||
```
|
||||
Node A Node B Random Node
|
||||
| | |
|
||||
|-- Shuffle ---------->| |
|
||||
| (origin=A, nodes, | |
|
||||
| TTL=6) | |
|
||||
| |-- Shuffle ------------>|
|
||||
| | (origin=A, nodes, |
|
||||
| | TTL=5) |
|
||||
| | |-- ...
|
||||
| | |-- (TTL reaches 0)
|
||||
| | |
|
||||
|<-- ShuffleReply ----|<-- ShuffleReply --------|
|
||||
| (random nodes) | (random nodes) |
|
||||
| | |
|
||||
|-- add_passive(nodes from reply) |
|
||||
```
|
||||
|
||||
## RPC Support (Optional Feature)
|
||||
|
||||
When the `rpc` feature is enabled, `GossipApi` can also operate remotely:
|
||||
|
||||
```rust
|
||||
// Server side
|
||||
gossip.listen(rpc_endpoint).await;
|
||||
|
||||
// Client side
|
||||
let api = GossipApi::connect(rpc_endpoint, addr);
|
||||
let topic = api.subscribe_and_join(topic_id, bootstrap).await?;
|
||||
```
|
||||
|
||||
This uses the `irpc`/`noq` crates for bidirectional streaming RPC. The `Join` request establishes a bidirectional stream:
|
||||
- Client → Server: `Command` messages (Broadcast, BroadcastNeighbors, JoinPeers)
|
||||
- Server → Client: `Event` messages (NeighborUp, NeighborDown, Received, Lagged)
|
||||
|
||||
## Channel Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Actor │
|
||||
│ │
|
||||
RPC/Local ──────►│ rpc_rx ◄─────────────────────────────────────│
|
||||
Commands │ local_rx ◄── HandleConnection, Shutdown │
|
||||
│ │
|
||||
│ in_event_tx ──► in_event_rx ────────────────│──► proto::State::handle()
|
||||
│ │ │
|
||||
│ ◄── OutEvent ────────────────────────────────│◄──── │
|
||||
│ │ │
|
||||
│ ├──► SendMessage ──► peer.send_tx │
|
||||
│ ├──► EmitEvent ──► topic.event_sender │
|
||||
│ ├──► ScheduleTimer ──► timers │
|
||||
│ ├──► DisconnectPeer ──► drop peer │
|
||||
│ └──► PeerData ──► address_lookup │
|
||||
│ │
|
||||
│ topic.event_sender ──► broadcast channel ────│──► GossipReceiver
|
||||
│ │
|
||||
│ command_rx ◄─── per-topic command streams ──│◄── GossipSender
|
||||
│ │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Configuration Defaults Summary
|
||||
|
||||
| Parameter | Default | Source |
|
||||
|-----------|---------|--------|
|
||||
| Active view capacity | 5 | HyParView paper (p9) |
|
||||
| Passive view capacity | 30 | HyParView paper (p9) |
|
||||
| Active random walk length | 6 | HyParView paper (p9) |
|
||||
| Passive random walk length | 3 | HyParView paper (p9) |
|
||||
| Shuffle random walk length | 6 | HyParView paper (p9) |
|
||||
| Shuffle active view count | 3 | HyParView paper (p9) |
|
||||
| Shuffle passive view count | 4 | HyParView paper (p9) |
|
||||
| Shuffle interval | 60s | Implementation choice |
|
||||
| Neighbor request timeout | 500ms | Implementation choice |
|
||||
| Graft timeout 1 | 80ms | Implementation choice |
|
||||
| Graft timeout 2 | 40ms | Implementation choice |
|
||||
| Dispatch timeout | 5ms | Implementation choice |
|
||||
| Optimization threshold | 7 hops | PlumTree paper (p12) |
|
||||
| Message cache retention | 30s | Implementation choice |
|
||||
| Message ID retention | 90s | Implementation choice |
|
||||
| Cache evict interval | 1s | Implementation choice |
|
||||
| Max message size | 4096 bytes | Implementation choice |
|
||||
| Send queue capacity | 64 messages | Implementation choice |
|
||||
| To-actor channel capacity | 64 messages | Implementation choice |
|
||||
| In-event channel capacity | 1024 messages | Implementation choice |
|
||||
| Topic event channel capacity | 256 events | Implementation choice |
|
||||
| Topic events default capacity | 2048 events | Implementation choice |
|
||||
| Topic commands channel capacity | 64 commands | Implementation choice |
|
||||
@@ -0,0 +1,176 @@
|
||||
# iroh-gossip: Utility Data Structures & Wire Format
|
||||
|
||||
## IndexSet (`proto::util::IndexSet`)
|
||||
|
||||
A wrapper around `indexmap::IndexSet` that provides random selection capabilities needed by HyParView:
|
||||
|
||||
```rust
|
||||
pub(crate) struct IndexSet<T> {
|
||||
inner: indexmap::IndexSet<T>,
|
||||
}
|
||||
```
|
||||
|
||||
### Key Operations
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `insert(value)` | Add element (returns false if already present) |
|
||||
| `remove(value)` | Remove by value (swap-remove, O(1)) |
|
||||
| `remove_index(index)` | Remove by index (swap-remove) |
|
||||
| `remove_random(rng)` | Remove a random element |
|
||||
| `pick_random(rng)` | Get reference to random element |
|
||||
| `pick_random_without(exclude, rng)` | Random element excluding certain elements |
|
||||
| `pick_random_index(rng)` | Random index |
|
||||
| `shuffled(rng)` | All elements in random order |
|
||||
| `shuffled_and_capped(len, rng)` | First `len` elements after shuffle |
|
||||
| `shuffled_without(exclude, rng)` | Random order excluding certain elements |
|
||||
| `shuffled_without_and_capped(exclude, len, rng)` | Capped shuffle excluding elements |
|
||||
| `iter_without(value)` | Iterator skipping a specific element |
|
||||
|
||||
These operations are critical for HyParView's random walks, shuffle exchanges, and passive view management.
|
||||
|
||||
## TimerMap (`proto::util::TimerMap`)
|
||||
|
||||
A priority queue of timer entries sorted by `Instant`, with stable ordering via a sequence counter:
|
||||
|
||||
```rust
|
||||
pub struct TimerMap<T> {
|
||||
heap: BinaryHeap<TimerMapEntry<T>>,
|
||||
seq: u64,
|
||||
}
|
||||
```
|
||||
|
||||
Used by the protocol state machine for scheduling future events (shuffles, graft timeouts, cache eviction). The networking layer wraps this in an async-friendly `Timers` type that can `wait_next()`.
|
||||
|
||||
### Key Operations
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `insert(instant, item)` | Schedule a timer |
|
||||
| `pop_before(limit)` | Pop the earliest entry if it's before `limit` |
|
||||
| `drain_until(from)` | Drain all entries up to a time |
|
||||
| `first()` | Get reference to earliest entry |
|
||||
|
||||
## TimeBoundCache (`proto::util::TimeBoundCache`)
|
||||
|
||||
A `HashMap` where entries expire after a specified `Instant`:
|
||||
|
||||
```rust
|
||||
pub struct TimeBoundCache<K, V> {
|
||||
map: HashMap<K, (Instant, V)>,
|
||||
expiry: TimerMap<K>,
|
||||
}
|
||||
```
|
||||
|
||||
Used by PlumTree for:
|
||||
- `received_messages: TimeBoundCache<MessageId, ()>` — deduplication
|
||||
- `cache: TimeBoundCache<MessageId, Gossip>` — message payload storage for Graft replies
|
||||
|
||||
### Key Operations
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `insert(key, value, expires)` | Insert with expiration |
|
||||
| `contains_key(key)` | Check existence |
|
||||
| `get(key)` | Get value |
|
||||
| `expires(key)` | Get expiration time |
|
||||
| `expire_until(instant)` | Remove all expired entries, returns count |
|
||||
| `len()` / `is_empty()` | Size queries |
|
||||
|
||||
The `expire_until` method correctly handles re-insertions: if a key is re-inserted with a later expiration time after being added to the expiry queue, the old expiry entry is ignored (not removed from the map).
|
||||
|
||||
## Wire Format
|
||||
|
||||
### Frame Encoding
|
||||
|
||||
Messages are encoded using `postcard` (a `no_std`-friendly, `serde`-compatible format) and sent as length-prefixed frames:
|
||||
|
||||
```
|
||||
┌──────────────┬──────────────┬─────────────────┐
|
||||
│ Length (u32) │ TopicHeader │ Message Payload │
|
||||
│ big-endian │ postcard │ postcard │
|
||||
└──────────────┴──────────────┴─────────────────┘
|
||||
```
|
||||
|
||||
### Stream Protocol
|
||||
|
||||
Each QUIC unidirectional stream is dedicated to a single topic. The stream begins with a `StreamHeader`:
|
||||
|
||||
```rust
|
||||
pub(crate) struct StreamHeader {
|
||||
pub(crate) topic_id: TopicId,
|
||||
}
|
||||
```
|
||||
|
||||
All subsequent frames on that stream carry messages for that topic. When a `Disconnect` message is sent, the stream is closed (via `finish()`).
|
||||
|
||||
### Message Types on Wire
|
||||
|
||||
```rust
|
||||
pub enum Message<PI> {
|
||||
Swarm(hyparview::Message<PI>), // Membership messages
|
||||
Gossip(plumtree::Message), // Broadcast messages
|
||||
}
|
||||
```
|
||||
|
||||
Where `PI` is `PublicKey` (32-byte ed25519 public key) in the networking layer.
|
||||
|
||||
The `MessageKind` classification is used for metrics:
|
||||
|
||||
| Kind | Message Types |
|
||||
|------|--------------|
|
||||
| `Data` | `Gossip` messages (actual content) |
|
||||
| `Control` | All Swarm messages, plus `Prune`, `Graft`, `IHave` |
|
||||
|
||||
### Message Size Limits
|
||||
|
||||
- Default max message size: 4096 bytes (minimum: 512)
|
||||
- The header size is computed at compile time via `postcard::experimental::serialized_size`
|
||||
- Actual payload capacity = `max_message_size - header_size`
|
||||
|
||||
The `SendLoop` checks message size before writing and returns `WriteError::TooLarge` if exceeded.
|
||||
|
||||
## PeerData & Address Propagation
|
||||
|
||||
The `PeerData` type is an opaque `Bytes` wrapper used in HyParView messages. In the `net` layer, it carries addressing information:
|
||||
|
||||
```rust
|
||||
struct AddrInfo {
|
||||
relay_url: Option<RelayUrl>,
|
||||
direct_addresses: BTreeSet<SocketAddr>,
|
||||
}
|
||||
```
|
||||
|
||||
This is serialized with `postcard` and passed as `PeerData` in `Join`, `ForwardJoin`, and `Neighbor` messages. When received, the `AddrInfo` is decoded and fed into `GossipAddressLookup`, which implements iroh's `AddressLookup` trait, allowing gossip-discovered addresses to be used for future connections.
|
||||
|
||||
## GossipAddressLookup
|
||||
|
||||
```rust
|
||||
pub(crate) struct GossipAddressLookup {
|
||||
endpoints: NodeMap, // Arc<RwLock<BTreeMap<EndpointId, StoredEndpointInfo>>>
|
||||
_task_handle: Arc<AbortOnDropHandle<()>>, // Background eviction task
|
||||
}
|
||||
```
|
||||
|
||||
Key behaviors:
|
||||
- **Merging**: When adding addresses for an already-known endpoint, new addresses are merged (union of direct addresses, relay URL is overwritten)
|
||||
- **Expiration**: Entries expire after 5 minutes, with eviction checks every 30 seconds
|
||||
- **Integration**: Implements `iroh::address_lookup::AddressLookup`, returning data with provenance "gossip"
|
||||
|
||||
## Dialer
|
||||
|
||||
```rust
|
||||
struct Dialer {
|
||||
endpoint: Endpoint,
|
||||
pending: JoinSet<(EndpointId, Option<Result<Connection, ConnectError>>)>,
|
||||
pending_dials: HashMap<EndpointId, CancellationToken>,
|
||||
}
|
||||
```
|
||||
|
||||
The `Dialer` manages outgoing connection attempts:
|
||||
- Queues a dial via `queue_dial(endpoint_id, alpn)`
|
||||
- Checks for pending dials to avoid duplicate connections
|
||||
- Supports cancellation of in-progress dials
|
||||
- Returns completed connections via `next_conn()`
|
||||
|
||||
When a dial succeeds, the connection is passed to `handle_connection()`. When a dial fails and the peer is not already active, a `PeerDisconnected` event is injected into the protocol state.
|
||||
@@ -0,0 +1,169 @@
|
||||
# iroh-gossip: Testing & Simulation
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
The crate includes two layers of testing:
|
||||
|
||||
### 1. Unit Tests (in source files)
|
||||
|
||||
Unit tests are embedded in each module file behind `#[cfg(test)]`:
|
||||
|
||||
| Module | Tests |
|
||||
|--------|-------|
|
||||
| `proto/hyparview.rs` | Not shown (would be in the file) |
|
||||
| `proto/plumtree.rs` | `optimize_tree`, `spoofed_messages_are_ignored`, `cache_is_evicted` |
|
||||
| `proto.rs` | `hyparview_smoke`, `plumtree_smoke`, `quit` |
|
||||
| `net.rs` | `gossip_net_smoke`, `subscription_cleanup` |
|
||||
| `api.rs` | `test_rpc`, `ensure_gossip_topic_is_sync` |
|
||||
| `proto/util.rs` | `indexset`, `timer_map`, `hex`, `time_bound_cache` |
|
||||
|
||||
### 2. Protocol Simulator (`proto::sim`)
|
||||
|
||||
The `sim` module (behind `test-utils` feature) provides a deterministic network simulator:
|
||||
|
||||
```rust
|
||||
// Available when feature = "test-utils"
|
||||
pub mod sim;
|
||||
```
|
||||
|
||||
This allows testing the protocol logic without any real networking, using seeded RNG for reproducibility.
|
||||
|
||||
The simulator creates a `Network` of virtual nodes, each running their own `proto::State`. Events are processed in discrete "trips" (round-trips), allowing controlled testing of protocol behavior.
|
||||
|
||||
### 3. Simulation Binary (`sim` feature)
|
||||
|
||||
The crate includes a CLI simulator (behind `simulator` feature) that can run large-scale simulations:
|
||||
|
||||
```
|
||||
cargo run --bin sim --features simulator
|
||||
```
|
||||
|
||||
This uses `rayon` for parallel execution and `comfy-table` for result output.
|
||||
|
||||
### 4. Integration Tests (`tests/sim.rs`)
|
||||
|
||||
Behind the `test-utils` feature, provides end-to-end protocol testing.
|
||||
|
||||
## Key Test Patterns
|
||||
|
||||
### Protocol-Level Smoke Test
|
||||
|
||||
From `proto.rs`:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn hyparview_smoke() {
|
||||
let rng = ChaCha12Rng::seed_from_u64(0);
|
||||
let mut config = Config::default();
|
||||
config.membership.active_view_capacity = 2;
|
||||
let mut network = Network::new(config.into(), rng);
|
||||
for i in 0..4 { network.insert(i); }
|
||||
let t: TopicId = [0u8; 32].into();
|
||||
|
||||
// Join nodes
|
||||
network.command(0, t, Command::Join(vec![1, 2]));
|
||||
network.command(1, t, Command::Join(vec![2]));
|
||||
network.command(2, t, Command::Join(vec![]));
|
||||
network.run_trips(3);
|
||||
|
||||
// Verify events and connections
|
||||
assert_eq!(network.events_sorted(), expected);
|
||||
assert_eq!(network.conns(), vec![(0, 1), (0, 2), (1, 2)]);
|
||||
assert!(network.check_synchronicity());
|
||||
}
|
||||
```
|
||||
|
||||
### PlumTree Optimization Test
|
||||
|
||||
From `plumtree.rs`:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn optimize_tree() {
|
||||
// When an IHave message arrives with fewer hops than the Gossip message,
|
||||
// and the difference exceeds optimization_threshold, the tree is restructured:
|
||||
// - The IHave sender is promoted to eager (Graft)
|
||||
// - The Gossip sender is demoted to lazy (Prune)
|
||||
}
|
||||
```
|
||||
|
||||
### Spoofed Message Test
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn spoofed_messages_are_ignored() {
|
||||
// Messages where MessageId != blake3(content) are silently discarded
|
||||
let message = Message::Gossip(Gossip {
|
||||
content: content.clone(),
|
||||
id: MessageId::from_content(b"wrong_content"), // Spoofed!
|
||||
scope: DeliveryScope::Swarm(Round(1)),
|
||||
});
|
||||
state.handle(InEvent::RecvMessage(2, message), now, &mut io);
|
||||
// No events are emitted
|
||||
}
|
||||
```
|
||||
|
||||
### Networking Smoke Test
|
||||
|
||||
From `net.rs`:
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn gossip_net_smoke() {
|
||||
// Creates 3 endpoints with a relay server
|
||||
// Subscribes and joins a topic
|
||||
// Broadcasts messages and verifies reception
|
||||
// Uses real QUIC connections via iroh
|
||||
}
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
The `Metrics` struct (in `src/metrics.rs`) uses `iroh_metrics::MetricsGroup`:
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Default, MetricsGroup)]
|
||||
#[metrics(name = "gossip")]
|
||||
pub struct Metrics {
|
||||
pub msgs_ctrl_sent: Counter,
|
||||
pub msgs_ctrl_recv: Counter,
|
||||
pub msgs_data_sent: Counter,
|
||||
pub msgs_data_recv: Counter,
|
||||
pub msgs_data_sent_size: Counter,
|
||||
pub msgs_data_recv_size: Counter,
|
||||
pub msgs_ctrl_sent_size: Counter,
|
||||
pub msgs_ctrl_recv_size: Counter,
|
||||
pub neighbor_up: Counter,
|
||||
pub neighbor_down: Counter,
|
||||
pub actor_tick_main: Counter,
|
||||
pub actor_tick_rx: Counter,
|
||||
pub actor_tick_endpoint: Counter,
|
||||
pub actor_tick_dialer: Counter,
|
||||
pub actor_tick_dialer_success: Counter,
|
||||
pub actor_tick_dialer_failure: Counter,
|
||||
pub actor_tick_in_event_rx: Counter,
|
||||
pub actor_tick_timers: Counter,
|
||||
}
|
||||
```
|
||||
|
||||
These are tracked both in the protocol state machine (for message counts) and in the actor event loop (for tick-level diagnostics). When the `metrics` feature is enabled, they are exported via Prometheus-compatible endpoints.
|
||||
|
||||
## References
|
||||
|
||||
### Academic Papers
|
||||
|
||||
- **HyParView**: Leitao, J., Pereira, J., & Rodrigues, L. (2007). "HyParView: A Membership Protocol for Reliable Gossip Multicast with Dense Coverage." [PDF](https://asc.di.fct.unl.pt/~jleitao/pdf/dsn07-leitao.pdf)
|
||||
- **PlumTree**: Leitao, J., Pereira, J., & Rodrigues, L. (2007). "Epidemic Broadcast Trees." [PDF](https://asc.di.fct.unl.pt/~jleitao/pdf/srds07-leitao.pdf)
|
||||
|
||||
### Implementation Reference
|
||||
|
||||
- Bartosz Sypytkowski's example implementation: [gist](https://gist.github.com/Horusiath/84fac596101b197da0546d1697580d99)
|
||||
|
||||
### Related Projects
|
||||
|
||||
- [iroh](https://docs.rs/iroh) — The networking library that iroh-gossip integrates with
|
||||
- [Earthstar](https://github.com/earthstar-project/earthstar) — Another PlumTree implementation referenced in code comments
|
||||
|
||||
### Crate Repository
|
||||
|
||||
- [github.com/n0-computer/iroh-gossip](https://github.com/n0-computer/iroh-gossip)
|
||||
40
docs/research/references/iroh/iroh-gossip/README.md
Normal file
40
docs/research/references/iroh/iroh-gossip/README.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# iroh-gossip Reference Documentation
|
||||
|
||||
This directory contains a deep-dive reference on how the `iroh-gossip` crate works, based on source code analysis of the repository at `/workspace/iroh-gossip`.
|
||||
|
||||
## Documents
|
||||
|
||||
| # | File | Topic |
|
||||
|---|------|-------|
|
||||
| 01 | [Overview & Architecture](01-overview-architecture.md) | Crate structure, module organization, design principles, features, dependencies |
|
||||
| 02 | [HyParView Membership](02-hyparview-membership.md) | Swarm membership protocol: active/passive views, join procedure, shuffle mechanism, failure recovery, PeerData |
|
||||
| 03 | [PlumTree Broadcast](03-plumtree-broadcast.md) | Epidemic broadcast trees: eager/lazy push, Graft/IHave/Prune, tree optimization, message deduplication, cache management |
|
||||
| 04 | [State & Topic Coordination](04-state-and-topic.md) | Multi-topic state management, topic lifecycle, event routing between HyParView and PlumTree |
|
||||
| 05 | [Net Actor & Networking](05-net-actor.md) | Actor model, event loop, connection management, Dialer, wire protocol, address lookup, topic state in the net layer |
|
||||
| 06 | [API & Data Flow](06-api-data-flow.md) | Public API types, subscription model, event/command flow, channel architecture, configuration defaults |
|
||||
| 07 | [Utilities & Wire Format](07-utilities-wire-format.md) | IndexSet, TimerMap, TimeBoundCache, serialization, PeerData/AddrInfo, Dialer internals |
|
||||
| 08 | [Testing & Metrics](08-testing-metrics-refs.md) | Test infrastructure, simulation, key test patterns, metrics, references |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Version
|
||||
`iroh-gossip` v0.97.0
|
||||
|
||||
### ALPN
|
||||
`/iroh-gossip/1`
|
||||
|
||||
### Core Protocols
|
||||
- **HyParView**: Hybrid partial view membership (active view = 5, passive view = 30 by default)
|
||||
- **PlumTree**: Epidemic broadcast trees (eager + lazy push with Graft/IHave optimization)
|
||||
|
||||
### Key Abstractions
|
||||
- **TopicId**: 32-byte identifier for a topic/swarm
|
||||
- **PeerIdentity**: Generic trait (instantiated as `PublicKey` in the net layer)
|
||||
- **PeerData**: Opaque bytes exchanged on join (carries `AddrInfo` in net layer)
|
||||
- **IO trait**: Interface for protocol output events (pure state machine, no IO)
|
||||
|
||||
### Wire Format
|
||||
- Postcard (serde) encoding over QUIC unidirectional streams
|
||||
- Length-prefixed frames (u32 length + postcard payload)
|
||||
- Stream header with TopicId
|
||||
- Max message size: 4096 bytes (configurable, minimum 512)
|
||||
Reference in New Issue
Block a user