alkdev/alknet
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(research): add nats-async and nats-server deep-dive references

Browse Source
This commit is contained in:
glm-5.1
2026-06-11 05:09:41 +00:00
parent f10dc23d13
commit ff4f544fa5
20 changed files with 5707 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
docs/research/references/nats.rs/nats-async/01-overview-and-architecture.md Normal file
View File

@@ -0,0 +1,170 @@
# async-nats: Overview & Architecture
**Crate**: `async-nats`
**Version**: 0.49.1
**Repository**: https://github.com/nats-io/nats.rs
**License**: Apache-2.0
**Rust Edition**: 2021
**MSRV**: 1.88.0
**Async Runtime**: Tokio
## What is async-nats?
`async-nats` is the official async Rust client for the [NATS messaging system](https://nats.io). It provides a Tokio-based asynchronous interface to NATS server features including:
- **Core NATS** — publish/subscribe, request/reply, queue groups
- **JetStream** — persistent stream-based messaging with at-least-once and exactly-once semantics
- **Key-Value Store** — KV abstraction built on JetStream streams
- **Object Store** — large-object storage built on JetStream streams
- **Service API** — microservice request/reply pattern with built-in PING/INFO/STATS verbs
The crate is positioned as the **core client** in the NATS Rust ecosystem. A separate project, [Orbit](https://github.com/synadia-io/orbit.rs), provides higher-level opinionated abstractions on top.
```
┌──────────────────────────────────────────────────────┐
│ Application code │
└──────────────┬───────────────────────────┬───────────┘
│ │
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Orbit crates │ uses │ async-nats (core) │
│ (opinionated, │──────▶│ (parity, stable, │
│ per-crate semver) │ │ protocol-level) │
└───────────────────┘ └─────────┬─────────┘
│
▼
┌─────────────┐
│ nats-server │
└─────────────┘
```
## Feature Flags
Features are extensive and control which subsystems are compiled:
| Feature | Default | Description |
|---------|---------|-------------|
| `jetstream` | ✅ | JetStream API (streams, consumers, publish) |
| `kv` | ✅ | Key-Value store (depends on `jetstream`) |
| `object-store` | ✅ | Object store (depends on `jetstream` + `crypto`) |
| `service` | ✅ | Service API (microservice pattern) |
| `nkeys` | ✅ | NKey/JWT authentication |
| `nuid` | ✅ | NUID-based unique ID generation |
| `crypto` | ✅ | Cryptographic primitives (SHA-256 for object store) |
| `websockets` | ✅ | WebSocket transport (`ws://`/`wss://`) |
| `ring` | ✅ | Use `ring` as TLS crypto backend |
| `aws-lc-rs` | ❌ | Use `aws-lc-rs` as TLS crypto backend |
| `fips` | ❌ | FIPS 140-2 compliant via `aws-lc-rs` |
| `chrono` | ❌ | Use `chrono` instead of `time` for datetime types |
| `server_2_10` | ✅ | Server 2.10+ features |
| `server_2_11` | ✅ | Server 2.11+ features |
| `server_2_12` | ✅ | Server 2.12+ features |
| `server_2_14` | ✅ | Server 2.14+ features |
| `experimental` | ❌ | Experimental features |
## Source Structure
```
async-nats/src/
├── lib.rs # Entry point: connect(), ServerInfo, Command, ClientOp, ServerOp,
│ ConnectionHandler, Subscriber, Event, ServerAddr, ConnectInfo
├── client.rs # Client struct, publish/subscribe/request/drain/flush APIs,
│ Request builder, Statistics, trait definitions
├── connection.rs # Framed connection: NATS protocol parser/serializer,
│ read/write buffer management, WebSocket adapter
├── connector.rs # Server pool, reconnection logic, TLS setup, DNS resolution,
│ authentication handshake
├── options.rs # ConnectOptions builder, auth methods, TLS config, callbacks
├── auth.rs # Auth struct (username, password, token, JWT, nkey, signature)
├── auth_utils.rs # Credentials file parsing (JWT + NKey seed)
├── message.rs # Message (inbound), OutboundMessage (outbound)
├── header.rs # HeaderMap, HeaderName, HeaderValue (NATS headers)
├── subject.rs # Subject type, ToSubject trait, SubjectError
├── status.rs # StatusCode enum (NATS status codes)
├── error.rs # Generic Error<K> type used throughout
├── datetime.rs # DateTime type (time or chrono backend)
├── id_generator.rs # Unique ID generation (NUID or rand fallback)
├── tls.rs # TLS configuration helper
├── crypto.rs # SHA-256 for object store integrity
├── jetstream/
│ ├── mod.rs # Module entry: new(), with_domain(), with_prefix()
│ ├── context.rs # Context: JetStream API (streams, consumers, KV, OS, publish)
│ ├── stream.rs # Stream handle, Config, Info, purge/delete/message ops
│ ├── consumer/
│ │ ├── mod.rs # Consumer trait, Info, Config base
│ │ ├── pull.rs # PullConsumer: batch fetch, sequence, messages stream
│ │ └── push.rs # PushConsumer: Ordered push consumer with auto-recreate
│ ├── publish.rs # PublishAck, PublishAckFuture, PublishMessage builder
│ ├── message.rs # JetStream Message (with ack methods), AckKind
│ ├── response.rs # Response<T> (Ok/Err) for JetStream API calls
│ ├── errors.rs # ErrorCode, Error for JetStream
│ ├── account.rs # Account info
│ ├── kv/
│ │ ├── mod.rs # Store: put/get/delete/purge/watch/history/keys
│ │ └── bucket.rs # Bucket Status
│ └── object_store/
│ └── mod.rs # ObjectStore: put/get/delete/watch/list/seal, Object (AsyncRead)
└── service/
├── mod.rs # Service, ServiceBuilder, Group, EndpointBuilder, Request
└── endpoint.rs # Endpoint stream, Stats, Info
```
## Architecture: Core Connection Model
The client uses a **single-connection, actor-model** design:
```
┌──────────────────────────────────────┐
Client (clone) ──▶│ mpsc::Sender<Command> │
(many handles) │ (bounded channel) │
└────────────┬────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ ConnectionHandler (tokio::task) │
│ - Receives Command from channel │
│ - Converts to ClientOp │
│ - Manages subscriptions map │
│ - Manages multiplexer (request/reply)│
│ - Pings server on interval │
│ - Handles reconnection │
└────────────┬──────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ Connection (framed TCP/TLS/WS) │
│ - Protocol parser (try_read_op) │
│ - Write buffer (VecDeque<Bytes>) │
│ - Vectored I/O support │
│ - Read buffer (BytesMut) │
└────────────┬──────────────────────────┘
│
▼
nats-server
```
### Key Design Decisions
1. **Cloneable Client**: `Client` is `Clone` (via `mpsc::Sender` clone), enabling shared use across tasks
2. **Single TCP connection**: All traffic (Core NATS, JetStream API, etc.) multiplexes over one connection
3. **Background task**: `ConnectionHandler` runs as a spawned Tokio task, bridging the mpsc channel to the TCP stream
4. **Automatic reconnection**: On disconnect, `Connector` retries servers from the pool with exponential backoff
5. **Subscription rehydration**: On reconnect, all active subscriptions are re-subscribed with adjusted `max` counts
6. **Multiplexer for request/reply**: A single wildcard subscription (`_INBOX.<id>.*`) multiplexes all pending request/reply correlations
## Dependencies (Key)
| Crate | Purpose |
|-------|---------|
| `tokio` | Async runtime, TCP, time, sync, io-util |
| `bytes` | Efficient byte buffer (`Bytes`, `BytesMut`) |
| `tokio-rustls` | TLS via rustls |
| `rustls-native-certs` | Load system root certificates |
| `serde` / `serde_json` | JSON serialization for JetStream API |
| `futures-util` | Stream trait, Sink trait, StreamExt |
| `tracing` | Structured logging |
| `thiserror` | Error derive macros |
| `memchr` | Fast substring search for protocol parsing |
| `portable-atomic` | Atomic types with portable-atomic fallback |
| `tokio-util` | `PollSender` for Sink implementation |
| `tokio-stream` | `ReceiverStream` adapter |

404
docs/research/references/nats.rs/nats-async/02-key-types-and-traits.md Normal file
View File

@@ -0,0 +1,404 @@
# async-nats: Key Types & Traits
## Core Types
### `Client`
The primary handle to a NATS connection. Cheaply cloneable (wraps `mpsc::Sender<Command>`).
```rust
#[derive(Clone, Debug)]
pub struct Client {
info: tokio::sync::watch::Receiver<Option<ServerInfo>>,
state: tokio::sync::watch::Receiver<State>,
sender: mpsc::Sender<Command>,
poll_sender: PollSender<Command>,
next_subscription_id: Arc<AtomicU64>,
subscription_capacity: usize,
inbox_prefix: Arc<str>,
request_timeout: Option<Duration>,
max_payload: Arc<AtomicUsize>,
connection_stats: Arc<Statistics>,
skip_subject_validation: bool,
}
```
**Key methods**:
- `publish(subject, payload)` — fire-and-forget publish
- `publish_with_headers(subject, headers, payload)` — publish with NATS headers
- `publish_with_reply(subject, reply, payload)` — publish with reply-to subject
- `subscribe(subject)` → `Subscriber` — subscribe to a subject
- `queue_subscribe(subject, queue_group)` → `Subscriber` — queue group subscription
- `request(subject, payload)` → `Message` — request/reply with default timeout
- `send_request(subject, request)` → `Message` — request with custom `Request` builder
- `flush()` — wait until all buffered writes are flushed to the server
- `drain()` — drain all subscriptions, flush, then close
- `force_reconnect()` — force a reconnection (e.g., to re-trigger auth)
- `new_inbox()` — generate a unique inbox subject (`_INBOX.<id>`)
- `server_info()` → `ServerInfo` — last known server info
- `connection_state()` → `State` — `Pending`/`Connected`/`Disconnected`
- `statistics()` → `Arc<Statistics>` — connection statistics (bytes, messages, connects)
- `max_payload()` → `usize` — server's max payload size
- `set_server_pool(addrs)` — replace the server pool for reconnection
- `server_pool()` — snapshot of current server pool
### `Subscriber`
A `Stream` yielding `Message` values from a subscription.
```rust
#[derive(Debug)]
pub struct Subscriber {
sid: u64,
receiver: mpsc::Receiver<Message>,
sender: mpsc::Sender<Command>,
}
```
Implements `futures_util::Stream<Item = Message>`. Methods:
- `unsubscribe()` — immediately unsubscribe
- `unsubscribe_after(n)` — unsubscribe after `n` total delivered messages
- `drain()` — unsubscribe after in-flight messages are delivered
**Drop behavior**: When a `Subscriber` is dropped, it spawns a task to send `Command::Unsubscribe` to the connection handler, ensuring the server is always notified.
### `Message`
An inbound NATS message:
```rust
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
### `OutboundMessage`
An outbound message for publishing (no status/description):
```rust
#[derive(Clone, Debug)]
pub struct OutboundMessage {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
}
```
### `Request`
Builder for request/reply calls:
```rust
#[derive(Default)]
pub struct Request {
pub payload: Option<Bytes>,
pub headers: Option<HeaderMap>,
pub timeout: Option<Option<Duration>>,
pub inbox: Option<String>,
}
```
Builder methods: `payload()`, `headers()`, `timeout()`, `inbox()`. The `inbox` field, when set, bypasses the multiplexer and uses a dedicated subscription instead.
### `ServerInfo`
Server metadata received during connection handshake:
```rust
#[derive(Debug, Deserialize, Default, Clone, Eq, PartialEq)]
pub struct ServerInfo {
pub server_id: String,
pub server_name: String,
pub host: String,
pub port: u16,
pub version: String,
pub auth_required: bool,
pub tls_required: bool,
pub max_payload: usize,
pub proto: i8,
pub client_id: u64,
pub go: String,
pub nonce: String,
pub connect_urls: Vec<String>,
pub client_ip: String,
pub headers: bool,
pub lame_duck_mode: bool,
pub cluster: Option<String>,
pub domain: Option<String>,
pub jetstream: bool,
}
```
### `ConnectInfo`
Client → server `CONNECT` message payload:
```rust
#[derive(Clone, Debug, Serialize)]
pub struct ConnectInfo {
pub verbose: bool,
pub pedantic: bool,
pub user_jwt: Option<String>,
pub nkey: Option<String>,
pub signature: Option<String>,
pub name: Option<String>,
pub echo: bool,
pub lang: String,
pub version: String,
pub protocol: Protocol, // Original(0) or Dynamic(1)
pub tls_required: bool,
pub user: Option<String>,
pub pass: Option<String>,
pub auth_token: Option<String>,
pub headers: bool,
pub no_responders: bool,
}
```
The client always sets: `verbose=false`, `pedantic=false`, `lang="rust"`, `protocol=Dynamic`, `headers=true`, `no_responders=true`.
### `Statistics`
Atomic connection statistics (shared via `Arc`):
```rust
#[derive(Default, Debug)]
pub struct Statistics {
pub in_bytes: AtomicU64,
pub out_bytes: AtomicU64,
pub in_messages: AtomicU64,
pub out_messages: AtomicU64,
pub connects: AtomicU64,
}
```
## Subject Types
### `Subject`
A validated NATS subject string (newtype over `String`):
```rust
// Usage:
let subject: Subject = "foo.bar.baz".into();
```
### `ToSubject` trait
Conversion trait for subjects:
```rust
pub trait ToSubject {
fn to_subject(self) -> Result<Subject, SubjectError>;
}
```
Implemented for `&str`, `String`, `Subject` directly.
### `SubjectError`
```rust
pub enum SubjectError {
InvalidFormat,
}
```
## Header Types
### `HeaderMap`
A multimap of header name → values:
```rust
pub struct HeaderMap {
inner: VecMap<HeaderName, Vec<HeaderValue>>,
}
```
Methods: `insert()`, `append()`, `get()`, `len()`, `is_empty()`, `iter()`, `to_bytes()`.
### `HeaderName`
Case-insensitive header name. Created via `FromStr`:
```rust
let name: HeaderName = "Nats-Expected-Last-Subject-Sequence".parse()?;
```
### `HeaderValue`
Header value string. Created via `FromStr` or `From<u64>`:
```rust
let val: HeaderValue = "some value".parse()?;
let val: HeaderValue = HeaderValue::from(42u64);
```
## Server Address Types
### `ServerAddr`
Wraps a `url::Url` with NATS-specific validation. Supports schemes: `nats://`, `tls://`, `ws://`, `wss://`. Default port is `4222`.
```rust
let addr: ServerAddr = "demo.nats.io".parse()?;
let addr: ServerAddr = "nats://demo.nats.io:4222".parse()?;
let addr: ServerAddr = "tls://demo.nats.io".parse()?;
```
### `ToServerAddrs` trait
Flexible server address input (single URL, `Vec`, slice, etc.):
```rust
pub trait ToServerAddrs {
type Iter: Iterator<Item = ServerAddr>;
fn to_server_addrs(&self) -> io::Result<Self::Iter>;
}
```
### `Server`
Metadata about a server in the pool:
```rust
pub struct Server {
pub addr: ServerAddr,
pub failed_attempts: usize,
pub did_connect: bool,
pub is_discovered: bool,
pub last_error: Option<String>,
}
```
## Event & State Types
### `Event`
Asynchronous notifications from the connection:
```rust
pub enum Event {
Connected,
Disconnected,
LameDuckMode,
Draining,
Closed,
SlowConsumer(u64), // subscription sid
ServerError(ServerError),
ClientError(ClientError),
}
```
Received via `ConnectOptions::event_callback()`.
### `State`
Connection state observable via `watch::Receiver`:
```rust
pub enum State {
Pending,
Connected,
Disconnected,
}
```
### `StatusCode`
NATS protocol status codes (e.g., `NO_RESPONDERS = 404`, `TIMEOUT = 408`).
## Error Types
All error types follow the pattern `Error<Kind>` from `crate::error`:
| Error Type | Kind | Used By |
|------------|------|---------|
| `ConnectError` | `ConnectErrorKind` | Connection establishment |
| `PublishError` | `PublishErrorKind` | Publish operations |
| `RequestError` | `RequestErrorKind` | Request/reply |
| `SubscribeError` | `SubscribeErrorKind` | Subscribe |
| `FlushError` | `FlushErrorKind` | Flush |
| `DrainError` | — | Drain |
### `ConnectErrorKind`
```rust
pub enum ConnectErrorKind {
ServerParse, // URL parsing failed
Dns, // DNS resolution failed
Authentication, // Auth signing failed
AuthorizationViolation, // Server rejected auth
TimedOut, // Connection handshake timeout
Tls, // TLS error
Io, // Other I/O error
MaxReconnects, // Exceeded max reconnect attempts
}
```
## Trait Definitions
The `client::traits` module defines abstract interfaces:
```rust
pub trait Publisher {
fn publish_with_reply(&self, subject, reply, payload) -> Future<Output = Result<(), PublishError>>;
fn publish_message(&self, msg: OutboundMessage) -> Future<Output = Result<(), PublishError>>;
}
pub trait Subscriber {
fn subscribe(&self, subject) -> Future<Output = Result<crate::Subscriber, SubscribeError>>;
}
pub trait Requester {
fn send_request(&self, subject, request: Request) -> Future<Output = Result<Message, RequestError>>;
}
pub trait TimeoutProvider {
fn timeout(&self) -> Option<Duration>;
}
```
`Client` implements all of these. The JetStream `Context` also implements them via delegation.
## Authentication Types
### `Auth`
Container for all authentication methods:
```rust
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
### `AuthError`
Simple string error for auth callback failures.
### `ReconnectToServer`
Returned by `reconnect_to_server_callback` to select a server and delay:
```rust
pub struct ReconnectToServer {
pub addr: ServerAddr,
pub delay: Option<Duration>,
}
```

278
docs/research/references/nats.rs/nats-async/03-protocol-and-wire-format.md Normal file
View File

@@ -0,0 +1,278 @@
# async-nats: NATS Protocol & Wire Format
## Protocol Overview
NATS uses a simple, text-based protocol over TCP. Messages are terminated with `\r\n`. The protocol is symmetric for client and server operations.
### Client → Server Operations (`ClientOp`)
```rust
pub(crate) enum ClientOp {
Publish { subject, payload, respond, headers },
Subscribe { sid, subject, queue_group },
Unsubscribe { sid, max },
Ping,
Pong,
Connect(ConnectInfo),
}
```
### Server → Client Operations (`ServerOp`)
```rust
pub(crate) enum ServerOp {
Ok,
Info(Box<ServerInfo>),
Ping,
Pong,
Error(ServerError),
Message { sid, subject, reply, payload, headers, status, description, length },
}
```
## Wire Format: Client Operations
### CONNECT
Sent immediately after receiving the first `INFO` from the server:
```
CONNECT {"verbose":false,"pedantic":false,...}\r\n
```
The JSON payload is `ConnectInfo` serialized inline on the same line.
### PUB (Publish without headers)
```
PUB <subject> [reply-to] <payload-size>\r\n
<payload>\r\n
```
Example:
```
PUB events.data INBOX.67 11\r\n
Hello World\r\n
```
### HPUB (Publish with headers)
When headers are present and non-empty:
```
HPUB <subject> [reply-to] <header-size> <total-size>\r\n
<headers>\r\n
<payload>\r\n
```
The `<total-size>` = `<header-size>` + `<payload-size>`.
Header block format:
```
NATS/1.0\r\n
Header-Name: Header-Value\r\n
Another-Header: Another-Value\r\n
\r\n
```
The version line (`NATS/1.0`) may include a status code and description:
```
NATS/1.0 404 No Messages\r\n
\r\n
```
### SUB (Subscribe)
```
SUB <subject> [queue-group] <sid>\r\n
```
The `sid` (subscription ID) is a client-assigned u64, unique per connection.
### UNSUB (Unsubscribe)
```
UNSUB <sid> [max]\r\n
```
The optional `max` tells the server to auto-unsubscribe after `max` messages are delivered.
### PING / PONG
```
PING\r\n
PONG\r\n
```
Client sends PING periodically (default every 60s). If 2+ pings are pending without PONG, the connection is considered dead.
## Wire Format: Server Operations
### INFO
First message sent by the server on connection:
```
INFO {"server_id":"NATSxxx","version":"2.10"...}\r\n
```
Also sent asynchronously when cluster topology changes.
### MSG (Message without headers)
```
MSG <subject> <sid> [reply-to] <payload-size>\r\n
<payload>\r\n
```
### HMSG (Message with headers)
```
HMSG <subject> <sid> [reply-to] <header-size> <total-size>\r\n
<headers + payload>\r\n
```
### +OK / -ERR
```
+OK\r\n
-ERR <description>\r\n
```
Sent only when `verbose=true` in `CONNECT`. The client always sets `verbose=false`, so `+OK` is not expected.
## Protocol Parser
The `Connection` struct handles all protocol parsing and serialization:
### Read Path (`try_read_op`)
1. Search for `\r\n` in `read_buf` using `memchr::memmem::find`
2. Inspect the first bytes to determine the operation type:
- `+OK` → `ServerOp::Ok`
- `PING` → `ServerOp::Ping`
- `PONG` → `ServerOp::Pong`
- `-ERR` → `ServerOp::Error(...)` (description is `trim_matches('\'')`)
- `INFO ` → `ServerOp::Info(...)` (serde_json deserialization)
- `MSG ` → Parse subject/sid/reply/size, then read payload
- `HMSG ` → Parse subject/sid/reply/header_len/total_len, then read headers + payload
3. For `MSG`/`HMSG`: if the full message body hasn't been read yet, return `None` (wait for more data)
4. For `HMSG`: parse the header block — extract version line (`NATS/1.0[ <status>[ <description>]]`), then key-value pairs (supports folded/multi-line header values)
### Write Path (`enqueue_write_op`)
Writes into a buffer strategy:
- **Small writes** (< 4096 bytes): flattened into `flattened_writes: BytesMut`
- **Large writes** (≥ 4096 bytes): appended as separate `Bytes` chunks in `write_buf: VecDeque<Bytes>`
This enables efficient vectored I/O when the underlying stream supports it.
### Write Flush Strategy
The `should_flush()` method returns:
- `Yes` — buffers empty but haven't flushed yet
- `May` — buffers not empty and haven't flushed
- `No` — already flushed or nothing to flush
The `ConnectionHandler` calls `poll_flush()` after processing commands, ensuring data is actually sent to the server.
## Vectored I/O
When `stream.is_write_vectored()` returns true, the connection uses `poll_write_vectored()` to write up to 64 `IoSlice`s at once. This is significantly more efficient for bursty publish patterns.
```rust
const WRITE_VECTORED_CHUNKS: usize = 64;
```
## WebSocket Transport
When the `websockets` feature is enabled, `WebSocketAdapter<T>` wraps `tokio_websockets::WebSocketStream<T>` to implement `AsyncRead + AsyncWrite`, making WebSocket connections transparent to the protocol layer.
```rust
#[cfg(feature = "websockets")]
pub(crate) struct WebSocketAdapter<T> {
pub(crate) inner: WebSocketStream<T>,
pub(crate) read_buf: BytesMut,
}
```
WebSocket connections use `ws://` or `wss://` scheme in the server URL. TLS for `wss://` is handled by the WebSocket library's built-in TLS support.
## Connection Lifecycle
### Initial Connection Flow
```
Client Server
│ │
│──── TCP connect ────────────────────▶ │
│◀──── INFO {server_id, nonce, ...} ─── │
│──── CONNECT {auth, ...} ──────────▶ │
│──── PING ─────────────────────────▶ │
│◀──── PONG (or -ERR) ─────────────── │
│ │
│ [connected, ConnectionHandler runs] │
```
If `tls_first` is enabled, TLS is established before reading INFO:
```
Client Server
│ │
│──── TCP connect ────────────────────▶ │
│──── TLS handshake ─────────────────▶ │
│◀──── TLS handshake ──────────────── │
│◀──── INFO {...} ──────────────────── │
│──── CONNECT + PING ────────────────▶ │
│◀──── PONG ────────────────────────── │
```
### Ping/Pong Keepalive
- Client sends PING every `ping_interval` (default 60s)
- Server responds with PONG
- If `pending_pings > MAX_PENDING_PINGS (2)`, connection is considered dead
- Any server operation resets the ping interval timer
### Reconnection Flow
On disconnect:
1. `handle_disconnect()` sends `Event::Disconnected` and sets state to `Disconnected`
2. `handle_reconnect()` calls `connector.connect()` which:
- Shuffles servers (unless `retain_servers_order`)
- Sorts by `failed_attempts` (ascending)
- Iterates through servers with exponential backoff delay
- On each server: DNS resolve → TCP connect → INFO → TLS (if needed) → CONNECT+PING → PONG
3. On success:
- Sends `Event::Connected`, sets state to `Connected`
- Removes closed subscriptions
- Re-subscribes all active subscriptions (with adjusted `max = max - delivered`)
- Re-subscribes the multiplexer (if active)
4. On failure with `MaxReconnects` reached, the handler loop exits
### Default Reconnect Delay
Exponential backoff capped at 4 seconds:
```rust
fn reconnect_delay_callback_default(attempts: usize) -> Duration {
if attempts <= 1 {
Duration::from_millis(0)
} else {
let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX);
cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), Duration::from_secs(4))
}
}
```
| Attempt | Delay |
|---------|-------|
| 1 | 0ms |
| 2 | 0ms |
| 3 | 2ms |
| 4 | 8ms |
| 5 | 32ms |
| 6 | 128ms |
| 7 | 512ms |
| 8 | 2048ms |
| 9+ | 4000ms (cap) |

221
docs/research/references/nats.rs/nats-async/04-connection-management.md Normal file
View File

@@ -0,0 +1,221 @@
# async-nats: Connection Management & Configuration
## ConnectOptions Builder
`ConnectOptions` provides a builder for all connection configuration:
```rust
let client = ConnectOptions::new()
.require_tls(true)
.ping_interval(Duration::from_secs(10))
.name("my-service")
.connect("demo.nats.io")
.await?;
```
### Authentication Methods
| Method | Description |
|--------|-------------|
| `with_token(token)` | Token-based auth |
| `with_user_and_password(user, pass)` | Username/password auth |
| `with_nkey(seed)` | NKey auth (requires `nkeys` feature) |
| `with_jwt(jwt, sign_cb)` | JWT + signing callback (requires `nkeys`) |
| `with_credentials_file(path)` | Load from `.creds` file (requires `nkeys`) |
| `with_credentials(creds_str)` | Parse credentials string (requires `nkeys`) |
| `with_auth_callback(cb)` | Dynamic auth callback receiving nonce, returning `Auth` |
The auth callback is the most flexible — it receives the server nonce and can return any combination of auth fields:
```rust
ConnectOptions::with_auth_callback(move |nonce| async move {
let mut auth = Auth::new();
auth.username = Some("user".to_string());
auth.password = Some("pass".to_string());
Ok(auth)
})
```
### TLS Configuration
| Option | Description |
|-------|-------------|
| `require_tls(bool)` | Require TLS for the connection |
| `tls_first()` | Establish TLS before INFO (requires server `handshake_first`) |
| `add_root_certificates(path)` | Load root CA certificates from PEM file |
| `add_client_certificate(cert, key)` | Load client certificate for mTLS |
| `tls_client_config(config)` | Pass a custom `rustls::ClientConfig` |
Two TLS crypto backends: `ring` (default) or `aws-lc-rs` (via feature flags). FIPS mode available via `aws-lc-rs` + `fips` features.
### Connection Behavior
| Option | Default | Description |
|--------|---------|-------------|
| `connection_timeout` | 5s | Timeout for full connection establishment |
| `request_timeout` | 10s | Default timeout for `Client::request` |
| `ping_interval` | 60s | How often client sends PING |
| `retry_on_initial_connect` | false | Return client immediately, connect in background |
| `max_reconnects` | None (unlimited) | Max consecutive reconnect attempts |
| `ignore_discovered_servers` | false | Ignore servers advertised in INFO |
| `retain_servers_order` | false | Don't shuffle server list on reconnect |
| `skip_subject_validation` | false | Skip whitespace validation on publish subjects |
| `subscription_capacity` | 65536 | mpsc channel capacity per subscription |
| `client_capacity` | 2048 | mpsc channel capacity for command sender |
| `custom_inbox_prefix` | `_INBOX` | Custom prefix for inbox subjects |
| `read_buffer_capacity` | 65535 | Initial size of the protocol read buffer |
| `local_address` | None | Local socket address to bind to |
| `no_echo` | false | Don't deliver messages published by this connection |
### Reconnection Callbacks
**`reconnect_delay_callback`**: Custom backoff strategy:
```rust
.reconnect_delay_callback(|attempts| {
Duration::from_millis(std::cmp::min((attempts * 100) as u64, 8000))
})
```
**`reconnect_to_server_callback`**: Select which server to connect to on each reconnect attempt:
```rust
.reconnect_to_server_callback(|servers, _info| async move {
servers.first().map(|s| ReconnectToServer {
addr: s.addr.clone(),
delay: Some(Duration::ZERO),
})
})
```
Receives `(Vec<Server>, ServerInfo)`, returns `Option<ReconnectToServer>`. If the returned server isn't in the pool, falls back to default selection.
**`event_callback`**: Receive async notifications:
```rust
.event_callback(|event| async move {
match event {
Event::Disconnected => println!("disconnected"),
Event::Connected => println!("connected"),
Event::SlowConsumer(sid) => eprintln!("slow consumer: {sid}"),
_ => {}
}
})
```
## Connection Handler Internals
### ProcessFut — The Core Event Loop
The `ConnectionHandler::process()` method creates a custom `Future` (`ProcessFut`) that drives the connection forward. Each `poll()` call:
1. **Check ping interval** — if timer ticked, send PING; if too many pending pings, disconnect
2. **Read server operations** — drain all available `ServerOp`s from `Connection::poll_read_op()`
3. **Process drain completions** — remove subscriptions that finished draining
4. **Handle commands** — receive up to 16 `Command`s from the mpsc channel and process them
5. **Write to socket** — flush the write buffer via `Connection::poll_write()`
6. **Flush** — call `poll_flush()` on the underlying stream when needed
7. **Check reconnect flag** — if `should_reconnect` is set, shut down and reconnect
```rust
const RECV_CHUNK_SIZE: usize = 16;
```
### Exit Reasons
The event loop exits with one of:
| Reason | Action |
|--------|--------|
| `Disconnected(Option<io::Error>)` | Attempt reconnection |
| `ReconnectRequested` | Shut down stream, attempt reconnection |
| `Closed` | Send `Event::Closed`, exit loop |
### Handle Disconnect & Reconnect
```rust
async fn handle_disconnect(&mut self) -> Result<(), ConnectError> {
self.pending_pings = 0;
self.connector.events_tx.try_send(Event::Disconnected).ok();
self.connector.state_tx.send(State::Disconnected).ok();
self.handle_reconnect().await
}
async fn handle_reconnect(&mut self) -> Result<(), ConnectError> {
let (info, connection) = self.connector.connect().await?;
self.connection = connection;
let _ = self.info_sender.send(Some(info));
// Remove closed subscriptions
self.subscriptions.retain(|_, sub| !sub.sender.is_closed());
// Re-subscribe all active subscriptions
for (sid, subscription) in &self.subscriptions {
self.connection.enqueue_write_op(&ClientOp::Subscribe {
sid: *sid,
subject: subscription.subject.to_owned(),
queue_group: subscription.queue_group.to_owned(),
});
if let Some(max) = subscription.max {
self.connection.enqueue_write_op(&ClientOp::Unsubscribe {
sid: *sid,
max: Some(max.saturating_sub(subscription.delivered)),
});
}
}
// Re-subscribe multiplexer if active
if let Some(multiplexer) = &self.multiplexer {
self.connection.enqueue_write_op(&ClientOp::Subscribe {
sid: MULTIPLEXER_SID,
subject: multiplexer.subject.to_owned(),
queue_group: None,
});
}
Ok(())
}
```
## Request/Reply Multiplexer
The client uses a **multiplexer** pattern for request/reply to avoid creating a separate subscription per request:
1. A single wildcard subscription is created on first request: `_INBOX.<random_id>.*`
2. Each request gets a unique token appended to the inbox: `_INBOX.<random_id>.<token>`
3. When a response arrives, the token is extracted from the subject and used to look up the `oneshot::Sender` in `multiplexer.senders`
4. The response is forwarded through the oneshot channel to the waiting `send_request()` future
```rust
struct Multiplexer {
subject: Subject, // _INBOX.<id>.*
prefix: Subject, // _INBOX.<id>.
senders: HashMap<String, oneshot::Sender<Message>>, // token → sender
}
```
The multiplexer subscription uses `sid = 0` (`MULTIPLEXER_SID`), which is separate from regular subscription IDs (which start at 1).
### Custom Inbox Bypass
If a `Request` has a custom `inbox` set, the multiplexer is bypassed — a dedicated subscription is created for that specific request, and the timeout/response logic is handled locally within `send_request()`.
## Server Pool Management
The `Connector` maintains a `Vec<Server>` pool. Servers can come from:
1. **Explicit URLs** — provided by the user at connect time
2. **Discovered servers** — advertised in `INFO.connect_urls` (unless `ignore_discovered_servers` is set)
On reconnection:
- Servers are shuffled (unless `retain_servers_order`)
- Sorted by `failed_attempts` (ascending) — prefer servers that haven't failed recently
- Each server is tried with exponential backoff delay
- On success: `failed_attempts` reset to 0, `did_connect` set to true
- On failure: `failed_attempts` incremented, `last_error` updated
### Dynamic Server Pool Updates
`Client::set_server_pool()` replaces the pool at runtime:
- Per-server state is preserved for servers that appear in both old and new pools
- The global reconnection attempt counter is reset
- Cannot mix WebSocket and non-WebSocket URLs
- Pool cannot be empty

373
docs/research/references/nats.rs/nats-async/05-jetstream.md Normal file
View File

@@ -0,0 +1,373 @@
# async-nats: JetStream
## Overview
JetStream is NATS' built-in persistence layer, providing stream-based messaging with at-least-once and exactly-once delivery semantics. The `async-nats` JetStream API is accessed through a `Context` object.
### Creating a Context
```rust
// Default context (prefix: $JS.API)
let jetstream = async_nats::jetstream::new(client);
// With domain (prefix: $JS.<domain>.API)
let jetstream = async_nats::jetstream::with_domain(client, "hub");
// With custom prefix
let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API");
// Builder with fine-grained control
let context = ContextBuilder::new()
.timeout(Duration::from_secs(5))
.api_prefix("MY.JS.API")
.max_ack_inflight(1000)
.backpressure_on_inflight(true)
.ack_timeout(Duration::from_secs(30))
.build(client);
```
## Context
```rust
#[derive(Debug, Clone)]
pub struct Context {
pub(crate) client: Client,
pub(crate) prefix: String,
pub(crate) timeout: Duration,
pub(crate) max_ack_semaphore: Arc<tokio::sync::Semaphore>,
pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver<Message>, OwnedSemaphorePermit)>,
pub(crate) backpressure_on_inflight: bool,
pub(crate) semaphore_capacity: usize,
}
```
### Publish Backpressure
The context uses a semaphore to limit the number of pending publish acknowledgments:
- `max_ack_inflight(n)` — sets semaphore capacity (default 5000)
- `backpressure_on_inflight(true)` — `publish()` waits for a permit when limit is reached
- `backpressure_on_inflight(false)` — `publish()` returns `MaxAckPending` error immediately when limit is reached
A background **acker task** monitors pending acks with a timeout (`ack_timeout`, default 30s), releasing permits when acks arrive or time out.
### JetStream API Request Pattern
All JetStream API calls follow the same pattern:
1. Build a subject from the prefix: `format!("{}.STREAM.CREATE.<name>", self.prefix)`
2. Serialize the request payload as JSON
3. Send a request via `client.send_request()` with the API subject
4. Deserialize the response as `Response<T>` (which is `Ok(T)` or `Err(ErrorCode)`)
## Streams
### Stream Handle
```rust
pub struct Stream<I = Info> {
context: Context,
info: I,
name: String,
}
```
`Stream<Info>` carries server-side info. `Stream<()>` is a lightweight handle that skips the INFO fetch. `Stream` (no generic) defaults to `Stream<Info>`.
### Stream Config
```rust
pub struct Config {
pub name: String,
pub description: Option<String>,
pub subjects: Vec<String>,
pub retention: RetentionPolicy,
pub max_consumers: i64,
pub max_messages: i64,
pub max_messages_per_subject: i64,
pub max_bytes: i64,
pub max_age: Duration,
pub max_messages_per_stream: i64,
pub max_msg_size: i32,
pub discard: DiscardPolicy,
pub discard_new_per_subject: bool,
pub storage: StorageType,
pub num_replicas: usize,
pub no_ack: bool,
pub duplicate_window: Duration,
pub placement: Option<Placement>,
pub mirror: Option<Source>,
pub sources: Option<Vec<Source>>,
pub sealed: bool,
pub allow_direct: bool,
pub allow_rollup_hdrs: bool,
// server_2_10 features:
pub compression: Option<Compression>,
pub first_sequence: Option<u64>,
pub subject_transform: Option<SubjectTransform>,
pub republish: Option<Republish>,
pub metadata: Option<HashMap<String, String>>,
}
```
### Stream Operations
| Method | Description |
|--------|-------------|
| `create_stream(config)` | Create a new stream |
| `get_stream(name)` | Get stream handle (with INFO) |
| `get_stream_no_info(name)` | Get lightweight handle (no server round-trip) |
| `get_or_create_stream(config)` | Get existing or create new |
| `delete_stream(name)` | Delete a stream |
| `update_stream(config)` | Update stream configuration |
| `create_or_update_stream(config)` | Update or create if not found |
| `stream_names()` | `Stream` of stream names (paginated) |
| `streams()` | `Stream` of stream info (paginated) |
| `stream_by_subject(subject)` | Find stream name containing subject |
### Stream Handle Methods
```rust
let stream: Stream = jetstream.get_stream("events").await?;
// Info
let info: Info = stream.info().await?; // Fresh info from server
let info: &Info = stream.cached_info(); // Cached info from last fetch
// Message operations
stream.get_raw_message(seq).await?; // Get raw message by sequence
stream.get_last_raw_message_by_subject(subj).await?; // Get last message for subject
stream.direct_get(seq).await?; // Direct get (if allow_direct)
stream.direct_get_last_for_subject(subj).await?; // Direct last by subject
stream.delete_message(seq).await?; // Delete a specific message
stream.purge().await?; // Purge all messages
stream.purge().filter(subj).await?; // Purge messages for subject
// Consumers
stream.create_consumer(config).await?; // Create consumer bound to stream
stream.get_consumer(name).await?; // Get existing consumer
stream.delete_consumer(name).await?; // Delete consumer
```
## Consumers
### Consumer Types
Two consumer types, each with distinct delivery models:
1. **Pull Consumer** (`pull::Config` / `PullConsumer`) — Client explicitly requests batches of messages
2. **Push Consumer** (`push::Config` / `PushConsumer`) — Server pushes messages to a deliver subject
### Pull Consumer
```rust
let consumer: PullConsumer = stream
.get_or_create_consumer("my-consumer", pull::Config {
durable_name: Some("my-consumer".to_string()),
..Default::default()
})
.await?;
```
**Key methods**:
- `consumer.batch(n).await?` — Fetch up to `n` messages (one-shot batch)
- `consumer.messages().await?` — Continuous `Stream` of messages
- `consumer.sequence(n).await?` — Continuous `Stream` of batches of `n` messages
- `consumer.fetch().max(n).expires(dur).await?` — Configurable fetch
Each message from a pull consumer is a `jetstream::Message` which has `ack()` methods.
### Push Consumer
Two push consumer variants:
1. **Standard** (`push::Config`) — messages delivered to a specific subject
2. **Ordered** (`push::OrderedConfig`) — auto-recreated on failure, with flow control
```rust
// Standard push
let consumer = stream.create_consumer(push::Config {
deliver_subject: "deliver.subject".to_string(),
durable_name: Some("push-consumer".to_string()),
..Default::default()
}).await?;
// Ordered push (no durable name, auto-recreates on failure)
let consumer = stream.create_consumer(push::OrderedConfig {
deliver_subject: client.new_inbox(),
filter_subject: "events.>".to_string(),
..Default::default()
}).await?;
```
### Consumer Config (Shared Fields)
```rust
pub struct Config {
// Pull fields
pub durable_name: Option<String>,
pub name: Option<String>,
// Push fields
pub deliver_subject: Option<String>,
pub deliver_group: Option<String>,
pub deliver_policy: DeliverPolicy,
pub opt_start_time: Option<DateTime>,
pub opt_start_sequence: Option<u64>,
pub ack_policy: AckPolicy,
pub ack_wait: Duration,
pub max_deliver: i64,
pub backoff: Vec<Duration>,
pub filter_subject: String,
pub filter_subjects: Vec<String>, // server_2_10+
pub replay_policy: ReplayPolicy,
pub rate_limit_bps: Option<u64>,
pub max_waiting: i64, // pull: max outstanding pull requests
pub max_ack_pending: i64,
pub flow_control: bool,
pub idle_heartbeat: Duration,
pub headers_only: bool,
pub num_replicas: usize,
pub mem_storage: bool,
pub description: Option<String>,
pub metadata: Option<HashMap<String, String>>,
pub inactive_threshold: Option<Duration>, // for ephemeral consumers
}
```
### Deliver Policy
```rust
pub enum DeliverPolicy {
All, // Deliver all messages
Last, // Deliver last message only
New, // Deliver only new messages
ByStartSequence { start_sequence: u64 },
ByStartTime { start_time: DateTime },
LastPerSubject, // Deliver last message per subject
}
```
### Ack Policy
```rust
pub enum AckPolicy {
None, // No acknowledgment needed
All, // Ack all messages up to this one
Explicit, // Ack each message individually
}
```
## JetStream Messages
### `jetstream::Message`
Wraps a core `Message` with JetStream-specific metadata:
```rust
pub struct Message {
pub message: crate::Message, // The underlying NATS message
pub ack_subject: Subject, // Subject for sending acks
pub stream: String, // Stream name
pub consumer: String, // Consumer name
pub stream_sequence: u64, // Sequence in stream
pub consumer_sequence: u64, // Sequence for this consumer
pub delivered: u64, // Delivery count
pub pending: u64, // Pending message count
pub published: DateTime, // Original publish time
}
```
### Ack Methods
```rust
// In-memory ack (non-persistent, fast)
message.ack().await?;
// Ack with specific type
message.ack_with(AckKind::Nak).await?;
message.ack_with(AckKind::Progress).await?;
message.ack_with(AckKind::Term).await?;
message.ack_with(AckKind::NakWithDelay(duration)).await?;
message.ack_with(AckKind::TermWithReason("reason")).await?;
```
### `AckKind`
```rust
pub enum AckKind {
Ack, // +ACK — message processed
Nak, // -NAK — re-deliver
Progress, // PRI — still working
Term, // +TERM — don't redeliver
NakWithDelay(Duration), // -NAK with re-delivery delay
TermWithReason(String), // +TERM with reason
}
```
## JetStream Publish
### `Context::publish()`
JetStream publish returns a `PublishAckFuture` — a future that resolves to a `PublishAck`:
```rust
let ack_future = jetstream.publish("events", "data".into()).await?;
let ack: PublishAck = ack_future.await?; // Wait for server acknowledgment
```
### `PublishAck`
```rust
pub struct PublishAck {
pub stream: String,
pub sequence: u64,
pub domain: String,
pub duplicate: bool,
}
```
### `PublishMessage` Builder
```rust
let ack = jetstream.send_publish(
"events",
PublishMessage::build()
.payload("data".into())
.message_id("uuid-123") // Deduplication ID
.expected_stream("events") // Fail if wrong stream
.expected_last_msg_id("prev-id")
.expected_last_sequence(42)
.headers(headers),
).await?;
```
## Pagination
Stream and consumer listing uses pagination internally:
```rust
pub struct StreamNames {
context: Context,
offset: usize,
page_request: Option<Request>,
streams: Vec<String>,
subject: Option<String>,
done: bool,
}
```
Implements `futures_util::Stream<Item = Result<String, Error>>`, lazily fetching pages as needed.
## Error Handling
JetStream errors follow the `Response<T>` pattern:
```rust
pub enum Response<T> {
Ok(T),
Err { error: ErrorCode },
}
```
`ErrorCode` carries the server's error code and description. Most JetStream-specific errors map to typed error enums (e.g., `CreateStreamError`, `ConsumerError`, etc.).

237
docs/research/references/nats.rs/nats-async/06-key-value-store.md Normal file
View File

@@ -0,0 +1,237 @@
# async-nats: Key-Value Store
## Overview
The Key-Value (KV) store is an abstraction built on top of JetStream streams. Each KV bucket is backed by a JetStream stream with the naming convention `KV_<bucket_name>`. Keys are mapped to subjects under the `$KV.<bucket>.<key>` prefix.
The KV feature requires `kv` (which implies `jetstream`).
## Store Handle
```rust
#[derive(Debug, Clone)]
pub struct Store {
pub name: String,
pub stream_name: String,
pub prefix: String, // $KV.<bucket>.
pub put_prefix: Option<String>, // For mirrored buckets
pub use_jetstream_prefix: bool, // Whether to prepend JS API prefix
pub stream: Stream,
}
```
## Bucket Config
```rust
#[derive(Debug, Clone, Default)]
pub struct Config {
pub bucket: String,
pub description: String,
pub max_value_size: i32,
pub history: i64, // Max historical entries per key (1-64)
pub max_age: Duration, // Max age of any entry
pub max_bytes: i64, // Total bucket size limit
pub storage: StorageType, // File or Memory
pub num_replicas: usize,
pub republish: Option<Republish>,
pub mirror: Option<Source>, // Mirror another bucket
pub sources: Option<Vec<Source>>,
pub mirror_direct: bool,
pub compression: bool, // server_2_10+
pub placement: Option<Placement>,
pub limit_markers: Option<Duration>, // server_2_11+
}
```
## Creating/Accessing Buckets
```rust
// Create a new bucket
let kv = jetstream.create_key_value(kv::Config {
bucket: "my-bucket".to_string(),
history: 10,
max_age: Duration::from_secs(3600),
..Default::default()
}).await?;
// Get an existing bucket
let kv = jetstream.get_key_value("my-bucket").await?;
// Create or update
let kv = jetstream.create_or_update_key_value(kv::Config { ... }).await?;
// Delete a bucket
jetstream.delete_key_value("my-bucket").await?;
```
## KV Operations
### Put
```rust
let revision: u64 = kv.put("key", "value".into()).await?;
```
Publishes to `$KV.<bucket>.<key>` (or with JS prefix). The JetStream stream stores it, and the returned sequence number serves as the revision.
### Get
```rust
let value: Option<Bytes> = kv.get("key").await?;
```
Returns `None` if the key doesn't exist or was deleted/purged. Uses either direct get (if `allow_direct`) or the standard message API.
### Entry
```rust
let entry: Option<Entry> = kv.entry("key").await?;
let entry: Option<Entry> = kv.entry_for_revision("key", 2).await?;
```
Returns full entry metadata:
```rust
pub struct Entry {
pub bucket: String,
pub key: String,
pub value: Bytes,
pub revision: u64,
pub created: DateTime,
pub delta: u64,
pub operation: Operation,
pub seen_current: bool,
}
```
### Create (Put if not exists)
```rust
let revision: u64 = kv.create("key", "value".into()).await?;
```
Uses `update` with `expected_last_subject_sequence = 0` (create-only). If the key exists and is deleted/purged, it's re-created.
### Update (Conditional Put)
```rust
let revision: u64 = kv.update("key", "value".into(), last_revision).await?;
```
Uses the `Nats-Expected-Last-Subject-Sequence` header for optimistic concurrency control. Only succeeds if the key's current revision matches.
### Delete
```rust
kv.delete("key").await?;
kv.delete_expect_revision("key", Some(revision)).await?;
```
Non-destructive — publishes a `DEL` marker message. The key appears deleted to `get()`, but history is preserved (up to `history` limit).
### Purge
```rust
kv.purge("key").await?;
kv.purge_with_ttl("key", Duration::from_secs(10)).await?;
kv.purge_expect_revision("key", Some(revision)).await?;
```
Destructive — publishes a `PURGE` marker with rollup header, removing all previous revisions of the key. Leaves a single purge entry.
### Watch
```rust
// Watch for new changes
let mut watch = kv.watch("key").await?;
// Watch with initial value
let mut watch = kv.watch_with_history("key").await?;
// Watch from specific revision
let mut watch = kv.watch_from_revision("key", 5).await?;
// Watch all keys
let mut watch = kv.watch_all().await?;
// Watch multiple keys (server_2_10+)
let mut watch = kv.watch_many(["foo", "bar"]).await?;
```
`Watch` implements `futures_util::Stream<Item = Result<Entry, WatcherError>>`.
Under the hood, each watch creates an **ordered push consumer** on the KV stream with:
- `filter_subject` matching `$KV.<bucket>.<key>`
- `replay_policy: Instant`
- Appropriate `deliver_policy`
### History
```rust
let mut history = kv.history("key").await?;
```
Returns a `Stream` of all past `Entry` values for a key (including deletes/purges).
### Keys
```rust
let mut keys = kv.keys().await?;
```
Returns a `Stream<String>` of all current keys. Uses a headers-only consumer with `LastPerSubject` deliver policy to efficiently scan the bucket.
## Entry Operations
```rust
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
pub enum Operation {
Put, // Value was put
Delete, // Value was deleted (DEL marker)
Purge, // Value was purged (PURGE marker with rollup)
}
```
The operation type is determined from the `KV-Operation` header (`PUT`, `DEL`, `PURGE`) or the `Nats-Marker-Reason` header (fallback for server-generated markers like `MaxAge`, `Purge`, `Remove`).
## Key and Bucket Name Validation
```rust
// Bucket: alphanumeric, dash, underscore only
VALID_BUCKET_RE: \A[a-zA-Z0-9_-]+\z
// Key: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots
VALID_KEY_RE: \A[-/_=\.a-zA-Z0-9]+\z
```
## Bucket Status
```rust
let status: Status = kv.status().await?;
```
Wraps stream info to provide bucket-level statistics (bucket name, message count, byte count, etc.).
## Mirrored Buckets
When a bucket is configured as a mirror of another (potentially in a different account/domain):
- `prefix` is set to `$KV.<mirror_bucket>.`
- `put_prefix` may be set to the source bucket's API prefix for cross-domain writes
- `use_jetstream_prefix` is adjusted based on whether the mirror is in the same domain
## KV → Stream Config Mapping
When creating a KV bucket, the `Config` is converted to a JetStream `stream::Config`:
| KV Config | Stream Config |
|-----------|---------------|
| `bucket` | `name = "KV_<bucket>"` |
| `subjects` | `["$KV.<bucket>.>"]` |
| `max_messages_per_subject` | `history` (max 64) |
| `max_age` | `max_age` |
| `max_bytes` | `max_bytes` |
| `storage` | `storage` |
| `num_replicas` | `num_replicas` |
| `republish` | `republish` |
| `mirror` | `mirror` |
| `discard` | `DiscardPolicy::New` |
| `allow_direct` | `true` |
| `allow_rollup_hdrs` | `true |
| `max_msg_size` | `max_value_size` |

245
docs/research/references/nats.rs/nats-async/07-object-store.md Normal file
View File

@@ -0,0 +1,245 @@
# async-nats: Object Store
## Overview
The Object Store provides large-object storage built on JetStream. Objects are chunked and stored as messages in a JetStream stream, with metadata stored separately. The stream is named `OBJ_<bucket_name>`.
The object-store feature requires `object-store` (which implies `jetstream` + `crypto`).
## ObjectStore Handle
```rust
#[derive(Clone)]
pub struct ObjectStore {
pub(crate) name: String,
pub(crate) stream: Stream,
}
```
## Object Store Config
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct Config {
pub bucket: String,
pub description: Option<String>,
pub max_age: Duration,
pub max_bytes: i64,
pub storage: StorageType,
pub num_replicas: usize,
pub compression: bool,
pub placement: Option<Placement>,
}
```
## Creating/Accessing Object Stores
```rust
// Create
let bucket = jetstream.create_object_store(object_store::Config {
bucket: "my-bucket".to_string(),
..Default::default()
}).await?;
// Get existing
let bucket = jetstream.get_object_store("my-bucket").await?;
// Delete
jetstream.delete_object_store("my-bucket").await?;
```
## Object Store Operations
### Put
```rust
let info: ObjectInfo = bucket.put("file.txt", &mut async_read).await?;
```
The put operation:
1. Reads data from any `AsyncRead + Unpin` source in chunks (default 128KB)
2. Each chunk is published to `$O.<bucket>.C.<nuid>` (chunk subject)
3. SHA-256 digest is computed incrementally
4. After all chunks, metadata is published to `$O.<bucket>.M.<encoded_name>` with a rollup header
5. If the object previously existed, old chunks are purged
### Get
```rust
let mut object: Object = bucket.get("file.txt").await?;
```
Returns an `Object` that implements `tokio::io::AsyncRead`:
```rust
let mut bytes = Vec::new();
object.read_to_end(&mut bytes).await?;
```
On read, the Object:
1. Creates an ordered push consumer on `$O.<bucket>.C.<nuid>`
2. Streams chunk messages, feeding bytes to the reader
3. Verifies SHA-256 digest after the last chunk
4. If digest doesn't match, returns `io::ErrorKind::InvalidData`
### Delete
```rust
bucket.delete("file.txt").await?;
```
Marks the object as deleted in metadata (sets `deleted = true`, `chunks = 0`, `size = 0`) with a rollup, then purges all chunk messages.
### Info
```rust
let info: ObjectInfo = bucket.info("file.txt").await?;
```
Fetches the last metadata message for the object (from `$O.<bucket>.M.<encoded_name>`).
### Watch
```rust
let mut watcher = bucket.watch().await?;
let mut watcher = bucket.watch_with_history().await?;
```
Returns a `Stream<Item = Result<ObjectInfo, WatcherError>>`. Uses an ordered push consumer on `$O.<bucket>.M.>`.
### List
```rust
let mut list = bucket.list().await?;
```
Returns a `Stream<Item = Result<ObjectInfo, ListerError>>`. Lists all non-deleted objects. Uses `DeliverPolicy::All` to replay all metadata.
### Seal
```rust
bucket.seal().await?;
```
Sets the underlying stream's `sealed = true`, preventing any further modifications.
### Links
```rust
// Link to another object (same or different bucket)
let info = bucket.add_link("link_name", &object).await?;
// Link to another bucket
let info = bucket.add_bucket_link("link_name", "other_bucket").await?;
```
Links are followed automatically when `get()` is called (one level deep). Cannot link to a deleted object or create a link to a link.
### Update Metadata
```rust
bucket.update_metadata("object", object_store::UpdateMetadata {
name: "new_name".to_string(),
description: Some("updated description".to_string()),
..Default::default()
}).await?;
```
If the name changes, old metadata is purged and new metadata is published.
## Object Types
### ObjectInfo
```rust
#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectInfo {
pub name: String,
pub description: Option<String>,
pub metadata: HashMap<String, String>,
pub headers: Option<HeaderMap>,
pub options: Option<ObjectOptions>,
pub bucket: String,
pub nuid: String,
pub size: usize,
pub chunks: usize,
pub modified: Option<DateTime>,
pub digest: Option<String>, // Format: "SHA-256=<base64url-digest>"
pub deleted: bool,
}
```
### ObjectMetadata
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectMetadata {
pub name: String,
pub description: Option<String>,
pub chunk_size: Option<usize>,
pub metadata: HashMap<String, String>,
pub headers: Option<HeaderMap>,
}
```
### ObjectLink
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectLink {
pub name: Option<String>, // None = bucket link, Some = object link
pub bucket: String,
}
```
### Object
```rust
pub struct Object {
pub info: ObjectInfo,
remaining_bytes: VecDeque<u8>,
has_pending_messages: bool,
digest: Option<Sha256>,
subscription: Option<crate::jetstream::consumer::push::Ordered>,
subscription_future: Option<BoxFuture<'static, Result<Ordered, StreamError>>>,
stream: Stream,
}
```
Implements `tokio::io::AsyncRead`. Lazy-creates the consumer on first read.
## Subject Naming Convention
| Purpose | Subject Pattern |
|---------|----------------|
| Chunks | `$O.<bucket>.C.<nuid>` |
| Metadata | `$O.<bucket>.M.<base64url-encoded-name>` |
Object names are base64url-encoded in metadata subjects to allow arbitrary characters (the raw name might contain characters invalid in NATS subjects).
## Validation
```rust
// Bucket: alphanumeric, dash, underscore only
BUCKET_NAME_RE: \A[a-zA-Z0-9_-]+\z
// Object name: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots
OBJECT_NAME_RE: \A[-/_=\.a-zA-Z0-9]+\z
```
## Data Integrity
The object store uses SHA-256 hashing (from the `crypto` module) to verify data integrity:
1. On `put()`: SHA-256 is computed incrementally as chunks are read. The digest is stored in `ObjectInfo.digest` as `"SHA-256=<base64url>"`.
2. On `get()` (via `AsyncRead`): SHA-256 is verified after the last chunk is read. If the computed digest doesn't match the stored digest, `io::ErrorKind::InvalidData` is returned.
```rust
// crypto module
pub(crate) struct Sha256 { ... }
impl Sha256 {
pub fn new() -> Self;
pub fn update(&mut self, data: &[u8]);
pub fn finish(self) -> [u8; 32];
}
```

272
docs/research/references/nats.rs/nats-async/08-service-api.md Normal file
View File

@@ -0,0 +1,272 @@
# async-nats: Service API
## Overview
The Service API provides a microservice request/reply pattern with built-in service discovery, health checking, and statistics. It follows the [NATS Micro v1 specification](https://github.com/nats-io/nats-architecture-design/blob/main/adr/ADR-33.md).
The `service` feature is required.
## Service
```rust
#[derive(Debug)]
pub struct Service {
endpoints_state: Arc<Mutex<Endpoints>>,
info: Info,
client: Client,
handle: JoinHandle<Result<(), Error>>,
shutdown_tx: Sender<()>,
subjects: Arc<Mutex<Vec<String>>>,
queue_group: String,
}
```
## Creating a Service
Via the `ServiceExt` trait on `Client`:
```rust
use async_nats::service::ServiceExt;
// Builder pattern
let mut service = client
.service_builder()
.description("product service")
.stats_handler(|endpoint, stats| serde_json::json!({ "endpoint": endpoint }))
.metadata(HashMap::from([("version".into(), "v2".into())]))
.queue_group("products-group")
.start("products", "1.0.0")
.await?;
// Direct config
let mut service = client
.add_service(service::Config {
name: "products".to_string(),
version: "1.0.0".to_string(),
description: Some("product service".to_string()),
stats_handler: None,
metadata: None,
queue_group: None,
})
.await?;
```
Service name must match `^[A-Za-z0-9\-_]+$`. Version must be valid SemVer.
## Service Verbs
Every service automatically subscribes to three verb subjects for discovery and monitoring:
| Verb | Subject Pattern | Purpose |
|------|----------------|---------|
| PING | `$SRV.PING`, `$SRV.PING.<name>`, `$SRV.PING.<name>.<id>` | Lightweight health check |
| INFO | `$SRV.INFO.<name>`, `$SRV.INFO.<name>.<id>` | Service metadata |
| STATS | `$SRV.STATS.<name>`, `$SRV.STATS.<name>.<id>` | Service + endpoint statistics |
A background task handles these verb requests and responds with JSON payloads.
## Service Config
```rust
#[derive(Serialize, Deserialize, Debug)]
pub struct Config {
pub name: String,
pub description: Option<String>,
pub version: String,
pub stats_handler: Option<StatsHandler>,
pub metadata: Option<HashMap<String, String>>,
pub queue_group: Option<String>,
}
```
## Adding Endpoints
```rust
// Simple endpoint
let mut endpoint = service.endpoint("get-products").await?;
// Endpoint with custom name and metadata
let endpoint = service
.endpoint_builder()
.name("api")
.metadata(HashMap::from([("auth".into(), "required".into())]))
.queue_group("custom-group")
.add("products")
.await?;
// Grouped endpoints
let v1 = service.group("v1");
let products = v1.endpoint("products").await?;
let orders = v1.endpoint("orders").await?;
// Nested groups
let v1_api = service.group("api").group("v1");
```
## Endpoint
```rust
pub struct Endpoint {
requests: Subscriber,
stats: Arc<Mutex<Endpoints>>,
client: Client,
endpoint: String,
shutdown: Option<ShutdownRx>,
shutdown_future: Option<ShutdownReceiverFuture>,
}
```
Implements `futures_util::Stream<Item = Request>`.
```rust
while let Some(request) = endpoint.next().await {
request.respond(Ok("response data".into())).await?;
}
```
## Service Request
```rust
#[derive(Debug)]
pub struct Request {
issued: Instant,
client: Client,
pub message: Message,
endpoint: String,
stats: Arc<Mutex<Endpoints>>,
}
```
### Responding
```rust
// Success
request.respond(Ok("result".into())).await?;
// Success with headers
request.respond_with_headers(Ok("result".into()), headers).await?;
// Error
request.respond(Err(service::error::Error {
code: 500,
status: "internal error".to_string(),
})).await?;
```
Error responses always include `Nats-Service-Error` and `Nats-Service-Error-Code` headers. If user-supplied headers contain these headers, they are overridden by the error values.
### Stats Tracking
Each response updates endpoint statistics:
- `requests` — total requests
- `processing_time` — cumulative processing time
- `average_processing_time` — average per request
- `errors` — error count
- `last_error` — last error details
## Service Info Types
### PingResponse
```rust
pub struct PingResponse {
pub kind: String, // "io.nats.micro.v1.ping_response"
pub name: String,
pub id: String,
pub version: String,
pub metadata: HashMap<String, String>,
}
```
### Info
```rust
pub struct Info {
pub kind: String, // "io.nats.micro.v1.info_response"
pub name: String,
pub id: String,
pub description: String,
pub version: String,
pub metadata: HashMap<String, String>,
pub endpoints: Vec<endpoint::Info>,
}
```
### Stats
```rust
pub struct Stats {
pub kind: String, // "io.nats.micro.v1.stats_response"
pub name: String,
pub id: String,
pub version: String,
pub started: DateTime,
pub endpoints: Vec<endpoint::Stats>,
}
```
### Endpoint Stats
```rust
pub struct endpoint::Stats {
pub name: String,
pub subject: String,
pub queue_group: String,
pub data: Option<serde_json::Value>, // Custom data from stats_handler
pub errors: u64,
pub processing_time: Duration,
pub average_processing_time: Duration,
pub requests: u64,
pub last_error: Option<error::Error>,
}
```
## Service Groups
Groups provide subject prefixing for endpoint organization:
```rust
let service = client.service_builder().start("api", "1.0.0").await?;
// Endpoints subscribe to "products" and "orders"
let products = service.endpoint("products").await?;
let orders = service.endpoint("orders").await?;
// Grouped: subscribe to "v1.products" and "v1.orders"
let v1 = service.group("v1");
let products = v1.endpoint("products").await?;
let orders = v1.endpoint("orders").await?;
// Nested: subscribe to "api.v1.products"
let api_v1 = service.group("api").group("v1");
let products = api_v1.endpoint("products").await?;
```
Each group can have its own queue group:
```rust
let v1 = service.group_with_queue_group("v1", "v1-workers");
```
## Stopping a Service
```rust
service.stop().await?;
```
Sends a shutdown signal and aborts the verb-handling task. Other service instances with the same name continue running.
## Resetting Stats
```rust
service.reset().await?;
```
Resets all endpoint statistics (errors, processing time, requests, average processing time) to zero.
## Querying Service State
```rust
let stats: HashMap<String, endpoint::Stats> = service.stats().await?;
let info: Info = service.info().await?;
```

312
docs/research/references/nats.rs/nats-async/09-quick-reference.md Normal file
View File

@@ -0,0 +1,312 @@
# async-nats: Quick Reference
## Connection
```rust
// Basic connect
let client = async_nats::connect("demo.nats.io").await?;
// With options
let client = async_nats::ConnectOptions::new()
.require_tls(true)
.name("my-service")
.ping_interval(Duration::from_secs(10))
.request_timeout(Some(Duration::from_secs(5)))
.connect("demo.nats.io")
.await?;
// Multiple servers
let client = async_nats::connect(vec![
"nats://server1:4222".parse()?,
"nats://server2:4222".parse()?,
]).await?;
// Background connect
let client = async_nats::ConnectOptions::new()
.retry_on_initial_connect()
.connect("demo.nats.io")
.await?;
```
## Core NATS: Publish
```rust
// Simple publish
client.publish("subject", "payload".into()).await?;
// With reply-to
client.publish_with_reply("subject", "reply-to", "payload".into()).await?;
// With headers
let mut headers = HeaderMap::new();
headers.insert("X-Custom", "value");
client.publish_with_headers("subject", headers, "payload".into()).await?;
// Full control
client.publish_with_reply_and_headers("subject", "reply-to", headers, "payload".into()).await?;
// Flush (ensure all published messages are sent)
client.flush().await?;
```
## Core NATS: Subscribe
```rust
use futures_util::StreamExt;
// Basic subscribe
let mut subscriber = client.subscribe("subject").await?;
// Queue group
let mut subscriber = client.queue_subscribe("subject", "group".into()).await?;
// Receive messages (Subscriber implements Stream)
while let Some(message) = subscriber.next().await {
println!("subject: {}, payload: {:?}", message.subject, message.payload);
}
// Unsubscribe
subscriber.unsubscribe().await?;
// Unsubscribe after N messages
subscriber.unsubscribe_after(10).await?;
// Drain (wait for in-flight, then unsubscribe)
subscriber.drain().await?;
```
## Core NATS: Request/Reply
```rust
// Simple request (uses default timeout)
let response = client.request("subject", "data".into()).await?;
// With custom timeout and headers
let request = async_nats::Request::new()
.payload("data".into())
.timeout(Some(Duration::from_secs(5)))
.headers(headers);
let response = client.send_request("subject", request).await?;
// Custom inbox (bypasses multiplexer)
let request = async_nats::Request::new()
.payload("data".into())
.inbox("custom-inbox".into());
let response = client.send_request("subject", request).await?;
```
## Message Structure
```rust
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
## JetStream
```rust
let jetstream = async_nats::jetstream::new(client);
// Publish (returns ack future)
let ack = jetstream.publish("events", "data".into()).await?;
let publish_ack = ack.await?;
// Stream management
let stream = jetstream.create_stream(stream::Config {
name: "events".to_string(),
subjects: vec!["events.>".to_string()],
max_messages: 10_000,
..Default::default()
}).await?;
let stream = jetstream.get_stream("events").await?;
let stream = jetstream.get_or_create_stream(config).await?;
jetstream.delete_stream("events").await?;
jetstream.update_stream(config).await?;
// Consumer management
let consumer: PullConsumer = stream.create_consumer(pull::Config {
durable_name: Some("my-consumer".to_string()),
..Default::default()
}).await?;
// Pull consumer: fetch messages
let mut messages = consumer.messages().await?;
while let Some(message) = messages.next().await {
let message = message?;
message.ack().await?;
}
// Push consumer (ordered)
let consumer = stream.create_consumer(push::OrderedConfig {
deliver_subject: client.new_inbox(),
filter_subject: "events.>".to_string(),
..Default::default()
}).await?;
let mut messages = consumer.messages().await?;
```
## Key-Value Store
```rust
let kv = jetstream.create_key_value(kv::Config {
bucket: "my-bucket".to_string(),
history: 10,
..Default::default()
}).await?;
// CRUD
let revision = kv.put("key", "value".into()).await?;
let revision = kv.create("key", "value".into()).await?;
let value: Option<Bytes> = kv.get("key").await?;
let entry: Option<Entry> = kv.entry("key").await?;
let revision = kv.update("key", "new-value".into(), revision).await?;
kv.delete("key").await?;
kv.purge("key").await?;
// Watch
let mut watch = kv.watch("key").await?;
let mut watch_all = kv.watch_all().await?;
// History & Keys
let mut history = kv.history("key").await?;
let mut keys = kv.keys().await?;
```
## Object Store
```rust
let bucket = jetstream.create_object_store(object_store::Config {
bucket: "files".to_string(),
..Default::default()
}).await?;
// Put (from any AsyncRead)
let info = bucket.put("file.txt", &mut file).await?;
// Get (returns AsyncRead)
let mut object = bucket.get("file.txt").await?;
let mut bytes = Vec::new();
object.read_to_end(&mut bytes).await?;
// Info, delete, list, watch
let info = bucket.info("file.txt").await?;
bucket.delete("file.txt").await?;
let mut list = bucket.list().await?;
let mut watch = bucket.watch().await?;
```
## Service API
```rust
use async_nats::service::ServiceExt;
use futures_util::StreamExt;
let mut service = client
.service_builder()
.description("product service")
.start("products", "1.0.0")
.await?;
let mut endpoint = service.endpoint("get").await?;
while let Some(request) = endpoint.next().await {
request.respond(Ok("result".into())).await?;
}
```
## Client State & Events
```rust
// Check connection state
match client.connection_state() {
State::Connected => {},
State::Disconnected => {},
State::Pending => {},
}
// Get server info
let info: ServerInfo = client.server_info();
println!("max_payload: {}", info.max_payload);
println!("jetstream: {}", info.jetstream);
// Get statistics
let stats = client.statistics();
println!("in_messages: {}", stats.in_messages.load(Ordering::Relaxed));
// Force reconnect
client.force_reconnect().await?;
// Server pool management
client.set_server_pool(["nats://s1:4222".parse()?, "nats://s2:4222".parse()?].as_slice()).await?;
let pool = client.server_pool().await?;
// Drain
client.drain().await?;
```
## Error Handling Patterns
```rust
// Connect errors
match async_nats::connect("server").await {
Err(e) => match e.kind() {
ConnectErrorKind::TimedOut => {},
ConnectErrorKind::Authentication => {},
ConnectErrorKind::AuthorizationViolation => {},
_ => {},
},
Ok(client) => {},
}
// Publish errors
match client.publish("subject", "data".into()).await {
Err(e) => match e.kind() {
PublishErrorKind::MaxPayloadExceeded => {},
PublishErrorKind::InvalidSubject => {},
PublishErrorKind::Send => {},
_ => {},
},
_ => {},
}
// Request errors
match client.request("subject", "data".into()).await {
Err(e) => match e.kind() {
RequestErrorKind::TimedOut => {},
RequestErrorKind::NoResponders => {},
RequestErrorKind::InvalidSubject => {},
RequestErrorKind::MaxPayloadExceeded => {},
_ => {},
},
Ok(message) => {},
}
```
## Feature Flag Quick Reference
| Feature | Enables | Default |
|---------|---------|---------|
| `jetstream` | JetStream streams, consumers, publish | ✅ |
| `kv` | Key-Value store (implies `jetstream`) | ✅ |
| `object-store` | Object store (implies `jetstream` + `crypto`) | ✅ |
| `service` | Service API | ✅ |
| `nkeys` | NKey/JWT authentication | ✅ |
| `nuid` | NUID-based ID generation | ✅ |
| `crypto` | SHA-256 (for object store) | ✅ |
| `websockets` | WebSocket transport | ✅ |
| `ring` | `ring` TLS crypto backend | ✅ |
| `aws-lc-rs` | `aws-lc-rs` TLS crypto backend | ❌ |
| `fips` | FIPS mode via `aws-lc-rs` | ❌ |
| `chrono` | `chrono` datetime instead of `time` | ❌ |
| `server_2_10` | Server 2.10+ features | ✅ |
| `server_2_11` | Server 2.11+ features | ✅ |
| `server_2_12` | Server 2.12+ features | ✅ |
| `server_2_14` | Server 2.14+ features | ✅ |

23
docs/research/references/nats.rs/nats-async/README.md Normal file
View File

@@ -0,0 +1,23 @@
# async-nats Reference Documentation
**Crate**: `async-nats` v0.49.1
**Source**: https://github.com/nats-io/nats.rs (`async-nats/` directory)
**License**: Apache-2.0
## Contents
| # | File | Topic |
|---|------|-------|
| 01 | [Overview & Architecture](01-overview-and-architecture.md) | Crate overview, feature flags, source structure, core connection model, dependency graph |
| 02 | [Key Types & Traits](02-key-types-and-traits.md) | `Client`, `Subscriber`, `Message`, `Request`, `ServerInfo`, `ConnectInfo`, `Statistics`, subject/header types, event/state types, error types, trait definitions |
| 03 | [Protocol & Wire Format](03-protocol-and-wire-format.md) | NATS wire protocol (PUB/HPUB/SUB/UNSUB/PING/PONG, MSG/HMSG/INFO/ERR), parser/serializer internals, vectored I/O, WebSocket transport, connection lifecycle, reconnection |
| 04 | [Connection Management](04-connection-management.md) | `ConnectOptions` builder, authentication methods, TLS configuration, reconnection callbacks, event callbacks, `ConnectionHandler` internals, multiplexer, server pool management |
| 05 | [JetStream](05-jetstream.md) | `Context` and `ContextBuilder`, streams, consumers (pull/push/ordered), JetStream messages and acks, publish with ack futures, pagination |
| 06 | [Key-Value Store](06-key-value-store.md) | KV `Store` handle, bucket CRUD, put/get/create/update/delete/purge, watch/history/keys, entry operations, mirrored buckets, KV-to-stream mapping |
| 07 | [Object Store](07-object-store.md) | `ObjectStore` handle, put/get/delete/watch/list/seal, links, `Object` (AsyncRead), chunking, SHA-256 integrity, subject naming |
| 08 | [Service API](08-service-api.md) | `Service` and `ServiceBuilder`, endpoints, groups, verb subscriptions (PING/INFO/STATS), request/respond with stats tracking |
| 09 | [Quick Reference](09-quick-reference.md) | Code examples for all major operations, feature flag reference |
## How This Documentation Was Produced
All information was derived by reading the source code of the `async-nats` crate at version 0.49.1 from the `nats.rs` repository. No external documentation was consulted — this is a ground-up reference based purely on the source.

200
docs/research/references/nats.rs/nats-server/01-overview-and-architecture.md Normal file
View File

@@ -0,0 +1,200 @@
# nats.rs: Overview and Architecture
**Version**: async-nats 0.49.1, nats-server 0.1.0
**Repository**: https://github.com/nats-io/nats.rs
**License**: Apache-2.0
**Rust Edition**: 2021
**MSRV**: 1.88.0
**Protocol**: NATS Client Protocol (INFO/CONNECT/PUB/SUB/UNSUB/PING/PONG)
## What It Is
The `nats.rs` repository contains the **official Rust client for NATS.io**, a high-performance messaging system. The active crate is **`async-nats`** — a fully async, Tokio-based NATS client. The deprecated `nats` crate (synchronous) receives security fixes only.
The `nats-server` crate is **not** an implementation of the NATS server. It is a **test harness** that spawns the Go-based `nats-server` binary for integration tests. The actual NATS server is a separate Go project at `github.com/nats-io/nats-server`.
Core design decisions:
- **Fully async** — all I/O is Tokio-based with async/await throughout
- **Cloneable Client handle** — `Client` is cheap to clone (Arc internals), all protocol work happens in a single `ConnectionHandler` task
- **Channel-based internal communication** — `Client` sends `Command` variants via `mpsc` channel to `ConnectionHandler`
- **Multiplexed request-reply** — one internal subscription handles all request-response patterns via inbox token routing
- **Automatic reconnection** — exponential backoff with configurable server pool rotation
- **Feature-gated subsystems** — JetStream, KV, Object Store, Service API, NKeys, WebSockets, and crypto backends are all optional
## Workspace Structure
```
nats.rs/
├── async-nats/ # Primary crate — async NATS client
│ ├── src/
│ │ ├── lib.rs # Entry point: connect(), ServerOp, ClientOp, Command, ConnectionHandler, Subscriber
│ │ ├── client.rs # Client handle: publish, subscribe, request, flush, drain
│ │ ├── connection.rs # Low-level I/O: protocol parsing, read/write buffers
│ │ ├── connector.rs # Connection establishment, reconnection, server pool
│ │ ├── options.rs # ConnectOptions builder
│ │ ├── auth.rs # Auth struct (credentials container)
│ │ ├── auth_utils.rs # Credential file parsing (.creds files)
│ │ ├── error.rs # Generic Error<Kind> type
│ │ ├── header.rs # HeaderMap — NATS message headers
│ │ ├── subject.rs # Subject type, ToSubject trait
│ │ ├── status.rs # StatusCode (100-999 NATS protocol codes)
│ │ ├── message.rs # Message and OutboundMessage types
│ │ ├── tls.rs # TLS configuration helpers
│ │ ├── crypto.rs # Crypto feature support
│ │ ├── id_generator.rs # NUID/rand-based unique ID generation
│ │ ├── datetime.rs # DateTime helpers for JetStream/Service
│ │ ├── jetstream/ # JetStream API (feature-gated)
│ │ │ ├── mod.rs # Module root, jetstream::new(), with_domain()
│ │ │ ├── context.rs # JetStream Context — streams, publishing, consumers
│ │ │ ├── stream.rs # Stream management, Config, Info, Consumer creation
│ │ │ ├── consumer/ # Pull, Push, Ordered consumers
│ │ │ ├── message.rs # JetStream Message with ack methods
│ │ │ ├── publish.rs # PublishAck
│ │ │ ├── response.rs # Response wrapper
│ │ │ ├── errors.rs # JetStream error codes
│ │ │ ├── account.rs # Account info
│ │ │ ├── kv/ # Key-Value store (feature: "kv")
│ │ │ └── object_store/ # Object store (feature: "object-store")
│ │ └── service/ # Service API (feature-gated)
│ │ ├── mod.rs # Service, ServiceBuilder
│ │ ├── endpoint.rs # Endpoint handling
│ │ └── error.rs # Service errors
│ ├── tests/ # Integration tests (require nats-server binary)
│ ├── examples/ # Runnable examples
│ └── benches/ # Criterion benchmarks
├── nats-server/ # Test harness — spawns Go nats-server for tests
│ ├── src/lib.rs # Server struct, run_server(), run_cluster()
│ └── configs/ # Server config files for tests
│ └── jetstream.conf
└── nats/ # DEPRECATED sync client — do not modify
```
## Architecture Diagram
```
┌──────────────────────────────────────────────────────────┐
│ Application Layer │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ JetStream│ │ KV │ │ Object │ │ Service │ │
│ │ Context │ │ Store │ │ Store │ │ API │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ └──────────────┴─────────────┴─────────────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ Client │ Cloneable handle │
│ │ (mpsc::Sender) │
│ └──────┬──────┘ │
│ │ Command channel │
└──────────────────────────┼────────────────────────────────┘
│
┌──────────────────────────┼────────────────────────────────┐
│ ConnectionHandler │
│ (single Tokio task) │
│ │ │
│ ┌───────────┐ ┌───────┴───────┐ ┌──────────────────┐ │
│ │Subscriptions│ │ Multiplexer │ │ Flush Observers │ │
│ │ HashMap │ │ (request-reply)│ │ │ │
│ └──────┬──────┘ └───────┬───────┘ └──────────────────┘ │
│ └────────────────┼ │
│ ┌──────┴──────┐ │
│ │ Connector │ Server pool, reconnect │
│ └──────┬──────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ Connection │ Protocol I/O │
│ │ (read/write)│ ServerOp / ClientOp │
│ └──────┬──────┘ │
└──────────────────────────┼────────────────────────────────┘
│
┌──────┴──────┐
│ NATS Server │ (Go binary, TCP/TLS/WS)
└─────────────┘
```
## Key Concepts
### Subject
NATS uses subject strings for message addressing. A `Subject` is a validated, immutable, UTF-8 string backed by `Bytes`. Subjects use dot-delimited tokens (e.g., `events.data.sensor1`). Wildcards `*` (single token) and `>` (multi-token suffix) are supported for subscriptions.
### ClientOp / ServerOp
The NATS client-server protocol is text-based with binary payloads. The client sends `ClientOp` variants (CONNECT, PUB/HPUB, SUB, UNSUB, PING, PONG) and receives `ServerOp` variants (INFO, MSG/HMSG, +OK, -ERR, PING, PONG).
### Command
Internal command type sent from `Client` to `ConnectionHandler` via `mpsc` channel. Includes Publish, Request, Subscribe, Unsubscribe, Flush, Drain, Reconnect, SetServerPool, ServerPool.
### Multiplexer
A single internal subscription (SID 0) that routes all request-reply responses. When a `Request` is made, a unique inbox token is registered in the multiplexer's sender map, and the response is dispatched to the corresponding `oneshot::Sender`.
### ConnectionHandler
A single Tokio task that drives all protocol I/O. It processes server operations from `Connection`, handles client commands from the `mpsc` channel, manages subscriptions, maintains ping/pong health, and orchestrates reconnection.
## nats-server Test Harness
The `nats-server` crate provides utilities for launching real NATS server instances in tests:
- `run_server(cfg)` — starts a single server with optional config
- `run_cluster(cfg)` — starts a 3-node cluster
- `Server` struct — holds the child process, cleans up on drop
- `Server::restart()` — kills and restarts the server process
- `Server::client_url()` — reads the INFO from the server to get the client URL
- `set_lame_duck_mode(server)` — sends LDM signal to the server process
The test harness spawns the Go `nats-server` binary via `std::process::Command`, using dynamic ports for parallel test execution. It auto-discovers the client URL by connecting to the server's TCP port and parsing the `INFO` JSON. On `Drop`, it kills the child process and cleans up JetStream storage directories.
## Feature Flags
```toml
# Default: everything enabled
default = ["server_2_10", "server_2_11", "server_2_12", "server_2_14",
"service", "ring", "jetstream", "nkeys", "crypto",
"object-store", "kv", "websockets", "nuid"]
# Subsystems
jetstream # JetStream API
kv # Key-Value store (requires jetstream)
object-store # Object store (requires jetstream + crypto)
service # Service API
# Crypto backends (pick one)
ring # Default crypto backend
aws-lc-rs # Alternative backend
fips # FIPS mode (requires aws-lc-rs)
# Auth
nkeys # NKey authentication
# Other
nuid # NUID-based ID generation (falls back to rand)
crypto # Encryption support
websockets # WebSocket transport
experimental # Experimental features
# Server version markers (enable version-specific API fields)
server_2_10
server_2_11
server_2_12
server_2_14
```
## Dependencies (Key)
| Dependency | Purpose |
|-----------|---------|
| `tokio` | Async runtime (macros, rt, net, sync, time, io-util) |
| `bytes` | Zero-copy byte buffers for payloads |
| `tokio-rustls` | TLS via rustls |
| `rustls-native-certs` | Load native TLS root certificates |
| `serde` / `serde_json` | JSON serialization for protocol messages and JetStream API |
| `memchr` | Fast CRLF search for protocol parsing |
| `futures-util` | Stream trait, Sink trait, StreamExt |
| `tracing` | Structured logging |
| `thiserror` | Error type derivation |
| `url` | URL parsing for server addresses |
| `portable-atomic` | Portable atomic operations |
## References
- [NATS Protocol Specification](https://docs.nats.io/reference/reference-protocols/nats-protocol)
- [NATS JetStream Documentation](https://docs.nats.io/nats-concepts/jetstream)
- [async-nats on docs.rs](https://docs.rs/async-nats)

281
docs/research/references/nats.rs/nats-server/02-protocol-and-wire-format.md Normal file
View File

@@ -0,0 +1,281 @@
# NATS Client Protocol and Wire Format
**Protocol**: NATS Client Protocol v1 (with dynamic reconfiguration)
**Transport**: TCP (port 4222), TLS, WebSocket (ws/wss)
## Protocol Overview
The NATS client-server protocol is a simple, text-based protocol with binary payload support. All operations are terminated with `\r\n`. Messages carry their payload length, allowing efficient binary data transfer.
### Connection Lifecycle
```
Client Server
│ │
│◄──────────── INFO {json} ────────────────────│ Server sends INFO first
│ │
│────────────── CONNECT {json} ────────────────►│ Client sends CONNECT
│────────────── PING ──────────────────────────►│ Client sends PING
│◄──────────── PONG ────────────────────────── │ Server confirms connection
│ │
│──── SUB/UNSUB/PUB/HPUB ──────────────────────►│ Normal operation
│◄─── MSG/HMSG/+OK/-ERR/PING ─────────────────│
│ │
```
## Server Operations (ServerOp)
These are operations received from the server. The `Connection` module parses these from the read buffer.
### INFO
Sent by the server upon connection and asynchronously when cluster topology changes.
```
INFO {json}\r\n
```
JSON fields (see `ServerInfo` struct):
| Field | Type | Description |
|-------|------|-------------|
| `server_id` | String | Unique server identifier |
| `server_name` | String | Generated server name |
| `host` | String | Cluster host |
| `port` | u16 | Cluster port |
| `version` | String | Server version |
| `auth_required` | bool | Authentication required |
| `tls_required` | bool | TLS required |
| `max_payload` | usize | Maximum payload size |
| `proto` | i8 | Protocol version (0 or 1) |
| `client_id` | u64 | Server-assigned client ID |
| `go` | String | Go build version |
| `nonce` | String | Nonce for nkey auth |
| `connect_urls` | Vec<String> | Cluster server URLs |
| `client_ip` | String | Client IP as seen by server |
| `headers` | bool | Server supports headers |
| `ldm` | bool | Lame duck mode |
| `cluster` | Option<String> | Cluster name |
| `domain` | Option<String> | NATS domain |
| `jetstream` | bool | JetStream enabled |
### MSG
Delivers a message to a subscription (no headers):
```
MSG <subject> <sid> [reply-to] <#bytes>\r\n
<payload>\r\n
```
### HMSG
Delivers a message with headers:
```
HMSG <subject> <sid> [reply-to] <#header-bytes> <#total-bytes>\r\n
<NATS/1.0 [status] [description]>\r\n
<header-name>: <header-value>\r\n
\r\n
<payload>\r\n
```
Header format follows the NATS/1.0 header spec:
- First line: `NATS/1.0` optionally followed by status code and description
- Subsequent lines: `name: value` headers
- Empty line separates headers from payload
- Header values may span multiple lines (continuation lines start with whitespace)
### PING / PONG
```
PING\r\n → Client responds with PONG
PONG\r\n → Acknowledges client's PING
```
### +OK / -ERR
```
+OK\r\n → Success acknowledgment (verbose mode)
-ERR <description>\r\n → Error from server
```
Common server errors:
- `authorization violation` → parsed as `ServerError::AuthorizationViolation`
- Other strings → `ServerError::Other(String)`
## Client Operations (ClientOp)
These are operations sent from the client to the server. The `Connection` module serializes these to the write buffer.
### CONNECT
Sent as the first client operation after receiving INFO. Contains authentication and capability information.
```
CONNECT {json}\r\n
```
JSON fields (see `ConnectInfo` struct):
| Field | Type | Description |
|-------|------|-------------|
| `verbose` | bool | Enable +OK acknowledgments (always false in this client) |
| `pedantic` | bool | Strict format checking (always false) |
| `jwt` | Option<String> | User JWT for auth |
| `nkey` | Option<String> | Public nkey for auth |
| `sig` | Option<String> | Signed nonce (Base64URL encoded) |
| `name` | Option<String> | Client name |
| `echo` | bool | Whether server should echo messages back |
| `lang` | String | Implementation language ("rust") |
| `version` | String | Client version |
| `protocol` | u8 | Protocol version (1 = dynamic) |
| `tls_required` | bool | TLS required |
| `user` | Option<String> | Username |
| `pass` | Option<String> | Password |
| `auth_token` | Option<String> | Auth token |
| `headers` | bool | Client supports headers (always true) |
| `no_responders` | bool | Client supports no-responders (always true) |
### PUB / HPUB
Publish a message:
```
PUB <subject> [reply-to] <#payload-bytes>\r\n
<payload>\r\n
```
Publish with headers:
```
HPUB <subject> [reply-to] <#header-bytes> <#total-bytes>\r\n
<NATS/1.0\r\n
<header-name>: <header-value>\r\n
\r\n
<payload>\r\n
```
### SUB
Subscribe to a subject:
```
SUB <subject> [queue-group] <sid>\r\n
```
### UNSUB
Unsubscribe from a subscription:
```
UNSUB <sid> [max]\r\n
```
The optional `max` parameter tells the server to auto-unsubscribe after receiving the specified number of messages.
### PING / PONG
```
PING\r\n → Health check / keepalive
PONG\r\n → Response to server PING
```
## Protocol Version
The `Protocol` enum has two variants:
| Value | Name | Description |
|-------|------|-------------|
| 0 | Original | Basic protocol |
| 1 | Dynamic | Supports async INFO for cluster topology changes, lame duck mode |
This client always sends `protocol: 1` (Dynamic), enabling:
- Asynchronous INFO messages with updated server lists
- Lame duck mode notifications
- Dynamic reconfiguration of cluster topology
## Wire Format Details
### Message Length Calculation
For plain `MSG`:
```
length = subject.len() + reply.map_or(0, |r| r.len()) + payload.len()
```
For `HMSG`:
```
length = subject.len() + reply.map_or(0, |r| r.len()) + header_len + payload.len()
```
Where `header_len` = serialized header bytes and `total_len` = `header_len + payload.len()`.
### Write Buffer Architecture
The `Connection` uses a two-tier write buffer:
1. **`flattened_writes`** (`BytesMut`) — for small writes (< 4096 bytes). Protocol headers, short commands, and small messages are flattened into this buffer for efficient sequential writing.
2. **`write_buf`** (`VecDeque<Bytes>`) — for large writes (>= 4096 bytes). Large payloads are appended as separate `Bytes` chunks. Supports vectored writes (`write_vectored`) when the underlying stream supports it, writing up to 64 chunks at once.
The soft limit for the total write buffer is 65,535 bytes (`SOFT_WRITE_BUF_LIMIT`). When exceeded, the `ConnectionHandler` stops processing new commands until the buffer drains.
### Read Buffer Architecture
The `Connection` uses a single `BytesMut` read buffer with configurable initial capacity (default 65,535 bytes). Protocol parsing uses `memchr::memmem::find` to locate CRLF delimiters efficiently. If a partial message is in the buffer, the parser returns `None` and waits for more data.
### Header Serialization
Headers are serialized in NATS/1.0 format:
```
NATS/1.0\r\n
Header-Name: Header-Value\r\n
Multi-Line-Header: value part 1\r\n
continuation of value\r\n
Another-Header: another value\r\n
\r\n
```
The `HeaderMap::to_bytes()` method handles this serialization, using `httparse`-compatible line folding for multi-line values.
### Status Codes in Headers
NATS status codes are embedded in the `HMSG` header version line:
```
NATS/1.0 404 No Messages\r\n
NATS/1.0 408 Request Timeout\r\n
NATS/1.0 503 No Responders\r\n
```
Common codes used by the client:
| Code | Constant | Meaning |
|------|----------|---------|
| 100 | `IDLE_HEARTBEAT` | JetStream idle heartbeat |
| 200 | `OK` | Success |
| 404 | `NOT_FOUND` | Message/stream not found |
| 408 | `TIMEOUT` | Request timeout |
| 409 | `REQUEST_TERMINATED` | Request terminated |
| 503 | `NO_RESPONDERS` | No responders available |
## Protocol Parsing Implementation
The `Connection::try_read_op()` method handles all protocol parsing:
1. Search for `\r\n` delimiter using `memchr::memmem::find`
2. Match the operation prefix:
- `+OK` → `ServerOp::Ok`
- `PING` → `ServerOp::Ping`
- `PONG` → `ServerOp::Pong`
- `-ERR` → parse error description → `ServerOp::Error`
- `INFO ` → parse JSON → `ServerOp::Info`
- `MSG ` → parse subject/sid/reply/length, read payload → `ServerOp::Message`
- `HMSG ` → parse headers + payload → `ServerOp::Message`
3. Unknown prefix → return `io::Error` with `InvalidInput`
For `MSG` and `HMSG`, if the complete payload isn't yet in the read buffer (checked via `len + payload_len + 4 > remaining`), the method returns `Ok(None)` and the buffer accumulates more data before retrying.
Non-UTF8 subjects in server messages are handled gracefully — the parser returns an `io::Error` rather than panicking, which is critical because the Go server does not enforce UTF-8 in subjects (regression fix for issue #1572).

443
docs/research/references/nats.rs/nats-server/03-key-types-and-traits.md Normal file
View File

@@ -0,0 +1,443 @@
# Key Types and Traits
This document covers the core data types in the `async-nats` crate that form the public API and internal plumbing.
## Public Types
### Client
**Location**: `client.rs`
`Client` is the primary user-facing type. It is a lightweight, cloneable handle to a NATS connection.
```rust
#[derive(Clone, Debug)]
pub struct Client {
info: tokio::sync::watch::Receiver<Option<ServerInfo>>,
state: tokio::sync::watch::Receiver<State>,
sender: mpsc::Sender<Command>,
poll_sender: PollSender<Command>,
next_subscription_id: Arc<AtomicU64>,
subscription_capacity: usize,
inbox_prefix: Arc<str>,
request_timeout: Option<Duration>,
max_payload: Arc<AtomicUsize>,
connection_stats: Arc<Statistics>,
skip_subject_validation: bool,
}
```
Key methods:
- `publish(subject, payload)` — fire-and-forget publish
- `publish_with_headers(subject, headers, payload)` — publish with NATS headers
- `publish_with_reply(subject, reply, payload)` — publish with reply subject
- `request(subject, payload)` — request-response (returns `Message`)
- `send_request(subject, request)` — request with `Request` builder
- `subscribe(subject)` — subscribe to a subject, returns `Subscriber`
- `queue_subscribe(subject, queue_group)` — subscribe as part of a queue group
- `flush()` — ensure all pending messages are written to the wire
- `drain()` — gracefully drain all subscriptions and close
- `force_reconnect()` — trigger immediate reconnection
- `new_inbox()` — generate a unique inbox subject for request-reply
- `server_info()` — get last received `ServerInfo`
- `max_payload()` — get server's maximum payload size
- `connection_state()` — get current connection `State`
- `statistics()` — get `Arc<Statistics>` for connection metrics
- `is_server_compatible(major, minor, patch)` — check server version compatibility
- `set_server_pool(addrs)` / `server_pool()` — manage server pool
`Client` also implements `Sink<OutboundMessage>` for backpressure-aware publishing.
### Subscriber
**Location**: `lib.rs`
A `Subscriber` receives messages from a single subscription. It implements `futures::Stream`.
```rust
#[derive(Debug)]
pub struct Subscriber {
sid: u64,
receiver: mpsc::Receiver<Message>,
sender: mpsc::Sender<Command>,
}
```
Key methods:
- `unsubscribe()` — unsubscribe and close the stream
- `unsubscribe_after(max)` — auto-unsubscribe after N messages
- `drain()` — gracefully drain remaining messages then close
On `Drop`, `Subscriber` automatically sends an `Unsubscribe` command and closes the receiver channel.
### Message
**Location**: `message.rs`
Represents an inbound NATS message.
```rust
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
### OutboundMessage
**Location**: `message.rs`
Represents a message to be published. No status/description fields (those are inbound-only).
```rust
#[derive(Clone, Debug)]
pub struct OutboundMessage {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
}
```
### Subject
**Location**: `subject.rs`
An immutable, validated UTF-8 string backed by `Bytes`. Used throughout the crate instead of raw `String` for subjects.
```rust
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Subject {
bytes: Bytes,
}
```
Implements `Deref<Target = str>`, `From<&str>`, `From<String>`, `TryFrom<Bytes>`, `Serialize`, `Deserialize`.
Validation methods:
- `is_valid()` — checks NATS subject rules (no leading/trailing dots, no consecutive dots, no whitespace)
- `validated(s)` — construct with validation, returns `Result<Subject, SubjectError>`
- `from_static_validated(s)` — const-time validation for static strings (compile-time panic on invalid)
### ToSubject Trait
**Location**: `subject.rs`
```rust
pub trait ToSubject {
fn to_subject(&self) -> Subject;
}
```
Implemented for `Subject`, `&'static str`, `String`. All methods accepting subjects are generic over `impl ToSubject`.
### HeaderMap
**Location**: `header.rs`
NATS message headers, modeled after the `http::header` crate.
```rust
#[derive(Clone, PartialEq, Eq, Debug, Default)]
pub struct HeaderMap {
inner: HashMap<HeaderName, Vec<HeaderValue>>,
}
```
Supports multiple values per header name (like HTTP). Key methods:
- `insert(name, value)` — replace all values for a name
- `append(name, value)` — add a value to a name
- `get(name)` — get the first value
- `get_all(name)` — get all values as an iterator
- `len()` / `is_empty()` — number of header entries
- `to_bytes()` — serialize to NATS/1.0 wire format
- `wire_len()` — size in wire format (for payload size checks)
### StatusCode
**Location**: `status.rs`
NATS status codes (100-999), structurally similar to HTTP status codes.
```rust
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
pub struct StatusCode(NonZeroU16);
```
Constants:
| Constant | Code | Meaning |
|----------|------|---------|
| `IDLE_HEARTBEAT` | 100 | JetStream idle heartbeat |
| `OK` | 200 | Success |
| `NOT_FOUND` | 404 | Not found |
| `TIMEOUT` | 408 | Timeout |
| `REQUEST_TERMINATED` | 409 | Request terminated |
| `NO_RESPONDERS` | 503 | No responders |
### ServerInfo
**Location**: `lib.rs`
Deserialized from the server's `INFO` JSON message. Contains server capabilities, connection details, and cluster information.
### ConnectInfo
**Location**: `lib.rs`
Serialized into the client's `CONNECT` JSON message. Contains authentication credentials, client capabilities, and protocol preferences.
### ServerAddr
**Location**: `lib.rs`
A validated NATS server URL, supporting schemes `nats://`, `tls://`, `ws://`, `wss://`.
```rust
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ServerAddr(Url);
```
Methods:
- `from_url(url)` — validate and create
- `tls_required()` — true for `tls://` scheme
- `is_websocket()` — true for `ws://` or `wss://`
- `host()` / `port()` / `scheme()` — URL component accessors
- `socket_addrs()` — async DNS resolution
- `username()` / `password()` — embedded credentials
### Auth
**Location**: `auth.rs`
Container for authentication credentials.
```rust
#[derive(Clone, Default)]
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
### Request
**Location**: `client.rs`
Builder for customized request-response operations.
```rust
#[derive(Default)]
pub struct Request {
pub payload: Option<Bytes>,
pub headers: Option<HeaderMap>,
pub timeout: Option<Option<Duration>>,
pub inbox: Option<String>,
}
```
### Statistics
**Location**: `client.rs`
Atomic connection statistics shared between Client and ConnectionHandler.
```rust
#[derive(Default, Debug)]
pub struct Statistics {
pub in_bytes: AtomicU64,
pub out_bytes: AtomicU64,
pub in_messages: AtomicU64,
pub out_messages: AtomicU64,
pub connects: AtomicU64,
}
```
### Event
**Location**: `lib.rs`
Events emitted by the client for connection lifecycle monitoring.
```rust
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Event {
Connected,
Disconnected,
LameDuckMode,
Draining,
Closed,
SlowConsumer(u64),
ServerError(ServerError),
ClientError(ClientError),
}
```
## Internal Types
### Command
**Location**: `lib.rs`
Internal commands sent from `Client` to `ConnectionHandler` via `mpsc` channel.
```rust
pub(crate) enum Command {
Publish(OutboundMessage),
Request { subject, payload, respond, headers, sender: oneshot::Sender<Message> },
Subscribe { sid, subject, queue_group, sender: mpsc::Sender<Message> },
Unsubscribe { sid, max: Option<u64> },
Flush { observer: oneshot::Sender<()> },
Drain { sid: Option<u64> },
Reconnect,
SetServerPool { servers: Vec<ServerAddr>, result: oneshot::Sender<Result<(), String>> },
ServerPool { result: oneshot::Sender<Vec<connector::Server>> },
}
```
### ClientOp / ServerOp
**Location**: `lib.rs`
Protocol-level operation types used by `Connection` for wire format parsing and serialization.
### Subscription (Internal)
**Location**: `lib.rs`
```rust
struct Subscription {
subject: Subject,
sender: mpsc::Sender<Message>,
queue_group: Option<String>,
delivered: u64,
max: Option<u64>,
}
```
### Multiplexer (Internal)
**Location**: `lib.rs`
```rust
struct Multiplexer {
subject: Subject, // Wildcard subscription subject (e.g., "_INBOX.xxx.*")
prefix: Subject, // Prefix for routing (e.g., "_INBOX.xxx.")
senders: HashMap<String, oneshot::Sender<Message>>, // token → sender
}
```
### Connection State
**Location**: `connection.rs`
```rust
#[derive(Debug, Eq, PartialEq, Clone)]
pub enum State {
Pending,
Connected,
Disconnected,
}
```
### Protocol
**Location**: `lib.rs`
```rust
#[derive(Serialize_repr, Deserialize_repr, PartialEq, Eq, Debug, Clone, Copy)]
#[repr(u8)]
pub enum Protocol {
Original = 0,
Dynamic = 1,
}
```
## Error Type Pattern
The crate uses a generic `Error<Kind>` type throughout. Every subsystem defines its own `ErrorKind` enum and a type alias:
```rust
// Define the kind enum
#[derive(Clone, Debug, PartialEq)]
pub enum PublishErrorKind {
MaxPayloadExceeded,
InvalidSubject,
Send,
}
// Define the error type alias
pub type PublishError = Error<PublishErrorKind>;
// Construct errors
PublishError::new(PublishErrorKind::MaxPayloadExceeded)
PublishError::with_source(PublishErrorKind::Send, io_error)
// Match on errors
if err.kind() == PublishErrorKind::MaxPayloadExceeded { ... }
```
Error kinds in the crate:
| Error Type | Kind Enum | Context |
|-----------|-----------|---------|
| `ConnectError` | `ConnectErrorKind` | Initial connection failures |
| `PublishError` | `PublishErrorKind` | Publish validation failures |
| `RequestError` | `RequestErrorKind` | Request-response failures |
| `SubscribeError` | `SubscribeErrorKind` | Subscription failures |
| `FlushError` | `FlushErrorKind` | Flush failures |
| `ServerPoolError` | `ServerPoolErrorKind` | Server pool query failures |
| `SetServerPoolError` | `SetServerPoolErrorKind` | Server pool modification failures |
## Trait Implementations
### Client Trait Interfaces
The `Client` implements several traits defined in `client::traits`:
```rust
// Publisher trait — publish with optional reply subject
trait Publisher {
fn publish_with_reply<S, R>(&self, subject: S, reply: R, payload: Bytes) -> impl Future<Output = Result<(), PublishError>>;
fn publish_message(&self, msg: OutboundMessage) -> impl Future<Output = Result<(), PublishError>>;
}
// Subscriber trait — subscribe to a subject
trait Subscriber {
fn subscribe<S>(&self, subject: S) -> impl Future<Output = Result<crate::Subscriber, SubscribeError>>;
}
// Requester trait — send request-response
trait Requester {
fn send_request<S>(&self, subject: S, request: Request) -> impl Future<Output = Result<Message, RequestError>>;
}
// TimeoutProvider trait — access request timeout
trait TimeoutProvider {
fn timeout(&self) -> Option<Duration>;
}
```
### ToServerAddrs Trait
**Location**: `lib.rs`
Converts various address types into server address iterators. Implemented for `ServerAddr`, `str`, `String`, `&[T]`, `Vec<T>`, `&[ServerAddr]`, and references.
### Sink<OutboundMessage>
`Client` implements `futures::Sink<OutboundMessage>` for backpressure-aware publishing through the `PollSender` adapter.
### Stream for Subscriber
`Subscriber` implements `futures::Stream` with `Item = Message`, delegating to the internal `mpsc::Receiver`.

338
docs/research/references/nats.rs/nats-server/04-connection-handler-and-data-flow.md Normal file
View File

@@ -0,0 +1,338 @@
# Connection Handler and Data Flow
This document covers the internal `ConnectionHandler` that drives all protocol I/O, and the data flow through the system.
## ConnectionHandler
**Location**: `lib.rs`
The `ConnectionHandler` is the heart of the client. It runs as a single Tokio task and manages all communication with the NATS server.
```rust
pub(crate) struct ConnectionHandler {
connection: Connection, // Low-level I/O
connector: Connector, // Server pool, reconnection
subscriptions: HashMap<u64, Subscription>, // Active subscriptions
multiplexer: Option<Multiplexer>, // Request-reply multiplexer
pending_pings: usize, // Unanswered PINGs
info_sender: tokio::sync::watch::Sender<Option<ServerInfo>>,
ping_interval: Interval, // Periodic PING timer
should_reconnect: bool, // Flag for forced reconnect
flush_observers: Vec<oneshot::Sender<()>>, // Pending flush callbacks
is_draining: bool, // Connection is draining
drain_pings: VecDeque<u64>, // SIDs being drained
}
```
## Data Flow: Publish
```
Application
│
│ client.publish("events.data", payload)
│
▼
Client
│ validates subject & payload size
│ sends Command::Publish(OutboundMessage) via mpsc channel
│
▼
ConnectionHandler::handle_command(Command::Publish)
│ increments out_messages, out_bytes statistics
│ calls connection.enqueue_write_op(&ClientOp::Publish { ... })
│
▼
Connection::enqueue_write_op
│ serializes to wire format:
│ "PUB events.data 11\r\n" or "HPUB events.data 23 34\r\n"
│ appends to flattened_writes or write_buf
│
▼
Connection::poll_write
│ uses vectored writes (64 chunks) if supported
│ or sequential writes otherwise
│
▼
Connection::poll_flush
│ flushes the TCP/TLS/WS stream
│ notifies flush_observers
│
▼
NATS Server (TCP/TLS/WebSocket)
```
## Data Flow: Subscribe
```
Application
│
│ client.subscribe("events.>")
│
▼
Client::subscribe
│ validates subject (always, regardless of skip_subject_validation)
│ allocates next sid via AtomicU64
│ creates mpsc channel for messages
│ sends Command::Subscribe { sid, subject, sender }
│ returns Subscriber { sid, receiver }
│
▼
ConnectionHandler::handle_command(Command::Subscribe)
│ creates Subscription { subject, sender, delivered: 0, max: None }
│ inserts into subscriptions HashMap
│ calls connection.enqueue_write_op(&ClientOp::Subscribe { sid, subject, queue_group })
│
▼
Connection::enqueue_write_op
│ serializes: "SUB events.> 42\r\n"
│
▼
Server sends MSG for matching subjects:
│
▼
ConnectionHandler::handle_server_op(ServerOp::Message { sid, subject, ... })
│ looks up sid in subscriptions HashMap
│ constructs Message { subject, reply, payload, headers, status, description }
│ tries subscription.sender.try_send(message)
│
├── Ok → increments subscription.delivered, checks max
├── Full → emits Event::SlowConsumer(sid)
└── Closed → removes subscription, sends ClientOp::Unsubscribe
│
▼
Subscriber::poll_next (Stream impl)
│ receives from mpsc::Receiver
│
▼
Application processes Message
```
## Data Flow: Request-Response
The request-response pattern uses the **multiplexer** — a single wildcard subscription that routes responses to their waiting requesters.
```
Application
│
│ client.request("service", payload)
│
▼
Client::send_request
│ validates subject & payload size
│ creates oneshot channel for response
│ generates unique inbox: "_INBOX.<nuid>.<token>"
│ sends Command::Request { subject, payload, respond, sender }
│
▼
ConnectionHandler::handle_command(Command::Request)
│ extracts token from respond subject (after last '.')
│ if no multiplexer exists:
│ creates Multiplexer with wildcard sub "_INBOX.<id>.*" (SID 0)
│ sends ClientOp::Subscribe { sid: 0, subject: "_INBOX.<id>.*" }
│ inserts token → oneshot::Sender in multiplexer.senders
│ sends ClientOp::Publish { subject, payload, respond: "<prefix><token>" }
│
▼
Server routes request to service:
│
▼
Service responds by publishing to the reply subject:
│
▼
ConnectionHandler::handle_server_op(ServerOp::Message { sid: 0, ... })
│ sid == MULTIPLEXER_SID (0), so enters multiplexer path
│ extracts token by stripping prefix from subject
│ looks up token in multiplexer.senders
│ sends Message via oneshot::Sender
│
▼
Client::send_request receives via oneshot::Receiver
│ applies timeout (default 10s)
│ checks for NO_RESPONDERS status (503)
│
▼
Application receives Message
```
### Custom Inbox Request
If the `Request` builder specifies a custom `inbox`, the flow is different:
- The client subscribes to the inbox directly (not via multiplexer)
- Publishes with the inbox as the reply subject
- Waits for the message on that subscription
- No multiplexer involvement
## Data Flow: Flush
```
Application
│
│ client.flush()
│
▼
Client::flush
│ creates oneshot channel
│ sends Command::Flush { observer }
│
▼
ConnectionHandler::handle_command(Command::Flush)
│ pushes observer into flush_observers Vec
│
▼
ProcessFut::poll (main loop)
│ after writing all pending data...
│ checks should_flush():
│ Yes (write buffers empty, not yet flushed) → poll_flush
│ May (write buffers not empty) → poll_flush
│ No (already flushed) → skip
│ on successful flush:
│ drains flush_observers, sending () to each
│
▼
Client::flush receives via oneshot::Receiver
```
## Data Flow: Drain
```
Application
│
│ client.drain() or subscriber.drain()
│
▼
Client::drain / Subscriber::drain
│ sends Command::Drain { sid: None } (whole client)
│ or Command::Drain { sid: Some(n) } (single subscription)
│
▼
ConnectionHandler::handle_command(Command::Drain)
│ if sid is Some:
│ pushes sid to drain_pings
│ sends ClientOp::Unsubscribe { sid, max: None }
│ if sid is None (whole client):
│ sets is_draining = true
│ emits Event::Draining
│ for each subscription: drain_pings.push(sid), Unsubscribe
│ sends ClientOp::Ping (to flush the UNSUB messages)
│
▼
ProcessFut::poll (main loop)
│ processes any remaining server messages
│ removes drained subscriptions from HashMap
│ if is_draining: returns ExitReason::Closed
│
▼
ConnectionHandler exits, emits Event::Closed
```
## Main Processing Loop
The `ConnectionHandler::process` method implements the core event loop via a custom `Future` (`ProcessFut`):
```rust
impl Future for ProcessFut<'_> {
type Output = ExitReason;
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
// 1. Check ping interval — send PING if due, disconnect if too many pending
while self.handler.ping_interval.poll_tick(cx).is_ready() {
if let Poll::Ready(exit) = self.ping() { return Poll::Ready(exit); }
}
// 2. Read all available server operations
loop {
match self.handler.connection.poll_read_op(cx) {
Poll::Pending => break,
Poll::Ready(Ok(Some(server_op))) => self.handler.handle_server_op(server_op),
Poll::Ready(Ok(None)) => return Poll::Ready(ExitReason::Disconnected(None)),
Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))),
}
}
// 3. Clean up drained subscriptions
while let Some(sid) = self.handler.drain_pings.pop_front() {
self.handler.subscriptions.remove(&sid);
}
// 4. If draining, exit
if self.handler.is_draining { return Poll::Ready(ExitReason::Closed); }
// 5. Process client commands (batch of up to 16)
// while write buffer not full
loop {
while !self.handler.connection.is_write_buf_full() {
match receiver.poll_recv_many(cx, recv_buf, 16) {
Poll::Pending => break,
Poll::Ready(1..) => { for cmd in recv_buf.drain(..) { handler.handle_command(cmd); } }
Poll::Ready(0) => return Poll::Ready(ExitReason::Closed),
}
}
// 6. Write pending data to stream
match self.handler.connection.poll_write(cx) {
Poll::Pending => break,
Poll::Ready(Ok(())) => continue, // write buffer empty, try more commands
Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))),
}
}
// 7. Flush stream and notify observers
match self.handler.connection.poll_flush(cx) { ... }
// 8. Check for forced reconnect
if mem::take(&mut self.handler.should_reconnect) {
return Poll::Ready(ExitReason::ReconnectRequested);
}
Poll::Pending
}
}
```
### Exit Reasons
The main loop exits for three reasons:
| Reason | Action |
|--------|--------|
| `Disconnected(Option<io::Error>)` | Attempt reconnection via `handle_disconnect()` |
| `ReconnectRequested` | Force reconnect (user-triggered) |
| `Closed` | Connection handler terminates, emit `Event::Closed` |
On disconnection, `handle_disconnect()` is called which:
1. Resets `pending_pings` to 0
2. Emits `Event::Disconnected`
3. Updates connection state to `Disconnected`
4. Calls `handle_reconnect()` which uses `Connector::connect()`
5. On successful reconnect, re-subscribes all active subscriptions
6. Re-subscribes the multiplexer wildcard if present
## Slow Consumer Handling
When a subscription's `mpsc::Sender` channel is full (the application isn't consuming messages fast enough):
1. `try_send` returns `TrySendError::Full`
2. The `ConnectionHandler` emits `Event::SlowConsumer(sid)`
3. The message is **dropped** (not queued)
4. The subscription remains active
When a subscription's receiver is dropped (application closed the stream):
1. `try_send` returns `TrySendError::Closed`
2. The subscription is removed from the HashMap
3. An `UNSUB` command is sent to the server
## Ping/Pong Health Check
The `ConnectionHandler` maintains a periodic PING interval (default 60 seconds):
1. `ping_interval` fires every N seconds
2. A `ClientOp::Ping` is enqueued
3. `pending_pings` counter increments
4. If `pending_pings > MAX_PENDING_PINGS (2)`, the connection is considered dead
5. When `ServerOp::Pong` is received, `pending_pings` decrements
6. Any server operation resets the ping interval timer
## Batched Command Processing
Commands from the `Client` are received in batches of up to 16 (`RECV_CHUNK_SIZE`) using `poll_recv_many`. This amortizes the cost of waking the task and enables pipelining multiple operations (e.g., publishing many messages) in a single poll cycle.

277
docs/research/references/nats.rs/nats-server/05-connection-and-reconnection.md Normal file
View File

@@ -0,0 +1,277 @@
# Connection and Reconnection
This document covers how connections are established, TLS handling, the server pool, and the reconnection mechanism.
## Connector
**Location**: `connector.rs`
The `Connector` manages the server pool and handles connection establishment and reconnection.
```rust
pub(crate) struct Connector {
servers: Vec<Server>, // Server pool with per-server metadata
options: ConnectorOptions, // Connection configuration
connect_stats: Arc<Statistics>, // Shared statistics
attempts: usize, // Global reconnection attempt counter
events_tx: mpsc::Sender<Event>, // Event channel
state_tx: watch::Sender<State>, // Connection state watcher
max_payload: Arc<AtomicUsize>, // Server's max payload
last_info: ServerInfo, // Last known server info
}
```
### Server Pool
Each server in the pool carries metadata:
```rust
#[derive(Debug, Clone)]
pub struct Server {
pub addr: ServerAddr,
pub failed_attempts: usize, // Consecutive failed attempts
pub did_connect: bool, // Ever successfully connected?
pub is_discovered: bool, // Discovered via INFO, not user-configured
pub last_error: Option<String>, // Last connection error
}
```
### ConnectorOptions
```rust
pub(crate) struct ConnectorOptions {
pub tls_required: bool,
pub certificates: Vec<PathBuf>,
pub client_cert: Option<PathBuf>,
pub client_key: Option<PathBuf>,
pub tls_client_config: Option<rustls::ClientConfig>,
pub tls_first: bool,
pub auth: Auth,
pub no_echo: bool,
pub connection_timeout: Duration, // Default: 5 seconds
pub name: Option<String>,
pub ignore_discovered_servers: bool,
pub retain_servers_order: bool,
pub read_buffer_capacity: u16, // Default: 65535
pub reconnect_delay_callback: Arc<dyn Fn(usize) -> Duration>,
pub auth_callback: Option<CallbackArg1<Vec<u8>, Result<Auth, AuthError>>>,
pub max_reconnects: Option<usize>,
pub local_address: Option<SocketAddr>,
pub reconnect_to_server_callback: Option<ReconnectToServerCallback>,
}
```
## Connection Establishment Flow
```
Connector::try_connect_to_server(addr)
│
├── 1. DNS resolution
│ server_addr.socket_addrs()
│
├── 2. For each resolved address:
│ │
│ ├── 2a. Connect with timeout
│ │ tokio::time::timeout(connection_timeout, try_connect_to(socket_addr, ...))
│ │
│ └── 2b. try_connect_to():
│ │
│ ├── Select transport:
│ │ ├── "ws" → WebSocket (tokio_websockets)
│ │ ├── "wss" → WebSocket over TLS
│ │ └── default → TCP (TcpStream)
│ │
│ ├── Optional: bind to local_address
│ ├── Set TCP_NODELAY
│ ├── Create Connection with read_buffer_capacity
│ │
│ ├── If tls_first: upgrade to TLS before INFO
│ │
│ ├── Read INFO from server
│ │
│ ├── If TLS required (by option, server, or URL scheme):
│ │ upgrade to TLS (rustls)
│ │
│ ├── Discover servers from INFO.connect_urls
│ │ (unless ignore_discovered_servers)
│ │
│ ├── Build ConnectInfo with auth:
│ │ ├── username/password (from Auth or URL)
│ │ ├── token (from Auth)
│ │ ├── nkey + signed nonce (feature: nkeys)
│ │ ├── JWT + signature callback (feature: nkeys)
│ │ └── auth_callback (custom async callback)
│ │
│ ├── Send CONNECT + PING
│ │
│ └── Wait for response:
│ ├── -ERR (authorization violation) → error
│ ├── PONG or +OK → success
│ └── EOF → error
│
└── 3. On success:
├── Reset attempt counter
├── Increment connects statistic
├── Emit Event::Connected
├── Update State::Connected
├── Store max_payload
├── Update per-server metadata (did_connect, failed_attempts)
└── Return (ServerInfo, Connection)
```
## TLS Handling
The client supports three TLS modes:
### 1. Standard TLS (INFO → TLS)
Default behavior. The client receives the `INFO` message in plaintext, then upgrades to TLS if:
- `tls_required` option is set
- Server's `INFO.tls_required` is true
- URL scheme is `tls://`
### 2. TLS First (TLS → INFO)
When `ConnectOptions::tls_first()` is enabled, the client establishes TLS before reading INFO. This requires the server to have `handshake_first` enabled. Useful for environments where plaintext INFO is not acceptable.
### 3. WebSocket TLS
For `wss://` URLs, TLS is handled by the WebSocket library (`tokio-websockets`) directly, not by the client's TLS layer.
### TLS Configuration
The client uses `rustls` via `tokio-rustls`. Configuration steps:
1. Load root certificates from system store (`rustls-native-certs`)
2. Optionally add custom root certificates from PEM files
3. Optionally configure client certificate and key for mTLS
4. Optionally pass a custom `rustls::ClientConfig`
Crypto backend is selectable via feature flags:
- `ring` (default)
- `aws-lc-rs`
- `fips` (requires aws-lc-rs)
## Reconnection
### Reconnection Trigger
Reconnection is triggered when:
1. I/O error during read or write (`ExitReason::Disconnected`)
2. Too many pending PINGs (no PONG received)
3. User calls `Client::force_reconnect()` (`ExitReason::ReconnectRequested`)
### Reconnection Flow
```
ConnectionHandler::handle_disconnect()
│
├── Reset pending_pings to 0
├── Emit Event::Disconnected
├── Update State::Disconnected
│
└── handle_reconnect()
│
└── Connector::connect()
│
└── Loop: try_connect()
│
├── If reconnect_to_server_callback is set:
│ │ Call callback with (server_pool, server_info)
│ │ If returns Some(ReconnectToServer):
│ │ Validate server is in pool
│ │ Use callback's delay or default backoff
│ │ Try connecting to selected server
│ └── If None or invalid: fall through to default
│
├── Default selection:
│ ├── Shuffle servers (unless retain_servers_order)
│ ├── Sort by failed_attempts (ascending)
│ └── Try each server in order
│
├── For each server:
│ ├── Increment attempts counter
│ ├── Check max_reconnects limit
│ ├── Apply reconnect delay (exponential backoff)
│ └── try_connect_to_server(addr)
│
├── On success:
│ ├── Reset attempts to 0
│ ├── Re-subscribe all active subscriptions
│ │ (filter out closed subscription channels)
│ ├── Re-subscribe multiplexer wildcard
│ └── Return (ServerInfo, Connection)
│
└── On failure:
├── Update per-server metadata (failed_attempts, last_error)
├── Auth errors → propagate immediately
└── Other errors → continue to next server
```
### Exponential Backoff
Default reconnect delay function:
```rust
fn reconnect_delay_callback_default(attempts: usize) -> Duration {
if attempts <= 1 {
Duration::from_millis(0)
} else {
let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX);
let max = Duration::from_secs(4);
cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), max)
}
}
```
| Attempt | Delay |
|---------|-------|
| 1 | 0ms |
| 2 | 0ms |
| 3 | 2ms |
| 4 | 4ms |
| 5 | 8ms |
| ... | ... |
| 13 | 4096ms |
| 14+ | 4000ms (capped) |
Custom delay functions can be provided via `ConnectOptions::reconnect_delay_callback()`.
### Server Pool Updates
The server pool is dynamic:
1. **Initial pool**: from `connect()` / `ConnectOptions::connect()` URL(s)
2. **Discovered servers**: added from `INFO.connect_urls` on each connection (unless `ignore_discovered_servers` is set)
3. **Runtime updates**: via `Client::set_server_pool()` — replaces the entire pool while preserving per-server state for servers that appear in both old and new pools
4. **Order**: servers are shuffled by default (random selection), unless `retain_servers_order` is set
### Max Reconnects
The `max_reconnects` option limits total reconnection attempts:
- `None` or `0` → unlimited (default)
- `Some(n)` → give up after `n` total attempts
- Counter is reset on successful connection and when `set_server_pool()` is called
## ConnectOptions Defaults
| Option | Default |
|--------|---------|
| `connection_timeout` | 5 seconds |
| `ping_interval` | 60 seconds |
| `sender_capacity` | 2048 |
| `subscription_capacity` | 65536 |
| `inbox_prefix` | `"_INBOX"` |
| `request_timeout` | 10 seconds |
| `retry_on_initial_connect` | false |
| `ignore_discovered_servers` | false |
| `retain_servers_order` | false |
| `read_buffer_capacity` | 65535 |
| `skip_subject_validation` | false |
| `no_echo` | false |
| `tls_required` | false |
| `tls_first` | false |
| `max_reconnects` | None (unlimited) |
## Background Connection
When `ConnectOptions::retry_on_initial_connect()` is enabled, the `connect()` function returns a `Client` immediately, before the connection is established. The connection is established in a background Tokio task. This means:
- `client.server_info()` returns `ServerInfo::default()` until connected
- `client.connection_state()` returns `State::Pending`
- Operations like `publish()` will queue in the command channel
- The `Client` becomes usable once the background task connects

472
docs/research/references/nats.rs/nats-server/06-jetstream-internals.md Normal file
View File

@@ -0,0 +1,472 @@
# JetStream Internals
This document covers the JetStream subsystem — how it provides stream-based messaging with persistence, consumer management, and higher-level APIs like KV and Object Store.
## JetStream Context
**Location**: `jetstream/context.rs`
The `Context` is the entry point to the JetStream API. It wraps a `Client` and provides stream management, publishing, and consumer operations.
```rust
#[derive(Debug, Clone)]
pub struct Context {
pub(crate) client: Client,
pub(crate) prefix: String, // API subject prefix (default: "$JS.API")
pub(crate) timeout: Duration, // Default request timeout
pub(crate) max_ack_semaphore: Arc<Semaphore>, // Limits in-flight ack waits
pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver<Message>, OwnedSemaphorePermit)>,
pub(crate) backpressure_on_inflight: bool,
}
```
### Context Creation
```rust
// Default context (prefix = "$JS.API")
let jetstream = async_nats::jetstream::new(client);
// With domain (prefix = "$JS.hub.API")
let jetstream = async_nats::jetstream::with_domain(client, "hub");
// With custom prefix
let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API");
// Builder pattern for more options
let jetstream = async_nats::jetstream::Context::builder(client)
.domain("hub")
.prefix("$JS.API")
.timeout(Duration::from_secs(30))
.max_ack_pending(256)
.backpressure_on_inflight(true)
.build();
```
### JetStream API Subject Convention
All JetStream API calls are request-response messages sent to subjects following the pattern:
```
$JS.API.<operation>.<stream-name>[.<consumer-name>]
```
Examples:
- `$JS.API.STREAM.CREATE.events` — create stream "events"
- `$JS.API.STREAM.INFO.events` — get stream info
- `$JS.API.CONSUMER.DURABLE.CREATE.events.myconsumer` — create durable consumer
- `$JS.API.CONSUMER.MSG.NEXT.events.myconsumer` — pull next message
With a domain, the prefix changes to `$JS.<domain>.API`.
## Stream Management
**Location**: `jetstream/stream.rs`
### Stream Config
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Config {
pub name: String,
pub subjects: Vec<String>, // Subject filter
pub retention: RetentionPolicy, // Limits, Interest, WorkQueue
pub max_consumers: i32,
pub max_messages: i64, // Per-stream message limit
pub max_messages_per_subject: i64,
pub max_bytes: i64, // Per-stream byte limit
pub max_age: Duration, // Message TTL
pub max_message_size: Option<i32>, // Max individual message size
pub storage: StorageType, // File or Memory
pub num_replicas: usize,
pub no_ack: bool, // Don't require ack
pub discard: DiscardPolicy, // Old or New
pub duplicate_window: Duration,
pub allow_rollup_hdrs: bool,
pub allow_direct: bool,
pub mirror: Option<External>,
pub sources: Vec<External>,
pub sealed: bool,
pub compression: Option<Compression>, // server_2_10+
pub first_sequence: Option<u64>, // server_2_11+
pub subject_transform: Option<SubjectTransform>, // server_2_12+
pub metadata: Option<HashMap<String, String>>, // server_2_10+
pub placement: Option<Placement>,
pub republish: Option<RePublish>,
}
```
### Stream Operations
Via `Context`:
| Method | API Subject | Description |
|--------|------------|-------------|
| `create_stream(config)` | `STREAM.CREATE.<name>` | Create a new stream |
| `get_stream(name)` | `STREAM.INFO.<name>` | Get existing stream |
| `get_or_create_stream(config)` | `STREAM.INFO` → `STREAM.CREATE` | Get or create |
| `delete_stream(name)` | `STREAM.DELETE.<name>` | Delete a stream |
| `update_stream(name, config)` | `STREAM.UPDATE.<name>` | Update stream config |
| `purge_stream(name)` | `STREAM.PURGE.<name>` | Purge all messages |
| `streams()` | `STREAM.LIST` | List all streams (paged iterator) |
| `stream_names()` | `STREAM.NAMES` | List stream names (paged iterator) |
| `account_info()` | `ACCOUNT.INFO` | Get account info |
Via `Stream`:
| Method | API Subject | Description |
|--------|------------|-------------|
| `info()` | `STREAM.INFO.<name>` | Refresh stream info |
| `purge()` | `STREAM.PURGE.<name>` | Purge messages |
| `delete()` | `STREAM.DELETE.<name>` | Delete this stream |
| `update(config)` | `STREAM.UPDATE.<name>` | Update config |
| `get_raw_message(seq)` | `STREAM.MSG.GET.<name>` | Get message by sequence (stored mode) |
| `get_last_message(subject)` | `STREAM.MSG.GET.<name>` | Get last message for subject (stored mode) |
| `direct_get_last(subject)` | `DIRECT.GET.<name>` | Direct get last (bypasses RAA) |
| `direct_get(seq)` | `DIRECT.GET.<name>` | Direct get by sequence |
| `delete_message(seq)` | `STREAM.MSG.DELETE.<name>` | Delete a specific message |
| `create_consumer(config)` | `CONSUMER.CREATE.<stream>` | Create consumer |
| `get_or_create_consumer(name, config)` | `CONSUMER.DURABLE.CREATE.<stream>.<name>` | Get or create durable |
| `get_consumer(name)` | `CONSUMER.INFO.<stream>.<name>` | Get existing consumer |
### Stream Info
```rust
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Info {
pub config: Config,
pub created: DateTime,
pub state: State, // Messages, bytes, first/last sequence, consumer count
pub cluster: Option<ClusterInfo>,
pub timestamp: DateTime,
pub leader: Option<String>,
pub subjects: Option<HashMap<String, u64>>, // Subject → message count
}
```
### Paged List Operations
Stream and consumer listing uses a paged iterator pattern:
```rust
// streams() returns an iterator that automatically pages
let mut streams = jetstream.streams();
while let Some(stream) = streams.next().await {
let stream = stream?;
// process stream
}
// stream_names() similarly pages
let mut names = jetstream.stream_names();
while let Some(name) = names.next().await {
println!("{}", name?);
}
```
The paged iterator sends an initial request with `offset: 0` and continues fetching pages until no more results are returned.
## Publishing
**Location**: `jetstream/context.rs`, `jetstream/publish.rs`
### Publish
```rust
// Basic publish (fire-and-forget)
jetstream.publish("events.data", "payload".into()).await?;
// Publish with custom message builder
jetstream.publish_message(
jetstream::message::PublishMessage::build()
.payload("data".into())
.message_id("unique-id") // Nats-Msg-Id header for dedup
.expected_last_message_id("prev") // Nats-Expected-Last-Msg-Id
.expected_last_sequence(42) // Nats-Expected-Last-Sequence
.expected_last_subject_sequence("events", 10) // Per-subject sequence
.header("Custom", "Value")
).await?;
```
### PublishAck
When a message is published to a JetStream stream, the server responds with a `PublishAck`:
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct PublishAck {
pub stream: String,
pub sequence: u64,
pub domain: Option<String>,
pub duplicate: bool,
}
```
### PublishAckFuture
Publishing returns a `PublishAckFuture` that resolves to `PublishAck`. The future uses a semaphore (`max_ack_semaphore`) to limit in-flight ack waits and prevent backpressure issues.
When `backpressure_on_inflight` is enabled, the publish operation blocks if there are too many pending acks, preventing the command channel from filling up with unbounded publish operations.
### Idempotent Publishing
Headers for exactly-once semantics:
| Header | Purpose |
|--------|---------|
| `Nats-Msg-Id` | Message ID for deduplication within the stream's duplicate window |
| `Nats-Expected-Last-Msg-Id` | Expected last message ID (conditional publish) |
| `Nats-Expected-Last-Sequence` | Expected last sequence number |
| `Nats-Expected-Last-Subject-Sequence` | Expected last sequence for a specific subject |
## Consumers
**Location**: `jetstream/consumer/`
### Consumer Types
| Type | Description |
|------|-------------|
| `PullConsumer` | Client pulls messages on demand |
| `PushConsumer` | Server pushes messages to a delivery subject |
| `OrderedConsumer` | Push consumer with automatic re-creation on failure |
### Consumer Config
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Config {
pub name: Option<String>,
pub durable_name: Option<String>,
pub description: Option<String>,
pub deliver_subject: Option<String>, // Push consumers only
pub ack_policy: AckPolicy,
pub ack_wait: Duration,
pub max_deliver: i64,
pub max_ack_pending: i32,
pub max_waiting: i32, // Pull consumers only
pub filter_subject: Option<String>,
pub replay_policy: ReplayPolicy,
pub sample_frequency: Option<i8>,
pub max_batch: i32, // Pull consumers
pub max_expires: Duration, // Pull consumers
pub inactive_threshold: Duration,
pub flow_control: bool, // Push consumers
pub heartbeat: Option<Duration>, // Push consumers
pub backoff: Vec<Duration>,
pub deliver_group: Option<String>,
pub num_replicas: usize,
pub mem_storage: bool,
pub metadata: Option<HashMap<String, String>>,
pub ack_markers: Option<Vec<String>>, // server_2_12+
}
```
### Pull Consumer
**Location**: `jetstream/consumer/pull.rs`
Pull consumers require explicit requests for messages:
```rust
// Batch request
let mut messages = consumer.messages().await?.take(100);
while let Some(message) = messages.next().await {
let message = message?;
message.ack().await?;
}
// Sequence-based batch
let mut batches = consumer.sequence(50)?.take(10);
while let Some(mut batch) = batches.try_next().await? {
while let Some(Ok(message)) = batch.next().await {
message.ack().await?;
}
}
// Single message fetch
let message = consumer.fetch().await?;
```
Pull requests are sent to: `$JS.API.CONSUMER.MSG.NEXT.<stream>.<consumer>`
The request payload is JSON:
```json
{"batch": 10, "expires": 5000, "no_wait": false}
```
### Push Consumer
**Location**: `jetstream/consumer/push.rs`
Push consumers receive messages automatically on a delivery subject. The client subscribes to the delivery subject and processes messages as they arrive.
Features:
- **Flow control** — server sends flow control messages, client responds to maintain delivery rate
- **Heartbeats** — idle heartbeats (status code 100) when no messages are available
- **Ordered consumers** — automatically recreated on delivery failures with correct sequence positioning
### Acknowledgment
**Location**: `jetstream/message.rs`
JetStream messages support multiple acknowledgment types:
```rust
pub enum AckKind {
Ack, // Ack (message processed)
Nack, // Nak (re-deliver)
Progress, // Progress (still working)
Next, // Next (ack + pull next)
Term, // Term (don't redeliver, remove from stream)
All, // Ack all messages up to this sequence
}
```
Methods on JetStream `Message`:
- `ack()` — simple acknowledgment
- `ack_with(kind)` — acknowledgment with specific type
- `double_ack()` — exactly-once ack (ACK + separate ack message)
- `nack()` — negative acknowledgment (request redelivery)
- `in_progress()` — progress indicator
- `term()` — terminate message (no redelivery)
## JetStream Message
**Location**: `jetstream/message.rs`
JetStream messages wrap core `Message` with metadata extracted from headers:
```rust
#[derive(Debug)]
pub struct Message {
pub message: crate::Message, // The underlying NATS message
pub context: Context, // JetStream context for acking
pub ack_pending: Arc<AtomicU64>, // Pending ack counter
}
impl Message {
pub fn info(&self) -> Result<Info, MessageInfoError> // Parse message info from headers
pub async fn ack(&self) -> Result<(), AckError>
pub async fn ack_with(&self, kind: AckKind) -> Result<(), AckError>
pub async fn double_ack(&self) -> Result<(), AckError>
pub async fn nack(&self) -> Result<(), AckError>
pub async fn in_progress(&self) -> Result<(), AckError>
pub async fn term(&self) -> Result<(), AckError>
}
```
Message info is extracted from the `HMSG` headers:
- `Nats-Stream` — stream name
- `Nats-Consumer` — consumer name
- `Nats-Delivered` — delivery count
- `Nats-Sequence` — stream sequence
- `Nats-Time-Stamp` — timestamp
- `Nats-Subject` — original subject
- `Nats-Pending-Messages` / `Nats-Pending-Bytes` — pending counts
## Key-Value Store
**Location**: `jetstream/kv/`
The KV store is a JetStream-based key-value API. Each bucket maps to a JetStream stream with specific configuration:
```rust
// Create a KV store
let kv = jetstream
.create_key_value(async_nats::jetstream::kv::Config {
bucket: "my_bucket".to_string(),
history: 5, // Max history per key (1-64)
ttl: Duration::from_secs(3600), // Key TTL
max_bytes: 1024 * 1024, // Max bucket size
storage: StorageType::File,
replicas: 1,
..Default::default()
})
.await?;
```
Under the hood:
- Each key is stored as a message with subject `$KV.<bucket>.<key>`
- Keys support wildcard patterns (`$KV.bucket.prefix.*`)
- History is managed via stream `max_messages_per_subject`
- TTL is managed via stream `max_age`
- `put(key, value)` publishes to the key subject
- `get(key)` reads the last message for the key subject
- `delete(key)` publishes an internal delete marker
- `purge(key)` uses stream purge API
- `watch()` subscribes to key changes and returns a `Watch` stream
- `keys()` / `history(key)` list keys and history
## Object Store
**Location**: `jetstream/object_store/`
The Object Store provides large object storage built on JetStream. Objects are chunked and stored across multiple messages in a stream.
```rust
// Create an object store
let store = jetstream
.create_object_store(async_nats::jetstream::object_store::Config {
bucket: "my_objects".to_string(),
..Default::default()
})
.await?;
// Put an object
let info = store.put("file.txt", stream).await?;
// Get an object
let mut object_stream = store.get("file.txt").await?;
```
Under the hood:
- Objects are chunked into ~128KB messages
- Metadata (object info) is stored as the first "chunk 0" message
- Each chunk is a message with subject `$OBJ.<bucket>.<object-nuid>.C<chunk-number>`
- Metadata includes: name, description, headers, size, chunks, digest (SHA-256)
- `get()` returns a stream of chunks
- Links allow referencing one object from another (like symlinks)
## JetStream Error Codes
**Location**: `jetstream/errors.rs`
Standard JetStream error codes returned by the server:
| Code | Constant | Description |
|------|----------|-------------|
| 10001 | `NOT_FOUND` | Resource not found |
| 10002 | `STREAM_NOT_FOUND` | Stream not found |
| 10003 | `CONSUMER_NOT_FOUND` | Consumer not found |
| 10004 | `REQUEST_NOT_FOUND` | Request not found |
| 10005 | `STREAM_WRONG_LAST_SEQ` | Wrong last sequence |
| 10006 | `STREAM_NAME_EXISTS` | Stream already exists |
| 10007 | `CONSUMER_NAME_EXISTS` | Consumer already exists |
| 10008 | `INSUFFICIENT_RESOURCES` | Insufficient resources |
| 10009 | `NO_MESSAGE_FOUND` | No message found |
| 10013 | `CONSUMER_EXISTS` | Consumer already exists (duplicate) |
| 10014 | `STREAM_NOT_CONFIGURED` | Stream not configured |
| 10015 | `CLUSTER_NOT_ACTIVE` | Cluster not active |
| 10016 | `CLUSTER_NOT_LEADER` | Not the cluster leader |
| 10017 | `CLUSTER_NOT_ENOUGH_PEERS` | Not enough peers |
| 10018 | `CLUSTER_INCOMPLETE` | Cluster incomplete |
| 10019 | `CONSUMER_DELETED` | Consumer was deleted |
| 10020 | `CONSUMER_BAD_ACK` | Bad acknowledgment |
| 10021 | `CONSUMER_BAD_SUBJECT` | Bad consumer subject |
| 10022 | `CONSUMER_DELETED_DRIFT` | Consumer deleted due to drift |
| ... | ... | Additional codes |
## Account
**Location**: `jetstream/account.rs`
The `Account` struct provides information about the JetStream account:
```rust
pub struct Account {
pub memory: i64,
pub storage: i64,
pub streams: i64,
pub consumers: i64,
pub limits: AccountLimits,
}
```

292
docs/research/references/nats.rs/nats-server/07-authentication-and-security.md Normal file
View File

@@ -0,0 +1,292 @@
# Authentication and Security
This document covers the authentication mechanisms, TLS configuration, and security-related features of the async-nats client.
## Authentication Methods
The NATS server supports multiple authentication methods. The client implements all of them.
### 1. Username/Password
The simplest authentication method.
```rust
// Via ConnectOptions
let client = ConnectOptions::with_user_and_password("user".into(), "pass".into())
.connect("nats://localhost")
.await?;
// Via URL
let client = connect("nats://user:pass@localhost:4222").await?;
```
These credentials are sent in the `CONNECT` message as `user` and `pass` fields.
### 2. Token Authentication
A single token used for authentication.
```rust
let client = ConnectOptions::with_token("my-token".into())
.connect("nats://localhost")
.await?;
```
Token is sent in the `CONNECT` message as `auth_token` field.
### 3. NKey Authentication
NKey-based authentication using Ed25519 key pairs. Requires the `nkeys` feature.
```rust
let seed = "SUANQDPB2RUOE4ETUA26CNX7FUKE5ZZKFCQIIW63OX225F2CO7UEXTM7ZY";
let client = ConnectOptions::with_nkey(seed.into())
.connect("nats://localhost")
.await?;
```
Flow:
1. Server sends `INFO` with a `nonce` field
2. Client creates a `KeyPair` from the seed
3. Client signs the nonce: `key_pair.sign(nonce.as_bytes())`
4. Client sends `CONNECT` with `nkey` (public key) and `sig` (Base64URL-encoded signature)
5. Server verifies the signature against the public key and nonce
### 4. JWT Authentication
User JWT with a signing callback. Requires the `nkeys` feature.
```rust
let key_pair = Arc::new(nkeys::KeyPair::from_seed(seed)?);
let jwt = load_jwt().await?;
let client = ConnectOptions::with_jwt(jwt, move |nonce| {
let key_pair = key_pair.clone();
async move { key_pair.sign(&nonce).map_err(AuthError::new) }
})
.connect("nats://localhost")
.await?;
```
Flow:
1. Server sends `INFO` with a `nonce` field
2. Client sends `CONNECT` with `jwt` (user JWT) and `sig` (Base64URL-encoded nonce signature)
3. The signing callback is async, allowing integration with external signing services (e.g., HSM)
### 5. Credentials File
Combines JWT and NKey from a `.creds` file. Requires the `nkeys` feature.
```rust
// From file
let client = ConnectOptions::with_credentials_file("path/to/my.creds")
.await?
.connect("nats://localhost")
.await?;
// From string
let client = ConnectOptions::with_credentials(creds_string)
.connect("nats://localhost")
.await?;
```
Credentials file format:
```
-----BEGIN NATS USER JWT-----
eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5...
------END NATS USER JWT------
************************* IMPORTANT *************************
NKEY Seed printed below can be used sign and prove identity.
-----BEGIN USER NKEY SEED-----
SUAIO3FHUX5PNV2LQIIP7TZ3N4L7TX3W53MQGEIVYFIGA635OZCKEYHFLM
------END USER NKEY SEED------
```
**Location**: `auth_utils.rs` handles parsing:
- `load_creds(path)` — async file read + parse
- `parse_jwt_and_key_from_creds(creds)` — extracts JWT and KeyPair from the string
### 6. Auth Callback
A custom async callback that receives the server nonce and returns an `Auth` struct. This is the most flexible mechanism.
```rust
let client = ConnectOptions::with_auth_callback(move |nonce| {
async move {
let mut auth = Auth::new();
auth.username = Some("user".to_string());
auth.password = Some("pass".to_string());
// Can also set jwt, nkey, signature, token
Ok(auth)
}
})
.connect("nats://localhost")
.await?;
```
The callback is invoked on each connection/reconnection, allowing dynamic credential refresh (e.g., refreshing JWTs from an auth server).
### 7. URL-Embedded Credentials
```rust
// Username and password in URL
let client = connect("nats://user:pass@localhost:4222").await?;
// Token in URL (username field)
let client = connect("nats://token@localhost:4222").await?;
```
## Auth Struct
**Location**: `auth.rs`
The `Auth` struct is a container for all authentication methods. Multiple fields can be set simultaneously:
```rust
#[derive(Clone, Default)]
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
Priority in `Connector::try_connect_to()`:
1. Auth callback overrides all other methods
2. NKey authentication (if `auth.nkey` is set)
3. JWT authentication (if `auth.jwt` is set)
4. Username/password/token from `Auth` struct
5. Username/password from URL
## TLS Configuration
### TLS Modes
| Mode | When | Description |
|------|------|-------------|
| None | Default | Plaintext connection |
| Standard | `tls_required` or server requires | TLS after INFO |
| TLS First | `tls_first` option | TLS before INFO |
| WebSocket | `wss://` URL | TLS handled by WebSocket library |
### TLS Setup
**Location**: `tls.rs`
The `config_tls()` function builds a `rustls::ClientConfig`:
1. Create `RootCertStore` and load native system certificates
2. Add custom root certificates from configured PEM files
3. Build `ClientConfig` with the chosen crypto provider:
- `ring` (default)
- `aws-lc-rs`
- `fips` (aws-lc-rs in FIPS mode)
4. If client certificate + key are configured, add them for mTLS
5. If a custom `rustls::ClientConfig` was provided, use it directly
### TLS First
```rust
let client = ConnectOptions::new()
.tls_first()
.connect("nats://localhost")
.await?;
```
This sets both `tls_first = true` and `tls_required = true`. The client performs TLS handshake before reading the `INFO` message. The server must have `handshake_first: true` in its configuration.
### Custom TLS Configuration
```rust
let tls_client = rustls::ClientConfig::builder()
.with_root_certificates(root_store)
.with_no_client_auth();
let client = ConnectOptions::new()
.require_tls(true)
.tls_client_config(tls_client)
.connect("nats://localhost")
.await?;
```
### mTLS (Mutual TLS)
```rust
let client = ConnectOptions::new()
.add_root_certificates("ca.pem".into())
.add_client_certificate("cert.pem".into(), "key.pem".into())
.connect("tls://localhost")
.await?;
```
## WebSocket Transport
Requires the `websockets` feature. Supports `ws://` and `wss://` schemes.
```rust
let client = connect("ws://localhost:8080").await?;
let client = connect("wss://localhost:443").await?;
```
Implementation uses `tokio-websockets` with a `WebSocketAdapter` that wraps the WebSocket stream to implement `AsyncRead + AsyncWrite`:
```rust
// WebSocketAdapter bridges WebSocket messages to byte streams
pub(crate) struct WebSocketAdapter<T> {
pub(crate) inner: WebSocketStream<T>,
pub(crate) read_buf: BytesMut, // Buffered incoming WebSocket messages
}
```
For `wss://`, TLS is configured within the WebSocket connector, not via the client's TLS layer.
## Security Considerations
### Nonce Signing
The server's `nonce` in the `INFO` message prevents replay attacks:
- Each connection gets a unique nonce
- The nonce must be signed with the client's private key
- The signature is verified server-side against the public key
### Authorization Violations
When the server sends `-ERR 'authorization violation'`:
- The client parses this as `ServerError::AuthorizationViolation`
- The `Connector` immediately propagates this error (does not retry)
- The error is converted to `ConnectErrorKind::AuthorizationViolation`
### Subject Validation
By default, the client validates subjects for protocol safety:
- **Publish subjects**: checked for emptiness and whitespace (can be disabled with `skip_subject_validation`)
- **Subscribe subjects**: always checked for emptiness, whitespace, leading/trailing dots, consecutive dots
- **Queue group names**: checked for emptiness and whitespace
The server enforces its own validation, but client-side checks prevent protocol-framing errors.
### Max Payload Size
The client checks payload size against the server's `max_payload` before publishing:
- For plain messages: `payload.len() > max_payload`
- For messages with headers: `headers.wire_len() + payload.len() > max_payload`
- Returns `PublishErrorKind::MaxPayloadExceeded` if exceeded
### No Echo
When `no_echo` is set, the `CONNECT` message includes `echo: false`. The server will not deliver messages published by this connection back to its own subscriptions. This prevents feedback loops.
### Lame Duck Mode
When the server enters lame duck mode (draining for shutdown):
1. Server sends `INFO` with `ldm: true`
2. Client emits `Event::LameDuckMode`
3. Application should gracefully close or reconnect to another server
The `nats-server` test harness provides `set_lame_duck_mode(server)` for testing this behavior.

347
docs/research/references/nats.rs/nats-server/08-test-harness.md Normal file
View File

@@ -0,0 +1,347 @@
# nats-server Test Harness
This document covers the `nats-server` crate — a test harness for spawning real NATS server instances in integration tests.
**Location**: `nats-server/src/lib.rs`
**Version**: 0.1.0
**License**: Apache-2.0
**Dependencies**: `lazy_static`, `regex`, `serde_json`, `nuid`, `rand`, `tokio-retry`
## What It Is
The `nats-server` crate is **not** a NATS server implementation. It is a thin test harness that:
- Spawns the Go-based `nats-server` binary as a child process
- Configures it for test use (dynamic ports, temp storage, log files)
- Discovers the client URL from the server's `INFO` protocol message
- Cleans up resources (JetStream storage, logs, PID files) on `Drop`
- Supports single servers and 3-node clusters
The actual NATS server must be installed separately (Go binary from `github.com/nats-io/nats-server`).
## Server Struct
```rust
pub struct Server {
inner: Inner,
}
struct Inner {
cfg: String, // Config file path
id: String, // Unique server ID (NUID)
port: Option<String>, // Explicit port (None = dynamic)
child: Child, // Child process handle
logfile: PathBuf, // Log file path in temp dir
pidfile: PathBuf, // PID file path in temp dir
}
```
## Public API
### run_server
```rust
pub fn run_server(cfg: &str) -> Server
```
Starts a single NATS server with optional config file.
- Uses dynamic port (`-1` flag) for parallel test execution
- Stores JetStream data in temp directory
- Writes logs to temp file: `nats-server-<id>.log`
- Writes PID to temp file: `nats-server-<id>.pid`
- If `cfg` is non-empty, passes `-c <cfg>` to the server
Example:
```rust
let server = nats_server::run_server("tests/configs/jetstream.conf");
let client = async_nats::connect(server.client_url()).await.unwrap();
```
### run_basic_server
```rust
pub fn run_basic_server() -> Server
```
Starts a server with no config (bare minimum). Equivalent to `run_server("")`.
### run_server_with_port
```rust
pub fn run_server_with_port(cfg: &str, port: Option<&str>) -> Server
```
Starts a server with an explicit port. If `None`, uses dynamic port.
### run_cluster
```rust
pub fn run_cluster<'a, C: IntoConfig<'a>>(cfg: C) -> Cluster
```
Starts a 3-node cluster with the given config.
- Allocates 3 random port ranges (base, base+100, base+200)
- Configures cluster routes between nodes
- Each node gets: `--cluster nats://127.0.0.1:<cluster_port>`, `--routes <other_routes>`, `--cluster_name cluster`, `-n nodeN`
- Waits 2 seconds for cluster formation and leader election
The `IntoConfig` trait allows passing either a single config string (applied to all 3 nodes) or an array of 3 configs (one per node):
```rust
// Same config for all nodes
let cluster = run_cluster("configs/jetstream.conf");
// Different configs per node
let cluster = run_cluster(["node1.conf", "node2.conf", "node3.conf"]);
```
### Cluster Struct
```rust
pub struct Cluster {
pub servers: Vec<Server>,
}
impl Cluster {
pub fn client_url(&self) -> String {
self.servers[0].client_url()
}
}
```
### Server Methods
```rust
impl Server {
pub fn restart(&mut self)
pub fn client_url(&self) -> String
pub fn client_port(&self) -> u16
pub fn client_url_with(&self, user: &str, pass: &str) -> String
pub fn client_url_with_token(&self, token: &str) -> String
pub fn client_pid(&self) -> usize
}
```
#### restart()
Kills the current server process, waits for it to exit, then restarts with the same config, port, and ID. Used for testing reconnection behavior.
#### client_url()
Connects to the server's TCP port, reads the `INFO` line, parses the JSON, and constructs a URL:
- `nats://localhost:<port>` for non-TLS
- `tls://localhost:<port>` for TLS-required servers
Polls the log file (up to 10 seconds) to discover the client address, since the port may be dynamically allocated.
#### client_pid()
Reads the PID file and returns the server process ID. Used for sending signals.
### set_lame_duck_mode
```rust
pub fn set_lame_duck_mode(s: &Server)
```
Sends the lame duck mode signal to the server:
```bash
nats-server --signal ldm=<pid>
```
### is_port_available
```rust
pub fn is_port_available(port: usize) -> bool
```
Tests if a TCP port is available by attempting to bind to it.
## Server Lifecycle
### Spawning
The `do_run` function constructs and spawns the server process:
```rust
fn do_run(cfg: &str, port: Option<&str>, id: Option<String>) -> Inner {
let id = id.unwrap_or_else(|| nuid::next().to_string());
let logfile = env::temp_dir().join(format!("nats-server-{id}.log"));
let pidfile = env::temp_dir().join(format!("nats-server-{id}.pid"));
let store_dir = env::temp_dir().join(format!("store-dir-{id}"));
let mut cmd = Command::new("nats-server");
cmd.arg("--store_dir").arg(store_dir.as_path())
.arg("-p");
match port {
Some(port) => cmd.arg(port),
None => cmd.arg("-1"), // Dynamic port
};
cmd.arg("-l").arg(logfile.as_os_str())
.arg("-P").arg(pidfile.as_os_str());
if !cfg.is_empty() {
cmd.arg("-c").arg(cfg);
}
let child = cmd.spawn().unwrap();
// ...
}
```
Key flags:
- `--store_dir` — JetStream storage directory in temp
- `-p -1` — Dynamic port allocation (or explicit port)
- `-l` — Log file path
- `-P` — PID file path
- `-c` — Config file path
### Cleanup (Drop)
```rust
impl Drop for Server {
fn drop(&mut self) {
self.inner.child.kill().unwrap();
self.inner.child.wait().unwrap();
if let Ok(log) = fs::read_to_string(self.inner.logfile.as_os_str()) {
// Clean up JetStream storage directory if found in log
if let Some(caps) = SD_RE.captures(&log) {
let sd = caps.get(1).map_or("", |m| m.as_str());
fs::remove_dir_all(sd).ok();
}
// Remove log file
fs::remove_file(self.inner.logfile.as_os_str()).ok();
}
}
}
```
The regex `SD_RE` matches the "Store Directory" line in the server log:
```
.+\sStore Directory:\s+"([^"]+)"
```
### Client URL Discovery
The `client_addr` method polls the log file to find the server's listen address:
```rust
fn client_addr(&self) -> String {
for _ in 0..100 { // 100 iterations × 500ms = 50s max
match fs::read_to_string(self.inner.logfile.as_os_str()) {
Ok(l) => {
if let Some(cre) = CLIENT_RE.captures(&l) {
return cre.get(1).unwrap().as_str()
.replace("0.0.0.0", "localhost");
} else {
thread::sleep(Duration::from_millis(500));
}
}
_ => thread::sleep(Duration::from_millis(500)),
}
}
panic!("no client addr info");
}
```
The regex `CLIENT_RE` matches:
```
.+\sclient connections on\s+(\S+)
```
After finding the address, `client_url()` connects to it and parses the `INFO` JSON to get the port and TLS requirements.
## Cluster Setup
The `run_cluster_node_with_port` function spawns a single cluster node:
```rust
fn run_cluster_node_with_port(
cfg: &str,
port: Option<&str>,
routes: Vec<usize>,
name: String,
cluster_name: String,
cluster: usize,
) -> Server
```
Additional flags for cluster nodes:
- `--routes nats://127.0.0.1:<port1>,nats://127.0.0.1:<port2>` — routes to other cluster members
- `--cluster nats://127.0.0.1:<cluster_port>` — cluster listen address
- `--cluster_name <name>` — cluster name for grouping
- `-n <name>` — server name
Port allocation for a cluster:
```
Base port: random in 3000..50000
Node 1: client_port=base, cluster_port=base+1
Node 2: client_port=base+100, cluster_port=base+101
Node 3: client_port=base+200, cluster_port=base+201
```
Each port is checked for availability with `is_port_available()`, including the +1 cluster port.
## JetStream Config
**Location**: `configs/jetstream.conf`
```conf
jetstream: {
strict: true,
max_mem_store: 8MiB,
max_file_store: 10GiB
}
```
This is the default test config for JetStream-enabled servers. It enables strict mode and sets memory/file storage limits suitable for testing.
## Test Usage Patterns
```rust
#[tokio::test]
async fn basic_test() {
let server = nats_server::run_server("configs/jetstream.conf");
let client = async_nats::connect(server.client_url()).await.unwrap();
// ... test logic ...
// Server cleaned up on drop
}
#[tokio::test]
async fn cluster_test() {
let cluster = nats_server::run_cluster("configs/jetstream.conf");
let client = async_nats::connect(cluster.client_url()).await.unwrap();
// ... test logic ...
}
#[tokio::test]
async fn reconnect_test() {
let mut server = nats_server::run_server("");
let client = async_nats::connect(server.client_url()).await.unwrap();
// Restart the server to test reconnection
server.restart();
// Client should reconnect automatically
client.publish("test", "data".into()).await.unwrap();
}
```
## Dependencies
| Dependency | Version | Purpose |
|-----------|---------|---------|
| `lazy_static` | 1.4.0 | Static regex initialization |
| `regex` | 1.7.1 | Log parsing (store directory, client address) |
| `url` | 2 | URL manipulation for client_url_with |
| `serde_json` | 1.0.104 | INFO JSON parsing |
| `nuid` | 0.5 | Unique server ID generation |
| `rand` | 0.10.1 | Random port selection |
| `tokio-retry` | 0.3.0 | Exponential backoff for cluster operations |
Note: `async-nats` is only a dev-dependency, used in the crate's own integration tests.

307
docs/research/references/nats.rs/nats-server/09-service-api-and-abstractions.md Normal file
View File

@@ -0,0 +1,307 @@
# Service API and Higher-Level Abstractions
This document covers the Service API and other higher-level abstractions built on top of the core NATS client.
## Service API
**Location**: `service/` (feature: `service`)
The Service API provides a framework for building NATS-based microservices with built-in monitoring, health checks, and statistics.
### Service
```rust
#[derive(Debug)]
pub struct Service {
client: Client,
info: Info,
endpoints: HashMap<String, Endpoint>,
started: DateTime,
stats_handler: Arc<dyn Fn(&str, &Stats) -> serde_json::Value + Send + Sync>,
stop_sender: mpsc::Sender<()>,
stop_receiver: Option<mpsc::Receiver<()>>,
}
```
### Creating a Service
```rust
use async_nats::service::ServiceExt;
let mut service = client
.service_builder()
.description("Product service")
.stats_handler(|endpoint, stats| {
serde_json::json!({
"endpoint": endpoint,
"requests": stats.num_requests,
"errors": stats.num_errors,
})
})
.start("products", "1.0.0")
.await?;
```
### ServiceBuilder
```rust
impl ServiceBuilder {
pub fn description(mut self, description: impl Into<String>) -> Self
pub fn stats_handler<F>(mut self, handler: F) -> Self
pub async fn start(self, name: impl Into<String>, version: impl Into<String>) -> Result<Service, ServiceError>
}
```
### Endpoints
A service exposes one or more endpoints, each handling requests on a specific subject:
```rust
// Add an endpoint
let mut endpoint = service
.endpoint("get_product")
.await?;
// Process requests
while let Some(request) = endpoint.next().await {
let request = request?;
// Handle the request
request.respond(serde_json::json!({ "id": 1, "name": "Widget" })).await?;
}
```
### Endpoint
**Location**: `service/endpoint.rs`
```rust
pub struct Endpoint {
subject: Subject,
queue_group: Option<String>,
info: EndpointInfo,
stats: Stats,
subscriber: Subscriber,
}
```
Implements `futures::Stream` yielding `ServiceRequest` objects.
### ServiceRequest
```rust
pub struct ServiceRequest {
pub subject: Subject,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub reply: Option<Subject>,
pub client: Client,
}
```
Methods:
- `respond(payload)` — send a response to the requester
- `respond_with_headers(payload, headers)` — send a response with headers
### Monitoring Subjects
The Service API automatically creates monitoring endpoints:
| Subject | Description |
|---------|-------------|
| `$SRV.PING` | Ping all services (returns service info) |
| `$SRV.PING.<name>` | Ping specific service by name |
| `$SRV.PING.<name>.<id>` | Ping specific service instance |
| `$SRV.INFO` | Get service info |
| `$SRV.STATS` | Get service statistics |
### Service Info
```rust
pub struct Info {
pub name: String,
pub id: String,
pub version: String,
pub description: String,
pub endpoints: Vec<EndpointInfo>,
}
```
### Stats
```rust
pub struct Stats {
pub num_requests: u64,
pub num_errors: u64,
pub last_error: Option<String>,
pub processing_time: Duration,
pub average_processing_time: Duration,
}
```
## ID Generation
**Location**: `id_generator.rs`
The client needs unique IDs for inbox subjects and other purposes.
### With `nuid` Feature (Default)
Uses the NUID library for high-performance, cryptographically strong, collision-resistant IDs:
```rust
pub(crate) fn next() -> String {
nuid::next().to_string()
}
```
NUID generates 22-character alphanumeric strings using a combination of a random prefix and a sequential counter.
### Without `nuid` Feature
Falls back to `rand`-based generation:
```rust
pub(crate) fn next() -> String {
rng()
.sample_iter(Alphanumeric)
.take(22)
.map(char::from)
.collect()
}
```
Both approaches produce 22-character alphanumeric strings, but NUID is more performant and has better collision resistance.
## Inbox Generation
The `Client::new_inbox()` method generates globally unique inbox subjects for request-reply:
```rust
pub fn new_inbox(&self) -> String {
format!("{}.{}", self.inbox_prefix, crate::id_generator::next())
}
```
Default prefix is `_INBOX`, producing subjects like `_INBOX.UaBG3f3q5NxX3KdNcRmF2f`.
Custom prefix via `ConnectOptions::custom_inbox_prefix()`:
```rust
let client = ConnectOptions::new()
.custom_inbox_prefix("MYAPP")
.connect("demo.nats.io")
.await?;
// Inbox subjects: MYAPP.UaBG3f3q5KdNcRmF2f
```
## DateTime Helpers
**Location**: `datetime.rs` (feature: `jetstream` or `service` or `chrono`)
Provides date/time types for JetStream and Service API timestamps:
- Uses the `time` crate by default
- Optionally uses `chrono` via the `chrono` feature flag
- Supports RFC 3339 formatting and parsing
- `DateTime` type wraps either `time::OffsetDateTime` or `chrono::DateTime<Utc>`
## Crypto Module
**Location**: `crypto.rs` (feature: `crypto`)
Provides encryption/decryption support used by the Object Store for server-side encryption.
## Subject Validation
**Location**: `lib.rs`
The client provides two levels of subject validation:
### is_valid_publish_subject
```rust
pub(crate) fn is_valid_publish_subject<T: AsRef<str>>(subject: T) -> bool
```
Checks for protocol safety only:
- Not empty
- No whitespace (space, tab, CR, LF) which would break protocol framing
Used for publish operations. Can be disabled with `skip_subject_validation`.
### is_valid_subject
```rust
pub(crate) fn is_valid_subject<T: AsRef<str>>(subject: T) -> bool
```
Checks structural validity:
- Not empty
- No leading/trailing dots
- No consecutive dots (`..`)
- No whitespace
Used for subscribe operations (always runs, matching Go/Java behavior).
### is_valid_queue_group
```rust
pub(crate) fn is_valid_queue_group(queue_group: &str) -> bool
```
Checks:
- Not empty
- No whitespace
## JetStream Name Validation
**Location**: `jetstream/mod.rs`
```rust
pub(crate) fn is_valid_name(name: &str) -> bool {
!name.is_empty()
&& name.bytes().all(|c| !c.is_ascii_whitespace() && c != b'.' && c != b'*' && c != b'>')
}
```
JetStream names (stream names, consumer names) must not contain:
- Whitespace
- Dots (`.`) — would conflict with subject delimiters
- Wildcards (`*`, `>`) — would conflict with subject wildcards
## CallbackArg1
**Location**: `options.rs`
A type-erased async callback wrapper used throughout the crate:
```rust
pub(crate) type AsyncCallbackArg1<A, T> =
Arc<dyn Fn(A) -> Pin<Box<dyn Future<Output = T> + Send + Sync + 'static>> + Send + Sync>;
#[derive(Clone)]
pub(crate) struct CallbackArg1<A, T>(AsyncCallbackArg1<A, T>);
impl<A, T> CallbackArg1<A, T> {
pub(crate) async fn call(&self, arg: A) -> T {
(self.0.as_ref())(arg).await
}
}
```
Used for:
- `event_callback` — `CallbackArg1<Event, ()>`
- `auth_callback` — `CallbackArg1<Vec<u8>, Result<Auth, AuthError>>`
- `reconnect_to_server_callback` — `CallbackArg1<(Vec<Server>, ServerInfo), Option<ReconnectToServer>>`
- `signature_callback` — `CallbackArg1<String, Result<String, AuthError>>`
## Version Compatibility Checking
The `Client::is_server_compatible` method checks if the server version meets a minimum requirement:
```rust
pub fn is_server_compatible(&self, major: i64, minor: i64, patch: i64) -> bool
```
This parses the server version string from `ServerInfo::version` using a regex and compares major/minor/patch components. Note: this checks the directly-connected server, not necessarily the JetStream leader.
The `server_2_10`, `server_2_11`, `server_2_12`, and `server_2_14` feature flags enable version-specific API fields and methods without runtime checks.

215
docs/research/references/nats.rs/nats-server/10-quick-reference.md Normal file
View File

@@ -0,0 +1,215 @@
# Quick Reference
## Crate Summary
| | |
|---|---|
| **Crate** | `async-nats` |
| **Version** | 0.49.1 |
| **Edition** | 2021 |
| **MSRV** | 1.88.0 |
| **License** | Apache-2.0 |
| **Runtime** | Tokio |
| **Protocol** | NATS Client Protocol v1 (Dynamic) |
| **TLS** | rustls (ring / aws-lc-rs / fips) |
| **WebSocket** | tokio-websockets (feature-gated) |
## Quick Start
```rust
use async_nats::connect;
use futures_util::StreamExt;
#[tokio::main]
async fn main() -> Result<(), async_nats::Error> {
let client = connect("demo.nats.io").await?;
// Publish
client.publish("events.data", "hello".into()).await?;
// Subscribe
let mut sub = client.subscribe("events.>").await?;
while let Some(msg) = sub.next().await {
println!("{:?}", msg);
}
// Request-Response
let response = client.request("service", "input".into()).await?;
Ok(())
}
```
## Architecture at a Glance
```
Client (cloneable handle, mpsc::Sender<Command>)
│
▼
ConnectionHandler (single Tokio task)
├── Subscriptions HashMap<u64, Subscription>
├── Multiplexer (request-reply, SID 0)
├── Flush Observers
└── Ping/Pong health check
│
▼
Connection (protocol I/O, read/write buffers)
│
▼
Connector (server pool, reconnection)
│
▼
NATS Server (Go binary, TCP/TLS/WebSocket)
```
## Key Types
| Type | Location | Purpose |
|------|----------|---------|
| `Client` | `client.rs` | Cloneable connection handle |
| `Subscriber` | `lib.rs` | Message stream (impl `futures::Stream`) |
| `Message` | `message.rs` | Inbound NATS message |
| `OutboundMessage` | `message.rs` | Outbound publish message |
| `Subject` | `subject.rs` | Validated subject string (backed by `Bytes`) |
| `HeaderMap` | `header.rs` | NATS message headers |
| `StatusCode` | `status.rs` | NATS protocol status codes |
| `ServerInfo` | `lib.rs` | Server INFO data |
| `ConnectInfo` | `lib.rs` | Client CONNECT data |
| `ServerAddr` | `lib.rs` | Validated server URL |
| `Auth` | `auth.rs` | Authentication credentials |
| `ConnectOptions` | `options.rs` | Connection configuration builder |
| `Event` | `lib.rs` | Connection lifecycle events |
| `State` | `connection.rs` | Connection state (Pending/Connected/Disconnected) |
| `Statistics` | `client.rs` | Atomic connection metrics |
| `Request` | `client.rs` | Request-response builder |
## JetStream Types
| Type | Location | Purpose |
|------|----------|---------|
| `jetstream::Context` | `jetstream/context.rs` | JetStream API entry point |
| `jetstream::stream::Stream` | `jetstream/stream.rs` | Stream management |
| `jetstream::stream::Config` | `jetstream/stream.rs` | Stream configuration |
| `jetstream::stream::Info` | `jetstream/stream.rs` | Stream info/state |
| `jetstream::consumer::PullConsumer` | `jetstream/consumer/pull.rs` | Pull-based consumer |
| `jetstream::consumer::PushConsumer` | `jetstream/consumer/push.rs` | Push-based consumer |
| `jetstream::consumer::Config` | `jetstream/consumer/mod.rs` | Consumer configuration |
| `jetstream::Message` | `jetstream/message.rs` | Message with ack methods |
| `jetstream::PublishAck` | `jetstream/publish.rs` | Publish acknowledgment |
| `jetstream::kv::Store` | `jetstream/kv/bucket.rs` | Key-Value store |
| `jetstream::object_store::ObjectStore` | `jetstream/object_store/mod.rs` | Object store |
| `jetstream::ErrorCode` | `jetstream/errors.rs` | JetStream error codes |
## Protocol Operations
### Client → Server (ClientOp)
| Op | Wire Format | Purpose |
|----|-----------|---------|
| `CONNECT` | `CONNECT {json}\r\n` | Authentication and capabilities |
| `PUB` | `PUB <subject> [reply] <len>\r\n<payload>\r\n` | Publish message |
| `HPUB` | `HPUB <subject> [reply] <hlen> <tlen>\r\n<hdrs><payload>\r\n` | Publish with headers |
| `SUB` | `SUB <subject> [queue] <sid>\r\n` | Subscribe |
| `UNSUB` | `UNSUB <sid> [max]\r\n` | Unsubscribe |
| `PING` | `PING\r\n` | Keepalive / health check |
| `PONG` | `PONG\r\n` | Response to server PING |
### Server → Client (ServerOp)
| Op | Wire Format | Purpose |
|----|-----------|---------|
| `INFO` | `INFO {json}\r\n` | Server capabilities, cluster info |
| `MSG` | `MSG <subj> <sid> [reply] <len>\r\n<payload>\r\n` | Deliver message |
| `HMSG` | `HMSG <subj> <sid> [reply] <hlen> <tlen>\r\n<hdrs><payload>\r\n` | Message with headers |
| `+OK` | `+OK\r\n` | Success (verbose mode) |
| `-ERR` | `-ERR <desc>\r\n` | Server error |
| `PING` | `PING\r\n` | Server health check |
| `PONG` | `PONG\r\n` | Ack client PING |
## Internal Commands (Command → ConnectionHandler)
| Command | Purpose |
|---------|---------|
| `Publish(OutboundMessage)` | Queue message for sending |
| `Request { subject, payload, respond, headers, sender }` | Request-response via multiplexer |
| `Subscribe { sid, subject, queue_group, sender }` | Create subscription |
| `Unsubscribe { sid, max }` | Remove subscription |
| `Flush { observer }` | Wait for write buffer flush |
| `Drain { sid }` | Gracefully drain (sub or whole client) |
| `Reconnect` | Force reconnection |
| `SetServerPool { servers, result }` | Replace server pool |
| `ServerPool { result }` | Query server pool |
## Feature Flags
| Feature | Default | Enables |
|---------|---------|---------|
| `jetstream` | ✓ | JetStream API (streams, consumers, publish) |
| `kv` | ✓ | Key-Value store (requires jetstream) |
| `object-store` | ✓ | Object store (requires jetstream + crypto) |
| `service` | ✓ | Service API |
| `nkeys` | ✓ | NKey/JWT authentication |
| `crypto` | ✓ | Encryption support |
| `websockets` | ✓ | WebSocket transport |
| `nuid` | ✓ | NUID ID generation |
| `ring` | ✓ | Ring crypto backend |
| `aws-lc-rs` | ✗ | Alternative crypto backend |
| `fips` | ✗ | FIPS mode (requires aws-lc-rs) |
| `chrono` | ✗ | Use chrono instead of time |
| `experimental` | ✗ | Experimental features |
| `server_2_10` | ✓ | Server 2.10+ API fields |
| `server_2_11` | ✓ | Server 2.11+ API fields |
| `server_2_12` | ✓ | Server 2.12+ API fields |
| `server_2_14` | ✓ | Server 2.14+ API fields |
## Connection Defaults
| Parameter | Default |
|-----------|---------|
| Connection timeout | 5 seconds |
| Ping interval | 60 seconds |
| Max pending pings | 2 |
| Request timeout | 10 seconds |
| Command channel capacity | 2048 |
| Subscription capacity | 65536 |
| Read buffer capacity | 65535 |
| Inbox prefix | `_INBOX` |
| Reconnect delay | Exponential (0ms → 4s cap) |
| Max reconnects | Unlimited |
| TLS required | Auto (server-dependent) |
## Error Hierarchy
```
ConnectError (ConnectErrorKind::ServerParse | Dns | Authentication | AuthorizationViolation | TimedOut | Tls | Io | MaxReconnects)
PublishError (PublishErrorKind::MaxPayloadExceeded | InvalidSubject | Send)
RequestError (RequestErrorKind::TimedOut | NoResponders | InvalidSubject | MaxPayloadExceeded | Other)
SubscribeError (SubscribeErrorKind::InvalidSubject | InvalidQueueName | Other)
FlushError (FlushErrorKind::SendError | FlushError)
```
## nats-server Test Harness
| Function | Description |
|----------|-------------|
| `run_server(cfg)` | Start single server with config |
| `run_basic_server()` | Start bare server |
| `run_cluster(cfg)` | Start 3-node cluster |
| `set_lame_duck_mode(s)` | Send LDM signal |
## JetStream API Subjects
| Operation | Subject Pattern |
|-----------|---------------|
| Create stream | `$JS.API.STREAM.CREATE.<name>` |
| Stream info | `$JS.API.STREAM.INFO.<name>` |
| Update stream | `$JS.API.STREAM.UPDATE.<name>` |
| Delete stream | `$JS.API.STREAM.DELETE.<name>` |
| Purge stream | `$JS.API.STREAM.PURGE.<name>` |
| List streams | `$JS.API.STREAM.LIST` |
| Create consumer | `$JS.API.CONSUMER.CREATE.<stream>` |
| Create durable | `$JS.API.CONSUMER.DURABLE.CREATE.<stream>.<name>` |
| Consumer info | `$JS.API.CONSUMER.INFO.<stream>.<name>` |
| Pull next | `$JS.API.CONSUMER.MSG.NEXT.<stream>.<name>` |
| Account info | `$JS.API.ACCOUNT.INFO` |
| Direct get | `$JS.API.DIRECT.GET.<name>` |