diff --git a/docs/research/references/nats.rs/nats-async/01-overview-and-architecture.md b/docs/research/references/nats.rs/nats-async/01-overview-and-architecture.md new file mode 100644 index 0000000..9535b6c --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/01-overview-and-architecture.md @@ -0,0 +1,170 @@ +# async-nats: Overview & Architecture + +**Crate**: `async-nats` +**Version**: 0.49.1 +**Repository**: https://github.com/nats-io/nats.rs +**License**: Apache-2.0 +**Rust Edition**: 2021 +**MSRV**: 1.88.0 +**Async Runtime**: Tokio + +## What is async-nats? + +`async-nats` is the official async Rust client for the [NATS messaging system](https://nats.io). It provides a Tokio-based asynchronous interface to NATS server features including: + +- **Core NATS** — publish/subscribe, request/reply, queue groups +- **JetStream** — persistent stream-based messaging with at-least-once and exactly-once semantics +- **Key-Value Store** — KV abstraction built on JetStream streams +- **Object Store** — large-object storage built on JetStream streams +- **Service API** — microservice request/reply pattern with built-in PING/INFO/STATS verbs + +The crate is positioned as the **core client** in the NATS Rust ecosystem. A separate project, [Orbit](https://github.com/synadia-io/orbit.rs), provides higher-level opinionated abstractions on top. + +``` + ┌──────────────────────────────────────────────────────┐ + │ Application code │ + └──────────────┬───────────────────────────┬───────────┘ + │ │ + ▼ ▼ + ┌───────────────────┐ ┌───────────────────┐ + │ Orbit crates │ uses │ async-nats (core) │ + │ (opinionated, │──────▶│ (parity, stable, │ + │ per-crate semver) │ │ protocol-level) │ + └───────────────────┘ └─────────┬─────────┘ + │ + ▼ + ┌─────────────┐ + │ nats-server │ + └─────────────┘ +``` + +## Feature Flags + +Features are extensive and control which subsystems are compiled: + +| Feature | Default | Description | +|---------|---------|-------------| +| `jetstream` | ✅ | JetStream API (streams, consumers, publish) | +| `kv` | ✅ | Key-Value store (depends on `jetstream`) | +| `object-store` | ✅ | Object store (depends on `jetstream` + `crypto`) | +| `service` | ✅ | Service API (microservice pattern) | +| `nkeys` | ✅ | NKey/JWT authentication | +| `nuid` | ✅ | NUID-based unique ID generation | +| `crypto` | ✅ | Cryptographic primitives (SHA-256 for object store) | +| `websockets` | ✅ | WebSocket transport (`ws://`/`wss://`) | +| `ring` | ✅ | Use `ring` as TLS crypto backend | +| `aws-lc-rs` | ❌ | Use `aws-lc-rs` as TLS crypto backend | +| `fips` | ❌ | FIPS 140-2 compliant via `aws-lc-rs` | +| `chrono` | ❌ | Use `chrono` instead of `time` for datetime types | +| `server_2_10` | ✅ | Server 2.10+ features | +| `server_2_11` | ✅ | Server 2.11+ features | +| `server_2_12` | ✅ | Server 2.12+ features | +| `server_2_14` | ✅ | Server 2.14+ features | +| `experimental` | ❌ | Experimental features | + +## Source Structure + +``` +async-nats/src/ +├── lib.rs # Entry point: connect(), ServerInfo, Command, ClientOp, ServerOp, +│ ConnectionHandler, Subscriber, Event, ServerAddr, ConnectInfo +├── client.rs # Client struct, publish/subscribe/request/drain/flush APIs, +│ Request builder, Statistics, trait definitions +├── connection.rs # Framed connection: NATS protocol parser/serializer, +│ read/write buffer management, WebSocket adapter +├── connector.rs # Server pool, reconnection logic, TLS setup, DNS resolution, +│ authentication handshake +├── options.rs # ConnectOptions builder, auth methods, TLS config, callbacks +├── auth.rs # Auth struct (username, password, token, JWT, nkey, signature) +├── auth_utils.rs # Credentials file parsing (JWT + NKey seed) +├── message.rs # Message (inbound), OutboundMessage (outbound) +├── header.rs # HeaderMap, HeaderName, HeaderValue (NATS headers) +├── subject.rs # Subject type, ToSubject trait, SubjectError +├── status.rs # StatusCode enum (NATS status codes) +├── error.rs # Generic Error type used throughout +├── datetime.rs # DateTime type (time or chrono backend) +├── id_generator.rs # Unique ID generation (NUID or rand fallback) +├── tls.rs # TLS configuration helper +├── crypto.rs # SHA-256 for object store integrity +├── jetstream/ +│ ├── mod.rs # Module entry: new(), with_domain(), with_prefix() +│ ├── context.rs # Context: JetStream API (streams, consumers, KV, OS, publish) +│ ├── stream.rs # Stream handle, Config, Info, purge/delete/message ops +│ ├── consumer/ +│ │ ├── mod.rs # Consumer trait, Info, Config base +│ │ ├── pull.rs # PullConsumer: batch fetch, sequence, messages stream +│ │ └── push.rs # PushConsumer: Ordered push consumer with auto-recreate +│ ├── publish.rs # PublishAck, PublishAckFuture, PublishMessage builder +│ ├── message.rs # JetStream Message (with ack methods), AckKind +│ ├── response.rs # Response (Ok/Err) for JetStream API calls +│ ├── errors.rs # ErrorCode, Error for JetStream +│ ├── account.rs # Account info +│ ├── kv/ +│ │ ├── mod.rs # Store: put/get/delete/purge/watch/history/keys +│ │ └── bucket.rs # Bucket Status +│ └── object_store/ +│ └── mod.rs # ObjectStore: put/get/delete/watch/list/seal, Object (AsyncRead) +└── service/ + ├── mod.rs # Service, ServiceBuilder, Group, EndpointBuilder, Request + └── endpoint.rs # Endpoint stream, Stats, Info +``` + +## Architecture: Core Connection Model + +The client uses a **single-connection, actor-model** design: + +``` + ┌──────────────────────────────────────┐ + Client (clone) ──▶│ mpsc::Sender │ + (many handles) │ (bounded channel) │ + └────────────┬────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────┐ + │ ConnectionHandler (tokio::task) │ + │ - Receives Command from channel │ + │ - Converts to ClientOp │ + │ - Manages subscriptions map │ + │ - Manages multiplexer (request/reply)│ + │ - Pings server on interval │ + │ - Handles reconnection │ + └────────────┬──────────────────────────┘ + │ + ▼ + ┌────────────────────────────────────────┐ + │ Connection (framed TCP/TLS/WS) │ + │ - Protocol parser (try_read_op) │ + │ - Write buffer (VecDeque) │ + │ - Vectored I/O support │ + │ - Read buffer (BytesMut) │ + └────────────┬──────────────────────────┘ + │ + ▼ + nats-server +``` + +### Key Design Decisions + +1. **Cloneable Client**: `Client` is `Clone` (via `mpsc::Sender` clone), enabling shared use across tasks +2. **Single TCP connection**: All traffic (Core NATS, JetStream API, etc.) multiplexes over one connection +3. **Background task**: `ConnectionHandler` runs as a spawned Tokio task, bridging the mpsc channel to the TCP stream +4. **Automatic reconnection**: On disconnect, `Connector` retries servers from the pool with exponential backoff +5. **Subscription rehydration**: On reconnect, all active subscriptions are re-subscribed with adjusted `max` counts +6. **Multiplexer for request/reply**: A single wildcard subscription (`_INBOX..*`) multiplexes all pending request/reply correlations + +## Dependencies (Key) + +| Crate | Purpose | +|-------|---------| +| `tokio` | Async runtime, TCP, time, sync, io-util | +| `bytes` | Efficient byte buffer (`Bytes`, `BytesMut`) | +| `tokio-rustls` | TLS via rustls | +| `rustls-native-certs` | Load system root certificates | +| `serde` / `serde_json` | JSON serialization for JetStream API | +| `futures-util` | Stream trait, Sink trait, StreamExt | +| `tracing` | Structured logging | +| `thiserror` | Error derive macros | +| `memchr` | Fast substring search for protocol parsing | +| `portable-atomic` | Atomic types with portable-atomic fallback | +| `tokio-util` | `PollSender` for Sink implementation | +| `tokio-stream` | `ReceiverStream` adapter | \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/02-key-types-and-traits.md b/docs/research/references/nats.rs/nats-async/02-key-types-and-traits.md new file mode 100644 index 0000000..1f2bd78 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/02-key-types-and-traits.md @@ -0,0 +1,404 @@ +# async-nats: Key Types & Traits + +## Core Types + +### `Client` + +The primary handle to a NATS connection. Cheaply cloneable (wraps `mpsc::Sender`). + +```rust +#[derive(Clone, Debug)] +pub struct Client { + info: tokio::sync::watch::Receiver>, + state: tokio::sync::watch::Receiver, + sender: mpsc::Sender, + poll_sender: PollSender, + next_subscription_id: Arc, + subscription_capacity: usize, + inbox_prefix: Arc, + request_timeout: Option, + max_payload: Arc, + connection_stats: Arc, + skip_subject_validation: bool, +} +``` + +**Key methods**: +- `publish(subject, payload)` — fire-and-forget publish +- `publish_with_headers(subject, headers, payload)` — publish with NATS headers +- `publish_with_reply(subject, reply, payload)` — publish with reply-to subject +- `subscribe(subject)` → `Subscriber` — subscribe to a subject +- `queue_subscribe(subject, queue_group)` → `Subscriber` — queue group subscription +- `request(subject, payload)` → `Message` — request/reply with default timeout +- `send_request(subject, request)` → `Message` — request with custom `Request` builder +- `flush()` — wait until all buffered writes are flushed to the server +- `drain()` — drain all subscriptions, flush, then close +- `force_reconnect()` — force a reconnection (e.g., to re-trigger auth) +- `new_inbox()` — generate a unique inbox subject (`_INBOX.`) +- `server_info()` → `ServerInfo` — last known server info +- `connection_state()` → `State` — `Pending`/`Connected`/`Disconnected` +- `statistics()` → `Arc` — connection statistics (bytes, messages, connects) +- `max_payload()` → `usize` — server's max payload size +- `set_server_pool(addrs)` — replace the server pool for reconnection +- `server_pool()` — snapshot of current server pool + +### `Subscriber` + +A `Stream` yielding `Message` values from a subscription. + +```rust +#[derive(Debug)] +pub struct Subscriber { + sid: u64, + receiver: mpsc::Receiver, + sender: mpsc::Sender, +} +``` + +Implements `futures_util::Stream`. Methods: +- `unsubscribe()` — immediately unsubscribe +- `unsubscribe_after(n)` — unsubscribe after `n` total delivered messages +- `drain()` — unsubscribe after in-flight messages are delivered + +**Drop behavior**: When a `Subscriber` is dropped, it spawns a task to send `Command::Unsubscribe` to the connection handler, ensuring the server is always notified. + +### `Message` + +An inbound NATS message: + +```rust +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct Message { + pub subject: Subject, + pub reply: Option, + pub payload: Bytes, + pub headers: Option, + pub status: Option, + pub description: Option, + pub length: usize, +} +``` + +### `OutboundMessage` + +An outbound message for publishing (no status/description): + +```rust +#[derive(Clone, Debug)] +pub struct OutboundMessage { + pub subject: Subject, + pub reply: Option, + pub payload: Bytes, + pub headers: Option, +} +``` + +### `Request` + +Builder for request/reply calls: + +```rust +#[derive(Default)] +pub struct Request { + pub payload: Option, + pub headers: Option, + pub timeout: Option>, + pub inbox: Option, +} +``` + +Builder methods: `payload()`, `headers()`, `timeout()`, `inbox()`. The `inbox` field, when set, bypasses the multiplexer and uses a dedicated subscription instead. + +### `ServerInfo` + +Server metadata received during connection handshake: + +```rust +#[derive(Debug, Deserialize, Default, Clone, Eq, PartialEq)] +pub struct ServerInfo { + pub server_id: String, + pub server_name: String, + pub host: String, + pub port: u16, + pub version: String, + pub auth_required: bool, + pub tls_required: bool, + pub max_payload: usize, + pub proto: i8, + pub client_id: u64, + pub go: String, + pub nonce: String, + pub connect_urls: Vec, + pub client_ip: String, + pub headers: bool, + pub lame_duck_mode: bool, + pub cluster: Option, + pub domain: Option, + pub jetstream: bool, +} +``` + +### `ConnectInfo` + +Client → server `CONNECT` message payload: + +```rust +#[derive(Clone, Debug, Serialize)] +pub struct ConnectInfo { + pub verbose: bool, + pub pedantic: bool, + pub user_jwt: Option, + pub nkey: Option, + pub signature: Option, + pub name: Option, + pub echo: bool, + pub lang: String, + pub version: String, + pub protocol: Protocol, // Original(0) or Dynamic(1) + pub tls_required: bool, + pub user: Option, + pub pass: Option, + pub auth_token: Option, + pub headers: bool, + pub no_responders: bool, +} +``` + +The client always sets: `verbose=false`, `pedantic=false`, `lang="rust"`, `protocol=Dynamic`, `headers=true`, `no_responders=true`. + +### `Statistics` + +Atomic connection statistics (shared via `Arc`): + +```rust +#[derive(Default, Debug)] +pub struct Statistics { + pub in_bytes: AtomicU64, + pub out_bytes: AtomicU64, + pub in_messages: AtomicU64, + pub out_messages: AtomicU64, + pub connects: AtomicU64, +} +``` + +## Subject Types + +### `Subject` + +A validated NATS subject string (newtype over `String`): + +```rust +// Usage: +let subject: Subject = "foo.bar.baz".into(); +``` + +### `ToSubject` trait + +Conversion trait for subjects: + +```rust +pub trait ToSubject { + fn to_subject(self) -> Result; +} +``` + +Implemented for `&str`, `String`, `Subject` directly. + +### `SubjectError` + +```rust +pub enum SubjectError { + InvalidFormat, +} +``` + +## Header Types + +### `HeaderMap` + +A multimap of header name → values: + +```rust +pub struct HeaderMap { + inner: VecMap>, +} +``` + +Methods: `insert()`, `append()`, `get()`, `len()`, `is_empty()`, `iter()`, `to_bytes()`. + +### `HeaderName` + +Case-insensitive header name. Created via `FromStr`: + +```rust +let name: HeaderName = "Nats-Expected-Last-Subject-Sequence".parse()?; +``` + +### `HeaderValue` + +Header value string. Created via `FromStr` or `From`: + +```rust +let val: HeaderValue = "some value".parse()?; +let val: HeaderValue = HeaderValue::from(42u64); +``` + +## Server Address Types + +### `ServerAddr` + +Wraps a `url::Url` with NATS-specific validation. Supports schemes: `nats://`, `tls://`, `ws://`, `wss://`. Default port is `4222`. + +```rust +let addr: ServerAddr = "demo.nats.io".parse()?; +let addr: ServerAddr = "nats://demo.nats.io:4222".parse()?; +let addr: ServerAddr = "tls://demo.nats.io".parse()?; +``` + +### `ToServerAddrs` trait + +Flexible server address input (single URL, `Vec`, slice, etc.): + +```rust +pub trait ToServerAddrs { + type Iter: Iterator; + fn to_server_addrs(&self) -> io::Result; +} +``` + +### `Server` + +Metadata about a server in the pool: + +```rust +pub struct Server { + pub addr: ServerAddr, + pub failed_attempts: usize, + pub did_connect: bool, + pub is_discovered: bool, + pub last_error: Option, +} +``` + +## Event & State Types + +### `Event` + +Asynchronous notifications from the connection: + +```rust +pub enum Event { + Connected, + Disconnected, + LameDuckMode, + Draining, + Closed, + SlowConsumer(u64), // subscription sid + ServerError(ServerError), + ClientError(ClientError), +} +``` + +Received via `ConnectOptions::event_callback()`. + +### `State` + +Connection state observable via `watch::Receiver`: + +```rust +pub enum State { + Pending, + Connected, + Disconnected, +} +``` + +### `StatusCode` + +NATS protocol status codes (e.g., `NO_RESPONDERS = 404`, `TIMEOUT = 408`). + +## Error Types + +All error types follow the pattern `Error` from `crate::error`: + +| Error Type | Kind | Used By | +|------------|------|---------| +| `ConnectError` | `ConnectErrorKind` | Connection establishment | +| `PublishError` | `PublishErrorKind` | Publish operations | +| `RequestError` | `RequestErrorKind` | Request/reply | +| `SubscribeError` | `SubscribeErrorKind` | Subscribe | +| `FlushError` | `FlushErrorKind` | Flush | +| `DrainError` | — | Drain | + +### `ConnectErrorKind` + +```rust +pub enum ConnectErrorKind { + ServerParse, // URL parsing failed + Dns, // DNS resolution failed + Authentication, // Auth signing failed + AuthorizationViolation, // Server rejected auth + TimedOut, // Connection handshake timeout + Tls, // TLS error + Io, // Other I/O error + MaxReconnects, // Exceeded max reconnect attempts +} +``` + +## Trait Definitions + +The `client::traits` module defines abstract interfaces: + +```rust +pub trait Publisher { + fn publish_with_reply(&self, subject, reply, payload) -> Future>; + fn publish_message(&self, msg: OutboundMessage) -> Future>; +} + +pub trait Subscriber { + fn subscribe(&self, subject) -> Future>; +} + +pub trait Requester { + fn send_request(&self, subject, request: Request) -> Future>; +} + +pub trait TimeoutProvider { + fn timeout(&self) -> Option; +} +``` + +`Client` implements all of these. The JetStream `Context` also implements them via delegation. + +## Authentication Types + +### `Auth` + +Container for all authentication methods: + +```rust +pub struct Auth { + pub jwt: Option, + pub nkey: Option, + pub signature_callback: Option>>, + pub signature: Option>, + pub username: Option, + pub password: Option, + pub token: Option, +} +``` + +### `AuthError` + +Simple string error for auth callback failures. + +### `ReconnectToServer` + +Returned by `reconnect_to_server_callback` to select a server and delay: + +```rust +pub struct ReconnectToServer { + pub addr: ServerAddr, + pub delay: Option, +} +``` \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/03-protocol-and-wire-format.md b/docs/research/references/nats.rs/nats-async/03-protocol-and-wire-format.md new file mode 100644 index 0000000..1a8003d --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/03-protocol-and-wire-format.md @@ -0,0 +1,278 @@ +# async-nats: NATS Protocol & Wire Format + +## Protocol Overview + +NATS uses a simple, text-based protocol over TCP. Messages are terminated with `\r\n`. The protocol is symmetric for client and server operations. + +### Client → Server Operations (`ClientOp`) + +```rust +pub(crate) enum ClientOp { + Publish { subject, payload, respond, headers }, + Subscribe { sid, subject, queue_group }, + Unsubscribe { sid, max }, + Ping, + Pong, + Connect(ConnectInfo), +} +``` + +### Server → Client Operations (`ServerOp`) + +```rust +pub(crate) enum ServerOp { + Ok, + Info(Box), + Ping, + Pong, + Error(ServerError), + Message { sid, subject, reply, payload, headers, status, description, length }, +} +``` + +## Wire Format: Client Operations + +### CONNECT + +Sent immediately after receiving the first `INFO` from the server: + +``` +CONNECT {"verbose":false,"pedantic":false,...}\r\n +``` + +The JSON payload is `ConnectInfo` serialized inline on the same line. + +### PUB (Publish without headers) + +``` +PUB [reply-to] \r\n +\r\n +``` + +Example: +``` +PUB events.data INBOX.67 11\r\n +Hello World\r\n +``` + +### HPUB (Publish with headers) + +When headers are present and non-empty: + +``` +HPUB [reply-to] \r\n +\r\n +\r\n +``` + +The `` = `` + ``. + +Header block format: +``` +NATS/1.0\r\n +Header-Name: Header-Value\r\n +Another-Header: Another-Value\r\n +\r\n +``` + +The version line (`NATS/1.0`) may include a status code and description: +``` +NATS/1.0 404 No Messages\r\n +\r\n +``` + +### SUB (Subscribe) + +``` +SUB [queue-group] \r\n +``` + +The `sid` (subscription ID) is a client-assigned u64, unique per connection. + +### UNSUB (Unsubscribe) + +``` +UNSUB [max]\r\n +``` + +The optional `max` tells the server to auto-unsubscribe after `max` messages are delivered. + +### PING / PONG + +``` +PING\r\n +PONG\r\n +``` + +Client sends PING periodically (default every 60s). If 2+ pings are pending without PONG, the connection is considered dead. + +## Wire Format: Server Operations + +### INFO + +First message sent by the server on connection: + +``` +INFO {"server_id":"NATSxxx","version":"2.10"...}\r\n +``` + +Also sent asynchronously when cluster topology changes. + +### MSG (Message without headers) + +``` +MSG [reply-to] \r\n +\r\n +``` + +### HMSG (Message with headers) + +``` +HMSG [reply-to] \r\n +\r\n +``` + +### +OK / -ERR + +``` ++OK\r\n +-ERR \r\n +``` + +Sent only when `verbose=true` in `CONNECT`. The client always sets `verbose=false`, so `+OK` is not expected. + +## Protocol Parser + +The `Connection` struct handles all protocol parsing and serialization: + +### Read Path (`try_read_op`) + +1. Search for `\r\n` in `read_buf` using `memchr::memmem::find` +2. Inspect the first bytes to determine the operation type: + - `+OK` → `ServerOp::Ok` + - `PING` → `ServerOp::Ping` + - `PONG` → `ServerOp::Pong` + - `-ERR` → `ServerOp::Error(...)` (description is `trim_matches('\'')`) + - `INFO ` → `ServerOp::Info(...)` (serde_json deserialization) + - `MSG ` → Parse subject/sid/reply/size, then read payload + - `HMSG ` → Parse subject/sid/reply/header_len/total_len, then read headers + payload +3. For `MSG`/`HMSG`: if the full message body hasn't been read yet, return `None` (wait for more data) +4. For `HMSG`: parse the header block — extract version line (`NATS/1.0[ [ ]]`), then key-value pairs (supports folded/multi-line header values) + +### Write Path (`enqueue_write_op`) + +Writes into a buffer strategy: +- **Small writes** (< 4096 bytes): flattened into `flattened_writes: BytesMut` +- **Large writes** (≥ 4096 bytes): appended as separate `Bytes` chunks in `write_buf: VecDeque` + +This enables efficient vectored I/O when the underlying stream supports it. + +### Write Flush Strategy + +The `should_flush()` method returns: +- `Yes` — buffers empty but haven't flushed yet +- `May` — buffers not empty and haven't flushed +- `No` — already flushed or nothing to flush + +The `ConnectionHandler` calls `poll_flush()` after processing commands, ensuring data is actually sent to the server. + +## Vectored I/O + +When `stream.is_write_vectored()` returns true, the connection uses `poll_write_vectored()` to write up to 64 `IoSlice`s at once. This is significantly more efficient for bursty publish patterns. + +```rust +const WRITE_VECTORED_CHUNKS: usize = 64; +``` + +## WebSocket Transport + +When the `websockets` feature is enabled, `WebSocketAdapter` wraps `tokio_websockets::WebSocketStream` to implement `AsyncRead + AsyncWrite`, making WebSocket connections transparent to the protocol layer. + +```rust +#[cfg(feature = "websockets")] +pub(crate) struct WebSocketAdapter { + pub(crate) inner: WebSocketStream, + pub(crate) read_buf: BytesMut, +} +``` + +WebSocket connections use `ws://` or `wss://` scheme in the server URL. TLS for `wss://` is handled by the WebSocket library's built-in TLS support. + +## Connection Lifecycle + +### Initial Connection Flow + +``` +Client Server + │ │ + │──── TCP connect ────────────────────▶ │ + │◀──── INFO {server_id, nonce, ...} ─── │ + │──── CONNECT {auth, ...} ──────────▶ │ + │──── PING ─────────────────────────▶ │ + │◀──── PONG (or -ERR) ─────────────── │ + │ │ + │ [connected, ConnectionHandler runs] │ +``` + +If `tls_first` is enabled, TLS is established before reading INFO: + +``` +Client Server + │ │ + │──── TCP connect ────────────────────▶ │ + │──── TLS handshake ─────────────────▶ │ + │◀──── TLS handshake ──────────────── │ + │◀──── INFO {...} ──────────────────── │ + │──── CONNECT + PING ────────────────▶ │ + │◀──── PONG ────────────────────────── │ +``` + +### Ping/Pong Keepalive + +- Client sends PING every `ping_interval` (default 60s) +- Server responds with PONG +- If `pending_pings > MAX_PENDING_PINGS (2)`, connection is considered dead +- Any server operation resets the ping interval timer + +### Reconnection Flow + +On disconnect: +1. `handle_disconnect()` sends `Event::Disconnected` and sets state to `Disconnected` +2. `handle_reconnect()` calls `connector.connect()` which: + - Shuffles servers (unless `retain_servers_order`) + - Sorts by `failed_attempts` (ascending) + - Iterates through servers with exponential backoff delay + - On each server: DNS resolve → TCP connect → INFO → TLS (if needed) → CONNECT+PING → PONG +3. On success: + - Sends `Event::Connected`, sets state to `Connected` + - Removes closed subscriptions + - Re-subscribes all active subscriptions (with adjusted `max = max - delivered`) + - Re-subscribes the multiplexer (if active) +4. On failure with `MaxReconnects` reached, the handler loop exits + +### Default Reconnect Delay + +Exponential backoff capped at 4 seconds: + +```rust +fn reconnect_delay_callback_default(attempts: usize) -> Duration { + if attempts <= 1 { + Duration::from_millis(0) + } else { + let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX); + cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), Duration::from_secs(4)) + } +} +``` + +| Attempt | Delay | +|---------|-------| +| 1 | 0ms | +| 2 | 0ms | +| 3 | 2ms | +| 4 | 8ms | +| 5 | 32ms | +| 6 | 128ms | +| 7 | 512ms | +| 8 | 2048ms | +| 9+ | 4000ms (cap) | \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/04-connection-management.md b/docs/research/references/nats.rs/nats-async/04-connection-management.md new file mode 100644 index 0000000..9cf0131 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/04-connection-management.md @@ -0,0 +1,221 @@ +# async-nats: Connection Management & Configuration + +## ConnectOptions Builder + +`ConnectOptions` provides a builder for all connection configuration: + +```rust +let client = ConnectOptions::new() + .require_tls(true) + .ping_interval(Duration::from_secs(10)) + .name("my-service") + .connect("demo.nats.io") + .await?; +``` + +### Authentication Methods + +| Method | Description | +|--------|-------------| +| `with_token(token)` | Token-based auth | +| `with_user_and_password(user, pass)` | Username/password auth | +| `with_nkey(seed)` | NKey auth (requires `nkeys` feature) | +| `with_jwt(jwt, sign_cb)` | JWT + signing callback (requires `nkeys`) | +| `with_credentials_file(path)` | Load from `.creds` file (requires `nkeys`) | +| `with_credentials(creds_str)` | Parse credentials string (requires `nkeys`) | +| `with_auth_callback(cb)` | Dynamic auth callback receiving nonce, returning `Auth` | + +The auth callback is the most flexible — it receives the server nonce and can return any combination of auth fields: + +```rust +ConnectOptions::with_auth_callback(move |nonce| async move { + let mut auth = Auth::new(); + auth.username = Some("user".to_string()); + auth.password = Some("pass".to_string()); + Ok(auth) +}) +``` + +### TLS Configuration + +| Option | Description | +|-------|-------------| +| `require_tls(bool)` | Require TLS for the connection | +| `tls_first()` | Establish TLS before INFO (requires server `handshake_first`) | +| `add_root_certificates(path)` | Load root CA certificates from PEM file | +| `add_client_certificate(cert, key)` | Load client certificate for mTLS | +| `tls_client_config(config)` | Pass a custom `rustls::ClientConfig` | + +Two TLS crypto backends: `ring` (default) or `aws-lc-rs` (via feature flags). FIPS mode available via `aws-lc-rs` + `fips` features. + +### Connection Behavior + +| Option | Default | Description | +|--------|---------|-------------| +| `connection_timeout` | 5s | Timeout for full connection establishment | +| `request_timeout` | 10s | Default timeout for `Client::request` | +| `ping_interval` | 60s | How often client sends PING | +| `retry_on_initial_connect` | false | Return client immediately, connect in background | +| `max_reconnects` | None (unlimited) | Max consecutive reconnect attempts | +| `ignore_discovered_servers` | false | Ignore servers advertised in INFO | +| `retain_servers_order` | false | Don't shuffle server list on reconnect | +| `skip_subject_validation` | false | Skip whitespace validation on publish subjects | +| `subscription_capacity` | 65536 | mpsc channel capacity per subscription | +| `client_capacity` | 2048 | mpsc channel capacity for command sender | +| `custom_inbox_prefix` | `_INBOX` | Custom prefix for inbox subjects | +| `read_buffer_capacity` | 65535 | Initial size of the protocol read buffer | +| `local_address` | None | Local socket address to bind to | +| `no_echo` | false | Don't deliver messages published by this connection | + +### Reconnection Callbacks + +**`reconnect_delay_callback`**: Custom backoff strategy: + +```rust +.reconnect_delay_callback(|attempts| { + Duration::from_millis(std::cmp::min((attempts * 100) as u64, 8000)) +}) +``` + +**`reconnect_to_server_callback`**: Select which server to connect to on each reconnect attempt: + +```rust +.reconnect_to_server_callback(|servers, _info| async move { + servers.first().map(|s| ReconnectToServer { + addr: s.addr.clone(), + delay: Some(Duration::ZERO), + }) +}) +``` + +Receives `(Vec, ServerInfo)`, returns `Option`. If the returned server isn't in the pool, falls back to default selection. + +**`event_callback`**: Receive async notifications: + +```rust +.event_callback(|event| async move { + match event { + Event::Disconnected => println!("disconnected"), + Event::Connected => println!("connected"), + Event::SlowConsumer(sid) => eprintln!("slow consumer: {sid}"), + _ => {} + } +}) +``` + +## Connection Handler Internals + +### ProcessFut — The Core Event Loop + +The `ConnectionHandler::process()` method creates a custom `Future` (`ProcessFut`) that drives the connection forward. Each `poll()` call: + +1. **Check ping interval** — if timer ticked, send PING; if too many pending pings, disconnect +2. **Read server operations** — drain all available `ServerOp`s from `Connection::poll_read_op()` +3. **Process drain completions** — remove subscriptions that finished draining +4. **Handle commands** — receive up to 16 `Command`s from the mpsc channel and process them +5. **Write to socket** — flush the write buffer via `Connection::poll_write()` +6. **Flush** — call `poll_flush()` on the underlying stream when needed +7. **Check reconnect flag** — if `should_reconnect` is set, shut down and reconnect + +```rust +const RECV_CHUNK_SIZE: usize = 16; +``` + +### Exit Reasons + +The event loop exits with one of: + +| Reason | Action | +|--------|--------| +| `Disconnected(Option)` | Attempt reconnection | +| `ReconnectRequested` | Shut down stream, attempt reconnection | +| `Closed` | Send `Event::Closed`, exit loop | + +### Handle Disconnect & Reconnect + +```rust +async fn handle_disconnect(&mut self) -> Result<(), ConnectError> { + self.pending_pings = 0; + self.connector.events_tx.try_send(Event::Disconnected).ok(); + self.connector.state_tx.send(State::Disconnected).ok(); + self.handle_reconnect().await +} + +async fn handle_reconnect(&mut self) -> Result<(), ConnectError> { + let (info, connection) = self.connector.connect().await?; + self.connection = connection; + let _ = self.info_sender.send(Some(info)); + + // Remove closed subscriptions + self.subscriptions.retain(|_, sub| !sub.sender.is_closed()); + + // Re-subscribe all active subscriptions + for (sid, subscription) in &self.subscriptions { + self.connection.enqueue_write_op(&ClientOp::Subscribe { + sid: *sid, + subject: subscription.subject.to_owned(), + queue_group: subscription.queue_group.to_owned(), + }); + if let Some(max) = subscription.max { + self.connection.enqueue_write_op(&ClientOp::Unsubscribe { + sid: *sid, + max: Some(max.saturating_sub(subscription.delivered)), + }); + } + } + + // Re-subscribe multiplexer if active + if let Some(multiplexer) = &self.multiplexer { + self.connection.enqueue_write_op(&ClientOp::Subscribe { + sid: MULTIPLEXER_SID, + subject: multiplexer.subject.to_owned(), + queue_group: None, + }); + } + Ok(()) +} +``` + +## Request/Reply Multiplexer + +The client uses a **multiplexer** pattern for request/reply to avoid creating a separate subscription per request: + +1. A single wildcard subscription is created on first request: `_INBOX..*` +2. Each request gets a unique token appended to the inbox: `_INBOX..` +3. When a response arrives, the token is extracted from the subject and used to look up the `oneshot::Sender` in `multiplexer.senders` +4. The response is forwarded through the oneshot channel to the waiting `send_request()` future + +```rust +struct Multiplexer { + subject: Subject, // _INBOX..* + prefix: Subject, // _INBOX.. + senders: HashMap>, // token → sender +} +``` + +The multiplexer subscription uses `sid = 0` (`MULTIPLEXER_SID`), which is separate from regular subscription IDs (which start at 1). + +### Custom Inbox Bypass + +If a `Request` has a custom `inbox` set, the multiplexer is bypassed — a dedicated subscription is created for that specific request, and the timeout/response logic is handled locally within `send_request()`. + +## Server Pool Management + +The `Connector` maintains a `Vec` pool. Servers can come from: +1. **Explicit URLs** — provided by the user at connect time +2. **Discovered servers** — advertised in `INFO.connect_urls` (unless `ignore_discovered_servers` is set) + +On reconnection: +- Servers are shuffled (unless `retain_servers_order`) +- Sorted by `failed_attempts` (ascending) — prefer servers that haven't failed recently +- Each server is tried with exponential backoff delay +- On success: `failed_attempts` reset to 0, `did_connect` set to true +- On failure: `failed_attempts` incremented, `last_error` updated + +### Dynamic Server Pool Updates + +`Client::set_server_pool()` replaces the pool at runtime: +- Per-server state is preserved for servers that appear in both old and new pools +- The global reconnection attempt counter is reset +- Cannot mix WebSocket and non-WebSocket URLs +- Pool cannot be empty \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/05-jetstream.md b/docs/research/references/nats.rs/nats-async/05-jetstream.md new file mode 100644 index 0000000..bae7ce3 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/05-jetstream.md @@ -0,0 +1,373 @@ +# async-nats: JetStream + +## Overview + +JetStream is NATS' built-in persistence layer, providing stream-based messaging with at-least-once and exactly-once delivery semantics. The `async-nats` JetStream API is accessed through a `Context` object. + +### Creating a Context + +```rust +// Default context (prefix: $JS.API) +let jetstream = async_nats::jetstream::new(client); + +// With domain (prefix: $JS..API) +let jetstream = async_nats::jetstream::with_domain(client, "hub"); + +// With custom prefix +let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API"); + +// Builder with fine-grained control +let context = ContextBuilder::new() + .timeout(Duration::from_secs(5)) + .api_prefix("MY.JS.API") + .max_ack_inflight(1000) + .backpressure_on_inflight(true) + .ack_timeout(Duration::from_secs(30)) + .build(client); +``` + +## Context + +```rust +#[derive(Debug, Clone)] +pub struct Context { + pub(crate) client: Client, + pub(crate) prefix: String, + pub(crate) timeout: Duration, + pub(crate) max_ack_semaphore: Arc, + pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver, OwnedSemaphorePermit)>, + pub(crate) backpressure_on_inflight: bool, + pub(crate) semaphore_capacity: usize, +} +``` + +### Publish Backpressure + +The context uses a semaphore to limit the number of pending publish acknowledgments: + +- `max_ack_inflight(n)` — sets semaphore capacity (default 5000) +- `backpressure_on_inflight(true)` — `publish()` waits for a permit when limit is reached +- `backpressure_on_inflight(false)` — `publish()` returns `MaxAckPending` error immediately when limit is reached + +A background **acker task** monitors pending acks with a timeout (`ack_timeout`, default 30s), releasing permits when acks arrive or time out. + +### JetStream API Request Pattern + +All JetStream API calls follow the same pattern: + +1. Build a subject from the prefix: `format!("{}.STREAM.CREATE.", self.prefix)` +2. Serialize the request payload as JSON +3. Send a request via `client.send_request()` with the API subject +4. Deserialize the response as `Response` (which is `Ok(T)` or `Err(ErrorCode)`) + +## Streams + +### Stream Handle + +```rust +pub struct Stream { + context: Context, + info: I, + name: String, +} +``` + +`Stream` carries server-side info. `Stream<()>` is a lightweight handle that skips the INFO fetch. `Stream` (no generic) defaults to `Stream`. + +### Stream Config + +```rust +pub struct Config { + pub name: String, + pub description: Option, + pub subjects: Vec, + pub retention: RetentionPolicy, + pub max_consumers: i64, + pub max_messages: i64, + pub max_messages_per_subject: i64, + pub max_bytes: i64, + pub max_age: Duration, + pub max_messages_per_stream: i64, + pub max_msg_size: i32, + pub discard: DiscardPolicy, + pub discard_new_per_subject: bool, + pub storage: StorageType, + pub num_replicas: usize, + pub no_ack: bool, + pub duplicate_window: Duration, + pub placement: Option, + pub mirror: Option, + pub sources: Option>, + pub sealed: bool, + pub allow_direct: bool, + pub allow_rollup_hdrs: bool, + // server_2_10 features: + pub compression: Option, + pub first_sequence: Option, + pub subject_transform: Option, + pub republish: Option, + pub metadata: Option>, +} +``` + +### Stream Operations + +| Method | Description | +|--------|-------------| +| `create_stream(config)` | Create a new stream | +| `get_stream(name)` | Get stream handle (with INFO) | +| `get_stream_no_info(name)` | Get lightweight handle (no server round-trip) | +| `get_or_create_stream(config)` | Get existing or create new | +| `delete_stream(name)` | Delete a stream | +| `update_stream(config)` | Update stream configuration | +| `create_or_update_stream(config)` | Update or create if not found | +| `stream_names()` | `Stream` of stream names (paginated) | +| `streams()` | `Stream` of stream info (paginated) | +| `stream_by_subject(subject)` | Find stream name containing subject | + +### Stream Handle Methods + +```rust +let stream: Stream = jetstream.get_stream("events").await?; + +// Info +let info: Info = stream.info().await?; // Fresh info from server +let info: &Info = stream.cached_info(); // Cached info from last fetch + +// Message operations +stream.get_raw_message(seq).await?; // Get raw message by sequence +stream.get_last_raw_message_by_subject(subj).await?; // Get last message for subject +stream.direct_get(seq).await?; // Direct get (if allow_direct) +stream.direct_get_last_for_subject(subj).await?; // Direct last by subject +stream.delete_message(seq).await?; // Delete a specific message +stream.purge().await?; // Purge all messages +stream.purge().filter(subj).await?; // Purge messages for subject + +// Consumers +stream.create_consumer(config).await?; // Create consumer bound to stream +stream.get_consumer(name).await?; // Get existing consumer +stream.delete_consumer(name).await?; // Delete consumer +``` + +## Consumers + +### Consumer Types + +Two consumer types, each with distinct delivery models: + +1. **Pull Consumer** (`pull::Config` / `PullConsumer`) — Client explicitly requests batches of messages +2. **Push Consumer** (`push::Config` / `PushConsumer`) — Server pushes messages to a deliver subject + +### Pull Consumer + +```rust +let consumer: PullConsumer = stream + .get_or_create_consumer("my-consumer", pull::Config { + durable_name: Some("my-consumer".to_string()), + ..Default::default() + }) + .await?; +``` + +**Key methods**: +- `consumer.batch(n).await?` — Fetch up to `n` messages (one-shot batch) +- `consumer.messages().await?` — Continuous `Stream` of messages +- `consumer.sequence(n).await?` — Continuous `Stream` of batches of `n` messages +- `consumer.fetch().max(n).expires(dur).await?` — Configurable fetch + +Each message from a pull consumer is a `jetstream::Message` which has `ack()` methods. + +### Push Consumer + +Two push consumer variants: + +1. **Standard** (`push::Config`) — messages delivered to a specific subject +2. **Ordered** (`push::OrderedConfig`) — auto-recreated on failure, with flow control + +```rust +// Standard push +let consumer = stream.create_consumer(push::Config { + deliver_subject: "deliver.subject".to_string(), + durable_name: Some("push-consumer".to_string()), + ..Default::default() +}).await?; + +// Ordered push (no durable name, auto-recreates on failure) +let consumer = stream.create_consumer(push::OrderedConfig { + deliver_subject: client.new_inbox(), + filter_subject: "events.>".to_string(), + ..Default::default() +}).await?; +``` + +### Consumer Config (Shared Fields) + +```rust +pub struct Config { + // Pull fields + pub durable_name: Option, + pub name: Option, + + // Push fields + pub deliver_subject: Option, + pub deliver_group: Option, + pub deliver_policy: DeliverPolicy, + pub opt_start_time: Option, + pub opt_start_sequence: Option, + pub ack_policy: AckPolicy, + pub ack_wait: Duration, + pub max_deliver: i64, + pub backoff: Vec, + pub filter_subject: String, + pub filter_subjects: Vec, // server_2_10+ + pub replay_policy: ReplayPolicy, + pub rate_limit_bps: Option, + pub max_waiting: i64, // pull: max outstanding pull requests + pub max_ack_pending: i64, + pub flow_control: bool, + pub idle_heartbeat: Duration, + pub headers_only: bool, + pub num_replicas: usize, + pub mem_storage: bool, + pub description: Option, + pub metadata: Option>, + pub inactive_threshold: Option, // for ephemeral consumers +} +``` + +### Deliver Policy + +```rust +pub enum DeliverPolicy { + All, // Deliver all messages + Last, // Deliver last message only + New, // Deliver only new messages + ByStartSequence { start_sequence: u64 }, + ByStartTime { start_time: DateTime }, + LastPerSubject, // Deliver last message per subject +} +``` + +### Ack Policy + +```rust +pub enum AckPolicy { + None, // No acknowledgment needed + All, // Ack all messages up to this one + Explicit, // Ack each message individually +} +``` + +## JetStream Messages + +### `jetstream::Message` + +Wraps a core `Message` with JetStream-specific metadata: + +```rust +pub struct Message { + pub message: crate::Message, // The underlying NATS message + pub ack_subject: Subject, // Subject for sending acks + pub stream: String, // Stream name + pub consumer: String, // Consumer name + pub stream_sequence: u64, // Sequence in stream + pub consumer_sequence: u64, // Sequence for this consumer + pub delivered: u64, // Delivery count + pub pending: u64, // Pending message count + pub published: DateTime, // Original publish time +} +``` + +### Ack Methods + +```rust +// In-memory ack (non-persistent, fast) +message.ack().await?; + +// Ack with specific type +message.ack_with(AckKind::Nak).await?; +message.ack_with(AckKind::Progress).await?; +message.ack_with(AckKind::Term).await?; +message.ack_with(AckKind::NakWithDelay(duration)).await?; +message.ack_with(AckKind::TermWithReason("reason")).await?; +``` + +### `AckKind` + +```rust +pub enum AckKind { + Ack, // +ACK — message processed + Nak, // -NAK — re-deliver + Progress, // PRI — still working + Term, // +TERM — don't redeliver + NakWithDelay(Duration), // -NAK with re-delivery delay + TermWithReason(String), // +TERM with reason +} +``` + +## JetStream Publish + +### `Context::publish()` + +JetStream publish returns a `PublishAckFuture` — a future that resolves to a `PublishAck`: + +```rust +let ack_future = jetstream.publish("events", "data".into()).await?; +let ack: PublishAck = ack_future.await?; // Wait for server acknowledgment +``` + +### `PublishAck` + +```rust +pub struct PublishAck { + pub stream: String, + pub sequence: u64, + pub domain: String, + pub duplicate: bool, +} +``` + +### `PublishMessage` Builder + +```rust +let ack = jetstream.send_publish( + "events", + PublishMessage::build() + .payload("data".into()) + .message_id("uuid-123") // Deduplication ID + .expected_stream("events") // Fail if wrong stream + .expected_last_msg_id("prev-id") + .expected_last_sequence(42) + .headers(headers), +).await?; +``` + +## Pagination + +Stream and consumer listing uses pagination internally: + +```rust +pub struct StreamNames { + context: Context, + offset: usize, + page_request: Option, + streams: Vec, + subject: Option, + done: bool, +} +``` + +Implements `futures_util::Stream>`, lazily fetching pages as needed. + +## Error Handling + +JetStream errors follow the `Response` pattern: + +```rust +pub enum Response { + Ok(T), + Err { error: ErrorCode }, +} +``` + +`ErrorCode` carries the server's error code and description. Most JetStream-specific errors map to typed error enums (e.g., `CreateStreamError`, `ConsumerError`, etc.). \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/06-key-value-store.md b/docs/research/references/nats.rs/nats-async/06-key-value-store.md new file mode 100644 index 0000000..bf6c230 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/06-key-value-store.md @@ -0,0 +1,237 @@ +# async-nats: Key-Value Store + +## Overview + +The Key-Value (KV) store is an abstraction built on top of JetStream streams. Each KV bucket is backed by a JetStream stream with the naming convention `KV_`. Keys are mapped to subjects under the `$KV..` prefix. + +The KV feature requires `kv` (which implies `jetstream`). + +## Store Handle + +```rust +#[derive(Debug, Clone)] +pub struct Store { + pub name: String, + pub stream_name: String, + pub prefix: String, // $KV.. + pub put_prefix: Option, // For mirrored buckets + pub use_jetstream_prefix: bool, // Whether to prepend JS API prefix + pub stream: Stream, +} +``` + +## Bucket Config + +```rust +#[derive(Debug, Clone, Default)] +pub struct Config { + pub bucket: String, + pub description: String, + pub max_value_size: i32, + pub history: i64, // Max historical entries per key (1-64) + pub max_age: Duration, // Max age of any entry + pub max_bytes: i64, // Total bucket size limit + pub storage: StorageType, // File or Memory + pub num_replicas: usize, + pub republish: Option, + pub mirror: Option, // Mirror another bucket + pub sources: Option>, + pub mirror_direct: bool, + pub compression: bool, // server_2_10+ + pub placement: Option, + pub limit_markers: Option, // server_2_11+ +} +``` + +## Creating/Accessing Buckets + +```rust +// Create a new bucket +let kv = jetstream.create_key_value(kv::Config { + bucket: "my-bucket".to_string(), + history: 10, + max_age: Duration::from_secs(3600), + ..Default::default() +}).await?; + +// Get an existing bucket +let kv = jetstream.get_key_value("my-bucket").await?; + +// Create or update +let kv = jetstream.create_or_update_key_value(kv::Config { ... }).await?; + +// Delete a bucket +jetstream.delete_key_value("my-bucket").await?; +``` + +## KV Operations + +### Put + +```rust +let revision: u64 = kv.put("key", "value".into()).await?; +``` + +Publishes to `$KV..` (or with JS prefix). The JetStream stream stores it, and the returned sequence number serves as the revision. + +### Get + +```rust +let value: Option = kv.get("key").await?; +``` + +Returns `None` if the key doesn't exist or was deleted/purged. Uses either direct get (if `allow_direct`) or the standard message API. + +### Entry + +```rust +let entry: Option = kv.entry("key").await?; +let entry: Option = kv.entry_for_revision("key", 2).await?; +``` + +Returns full entry metadata: + +```rust +pub struct Entry { + pub bucket: String, + pub key: String, + pub value: Bytes, + pub revision: u64, + pub created: DateTime, + pub delta: u64, + pub operation: Operation, + pub seen_current: bool, +} +``` + +### Create (Put if not exists) + +```rust +let revision: u64 = kv.create("key", "value".into()).await?; +``` + +Uses `update` with `expected_last_subject_sequence = 0` (create-only). If the key exists and is deleted/purged, it's re-created. + +### Update (Conditional Put) + +```rust +let revision: u64 = kv.update("key", "value".into(), last_revision).await?; +``` + +Uses the `Nats-Expected-Last-Subject-Sequence` header for optimistic concurrency control. Only succeeds if the key's current revision matches. + +### Delete + +```rust +kv.delete("key").await?; +kv.delete_expect_revision("key", Some(revision)).await?; +``` + +Non-destructive — publishes a `DEL` marker message. The key appears deleted to `get()`, but history is preserved (up to `history` limit). + +### Purge + +```rust +kv.purge("key").await?; +kv.purge_with_ttl("key", Duration::from_secs(10)).await?; +kv.purge_expect_revision("key", Some(revision)).await?; +``` + +Destructive — publishes a `PURGE` marker with rollup header, removing all previous revisions of the key. Leaves a single purge entry. + +### Watch + +```rust +// Watch for new changes +let mut watch = kv.watch("key").await?; +// Watch with initial value +let mut watch = kv.watch_with_history("key").await?; +// Watch from specific revision +let mut watch = kv.watch_from_revision("key", 5).await?; +// Watch all keys +let mut watch = kv.watch_all().await?; +// Watch multiple keys (server_2_10+) +let mut watch = kv.watch_many(["foo", "bar"]).await?; +``` + +`Watch` implements `futures_util::Stream>`. + +Under the hood, each watch creates an **ordered push consumer** on the KV stream with: +- `filter_subject` matching `$KV..` +- `replay_policy: Instant` +- Appropriate `deliver_policy` + +### History + +```rust +let mut history = kv.history("key").await?; +``` + +Returns a `Stream` of all past `Entry` values for a key (including deletes/purges). + +### Keys + +```rust +let mut keys = kv.keys().await?; +``` + +Returns a `Stream` of all current keys. Uses a headers-only consumer with `LastPerSubject` deliver policy to efficiently scan the bucket. + +## Entry Operations + +```rust +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum Operation { + Put, // Value was put + Delete, // Value was deleted (DEL marker) + Purge, // Value was purged (PURGE marker with rollup) +} +``` + +The operation type is determined from the `KV-Operation` header (`PUT`, `DEL`, `PURGE`) or the `Nats-Marker-Reason` header (fallback for server-generated markers like `MaxAge`, `Purge`, `Remove`). + +## Key and Bucket Name Validation + +```rust +// Bucket: alphanumeric, dash, underscore only +VALID_BUCKET_RE: \A[a-zA-Z0-9_-]+\z + +// Key: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots +VALID_KEY_RE: \A[-/_=\.a-zA-Z0-9]+\z +``` + +## Bucket Status + +```rust +let status: Status = kv.status().await?; +``` + +Wraps stream info to provide bucket-level statistics (bucket name, message count, byte count, etc.). + +## Mirrored Buckets + +When a bucket is configured as a mirror of another (potentially in a different account/domain): + +- `prefix` is set to `$KV..` +- `put_prefix` may be set to the source bucket's API prefix for cross-domain writes +- `use_jetstream_prefix` is adjusted based on whether the mirror is in the same domain + +## KV → Stream Config Mapping + +When creating a KV bucket, the `Config` is converted to a JetStream `stream::Config`: + +| KV Config | Stream Config | +|-----------|---------------| +| `bucket` | `name = "KV_"` | +| `subjects` | `["$KV..>"]` | +| `max_messages_per_subject` | `history` (max 64) | +| `max_age` | `max_age` | +| `max_bytes` | `max_bytes` | +| `storage` | `storage` | +| `num_replicas` | `num_replicas` | +| `republish` | `republish` | +| `mirror` | `mirror` | +| `discard` | `DiscardPolicy::New` | +| `allow_direct` | `true` | +| `allow_rollup_hdrs` | `true | +| `max_msg_size` | `max_value_size` | \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/07-object-store.md b/docs/research/references/nats.rs/nats-async/07-object-store.md new file mode 100644 index 0000000..8845a56 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/07-object-store.md @@ -0,0 +1,245 @@ +# async-nats: Object Store + +## Overview + +The Object Store provides large-object storage built on JetStream. Objects are chunked and stored as messages in a JetStream stream, with metadata stored separately. The stream is named `OBJ_`. + +The object-store feature requires `object-store` (which implies `jetstream` + `crypto`). + +## ObjectStore Handle + +```rust +#[derive(Clone)] +pub struct ObjectStore { + pub(crate) name: String, + pub(crate) stream: Stream, +} +``` + +## Object Store Config + +```rust +#[derive(Debug, Default, Clone, Serialize, Deserialize)] +pub struct Config { + pub bucket: String, + pub description: Option, + pub max_age: Duration, + pub max_bytes: i64, + pub storage: StorageType, + pub num_replicas: usize, + pub compression: bool, + pub placement: Option, +} +``` + +## Creating/Accessing Object Stores + +```rust +// Create +let bucket = jetstream.create_object_store(object_store::Config { + bucket: "my-bucket".to_string(), + ..Default::default() +}).await?; + +// Get existing +let bucket = jetstream.get_object_store("my-bucket").await?; + +// Delete +jetstream.delete_object_store("my-bucket").await?; +``` + +## Object Store Operations + +### Put + +```rust +let info: ObjectInfo = bucket.put("file.txt", &mut async_read).await?; +``` + +The put operation: +1. Reads data from any `AsyncRead + Unpin` source in chunks (default 128KB) +2. Each chunk is published to `$O..C.` (chunk subject) +3. SHA-256 digest is computed incrementally +4. After all chunks, metadata is published to `$O..M.` with a rollup header +5. If the object previously existed, old chunks are purged + +### Get + +```rust +let mut object: Object = bucket.get("file.txt").await?; +``` + +Returns an `Object` that implements `tokio::io::AsyncRead`: + +```rust +let mut bytes = Vec::new(); +object.read_to_end(&mut bytes).await?; +``` + +On read, the Object: +1. Creates an ordered push consumer on `$O..C.` +2. Streams chunk messages, feeding bytes to the reader +3. Verifies SHA-256 digest after the last chunk +4. If digest doesn't match, returns `io::ErrorKind::InvalidData` + +### Delete + +```rust +bucket.delete("file.txt").await?; +``` + +Marks the object as deleted in metadata (sets `deleted = true`, `chunks = 0`, `size = 0`) with a rollup, then purges all chunk messages. + +### Info + +```rust +let info: ObjectInfo = bucket.info("file.txt").await?; +``` + +Fetches the last metadata message for the object (from `$O..M.`). + +### Watch + +```rust +let mut watcher = bucket.watch().await?; +let mut watcher = bucket.watch_with_history().await?; +``` + +Returns a `Stream>`. Uses an ordered push consumer on `$O..M.>`. + +### List + +```rust +let mut list = bucket.list().await?; +``` + +Returns a `Stream>`. Lists all non-deleted objects. Uses `DeliverPolicy::All` to replay all metadata. + +### Seal + +```rust +bucket.seal().await?; +``` + +Sets the underlying stream's `sealed = true`, preventing any further modifications. + +### Links + +```rust +// Link to another object (same or different bucket) +let info = bucket.add_link("link_name", &object).await?; + +// Link to another bucket +let info = bucket.add_bucket_link("link_name", "other_bucket").await?; +``` + +Links are followed automatically when `get()` is called (one level deep). Cannot link to a deleted object or create a link to a link. + +### Update Metadata + +```rust +bucket.update_metadata("object", object_store::UpdateMetadata { + name: "new_name".to_string(), + description: Some("updated description".to_string()), + ..Default::default() +}).await?; +``` + +If the name changes, old metadata is purged and new metadata is published. + +## Object Types + +### ObjectInfo + +```rust +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] +pub struct ObjectInfo { + pub name: String, + pub description: Option, + pub metadata: HashMap, + pub headers: Option, + pub options: Option, + pub bucket: String, + pub nuid: String, + pub size: usize, + pub chunks: usize, + pub modified: Option, + pub digest: Option, // Format: "SHA-256=" + pub deleted: bool, +} +``` + +### ObjectMetadata + +```rust +#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)] +pub struct ObjectMetadata { + pub name: String, + pub description: Option, + pub chunk_size: Option, + pub metadata: HashMap, + pub headers: Option, +} +``` + +### ObjectLink + +```rust +#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)] +pub struct ObjectLink { + pub name: Option, // None = bucket link, Some = object link + pub bucket: String, +} +``` + +### Object + +```rust +pub struct Object { + pub info: ObjectInfo, + remaining_bytes: VecDeque, + has_pending_messages: bool, + digest: Option, + subscription: Option, + subscription_future: Option>>, + stream: Stream, +} +``` + +Implements `tokio::io::AsyncRead`. Lazy-creates the consumer on first read. + +## Subject Naming Convention + +| Purpose | Subject Pattern | +|---------|----------------| +| Chunks | `$O..C.` | +| Metadata | `$O..M.` | + +Object names are base64url-encoded in metadata subjects to allow arbitrary characters (the raw name might contain characters invalid in NATS subjects). + +## Validation + +```rust +// Bucket: alphanumeric, dash, underscore only +BUCKET_NAME_RE: \A[a-zA-Z0-9_-]+\z + +// Object name: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots +OBJECT_NAME_RE: \A[-/_=\.a-zA-Z0-9]+\z +``` + +## Data Integrity + +The object store uses SHA-256 hashing (from the `crypto` module) to verify data integrity: + +1. On `put()`: SHA-256 is computed incrementally as chunks are read. The digest is stored in `ObjectInfo.digest` as `"SHA-256="`. +2. On `get()` (via `AsyncRead`): SHA-256 is verified after the last chunk is read. If the computed digest doesn't match the stored digest, `io::ErrorKind::InvalidData` is returned. + +```rust +// crypto module +pub(crate) struct Sha256 { ... } +impl Sha256 { + pub fn new() -> Self; + pub fn update(&mut self, data: &[u8]); + pub fn finish(self) -> [u8; 32]; +} +``` \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/08-service-api.md b/docs/research/references/nats.rs/nats-async/08-service-api.md new file mode 100644 index 0000000..2debb28 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/08-service-api.md @@ -0,0 +1,272 @@ +# async-nats: Service API + +## Overview + +The Service API provides a microservice request/reply pattern with built-in service discovery, health checking, and statistics. It follows the [NATS Micro v1 specification](https://github.com/nats-io/nats-architecture-design/blob/main/adr/ADR-33.md). + +The `service` feature is required. + +## Service + +```rust +#[derive(Debug)] +pub struct Service { + endpoints_state: Arc>, + info: Info, + client: Client, + handle: JoinHandle>, + shutdown_tx: Sender<()>, + subjects: Arc>>, + queue_group: String, +} +``` + +## Creating a Service + +Via the `ServiceExt` trait on `Client`: + +```rust +use async_nats::service::ServiceExt; + +// Builder pattern +let mut service = client + .service_builder() + .description("product service") + .stats_handler(|endpoint, stats| serde_json::json!({ "endpoint": endpoint })) + .metadata(HashMap::from([("version".into(), "v2".into())])) + .queue_group("products-group") + .start("products", "1.0.0") + .await?; + +// Direct config +let mut service = client + .add_service(service::Config { + name: "products".to_string(), + version: "1.0.0".to_string(), + description: Some("product service".to_string()), + stats_handler: None, + metadata: None, + queue_group: None, + }) + .await?; +``` + +Service name must match `^[A-Za-z0-9\-_]+$`. Version must be valid SemVer. + +## Service Verbs + +Every service automatically subscribes to three verb subjects for discovery and monitoring: + +| Verb | Subject Pattern | Purpose | +|------|----------------|---------| +| PING | `$SRV.PING`, `$SRV.PING.`, `$SRV.PING..` | Lightweight health check | +| INFO | `$SRV.INFO.`, `$SRV.INFO..` | Service metadata | +| STATS | `$SRV.STATS.`, `$SRV.STATS..` | Service + endpoint statistics | + +A background task handles these verb requests and responds with JSON payloads. + +## Service Config + +```rust +#[derive(Serialize, Deserialize, Debug)] +pub struct Config { + pub name: String, + pub description: Option, + pub version: String, + pub stats_handler: Option, + pub metadata: Option>, + pub queue_group: Option, +} +``` + +## Adding Endpoints + +```rust +// Simple endpoint +let mut endpoint = service.endpoint("get-products").await?; + +// Endpoint with custom name and metadata +let endpoint = service + .endpoint_builder() + .name("api") + .metadata(HashMap::from([("auth".into(), "required".into())])) + .queue_group("custom-group") + .add("products") + .await?; + +// Grouped endpoints +let v1 = service.group("v1"); +let products = v1.endpoint("products").await?; +let orders = v1.endpoint("orders").await?; + +// Nested groups +let v1_api = service.group("api").group("v1"); +``` + +## Endpoint + +```rust +pub struct Endpoint { + requests: Subscriber, + stats: Arc>, + client: Client, + endpoint: String, + shutdown: Option, + shutdown_future: Option, +} +``` + +Implements `futures_util::Stream`. + +```rust +while let Some(request) = endpoint.next().await { + request.respond(Ok("response data".into())).await?; +} +``` + +## Service Request + +```rust +#[derive(Debug)] +pub struct Request { + issued: Instant, + client: Client, + pub message: Message, + endpoint: String, + stats: Arc>, +} +``` + +### Responding + +```rust +// Success +request.respond(Ok("result".into())).await?; + +// Success with headers +request.respond_with_headers(Ok("result".into()), headers).await?; + +// Error +request.respond(Err(service::error::Error { + code: 500, + status: "internal error".to_string(), +})).await?; +``` + +Error responses always include `Nats-Service-Error` and `Nats-Service-Error-Code` headers. If user-supplied headers contain these headers, they are overridden by the error values. + +### Stats Tracking + +Each response updates endpoint statistics: +- `requests` — total requests +- `processing_time` — cumulative processing time +- `average_processing_time` — average per request +- `errors` — error count +- `last_error` — last error details + +## Service Info Types + +### PingResponse + +```rust +pub struct PingResponse { + pub kind: String, // "io.nats.micro.v1.ping_response" + pub name: String, + pub id: String, + pub version: String, + pub metadata: HashMap, +} +``` + +### Info + +```rust +pub struct Info { + pub kind: String, // "io.nats.micro.v1.info_response" + pub name: String, + pub id: String, + pub description: String, + pub version: String, + pub metadata: HashMap, + pub endpoints: Vec, +} +``` + +### Stats + +```rust +pub struct Stats { + pub kind: String, // "io.nats.micro.v1.stats_response" + pub name: String, + pub id: String, + pub version: String, + pub started: DateTime, + pub endpoints: Vec, +} +``` + +### Endpoint Stats + +```rust +pub struct endpoint::Stats { + pub name: String, + pub subject: String, + pub queue_group: String, + pub data: Option, // Custom data from stats_handler + pub errors: u64, + pub processing_time: Duration, + pub average_processing_time: Duration, + pub requests: u64, + pub last_error: Option, +} +``` + +## Service Groups + +Groups provide subject prefixing for endpoint organization: + +```rust +let service = client.service_builder().start("api", "1.0.0").await?; + +// Endpoints subscribe to "products" and "orders" +let products = service.endpoint("products").await?; +let orders = service.endpoint("orders").await?; + +// Grouped: subscribe to "v1.products" and "v1.orders" +let v1 = service.group("v1"); +let products = v1.endpoint("products").await?; +let orders = v1.endpoint("orders").await?; + +// Nested: subscribe to "api.v1.products" +let api_v1 = service.group("api").group("v1"); +let products = api_v1.endpoint("products").await?; +``` + +Each group can have its own queue group: + +```rust +let v1 = service.group_with_queue_group("v1", "v1-workers"); +``` + +## Stopping a Service + +```rust +service.stop().await?; +``` + +Sends a shutdown signal and aborts the verb-handling task. Other service instances with the same name continue running. + +## Resetting Stats + +```rust +service.reset().await?; +``` + +Resets all endpoint statistics (errors, processing time, requests, average processing time) to zero. + +## Querying Service State + +```rust +let stats: HashMap = service.stats().await?; +let info: Info = service.info().await?; +``` \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/09-quick-reference.md b/docs/research/references/nats.rs/nats-async/09-quick-reference.md new file mode 100644 index 0000000..adb85b4 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/09-quick-reference.md @@ -0,0 +1,312 @@ +# async-nats: Quick Reference + +## Connection + +```rust +// Basic connect +let client = async_nats::connect("demo.nats.io").await?; + +// With options +let client = async_nats::ConnectOptions::new() + .require_tls(true) + .name("my-service") + .ping_interval(Duration::from_secs(10)) + .request_timeout(Some(Duration::from_secs(5))) + .connect("demo.nats.io") + .await?; + +// Multiple servers +let client = async_nats::connect(vec![ + "nats://server1:4222".parse()?, + "nats://server2:4222".parse()?, +]).await?; + +// Background connect +let client = async_nats::ConnectOptions::new() + .retry_on_initial_connect() + .connect("demo.nats.io") + .await?; +``` + +## Core NATS: Publish + +```rust +// Simple publish +client.publish("subject", "payload".into()).await?; + +// With reply-to +client.publish_with_reply("subject", "reply-to", "payload".into()).await?; + +// With headers +let mut headers = HeaderMap::new(); +headers.insert("X-Custom", "value"); +client.publish_with_headers("subject", headers, "payload".into()).await?; + +// Full control +client.publish_with_reply_and_headers("subject", "reply-to", headers, "payload".into()).await?; + +// Flush (ensure all published messages are sent) +client.flush().await?; +``` + +## Core NATS: Subscribe + +```rust +use futures_util::StreamExt; + +// Basic subscribe +let mut subscriber = client.subscribe("subject").await?; + +// Queue group +let mut subscriber = client.queue_subscribe("subject", "group".into()).await?; + +// Receive messages (Subscriber implements Stream) +while let Some(message) = subscriber.next().await { + println!("subject: {}, payload: {:?}", message.subject, message.payload); +} + +// Unsubscribe +subscriber.unsubscribe().await?; + +// Unsubscribe after N messages +subscriber.unsubscribe_after(10).await?; + +// Drain (wait for in-flight, then unsubscribe) +subscriber.drain().await?; +``` + +## Core NATS: Request/Reply + +```rust +// Simple request (uses default timeout) +let response = client.request("subject", "data".into()).await?; + +// With custom timeout and headers +let request = async_nats::Request::new() + .payload("data".into()) + .timeout(Some(Duration::from_secs(5))) + .headers(headers); +let response = client.send_request("subject", request).await?; + +// Custom inbox (bypasses multiplexer) +let request = async_nats::Request::new() + .payload("data".into()) + .inbox("custom-inbox".into()); +let response = client.send_request("subject", request).await?; +``` + +## Message Structure + +```rust +pub struct Message { + pub subject: Subject, + pub reply: Option, + pub payload: Bytes, + pub headers: Option, + pub status: Option, + pub description: Option, + pub length: usize, +} +``` + +## JetStream + +```rust +let jetstream = async_nats::jetstream::new(client); + +// Publish (returns ack future) +let ack = jetstream.publish("events", "data".into()).await?; +let publish_ack = ack.await?; + +// Stream management +let stream = jetstream.create_stream(stream::Config { + name: "events".to_string(), + subjects: vec!["events.>".to_string()], + max_messages: 10_000, + ..Default::default() +}).await?; + +let stream = jetstream.get_stream("events").await?; +let stream = jetstream.get_or_create_stream(config).await?; +jetstream.delete_stream("events").await?; +jetstream.update_stream(config).await?; + +// Consumer management +let consumer: PullConsumer = stream.create_consumer(pull::Config { + durable_name: Some("my-consumer".to_string()), + ..Default::default() +}).await?; + +// Pull consumer: fetch messages +let mut messages = consumer.messages().await?; +while let Some(message) = messages.next().await { + let message = message?; + message.ack().await?; +} + +// Push consumer (ordered) +let consumer = stream.create_consumer(push::OrderedConfig { + deliver_subject: client.new_inbox(), + filter_subject: "events.>".to_string(), + ..Default::default() +}).await?; +let mut messages = consumer.messages().await?; +``` + +## Key-Value Store + +```rust +let kv = jetstream.create_key_value(kv::Config { + bucket: "my-bucket".to_string(), + history: 10, + ..Default::default() +}).await?; + +// CRUD +let revision = kv.put("key", "value".into()).await?; +let revision = kv.create("key", "value".into()).await?; +let value: Option = kv.get("key").await?; +let entry: Option = kv.entry("key").await?; +let revision = kv.update("key", "new-value".into(), revision).await?; +kv.delete("key").await?; +kv.purge("key").await?; + +// Watch +let mut watch = kv.watch("key").await?; +let mut watch_all = kv.watch_all().await?; + +// History & Keys +let mut history = kv.history("key").await?; +let mut keys = kv.keys().await?; +``` + +## Object Store + +```rust +let bucket = jetstream.create_object_store(object_store::Config { + bucket: "files".to_string(), + ..Default::default() +}).await?; + +// Put (from any AsyncRead) +let info = bucket.put("file.txt", &mut file).await?; + +// Get (returns AsyncRead) +let mut object = bucket.get("file.txt").await?; +let mut bytes = Vec::new(); +object.read_to_end(&mut bytes).await?; + +// Info, delete, list, watch +let info = bucket.info("file.txt").await?; +bucket.delete("file.txt").await?; +let mut list = bucket.list().await?; +let mut watch = bucket.watch().await?; +``` + +## Service API + +```rust +use async_nats::service::ServiceExt; +use futures_util::StreamExt; + +let mut service = client + .service_builder() + .description("product service") + .start("products", "1.0.0") + .await?; + +let mut endpoint = service.endpoint("get").await?; + +while let Some(request) = endpoint.next().await { + request.respond(Ok("result".into())).await?; +} +``` + +## Client State & Events + +```rust +// Check connection state +match client.connection_state() { + State::Connected => {}, + State::Disconnected => {}, + State::Pending => {}, +} + +// Get server info +let info: ServerInfo = client.server_info(); +println!("max_payload: {}", info.max_payload); +println!("jetstream: {}", info.jetstream); + +// Get statistics +let stats = client.statistics(); +println!("in_messages: {}", stats.in_messages.load(Ordering::Relaxed)); + +// Force reconnect +client.force_reconnect().await?; + +// Server pool management +client.set_server_pool(["nats://s1:4222".parse()?, "nats://s2:4222".parse()?].as_slice()).await?; +let pool = client.server_pool().await?; + +// Drain +client.drain().await?; +``` + +## Error Handling Patterns + +```rust +// Connect errors +match async_nats::connect("server").await { + Err(e) => match e.kind() { + ConnectErrorKind::TimedOut => {}, + ConnectErrorKind::Authentication => {}, + ConnectErrorKind::AuthorizationViolation => {}, + _ => {}, + }, + Ok(client) => {}, +} + +// Publish errors +match client.publish("subject", "data".into()).await { + Err(e) => match e.kind() { + PublishErrorKind::MaxPayloadExceeded => {}, + PublishErrorKind::InvalidSubject => {}, + PublishErrorKind::Send => {}, + _ => {}, + }, + _ => {}, +} + +// Request errors +match client.request("subject", "data".into()).await { + Err(e) => match e.kind() { + RequestErrorKind::TimedOut => {}, + RequestErrorKind::NoResponders => {}, + RequestErrorKind::InvalidSubject => {}, + RequestErrorKind::MaxPayloadExceeded => {}, + _ => {}, + }, + Ok(message) => {}, +} +``` + +## Feature Flag Quick Reference + +| Feature | Enables | Default | +|---------|---------|---------| +| `jetstream` | JetStream streams, consumers, publish | ✅ | +| `kv` | Key-Value store (implies `jetstream`) | ✅ | +| `object-store` | Object store (implies `jetstream` + `crypto`) | ✅ | +| `service` | Service API | ✅ | +| `nkeys` | NKey/JWT authentication | ✅ | +| `nuid` | NUID-based ID generation | ✅ | +| `crypto` | SHA-256 (for object store) | ✅ | +| `websockets` | WebSocket transport | ✅ | +| `ring` | `ring` TLS crypto backend | ✅ | +| `aws-lc-rs` | `aws-lc-rs` TLS crypto backend | ❌ | +| `fips` | FIPS mode via `aws-lc-rs` | ❌ | +| `chrono` | `chrono` datetime instead of `time` | ❌ | +| `server_2_10` | Server 2.10+ features | ✅ | +| `server_2_11` | Server 2.11+ features | ✅ | +| `server_2_12` | Server 2.12+ features | ✅ | +| `server_2_14` | Server 2.14+ features | ✅ | \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-async/README.md b/docs/research/references/nats.rs/nats-async/README.md new file mode 100644 index 0000000..2f5d619 --- /dev/null +++ b/docs/research/references/nats.rs/nats-async/README.md @@ -0,0 +1,23 @@ +# async-nats Reference Documentation + +**Crate**: `async-nats` v0.49.1 +**Source**: https://github.com/nats-io/nats.rs (`async-nats/` directory) +**License**: Apache-2.0 + +## Contents + +| # | File | Topic | +|---|------|-------| +| 01 | [Overview & Architecture](01-overview-and-architecture.md) | Crate overview, feature flags, source structure, core connection model, dependency graph | +| 02 | [Key Types & Traits](02-key-types-and-traits.md) | `Client`, `Subscriber`, `Message`, `Request`, `ServerInfo`, `ConnectInfo`, `Statistics`, subject/header types, event/state types, error types, trait definitions | +| 03 | [Protocol & Wire Format](03-protocol-and-wire-format.md) | NATS wire protocol (PUB/HPUB/SUB/UNSUB/PING/PONG, MSG/HMSG/INFO/ERR), parser/serializer internals, vectored I/O, WebSocket transport, connection lifecycle, reconnection | +| 04 | [Connection Management](04-connection-management.md) | `ConnectOptions` builder, authentication methods, TLS configuration, reconnection callbacks, event callbacks, `ConnectionHandler` internals, multiplexer, server pool management | +| 05 | [JetStream](05-jetstream.md) | `Context` and `ContextBuilder`, streams, consumers (pull/push/ordered), JetStream messages and acks, publish with ack futures, pagination | +| 06 | [Key-Value Store](06-key-value-store.md) | KV `Store` handle, bucket CRUD, put/get/create/update/delete/purge, watch/history/keys, entry operations, mirrored buckets, KV-to-stream mapping | +| 07 | [Object Store](07-object-store.md) | `ObjectStore` handle, put/get/delete/watch/list/seal, links, `Object` (AsyncRead), chunking, SHA-256 integrity, subject naming | +| 08 | [Service API](08-service-api.md) | `Service` and `ServiceBuilder`, endpoints, groups, verb subscriptions (PING/INFO/STATS), request/respond with stats tracking | +| 09 | [Quick Reference](09-quick-reference.md) | Code examples for all major operations, feature flag reference | + +## How This Documentation Was Produced + +All information was derived by reading the source code of the `async-nats` crate at version 0.49.1 from the `nats.rs` repository. No external documentation was consulted — this is a ground-up reference based purely on the source. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/01-overview-and-architecture.md b/docs/research/references/nats.rs/nats-server/01-overview-and-architecture.md new file mode 100644 index 0000000..e6c7d0d --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/01-overview-and-architecture.md @@ -0,0 +1,200 @@ +# nats.rs: Overview and Architecture + +**Version**: async-nats 0.49.1, nats-server 0.1.0 +**Repository**: https://github.com/nats-io/nats.rs +**License**: Apache-2.0 +**Rust Edition**: 2021 +**MSRV**: 1.88.0 +**Protocol**: NATS Client Protocol (INFO/CONNECT/PUB/SUB/UNSUB/PING/PONG) + +## What It Is + +The `nats.rs` repository contains the **official Rust client for NATS.io**, a high-performance messaging system. The active crate is **`async-nats`** — a fully async, Tokio-based NATS client. The deprecated `nats` crate (synchronous) receives security fixes only. + +The `nats-server` crate is **not** an implementation of the NATS server. It is a **test harness** that spawns the Go-based `nats-server` binary for integration tests. The actual NATS server is a separate Go project at `github.com/nats-io/nats-server`. + +Core design decisions: +- **Fully async** — all I/O is Tokio-based with async/await throughout +- **Cloneable Client handle** — `Client` is cheap to clone (Arc internals), all protocol work happens in a single `ConnectionHandler` task +- **Channel-based internal communication** — `Client` sends `Command` variants via `mpsc` channel to `ConnectionHandler` +- **Multiplexed request-reply** — one internal subscription handles all request-response patterns via inbox token routing +- **Automatic reconnection** — exponential backoff with configurable server pool rotation +- **Feature-gated subsystems** — JetStream, KV, Object Store, Service API, NKeys, WebSockets, and crypto backends are all optional + +## Workspace Structure + +``` +nats.rs/ +├── async-nats/ # Primary crate — async NATS client +│ ├── src/ +│ │ ├── lib.rs # Entry point: connect(), ServerOp, ClientOp, Command, ConnectionHandler, Subscriber +│ │ ├── client.rs # Client handle: publish, subscribe, request, flush, drain +│ │ ├── connection.rs # Low-level I/O: protocol parsing, read/write buffers +│ │ ├── connector.rs # Connection establishment, reconnection, server pool +│ │ ├── options.rs # ConnectOptions builder +│ │ ├── auth.rs # Auth struct (credentials container) +│ │ ├── auth_utils.rs # Credential file parsing (.creds files) +│ │ ├── error.rs # Generic Error type +│ │ ├── header.rs # HeaderMap — NATS message headers +│ │ ├── subject.rs # Subject type, ToSubject trait +│ │ ├── status.rs # StatusCode (100-999 NATS protocol codes) +│ │ ├── message.rs # Message and OutboundMessage types +│ │ ├── tls.rs # TLS configuration helpers +│ │ ├── crypto.rs # Crypto feature support +│ │ ├── id_generator.rs # NUID/rand-based unique ID generation +│ │ ├── datetime.rs # DateTime helpers for JetStream/Service +│ │ ├── jetstream/ # JetStream API (feature-gated) +│ │ │ ├── mod.rs # Module root, jetstream::new(), with_domain() +│ │ │ ├── context.rs # JetStream Context — streams, publishing, consumers +│ │ │ ├── stream.rs # Stream management, Config, Info, Consumer creation +│ │ │ ├── consumer/ # Pull, Push, Ordered consumers +│ │ │ ├── message.rs # JetStream Message with ack methods +│ │ │ ├── publish.rs # PublishAck +│ │ │ ├── response.rs # Response wrapper +│ │ │ ├── errors.rs # JetStream error codes +│ │ │ ├── account.rs # Account info +│ │ │ ├── kv/ # Key-Value store (feature: "kv") +│ │ │ └── object_store/ # Object store (feature: "object-store") +│ │ └── service/ # Service API (feature-gated) +│ │ ├── mod.rs # Service, ServiceBuilder +│ │ ├── endpoint.rs # Endpoint handling +│ │ └── error.rs # Service errors +│ ├── tests/ # Integration tests (require nats-server binary) +│ ├── examples/ # Runnable examples +│ └── benches/ # Criterion benchmarks +├── nats-server/ # Test harness — spawns Go nats-server for tests +│ ├── src/lib.rs # Server struct, run_server(), run_cluster() +│ └── configs/ # Server config files for tests +│ └── jetstream.conf +└── nats/ # DEPRECATED sync client — do not modify +``` + +## Architecture Diagram + +``` +┌──────────────────────────────────────────────────────────┐ +│ Application Layer │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ JetStream│ │ KV │ │ Object │ │ Service │ │ +│ │ Context │ │ Store │ │ Store │ │ API │ │ +│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ +│ └──────────────┴─────────────┴─────────────┘ │ +│ │ │ +│ ┌──────┴──────┐ │ +│ │ Client │ Cloneable handle │ +│ │ (mpsc::Sender) │ +│ └──────┬──────┘ │ +│ │ Command channel │ +└──────────────────────────┼────────────────────────────────┘ + │ +┌──────────────────────────┼────────────────────────────────┐ +│ ConnectionHandler │ +│ (single Tokio task) │ +│ │ │ +│ ┌───────────┐ ┌───────┴───────┐ ┌──────────────────┐ │ +│ │Subscriptions│ │ Multiplexer │ │ Flush Observers │ │ +│ │ HashMap │ │ (request-reply)│ │ │ │ +│ └──────┬──────┘ └───────┬───────┘ └──────────────────┘ │ +│ └────────────────┼ │ +│ ┌──────┴──────┐ │ +│ │ Connector │ Server pool, reconnect │ +│ └──────┬──────┘ │ +│ │ │ +│ ┌──────┴──────┐ │ +│ │ Connection │ Protocol I/O │ +│ │ (read/write)│ ServerOp / ClientOp │ +│ └──────┬──────┘ │ +└──────────────────────────┼────────────────────────────────┘ + │ + ┌──────┴──────┐ + │ NATS Server │ (Go binary, TCP/TLS/WS) + └─────────────┘ +``` + +## Key Concepts + +### Subject +NATS uses subject strings for message addressing. A `Subject` is a validated, immutable, UTF-8 string backed by `Bytes`. Subjects use dot-delimited tokens (e.g., `events.data.sensor1`). Wildcards `*` (single token) and `>` (multi-token suffix) are supported for subscriptions. + +### ClientOp / ServerOp +The NATS client-server protocol is text-based with binary payloads. The client sends `ClientOp` variants (CONNECT, PUB/HPUB, SUB, UNSUB, PING, PONG) and receives `ServerOp` variants (INFO, MSG/HMSG, +OK, -ERR, PING, PONG). + +### Command +Internal command type sent from `Client` to `ConnectionHandler` via `mpsc` channel. Includes Publish, Request, Subscribe, Unsubscribe, Flush, Drain, Reconnect, SetServerPool, ServerPool. + +### Multiplexer +A single internal subscription (SID 0) that routes all request-reply responses. When a `Request` is made, a unique inbox token is registered in the multiplexer's sender map, and the response is dispatched to the corresponding `oneshot::Sender`. + +### ConnectionHandler +A single Tokio task that drives all protocol I/O. It processes server operations from `Connection`, handles client commands from the `mpsc` channel, manages subscriptions, maintains ping/pong health, and orchestrates reconnection. + +## nats-server Test Harness + +The `nats-server` crate provides utilities for launching real NATS server instances in tests: + +- `run_server(cfg)` — starts a single server with optional config +- `run_cluster(cfg)` — starts a 3-node cluster +- `Server` struct — holds the child process, cleans up on drop +- `Server::restart()` — kills and restarts the server process +- `Server::client_url()` — reads the INFO from the server to get the client URL +- `set_lame_duck_mode(server)` — sends LDM signal to the server process + +The test harness spawns the Go `nats-server` binary via `std::process::Command`, using dynamic ports for parallel test execution. It auto-discovers the client URL by connecting to the server's TCP port and parsing the `INFO` JSON. On `Drop`, it kills the child process and cleans up JetStream storage directories. + +## Feature Flags + +```toml +# Default: everything enabled +default = ["server_2_10", "server_2_11", "server_2_12", "server_2_14", + "service", "ring", "jetstream", "nkeys", "crypto", + "object-store", "kv", "websockets", "nuid"] + +# Subsystems +jetstream # JetStream API +kv # Key-Value store (requires jetstream) +object-store # Object store (requires jetstream + crypto) +service # Service API + +# Crypto backends (pick one) +ring # Default crypto backend +aws-lc-rs # Alternative backend +fips # FIPS mode (requires aws-lc-rs) + +# Auth +nkeys # NKey authentication + +# Other +nuid # NUID-based ID generation (falls back to rand) +crypto # Encryption support +websockets # WebSocket transport +experimental # Experimental features + +# Server version markers (enable version-specific API fields) +server_2_10 +server_2_11 +server_2_12 +server_2_14 +``` + +## Dependencies (Key) + +| Dependency | Purpose | +|-----------|---------| +| `tokio` | Async runtime (macros, rt, net, sync, time, io-util) | +| `bytes` | Zero-copy byte buffers for payloads | +| `tokio-rustls` | TLS via rustls | +| `rustls-native-certs` | Load native TLS root certificates | +| `serde` / `serde_json` | JSON serialization for protocol messages and JetStream API | +| `memchr` | Fast CRLF search for protocol parsing | +| `futures-util` | Stream trait, Sink trait, StreamExt | +| `tracing` | Structured logging | +| `thiserror` | Error type derivation | +| `url` | URL parsing for server addresses | +| `portable-atomic` | Portable atomic operations | + +## References + +- [NATS Protocol Specification](https://docs.nats.io/reference/reference-protocols/nats-protocol) +- [NATS JetStream Documentation](https://docs.nats.io/nats-concepts/jetstream) +- [async-nats on docs.rs](https://docs.rs/async-nats) \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/02-protocol-and-wire-format.md b/docs/research/references/nats.rs/nats-server/02-protocol-and-wire-format.md new file mode 100644 index 0000000..fa7d0ca --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/02-protocol-and-wire-format.md @@ -0,0 +1,281 @@ +# NATS Client Protocol and Wire Format + +**Protocol**: NATS Client Protocol v1 (with dynamic reconfiguration) +**Transport**: TCP (port 4222), TLS, WebSocket (ws/wss) + +## Protocol Overview + +The NATS client-server protocol is a simple, text-based protocol with binary payload support. All operations are terminated with `\r\n`. Messages carry their payload length, allowing efficient binary data transfer. + +### Connection Lifecycle + +``` +Client Server + │ │ + │◄──────────── INFO {json} ────────────────────│ Server sends INFO first + │ │ + │────────────── CONNECT {json} ────────────────►│ Client sends CONNECT + │────────────── PING ──────────────────────────►│ Client sends PING + │◄──────────── PONG ────────────────────────── │ Server confirms connection + │ │ + │──── SUB/UNSUB/PUB/HPUB ──────────────────────►│ Normal operation + │◄─── MSG/HMSG/+OK/-ERR/PING ─────────────────│ + │ │ +``` + +## Server Operations (ServerOp) + +These are operations received from the server. The `Connection` module parses these from the read buffer. + +### INFO + +Sent by the server upon connection and asynchronously when cluster topology changes. + +``` +INFO {json}\r\n +``` + +JSON fields (see `ServerInfo` struct): + +| Field | Type | Description | +|-------|------|-------------| +| `server_id` | String | Unique server identifier | +| `server_name` | String | Generated server name | +| `host` | String | Cluster host | +| `port` | u16 | Cluster port | +| `version` | String | Server version | +| `auth_required` | bool | Authentication required | +| `tls_required` | bool | TLS required | +| `max_payload` | usize | Maximum payload size | +| `proto` | i8 | Protocol version (0 or 1) | +| `client_id` | u64 | Server-assigned client ID | +| `go` | String | Go build version | +| `nonce` | String | Nonce for nkey auth | +| `connect_urls` | Vec | Cluster server URLs | +| `client_ip` | String | Client IP as seen by server | +| `headers` | bool | Server supports headers | +| `ldm` | bool | Lame duck mode | +| `cluster` | Option | Cluster name | +| `domain` | Option | NATS domain | +| `jetstream` | bool | JetStream enabled | + +### MSG + +Delivers a message to a subscription (no headers): + +``` +MSG [reply-to] <#bytes>\r\n +\r\n +``` + +### HMSG + +Delivers a message with headers: + +``` +HMSG [reply-to] <#header-bytes> <#total-bytes>\r\n +\r\n +: \r\n +\r\n +\r\n +``` + +Header format follows the NATS/1.0 header spec: +- First line: `NATS/1.0` optionally followed by status code and description +- Subsequent lines: `name: value` headers +- Empty line separates headers from payload +- Header values may span multiple lines (continuation lines start with whitespace) + +### PING / PONG + +``` +PING\r\n → Client responds with PONG +PONG\r\n → Acknowledges client's PING +``` + +### +OK / -ERR + +``` ++OK\r\n → Success acknowledgment (verbose mode) +-ERR \r\n → Error from server +``` + +Common server errors: +- `authorization violation` → parsed as `ServerError::AuthorizationViolation` +- Other strings → `ServerError::Other(String)` + +## Client Operations (ClientOp) + +These are operations sent from the client to the server. The `Connection` module serializes these to the write buffer. + +### CONNECT + +Sent as the first client operation after receiving INFO. Contains authentication and capability information. + +``` +CONNECT {json}\r\n +``` + +JSON fields (see `ConnectInfo` struct): + +| Field | Type | Description | +|-------|------|-------------| +| `verbose` | bool | Enable +OK acknowledgments (always false in this client) | +| `pedantic` | bool | Strict format checking (always false) | +| `jwt` | Option | User JWT for auth | +| `nkey` | Option | Public nkey for auth | +| `sig` | Option | Signed nonce (Base64URL encoded) | +| `name` | Option | Client name | +| `echo` | bool | Whether server should echo messages back | +| `lang` | String | Implementation language ("rust") | +| `version` | String | Client version | +| `protocol` | u8 | Protocol version (1 = dynamic) | +| `tls_required` | bool | TLS required | +| `user` | Option | Username | +| `pass` | Option | Password | +| `auth_token` | Option | Auth token | +| `headers` | bool | Client supports headers (always true) | +| `no_responders` | bool | Client supports no-responders (always true) | + +### PUB / HPUB + +Publish a message: + +``` +PUB [reply-to] <#payload-bytes>\r\n +\r\n +``` + +Publish with headers: + +``` +HPUB [reply-to] <#header-bytes> <#total-bytes>\r\n +: \r\n +\r\n +\r\n +``` + +### SUB + +Subscribe to a subject: + +``` +SUB [queue-group] \r\n +``` + +### UNSUB + +Unsubscribe from a subscription: + +``` +UNSUB [max]\r\n +``` + +The optional `max` parameter tells the server to auto-unsubscribe after receiving the specified number of messages. + +### PING / PONG + +``` +PING\r\n → Health check / keepalive +PONG\r\n → Response to server PING +``` + +## Protocol Version + +The `Protocol` enum has two variants: + +| Value | Name | Description | +|-------|------|-------------| +| 0 | Original | Basic protocol | +| 1 | Dynamic | Supports async INFO for cluster topology changes, lame duck mode | + +This client always sends `protocol: 1` (Dynamic), enabling: +- Asynchronous INFO messages with updated server lists +- Lame duck mode notifications +- Dynamic reconfiguration of cluster topology + +## Wire Format Details + +### Message Length Calculation + +For plain `MSG`: +``` +length = subject.len() + reply.map_or(0, |r| r.len()) + payload.len() +``` + +For `HMSG`: +``` +length = subject.len() + reply.map_or(0, |r| r.len()) + header_len + payload.len() +``` + +Where `header_len` = serialized header bytes and `total_len` = `header_len + payload.len()`. + +### Write Buffer Architecture + +The `Connection` uses a two-tier write buffer: + +1. **`flattened_writes`** (`BytesMut`) — for small writes (< 4096 bytes). Protocol headers, short commands, and small messages are flattened into this buffer for efficient sequential writing. + +2. **`write_buf`** (`VecDeque`) — for large writes (>= 4096 bytes). Large payloads are appended as separate `Bytes` chunks. Supports vectored writes (`write_vectored`) when the underlying stream supports it, writing up to 64 chunks at once. + +The soft limit for the total write buffer is 65,535 bytes (`SOFT_WRITE_BUF_LIMIT`). When exceeded, the `ConnectionHandler` stops processing new commands until the buffer drains. + +### Read Buffer Architecture + +The `Connection` uses a single `BytesMut` read buffer with configurable initial capacity (default 65,535 bytes). Protocol parsing uses `memchr::memmem::find` to locate CRLF delimiters efficiently. If a partial message is in the buffer, the parser returns `None` and waits for more data. + +### Header Serialization + +Headers are serialized in NATS/1.0 format: + +``` +NATS/1.0\r\n +Header-Name: Header-Value\r\n +Multi-Line-Header: value part 1\r\n + continuation of value\r\n +Another-Header: another value\r\n +\r\n +``` + +The `HeaderMap::to_bytes()` method handles this serialization, using `httparse`-compatible line folding for multi-line values. + +### Status Codes in Headers + +NATS status codes are embedded in the `HMSG` header version line: + +``` +NATS/1.0 404 No Messages\r\n +NATS/1.0 408 Request Timeout\r\n +NATS/1.0 503 No Responders\r\n +``` + +Common codes used by the client: + +| Code | Constant | Meaning | +|------|----------|---------| +| 100 | `IDLE_HEARTBEAT` | JetStream idle heartbeat | +| 200 | `OK` | Success | +| 404 | `NOT_FOUND` | Message/stream not found | +| 408 | `TIMEOUT` | Request timeout | +| 409 | `REQUEST_TERMINATED` | Request terminated | +| 503 | `NO_RESPONDERS` | No responders available | + +## Protocol Parsing Implementation + +The `Connection::try_read_op()` method handles all protocol parsing: + +1. Search for `\r\n` delimiter using `memchr::memmem::find` +2. Match the operation prefix: + - `+OK` → `ServerOp::Ok` + - `PING` → `ServerOp::Ping` + - `PONG` → `ServerOp::Pong` + - `-ERR` → parse error description → `ServerOp::Error` + - `INFO ` → parse JSON → `ServerOp::Info` + - `MSG ` → parse subject/sid/reply/length, read payload → `ServerOp::Message` + - `HMSG ` → parse headers + payload → `ServerOp::Message` +3. Unknown prefix → return `io::Error` with `InvalidInput` + +For `MSG` and `HMSG`, if the complete payload isn't yet in the read buffer (checked via `len + payload_len + 4 > remaining`), the method returns `Ok(None)` and the buffer accumulates more data before retrying. + +Non-UTF8 subjects in server messages are handled gracefully — the parser returns an `io::Error` rather than panicking, which is critical because the Go server does not enforce UTF-8 in subjects (regression fix for issue #1572). \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/03-key-types-and-traits.md b/docs/research/references/nats.rs/nats-server/03-key-types-and-traits.md new file mode 100644 index 0000000..a246ad4 --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/03-key-types-and-traits.md @@ -0,0 +1,443 @@ +# Key Types and Traits + +This document covers the core data types in the `async-nats` crate that form the public API and internal plumbing. + +## Public Types + +### Client + +**Location**: `client.rs` + +`Client` is the primary user-facing type. It is a lightweight, cloneable handle to a NATS connection. + +```rust +#[derive(Clone, Debug)] +pub struct Client { + info: tokio::sync::watch::Receiver>, + state: tokio::sync::watch::Receiver, + sender: mpsc::Sender, + poll_sender: PollSender, + next_subscription_id: Arc, + subscription_capacity: usize, + inbox_prefix: Arc, + request_timeout: Option, + max_payload: Arc, + connection_stats: Arc, + skip_subject_validation: bool, +} +``` + +Key methods: +- `publish(subject, payload)` — fire-and-forget publish +- `publish_with_headers(subject, headers, payload)` — publish with NATS headers +- `publish_with_reply(subject, reply, payload)` — publish with reply subject +- `request(subject, payload)` — request-response (returns `Message`) +- `send_request(subject, request)` — request with `Request` builder +- `subscribe(subject)` — subscribe to a subject, returns `Subscriber` +- `queue_subscribe(subject, queue_group)` — subscribe as part of a queue group +- `flush()` — ensure all pending messages are written to the wire +- `drain()` — gracefully drain all subscriptions and close +- `force_reconnect()` — trigger immediate reconnection +- `new_inbox()` — generate a unique inbox subject for request-reply +- `server_info()` — get last received `ServerInfo` +- `max_payload()` — get server's maximum payload size +- `connection_state()` — get current connection `State` +- `statistics()` — get `Arc` for connection metrics +- `is_server_compatible(major, minor, patch)` — check server version compatibility +- `set_server_pool(addrs)` / `server_pool()` — manage server pool + +`Client` also implements `Sink` for backpressure-aware publishing. + +### Subscriber + +**Location**: `lib.rs` + +A `Subscriber` receives messages from a single subscription. It implements `futures::Stream`. + +```rust +#[derive(Debug)] +pub struct Subscriber { + sid: u64, + receiver: mpsc::Receiver, + sender: mpsc::Sender, +} +``` + +Key methods: +- `unsubscribe()` — unsubscribe and close the stream +- `unsubscribe_after(max)` — auto-unsubscribe after N messages +- `drain()` — gracefully drain remaining messages then close + +On `Drop`, `Subscriber` automatically sends an `Unsubscribe` command and closes the receiver channel. + +### Message + +**Location**: `message.rs` + +Represents an inbound NATS message. + +```rust +#[derive(Clone, Debug, Serialize, Deserialize)] +pub struct Message { + pub subject: Subject, + pub reply: Option, + pub payload: Bytes, + pub headers: Option, + pub status: Option, + pub description: Option, + pub length: usize, +} +``` + +### OutboundMessage + +**Location**: `message.rs` + +Represents a message to be published. No status/description fields (those are inbound-only). + +```rust +#[derive(Clone, Debug)] +pub struct OutboundMessage { + pub subject: Subject, + pub reply: Option, + pub payload: Bytes, + pub headers: Option, +} +``` + +### Subject + +**Location**: `subject.rs` + +An immutable, validated UTF-8 string backed by `Bytes`. Used throughout the crate instead of raw `String` for subjects. + +```rust +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct Subject { + bytes: Bytes, +} +``` + +Implements `Deref`, `From<&str>`, `From`, `TryFrom`, `Serialize`, `Deserialize`. + +Validation methods: +- `is_valid()` — checks NATS subject rules (no leading/trailing dots, no consecutive dots, no whitespace) +- `validated(s)` — construct with validation, returns `Result` +- `from_static_validated(s)` — const-time validation for static strings (compile-time panic on invalid) + +### ToSubject Trait + +**Location**: `subject.rs` + +```rust +pub trait ToSubject { + fn to_subject(&self) -> Subject; +} +``` + +Implemented for `Subject`, `&'static str`, `String`. All methods accepting subjects are generic over `impl ToSubject`. + +### HeaderMap + +**Location**: `header.rs` + +NATS message headers, modeled after the `http::header` crate. + +```rust +#[derive(Clone, PartialEq, Eq, Debug, Default)] +pub struct HeaderMap { + inner: HashMap>, +} +``` + +Supports multiple values per header name (like HTTP). Key methods: +- `insert(name, value)` — replace all values for a name +- `append(name, value)` — add a value to a name +- `get(name)` — get the first value +- `get_all(name)` — get all values as an iterator +- `len()` / `is_empty()` — number of header entries +- `to_bytes()` — serialize to NATS/1.0 wire format +- `wire_len()` — size in wire format (for payload size checks) + +### StatusCode + +**Location**: `status.rs` + +NATS status codes (100-999), structurally similar to HTTP status codes. + +```rust +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +pub struct StatusCode(NonZeroU16); +``` + +Constants: +| Constant | Code | Meaning | +|----------|------|---------| +| `IDLE_HEARTBEAT` | 100 | JetStream idle heartbeat | +| `OK` | 200 | Success | +| `NOT_FOUND` | 404 | Not found | +| `TIMEOUT` | 408 | Timeout | +| `REQUEST_TERMINATED` | 409 | Request terminated | +| `NO_RESPONDERS` | 503 | No responders | + +### ServerInfo + +**Location**: `lib.rs` + +Deserialized from the server's `INFO` JSON message. Contains server capabilities, connection details, and cluster information. + +### ConnectInfo + +**Location**: `lib.rs` + +Serialized into the client's `CONNECT` JSON message. Contains authentication credentials, client capabilities, and protocol preferences. + +### ServerAddr + +**Location**: `lib.rs` + +A validated NATS server URL, supporting schemes `nats://`, `tls://`, `ws://`, `wss://`. + +```rust +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct ServerAddr(Url); +``` + +Methods: +- `from_url(url)` — validate and create +- `tls_required()` — true for `tls://` scheme +- `is_websocket()` — true for `ws://` or `wss://` +- `host()` / `port()` / `scheme()` — URL component accessors +- `socket_addrs()` — async DNS resolution +- `username()` / `password()` — embedded credentials + +### Auth + +**Location**: `auth.rs` + +Container for authentication credentials. + +```rust +#[derive(Clone, Default)] +pub struct Auth { + pub jwt: Option, + pub nkey: Option, + pub signature_callback: Option>>, + pub signature: Option>, + pub username: Option, + pub password: Option, + pub token: Option, +} +``` + +### Request + +**Location**: `client.rs` + +Builder for customized request-response operations. + +```rust +#[derive(Default)] +pub struct Request { + pub payload: Option, + pub headers: Option, + pub timeout: Option>, + pub inbox: Option, +} +``` + +### Statistics + +**Location**: `client.rs` + +Atomic connection statistics shared between Client and ConnectionHandler. + +```rust +#[derive(Default, Debug)] +pub struct Statistics { + pub in_bytes: AtomicU64, + pub out_bytes: AtomicU64, + pub in_messages: AtomicU64, + pub out_messages: AtomicU64, + pub connects: AtomicU64, +} +``` + +### Event + +**Location**: `lib.rs` + +Events emitted by the client for connection lifecycle monitoring. + +```rust +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Event { + Connected, + Disconnected, + LameDuckMode, + Draining, + Closed, + SlowConsumer(u64), + ServerError(ServerError), + ClientError(ClientError), +} +``` + +## Internal Types + +### Command + +**Location**: `lib.rs` + +Internal commands sent from `Client` to `ConnectionHandler` via `mpsc` channel. + +```rust +pub(crate) enum Command { + Publish(OutboundMessage), + Request { subject, payload, respond, headers, sender: oneshot::Sender }, + Subscribe { sid, subject, queue_group, sender: mpsc::Sender }, + Unsubscribe { sid, max: Option }, + Flush { observer: oneshot::Sender<()> }, + Drain { sid: Option }, + Reconnect, + SetServerPool { servers: Vec, result: oneshot::Sender> }, + ServerPool { result: oneshot::Sender> }, +} +``` + +### ClientOp / ServerOp + +**Location**: `lib.rs` + +Protocol-level operation types used by `Connection` for wire format parsing and serialization. + +### Subscription (Internal) + +**Location**: `lib.rs` + +```rust +struct Subscription { + subject: Subject, + sender: mpsc::Sender, + queue_group: Option, + delivered: u64, + max: Option, +} +``` + +### Multiplexer (Internal) + +**Location**: `lib.rs` + +```rust +struct Multiplexer { + subject: Subject, // Wildcard subscription subject (e.g., "_INBOX.xxx.*") + prefix: Subject, // Prefix for routing (e.g., "_INBOX.xxx.") + senders: HashMap>, // token → sender +} +``` + +### Connection State + +**Location**: `connection.rs` + +```rust +#[derive(Debug, Eq, PartialEq, Clone)] +pub enum State { + Pending, + Connected, + Disconnected, +} +``` + +### Protocol + +**Location**: `lib.rs` + +```rust +#[derive(Serialize_repr, Deserialize_repr, PartialEq, Eq, Debug, Clone, Copy)] +#[repr(u8)] +pub enum Protocol { + Original = 0, + Dynamic = 1, +} +``` + +## Error Type Pattern + +The crate uses a generic `Error` type throughout. Every subsystem defines its own `ErrorKind` enum and a type alias: + +```rust +// Define the kind enum +#[derive(Clone, Debug, PartialEq)] +pub enum PublishErrorKind { + MaxPayloadExceeded, + InvalidSubject, + Send, +} + +// Define the error type alias +pub type PublishError = Error; + +// Construct errors +PublishError::new(PublishErrorKind::MaxPayloadExceeded) +PublishError::with_source(PublishErrorKind::Send, io_error) + +// Match on errors +if err.kind() == PublishErrorKind::MaxPayloadExceeded { ... } +``` + +Error kinds in the crate: + +| Error Type | Kind Enum | Context | +|-----------|-----------|---------| +| `ConnectError` | `ConnectErrorKind` | Initial connection failures | +| `PublishError` | `PublishErrorKind` | Publish validation failures | +| `RequestError` | `RequestErrorKind` | Request-response failures | +| `SubscribeError` | `SubscribeErrorKind` | Subscription failures | +| `FlushError` | `FlushErrorKind` | Flush failures | +| `ServerPoolError` | `ServerPoolErrorKind` | Server pool query failures | +| `SetServerPoolError` | `SetServerPoolErrorKind` | Server pool modification failures | + +## Trait Implementations + +### Client Trait Interfaces + +The `Client` implements several traits defined in `client::traits`: + +```rust +// Publisher trait — publish with optional reply subject +trait Publisher { + fn publish_with_reply(&self, subject: S, reply: R, payload: Bytes) -> impl Future>; + fn publish_message(&self, msg: OutboundMessage) -> impl Future>; +} + +// Subscriber trait — subscribe to a subject +trait Subscriber { + fn subscribe(&self, subject: S) -> impl Future>; +} + +// Requester trait — send request-response +trait Requester { + fn send_request(&self, subject: S, request: Request) -> impl Future>; +} + +// TimeoutProvider trait — access request timeout +trait TimeoutProvider { + fn timeout(&self) -> Option; +} +``` + +### ToServerAddrs Trait + +**Location**: `lib.rs` + +Converts various address types into server address iterators. Implemented for `ServerAddr`, `str`, `String`, `&[T]`, `Vec`, `&[ServerAddr]`, and references. + +### Sink + +`Client` implements `futures::Sink` for backpressure-aware publishing through the `PollSender` adapter. + +### Stream for Subscriber + +`Subscriber` implements `futures::Stream` with `Item = Message`, delegating to the internal `mpsc::Receiver`. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/04-connection-handler-and-data-flow.md b/docs/research/references/nats.rs/nats-server/04-connection-handler-and-data-flow.md new file mode 100644 index 0000000..bd929ed --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/04-connection-handler-and-data-flow.md @@ -0,0 +1,338 @@ +# Connection Handler and Data Flow + +This document covers the internal `ConnectionHandler` that drives all protocol I/O, and the data flow through the system. + +## ConnectionHandler + +**Location**: `lib.rs` + +The `ConnectionHandler` is the heart of the client. It runs as a single Tokio task and manages all communication with the NATS server. + +```rust +pub(crate) struct ConnectionHandler { + connection: Connection, // Low-level I/O + connector: Connector, // Server pool, reconnection + subscriptions: HashMap, // Active subscriptions + multiplexer: Option, // Request-reply multiplexer + pending_pings: usize, // Unanswered PINGs + info_sender: tokio::sync::watch::Sender>, + ping_interval: Interval, // Periodic PING timer + should_reconnect: bool, // Flag for forced reconnect + flush_observers: Vec>, // Pending flush callbacks + is_draining: bool, // Connection is draining + drain_pings: VecDeque, // SIDs being drained +} +``` + +## Data Flow: Publish + +``` +Application + │ + │ client.publish("events.data", payload) + │ + ▼ +Client + │ validates subject & payload size + │ sends Command::Publish(OutboundMessage) via mpsc channel + │ + ▼ +ConnectionHandler::handle_command(Command::Publish) + │ increments out_messages, out_bytes statistics + │ calls connection.enqueue_write_op(&ClientOp::Publish { ... }) + │ + ▼ +Connection::enqueue_write_op + │ serializes to wire format: + │ "PUB events.data 11\r\n" or "HPUB events.data 23 34\r\n" + │ appends to flattened_writes or write_buf + │ + ▼ +Connection::poll_write + │ uses vectored writes (64 chunks) if supported + │ or sequential writes otherwise + │ + ▼ +Connection::poll_flush + │ flushes the TCP/TLS/WS stream + │ notifies flush_observers + │ + ▼ +NATS Server (TCP/TLS/WebSocket) +``` + +## Data Flow: Subscribe + +``` +Application + │ + │ client.subscribe("events.>") + │ + ▼ +Client::subscribe + │ validates subject (always, regardless of skip_subject_validation) + │ allocates next sid via AtomicU64 + │ creates mpsc channel for messages + │ sends Command::Subscribe { sid, subject, sender } + │ returns Subscriber { sid, receiver } + │ + ▼ +ConnectionHandler::handle_command(Command::Subscribe) + │ creates Subscription { subject, sender, delivered: 0, max: None } + │ inserts into subscriptions HashMap + │ calls connection.enqueue_write_op(&ClientOp::Subscribe { sid, subject, queue_group }) + │ + ▼ +Connection::enqueue_write_op + │ serializes: "SUB events.> 42\r\n" + │ + ▼ +Server sends MSG for matching subjects: + │ + ▼ +ConnectionHandler::handle_server_op(ServerOp::Message { sid, subject, ... }) + │ looks up sid in subscriptions HashMap + │ constructs Message { subject, reply, payload, headers, status, description } + │ tries subscription.sender.try_send(message) + │ + ├── Ok → increments subscription.delivered, checks max + ├── Full → emits Event::SlowConsumer(sid) + └── Closed → removes subscription, sends ClientOp::Unsubscribe + │ + ▼ +Subscriber::poll_next (Stream impl) + │ receives from mpsc::Receiver + │ + ▼ +Application processes Message +``` + +## Data Flow: Request-Response + +The request-response pattern uses the **multiplexer** — a single wildcard subscription that routes responses to their waiting requesters. + +``` +Application + │ + │ client.request("service", payload) + │ + ▼ +Client::send_request + │ validates subject & payload size + │ creates oneshot channel for response + │ generates unique inbox: "_INBOX.." + │ sends Command::Request { subject, payload, respond, sender } + │ + ▼ +ConnectionHandler::handle_command(Command::Request) + │ extracts token from respond subject (after last '.') + │ if no multiplexer exists: + │ creates Multiplexer with wildcard sub "_INBOX..*" (SID 0) + │ sends ClientOp::Subscribe { sid: 0, subject: "_INBOX..*" } + │ inserts token → oneshot::Sender in multiplexer.senders + │ sends ClientOp::Publish { subject, payload, respond: "" } + │ + ▼ +Server routes request to service: + │ + ▼ +Service responds by publishing to the reply subject: + │ + ▼ +ConnectionHandler::handle_server_op(ServerOp::Message { sid: 0, ... }) + │ sid == MULTIPLEXER_SID (0), so enters multiplexer path + │ extracts token by stripping prefix from subject + │ looks up token in multiplexer.senders + │ sends Message via oneshot::Sender + │ + ▼ +Client::send_request receives via oneshot::Receiver + │ applies timeout (default 10s) + │ checks for NO_RESPONDERS status (503) + │ + ▼ +Application receives Message +``` + +### Custom Inbox Request + +If the `Request` builder specifies a custom `inbox`, the flow is different: +- The client subscribes to the inbox directly (not via multiplexer) +- Publishes with the inbox as the reply subject +- Waits for the message on that subscription +- No multiplexer involvement + +## Data Flow: Flush + +``` +Application + │ + │ client.flush() + │ + ▼ +Client::flush + │ creates oneshot channel + │ sends Command::Flush { observer } + │ + ▼ +ConnectionHandler::handle_command(Command::Flush) + │ pushes observer into flush_observers Vec + │ + ▼ +ProcessFut::poll (main loop) + │ after writing all pending data... + │ checks should_flush(): + │ Yes (write buffers empty, not yet flushed) → poll_flush + │ May (write buffers not empty) → poll_flush + │ No (already flushed) → skip + │ on successful flush: + │ drains flush_observers, sending () to each + │ + ▼ +Client::flush receives via oneshot::Receiver +``` + +## Data Flow: Drain + +``` +Application + │ + │ client.drain() or subscriber.drain() + │ + ▼ +Client::drain / Subscriber::drain + │ sends Command::Drain { sid: None } (whole client) + │ or Command::Drain { sid: Some(n) } (single subscription) + │ + ▼ +ConnectionHandler::handle_command(Command::Drain) + │ if sid is Some: + │ pushes sid to drain_pings + │ sends ClientOp::Unsubscribe { sid, max: None } + │ if sid is None (whole client): + │ sets is_draining = true + │ emits Event::Draining + │ for each subscription: drain_pings.push(sid), Unsubscribe + │ sends ClientOp::Ping (to flush the UNSUB messages) + │ + ▼ +ProcessFut::poll (main loop) + │ processes any remaining server messages + │ removes drained subscriptions from HashMap + │ if is_draining: returns ExitReason::Closed + │ + ▼ +ConnectionHandler exits, emits Event::Closed +``` + +## Main Processing Loop + +The `ConnectionHandler::process` method implements the core event loop via a custom `Future` (`ProcessFut`): + +```rust +impl Future for ProcessFut<'_> { + type Output = ExitReason; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + // 1. Check ping interval — send PING if due, disconnect if too many pending + while self.handler.ping_interval.poll_tick(cx).is_ready() { + if let Poll::Ready(exit) = self.ping() { return Poll::Ready(exit); } + } + + // 2. Read all available server operations + loop { + match self.handler.connection.poll_read_op(cx) { + Poll::Pending => break, + Poll::Ready(Ok(Some(server_op))) => self.handler.handle_server_op(server_op), + Poll::Ready(Ok(None)) => return Poll::Ready(ExitReason::Disconnected(None)), + Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))), + } + } + + // 3. Clean up drained subscriptions + while let Some(sid) = self.handler.drain_pings.pop_front() { + self.handler.subscriptions.remove(&sid); + } + + // 4. If draining, exit + if self.handler.is_draining { return Poll::Ready(ExitReason::Closed); } + + // 5. Process client commands (batch of up to 16) + // while write buffer not full + loop { + while !self.handler.connection.is_write_buf_full() { + match receiver.poll_recv_many(cx, recv_buf, 16) { + Poll::Pending => break, + Poll::Ready(1..) => { for cmd in recv_buf.drain(..) { handler.handle_command(cmd); } } + Poll::Ready(0) => return Poll::Ready(ExitReason::Closed), + } + } + + // 6. Write pending data to stream + match self.handler.connection.poll_write(cx) { + Poll::Pending => break, + Poll::Ready(Ok(())) => continue, // write buffer empty, try more commands + Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))), + } + } + + // 7. Flush stream and notify observers + match self.handler.connection.poll_flush(cx) { ... } + + // 8. Check for forced reconnect + if mem::take(&mut self.handler.should_reconnect) { + return Poll::Ready(ExitReason::ReconnectRequested); + } + + Poll::Pending + } +} +``` + +### Exit Reasons + +The main loop exits for three reasons: + +| Reason | Action | +|--------|--------| +| `Disconnected(Option)` | Attempt reconnection via `handle_disconnect()` | +| `ReconnectRequested` | Force reconnect (user-triggered) | +| `Closed` | Connection handler terminates, emit `Event::Closed` | + +On disconnection, `handle_disconnect()` is called which: +1. Resets `pending_pings` to 0 +2. Emits `Event::Disconnected` +3. Updates connection state to `Disconnected` +4. Calls `handle_reconnect()` which uses `Connector::connect()` +5. On successful reconnect, re-subscribes all active subscriptions +6. Re-subscribes the multiplexer wildcard if present + +## Slow Consumer Handling + +When a subscription's `mpsc::Sender` channel is full (the application isn't consuming messages fast enough): + +1. `try_send` returns `TrySendError::Full` +2. The `ConnectionHandler` emits `Event::SlowConsumer(sid)` +3. The message is **dropped** (not queued) +4. The subscription remains active + +When a subscription's receiver is dropped (application closed the stream): + +1. `try_send` returns `TrySendError::Closed` +2. The subscription is removed from the HashMap +3. An `UNSUB` command is sent to the server + +## Ping/Pong Health Check + +The `ConnectionHandler` maintains a periodic PING interval (default 60 seconds): + +1. `ping_interval` fires every N seconds +2. A `ClientOp::Ping` is enqueued +3. `pending_pings` counter increments +4. If `pending_pings > MAX_PENDING_PINGS (2)`, the connection is considered dead +5. When `ServerOp::Pong` is received, `pending_pings` decrements +6. Any server operation resets the ping interval timer + +## Batched Command Processing + +Commands from the `Client` are received in batches of up to 16 (`RECV_CHUNK_SIZE`) using `poll_recv_many`. This amortizes the cost of waking the task and enables pipelining multiple operations (e.g., publishing many messages) in a single poll cycle. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/05-connection-and-reconnection.md b/docs/research/references/nats.rs/nats-server/05-connection-and-reconnection.md new file mode 100644 index 0000000..b3b451a --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/05-connection-and-reconnection.md @@ -0,0 +1,277 @@ +# Connection and Reconnection + +This document covers how connections are established, TLS handling, the server pool, and the reconnection mechanism. + +## Connector + +**Location**: `connector.rs` + +The `Connector` manages the server pool and handles connection establishment and reconnection. + +```rust +pub(crate) struct Connector { + servers: Vec, // Server pool with per-server metadata + options: ConnectorOptions, // Connection configuration + connect_stats: Arc, // Shared statistics + attempts: usize, // Global reconnection attempt counter + events_tx: mpsc::Sender, // Event channel + state_tx: watch::Sender, // Connection state watcher + max_payload: Arc, // Server's max payload + last_info: ServerInfo, // Last known server info +} +``` + +### Server Pool + +Each server in the pool carries metadata: + +```rust +#[derive(Debug, Clone)] +pub struct Server { + pub addr: ServerAddr, + pub failed_attempts: usize, // Consecutive failed attempts + pub did_connect: bool, // Ever successfully connected? + pub is_discovered: bool, // Discovered via INFO, not user-configured + pub last_error: Option, // Last connection error +} +``` + +### ConnectorOptions + +```rust +pub(crate) struct ConnectorOptions { + pub tls_required: bool, + pub certificates: Vec, + pub client_cert: Option, + pub client_key: Option, + pub tls_client_config: Option, + pub tls_first: bool, + pub auth: Auth, + pub no_echo: bool, + pub connection_timeout: Duration, // Default: 5 seconds + pub name: Option, + pub ignore_discovered_servers: bool, + pub retain_servers_order: bool, + pub read_buffer_capacity: u16, // Default: 65535 + pub reconnect_delay_callback: Arc Duration>, + pub auth_callback: Option, Result>>, + pub max_reconnects: Option, + pub local_address: Option, + pub reconnect_to_server_callback: Option, +} +``` + +## Connection Establishment Flow + +``` +Connector::try_connect_to_server(addr) + │ + ├── 1. DNS resolution + │ server_addr.socket_addrs() + │ + ├── 2. For each resolved address: + │ │ + │ ├── 2a. Connect with timeout + │ │ tokio::time::timeout(connection_timeout, try_connect_to(socket_addr, ...)) + │ │ + │ └── 2b. try_connect_to(): + │ │ + │ ├── Select transport: + │ │ ├── "ws" → WebSocket (tokio_websockets) + │ │ ├── "wss" → WebSocket over TLS + │ │ └── default → TCP (TcpStream) + │ │ + │ ├── Optional: bind to local_address + │ ├── Set TCP_NODELAY + │ ├── Create Connection with read_buffer_capacity + │ │ + │ ├── If tls_first: upgrade to TLS before INFO + │ │ + │ ├── Read INFO from server + │ │ + │ ├── If TLS required (by option, server, or URL scheme): + │ │ upgrade to TLS (rustls) + │ │ + │ ├── Discover servers from INFO.connect_urls + │ │ (unless ignore_discovered_servers) + │ │ + │ ├── Build ConnectInfo with auth: + │ │ ├── username/password (from Auth or URL) + │ │ ├── token (from Auth) + │ │ ├── nkey + signed nonce (feature: nkeys) + │ │ ├── JWT + signature callback (feature: nkeys) + │ │ └── auth_callback (custom async callback) + │ │ + │ ├── Send CONNECT + PING + │ │ + │ └── Wait for response: + │ ├── -ERR (authorization violation) → error + │ ├── PONG or +OK → success + │ └── EOF → error + │ + └── 3. On success: + ├── Reset attempt counter + ├── Increment connects statistic + ├── Emit Event::Connected + ├── Update State::Connected + ├── Store max_payload + ├── Update per-server metadata (did_connect, failed_attempts) + └── Return (ServerInfo, Connection) +``` + +## TLS Handling + +The client supports three TLS modes: + +### 1. Standard TLS (INFO → TLS) +Default behavior. The client receives the `INFO` message in plaintext, then upgrades to TLS if: +- `tls_required` option is set +- Server's `INFO.tls_required` is true +- URL scheme is `tls://` + +### 2. TLS First (TLS → INFO) +When `ConnectOptions::tls_first()` is enabled, the client establishes TLS before reading INFO. This requires the server to have `handshake_first` enabled. Useful for environments where plaintext INFO is not acceptable. + +### 3. WebSocket TLS +For `wss://` URLs, TLS is handled by the WebSocket library (`tokio-websockets`) directly, not by the client's TLS layer. + +### TLS Configuration +The client uses `rustls` via `tokio-rustls`. Configuration steps: +1. Load root certificates from system store (`rustls-native-certs`) +2. Optionally add custom root certificates from PEM files +3. Optionally configure client certificate and key for mTLS +4. Optionally pass a custom `rustls::ClientConfig` + +Crypto backend is selectable via feature flags: +- `ring` (default) +- `aws-lc-rs` +- `fips` (requires aws-lc-rs) + +## Reconnection + +### Reconnection Trigger + +Reconnection is triggered when: +1. I/O error during read or write (`ExitReason::Disconnected`) +2. Too many pending PINGs (no PONG received) +3. User calls `Client::force_reconnect()` (`ExitReason::ReconnectRequested`) + +### Reconnection Flow + +``` +ConnectionHandler::handle_disconnect() + │ + ├── Reset pending_pings to 0 + ├── Emit Event::Disconnected + ├── Update State::Disconnected + │ + └── handle_reconnect() + │ + └── Connector::connect() + │ + └── Loop: try_connect() + │ + ├── If reconnect_to_server_callback is set: + │ │ Call callback with (server_pool, server_info) + │ │ If returns Some(ReconnectToServer): + │ │ Validate server is in pool + │ │ Use callback's delay or default backoff + │ │ Try connecting to selected server + │ └── If None or invalid: fall through to default + │ + ├── Default selection: + │ ├── Shuffle servers (unless retain_servers_order) + │ ├── Sort by failed_attempts (ascending) + │ └── Try each server in order + │ + ├── For each server: + │ ├── Increment attempts counter + │ ├── Check max_reconnects limit + │ ├── Apply reconnect delay (exponential backoff) + │ └── try_connect_to_server(addr) + │ + ├── On success: + │ ├── Reset attempts to 0 + │ ├── Re-subscribe all active subscriptions + │ │ (filter out closed subscription channels) + │ ├── Re-subscribe multiplexer wildcard + │ └── Return (ServerInfo, Connection) + │ + └── On failure: + ├── Update per-server metadata (failed_attempts, last_error) + ├── Auth errors → propagate immediately + └── Other errors → continue to next server +``` + +### Exponential Backoff + +Default reconnect delay function: + +```rust +fn reconnect_delay_callback_default(attempts: usize) -> Duration { + if attempts <= 1 { + Duration::from_millis(0) + } else { + let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX); + let max = Duration::from_secs(4); + cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), max) + } +} +``` + +| Attempt | Delay | +|---------|-------| +| 1 | 0ms | +| 2 | 0ms | +| 3 | 2ms | +| 4 | 4ms | +| 5 | 8ms | +| ... | ... | +| 13 | 4096ms | +| 14+ | 4000ms (capped) | + +Custom delay functions can be provided via `ConnectOptions::reconnect_delay_callback()`. + +### Server Pool Updates + +The server pool is dynamic: + +1. **Initial pool**: from `connect()` / `ConnectOptions::connect()` URL(s) +2. **Discovered servers**: added from `INFO.connect_urls` on each connection (unless `ignore_discovered_servers` is set) +3. **Runtime updates**: via `Client::set_server_pool()` — replaces the entire pool while preserving per-server state for servers that appear in both old and new pools +4. **Order**: servers are shuffled by default (random selection), unless `retain_servers_order` is set + +### Max Reconnects + +The `max_reconnects` option limits total reconnection attempts: +- `None` or `0` → unlimited (default) +- `Some(n)` → give up after `n` total attempts +- Counter is reset on successful connection and when `set_server_pool()` is called + +## ConnectOptions Defaults + +| Option | Default | +|--------|---------| +| `connection_timeout` | 5 seconds | +| `ping_interval` | 60 seconds | +| `sender_capacity` | 2048 | +| `subscription_capacity` | 65536 | +| `inbox_prefix` | `"_INBOX"` | +| `request_timeout` | 10 seconds | +| `retry_on_initial_connect` | false | +| `ignore_discovered_servers` | false | +| `retain_servers_order` | false | +| `read_buffer_capacity` | 65535 | +| `skip_subject_validation` | false | +| `no_echo` | false | +| `tls_required` | false | +| `tls_first` | false | +| `max_reconnects` | None (unlimited) | + +## Background Connection + +When `ConnectOptions::retry_on_initial_connect()` is enabled, the `connect()` function returns a `Client` immediately, before the connection is established. The connection is established in a background Tokio task. This means: +- `client.server_info()` returns `ServerInfo::default()` until connected +- `client.connection_state()` returns `State::Pending` +- Operations like `publish()` will queue in the command channel +- The `Client` becomes usable once the background task connects \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/06-jetstream-internals.md b/docs/research/references/nats.rs/nats-server/06-jetstream-internals.md new file mode 100644 index 0000000..a0cca57 --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/06-jetstream-internals.md @@ -0,0 +1,472 @@ +# JetStream Internals + +This document covers the JetStream subsystem — how it provides stream-based messaging with persistence, consumer management, and higher-level APIs like KV and Object Store. + +## JetStream Context + +**Location**: `jetstream/context.rs` + +The `Context` is the entry point to the JetStream API. It wraps a `Client` and provides stream management, publishing, and consumer operations. + +```rust +#[derive(Debug, Clone)] +pub struct Context { + pub(crate) client: Client, + pub(crate) prefix: String, // API subject prefix (default: "$JS.API") + pub(crate) timeout: Duration, // Default request timeout + pub(crate) max_ack_semaphore: Arc, // Limits in-flight ack waits + pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver, OwnedSemaphorePermit)>, + pub(crate) backpressure_on_inflight: bool, +} +``` + +### Context Creation + +```rust +// Default context (prefix = "$JS.API") +let jetstream = async_nats::jetstream::new(client); + +// With domain (prefix = "$JS.hub.API") +let jetstream = async_nats::jetstream::with_domain(client, "hub"); + +// With custom prefix +let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API"); + +// Builder pattern for more options +let jetstream = async_nats::jetstream::Context::builder(client) + .domain("hub") + .prefix("$JS.API") + .timeout(Duration::from_secs(30)) + .max_ack_pending(256) + .backpressure_on_inflight(true) + .build(); +``` + +### JetStream API Subject Convention + +All JetStream API calls are request-response messages sent to subjects following the pattern: + +``` +$JS.API..[.] +``` + +Examples: +- `$JS.API.STREAM.CREATE.events` — create stream "events" +- `$JS.API.STREAM.INFO.events` — get stream info +- `$JS.API.CONSUMER.DURABLE.CREATE.events.myconsumer` — create durable consumer +- `$JS.API.CONSUMER.MSG.NEXT.events.myconsumer` — pull next message + +With a domain, the prefix changes to `$JS..API`. + +## Stream Management + +**Location**: `jetstream/stream.rs` + +### Stream Config + +```rust +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)] +pub struct Config { + pub name: String, + pub subjects: Vec, // Subject filter + pub retention: RetentionPolicy, // Limits, Interest, WorkQueue + pub max_consumers: i32, + pub max_messages: i64, // Per-stream message limit + pub max_messages_per_subject: i64, + pub max_bytes: i64, // Per-stream byte limit + pub max_age: Duration, // Message TTL + pub max_message_size: Option, // Max individual message size + pub storage: StorageType, // File or Memory + pub num_replicas: usize, + pub no_ack: bool, // Don't require ack + pub discard: DiscardPolicy, // Old or New + pub duplicate_window: Duration, + pub allow_rollup_hdrs: bool, + pub allow_direct: bool, + pub mirror: Option, + pub sources: Vec, + pub sealed: bool, + pub compression: Option, // server_2_10+ + pub first_sequence: Option, // server_2_11+ + pub subject_transform: Option, // server_2_12+ + pub metadata: Option>, // server_2_10+ + pub placement: Option, + pub republish: Option, +} +``` + +### Stream Operations + +Via `Context`: + +| Method | API Subject | Description | +|--------|------------|-------------| +| `create_stream(config)` | `STREAM.CREATE.` | Create a new stream | +| `get_stream(name)` | `STREAM.INFO.` | Get existing stream | +| `get_or_create_stream(config)` | `STREAM.INFO` → `STREAM.CREATE` | Get or create | +| `delete_stream(name)` | `STREAM.DELETE.` | Delete a stream | +| `update_stream(name, config)` | `STREAM.UPDATE.` | Update stream config | +| `purge_stream(name)` | `STREAM.PURGE.` | Purge all messages | +| `streams()` | `STREAM.LIST` | List all streams (paged iterator) | +| `stream_names()` | `STREAM.NAMES` | List stream names (paged iterator) | +| `account_info()` | `ACCOUNT.INFO` | Get account info | + +Via `Stream`: + +| Method | API Subject | Description | +|--------|------------|-------------| +| `info()` | `STREAM.INFO.` | Refresh stream info | +| `purge()` | `STREAM.PURGE.` | Purge messages | +| `delete()` | `STREAM.DELETE.` | Delete this stream | +| `update(config)` | `STREAM.UPDATE.` | Update config | +| `get_raw_message(seq)` | `STREAM.MSG.GET.` | Get message by sequence (stored mode) | +| `get_last_message(subject)` | `STREAM.MSG.GET.` | Get last message for subject (stored mode) | +| `direct_get_last(subject)` | `DIRECT.GET.` | Direct get last (bypasses RAA) | +| `direct_get(seq)` | `DIRECT.GET.` | Direct get by sequence | +| `delete_message(seq)` | `STREAM.MSG.DELETE.` | Delete a specific message | +| `create_consumer(config)` | `CONSUMER.CREATE.` | Create consumer | +| `get_or_create_consumer(name, config)` | `CONSUMER.DURABLE.CREATE..` | Get or create durable | +| `get_consumer(name)` | `CONSUMER.INFO..` | Get existing consumer | + +### Stream Info + +```rust +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct Info { + pub config: Config, + pub created: DateTime, + pub state: State, // Messages, bytes, first/last sequence, consumer count + pub cluster: Option, + pub timestamp: DateTime, + pub leader: Option, + pub subjects: Option>, // Subject → message count +} +``` + +### Paged List Operations + +Stream and consumer listing uses a paged iterator pattern: + +```rust +// streams() returns an iterator that automatically pages +let mut streams = jetstream.streams(); +while let Some(stream) = streams.next().await { + let stream = stream?; + // process stream +} + +// stream_names() similarly pages +let mut names = jetstream.stream_names(); +while let Some(name) = names.next().await { + println!("{}", name?); +} +``` + +The paged iterator sends an initial request with `offset: 0` and continues fetching pages until no more results are returned. + +## Publishing + +**Location**: `jetstream/context.rs`, `jetstream/publish.rs` + +### Publish + +```rust +// Basic publish (fire-and-forget) +jetstream.publish("events.data", "payload".into()).await?; + +// Publish with custom message builder +jetstream.publish_message( + jetstream::message::PublishMessage::build() + .payload("data".into()) + .message_id("unique-id") // Nats-Msg-Id header for dedup + .expected_last_message_id("prev") // Nats-Expected-Last-Msg-Id + .expected_last_sequence(42) // Nats-Expected-Last-Sequence + .expected_last_subject_sequence("events", 10) // Per-subject sequence + .header("Custom", "Value") +).await?; +``` + +### PublishAck + +When a message is published to a JetStream stream, the server responds with a `PublishAck`: + +```rust +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)] +pub struct PublishAck { + pub stream: String, + pub sequence: u64, + pub domain: Option, + pub duplicate: bool, +} +``` + +### PublishAckFuture + +Publishing returns a `PublishAckFuture` that resolves to `PublishAck`. The future uses a semaphore (`max_ack_semaphore`) to limit in-flight ack waits and prevent backpressure issues. + +When `backpressure_on_inflight` is enabled, the publish operation blocks if there are too many pending acks, preventing the command channel from filling up with unbounded publish operations. + +### Idempotent Publishing + +Headers for exactly-once semantics: + +| Header | Purpose | +|--------|---------| +| `Nats-Msg-Id` | Message ID for deduplication within the stream's duplicate window | +| `Nats-Expected-Last-Msg-Id` | Expected last message ID (conditional publish) | +| `Nats-Expected-Last-Sequence` | Expected last sequence number | +| `Nats-Expected-Last-Subject-Sequence` | Expected last sequence for a specific subject | + +## Consumers + +**Location**: `jetstream/consumer/` + +### Consumer Types + +| Type | Description | +|------|-------------| +| `PullConsumer` | Client pulls messages on demand | +| `PushConsumer` | Server pushes messages to a delivery subject | +| `OrderedConsumer` | Push consumer with automatic re-creation on failure | + +### Consumer Config + +```rust +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)] +pub struct Config { + pub name: Option, + pub durable_name: Option, + pub description: Option, + pub deliver_subject: Option, // Push consumers only + pub ack_policy: AckPolicy, + pub ack_wait: Duration, + pub max_deliver: i64, + pub max_ack_pending: i32, + pub max_waiting: i32, // Pull consumers only + pub filter_subject: Option, + pub replay_policy: ReplayPolicy, + pub sample_frequency: Option, + pub max_batch: i32, // Pull consumers + pub max_expires: Duration, // Pull consumers + pub inactive_threshold: Duration, + pub flow_control: bool, // Push consumers + pub heartbeat: Option, // Push consumers + pub backoff: Vec, + pub deliver_group: Option, + pub num_replicas: usize, + pub mem_storage: bool, + pub metadata: Option>, + pub ack_markers: Option>, // server_2_12+ +} +``` + +### Pull Consumer + +**Location**: `jetstream/consumer/pull.rs` + +Pull consumers require explicit requests for messages: + +```rust +// Batch request +let mut messages = consumer.messages().await?.take(100); +while let Some(message) = messages.next().await { + let message = message?; + message.ack().await?; +} + +// Sequence-based batch +let mut batches = consumer.sequence(50)?.take(10); +while let Some(mut batch) = batches.try_next().await? { + while let Some(Ok(message)) = batch.next().await { + message.ack().await?; + } +} + +// Single message fetch +let message = consumer.fetch().await?; +``` + +Pull requests are sent to: `$JS.API.CONSUMER.MSG.NEXT..` + +The request payload is JSON: +```json +{"batch": 10, "expires": 5000, "no_wait": false} +``` + +### Push Consumer + +**Location**: `jetstream/consumer/push.rs` + +Push consumers receive messages automatically on a delivery subject. The client subscribes to the delivery subject and processes messages as they arrive. + +Features: +- **Flow control** — server sends flow control messages, client responds to maintain delivery rate +- **Heartbeats** — idle heartbeats (status code 100) when no messages are available +- **Ordered consumers** — automatically recreated on delivery failures with correct sequence positioning + +### Acknowledgment + +**Location**: `jetstream/message.rs` + +JetStream messages support multiple acknowledgment types: + +```rust +pub enum AckKind { + Ack, // Ack (message processed) + Nack, // Nak (re-deliver) + Progress, // Progress (still working) + Next, // Next (ack + pull next) + Term, // Term (don't redeliver, remove from stream) + All, // Ack all messages up to this sequence +} +``` + +Methods on JetStream `Message`: +- `ack()` — simple acknowledgment +- `ack_with(kind)` — acknowledgment with specific type +- `double_ack()` — exactly-once ack (ACK + separate ack message) +- `nack()` — negative acknowledgment (request redelivery) +- `in_progress()` — progress indicator +- `term()` — terminate message (no redelivery) + +## JetStream Message + +**Location**: `jetstream/message.rs` + +JetStream messages wrap core `Message` with metadata extracted from headers: + +```rust +#[derive(Debug)] +pub struct Message { + pub message: crate::Message, // The underlying NATS message + pub context: Context, // JetStream context for acking + pub ack_pending: Arc, // Pending ack counter +} + +impl Message { + pub fn info(&self) -> Result // Parse message info from headers + pub async fn ack(&self) -> Result<(), AckError> + pub async fn ack_with(&self, kind: AckKind) -> Result<(), AckError> + pub async fn double_ack(&self) -> Result<(), AckError> + pub async fn nack(&self) -> Result<(), AckError> + pub async fn in_progress(&self) -> Result<(), AckError> + pub async fn term(&self) -> Result<(), AckError> +} +``` + +Message info is extracted from the `HMSG` headers: +- `Nats-Stream` — stream name +- `Nats-Consumer` — consumer name +- `Nats-Delivered` — delivery count +- `Nats-Sequence` — stream sequence +- `Nats-Time-Stamp` — timestamp +- `Nats-Subject` — original subject +- `Nats-Pending-Messages` / `Nats-Pending-Bytes` — pending counts + +## Key-Value Store + +**Location**: `jetstream/kv/` + +The KV store is a JetStream-based key-value API. Each bucket maps to a JetStream stream with specific configuration: + +```rust +// Create a KV store +let kv = jetstream + .create_key_value(async_nats::jetstream::kv::Config { + bucket: "my_bucket".to_string(), + history: 5, // Max history per key (1-64) + ttl: Duration::from_secs(3600), // Key TTL + max_bytes: 1024 * 1024, // Max bucket size + storage: StorageType::File, + replicas: 1, + ..Default::default() + }) + .await?; +``` + +Under the hood: +- Each key is stored as a message with subject `$KV..` +- Keys support wildcard patterns (`$KV.bucket.prefix.*`) +- History is managed via stream `max_messages_per_subject` +- TTL is managed via stream `max_age` +- `put(key, value)` publishes to the key subject +- `get(key)` reads the last message for the key subject +- `delete(key)` publishes an internal delete marker +- `purge(key)` uses stream purge API +- `watch()` subscribes to key changes and returns a `Watch` stream +- `keys()` / `history(key)` list keys and history + +## Object Store + +**Location**: `jetstream/object_store/` + +The Object Store provides large object storage built on JetStream. Objects are chunked and stored across multiple messages in a stream. + +```rust +// Create an object store +let store = jetstream + .create_object_store(async_nats::jetstream::object_store::Config { + bucket: "my_objects".to_string(), + ..Default::default() + }) + .await?; + +// Put an object +let info = store.put("file.txt", stream).await?; + +// Get an object +let mut object_stream = store.get("file.txt").await?; +``` + +Under the hood: +- Objects are chunked into ~128KB messages +- Metadata (object info) is stored as the first "chunk 0" message +- Each chunk is a message with subject `$OBJ...C` +- Metadata includes: name, description, headers, size, chunks, digest (SHA-256) +- `get()` returns a stream of chunks +- Links allow referencing one object from another (like symlinks) + +## JetStream Error Codes + +**Location**: `jetstream/errors.rs` + +Standard JetStream error codes returned by the server: + +| Code | Constant | Description | +|------|----------|-------------| +| 10001 | `NOT_FOUND` | Resource not found | +| 10002 | `STREAM_NOT_FOUND` | Stream not found | +| 10003 | `CONSUMER_NOT_FOUND` | Consumer not found | +| 10004 | `REQUEST_NOT_FOUND` | Request not found | +| 10005 | `STREAM_WRONG_LAST_SEQ` | Wrong last sequence | +| 10006 | `STREAM_NAME_EXISTS` | Stream already exists | +| 10007 | `CONSUMER_NAME_EXISTS` | Consumer already exists | +| 10008 | `INSUFFICIENT_RESOURCES` | Insufficient resources | +| 10009 | `NO_MESSAGE_FOUND` | No message found | +| 10013 | `CONSUMER_EXISTS` | Consumer already exists (duplicate) | +| 10014 | `STREAM_NOT_CONFIGURED` | Stream not configured | +| 10015 | `CLUSTER_NOT_ACTIVE` | Cluster not active | +| 10016 | `CLUSTER_NOT_LEADER` | Not the cluster leader | +| 10017 | `CLUSTER_NOT_ENOUGH_PEERS` | Not enough peers | +| 10018 | `CLUSTER_INCOMPLETE` | Cluster incomplete | +| 10019 | `CONSUMER_DELETED` | Consumer was deleted | +| 10020 | `CONSUMER_BAD_ACK` | Bad acknowledgment | +| 10021 | `CONSUMER_BAD_SUBJECT` | Bad consumer subject | +| 10022 | `CONSUMER_DELETED_DRIFT` | Consumer deleted due to drift | +| ... | ... | Additional codes | + +## Account + +**Location**: `jetstream/account.rs` + +The `Account` struct provides information about the JetStream account: + +```rust +pub struct Account { + pub memory: i64, + pub storage: i64, + pub streams: i64, + pub consumers: i64, + pub limits: AccountLimits, +} +``` \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/07-authentication-and-security.md b/docs/research/references/nats.rs/nats-server/07-authentication-and-security.md new file mode 100644 index 0000000..c6fec23 --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/07-authentication-and-security.md @@ -0,0 +1,292 @@ +# Authentication and Security + +This document covers the authentication mechanisms, TLS configuration, and security-related features of the async-nats client. + +## Authentication Methods + +The NATS server supports multiple authentication methods. The client implements all of them. + +### 1. Username/Password + +The simplest authentication method. + +```rust +// Via ConnectOptions +let client = ConnectOptions::with_user_and_password("user".into(), "pass".into()) + .connect("nats://localhost") + .await?; + +// Via URL +let client = connect("nats://user:pass@localhost:4222").await?; +``` + +These credentials are sent in the `CONNECT` message as `user` and `pass` fields. + +### 2. Token Authentication + +A single token used for authentication. + +```rust +let client = ConnectOptions::with_token("my-token".into()) + .connect("nats://localhost") + .await?; +``` + +Token is sent in the `CONNECT` message as `auth_token` field. + +### 3. NKey Authentication + +NKey-based authentication using Ed25519 key pairs. Requires the `nkeys` feature. + +```rust +let seed = "SUANQDPB2RUOE4ETUA26CNX7FUKE5ZZKFCQIIW63OX225F2CO7UEXTM7ZY"; +let client = ConnectOptions::with_nkey(seed.into()) + .connect("nats://localhost") + .await?; +``` + +Flow: +1. Server sends `INFO` with a `nonce` field +2. Client creates a `KeyPair` from the seed +3. Client signs the nonce: `key_pair.sign(nonce.as_bytes())` +4. Client sends `CONNECT` with `nkey` (public key) and `sig` (Base64URL-encoded signature) +5. Server verifies the signature against the public key and nonce + +### 4. JWT Authentication + +User JWT with a signing callback. Requires the `nkeys` feature. + +```rust +let key_pair = Arc::new(nkeys::KeyPair::from_seed(seed)?); +let jwt = load_jwt().await?; + +let client = ConnectOptions::with_jwt(jwt, move |nonce| { + let key_pair = key_pair.clone(); + async move { key_pair.sign(&nonce).map_err(AuthError::new) } +}) +.connect("nats://localhost") +.await?; +``` + +Flow: +1. Server sends `INFO` with a `nonce` field +2. Client sends `CONNECT` with `jwt` (user JWT) and `sig` (Base64URL-encoded nonce signature) +3. The signing callback is async, allowing integration with external signing services (e.g., HSM) + +### 5. Credentials File + +Combines JWT and NKey from a `.creds` file. Requires the `nkeys` feature. + +```rust +// From file +let client = ConnectOptions::with_credentials_file("path/to/my.creds") + .await? + .connect("nats://localhost") + .await?; + +// From string +let client = ConnectOptions::with_credentials(creds_string) + .connect("nats://localhost") + .await?; +``` + +Credentials file format: +``` +-----BEGIN NATS USER JWT----- +eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5... +------END NATS USER JWT------ + +************************* IMPORTANT ************************* +NKEY Seed printed below can be used sign and prove identity. + +-----BEGIN USER NKEY SEED----- +SUAIO3FHUX5PNV2LQIIP7TZ3N4L7TX3W53MQGEIVYFIGA635OZCKEYHFLM +------END USER NKEY SEED------ +``` + +**Location**: `auth_utils.rs` handles parsing: +- `load_creds(path)` — async file read + parse +- `parse_jwt_and_key_from_creds(creds)` — extracts JWT and KeyPair from the string + +### 6. Auth Callback + +A custom async callback that receives the server nonce and returns an `Auth` struct. This is the most flexible mechanism. + +```rust +let client = ConnectOptions::with_auth_callback(move |nonce| { + async move { + let mut auth = Auth::new(); + auth.username = Some("user".to_string()); + auth.password = Some("pass".to_string()); + // Can also set jwt, nkey, signature, token + Ok(auth) + } +}) +.connect("nats://localhost") +.await?; +``` + +The callback is invoked on each connection/reconnection, allowing dynamic credential refresh (e.g., refreshing JWTs from an auth server). + +### 7. URL-Embedded Credentials + +```rust +// Username and password in URL +let client = connect("nats://user:pass@localhost:4222").await?; + +// Token in URL (username field) +let client = connect("nats://token@localhost:4222").await?; +``` + +## Auth Struct + +**Location**: `auth.rs` + +The `Auth` struct is a container for all authentication methods. Multiple fields can be set simultaneously: + +```rust +#[derive(Clone, Default)] +pub struct Auth { + pub jwt: Option, + pub nkey: Option, + pub signature_callback: Option>>, + pub signature: Option>, + pub username: Option, + pub password: Option, + pub token: Option, +} +``` + +Priority in `Connector::try_connect_to()`: +1. Auth callback overrides all other methods +2. NKey authentication (if `auth.nkey` is set) +3. JWT authentication (if `auth.jwt` is set) +4. Username/password/token from `Auth` struct +5. Username/password from URL + +## TLS Configuration + +### TLS Modes + +| Mode | When | Description | +|------|------|-------------| +| None | Default | Plaintext connection | +| Standard | `tls_required` or server requires | TLS after INFO | +| TLS First | `tls_first` option | TLS before INFO | +| WebSocket | `wss://` URL | TLS handled by WebSocket library | + +### TLS Setup + +**Location**: `tls.rs` + +The `config_tls()` function builds a `rustls::ClientConfig`: + +1. Create `RootCertStore` and load native system certificates +2. Add custom root certificates from configured PEM files +3. Build `ClientConfig` with the chosen crypto provider: + - `ring` (default) + - `aws-lc-rs` + - `fips` (aws-lc-rs in FIPS mode) +4. If client certificate + key are configured, add them for mTLS +5. If a custom `rustls::ClientConfig` was provided, use it directly + +### TLS First + +```rust +let client = ConnectOptions::new() + .tls_first() + .connect("nats://localhost") + .await?; +``` + +This sets both `tls_first = true` and `tls_required = true`. The client performs TLS handshake before reading the `INFO` message. The server must have `handshake_first: true` in its configuration. + +### Custom TLS Configuration + +```rust +let tls_client = rustls::ClientConfig::builder() + .with_root_certificates(root_store) + .with_no_client_auth(); + +let client = ConnectOptions::new() + .require_tls(true) + .tls_client_config(tls_client) + .connect("nats://localhost") + .await?; +``` + +### mTLS (Mutual TLS) + +```rust +let client = ConnectOptions::new() + .add_root_certificates("ca.pem".into()) + .add_client_certificate("cert.pem".into(), "key.pem".into()) + .connect("tls://localhost") + .await?; +``` + +## WebSocket Transport + +Requires the `websockets` feature. Supports `ws://` and `wss://` schemes. + +```rust +let client = connect("ws://localhost:8080").await?; +let client = connect("wss://localhost:443").await?; +``` + +Implementation uses `tokio-websockets` with a `WebSocketAdapter` that wraps the WebSocket stream to implement `AsyncRead + AsyncWrite`: + +```rust +// WebSocketAdapter bridges WebSocket messages to byte streams +pub(crate) struct WebSocketAdapter { + pub(crate) inner: WebSocketStream, + pub(crate) read_buf: BytesMut, // Buffered incoming WebSocket messages +} +``` + +For `wss://`, TLS is configured within the WebSocket connector, not via the client's TLS layer. + +## Security Considerations + +### Nonce Signing + +The server's `nonce` in the `INFO` message prevents replay attacks: +- Each connection gets a unique nonce +- The nonce must be signed with the client's private key +- The signature is verified server-side against the public key + +### Authorization Violations + +When the server sends `-ERR 'authorization violation'`: +- The client parses this as `ServerError::AuthorizationViolation` +- The `Connector` immediately propagates this error (does not retry) +- The error is converted to `ConnectErrorKind::AuthorizationViolation` + +### Subject Validation + +By default, the client validates subjects for protocol safety: +- **Publish subjects**: checked for emptiness and whitespace (can be disabled with `skip_subject_validation`) +- **Subscribe subjects**: always checked for emptiness, whitespace, leading/trailing dots, consecutive dots +- **Queue group names**: checked for emptiness and whitespace + +The server enforces its own validation, but client-side checks prevent protocol-framing errors. + +### Max Payload Size + +The client checks payload size against the server's `max_payload` before publishing: +- For plain messages: `payload.len() > max_payload` +- For messages with headers: `headers.wire_len() + payload.len() > max_payload` +- Returns `PublishErrorKind::MaxPayloadExceeded` if exceeded + +### No Echo + +When `no_echo` is set, the `CONNECT` message includes `echo: false`. The server will not deliver messages published by this connection back to its own subscriptions. This prevents feedback loops. + +### Lame Duck Mode + +When the server enters lame duck mode (draining for shutdown): +1. Server sends `INFO` with `ldm: true` +2. Client emits `Event::LameDuckMode` +3. Application should gracefully close or reconnect to another server + +The `nats-server` test harness provides `set_lame_duck_mode(server)` for testing this behavior. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/08-test-harness.md b/docs/research/references/nats.rs/nats-server/08-test-harness.md new file mode 100644 index 0000000..455c9eb --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/08-test-harness.md @@ -0,0 +1,347 @@ +# nats-server Test Harness + +This document covers the `nats-server` crate — a test harness for spawning real NATS server instances in integration tests. + +**Location**: `nats-server/src/lib.rs` +**Version**: 0.1.0 +**License**: Apache-2.0 +**Dependencies**: `lazy_static`, `regex`, `serde_json`, `nuid`, `rand`, `tokio-retry` + +## What It Is + +The `nats-server` crate is **not** a NATS server implementation. It is a thin test harness that: +- Spawns the Go-based `nats-server` binary as a child process +- Configures it for test use (dynamic ports, temp storage, log files) +- Discovers the client URL from the server's `INFO` protocol message +- Cleans up resources (JetStream storage, logs, PID files) on `Drop` +- Supports single servers and 3-node clusters + +The actual NATS server must be installed separately (Go binary from `github.com/nats-io/nats-server`). + +## Server Struct + +```rust +pub struct Server { + inner: Inner, +} + +struct Inner { + cfg: String, // Config file path + id: String, // Unique server ID (NUID) + port: Option, // Explicit port (None = dynamic) + child: Child, // Child process handle + logfile: PathBuf, // Log file path in temp dir + pidfile: PathBuf, // PID file path in temp dir +} +``` + +## Public API + +### run_server + +```rust +pub fn run_server(cfg: &str) -> Server +``` + +Starts a single NATS server with optional config file. + +- Uses dynamic port (`-1` flag) for parallel test execution +- Stores JetStream data in temp directory +- Writes logs to temp file: `nats-server-.log` +- Writes PID to temp file: `nats-server-.pid` +- If `cfg` is non-empty, passes `-c ` to the server + +Example: +```rust +let server = nats_server::run_server("tests/configs/jetstream.conf"); +let client = async_nats::connect(server.client_url()).await.unwrap(); +``` + +### run_basic_server + +```rust +pub fn run_basic_server() -> Server +``` + +Starts a server with no config (bare minimum). Equivalent to `run_server("")`. + +### run_server_with_port + +```rust +pub fn run_server_with_port(cfg: &str, port: Option<&str>) -> Server +``` + +Starts a server with an explicit port. If `None`, uses dynamic port. + +### run_cluster + +```rust +pub fn run_cluster<'a, C: IntoConfig<'a>>(cfg: C) -> Cluster +``` + +Starts a 3-node cluster with the given config. + +- Allocates 3 random port ranges (base, base+100, base+200) +- Configures cluster routes between nodes +- Each node gets: `--cluster nats://127.0.0.1:`, `--routes `, `--cluster_name cluster`, `-n nodeN` +- Waits 2 seconds for cluster formation and leader election + +The `IntoConfig` trait allows passing either a single config string (applied to all 3 nodes) or an array of 3 configs (one per node): + +```rust +// Same config for all nodes +let cluster = run_cluster("configs/jetstream.conf"); + +// Different configs per node +let cluster = run_cluster(["node1.conf", "node2.conf", "node3.conf"]); +``` + +### Cluster Struct + +```rust +pub struct Cluster { + pub servers: Vec, +} + +impl Cluster { + pub fn client_url(&self) -> String { + self.servers[0].client_url() + } +} +``` + +### Server Methods + +```rust +impl Server { + pub fn restart(&mut self) + pub fn client_url(&self) -> String + pub fn client_port(&self) -> u16 + pub fn client_url_with(&self, user: &str, pass: &str) -> String + pub fn client_url_with_token(&self, token: &str) -> String + pub fn client_pid(&self) -> usize +} +``` + +#### restart() + +Kills the current server process, waits for it to exit, then restarts with the same config, port, and ID. Used for testing reconnection behavior. + +#### client_url() + +Connects to the server's TCP port, reads the `INFO` line, parses the JSON, and constructs a URL: +- `nats://localhost:` for non-TLS +- `tls://localhost:` for TLS-required servers + +Polls the log file (up to 10 seconds) to discover the client address, since the port may be dynamically allocated. + +#### client_pid() + +Reads the PID file and returns the server process ID. Used for sending signals. + +### set_lame_duck_mode + +```rust +pub fn set_lame_duck_mode(s: &Server) +``` + +Sends the lame duck mode signal to the server: + +```bash +nats-server --signal ldm= +``` + +### is_port_available + +```rust +pub fn is_port_available(port: usize) -> bool +``` + +Tests if a TCP port is available by attempting to bind to it. + +## Server Lifecycle + +### Spawning + +The `do_run` function constructs and spawns the server process: + +```rust +fn do_run(cfg: &str, port: Option<&str>, id: Option) -> Inner { + let id = id.unwrap_or_else(|| nuid::next().to_string()); + let logfile = env::temp_dir().join(format!("nats-server-{id}.log")); + let pidfile = env::temp_dir().join(format!("nats-server-{id}.pid")); + let store_dir = env::temp_dir().join(format!("store-dir-{id}")); + + let mut cmd = Command::new("nats-server"); + cmd.arg("--store_dir").arg(store_dir.as_path()) + .arg("-p"); + + match port { + Some(port) => cmd.arg(port), + None => cmd.arg("-1"), // Dynamic port + }; + + cmd.arg("-l").arg(logfile.as_os_str()) + .arg("-P").arg(pidfile.as_os_str()); + + if !cfg.is_empty() { + cmd.arg("-c").arg(cfg); + } + + let child = cmd.spawn().unwrap(); + // ... +} +``` + +Key flags: +- `--store_dir` — JetStream storage directory in temp +- `-p -1` — Dynamic port allocation (or explicit port) +- `-l` — Log file path +- `-P` — PID file path +- `-c` — Config file path + +### Cleanup (Drop) + +```rust +impl Drop for Server { + fn drop(&mut self) { + self.inner.child.kill().unwrap(); + self.inner.child.wait().unwrap(); + + if let Ok(log) = fs::read_to_string(self.inner.logfile.as_os_str()) { + // Clean up JetStream storage directory if found in log + if let Some(caps) = SD_RE.captures(&log) { + let sd = caps.get(1).map_or("", |m| m.as_str()); + fs::remove_dir_all(sd).ok(); + } + // Remove log file + fs::remove_file(self.inner.logfile.as_os_str()).ok(); + } + } +} +``` + +The regex `SD_RE` matches the "Store Directory" line in the server log: +``` +.+\sStore Directory:\s+"([^"]+)" +``` + +### Client URL Discovery + +The `client_addr` method polls the log file to find the server's listen address: + +```rust +fn client_addr(&self) -> String { + for _ in 0..100 { // 100 iterations × 500ms = 50s max + match fs::read_to_string(self.inner.logfile.as_os_str()) { + Ok(l) => { + if let Some(cre) = CLIENT_RE.captures(&l) { + return cre.get(1).unwrap().as_str() + .replace("0.0.0.0", "localhost"); + } else { + thread::sleep(Duration::from_millis(500)); + } + } + _ => thread::sleep(Duration::from_millis(500)), + } + } + panic!("no client addr info"); +} +``` + +The regex `CLIENT_RE` matches: +``` +.+\sclient connections on\s+(\S+) +``` + +After finding the address, `client_url()` connects to it and parses the `INFO` JSON to get the port and TLS requirements. + +## Cluster Setup + +The `run_cluster_node_with_port` function spawns a single cluster node: + +```rust +fn run_cluster_node_with_port( + cfg: &str, + port: Option<&str>, + routes: Vec, + name: String, + cluster_name: String, + cluster: usize, +) -> Server +``` + +Additional flags for cluster nodes: +- `--routes nats://127.0.0.1:,nats://127.0.0.1:` — routes to other cluster members +- `--cluster nats://127.0.0.1:` — cluster listen address +- `--cluster_name ` — cluster name for grouping +- `-n ` — server name + +Port allocation for a cluster: +``` +Base port: random in 3000..50000 +Node 1: client_port=base, cluster_port=base+1 +Node 2: client_port=base+100, cluster_port=base+101 +Node 3: client_port=base+200, cluster_port=base+201 +``` + +Each port is checked for availability with `is_port_available()`, including the +1 cluster port. + +## JetStream Config + +**Location**: `configs/jetstream.conf` + +```conf +jetstream: { + strict: true, + max_mem_store: 8MiB, + max_file_store: 10GiB +} +``` + +This is the default test config for JetStream-enabled servers. It enables strict mode and sets memory/file storage limits suitable for testing. + +## Test Usage Patterns + +```rust +#[tokio::test] +async fn basic_test() { + let server = nats_server::run_server("configs/jetstream.conf"); + let client = async_nats::connect(server.client_url()).await.unwrap(); + // ... test logic ... + // Server cleaned up on drop +} + +#[tokio::test] +async fn cluster_test() { + let cluster = nats_server::run_cluster("configs/jetstream.conf"); + let client = async_nats::connect(cluster.client_url()).await.unwrap(); + // ... test logic ... +} + +#[tokio::test] +async fn reconnect_test() { + let mut server = nats_server::run_server(""); + let client = async_nats::connect(server.client_url()).await.unwrap(); + + // Restart the server to test reconnection + server.restart(); + + // Client should reconnect automatically + client.publish("test", "data".into()).await.unwrap(); +} +``` + +## Dependencies + +| Dependency | Version | Purpose | +|-----------|---------|---------| +| `lazy_static` | 1.4.0 | Static regex initialization | +| `regex` | 1.7.1 | Log parsing (store directory, client address) | +| `url` | 2 | URL manipulation for client_url_with | +| `serde_json` | 1.0.104 | INFO JSON parsing | +| `nuid` | 0.5 | Unique server ID generation | +| `rand` | 0.10.1 | Random port selection | +| `tokio-retry` | 0.3.0 | Exponential backoff for cluster operations | + +Note: `async-nats` is only a dev-dependency, used in the crate's own integration tests. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/09-service-api-and-abstractions.md b/docs/research/references/nats.rs/nats-server/09-service-api-and-abstractions.md new file mode 100644 index 0000000..427359e --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/09-service-api-and-abstractions.md @@ -0,0 +1,307 @@ +# Service API and Higher-Level Abstractions + +This document covers the Service API and other higher-level abstractions built on top of the core NATS client. + +## Service API + +**Location**: `service/` (feature: `service`) + +The Service API provides a framework for building NATS-based microservices with built-in monitoring, health checks, and statistics. + +### Service + +```rust +#[derive(Debug)] +pub struct Service { + client: Client, + info: Info, + endpoints: HashMap, + started: DateTime, + stats_handler: Arc serde_json::Value + Send + Sync>, + stop_sender: mpsc::Sender<()>, + stop_receiver: Option>, +} +``` + +### Creating a Service + +```rust +use async_nats::service::ServiceExt; + +let mut service = client + .service_builder() + .description("Product service") + .stats_handler(|endpoint, stats| { + serde_json::json!({ + "endpoint": endpoint, + "requests": stats.num_requests, + "errors": stats.num_errors, + }) + }) + .start("products", "1.0.0") + .await?; +``` + +### ServiceBuilder + +```rust +impl ServiceBuilder { + pub fn description(mut self, description: impl Into) -> Self + pub fn stats_handler(mut self, handler: F) -> Self + pub async fn start(self, name: impl Into, version: impl Into) -> Result +} +``` + +### Endpoints + +A service exposes one or more endpoints, each handling requests on a specific subject: + +```rust +// Add an endpoint +let mut endpoint = service + .endpoint("get_product") + .await?; + +// Process requests +while let Some(request) = endpoint.next().await { + let request = request?; + // Handle the request + request.respond(serde_json::json!({ "id": 1, "name": "Widget" })).await?; +} +``` + +### Endpoint + +**Location**: `service/endpoint.rs` + +```rust +pub struct Endpoint { + subject: Subject, + queue_group: Option, + info: EndpointInfo, + stats: Stats, + subscriber: Subscriber, +} +``` + +Implements `futures::Stream` yielding `ServiceRequest` objects. + +### ServiceRequest + +```rust +pub struct ServiceRequest { + pub subject: Subject, + pub payload: Bytes, + pub headers: Option, + pub reply: Option, + pub client: Client, +} +``` + +Methods: +- `respond(payload)` — send a response to the requester +- `respond_with_headers(payload, headers)` — send a response with headers + +### Monitoring Subjects + +The Service API automatically creates monitoring endpoints: + +| Subject | Description | +|---------|-------------| +| `$SRV.PING` | Ping all services (returns service info) | +| `$SRV.PING.` | Ping specific service by name | +| `$SRV.PING..` | Ping specific service instance | +| `$SRV.INFO` | Get service info | +| `$SRV.STATS` | Get service statistics | + +### Service Info + +```rust +pub struct Info { + pub name: String, + pub id: String, + pub version: String, + pub description: String, + pub endpoints: Vec, +} +``` + +### Stats + +```rust +pub struct Stats { + pub num_requests: u64, + pub num_errors: u64, + pub last_error: Option, + pub processing_time: Duration, + pub average_processing_time: Duration, +} +``` + +## ID Generation + +**Location**: `id_generator.rs` + +The client needs unique IDs for inbox subjects and other purposes. + +### With `nuid` Feature (Default) + +Uses the NUID library for high-performance, cryptographically strong, collision-resistant IDs: + +```rust +pub(crate) fn next() -> String { + nuid::next().to_string() +} +``` + +NUID generates 22-character alphanumeric strings using a combination of a random prefix and a sequential counter. + +### Without `nuid` Feature + +Falls back to `rand`-based generation: + +```rust +pub(crate) fn next() -> String { + rng() + .sample_iter(Alphanumeric) + .take(22) + .map(char::from) + .collect() +} +``` + +Both approaches produce 22-character alphanumeric strings, but NUID is more performant and has better collision resistance. + +## Inbox Generation + +The `Client::new_inbox()` method generates globally unique inbox subjects for request-reply: + +```rust +pub fn new_inbox(&self) -> String { + format!("{}.{}", self.inbox_prefix, crate::id_generator::next()) +} +``` + +Default prefix is `_INBOX`, producing subjects like `_INBOX.UaBG3f3q5NxX3KdNcRmF2f`. + +Custom prefix via `ConnectOptions::custom_inbox_prefix()`: +```rust +let client = ConnectOptions::new() + .custom_inbox_prefix("MYAPP") + .connect("demo.nats.io") + .await?; +// Inbox subjects: MYAPP.UaBG3f3q5KdNcRmF2f +``` + +## DateTime Helpers + +**Location**: `datetime.rs` (feature: `jetstream` or `service` or `chrono`) + +Provides date/time types for JetStream and Service API timestamps: + +- Uses the `time` crate by default +- Optionally uses `chrono` via the `chrono` feature flag +- Supports RFC 3339 formatting and parsing +- `DateTime` type wraps either `time::OffsetDateTime` or `chrono::DateTime` + +## Crypto Module + +**Location**: `crypto.rs` (feature: `crypto`) + +Provides encryption/decryption support used by the Object Store for server-side encryption. + +## Subject Validation + +**Location**: `lib.rs` + +The client provides two levels of subject validation: + +### is_valid_publish_subject + +```rust +pub(crate) fn is_valid_publish_subject>(subject: T) -> bool +``` + +Checks for protocol safety only: +- Not empty +- No whitespace (space, tab, CR, LF) which would break protocol framing + +Used for publish operations. Can be disabled with `skip_subject_validation`. + +### is_valid_subject + +```rust +pub(crate) fn is_valid_subject>(subject: T) -> bool +``` + +Checks structural validity: +- Not empty +- No leading/trailing dots +- No consecutive dots (`..`) +- No whitespace + +Used for subscribe operations (always runs, matching Go/Java behavior). + +### is_valid_queue_group + +```rust +pub(crate) fn is_valid_queue_group(queue_group: &str) -> bool +``` + +Checks: +- Not empty +- No whitespace + +## JetStream Name Validation + +**Location**: `jetstream/mod.rs` + +```rust +pub(crate) fn is_valid_name(name: &str) -> bool { + !name.is_empty() + && name.bytes().all(|c| !c.is_ascii_whitespace() && c != b'.' && c != b'*' && c != b'>') +} +``` + +JetStream names (stream names, consumer names) must not contain: +- Whitespace +- Dots (`.`) — would conflict with subject delimiters +- Wildcards (`*`, `>`) — would conflict with subject wildcards + +## CallbackArg1 + +**Location**: `options.rs` + +A type-erased async callback wrapper used throughout the crate: + +```rust +pub(crate) type AsyncCallbackArg1 = + Arc Pin + Send + Sync + 'static>> + Send + Sync>; + +#[derive(Clone)] +pub(crate) struct CallbackArg1(AsyncCallbackArg1); + +impl CallbackArg1 { + pub(crate) async fn call(&self, arg: A) -> T { + (self.0.as_ref())(arg).await + } +} +``` + +Used for: +- `event_callback` — `CallbackArg1` +- `auth_callback` — `CallbackArg1, Result>` +- `reconnect_to_server_callback` — `CallbackArg1<(Vec, ServerInfo), Option>` +- `signature_callback` — `CallbackArg1>` + +## Version Compatibility Checking + +The `Client::is_server_compatible` method checks if the server version meets a minimum requirement: + +```rust +pub fn is_server_compatible(&self, major: i64, minor: i64, patch: i64) -> bool +``` + +This parses the server version string from `ServerInfo::version` using a regex and compares major/minor/patch components. Note: this checks the directly-connected server, not necessarily the JetStream leader. + +The `server_2_10`, `server_2_11`, `server_2_12`, and `server_2_14` feature flags enable version-specific API fields and methods without runtime checks. \ No newline at end of file diff --git a/docs/research/references/nats.rs/nats-server/10-quick-reference.md b/docs/research/references/nats.rs/nats-server/10-quick-reference.md new file mode 100644 index 0000000..ce22752 --- /dev/null +++ b/docs/research/references/nats.rs/nats-server/10-quick-reference.md @@ -0,0 +1,215 @@ +# Quick Reference + +## Crate Summary + +| | | +|---|---| +| **Crate** | `async-nats` | +| **Version** | 0.49.1 | +| **Edition** | 2021 | +| **MSRV** | 1.88.0 | +| **License** | Apache-2.0 | +| **Runtime** | Tokio | +| **Protocol** | NATS Client Protocol v1 (Dynamic) | +| **TLS** | rustls (ring / aws-lc-rs / fips) | +| **WebSocket** | tokio-websockets (feature-gated) | + +## Quick Start + +```rust +use async_nats::connect; +use futures_util::StreamExt; + +#[tokio::main] +async fn main() -> Result<(), async_nats::Error> { + let client = connect("demo.nats.io").await?; + + // Publish + client.publish("events.data", "hello".into()).await?; + + // Subscribe + let mut sub = client.subscribe("events.>").await?; + while let Some(msg) = sub.next().await { + println!("{:?}", msg); + } + + // Request-Response + let response = client.request("service", "input".into()).await?; + + Ok(()) +} +``` + +## Architecture at a Glance + +``` +Client (cloneable handle, mpsc::Sender) + │ + ▼ +ConnectionHandler (single Tokio task) + ├── Subscriptions HashMap + ├── Multiplexer (request-reply, SID 0) + ├── Flush Observers + └── Ping/Pong health check + │ + ▼ +Connection (protocol I/O, read/write buffers) + │ + ▼ +Connector (server pool, reconnection) + │ + ▼ +NATS Server (Go binary, TCP/TLS/WebSocket) +``` + +## Key Types + +| Type | Location | Purpose | +|------|----------|---------| +| `Client` | `client.rs` | Cloneable connection handle | +| `Subscriber` | `lib.rs` | Message stream (impl `futures::Stream`) | +| `Message` | `message.rs` | Inbound NATS message | +| `OutboundMessage` | `message.rs` | Outbound publish message | +| `Subject` | `subject.rs` | Validated subject string (backed by `Bytes`) | +| `HeaderMap` | `header.rs` | NATS message headers | +| `StatusCode` | `status.rs` | NATS protocol status codes | +| `ServerInfo` | `lib.rs` | Server INFO data | +| `ConnectInfo` | `lib.rs` | Client CONNECT data | +| `ServerAddr` | `lib.rs` | Validated server URL | +| `Auth` | `auth.rs` | Authentication credentials | +| `ConnectOptions` | `options.rs` | Connection configuration builder | +| `Event` | `lib.rs` | Connection lifecycle events | +| `State` | `connection.rs` | Connection state (Pending/Connected/Disconnected) | +| `Statistics` | `client.rs` | Atomic connection metrics | +| `Request` | `client.rs` | Request-response builder | + +## JetStream Types + +| Type | Location | Purpose | +|------|----------|---------| +| `jetstream::Context` | `jetstream/context.rs` | JetStream API entry point | +| `jetstream::stream::Stream` | `jetstream/stream.rs` | Stream management | +| `jetstream::stream::Config` | `jetstream/stream.rs` | Stream configuration | +| `jetstream::stream::Info` | `jetstream/stream.rs` | Stream info/state | +| `jetstream::consumer::PullConsumer` | `jetstream/consumer/pull.rs` | Pull-based consumer | +| `jetstream::consumer::PushConsumer` | `jetstream/consumer/push.rs` | Push-based consumer | +| `jetstream::consumer::Config` | `jetstream/consumer/mod.rs` | Consumer configuration | +| `jetstream::Message` | `jetstream/message.rs` | Message with ack methods | +| `jetstream::PublishAck` | `jetstream/publish.rs` | Publish acknowledgment | +| `jetstream::kv::Store` | `jetstream/kv/bucket.rs` | Key-Value store | +| `jetstream::object_store::ObjectStore` | `jetstream/object_store/mod.rs` | Object store | +| `jetstream::ErrorCode` | `jetstream/errors.rs` | JetStream error codes | + +## Protocol Operations + +### Client → Server (ClientOp) + +| Op | Wire Format | Purpose | +|----|-----------|---------| +| `CONNECT` | `CONNECT {json}\r\n` | Authentication and capabilities | +| `PUB` | `PUB [reply] \r\n\r\n` | Publish message | +| `HPUB` | `HPUB [reply] \r\n\r\n` | Publish with headers | +| `SUB` | `SUB [queue] \r\n` | Subscribe | +| `UNSUB` | `UNSUB [max]\r\n` | Unsubscribe | +| `PING` | `PING\r\n` | Keepalive / health check | +| `PONG` | `PONG\r\n` | Response to server PING | + +### Server → Client (ServerOp) + +| Op | Wire Format | Purpose | +|----|-----------|---------| +| `INFO` | `INFO {json}\r\n` | Server capabilities, cluster info | +| `MSG` | `MSG [reply] \r\n\r\n` | Deliver message | +| `HMSG` | `HMSG [reply] \r\n\r\n` | Message with headers | +| `+OK` | `+OK\r\n` | Success (verbose mode) | +| `-ERR` | `-ERR \r\n` | Server error | +| `PING` | `PING\r\n` | Server health check | +| `PONG` | `PONG\r\n` | Ack client PING | + +## Internal Commands (Command → ConnectionHandler) + +| Command | Purpose | +|---------|---------| +| `Publish(OutboundMessage)` | Queue message for sending | +| `Request { subject, payload, respond, headers, sender }` | Request-response via multiplexer | +| `Subscribe { sid, subject, queue_group, sender }` | Create subscription | +| `Unsubscribe { sid, max }` | Remove subscription | +| `Flush { observer }` | Wait for write buffer flush | +| `Drain { sid }` | Gracefully drain (sub or whole client) | +| `Reconnect` | Force reconnection | +| `SetServerPool { servers, result }` | Replace server pool | +| `ServerPool { result }` | Query server pool | + +## Feature Flags + +| Feature | Default | Enables | +|---------|---------|---------| +| `jetstream` | ✓ | JetStream API (streams, consumers, publish) | +| `kv` | ✓ | Key-Value store (requires jetstream) | +| `object-store` | ✓ | Object store (requires jetstream + crypto) | +| `service` | ✓ | Service API | +| `nkeys` | ✓ | NKey/JWT authentication | +| `crypto` | ✓ | Encryption support | +| `websockets` | ✓ | WebSocket transport | +| `nuid` | ✓ | NUID ID generation | +| `ring` | ✓ | Ring crypto backend | +| `aws-lc-rs` | ✗ | Alternative crypto backend | +| `fips` | ✗ | FIPS mode (requires aws-lc-rs) | +| `chrono` | ✗ | Use chrono instead of time | +| `experimental` | ✗ | Experimental features | +| `server_2_10` | ✓ | Server 2.10+ API fields | +| `server_2_11` | ✓ | Server 2.11+ API fields | +| `server_2_12` | ✓ | Server 2.12+ API fields | +| `server_2_14` | ✓ | Server 2.14+ API fields | + +## Connection Defaults + +| Parameter | Default | +|-----------|---------| +| Connection timeout | 5 seconds | +| Ping interval | 60 seconds | +| Max pending pings | 2 | +| Request timeout | 10 seconds | +| Command channel capacity | 2048 | +| Subscription capacity | 65536 | +| Read buffer capacity | 65535 | +| Inbox prefix | `_INBOX` | +| Reconnect delay | Exponential (0ms → 4s cap) | +| Max reconnects | Unlimited | +| TLS required | Auto (server-dependent) | + +## Error Hierarchy + +``` +ConnectError (ConnectErrorKind::ServerParse | Dns | Authentication | AuthorizationViolation | TimedOut | Tls | Io | MaxReconnects) +PublishError (PublishErrorKind::MaxPayloadExceeded | InvalidSubject | Send) +RequestError (RequestErrorKind::TimedOut | NoResponders | InvalidSubject | MaxPayloadExceeded | Other) +SubscribeError (SubscribeErrorKind::InvalidSubject | InvalidQueueName | Other) +FlushError (FlushErrorKind::SendError | FlushError) +``` + +## nats-server Test Harness + +| Function | Description | +|----------|-------------| +| `run_server(cfg)` | Start single server with config | +| `run_basic_server()` | Start bare server | +| `run_cluster(cfg)` | Start 3-node cluster | +| `set_lame_duck_mode(s)` | Send LDM signal | + +## JetStream API Subjects + +| Operation | Subject Pattern | +|-----------|---------------| +| Create stream | `$JS.API.STREAM.CREATE.` | +| Stream info | `$JS.API.STREAM.INFO.` | +| Update stream | `$JS.API.STREAM.UPDATE.` | +| Delete stream | `$JS.API.STREAM.DELETE.` | +| Purge stream | `$JS.API.STREAM.PURGE.` | +| List streams | `$JS.API.STREAM.LIST` | +| Create consumer | `$JS.API.CONSUMER.CREATE.` | +| Create durable | `$JS.API.CONSUMER.DURABLE.CREATE..` | +| Consumer info | `$JS.API.CONSUMER.INFO..` | +| Pull next | `$JS.API.CONSUMER.MSG.NEXT..` | +| Account info | `$JS.API.ACCOUNT.INFO` | +| Direct get | `$JS.API.DIRECT.GET.` | \ No newline at end of file