# NATS Client Protocol and Wire Format **Protocol**: NATS Client Protocol v1 (with dynamic reconfiguration) **Transport**: TCP (port 4222), TLS, WebSocket (ws/wss) ## Protocol Overview The NATS client-server protocol is a simple, text-based protocol with binary payload support. All operations are terminated with `\r\n`. Messages carry their payload length, allowing efficient binary data transfer. ### Connection Lifecycle ``` Client Server │ │ │◄──────────── INFO {json} ────────────────────│ Server sends INFO first │ │ │────────────── CONNECT {json} ────────────────►│ Client sends CONNECT │────────────── PING ──────────────────────────►│ Client sends PING │◄──────────── PONG ────────────────────────── │ Server confirms connection │ │ │──── SUB/UNSUB/PUB/HPUB ──────────────────────►│ Normal operation │◄─── MSG/HMSG/+OK/-ERR/PING ─────────────────│ │ │ ``` ## Server Operations (ServerOp) These are operations received from the server. The `Connection` module parses these from the read buffer. ### INFO Sent by the server upon connection and asynchronously when cluster topology changes. ``` INFO {json}\r\n ``` JSON fields (see `ServerInfo` struct): | Field | Type | Description | |-------|------|-------------| | `server_id` | String | Unique server identifier | | `server_name` | String | Generated server name | | `host` | String | Cluster host | | `port` | u16 | Cluster port | | `version` | String | Server version | | `auth_required` | bool | Authentication required | | `tls_required` | bool | TLS required | | `max_payload` | usize | Maximum payload size | | `proto` | i8 | Protocol version (0 or 1) | | `client_id` | u64 | Server-assigned client ID | | `go` | String | Go build version | | `nonce` | String | Nonce for nkey auth | | `connect_urls` | Vec | Cluster server URLs | | `client_ip` | String | Client IP as seen by server | | `headers` | bool | Server supports headers | | `ldm` | bool | Lame duck mode | | `cluster` | Option | Cluster name | | `domain` | Option | NATS domain | | `jetstream` | bool | JetStream enabled | ### MSG Delivers a message to a subscription (no headers): ``` MSG [reply-to] <#bytes>\r\n \r\n ``` ### HMSG Delivers a message with headers: ``` HMSG [reply-to] <#header-bytes> <#total-bytes>\r\n \r\n : \r\n \r\n \r\n ``` Header format follows the NATS/1.0 header spec: - First line: `NATS/1.0` optionally followed by status code and description - Subsequent lines: `name: value` headers - Empty line separates headers from payload - Header values may span multiple lines (continuation lines start with whitespace) ### PING / PONG ``` PING\r\n → Client responds with PONG PONG\r\n → Acknowledges client's PING ``` ### +OK / -ERR ``` +OK\r\n → Success acknowledgment (verbose mode) -ERR \r\n → Error from server ``` Common server errors: - `authorization violation` → parsed as `ServerError::AuthorizationViolation` - Other strings → `ServerError::Other(String)` ## Client Operations (ClientOp) These are operations sent from the client to the server. The `Connection` module serializes these to the write buffer. ### CONNECT Sent as the first client operation after receiving INFO. Contains authentication and capability information. ``` CONNECT {json}\r\n ``` JSON fields (see `ConnectInfo` struct): | Field | Type | Description | |-------|------|-------------| | `verbose` | bool | Enable +OK acknowledgments (always false in this client) | | `pedantic` | bool | Strict format checking (always false) | | `jwt` | Option | User JWT for auth | | `nkey` | Option | Public nkey for auth | | `sig` | Option | Signed nonce (Base64URL encoded) | | `name` | Option | Client name | | `echo` | bool | Whether server should echo messages back | | `lang` | String | Implementation language ("rust") | | `version` | String | Client version | | `protocol` | u8 | Protocol version (1 = dynamic) | | `tls_required` | bool | TLS required | | `user` | Option | Username | | `pass` | Option | Password | | `auth_token` | Option | Auth token | | `headers` | bool | Client supports headers (always true) | | `no_responders` | bool | Client supports no-responders (always true) | ### PUB / HPUB Publish a message: ``` PUB [reply-to] <#payload-bytes>\r\n \r\n ``` Publish with headers: ``` HPUB [reply-to] <#header-bytes> <#total-bytes>\r\n : \r\n \r\n \r\n ``` ### SUB Subscribe to a subject: ``` SUB [queue-group] \r\n ``` ### UNSUB Unsubscribe from a subscription: ``` UNSUB [max]\r\n ``` The optional `max` parameter tells the server to auto-unsubscribe after receiving the specified number of messages. ### PING / PONG ``` PING\r\n → Health check / keepalive PONG\r\n → Response to server PING ``` ## Protocol Version The `Protocol` enum has two variants: | Value | Name | Description | |-------|------|-------------| | 0 | Original | Basic protocol | | 1 | Dynamic | Supports async INFO for cluster topology changes, lame duck mode | This client always sends `protocol: 1` (Dynamic), enabling: - Asynchronous INFO messages with updated server lists - Lame duck mode notifications - Dynamic reconfiguration of cluster topology ## Wire Format Details ### Message Length Calculation For plain `MSG`: ``` length = subject.len() + reply.map_or(0, |r| r.len()) + payload.len() ``` For `HMSG`: ``` length = subject.len() + reply.map_or(0, |r| r.len()) + header_len + payload.len() ``` Where `header_len` = serialized header bytes and `total_len` = `header_len + payload.len()`. ### Write Buffer Architecture The `Connection` uses a two-tier write buffer: 1. **`flattened_writes`** (`BytesMut`) — for small writes (< 4096 bytes). Protocol headers, short commands, and small messages are flattened into this buffer for efficient sequential writing. 2. **`write_buf`** (`VecDeque`) — for large writes (>= 4096 bytes). Large payloads are appended as separate `Bytes` chunks. Supports vectored writes (`write_vectored`) when the underlying stream supports it, writing up to 64 chunks at once. The soft limit for the total write buffer is 65,535 bytes (`SOFT_WRITE_BUF_LIMIT`). When exceeded, the `ConnectionHandler` stops processing new commands until the buffer drains. ### Read Buffer Architecture The `Connection` uses a single `BytesMut` read buffer with configurable initial capacity (default 65,535 bytes). Protocol parsing uses `memchr::memmem::find` to locate CRLF delimiters efficiently. If a partial message is in the buffer, the parser returns `None` and waits for more data. ### Header Serialization Headers are serialized in NATS/1.0 format: ``` NATS/1.0\r\n Header-Name: Header-Value\r\n Multi-Line-Header: value part 1\r\n continuation of value\r\n Another-Header: another value\r\n \r\n ``` The `HeaderMap::to_bytes()` method handles this serialization, using `httparse`-compatible line folding for multi-line values. ### Status Codes in Headers NATS status codes are embedded in the `HMSG` header version line: ``` NATS/1.0 404 No Messages\r\n NATS/1.0 408 Request Timeout\r\n NATS/1.0 503 No Responders\r\n ``` Common codes used by the client: | Code | Constant | Meaning | |------|----------|---------| | 100 | `IDLE_HEARTBEAT` | JetStream idle heartbeat | | 200 | `OK` | Success | | 404 | `NOT_FOUND` | Message/stream not found | | 408 | `TIMEOUT` | Request timeout | | 409 | `REQUEST_TERMINATED` | Request terminated | | 503 | `NO_RESPONDERS` | No responders available | ## Protocol Parsing Implementation The `Connection::try_read_op()` method handles all protocol parsing: 1. Search for `\r\n` delimiter using `memchr::memmem::find` 2. Match the operation prefix: - `+OK` → `ServerOp::Ok` - `PING` → `ServerOp::Ping` - `PONG` → `ServerOp::Pong` - `-ERR` → parse error description → `ServerOp::Error` - `INFO ` → parse JSON → `ServerOp::Info` - `MSG ` → parse subject/sid/reply/length, read payload → `ServerOp::Message` - `HMSG ` → parse headers + payload → `ServerOp::Message` 3. Unknown prefix → return `io::Error` with `InvalidInput` For `MSG` and `HMSG`, if the complete payload isn't yet in the read buffer (checked via `len + payload_len + 4 > remaining`), the method returns `Ok(None)` and the buffer accumulates more data before retrying. Non-UTF8 subjects in server messages are handled gracefully — the parser returns an `io::Error` rather than panicking, which is critical because the Go server does not enforce UTF-8 in subjects (regression fix for issue #1572).