docs(research): add nats-async and nats-server deep-dive references
This commit is contained in:
@@ -0,0 +1,237 @@
|
||||
# async-nats: Key-Value Store
|
||||
|
||||
## Overview
|
||||
|
||||
The Key-Value (KV) store is an abstraction built on top of JetStream streams. Each KV bucket is backed by a JetStream stream with the naming convention `KV_<bucket_name>`. Keys are mapped to subjects under the `$KV.<bucket>.<key>` prefix.
|
||||
|
||||
The KV feature requires `kv` (which implies `jetstream`).
|
||||
|
||||
## Store Handle
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Store {
|
||||
pub name: String,
|
||||
pub stream_name: String,
|
||||
pub prefix: String, // $KV.<bucket>.
|
||||
pub put_prefix: Option<String>, // For mirrored buckets
|
||||
pub use_jetstream_prefix: bool, // Whether to prepend JS API prefix
|
||||
pub stream: Stream,
|
||||
}
|
||||
```
|
||||
|
||||
## Bucket Config
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct Config {
|
||||
pub bucket: String,
|
||||
pub description: String,
|
||||
pub max_value_size: i32,
|
||||
pub history: i64, // Max historical entries per key (1-64)
|
||||
pub max_age: Duration, // Max age of any entry
|
||||
pub max_bytes: i64, // Total bucket size limit
|
||||
pub storage: StorageType, // File or Memory
|
||||
pub num_replicas: usize,
|
||||
pub republish: Option<Republish>,
|
||||
pub mirror: Option<Source>, // Mirror another bucket
|
||||
pub sources: Option<Vec<Source>>,
|
||||
pub mirror_direct: bool,
|
||||
pub compression: bool, // server_2_10+
|
||||
pub placement: Option<Placement>,
|
||||
pub limit_markers: Option<Duration>, // server_2_11+
|
||||
}
|
||||
```
|
||||
|
||||
## Creating/Accessing Buckets
|
||||
|
||||
```rust
|
||||
// Create a new bucket
|
||||
let kv = jetstream.create_key_value(kv::Config {
|
||||
bucket: "my-bucket".to_string(),
|
||||
history: 10,
|
||||
max_age: Duration::from_secs(3600),
|
||||
..Default::default()
|
||||
}).await?;
|
||||
|
||||
// Get an existing bucket
|
||||
let kv = jetstream.get_key_value("my-bucket").await?;
|
||||
|
||||
// Create or update
|
||||
let kv = jetstream.create_or_update_key_value(kv::Config { ... }).await?;
|
||||
|
||||
// Delete a bucket
|
||||
jetstream.delete_key_value("my-bucket").await?;
|
||||
```
|
||||
|
||||
## KV Operations
|
||||
|
||||
### Put
|
||||
|
||||
```rust
|
||||
let revision: u64 = kv.put("key", "value".into()).await?;
|
||||
```
|
||||
|
||||
Publishes to `$KV.<bucket>.<key>` (or with JS prefix). The JetStream stream stores it, and the returned sequence number serves as the revision.
|
||||
|
||||
### Get
|
||||
|
||||
```rust
|
||||
let value: Option<Bytes> = kv.get("key").await?;
|
||||
```
|
||||
|
||||
Returns `None` if the key doesn't exist or was deleted/purged. Uses either direct get (if `allow_direct`) or the standard message API.
|
||||
|
||||
### Entry
|
||||
|
||||
```rust
|
||||
let entry: Option<Entry> = kv.entry("key").await?;
|
||||
let entry: Option<Entry> = kv.entry_for_revision("key", 2).await?;
|
||||
```
|
||||
|
||||
Returns full entry metadata:
|
||||
|
||||
```rust
|
||||
pub struct Entry {
|
||||
pub bucket: String,
|
||||
pub key: String,
|
||||
pub value: Bytes,
|
||||
pub revision: u64,
|
||||
pub created: DateTime,
|
||||
pub delta: u64,
|
||||
pub operation: Operation,
|
||||
pub seen_current: bool,
|
||||
}
|
||||
```
|
||||
|
||||
### Create (Put if not exists)
|
||||
|
||||
```rust
|
||||
let revision: u64 = kv.create("key", "value".into()).await?;
|
||||
```
|
||||
|
||||
Uses `update` with `expected_last_subject_sequence = 0` (create-only). If the key exists and is deleted/purged, it's re-created.
|
||||
|
||||
### Update (Conditional Put)
|
||||
|
||||
```rust
|
||||
let revision: u64 = kv.update("key", "value".into(), last_revision).await?;
|
||||
```
|
||||
|
||||
Uses the `Nats-Expected-Last-Subject-Sequence` header for optimistic concurrency control. Only succeeds if the key's current revision matches.
|
||||
|
||||
### Delete
|
||||
|
||||
```rust
|
||||
kv.delete("key").await?;
|
||||
kv.delete_expect_revision("key", Some(revision)).await?;
|
||||
```
|
||||
|
||||
Non-destructive — publishes a `DEL` marker message. The key appears deleted to `get()`, but history is preserved (up to `history` limit).
|
||||
|
||||
### Purge
|
||||
|
||||
```rust
|
||||
kv.purge("key").await?;
|
||||
kv.purge_with_ttl("key", Duration::from_secs(10)).await?;
|
||||
kv.purge_expect_revision("key", Some(revision)).await?;
|
||||
```
|
||||
|
||||
Destructive — publishes a `PURGE` marker with rollup header, removing all previous revisions of the key. Leaves a single purge entry.
|
||||
|
||||
### Watch
|
||||
|
||||
```rust
|
||||
// Watch for new changes
|
||||
let mut watch = kv.watch("key").await?;
|
||||
// Watch with initial value
|
||||
let mut watch = kv.watch_with_history("key").await?;
|
||||
// Watch from specific revision
|
||||
let mut watch = kv.watch_from_revision("key", 5).await?;
|
||||
// Watch all keys
|
||||
let mut watch = kv.watch_all().await?;
|
||||
// Watch multiple keys (server_2_10+)
|
||||
let mut watch = kv.watch_many(["foo", "bar"]).await?;
|
||||
```
|
||||
|
||||
`Watch` implements `futures_util::Stream<Item = Result<Entry, WatcherError>>`.
|
||||
|
||||
Under the hood, each watch creates an **ordered push consumer** on the KV stream with:
|
||||
- `filter_subject` matching `$KV.<bucket>.<key>`
|
||||
- `replay_policy: Instant`
|
||||
- Appropriate `deliver_policy`
|
||||
|
||||
### History
|
||||
|
||||
```rust
|
||||
let mut history = kv.history("key").await?;
|
||||
```
|
||||
|
||||
Returns a `Stream` of all past `Entry` values for a key (including deletes/purges).
|
||||
|
||||
### Keys
|
||||
|
||||
```rust
|
||||
let mut keys = kv.keys().await?;
|
||||
```
|
||||
|
||||
Returns a `Stream<String>` of all current keys. Uses a headers-only consumer with `LastPerSubject` deliver policy to efficiently scan the bucket.
|
||||
|
||||
## Entry Operations
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
|
||||
pub enum Operation {
|
||||
Put, // Value was put
|
||||
Delete, // Value was deleted (DEL marker)
|
||||
Purge, // Value was purged (PURGE marker with rollup)
|
||||
}
|
||||
```
|
||||
|
||||
The operation type is determined from the `KV-Operation` header (`PUT`, `DEL`, `PURGE`) or the `Nats-Marker-Reason` header (fallback for server-generated markers like `MaxAge`, `Purge`, `Remove`).
|
||||
|
||||
## Key and Bucket Name Validation
|
||||
|
||||
```rust
|
||||
// Bucket: alphanumeric, dash, underscore only
|
||||
VALID_BUCKET_RE: \A[a-zA-Z0-9_-]+\z
|
||||
|
||||
// Key: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots
|
||||
VALID_KEY_RE: \A[-/_=\.a-zA-Z0-9]+\z
|
||||
```
|
||||
|
||||
## Bucket Status
|
||||
|
||||
```rust
|
||||
let status: Status = kv.status().await?;
|
||||
```
|
||||
|
||||
Wraps stream info to provide bucket-level statistics (bucket name, message count, byte count, etc.).
|
||||
|
||||
## Mirrored Buckets
|
||||
|
||||
When a bucket is configured as a mirror of another (potentially in a different account/domain):
|
||||
|
||||
- `prefix` is set to `$KV.<mirror_bucket>.`
|
||||
- `put_prefix` may be set to the source bucket's API prefix for cross-domain writes
|
||||
- `use_jetstream_prefix` is adjusted based on whether the mirror is in the same domain
|
||||
|
||||
## KV → Stream Config Mapping
|
||||
|
||||
When creating a KV bucket, the `Config` is converted to a JetStream `stream::Config`:
|
||||
|
||||
| KV Config | Stream Config |
|
||||
|-----------|---------------|
|
||||
| `bucket` | `name = "KV_<bucket>"` |
|
||||
| `subjects` | `["$KV.<bucket>.>"]` |
|
||||
| `max_messages_per_subject` | `history` (max 64) |
|
||||
| `max_age` | `max_age` |
|
||||
| `max_bytes` | `max_bytes` |
|
||||
| `storage` | `storage` |
|
||||
| `num_replicas` | `num_replicas` |
|
||||
| `republish` | `republish` |
|
||||
| `mirror` | `mirror` |
|
||||
| `discard` | `DiscardPolicy::New` |
|
||||
| `allow_direct` | `true` |
|
||||
| `allow_rollup_hdrs` | `true |
|
||||
| `max_msg_size` | `max_value_size` |
|
||||
Reference in New Issue
Block a user