alkdev/alknet
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

greenfield: clean slate for ALPN-as-service pivot

Browse Source 
Delete old source crates (alknet-core, alknet, alknet-napi), old
architecture docs (ADRs, specs, open questions), old research docs
(phase2, event-sourcing, feasibility, etc.), old tasks, and obsolete
reference material (gitserver/MPL, honker, nats, rustfs, polyglot,
keystone, distributed-identity).

Keep: alknet-secret (standalone, compiles), pivot docs, iroh and ssh
references, rudolfs reference (MIT/Apache, fork candidate), ops docs,
sdd_process.md, and licenses.

Previous implementation preserved at /workspace/@alkdev/alknet-main/
for reference during porting.

Workspace compiles: cargo check + 14 tests pass for alknet-secret.
This commit is contained in:
glm-5.1
2026-06-15 12:08:08 +00:00
parent d003a4f4ec
commit b5a4600d74
261 changed files with 138 additions and 53794 deletions
Download Patch File Download Diff File Expand all files Collapse all files

771
docs/research/references/distributed-identity/distributed-identity-reference.md
View File

@@ -1,771 +0,0 @@
# Research: Distributed Identity, Smart Contract ACL, and Decentralized Git
> Status: Research Reference
> Created: 2026-06-08
> Scope: Decentralized git hosting, distributed identity, smart contract-based access control, and their relevance to alknet
## Table of Contents
1. [Executive Summary](#1-executive-summary)
2. [Source Concept: NFT-Based Decentralized Git](#2-source-concept-nft-based-decentralized-git)
3. [Existing Projects](#3-existing-projects)
4. [Identity on the Blockchain](#4-identity-on-the-blockchain)
5. [Access Control Models for Distributed Git](#5-access-control-models-for-distributed-git)
6. [Cryptographic Identity Mapping](#6-cryptographic-identity-mapping)
7. [Gossip Protocols for Repo Synchronization](#7-gossip-protocols-for-repo-synchronization)
8. [Relevance to Alknet](#8-relevance-to-alknet)
9. [References](#9-references)
---
## 1. Executive Summary
This document researches distributed identity systems, smart contract-based access control, and decentralized git platforms to inform alknet's architecture. The source concept — a decentralized, censorship-resistant git hosting platform using NFTs (ERC-721) for identity and smart contracts for ACL — directly inspired some of alknet's cryptographic identity and key derivation ideas. The research reveals several key findings:
**Key Findings:**
1. **Radicle is the most mature decentralized git system** and provides the closest production reference for alknet's architecture, particularly in Ed25519 identity, gossip-based replication, and self-certifying repositories. However, Radicle lacks the smart contract/on-chain ACL layer that the source concept envisions.
2. **Smart contract ACL is feasible but introduces latency trade-offs.** On-chain identity verification costs 0.5-5 seconds per look-up on L2s, making it unsuitable as a hot path. The correct pattern is on-chain registration + local cache, which aligns with alknet's `StorageIdentityProvider` approach.
3. **alknet's BIP39/SLIP-0010 key derivation already spans both worlds.** The `m/74'/0'/0'/0'` path for Ed25519 identity and `m/44'/60'/0'/0/0` for Ethereum signing means the same seed phrase that governs alknet authentication can also sign on-chain transactions — no separate wallet needed.
4. **The Identity + IdentityProvider model maps directly to decentralized identity.** `ConfigIdentityProvider` is the local-only mode (Radicle-like); `StorageIdentityProvider` is the cached mode (on-chain ACL mirrored to SQLite); a future `OnChainIdentityProvider` could verify against smart contracts.
5. **Domain events vs. integration events (from alknet's event sourcing research) is the correct pattern** for synchronizing on-chain state to local nodes. On-chain events are the source of truth; honker streams carry the projected local state.
---
## 2. Source Concept: NFT-Based Decentralized Git
The originating concept for this research is a decentralized, censorship-resistant git hosting platform built on the following principles:
### 2.1 Core Architecture
| Component | Mechanism | Purpose |
|-----------|----------|---------|
| **Org/User Identity** | Transferable ERC-721 tokens | Organizations and users are NFTs; ownership is on-chain and transferable |
| **Repository Identity** | ERC-721 tokens owned by org/user tokens | Repos are NFTs with a `mapping(address => Role)` ACL |
| **Replicators** | User/org nodes listing replicated repos + public endpoints | Decentralized hosting; replicators choose what to mirror |
| **Gossip Protocol** | Push/pull notifications about repo updates | Replicators learn about new commits from tracked repos |
| **Push Authorization** | Identity's on-chain ACL verified by replicator | No central authority can ban; replicators individually verify write privileges |
| **Funding Model** | After-the-fact Patreon-like contributions | Replicators receive donations; no paywall for access |
### 2.2 Key Design Properties
- **No central authority**: No single entity can ban an org, user, or repo
- **Individual replicator choice**: Each replicator independently decides what to replicate and whose pushes to accept
- **Transferable identity**: Selling the org NFT transfers all repos and access permissions
- **Self-certifying data**: Git content addresses + on-chain identity = verifiable data provenance
### 2.3 Critical Gaps in the Source Concept
| Gap | Issue | Solution Pattern |
|-----|-------|-----------------|
| **Hot path latency** | On-chain ACL look-up per push is too slow | Cache ACL locally; sync from chain events |
| **Key rotation** | If the private key controlling the NFT is lost, the identity is lost | Multi-delegate thresholds (like Radicle) + social recovery |
| **Fork/namespace collisions** | Multiple repos with same name under different orgs | Use on-chain IDs (token IDs) not human-readable names as the authoritative identifier |
| **Gas costs** | Every ACL change costs gas | Batch updates; use L2s (Base, Arbitrum); delegate to replicator-level local ACL |
| **Revocation propagation** | Revoking write access must propagate to all replicators | Event-driven: on-chain Revoked event → gossip notification → local ACL update |
---
## 3. Existing Projects
### 3.1 Radicle (radicle.xyz)
**Overview**: Radicle is an open-source, peer-to-peer code collaboration stack built on Git. It is the most mature decentralized git system currently in production (v1.x, Heartwood release).
#### Identity System
| Feature | Implementation |
|---------|---------------|
| **Node ID (NID)** | Ed25519 public key encoded as a DID (`did:key:z6Mk...`) |
| **Key format** | Ed25519 (same curve as alknet) |
| **Storage** | SSH-format key files; `MemorySigner` holds decrypted key in RAM |
| **Multi-device** | Currently one key per device (per RIP-0002); multi-device via threshold delegates is in development |
| **Identity Document** | JSON document stored in Git, listing delegates (DIDs) and a threshold for canonical updates |
**Relevance to alknet**: Radicle's NID system is architecturally very close to alknet's Ed25519-based identity. Both use:
- Ed25519 as the primary key type
- A single seed/identity as the root of trust
- DID-like identifiers for inter-node communication
- Cryptographic signatures for data verification
**Key difference**: Radicle uses pure Ed25519 keypairs directly (no hierarchical derivation), while alknet derives Ed25519 keys from a BIP39 seed phrase via SLIP-0010. This gives alknet the ability to derive multiple keys from a single root and to derive Ethereum signing keys from the same seed.
#### Gossip Protocol
Radicle uses a custom gossip protocol with three message types:
| Message Type | Purpose | Content |
|-------------|---------|---------|
| **Node Announcement** | Peer discovery | Node ID, alias, addresses, capabilities, timestamp |
| **Inventory Announcement** | Repo discovery | List of RepoIDs being seeded, timestamp |
| **Reference Announcement** | Repo update notification | RepoID + updated signed refs, timestamp |
Each announcement includes a cryptographic signature and timestamp, enabling verification before relay. Messages are dropped on re-encounter (epidemic-style deduplication). Bootstrap nodes seed peer discovery.
**Comparison with alknet's call protocol**: Radicle's gossip is metadata-only; actual data transfer uses Git protocol. alknet's approach uses a call protocol (`EventEnvelope`) for both metadata and operation invocation. The gossip pattern could be layered on top of alknet's call protocol as a subscription-based integration event mechanism.
#### Self-Certifying Repositories
Radicle repositories are **self-certifying**:
- The Repository ID (RID) is derived from the initial identity document hash
- All actions (commits, issue comments, patches) are cryptographically signed
- **Delegates** are public keys authorized to update the identity document
- A **threshold** defines how many delegates must sign for an update to be canonical
- Canonical branches are established dynamically based on signature thresholds
This eliminates the need for a central authority to determine "which version is correct."
**Relevance**: alknet's on-chain ACL concept (from the source) can use this threshold model. Instead of a single NFT owner dictating the canonical branch, a threshold of delegates can be required — this mirrors the `narrowed_scopes` / `DelegatesEdge` model in alknet's ACL graph.
#### Collaborative Objects (COBs)
COBs are Radicle's mechanism for distributed social artifacts (issues, patches, code review):
- Stored as Git objects in `refs/cobs/<type>/<object-id>` namespace
- Use CRDT DAG (Directed Acyclic Graph) for conflict-free merging
- All operations are Ed25519-signed by their author
- SQLite cache (`cobs.db`) provides indexed queries without traversing Git history
**Relevance**: COBs demonstrate that complex social data can be stored in Git with CRDT semantics. alknet's `alknet-storage` metagraph + honker streams could serve a similar role for distributed state, with the key difference being that alknet's state store is SQLite-backed rather than Git-backed, making it more efficient for real-time operations.
#### Summary Assessment
| Dimension | Radicle | alknet (proposed) |
|-----------|---------|-------------------|
| **Identity** | Ed25519 keypair (DID) | Ed25519 from SLIP-0010 + Ethereum key from same seed |
| **Naming** | No global naming; NID is identifier | On-chain NFT ID + human-readable name (via ENS or custom) |
| **Access Control** | Threshold delegates in identity doc | Smart contract ACL + local graph cache |
| **Replication** | Gossip for metadata, Git for data | Call protocol + (future) gossip subscriptions |
| **Data Storage** | Git objects + SQLite cache | SQLite (metagraph/honker) + Git-compatible |
| **Censorship Resistance** | P2P, no authority | P2P + on-chain identity (uncensorable registration) |
| **Funding Model** | Community-funded seed nodes | After-the-fact contributions (replicators) |
### 3.2 ForgeFed (Forgejo Federation)
**Overview**: ForgeFed is an ActivityPub-based federation protocol for software forges. It enables Gitea/Forgejo instances to interoperate — users on one instance can open issues and submit PRs on another without creating separate accounts.
| Feature | Details |
|---------|---------|
| **Protocol** | ActivityPub (same as Mastodon, PeerTube) |
| **Identity** | Web-based (user@example.com format, like email) |
| **ACL** | Per-instance ACL; no on-chain verification |
| **Censorship Resistance** | Limited; instances can block each other |
| **Status** | Forgejo implementing; Vervis is reference implementation |
**Relevance to alknet**: ForgeFed shows how federation works without blockchain. It uses ActivityPub for cross-instance communication, which is analogous to alknet's call protocol for cross-node communication. However, ForgeFed relies on instance-level trust (each Forgejo admin controls their instance), while alknet's concept uses on-chain identity for trust.
**Key takeaway**: ForgeFed's federation model is complementary, not competitive, with blockchain identity. An alknet node could expose a ForgeFed-compatible interface for interop with existing forges while using on-chain identity for internal trust decisions.
### 3.3 Git-Based Smart Contract Projects
| Project | Chain | Approach | Status |
|---------|-------|----------|--------|
| **GitBross** | Solana/Arbitrum + IPFS | Repos backed up to IPFS; smart contracts for metadata | Active |
| **GitLike** | Ethereum + IPFS | Browser-based decentralized VCS | Experimental |
| **Statik** | IPFS | Version control on IPFS with content-addressed storage | Experimental |
| **PineSU** | Ethereum | Git repos + blockchain for integrity/timestamping | Research paper |
**Common patterns**:
- IPFS for content-addressed storage of git objects
- Smart contracts for metadata (ownership, ACL, provenance)
- Ethereum or L2 for on-chain verification
- Git bridge tools that push to both IPFS and traditional remotes
**Key insight**: None of these projects have achieved widespread adoption. The main challenges are:
1. **Performance**: IPFS retrieval is slower than centralized git hosting
2. **UX**: Browser-based git clients lack feature parity with CLI tools
3. **Incentives**: No sustainable funding model for replicators
alknet's approach of using traditional git remotes with a smart contract ACL overlay avoids the IPFS performance trap while still providing censorship resistance.
### 3.4 NFT-Based Access Control Systems
Several projects use NFTs (ERC-721) for access gating:
| Pattern | Mechanism | Example |
|---------|-----------|---------|
| **Token-gated content** | Wallet verification proves NFT ownership before granting access | NFT-gated websites, Discord roles |
| **Role-based ACL via NFT** | NFTs represent roles; smart contract checks `balanceOf(address) > 0` | Token-gated DAOs, access-controlled channels |
| **Namespace NFTs** | Each NFT represents a namespace/org; sub-rights derive from ownership | ENS domains, NFT-based guild systems |
**Solidity Pattern for Repository ACL**:
```solidity
// Simplified example: NFT-based org/repo with on-chain ACL
contract OrgToken is ERC721 {
struct Org {
address owner;
mapping(address => Role) members; // ACL mapping
}
struct Repo {
uint256 orgTokenId; // Owning org
mapping(address => Permission) collaborators;
}
function canPush(uint256 repoId, address user) external view returns (bool) {
Repo storage repo = repos[repoId];
// Check direct permission
if (repo.collaborators[user] >= Permission.Write) return true;
// Check org membership
Org storage org = orgs[repo.orgTokenId];
if (org.members[user] >= Role.Member) return true;
return false;
}
}
```
**Performance considerations**: A `canPush()` check on L2 (Base, Arbitrum) costs ~0.001-0.01 USD and takes 0.5-2 seconds. This is acceptable for occasional operations (repo creation, ACL changes) but not for per-push verification. Caching is essential.
**Relevance to alknet**: The mapping from on-chain ACL to alknet's local ACL graph is direct:
- ERC-721 token ID → `PrincipalNode` in alknet's ACL metagraph
- `collaborators` mapping → `DelegatesEdge` with `narrowed_scopes`
- `canPush()` → alknet's `check_access()` function
---
## 4. Identity on the Blockchain
### 4.1 ERC-721 as Identity/Namespace Tokens
**How it works**: Each unique identity (org, user, namespace) is an ERC-721 NFT. The token ID is the on-chain identifier; metadata (display name, avatar, public key) is stored off-chain (IPFS or DNS).
**Advantages**:
- Inherent transferability (sell/gift an org identity)
- On-chain ownership verification
- Metadata can include cryptographic public keys for off-chain verification
- Composable with other on-chain protocols (DAO governance, treasury)
**Disadvantages**:
- Gas costs for every state change
- Key rotation requires a transaction (can't just change a local file)
- Metadata availability depends on off-chain storage
- Privacy: all ACL changes are public on-chain
**Resolution pattern**: Use on-chain registration as the root of trust, but resolve identity locally via cached data. This is exactly how DNS works — the zone file is authoritative, but resolvers cache it.
### 4.2 ENS (Ethereum Name Service) as a Naming Layer
**Overview**: ENS maps human-readable names (e.g., `alice.eth`) to machine-readable identifiers (Ethereum addresses, content hashes, text records).
| Feature | Implementation |
|---------|---------------|
| **Name resolution** | `alice.eth` → Ethereum address (NFT owner) |
| **Text records** | Store arbitrary key-value data (avatar, email, public key, SSH key) |
| **Subdomains** | `git.alice.eth` can point to a replicator endpoint |
| **Resolver** | Smart contract that returns records for a name |
| **Off-chain look-up** | CCIP-read (EIP-3668) allows resolving names via external data |
**Relevance to alknet**: ENS text records can store alknet node identifiers:
- `alk.id` text record → alknet Node ID (Ed25519 public key fingerprint)
- `alk.pubkey` text record → Ed25519 public key (for SSH authentication)
- `alk.replicator` text record → endpoint URL (for repo discovery)
This creates a human-friendly naming overlay on top of alknet's cryptographic identifiers. Combined with DNS TXT records (alknet's planned DNS naming layer), it provides multiple resolution paths.
**Limitation**: ENS resolution requires an Ethereum RPC call, which adds latency. For production use, ENS data should be cached locally and refreshed periodically, similar to DNS TTLs.
### 4.3 Smart Contracts as ACL/Naming Services
**Pattern**: A smart contract stores the ACL mapping and provides a view function for verification. This is the "source of truth" that local caches sync from.
```
On-chain ACL contract (source of truth)
│
│ events: RoleGranted, RoleRevoked, RepoCreated, etc.
│
▼
alknet-storage (local cache)
├── ACL metagraph (PrincipalNode + DelegatesEdge)
├── Synced from on-chain events
└── Used for hot-path access checks
```
**Event-driven sync pattern** (critical for alknet):
1. Smart contract emits `RoleGranted(address, repoId, role)` event
2. alknet head node listens to these events (via Ethereum log subscription)
3. Event is projected into the ACL metagraph as a `DelegatesEdge` with `narrowed_scopes`
4. Local access checks use the metagraph (fast, SQLite)
5. Periodic consistency check ensures local cache matches on-chain state
This maps directly to alknet's **event boundary discipline**:
- On-chain events = external source of truth (like domain events from another service)
- ACL metagraph = local projection (like an integration event or read model)
- Honker stream `acl:updated` = notification that the local cache changed (integration event)
### 4.4 Decentralized Identity Standards
#### W3C DIDs (Decentralized Identifiers)
**Overview**: DIDs are a W3C standard for verifiable, self-sovereign digital identifiers. A DID is a URI that resolves to a DID Document describing how to interact with the identity holder.
| DID Method | Resolution | Key Type | Use Case |
|-----------|-----------|----------|----------|
| `did:key` | Static (no registry) | Ed25519, secp256k1, etc. | Radicle uses this; self-certifying |
| `did:ethr` | Ethereum registry | secp256k1 | Blockchain-verifiable identity |
| `did:web` | DNS/web server | Any | Traditional web PKI bridge |
| `did:ion` | Bitcoin Sidetree | secp256k1 | Microsoft's DID system |
**Relevance**: Radicle uses `did:key` with Ed25519 keys. alknet could use `did:key` for local identity (same key type!) and extend to `did:ethr` for on-chain identity, using the same seed phrase to derive both keys.
#### Verifiable Credentials (VCs)
**Overview**: VCs are tamper-evident, cryptographically secure attestations issued by a trusted authority. Think of them as digital certificates (driver's license, degree) that the holder presents to a verifier.
**Application to git access**: A VC could attest that "this Ed25519 public key has write access to repo X." The issuer is the org's NFT contract (or a delegate). VCs can be verified off-chain, reducing on-chain transaction costs.
**alknet mapping**: VCs are analogous to alknet's `Identity` struct with `scopes` and `resources`. A VC issuance maps to the creation of a `DelegatesEdge` in the ACL graph. The key difference is that VCs are bearer tokens (anyone who holds one can present it), while alknet's ACL is graph-based (the principal must be connected to the resource via edges).
---
## 5. Access Control Models for Distributed Git
### 5.1 Git's Own ACL Model
Git has limited built-in ACL. Access control is typically enforced at the transport layer:
| Mechanism | Layer | Scope |
|-----------|-------|-------|
| **`pre-receive` hook** | Server-side | Reject pushes based on branch, author, file patterns |
| **`update` hook** | Server-side | Per-ref checks (branch-level protection) |
| **`post-receive` hook** | Server-side | Post-push actions (notifications, CI triggers) |
| **SSH key mapping** | Transport | `authorized_keys` → system user → filesystem permissions |
| **HTTP basic auth** | Transport | Username/password → Git smart HTTP |
| **Gitolite** | Server-side | Config-file-based ACL mapping SSH keys to repos and permissions |
**Gitolite pattern** (most relevant for distributed git):
- `~/.ssh/authorized_keys` maps SSH keys to Gitolite users
- `~/.gitolite/conf/gitolite.conf` defines repos and permissions
- Permission levels: `R` (read), `RW` (read+write), `RW+` (read+write+force-push)
- Wildcard repos: `CREATOR/..*` — users can create repos matching patterns
**alknet mapping**: Gitolite's config file is the analog of alknet's ACL metagraph. The key difference is that Gitolite is centralized (one config file), while alknet's ACL can be distributed (synced from on-chain events).
### 5.2 Decentralized Write Permission Without Central Authority
In a truly decentralized system, no single node controls access. Several patterns exist:
#### Pattern 1: Self-Certifying Repositories (Radicle)
- The repo creator defines an identity document listing delegates
- Delegates are Ed25519 public keys with a threshold
- Only delegate signatures on refs are considered canonical
- Replicators accept any push but only replicate refs signed by sufficient delegates
**Trade-off**: Simple, no on-chain costs, but no mechanism for human-readable names or transferable ownership.
#### Pattern 2: On-Chain ACL (Source Concept)
- Smart contract stores `mapping(address => Role)` for each repo
- Replicators verify pusher's address against the contract before accepting
- Ownership is transferable (the NFT can be sold)
- Gas costs for setup and ACL changes
**Trade-off**: Transferable ownership and verifiable ACL, but requires Ethereum interaction and introduces latency.
#### Pattern 3: Hybrid — On-Chain Root + Local Cache
- On-chain contract defines who owns each org/repo NFT
- Local ACL graph caches on-chain state and adds local rules
- Hot-path checks use local cache (SQLite, fast)
- Cold-path operations (ACL changes, ownership transfers) go on-chain
- Local cache is periodically verified against on-chain state
**This is the recommended pattern for alknet.** It combines:
- On-chain censorship resistance (no single authority can revoke identity)
- Local performance (ACL checks are SQLite-fast)
- Transferable ownership (NFT can be sold/transferred on-chain)
- Graceful degradation (local ACL still works when chain is unavailable)
### 5.3 Radicle's Approach to Identity and Verification
Radicle's identity model has specific properties worth detailed comparison:
| Property | Radicle | alknet (proposed) |
|----------|---------|-------------------|
| **Identity root** | Ed25519 keypair (generated locally) | BIP39 seed phrase → SLIP-0010 derivation |
| **Identity document** | JSON in Git, signed by delegates | On-chain NFT + local ACL metagraph |
| **Delegate model** | Threshold of N public keys | Threshold of N delegates (on-chain or local) |
| **Key rotation** | Add/remove delegates via identity doc update | Transfer NFT to new address; update local keys |
| **Multi-device** | One key per device (RIP-0002) | One key per device derived from same seed (`m/74'/0'/0'/{n}'`) |
| **Namespace collision** | RID is content-hash, collision-free | NFT token ID is unique; human names via ENS |
| **Revocation** | Remove delegate from identity doc | On-chain ACL change + local cache update |
| **Verification** | Signature verification against delegate list | Signature verification + on-chain ACL check |
**alknet advantage**: Deriving multiple keys from one seed means:
- Multi-device support is built-in (derive a key per device)
- No "one key per identity" limitation
- The same seed provides identity keys, encryption keys, SSH keys, and Ethereum signing keys
- Key rotation for a single device is: derive a new key from the next index, updated locally
**alknet challenge**: If the seed phrase is lost, all derived keys are lost. Mitigation strategies:
- Social recovery (N-of-M threshold: trusted contacts hold shards)
- Hardware security module (HSM) protection for the seed
- Multi-sig on key operations (require threshold of devices to authorize)
---
## 6. Cryptographic Identity Mapping
### 6.1 Ed25519 Keys (alknet's Key Type)
alknet uses Ed25519 as the primary key type for:
- SSH authentication (fingerprint-based verification)
- Node identity (Node IDs are Ed25519 public keys)
- Channel signing (call protocol event signatures)
**Relevant properties of Ed25519**:
- 32-byte public key, 64-byte private key (or 32-byte seed + 32-byte public key)
- Deterministic signatures (same message, same key → same signature)
- Fast verification (~3x faster than secp256k1)
- Used in SSH (since OpenSSH 6.5), Tor onion services, Signal
**SLIP-0010 derivation** (what alknet uses):
- SLIP-0010 generalizes BIP-32 to non-secp256k1 curves
- Ed25519 derivation uses **hardened keys only** (cannot derive child public keys from parent public key)
- This means: the master seed must be available to derive any child key
- alknet's secret service holds the seed in RAM and derives keys on demand
### 6.2 Blockchain Private Keys vs SSH Keys
The key question for mapping blockchain identity to git access is: **how does an Ed25519 SSH key relate to a secp256k1 Ethereum key?**
| Key Type | Curve | Use Case | alknet Derivation Path |
|----------|-------|----------|----------------------|
| Identity key | Ed25519 | SSH auth, node identity | `m/74'/0'/0'/0'` |
| Device key | Ed25519 | Per-device identity | `m/74'/0'/0'/{n}'` |
| SSH host key | Ed25519 | Server identity | `m/74'/0'/1'/0'` |
| Encryption key | AES-256-GCM | External credential encryption | `m/74'/2'/0'/0'` |
| Ethereum key | secp256k1 | Smart contract signing | `m/44'/60'/0'/0/0` |
**The bridge**: Both keys derive from the **same BIP39 seed phrase**. The secret service can sign an Ethereum transaction using the secp256k1 key and also authenticate SSH using the Ed25519 key. This creates a cryptographically linked identity pair:
- On-chain identity (Ethereum address derived from `m/44'/60'/0'/0/0`)
- Off-chain identity (Ed25519 key derived from `m/74'/0'/0'/0'`)
**Binding them**: To prove that the Ed25519 key and the Ethereum key belong to the same entity:
1. Sign a message with the Ed25519 key: `"I, <Ed25519-pubkey>, attest that my on-chain identity is <Ethereum-address>"`
2. Store this attestation on-chain (in the org/user NFT metadata)
3. Anyone can verify: the on-chain address owns the NFT, and the attestation links the SSH key to that address
This is the **key binding mechanism** that connects alknet's SSH-based authentication to on-chain identity.
### 6.3 Deriving Repository Access from On-Chain Identity
The complete flow for a push operation in a decentralized git system with on-chain ACL:
```
1. Client connects to replicator via SSH
2. SSH auth succeeds (Ed25519 key verified by alknet IdentityProvider)
3. Client pushes to repo X
4. Replicator checks:
a. Local ACL metagraph: does this Ed25519 key have write access to repo X?
b. If local ACL is stale, re-verify against on-chain contract
5. If authorized: accept push, gossip update to other replicators
6. If not: reject with "access denied"
```
**Optimization**: Step 4b is rarely needed if the local ACL cache is kept fresh via event subscriptions. The on-chain contract emits events on ACL changes, and the head node's sync process projects these into the local ACL metagraph.
**alknet's existing support for this flow**:
| Component | Role |
|-----------|------|
| `IdentityProvider` trait | Resolves Ed25519 fingerprint → `Identity` with scopes/resources |
| `ConfigIdentityProvider` | Local-only: reads from `authorized_keys` config |
| `StorageIdentityProvider` | SQLite-backed: queries `peer_credentials` + ACL metagraph |
| `OnChainIdentityProvider` (future) | Verifies against on-chain ACL, falls back to local cache |
| `AuthProtocol` (irpc) | `VerifyPubkey` → `Identity` resolution |
| `CheckAccess` (irpc) | `Identity` + operation → access verification using ACL graph |
| `OperationSpec.access_control` | Declarative access requirements per operation |
---
## 7. Gossip Protocols for Repo Synchronization
### 7.1 Epidemic/Gossip Protocol Fundamentals
Gossip protocols are decentralized dissemination mechanisms inspired by how rumors spread in social networks. Key properties:
- **Eventual consistency**: All nodes eventually receive all updates
- **Fault tolerance**: Works even when nodes join/leave randomly
- **Scalability**: O(log N) time to reach all nodes in a network of N nodes
- **No single point of failure**: No coordinator node
### 7.2 Radicle's Gossip Protocol
Radicle uses three message types (detailed in Section 3.1):
- **Node Announcements**: Peer discovery (who's online, where to reach them)
- **Inventory Announcements**: Repo discovery (what repos each node seeds)
- **Reference Announcements**: Update notifications (new commits, new COB operations)
**Anti-entropy mechanism**: Nodes periodically exchange state summaries to ensure they haven't missed any updates. This is similar to Merkle tree-based reconciliation in distributed databases.
**Relevance to alknet**: alknet's call protocol subscription model (`call.requested` with `OperationType::Subscription`) can serve as the transport for gossip messages. The key difference is that alknet's call protocol is request-response oriented, while gossip is push-based. A gossip layer on top of the call protocol would work as follows:
```
alknet gossip layer:
1. Subscribe to `/{node}/gossip/announce` on known peers
2. Receive NodeAnnouncement, InventoryAnnouncement, RefAnnouncement events
3. Forward announcements to other connected peers (with deduplication)
4. For RefAnnouncements of tracked repos, trigger git fetch
```
### 7.3 Alternative: CRDT-Based Sync
Instead of gossip + git fetch, some systems use CRDTs for repository synchronization:
- **Advantages**: No merge conflicts, automatic convergence
- **Disadvantages**: Large metadata overhead, complex implementation, doesn't map directly to git's object model
**Recommendation for alknet**: Start with gossip + git fetch (as Radicle does) and consider CRDT-based sync for specific metadata (e.g., ACL state, org metadata) while keeping git data as-is. The ACL metagraph changes can propagate via honker streams (which are effectively a form of CRDT merge).
---
## 8. Relevance to Alknet
### 8.1 Identity + IdentityProvider Model
alknet's existing `Identity` struct and `IdentityProvider` trait are already designed for this use case:
```rust
pub struct Identity {
pub id: String, // Fingerprint or UUID
pub scopes: Vec<String>, // Permission scopes
pub resources: Option<HashMap<String, Vec<String>>>, // Resource-level access
}
```
The `id` field serves dual purpose:
- **Config-based auth**: SSH fingerprint (e.g., `SHA256:abc123...`)
- **Storage-based auth**: Account UUID (e.g., `acc_0123456789`)
**Extended for on-chain identity**, the `id` field could also be:
- **On-chain auth**: Ethereum address (e.g., `0x1234...`) or NFT token ID (e.g., `token_42`)
The `IdentityProvider` trait naturally extends:
```rust
trait IdentityProvider: Send + Sync {
fn resolve_from_fingerprint(&self, fingerprint: &str) -> Option<Identity>;
fn resolve_from_token(&self, token: &[u8]) -> Option<Identity>;
}
// Future extension:
// OnChainIdentityProvider resolves Ethereum address + Ed25519 binding
// from on-chain ACL contract, with local metagraph cache
```
### 8.2 OperationRegistry Extension with On-Chain Verification
alknet's `OperationSpec` includes `access_control` fields:
```rust
pub struct AccessControl {
pub required_scopes: Vec<String>,
pub required_scopes_any: Option<Vec<String>>,
pub resource_type: Option<String>,
pub resource_action: Option<String>,
}
```
For on-chain verification, a new `access_control` mode could be added:
```rust
pub enum AccessControlMode {
Local, // Check against local ACL metagraph (current)
OnChain, // Verify against on-chain contract (future)
CachedOnChain, // Check local cache first, verify on-chain on miss/stale (recommended)
}
```
The `AccessControl` struct gains a `mode` field defaulting to `Local`. This is additive and doesn't change existing behavior.
### 8.3 Git Service Adapter for Decentralized Replication
alknet's application service pattern (from services.md) can accommodate a `GitService`:
```rust
#[rpc_requests(message = GitMessage)]
enum GitProtocol {
#[rpc(tx=oneshot::Sender<RepoInfo>)]
#[wrap(GetRepo)]
GetRepo { repo_id: String },
#[rpc(tx=oneshot::Sender<Vec<RepoInfo>>)]
#[wrap(ListRepos)]
ListRepos { org: Option<String> },
#[rpc(tx=oneshot::Sender<bool>)]
#[wrap(CanPush)]
CanPush { repo_id: String, identity: Identity },
#[rpc(tx=oneshot::Sender<()>)]
#[wrap(UpdateMirror)]
UpdateMirror { repo_id: String, refs: Vec<RefUpdate> },
#[rpc(tx=mpsc::Sender<RefAnnouncement>)]
#[wrap(SubscribeRefs)]
SubscribeRefs { repo_ids: Vec<String> },
}
```
This service:
- **Registers with the call protocol** as `/head/git/*`
- **Uses `StorageIdentityProvider`** for `CanPush` checks (with ACL metagraph)
- **Manages git mirrors** (git bare repos on the local filesystem)
- **Propagates updates** via `SubscribeRefs` (which maps to honker stream subscriptions → call protocol integration events)
### 8.4 CredentialProvider Role
The existing `CredentialProvider` pattern in alknet (used for outbound authentication TO external services) maps to:
| Use Case | CredentialProvider Implementation |
|----------|----------------------------------|
| Push to GitHub/GitLab | SSH key from alknet identity, or OAuth token from external source |
| Push to on-chain repo | Ed25519 key derived from seed (signs the push) + Ethereum key (signs on-chain attestation) |
| Authenticate to replicator | Ed25519 key (SSH auth via `IdentityProvider`) |
| Decrypt stored credentials | AES-256-GCM key derived from seed via `SecretProtocol` |
### 8.5 Domain Events vs. Integration Events (Distributed Git Context)
alknet's event boundary discipline (from event sourcing research and ADR-032) is critical for the distributed git scenario:
| Event Type | Source | Consumer | Boundary | Git Analog |
|-----------|--------|----------|----------|------------|
| **Domain events** (honker) | Local service | Same service | Internal | Git object creation/update in local repo |
| **Integration events** (call protocol) | Projected from domain events | Other nodes/services | Cross-node | Push notification, gossip announcement |
| **On-chain events** (smart contract) | Ethereum log | Head node sync process | External source | ACL change on blockchain |
| **Notifications** (honker) | Service | Any subscriber | Cross-service | "Repo X was updated" (thin, ID-only) |
**The flow for a decentralized git push**:
```
1. Client pushes to replicator
2. Replicator's GitService receives push
3. GitService publishes domain event: "repo:refs-updated" (honker stream)
4. Integration event projected: "call.responded" with repo update (call protocol)
5. Replicator gossips "RefAnnouncement" to tracked peers (call protocol subscription)
6. On-chain: if this push creates a new branch, optionally emit on-chain attestation
7. Peer replicators fetch updated refs (git protocol) and update their mirrors
```
**The flow for an ACL change**:
```
1. Org admin calls smart contract: grantWrite(repoId, newUserAddress)
2. Smart contract emits RoleGranted event
3. Head node's sync process detects the event (Ethereum log subscription)
4. Sync process calls StorageService: add DelegatesEdge to ACL metagraph
5. StorageService publishes domain event: "acl:updated" (honker stream)
6. Integration event projected: notify replicators of ACL change (call protocol)
7. Replicators update their local ACL cache
```
This cleanly separates:
- **On-chain events** (smart contract logs) = external source of truth
- **Local projections** (ACL metagraph) = cached view for fast access checks
- **Integration events** (call protocol) = cross-node notification mechanism
- **Domain events** (honker streams) = internal state management
### 8.6 Practical Integration Path
For alknet to support the decentralized git concept, the integration path is:
#### Phase 1: Foundation (Current Architecture)
- `IdentityProvider` trait supports multiple backends ✓
- `StorageIdentityProvider` queries `peer_credentials` + ACL graph ✓
- `SecretProtocol` derives Ed25519 and secp256k1 keys from same seed ✓
- `OperationSpec.access_control` supports scope-based checks ✓
#### Phase 2: Git Service (Additive)
- Add `GitProtocol` irpc service for repo management
- Implement `GitService` as an application service (like DockerService, NodeService)
- Map `CanPush` to ACL metagraph traversal
- Implement `pre-receive` hook that calls alknet's `CheckAccess` irpc
#### Phase 3: On-Chain ACL (Additive, Requires External Dependencies)
- Add `OnChainIdentityProvider` that:
1. Resolves Ed25519 fingerprint → Ethereum address (via attestation stored in NFT metadata)
2. Checks on-chain ACL contract for access rights
3. Caches results in local ACL metagraph
4. Subscribes to on-chain events for ACL changes
- Add `AccessControlMode::CachedOnChain` to `OperationSpec`
- Add `WalletProtocol` irpc service for signing on-chain transactions
#### Phase 4: Gossip and Replication (Additive)
- Add gossip message types to call protocol (`NodeAnnouncement`, `RepoAnnouncement`, `RefAnnouncement`)
- Implement `SubscribeRefs` streaming operation for repo update subscriptions
- Add replicator service that seeds repos and responds to gossip
Each phase is additive and doesn't require changes to earlier phases. The architecture supports this incremental extension because:
1. `IdentityProvider` is a trait — new implementations are additive
2. `OperationSpec.access_control` is a struct — new fields are additive
3. Application services register with the call protocol — new services don't change core
4. Honker streams are internal — new streams are additive
---
## 9. References
### Decentralized Git Platforms
- **Radicle Protocol Guide**: https://radicle.dev/guides/protocol — Comprehensive documentation of Radicle's identity system, gossip protocol, replication, and self-certifying repositories
- **Radicle Heartwood (source)**: https://github.com/radicle-dev/heartwood — Reference implementation in Rust
- **RIP-0002 Identity**: Radicle Improvement Proposal for identity documents and delegate thresholds
- **radicle-crypto crate**: Ed25519 key types, SSH encoding, keystore (DeepWiki analysis: https://deepwiki.com/radicle-dev/heartwood/7.1-radicle-crypto)
- **ForgeFed**: https://forgefed.org/ — ActivityPub-based federation protocol for forges (Forgejo, Gitea integration)
- **GitLike**: https://gitlike.dev/ — Browser-based decentralized VCS using IPFS and Ethereum
- **GitBross**: https://gitbross.com/ — Decentralized Git platform using Solana, Arbitrum, and IPFS
- **PineSU**: IEEE paper on Git + Ethereum integration for trusted information sharing
### Blockchain Identity and Naming
- **ERC-721 Standard**: https://ethereum.org/developers/docs/standards/tokens/erc-721 — Non-fungible token standard
- **ENS (Ethereum Name Service)**: https://docs.ens.domains/ — Decentralized naming on Ethereum
- **W3C DID Primer**: https://w3c-ccg.github.io/did-primer/ — Decentralized Identifiers overview
- **W3C Verifiable Credentials**: https://www.w3.org/TR/vc-data-model/ — VC specification
- **EIP-3668 (CCIP-Read)**: Off-chain data lookup for ENS, enabling smart contracts to verify off-chain data
### Access Control
- **Git Hooks**: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks — Server-side hooks for git access control
- **Gitolite**: Config-file-based SSH key → repo permission mapping
- **Token-Gated Access Control**: https://chainscorelabs.com/guides/ — Patterns for ERC-721/ERC-1155 token-gated access
- **ChainGuard**: Blockchain-based authentication and access control (academic paper)
### Cryptographic Key Management
- **SLIP-0010**: https://slips.readthedocs.io/en/latest/slip-0010/ — Universal private key derivation from master private key (Ed25519, secp256k1, NIST P-256)
- **BIP-0032**: Hierarchical Deterministic Wallets
- **BIP-0039**: Mnemonic code for generating deterministic keys
- **SLIP-0044**: Registered coin types for BIP-0044 (alknet uses unallocated `74'`)
- **Ed25519**: Bernstein's Edwards-curve Digital Signature Algorithm
### Gossip Protocols
- **Gossip Protocol Fundamentals**: https://www.geeksforgeeks.org/distributed-systems/gossip-protocol-in-disrtibuted-systems/ — Epidemic-style information dissemination
- **libgossip**: C++17 implementation for decentralized node discovery and metadata propagation
- **Bitcoin Gossip**: Used in Bitcoin for transaction and block propagation
- **Secure Scuttlebutt (SSB)**: Inspiration for Radicle's gossip model
### Alknet Architecture Documents (Internal)
- **core.md**: Transport, call protocol, auth, services, DNS
- **services.md**: irpc service architecture, OperationEnv, Identity, auth/secret/config protocols
- **storage.md**: Metagraph data model, ACL as metagraph, identity tables, honker integration
- **integration-plan.md**: Phase 0-4 integration plan, ADRs 026-034
- **ADR-029**: Identity as core type (`Identity { id, scopes, resources }` + `IdentityProvider` trait)
- **ADR-032**: Event boundary discipline (domain events vs. integration events vs. service calls)
### Radicle-Specific Documentation
- **Radicle COBs (Collaborative Objects)**: CRDT-based distributed issues/patches stored as Git objects — https://deepwiki.com/radicle-dev/heartwood/6.1-collaborative-objects-(cobs)
- **Radicle Identity Documents**: Delegates, thresholds, and self-certifying repo identity — RIP-0002
- **Radicle Signed Refs**: Vulnerability disclosure (2026-03) on replay attacks in signed references

716
docs/research/references/gitserver/gitserver-reference.md
View File

@@ -1,716 +0,0 @@
# Gitserver Reference Document
> **Source**: <https://github.com/WJQSERVER/gitserver> (cloned at `/workspace/gitserver/`)
> **Version**: 0.0.3 (workspace Cargo.toml)
> **License**: MPL-2.0 (primary); upstream portions MIT (preserved in UPSTREAM-LICENSE)
> **Upstream origin**: <https://github.com/ggueret/git-server>
> **Date researched**: 2026-06-08
> **Purpose**: Evaluate gitserver as a basis for a git service adapter within alknet
---
## 1. Architecture Overview
### 1.1 What is gitserver?
Gitserver is a **Rust-native Git Smart HTTP server** that does not require an installed `git` binary at runtime. All Git operations (ref advertisement, pack generation, receive-pack) are implemented via the [gitoxide](https://github.com/GitoxideLabs/gitoxide) (`gix`) crate. It supports both Git protocol v1 and v2, including shallow clones and multi-ack negotiation.
The project follows a **library-first design**: `gitserver-core` and `gitserver-http` are reusable libraries, while the `gitserver` binary is a thin CLI wrapper for standalone deployment.
### 1.2 Crate Structure
```
crates/
├── gitserver-core/ # Git protocol operations (no HTTP dependency)
│ ├── backend.rs # GitBackend: unified interface for refs/pack/receive-pack
│ ├── discovery.rs # RepoStore: filesystem-based repo discovery
│ ├── dynamic_registry.rs # DynamicRepoRegistry, RepoResolver, MutableRepoRegistry traits
│ ├── error.rs # Error types (RepoNotFound, PathTraversal, Protocol, Git, Io)
│ ├── pack.rs # UploadPackRequest parsing, pack generation with side-band-64k
│ ├── path.rs # Path safety: resolve_repo_path (normalize + canonicalize)
│ ├── pktline.rs # pkt-line encoding/decoding utilities
│ ├── protocol_v2.rs # Git protocol v2: ls-refs, fetch, shallow, stateless-rpc
│ ├── receive_pack.rs # receive-pack: ref advertisement, pack reception, fast-forward validation
│ └── refs.rs # Protocol v1 ref advertisement
├── gitserver-http/ # Axum HTTP layer
│ ├── error.rs # AppError enum → HTTP status codes
│ ├── handlers.rs # Route handlers: info_refs, upload_pack, receive_pack, healthz, list
│ ├── lib.rs # router() function + public re-exports
│ └── state.rs # SharedState (RepoMode, AuthConfig, ServicePolicy, draining flag)
├── gitserver/ # CLI binary (thin wrapper)
│ └── main.rs # CLI args, RepoStore discovery, Axum server, graceful shutdown
└── gitserver-bench/ # Performance benchmarks (not published)
```
### 1.3 Key Dependencies
| Dependency | Version | Purpose |
|---|---|---|
| `gix` | 0.80.0 | Native Git repository operations (open refs, object store, rev-walk) |
| `gix-pack` | 0.67.0 | Pack file writing (receive-pack) |
| `axum` | 0.8.8 | HTTP routing and handlers |
| `tokio` | 1.50.0 | Async runtime, channels, IO |
| `miniz_oxide` | 0.8 | Zlib compression for pack objects |
| `sha1` | 0.10 | Pack checksum |
| `flate2` | 1 | Gzip response compression |
| `zstd` | 0.13 | Zstd response compression |
| `base64` | 0.22 | HTTP Basic auth decoding |
| `subtle` | 2 | Constant-time comparison (auth) |
| `clap` | 4.6.0 | CLI argument parsing |
### 1.4 Request Flow
#### Clone/Fetch (Protocol v1)
```
Client → GET /{repo}/info/refs?service=git-upload-pack
→ Server: resolve repo, verify auth, advertise_refs()
← Ref advertisement response
Client → POST /{repo}/git-upload-pack
→ Server: parse UploadPackRequest, generate_pack()
← Streamed side-band-64k pack response
```
#### Clone/Fetch (Protocol v2)
```
Client → GET /{repo}/info/refs (git-protocol: version=2)
← Capabilities advertisement
Client → POST /{repo}/git-upload-pack (git-protocol: version=2)
→ Server: parse_command_request() → ls-refs or fetch
← ls-refs result or streamed packfile
```
#### Push (receive-pack, must be enabled)
```
Client → GET /{repo}/info/refs?service=git-receive-pack
← Ref advertisement
Client → POST /{repo}/git-receive-pack
→ Server: parse commands, write pack, validate fast-forward, update refs
← Status report (ok/ng per ref)
```
---
## 2. Protocol Support
### 2.1 Smart HTTP Git Protocol
Gitserver implements the **Git Smart HTTP protocol** (RFC-like, de facto standard). This is the standard protocol used by `git clone http://...`, `git fetch`, and `git push` over HTTP.
**Supported endpoints:**
| Method | Endpoint | Protocol Version | Description |
|---|---|---|---|
| GET | `/healthz` | — | Health check (no auth) |
| GET | `/` | — | JSON repository listing (auth required if configured) |
| GET | `/{repo}/info/refs?service=git-upload-pack` | v1 | Ref advertisement for clone/fetch |
| GET | `/{repo}/info/refs?service=git-receive-pack` | v1 | Ref advertisement for push (disabled by default) |
| POST | `/{repo}/git-upload-pack` | v1 | Pack negotiation and transfer |
| POST | `/{repo}/git-receive-pack` | v1 | Push operations (disabled by default) |
| GET | `/{repo}/info/refs` with `git-protocol: version=2` | v2 | Capabilities advertisement |
| POST | `/{repo}/git-upload-pack` with `git-protocol: version=2` | v2 | `ls-refs` and `fetch` commands |
### 2.2 Git Operations
| Operation | Supported | Notes |
|---|---|---|
| `git clone` | ✓ | Both v1 and v2 |
| `git fetch` | ✓ | Multi-ack, multi-ack-detailed negotiation |
| `git push` | ✓ (opt-in) | Via `--enable-receive-pack` or `ServicePolicy.receive_pack: true` |
| Shallow clone | ✓ | Protocol v2 `fetch` with `deepen` |
| OFS_DELTA | ✓ | Offset delta compression in packs |
| Side-band-64k | ✓ | Multiplexed progress/pack data |
| Response compression | ✓ | Gzip and Zstd on ref advertisement |
### 2.3 Push Restrictions
When receive-pack is enabled, the following restrictions apply:
- **Fast-forward only**: Branch updates under `refs/heads/*` must be fast-forward (old commit is ancestor of new)
- **No ref deletion**: New OID cannot be the zero OID
- **No tag overwrite**: Updating an existing tag is rejected
- **Commits only**: Branch tips must point to commit objects
- **Timeouts**: 300s total, 30s idle
### 2.4 SSH Git Protocol
Gitserver does **not** support SSH Git protocol. It is HTTP-only. SSH git access would require a separate implementation or integration layer (see Section 6).
---
## 3. Interface Pattern Analysis
### 3.1 HTTP Handler Architecture
Gitserver's HTTP layer follows a clean handler pattern:
```rust
// gitserver-http/src/lib.rs
pub fn router(state: SharedState) -> Router {
Router::new()
.route("/healthz", get(handlers::healthz))
.route("/", get(handlers::list_repos))
.route("/{*path}", get(handlers::info_refs_dispatch))
.route("/{*path}", post(handlers::rpc_dispatch))
.with_state(state)
}
```
The `SharedState` is an Axum state object containing:
- `RepoMode` — either `Discovered(Arc<RwLock<RepoStore>>)` or `Dynamic { resolver, registry }`
- `AuthConfig` — optional Basic and/or Bearer authentication
- `ServicePolicy` — toggle for upload_pack, upload_pack_v2, receive_pack
- `draining: Arc<AtomicBool>` — graceful shutdown flag
Each handler follows this pattern:
1. Check `draining` flag → 503 if shutting down
2. Check `ServicePolicy` → 404 if service disabled
3. Authenticate request via `require_auth()` → 401 if credentials missing/invalid
4. Resolve repository via `SharedState::resolve()` → 404 if not found
5. Execute git operation via `GitBackend`
6. Return streaming or buffered response
### 3.2 Mapping to alknet's MessageInterface
Gitserver's `SharedState` + handler pattern maps closely to alknet's proposed `MessageInterface` trait:
```rust
// alknet's proposed MessageInterface
async fn handle_request(&self, request: InterfaceRequest) -> Result<InterfaceResponse>;
```
Gitserver's handler flow is essentially:
1. Receive HTTP request (analogous to `InterfaceRequest`)
2. Extract operation path, auth, and body
3. Dispatch to the appropriate Git operation
4. Return HTTP response (analogous to `InterfaceResponse`)
### 3.3 Low-Level Handler API
Gitserver also exposes handler functions that can be called directly without going through the Axum router:
```rust
use gitserver_http::handlers::{info_refs_endpoint, ServiceKind};
let response = info_refs_endpoint(
&state,
"my-project.git",
ServiceKind::UploadPack,
HeaderMap::new(),
).await?;
```
This is significant for alknet integration — it means the git logic can be invoked programmatically without HTTP routing.
---
## 4. Authentication
### 4.1 Current Auth Model
Gitserver supports two HTTP authentication mechanisms, both optional:
```rust
pub struct AuthConfig {
pub basic: Option<BasicAuthConfig>,
pub bearer_token: Option<String>,
}
pub struct BasicAuthConfig {
pub username: String,
pub password: String,
}
```
**Key characteristics:**
- Both can be configured simultaneously; **either one passing is sufficient**
- Basic auth uses **constant-time comparison** (`subtle` crate) to prevent timing attacks
- Bearer token is compared directly (suitable for generated tokens)
- Failed auth returns `401 Unauthorized` with `WWW-Authenticate: Basic realm="gitserver", Bearer`
- `GET /healthz` is **unauthenticated** (always accessible)
- Auth is **global** (same credentials for all repositories) — no per-repo or per-user ACL
### 4.2 Auth Flow in Handlers
```rust
fn require_auth(store: &SharedState, headers: &HeaderMap) -> Result<(), AppError> {
let auth = store.auth();
if auth.basic.is_none() && auth.bearer_token.is_none() {
return Ok(()); // No auth configured → allow all
}
let value = headers.get(AUTHORIZATION)...;
// Try Bearer first, then Basic
// Constant-time comparison for Basic
}
```
### 4.3 Mapping to alknet Identity
alknet's `IdentityProvider` resolves credentials to an `Identity`. The mapping would be:
| gitserver auth | alknet equivalent | Resolution path |
|---|---|---|
| No auth | `Identity::anonymous()` or reject | Configurable policy |
| Basic auth (username/password) | `IdentityProvider::resolve_from_token()` | Map to AuthToken or direct lookup |
| Bearer token | `IdentityProvider::resolve_from_token()` | Token is already in the right format |
The key gap is that gitserver's auth is **single-credential, global**, while alknet needs **per-identity, per-repository** access control. Integration would require:
1. Replacing `AuthConfig` with alknet's `IdentityProvider`
2. Extracting identity from the `Authorization` header
3. Checking per-repo ACL based on resolved `Identity`
---
## 5. Storage
### 5.1 Filesystem-Based Storage
Gitserver currently stores repositories as **bare Git repositories on the local filesystem**. The storage model is:
```
ROOT/
├── project-a.git/ # bare repository
│ ├── HEAD
│ ├── objects/
│ ├── refs/
│ └── description
├── org/
│ └── project-b.git/ # nested repository (up to max_depth)
└── ...
```
The `RepoStore::discover(root, max_depth)` function:
1. Canonicalizes the root path
2. Recursively walks subdirectories up to `max_depth`
3. Attempts `gix::open(path)` on each directory
4. If `repo.is_bare()`, adds it as a `RepoInfo`
5. Path traversal protection via lexical normalization + `canonicalize()` double-check
The `DynamicRepoRegistry` allows programmatic registration/unregistration of repos at runtime, validated by `gix::open()` confirming the path is a bare repo.
### 5.2 Storage Abstraction Points
The key storage interaction points in the codebase are:
| Component | Storage Pattern |
|---|---|
| `RepoStore::discover()` | Filesystem scan (local directory tree) |
| `DynamicRepoRegistry` | In-memory registry with filesystem-backed paths |
| `GitBackend::new(repo_path)` | Opens a local bare repo via `gix::open()` |
| `receive_pack::write_pack()` | Writes pack to `objects/pack/` via `gix_pack::Bundle::write_to_directory()` |
| `path::resolve_repo_path()` | Canonical path resolution + traversal protection |
**All storage operations assume a local filesystem path.** There is no abstraction for remote or object storage backends.
### 5.3 Rustfs (S3-Compatible) Integration Feasibility
Git operations fundamentally require **a local filesystem** — `gix::open()` expects a directory with the standard `.git` layout (objects, refs, HEAD, etc.). Rustfs (S3-compatible) cannot serve as a **direct** storage backend for gitoxide's repository operations because:
1. `gix::open()` requires a local path — it reads `HEAD`, refs, and object packs from the filesystem
2. Pack generation (`generate_pack()`) streams objects from the local ODB
3. Receive-pack writes pack files to the local `objects/pack/` directory
4. Reference updates use `gix::Repository::edit_references()` which operates on the local refstore
However, rustfs **could** be used in several supporting roles:
| Integration Approach | Description | Feasibility |
|---|---|---|
| **Repo sync backend** | Store bare repo tarballs in rustfs; sync to local disk on demand | High — sync from S3 to local FS before serving |
| **Backup/archive** | Push repo backups to rustfs buckets | High — out-of-band backup |
| **Git LFS storage** | Store large file objects in rustfs via Git LFS | Medium — requires LFS server implementation |
| **Object store proxy** | Cache layer: serve from local FS, sync to/from rustfs | Medium — needs repo lifecycle management |
| **Direct S3 repo** | Custom `gix` object backend reading from S3 | Low — would require deep gitoxide customization |
The most practical approach: **use rustfs as a backing store for repository synchronization**. Gitserver would always operate on local filesystem paths, but a separate component would manage syncing repos to/from rustfs buckets.
---
## 6. SSH Support
### 6.1 Current State
Gitserver has **no SSH transport capability**. It only implements the HTTP Smart Git protocol. Adding SSH support would require implementing the Git SSH protocol, which is a different wire format:
| Aspect | Smart HTTP | SSH |
|---|---|---|
| Transport | HTTP (request/response) | Persistent SSH channel |
| Service discovery | `GET /info/refs?service=git-upload-pack` | `ssh://host/git-upload-pack 'repo'` |
| Protocol framing | pkt-line over HTTP | pkt-line over SSH channel |
| Authentication | HTTP Authorization header | SSH key-based |
| Multiplexing | HTTP/2 or separate connections | Multiple SSH channels |
### 6.2 How Git over SSH Works
The Git SSH protocol uses SSH as a transport for the same `git-upload-pack` and `git-receive-pack` commands:
```
Client connects via SSH → server executes git-upload-pack or git-receive-pack
Client ← SSH channel → Server (bidirectional pkt-line stream)
```
### 6.3 Integration with alknet's SSH Interface
alknet's SSH interface (`SshInterface`) is a `StreamInterface` — it accepts a persistent byte stream and multiplexes it into channels. This maps naturally to Git over SSH:
**Approach: Git as an alknet operation over SSH**
```
alknet SSH session
│
├─ Channel: call protocol (operations)
│
└─ Channel: git-upload-pack
OR git-receive-pack
│
▼
gitserver-core protocol logic
(ref advertisement, pack generation, receive-pack)
```
This would work by:
1. The SSH interface receives a connection with a request like `git-upload-pack '/repos/project.git'`
2. alknet resolves the identity from the SSH key fingerprint
3. Checks ACL: does this identity have read/write access to this repo?
4. Invokes `gitserver-core` functions directly (no HTTP needed):
- `refs::advertise_refs()` → send over SSH channel
- `pack::generate_pack()` → stream over SSH channel
- `receive_pack::receive_pack()` → read/write over SSH channel
**Key advantage**: Since `gitserver-core` has no HTTP dependency, it can be used directly over SSH channels without the HTTP overhead. The `GitBackend` API is transport-agnostic.
### 6.4 Alternative: Dedicated Git SSH Adapter
A simpler approach that doesn't require modifying the SSH channel multiplexing:
```
alknet SSH session → call protocol → operation "git/upload-pack" →
→ GitAdapter::upload_pack(repo, wants, haves) → streaming response
```
This treats Git operations as alknet call operations, where the SSH interface is the transport but Git operations are invoked via the call protocol rather than raw SSH channels. This is more aligned with alknet's architecture but requires adapting the Git protocol to the call protocol's request/response model (potentially with streaming).
---
## 7. Relevance to alknet
### 7.1 Mapping to alknet's Interface Model
Gitserver is a textbook **`MessageInterface`** implementation:
| alknet MessageInterface | Gitserver Equivalent |
|---|---|
| `handle_request(InterfaceRequest)` | `info_refs_dispatch()` / `rpc_dispatch()` |
| `InterfaceRequest.operation_path` | URL path (`/{repo}/info/refs`, `/{repo}/git-upload-pack`) |
| `InterfaceRequest.auth_token` | `Authorization` header → `require_auth()` |
| `InterfaceRequest.input` | Request body (pack negotiation data) |
| `InterfaceResponse.result` | HTTP response body (ref advertisement, pack data) |
| `InterfaceResponse.status` | HTTP status code |
| `InterfaceResponse.headers` | Content-Type, Cache-Control, etc. |
However, gitserver **manages its own transport** (Axum HTTP server), which is exactly the `MessageInterface` pattern described in alknet's interface model: "MessageInterface implementations manage their own transport. They don't need the Transport trait because they're not wrapping a generic byte stream — they ARE the transport+interface combined."
### 7.2 Git as an alknet Operation
Git operations could be mapped to alknet's call protocol namespace:
```
Namespace: "git"
Operations:
- git/list → List available repositories
- git/info-refs → Get ref advertisement for a repo
- git/upload-pack → Clone/fetch (streaming response)
- git/receive-pack → Push (streaming request+response)
- git/ls-refs → Protocol v2 ls-refs
- git/fetch → Protocol v2 fetch
```
**Challenge**: Git operations are **streaming and bidirectional** (especially fetch negotiation and receive-pack), while alknet's call protocol is currently defined as request→response. This needs design consideration:
| Operation | Direction | Stream Duration | alknet Fit |
|---|---|---|---|
| `git/list` | Request → Response | Short | Direct fit |
| `git/info-refs` | Request → Response | Short | Direct fit |
| `git/upload-pack` | Request → Streaming Response | Long | Needs streaming response support |
| `git/receive-pack` | Streaming Request → Streaming Response | Long | Needs bidirectional streaming |
### 7.3 Proposed GitAdapter Architecture
```
┌─────────────────────────────────────────────────────────┐
│ alknet node │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ HttpInterface│ │ SshInterface │ │ DNS/other │ │
│ │ (Message) │ │ (Stream) │ │ (Message) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ OperationRegistry │ │
│ │ "git/list" → GitAdapter::list_repos() │ │
│ │ "git/upload-pack" → GitAdapter::upload_pack() │ │
│ │ "git/receive-pack" → GitAdapter::receive_pack() │ │
│ └──────────────┬───────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────┐ │
│ │ GitAdapter │ │
│ │ - SharedState (repos, auth) │ │
│ │ - GitBackend (protocol ops) │ │
│ │ - IdentityProvider (auth) │ │
│ │ - RepoResolver (filesystem) │ │
│ └──────────────┬───────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────┐ ┌────────────────┐ │
│ │ Local filesystem │ │ Rustfs sync │ │
│ │ (bare git repos) │ │ (S3 backend) │ │
│ └──────────────────────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
### 7.4 Auth Integration: alknet Identity → Gitserver Auth
**Current gitserver auth** (single global credential):
```rust
AuthConfig {
basic: Option<BasicAuthConfig>, // one username/password
bearer_token: Option<String>, // one token
}
```
**Proposed alknet integration** (per-identity, per-repo):
```rust
struct GitAdapter {
identity_provider: Arc<dyn IdentityProvider>,
repo_resolver: Arc<dyn RepoResolver>,
backend_factory: Arc<dyn GitBackendFactory>,
acl: Arc<dyn GitAcl>,
}
impl GitAdapter {
async fn handle_request(
&self,
request: InterfaceRequest,
) -> Result<InterfaceResponse> {
// 1. Resolve identity from auth token
let identity = self.identity_provider
.resolve_from_token(request.auth_token)?;
// 2. Parse git operation from path
let operation = parse_git_operation(&request.operation_path)?;
// 3. Check ACL
self.acl.check_access(&identity, &operation.repo, operation.access_type)?;
// 4. Dispatch to gitserver-core logic
// ...
}
}
```
**ACL design** (per-repo, per-operation):
```rust
enum GitAccess {
Read, // clone, fetch
Write, // push
}
trait GitAcl: Send + Sync {
fn check_access(
&self,
identity: &Identity,
repo: &str,
access: GitAccess,
) -> Result<()>;
}
```
### 7.5 Storage Integration with Rustfs
**Recommended approach**: Rustfs as a sync backend:
```rust
trait RepoStorage: Send + Sync {
/// Ensure a local working copy exists for the given repo.
/// May involve syncing from S3 (rustfs) to local disk.
async fn ensure_local(&self, repo: &str) -> Result<PathBuf>;
/// Sync local changes back to S3 (rustfs) after a push.
async fn sync_to_remote(&self, repo: &str) -> Result<()>;
/// List available repos (may consult S3 bucket listing).
async fn list_repos(&self) -> Result<Vec<RepoInfo>>;
}
```
The flow would be:
1. `GitAdapter` receives a request for repo `X`
2. `RepoStorage::ensure_local("X")` checks if the repo exists on local disk; if not, syncs from rustfs
3. Git operations run on the local filesystem (using `gitserver-core` directly)
4. After push operations, `RepoStorage::sync_to_remote("X")` pushes updates to rustfs
This maintains gitserver's requirement for a local filesystem while leveraging rustfs for durability and distribution.
### 7.6 Operation Mapping
| Git Operation | alknet Namespace | alknet Op | Input | Output | Stream? |
|---|---|---|---|---|---|
| List repos | `git` | `list` | `{}` | `[RepoInfo]` | No |
| Ref advertisement (v1) | `git` | `info-refs` | `{repo, service: "upload-pack" \| "receive-pack"}` | Binary ref advertisement | No |
| Ref capabilities (v2) | `git` | `capabilities` | `{repo}` | Binary capabilities | No |
| Ls-refs (v2) | `git` | `ls-refs` | `{repo, peel, symrefs, ref_prefixes}` | Binary ref listing | No |
| Clone/Fetch | `git` | `upload-pack` | `{repo, wants, haves, done, ...}` | Streamed pack data | Yes (response) |
| Push | `git` | `receive-pack` | `{repo, commands, pack_data}` | Status report | Yes (both) |
### 7.7 What gitserver-core Provides Directly
The most valuable integration point is `gitserver-core` — the HTTP-free protocol library:
```rust
// Direct usage without HTTP
use gitserver_core::backend::GitBackend;
use gitserver_core::discovery::RepoStore;
use gitserver_core::pack::{UploadPackRequest, UploadPackCapabilities, ShallowRequest};
use gitserver_core::protocol_v2;
// Repository discovery
let store = RepoStore::discover("./repos".into(), 3)?;
let repo = store.resolve("my-project.git")?;
// Protocol v1 ref advertisement
let backend = GitBackend::new(repo.absolute_path.clone());
let refs = backend.advertise_refs()?;
// Pack generation (streaming)
let request = UploadPackRequest { wants, haves, done, ... };
let pack_stream = backend.upload_pack(&request).await?;
// Receive-pack (push)
let result = backend.receive_pack(request_stream).await?;
// Protocol v2
let capabilities = protocol_v2::advertise_capabilities();
let ls_refs_output = protocol_v2::ls_refs(&repo_path, &ls_refs_request)?;
let fetch_output = backend.upload_pack(&fetch_request.upload_request).await?;
```
These functions can be called from any async context — SSH channel handler, alknet operation handler, HTTP handler — without going through the Axum HTTP layer.
---
## 8. Integration Recommendations
### 8.1 Recommended Integration Strategy
**Phase 1: HTTP Gateway (MessageInterface)**
Embed gitserver-http's Axum router into alknet's HTTP interface. This provides immediate Git-over-HTTP capability:
```rust
// In alknet's HttpInterface::handle_request()
// Route: /git/* → gitserver router
let git_app = gitserver_http::router(git_state);
let app = Router::new()
.nest("/git", git_app) // Mount git under /git
.route("/v1/{namespace}/{op}", post(operation_handler));
```
This works because gitserver is designed to be nested into existing Axum apps. Auth integration would replace `AuthConfig` with alknet's `IdentityProvider`.
**Phase 2: SSH Git Adapter (StreamInterface)**
Use `gitserver-core` directly within alknet's SSH interface for Git-over-SSH:
```rust
// In alknet's SshInterface channel handler
// SSH channel request: "git-upload-pack '/repos/project.git'"
let backend = GitBackend::new(repo_path);
let refs = backend.advertise_refs()?;
// Send refs over SSH channel
// Stream pack data over SSH channel
```
**Phase 3: Call Protocol Operations (OperationRegistry)**
Register Git operations in the operation registry for access via any interface:
```rust
registry.register(GitListRepos::new(adapter.clone()));
registry.register(GitUploadPack::new(adapter.clone()));
registry.register(GitReceivePack::new(adapter.clone()));
```
### 8.2 Key Modifications Needed
1. **Auth replacement**: Replace `AuthConfig` with `IdentityProvider`-based auth in `handlers.rs`'s `require_auth()` function
2. **ACL addition**: Add per-repo, per-identity access control (gitserver currently has none)
3. **RepoResolver abstraction**: Replace `RepoStore`/`DynamicRepoRegistry` with alknet's `RepoResolver` that integrates with rustfs sync
4. **Streaming response support**: Adapt alknet's call protocol for streaming (large pack files)
5. **Bidirectional streaming**: For receive-pack, the call protocol needs to support bidirectional streaming
### 8.3 Risks and Mitigations
| Risk | Mitigation |
|---|---|
| gitserver requires local filesystem | Use rustfs as sync backend; maintain local working copies |
| Auth is global (single credential) | Fork/modify `require_auth()` to use `IdentityProvider` |
| No per-repo ACL | Add `GitAcl` trait in the adapter layer |
| MPL-2.0 license requires modifications to be under MPL-2.0 | Acceptable for alknet (MPL-2.0 is file-level copyleft) |
| Large pack files may not fit alknet's message size limits | Implement streaming response in the call protocol |
| gitoxide version coupling | Pin `gix = "0.80.0"` as gitserver does |
### 8.4 License Considerations
- **Primary license**: MPL-2.0 (file-level copyleft)
- **Upstream portions**: MIT (preserved in UPSTREAM-LICENSE)
- **Implication**: Modifications to gitserver's `.rs` files must remain under MPL-2.0. Linking from alknet code is unrestricted.
- **Recommendation**: Use gitserver as a library dependency. If alknet-specific auth/ACL modifications are needed, contribute them upstream or maintain them as separate files under MPL-2.0.
---
## 9. Summary
### 9.1 Key Findings
1. **gitserver is a well-structured, library-first Rust Git Smart HTTP server** with clean separation between protocol logic (`gitserver-core`) and HTTP transport (`gitserver-http`).
2. **Protocol support is comprehensive**: Git Smart HTTP v1 and v2, clone, fetch, push (opt-in), shallow clones, delta compression, streaming pack generation.
3. **No SSH support exists**, but `gitserver-core` is transport-agnostic and can serve Git operations over any channel.
4. **Auth is simple but limited**: single global Basic/Bearer credential, no per-repo or per-user ACL.
5. **Storage is local-filesystem only**: `gix::open()` requires a local path. S3/rustfs integration requires a sync-to-local approach.
6. **The library design enables direct integration**: `GitBackend` and protocol functions can be called without HTTP.
### 9.2 Recommendation
**Use `gitserver-core` as alknet's Git protocol engine.** The core crate provides all Git protocol operations (ref advertisement, pack generation, receive-pack, protocol v2) without any HTTP dependency. This allows alknet to expose Git services through any interface (HTTP, SSH, call protocol) while maintaining a single protocol implementation.
**Use `gitserver-http` as alknet's Git HTTP interface** by nesting its Axum router under alknet's HTTP interface, with auth replaced by `IdentityProvider`.
**Design a `GitAdapter`** that wraps `gitserver-core` and integrates with alknet's `OperationRegistry`, `IdentityProvider`, and rustfs-backed storage.
### 9.3 Next Steps
1. Fork or vendor `gitserver-core` and `gitserver-http` into alknet's dependency tree
2. Design the `GitAdapter` trait with `IdentityProvider` auth and `GitAcl` access control
3. Implement Phase 1: HTTP gateway with nested Axum router and `IdentityProvider` auth
4. Implement `RepoStorage` trait with rustfs sync-to-local strategy
5. Design streaming extensions to alknet's call protocol for pack file transfer
6. Evaluate Phase 2: SSH Git adapter using `gitserver-core` directly over SSH channels
---
## References
- [gitserver README](https://github.com/WJQSERVER/gitserver) — project overview, quick start, CLI usage
- [gitserver Architecture docs](docs/en/architecture.md) — crate responsibilities, request flows
- [gitserver Library docs](docs/en/library.md) — embedding, dynamic registration, auth config
- [gitserver API Reference](docs/en/api.md) — REST endpoints, protocol details, error codes
- [alknet Interface Model](../../phase2/interface-model.md) — StreamInterface/MessageInterface design
- [gitoxide](https://github.com/GitoxideLabs/gitoxide) — underlying Git implementation library

857
docs/research/references/honker/honker-reference.md
View File

@@ -1,857 +0,0 @@
# Research: Honker — SQLite Pub/Sub, Queue, and Notification Extension
## Key Findings
- **Honker is a Rust-based SQLite extension** that adds Postgres-style NOTIFY/LISTEN semantics plus durable pub/sub, task queues, and event streams entirely within SQLite. It eliminates the need for a separate message broker (Redis, Kafka) when SQLite is the primary datastore.
- **Three core primitives**: `notify/listen` (ephemeral pub/sub), `stream` (durable pub/sub with per-consumer offsets), and `queue` (at-least-once work queue with retries, priority, delayed jobs, and dead-letter handling). All three are SQL INSERTs inside your transaction — business write and side-effect commit or roll back together.
- **Wake mechanism**: Uses `PRAGMA data_version` polling at 1ms granularity to detect commits, achieving ~1-2ms median cross-process wake latency without requiring a daemon or broker. A single thread per database fans out to N subscribers via bounded channels.
- **Single-machine, single-writer model**: Designed for self-hosted deployments. Not distributed — no multi-node replication. This maps perfectly to alknet's per-node architecture where domain events are internal to a service boundary (ADR-032).
- **Comprehensive SQL API**: 30+ SQL scalar functions (`honker_enqueue`, `honker_claim_batch`, `honker_ack_batch`, `honker_stream_publish`, `honker_stream_read_since`, `honker_stream_save_offset`, `notify`, `honker_lock_acquire`, `honker_rate_limit_try`, `honker_scheduler_register`, etc.) registered as a loadable SQLite extension. Any language that can `SELECT load_extension('honker')` gets the same features.
- **Rust core (`honker-core`)**: All SQL implementations live in a shared Rust crate consumed by the loadable extension, PyO3 Python binding, napi-rs Node binding, and other language wrappers. One source of truth for the SQL — no behavioral drift across bindings.
- **License**: Apache 2.0 / MIT dual-license. Fully permissive for integration.
**Recommendation**: Adopt honker's patterns directly in `alknet-storage`. The `honker` crate (or `honker-core` for a Rust-native integration) should be a dependency of `alknet-storage`. Honker's single-node model aligns with alknet's event boundary discipline — domain events stay within the service boundary, and cross-node events go through the call protocol. For production deployments that use Postgres instead of SQLite, the same patterns (queue/claim, stream/subscribe, notify/listen) can be replicated using Postgres features, but honker's built-in retry, visibility timeout, and scheduling would need to be reimplemented.
---
## 1. Architecture
### What Is Honker?
Honker is a **SQLite extension + language bindings** that adds Postgres-style `NOTIFY`/`LISTEN` semantics to SQLite, with built-in durable pub/sub, task queues, and event streams — without requiring a client-polling loop, a daemon, or a separate broker.
**Core idea**: If SQLite is your primary datastore, your queue should live in the same file. `INSERT INTO orders` and `queue.enqueue(...)` commit in the same transaction. Rollback drops both.
**Implementation language**: Rust. The shared engine is `honker-core`, a plain Rust `rlib` crate. Language bindings (Python via PyO3, Node via napi-rs, Go via CGo, Ruby via C extension, .NET via P/Invoke, JVM via JNI, Kotlin wrapper, Elixir via NIF, C++ via header-only wrapper) are thin wrappers around the loadable extension's SQL functions.
**How it works as a SQLite extension**: The `honker-extension` crate compiles to `libhonker_ext.{so,dylib,dll}`. Any SQLite 3.9+ client loads it:
```sql
.load ./libhonker_ext
SELECT honker_bootstrap();
```
This creates the schema tables (`_honker_live`, `_honker_dead`, `_honker_notifications`, `_honker_stream`, `_honker_stream_consumers`, `_honker_locks`, `_honker_rate_limits`, `_honker_scheduler_tasks`, `_honker_results`) and registers all SQL scalar functions. The extension and Python/binding tables are shared, so a Python worker can claim jobs any other language pushed via the extension.
### Crate Structure
```
honker-core/ # Rust rlib shared across all bindings (published on crates.io)
honker-extension/ # SQLite loadable extension (cdylib, published on crates.io)
packages/
honker/ # Python package (PyO3 cdylib + Queue/Stream/Outbox/Scheduler)
honker-node/ # napi-rs Node.js binding
honker-rs/ # Ergonomic Rust wrapper
honker-go/ # Go binding
honker-ruby/ # Ruby binding
honker-bun/ # Bun binding
honker-ex/ # Elixir binding
honker-cpp/ # C++ binding
honker-dotnet/ # .NET / C# binding
honker-jvm/ # JVM / Java-compatible binding
honker-kotlin/ # Kotlin convenience wrapper
```
### Wake Path Architecture
The fundamental challenge for any SQLite-based pub/sub system: SQLite has no wire protocol or server-push. Consumers must initiate reads. Honker solves this with a **single-digit-microsecond `PRAGMA data_version` read**:
1. **One PRAGMA-poll thread per `Database`** queries `data_version` every 1ms
2. Counter change → fan out a tick to each subscriber's bounded channel (capacity 1 — coalesces redundant wakes)
3. Each subscriber runs `SELECT … WHERE id > last_seen` against a partial index, yields rows, returns to wait
4. 100 subscribers = 1 poll thread. Idle listeners run zero SQL queries.
Idle cost: ~3.5µs per `PRAGMA data_version` query, ~3.5ms/sec total at 1kHz. A 5-second paranoia poll exists as a fallback only if the update watcher cannot fire.
**Three backend options** (controlled by `WatcherBackend` enum):
- **Polling** (default, stable): `PRAGMA data_version` every 1ms. Correct on all platforms.
- **Kernel** (experimental, `kernel-watcher` Cargo feature): Uses `notify-rs` filesystem events. Fires on every filesystem write. May produce spurious/missed wakes. Dead-man's switch for file replacement.
- **SHM fast path** (experimental, `shm-fast-path` Cargo feature): Memory-maps the `-shm` WAL index file and reads `iChange` at offset 8 at ~100µs cadence. WAL-mode only. Dead-man's switch for file replacement.
**Dead-man's switch**: All backends check file identity `(dev, ino)` / `(volume_serial, file_index)` every ~100ms. If the database file is replaced (atomic rename, litestream restore, volume remount), the watcher panics with a clear error message. Subscribers see an error from `update_events()` instead of hanging silently.
### SharedUpdateWatcher
```rust
pub struct SharedUpdateWatcher {
watcher: Mutex<Option<UpdateWatcher>>, // background poll thread
senders: Arc<Mutex<HashMap<u64, SyncSender<()>>>>, // fan-out channels
next_id: AtomicU64,
}
```
- `subscribe()` → `(u64, Receiver<()>)` — register a channel; capacity 1
- `unsubscribe(id)` — remove channel; receiver sees `Err(RecvError)`
- `close()` — join the poll thread, clear all subscribers
- Wakes are idempotent "go re-read state" signals. Dropped redundant wakes never lose data.
---
## 2. Core Capabilities
### 2.1 Notify/Listen — Ephemeral Pub/Sub
**What it is**: Fire-and-forget notifications to channel subscribers. Like `pg_notify` but with table-backed persistence until explicitly pruned.
**How it works**:
- `notify(channel, payload)` is a SQL scalar function that INSERTs into `_honker_notifications` and returns the row id. Runs inside the caller's open transaction — rollbacks drop the notification.
- `db.listen(channel)` or `db.updateEvents()` in Node — registers a subscriber that wakes on any database commit, then filters by channel in the `SELECT` path.
- Listeners attach at current `MAX(id)`; **history is not replayed**. This is the key distinction from streams.
**Schema**:
```sql
CREATE TABLE _honker_notifications (
id INTEGER PRIMARY KEY AUTOINCREMENT,
channel TEXT NOT NULL,
payload TEXT NOT NULL,
created_at INTEGER NOT NULL DEFAULT (unixepoch())
);
CREATE INDEX _honker_notifications_recent ON _honker_notifications(channel, id);
```
**Key characteristics**:
- Not auto-pruned. Call `db.prune_notifications(older_than_s=…, max_keep=…)` from a scheduled task.
- Over-triggering is by design: a `data_version` change wakes every subscriber on that database, not just the matching channel. Each wasted wake = one indexed SELECT (microseconds). A missed wake = a silent correctness bug.
- Payload must be valid JSON for cross-language compatibility.
### 2.2 Queue — At-Least-Once Work Queue
**What it is**: Durable, at-least-once delivery work queue with retries, priority, delayed jobs, task expiration, dead-letter handling, named locks, and rate-limiting.
**Schema (single-table hybrid)**:
```sql
CREATE TABLE _honker_live (
id INTEGER PRIMARY KEY AUTOINCREMENT,
queue TEXT NOT NULL,
payload TEXT NOT NULL,
state TEXT NOT NULL DEFAULT 'pending', -- 'pending' | 'processing'
priority INTEGER NOT NULL DEFAULT 0,
run_at INTEGER NOT NULL DEFAULT (unixepoch()), -- for delayed jobs
worker_id TEXT,
claim_expires_at INTEGER, -- visibility timeout
attempts INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 3,
created_at INTEGER NOT NULL DEFAULT (unixepoch()),
expires_at INTEGER -- job expiration
);
CREATE INDEX _honker_live_claim
ON _honker_live(queue, priority DESC, run_at, id)
WHERE state IN ('pending', 'processing');
CREATE TABLE _honker_dead (
id INTEGER PRIMARY KEY,
queue TEXT NOT NULL,
payload TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0,
run_at INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
created_at INTEGER NOT NULL DEFAULT (unixepoch()),
died_at INTEGER NOT NULL DEFAULT (unixepoch())
);
```
**Claim/ack/nack model**:
| Operation | SQL | Notes |
|-----------|-----|-------|
| Enqueue | `INSERT INTO _honker_live (queue, payload, run_at, priority, max_attempts, expires_at) VALUES (…)` | Returns auto-increment id |
| Claim | `UPDATE _honker_live SET state='processing', worker_id=?, claim_expires_at=unixepoch()+?, attempts=attempts+1 WHERE id IN (SELECT id FROM _honker_live WHERE queue=? AND state IN ('pending','processing') AND (expires_at IS NULL OR expires_at > unixepoch()) AND ((state='pending' AND run_at <= unixepoch()) OR (state='processing' AND claim_expires_at < unixepoch())) ORDER BY priority DESC, run_at ASC, id ASC LIMIT ?) RETURNING …` | One `UPDATE … RETURNING` via partial index |
| Ack | `DELETE FROM _honker_live WHERE id=? AND worker_id=? AND claim_expires_at >= unixepoch() RETURNING id` | Returns 1 if claim still valid, 0 if expired |
| Retry | `UPDATE _honker_live SET state='pending', run_at=unixepoch()+?, worker_id=NULL, claim_expires_at=NULL WHERE id=?` + notify on queue channel | If `attempts >= max_attempts`, DELETE from `_honker_live` and INSERT into `_honker_dead` |
| Fail | `DELETE FROM _honker_live WHERE id=? AND worker_id=? AND claim_expires_at >= unixepoch() RETURNING …` + `INSERT INTO _honker_dead` | Unconditionally move to dead letter |
| Heartbeat | `UPDATE _honker_live SET claim_expires_at=unixepoch()+? WHERE id=? AND worker_id=? AND state='processing'` | Extend claim for long-running handlers |
| Cancel | `DELETE FROM _honker_live WHERE id=? AND state IN ('pending', 'processing')` | Idempotent |
**Visibility timeout**: Default 300 seconds (`claim_expires_at = unixepoch() + 300`). If a worker crashes mid-job, the claim expires and another worker reclaims. `attempts` increments. After `max_attempts` (default 3), the row moves to `_honker_dead`.
**Priority**: Higher `priority` value = claimed first. The partial index on `(queue, priority DESC, run_at, id)` ensures claim path is bounded by working-set size, not history size.
**Delayed jobs**: Set `run_at` to a future timestamp. Workers only claim rows where `run_at <= unixepoch()`. The `run_at` deadline also wakes sleeping workers through `honker_queue_next_claim_at()`.
**Task expiration**: Set `expires_at` on enqueue. Expired jobs are filtered from the claim path. Call `queue.sweep_expired()` to move them to `_honker_dead` with `last_error='expired'`.
**Named locks**: `honker_lock_acquire(name, owner, ttl_s)` → 1 (got it) or 0 (held). `honker_lock_release(name, owner)` → 1 (released) or 0 (not yours). Uses `_honker_locks` table with TTL-based expiration. Primary use case: cron tasks that shouldn't overlap (leader election).
**Rate limiting**: `honker_rate_limit_try(name, limit, per)` → 1 (under limit) or 0 (at limit). Fixed-window counter. Rejected calls don't inflate the count.
**Batch operations**: `honker_claim_batch(queue, worker_id, n, timeout_s)` returns a JSON array of claimed jobs. `honker_ack_batch('[1,2,3]', worker_id)` acks multiple jobs. Ack is per-transaction for batch — honest bool return.
**Task result storage**: `honker_enqueue()` returns the job id. Workers can persist return values via `honker_result_save(id, value, ttl_s)`. Callers await results with `queue.wait_result(id, timeout)`. Opt-in (default `save_result=False`).
**Claim iterator pattern**:
```python
async for job in q.claim("worker-1"):
try:
send(job.payload)
job.ack()
except Exception as e:
job.retry(delay_s=60, error=str(e))
```
Each iteration is `claim_batch(worker_id, 1)`. Wakes on database update from any process, or when the next `run_at` / reclaim deadline arrives. 5-second paranoia poll is the only fallback.
**Queue notifications**: Each enqueue also fires a notification on `honker:<queue>` channel so workers wake immediately without waiting for the next poll cycle.
### 2.3 Stream — Durable Pub/Sub with Per-Consumer Offsets
**What it is**: Durable event stream where each named consumer tracks its own offset. Events persist until explicitly pruned. At-least-once delivery with configurable offset flush cadence.
**Schema**:
```sql
CREATE TABLE _honker_stream (
offset INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
key TEXT,
payload TEXT NOT NULL,
created_at INTEGER NOT NULL DEFAULT (unixepoch())
);
CREATE INDEX _honker_stream_topic
ON _honker_stream(topic, offset);
CREATE TABLE _honker_stream_consumers (
name TEXT NOT NULL,
topic TEXT NOT NULL,
offset INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (name, topic)
);
```
**API**:
| Function | Returns | Notes |
|----------|---------|-------|
| `honker_stream_publish(topic, key_or_null, payload_json)` | `offset` | INSERTs into `_honker_stream` + fires notification on `honker:stream:<topic>` |
| `honker_stream_read_since(topic, offset, limit)` | JSON array | Reads rows where `offset > ?` ordered by offset |
| `honker_stream_save_offset(consumer, topic, offset)` | 1 or 0 | Monotonic upsert — never rewinds. 1 = advanced, 0 = existing offset ≥ new |
| `honker_stream_get_offset(consumer, topic)` | offset or 0 | Returns saved offset for consumer/topic pair |
**Python binding**:
```python
stream = db.stream("user-events")
stream.publish({"user_id": uid, "change": "name"}, tx=tx)
async for event in stream.subscribe(consumer="dashboard"):
await push_to_browser(event)
```
**Subscribe behavior**:
1. Replay rows past `offset > saved_offset` in batches (default 1000 rows)
2. Transition to live delivery on commit wake
3. Auto-save offset at most every 1000 events or every 1 second (whichever first)
4. At-least-once: a crash re-delivers in-flight events up to the last flushed offset
5. Override auto-save with `save_every_n=` / `save_every_s=`; set both to 0 for manual control
**Transaction coupling**: `stream.publish(payload, tx=tx)` inserts into `_honker_stream` inside the caller's transaction. Rollback drops the event. This is the transactional outbox pattern without a separate dispatch table.
### 2.4 Scheduler — Time-Triggered Cron Tasks
**Schema**:
```sql
CREATE TABLE _honker_scheduler_tasks (
name TEXT PRIMARY KEY,
queue TEXT NOT NULL,
cron_expr TEXT NOT NULL,
payload TEXT NOT NULL,
priority INTEGER NOT NULL DEFAULT 0,
expires_s INTEGER,
next_fire_at INTEGER NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1
);
```
**API**:
```sql
SELECT honker_scheduler_register('nightly', 'backups', '0 3 * * *', '"go"', 0, NULL);
SELECT honker_scheduler_tick(unixepoch()); -- JSON: fires due
SELECT honker_scheduler_soonest(); -- min next_fire_at
SELECT honker_scheduler_unregister('nightly'); -- 1 = deleted
SELECT honker_scheduler_pause('nightly'); -- 1 = paused
SELECT honker_scheduler_resume('nightly'); -- 1 = resumed
SELECT honker_scheduler_list(); -- JSON array of all schedules
SELECT honker_scheduler_update('nightly', '0 4 * * *', NULL, NULL, NULL, 0);
```
Supports: 5-field cron, 6-field cron (with seconds), `@every <n><unit>` interval expressions.
**Leader election via named lock**: `db.lock('honker-scheduler', ttl=60)`. Two scheduler processes can't both fire. The lock is heartbeat-refreshed every 30s.
**Missed-fire catch-up**: If the scheduler was down for 4 hours with an hourly schedule, the first iteration fires all 4 missed boundaries (with `expires=` to drop stale ones).
**Fires = enqueue**: The scheduler never runs handlers. It enqueues into the task queue. Regular workers consume.
### 2.5 Outbox Pattern
The `outbox` is a convenience wrapper around the `Queue` primitive:
```python
db.outbox("emails", delivery=send_email)
db.outbox("emails").enqueue({"to": "alice@example.com"}, tx=tx)
db.outbox("emails").run_worker("worker-1")
```
Failures retry with exponential backoff (`base_backoff_s * 2^(attempts-1)`) up to `max_attempts`, then land in `_honker_dead`.
---
## 3. Persistence and Reliability
### Durability Guarantees
- **Atomic commit**: Business write + side-effect enqueue/event/notify commit together or roll back together. This is SQLite ACID — the transactional outbox pattern is built into the primitives, not bolted on.
- **SIGKILL safety**: Verified in `tests/test_crash_recovery.py`. Subprocess killed pre-COMMIT → `PRAGMA integrity_check == 'ok'`, zero in-flight rows, no stale write lock, queue round-trip works post-crash.
- **Worker crash recovery**: If a worker crashes mid-job, the claim expires after `visibility_timeout_s` (default 300s) and another worker reclaims. `attempts` increments on each claim. After `max_attempts` (default 3), the row moves to `_honker_dead`.
- **Stream at-least-once**: Offsets auto-flush every 1000 events or 1 second. A crash re-delivers in-flight events up to the last flushed offset. The crash window is bounded by the flush thresholds.
- **Notify has no replay**: Listeners attach at `MAX(id)`. Pruned events are gone. For durable replay, use streams.
### WAL Mode
Recommended default (`journal_mode = WAL`). Gives concurrent readers with one writer and efficient fsync batching (`wal_autocheckpoint = 10000`). Other journal modes work but lose WAL's concurrent-read-while-writing property. Wake detection (`PRAGMA data_version`) works in all journal modes.
### What Happens on Crash
| Scenario | Result |
|----------|--------|
| Process SIGKILL mid-TRANSACTION | SQLite atomic-commit rollback. In-flight write did not land. Fresh process can acquire write lock immediately. |
| Worker process crash mid-job | Claim expires after visibility_timeout. Another worker reclaims. `attempts` increments. |
| Stream consumer crash | Resumes from last auto-saved offset (at-least-once). Pending offset is lost. |
| Database file replaced (litestream restore) | Watcher panics with clear error message. All subscribers see error from update_events(). Must reopen database. |
### What Honker Does NOT Provide
- **Multi-writer replication**: SQLite's locking is for single-host. Two servers writing one `.db` over NFS will corrupt it. Shard by file or switch to Postgres.
- **In-memory database support**: `:memory:` creates a separate database per connection, splitting writer/readers/watchers. Use temp file-backed `.db` for tests.
- **Cross-node distribution**: Honker is single-machine. No built-in mechanism for distributing events across nodes. (This is intentional — see alknet relevance below.)
- **Task pipelines/chains/groups/chords**. Deliberately not built.
- **Workflow orchestration with DAGs**. Deliberately not built.
- **Ordering guarantees across queues**. Each queue is independent.
- **Exactly-once delivery**. Honker provides at-least-once. Idempotent handlers are the user's responsibility.
---
## 4. Performance
### Benchmarks (M-series, release build, median of 3)
| Operation | Throughput |
|-----------|-----------|
| enqueue (1/tx) | ~8,000/sec |
| enqueue (100/tx) | ~110,000/sec |
| claim + ack (individual) | ~4,500/sec |
| claim_batch + ack_batch (32) | ~75,000/sec |
| claim_batch + ack_batch (128) | ~110,000/sec |
| async iter end-to-end | ~6,500/sec |
| stream replay | ~1,000,000/sec |
| stream live e2e p50 | 0.24ms |
| stream live e2e p99 | 8ms |
### Cross-Process Wake Latency
Median ~1-2ms on M-series, bounded by the 1ms `PRAGMA data_version` poll cadence. 600-second soak test under sustained ~75 commits/sec showed zero missed wakes, zero drift, `PRAGMA integrity_check = ok`.
### Claim Performance at Scale
With 100,000 dead rows in `_honker_dead`:
| Operation | Claim+ack |
|-----------|-----------|
| 0 dead rows (fresh DB) | ~4,000/sec |
| 100k dead rows | ~3,500/sec |
The partial index `(queue, priority DESC, run_at, id) WHERE state IN ('pending','processing')` keeps the claim hot path bounded by working-set size, not history size.
### How It Compares to Polling
Prior to honker's wake mechanism, the alternative would be application-level polling (e.g., `SELECT … WHERE id > last_seen` every N seconds). Honker replaces this with a single-digit-microsecond PRAGMA read. 100 subscribers still = 1 poll thread. The over-triggering trade-off (waking all subscribers on any commit) is explicitly chosen over potentially missing a wake.
---
## 5. SQLite Integration
### Loading the Extension
```sql
-- Any SQLite 3.9+ client
.load ./libhonker_ext
SELECT honker_bootstrap();
```
`honker_bootstrap()` is idempotent — it runs `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` for all schema tables.
### Compile/Load Flags
For Rust integration via `rusqlite`:
```toml
[dependencies]
honker-core = "0.2.3"
rusqlite = { version = "0.39.0", features = ["functions", "hooks"] }
```
Then in Rust:
```rust
use honker_core::{attach_notify, attach_honker_functions, bootstrap_honker_schema, open_conn};
let conn = open_conn("app.db", true)?; // true = install notify
attach_honker_functions(&conn)?;
bootstrap_honker_schema(&conn)?;
```
For the loadable extension:
```bash
cargo build --release -p honker-extension
# Produces: target/release/libhonker_ext.so (or .dylib, .dll)
```
### Rust Crate Usage
```rust
use honker_core::SharedUpdateWatcher;
let watcher = SharedUpdateWatcher::new(db_path.clone());
let (sub_id, rx) = watcher.subscribe();
// In a loop:
match rx.recv_timeout(Duration::from_secs(5)) {
Ok(()) => { /* re-read state from SQLite */ },
Err(RecvTimeoutError::Timeout) => { /* paranoia poll */ },
Err(RecvTimeoutError::Disconnected) => { /* watcher died, reopen */ },
}
watcher.unsubscribe(sub_id);
watcher.close()?;
```
### Using with ORM Connections
Load `libhonker_ext` on the ORM's connection and call `honker_bootstrap()` inside the ORM's transaction:
```python
# SQLAlchemy
@event.listens_for(engine, "connect")
def _load_honker(conn, _):
honker.load_extension(conn)
conn.execute("SELECT honker_bootstrap()")
with Session(engine) as s, s.begin():
s.add(Order(user_id=42))
s.execute(text("SELECT honker_enqueue(:q, :p, NULL, NULL, 0, 3, NULL)"),
{"q": "emails", "p": '{"to":"alice"}'})
```
### PRAGMA Defaults
Applied on every connection opened via `open_conn`:
```sql
PRAGMA journal_mode = WAL;
PRAGMA synchronous = NORMAL; -- fsync WAL at checkpoint, not every commit
PRAGMA busy_timeout = 5000; -- wait up to 5s for writer lock
PRAGMA foreign_keys = ON;
PRAGMA cache_size = -32000; -- 32MB page cache (default was 2MB)
PRAGMA temp_store = MEMORY; -- temp B-trees in RAM
PRAGMA wal_autocheckpoint = 10000; -- fsync every 10k WAL pages
```
---
## 6. Complete API Surface
### Notification Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `notify(channel, payload)` | Scalar, 2 args | `rowid` | INSERTs into `_honker_notifications`, returns auto-generated id |
### Queue Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_bootstrap()` | 0 args | `1` | Creates all schema tables/indexes. Idempotent. |
| `honker_enqueue(queue, payload, run_at_or_null, delay_or_null, priority, max_attempts, expires_or_null)` | 7 args | `id` | INSERTs job. Delay overrides run_at. |
| `honker_claim_batch(queue, worker_id, n, timeout_s)` | 4 args | JSON array | Claims up to `n` jobs. Each gets `claim_expires_at = now + timeout_s`. |
| `honker_ack_batch(ids_json, worker_id)` | 2 args | `count` | ACKs (DELETEs) claimed jobs. `ids_json` is `[1,2,3]`. |
| `honker_ack(job_id, worker_id)` | 2 args | `1` or `0` | Single-job ack. Returns 0 if claim expired. |
| `honker_retry(job_id, worker_id, delay_s, error)` | 4 args | `1` or `0` | Retries (flips back to pending) or fails to dead if `attempts >= max_attempts`. |
| `honker_fail(job_id, worker_id, error)` | 3 args | `1` or `0` | Unconditionally moves to `_honker_dead`. |
| `honker_heartbeat(job_id, worker_id, extend_s)` | 3 args | `1` or `0` | Extends claim for long-running handlers. |
| `honker_cancel(job_id)` | 1 arg | `1` or `0` | Removes pending/processing row. Idempotent. |
| `honker_get_job(job_id)` | 1 arg | JSON or `""` | Read job state. Pure read. |
| `honker_sweep_expired(queue)` | 1 arg | `count` | Moves expired pending jobs to `_honker_dead`. |
| `honker_queue_next_claim_at(queue)` | 1 arg | `unix_ts` or `0` | Earliest future deadline (run_at or claim_expires_at + 1). |
### Stream Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_stream_publish(topic, key_or_null, payload_json)` | 3 args | `offset` | INSERTs event + fires notification |
| `honker_stream_read_since(topic, offset, limit)` | 3 args | JSON array | Reads events with `offset > ?` |
| `honker_stream_save_offset(consumer, topic, offset)` | 3 args | `1` or `0` | Monotonic upsert. 0 = existing offset ≥ new |
| `honker_stream_get_offset(consumer, topic)` | 2 args | `offset` or `0` | Returns saved offset |
### Lock Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_lock_acquire(name, owner, ttl_s)` | 3 args | `1` or `0` | 1 = acquired, 0 = held |
| `honker_lock_release(name, owner)` | 2 args | `1` or `0` | 1 = released, 0 = not yours |
### Rate Limit Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_rate_limit_try(name, limit, per)` | 3 args | `1` or `0` | 1 = under limit, 0 = at limit |
| `honker_rate_limit_sweep(older_than_s)` | 1 arg | `count` | Prunes expired windows |
### Scheduler Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_scheduler_register(name, queue, cron_expr, payload, priority, expires_s_or_null)` | 6 args | `1` | Upserts task. Computes next_fire_at. |
| `honker_scheduler_unregister(name)` | 1 arg | `0` or `1` | Deletes task. |
| `honker_scheduler_tick(now_unix)` | 1 arg | JSON array | Fires due tasks, enqueues payloads, advances next_fire_at. |
| `honker_scheduler_soonest()` | 0 args | `unix_ts` or `0` | Earliest next_fire_at for sleep duration calculation. |
| `honker_scheduler_pause(name)` | 1 arg | `0` or `1` | Toggles `enabled = 0`. |
| `honker_scheduler_resume(name)` | 1 arg | `0` or `1` | Toggles `enabled = 1`. |
| `honker_scheduler_list()` | 0 args | JSON array | Returns all schedules with state. |
| `honker_scheduler_update(name, cron_expr_or_null, payload_or_null, priority_or_null, expires_s_or_null, touch_expires)` | 6 args | `0` or `1` | Mutates schedule fields. Recomputes next_fire_at if cron_expr changed. |
| `honker_cron_next_after(expr, from_unix)` | 2 args | `unix_ts` | Pure deterministic function. 5-field, 6-field, or `@every <n><unit>`. |
### Result Functions
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_result_save(job_id, value_json, ttl_s)` | 3 args | `1` | UPSERTs result. `ttl_s=0` = no expiration. |
| `honker_result_get(job_id)` | 1 arg | `value` or `NULL` | Returns result or NULL if expired/missing. |
| `honker_result_sweep()` | 0 args | `count` | Prunes expired result rows. |
### Watcher Functions (Extension ABI)
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_update_watcher_open(db_path, backend)` | 2 SQL args | `id` | Opens a watcher handle. For Elixir and extension consumers. |
| `honker_update_watcher_wait(id, timeout_ms)` | 2 SQL args | `1`/`0`/`-1` | 1 = update observed, 0 = timeout, -1 = disconnected |
| `honker_update_watcher_close(id)` | 1 SQL arg | `1` | Closes watcher handle. |
C ABI (for Go, .NET, C++, Ruby bindings that route through the extension):
| Function | Signature | Returns | Notes |
|----------|-----------|---------|-------|
| `honker_watcher_open(db_path, backend, err_buf, err_buf_len)` | C ABI | `*mut HonkerWatcherHandle` | Opens a core-backed update watcher. |
| `honker_watcher_wait(handle, timeout_ms)` | C ABI | `1`/`0`/`-1`/`-2` | 1 = update, 0 = timeout, -1 = closed, -2 = panic |
| `honker_watcher_close(handle)` | C ABI | void | Closes and frees the handle. |
### Tables
| Table | Purpose |
|-------|---------|
| `_honker_live` | Pending + processing jobs. Partial index for fast claims. |
| `_honker_dead` | Terminal jobs (retry-exhausted or explicitly failed). Never scanned by claim path. |
| `_honker_notifications` | Ephemeral notify/listen messages. Not auto-pruned. |
| `_honker_stream` | Durable stream events with auto-incrementing offsets. |
| `_honker_stream_consumers` | Per-consumer stream offsets. Monotonic upsert. |
| `_honker_locks` | Named advisory locks with TTL expiration. |
| `_honker_rate_limits` | Fixed-window rate limit counters. |
| `_honker_scheduler_tasks` | Cron/schedule task definitions with next_fire_at. |
| `_honker_results` | Task result storage with TTL expiration. |
---
## 7. Comparison to Postgres (pg_notify)
| Feature | Honker | pg_notify |
|---------|--------|-----------|
| **Delivery model** | Table-backed `INSERT` in transaction | In-memory NOTIFY with LISTEN callback |
| **Persistence** | Rows survive restart. Not auto-pruned. | Ephemeral — lost on restart, not replayed. |
| **Transactional coupling** | `notify(channel, payload)` inside `BEGIN IMMEDIATE; INSERT; COMMIT` — atomic with business write | NOTIFY fires at COMMIT inside the same transaction. Atomic with business write. |
| **Retry / visibility timeout** | Queue has `claim_expires_at`, `attempts`, `max_attempts`, dead-letter. | No retry. No visibility timeout. |
| **Delayed delivery** | `run_at` for scheduled delivery. Jobs only claimable after deadline. | No scheduling. |
| **Cross-process wake** | `PRAGMA data_version` polling at ~1ms cadence. SharedUpdateWatcher fans out to N subscribers. | Postgres notifies listeners via its inter-process communication. |
| **Priority** | Queue priority via partial index `(queue, priority DESC, run_at, id)`. | No priority. |
| **Rate limiting** | Built-in fixed-window `rate_limit_try`. | No rate limiting. |
| **Named locks** | TTL-based advisory locks in `_honker_locks`. | `pg_advisory_lock` (similar concept, different implementation). |
| **Cron scheduling** | Built-in scheduler with 5-field/6-field cron + `@every` intervals. | Needs pg-boss/Oban/cron extension. |
| **Stream offsets** | Per-consumer tracked offsets with monotonic upsert. | No built-in stream offsets. |
| **Multi-process** | Single-machine, single-writer. | Multi-process, multi-writer natively. |
| **Durability** | SQLite ACID. WAL mode for concurrent readers. | Postgres ACID. Full write-ahead logging. |
**What honker gives you that pg_notify alone doesn't**:
1. **Retry with exponential backoff** — automatic re-delivery on failure
2. **Visibility timeout** — crashed workers don't permanently lose messages
3. **Dead-letter queue** — exhausted retries land in `_honker_dead` for inspection
4. **Delayed jobs** — `run_at` for future delivery
5. **Prioritization** — `priority` column in claim index
6. **Transactional outbox** — business write + enqueue/event in one transaction, without adding Redis/Celery
7. **Task result storage** — workers can persist return values; callers can await results
8. **Durable streams** — per-consumer offsets with at-least-once delivery
9. **Cron scheduling** — built-in periodic tasks with leader election
10. **Named locks and rate limiting** — built-in coordination primitives
**What you'd need to add if you used Postgres instead**: pg-boss, Oban, or similar PgBoss-style packages provide many of these features, but they require Postgres as the database. Honker exists for the case where SQLite is already the primary datastore.
---
## 8. Comparison to Other Message Systems
| Feature | Honker | Redis Pub/Sub | NATS | Kafka |
|---------|--------|--------------|------|-------|
| **Persistence** | SQLite tables (disk) | In-memory only (unless RDB/AOF) | In-memory (JetStream adds persistence) | Persistent log |
| **Transactional coupling** | Business write + enqueue in one tx | Not atomic with business data | Not atomic with business data | Not atomic with business data |
| **Delivery guarantee** | At-least-once | At-most-once (fire-and-forget) | At-most-once (core); at-least-once (JetStream) | At-least-once with consumer offsets |
| **Retry/visibility** | Built-in (claim timeout, retry, dead-letter) | None (messages disappear if no consumer) | None (core); redelivery (JetStream) | Consumer group offsets |
| **Priority** | Yes (partial index) | No | No | No |
| **Delayed delivery** | Yes (`run_at`) | No (requires sorted sets hack) | No | No (requires time-based logic) |
| **Single-node complexity** | Zero — just a `.db` file | Requires Redis server | Requires NATS server | Requires Kafka cluster |
| **Cross-process wake latency** | 1-2ms | ~0.1ms | ~0.1ms | ~1-5ms |
| **Cross-node distribution** | None (single machine) | Pub/Sub is fan-out to connected clients | JetStream supports clustering | Built for distributed |
| **Dependency** | SQLite (already in your stack) | Additional server | Additional server | Additional cluster |
| **Schema coupling** | Same file as business data — dual-write impossible | Separate system — dual-write risk | Separate system — dual-write risk | Separate system — dual-write risk |
| **Language support** | Python, Node, Rust, Go, Ruby, Bun, Elixir, C++, .NET, JVM, Kotlin | Many (but protocol, not SQL) | 40+ client libraries | Many client libraries |
| **Dead-letter queue** | Built-in `_honker_dead` | None | JetStream has DLQ | DLQ via configuration |
**When honker is the right choice**: SQLite is already your primary datastore, and you need pub/sub + queue + scheduling without introducing Redis/Celery/NATS. The dual-write problem between your business tables and the queue disappears.
**When honker is NOT the right choice**: Multi-node deployments, multi-writer sharding, need for cross-datacenter replication, or workloads exceeding single-machine throughput.
---
## 9. Relevance to Alknet
### 9.1 Alignment with Event Boundary Discipline (ADR-032)
ADR-032 defines three communication layers:
```
Call Protocol (Layer 3, external, JSON)
└── irpc Service (Layer 3, internal, postcard)
└── Honker Streams (Domain events, within service boundary)
```
**Honker's single-machine model is exactly right for the bottom layer.** Domain events in alknet are internal to the service that owns that data — `nodes:created`, `edges:deleted`, `accounts:updated`. These never cross the service boundary without projection into a call protocol `EventEnvelope`.
The integration plan (Phase 2.2) explicitly lists honker integration patterns for alknet-storage:
| Feature | Use Case |
|---------|----------|
| `stream_publish` / `subscribe` | Durable pub/sub for node/edge/membership changes |
| `notify` / `listen` | Ephemeral pub/sub for real-time control channel events |
| `queue` / `claim` / `ack` | Task queue for async operations |
### 9.2 Patterns from Honker for alknet-storage Adoption
**Map honker's primitives to alknet-storage's internal events**:
| Alknet Domain Event | Honker Primitive | Stream Name |
|---------------------|------------------|-------------|
| Node created | `stream.publish("nodes:created", ...)` | `nodes:created` |
| Node updated | `stream.publish("nodes:updated", ...)` | `nodes:updated` |
| Node deleted | `stream.publish("nodes:deleted", ...)` | `nodes:deleted` |
| Edge created | `stream.publish("edges:created", ...)` | `edges:created` |
| Account updated | `stream.publish("accounts:updated", ...)` | `accounts:updated` |
| ACL rule changed | `stream.publish("acl:changed", ...)` | `acl:changed` |
**Map honker's task queue to alknet's async operations**:
| Alknet Async Task | Honker Queue |
|-------------------|-------------|
| Key rotation | `queue("key-rotation")` |
| Certificate renewal | `queue("cert-renewal")` |
| Audit log archival | `queue("audit-archival")` |
| Node encryption/decryption | `queue("node-crypto")` |
**Map honker's notify/listen to real-time events**:
| Alknet Real-Time Event | Honker Channel |
|------------------------|---------------|
| SSH connection opened | `notify("ssh:connected", ...)` |
| Config reload triggered | `notify("config:reload", ...)` |
| Forwarding rule activated | `notify("forwarding:activated", ...)` |
### 9.3 Replicating Honker Patterns with Postgres for Production
If alknet-storage is backed by Postgres in production deployments (the storage spec mentions `rusqlite` but leaves room for alternative backends), the following Postgres equivalents would be needed:
| Honker Primitive | Postgres Equivalent | What's Lost |
|-----------------|---------------------|-------------|
| `notify/listen` | `pg_notify` + `LISTEN` | Postgres NOTIFY is ephemeral (lost on restart). Honker's table-backed notifications persist. Need to add a `_notifications` table and polling. |
| `stream_publish/subscribe` | `pg_notify` + consumer offset table | No built-in per-consumer offset tracking. Would need a `_stream_consumers` table and polling/cursor logic. |
| `queue/claim/ack` | pg-boss / Oban | These exist and are production-quality. Honker's simplicity (one table, partial index) is lost. Need a dependency on Oban or pg-boss. |
| `run_at` (delayed jobs) | Oban's `scheduled_at` / pg-boss's `startAfter` | Available in both. |
| `claim_expires_at` (visibility timeout) | Oban's `attempted_at` + `max_attempts` | Available in both. |
| `honker_lock_acquire/release` | `pg_advisory_lock` | Built-in, similar concept. |
| `honker_rate_limit_try` | Custom table or Redis | Postgres has no built-in rate limiting. |
| Transactional coupling | Same tx | Naturally available: `INSERT INTO orders ...; INSERT INTO _honker_live ...;` both in the same Postgres tx. |
| Scheduler | pg-boss `schedule()` or Oban's `Oban.insert(CronWorker, ...)` | Available in both. |
**What would be lost switching to Postgres + pg-boss/Oban**:
- **Schema simplicity**: Honker uses 2 tables for 90% of queue operations. pg-boss uses more tables. Oban uses per-queue tables.
- **Zero-dependency**: Honker is a SQLite extension. No Redis, no Celery, no broker. pg-boss requires Postgres. Oban requires Postgres + Elixir.
- **Cross-language transparency**: Any SQLite client can `SELECT load_extension('honker')` and get the same features. Postgres requires language-specific client libraries.
- **File-based deployment**: Copy the `.db` file. Done. Postgres requires a server.
**Recommendation for alknet-storage**: Start with honker on SQLite for self-hosted/edge deployments. For production Postgres deployments, create an abstraction layer in `alknet-storage` that implements the same `EventStream`, `TaskQueue`, and `NotificationChannel` traits against both backends. The honker-on-SQLite implementation is the reference; the Postgres implementation uses `pg_notify` + offset tables + Oban/pg-boss.
### 9.4 Honker's Queue/Claim Model and alknet's Call Protocol
The call protocol's `EventEnvelope` frames are the integration boundary (ADR-033). When a domain event needs to cross node boundaries, it must be projected:
```
Honker stream event (internal)
→ Projection function
→ EventEnvelope frame (external, call protocol)
→ Transported over SSH/QUIC/DNS
→ Received by remote node
→ May trigger local Honker stream event on remote node
```
The **queue/claim model maps to async call protocol operations**:
1. **call.requested** → Honker `queue.enqueue({"operation": "/head/auth/verify", "input": {...}})`
2. **Worker claims the job** → Like a worker process picking up a call request
3. **job.ack()** → call.responded with the result
4. **job.retry()** → Call timeout / retry logic (but this is at the transport layer, not the queue)
5. **job fails → _honker_dead** → Dead letter equivalent for failed call protocol operations
The **key difference**: alknet's call protocol is synchronous request-response at the transport layer, while honker's queue is async at-least-once. They serve different purposes:
- **Call protocol**: "I need you to verify this pubkey NOW" (synchronous, cross-node)
- **Honker queue**: "Process this key rotation in the background" (asynchronous, within-node)
For **cross-node task distribution**, honker's queue should NOT be the transport. Instead:
1. A domain event (honker stream) in the storage service triggers a projection
2. The projection creates an `EventEnvelope` frame
3. The call protocol delivers it to remote nodes
4. Remote nodes may enqueue it into their own honker queues for local processing
### 9.5 Cross-Node Event Distribution
**Honker is single-node by design.** This is correct for alknet's architecture because:
1. **Domain events stay within the service boundary** (ADR-032). Honker streams are for internal state reconstruction, not cross-node distribution.
2. **Integration events cross boundaries via the call protocol.** When a domain event in the storage service needs to be communicated to another node, it's projected into an `EventEnvelope` frame and sent over the wire.
3. **Each node has its own `.db` file** with its own honker streams. This is a feature, not a limitation — it enforces the event boundary discipline.
The bridge pattern:
```
Node A (storage service):
1. Business write INSERTs into SQLite
2. stream.publish("nodes:created", {node_id: 42}) in same tx
3. A local subscriber detects the event
4. Projects it into EventEnvelope {operation: "/head/nodes/created", data: {node_id: 42}}
5. Sends via call protocol over SSH/QUIC/DNS to Node B
Node B (receiver):
1. Receives EventEnvelope via call protocol
2. Enqueues locally: queue("incoming-events").enqueue({source: "node-A", event: ...})
3. Or publishes locally: stream.publish("remote:nodes:created", {node_id: 42})
```
This preserves the three-layer model while respecting honker's single-machine design.
### 9.6 Honker Patterns and Integration Plan Mapping
The integration plan (Phase 2.2, alknet-storage) references these honker patterns. Here's the direct mapping:
| Plan Reference | Honker Primitive | Implementation Notes |
|---------------|-----------------|---------------------|
| `stream_publish/subscribe` | `db.stream("topic").publish(data, tx=tx)` + `async for event in stream.subscribe(consumer="name")` | Used for domain events within alknet-storage. Each metagraph change (node/edge created/updated/deleted) publishes to a stream. Consumers (local reactive logic, SSE endpoints) subscribe. |
| `notify/listen` | `tx.notify("channel", data)` + `async for n in db.listen("channel")` | Used for ephemeral real-time signals. SSH connection events, config reload triggers, forwarding rule activation. No persistence needed. |
| `queue/claim` | `queue.enqueue(data, tx=tx)` + `async for job in queue.claim(worker_id)` | Used for background tasks. Key rotation, certificate renewal, audit log archival, batch operations. The `tx=tx` parameter ensures atomicity with business writes. |
**Implementation approach for alknet-storage (Rust)**:
Use `honker-core` directly (not the Python binding). The Rust crate exposes:
- `open_conn(path, install_notify)` — open a connection with PRAGMA defaults
- `attach_honker_functions(&conn)` — register all SQL functions
- `bootstrap_honker_schema(&conn)` — create tables
- `SharedUpdateWatcher::new(db_path)` — the wake listener
Or load the extension via `rusqlite`:
```rust
use rusqlite::Connection;
let conn = Connection::open("alknet.db")?;
conn.load_extension("libhonker_ext", None)?;
conn.execute_batch("SELECT honker_bootstrap()")?;
```
### 9.7 Rust Integration: honker-core vs honker-rs
Two options for Rust integration in alknet-storage:
**Option A: Use `honker-core` directly**
The `honker-core` crate provides:
- `attach_notify(&conn)` — `_honker_notifications` table + `notify()` SQL function
- `attach_honker_functions(&conn)` — all `honker_*` SQL functions
- `bootstrap_honker_schema(&conn)` — all table/index creation
- `SharedUpdateWatcher` — the wake mechanism
- `open_conn(path, install_notify)` — connection factory with PRAGMA defaults
This gives you raw SQL access. You call `conn.query_row("SELECT honker_enqueue(…)")` etc. Maximum control, minimum abstraction.
**Option B: Use `honker-rs` ergonomic wrapper**
The `packages/honker-rs` crate provides:
- `Database::open(path)` — opens `system.db`
- `db.queue("name")` — `Queue` handle with `.enqueue()`, `.claim_batch()`, `.ack_batch()`
- `db.stream("name")` — `Stream` handle with `.publish()`, `.subscribe()`
- `db.listen("channel")` — async listener
- `db.outbox("name", delivery_fn)` — outbox pattern
- `db.lock("name", owner, ttl)` — named lock
- `db.scheduler()` — cron scheduler
**Recommendation**: Start with `honker-core` + direct SQL. The schema and functions are stable and well-tested. Wrap in application-level methods as needed. `honker-rs` may not expose all features (e.g., the scheduler pause/resume/list/update functions added in Phase Mantle). Using `honker-core` gives maximum flexibility while maintaining a single source of truth for SQL behavior.
---
## 10. Open Questions for Alknet
1. **Should alknet-storage bundle honker as a Rust crate dependency, or load the extension at runtime?**
- Bundling `honker-core` gives compile-time verification. Loading the extension requires shipping `libhonker_ext.so/.dylib/.dll` alongside the binary.
- Recommendation: Bundle `honker-core` as a crate dependency for the Rust implementation. Extension loading is for language bindings that can't link Rust code directly.
2. **Should the `alknet-storage` crate depend on `honker` (the Python package) or `honker-core` (the Rust rlib)?**
- `honker-core` (Rust rlib) — correct choice for a Rust crate. `honker` is the Python binding.
- The Crate dependency in storage.md currently lists `honker = "0.x"`. This should be `honker-core = "0.2"`.
3. **How does the Rust `SharedUpdateWatcher` integrate with tokio?**
- `SharedUpdateWatcher::subscribe()` returns a `std::sync::mpsc::Receiver<()>`, which is blocking. For tokio integration, wrap in `tokio::task::spawn_blocking` or use `tokio::sync::mpsc` as a bridge.
- Alternatively, use `UpdateWatcher::spawn()` directly and convert ticks to tokio notifications.
4. **Should alknet-storage abstract over honker-specific table names?**
- Honker prefixes all internal tables with `_honker_` (e.g., `_honker_live`, `_honker_stream`). Alknet-storage should treat these as honker's internal schema and not directly query them for application logic.
- Application-level tables (like `nodes`, `edges`, `accounts`) should use their own namespacing convention. Honker's tables coexist in the same `.db` file.
5. **Multi-tenant support**: Honker queues and streams are identified by name strings (e.g., `"emails"`, `"user-events"`). For alknet's multi-tenant model (system DB vs tenant DB), each tenant gets its own `.db` file with its own honker tables. Cross-tenant events must go through the call protocol — never by direct honker stream subscription across database files.
6. **Database file management**: Alknet-storage's system DB (`system.db`) and tenant DBs (`tenant-{orgId}.db`) should each have their own honker instance. The `SharedUpdateWatcher` is per-database, so 100 active tenants = 100 poll threads. This is fine for the expected alknet deployment size, but worth monitoring thread count in large deployments.
---
## 11. License and Maturity
- **License**: Apache 2.0 OR MIT (dual-licensed). Fully permissive for integration.
- **Maturity**: Alpha software (noted in README). Better than experimental but not beta-quality yet.
- **Status**: Active development. Regular commits. Cross-language interop tests. 180+ Python tests, 12+ Rust tests. Crash recovery verified. 600-second soak test under sustained writes.
- **Breaking changes risk**: The project is pre-1.0. Some table names still reference "joblite" and "litenotify" in the CHANGELOG (historical names). Current names use `_honker_` prefix. The API surface is stabilizing but may change.
- **Recommendation**: Pin to a specific `honker-core` version in `alknet-storage`'s `Cargo.toml`. The schema migration path (seen in `bootstrap_honker_schema`'s ALTER TABLE for `enabled` column) shows the project handles migrations.
---
## References
- [Honker GitHub Repository](https://github.com/russellromney/honker) — Primary source for all code and documentation
- [Honker README](https://github.com/russellromney/honker/blob/main/README.md) — Feature overview, quick start, architecture, performance
- [Honker BINDINGS.md](https://github.com/russellromney/honker/blob/main/BINDINGS.md) — Language binding support matrix
- [Honker ROADMAP.md](https://github.com/russellromney/honker/blob/main/ROADMAP.md) — Future work phases, planned features (singleton/dedup, state events, queue stats, per-queue config)
- [Honker CHANGELOG.md](https://github.com/russellromney/honker/blob/main/CHANGELOG.md) — Detailed history of all changes, performance passes, and architecture decisions
- [Honker honker-core/src/lib.rs](https://github.com/russellromney/honker/blob/main/honker-core/src/lib.rs) — Core Rust implementation: Writer, Readers, UpdateWatcher, SharedUpdateWatcher, schema, PRAGMAs
- [Honker honker-core/src/honker_ops.rs](https://github.com/russellromney/honker/blob/main/honker-core/src/honker_ops.rs) — All SQL function implementations: enqueue, claim, ack, retry, stream, lock, rate limit, scheduler
- [Honker honker-extension/src/lib.rs](https://github.com/russellromney/honker/blob/main/honker-extension/src/lib.rs) — Loadable extension entry point and C ABI for watcher
- [alknet ADR-032: Event Boundary Discipline](../../architecture/decisions/032-event-boundary-discipline.md) — Domain events stay within service boundary
- [alknet Integration Plan](../../research/integration-plan.md) — Phase 2.2: alknet-storage honker integration
- [alknet Storage Spec](../../architecture/storage.md) — alknet-storage crate design and honker integration table

170
docs/research/references/nats.rs/nats-async/01-overview-and-architecture.md
View File

@@ -1,170 +0,0 @@
# async-nats: Overview & Architecture
**Crate**: `async-nats`
**Version**: 0.49.1
**Repository**: https://github.com/nats-io/nats.rs
**License**: Apache-2.0
**Rust Edition**: 2021
**MSRV**: 1.88.0
**Async Runtime**: Tokio
## What is async-nats?
`async-nats` is the official async Rust client for the [NATS messaging system](https://nats.io). It provides a Tokio-based asynchronous interface to NATS server features including:
- **Core NATS** — publish/subscribe, request/reply, queue groups
- **JetStream** — persistent stream-based messaging with at-least-once and exactly-once semantics
- **Key-Value Store** — KV abstraction built on JetStream streams
- **Object Store** — large-object storage built on JetStream streams
- **Service API** — microservice request/reply pattern with built-in PING/INFO/STATS verbs
The crate is positioned as the **core client** in the NATS Rust ecosystem. A separate project, [Orbit](https://github.com/synadia-io/orbit.rs), provides higher-level opinionated abstractions on top.
```
┌──────────────────────────────────────────────────────┐
│ Application code │
└──────────────┬───────────────────────────┬───────────┘
│ │
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Orbit crates │ uses │ async-nats (core) │
│ (opinionated, │──────▶│ (parity, stable, │
│ per-crate semver) │ │ protocol-level) │
└───────────────────┘ └─────────┬─────────┘
│
▼
┌─────────────┐
│ nats-server │
└─────────────┘
```
## Feature Flags
Features are extensive and control which subsystems are compiled:
| Feature | Default | Description |
|---------|---------|-------------|
| `jetstream` | ✅ | JetStream API (streams, consumers, publish) |
| `kv` | ✅ | Key-Value store (depends on `jetstream`) |
| `object-store` | ✅ | Object store (depends on `jetstream` + `crypto`) |
| `service` | ✅ | Service API (microservice pattern) |
| `nkeys` | ✅ | NKey/JWT authentication |
| `nuid` | ✅ | NUID-based unique ID generation |
| `crypto` | ✅ | Cryptographic primitives (SHA-256 for object store) |
| `websockets` | ✅ | WebSocket transport (`ws://`/`wss://`) |
| `ring` | ✅ | Use `ring` as TLS crypto backend |
| `aws-lc-rs` | ❌ | Use `aws-lc-rs` as TLS crypto backend |
| `fips` | ❌ | FIPS 140-2 compliant via `aws-lc-rs` |
| `chrono` | ❌ | Use `chrono` instead of `time` for datetime types |
| `server_2_10` | ✅ | Server 2.10+ features |
| `server_2_11` | ✅ | Server 2.11+ features |
| `server_2_12` | ✅ | Server 2.12+ features |
| `server_2_14` | ✅ | Server 2.14+ features |
| `experimental` | ❌ | Experimental features |
## Source Structure
```
async-nats/src/
├── lib.rs # Entry point: connect(), ServerInfo, Command, ClientOp, ServerOp,
│ ConnectionHandler, Subscriber, Event, ServerAddr, ConnectInfo
├── client.rs # Client struct, publish/subscribe/request/drain/flush APIs,
│ Request builder, Statistics, trait definitions
├── connection.rs # Framed connection: NATS protocol parser/serializer,
│ read/write buffer management, WebSocket adapter
├── connector.rs # Server pool, reconnection logic, TLS setup, DNS resolution,
│ authentication handshake
├── options.rs # ConnectOptions builder, auth methods, TLS config, callbacks
├── auth.rs # Auth struct (username, password, token, JWT, nkey, signature)
├── auth_utils.rs # Credentials file parsing (JWT + NKey seed)
├── message.rs # Message (inbound), OutboundMessage (outbound)
├── header.rs # HeaderMap, HeaderName, HeaderValue (NATS headers)
├── subject.rs # Subject type, ToSubject trait, SubjectError
├── status.rs # StatusCode enum (NATS status codes)
├── error.rs # Generic Error<K> type used throughout
├── datetime.rs # DateTime type (time or chrono backend)
├── id_generator.rs # Unique ID generation (NUID or rand fallback)
├── tls.rs # TLS configuration helper
├── crypto.rs # SHA-256 for object store integrity
├── jetstream/
│ ├── mod.rs # Module entry: new(), with_domain(), with_prefix()
│ ├── context.rs # Context: JetStream API (streams, consumers, KV, OS, publish)
│ ├── stream.rs # Stream handle, Config, Info, purge/delete/message ops
│ ├── consumer/
│ │ ├── mod.rs # Consumer trait, Info, Config base
│ │ ├── pull.rs # PullConsumer: batch fetch, sequence, messages stream
│ │ └── push.rs # PushConsumer: Ordered push consumer with auto-recreate
│ ├── publish.rs # PublishAck, PublishAckFuture, PublishMessage builder
│ ├── message.rs # JetStream Message (with ack methods), AckKind
│ ├── response.rs # Response<T> (Ok/Err) for JetStream API calls
│ ├── errors.rs # ErrorCode, Error for JetStream
│ ├── account.rs # Account info
│ ├── kv/
│ │ ├── mod.rs # Store: put/get/delete/purge/watch/history/keys
│ │ └── bucket.rs # Bucket Status
│ └── object_store/
│ └── mod.rs # ObjectStore: put/get/delete/watch/list/seal, Object (AsyncRead)
└── service/
├── mod.rs # Service, ServiceBuilder, Group, EndpointBuilder, Request
└── endpoint.rs # Endpoint stream, Stats, Info
```
## Architecture: Core Connection Model
The client uses a **single-connection, actor-model** design:
```
┌──────────────────────────────────────┐
Client (clone) ──▶│ mpsc::Sender<Command> │
(many handles) │ (bounded channel) │
└────────────┬────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ ConnectionHandler (tokio::task) │
│ - Receives Command from channel │
│ - Converts to ClientOp │
│ - Manages subscriptions map │
│ - Manages multiplexer (request/reply)│
│ - Pings server on interval │
│ - Handles reconnection │
└────────────┬──────────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ Connection (framed TCP/TLS/WS) │
│ - Protocol parser (try_read_op) │
│ - Write buffer (VecDeque<Bytes>) │
│ - Vectored I/O support │
│ - Read buffer (BytesMut) │
└────────────┬──────────────────────────┘
│
▼
nats-server
```
### Key Design Decisions
1. **Cloneable Client**: `Client` is `Clone` (via `mpsc::Sender` clone), enabling shared use across tasks
2. **Single TCP connection**: All traffic (Core NATS, JetStream API, etc.) multiplexes over one connection
3. **Background task**: `ConnectionHandler` runs as a spawned Tokio task, bridging the mpsc channel to the TCP stream
4. **Automatic reconnection**: On disconnect, `Connector` retries servers from the pool with exponential backoff
5. **Subscription rehydration**: On reconnect, all active subscriptions are re-subscribed with adjusted `max` counts
6. **Multiplexer for request/reply**: A single wildcard subscription (`_INBOX.<id>.*`) multiplexes all pending request/reply correlations
## Dependencies (Key)
| Crate | Purpose |
|-------|---------|
| `tokio` | Async runtime, TCP, time, sync, io-util |
| `bytes` | Efficient byte buffer (`Bytes`, `BytesMut`) |
| `tokio-rustls` | TLS via rustls |
| `rustls-native-certs` | Load system root certificates |
| `serde` / `serde_json` | JSON serialization for JetStream API |
| `futures-util` | Stream trait, Sink trait, StreamExt |
| `tracing` | Structured logging |
| `thiserror` | Error derive macros |
| `memchr` | Fast substring search for protocol parsing |
| `portable-atomic` | Atomic types with portable-atomic fallback |
| `tokio-util` | `PollSender` for Sink implementation |
| `tokio-stream` | `ReceiverStream` adapter |

404
docs/research/references/nats.rs/nats-async/02-key-types-and-traits.md
View File

@@ -1,404 +0,0 @@
# async-nats: Key Types & Traits
## Core Types
### `Client`
The primary handle to a NATS connection. Cheaply cloneable (wraps `mpsc::Sender<Command>`).
```rust
#[derive(Clone, Debug)]
pub struct Client {
info: tokio::sync::watch::Receiver<Option<ServerInfo>>,
state: tokio::sync::watch::Receiver<State>,
sender: mpsc::Sender<Command>,
poll_sender: PollSender<Command>,
next_subscription_id: Arc<AtomicU64>,
subscription_capacity: usize,
inbox_prefix: Arc<str>,
request_timeout: Option<Duration>,
max_payload: Arc<AtomicUsize>,
connection_stats: Arc<Statistics>,
skip_subject_validation: bool,
}
```
**Key methods**:
- `publish(subject, payload)` — fire-and-forget publish
- `publish_with_headers(subject, headers, payload)` — publish with NATS headers
- `publish_with_reply(subject, reply, payload)` — publish with reply-to subject
- `subscribe(subject)` → `Subscriber` — subscribe to a subject
- `queue_subscribe(subject, queue_group)` → `Subscriber` — queue group subscription
- `request(subject, payload)` → `Message` — request/reply with default timeout
- `send_request(subject, request)` → `Message` — request with custom `Request` builder
- `flush()` — wait until all buffered writes are flushed to the server
- `drain()` — drain all subscriptions, flush, then close
- `force_reconnect()` — force a reconnection (e.g., to re-trigger auth)
- `new_inbox()` — generate a unique inbox subject (`_INBOX.<id>`)
- `server_info()` → `ServerInfo` — last known server info
- `connection_state()` → `State` — `Pending`/`Connected`/`Disconnected`
- `statistics()` → `Arc<Statistics>` — connection statistics (bytes, messages, connects)
- `max_payload()` → `usize` — server's max payload size
- `set_server_pool(addrs)` — replace the server pool for reconnection
- `server_pool()` — snapshot of current server pool
### `Subscriber`
A `Stream` yielding `Message` values from a subscription.
```rust
#[derive(Debug)]
pub struct Subscriber {
sid: u64,
receiver: mpsc::Receiver<Message>,
sender: mpsc::Sender<Command>,
}
```
Implements `futures_util::Stream<Item = Message>`. Methods:
- `unsubscribe()` — immediately unsubscribe
- `unsubscribe_after(n)` — unsubscribe after `n` total delivered messages
- `drain()` — unsubscribe after in-flight messages are delivered
**Drop behavior**: When a `Subscriber` is dropped, it spawns a task to send `Command::Unsubscribe` to the connection handler, ensuring the server is always notified.
### `Message`
An inbound NATS message:
```rust
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
### `OutboundMessage`
An outbound message for publishing (no status/description):
```rust
#[derive(Clone, Debug)]
pub struct OutboundMessage {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
}
```
### `Request`
Builder for request/reply calls:
```rust
#[derive(Default)]
pub struct Request {
pub payload: Option<Bytes>,
pub headers: Option<HeaderMap>,
pub timeout: Option<Option<Duration>>,
pub inbox: Option<String>,
}
```
Builder methods: `payload()`, `headers()`, `timeout()`, `inbox()`. The `inbox` field, when set, bypasses the multiplexer and uses a dedicated subscription instead.
### `ServerInfo`
Server metadata received during connection handshake:
```rust
#[derive(Debug, Deserialize, Default, Clone, Eq, PartialEq)]
pub struct ServerInfo {
pub server_id: String,
pub server_name: String,
pub host: String,
pub port: u16,
pub version: String,
pub auth_required: bool,
pub tls_required: bool,
pub max_payload: usize,
pub proto: i8,
pub client_id: u64,
pub go: String,
pub nonce: String,
pub connect_urls: Vec<String>,
pub client_ip: String,
pub headers: bool,
pub lame_duck_mode: bool,
pub cluster: Option<String>,
pub domain: Option<String>,
pub jetstream: bool,
}
```
### `ConnectInfo`
Client → server `CONNECT` message payload:
```rust
#[derive(Clone, Debug, Serialize)]
pub struct ConnectInfo {
pub verbose: bool,
pub pedantic: bool,
pub user_jwt: Option<String>,
pub nkey: Option<String>,
pub signature: Option<String>,
pub name: Option<String>,
pub echo: bool,
pub lang: String,
pub version: String,
pub protocol: Protocol, // Original(0) or Dynamic(1)
pub tls_required: bool,
pub user: Option<String>,
pub pass: Option<String>,
pub auth_token: Option<String>,
pub headers: bool,
pub no_responders: bool,
}
```
The client always sets: `verbose=false`, `pedantic=false`, `lang="rust"`, `protocol=Dynamic`, `headers=true`, `no_responders=true`.
### `Statistics`
Atomic connection statistics (shared via `Arc`):
```rust
#[derive(Default, Debug)]
pub struct Statistics {
pub in_bytes: AtomicU64,
pub out_bytes: AtomicU64,
pub in_messages: AtomicU64,
pub out_messages: AtomicU64,
pub connects: AtomicU64,
}
```
## Subject Types
### `Subject`
A validated NATS subject string (newtype over `String`):
```rust
// Usage:
let subject: Subject = "foo.bar.baz".into();
```
### `ToSubject` trait
Conversion trait for subjects:
```rust
pub trait ToSubject {
fn to_subject(self) -> Result<Subject, SubjectError>;
}
```
Implemented for `&str`, `String`, `Subject` directly.
### `SubjectError`
```rust
pub enum SubjectError {
InvalidFormat,
}
```
## Header Types
### `HeaderMap`
A multimap of header name → values:
```rust
pub struct HeaderMap {
inner: VecMap<HeaderName, Vec<HeaderValue>>,
}
```
Methods: `insert()`, `append()`, `get()`, `len()`, `is_empty()`, `iter()`, `to_bytes()`.
### `HeaderName`
Case-insensitive header name. Created via `FromStr`:
```rust
let name: HeaderName = "Nats-Expected-Last-Subject-Sequence".parse()?;
```
### `HeaderValue`
Header value string. Created via `FromStr` or `From<u64>`:
```rust
let val: HeaderValue = "some value".parse()?;
let val: HeaderValue = HeaderValue::from(42u64);
```
## Server Address Types
### `ServerAddr`
Wraps a `url::Url` with NATS-specific validation. Supports schemes: `nats://`, `tls://`, `ws://`, `wss://`. Default port is `4222`.
```rust
let addr: ServerAddr = "demo.nats.io".parse()?;
let addr: ServerAddr = "nats://demo.nats.io:4222".parse()?;
let addr: ServerAddr = "tls://demo.nats.io".parse()?;
```
### `ToServerAddrs` trait
Flexible server address input (single URL, `Vec`, slice, etc.):
```rust
pub trait ToServerAddrs {
type Iter: Iterator<Item = ServerAddr>;
fn to_server_addrs(&self) -> io::Result<Self::Iter>;
}
```
### `Server`
Metadata about a server in the pool:
```rust
pub struct Server {
pub addr: ServerAddr,
pub failed_attempts: usize,
pub did_connect: bool,
pub is_discovered: bool,
pub last_error: Option<String>,
}
```
## Event & State Types
### `Event`
Asynchronous notifications from the connection:
```rust
pub enum Event {
Connected,
Disconnected,
LameDuckMode,
Draining,
Closed,
SlowConsumer(u64), // subscription sid
ServerError(ServerError),
ClientError(ClientError),
}
```
Received via `ConnectOptions::event_callback()`.
### `State`
Connection state observable via `watch::Receiver`:
```rust
pub enum State {
Pending,
Connected,
Disconnected,
}
```
### `StatusCode`
NATS protocol status codes (e.g., `NO_RESPONDERS = 404`, `TIMEOUT = 408`).
## Error Types
All error types follow the pattern `Error<Kind>` from `crate::error`:
| Error Type | Kind | Used By |
|------------|------|---------|
| `ConnectError` | `ConnectErrorKind` | Connection establishment |
| `PublishError` | `PublishErrorKind` | Publish operations |
| `RequestError` | `RequestErrorKind` | Request/reply |
| `SubscribeError` | `SubscribeErrorKind` | Subscribe |
| `FlushError` | `FlushErrorKind` | Flush |
| `DrainError` | — | Drain |
### `ConnectErrorKind`
```rust
pub enum ConnectErrorKind {
ServerParse, // URL parsing failed
Dns, // DNS resolution failed
Authentication, // Auth signing failed
AuthorizationViolation, // Server rejected auth
TimedOut, // Connection handshake timeout
Tls, // TLS error
Io, // Other I/O error
MaxReconnects, // Exceeded max reconnect attempts
}
```
## Trait Definitions
The `client::traits` module defines abstract interfaces:
```rust
pub trait Publisher {
fn publish_with_reply(&self, subject, reply, payload) -> Future<Output = Result<(), PublishError>>;
fn publish_message(&self, msg: OutboundMessage) -> Future<Output = Result<(), PublishError>>;
}
pub trait Subscriber {
fn subscribe(&self, subject) -> Future<Output = Result<crate::Subscriber, SubscribeError>>;
}
pub trait Requester {
fn send_request(&self, subject, request: Request) -> Future<Output = Result<Message, RequestError>>;
}
pub trait TimeoutProvider {
fn timeout(&self) -> Option<Duration>;
}
```
`Client` implements all of these. The JetStream `Context` also implements them via delegation.
## Authentication Types
### `Auth`
Container for all authentication methods:
```rust
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
### `AuthError`
Simple string error for auth callback failures.
### `ReconnectToServer`
Returned by `reconnect_to_server_callback` to select a server and delay:
```rust
pub struct ReconnectToServer {
pub addr: ServerAddr,
pub delay: Option<Duration>,
}
```

278
docs/research/references/nats.rs/nats-async/03-protocol-and-wire-format.md
View File

@@ -1,278 +0,0 @@
# async-nats: NATS Protocol & Wire Format
## Protocol Overview
NATS uses a simple, text-based protocol over TCP. Messages are terminated with `\r\n`. The protocol is symmetric for client and server operations.
### Client → Server Operations (`ClientOp`)
```rust
pub(crate) enum ClientOp {
Publish { subject, payload, respond, headers },
Subscribe { sid, subject, queue_group },
Unsubscribe { sid, max },
Ping,
Pong,
Connect(ConnectInfo),
}
```
### Server → Client Operations (`ServerOp`)
```rust
pub(crate) enum ServerOp {
Ok,
Info(Box<ServerInfo>),
Ping,
Pong,
Error(ServerError),
Message { sid, subject, reply, payload, headers, status, description, length },
}
```
## Wire Format: Client Operations
### CONNECT
Sent immediately after receiving the first `INFO` from the server:
```
CONNECT {"verbose":false,"pedantic":false,...}\r\n
```
The JSON payload is `ConnectInfo` serialized inline on the same line.
### PUB (Publish without headers)
```
PUB <subject> [reply-to] <payload-size>\r\n
<payload>\r\n
```
Example:
```
PUB events.data INBOX.67 11\r\n
Hello World\r\n
```
### HPUB (Publish with headers)
When headers are present and non-empty:
```
HPUB <subject> [reply-to] <header-size> <total-size>\r\n
<headers>\r\n
<payload>\r\n
```
The `<total-size>` = `<header-size>` + `<payload-size>`.
Header block format:
```
NATS/1.0\r\n
Header-Name: Header-Value\r\n
Another-Header: Another-Value\r\n
\r\n
```
The version line (`NATS/1.0`) may include a status code and description:
```
NATS/1.0 404 No Messages\r\n
\r\n
```
### SUB (Subscribe)
```
SUB <subject> [queue-group] <sid>\r\n
```
The `sid` (subscription ID) is a client-assigned u64, unique per connection.
### UNSUB (Unsubscribe)
```
UNSUB <sid> [max]\r\n
```
The optional `max` tells the server to auto-unsubscribe after `max` messages are delivered.
### PING / PONG
```
PING\r\n
PONG\r\n
```
Client sends PING periodically (default every 60s). If 2+ pings are pending without PONG, the connection is considered dead.
## Wire Format: Server Operations
### INFO
First message sent by the server on connection:
```
INFO {"server_id":"NATSxxx","version":"2.10"...}\r\n
```
Also sent asynchronously when cluster topology changes.
### MSG (Message without headers)
```
MSG <subject> <sid> [reply-to] <payload-size>\r\n
<payload>\r\n
```
### HMSG (Message with headers)
```
HMSG <subject> <sid> [reply-to] <header-size> <total-size>\r\n
<headers + payload>\r\n
```
### +OK / -ERR
```
+OK\r\n
-ERR <description>\r\n
```
Sent only when `verbose=true` in `CONNECT`. The client always sets `verbose=false`, so `+OK` is not expected.
## Protocol Parser
The `Connection` struct handles all protocol parsing and serialization:
### Read Path (`try_read_op`)
1. Search for `\r\n` in `read_buf` using `memchr::memmem::find`
2. Inspect the first bytes to determine the operation type:
- `+OK` → `ServerOp::Ok`
- `PING` → `ServerOp::Ping`
- `PONG` → `ServerOp::Pong`
- `-ERR` → `ServerOp::Error(...)` (description is `trim_matches('\'')`)
- `INFO ` → `ServerOp::Info(...)` (serde_json deserialization)
- `MSG ` → Parse subject/sid/reply/size, then read payload
- `HMSG ` → Parse subject/sid/reply/header_len/total_len, then read headers + payload
3. For `MSG`/`HMSG`: if the full message body hasn't been read yet, return `None` (wait for more data)
4. For `HMSG`: parse the header block — extract version line (`NATS/1.0[ <status>[ <description>]]`), then key-value pairs (supports folded/multi-line header values)
### Write Path (`enqueue_write_op`)
Writes into a buffer strategy:
- **Small writes** (< 4096 bytes): flattened into `flattened_writes: BytesMut`
- **Large writes** (≥ 4096 bytes): appended as separate `Bytes` chunks in `write_buf: VecDeque<Bytes>`
This enables efficient vectored I/O when the underlying stream supports it.
### Write Flush Strategy
The `should_flush()` method returns:
- `Yes` — buffers empty but haven't flushed yet
- `May` — buffers not empty and haven't flushed
- `No` — already flushed or nothing to flush
The `ConnectionHandler` calls `poll_flush()` after processing commands, ensuring data is actually sent to the server.
## Vectored I/O
When `stream.is_write_vectored()` returns true, the connection uses `poll_write_vectored()` to write up to 64 `IoSlice`s at once. This is significantly more efficient for bursty publish patterns.
```rust
const WRITE_VECTORED_CHUNKS: usize = 64;
```
## WebSocket Transport
When the `websockets` feature is enabled, `WebSocketAdapter<T>` wraps `tokio_websockets::WebSocketStream<T>` to implement `AsyncRead + AsyncWrite`, making WebSocket connections transparent to the protocol layer.
```rust
#[cfg(feature = "websockets")]
pub(crate) struct WebSocketAdapter<T> {
pub(crate) inner: WebSocketStream<T>,
pub(crate) read_buf: BytesMut,
}
```
WebSocket connections use `ws://` or `wss://` scheme in the server URL. TLS for `wss://` is handled by the WebSocket library's built-in TLS support.
## Connection Lifecycle
### Initial Connection Flow
```
Client Server
│ │
│──── TCP connect ────────────────────▶ │
│◀──── INFO {server_id, nonce, ...} ─── │
│──── CONNECT {auth, ...} ──────────▶ │
│──── PING ─────────────────────────▶ │
│◀──── PONG (or -ERR) ─────────────── │
│ │
│ [connected, ConnectionHandler runs] │
```
If `tls_first` is enabled, TLS is established before reading INFO:
```
Client Server
│ │
│──── TCP connect ────────────────────▶ │
│──── TLS handshake ─────────────────▶ │
│◀──── TLS handshake ──────────────── │
│◀──── INFO {...} ──────────────────── │
│──── CONNECT + PING ────────────────▶ │
│◀──── PONG ────────────────────────── │
```
### Ping/Pong Keepalive
- Client sends PING every `ping_interval` (default 60s)
- Server responds with PONG
- If `pending_pings > MAX_PENDING_PINGS (2)`, connection is considered dead
- Any server operation resets the ping interval timer
### Reconnection Flow
On disconnect:
1. `handle_disconnect()` sends `Event::Disconnected` and sets state to `Disconnected`
2. `handle_reconnect()` calls `connector.connect()` which:
- Shuffles servers (unless `retain_servers_order`)
- Sorts by `failed_attempts` (ascending)
- Iterates through servers with exponential backoff delay
- On each server: DNS resolve → TCP connect → INFO → TLS (if needed) → CONNECT+PING → PONG
3. On success:
- Sends `Event::Connected`, sets state to `Connected`
- Removes closed subscriptions
- Re-subscribes all active subscriptions (with adjusted `max = max - delivered`)
- Re-subscribes the multiplexer (if active)
4. On failure with `MaxReconnects` reached, the handler loop exits
### Default Reconnect Delay
Exponential backoff capped at 4 seconds:
```rust
fn reconnect_delay_callback_default(attempts: usize) -> Duration {
if attempts <= 1 {
Duration::from_millis(0)
} else {
let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX);
cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), Duration::from_secs(4))
}
}
```
| Attempt | Delay |
|---------|-------|
| 1 | 0ms |
| 2 | 0ms |
| 3 | 2ms |
| 4 | 8ms |
| 5 | 32ms |
| 6 | 128ms |
| 7 | 512ms |
| 8 | 2048ms |
| 9+ | 4000ms (cap) |

221
docs/research/references/nats.rs/nats-async/04-connection-management.md
View File

@@ -1,221 +0,0 @@
# async-nats: Connection Management & Configuration
## ConnectOptions Builder
`ConnectOptions` provides a builder for all connection configuration:
```rust
let client = ConnectOptions::new()
.require_tls(true)
.ping_interval(Duration::from_secs(10))
.name("my-service")
.connect("demo.nats.io")
.await?;
```
### Authentication Methods
| Method | Description |
|--------|-------------|
| `with_token(token)` | Token-based auth |
| `with_user_and_password(user, pass)` | Username/password auth |
| `with_nkey(seed)` | NKey auth (requires `nkeys` feature) |
| `with_jwt(jwt, sign_cb)` | JWT + signing callback (requires `nkeys`) |
| `with_credentials_file(path)` | Load from `.creds` file (requires `nkeys`) |
| `with_credentials(creds_str)` | Parse credentials string (requires `nkeys`) |
| `with_auth_callback(cb)` | Dynamic auth callback receiving nonce, returning `Auth` |
The auth callback is the most flexible — it receives the server nonce and can return any combination of auth fields:
```rust
ConnectOptions::with_auth_callback(move |nonce| async move {
let mut auth = Auth::new();
auth.username = Some("user".to_string());
auth.password = Some("pass".to_string());
Ok(auth)
})
```
### TLS Configuration
| Option | Description |
|-------|-------------|
| `require_tls(bool)` | Require TLS for the connection |
| `tls_first()` | Establish TLS before INFO (requires server `handshake_first`) |
| `add_root_certificates(path)` | Load root CA certificates from PEM file |
| `add_client_certificate(cert, key)` | Load client certificate for mTLS |
| `tls_client_config(config)` | Pass a custom `rustls::ClientConfig` |
Two TLS crypto backends: `ring` (default) or `aws-lc-rs` (via feature flags). FIPS mode available via `aws-lc-rs` + `fips` features.
### Connection Behavior
| Option | Default | Description |
|--------|---------|-------------|
| `connection_timeout` | 5s | Timeout for full connection establishment |
| `request_timeout` | 10s | Default timeout for `Client::request` |
| `ping_interval` | 60s | How often client sends PING |
| `retry_on_initial_connect` | false | Return client immediately, connect in background |
| `max_reconnects` | None (unlimited) | Max consecutive reconnect attempts |
| `ignore_discovered_servers` | false | Ignore servers advertised in INFO |
| `retain_servers_order` | false | Don't shuffle server list on reconnect |
| `skip_subject_validation` | false | Skip whitespace validation on publish subjects |
| `subscription_capacity` | 65536 | mpsc channel capacity per subscription |
| `client_capacity` | 2048 | mpsc channel capacity for command sender |
| `custom_inbox_prefix` | `_INBOX` | Custom prefix for inbox subjects |
| `read_buffer_capacity` | 65535 | Initial size of the protocol read buffer |
| `local_address` | None | Local socket address to bind to |
| `no_echo` | false | Don't deliver messages published by this connection |
### Reconnection Callbacks
**`reconnect_delay_callback`**: Custom backoff strategy:
```rust
.reconnect_delay_callback(|attempts| {
Duration::from_millis(std::cmp::min((attempts * 100) as u64, 8000))
})
```
**`reconnect_to_server_callback`**: Select which server to connect to on each reconnect attempt:
```rust
.reconnect_to_server_callback(|servers, _info| async move {
servers.first().map(|s| ReconnectToServer {
addr: s.addr.clone(),
delay: Some(Duration::ZERO),
})
})
```
Receives `(Vec<Server>, ServerInfo)`, returns `Option<ReconnectToServer>`. If the returned server isn't in the pool, falls back to default selection.
**`event_callback`**: Receive async notifications:
```rust
.event_callback(|event| async move {
match event {
Event::Disconnected => println!("disconnected"),
Event::Connected => println!("connected"),
Event::SlowConsumer(sid) => eprintln!("slow consumer: {sid}"),
_ => {}
}
})
```
## Connection Handler Internals
### ProcessFut — The Core Event Loop
The `ConnectionHandler::process()` method creates a custom `Future` (`ProcessFut`) that drives the connection forward. Each `poll()` call:
1. **Check ping interval** — if timer ticked, send PING; if too many pending pings, disconnect
2. **Read server operations** — drain all available `ServerOp`s from `Connection::poll_read_op()`
3. **Process drain completions** — remove subscriptions that finished draining
4. **Handle commands** — receive up to 16 `Command`s from the mpsc channel and process them
5. **Write to socket** — flush the write buffer via `Connection::poll_write()`
6. **Flush** — call `poll_flush()` on the underlying stream when needed
7. **Check reconnect flag** — if `should_reconnect` is set, shut down and reconnect
```rust
const RECV_CHUNK_SIZE: usize = 16;
```
### Exit Reasons
The event loop exits with one of:
| Reason | Action |
|--------|--------|
| `Disconnected(Option<io::Error>)` | Attempt reconnection |
| `ReconnectRequested` | Shut down stream, attempt reconnection |
| `Closed` | Send `Event::Closed`, exit loop |
### Handle Disconnect & Reconnect
```rust
async fn handle_disconnect(&mut self) -> Result<(), ConnectError> {
self.pending_pings = 0;
self.connector.events_tx.try_send(Event::Disconnected).ok();
self.connector.state_tx.send(State::Disconnected).ok();
self.handle_reconnect().await
}
async fn handle_reconnect(&mut self) -> Result<(), ConnectError> {
let (info, connection) = self.connector.connect().await?;
self.connection = connection;
let _ = self.info_sender.send(Some(info));
// Remove closed subscriptions
self.subscriptions.retain(|_, sub| !sub.sender.is_closed());
// Re-subscribe all active subscriptions
for (sid, subscription) in &self.subscriptions {
self.connection.enqueue_write_op(&ClientOp::Subscribe {
sid: *sid,
subject: subscription.subject.to_owned(),
queue_group: subscription.queue_group.to_owned(),
});
if let Some(max) = subscription.max {
self.connection.enqueue_write_op(&ClientOp::Unsubscribe {
sid: *sid,
max: Some(max.saturating_sub(subscription.delivered)),
});
}
}
// Re-subscribe multiplexer if active
if let Some(multiplexer) = &self.multiplexer {
self.connection.enqueue_write_op(&ClientOp::Subscribe {
sid: MULTIPLEXER_SID,
subject: multiplexer.subject.to_owned(),
queue_group: None,
});
}
Ok(())
}
```
## Request/Reply Multiplexer
The client uses a **multiplexer** pattern for request/reply to avoid creating a separate subscription per request:
1. A single wildcard subscription is created on first request: `_INBOX.<random_id>.*`
2. Each request gets a unique token appended to the inbox: `_INBOX.<random_id>.<token>`
3. When a response arrives, the token is extracted from the subject and used to look up the `oneshot::Sender` in `multiplexer.senders`
4. The response is forwarded through the oneshot channel to the waiting `send_request()` future
```rust
struct Multiplexer {
subject: Subject, // _INBOX.<id>.*
prefix: Subject, // _INBOX.<id>.
senders: HashMap<String, oneshot::Sender<Message>>, // token → sender
}
```
The multiplexer subscription uses `sid = 0` (`MULTIPLEXER_SID`), which is separate from regular subscription IDs (which start at 1).
### Custom Inbox Bypass
If a `Request` has a custom `inbox` set, the multiplexer is bypassed — a dedicated subscription is created for that specific request, and the timeout/response logic is handled locally within `send_request()`.
## Server Pool Management
The `Connector` maintains a `Vec<Server>` pool. Servers can come from:
1. **Explicit URLs** — provided by the user at connect time
2. **Discovered servers** — advertised in `INFO.connect_urls` (unless `ignore_discovered_servers` is set)
On reconnection:
- Servers are shuffled (unless `retain_servers_order`)
- Sorted by `failed_attempts` (ascending) — prefer servers that haven't failed recently
- Each server is tried with exponential backoff delay
- On success: `failed_attempts` reset to 0, `did_connect` set to true
- On failure: `failed_attempts` incremented, `last_error` updated
### Dynamic Server Pool Updates
`Client::set_server_pool()` replaces the pool at runtime:
- Per-server state is preserved for servers that appear in both old and new pools
- The global reconnection attempt counter is reset
- Cannot mix WebSocket and non-WebSocket URLs
- Pool cannot be empty

373
docs/research/references/nats.rs/nats-async/05-jetstream.md
View File

@@ -1,373 +0,0 @@
# async-nats: JetStream
## Overview
JetStream is NATS' built-in persistence layer, providing stream-based messaging with at-least-once and exactly-once delivery semantics. The `async-nats` JetStream API is accessed through a `Context` object.
### Creating a Context
```rust
// Default context (prefix: $JS.API)
let jetstream = async_nats::jetstream::new(client);
// With domain (prefix: $JS.<domain>.API)
let jetstream = async_nats::jetstream::with_domain(client, "hub");
// With custom prefix
let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API");
// Builder with fine-grained control
let context = ContextBuilder::new()
.timeout(Duration::from_secs(5))
.api_prefix("MY.JS.API")
.max_ack_inflight(1000)
.backpressure_on_inflight(true)
.ack_timeout(Duration::from_secs(30))
.build(client);
```
## Context
```rust
#[derive(Debug, Clone)]
pub struct Context {
pub(crate) client: Client,
pub(crate) prefix: String,
pub(crate) timeout: Duration,
pub(crate) max_ack_semaphore: Arc<tokio::sync::Semaphore>,
pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver<Message>, OwnedSemaphorePermit)>,
pub(crate) backpressure_on_inflight: bool,
pub(crate) semaphore_capacity: usize,
}
```
### Publish Backpressure
The context uses a semaphore to limit the number of pending publish acknowledgments:
- `max_ack_inflight(n)` — sets semaphore capacity (default 5000)
- `backpressure_on_inflight(true)` — `publish()` waits for a permit when limit is reached
- `backpressure_on_inflight(false)` — `publish()` returns `MaxAckPending` error immediately when limit is reached
A background **acker task** monitors pending acks with a timeout (`ack_timeout`, default 30s), releasing permits when acks arrive or time out.
### JetStream API Request Pattern
All JetStream API calls follow the same pattern:
1. Build a subject from the prefix: `format!("{}.STREAM.CREATE.<name>", self.prefix)`
2. Serialize the request payload as JSON
3. Send a request via `client.send_request()` with the API subject
4. Deserialize the response as `Response<T>` (which is `Ok(T)` or `Err(ErrorCode)`)
## Streams
### Stream Handle
```rust
pub struct Stream<I = Info> {
context: Context,
info: I,
name: String,
}
```
`Stream<Info>` carries server-side info. `Stream<()>` is a lightweight handle that skips the INFO fetch. `Stream` (no generic) defaults to `Stream<Info>`.
### Stream Config
```rust
pub struct Config {
pub name: String,
pub description: Option<String>,
pub subjects: Vec<String>,
pub retention: RetentionPolicy,
pub max_consumers: i64,
pub max_messages: i64,
pub max_messages_per_subject: i64,
pub max_bytes: i64,
pub max_age: Duration,
pub max_messages_per_stream: i64,
pub max_msg_size: i32,
pub discard: DiscardPolicy,
pub discard_new_per_subject: bool,
pub storage: StorageType,
pub num_replicas: usize,
pub no_ack: bool,
pub duplicate_window: Duration,
pub placement: Option<Placement>,
pub mirror: Option<Source>,
pub sources: Option<Vec<Source>>,
pub sealed: bool,
pub allow_direct: bool,
pub allow_rollup_hdrs: bool,
// server_2_10 features:
pub compression: Option<Compression>,
pub first_sequence: Option<u64>,
pub subject_transform: Option<SubjectTransform>,
pub republish: Option<Republish>,
pub metadata: Option<HashMap<String, String>>,
}
```
### Stream Operations
| Method | Description |
|--------|-------------|
| `create_stream(config)` | Create a new stream |
| `get_stream(name)` | Get stream handle (with INFO) |
| `get_stream_no_info(name)` | Get lightweight handle (no server round-trip) |
| `get_or_create_stream(config)` | Get existing or create new |
| `delete_stream(name)` | Delete a stream |
| `update_stream(config)` | Update stream configuration |
| `create_or_update_stream(config)` | Update or create if not found |
| `stream_names()` | `Stream` of stream names (paginated) |
| `streams()` | `Stream` of stream info (paginated) |
| `stream_by_subject(subject)` | Find stream name containing subject |
### Stream Handle Methods
```rust
let stream: Stream = jetstream.get_stream("events").await?;
// Info
let info: Info = stream.info().await?; // Fresh info from server
let info: &Info = stream.cached_info(); // Cached info from last fetch
// Message operations
stream.get_raw_message(seq).await?; // Get raw message by sequence
stream.get_last_raw_message_by_subject(subj).await?; // Get last message for subject
stream.direct_get(seq).await?; // Direct get (if allow_direct)
stream.direct_get_last_for_subject(subj).await?; // Direct last by subject
stream.delete_message(seq).await?; // Delete a specific message
stream.purge().await?; // Purge all messages
stream.purge().filter(subj).await?; // Purge messages for subject
// Consumers
stream.create_consumer(config).await?; // Create consumer bound to stream
stream.get_consumer(name).await?; // Get existing consumer
stream.delete_consumer(name).await?; // Delete consumer
```
## Consumers
### Consumer Types
Two consumer types, each with distinct delivery models:
1. **Pull Consumer** (`pull::Config` / `PullConsumer`) — Client explicitly requests batches of messages
2. **Push Consumer** (`push::Config` / `PushConsumer`) — Server pushes messages to a deliver subject
### Pull Consumer
```rust
let consumer: PullConsumer = stream
.get_or_create_consumer("my-consumer", pull::Config {
durable_name: Some("my-consumer".to_string()),
..Default::default()
})
.await?;
```
**Key methods**:
- `consumer.batch(n).await?` — Fetch up to `n` messages (one-shot batch)
- `consumer.messages().await?` — Continuous `Stream` of messages
- `consumer.sequence(n).await?` — Continuous `Stream` of batches of `n` messages
- `consumer.fetch().max(n).expires(dur).await?` — Configurable fetch
Each message from a pull consumer is a `jetstream::Message` which has `ack()` methods.
### Push Consumer
Two push consumer variants:
1. **Standard** (`push::Config`) — messages delivered to a specific subject
2. **Ordered** (`push::OrderedConfig`) — auto-recreated on failure, with flow control
```rust
// Standard push
let consumer = stream.create_consumer(push::Config {
deliver_subject: "deliver.subject".to_string(),
durable_name: Some("push-consumer".to_string()),
..Default::default()
}).await?;
// Ordered push (no durable name, auto-recreates on failure)
let consumer = stream.create_consumer(push::OrderedConfig {
deliver_subject: client.new_inbox(),
filter_subject: "events.>".to_string(),
..Default::default()
}).await?;
```
### Consumer Config (Shared Fields)
```rust
pub struct Config {
// Pull fields
pub durable_name: Option<String>,
pub name: Option<String>,
// Push fields
pub deliver_subject: Option<String>,
pub deliver_group: Option<String>,
pub deliver_policy: DeliverPolicy,
pub opt_start_time: Option<DateTime>,
pub opt_start_sequence: Option<u64>,
pub ack_policy: AckPolicy,
pub ack_wait: Duration,
pub max_deliver: i64,
pub backoff: Vec<Duration>,
pub filter_subject: String,
pub filter_subjects: Vec<String>, // server_2_10+
pub replay_policy: ReplayPolicy,
pub rate_limit_bps: Option<u64>,
pub max_waiting: i64, // pull: max outstanding pull requests
pub max_ack_pending: i64,
pub flow_control: bool,
pub idle_heartbeat: Duration,
pub headers_only: bool,
pub num_replicas: usize,
pub mem_storage: bool,
pub description: Option<String>,
pub metadata: Option<HashMap<String, String>>,
pub inactive_threshold: Option<Duration>, // for ephemeral consumers
}
```
### Deliver Policy
```rust
pub enum DeliverPolicy {
All, // Deliver all messages
Last, // Deliver last message only
New, // Deliver only new messages
ByStartSequence { start_sequence: u64 },
ByStartTime { start_time: DateTime },
LastPerSubject, // Deliver last message per subject
}
```
### Ack Policy
```rust
pub enum AckPolicy {
None, // No acknowledgment needed
All, // Ack all messages up to this one
Explicit, // Ack each message individually
}
```
## JetStream Messages
### `jetstream::Message`
Wraps a core `Message` with JetStream-specific metadata:
```rust
pub struct Message {
pub message: crate::Message, // The underlying NATS message
pub ack_subject: Subject, // Subject for sending acks
pub stream: String, // Stream name
pub consumer: String, // Consumer name
pub stream_sequence: u64, // Sequence in stream
pub consumer_sequence: u64, // Sequence for this consumer
pub delivered: u64, // Delivery count
pub pending: u64, // Pending message count
pub published: DateTime, // Original publish time
}
```
### Ack Methods
```rust
// In-memory ack (non-persistent, fast)
message.ack().await?;
// Ack with specific type
message.ack_with(AckKind::Nak).await?;
message.ack_with(AckKind::Progress).await?;
message.ack_with(AckKind::Term).await?;
message.ack_with(AckKind::NakWithDelay(duration)).await?;
message.ack_with(AckKind::TermWithReason("reason")).await?;
```
### `AckKind`
```rust
pub enum AckKind {
Ack, // +ACK — message processed
Nak, // -NAK — re-deliver
Progress, // PRI — still working
Term, // +TERM — don't redeliver
NakWithDelay(Duration), // -NAK with re-delivery delay
TermWithReason(String), // +TERM with reason
}
```
## JetStream Publish
### `Context::publish()`
JetStream publish returns a `PublishAckFuture` — a future that resolves to a `PublishAck`:
```rust
let ack_future = jetstream.publish("events", "data".into()).await?;
let ack: PublishAck = ack_future.await?; // Wait for server acknowledgment
```
### `PublishAck`
```rust
pub struct PublishAck {
pub stream: String,
pub sequence: u64,
pub domain: String,
pub duplicate: bool,
}
```
### `PublishMessage` Builder
```rust
let ack = jetstream.send_publish(
"events",
PublishMessage::build()
.payload("data".into())
.message_id("uuid-123") // Deduplication ID
.expected_stream("events") // Fail if wrong stream
.expected_last_msg_id("prev-id")
.expected_last_sequence(42)
.headers(headers),
).await?;
```
## Pagination
Stream and consumer listing uses pagination internally:
```rust
pub struct StreamNames {
context: Context,
offset: usize,
page_request: Option<Request>,
streams: Vec<String>,
subject: Option<String>,
done: bool,
}
```
Implements `futures_util::Stream<Item = Result<String, Error>>`, lazily fetching pages as needed.
## Error Handling
JetStream errors follow the `Response<T>` pattern:
```rust
pub enum Response<T> {
Ok(T),
Err { error: ErrorCode },
}
```
`ErrorCode` carries the server's error code and description. Most JetStream-specific errors map to typed error enums (e.g., `CreateStreamError`, `ConsumerError`, etc.).

237
docs/research/references/nats.rs/nats-async/06-key-value-store.md
View File

@@ -1,237 +0,0 @@
# async-nats: Key-Value Store
## Overview
The Key-Value (KV) store is an abstraction built on top of JetStream streams. Each KV bucket is backed by a JetStream stream with the naming convention `KV_<bucket_name>`. Keys are mapped to subjects under the `$KV.<bucket>.<key>` prefix.
The KV feature requires `kv` (which implies `jetstream`).
## Store Handle
```rust
#[derive(Debug, Clone)]
pub struct Store {
pub name: String,
pub stream_name: String,
pub prefix: String, // $KV.<bucket>.
pub put_prefix: Option<String>, // For mirrored buckets
pub use_jetstream_prefix: bool, // Whether to prepend JS API prefix
pub stream: Stream,
}
```
## Bucket Config
```rust
#[derive(Debug, Clone, Default)]
pub struct Config {
pub bucket: String,
pub description: String,
pub max_value_size: i32,
pub history: i64, // Max historical entries per key (1-64)
pub max_age: Duration, // Max age of any entry
pub max_bytes: i64, // Total bucket size limit
pub storage: StorageType, // File or Memory
pub num_replicas: usize,
pub republish: Option<Republish>,
pub mirror: Option<Source>, // Mirror another bucket
pub sources: Option<Vec<Source>>,
pub mirror_direct: bool,
pub compression: bool, // server_2_10+
pub placement: Option<Placement>,
pub limit_markers: Option<Duration>, // server_2_11+
}
```
## Creating/Accessing Buckets
```rust
// Create a new bucket
let kv = jetstream.create_key_value(kv::Config {
bucket: "my-bucket".to_string(),
history: 10,
max_age: Duration::from_secs(3600),
..Default::default()
}).await?;
// Get an existing bucket
let kv = jetstream.get_key_value("my-bucket").await?;
// Create or update
let kv = jetstream.create_or_update_key_value(kv::Config { ... }).await?;
// Delete a bucket
jetstream.delete_key_value("my-bucket").await?;
```
## KV Operations
### Put
```rust
let revision: u64 = kv.put("key", "value".into()).await?;
```
Publishes to `$KV.<bucket>.<key>` (or with JS prefix). The JetStream stream stores it, and the returned sequence number serves as the revision.
### Get
```rust
let value: Option<Bytes> = kv.get("key").await?;
```
Returns `None` if the key doesn't exist or was deleted/purged. Uses either direct get (if `allow_direct`) or the standard message API.
### Entry
```rust
let entry: Option<Entry> = kv.entry("key").await?;
let entry: Option<Entry> = kv.entry_for_revision("key", 2).await?;
```
Returns full entry metadata:
```rust
pub struct Entry {
pub bucket: String,
pub key: String,
pub value: Bytes,
pub revision: u64,
pub created: DateTime,
pub delta: u64,
pub operation: Operation,
pub seen_current: bool,
}
```
### Create (Put if not exists)
```rust
let revision: u64 = kv.create("key", "value".into()).await?;
```
Uses `update` with `expected_last_subject_sequence = 0` (create-only). If the key exists and is deleted/purged, it's re-created.
### Update (Conditional Put)
```rust
let revision: u64 = kv.update("key", "value".into(), last_revision).await?;
```
Uses the `Nats-Expected-Last-Subject-Sequence` header for optimistic concurrency control. Only succeeds if the key's current revision matches.
### Delete
```rust
kv.delete("key").await?;
kv.delete_expect_revision("key", Some(revision)).await?;
```
Non-destructive — publishes a `DEL` marker message. The key appears deleted to `get()`, but history is preserved (up to `history` limit).
### Purge
```rust
kv.purge("key").await?;
kv.purge_with_ttl("key", Duration::from_secs(10)).await?;
kv.purge_expect_revision("key", Some(revision)).await?;
```
Destructive — publishes a `PURGE` marker with rollup header, removing all previous revisions of the key. Leaves a single purge entry.
### Watch
```rust
// Watch for new changes
let mut watch = kv.watch("key").await?;
// Watch with initial value
let mut watch = kv.watch_with_history("key").await?;
// Watch from specific revision
let mut watch = kv.watch_from_revision("key", 5).await?;
// Watch all keys
let mut watch = kv.watch_all().await?;
// Watch multiple keys (server_2_10+)
let mut watch = kv.watch_many(["foo", "bar"]).await?;
```
`Watch` implements `futures_util::Stream<Item = Result<Entry, WatcherError>>`.
Under the hood, each watch creates an **ordered push consumer** on the KV stream with:
- `filter_subject` matching `$KV.<bucket>.<key>`
- `replay_policy: Instant`
- Appropriate `deliver_policy`
### History
```rust
let mut history = kv.history("key").await?;
```
Returns a `Stream` of all past `Entry` values for a key (including deletes/purges).
### Keys
```rust
let mut keys = kv.keys().await?;
```
Returns a `Stream<String>` of all current keys. Uses a headers-only consumer with `LastPerSubject` deliver policy to efficiently scan the bucket.
## Entry Operations
```rust
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
pub enum Operation {
Put, // Value was put
Delete, // Value was deleted (DEL marker)
Purge, // Value was purged (PURGE marker with rollup)
}
```
The operation type is determined from the `KV-Operation` header (`PUT`, `DEL`, `PURGE`) or the `Nats-Marker-Reason` header (fallback for server-generated markers like `MaxAge`, `Purge`, `Remove`).
## Key and Bucket Name Validation
```rust
// Bucket: alphanumeric, dash, underscore only
VALID_BUCKET_RE: \A[a-zA-Z0-9_-]+\z
// Key: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots
VALID_KEY_RE: \A[-/_=\.a-zA-Z0-9]+\z
```
## Bucket Status
```rust
let status: Status = kv.status().await?;
```
Wraps stream info to provide bucket-level statistics (bucket name, message count, byte count, etc.).
## Mirrored Buckets
When a bucket is configured as a mirror of another (potentially in a different account/domain):
- `prefix` is set to `$KV.<mirror_bucket>.`
- `put_prefix` may be set to the source bucket's API prefix for cross-domain writes
- `use_jetstream_prefix` is adjusted based on whether the mirror is in the same domain
## KV → Stream Config Mapping
When creating a KV bucket, the `Config` is converted to a JetStream `stream::Config`:
| KV Config | Stream Config |
|-----------|---------------|
| `bucket` | `name = "KV_<bucket>"` |
| `subjects` | `["$KV.<bucket>.>"]` |
| `max_messages_per_subject` | `history` (max 64) |
| `max_age` | `max_age` |
| `max_bytes` | `max_bytes` |
| `storage` | `storage` |
| `num_replicas` | `num_replicas` |
| `republish` | `republish` |
| `mirror` | `mirror` |
| `discard` | `DiscardPolicy::New` |
| `allow_direct` | `true` |
| `allow_rollup_hdrs` | `true |
| `max_msg_size` | `max_value_size` |

245
docs/research/references/nats.rs/nats-async/07-object-store.md
View File

@@ -1,245 +0,0 @@
# async-nats: Object Store
## Overview
The Object Store provides large-object storage built on JetStream. Objects are chunked and stored as messages in a JetStream stream, with metadata stored separately. The stream is named `OBJ_<bucket_name>`.
The object-store feature requires `object-store` (which implies `jetstream` + `crypto`).
## ObjectStore Handle
```rust
#[derive(Clone)]
pub struct ObjectStore {
pub(crate) name: String,
pub(crate) stream: Stream,
}
```
## Object Store Config
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct Config {
pub bucket: String,
pub description: Option<String>,
pub max_age: Duration,
pub max_bytes: i64,
pub storage: StorageType,
pub num_replicas: usize,
pub compression: bool,
pub placement: Option<Placement>,
}
```
## Creating/Accessing Object Stores
```rust
// Create
let bucket = jetstream.create_object_store(object_store::Config {
bucket: "my-bucket".to_string(),
..Default::default()
}).await?;
// Get existing
let bucket = jetstream.get_object_store("my-bucket").await?;
// Delete
jetstream.delete_object_store("my-bucket").await?;
```
## Object Store Operations
### Put
```rust
let info: ObjectInfo = bucket.put("file.txt", &mut async_read).await?;
```
The put operation:
1. Reads data from any `AsyncRead + Unpin` source in chunks (default 128KB)
2. Each chunk is published to `$O.<bucket>.C.<nuid>` (chunk subject)
3. SHA-256 digest is computed incrementally
4. After all chunks, metadata is published to `$O.<bucket>.M.<encoded_name>` with a rollup header
5. If the object previously existed, old chunks are purged
### Get
```rust
let mut object: Object = bucket.get("file.txt").await?;
```
Returns an `Object` that implements `tokio::io::AsyncRead`:
```rust
let mut bytes = Vec::new();
object.read_to_end(&mut bytes).await?;
```
On read, the Object:
1. Creates an ordered push consumer on `$O.<bucket>.C.<nuid>`
2. Streams chunk messages, feeding bytes to the reader
3. Verifies SHA-256 digest after the last chunk
4. If digest doesn't match, returns `io::ErrorKind::InvalidData`
### Delete
```rust
bucket.delete("file.txt").await?;
```
Marks the object as deleted in metadata (sets `deleted = true`, `chunks = 0`, `size = 0`) with a rollup, then purges all chunk messages.
### Info
```rust
let info: ObjectInfo = bucket.info("file.txt").await?;
```
Fetches the last metadata message for the object (from `$O.<bucket>.M.<encoded_name>`).
### Watch
```rust
let mut watcher = bucket.watch().await?;
let mut watcher = bucket.watch_with_history().await?;
```
Returns a `Stream<Item = Result<ObjectInfo, WatcherError>>`. Uses an ordered push consumer on `$O.<bucket>.M.>`.
### List
```rust
let mut list = bucket.list().await?;
```
Returns a `Stream<Item = Result<ObjectInfo, ListerError>>`. Lists all non-deleted objects. Uses `DeliverPolicy::All` to replay all metadata.
### Seal
```rust
bucket.seal().await?;
```
Sets the underlying stream's `sealed = true`, preventing any further modifications.
### Links
```rust
// Link to another object (same or different bucket)
let info = bucket.add_link("link_name", &object).await?;
// Link to another bucket
let info = bucket.add_bucket_link("link_name", "other_bucket").await?;
```
Links are followed automatically when `get()` is called (one level deep). Cannot link to a deleted object or create a link to a link.
### Update Metadata
```rust
bucket.update_metadata("object", object_store::UpdateMetadata {
name: "new_name".to_string(),
description: Some("updated description".to_string()),
..Default::default()
}).await?;
```
If the name changes, old metadata is purged and new metadata is published.
## Object Types
### ObjectInfo
```rust
#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectInfo {
pub name: String,
pub description: Option<String>,
pub metadata: HashMap<String, String>,
pub headers: Option<HeaderMap>,
pub options: Option<ObjectOptions>,
pub bucket: String,
pub nuid: String,
pub size: usize,
pub chunks: usize,
pub modified: Option<DateTime>,
pub digest: Option<String>, // Format: "SHA-256=<base64url-digest>"
pub deleted: bool,
}
```
### ObjectMetadata
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectMetadata {
pub name: String,
pub description: Option<String>,
pub chunk_size: Option<usize>,
pub metadata: HashMap<String, String>,
pub headers: Option<HeaderMap>,
}
```
### ObjectLink
```rust
#[derive(Debug, Default, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectLink {
pub name: Option<String>, // None = bucket link, Some = object link
pub bucket: String,
}
```
### Object
```rust
pub struct Object {
pub info: ObjectInfo,
remaining_bytes: VecDeque<u8>,
has_pending_messages: bool,
digest: Option<Sha256>,
subscription: Option<crate::jetstream::consumer::push::Ordered>,
subscription_future: Option<BoxFuture<'static, Result<Ordered, StreamError>>>,
stream: Stream,
}
```
Implements `tokio::io::AsyncRead`. Lazy-creates the consumer on first read.
## Subject Naming Convention
| Purpose | Subject Pattern |
|---------|----------------|
| Chunks | `$O.<bucket>.C.<nuid>` |
| Metadata | `$O.<bucket>.M.<base64url-encoded-name>` |
Object names are base64url-encoded in metadata subjects to allow arbitrary characters (the raw name might contain characters invalid in NATS subjects).
## Validation
```rust
// Bucket: alphanumeric, dash, underscore only
BUCKET_NAME_RE: \A[a-zA-Z0-9_-]+\z
// Object name: alphanumeric, dash, slash, underscore, equals, dot; no leading/trailing dots
OBJECT_NAME_RE: \A[-/_=\.a-zA-Z0-9]+\z
```
## Data Integrity
The object store uses SHA-256 hashing (from the `crypto` module) to verify data integrity:
1. On `put()`: SHA-256 is computed incrementally as chunks are read. The digest is stored in `ObjectInfo.digest` as `"SHA-256=<base64url>"`.
2. On `get()` (via `AsyncRead`): SHA-256 is verified after the last chunk is read. If the computed digest doesn't match the stored digest, `io::ErrorKind::InvalidData` is returned.
```rust
// crypto module
pub(crate) struct Sha256 { ... }
impl Sha256 {
pub fn new() -> Self;
pub fn update(&mut self, data: &[u8]);
pub fn finish(self) -> [u8; 32];
}
```

272
docs/research/references/nats.rs/nats-async/08-service-api.md
View File

@@ -1,272 +0,0 @@
# async-nats: Service API
## Overview
The Service API provides a microservice request/reply pattern with built-in service discovery, health checking, and statistics. It follows the [NATS Micro v1 specification](https://github.com/nats-io/nats-architecture-design/blob/main/adr/ADR-33.md).
The `service` feature is required.
## Service
```rust
#[derive(Debug)]
pub struct Service {
endpoints_state: Arc<Mutex<Endpoints>>,
info: Info,
client: Client,
handle: JoinHandle<Result<(), Error>>,
shutdown_tx: Sender<()>,
subjects: Arc<Mutex<Vec<String>>>,
queue_group: String,
}
```
## Creating a Service
Via the `ServiceExt` trait on `Client`:
```rust
use async_nats::service::ServiceExt;
// Builder pattern
let mut service = client
.service_builder()
.description("product service")
.stats_handler(|endpoint, stats| serde_json::json!({ "endpoint": endpoint }))
.metadata(HashMap::from([("version".into(), "v2".into())]))
.queue_group("products-group")
.start("products", "1.0.0")
.await?;
// Direct config
let mut service = client
.add_service(service::Config {
name: "products".to_string(),
version: "1.0.0".to_string(),
description: Some("product service".to_string()),
stats_handler: None,
metadata: None,
queue_group: None,
})
.await?;
```
Service name must match `^[A-Za-z0-9\-_]+$`. Version must be valid SemVer.
## Service Verbs
Every service automatically subscribes to three verb subjects for discovery and monitoring:
| Verb | Subject Pattern | Purpose |
|------|----------------|---------|
| PING | `$SRV.PING`, `$SRV.PING.<name>`, `$SRV.PING.<name>.<id>` | Lightweight health check |
| INFO | `$SRV.INFO.<name>`, `$SRV.INFO.<name>.<id>` | Service metadata |
| STATS | `$SRV.STATS.<name>`, `$SRV.STATS.<name>.<id>` | Service + endpoint statistics |
A background task handles these verb requests and responds with JSON payloads.
## Service Config
```rust
#[derive(Serialize, Deserialize, Debug)]
pub struct Config {
pub name: String,
pub description: Option<String>,
pub version: String,
pub stats_handler: Option<StatsHandler>,
pub metadata: Option<HashMap<String, String>>,
pub queue_group: Option<String>,
}
```
## Adding Endpoints
```rust
// Simple endpoint
let mut endpoint = service.endpoint("get-products").await?;
// Endpoint with custom name and metadata
let endpoint = service
.endpoint_builder()
.name("api")
.metadata(HashMap::from([("auth".into(), "required".into())]))
.queue_group("custom-group")
.add("products")
.await?;
// Grouped endpoints
let v1 = service.group("v1");
let products = v1.endpoint("products").await?;
let orders = v1.endpoint("orders").await?;
// Nested groups
let v1_api = service.group("api").group("v1");
```
## Endpoint
```rust
pub struct Endpoint {
requests: Subscriber,
stats: Arc<Mutex<Endpoints>>,
client: Client,
endpoint: String,
shutdown: Option<ShutdownRx>,
shutdown_future: Option<ShutdownReceiverFuture>,
}
```
Implements `futures_util::Stream<Item = Request>`.
```rust
while let Some(request) = endpoint.next().await {
request.respond(Ok("response data".into())).await?;
}
```
## Service Request
```rust
#[derive(Debug)]
pub struct Request {
issued: Instant,
client: Client,
pub message: Message,
endpoint: String,
stats: Arc<Mutex<Endpoints>>,
}
```
### Responding
```rust
// Success
request.respond(Ok("result".into())).await?;
// Success with headers
request.respond_with_headers(Ok("result".into()), headers).await?;
// Error
request.respond(Err(service::error::Error {
code: 500,
status: "internal error".to_string(),
})).await?;
```
Error responses always include `Nats-Service-Error` and `Nats-Service-Error-Code` headers. If user-supplied headers contain these headers, they are overridden by the error values.
### Stats Tracking
Each response updates endpoint statistics:
- `requests` — total requests
- `processing_time` — cumulative processing time
- `average_processing_time` — average per request
- `errors` — error count
- `last_error` — last error details
## Service Info Types
### PingResponse
```rust
pub struct PingResponse {
pub kind: String, // "io.nats.micro.v1.ping_response"
pub name: String,
pub id: String,
pub version: String,
pub metadata: HashMap<String, String>,
}
```
### Info
```rust
pub struct Info {
pub kind: String, // "io.nats.micro.v1.info_response"
pub name: String,
pub id: String,
pub description: String,
pub version: String,
pub metadata: HashMap<String, String>,
pub endpoints: Vec<endpoint::Info>,
}
```
### Stats
```rust
pub struct Stats {
pub kind: String, // "io.nats.micro.v1.stats_response"
pub name: String,
pub id: String,
pub version: String,
pub started: DateTime,
pub endpoints: Vec<endpoint::Stats>,
}
```
### Endpoint Stats
```rust
pub struct endpoint::Stats {
pub name: String,
pub subject: String,
pub queue_group: String,
pub data: Option<serde_json::Value>, // Custom data from stats_handler
pub errors: u64,
pub processing_time: Duration,
pub average_processing_time: Duration,
pub requests: u64,
pub last_error: Option<error::Error>,
}
```
## Service Groups
Groups provide subject prefixing for endpoint organization:
```rust
let service = client.service_builder().start("api", "1.0.0").await?;
// Endpoints subscribe to "products" and "orders"
let products = service.endpoint("products").await?;
let orders = service.endpoint("orders").await?;
// Grouped: subscribe to "v1.products" and "v1.orders"
let v1 = service.group("v1");
let products = v1.endpoint("products").await?;
let orders = v1.endpoint("orders").await?;
// Nested: subscribe to "api.v1.products"
let api_v1 = service.group("api").group("v1");
let products = api_v1.endpoint("products").await?;
```
Each group can have its own queue group:
```rust
let v1 = service.group_with_queue_group("v1", "v1-workers");
```
## Stopping a Service
```rust
service.stop().await?;
```
Sends a shutdown signal and aborts the verb-handling task. Other service instances with the same name continue running.
## Resetting Stats
```rust
service.reset().await?;
```
Resets all endpoint statistics (errors, processing time, requests, average processing time) to zero.
## Querying Service State
```rust
let stats: HashMap<String, endpoint::Stats> = service.stats().await?;
let info: Info = service.info().await?;
```

312
docs/research/references/nats.rs/nats-async/09-quick-reference.md
View File

@@ -1,312 +0,0 @@
# async-nats: Quick Reference
## Connection
```rust
// Basic connect
let client = async_nats::connect("demo.nats.io").await?;
// With options
let client = async_nats::ConnectOptions::new()
.require_tls(true)
.name("my-service")
.ping_interval(Duration::from_secs(10))
.request_timeout(Some(Duration::from_secs(5)))
.connect("demo.nats.io")
.await?;
// Multiple servers
let client = async_nats::connect(vec![
"nats://server1:4222".parse()?,
"nats://server2:4222".parse()?,
]).await?;
// Background connect
let client = async_nats::ConnectOptions::new()
.retry_on_initial_connect()
.connect("demo.nats.io")
.await?;
```
## Core NATS: Publish
```rust
// Simple publish
client.publish("subject", "payload".into()).await?;
// With reply-to
client.publish_with_reply("subject", "reply-to", "payload".into()).await?;
// With headers
let mut headers = HeaderMap::new();
headers.insert("X-Custom", "value");
client.publish_with_headers("subject", headers, "payload".into()).await?;
// Full control
client.publish_with_reply_and_headers("subject", "reply-to", headers, "payload".into()).await?;
// Flush (ensure all published messages are sent)
client.flush().await?;
```
## Core NATS: Subscribe
```rust
use futures_util::StreamExt;
// Basic subscribe
let mut subscriber = client.subscribe("subject").await?;
// Queue group
let mut subscriber = client.queue_subscribe("subject", "group".into()).await?;
// Receive messages (Subscriber implements Stream)
while let Some(message) = subscriber.next().await {
println!("subject: {}, payload: {:?}", message.subject, message.payload);
}
// Unsubscribe
subscriber.unsubscribe().await?;
// Unsubscribe after N messages
subscriber.unsubscribe_after(10).await?;
// Drain (wait for in-flight, then unsubscribe)
subscriber.drain().await?;
```
## Core NATS: Request/Reply
```rust
// Simple request (uses default timeout)
let response = client.request("subject", "data".into()).await?;
// With custom timeout and headers
let request = async_nats::Request::new()
.payload("data".into())
.timeout(Some(Duration::from_secs(5)))
.headers(headers);
let response = client.send_request("subject", request).await?;
// Custom inbox (bypasses multiplexer)
let request = async_nats::Request::new()
.payload("data".into())
.inbox("custom-inbox".into());
let response = client.send_request("subject", request).await?;
```
## Message Structure
```rust
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
## JetStream
```rust
let jetstream = async_nats::jetstream::new(client);
// Publish (returns ack future)
let ack = jetstream.publish("events", "data".into()).await?;
let publish_ack = ack.await?;
// Stream management
let stream = jetstream.create_stream(stream::Config {
name: "events".to_string(),
subjects: vec!["events.>".to_string()],
max_messages: 10_000,
..Default::default()
}).await?;
let stream = jetstream.get_stream("events").await?;
let stream = jetstream.get_or_create_stream(config).await?;
jetstream.delete_stream("events").await?;
jetstream.update_stream(config).await?;
// Consumer management
let consumer: PullConsumer = stream.create_consumer(pull::Config {
durable_name: Some("my-consumer".to_string()),
..Default::default()
}).await?;
// Pull consumer: fetch messages
let mut messages = consumer.messages().await?;
while let Some(message) = messages.next().await {
let message = message?;
message.ack().await?;
}
// Push consumer (ordered)
let consumer = stream.create_consumer(push::OrderedConfig {
deliver_subject: client.new_inbox(),
filter_subject: "events.>".to_string(),
..Default::default()
}).await?;
let mut messages = consumer.messages().await?;
```
## Key-Value Store
```rust
let kv = jetstream.create_key_value(kv::Config {
bucket: "my-bucket".to_string(),
history: 10,
..Default::default()
}).await?;
// CRUD
let revision = kv.put("key", "value".into()).await?;
let revision = kv.create("key", "value".into()).await?;
let value: Option<Bytes> = kv.get("key").await?;
let entry: Option<Entry> = kv.entry("key").await?;
let revision = kv.update("key", "new-value".into(), revision).await?;
kv.delete("key").await?;
kv.purge("key").await?;
// Watch
let mut watch = kv.watch("key").await?;
let mut watch_all = kv.watch_all().await?;
// History & Keys
let mut history = kv.history("key").await?;
let mut keys = kv.keys().await?;
```
## Object Store
```rust
let bucket = jetstream.create_object_store(object_store::Config {
bucket: "files".to_string(),
..Default::default()
}).await?;
// Put (from any AsyncRead)
let info = bucket.put("file.txt", &mut file).await?;
// Get (returns AsyncRead)
let mut object = bucket.get("file.txt").await?;
let mut bytes = Vec::new();
object.read_to_end(&mut bytes).await?;
// Info, delete, list, watch
let info = bucket.info("file.txt").await?;
bucket.delete("file.txt").await?;
let mut list = bucket.list().await?;
let mut watch = bucket.watch().await?;
```
## Service API
```rust
use async_nats::service::ServiceExt;
use futures_util::StreamExt;
let mut service = client
.service_builder()
.description("product service")
.start("products", "1.0.0")
.await?;
let mut endpoint = service.endpoint("get").await?;
while let Some(request) = endpoint.next().await {
request.respond(Ok("result".into())).await?;
}
```
## Client State & Events
```rust
// Check connection state
match client.connection_state() {
State::Connected => {},
State::Disconnected => {},
State::Pending => {},
}
// Get server info
let info: ServerInfo = client.server_info();
println!("max_payload: {}", info.max_payload);
println!("jetstream: {}", info.jetstream);
// Get statistics
let stats = client.statistics();
println!("in_messages: {}", stats.in_messages.load(Ordering::Relaxed));
// Force reconnect
client.force_reconnect().await?;
// Server pool management
client.set_server_pool(["nats://s1:4222".parse()?, "nats://s2:4222".parse()?].as_slice()).await?;
let pool = client.server_pool().await?;
// Drain
client.drain().await?;
```
## Error Handling Patterns
```rust
// Connect errors
match async_nats::connect("server").await {
Err(e) => match e.kind() {
ConnectErrorKind::TimedOut => {},
ConnectErrorKind::Authentication => {},
ConnectErrorKind::AuthorizationViolation => {},
_ => {},
},
Ok(client) => {},
}
// Publish errors
match client.publish("subject", "data".into()).await {
Err(e) => match e.kind() {
PublishErrorKind::MaxPayloadExceeded => {},
PublishErrorKind::InvalidSubject => {},
PublishErrorKind::Send => {},
_ => {},
},
_ => {},
}
// Request errors
match client.request("subject", "data".into()).await {
Err(e) => match e.kind() {
RequestErrorKind::TimedOut => {},
RequestErrorKind::NoResponders => {},
RequestErrorKind::InvalidSubject => {},
RequestErrorKind::MaxPayloadExceeded => {},
_ => {},
},
Ok(message) => {},
}
```
## Feature Flag Quick Reference
| Feature | Enables | Default |
|---------|---------|---------|
| `jetstream` | JetStream streams, consumers, publish | ✅ |
| `kv` | Key-Value store (implies `jetstream`) | ✅ |
| `object-store` | Object store (implies `jetstream` + `crypto`) | ✅ |
| `service` | Service API | ✅ |
| `nkeys` | NKey/JWT authentication | ✅ |
| `nuid` | NUID-based ID generation | ✅ |
| `crypto` | SHA-256 (for object store) | ✅ |
| `websockets` | WebSocket transport | ✅ |
| `ring` | `ring` TLS crypto backend | ✅ |
| `aws-lc-rs` | `aws-lc-rs` TLS crypto backend | ❌ |
| `fips` | FIPS mode via `aws-lc-rs` | ❌ |
| `chrono` | `chrono` datetime instead of `time` | ❌ |
| `server_2_10` | Server 2.10+ features | ✅ |
| `server_2_11` | Server 2.11+ features | ✅ |
| `server_2_12` | Server 2.12+ features | ✅ |
| `server_2_14` | Server 2.14+ features | ✅ |

23
docs/research/references/nats.rs/nats-async/README.md
View File

@@ -1,23 +0,0 @@
# async-nats Reference Documentation
**Crate**: `async-nats` v0.49.1
**Source**: https://github.com/nats-io/nats.rs (`async-nats/` directory)
**License**: Apache-2.0
## Contents
| # | File | Topic |
|---|------|-------|
| 01 | [Overview & Architecture](01-overview-and-architecture.md) | Crate overview, feature flags, source structure, core connection model, dependency graph |
| 02 | [Key Types & Traits](02-key-types-and-traits.md) | `Client`, `Subscriber`, `Message`, `Request`, `ServerInfo`, `ConnectInfo`, `Statistics`, subject/header types, event/state types, error types, trait definitions |
| 03 | [Protocol & Wire Format](03-protocol-and-wire-format.md) | NATS wire protocol (PUB/HPUB/SUB/UNSUB/PING/PONG, MSG/HMSG/INFO/ERR), parser/serializer internals, vectored I/O, WebSocket transport, connection lifecycle, reconnection |
| 04 | [Connection Management](04-connection-management.md) | `ConnectOptions` builder, authentication methods, TLS configuration, reconnection callbacks, event callbacks, `ConnectionHandler` internals, multiplexer, server pool management |
| 05 | [JetStream](05-jetstream.md) | `Context` and `ContextBuilder`, streams, consumers (pull/push/ordered), JetStream messages and acks, publish with ack futures, pagination |
| 06 | [Key-Value Store](06-key-value-store.md) | KV `Store` handle, bucket CRUD, put/get/create/update/delete/purge, watch/history/keys, entry operations, mirrored buckets, KV-to-stream mapping |
| 07 | [Object Store](07-object-store.md) | `ObjectStore` handle, put/get/delete/watch/list/seal, links, `Object` (AsyncRead), chunking, SHA-256 integrity, subject naming |
| 08 | [Service API](08-service-api.md) | `Service` and `ServiceBuilder`, endpoints, groups, verb subscriptions (PING/INFO/STATS), request/respond with stats tracking |
| 09 | [Quick Reference](09-quick-reference.md) | Code examples for all major operations, feature flag reference |
## How This Documentation Was Produced
All information was derived by reading the source code of the `async-nats` crate at version 0.49.1 from the `nats.rs` repository. No external documentation was consulted — this is a ground-up reference based purely on the source.

200
docs/research/references/nats.rs/nats-server/01-overview-and-architecture.md
View File

@@ -1,200 +0,0 @@
# nats.rs: Overview and Architecture
**Version**: async-nats 0.49.1, nats-server 0.1.0
**Repository**: https://github.com/nats-io/nats.rs
**License**: Apache-2.0
**Rust Edition**: 2021
**MSRV**: 1.88.0
**Protocol**: NATS Client Protocol (INFO/CONNECT/PUB/SUB/UNSUB/PING/PONG)
## What It Is
The `nats.rs` repository contains the **official Rust client for NATS.io**, a high-performance messaging system. The active crate is **`async-nats`** — a fully async, Tokio-based NATS client. The deprecated `nats` crate (synchronous) receives security fixes only.
The `nats-server` crate is **not** an implementation of the NATS server. It is a **test harness** that spawns the Go-based `nats-server` binary for integration tests. The actual NATS server is a separate Go project at `github.com/nats-io/nats-server`.
Core design decisions:
- **Fully async** — all I/O is Tokio-based with async/await throughout
- **Cloneable Client handle** — `Client` is cheap to clone (Arc internals), all protocol work happens in a single `ConnectionHandler` task
- **Channel-based internal communication** — `Client` sends `Command` variants via `mpsc` channel to `ConnectionHandler`
- **Multiplexed request-reply** — one internal subscription handles all request-response patterns via inbox token routing
- **Automatic reconnection** — exponential backoff with configurable server pool rotation
- **Feature-gated subsystems** — JetStream, KV, Object Store, Service API, NKeys, WebSockets, and crypto backends are all optional
## Workspace Structure
```
nats.rs/
├── async-nats/ # Primary crate — async NATS client
│ ├── src/
│ │ ├── lib.rs # Entry point: connect(), ServerOp, ClientOp, Command, ConnectionHandler, Subscriber
│ │ ├── client.rs # Client handle: publish, subscribe, request, flush, drain
│ │ ├── connection.rs # Low-level I/O: protocol parsing, read/write buffers
│ │ ├── connector.rs # Connection establishment, reconnection, server pool
│ │ ├── options.rs # ConnectOptions builder
│ │ ├── auth.rs # Auth struct (credentials container)
│ │ ├── auth_utils.rs # Credential file parsing (.creds files)
│ │ ├── error.rs # Generic Error<Kind> type
│ │ ├── header.rs # HeaderMap — NATS message headers
│ │ ├── subject.rs # Subject type, ToSubject trait
│ │ ├── status.rs # StatusCode (100-999 NATS protocol codes)
│ │ ├── message.rs # Message and OutboundMessage types
│ │ ├── tls.rs # TLS configuration helpers
│ │ ├── crypto.rs # Crypto feature support
│ │ ├── id_generator.rs # NUID/rand-based unique ID generation
│ │ ├── datetime.rs # DateTime helpers for JetStream/Service
│ │ ├── jetstream/ # JetStream API (feature-gated)
│ │ │ ├── mod.rs # Module root, jetstream::new(), with_domain()
│ │ │ ├── context.rs # JetStream Context — streams, publishing, consumers
│ │ │ ├── stream.rs # Stream management, Config, Info, Consumer creation
│ │ │ ├── consumer/ # Pull, Push, Ordered consumers
│ │ │ ├── message.rs # JetStream Message with ack methods
│ │ │ ├── publish.rs # PublishAck
│ │ │ ├── response.rs # Response wrapper
│ │ │ ├── errors.rs # JetStream error codes
│ │ │ ├── account.rs # Account info
│ │ │ ├── kv/ # Key-Value store (feature: "kv")
│ │ │ └── object_store/ # Object store (feature: "object-store")
│ │ └── service/ # Service API (feature-gated)
│ │ ├── mod.rs # Service, ServiceBuilder
│ │ ├── endpoint.rs # Endpoint handling
│ │ └── error.rs # Service errors
│ ├── tests/ # Integration tests (require nats-server binary)
│ ├── examples/ # Runnable examples
│ └── benches/ # Criterion benchmarks
├── nats-server/ # Test harness — spawns Go nats-server for tests
│ ├── src/lib.rs # Server struct, run_server(), run_cluster()
│ └── configs/ # Server config files for tests
│ └── jetstream.conf
└── nats/ # DEPRECATED sync client — do not modify
```
## Architecture Diagram
```
┌──────────────────────────────────────────────────────────┐
│ Application Layer │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ JetStream│ │ KV │ │ Object │ │ Service │ │
│ │ Context │ │ Store │ │ Store │ │ API │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ └──────────────┴─────────────┴─────────────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ Client │ Cloneable handle │
│ │ (mpsc::Sender) │
│ └──────┬──────┘ │
│ │ Command channel │
└──────────────────────────┼────────────────────────────────┘
│
┌──────────────────────────┼────────────────────────────────┐
│ ConnectionHandler │
│ (single Tokio task) │
│ │ │
│ ┌───────────┐ ┌───────┴───────┐ ┌──────────────────┐ │
│ │Subscriptions│ │ Multiplexer │ │ Flush Observers │ │
│ │ HashMap │ │ (request-reply)│ │ │ │
│ └──────┬──────┘ └───────┬───────┘ └──────────────────┘ │
│ └────────────────┼ │
│ ┌──────┴──────┐ │
│ │ Connector │ Server pool, reconnect │
│ └──────┬──────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ Connection │ Protocol I/O │
│ │ (read/write)│ ServerOp / ClientOp │
│ └──────┬──────┘ │
└──────────────────────────┼────────────────────────────────┘
│
┌──────┴──────┐
│ NATS Server │ (Go binary, TCP/TLS/WS)
└─────────────┘
```
## Key Concepts
### Subject
NATS uses subject strings for message addressing. A `Subject` is a validated, immutable, UTF-8 string backed by `Bytes`. Subjects use dot-delimited tokens (e.g., `events.data.sensor1`). Wildcards `*` (single token) and `>` (multi-token suffix) are supported for subscriptions.
### ClientOp / ServerOp
The NATS client-server protocol is text-based with binary payloads. The client sends `ClientOp` variants (CONNECT, PUB/HPUB, SUB, UNSUB, PING, PONG) and receives `ServerOp` variants (INFO, MSG/HMSG, +OK, -ERR, PING, PONG).
### Command
Internal command type sent from `Client` to `ConnectionHandler` via `mpsc` channel. Includes Publish, Request, Subscribe, Unsubscribe, Flush, Drain, Reconnect, SetServerPool, ServerPool.
### Multiplexer
A single internal subscription (SID 0) that routes all request-reply responses. When a `Request` is made, a unique inbox token is registered in the multiplexer's sender map, and the response is dispatched to the corresponding `oneshot::Sender`.
### ConnectionHandler
A single Tokio task that drives all protocol I/O. It processes server operations from `Connection`, handles client commands from the `mpsc` channel, manages subscriptions, maintains ping/pong health, and orchestrates reconnection.
## nats-server Test Harness
The `nats-server` crate provides utilities for launching real NATS server instances in tests:
- `run_server(cfg)` — starts a single server with optional config
- `run_cluster(cfg)` — starts a 3-node cluster
- `Server` struct — holds the child process, cleans up on drop
- `Server::restart()` — kills and restarts the server process
- `Server::client_url()` — reads the INFO from the server to get the client URL
- `set_lame_duck_mode(server)` — sends LDM signal to the server process
The test harness spawns the Go `nats-server` binary via `std::process::Command`, using dynamic ports for parallel test execution. It auto-discovers the client URL by connecting to the server's TCP port and parsing the `INFO` JSON. On `Drop`, it kills the child process and cleans up JetStream storage directories.
## Feature Flags
```toml
# Default: everything enabled
default = ["server_2_10", "server_2_11", "server_2_12", "server_2_14",
"service", "ring", "jetstream", "nkeys", "crypto",
"object-store", "kv", "websockets", "nuid"]
# Subsystems
jetstream # JetStream API
kv # Key-Value store (requires jetstream)
object-store # Object store (requires jetstream + crypto)
service # Service API
# Crypto backends (pick one)
ring # Default crypto backend
aws-lc-rs # Alternative backend
fips # FIPS mode (requires aws-lc-rs)
# Auth
nkeys # NKey authentication
# Other
nuid # NUID-based ID generation (falls back to rand)
crypto # Encryption support
websockets # WebSocket transport
experimental # Experimental features
# Server version markers (enable version-specific API fields)
server_2_10
server_2_11
server_2_12
server_2_14
```
## Dependencies (Key)
| Dependency | Purpose |
|-----------|---------|
| `tokio` | Async runtime (macros, rt, net, sync, time, io-util) |
| `bytes` | Zero-copy byte buffers for payloads |
| `tokio-rustls` | TLS via rustls |
| `rustls-native-certs` | Load native TLS root certificates |
| `serde` / `serde_json` | JSON serialization for protocol messages and JetStream API |
| `memchr` | Fast CRLF search for protocol parsing |
| `futures-util` | Stream trait, Sink trait, StreamExt |
| `tracing` | Structured logging |
| `thiserror` | Error type derivation |
| `url` | URL parsing for server addresses |
| `portable-atomic` | Portable atomic operations |
## References
- [NATS Protocol Specification](https://docs.nats.io/reference/reference-protocols/nats-protocol)
- [NATS JetStream Documentation](https://docs.nats.io/nats-concepts/jetstream)
- [async-nats on docs.rs](https://docs.rs/async-nats)

281
docs/research/references/nats.rs/nats-server/02-protocol-and-wire-format.md
View File

@@ -1,281 +0,0 @@
# NATS Client Protocol and Wire Format
**Protocol**: NATS Client Protocol v1 (with dynamic reconfiguration)
**Transport**: TCP (port 4222), TLS, WebSocket (ws/wss)
## Protocol Overview
The NATS client-server protocol is a simple, text-based protocol with binary payload support. All operations are terminated with `\r\n`. Messages carry their payload length, allowing efficient binary data transfer.
### Connection Lifecycle
```
Client Server
│ │
│◄──────────── INFO {json} ────────────────────│ Server sends INFO first
│ │
│────────────── CONNECT {json} ────────────────►│ Client sends CONNECT
│────────────── PING ──────────────────────────►│ Client sends PING
│◄──────────── PONG ────────────────────────── │ Server confirms connection
│ │
│──── SUB/UNSUB/PUB/HPUB ──────────────────────►│ Normal operation
│◄─── MSG/HMSG/+OK/-ERR/PING ─────────────────│
│ │
```
## Server Operations (ServerOp)
These are operations received from the server. The `Connection` module parses these from the read buffer.
### INFO
Sent by the server upon connection and asynchronously when cluster topology changes.
```
INFO {json}\r\n
```
JSON fields (see `ServerInfo` struct):
| Field | Type | Description |
|-------|------|-------------|
| `server_id` | String | Unique server identifier |
| `server_name` | String | Generated server name |
| `host` | String | Cluster host |
| `port` | u16 | Cluster port |
| `version` | String | Server version |
| `auth_required` | bool | Authentication required |
| `tls_required` | bool | TLS required |
| `max_payload` | usize | Maximum payload size |
| `proto` | i8 | Protocol version (0 or 1) |
| `client_id` | u64 | Server-assigned client ID |
| `go` | String | Go build version |
| `nonce` | String | Nonce for nkey auth |
| `connect_urls` | Vec<String> | Cluster server URLs |
| `client_ip` | String | Client IP as seen by server |
| `headers` | bool | Server supports headers |
| `ldm` | bool | Lame duck mode |
| `cluster` | Option<String> | Cluster name |
| `domain` | Option<String> | NATS domain |
| `jetstream` | bool | JetStream enabled |
### MSG
Delivers a message to a subscription (no headers):
```
MSG <subject> <sid> [reply-to] <#bytes>\r\n
<payload>\r\n
```
### HMSG
Delivers a message with headers:
```
HMSG <subject> <sid> [reply-to] <#header-bytes> <#total-bytes>\r\n
<NATS/1.0 [status] [description]>\r\n
<header-name>: <header-value>\r\n
\r\n
<payload>\r\n
```
Header format follows the NATS/1.0 header spec:
- First line: `NATS/1.0` optionally followed by status code and description
- Subsequent lines: `name: value` headers
- Empty line separates headers from payload
- Header values may span multiple lines (continuation lines start with whitespace)
### PING / PONG
```
PING\r\n → Client responds with PONG
PONG\r\n → Acknowledges client's PING
```
### +OK / -ERR
```
+OK\r\n → Success acknowledgment (verbose mode)
-ERR <description>\r\n → Error from server
```
Common server errors:
- `authorization violation` → parsed as `ServerError::AuthorizationViolation`
- Other strings → `ServerError::Other(String)`
## Client Operations (ClientOp)
These are operations sent from the client to the server. The `Connection` module serializes these to the write buffer.
### CONNECT
Sent as the first client operation after receiving INFO. Contains authentication and capability information.
```
CONNECT {json}\r\n
```
JSON fields (see `ConnectInfo` struct):
| Field | Type | Description |
|-------|------|-------------|
| `verbose` | bool | Enable +OK acknowledgments (always false in this client) |
| `pedantic` | bool | Strict format checking (always false) |
| `jwt` | Option<String> | User JWT for auth |
| `nkey` | Option<String> | Public nkey for auth |
| `sig` | Option<String> | Signed nonce (Base64URL encoded) |
| `name` | Option<String> | Client name |
| `echo` | bool | Whether server should echo messages back |
| `lang` | String | Implementation language ("rust") |
| `version` | String | Client version |
| `protocol` | u8 | Protocol version (1 = dynamic) |
| `tls_required` | bool | TLS required |
| `user` | Option<String> | Username |
| `pass` | Option<String> | Password |
| `auth_token` | Option<String> | Auth token |
| `headers` | bool | Client supports headers (always true) |
| `no_responders` | bool | Client supports no-responders (always true) |
### PUB / HPUB
Publish a message:
```
PUB <subject> [reply-to] <#payload-bytes>\r\n
<payload>\r\n
```
Publish with headers:
```
HPUB <subject> [reply-to] <#header-bytes> <#total-bytes>\r\n
<NATS/1.0\r\n
<header-name>: <header-value>\r\n
\r\n
<payload>\r\n
```
### SUB
Subscribe to a subject:
```
SUB <subject> [queue-group] <sid>\r\n
```
### UNSUB
Unsubscribe from a subscription:
```
UNSUB <sid> [max]\r\n
```
The optional `max` parameter tells the server to auto-unsubscribe after receiving the specified number of messages.
### PING / PONG
```
PING\r\n → Health check / keepalive
PONG\r\n → Response to server PING
```
## Protocol Version
The `Protocol` enum has two variants:
| Value | Name | Description |
|-------|------|-------------|
| 0 | Original | Basic protocol |
| 1 | Dynamic | Supports async INFO for cluster topology changes, lame duck mode |
This client always sends `protocol: 1` (Dynamic), enabling:
- Asynchronous INFO messages with updated server lists
- Lame duck mode notifications
- Dynamic reconfiguration of cluster topology
## Wire Format Details
### Message Length Calculation
For plain `MSG`:
```
length = subject.len() + reply.map_or(0, |r| r.len()) + payload.len()
```
For `HMSG`:
```
length = subject.len() + reply.map_or(0, |r| r.len()) + header_len + payload.len()
```
Where `header_len` = serialized header bytes and `total_len` = `header_len + payload.len()`.
### Write Buffer Architecture
The `Connection` uses a two-tier write buffer:
1. **`flattened_writes`** (`BytesMut`) — for small writes (< 4096 bytes). Protocol headers, short commands, and small messages are flattened into this buffer for efficient sequential writing.
2. **`write_buf`** (`VecDeque<Bytes>`) — for large writes (>= 4096 bytes). Large payloads are appended as separate `Bytes` chunks. Supports vectored writes (`write_vectored`) when the underlying stream supports it, writing up to 64 chunks at once.
The soft limit for the total write buffer is 65,535 bytes (`SOFT_WRITE_BUF_LIMIT`). When exceeded, the `ConnectionHandler` stops processing new commands until the buffer drains.
### Read Buffer Architecture
The `Connection` uses a single `BytesMut` read buffer with configurable initial capacity (default 65,535 bytes). Protocol parsing uses `memchr::memmem::find` to locate CRLF delimiters efficiently. If a partial message is in the buffer, the parser returns `None` and waits for more data.
### Header Serialization
Headers are serialized in NATS/1.0 format:
```
NATS/1.0\r\n
Header-Name: Header-Value\r\n
Multi-Line-Header: value part 1\r\n
continuation of value\r\n
Another-Header: another value\r\n
\r\n
```
The `HeaderMap::to_bytes()` method handles this serialization, using `httparse`-compatible line folding for multi-line values.
### Status Codes in Headers
NATS status codes are embedded in the `HMSG` header version line:
```
NATS/1.0 404 No Messages\r\n
NATS/1.0 408 Request Timeout\r\n
NATS/1.0 503 No Responders\r\n
```
Common codes used by the client:
| Code | Constant | Meaning |
|------|----------|---------|
| 100 | `IDLE_HEARTBEAT` | JetStream idle heartbeat |
| 200 | `OK` | Success |
| 404 | `NOT_FOUND` | Message/stream not found |
| 408 | `TIMEOUT` | Request timeout |
| 409 | `REQUEST_TERMINATED` | Request terminated |
| 503 | `NO_RESPONDERS` | No responders available |
## Protocol Parsing Implementation
The `Connection::try_read_op()` method handles all protocol parsing:
1. Search for `\r\n` delimiter using `memchr::memmem::find`
2. Match the operation prefix:
- `+OK` → `ServerOp::Ok`
- `PING` → `ServerOp::Ping`
- `PONG` → `ServerOp::Pong`
- `-ERR` → parse error description → `ServerOp::Error`
- `INFO ` → parse JSON → `ServerOp::Info`
- `MSG ` → parse subject/sid/reply/length, read payload → `ServerOp::Message`
- `HMSG ` → parse headers + payload → `ServerOp::Message`
3. Unknown prefix → return `io::Error` with `InvalidInput`
For `MSG` and `HMSG`, if the complete payload isn't yet in the read buffer (checked via `len + payload_len + 4 > remaining`), the method returns `Ok(None)` and the buffer accumulates more data before retrying.
Non-UTF8 subjects in server messages are handled gracefully — the parser returns an `io::Error` rather than panicking, which is critical because the Go server does not enforce UTF-8 in subjects (regression fix for issue #1572).

443
docs/research/references/nats.rs/nats-server/03-key-types-and-traits.md
View File

@@ -1,443 +0,0 @@
# Key Types and Traits
This document covers the core data types in the `async-nats` crate that form the public API and internal plumbing.
## Public Types
### Client
**Location**: `client.rs`
`Client` is the primary user-facing type. It is a lightweight, cloneable handle to a NATS connection.
```rust
#[derive(Clone, Debug)]
pub struct Client {
info: tokio::sync::watch::Receiver<Option<ServerInfo>>,
state: tokio::sync::watch::Receiver<State>,
sender: mpsc::Sender<Command>,
poll_sender: PollSender<Command>,
next_subscription_id: Arc<AtomicU64>,
subscription_capacity: usize,
inbox_prefix: Arc<str>,
request_timeout: Option<Duration>,
max_payload: Arc<AtomicUsize>,
connection_stats: Arc<Statistics>,
skip_subject_validation: bool,
}
```
Key methods:
- `publish(subject, payload)` — fire-and-forget publish
- `publish_with_headers(subject, headers, payload)` — publish with NATS headers
- `publish_with_reply(subject, reply, payload)` — publish with reply subject
- `request(subject, payload)` — request-response (returns `Message`)
- `send_request(subject, request)` — request with `Request` builder
- `subscribe(subject)` — subscribe to a subject, returns `Subscriber`
- `queue_subscribe(subject, queue_group)` — subscribe as part of a queue group
- `flush()` — ensure all pending messages are written to the wire
- `drain()` — gracefully drain all subscriptions and close
- `force_reconnect()` — trigger immediate reconnection
- `new_inbox()` — generate a unique inbox subject for request-reply
- `server_info()` — get last received `ServerInfo`
- `max_payload()` — get server's maximum payload size
- `connection_state()` — get current connection `State`
- `statistics()` — get `Arc<Statistics>` for connection metrics
- `is_server_compatible(major, minor, patch)` — check server version compatibility
- `set_server_pool(addrs)` / `server_pool()` — manage server pool
`Client` also implements `Sink<OutboundMessage>` for backpressure-aware publishing.
### Subscriber
**Location**: `lib.rs`
A `Subscriber` receives messages from a single subscription. It implements `futures::Stream`.
```rust
#[derive(Debug)]
pub struct Subscriber {
sid: u64,
receiver: mpsc::Receiver<Message>,
sender: mpsc::Sender<Command>,
}
```
Key methods:
- `unsubscribe()` — unsubscribe and close the stream
- `unsubscribe_after(max)` — auto-unsubscribe after N messages
- `drain()` — gracefully drain remaining messages then close
On `Drop`, `Subscriber` automatically sends an `Unsubscribe` command and closes the receiver channel.
### Message
**Location**: `message.rs`
Represents an inbound NATS message.
```rust
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Message {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub status: Option<StatusCode>,
pub description: Option<String>,
pub length: usize,
}
```
### OutboundMessage
**Location**: `message.rs`
Represents a message to be published. No status/description fields (those are inbound-only).
```rust
#[derive(Clone, Debug)]
pub struct OutboundMessage {
pub subject: Subject,
pub reply: Option<Subject>,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
}
```
### Subject
**Location**: `subject.rs`
An immutable, validated UTF-8 string backed by `Bytes`. Used throughout the crate instead of raw `String` for subjects.
```rust
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub struct Subject {
bytes: Bytes,
}
```
Implements `Deref<Target = str>`, `From<&str>`, `From<String>`, `TryFrom<Bytes>`, `Serialize`, `Deserialize`.
Validation methods:
- `is_valid()` — checks NATS subject rules (no leading/trailing dots, no consecutive dots, no whitespace)
- `validated(s)` — construct with validation, returns `Result<Subject, SubjectError>`
- `from_static_validated(s)` — const-time validation for static strings (compile-time panic on invalid)
### ToSubject Trait
**Location**: `subject.rs`
```rust
pub trait ToSubject {
fn to_subject(&self) -> Subject;
}
```
Implemented for `Subject`, `&'static str`, `String`. All methods accepting subjects are generic over `impl ToSubject`.
### HeaderMap
**Location**: `header.rs`
NATS message headers, modeled after the `http::header` crate.
```rust
#[derive(Clone, PartialEq, Eq, Debug, Default)]
pub struct HeaderMap {
inner: HashMap<HeaderName, Vec<HeaderValue>>,
}
```
Supports multiple values per header name (like HTTP). Key methods:
- `insert(name, value)` — replace all values for a name
- `append(name, value)` — add a value to a name
- `get(name)` — get the first value
- `get_all(name)` — get all values as an iterator
- `len()` / `is_empty()` — number of header entries
- `to_bytes()` — serialize to NATS/1.0 wire format
- `wire_len()` — size in wire format (for payload size checks)
### StatusCode
**Location**: `status.rs`
NATS status codes (100-999), structurally similar to HTTP status codes.
```rust
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
pub struct StatusCode(NonZeroU16);
```
Constants:
| Constant | Code | Meaning |
|----------|------|---------|
| `IDLE_HEARTBEAT` | 100 | JetStream idle heartbeat |
| `OK` | 200 | Success |
| `NOT_FOUND` | 404 | Not found |
| `TIMEOUT` | 408 | Timeout |
| `REQUEST_TERMINATED` | 409 | Request terminated |
| `NO_RESPONDERS` | 503 | No responders |
### ServerInfo
**Location**: `lib.rs`
Deserialized from the server's `INFO` JSON message. Contains server capabilities, connection details, and cluster information.
### ConnectInfo
**Location**: `lib.rs`
Serialized into the client's `CONNECT` JSON message. Contains authentication credentials, client capabilities, and protocol preferences.
### ServerAddr
**Location**: `lib.rs`
A validated NATS server URL, supporting schemes `nats://`, `tls://`, `ws://`, `wss://`.
```rust
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ServerAddr(Url);
```
Methods:
- `from_url(url)` — validate and create
- `tls_required()` — true for `tls://` scheme
- `is_websocket()` — true for `ws://` or `wss://`
- `host()` / `port()` / `scheme()` — URL component accessors
- `socket_addrs()` — async DNS resolution
- `username()` / `password()` — embedded credentials
### Auth
**Location**: `auth.rs`
Container for authentication credentials.
```rust
#[derive(Clone, Default)]
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
### Request
**Location**: `client.rs`
Builder for customized request-response operations.
```rust
#[derive(Default)]
pub struct Request {
pub payload: Option<Bytes>,
pub headers: Option<HeaderMap>,
pub timeout: Option<Option<Duration>>,
pub inbox: Option<String>,
}
```
### Statistics
**Location**: `client.rs`
Atomic connection statistics shared between Client and ConnectionHandler.
```rust
#[derive(Default, Debug)]
pub struct Statistics {
pub in_bytes: AtomicU64,
pub out_bytes: AtomicU64,
pub in_messages: AtomicU64,
pub out_messages: AtomicU64,
pub connects: AtomicU64,
}
```
### Event
**Location**: `lib.rs`
Events emitted by the client for connection lifecycle monitoring.
```rust
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Event {
Connected,
Disconnected,
LameDuckMode,
Draining,
Closed,
SlowConsumer(u64),
ServerError(ServerError),
ClientError(ClientError),
}
```
## Internal Types
### Command
**Location**: `lib.rs`
Internal commands sent from `Client` to `ConnectionHandler` via `mpsc` channel.
```rust
pub(crate) enum Command {
Publish(OutboundMessage),
Request { subject, payload, respond, headers, sender: oneshot::Sender<Message> },
Subscribe { sid, subject, queue_group, sender: mpsc::Sender<Message> },
Unsubscribe { sid, max: Option<u64> },
Flush { observer: oneshot::Sender<()> },
Drain { sid: Option<u64> },
Reconnect,
SetServerPool { servers: Vec<ServerAddr>, result: oneshot::Sender<Result<(), String>> },
ServerPool { result: oneshot::Sender<Vec<connector::Server>> },
}
```
### ClientOp / ServerOp
**Location**: `lib.rs`
Protocol-level operation types used by `Connection` for wire format parsing and serialization.
### Subscription (Internal)
**Location**: `lib.rs`
```rust
struct Subscription {
subject: Subject,
sender: mpsc::Sender<Message>,
queue_group: Option<String>,
delivered: u64,
max: Option<u64>,
}
```
### Multiplexer (Internal)
**Location**: `lib.rs`
```rust
struct Multiplexer {
subject: Subject, // Wildcard subscription subject (e.g., "_INBOX.xxx.*")
prefix: Subject, // Prefix for routing (e.g., "_INBOX.xxx.")
senders: HashMap<String, oneshot::Sender<Message>>, // token → sender
}
```
### Connection State
**Location**: `connection.rs`
```rust
#[derive(Debug, Eq, PartialEq, Clone)]
pub enum State {
Pending,
Connected,
Disconnected,
}
```
### Protocol
**Location**: `lib.rs`
```rust
#[derive(Serialize_repr, Deserialize_repr, PartialEq, Eq, Debug, Clone, Copy)]
#[repr(u8)]
pub enum Protocol {
Original = 0,
Dynamic = 1,
}
```
## Error Type Pattern
The crate uses a generic `Error<Kind>` type throughout. Every subsystem defines its own `ErrorKind` enum and a type alias:
```rust
// Define the kind enum
#[derive(Clone, Debug, PartialEq)]
pub enum PublishErrorKind {
MaxPayloadExceeded,
InvalidSubject,
Send,
}
// Define the error type alias
pub type PublishError = Error<PublishErrorKind>;
// Construct errors
PublishError::new(PublishErrorKind::MaxPayloadExceeded)
PublishError::with_source(PublishErrorKind::Send, io_error)
// Match on errors
if err.kind() == PublishErrorKind::MaxPayloadExceeded { ... }
```
Error kinds in the crate:
| Error Type | Kind Enum | Context |
|-----------|-----------|---------|
| `ConnectError` | `ConnectErrorKind` | Initial connection failures |
| `PublishError` | `PublishErrorKind` | Publish validation failures |
| `RequestError` | `RequestErrorKind` | Request-response failures |
| `SubscribeError` | `SubscribeErrorKind` | Subscription failures |
| `FlushError` | `FlushErrorKind` | Flush failures |
| `ServerPoolError` | `ServerPoolErrorKind` | Server pool query failures |
| `SetServerPoolError` | `SetServerPoolErrorKind` | Server pool modification failures |
## Trait Implementations
### Client Trait Interfaces
The `Client` implements several traits defined in `client::traits`:
```rust
// Publisher trait — publish with optional reply subject
trait Publisher {
fn publish_with_reply<S, R>(&self, subject: S, reply: R, payload: Bytes) -> impl Future<Output = Result<(), PublishError>>;
fn publish_message(&self, msg: OutboundMessage) -> impl Future<Output = Result<(), PublishError>>;
}
// Subscriber trait — subscribe to a subject
trait Subscriber {
fn subscribe<S>(&self, subject: S) -> impl Future<Output = Result<crate::Subscriber, SubscribeError>>;
}
// Requester trait — send request-response
trait Requester {
fn send_request<S>(&self, subject: S, request: Request) -> impl Future<Output = Result<Message, RequestError>>;
}
// TimeoutProvider trait — access request timeout
trait TimeoutProvider {
fn timeout(&self) -> Option<Duration>;
}
```
### ToServerAddrs Trait
**Location**: `lib.rs`
Converts various address types into server address iterators. Implemented for `ServerAddr`, `str`, `String`, `&[T]`, `Vec<T>`, `&[ServerAddr]`, and references.
### Sink<OutboundMessage>
`Client` implements `futures::Sink<OutboundMessage>` for backpressure-aware publishing through the `PollSender` adapter.
### Stream for Subscriber
`Subscriber` implements `futures::Stream` with `Item = Message`, delegating to the internal `mpsc::Receiver`.

338
docs/research/references/nats.rs/nats-server/04-connection-handler-and-data-flow.md
View File

@@ -1,338 +0,0 @@
# Connection Handler and Data Flow
This document covers the internal `ConnectionHandler` that drives all protocol I/O, and the data flow through the system.
## ConnectionHandler
**Location**: `lib.rs`
The `ConnectionHandler` is the heart of the client. It runs as a single Tokio task and manages all communication with the NATS server.
```rust
pub(crate) struct ConnectionHandler {
connection: Connection, // Low-level I/O
connector: Connector, // Server pool, reconnection
subscriptions: HashMap<u64, Subscription>, // Active subscriptions
multiplexer: Option<Multiplexer>, // Request-reply multiplexer
pending_pings: usize, // Unanswered PINGs
info_sender: tokio::sync::watch::Sender<Option<ServerInfo>>,
ping_interval: Interval, // Periodic PING timer
should_reconnect: bool, // Flag for forced reconnect
flush_observers: Vec<oneshot::Sender<()>>, // Pending flush callbacks
is_draining: bool, // Connection is draining
drain_pings: VecDeque<u64>, // SIDs being drained
}
```
## Data Flow: Publish
```
Application
│
│ client.publish("events.data", payload)
│
▼
Client
│ validates subject & payload size
│ sends Command::Publish(OutboundMessage) via mpsc channel
│
▼
ConnectionHandler::handle_command(Command::Publish)
│ increments out_messages, out_bytes statistics
│ calls connection.enqueue_write_op(&ClientOp::Publish { ... })
│
▼
Connection::enqueue_write_op
│ serializes to wire format:
│ "PUB events.data 11\r\n" or "HPUB events.data 23 34\r\n"
│ appends to flattened_writes or write_buf
│
▼
Connection::poll_write
│ uses vectored writes (64 chunks) if supported
│ or sequential writes otherwise
│
▼
Connection::poll_flush
│ flushes the TCP/TLS/WS stream
│ notifies flush_observers
│
▼
NATS Server (TCP/TLS/WebSocket)
```
## Data Flow: Subscribe
```
Application
│
│ client.subscribe("events.>")
│
▼
Client::subscribe
│ validates subject (always, regardless of skip_subject_validation)
│ allocates next sid via AtomicU64
│ creates mpsc channel for messages
│ sends Command::Subscribe { sid, subject, sender }
│ returns Subscriber { sid, receiver }
│
▼
ConnectionHandler::handle_command(Command::Subscribe)
│ creates Subscription { subject, sender, delivered: 0, max: None }
│ inserts into subscriptions HashMap
│ calls connection.enqueue_write_op(&ClientOp::Subscribe { sid, subject, queue_group })
│
▼
Connection::enqueue_write_op
│ serializes: "SUB events.> 42\r\n"
│
▼
Server sends MSG for matching subjects:
│
▼
ConnectionHandler::handle_server_op(ServerOp::Message { sid, subject, ... })
│ looks up sid in subscriptions HashMap
│ constructs Message { subject, reply, payload, headers, status, description }
│ tries subscription.sender.try_send(message)
│
├── Ok → increments subscription.delivered, checks max
├── Full → emits Event::SlowConsumer(sid)
└── Closed → removes subscription, sends ClientOp::Unsubscribe
│
▼
Subscriber::poll_next (Stream impl)
│ receives from mpsc::Receiver
│
▼
Application processes Message
```
## Data Flow: Request-Response
The request-response pattern uses the **multiplexer** — a single wildcard subscription that routes responses to their waiting requesters.
```
Application
│
│ client.request("service", payload)
│
▼
Client::send_request
│ validates subject & payload size
│ creates oneshot channel for response
│ generates unique inbox: "_INBOX.<nuid>.<token>"
│ sends Command::Request { subject, payload, respond, sender }
│
▼
ConnectionHandler::handle_command(Command::Request)
│ extracts token from respond subject (after last '.')
│ if no multiplexer exists:
│ creates Multiplexer with wildcard sub "_INBOX.<id>.*" (SID 0)
│ sends ClientOp::Subscribe { sid: 0, subject: "_INBOX.<id>.*" }
│ inserts token → oneshot::Sender in multiplexer.senders
│ sends ClientOp::Publish { subject, payload, respond: "<prefix><token>" }
│
▼
Server routes request to service:
│
▼
Service responds by publishing to the reply subject:
│
▼
ConnectionHandler::handle_server_op(ServerOp::Message { sid: 0, ... })
│ sid == MULTIPLEXER_SID (0), so enters multiplexer path
│ extracts token by stripping prefix from subject
│ looks up token in multiplexer.senders
│ sends Message via oneshot::Sender
│
▼
Client::send_request receives via oneshot::Receiver
│ applies timeout (default 10s)
│ checks for NO_RESPONDERS status (503)
│
▼
Application receives Message
```
### Custom Inbox Request
If the `Request` builder specifies a custom `inbox`, the flow is different:
- The client subscribes to the inbox directly (not via multiplexer)
- Publishes with the inbox as the reply subject
- Waits for the message on that subscription
- No multiplexer involvement
## Data Flow: Flush
```
Application
│
│ client.flush()
│
▼
Client::flush
│ creates oneshot channel
│ sends Command::Flush { observer }
│
▼
ConnectionHandler::handle_command(Command::Flush)
│ pushes observer into flush_observers Vec
│
▼
ProcessFut::poll (main loop)
│ after writing all pending data...
│ checks should_flush():
│ Yes (write buffers empty, not yet flushed) → poll_flush
│ May (write buffers not empty) → poll_flush
│ No (already flushed) → skip
│ on successful flush:
│ drains flush_observers, sending () to each
│
▼
Client::flush receives via oneshot::Receiver
```
## Data Flow: Drain
```
Application
│
│ client.drain() or subscriber.drain()
│
▼
Client::drain / Subscriber::drain
│ sends Command::Drain { sid: None } (whole client)
│ or Command::Drain { sid: Some(n) } (single subscription)
│
▼
ConnectionHandler::handle_command(Command::Drain)
│ if sid is Some:
│ pushes sid to drain_pings
│ sends ClientOp::Unsubscribe { sid, max: None }
│ if sid is None (whole client):
│ sets is_draining = true
│ emits Event::Draining
│ for each subscription: drain_pings.push(sid), Unsubscribe
│ sends ClientOp::Ping (to flush the UNSUB messages)
│
▼
ProcessFut::poll (main loop)
│ processes any remaining server messages
│ removes drained subscriptions from HashMap
│ if is_draining: returns ExitReason::Closed
│
▼
ConnectionHandler exits, emits Event::Closed
```
## Main Processing Loop
The `ConnectionHandler::process` method implements the core event loop via a custom `Future` (`ProcessFut`):
```rust
impl Future for ProcessFut<'_> {
type Output = ExitReason;
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
// 1. Check ping interval — send PING if due, disconnect if too many pending
while self.handler.ping_interval.poll_tick(cx).is_ready() {
if let Poll::Ready(exit) = self.ping() { return Poll::Ready(exit); }
}
// 2. Read all available server operations
loop {
match self.handler.connection.poll_read_op(cx) {
Poll::Pending => break,
Poll::Ready(Ok(Some(server_op))) => self.handler.handle_server_op(server_op),
Poll::Ready(Ok(None)) => return Poll::Ready(ExitReason::Disconnected(None)),
Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))),
}
}
// 3. Clean up drained subscriptions
while let Some(sid) = self.handler.drain_pings.pop_front() {
self.handler.subscriptions.remove(&sid);
}
// 4. If draining, exit
if self.handler.is_draining { return Poll::Ready(ExitReason::Closed); }
// 5. Process client commands (batch of up to 16)
// while write buffer not full
loop {
while !self.handler.connection.is_write_buf_full() {
match receiver.poll_recv_many(cx, recv_buf, 16) {
Poll::Pending => break,
Poll::Ready(1..) => { for cmd in recv_buf.drain(..) { handler.handle_command(cmd); } }
Poll::Ready(0) => return Poll::Ready(ExitReason::Closed),
}
}
// 6. Write pending data to stream
match self.handler.connection.poll_write(cx) {
Poll::Pending => break,
Poll::Ready(Ok(())) => continue, // write buffer empty, try more commands
Poll::Ready(Err(err)) => return Poll::Ready(ExitReason::Disconnected(Some(err))),
}
}
// 7. Flush stream and notify observers
match self.handler.connection.poll_flush(cx) { ... }
// 8. Check for forced reconnect
if mem::take(&mut self.handler.should_reconnect) {
return Poll::Ready(ExitReason::ReconnectRequested);
}
Poll::Pending
}
}
```
### Exit Reasons
The main loop exits for three reasons:
| Reason | Action |
|--------|--------|
| `Disconnected(Option<io::Error>)` | Attempt reconnection via `handle_disconnect()` |
| `ReconnectRequested` | Force reconnect (user-triggered) |
| `Closed` | Connection handler terminates, emit `Event::Closed` |
On disconnection, `handle_disconnect()` is called which:
1. Resets `pending_pings` to 0
2. Emits `Event::Disconnected`
3. Updates connection state to `Disconnected`
4. Calls `handle_reconnect()` which uses `Connector::connect()`
5. On successful reconnect, re-subscribes all active subscriptions
6. Re-subscribes the multiplexer wildcard if present
## Slow Consumer Handling
When a subscription's `mpsc::Sender` channel is full (the application isn't consuming messages fast enough):
1. `try_send` returns `TrySendError::Full`
2. The `ConnectionHandler` emits `Event::SlowConsumer(sid)`
3. The message is **dropped** (not queued)
4. The subscription remains active
When a subscription's receiver is dropped (application closed the stream):
1. `try_send` returns `TrySendError::Closed`
2. The subscription is removed from the HashMap
3. An `UNSUB` command is sent to the server
## Ping/Pong Health Check
The `ConnectionHandler` maintains a periodic PING interval (default 60 seconds):
1. `ping_interval` fires every N seconds
2. A `ClientOp::Ping` is enqueued
3. `pending_pings` counter increments
4. If `pending_pings > MAX_PENDING_PINGS (2)`, the connection is considered dead
5. When `ServerOp::Pong` is received, `pending_pings` decrements
6. Any server operation resets the ping interval timer
## Batched Command Processing
Commands from the `Client` are received in batches of up to 16 (`RECV_CHUNK_SIZE`) using `poll_recv_many`. This amortizes the cost of waking the task and enables pipelining multiple operations (e.g., publishing many messages) in a single poll cycle.

277
docs/research/references/nats.rs/nats-server/05-connection-and-reconnection.md
View File

@@ -1,277 +0,0 @@
# Connection and Reconnection
This document covers how connections are established, TLS handling, the server pool, and the reconnection mechanism.
## Connector
**Location**: `connector.rs`
The `Connector` manages the server pool and handles connection establishment and reconnection.
```rust
pub(crate) struct Connector {
servers: Vec<Server>, // Server pool with per-server metadata
options: ConnectorOptions, // Connection configuration
connect_stats: Arc<Statistics>, // Shared statistics
attempts: usize, // Global reconnection attempt counter
events_tx: mpsc::Sender<Event>, // Event channel
state_tx: watch::Sender<State>, // Connection state watcher
max_payload: Arc<AtomicUsize>, // Server's max payload
last_info: ServerInfo, // Last known server info
}
```
### Server Pool
Each server in the pool carries metadata:
```rust
#[derive(Debug, Clone)]
pub struct Server {
pub addr: ServerAddr,
pub failed_attempts: usize, // Consecutive failed attempts
pub did_connect: bool, // Ever successfully connected?
pub is_discovered: bool, // Discovered via INFO, not user-configured
pub last_error: Option<String>, // Last connection error
}
```
### ConnectorOptions
```rust
pub(crate) struct ConnectorOptions {
pub tls_required: bool,
pub certificates: Vec<PathBuf>,
pub client_cert: Option<PathBuf>,
pub client_key: Option<PathBuf>,
pub tls_client_config: Option<rustls::ClientConfig>,
pub tls_first: bool,
pub auth: Auth,
pub no_echo: bool,
pub connection_timeout: Duration, // Default: 5 seconds
pub name: Option<String>,
pub ignore_discovered_servers: bool,
pub retain_servers_order: bool,
pub read_buffer_capacity: u16, // Default: 65535
pub reconnect_delay_callback: Arc<dyn Fn(usize) -> Duration>,
pub auth_callback: Option<CallbackArg1<Vec<u8>, Result<Auth, AuthError>>>,
pub max_reconnects: Option<usize>,
pub local_address: Option<SocketAddr>,
pub reconnect_to_server_callback: Option<ReconnectToServerCallback>,
}
```
## Connection Establishment Flow
```
Connector::try_connect_to_server(addr)
│
├── 1. DNS resolution
│ server_addr.socket_addrs()
│
├── 2. For each resolved address:
│ │
│ ├── 2a. Connect with timeout
│ │ tokio::time::timeout(connection_timeout, try_connect_to(socket_addr, ...))
│ │
│ └── 2b. try_connect_to():
│ │
│ ├── Select transport:
│ │ ├── "ws" → WebSocket (tokio_websockets)
│ │ ├── "wss" → WebSocket over TLS
│ │ └── default → TCP (TcpStream)
│ │
│ ├── Optional: bind to local_address
│ ├── Set TCP_NODELAY
│ ├── Create Connection with read_buffer_capacity
│ │
│ ├── If tls_first: upgrade to TLS before INFO
│ │
│ ├── Read INFO from server
│ │
│ ├── If TLS required (by option, server, or URL scheme):
│ │ upgrade to TLS (rustls)
│ │
│ ├── Discover servers from INFO.connect_urls
│ │ (unless ignore_discovered_servers)
│ │
│ ├── Build ConnectInfo with auth:
│ │ ├── username/password (from Auth or URL)
│ │ ├── token (from Auth)
│ │ ├── nkey + signed nonce (feature: nkeys)
│ │ ├── JWT + signature callback (feature: nkeys)
│ │ └── auth_callback (custom async callback)
│ │
│ ├── Send CONNECT + PING
│ │
│ └── Wait for response:
│ ├── -ERR (authorization violation) → error
│ ├── PONG or +OK → success
│ └── EOF → error
│
└── 3. On success:
├── Reset attempt counter
├── Increment connects statistic
├── Emit Event::Connected
├── Update State::Connected
├── Store max_payload
├── Update per-server metadata (did_connect, failed_attempts)
└── Return (ServerInfo, Connection)
```
## TLS Handling
The client supports three TLS modes:
### 1. Standard TLS (INFO → TLS)
Default behavior. The client receives the `INFO` message in plaintext, then upgrades to TLS if:
- `tls_required` option is set
- Server's `INFO.tls_required` is true
- URL scheme is `tls://`
### 2. TLS First (TLS → INFO)
When `ConnectOptions::tls_first()` is enabled, the client establishes TLS before reading INFO. This requires the server to have `handshake_first` enabled. Useful for environments where plaintext INFO is not acceptable.
### 3. WebSocket TLS
For `wss://` URLs, TLS is handled by the WebSocket library (`tokio-websockets`) directly, not by the client's TLS layer.
### TLS Configuration
The client uses `rustls` via `tokio-rustls`. Configuration steps:
1. Load root certificates from system store (`rustls-native-certs`)
2. Optionally add custom root certificates from PEM files
3. Optionally configure client certificate and key for mTLS
4. Optionally pass a custom `rustls::ClientConfig`
Crypto backend is selectable via feature flags:
- `ring` (default)
- `aws-lc-rs`
- `fips` (requires aws-lc-rs)
## Reconnection
### Reconnection Trigger
Reconnection is triggered when:
1. I/O error during read or write (`ExitReason::Disconnected`)
2. Too many pending PINGs (no PONG received)
3. User calls `Client::force_reconnect()` (`ExitReason::ReconnectRequested`)
### Reconnection Flow
```
ConnectionHandler::handle_disconnect()
│
├── Reset pending_pings to 0
├── Emit Event::Disconnected
├── Update State::Disconnected
│
└── handle_reconnect()
│
└── Connector::connect()
│
└── Loop: try_connect()
│
├── If reconnect_to_server_callback is set:
│ │ Call callback with (server_pool, server_info)
│ │ If returns Some(ReconnectToServer):
│ │ Validate server is in pool
│ │ Use callback's delay or default backoff
│ │ Try connecting to selected server
│ └── If None or invalid: fall through to default
│
├── Default selection:
│ ├── Shuffle servers (unless retain_servers_order)
│ ├── Sort by failed_attempts (ascending)
│ └── Try each server in order
│
├── For each server:
│ ├── Increment attempts counter
│ ├── Check max_reconnects limit
│ ├── Apply reconnect delay (exponential backoff)
│ └── try_connect_to_server(addr)
│
├── On success:
│ ├── Reset attempts to 0
│ ├── Re-subscribe all active subscriptions
│ │ (filter out closed subscription channels)
│ ├── Re-subscribe multiplexer wildcard
│ └── Return (ServerInfo, Connection)
│
└── On failure:
├── Update per-server metadata (failed_attempts, last_error)
├── Auth errors → propagate immediately
└── Other errors → continue to next server
```
### Exponential Backoff
Default reconnect delay function:
```rust
fn reconnect_delay_callback_default(attempts: usize) -> Duration {
if attempts <= 1 {
Duration::from_millis(0)
} else {
let exp: u32 = (attempts - 1).try_into().unwrap_or(u32::MAX);
let max = Duration::from_secs(4);
cmp::min(Duration::from_millis(2_u64.saturating_pow(exp)), max)
}
}
```
| Attempt | Delay |
|---------|-------|
| 1 | 0ms |
| 2 | 0ms |
| 3 | 2ms |
| 4 | 4ms |
| 5 | 8ms |
| ... | ... |
| 13 | 4096ms |
| 14+ | 4000ms (capped) |
Custom delay functions can be provided via `ConnectOptions::reconnect_delay_callback()`.
### Server Pool Updates
The server pool is dynamic:
1. **Initial pool**: from `connect()` / `ConnectOptions::connect()` URL(s)
2. **Discovered servers**: added from `INFO.connect_urls` on each connection (unless `ignore_discovered_servers` is set)
3. **Runtime updates**: via `Client::set_server_pool()` — replaces the entire pool while preserving per-server state for servers that appear in both old and new pools
4. **Order**: servers are shuffled by default (random selection), unless `retain_servers_order` is set
### Max Reconnects
The `max_reconnects` option limits total reconnection attempts:
- `None` or `0` → unlimited (default)
- `Some(n)` → give up after `n` total attempts
- Counter is reset on successful connection and when `set_server_pool()` is called
## ConnectOptions Defaults
| Option | Default |
|--------|---------|
| `connection_timeout` | 5 seconds |
| `ping_interval` | 60 seconds |
| `sender_capacity` | 2048 |
| `subscription_capacity` | 65536 |
| `inbox_prefix` | `"_INBOX"` |
| `request_timeout` | 10 seconds |
| `retry_on_initial_connect` | false |
| `ignore_discovered_servers` | false |
| `retain_servers_order` | false |
| `read_buffer_capacity` | 65535 |
| `skip_subject_validation` | false |
| `no_echo` | false |
| `tls_required` | false |
| `tls_first` | false |
| `max_reconnects` | None (unlimited) |
## Background Connection
When `ConnectOptions::retry_on_initial_connect()` is enabled, the `connect()` function returns a `Client` immediately, before the connection is established. The connection is established in a background Tokio task. This means:
- `client.server_info()` returns `ServerInfo::default()` until connected
- `client.connection_state()` returns `State::Pending`
- Operations like `publish()` will queue in the command channel
- The `Client` becomes usable once the background task connects

472
docs/research/references/nats.rs/nats-server/06-jetstream-internals.md
View File

@@ -1,472 +0,0 @@
# JetStream Internals
This document covers the JetStream subsystem — how it provides stream-based messaging with persistence, consumer management, and higher-level APIs like KV and Object Store.
## JetStream Context
**Location**: `jetstream/context.rs`
The `Context` is the entry point to the JetStream API. It wraps a `Client` and provides stream management, publishing, and consumer operations.
```rust
#[derive(Debug, Clone)]
pub struct Context {
pub(crate) client: Client,
pub(crate) prefix: String, // API subject prefix (default: "$JS.API")
pub(crate) timeout: Duration, // Default request timeout
pub(crate) max_ack_semaphore: Arc<Semaphore>, // Limits in-flight ack waits
pub(crate) ack_sender: mpsc::Sender<(oneshot::Receiver<Message>, OwnedSemaphorePermit)>,
pub(crate) backpressure_on_inflight: bool,
}
```
### Context Creation
```rust
// Default context (prefix = "$JS.API")
let jetstream = async_nats::jetstream::new(client);
// With domain (prefix = "$JS.hub.API")
let jetstream = async_nats::jetstream::with_domain(client, "hub");
// With custom prefix
let jetstream = async_nats::jetstream::with_prefix(client, "JS.acc@hub.API");
// Builder pattern for more options
let jetstream = async_nats::jetstream::Context::builder(client)
.domain("hub")
.prefix("$JS.API")
.timeout(Duration::from_secs(30))
.max_ack_pending(256)
.backpressure_on_inflight(true)
.build();
```
### JetStream API Subject Convention
All JetStream API calls are request-response messages sent to subjects following the pattern:
```
$JS.API.<operation>.<stream-name>[.<consumer-name>]
```
Examples:
- `$JS.API.STREAM.CREATE.events` — create stream "events"
- `$JS.API.STREAM.INFO.events` — get stream info
- `$JS.API.CONSUMER.DURABLE.CREATE.events.myconsumer` — create durable consumer
- `$JS.API.CONSUMER.MSG.NEXT.events.myconsumer` — pull next message
With a domain, the prefix changes to `$JS.<domain>.API`.
## Stream Management
**Location**: `jetstream/stream.rs`
### Stream Config
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Config {
pub name: String,
pub subjects: Vec<String>, // Subject filter
pub retention: RetentionPolicy, // Limits, Interest, WorkQueue
pub max_consumers: i32,
pub max_messages: i64, // Per-stream message limit
pub max_messages_per_subject: i64,
pub max_bytes: i64, // Per-stream byte limit
pub max_age: Duration, // Message TTL
pub max_message_size: Option<i32>, // Max individual message size
pub storage: StorageType, // File or Memory
pub num_replicas: usize,
pub no_ack: bool, // Don't require ack
pub discard: DiscardPolicy, // Old or New
pub duplicate_window: Duration,
pub allow_rollup_hdrs: bool,
pub allow_direct: bool,
pub mirror: Option<External>,
pub sources: Vec<External>,
pub sealed: bool,
pub compression: Option<Compression>, // server_2_10+
pub first_sequence: Option<u64>, // server_2_11+
pub subject_transform: Option<SubjectTransform>, // server_2_12+
pub metadata: Option<HashMap<String, String>>, // server_2_10+
pub placement: Option<Placement>,
pub republish: Option<RePublish>,
}
```
### Stream Operations
Via `Context`:
| Method | API Subject | Description |
|--------|------------|-------------|
| `create_stream(config)` | `STREAM.CREATE.<name>` | Create a new stream |
| `get_stream(name)` | `STREAM.INFO.<name>` | Get existing stream |
| `get_or_create_stream(config)` | `STREAM.INFO` → `STREAM.CREATE` | Get or create |
| `delete_stream(name)` | `STREAM.DELETE.<name>` | Delete a stream |
| `update_stream(name, config)` | `STREAM.UPDATE.<name>` | Update stream config |
| `purge_stream(name)` | `STREAM.PURGE.<name>` | Purge all messages |
| `streams()` | `STREAM.LIST` | List all streams (paged iterator) |
| `stream_names()` | `STREAM.NAMES` | List stream names (paged iterator) |
| `account_info()` | `ACCOUNT.INFO` | Get account info |
Via `Stream`:
| Method | API Subject | Description |
|--------|------------|-------------|
| `info()` | `STREAM.INFO.<name>` | Refresh stream info |
| `purge()` | `STREAM.PURGE.<name>` | Purge messages |
| `delete()` | `STREAM.DELETE.<name>` | Delete this stream |
| `update(config)` | `STREAM.UPDATE.<name>` | Update config |
| `get_raw_message(seq)` | `STREAM.MSG.GET.<name>` | Get message by sequence (stored mode) |
| `get_last_message(subject)` | `STREAM.MSG.GET.<name>` | Get last message for subject (stored mode) |
| `direct_get_last(subject)` | `DIRECT.GET.<name>` | Direct get last (bypasses RAA) |
| `direct_get(seq)` | `DIRECT.GET.<name>` | Direct get by sequence |
| `delete_message(seq)` | `STREAM.MSG.DELETE.<name>` | Delete a specific message |
| `create_consumer(config)` | `CONSUMER.CREATE.<stream>` | Create consumer |
| `get_or_create_consumer(name, config)` | `CONSUMER.DURABLE.CREATE.<stream>.<name>` | Get or create durable |
| `get_consumer(name)` | `CONSUMER.INFO.<stream>.<name>` | Get existing consumer |
### Stream Info
```rust
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Info {
pub config: Config,
pub created: DateTime,
pub state: State, // Messages, bytes, first/last sequence, consumer count
pub cluster: Option<ClusterInfo>,
pub timestamp: DateTime,
pub leader: Option<String>,
pub subjects: Option<HashMap<String, u64>>, // Subject → message count
}
```
### Paged List Operations
Stream and consumer listing uses a paged iterator pattern:
```rust
// streams() returns an iterator that automatically pages
let mut streams = jetstream.streams();
while let Some(stream) = streams.next().await {
let stream = stream?;
// process stream
}
// stream_names() similarly pages
let mut names = jetstream.stream_names();
while let Some(name) = names.next().await {
println!("{}", name?);
}
```
The paged iterator sends an initial request with `offset: 0` and continues fetching pages until no more results are returned.
## Publishing
**Location**: `jetstream/context.rs`, `jetstream/publish.rs`
### Publish
```rust
// Basic publish (fire-and-forget)
jetstream.publish("events.data", "payload".into()).await?;
// Publish with custom message builder
jetstream.publish_message(
jetstream::message::PublishMessage::build()
.payload("data".into())
.message_id("unique-id") // Nats-Msg-Id header for dedup
.expected_last_message_id("prev") // Nats-Expected-Last-Msg-Id
.expected_last_sequence(42) // Nats-Expected-Last-Sequence
.expected_last_subject_sequence("events", 10) // Per-subject sequence
.header("Custom", "Value")
).await?;
```
### PublishAck
When a message is published to a JetStream stream, the server responds with a `PublishAck`:
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct PublishAck {
pub stream: String,
pub sequence: u64,
pub domain: Option<String>,
pub duplicate: bool,
}
```
### PublishAckFuture
Publishing returns a `PublishAckFuture` that resolves to `PublishAck`. The future uses a semaphore (`max_ack_semaphore`) to limit in-flight ack waits and prevent backpressure issues.
When `backpressure_on_inflight` is enabled, the publish operation blocks if there are too many pending acks, preventing the command channel from filling up with unbounded publish operations.
### Idempotent Publishing
Headers for exactly-once semantics:
| Header | Purpose |
|--------|---------|
| `Nats-Msg-Id` | Message ID for deduplication within the stream's duplicate window |
| `Nats-Expected-Last-Msg-Id` | Expected last message ID (conditional publish) |
| `Nats-Expected-Last-Sequence` | Expected last sequence number |
| `Nats-Expected-Last-Subject-Sequence` | Expected last sequence for a specific subject |
## Consumers
**Location**: `jetstream/consumer/`
### Consumer Types
| Type | Description |
|------|-------------|
| `PullConsumer` | Client pulls messages on demand |
| `PushConsumer` | Server pushes messages to a delivery subject |
| `OrderedConsumer` | Push consumer with automatic re-creation on failure |
### Consumer Config
```rust
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Config {
pub name: Option<String>,
pub durable_name: Option<String>,
pub description: Option<String>,
pub deliver_subject: Option<String>, // Push consumers only
pub ack_policy: AckPolicy,
pub ack_wait: Duration,
pub max_deliver: i64,
pub max_ack_pending: i32,
pub max_waiting: i32, // Pull consumers only
pub filter_subject: Option<String>,
pub replay_policy: ReplayPolicy,
pub sample_frequency: Option<i8>,
pub max_batch: i32, // Pull consumers
pub max_expires: Duration, // Pull consumers
pub inactive_threshold: Duration,
pub flow_control: bool, // Push consumers
pub heartbeat: Option<Duration>, // Push consumers
pub backoff: Vec<Duration>,
pub deliver_group: Option<String>,
pub num_replicas: usize,
pub mem_storage: bool,
pub metadata: Option<HashMap<String, String>>,
pub ack_markers: Option<Vec<String>>, // server_2_12+
}
```
### Pull Consumer
**Location**: `jetstream/consumer/pull.rs`
Pull consumers require explicit requests for messages:
```rust
// Batch request
let mut messages = consumer.messages().await?.take(100);
while let Some(message) = messages.next().await {
let message = message?;
message.ack().await?;
}
// Sequence-based batch
let mut batches = consumer.sequence(50)?.take(10);
while let Some(mut batch) = batches.try_next().await? {
while let Some(Ok(message)) = batch.next().await {
message.ack().await?;
}
}
// Single message fetch
let message = consumer.fetch().await?;
```
Pull requests are sent to: `$JS.API.CONSUMER.MSG.NEXT.<stream>.<consumer>`
The request payload is JSON:
```json
{"batch": 10, "expires": 5000, "no_wait": false}
```
### Push Consumer
**Location**: `jetstream/consumer/push.rs`
Push consumers receive messages automatically on a delivery subject. The client subscribes to the delivery subject and processes messages as they arrive.
Features:
- **Flow control** — server sends flow control messages, client responds to maintain delivery rate
- **Heartbeats** — idle heartbeats (status code 100) when no messages are available
- **Ordered consumers** — automatically recreated on delivery failures with correct sequence positioning
### Acknowledgment
**Location**: `jetstream/message.rs`
JetStream messages support multiple acknowledgment types:
```rust
pub enum AckKind {
Ack, // Ack (message processed)
Nack, // Nak (re-deliver)
Progress, // Progress (still working)
Next, // Next (ack + pull next)
Term, // Term (don't redeliver, remove from stream)
All, // Ack all messages up to this sequence
}
```
Methods on JetStream `Message`:
- `ack()` — simple acknowledgment
- `ack_with(kind)` — acknowledgment with specific type
- `double_ack()` — exactly-once ack (ACK + separate ack message)
- `nack()` — negative acknowledgment (request redelivery)
- `in_progress()` — progress indicator
- `term()` — terminate message (no redelivery)
## JetStream Message
**Location**: `jetstream/message.rs`
JetStream messages wrap core `Message` with metadata extracted from headers:
```rust
#[derive(Debug)]
pub struct Message {
pub message: crate::Message, // The underlying NATS message
pub context: Context, // JetStream context for acking
pub ack_pending: Arc<AtomicU64>, // Pending ack counter
}
impl Message {
pub fn info(&self) -> Result<Info, MessageInfoError> // Parse message info from headers
pub async fn ack(&self) -> Result<(), AckError>
pub async fn ack_with(&self, kind: AckKind) -> Result<(), AckError>
pub async fn double_ack(&self) -> Result<(), AckError>
pub async fn nack(&self) -> Result<(), AckError>
pub async fn in_progress(&self) -> Result<(), AckError>
pub async fn term(&self) -> Result<(), AckError>
}
```
Message info is extracted from the `HMSG` headers:
- `Nats-Stream` — stream name
- `Nats-Consumer` — consumer name
- `Nats-Delivered` — delivery count
- `Nats-Sequence` — stream sequence
- `Nats-Time-Stamp` — timestamp
- `Nats-Subject` — original subject
- `Nats-Pending-Messages` / `Nats-Pending-Bytes` — pending counts
## Key-Value Store
**Location**: `jetstream/kv/`
The KV store is a JetStream-based key-value API. Each bucket maps to a JetStream stream with specific configuration:
```rust
// Create a KV store
let kv = jetstream
.create_key_value(async_nats::jetstream::kv::Config {
bucket: "my_bucket".to_string(),
history: 5, // Max history per key (1-64)
ttl: Duration::from_secs(3600), // Key TTL
max_bytes: 1024 * 1024, // Max bucket size
storage: StorageType::File,
replicas: 1,
..Default::default()
})
.await?;
```
Under the hood:
- Each key is stored as a message with subject `$KV.<bucket>.<key>`
- Keys support wildcard patterns (`$KV.bucket.prefix.*`)
- History is managed via stream `max_messages_per_subject`
- TTL is managed via stream `max_age`
- `put(key, value)` publishes to the key subject
- `get(key)` reads the last message for the key subject
- `delete(key)` publishes an internal delete marker
- `purge(key)` uses stream purge API
- `watch()` subscribes to key changes and returns a `Watch` stream
- `keys()` / `history(key)` list keys and history
## Object Store
**Location**: `jetstream/object_store/`
The Object Store provides large object storage built on JetStream. Objects are chunked and stored across multiple messages in a stream.
```rust
// Create an object store
let store = jetstream
.create_object_store(async_nats::jetstream::object_store::Config {
bucket: "my_objects".to_string(),
..Default::default()
})
.await?;
// Put an object
let info = store.put("file.txt", stream).await?;
// Get an object
let mut object_stream = store.get("file.txt").await?;
```
Under the hood:
- Objects are chunked into ~128KB messages
- Metadata (object info) is stored as the first "chunk 0" message
- Each chunk is a message with subject `$OBJ.<bucket>.<object-nuid>.C<chunk-number>`
- Metadata includes: name, description, headers, size, chunks, digest (SHA-256)
- `get()` returns a stream of chunks
- Links allow referencing one object from another (like symlinks)
## JetStream Error Codes
**Location**: `jetstream/errors.rs`
Standard JetStream error codes returned by the server:
| Code | Constant | Description |
|------|----------|-------------|
| 10001 | `NOT_FOUND` | Resource not found |
| 10002 | `STREAM_NOT_FOUND` | Stream not found |
| 10003 | `CONSUMER_NOT_FOUND` | Consumer not found |
| 10004 | `REQUEST_NOT_FOUND` | Request not found |
| 10005 | `STREAM_WRONG_LAST_SEQ` | Wrong last sequence |
| 10006 | `STREAM_NAME_EXISTS` | Stream already exists |
| 10007 | `CONSUMER_NAME_EXISTS` | Consumer already exists |
| 10008 | `INSUFFICIENT_RESOURCES` | Insufficient resources |
| 10009 | `NO_MESSAGE_FOUND` | No message found |
| 10013 | `CONSUMER_EXISTS` | Consumer already exists (duplicate) |
| 10014 | `STREAM_NOT_CONFIGURED` | Stream not configured |
| 10015 | `CLUSTER_NOT_ACTIVE` | Cluster not active |
| 10016 | `CLUSTER_NOT_LEADER` | Not the cluster leader |
| 10017 | `CLUSTER_NOT_ENOUGH_PEERS` | Not enough peers |
| 10018 | `CLUSTER_INCOMPLETE` | Cluster incomplete |
| 10019 | `CONSUMER_DELETED` | Consumer was deleted |
| 10020 | `CONSUMER_BAD_ACK` | Bad acknowledgment |
| 10021 | `CONSUMER_BAD_SUBJECT` | Bad consumer subject |
| 10022 | `CONSUMER_DELETED_DRIFT` | Consumer deleted due to drift |
| ... | ... | Additional codes |
## Account
**Location**: `jetstream/account.rs`
The `Account` struct provides information about the JetStream account:
```rust
pub struct Account {
pub memory: i64,
pub storage: i64,
pub streams: i64,
pub consumers: i64,
pub limits: AccountLimits,
}
```

292
docs/research/references/nats.rs/nats-server/07-authentication-and-security.md
View File

@@ -1,292 +0,0 @@
# Authentication and Security
This document covers the authentication mechanisms, TLS configuration, and security-related features of the async-nats client.
## Authentication Methods
The NATS server supports multiple authentication methods. The client implements all of them.
### 1. Username/Password
The simplest authentication method.
```rust
// Via ConnectOptions
let client = ConnectOptions::with_user_and_password("user".into(), "pass".into())
.connect("nats://localhost")
.await?;
// Via URL
let client = connect("nats://user:pass@localhost:4222").await?;
```
These credentials are sent in the `CONNECT` message as `user` and `pass` fields.
### 2. Token Authentication
A single token used for authentication.
```rust
let client = ConnectOptions::with_token("my-token".into())
.connect("nats://localhost")
.await?;
```
Token is sent in the `CONNECT` message as `auth_token` field.
### 3. NKey Authentication
NKey-based authentication using Ed25519 key pairs. Requires the `nkeys` feature.
```rust
let seed = "SUANQDPB2RUOE4ETUA26CNX7FUKE5ZZKFCQIIW63OX225F2CO7UEXTM7ZY";
let client = ConnectOptions::with_nkey(seed.into())
.connect("nats://localhost")
.await?;
```
Flow:
1. Server sends `INFO` with a `nonce` field
2. Client creates a `KeyPair` from the seed
3. Client signs the nonce: `key_pair.sign(nonce.as_bytes())`
4. Client sends `CONNECT` with `nkey` (public key) and `sig` (Base64URL-encoded signature)
5. Server verifies the signature against the public key and nonce
### 4. JWT Authentication
User JWT with a signing callback. Requires the `nkeys` feature.
```rust
let key_pair = Arc::new(nkeys::KeyPair::from_seed(seed)?);
let jwt = load_jwt().await?;
let client = ConnectOptions::with_jwt(jwt, move |nonce| {
let key_pair = key_pair.clone();
async move { key_pair.sign(&nonce).map_err(AuthError::new) }
})
.connect("nats://localhost")
.await?;
```
Flow:
1. Server sends `INFO` with a `nonce` field
2. Client sends `CONNECT` with `jwt` (user JWT) and `sig` (Base64URL-encoded nonce signature)
3. The signing callback is async, allowing integration with external signing services (e.g., HSM)
### 5. Credentials File
Combines JWT and NKey from a `.creds` file. Requires the `nkeys` feature.
```rust
// From file
let client = ConnectOptions::with_credentials_file("path/to/my.creds")
.await?
.connect("nats://localhost")
.await?;
// From string
let client = ConnectOptions::with_credentials(creds_string)
.connect("nats://localhost")
.await?;
```
Credentials file format:
```
-----BEGIN NATS USER JWT-----
eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5...
------END NATS USER JWT------
************************* IMPORTANT *************************
NKEY Seed printed below can be used sign and prove identity.
-----BEGIN USER NKEY SEED-----
SUAIO3FHUX5PNV2LQIIP7TZ3N4L7TX3W53MQGEIVYFIGA635OZCKEYHFLM
------END USER NKEY SEED------
```
**Location**: `auth_utils.rs` handles parsing:
- `load_creds(path)` — async file read + parse
- `parse_jwt_and_key_from_creds(creds)` — extracts JWT and KeyPair from the string
### 6. Auth Callback
A custom async callback that receives the server nonce and returns an `Auth` struct. This is the most flexible mechanism.
```rust
let client = ConnectOptions::with_auth_callback(move |nonce| {
async move {
let mut auth = Auth::new();
auth.username = Some("user".to_string());
auth.password = Some("pass".to_string());
// Can also set jwt, nkey, signature, token
Ok(auth)
}
})
.connect("nats://localhost")
.await?;
```
The callback is invoked on each connection/reconnection, allowing dynamic credential refresh (e.g., refreshing JWTs from an auth server).
### 7. URL-Embedded Credentials
```rust
// Username and password in URL
let client = connect("nats://user:pass@localhost:4222").await?;
// Token in URL (username field)
let client = connect("nats://token@localhost:4222").await?;
```
## Auth Struct
**Location**: `auth.rs`
The `Auth` struct is a container for all authentication methods. Multiple fields can be set simultaneously:
```rust
#[derive(Clone, Default)]
pub struct Auth {
pub jwt: Option<String>,
pub nkey: Option<String>,
pub signature_callback: Option<CallbackArg1<String, Result<String, AuthError>>>,
pub signature: Option<Vec<u8>>,
pub username: Option<String>,
pub password: Option<String>,
pub token: Option<String>,
}
```
Priority in `Connector::try_connect_to()`:
1. Auth callback overrides all other methods
2. NKey authentication (if `auth.nkey` is set)
3. JWT authentication (if `auth.jwt` is set)
4. Username/password/token from `Auth` struct
5. Username/password from URL
## TLS Configuration
### TLS Modes
| Mode | When | Description |
|------|------|-------------|
| None | Default | Plaintext connection |
| Standard | `tls_required` or server requires | TLS after INFO |
| TLS First | `tls_first` option | TLS before INFO |
| WebSocket | `wss://` URL | TLS handled by WebSocket library |
### TLS Setup
**Location**: `tls.rs`
The `config_tls()` function builds a `rustls::ClientConfig`:
1. Create `RootCertStore` and load native system certificates
2. Add custom root certificates from configured PEM files
3. Build `ClientConfig` with the chosen crypto provider:
- `ring` (default)
- `aws-lc-rs`
- `fips` (aws-lc-rs in FIPS mode)
4. If client certificate + key are configured, add them for mTLS
5. If a custom `rustls::ClientConfig` was provided, use it directly
### TLS First
```rust
let client = ConnectOptions::new()
.tls_first()
.connect("nats://localhost")
.await?;
```
This sets both `tls_first = true` and `tls_required = true`. The client performs TLS handshake before reading the `INFO` message. The server must have `handshake_first: true` in its configuration.
### Custom TLS Configuration
```rust
let tls_client = rustls::ClientConfig::builder()
.with_root_certificates(root_store)
.with_no_client_auth();
let client = ConnectOptions::new()
.require_tls(true)
.tls_client_config(tls_client)
.connect("nats://localhost")
.await?;
```
### mTLS (Mutual TLS)
```rust
let client = ConnectOptions::new()
.add_root_certificates("ca.pem".into())
.add_client_certificate("cert.pem".into(), "key.pem".into())
.connect("tls://localhost")
.await?;
```
## WebSocket Transport
Requires the `websockets` feature. Supports `ws://` and `wss://` schemes.
```rust
let client = connect("ws://localhost:8080").await?;
let client = connect("wss://localhost:443").await?;
```
Implementation uses `tokio-websockets` with a `WebSocketAdapter` that wraps the WebSocket stream to implement `AsyncRead + AsyncWrite`:
```rust
// WebSocketAdapter bridges WebSocket messages to byte streams
pub(crate) struct WebSocketAdapter<T> {
pub(crate) inner: WebSocketStream<T>,
pub(crate) read_buf: BytesMut, // Buffered incoming WebSocket messages
}
```
For `wss://`, TLS is configured within the WebSocket connector, not via the client's TLS layer.
## Security Considerations
### Nonce Signing
The server's `nonce` in the `INFO` message prevents replay attacks:
- Each connection gets a unique nonce
- The nonce must be signed with the client's private key
- The signature is verified server-side against the public key
### Authorization Violations
When the server sends `-ERR 'authorization violation'`:
- The client parses this as `ServerError::AuthorizationViolation`
- The `Connector` immediately propagates this error (does not retry)
- The error is converted to `ConnectErrorKind::AuthorizationViolation`
### Subject Validation
By default, the client validates subjects for protocol safety:
- **Publish subjects**: checked for emptiness and whitespace (can be disabled with `skip_subject_validation`)
- **Subscribe subjects**: always checked for emptiness, whitespace, leading/trailing dots, consecutive dots
- **Queue group names**: checked for emptiness and whitespace
The server enforces its own validation, but client-side checks prevent protocol-framing errors.
### Max Payload Size
The client checks payload size against the server's `max_payload` before publishing:
- For plain messages: `payload.len() > max_payload`
- For messages with headers: `headers.wire_len() + payload.len() > max_payload`
- Returns `PublishErrorKind::MaxPayloadExceeded` if exceeded
### No Echo
When `no_echo` is set, the `CONNECT` message includes `echo: false`. The server will not deliver messages published by this connection back to its own subscriptions. This prevents feedback loops.
### Lame Duck Mode
When the server enters lame duck mode (draining for shutdown):
1. Server sends `INFO` with `ldm: true`
2. Client emits `Event::LameDuckMode`
3. Application should gracefully close or reconnect to another server
The `nats-server` test harness provides `set_lame_duck_mode(server)` for testing this behavior.

347
docs/research/references/nats.rs/nats-server/08-test-harness.md
View File

@@ -1,347 +0,0 @@
# nats-server Test Harness
This document covers the `nats-server` crate — a test harness for spawning real NATS server instances in integration tests.
**Location**: `nats-server/src/lib.rs`
**Version**: 0.1.0
**License**: Apache-2.0
**Dependencies**: `lazy_static`, `regex`, `serde_json`, `nuid`, `rand`, `tokio-retry`
## What It Is
The `nats-server` crate is **not** a NATS server implementation. It is a thin test harness that:
- Spawns the Go-based `nats-server` binary as a child process
- Configures it for test use (dynamic ports, temp storage, log files)
- Discovers the client URL from the server's `INFO` protocol message
- Cleans up resources (JetStream storage, logs, PID files) on `Drop`
- Supports single servers and 3-node clusters
The actual NATS server must be installed separately (Go binary from `github.com/nats-io/nats-server`).
## Server Struct
```rust
pub struct Server {
inner: Inner,
}
struct Inner {
cfg: String, // Config file path
id: String, // Unique server ID (NUID)
port: Option<String>, // Explicit port (None = dynamic)
child: Child, // Child process handle
logfile: PathBuf, // Log file path in temp dir
pidfile: PathBuf, // PID file path in temp dir
}
```
## Public API
### run_server
```rust
pub fn run_server(cfg: &str) -> Server
```
Starts a single NATS server with optional config file.
- Uses dynamic port (`-1` flag) for parallel test execution
- Stores JetStream data in temp directory
- Writes logs to temp file: `nats-server-<id>.log`
- Writes PID to temp file: `nats-server-<id>.pid`
- If `cfg` is non-empty, passes `-c <cfg>` to the server
Example:
```rust
let server = nats_server::run_server("tests/configs/jetstream.conf");
let client = async_nats::connect(server.client_url()).await.unwrap();
```
### run_basic_server
```rust
pub fn run_basic_server() -> Server
```
Starts a server with no config (bare minimum). Equivalent to `run_server("")`.
### run_server_with_port
```rust
pub fn run_server_with_port(cfg: &str, port: Option<&str>) -> Server
```
Starts a server with an explicit port. If `None`, uses dynamic port.
### run_cluster
```rust
pub fn run_cluster<'a, C: IntoConfig<'a>>(cfg: C) -> Cluster
```
Starts a 3-node cluster with the given config.
- Allocates 3 random port ranges (base, base+100, base+200)
- Configures cluster routes between nodes
- Each node gets: `--cluster nats://127.0.0.1:<cluster_port>`, `--routes <other_routes>`, `--cluster_name cluster`, `-n nodeN`
- Waits 2 seconds for cluster formation and leader election
The `IntoConfig` trait allows passing either a single config string (applied to all 3 nodes) or an array of 3 configs (one per node):
```rust
// Same config for all nodes
let cluster = run_cluster("configs/jetstream.conf");
// Different configs per node
let cluster = run_cluster(["node1.conf", "node2.conf", "node3.conf"]);
```
### Cluster Struct
```rust
pub struct Cluster {
pub servers: Vec<Server>,
}
impl Cluster {
pub fn client_url(&self) -> String {
self.servers[0].client_url()
}
}
```
### Server Methods
```rust
impl Server {
pub fn restart(&mut self)
pub fn client_url(&self) -> String
pub fn client_port(&self) -> u16
pub fn client_url_with(&self, user: &str, pass: &str) -> String
pub fn client_url_with_token(&self, token: &str) -> String
pub fn client_pid(&self) -> usize
}
```
#### restart()
Kills the current server process, waits for it to exit, then restarts with the same config, port, and ID. Used for testing reconnection behavior.
#### client_url()
Connects to the server's TCP port, reads the `INFO` line, parses the JSON, and constructs a URL:
- `nats://localhost:<port>` for non-TLS
- `tls://localhost:<port>` for TLS-required servers
Polls the log file (up to 10 seconds) to discover the client address, since the port may be dynamically allocated.
#### client_pid()
Reads the PID file and returns the server process ID. Used for sending signals.
### set_lame_duck_mode
```rust
pub fn set_lame_duck_mode(s: &Server)
```
Sends the lame duck mode signal to the server:
```bash
nats-server --signal ldm=<pid>
```
### is_port_available
```rust
pub fn is_port_available(port: usize) -> bool
```
Tests if a TCP port is available by attempting to bind to it.
## Server Lifecycle
### Spawning
The `do_run` function constructs and spawns the server process:
```rust
fn do_run(cfg: &str, port: Option<&str>, id: Option<String>) -> Inner {
let id = id.unwrap_or_else(|| nuid::next().to_string());
let logfile = env::temp_dir().join(format!("nats-server-{id}.log"));
let pidfile = env::temp_dir().join(format!("nats-server-{id}.pid"));
let store_dir = env::temp_dir().join(format!("store-dir-{id}"));
let mut cmd = Command::new("nats-server");
cmd.arg("--store_dir").arg(store_dir.as_path())
.arg("-p");
match port {
Some(port) => cmd.arg(port),
None => cmd.arg("-1"), // Dynamic port
};
cmd.arg("-l").arg(logfile.as_os_str())
.arg("-P").arg(pidfile.as_os_str());
if !cfg.is_empty() {
cmd.arg("-c").arg(cfg);
}
let child = cmd.spawn().unwrap();
// ...
}
```
Key flags:
- `--store_dir` — JetStream storage directory in temp
- `-p -1` — Dynamic port allocation (or explicit port)
- `-l` — Log file path
- `-P` — PID file path
- `-c` — Config file path
### Cleanup (Drop)
```rust
impl Drop for Server {
fn drop(&mut self) {
self.inner.child.kill().unwrap();
self.inner.child.wait().unwrap();
if let Ok(log) = fs::read_to_string(self.inner.logfile.as_os_str()) {
// Clean up JetStream storage directory if found in log
if let Some(caps) = SD_RE.captures(&log) {
let sd = caps.get(1).map_or("", |m| m.as_str());
fs::remove_dir_all(sd).ok();
}
// Remove log file
fs::remove_file(self.inner.logfile.as_os_str()).ok();
}
}
}
```
The regex `SD_RE` matches the "Store Directory" line in the server log:
```
.+\sStore Directory:\s+"([^"]+)"
```
### Client URL Discovery
The `client_addr` method polls the log file to find the server's listen address:
```rust
fn client_addr(&self) -> String {
for _ in 0..100 { // 100 iterations × 500ms = 50s max
match fs::read_to_string(self.inner.logfile.as_os_str()) {
Ok(l) => {
if let Some(cre) = CLIENT_RE.captures(&l) {
return cre.get(1).unwrap().as_str()
.replace("0.0.0.0", "localhost");
} else {
thread::sleep(Duration::from_millis(500));
}
}
_ => thread::sleep(Duration::from_millis(500)),
}
}
panic!("no client addr info");
}
```
The regex `CLIENT_RE` matches:
```
.+\sclient connections on\s+(\S+)
```
After finding the address, `client_url()` connects to it and parses the `INFO` JSON to get the port and TLS requirements.
## Cluster Setup
The `run_cluster_node_with_port` function spawns a single cluster node:
```rust
fn run_cluster_node_with_port(
cfg: &str,
port: Option<&str>,
routes: Vec<usize>,
name: String,
cluster_name: String,
cluster: usize,
) -> Server
```
Additional flags for cluster nodes:
- `--routes nats://127.0.0.1:<port1>,nats://127.0.0.1:<port2>` — routes to other cluster members
- `--cluster nats://127.0.0.1:<cluster_port>` — cluster listen address
- `--cluster_name <name>` — cluster name for grouping
- `-n <name>` — server name
Port allocation for a cluster:
```
Base port: random in 3000..50000
Node 1: client_port=base, cluster_port=base+1
Node 2: client_port=base+100, cluster_port=base+101
Node 3: client_port=base+200, cluster_port=base+201
```
Each port is checked for availability with `is_port_available()`, including the +1 cluster port.
## JetStream Config
**Location**: `configs/jetstream.conf`
```conf
jetstream: {
strict: true,
max_mem_store: 8MiB,
max_file_store: 10GiB
}
```
This is the default test config for JetStream-enabled servers. It enables strict mode and sets memory/file storage limits suitable for testing.
## Test Usage Patterns
```rust
#[tokio::test]
async fn basic_test() {
let server = nats_server::run_server("configs/jetstream.conf");
let client = async_nats::connect(server.client_url()).await.unwrap();
// ... test logic ...
// Server cleaned up on drop
}
#[tokio::test]
async fn cluster_test() {
let cluster = nats_server::run_cluster("configs/jetstream.conf");
let client = async_nats::connect(cluster.client_url()).await.unwrap();
// ... test logic ...
}
#[tokio::test]
async fn reconnect_test() {
let mut server = nats_server::run_server("");
let client = async_nats::connect(server.client_url()).await.unwrap();
// Restart the server to test reconnection
server.restart();
// Client should reconnect automatically
client.publish("test", "data".into()).await.unwrap();
}
```
## Dependencies
| Dependency | Version | Purpose |
|-----------|---------|---------|
| `lazy_static` | 1.4.0 | Static regex initialization |
| `regex` | 1.7.1 | Log parsing (store directory, client address) |
| `url` | 2 | URL manipulation for client_url_with |
| `serde_json` | 1.0.104 | INFO JSON parsing |
| `nuid` | 0.5 | Unique server ID generation |
| `rand` | 0.10.1 | Random port selection |
| `tokio-retry` | 0.3.0 | Exponential backoff for cluster operations |
Note: `async-nats` is only a dev-dependency, used in the crate's own integration tests.

307
docs/research/references/nats.rs/nats-server/09-service-api-and-abstractions.md
View File

@@ -1,307 +0,0 @@
# Service API and Higher-Level Abstractions
This document covers the Service API and other higher-level abstractions built on top of the core NATS client.
## Service API
**Location**: `service/` (feature: `service`)
The Service API provides a framework for building NATS-based microservices with built-in monitoring, health checks, and statistics.
### Service
```rust
#[derive(Debug)]
pub struct Service {
client: Client,
info: Info,
endpoints: HashMap<String, Endpoint>,
started: DateTime,
stats_handler: Arc<dyn Fn(&str, &Stats) -> serde_json::Value + Send + Sync>,
stop_sender: mpsc::Sender<()>,
stop_receiver: Option<mpsc::Receiver<()>>,
}
```
### Creating a Service
```rust
use async_nats::service::ServiceExt;
let mut service = client
.service_builder()
.description("Product service")
.stats_handler(|endpoint, stats| {
serde_json::json!({
"endpoint": endpoint,
"requests": stats.num_requests,
"errors": stats.num_errors,
})
})
.start("products", "1.0.0")
.await?;
```
### ServiceBuilder
```rust
impl ServiceBuilder {
pub fn description(mut self, description: impl Into<String>) -> Self
pub fn stats_handler<F>(mut self, handler: F) -> Self
pub async fn start(self, name: impl Into<String>, version: impl Into<String>) -> Result<Service, ServiceError>
}
```
### Endpoints
A service exposes one or more endpoints, each handling requests on a specific subject:
```rust
// Add an endpoint
let mut endpoint = service
.endpoint("get_product")
.await?;
// Process requests
while let Some(request) = endpoint.next().await {
let request = request?;
// Handle the request
request.respond(serde_json::json!({ "id": 1, "name": "Widget" })).await?;
}
```
### Endpoint
**Location**: `service/endpoint.rs`
```rust
pub struct Endpoint {
subject: Subject,
queue_group: Option<String>,
info: EndpointInfo,
stats: Stats,
subscriber: Subscriber,
}
```
Implements `futures::Stream` yielding `ServiceRequest` objects.
### ServiceRequest
```rust
pub struct ServiceRequest {
pub subject: Subject,
pub payload: Bytes,
pub headers: Option<HeaderMap>,
pub reply: Option<Subject>,
pub client: Client,
}
```
Methods:
- `respond(payload)` — send a response to the requester
- `respond_with_headers(payload, headers)` — send a response with headers
### Monitoring Subjects
The Service API automatically creates monitoring endpoints:
| Subject | Description |
|---------|-------------|
| `$SRV.PING` | Ping all services (returns service info) |
| `$SRV.PING.<name>` | Ping specific service by name |
| `$SRV.PING.<name>.<id>` | Ping specific service instance |
| `$SRV.INFO` | Get service info |
| `$SRV.STATS` | Get service statistics |
### Service Info
```rust
pub struct Info {
pub name: String,
pub id: String,
pub version: String,
pub description: String,
pub endpoints: Vec<EndpointInfo>,
}
```
### Stats
```rust
pub struct Stats {
pub num_requests: u64,
pub num_errors: u64,
pub last_error: Option<String>,
pub processing_time: Duration,
pub average_processing_time: Duration,
}
```
## ID Generation
**Location**: `id_generator.rs`
The client needs unique IDs for inbox subjects and other purposes.
### With `nuid` Feature (Default)
Uses the NUID library for high-performance, cryptographically strong, collision-resistant IDs:
```rust
pub(crate) fn next() -> String {
nuid::next().to_string()
}
```
NUID generates 22-character alphanumeric strings using a combination of a random prefix and a sequential counter.
### Without `nuid` Feature
Falls back to `rand`-based generation:
```rust
pub(crate) fn next() -> String {
rng()
.sample_iter(Alphanumeric)
.take(22)
.map(char::from)
.collect()
}
```
Both approaches produce 22-character alphanumeric strings, but NUID is more performant and has better collision resistance.
## Inbox Generation
The `Client::new_inbox()` method generates globally unique inbox subjects for request-reply:
```rust
pub fn new_inbox(&self) -> String {
format!("{}.{}", self.inbox_prefix, crate::id_generator::next())
}
```
Default prefix is `_INBOX`, producing subjects like `_INBOX.UaBG3f3q5NxX3KdNcRmF2f`.
Custom prefix via `ConnectOptions::custom_inbox_prefix()`:
```rust
let client = ConnectOptions::new()
.custom_inbox_prefix("MYAPP")
.connect("demo.nats.io")
.await?;
// Inbox subjects: MYAPP.UaBG3f3q5KdNcRmF2f
```
## DateTime Helpers
**Location**: `datetime.rs` (feature: `jetstream` or `service` or `chrono`)
Provides date/time types for JetStream and Service API timestamps:
- Uses the `time` crate by default
- Optionally uses `chrono` via the `chrono` feature flag
- Supports RFC 3339 formatting and parsing
- `DateTime` type wraps either `time::OffsetDateTime` or `chrono::DateTime<Utc>`
## Crypto Module
**Location**: `crypto.rs` (feature: `crypto`)
Provides encryption/decryption support used by the Object Store for server-side encryption.
## Subject Validation
**Location**: `lib.rs`
The client provides two levels of subject validation:
### is_valid_publish_subject
```rust
pub(crate) fn is_valid_publish_subject<T: AsRef<str>>(subject: T) -> bool
```
Checks for protocol safety only:
- Not empty
- No whitespace (space, tab, CR, LF) which would break protocol framing
Used for publish operations. Can be disabled with `skip_subject_validation`.
### is_valid_subject
```rust
pub(crate) fn is_valid_subject<T: AsRef<str>>(subject: T) -> bool
```
Checks structural validity:
- Not empty
- No leading/trailing dots
- No consecutive dots (`..`)
- No whitespace
Used for subscribe operations (always runs, matching Go/Java behavior).
### is_valid_queue_group
```rust
pub(crate) fn is_valid_queue_group(queue_group: &str) -> bool
```
Checks:
- Not empty
- No whitespace
## JetStream Name Validation
**Location**: `jetstream/mod.rs`
```rust
pub(crate) fn is_valid_name(name: &str) -> bool {
!name.is_empty()
&& name.bytes().all(|c| !c.is_ascii_whitespace() && c != b'.' && c != b'*' && c != b'>')
}
```
JetStream names (stream names, consumer names) must not contain:
- Whitespace
- Dots (`.`) — would conflict with subject delimiters
- Wildcards (`*`, `>`) — would conflict with subject wildcards
## CallbackArg1
**Location**: `options.rs`
A type-erased async callback wrapper used throughout the crate:
```rust
pub(crate) type AsyncCallbackArg1<A, T> =
Arc<dyn Fn(A) -> Pin<Box<dyn Future<Output = T> + Send + Sync + 'static>> + Send + Sync>;
#[derive(Clone)]
pub(crate) struct CallbackArg1<A, T>(AsyncCallbackArg1<A, T>);
impl<A, T> CallbackArg1<A, T> {
pub(crate) async fn call(&self, arg: A) -> T {
(self.0.as_ref())(arg).await
}
}
```
Used for:
- `event_callback` — `CallbackArg1<Event, ()>`
- `auth_callback` — `CallbackArg1<Vec<u8>, Result<Auth, AuthError>>`
- `reconnect_to_server_callback` — `CallbackArg1<(Vec<Server>, ServerInfo), Option<ReconnectToServer>>`
- `signature_callback` — `CallbackArg1<String, Result<String, AuthError>>`
## Version Compatibility Checking
The `Client::is_server_compatible` method checks if the server version meets a minimum requirement:
```rust
pub fn is_server_compatible(&self, major: i64, minor: i64, patch: i64) -> bool
```
This parses the server version string from `ServerInfo::version` using a regex and compares major/minor/patch components. Note: this checks the directly-connected server, not necessarily the JetStream leader.
The `server_2_10`, `server_2_11`, `server_2_12`, and `server_2_14` feature flags enable version-specific API fields and methods without runtime checks.

215
docs/research/references/nats.rs/nats-server/10-quick-reference.md
View File

@@ -1,215 +0,0 @@
# Quick Reference
## Crate Summary
| | |
|---|---|
| **Crate** | `async-nats` |
| **Version** | 0.49.1 |
| **Edition** | 2021 |
| **MSRV** | 1.88.0 |
| **License** | Apache-2.0 |
| **Runtime** | Tokio |
| **Protocol** | NATS Client Protocol v1 (Dynamic) |
| **TLS** | rustls (ring / aws-lc-rs / fips) |
| **WebSocket** | tokio-websockets (feature-gated) |
## Quick Start
```rust
use async_nats::connect;
use futures_util::StreamExt;
#[tokio::main]
async fn main() -> Result<(), async_nats::Error> {
let client = connect("demo.nats.io").await?;
// Publish
client.publish("events.data", "hello".into()).await?;
// Subscribe
let mut sub = client.subscribe("events.>").await?;
while let Some(msg) = sub.next().await {
println!("{:?}", msg);
}
// Request-Response
let response = client.request("service", "input".into()).await?;
Ok(())
}
```
## Architecture at a Glance
```
Client (cloneable handle, mpsc::Sender<Command>)
│
▼
ConnectionHandler (single Tokio task)
├── Subscriptions HashMap<u64, Subscription>
├── Multiplexer (request-reply, SID 0)
├── Flush Observers
└── Ping/Pong health check
│
▼
Connection (protocol I/O, read/write buffers)
│
▼
Connector (server pool, reconnection)
│
▼
NATS Server (Go binary, TCP/TLS/WebSocket)
```
## Key Types
| Type | Location | Purpose |
|------|----------|---------|
| `Client` | `client.rs` | Cloneable connection handle |
| `Subscriber` | `lib.rs` | Message stream (impl `futures::Stream`) |
| `Message` | `message.rs` | Inbound NATS message |
| `OutboundMessage` | `message.rs` | Outbound publish message |
| `Subject` | `subject.rs` | Validated subject string (backed by `Bytes`) |
| `HeaderMap` | `header.rs` | NATS message headers |
| `StatusCode` | `status.rs` | NATS protocol status codes |
| `ServerInfo` | `lib.rs` | Server INFO data |
| `ConnectInfo` | `lib.rs` | Client CONNECT data |
| `ServerAddr` | `lib.rs` | Validated server URL |
| `Auth` | `auth.rs` | Authentication credentials |
| `ConnectOptions` | `options.rs` | Connection configuration builder |
| `Event` | `lib.rs` | Connection lifecycle events |
| `State` | `connection.rs` | Connection state (Pending/Connected/Disconnected) |
| `Statistics` | `client.rs` | Atomic connection metrics |
| `Request` | `client.rs` | Request-response builder |
## JetStream Types
| Type | Location | Purpose |
|------|----------|---------|
| `jetstream::Context` | `jetstream/context.rs` | JetStream API entry point |
| `jetstream::stream::Stream` | `jetstream/stream.rs` | Stream management |
| `jetstream::stream::Config` | `jetstream/stream.rs` | Stream configuration |
| `jetstream::stream::Info` | `jetstream/stream.rs` | Stream info/state |
| `jetstream::consumer::PullConsumer` | `jetstream/consumer/pull.rs` | Pull-based consumer |
| `jetstream::consumer::PushConsumer` | `jetstream/consumer/push.rs` | Push-based consumer |
| `jetstream::consumer::Config` | `jetstream/consumer/mod.rs` | Consumer configuration |
| `jetstream::Message` | `jetstream/message.rs` | Message with ack methods |
| `jetstream::PublishAck` | `jetstream/publish.rs` | Publish acknowledgment |
| `jetstream::kv::Store` | `jetstream/kv/bucket.rs` | Key-Value store |
| `jetstream::object_store::ObjectStore` | `jetstream/object_store/mod.rs` | Object store |
| `jetstream::ErrorCode` | `jetstream/errors.rs` | JetStream error codes |
## Protocol Operations
### Client → Server (ClientOp)
| Op | Wire Format | Purpose |
|----|-----------|---------|
| `CONNECT` | `CONNECT {json}\r\n` | Authentication and capabilities |
| `PUB` | `PUB <subject> [reply] <len>\r\n<payload>\r\n` | Publish message |
| `HPUB` | `HPUB <subject> [reply] <hlen> <tlen>\r\n<hdrs><payload>\r\n` | Publish with headers |
| `SUB` | `SUB <subject> [queue] <sid>\r\n` | Subscribe |
| `UNSUB` | `UNSUB <sid> [max]\r\n` | Unsubscribe |
| `PING` | `PING\r\n` | Keepalive / health check |
| `PONG` | `PONG\r\n` | Response to server PING |
### Server → Client (ServerOp)
| Op | Wire Format | Purpose |
|----|-----------|---------|
| `INFO` | `INFO {json}\r\n` | Server capabilities, cluster info |
| `MSG` | `MSG <subj> <sid> [reply] <len>\r\n<payload>\r\n` | Deliver message |
| `HMSG` | `HMSG <subj> <sid> [reply] <hlen> <tlen>\r\n<hdrs><payload>\r\n` | Message with headers |
| `+OK` | `+OK\r\n` | Success (verbose mode) |
| `-ERR` | `-ERR <desc>\r\n` | Server error |
| `PING` | `PING\r\n` | Server health check |
| `PONG` | `PONG\r\n` | Ack client PING |
## Internal Commands (Command → ConnectionHandler)
| Command | Purpose |
|---------|---------|
| `Publish(OutboundMessage)` | Queue message for sending |
| `Request { subject, payload, respond, headers, sender }` | Request-response via multiplexer |
| `Subscribe { sid, subject, queue_group, sender }` | Create subscription |
| `Unsubscribe { sid, max }` | Remove subscription |
| `Flush { observer }` | Wait for write buffer flush |
| `Drain { sid }` | Gracefully drain (sub or whole client) |
| `Reconnect` | Force reconnection |
| `SetServerPool { servers, result }` | Replace server pool |
| `ServerPool { result }` | Query server pool |
## Feature Flags
| Feature | Default | Enables |
|---------|---------|---------|
| `jetstream` | ✓ | JetStream API (streams, consumers, publish) |
| `kv` | ✓ | Key-Value store (requires jetstream) |
| `object-store` | ✓ | Object store (requires jetstream + crypto) |
| `service` | ✓ | Service API |
| `nkeys` | ✓ | NKey/JWT authentication |
| `crypto` | ✓ | Encryption support |
| `websockets` | ✓ | WebSocket transport |
| `nuid` | ✓ | NUID ID generation |
| `ring` | ✓ | Ring crypto backend |
| `aws-lc-rs` | ✗ | Alternative crypto backend |
| `fips` | ✗ | FIPS mode (requires aws-lc-rs) |
| `chrono` | ✗ | Use chrono instead of time |
| `experimental` | ✗ | Experimental features |
| `server_2_10` | ✓ | Server 2.10+ API fields |
| `server_2_11` | ✓ | Server 2.11+ API fields |
| `server_2_12` | ✓ | Server 2.12+ API fields |
| `server_2_14` | ✓ | Server 2.14+ API fields |
## Connection Defaults
| Parameter | Default |
|-----------|---------|
| Connection timeout | 5 seconds |
| Ping interval | 60 seconds |
| Max pending pings | 2 |
| Request timeout | 10 seconds |
| Command channel capacity | 2048 |
| Subscription capacity | 65536 |
| Read buffer capacity | 65535 |
| Inbox prefix | `_INBOX` |
| Reconnect delay | Exponential (0ms → 4s cap) |
| Max reconnects | Unlimited |
| TLS required | Auto (server-dependent) |
## Error Hierarchy
```
ConnectError (ConnectErrorKind::ServerParse | Dns | Authentication | AuthorizationViolation | TimedOut | Tls | Io | MaxReconnects)
PublishError (PublishErrorKind::MaxPayloadExceeded | InvalidSubject | Send)
RequestError (RequestErrorKind::TimedOut | NoResponders | InvalidSubject | MaxPayloadExceeded | Other)
SubscribeError (SubscribeErrorKind::InvalidSubject | InvalidQueueName | Other)
FlushError (FlushErrorKind::SendError | FlushError)
```
## nats-server Test Harness
| Function | Description |
|----------|-------------|
| `run_server(cfg)` | Start single server with config |
| `run_basic_server()` | Start bare server |
| `run_cluster(cfg)` | Start 3-node cluster |
| `set_lame_duck_mode(s)` | Send LDM signal |
## JetStream API Subjects
| Operation | Subject Pattern |
|-----------|---------------|
| Create stream | `$JS.API.STREAM.CREATE.<name>` |
| Stream info | `$JS.API.STREAM.INFO.<name>` |
| Update stream | `$JS.API.STREAM.UPDATE.<name>` |
| Delete stream | `$JS.API.STREAM.DELETE.<name>` |
| Purge stream | `$JS.API.STREAM.PURGE.<name>` |
| List streams | `$JS.API.STREAM.LIST` |
| Create consumer | `$JS.API.CONSUMER.CREATE.<stream>` |
| Create durable | `$JS.API.CONSUMER.DURABLE.CREATE.<stream>.<name>` |
| Consumer info | `$JS.API.CONSUMER.INFO.<stream>.<name>` |
| Pull next | `$JS.API.CONSUMER.MSG.NEXT.<stream>.<name>` |
| Account info | `$JS.API.ACCOUNT.INFO` |
| Direct get | `$JS.API.DIRECT.GET.<name>` |

963
docs/research/references/openstack-keystone/keystone-reference.md
View File

@@ -1,963 +0,0 @@
# OpenStack Keystone Identity Service — Reference Document
> Status: Research reference
> Created: 2026-06-08
> Context: alknet auth/identity system design; rustfs S3-compatible store with Keystone auth
## 1. Overview
OpenStack Keystone is the identity service for the OpenStack cloud platform. It
provides authentication, authorization, and service discovery via a RESTful HTTP
API. Every other OpenStack service (Nova, Neutron, Cinder, Swift, etc.) depends
on Keystone for token validation and access control.
Key responsibilities:
| Responsibility | Description |
|---|---|
| **Authentication** | Verify identity via passwords, tokens, TOTP, SAML, OIDC, application credentials |
| **Authorization** | Role-based access control (RBAC) across projects, domains, and system scope |
| **Service Catalog** | Registry of available services and their endpoint URLs |
| **Token Management** | Issue, validate, and revoke bearer tokens with scoped authorization |
| **Federation** | Accept identity assertions from external IdPs (SAML, OIDC) |
| **Trust Delegation** | Allow users to delegate limited authority to other users |
---
## 2. Core Concepts
### 2.1 Domains
A **domain** is a top-level namespace that contains users, groups, and projects.
Domains provide administrative isolation: a domain administrator can manage
users and projects within their domain but not across domains.
- Domains were introduced in the Identity API v3 (the "v3" API).
- Before domains, OpenStack used "tenants" (v2 API) — projects are the v3
equivalent, but domains add a containment boundary.
- Every user, group, and project belongs to exactly one domain.
- The `Default` domain is created automatically and holds all v2-compatible
resources.
**Key property**: Domains are the unit of administrative delegation. A domain
admin can create/delete users, groups, and projects within their domain.
### 2.2 Projects
A **project** is a container for resources — compute instances, storage volumes,
networks, etc. Projects are the primary scope for authorization in OpenStack.
- Projects group resources: "who can see/use these VMs and volumes?"
- Projects belong to a domain.
- Projects are the primary unit for role assignment and token scoping.
- Projects can be hierarchical (parent/child) with inherited role assignments.
**Key property**: A project-scoped token lets you operate on resources within
that project. You cannot use a project-scoped token to access resources in a
different project.
### 2.3 Users
A **user** represents a digital identity — a person, system account, or service
account that can authenticate and be authorized.
- Users belong to a domain.
- Users can have multiple authentication methods (password, TOTP, application
credentials, federated identity).
- Users can be members of groups.
- Users receive role assignments on projects, domains, or system scope.
### 2.4 Groups
A **group** is a named collection of users. Groups simplify role management: you
assign a role to a group on a project, and every user in the group inherits that
role.
- Groups belong to a domain.
- Groups are used for role assignment: `group:X → role:member → project:Y`.
- Federation mappings often resolve external IdP groups to local Keystone groups.
### 2.5 Roles
A **role** is a named permission set. Roles by themselves don't define what
operations are allowed — they are labels that policy files map to API operations.
- Roles are assigned by binding an actor (user or group) to a target (project,
domain, or system) with a role.
- Assignment format: `{actor, role, target}` — e.g., `{user:alice, member,
project:engineering}`.
- OpenStack defines default roles: `admin`, `member`, `reader`.
- Custom roles can be created. Policy files (policy.yaml) map roles to API
operations.
- **Implied roles**: one role can imply another (e.g., `admin` implies `member`
implies `reader`).
- **Inherited roles**: a role assigned on a domain with `inherited_to_projects`
flag propagates to all projects within that domain.
### 2.6 Endpoints
An **endpoint** is a network-accessible URL for an OpenStack service. Each
service registers one or more endpoints in Keystone's service catalog.
- Endpoints have an **interface** type:
- `public` — for end users (public network)
- `internal` — for service-to-service communication (internal network)
- `admin` — for administrative operations (restricted network)
- Endpoints have a **region** attribute for multi-region deployments.
- Endpoint URLs can contain template variables like `$(project_id)s` that are
resolved at token time.
### 2.7 Service Catalog
The **service catalog** is a registry of all services available in the
deployment and their endpoints. It is included in token responses and is
available via `GET /v3/auth/catalog`.
- A service has a `type` (e.g., `identity`, `compute`, `object-store`) and a
`name` (e.g., `keystone`, `nova`, `swift`).
- The `type` follows the [service-types authority][] — it identifies the API
contract, not the implementation version.
- The service catalog in a token is filtered by scope: a project-scoped token
shows only endpoints relevant to that project.
- Endpoint filtering allows administrators to restrict which endpoints are
visible to specific projects via project-endpoint associations or endpoint
groups.
[service-types authority]: https://service-types.openstack.org/
**Example service catalog entry:**
```json
{
"catalog": [
{
"name": "Keystone",
"type": "identity",
"endpoints": [
{
"interface": "public",
"url": "https://identity.example.com:5000/"
},
{
"interface": "internal",
"url": "https://identity.internal:5000/"
},
{
"interface": "admin",
"url": "https://identity.admin:5000/"
}
]
}
]
}
```
---
## 3. Token Lifecycle
### 3.1 Token Types by Scope
| Token Type | Scope | Contains | Use Case |
|---|---|---|---|
| **Unscoped** | None | User identity only, no roles, no catalog | Prove identity for subsequent scoped auth |
| **Project-scoped** | Project | Roles, catalog, project info | Operate on project resources (VMs, volumes) |
| **Domain-scoped** | Domain | Roles, catalog, domain info | Manage users/projects within a domain |
| **System-scoped** | System | Roles, catalog, system info | Cloud-wide admin operations |
| **Trust-scoped** | Trust | Delegated roles, trust metadata | Act on behalf of another user |
### 3.2 Authentication Flow
```
1. Client → POST /v3/auth/tokens (with credentials)
2. Keystone validates credentials
3. Keystone issues token:
- Token ID returned in X-Subject-Token header
- Token body (JSON) returned in response body
4. Client uses token: X-Auth-Token: <token_id> on subsequent requests
5. Services validate token:
- Option A: Local validation (Fernet/JWS — self-contained)
- Option B: Call Keystone to validate (UUID tokens)
```
### 3.3 Token Providers
| Provider | Format | Persistence | Size | Security |
|---|---|---|---|---|
| **Fernet** (default) | AES256-encrypted ciphertext + SHA256 HMAC | None (self-contained) | ~200 bytes | Symmetric keys; only Keystone can decrypt |
| **JWS** | JSON Web Signature (ES256) | None (self-contained) | ~800 bytes | Asymmetric keys; anyone can verify signature, payload is readable |
| **UUID** (legacy) | Random UUID string | Database (must be stored) | ~32 bytes | Requires database lookup for validation |
**Fernet tokens** are the recommended default. They are:
- Self-contained: no database persistence needed.
- Encrypted: the token payload is opaque to clients.
- Compact: much smaller than JWS tokens.
- Key rotation: Fernet keys are rotated using `keystone-manage fernet_rotate`.
**JWS tokens** are appropriate when:
- You want asymmetric key verification (services can validate without sharing
symmetric keys).
- You're comfortable with the payload being readable by anyone who has the token.
### 3.4 Token Contents
A project-scoped token contains:
```json
{
"token": {
"methods": ["password"],
"user": {
"id": "aaa...",
"name": "alice",
"domain": { "id": "default", "name": "Default" }
},
"project": {
"id": "bbb...",
"name": "engineering",
"domain": { "id": "default", "name": "Default" }
},
"roles": [
{ "id": "ccc...", "name": "member" },
{ "id": "ddd...", "name": "reader" }
],
"catalog": [ ... ],
"expires_at": "2026-06-08T12:00:00.000000Z",
"issued_at": "2026-06-08T11:00:00.000000Z",
"audit_ids": ["eeee..."],
"is_domain": false
}
}
```
Key fields:
- `methods`: Authentication methods used (e.g., `["password"]` or
`["password", "totp"]` for MFA).
- `user`: Who the token belongs to.
- `project` / `domain` / `system`: The authorization scope.
- `roles`: The roles assigned to the user within the scope.
- `catalog`: Service catalog (absent in unscoped tokens).
- `expires_at` / `issued_at`: Token validity window.
- `audit_ids`: Chain of audit IDs for tracking token derivation.
### 3.5 Token Validation
When a service receives a request with a token:
1. Extract `X-Auth-Token` header.
2. For Fernet tokens: decrypt with local Fernet key, parse payload, verify
expiration. Check revocation events.
3. For JWS tokens: verify signature with public key, parse payload, verify
expiration. Check revocation events.
4. For UUID tokens: call Keystone to validate. (Deprecated, but still supported.)
Keystone middleware (`keystonemiddleware`) handles this automatically for
OpenStack services.
### 3.6 Token Revocation
Tokens can be revoked explicitly (`DELETE /v3/auth/tokens`) or implicitly via
revocation events triggered by:
- User account disabled
- Domain disabled
- Project disabled
- Password changed (invalidates all tokens for that user)
- Role assignment changed (invalidates tokens for the affected scope)
Revocation events use pattern matching for efficiency — a single event can
invalidate many tokens (e.g., all tokens for a user, or all tokens for a project).
---
## 4. Scoping
### 4.1 Unscoped → Scoped Flow
The typical authentication flow is two-step:
1. **Authenticate** → receive an **unscoped token** (proves identity, no
authorization).
2. **Re-authenticate with scope** → receive a **scoped token** (proves identity
+ authorization).
```bash
# Step 1: Get unscoped token
curl -X POST /v3/auth/tokens -d '{
"auth": {
"identity": {
"methods": ["password"],
"password": { "user": { "name": "alice", "password": "..." } }
}
}
}'
# Step 2: Get project-scoped token using unscoped token
curl -X POST /v3/auth/tokens -d '{
"auth": {
"identity": {
"methods": ["token"],
"token": { "id": "<unscoped_token>" }
},
"scope": {
"project": { "name": "engineering", "domain": { "name": "Default" } }
}
}
}'
```
### 4.2 Scope Types and Authorization
| Scope | Token Can Do | Token Cannot Do |
|---|---|---|
| **Project** | Operate on project resources (VMs, storage, networks) | Manage domain users, system-wide operations |
| **Domain** | Manage users/projects within that domain | Operate on project resources (without project scope) |
| **System** | Cloud-wide admin: manage endpoints, services, hypervisor info | Project-specific resource operations |
| **None (unscoped)** | Prove identity to Keystone | Access any service resources |
A project-scoped token **cannot** be reused in a different project. Each scope
is a separate token. This is a deliberate security design: token scope limits
the blast radius of a compromised token.
### 4.3 Design Rationale
The scoping model exists because:
1. **Principle of least privilege**: Users authenticate once (expensive), then
get narrowly scoped tokens (cheap) for each operation context.
2. **Multi-tenancy**: A cloud serves many organizations; project scoping
prevents cross-tenant access.
3. **Administrative separation**: Domain admins manage users; system admins
manage infrastructure. Different scopes for different jobs.
---
## 5. Role-Based Access Control (RBAC)
### 5.1 Role Assignments
A role assignment binds an **actor** (user or group) to a **role** on a
**target** (project, domain, or system).
The four assignment types:
| Assignment | Actor | Target | Example |
|---|---|---|---|
| User → Project | User | Project | Alice is `member` of `engineering` |
| Group → Project | Group | Project | `dev-team` group is `member` of `engineering` |
| User → Domain | User | Domain | Alice is `admin` of `acme-domain` |
| Group → Domain | Group | Domain | `ops-team` group is `admin` of `acme-domain` |
Plus **system** role assignments for cloud-wide operations.
### 5.2 Effective Role Assignments
When querying role assignments with `effective=True`, Keystone resolves:
1. **Direct assignments**: Roles explicitly granted.
2. **Group memberships**: Roles inherited from groups the user belongs to.
3. **Inherited roles**: Roles from parent projects or domains (via
`inherited_to_projects` flag).
4. **Implied roles**: Roles implied by other roles (e.g., `admin` → `member`
→ `reader`).
### 5.3 Policy Enforcement
Keystone uses `oslo.policy` for policy enforcement. Each OpenStack service
defines policy rules in `policy.yaml` files. A rule maps an API operation to a
check string:
```yaml
"identity:create_project": "role:admin and domain_id:%(target.domain.id)s"
"identity:list_projects": "role:reader"
"identity:update_project": "role:admin or project_id:%(target.project.id)s"
```
Policy rules can check:
- Role membership (`role:admin`)
- Scope type (`system_scope:all`, `domain_id:...`)
- Resource ownership (`user_id:%(target.user.id)s`)
- Arbitrary target attributes
### 5.4 Scope Enforcement in Policy
Since the Rocky release, policies can require specific token scopes:
```yaml
# System-scoped token required
"identity:list_projects": "role:reader and system_scope:all"
# Project-scoped token required
"nova:create_server": "role:member and project_id:%(target.project.id)s"
```
This prevents:
- Using a project-scoped token for system operations.
- Using a system-scoped token for project operations (without a project context).
---
## 6. Trust Delegation (OS-TRUST)
### 6.1 Overview
Trusts allow one user (**trustor**) to delegate a subset of their authority to
another user (**trustee**) for a limited scope and duration, without sharing
credentials.
**Key properties of a trust:**
| Property | Description |
|---|---|
| `trustor_user_id` | User creating the trust (delegating authority) |
| `trustee_user_id` | User receiving the delegation |
| `project_id` | Project scope for the delegated authority |
| `roles` | Subset of trustor's roles being delegated |
| `impersonation` | If `true`, tokens appear to come from the trustor |
| `expires_at` | Optional expiration timestamp |
| `remaining_uses` | Optional limit on how many tokens can be created from this trust |
| `allow_redelegation` | Whether the trustee can create sub-trusts |
| `redelegation_count` | Maximum depth of redelegation chain |
### 6.2 Trust-Scoped Tokens
When a trustee authenticates using a trust:
1. The trustee authenticates with their own credentials.
2. They specify `trust_id` in the auth request.
3. Keystone issues a **trust-scoped token** with:
- Roles: the intersection of the trust's roles and the trustor's current
roles (if trustor lost a role, the trust is invalidated).
- `OS-TRUST:trust` section in the token body containing trust metadata.
If `impersonation=true`, the token's `user` field shows the trustor — the
trustee acts as the trustor. If `impersonation=false`, the token's `user`
field shows the trustee.
### 6.3 Trust Delegation Chains
Trusts support **redelegation**: a trustee can create a new trust delegating to
a third party. This creates a trust chain:
```
Trustor → Trust(A) → Trustee1
Trustee1 → Trust(B) → Trustee2 (redelegation)
```
Delegation depth is controlled by:
- `allow_redelegation: true/false`
- `redelegation_count: N` (decremented on each redelegation; default max is 3)
**Security constraints:**
- The redelegated trust's roles must be a subset of the original trustor's
roles (not the intermediate trustee's).
- If `impersonation=false` in the source trust, the redelegated trust cannot
set `impersonation=true`.
- Application credentials cannot create or delete trusts (prevents automated
escalation chains).
### 6.4 Automatic Trust Revocation
Trusts are automatically revoked (soft-deleted) when:
- The trustor is deleted.
- The trustee is deleted.
- The project is deleted.
- The trust expires (`expires_at`).
- The remaining uses are exhausted (`remaining_uses` reaches 0).
- The trustor loses a role that was delegated in the trust.
---
## 7. Application Credentials
### 7.1 Overview
Application credentials allow users to create long-lived, restricted credentials
for applications without exposing their password. This is especially important
for users whose identity comes from LDAP or SSO — applications can't use their
password.
**Key properties:**
| Property | Description |
|---|---|
| `name` | Unique name within the user's application credentials |
| `secret` | Auto-generated or user-provided secret (hashed on storage, shown once) |
| `project_id` | Project scope (always the user's current project) |
| `roles` | Subset of the user's roles on the project (cannot exceed user's roles) |
| `expires_at` | Optional expiration timestamp |
| `unrestricted` | `false` by default — restricted from creating/deleting other app creds and trusts |
### 7.2 Authentication with Application Credentials
```bash
# Auth with application credential ID + secret
curl -X POST /v3/auth/tokens -d '{
"auth": {
"identity": {
"methods": ["application_credential"],
"application_credential": {
"id": "aa809205ed614a0e854bac92c0768bb9",
"secret": "oKce6DOC_WcZoE13l3eX..."
}
}
}
}'
```
Or by name + user:
```bash
"application_credential": {
"name": "monitoring",
"user": { "name": "glance", "domain": { "name": "Default" } },
"secret": "securesecret"
}
```
### 7.3 Restriction Model
By default (`unrestricted=false`), application credentials **cannot**:
- Create or delete other application credentials.
- Create or delete trusts.
- List other application credentials.
This prevents a compromised app credential from regenerating itself or escalating
privileges. Setting `unrestricted=true` removes these restrictions, but adds
risk.
### 7.4 Rotation
Application credentials support **zero-downtime rotation**:
1. Create a new application credential (names must be unique per user).
2. Update the application configuration with the new ID/secret.
3. Delete the old application credential.
Multiple application credentials can coexist for the same user+project,
enabling seamless transitions.
### 7.5 Invalidation
Application credentials are automatically invalidated when:
- The user is deleted or disabled.
- The user's role assignment on the project changes (roles are checked at
auth time against the user's current roles).
- The project is deleted or disabled.
- The credential expires (`expires_at`).
- The credential is explicitly deleted.
---
## 8. Federation
### 8.1 Overview
Keystone's federation module allows external Identity Providers (IdPs) to
authenticate users, with Keystone acting as a Service Provider (SP). Keystone
maps the external identity to local users, groups, and roles.
**Supported protocols:**
| Protocol | Module | Use Case |
|---|---|---|
| **SAML 2.0** | mod_shib / mod_auth_mellon | Enterprise SSO |
| **OpenID Connect** | mod_auth_openidc | OAuth2/OIDC providers (Google, Keycloak, Okta) |
| **Mapped** | Custom auth module | Any HTTP auth module |
| **K2K** | Keystone-to-Keystone | Multi-cloud federation between OpenStack deployments |
### 8.2 Federation Architecture
```
┌──────────────────┐
│ External IdP │
│ (SAML/OIDC/...) │
└────────┬────────┘
│
SAML assertion or
OIDC claims
│
▼
┌──────────┐ HTTPD auth module ┌───────────────┐
│ Browser │ ───────────────────────▶│ Apache/Nginx │
│ or CLI │ (mod_shib / │ + auth module │
└──────────┘ mod_auth_openidc) └───────┬────────┘
│
REMOTE_USER header
+ other attributes
│
▼
┌──────────────────┐
│ Keystone │
│ (SP) │
│ │
│ 1. Lookup IdP │
│ 2. Apply mapping│
│ │ remote attrs │
│ │ → local user,│
│ │ groups, │
│ │ roles │
│ 3. Issue token │
└──────────────────┘
```
### 8.3 Key Federation Components
1. **Identity Provider** object — represents the external IdP in Keystone.
Has `remote_ids` (entity IDs) that Keystone uses to match incoming
requests.
2. **Mapping** — a set of rules that transform attributes from the external IdP
into Keystone-local user properties and group memberships. Mappings can:
- Map remote users to local users (by name, email, or other attributes).
- Assign users to local groups (inherit group role assignments).
- Dynamically create projects based on remote attributes.
- Support complex condition logic.
3. **Protocol** — links an Identity Provider to a Mapping. Supported values:
`saml2`, `openid`, `mapped`, or custom.
4. **Mapping rule example:**
```json
[{
"local": [{
"user": { "name": "{0}" },
"group": { "domain": { "name": "Default" }, "name": "federated_users" }
}],
"remote": [{ "type": "REMOTE_USER" }]
}]
```
This maps all authenticated external users to a local user (named by the
`REMOTE_USER` attribute) and adds them to the `federated_users` group.
### 8.4 Federation Token Flow
1. User authenticates with the external IdP.
2. The HTTPD auth module (Apache/Nginx) validates the assertion and sets
`REMOTE_USER` and other headers.
3. Keystone receives the request at `/v3/OS-FEDERATION/identity_providers/{idp}/protocols/{protocol}/auth`.
4. Keystone applies the mapping rules to produce a local user + groups + roles.
5. Keystone issues a **federated unscoped token**.
6. The user can then exchange it for a scoped token (project, domain, or
system) just like any other unscoped token.
### 8.5 Identity Provider (Keystone as IdP)
Keystone can also act as an **Identity Provider** (SAML IdP), allowing it to
authenticate users from other OpenStack deployments (K2K federation) or other
SAML SPs.
---
## 9. Service Catalog Deep Dive
### 9.1 Service Registration
Services are registered with Keystone via the API:
```bash
openstack service create --name nova --description "Compute" compute
openstack endpoint create --region RegionOne compute public https://nova.example.com:8774/
openstack endpoint create --region RegionOne compute internal https://nova.internal:8774/
openstack endpoint create --region RegionOne compute admin https://nova.admin:8774/
```
### 9.2 Catalog Filtering
The catalog returned in a token is filtered by:
1. **Scope**: A project-scoped token includes endpoints filtered by
project-endpoint associations.
2. **Endpoint groups**: Admins can define endpoint groups (filtered by service
type, region, or interface) and associate them with projects.
3. **Enabled/disabled**: Disabled services and endpoints don't appear in the
catalog.
4. **Interface visibility**: `public`, `internal`, and `admin` endpoints serve
different audiences.
### 9.3 URL Templating
Endpoint URLs support template variables:
- `$(project_id)s` — replaced with the token's project ID
- `$(user_id)s` — replaced with the token's user ID
Example:
```
https://object-store.example.com/v1/KEY_$(project_id)s
```
When a project-scoped token is issued, the catalog resolves this to:
```
https://object-store.example.com/v1/KEY_d12af07f4e2c4390a21acc31517ebec9
```
### 9.4 Client Discovery
An OpenStack client authenticates with Keystone, receives a token (which
includes the service catalog), and then uses the catalog to discover the URL
for any service it needs:
```python
# After authentication, the catalog is in the token response:
for service in token['catalog']:
if service['type'] == 'compute':
for endpoint in service['endpoints']:
if endpoint['interface'] == 'public':
nova_url = endpoint['url']
break
```
This is how every OpenStack client discovers service endpoints — they never
hardcode URLs. They authenticate once, get the catalog, and dynamically route
to the correct endpoint.
---
## 10. Mapping to alknet Concepts
### 10.1 Concept Comparison Table
| Keystone Concept | alknet Concept | Notes |
|---|---|---|
| Domain | (Not directly mapped) | alknet is single-tenant/small-team focused; no need for domain-level admin boundaries yet |
| Project | `Identity.resources` | Projects scope resources; alknet's `resources: HashMap<String, Vec<String>>` serves a similar scoping purpose |
| User | `Identity.id` | Keystone users ↔ alknet identities (fingerprint or UUID) |
| Group | (Not directly mapped) | Could be added via `Identity.scopes` patterns or a groups concept in alknet-storage |
| Role | `Identity.scopes` | Keystone roles map to alknet scopes: `["relay:connect", "service:gitea:read"]` ≈ role assignments |
| Token (scoped) | `AuthToken` + scoped permissions | alknet's AuthToken proves identity + timestamp; scopes come from IdentityProvider lookup |
| Service Catalog | `OperationRegistry` + OpenAPI spec generation | Both solve service discovery; Keystone is runtime API catalog, alknet generates from OpenAPI |
| Trust Delegation | (Potential future model) | alknet doesn't have delegation yet; trust model could inspire future `DelegationToken` |
| Application Credentials | API keys in `api_keys` table | alknet's `api_keys` table parallels app creds: long-lived, scoped, user-bound |
| Federation (SAML/OIDC) | Phase D OIDC provider aspiration | alknet wants to *be* an OIDC provider; Keystone consumes external IdPs |
| Service Endpoint | (Implicit in OperationEnv) | alknet operations are discovered via registry, not external endpoint lookup |
| Policy (policy.yaml) | `ForwardingPolicy` + call protocol ACL | Both enforce "who can do what where"; alknet is code-based, not YAML-configured |
### 10.2 What to Adopt from Keystone
#### 10.2.1 Scoped Tokens (Strong Adopt)
**Keystone pattern**: Unscoped → project/domain/system scoped token flow.
**alknet application**: Currently, `AuthToken` proves identity with a timestamp.
`Identity.scopes` and `Identity.resources` are resolved *after* token
verification by `IdentityProvider`. This is analogous to Keystone's flow:
| Keystone | alknet |
|---|---|
| Unscoped token (identity only) | AuthToken (proves key possession + timestamp) |
| Scoped token (identity + roles + catalog) | Identity (resolved by IdentityProvider with scopes + resources) |
| Re-auth with scope | Not needed — alknet scopes come from the `IdentityProvider` lookup |
**Recommendation**: alknet's current model is already similar to Keystone's, but
more streamlined. alknet doesn't need a separate "re-auth with scope" step
because the `IdentityProvider` resolution *is* the scoping step. However,
consider adding explicit scope fields to the token in the future for
multi-tenant deployments.
#### 10.2.2 Service Catalog Pattern (Strong Adopt)
**Keystone pattern**: Services register endpoints; clients discover them from
the token/catalog.
**alknet application**: The `OperationRegistry` + `OpenAPIServiceRegistry`
serves a similar purpose:
- Keystone: `POST /v3/auth/tokens` → response includes catalog of services
and URLs.
- alknet: `OperationRegistry` knows all available operations; `FromOpenAPI`
generates them from specs.
**Key difference**: In Keystone, the catalog is returned *with the token* and
is dynamic (filtered by project scope). In alknet, the registry is built at
startup from configuration, and access control is enforced per-operation in the
call protocol.
**Recommendation**: Consider adding a "service discovery" operation to the
call protocol — a way for clients to ask "what operations are available to me?"
This would be analogous to Keystone's `GET /v3/auth/catalog`.
#### 10.2.3 Role Hierarchies and Implied Roles (Moderate Adopt)
**Keystone pattern**: Roles can imply other roles (`admin` → `member` →
`reader`). Role assignments on domains propagate to projects via inheritance.
**alknet application**: Currently, alknet's scopes are flat strings. Consider:
```
admin:service:* → implies → member:service:* → implies → reader:service:*
```
This would simplify scope assignment in the `IdentityProvider`: grant `admin:service:*`
and automatically get `member` and `reader` permissions.
**Recommendation**: Implement implied scopes as a Phase 2+ feature when
alknet-storage adds the ACL graph. Don't over-engineer in Phase 1.
#### 10.2.4 Application Credentials (Strong Adopt — alreded parallels)
**Keystone pattern**: Password-less auth with restricted capabilities, tied to a
user and project, with expiration and rotation support.
**alknet application**: The `api_keys` table in alknet-storage is exactly this:
| Keystone App Credential | alknet API Key |
|---|---|
| `id` + `secret` | `key_prefix` + `key_hash` |
| `roles` (subset of user's roles) | `scopes` (subset of account's scopes) |
| `project_id` (scope) | Account-scoped |
| `expires_at` | `expires_at` |
| `unrestricted` | (not yet implemented) |
| Rotation via create-new-then-delete | (not yet implemented) |
**Recommendation**: Add the `unrestricted` concept to API keys — by default,
API keys should NOT be able to create or delete other API keys or modify
account settings. Also add rotation support (create new key, update config,
delete old key).
#### 10.2.5 Trust Delegation (Future Consideration)
**Keystone pattern**: Trustor delegates limited authority to trustee with
impersonation, expiration, usage limits, and redelegation chains.
**alknet application**: alknet doesn't have this yet, but it could be useful
for:
- **Service-to-service auth**: An alknet node delegates limited authority to a
service wrapper (e.g., "let the rustfs wrapper access S3 on my behalf for 1
hour").
- **Temporary access grants**: "Give Alice access to the `engineering` scope
for 24 hours."
- **Impersonation for audit**: Trusted services acting on behalf of a user,
with the user's identity appearing in audit logs.
**Recommendation**: Design a `DelegationToken` or `Trust` model when
alknet-storage is built. The trust model — trustor, trustee, roles, expiration,
remaining_uses — is a good template.
#### 10.2.6 Federation (Phase D Alignment)
**Keystone pattern**: External IdPs (SAML, OIDC) authenticate users; Keystone
maps them to local identities via mapping rules.
**alknet application**: Phase D of `credential-provider.md` envisions alknet
*as* an OIDC provider for self-hosted services. This is the **inverse** of
Keystone's federation model:
- Keystone: external IdP → Keystone (SP) → local identity
- alknet Phase D: alknet (IdP) → rustfs/gitea (SP) → local identity on self-hosted service
**Key learning from Keystone's federation model**:
1. **Mapping rules** are critical. Keystone's mapping engine (`local` ← `remote`)
is how IdP attributes become local roles. alknet will need the inverse:
`Identity.scopes` → OIDC claims → rustfs/gitea policies.
2. **Group membership from federation** is temporary by default (valid for
token lifetime). alknet should consider whether federated identities are
permanent or session-scoped.
3. **Multiple IdP support**: Keystone can consume from multiple external IdPs.
alknet Phase D should support multiple SPs (multiple self-hosted services)
consuming from one alknet IdP.
**Recommendation**: When building Phase D, study Keystone's mapping rule
format. alknet will need a similar concept: `alknet.scope → oidc.claim →
service.policy`. This could be part of the `CredentialProvider` or a new
`IdentityMappingProvider`.
### 10.3 What NOT to Adopt from Keystone
#### 10.3.1 Domains (Not Needed)
Keystone's domain model is designed for multi-tenant cloud hosting where
different organizations share the same OpenStack deployment. alknet is designed
for self-hosted, single-organization or small-team deployments. The domain
concept adds complexity that doesn't justify itself in alknet's use case.
alknet's `Identity.resources` already provides a lightweight scoping mechanism
that covers the "which resources does this identity have access to" use case
without the overhead of a domain hierarchy.
#### 10.3.2 Separate Policy Engine (Over-Engineering)
Keystone's `oslo.policy` is a full YAML-based policy engine with complex rule
combinations (`role:admin AND domain_id:%(target.domain.id)s OR
project_id:%(target.project.id)s`). alknet's authorization model is
programmatic (Rust code in `ForwardingPolicy` and call protocol handlers), not
configured via YAML. This is appropriate for alknet's size and complexity.
**If** alknet needs configurable policies in the future (e.g., admin-editable
ACL rules stored in the database), a simple rule engine would suffice — not the
full oslo.policy model.
#### 10.3.3 Multiple Token/Scope Types (Unnecessary Complexity)
Keystone has separate token types for project/domain/system scope. alknet's
`AuthToken` is already simpler: it proves identity + timestamp, and the
`IdentityProvider` resolves scopes. There's no need for alknet to issue
different token types for different scopes.
If multi-tenancy is added in the future, the `Identity.resources` map can
encode project equivalents without needing a separate token type.
#### 10.3.3 Service Endpoint Registration (Unnecessary)
Keystone requires every service to register its endpoints in the catalog
before it can be discovered. alknet services are registered programmatically
(via `OperationRegistry::register()`) at startup, not via a central API. The
`OperationRegistry` is built from configuration and OpenAPI specs, not from a
catalog service.
This is appropriate for alknet's architecture: services are known at deploy
time, not dynamically registered. If dynamic service discovery is needed later,
a simple registry operation in the call protocol would suffice.
---
## 11. Summary of Recommendations
| Keystone Concept | Adoption Level | alknet Implementation |
|---|---|---|
| **Scoped tokens** | ✅ Strong Adopt | Already present in IdentityProvider resolution (AuthToken → Identity with scopes/resources) |
| **Service catalog** | ✅ Strong Adopt | `OperationRegistry` + `FromOpenAPI`; consider adding "list operations" discovery |
| **Application credentials** | ✅ Strong Adopt | `api_keys` table parallels exactly; add `unrestricted` flag and rotation support |
| **Role hierarchies / implied roles** | ⚡ Moderate | Implied scope hierarchies in Phase 2+ when ACL graph is built |
| **Trust delegation** | ⚡ Moderate | Design `DelegationToken` model for service-to-service and temporary access in Phase 2+ |
| **Federation mapping** | ⚡ Moderate | Phase D: adopt `scope → claim → policy` mapping pattern for OIDC provider |
| **Token revocation events** | ⚡ Moderate | Consider pattern-matching revocation for efficiency when alknet-storage supports it |
| **Domains** | ❌ Skip | alknet is self-hosted/small-team; `Identity.resources` provides lightweight scoping |
| **oslo.policy (YAML-based)** | ❌ Skip | alknet uses programmatic auth (Rust code); add simple rule engine only if needed |
| **Multiple token types** | ❌ Skip | One token type with scope resolution via `IdentityProvider` is sufficient |
| **Endpoint registration API** | ❌ Skip | `OperationRegistry` is configured at startup, not via a catalog API |
---
## 12. References
- [Keystone Architecture — OpenStack Docs](https://docs.openstack.org/keystone/2024.2/getting-started/architecture.html)
- [Keystone Tokens Overview](https://docs.openstack.org/keystone/latest/admin/tokens-overview.html)
- [Keystone Service Catalog Overview](https://docs.openstack.org/keystone/latest/contributor/service-catalog.html)
- [Keystone Trusts Documentation](https://docs.openstack.org/keystone/latest/user/trusts.html)
- [Keystone Application Credentials](https://docs.openstack.org/keystone/queens/user/application_credentials.html)
- [Keystone Federation Configuration](https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html)
- [Keystone RBAC and Authorization — DeepWiki](https://deepwiki.com/openstack/keystone/4-authorization-and-access-control)
- [Keystone Authentication and Token Management — DeepWiki](https://deepwiki.com/openstack/keystone/3-authentication-and-token-management)
- [Keystone Trust Delegation — DeepWiki](https://deepwiki.com/openstack/keystone/4.4-trust-delegation)
- [Keystone Service Catalog — DeepWiki](https://deepwiki.com/openstack/keystone/5.4-service-catalog)
- [Keystone Token Revocation — DeepWiki](https://deepwiki.com/openstack/keystone/3.4-token-revocation)
- [Understanding OpenStack Keystone: Scoped vs. Unscoped Tokens](https://osie.io/blog/understanding-openstack-keystone-scoped-vs-unscoped-tokens)
- [Trust Delegation in OpenStack Using Keystone Trusts](https://blog.zhaw.ch/icclab/trust-delegation-in-openstack-using-keystone-trusts/)
- [OpenStack Knowledge: Keystone Federation](https://github.com/stackers-network/openstack-knowledge/blob/main/core/identity/federation.md)
- [alknet identity.md](../../architecture/identity.md)
- [alknet auth.md](../../architecture/auth.md)
- [alknet credential-provider.md](../phase2/credential-provider.md)

137
docs/research/references/polyglot/01_overview.md
View File

@@ -1,137 +0,0 @@
# Polyglot: Research Overview
**Library**: `polyglot-sql` (Rust crate) / `@polyglot-sql/sdk` (TypeScript/WASM) / `polyglot-sql` (Python)
**Repository**: <https://github.com/tobilg/polyglot>
**Current Version**: 0.4.4 (as of 2026-06-03)
**License**: MIT (+ sqlglot MIT for test fixtures)
**Author**: Tobias G. (tobilg)
**Inspiration**: Python [sqlglot](https://github.com/tobymao/sqlglot) by Toby Mao
---
## 1. What Is Polyglot?
Polyglot is a **SQL transpiler** — it parses SQL from one database dialect into an AST, and generates SQL for a different dialect. It is **not** a database driver, ORM, query executor, or connection pool. Its core purpose is **dialect-agnostic SQL manipulation**: parse, transform, validate, format, and transpile SQL across 32+ database dialects.
### Key Capabilities
| Capability | Description |
|---|---|
| **Parse** | Convert SQL string → typed AST with 200+ expression node types |
| **Generate** | Convert AST → SQL string for any supported dialect |
| **Transpile** | Convert SQL from dialect A → dialect B in one call |
| **Format** | Pretty-print SQL with configurable guard rails |
| **Build** | Construct SQL programmatically via a fluent builder API |
| **Validate** | Syntax + semantic validation with error positions |
| **Lineage** | Trace column lineage through queries; generate OpenLineage payloads |
| **Diff** | AST-aware diff between two SQL expressions |
| **Traverse** | DFS/BFS iterators, predicate queries, and transforms on the AST |
### Supported Dialects (32)
Athena, BigQuery, ClickHouse, CockroachDB, Databricks, Doris, Dremio, Drill, Druid, DuckDB, Dune, Exasol, Fabric, Hive, Materialize, MySQL, Oracle, PostgreSQL, Presto, Redshift, RisingWave, SingleStore, Snowflake, Solr, Spark, SQLite, StarRocks, Tableau, Teradata, TiDB, Trino, TSQL
Plus a `Generic` dialect for standard SQL.
### Language Bindings
| Binding | Package | Delivery |
|---|---|---|
| **Rust** | `polyglot-sql` on crates.io | Native Rust crate |
| **TypeScript/WASM** | `@polyglot-sql/sdk` on npm | WASM module + JS wrapper |
| **Python** | `polyglot-sql` on PyPI | PyO3 native extension |
| **Go** | `github.com/tobilg/polyglot/packages/go` | PureGo wrapper over C FFI |
| **C FFI** | Built from `polyglot-sql-ffi` | `.so` / `.dylib` / `.dll` + `.a` / `.lib` + header |
---
## 2. Core Philosophy & Design Principles
1. **Pipeline architecture**: SQL → Tokenize → Parse → AST → Transform → Generate → SQL string. Each stage is independently configurable per dialect.
2. **Ported from Python sqlglot**: The Rust implementation is a faithful port of the Python `sqlglot` library, maintaining compatibility with its test fixtures (10,220+ fixture cases at 100% pass rate). The architecture, expression types, transformation rules, and dialect behaviors mirror the Python original.
3. **No runtime database connection**: Polyglot never connects to a database. It operates purely on SQL strings and ASTs. This makes it safe for sandboxed environments (WASM, serverless) and suitable for build-time / CI-time SQL analysis.
4. **Feature-gated compilation**: Each dialect is behind a Cargo feature flag (`dialect-postgresql`, `dialect-mysql`, etc.), so users compiling for constrained targets (WASM) can include only what they need. The `default` feature set includes everything.
5. **Stack safety**: The `stacker` feature (default-on for native builds) grows the stack on deeply nested inputs, preventing stack overflow from pathological SQL. WASM builds opt out since `stacker` doesn't work there.
6. **Guard rails**: Format/guard options limit input size (16 MiB default), token count (1M), AST node count (1M), and set-operation chain depth (256) to prevent resource exhaustion.
7. **Performance-first**: Built in Rust for speed. Benchmarks show 8–19× speedup over the Python `sqlglot` for transpilation, with generation at ~86× faster. The WASM build enables near-native performance in browsers.
---
## 3. How It Differs from Database Abstraction Layers
**Critical distinction**: Polyglot is a **SQL dialect transpiler**, not a database abstraction layer. It does not:
- Connect to databases
- Execute queries
- Manage connection pools
- Handle migrations (no `CREATE TABLE` schema evolution management)
- Map Rust types to database types
- Provide an ORM-like interface
- Handle async I/O
Instead, it focuses purely on **SQL text manipulation**: parsing, analyzing, transforming, and generating SQL strings. This makes it complementary to (not competing with) libraries like Diesel, SQLx, or SeaORM.
---
## 4. Performance Characteristics
From the project's benchmark suite (polyglot-sql v0.1.2 vs sqlglot v28.10.1):
| Operation | Speedup Range |
|---|---|
| Parse (SQL → AST) | 10–13× faster |
| Generate (AST → SQL) | 77–101× faster |
| Roundtrip (parse → generate → re-parse) | 13–15× faster |
| Transpile (full cross-dialect) | 1.6× (simple) to 19× (complex BigQuery→Snowflake) |
| Geometric mean | **8.70×** |
Parse benchmarks (v0.4.x, native Rust):
| Query | Mean |
|---|---|
| short (SELECT a, b, c) | 51.28 μs |
| medium (5 cols, JOIN, GROUP BY) | 259.61 μs |
| complex (3 CTEs, subquery) | 268.59 μs – 1.03 ms |
---
## 5. Project Maturity Indicators
| Indicator | Status |
|---|---|
| **Version** | 0.4.4 (pre-1.0, active development) |
| **Test coverage** | 18,745 test cases at 100% pass rate |
| **crates.io downloads** | ~4,738 total (as of mid-2026) |
| **Dependent crates** | 2 (via entdb) |
| **Release cadence** | Frequent patch releases (0.4.2, 0.4.3, 0.4.4 in quick succession) |
| **Source code size** | ~241K lines of Rust in core crate |
| **Fuzzing** | Supported via `cargo +nightly fuzz` |
| **CI** | Full test suite + FFI + Python + WASM |
| **Documentation** | Rust API docs (docs.rs), TypeScript docs, Python docs, playground |
| **Breaking changes** | Possible before 1.0; semver suggests API instability |
---
## 6. License
- **MIT License** for the Polyglot code itself
- **sqlglot MIT License** for the test fixtures derived from the Python project
- Both are permissive, suitable for commercial use
---
## References
- <https://github.com/tobilg/polyglot> — Main repository
- <https://crates.io/crates/polyglot-sql> — Rust crate on crates.io
- <https://www.npmjs.com/package/@polyglot-sql/sdk> — TypeScript SDK on npm
- <https://pypi.org/project/polyglot-sql/> — Python bindings on PyPI
- <https://docs.rs/polyglot-sql/latest/polyglot_sql/> — Rust API documentation
- <https://polyglot-playground.gh.tobilg.com/> — Interactive playground
- <https://github.com/tobymao/sqlglot> — Original Python inspiration

720
docs/research/references/polyglot/02_architecture.md
View File

@@ -1,720 +0,0 @@
# Polyglot: Architecture Deep Dive
---
## 1. Workspace Structure
The repository is organized as a Cargo workspace with 5 crates and supporting packages:
```
polyglot/
├── crates/
│ ├── polyglot-sql/ # Core Rust library (~241K LOC)
│ │ └── src/
│ │ ├── lib.rs # Public API, top-level functions
│ │ ├── tokens.rs # Tokenizer (lexer)
│ │ ├── parser.rs # Recursive-descent parser (~62K LOC)
│ │ ├── expressions.rs # AST node types (~15K LOC)
│ │ ├── generator.rs # SQL code generator (~39K LOC)
│ │ ├── dialects/ # 33 dialect implementations
│ │ │ ├── mod.rs # Dialect trait, Dialect struct, CustomDialectBuilder
│ │ │ ├── generic.rs # Base/standard SQL dialect
│ │ │ ├── postgres.rs # PostgreSQL (~1.9K LOC)
│ │ │ ├── mysql.rs # MySQL
│ │ │ ├── sqlite.rs # SQLite
│ │ │ ├── bigquery.rs # BigQuery
│ │ │ ├── ... (32 total)
│ │ ├── builder.rs # Fluent query builder API
│ │ ├── transforms.rs # Cross-dialect transform functions
│ │ ├── validation.rs # Syntax + semantic validation
│ │ ├── schema.rs # Schema representation
│ │ ├── scope.rs # Scope analysis
│ │ ├── resolver.rs # Column resolution
│ │ ├── lineage.rs # Column lineage tracking
│ │ ├── openlineage.rs # OpenLineage payload generation
│ │ ├── diff.rs # AST diff (ChangeDistiller algorithm)
│ │ ├── planner.rs # Logical query plan
│ │ ├── optimizer/ # Query optimizer modules
│ │ │ ├── annotate_types.rs # Type annotation
│ │ │ ├── qualify_columns.rs # Column qualification
│ │ │ ├── qualify_tables.rs # Table qualification
│ │ │ ├── pushdown_predicates.rs
│ │ │ ├── pushdown_projections.rs
│ │ │ ├── eliminate_joins.rs
│ │ │ ├── eliminate_ctes.rs
│ │ │ ├── simplify.rs
│ │ │ └── ...
│ │ ├── traversal.rs # DFS/BFS visitors, AST predicates
│ │ ├── ast_transforms.rs # AST manipulation utilities
│ │ ├── error.rs # Error types
│ │ └── time.rs # Time format conversion
│ ├── polyglot-sql-function-catalogs/ # Optional dialect function catalogs
│ ├── polyglot-sql-wasm/ # WASM bindings (wasm-pack)
│ ├── polyglot-sql-ffi/ # C FFI bindings (cbindgen)
│ └── polyglot-sql-python/ # Python bindings (PyO3 + maturin)
├── packages/
│ ├── sdk/ # TypeScript SDK (@polyglot-sql/sdk)
│ ├── go/ # Go SDK (PureGo wrapper over FFI)
│ ├── documentation/ # TypeScript API docs site
│ ├── playground/ # Browser playground (React 19, Vite)
│ └── python-docs/ # Python API docs
├── examples/
│ ├── rust/ # Rust usage example
│ ├── typescript/ # TypeScript SDK example
│ └── c/ # C FFI usage example
└── tools/
├── sqlglot-compare/ # Fixture extraction & comparison
└── bench-compare/ # Performance benchmarks
```
---
## 2. Data Flow Pipeline
```
┌──────────────────────────────────────────────────────────────────────┐
│ SQL String (source dialect) │
└──────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Tokenizer (tokens.rs) │
│ • Dialect-specific lexing rules (quotes, comments, keywords) │
│ • Configurable via TokenizerConfig per dialect │
│ • Produces Vec<Token> with type, text, and Span (line/col/offset) │
└──────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Parser (parser.rs, ~62K LOC) │
│ • Recursive-descent with precedence climbing │
│ • Dialect-aware parsing (custom keywords, syntax rules) │
│ • Produces Expression AST tree │
│ • Stack safety via `stacker` feature (default-on) │
└──────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Expression AST (expressions.rs) │
│ • Single tagged enum with 150+ variants │
│ • Each variant has its own struct (Select, Insert, Function, etc.) │
│ • Box<Variant> keeps enum size to 2 words (tag + pointer) │
│ • Serializable via serde (derive Serialize/Deserialize) │
│ • Optional TypeScript type generation via `ts-rs` feature flag │
└──────────────────────────┬──────────────────────────────────────────┘
│
┌────┴────┐
│ │
┌─────────┘ └──────────┐
│ │
▼ ▼
┌────────────────────────┐ ┌────────────────────────────────────┐
│ Transform Pipeline │ │ Semantic / Analysis Modules │
│ (transpile path) │ │ • validation.rs → syntax checks │
│ │ │ • schema.rs → column/type lookup │
│ 1. preprocess() │ │ • scope.rs → scope analysis │
│ (whole-tree rewrites│ │ • resolver.rs → column resolution │
│ like eliminate_ │ │ • lineage.rs → column lineage │
│ qualify) │ │ • openlineage.rs → OL payloads │
│ │ │ • optimizer/ → query optimization │
│ 2. transform_expr() │ │ • diff.rs → AST diff │
│ (per-node rewrites │ │ • planner.rs → logical plan DAG │
│ per dialect) │ │ • traversal.rs → DFS/BFS visitors │
│ │ │
│ 3. Generator │ │
│ (AST → SQL string) │ │
└───────────┬────────────┘ └────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────┐
│ SQL String (target dialect) │
└──────────────────────────────────────────────────────────────────────┘
```
---
## 3. Core Abstractions
### 3.1 Expression AST
The central type is `Expression`, a large tagged enum with one variant per SQL construct:
```rust
pub enum Expression {
// Literals
Literal(Box<Literal>),
Boolean(BooleanLiteral),
Null(Null),
// Identifiers
Identifier(Identifier),
Column(Box<Column>),
Table(Box<TableRef>),
Star(Star),
// Queries
Select(Box<Select>),
Union(Box<Union>),
Intersect(Box<Intersect>),
Except(Box<Except>),
Subquery(Box<Subquery>),
// DML
Insert(Box<Insert>),
Update(Box<Update>),
Delete(Box<Delete>),
Copy(Box<CopyStmt>),
// Binary/Unary operators
And(Box<BinaryOp>),
Or(Box<BinaryOp>),
Add(Box<BinaryOp>),
Eq(Box<BinaryOp>),
// ... 30+ operator variants
// Functions
Function(Box<Function>),
AggregateFunction(Box<AggregateFunction>),
WindowFunction(Box<WindowFunction>),
// Clauses
From(Box<From>),
Join(Box<Join>),
Where(Box<Where>),
OrderBy(Box<OrderBy>),
// ...
// ~150 total variants
}
```
Key design choices:
- **Boxed variants**: Most variants wrap their payload in `Box` to keep `size_of::<Expression>()` at 2 words (16 bytes on 64-bit).
- **Serde support**: `#[derive(Serialize, Deserialize)]` for JSON serialization across FFI/WASM boundaries.
- **TypeScript types**: Optional `ts-rs` feature generates TypeScript interfaces.
- **Convenience methods**: `Expression::column()`, `Expression::number()`, `Expression::sql()`, `Expression::sql_for()`.
### 3.2 DialectType Enum
```rust
pub enum DialectType {
Generic, PostgreSQL, MySQL, BigQuery, Snowflake, DuckDB, SQLite,
Hive, Spark, Trino, Presto, Redshift, TSQL, Oracle, ClickHouse,
Databricks, Athena, Teradata, Doris, StarRocks, Materialize,
RisingWave, SingleStore, CockroachDB, TiDB, Druid, Solr, Tableau,
Dune, Fabric, Drill, Dremio, Exasol, DataFusion,
}
```
- Implements `FromStr` with aliases (e.g., `"mssql"` → `TSQL`, `"cockroach"` → `CockroachDB`)
- Each variant maps to a feature-gated dialect module
- Custom dialects can be registered at runtime via `CustomDialectBuilder`
### 3.3 DialectImpl Trait
```rust
pub trait DialectImpl {
fn dialect_type(&self) -> DialectType;
fn tokenizer_config(&self) -> TokenizerConfig { /* default */ }
fn generator_config(&self) -> GeneratorConfig { /* default */ }
fn generator_config_for_expr(&self, _expr: &Expression) -> GeneratorConfig { /* default */ }
fn transform_expr(&self, expr: Expression) -> Result<Expression> { Ok(expr) }
fn preprocess(&self, expr: Expression) -> Result<Expression> { Ok(expr) }
}
```
Each dialect implements this trait to provide:
1. **Tokenizer config**: Identifier quoting characters, string delimiters, keyword overrides, comment styles, hex number support
2. **Generator config**: 30+ flags controlling SQL output (identifier quote style, function casing, `LIMIT` vs `TOP` vs `FETCH FIRST`, etc.)
3. **Per-node transform**: Dialect-specific expression rewrites (e.g., PostgreSQL transforms `IFNULL` → `COALESCE`, SQLite transforms `TRY_CAST` → `CAST`)
4. **Whole-tree preprocess**: Structural rewrites that need full-tree context (e.g., eliminating `QUALIFY` for dialects that don't support it)
### 3.4 Dialect Struct (High-Level API)
```rust
pub struct Dialect {
dialect_type: DialectType,
tokenizer: Tokenizer,
generator_config: Arc<GeneratorConfig>,
transformer: Box<dyn Fn(Expression) -> Result<Expression> + Send + Sync>,
generator_config_for_expr: Option<Box<dyn Fn(&Expression) -> GeneratorConfig + Send + Sync>>,
custom_preprocess: Option<Box<dyn Fn(Expression) -> Result<Expression> + Send + Sync>>,
}
```
The `Dialect` struct bundles all dialect-specific state and provides the primary API:
```rust
// Parse SQL
let ast = dialect.parse("SELECT 1")?;
// Generate SQL from AST
let sql = dialect.generate(&ast[0])?;
// Transpile between dialects
let results = dialect.transpile("SELECT IFNULL(a,b) FROM t", DialectType::PostgreSQL)?;
// Tokenize
let tokens = dialect.tokenize("SELECT 1")?;
```
### 3.5 CustomDialectBuilder
For runtime-extensible dialect support:
```rust
use polyglot_sql::dialects::{CustomDialectBuilder, Dialect, DialectType};
use polyglot_sql::generator::NormalizeFunctions;
// Register a custom dialect inheriting from PostgreSQL
CustomDialectBuilder::new("my_postgres")
.based_on(DialectType::PostgreSQL)
.generator_config_modifier(|gc| {
gc.normalize_functions = NormalizeFunctions::Lower;
})
.register()?;
let d = Dialect::get_by_name("my_postgres").unwrap();
// Use like any built-in dialect
```
---
## 4. Dialect Implementation Details
### 4.1 PostgreSQL (`postgres.rs`, ~1,879 LOC)
**Tokenizer:**
- `$$` string literals (dollar-quoting)
- Double-quote identifier quoting
- Nested block comments
- `EXEC` treated as generic command
**Generator config highlights:**
- `identifier_quote: '"'` (double quotes)
- `single_string_interval: true` (`INTERVAL '1 day'`)
- `parameter_token: "$"` (`$1`, `$2` placeholders)
- `supports_select_into: true`
- `supports_window_exclude: true`
- `can_implement_array_any: true`
**Transform examples:**
- `IFNULL(a, b)` → `COALESCE(a, b)`
- `RAND()` → `RANDOM()`
- `DATEDIFF(day, a, b)` → `CAST(b - a AS INT)` (date subtraction)
- `JSON_EXTRACT(a, '$.x')` → `a #> '{x}'` (arrow syntax)
- `JSON_EXTRACT_SCALAR(a, '$.x')` → `a #>> '{x}'`
- `DATE_ADD` / `DATE_SUB` → `+` / `-` interval arithmetic
- Type mappings: `TINYINT` → `SMALLINT`, `FLOAT` → `REAL`, `DOUBLE` → `DOUBLE PRECISION`
- `ILIKE` preserved (native PostgreSQL)
- `RegexpLike` → `~` operator, `RegexpILike` → `~*` operator
### 4.2 SQLite (`sqlite.rs`, ~750 LOC)
**Tokenizer:**
- Supports `"`, `[`, `` ` `` as identifier quote characters
- No nested comments
- Hex number literals (`0xCC`)
**Generator config:**
- `identifier_quote: '"'` (double quotes)
- `supports_table_alias_columns: false`
- `json_key_value_pair_sep: ","` (comma-style `JSON_OBJECT`)
**Transform examples:**
- `NVL(a, b)` → `IFNULL(a, b)`
- `TRY_CAST(x AS t)` → `CAST(x AS t)` (no try-cast)
- `RANDOM()` → function
- `ILIKE` → `LOWER(left) LIKE LOWER(right)` (no native ILIKE)
- `CountIf(cond)` → `SUM(IIF(cond, 1, 0))`
- `CEIL(x)` → function form
- `DATE_TRUNC(unit, col)` → various strftime patterns
- `DATE_DIFF` → `juliandiff` patterns
### 4.3 MySQL (`mysql.rs`)
**Tokenizer:** Backtick identifiers, `#` comments
**Generator:** Backtick quoting, `LIMIT` syntax, `CONCAT()` instead of `||`
**Transforms:** `COALESCE(a,b)` ← `IFNULL(a,b)`, `||` → `CONCAT()` (string concat), etc.
### 4.4 BigQuery (`bigquery.rs`)
**Tokenizer:** Backtick identifiers, `QUALIFY` keyword
**Generator:** Backtick quoting, `STRUCT` types, `QUALIFY` clause, `DATE_DIFF` syntax
**Transforms:** Complex date/timestamp function mappings, `UNNEST` handling, `APPROX_COUNT_DISTINCT` → `APPROX_COUNT_DISTINCT`
### 4.5 How Transpilation Works
The full transpilation pipeline:
```
Input SQL (source dialect)
│
▼
Source Dialect Tokenizer
│
▼
Parser (dialect-aware)
│
▼
Expression AST
│
▼
Source Dialect::preprocess() ← whole-tree rewrites
│
▼
Source Dialect::transform_expr() ← per-node rewrites (recursive, bottom-up)
│
▼
Normalized AST
│
▼
Target Dialect Generator
│
▼
Output SQL (target dialect)
```
The transform pipeline uses an explicit task stack (not recursive calls) for the hot paths to avoid stack overflow. The `stacker` crate provides additional stack-growth protection.
Key cross-dialect transforms include:
- Function renaming: `IFNULL` ↔ `COALESCE` ↔ `NVL`, `DATEDIFF` ↔ date arithmetic, `STRING_AGG` ↔ `GROUP_CONCAT`
- Type mapping: `TINYINT` ↔ `SMALLINT`, `FLOAT` ↔ `REAL`, `JSON` ↔ `JSONB`
- Syntax conversion: `LIMIT` ↔ `TOP` ↔ `FETCH FIRST`, `||` (concat) ↔ `CONCAT()`, `SELECT INTO` ↔ `CREATE TABLE AS`
- Boolean handling: `BOOL_AND`/`BOOL_OR` ↔ `MIN`/`MAX`-over-`CASE`
- JSON operators: `JSON_EXTRACT` ↔ `#>`/`#>>` ↔ `->`/`->>` (PostgreSQL arrow syntax)
---
## 5. Fluent Builder API
The builder module (`builder.rs`, ~3.3K LOC) provides a type-safe, ergonomic way to construct SQL expressions without string interpolation:
```rust
use polyglot_sql::builder::*;
// SELECT id, name FROM users WHERE age > 18 ORDER BY name LIMIT 10
let expr = select(["id", "name"])
.from("users")
.where_(col("age").gt(lit(18)))
.order_by(["name"])
.limit(10)
.build();
// INSERT
let ins = insert_into("users")
.columns(["id", "name"])
.values([lit(1), lit("Alice")])
.build();
// CASE expression
let expr = case()
.when(col("x").gt(lit(0)), lit("positive"))
.else_(lit("non-positive"))
.build();
// Set operations
let expr = union_all(
select(["id"]).from("a"),
select(["id"]).from("b"),
).order_by(["id"]).limit(5).build();
```
Expression helpers:
- `col("users.id")` — column reference (splits on last `.`)
- `lit(42)`, `lit("hello")`, `lit(3.14)`, `lit(true)` — literals
- `func("COALESCE", [col("a"), col("b")])` — function calls
- Operator chain: `col("age").gte(lit(18)).and(col("status").eq(lit("active")))`
The builder generates an `Expression` AST that can then be serialized to any dialect via `generate()`.
---
## 6. Validation and Schema-Aware Analysis
### 6.1 Syntax Validation
```rust
use polyglot_sql::{validate, DialectType};
let result = validate("SELECT * FORM users", DialectType::Generic);
// result.valid == false
// result.errors contain line/column/message/error codes
```
Error codes:
- `E001` — Syntax error
- `E002` — Tokenization error
- `E003` — Parse error
- `E004` — Invalid expression (not a valid statement)
- `E005` — Trailing comma in strict mode
### 6.2 Schema-Aware Validation
```rust
use polyglot_sql::{
validate_with_schema, DialectType, SchemaColumn, SchemaTable,
SchemaValidationOptions, ValidationSchema,
};
let schema = ValidationSchema {
strict: Some(true),
tables: vec![
SchemaTable {
name: "users".into(),
columns: vec![
SchemaColumn { name: "id".into(), data_type: "integer".into(), nullable: Some(false), primary_key: true, unique: false, references: None },
SchemaColumn { name: "email".into(), data_type: "varchar".into(), nullable: Some(false), primary_key: false, unique: true, references: None },
],
// ...
},
],
};
let opts = SchemaValidationOptions { check_types: true, check_references: true, strict: None, semantic: true };
let result = validate_with_schema("SELECT id FROM users WHERE email = 1", DialectType::Generic, &schema, &opts);
// result.valid == false (type mismatch: email is varchar, compared to integer)
```
Schema-aware error codes:
- `E200`/`E201` — Unknown table/column
- `E210`–`E217`, `W210`–`W216` — Type checks
- `E220`, `E221`, `W220`, `W221`, `W222` — Reference/FK checks
### 6.3 Function Catalogs
Optional feature-gated function catalogs (currently ClickHouse and DuckDB) provide known function signatures for semantic type checking:
```toml
polyglot-sql = { version = "0.4", features = ["function-catalog-clickhouse"] }
```
---
## 7. Column Lineage & OpenLineage
### 7.1 Column Lineage
Trace how columns flow through a query:
```rust
use polyglot_sql::{parse, DialectType};
use polyglot_sql::lineage::get_column_lineage;
let ast = parse("SELECT a + b AS total FROM t", DialectType::Generic).unwrap();
let lineage = get_column_lineage(&ast[0], /* schema */ None, DialectType::Generic);
// lineage tells you that "total" depends on columns "a" and "b" from table "t"
```
### 7.2 OpenLineage Payload Generation
```rust
use polyglot_sql::openlineage::{generate_run_event, OpenLineageOptions, OpenLineageDatasetId};
let opts = OpenLineageOptions {
dialect: DialectType::PostgreSQL,
producer: "my-app".into(),
dataset_namespace: Some("mydb".into()),
// ...
};
let event = generate_run_event("SELECT * FROM users", &opts)?;
// event is a JSON-serializable OpenLineage RunEvent with columnLineage facets
```
---
## 8. Error Handling
### 8.1 Error Types
```rust
pub enum Error {
Tokenize { message: String, line: usize, column: usize, start: usize, end: usize },
Parse { message: String, line: usize, column: usize, start: usize, end: usize },
Generate(String),
Unsupported { feature: String, dialect: String },
Syntax { message: String, line: usize, column: usize, start: usize, end: usize },
Internal(String),
}
```
All position-bearing errors include:
- `line` — 1-based line number
- `column` — 1-based column number
- `start` / `end` — byte offsets (0-based, end exclusive)
```rust
let err = Error::parse("Unexpected token", 3, 15, 42, 44);
assert_eq!(err.line(), Some(3));
assert_eq!(err.column(), Some(15));
assert_eq!(err.start(), Some(42));
```
### 8.2 Validation Errors
```rust
pub struct ValidationError {
pub message: String,
pub line: Option<usize>,
pub column: Option<usize>,
pub severity: ValidationSeverity, // Error or Warning
pub code: String, // e.g., "E001", "E200"
pub start: Option<usize>,
pub end: Option<usize>,
}
pub struct ValidationResult {
pub valid: bool,
pub errors: Vec<ValidationError>,
}
```
### 8.3 Guard Rail Errors
Format operations have configurable guard limits that return structured errors:
- `E_GUARD_INPUT_TOO_LARGE` — input exceeds `max_input_bytes`
- `E_GUARD_TOKEN_BUDGET_EXCEEDED` — token count exceeds `max_tokens`
- `E_GUARD_AST_BUDGET_EXCEEDED` — AST node count exceeds `max_ast_nodes`
- `E_GUARD_SET_OP_CHAIN_EXCEEDED` — UNION/INTERSECT/EXCEPT chain exceeds `max_set_op_chain`
---
## 9. AST Traversal & Analysis
### 9.1 Traversal
```rust
use polyglot_sql::{parse, DialectType};
use polyglot_sql::traversal::*;
let ast = parse("SELECT a, b FROM t WHERE x > 1", DialectType::Generic).unwrap();
let columns = get_columns(&ast[0]); // ["a", "b", "x"]
let tables = get_tables(&ast[0]); // ["t"]
```
Available predicates (70+):
- `is_select`, `is_insert`, `is_update`, `is_delete`, `is_ddl`
- `is_join`, `is_where`, `is_group_by`, `is_order_by`, `is_limit`
- `is_function`, `is_aggregate`, `is_subquery`, `is_cte`
- `is_comparison`, `is_logical`, `is_arithmetic`
- `contains_subquery`, `contains_aggregate`, `contains_window_function`
Iterators: `DfsIter`, `BfsIter` for depth-first and breadth-first traversal.
### 9.2 AST Transforms
```rust
use polyglot_sql::ast_transforms::*;
// Rename tables
let renamed = rename_tables(expr, &[("old_name", "new_name")]);
// Add WHERE condition
let filtered = add_where(expr, col("active").eq(lit(true)));
// Remove LIMIT/OFFSET
let unlimited = remove_limit_offset(expr);
```
### 9.3 AST Diff
```rust
use polyglot_sql::diff::{diff, diff_with_config, DiffConfig};
let edits = diff(&source_expr, &target_expr, true);
for edit in &edits {
if edit.is_change() {
println!("{:?}", edit);
}
}
```
Uses the ChangeDistiller algorithm with Dice coefficient matching for structural comparison.
### 9.4 Logical Planner
```rust
use polyglot_sql::planner::Plan;
let plan = Plan::from_expression(&expr);
// plan.root is a Step DAG
// plan.leaves() returns leaf steps
// plan.dag() returns the dependency graph
```
Step kinds: Scan, Filter, Project, Aggregate, Join, Sort, Limit, etc.
---
## 10. Optimizer Modules
The optimizer is available behind the `semantic` feature flag:
| Module | Purpose |
|---|---|
| `qualify_columns.rs` | Resolve unqualified column references to table.column |
| `qualify_tables.rs` | Expand table names with schema/catalog |
| `annotate_types.rs` | Infer and annotate expression types |
| `pushdown_predicates.rs` | Push WHERE conditions into JOINs |
| `pushdown_projections.rs` | Reduce columns to only what's needed |
| `eliminate_joins.rs` | Remove unnecessary JOINs |
| `eliminate_ctes.rs` | Inline single-use CTEs |
| `simplify.rs` | Simplify boolean expressions, constant folding |
| `normalize.rs` | Expression normalization |
| `canonicalize.rs` | Query canonicalization |
| `subquery.rs` | Subquery analysis |
---
## 11. Async Support
**Polyglot does not use async I/O** — it is a pure computational library. All operations are synchronous and CPU-bound:
- `parse()` — synchronous
- `generate()` — synchronous
- `transpile()` — synchronous
- `validate()` — synchronous
- `format()` — synchronous
This is by design: Polyglot operates on SQL strings in memory, with no network or filesystem I/O. For use in async contexts (Tokio, async-std), callers should use `tokio::task::spawn_blocking()` or similar to offload CPU-heavy parsing/transpilation to a blocking thread pool.
---
## 12. Feature Flags
| Flag | Description | Default |
|---|---|---|
| `all-dialects` | Enable all 32 dialect parsers | ✅ |
| `generate` | SQL generation from AST | ✅ |
| `transpile` | Cross-dialect transpilation (implies `generate`) | ✅ |
| `builder` | Fluent query builder API (implies `generate`) | ✅ |
| `ast-tools` | AST inspection & transform utilities | ✅ |
| `semantic` | Schema, resolver, lineage, optimizer, validation | ✅ |
| `openlineage` | OpenLineage payload generation (implies `semantic`) | ✅ |
| `diff` | AST diff support (implies `generate`) | ✅ |
| `planner` | Logical planning helpers | ✅ |
| `time` | Time-format conversion helpers | ✅ |
| `stacker` | Stack-growth protection for native builds | ✅ |
| `bindings` | TypeScript type generation via `ts-rs` | ❌ |
| `dialect-postgresql` | PostgreSQL dialect only | — |
| `dialect-mysql` | MySQL dialect only | — |
| ... (one per dialect) | Individual dialect selector | — |
| `function-catalog-clickhouse` | ClickHouse function catalog | ❌ |
| `function-catalog-duckdb` | DuckDB function catalog | ❌ |
| `function-catalog-all-dialects` | All function catalogs | ❌ |
Minimal WASM build (for constrained targets):
```toml
polyglot-sql = { version = "0.4", default-features = false, features = ["generate", "transpile", "dialect-postgresql", "dialect-mysql"] }
```
---
## References
- Source code examined: `/workspace/polyglot/crates/polyglot-sql/src/` (~241K LOC)
- Architecture documentation: `/workspace/polyglot/docs/sqlglot-architecture.md`
- Benchmark results: `/workspace/polyglot/docs/benchmark.md`
- README: `/workspace/polyglot/README.md`, `/workspace/polyglot/crates/polyglot-sql/README.md`
- CHANGELOG: `/workspace/polyglot/CHANGELOG.md`

294
docs/research/references/polyglot/03_analysis.md
View File

@@ -1,294 +0,0 @@
# Polyglot: Suitability Analysis & Comparisons
---
## 1. What Polyglot Is NOT
Before evaluating suitability, it's essential to understand what Polyglot **does not** do:
| NOT a... | Because |
|---|---|
| **Database driver** | No connection management, no query execution, no result set handling |
| **ORM** | No object-relational mapping, no model definitions, no active record pattern |
| **Migration tool** | No `CREATE TABLE` evolution management, no up/down migrations framework |
| **Type mapper** | No Rust type → SQL type mapping, no `FromRow` derives |
| **Connection pool** | No async I/O, no TCP connections, no TLS |
| **Query executor** | Never connects to a database; operates purely on SQL text |
**Polyglot is a SQL dialect transpiler.** It converts SQL strings between database dialects. Period.
---
## 2. Suitability Assessment for Multi-Database Storage Layer
### 2.1 What Polyglot CAN Do for a Multi-DB Project
| Use Case | Polyglot Support | Maturity |
|---|---|---|
| **SQL dialect translation** | ✅ Core purpose; 32 dialects with 100% test pass rate | Mature |
| **SQL pretty-printing** | ✅ Built-in format with guard rails | Mature |
| **SQL syntax validation** | ✅ Line/column error positions, error codes | Mature |
| **Schema-aware validation** | ✅ Table/column/type checking with `ValidationSchema` | Moderate |
| **Column lineage tracing** | ✅ `get_column_lineage()` for data lineage | Moderate |
| **OpenLineage payloads** | ✅ `RunEvent` and `DatasetFacet` generation | Early but functional |
| **Query builder** | ✅ Fluent API for SELECT/INSERT/UPDATE/DELETE | Usable but not as rich as query-builder-first libraries |
| **AST diff** | ✅ ChangeDistiller-based structural diff | Functional |
| **Logical planning** | ✅ Basic DAG plan extraction | Early stage |
| **Query optimization** | ✅ Column qualification, predicate pushdown, join elimination | Moderate |
| **Custom dialect registration** | ✅ `CustomDialectBuilder` for runtime extension | Functional |
### 2.2 What Polyglot CANNOT Do for a Multi-DB Project
| Need | Polyglot Support | Alternative |
|---|---|---|
| **Execute queries** | ❌ No | Use sqlx, diesel, or sea-orm |
| **Connection pooling** | ❌ No | Use deadpool, bb8, or sqlx built-in |
| **Async I/O** | ❌ Synchronous only | Wrap in `spawn_blocking()` |
| **Type-safe query building** | ⚠️ Partial (builder API returns strings) | Use diesel or sea-orm for compile-time checks |
| **Schema migration management** | ❌ No | Use diesel migrations, sqlx migrations, or refinery |
| **Row mapping / deserialization** | ❌ No | Use sqlx `FromRow`, diesel `Queryable` |
| **Runtime type mapping** | ⚠️ Limited (DataType enum, no Rust type bridge) | Build your own layer |
| **Database-specific DDL generation** | ⚠️ Parses/generates DDL but no migration framework | Use as a building block |
| **Transaction management** | ❌ No | Use sqlx or diesel |
### 2.3 Integration Pattern: Polyglot as a SQL Dialect Layer
The most natural integration pattern for a multi-database storage layer:
```
┌──────────────────────────────────────────────┐
│ Application Logic │
├──────────────────────────────────────────────┤
│ Query Builder / ORM Layer │
│ (diesel / sea-orm / custom) │
├──────────────────────┬───────────────────────┤
│ │ │
│ Polyglot Layer │ Direct SQL │
│ (transpile, │ (no translation │
│ validate, │ needed) │
│ format) │ │
├──────────────────────┴───────────────────────┤
│ Database Driver Layer │
│ (sqlx / diesel / tungstenite) │
├──────────────────────────────────────────────┤
│ PostgreSQL │ MySQL │ SQLite │
└──────────────────────────────────────────────┘
```
In this pattern, Polyglot sits **above** the database drivers, translating SQL from a canonical dialect to the target database's dialect before execution. It does **not** replace the drivers.
---
## 3. Comparison with Other Rust SQL Libraries
### 3.1 Feature Comparison Matrix
| Feature | **Polyglot** | **Diesel** | **SQLx** | **SeaORM** | **sqlparser-rs** |
|---|---|---|---|---|---|
| **Primary Purpose** | SQL transpilation | ORM / query builder | Async DB driver | Async ORM | SQL parsing |
| **SQL Parsing** | ✅ Full AST (200+ node types) | ✅ DSL-based | ❌ No | ❌ No | ✅ Full AST |
| **SQL Generation** | ✅ Multi-dialect | ✅ Via DSL | ❌ No | ❌ No | ⚠️ Limited |
| **Cross-dialect Transpilation** | ✅ 32 dialects | ❌ No | ❌ No | ❌ No | ❌ No |
| **Query Builder** | ⚠️ Fluent, string-based | ✅ Type-safe DSL | ❌ No | ✅ Type-safe | ❌ No |
| **Async I/O** | ❌ No (sync only) | ❌ Diesel 1.x is sync | ✅ Native async | ✅ Native async | ❌ No |
| **Type-safe Queries** | ❌ No (runtime) | ✅ Compile-time | ❌ No | ✅ Compile-time | ❌ No |
| **Connection Pool** | ❌ No | ❌ No (Diesel 2.x via r2d2) | ✅ Built-in | ✅ Built-in | ❌ No |
| **Migration Support** | ❌ No | ✅ Built-in | ❌ No | ✅ Built-in | ❌ No |
| **Database Execution** | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No |
| **Schema Validation** | ✅ Via ValidationSchema | ✅ Compile-time | ❌ No | ⚠️ Limited | ❌ No |
| **Column Lineage** | ✅ Built-in | ❌ No | ❌ No | ❌ No | ❌ No |
| **AST Diff** | ✅ Built-in | ❌ No | ❌ No | ❌ No | ❌ No |
| **Dialects Supported** | 32 | 4 (PG, MySQL, SQLite, MSSQL) | N/A | N/A | 1 (ANSI SQL) |
| **License** | MIT | MIT/Apache-2.0 | MIT/Apache-2.0 | MIT | MIT/Apache-2.0 |
| **Maturity** | v0.4.4 (pre-1.0) | v2.2 (stable) | v0.8 (stable) | v1.1 (stable) | v0.49 (mature) |
### 3.2 Polyglot vs Diesel
| Aspect | Polyglot | Diesel |
|---|---|---|
| **Philosophy** | Parse any SQL → AST → generate any dialect | Type-safe DSL → SQL for specific databases |
| **Type Safety** | Runtime (string-based) | Compile-time (macro-based) |
| **Query Building** | `select(["col"]).from("t").where_(...)` → `Expression` AST | `schema::table::dsl::col.filter(...)` → SQL |
| **Dialect Breadth** | 32 dialects | 4 (PostgreSQL, MySQL, SQLite, MSSQL) |
| **Database Execution** | None (SQL text only) | Full CRUD with connection management |
| **Migrations** | None | Built-in migration framework |
| **When to use** | You need cross-dialect SQL translation, validation, lineage | You need type-safe queries with database execution |
**Verdict**: Polyglot and Diesel are **complementary**, not competing. Use Diesel for type-safe database interaction; use Polyglot when you need to translate SQL between dialects or analyze SQL without executing it.
### 3.3 Polyglot vs SQLx
| Aspect | Polyglot | SQLx |
|---|---|---|
| **Philosophy** | SQL manipulation without execution | Async database driver with compile-time query checking |
| **Async** | Synchronous only | Fully async |
| **Query Checking** | Runtime validation against schema | Compile-time `query!()` macro |
| **Database Support** | 32 dialects (parsing) | PostgreSQL, MySQL, SQLite (execution) |
| **When to use** | SQL transformation/analysis | Database interaction with async Rust |
**Verdict**: SQLx is for executing queries against databases. Polyglot is for transforming SQL text. They solve entirely different problems.
### 3.4 Polyglot vs SeaORM
| Aspect | Polyglot | SeaORM |
|---|---|---|
| **Philosophy** | SQL transpilation | Async ORM built on SQLx |
| **Async** | No | Yes |
| **Model Definition** | None | Entity models via macros |
| **Relationships** | None | Has-one, has-many, many-to-many |
| **When to use** | SQL dialect conversion | Database CRUD with relationships |
**Verdict**: Same as SQLx — complementary, not competing.
### 3.5 Polyglot vs sqlparser-rs
| Aspect | Polyglot | sqlparser-rs |
|---|---|---|
| **Parsing** | ✅ Full (200+ node types) | ✅ Full (ANSI SQL + some dialects) |
| **Generation** | ✅ Multi-dialect generation | ⚠️ Limited round-trip |
| **Transpilation** | ✅ Cross-dialect transforms | ❌ No |
| **Dialects** | 32 | Primarily ANSI SQL |
| **Validation** | ✅ With error positions | ❌ Parse errors only |
| **Builder** | ✅ Fluent API | ❌ No |
| **Lineage** | ✅ Built-in | ❌ No |
| **Diff** | ✅ Built-in | ❌ No |
| **Maturity** | v0.4.4 | v0.49 (more established) |
**Verdict**: sqlparser-rs is a mature parser for ANSI SQL. Polyglot offers significantly more: transpilation, 32 dialects, validation, lineage, diff, and a builder API. If you need dialect translation, Polyglot is the clear choice. If you only need ANSI SQL parsing and don't need generation/transpilation, sqlparser-rs may suffice with less overhead.
### 3.6 Polyglot vs Python sqlglot
| Aspect | Polyglot (Rust) | sqlglot (Python) |
|---|---|---|
| **Performance** | 8–19× faster (transpile), ~86× faster (generate) | Baseline |
| **Language** | Rust | Python |
| **Feature Parity** | ~95% of sqlglot's transpilation | Full feature set |
| **Optimizer** | Column qualification, predicate pushdown (moderate) | Full optimizer (column pruning, join elimination, etc.) |
| **Execution** | ❌ No | ⚠️ Limited (can execute against some engines) |
| **Test Compatibility** | 10,220+ sqlglot fixture cases at 100% | Original test suite |
| **Deployment** | Native binary / WASM / Python / Go | Python package |
**Verdict**: Polyglot is the performance-oriented port of sqlglot. It covers the core transpilation use case at near-full feature parity. The Python sqlglot has a more mature optimizer and some execution capabilities, but Polyglot is catching up rapidly (0.4.x adds lineage, OpenLineage, schema validation, and more).
---
## 4. Limitations and Gotchas
### 4.1 Current Limitations
| Limitation | Impact | Mitigation |
|---|---|---|
| **Pre-1.0 API** | Breaking changes possible between minor versions | Pin exact version in Cargo.toml |
| **No query execution** | Cannot run SQL against databases | Use alongside sqlx/diesel |
| **No async** | Blocking in async contexts | Wrap in `spawn_blocking()` |
| **No migration framework** | Cannot manage schema evolution | Use diesel migrations or refinery |
| **No Rust type mapping** | `DataType` enum doesn't map to Rust types | Build your own type bridge |
| **Builder returns Expression** | Builder doesn't produce type-safe queries | Accept runtime nature; pair with runtime validation |
| **Optimizer is early** | Limited optimization passes vs Python sqlglot | Most useful passes exist (qualify_columns, pushdown_predicates) |
| **WASM lacks `stacker`** | Deeply nested SQL may overflow stack in browser | Set format guard limits; consider web workers |
| **Custom dialects are global** | `CustomDialectBuilder` uses a global `RwLock` registry | Fine for most apps; not ideal for per-request isolation |
| **No prepared statement support** | Cannot generate `?` placeholders for parameterized queries | Build queries as strings; use sqlx for parameterization |
### 4.2 Gotchas
1. **`Dialect::get()` creates a new instance each call**: The `Dialect` struct bundles tokenizer + generator config + transformer. For hot loops, cache the `Dialect` instance rather than calling `Dialect::get()` repeatedly. (The overhead is minimal but non-zero.)
2. **Transpilation is not always invertible**: Some dialects have features that don't exist in others (e.g., BigQuery's `QUALIFY`, PostgreSQL's `ILIKE`, TSQL's `TOP`). Transpiling `A → B` and then `B → A` may lose information.
3. **Function transformation depth**: The transform pipeline processes per-node bottom-up. Some transformations require multi-pass processing (handled by `preprocess()`), but edge cases may require manual intervention.
4. **AST is not a stable serialization format**: The `Expression` enum and its inner structs may change between versions. If you serialize ASTs to JSON, expect breaking changes across minor versions.
5. **Feature flags are cumulative**: `transpile` implies `generate`, `openlineage` implies `semantic`, etc. For minimal builds, use `default-features = false` and select only what you need.
6. **Global custom dialect registry**: Custom dialects registered via `CustomDialectBuilder::register()` are stored in a global `RwLock<HashMap>`. This means they persist for the lifetime of the process and are visible across threads. Call `unregister_custom_dialect()` to remove them.
7. **Parser is permissive**: The parser accepts many SQL constructs that some databases reject. Validation (via `validate()` or `validate_with_schema()`) can catch some issues, but it's not a substitute for database-level error checking.
8. **No `?` placeholder generation**: Polyglot doesn't generate parameterized query placeholders. For prepared statements, you'll need to handle parameter binding yourself with your database driver.
9. **Schema validation requires manual schema definition**: The `ValidationSchema` struct must be populated manually — there's no automatic schema introspection from a live database.
---
## 5. Production-Readiness Assessment
### 5.1 Strengths
| Area | Rating | Notes |
|---|---|---|
| **Transpilation accuracy** | ⭐⭐⭐⭐⭐ | 10,220+ fixture cases at 100% pass rate |
| **Performance** | ⭐⭐⭐⭐⭐ | 8–19× faster than Python sqlglot |
| **Dialect coverage** | ⭐⭐⭐⭐⭐ | 32 dialects covering all major databases |
| **API ergonomics** | ⭐⭐⭐⭐ | Clean public API; builder is pleasant |
| **Error reporting** | ⭐⭐⭐⭐ | Line/column/byte-offset positions |
| **WASM support** | ⭐⭐⭐⭐ | Full feature set in browser |
| **Multi-language bindings** | ⭐⭐⭐⭐⭐ | Rust, TypeScript, Python, Go, C FFI |
| **Documentation** | ⭐⭐⭐ | Rust API docs exist; could use more guides |
| **Test coverage** | ⭐⭐⭐⭐⭐ | 18,745 test cases |
| **Fuzzing** | ⭐⭐⭐⭐ | Supported via `cargo fuzz` |
### 5.2 Risks
| Risk | Severity | Mitigation |
|---|---|---|
| **Pre-1.0 breaking changes** | Medium | Pin version; monitor CHANGELOG |
| **Single maintainer** | Medium | Code is well-structured; community could fork |
| **Limited optimizer** | Low | Core passes exist; Python sqlglot is reference |
| **No query execution** | Low (by design) | Combine with sqlx/diesel |
| **WASM stack limits** | Low | Set guard rails; use web workers |
### 5.3 Overall Assessment
**Polyglot is production-viable for SQL transpilation and analysis tasks**, with caveats:
- ✅ **Use for**: SQL dialect translation, SQL linting/validation, column lineage, pretty-printing, AST analysis, cross-database query migration
- ⚠️ **Use with caution for**: Query building (no type safety), optimization (partial coverage)
- ❌ **Don't use for**: Database execution, connection management, migrations, type-safe queries
For a multi-database storage layer, the recommended pattern is:
```
Application → Polyglot (transpile SQL to target dialect) → sqlx/diesel (execute)
```
---
## 6. Recommendation
### When to Adopt Polyglot
1. **You need to support multiple database backends with different SQL dialects** and want to write queries once in a canonical dialect, then transpile to the target at runtime.
2. **You need SQL validation or analysis** (lineage, schema checking) without executing queries.
3. **You need SQL pretty-printing or formatting** with configurable guard rails.
4. **You need column lineage tracking** for data governance or OpenLineage integration.
5. **You need to parse and analyze SQL** in a Rust/WASM/Python/Go context without connecting to a database.
### When NOT to Adopt Polyglot
1. **You need type-safe query building** — use Diesel or SeaORM instead.
2. **You need async database execution** — use SQLx or SeaORM instead.
3. **You need schema migrations** — use Diesel migrations, sqlx migrations, or Refinery instead.
4. **You only need PostgreSQL** (or a single dialect) — a simpler parser may suffice.
5. **You need Rust type → SQL type mapping** — Polyglot doesn't provide this.
### Suggested Adoption Strategy
For a multi-database storage layer:
1. **Use Polyglot for SQL transpilation**: Write queries in a canonical dialect (e.g., PostgreSQL-compatible), transpile to the target dialect at runtime.
2. **Use SQLx for database execution**: Handle connections, pooling, and async I/O.
3. **Use Polyglot for validation**: Validate user-provided SQL before execution.
4. **Use Polyglot for lineage**: Trace column flow for data governance.
5. **Build a thin integration layer** that combines Polyglot's transpilation with SQLx's execution.
---
## References
- <https://github.com/tobilg/polyglot> — Main repository
- <https://crates.io/crates/polyglot-sql> — Rust crate (v0.4.4)
- <https://docs.rs/polyglot-sql/latest/polyglot_sql/> — Rust API docs
- <https://github.com/tobymao/sqlglot> — Python inspiration
- <https://lib.rs/crates/polyglot-sql> — Package metadata
- Local source: `/workspace/polyglot/`

765
docs/research/references/rustfs/rustfs-events-select.md
View File

@@ -1,765 +0,0 @@
# RustFS Event Notification System & S3 Select Reference
> **Companion document**: This extends [rustfs-reference.md](./rustfs-reference.md) which covers auth, architecture, and credential mapping. This document focuses on the **event notification system** and **S3 Select** feature.
**Date**: 2026-06-08
**RustFS version**: Based on source at `/workspace/rustfs/` (commit-level snapshot)
**Purpose**: Evaluate rustfs event notification and S3 Select for alknet integration
---
## Table of Contents
1. [Event Notification System](#1-event-notification-system)
2. [Event Types & Structure](#2-event-types--structure)
3. [Notification Targets](#3-notification-targets)
4. [Configuration & Rule Engine](#4-configuration--rule-engine)
5. [Pipeline & Delivery](#5-pipeline--delivery)
6. [Live Event Stream](#6-live-event-stream)
7. [S3 Select](#7-s3-select)
8. [Mapping to alknet](#8-mapping-to-alknet)
9. [References](#9-references)
---
## 1. Event Notification System
### 1.1 Architecture Overview
RustFS implements a full S3-compatible bucket notification system. The architecture follows a layered pattern:
```
┌──────────────────────────────────────────────────────────┐
│ S3 API Layer │
│ (PutObject, DeleteObject, CopyObject, etc.) │
└─────────────┬────────────────────────────────────────────┘
│ emits EventArgs
▼
┌──────────────────────────────────────────────────────────┐
│ ECStore (event_notification.rs) │
│ - send_event() hook (global OnceLock dispatch) │
│ - registers dispatch callback during init │
└─────────────┬────────────────────────────────────────────┘
│ converts EventArgs → NotifyEventArgs
▼
┌──────────────────────────────────────────────────────────┐
│ rustfs_notify (NotificationSystem) │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ NotifyPipeline│──▶│ NotifyRuleEngine│─▶│ EventNotifier │ │
│ │ (broadcast │ │ (match rules) │ │ (send to │ │
│ │ + history) │ │ │ │ targets) │ │
│ └──────────────┘ └──────────────┘ └──────┬────────┘ │
│ │ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────▼────────┐ │
│ │BucketConfigM │ │ NotifyConfigM │ │ TargetList │ │
│ │ anager │ │ anager │ │ (Webhook, │ │
│ └──────────────┘ └──────────────┘ │ Kafka, AMQP, │ │
│ │ NATS, Redis, │ │
│ │ MQTT, MySQL, │ │
│ │ Postgres, │ │
│ │ Pulsar) │ │
│ └───────────────┘ │
└──────────────────────────────────────────────────────────┘
```
### 1.2 Key Crates
| Crate | Purpose |
|-------|---------|
| `rustfs_notify` | Core notification orchestration: `Event`, `EventArgs`, `EventNotifier`, `NotifyPipeline`, `NotificationSystem`, rule engine, bucket config management |
| `rustfs_targets` | Target implementations (Webhook, Kafka, AMQP, NATS, Redis, MQTT, MySQL, PostgreSQL, Pulsar) + `Target` trait, `QueueStore`, TLS hot-reload |
| `rustfs_s3_types` | `EventName` enum with all S3 event type definitions, serialization, mask/bitfield support |
| `rustfs_ecstore` | Storage layer; `event_notification.rs` provides the dispatch hook that bridges ecstore events to the notify system |
| `rustfs_config` | Configuration for each target type (Env vars, KVS parsing, subsystem names) |
### 1.3 Initialization Flow
1. `rustfs/server/event.rs::init_event_notifier()` runs at startup
2. If notify module is enabled (`RUSTFS_NOTIFY_ENABLE=true`), it calls `rustfs_notify::initialize(config)` which:
- Creates a `NotificationSystem` with `EventNotifier`, `TargetRegistry`, and config
- Loads all target configurations from the config store
- Initializes each target (connects, health-checks, starts stream replay workers)
3. An ECStore dispatch hook is installed via `register_event_dispatch_hook()` which:
- Converts `ecstore::EventArgs` → `notify::EventArgs`
- Parses `EventName` from string
- Spawns an async task to call `notifier_global::notify(args)`
### 1.4 Module Toggle
The notification system respects a module enable/disable flag:
- Environment variable: `RUSTFS_NOTIFY_ENABLE` (default: `DEFAULT_NOTIFY_ENABLE`)
- When disabled, only the **live event stream** is initialized (no targets are loaded)
- This allows in-process event subscription without external delivery
---
## 2. Event Types & Structure
### 2.1 EventName Enum
Defined in `rustfs_s3_types::EventName`. All S3-standard event types plus RustFS extensions:
| Category | Events |
|----------|--------|
| **ObjectAccessed** | `s3:ObjectAccessed:Get`, `s3:ObjectAccessed:Head`, `s3:ObjectAccessed:GetRetention`, `s3:ObjectAccessed:GetLegalHold`, `s3:ObjectAccessed:Attributes` |
| **ObjectCreated** | `s3:ObjectCreated:Put`, `s3:ObjectCreated:Post`, `s3:ObjectCreated:Copy`, `s3:ObjectCreated:CompleteMultipartUpload`, `s3:ObjectCreated:PutRetention`, `s3:ObjectCreated:PutLegalHold` |
| **ObjectRemoved** | `s3:ObjectRemoved:Delete`, `s3:ObjectRemoved:DeleteMarkerCreated`, `s3:ObjectRemoved:DeleteAllVersions`, `s3:ObjectRemoved:NoOP` |
| **ObjectTagging** | `s3:ObjectTagging:Put`, `s3:ObjectTagging:Delete` |
| **ObjectAcl** | `s3:ObjectAcl:Put` |
| **ObjectReplication** | `s3:Replication:OperationFailedReplication`, `s3:Replication:OperationCompletedReplication`, `s3:Replication:OperationMissedThreshold`, `s3:Replication:OperationReplicatedAfterThreshold`, `s3:Replication:OperationNotTracked` |
| **ObjectRestore** | `s3:ObjectRestore:Post`, `s3:ObjectRestore:Completed` |
| **ObjectTransition** | `s3:ObjectTransition:Failed`, `s3:ObjectTransition:Complete` |
| **Lifecycle** | `s3:LifecycleExpiration:Delete`, `s3:LifecycleExpiration:DeleteMarkerCreated`, `s3:LifecycleDelMarkerExpiration:Delete`, `s3:LifecycleTransition` |
| **Bucket** | `s3:BucketCreated:*`, `s3:BucketRemoved:*` |
| **Scanner** | `s3:Scanner:ManyVersions`, `s3:Scanner:LargeVersions`, `s3:Scanner:BigPrefix` |
| **IntelligentTiering** | `s3:IntelligentTiering` |
| **Compound (wildcard)** | `s3:ObjectAccessed:*`, `s3:ObjectCreated:*`, `s3:ObjectRemoved:*`, `s3:ObjectTagging:*`, `s3:Replication:*`, `s3:ObjectRestore:*`, `s3:LifecycleExpiration:*`, `s3:ObjectTransition:*`, `s3:Scanner:*`, `Everything` |
| **Internal** | `ObjectRemovedAbortMultipartUpload`, `ObjectCreatedCreateMultipartUpload`, `ObjectRemovedDeleteObjects` |
### 2.2 Event Schema Versioning
The `event_schema_version` function returns different versions based on event type:
| Version | Events |
|---------|--------|
| `2.1` | ObjectCreated/Removed/Accessed base events |
| `2.2` | Replication events |
| `2.3` | Tagging, ACL, Restore, Lifecycle, IntelligentTiering events |
### 2.3 Event Record Structure (`rustfs_notify::Event`)
```rust
pub struct Event {
pub event_version: String, // e.g., "2.1", "2.2", "2.3"
pub event_source: String, // "rustfs:s3"
pub aws_region: String,
pub event_time: DateTime<Utc>,
pub event_name: EventName,
pub user_identity: Identity, // { principal_id: String }
pub request_parameters: HashMap<String, String>,
pub response_elements: HashMap<String, String>,
pub s3: Metadata, // See below
pub glacier_event_data: Option<GlacierEventData>,
pub source: Source, // { host, port, user_agent }
}
pub struct Metadata {
pub schema_version: String, // "1.0"
pub configuration_id: String,
pub bucket: Bucket, // { name, owner_identity, arn }
pub object: Object, // See below
}
pub struct Object {
pub key: String, // URL-encoded object key
pub size: Option<i64>,
pub e_tag: Option<String>,
pub content_type: Option<String>,
pub user_metadata: Option<HashMap<String, String>>,
pub version_id: Option<String>,
pub sequencer: String, // Monotonic event sequence ID
}
```
- The `key` field is URL-encoded (form-urlencoded)
- `sequencer` is derived from `ObjectInfo.mod_time` nanosecond timestamp, ensuring ordering
- `user_metadata` filters out keys starting with `x-amz-meta-internal-`
- For removed events, `size`, `e_tag`, `content_type`, and `user_metadata` are omitted
### 2.4 EventArgs Builder
Events are constructed via `EventArgsBuilder`:
```rust
let args = EventArgsBuilder::new(EventName::ObjectCreatedPut, "my-bucket", object_info)
.host("10.0.0.1")
.port(9000)
.user_agent("alknet-storage/1.0")
.req_param("principalId", "user-123")
.version_id("v2")
.build();
let event = Event::new(args);
```
The builder pattern ensures all required fields are provided and allows optional fields.
---
## 3. Notification Targets
### 3.1 Target Trait
All targets implement `rustfs_targets::Target<E>`:
```rust
#[async_trait]
pub trait Target<E>: Send + Sync + 'static
where E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned
{
fn id(&self) -> TargetID;
fn name(&self) -> String;
async fn is_active(&self) -> Result<bool, TargetError>;
async fn save(&self, event: Arc<EntityTarget<E>>) -> Result<(), TargetError>;
async fn send_raw_from_store(&self, key: Key, body: Vec<u8>, meta: QueuedPayloadMeta) -> Result<(), TargetError>;
async fn send_from_store(&self, key: Key) -> Result<(), TargetError>;
async fn close(&self) -> Result<(), TargetError>;
fn store(&self) -> Option<&(dyn Store<QueuedPayload, ...>)>;
fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync>;
async fn init(&self) -> Result<(), TargetError>;
fn is_enabled(&self) -> bool;
fn delivery_snapshot(&self) -> TargetDeliverySnapshot;
fn record_final_failure(&self);
}
```
### 3.2 Supported Targets
| Target | Crate Module | Protocol | Queue Store | TLS/mTLS | SASL | Notes |
|--------|-------------|----------|-------------|----------|------|-------|
| **Webhook** | `targets::webhook` | HTTP POST | Yes (file) | Yes (CA, client cert, skip_verify) | Bearer token | Health check via HEAD to `/`; TLS hot-reload |
| **Kafka** | `targets::kafka` | Kafka Produce | Yes (file) | Yes (CA, client cert) | PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 | Uses `rustfs_kafka_async`; acknowledgments configurable (-1, 0, 1) |
| **AMQP** | `targets::amqp` | AMQP 0-9-1 | Yes (file) | Yes (CA, client cert via amqps://) | Username/password (in URL or config) | Uses `lapin`; publisher confirms; persistent delivery mode |
| **NATS** | `targets::nats` | NATS Publish | Yes (file) | Yes (CA, client cert) | Token, username/password, credentials file | Subject-based routing |
| **Redis** | `targets::redis` | Redis Pub/Sub | Yes (file) | Yes (CA, client cert, insecure) | Password | Channel publish; connection pooling |
| **MQTT** | `targets::mqtt` | MQTT v5 | Yes (file) | Yes (CA, client cert) | Username/password | Uses `rumqttc`; QoS 0/1; WebSocket path allowlist |
| **MySQL** | `targets::mysql` | MySQL INSERT | Yes (file) | Yes (CA, client cert) | Username/password | Namespace or access format; connection pooling |
| **PostgreSQL** | `targets::postgres` | PostgreSQL INSERT/UPSERT | Yes (file) | Yes (CA, client cert) | Username/password (DSN) | Namespace (UPSERT) or access (append) format; `deadpool-postgres` pooling |
| **Pulsar** | `targets::pulsar` | Pulsar Produce | Yes (file) | Yes (CA, client cert) | Token, OAuth2 | Topic-based; persistent or non-persistent |
**Note**: Elasticsearch is listed as a subsystem constant (`notify_elasticsearch`) but marked `#[allow(dead_code)]`, indicating it's planned but not yet implemented.
### 3.3 Target Identification (ARN)
Each target has a `TargetID` (format: `ID:Name`, e.g., `1:webhook`) and an `ARN` (format: `arn:rustfs:sqs:{region}:{id}:{name}`, e.g., `arn:rustfs:sqs:us-east-1:1:webhook`).
Default partition: `rustfs`, default service: `sqs`.
### 3.4 Queue Store (Persistent Delivery)
Targets that have a `queue_dir` configured use a persistent store for at-least-once delivery:
- Events are first persisted to the queue store, then sent
- If the target is unreachable, events remain in the store and are replayed when connectivity recovers
- Queue store format: `RQP1` magic + metadata length (LE u32) + JSON metadata + raw body
- `QueuedPayload` structure includes: event_name, bucket_name, object_name, content_type, queued_at_unix_ms, payload_len
- Extension: `notify_store` (`.nqs`) for notification events, `audit_store` for audit logs
### 3.5 Delivery Payload Format (`TargetLog`)
```rust
// Serialized as JSON when delivering to targets
struct TargetLog {
event_name: EventName,
key: String, // "{bucket}/{decoded_object_name}"
records: Vec<E>, // For AMQP/NATS: includes full EntityTarget records
// For others: includes serialized Event data
}
```
For AMQP and NATS targets, `build_queued_payload_with_records()` is used, which includes cloned `EntityTarget` records. For other targets, `build_queued_payload()` serializes just the event data.
### 3.6 Concurrency Controls
| Parameter | Default | Env Var |
|-----------|---------|---------|
| Target stream concurrency | 20 | `RUSTFS_NOTIFY_TARGET_STREAM_CONCURRENCY` |
| Send concurrency (inflight limit) | 64 | `RUSTFS_NOTIFY_SEND_CONCURRENCY` |
### 3.7 TLS Hot-Reload
All targets that support TLS (webhook, Kafka, AMQP, NATS, MySQL, PostgreSQL, MQTT) implement `ReloadableTargetTls`:
- A background coordinator polls TLS files for changes
- When fingerprint changes are detected, new material (HTTP client, producer, connection) is built
- Applied via `apply_tls_material()` without requiring a restart
- Supports CA certificates, client certificates, and client keys
---
## 4. Configuration & Rule Engine
### 4.1 Bucket Notification Configuration (XML)
Configuration follows the S3 `NotificationConfiguration` XML schema:
```xml
<NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<QueueConfiguration>
<Id>my-notification</Id>
<Queue>arn:rustfs:sqs:us-east-1:1:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:Delete</Event>
<Filter>
<S3Key>
<FilterRule>
<Name>prefix</Name>
<Value>uploads/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.csv</Value>
</FilterRule>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>
```
The XML is parsed via `quick_xml` into `NotificationConfiguration` → `QueueConfig` → validated → converted to `BucketNotificationConfig` → `RulesMap`.
Key validation rules:
- Lambda and Topic configurations are **not supported** (return `UnsupportedConfiguration` error)
- Only `QueueConfiguration` is supported (maps to all target types, not just SQS)
- One prefix filter and one suffix filter maximum
- Filter values: ≤1024 chars, no `.` or `..` segments, no `\`, valid UTF-8
- No duplicate event names within a queue config
- ARN must exist in the configured target list
### 4.2 RulesMap
`RulesMap` maps `EventName` → `PatternRules` → `TargetIdSet`:
- Compound events (like `ObjectCreatedAll`) are **expanded** into specific events on insertion
- Pattern matching: prefix/suffix wildcards (e.g., `uploads/*.csv`)
- URL-encoded keys are matched against both encoded and decoded patterns
- Bitmask-based fast path: `total_events_mask` enables O(1) `has_subscriber()` checks
### 4.3 Dynamically Reconfigurable
- `NotificationSystem::set_target_config()` — add/update a target
- `NotificationSystem::remove_target_config()` — remove a target
- `NotificationSystem::load_bucket_notification_config()` — load per-bucket rules
- `NotificationSystem::remove_bucket_notification_config()` — remove per-bucket rules
- `NotificationSystem::reload_config()` — reload from a new `Config` object
- All changes trigger automatic re-initialization of affected targets
---
## 5. Pipeline & Delivery
### 5.1 Event Flow
```
ECStore operation
↓
ecstore::event_notification::send_event(EventArgs)
↓ (OnceLock dispatch hook)
convert EventArgs → notify::EventArgs
↓ spawn
notifier_global::notify(EventArgs)
↓
NotificationSystem::send_event(Arc<Event>)
↓
NotifyPipeline::send_event()
├── LiveEventHistory::record() (in-memory, last 1024 events)
├── broadcast::send() (tokio broadcast channel, capacity 1024)
└── EventNotifier::send() (async, rule-matched delivery)
├── RuleEngine::match_targets(bucket, event_name, object_key)
└── For each matched target:
├── EntityTarget construction
├── If queue_store: persist then async send
└── If no queue_store: immediate async send
```
### 5.2 Live Event Stream
The `NotifyPipeline` provides an in-process event stream via `tokio::sync::broadcast`:
```rust
// Subscribe to live events
let rx = system.subscribe_live_events();
// Check if there are live listeners
system.has_live_listeners();
// Get recent events since a sequence number
system.recent_live_events_since(after_sequence, limit) → LiveEventBatch
```
- Broadcast channel capacity: 1024
- `LiveEventHistory` stores last 1024 events with monotonic sequence numbers
- `LiveEventBatch` includes `events: Vec<Arc<Event>>`, `next_sequence: u64`, `truncated: bool`
### 5.3 Metrics
`NotificationMetrics` tracks:
- Processing count (in-flight)
- Processed count (completed)
- Failed count
- Skipped count (no matching targets)
Per-target `TargetDeliverySnapshot`:
- `total_messages`
- `failed_messages`
- `queue_length`
---
## 6. Live Event Stream
### 6.1 In-Process Subscription
The live event stream is useful for alknet because it provides a **push-based** event feed without requiring external message brokers:
```rust
// This can be used from within the same process
let mut rx = notification_system.subscribe_live_events();
while let Ok(event) = rx.recv().await {
// event: Arc<Event> — full S3 event record
println!("Event: {} on {}/{}", event.event_name, event.s3.bucket.name, event.s3.object.key);
}
```
### 6.2 Event History Replay
The `LiveEventHistory` supports catch-up subscriptions:
```rust
// Get events since sequence number 42
let batch = system.recent_live_events_since(42, 100).await;
// batch.next_sequence → next sequence to request
// batch.truncated → whether there are more events
// batch.events → Vec<Arc<Event>>
```
---
## 7. S3 Select
### 7.1 Architecture Overview
RustFS implements S3 Select using **Apache DataFusion** as the SQL engine:
```
SelectObjectContentRequest
↓ validation (expression type, input/output format, scan range)
↓ preflight (get object info, validate SSE headers)
↓ create EcObjectStore (DataFusion ObjectStore adapter)
↓ get_global_db(input) → QueryDispatcher
↓ Query::new(Context, expression) → execute
↓ DataFusion SQL parser → logical plan → optimized → physical plan → RecordBatch stream
↓ SelectOutputEncoder → CSV or JSON → chunked (128KB) → event stream
```
### 7.2 Key Crates
| Crate | Purpose |
|-------|---------|
| `rustfs_s3select_api` | Query error types, `Context`, `Query`, `QueryResult`, `DatabaseManagerSystem` trait, object store |
| `rustfs_s3select_query` | SQL implementation: parser, analyzer, optimizer, function manager, execution, dispatcher |
### 7.3 SQL Engine
- **Parser**: Custom `RustFsDialect` + `ExtParser` extending DataFusion's SQL parser
- **Supports**: Single SELECT statements only (multi-statement is rejected)
- **Optimizer**: `CascadeOptimizerBuilder` (DataFusion's default rule set)
- **Scheduler**: `LocalScheduler` (single-node execution)
- **Functions**: All of DataFusion's built-in scalar, aggregate, and window functions
### 7.4 Input Formats
| Format | Support | Notes |
|--------|---------|-------|
| **CSV** | ✅ Full | `FileHeaderInfo` (NONE, USE, IGNORE), custom delimiters, quote chars, comment chars, record delimiters |
| **JSON (LINES)** | ✅ Full | NDJSON line-by-line streaming |
| **JSON (DOCUMENT)** | ✅ Limited | Max 128 MiB (OOM guard); no scan range support |
| **Parquet** | ✅ Full | Columnar format |
| **Compression** | ❌ Not supported | Only `NONE` compression currently accepted |
### 7.5 Output Formats
| Format | Options |
|--------|---------|
| **CSV** | Custom field delimiter, quote character, quote escape, record delimiter, quote fields (ALWAYS/ASNEEDED) |
| **JSON** | Line-delimited (NDJSON); custom record delimiter |
### 7.6 Expression Limitations
- Max expression size: 256 KiB (`MAX_SELECT_EXPRESSION_BYTES`)
- Expression type must be `SQL`
- No `AllowQuotedRecordDelimiter` support for CSV
- Scan ranges:
- CSV: supported
- JSON LINES: supported
- JSON DOCUMENT: **not supported**
- Parquet: supported
- Range must be valid (start < end, start < object size)
### 7.7 Object Store Integration
`EcObjectStore` implements DataFusion's `ObjectStore` trait, adapting rustfs's ECStore for query execution:
- Handles `GET` with optional byte ranges (scan range)
- JSON DOCUMENT mode: entire file buffered for DOM parsing, then flattened to NDJSON
- JSON sub-path extraction: `FROM s3object.some.path` navigates to the key before flattening
- Respects SSE-C headers for encrypted objects
### 7.8 Streaming Response
Results are streamed as S3 event types:
1. `Cont` event (continuation marker)
2. `Records` events (128KB chunks)
3. `Progress` events (if `RequestProgress.Enabled=true`) — currently only `BytesReturned` populated
4. `Stats` event (final)
5. `End` event
### 7.9 Error Mapping
| QueryError | S3 Error |
|-----------|----------|
| `Parser` | `ParseSelectFailure` (400) |
| `MultiStatement` | `UnsupportedSqlStructure` |
| `NotImplemented` | `NotImplemented` |
| `Datafusion` (scan range) | `InvalidRequestParameter` |
| `Datafusion` (missing binding) | `EvaluatorBindingDoesNotExist` |
| `Datafusion` (other) | `UnsupportedSqlOperation` |
| `StoreError` (bucket not found) | `NoSuchBucket` |
| `StoreError` (object not found) | `NoSuchKey` |
| `StoreError` (other) | `InternalError` |
---
## 8. Mapping to alknet
### 8.1 rustfs Events → alknet Integration Events
rustfs events are **integration events from rustfs's perspective** and remain **integration events from alknet's perspective**. This is the correct cross-boundary classification per ADR-032.
#### Event Projection: `rustfs::BucketNotificationEvent` → `alknet::EventEnvelope`
Suggested namespace and operation mapping:
| rustfs EventName | alknet Namespace | alknet Operation |
|------------------|-----------------|-----------------|
| `s3:ObjectCreated:Put` | `storage.object` | `created.put` |
| `s3:ObjectCreated:Post` | `storage.object` | `created.post` |
| `s3:ObjectCreated:Copy` | `storage.object` | `created.copy` |
| `s3:ObjectCreated:CompleteMultipartUpload` | `storage.object` | `created.multipart-complete` |
| `s3:ObjectRemoved:Delete` | `storage.object` | `removed.delete` |
| `s3:ObjectRemoved:DeleteMarkerCreated` | `storage.object` | `removed.delete-marker-created` |
| `s3:ObjectAccessed:Get` | `storage.object` | `accessed.get` |
| `s3:ObjectAccessed:Head` | `storage.object` | `accessed.head` |
| `s3:BucketCreated:*` | `storage.bucket` | `created` |
| `s3:BucketRemoved:*` | `storage.bucket` | `removed` |
The full `Event` record from rustfs should be preserved in the `EventEnvelope.payload` field for traceability, while a normalized `metadata` extraction provides fast-path access:
```rust
// Pseudocode for mapping
fn project_rustfs_event(event: &rustfs_notify::Event) -> alknet::EventEnvelope {
let namespace = if event.event_name == EventName::BucketCreated || event.event_name == EventName::BucketRemoved {
"storage.bucket"
} else {
"storage.object"
};
let operation = event.event_name.as_str() // "s3:ObjectCreated:Put"
.strip_prefix("s3:") // "ObjectCreated:Put"
.unwrap_or("unknown")
.to_lowercase()
.replace(':',, ".");
EventEnvelope {
id: uuid::Uuid::new_v4(),
namespace: namespace.into(),
operation: operation.into(), // e.g., "objectcreated.put"
timestamp: event.event_time,
source: "rustfs".into(),
metadata: json!({
"bucket": event.s3.bucket.name,
"key": event.s3.object.key,
"size": event.s3.object.size,
"eTag": event.s3.object.e_tag,
"versionId": event.s3.object.version_id,
"sequencer": event.s3.object.sequencer,
"principalId": event.user_identity.principal_id,
}),
payload: serde_json::to_value(event).ok(),
}
}
```
### 8.2 Subscription Architecture
#### Option A: In-Process Live Event Stream (Recommended)
Since alknet and rustfs share the same process, alknet can subscribe to the live event stream directly:
```rust
// In alknet's initialization
let notification_system = rustfs_notify::notification_system().unwrap();
let mut event_rx = notification_system.subscribe_live_events();
// In alknet's event loop
tokio::spawn(async move {
while let Ok(event) = event_rx.recv().await {
let envelope = project_rustfs_event(&event);
alknet::honker::publish(envelope).await;
}
});
```
**Advantages**:
- Zero-latency, zero-serialization overhead
- No network hop
- Direct access to `Arc<Event>` in-process
- alknet's Honker streams get events immediately
**Considerations**:
- `has_live_listeners()` can be checked before performing expensive event construction
- The broadcast channel capacity is 1024; slow consumers will miss events (acceptable for integration events)
- `recent_live_events_since()` allows catch-up after reconnection
#### Option B: External Target via Webhook/Kafka/etc.
If alknet runs as a separate process, configure a webhook or Kafka target pointing to alknet's event ingestion endpoint:
```json
{
"notify_webhook": {
"1": {
"enable": true,
"endpoint": "https://alknet.internal/events/rustfs",
"auth_token": "Bearer alknet-secret"
}
}
}
```
**Advantages**:
- Decoupled deployment
- RustFS's queue store provides at-least-once delivery
**Considerations**:
- Network latency and serialization overhead
- Need to handle deduplication (at-least-once means possible duplicates)
- Queue store provides durability if alknet is temporarily unavailable
#### Option C: Hybrid — Live Stream + Webhook Fallback
For maximum reliability:
1. In-process live stream for low-latency event propagation
2. Webhook/Kafka target as a fallback for events missed during restarts
3. Use `sequentor` ordering to detect gaps
### 8.3 S3 Select → alknet Operations
S3 Select can be exposed as an alknet operation:
| alknet Operation | Description |
|-----------------|-------------|
| `storage.select` | Run an S3 Select SQL query on an object |
| `storage.select-status` | Check Select availability (optional) |
```rust
// Example alknet call protocol operation
fn handle_storage_select(params: StorageSelectParams) -> Result<StorageSelectResult, Error> {
// 1. Construct SelectObjectContentInput
// 2. Call existing rustfs SelectObjectContent handler
// 3. Stream results back through alknet call protocol
}
```
#### Use Cases for alknet
1. **Metagraph Queries**: Query stored metagraph JSON/CSV objects without downloading them entirely
```sql
SELECT s.name, s.version FROM S3Object s WHERE s.type = 'service'
```
2. **Log Analytics**: Query structured log data stored in S3
```sql
SELECT COUNT(*) as cnt, s.level FROM S3Object s WHERE s.timestamp > '2026-01-01' GROUP BY s.level
```
3. **Ad-hoc Data Exploration**: Quick data inspection without full downloads
```sql
SELECT * FROM S3Object s LIMIT 100
```
4. **Aggregation Pipelines**: Pre-process data before moving to alknet's internal stores
### 8.4 ADR-032 Implications: Cross-Boundary Event Flow
Per ADR-032, rustfs events are **integration events** — they represent facts about state changes that have already happened in the storage system boundary. When alknet consumes them:
```
┌─────────────┐ ┌─────────────┐
│ rustfs │ │ alknet │
│ (bounded │ integration │ (bounded │
│ context) │───── event ─────────▶│ context) │
│ │ │ │
│ S3 Object │ EventEnvelope │ Honker │
│ Created/ │ namespace: │ Stream │
│ Removed/ │ "storage.object" │ Subscriber │
│ Accessed │ operation: │ │
│ │ "created.put" │ Call │
│ │ │ Protocol │
│ S3 Select │ storage.select │ Operation │
│ Results │◀──── call ──────────│ │
└─────────────┘ └─────────────┘
```
Key points:
1. **Events flow inward**: rustfs → alknet (integration events entering alknet's boundary)
2. **Calls flow outward**: alknet → rustfs (alknet initiates S3 Select as a call)
3. **No shared domain model**: alknet shouldn't reference rustfs's `Event` struct directly in its domain; it projects into its own `EventEnvelope` format
4. **Eventual consistency**: rustfs notifications may arrive out of order; `sequentor` field provides ordering within a bucket
5. **At-least-once delivery**: If using webhook/Kafka targets, duplicate events are possible; alknet must be idempotent
6. **No orchestration across boundaries**: alknet doesn't tell rustfs to emit events; it subscribes to events rustfs naturally produces
### 8.5 Implementation Recommendations
1. **Short-term**: Use the **in-process live event stream** to subscribe to rustfs events and re-emit them through alknet's Honker system. This gives immediate value with minimal integration work.
2. **Medium-term**: Add a **webhook notification target** pointing at an alknet HTTP endpoint for redundancy. Configure bucket notification rules via the S3 API (PutBucketNotificationConfiguration).
3. **Long-term**: Consider implementing an **alknet NATS target** that directly publishes events into alknet's NATS infrastructure, bypassing the HTTP layer entirely for lower latency.
4. **S3 Select**: Expose via alknet's call protocol as `storage.select`. The existing `execute_select_object_content` function can be called directly as a library function since alknet and rustfs share the same process.
5. **Event schema versioning**: Store the `event_version` field from rustfs events in alknet's `EventEnvelope.metadata` to handle future schema evolution.
---
## 9. References
### Source Code Locations
| Component | Path |
|-----------|------|
| Event structure | `/crates/notify/src/event.rs` |
| EventName enum | `/crates/s3-types/src/event_name.rs` |
| NotifyPipeline + LiveEventHistory | `/crates/notify/src/pipeline.rs` |
| EventNotifier + TargetList | `/crates/notify/src/notifier.rs` |
| NotificationSystem | `/crates/notify/src/integration.rs` |
| Rule engine | `/crates/notify/src/rule_engine.rs` |
| RulesMap | `/crates/notify/src/rules/rules_map.rs` |
| Bucket notification config | `/crates/notify/src/rules/config.rs` |
| XML notification config | `/crates/notify/src/rules/xml_config.rs` |
| Target trait + QueuedPayload | `/crates/targets/src/target/mod.rs` |
| Webhook target | `/crates/targets/src/target/webhook.rs` |
| Kafka target | `/crates/targets/src/target/kafka.rs` |
| AMQP target | `/crates/targets/src/target/amqp.rs` |
| NATS target | `/crates/targets/src/target/nats.rs` |
| Redis target | `/crates/targets/src/target/redis.rs` |
| MQTT target | `/crates/targets/src/target/mqtt.rs` |
| MySQL target | `/crates/targets/src/target/mysql.rs` |
| PostgreSQL target | `/crates/targets/src/target/postgres.rs` |
| Pulsar target | `/crates/targets/src/target/pulsar.rs` |
| ARN + TargetID | `/crates/targets/src/arn.rs` |
| ECStore event dispatch | `/crates/ecstore/src/event_notification.rs` |
| Server event init | `/rustfs/src/server/event.rs` |
| S3 Select handler | `/rustfs/src/app/select_object.rs` |
| S3 Select query engine | `/crates/s3select-query/src/` |
| S3 Select API | `/crates/s3select-api/src/` |
| S3 Select object store | `/crates/s3select-api/src/object_store.rs` |
| Config subsystem names | `/crates/config/src/notify/mod.rs` |
### AWS S3 Documentation
- [S3 Event Notification Configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html)
- [S3 Select Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html)
### Internal References
- `/workspace/@alkdev/alknet/docs/research/references/rustfs/rustfs-reference.md` — Companion document covering auth, architecture, and credential mapping

732
docs/research/references/rustfs/rustfs-reference.md
View File

@@ -1,732 +0,0 @@
# RustFS Reference Document
> Status: Research Complete
> Last updated: 2026-06-08
> Source: /workspace/rustfs/ (cloned repository, v1.0.0-beta.7)
> Context: alknet internal service integration research
---
## 1. Architecture Overview
### What is RustFS?
RustFS is a high-performance, distributed, S3-compatible object storage system written in Rust. It is an Apache 2.0-licensed alternative to MinIO that combines S3 API compatibility with OpenStack Swift/Keystone support, designed for data lake, AI, and big data workloads.
**Key characteristics:**
- Language: Rust (edition 2024, MSRV 1.95.0)
- License: Apache 2.0 (no AGPL restrictions)
- Workspace: 57 crates in a flat `crates/` layout
- Main binary: `rustfs/` (75K lines); core engine: `crates/ecstore/` (87K lines)
- Version: 1.0.0-beta.7
### Ports and Endpoints
| Port | Purpose |
|------|---------|
| 9000 | S3 API (primary data path) + Admin API (`/minio/` prefix) |
| 9001 | Web Console UI |
### Request Flow
```
HTTP request
→ server (TLS, auth, routing, compression)
→ app/object_usecase (validation, policy, lifecycle)
→ storage/ecfs (erasure coding, encryption, checksums)
→ ecstore (disk pool selection, data distribution)
→ rio (reader pipeline: encrypt → compress → hash → write)
→ io-core (zero-copy I/O, buffer pool, direct I/O)
→ local disk / remote disk via RPC
```
### Key Crate Map (Security & Auth Focus)
| Crate | Lines | Purpose |
|-------|-------|---------|
| `credentials` | 713 | Credential types (access key / secret key), global credentials |
| `signer` | 1.4K | AWS Signature V4 request signing |
| `iam` | 9.0K | Identity and Access Management (users, groups, policies, OIDC) |
| `policy` | 8.8K | S3 bucket/IAM policy engine |
| `keystone` | 1.9K | OpenStack Keystone auth integration |
| `appauth` | 143 | Application-level auth tokens |
| `crypto` | 1.6K | Encryption primitives |
| `kms` | 8.1K | Key management service integration |
| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
| `s3-ops` | — | S3 operation definitions and mapping |
| `s3-types` | — | S3 event type definitions |
### Startup Sequence (Auth-Relevant Steps)
1. Environment variable compatibility (`MINIO_*` → `RUSTFS_*`)
2. Tokio runtime construction
3. CLI argument parsing
4. Config parsing, credentials/endpoints initialization
5. HTTP server start (S3 API + optional console)
6. ECStore initialization
7. **Steps 13: Bucket metadata, IAM, Keystone, OIDC** initialization
8. FullReady → serving requests
---
## 2. S3 API Compatibility
### Supported S3 Operations
RustFS implements a substantial subset of the S3 API via the `s3s` crate (a fork/custom build at `https://github.com/rustfs/s3s`). Based on the feature status table and crate structure:
| Category | Status | Details |
|----------|--------|---------|
| Core Object Ops (GET/PUT/DELETE/HEAD) | ✅ Available | Primary data path |
| Multipart Upload | ✅ Available | Upload, download, multipart |
| Versioning | ✅ Available | Object versioning |
| Bucket Operations | ✅ Available | Create, list, delete, metadata |
| Logging | ✅ Available | Access logging |
| Event Notifications | ✅ Available | Webhook, Kafka, AMQP, MQTT, NATS targets |
| Bitrot Protection | ✅ Available | Checskums at storage layer |
| Single Node Mode | ✅ Available | Single-node deployment |
| Bucket Replication | ✅ Available | Cross-region replication |
| KMS | 🚧 Under Testing | Key management service |
| Lifecycle Management | 🚧 Under Testing | Object lifecycle rules |
| Distributed Mode | 🚧 Under Testing | Multi-node erasure coding |
| Admin API | ✅ Available | `/minio/` prefix, 30+ handler modules |
| Console | ✅ Available | Web UI on port 9001 |
| S3 Select | ✅ Available | `s3select-api` + `s3select-query` crates |
| WebDAV | ✅ Available | `protocols` crate, `dav-server` |
| FTP/FTPS | ✅ Available | `libunftp`, `suppaftp` |
| SFTP | — | `russh` + `russh-sftp` crate deps |
### Authentication Methods
RustFS supports multiple authentication methods (derived from `auth.rs`):
| Auth Type | Constant | Detection |
|-----------|----------|-----------|
| AWS Signature V4 (header) | `Signed` | `Authorization: AWS4-HMAC-SHA256 ...` |
| AWS Signature V4 (query) | `Presigned` | `X-Amz-Credential` in query |
| AWS Signature V2 (header) | `SignedV2` | `Authorization: AWS ...` |
| AWS Signature V2 (query) | `PresignedV2` | `AWSAccessKeyId` in query |
| Streaming V4 | `StreamingSigned` | `x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD` |
| Streaming V4 (trailer) | `StreamingSignedTrailer` | `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER` |
| Unsigned payload (trailer) | `StreamingUnsignedTrailer` | `STREAMING-UNSIGNED-PAYLOAD-TRAILER` |
| POST policy | `PostPolicy` | `multipart/form-data` content type |
| Bearer JWT | `JWT` | `Authorization: Bearer ...` |
| STS | `STS` | `Action` header presence |
| Anonymous | `Anonymous` | No `Authorization` header |
| Keystone token | — | `X-Auth-Token` header (via middleware) |
### S3 Request Signing
The `rustfs-signer` crate implements AWS Signature V4. The general flow:
1. Client computes a canonical request (method + path + query + headers + payload hash)
2. Client creates a string to sign (algorithm + timestamp + credential scope + canonical request hash)
3. Client computes HMAC-SHA256 signature using the secret key
4. Client sends the `Authorization` header with the signature
---
## 3. OpenStack Swift and Keystone Integration
### Swift API
RustFS provides an **OpenStack Swift-compatible API** as an opt-in feature (behind the `swift` cargo feature flag). This is implemented in `crates/protocols/src/swift/`.
**Swift API endpoint pattern:** `/v1/AUTH_{project_id}/...`
**Supported Swift operations:**
- Container CRUD (create, list, delete, metadata)
- Object CRUD with streaming downloads
- Keystone token authentication
- Multi-tenant isolation with SHA256-based bucket prefixing
- Server-side object copy (COPY method)
- HTTP Range requests (206/416 responses)
- Custom metadata (X-Object-Meta-*, X-Container-Meta-*)
**Not yet implemented:** Account-level ops, large object support (>5GB), object versioning, container ACLs/CORS, TempURL, XML/plain-text response formats.
**Tenant isolation:** Swift containers are mapped to S3 buckets with a secure hash prefix:
```
Swift: /v1/AUTH_abc123/mycontainer
→ S3 Bucket: {sha256(abc123)[0:16]}-mycontainer
```
### Keystone Authentication — Complete Flow
This is the most auth-relevant subsystem for alknet integration.
#### Configuration (Environment Variables)
| Variable | Description | Default |
|----------|-------------|---------|
| `RUSTFS_KEYSTONE_ENABLE` | Enable Keystone auth | `false` |
| `RUSTFS_KEYSTONE_AUTH_URL` | Keystone endpoint URL | (required) |
| `RUSTFS_KEYSTONE_VERSION` | API version (`v3` or `v2.0`) | `v3` |
| `RUSTFS_KEYSTONE_ADMIN_USER` | Admin username | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_PASSWORD` | Admin password | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_PROJECT` | Admin project/tenant | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_DOMAIN` | Admin domain | `Default` |
| `RUSTFS_KEYSTONE_VERIFY_SSL` | Verify TLS certificates | `true` |
| `RUSTFS_KEYSTONE_ENABLE_CACHE` | Enable token caching | `true` |
| `RUSTFS_KEYSTONE_CACHE_SIZE` | Token cache capacity | `10000` |
| `RUSTFS_KEYSTONE_CACHE_TTL` | Token cache TTL (seconds) | `300` |
| `RUSTFS_KEYSTONE_TENANT_PREFIX` | Enable tenant project prefixing | `true` |
| `RUSTFS_KEYSTONE_IMPLICIT_TENANTS` | Auto-create tenants | `true` |
| `RUSTFS_KEYSTONE_TIMEOUT` | Request timeout (seconds) | `30` |
#### Architecture: Component Stack
```
KeystoneClient (HTTP calls to Keystone v3 API)
↓
KeystoneAuthProvider (Authentication + Caching via moka::future::Cache)
↓
KeystoneAuthMiddleware (Tower layer, intercepts HTTP requests)
↓ (task-local: KEYSTONE_CREDENTIALS)
IAMAuth → check_key_valid (Authorization)
↓
RustFS Credentials (access_key starts with "keystone:")
```
#### Authentication Flow
**Request with `X-Auth-Token` header:**
1. **Middleware intercepts:** `KeystoneAuthMiddleware` extracts `X-Auth-Token` header
2. **Cache check:** Token cache hit → return cached credentials (~1-2ms)
3. **Token validation:** Cache miss → `KeystoneClient.validate_token()` → `GET /v3/auth/tokens` with `X-Auth-Token` and `X-Subject-Token` headers
4. **Token parsing:** Parse `KeystoneToken` (user_id, username, project_id, project_name, domain, roles, expires_at)
5. **Credential mapping:** Convert to `Credentials` struct:
- `access_key`: `keystone:<user_id>` (special prefix identifies Keystone users)
- `secret_key`: `""` (empty — bypasses AWS SigV4 verification)
- `session_token`: the Keystone token string
- `parent_user`: Keystone username
- `groups`: roles list
- `claims`: JSON map with `keystone_user_id`, `keystone_project_id`, `keystone_roles`, `auth_source: "keystone"`
6. **Task-local storage:** Store credentials in `KEYSTONE_CREDENTIALS` task-local (async-scoped to request)
7. **Auth bypass:** IAMAuth detects `keystone:` prefix → returns empty secret key, bypassing SigV4
8. **Authorization:** `check_key_valid()` retrieves credentials from task-local storage
9. **Role check:** `admin` or `reseller_admin` roles → `is_owner=true`; other roles → `is_owner=false`
**Request without `X-Auth-Token`:**
1. Middleware passes through unchanged
2. Standard AWS SigV4 authentication proceeds
3. IAM validation as normal
**Invalid token:**
1. Middleware returns `401 Unauthorized` immediately with XML error body
2. **No fallback** to standard S3 auth
#### EC2 Credentials
RustFS also supports Keystone EC2 credentials for S3 API compatibility:
- `POST /v3/ec2tokens` with `{access, signature, data}` validates EC2-style credentials
- `GET /v3/users/{user_id}/credentials/OS-EC2` lists EC2 credentials for a user
- Access key format: `user_id:project_id` or `user_id`
#### Role Mapping (Keystone → RustFS)
| Keystone Role | RustFS Policy | Permissions |
|---------------|---------------|-------------|
| `admin` | AdminPolicy | Full access (`s3:*`) |
| `Admin` | AdminPolicy | Full access |
| `Member` | ReadWritePolicy | Read/write |
| `_member_` | ReadOnlyPolicy | Read-only |
| `ResellerAdmin` | AdminPolicy | Full access |
| `SwiftOperator` | ReadWritePolicy | Read/write |
| `objectstore:admin` | AdminPolicy | Full access |
| `objectstore:creator` | ReadWritePolicy | Read/write |
Custom role mappings can be added programmatically via `KeystoneIdentityMapper::add_role_mapping()`.
#### Multi-Tenancy
When `RUSTFS_KEYSTONE_TENANT_PREFIX=true`:
- Bucket creation: `mybucket` → stored as `project_id:mybucket`
- Bucket listing: filtered by project_id
- Access control: users can only access their project's buckets
---
## 4. Authentication Model — Complete Reference
### Credentials Struct
The core `Credentials` struct (in `rustfs-credentials`):
```rust
pub struct Credentials {
pub access_key: String, // S3 access key (or "keystone:<user_id>")
pub secret_key: String, // S3 secret key (empty for Keystone)
pub session_token: String, // STS session token / Keystone token
pub expiration: Option<OffsetDateTime>, // Token expiration
pub status: String, // "active" or "off"
pub parent_user: String, // Parent user for STS/service accounts
pub groups: Option<Vec<String>>, // Group membership
pub claims: Option<HashMap<String, Value>>, // JWT/Keystone claims
pub name: Option<String>, // Human-readable name
pub description: Option<String>,
}
```
Key methods:
- `is_expired()` — checks if the credential's expiration has passed
- `is_temp()` — true if `session_token` is non-empty and not expired
- `is_service_account()` — true if claims contain `sa-policy` key and `parent_user` is non-empty
- `is_valid()` — access_key >= 3 chars, secret_key >= 8 chars, not expired, status != "off"
- Default credentials: `rustfsadmin` / `rustfsadmin` (env vars: `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY`)
### IAM System
The IAM system (`rustfs-iam`) manages:
- **Users and groups** with RBAC
- **Service accounts** and API key authentication
- **Policy engine** with fine-grained S3-style permissions
- **LDAP/Active Directory** integration
- **Session management** and token validation
- **OIDC integration** (full OpenID Connect with PKCE)
The IAM system is initialized as a singleton (`IAM_SYS`) backed by an `ObjectStore` (persisted in the S3 storage itself). Lookups go through `IamSys::check_key(access_key)` which loads from cache or disk.
### OIDC Support
RustFS has comprehensive OIDC support (`rustfs-iam` → `oidc.rs`):
**Configuration (environment variables):**
- `RUSTFS_IDENTITY_OPENID_ENABLE=on`
- `RUSTFS_IDENTITY_OPENID_CONFIG_URL` — OIDC discovery URL
- `RUSTFS_IDENTITY_OPENID_CLIENT_ID` — OAuth2 client ID
- `RUSTFS_IDENTITY_OPENID_CLIENT_SECRET` — OAuth2 client secret
- `RUSTFS_IDENTITY_OPENID_SCOPES` — comma-separated scopes (default: `openid,profile,email`)
- `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` — claim for group membership
- `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM` — claim for role mapping (Microsoft Entra ID app roles)
- `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` — primary claim for policy mapping
- `RUSTFS_IDENTITY_OPENID_CLAIM_PREFIX` — prefix for claim-to-policy mapping
- `RUSTFS_IDENTITY_OPENID_REDIRECT_URI` — callback URL
- `RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC` — allow dynamic redirect URIs
**Features:**
- Authorization Code flow with PKCE
- OIDC discovery and JWKS auto-refresh
- Multiple OIDC providers (suffixed env vars like `_PRIMARY`, `_SECONDARY`)
- ID token verification (signature, issuer, audience, expiry)
- `AssumeRoleWithWebIdentity` flow (JWT directly, no browser)
- Roles and groups claim mapping to RustFS IAM policies
- Provider-specific configuration (Microsoft Entra ID roles claim support)
**OIDC Claims → RustFS Policy Mapping:**
```json
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["admin:*"],
"Resource": ["arn:aws:s3:::*"],
"Condition": {
"ForAnyValue:StringEquals": {
"jwt:roles": ["RustFS.ConsoleAdmin"]
}
}
}]
}
```
### RPC Authentication
RustFS uses a derived RPC secret for inter-node communication:
- Environment variable: `RUSTFS_RPC_SECRET` (explicit) or derived from `access_key + secret_key` via HMAC-SHA256
- Uses a `0xFFFFFFFFFFFFFFFF` mask for the signing context
- Base64url-encoded (no padding) output
---
## 5. Docker Deployment
### Simple Deployment
```yaml
# docker-compose-simple.yml
services:
rustfs:
image: rustfs/rustfs:latest
ports:
- "9000:9000" # S3 API
- "9001:9001" # Console
environment:
- RUSTFS_VOLUMES=/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_OBS_LOGGER_LEVEL=info
volumes:
- rustfs_data_0:/data/rustfs0
- rustfs_data_1:/data/rustfs1
- rustfs_data_2:/data/rustfs2
- rustfs_data_3:/data/rustfs3
```
### Full Deployment (with Observability)
```yaml
# docker-compose.yml (with --profile observability)
services:
rustfs:
# ... same as above, plus:
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
otel-collector: # OpenTelemetry collector
tempo: # Distributed tracing
jaeger: # Jaeger UI
prometheus: # Metrics
loki: # Logs
grafana: # Dashboards
nginx: # Reverse proxy (optional, --profile proxy)
```
### Dockerfile
- Base: Alpine 3.23.4
- Runs as non-root user `rustfs` (UID/GID 10001:10001)
- Single binary: `/usr/bin/rustfs`
- Entrypoint: `/entrypoint.sh` (processes volumes, log dirs, default credential warnings)
- Health check: HTTP/HTTPS `/health` on port 9000, `/rustfs/console/health` on 9001
- Supports TLS via `RUSTFS_TLS_PATH=/opt/tls` with `rustfs_cert.pem` + `rustfs_key.pem` + optional `ca.crt`
### Keystone-Enabled Deployment
```bash
docker run -d \
-p 9000:9000 -p 9001:9001 \
-e RUSTFS_ACCESS_KEY=admin \
-e RUSTFS_SECRET_KEY=adminsecret \
-e RUSTFS_KEYSTONE_ENABLE=true \
-e RUSTFS_KEYSTONE_AUTH_URL=http://keystone:5000 \
-e RUSTFS_KEYSTONE_VERSION=v3 \
-e RUSTFS_KEYSTONE_ADMIN_USER=admin \
-e RUSTFS_KEYSTONE_ADMIN_PASSWORD=secret \
-e RUSTFS_KEYSTONE_ADMIN_PROJECT=admin \
-e RUSTFS_KEYSTONE_ADMIN_DOMAIN=Default \
-v /data:/data \
rustfs/rustfs:latest
```
### Webhook Notification
```bash
docker run -d --name rustfs -p 9000:9000 \
-e RUSTFS_NOTIFY_ENABLE=true \
-e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
-e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://host:3020/webhook \
-e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
rustfs/rustfs:latest
```
---
## 6. SDK/Client Libraries — Rust S3 Clients
### aws-sdk-s3 (Official AWS SDK for Rust)
RustFS itself uses `aws-sdk-s3` (v1.135.0) as a dependency — this is the most mature Rust S3 client:
```toml
aws-sdk-s3 = { version = "1.135.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-config = { version = "1.8.18" }
aws-credential-types = { version = "1.2.14" }
```
**Pros:** Full S3 API coverage, SigV4/SigV4a signing, async, production-tested
**Cons:** Heavy dependency (pulls in significant AWS SDK surface area), AWS-centric abstractions
### s3s (RustFS's own S3 framework)
RustFS uses a custom `s3s` crate (`https://github.com/rustfs/s3s`, with `minio` feature):
```toml
s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
```
This provides S3 request/response types, routing, and the `S3Auth` trait used by RustFS's `IAMAuth`.
### rust-s3 ( Community)
Not used by RustFS, but worth noting as an alternative:
- Crate: `rust-s3` / `s3`
- Simpler API than aws-sdk-s3
- Supports MinIO-compatible endpoints
- Less complete S3 operation coverage
### Recommendation for alknet
For alknet's S3 adapter:
- **Internal use**: aws-sdk-s3, configured with custom endpoint pointing to rustfs
- **Request signing**: If building a lightweight adapter, extract just the signing logic from `rustfs-signer` or use `aws-smithy-runtime` directly
- **The CredentialSet::S3AccessKey variant** (from alknet's credential-provider.md) maps directly to RustFS's `access_key + secret_key` pair; no additional transformation needed
---
## 7. Relevance to Alknet
### 7.1 RustFS as an Internal Object Store Behind Alknet's HTTP Interface
**Architecture:**
```
Client (any S3 SDK)
→ Alknet HTTP adapter (port 443/80 with HTTPS termination)
→ RustFS (port 9000, Docker network, not exposed externally)
→ Disk storage (/data volumes)
```
**Deployment pattern:** RustFS runs as a Docker container on the same Docker network as alknet, listening only on the internal network. Alknet's HTTP interface reverse-proxies S3 API calls to rustfs.
**Reverse proxy considerations:**
- Alknet would forward `Host`, `Authorization`, `X-Auth-Token`, `X-Amz-*` headers unchanged
- RustFS needs the real client IP for S3 policy `SourceIp` conditions; alknet should set `X-Forwarded-For` and configure `RUSTFS_TRUSTED_PROXIES` or use rustfs's `trusted-proxies` crate
- Health check: Alknet proxies `/health` → rustfs:9000
- RustFS supports `X-Forwarded-Proto` for TLS offloading via its `trusted-proxies` crate
**Why behind alknet rather than standalone:**
1. Unified TLS termination at alknet
2. alknet can inject auth headers (e.g., OIDC tokens) before forwarding
3. alknet can enforce rate limiting and access control
4. Network isolation — rustfs only accessible via alknet
**Webhook integration:** RustFS can POST events to alknet via its notification system:
```bash
RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://alknet:3020/webhook
```
### 7.2 Mapping S3 Auth to Alknet's CredentialProvider/CredentialSet
The alknet `CredentialSet` enum directly models the S3 auth pattern:
| RustFS Auth Method | Alknet CredentialSet Variant | Mapping |
|---|---|---|
| Access key + secret key (SigV4) | `S3AccessKey { access_key, secret_key, session_token }` | Direct 1:1 mapping; access_key and secret_key are the S3 credential pair |
| Keystone X-Auth-Token | `OidcToken { access_token, ... }` | Keystone token → OIDC access_token; expires_at maps to Keystone token expiration |
| STS AssumeRole session | `S3AccessKey { ..., session_token: Some(...) }` | STS temporary credentials with session token |
| OIDC (browser flow) | `OidcToken { access_token, refresh_token, expires_at }` | Direct mapping |
| Admin default credentials | `S3AccessKey { access_key: "rustfsadmin", secret_key: "rustfsadmin" }` | Service-level credential |
**S3 Request Signing (Phase C in credential-provider.md):**
The `S3AccessKey` variant contains the raw credential data. The signing computation itself is separate — it's a utility function `s3_sign(credential: &S3AccessKey, request: &HttpRequest) -> SignedRequest` that should live in a shared `alknet-s3` utility crate, not in `CredentialSet`. This matches OpenQ-04 in the credential-provider doc.
**For alknet's `S3CredentialManager`:**
```rust
impl CredentialManager for S3CredentialManager {
fn refresh(&self, current: &CredentialSet) -> Option<CredentialSet> {
// If we have an STS session token, check expiration
// and re-AssumeRole if needed
}
fn is_expired(&self, current: &CredentialSet) -> bool {
match current {
CredentialSet::S3AccessKey { session_token: Some(t), .. }
if !t.is_empty() => check_sts_expiration(t),
CredentialSet::OidcToken { expires_at: Some(ts), .. }
=> *ts < now(),
_ => false, // Static keys don't expire
}
}
fn provision(&self, identity: &Identity) -> Option<CredentialSet> {
// Create a rustfs IAM access key for this alknet identity
// via the rustfs admin API
}
}
```
### 7.3 Alknet as an OIDC Provider for RustFS (Phase D)
This is the most strategically important integration point. RustFS already has complete OIDC support — it just needs an OIDC provider to trust.
**How it would work:**
1. **alknet exposes OIDC endpoints** (via call protocol HTTP adapter or a dedicated `/oidc/` path):
- `GET /.well-known/openid-configuration` — discovery document
- `GET /oidc/authorize` — authorization endpoint
- `POST /oidc/token` — token exchange
- `GET /oidc/userinfo` — user info
- `GET /oidc/jwks` — JSON Web Key Set
- `GET /oidc/logout` — RP-initiated logout
2. **alknet's Identity maps to OIDC claims:**
- `sub` → `Identity.id` (SSH fingerprint or account UUID)
- `email` → from account metadata (if available)
- `username` → display name or `Identity.id`
- `groups` → `Identity.scopes` (e.g., `["s3:admin", "s3:readwrite"]`)
- `roles` → derived from scopes (e.g., `scope "s3:admin"` → role `"admin"`)
3. **RustFS configuration** (pointing at alknet):
```bash
RUSTFS_IDENTITY_OPENID_ENABLE=on
RUSTFS_IDENTITY_OPENID_CONFIG_URL=https://alknet:443/.well-known/openid-configuration
RUSTFS_IDENTITY_OPENID_CLIENT_ID=alknet-rustfs-client
RUSTFS_IDENTITY_OPENID_CLIENT_SECRET=<auto-generated>
RUSTFS_IDENTITY_OPENID_SCOPES=openid,profile,email,groups
RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM=groups
RUSTFS_IDENTITY_OPENID_ROLES_CLAIM=roles
```
4. **Authentication flow:**
- User connects to alknet (via SSH/WebTransport/HTTP)
- alknet resolves identity → `Identity { id, scopes, resources }`
- User requests access to rustfs console
- Browser redirects to alknet's OIDC authorize endpoint
- alknet issues authorization code → token exchange → ID token
- RustFS verifies the ID token using alknet's JWKS endpoint
- RustFS maps `groups` and `roles` claims to IAM policies
5. **For `AssumeRoleWithWebIdentity` (programmatic access):**
- alknet issues a JWT directly to the client
- Client presents JWT to RustFS via `Action=AssumeRoleWithWebIdentity`
- RustFS calls `OidcSys::verify_web_identity_token()` which:
- Decodes JWT payload to get `iss` claim
- Finds matching OIDC provider (alknet)
- Verifies signature, issuer, audience, expiry
- Extracts claims → maps to RustFS policies
**This eliminates stored credentials entirely** — alknet identities authenticate directly to rustfs via OIDC, no `S3AccessKey` needed.
### 7.4 Alknet RustFS Adapter Architecture
An alknet HTTP/HTTPS adapter for the S3 API would look like:
```
alknet HTTP adapter
├── Route: /s3/* → reverse proxy to rustfs:9000
│ ├── Preserve all S3 headers (Authorization, X-Amz-*, X-Auth-Token, Content-*)
│ ├── Set X-Forwarded-For, X-Forwarded-Proto
│ ├── Optionally inject X-Auth-Token from alknet Identity
│ └── Response streaming (for large object downloads)
├── Route: /s3/health → rustfs:9000/health (health check)
└── Route: /s3/admin/* → rustfs:9000/minio/* (admin API)
```
**Key considerations:**
- S3 requests can be very large (multipart uploads, 5TB+ objects). The adapter must support streaming both request and response bodies without buffering.
- `X-Forwarded-For` must be set so rustfs can evaluate `SourceIp` condition keys in bucket policies.
- RustFS already handles `X-Forwarded-Proto` for HTTPS offloading via its `trusted-proxies` crate.
- For OIDC integration, the adapter doesn't need to modify auth headers — rustfs handles OIDC token validation itself when pointed at alknet's OIDC endpoint.
**Alknet's `OpenAPIServiceRegistry` integration:**
Since rustfs exposes an S3 API, alknet could auto-register S3 operations via an OpenAPI spec or hardcoded operation specs:
```rust
// In alknet's service registry:
let s3_ops = FromOpenAPI(s3_openapi_spec, config);
// Where config.auth = CredentialSet::S3AccessKey { access_key, secret_key, session_token: None }
// Or: config.auth = CredentialSet::OidcToken { access_token, refresh_token, expires_at }
```
---
## 8. Key RustFS Source Files for Reference
| File | Purpose |
|------|---------|
| `crates/credentials/src/credentials.rs` | `Credentials` struct, global credentials, key generation |
| `crates/credentials/src/constants.rs` | Default access/secret keys, IAM policy constants |
| `crates/signer/` | AWS Signature V4 implementation |
| `crates/keystone/src/config.rs` | Keystone configuration from env vars |
| `crates/keystone/src/client.rs` | Keystone v3 API client (token validation, EC2 creds, admin auth) |
| `crates/keystone/src/auth.rs` | `KeystoneAuthProvider` (token → `Credentials` mapping) |
| `crates/keystone/src/middleware.rs` | Tower middleware extracting `X-Auth-Token`, task-local storage |
| `crates/keystone/src/identity.rs` | `KeystoneIdentityMapper` (role → policy, tenant prefix) |
| `crates/iam/src/oidc.rs` | Complete OIDC system (discovery, PKCE, token exchange, JWT verification) |
| `crates/iam/src/sys.rs` | `IamSys` (IAM singleton, user/key management) |
| `crates/policy/` | S3 bucket/IAM policy evaluation engine |
| `rustfs/src/auth.rs` | `IAMAuth`, `check_key_valid`, auth type detection, condition values |
| `rustfs/src/server/` | HTTP server, TLS, routing, middleware stack |
| `crates/protocols/src/swift/` | OpenStack Swift API implementation |
| `Dockerfile` / `docker-compose-simple.yml` | Deployment configuration |
---
## 9. Configuration Quick Reference
### RustFS Docker Environment Variables (Auth-Relevant)
| Variable | Description | Default |
|----------|-------------|---------|
| `RUSTFS_ACCESS_KEY` | Root access key | `rustfsadmin` |
| `RUSTFS_SECRET_KEY` | Root secret key | `rustfsadmin` |
| `RUSTFS_ADDRESS` | S3 API listen address | `0.0.0.0:9000` |
| `RUSTFS_CONSOLE_ADDRESS` | Console listen address | `0.0.0.0:9001` |
| `RUSTFS_CONSOLE_ENABLE` | Enable web console | `true` |
| `RUSTFS_TLS_PATH` | TLS certificate directory | (none, HTTP) |
| `RUSTFS_KEYSTONE_ENABLE` | Enable Keystone auth | `false` |
| `RUSTFS_KEYSTONE_AUTH_URL` | Keystone v3 endpoint | (required if enabled) |
| `RUSTFS_KEYSTONE_VERSION` | Keystone API version | `v3` |
| `RUSTFS_KEYSTONE_ADMIN_USER` | Keystone admin user | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_PASSWORD` | Keystone admin password | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_PROJECT` | Keystone admin project | (optional) |
| `RUSTFS_KEYSTONE_ADMIN_DOMAIN` | Keystone admin domain | `Default` |
| `RUSTFS_KEYSTONE_VERIFY_SSL` | Verify Keystone TLS | `true` |
| `RUSTFS_KEYSTONE_CACHE_SIZE` | Token cache size | `10000` |
| `RUSTFS_KEYSTONE_CACHE_TTL` | Token cache TTL (sec) | `300` |
| `RUSTFS_KEYSTONE_TENANT_PREFIX` | Enable tenant prefixing | `true` |
| `RUSTFS_IDENTITY_OPENID_ENABLE` | Enable OIDC | `off` |
| `RUSTFS_IDENTITY_OPENID_CONFIG_URL` | OIDC discovery URL | (required) |
| `RUSTFS_IDENTITY_OPENID_CLIENT_ID` | OIDC client ID | (required) |
| `RUSTFS_IDENTITY_OPENID_CLIENT_SECRET` | OIDC client secret | (optional) |
| `RUSTFS_IDENTITY_OPENID_SCOPES` | OIDC scopes | `openid,profile,email` |
| `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` | Groups claim name | `groups` |
| `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM` | Roles claim name | (empty, opt-in) |
| `RUSTFS_RPC_SECRET` | Inter-node RPC auth secret | (derived from keys) |
| `RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY` | Enable webhook notifications | `off` |
| `RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY` | Webhook URL | (required) |
---
## 10. Summary of Integration Paths
### Phase A (Immediate): Static S3 Credentials
- Deploy rustfs as a Docker service next to alknet
- Configure `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY`
- alknet stores these as `CredentialSet::S3AccessKey`
- alknet's HTTP adapter reverse-proxies S3 calls to rustfs
- Use `aws-sdk-s3` or `rust-s3` as the client library
**Effort:** Low. No auth changes in either system.
### Phase B: OIDC via External Provider
- Configure rustfs `RUSTFS_IDENTITY_OPENID_*` to point at an external OIDC provider (e.g., Keycloak, Authentik, Microsoft Entra ID)
- alknet can still manage its own auth independently
- Both systems trust the same OIDC provider
**Effort:** Low. Configuration-only change in rustfs.
### Phase C: Managed Credentials
- alknet provisions rustfs access keys via admin API (`/minio/` endpoints)
- `S3CredentialManager` handles session token rotation
- Identity-bound credentials: alknet creates per-user access keys in rustfs IAM
**Effort:** Medium. Requires admin API client, credential lifecycle management.
### Phase D: Alknet as OIDC Provider (Target State)
- alknet exposes OIDC endpoints (`.well-known/openid-configuration`, `/oidc/authorize`, `/oidc/token`, `/oidc/jwks`)
- rustfs trusts alknet as its OIDC provider
- `Identity.scopes` maps to rustfs IAM policies (e.g., `s3:admin` → admin policy)
- No stored S3 credentials — users authenticate directly via alknet identity
- `AssumeRoleWithWebIdentity` for programmatic access
**Effort:** High. Requires building OIDC authorization server in alknet. This is the most elegant but most complex path.
---
## References
- [RustFS GitHub](https://github.com/rustfs/rustfs) — v1.0.0-beta.7
- [RustFS Documentation](https://docs.rustfs.com)
- [RustFS Keystone README](file:///workspace/rustfs/crates/keystone/README.md) — comprehensive Keystone integration docs
- [RustFS OIDC implementation](file:///workspace/rustfs/crates/iam/src/oidc.rs) — full OIDC client with PKCE, discovery, JWKS refresh
- [RustFS auth.rs](file:///workspace/rustfs/rustfs/src/auth.rs) — IAMAuth, check_key_valid, auth type detection
- [alknet credential-provider.md](file:///workspace/@alkdev/alknet/docs/research/phase2/credential-provider.md) — alknet's outbound auth design
- [alknet identity.md](file:///workspace/@alkdev/alknet/docs/architecture/identity.md) — alknet's inbound auth design