--- status: draft last_updated: 2026-06-14 --- # Operations ## What It Is The operations component covers everything related to running the proxy in production: rate limiting, logging (fail2ban integration), health checks, systemd integration, and graceful shutdown. ## Why It Exists A reverse proxy that can't be monitored, rate-limited, or gracefully restarted is not production-ready. These concerns are cross-cutting — they affect the proxy handler, the TLS layer, and the config system. ## Rate Limiting ### Requirements - Limit requests per IP address (replacing nginx's `limit_req_zone`) - Default: 10 requests/second with burst of 20 (matching current nginx config) - Configurable via DynamicConfig (no restart needed) - Must produce logs that fail2ban can consume ### Design The rate limiter runs as axum middleware before the proxy handler. It uses a token bucket algorithm per client IP, matching nginx's `limit_req burst` semantics. The client IP for rate limiting is determined **exclusively** from `ConnectInfo` — the TCP peer address set before TLS handshake. Client-supplied `X-Forwarded-For` headers must not be consulted because the rate limiter runs before the proxy handler injects trusted headers. See ADR-025. Rate limits are global per-IP in Phase 1 (not per-site). A request from IP address X counts against the same bucket regardless of which site it targets. Per-site rate limits may be added in Phase 2. The token bucket uses **nodelay** semantics matching nginx's `limit_req burst nodelay`: when the bucket is empty, the request is immediately rejected with 429 — requests are not queued. Tokens are added at a rate of `requests_per_second` (1 token every 1000ms / requests_per_second), and the bucket capacity is the `burst` value. When a request exceeds the rate limit, the middleware returns `429 Too Many Requests` and logs the event with structured fields. ### State Eviction The per-IP token bucket state grows over time as new IPs are seen. A background task runs every 60 seconds (configurable) and removes entries whose last access timestamp is older than a configurable eviction age (default: 300 seconds / 5 minutes). This prevents unbounded memory growth while preserving recent entries that may still receive traffic. ### Config Reload Behavior When rate limit parameters change (e.g., from 10 req/s burst 20 to 20 req/s burst 40), the behavior is: 1. New `DynamicConfig` is swapped in via ArcSwap. 2. On the next request from an existing IP, the rate limiter reads the current `DynamicConfig` for rate/burst parameters. 3. The token bucket refills using the new rate, and its capacity is set to the new burst maximum. 4. If the current token count exceeds the new burst maximum, it is capped to the new burst maximum. The HashMap is **not** cleared — this avoids creating a rate-limiting gap. Existing buckets adopt new parameters on their next request. The eviction task continues removing stale entries independently. ### IPv6 Rate Limiting IPv6 addresses have a vastly larger address space than IPv4. Rate limiting per individual IPv6 address (`/128`) is ineffective against attackers who can generate many addresses within a `/64` prefix. - **IPv4**: Rate limited per individual address (`/32`). - **IPv6**: Rate limited per `/64` prefix. All addresses in the same `/64` share the same token bucket. This matches RFC 4941 privacy extension boundaries and common anti-abuse practice. The rate limiter normalizes IPv6 addresses to their `/64` prefix before bucket lookup. ### Fail2ban Integration Rate limit events are logged in a structured format that a custom fail2ban filter can parse. See [ADR-007](decisions/007-custom-log-format.md) for the format decision. The log format uses `key=value` pairs with a `RATE_LIMIT` prefix: ``` RATE_LIMIT client_ip=203.0.113.50 host=Y.Z path=/W status=429 ``` A corresponding fail2ban filter and jail configuration are provided as part of the deployment documentation. ## Logging ### Structure All logs use `tracing` with structured fields. The proxy outputs two types of log entries: 1. **Access logs**: Every proxied request is logged at `info` level with structured fields. Access logging is **always-on** — it is the primary observability mechanism for the proxy and is required for fail2ban integration. There is no configuration option to disable access logging. ``` REQUEST client_ip=203.0.113.50 host=git.alk.dev method=GET path=/user/repo status=200 upstream=127.0.0.1:3000 duration_ms=45 ``` 2. **Event logs**: Rate limits, TLS errors, upstream failures, config reloads, etc. ``` RATE_LIMIT client_ip=203.0.113.50 host=git.alk.dev path=/login status=429 UPSTREAM_ERROR host=git.alk.dev upstream=127.0.0.1:3000 error="connection refused" CONFIG_RELOAD status=success sites=1 ``` ### Output Logs are written to two destinations simultaneously: - **File** (primary): `/var/log/reverse-proxy/access.log` — the authoritative source for fail2ban consumption. File logging is always enabled when the `log_file_path` config is set. See ADR-020 for the rationale behind file-primary logging. - **stdout/stderr**: Always-on, for `docker logs`, `journalctl`, and development use. Structured in the same format as the file output. Both output destinations use `with_ansi(false)` to disable ANSI escape codes. This is critical for fail2ban log parsing (ANSI codes break regex matching) and for clean output in Docker containers where a terminal is not attached. See ADR-024. The `tracing-subscriber` layer configuration supports both simultaneously via `Layer` composition. Both output destinations must respect the `format` config value: when `format = "json"`, both file and stdout output must use JSON formatting. When `format = "text"`, both use text formatting. The format must not be silently ignored in any output path (see Security Review C4). ### File Logging and fail2ban File logging is the primary integration point for fail2ban. A log file on a volume mount is simpler and more reliable than parsing Docker log drivers or journald — no log driver configuration, no format conversion, no risk of dropping events. In container deployments, the log directory is volume-mounted so fail2ban on the host can read it directly: ```yaml volumes: - /var/log/reverse-proxy:/var/log/reverse-proxy ``` A corresponding fail2ban filter definition and jail configuration are provided as part of the deployment documentation. **Filter regex note**: The fail2ban `failregex` pattern matches `RATE_LIMIT` without a `^` anchor because log lines have a timestamp/level prefix before the `RATE_LIMIT` keyword. The pattern `RATE_LIMIT client_ip= host=\S+ path=\S+ status=\d+` matches the rate limit event anywhere in the line, which correctly handles the structured log format. ### Log Levels | Level | Use | |-------|-----| | `error` | Unrecoverable failures (TLS handshake failure, config validation) | | `warn` | Rate limit exceeded, upstream unreachable, upstream timeout | | `info` | Access logs, config reloads, ACME events, startup/shutdown | | `debug` | Request/response headers, connection details | | `trace` | Detailed protocol-level information | Configurable via `log_level` in StaticConfig. ## Health Check ### Local Health Check Port The health check endpoint is served on a separate local port (default: 9900), bound to `127.0.0.1` only. It is not served on the main HTTPS listener — health checking is an operational concern that does not belong on the public-facing proxy. See ADR-013 and ADR-022. ``` GET http://127.0.0.1:9900/health → 200 OK (empty body) ``` The port is configurable via `health_check_port` in StaticConfig. Setting it to `0` disables the health check listener entirely. The admin HTTP endpoint's `status` command provides an additional health/status mechanism that returns process information: ``` GET http://127.0.0.1:9900/admin/status Authorization: Bearer → {"status": "ok", "uptime_secs": 1234, "sites": 2} ``` Requires Bearer token authentication. See ADR-028 for details. ### What It Checks - Process is running and the tokio runtime is responsive - Config is loaded (StaticConfig and DynamicConfig are initialized) It does **not** check upstream reachability. The health check answers "is the proxy process healthy?", not "is the upstream reachable?" — upstream health is a separate concern that would produce 502/504 responses in the proxy handler. It also does **not** verify TLS configuration — that is the responsibility of external monitoring tools that connect to the public HTTPS port directly. ### Future Extensions - `/health/ready` — readiness check that includes upstream reachability - Prometheus metrics at `/metrics` ## Systemd Integration The proxy can also run as a bare binary via systemd (alternative to container deployment). The systemd unit file is provided for this use case. ### Unit File ```ini [Unit] Description=Reverse Proxy After=network.target Wants=network-online.target [Service] Type=notify NotifyAccess=all ExecStart=/usr/local/bin/reverse-proxy --config /etc/reverse-proxy/config.toml Restart=on-failure RestartSec=5 # Security hardening NoNewPrivileges=yes ProtectSystem=strict ProtectHome=yes PrivateTmp=yes ReadWritePaths=/var/lib/reverse-proxy /var/log/reverse-proxy # ACME challenge cache directory StateDirectory=reverse-proxy [Install] WantedBy=multi-user.target ``` The proxy signals readiness to systemd via `sd_notify` after binding listeners and completing the initial configuration load. ## Graceful Shutdown ### Signal Handling The proxy handles three signals via `signal-hook` (see [ADR-009](decisions/009-signal-handling.md)): - **SIGTERM / SIGINT**: Graceful shutdown. Stop accepting new connections, wait for in-flight requests to complete (up to a configurable timeout), then exit. - **SIGHUP**: Config reload. Re-read the config file, validate, and swap DynamicConfig if valid. No feedback on success or failure. - **Admin HTTP endpoint**: Send `POST /admin/reload` with a Bearer token to the health check listener (`http://127.0.0.1:9900/admin/reload`). Returns structured response indicating success or failure. See ADR-028 for details. ### SIGHUP for Config Reload SIGHUP triggers config reload (see [config.md](config.md) for details). The process does not exit on SIGHUP. ### Admin HTTP Endpoint for Config Reload The admin HTTP endpoint provides programmatic config reload with feedback and authentication. This is useful for CI/CD pipelines and automation tools. See ADR-028 for the rationale and security model. **Authentication**: All `/admin/*` endpoints require a Bearer token in the `Authorization` header. The token is compared against a SHA-256 hash of the admin key file contents using constant-time comparison (`subtle::ConstantTimeEq`) to prevent timing attacks. **Key management**: The admin key is stored in a file on disk (specified by `admin_key_path` in StaticConfig, default: `/etc/reverse-proxy/admin-key`). The proxy reads this file once at startup, hashes its contents with SHA-256, and stores only the hash in memory. The plaintext key is never held in memory after startup. Setting `admin_key_path` to an empty string disables all admin endpoints. **Endpoints**: | Method | Path | Description | |--------|------|-------------| | POST | `/admin/reload` | Trigger config reload. Returns `{"status": "ok"}` or `{"status": "error", "message": "..."}` | | GET | `/admin/status` | Return process info: `{"status": "ok", "uptime_secs": N, "sites": N}` | | POST | `/admin/rotate-key` | Generate new random key, return it, replace stored hash | **Error responses**: Admin endpoint errors return generic messages — no filesystem paths, no config structure details. Full error information is logged server-side only. When admin endpoints are disabled (empty `admin_key_path`), `/admin/*` returns 404 (endpoint appears not to exist). Wrong or missing Bearer token returns 401. **Key rotation**: `/admin/rotate-key` generates a new 256-bit random key, returns the plaintext key in the response, and replaces the stored SHA-256 hash in memory. In-memory rotation does **not** persist across restarts — on restart, the proxy re-reads the key file. To make rotation permanent, the operator must also update the key file on disk. **Concurrency**: Reload operations are serialized (see Config Reload section in config.md). Multiple concurrent `/admin/reload` requests are handled correctly. **Config file TOCTOU protection**: Both SIGHUP and admin HTTP reload compare the config file's modification timestamp before and after reading. If the file changed during the read, the reload is rejected with a "please retry" message. Operators should use atomic file replacement (write to a temp file, then `rename()` over the target) for safe config rotation. See ADR-029. **Reload validation consistency**: The `cli_allow_wildcard_bind` flag is stored in `ConfigReloadHandle` at startup, ensuring that reload validation uses the same flag as startup validation. If the proxy was started with `--allow-wildcard-bind`, reload will also allow wildcard bind addresses. See ADR-030. ### Shutdown Sequence On SIGTERM or SIGINT, the proxy performs a graceful shutdown: 1. **Stop accepting new connections** — Close all TCP listening sockets. No new connections are accepted. 2. **Close idle keep-alive connections** — Send `Connection: close` on any idle connections in the keep-alive pool. 3. **Wait for in-flight requests** — Up to `shutdown_timeout_secs` (default: 30) for active requests to complete. The proxy tracks in-flight requests using an atomic counter: each request **must** increment the counter when it begins and decrement when it completes (via guard drop). The increment must happen before the request task is spawned — if the counter is not incremented, the drain logic is broken (see Security Review C2). During drain, the proxy polls the counter every 100ms and exits early when it reaches zero. If the timeout expires before all requests complete, the proxy logs how many in-flight requests remain and proceeds to force-close. Server tasks are joined (not aborted) so that in-flight requests can drain normally. Only after the timeout expires are remaining tasks aborted. 4. **Force-close remaining connections** — After the timeout, any remaining connections are forcefully closed via TCP RST. 5. **Cancel background tasks** — ACME renewal tasks, rate limiter eviction task, and rate limiter eviction task are all cancelled. 6. **Exit with code 0**. The `shutdown_timeout_secs` is configurable in StaticConfig (default: 30 seconds). See config.md for details. ## File Descriptor Budget The connection semaphore (shared across all listeners, review #010 C4) caps concurrent client TLS connections at exactly `max_connections` process-wide. Each connection holds one FD, and the process needs additional FDs for the HTTPS/HTTP/health listener sockets, the log file, ACME renewal sockets, epoll/timer FDs, and stdin/stdout/stderr. At startup and on config reload, the proxy reads the process's soft `RLIMIT_NOFILE` (Unix; skipped when the limit is `RLIM_INFINITY`) and rejects the configuration if: ``` max_connections + 64 (reserved FDs) > soft RLIMIT_NOFILE ``` This guard exists because the semaphore is the intended protection against FD exhaustion — but before review #010 C2, nothing stopped it from being configured at or above the real ceiling, guaranteeing `EMFILE` under load (the review #010 incident shape). A soft limit at or below the Docker default (1024) also triggers a prominent startup warning even when the budget fits. ### Recommended baselines | Environment | nofile | max_connections | Rationale | |-------------|--------|-----------------|-----------| | Docker / docker-compose | 8192 (`ulimits.nofile`) | 800–1024 | Working production baseline from review #010 M1: connection FDs ≈ 10% of the limit | | systemd | 8192 (`LimitNOFILE=`) | 800–1024 | Same headroom; set in the unit file's `[Service]` section | | Bare metal (high traffic) | ≥ 2 × max_connections | sized to traffic | Keep ≥ 50% headroom for non-connection FDs and bursts | Check what the running process actually sees: ```bash cat /proc/$(pidof reverse-proxy)/limits | grep "open files" ``` ## Deployment ### Binary Single static binary, no runtime dependencies: ```bash cargo build --release # Produces: target/release/reverse-proxy ``` The binary is self-contained — no system libraries beyond libc for DNS resolution. The `aws_lc_rs` crypto provider is statically linked. ### Configuration ```bash # Config file /etc/reverse-proxy/config.toml # ACME cache directory /var/lib/reverse-proxy/acme-cache/ # Log directory (optional, for fail2ban) /var/log/reverse-proxy/ ``` ### CLI ```bash reverse-proxy [OPTIONS] Options: --config Path to config file [default: /etc/reverse-proxy/config.toml] --validate Validate config and exit --allow-wildcard-bind Permit 0.0.0.0 as a bind address (for container deployments) --help Show help --version Show version ``` ## Container Deployment ### Rationale The proxy runs in a minimal Docker container for defense-in-depth. Even if an attacker finds a logic-level vulnerability, they must also escape the container boundary. Combined with Rust's memory safety, this provides two independent barriers against exploitation. See ADR-020 for the full rationale. ### Container Image Multi-stage build: compile in `rust:alpine`, run in `alpine` (or `scratch` for absolute minimum). The final image contains only the static binary and necessary runtime files. No shell, no package manager, no unnecessary tools. The binary is compiled against the `x86_64-unknown-linux-musl` target for static linking. The `aws_lc_rs` crypto provider is statically linked — no OpenSSL dependency. ### Networking The proxy supports flexible upstream addressing — no assumption about upstream localality: | Deployment | Upstream Address | Example | |------------|-----------------|---------| | Same-host, shared Docker network | Docker DNS name | `gitea:3000` | | Same-host, host networking | Loopback | `127.0.0.1:3000` | | Different host, LAN | LAN IP | `10.0.0.5:3000` | | Different host, VPN/tunnel | Tunnel endpoint | Varies by tunnel config | In container deployments, the proxy binds `0.0.0.0` inside the container and Docker publishes specific ports to the host IP. The `allow_wildcard_bind` override is required for this configuration (see ADR-016, ADR-020). ### Volume Mounts | Container Path | Host Path | Purpose | |---------------|-----------|---------| | `/etc/reverse-proxy/config.toml` | Config file (read-only) | Proxy configuration | | `/var/lib/reverse-proxy/acme-cache/` | ACME state directory | Certificate persistence across restarts | | `/var/log/reverse-proxy/` | Log directory | fail2ban reads from host | | `/etc/reverse-proxy/admin-key` | Admin key file | Bearer token for admin endpoints (read-only) | ### Docker Compose Example This example shows the reverse proxy alongside a Gitea container on a shared Docker network. Real IPs, secrets, and domain names are replaced with placeholders. ```yaml services: reverse-proxy: build: . container_name: reverse-proxy restart: unless-stopped ports: - "203.0.113.10:80:80" # HTTP redirect - "203.0.113.10:443:443" # HTTPS volumes: - /etc/reverse-proxy/config.toml:/etc/reverse-proxy/config.toml:ro - /var/lib/reverse-proxy/acme-cache:/var/lib/reverse-proxy/acme-cache - /var/log/reverse-proxy:/var/log/reverse-proxy - /etc/reverse-proxy/admin-key:/etc/reverse-proxy/admin-key:ro networks: - proxy-net healthcheck: test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:9900/health"] interval: 30s timeout: 5s retries: 3 gitea: image: gitea/gitea:latest container_name: gitea restart: unless-stopped ports: - "203.0.113.10:22:2222" # Git SSH volumes: - /opt/gitea:/data networks: - proxy-net - gitea-db-net gitea-db: image: postgres:16-alpine container_name: gitea-db restart: unless-stopped environment: POSTGRES_USER: admin POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_DB: gitea volumes: - gitea-db:/var/lib/postgresql/data networks: - gitea-db-net networks: proxy-net: gitea-db-net: volumes: gitea-db: ``` Corresponding proxy config (inside the container): ```toml allow_wildcard_bind = true health_check_port = 9900 admin_key_path = "/etc/reverse-proxy/admin-key" [logging] level = "info" format = "text" log_file_path = "/var/log/reverse-proxy/access.log" [rate_limit] requests_per_second = 10 burst = 20 [body] limit_bytes = 104857600 [[listeners]] bind_addr = "0.0.0.0" http_port = 80 https_port = 443 [listeners.tls] mode = "acme" acme_domains = ["git.example.com"] acme_cache_dir = "/var/lib/reverse-proxy/acme-cache" acme_directory = "production" acme_contact = "mailto:admin@example.com" [[listeners.sites]] host = "git.example.com" upstream = "gitea:3000" # Docker DNS resolves this ``` ### fail2ban Integration In container deployments, fail2ban runs on the host and reads the proxy's log file from the volume mount: ``` /var/log/reverse-proxy/access.log → fail2ban filter → iptables/nftables ``` This is simpler and more reliable than parsing Docker log drivers. The log file is the authoritative source for rate limit events and access logs. ### Health Check Docker's native `HEALTHCHECK` uses the local health endpoint: ```dockerfile HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ CMD wget -q --spider http://127.0.0.1:9900/health || exit 1 ``` No port publishing is needed — the health check runs inside the container. There is no `/health` route on the main HTTPS listener. ### SSH Traffic SSH traffic for Git operations is not proxied through the reverse proxy. It continues to be routed directly to the Gitea container via Docker port publishing (e.g., `203.0.113.10:22:2222`), matching the current deployment pattern. ## Startup Sequence The proxy starts components in a specific order to ensure fail-fast behavior and correct dependency initialization: 1. **Parse and validate config** — Read the TOML config file, deserialize into `StaticConfig` and `DynamicConfig`, and validate all rules. If validation fails, exit with non-zero code and log errors. No ports are bound. 2. **Initialize DynamicConfig** — Load sites, rate limits, and body limits into `ArcSwap`. 3. **Initialize shared state** — Create the rate limiter `HashMap`, the shared `hyper::Client`, and the `tracing-subscriber` with file and stdout layers. 4. **Bind health check port** (if enabled) — Bind `127.0.0.1:{health_check_port}`. Fail-fast if bind fails. 5. **Read admin key** (if enabled) — Read the admin key file, compute SHA-256 hash, and store the hash in memory for Bearer token authentication. If the file is not readable, log a warning and disable admin endpoints (but continue starting — admin endpoints are non-critical). 6. **Bind all listener ports** — For each listener: bind HTTP port (if enabled) and HTTPS port. If any bind fails, fail-fast and exit. All ports are bound before proceeding. 7. **Load TLS configuration** — For each listener: load manual certificates or initialize ACME state machine. If manual certificate loading fails, fail-fast and exit. For ACME: if no cached certificate exists and ACME provisioning fails, fail-fast and exit. 8. **Start TCP listeners** — Begin accepting connections on all bound ports. 9. **Start background tasks** — ACME renewal tasks (per listener in ACME mode), rate limiter eviction task, signal handler task. 10. **Signal readiness** — Send `sd_notify("READY=1")` to systemd (if running under systemd). **Failure semantics**: **Fail-fast**. If any step fails, the process exits with a non-zero code. The proxy does not partially start. All ports are bound before any connections are accepted. ## Design Decisions All design decisions are documented as ADRs in [decisions/](decisions/). | ADR | Decision | Summary | |-----|----------|---------| | [001](decisions/001-rust-axum.md) | Rust with axum | Memory safety; single binary deployment | | [006](decisions/006-rate-limiting-approach.md) | Token bucket rate limiting | In-memory per-IP token bucket matching nginx burst semantics | | [007](decisions/007-custom-log-format.md) | Custom structured log format | key=value pairs with RATE_LIMIT prefix for fail2ban | | [009](decisions/009-signal-handling.md) | Signal handling strategy | signal-hook for SIGTERM/SIGINT/SIGHUP | | [013](decisions/013-health-check-port.md) | Health check on separate local port | Localhost-only HTTP health check, configurable port | | [014](decisions/014-unix-socket-reload.md) | ~~Unix domain socket config reload API~~ | ~~Programmatic reload with success/failure feedback~~ (Superseded by ADR-028) | | [020](decisions/020-container-deployment.md) | Container deployment model | Defense-in-depth via container isolation; file-primary logging | | [024](decisions/024-ansi-disabled-logging.md) | ANSI-disabled logging | All log output uses `with_ansi(false)` for fail2ban and Docker compatibility | | [025](decisions/025-rate-limiter-ip-source.md) | Rate limiter IP source | ConnectInfo only, never client-supplied X-Forwarded-For | | [027](decisions/027-admin-socket-resource-limits.md) | ~~Admin socket resource limits~~ | ~~5s read timeout, 4096 byte line length limit~~ (Deprecated by ADR-028) | | [028](decisions/028-admin-http-api.md) | Authenticated HTTP admin API | Bearer token auth on health check port; replaces Unix domain socket | | [029](decisions/029-config-reload-toctou.md) | Config file TOCTOU mitigation | mtime check before and after config read; reject reload if file changed | | [030](decisions/030-wildcard-flag-consistency.md) | Store cli_allow_wildcard_bind in ConfigReloadHandle | Consistent validation between startup and reload | ## Open Questions Open questions are tracked in [open-questions.md](open-questions.md). Key questions affecting this document: - ~~**OQ-03**: Should the health check endpoint be on a separate port?~~ (resolved — ADR-013: separate local port, default 9900, localhost only) - ~~**OQ-08**: Should `/health` use a less common path?~~ (resolved — ADR-022: no `/health` route on the main listener at all; health checking is via port 9900 and admin HTTP endpoint only) - ~~**OQ-12**: Should request access logging be mandatory or optional?~~ (resolved — access logging is mandatory and always-on at `info` level; no configuration option to disable it) - **OQ-14**: Should rate limiter eviction interval and max age be configurable? (see [open-questions.md](open-questions.md)) - **OQ-15**: Should admin key rotation persist across restarts? (see [open-questions.md](open-questions.md))