From 80d1fd0fb3b8c4fa21b2694bc4e71a569bb71730 Mon Sep 17 00:00:00 2001 From: "glm-5.1" Date: Fri, 12 Jun 2026 13:17:39 +0000 Subject: [PATCH] Update architecture docs to address security review #003 findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add three ADRs (025-027) and update five spec documents to close gaps identified in the security and bug review: - ADR-025: Rate limiter IP source must be ConnectInfo only (C1 fix) - ADR-026: Connector timeout ceiling of 30s for per-site timeouts (C3 fix) - ADR-027: Admin socket resource limits — 5s timeout, 4096 byte line limit (W4 fix) Spec changes: - proxy.md: add rate limiter IP source section, URI error handling constraint, connector ceiling description, renumber sections - operations.md: add ConnectInfo-only IP source, in-flight counter architectural requirement (C2), JSON format guarantee (C4), admin socket resource limits, 100ms drain polling interval - config.md: fix http_port type u32→u16 (W12), tighten upstream host validation (W1), tighten ACME contact validation (W2), add X-Forwarded-Proto cross-reference, clarify alknet ADR-030 reference - overview.md: fix ambiguous C1 reference, add ADR/OQ cross-references - open-questions.md: update OQ-09 resolution, add OQ-13 (acme_contact Vec) and OQ-14 (eviction configurability) - README.md: add ADR-025/026/027 and OQ-13/14, update doc statuses to draft Also fix reviewer findings: alknet ADR-030 scope clarification, RFC 2616 reference updated to RFC 7230. --- docs/architecture/README.md | 17 ++-- docs/architecture/config.md | 48 +++++++--- .../decisions/025-rate-limiter-ip-source.md | 93 +++++++++++++++++++ .../026-connector-timeout-ceiling.md | 90 ++++++++++++++++++ .../027-admin-socket-resource-limits.md | 77 +++++++++++++++ docs/architecture/open-questions.md | 48 ++++++++-- docs/architecture/operations.md | 42 +++++++-- docs/architecture/overview.md | 12 ++- docs/architecture/proxy.md | 58 ++++++++---- 9 files changed, 432 insertions(+), 53 deletions(-) create mode 100644 docs/architecture/decisions/025-rate-limiter-ip-source.md create mode 100644 docs/architecture/decisions/026-connector-timeout-ceiling.md create mode 100644 docs/architecture/decisions/027-admin-socket-resource-limits.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index f5c38cf..7752f71 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -25,11 +25,11 @@ connections remain HTTP/1.1. | Document | Status | Description | |----------|--------|-------------| -| [overview.md](overview.md) | Reviewed | Vision, scope, crate dependencies, exports | -| [proxy.md](proxy.md) | Reviewed | Reverse proxy handler, request flow, header injection | +| [overview.md](overview.md) | Draft | Vision, scope, crate dependencies, exports | +| [proxy.md](proxy.md) | Draft | Reverse proxy handler, request flow, header injection | | [tls.md](tls.md) | Reviewed | TLS termination, ACME, manual certs, SNI, ALPN | -| [config.md](config.md) | Reviewed | TOML config format, static/dynamic split, ArcSwap reload | -| [operations.md](operations.md) | Reviewed | Rate limiting, logging, health check, systemd, shutdown | +| [config.md](config.md) | Draft | TOML config format, static/dynamic split, ArcSwap reload | +| [operations.md](operations.md) | Draft | Rate limiting, logging, health check, systemd, shutdown | ## ADR Table @@ -59,6 +59,9 @@ connections remain HTTP/1.1. | [022](decisions/022-health-check-scope.md) | Health Check Scope — Local Port and Admin Socket Only | Accepted | | [023](decisions/023-http2-client-facing.md) | HTTP/2 Client-Facing Support | Accepted | | [024](decisions/024-ansi-disabled-logging.md) | ANSI-Disabled Logging for Container Deployments | Accepted | +| [025](decisions/025-rate-limiter-ip-source.md) | Rate Limiter IP Source — ConnectInfo Only | Accepted | +| [026](decisions/026-connector-timeout-ceiling.md) | Connector Timeout Ceiling for Per-Site Timeouts | Accepted | +| [027](decisions/027-admin-socket-resource-limits.md) | Admin Socket Resource Limits | Accepted | ## Open Questions @@ -74,10 +77,12 @@ See [open-questions.md](open-questions.md) for the full tracker. | ~~OQ-06~~ | ~~Should upstream timeouts be configurable per-site?~~ | ~~low~~ | **resolved** (ADR-015) | | ~~OQ-07~~ | ~~Should per-site TLS overrides be supported for mixed ACME/manual domains?~~ | ~~low~~ | **resolved** (ADR-019) | | ~~OQ-08~~ | ~~Should `/health` use a less common path to avoid upstream collision?~~ | ~~medium~~ | **resolved** (ADR-022: no `/health` route on main listener) | -| ~~OQ-09~~ | ~~How should `upstream_connect_timeout_secs` be enforced?~~ | ~~medium~~ | **resolved** (implementation gap — ADR-015 already decides this) | +| ~~OQ-09~~ | ~~How should `upstream_connect_timeout_secs` be enforced?~~ | ~~medium~~ | **resolved** (ADR-026: 30s connector ceiling) | | ~~OQ-10~~ | ~~Should ACME contact email be a required config field?~~ | ~~high~~ | **resolved** (already specified in config.md; implementation bug C2) | | ~~OQ-11~~ | ~~How should `X-Forwarded-Proto` be derived per-listener?~~ | ~~medium~~ | **resolved** (hardcoded `https` is correct for TLS-terminating proxy) | | ~~OQ-12~~ | ~~Should request access logging be mandatory or optional?~~ | ~~high~~ | **resolved** (mandatory, always-on per operations.md) | +| OQ-13 | Should `acme_contact` support multiple email addresses? | low | open | +| OQ-14 | Should rate limiter eviction interval and max age be configurable? | low | open | ## Document Lifecycle diff --git a/docs/architecture/config.md b/docs/architecture/config.md index 32f0115..080abaf 100644 --- a/docs/architecture/config.md +++ b/docs/architecture/config.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -75,9 +75,9 @@ config.toml ## Static vs Dynamic Configuration -This split follows the pattern established in alknet (ADR-030) and adapted -for our simpler use case. See ADR-019 for the rationale behind the -`[[listeners]]` configuration format. +This split follows the pattern established in alknet (alknet ADR-030, not +this project) and adapted for our simpler use case. See ADR-019 for the +rationale behind the `[[listeners]]` configuration format. ### StaticConfig @@ -114,16 +114,23 @@ will be handled via signal-based or built-in rotation. | Field | Type | Description | |-------|------|-------------| | `bind_addr` | `String` | IP address to bind to (must be explicit, no `0.0.0.0`; see ADR-016) | -| `http_port` | `u32` | Port for HTTP→HTTPS redirect (default: `80`; set to `0` to disable; valid values: 0 or 1–65535) | +| `http_port` | `u16` | Port for HTTP→HTTPS redirect (default: `80`; set to `0` to disable; valid values: 0 or 1–65535). Note: the implementation currently uses `u32`; this must be changed to `u16` to match the architecture spec (see Security Review W12). | | `https_port` | `u16` | Port for TLS listener (default: `443`) | | `tls.mode` | `"acme"` or `"manual"` | Certificate provisioning mode | | `tls.acme_domains` | `Vec` | Domains for ACME SAN certificate (ACME mode only) | | `tls.acme_cache_dir` | `String` | ACME state cache directory | | `tls.acme_directory` | `"production"` or `"staging"` | Let's Encrypt directory | -| `tls.acme_contact` | `String` | Contact email for ACME registration (e.g., `"mailto:admin@example.com"`). Required for production; Let's Encrypt rejects registrations without a contact email. See OQ-10. | +| `tls.acme_contact` | `String` | Contact email for ACME registration (e.g., `"mailto:admin@example.com"`). Required for production; Let's Encrypt rejects registrations without a contact email. Must contain a non-empty email after `mailto:` with an `@` sign. See OQ-10, OQ-13. | | `tls.cert_path` | `String` | Certificate file path (manual mode only) | | `tls.key_path` | `String` | Private key file path (manual mode only) | +**Note on `X-Forwarded-Proto`**: The `X-Forwarded-Proto` header is derived +from which listener port received the request: `https` for requests on the +listener's `https_port`, `http` for requests on the `http_port`. In practice, +since the HTTP listener sends a 301 redirect rather than proxying, +`X-Forwarded-Proto` is always `"https"` for proxied requests. See proxy.md and +OQ-11. + **Why listeners are static:** Each listener requires binding a TCP socket and constructing a TLS acceptor — operations that fundamentally require a restart. Changing a listener's bind address, TLS mode, or certificate configuration @@ -401,14 +408,23 @@ On startup, the config is validated: 16. Site `host` values must be valid hostnames (not IP addresses, not including ports). Hostnames are normalized to lowercase during validation. 17. `upstream` must be in `host:port` format where `port` is a required integer - 1–65535. Examples: `gitea:3000`, `127.0.0.1:3000`, `[::1]:3000`. Invalid - examples: `gitea` (missing port), `http://gitea:3000` (includes scheme), - `10.0.0.5` (missing port). The `upstream_scheme` field handles the protocol. + 1–65535 and the host part must be a valid DNS hostname or IP address. + IPv6 addresses must use bracket notation (e.g., `[::1]:3000`). Values + like `!!!bad!!!:3000` or `@#$%:8080` are rejected. The host part is + validated as follows: bracket-enclosed values are parsed as IPv6 + addresses; otherwise the host part must parse as a valid `IpAddr` or + pass `is_valid_hostname` validation (same rules as site `host` values). + Examples: `gitea:3000`, `127.0.0.1:3000`, `[::1]:3000`. Invalid examples: + `gitea` (missing port), `http://gitea:3000` (includes scheme), `10.0.0.5` + (missing port), `!!!bad!!!:3000` (invalid host part). The + `upstream_scheme` field handles the protocol. 18. `upstream_scheme` values are case-sensitive: only `"http"` or `"https"` (lowercase). Default is `"http"`. -19. In ACME mode, `tls.acme_contact` must be a valid `mailto:` URI - (e.g., `"mailto:admin@example.com"`). Let's Encrypt requires a contact - email for production certificate requests. +19. In ACME mode, `tls.acme_contact` must be a valid `mailto:` URI with a + non-empty email address containing an `@` sign + (e.g., `"mailto:admin@example.com"`). Values like `"mailto:"` (empty + email) or `"mailto:user"` (no `@`) are rejected. Let's Encrypt requires + a contact email for production certificate requests. On SIGHUP reload, the same validation applies. If the new config fails validation, the reload is rejected and the old config remains active. An error @@ -434,6 +450,8 @@ All design decisions are documented as ADRs in [decisions/](decisions/). | [016](decisions/016-explicit-bind-address.md) | Explicit bind address required | Rejects `0.0.0.0` to prevent accidental exposure | | [019](decisions/019-multi-config-listeners.md) | Multi-config listeners | `[[listeners]]` supporting both dedicated-IP and shared-IP deployment models | | [020](decisions/020-container-deployment.md) | Container deployment model | Flexible upstream addressing; `allow_wildcard_bind` override for containers | +| [026](decisions/026-connector-timeout-ceiling.md) | Connector timeout ceiling | 30s ceiling on connector, per-site timeout via tokio::time::timeout | +| [027](decisions/027-admin-socket-resource-limits.md) | Admin socket resource limits | 5s read timeout, 4096 byte line length limit | ## Open Questions @@ -443,4 +461,8 @@ questions affecting this document: - ~~**OQ-04**: Should config reload support a Unix domain socket API in addition to SIGHUP?~~ (resolved — ADR-014: Unix domain socket admin API added) - ~~**OQ-07**: Should per-site TLS overrides be supported for mixed ACME/manual - domains?~~ (resolved — ADR-019: `[[listeners]]` with per-listener TLS config) \ No newline at end of file + domains?~~ (resolved — ADR-019: `[[listeners]]` with per-listener TLS config) +- **OQ-13**: Should `acme_contact` support multiple email addresses? (see + [open-questions.md](open-questions.md)) +- **OQ-14**: Should rate limiter eviction interval and max age be configurable? + (see [open-questions.md](open-questions.md)) \ No newline at end of file diff --git a/docs/architecture/decisions/025-rate-limiter-ip-source.md b/docs/architecture/decisions/025-rate-limiter-ip-source.md new file mode 100644 index 0000000..3e1450c --- /dev/null +++ b/docs/architecture/decisions/025-rate-limiter-ip-source.md @@ -0,0 +1,93 @@ +# ADR-025: Rate Limiter IP Source Must Be ConnectInfo Only + +## Status + +Accepted + +## Context + +The rate limiter identifies clients by IP address to enforce per-IP token bucket +limits. The question is: what is the authoritative source of the client IP for +rate limiting? + +Two potential sources exist: + +1. **`ConnectInfo`**: The TCP peer address, extracted from + `TcpStream::peer_addr()` before TLS handshake and propagated to axum via + `ConnectInfoService`. This is the real client IP at the TCP level. + +2. **`X-Forwarded-For` header**: A client-supplied HTTP header that may contain + an IP address. This header is set by the proxy handler *after* rate limiting + (the rate limiter runs as middleware before the handler), so the value + present during rate limit checks is whatever the client sent — completely + untrusted. + +ADR-021 already establishes that the proxy is an edge proxy and client-supplied +`X-Forwarded-For` headers are untrusted. The proxy handler replaces +`X-Forwarded-For` with the `ConnectInfo` IP before forwarding upstream +specifically to prevent spoofing. However, ADR-021 only addresses the proxy +handler's header injection — it does not specify the rate limiter's IP source. + +The current implementation checks `X-Forwarded-For` *first* and falls back to +`ConnectInfo`, which creates two attack vectors: + +- **Rate limit bypass**: A client sends each request with a different random + `X-Forwarded-For` value. Every request appears to come from a different IP, + evading the per-IP token bucket entirely. +- **Denial-of-service via IP spoofing**: A client sends requests with + `X-Forwarded-For: `. The victim's bucket is depleted and their + legitimate requests receive 429 responses. + +## Decision + +The rate limiter must use `ConnectInfo` as the **sole** source of +client IP addresses. `X-Forwarded-For` must not be consulted for rate limiting +purposes. + +The rate limiter runs as middleware before the proxy handler. At that point in +the request lifecycle, no proxy headers have been injected — any `X-Forwarded-For` +header present is from the client and is untrusted (ADR-021). + +`ConnectInfo` is always present because each listener populates it +via `into_make_service_with_connect_info::()`. If `ConnectInfo` is +absent (which should not happen in normal operation), the request should be +rejected rather than falling back to an untrusted header. + +## Rationale + +- The proxy is the edge proxy (ADR-021). Client-supplied headers are + untrusted at the edge. +- Rate limiting is a security mechanism — it must use the most trustworthy + IP source available. `ConnectInfo` is set by the kernel's TCP stack, not by + the client. +- The rate limiter's position in the middleware stack (before the handler) + means it sees raw client headers, not the replaced `X-Forwarded-For` that the + proxy handler injects. +- Falling back to `X-Forwarded-For` when `ConnectInfo` is absent creates a + downgrade attack — an attacker could potentially strip `ConnectInfo` from + extensions to force the fallback path. Better to reject than to accept an + untrusted IP. + +## Consequences + +**Positive:** +- Rate limiting cannot be bypassed via header spoofing +- Rate limiting cannot be weaponized to DoS legitimate clients +- Consistent with ADR-021's edge proxy trust model +- No ambiguity about the IP source — one source, one code path + +**Negative:** +- If the proxy is ever placed behind a trusted CDN or load balancer, + `ConnectInfo` will reflect the CDN's IP, not the end client's. This would + require a "trusted proxies" configuration (already noted as a Phase 2 + consideration in ADR-021). +- Requests without `ConnectInfo` are rejected. This should not happen in + normal operation but adds a hard failure mode. + +## References + +- [proxy.md](../proxy.md) — Proxy header injection, request flow +- [operations.md](../operations.md) — Rate limiting design +- ADR-006 — Token bucket rate limiting +- ADR-021 — X-Forwarded-For edge proxy model +- Security Review C1 — Rate limiter X-Forwarded-For spoofing vulnerability \ No newline at end of file diff --git a/docs/architecture/decisions/026-connector-timeout-ceiling.md b/docs/architecture/decisions/026-connector-timeout-ceiling.md new file mode 100644 index 0000000..913cd80 --- /dev/null +++ b/docs/architecture/decisions/026-connector-timeout-ceiling.md @@ -0,0 +1,90 @@ +# ADR-026: Connector Timeout Ceiling for Per-Site Timeouts + +## Status + +Accepted + +## Context + +ADR-015 specifies per-site upstream connect timeout configuration with a default +of 5 seconds. The proxy enforces connect timeouts using two mechanisms: + +1. **`tokio::time::timeout`**: Wraps the entire `client.request()` call with the + per-site `upstream_connect_timeout_secs` value. +2. **`HttpConnector::set_connect_timeout()`**: Sets the TCP-level connect timeout + on the hyper `HttpConnector` inside the shared client. + +The problem: the HTTP connector's `set_connect_timeout()` is set once at client +creation time and applies to all requests through that client. The current +implementation hardcodes this to 5 seconds. Since the connector's internal +timeout fires before `tokio::time::timeout`, any per-site connect timeout +greater than 5 seconds is silently capped — the connector times out at 5s +regardless of the configured value. + +Three approaches exist: + +1. **Raise the connector timeout to a high ceiling**: Set the connector's + `set_connect_timeout` to a value higher than any reasonable per-site timeout + (e.g., 30s). Let `tokio::time::timeout` enforce the actual per-site limit. + The connector timeout becomes a safety ceiling, not the primary enforcement + mechanism. + +2. **Remove the connector timeout entirely**: Set `set_connect_timeout` to + `None` and rely solely on `tokio::time::timeout`. This removes one layer + of timeout enforcement but simplifies the model. + +3. **Create per-site client instances**: Each site gets its own hyper Client + with its own connector configured with the site's connect timeout. This is + the most precise approach but creates many client instances and connection + pools, increasing resource usage. + +## Decision + +Use approach 1: set the connector timeout to a high ceiling value (30 seconds) +and let `tokio::time::timeout` enforce the actual per-site connect timeout. + +The connector timeout serves as a safety ceiling — it ensures that even if the +`tokio::time::timeout` wrapper fails or is misconfigured, TCP connections +cannot hang indefinitely. The ceiling of 30s is well above the default 5s and +any reasonable per-site override. + +The `tokio::time::timeout` wrapper with the per-site value is the primary +enforcement mechanism. It fires at the correct per-site threshold. + +## Rationale + +- The shared client architecture (ADR-017) means one connector timeout for all + sites. Creating per-site clients would undermine connection pooling and + increase resource usage. +- A ceiling approach preserves the defense-in-depth benefit of two timeout + layers while allowing per-site values to actually work. +- 30s is a reasonable ceiling — no legitimate upstream connect should take + longer than 30s. Sites that need a higher connect timeout can set the ceiling + even higher if needed, but 30s covers all practical cases. +- Removing the connector timeout (approach 2) removes the safety ceiling + entirely. If `tokio::time::timeout` has a bug or is misapplied, TCP connects + could hang indefinitely. The ceiling provides a backstop. +- The HTTPS client uses `HttpsConnector`, which wraps the + `HttpConnector`. The same ceiling applies to both HTTP and HTTPS clients. + +## Consequences + +**Positive:** +- Per-site connect timeouts work as documented (ADR-015) +- Maintains defense-in-depth with two timeout layers +- No change to the shared client / connection pooling architecture +- Simple implementation: change one constant + +**Negative:** +- The connector timeout no longer matches the default connect timeout (5s + default vs. 30s ceiling). Operators who read the connector timeout might + be confused — documentation must make the ceiling role clear. +- If a site needs a connect timeout > 30s, the ceiling must be raised. This + is unlikely in practice but creates a hidden upper bound. + +## References + +- [proxy.md](../proxy.md) — Upstream connection, per-site timeouts +- ADR-015 — Per-site upstream timeouts with defaults +- ADR-017 — Upstream connection defaults +- Security Review C3 — Connect timeout silently capped at 5s \ No newline at end of file diff --git a/docs/architecture/decisions/027-admin-socket-resource-limits.md b/docs/architecture/decisions/027-admin-socket-resource-limits.md new file mode 100644 index 0000000..6d45618 --- /dev/null +++ b/docs/architecture/decisions/027-admin-socket-resource-limits.md @@ -0,0 +1,77 @@ +# ADR-027: Admin Socket Resource Limits + +## Status + +Accepted + +## Context + +The admin Unix domain socket (ADR-014) accepts connections from local +processes and reads one newline-terminated command per connection. The current +implementation has no read timeout and no line length limit. + +Two attack vectors exist for processes with access to the admin socket +(controlled by Unix file permissions): + +1. **Connection hold**: A client connects and sends no data, keeping a + connection and tokio task open indefinitely. An attacker with socket access + can open many such connections to exhaust file descriptors or task slots. + +2. **Unbounded memory allocation**: A client sends data without a newline, + causing `read_line` to buffer indefinitely. This allows unbounded memory + consumption from a single connection. + +While the admin socket is protected by Unix file permissions (only processes +with access to the socket file can connect), defense-in-depth warrants basic +resource limits. The socket is a local diagnostic interface, not a +high-performance endpoint — strict limits are appropriate. + +## Decision + +Apply two resource limits to admin socket connections: + +1. **Read timeout**: 5 seconds per connection. If no complete command is + received within 5 seconds, the connection is closed and a timeout error + is logged at `debug` level. + +2. **Line length limit**: 4096 bytes per command. If a client sends more + than 4096 bytes without a newline, the connection is closed and the event + is logged at `warn` level. The longest valid command (`reload`) is 6 bytes — + 4096 bytes provides ample room for future commands while preventing abuse. + +Both limits apply per connection. The admin socket creates one tokio task per +connection, so the limits also bound per-connection resource usage. + +## Rationale + +- Defense-in-depth: even with Unix permission protection, basic resource + limits prevent accidental or deliberate resource exhaustion. +- 5 seconds is generous for a local socket — a local process should be able + to send a command within milliseconds. The timeout handles stuck clients and + network issues on tunnel-mounted sockets. +- 4096 bytes is 600x the longest current command. It accommodates future + multi-field commands without allowing abuse. +- These limits have no impact on legitimate admin socket usage — the current + commands (`reload`, `status`) are tiny and complete instantly. + +## Consequences + +**Positive:** +- No unbounded memory allocation from admin socket connections +- No indefinite connection holding +- Simple implementation: `tokio::time::timeout` + `tokio::io::take` (or + `read_until` with a byte limit) +- No impact on legitimate usage + +**Negative:** +- If a future admin command requires more than 4096 bytes (unlikely), the + limit must be raised. +- A 5-second timeout could cause issues if the admin socket is accessed via + a slow network tunnel. In practice, this is unlikely — admin socket access + is typically local. + +## References + +- ADR-014 — Unix domain socket config reload API +- [operations.md](../operations.md) — Admin socket protocol +- Security Review W4 — Admin socket has no read timeout or line length limit \ No newline at end of file diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index 814302a..b1f0c4e 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -111,18 +111,21 @@ last_updated: 2026-06-12 ### ~~OQ-09: How should `upstream_connect_timeout_secs` be enforced?~~ -- **Origin**: Implementation review finding W4, ADR-015, ADR-017 +- **Origin**: Implementation review finding W4, ADR-015, ADR-017, Security + Review C3 - **Status**: resolved - **Priority**: medium - **Resolution**: Implemented using a two-phase `tokio::time::timeout` approach. The inner timeout uses the per-site `upstream_connect_timeout_secs` (default 5s) for the connect + first-byte phase, and the outer timeout uses `upstream_request_timeout_secs` (default 60s) for the full request/response - cycle. Additionally, `HttpConnector::set_connect_timeout()` enforces the - TCP-level connect timeout on both HTTP and HTTPS clients. The implementation - is in `handler.rs` and `create_http_client()`/`create_https_client()`. - No new ADR needed; the decision was already made in ADR-015. -- **Cross-references**: ADR-015, ADR-017 + cycle. The shared `HttpConnector` uses a 30-second connect timeout ceiling + via `set_connect_timeout()` — this is a safety backstop, not the primary + enforcement mechanism. The per-site `tokio::time::timeout` enforces the + actual connect timeout. This ensures per-site values >5s work correctly + (previously the hardcoded 5s connector timeout silently capped them). See + ADR-026. +- **Cross-references**: ADR-015, ADR-017, ADR-026 ### ~~OQ-10: Should ACME contact email be a required config field?~~ @@ -168,4 +171,33 @@ last_updated: 2026-06-12 disable access logging." The `log_request!` macro exists in the codebase but is not called — this is an implementation gap (W13), not an architectural question. No ADR needed; ADR-007 already covers the log format. -- **Cross-references**: ADR-007 \ No newline at end of file +- **Cross-references**: ADR-007 + +## Configuration + +### OQ-13: Should `acme_contact` support multiple email addresses? + +- **Origin**: Security Review S9, [config.md](config.md), [tls.md](tls.md) +- **Status**: open +- **Priority**: low +- **Details**: `acme_contact` is currently a single `String`, but ACME supports + multiple contact emails. The `AcmeTlsConfig.contact` field in the + implementation is already `Vec`, and the single config value is + wrapped in `vec![...]`. Changing `acme_contact` to `Vec` in the + config schema would provide consistency with the ACME protocol. However, + this is a config format change that requires migration documentation and + backward compatibility considerations. For Phase 1, a single email is + sufficient. +- **Cross-references**: ADR-004 + +### OQ-14: Should rate limiter eviction interval and max age be configurable? + +- **Origin**: Security Review S2, [operations.md](operations.md) +- **Status**: open +- **Priority**: low +- **Details**: The eviction task interval (60s) and max age (300s) are + currently hardcoded. In high-traffic deployments, a shorter interval or + longer max age might be desirable. These would be dynamic config fields + (hot-reloadable via ArcSwap) if added. For Phase 1, the hardcoded values + are reasonable defaults. +- **Cross-references**: ADR-006 \ No newline at end of file diff --git a/docs/architecture/operations.md b/docs/architecture/operations.md index eb57862..fdda377 100644 --- a/docs/architecture/operations.md +++ b/docs/architecture/operations.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -32,6 +32,12 @@ The rate limiter runs as axum middleware before the proxy handler. It uses a token bucket algorithm per client IP, matching nginx's `limit_req burst` semantics. +The client IP for rate limiting is determined **exclusively** from +`ConnectInfo` — the TCP peer address set before TLS handshake. +Client-supplied `X-Forwarded-For` headers must not be consulted because the +rate limiter runs before the proxy handler injects trusted headers. See +ADR-025. + Rate limits are global per-IP in Phase 1 (not per-site). A request from IP address X counts against the same bucket regardless of which site it targets. Per-site rate limits may be added in Phase 2. @@ -142,6 +148,11 @@ ADR-024. The `tracing-subscriber` layer configuration supports both simultaneously via `Layer` composition. +Both output destinations must respect the `format` config value: when +`format = "json"`, both file and stdout output must use JSON formatting. +When `format = "text"`, both use text formatting. The format must not be +silently ignored in any output path (see Security Review C4). + ### File Logging and fail2ban File logging is the primary integration point for fail2ban. A log file on a @@ -286,6 +297,11 @@ rationale. one newline-terminated command, receives one newline-terminated JSON response, then the server closes the connection. - **Message framing**: Newline-delimited (`\n`). Responses end with `\n`. +- **Resource limits** (see ADR-027): + - Read timeout: 5 seconds. Connections that send no complete command within + 5 seconds are closed. The timeout is logged at `debug` level. + - Line length limit: 4096 bytes. Connections that send more than 4096 bytes + without a newline are closed. The event is logged at `warn` level. - **Commands**: - `reload` — Re-read config file, validate, and swap DynamicConfig. Returns `{"status": "ok"}` or `{"status": "error", "message": "..."}`. @@ -309,9 +325,17 @@ On SIGTERM or SIGINT, the proxy performs a graceful shutdown: 2. **Close idle keep-alive connections** — Send `Connection: close` on any idle connections in the keep-alive pool. 3. **Wait for in-flight requests** — Up to `shutdown_timeout_secs` (default: 30) - for active requests to complete. Server tasks are joined (not aborted) so - that in-flight requests can drain normally. Only after the timeout expires - are remaining tasks aborted. + for active requests to complete. The proxy tracks in-flight requests using + an atomic counter: each request **must** increment the counter when it + begins and decrement when it completes (via guard drop). The increment + must happen before the request task is spawned — if the counter is not + incremented, the drain logic is broken (see Security Review C2). During + drain, the proxy polls the counter every 100ms and exits early + when it reaches zero. If the timeout expires before all requests complete, + the proxy logs how many in-flight requests remain and proceeds to + force-close. Server tasks are joined (not aborted) so that in-flight + requests can drain normally. Only after the timeout expires are remaining + tasks aborted. 4. **Force-close remaining connections** — After the timeout, any remaining connections are forcefully closed via TCP RST. 5. **Cancel background tasks** — ACME renewal tasks, rate limiter eviction task, @@ -592,11 +616,13 @@ All design decisions are documented as ADRs in [decisions/](decisions/). | [014](decisions/014-unix-socket-reload.md) | Unix domain socket config reload API | Programmatic reload with success/failure feedback | | [020](decisions/020-container-deployment.md) | Container deployment model | Defense-in-depth via container isolation; file-primary logging | | [024](decisions/024-ansi-disabled-logging.md) | ANSI-disabled logging | All log output uses `with_ansi(false)` for fail2ban and Docker compatibility | +| [025](decisions/025-rate-limiter-ip-source.md) | Rate limiter IP source | ConnectInfo only, never client-supplied X-Forwarded-For | +| [027](decisions/027-admin-socket-resource-limits.md) | Admin socket resource limits | 5s read timeout, 4096 byte line length limit | ## Open Questions -Open questions are tracked in [open-questions.md](open-questions.md). All -questions affecting this document have been resolved: +Open questions are tracked in [open-questions.md](open-questions.md). Key +questions affecting this document: - ~~**OQ-03**: Should the health check endpoint be on a separate port?~~ (resolved — ADR-013: separate local port, default 9900, localhost only) @@ -605,4 +631,6 @@ questions affecting this document have been resolved: 9900 and admin socket only) - ~~**OQ-12**: Should request access logging be mandatory or optional?~~ (resolved — access logging is mandatory and always-on at `info` level; no configuration - option to disable it) \ No newline at end of file + option to disable it) +- **OQ-14**: Should rate limiter eviction interval and max age be configurable? + (see [open-questions.md](open-questions.md)) \ No newline at end of file diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 7e16d18..3b5c7ec 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -133,7 +133,8 @@ but all routers share `Arc>` and `Arc>>` via axum State. Site routing is global: the `Host` header is matched against a single routing table collected from all listeners' site definitions. Hostnames must be unique across all -listeners — see C1 resolution in the architecture review. +listeners. Hostnames must be unique across all listeners — see Security & Bug +Review #003, finding C1, resolved by ADR-025. In container deployments (ADR-020), the proxy runs in a minimal container with `0.0.0.0` bind address and Docker port publishing. Upstream addresses use Docker @@ -217,6 +218,9 @@ All design decisions are documented as ADRs in [decisions/](decisions/). | [022](decisions/022-health-check-scope.md) | Health check scope — local port and admin socket only | No `/health` route on main listener; health check is port 9900/admin socket only | | [023](decisions/023-http2-client-facing.md) | HTTP/2 client-facing support | ALPN-based protocol detection; HTTP/2 to clients, HTTP/1.1 to upstreams | | [024](decisions/024-ansi-disabled-logging.md) | ANSI-disabled logging | All log output uses `with_ansi(false)` for fail2ban and Docker compatibility | +| [025](decisions/025-rate-limiter-ip-source.md) | Rate limiter IP source | ConnectInfo only, never client-supplied X-Forwarded-For | +| [026](decisions/026-connector-timeout-ceiling.md) | Connector timeout ceiling | 30s ceiling on connector, per-site timeout via tokio::time::timeout | +| [027](decisions/027-admin-socket-resource-limits.md) | Admin socket resource limits | 5s read timeout, 4096 byte line length limit | ## Open Questions @@ -227,4 +231,6 @@ questions affecting this document have been resolved: - ~~**OQ-03**: Should the health check endpoint be on a separate port?~~ (resolved — ADR-013) - ~~**OQ-05**: Should the proxy bind to multiple addresses?~~ (resolved — single `bind_addr` per listener) - ~~**OQ-07**: Should per-site TLS overrides be supported for mixed ACME/manual domains?~~ (resolved — ADR-019: `[[listeners]]` with per-listener TLS config) -- ~~**OQ-08**: Should `/health` use a less common path?~~ (resolved — ADR-022: no `/health` route on main listener; health check is port 9900/admin socket only) \ No newline at end of file +- ~~**OQ-08**: Should `/health` use a less common path?~~ (resolved — ADR-022: no `/health` route on main listener; health check is port 9900/admin socket only) +- **OQ-13**: Should `acme_contact` support multiple email addresses? (see [open-questions.md](open-questions.md)) +- **OQ-14**: Should rate limiter eviction interval and max age be configurable? (see [open-questions.md](open-questions.md)) \ No newline at end of file diff --git a/docs/architecture/proxy.md b/docs/architecture/proxy.md index 9447bfb..34ca97b 100644 --- a/docs/architecture/proxy.md +++ b/docs/architecture/proxy.md @@ -1,5 +1,5 @@ --- -status: reviewed +status: draft last_updated: 2026-06-12 --- @@ -46,7 +46,7 @@ Incoming HTTPS request (HTTP/1.1 or HTTP/2) ▼ ┌─────────────────┐ │ Rate Limiting │ ← tower middleware layer -│ Middleware │ +│ Middleware │ ← IP from ConnectInfo only (ADR-025) └───────┬─────────┘ │ ▼ @@ -116,7 +116,21 @@ port (default: 9900, bound to `127.0.0.1` only) and the admin socket's `status` command — not by intercepting traffic on the public-facing proxy. See ADR-013 and ADR-022. -### 2. Proxy Header Injection +### 2. Rate Limiter IP Source + +The rate limiting middleware runs **before** the proxy handler. At that point, +no proxy headers have been injected — any `X-Forwarded-For` header present is +from the client and is untrusted. The rate limiter must use +`ConnectInfo` as the **sole** source of client IP addresses. +Client-supplied `X-Forwarded-For` headers must not be consulted for rate +limiting. See ADR-025. + +`ConnectInfo` is always present because each listener populates it +via `into_make_service_with_connect_info::()`. If `ConnectInfo` +is absent, the request must be rejected rather than falling back to an +untrusted header. + +### 3. Proxy Header Injection Headers are injected before forwarding. The proxy is an **edge proxy** — it sits directly in front of the internet with no trusted proxies upstream. This @@ -135,12 +149,16 @@ extracting `TcpStream::peer_addr()` before wrapping the connection in `TlsStream`. Each listener provides this information to its axum Router via `axum::ServiceExt::into_make_service_with_connect_info::()`. -### 3. Request Forwarding +### 4. Request Forwarding The proxy handler constructs a new request to the upstream: 1. Build the upstream URI using the site's `upstream_scheme` and `upstream` - address, preserving the original path and query string + address, preserving the original path and query string. **If URI + construction fails** (e.g., the resulting URI is malformed), the proxy must + return 502 Bad Gateway and log the error at `warn` level. The proxy must + never silently drop parts of the URI (such as the query string) — a + malformed upstream URI is an error, not a recoverable condition. 2. Copy the request method, headers, and body from the original 3. Inject proxy headers (X-Real-IP, X-Forwarded-For, X-Forwarded-Proto) 4. Remove hop-by-hop headers (Connection, Keep-Alive, Transfer-Encoding, etc.) @@ -173,12 +191,12 @@ specified, defaults of 5s connect and 60s request are used. Both timeouts are enforced using `tokio::time::timeout`, with the connect timeout nested inside the request timeout to ensure the overall deadline is respected. -### 4. Header Handling +### 5. Header Handling The proxy must handle request and response headers correctly to avoid security issues and protocol violations. -**Headers removed before forwarding (hop-by-hop headers per RFC 2616 §13.5.1):** +**Headers removed before forwarding (hop-by-hop headers per RFC 7230 §6.1):** - `Connection` - `Keep-Alive` @@ -217,7 +235,7 @@ exceptions: - The `Server` header is removed (defense-in-depth: hiding upstream identity) - The proxy does not add a `Server` header to responses -### 5. Error Handling +### 6. Error Handling All error responses use plain text bodies with no proxy version or identity information. No upstream error details are included. Response format: @@ -237,7 +255,7 @@ information. No upstream error details are included. Response format: | Unknown Host header | 404 Not Found | `Not Found` | No matching site definition | | Missing Host header (and no URI host) | 400 Bad Request | `Bad Request` | Required for routing; HTTP/2 clients use `:authority` | -### 6. HTTP → HTTPS Redirect +### 7. HTTP → HTTPS Redirect A separate HTTP listener on port 80 (per listener) handles redirect. It reads the `Host` header from the incoming request and returns a 301 Permanent Redirect @@ -280,9 +298,13 @@ Two shared hyper Client instances handle upstream connections: - **HTTPS client** (`Client, Body>`): For `https://` upstreams, using `hyper-rustls` with system native certificates -Both clients enforce the per-site connect timeout (default 5s) at the TCP level -via `HttpConnector::set_connect_timeout()` and the overall request timeout -(default 60s) via `tokio::time::timeout`. +Both clients use a shared `HttpConnector` with a connect timeout ceiling +(30 seconds) set via `HttpConnector::set_connect_timeout()`. This ceiling +ensures TCP connections cannot hang indefinitely even if the per-site +`tokio::time::timeout` wrapper fails. The per-site connect timeout (default +5s) is enforced by `tokio::time::timeout`, which fires at the correct +per-site threshold. The connector ceiling is a safety backstop, not the +primary enforcement mechanism. See ADR-026. ## Body Size Limit @@ -306,11 +328,13 @@ All design decisions are documented as ADRs in [decisions/](decisions/). | [018](decisions/018-body-size-limit.md) | Request body size limit | 100 MB default matching nginx, Gitea push compatibility | | [021](decisions/021-x-forwarded-for-edge-proxy.md) | X-Forwarded-For edge proxy model | Replace, don't append — proxy is the edge, no trusted upstream proxies | | [023](decisions/023-http2-client-facing.md) | HTTP/2 client-facing support | ALPN-based protocol detection; HTTP/2 to clients, HTTP/1.1 to upstreams | +| [025](decisions/025-rate-limiter-ip-source.md) | Rate limiter IP source | ConnectInfo only, never client-supplied X-Forwarded-For | +| [026](decisions/026-connector-timeout-ceiling.md) | Connector timeout ceiling | 30s ceiling on connector, per-site timeout via tokio::time::timeout | ## Open Questions -Open questions are tracked in [open-questions.md](open-questions.md). All -questions affecting this document have been resolved: +Open questions are tracked in [open-questions.md](open-questions.md). Key +questions affecting this document: - ~~**OQ-06**: Should upstream timeouts be configurable per-site?~~ (resolved — ADR-015: per-site timeout overrides with defaults) @@ -318,5 +342,7 @@ questions affecting this document have been resolved: upstream collision?~~ (resolved — ADR-022: no `/health` route on the main listener; health checking is via port 9900 and admin socket only) - ~~**OQ-09**: How should `upstream_connect_timeout_secs` be enforced?~~ - (resolved — two-phase timeout with `tokio::time::timeout`; connect timeout - nested inside request timeout; TCP-level `set_connect_timeout` on connector) \ No newline at end of file + (resolved — ADR-026: 30s connector ceiling, per-site timeout via + `tokio::time::timeout`) +- **OQ-13**: Should `acme_contact` support multiple email addresses? (see + [open-questions.md](open-questions.md)) \ No newline at end of file