tls_handshake_timeout_secs (default 10s, must be > 0) wraps
tls_acceptor.accept() in tokio::time::timeout so stalled handshakes
(slowloris / crawler slow-handshake vector) can no longer hold an FD +
semaphore permit indefinitely. Config plumbing through FullConfig /
StaticConfig / validation / reload diff + docs.
HTTP/2 requests carry the client's host only in the :authority
pseudo-header, with no literal Host header. build_upstream_request
copied headers verbatim, so the upstream request had no Host, and
hyper-util's legacy client (set_host=true default) filled it in from
the upstream URI, sending Host: 127.0.0.1:3000 instead of the client's
host. Backends like Gitea build absolute URLs from that host, causing
the admin self-check mismatch and intermittent wrong asset URLs.
Reconstruct Host from the request URI authority for HTTP/2, preserve
the client's Host header for HTTP/1.1, and let the URI authority win
if both are present.
C2: IdleTrackingService decremented in_flight when the handler returned
the Response, but for streaming responses (e.g. git clone) the body is
still in flight, so the watchdog could kill the connection mid-stream.
Fix: wrap the response body in IdleTrackingBody which owns the decrement,
releasing it on body EOF/error/drop (guarded against double-decrement by
an AtomicBool) and calling touch() on each data frame. The handler-level
decrement now only happens on the Err branch. Verified by a reproducer
test (streaming_body_not_killed_by_idle_watchdog) that streams 15 chunks
over 3s with a 500ms idle timeout — red on 4ab8c51, green with this fix.
C3: the two pre-existing idle-timeout tests pointed at 127.0.0.1:18080
where nothing listens, so they validated watchdog behaviour against an
immediate 504 error response (HTTP/1.1 prefix + in_flight==1 hold for
a 504 just as for a 200) and never exercised a real upstream round-trip.
Rewired them to spawn a real TestUpstream and assert HTTP/1.1 200 OK.
Removed the dead no-arg make_proxy_router/start_test_https_server wrappers.
Adds http-body as a direct dep (for the Frame/Body trait) and
tokio-stream as a dev-dep (for the slow-stream test helper).
cargo test: 226 unit + 40 integration green; cargo clippy --all-targets clean.
C1 (deploy to dev1) remains open and out of scope for this repo session.
Documents two issues discovered 2026-08-19:
- C1: review #008 fix (commit 4ab8c51) never deployed to dev1; the
running binary still leaks FDs (433/1024 after 8 days, ~9 FDs/min).
- C2: the watchdog in IdleTrackingService decrements in_flight when
the handler returns the Response, but for streaming responses (e.g.
git clone) the body is still in flight, so the watchdog can kill
the connection mid-stream. Requires an IdleTrackingBody wrapper
that owns the decrement until the body reaches EOF or is dropped.
The C1 fix from review #007 added keep_alive_interval(15s) +
keep_alive_timeout(60s) on the HTTP/2 builders. However,
keep_alive_timeout is the timeout for receiving a PONG response to a
PING, not an idle timeout. Well-behaved HTTP/2 clients (including
crawlers) respond to PINGs, resetting the timeout indefinitely. After
6 days of uptime, the proxy had 930+ idle connections (some 145 hours
old) that were never reaped.
Fix: add a custom idle timeout that tracks real request activity and
closes connections with no in-flight requests for longer than the
configured timeout, regardless of PING/PONG activity.
- IdleState: per-connection last_activity + in_flight counter
- IdleTrackingService: tower Service wrapper that updates last_activity
and in_flight on call() and on response completion (prevents killing
long-running requests like large git clones)
- idle_watchdog: races against serve_connection in tokio::select!,
fires only when in_flight == 0 and idle_for >= timeout
- HTTP/1.1 path: header_read_timeout already covers between-request
idle (verified in hyper source); watchdog is defense-in-depth
- HTTP/2 path: watchdog is the primary fix (hyper has no native
request-activity-based idle timeout)
Also refactored build_manual_server_config to extract
build_manual_server_config_from_certs for testability (in-memory
certs/keys for integration tests with rcgen).
Tests: 8 unit tests for IdleState/watchdog logic, 2 integration tests
verifying idle connections are closed and active ones are not.
Closes review #008.
Critical:
- C1: Set server-side idle/keep-alive timeouts on both hyper builders
(http2 keep_alive_interval=15s + keep_alive_timeout, http1
header_read_timeout). Both builders now set TokioTimer (required to
avoid runtime panic). Prevents FD exhaustion from abandoned TLS
connections — the root cause of the 2026-07-24 outage.
- C2: Add Semaphore(max_connections) gating the accept loop. Provides
backpressure via OS TCP backlog when all permits are taken.
Warnings:
- W1: Add SIGUSR1 log-reopen handler. New ReopenableFileWriter
(Arc<ArcSwap<File>> via custom MakeWriter) atomically swaps the log
file. Enables postrotate logrotate without copytruncate, which caused
the 1.15GB sparse file that wedged fail2ban.
- W2: Set pool_max_idle_per_host(10) on both upstream clients, bounding
idle upstream connections per host.
- W3: Add connection_idle_timeout_secs to StaticConfig (default 60).
- W4: Add max_connections to StaticConfig (default 1024).
Both new fields are validated (> 0) and included in static config drift
detection on reload. Docs (config.md, README, ADR-009) updated.
- Add reverse-proxy-4xx and reverse-proxy-badbots fail2ban filters
- Set backend=auto and ignoreip on all jails (fixes silent no-match
when defaults-debian.conf inherits systemd backend)
- Document three-jail setup and REQUEST log format in README
- Add review #007 covering connection lifecycle, logging, and deployment
drift triggered by the 2026-07-24 FD exhaustion incident
Extract shared read_and_validate_config() with before/after mtime check
to detect mid-write config file changes. Add ReloadError enum with
FileChangedDuringRead variant. Return HTTP 409 Conflict on mtime change
from admin reload endpoint. Store cli_allow_wildcard_bind in
ConfigReloadHandle and use it in reload() validation instead of hardcoded
false. Update all ConfigReloadHandle::new() call sites.
Remove src/admin/socket.rs and replace with Bearer token HTTP auth on the
health check listener (port 9900). New src/admin/auth.rs provides SHA-256
key hashing with constant-time comparison; src/admin/handler.rs implements
/admin/reload, /admin/status, and /admin/rotate-key. Admin endpoints return
404 when disabled (empty admin_key_path), 401 on bad auth. Config field
renamed admin_socket_path → admin_key_path. Deployment files updated for
key file mount instead of socket directory.
Security review #005 identified critical vulnerabilities in the Unix domain
socket admin API (C1 symlink race, C2 no auth, C3 info leak, W1-W7, S1-S6).
ADR-028 (already accepted) replaces the socket with an authenticated HTTP
admin API on the health check port. This commit adds the remaining spec work:
- ADR-029: Config file TOCTOU mitigation (mtime check on reload)
- ADR-030: Store cli_allow_wildcard_bind in ConfigReloadHandle for consistent
reload validation
- Implementation tasks for the admin HTTP migration (fix/admin-http-api),
TOCTOU fix (fix/config-reload-toctou), and wildcard flag fix
(fix/wildcard-flag-reload)
- Updated review #005 status to resolved with per-finding disposition
- Resolved OQ-16: POST for state-changing admin endpoints, GET for read-only
- Updated all architecture docs to reference new ADRs, use admin_key_path
instead of admin_socket_path, and reflect POST method for /admin/reload
Add host part validation to is_valid_upstream: IPv4/IPv6 addresses must parse
as valid IpAddr, bracket-enclosed hosts must parse as IPv6, DNS names must
pass is_valid_hostname. Previously, values like '!!!bad!!!:3000' would pass.
Removed #[non_exhaustive] from TlsMode and the wildcard _ arm in the
match tls_mode block in main.rs. Since setup_tls already rejects unknown
modes with bail!, the wildcard was unreachable dead code. Removing it
ensures the compiler catches future TlsMode variant additions. Added
defense-in-depth count mismatch check after the acceptor loop to catch
any silent listener/acceptor mismatch from zip truncation.
glm-5.3-flash
glm-5.2
glm-5.1