reverse-proxy/docs/reviews/007-connection-lifecycle-and-deployment-drift.md

status, last_updated, reviewed_code, reviewer, based_on, trigger

status	last_updated	reviewed_code	reviewer	based_on	trigger
open	2026-07-28	src/server.rs src/proxy/handler.rs src/proxy/mod.rs src/main.rs src/shutdown.rs src/logging/mod.rs src/logging/format.rs src/config/static_config.rs src/config/mod.rs src/health.rs src/rate_limit/mod.rs deploy/Dockerfile deploy/docker-compose.yml deploy/fail2ban/jail.d/reverse-proxy.conf deploy/fail2ban/filter.d/reverse-proxy.conf deploy/fail2ban/filter.d/reverse-proxy-4xx.conf deploy/fail2ban/filter.d/reverse-proxy-badbots.conf	code-reviewer	docs/reviews/006-attack-surface-review.md	Production incident (reverse-proxy FD exhaustion, 2026-07-24) and follow-up fail2ban audit

Operational Review #007 — Connection Lifecycle, Logging, and Deployment Drift

Purpose

This review was triggered by a production incident where the reverse-proxy container exhausted its 1024 file-descriptor limit after ~6 weeks of uptime, making a public Gitea instance unresponsive. A subsequent fail2ban audit revealed that the HTTP-level ban protection had been silently broken since the nginx→reverse-proxy migration (0 bans ever from the reverse-proxy jail).

The review examines the current HEAD (f6e62a3, 2026-06-15) source for connection-lifecycle, logging, and deployment issues that contributed to the incident or were uncovered during the audit. It also catalogs the gap between a deployed binary (built from cfe0ae5, 2026-06-15 — before the admin socket removal) and HEAD, since that production server has not yet been updated.

Findings are grouped into:

• Critical — caused or directly contributed to the production outage
• Warning — operational risks that will recur or cause future outages
• Suggestion — improvements to robustness, observability, or maintainability

Each finding includes the code location in HEAD, why it matters, and a recommended fix. Findings marked [new code required] need source changes; those marked [deploy existing] are already fixed in HEAD and only require shipping a new image.

---

Critical Findings

C1. No server-side idle/keep-alive timeout on TLS connections [new code required]

Location: src/server.rs:102-125

The HTTPS listener spawns a task per accepted connection and calls serve_connection / serve_connection_with_upgrades on the hyper builder. Neither the http2::Builder (line 103) nor the auto::Builder (line 112) sets any keep-alive or idle timeout:

// line 102-110 (HTTP/2 path)
let mut builder = hyper::server::conn::http2::Builder::new(TokioExecutor::new());
if let Err(e) = builder
    .enable_connect_protocol()
    .serve_connection(io, svc)   // no keep_alive_interval / keep_alive_timeout
    .await
{ ... }

// line 111-125 (HTTP/1.1 + auto path)
let mut builder = hyper_util::server::conn::auto::Builder::new(TokioExecutor::new());
builder.http2().enable_connect_protocol();
if let Err(e) = builder
    .serve_connection_with_upgrades(io, svc)   // no http1 keep_alive_timeout
    .await
{ ... }

Impact: This is the primary root cause of the FD exhaustion. A connection that the client opens but never cleanly closes (half-open TCP, abandoned scanner socket, or a bot that stops sending without FIN) keeps its spawned task + socket FD alive forever. Over 6 weeks of bot traffic (a crawler blasting hundreds of /raw/commit/... and /blame/commit/... URLs), abandoned TLS connections accumulated until the process hit its 1024 soft FD limit. At the time of the incident, the container had exactly 1014 socket: FDs open — all held by spawned connection tasks with no timeout to reap them.

The proxy became unable to accept new connections:

ERROR reverse_proxy::server: failed to accept TCP connection error=No file descriptors available (os error 24)

Every new HTTPS request to the proxied site timed out, while the upstream service itself (on 127.0.0.1:3000) remained healthy — only the proxy was broken.

Recommendation: Set idle/keep-alive timeouts on both builders:

use std::time::Duration;
const SERVER_IDLE_TIMEOUT: Duration = Duration::from_secs(60);

// HTTP/2 path
let mut builder = hyper::server::conn::http2::Builder::new(TokioExecutor::new());
builder
    .keep_alive_interval(Some(Duration::from_secs(15)))
    .keep_alive_timeout(SERVER_IDLE_TIMEOUT)
    .enable_connect_protocol()
    .serve_connection(io, svc)

// HTTP/1.1 + auto path
let mut builder = hyper_util::server::conn::auto::Builder::new(TokioExecutor::new());
builder
    .http1().keep_alive_timeout(SERVER_IDLE_TIMEOUT)
    .http2().keep_alive_interval(Some(Duration::from_secs(15)))
        .keep_alive_timeout(SERVER_IDLE_TIMEOUT)
    .enable_connect_protocol();
builder.serve_connection_with_upgrades(io, svc)

This makes idle connections close after 60s of inactivity (with HTTP/2 keep-alive pings at 15s to detect dead peers faster). The timeout should be configurable via StaticConfig (e.g. connection_idle_timeout_secs, default 60).

Severity rationale: This caused a complete outage of the public Gitea instance. It will recur on any deployment with sustained traffic, with a time-to-failure proportional to the FD limit and traffic pattern.

---

C2. No concurrency cap on accepted connections [new code required]

Location: src/server.rs:65-127

The accept loop calls tokio::spawn for every accepted connection with no bound on the number of concurrent tasks:

loop {
    tokio::select! {
        accept_result = tcp_listener.accept() => {
            // ...
            tokio::spawn(async move {
                let _guard = InFlightGuard::new(in_flight.clone());
                // ... serve connection ...
            });
        }
        // ...
    }
}

The InFlightCounter (lines 17-54) only counts active connections for graceful-shutdown draining — it does not limit them. There is no semaphore, no max_connections config, and no backpressure on the accept loop. An attacker (or a misbehaving crawler) can open thousands of simultaneous TLS connections, each consuming a task + FD + TLS state, with nothing to stop the process from exhausting FDs or memory.

This is the same class of issue flagged as W1 in review #005 ("No connection concurrency limit"), but that finding was scoped to the admin socket and declared "eliminated" by removing the admin socket (ADR-028, line 168). The public HTTPS listener was never bounded. Review #006 (attack surface review) section 1.1 noted "No resource limiting beyond OS TCP backlog" but did not escalate it.

Recommendation: Add a tokio::sync::Semaphore that gates connection acceptance, with a configurable max_connections (default e.g. 1024):

use tokio::sync::Semaphore;
use std::sync::Arc;

let conn_sem = Arc::new(Semaphore::new(max_connections));

loop {
    tokio::select! {
        accept_result = tcp_listener.accept() => {
            let (tcp_stream, remote_addr) = // ...;
            let permit = conn_sem.clone().acquire_owned().await.unwrap();
            let in_flight = in_flight.clone();
            let tls_acceptor = tls_acceptor.clone();
            let router = router.clone();

            tokio::spawn(async move {
                let _guard = InFlightGuard::new(in_flight.clone());
                let _permit = permit;  // released when task ends
                // ... serve connection ...
            });
        }
        // ...
    }
}

The semaphore acquire_owned().await blocks the accept loop when all permits are taken, providing natural backpressure (the OS TCP backlog holds pending connections). Combined with C1's idle timeout, this bounds both concurrent and leaked connections.

Severity rationale: Without this, the FD limit is the only backstop, and hitting it takes down the entire service. With C1 fixed, this becomes less urgent, but it's still needed for defense against intentional FD exhaustion attacks and to prevent memory unbounded growth under load.

---

Warning Findings

W1. No SIGHUP log-reopen support (sparse-file hazard) [new code required]

Location: src/shutdown.rs:70-73, src/logging/mod.rs:36-37,74-75

SIGHUP is handled exclusively as a config-reload signal (shutdown.rs:70-73):

SIGHUP => {
    tracing::info!(event = "SIGNAL", signal = "SIGHUP");
    handle_sighup_reload(&reload_handle, &config_path).await;
}

The log file is opened once at startup with File::create(path) and held as an Arc<File> writer (logging/mod.rs:36-37,74-75). There is no mechanism to reopen the log file — not on SIGHUP, not on any signal.

Impact: This breaks logrotate and any external log rotation that relies on moving/renaming the file. The standard approaches are:

1. copytruncate — logrotate copies the file, then truncates it to zero. The process keeps its FD and offset. But: if the process's FD offset is large (e.g. the file was 1GB), truncate -s 0 resets the file size to 0 while the process's FD offset stays at 1GB. The next write lands at offset 1GB, creating a sparse file with a 1GB hole. This is exactly what happened during the incident response: after truncate -s 0 on the 1.15GB access log, the file immediately reported 1.15GB apparent size (with only 8KB of real blocks). Fail2ban then tried to scan this "1.15GB" file, loaded the sparse hole into memory, and bloated to 7.3GB RSS, wedging the fail2ban server.
2. postrotate with signal — logrotate renames the file, then signals the process to reopen. This is the standard nginx/apache pattern. But the proxy doesn't support log-reopen on any signal (SIGHUP is config reload).

The production logrotate config currently uses copytruncate with maxsize 100M to bound the offset growth, but this is a workaround, not a fix. The sparse file will still appear whenever the log is truncated while the offset is high.

Recommendation: Add a log-reopen signal handler. Options:

• Option A (preferred): Use SIGUSR1 for log reopen (the conventional choice, used by nginx, Apache, etc.). Keep SIGHUP for config reload. The handler closes the old Arc<File> and opens a new one at the same path, swapping it atomically. This enables standard postrotate logrotate configs without copytruncate.

• Option B: Reuse SIGHUP to do both config reload AND log reopen. Less clean (conflates two operations) but avoids adding a new signal handler.

In either case, the Arc<File> writer in logging/mod.rs needs to become an Arc<ArcSwap<File>> or similar so the reopen can swap the writer atomically without restarting the tracing subscriber. The tracing_subscriber MakeWriter trait is the clean way to support this (implement a custom writer that holds an Arc<ArcSwap<File>> and reads the current file on each write).

Severity rationale: This caused the fail2ban wedge during incident response and makes log rotation fragile. It won't cause the original outage on its own, but it complicates recovery and will recur on every log rotation.

---

W2. Upstream client pool has no max-idle-per-host bound [new code required]

Location: src/proxy/handler.rs:232-258

Both the HTTP and HTTPS upstream clients set pool_idle_timeout(90s) but do not set pool_max_idle_per_host:

pub fn create_http_client() -> Client<HttpConnector, Body> {
    let mut connector = HttpConnector::new();
    connector.set_connect_timeout(Some(Duration::from_secs(CONNECT_TIMEOUT_CEILING_SECS)));
    Client::builder(TokioExecutor::new())
        .pool_idle_timeout(Duration::from_secs(90))
        // no .pool_max_idle_per_host(...)
        .build(connector)
}

hyper-util's default pool_max_idle_per_host is unbounded (or very high, depending on version). For a single-upstream deployment (e.g. a proxy where everything routes to 127.0.0.1:3000), every concurrent client request can leave an idle connection in the pool, and all of them are to the same host. Under burst traffic (e.g. a crawler hitting 20+ URLs in rapid succession), the pool accumulates idle upstream connections, each holding a socket FD on the upstream side.

This is a secondary contributor to the FD exhaustion — the 1014 leaked FDs were primarily client-facing TLS sockets (C1), but upstream pool connections also consume FDs and have no bound.

Recommendation: Set pool_max_idle_per_host to a reasonable value (e.g. 10-20) on both clients:

Client::builder(TokioExecutor::new())
    .pool_idle_timeout(Duration::from_secs(90))
    .pool_max_idle_per_host(10)
    .build(connector)

This bounds the upstream pool to at most 10 idle connections to any single upstream, which is plenty for a single-user Gitea instance.

---

W3. No connection idle timeout is configurable [new code required]

Location: src/config/static_config.rs (no such field)

There is no config option for server-side connection idle timeout. Even after C1 is fixed with hardcoded timeouts, operators should be able to tune this without a rebuild. A site serving long-lived HTTP/2 streams (e.g. Gitea's event-source endpoints) might need a longer timeout than a static-file site.

Recommendation: Add connection_idle_timeout_secs to StaticConfig (default 60), and use it for the keep-alive timeouts in C1. Document it in the config schema and README.

---

W4. No max_connections is configurable [new code required]

Location: src/config/static_config.rs (no such field)

Related to C2. The concurrency cap should be operator-tunable, not hardcoded. A high-traffic deployment might need more than 1024; a low-memory deployment might want fewer.

Recommendation: Add max_connections to StaticConfig (default 1024), wire it to the Semaphore in C2.

---

W5. Deployed binary predates admin socket removal — config format drift [deploy existing]

Location: Production /etc/reverse-proxy/config.toml vs src/config/static_config.rs at HEAD

The production binary was built from cfe0ae5 (2026-06-15 05:23 UTC), which is the commit immediately before 3ea3f56 ("Replace Unix socket admin API with authenticated HTTP admin API"). The live config still uses:

admin_socket_path = "/run/reverse-proxy/admin.sock"

At HEAD (f6e62a3), this field was replaced with:

admin_key_path = "/etc/reverse-proxy/admin-key"

The deployed binary has the Unix socket admin API (with the vulnerabilities documented in review #005: symlink race C1, no auth C2, info leak C3, no concurrency limit W1). HEAD replaced it with the authenticated HTTP admin API (ADR-028).

Commits in HEAD not yet deployed (4 commits, 3ea3f56..f6e62a3):

1. 3ea3f56 — Replace Unix socket admin API with authenticated HTTP admin API
2. c6dda71 — Add mtime TOCTOU check and wildcard flag to ConfigReloadHandle (ADR-029/030: fixes W2 config-reload TOCTOU and W5 wildcard-flag drift from review #005)
3. 143ebaa — Add wildcard bind acceptance/rejection tests for reload path
4. f6e62a3 — Update README/AGENTS.md for admin HTTP API

Impact: The production server is running with known admin-API vulnerabilities (review #005 C1-C3) that are already fixed in source. The config file uses a field name that HEAD doesn't recognize, so deploying HEAD as-is would break admin functionality until the config is migrated.

Recommendation: When deploying the new image (after C1/C2/W1-W4 are fixed), the config migration is:

1. Replace admin_socket_path = "..." with admin_key_path = "/etc/reverse-proxy/admin-key" (or set to empty string "" to disable admin endpoints)
2. If enabling admin: openssl rand -hex 32 > /etc/reverse-proxy/admin-key && chmod 600 /etc/reverse-proxy/admin-key
3. Remove the /run/reverse-proxy volume mount from docker-compose (no longer needed for the admin socket)
4. Add /etc/reverse-proxy/admin-key:/etc/reverse-proxy/admin-key:ro volume mount if admin is enabled

This is a deploy existing fix — no new code needed, just ship HEAD (plus the C1/C2/W1-W4 fixes) and migrate the config.

---

W6. fail2ban jail backend mismatch (already fixed in project, but worth documenting) [deploy existing]

Location: deploy/fail2ban/jail.d/reverse-proxy.conf (already updated)

The original jail config did not set backend =, so it inherited backend = systemd from /etc/fail2ban/jail.d/defaults-debian.conf. The systemd backend ignores logpath and reads journald — but the reverse-proxy uses Docker's json-file driver and writes to a file, not journald. Result: the jail silently matched nothing (0 bans ever despite 3830 RATE_LIMIT lines in the access log).

This is now fixed in the project's deploy/fail2ban/ files (backend = auto added). Documented here so future deployments don't regress.

Recommendation: Already addressed. No further action needed beyond ensuring the updated deploy/fail2ban/ files are used in future deployments.

---

Suggestion Findings

S1. Add FD usage / connection count metrics [new code required]

Location: src/server.rs (InFlightCounter), src/health.rs

The InFlightCounter already tracks active connections for graceful shutdown. Expose it via the /health endpoint (or a new /metrics endpoint) so operators and monitoring can see connection count trends. Additionally, log periodic FD usage (read /proc/self/fd count on Linux) so FD exhaustion is detected before it hits the limit.

Recommendation:

• Add in_flight_connections to the /health JSON response
• Add a periodic (every 60s) info log line: in_flight=N fds=M where M is the count of open FDs from /proc/self/fd
• Optionally add a warning log when FDs exceed 80% of the soft limit

This would have given early warning of the FD exhaustion weeks before the outage.

---

S2. Default rate limit (10 rps / burst 20) is permissive for single-user Gitea [config, no code change]

Location: Production /etc/reverse-proxy/config.toml

The live config has requests_per_second = 10, burst = 20. This is per-IP, so a crawler gets 20 burst + 10/s indefinitely before seeing a 429. For a single-user Gitea instance, this is generous — legitimate traffic (git clone, web UI browsing) rarely exceeds 2-3 rps.

The permissive rate limit meant the crawler could blast hundreds of requests per minute, each opening a TLS connection that (per C1) never timed out, accelerating the FD exhaustion. With C1 fixed, the impact is reduced, but tighter rate limits would further reduce crawler pressure and the volume of RATE_LIMIT log lines fail2ban must process.

Recommendation: Consider requests_per_second = 3, burst = 5 for single-user deployments. This is a config change, not a code change, and should be tuned per deployment. The fail2ban reverse-proxy jail (maxretry=10, findtime=60s) will ban IPs that consistently exceed even the tighter limit.

---

S3. Log file grows unbounded between logrotate runs [config/ops, partially addressed]

Location: Production /etc/logrotate.d/reverse-proxy

The logrotate config uses maxsize 100M with copytruncate. This bounds the file to ~100MB before rotation. However, as documented in W1, copytruncate on a file with a high FD offset creates a sparse file. The logrotate will trigger on the 100MB apparent size, but the actual disk usage may be far smaller (sparse hole), and the truncated file will immediately re-report a large apparent size.

Recommendation: This is fully resolved by W1 (SIGHUP log-reopen + postrotate logrotate). Until W1 is implemented, the workaround is to restart the proxy in a logrotate postrotate script:

postrotate
    docker restart reverse-proxy 2>/dev/null || true
endscript

This is heavy-handed (brief downtime on each rotation) but avoids the sparse file problem. The proper fix is W1.

---

S4. Document the connection-lifecycle config in the README [docs, after C1/W3/W4]

Location: README.md — Configuration section

Once connection_idle_timeout_secs (W3) and max_connections (W4) are added to StaticConfig, document them in the README's static config table and the architecture docs (docs/architecture/config.md).

Include a "Tuning for your traffic" section with guidance:

• Single-user / low-traffic: max_connections=256, connection_idle_timeout_secs=60
• Multi-user / high-traffic: max_connections=2048, connection_idle_timeout_secs=120
• Behind a load balancer: higher max_connections, shorter idle timeout

---

S5. Add integration test for connection cleanup under idle timeout [new code required, after C1]

Location: tests/integration_test.rs

After C1 is implemented, add an integration test that:

1. Opens a TLS connection to the proxy
2. Sends no data (or an incomplete request)
3. Waits for connection_idle_timeout_secs + 5s
4. Verifies the connection is closed (socket FD released)

This prevents regressions in the idle-timeout cleanup. The test can use rcgen for a self-signed cert (existing test pattern) and a raw TcpStream to control the connection lifecycle.

---

Deployment Drift Summary

The production server is running a binary built from cfe0ae5 (2026-06-15 05:23 UTC). HEAD is f6e62a3 (2026-06-15 06:34 UTC). The 4 commits between them are:

Commit	What it fixes	Deploy status
3ea3f56	Admin socket → authenticated HTTP API (review #005 C1-C3, W1)	Not deployed
c6dda71	Config reload mtime TOCTOU + wildcard flag drift (review #005 W2, W5)	Not deployed
143ebaa	Tests for reload wildcard path	Not deployed
f6e62a3	Docs for admin HTTP API	Not deployed

Additionally, the following source-level issues identified in this review are not yet fixed in HEAD and require new code:

Finding	Fix needed	Severity
C1 — no server idle timeout	keep_alive_timeout on builders	Critical
C2 — no concurrency cap	Semaphore on accept loop	Critical
W1 — no SIGHUP log reopen	SIGUSR1 handler + ArcSwap writer	Warning
W2 — no upstream pool max-idle	pool_max_idle_per_host on clients	Warning
W3 — no idle timeout config	StaticConfig field	Warning
W4 — no max_connections config	StaticConfig field	Warning
S1 — FD/connection metrics	health endpoint + periodic log	Suggestion
S5 — idle timeout integration test	new test	Suggestion

Recommended Migration Sequence

When the source fixes (C1, C2, W1-W4) are implemented and tested:

1. Build new image from HEAD+fixes
2. Migrate config:
   • admin_socket_path → admin_key_path (W5)
   • Add connection_idle_timeout_secs = 60 (W3)
   • Add max_connections = 1024 (W4)
   • Optionally tighten rate limit to 3 rps / burst 5 (S2)
3. Create admin key (if enabling admin API): openssl rand -hex 32 > /etc/reverse-proxy/admin-key
4. Update docker-compose: remove /run/reverse-proxy mount, add admin-key mount
5. Update logrotate: switch from copytruncate to postrotate + kill -USR1 $(pidof reverse-proxy) (W1)
6. Deploy: docker compose up -d (pulls new image, restarts with new config)
7. Verify: check health, FD count, fail2ban jails, make a test request
8. Monitor: watch the new FD/connection metrics (S1) for the first 24h

References

• Review #005 — admin socket vulnerabilities (fixed in HEAD by ADR-028)
• Review #006 — attack surface enumeration (noted "no resource limiting" at 1.1)
• ADR-028 — admin HTTP API replacement for Unix socket
• ADR-029/030 — config reload TOCTOU and wildcard flag