mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-22 15:06:32 +00:00
fix(server): keep idle connections open longer than Traefik's upstream timeout
The 65 s default sat above nginx's 60 s but below the 90 s Traefik keeps an idle upstream connection, so a default Traefik deployment could still reuse a socket Node had just closed and answer that request with a 502. 95 s clears both, and the headers timeout one second above it stays well under Node's 300 s request timeout and the 290000 cap the schema accepts. Also moves the stray RawEnv import in boot-derive.ts up to the other import.
This commit is contained in:
+1
-1
@@ -24,7 +24,7 @@ ALLOWED_ORIGINS=https://trek.example.com # Comma-separated origins for CORS and
|
||||
FORCE_HTTPS=false # Optional. When true: HTTPS redirect + HSTS + CSP upgrade-insecure-requests + secure cookies. Only behind a TLS proxy.
|
||||
# HSTS_INCLUDE_SUBDOMAINS=false # When true: adds includeSubDomains to the HSTS header. Only effective when HSTS is active (FORCE_HTTPS=true or NODE_ENV=production). Leave false if you run other services on sibling subdomains over plain HTTP.
|
||||
COOKIE_SECURE=true # Auto-derived (true when NODE_ENV=production or FORCE_HTTPS=true). Set false to force cookies over plain HTTP.
|
||||
# HTTP_KEEP_ALIVE_TIMEOUT_MS=65000 # How long an idle keep-alive connection stays open. Keep it above your reverse proxy's upstream idle timeout (60 s for nginx and most load balancers, 90 s for Traefik) or the proxy answers some requests with a 502. 1000 to 290000. (default: 65000)
|
||||
# HTTP_KEEP_ALIVE_TIMEOUT_MS=95000 # How long an idle keep-alive connection stays open. Keep it above your reverse proxy's upstream idle timeout (60 s for nginx and most load balancers, 90 s for Traefik) or the proxy answers some requests with a 502. 1000 to 290000. (default: 95000)
|
||||
TRUST_PROXY=1 # Trusted proxy hops (parseInt or 1). Active in production by default; off in dev unless set. Needed for FORCE_HTTPS.
|
||||
ALLOW_INTERNAL_NETWORK=false # Allow outbound requests to private/RFC1918 IPs (e.g. Immich hosted on your LAN). Loopback and link-local addresses are always blocked.
|
||||
# ALLOW_LINK_LOCAL_IPS=169.254.1.2 # Single link-local addresses TREK may reach after all, comma-separated, e.g. the host gateway of a rootless Podman container. OIDC reaches a listed address directly; other integrations also need ALLOW_INTERNAL_NETWORK=true. 169.254.169.x and 169.254.170.x (cloud metadata) cannot be listed.
|
||||
|
||||
+1
-1
@@ -96,7 +96,7 @@ SQLite is the engine today and Postgres is the planned second one. Code under `s
|
||||
- **Auth is default-deny**: the global auth guard and the MFA policy guard run as `APP_GUARD`s, followed by the managed-install guard (`managed.guard.ts`), which refuses the routes a centrally administered install withholds from its own admin — it runs last so strangers still get a 401, never a route-revealing 403. Routes opt out with `@Public()`/`@OptionalAuth()`. The global guard stands down for routes declaring their own guard chain, so `@UseGuards(JwtAuthGuard, TripAccessGuard)` + `@RequirePermission` stacks work. Session is a cookie (`trek_session`); `@CurrentUser()` reads the user. **Never gate a multipart upload with a guard** — guards run before the parser and the client sees ECONNRESET instead of your 403; check in the handler.
|
||||
- **Logging**: `src/nest/audit/audit-log.logger.ts` (`logInfo`/`logWarn`/`logError`/`logDebug`) prints each line to the console and queues it for `data/logs/trek.log`, which `log-file.ts` appends in batches off the event loop and rotates at 10 MB over five files; `index.ts` flushes what is queued on exit. Never write the log file synchronously from a request path.
|
||||
- **Request correlation** (`src/nest/common/request-correlation.ts`): every HTTP request runs under an id (a well-formed incoming `X-Request-Id`, else a fresh UUID) that the response carries in `X-Request-Id`, and every log line written inside it is tagged `[http <id>]`. The exception filter leaves a 5xx on `res.locals` so the access log writes the request line and the stack as one entry. The other entry points (MCP tool/resource/prompt calls via the registry's `around` seam, WebSocket messages in `TrekWsAdapter`, plugin RPC dispatches in the supervisor, cron ticks in `CronRegistrarService`) each run under their own id through `traceEntry()` (`nest/audit/entry-trace.logger.ts`), which logs one line per call: debug on success, debug as `skipped` when the call declines its work on purpose (a cron tick whose lease another process holds), warn or error with the reason on failure. A new entry point goes through `traceEntry()` too.
|
||||
- **Process safety nets** (`src/index.ts`, `src/shutdown.ts`): `unhandledRejection` and `uncaughtException` are logged to trek.log and end in the same orderly shutdown as SIGTERM, with exit code 1; a second fatal error during it exits at once. A shutdown first marks `ReadinessService` (`nest/health/`) as draining, so `/api/health/ready` answers 503 for the whole drain. The HTTP server keeps idle keep-alive connections for `HTTP_KEEP_ALIVE_TIMEOUT_MS` (65 s, above the usual proxy idle timeout) with the headers timeout one second above it (`applyProxyTimeouts` in `bootstrap.ts`).
|
||||
- **Process safety nets** (`src/index.ts`, `src/shutdown.ts`): `unhandledRejection` and `uncaughtException` are logged to trek.log and end in the same orderly shutdown as SIGTERM, with exit code 1; a second fatal error during it exits at once. A shutdown first marks `ReadinessService` (`nest/health/`) as draining, so `/api/health/ready` answers 503 for the whole drain. The HTTP server keeps idle keep-alive connections for `HTTP_KEEP_ALIVE_TIMEOUT_MS` (95 s, above the 60 s of nginx and the 90 s of Traefik) with the headers timeout one second above it (`applyProxyTimeouts` in `bootstrap.ts`).
|
||||
- **Process-local state sits behind injectable ports**, each with an in-memory implementation that keeps today's behaviour: `RateLimitStore` (`nest/common/rate-limit.store.ts`, provided by `RateLimitModule`), `PendingCodeStore` (`nest/oauth/oauth.pending-codes.ts`), `OidcFlowStore` (`nest/oidc/oidc-flow.store.ts`), `PermissionsCacheStore` (`nest/permissions/permissions-cache.ts`) and `RoomRegistry` (`nest/realtime/ws-state.ts`). The owning module provides the implementation. Every port a shared store could back returns promises (a store in the database or on the network is async); `RoomRegistry` stays synchronous because membership holds live sockets, and several processes would need a fan-out in front of `broadcast()` instead. The process-wide ports (`PendingCodeStore`, `PermissionsCacheStore`, `RoomRegistry`) also have readers outside the container (the pending-code sweep, the backup restore's cache flush, the broadcast functions, hand-built consumers), so their module installs the store the container resolved in a `ProcessStoreSlot` (`nest/common/process-store-slot.ts`) those readers use: overriding the provider changes every reader. New process-local state follows the same shape; TREK still runs as one process.
|
||||
- **`src/middleware/globalMiddleware.ts`** is the only Express middleware left (helmet/CSP, CORS, HSTS, forced-HTTPS, logging, cookies).
|
||||
|
||||
|
||||
@@ -8,16 +8,18 @@
|
||||
* each side touches. A value that pre-container code needs, or that tests
|
||||
* mutate mid-lifetime, belongs in derive.ts instead.
|
||||
*/
|
||||
import type { RawEnv } from './derive';
|
||||
import { parseBool, positiveIntOr } from './parsers';
|
||||
|
||||
/**
|
||||
* Longer than the idle timeout of the common reverse proxies and load
|
||||
* balancers (60 s), so the proxy closes an idle upstream connection before
|
||||
* Node does. The other way round the proxy reuses a socket Node just closed
|
||||
* and answers that request with a 502.
|
||||
* Longer than the upstream idle timeout of the common reverse proxies and load
|
||||
* balancers (60 s for nginx and most load balancers, 90 s for Traefik), so the
|
||||
* proxy closes an idle upstream connection before Node does. The other way
|
||||
* round the proxy reuses a socket Node just closed and answers that request
|
||||
* with a 502. The headers timeout follows one second above, still well under
|
||||
* Node's 300 s request timeout.
|
||||
*/
|
||||
export const DEFAULT_KEEP_ALIVE_TIMEOUT_MS = 65_000;
|
||||
import type { RawEnv } from './derive';
|
||||
export const DEFAULT_KEEP_ALIVE_TIMEOUT_MS = 95_000;
|
||||
|
||||
/** What the pre-init Express layer and the HTTP server freeze when buildApp() runs. */
|
||||
export function deriveHttpBoot(raw: RawEnv) {
|
||||
|
||||
@@ -50,15 +50,15 @@ describe('AppConfigModule in the real buildApp()', () => {
|
||||
});
|
||||
|
||||
it("the HTTP server outlasts a proxy's idle timeout, with the headers timeout above it", async () => {
|
||||
app = await buildApp();
|
||||
expect(getHttpServer().keepAliveTimeout).toBe(65_000);
|
||||
expect(getHttpServer().headersTimeout).toBe(66_000);
|
||||
await app.close();
|
||||
|
||||
process.env.HTTP_KEEP_ALIVE_TIMEOUT_MS = '95000';
|
||||
app = await buildApp();
|
||||
expect(getHttpServer().keepAliveTimeout).toBe(95_000);
|
||||
expect(getHttpServer().headersTimeout).toBe(96_000);
|
||||
await app.close();
|
||||
|
||||
process.env.HTTP_KEEP_ALIVE_TIMEOUT_MS = '120000';
|
||||
app = await buildApp();
|
||||
expect(getHttpServer().keepAliveTimeout).toBe(120_000);
|
||||
expect(getHttpServer().headersTimeout).toBe(121_000);
|
||||
});
|
||||
|
||||
it('boot-stable snapshots re-derive per app build (mutate → rebuild → new value)', async () => {
|
||||
|
||||
@@ -76,9 +76,9 @@ describe('deriveHttp', () => {
|
||||
expect(deriveHttpBoot({}).trustProxy).toBe(1);
|
||||
});
|
||||
|
||||
it("HTTP_KEEP_ALIVE_TIMEOUT_MS: 65 s unless set (above the common proxies' 60 s idle timeout)", () => {
|
||||
expect(deriveHttpBoot({}).keepAliveTimeoutMs).toBe(65_000);
|
||||
expect(deriveHttpBoot({ HTTP_KEEP_ALIVE_TIMEOUT_MS: '95000' }).keepAliveTimeoutMs).toBe(95_000);
|
||||
it("HTTP_KEEP_ALIVE_TIMEOUT_MS: 95 s unless set (above nginx's 60 s and Traefik's 90 s idle timeout)", () => {
|
||||
expect(deriveHttpBoot({}).keepAliveTimeoutMs).toBe(95_000);
|
||||
expect(deriveHttpBoot({ HTTP_KEEP_ALIVE_TIMEOUT_MS: '120000' }).keepAliveTimeoutMs).toBe(120_000);
|
||||
});
|
||||
|
||||
it('boolean switches accept the whole boolean-like family (unified semantics)', () => {
|
||||
|
||||
@@ -170,7 +170,7 @@ full explanation.
|
||||
| `FORCE_HTTPS` | When `true`: 301-redirects HTTP→HTTPS, sends HSTS (`max-age=31536000`), adds CSP `upgrade-insecure-requests`, forces cookie `secure` flag. Only useful behind a TLS proxy. Your proxy must send `X-Forwarded-Proto: https`. | `false` |
|
||||
| `HSTS_INCLUDE_SUBDOMAINS` | When `true`: adds the `includeSubDomains` directive to the HSTS header, extending HTTPS enforcement to all subdomains. Only effective when HSTS is active (`FORCE_HTTPS=true` or `NODE_ENV=production`). Leave `false` if you run other services on sibling subdomains over plain HTTP. | `false` |
|
||||
| `TRUST_PROXY` | Number of trusted proxy hops. Tells Express how far into `X-Forwarded-For` to look for the real client IP, and where to read `X-Forwarded-Proto`. Count your hops: with two proxies in front of TREK and `TRUST_PROXY=1`, the IP on every audit row is the inner proxy's. `0` trusts nothing and always uses the socket address. Not required for the `FORCE_HTTPS` redirect, which reads the `X-Forwarded-Proto` header directly — set it for correct client IPs in the audit log, and for the `COOKIE_SECURE` auto-derivation, which goes through `req.secure`. | `1` (production) |
|
||||
| `HTTP_KEEP_ALIVE_TIMEOUT_MS` | How long (in milliseconds) the server keeps an idle keep-alive connection open; the headers timeout follows one second above it. Keep it above your reverse proxy's upstream idle timeout (60 s for nginx and most load balancers, 90 s for Traefik), otherwise the proxy reuses a connection the server just closed and answers that request with a 502. Between `1000` and `290000`. | `65000` |
|
||||
| `HTTP_KEEP_ALIVE_TIMEOUT_MS` | How long (in milliseconds) the server keeps an idle keep-alive connection open; the headers timeout follows one second above it. Keep it above your reverse proxy's upstream idle timeout (60 s for nginx and most load balancers, 90 s for Traefik), otherwise the proxy reuses a connection the server just closed and answers that request with a 502. Between `1000` and `290000`. | `95000` |
|
||||
| `COOKIE_SECURE` | Controls the `secure` flag on the `trek_session` cookie. Auto-derived as `true` when `NODE_ENV=production`, when `FORCE_HTTPS=true`, or when the request itself arrived over TLS on the outermost hop (`X-Forwarded-Proto: https` with `TRUST_PROXY` set). Set to `false` only as an escape hatch for LAN testing without TLS — not recommended in production. | auto |
|
||||
|
||||
> **Warning:** `FORCE_HTTPS=true` behind a proxy that does not forward `X-Forwarded-Proto: https` causes a
|
||||
|
||||
Reference in New Issue
Block a user