Commit Graph

90 Commits

Author SHA1 Message Date
Maurice 59adaf2f0c test(db): keep the legacy upgrade suites on SQLite by name
They boot on a SQLite file the retired positional runner built, which no
Postgres database ever came from, so they belong in SQLITE_ONLY rather
than in the snapshot baseline. The lane scan now tells such a suite apart
from the snapshot mock: it mocks src/db/database onto a database of its
own, never names createSnapshotTestDb, and is named in SQLITE_ONLY with
its reason. A snapshot suite added to SQLITE_ONLY stays in the baseline.
2026-11-17 17:55:42 +01:00
Maurice fa193a400e docs(db): every transaction goes through UnitOfWork, checks read inside it 2026-11-17 14:14:49 +01:00
Maurice 4eb2c3be58 docs(db): moving an existing install to Postgres, and the refusal of an empty database beside one 2026-11-17 13:41:21 +01:00
Maurice 8e85fb1c2c docs(db): plugin data on Postgres, its SQL rules and one TREK per database
The plugin-sdk README says what dialect a plugin writes when TREK runs on
Postgres, what the guard refuses and how the limits hold there. The wiki's
Postgres page describes the plugin schemas, their backup and restore and
the one-instance-per-database limit, and its key rotation path now names
the file where it is.
2026-11-17 13:00:55 +01:00
Maurice 20fb054d27 docs(db): Postgres 14 to 18, the database role, restore limits and the manual rollback
- Database-Postgres: supported versions, the version 18 client and the tool
  search order, sslrootcert with a client before 16, how to create a role
  that is not a superuser and move an older compose install to one, a
  current schema other than public, and the space a restore needs.
- Updating: the manual rollback clears the whole schema and loads the copy
  in one transaction, as TREK's restore does; dropping only the tables left
  the collation, the trigger functions and the sequences behind. A unit
  test keeps the printed statement equal to CLEAR_SCHEMA_SQL, and the boot's
  rollback hint points at that section.
- Troubleshooting, Install-Docker-Compose, Environment-Variables, README
  and server/CLAUDE.md follow.
2026-11-17 12:24:07 +01:00
Maurice 40f6efbf5b docs(db): describe the discovered Postgres lane and the 14/18 job 2026-11-17 12:05:17 +01:00
Maurice 16da5dd685 docs(db): the routine for a migration after the Postgres cut 2026-11-17 11:48:18 +01:00
Isaias Tavares 526437926a test(db): run the suites that boot the app on Postgres in their own lane
npm run test:pg --workspace=server (vitest.pg.config.ts) runs every suite
that boots the app through buildApp() on the real database module against
Postgres, each test file on its own copy of one template database the
engine's bootstrap migrates once per run. TREK_TEST_DB_URL names the server;
without it the lane fails instead of passing on nothing. The CI job
'Server: Postgres Tests' runs it on a postgres:17 service and is part of
ci-ok.

tests/postgres/ holds what only Postgres needs proving:
- PARITY-PG: MikroORM's comparator finds nothing between the bootstrapped
  database and the entities. What it reports that changes nothing (key
  columns it would make nullable, defaults and checks Postgres spells its
  own way, the hand-ported triggers) is proven so: the spelling statements
  run in a rolled-back transaction and every default and check must read
  the same after, and every ported trigger must exist.
- the write lock: concurrent read-then-insert units of work never see the
  same count, and counts are numbers; a search ignores case.

tours-hike and tours-plan-types stay on SQLite only: they move a faked
setTimeout on until a tool call ends, which a database over the network
does not finish within.
2026-11-04 22:45:32 -03:00
Isaias Tavares fc9e634d86 feat(db): back up, restore and copy a Postgres database before migrating it
On Postgres the backup port dumps with pg_dump and restores by turning the
dump into a script with pg_restore and having psql clear the current schema
and run it in one transaction. A dump only recreates what it holds, so a
plain pg_restore --clean would leave a later migration's tables behind and
break the migrations the reopen runs; a restore that fails leaves the
database as it was. A restore refuses a pg_restore or psql older than
17.6/16.10/15.14/14.19, whose script would lack the \restrict line that
keeps a psql meta-command smuggled into an uploaded dump from running
(CVE-2025-8714). A restore
refuses a SQLite archive, a dump without TREK's migration history or one a
newer TREK wrote before anything is replaced, and keeps a dump of the
database it replaces.

RESTORE_FROM_BACKUP loads the archive's dump the same way into a database
that holds no table yet, and is ignored with a warning once it holds one.

Before pending migrations the boot now takes a pg_dump copy to
data/pre-migrate-<last migration>-<time>.pgdump under the same
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) policy as the SQLite file, instead of
refusing; a copy that cannot be written refuses the boot.

The tools run by absolute path (TREK_DB_PG_BIN_DIR, or the usual
directories), never through PATH, with the password in PGPASSWORD; an
sslmode that asks for SSL verifies the certificate as the app pool does.
Dumps are written beside their target and renamed into place, readable by
their owner only, and a tool's stderr reaches errors without the lines that
quote SQL or rows. The image installs postgresql-client-17 from Debian 13.
2026-11-04 22:45:31 -03:00
Isaias Tavares 515caffa79 feat(db): keep SQLite's write and search semantics on Postgres
- UnitOfWork transactions take one advisory lock for the database on
  Postgres (DatabaseEngine.serializeWrites, pg_advisory_xact_lock), so a
  MAX(...) + 1 read inside one cannot race another, as with SQLite's single
  writer. SQLite needs nothing.
- ilike(platform, refs) builds the case-insensitive search condition: LIKE
  on SQLite, ILIKE on Postgres, where LIKE is exact. The place searches use it.
- Plugin data erasure runs each statement in its own savepoint: on Postgres a
  failed statement would abort the user's whole erasure transaction.
2026-11-04 22:45:14 -03:00
Isaias Tavares a77705984e feat(db): create and migrate the schema on a Postgres database
With TREK_DB_URL set, TREK now boots on Postgres; without it nothing changes.

- The database engine contract gains open, close, reopen and the schema
  bootstrap; DatabaseLifecycle delegates to the engine resolveEngine() picks.
- An empty Postgres database gets server/assets/postgres/baseline.sql
  (generated from the entities by gen:pg-baseline, with the reference rows
  the migrations up to its cut insert: tour types, provider catalogues,
  addons, app settings) and every migration up to its cut recorded as
  executed, in one transaction. A database holding tables
  without TREK's history is refused before anything is written; a migrated one
  runs the newer-database refusal and its pending migrations, which refuse
  without a copy unless TREK_DB_PRE_MIGRATE_SNAPSHOT=false.
- INTEGER columns are bigint on Postgres (SQLite's INTEGER is 64-bit: an
  epoch-millisecond column overflowed int4), and the engine's driver reads
  bigint and numeric back as numbers, refusing values past 2^53.
- The two SQLite triggers are ported to PL/pgSQL; check:pg-baseline fails
  while a SQLite trigger has no port, and while the baseline drifts from the
  entities before a later migration freezes it.
- Backups and restores on Postgres refuse until they can dump and load one.
- Migration helpers read the catalog per engine (dialect/catalog.ts), and
  the email case index no longer needs group_concat.
- CI runs check:pg-baseline and boots the image on postgres:17 twice.
2026-11-04 22:45:02 -03:00
Isaias Tavares 25f5e34558 feat(db): choose the database engine from TREK_DB_URL
A port for the engine TREK runs on, without changing what it runs on.
src/db/engine/ names the engine (SqliteEngine, PostgresEngine) and builds
the MikroORM config from it; src/mikro-orm.config.ts exports that config for
the CLI and the Nest container alike. Without TREK_DB_URL the config is the
one it was: the shared SQLite driver on TREK_DB_FILE or data/travel.db.

- TREK_DB_URL takes a postgres:// or postgresql:// URL and is masked in the
  boot report; TREK_DB_POOL_MAX sizes the pool (1 to 100, default 10).
  TREK_DB_URL beside TREK_DB_FILE, TREK_DB_JOURNAL_MODE or
  TREK_DB_SYNCHRONOUS refuses the boot as an ambiguous configuration.
- MikroORM builds the pg pool from the URL's host, port, user, password and
  database and drops the rest of the query, so the engine reads sslmode and
  sslrootcert itself. require, verify-ca and verify-full all verify the
  server's certificate, as the pg driver reads them; a mode that would not
  verify, or a misspelt one, refuses.
- Postgres cannot run TREK yet, so a boot with TREK_DB_URL is refused with
  that reason, in the boot check and again where the config is built, before
  anything connects.
- @mikro-orm/postgresql moves to dependencies; pg is plain JavaScript.
- lint:engine-dispatch (CI) holds the engine checks outside src/db/dialect/
  and src/db/engine/ (isSqlite(, isPostgres(, an instanceof on a platform, a
  comparison on the engine's name) to a baseline that starts and stays empty.
2026-11-04 22:45:02 -03:00
Isaias Tavares 62912e5db1 fix(db): run every repository statement the Postgres probe reaches
The probe's baseline is empty now: no repository method Postgres refuses,
none it cannot measure.

- reservations.accommodation_id (TEXT) is cast where it is compared with
  day_accommodations.id (INTEGER), in the four reservation reads and the two
  stay reads; SQLite coerced the bare comparison, Postgres refuses it. The
  joined RS18/RR1 read moves to reservation-join.ts beside the repository.
- The usage counters qualify the target row's column in DO UPDATE SET, which
  Postgres reads as ambiguous with `excluded` otherwise.
- ReservationDayPositions.upsertScoped upserts on its primary key instead of
  INSERT OR REPLACE; position is the only other column and nothing references
  the table, so the row left behind is the same.
- PRAGMA wal_checkpoint and VACUUM INTO move from MaintenanceRepository to
  SqliteMaintenance under src/db/engine/sqlite/, the SQLite engine's folder,
  where lint:dialect allows PRAGMA.
- The places list takes category and tag ids as numbers: the service converts
  the query text with toRowId and answers an empty list for one that is no
  id, instead of binding the text to an integer column.
- The probe samples the `present` flag of a partial update's pair as true, so
  the four update methods that sent no SQL are measured, and it now holds
  SQLSTATE class 22 too: a value of the wrong type reaching a column is the
  code's, since the sampler sends what each parameter declares.
2026-11-04 22:10:18 -03:00
Maurice bf26175724 docs(server): say which builds refuse a migrated database and which do not
The pre-migrate snapshot docstring said an older release refuses a database a newer TREK migrated. Only images from 5.0 on do; 4.3.x and earlier start on it without a warning. server/CLAUDE.md said anything the legacy baseline cannot map refuses the boot, which holds only above step 258.
2026-10-13 21:10:41 +02:00
Maurice ef68c62d80 build(server): the response-contract exempt marker only excuses a handler that writes through @Res()
A response-contract-exempt comment counted any handler as covered, whatever it returned, so one comment line let a JSON handler past the ratchet. The marker now counts only on a handler taking @Res() or @Next(). The five vacay holiday proxies return the provider's JSON, so they lose their markers and are counted again until they declare a schema.
2026-10-10 20:16:14 +02:00
Maurice 8fb888c8a3 build(server): lint:service-http counts envelopes in every domain file and every spelling of a branch on one
The ratchet only read *.service.ts for status literals and one branch shape in controllers and tools, so an envelope in a helper, impl or provider file, a branch written as result.error === 'not_found', and RPC handlers all went uncounted. It now counts the literals in every file under src/nest outside the HTTP layer, and comparisons, negations, optional chains and ternaries on a returned error in controllers, tools and RPC handlers. The baseline takes what the wider count finds.
2026-10-10 20:16:13 +02:00
Maurice ebd8a4f8c9 build(server): lint:boundaries counts the RPC and MCP layers as domains and sees getRepository calls
src/nest-rpc, src/nest-mcp and src/mcp import domains and are imported by them, but the domain map only knew src/nest, so their edges dropped out of domainCycles and their repository injections out of foreignRepositories. Each layer now counts as one domain. The repository rule reads the syntax tree for @InjectRepository(X) and for getRepository(X) on an EntityManager, so a comment no longer counts and a foreign read through the EntityManager no longer passes. The baseline takes the cycles and the repository reaches the wider view finds; settings -> Users goes, it was a comment.
2026-10-10 20:16:13 +02:00
Maurice bb6eb1ab89 fix(mcp): the six write tools that never had a demo check stay open to demo accounts under the central gate
sync_trip_documents and the five manual school-holiday tools carried no demo check of their own before the gate moved into the registry, so a demo account could call them. The gate now lets exactly those through, and a test pins the list. upload_trip_file keeps the gate's order: a demo account is refused before its trip access is checked, where the tool used to answer a trip it cannot see with the no-access result first.
2026-10-10 20:16:13 +02:00
Maurice 4427c435a8 ci(server): module mocks, hand-built services and test file length in tests/ may only shrink
lint:test-mocks counts vi.mock/vi.doMock of src/config, src/nest/realtime and src/db per file under tests/, lint:test-new-service counts new XService(/new XRepository( there with every class whose constructor takes nothing allowed (derived from src/ and tests/helpers/), and lint:size holds files under tests/ to 2000 lines against their own baseline. Both counters share scripts/lib/count-ratchet.mjs, which gains a prepare hook for a check that reads the tree first. The three run as steps of the existing server lint job; the CLAUDE.md files and the module guide describe the test helpers they point to.
2026-10-10 20:16:12 +02:00
Maurice fa0edfd714 test(server): suites record broadcasts on a FakeRealtimeService instead of mocking src/websocket
A hand-built service gets a FakeRealtimeService where it took a RealtimeService, a Nest testing module overrides the provider, and a suite over buildApp() spies on the container's instance (spyOnRealtime). The 167 vi.mock factories on src/websocket are gone, RealtimeService reads the socket registry in ws-state.ts directly, and the re-export stub src/websocket.ts is deleted. The facade's own suite now checks what reaches a socket instead of which module function it called.
2026-10-10 20:16:10 +02:00
Maurice 87071bf738 feat(server): @ResponseContract checks a handler's responses against its shared schema in tests
The decorator records the @trek/shared response schema a route answers
with. Under NODE_ENV=test it adds an interceptor that parses every response
and fails the request with a logged 500 naming the handler and the issues,
so the e2e and integration suites catch a response that drifted from its
contract; in production nothing is added to the request path and the body
is never rewritten. lint:response-contracts holds the handlers still without
one to a per-file baseline that only shrinks, and runs in server-quality.
2026-10-10 20:16:08 +02:00
Maurice 8d716aec2f refactor(budget): money arithmetic lives in shared and the settlement uses it
toMinor, sumMinor, splitEqualShares and currencyDecimals move to
shared/src/money. The budget service and its MCP tools call them instead of
private copies, and the share table both sides were pinned to becomes the
spec of the shared function. Stored values and every computed figure stay
the same.
2026-10-10 20:16:07 +02:00
Maurice e7c1acfb47 chore(server): lint:mcp-zod and lint:service-http hold inline tool zod and error envelopes to a shrinking baseline 2026-10-10 20:16:07 +02:00
Maurice dc42fe9a5a chore(server): lint:boundaries refuses a provider that imports the orchestrator above it 2026-10-10 20:16:02 +02:00
Maurice cd0c27921c build(server): lint:boundaries holds foreign repository injections to a baseline
A domain injecting the repository of a table another domain owns couples
the two below the service layer, and nothing counted it. The new
foreignRepositories rule reads every @InjectRepository under src/nest,
looks the entity up in scripts/repository-owners.json (an injected entity
without an owner stops the run) and holds each domain/entity pair to the
baseline, which only shrinks: 209 pairs today.
2026-10-10 20:16:00 +02:00
Maurice 1a9855cf70 refactor(mcp): the demo-mode write block runs once, in the registry's tool gate
Every write tool used to open with its own demo check, 196 copies across 30
tool files, most of them injecting AuthService for nothing else. The
registry now takes a toolGate (built from the container) that runs before
each registered tool handler, and TREK's gate refuses a demo account on
every tool not annotated read-only with the same canned result. The
per-tool checks, the AuthService injections, AuthService.isDemoUser and the
AuthModule imports that only fed them are gone; ESLint refuses a demo check
in a tool file, and a registry-wide test calls every write tool as the
demo account.
2026-10-10 20:15:57 +02:00
Maurice 3b58694eb9 refactor(settings): instance settings are named through one typed register
app-settings.registry.ts lists every app_settings key the server reads by name with its shape, default, environment override and flags. The admin form's keys, its masked and encrypted handling and the public config's reads derive from it, isOidcConfigured() replaces the three issuer/client-id checks in AuthService, and OidcService resolves its values through the same rule while still requiring the client secret. Every literal getValue() read goes through readAppSetting()/resolveAppSetting(), and ESLint refuses a new one.
2026-10-10 20:15:56 +02:00
Maurice 4a3c97e500 docs(server): name the query API per statement shape and hold QueryBuilder calls to a baseline
server/CLAUDE.md now says which API new repository code uses: Kysely for join, aggregate and set-based statements, the entity methods for single-table CRUD, no new QueryBuilder calls. The TrekRepository docstring no longer calls Kysely an escape hatch. lint:query-api counts this.qb()/createQueryBuilder() per repository file against a baseline that may only shrink and runs in the server CI job.

The same file documents the JSON column and timestamp rules from the two changes before.
2026-10-10 20:15:56 +02:00
Maurice b1da50466d ci(server): strict type errors per file may only shrink
tsconfig.strict.json is the build config with strictNullChecks and
noImplicitAny on. lint:strict runs tsc over it and holds the errors per file
to scripts/strict-baseline.json (204 in 109 files today): a file may only
lose errors, a new file starts at none, and an entry above its count fails
until --update lowers it. An error outside any file or an unreadable tsc run
fails closed. It runs in the server job; the build itself stays on
strict: false.
2026-10-10 20:15:55 +02:00
Maurice 1ea435d730 ci: server and shared lint stop rewriting files, and Prettier is checked per file
shared's lint ran eslint --fix with prettier/prettier as an error, so in
lint-prettier.yml the format:check after it only ever saw files it had just
rewritten, and the server's format scripts pointed at a test/ directory that
does not exist. Now lint is check-only in both (lint:fix applies fixes), the
server scripts cover src/ and tests/, and a lint:format ratchet on the shared
library holds every file outside scripts/format-baseline.json to Prettier in
both workspaces: in the server job of test.yml and in lint-prettier.yml,
which keeps providing the lint check. Formatting leaves ESLint in shared, and
the unused eslint-plugin-prettier goes. Once the tree is reformatted in one
commit, --update empties the baselines and the check is a full prettier
--check.
2026-10-10 20:15:54 +02:00
Maurice c9f66156d3 ci(server): lint:warnings holds the ESLint warnings per rule to a baseline
Server lint ran without --max-warnings or a baseline, so the rules left on
warn could grow without limit. lint:warnings runs ESLint with the server's
own config through the shared tally, fails on any error or inline rule
config, and holds the warnings per rule for src, tests and eslint-disable'd
messages to scripts/eslint-baseline.json, which only falls. It replaces
lint:check in CI. no-await-in-loop, no-non-null-assertion and no-console come
in as warnings held at today's counts. npm run lint no longer rewrites files;
lint:fix does.
2026-10-10 12:08:37 +02:00
Maurice 68504b337f refactor(scheduling): take a cron lease in one conditional upsert
acquire() made sure a row existed and then raced on a conditional UPDATE,
two statements the registrar wrapped in a transaction of its own. lint:tx
cannot follow a repository reached through a local variable, so it
reported the method. One INSERT ... ON CONFLICT DO UPDATE ... WHERE now
takes a free lease, renews the holder's and leaves anyone else's alone,
atomically on SQLite and on Postgres, and needs no transaction.
2026-10-08 19:05:03 +02:00
Maurice 1bba65634c test(server): seed and read test data through ORM factories
The server suite seeds and reads its database through some 5,500 raw
better-sqlite3 statements on the sync handle. They restate the schema by
hand, skip the repositories the app goes through and run on SQLite only,
so nothing in the suite could ever exercise the dialect layer on another
engine.

tests/helpers/factories/ adds typed factories per domain (users, trips and
days, places, assignments, bookings, budget, packing, to-dos, files,
journeys, tours, collections, notifications, settings, plugins, tokens,
atlas, vacay, photos, collab) and generic readers (findRow, findRows,
countRows, updateRows, deleteRows) that go through MikroORM in a request
context of their own, against the app's ORM or a createTestOrm handle.
They refuse a write to a read-only foreign-key twin, which MikroORM would
otherwise drop without a word. The README there is the conversion guide.

The categories integration suite and the tags and calendar-feed e2e suites
are converted in full as worked examples; the tags suite also moves from
hand-rolled tables to the migrated snapshot.

lint:test-sql (scripts/test-sql-ratchet.mjs) holds the raw statements left
to a per-file baseline: growth and stale entries fail, files whose subject
is raw SQL are exempt with a reason, and a single statement can be allowed
with a test-sql-allow comment. CI runs it next to lint:boundaries.
2026-10-08 18:44:39 +02:00
Maurice 648f63c626 fix(sessions): keep an ended session's row until it expires
A session renewed from a token issued before sessions were tracked has
an id derived from that token, and the renewal inserts it only when no
row by that id exists. The nightly purge deleted revoked rows straight
away, so the next request with the still valid old token inserted the
session again as an active one and got a fresh tracked token with a new
lifetime. A logout, an ended session or a sign out of the others held
only until the purge ran.

The purge now deletes expired rows only, and a revoked row stays until
its own expiry. A derived row is recorded with an expiry no earlier
than the old token's, so by the time it goes the old token is refused
by its signature check anyway.
2026-10-08 18:44:08 +02:00
Maurice 0dde01baa2 fix(oidc): sign in after SSO on a success flag, not the deprecated token
The token in the sign-in bodies was declared deprecated and "read by
nothing of ours", but the web client still decided an SSO sign-in by
reading data.token from GET /api/auth/oidc/exchange, an endpoint the
deprecation notice left out. Removing the field later, as documented,
would have broken SSO sign-in.

The exchange now answers { success: true, token } and the login hook
signs in when the response is OK and success is true, so it never reads
token. The field stays for API clients and is now on the deprecation
list in the API docs, the controller comments, server/CLAUDE.md and the
wiki (Security-Hardening, OIDC-SSO). Adding a field to the body breaks
no existing client.
2026-10-08 18:44:00 +02:00
Maurice 9aab9e7513 feat(auth): track sign-in sessions so a session can end before it expires
A session token was a signed JWT and nothing else. Logout only cleared
the cookie, a 30-day "remember me" token stayed valid wherever a copy of
it had ended up, and nobody could sign out one device short of changing
the password or rotating the JWT secret for everyone.

Every session token now carries an id (jti) naming a row in the new
user_sessions table (id, user_id, created_at, last_seen_at, expires_at,
revoked_at, user_agent cut to 256 characters; no address is kept). The
rows live behind UserSessionsRepository and the new sessions domain,
whose SessionsService is the one issuer: AuthService.generateToken and
the OIDC sign-in both go through it, and sliding renewal re-signs a
tracked token under the same id and moves its expiry. verifyJwtAndLoadUser
takes the repository and refuses a token whose session is revoked,
expired or gone, refreshing last_seen_at at most every five minutes.
A token without a jti was issued before this change; it passes until its
own expiry (or a password change), so the upgrade signs nobody out, and
renewal turns it into a tracked session.

Sessions end on logout (the one the request names, verified even when
expired), on a password change (all, the device gets a new one), a
password reset, an admin setting a password, reset-admin.js, an MFA
disable (all but the current one), an admin clearing a user's MFA, and
with the account. GET /api/auth/sessions lists the active sessions,
DELETE /api/auth/sessions/:id ends one and POST
/api/auth/sessions/revoke-others ends all but the current, with their
contracts in @trek/shared and two new audit actions. There is no UI for
them yet. MCP exposes no account or session tools, so there is no
parallel tool to change; an MCP call made with a session JWT as bearer
goes through the same check.

The token in the login, register, demo-login, MFA and passkey bodies
stays for API clients but is marked deprecated in the API docs and in
the client type, which never read it.

A nightly session-purge job on CronRegistrarService deletes the expired
and revoked rows. The migration is 042500. The sessions domain gets its
own coverage entry at the 80 floor until a full run measures it.
2026-10-08 18:43:06 +02:00
Maurice 8018f10273 feat(server): gate writes outside a transaction with lint:tx
The rule that a multi-statement write runs in uow.transactional was
written down but nothing held it. scripts/tx-writes.mjs reads every class
under src/ with the TypeScript parser and counts, per method, the writes
it issues outside a transactional callback: repository calls that build
an INSERT, UPDATE or DELETE, calls to methods that write (on the class
itself or on a constructor-injected one), and a transaction holding
writes as one. A write in a loop counts twice, of two branches only the
larger counts, local helpers count where they are called, and a helper
whose every call sits inside a transaction is not reported. --explain
lists the line of every call that counted.

Today's 102 reported methods sit in scripts/tx-writes-baseline.json and
may only shrink; most are syncs with network I/O between their writes,
plugin lifecycle steps and adapters calling two service writes. A new
method fails CI, and an entry above what its method holds fails until
--update lowers it, the same ratchet as lint:size and lint:boundaries.

Two JSDoc tags mark the deliberate cases with their reason in the tag:
@txStandalone for a write that stands on its own whatever the caller does
(the audit row, notification sends, the booking price mirror, the
assignment journey refresh, the quota retention purge), @txIndependent
for writes complete one by one on purpose (the OAuth token retention
sweep, the notification claim and its release). Runs in CI after
lint:boundaries; documented in server/CLAUDE.md, the root CLAUDE.md and
the wiki's dev setup page.
2026-10-08 18:41:57 +02:00
Maurice d3be6fb2b8 fix(plugins): hold exec scripts to the time budget between statements
The time budget covered query() and tx() only. exec() without bound args
handed the plugin's script to Database#exec, which runs any number of
statements in one call with nothing checked between them, so a long
script held the host loop as long as it liked.

An exec script now runs one statement at a time. sql-script.ts cuts it
after every semicolon outside a string, a quoted identifier or a
comment; a piece SQLite reports as incomplete input (a trigger body cut
at its inner semicolon) is joined with the next until it prepares. The
clock is read after every statement, and past the budget the call
throws. As with Database#exec, the statements before that point stay
applied, and an error is SQLite's own; a transaction the script opened
itself is rolled back so the connection is not left inside it.

exec() with bound args is one statement and stays as it was. A single
statement cannot be bounded at all on the host thread, a write such as
INSERT ... SELECT over a cross join included, and the SDK README and the
wiki now say so next to the cross-join read. migrate() keeps running
unbudgeted on purpose: a migration stopped part of the way is rolled back
and fails the same way on every later start, leaving the plugin unable to
update its schema.

The ctx.db rows of the SDK README and the plugin wiki pages describe the
budget for scripts, and the lines touched there and in plugin-sdk's db
comment lose their em dashes.
2026-10-08 18:41:02 +02:00
Maurice 9dec36995e feat(plugins): extend the output allowlist to child rows and whole-row read models
The output contract cut only the top level of an entity result. A derived
key's value went out untouched, so the children that are whole rows still
carried every column: a day's notes_items are full day_notes entities and
a reservation's endpoints are SELECT e.* on reservation_endpoints. About
fifty methods were also left as readModel and passed through whole, many
of them backed by SELECT * reads (day_accommodations, day_notes, todos,
tags, packing bags, journal entries, collab, collections, vacay plans).
A new column on any of those tables reached every plugin by default,
which is what the contract was meant to stop and what the SDK README and
the wiki already claimed it did.

PLUGIN_ENTITY_NESTED now names, per entity, the derived keys that hold
rows of another entity, and the shaper cuts those rows down to their own
field list, recursively. dayNote and reservationEndpoint are the child
tables that were whole rows; day.assignments and place.tags are held to
the assignment and tag lists too, although their reads already name
their columns. The remaining derived children (travelers, participants,
ratings, budget members, payers and receipts, packing recipients,
attachments, voters, bag members) are built field by field, which the
contract comment now states as the rule for a derived key without a
nested entity.

The whole-row read models get entity contracts: accommodation, dayNote,
todo, tag, category, packingBag, fileLink, collabNote, collabPoll,
collabMessage, journey, journalEntry, bucketItem and collection, and
vacayPlan with its holiday calendars. collections.listMine,
collections.get and vacay.mine return envelopes around rows; an entity
with a null table describes those, so their rows are reached through the
same nesting. Every list holds exactly the keys each read returned
before, so no plugin loses a field. What stays readModel is built field
by field from named columns, and each such entry says what it returns:
setBagMembers, atlas.visited, collections.savePlace and copyToTrip, the
two vacay toggles and journal.addEntryPhoto.

The schema test now covers every child and new table, and calls each
newly shaped method on the real services, comparing nested rows key for
key so the only field allowed to disappear is a withheld one. A further
case proves the nested rows it compares are not empty. gen:plugin-facts
publishes PLUGIN_ENTITY_NESTED to the SDK, and the SDK README, the plugin
wiki pages and server/CLAUDE.md describe the nested rows and list the new
entities.
2026-10-08 18:41:02 +02:00
Maurice 0c9e1cd639 fix(plugins): hold plugin SQL to a time budget between rows
A plugin's own database runs on the host's event loop, and the only bounds
were a regex denylist, the 256 MB quota and a 100,000 row cap. A select
whose every row is expensive, or a long scan under the cap, could hold the
loop for as long as it liked.

query() and tx() now get a wall-clock budget of 2 s, read between the rows
a statement yields and between the statements of a batch. Past it the call
throws, and a tx rolls back whole like any other failure inside it. The
row cap and the budget are overridable per PluginDataDb for tests; the
defaults are what production runs.

better-sqlite3 has no interrupt or progress handler, so a statement that
yields nothing until it is done, such as an aggregate over a cross join,
still runs to its first row. Only moving plugin SQL off the host thread can
bound that; the SDK README and the plugin wiki state the budget and this
limit.
2026-10-08 18:41:01 +02:00
Maurice 95f9f031c4 feat(plugins): cut entity results down to a published field list
The entity reads of the plugin RPC surface handed plugins whatever their
query produced, mostly SELECT t.*, with credentials stripped by a denylist
(withoutFeedToken). Every column a migration added reached every plugin
holding the read grant, and renaming one was an unversioned break of a
published API.

protocol/output-contract.ts now states, for every wire method, what its
result is: rows of one of the nine entities the SDK types (trip, place,
day, reservation, packingItem, tripFile, budgetItem, assignment, user), a
value the host builds itself, or a read model passed on as the REST route
returns it. The router applies it to each result on the way out, so an
entity row keeps exactly the fields plugins received when the list was
drawn, in the same order and with the same values, minus the trip's
feed_token (listMine, create and update used to carry it as null). The
table is keyed by every method, so a new one does not compile without an
entry.

output-contract.schema.test.ts holds the lists to the migrated schema: a
new column on a whole-row table fails until it is published or withheld,
and each entity method is called on the real services to prove the
contract drops nothing but the withheld fields.

gen:plugin-facts publishes the lists to the SDK as PLUGIN_ENTITY_FIELDS
and PLUGIN_METHOD_RESULT, and the SDK's entity interfaces are type-checked
against them. Place.day_id, which no read ever delivered, is deprecated
rather than removed. KNOWN_ADDONS is generated from ADDON_IDS as well,
which adds the dawarich, roadtrip and tours ids the hand-kept copy lacked.
The SDK goes to 1.9.0, and its README and the plugin wiki pages describe
what a ctx call returns.
2026-10-08 18:41:01 +02:00
Maurice aa8d9b4037 refactor(server): inject the transit base URL and the kitinerary paths as config tokens
The ownership rule says a boot-stable value that only Nest providers read
belongs to a registerAs token, yet TransitService froze TRANSIT_API_URL in a
module-top readEnv() const and KitineraryExtractorService read
KITINERARY_EXTRACTOR_PATH and PATH through readEnv() in its probe. Both now
come from tokens: transitConfig and kitineraryConfig, derived in
boot-derive.ts and dropped from deriveIntegrations, so readEnv() no longer
reads those variables at all. TransitModule and KitineraryExtractorModule
import AppConfigModule, which keeps them buildable on their own in tests.

Behaviour is unchanged: the same defaults, the same trailing-slash strip and
PATH split, now frozen per built app instead of at import. The app-config
README names each boot-stable variable that stays on readEnv() together with
the reader the container cannot inject into.
2026-10-08 18:41:01 +02:00
Maurice 558203213c fix(server): keep idle connections open longer than Traefik's upstream timeout
The 65 s default sat above nginx's 60 s but below the 90 s Traefik keeps an
idle upstream connection, so a default Traefik deployment could still reuse a
socket Node had just closed and answer that request with a 502. 95 s clears
both, and the headers timeout one second above it stays well under Node's
300 s request timeout and the 290000 cap the schema accepts.

Also moves the stray RawEnv import in boot-derive.ts up to the other import.
2026-10-08 18:41:01 +02:00
Maurice b3a085df3f refactor(server): make the state ports async and route every reader through the provided store
The process-state ports were meant to let a store shared between
processes replace the in-memory one, but they could not. The readers
outside the container (the broadcast functions, the backup restore's
permissions-cache flush, the pending-code sweep, hand-built consumers)
went straight to the module-level in-memory instances, so an overridden
provider was used by the gateway and the services while the broadcasts
and the restore flush kept using the old store. And every port was
synchronous, while a store in the database or on the network is async,
so plugging one in would have changed every signature anyway.

RateLimitStore, PendingCodeStore, OidcFlowStore and
PermissionsCacheStore now return promises, and RateLimitService,
OauthService, OidcService, PermissionsService and their callers await
them. The in-memory stores still decide synchronously, so who gets a
429, a code or a cached level is unchanged. RoomRegistry stays
synchronous: membership holds live sockets and broadcast() is a
synchronous call every domain makes, so several processes need a
fan-out in front of broadcast(), not a different place for the sets.

The process-wide ports get a ProcessStoreSlot. PermissionsModule,
OauthModule and RealtimeGatewayModule install the store the container
resolved and release it on shutdown, and the free functions, the sweep
and hand-built consumers read the slot, so overriding the provider
changes every reader. Tests provide fake stores and check that the
broadcasts, the restore flush and the sweep use them.

Two callers needed more than an await. The transit MCP tools counted in
a RateLimitService of their own that no provider could reach; they now
take the one RateLimitModule provides (their mcp_transit_* buckets never
overlap the REST ones). The link preview claimed its in-flight slot only
after the budget check, which now yields, so simultaneous askers would
each have been charged; it now claims the slot first, and an asker that
joined someone out of budget asks again on its own budget.

The unused putPendingCode/takePendingCode helpers are gone.
2026-10-08 18:41:01 +02:00
Maurice fa0e6a3eae refactor(server): take the storage roots from DataPathsService
The data-directory change said Nest classes inject DataPathsService, but
only AdminService did. StorageRegistryService still read the built-in
uploads and backups roots and the scratch dir from module constants
frozen at import, and StorageJobsService read the scratch dir the same
way.

The registry now takes DataPathsService through its constructor (a
hand-built one resolves the same layout), and the jobs service asks the
registry for its temp dir. The plain helper modules keep calling
resolveDataPaths() because they run without a container or are shared
with code that does: config.ts, the database path, index.ts, the logger,
the backup functions and the plugin trees, which the out-of-process
supervisor reads too. The app-config README now lists both sides, and
the nest README, server/CLAUDE.md and the provider's own comment say the
same.
2026-10-08 18:40:52 +02:00
Maurice d1b461a37f docs(server): name HTTP_KEEP_ALIVE_TIMEOUT_MS as an httpConfig variable
The keep-alive change moved HTTP_KEEP_ALIVE_TIMEOUT_MS onto the httpConfig
token, but the app-config README still listed only TRUST_PROXY and
HSTS_INCLUDE_SUBDOMAINS for it and left the variable out of the
boot-stable list. Also drops the dashes from two lines the recent
scheduling and entity changes rewrote.
2026-10-08 18:40:37 +02:00
Maurice a72497b3c4 fix(server): trace a cron tick lost to another process as skipped
A tick whose lease another process held returned from runLeased without
running, and traceEntry still wrote `cron <name> ok`, so both processes
logged the tick as run and the per-tick line could not tell which
replica did the work. runLeased now says whether the tick ran, and
traceEntry takes a `skipped` option that logs such a call at debug as
`cron <name> skipped <ms>: lease held by another process`.

Taking the lease is two statements (make sure the row exists, then the
conditional UPDATE that decides the race), so it now runs inside
UnitOfWork.transactional like every other multi-statement write.
2026-10-08 18:40:37 +02:00
Maurice 06dfdc6943 refactor(server): put process-local state behind injectable stores
Rate-limit counters, OAuth pending codes, OIDC login states and codes,
the permissions cache and the WebSocket rooms were Maps on module or
instance scope. That is right while TREK runs as one process, but nothing
could be swapped without editing every consumer.

Each now sits behind a port with an in-memory implementation that keeps
the old behaviour exactly: RateLimitStore, PendingCodeStore,
OidcFlowStore, PermissionsCacheStore and RoomRegistry. The owning module
provides the implementation with the same lifetime the state had
(process-wide for the pending codes, the permissions cache and the rooms,
per instance for the rate limits and the OIDC flows), the consumers take
it through their constructor, and a hand-built consumer falls back to the
same in-memory store, so tests and the free-function paths (the backup
restore's cache flush, the broadcasts) see what they saw before. A store
shared between processes can replace a provider later without touching
the services.
2026-10-08 18:40:37 +02:00
Maurice 5cb87a4df5 feat(server): add last-resort error handlers, a draining readiness probe and proxy keep-alive
Three small gaps in how the process behaves at its edges:

- An unhandled rejection or uncaught exception crashed the process with
  the stack on stderr only. index.ts now logs it to trek.log and runs the
  same orderly shutdown a SIGTERM gets, exiting with 1; a second fatal
  error during that shutdown exits at once.
- /api/health/ready kept answering 200 while a SIGTERM was draining the
  process, so an orchestrator sent traffic for another probe period. A
  shutdown now marks the new ReadinessService first (and Nest's own
  shutdown hook marks it too), and the probe answers 503 from then on.
- Node closes an idle keep-alive connection after 5 seconds while reverse
  proxies keep upstream connections for about a minute, which shows up as
  intermittent 502s. The server now keeps them for 65 seconds, configurable
  through HTTP_KEEP_ALIVE_TIMEOUT_MS (owned by the httpConfig token), with
  the headers timeout one second above.
2026-10-08 18:40:37 +02:00
Maurice c5bc6798a9 feat(server): take a database lease before a cron tick runs
waitForCompletion only kept a tick from overlapping itself inside one
process. Two processes on the same database (a rolling update's overlap,
a second replica, a dev container pointed at the same file) both ran
every job: two auto-backups, double syncs and retention sweeps, and
reminder runs racing each other.

A new migration adds scheduler_leases, and CronRegistrarService now runs
every registered tick through SchedulerLeasesRepository.acquire: one
conditional UPDATE moves the job's row to this process when it is free
(no holder, a lapsed holder, or this process already), so exactly one
process wins. A running tick renews its lease every 20 seconds, a crashed
holder frees the job after a minute, and a finished tick keeps the lease
for 30 seconds so a peer whose timer fires late does not run the same
tick again. The per-row reminder claims stay as they are; jobs need no
change of their own.
2026-10-08 18:40:15 +02:00