diff --git a/server/CLAUDE.md b/server/CLAUDE.md index 1b5dcdbce..0a445be25 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -85,7 +85,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Thin controller → injectable provider.** Never resurrect a plain function-module service; don't grow god services — split a domain into several modules when it outgrows one class (trips and auth already are). - **Inject, don't reach for module globals.** No imports of the global `db` proxy from `nest/` layers at all any more — the last production importer converted onto a repository, and the export now survives only so ~190 test files' `vi.mock('…/db/database', …)` factories keep typechecking; no new code depending on the `ws` singleton or event-sink globals. There is no `DatabaseService` to inject either — take the repository(ies) you need plus `UnitOfWork` for transactions. Modules must be importable without side effects. -- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). The sequence lives in the service, never in a controller, MCP tool or RPC handler, so every surface calls the one method that holds the transaction (`ReservationsService.createWithCost` is the shape: side effects such as broadcasts come back to the caller and go out after the commit). A position or id taken as `MAX(...) + 1` is read inside the same transaction as the insert that uses it: the transaction holds the connection, so two requests at once cannot read the same MAX. `lint:tx` holds the rest. Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository`, `AdminSeeder` and `CategorySeeder` are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`. +- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). The sequence lives in the service, never in a controller, MCP tool or RPC handler, so every surface calls the one method that holds the transaction (`ReservationsService.createWithCost` is the shape: side effects such as broadcasts come back to the caller and go out after the commit). A position or id taken as `MAX(...) + 1` is read inside the same transaction as the insert that uses it: the transaction holds the connection, so two requests at once cannot read the same MAX. On Postgres, where a pool runs transactions side by side, `UnitOfWork` first takes one advisory lock for the database (`DatabaseEngine.serializeWrites`), so the same holds; a transaction opened any other way does not get it. `lint:tx` holds the rest. Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository`, `AdminSeeder` and `CategorySeeder` are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`. - **Migrations are append-only.** A migration is identified by its class name in `mikro_orm_migrations`, and files sort by the timestamp in that name. Never rename or delete one that has shipped, and never edit its body; add a new one. No env interpolation in SQL. (The old positional `schema_version` array and the hand-kept `db/migrations.ts` test builder are both gone; the numbered migrations are the only schema history, for tests too.) A new foreign key comes with an index that leads with its column(s), or deleting the parent scans the child table (a partial index counts only when its WHERE clause is `IS NOT NULL` on those columns); `tests/unit/db/foreign-key-indexes.test.ts` fails on one without. - **Every write endpoint validates** through the Zod pipe with a schema from `@trek/shared` — no `Record` + ad-hoc checks. Also verify every referenced id exists and belongs to the same trip. - **Every outbound `fetch`** gets a timeout, a response-size cap and boundary validation: no `as`-casting provider responses, no silent `catch → null`. User-influenced URLs go through the SSRF guard. Never fall back to another user's API key. ESLint requires an init with a `signal` on `fetch`, `safeFetch`, `safeFetchFollow` and `safeFetchAdminConfigured` alike, or a literal `timeoutMs` in the options object of `safeFetch`/`safeFetchFollow`; an init spelled `undefined`, `null` or `void 0` counts as none (`safeFetchLlm` takes `LLM_TIMEOUT_MS` inside). Behind that, the guarded wrappers (`utils/ssrfGuard.ts`) cap every body at `DEFAULT_MAX_RESPONSE_BYTES` (64 MB) and, for a call without a signal, wait `DEFAULT_RESPONSE_TIMEOUT_MS` (30 s) for the headers. Pass a tighter `maxBytes` when you buffer a small answer, `timeoutMs` for a deadline over the whole exchange, and `maxBytes: null` only where the body is streamed on or counted by the caller (`pipeAsset`, the doc-sync transfers, the Ollama pull). @@ -105,6 +105,7 @@ SQLite is the default engine and Postgres the second (`TREK_DB_URL`, see Engines - **Insert or ignore / upsert**: `.onConflict((oc) => oc.columns([...]).doNothing())` or `.doUpdateSet({...})`, or `em.upsert`. Name the conflict target where there is one. Never `INSERT OR IGNORE`/`INSERT OR REPLACE`. - **The new id**: `.returning('id').executeTakeFirstOrThrow()`. Never `InsertResult.insertId` or `last_insert_rowid()`; both are SQLite's rowid and undefined on Postgres. - **Time**: `currentTimestamp`/`currentTimestampKysely`, `nowMinusDays`, `nowMinusHours`, `nowPlusSeconds`/`nowPlusSecondsKysely`, `nowDateOffset`, `dateOf`, `dateAdd`, `unixEpochToIsoKysely` from `src/db/dialect/sql-functions.ts` (the MikroORM `raw()` helpers) and `src/db/dialect/kysely-functions.ts` (their Kysely-expression twins). Never `datetime('now')`, `strftime` or `julianday`. +- **Search that ignores case**: `ilike(platform, refs)` builds the `(a LIKE ? ESCAPE '\' OR …)` condition, `ILIKE` on Postgres, where `LIKE` is exact. A bare `LIKE` is for a prefix that is exact on both engines (a settings key, a storage path). - **Strings and casts**: `concat`/`concatKysely` instead of `||`, `substring`/`substringKysely`, `lower`/`lowerTrim`, `collateNoCase`, `castInteger`/`castIntegerKysely`, and `startsWithIsoDate`/`startsWithIsoDateKysely` instead of `GLOB`. - **JSON columns** are read whole and parsed in the service; no `json_extract` in SQL. Each one is declared once in `src/db/json-columns.ts` (its Zod shape and the fallback a reader gets for an empty or broken row) and read with `decodeJson`/`decodeJsonResult`, written from a server-built value with `encodeJson` (`src/utils/json-column.ts`), never with a bare `JSON.parse` at the call site: one column, one answer to a broken row, logged. - **Timestamps** in a DATETIME column are the canonical `YYYY-MM-DD HH:MM:SS` UTC text, never `toISOString()`: in SQL the two spellings do not compare. A value written from JS is a `DbTimestamp` from `dbNow()` (`src/db/types`), and a stored value is read back as an instant with `parseDbTimestamp()` (a bare `new Date()` reads the canonical text as local time). Where a response has always carried the ISO spelling, convert on the way out (`utcSuffix`), not in the column. diff --git a/server/scripts/pg-probe/helper-cases.ts b/server/scripts/pg-probe/helper-cases.ts index 3974ec3d7..92b5b7936 100644 --- a/server/scripts/pg-probe/helper-cases.ts +++ b/server/scripts/pg-probe/helper-cases.ts @@ -44,6 +44,7 @@ import { dateOf, dayDistance, foundAgainState, + ilike, lower, lowerParam, lowerTrim, @@ -269,6 +270,18 @@ export const HELPER_CASES: readonly HelperCase[] = [ raw: (p) => rawOf(dayDistance(p, 'a', '2026-09-24')), expect: { equals: 3 }, }, + { + name: 'ilike matches in other case', + row: { a: 'Probe Place' }, + raw: (p) => ({ sql: ilike(p, ['a']), params: ['%PLACE%'] }), + expect: { equals: true }, + }, + { + name: 'ilike keeps the escaped wildcard literal', + row: { a: 'Probe Place' }, + raw: (p) => ({ sql: ilike(p, ['a']), params: ['probe\\_place'] }), + expect: { equals: false }, + }, { name: 'collateNoCase', row: { a: 'Probe' }, diff --git a/server/scripts/size-baseline.json b/server/scripts/size-baseline.json index 13cd417b2..013f14ac9 100644 --- a/server/scripts/size-baseline.json +++ b/server/scripts/size-baseline.json @@ -1,7 +1,7 @@ { "scripts/db-call-graph.mjs": 1036, "scripts/generate-entities.ts": 2259, - "src/db/repositories/Places.repository.ts": 1437, + "src/db/repositories/Places.repository.ts": 1435, "src/db/repositories/Reservations.repository.ts": 1563, "src/db/repositories/Trips.repository.ts": 1121, "src/db/repositories/Users.repository.ts": 1445, diff --git a/server/src/db/dialect/sql-functions.ts b/server/src/db/dialect/sql-functions.ts index a511960f3..55abdd14e 100644 --- a/server/src/db/dialect/sql-functions.ts +++ b/server/src/db/dialect/sql-functions.ts @@ -620,6 +620,17 @@ export function collateNoCase(platform: Platform, ref: string): RawQueryFragment return unsupported(platform); } +/** + * A search that ignores case over `refs`: `(a LIKE ? ESCAPE '\' OR b LIKE ? + * ...)`, one bound pattern per column, the caller escaping `%`, `_` and `\`. + * SQLite's `LIKE` already ignores case (ASCII letters only); Postgres's is + * exact, so it gets `ILIKE`, which folds every letter. + */ +export function ilike(platform: Platform, refs: readonly string[]): string { + const operator = isSqlite(platform) ? 'LIKE' : isPostgres(platform) ? 'ILIKE' : unsupported(platform); + return `(${refs.map((ref) => `${column(ref)} ${operator} ? ESCAPE '\\'`).join(' OR ')})`; +} + /** * `datetime('now', '-N hours')` — the hour-granularity sibling of * `nowMinusDays()` above, for `reminder-jobs.service.ts#todoTick`'s diff --git a/server/src/db/engine/engine.ts b/server/src/db/engine/engine.ts index 050026fde..821c051e5 100644 --- a/server/src/db/engine/engine.ts +++ b/server/src/db/engine/engine.ts @@ -1,5 +1,5 @@ import type { AnyOrm } from '../orm'; -import type { Options } from '@mikro-orm/core'; +import type { EntityManager, Options } from '@mikro-orm/core'; /** The databases TREK can be pointed at. MySQL/MariaDB are out of scope (server/CLAUDE.md). */ type EngineId = 'sqlite' | 'postgres'; @@ -37,4 +37,10 @@ export interface DatabaseEngine { * archive is the copy). */ schemaBootstrap(orm: AnyOrm, options: { snapshot: boolean }): Promise; + /** + * Runs first in every `UnitOfWork` transaction, on its EntityManager: makes + * the transactions that write run one at a time, as SQLite's single writer + * does, so a `MAX(...) + 1` read inside one cannot race another. + */ + serializeWrites(em: EntityManager): Promise; } diff --git a/server/src/db/engine/postgres/postgres-engine.ts b/server/src/db/engine/postgres/postgres-engine.ts index f2e815421..2df0315bf 100644 --- a/server/src/db/engine/postgres/postgres-engine.ts +++ b/server/src/db/engine/postgres/postgres-engine.ts @@ -3,6 +3,7 @@ import type { DatabaseEngine } from '../engine'; import { redactUrl } from '../redact-url'; import { TrekPostgreSqlDriver } from './postgres-driver'; import { bootstrapPostgresSchema, defaultPostgresBootstrapDeps, type PostgresBootstrapDeps } from './schema-bootstrap'; +import type { EntityManager } from '@mikro-orm/core'; import { readFileSync } from 'node:fs'; @@ -53,6 +54,9 @@ export function sslOptionFor(url: string): PgSslOption | undefined { } } +/** The advisory lock key every write transaction takes: "TREK" in ASCII. */ +const TREK_WRITE_LOCK = 0x5452454b; + /** * The Postgres engine: MikroORM's own `pg` driver on `TREK_DB_URL`, with a * pool of at most `TREK_DB_POOL_MAX` connections that keeps none open while @@ -80,6 +84,17 @@ export class PostgresEngine implements DatabaseEngine { return Promise.reject(new Error('Restoring a backup into a Postgres database is not available in this release.')); } + /** + * `pg_advisory_xact_lock` on one key for the whole database: held until the + * transaction commits or rolls back, taken again by a savepoint in the same + * session without waiting, and shared by every process on the database. + */ + async serializeWrites(em: EntityManager): Promise { + await em + .getConnection() + .execute('select pg_advisory_xact_lock(?)', [TREK_WRITE_LOCK], 'run', em.getTransactionContext()); + } + async schemaBootstrap(orm: AnyOrm): Promise { await bootstrapPostgresSchema(orm, this.bootstrapDeps()); await finishSchemaBootstrap(orm); diff --git a/server/src/db/engine/sqlite/sqlite-engine.ts b/server/src/db/engine/sqlite/sqlite-engine.ts index 95533e684..0f3523298 100644 --- a/server/src/db/engine/sqlite/sqlite-engine.ts +++ b/server/src/db/engine/sqlite/sqlite-engine.ts @@ -3,6 +3,7 @@ import { resolveDbPath } from '../../db-path'; import { runSchemaBootstrap, type AnyOrm } from '../../orm'; import { SharedSqliteDriver } from '../../orm-driver'; import type { DatabaseEngine } from '../engine'; +import type { EntityManager } from '@mikro-orm/core'; /** * The SQLite file engine, TREK's default: one better-sqlite3 handle that @@ -40,4 +41,9 @@ export class SqliteEngine implements DatabaseEngine { schemaBootstrap(orm: AnyOrm, options: { snapshot: boolean }): Promise { return runSchemaBootstrap(orm, options); } + + serializeWrites(_em: EntityManager): Promise { + // One connection, one writer: SQLite serializes the transactions already. + return Promise.resolve(); + } } diff --git a/server/src/db/repositories/MaintenanceRepository.ts b/server/src/db/repositories/MaintenanceRepository.ts index 89fc10dee..af20dc710 100644 --- a/server/src/db/repositories/MaintenanceRepository.ts +++ b/server/src/db/repositories/MaintenanceRepository.ts @@ -69,14 +69,20 @@ export class MaintenanceRepository { * otherwise, so passing it through is correct both inside and outside a * transaction. Each statement is independently best-effort: a * slimmed-down test schema may not carry one or more of these tables at - * all, matching the legacy per-table try/catch exactly. + * all, matching the legacy per-table try/catch exactly. Each one runs in + * its own savepoint (a nested `em.transactional`), because on Postgres a + * failed statement aborts the whole transaction it is in: the savepoint + * rolls back that one statement and the user's erasure goes on. */ async deletePluginUserData(userId: number): Promise { this.validateRequestContext(); - const ctx = this.em.getTransactionContext(); for (const table of ['plugin_user_config', 'plugin_oauth_tokens', 'plugin_oauth_state']) { try { - await this.em.getConnection().execute(`DELETE FROM ${table} WHERE user_id = ?`, [userId], 'run', ctx); + await this.em.transactional((em) => + em + .getConnection() + .execute(`DELETE FROM ${table} WHERE user_id = ?`, [userId], 'run', em.getTransactionContext()), + ); } catch { /* table absent (slim schema) */ } diff --git a/server/src/db/repositories/Places.repository.ts b/server/src/db/repositories/Places.repository.ts index 4941a1582..98257e2d8 100644 --- a/server/src/db/repositories/Places.repository.ts +++ b/server/src/db/repositories/Places.repository.ts @@ -1,6 +1,6 @@ import type { Places } from '../entities/Places.entity'; import { wrapOptionalLongitude } from '@trek/shared'; -import { absDifference, coalesceParam, columnRef, currentTimestamp, lowerTrim } from '../dialect/sql-functions'; +import { absDifference, coalesceParam, columnRef, currentTimestamp, ilike, lowerTrim } from '../dialect/sql-functions'; import { type AssertRowKeys } from './_shared/rows'; import { TrekRepository } from './_shared/trek-repository'; import { tourPlaceTextPatch, type TourPlaceWrite } from './tour-place-write'; @@ -680,10 +680,8 @@ export class PlacesRepository extends TrekRepository { .where('p.trip_id = ?', [trip_id]); if (filters.searchPattern) { - qb.andWhere( - "(p.name LIKE ? ESCAPE '\\' OR p.address LIKE ? ESCAPE '\\' OR p.description LIKE ? ESCAPE '\\')", - [filters.searchPattern, filters.searchPattern, filters.searchPattern], - ); + const fields = ['p.name', 'p.address', 'p.description']; + qb.andWhere(ilike(platform, fields), fields.map(() => filters.searchPattern)); } if (filters.category !== undefined) { qb.andWhere('p.category_id = ?', [filters.category]); diff --git a/server/src/db/repositories/Trips.repository.ts b/server/src/db/repositories/Trips.repository.ts index cba9187ec..e91226ae1 100644 --- a/server/src/db/repositories/Trips.repository.ts +++ b/server/src/db/repositories/Trips.repository.ts @@ -1,6 +1,6 @@ import { QueryFlag } from '@mikro-orm/core'; import type { Trips } from '../entities/Trips.entity'; -import { coalesceParam, currentTimestamp, nowDateOffset } from '../dialect/sql-functions'; +import { coalesceParam, currentTimestamp, ilike, nowDateOffset } from '../dialect/sql-functions'; import type { AssertRowKeys } from './_shared/rows'; import { TrekRepository } from './_shared/trek-repository'; import { tripAccessExpr } from './_shared/trip-access'; @@ -174,7 +174,7 @@ export class TripsRepository extends TrekRepository { return await this.accessibleTripsQuery(user_id) .join('t.places_collection', 'p') .select(['p.trip', 'p.name']) - .andWhere("(p.name LIKE ? ESCAPE '\\' OR p.address LIKE ? ESCAPE '\\')", [likePattern, likePattern]) + .andWhere(ilike(this.getEntityManager().getPlatform(), ['p.name', 'p.address']), [likePattern, likePattern]) .orderBy([{ 'p.trip': 'asc' }, { 'p.name': 'asc' }]) .limit(500) // The limit is on the joined place rows, like the legacy statement's. diff --git a/server/src/nest/database/unit-of-work.ts b/server/src/nest/database/unit-of-work.ts index 8b2e9b600..7b4002e60 100644 --- a/server/src/nest/database/unit-of-work.ts +++ b/server/src/nest/database/unit-of-work.ts @@ -1,3 +1,4 @@ +import { resolveEngine } from '../../db/engine'; import { EntityManager } from '@mikro-orm/core'; import { Injectable } from '@nestjs/common'; @@ -17,12 +18,22 @@ import { Injectable } from '@nestjs/common'; * * Never awaits network I/O inside: the SQLite connection is one Kysely client * with a mutex, and an open transaction holds it for every other request. + * + * The services read `MAX(...) + 1` inside these transactions on the strength + * of SQLite's single writer. On Postgres a pool of connections would let two + * of them read the same MAX, so every transaction first takes the engine's + * write lock (`DatabaseEngine.serializeWrites`), which keeps that guarantee. */ @Injectable() export class UnitOfWork { + private readonly engine = resolveEngine(); + constructor(private readonly em: EntityManager) {} transactional(fn: () => Promise): Promise { - return this.em.transactional(() => fn()); + return this.em.transactional(async (em) => { + await this.engine.serializeWrites(em); + return fn(); + }); } } diff --git a/server/tests/unit/db/dialect/sql-functions.postgres.test.ts b/server/tests/unit/db/dialect/sql-functions.postgres.test.ts index b26f0c54e..6c4d3e89e 100644 --- a/server/tests/unit/db/dialect/sql-functions.postgres.test.ts +++ b/server/tests/unit/db/dialect/sql-functions.postgres.test.ts @@ -34,6 +34,7 @@ import { dateOf, dayDistance, foundAgainState, + ilike, lower, lowerParam, lowerTrim, @@ -175,6 +176,9 @@ describe('sql-functions (postgres)', () => { 'CAST(trunc(CAST(vr.accommodation_id AS numeric)) AS integer)', ); expect(collateNoCase(pg, 'name').sql).toBe('name COLLATE "nocase"'); + expect(ilike(pg, ['p.name', 'p.address'])).toBe("(p.name ILIKE ? ESCAPE '\\' OR p.address ILIKE ? ESCAPE '\\')"); + expect(ilike(new SqlitePlatform(), ['p.name'])).toBe("(p.name LIKE ? ESCAPE '\\')"); + expect(() => ilike(pg, ['p.name; drop'])).toThrow(/not a column reference/); const whileSame = coalesceOverrideWhileSame(pg, null, 'u.tls', 'u.url', 'https://a'); expect(whileSame.sql).toBe( 'CASE WHEN u.url IS NOT DISTINCT FROM ? THEN COALESCE(?, u.tls) ELSE COALESCE(?, 0) END', diff --git a/server/tests/unit/db/engine/engine.test.ts b/server/tests/unit/db/engine/engine.test.ts index 69343b339..c04d75dd3 100644 --- a/server/tests/unit/db/engine/engine.test.ts +++ b/server/tests/unit/db/engine/engine.test.ts @@ -4,11 +4,12 @@ import { TrekPostgreSqlDriver } from '../../../../src/db/engine/postgres/postgre import { sslOptionFor } from '../../../../src/db/engine/postgres/postgres-engine'; import { redactUrl } from '../../../../src/db/engine/redact-url'; import { SharedSqliteDriver } from '../../../../src/db/orm-driver'; +import type { EntityManager } from '@mikro-orm/core'; import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import path from 'node:path'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; const PG_URL = 'postgresql://trek:s3cret@db.example:5432/trek'; @@ -113,4 +114,15 @@ describe('buildOrmConfig', () => { expect(() => engine.close()).not.toThrow(); await expect(engine.reopen()).rejects.toThrow(/not available in this release/); }); + + it('ENGINE-033: Postgres serializes write transactions with one advisory lock taken in the transaction, SQLite needs none', async () => { + const execute = vi.fn(async () => []); + const trx = { trx: true }; + const em = { getConnection: () => ({ execute }), getTransactionContext: () => trx } as unknown as EntityManager; + await resolveEngine(deriveDb({ TREK_DB_URL: PG_URL })).serializeWrites(em); + expect(execute).toHaveBeenCalledWith('select pg_advisory_xact_lock(?)', [0x5452454b], 'run', trx); + execute.mockClear(); + await resolveEngine().serializeWrites(em); + expect(execute).not.toHaveBeenCalled(); + }); });