feat(db): keep SQLite's write and search semantics on Postgres

- UnitOfWork transactions take one advisory lock for the database on
  Postgres (DatabaseEngine.serializeWrites, pg_advisory_xact_lock), so a
  MAX(...) + 1 read inside one cannot race another, as with SQLite's single
  writer. SQLite needs nothing.
- ilike(platform, refs) builds the case-insensitive search condition: LIKE
  on SQLite, ILIKE on Postgres, where LIKE is exact. The place searches use it.
- Plugin data erasure runs each statement in its own savepoint: on Postgres a
  failed statement would abort the user's whole erasure transaction.
This commit is contained in:
Isaias Tavares
2026-11-02 14:58:29 -03:00
parent a77705984e
commit 515caffa79
13 changed files with 98 additions and 15 deletions
+2 -1
View File
@@ -85,7 +85,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
- **Thin controller → injectable provider.** Never resurrect a plain function-module service; don't grow god services — split a domain into several modules when it outgrows one class (trips and auth already are).
- **Inject, don't reach for module globals.** No imports of the global `db` proxy from `nest/` layers at all any more — the last production importer converted onto a repository, and the export now survives only so ~190 test files' `vi.mock('…/db/database', …)` factories keep typechecking; no new code depending on the `ws` singleton or event-sink globals. There is no `DatabaseService` to inject either — take the repository(ies) you need plus `UnitOfWork` for transactions. Modules must be importable without side effects.
- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). The sequence lives in the service, never in a controller, MCP tool or RPC handler, so every surface calls the one method that holds the transaction (`ReservationsService.createWithCost` is the shape: side effects such as broadcasts come back to the caller and go out after the commit). A position or id taken as `MAX(...) + 1` is read inside the same transaction as the insert that uses it: the transaction holds the connection, so two requests at once cannot read the same MAX. `lint:tx` holds the rest. Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository`, `AdminSeeder` and `CategorySeeder` are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`.
- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). The sequence lives in the service, never in a controller, MCP tool or RPC handler, so every surface calls the one method that holds the transaction (`ReservationsService.createWithCost` is the shape: side effects such as broadcasts come back to the caller and go out after the commit). A position or id taken as `MAX(...) + 1` is read inside the same transaction as the insert that uses it: the transaction holds the connection, so two requests at once cannot read the same MAX. On Postgres, where a pool runs transactions side by side, `UnitOfWork` first takes one advisory lock for the database (`DatabaseEngine.serializeWrites`), so the same holds; a transaction opened any other way does not get it. `lint:tx` holds the rest. Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository`, `AdminSeeder` and `CategorySeeder` are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`.
- **Migrations are append-only.** A migration is identified by its class name in `mikro_orm_migrations`, and files sort by the timestamp in that name. Never rename or delete one that has shipped, and never edit its body; add a new one. No env interpolation in SQL. (The old positional `schema_version` array and the hand-kept `db/migrations.ts` test builder are both gone; the numbered migrations are the only schema history, for tests too.) A new foreign key comes with an index that leads with its column(s), or deleting the parent scans the child table (a partial index counts only when its WHERE clause is `IS NOT NULL` on those columns); `tests/unit/db/foreign-key-indexes.test.ts` fails on one without.
- **Every write endpoint validates** through the Zod pipe with a schema from `@trek/shared` — no `Record<string, unknown>` + ad-hoc checks. Also verify every referenced id exists and belongs to the same trip.
- **Every outbound `fetch`** gets a timeout, a response-size cap and boundary validation: no `as`-casting provider responses, no silent `catch → null`. User-influenced URLs go through the SSRF guard. Never fall back to another user's API key. ESLint requires an init with a `signal` on `fetch`, `safeFetch`, `safeFetchFollow` and `safeFetchAdminConfigured` alike, or a literal `timeoutMs` in the options object of `safeFetch`/`safeFetchFollow`; an init spelled `undefined`, `null` or `void 0` counts as none (`safeFetchLlm` takes `LLM_TIMEOUT_MS` inside). Behind that, the guarded wrappers (`utils/ssrfGuard.ts`) cap every body at `DEFAULT_MAX_RESPONSE_BYTES` (64 MB) and, for a call without a signal, wait `DEFAULT_RESPONSE_TIMEOUT_MS` (30 s) for the headers. Pass a tighter `maxBytes` when you buffer a small answer, `timeoutMs` for a deadline over the whole exchange, and `maxBytes: null` only where the body is streamed on or counted by the caller (`pipeAsset`, the doc-sync transfers, the Ollama pull).
@@ -105,6 +105,7 @@ SQLite is the default engine and Postgres the second (`TREK_DB_URL`, see Engines
- **Insert or ignore / upsert**: `.onConflict((oc) => oc.columns([...]).doNothing())` or `.doUpdateSet({...})`, or `em.upsert`. Name the conflict target where there is one. Never `INSERT OR IGNORE`/`INSERT OR REPLACE`.
- **The new id**: `.returning('id').executeTakeFirstOrThrow()`. Never `InsertResult.insertId` or `last_insert_rowid()`; both are SQLite's rowid and undefined on Postgres.
- **Time**: `currentTimestamp`/`currentTimestampKysely`, `nowMinusDays`, `nowMinusHours`, `nowPlusSeconds`/`nowPlusSecondsKysely`, `nowDateOffset`, `dateOf`, `dateAdd`, `unixEpochToIsoKysely` from `src/db/dialect/sql-functions.ts` (the MikroORM `raw()` helpers) and `src/db/dialect/kysely-functions.ts` (their Kysely-expression twins). Never `datetime('now')`, `strftime` or `julianday`.
- **Search that ignores case**: `ilike(platform, refs)` builds the `(a LIKE ? ESCAPE '\' OR …)` condition, `ILIKE` on Postgres, where `LIKE` is exact. A bare `LIKE` is for a prefix that is exact on both engines (a settings key, a storage path).
- **Strings and casts**: `concat`/`concatKysely` instead of `||`, `substring`/`substringKysely`, `lower`/`lowerTrim`, `collateNoCase`, `castInteger`/`castIntegerKysely`, and `startsWithIsoDate`/`startsWithIsoDateKysely` instead of `GLOB`.
- **JSON columns** are read whole and parsed in the service; no `json_extract` in SQL. Each one is declared once in `src/db/json-columns.ts` (its Zod shape and the fallback a reader gets for an empty or broken row) and read with `decodeJson`/`decodeJsonResult`, written from a server-built value with `encodeJson` (`src/utils/json-column.ts`), never with a bare `JSON.parse` at the call site: one column, one answer to a broken row, logged.
- **Timestamps** in a DATETIME column are the canonical `YYYY-MM-DD HH:MM:SS` UTC text, never `toISOString()`: in SQL the two spellings do not compare. A value written from JS is a `DbTimestamp` from `dbNow()` (`src/db/types`), and a stored value is read back as an instant with `parseDbTimestamp()` (a bare `new Date()` reads the canonical text as local time). Where a response has always carried the ISO spelling, convert on the way out (`utcSuffix`), not in the column.
+13
View File
@@ -44,6 +44,7 @@ import {
dateOf,
dayDistance,
foundAgainState,
ilike,
lower,
lowerParam,
lowerTrim,
@@ -269,6 +270,18 @@ export const HELPER_CASES: readonly HelperCase[] = [
raw: (p) => rawOf(dayDistance(p, 'a', '2026-09-24')),
expect: { equals: 3 },
},
{
name: 'ilike matches in other case',
row: { a: 'Probe Place' },
raw: (p) => ({ sql: ilike(p, ['a']), params: ['%PLACE%'] }),
expect: { equals: true },
},
{
name: 'ilike keeps the escaped wildcard literal',
row: { a: 'Probe Place' },
raw: (p) => ({ sql: ilike(p, ['a']), params: ['probe\\_place'] }),
expect: { equals: false },
},
{
name: 'collateNoCase',
row: { a: 'Probe' },
+1 -1
View File
@@ -1,7 +1,7 @@
{
"scripts/db-call-graph.mjs": 1036,
"scripts/generate-entities.ts": 2259,
"src/db/repositories/Places.repository.ts": 1437,
"src/db/repositories/Places.repository.ts": 1435,
"src/db/repositories/Reservations.repository.ts": 1563,
"src/db/repositories/Trips.repository.ts": 1121,
"src/db/repositories/Users.repository.ts": 1445,
+11
View File
@@ -620,6 +620,17 @@ export function collateNoCase(platform: Platform, ref: string): RawQueryFragment
return unsupported(platform);
}
/**
* A search that ignores case over `refs`: `(a LIKE ? ESCAPE '\' OR b LIKE ?
* ...)`, one bound pattern per column, the caller escaping `%`, `_` and `\`.
* SQLite's `LIKE` already ignores case (ASCII letters only); Postgres's is
* exact, so it gets `ILIKE`, which folds every letter.
*/
export function ilike(platform: Platform, refs: readonly string[]): string {
const operator = isSqlite(platform) ? 'LIKE' : isPostgres(platform) ? 'ILIKE' : unsupported(platform);
return `(${refs.map((ref) => `${column(ref)} ${operator} ? ESCAPE '\\'`).join(' OR ')})`;
}
/**
* `datetime('now', '-N hours')` — the hour-granularity sibling of
* `nowMinusDays()` above, for `reminder-jobs.service.ts#todoTick`'s
+7 -1
View File
@@ -1,5 +1,5 @@
import type { AnyOrm } from '../orm';
import type { Options } from '@mikro-orm/core';
import type { EntityManager, Options } from '@mikro-orm/core';
/** The databases TREK can be pointed at. MySQL/MariaDB are out of scope (server/CLAUDE.md). */
type EngineId = 'sqlite' | 'postgres';
@@ -37,4 +37,10 @@ export interface DatabaseEngine {
* archive is the copy).
*/
schemaBootstrap(orm: AnyOrm, options: { snapshot: boolean }): Promise<void>;
/**
* Runs first in every `UnitOfWork` transaction, on its EntityManager: makes
* the transactions that write run one at a time, as SQLite's single writer
* does, so a `MAX(...) + 1` read inside one cannot race another.
*/
serializeWrites(em: EntityManager): Promise<void>;
}
@@ -3,6 +3,7 @@ import type { DatabaseEngine } from '../engine';
import { redactUrl } from '../redact-url';
import { TrekPostgreSqlDriver } from './postgres-driver';
import { bootstrapPostgresSchema, defaultPostgresBootstrapDeps, type PostgresBootstrapDeps } from './schema-bootstrap';
import type { EntityManager } from '@mikro-orm/core';
import { readFileSync } from 'node:fs';
@@ -53,6 +54,9 @@ export function sslOptionFor(url: string): PgSslOption | undefined {
}
}
/** The advisory lock key every write transaction takes: "TREK" in ASCII. */
const TREK_WRITE_LOCK = 0x5452454b;
/**
* The Postgres engine: MikroORM's own `pg` driver on `TREK_DB_URL`, with a
* pool of at most `TREK_DB_POOL_MAX` connections that keeps none open while
@@ -80,6 +84,17 @@ export class PostgresEngine implements DatabaseEngine {
return Promise.reject(new Error('Restoring a backup into a Postgres database is not available in this release.'));
}
/**
* `pg_advisory_xact_lock` on one key for the whole database: held until the
* transaction commits or rolls back, taken again by a savepoint in the same
* session without waiting, and shared by every process on the database.
*/
async serializeWrites(em: EntityManager): Promise<void> {
await em
.getConnection()
.execute('select pg_advisory_xact_lock(?)', [TREK_WRITE_LOCK], 'run', em.getTransactionContext());
}
async schemaBootstrap(orm: AnyOrm): Promise<void> {
await bootstrapPostgresSchema(orm, this.bootstrapDeps());
await finishSchemaBootstrap(orm);
@@ -3,6 +3,7 @@ import { resolveDbPath } from '../../db-path';
import { runSchemaBootstrap, type AnyOrm } from '../../orm';
import { SharedSqliteDriver } from '../../orm-driver';
import type { DatabaseEngine } from '../engine';
import type { EntityManager } from '@mikro-orm/core';
/**
* The SQLite file engine, TREK's default: one better-sqlite3 handle that
@@ -40,4 +41,9 @@ export class SqliteEngine implements DatabaseEngine {
schemaBootstrap(orm: AnyOrm, options: { snapshot: boolean }): Promise<void> {
return runSchemaBootstrap(orm, options);
}
serializeWrites(_em: EntityManager): Promise<void> {
// One connection, one writer: SQLite serializes the transactions already.
return Promise.resolve();
}
}
@@ -69,14 +69,20 @@ export class MaintenanceRepository {
* otherwise, so passing it through is correct both inside and outside a
* transaction. Each statement is independently best-effort: a
* slimmed-down test schema may not carry one or more of these tables at
* all, matching the legacy per-table try/catch exactly.
* all, matching the legacy per-table try/catch exactly. Each one runs in
* its own savepoint (a nested `em.transactional`), because on Postgres a
* failed statement aborts the whole transaction it is in: the savepoint
* rolls back that one statement and the user's erasure goes on.
*/
async deletePluginUserData(userId: number): Promise<void> {
this.validateRequestContext();
const ctx = this.em.getTransactionContext();
for (const table of ['plugin_user_config', 'plugin_oauth_tokens', 'plugin_oauth_state']) {
try {
await this.em.getConnection().execute(`DELETE FROM ${table} WHERE user_id = ?`, [userId], 'run', ctx);
await this.em.transactional((em) =>
em
.getConnection()
.execute(`DELETE FROM ${table} WHERE user_id = ?`, [userId], 'run', em.getTransactionContext()),
);
} catch {
/* table absent (slim schema) */
}
@@ -1,6 +1,6 @@
import type { Places } from '../entities/Places.entity';
import { wrapOptionalLongitude } from '@trek/shared';
import { absDifference, coalesceParam, columnRef, currentTimestamp, lowerTrim } from '../dialect/sql-functions';
import { absDifference, coalesceParam, columnRef, currentTimestamp, ilike, lowerTrim } from '../dialect/sql-functions';
import { type AssertRowKeys } from './_shared/rows';
import { TrekRepository } from './_shared/trek-repository';
import { tourPlaceTextPatch, type TourPlaceWrite } from './tour-place-write';
@@ -680,10 +680,8 @@ export class PlacesRepository extends TrekRepository<Places> {
.where('p.trip_id = ?', [trip_id]);
if (filters.searchPattern) {
qb.andWhere(
"(p.name LIKE ? ESCAPE '\\' OR p.address LIKE ? ESCAPE '\\' OR p.description LIKE ? ESCAPE '\\')",
[filters.searchPattern, filters.searchPattern, filters.searchPattern],
);
const fields = ['p.name', 'p.address', 'p.description'];
qb.andWhere(ilike(platform, fields), fields.map(() => filters.searchPattern));
}
if (filters.category !== undefined) {
qb.andWhere('p.category_id = ?', [filters.category]);
@@ -1,6 +1,6 @@
import { QueryFlag } from '@mikro-orm/core';
import type { Trips } from '../entities/Trips.entity';
import { coalesceParam, currentTimestamp, nowDateOffset } from '../dialect/sql-functions';
import { coalesceParam, currentTimestamp, ilike, nowDateOffset } from '../dialect/sql-functions';
import type { AssertRowKeys } from './_shared/rows';
import { TrekRepository } from './_shared/trek-repository';
import { tripAccessExpr } from './_shared/trip-access';
@@ -174,7 +174,7 @@ export class TripsRepository extends TrekRepository<Trips> {
return await this.accessibleTripsQuery(user_id)
.join('t.places_collection', 'p')
.select(['p.trip', 'p.name'])
.andWhere("(p.name LIKE ? ESCAPE '\\' OR p.address LIKE ? ESCAPE '\\')", [likePattern, likePattern])
.andWhere(ilike(this.getEntityManager().getPlatform(), ['p.name', 'p.address']), [likePattern, likePattern])
.orderBy([{ 'p.trip': 'asc' }, { 'p.name': 'asc' }])
.limit(500)
// The limit is on the joined place rows, like the legacy statement's.
+12 -1
View File
@@ -1,3 +1,4 @@
import { resolveEngine } from '../../db/engine';
import { EntityManager } from '@mikro-orm/core';
import { Injectable } from '@nestjs/common';
@@ -17,12 +18,22 @@ import { Injectable } from '@nestjs/common';
*
* Never awaits network I/O inside: the SQLite connection is one Kysely client
* with a mutex, and an open transaction holds it for every other request.
*
* The services read `MAX(...) + 1` inside these transactions on the strength
* of SQLite's single writer. On Postgres a pool of connections would let two
* of them read the same MAX, so every transaction first takes the engine's
* write lock (`DatabaseEngine.serializeWrites`), which keeps that guarantee.
*/
@Injectable()
export class UnitOfWork {
private readonly engine = resolveEngine();
constructor(private readonly em: EntityManager) {}
transactional<T>(fn: () => Promise<T>): Promise<T> {
return this.em.transactional(() => fn());
return this.em.transactional(async (em) => {
await this.engine.serializeWrites(em);
return fn();
});
}
}
@@ -34,6 +34,7 @@ import {
dateOf,
dayDistance,
foundAgainState,
ilike,
lower,
lowerParam,
lowerTrim,
@@ -175,6 +176,9 @@ describe('sql-functions (postgres)', () => {
'CAST(trunc(CAST(vr.accommodation_id AS numeric)) AS integer)',
);
expect(collateNoCase(pg, 'name').sql).toBe('name COLLATE "nocase"');
expect(ilike(pg, ['p.name', 'p.address'])).toBe("(p.name ILIKE ? ESCAPE '\\' OR p.address ILIKE ? ESCAPE '\\')");
expect(ilike(new SqlitePlatform(), ['p.name'])).toBe("(p.name LIKE ? ESCAPE '\\')");
expect(() => ilike(pg, ['p.name; drop'])).toThrow(/not a column reference/);
const whileSame = coalesceOverrideWhileSame(pg, null, 'u.tls', 'u.url', 'https://a');
expect(whileSame.sql).toBe(
'CASE WHEN u.url IS NOT DISTINCT FROM ? THEN COALESCE(?, u.tls) ELSE COALESCE(?, 0) END',
+13 -1
View File
@@ -4,11 +4,12 @@ import { TrekPostgreSqlDriver } from '../../../../src/db/engine/postgres/postgre
import { sslOptionFor } from '../../../../src/db/engine/postgres/postgres-engine';
import { redactUrl } from '../../../../src/db/engine/redact-url';
import { SharedSqliteDriver } from '../../../../src/db/orm-driver';
import type { EntityManager } from '@mikro-orm/core';
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { describe, expect, it } from 'vitest';
import { describe, expect, it, vi } from 'vitest';
const PG_URL = 'postgresql://trek:s3cret@db.example:5432/trek';
@@ -113,4 +114,15 @@ describe('buildOrmConfig', () => {
expect(() => engine.close()).not.toThrow();
await expect(engine.reopen()).rejects.toThrow(/not available in this release/);
});
it('ENGINE-033: Postgres serializes write transactions with one advisory lock taken in the transaction, SQLite needs none', async () => {
const execute = vi.fn(async () => []);
const trx = { trx: true };
const em = { getConnection: () => ({ execute }), getTransactionContext: () => trx } as unknown as EntityManager;
await resolveEngine(deriveDb({ TREK_DB_URL: PG_URL })).serializeWrites(em);
expect(execute).toHaveBeenCalledWith('select pg_advisory_xact_lock(?)', [0x5452454b], 'run', trx);
execute.mockClear();
await resolveEngine().serializeWrites(em);
expect(execute).not.toHaveBeenCalled();
});
});