diff --git a/server/CLAUDE.md b/server/CLAUDE.md index dd5c62066..22bbd3c40 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -67,7 +67,7 @@ Single test: `npx vitest run tests/unit/nest/weather.controller.test.ts`, or `np **Build emits even with type errors** (`scripts/build.mjs` catches `tsc` failures and warns). `npm run build` succeeding does **not** mean the code typechecks — run `npm run typecheck`. -**`lint:dialect`** (`scripts/dialect-lint.mjs`) reads every string and template literal under `src/` (comments do not count) and every `.insertId` read, and counts the SQLite-only spellings per file and rule: `INSERT OR IGNORE`/`REPLACE`, `datetime()`/`julianday()`/`unixepoch()` (also as the bare name handed to `eb.fn`), `strftime`, `json_extract` and its siblings, `GLOB`, `last_insert_rowid`, `AUTOINCREMENT`, a `PRAGMA` statement outside the `src/db/*.ts` lifecycle files and the migrations, and `||` concatenation in SQL. The counts are held to `scripts/dialect-baseline.json`; a new hit fails, and so does an entry above what its file holds now, so the change that removes one runs `npm run lint:dialect -- --update` (it never raises or adds an entry). The dialect layer `src/db/dialect/` is not scanned, and neither are the migrations up to `FROZEN_MIGRATIONS_THROUGH` in the script; a newer migration is held to the DML rules. A migration after the Postgres baseline's cut (`through` in `assets/postgres/baseline.json`, read on every run) is also held to two rules no baseline entry can excuse: `ddl` (a CREATE/ALTER/DROP of a table, index, trigger or view, or a RENAME, outside `engineDdl`; a plain string as `addColumnIfMissing`'s column; a call to the SQLite-only `tableSql` or `createPlaceRegionsFollowPlaceTrigger`) and `sqlite-ddl` (DATETIME, AUTOINCREMENT, INTEGER PRIMARY KEY, WITHOUT ROWID, PRAGMA or a SQLite-only function in an `engineDdl` postgres member). The `sqlite` member is SQLite's own and not scanned. The routine under SQL dialect, Migrations, is what satisfies them. +**`lint:dialect`** (`scripts/dialect-lint.mjs`) reads every string and template literal under `src/` (comments do not count), every `.insertId` read and every MikroORM `$like`, and counts the SQLite-only spellings per file and rule: `INSERT OR IGNORE`/`REPLACE`, `datetime()`/`julianday()`/`unixepoch()` (also as the bare name handed to `eb.fn`), `strftime`, `json_extract` and its siblings (`json_object()`, `json_array()`, `json_valid()`, `json_type()`, `json_quote()` as calls), the SQLite-only functions `group_concat`, `ifnull`, `instr`, `iif`, `printf`, `typeof` (as calls or as the bare name handed to `eb.fn`) and `total(`/`changes(`, `GLOB`, `last_insert_rowid`, `AUTOINCREMENT`, a `PRAGMA` statement outside the `src/db/*.ts` lifecycle files and the migrations, `||` concatenation in SQL, and `like`: LIKE in SQL text, Kysely's `'like'`/`'not like'` operator and `$like`, which ignore case on SQLite only. A LIKE that is exact on purpose (a settings key, a storage path) carries a `// like-exact: ` comment on its line or the line above; a case-insensitive search uses `ilike()`. The counts are held to `scripts/dialect-baseline.json`; a new hit fails, and so does an entry above what its file holds now, so the change that removes one runs `npm run lint:dialect -- --update` (it never raises or adds an entry). The dialect layer `src/db/dialect/` and the SQLite engine's own code `src/db/engine/sqlite/` are not scanned, and neither are the migrations up to `FROZEN_MIGRATIONS_THROUGH` in the script; a newer migration is held to the DML rules (`like` only after the cut). A migration after the Postgres baseline's cut (`through` in `assets/postgres/baseline.json`, read on every run, or `POSTGRES_FIRST_CUT` in the script when that comes first, so regenerating the baseline over a migration never takes it off these rules) is also held to rules no baseline entry can excuse: `ddl` (a CREATE/ALTER/DROP of a table, index, trigger or view, or a RENAME, outside `engineDdl`; a plain string as `addColumnIfMissing`'s column; a call to the SQLite-only `tableSql` or `createPlaceRegionsFollowPlaceTrigger`), `sqlite-ddl` (DATETIME, AUTOINCREMENT, INTEGER PRIMARY KEY, WITHOUT ROWID, PRAGMA or a SQLite-only function in an `engineDdl` postgres member), `sqlite-member` (an `engineDdl` sqlite member holding anything but inline schema statements: CREATE/DROP of a table, index or view, ALTER TABLE, PRAGMA, and for a rebuild the `INSERT INTO ... SELECT` and the `sqlite_sequence` statements; a trigger or a data change there would never reach Postgres), `empty-postgres` (SQLite statements with an empty postgres member and no `// sqlite-only: ` comment above it) and `best-effort` (`execBestEffort`, `bestEffort`, `execUnlessTableMissing`, whose swallowed failure aborts the transaction every pending migration runs in on Postgres). The routine under SQL dialect, Migrations, is what satisfies them. **`probe:pg`** (`scripts/pg-probe.ts`, CI job `server-postgres`, on Postgres 14 and 18) measures what of the SQL runs on Postgres. Given an empty database (`TREK_PG_PROBE_URL`, or `--url=`; it refuses one holding tables it did not create) it derives a Postgres schema from the entities with MikroORM's schema generator (timestamps stay TEXT, integers are `bigint`, SQLite-only defaults and `COLLATE NOCASE` get their Postgres spelling, no foreign keys), runs every dialect helper on an in-memory SQLite and on Postgres and checks both against the same expectation (`scripts/pg-probe/helper-cases.ts`; every case must pass), then calls every public method of every class in `src/db/repositories/*.ts` once. The arguments come from the parameter types, read with the TypeScript checker (`pg-probe/arg-samples.ts`), and the calls go through a MikroORM Postgres driver that runs each statement alone in a rolled-back transaction with a statement timeout and records whether Postgres accepted it (`pg-probe/recorder.ts`), so every statement meets empty tables and nothing persists. What it measures is the statements each method reaches with those sample arguments against empty tables, not every statement the code can send. The distinct statements Postgres refuses as SQL or over a value's type (SQLSTATE class 42, 0A or 22: syntax, an unknown function or column, a datatype mismatch, an unsupported feature, a string reaching an integer or date column) are held per `Class.method` to `failing` in `scripts/pg-probe-baseline.json`: a method may fail no more than its entry, and an entry above what it fails now fails too. Class 22 counts because the sampler sends what each parameter declares (a `number` is 1, a `string` named like a date is a date, the `present` flag of a partial update's pair is true), so a value of the wrong type reaching a column is the code's: a parameter typed `string` bound to an integer column compares on SQLite and is refused on Postgres; type it as the column is and convert at the service (`toRowId`). A constraint the sample values break (class 23) or the empty tables is reported in the log and the step summary but not held. The methods whose SQL the probe did not see (it could not build their arguments, they sent no SQL because they threw first or never reach the database, or they timed out) are held to the same file's `uncovered` list: a method outside the list that goes unmeasured fails the run, and so does a listed method that is measured now or gone, so the list only shrinks and a new method the sampler cannot reach does not land unchecked. Fix such a method by giving it parameters the sampler can build or by making it reach its SQL with sample values. The job uploads the measured baseline and a JSON report per major (`pg-probe-baseline-14`, `pg-probe-baseline-18` artifacts; the baseline holds for both), so the change that fixes a statement copies the lowered baseline from there or runs `npm run probe:pg -- --update` against a local Postgres (`docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=probe postgres:17`); `--update` never raises or adds an entry in either part. While the baseline's `failing` or `uncovered` is `null` it has never been measured, and the run fails (outside `--update`) until the first measurement from the artifact is committed, so the gate cannot land on `dev` holding nothing. A run that called no repository method fails too, so a broken plan can neither pass nor seed an empty baseline. The unit tests under `tests/unit/scripts/pg-probe/` cover the probe's logic, including every helper case on SQLite. @@ -108,7 +108,7 @@ SQLite is the default engine and Postgres the second (`TREK_DB_URL`, see Engines - **Insert or ignore / upsert**: `.onConflict((oc) => oc.columns([...]).doNothing())` or `.doUpdateSet({...})`, or `em.upsert`. Name the conflict target where there is one. Never `INSERT OR IGNORE`/`INSERT OR REPLACE`. - **The new id**: `.returning('id').executeTakeFirstOrThrow()`. Never `InsertResult.insertId` or `last_insert_rowid()`; both are SQLite's rowid and undefined on Postgres. - **Time**: `currentTimestamp`/`currentTimestampKysely`, `nowMinusDays`, `nowMinusHours`, `nowPlusSeconds`/`nowPlusSecondsKysely`, `nowDateOffset`, `dateOf`, `dateAdd`, `unixEpochToIsoKysely` from `src/db/dialect/sql-functions.ts` (the MikroORM `raw()` helpers) and `src/db/dialect/kysely-functions.ts` (their Kysely-expression twins). Never `datetime('now')`, `strftime` or `julianday`. -- **Search that ignores case**: `ilike(platform, refs)` builds the `(a LIKE ? ESCAPE '\' OR …)` condition, `ILIKE` on Postgres, where `LIKE` is exact. A bare `LIKE` is for a prefix that is exact on both engines (a settings key, a storage path). +- **Search that ignores case**: `ilike(platform, refs)` builds the `(a LIKE ? ESCAPE '\' OR …)` condition, `ILIKE` on Postgres, where `LIKE` is exact. A bare `LIKE` is for a prefix that is exact on both engines (a settings key, a storage path), and says so in a `// like-exact: ` comment (`lint:dialect`). - **No NUL in text**: Postgres text cannot hold U+0000, so TREK stores it on neither engine. `src/db/nul-free.ts` drops it where every value leaves for the database (both MikroORM platforms, both Kysely dialects, a plugin's own SQLite file, the move from SQLite), filters included, so nothing above the database layer handles it and the same request leaves the same row on either engine. - **Dates bound as dates**: SQLite rolls an impossible day (`2024-02-30`) over into the next month where Postgres refuses to cast it; a helper that binds a value either engine reads as a date (`dayDistance`) refuses anything `isCalendarDate` does not accept, on both engines, and its caller checks first. - **Strings and casts**: `concat`/`concatKysely` instead of `||`, `substring`/`substringKysely`, `lower`/`lowerTrim`, `collateNoCase`, `castInteger`/`castIntegerKysely`, and `startsWithIsoDate`/`startsWithIsoDateKysely` instead of `GLOB`. @@ -117,10 +117,10 @@ SQLite is the default engine and Postgres the second (`TREK_DB_URL`, see Engines - **PRAGMA** belongs to the `src/db/` lifecycle files (`database.ts`, `connection.ts`, `durability.ts`, the migrations) and to the SQLite engine's folder, `src/db/engine/sqlite/`, where `SqliteMaintenance` holds the file's own maintenance (`wal_checkpoint`, `VACUUM INTO`) for the backup port. `MaintenanceRepository` keeps only what runs on every engine (the readiness ping, the plugin-table erasure). - **A missing helper** goes into `sql-functions.ts` (or `kysely-functions.ts` for a Kysely expression), dispatching on the live platform through `src/db/dialect/platform.ts`: a SQLite branch and a Postgres branch (one spelling under `isKnownPlatform` where both engines read it alike), `unsupported(platform)` for anything else. The Postgres branch keeps the SQLite storage model, so a clock helper renders the same `YYYY-MM-DD HH:MM:SS` UTC text. Pin the Postgres text in `tests/unit/db/dialect/sql-functions.postgres.test.ts` and add a case to `scripts/pg-probe/helper-cases.ts` so the probe runs it on both engines. `src/db/dialect/` is the only place allowed to spell an engine's own SQL. - **Migrations** that have shipped are frozen as they are. The ones up to the Postgres baseline's cut (`through` in `assets/postgres/baseline.json`) never run on Postgres; every later one runs on both engines, and its Postgres result must be exactly what MikroORM generates from the entities (`bigint`, `bigserial` ids, `__foreign` keys, TEXT timestamps with the `PG_NOW_TEXT_DEFAULT` default), which PARITY-PG holds. Nobody writes that by hand. A schema change is: - 1. Add the migration with the next timestamp and write its schema statements as `engineDdl(this, { sqlite: [...], postgres: [] })` from `src/db/migration-utils.ts`, the SQLite side by hand (a table rebuild stays a rebuild there). A column for `addColumnIfMissing` is `{ sqlite: 'note TEXT', postgres: '...' }`. Data statements go through `this.addSql` after it, by the rules above; `engineDdl` queues like `addSql`, so a `this.execute` would run before it. + 1. Add the migration with the next timestamp and write its schema statements as `engineDdl(this, { sqlite: [...], postgres: [] })` from `src/db/migration-utils.ts`, the SQLite side by hand (a table rebuild stays a rebuild there). A column for `addColumnIfMissing` is `{ sqlite: 'note TEXT', postgres: '...' }`. The sqlite member holds the schema change alone (a trigger or a data change there would never reach Postgres). Data statements go through `this.addSql` after it, by the rules above; `engineDdl` queues like `addSql`, so a `this.execute` would run before it. A migration after the cut never uses the best-effort helpers: a failed statement aborts the Postgres migration transaction, so write one that cannot fail (a guard in its WHERE clause, `tableExists`/`columnNames` first). 2. Bring the entities and the table types to the migrated schema: `npm run gen:entities`, `npm run gen:db-types`. - 3. Run `npm run gen:pg-migration` (with `TREK_TEST_DB_URL` naming a Postgres server it may create a scratch database on, or `--url=`). It builds a database the way a boot does up to the new migration, compares it with the entities by PARITY-PG's own rules (`src/db/engine/postgres/entity-schema-diff.ts`) and prints the `postgres: [...]` member to paste. Nothing to do is printed as `postgres: [],`. - 4. Push. `lint:dialect` fails a schema statement outside `engineDdl` or a SQLite spelling in its postgres member, and the Postgres lane runs the migration and PARITY-PG. + 3. Run `npm run gen:pg-migration` (with `TREK_TEST_DB_URL` naming a Postgres server it may create a scratch database on, or `--url=`). It builds a database the way a boot does up to the new migration, compares it with the entities by PARITY-PG's own rules (`src/db/engine/postgres/entity-schema-diff.ts`) and prints the `postgres: [...]` member to paste. Nothing to do is printed as `postgres: [],`; keep it only with a `// sqlite-only: ` comment above it. + 4. Push. `lint:dialect` fails a schema statement outside `engineDdl`, a SQLite spelling in its postgres member, anything but the schema change in its sqlite member and an unexplained empty postgres member, and the Postgres lane runs the migration and PARITY-PG. The catalog helpers (`tableExists`, `columnNames`, which read `src/db/dialect/catalog.ts`) answer on both engines; `tableSql` and `createPlaceRegionsFollowPlaceTrigger` are SQLite's only. A migration never branches on the engine itself (`lint:engine-dispatch`). The baseline is not regenerated for a migration after its cut: an install created from the old one only gets the change through the migration, so `check:pg-baseline` then verifies the manifest and lists the migrations after the cut, and nothing moves the cut on its own. A trigger after the cut is not yet supported by PARITY-PG's rules (only `POSTGRES_TRIGGERS` are known); add one only together with that. - **Storage model**: the generated Kysely types describe what SQLite stores (0/1 integers for flags, TEXT timestamps, JSON in TEXT). Convert at the service boundary, as the contracts already require. diff --git a/server/scripts/dialect-lint.mjs b/server/scripts/dialect-lint.mjs index 76b0abb13..43681c630 100644 --- a/server/scripts/dialect-lint.mjs +++ b/server/scripts/dialect-lint.mjs @@ -2,35 +2,45 @@ /* * lint:dialect: SQLite-only SQL may only shrink. * - * The server runs on SQLite today and is meant to run on Postgres next. What - * stands in the way is SQL that only SQLite understands, spelled out inline: - * INSERT OR IGNORE, datetime('now'), strftime, json_extract, GLOB, - * last_insert_rowid, PRAGMA, AUTOINCREMENT and `||` string concatenation. + * The server runs on SQLite and on Postgres. What stands in the way is SQL + * that only SQLite understands, spelled out inline: INSERT OR IGNORE, + * datetime('now'), strftime, json_extract, group_concat, ifnull and the other + * SQLite-only functions, GLOB, last_insert_rowid, PRAGMA, AUTOINCREMENT and + * `||` string concatenation, and LIKE, which ignores case on SQLite only. * Each of them has a dialect-neutral form: the query builder's onConflict(), * returning('id'), or a helper in src/db/dialect/ (sql-functions.ts and its - * Kysely twin kysely-functions.ts), which dispatches on the live platform - * and is the one place allowed to spell an engine's own forms. + * Kysely twin kysely-functions.ts, ilike() for a search), which dispatches on + * the live platform and is the one place allowed to spell an engine's own + * forms. A LIKE that is exact on purpose (a key or path prefix) carries a + * `like-exact: ` comment on its line or the line above. * * The check reads every string and template literal under src/ (comments - * never count) and every `.insertId` read, counts the hits per file and rule, - * and holds them to scripts/dialect-baseline.json. A new hit fails, and so - * does a baseline entry above what its file holds now, so the change that - * removes one lowers the baseline with --update in the same commit. + * never count), every `.insertId` read and every MikroORM `$like`, counts the + * hits per file and rule, and holds them to scripts/dialect-baseline.json. A + * new hit fails, and so does a baseline entry above what its file holds now, + * so the change that removes one lowers the baseline with --update in the + * same commit. * - * Not scanned: src/db/dialect/ (the dialect layer itself) and - * the migrations that have shipped (append-only, never edited again; the last - * one is FROZEN_MIGRATIONS_THROUGH). A newer migration is scanned for the - * DML rules. PRAGMA is allowed in the src/db/ lifecycle files (src/db/*.ts - * and the migrations), nowhere else. + * Not scanned: src/db/dialect/ (the dialect layer itself), the SQLite + * engine's own code (src/db/engine/sqlite/) and the migrations that have + * shipped (append-only, never edited again; the last one is + * FROZEN_MIGRATIONS_THROUGH). A newer migration is scanned for the DML rules. + * PRAGMA is allowed in the src/db/ lifecycle files (src/db/*.ts and the + * migrations), nowhere else. * * A migration after the Postgres baseline's cut (`through` in - * assets/postgres/baseline.json) runs on both engines, so its schema - * statements are held too (POST_CUT_RULES): each one goes through - * engineDdl(this, { sqlite: [...], postgres: [...] }), each column handed to - * addColumnIfMissing is { sqlite, postgres }, the SQLite-only helpers of - * migration-utils are not called, and a postgres member holds no SQLite - * spelling. These rules have no allowance: the baseline file may not hold an - * entry for them. + * assets/postgres/baseline.json, or POSTGRES_FIRST_CUT when that comes + * first: moving the manifest's cut never takes a migration off the Postgres + * installs that already run it) runs on both engines, so it is held to + * POST_CUT_RULES as well: each schema statement goes through + * engineDdl(this, { sqlite: [...], postgres: [...] }), whose sqlite member + * holds nothing but the schema change and whose postgres member is filled (or + * says why not) and holds no SQLite spelling; each column handed to + * addColumnIfMissing is { sqlite, postgres }; the SQLite-only helpers of + * migration-utils are not called, and neither are the best-effort ones, which + * abort the migration transaction on Postgres when they fail; and LIKE counts + * there too. The post-cut rules have no allowance: the baseline file may not + * hold an entry for them. * * npm run lint:dialect check against the baseline (CI) * npm run lint:dialect -- --update lower the baseline to what the files hold now; @@ -52,9 +62,22 @@ export const FROZEN_MIGRATIONS_THROUGH = 'Migration20200101042400'; /** Where the cut is read from: the migrations after its `through` run on Postgres too. */ export const POSTGRES_BASELINE_MANIFEST = 'assets/postgres/baseline.json'; +/** + * The cut Postgres support shipped with. A Postgres install can stand at it, + * and its next boot runs every later migration through the migrator, so a + * migration after it is held to POST_CUT_RULES for good, wherever the + * manifest's `through` moves later: regenerating the baseline over a migration + * does not take it off the Postgres installs that already exist. Never + * change this. + */ +export const POSTGRES_FIRST_CUT = 'Migration20200101046440'; + /** The dialect layer: the one place that may spell an engine's own SQL. */ const DIALECT_LAYER = 'src/db/dialect/'; +/** The SQLite engine's own code, which runs on SQLite alone. */ +const SQLITE_ENGINE = 'src/db/engine/sqlite/'; + const SQL_KEYWORDS = /\b(SELECT|UPDATE|INSERT|DELETE|WHERE|SET|CASE|WHEN|COALESCE|CAST|VALUES|FROM)\b/i; /** @@ -84,14 +107,30 @@ export const RULES = [ }, { name: 'json', - test: (text) => /\bjson_(extract|each|set|patch|insert|replace|remove|group_array|group_object)\b/i.test(text), + test: (text) => + /\bjson_(extract|each|set|patch|insert|replace|remove|group_array|group_object)\b/i.test(text) || + // These read as words elsewhere (an LLM API's json_object mode), so only the call counts. + /\bjson_(object|array|valid|type|quote)\s*\(/i.test(text), advice: 'read the JSON column and parse it in the service, or add a helper to sql-functions.ts', migrations: true, }, + { + name: 'sqlite-function', + // The call in SQL text, or the bare name handed to eb.fn('ifnull', [...]). + // `total` and `changes` are words and column names too, so they count + // only as a call with the parenthesis right after the name. + test: (text) => + /\b(group_concat|ifnull|instr|iif|printf|typeof)\s*\(|\b(total|changes)\(/i.test(text) || + /^(group_concat|ifnull|instr|iif|printf)$/i.test(text.trim()), + advice: + 'group_concat, ifnull, instr, iif, printf, typeof, total and changes are SQLite-only: use coalesce (both ' + + 'engines), aggregate in the service, or add a helper to sql-functions.ts that spells both engines', + migrations: true, + }, { name: 'glob', test: (text) => /\bGLOB\b/.test(text) || text.trim().toLowerCase() === 'glob', - advice: 'use LIKE, or startsWithIsoDate and its siblings in sql-functions.ts', + advice: 'use startsWithIsoDate and its siblings, or ilike(), from sql-functions.ts', migrations: true, }, { @@ -118,6 +157,27 @@ export const RULES = [ test: (text, inSqlTag) => text.includes('||') && (inSqlTag || SQL_KEYWORDS.test(text)), advice: 'use concat/concatKysely from sql-functions.ts', migrations: true, + // Postgres spells `||` itself, so a postgres member may hold it. + postgresSpells: true, + }, + { + name: 'like', + // Kysely's 'like'/'not like' operator, LIKE in a sql tag, or LIKE before a + // SQL operand (a bound value, a quoted pattern, a call) in other SQL text, + // so that prose ("a name like ...") does not count. MikroORM's `$like` + // key is counted where the object is read (scanSource). + test: (text, inSqlTag) => + /^(not\s+)?like$/i.test(text.trim()) || + (inSqlTag && /\bLIKE\b/i.test(text)) || + (/\bLIKE\s+(?:\?|'|:\w|\$\d|\w+\s*\()/i.test(text) && SQL_KEYWORDS.test(text)), + advice: + 'LIKE ignores case on SQLite and not on Postgres: build a search with ilike(platform, refs) from ' + + 'sql-functions.ts. A match that is exact on purpose (a settings key, a storage path) keeps LIKE and says so ' + + 'in a `// like-exact: ` comment on its line or the line above', + // A migration up to the cut runs on SQLite alone; a later one runs on both. + migrations: 'post-cut', + postgresSpells: true, + marker: /like-exact:\s*\S/, }, ]; @@ -140,9 +200,32 @@ const SQLITE_ONLY_DDL = /\bDATETIME\b|\bAUTOINCREMENT\b|\bINTEGER\s+PRIMARY\s+KE /** The migration-utils helpers that only speak SQLite. */ const SQLITE_ONLY_HELPERS = new Set(['tableSql', 'createPlaceRegionsFollowPlaceTrigger']); +/** + * The migration-utils helpers that swallow a failed statement. On Postgres the + * pending migrations run in one transaction, and a failed statement aborts it, + * so they cannot be best-effort there. + */ +const BEST_EFFORT_HELPERS = new Set(['execBestEffort', 'bestEffort', 'execUnlessTableMissing']); + /** The calls whose { sqlite, postgres } argument holds the DDL of each engine. */ const ENGINE_DDL_CALLS = new Set(['engineDdl', 'addColumnIfMissing']); +/** + * What an engineDdl sqlite member may hold, statement by statement: a schema + * change (CREATE/DROP of a table, index or view, an ALTER TABLE), a PRAGMA, + * and what a table rebuild needs besides: the INSERT ... SELECT that copies + * the old table into a new one the same member creates, and the + * sqlite_sequence bookkeeping that carries its AUTOINCREMENT counter over. + */ +const SQLITE_SCHEMA_STATEMENT = + /^(?:(?:CREATE|DROP)\s+(?:UNIQUE\s+)?(?:TEMP(?:ORARY)?\s+)?(?:TABLE|INDEX|VIEW)\b|ALTER\s+TABLE\b|PRAGMA\b)/i; +const SQLITE_SEQUENCE_STATEMENT = /^(?:UPDATE|DELETE\s+FROM|INSERT\s+INTO)\s+["`]?sqlite_sequence\b/i; +const CREATED_TABLE = /^CREATE\s+(?:TEMP(?:ORARY)?\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?["`]?(\w+)/i; +const TABLE_COPY = /^INSERT\s+INTO\s+["`]?(\w+)["`]?[\s(][\s\S]*\bSELECT\b/i; + +/** The comment that names why an engineDdl with SQLite statements leaves Postgres nothing to do. */ +const SQLITE_ONLY_MARKER = /sqlite-only:\s*\S/; + /** * The rules for a migration after the Postgres baseline's cut. A new * migration starts at zero: the baseline file may not hold an entry for them. @@ -160,6 +243,34 @@ export const POST_CUT_RULES = [ 'the postgres member is what Postgres runs, and DATETIME, AUTOINCREMENT, INTEGER PRIMARY KEY, WITHOUT ROWID, ' + 'PRAGMA and the SQLite-only functions are not Postgres: replace it with what npm run gen:pg-migration prints', }, + { + name: 'sqlite-member', + label: 'statement(s) in an engineDdl sqlite member that are not a schema change', + advice: + 'what the sqlite member holds never runs on Postgres, so it holds the schema change alone: CREATE/DROP of a ' + + 'table, index or view, ALTER TABLE, PRAGMA, and for a table rebuild the INSERT INTO ... SELECT ' + + 'and the sqlite_sequence statements, each written inline as a string. Move a data change (UPDATE, DELETE, ' + + 'INSERT) into this.addSql after the engineDdl call, where it runs on both engines under the DML rules. A ' + + 'trigger after the cut has no Postgres port (server/CLAUDE.md, SQL dialect, Migrations): do that work in the ' + + 'repository instead', + }, + { + name: 'empty-postgres', + label: 'engineDdl call(s) with SQLite statements and an empty postgres member', + advice: + 'fill the postgres member with what npm run gen:pg-migration prints. When it prints postgres: [] because the ' + + "change is SQLite's alone (a rebuild that leaves the Postgres schema as it is), say why in a " + + '`// sqlite-only: ` comment directly above the postgres member', + }, + { + name: 'best-effort', + label: 'best-effort step(s), which abort every pending migration on Postgres when they fail', + advice: + 'execBestEffort, bestEffort and execUnlessTableMissing swallow a failed statement, but on Postgres the pending ' + + 'migrations run in one transaction that a failed statement aborts, so the next statement and the history row ' + + 'fail and the boot is refused. Write a statement that cannot fail instead: guard it in its WHERE clause, or ' + + 'check tableExists/columnNames from src/db/migration-utils.ts first', + }, ]; const POST_CUT_RULE_NAMES = new Set(POST_CUT_RULES.map((rule) => rule.name)); @@ -173,15 +284,16 @@ function migrationId(name) { /** * 'skip', 'migration' (a newer one, DML rules only), 'post-cut' (one after - * the Postgres baseline's `through`: the DML rules and POST_CUT_RULES) or - * 'source'. + * the Postgres baseline's `through` or after POSTGRES_FIRST_CUT, whichever + * comes first: the DML rules and POST_CUT_RULES) or 'source'. */ export function classify(file, through) { - if (file.startsWith(DIALECT_LAYER)) return 'skip'; + if (file.startsWith(DIALECT_LAYER) || file.startsWith(SQLITE_ENGINE)) return 'skip'; const migration = /^src\/db\/migrations\/(Migration\d+)_/.exec(file); if (migration) { if (migration[1] <= FROZEN_MIGRATIONS_THROUGH) return 'skip'; - return migration[1] > migrationId(through) ? 'post-cut' : 'migration'; + const cut = [migrationId(through), POSTGRES_FIRST_CUT].sort()[0]; + return migration[1] > cut ? 'post-cut' : 'migration'; } if (file.startsWith('src/db/migrations/')) return 'skip'; return 'source'; @@ -239,6 +351,75 @@ function isSqlTag(node) { return ts.isIdentifier(tag) && tag.text === 'sql'; } +/** The text of a string or template literal, or null. An interpolation is code, scanned on its own. */ +function literalText(node) { + if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) return node.text; + if (ts.isTemplateExpression(node)) { + // An interpolated value reads as what it stands for in SQL, a bound value. + return [node.head.text, ...node.templateSpans.map((span) => span.literal.text)].join(' ? '); + } + return null; +} + +/** The statements of one SQL text, `--` comments and empty ones left out. */ +function statementsOf(text) { + return text + .replace(/--[^\n]*/g, ' ') + .split(';') + .map((statement) => statement.trim()) + .filter((statement) => statement.length > 0); +} + +/** The `sqlite` or `postgres` property of a per-engine object. */ +function engineMember(object, name) { + return object.properties.find( + (property) => ts.isPropertyAssignment(property) && ts.isIdentifier(property.name) && property.name.text === name, + ); +} + +/** + * How many literals of an engineDdl sqlite member hold a statement that is + * not a schema change (SQLITE_SCHEMA_STATEMENT, the sequence bookkeeping, or + * a copy into a table the same member creates). A member that is not an array + * literal, and an element that is not a literal, count too: what the check + * cannot read, it cannot let through. + */ +function sqliteMemberProblems(initializer) { + if (!ts.isArrayLiteralExpression(initializer)) return 1; + const texts = initializer.elements.map(literalText); + const created = new Set( + texts + .flatMap((text) => (text === null ? [] : statementsOf(text))) + .map((statement) => CREATED_TABLE.exec(statement)?.[1]?.toLowerCase()) + .filter(Boolean), + ); + const isSchemaChange = (statement) => + SQLITE_SCHEMA_STATEMENT.test(statement) || + SQLITE_SEQUENCE_STATEMENT.test(statement) || + created.has(TABLE_COPY.exec(statement)?.[1]?.toLowerCase()); + return texts.filter((text) => text === null || !statementsOf(text).every(isSchemaChange)).length; +} + +/** + * Whether an engineDdl call hands SQLite statements and Postgres none, with no + * `sqlite-only: ` comment above its postgres member. + */ +function emptyPostgresUnexplained(object, sourceText) { + const sqlite = engineMember(object, 'sqlite'); + const postgres = engineMember(object, 'postgres'); + if (!ts.isArrayLiteralExpression(sqlite.initializer) || sqlite.initializer.elements.length === 0) return false; + if (!ts.isArrayLiteralExpression(postgres.initializer) || postgres.initializer.elements.length > 0) return false; + const comments = ts.getLeadingCommentRanges(sourceText, postgres.getFullStart()) ?? []; + return !comments.some((range) => SQLITE_ONLY_MARKER.test(sourceText.slice(range.pos, range.end))); +} + +/** Whether `rule` applies to a file of `kind` ('source', 'migration' or 'post-cut'). */ +function ruleApplies(rule, kind) { + if (kind === 'source') return true; + if (rule.migrations === 'post-cut') return kind === 'post-cut'; + return rule.migrations; +} + /** * Hits per rule in one file's source. `through` is the Postgres baseline's * cut; a migration after it is held to POST_CUT_RULES as well. @@ -252,26 +433,33 @@ export function scanSource(file, text, through) { counts[rule] = (counts[rule] ?? 0) + 1; }; const sf = ts.createSourceFile(file, text, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS); + const lines = text.split('\n'); + // A rule's marker comment excuses a hit on its own line or the line above. + const marked = (rule, node) => { + if (!rule.marker) return false; + const line = sf.getLineAndCharacterOfPosition(node.getStart(sf)).line; + return rule.marker.test(lines[line]) || (line > 0 && rule.marker.test(lines[line - 1])); + }; // `engine` is the member of an engineDdl or addColumnIfMissing argument a - // literal sits in: 'sqlite' (SQLite's own DDL, held to nothing), 'postgres' - // (no SQLite spelling) or null (anywhere else). - const checkText = (literal, inSqlTag, engine) => { + // literal sits in: 'sqlite' (SQLite's own DDL, whose shape the engineDdl + // call checks), 'postgres' (no SQLite spelling) or null (anywhere else). + const checkText = (literal, node, inSqlTag, engine) => { if (engine === 'sqlite') return; if (engine === 'postgres') { - // Postgres spells `||` itself; every other rule names a form only SQLite has. const sqliteOnly = - SQLITE_ONLY_DDL.test(literal) || RULES.some((rule) => rule.name !== 'concat' && rule.test(literal, inSqlTag)); + SQLITE_ONLY_DDL.test(literal) || RULES.some((rule) => !rule.postgresSpells && rule.test(literal, inSqlTag)); if (sqliteOnly) add('sqlite-ddl'); return; } if (postCut && DDL_STATEMENT.test(literal)) add('ddl'); for (const rule of RULES) { - if (kind !== 'source' && !rule.migrations) continue; + if (!ruleApplies(rule, kind)) continue; if (rule.name === 'pragma' && pragmaAllowed(file)) continue; - if (rule.test(literal, inSqlTag)) add(rule.name); + if (rule.test(literal, inSqlTag) && !marked(rule, node)) add(rule.name); } }; + const like = RULES.find((rule) => rule.name === 'like'); const visitPerEngine = (object, inSqlTag) => { for (const property of object.properties) { @@ -289,22 +477,39 @@ export function scanSource(file, text, through) { if (postCut && engine === null && ts.isCallExpression(node)) { const name = calleeName(node); if (SQLITE_ONLY_HELPERS.has(name)) add('ddl'); + if (BEST_EFFORT_HELPERS.has(name)) add('best-effort'); // A plain string would hand Postgres the SQLite spelling of the column. if (name === 'addColumnIfMissing' && !isPerEngineObject(node.arguments[3])) add('ddl'); if (ENGINE_DDL_CALLS.has(name)) { visit(node.expression, inSqlTag, null); for (const argument of node.arguments) { - if (isPerEngineObject(argument)) visitPerEngine(argument, inSqlTag); - else visit(argument, inSqlTag, null); + if (!isPerEngineObject(argument)) { + visit(argument, inSqlTag, null); + continue; + } + if (name === 'engineDdl') { + const problems = sqliteMemberProblems(engineMember(argument, 'sqlite').initializer); + for (let i = 0; i < problems; i++) add('sqlite-member'); + if (emptyPostgresUnexplained(argument, text)) add('empty-postgres'); + } + visitPerEngine(argument, inSqlTag); } return; } } - if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) { - checkText(node.text, inSqlTag, engine); - } else if (ts.isTemplateExpression(node)) { - // The literal parts only: an interpolated value is code, scanned on its own. - checkText([node.head.text, ...node.templateSpans.map((span) => span.literal.text)].join(' '), inSqlTag, engine); + const literal = literalText(node); + if (literal !== null) { + checkText(literal, node, inSqlTag, engine); + } else if ( + engine === null && + ruleApplies(like, kind) && + (ts.isPropertyAssignment(node) || ts.isShorthandPropertyAssignment(node)) && + (ts.isIdentifier(node.name) || ts.isStringLiteral(node.name)) && + node.name.text === '$like' && + !marked(like, node) + ) { + // MikroORM's `{ title: { $like: ... } }` is the same LIKE as in SQL text. + add('like'); } else if (ts.isPropertyAccessExpression(node) && node.name.text === 'insertId') { add(INSERT_ID_RULE); } else if ( diff --git a/server/src/db/pre-migrate-snapshot.ts b/server/src/db/pre-migrate-snapshot.ts index 03f2b25e0..49ba16d04 100644 --- a/server/src/db/pre-migrate-snapshot.ts +++ b/server/src/db/pre-migrate-snapshot.ts @@ -45,6 +45,7 @@ async function databaseFile(connection: Connection): Promise { /** A fresh install: nothing but the migrator's own bookkeeping table yet. */ async function holdsAnything(connection: Connection): Promise { const rows: unknown[] = await connection.execute( + // like-exact: SQLite's own `sqlite_` tables, on the SQLite file this snapshot is for. `SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' AND name <> 'mikro_orm_migrations' LIMIT 1`, ); return rows.length > 0; diff --git a/server/src/db/repositories/AppSettings.repository.ts b/server/src/db/repositories/AppSettings.repository.ts index 657a17d87..543df0dcd 100644 --- a/server/src/db/repositories/AppSettings.repository.ts +++ b/server/src/db/repositories/AppSettings.repository.ts @@ -81,6 +81,7 @@ export class AppSettingsRepository extends TrekRepository { * a prefix that itself contains `%`/`_`. */ async findByKeyPrefix(prefix: string): Promise { + // like-exact: setting keys are lowercase identifiers, matched by their exact prefix on both engines. const rows = await this.find({ key: { $like: `${prefix}%` } }); return rows.map((row) => toRow(row) as AppSettingsRow); } diff --git a/server/src/db/repositories/Settings.repository.ts b/server/src/db/repositories/Settings.repository.ts index 0a65440a5..d22a167e0 100644 --- a/server/src/db/repositories/Settings.repository.ts +++ b/server/src/db/repositories/Settings.repository.ts @@ -87,6 +87,7 @@ export class SettingsRepository extends TrekRepository { * parameter — never interpolated into SQL text. */ async deleteByKeyPrefix(prefix: string): Promise { + // like-exact: a plugin's setting keys start with its exact `plugin::` prefix on both engines. return await this.nativeDelete({ key: { $like: `${prefix}%` } }); } } diff --git a/server/src/db/repositories/TrekPhotos.repository.ts b/server/src/db/repositories/TrekPhotos.repository.ts index 1081135dd..5ee7f02ea 100644 --- a/server/src/db/repositories/TrekPhotos.repository.ts +++ b/server/src/db/repositories/TrekPhotos.repository.ts @@ -201,12 +201,14 @@ export class TrekPhotosRepository extends TrekRepository { * never match `LIKE`, so every returned value is a real string. */ async listByFilePathPrefix(prefix: string): Promise { + // like-exact: a storage path prefix, exact on both engines. const rows = await this.find({ file_path: { $like: `${prefix}%` } }, { fields: ['file_path'] }); return rows.map((row) => row.file_path).filter((path): path is string => path != null); } /** TH2 — `SELECT thumbnail_path FROM trek_photos WHERE thumbnail_path LIKE 'journey/thumbs/%'`, same shape as TH1 on the sibling column. */ async listByThumbnailPathPrefix(prefix: string): Promise { + // like-exact: a storage path prefix, exact on both engines. const rows = await this.find({ thumbnail_path: { $like: `${prefix}%` } }, { fields: ['thumbnail_path'] }); return rows.map((row) => row.thumbnail_path).filter((path): path is string => path != null); } diff --git a/server/src/nest/storage/storage-reclaim-source.service.ts b/server/src/nest/storage/storage-reclaim-source.service.ts index 08402f3d5..5bbfcd748 100644 --- a/server/src/nest/storage/storage-reclaim-source.service.ts +++ b/server/src/nest/storage/storage-reclaim-source.service.ts @@ -129,7 +129,8 @@ export function likeAny( const terms = containing === null ? fields.map((field) => ({ [field]: { $ne: null } })) - : fields.flatMap((field) => containing.map((run) => ({ [lower(platform, field)]: { $like: `%${run}%` } }))); + : // like-exact: the column is lowercased and the runs are lowercase, so the match ignores case on both engines. + fields.flatMap((field) => containing.map((run) => ({ [lower(platform, field)]: { $like: `%${run}%` } }))); if (terms.length > MAX_LIKE_TERMS) { logWarn(`[storage] a reference lookup asked for ${terms.length} LIKE terms, more than ${MAX_LIKE_TERMS}`); throw new Error(`a reference lookup may hold at most ${MAX_LIKE_TERMS} LIKE terms`); diff --git a/server/tests/unit/scripts/dialect-lint.test.ts b/server/tests/unit/scripts/dialect-lint.test.ts index 36a5c051f..5e424b3fe 100644 --- a/server/tests/unit/scripts/dialect-lint.test.ts +++ b/server/tests/unit/scripts/dialect-lint.test.ts @@ -90,6 +90,14 @@ describe('dialect-lint.mjs', () => { ['concat', 'void "SELECT first || \' \' || last FROM users";'], ['insert-id', 'declare const r: { insertId: bigint };\nvoid Number(r.insertId);'], ['insert-id', 'declare const r: { insertId: bigint };\nconst { insertId } = r;\nvoid insertId;'], + ['json', "void 'SELECT json_object(?, a) FROM t';"], + ['sqlite-function', 'void "SELECT trip_id, group_concat(name, \', \') FROM places GROUP BY trip_id";'], + ['sqlite-function', "void 'SELECT ifnull(SUM(cost), 0) AS total FROM places WHERE trip_id = ?';"], + ['sqlite-function', "declare function fn(name: string, args: string[]): void;\nfn('ifnull', ['a', 'b']);"], + ['sqlite-function', "void 'SELECT total(cost) FROM places';"], + ['like', "declare function find(where: unknown): void;\nfind({ title: { $like: '%paris%' } });"], + ['like', "declare function where(c: string, op: string, v: string): void;\nwhere('title', 'like', '%paris%');"], + ['like', "void 'SELECT id FROM collections WHERE title LIKE ?';"], ])('DIALECT-002: fails a new %s hit and names the rule', (rule, code) => { const { status, out } = run(serverRoot({ [REPO]: `${code}\n` })); expect(status).toBe(1); @@ -109,20 +117,49 @@ describe('dialect-lint.mjs', () => { expect(run(serverRoot({ 'src/nest/probe/probe.controller.ts': code })).status).toBe(0); }); + it('DIALECT-017: leaves prose that reads like SQL alone, and a LIKE marked exact', () => { + const prose = [ + 'export const a = \'For a question like "what if it rains", look up the weather from the forecast\';', + "export const b = 'Pass the total (the places the filter matches) and the changes (name, route)';", + "export const c = { type: 'json_object' };", + '', + ].join('\n'); + expect(run(serverRoot({ 'src/nest/probe/probe.mcp.ts': prose })).status).toBe(0); + const exact = [ + 'declare function find(where: unknown): void;', + '// like-exact: setting keys match their exact prefix on both engines.', + "find({ key: { $like: 'perm_%' } });", + "find({ key: { $like: 'plugin:%' } }); // like-exact: a plugin's key prefix", + '', + ].join('\n'); + expect(run(serverRoot({ [REPO]: exact })).status).toBe(0); + // A marker without a reason excuses nothing. + const bare = "declare function find(where: unknown): void;\n// like-exact:\nfind({ key: { $like: 'a%' } });\n"; + expect(run(serverRoot({ [REPO]: bare })).status).toBe(1); + }); + it('DIALECT-004: never scans the dialect layer or a migration that has shipped', () => { const dir = serverRoot({ 'src/db/dialect/sql-functions.ts': "export const now = `datetime('now')`;\nexport const s = 'strftime';\n", 'src/db/dialect/kysely-functions.ts': "export const g = 'glob';\nexport const s = 'strftime';\n", 'src/db/migrations/Migration20200101000000_baseline_schema.ts': "export const t = 'CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT, at TEXT DEFAULT (datetime(\\'now\\')))';\n", + 'src/db/engine/sqlite/sqlite-source.ts': "export const q = 'select distinct typeof(a) from t';\n", }); expect(run(dir).status).toBe(0); }); it('DIALECT-005: a newer migration up to the Postgres cut is held to the DML rules but may spell its own DDL', () => { const migration = 'src/db/migrations/Migration20200101043000_probe.ts'; - const ddl = - "export const t = 'CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT)';\nexport const p = 'PRAGMA foreign_keys = OFF';\n"; + // It runs on SQLite alone, so a LIKE and a best-effort step are SQLite's business there. + const ddl = [ + "export const t = 'CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT)';", + "export const p = 'PRAGMA foreign_keys = OFF';", + "export const l = 'UPDATE t SET a = 1 WHERE b LIKE ?';", + 'declare function execBestEffort(m: unknown, s: string): void;', + "execBestEffort(null, 'UPDATE t SET a = 1');", + '', + ].join('\n'); expect(run(serverRoot({ [migration]: ddl })).status).toBe(0); const dml = "export const s = 'INSERT OR IGNORE INTO t (a) VALUES (1)';\n"; @@ -183,7 +220,7 @@ describe('dialect-lint.mjs', () => { const POST_CUT = 'src/db/migrations/Migration20200101046450_probe.ts'; const migration = (body: string) => [ - "import { addColumnIfMissing, engineDdl, tableSql } from '../migration-utils';", + "import { addColumnIfMissing, bestEffort, engineDdl, execBestEffort, execUnlessTableMissing, tableSql } from '../migration-utils';", 'export class Probe {', ' async up(this: never): Promise {', body, @@ -243,18 +280,106 @@ describe('dialect-lint.mjs', () => { it("DIALECT-014: Postgres's own || in a postgres member and SQLite DDL in a sqlite member pass", () => { const body = [ - " engineDdl(this, { sqlite: ['CREATE TRIGGER t AFTER UPDATE ON a BEGIN SELECT 1; END'], postgres: [", + " engineDdl(this, { sqlite: ['CREATE TABLE t (a INTEGER PRIMARY KEY AUTOINCREMENT, b TEXT)'], postgres: [", " 'create function f() returns trigger language plpgsql as $$ begin new.a := new.b || new.c; return new; end $$',", ' ] });', ].join('\n'); expect(run(serverRoot({ [POST_CUT]: migration(body) })).status).toBe(0); }); - it('DIALECT-015: the cut is read from the Postgres manifest, and moves with it', () => { - const body = " this.addSql('CREATE INDEX i ON t (a)');"; - const later = JSON.stringify({ through: 'Migration20200101046450_probe', migrations: [] }); - expect(run(serverRoot({ [POST_CUT]: migration(body), [MANIFEST]: later })).status).toBe(0); + it("DIALECT-015: moving the manifest's cut over a migration does not take it off the post-cut rules", () => { + const body = " this.addSql('ALTER TABLE trips ADD COLUMN archived_at DATETIME');"; + // npm run gen:pg-baseline after the migration landed: the manifest now stands for it. + const moved = JSON.stringify({ through: 'Migration20200101046450_probe', migrations: [] }); + const { status, out } = run(serverRoot({ [POST_CUT]: migration(body), [MANIFEST]: moved })); + expect(status).toBe(1); + expect(out).toContain(`FAIL ${POST_CUT}: 1 schema statement(s) outside engineDdl, 0 allowed.`); expect(run(serverRoot({ [POST_CUT]: migration(body) })).status).toBe(1); + // A cut moved back holds more migrations to the post-cut rules, never fewer. + const earlier = JSON.stringify({ through: 'Migration20200101043000_probe', migrations: [] }); + const between = 'src/db/migrations/Migration20200101044000_probe.ts'; + expect(run(serverRoot({ [between]: migration(body), [MANIFEST]: earlier })).status).toBe(1); + }); + + it('DIALECT-018: a table rebuild in a sqlite member passes, its counter bookkeeping included', () => { + const body = [ + ' engineDdl(this, {', + ' sqlite: [', + " 'CREATE TABLE t_new (id INTEGER PRIMARY KEY AUTOINCREMENT, a REAL)',", + " 'INSERT INTO t_new (id, a) SELECT id, a FROM t',", + " \"UPDATE sqlite_sequence SET seq = (SELECT seq FROM sqlite_sequence WHERE name = 't') WHERE name = 't_new'\",", + " \"INSERT INTO sqlite_sequence (name, seq) SELECT 't_new', seq FROM sqlite_sequence WHERE name = 't'\",", + " 'DROP TABLE t',", + " 'ALTER TABLE t_new RENAME TO t',", + " 'CREATE INDEX idx_t_a ON t(a)',", + ' ],', + ' postgres: [\'alter table "t" alter column "a" type double precision\'],', + ' });', + ].join('\n'); + const { status, out } = run(serverRoot({ [POST_CUT]: migration(body) })); + expect(out).toBe('dialect: 0 file(s) hold 0 SQLite-only spelling(s) at their baseline\n'); + expect(status).toBe(0); + }); + + it.each([ + [ + 'a trigger', + "'CREATE TRIGGER trg AFTER UPDATE ON places BEGIN UPDATE trips SET updated_at = 1 WHERE id = NEW.trip_id; END'", + ], + ['a data backfill', '"UPDATE reservations SET day = strftime(\'%Y-%m-%d\', reservation_time)"'], + ['a delete', "'DELETE FROM t WHERE a IS NULL'"], + ['an insert of values', '"INSERT INTO t (a) VALUES (\'x\')"'], + ['a copy into a table the member does not create', "'INSERT INTO other SELECT * FROM t'"], + ['a schema change with a data change behind it', "'CREATE INDEX i ON t(a); UPDATE t SET a = 1'"], + ['a statement built elsewhere', 'STATEMENT'], + ])('DIALECT-019: fails %s in a sqlite member and says where it belongs', (_what, statement) => { + const body = [ + " const STATEMENT = 'UPDATE t SET a = 1';", + ` engineDdl(this, { sqlite: [${statement}], postgres: ['create index "i" on "t" ("a")'] });`, + ].join('\n'); + const { status, out } = run(serverRoot({ [POST_CUT]: migration(body) })); + expect(status).toBe(1); + expect(out).toContain( + `FAIL ${POST_CUT}: 1 statement(s) in an engineDdl sqlite member that are not a schema change, 0 allowed.`, + ); + expect(out).toContain('into this.addSql after the engineDdl call'); + }); + + it('DIALECT-020: SQLite statements with an empty postgres member need a sqlite-only reason', () => { + const empty = " engineDdl(this, { sqlite: ['CREATE INDEX i ON t(a)'], postgres: [] });"; + const { status, out } = run(serverRoot({ [POST_CUT]: migration(empty) })); + expect(status).toBe(1); + expect(out).toContain( + `FAIL ${POST_CUT}: 1 engineDdl call(s) with SQLite statements and an empty postgres member, 0 allowed.`, + ); + expect(out).toContain('// sqlite-only: '); + const explained = [ + " engineDdl(this, { sqlite: ['CREATE INDEX i ON t(a)'],", + ' // sqlite-only: the index exists on Postgres since the baseline.', + ' postgres: [] });', + ].join('\n'); + expect(run(serverRoot({ [POST_CUT]: migration(explained) })).status).toBe(0); + }); + + it.each([ + [ + 'execBestEffort', + "await execBestEffort(this, 'UPDATE budget_items SET amount_cents = CAST(amount AS INTEGER) * 100');", + ], + ['bestEffort', "await bestEffort(async () => { this.addSql('UPDATE t SET a = 1'); });"], + ['execUnlessTableMissing', "await execUnlessTableMissing(this, 'UPDATE t SET a = 1');"], + ])('DIALECT-021: fails %s, which aborts every pending migration on Postgres', (_what, body) => { + const { status, out } = run(serverRoot({ [POST_CUT]: migration(` ${body}`) })); + expect(status).toBe(1); + expect(out).toContain(`FAIL ${POST_CUT}: 1 best-effort step(s)`); + expect(out).toContain('guard it in its WHERE clause'); + }); + + it('DIALECT-022: a LIKE in a post-cut migration counts, as everywhere that runs on Postgres', () => { + const body = " this.addSql('UPDATE t SET a = 1 WHERE b LIKE ?');"; + const { status, out } = run(serverRoot({ [POST_CUT]: migration(body) })); + expect(status).toBe(1); + expect(out).toContain(`FAIL ${POST_CUT}: 1 SQLite-only like hit(s), 0 allowed.`); }); it('DIALECT-016: fails closed on a missing or malformed manifest, and on a baselined allowance', () => { @@ -268,7 +393,7 @@ describe('dialect-lint.mjs', () => { expect(broken.status).toBe(1); expect(broken.out).toContain('must name the last migration it stands for'); - for (const rule of ['ddl', 'sqlite-ddl']) { + for (const rule of ['ddl', 'sqlite-ddl', 'sqlite-member', 'empty-postgres', 'best-effort']) { const allowed = run(serverRoot(files, { [POST_CUT]: { [rule]: 1 } })); expect(allowed.status).toBe(1); expect(allowed.out).toContain(`holds an allowance for ${rule}`);