build(server): hold SQLite-only SQL spellings to a shrinking baseline

ESLint keeps raw SQL out of the repositories, but nothing looked at the rest
of src/: the entities, the domain services, the src/db lifecycle files or a
migration added from now on. scripts/dialect-lint.mjs (npm run lint:dialect)
reads every string and template literal under src/, skipping comments, and
counts INSERT OR IGNORE/REPLACE, datetime(), strftime, json_extract, GLOB,
last_insert_rowid, AUTOINCREMENT, PRAGMA outside the lifecycle files, ||
concatenation in SQL and .insertId reads. scripts/dialect-baseline.json
holds today's thirteen hits; the counts may only go down, like lint:size.
The dialect layer is not scanned, and neither are the migrations that have
shipped.

The four seeders that still spelled INSERT OR IGNORE (addons, photo and
document providers, the schema_version marker) are Kysely inserts that do
nothing on conflict now, so a re-run still leaves an operator's rows alone.
They leave the SQL-string exemption list in eslint.config.mjs and their pins
in the selector test.
This commit is contained in:
Maurice
2026-10-08 15:21:17 +02:00
parent c524454f47
commit 39bb8f9619
14 changed files with 689 additions and 64 deletions
+4 -1
View File
@@ -21,6 +21,7 @@ node scripts/coverage-thresholds.mjs # after test:coverage — prints the ratch
npm run db:call-graph -- --sync --tx [--domains nest/days,…] # sync DB-touching methods + transaction sites (exit 1 if any)
npm run lint:size # no file in src/ or scripts/ past 1000 lines, longer ones only shrink (CI gate)
npm run lint:boundaries # import cycles and cross-domain reach-ins may only shrink (CI gate)
npm run lint:dialect # SQLite-only SQL spellings under src/ may only shrink (CI gate)
npm run contracts:dto-open # open shapes in the createZodDto schemas may only shrink (runs in the unit tests)
```
@@ -34,6 +35,8 @@ Single test: `npx vitest run tests/unit/nest/weather.controller.test.ts`, or `np
**Build emits even with type errors** (`scripts/build.mjs` catches `tsc` failures and warns). `npm run build` succeeding does **not** mean the code typechecks — run `npm run typecheck`.
**`lint:dialect`** (`scripts/dialect-lint.mjs`) reads every string and template literal under `src/` (comments do not count) and every `.insertId` read, and counts the SQLite-only spellings per file and rule: `INSERT OR IGNORE`/`REPLACE`, `datetime()`/`julianday()`, `strftime`, `json_extract` and its siblings, `GLOB`, `last_insert_rowid`, `AUTOINCREMENT`, a `PRAGMA` statement outside the `src/db/*.ts` lifecycle files and the migrations, and `||` concatenation in SQL. The counts are held to `scripts/dialect-baseline.json`; a new hit fails, and so does an entry above what its file holds now, so the change that removes one runs `npm run lint:dialect -- --update` (it never raises or adds an entry). `src/db/dialect/sql-functions.ts` is not scanned, and neither are the migrations up to `FROZEN_MIGRATIONS_THROUGH` in the script; a newer migration is held to the DML rules, while its DDL stays engine specific.
## Architecture invariants
Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`controller` + `service` + `module`, registered in `src/nest/app.module.ts`); business logic lives in the injectable `.service.ts`, pure halves in `.helpers.ts`, SQL in a `.repository.ts` where one exists. There is no legacy service layer — ESLint errors on any `services/` import. **`weather/` is the reference module shape; `collections.controller.ts` is the reference for body validation and enumeration-safe access checks.**
@@ -50,7 +53,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
- **Thin controller → injectable provider.** Never resurrect a plain function-module service; don't grow god services — split a domain into several modules when it outgrows one class (trips and auth already are).
- **Inject, don't reach for module globals.** No imports of the global `db` proxy from `nest/` layers at all any more — the last production importer converted onto a repository, and the export now survives only so ~190 test files' `vi.mock('…/db/database', …)` factories keep typechecking; no new code depending on the `ws` singleton or event-sink globals. There is no `DatabaseService` to inject either — take the repository(ies) you need plus `UnitOfWork` for transactions. Modules must be importable without side effects.
- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository` and the six existing seeders are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`.
- **Transactions are not optional.** Multi-statement writes go in `await this.uow.transactional(async () => …)` (`UnitOfWork` from `nest/database`, provided by the global `OrmModule`); never hand-roll `BEGIN`/`COMMIT` and never call `db.transaction()` (the Phase 1 async sweep removes it). Every DB-touching method is `async`; `@typescript-eslint/no-floating-promises` and `no-misused-promises` are errors, so an unawaited call or a `Promise` in a condition fails lint. Bind values with `?`; identifiers only via a literal allow-list. In `src/db/repositories/` and `src/db/seeders/`, ESLint refuses raw SQL in any spelling: `raw()`, the `sql` tag, and SQL text handed to `execute`/`run`/`all`/`get`. Build the statement with the query builder; `MaintenanceRepository`, `AdminSeeder` and `CategorySeeder` are the named exceptions in `eslint.config.mjs`, and `tests/unit/eslint-rules/restricted-syntax-selectors.test.ts` pins how many SQL strings each still holds, so they can only lose SQL, never gain it. An insert that needs the new id reads it with `.returning('id')`: `InsertResult.insertId` is SQLite's `last_insert_rowid` and undefined on Postgres, and ESLint refuses it in `src/db/repositories/` and `src/db/seeders/`.
- **Migrations are append-only.** A migration is identified by its class name in `mikro_orm_migrations`, and files sort by the timestamp in that name. Never rename or delete one that has shipped, and never edit its body; add a new one. No env interpolation in SQL. (The old positional `schema_version` array and the hand-kept `db/migrations.ts` test builder are both gone; the numbered migrations are the only schema history, for tests too.) A new foreign key comes with an index that leads with its column(s), or deleting the parent scans the child table (a partial index counts only when its WHERE clause is `IS NOT NULL` on those columns); `tests/unit/db/foreign-key-indexes.test.ts` fails on one without.
- **Every write endpoint validates** through the Zod pipe with a schema from `@trek/shared` — no `Record<string, unknown>` + ad-hoc checks. Also verify every referenced id exists and belongs to the same trip.
- **Every outbound `fetch`** gets a timeout, a response-size cap and boundary validation: no `as`-casting provider responses, no silent `catch → null`. User-influenced URLs go through the SSRF guard. Never fall back to another user's API key. ESLint requires an init with a `signal` on `fetch`, `safeFetch`, `safeFetchFollow` and `safeFetchAdminConfigured` alike, or a literal `timeoutMs` in the options object of `safeFetch`/`safeFetchFollow`; an init spelled `undefined`, `null` or `void 0` counts as none (`safeFetchLlm` takes `LLM_TIMEOUT_MS` inside). Behind that, the guarded wrappers (`utils/ssrfGuard.ts`) cap every body at `DEFAULT_MAX_RESPONSE_BYTES` (64 MB) and, for a call without a signal, wait `DEFAULT_RESPONSE_TIMEOUT_MS` (30 s) for the headers. Pass a tighter `maxBytes` when you buffer a small answer, `timeoutMs` for a deadline over the whole exchange, and `maxBytes: null` only where the body is streamed on or counted by the caller (`pipeAsset`, the doc-sync transfers, the Ollama pull).
+3 -7
View File
@@ -360,21 +360,17 @@ export default tseslint.config(
// same and passed unseen, in the seeders too, which no repository rule
// reached. The files below already hold such strings and are named one by
// one, so the list can only shrink: MaintenanceRepository is the SQLite
// maintenance adapter (PRAGMA, VACUUM INTO), and every seeder but
// DatabaseSeeder writes its rows as SQL text (four of them with the
// SQLite-only INSERT OR IGNORE). A new seeder starts on the query builder.
// maintenance adapter (PRAGMA, VACUUM INTO), and AdminSeeder and
// CategorySeeder still write their rows as SQL text. A new seeder starts on
// the query builder.
// tests/unit/eslint-rules/restricted-syntax-selectors.test.ts pins how many
// SQL strings each of them holds, so new SQL text in an exempt file fails
// there, and a file with none left has to leave this list.
files: ['src/db/repositories/**/*.ts', 'src/db/seeders/**/*.ts'],
ignores: [
'src/db/repositories/MaintenanceRepository.ts',
'src/db/seeders/AddonSeeder.ts',
'src/db/seeders/AdminSeeder.ts',
'src/db/seeders/CategorySeeder.ts',
'src/db/seeders/DocumentProviderSeeder.ts',
'src/db/seeders/PhotoProviderSeeder.ts',
'src/db/seeders/SchemaVersionSeeder.ts',
],
rules: {
'no-restricted-syntax': [
+1
View File
@@ -17,6 +17,7 @@
"db:call-graph": "node scripts/db-call-graph.mjs",
"lint:size": "node scripts/size-lint.mjs",
"lint:boundaries": "node scripts/import-boundaries.mjs",
"lint:dialect": "node scripts/dialect-lint.mjs",
"contracts:dto-open": "node scripts/dto-open-shapes.mjs",
"gen:plugin-facts": "node --import tsx scripts/gen-plugin-facts.ts",
"check:plugin-facts": "node --import tsx scripts/gen-plugin-facts.ts --check",
+38
View File
@@ -0,0 +1,38 @@
{
"src/db/document-provider-seed.ts": {
"insert-or": 2
},
"src/db/entities/IdempotencyKeys.entity.ts": {
"strftime": 1
},
"src/db/entities/PlaceShadowPicks.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginCapabilityAudit.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginEgressHosts.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginEntityMetadata.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginOauthTokens.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginScheduledTasks.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginUserConfig.entity.ts": {
"datetime": 1
},
"src/db/entities/PluginUserErasureQueue.entity.ts": {
"datetime": 1
},
"src/db/repositories/MaintenanceRepository.ts": {
"pragma": 1
},
"src/nest/backup/backup-archive.ts": {
"pragma": 1
}
}
+311
View File
@@ -0,0 +1,311 @@
#!/usr/bin/env node
/*
* lint:dialect: SQLite-only SQL may only shrink.
*
* The server runs on SQLite today and is meant to run on Postgres next. What
* stands in the way is SQL that only SQLite understands, spelled out inline:
* INSERT OR IGNORE, datetime('now'), strftime, json_extract, GLOB,
* last_insert_rowid, PRAGMA, AUTOINCREMENT and `||` string concatenation.
* Each of them has a dialect-neutral form: the query builder's onConflict(),
* returning('id'), or a helper in src/db/dialect/sql-functions.ts, which
* dispatches on the live platform and is the one file allowed to spell the
* SQLite forms.
*
* The check reads every string and template literal under src/ (comments
* never count) and every `.insertId` read, counts the hits per file and rule,
* and holds them to scripts/dialect-baseline.json. A new hit fails, and so
* does a baseline entry above what its file holds now, so the change that
* removes one lowers the baseline with --update in the same commit.
*
* Not scanned: src/db/dialect/sql-functions.ts (the dialect layer itself) and
* the migrations that have shipped (append-only, never edited again; the last
* one is FROZEN_MIGRATIONS_THROUGH). A newer migration is scanned for the
* DML rules; its DDL may still be engine specific. PRAGMA is allowed in the
* src/db/ lifecycle files (src/db/*.ts and the migrations), nowhere else.
*
* npm run lint:dialect check against the baseline (CI)
* npm run lint:dialect -- --update lower the baseline to what the files hold now;
* it never raises or adds an entry
*
* --dir=<path> points the check at another server root (the unit tests use it).
*/
import { createRequire } from 'node:module';
import { existsSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from 'node:fs';
import { join, relative, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
const require = createRequire(import.meta.url);
const ts = require('typescript');
/** The last migration that shipped before this check existed. Later ones are scanned. */
export const FROZEN_MIGRATIONS_THROUGH = 'Migration20200101042400';
/** The dialect layer: the one file that may spell SQLite's own SQL. */
const DIALECT_LAYER = 'src/db/dialect/sql-functions.ts';
const SQL_KEYWORDS = /\b(SELECT|UPDATE|INSERT|DELETE|WHERE|SET|CASE|WHEN|COALESCE|CAST|VALUES|FROM)\b/i;
/**
* Each rule: a name, a test over one literal's text, the advice the failure
* prints, and whether it applies to a migration newer than the frozen ones.
*/
export const RULES = [
{
name: 'insert-or',
test: (text) => /\bINSERT\s+OR\s+(IGNORE|REPLACE)\b|\bREPLACE\s+INTO\b/i.test(text),
advice: "use the query builder's onConflict((oc) => oc.columns([...]).doNothing() / .doUpdateSet(...))",
migrations: true,
},
{
name: 'datetime',
test: (text) => /\b(datetime|julianday|unixepoch)\s*\(/i.test(text),
advice: 'use currentTimestamp/nowMinusDays/nowPlusSeconds and friends from sql-functions.ts',
migrations: true,
},
{
name: 'strftime',
test: (text) => /\bstrftime\b/i.test(text),
advice: 'use unixEpochToIsoKysely or another sql-functions.ts helper',
migrations: true,
},
{
name: 'json',
test: (text) => /\bjson_(extract|each|set|patch|insert|replace|remove|group_array|group_object)\b/i.test(text),
advice: 'read the JSON column and parse it in the service, or add a helper to sql-functions.ts',
migrations: true,
},
{
name: 'glob',
test: (text) => /\bGLOB\b/.test(text) || text.trim().toLowerCase() === 'glob',
advice: 'use LIKE, or startsWithIsoDate and its siblings in sql-functions.ts',
migrations: true,
},
{
name: 'last-insert-rowid',
test: (text) => /\blast_insert_rowid\b/i.test(text),
advice: "read the new id with .returning('id')",
migrations: true,
},
{
name: 'autoincrement',
test: (text) => /\bAUTOINCREMENT\b/i.test(text),
advice: 'tables are created by migrations, not by query-builder code',
migrations: false,
},
{
name: 'pragma',
// The statement, `PRAGMA name`: the bare word is also an HTTP header.
test: (text) => /\bPRAGMA\s+[a-z_]/i.test(text),
advice: 'PRAGMA belongs in the src/db/ lifecycle files (connection, durability, migrations)',
migrations: false,
},
{
name: 'concat',
test: (text, inSqlTag) => text.includes('||') && (inSqlTag || SQL_KEYWORDS.test(text)),
advice: 'use concat/concatKysely from sql-functions.ts',
migrations: true,
},
];
/** `.insertId` reads, the JS face of last_insert_rowid. */
const INSERT_ID_RULE = 'insert-id';
const SOURCE = /\.ts$/;
const TEST = /\.(?:test|spec)\./;
/** 'skip', 'migration' (a newer one, DML rules only) or 'source'. */
export function classify(file) {
if (file === DIALECT_LAYER) return 'skip';
const migration = /^src\/db\/migrations\/(Migration\d+)_/.exec(file);
if (migration) return migration[1] <= FROZEN_MIGRATIONS_THROUGH ? 'skip' : 'migration';
if (file.startsWith('src/db/migrations/')) return 'skip';
return 'source';
}
/** PRAGMA is fine in src/db/*.ts (connection, durability, baseline, snapshot) and the migrations. */
export function pragmaAllowed(file) {
return /^src\/db\/[^/]+\.ts$/.test(file) || file.startsWith('src/db/migrations/');
}
function isSqlTag(node) {
let tag = node.tag;
if (ts.isPropertyAccessExpression(tag)) tag = tag.expression;
return ts.isIdentifier(tag) && tag.text === 'sql';
}
/** Hits per rule in one file's source. */
export function scanSource(file, text) {
const kind = classify(file);
const counts = {};
if (kind === 'skip') return counts;
const add = (rule) => {
counts[rule] = (counts[rule] ?? 0) + 1;
};
const sf = ts.createSourceFile(file, text, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS);
const checkText = (literal, inSqlTag) => {
for (const rule of RULES) {
if (kind === 'migration' && !rule.migrations) continue;
if (rule.name === 'pragma' && pragmaAllowed(file)) continue;
if (rule.test(literal, inSqlTag)) add(rule.name);
}
};
const visit = (node, inSqlTag) => {
if (ts.isTaggedTemplateExpression(node) && isSqlTag(node)) {
ts.forEachChild(node, (child) => visit(child, true));
return;
}
if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) {
checkText(node.text, inSqlTag);
} else if (ts.isTemplateExpression(node)) {
// The literal parts only: an interpolated value is code, scanned on its own.
checkText([node.head.text, ...node.templateSpans.map((span) => span.literal.text)].join(' '), inSqlTag);
} else if (ts.isPropertyAccessExpression(node) && node.name.text === 'insertId') {
add(INSERT_ID_RULE);
} else if (
ts.isBindingElement(node) &&
((node.propertyName && ts.isIdentifier(node.propertyName) && node.propertyName.text === 'insertId') ||
(!node.propertyName && ts.isIdentifier(node.name) && node.name.text === 'insertId'))
) {
add(INSERT_ID_RULE);
}
ts.forEachChild(node, (child) => visit(child, inSqlTag));
};
visit(sf, false);
return counts;
}
function walk(dir, files = []) {
for (const name of readdirSync(dir)) {
if (name === 'node_modules') continue;
const path = join(dir, name);
if (statSync(path).isDirectory()) walk(path, files);
else if (SOURCE.test(name) && !TEST.test(name) && !name.endsWith('.d.ts')) files.push(path);
}
return files;
}
/** `{ file: { rule: n } }` for every file under src/ with at least one hit. */
export function scan(serverDir) {
const src = join(serverDir, 'src');
if (!existsSync(src) || !statSync(src).isDirectory()) {
throw new Error(`src/ does not exist under ${serverDir}: the check would pass without looking at anything`);
}
const result = {};
for (const path of walk(src)) {
const file = relative(serverDir, path).split('\\').join('/');
const counts = scanSource(file, readFileSync(path, 'utf8'));
if (Object.keys(counts).length > 0) result[file] = sortKeys(counts);
}
return sortKeys(result);
}
function sortKeys(obj) {
return Object.fromEntries(Object.entries(obj).sort(([a], [b]) => a.localeCompare(b)));
}
const RULE_NAMES = new Set([...RULES.map((r) => r.name), INSERT_ID_RULE]);
/** The committed baseline. Missing or malformed stops the run rather than reading as empty. */
export function readBaseline(path) {
let parsed;
try {
parsed = JSON.parse(readFileSync(path, 'utf8'));
} catch (err) {
throw new Error(`scripts/dialect-baseline.json cannot be read: ${err.message}`);
}
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('scripts/dialect-baseline.json must be an object of file paths to { rule: count }');
}
for (const [file, rules] of Object.entries(parsed)) {
if (rules === null || typeof rules !== 'object' || Array.isArray(rules)) {
throw new Error(`scripts/dialect-baseline.json: ${file} must map rule names to counts`);
}
for (const [rule, n] of Object.entries(rules)) {
if (!RULE_NAMES.has(rule)) throw new Error(`scripts/dialect-baseline.json: ${file} names an unknown rule "${rule}"`);
if (!Number.isInteger(n) || n <= 0) {
throw new Error(`scripts/dialect-baseline.json: ${file} ${rule} holds ${JSON.stringify(n)}, expected a positive integer`);
}
}
}
return parsed;
}
/** The baseline lowered to what the files hold now. Never raises an entry, never adds one. */
export function lowerBaseline(baseline, counts) {
const lowered = {};
for (const [file, rules] of Object.entries(baseline)) {
const kept = {};
for (const [rule, allowed] of Object.entries(rules)) {
const now = counts[file]?.[rule] ?? 0;
if (now > 0) kept[rule] = Math.min(allowed, now);
}
if (Object.keys(kept).length > 0) lowered[file] = sortKeys(kept);
}
return sortKeys(lowered);
}
/** New hits over the baseline, and baseline entries above what their file holds now. */
export function compare(baseline, counts) {
const grown = [];
for (const [file, rules] of Object.entries(counts)) {
for (const [rule, n] of Object.entries(rules)) {
const allowed = baseline[file]?.[rule] ?? 0;
if (n > allowed) grown.push({ file, rule, n, allowed });
}
}
const lowered = lowerBaseline(baseline, counts);
const stale = [];
for (const [file, rules] of Object.entries(baseline)) {
for (const [rule, allowed] of Object.entries(rules)) {
const now = lowered[file]?.[rule] ?? 0;
if (now !== allowed) stale.push({ file, rule, allowed, now: counts[file]?.[rule] ?? 0 });
}
}
return { grown, stale };
}
function adviceFor(rule) {
if (rule === INSERT_ID_RULE) return "read the new id with .returning('id'); insertId is undefined on Postgres";
return RULES.find((r) => r.name === rule).advice;
}
function main(argv) {
const dirArg = argv.find((a) => a.startsWith('--dir='));
const serverDir = dirArg ? resolve(dirArg.slice('--dir='.length)) : fileURLToPath(new URL('..', import.meta.url));
const baselinePath = join(serverDir, 'scripts', 'dialect-baseline.json');
const update = argv.includes('--update');
const counts = scan(serverDir);
let baseline = readBaseline(baselinePath);
if (update) {
baseline = lowerBaseline(baseline, counts);
writeFileSync(baselinePath, JSON.stringify(baseline, null, 2) + '\n');
}
const { grown, stale } = compare(baseline, counts);
for (const { file, rule, n, allowed } of grown) {
console.error(`FAIL ${file}: ${n} SQLite-only ${rule} hit(s), ${allowed} allowed. ${adviceFor(rule)}.`);
}
for (const { file, rule, allowed, now } of stale) {
console.error(`FAIL ${file} is held at ${allowed} ${rule} hit(s) in scripts/dialect-baseline.json, but has ${now} now.`);
}
if (stale.length) {
console.error('Run npm run lint:dialect -- --update to lower the baseline with the change that removed them.');
}
const total = Object.values(baseline).reduce((sum, rules) => sum + Object.values(rules).reduce((a, b) => a + b, 0), 0);
console.log(`dialect: ${Object.keys(baseline).length} file(s) hold ${total} SQLite-only spelling(s) at their baseline`);
return grown.length || stale.length ? 1 : 0;
}
const isCli =
Boolean(process.argv[1]) && existsSync(process.argv[1]) && realpathSync(process.argv[1]) === fileURLToPath(import.meta.url);
if (isCli) {
try {
process.exitCode = main(process.argv.slice(2));
} catch (err) {
console.error(`FAIL ${err.message}`);
process.exitCode = 1;
}
}
+12 -8
View File
@@ -1,5 +1,7 @@
import type { EntityManager } from '@mikro-orm/core';
import type { SqlEntityManager } from '@mikro-orm/sql';
import { Seeder } from '@mikro-orm/seeder';
import type { DB } from '../kysely/db';
interface AddonRow {
id: string;
@@ -154,18 +156,20 @@ const DEFAULT_ADDONS: AddonRow[] = [
/**
* Seeds the addon shelf.
*
* `INSERT OR IGNORE` per row, not a table-level emptiness check: an operator who
* has toggled an addon keeps their choice, and a release that adds a new addon
* still gets its tile on an existing install.
* One insert per row that does nothing when the id is already there, not a
* table-level emptiness check: an operator who has toggled an addon keeps their
* choice, and a release that adds a new addon still gets its tile on an
* existing install.
*/
export class AddonSeeder extends Seeder {
async run(em: EntityManager): Promise<void> {
const connection = em.getConnection();
const db = (em as SqlEntityManager).getKysely<Pick<DB, 'addons'>>();
for (const addon of DEFAULT_ADDONS) {
await connection.execute(
'INSERT OR IGNORE INTO addons (id, name, description, type, icon, enabled, sort_order) VALUES (?, ?, ?, ?, ?, ?, ?)',
[addon.id, addon.name, addon.description, addon.type, addon.icon, addon.enabled, addon.sort_order],
);
await db
.insertInto('addons')
.values({ ...addon })
.onConflict((oc) => oc.doNothing())
.execute();
}
console.log('Default addons seeded');
}
+25 -12
View File
@@ -1,32 +1,45 @@
import { DOCUMENT_PROVIDERS, DOCUMENT_PROVIDER_FIELDS } from '../document-provider-seed';
import type { EntityManager } from '@mikro-orm/core';
import type { SqlEntityManager } from '@mikro-orm/sql';
import { Seeder } from '@mikro-orm/seeder';
import type { DB } from '../kysely/db';
/**
* Seeds the document providers.
*
* The rows come from `document-provider-seed.ts`, the same constants the
* migration that introduces the tables uses, so a fresh install and an upgraded
* one end up with identical rows.
* one end up with identical rows. A row that is already there is left alone
* (the insert does nothing on conflict), so an operator's edits survive.
*/
export class DocumentProviderSeeder extends Seeder {
async run(em: EntityManager): Promise<void> {
const connection = em.getConnection();
const db = (em as SqlEntityManager).getKysely<Pick<DB, 'document_providers' | 'document_provider_fields'>>();
for (const p of DOCUMENT_PROVIDERS) {
await connection.execute(
'INSERT OR IGNORE INTO document_providers (id, name, description, icon, enabled, sort_order) VALUES (?, ?, ?, ?, 0, ?)',
[p.id, p.name, p.description, p.icon, p.sort_order],
);
await db
.insertInto('document_providers')
.values({ id: p.id, name: p.name, description: p.description, icon: p.icon, enabled: 0, sort_order: p.sort_order })
.onConflict((oc) => oc.doNothing())
.execute();
}
for (const f of DOCUMENT_PROVIDER_FIELDS) {
await connection.execute(
`INSERT OR IGNORE INTO document_provider_fields
(provider_id, field_key, label, input_type, placeholder, hint, required, secret, sort_order)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
[f.provider_id, f.field_key, f.label, f.input_type, f.placeholder, f.hint, f.required, f.secret, f.sort_order],
);
await db
.insertInto('document_provider_fields')
.values({
provider_id: f.provider_id,
field_key: f.field_key,
label: f.label,
input_type: f.input_type,
placeholder: f.placeholder,
hint: f.hint,
required: f.required,
secret: f.secret,
sort_order: f.sort_order,
})
.onConflict((oc) => oc.doNothing())
.execute();
}
}
}
+26 -24
View File
@@ -1,5 +1,7 @@
import type { EntityManager } from '@mikro-orm/core';
import type { SqlEntityManager } from '@mikro-orm/sql';
import { Seeder } from '@mikro-orm/seeder';
import type { DB } from '../kysely/db';
interface PhotoProviderRow {
id: string;
@@ -157,37 +159,37 @@ const FIELDS: PhotoProviderFieldRow[] = [
},
];
/** Seeds the photo providers and the fields their connection form asks for. */
/** Seeds the photo providers and the fields their connection form asks for, leaving a row that is already there alone. */
export class PhotoProviderSeeder extends Seeder {
async run(em: EntityManager): Promise<void> {
const connection = em.getConnection();
const db = (em as SqlEntityManager).getKysely<Pick<DB, 'photo_providers' | 'photo_provider_fields'>>();
for (const p of PROVIDERS) {
await connection.execute(
'INSERT OR IGNORE INTO photo_providers (id, name, description, icon, enabled, sort_order) VALUES (?, ?, ?, ?, ?, ?)',
[p.id, p.name, p.description, p.icon, p.enabled, p.sort_order],
);
await db
.insertInto('photo_providers')
.values({ id: p.id, name: p.name, description: p.description, icon: p.icon, enabled: p.enabled, sort_order: p.sort_order })
.onConflict((oc) => oc.doNothing())
.execute();
}
for (const f of FIELDS) {
await connection.execute(
`INSERT OR IGNORE INTO photo_provider_fields
(provider_id, field_key, label, input_type, placeholder, hint, required, secret, settings_key, payload_key, sort_order)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
[
f.provider_id,
f.field_key,
f.label,
f.input_type,
f.placeholder,
f.hint,
f.required,
f.secret,
f.settings_key,
f.payload_key,
f.sort_order,
],
);
await db
.insertInto('photo_provider_fields')
.values({
provider_id: f.provider_id,
field_key: f.field_key,
label: f.label,
input_type: f.input_type,
placeholder: f.placeholder,
hint: f.hint,
required: f.required,
secret: f.secret,
settings_key: f.settings_key,
payload_key: f.payload_key,
sort_order: f.sort_order,
})
.onConflict((oc) => oc.doNothing())
.execute();
}
}
}
+9 -7
View File
@@ -1,5 +1,7 @@
import type { EntityManager } from '@mikro-orm/core';
import type { SqlEntityManager } from '@mikro-orm/sql';
import { Seeder } from '@mikro-orm/seeder';
import type { DB } from '../kysely/db';
/**
* The version the hand-written `db/migrations.ts` chain ended on — the highest
@@ -16,17 +18,17 @@ const LEGACY_SCHEMA_VERSION = 258;
* `mikro_orm_migrations` now — but it is still an entity and still carries the
* row an installation upgraded from the hand-written chain would have.
*
* `INSERT OR IGNORE`, like every other seeder here: seeding runs again whenever
* An insert that does nothing on conflict, like every other seeder here: seeding runs again whenever
* the schema bootstrap does, and a backup restore re-runs it against a database
* that already has this row.
*/
export class SchemaVersionSeeder extends Seeder {
async run(em: EntityManager): Promise<void> {
const connection = em.getConnection();
await connection.execute('INSERT OR IGNORE INTO schema_version (id, version) VALUES (?, ?)', [
1,
LEGACY_SCHEMA_VERSION,
]);
await (em as SqlEntityManager)
.getKysely<Pick<DB, 'schema_version'>>()
.insertInto('schema_version')
.values({ id: 1, version: LEGACY_SCHEMA_VERSION })
.onConflict((oc) => oc.doNothing())
.execute();
}
}
@@ -0,0 +1,82 @@
/**
* The addon, photo-provider and document-provider seeders run again on every
* schema bootstrap and on a backup restore. They used to spell SQLite's
* `INSERT OR IGNORE`; they are query-builder inserts that do nothing on
* conflict now, and these tests pin the behaviour that has to survive that:
* a second run adds nothing, and a row an operator changed keeps the change.
*/
import { AddonSeeder } from '../../../src/db/seeders/AddonSeeder';
import { DocumentProviderSeeder } from '../../../src/db/seeders/DocumentProviderSeeder';
import { PhotoProviderSeeder } from '../../../src/db/seeders/PhotoProviderSeeder';
import { createMigrationOrm, migratorOf, rawExec, rawQuery } from '../../helpers/migration-step';
import type { MikroORM } from '@mikro-orm/sqlite';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
describe('seeders that leave existing rows alone', () => {
let orm: MikroORM;
beforeEach(async () => {
orm = await createMigrationOrm();
await migratorOf(orm).up();
});
afterEach(async () => {
await orm.close(true);
});
const count = async (table: string) =>
(await rawQuery<{ n: number }>(orm, `SELECT COUNT(*) AS n FROM ${table}`))[0].n;
it('SEEDIGN-001: AddonSeeder adds nothing the second time and keeps an operator toggle', async () => {
await new AddonSeeder().run(orm.em);
const seeded = await count('addons');
expect(seeded).toBeGreaterThan(0);
await rawExec(orm, "UPDATE addons SET enabled = 1 - enabled, sort_order = 99 WHERE id = 'mcp'");
const [before] = await rawQuery<{ enabled: number }>(orm, "SELECT enabled FROM addons WHERE id = 'mcp'");
await new AddonSeeder().run(orm.em);
expect(await count('addons')).toBe(seeded);
expect(await rawQuery(orm, "SELECT enabled, sort_order FROM addons WHERE id = 'mcp'")).toEqual([
{ enabled: before.enabled, sort_order: 99 },
]);
});
it('SEEDIGN-002: AddonSeeder restores a deleted default addon', async () => {
await new AddonSeeder().run(orm.em);
await rawExec(orm, "DELETE FROM addons WHERE id = 'collab'");
await new AddonSeeder().run(orm.em);
expect(await rawQuery(orm, "SELECT id, type, icon FROM addons WHERE id = 'collab'")).toEqual([
{ id: 'collab', type: 'trip', icon: 'Users' },
]);
});
it('SEEDIGN-003: PhotoProviderSeeder seeds providers and fields once, keeping a renamed provider', async () => {
await new PhotoProviderSeeder().run(orm.em);
const providers = await count('photo_providers');
const fields = await count('photo_provider_fields');
expect(providers).toBeGreaterThan(0);
expect(fields).toBeGreaterThan(0);
await rawExec(orm, "UPDATE photo_providers SET name = 'Renamed' WHERE id = 'synologyphotos'");
await new PhotoProviderSeeder().run(orm.em);
expect(await count('photo_providers')).toBe(providers);
expect(await count('photo_provider_fields')).toBe(fields);
expect(await rawQuery(orm, "SELECT name FROM photo_providers WHERE id = 'synologyphotos'")).toEqual([{ name: 'Renamed' }]);
});
it('SEEDIGN-004: DocumentProviderSeeder seeds providers disabled and fields once', async () => {
await new DocumentProviderSeeder().run(orm.em);
const providers = await count('document_providers');
const fields = await count('document_provider_fields');
expect(providers).toBeGreaterThan(0);
expect(await rawQuery(orm, 'SELECT DISTINCT enabled FROM document_providers')).toEqual([{ enabled: 0 }]);
await new DocumentProviderSeeder().run(orm.em);
expect(await count('document_providers')).toBe(providers);
expect(await count('document_provider_fields')).toBe(fields);
});
});
@@ -141,12 +141,8 @@ declare const repo: { run(sql: string, params?: unknown[]): Promise<unknown>; ge
// the ignores entry) has to follow.
const EXEMPT_SQL_STRINGS: Record<string, number> = {
'src/db/repositories/MaintenanceRepository.ts': 4,
'src/db/seeders/AddonSeeder.ts': 1,
'src/db/seeders/AdminSeeder.ts': 2,
'src/db/seeders/CategorySeeder.ts': 2,
'src/db/seeders/DocumentProviderSeeder.ts': 2,
'src/db/seeders/PhotoProviderSeeder.ts': 2,
'src/db/seeders/SchemaVersionSeeder.ts': 1,
};
const SQL_MESSAGE_PREFIX = 'A SQL string handed to the connection';
@@ -0,0 +1,176 @@
/**
* scripts/dialect-lint.mjs is the CI gate that holds SQLite-only SQL
* (INSERT OR IGNORE, datetime(), strftime, json_extract, GLOB, PRAGMA,
* AUTOINCREMENT, `||` concatenation, insertId) to its baseline. A gate nobody
* tests passes forever once it breaks, so this runs the real script as a
* child process against throwaway server roots (--dir): it must fail on each
* spelling, leave comments and the dialect layer alone, and refuse to run on
* a broken setup instead of passing silently.
*/
import { execFileSync } from 'node:child_process';
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
const SCRIPT = path.join(__dirname, '../../../scripts/dialect-lint.mjs');
interface ExecError {
status: number | null;
stdout: string;
stderr: string;
}
const roots: string[] = [];
function serverRoot(files: Record<string, string>, baseline: unknown = {}): string {
// null writes no baseline file at all.
const dir = mkdtempSync(path.join(tmpdir(), 'trek-dialect-lint-'));
roots.push(dir);
mkdirSync(path.join(dir, 'src'), { recursive: true });
mkdirSync(path.join(dir, 'scripts'), { recursive: true });
for (const [file, text] of Object.entries(files)) {
mkdirSync(path.dirname(path.join(dir, file)), { recursive: true });
writeFileSync(path.join(dir, file), text);
}
if (baseline !== null) {
writeFileSync(
path.join(dir, 'scripts/dialect-baseline.json'),
typeof baseline === 'string' ? baseline : JSON.stringify(baseline),
);
}
return dir;
}
function run(dir: string, ...args: string[]): { status: number | null; out: string } {
try {
const stdout = execFileSync(process.execPath, [SCRIPT, `--dir=${dir}`, ...args], {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
});
return { status: 0, out: stdout };
} catch (err) {
const { status, stdout, stderr } = err as ExecError;
return { status, out: `${stdout}${stderr}` };
}
}
const REPO = 'src/db/repositories/Probe.repository.ts';
afterEach(() => {
for (const dir of roots.splice(0)) rmSync(dir, { recursive: true, force: true });
});
describe('dialect-lint.mjs', () => {
it('DIALECT-001: passes a tree with no SQLite-only spelling', () => {
const { status, out } = run(serverRoot({ [REPO]: "export const q = 'SELECT id FROM trips WHERE id = ?';\n" }));
expect(status).toBe(0);
expect(out).toContain('dialect: 0 file(s) hold 0 SQLite-only spelling(s)');
});
it.each([
['insert-or', "void 'INSERT OR IGNORE INTO t (a) VALUES (?)';"],
['insert-or', 'void `INSERT OR REPLACE INTO t (a) VALUES (?)`;'],
['datetime', "void \"UPDATE t SET at = datetime('now')\";"],
['strftime', "void `SELECT strftime('%s', 'now')`;"],
['json', "void 'SELECT json_extract(settings, ?) FROM users';"],
['glob', "void 'SELECT id FROM t WHERE name GLOB ?';"],
['glob', "declare function where(c: string, op: string, v: string): void;\nwhere('name', 'glob', 'a*');"],
['last-insert-rowid', "void 'SELECT last_insert_rowid()';"],
['autoincrement', "void 'id INTEGER PRIMARY KEY AUTOINCREMENT';"],
['pragma', "void 'PRAGMA foreign_keys = OFF';"],
['concat', "declare const sql: (s: TemplateStringsArray, ...v: unknown[]) => unknown;\nvoid sql`a || b`;"],
['concat', "void \"SELECT first || ' ' || last FROM users\";"],
['insert-id', 'declare const r: { insertId: bigint };\nvoid Number(r.insertId);'],
['insert-id', 'declare const r: { insertId: bigint };\nconst { insertId } = r;\nvoid insertId;'],
])('DIALECT-002: fails a new %s hit and names the rule', (rule, code) => {
const { status, out } = run(serverRoot({ [REPO]: `${code}\n` }));
expect(status).toBe(1);
expect(out).toContain(`FAIL ${REPO}: 1 SQLite-only ${rule} hit(s), 0 allowed.`);
});
it('DIALECT-003: leaves comments, ordinary strings and the HTTP Pragma header alone', () => {
const code = [
'// INSERT OR IGNORE INTO t, datetime(\'now\'), PRAGMA foreign_keys',
'/** strftime and json_extract and GLOB and AUTOINCREMENT */',
"declare const res: { set(k: string, v: string): void };",
"res.set('Pragma', 'no-cache');",
"export const fallback = (a: string | null, b: string) => a || b;",
"export const label = 'one || two';",
'',
].join('\n');
expect(run(serverRoot({ 'src/nest/probe/probe.controller.ts': code })).status).toBe(0);
});
it('DIALECT-004: never scans the dialect layer or a migration that has shipped', () => {
const dir = serverRoot({
'src/db/dialect/sql-functions.ts': "export const now = `datetime('now')`;\nexport const s = 'strftime';\n",
'src/db/migrations/Migration20200101000000_baseline_schema.ts':
"export const t = 'CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT, at TEXT DEFAULT (datetime(\\'now\\')))';\n",
});
expect(run(dir).status).toBe(0);
});
it('DIALECT-005: a newer migration is held to the DML rules but may spell its own DDL and PRAGMA', () => {
const migration = 'src/db/migrations/Migration20990101000000_probe.ts';
const ddl = "export const t = 'CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT)';\nexport const p = 'PRAGMA foreign_keys = OFF';\n";
expect(run(serverRoot({ [migration]: ddl })).status).toBe(0);
const dml = "export const s = 'INSERT OR IGNORE INTO t (a) VALUES (1)';\n";
const { status, out } = run(serverRoot({ [migration]: dml }));
expect(status).toBe(1);
expect(out).toContain(`FAIL ${migration}: 1 SQLite-only insert-or hit(s), 0 allowed.`);
});
it('DIALECT-006: PRAGMA is allowed in the src/db lifecycle files, not in a repository or a domain', () => {
const pragma = "export const p = 'PRAGMA wal_checkpoint(TRUNCATE)';\n";
expect(run(serverRoot({ 'src/db/durability.ts': pragma })).status).toBe(0);
expect(run(serverRoot({ [REPO]: pragma })).status).toBe(1);
expect(run(serverRoot({ 'src/nest/backup/probe.ts': pragma })).status).toBe(1);
});
it('DIALECT-007: holds a baselined file at its count, fails a grown one and a stale entry', () => {
const one = "export const a = 'INSERT OR IGNORE INTO t (a) VALUES (1)';\n";
const two = `${one}export const b = 'INSERT OR IGNORE INTO t (a) VALUES (2)';\n`;
expect(run(serverRoot({ [REPO]: one }, { [REPO]: { 'insert-or': 1 } })).status).toBe(0);
expect(run(serverRoot({ [REPO]: two }, { [REPO]: { 'insert-or': 1 } })).status).toBe(1);
const stale = run(serverRoot({ [REPO]: one }, { [REPO]: { 'insert-or': 2 } }));
expect(stale.status).toBe(1);
expect(stale.out).toContain(`FAIL ${REPO} is held at 2 insert-or hit(s) in scripts/dialect-baseline.json, but has 1 now.`);
expect(stale.out).toContain('npm run lint:dialect -- --update');
});
it('DIALECT-008: --update only lowers, drops what is gone, and never adds an entry', () => {
const one = "export const a = 'INSERT OR IGNORE INTO t (a) VALUES (1)';\n";
const dir = serverRoot(
{ [REPO]: one, 'src/nest/new.ts': "export const p = 'PRAGMA user_version';\n" },
{ [REPO]: { 'insert-or': 3, pragma: 1 }, 'src/gone.ts': { strftime: 1 } },
);
const { status } = run(dir, '--update');
const baseline: unknown = JSON.parse(readFileSync(path.join(dir, 'scripts/dialect-baseline.json'), 'utf8'));
expect(baseline).toEqual({ [REPO]: { 'insert-or': 1 } });
// The new file's hit is still not allowed after the update.
expect(status).toBe(1);
});
it('DIALECT-009: fails closed on a missing, unparsable or malformed baseline', () => {
const files = { [REPO]: 'export {};\n' };
const missing = run(serverRoot(files, null));
expect(missing.status).toBe(1);
expect(missing.out).toContain('dialect-baseline.json cannot be read');
expect(run(serverRoot(files, '{ not json')).status).toBe(1);
expect(run(serverRoot(files, '[]')).status).toBe(1);
expect(run(serverRoot(files, { [REPO]: 3 })).status).toBe(1);
expect(run(serverRoot(files, { [REPO]: { 'no-such-rule': 1 } })).status).toBe(1);
expect(run(serverRoot(files, { [REPO]: { pragma: 0 } })).status).toBe(1);
});
it('DIALECT-010: fails closed when src/ is missing', () => {
const dir = serverRoot({ [REPO]: 'export {};\n' });
rmSync(path.join(dir, 'src'), { recursive: true });
const { status, out } = run(dir);
expect(status).toBe(1);
expect(out).toContain('src/ does not exist');
});
});