From 85b773420c850e3e69d81c2c97dd7d134d08c574 Mon Sep 17 00:00:00 2001 From: Maurice Date: Wed, 18 Nov 2026 15:22:37 +0100 Subject: [PATCH] fix(db): say what a schema difference means when the move to Postgres refuses it The refusal listed the differences and nothing else. A table or column only one side has now comes with what that means: the file is at this version's level, the start that brought it there added every column it could, the start log names the rest, and the operator keeps running on SQLite and reports it. A column of TREK's own schema the move cannot copy is listed apart and named a bug in TREK, since the file is not at fault. The server guide describes the gap, the repair, the check and the refusal. --- server/CLAUDE.md | 4 +- .../db/engine/postgres/sqlite-move/catalog.ts | 22 ++++---- .../db/engine/postgres/sqlite-move/move.ts | 50 +++++++++++++++++-- .../engine/postgres/sqlite-move/move.test.ts | 43 +++++++++++++++- .../engine/postgres/sqlite-move/order.test.ts | 8 ++- 5 files changed, 108 insertions(+), 19 deletions(-) diff --git a/server/CLAUDE.md b/server/CLAUDE.md index c77319f03..82ba178a8 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -81,8 +81,8 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire`, one conditional upsert that decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send. - **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL runs in one store per engine behind the `PluginDataStore` port (`src/db/engine/plugin-data.ts`), each in its engine's folder: `SqlitePluginDataStore` (`src/db/engine/sqlite/plugin-data-store.ts`) and `PostgresPluginDataStore` (`src/db/engine/postgres/plugin-data-store.ts`). Both hold it to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed), and refuse a call that breaks the contract with the same words, taken from `PLUGIN_DATA_ERRORS` beside the caps in `src/db/engine/plugin-data-contract.ts` (`tests/unit/db/engine/plugin-data-contract.test.ts` runs every such call on both stores and fails on one answered differently; a closed store refuses every call). Transaction control is refused in every statement of every call on both engines with one text, `PLUGIN_DATA_ERRORS.txControl` (`refuseTransactionControl` on SQLite and in the SDK, the guard on Postgres). The SDK's dev harness and mock host throw the same texts and hold the same caps from a byte-identical copy of that file, held by a parity test in `plugin-sdk/`. Where that data lives is the engine's: `PLUGIN_DATA` (`src/nest/plugins/host/plugin-data.backend.ts`) binds the SQLite file per plugin, or on Postgres a schema per plugin (`trek_plugin_`) whose SQL is kept inside that schema in layers that `src/db/engine/postgres/plugin-sql-guard.ts`'s header names, the ones after the guard enforced by the database: no plugin SQL at all while TREK's role is privileged in the sense of `PRIVILEGED_ROLE_SQL` (`src/db/engine/postgres/plugin-data-checks.ts`, read once per pool connection, and the boot log says why); the guard, which refuses what it would read differently from the Postgres lexer, other schemas, the catalogs, temporary objects, `SELECT ... INTO`, the `reg*` types, the functions that look an object up by name, a read of a system column (`xmin`, `ctid`, ...; the name may still be given with AS), the `xid` types where the word stands for the type, a collation or access method beyond the ones every server has, and every function of `pg_catalog` outside the allowlist `assets/postgres/plugin-functions.txt`, read by `src/db/engine/postgres/plugin-sql-functions.ts` (its catalog list, `assets/postgres/catalog-functions.txt`, is held to `pg_proc` by PGPLUG-021, and PGPLUG-022 fails on an allowed function with an overload that takes or returns a `reg*` type or `aclitem`, which would turn a text argument into a catalog lookup, and on one that makes a transaction id out of other values); the settings that decide how values print and parse (time zone UTC, the locales `C`, the date style, ...), pinned with `SET LOCAL` in every plugin transaction (`src/db/engine/postgres/plugin-session.ts`; PGPLUG-023 fails on a client setting of those categories a new Postgres adds until it is pinned or named as harmless); the objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX changes, resolved by Postgres (`to_regclass`/`to_regtype`) right before the statement and refused outside the plugin's schema, so an unqualified `int4` is `pg_catalog`'s and refused even on a superuser connection; one statement per round trip over the extended query protocol, which the store sends on a pool client it checks out for each call and runs its transaction on directly; `pg_temp` last on the plugin's and on every pool connection's `search_path`, `session-search-path.ts`, with the store dropping its client's temporary tables and other session state before the client goes back to the pool; and a quota over every relation of the schema with its TOAST table plus a cap on its catalog objects (`PLUGIN_SCHEMA_MAX_OBJECTS`). Plugin SQL, an argument or a migration id holding NUL is refused on both engines with one error (`refuseNulSql`/`refuseNulValues` in the same contract file, so the SDK refuses it too), since Postgres text cannot hold it and a plugin must answer the same on either; a NUL the plugin's SQLite SQL builds itself (`char(0)`) is not caught, and the move to Postgres refuses such a row; the plugin's SQL is never translated, and `db.engine` tells the plugin which dialect to write. Backups and restores carry the plugin schemas the role owns (`pg-tools.ts`). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. -- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot. It refuses the boot when `schema_version` is not exactly one row holding an integer ≥ 1, or is above the last mapped step (258); a value from 1 to 258 is read as dev's steps of those numbers, so a pre-ORM feature-branch database whose steps differ is not detected (the Tours branch is the one case it cuts back). Recorded `mikro_orm_migrations` names this build does not ship refuse it too (`refuseNewerDatabase`), whether a newer release or a development build wrote them. 4.3.x and earlier have no such check and start on a database 5.x migrated. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--