From 40f6efbf5baa2171582f4c5fbbd57c640b33eb1f Mon Sep 17 00:00:00 2001 From: Maurice Date: Tue, 17 Nov 2026 11:47:44 +0100 Subject: [PATCH] docs(db): describe the discovered Postgres lane and the 14/18 job --- CLAUDE.md | 4 ++-- server/CLAUDE.md | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 41cb44163..5f8285b26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,11 +137,11 @@ These principles come out of a full-repo audit and shape all new code: ## Quality gates in CI -`.github/workflows/test.yml` runs, per package: typecheck (server also `typecheck:tests`, client also the strict-mode ratchet `lint:strict`), lint (client and server through `lint:warnings`, which also holds the warning count per rule), `lint:pages`, `lint:rtl`, `lint:size` (client and server), the server's `lint:strict`, `lint:boundaries`, `lint:tx`, `lint:dialect`, `lint:engine-dispatch`, `lint:test-sql`, `lint:test-mocks`, `lint:test-new-service`, `lint:query-api`, `lint:mcp-zod`, `lint:service-http` and `lint:response-contracts`, the client's and server's `lint:format`, `lint:layers`, `lint:offline`, `lint:dup`, `lint:pairs`, `lint:skips`, `lint:i18n-keys` and `theme:lint`, shared's `contracts:open`, the repo-wide `lint:knip` (in server-lint-checks), `check:plugin-facts`, `i18n:parity:strict` (with the untranslated-strings ratchet), the S3 contract suite, the Postgres probe (`postgres-probe`: every dialect helper on SQLite and Postgres, and the statements every repository method reaches with sample arguments against a Postgres service; the SQL errors per method and the list of methods it cannot measure are held to a baseline that only shrinks and fails while it is still unmeasured; see `server/CLAUDE.md`), the Postgres lane (`server-postgres-tests`: the suites that boot the app, PARITY-PG and the write lock on a Postgres service), the browser end-to-end flows (Chromium and WebKit), a client build with `check:gl-split`, coverage for all four packages, then a SonarCloud scan. It also builds the Docker image and boots it until `/api/health/ready` answers (`docker-image`), and lints and renders the Helm chart against the Kubernetes schemas and runs `docker compose config` on every compose file (`deploy-manifests`). `lint-prettier.yml` additionally runs `lint` + `lint:format` on `shared/`, both check-only. `plugin-sdk` has no lint script and is outside the eslint gates. **Not in CI**: the screenshot and help-media Playwright projects; run them locally. There are no git hooks; nothing runs before a commit except you. Other workflows build Docker images, scan them, and publish `trek-plugin-sdk` on `plugin-sdk-v*` tags. +`.github/workflows/test.yml` runs, per package: typecheck (server also `typecheck:tests`, client also the strict-mode ratchet `lint:strict`), lint (client and server through `lint:warnings`, which also holds the warning count per rule), `lint:pages`, `lint:rtl`, `lint:size` (client and server), the server's `lint:strict`, `lint:boundaries`, `lint:tx`, `lint:dialect`, `lint:engine-dispatch`, `lint:test-sql`, `lint:test-mocks`, `lint:test-new-service`, `lint:query-api`, `lint:mcp-zod`, `lint:service-http` and `lint:response-contracts`, the client's and server's `lint:format`, `lint:layers`, `lint:offline`, `lint:dup`, `lint:pairs`, `lint:skips`, `lint:i18n-keys` and `theme:lint`, shared's `contracts:open`, the repo-wide `lint:knip` (in server-lint-checks), `check:plugin-facts`, `i18n:parity:strict` (with the untranslated-strings ratchet), the S3 contract suite, TREK on Postgres (`server-postgres`, once per supported major, 14 and 18: first the Postgres probe, every dialect helper on SQLite and Postgres and the statements every repository method reaches with sample arguments, its SQL errors per method and the methods it cannot measure held to a baseline that only shrinks and fails while it is still unmeasured; then the Postgres lane, every suite that boots the app on the real database module, found by `server/tests/postgres/lane.ts` rather than listed, plus PARITY-PG and the write lock; see `server/CLAUDE.md`), the browser end-to-end flows (Chromium and WebKit), a client build with `check:gl-split`, coverage for all four packages, then a SonarCloud scan. It also builds the Docker image and boots it until `/api/health/ready` answers, on SQLite and on Postgres 18 (`docker-image`), and lints and renders the Helm chart against the Kubernetes schemas and runs `docker compose config` on every compose file (`deploy-manifests`). `lint-prettier.yml` additionally runs `lint` + `lint:format` on `shared/`, both check-only. `plugin-sdk` has no lint script and is outside the eslint gates. **Not in CI**: the screenshot and help-media Playwright projects; run them locally. There are no git hooks; nothing runs before a commit except you. Other workflows build Docker images, scan them, and publish `trek-plugin-sdk` on `plugin-sdk-v*` tags. - **Job names read "package: what"** (`Server: Types`, `Client: Tests 3/7`), and the list at the top of `test.yml` says what each job holds. A new job follows the scheme and joins that list, and the job-count note there says when a long job needs a slot freed first. - **Tests run beside the gates, in shards.** The client and server suites run in slices on parallel runners (`client-tests`, `server-tests`, through each package's `vitest.shard.config.ts`), and `client-coverage`/`server-coverage` merge their blob reports and check the thresholds against the whole. A red slice is usually a failing test, a red merge job usually a threshold. -- **`ci-ok` is the required check.** The branch ruleset requires `ci-ok` (plus `lint` and `check-target` from their own workflows), and `ci-ok` needs every job above except the scan. It fails if any of them failed or was cancelled and passes if they passed or were skipped. A new gate job counts only once it is listed in `ci-ok`'s `needs:`. +- **`ci-ok` is the required check.** The branch ruleset requires `ci-ok` (plus `lint` and `check-target` from their own workflows), and `ci-ok` needs every job above except the scan. It fails if any of them failed or was cancelled and passes if they passed or were skipped. A new gate job counts only once it is listed in `ci-ok`'s `needs:`, and `scripts/ci/workflow-jobs.test.mjs` (in `test:ci-scripts`) fails while a job other than the scan is missing there. - **Guides name paths that exist**: the `changes` job runs `scripts/ci/doc-paths.mjs`, which resolves every backtick path in the CLAUDE.md files, `README.md`, `server/src/nest/README.md` and the wiki against the tracked files; the missing ones are listed in `scripts/ci/doc-paths-baseline.json`, which only shrinks (`--update` after a fix). - **What runs is decided by `changes`, not by `on.paths`**: `scripts/ci/changed-areas.mjs` maps the changed files to the `code`, `image` and `deploy` areas (tests: `npm run test:ci-scripts`). A path filter on the workflow would leave `ci-ok` unreported on a docs-only PR and block it. Add a new top-level file the build reads to the right area there. - **Runners and Node**: jobs run on `ubuntu-24.04`, not `ubuntu-latest`, and every `setup-node` reads `.nvmrc`. Third-party actions (anything outside `actions/*`) are pinned by commit SHA with the tag in a comment; Dependabot keeps both current. diff --git a/server/CLAUDE.md b/server/CLAUDE.md index c2e093d94..0d5a31989 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -29,8 +29,8 @@ npm run lint:size # no file in src/ or scripts/ past 1000 lines, none in npm run lint:boundaries # import cycles and cross-domain reach-ins may only shrink (CI gate) npm run lint:dialect # SQLite-only SQL spellings under src/ may only shrink (CI gate) npm run lint:engine-dispatch # isSqlite(/isPostgres(/platform instanceof outside src/db/dialect/ and src/db/engine/ may only shrink (CI gate) -npm run probe:pg # dialect helpers + the repository statements it reaches against Postgres (CI job postgres-probe; needs TREK_PG_PROBE_URL) -npm run test:pg # the Postgres lane (CI job server-postgres-tests; needs TREK_TEST_DB_URL) +npm run probe:pg # dialect helpers + the repository statements it reaches against Postgres (CI job server-postgres; needs TREK_PG_PROBE_URL) +npm run test:pg # the Postgres lane (CI job server-postgres, on 14 and 18; needs TREK_TEST_DB_URL) npm run lint:tx # methods writing more than once outside one transaction may only shrink (CI gate) npm run lint:test-sql # raw better-sqlite3 statements in tests/ may only shrink (CI gate) npm run lint:test-mocks # vi.mock of src/config, src/nest/realtime and src/db in tests/ may only shrink (CI gate) @@ -68,7 +68,7 @@ Single test: `npx vitest run tests/unit/nest/weather.controller.test.ts`, or `np **`lint:dialect`** (`scripts/dialect-lint.mjs`) reads every string and template literal under `src/` (comments do not count) and every `.insertId` read, and counts the SQLite-only spellings per file and rule: `INSERT OR IGNORE`/`REPLACE`, `datetime()`/`julianday()`/`unixepoch()` (also as the bare name handed to `eb.fn`), `strftime`, `json_extract` and its siblings, `GLOB`, `last_insert_rowid`, `AUTOINCREMENT`, a `PRAGMA` statement outside the `src/db/*.ts` lifecycle files and the migrations, and `||` concatenation in SQL. The counts are held to `scripts/dialect-baseline.json`; a new hit fails, and so does an entry above what its file holds now, so the change that removes one runs `npm run lint:dialect -- --update` (it never raises or adds an entry). The dialect layer `src/db/dialect/` is not scanned, and neither are the migrations up to `FROZEN_MIGRATIONS_THROUGH` in the script; a newer migration is held to the DML rules. A migration after the Postgres baseline's cut (`through` in `assets/postgres/baseline.json`, read on every run) is also held to two rules no baseline entry can excuse: `ddl` (a CREATE/ALTER/DROP of a table, index, trigger or view, or a RENAME, outside `engineDdl`; a plain string as `addColumnIfMissing`'s column; a call to the SQLite-only `tableSql` or `createPlaceRegionsFollowPlaceTrigger`) and `sqlite-ddl` (DATETIME, AUTOINCREMENT, INTEGER PRIMARY KEY, WITHOUT ROWID, PRAGMA or a SQLite-only function in an `engineDdl` postgres member). The `sqlite` member is SQLite's own and not scanned. The routine under SQL dialect, Migrations, is what satisfies them. -**`probe:pg`** (`scripts/pg-probe.ts`, CI job `postgres-probe`) measures what of the SQL runs on Postgres. Given an empty database (`TREK_PG_PROBE_URL`, or `--url=`; it refuses one holding tables it did not create) it derives a Postgres schema from the entities with MikroORM's schema generator (timestamps stay TEXT, integers are `bigint`, SQLite-only defaults and `COLLATE NOCASE` get their Postgres spelling, no foreign keys), runs every dialect helper on an in-memory SQLite and on Postgres and checks both against the same expectation (`scripts/pg-probe/helper-cases.ts`; every case must pass), then calls every public method of every class in `src/db/repositories/*.ts` once. The arguments come from the parameter types, read with the TypeScript checker (`pg-probe/arg-samples.ts`), and the calls go through a MikroORM Postgres driver that runs each statement alone in a rolled-back transaction with a statement timeout and records whether Postgres accepted it (`pg-probe/recorder.ts`), so every statement meets empty tables and nothing persists. What it measures is the statements each method reaches with those sample arguments against empty tables, not every statement the code can send. The distinct statements Postgres refuses as SQL or over a value's type (SQLSTATE class 42, 0A or 22: syntax, an unknown function or column, a datatype mismatch, an unsupported feature, a string reaching an integer or date column) are held per `Class.method` to `failing` in `scripts/pg-probe-baseline.json`: a method may fail no more than its entry, and an entry above what it fails now fails too. Class 22 counts because the sampler sends what each parameter declares (a `number` is 1, a `string` named like a date is a date, the `present` flag of a partial update's pair is true), so a value of the wrong type reaching a column is the code's: a parameter typed `string` bound to an integer column compares on SQLite and is refused on Postgres; type it as the column is and convert at the service (`toRowId`). A constraint the sample values break (class 23) or the empty tables is reported in the log and the step summary but not held. The methods whose SQL the probe did not see (it could not build their arguments, they sent no SQL because they threw first or never reach the database, or they timed out) are held to the same file's `uncovered` list: a method outside the list that goes unmeasured fails the run, and so does a listed method that is measured now or gone, so the list only shrinks and a new method the sampler cannot reach does not land unchecked. Fix such a method by giving it parameters the sampler can build or by making it reach its SQL with sample values. The job uploads the measured baseline and a JSON report (`pg-probe-baseline` artifact), so the change that fixes a statement copies the lowered baseline from there or runs `npm run probe:pg -- --update` against a local Postgres (`docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=probe postgres:17`); `--update` never raises or adds an entry in either part. While the baseline's `failing` or `uncovered` is `null` it has never been measured, and the run fails (outside `--update`) until the first measurement from the artifact is committed, so the gate cannot land on `dev` holding nothing. A run that called no repository method fails too, so a broken plan can neither pass nor seed an empty baseline. The unit tests under `tests/unit/scripts/pg-probe/` cover the probe's logic, including every helper case on SQLite. +**`probe:pg`** (`scripts/pg-probe.ts`, CI job `server-postgres`, on Postgres 14 and 18) measures what of the SQL runs on Postgres. Given an empty database (`TREK_PG_PROBE_URL`, or `--url=`; it refuses one holding tables it did not create) it derives a Postgres schema from the entities with MikroORM's schema generator (timestamps stay TEXT, integers are `bigint`, SQLite-only defaults and `COLLATE NOCASE` get their Postgres spelling, no foreign keys), runs every dialect helper on an in-memory SQLite and on Postgres and checks both against the same expectation (`scripts/pg-probe/helper-cases.ts`; every case must pass), then calls every public method of every class in `src/db/repositories/*.ts` once. The arguments come from the parameter types, read with the TypeScript checker (`pg-probe/arg-samples.ts`), and the calls go through a MikroORM Postgres driver that runs each statement alone in a rolled-back transaction with a statement timeout and records whether Postgres accepted it (`pg-probe/recorder.ts`), so every statement meets empty tables and nothing persists. What it measures is the statements each method reaches with those sample arguments against empty tables, not every statement the code can send. The distinct statements Postgres refuses as SQL or over a value's type (SQLSTATE class 42, 0A or 22: syntax, an unknown function or column, a datatype mismatch, an unsupported feature, a string reaching an integer or date column) are held per `Class.method` to `failing` in `scripts/pg-probe-baseline.json`: a method may fail no more than its entry, and an entry above what it fails now fails too. Class 22 counts because the sampler sends what each parameter declares (a `number` is 1, a `string` named like a date is a date, the `present` flag of a partial update's pair is true), so a value of the wrong type reaching a column is the code's: a parameter typed `string` bound to an integer column compares on SQLite and is refused on Postgres; type it as the column is and convert at the service (`toRowId`). A constraint the sample values break (class 23) or the empty tables is reported in the log and the step summary but not held. The methods whose SQL the probe did not see (it could not build their arguments, they sent no SQL because they threw first or never reach the database, or they timed out) are held to the same file's `uncovered` list: a method outside the list that goes unmeasured fails the run, and so does a listed method that is measured now or gone, so the list only shrinks and a new method the sampler cannot reach does not land unchecked. Fix such a method by giving it parameters the sampler can build or by making it reach its SQL with sample values. The job uploads the measured baseline and a JSON report per major (`pg-probe-baseline-14`, `pg-probe-baseline-18` artifacts; the baseline holds for both), so the change that fixes a statement copies the lowered baseline from there or runs `npm run probe:pg -- --update` against a local Postgres (`docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=probe postgres:17`); `--update` never raises or adds an entry in either part. While the baseline's `failing` or `uncovered` is `null` it has never been measured, and the run fails (outside `--update`) until the first measurement from the artifact is committed, so the gate cannot land on `dev` holding nothing. A run that called no repository method fails too, so a broken plan can neither pass nor seed an empty baseline. The unit tests under `tests/unit/scripts/pg-probe/` cover the probe's logic, including every helper case on SQLite. ## Architecture invariants @@ -81,7 +81,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot. It refuses the boot when `schema_version` is not exactly one row holding an integer ≥ 1, or is above the last mapped step (258); a value from 1 to 258 is read as dev's steps of those numbers, so a pre-ORM feature-branch database whose steps differ is not detected (the Tours branch is the one case it cuts back). Recorded `mikro_orm_migrations` names this build does not ship refuse it too (`refuseNewerDatabase`), whether a newer release or a development build wrote them. 4.3.x and earlier have no such check and start on a database 5.x migrated. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--