From adcda60f7d2a35c3fc3aa754875814a29c05377b Mon Sep 17 00:00:00 2001 From: Maurice Date: Tue, 17 Nov 2026 22:41:57 +0000 Subject: [PATCH] ci(postgres): run the Postgres lane in three slices per major Since the lane runs every suite that boots the app it takes about ten minutes on Postgres 18 and about 25 on 14, past the job's 20 minute timeout. Each major now runs it in three slices (--shard=i/3), each on its own runner and service with its own template; the probe and its artifact stay once per major, in the first slice. ci-ok still needs server-postgres, whose result is failure when any slice fails or is cancelled, so every slice is required. --- .github/workflows/test.yml | 47 +++++++++++++++++++++++--------------- CLAUDE.md | 2 +- server/CLAUDE.md | 2 +- server/vitest.pg.config.ts | 4 ++-- 4 files changed, 32 insertions(+), 23 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9109af968..a8e60f06b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -31,11 +31,11 @@ permissions: # thresholds, uploads lcov # Server: S3 Contract (MinIO) # the storage driver against MinIO -# Server: Postgres 14/18 per supported major: the probe (the repository -# SQL against Postgres), then the lane (every suite -# that boots the app on the real database module, -# plus PARITY-PG and the write lock; -# vitest.pg.config.ts) +# Server: Postgres 14/18 per supported major, in three slices: the probe +# (the repository SQL against Postgres, first slice +# only), then a slice of the lane (every suite that +# boots the app on the real database module, plus +# PARITY-PG and the write lock; vitest.pg.config.ts) # Client: Types & Checks typecheck and the strict ratchet; then the page # pattern and the source ratchets (RTL, size, # Prettier, layers, offline path, duplication, @@ -69,13 +69,16 @@ permissions: # test jobs feeds the scan. # - Job count: GitHub runs at most 20 jobs of the organisation at once and # does not say in which order it starts queued ones. A code change starts -# the 14 shards and about a dozen other jobs; eight of those (parity, the -# SDK, the bundle, S3, the two Postgres majors, shared coverage, the -# chart) are done in a few minutes, so what queues at the start waits about that long. The -# jobs that run for minutes (shards, gates, e2e, the image) must not -# outnumber the 20 slots, or one of them waits for another to finish and -# the run grows by a whole job: a layout with six gate jobs did exactly -# that. Check the count before adding a long job or a shard. +# the 14 shards, the six Postgres slices and about a dozen other jobs; six +# of those (parity, the SDK, the bundle, S3, shared coverage, the chart) +# are done in a few minutes, so what queues at the start waits about that +# long. The jobs that run for minutes (shards, Postgres slices, gates, +# e2e, the image) should not outnumber the 20 slots, or one of them waits +# for another to finish and the run grows by a whole job: a layout with +# six gate jobs did exactly that. The Postgres slices go past that count +# because the lane needs the runner time either way: fewer slices would +# not shorten the run, only push each slice past its timeout. Check the +# count before adding a long job or a shard. # - Scan `needs:` the gates and the four coverage jobs. A coverage job # failing must stop the scan — a missing lcov file would make every line # of that package report as uncovered, and the dashboard would show a @@ -776,12 +779,16 @@ jobs: # without TREK_TEST_DB_URL, and TREK_TEST_PG_MAJOR makes it refuse a service # of another major. It runs after a red probe too, so one run shows both. # - # About four minutes per major: a minute of setup, the probe in about one, - # the lane in about two on four workers. Past one runner's worth, add a - # shard axis to the matrix and pass --shard=/; each slice builds its - # own template. + # The lane runs the whole server suite that boots the app, about ten + # minutes on 18 and well past the job's twenty on 14, so each major runs it + # in three slices (--shard=/3), each on a runner and a service of its + # own; each slice builds its own template in global setup. The probe runs + # once per major, in the first slice. ci-ok needs this job, and a matrix + # job's result is failure when any one slice fails or is cancelled, so + # every slice is required. Once a slice nears the timeout, add one to the + # shard axis and to the --shard divisor together. server-postgres: - name: 'Server: Postgres ${{ matrix.postgres }}' + name: 'Server: Postgres ${{ matrix.postgres }} (${{ matrix.shard }}/3)' runs-on: ubuntu-24.04 timeout-minutes: 20 needs: changes @@ -790,6 +797,7 @@ jobs: fail-fast: false matrix: postgres: [14, 18] + shard: [1, 2, 3] services: postgres: image: postgres:${{ matrix.postgres }} @@ -823,6 +831,7 @@ jobs: run: npm run build --workspace=shared - name: Run the Postgres probe + if: matrix.shard == 1 env: TREK_PG_PROBE_URL: postgres://postgres:lane@127.0.0.1:5432/trek_probe run: >- @@ -831,7 +840,7 @@ jobs: --report="$RUNNER_TEMP/pg-probe-report.json" - name: Upload the measured baseline and the report - if: ${{ !cancelled() }} + if: ${{ !cancelled() && matrix.shard == 1 }} uses: actions/upload-artifact@v6 with: name: pg-probe-baseline-${{ matrix.postgres }} @@ -846,7 +855,7 @@ jobs: env: TREK_TEST_DB_URL: postgres://postgres:lane@127.0.0.1:5432/postgres TREK_TEST_PG_MAJOR: ${{ matrix.postgres }} - run: npm run test:pg --workspace=server + run: npm run test:pg --workspace=server -- --shard=${{ matrix.shard }}/3 # The server suite in seven slices. vitest.shard.config.ts is vitest.config.ts # without the thresholds and the coverage reports: a slice's blob carries diff --git a/CLAUDE.md b/CLAUDE.md index 5f8285b26..dd887f380 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,7 +137,7 @@ These principles come out of a full-repo audit and shape all new code: ## Quality gates in CI -`.github/workflows/test.yml` runs, per package: typecheck (server also `typecheck:tests`, client also the strict-mode ratchet `lint:strict`), lint (client and server through `lint:warnings`, which also holds the warning count per rule), `lint:pages`, `lint:rtl`, `lint:size` (client and server), the server's `lint:strict`, `lint:boundaries`, `lint:tx`, `lint:dialect`, `lint:engine-dispatch`, `lint:test-sql`, `lint:test-mocks`, `lint:test-new-service`, `lint:query-api`, `lint:mcp-zod`, `lint:service-http` and `lint:response-contracts`, the client's and server's `lint:format`, `lint:layers`, `lint:offline`, `lint:dup`, `lint:pairs`, `lint:skips`, `lint:i18n-keys` and `theme:lint`, shared's `contracts:open`, the repo-wide `lint:knip` (in server-lint-checks), `check:plugin-facts`, `i18n:parity:strict` (with the untranslated-strings ratchet), the S3 contract suite, TREK on Postgres (`server-postgres`, once per supported major, 14 and 18: first the Postgres probe, every dialect helper on SQLite and Postgres and the statements every repository method reaches with sample arguments, its SQL errors per method and the methods it cannot measure held to a baseline that only shrinks and fails while it is still unmeasured; then the Postgres lane, every suite that boots the app on the real database module, found by `server/tests/postgres/lane.ts` rather than listed, plus PARITY-PG and the write lock; see `server/CLAUDE.md`), the browser end-to-end flows (Chromium and WebKit), a client build with `check:gl-split`, coverage for all four packages, then a SonarCloud scan. It also builds the Docker image and boots it until `/api/health/ready` answers, on SQLite and on Postgres 18 (`docker-image`), and lints and renders the Helm chart against the Kubernetes schemas and runs `docker compose config` on every compose file (`deploy-manifests`). `lint-prettier.yml` additionally runs `lint` + `lint:format` on `shared/`, both check-only. `plugin-sdk` has no lint script and is outside the eslint gates. **Not in CI**: the screenshot and help-media Playwright projects; run them locally. There are no git hooks; nothing runs before a commit except you. Other workflows build Docker images, scan them, and publish `trek-plugin-sdk` on `plugin-sdk-v*` tags. +`.github/workflows/test.yml` runs, per package: typecheck (server also `typecheck:tests`, client also the strict-mode ratchet `lint:strict`), lint (client and server through `lint:warnings`, which also holds the warning count per rule), `lint:pages`, `lint:rtl`, `lint:size` (client and server), the server's `lint:strict`, `lint:boundaries`, `lint:tx`, `lint:dialect`, `lint:engine-dispatch`, `lint:test-sql`, `lint:test-mocks`, `lint:test-new-service`, `lint:query-api`, `lint:mcp-zod`, `lint:service-http` and `lint:response-contracts`, the client's and server's `lint:format`, `lint:layers`, `lint:offline`, `lint:dup`, `lint:pairs`, `lint:skips`, `lint:i18n-keys` and `theme:lint`, shared's `contracts:open`, the repo-wide `lint:knip` (in server-lint-checks), `check:plugin-facts`, `i18n:parity:strict` (with the untranslated-strings ratchet), the S3 contract suite, TREK on Postgres (`server-postgres`, once per supported major, 14 and 18, each in three slices: first the Postgres probe (in the first slice), every dialect helper on SQLite and Postgres and the statements every repository method reaches with sample arguments, its SQL errors per method and the methods it cannot measure held to a baseline that only shrinks and fails while it is still unmeasured; then the Postgres lane, every suite that boots the app on the real database module, found by `server/tests/postgres/lane.ts` rather than listed, plus PARITY-PG and the write lock; see `server/CLAUDE.md`), the browser end-to-end flows (Chromium and WebKit), a client build with `check:gl-split`, coverage for all four packages, then a SonarCloud scan. It also builds the Docker image and boots it until `/api/health/ready` answers, on SQLite and on Postgres 18 (`docker-image`), and lints and renders the Helm chart against the Kubernetes schemas and runs `docker compose config` on every compose file (`deploy-manifests`). `lint-prettier.yml` additionally runs `lint` + `lint:format` on `shared/`, both check-only. `plugin-sdk` has no lint script and is outside the eslint gates. **Not in CI**: the screenshot and help-media Playwright projects; run them locally. There are no git hooks; nothing runs before a commit except you. Other workflows build Docker images, scan them, and publish `trek-plugin-sdk` on `plugin-sdk-v*` tags. - **Job names read "package: what"** (`Server: Types`, `Client: Tests 3/7`), and the list at the top of `test.yml` says what each job holds. A new job follows the scheme and joins that list, and the job-count note there says when a long job needs a slot freed first. - **Tests run beside the gates, in shards.** The client and server suites run in slices on parallel runners (`client-tests`, `server-tests`, through each package's `vitest.shard.config.ts`), and `client-coverage`/`server-coverage` merge their blob reports and check the thresholds against the whole. A red slice is usually a failing test, a red merge job usually a threshold. diff --git a/server/CLAUDE.md b/server/CLAUDE.md index b03429ce2..13f6fb156 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -82,7 +82,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). Where that data lives is the engine's: `PLUGIN_DATA` (`src/nest/plugins/host/plugin-data.backend.ts`) binds the SQLite file per plugin, or on Postgres a schema per plugin (`trek_plugin_`, `src/db/engine/postgres/plugin-data-store.ts`) whose SQL is kept inside that schema in layers that `src/db/engine/postgres/plugin-sql-guard.ts`'s header names, the ones after the guard enforced by the database: no plugin SQL at all while TREK's role is privileged in the sense of `PRIVILEGED_ROLE_SQL` (`src/db/engine/postgres/plugin-data-checks.ts`, read once per pool connection, and the boot log says why); the guard, which refuses what it would read differently from the Postgres lexer, other schemas, the catalogs, temporary objects, `SELECT ... INTO`, the `reg*` types, the functions that look an object up by name, and every function of `pg_catalog` outside the allowlist in `src/db/engine/postgres/plugin-sql-functions.ts` (its catalog list, `assets/postgres/catalog-functions.txt`, is held to `pg_proc` by PGPLUG-021); the objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX changes, resolved by Postgres (`to_regclass`/`to_regtype`) right before the statement and refused outside the plugin's schema, so an unqualified `int4` is `pg_catalog`'s and refused even on a superuser connection; one statement per round trip over the extended query protocol, which the store sends on a pool client it checks out for each call and runs its transaction on directly; `pg_temp` last on the plugin's and on every pool connection's `search_path`, `session-search-path.ts`, with the store dropping its client's temporary tables and other session state before the client goes back to the pool; and a quota over every relation of the schema with its TOAST table plus a cap on its catalog objects (`PLUGIN_SCHEMA_MAX_OBJECTS`). A plugin argument holding NUL is refused there, since Postgres text cannot hold it, while the plugin's SQLite file binds it exactly; the plugin's SQL is never translated, and `db.engine` tells the plugin which dialect to write. Backups and restores carry the plugin schemas the role owns (`pg-tools.ts`). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot. It refuses the boot when `schema_version` is not exactly one row holding an integer ≥ 1, or is above the last mapped step (258); a value from 1 to 258 is read as dev's steps of those numbers, so a pre-ORM feature-branch database whose steps differ is not detected (the Tours branch is the one case it cuts back). Recorded `mikro_orm_migrations` names this build does not ship refuse it too (`refuseNewerDatabase`), whether a newer release or a development build wrote them. 4.3.x and earlier have no such check and start on a database 5.x migrated. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--