From 636fc7d0a6501ca657342077883aa479ccac0d4d Mon Sep 17 00:00:00 2001 From: Maurice Date: Tue, 17 Nov 2026 23:30:29 +0000 Subject: [PATCH] fix(db): refuse the column notation without parentheses and aclitem in plugin SQL Postgres reads x.f as the call f(x) when x has no column f. The guard counted that spelling only after ")." or "].", so over a function in FROM that returns a single value, or a parameter, the allowlist was bypassed: "select g.obj_description from generate_series(...) g" returned the comments on every object of the database, and g.current_schemas, g.txid_status and $1.ts_token_type passed as well. A refused catalog function after a dot is now refused too, except the names in COLUMN_NAMES (money, name, oid, version, xid, xml), which take no argument or only convert it to their type, so t.name keeps working. PGPLUG-021 holds that list to pg_proc on the lane's server. The type aclitem names roles: '=r'::aclitem printed the cluster's first superuser and 'x=r/x'::aclitem told whether a role exists. It is refused in every spelling like the reg* types. The allowlist itself moves to assets/postgres/plugin-functions.txt, beside the catalog list it is checked against. As TypeScript it spelled 'like' and 'json_each', names of Postgres functions that the SQLite dialect gate reads as SQLite SQL. --- plugin-sdk/README.md | 17 +- server/CLAUDE.md | 2 +- server/assets/postgres/plugin-functions.txt | 424 +++++++++++++++ .../engine/postgres/plugin-sql-functions.ts | 491 +++--------------- .../db/engine/postgres/plugin-sql-guard.ts | 38 +- server/tests/postgres/plugin-data.test.ts | 27 +- .../postgres/plugin-sql-functions.test.ts | 30 +- .../engine/postgres/plugin-sql-guard.test.ts | 6 + wiki/Database-Postgres.md | 2 +- 9 files changed, 588 insertions(+), 449 deletions(-) create mode 100644 server/assets/postgres/plugin-functions.txt diff --git a/plugin-sdk/README.md b/plugin-sdk/README.md index 633a32f6c..e7c7ce130 100644 --- a/plugin-sdk/README.md +++ b/plugin-sdk/README.md @@ -219,18 +219,21 @@ What changes on Postgres: aggregates and window functions (`count`, `sum`, `string_agg`, `row_number`, `lag`, ...), UUIDs (`gen_random_uuid`), ranges, geometry, network addresses and full text search (`to_tsvector`, `ts_rank`, ...). Any other function of Postgres's catalog is refused - however you spell the call (`f(...)`, `"f"(...)`, the column notation `(x).f`): the ones - that read the catalogs or the server rather than their arguments (`obj_description`, - `col_description`, `format_type`, `version`, `current_database`, `inet_server_addr`, - `txid_current`, ...), and so is every new function a later Postgres adds until TREK has + however you spell the call (`f(...)`, `"f"(...)`, the column notation `(x).f`, `t.f`, + `$1.f`): the ones that read the catalogs or the server rather than their arguments + (`obj_description`, `col_description`, `format_type`, `version`, `current_database`, + `inet_server_addr`, `txid_current`, ...), and so is every new function a later Postgres adds until TREK has decided about it. The keywords `current_user`, `session_user`, `current_role`, `user`, `system_user`, `current_catalog` and `current_schema` are refused too. `coalesce`, `nullif`, `greatest`, `least`, `CASE` and `CAST` are syntax and always work. The complete - list is `PLUGIN_FUNCTIONS` in TREK's - [`plugin-sql-functions.ts`](https://github.com/liketrek/TREK/blob/main/server/src/db/engine/postgres/plugin-sql-functions.ts). + list is TREK's + [`plugin-functions.txt`](https://github.com/liketrek/TREK/blob/main/server/assets/postgres/plugin-functions.txt). A table or type of yours may still share a refused function's name where it cannot be a call (`INSERT INTO version (...)`, `CREATE TABLE version (...)`, `REFERENCES version (id)`); - as the table of `CREATE INDEX ... ON` it cannot, so name it differently. + as the table of `CREATE INDEX ... ON` it cannot, so name it differently. Postgres reads + `t.f` as the call `f(t)` when `t` has no column `f`, so a column named like a refused + function cannot follow a table's name and a dot either (`name`, `version`, `money`, `oid`, + `xid` and `xml` can); write it without the table's name, or name it differently. - **You change your own objects only.** Right before an `ALTER`, `DROP`, `REFRESH`, `TRUNCATE` or `CREATE INDEX ... ON` runs, Postgres resolves the object it names (and the tables a `CREATE`/`ALTER TABLE` ties to with `REFERENCES`, `INHERITS`, `PARTITION OF`, diff --git a/server/CLAUDE.md b/server/CLAUDE.md index 13f6fb156..72b483652 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -79,7 +79,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; tests hand in a `FakeRealtimeService` (`tests/helpers/fake-realtime.ts`) or spy on the container's instance instead of mocking the transport module. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change. - **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. The demo-mode write block is the registry's `toolGate` (`trekDemoToolGate`): every tool not annotated read-only refuses a demo account there (except the write tools in `DEMO_UNGATED_WRITE_TOOLS`, which never had a demo check), so a tool never checks demo mode itself (ESLint refuses it in `*.mcp.ts`). MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change. - **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire`, one conditional upsert that decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send. -- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). Where that data lives is the engine's: `PLUGIN_DATA` (`src/nest/plugins/host/plugin-data.backend.ts`) binds the SQLite file per plugin, or on Postgres a schema per plugin (`trek_plugin_`, `src/db/engine/postgres/plugin-data-store.ts`) whose SQL is kept inside that schema in layers that `src/db/engine/postgres/plugin-sql-guard.ts`'s header names, the ones after the guard enforced by the database: no plugin SQL at all while TREK's role is privileged in the sense of `PRIVILEGED_ROLE_SQL` (`src/db/engine/postgres/plugin-data-checks.ts`, read once per pool connection, and the boot log says why); the guard, which refuses what it would read differently from the Postgres lexer, other schemas, the catalogs, temporary objects, `SELECT ... INTO`, the `reg*` types, the functions that look an object up by name, and every function of `pg_catalog` outside the allowlist in `src/db/engine/postgres/plugin-sql-functions.ts` (its catalog list, `assets/postgres/catalog-functions.txt`, is held to `pg_proc` by PGPLUG-021); the objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX changes, resolved by Postgres (`to_regclass`/`to_regtype`) right before the statement and refused outside the plugin's schema, so an unqualified `int4` is `pg_catalog`'s and refused even on a superuser connection; one statement per round trip over the extended query protocol, which the store sends on a pool client it checks out for each call and runs its transaction on directly; `pg_temp` last on the plugin's and on every pool connection's `search_path`, `session-search-path.ts`, with the store dropping its client's temporary tables and other session state before the client goes back to the pool; and a quota over every relation of the schema with its TOAST table plus a cap on its catalog objects (`PLUGIN_SCHEMA_MAX_OBJECTS`). A plugin argument holding NUL is refused there, since Postgres text cannot hold it, while the plugin's SQLite file binds it exactly; the plugin's SQL is never translated, and `db.engine` tells the plugin which dialect to write. Backups and restores carry the plugin schemas the role owns (`pg-tools.ts`). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. +- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). Where that data lives is the engine's: `PLUGIN_DATA` (`src/nest/plugins/host/plugin-data.backend.ts`) binds the SQLite file per plugin, or on Postgres a schema per plugin (`trek_plugin_`, `src/db/engine/postgres/plugin-data-store.ts`) whose SQL is kept inside that schema in layers that `src/db/engine/postgres/plugin-sql-guard.ts`'s header names, the ones after the guard enforced by the database: no plugin SQL at all while TREK's role is privileged in the sense of `PRIVILEGED_ROLE_SQL` (`src/db/engine/postgres/plugin-data-checks.ts`, read once per pool connection, and the boot log says why); the guard, which refuses what it would read differently from the Postgres lexer, other schemas, the catalogs, temporary objects, `SELECT ... INTO`, the `reg*` types, the functions that look an object up by name, and every function of `pg_catalog` outside the allowlist `assets/postgres/plugin-functions.txt`, read by `src/db/engine/postgres/plugin-sql-functions.ts` (its catalog list, `assets/postgres/catalog-functions.txt`, is held to `pg_proc` by PGPLUG-021); the objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX changes, resolved by Postgres (`to_regclass`/`to_regtype`) right before the statement and refused outside the plugin's schema, so an unqualified `int4` is `pg_catalog`'s and refused even on a superuser connection; one statement per round trip over the extended query protocol, which the store sends on a pool client it checks out for each call and runs its transaction on directly; `pg_temp` last on the plugin's and on every pool connection's `search_path`, `session-search-path.ts`, with the store dropping its client's temporary tables and other session state before the client goes back to the pool; and a quota over every relation of the schema with its TOAST table plus a cap on its catalog objects (`PLUGIN_SCHEMA_MAX_OBJECTS`). A plugin argument holding NUL is refused there, since Postgres text cannot hold it, while the plugin's SQLite file binds it exactly; the plugin's SQL is never translated, and `db.engine` tells the plugin which dialect to write. Backups and restores carry the plugin schemas the role owns (`pg-tools.ts`). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot. It refuses the boot when `schema_version` is not exactly one row holding an integer ≥ 1, or is above the last mapped step (258); a value from 1 to 258 is read as dev's steps of those numbers, so a pre-ORM feature-branch database whose steps differ is not detected (the Tours branch is the one case it cuts back). Recorded `mikro_orm_migrations` names this build does not ship refuse it too (`refuseNewerDatabase`), whether a newer release or a development build wrote them. 4.3.x and earlier have no such check and start on a database 5.x migrated. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--