From 17c9c0758fcd3f8f8f61c2f40f41a7db61ebcb85 Mon Sep 17 00:00:00 2001 From: Maurice Date: Tue, 17 Nov 2026 13:00:55 +0100 Subject: [PATCH] docs(db): plugin data on Postgres, its SQL rules and one TREK per database The plugin-sdk README says what dialect a plugin writes when TREK runs on Postgres, what the guard refuses and how the limits hold there. The wiki's Postgres page describes the plugin schemas, their backup and restore and the one-instance-per-database limit, and its key rotation path now names the file where it is. --- plugin-sdk/README.md | 46 +++++++++++++++++++++++++++++++++++++- server/CLAUDE.md | 2 +- wiki/Database-Postgres.md | 17 ++++++++++++-- wiki/Plugin-Development.md | 2 +- wiki/Plugin-Permissions.md | 2 +- 5 files changed, 63 insertions(+), 6 deletions(-) diff --git a/plugin-sdk/README.md b/plugin-sdk/README.md index 902afe240..9dafe148f 100644 --- a/plugin-sdk/README.md +++ b/plugin-sdk/README.md @@ -169,6 +169,50 @@ Every ctx method falls into one of three kinds, listed per method in the exporte answer, an access token) or data you own (`ctx.db`, `ctx.meta`, another plugin's answer), returned as it is. +### Your own database on Postgres + +`ctx.db` is a SQLite file of your own on a default TREK. When the instance runs on +PostgreSQL (`TREK_DB_URL`), it is a schema of your own in TREK's database instead, +named `trek_plugin_` with each dash an underscore, and no SQLite file is +written. TREK hands your SQL to the engine as you wrote it and translates nothing, so +ask which one you are on and write that dialect: + +```js +const pg = (await ctx.db.engine()) === 'postgres' +await ctx.db.migrate('001', pg + ? 'CREATE TABLE notes (id bigserial PRIMARY KEY, body text NOT NULL)' + : 'CREATE TABLE notes (id INTEGER PRIMARY KEY AUTOINCREMENT, body TEXT NOT NULL)') +await ctx.db.exec(pg ? 'INSERT INTO notes (body) VALUES ($1)' : 'INSERT INTO notes (body) VALUES (?)', body) +``` + +What changes on Postgres: + +- **Placeholders** are `$1`, `$2`, ..., not `?`. A `?` with bound args is refused with a + message that says so. +- **Types and functions** are Postgres's: `bigserial`/`GENERATED ... AS IDENTITY` instead + of `AUTOINCREMENT`, `now()` instead of `datetime('now')`, `ON CONFLICT` works on both. + A `bigint` comes back as a number, and one past 2^53 is refused. A boolean comes back as + `true`/`false`, not `1`/`0`. +- **Your SQL stays in your schema.** Only your schema is on the `search_path`, and a + statement is refused before it runs when it names another schema (`public.users`, + another plugin's), anything starting with `pg_` or `information_schema`, or a function + that runs SQL from a string, reads settings or reaches an object by a name in a string + (`query_to_xml`, `ts_stat`, `set_config`, `current_setting`, `nextval`, `setval`, the + `reg*` casts, `lo_*`, `dblink`). An alias that shares a schema's name is refused too, so + pick another. +- **Statements**: `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `WITH`, `VALUES`, + `TABLE`, `TRUNCATE`, `REFRESH`, and `CREATE`/`ALTER`/`DROP` of a table, index, view, + materialized view, sequence or type. No functions, triggers, procedures, `DO` blocks, + temporary tables, `SELECT ... INTO`, `COPY`, `SET`, transaction control (`tx()` makes a + batch atomic), `WITH RECURSIVE`, and no `ALTER` of a schema, owner or tablespace. + Dollar-quoted strings, `E'...'` and `U&'...'` are refused; write plain single-quoted + strings. +- **Limits**: the same row cap and quota as on SQLite. The 2 s budget is a + `statement_timeout` here, so it also stops a single long statement, and an `exec` with + bound args is held to it too; `migrate` stays untimed. A write that takes your schema past + 256 MB and grows it is rolled back; a delete still runs. Rows deleted or rolled back keep + their space until Postgres vacuums them. + ## Settings Declare settings in `trek-plugin.json`; TREK renders the form, you write no settings UI. @@ -252,7 +296,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against | `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` | | `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` | | RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` | -| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did, and a transaction the script opened itself is rolled back; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded | +| `ctx.db` (your own database) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did, and a transaction the script opened itself is rolled back; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded. On a TREK running on Postgres the same caps hold, and the budget also stops a single statement (see [Your own database on Postgres](#your-own-database-on-postgres)) | | `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) | | Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes | | Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes | diff --git a/server/CLAUDE.md b/server/CLAUDE.md index bca0c0143..f7f3837f7 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -78,7 +78,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; tests hand in a `FakeRealtimeService` (`tests/helpers/fake-realtime.ts`) or spy on the container's instance instead of mocking the transport module. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change. - **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. The demo-mode write block is the registry's `toolGate` (`trekDemoToolGate`): every tool not annotated read-only refuses a demo account there (except the write tools in `DEMO_UNGATED_WRITE_TOOLS`, which never had a demo check), so a tool never checks demo mode itself (ESLint refuses it in `*.mcp.ts`). MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change. - **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire`, one conditional upsert that decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send. -- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. +- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). Where that data lives is the engine's: `PLUGIN_DATA` (`src/nest/plugins/host/plugin-data.backend.ts`) binds the SQLite file per plugin, or on Postgres a schema per plugin (`trek_plugin_`, `src/db/engine/postgres/plugin-data-store.ts`) whose SQL `src/db/engine/postgres/plugin-sql-guard.ts` keeps inside that schema before it runs; the plugin's SQL is never translated, and `db.engine` tells the plugin which dialect to write. Backups and restores carry the plugin schemas the role owns (`pg-tools.ts`). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot. It refuses the boot when `schema_version` is not exactly one row holding an integer ≥ 1, or is above the last mapped step (258); a value from 1 to 258 is read as dev's steps of those numbers, so a pre-ORM feature-branch database whose steps differ is not detected (the Tours branch is the one case it cuts back). Recorded `mikro_orm_migrations` names this build does not ship refuse it too (`refuseNewerDatabase`), whether a newer release or a development build wrote them. 4.3.x and earlier have no such check and start on a database 5.x migrated. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--