From cdc31967b2dba6ed2acf5663d4b9f8f828789ad4 Mon Sep 17 00:00:00 2001 From: Maurice Date: Wed, 18 Nov 2026 13:20:10 +0100 Subject: [PATCH] docs(db): describe the final Postgres, plugin data and deployment behaviour wiki/Database-Postgres.md, server/CLAUDE.md and the plugin SDK docs now describe what plugin SQL may name on Postgres, the pinned output settings, the shared contract (NUL and transaction control refused on every engine), the move's refusals, the Postgres lane in two slices per major on 14 and 18, and that TREK runs as one instance on either engine. server/.env.example names the version 18 client tools the image ships, and the troubleshooting entry and the Helm values name the full set of roles a restore refuses. A doc-paths baseline entry that no longer applies is removed. --- charts/trek/values.yaml | 4 +-- plugin-sdk/CLAUDE.md | 1 + plugin-sdk/README.md | 57 +++++++++++++++++++++++------- scripts/ci/doc-paths-baseline.json | 1 - server/.env.example | 2 +- server/CLAUDE.md | 7 ++-- wiki/Database-Postgres.md | 12 ++++--- wiki/Troubleshooting.md | 2 +- 8 files changed, 62 insertions(+), 24 deletions(-) diff --git a/charts/trek/values.yaml b/charts/trek/values.yaml index 8e20fb6d4..cb95d2833 100644 --- a/charts/trek/values.yaml +++ b/charts/trek/values.yaml @@ -188,8 +188,8 @@ env: # under existingSecretKey and the chart writes nothing. existingSecret wins over url. The # variable is required when either is set: a missing Secret key stops the pod instead of # starting it on SQLite. The role in the URL owns TREK's database and must not be a superuser -# (nor hold pg_execute_server_program or pg_write_server_files): a restore runs the backup's -# SQL as that role and refuses one that could reach the database host. The wiki page shows the +# (nor a member of one, nor hold a right that reaches the server's files or programs): a +# restore runs the backup's SQL as that role and refuses one that could reach the database host. The wiki page shows the # SQL that creates such a role. PostgreSQL 14 to 18 is supported. # url: "postgres://trek:@postgres.example.internal:5432/trek?sslmode=require" database: diff --git a/plugin-sdk/CLAUDE.md b/plugin-sdk/CLAUDE.md index 6b23fae48..ef0a8abb3 100644 --- a/plugin-sdk/CLAUDE.md +++ b/plugin-sdk/CLAUDE.md @@ -32,6 +32,7 @@ Because the package ships standalone it cannot import server code, so several fi |---|---|---| | `src/generated/host-facts.ts` (permissions, RPC methods, hooks, event catalog, `KNOWN_ADDONS`, the output contract `PLUGIN_ENTITY_FIELDS` / `PLUGIN_ENTITY_NESTED` / `PLUGIN_METHOD_RESULT`) | `server/src/nest/plugins/protocol/envelope.ts`, `protocol/output-contract.ts` and `server/src/addons.ts` via `server/scripts/gen-plugin-facts.ts` (**machine-written, never hand-edit**) | server `check:plugin-facts` CI gate; the entity interfaces in `src/index.ts` are type-checked against `PLUGIN_ENTITY_FIELDS` | | `src/egress-policy.ts` (pure helpers, kept **byte-identical**) | `server/src/nest/plugins/runtime/egress-policy.ts` | `test/permissions-parity.test.ts` | +| `src/plugin-data-contract.ts` (the `ctx.db` contract `dev` and the mock host apply: the caps, the refusal texts, the NUL refusal and the transaction-control refusal, the whole file **byte-identical**) | `server/src/db/engine/plugin-data-contract.ts` | `test/permissions-parity.test.ts` | | `src/manifest.ts` (validation rules incl. the `trek` version range; its patterns, sets, caps and `PLUGIN_API_VERSION` come from `generated/host-facts.ts`, written from `server/src/nest/plugins/protocol/manifest-rules.ts`) | `server/src/nest/plugins/install/manifest.ts` | the verdicts: `test/fixtures/manifest-corpus.ts`, run here by `test/manifest-corpus.test.ts` and through both validators by `server/tests/unit/plugins/manifest-parity.test.ts` (the SDK may only be stricter, case by case) | | `src/index.ts` (the plugin API types) | `server/src/nest/plugins/runtime/plugin-sdk.ts` (the child's copy) | `test/host-types.test-d.ts` (vitest typecheck mode, on inside the monorepo): plain shapes equal, ctx-carrying shapes and every ctx namespace with the same members; `test/host-types-names.test.ts` makes it list every shared name | | `src/cli/checks/*` (registry gates) | TREK-Plugins registry repo (`scripts/validate-entry.mjs`, `check-readme.mjs`) | `test/checks-parity.test.ts` (runs the registry's real script; `TREK_PLUGINS_REPO` points at a checkout) | diff --git a/plugin-sdk/README.md b/plugin-sdk/README.md index e7c7ce130..a085f37c7 100644 --- a/plugin-sdk/README.md +++ b/plugin-sdk/README.md @@ -217,8 +217,12 @@ What changes on Postgres: `varchar(20)`, `int8`, ...), JSON and JSONB (`jsonb_build_object`, `jsonb_set`, `jsonb_path_query`, ...), arrays (`unnest`, `array_agg`, `generate_series`, ...), aggregates and window functions (`count`, `sum`, `string_agg`, `row_number`, `lag`, ...), - UUIDs (`gen_random_uuid`), ranges, geometry, network addresses and full text search - (`to_tsvector`, `ts_rank`, ...). Any other function of Postgres's catalog is refused + UUIDs (`gen_random_uuid`), ranges, geometry, network addresses, and full text search over + vectors you build yourself (`array_to_tsvector`, `'a & b'::tsquery`, `ts_rank`, `@@`). The + text search functions that take a configuration (`to_tsvector`, `to_tsquery`, + `plainto_tsquery`, `websearch_to_tsquery`, `ts_headline`, ...) are refused: Postgres looks + the configuration up by its name, which would tell you what else the database holds. Any + other function of Postgres's catalog is refused however you spell the call (`f(...)`, `"f"(...)`, the column notation `(x).f`, `t.f`, `$1.f`): the ones that read the catalogs or the server rather than their arguments (`obj_description`, `col_description`, `format_type`, `version`, `current_database`, @@ -232,8 +236,33 @@ What changes on Postgres: call (`INSERT INTO version (...)`, `CREATE TABLE version (...)`, `REFERENCES version (id)`); as the table of `CREATE INDEX ... ON` it cannot, so name it differently. Postgres reads `t.f` as the call `f(t)` when `t` has no column `f`, so a column named like a refused - function cannot follow a table's name and a dot either (`name`, `version`, `money`, `oid`, - `xid` and `xml` can); write it without the table's name, or name it differently. + function cannot follow a table's name and a dot either (`name`, `version`, `money`, `oid` + and `xml` can); write it without the table's name, or name it differently. +- **No system columns.** Reading `xmin`, `xmax`, `cmin`, `cmax`, `ctid` or `tableoid` is + refused, quoted or not, after a table's name or alone: they show the server's transaction + ids or where a row is stored, and SQLite has none of them. For optimistic locking keep a + version or `updated_at` column of your own. A value of yours may still carry such a name + where you give it, after `AS` or in a `WITH` query's or an alias's column list + (`SELECT min(lng) AS xmin, max(lng) AS xmax FROM places`), and you read it from the + result; the SQL itself cannot refer to it by that name (`ORDER BY xmin` is refused). +- **No transaction ids.** The types `xid` and `xid8` (and their arrays) are refused where the + word stands for the type: `'3'::xid`, `CAST(x AS xid)`, `xid '3'`, `xid(x)`, a column + declared `xid`, and `x.xid`, which Postgres reads as the cast `xid(x)` when `x` has no + column `xid`. `age()` of one is how many transactions the server ran. A column of yours + may be called `xid` (`CREATE TABLE payments (id bigserial PRIMARY KEY, xid text)`, `SELECT + xid FROM payments WHERE xid = $1`); name it without its table, and rename it where you + need `t.xid` or `excluded.xid` in an upsert. +- **Collations and access methods every server has.** After `COLLATE` only `"C"`, + `"POSIX"`, `"default"` and `ucs_basic` are accepted, and after `USING` (or `SET ACCESS + METHOD`) only `btree`, `hash`, `gist`, `gin`, `spgist`, `brin` and, for a table, `heap`: + which others exist depends on the server's locales, ICU and extensions. Text sorts by the + database's collation; `ORDER BY name COLLATE "C"` sorts by code point, as SQLite does. +- **Values print the same on every server.** Each call runs with the time zone UTC and with + Postgres's built-in defaults for the date style, the interval style, the money, number and + month name formats (`lc_monetary`, `lc_numeric`, `lc_time` are `C`), float, bytea and xml + output and the text search configuration (`simple`), whatever the server is set to: + `now()::text` ends in `+00`, `date_trunc('day', now())` cuts at UTC midnight, and + `12.5::money` prints `$12.50`. - **You change your own objects only.** Right before an `ALTER`, `DROP`, `REFRESH`, `TRUNCATE` or `CREATE INDEX ... ON` runs, Postgres resolves the object it names (and the tables a `CREATE`/`ALTER TABLE` ties to with `REFERENCES`, `INHERITS`, `PARTITION OF`, @@ -260,13 +289,17 @@ What changes on Postgres: - **One statement at a time.** Each statement goes to Postgres on its own, over the protocol that runs exactly one: `query` and each `tx` entry take one statement, and an `exec` script is split at its semicolons and sent statement by statement. -- **No NUL in text.** Postgres text cannot hold the character U+0000. A bound argument - holding one (a string, or a string in an array) and a `migrate` id holding one are refused - with an error that names it, and so is SQL text with one anywhere, a string literal - included; TREK never drops or changes the character, since `'a\0b'` and `'ab'` are - different keys. Bind binary data as a `Buffer` into a `bytea` column, and use another - separator for composite keys. On SQLite your arguments are bound exactly as you send - them, NUL included, as they always were. +- **No NUL in text, on either engine.** Postgres text cannot hold the character U+0000, so + TREK refuses it on SQLite as well, and `trek-plugin dev` and the mock host refuse it the + same way. A bound argument holding one (a string, or a string in an array or an object) + and a `migrate` id holding one are refused with an error that names it, and so is SQL text + with one anywhere, a string literal included; TREK never drops or changes the character, + since `'a\0b'` and `'ab'` are different keys. Bind binary data as a `Buffer` (a `BLOB` on + SQLite, `bytea` on Postgres), and use another separator for composite keys. TREK does not + see a NUL your SQL builds itself: on SQLite `char(0)` or a BLOB cast to text still stores + one, while Postgres refuses `chr(0)`, and moving such an install to Postgres refuses the + row. Keep binary data in a `BLOB`. Rows an older TREK stored in your SQLite file with a + NUL are still read as they are. - **Limits**: the same row cap and quota as on SQLite. The 2 s budget is a `statement_timeout` here, so it also stops a single long statement, and an `exec` with bound args is held to it too; `migrate` stays untimed. The quota counts every relation of @@ -360,7 +393,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against | `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` | | `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` | | RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` | -| `ctx.db` (your own database) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did, and a transaction the script opened itself is rolled back; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded. On a TREK running on Postgres the same caps hold, and the budget also stops a single statement (see [Your own database on Postgres](#your-own-database-on-postgres)) | +| `ctx.db` (your own database) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. Transaction control (`BEGIN`, `START`, `COMMIT`, `END`, `ROLLBACK`, `ABORT`, `SAVEPOINT`, `RELEASE`) is refused in every statement of every call, on both engines and in `dev`, with the error `PLUGIN_DATA_ERRORS.txControl`: each call runs in a transaction of its own, and `tx` makes a batch atomic. A named parameter written directly before a parenthesis (`$a(1)`) is refused too (`PLUGIN_DATA_ERRORS.parameterBeforeParenthesis`): the SQLite under `dev` reads it as one token up to the closing parenthesis, and TREK's engines refuse it anyway. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded. On a TREK running on Postgres the same caps hold, and the budget also stops a single statement (see [Your own database on Postgres](#your-own-database-on-postgres)) | | `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) | | Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes | | Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes | diff --git a/scripts/ci/doc-paths-baseline.json b/scripts/ci/doc-paths-baseline.json index 98ba3324b..69458ea17 100644 --- a/scripts/ci/doc-paths-baseline.json +++ b/scripts/ci/doc-paths-baseline.json @@ -3,7 +3,6 @@ "plugin-sdk/CLAUDE.md :: protocol/output-contract.ts", "plugin-sdk/CLAUDE.md :: scripts/validate-entry.mjs", "server/CLAUDE.md :: db/migrations.ts", - "server/CLAUDE.md :: host/plugin-data.service.ts", "server/CLAUDE.md :: host/rpc-host.ts", "server/CLAUDE.md :: pg-probe/arg-samples.ts", "server/CLAUDE.md :: pg-probe/recorder.ts", diff --git a/server/.env.example b/server/.env.example index be19be9c2..1264bf13f 100644 --- a/server/.env.example +++ b/server/.env.example @@ -53,7 +53,7 @@ DEMO_MODE=false # Demo mode: resets data hourly # BACKUP_UPLOAD_LIMIT_MB=500 # Max size (MB) of a backup archive you can upload when restoring. Raise it if your backup exceeds 500 MB. If you sit behind a reverse proxy, raise its upload limit too (e.g. nginx client_max_body_size). # TREK_DB_URL=postgres://trek:password@db:5432/trek # Run on a Postgres database instead of the SQLite file in data/. TREK creates its schema in an empty database and refuses one holding tables it did not create. sslmode=disable, require, verify-ca or verify-full (the last three always verify the certificate; sslrootcert= names the CA file). Cannot be combined with TREK_DB_FILE, TREK_DB_JOURNAL_MODE or TREK_DB_SYNCHRONOUS. -# TREK_DB_PG_BIN_DIR=/usr/lib/postgresql/17/bin # The directory holding pg_dump, pg_restore and psql, which back up and restore a Postgres database. Unset, TREK looks in /usr/lib/postgresql/<17..14>/bin, /usr/local/bin and /usr/bin (the Docker image installs the version 17 tools, which cannot dump a newer server). Unused on SQLite. +# TREK_DB_PG_BIN_DIR=/usr/lib/postgresql/18/bin # The directory holding pg_dump, pg_restore and psql, which back up and restore a Postgres database. Unset, TREK looks in /usr/lib/postgresql/<18..14>/bin, newest first, then /usr/local/bin and /usr/bin (the Docker image installs the version 18 tools, which work with every supported server; tools older than the server are refused). Unused on SQLite. # TREK_DB_POOL_MAX=10 # Most connections TREK opens to the Postgres database (1 to 100). Unused on SQLite. (default: 10) # TREK_DB_JOURNAL_MODE=WAL # SQLite journal mode (DELETE, TRUNCATE, PERSIST, MEMORY, WAL, OFF). WAL is fastest and the right choice on local disk. Set DELETE if the data directory lives on network storage (Azure App Service, an SMB/NFS volume on a NAS) — WAL coordinates writers through a shared-memory file that those filesystems do not implement reliably. The mode is stored inside the database file, so it survives a restart; changing this applies on the next boot. # TREK_DB_PRE_MIGRATE_SNAPSHOT=true # Copy the database to data/pre-migrate--