mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-11-20 02:56:10 +00:00
docs(db): describe the final Postgres, plugin data and deployment behaviour
wiki/Database-Postgres.md, server/CLAUDE.md and the plugin SDK docs now describe what plugin SQL may name on Postgres, the pinned output settings, the shared contract (NUL and transaction control refused on every engine), the move's refusals, the Postgres lane in two slices per major on 14 and 18, and that TREK runs as one instance on either engine. server/.env.example names the version 18 client tools the image ships, and the troubleshooting entry and the Helm values name the full set of roles a restore refuses. A doc-paths baseline entry that no longer applies is removed.
This commit is contained in:
@@ -188,8 +188,8 @@ env:
|
||||
# under existingSecretKey and the chart writes nothing. existingSecret wins over url. The
|
||||
# variable is required when either is set: a missing Secret key stops the pod instead of
|
||||
# starting it on SQLite. The role in the URL owns TREK's database and must not be a superuser
|
||||
# (nor hold pg_execute_server_program or pg_write_server_files): a restore runs the backup's
|
||||
# SQL as that role and refuses one that could reach the database host. The wiki page shows the
|
||||
# (nor a member of one, nor hold a right that reaches the server's files or programs): a
|
||||
# restore runs the backup's SQL as that role and refuses one that could reach the database host. The wiki page shows the
|
||||
# SQL that creates such a role. PostgreSQL 14 to 18 is supported.
|
||||
# url: "postgres://trek:<password>@postgres.example.internal:5432/trek?sslmode=require"
|
||||
database:
|
||||
|
||||
@@ -32,6 +32,7 @@ Because the package ships standalone it cannot import server code, so several fi
|
||||
|---|---|---|
|
||||
| `src/generated/host-facts.ts` (permissions, RPC methods, hooks, event catalog, `KNOWN_ADDONS`, the output contract `PLUGIN_ENTITY_FIELDS` / `PLUGIN_ENTITY_NESTED` / `PLUGIN_METHOD_RESULT`) | `server/src/nest/plugins/protocol/envelope.ts`, `protocol/output-contract.ts` and `server/src/addons.ts` via `server/scripts/gen-plugin-facts.ts` (**machine-written, never hand-edit**) | server `check:plugin-facts` CI gate; the entity interfaces in `src/index.ts` are type-checked against `PLUGIN_ENTITY_FIELDS` |
|
||||
| `src/egress-policy.ts` (pure helpers, kept **byte-identical**) | `server/src/nest/plugins/runtime/egress-policy.ts` | `test/permissions-parity.test.ts` |
|
||||
| `src/plugin-data-contract.ts` (the `ctx.db` contract `dev` and the mock host apply: the caps, the refusal texts, the NUL refusal and the transaction-control refusal, the whole file **byte-identical**) | `server/src/db/engine/plugin-data-contract.ts` | `test/permissions-parity.test.ts` |
|
||||
| `src/manifest.ts` (validation rules incl. the `trek` version range; its patterns, sets, caps and `PLUGIN_API_VERSION` come from `generated/host-facts.ts`, written from `server/src/nest/plugins/protocol/manifest-rules.ts`) | `server/src/nest/plugins/install/manifest.ts` | the verdicts: `test/fixtures/manifest-corpus.ts`, run here by `test/manifest-corpus.test.ts` and through both validators by `server/tests/unit/plugins/manifest-parity.test.ts` (the SDK may only be stricter, case by case) |
|
||||
| `src/index.ts` (the plugin API types) | `server/src/nest/plugins/runtime/plugin-sdk.ts` (the child's copy) | `test/host-types.test-d.ts` (vitest typecheck mode, on inside the monorepo): plain shapes equal, ctx-carrying shapes and every ctx namespace with the same members; `test/host-types-names.test.ts` makes it list every shared name |
|
||||
| `src/cli/checks/*` (registry gates) | TREK-Plugins registry repo (`scripts/validate-entry.mjs`, `check-readme.mjs`) | `test/checks-parity.test.ts` (runs the registry's real script; `TREK_PLUGINS_REPO` points at a checkout) |
|
||||
|
||||
+45
-12
@@ -217,8 +217,12 @@ What changes on Postgres:
|
||||
`varchar(20)`, `int8`, ...), JSON and JSONB (`jsonb_build_object`, `jsonb_set`,
|
||||
`jsonb_path_query`, ...), arrays (`unnest`, `array_agg`, `generate_series`, ...),
|
||||
aggregates and window functions (`count`, `sum`, `string_agg`, `row_number`, `lag`, ...),
|
||||
UUIDs (`gen_random_uuid`), ranges, geometry, network addresses and full text search
|
||||
(`to_tsvector`, `ts_rank`, ...). Any other function of Postgres's catalog is refused
|
||||
UUIDs (`gen_random_uuid`), ranges, geometry, network addresses, and full text search over
|
||||
vectors you build yourself (`array_to_tsvector`, `'a & b'::tsquery`, `ts_rank`, `@@`). The
|
||||
text search functions that take a configuration (`to_tsvector`, `to_tsquery`,
|
||||
`plainto_tsquery`, `websearch_to_tsquery`, `ts_headline`, ...) are refused: Postgres looks
|
||||
the configuration up by its name, which would tell you what else the database holds. Any
|
||||
other function of Postgres's catalog is refused
|
||||
however you spell the call (`f(...)`, `"f"(...)`, the column notation `(x).f`, `t.f`,
|
||||
`$1.f`): the ones that read the catalogs or the server rather than their arguments
|
||||
(`obj_description`, `col_description`, `format_type`, `version`, `current_database`,
|
||||
@@ -232,8 +236,33 @@ What changes on Postgres:
|
||||
call (`INSERT INTO version (...)`, `CREATE TABLE version (...)`, `REFERENCES version (id)`);
|
||||
as the table of `CREATE INDEX ... ON` it cannot, so name it differently. Postgres reads
|
||||
`t.f` as the call `f(t)` when `t` has no column `f`, so a column named like a refused
|
||||
function cannot follow a table's name and a dot either (`name`, `version`, `money`, `oid`,
|
||||
`xid` and `xml` can); write it without the table's name, or name it differently.
|
||||
function cannot follow a table's name and a dot either (`name`, `version`, `money`, `oid`
|
||||
and `xml` can); write it without the table's name, or name it differently.
|
||||
- **No system columns.** Reading `xmin`, `xmax`, `cmin`, `cmax`, `ctid` or `tableoid` is
|
||||
refused, quoted or not, after a table's name or alone: they show the server's transaction
|
||||
ids or where a row is stored, and SQLite has none of them. For optimistic locking keep a
|
||||
version or `updated_at` column of your own. A value of yours may still carry such a name
|
||||
where you give it, after `AS` or in a `WITH` query's or an alias's column list
|
||||
(`SELECT min(lng) AS xmin, max(lng) AS xmax FROM places`), and you read it from the
|
||||
result; the SQL itself cannot refer to it by that name (`ORDER BY xmin` is refused).
|
||||
- **No transaction ids.** The types `xid` and `xid8` (and their arrays) are refused where the
|
||||
word stands for the type: `'3'::xid`, `CAST(x AS xid)`, `xid '3'`, `xid(x)`, a column
|
||||
declared `xid`, and `x.xid`, which Postgres reads as the cast `xid(x)` when `x` has no
|
||||
column `xid`. `age()` of one is how many transactions the server ran. A column of yours
|
||||
may be called `xid` (`CREATE TABLE payments (id bigserial PRIMARY KEY, xid text)`, `SELECT
|
||||
xid FROM payments WHERE xid = $1`); name it without its table, and rename it where you
|
||||
need `t.xid` or `excluded.xid` in an upsert.
|
||||
- **Collations and access methods every server has.** After `COLLATE` only `"C"`,
|
||||
`"POSIX"`, `"default"` and `ucs_basic` are accepted, and after `USING` (or `SET ACCESS
|
||||
METHOD`) only `btree`, `hash`, `gist`, `gin`, `spgist`, `brin` and, for a table, `heap`:
|
||||
which others exist depends on the server's locales, ICU and extensions. Text sorts by the
|
||||
database's collation; `ORDER BY name COLLATE "C"` sorts by code point, as SQLite does.
|
||||
- **Values print the same on every server.** Each call runs with the time zone UTC and with
|
||||
Postgres's built-in defaults for the date style, the interval style, the money, number and
|
||||
month name formats (`lc_monetary`, `lc_numeric`, `lc_time` are `C`), float, bytea and xml
|
||||
output and the text search configuration (`simple`), whatever the server is set to:
|
||||
`now()::text` ends in `+00`, `date_trunc('day', now())` cuts at UTC midnight, and
|
||||
`12.5::money` prints `$12.50`.
|
||||
- **You change your own objects only.** Right before an `ALTER`, `DROP`, `REFRESH`,
|
||||
`TRUNCATE` or `CREATE INDEX ... ON` runs, Postgres resolves the object it names (and the
|
||||
tables a `CREATE`/`ALTER TABLE` ties to with `REFERENCES`, `INHERITS`, `PARTITION OF`,
|
||||
@@ -260,13 +289,17 @@ What changes on Postgres:
|
||||
- **One statement at a time.** Each statement goes to Postgres on its own, over the protocol
|
||||
that runs exactly one: `query` and each `tx` entry take one statement, and an `exec`
|
||||
script is split at its semicolons and sent statement by statement.
|
||||
- **No NUL in text.** Postgres text cannot hold the character U+0000. A bound argument
|
||||
holding one (a string, or a string in an array) and a `migrate` id holding one are refused
|
||||
with an error that names it, and so is SQL text with one anywhere, a string literal
|
||||
included; TREK never drops or changes the character, since `'a\0b'` and `'ab'` are
|
||||
different keys. Bind binary data as a `Buffer` into a `bytea` column, and use another
|
||||
separator for composite keys. On SQLite your arguments are bound exactly as you send
|
||||
them, NUL included, as they always were.
|
||||
- **No NUL in text, on either engine.** Postgres text cannot hold the character U+0000, so
|
||||
TREK refuses it on SQLite as well, and `trek-plugin dev` and the mock host refuse it the
|
||||
same way. A bound argument holding one (a string, or a string in an array or an object)
|
||||
and a `migrate` id holding one are refused with an error that names it, and so is SQL text
|
||||
with one anywhere, a string literal included; TREK never drops or changes the character,
|
||||
since `'a\0b'` and `'ab'` are different keys. Bind binary data as a `Buffer` (a `BLOB` on
|
||||
SQLite, `bytea` on Postgres), and use another separator for composite keys. TREK does not
|
||||
see a NUL your SQL builds itself: on SQLite `char(0)` or a BLOB cast to text still stores
|
||||
one, while Postgres refuses `chr(0)`, and moving such an install to Postgres refuses the
|
||||
row. Keep binary data in a `BLOB`. Rows an older TREK stored in your SQLite file with a
|
||||
NUL are still read as they are.
|
||||
- **Limits**: the same row cap and quota as on SQLite. The 2 s budget is a
|
||||
`statement_timeout` here, so it also stops a single long statement, and an `exec` with
|
||||
bound args is held to it too; `migrate` stays untimed. The quota counts every relation of
|
||||
@@ -360,7 +393,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against
|
||||
| `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` |
|
||||
| `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` |
|
||||
| RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` |
|
||||
| `ctx.db` (your own database) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did, and a transaction the script opened itself is rolled back; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded. On a TREK running on Postgres the same caps hold, and the budget also stops a single statement (see [Your own database on Postgres](#your-own-database-on-postgres)) |
|
||||
| `ctx.db` (your own database) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops before its next statement once the time is up (the statements that ran stay applied, as they always did; a script whose last statement has run counts as done, however long it took). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. Transaction control (`BEGIN`, `START`, `COMMIT`, `END`, `ROLLBACK`, `ABORT`, `SAVEPOINT`, `RELEASE`) is refused in every statement of every call, on both engines and in `dev`, with the error `PLUGIN_DATA_ERRORS.txControl`: each call runs in a transaction of its own, and `tx` makes a batch atomic. A named parameter written directly before a parenthesis (`$a(1)`) is refused too (`PLUGIN_DATA_ERRORS.parameterBeforeParenthesis`): the SQLite under `dev` reads it as one token up to the closing parenthesis, and TREK's engines refuse it anyway. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded. On a TREK running on Postgres the same caps hold, and the budget also stops a single statement (see [Your own database on Postgres](#your-own-database-on-postgres)) |
|
||||
| `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) |
|
||||
| Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes |
|
||||
| Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes |
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
"plugin-sdk/CLAUDE.md :: protocol/output-contract.ts",
|
||||
"plugin-sdk/CLAUDE.md :: scripts/validate-entry.mjs",
|
||||
"server/CLAUDE.md :: db/migrations.ts",
|
||||
"server/CLAUDE.md :: host/plugin-data.service.ts",
|
||||
"server/CLAUDE.md :: host/rpc-host.ts",
|
||||
"server/CLAUDE.md :: pg-probe/arg-samples.ts",
|
||||
"server/CLAUDE.md :: pg-probe/recorder.ts",
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ DEMO_MODE=false # Demo mode: resets data hourly
|
||||
# BACKUP_UPLOAD_LIMIT_MB=500 # Max size (MB) of a backup archive you can upload when restoring. Raise it if your backup exceeds 500 MB. If you sit behind a reverse proxy, raise its upload limit too (e.g. nginx client_max_body_size).
|
||||
|
||||
# TREK_DB_URL=postgres://trek:password@db:5432/trek # Run on a Postgres database instead of the SQLite file in data/. TREK creates its schema in an empty database and refuses one holding tables it did not create. sslmode=disable, require, verify-ca or verify-full (the last three always verify the certificate; sslrootcert= names the CA file). Cannot be combined with TREK_DB_FILE, TREK_DB_JOURNAL_MODE or TREK_DB_SYNCHRONOUS.
|
||||
# TREK_DB_PG_BIN_DIR=/usr/lib/postgresql/17/bin # The directory holding pg_dump, pg_restore and psql, which back up and restore a Postgres database. Unset, TREK looks in /usr/lib/postgresql/<17..14>/bin, /usr/local/bin and /usr/bin (the Docker image installs the version 17 tools, which cannot dump a newer server). Unused on SQLite.
|
||||
# TREK_DB_PG_BIN_DIR=/usr/lib/postgresql/18/bin # The directory holding pg_dump, pg_restore and psql, which back up and restore a Postgres database. Unset, TREK looks in /usr/lib/postgresql/<18..14>/bin, newest first, then /usr/local/bin and /usr/bin (the Docker image installs the version 18 tools, which work with every supported server; tools older than the server are refused). Unused on SQLite.
|
||||
# TREK_DB_POOL_MAX=10 # Most connections TREK opens to the Postgres database (1 to 100). Unused on SQLite. (default: 10)
|
||||
# TREK_DB_JOURNAL_MODE=WAL # SQLite journal mode (DELETE, TRUNCATE, PERSIST, MEMORY, WAL, OFF). WAL is fastest and the right choice on local disk. Set DELETE if the data directory lives on network storage (Azure App Service, an SMB/NFS volume on a NAS) — WAL coordinates writers through a shared-memory file that those filesystems do not implement reliably. The mode is stored inside the database file, so it survives a restart; changing this applies on the next boot.
|
||||
# TREK_DB_PRE_MIGRATE_SNAPSHOT=true # Copy the database to data/pre-migrate-<schema>-<time>.db (a .pgdump made with pg_dump on Postgres) before a startup migrates it, so the previous image can run on the copy again. A copy that cannot be written stops the startup; false migrates without one.
|
||||
|
||||
+4
-3
File diff suppressed because one or more lines are too long
@@ -2,7 +2,7 @@
|
||||
|
||||
By default TREK keeps all data in one SQLite file, `travel.db`, in the data directory. Setting `TREK_DB_URL` runs TREK on a PostgreSQL database instead. Without that variable nothing changes, and every page of this wiki that describes `travel.db` still applies to an install on SQLite.
|
||||
|
||||
> **Status:** Postgres support is new. It has been checked end to end against PostgreSQL 17 in Docker (first boot, sign-in, trips, backup and restore, the copy before a migration, `RESTORE_FROM_BACKUP`). The same checks have been run against a Supabase project over its direct connection; see [Supabase](#supabase) and the [Manual verification checklist](#manual-verification-checklist) below.
|
||||
> **Status:** Postgres support is new. On every pull request, CI runs the server test suites that start the app against PostgreSQL 14 and 18, the oldest and newest supported versions. It has also been checked by hand end to end against PostgreSQL 17 in Docker (first boot, sign-in, trips, backup and restore, the copy before a migration, `RESTORE_FROM_BACKUP`). The same checks have been run against a Supabase project over its direct connection; see [Supabase](#supabase) and the [Manual verification checklist](#manual-verification-checklist) below.
|
||||
|
||||
## When to use it
|
||||
|
||||
@@ -128,7 +128,9 @@ TREK ships a command that copies a SQLite install into an empty Postgres databas
|
||||
|
||||
It prints a summary: the tables and rows it copied, the plugin schemas it filled, every value it changed with `--drop-nul`, and anything it could not carry over.
|
||||
|
||||
**NUL characters in TREK's data.** Postgres text cannot hold the character U+0000 (NUL). TREK no longer stores it on either engine and drops it on its next write, but a `travel.db` from an older version may still hold one. The command changes nothing silently, so by default it refuses such a value and lists it by table, row and column. To drop the character during the move, the way TREK would on its next write, run the command again with `--drop-nul`. The summary then lists every value it changed. If two values in a unique column differed only by a NUL, dropping it makes them equal. The command then stops, names the key and lists the values it changed in that column. Change or remove one of the two rows in the SQLite file, with TREK stopped, and run the command again. In JSON, `--drop-nul` removes only the escaped character (`\u0000`); everything else in the value stays as it was written.
|
||||
**NUL characters in TREK's data.** Postgres text cannot hold the character U+0000 (NUL). TREK no longer stores it on either engine and drops it on its next write, but a `travel.db` from an older version may still hold one. The command changes nothing silently, so by default it refuses such a value and lists it by table, row and column. To drop the character during the move, the way TREK would on its next write, run the command again with `--drop-nul`. The summary then lists every value it changed. If two values in a unique column differed only by a NUL, dropping it makes them equal. The command then stops and names the key and the value the two rows share. It lists the values it changed in the columns that key reads (for a key over an expression such as `lower(email)`, the columns the expression reads) whose text that value holds, so you can check those rows first; a key can also break for a reason that has nothing to do with a NUL (see below). Change or remove one of the two rows in the SQLite file, with TREK stopped, and run the command again. JSON holds a NUL either escaped (`\u0000`) or, in a value written by hand, as the raw character; the command refuses both as a NUL, and `--drop-nul` removes both, the way TREK drops it on its next write. Everything else in the value stays as it was written. A JSON value that is not valid even without its NUL characters is refused as invalid JSON.
|
||||
|
||||
**Values the SQLite file accepted and Postgres does not.** A few values pass SQLite's checks but not Postgres's: two rows that a unique key on Postgres holds equal while the SQLite file kept them apart (an install whose case-insensitive email index is still pending holds two accounts whose addresses differ only in case; SQLite's `lower()` changes only the letters A to Z), a value outside a column's allowed set, or JSON holding an escape of half a UTF-16 surrogate pair (`\ud800` alone). The command refuses the move, names the table and the key or value Postgres reports, and says what to do: correct or remove the row in the SQLite file, with TREK stopped, and run the command again. For a plugin's table, that row is in the plugin's `plugin.db`. A value Postgres refuses by its type once the command has let it through is a gap in the command, not in your data: the command stops, writes nothing, and asks you to report it.
|
||||
|
||||
**Text that is not UTF-8.** SQLite stores the bytes of a text value as it is given them, so a value written by hand with the `sqlite3` tool can hold bytes that are not valid UTF-8. Postgres text cannot hold such bytes, and reading them as text would replace them. The command refuses such a value, in TREK's data and in plugin data, with or without `--drop-nul`, and lists it by table, row and column with its bytes in hex. Correct the value in the SQLite file, with TREK stopped, and run the command again.
|
||||
|
||||
@@ -202,14 +204,16 @@ A plugin with the `db:own` permission keeps its own database. On SQLite that is
|
||||
- **TREK's role creates the schemas**, so it needs the `CREATE` privilege on the database, which the role that owns the database has.
|
||||
- **No plugin SQL on a privileged role.** While `TREK_DB_URL` connects as a role TREK refuses restores for (a superuser or a member of one, or a role that may reach the server's files or programs; see [A role that is not a superuser](#a-role-that-is-not-a-superuser)), every plugin database call fails with "Plugin databases (db:own) are refused on Postgres ...", and the boot log says so. A plugin's SQL runs as TREK's role, and such a role could change the system catalogs for the whole server. Connect TREK as a role that owns its database and is not a superuser, and restart; TREK reads the role once per pool connection.
|
||||
- **A plugin's SQL stays in its schema**, in layers the database enforces as well as TREK's own check:
|
||||
- TREK refuses a statement before it runs when it names another schema, the system catalogs, a function that could reach past the schema, or a type that looks an object up by name (`regclass`, `regtype`, `regconfig` and the other `reg*` types, `aclitem`, `has_table_privilege` and its siblings). Of PostgreSQL's own functions a plugin may call only those on an allowlist, the ones that work on their arguments alone (string, number, date and time, JSON, array, aggregate, window, UUID, range, geometry, network address and full text search functions). Every other one is refused, so a function that reads the catalogs or the server (`obj_description`, `format_type`, `version()`, `current_database()`, `inet_server_addr()`, ...) cannot reveal the names or comments of TREK's objects, and neither can a function a later PostgreSQL adds; the keywords `current_user`, `session_user` and their siblings are refused too. It also refuses temporary objects (`CREATE TEMP`/`TEMPORARY` and `SELECT ... INTO TEMP`; a column named `temp` is fine), `INTO` anywhere but after the `INSERT` or `MERGE` that is the statement's verb, hexadecimal, octal and binary numbers, and control characters or non-ASCII whitespace outside strings, which Postgres reads differently from the check.
|
||||
- TREK refuses a statement before it runs when it names another schema, the system catalogs, a function that could reach past the schema, or a type that looks an object up by name (`regclass`, `regtype`, `regconfig` and the other `reg*` types, `aclitem`, `has_table_privilege` and its siblings). Of PostgreSQL's own functions a plugin may call only those on an allowlist, the ones that work on their arguments alone (string, number, date and time, JSON, array, aggregate, window, UUID, range, geometry, network address functions, and full text search over vectors the plugin builds itself). The text search functions that take a configuration name (`to_tsvector`, `to_tsquery`, `ts_headline`, ...) are refused, since the name is looked up in the catalogs. Every other one is refused, so a function that reads the catalogs or the server (`obj_description`, `format_type`, `version()`, `current_database()`, `inet_server_addr()`, ...) cannot reveal the names or comments of TREK's objects, and neither can a function a later PostgreSQL adds; the keywords `current_user`, `session_user` and their siblings are refused too. A read of a system column (`xmin`, `xmax`, `cmin`, `cmax`, `ctid`, `tableoid`) is refused, since they show the server's transaction ids and where rows are stored and SQLite does not have them; a value of the plugin's own may still carry such a name where it is given (`min(lng) AS xmin`). The transaction id types `xid` and `xid8` are refused where the word stands for the type (`'3'::xid`, `xid(x)`, `x.xid`, a column of that type), since `age()` of one shows the server's transaction counter; a column of the plugin's own may be called `xid` and is then named without its table. A collation other than `"C"`, `"POSIX"`, `"default"` and `ucs_basic`, and an access method other than the built-in ones (`btree`, `hash`, `gist`, `gin`, `spgist`, `brin`, `heap`), are refused, since which others exist depends on the server's locales and extensions. It also refuses temporary objects (`CREATE TEMP`/`TEMPORARY` and `SELECT ... INTO TEMP`; a column named `temp` is fine), `INTO` anywhere but after the `INSERT` or `MERGE` that is the statement's verb, hexadecimal, octal and binary numbers, and control characters or non-ASCII whitespace outside strings, which Postgres reads differently from the check.
|
||||
- Right before an `ALTER`, `DROP`, `REFRESH`, `TRUNCATE` or `CREATE INDEX ... ON`, Postgres itself resolves the objects the statement changes, and TREK refuses it unless each one lies in the plugin's schema. An unqualified `int4` or `text` is the built-in type and is refused, whatever the role may do.
|
||||
- Each statement is sent on its own over the extended query protocol, which runs one statement and no more.
|
||||
- Every connection of TREK's pool searches the temporary schema last, and a plugin call drops the temporary tables and other session state of the connection it used before handing it back, so a temporary table can never stand in for one of TREK's.
|
||||
- A call that writes is held to the quota (256 MB, counted over every table, index, sequence and materialized view of the schema with their TOAST storage) and to 5,000 catalog objects (relations, types, constraints, enum labels), so objects that take no disk space are bounded too.
|
||||
|
||||
The rules are in the [plugin-sdk README](https://github.com/liketrek/TREK/blob/main/plugin-sdk/README.md#your-own-database-on-postgres).
|
||||
- **No NUL character in plugin text.** Postgres text cannot hold U+0000. A plugin argument holding one is refused with an error that names the reason, instead of being changed. On SQLite a plugin's own file keeps storing and finding it as before. TREK's own data holds no NUL on either engine.
|
||||
- **Values print the same on every server.** Every plugin transaction sets the time zone to UTC and the date style, the locale for money, numbers and dates, the float, bytea and xml output and the text search configuration to PostgreSQL's built-in defaults, so what a plugin reads does not depend on how the server is configured. The settings end with the transaction. The order of text follows the database's collation; a plugin that wants SQLite's order (by code point) writes `ORDER BY name COLLATE "C"`.
|
||||
- **No NUL character in plugin text, on either engine.** Postgres text cannot hold U+0000. SQL, a plugin argument or a migration id holding one is refused with an error that names the reason, instead of being changed, on SQLite as on Postgres, and `trek-plugin dev` refuses it the same way. SQL that builds a NUL itself (`char(0)`, a BLOB cast to text) still stores one on SQLite, while Postgres refuses to build it, and the move to Postgres refuses such a row; a plugin keeps binary data as a BLOB instead. Rows an older TREK stored in a plugin's SQLite file with a NUL are still read as they are. TREK's own data holds no NUL on either engine.
|
||||
- **No transaction control in plugin SQL, on either engine.** Every plugin call runs in a transaction of its own and `tx()` makes a batch atomic, so `BEGIN`, `START`, `COMMIT`, `END`, `ROLLBACK`, `ABORT`, `SAVEPOINT` and `RELEASE` are refused in every statement of a query, an exec script, a tx batch or a migration, with the same error on SQLite and Postgres and in `trek-plugin dev`.
|
||||
- **A plugin writes Postgres SQL** on a Postgres install, not SQLite SQL. TREK does not translate it. A plugin written for SQLite only fails its `db:own` calls with the database's error; ask its author for Postgres support (`ctx.db.engine()` tells a plugin which engine it runs on).
|
||||
- **Plugin calls share TREK's pool**, at most half of `TREK_DB_POOL_MAX` at once, one at a time per plugin. Each call takes a connection of its own from the pool for its transaction and gives it back when it ends.
|
||||
- **Uninstalling a plugin with "delete data"** drops its schema.
|
||||
|
||||
@@ -696,7 +696,7 @@ See [Notifications](Notifications#web-push) for the details.
|
||||
|
||||
## Postgres: a restore is refused because of the database role
|
||||
|
||||
**Cause:** `TREK_DB_URL` connects as a superuser, or as a role that holds `pg_execute_server_program` or `pg_write_server_files`. A backup is a SQL script that runs as this role, so TREK refuses to restore it with a role that could run commands or write files on the database server. TREK also logs a warning about such a role at startup.
|
||||
**Cause:** `TREK_DB_URL` connects as a role that could reach the database server from SQL: a superuser or a member of one, a role that holds `pg_execute_server_program`, `pg_read_server_files` or `pg_write_server_files` or may run a server file function such as `lo_export`, from PostgreSQL 15 on a role granted ALTER SYSTEM or a superuser-only setting, or on PostgreSQL 14 and 15 a role with CREATEROLE or a member of one. A backup is a SQL script that runs as this role, so TREK refuses to restore it with a role that could run commands or read or write files on the database server. Plugin databases (`db:own`) are refused for the same roles. TREK also logs a warning about such a role at startup.
|
||||
|
||||
**Fix:** Connect TREK as a role that owns its database and is not a superuser. See [Database-Postgres](Database-Postgres#a-role-that-is-not-a-superuser), which also shows how to move an install made with an older `docker-compose.postgres.yml`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user