Commit Graph

26 Commits

Author SHA1 Message Date
Maurice 636fc7d0a6 fix(db): refuse the column notation without parentheses and aclitem in plugin SQL
Postgres reads x.f as the call f(x) when x has no column f. The guard counted
that spelling only after ")." or "].", so over a function in FROM that
returns a single value, or a parameter, the allowlist was bypassed:
"select g.obj_description from generate_series(...) g" returned the comments
on every object of the database, and g.current_schemas, g.txid_status and
$1.ts_token_type passed as well. A refused catalog function after a dot is
now refused too, except the names in COLUMN_NAMES (money, name, oid, version,
xid, xml), which take no argument or only convert it to their type, so t.name
keeps working. PGPLUG-021 holds that list to pg_proc on the lane's server.

The type aclitem names roles: '=r'::aclitem printed the cluster's first
superuser and 'x=r/x'::aclitem told whether a role exists. It is refused in
every spelling like the reg* types.

The allowlist itself moves to assets/postgres/plugin-functions.txt, beside
the catalog list it is checked against. As TypeScript it spelled 'like' and
'json_each', names of Postgres functions that the SQLite dialect gate reads
as SQLite SQL.
2026-11-17 23:30:29 +00:00
Maurice bc6728678b fix(db): let the move to Postgres change nothing silently
The move stripped NUL from a plugin's own text, merging keys that were
distinct, and turned a plugin's values into another type (text in a BLOB
column into bytea, numbers in an untyped column into text). Its checksum
was built from the converted values, so both passed as verified.

A plugin's data now moves exactly as the Postgres plugin store would
hand it back, or not at all: a NUL, a value of another storage class
than its column, or an integer past 2^53 refuses the move before
anything is written, listing plugin, table, row and column. A column
left open by SQLite's affinity takes the kind its values need, so text
in a BLOB column stays text.

TREK's own text holding a NUL from an older install is refused by
default too. --drop-nul drops it, lists every changed value in the
summary, and names a unique key that dropping a NUL broke.

Every table's checksum is now built from the values read from SQLite,
so a conversion that changes a value fails the verification.
2026-11-17 23:28:41 +01:00
Maurice 2edf5f807a fix(db): let a plugin call only an allowlist of Postgres's catalog functions
The plugin SQL guard refused functions by name, so every function that reads
the catalogs or the server and was not listed passed: obj_description,
col_description and shobj_description returned the comments on TREK's
tables, columns, database and roles by OID, format_type named a type by its
OID, and version(), current_database(), inet_server_addr() and the session
keywords described the server.

A call to a pg_catalog function is now refused unless the function is on
PLUGIN_FUNCTIONS (string, number, date and time, JSON, array, aggregate,
window, UUID, range, geometry, network and text search functions), in
either spelling of a call, f(...) and the column notation (x).f. The
catalog's function names of Postgres 14 to 18 are kept in
assets/postgres/catalog-functions.txt, and PGPLUG-021 fails on the lane
when the server has a function that list does not name, so a new major
cannot widen what a plugin may call. The session keywords current_user,
session_user, current_role, user, system_user, current_catalog and
current_schema are refused unquoted.
2026-11-17 23:22:59 +01:00
Maurice f1ea716305 fix(db): hold plugin SQL on Postgres in layers the database enforces
The token guard kept leaking: on a superuser connection an unqualified
`alter type int4 rename to int4x` renamed pg_catalog's type for the whole
server, sequences and catalog-only objects grew the database past the quota,
regtype/regconfig/regdictionary looked TREK's objects up by name, and a
column alias named insert or merge in front of INTO made a table. Instead
of a longer word list, the store now asks the database:

- No plugin SQL while TREK's role is privileged in the sense of the restore
  check (PRIVILEGED_ROLE_SQL, imported from server-checks.ts): every db:own
  call fails with PRIVILEGED_ROLE_PLUGIN_REFUSAL, read once per pool
  connection, and the boot log says why.
- The objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX ... ON
  changes, and the tables a CREATE or ALTER TABLE ties to its own, are
  resolved with to_regclass/to_regtype in the transaction right before the
  statement, and it is refused unless each one found lies in the plugin's
  schema. An unqualified int4, text or box is pg_catalog's and refused even
  with the role check off.
- The quota counts every relation of the schema with its TOAST table, and a
  schema holds at most PLUGIN_SCHEMA_MAX_OBJECTS catalog objects (relations,
  types, constraints, enum labels).
- The guard refuses every reg* type and its array in every spelling, the
  has_*_privilege functions, currtid2 and the BRIN/GIN maintenance
  functions; INTO only after the INSERT or MERGE that is the statement's
  verb or a WITH body's; TEMP only where it makes a temporary object
  (CREATE TEMP/TEMPORARY, SELECT ... INTO TEMP), so a column named temp
  works again; a base type; and a NUL anywhere in the SQL text.
- A bound value or a migration id holding NUL is refused with the reason
  instead of being stripped, so the plugin's keys never change meaning.

Every bypass from the re-review is a test on the real database, on the
role the plugin data is meant to run as and on a superuser connection with
the role check off.
2026-11-17 23:04:12 +01:00
Maurice 9808e113e4 fix(db): refuse restores for CREATEROLE members, server file grants and parameter grants
The privileged-role check missed three ways a role can reach the database
host from SQL. Before PostgreSQL 16 a member of a CREATEROLE role sets its
role to it and grants itself pg_execute_server_program, since role
attributes are not inherited but the grant checks the role in effect. A role
granted EXECUTE on lo_export, lo_import, pg_read_file, pg_ls_dir and their
siblings reads or writes server files, and lo_export keeps the file after the
transaction rolls back. From PostgreSQL 15 on a role can be granted ALTER
SYSTEM on a setting or SET on a superuser-only one.

PRIVILEGED_ROLE_SQL now counts all three. The parameter grants are read
through query_to_xml inside a CASE taken only on 15 or later, because a
statement naming pg_parameter_acl does not parse on 14. A SET grant on a
setting every role may change anyway, and pg_monitor, stay ordinary. The
lane test creates each kind of role and shows the escalation the check is
for on the server it runs against.
2026-11-17 23:04:11 +01:00
Maurice e6f4f7a5f3 fix(db): leave nothing behind when a first Postgres boot fails, so RESTORE_FROM_BACKUP still runs
createFromBaseline created the migration history's table through MikroORM's
storage before, and outside, the transaction that runs the baseline. Without a
master transaction MikroORM commits that table on its own, so a first boot that
failed in the baseline (a server without ICU, a timeout, a dropped connection)
left an empty mikro_orm_migrations behind. The next start with
RESTORE_FROM_BACKUP took that for a database in use, skipped the restore with a
warning and brought up an empty instance, after which the restore never ran.

The table is now created on the baseline's own transaction, so a failed first
boot leaves the database empty. The boot restore also treats a lone migration
history that records nothing as no database, which a restore clears anyway; any
other answer keeps it, so the check fails closed. Database-Postgres says both.
2026-11-17 23:04:11 +01:00
Maurice 77d90be4f8 docs(db): describe plugin SQL on its own pool client instead of the marker hook
The guard's header, server/CLAUDE.md and the Postgres wiki page named
extended-query.ts and a hook on every pool connection. They now say that the
store sends each plugin statement over the extended protocol on a client it
checks out per call, and drops that client's temporary tables and session
state before handing it back.
2026-11-17 23:04:10 +01:00
Maurice a884539a2b fix(db): run plugin SQL on a pool client of its own instead of patching the pool
Plugin statements on Postgres went over the extended query protocol through a
marker parameter and a hook that replaced query() on every client of TREK's
pool. That kept a plugin to one statement per round trip, but it changed
TREK's own hot path for a concern only plugin data has.

The plugin data store now checks a pg client out of the ORM's pool for each
transaction, through MikroORM's getNativeClient, and runs the transaction on
it directly. Every plugin write goes as a query with queryMode 'extended' and
every read through a pg-cursor on that client, so Postgres parses exactly one
statement either way. The settings the transaction makes are local, and
before the client goes back to the pool the store drops temporary tables,
cursors, listeners, session advisory locks and sequence state, or closes a
client it cannot bring back to that state. The row cap, quota, time budget,
untimed migrate, per-plugin queue, pool share and script commits are as
before, and plugin args are still sent without NUL.

The marker, the hook and its install in the Postgres driver are gone.
2026-11-17 23:04:10 +01:00
Maurice f42ef5246c fix(db): store no NUL character on either engine
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
2026-11-17 23:04:08 +01:00
Maurice 2983971c8b fix(db): refuse restores for every role that can reach the database host
The privileged-role check looked only at the role's own rolsuper and at
pg_execute_server_program and pg_write_server_files. A role that is no
superuser itself but a member of one passed it and could SET ROLE to the
superuser from a crafted archive; on PostgreSQL 14 and 15 a role with
CREATEROLE could grant itself pg_execute_server_program; and
pg_read_server_files let COPY read the server's files into TREK's tables.

The check now counts membership in any superuser role, the three
predefined server roles, and CREATEROLE before 16. The boot warning and
the restore refusal describe the same list from one constant. A Postgres
lane suite creates each kind of role and evaluates the check as it, and
shows on the server it runs against whether CREATEROLE can grant itself
pg_execute_server_program.
2026-11-17 23:04:07 +01:00
Maurice e6a0a39a51 fix(db): read plugin SQL the way the Postgres lexer does and refuse INTO and TEMP anywhere
The guard ended a -- comment at LF only, while Postgres ends it at CR as
well, so "select 1 --\rfrom public.users" hid a qualified name from the
guard that Postgres then read, and with no args a second statement after
the CR ran as TREK's role. It also only looked for INTO at the top level
of a SELECT, so "(select ... into temp users)" made a temporary table that
shadowed TREK's own users table on the pooled connection.

The tokenizer now skips exactly what Postgres skips as whitespace, ends a
line comment at CR or LF, and refuses any character the two read
differently (other control characters, non-ASCII whitespace such as
U+00A0, U+FEFF and U+2028, and the non-decimal and underscored numbers
Postgres before 16 reads another way) instead of guessing. INTO is refused
anywhere but after INSERT or MERGE, at any depth, and the words TEMP and
TEMPORARY anywhere outside a string or quoted name.

The guard tests carry both payloads and their variants; the Postgres lane
runs them against the real database, proves a second statement in one
text is refused by Postgres over the extended protocol, and that a
temporary table never shadows TREK's or a plugin's tables.
2026-11-17 23:04:07 +01:00
Maurice 87612c0c79 fix(db): run plugin SQL one statement at a time and search pg_temp last
The plugin SQL guard was the only thing between a plugin and TREK's tables
on Postgres, and two gaps in it were enough to leave the plugin schema.
Two layers now hold even when the guard is fooled.

A plugin write with no bound args went over the simple query protocol,
which runs every statement in its text. Plugin statements now go through
extendedQuery, which pg sends over the extended protocol whatever the
args, so a second statement is a syntax error instead of a statement that
runs. A marker parameter carries the request to the pool's clients, and a
client without the hook binds one value too many, so it fails closed.

pg_temp is searched before the search_path unless the path names it, and
a temporary table lives as long as its pooled connection, so one left
behind would stand in for users, sessions and every other table for the
next caller. Every connection of TREK's pool now puts pg_temp last on the
search_path the role configured, and a plugin transaction sets its schema
with pg_temp last.
2026-11-17 23:04:07 +01:00
Maurice cf9172f9e7 fix(db): type the move's value conversion and client for the server's compiler
Formats the two files the format gate listed.
2026-11-17 23:03:59 +01:00
Maurice 057748c47a fix(db): set no deferred references once the move refused a value
Nothing is inserted after the first refused value, so the self-references held back had no rows to land on, and the move failed on them instead of listing the values (MOVE-006).
2026-11-17 23:03:58 +01:00
Maurice 5316362e9e fix(db): open the move's SQLite source through openDatabase, keep the drop failure as the cause
ESLint refused both: better-sqlite3 is imported only in src/db/connection.ts and its named neighbours, and an error thrown in a catch carries what that catch caught.
2026-11-17 23:03:58 +01:00
Maurice 410d2393e0 fix(db): log the demo baseline refusal through the logger, read the tool versions at once 2026-11-17 23:03:56 +01:00
Maurice 55607ff2f9 feat(db): move a SQLite install into an empty Postgres database
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.
2026-11-17 23:03:54 +01:00
Maurice 1ea08dcf0a fix(db): refuse a Postgres boot that would leave a SQLite install behind
Setting TREK_DB_URL on an existing SQLite install booted a new, empty
instance beside its data. The Postgres bootstrap now refuses an empty
database, or a migrated one without a single account, while the data
directory's travel.db holds accounts, and names the move command. A file
that cannot be read counts as holding them. TREK_DB_IGNORE_SQLITE_FILE=true
starts anyway, with a warning; the file is opened read-only and only when
the database holds no TREK data.
2026-11-17 23:03:54 +01:00
Maurice 7340087353 fix(db): carry the plugin data schemas in Postgres backups and restores
pg_dump now names the trek_plugin_ schemas TREK's role owns beside its own
schema. A restore reads the dump's plugin schemas from its table of
contents, drops the plugin schemas the role owns, creates the dump's and
loads them with TREK's schema, all in the one psql transaction. Schemas of
another role and every other schema of the database are never read or
dropped.

The dump's migration history is read from TREK's mikro_orm_migrations only,
so a plugin table of that name cannot pass for it.
2026-11-17 23:03:53 +01:00
Maurice b144781e9f feat(db): keep a plugin's own data in a Postgres schema of its own
A plugin with db:own got a SQLite WAL file in the data volume whatever the
engine, so a Postgres install still needed a local disk (discussion #386).
This adds the plugin data port (db/engine/plugin-data.ts) and its Postgres
side: one schema per plugin, trek_plugin_<id>, reached through TREK's pool.

Every call runs in a transaction that creates the schema if missing, puts
only it on the search_path and sets statement_timeout to the call's budget.
Reads stream through a cursor and stop at the row cap, a write that grows
the schema past the quota rolls back, and the calls of one plugin run one at
a time on at most half of the pool.

TREK's role may read every table, and a role per plugin would need
CREATEROLE, so plugin-sql-guard.ts checks each statement first: it lexes the
SQL as Postgres does and refuses another schema as a qualifier, the pg_
catalogs and functions, functions that run SQL from a string or reach an
object by a name in one, temporary tables, and every statement but DML and
DDL on tables, indexes, views, sequences and types.
2026-11-17 23:03:53 +01:00
Maurice a26db47cb6 refactor(db): keep the touched files within their size baselines
The users identity guard moves to repositories/_shared/user-identity.ts
(Users.repository.ts re-exports UserIdentityTakenError), the flag text pass
of the entity generator to scripts/generate-entities-flags.ts, and the
Wallet pass query takes the repository's EntityManager. The size baseline
of Users.repository.ts is lowered to what it holds now (1445 to 1421); a
few exports only their own file used are no longer exported.
2026-11-17 23:03:53 +01:00
Maurice c3f6177be1 fix(db): sort NULLs where SQLite sorts them on Postgres
SQLite takes NULL as the smallest value and Postgres as the largest, so any
list ordered by a nullable column (bookings without a time, trips without
dates) came back in another order on Postgres. Instead of every ORDER BY
spelling its NULL position, the Postgres engine renders SQLite's wherever a
statement names none: a Kysely query compiler for this.kysely() statements
and a MikroORM platform for find() and the query builder. An explicit
position is kept.

A new Postgres-lane suite checks the answers the probe cannot see: the NULL
order, a stay linked as '14.0' in the public reads, the traveller owner
flag, the identity errors for a lost race, and the plugin caps under a race
on the pool.
2026-11-17 23:03:52 +01:00
Maurice 633e9bcdc1 docs(db): Postgres 14 to 18, the database role, restore limits and the manual rollback
- Database-Postgres: supported versions, the version 18 client and the tool
  search order, sslrootcert with a client before 16, how to create a role
  that is not a superuser and move an older compose install to one, a
  current schema other than public, and the space a restore needs.
- Updating: the manual rollback clears the whole schema and loads the copy
  in one transaction, as TREK's restore does; dropping only the tables left
  the collation, the trigger functions and the sequences behind. A unit
  test keeps the printed statement equal to CLEAR_SCHEMA_SQL, and the boot's
  rollback hint points at that section.
- Troubleshooting, Install-Docker-Compose, Environment-Variables, README
  and server/CLAUDE.md follow.
2026-11-17 23:03:52 +01:00
Maurice 116ba52fbb fix(db): check the Postgres server and tools before a dump or a restore
- The client tools are looked for newest first, from 18 down: a client
  handles every server up to its own major and refuses a newer one. A dump
  or a restore now refuses a client older than the server up front, naming
  both versions, and the boot refuses a server outside PostgreSQL 14 to 18.
- sslrootcert=system is used only by a client from libpq 16 on; an older
  client is refused with a message that names sslrootcert and the bin dir.
- The restore's CVE-2025-8714 table covers 18.
- A restore refuses a role that is a superuser or may run programs or write
  files on the database host, since the archive's SQL runs as that role;
  the boot warns about such a role.
- pg_restore's script is streamed to its work file and stopped once it
  passes BACKUP_MAX_DECOMPRESSED_MB, so a small dump can no longer expand
  without limit on the data volume.
- A failed tool's error keeps only the summarized stderr; its cause names
  the exit code and signal instead of carrying the process error, whose
  message and stderr quoted row data into the log.
- The migration history is looked for in the current schema, where the
  tables and the backups are, so a role whose search_path is not public
  boots a second time.
2026-11-17 23:03:51 +01:00
Maurice 0f338d5484 refactor(db): move PARITY-PG's schema comparison into the Postgres engine
Which comparator statements change nothing (drop not null on a live primary
key, spelling-only defaults and checks proven by a rolled-back fingerprint, the
hand-ported triggers) is now one module the parity test and the migration
generator share, with unit tests for the sorting and the rollback.
2026-11-17 23:03:49 +01:00
Isaias Tavares 2e3762afe1 feat(db): run TREK on PostgreSQL
TREK runs on PostgreSQL when TREK_DB_URL is set, and on SQLite as before
otherwise: an engine port under src/db/engine, the dialect helpers under
src/db/dialect, a Postgres baseline for the migrations up to the cut,
pg_dump/pg_restore backups, a probe of every repository method against a
real server and a Postgres test lane in CI.
2026-11-17 23:03:33 +01:00