The plugin SQL guard was the only thing between a plugin and TREK's tables
on Postgres, and two gaps in it were enough to leave the plugin schema.
Two layers now hold even when the guard is fooled.
A plugin write with no bound args went over the simple query protocol,
which runs every statement in its text. Plugin statements now go through
extendedQuery, which pg sends over the extended protocol whatever the
args, so a second statement is a syntax error instead of a statement that
runs. A marker parameter carries the request to the pool's clients, and a
client without the hook binds one value too many, so it fails closed.
pg_temp is searched before the search_path unless the path names it, and
a temporary table lives as long as its pooled connection, so one left
behind would stand in for users, sessions and every other table for the
next caller. Every connection of TREK's pool now puts pg_temp last on the
search_path the role configured, and a plugin transaction sets its schema
with pg_temp last.
The CI runner gives a child process a socket for stdout, which /dev/stdout cannot open, so every case whose pg_restore streams its script failed there. The COPY terminators in PGTOOLS-020 are spelled as a dump spells them.
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.
Setting TREK_DB_URL on an existing SQLite install booted a new, empty
instance beside its data. The Postgres bootstrap now refuses an empty
database, or a migrated one without a single account, while the data
directory's travel.db holds accounts, and names the move command. A file
that cannot be read counts as holding them. TREK_DB_IGNORE_SQLITE_FILE=true
starts anyway, with a warning; the file is opened read-only and only when
the database holds no TREK data.
Covers what every call sets up, the schema check before plugin SQL runs, the
row cap, the time budget between chunks and at statement_timeout, the quota,
a script committed statement by statement, the tx and migrate shapes and one
call at a time per plugin, without a database. The Postgres lane runs the
same store on a real one.
pg_dump now names the trek_plugin_ schemas TREK's role owns beside its own
schema. A restore reads the dump's plugin schemas from its table of
contents, drops the plugin schemas the role owns, creates the dump's and
loads them with TREK's schema, all in the one psql transaction. Schemas of
another role and every other schema of the database are never read or
dropped.
The dump's migration history is read from TREK's mikro_orm_migrations only,
so a plugin table of that name cannot pass for it.
A plugin with db:own got a SQLite WAL file in the data volume whatever the
engine, so a Postgres install still needed a local disk (discussion #386).
This adds the plugin data port (db/engine/plugin-data.ts) and its Postgres
side: one schema per plugin, trek_plugin_<id>, reached through TREK's pool.
Every call runs in a transaction that creates the schema if missing, puts
only it on the search_path and sets statement_timeout to the call's budget.
Reads stream through a cursor and stop at the row cap, a write that grows
the schema past the quota rolls back, and the calls of one plugin run one at
a time on at most half of the pool.
TREK's role may read every table, and a role per plugin would need
CREATEROLE, so plugin-sql-guard.ts checks each statement first: it lexes the
SQL as Postgres does and refuses another schema as a qualifier, the pg_
catalogs and functions, functions that run SQL from a string or reach an
object by a name in one, temporary tables, and every statement but DML and
DDL on tables, indexes, views, sequences and types.
SQLite takes NULL as the smallest value and Postgres as the largest, so any
list ordered by a nullable column (bookings without a time, trips without
dates) came back in another order on Postgres. Instead of every ORDER BY
spelling its NULL position, the Postgres engine renders SQLite's wherever a
statement names none: a Kysely query compiler for this.kysely() statements
and a MikroORM platform for find() and the query builder. An explicit
position is kept.
A new Postgres-lane suite checks the answers the probe cannot see: the NULL
order, a stay linked as '14.0' in the public reads, the traveller owner
flag, the identity errors for a lost race, and the plugin caps under a race
on the pool.
- Database-Postgres: supported versions, the version 18 client and the tool
search order, sslrootcert with a client before 16, how to create a role
that is not a superuser and move an older compose install to one, a
current schema other than public, and the space a restore needs.
- Updating: the manual rollback clears the whole schema and loads the copy
in one transaction, as TREK's restore does; dropping only the tables left
the collation, the trigger functions and the sequences behind. A unit
test keeps the printed statement equal to CLEAR_SCHEMA_SQL, and the boot's
rollback hint points at that section.
- Troubleshooting, Install-Docker-Compose, Environment-Variables, README
and server/CLAUDE.md follow.
- The client tools are looked for newest first, from 18 down: a client
handles every server up to its own major and refuses a newer one. A dump
or a restore now refuses a client older than the server up front, naming
both versions, and the boot refuses a server outside PostgreSQL 14 to 18.
- sslrootcert=system is used only by a client from libpq 16 on; an older
client is refused with a message that names sslrootcert and the bin dir.
- The restore's CVE-2025-8714 table covers 18.
- A restore refuses a role that is a superuser or may run programs or write
files on the database host, since the archive's SQL runs as that role;
the boot warns about such a role.
- pg_restore's script is streamed to its work file and stopped once it
passes BACKUP_MAX_DECOMPRESSED_MB, so a small dump can no longer expand
without limit on the data volume.
- A failed tool's error keeps only the summarized stderr; its cause names
the exit code and signal instead of carrying the process error, whose
message and stderr quoted row data into the log.
- The migration history is looked for in the current schema, where the
tables and the backups are, so a role whose search_path is not public
boots a second time.
A frozen baseline still has its manifest verified, and the check now lists the
migrations that run on Postgres after the cut and what holds them. Every
problem it reports says what to do about it.
Which comparator statements change nothing (drop not null on a live primary
key, spelling-only defaults and checks proven by a rolled-back fingerprint, the
hand-ported triggers) is now one module the parity test and the migration
generator share, with unit tests for the sorting and the rollback.
createStandaloneOrm builds the SQLite schema snapshot, so it now refuses a
config that names another driver instead of silently building a snapshot of
the wrong engine. runSchemaBootstrap takes its snapshot flag as a required
argument: the reopen after a restore is the only caller that skips the copy,
and an omitted flag should not be able to pick a default for it.
The Postgres driver's number parsers say which values they refuse: NaN and
the infinities are reported as not finite instead of as beyond 2^53, which
was the wrong reason. The driver's header comment is folded into the constant
docs it described.
The backup, boot-restore and bootstrap tests stopped hand-writing their
stand-in tools and use tests/helpers/fake-pg-tools.ts, so one definition of
what a stand-in logs and answers covers all four files. The assertions about
adapter behaviour are unchanged.
PGBACKUP-002's refusals become it.each, each keeping its own message check:
a missing archive keeps its exact status test (PGBACKUP-002a), and the
accepted dump is PGBACKUP-002. Five refusals are covered in all: missing,
SQLite, newer, no history, and not a dump; the prompt counted six.
boot-restore hands its tables to the psql stand-in directly rather than
through a file the test writes after the plan is built. The bootstrap test
moves its Logger spy into beforeEach.
Four test files hand-wrote the stand-in pg_dump, pg_restore and psql
scripts, each in its own shape, and read their arguments by position ($4,
$6, args[3], slice(0, 5)). A change to what the tools are handed would have
meant finding every one of them.
tests/helpers/fake-pg-tools.ts installs the stand-ins: each logs its call,
and what it does is a named snippet (answersSchema, answersTables,
writesFile, writesRestrictedScript, listsMigrations, failsWith, touches,
copiesFileEndingWith). The output file is found by its --file= argument.
readToolCall and flagValues read a call back by name.
pg-tools.test.ts moves onto it. Argument checks use arrayContaining and
named flags; order is asserted only where it is the subject (the clear
script runs before the load, `--` precedes the dump path). PGTOOLS-007 and
PGTOOLS-009 split their scenarios into it.each, keeping their IDs as
suffixes.
--check migrated a temporary SQLite database twice: once inside the
generation and again to read the triggers for the port check. The generation
now returns the triggers it already read, so a check is one migration.
main is split into writeMode and checkMode, and the frozen-or-not decision
lives in one place, checkBaseline. A baseline whose `through` is the newest
migration must match the entities and port every SQLite trigger; a baseline
frozen behind a newer migration is only checked by its manifest. The old
code made the same decision twice, once for the match and once for the
triggers, and the two could drift apart.
The migrated-database read is split into listing the objects and reading
them, so the temporary file is removed in one place.
Manifest parsing uses a type guard, isBaselineManifest, instead of casts
after a check, and PGBASE-004 pins what it refuses.
schema-adjust.ts mixed the statement splitter, the trigger ports and the
column adjustments, and every caller repeated `meta.props as EntityProperty[]
as ColumnMeta[]` to reach them. Now:
- sql-split.ts holds splitStatements. Its scope is written down: `;` outside
quotes, '/" quoted text and the $$ body, and the session-setting lines it
drops. A tagged dollar quote ($tag$) is refused with an error instead of
being split inside, which the splitter could not do correctly.
- postgres-triggers.ts holds the hand-written trigger ports, which the
baseline and the parity test read by name.
- schema-adjust.ts holds the column adjustments. applyPostgresColumnAdjustments
(renamed from adjustColumns, which hid that it mutates its input) takes an
entity's props directly: ColumnMeta now reads customType as unknown, so
the entity's property metadata satisfies it without a cast. Each adjustment
is a small function that returns whether it changed the column.
The timestamp check is now `instanceof DbTimestampType` rather than a compare
on the constructor's name, which would have broken silently on a rename.
pg-tools.ts held six concerns in 366 lines: tool resolution and process
running, the libpq connection environment, the CVE-2025-8714 version policy,
the CLEAR-SCHEMA SQL and the operations themselves. Each now has its own
module, and pg-tools.ts keeps only the operations the backup and the boot
restore call, so their import paths do not change.
- pg-process.ts: PgTool, resolvePgTool, pgConnectionEnv, PgToolMissingError
and run (stderr is summarized there, as before).
- restore-safety.ts: hasRestrictFix and the refusal. The minimum versions in
the error are built from the RESTRICT_SINCE table, so the message cannot
drift from the policy.
- clear-schema.ts: the SQL that empties the schema before a load, unchanged.
The version floor also no longer keeps its own OLDEST_MAJOR and
LAST_MAJOR_WITHOUT_IT constants: both are read from the table. Behaviour is
unchanged. The literal /dev/null is os.devNull, and the COPY parsing of a
dump's migration history is its own function.
pgRestore read the whole pg_restore --file script into one string to find
TREK's migration table before clearing the schema. The script carries every
COPY row of the dump, so a large database made the read exceed V8's maximum
string length before the check ran, and the restore failed with an
unhelpful error instead of loading or refusing.
The check now streams the script with readline and stops at the first
matching line. The match itself is unchanged: the same CREATE TABLE line for
the same quoted schema, so a dump without TREK's history is still refused
before anything is cleared. The regex's escaping is a named helper.
PGTOOLS-005b pins that the script is never read whole.
pg_dump copied every schema of the database, so on a hosted Postgres such
as Supabase a backup carried its auth and storage schemas too, and a
restore failed on them. A dump now holds the current schema only, a
restore loads only that schema, and a dump without TREK's tables in it is
refused before anything is replaced.
On Postgres the backup port dumps with pg_dump and restores by turning the
dump into a script with pg_restore and having psql clear the current schema
and run it in one transaction. A dump only recreates what it holds, so a
plain pg_restore --clean would leave a later migration's tables behind and
break the migrations the reopen runs; a restore that fails leaves the
database as it was. A restore refuses a pg_restore or psql older than
17.6/16.10/15.14/14.19, whose script would lack the \restrict line that
keeps a psql meta-command smuggled into an uploaded dump from running
(CVE-2025-8714). A restore
refuses a SQLite archive, a dump without TREK's migration history or one a
newer TREK wrote before anything is replaced, and keeps a dump of the
database it replaces.
RESTORE_FROM_BACKUP loads the archive's dump the same way into a database
that holds no table yet, and is ignored with a warning once it holds one.
Before pending migrations the boot now takes a pg_dump copy to
data/pre-migrate-<last migration>-<time>.pgdump under the same
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) policy as the SQLite file, instead of
refusing; a copy that cannot be written refuses the boot.
The tools run by absolute path (TREK_DB_PG_BIN_DIR, or the usual
directories), never through PATH, with the password in PGPASSWORD; an
sslmode that asks for SSL verifies the certificate as the app pool does.
Dumps are written beside their target and renamed into place, readable by
their owner only, and a tool's stderr reaches errors without the lines that
quote SQL or rows. The image installs postgresql-client-17 from Debian 13.
With TREK_DB_URL set, TREK now boots on Postgres; without it nothing changes.
- The database engine contract gains open, close, reopen and the schema
bootstrap; DatabaseLifecycle delegates to the engine resolveEngine() picks.
- An empty Postgres database gets server/assets/postgres/baseline.sql
(generated from the entities by gen:pg-baseline, with the reference rows
the migrations up to its cut insert: tour types, provider catalogues,
addons, app settings) and every migration up to its cut recorded as
executed, in one transaction. A database holding tables
without TREK's history is refused before anything is written; a migrated one
runs the newer-database refusal and its pending migrations, which refuse
without a copy unless TREK_DB_PRE_MIGRATE_SNAPSHOT=false.
- INTEGER columns are bigint on Postgres (SQLite's INTEGER is 64-bit: an
epoch-millisecond column overflowed int4), and the engine's driver reads
bigint and numeric back as numbers, refusing values past 2^53.
- The two SQLite triggers are ported to PL/pgSQL; check:pg-baseline fails
while a SQLite trigger has no port, and while the baseline drifts from the
entities before a later migration freezes it.
- Backups and restores on Postgres refuse until they can dump and load one.
- Migration helpers read the catalog per engine (dialect/catalog.ts), and
the email case index no longer needs group_concat.
- CI runs check:pg-baseline and boots the image on postgres:17 twice.