Commit Graph

20 Commits

Author SHA1 Message Date
Maurice 36f64c51c8 feat(db): move a SQLite install into an empty Postgres database
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.
2026-11-17 13:41:20 +01:00
Maurice d177227519 fix(db): refuse a Postgres boot that would leave a SQLite install behind
Setting TREK_DB_URL on an existing SQLite install booted a new, empty
instance beside its data. The Postgres bootstrap now refuses an empty
database, or a migrated one without a single account, while the data
directory's travel.db holds accounts, and names the move command. A file
that cannot be read counts as holding them. TREK_DB_IGNORE_SQLITE_FILE=true
starts anyway, with a warning; the file is opened read-only and only when
the database holds no TREK data.
2026-11-17 13:41:12 +01:00
Maurice 9f7efd49ea test(db): the Postgres plugin data store against a stand-in client
Covers what every call sets up, the schema check before plugin SQL runs, the
row cap, the time budget between chunks and at statement_timeout, the quota,
a script committed statement by statement, the tx and migrate shapes and one
call at a time per plugin, without a database. The Postgres lane runs the
same store on a real one.
2026-11-17 13:02:48 +01:00
Maurice b82cd03a28 fix(db): carry the plugin data schemas in Postgres backups and restores
pg_dump now names the trek_plugin_ schemas TREK's role owns beside its own
schema. A restore reads the dump's plugin schemas from its table of
contents, drops the plugin schemas the role owns, creates the dump's and
loads them with TREK's schema, all in the one psql transaction. Schemas of
another role and every other schema of the database are never read or
dropped.

The dump's migration history is read from TREK's mikro_orm_migrations only,
so a plugin table of that name cannot pass for it.
2026-11-17 13:00:55 +01:00
Maurice b345c92493 feat(db): keep a plugin's own data in a Postgres schema of its own
A plugin with db:own got a SQLite WAL file in the data volume whatever the
engine, so a Postgres install still needed a local disk (discussion #386).
This adds the plugin data port (db/engine/plugin-data.ts) and its Postgres
side: one schema per plugin, trek_plugin_<id>, reached through TREK's pool.

Every call runs in a transaction that creates the schema if missing, puts
only it on the search_path and sets statement_timeout to the call's budget.
Reads stream through a cursor and stop at the row cap, a write that grows
the schema past the quota rolls back, and the calls of one plugin run one at
a time on at most half of the pool.

TREK's role may read every table, and a role per plugin would need
CREATEROLE, so plugin-sql-guard.ts checks each statement first: it lexes the
SQL as Postgres does and refuses another schema as a qualifier, the pg_
catalogs and functions, functions that run SQL from a string or reach an
object by a name in one, temporary tables, and every statement but DML and
DDL on tables, indexes, views, sequences and types.
2026-11-17 13:00:39 +01:00
Maurice 7e8c74b8ff fix(db): sort NULLs where SQLite sorts them on Postgres
SQLite takes NULL as the smallest value and Postgres as the largest, so any
list ordered by a nullable column (bookings without a time, trips without
dates) came back in another order on Postgres. Instead of every ORDER BY
spelling its NULL position, the Postgres engine renders SQLite's wherever a
statement names none: a Kysely query compiler for this.kysely() statements
and a MikroORM platform for find() and the query builder. An explicit
position is kept.

A new Postgres-lane suite checks the answers the probe cannot see: the NULL
order, a stay linked as '14.0' in the public reads, the traveller owner
flag, the identity errors for a lost race, and the plugin caps under a race
on the pool.
2026-11-17 12:24:07 +01:00
Maurice 20fb054d27 docs(db): Postgres 14 to 18, the database role, restore limits and the manual rollback
- Database-Postgres: supported versions, the version 18 client and the tool
  search order, sslrootcert with a client before 16, how to create a role
  that is not a superuser and move an older compose install to one, a
  current schema other than public, and the space a restore needs.
- Updating: the manual rollback clears the whole schema and loads the copy
  in one transaction, as TREK's restore does; dropping only the tables left
  the collation, the trigger functions and the sequences behind. A unit
  test keeps the printed statement equal to CLEAR_SCHEMA_SQL, and the boot's
  rollback hint points at that section.
- Troubleshooting, Install-Docker-Compose, Environment-Variables, README
  and server/CLAUDE.md follow.
2026-11-17 12:24:07 +01:00
Maurice 6f4dfc486f fix(db): check the Postgres server and tools before a dump or a restore
- The client tools are looked for newest first, from 18 down: a client
  handles every server up to its own major and refuses a newer one. A dump
  or a restore now refuses a client older than the server up front, naming
  both versions, and the boot refuses a server outside PostgreSQL 14 to 18.
- sslrootcert=system is used only by a client from libpq 16 on; an older
  client is refused with a message that names sslrootcert and the bin dir.
- The restore's CVE-2025-8714 table covers 18.
- A restore refuses a role that is a superuser or may run programs or write
  files on the database host, since the archive's SQL runs as that role;
  the boot warns about such a role.
- pg_restore's script is streamed to its work file and stopped once it
  passes BACKUP_MAX_DECOMPRESSED_MB, so a small dump can no longer expand
  without limit on the data volume.
- A failed tool's error keeps only the summarized stderr; its cause names
  the exit code and signal instead of carrying the process error, whose
  message and stderr quoted row data into the log.
- The migration history is looked for in the current schema, where the
  tables and the backups are, so a role whose search_path is not public
  boots a second time.
2026-11-17 12:23:51 +01:00
Maurice 53d875aecb fix(db): keep check:pg-baseline speaking once the baseline is frozen
A frozen baseline still has its manifest verified, and the check now lists the
migrations that run on Postgres after the cut and what holds them. Every
problem it reports says what to do about it.
2026-11-17 11:48:18 +01:00
Maurice 8c5e138150 refactor(db): move PARITY-PG's schema comparison into the Postgres engine
Which comparator statements change nothing (drop not null on a live primary
key, spelling-only defaults and checks proven by a rolled-back fingerprint, the
hand-ported triggers) is now one module the parity test and the migration
generator share, with unit tests for the sorting and the rollback.
2026-11-17 11:48:17 +01:00
Isaias Tavares 1c3dc93716 refactor(db): keep the SQLite-only helpers and the Postgres number parsers explicit
createStandaloneOrm builds the SQLite schema snapshot, so it now refuses a
config that names another driver instead of silently building a snapshot of
the wrong engine. runSchemaBootstrap takes its snapshot flag as a required
argument: the reopen after a restore is the only caller that skips the copy,
and an omitted flag should not be able to pick a default for it.

The Postgres driver's number parsers say which values they refuse: NaN and
the infinities are reported as not finite instead of as beyond 2^53, which
was the wrong reason. The driver's header comment is folded into the constant
docs it described.
2026-11-05 01:02:04 -03:00
Isaias Tavares b8c9184dff test(db): move the backup, boot-restore and bootstrap tests onto the fake-tool helper
The backup, boot-restore and bootstrap tests stopped hand-writing their
stand-in tools and use tests/helpers/fake-pg-tools.ts, so one definition of
what a stand-in logs and answers covers all four files. The assertions about
adapter behaviour are unchanged.

PGBACKUP-002's refusals become it.each, each keeping its own message check:
a missing archive keeps its exact status test (PGBACKUP-002a), and the
accepted dump is PGBACKUP-002. Five refusals are covered in all: missing,
SQLite, newer, no history, and not a dump; the prompt counted six.

boot-restore hands its tables to the psql stand-in directly rather than
through a file the test writes after the plan is built. The bootstrap test
moves its Logger spy into beforeEach.
2026-11-05 00:42:11 -03:00
Isaias Tavares f3e6480606 test(db): drive the Postgres tool tests from one fake-tool helper
Four test files hand-wrote the stand-in pg_dump, pg_restore and psql
scripts, each in its own shape, and read their arguments by position ($4,
$6, args[3], slice(0, 5)). A change to what the tools are handed would have
meant finding every one of them.

tests/helpers/fake-pg-tools.ts installs the stand-ins: each logs its call,
and what it does is a named snippet (answersSchema, answersTables,
writesFile, writesRestrictedScript, listsMigrations, failsWith, touches,
copiesFileEndingWith). The output file is found by its --file= argument.
readToolCall and flagValues read a call back by name.

pg-tools.test.ts moves onto it. Argument checks use arrayContaining and
named flags; order is asserted only where it is the subject (the clear
script runs before the load, `--` precedes the dump path). PGTOOLS-007 and
PGTOOLS-009 split their scenarios into it.each, keeping their IDs as
suffixes.
2026-11-05 00:42:11 -03:00
Isaias Tavares 5503db4883 refactor(db): generate the Postgres baseline in one pass, check it in two modes
--check migrated a temporary SQLite database twice: once inside the
generation and again to read the triggers for the port check. The generation
now returns the triggers it already read, so a check is one migration.

main is split into writeMode and checkMode, and the frozen-or-not decision
lives in one place, checkBaseline. A baseline whose `through` is the newest
migration must match the entities and port every SQLite trigger; a baseline
frozen behind a newer migration is only checked by its manifest. The old
code made the same decision twice, once for the match and once for the
triggers, and the two could drift apart.

The migrated-database read is split into listing the objects and reading
them, so the temporary file is removed in one place.

Manifest parsing uses a type guard, isBaselineManifest, instead of casts
after a check, and PGBASE-004 pins what it refuses.
2026-11-05 00:36:42 -03:00
Isaias Tavares 33a18f29dd refactor(db): split the Postgres schema adjustments by concern
schema-adjust.ts mixed the statement splitter, the trigger ports and the
column adjustments, and every caller repeated `meta.props as EntityProperty[]
as ColumnMeta[]` to reach them. Now:

- sql-split.ts holds splitStatements. Its scope is written down: `;` outside
  quotes, '/" quoted text and the $$ body, and the session-setting lines it
  drops. A tagged dollar quote ($tag$) is refused with an error instead of
  being split inside, which the splitter could not do correctly.
- postgres-triggers.ts holds the hand-written trigger ports, which the
  baseline and the parity test read by name.
- schema-adjust.ts holds the column adjustments. applyPostgresColumnAdjustments
  (renamed from adjustColumns, which hid that it mutates its input) takes an
  entity's props directly: ColumnMeta now reads customType as unknown, so
  the entity's property metadata satisfies it without a cast. Each adjustment
  is a small function that returns whether it changed the column.

The timestamp check is now `instanceof DbTimestampType` rather than a compare
on the constructor's name, which would have broken silently on a rename.
2026-11-05 00:32:18 -03:00
Isaias Tavares 1c73ac4c35 refactor(db): split the Postgres client-tool module by concern
pg-tools.ts held six concerns in 366 lines: tool resolution and process
running, the libpq connection environment, the CVE-2025-8714 version policy,
the CLEAR-SCHEMA SQL and the operations themselves. Each now has its own
module, and pg-tools.ts keeps only the operations the backup and the boot
restore call, so their import paths do not change.

- pg-process.ts: PgTool, resolvePgTool, pgConnectionEnv, PgToolMissingError
  and run (stderr is summarized there, as before).
- restore-safety.ts: hasRestrictFix and the refusal. The minimum versions in
  the error are built from the RESTRICT_SINCE table, so the message cannot
  drift from the policy.
- clear-schema.ts: the SQL that empties the schema before a load, unchanged.

The version floor also no longer keeps its own OLDEST_MAJOR and
LAST_MAJOR_WITHOUT_IT constants: both are read from the table. Behaviour is
unchanged. The literal /dev/null is os.devNull, and the COPY parsing of a
dump's migration history is its own function.
2026-11-05 00:20:22 -03:00
Isaias Tavares ae005fc582 fix(db): check the restore script for TREK's tables line by line
pgRestore read the whole pg_restore --file script into one string to find
TREK's migration table before clearing the schema. The script carries every
COPY row of the dump, so a large database made the read exceed V8's maximum
string length before the check ran, and the restore failed with an
unhelpful error instead of loading or refusing.

The check now streams the script with readline and stops at the first
matching line. The match itself is unchanged: the same CREATE TABLE line for
the same quoted schema, so a dump without TREK's history is still refused
before anything is cleared. The regex's escaping is a named helper.

PGTOOLS-005b pins that the script is never read whole.
2026-11-05 00:13:04 -03:00
Isaias Tavares c72b680fec fix(db): keep Postgres backups and restores inside TREK's schema
pg_dump copied every schema of the database, so on a hosted Postgres such
as Supabase a backup carried its auth and storage schemas too, and a
restore failed on them. A dump now holds the current schema only, a
restore loads only that schema, and a dump without TREK's tables in it is
refused before anything is replaced.
2026-11-04 22:45:31 -03:00
Isaias Tavares fc9e634d86 feat(db): back up, restore and copy a Postgres database before migrating it
On Postgres the backup port dumps with pg_dump and restores by turning the
dump into a script with pg_restore and having psql clear the current schema
and run it in one transaction. A dump only recreates what it holds, so a
plain pg_restore --clean would leave a later migration's tables behind and
break the migrations the reopen runs; a restore that fails leaves the
database as it was. A restore refuses a pg_restore or psql older than
17.6/16.10/15.14/14.19, whose script would lack the \restrict line that
keeps a psql meta-command smuggled into an uploaded dump from running
(CVE-2025-8714). A restore
refuses a SQLite archive, a dump without TREK's migration history or one a
newer TREK wrote before anything is replaced, and keeps a dump of the
database it replaces.

RESTORE_FROM_BACKUP loads the archive's dump the same way into a database
that holds no table yet, and is ignored with a warning once it holds one.

Before pending migrations the boot now takes a pg_dump copy to
data/pre-migrate-<last migration>-<time>.pgdump under the same
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) policy as the SQLite file, instead of
refusing; a copy that cannot be written refuses the boot.

The tools run by absolute path (TREK_DB_PG_BIN_DIR, or the usual
directories), never through PATH, with the password in PGPASSWORD; an
sslmode that asks for SSL verifies the certificate as the app pool does.
Dumps are written beside their target and renamed into place, readable by
their owner only, and a tool's stderr reaches errors without the lines that
quote SQL or rows. The image installs postgresql-client-17 from Debian 13.
2026-11-04 22:45:31 -03:00
Isaias Tavares a77705984e feat(db): create and migrate the schema on a Postgres database
With TREK_DB_URL set, TREK now boots on Postgres; without it nothing changes.

- The database engine contract gains open, close, reopen and the schema
  bootstrap; DatabaseLifecycle delegates to the engine resolveEngine() picks.
- An empty Postgres database gets server/assets/postgres/baseline.sql
  (generated from the entities by gen:pg-baseline, with the reference rows
  the migrations up to its cut insert: tour types, provider catalogues,
  addons, app settings) and every migration up to its cut recorded as
  executed, in one transaction. A database holding tables
  without TREK's history is refused before anything is written; a migrated one
  runs the newer-database refusal and its pending migrations, which refuse
  without a copy unless TREK_DB_PRE_MIGRATE_SNAPSHOT=false.
- INTEGER columns are bigint on Postgres (SQLite's INTEGER is 64-bit: an
  epoch-millisecond column overflowed int4), and the engine's driver reads
  bigint and numeric back as numbers, refusing values past 2^53.
- The two SQLite triggers are ported to PL/pgSQL; check:pg-baseline fails
  while a SQLite trigger has no port, and while the baseline drifts from the
  entities before a later migration freezes it.
- Backups and restores on Postgres refuse until they can dump and load one.
- Migration helpers read the catalog per engine (dialect/catalog.ts), and
  the email case index no longer needs group_concat.
- CI runs check:pg-baseline and boots the image on postgres:17 twice.
2026-11-04 22:45:02 -03:00