Commit Graph

1619 Commits

Author SHA1 Message Date
Maurice 1054d79f9e fix(db): pass the chunk size and the abort options through the NUL-free connection
The wrapper took the chunk size of a stream as optional, which lint:strict refuses, and dropped the abort options Kysely hands executeQuery and streamQuery, so a cancelled query kept running.
2026-11-17 21:42:10 +01:00
Maurice 459c4454ad test: import vi in the trip members suite and type the move's SQLite ORM options 2026-11-17 21:42:09 +01:00
Maurice 0460b08a79 test: send the reset and seeding steps one by one or together, not awaited in a loop
resetDatabase empties tables and puts back the default categories one after another through the repo's oneByOne helper, where the order can matter, and the default addons and photo providers together; the MCP suites switch their addons on together and create ordered stops through oneByOne. The warnings this brings below the baseline, and the no-explicit-any and no-unused-vars counts earlier commits lowered, are lowered in scripts/eslint-baseline.json.
2026-11-17 21:42:09 +01:00
Maurice 9327dc309a fix(db): refuse to regenerate a frozen Postgres baseline and check its trigger ports once frozen
gen:pg-baseline rewrote the baseline with its cut at the newest migration on
disk, so after a migration landed after the cut, running it moved the cut over
that migration while Postgres installs created from the old baseline would
still run it through the migrator. It now refuses while a migration follows the
committed cut, names those migrations and the routine, and moves the cut only
with --move-cut (lint:dialect keeps such a migration under the post-cut rules
either way).

check:pg-baseline stopped comparing SQLite triggers with POSTGRES_TRIGGERS once
the baseline was frozen, so a trigger a later migration created on SQLite went
unnoticed. The comparison now runs in both states, with advice that points at
the migration when the baseline is frozen.
2026-11-17 21:39:27 +01:00
Maurice f4a6411511 fix(db): hold SQLite-only functions, LIKE and the sqlite member of a post-cut migration in lint:dialect
lint:dialect missed several classes of SQL that SQLite reads and Postgres does
not, or reads differently:

- group_concat, ifnull, instr, iif, printf, typeof, total(), changes() and
  json_object/json_array/json_valid/json_type/json_quote, as calls in SQL text
  and as the bare name handed to eb.fn.
- LIKE in SQL text, Kysely's 'like' operator and MikroORM's $like, which ignore
  case on SQLite only. A LIKE that is exact on purpose now says so in a
  like-exact comment; the six existing ones (setting keys, storage paths, the
  lowercased reference search, sqlite_master) carry it.
- The sqlite member of engineDdl was not read at all, so a trigger or a data
  backfill put there never reached Postgres. It may now hold only the schema
  change and a rebuild's copy and sqlite_sequence bookkeeping, and SQLite
  statements with an empty postgres member need a sqlite-only reason.
- execBestEffort, bestEffort and execUnlessTableMissing in a post-cut
  migration: on Postgres a swallowed failure aborts the migration transaction.
- The post-cut rules follow POSTGRES_FIRST_CUT as well as the manifest, so
  regenerating the baseline over a new migration no longer releases it.

src/db/engine/sqlite/ is SQLite's own code and is no longer scanned.
2026-11-17 21:36:21 +01:00
Maurice dc75c200b1 fix(db): count every spelling of an engine check in lint:engine-dispatch
The gate matched only isSqlite(/isPostgres(, instanceof on a platform class
and a comparison on a property named engine or id. A destructured engine name,
the literal on the left, a switch over the engine, instanceof on a driver class
and a comparison with a driver class all passed with an empty baseline, and
standalone-orm.ts already held one such check. The regex now matches every one
of them, the guard in standalone-orm.ts moves into the engine port as
assertSqliteOrmConfig, and the advice names where a check belongs.
2026-11-17 21:27:41 +01:00
Maurice 56a1a832b2 test(db): list the carry-over migration among the ones without a legacy step 2026-11-17 21:27:13 +01:00
Maurice 5ff5e76603 fix(db): store no NUL character on either engine
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
2026-11-17 21:24:00 +01:00
Maurice 685506d067 fix(storage): find a stored reference in any case on Postgres too
The release queue narrows each owner's rows with LIKE '%run%' before it
parses them, and relied on SQLite's LIKE ignoring ASCII case. On Postgres
LIKE is exact, so a row naming an object in uppercase was not found and
the release deleted an object that row still names. likeAny now matches
LOWER(column) against the lowercase run, which reads alike on both
engines. RELI-010 fails on Postgres without it.
2026-11-17 21:23:39 +01:00
Maurice 143e2ae447 fix(db): find no nearest day for a date the calendar does not have
The nearest-day fallback of a booking's time cast the bound date to a
timestamp on Postgres, which refuses 2024-02-30 with a 500, while SQLite's
JULIANDAY rolled it over into March and placed the booking there.
dayDistance now refuses anything but a calendar date on both engines
(isCalendarDate), so a caller that forgets the check fails on SQLite too,
and findNearestDayId answers no day for such a date. The size baseline of
Reservations.repository.ts drops by one line.
2026-11-17 21:23:31 +01:00
Maurice e22210aca8 fix(db): refuse a fraction for an integer column at the request boundary
Every integer column is bigint on Postgres. SQLite stored a fraction sent
to one (a packing quantity of 1.5, a budget item for 2.5 days, an id of
3.5) as a REAL, while Postgres rounded it in a statement MikroORM renders
and refused it with a 500 in a Kysely statement, so the engines held
different data for the same request.

The request schemas of those fields now take whole numbers only, and
INTCOL (tests/unit/contracts/integer-columns.test.ts) holds the class: it
reads the integer columns from the entities and fails on any DTO field or
inline MCP tool field of that name that takes any number.
2026-11-17 21:02:25 +01:00
Maurice 8cfd18e6da fix(db): refuse restores for every role that can reach the database host
The privileged-role check looked only at the role's own rolsuper and at
pg_execute_server_program and pg_write_server_files. A role that is no
superuser itself but a member of one passed it and could SET ROLE to the
superuser from a crafted archive; on PostgreSQL 14 and 15 a role with
CREATEROLE could grant itself pg_execute_server_program; and
pg_read_server_files let COPY read the server's files into TREK's tables.

The check now counts membership in any superuser role, the three
predefined server roles, and CREATEROLE before 16. The boot warning and
the restore refusal describe the same list from one constant. A Postgres
lane suite creates each kind of role and evaluates the check as it, and
shows on the server it runs against whether CREATEROLE can grant itself
pg_execute_server_program.
2026-11-17 20:45:53 +01:00
Maurice f19aae6821 fix(db): read plugin SQL the way the Postgres lexer does and refuse INTO and TEMP anywhere
The guard ended a -- comment at LF only, while Postgres ends it at CR as
well, so "select 1 --\rfrom public.users" hid a qualified name from the
guard that Postgres then read, and with no args a second statement after
the CR ran as TREK's role. It also only looked for INTO at the top level
of a SELECT, so "(select ... into temp users)" made a temporary table that
shadowed TREK's own users table on the pooled connection.

The tokenizer now skips exactly what Postgres skips as whitespace, ends a
line comment at CR or LF, and refuses any character the two read
differently (other control characters, non-ASCII whitespace such as
U+00A0, U+FEFF and U+2028, and the non-decimal and underscored numbers
Postgres before 16 reads another way) instead of guessing. INTO is refused
anywhere but after INSERT or MERGE, at any depth, and the words TEMP and
TEMPORARY anywhere outside a string or quoted name.

The guard tests carry both payloads and their variants; the Postgres lane
runs them against the real database, proves a second statement in one
text is refused by Postgres over the extended protocol, and that a
temporary table never shadows TREK's or a plugin's tables.
2026-11-17 20:45:46 +01:00
Maurice 861f6ba789 fix(db): run plugin SQL one statement at a time and search pg_temp last
The plugin SQL guard was the only thing between a plugin and TREK's tables
on Postgres, and two gaps in it were enough to leave the plugin schema.
Two layers now hold even when the guard is fooled.

A plugin write with no bound args went over the simple query protocol,
which runs every statement in its text. Plugin statements now go through
extendedQuery, which pg sends over the extended protocol whatever the
args, so a second statement is a syntax error instead of a statement that
runs. A marker parameter carries the request to the pool's clients, and a
client without the hook binds one value too many, so it fails closed.

pg_temp is searched before the search_path unless the path names it, and
a temporary table lives as long as its pooled connection, so one left
behind would stand in for users, sessions and every other table for the
next caller. Every connection of TREK's pool now puts pg_temp last on the
search_path the role configured, and a plugin transaction sets its schema
with pg_temp last.
2026-11-17 20:45:39 +01:00
Maurice fcddde71a7 test(db): run the notifications and advanced packing MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The tag notification case waits for the fire-and-forget row with
vi.waitFor instead of one turn of the event loop: on SQLite the insert
lands within that turn, on Postgres it needs a few round trips more.
2026-11-17 20:13:29 +01:00
Maurice 4284a90b03 test(db): run the trips and vacay MCP suites and the route inventory on the real database module
The MCP suites seed through the booted container's ORM instead of the
SQLite snapshot mock and reset through resetDatabase. The route inventory
never touched the database and only mocked it; without the mock it
compiles AppModule over the real module, so it runs in the Postgres lane
as well.
2026-11-17 20:06:54 +01:00
Maurice 475597a6d7 test(db): run the transports, trip members and trip warnings MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The trip warnings suite seeded through the factories but only closed
the snapshot handle; it now boots the container to reach the ORM the reset
needs.
2026-11-17 19:56:40 +01:00
Maurice 4f8ae98e7d test(db): run the share, tags, maps and weather, and transit MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The transit suite saves its permission levels through the
container's PermissionsService in a request context instead of one built
by hand over the SQLite handle.
2026-11-17 19:46:27 +01:00
Maurice 65d55add5a test(db): run the reservations, road trip and settings MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The road trip suite saves its permission levels through the
container's PermissionsService in a request context instead of one built
by hand over the SQLite handle.
2026-11-17 19:36:14 +01:00
Maurice 17fff1a152 test(db): run the packing, places and plugin MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The plugin search suite seeded nothing through the snapshot handle
and only closed it; it now boots the container to reach the ORM the reset
needs.
2026-11-17 19:26:01 +01:00
Maurice f795508fcb test(db): run the feeds and invite, journey, leg transport mode, memories and notes MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well.
2026-11-17 19:09:36 +01:00
Maurice 8254ffd23b test(db): run the days, accommodations, files and help MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The collab sub-feature case in the help suite toggles through the
container's AddonsService in a request context instead of one built by
hand over the SQLite handle.
2026-11-17 18:49:07 +01:00
Maurice b6618eac73 test(db): run the budget and collections MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The settlement parity cases call the
container's BudgetController in a request context instead of one built by
hand over the SQLite handle, and the collections suite drops its raw
per-table DELETE for the same reset.
2026-11-17 18:30:12 +01:00
Maurice f13d60e7e9 test(db): run the AirTrail, atlas expanded, collab and categories MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The PermissionsService the AirTrail cases save
through and the CategoriesMcp and DemoService the categories write
harness attaches are the container's own, called in a request context,
instead of instances built by hand over the SQLite handle.

include_coords compares the coordinates as a set: getTravelStats reads
them through a SELECT DISTINCT without an order, and Postgres returns
them in hash order.
2026-11-17 18:30:05 +01:00
Maurice 1b16fd1683 test(db): run the demo tool gate, addon gating, atlas and assignments MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The tour the assignment cases put on a day is
made with makeTour instead of a raw insert.
2026-11-17 18:20:16 +01:00
Maurice 59adaf2f0c test(db): keep the legacy upgrade suites on SQLite by name
They boot on a SQLite file the retired positional runner built, which no
Postgres database ever came from, so they belong in SQLITE_ONLY rather
than in the snapshot baseline. The lane scan now tells such a suite apart
from the snapshot mock: it mocks src/db/database onto a database of its
own, never names createSnapshotTestDb, and is named in SQLITE_ONLY with
its reason. A snapshot suite added to SQLITE_ONLY stays in the baseline.
2026-11-17 17:55:42 +01:00
Maurice 9631a59c2b test(todo): run the todo MCP tool suite on the real database module
Seeds through the booted container's ORM instead of a second ORM over the
SQLite snapshot mock, and resets through resetDatabase, so it runs in the
Postgres lane as well.
2026-11-17 15:36:13 +01:00
Maurice 40385267fc fix(db): keep the stay filter inside src/db and cover the flag type's missing read
src/db may not import src/nest, so the '<id>.0' shape is spelled where the
filter is. The flag type's undefined read had no test, which left the types
folder below its branch threshold.
2026-11-17 15:33:15 +01:00
Maurice 6010694c08 test(db): reset a suite's database between tests on either engine
resetTestDb needs the SQLite snapshot mock's better-sqlite3 handle, so a
suite resetting with it can only run on SQLite. resetDatabase takes the
booted app's ORM and leaves the same state on SQLite and Postgres: every
table but the kept ones emptied (read from the live schema), sequences left
counting, seedDefaults' rows back through the ORM, and on the first call the
users sequence restarted as the snapshot mock normalises it.
2026-11-17 15:22:48 +01:00
Maurice 03ee902c3d fix(db): type the move's value conversion and client for the server's compiler
Formats the two files the format gate listed.
2026-11-17 15:15:48 +01:00
Maurice 6a9c1c287f test(auth): a plugin table missing from the schema fails the user erasure
The erasure no longer skips a missing plugin table in a savepoint of its own (61ea6b648); USER-CLEANUP-005 still expected the skip.
2026-11-17 14:57:24 +01:00
Maurice d253267e49 test(db): the move command's own output and target ORM 2026-11-17 14:49:42 +01:00
Maurice da7d4ff237 feat(notifications): know when the sends a write left running have finished
Most writes start their notification without waiting for it. NotificationsService now tracks each send until it settles, and settled() resolves once every send started before it has finished. The channel check moves to channel-gate.ts, which keeps the service under the size limit.
2026-11-17 14:33:07 +01:00
Maurice f508e0b4c2 test(demo): read the baseline refusal from the logger 2026-11-17 14:33:06 +01:00
Maurice 7e4561b469 test(db): name the restore's plugin schemas as plugin ids allow 2026-11-17 14:24:27 +01:00
Maurice 40e904e237 test(integrations): give the AirTrail suites' UnitOfWork its engine 2026-11-17 14:24:09 +01:00
Maurice c848429820 test(db): no await inside a loop in the Postgres tests 2026-11-17 14:23:13 +01:00
Maurice c46e34dc3a test(db): stand-in pg tools write --file=- to their own stdout
The CI runner gives a child process a socket for stdout, which /dev/stdout cannot open, so every case whose pg_restore streams its script failed there. The COPY terminators in PGTOOLS-020 are spelled as a dump spells them.
2026-11-17 14:23:12 +01:00
Maurice 203c7dee7e feat(lint): refuse a transaction outside UnitOfWork under src/
trek/transactions-through-unit-of-work reads the receiver's type and reports
transactional() on anything but a UnitOfWork (an untyped receiver
included), and begin() or transaction() on an EntityManager, a connection
or Kysely. src/db/engine/ and the two boot-time schema files are left out.
It runs with the rest of the server's ESLint, so lint:warnings fails on a
new one, and its message names UnitOfWork, writeTransaction and the rule in
server/CLAUDE.md.
2026-11-17 14:14:49 +01:00
Maurice 0eeee1e884 fix(db): read the checks a write relies on in the transaction that writes
On Postgres a check read before a transaction, or outside one, no longer
holds at the write that follows: the pool runs other requests' writes in
between. Moved into the unit of work that writes:

- the last-admin guard of an admin's role edit and of an account deletion
  (deleteUserCompletely takes a precondition run in its transaction)
- the login lockout guards of the auth settings save and the SSO settings
  save
- the first-account check of an SSO sign-up, which starts over as an
  ordinary sign-up when another account won
- the OAuth client caps, the token cap and the mail intake limits
- the year check before a vacay year is opened, and the label name check of
  a rename

A password sign-up that loses the last use of an invite is now refused, as
the SSO sign-up already was, and a reset link is burned by one conditional
update, so two resets with one link cannot both change the password.
2026-11-17 14:14:40 +01:00
Maurice 61ea6b648c fix(db): take the write lock in every transaction that writes
UnitOfWork took the engine's write lock, but four repositories opened
transactions of their own: the demo addon set, the plugin user erasure's
savepoints, and the plugin metadata and scheduler cap checks. On Postgres
those ran beside the others on the pool.

writeTransaction (src/db/engine/write-transaction.ts) is now the one
transaction that writes: UnitOfWork wraps it, and the demo seed and reset,
which run outside the container, call it with the engine the environment
names. The capped plugin writes run in their caller's unit of work instead
of a transaction of their own, and the plugin erasure runs in the erasure's
transaction without savepoints, since its three tables come with the
migrations on both engines. lint:tx counts writeTransaction as a
transaction.
2026-11-17 14:14:29 +01:00
Maurice f311fed5c9 test(db): the key rotation on Postgres, and its refusals before a key is asked 2026-11-17 13:54:25 +01:00
Maurice 7065f87dc0 fix(db): rotate the encryption key on Postgres through the server's engine
The rotation ran on the SQLite file only and refused when TREK_DB_URL was
set. Its statements now go through a small store per engine: better-sqlite3
for the file, unchanged, and on Postgres the server's own MikroORM
connection, in one transaction that takes the engine's write lock first. The
Postgres path checks that it reaches a database holding TREK's tables and
that the data directory is writable before it asks for a key, writes a
pg_dump copy before it changes anything, refuses DB_PATH or TREK_DB_FILE
beside the URL, and reads the jsonb addon config the driver hands back
parsed. The SQLite transaction is now BEGIN IMMEDIATE, since better-sqlite3
takes no async function.
2026-11-17 13:54:25 +01:00
Maurice 36f64c51c8 feat(db): move a SQLite install into an empty Postgres database
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.
2026-11-17 13:41:20 +01:00
Maurice d177227519 fix(db): refuse a Postgres boot that would leave a SQLite install behind
Setting TREK_DB_URL on an existing SQLite install booted a new, empty
instance beside its data. The Postgres bootstrap now refuses an empty
database, or a migrated one without a single account, while the data
directory's travel.db holds accounts, and names the move command. A file
that cannot be read counts as holding them. TREK_DB_IGNORE_SQLITE_FILE=true
starts anyway, with a warning; the file is opened read-only and only when
the database holds no TREK data.
2026-11-17 13:41:12 +01:00
Maurice 9f7efd49ea test(db): the Postgres plugin data store against a stand-in client
Covers what every call sets up, the schema check before plugin SQL runs, the
row cap, the time budget between chunks and at statement_timeout, the quota,
a script committed statement by statement, the tx and migrate shapes and one
call at a time per plugin, without a database. The Postgres lane runs the
same store on a real one.
2026-11-17 13:02:48 +01:00
Maurice b82cd03a28 fix(db): carry the plugin data schemas in Postgres backups and restores
pg_dump now names the trek_plugin_ schemas TREK's role owns beside its own
schema. A restore reads the dump's plugin schemas from its table of
contents, drops the plugin schemas the role owns, creates the dump's and
loads them with TREK's schema, all in the one psql transaction. Schemas of
another role and every other schema of the database are never read or
dropped.

The dump's migration history is read from TREK's mikro_orm_migrations only,
so a plugin table of that name cannot pass for it.
2026-11-17 13:00:55 +01:00
Maurice 3460511fa4 feat(plugins): store plugin data on the engine TREK runs on
PLUGIN_DATA binds the plugin data backend to the engine (the same
{ sqlite, postgres }[engine.id] binding the backup port uses): the SQLite
file per plugin, unchanged, or the Postgres schema per plugin. The host
opens a plugin's store through it and an uninstall that deletes data erases
through it, which drops the schema on Postgres. Nothing outside db/engine
asks which engine runs.

A plugin's SQL is passed through as written, so a plugin needs to know the
dialect: db.engine (db:own) answers 'sqlite' or 'postgres', and the SDK
exposes it as ctx.db.engine(), in the mock host (dbEngine option) and in
dev. The plugin facts are regenerated and the SDK goes to 1.10.0.

The Postgres lane's PGPLUG suite installs, uses and erases plugin data on
Postgres and shows that a plugin reaches neither TREK's users table nor
another plugin's schema.
2026-11-17 13:00:47 +01:00
Maurice b345c92493 feat(db): keep a plugin's own data in a Postgres schema of its own
A plugin with db:own got a SQLite WAL file in the data volume whatever the
engine, so a Postgres install still needed a local disk (discussion #386).
This adds the plugin data port (db/engine/plugin-data.ts) and its Postgres
side: one schema per plugin, trek_plugin_<id>, reached through TREK's pool.

Every call runs in a transaction that creates the schema if missing, puts
only it on the search_path and sets statement_timeout to the call's budget.
Reads stream through a cursor and stop at the row cap, a write that grows
the schema past the quota rolls back, and the calls of one plugin run one at
a time on at most half of the pool.

TREK's role may read every table, and a role per plugin would need
CREATEROLE, so plugin-sql-guard.ts checks each statement first: it lexes the
SQL as Postgres does and refuses another schema as a qualifier, the pg_
catalogs and functions, functions that run SQL from a string or reach an
object by a name in one, temporary tables, and every statement but DML and
DDL on tables, indexes, views, sequences and types.
2026-11-17 13:00:39 +01:00
Maurice 7a9a7c1831 refactor(db): keep the touched files within their size baselines
The users identity guard moves to repositories/_shared/user-identity.ts
(Users.repository.ts re-exports UserIdentityTakenError), the flag text pass
of the entity generator to scripts/generate-entities-flags.ts, and the
Wallet pass query takes the repository's EntityManager. The size baseline
of Users.repository.ts is lowered to what it holds now (1445 to 1421); a
few exports only their own file used are no longer exported.
2026-11-17 12:24:08 +01:00