Commit Graph

55 Commits

Author SHA1 Message Date
Maurice 636fc7d0a6 fix(db): refuse the column notation without parentheses and aclitem in plugin SQL
Postgres reads x.f as the call f(x) when x has no column f. The guard counted
that spelling only after ")." or "].", so over a function in FROM that
returns a single value, or a parameter, the allowlist was bypassed:
"select g.obj_description from generate_series(...) g" returned the comments
on every object of the database, and g.current_schemas, g.txid_status and
$1.ts_token_type passed as well. A refused catalog function after a dot is
now refused too, except the names in COLUMN_NAMES (money, name, oid, version,
xid, xml), which take no argument or only convert it to their type, so t.name
keeps working. PGPLUG-021 holds that list to pg_proc on the lane's server.

The type aclitem names roles: '=r'::aclitem printed the cluster's first
superuser and 'x=r/x'::aclitem told whether a role exists. It is refused in
every spelling like the reg* types.

The allowlist itself moves to assets/postgres/plugin-functions.txt, beside
the catalog list it is checked against. As TypeScript it spelled 'like' and
'json_each', names of Postgres functions that the SQLite dialect gate reads
as SQLite SQL.
2026-11-17 23:30:29 +00:00
Maurice 621b2f17df test(db): run the move's NUL and plugin type rules on a real Postgres
The plugin in the move suite gains a BLOB column holding text and an
untyped column holding numbers, and the plugin store must read both back
as they were stored. A new suite moves a file whose plugin and own rows
hold NUL: the plugin's refuses before anything is written, TREK's
refuses until --drop-nul, a username that dropping the NUL makes
duplicate is named with users_username_unique, and a move without a
collision lists the value it changed and stores it without the NUL.
2026-11-17 23:28:41 +01:00
Maurice 2edf5f807a fix(db): let a plugin call only an allowlist of Postgres's catalog functions
The plugin SQL guard refused functions by name, so every function that reads
the catalogs or the server and was not listed passed: obj_description,
col_description and shobj_description returned the comments on TREK's
tables, columns, database and roles by OID, format_type named a type by its
OID, and version(), current_database(), inet_server_addr() and the session
keywords described the server.

A call to a pg_catalog function is now refused unless the function is on
PLUGIN_FUNCTIONS (string, number, date and time, JSON, array, aggregate,
window, UUID, range, geometry, network and text search functions), in
either spelling of a call, f(...) and the column notation (x).f. The
catalog's function names of Postgres 14 to 18 are kept in
assets/postgres/catalog-functions.txt, and PGPLUG-021 fails on the lane
when the server has a function that list does not name, so a new major
cannot widen what a plugin may call. The session keywords current_user,
session_user, current_role, user, system_user, current_catalog and
current_schema are refused unquoted.
2026-11-17 23:22:59 +01:00
Maurice f1ea716305 fix(db): hold plugin SQL on Postgres in layers the database enforces
The token guard kept leaking: on a superuser connection an unqualified
`alter type int4 rename to int4x` renamed pg_catalog's type for the whole
server, sequences and catalog-only objects grew the database past the quota,
regtype/regconfig/regdictionary looked TREK's objects up by name, and a
column alias named insert or merge in front of INTO made a table. Instead
of a longer word list, the store now asks the database:

- No plugin SQL while TREK's role is privileged in the sense of the restore
  check (PRIVILEGED_ROLE_SQL, imported from server-checks.ts): every db:own
  call fails with PRIVILEGED_ROLE_PLUGIN_REFUSAL, read once per pool
  connection, and the boot log says why.
- The objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX ... ON
  changes, and the tables a CREATE or ALTER TABLE ties to its own, are
  resolved with to_regclass/to_regtype in the transaction right before the
  statement, and it is refused unless each one found lies in the plugin's
  schema. An unqualified int4, text or box is pg_catalog's and refused even
  with the role check off.
- The quota counts every relation of the schema with its TOAST table, and a
  schema holds at most PLUGIN_SCHEMA_MAX_OBJECTS catalog objects (relations,
  types, constraints, enum labels).
- The guard refuses every reg* type and its array in every spelling, the
  has_*_privilege functions, currtid2 and the BRIN/GIN maintenance
  functions; INTO only after the INSERT or MERGE that is the statement's
  verb or a WITH body's; TEMP only where it makes a temporary object
  (CREATE TEMP/TEMPORARY, SELECT ... INTO TEMP), so a column named temp
  works again; a base type; and a NUL anywhere in the SQL text.
- A bound value or a migration id holding NUL is refused with the reason
  instead of being stripped, so the plugin's keys never change meaning.

Every bypass from the re-review is a test on the real database, on the
role the plugin data is meant to run as and on a superuser connection with
the role check off.
2026-11-17 23:04:12 +01:00
Maurice 9808e113e4 fix(db): refuse restores for CREATEROLE members, server file grants and parameter grants
The privileged-role check missed three ways a role can reach the database
host from SQL. Before PostgreSQL 16 a member of a CREATEROLE role sets its
role to it and grants itself pg_execute_server_program, since role
attributes are not inherited but the grant checks the role in effect. A role
granted EXECUTE on lo_export, lo_import, pg_read_file, pg_ls_dir and their
siblings reads or writes server files, and lo_export keeps the file after the
transaction rolls back. From PostgreSQL 15 on a role can be granted ALTER
SYSTEM on a setting or SET on a superuser-only one.

PRIVILEGED_ROLE_SQL now counts all three. The parameter grants are read
through query_to_xml inside a CASE taken only on 15 or later, because a
statement naming pg_parameter_acl does not parse on 14. A SET grant on a
setting every role may change anyway, and pg_monitor, stay ordinary. The
lane test creates each kind of role and shows the escalation the check is
for on the server it runs against.
2026-11-17 23:04:11 +01:00
Maurice e6f4f7a5f3 fix(db): leave nothing behind when a first Postgres boot fails, so RESTORE_FROM_BACKUP still runs
createFromBaseline created the migration history's table through MikroORM's
storage before, and outside, the transaction that runs the baseline. Without a
master transaction MikroORM commits that table on its own, so a first boot that
failed in the baseline (a server without ICU, a timeout, a dropped connection)
left an empty mikro_orm_migrations behind. The next start with
RESTORE_FROM_BACKUP took that for a database in use, skipped the restore with a
warning and brought up an empty instance, after which the restore never ran.

The table is now created on the baseline's own transaction, so a failed first
boot leaves the database empty. The boot restore also treats a lone migration
history that records nothing as no database, which a restore clears anyway; any
other answer keeps it, so the check fails closed. Database-Postgres says both.
2026-11-17 23:04:11 +01:00
Maurice a884539a2b fix(db): run plugin SQL on a pool client of its own instead of patching the pool
Plugin statements on Postgres went over the extended query protocol through a
marker parameter and a hook that replaced query() on every client of TREK's
pool. That kept a plugin to one statement per round trip, but it changed
TREK's own hot path for a concern only plugin data has.

The plugin data store now checks a pg client out of the ORM's pool for each
transaction, through MikroORM's getNativeClient, and runs the transaction on
it directly. Every plugin write goes as a query with queryMode 'extended' and
every read through a pg-cursor on that client, so Postgres parses exactly one
statement either way. The settings the transaction makes are local, and
before the client goes back to the pool the store drops temporary tables,
cursors, listeners, session advisory locks and sequence state, or closes a
client it cannot bring back to that state. The row cap, quota, time budget,
untimed migrate, per-plugin queue, pool share and script commits are as
before, and plugin args are still sent without NUL.

The marker, the hook and its install in the Postgres driver are gone.
2026-11-17 23:04:10 +01:00
Maurice 223a7605ed test: import vi in the trip members suite and type the move's SQLite ORM options 2026-11-17 23:04:10 +01:00
Maurice f42ef5246c fix(db): store no NUL character on either engine
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
2026-11-17 23:04:08 +01:00
Maurice 2983971c8b fix(db): refuse restores for every role that can reach the database host
The privileged-role check looked only at the role's own rolsuper and at
pg_execute_server_program and pg_write_server_files. A role that is no
superuser itself but a member of one passed it and could SET ROLE to the
superuser from a crafted archive; on PostgreSQL 14 and 15 a role with
CREATEROLE could grant itself pg_execute_server_program; and
pg_read_server_files let COPY read the server's files into TREK's tables.

The check now counts membership in any superuser role, the three
predefined server roles, and CREATEROLE before 16. The boot warning and
the restore refusal describe the same list from one constant. A Postgres
lane suite creates each kind of role and evaluates the check as it, and
shows on the server it runs against whether CREATEROLE can grant itself
pg_execute_server_program.
2026-11-17 23:04:07 +01:00
Maurice e6a0a39a51 fix(db): read plugin SQL the way the Postgres lexer does and refuse INTO and TEMP anywhere
The guard ended a -- comment at LF only, while Postgres ends it at CR as
well, so "select 1 --\rfrom public.users" hid a qualified name from the
guard that Postgres then read, and with no args a second statement after
the CR ran as TREK's role. It also only looked for INTO at the top level
of a SELECT, so "(select ... into temp users)" made a temporary table that
shadowed TREK's own users table on the pooled connection.

The tokenizer now skips exactly what Postgres skips as whitespace, ends a
line comment at CR or LF, and refuses any character the two read
differently (other control characters, non-ASCII whitespace such as
U+00A0, U+FEFF and U+2028, and the non-decimal and underscored numbers
Postgres before 16 reads another way) instead of guessing. INTO is refused
anywhere but after INSERT or MERGE, at any depth, and the words TEMP and
TEMPORARY anywhere outside a string or quoted name.

The guard tests carry both payloads and their variants; the Postgres lane
runs them against the real database, proves a second statement in one
text is refused by Postgres over the extended protocol, and that a
temporary table never shadows TREK's or a plugin's tables.
2026-11-17 23:04:07 +01:00
Maurice ff57883e3d test(db): run the notifications and advanced packing MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The tag notification case waits for the fire-and-forget row with
vi.waitFor instead of one turn of the event loop: on SQLite the insert
lands within that turn, on Postgres it needs a few round trips more.
2026-11-17 23:04:06 +01:00
Maurice cd1a3fd433 test(realtime): run the websocket wire envelope suite on the real database module
It takes the ORM from the app buildApp boots, seeds through the factories
and resets through resetDatabase instead of the SQLite snapshot mock, so
it runs in the Postgres lane as well.
2026-11-17 23:04:06 +01:00
Maurice 377f22972b test(db): run the trips and vacay MCP suites and the route inventory on the real database module
The MCP suites seed through the booted container's ORM instead of the
SQLite snapshot mock and reset through resetDatabase. The route inventory
never touched the database and only mocked it; without the mock it
compiles AppModule over the real module, so it runs in the Postgres lane
as well.
2026-11-17 23:04:06 +01:00
Maurice fff7d123f6 test(db): run the transports, trip members and trip warnings MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The trip warnings suite seeded through the factories but only closed
the snapshot handle; it now boots the container to reach the ORM the reset
needs.
2026-11-17 23:04:06 +01:00
Maurice bedb70c479 test(db): run the share, tags, maps and weather, and transit MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The transit suite saves its permission levels through the
container's PermissionsService in a request context instead of one built
by hand over the SQLite handle.
2026-11-17 23:04:06 +01:00
Maurice 3b8270edc3 test(db): run the reservations, road trip and settings MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The road trip suite saves its permission levels through the
container's PermissionsService in a request context instead of one built
by hand over the SQLite handle.
2026-11-17 23:04:05 +01:00
Maurice 6137aaf716 test(db): run the packing, places and plugin MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The plugin search suite seeded nothing through the snapshot handle
and only closed it; it now boots the container to reach the ORM the reset
needs.
2026-11-17 23:04:05 +01:00
Maurice 97a3389409 test(db): run the feeds and invite, journey, leg transport mode, memories and notes MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well.
2026-11-17 23:04:05 +01:00
Maurice 95ade9cbb2 test(db): run the days, accommodations, files and help MCP suites on the real database module
They seed through the booted container's ORM instead of the SQLite snapshot
mock and reset through resetDatabase, so they run in the Postgres lane as
well. The collab sub-feature case in the help suite toggles through the
container's AddonsService in a request context instead of one built by
hand over the SQLite handle.
2026-11-17 23:04:05 +01:00
Maurice c4c278d2f3 test(db): run the budget and collections MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The settlement parity cases call the
container's BudgetController in a request context instead of one built by
hand over the SQLite handle, and the collections suite drops its raw
per-table DELETE for the same reset.
2026-11-17 23:04:05 +01:00
Maurice 2cb1964863 test(db): run the AirTrail, atlas expanded, collab and categories MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The PermissionsService the AirTrail cases save
through and the CategoriesMcp and DemoService the categories write
harness attaches are the container's own, called in a request context,
instead of instances built by hand over the SQLite handle.

include_coords compares the coordinates as a set: getTravelStats reads
them through a SELECT DISTINCT without an order, and Postgres returns
them in hash order.
2026-11-17 23:04:05 +01:00
Maurice 876cc5e0f0 test(db): run the demo tool gate, addon gating, atlas and assignments MCP suites on the real database module
They seed through the booted container's ORM instead of a second ORM over
the SQLite snapshot mock, and reset through resetDatabase, so they run in
the Postgres lane as well. The tour the assignment cases put on a day is
made with makeTour instead of a raw insert.
2026-11-17 23:04:04 +01:00
Maurice 1cebddac0e test(db): run the trips and vacay suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 78bfa8183a test(db): run the share, system notices and trip access primitives suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 43af349391 test(db): run the security, settings, tags and todo suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 9c35343d9d test(db): run the scheduler gate, storage health context and uploads suites on the real database module
They drop the SQLite snapshot mock and, where they seed, reset through
resetDatabase and seed through the ORM factories, so they run in the
Postgres lane as well.
2026-11-17 23:04:03 +01:00
Maurice b0b1986377 test(db): keep the legacy upgrade suites on SQLite by name
They boot on a SQLite file the retired positional runner built, which no
Postgres database ever came from, so they belong in SQLITE_ONLY rather
than in the snapshot baseline. The lane scan now tells such a suite apart
from the snapshot mock: it mocks src/db/database onto a database of its
own, never names createSnapshotTestDb, and is named in SQLITE_ONLY with
its reason. A snapshot suite added to SQLITE_ONLY stays in the baseline.
2026-11-17 23:04:03 +01:00
Maurice f11f021e15 test(db): run the Immich, Synology and unified memories suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.

IMMICH-057 and SYNO-035 need a shared photo row whose trip is gone, which
the foreign key's cascade rules out on a database that enforces it; they
write it with SQLite's foreign keys off as before and are skipped on
Postgres, which has no such switch.
2026-11-17 23:04:03 +01:00
Maurice 6e7336b8d7 test(db): run the MCP and OAuth suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well. The OAuth service they drove by hand over a second ORM on the
SQLite handle is now the container's own, called in a request context.

The MCP audit cases wait for the mcp.tool_call row: the transport writes
it without awaiting it, so on Postgres it can land after the response.
2026-11-17 23:04:03 +01:00
Maurice 927fcab67f test(db): run the places and incoming leg mode suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.

The two cases about a NUL byte in a place name move to
places-nul-sqlite.test.ts, listed in SQLITE_ONLY: they pin SQLite's value
quoting, and Postgres text cannot hold U+0000. The control characters
0x01 to 0x1F stay in places.test.ts and run on both engines.
2026-11-17 23:04:03 +01:00
Maurice c51f4f78bb test(db): run the packing, reservations and journey suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 12f4a40d7c test(db): run the OIDC, passkey, profile, notifications and ORM request context seam suites on the real database module
They reset through resetDatabase (the seam suite needs no reset) and seed
through the ORM factories instead of the SQLite snapshot mock, so they run
in the Postgres lane as well.
2026-11-17 23:04:02 +01:00
Maurice 3a6e91e26b test(db): run the journey place order, maps, misc and local OIDC provider suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 680ce7324a test(db): run the admin and auth suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well, the user erasure cases included.
2026-11-17 23:04:02 +01:00
Maurice f5fbd60b5e test(db): run the files and collab suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 8c1395c8f1 test(db): run the atlas and backup suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 19bfa77ace test(db): run the assignments, day notes and async closure guard suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:01 +01:00
Maurice 2fb041a428 test(db): run the feeds, atlas geocode, collections and Immich suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:01 +01:00
Maurice be1983c9a4 test(db): run the boot, boot sweep, demo seed, AirTrail and doc sync context suites on the real database module
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. The demo seed suite spies on the real runDemoSeed
export instead of replacing the database module to wrap it.
2026-11-17 23:04:01 +01:00
Maurice b2384892a8 test(db): run the session revocation, api docs, app config and help suites on the real database module
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. sessionRowsOf reads a user's session rows through
a booted app's ORM, next to sessionRows for the suites that still hand in
a better-sqlite3 handle.
2026-11-17 23:04:01 +01:00
Maurice 43e30ca28e test(realtime): run the websocket connection suite on the real database module
Seeds through the ORM factories, resets through resetDatabase and mints
ws-tokens through the container's TokenService instead of one built by hand
over the SQLite snapshot mock, so it runs in the Postgres lane as well.
2026-11-17 23:04:01 +01:00
Maurice 42f0705f07 test(todo): run the todo MCP tool suite on the real database module
Seeds through the booted container's ORM instead of a second ORM over the
SQLite snapshot mock, and resets through resetDatabase, so it runs in the
Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 76dfbe0a00 test(days): run the days suite on the real database module
Seeds through the ORM factories and resets through resetDatabase instead of
the SQLite snapshot mock, so it runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 424fa538e0 test(budget): run the budget suite on the real database module
Seeds through the ORM factories and resets through resetDatabase instead of
the SQLite snapshot mock, so it runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 863ff62b6c test(categories): run the categories suite on the real database module
It resets through resetDatabase instead of the SQLite snapshot mock, so it
runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 5938609f69 test(db): no await inside a loop in the Postgres tests 2026-11-17 23:03:57 +01:00
Maurice f57fb7f9de test(db): the key rotation on Postgres, and its refusals before a key is asked 2026-11-17 23:03:55 +01:00
Maurice fc37eb95d1 test(db): move a factory-seeded SQLite install into Postgres and read it back through the app 2026-11-17 23:03:55 +01:00
Maurice 910c98b971 feat(plugins): store plugin data on the engine TREK runs on
PLUGIN_DATA binds the plugin data backend to the engine (the same
{ sqlite, postgres }[engine.id] binding the backup port uses): the SQLite
file per plugin, unchanged, or the Postgres schema per plugin. The host
opens a plugin's store through it and an uninstall that deletes data erases
through it, which drops the schema on Postgres. Nothing outside db/engine
asks which engine runs.

A plugin's SQL is passed through as written, so a plugin needs to know the
dialect: db.engine (db:own) answers 'sqlite' or 'postgres', and the SDK
exposes it as ctx.db.engine(), in the mock host (dbEngine option) and in
dev. The plugin facts are regenerated and the SDK goes to 1.10.0.

The Postgres lane's PGPLUG suite installs, uses and erases plugin data on
Postgres and shows that a plugin reaches neither TREK's users table nor
another plugin's schema.
2026-11-17 23:03:53 +01:00