Commit Graph

329 Commits

Author SHA1 Message Date
Maurice ca7cdcebcc fix(db): add every column an upgraded pre-ORM install can lack
The legacy baseline records the positional runner's steps up to
schema_version as executed without running them, so a column that runner
never added stays missing for good. Two things in its history left such
gaps: a step inserted below a version a build had already shipped (step 53,
users.mfa_backup_codes, so turning on MFA failed there) and a column added
to createTables() with no step (users.immich_access_token).

Migration20200101046460 adds, where missing, every column the schema held
at legacy step 258 that head still holds and ALTER TABLE ADD COLUMN can
add, declared as head declares it; on Postgres every one of them is in the
baseline, so nothing is added there. The list is derived again from the
migrations in tests/unit/db/legacy-column-repair.test.ts, which also adds
all of them to a database lacking them and upgrades a v244 install shaped
like the one whose move to Postgres was refused.
2026-11-18 15:22:38 +01:00
Maurice 21f4f7d5de docs(server): plain wording in the comments and test titles this change added
Comments and test titles added on this branch cited review labels and
planning documents that do not exist in the repository, and some used
dashes as separators. They now say what the code does and why in plain
sentences, and the Postgres lane's suites share one header format. Test
ids stay as they were.
2026-11-18 13:20:09 +01:00
Maurice c970245e0d fix(plugins): hold plugin data on Postgres to its own schema and to one contract on both engines
The plugin SQL guard now refuses what still reached outside a plugin's
schema or depended on the server:
- reads of the system columns (xmin, xmax, cmin, cmax, ctid, tableoid),
  which show transaction ids and storage layout and do not exist on
  SQLite; a value of the plugin's own may still be named xmin with AS
- the xid and xid8 types wherever the word stands for the type, since
  age() of an xid is the cluster's transaction counter
- the text search functions that coerce a text argument to regconfig, a
  catalog lookup by name, which leave the allowlist
- collations and access methods outside the set every server has, which
  could tell whether a locale or an extension exists

Every plugin transaction pins, with SET LOCAL, the settings that decide
how values print and parse (TimeZone UTC, DateStyle, the C locales, float,
bytea and xml output, the text search configuration), so a plugin's output
neither depends on nor reveals the server's configuration.

Both stores now keep one contract. Its texts and caps live once, in
plugin-data-contract.ts, and both stores throw only those: NUL in SQL,
an argument or a migration id is refused on SQLite as on Postgres,
transaction control is refused in every call on both engines (tx() is
how a batch becomes atomic), a closed store refuses every call, and the
Postgres store answers the cases better-sqlite3 names with its words.
The SQLite store moves beside the Postgres one as SqlitePluginDataStore
in db/engine/sqlite, takes its file from the backend so src/db never
imports src/nest, and no longer needs the driver exemption. A contract
test runs every violation on both stores and fails when they answer
differently. The Postgres lane holds the allowlist, the pinned settings
and the SQL plugins commonly write against a real database.
2026-11-18 13:20:07 +01:00
Maurice 6dfba51f21 fix(api): read body, query and handler-parsed ids with toRowId instead of Number() and parseInt
The services and controllers that took an id from a request with Number(), parseInt() or not at all now use toRowId. That covers admin users, notifications (and their limit and offset), trip members and guests, day notes, assignments, memories (photo owner, album link, body photo_id), photos, mail-ingest, the assistant id parsers, dawarich, registration invites, the plugin contribution routes and the collab reply.

The accommodation update body is open, so AccommodationsService reads the place and day refs of every stay write with toRowId and answers a refused one with the create schema's 400.

AdminService.deleteUser checked the caller with parseInt and deleted with Number(), so an admin could delete their own account by spelling its id in hex; both now use one parse.

The OAuth session revoke called Number(id) first and so revoked session 5 for '0x5'; the service now reads the id as sent.
2026-11-18 13:20:06 +01:00
Maurice 3973741509 fix(shared): refuse a malformed row id in every request schema
Request fields that hold a row id took z.union([z.number(), z.string()]), a bare z.string() or z.array(z.number()), so '0x5', '5e0', 1.5 and 1e20 reached the handler and from there SQL, where SQLite matches nothing, Postgres 14 and 15 refuse the cast and 16 and later read hex and underscores as a number. Two primitives join idParamSchema: rowIdInputSchema takes a safe integer or its digits and keeps the value as sent, so field types do not change; rowIdStringSchema takes digits only, for multipart fields. Both refuse with 'Invalid id'. The accommodation, assignment, reservation position, admin invite, collab reply, vacay, file and memories photo ids now use them, and every list of ids is z.array(z.number().int()). idParamSchema takes only the decimal spelling or a whole number, and the trip-journal-link and tour library ids use it. A body id spelled '0x5' is now a 400 on SQLite too.
2026-11-18 13:20:05 +01:00
Maurice 7f6f2eb97c fix(db): read every request row id through toRowId, at the trip guards and the repository lookups
toRowId moves to src/db/row-id.ts so repositories can use it (src/db sits below src/nest), together with a new withRowId helper; src/nest/common/row-id.ts re-exports both. Its docstring is rewritten to state the rule plainly. TripsRepository read its number | string trip ids raw: SQLite matched '0x1' against nothing, Postgres 14 and 15 refused the cast with a 500, and 16 and later read it as trip 1. Its access reads, DaysRepository.existsInTrip and the packing template lookups now parse the id first, so an id toRowId refuses is no row on every engine. TripAccessGuard, TripOwnerGuard and JourneyAssistantGuard used Number(), which read '0x1' and '1e0' as row 1; they now use toRowId.
2026-11-18 13:20:05 +01:00
Maurice e8f808b67f fix(reservations): parse every referenced id in a booking body with toRowId
The booking body is not schema-checked, so day_id, end_day_id, place_id,
assignment_id, accommodation_id and the create_accommodation ids can arrive
as any string. The reference guards passed them to SQL as they came. SQLite
compares '0x5', '5e0' or 'abc' against an integer id and matches nothing
(or, for '5e0', the row), so the guard answered "Unknown reference". Postgres
refuses the cast on 14 and 15 and reads hex as a number from 16 on, so the
same body answered a 500 on one major and reached a real row on another;
the Postgres 14 lane caught it in L2.

Both guards now read each id through toRowId, the rule every other untrusted
row id follows: an id it refuses resolves to nothing, so referencesOutsideTrip
does not name it and unresolvedReferences does, and the answer is the same
400 on every engine. create() reads the payload's accommodation_id through
toRowId as update() already did, instead of storing '0x5.0', which the
reads' join could not cast back on Postgres.

ReservationRow and ReservationTraveler move to reservation-rows.ts, the way
reservation-inputs.ts holds the input shapes, so the service stays within
its size baseline; the service exports them on and it is lowered to 1166.
2026-11-17 22:49:55 +00:00
Maurice 70108e5694 test(memories): type the Synology fakes' request init as RequestInit
The fakes that answer safeFetch by API took their init as any; safeFetch
declares it as RequestInit, which is what they read (the body as
URLSearchParams or text). The SSRF album case had gained one more of them,
which put the tests' no-explicit-any count one over its baseline. All five
are typed now, and the baseline is lowered to the 1545 the tests hold.
2026-11-17 22:49:55 +00:00
Maurice 9b1bd6e435 test(memories): answer the SSRF album case by API, not by call order 2026-11-17 23:04:12 +01:00
Maurice 13e81c04ab fix(db): drop NUL from every request value before any logic runs
The database stores no NUL on either engine, but the application still held
the value a client sent. A key or a check made from it was then a different
value from the one the database compared: `victim@x.test` followed by NUL
was a fresh key for the per-email password-reset throttle while the lookup
found the victim's row, so every spelling mailed a new link and voided the
last one.

Every input now loses its NUL where it enters TREK: the nulFreeRequest
middleware after the body parsers (path, query string, body, cookies), the
first global pipe for what a route parses itself (a multipart body), the
websocket adapter for every frame, and the MCP registry through a new
host-supplied input mapper, since /mcp keeps its raw body. Headers need
nothing: Node refuses a NUL in one with a 400. A string under a key naming a
password stays as sent: it is only ever hashed, and stripping it would lock
out an account whose password was set with one. The stripping in the
database layer stays as the backstop for what does not come from a request.
2026-11-17 23:04:11 +01:00
Maurice f42ef5246c fix(db): store no NUL character on either engine
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
2026-11-17 23:04:08 +01:00
Maurice e1a5864406 fix(storage): find a stored reference in any case on Postgres too
The release queue narrows each owner's rows with LIKE '%run%' before it
parses them, and relied on SQLite's LIKE ignoring ASCII case. On Postgres
LIKE is exact, so a row naming an object in uppercase was not found and
the release deleted an object that row still names. likeAny now matches
LOWER(column) against the lowercase run, which reads alike on both
engines. RELI-010 fails on Postgres without it.
2026-11-17 23:04:08 +01:00
Maurice 62c03d71bf fix(db): find no nearest day for a date the calendar does not have
The nearest-day fallback of a booking's time cast the bound date to a
timestamp on Postgres, which refuses 2024-02-30 with a 500, while SQLite's
JULIANDAY rolled it over into March and placed the booking there.
dayDistance now refuses anything but a calendar date on both engines
(isCalendarDate), so a caller that forgets the check fails on SQLite too,
and findNearestDayId answers no day for such a date. The size baseline of
Reservations.repository.ts drops by one line.
2026-11-17 23:04:08 +01:00
Maurice f4ce2c2bd0 fix(db): keep a carried-over half day as a half on both engines
A vacation day can be taken as a half day, so what a year carries into
the next can end in a half. vacay_user_years.carried_over was an INTEGER
column: SQLite stored 29.5 as a REAL, Postgres (bigint) rounded it to 30,
and the move refused such a SQLite value. The column becomes REAL on
SQLite (a table rebuild that keeps every id and the AUTOINCREMENT counter)
and double precision on Postgres, through the first migration after the
Postgres baseline's cut. VACAY-046 books a half day and reads the
carry-over on both engines.
2026-11-17 23:04:08 +01:00
Maurice d643f8c329 test(memories): answer the session retry cases by API, not by call order
The album list asks three sources at once; which call reaches the fake first
depends on how database reads interleave, which differs between the engines.
The fake now answers by API and category and the cases assert the retry with
a fresh session itself.
2026-11-17 23:04:08 +01:00
Maurice 1b270806a1 test(share): write the place photo cache through the app's own ORM
The suite built the cache over a second SQLite ORM, so on Postgres its rows
went to a database the app never reads.
2026-11-17 23:04:07 +01:00
Maurice caae538056 test(db): merge the factory imports of the newly ported integration suites
One import per module, and the resetTestDb import that no longer has a
caller dropped.
2026-11-17 23:04:04 +01:00
Maurice 1cebddac0e test(db): run the trips and vacay suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 78bfa8183a test(db): run the share, system notices and trip access primitives suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 43af349391 test(db): run the security, settings, tags and todo suites on the real database module
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
2026-11-17 23:04:04 +01:00
Maurice 9c35343d9d test(db): run the scheduler gate, storage health context and uploads suites on the real database module
They drop the SQLite snapshot mock and, where they seed, reset through
resetDatabase and seed through the ORM factories, so they run in the
Postgres lane as well.
2026-11-17 23:04:03 +01:00
Maurice f11f021e15 test(db): run the Immich, Synology and unified memories suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.

IMMICH-057 and SYNO-035 need a shared photo row whose trip is gone, which
the foreign key's cascade rules out on a database that enforces it; they
write it with SQLite's foreign keys off as before and are skipped on
Postgres, which has no such switch.
2026-11-17 23:04:03 +01:00
Maurice 6e7336b8d7 test(db): run the MCP and OAuth suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well. The OAuth service they drove by hand over a second ORM on the
SQLite handle is now the container's own, called in a request context.

The MCP audit cases wait for the mcp.tool_call row: the transport writes
it without awaiting it, so on Postgres it can land after the response.
2026-11-17 23:04:03 +01:00
Maurice 927fcab67f test(db): run the places and incoming leg mode suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.

The two cases about a NUL byte in a place name move to
places-nul-sqlite.test.ts, listed in SQLITE_ONLY: they pin SQLite's value
quoting, and Postgres text cannot hold U+0000. The control characters
0x01 to 0x1F stay in places.test.ts and run on both engines.
2026-11-17 23:04:03 +01:00
Maurice c51f4f78bb test(db): run the packing, reservations and journey suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 12f4a40d7c test(db): run the OIDC, passkey, profile, notifications and ORM request context seam suites on the real database module
They reset through resetDatabase (the seam suite needs no reset) and seed
through the ORM factories instead of the SQLite snapshot mock, so they run
in the Postgres lane as well.
2026-11-17 23:04:02 +01:00
Maurice 3a6e91e26b test(db): run the journey place order, maps, misc and local OIDC provider suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 680ce7324a test(db): run the admin and auth suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well, the user erasure cases included.
2026-11-17 23:04:02 +01:00
Maurice f5fbd60b5e test(db): run the files and collab suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 8c1395c8f1 test(db): run the atlas and backup suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:02 +01:00
Maurice 19bfa77ace test(db): run the assignments, day notes and async closure guard suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:01 +01:00
Maurice 2fb041a428 test(db): run the feeds, atlas geocode, collections and Immich suites on the real database module
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
2026-11-17 23:04:01 +01:00
Maurice be1983c9a4 test(db): run the boot, boot sweep, demo seed, AirTrail and doc sync context suites on the real database module
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. The demo seed suite spies on the real runDemoSeed
export instead of replacing the database module to wrap it.
2026-11-17 23:04:01 +01:00
Maurice b2384892a8 test(db): run the session revocation, api docs, app config and help suites on the real database module
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. sessionRowsOf reads a user's session rows through
a booted app's ORM, next to sessionRows for the suites that still hand in
a better-sqlite3 handle.
2026-11-17 23:04:01 +01:00
Maurice 76dfbe0a00 test(days): run the days suite on the real database module
Seeds through the ORM factories and resets through resetDatabase instead of
the SQLite snapshot mock, so it runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 424fa538e0 test(budget): run the budget suite on the real database module
Seeds through the ORM factories and resets through resetDatabase instead of
the SQLite snapshot mock, so it runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 863ff62b6c test(categories): run the categories suite on the real database module
It resets through resetDatabase instead of the SQLite snapshot mock, so it
runs in the Postgres lane as well.
2026-11-17 23:04:00 +01:00
Maurice 900f2bfc30 test(realtime): read each parity case after its notification sends have settled
On Postgres a fire-and-forget notification still ran when the case read its recording, so it landed in the next case. The harness now waits for NotificationsService.settled() after each write and each exemption; a late send of any other kind still fails the next case.
2026-11-17 23:03:58 +01:00
Maurice 910c98b971 feat(plugins): store plugin data on the engine TREK runs on
PLUGIN_DATA binds the plugin data backend to the engine (the same
{ sqlite, postgres }[engine.id] binding the backup port uses): the SQLite
file per plugin, unchanged, or the Postgres schema per plugin. The host
opens a plugin's store through it and an uninstall that deletes data erases
through it, which drops the schema on Postgres. Nothing outside db/engine
asks which engine runs.

A plugin's SQL is passed through as written, so a plugin needs to know the
dialect: db.engine (db:own) answers 'sqlite' or 'postgres', and the SDK
exposes it as ctx.db.engine(), in the mock host (dbEngine option) and in
dev. The plugin facts are regenerated and the SDK goes to 1.10.0.

The Postgres lane's PGPLUG suite installs, uses and erases plugin data on
Postgres and shows that a plugin reaches neither TREK's users table nor
another plugin's schema.
2026-11-17 23:03:53 +01:00
Isaias Tavares 2e3762afe1 feat(db): run TREK on PostgreSQL
TREK runs on PostgreSQL when TREK_DB_URL is set, and on SQLite as before
otherwise: an engine port under src/db/engine, the dialect helpers under
src/db/dialect, a Postgres baseline for the migrations up to the cut,
pg_dump/pg_restore backups, a probe of every repository method against a
real server and a Postgres test lane in CI.
2026-11-17 23:03:33 +01:00
Mariano 33d074c786 feat(places): tags on places, owned by the trip
A place keeps its one category, which decides its marker, and can now carry
any number of tags as extra labels to filter by.

- Tags belong to the trip (tags.trip_id): every member sees the same list and
  whoever may edit the trip's places adds, renames and deletes. Personal tags
  stay what they were.
- The place form picks them on desktop and mobile through one shared hook,
  usePlaceTags, and can add a new tag to the trip on the spot.
- The existing category filter of the places list lists the tags as well.
- Tags show on the place (inspector pills, phone place sheet); the marker is
  untouched.
- MCP: create_place, create_and_assign_place and update_place take `tags`,
  and list/create/update/delete_trip_tag manage a trip's tags.
- /api/trips/:tripId/tags and the tripTags:changed event keep every client
  of the trip in step.
2026-11-17 08:44:10 +01:00
Maurice 2a134babbc fix(trips): keep a visit's own stay when a trip is copied
insertAssignmentCopy took stay_minutes but never wrote it, so a copied trip
fell back to the place's duration for every visit.
2026-11-16 10:45:24 +01:00
Maurice b8966330e0 test: keep the bringer and panel width changes within the lint and RTL ratchets
The bringer tests read their items through a small type instead of any, the
panel hook names its two ceilings after the panels rather than the sides, and
the stored-width test runs on a window wide enough for the map's share.
2026-11-15 12:22:12 +01:00
Maurice 7bbb5a9f71 fix(packing): anyone on the trip can be put down as the bringer of a shared item, and taken off again (#3098)
A common item only let you pledge yourself. Nobody else could be named,
and a bringer could never be removed, so an item that went personal and back
kept its owner's avatar for good. The item menu now lists every member to
tick on or off, a click on the avatar opens it there, and the add route takes
an optional user_id limited to the trip's members. Taking the owner off
clears the owner. The remove route now checks visibility for the caller, not
for the user named in the path.
2026-11-15 11:00:21 +01:00
Maurice bac4599d64 fix(roadtrip): keep the stay on each visit, not on the place (#3092)
A nullable day_assignments.stay_minutes holds the stay of one visit; a visit without one keeps reading the place's duration, so existing trips drive as before. REST PUT /trips/:tripId/assignments/:id/stay and the MCP tool set_assignment_stay write it with day_edit.
2026-11-14 17:48:57 +01:00
Maurice 6789ee1cb6 refactor(server): keep the grown files at their size baselines; centre the inbox dialog
- auth.service: how an admin form value is stored (validated, encrypted, a
  masked echo kept) moves to the app settings registry as storedAdminValue
  and isMaskEcho.
- reservations.service: a booking's origin (withoutOrigin, originColumns)
  moves to reservation-origin.ts, its input shapes to reservation-inputs.ts,
  exported on from the service so importers stay as they were.
- Reservations.repository: the joined projection, the origin lookup and the
  origin columns move to _shared/, like the Wallet pass context before.
- test-sql: MailIngestApplyService.prepare is marked as no SQL statement.
- The pending bookings dialog opens centred, and says 'nothing waiting' once.
2026-11-13 20:28:17 +01:00
Maurice c13c288f00 test(broadcast-parity): cases for the mail import's write tools
add, move, file, discard and undo of a mail booking and the import
settings each get a case with their REST twin. A trip expectation may
name its own room, for the move that announces in two trips.
2026-11-13 19:31:14 +01:00
Maurice b9231ffb53 feat(mcp): manage the trip's shared links over MCP (#3076)
list_collab_links, create_collab_link, update_collab_link and
delete_collab_link run the Links panel's REST routes as tools: the shared
body contracts, trip access, collab_edit on writes and the collab🔗*
broadcasts, behind the links sub-feature flag. Unit, e2e and broadcast
parity cases cover them; the wiki lists the tools.
2026-11-13 10:17:15 +01:00
Maurice 5345b50d17 test(vacay): the holiday provider cases live in the existing suites again
The two new test files mocked the db module and built VacayService by hand,
which the test-mocks and new-service ratchets refuse for files without a
baseline entry. The cases move back into vacay.service.test.ts and
vacay.test.ts unchanged; VACAY-037 asks for a year no other case caches. The
unrelated admin addon test change is reverted and two comments name the new
provider.
2026-11-11 21:06:29 +00:00
Luffy a3622e91b6 feat(vacay): implement Nager.Holidays v4 integration and adapt holiday data structure 2026-11-11 21:04:07 +00:00