Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
The release queue narrows each owner's rows with LIKE '%run%' before it
parses them, and relied on SQLite's LIKE ignoring ASCII case. On Postgres
LIKE is exact, so a row naming an object in uppercase was not found and
the release deleted an object that row still names. likeAny now matches
LOWER(column) against the lowercase run, which reads alike on both
engines. RELI-010 fails on Postgres without it.
The nearest-day fallback of a booking's time cast the bound date to a
timestamp on Postgres, which refuses 2024-02-30 with a 500, while SQLite's
JULIANDAY rolled it over into March and placed the booking there.
dayDistance now refuses anything but a calendar date on both engines
(isCalendarDate), so a caller that forgets the check fails on SQLite too,
and findNearestDayId answers no day for such a date. The size baseline of
Reservations.repository.ts drops by one line.
A vacation day can be taken as a half day, so what a year carries into
the next can end in a half. vacay_user_years.carried_over was an INTEGER
column: SQLite stored 29.5 as a REAL, Postgres (bigint) rounded it to 30,
and the move refused such a SQLite value. The column becomes REAL on
SQLite (a table rebuild that keeps every id and the AUTOINCREMENT counter)
and double precision on Postgres, through the first migration after the
Postgres baseline's cut. VACAY-046 books a half day and reads the
carry-over on both engines.
The album list asks three sources at once; which call reaches the fake first
depends on how database reads interleave, which differs between the engines.
The fake now answers by API and category and the cases assert the retry with
a fresh session itself.
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
They reset through resetDatabase and seed through the ORM factories on the
booted app's ORM instead of the SQLite snapshot mock, so they run in the
Postgres lane as well.
They drop the SQLite snapshot mock and, where they seed, reset through
resetDatabase and seed through the ORM factories, so they run in the
Postgres lane as well.
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
IMMICH-057 and SYNO-035 need a shared photo row whose trip is gone, which
the foreign key's cascade rules out on a database that enforces it; they
write it with SQLite's foreign keys off as before and are skipped on
Postgres, which has no such switch.
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well. The OAuth service they drove by hand over a second ORM on the
SQLite handle is now the container's own, called in a request context.
The MCP audit cases wait for the mcp.tool_call row: the transport writes
it without awaiting it, so on Postgres it can land after the response.
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well.
The two cases about a NUL byte in a place name move to
places-nul-sqlite.test.ts, listed in SQLITE_ONLY: they pin SQLite's value
quoting, and Postgres text cannot hold U+0000. The control characters
0x01 to 0x1F stay in places.test.ts and run on both engines.
They reset through resetDatabase (the seam suite needs no reset) and seed
through the ORM factories instead of the SQLite snapshot mock, so they run
in the Postgres lane as well.
They reset through resetDatabase and seed through the ORM factories
instead of the SQLite snapshot mock, so they run in the Postgres lane as
well, the user erasure cases included.
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. The demo seed suite spies on the real runDemoSeed
export instead of replacing the database module to wrap it.
They boot the app without the SQLite snapshot mock, so they run in the
Postgres lane as well. sessionRowsOf reads a user's session rows through
a booted app's ORM, next to sessionRows for the suites that still hand in
a better-sqlite3 handle.
On Postgres a fire-and-forget notification still ran when the case read its recording, so it landed in the next case. The harness now waits for NotificationsService.settled() after each write and each exemption; a late send of any other kind still fails the next case.
PLUGIN_DATA binds the plugin data backend to the engine (the same
{ sqlite, postgres }[engine.id] binding the backup port uses): the SQLite
file per plugin, unchanged, or the Postgres schema per plugin. The host
opens a plugin's store through it and an uninstall that deletes data erases
through it, which drops the schema on Postgres. Nothing outside db/engine
asks which engine runs.
A plugin's SQL is passed through as written, so a plugin needs to know the
dialect: db.engine (db:own) answers 'sqlite' or 'postgres', and the SDK
exposes it as ctx.db.engine(), in the mock host (dbEngine option) and in
dev. The plugin facts are regenerated and the SDK goes to 1.10.0.
The Postgres lane's PGPLUG suite installs, uses and erases plugin data on
Postgres and shows that a plugin reaches neither TREK's users table nor
another plugin's schema.
A place keeps its one category, which decides its marker, and can now carry
any number of tags as extra labels to filter by.
- Tags belong to the trip (tags.trip_id): every member sees the same list and
whoever may edit the trip's places adds, renames and deletes. Personal tags
stay what they were.
- The place form picks them on desktop and mobile through one shared hook,
usePlaceTags, and can add a new tag to the trip on the spot.
- The existing category filter of the places list lists the tags as well.
- Tags show on the place (inspector pills, phone place sheet); the marker is
untouched.
- MCP: create_place, create_and_assign_place and update_place take `tags`,
and list/create/update/delete_trip_tag manage a trip's tags.
- /api/trips/:tripId/tags and the tripTags:changed event keep every client
of the trip in step.
The bringer tests read their items through a small type instead of any, the
panel hook names its two ceilings after the panels rather than the sides, and
the stored-width test runs on a window wide enough for the map's share.
A common item only let you pledge yourself. Nobody else could be named,
and a bringer could never be removed, so an item that went personal and back
kept its owner's avatar for good. The item menu now lists every member to
tick on or off, a click on the avatar opens it there, and the add route takes
an optional user_id limited to the trip's members. Taking the owner off
clears the owner. The remove route now checks visibility for the caller, not
for the user named in the path.
A nullable day_assignments.stay_minutes holds the stay of one visit; a visit without one keeps reading the place's duration, so existing trips drive as before. REST PUT /trips/:tripId/assignments/:id/stay and the MCP tool set_assignment_stay write it with day_edit.
- auth.service: how an admin form value is stored (validated, encrypted, a
masked echo kept) moves to the app settings registry as storedAdminValue
and isMaskEcho.
- reservations.service: a booking's origin (withoutOrigin, originColumns)
moves to reservation-origin.ts, its input shapes to reservation-inputs.ts,
exported on from the service so importers stay as they were.
- Reservations.repository: the joined projection, the origin lookup and the
origin columns move to _shared/, like the Wallet pass context before.
- test-sql: MailIngestApplyService.prepare is marked as no SQL statement.
- The pending bookings dialog opens centred, and says 'nothing waiting' once.
add, move, file, discard and undo of a mail booking and the import
settings each get a case with their REST twin. A trip expectation may
name its own room, for the move that announces in two trips.
list_collab_links, create_collab_link, update_collab_link and
delete_collab_link run the Links panel's REST routes as tools: the shared
body contracts, trip access, collab_edit on writes and the collab🔗*
broadcasts, behind the links sub-feature flag. Unit, e2e and broadcast
parity cases cover them; the wiki lists the tools.
The two new test files mocked the db module and built VacayService by hand,
which the test-mocks and new-service ratchets refuse for files without a
baseline entry. The cases move back into vacay.service.test.ts and
vacay.test.ts unchanged; VACAY-037 asks for a year no other case caches. The
unrelated admin addon test change is reverted and two comments name the new
provider.
The settings for the database were one flat record whose url and pool size
were meaningful only for Postgres, so every reader had to re-check the URL.
They are now a discriminated union per engine (DbEngineSettings): the
Postgres arm carries its URL and a pool size already capped at the ceiling,
and the SQLite arm carries nothing else. The env schema uses the same ceiling
constant, so the boot check and the derivation cannot drift apart.
The engine was resolved from the environment by every caller that needed it
(the lifecycle, the unit of work, the backup adapters), each building its own
instance. The engine is now provided once per container by
DatabaseEngineModule and injected everywhere, so the state an engine keeps
(Postgres's reopen hook) is the same wherever it is used. resolveEngine stays
for the MikroORM CLI config and the DI factory, and nothing else calls it.
The engine's reopen no longer carries a `reopenSwapsConnection` flag. The
engine itself decides what a reopen needs: SQLite closes and reconnects the
ORM's connection before the schema bootstrap (its handle is swapped under
Kysely's cache), and Postgres only migrates, because a restore replaces the
contents of the database the pool already reaches. The Postgres pool is never
closed on a reopen, since ending it waits for connections lent out during the
restore.
The backup ports answer every method the same way. verify() now throws a
DomainError for a refused archive on both engines, and the restore path maps
it back to the same { success: false, error, status } result, so the HTTP
response of a refused restore (status and body) is unchanged. The restore
step that swaps the database and then always reopens it is shared in
nest/backup/database-swap.ts, with the error-description helper in
nest/common, instead of two copies that could drift.
The probe's baseline is empty now: no repository method Postgres refuses,
none it cannot measure.
- reservations.accommodation_id (TEXT) is cast where it is compared with
day_accommodations.id (INTEGER), in the four reservation reads and the two
stay reads; SQLite coerced the bare comparison, Postgres refuses it. The
joined RS18/RR1 read moves to reservation-join.ts beside the repository.
- The usage counters qualify the target row's column in DO UPDATE SET, which
Postgres reads as ambiguous with `excluded` otherwise.
- ReservationDayPositions.upsertScoped upserts on its primary key instead of
INSERT OR REPLACE; position is the only other column and nothing references
the table, so the row left behind is the same.
- PRAGMA wal_checkpoint and VACUUM INTO move from MaintenanceRepository to
SqliteMaintenance under src/db/engine/sqlite/, the SQLite engine's folder,
where lint:dialect allows PRAGMA.
- The places list takes category and tag ids as numbers: the service converts
the query text with toRowId and answers an empty list for one that is no
id, instead of binding the text to an integer column.
- The probe samples the `present` flag of a partial update's pair as true, so
the four update methods that sent no SQL are measured, and it now holds
SQLSTATE class 22 too: a value of the wrong type reaching a column is the
code's, since the sampler sends what each parameter declares.
Replacing a cover or an avatar and deleting a note file now hand the old
bytes to the release queue instead of deleting them in the request; the
integration tests check the queue row and that the bytes are still
there. Release mocks answer with a count, the reclaim e2e sends its
refused bodies one after another, and a stale-count reload that fails is
told to the admin instead of swallowed.
A delete or a replace no longer deletes inside the request. The owner's call
queues the keys in storage_release_queue, due after RELEASE_DELAY_MS (26 h:
the assistant's 24 h approval window plus a margin), and StorageReleaseJob
takes the due rows every ten minutes, at most 200 a tick and 50 keys a
question, asks every reference source again and deletes what none names.
It refuses what the nightly pass refuses (StorageReclaimGuardService: the
ledger, the required sources, a shared location, another installation's
owner marker, cached per category), and waits while a restore or a backup
runs or another storage job holds the slot. An assistant undo or a late
write that names the old value again keeps its object.
The install id lives in data/.trek-install-id, outside any backup and never
written by a restore. At boot and after every reopen the database is
reconciled with it: a database naming another installation (a boot restore,
a restart after a failed restore, a copied travel.db) loses its switch,
confirmations, reports, marks and queue before anything reads them.
A confirmation now stores the count the admin was shown and when; it covers
up to one and a half times that count (at least 100 more) for 14 days, is
used up once the share is back under a fifth, and a count the last report
no longer holds is refused as reclaim_confirm_stale. The admin state folds
a live confirmation in, so the prompt goes at once.
A pass reads each source once for every category and drops each category's
keys once judged; right before a batch it reads the category's sources
again, whole and in memory, instead of a LIKE per key. A narrowed read holds
at most 50 LIKE terms. A pass held off by a backup reports backup_running,
and a category the ledger never reclaims reports not_reclaimed even after a
stop.
Leaving every replaced cover, place image and deleted journey photo to the
nightly pass kept them publicly fetchable by URL for a week at least, and
for good while deletion was off. Owners now hand the values a delete or a
replace left behind to StorageReleaseService once their write committed.
It asks every registered reference source, with the same parser as the
nightly pass, and deletes at once what no row names any more: an object a
trip copy, a journey or a saved place still shows stays, which the old
inline deletes got wrong.
The release holds the pass's guards (reclaimable categories at their root
only, never an owner marker, never while a restore runs, never with a
required source missing or an overlapping location), never fails the
request and logs at debug. It runs for trip covers (update, upload, trip
delete), collection covers and saved-place images, place images (update,
bulk update, delete), journey covers and journey or gallery photos
(their thumbnail goes with a file that went), avatars (change, delete,
account deletion), trip files (permanent delete, empty trash) and collab
note and chat attachments. A trip file row now goes before its bytes, so a
failing backend no longer fails the delete. The objects of a deleted trip's
places and files are left to the nightly pass: their rows are other
domains'.
PlacesService drops the collection places repository nothing read any
more. Size, warning and boundary baselines follow the shrunk files.
The reclaim e2e and the source-registration check mocked src/db/database,
which lint:test-mocks holds at zero for new files; both now boot buildApp
against the worker's snapshot db. The key parser drops its non-null
assertions (the indexed reads are typed string already), and the e2e sends
its refused bodies at once instead of awaiting inside a loop, so the eslint
ratchet stays where it was.
Replacing a trip or list cover, changing or deleting a place or saved-place
image and deleting a journey photo removed the bytes at once, without asking
whether a trip copy, a journey, a saved place or another photo row still
named them (#2924). Those paths now leave the object in place and the reclaim
job removes it once no row names it. A journey photo the user deletes takes
its local photo row with it when nothing else links it, so its bytes become
reclaimable instead of being kept by a row nobody sees.
A place image copied from a trip file is still dropped at once when the
update that would name it fails: that copy was never anyone else's.
Size and warning baselines follow the shrunk files.
Domains whose rows name stored objects register a reference source with
StorageReclaimRegistryService; storage never imports them. A ledger in
storage decides per category whether it is reclaimed at all and which
sources must have answered first, and a category whose resolved location
overlaps another one is left alone.
A nightly pass at 04:05 lists the root level of files, journey, covers,
avatars and places, skips objects younger than a day or without a
modification time, asks every source in pages of 500 and marks the rest.
Only with deletion switched on, and only for marks a week old, it asks every
source about the single key again and deletes through StorageService. A
category is skipped and its marks start over when a source is missing or
fails, the listing fails, or more than a fifth of its objects look
unreferenced. A pass shares the one-job slot with backfills and migrations,
never runs on a demo instance and stops when the database is reopened.
Admins switch deletion and run a pass on demand under
/api/admin/storage/reclaim; both are audited, and GET /api/admin/storage
carries the switch and the last report. Deletion is off after install and
upgrade.
The Atlas server no longer keeps the admin-1 regions of every country in memory. Each country's regions are now a pre-gzipped file of their own, sent to the map as they are.
**Why:** `GET /addons/atlas/regions/geo?countries=…` built `admin1Store` (every country's region text, kept until restart), then joined the requested countries, `JSON.parse`d them, and Nest stringified them again. On the 512 MB – 1 GB boxes TREK runs on, the first z≥5 view could OOM the server (same shape as #1576).
**What changes:**
- `server/assets/atlas/admin1/<ISO_A2>.geojson.gz`: 198 files, 14.85 MB, the same 3,447 features as `admin1.geojson.gz`.
- New `GET /addons/atlas/regions/geo/:code` with a Zod contract (`regionGeoParamsSchema` in `@trek/shared`: two letters, any case). It sends the country's file as it is, with `Content-Encoding: gzip` and the current 1-day `Cache-Control`, as `countries/geo` does. A country without regions gets an empty FeatureCollection. Anything that is not two letters gets a 400.
- `useAtlas.ts` fetches one URL per uncached country and still rebuilds the layer once. Each country has a stable URL, so a country evicted by `REGION_CACHE_MAX` comes back from the browser cache.
- `getRegionFeatures` reads only the country it needs. `admin1Store` is removed.
- `?countries=` keeps working for cached PWA clients. It joins the per-country files as bytes, without a parse.
- The migrations are untouched. `admin1.geojson.gz` still ships, only for `Migration20200101021500` and `Migration20200101031500`. This costs ~15 MB of disk in the image and no RAM. `build-atlas-geo.mjs` writes both, and ATLAS-BUNDLE-008 keeps them equal.
**Memory** (real `atlas-geo` module, first z≥5 view over Europe = 49 country codes, JS heap + Buffers):
| | peak | kept after |
|---|---|---|
| before, first view (builds the store) | +372 MB | +61 MB |
| before, later views | +115 MB | 0 |
| after, `/regions/geo/:code` | +4.7 MB | 0 |
| after, legacy `?countries=` | +59 MB | 0 |
**Tests:**
- New: `atlas-region-files.test.ts` (moved from `atlas.service.test.ts`, plus join and dedupe cases), the `regionGeoParamsSchema` spec, ATLAS-BUNDLE-007 (the file format the byte join depends on), ATLAS-BUNDLE-008 (bundle and per-country files are equal), ATLAS-016/017/018 (the new route over HTTP, including the 400).
- The `getVisitedRegions` tests (ATLAS-UNIT-019..024) used to pass only when `getRegionGeo`'s `beforeAll` had built the store first. They now also pass on their own.
- The shared, server and client gates pass. Baselines go down only where this change removed code (`no-console`, `no-explicit-any`, `no-swallowed-catch`, file sizes).
**Trade-offs:**
- A first zoom over Europe sends ~49 small parallel requests instead of 1. Over HTTP/1.1 the browser queues them 6 at a time. The total bytes are the same (4.4 MB gz).
- The per-country URLs are unversioned and cached for a day, like `countries/geo`. After an upgrade that changes a country's regions, a browser can show the old shapes for up to 24 h.
Approved on Discord in #github-pr: <paste the thread link here>
Related: #1576, #2559
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] I have read the [Contributing Guidelines](https://github.com/liketrek/TREK/wiki/Contributing)
- [ ] My branch is [up to date with `dev`](https://github.com/liketrek/TREK/wiki/Development-environment#3-keep-your-fork-up-to-date)
- [x] This PR targets the `dev` branch, not `main` *(wiki-only PRs are exempt)*
- [x] I have tested my changes locally
- [x] I have added/updated tests that prove my fix is effective or that my feature works
- [x] I have updated documentation if needed