Commit Graph

1542 Commits

Author SHA1 Message Date
Isaias Tavares b8c9184dff test(db): move the backup, boot-restore and bootstrap tests onto the fake-tool helper
The backup, boot-restore and bootstrap tests stopped hand-writing their
stand-in tools and use tests/helpers/fake-pg-tools.ts, so one definition of
what a stand-in logs and answers covers all four files. The assertions about
adapter behaviour are unchanged.

PGBACKUP-002's refusals become it.each, each keeping its own message check:
a missing archive keeps its exact status test (PGBACKUP-002a), and the
accepted dump is PGBACKUP-002. Five refusals are covered in all: missing,
SQLite, newer, no history, and not a dump; the prompt counted six.

boot-restore hands its tables to the psql stand-in directly rather than
through a file the test writes after the plan is built. The bootstrap test
moves its Logger spy into beforeEach.
2026-11-05 00:42:11 -03:00
Isaias Tavares f3e6480606 test(db): drive the Postgres tool tests from one fake-tool helper
Four test files hand-wrote the stand-in pg_dump, pg_restore and psql
scripts, each in its own shape, and read their arguments by position ($4,
$6, args[3], slice(0, 5)). A change to what the tools are handed would have
meant finding every one of them.

tests/helpers/fake-pg-tools.ts installs the stand-ins: each logs its call,
and what it does is a named snippet (answersSchema, answersTables,
writesFile, writesRestrictedScript, listsMigrations, failsWith, touches,
copiesFileEndingWith). The output file is found by its --file= argument.
readToolCall and flagValues read a call back by name.

pg-tools.test.ts moves onto it. Argument checks use arrayContaining and
named flags; order is asserted only where it is the subject (the clear
script runs before the load, `--` precedes the dump path). PGTOOLS-007 and
PGTOOLS-009 split their scenarios into it.each, keeping their IDs as
suffixes.
2026-11-05 00:42:11 -03:00
Isaias Tavares 5503db4883 refactor(db): generate the Postgres baseline in one pass, check it in two modes
--check migrated a temporary SQLite database twice: once inside the
generation and again to read the triggers for the port check. The generation
now returns the triggers it already read, so a check is one migration.

main is split into writeMode and checkMode, and the frozen-or-not decision
lives in one place, checkBaseline. A baseline whose `through` is the newest
migration must match the entities and port every SQLite trigger; a baseline
frozen behind a newer migration is only checked by its manifest. The old
code made the same decision twice, once for the match and once for the
triggers, and the two could drift apart.

The migrated-database read is split into listing the objects and reading
them, so the temporary file is removed in one place.

Manifest parsing uses a type guard, isBaselineManifest, instead of casts
after a check, and PGBASE-004 pins what it refuses.
2026-11-05 00:36:42 -03:00
Isaias Tavares 33a18f29dd refactor(db): split the Postgres schema adjustments by concern
schema-adjust.ts mixed the statement splitter, the trigger ports and the
column adjustments, and every caller repeated `meta.props as EntityProperty[]
as ColumnMeta[]` to reach them. Now:

- sql-split.ts holds splitStatements. Its scope is written down: `;` outside
  quotes, '/" quoted text and the $$ body, and the session-setting lines it
  drops. A tagged dollar quote ($tag$) is refused with an error instead of
  being split inside, which the splitter could not do correctly.
- postgres-triggers.ts holds the hand-written trigger ports, which the
  baseline and the parity test read by name.
- schema-adjust.ts holds the column adjustments. applyPostgresColumnAdjustments
  (renamed from adjustColumns, which hid that it mutates its input) takes an
  entity's props directly: ColumnMeta now reads customType as unknown, so
  the entity's property metadata satisfies it without a cast. Each adjustment
  is a small function that returns whether it changed the column.

The timestamp check is now `instanceof DbTimestampType` rather than a compare
on the constructor's name, which would have broken silently on a rename.
2026-11-05 00:32:18 -03:00
Isaias Tavares 1c73ac4c35 refactor(db): split the Postgres client-tool module by concern
pg-tools.ts held six concerns in 366 lines: tool resolution and process
running, the libpq connection environment, the CVE-2025-8714 version policy,
the CLEAR-SCHEMA SQL and the operations themselves. Each now has its own
module, and pg-tools.ts keeps only the operations the backup and the boot
restore call, so their import paths do not change.

- pg-process.ts: PgTool, resolvePgTool, pgConnectionEnv, PgToolMissingError
  and run (stderr is summarized there, as before).
- restore-safety.ts: hasRestrictFix and the refusal. The minimum versions in
  the error are built from the RESTRICT_SINCE table, so the message cannot
  drift from the policy.
- clear-schema.ts: the SQL that empties the schema before a load, unchanged.

The version floor also no longer keeps its own OLDEST_MAJOR and
LAST_MAJOR_WITHOUT_IT constants: both are read from the table. Behaviour is
unchanged. The literal /dev/null is os.devNull, and the COPY parsing of a
dump's migration history is its own function.
2026-11-05 00:20:22 -03:00
Isaias Tavares ae005fc582 fix(db): check the restore script for TREK's tables line by line
pgRestore read the whole pg_restore --file script into one string to find
TREK's migration table before clearing the schema. The script carries every
COPY row of the dump, so a large database made the read exceed V8's maximum
string length before the check ran, and the restore failed with an
unhelpful error instead of loading or refusing.

The check now streams the script with readline and stops at the first
matching line. The match itself is unchanged: the same CREATE TABLE line for
the same quoted schema, so a dump without TREK's history is still refused
before anything is cleared. The regex's escaping is a named helper.

PGTOOLS-005b pins that the script is never read whole.
2026-11-05 00:13:04 -03:00
Isaias Tavares 293cdbc353 fix(assistant): record an unrun change as failed before the stream ends
A decision whose change never ran was written as failed only after the response stream had closed, so a client reading the thread again at once could still find it approved. On the in-memory SQLite database the write won the race; the Postgres lane lost it (ASTE2E-014). The failure is now recorded before the close, in both decision paths, and the stream still closes when that write fails. ASTE2E-026 slows the write down to pin it.
2026-11-04 23:56:09 -03:00
Isaias Tavares 63a2c9db00 docs(db): drop a migration count and a planning reference from comments
The baseline's count went stale with the first migration after the cut, and the lane test pointed at a document outside the repository.
2026-11-04 23:40:13 -03:00
Isaias Tavares 526437926a test(db): run the suites that boot the app on Postgres in their own lane
npm run test:pg --workspace=server (vitest.pg.config.ts) runs every suite
that boots the app through buildApp() on the real database module against
Postgres, each test file on its own copy of one template database the
engine's bootstrap migrates once per run. TREK_TEST_DB_URL names the server;
without it the lane fails instead of passing on nothing. The CI job
'Server: Postgres Tests' runs it on a postgres:17 service and is part of
ci-ok.

tests/postgres/ holds what only Postgres needs proving:
- PARITY-PG: MikroORM's comparator finds nothing between the bootstrapped
  database and the entities. What it reports that changes nothing (key
  columns it would make nullable, defaults and checks Postgres spells its
  own way, the hand-ported triggers) is proven so: the spelling statements
  run in a rolled-back transaction and every default and check must read
  the same after, and every ported trigger must exist.
- the write lock: concurrent read-then-insert units of work never see the
  same count, and counts are numbers; a search ignores case.

tours-hike and tours-plan-types stay on SQLite only: they move a faked
setTimeout on until a tool call ends, which a database over the network
does not finish within.
2026-11-04 22:45:32 -03:00
Isaias Tavares 3f1a786313 fix(db): keep the Postgres pool across a restore
The reopen after a restore closed the ORM's pool and waited for it to end.
Ending a pg pool waits for every connection it has lent out, and one lent
to a job while the restore ran was never handed back, so the restore
request never returned and every later query failed on a destroyed driver.
A Postgres restore replaces the contents of the same database, so the pool
now stays and the reopen only migrates; SQLite still reconnects around its
swapped file.
2026-11-04 22:45:31 -03:00
Isaias Tavares c72b680fec fix(db): keep Postgres backups and restores inside TREK's schema
pg_dump copied every schema of the database, so on a hosted Postgres such
as Supabase a backup carried its auth and storage schemas too, and a
restore failed on them. A dump now holds the current schema only, a
restore loads only that schema, and a dump without TREK's tables in it is
refused before anything is replaced.
2026-11-04 22:45:31 -03:00
Isaias Tavares fc9e634d86 feat(db): back up, restore and copy a Postgres database before migrating it
On Postgres the backup port dumps with pg_dump and restores by turning the
dump into a script with pg_restore and having psql clear the current schema
and run it in one transaction. A dump only recreates what it holds, so a
plain pg_restore --clean would leave a later migration's tables behind and
break the migrations the reopen runs; a restore that fails leaves the
database as it was. A restore refuses a pg_restore or psql older than
17.6/16.10/15.14/14.19, whose script would lack the \restrict line that
keeps a psql meta-command smuggled into an uploaded dump from running
(CVE-2025-8714). A restore
refuses a SQLite archive, a dump without TREK's migration history or one a
newer TREK wrote before anything is replaced, and keeps a dump of the
database it replaces.

RESTORE_FROM_BACKUP loads the archive's dump the same way into a database
that holds no table yet, and is ignored with a warning once it holds one.

Before pending migrations the boot now takes a pg_dump copy to
data/pre-migrate-<last migration>-<time>.pgdump under the same
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) policy as the SQLite file, instead of
refusing; a copy that cannot be written refuses the boot.

The tools run by absolute path (TREK_DB_PG_BIN_DIR, or the usual
directories), never through PATH, with the password in PGPASSWORD; an
sslmode that asks for SSL verifies the certificate as the app pool does.
Dumps are written beside their target and renamed into place, readable by
their owner only, and a tool's stderr reaches errors without the lines that
quote SQL or rows. The image installs postgresql-client-17 from Debian 13.
2026-11-04 22:45:31 -03:00
Isaias Tavares 515caffa79 feat(db): keep SQLite's write and search semantics on Postgres
- UnitOfWork transactions take one advisory lock for the database on
  Postgres (DatabaseEngine.serializeWrites, pg_advisory_xact_lock), so a
  MAX(...) + 1 read inside one cannot race another, as with SQLite's single
  writer. SQLite needs nothing.
- ilike(platform, refs) builds the case-insensitive search condition: LIKE
  on SQLite, ILIKE on Postgres, where LIKE is exact. The place searches use it.
- Plugin data erasure runs each statement in its own savepoint: on Postgres a
  failed statement would abort the user's whole erasure transaction.
2026-11-04 22:45:14 -03:00
Isaias Tavares a77705984e feat(db): create and migrate the schema on a Postgres database
With TREK_DB_URL set, TREK now boots on Postgres; without it nothing changes.

- The database engine contract gains open, close, reopen and the schema
  bootstrap; DatabaseLifecycle delegates to the engine resolveEngine() picks.
- An empty Postgres database gets server/assets/postgres/baseline.sql
  (generated from the entities by gen:pg-baseline, with the reference rows
  the migrations up to its cut insert: tour types, provider catalogues,
  addons, app settings) and every migration up to its cut recorded as
  executed, in one transaction. A database holding tables
  without TREK's history is refused before anything is written; a migrated one
  runs the newer-database refusal and its pending migrations, which refuse
  without a copy unless TREK_DB_PRE_MIGRATE_SNAPSHOT=false.
- INTEGER columns are bigint on Postgres (SQLite's INTEGER is 64-bit: an
  epoch-millisecond column overflowed int4), and the engine's driver reads
  bigint and numeric back as numbers, refusing values past 2^53.
- The two SQLite triggers are ported to PL/pgSQL; check:pg-baseline fails
  while a SQLite trigger has no port, and while the baseline drifts from the
  entities before a later migration freezes it.
- Backups and restores on Postgres refuse until they can dump and load one.
- Migration helpers read the catalog per engine (dialect/catalog.ts), and
  the email case index no longer needs group_concat.
- CI runs check:pg-baseline and boots the image on postgres:17 twice.
2026-11-04 22:45:02 -03:00
Isaias Tavares 25f5e34558 feat(db): choose the database engine from TREK_DB_URL
A port for the engine TREK runs on, without changing what it runs on.
src/db/engine/ names the engine (SqliteEngine, PostgresEngine) and builds
the MikroORM config from it; src/mikro-orm.config.ts exports that config for
the CLI and the Nest container alike. Without TREK_DB_URL the config is the
one it was: the shared SQLite driver on TREK_DB_FILE or data/travel.db.

- TREK_DB_URL takes a postgres:// or postgresql:// URL and is masked in the
  boot report; TREK_DB_POOL_MAX sizes the pool (1 to 100, default 10).
  TREK_DB_URL beside TREK_DB_FILE, TREK_DB_JOURNAL_MODE or
  TREK_DB_SYNCHRONOUS refuses the boot as an ambiguous configuration.
- MikroORM builds the pg pool from the URL's host, port, user, password and
  database and drops the rest of the query, so the engine reads sslmode and
  sslrootcert itself. require, verify-ca and verify-full all verify the
  server's certificate, as the pg driver reads them; a mode that would not
  verify, or a misspelt one, refuses.
- Postgres cannot run TREK yet, so a boot with TREK_DB_URL is refused with
  that reason, in the boot check and again where the config is built, before
  anything connects.
- @mikro-orm/postgresql moves to dependencies; pg is plain JavaScript.
- lint:engine-dispatch (CI) holds the engine checks outside src/db/dialect/
  and src/db/engine/ (isSqlite(, isPostgres(, an instanceof on a platform, a
  comparison on the engine's name) to a baseline that starts and stays empty.
2026-11-04 22:45:02 -03:00
Isaias Tavares 62912e5db1 fix(db): run every repository statement the Postgres probe reaches
The probe's baseline is empty now: no repository method Postgres refuses,
none it cannot measure.

- reservations.accommodation_id (TEXT) is cast where it is compared with
  day_accommodations.id (INTEGER), in the four reservation reads and the two
  stay reads; SQLite coerced the bare comparison, Postgres refuses it. The
  joined RS18/RR1 read moves to reservation-join.ts beside the repository.
- The usage counters qualify the target row's column in DO UPDATE SET, which
  Postgres reads as ambiguous with `excluded` otherwise.
- ReservationDayPositions.upsertScoped upserts on its primary key instead of
  INSERT OR REPLACE; position is the only other column and nothing references
  the table, so the row left behind is the same.
- PRAGMA wal_checkpoint and VACUUM INTO move from MaintenanceRepository to
  SqliteMaintenance under src/db/engine/sqlite/, the SQLite engine's folder,
  where lint:dialect allows PRAGMA.
- The places list takes category and tag ids as numbers: the service converts
  the query text with toRowId and answers an empty list for one that is no
  id, instead of binding the text to an integer column.
- The probe samples the `present` flag of a partial update's pair as true, so
  the four update methods that sent no SQL are measured, and it now holds
  SQLSTATE class 22 too: a value of the wrong type reaching a column is the
  code's, since the sampler sends what each parameter declares.
2026-11-04 22:10:18 -03:00
Maurice 67ca58f2a5 fix(mcp): a transport priced without a category is stored under its Costs key 2026-11-04 15:12:20 +01:00
Maurice 9eab19ea6c fix(mcp): hold two more tests to the fixed Costs keys
The budget prompt and the price link of a new transport now read the stored key.
2026-11-04 15:00:03 +01:00
Maurice 50ea9a74c9 fix(mcp): keep budget categories to the Costs keys, and say what a journal entry's visibility does
An assistant writing expenses through the MCP was told "e.g. Accommodation, Food,
Transport" and stored exactly that, so "Food" and "food" sat side by side in the
MCP totals while the Costs tab already read both as one. A typed category in any
case, or a common name like Hotel or Flight, is now stored as its fixed key on
every path (REST, MCP, linked booking costs), the totals the MCP reads group by
that key, and the tools name the real keys. The client reads categories through
the same shared mapping. Any other text is still kept as given.

The journal tools said that a private entry stays off the share link. It never
did: the share link shows every entry that is not a draft, whatever its
visibility, and the web editor has no visibility at all. The tools now say so
and point to is_draft, which is what keeps an entry to the contributors.

The wiki explains why the location button needs HTTPS and how to run TREK behind
an HTTPS tunnel while still using it over plain HTTP at home, and that Android's
photo picker drops a photo's location while Immich and Synology keep it.
2026-11-04 14:26:25 +01:00
Maurice 178df0fbe3 fix(map): keep the world-copy fix inside the CI ratchets
Moves the Leaflet map's event bridges (click, context menu, viewport, POI drop
and the longitude-frame follower) out of MapView into leafletMapBridges, and
shares one POI drop target between the Leaflet and GL maps, so both map files
shrink instead of growing. The repair migration does its arithmetic in one
UPDATE per table, and the new tests drop their non-null assertions and loops.
2026-11-04 11:37:52 +01:00
Maurice 51af575c9a fix(map): keep places and routes on one world, and start new installs with Journey and Collections
A right-click, click or drag on a repeated world copy reported the longitude it
has on that copy (274.9 for -85.1). The place was saved there, the geocoder
found nothing, and a route to a place on the real world ran across the whole
map (#2976). Longitudes are now brought back onto [-180, 180] where the map
hands them out (Leaflet and GL), where they are stored (places, booking
endpoints, collection places, so REST, MCP, plugin RPC and importers alike)
and before reverse geocoding. A migration moves the ones already stored.

Leaflet draws a layer at its literal longitude only, so a trip from Los
Angeles to Auckland had its pins at opposite ends of the world while the
flight ran off the edge to the next copy (#2975). A trip that crosses the
antimeridian is now drawn in one longitude frame, seamed in the widest
stretch it leaves empty, so pins, routes, flights and every overlay lie on
one copy and the map opens across the Pacific. Every other trip has no frame
and draws exactly as before. The GL day fit also takes the short way.

A new install now starts with Journey and Collections switched on, next to
the trip addons (all but Road trip), Atlas, Vacay and the Naver import that
already were. Existing installs keep their addons as they are.

Closes #2975
Closes #2976
2026-11-04 10:56:36 +01:00
Maurice 154620b99a fix(settings): format two settings tests and keep the plugin panel at its size 2026-11-03 23:38:52 +00:00
Maurice adbe3a40f6 fix(settings): lint, size and test follow-ups for the date format
The admin reset link is a component of its own instead of one made during render, the version line of the plugin panels moved into small modules, the new settings rows hand their save on as void, the date picker takes an accessible name for fields without a visible label (the Vacay hire date), the Vacay and poll tests type through the shared picker, and the theme, size and lint baselines are lowered to what the files hold now.
2026-11-03 23:38:52 +00:00
Maurice 44c6fbf372 fix(settings): let every date follow a date format of the user's choice
Every date in TREK was written in the interface language's format, and
English is en-US, so an English interface always showed and read
mm/dd/yyyy with no way to change it. A few native date inputs followed
the browser instead, so one screen could mix two formats.

date_format is now a user setting with an admin default, next to the
time format: automatic (the language's format, as before), YYYY-MM-DD,
YYYY/MM/DD, DD.MM.YYYY, DD/MM/YYYY or MM/DD/YYYY. The date picker shows,
pre-fills, hints and reads typed dates in it, and every date written in
numbers across the app (lists, cards, admin panels, CSV and PDF exports)
goes through one formatter. Dates written with a month name keep the
language's wording. The remaining native date inputs are the shared
picker now. The display settings MCP tool reads and writes the key.

Closes #2971
2026-11-03 23:38:37 +00:00
Maurice c6149cd64d fix(costs): move the settlement currency setup into the money helpers and hold the size baselines 2026-11-03 22:16:41 +00:00
Maurice 4585988dfd fix(costs): keep convertBooked at five parameters and update the tests to as-typed display rows
The display-currency check is its own helper (inDisplayCurrency), used by the Costs model and the shared trip page. The tests that pinned #2525's reading of a display-currency bill through the trip currency now pin the as-typed reading, which the balances use too; read in the trip currency, such a bill and its repayment may stay a euro cent apart.
2026-11-03 22:16:41 +00:00
Maurice a37a9c0fa8 fix(costs): count an expense in the display currency as typed
On a yen trip read in euros, an expense entered in euros was booked through the yen at the rate frozen on entry and shown back at today's rate, so 1,081.09 EUR read 1,078.86 EUR a day later, and repaying exactly what was typed left 2.23 EUR owed.

Rows in the display currency now net apart in whole display cents, as entered, and the lists show them as typed. A payment in the display currency first pays what is owed between the two in such rows; only the rest goes through the trip currency at its frozen rate, so a trip-currency debt settled in the display currency stays settled when rates move (#1445). Every other row still nets in trip cents and converts once (#1382), so the balances add up to zero in both halves.

Closes #2965
2026-11-03 22:16:41 +00:00
Maurice e798eb712e fix(assistant): run a blind call, load its group, and say why when its arguments fail
Refusing every call to a tool of a group not loaded in the turn also stopped calls whose arguments were fine. The call now runs as before; its group is loaded at once, so the next step offers the schema, and when the arguments fail the MCP check the error tells the model the schema is there now and to send the call again.
2026-11-03 22:09:54 +01:00
Maurice 0dae908e05 fix(assistant): send back a call to a tool whose group the turn has not loaded
The model could call a tool it knew from an earlier turn without loading its group in this one. TREK ran it, but the model had never seen its schema in the turn, so numbers, arrays and objects came as strings and required arguments were missing, and the MCP check refused it with -32602 (mostly the Tours tools).

Such a call no longer runs: its group is loaded for the rest of the turn and the call is answered with a tool error asking for it again with the schema the next step now offers.

Closes #2967
2026-11-03 21:48:37 +01:00
Maurice 01c7bc9de2 fix(tours): plural toast, repo route and test updates for the waypoint import
The places toast is a plural group in every language, the points import goes through tourRepo instead of src/api, the hook and dialog keep their complexity, and the tests that checked the old confirm call or the GPX left-out count follow the new behaviour.
2026-11-03 20:42:14 +01:00
Maurice 2e683f8609 fix(tours): keep the waypoints of a GPX/KML file that also holds a track
With the Tours addon on, a file picked or dropped on the places column goes
into the Tours import as soon as it holds one line, and that import reads
lines only. A Sygic or Trip-o-matic export (a route plus its stops) came in
as one tour and every place was dropped without a word.

The preview now counts a GPX file's waypoints as it already counted a KML
file's pins, and the dialog offers to add those points as places of the
trip, on by default. Confirming runs the Tours import for the lines and the
places import (points only) for the same files. A file whose tracks the
trip already holds can still bring its points.

Closes #2969
2026-11-03 20:19:43 +01:00
Maurice 91bc1944dc Managed mode: keep the admin settings and addons ratchets green
- The phone's OIDC card moves into MAdminOidcCard, so the settings section
  is back within its complexity limit after the managed-mode conditions.
- enabledPhotoProviderIds reads the enabled ids straight from the
  repository and has a test of its own, which brings the addons coverage
  back to its threshold.
2026-11-03 08:44:45 +01:00
Maurice 59e68d81b2 Managed mode: satisfy the strict, loop and test ratchets
- enabledPhotoProviderIds skips rows without an id instead of widening the type.
- The listed events read each addon once, in parallel, not in a loop.
- The narrowing test reuses the suite's service and swaps the addon answers.
- PlaceContribPanel keeps its complexity by naming the standalone check.
2026-11-03 08:44:45 +01:00
Maurice 86513fda0d Managed mode: hand place search, routing, plugins and docs to the operator
On a managed install the operator runs place search and routing, the plugin
runtime, the integrations and the documentation, so the admin and the users
no longer see switches and guides for them:

- The place search and routing card, the transit provider and the search log
  are gone from the admin settings (desktop and mobile). places_provider is a
  managed-locked setting, and the transit provider, search log and wanderer
  routes refuse writes there. Share place details stays the admin's, as a card
  of its own, hidden when TREK_PLACES_CONTRIBUTE switches it off.
- File types and trip reminders move to the right column, so the columns stay
  balanced.
- Plugins disappear from the admin and the user settings, wanderer from Tours.
- The Help & Docs page and its links are not offered. The help center stays and
  leaves out guides about what the operator runs (keys, OIDC, backups, storage,
  releases, plugins, document sync) and about integrations that are off.
- Mapbox GL is offered only with the operator's token, Amap tiles only where
  the operator set Amap up.
- The mobile admin no longer shows OIDC and the WebAuthn relying party fields,
  matching the desktop tab.

For every install:

- Notification matrices list only events the install can raise: no plugin row
  without the plugin runtime, no Vacay, collection, photo, Synology, wanderer or
  packing rows while those are off. On a managed install the release and
  replica alerts are left out.
- The journey entry editor offers External photos only where a photo provider
  is switched on, as the phone already did.
- app-config is asked again after a failed answer. A page opened while the
  server restarted kept every default for the session, which on a managed
  install brought back the operator's settings.
2026-11-03 08:44:45 +01:00
Maurice 97188c16d1 fix(place-contrib): mark the independent writes and use TREK's OSM id form in the catch-up test 2026-11-02 14:11:27 +01:00
Maurice 247d5fc519 feat(services): get the key at startup and share earlier place details
An installation now asks for its TREK services key a few minutes after
it starts, and the daily check registers a missing key as well as
renewing a due one. Until now nothing happened before someone searched
for a place or opened the admin settings, so "Share place details"
stayed unavailable on a fresh or updated instance.

When an admin agrees to share place details, what users typed into
places since TREK 5.1.0 goes out too, not only what they type from then
on. The origin of every field is recorded since 5.1.0 whatever the
decision, so a catch-up walks those places page by page before each
send run: only values that are still the ones a user typed, only for
fields switched on, only for places the service can key, and only
while the queue holds fewer than 200 values. It goes on where it
stopped after a restart, never replaces a newer value a save queued,
and ends on switching off, declining or withdrawing everything. A field
switched on later catches up as well, under an agreement given with the
new prompt sentence that says so; an agreement from 5.1.0 does not.

Queuing moves into PlaceContribQueue so a save and the catch-up take
the same path.
2026-11-02 13:56:32 +01:00
Maurice 8659262e59 fix(services): let slow handshakes reach the TREK services
Node gives each address of a dual-stack name 250 ms before it moves on.
From Australia that is shorter than one round trip to Europe, so on a
network without IPv6 the IPv4 handshake to places.liketrek.com was cut
off, the IPv6 attempt failed at once and every request ended in
ETIMEDOUT. The installation never got its services key, sharing place
details stayed unavailable and searches fell back to OpenStreetMap.

Raise the per-address attempt timeout to 2 s at startup, keeping a
larger value an operator already set, and log the reason behind
undici's bare "fetch failed" for each address tried.
2026-11-02 12:12:48 +01:00
Maurice 327ebc354f fix(push): accept Google's jmt17.google.com push endpoints
Chrome on Android increasingly hands out push subscriptions on
jmt17.google.com, which the push service allowlist did not know, so
turning push on failed with a 400. The host is added as an exact entry,
next to the two FCM hosts.

Closes #2939
2026-11-01 22:51:27 +01:00
roopesh 1625a10424 fix(server): refuse unwritable database in maintenance scripts (#2933) 2026-11-01 22:26:40 +01:00
Maurice bf8e04741d test(storage): follow the release queue in the tests that pinned an immediate delete
Replacing a cover or an avatar and deleting a note file now hand the old
bytes to the release queue instead of deleting them in the request; the
integration tests check the queue row and that the bytes are still
there. Release mocks answer with a count, the reclaim e2e sends its
refused bodies one after another, and a stale-count reload that fails is
told to the admin instead of swallowed.
2026-11-01 18:46:58 +01:00
Maurice 0b598680ec test(storage): spell the pending-reset case title without a stray quote 2026-11-01 18:32:43 +01:00
Maurice 13332cf06b fix(storage): a pass on a database this installation does not recognise only reports
While a restore's reset is still pending, or the database names another
installation, the nightly pass judges and reports but deletes nothing,
whatever the restored switch says.
2026-11-01 18:32:33 +01:00
Maurice bef8fc20f1 fix(storage): reset reclaim state after any restore, re-check owners often, share release ticks
- A restore (admin panel or RESTORE_FROM_BACKUP) now leaves
  data/.trek-reclaim-reset before it swaps the database in. Boot reconcile
  resets the reclaim state when it finds the file, whatever install id the
  database names, and removes it afterwards; a successful reopen removes it
  too. A restart after a failed reopen, or a boot restore of this
  installation's own backup, no longer keeps the old switch, marks and
  confirmations. The id mismatch reset stays.
- The release job skips a tick (install_mismatch) while the database names
  another installation or a reset is still pending.
- An allowed owner verdict is cached for 15 minutes, a refusal for 26 hours.
  The local driver reads a list prefix as a folder, so the category root is
  still listed whole, at most once per category per window.
- Each tick reads every allowed category's due rows separately, an equal
  share of the 200 each, so a category whose deletes keep failing cannot
  starve the others.
- Rows dropped after 30 days overdue and releases refused by a full queue
  are logged at warn with their counts.
2026-11-01 18:31:41 +01:00
Maurice 1c6a175ebc fix(storage): queue what a delete left unnamed past every undo, and tell a copied database apart
A delete or a replace no longer deletes inside the request. The owner's call
queues the keys in storage_release_queue, due after RELEASE_DELAY_MS (26 h:
the assistant's 24 h approval window plus a margin), and StorageReleaseJob
takes the due rows every ten minutes, at most 200 a tick and 50 keys a
question, asks every reference source again and deletes what none names.
It refuses what the nightly pass refuses (StorageReclaimGuardService: the
ledger, the required sources, a shared location, another installation's
owner marker, cached per category), and waits while a restore or a backup
runs or another storage job holds the slot. An assistant undo or a late
write that names the old value again keeps its object.

The install id lives in data/.trek-install-id, outside any backup and never
written by a restore. At boot and after every reopen the database is
reconciled with it: a database naming another installation (a boot restore,
a restart after a failed restore, a copied travel.db) loses its switch,
confirmations, reports, marks and queue before anything reads them.

A confirmation now stores the count the admin was shown and when; it covers
up to one and a half times that count (at least 100 more) for 14 days, is
used up once the share is back under a fifth, and a count the last report
no longer holds is refused as reclaim_confirm_stale. The admin state folds
a live confirmation in, so the prompt goes at once.

A pass reads each source once for every category and drops each category's
keys once judged; right before a batch it reads the category's sources
again, whole and in memory, instead of a LIKE per key. A narrowed read holds
at most 50 LIKE terms. A pass held off by a backup reports backup_running,
and a category the ledger never reclaims reports not_reclaimed even after a
stop.
2026-11-01 18:11:43 +01:00
Maurice 59a6d92cea fix(storage): type the key search run, split the reclaim view per category
The longest-run helper typed its reduce as never, reclaimView grew past
the complexity limit (one function per category line now), and a test
read a promise as a boolean.
2026-11-01 16:37:00 +01:00
Maurice d17457de56 fix(storage): delete what a delete or a replace left unnamed right away again
Leaving every replaced cover, place image and deleted journey photo to the
nightly pass kept them publicly fetchable by URL for a week at least, and
for good while deletion was off. Owners now hand the values a delete or a
replace left behind to StorageReleaseService once their write committed.
It asks every registered reference source, with the same parser as the
nightly pass, and deletes at once what no row names any more: an object a
trip copy, a journey or a saved place still shows stays, which the old
inline deletes got wrong.

The release holds the pass's guards (reclaimable categories at their root
only, never an owner marker, never while a restore runs, never with a
required source missing or an overlapping location), never fails the
request and logs at debug. It runs for trip covers (update, upload, trip
delete), collection covers and saved-place images, place images (update,
bulk update, delete), journey covers and journey or gallery photos
(their thumbnail goes with a file that went), avatars (change, delete,
account deletion), trip files (permanent delete, empty trash) and collab
note and chat attachments. A trip file row now goes before its bytes, so a
failing backend no longer fails the delete. The objects of a deleted trip's
places and files are left to the nightly pass: their rows are other
domains'.

PlacesService drops the collection places repository nothing read any
more. Size, warning and boundary baselines follow the shrunk files.
2026-11-01 16:18:21 +01:00
Maurice 790fccb92c fix(storage): judge unreferenced files without a brake anyone can trip
The 20 % brake skipped a category and wiped its marks every night once a
fifth of it looked unreferenced. Any user could push a small install over
that line, a backlog from #2924 started braked, and a second install on the
same bucket below the line was not caught at all. It is gone:

- Each pass leaves a trek-owner-<install id> marker at the root of every
  category it judges. A category holding another install's marker is
  skipped as foreign_install, one where the marker cannot be written as
  owner_unknown. Backups leave the markers out.
- The install id lives in app_settings outside the storage.* prefix. The
  switch is stored as that id, so a database from another install reads as
  off, and every reopen (a restore) writes this install's id back, turns
  deletion off and drops the marks, confirmations and reports.
- More than a fifth unreferenced still marks and reports, but deletes only
  after an admin confirmed the count shown (POST .../reclaim/confirm). A
  larger count later waits again.

A skip no longer touches the marks, so a failed listing or lookup does not
restart the week. Check now runs in the background and answers 202; the
admin state says running and keeps the last deleting report apart, so a
report-only check never resets what was freed. A pass that deletes writes a
system audit entry, a pass that stops or fails keeps its partial counts and
marks the rest interrupted, a backend error stops deleting in its category,
and deleting stops for the rest of the pass once the switch is off or a
backup is reading the uploads. The usage scan waits for a pass still
running.

Sources read each category once per pass and answer the pre-delete
re-check with a query narrowed to the keys' plain runs. Bare names keep
their raw spelling (a '#', '?' or trailing blank is part of an upload's
name), and local roots are resolved through their symlinks before the
overlap check. A pass lists only a category's root.
2026-11-01 16:18:08 +01:00
Maurice 382797febc fix(storage): let the app boot with the reclaim routes and keep the ratchets where they were
The two admin routes go on the managed allow list, without which buildApp
refused to start. The reclaim calls move to their own api module so
client.ts stays at its size, the phone card sizes its text through the
type tiers, and the tests drop a hand-built service, a non-null assertion
and a promise read as a boolean.
2026-11-01 14:44:53 +01:00
Maurice b31666d1c9 fix(storage): re-check a batch right before its delete, and drop reads nothing calls
The last question to the sources now comes after the object's own check,
so nothing slower than the delete sits between the answer and the delete,
and it is asked once per batch instead of once per object: a pass read
every source's columns for each object it was about to remove. The
collection reads the removed inline reclaims used are gone with them.
2026-11-01 14:23:04 +01:00
Maurice 19e4f0dadf fix(storage): keep the reclaim pass and a backup restore apart
A restore wipes and rehydrates the uploads under a swapped database, but it
never took the storage job slot, so a reclaim pass could start or carry on
beside it, and a delete already past the generation check could land on an
object the restore had just put back. BackupService now marks a restore as
running on StorageJobsService and waits for a pass in flight to let go; a
pass refuses to start while a restore runs and stops at its next check when
one begins. Backfills and migrations are not affected. The service test also
reads its fake source sets through a helper instead of non-null assertions.
2026-11-01 14:20:11 +01:00