Commit Graph

69 Commits

Author SHA1 Message Date
Maurice d0d3f455a8 fix(planner): let the day list see which stop a booking wrote
The row shapers name every column they pass on, and accommodation_id was in
none of them. So the field the day list hides on never reached the client, and
the booked night was listed as an ordinary place next to the overnight block
that already named it. The stop was correct in the table the whole time; the
three read paths just dropped it on the way out.

The stamp on the freshly written stop had the same hole from the other side: the
INSERT went in without the booking id and an UPDATE added it afterwards, so the
row handed back in the answer, the one the client that booked the night puts
straight into its store, carried none. It goes in with the INSERT now.

The shared link gains the same filter. It has no road trip view to need the stop
and already shows the booking as its own chip on the day, so the row was only
ever the same hotel twice.
2026-09-13 12:59:19 +02:00
Maurice 43a6c30325 feat(places): remember the Amap id a place was found by
The provider produced amap_poi_id, but nothing stored it, so a saved
Amap place could never be reopened against Amap. The column now travels
with google_place_id and osm_id through the shared contract, the place
insert and update, duplicate matching, the assignment projections, the
trip copy and the three place MCP tools.
2026-09-12 18:03:10 +02:00
A1phaN 57eac32f9f feat(map): amap as map api provider and display template 2026-09-12 18:02:57 +02:00
Maurice 491cfccc56 feat(roadtrip): add daily scheduling and configurable day endings 2026-09-11 22:28:03 +02:00
Maurice 96ab748f34 feat(roadtrip): let one stop say how full it fills
The fill level was one figure for the whole trip, and a trip does not have one.
A motorway rapid charger is worth about 80 % because the last fifth costs as
long again; the charger at the hotel is worth all of it because the car stands
there all night. Between those two the difference is a whole leg, and a single
setting has to be wrong about one of them.

So the badge on a refuelling stop is now the way in: it shows what that stop
will actually fill to, its own figure or the traveller's, and opens a panel of
the six answers people actually give. Under them it says what the choice buys in
kilometres, because 80 % is an abstraction and "about 290 km from here" is the
thing being decided. Picking the figure a stop already has hands it back to the
setting, the same gesture the kind picker uses to un-pick a kind.

The value lives in places.fill_percent, beside stop_type, because it is a
property of the stop rather than of the person. Null means "follow the setting",
which is every stop that predates the column and every stop nobody has an
opinion about, so nothing changes for a trip that never touches it.

Two pieces of shared machinery came out of it rather than being copied. The
popover chrome — measure before showing, flip when the window is short, close on
an outside click, Escape or a scroll — is one component now, used by the kind
picker and this one. So is the assembly of the range from seven settings, which
the routes hook and the rail both need and which would have drifted the first
time a field was added.
2026-09-10 21:40:51 +02:00
Maurice 77162aa8d5 feat(roadtrip): avoid tolls, motorways and ferries, through a second engine
The switch a road tripper keeps asking for has never worked. OSRM's `exclude` is the only
avoidance TREK has, and the two public hosts it ships with answer it HTTP 400 on every
leg: their car profile is built without excludable classes, and their own Lua says why
("no memory for classes on demo server"). So "drive this without the tolls" produced
nothing at all on a default install, and the fallback inside the alternatives bar quietly
gave up after one refusal per host.

FOSSGIS runs a public Valhalla on the whole planet next to that OSRM, keyless, and there
avoidance is a runtime weighting rather than a profile that has to be rebuilt. This adds
it as a second engine asked only the questions the first cannot answer, and puts three
switches under the road trip settings that change the road itself rather than offering an
alternative beside it. Paris to Lyon measured 465.8 km / 297 min with tolls against
492.0 km / 454 min without; Hamburg to Copenhagen comes back ferry-free at 469.8 km,
which OSRM cannot produce at all — where a ferry IS the crossing it answers NoRoute
rather than driving round.

Four things the docs would have got wrong, all measured against the live instance:

- `format=osrm` does not carry `has_toll`. That flag is the entire reason for doing this,
  and it exists only in the native answer — which is also 5.7x smaller, because the OSRM
  shape drags every intersection along. So the adapter translates instead of borrowing a
  familiar envelope.
- Without a `radius` Valhalla refuses a point it cannot bind to a road, where OSRM still
  snaps one. Vienna's Stephansdom, in a pedestrian zone, answers error 442 until a search
  radius is allowed. TREK is full of such places, so every location carries one.
- Ten locations is a hard cap and a road trip day is bundled into one request. A day over
  it is split into runs overlapping by one point, which is exact rather than approximate:
  a measured six-stop day came back within two metres and under a second of the same day
  asked whole.
- Its drive times run 5.4 % above OSRM's over ten European legs and 36 % above on one, and
  those times feed the arrival clocks of a day. So a day is answered by one engine or by
  neither, never by both, and a piece that will not answer loses the whole day rather than
  half of it — the schedule chains leg times blindly and would otherwise print a complete,
  plausible, silently wrong timetable.

Because the weighting can fail, the honest surface is the answer rather than the request.
`use_tolls: 0` still returns a tolled road where there is no untolled crossing, so what
the road actually turned out to be is read back per leg, and a day that could not avoid
what it was asked to avoid says so on its own header instead of wearing a label the road
disproves. The switches disable themselves, with a reason, on an instance pointed at its
own OSRM: that operator chose against public hosts, and a second one they never asked for
would undo the choice.

Two things that would have broken quietly and did not:

- The route cache keys on `profile:unit:coords` and is shared with the day planner, the
  studio and the booking geometry. Without the avoidance in the key, flipping a switch
  returns the old road out of cache in under a millisecond — and worse, an avoiding answer
  stored under the plain key hands those other callers a road nobody drives.
- `snapped` is OSRM-only and carries the dashed kerb-to-door spur from #2273. Valhalla
  echoes the coordinates it was given and never says where it bound them, but the ends of
  a leg's line are exactly those points: measured against Füssen, its first vertex matches
  OSRM's snapped location to the digit.

The alternatives bar gets the same engine, which finally makes its "no motorway" and "no
tolls" offers appear on a default install, plus a ferry-free one. `sameRoad` now compares
geometry rather than numbers there, because two engines' distances for the same road sit
kilometres apart and the old 200 m tolerance called that a discovery.

Desktop only, like the whole road trip mode: nothing under client/src/mobile renders it.
2026-09-09 19:03:00 +02:00
Maurice 7e3b9fd039 fix(collab): keep every panel on screen, and stop leaking spool and links
The desktop layout rendered notes and links as a pair and dropped whichever of
the two was on its own, so turning the new Links feature off on an otherwise
default install made the Notes panel disappear. Both rows are now split by kind
and rendered only when they hold something, and one function decides what a
panel id renders, because the branch that lost a panel did so by not listing it.

The message and its attachment rows are written in one transaction. The caller
has already committed the image bytes by then, so a message that landed without
them left those bytes with nothing pointing at them and nothing to sweep them.
Deleting one now drops the blobs after the rows are gone rather than before.

A body the Zod pipe rejects used to leave up to four spooled images on disk:
pipes run after interceptors, so multer had already written them and the
handler's own cleanup was never reached. An interceptor removes them on any
failure before the handler, which also covers every other schema mismatch, and
the length check that could never fire is gone.

Adding, pinning and deleting a link now says something when it fails instead of
raising an unhandled rejection, the list has real types, and the rejected-image
toast is translated in all 23 locales rather than shipping English everywhere.
The picking, previewing and clearing of chat images moved into one hook both
shells use: it was copied into two, and both rebuilt every preview on each
change and leaked an object URL per image per pick.

Tests: the four link routes end to end with the validation pipe the harness was
missing, the image filter against the disguised-extension case, and the layout
combinations that drop a panel.
2026-09-09 00:11:00 +02:00
Maurice 9a0a6f8c3b fix(settings): the routing endpoint belongs to the operator
routing_base_url arrived on this branch and was assigned to neither managed
set, which MANAGED-KEYS-001 exists to catch. It is locked, alongside the other
infrastructure endpoints: the host it names is folded into the CSP once at boot
(routingCspOrigins), so a customer who repoints it lands in a policy that does
not know their router and the browser blocks the requests with nothing to
catch. MANAGED-KEYS-005 pins the list verbatim and is updated with it.
2026-09-07 18:36:54 +02:00
Maurice e11ae6f056 feat(roadtrip): shape the drive — vias, alternatives and stops along the way
The road-trip addon's second half: the parts that let a planned drive be
argued with rather than only read.

Reshaping. A via records which stop it follows by position, and that position
is resolved fresh every time the day is routed — so inserting, reordering or
removing a stop moved the ground under every via after it, and the drive
silently snapped back to the road it had been steered away from while the
handle still sat on the map where it was left. The correction is computed
against the day as it stands and written with the same change, in one
transaction: a half-applied re-anchoring mixes two numbering schemes in one
day. The one judgement that needs geometry — which half of a split leg a via
belongs to when a stop lands in the middle of it — is measured on the road
actually driven, so a fuel stop added to a bent leg keeps the bend on the side
it belongs to.

Alternatives. OSRM answers most long legs with exactly one route, which read as
"there is no other way" when a perfectly good slower road existed. When it
offers nothing, the same drive is asked for again without the motorway, and
failing that without the tolls — one question at a time, stopping as soon as
one lands. The offer says what it is rather than how much slower.

Stops along the way. Everything the corridor search can find is a pause, not a
destination: fuel, charging, rest areas, campsites, food and sights all sit
inside the leg they were found on, carry their own colour and icon, and are
left out of every stop count. That colour is defined once and read by the rail,
the corridor list, the map pin and the picker — and it replaced the chains'
logos, which turned a corridor full of petrol stations into a row of
advertisements and left the brands with no logo on file looking like something
else entirely.

How long a stop takes was readable in the rail and writable nowhere; new places
were given an invented hour that nobody had entered. It is now set from the
rail, cleared as a zero because the update statement folds a null into "leave
it alone", and new places start with nothing.

Both columns are rebuilt on one type scale, anchored to the day plan's own
sizes so switching tabs does not change the size of the text beside it. The
rail reads as trip totals, day cards and a chain of stops with the drives
between them; the corridor column as what to look for, how the search is
getting on, and what it found. Six category pills over three lines became one
dropdown, which is the space the results wanted.

Also: the offered routes are framed on the map when the picker opens (`fitKey`
carries no coordinates and each renderer reads it as "the selected day"), road
trip mode does not cluster at all so the whole drive stays readable zoomed out,
and the drive-time labels finally render in the app's own typeface instead of
Leaflet's Helvetica. All of it in both renderers.
2026-09-07 18:36:54 +02:00
Maurice c59425947a fix(server): unicode-safe content-disposition for gpx, ics, feed and wallet downloads (#2165) 2026-09-01 17:18:10 +02:00
Maurice fd6381c814 feat(api): add a versioned read-only public API for integrations
TREK's REST surface is internal: it answers a session cookie, carries no
version, and changes with the client. That is fine for our own frontend and
useless for anyone building against us — the Dawarich integration people have
been asking for since April needs something it can rely on not moving.

So this adds a small one rather than opening the existing 549 endpoints:

  GET /api/v1/trips           the trips the caller can reach
  GET /api/v1/trips/:id       one trip, ?include= picks the sections

Auth reuses the trek_ tokens users already manage under Settings, via Bearer
or X-API-Key. No new credential type, no new table — the guard calls the same
hash lookup the MCP transport does. It deliberately refuses a session JWT and,
for now, OAuth bearers: a credential whose scopes this surface does not yet
interpret is worse than one it rejects outright.

Access is decided per trip against canAccessTrip, the same owner-or-member
predicate as everywhere else, and a trip the caller may not read answers 404
identically to one that does not exist — a 403 would confirm the id is real and
turn the endpoint into a way to count someone else's trips.

The payload is not the rows. Ids, foreign keys and order_index stay inside;
accommodation day ids resolve to ISO dates, places arrive in planned order, and
every dated thing carries its date so a consumer can join on days rather than
on ids it has no way to look up.
2026-08-28 15:23:05 +02:00
Maurice 1f66ef1d32 fix(maps): let users and admins enter a CARTO API key (#2060)
CARTO stamps an "API KEY REQUIRED" watermark onto every basemap tile
fetched without a key since 26.08.2026, and both CartoDB presets plus the
built-in fallback are CARTO. The key is free and needs no account, so add
it as a setting rather than dropping the provider.

carto_api_key sits next to mapbox_access_token: per user in Settings/Map,
instance-wide under Admin/User Defaults, encrypted at rest, injected from
CARTO_API_KEY on managed instances. Unlike Mapbox it never switches the
map provider, since a raster key says nothing about the renderer.

The key is appended as ?key= at render time instead of being stored in the
template, so a saved URL stays portable and survives a rotation. That is
also what reaches the four maps that pinned CARTO and ignored the user's
tile setting: atlas, collections, the public trip share and the public
journey. The two share pages read it from their token-gated payload, the
same way baseCurrency already travels.

Also lifts the release notice above page chrome. Its overlay sat at
z-index 50, under the Vacay mode toolbar at 61, so the dialog was neither
on top nor modal. Adds a six-step layer scale to index.css and moves the
three notice shells onto it.

Closes #2054
Closes #2052
2026-08-27 12:06:55 +02:00
Maurice 62b13a278e refactor(common): one shared comparator instead of three inline copies
The coverage ratchet caught something worth keeping: an inline
`(a, b) => (a < b ? -1 : a > b ? 1 : 0)` is a new function with four branches at
every call site, and two of the three in src/nest/common sit on error paths that
a passing test never walks. Three restatements of the default sort order were
enough to pull the domain under its pinned thresholds (functions 97.8% against
99, branches 87.92% against 89).

Writing tests to walk those error paths would have been solving the wrong
problem. byCodeUnit says the same thing once, is covered once, and the callers
read better for it.

Its test asserts equality with a bare `.sort()` rather than hand-written
expectations, since "identical to the default order" is the whole contract. It
also pins what byCodeUnit is NOT: `['Zelt', 'Übernachtung']` keeps that order,
where localeCompare swaps it — so the next person reaching for a comparator on a
list a human reads picks the other one.
2026-08-25 19:26:38 +02:00
Maurice 1ad2e92142 fix: make the app keyboard-reachable, and close four DoS-able patterns
The bulk of what SonarQube still had open on the reliability side, read site by
site rather than swept. Two clusters are worth calling out because they are
defects, not lint.

**Four regexes could be made to hang the process.** The worst is
`/.+@.+\..+/` in the forgot-password path: three unbounded `.+` that all
overlap, unanchored, on an endpoint that needs no login and whose schema is a
bare z.string() under a 100kb body cap. A thousand '@' characters had not
finished after 100 seconds — one request pins the event loop, and Node has one.
EMAIL_REGEX in auth.helpers is the same shape on register and profile update
(310ms for 8kb). The two tag strippers over an uploaded KML description and over
Commons metadata took 0.52s on a 16k run of '<'. Each rewrite was proven
equivalent by brute force against the original — millions of strings for the
coordinate patterns, 671,876 for the tag stripper — because a subtly different
regex on a parsing path is worse than the finding.

`<[^>]+>` could not be fixed in the pattern: `[^>]` matches `<` too, so the only
unambiguous form changes what happens to malformed markup. It is now a scan in
common/stripHtmlTags.ts, quirks included — `<>` is left alone, `<a<b>` is one
tag — with the equivalence pinned by a test that compares against the old regex
over every string up to length six.

**And roughly 500 places could only be reached with a mouse.** Clickable divs
became real buttons where the markup allowed (52 of them), took
role + tabIndex + Enter/Space where they had to stay divs, and modal backdrops
took role="presentation" — the honest role, since a backdrop is a mouse shortcut
for a Cancel control that is already in the tab order, and making it a button
plants a nameless full-screen tab stop around its own contents.

Alongside those: `.sort()` calls that now say the order they always had, `{n &&
<X/>}` guards that rendered a literal "0" for a zero-byte file and for a place at
latitude 0, an optimistic budget write that stored a boolean where the contract
and both transports use 0/1, and the Number statics.

sonar-project.properties documents the six that were deliberately left: two wiki
patterns with no overlapping quantifier pair whose "fix" would stop parsing
`[see [1]](url)`, four tar header fields bounded at 12-155 bytes by the format,
two djb2 hashes where the 32-bit wrap is the algorithm, two `| 0` that are also
coercing undefined to 0 on a path where the API sends no user_id, and the MCP
session maps Sonar reads as empty because they are filled from another module.
2026-08-25 19:26:38 +02:00
Maurice 52d1a0720f fix: say out loud what every sort was already doing
Sonar's S2871 flags a bare .sort(), which orders by UTF-16 code unit rather than
by value. All 30 remaining sites turned out to sort ASCII strings — ISO dates,
currency and country codes, setting keys, route names, OAuth scopes, zip entry
names — where that order is already the intended one. Each says so explicitly
now, with a comparator that reproduces it exactly rather than a localeCompare
that would quietly reorder things.

That mattered more than it sounds. Several of these feed something other than a
display: useAtlas joins the sorted codes into the signature that decides whether
to rebuild the region layer, daysSlice re-pins sorted dates positionally onto
reordered days, and three timeline views pick a day's colour by indexOf into the
sorted array. Switching any of them to a locale collator would have been a silent
regression.

Two smaller ones came with it. The .map(placeBtn) calls in FileManagerAssignModal
pass the index as a second argument, which placeBtn declares and uses for its
React key — correct already, now written out as .map((p, i) => placeBtn(p, i)).

And budgetSlice's optimistic paid-toggle stored the raw boolean behind
`paid as unknown as number`, while the schema declares z.number() and both the
REST response and the WebSocket event carry 0/1. So the optimistic state and the
confirmed state disagreed on type until a refresh. It normalises now, and the two
tests that asserted `true` assert the contract.

Four `| 0` truncations are listed in sonar-project.properties instead of changed.
Two are a djb2 string hash where the 32-bit wrap is the algorithm. The other two
look like plain truncation of a row id, but `| 0` is also coercing undefined to
0 there — the settlement flows the API returns carry no user_id, so that is the
normal path, and Math.trunc returns NaN and crashes the lookup.
2026-08-25 19:26:38 +02:00
Maurice 23196aa038 refactor(demo): keep the demo password in one place
It was written out twice — once in the seeder that creates the account, once
in the config payload the login screen reads it from. The two have to match
exactly or the published demo credentials stop working, which is a poor thing
to leave to copy-paste. It moves next to DEMO_EMAIL_PRIMARY, which the same
comment block already argues for.

Also clears a Sonar S2068 "hard-coded password" on the auth service: the value
is public on purpose, but a literal next to a `demo_password` key is exactly
what the rule looks for.
2026-08-25 11:02:39 +02:00
Maurice 38f3cb3101 fix(server): the second pass over this branch's own fixes
An adversarial review of what had already landed here turned up three that
were wrong, and they are the reason for this commit.

The CSP change removed 'unsafe-eval', which is not leftover: heic-to's
libheif build initialises embind through new Function(), and without it every
iPhone .heic upload fails in the browser. 'wasm-unsafe-eval' alone was tried
once already and was not enough (93b51a0b). Put back, with the reason written
down this time and the test pinning it rather than its absence.

The idempotency in-flight guard never fired. finalize() runs when the
handler's observable completes, but Nest writes the response - and therefore
stores the row - several microtasks later, so the waiting request always woke
up before the row existed and ran the handler a second time. It releases from
the point the row is written now; finalize is the backstop for handlers that
answer another way. The test drove the two in the opposite order from Nest,
which is why it passed either way.

And the trip-scoping check treated an id that resolves to nothing like one
belonging to someone else. accommodation_id has no foreign key, so shortening
a trip's dates leaves a booking pointing at a gap - which update() has always
healed by clearing the link, and which would have become a permanent 400.

The rest: a reply to a soft-deleted chat message is refused and the create
path blanks the quote like the list path does; the ws-token buckets are keyed
by account rather than by IP, where every user behind one NAT shared a
ceiling; the weather and vacay caps read through the stream, since a chunked
response has no content-length to check; the Google host gate takes country
domains again; abandoned response bodies are cancelled instead of left to GC;
the boot failure prints its errno; and a share link that is not yours to read
now says so instead of reporting that the journey has none.

TICKETJSON gets a real repair: the in-place GLOB fix only helps installs that
have not run it yet, so an appended migration puts back the note text the
case-insensitive LIKE cut off, for rows where what it stored is not valid
JSON. A genuine receipt always parses, so it cannot be touched.

Wiki: the audit log records the IP Express resolves after TRUST_PROXY, not
the leftmost X-Forwarded-For entry a caller can write. That is the right way
round, but it means the hop count has to be accurate - said plainly in both
places now.
2026-08-23 11:59:12 +02:00
Maurice 8e5fdaa877 fix(server): tighten the guard, audit and token paths
Permissions cached an empty map before it had loaded, so a DB error during
the load cached permissive defaults and every later check answered from that
cache - fail-open, and it never expired. The rate limiter grew its per-IP map
forever; the GC the legacy version had was dropped in the fold. Idempotency
did a check-then-insert, so two concurrent duplicates both ran the handler.

Audit rows recorded whatever X-Forwarded-For said, which the client writes,
and the request-body debug logger redacted by exact key match - it missed
refresh tokens, api keys and PKCE verifiers. FORCE_HTTPS built its redirect
from the client's Host header. CSP asked for unsafe-eval on top of the
wasm-unsafe-eval it actually needs.

The OAuth deny path redirected to an unvalidated URI, ws-token minting had
no rate limit at all, and PKCE was skipped when code_verifier was absent
rather than refused. An MCP session resumed with a token whose scopes had
since been reduced kept the capability set it was created with, and the
session cap was checked before the async registration that increments it.

Plugin frame containment was lexical, so a symlink out of the directory was
never noticed, and instance config accepted keys the manifest never
declared.
2026-08-23 11:59:12 +02:00
Maurice b163c1e05d fix(server): hold DELETE bodies to the same contract as every other write
The body-contract boot gate only looked at POST/PUT/PATCH, so a DELETE that
reads a body slipped past it. Two do; one was already typed, the other read
photo_id straight off an untyped Record. Give it a contract from shared like
everything else, then add DELETE to the gate so the next one cannot repeat it.
2026-08-23 11:59:12 +02:00
Maurice 3f8a7a1ea9 fix(feeds): keep subscribable calendars working under default-deny
The ICS feed routes are documented as public - the secret token in the URL
is the credential - but nothing in the code said so, and the guard this
release makes global is default-deny. Every calendar client subscribed to
/api/feed/trip/<token>.ics would have started getting 401 on upgrade, and
a calendar app that stops fetching just goes quiet.

The e2e suite asserted anonymous access the whole time. It booted
[DatabaseModule, RealtimeModule, FeedsModule], and the APP_GUARDs live in
AppModule, so it was asserting against an app that had no opinion on
anonymity. It boots the real guard stack now, and fails without the fix.

The boot ratchet could not have caught this either: it only looked at routes
carrying @Public(), and GlobalAuthGuard stands down for ANY declared chain -
so a chain of guards that refuse a caller but never resolve one answers
strangers, invisibly. Split the inventory in two and write that half down as
well. It turns up two routes nobody had listed: the passkey login ceremony,
which genuinely runs before a session exists.
2026-08-23 11:59:12 +02:00
jubnl 305b113d5f Feat/storage driver (#1949)
* feat(storage): storage key validation, driver contract suite, LocalDriver

* feat(storage): MirrorDriver composite backend

* feat(server): add TREK_UPLOADS_DIR env var for the uploads root

* feat(storage): settings-backed registry, StorageService facade, module wiring

* test(storage): pin src/nest/storage coverage thresholds

* feat(storage): multer upload options factory

* refactor(files): spool trip-file uploads through storage

* refactor(collab): spool note attachments through storage; normalize files/ prefix

* refactor(journey): spool journey media uploads through storage

* refactor(trips): spool trip cover upload through storage

* refactor(places): spool place-image and collection uploads through storage

* refactor(auth): spool avatar upload through storage

* fix(server): consolidate allowed_file_types onto AllowedFileTypesService

* fix(storage): default defParamCharset to utf8 in the upload factory

* test(db): re-point the trailing-migration guard at the trip_files prefix strip

* fix(storage): age-gate the boot spool sweep

* test(files): key FILE-P02 residue checks on upload size

* test(storage): raise src/nest/storage coverage pin

* refactor(platform): serve /uploads routes through storage

* refactor(files): serve trip-file downloads through storage

* refactor(memories): serve photo bytes through storage

* refactor(maps): stream place-photo bytes through storage

* refactor(share): stream shared place-photo bytes through storage

* refactor(journey): serve public journey assets through storage

* fix(share): answer place-photo cache miss with 204 like the maps proxy

* test(server): close serving-swap gaps surfaced by the full tier

* test(storage): raise src/nest/storage branches pin

* refactor(place-photos): route place-photo cache through storage

* refactor(places): reclaim place images and collection covers through storage

* refactor(trips): trip covers through storage

* refactor(auth): avatar reclaim through storage

* refactor(collab): note-file deletion through storage

* refactor(files): trash deletion through storage

* refactor(memories): trek photo cache internals through storage

* refactor(memories): thumbnail service category-addressed

* refactor(memories): immich upload reads through storage; drop uploads-root

* refactor(journey): photo deletion through storage

* fix(memories): reclaim orphaned trek photo cache objects in the ttl sweep

* feat(journey): journey-thumbs orphan sweep and delete-site derived-artifact reclaim

* test(server): close slice-4 gaps surfaced by the full tier

* test(coverage): raise slice-4 domain pins

* refactor(backup): list, existence and delete through storage

* refactor(backup): download through storage sendToResponse

* refactor(backup): create backup through per-category storage enumeration

* refactor(backup): restore and upload-restore through storage

* refactor(backup): auto-backup retention through storage

* refactor(server): boot dir-creation to driver init

* test(server): close slice-5 gaps surfaced by the coverage tier

* test(coverage): raise slice-5 backup pin

* test(server): resolve async backupFileExists mocks for the tests typecheck

* test(server): pin ranged-read stat and zero-byte roundtrip in the storage driver contract

* feat(server): TREK_S3_* env wiring — schema, s3 namespace, all-or-nothing boot rule

* feat(server): S3 driver core — aws-lite client seam, stat/delete, driver-owned deadlines

* refactor(server): type the aws-lite S3 client via s3-types reference

* feat(server): S3 driver getStream and list — ranged full-object stat, iterator pagination

* feat(server): S3 driver put — bounded-peek routing, Upload inactivity deadline

* fix(server): S3 driver Upload monitoring — Transform interposition, stream cleanup

* feat(server): env-declared s3-main storage backend; settings-declared s3 rejected

* test(server): env-gated S3 driver contract suite against MinIO

* fix(server): S3 driver zero-byte put — work around aws4 duplicate Content-Length signing defect

* fix(server): scope the zero-byte put workaround to buffered bodies; bind test MinIO to localhost

* ci(server): MinIO-backed S3 driver contract job

* ci(server): align s3-contract shim comment and MinIO bind with references

* docs: S3 storage backend configuration and off-box backup mirroring

* docs: restore README formatting, keep S3 additions only

* ci(server): make live-MinIO contract runs manual-only (AIStor licensing)

MinIO's community line is discontinued entirely; the maintained successor, AIStor, requires a license key that can't safely live in a shared CI runner. The s3-contract CI job is removed; live-provider contract runs against MinIO/AIStor are now a manual, license-gated procedure documented in docker-compose.minio-test.yml's header.

* chore(server): final-review polish — stale CI refs, dockerignore key, boundary pins, zero-byte tempfile contract case

* expose admin ui

* feat(shared): storage backend type registry, wire schemas, category relocation

* feat(server): accept settings-declared s3 backends — shared-schema options, decrypt-at-build, registry preview()

* feat(server): remove TREK_S3_*/TREK_UPLOADS_DIR — tombstone boot rule, built-in override as the relocation path

* feat(server): deriveSecurity.encryptionKeySet and storage secret plumbing (gate, encrypt, unmask, mask, redact)

* chore(server): drop storage-secrets import that only Task 8's probe helper will use

* feat(server): seed-once storage-config.json boot import — fail closed on any invalid seed

* feat(server): registry snapshot() — effective storage config with provenance

* feat(server): api/admin/storage GET/PUT — masked secrets, preview-gated saves, wired reload, audits

* feat(server): api/admin/storage/test — per-target ephemeral probes, mirror targets probed individually

* test(server): e2e coverage for api/admin/storage — guards, managed refusal, envelope, probes

* test(server): give e2e harness DBs the app_settings table the storage registry reads at boot

* test(coverage): raise pins after the storage admin-config slice

* fix(server): final-review polish — encryption-gate wording, all-field sentinel guard, README/e2e notes

* test(server): storage slice-1 review follow-ups — mirror s3-replica probe decrypt, all-encrypted seed without key, test-route 401

* feat(i18n): storage admin domain + tab label in every locale

* feat(client): adminApi storage endpoints — getStorage, updateStorage, testStorageBackend

* feat(client): storage admin draft-document helpers + relative-time util

* feat(client): schema-driven storage BackendForm — fields by kind, mask echo, encryption banner

* feat(client): admin storage panel — backends list, category map, health strip, one-PUT save

* feat(client): storage tab on the admin page

* feat(client): storage admin panel and section on the phone shell

* fix(client): block renaming a storage backend onto an existing name

* feat(server): allow mirrors on any storage category — lift the backups-only rule

* feat(i18n): replicas-on-primary mirror strings in every locale

* feat(client): mirror fold/synthesize model helpers for replicas-on-primary

* feat(client): BackendForm mirror-targets composer; raw mirror type hidden

* feat(client): desktop storage panel folds mirrors into their primaries

* feat(client): phone storage panel folds mirrors into their primaries

* fix(client): strip removed backends from remaining mirror replicas — no phantom refs

* feat(storage): retire photos as a configurable category — served-only via the ServedCategory seam

* feat(i18n): descriptive storage category names and descriptions in every locale

* feat(client): descriptive category names, descriptions and id badges in the storage panel

* docs(storage): fix stale nine-category comments after the eight-category shrink

* docs: storage configuration via the admin UI and seed file — README + compose

* docs(wiki): Admin Storage page and storage cross-links

* docs(server): storage domain blueprint entry; TREK_PLACE_PHOTO_DIR tombstone clarifier

* refactor(server): drop the storage env-var tombstones — the variables never shipped

* docs: storage config is simply new in v4 — drop the removed-variables story

* feat(shared): backfill-status and usage wire schemas on the storage admin state

* feat(server): MirrorDriver.backfill — prefix-scoped, size-compared, cancellable replica catch-up

* feat(server): one-at-a-time storage backfill jobs + the storage events bridge

* feat(server): storage usage scan — nightly cron, on-demand service, persisted with computedAt

* feat(server): replica_failure admin notifications — events bridge, hourly debounce, channel registration

* feat(i18n): replica-failure notification text in every locale

* feat(server): storage backfill and stats-refresh admin endpoints

* feat(client): adminApi backfill and stats-refresh endpoints

* feat(i18n): storage sync and usage strings in every locale

* feat(client): desktop storage panel — sync now, usage display, backfill polling

* feat(client): phone storage panel — sync now, usage display, backfill polling

* fix(client): per-category usage line on the phone storage panel

* docs: sync now, usage stats and replica-failure notifications

* test(server): admin scope now includes replica_failure, not just version_available

Commit 6db135da added replica_failure to ADMIN_SCOPED_EVENTS but left
NPREF-005b asserting the admin scope contained only version_available.
Update the expectation to match.

* test(server): cover storage backfill/stats/jobs branches to hold the coverage pin

* fix(client): guard storage admin polling against stale responses racing a save

A poll GET issued before a save's PUT resolves could land after the save
applied its fresher response, overwriting it with the pre-save world and
re-deriving the draft from that stale snapshot. useStorageAdmin now tracks
a state-application sequence number and drops any refreshState() response
whose sequence has moved on, or that resolves while a save is in flight.
Also guard the fire-and-forget refreshState() calls in startBackfill/
cancelBackfill against unhandled rejections, matching the poll's style.

* fix(server): report 404 before 409 when a backfill start names an unknown mirror

startBackfill checked the global busy flag before validating that :name is
a mirror routed by some category, so POST .../backends/garbage/backfill
answered 409 instead of 404 whenever a sync was already running. Resolve
and validate the target first (BackfillTargetError -> 404), then check
busy (BackfillBusyError -> 409).

* feat(client): two-column category grid on the storage panel

* feat(server): plugin files.getContent reads through the storage layer

* feat(server): plugin files.create writes through the storage layer

* refactor(server): drop FilesService.resolveFilePath, last direct-fs seam in the plugin surface

* fix(server): map StorageInvalidKeyError to the plugin file accessibility envelope

* feat(storage): allow credentialed backends without an explicit ENCRYPTION_KEY

Secrets are still always encrypted at rest; without an explicit key the
implicit data-directory key is used. Removes the plaintext-secret refusal
from the admin PUT and the seed import, the encryptionReady GET field, the
save-blocking banner in both admin shells, and the now-consumerless
security.encryptionKeySet env derivation. Docs now recommend (not require)
an explicit key, since the implicit key rides inside backup archives.

* test(client): pin the stale-poll release to the pre-save GET in FE-ADMIN-STOR-024

The stub deferred EVERY poll GET and rebound the release handle to the
newest one, so a tick landing after the save was released with the
pre-save payload — a response no real backend can produce, which the seq
guard rightly applied. Only the first (genuinely stale) GET is held now;
later GETs answer with the world the mock server currently holds. Also
drops the encryptionReady fixture field removed from the schema in the
previous commit.

* fix(storage): backfill puts carry content-type, stat failures report op stat, progress clamps

* feat(storage): category migration job — copy, flip, delta sweep, reclaimable tally

* feat(storage): migration start/cancel endpoints and client API

* test(client): storage admin fixtures carry the new migrations field

* fix(storage): migration POST answers 200 and validates its body via the shared schema

* fix(storage): cancel running storage jobs when a config save removes their backend

* fix(notifications): in-app replica-failure row carries the suppressed count

* feat(client): migration prompt, queue, and progress on the storage panel

* fix(client): migration queue holds a synchronous single-flight lock

* feat(client): migration prompt and progress on the mobile storage panel

* fix(server): register category-migration routes in the managed-route allow list

StorageAdminController.migrationStart/migrationCancel were added under the
controller-level @ManagedForbidden() decorator but never added to
MANAGED_ROUTE_ALLOW_LIST, so validateManagedRoutes() threw "Managed surface
changed" on every buildApp() call, failing 46 test files (any test booting
the real app).

* test(server): cover category-migration copy/sweep branches to hold the storage coverage pin

Adds MIG-005 (a target object already matching size/key is skipped in the
copy phase itself, not just the delta sweep) and MIG-006 (a copy failure
during the delta sweep is counted without undoing the flip that already
happened). src/nest/storage/**'s branch coverage was 83.91%, just under the
84% pin.

* docs(storage): category migration — move existing objects on reassign

* fix(storage): close the pre-flip cancel window and cancel re-routed migrations

A cancel landing between the last copy iteration and the category flip was
silently ignored, and a config save that re-routed the migrating category to
a third, still-defined backend never stopped the in-flight migration. Also
guards assignCategory against writing an unknown backend name, and fixes a
missing `suppressed` notification param being treated as truthy suppression.

* fix(client): migrations keep mirrors, queue waits on backfills, sweep failures surface

Reassigning a category onto a mirrored primary now posts the migration to the
mirror's wire name instead of the bare primary, so replication survives the
move. The queued-migration effect also waits out a running backfill (the
server's one-job rule spans both), and a done migration with sweep failures
now surfaces a doneFailures line in both shells, translated into every locale.

* ci(server): run the S3 contract suite against MinIO

Resurrects the s3-contract job removed in 6fbff835, adapted for AIStor:
writes the MINIO_LICENSE_KEY secret to a license file, boots the licensed
container, and runs the contract suite hermetically (no --coverage) so the
storage coverage pin keeps holding from unit tests alone. Gated to same-repo
PRs since fork PRs have no access to the license secret.

* fix(server): backup archives uploads from a remote primary

createBackup() enumerated uploads through withLocalFile, which downloads a
remote object to a temp file, hands it to the callback, then deletes it
before archiver ever reads it — archiver reads entries lazily during
finalize(), so a remote (S3) primary silently backed up zero upload bytes.

Add StorageService.getLocalPathOrNull(category, name): a narrow locality
probe that returns a real path only when one exists on disk right now (the
fs-existence check is the fail-safe half of the contract — a driver-reported
path that vanished between list() and here falls back too, not just a
path-less remote driver). The enumeration loop uses it directly when
available (zero-copy, unchanged for a local primary) and otherwise streams
the object into a per-backup staging dir under the backups spool, cleaned up
in the existing finally alongside zipSpool/dbSnap.

Also wire archive.on('warning', reject): archiver treats a missing/unreadable
entry as a warning, not an error, and skips it by default — which would
silently drop the entry from an otherwise "successful" backup.

* fix(server): client aborts stop 500ing and leaking streams

storage.service.ts sendToResponse: the local res.sendFile branch now treats
ECONNABORTED/ECONNRESET/ERR_STREAM_PREMATURE_CLOSE/syscall=write as success,
mirroring res.sendFile's own default callback (the contract
storageStaticHandler already implements caller-side). The remote branch
replaces the hand-rolled pipe+Promise with pipeline(stream, res), which
destroys the source stream on any failure — fixing the leaked S3 keep-alive
socket on a client abort — while swallowing the same client-abort error set;
a real source error still rejects so callers keep their miss contracts.

This centralizes the fix for every caller that streams through
sendToResponse: files-download, journey-public, photo-resolver,
trek-photo-cache, backup downloads, and /uploads/photos.

trek-exception.filter.ts now no-ops (destroying the socket) when headers are
already sent, so no filter path ever writes onto a response in flight.

maps.controller.ts and share.controller.ts's place-photo bytes proxies
switch their direct getStream+pipe to pipeline() with the same abort-tolerant
catch, keeping the existing 204-empty fallback for pre-header failures.

* fix(storage): mirror read fallback consults replicas on primary miss

MirrorDriver.getStream/stat now fall back to replicas on a primary miss
(StorageNotFoundError / null stat), not just an infra error — binding
decision: a read succeeds if ANY member holds the object. Only a
deterministic StorageInvalidKeyError skips fallback.

stat() gets a dedicated fallback loop instead of reusing fromReplicas:
a replica returning null means "keep looking," which the generic
catch-and-continue helper would have short-circuited on.

StorageService.sendToResponse/withLocalFile stop treating composite
driver.stat() as proof the object is at localPath; both now fs-check
the path itself and fall through to the stream branch when it's
missing, which is what actually reaches a MirrorDriver's replicas.

* feat(storage): remote serving parity — content-type, validators, range

Local backends get content type, ETag/Last-Modified, conditional GET, Range
and HEAD for free from res.sendFile; the proxying branch every remote
backend (and a local primary whose file vanished) falls through to had none
of it — every request re-streamed the whole object with no cache validator
and no way to seek.

sendStreamed() reproduces the contract from ONE pre-flight driver.stat():
a miss throws before any header is set (the caller's 404/204/next()
contract), and that same stat then answers 304s, 416s, HEADs and suffix
ranges without ever opening a body. The decisions live in a new pure module,
http-serving.ts:

- validatorsFor: the driver's ETag, else express/send's weak
  W/"<size hex>-<mtime hex>". mtimeMs 0 (S3 can report it) means no
  Last-Modified and no derived tag — serving uncached beats a 1970 header
  that reads as fresh forever.
- isNotModified: If-None-Match (weak compare, *, list) wins over
  If-Modified-Since, per RFC 9110 §13.2.2.
- decideRange: single range only, inclusive clamp, suffix computed from the
  pre-stat size, unsatisfiable → 416, If-Range strong-exact or full entity.
  A multi-range or malformed spec is answered with the full object rather
  than multipart/byteranges. If-Match/If-Unmodified-Since stay out of scope.

Range math uses the pre-stat size as the single source of truth, the ranged
getStream stat is never re-consulted, and the pipeline abort semantics are
unchanged.

S3Driver now surfaces the bucket's ETag on stat()/getStream() (on a ranged
GET it is still the full object's tag). aws-lite strips the tag's
surrounding quotes — verified live against AIStor RELEASE.2026-08-07 — and
an unquoted entity-tag is not a valid HTTP ETag, so the driver restores
exactly the quotes the SDK removed and nothing else.

* fix(server): key rotation covers storage.backends

The S3 storage backend's secretAccessKey is encrypted at rest inside the
app_settings row keyed storage.backends, but scripts/migrate-encryption.ts
didn't know about it — after a key rotation that secret stayed encrypted
under the old key and silently failed to decrypt on the next read.

The script now re-encrypts secretAccessKey for every s3 backend inside the
storage.backends JSON array, using a pinned copy of the secret-field list
(it deliberately can't import @trek/shared). The parity test pins that copy
against storageSecretFields('s3') so a new secret field can't be missed
silently.

* fix(server): restore reloads the storage registry

restoreFromZip closed and reopened the DB but never reloaded the
storage registry, so uploads rehydration ran against the STALE
pre-restore driver/category map instead of the config the restored
archive's own travel.db carries (audit #4, major).

StorageService gains reloadConfig(), a narrow passthrough to the
(unexported) registry's reload() — same shape as health(). backup.impl's
restoreFromZip now calls it right after reinitialize() and right before
the uploads wipe/rehydrateUploads/plugin staging, so rehydrated bytes
land where the restored config says. When reopen fails (reinitFailed),
reload and rehydration are skipped entirely — already reported as
"restart required", and there is no live DB handle for the registry to
read anyway.

* fix(server): category migration rewrites keys across prefix changes

Category migration copied objects to the destination backend using the
SOURCE key verbatim. A mode-A ('' prefix) source migrating onto a
prefixed backend (or vice versa) landed objects at keys the registry
could never resolve post-flip (audit #8).

Extracts StorageRegistryService.keyPrefixFor(category, backendName) from
build()'s inline conditional so startMigration can compute the
destination prefix before the flip. runMigration now maps every
destination key (destPrefix + key.slice(sourcePrefix.length)) in the
copy skip-check, the put, and the delta sweep; the reclaimable tally
stays source-keyed. Equal-prefix migrations are unaffected (identity
mapping).

* fix(plugins): await the now-async place/collection services in RPCs

PlacesService.update/remove and CollectionsService.deletePlace became
async when they started deleting storage objects, but their plugin RPC
handlers still called them synchronously: the null/false refusal check
compared against an always-truthy Promise, and places.update broadcast
a Promise instead of the resolved place. Both handlers now await the
service call; mapCollectionError is async-aware (accepts a sync-or-
async thunk) so the sync callers (create/update/savePlace/copyToTrip)
keep working through the same path. Verified the RPC host already
awaits handler results, so no change was needed there.

* fix(storage): backfill Sync now gains a deletion sweep

The Sync now backfill copied primary objects to replicas but never removed
replica objects the primary no longer has, leaving mirror read-fallback able
to serve stale data indefinitely after a failed replica delete. backfill()
now sweeps each replica post-copy: any replica key absent from the primary
key-set is re-checked against primary.stat (a race gate against the primary
regaining the key) before being deleted. Adds a deleted counter end-to-end
(driver, jobs service, shared schema, both admin shells, all 23 locales) and
a new ReplicaFailure 'list' op for a replica whose listing itself fails
during the sweep.

* feat(storage): optimistic concurrency for admin config saves

A stale admin PUT of storage settings could silently undo a category
migration's just-flipped assignment. Adds a version counter
(app_settings row storage.config_version) bumped inside the same
transaction by both writers — applyConfig and the migration flip
(assignCategory). applyConfig compares the submitted version first
and throws StorageConflictError, mapped to 409 ahead of the blanket
400. The client carries the draft's loaded version through every
local edit and maps a 409 to a distinct "settings changed underneath
you" message, refreshing state while preserving the dirty draft for
review.

* fix(storage): surface a failed config load + redact secretAccessKey

Registry load() now records the exact parse/validate failure as
lastLoadError() instead of silently falling back to last-good config or
built-in defaults; StorageAdminService exposes it as configError on the
admin state (storageAdminStateSchema extended additively), and
AdminStoragePanel renders it as a warning banner naming the error. Save
stays enabled — it's the recovery path — but an admin can no longer save
over a broken stored config without knowing.

Also adds 'secretaccesskey' to globalMiddleware's SENSITIVE_KEYS request-log
redaction list, so storage admin PUT bodies never leak the S3 secret into
debug logs.

* fix(client): admin storage panel state races

Migration prompt now recomputes computeMigrationCandidates(draft, state)
on render and again at confirm time, so a stale open-time snapshot never
drives stripCategories or the migration queue; adds a Cancel button to
the dialog. A failed startMigration POST explicitly clears the remaining
queue with a toast naming the dropped categories and refreshes state,
instead of leaving it invisibly stranded; a "Queued: X, Y" line renders
while the queue is non-empty. Testing a mirrored row now expands the
draft primary + replicas client-side and probes each individually,
instead of sending the mirror stub to the server (which resolved
targets from its own live snapshot, silently testing saved options over
unsaved draft edits). A completed/cancelled/errored backfill now renders
its terminal status alongside the Sync now button instead of hiding it,
so it can be re-run without a page reload.

* fix(storage): refuse shared replica backends so the sync sweep cannot cross categories

A mirror's "Sync now" sweep makes each replica MATCH the primary: it lists the
replica under every prefix the mirror's categories occupy and deletes the keys
the primary doesn't hold. The backups category's prefix is '' — the replica's
entire root — so making a category-serving backend (say uploads-local) a
replica of a backups mirror meant the first sync deleted every user upload.
Nothing forbade the config: validateConfig only checked that mirror targets
existed and weren't nested, and the admin picker offered every backend.

The sweep is correct, so the fix is at the config level:

- validateConfig refuses a backend that is both a mirror replica and a
  category target (directly, or as the primary of a mirror a category routes
  to), and refuses a backend replicating two mirrors whose swept prefixes
  overlap — '' overlaps everything. Disjoint prefixes stay legal.
- assignCategory (the migration flip, the other writer of the settings rows)
  refuses routing a category onto a backend that is currently a replica.
- The admin mirror-target picker (both shells) only offers backends that serve
  no category; already-selected targets stay listed so they can be unchecked.
  A category assigned to a replica still surfaces the server's 400 verbatim.

Existing shared-replica fixtures were rewritten onto dedicated replicas.

* fix(client): a 409 conflict save is recoverable without a page reload

After a 409 the admin storage panel was a dead end. refreshState leaves a
dirty draft's version alone on purpose (audit #7: the background poll must
never float a mid-edit draft past a migration flip it never carried), setDraft
re-attaches that same version to every later edit, and save() resubmits it —
so every retry 409s again and only a full page reload got the operator out,
while the banner told them to "review and save again".

The version block stays. Instead the conflict banner now offers the one action
that can clear it: discard the local edits and reload the saved settings —
a fresh GET applied through applyState, re-deriving the draft (and its
version) and clearing dirty/saveError. Bumping the version in refreshState,
or only in the 409 branch, was rejected: either resurrects audit #7, since a
full-document save would then revert a flip the draft never carried.

storage.saveConflict now points at that action instead of implying a plain
re-save works (all 23 locales). The escape lives in the shared hook, so the
phone panel renders it too.

* fix(client): deflake the queued-migration-waits-out-backfill panel test

FE-ADMIN-STOR-037 seeds a running backfill, so the panel polls GET from
first render — but the stub served the post-save world (files: off-box)
before the PUT had landed, which no real server would do. A poll
resolving while the confirm dialog was open made the confirm-time
candidate recompute see from === to, so nothing was queued and the
migration never POSTed.

The GET stub now stays faithful (pre-save world until the PUT lands,
usage included so candidates keep their object counts), and the test
deterministically forces the once-flaky interleaving: it waits for
polls to land while the dialog is open and asserts the candidate
survives them.

* fix(storage): surface the underlying cause in probe target errors

The admin storage Test probe reported only the driver wrapper's outer
message ("put failed for '…' on 'probe-s3'"), dropping the cause that
says WHY — connection refused vs missing bucket. Flatten the error's
cause chain into the probe result, appending an S3-style code when it
isn't already in the message.

* fix(storage): surface the underlying cause in replica-failure reports

Same defect as the probe fix, one layer over: MirrorDriver's
reportReplicaFailure captured only the wrapper's message, so
replica-failure notifications said "put failed" without the why.
Move the probe's cause-chain flattener into storage.types.ts as
describeError and use it in both call sites.

* fix(storage): strip ContentType from multipart Upload calls (MalformedXML)

Every multipart S3 put carrying a contentType — every backup zip and
photo over the 10MB threshold — failed with 400 MalformedXML. Upstream
@aws-lite/s3's Upload leaks the ContentType param into its final
CompleteMultipartUpload headers ({ ...xml, ...paramHeaders }), clobbering
application/xml, which flips @aws-lite/client's payload serializer to
JSON — so the server receives a JSON parts list where XML belongs.
Confirmed live against MinIO AIStor; any S3 refuses the same.

uploadMonitored now drops ContentType before s3.Upload. Multipart
objects are stored without a content-type, which TREK never reads back
(serving derives Content-Type from the key/caller, never object
metadata); single-request PutObject paths keep sending it.

Also closes the coverage gap that let this ship: the live contract
suite never crossed MULTIPART_THRESHOLD, so s3.Upload had never run
against a real server — it now round-trips a >10MB put carrying a
contentType.

---------

Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-08-22 11:33:55 +02:00
Maurice 3a0604b3f5 feat(config): put the operator-owned surfaces behind TREK_MANAGED (#1946)
* feat(config): put the operator-owned surfaces behind TREK_MANAGED

The switch landed last release with nothing behind it. This is what it turns
off, plus the four fixes that came out of walking every one of these surfaces.

Nothing here changes anything for an install that does not set the flag. That
is not a hope, it is what managed-inert.test.ts asserts: the mode stays off for
any unparseable value, no other variable can turn it on, every settings key
stays writable, and splitManagedKeys hands back the same object rather than a
copy so no write path can quietly diverge on self-hosted installs only.

WHAT THE MODE WITHHOLDS

Fourteen routes carry @ManagedForbidden: key validation, static MCP tokens, JWT
rotation, the OIDC admin pair, restore and auto-backup, plugin sideload and
reload, the Ollama model list and pull, and the SMTP test send. Each names the
reason it is not the instance admin's, and the reason is mandatory.

Eighteen settings keys are filtered instead of refused. SMTP, the OIDC toggles
and the WebAuthn pair share one handler with keys the admin does own, and the
llm_* family is reachable through per-user settings, the instance defaults and
the addon config alike, so a route-level refusal would be both too coarse and
incomplete. A locked key is skipped and reported as managed_keys; the rest of
the save goes through. Refusing the request would take the registration toggles
down with the SMTP block.

GET /api/auth/me/settings stops returning the three API keys in the clear. It
is the only endpoint that decrypts a stored credential back to the caller, which
is fine when the admin pasted it in and wrong when the operator supplied it.

Two multipart routes check in the handler rather than the guard: a guard throws
before the parser drains the body and the client gets an ECONNRESET instead of
the 403 (PROFILE-015). They still carry the marker so the boot gate sees them.

WHAT KEEPS IT FROM DRIFTING

validate-managed-routes.ts refuses to boot on a marking that is not on the
reviewed list, a list entry whose route lost its marking, or a marking with no
reason. The third case has no equivalent in the guard gate next door and is the
one that matters most: a blank reason destroys the only record of why a control
was taken away, and nothing recovers that later. It found two wrong entries in
my own list while I was writing this.

managed-keys.test.ts asserts that every key in ADMIN_SETTINGS_KEYS and
DEFAULTABLE_USER_SETTING_KEYS belongs to exactly one of two sets. A key added to
either list belongs to neither until somebody decides, and that is the failure
this prevents: the lists grow, the mode does not.

PLACES_API_BASE AND PLACES_API_KEY

The nine Places endpoints spell out googleapis.com, so an install that wants
those calls to leave through something of its own has no way to say so. One
variable, substituted where every call already funnels through. A second one
supplies the key, and when it is set the users table is not consulted at all —
a per-user key would route around whatever the endpoint counts, and those
columns are readable by any admin of the instance.

FOUR FIXES THAT ARE NOT ABOUT THE MODE

An admin setting somebody's password now bumps password_version and clears
their MCP tokens. An admin sets a password for one reason, and until now
verifyJwtAndLoadUser kept accepting every cookie the intruder already had: the
one action taken to lock them out was the one action that did not.

The four OIDC provider calls go through the SSRF guard. Three are worse than
they look — the token, userinfo and JWKS URLs come out of the discovery
response, so they are chosen by the other end, not by an admin. safeFetchLlm is
renamed safeFetchAdminConfigured and keeps its old name as an alias, because it
is the variant that allows loopback and LAN: a self-hoster with Authentik on
localhost:9000 must keep working. The token exchange follows no redirect at all,
since one would hand client_secret to a second host.

DELETE /api/admin/users/:id/mfa clears another account's TOTP. The passkey half
has existed since passkeys landed; the TOTP half never did, which left
"somebody on the trip lost their phone" with no answer short of an operator in
the database. Never for the caller themselves.

TREK_PLUGIN_PERMISSIONS=off refuses to boot when the install is centrally
administered. A warning is right for a machine its owner chose to trust; where
the person who set the flag and the people whose data is in the database are
not the same person, fail closed.

TEXTS

Four strings lose the phrase "self-hosted" because they describe somebody
else's software — an Ollama endpoint, an AirTrail instance, a service a plugin
talks to — and read better without it in both modes. The one string that
describes TREK itself keeps it, and the mode picks a second key instead, so
self-hosters lose nothing.

A grep gate in shared/ keeps the canonical locale honest, because the parity
check compares the file set and the top-level keys and never looks at a value:
re-wording a string is invisible to CI today, and so is putting it back.

* feat(maps): supply the map credential with the instance when centrally administered

MAPBOX_ACCESS_TOKEN joins PLACES_API_KEY as an operator-supplied value.
SettingsService injects it when the settings are read, and picks GL as the
provider if the reader is still on Leaflet, since a token is always present.

The key moves from the customer set to the locked set. It is the one locked key
whose value is public by design — a pk.* token has to reach the browser to be
useful — so this is not about secrecy: a per-user save landing on top of the
instance token would only break the map for that user. Both settings fields for
it go away with it.

* fix(auth): count the operator key when reporting whether Google features work

has_maps_key only looked at the users table, but getMapsKey prefers the
PLACES_API_KEY env over anything stored there. On an install that supplies the
key that way and has an empty column, every lookup worked while the client was
told there was no key and hid the search, the autocomplete and the place photos.

Found by walking the running instance rather than the code: the map came up,
the search box did not.

* feat(admin): hide the settings a centrally administered install cannot act on

A control that answers 403 is worse than no control: the AI section rendered the
refusal as "Request failed with status code 403", which tells an admin they broke
something rather than that the decision is not theirs.

So the surfaces go rather than being disabled in place:

- AI parsing and AirTrail leave the addon list, and the AI addon route now
  refuses outright. The endpoint, the model and what a document costs come with
  the instance, so there is no instance decision left — not even whether it runs.
- Immich and Synology Photos leave it too. Both connect to a server the reader
  runs at home, and there is no route from here to one.
- The SMTP block, the admin webhook and ntfy targets, and the passkey RP ID and
  origins follow the same rule: they configure something the operator owns, and
  a wrong RP ID on a shared parent domain reaches past this instance entirely.
- The About tab goes. It is about the project — what TREK is, where to file a
  bug, where to support it — and is aimed at the person running the software.
  The version and the link to the source move to the sidebar footer, on both the
  desktop page and the mobile twin, because AGPL 13 wants that offer wherever
  people reach the software over a network.
- The update check stops asking GitHub and reports nothing available.

Separately, the version footers lose the "self-hosted" suffix everywhere, not
just here. It read as a label on the software rather than on the install, and
the version alone says what it needs to.

* feat(config): refuse to boot centrally administered without an at-rest key

Measured on a running instance rather than assumed: with ENCRYPTION_KEY unset,
the at-rest key falls back to a file in the data volume, and backupService
bundles that file into every archive it builds. The archive is downloadable by
any instance admin, so the encryption ships with its own key in the same zip.

The skip in backupService already handles this — it only omits the file when the
key came from the env — but nothing made sure the env actually carried one.
Provisioning sets it; forgetting it was silent.

validateEnvAtBoot now carries a second kind of check next to the schema: cross-
field rules that only apply with the flag set, for combinations where every value
is individually valid and the result is wrong. Empty without the flag, so an
install that never asked to be centrally administered cannot meet it.

* feat(client): add an attachment point for install-specific screens

Two empty exports on the client, an empty module on the server, and the four
places that read them. Nothing here adds a route, a nav entry or an endpoint —
the arrays are empty and the module contributes nothing, which managed-ext.test
pins so it stays that way.

Read unconditionally, never behind a flag. On the server a conditional import
would tie the module graph to an environment variable, and the graph is built
once per app while the flag is read live; on the client an empty array costs
nothing to spread, so every build walks the same path instead of one build
walking a path nobody tests.

Anything attached here follows the same rules as the rest of src/nest and
src/components: a real module with a controller and a provider, bodies through
the Zod pipe, routes under the global guards, tests.

* test(shared): move the wording gate out of src so shared typechecks again

CI installs shared with its own dependencies only (npm ci --workspace shared),
so there is no @types/node and the `node:fs` import inside src/ failed to
compile. The tests passed the whole time — vitest supplies the module — which is
why this only showed up in the typecheck step.

The walk moves to scripts/, next to i18n-parity.mjs, and the spec imports it the
same way that one does. It also runs on its own now
(`node scripts/i18n-selfhost-wording.mjs`), which is what you want when CI names
a line you have to go find.

* test(server): repair the call sites typecheck:tests caught

tests/ sits outside the build tsconfig, so none of this showed up in a green
test run — which is exactly the gap `npm run typecheck:tests` exists to close.
Three kinds, all mine:

- Seven `import type` paths for RuntimeEnvService carried one `../` too many.
  Type-only imports are erased, so vitest never resolved them and nothing failed.
- VersionCheckJob and PluginsController grew a constructor parameter and eleven
  hand-wired `new` calls kept passing the old count. The argument arrived as
  undefined and no assertion happened to read it.
- ConditionContext grew `managed`, and the shared context object in the
  conditions suite did not.

The condition also had no test of its own, so it has one now: both directions,
since the registry uses `is: false` to keep a notice away and a notice that only
belongs on a centrally administered install would use `is: true`.
2026-08-15 16:42:27 +02:00
Maurice bfb3bcd63c feat(config): mark an install as centrally administered
Some installs are run by somebody other than the person who administers them.
The operator owns the credentials, the configuration and the upgrade schedule,
and the admin of the instance is a user of the result rather than the one who
set it up. Nothing in the app knows that today, so it keeps offering settings
whose values are not the admin's to change, and the honest answer to half of
them would be a 403 the UI never anticipated.

TREK_MANAGED says it once. It is a single boolean and the surfaces that care
read it themselves, so adding one never comes back to the config layer.

Runtime-toggled, in the same class as DEMO_MODE, rather than a registerAs
token: roughly sixty tests mutate env mid-lifetime, and a value frozen per
built app breaks all of them. `=== true` on the way in, so an unset, empty or
misspelled value means off — a managed install sets it deliberately, and a typo
must not quietly take settings away from an admin who never asked for it.

The guard keys on a decorator, not on a path. MfaPolicyGuard documents where
the two literal path lists it replaced ended up: they stopped being maintained,
and every endpoint added afterwards silently fell on the wrong side. A marker
on the handler cannot drift away from the handler. It carries a mandatory
reason, which is for the next reader of the source rather than for the client,
so the marks stay auditable.

It registers third, after the auth and MFA guards, so a stranger still gets the
401 those two would have given them instead of a 403 that would confirm the
route exists. And it checks the mode before it touches metadata, because it
runs on every request of every install and almost all of them are self-hosted.

One 403 body, shared by REST and MCP. DEMO_MODE grew two texts for the same
condition and neither can be changed now without hunting for the other.

The flag reaches the client through app-config so the UI can stop offering what
the server would refuse anyway, but it is deliberately not persisted the way
demo_mode is: two installs can share a browser profile, and a stale "this one
is managed" would take settings away on an install that never set it.

No route carries the marker yet. The per-domain marks follow. Unset, which is
every self-hosted install, nothing about this changes anything.
2026-08-15 11:45:11 +02:00
Ida Rogalska edb21b681f read: project incoming_leg_transport_mode via rowShape + getAssignmentWithPlace 2026-08-12 09:08:47 +02:00
jubnl 954a3ca0a7 refactor(server): serve /mcp from a Nest controller; delete auth, audit, addons and oauth bridges 2026-08-11 23:01:19 +02:00
jubnl 1390bfbe06 refactor(server): move health, discovery and consent-COOP behind the container 2026-08-11 22:23:51 +02:00
jubnl 8b7b91d814 refactor(server): let user-cleanup inject BudgetService and delete budget.bridge 2026-08-11 16:42:49 +02:00
jubnl de5d196344 refactor(server): move the cache sweeps, idempotency purge and airtrail poll in-container 2026-08-10 22:02:54 +02:00
Maurice ba7d55f999 Finish the contract ratchet, the upload bridge and the client leftovers
BODY_CONTRACT_ALLOW_LIST is empty. The last seventeen handlers type their
body with a createZodDto class, so the fail-closed boot gate has nothing left
to forgive; it still refuses a boot for a new handler that reads @Body()
without one.

Several of those contracts describe rather than constrain, and that is the
whole care in this change. The pipe runs before the handler, so a schema
stricter than the handler moves a user-visible answer:

- TagsController.create answers { error: 'Tag name is required' }, pinned
  byte-for-byte in two tests, and the pipe formats as `field: message`. The
  DTO leaves name optional and the handler keeps the check that owns the
  message. Same shape for the booking-import confirm body, which additionally
  gains per-item validation the handler never did.
- PluginsController.activate and install answer 503 when plugins are disabled,
  and check that BEFORE looking at the body. A schema that rejected first
  would turn that into a 400 for an operator whose server has plugins off.
- setEgressHosts reads a non-array `hosts` as "clear the list", which is the
  documented reset. Typing it as string[] would have removed a working admin
  action rather than tightened anything.
- PluginRoutesController.route answers { route: null } at 200 for everything
  it dislikes, so the client falls back to straight lines. A 400 there would
  turn a graceful degradation into a visible error.

files.bridge is gone. The multer options are built by
MulterModule.registerAsync from an AllowedFileTypesService leaf, so the
fileFilter reads the operator's extension list from the container instead of
from a second FilesService built outside it. Nest injects
MULTER_MODULE_OPTIONS with @Optional(), so a token that fails to resolve is
not a boot error: it silently falls back to defaults, with no filter and no
size cap. FILE-020 already proved that for trip files by uploading an .svg
over HTTP and expecting 400; journey uploads had no such case, and now do.
VIDEO_UPLOAD is untouched, including the filename callback that forces .jpg
regardless of what the client sends, which guards the public share proxy.

PluginFrame gets an onError. A frame whose document fails to load left a
blank element at full height, which reads as a hung app rather than as a
plugin that is gone.

typecheck:tests runs in CI now, with the test tree at zero errors. tz-lookup
is left as it is: the build puts it in a lazy helpers chunk, not the entry
chunk, and the remaining split would trade a synchronous render for an async
one to move 172 kB that is already off the critical path.
2026-08-10 19:54:46 +02:00
Maurice 328b5fcb2b One Nominatim client for atlas and maps
Atlas throttled to the 1.1s OSM asks for and cached its answers in a 50k
map; maps did neither, so five interactive routes fired straight at the
service while the two atlas backfill loops politely waited their turn.
From Nominatim's side that is one instance ignoring the rate limit.

Both go through geo/nominatim.client.ts now. Three things the naive merge
would have got wrong:

- The shared client sends UA, the instance-identifying User-Agent, not the
  generic string atlas used. A shared generic UA is throttled harder.
- One cursor across five interactive routes and two background loops would
  have put a keystroke behind however many places the backfill enqueued, so
  there are two lanes over the one cursor and background yields.
- The cache key carries the query shape. Atlas asks at zoom 3 and 8, maps at
  18; a coordinate-only key answers a street lookup with a country.

The throttle cursor and the cache stay module state, the way the permissions
and FX caches do, because the bridges build their collaborators outside the
container and a second copy of the cursor is the bug being removed here.
GeocodingService owns only the cleanup timer's lifecycle, which atlas used to
start as an import-time side effect.

Also folds the three haversine implementations into common/geo.ts, keeping
the clamped form: asin of a value a hair over 1 is NaN, and a NaN distance
fails every comparison it feeds instead of throwing.

tests/setup.ts zeroes the interval. Nineteen maps cases drive these paths
back to back with real timers and were sleeping nineteen seconds for it.
2026-08-10 17:08:54 +02:00
Maurice 55da861de6 Put the reservation imports behind one door
Two controllers declared the identical @Controller string from two modules and
each rebuilt the trip access check its own way: booking-import ran it as a
guard chain, integrations ran it by hand inside the handler, and
BookingImportService carried a third copy that nothing called at all.

They never actually collided - the four file-upload sub-paths and the one
AirTrail sub-path are disjoint - so this is not a bug fix. It is one access
path instead of three implementations of the same one, and the dead copy is
gone.

The addon requirement sits on the AirTrail handler rather than the class.
Class-level would have 404d the four upload routes whenever AirTrail happened
to be off. AddonGuard still runs first in the chain, because a disabled addon
has to answer 404 to an anonymous caller before anything answers 401, and
ADDON-GUARD-006 pins both halves of that.

One deliberate behaviour change: the AirTrail route now gets its trip 404 from
the guard, ahead of the body 400, because a guard runs before the pipe. That
is the ordering every other trip-scoped controller already has and nothing
pinned the other way for this route.

The class rename would have stopped the server booting: the body-contract gate
keys on ControllerClass.methodName and throws on stale entries. Three entries
renamed, and the list re-sorted - GATE-006 caught that the rename had broken
the ordering it asserts.

booking-import lost the controller from its coverage denominator, so the
branch floor was about to fall. It goes up instead: the off-request parse had
no case for its catch arm, so a throw would have left the job on 'running' and
the widget spinning, and the binary probe behind /api/health/features had no
test at all - that one boolean decides whether the client offers booking
import.
2026-08-10 17:08:54 +02:00
Maurice ffb73fac20 Turn protection from opt-in into default-deny
Nothing in src/nest/** was an APP_GUARD. Every protected route said so
itself, 62 times, which means a forgotten @UseGuards was a silent bypass
rather than an error. GlobalAuthGuard inverts that: a route is authenticated
unless it carries @Public() or @OptionalAuth(), or declares its own guard
chain.

Reading the declared chain is what makes this behaviour-neutral. Nest runs
global guards before route guards, so without that check the global guard
would 401 an anonymous request to an addon-gated controller before its
AddonGuard could answer 404 - inverting an order four e2e suites pin, and
leaking which addons are installed. A controller that declared a chain keeps
deciding for itself.

It is a new class rather than JwtAuthGuard gaining a Reflector: that guard is
constructed positionally in its own test with a context stub that has no
getHandler, and the two want opposite behaviour anyway.

@Public() takes a required reason, and the reasons are the diff worth
reading: the invite token IS the credential, help assets are loaded by <img>
which cannot send one, logout has to work with an expired session or nobody
can sign out.

The real deliverable is the inventory. validateRouteGuards runs in buildApp
next to the body-contract gate and refuses to start when a route answers
anonymously without being on a reviewed list of 25 - or when a listed one no
longer does. An allow list nobody prunes stops being a list.

Default-deny found two controllers on its own: SharedController and
OidcController both sit in a file next to a guarded controller, so a
file-level search for unguarded ones missed them. 37 tests went red and named
them. That is the mechanism working before the change shipped, not after.
2026-08-10 13:38:26 +02:00
Maurice ac49191ec8 Describe the journey bodies instead of tightening them
Sixteen of the thirty-four entries in the body-contract allow list were the
JourneyController, and a strict schema already existed for six of them. That
is the trap: eight of these handlers validate their own body and answer with
their own text - 'Title is required', 'trip_id required', 'provider and
asset_id required', 'entry_date is required', 'user_id required' - and those
bodies are pinned by e2e and integration cases. Adopting the strict schemas
would have the pipe answer first, with a different body. That is a
client-visible change wearing the costume of a refactor.

So the schemas follow the memories doctrine instead: loose objects, and every
field the handler type-checks itself is unknown here. They describe the shape
that already works, which is what the boot gate needs, and every hand-rolled
400 keeps answering. The spec inverts accordingly - it now asserts that each
schema ACCEPTS what the handler still rejects, which is the property that
makes the claim checkable.

Two of them are optional at the top level, and that is not decoration: a
multipart upload with files and no text fields arrives with no body at all,
and JOURNEY-INT-045 pins 'No files uploaded' for it. Without the optional the
pipe answered 'body: Invalid input: expected object' instead. The suite
caught it.

Backup joins them - its schema was already a passthrough, because the service
normalises the body itself.

Allow list: 34 entries down to 17. Tags and the two OAuth entries stay for
now; tags reads @Body('name') per property, which is a signature change
rather than a contract one.
2026-08-10 13:38:26 +02:00
Maurice 5a9ded6676 Move the calendar export off the trip aggregate
exportICS and its four module helpers were 330 lines of RFC 5545 inside
TripsService, and FeedsService had to import the whole trip aggregate to get
an ICS string out of it. They move to a calendar domain unchanged - same
statements, same escaping, same folding, same VTIMEZONE fallback - so the
emitted bytes are identical on both the download and the feed.

The 21 unit cases move with the code and keep every assertion, which is what
makes that claim checkable rather than asserted. FeedsModule now imports
CalendarModule instead of TripsModule, and its e2e container drops from the
trip aggregate to two small modules.

NotFoundError/ValidationError move to common/ so a second domain can raise
them without importing the first one for its error types; trips re-exports
them, so the nine files that import them from there are untouched.

Fixes a harness that tsc cannot see on the way past: mcp-test-controllers
built TripsService with 12 arguments for 14 parameters, leaving unsplash and
userCleanup undefined.
2026-08-10 13:38:26 +02:00
Maurice 97f524cdfd feat(server): give the memories endpoints their Zod contracts
The last nine memories entries in the body-contract allow list are gone: three
each on the immich, synology and unified controllers. They read @Body() as
Record<string, unknown> and coerced by hand.

The schemas are deliberately permissive and loose, and that is the point rather
than a shortcut. Paging arrives as `2` or `'2'` because the controller does
`Number(page) || 1`; synology_skip_ssl arrives as a boolean or the string
'true'; auto_upload is accepted in any shape because only a real boolean is
acted on. A schema that described the ideal body instead of the real one would
turn working clients into 400s, and parity outranks tightening. Narrowing them
is a separate, client-visible change.

Loose objects for the same reason: a strict object strips unknown keys, and
`selections[]` goes straight to the services as `Selection[]` - stripping there
would be a silent data loss the day a provider adds a field.

This unblocks be-memories-fold, which could not move a controller whose body had
no contract to move with it.
2026-08-08 12:43:29 +02:00
Maurice 9c46f31c57 refactor(server): fold airports and distance, drop the boot-time backfill cycle (#1834)
Two thirds of be-weather-airports-fold. weatherService (513 LOC, a process
cache and a setInterval at import) is left for its own PR.

airportService split along the line that was already there. The dataset lookup
touches no database — it reads one JSON file and keeps it in memory — and both
MCP registrars call it from outside the container, so findByIata/searchAirports
are free functions in nest/airports/airports.data.ts. backfillFlightEndpoints
needs the connection and became a method on AirportsService.

That backfill was the interesting part. db/database.ts ran it at module eval
through a bare require() back into services/, so the module graph closed a cycle
that only held because `db` happened to be initialised by the time the require
resolved. It runs in onApplicationBootstrap now: it repairs rows, nothing boots
on it, and the container is fully built by then.

The cycle had a visible cost. tests/setup.console-noise.ts carried a suppression
for `Cannot find module '../services/airportService'`, because vitest's
transform pipeline could not resolve that require at all — it fired in roughly
sixty test files. That suppression is now narrowed to the only case left: an e2e
TestingModule whose fixture DB has no reservations table.

distanceService split the same way: haversineKm is arithmetic and moved to
common/geo.ts; getFlightDistanceKm reads the database and became a private
method on AuthService, its only caller.

AirportsModule imports DatabaseModule explicitly. @Global only reaches modules
that are in the graph, and the airports e2e builds its container from
AirportsModule alone — exactly the trap the nest README now documents.

The search ranking gets its first direct tests. It sat in services/, outside the
coverage gate, and only ever ran incidentally through the route; moving it into
src/nest made the gap measurable (16% branches). The ten cases pin the scoring
ladder — exact IATA short-circuit, ICAO over prefix, IATA prefix over city
prefix, city prefix over substring — and the missing-dataset fallback, which no
test had reached. 90% branches now.
2026-08-07 21:07:14 +02:00
Maurice 4e6b45016a refactor(server): move the KML importer and place-image helpers into places/ (#1832)
Two of the three files in be-places-helpers-fold. placePhotoCache is left for
its own PR — it holds three module-level caches and its sweep hangs off the
cron, so it needs both a provider with real state and a second entry in the
scheduler contract.

kmlImport.ts is 212 lines with no state, no database and no I/O — parsing,
sanitising and coordinate arithmetic. It is nest/places/kml-import.helpers.ts
now, unchanged, next to the places.helpers.ts that already consumes it.

placeImage.ts moves to nest/places/place-image.ts as free functions. Making
reclaimPlaceImage a provider would have split it from PLACE_IMAGES_DIR, which
cannot be injected: nest/common/place-image-upload.ts reads it at module scope
to configure multer's storage, before any container exists.

That constant is the one thing here that could have broken quietly:

  path.resolve(__dirname, '../../uploads/places')

__dirname moves with the file. From src/services that resolved to
<server>/uploads/places; from src/nest/places the same string would have
resolved to <server>/src/uploads/places — a directory multer would have happily
created, leaving every uploaded place image in the wrong place and every
existing one unreachable. The path is now '../../../uploads/places', verified to
resolve to the identical absolute directory, and uploads-dirs.test.ts covers it.
2026-08-07 19:58:12 +02:00
Maurice 22d178d2f9 refactor(server): one addon guard, one demo-mode condition (#1829)
Two pieces of copy-paste in the guard layer.

Three addon guards — collections, airtrail, journey — did the same thing and
differed only in the addon id they read and the word leading the 404. They are
one AddonGuard plus a @RequireAddon(addonId, label) decorator now, read through
the Reflector. A handler-level decorator overrides the controller's, and a route
without the decorator is not gated at all.

The demo-mode upload block was copy-pasted into six handlers, each carrying its
own copy of the message. All six call isDemoWriteBlocked now, against the same
DEMO_WRITE_ERROR body.

That one is deliberately NOT a guard, though all six sites are handlers and a
guard is the obvious shape — and this PR did build one first. Guards run before
the multipart parser, so throwing from one leaves the request body unread and
Node resets the connection: the client gets ECONNRESET instead of the 403.
PROFILE-015 caught it. The check has to sit inside the handler, after the
interceptor has drained the upload, for the response to arrive at all. The
function carries that reasoning so the next person does not rediscover it.

Two smaller findings on the way there.

The DemoWriteGuard attempt broke eight e2e suites that never touch demo mode:
AuthModule listed it as a provider, and RuntimeEnvService was not resolvable
inside a TestingModule built around one domain module. AppConfigModule is
@Global, but @Global only applies to modules that are in the graph. The five
modules whose controllers now inject RuntimeEnvService import it explicitly, the
same way PermissionsModule is handled.

And both new guards get a test asserting the decorators are actually applied.
Once a check moves out of a handler, that handler's own tests stop noticing if
the decorator is dropped — the assertions on __guards__ and the @RequireAddon
metadata are what keeps the four upload endpoints and the four addon route
groups gated, and gated on the right addon.
2026-08-07 18:42:53 +02:00
Maurice 9814480a96 refactor(server): move the JWT verify into nest/auth, delete four dead middlewares (#1828)
middleware/auth.ts exported seven things. Two of them — extractToken and
verifyJwtAndLoadUser — are the canonical session check every auth surface goes
through. The other five (authenticate, requireCookieAuth, optionalAuth,
adminOnly, demoUploadBlock) are Express middlewares with no production caller
left; the guards in nest/auth replaced them and only comments still mention
them.

The two live functions move to nest/auth/jwt-verify.ts and the file goes, along
with middleware/validate.ts (nothing but its own test imported it) and
middleware/idempotency.ts (its only caller was `authenticate`, and
IdempotencyInterceptor has replaced it). AuthRequest and OptionalAuthRequest go
with them — src/typings/express.d.ts is what types req.user now.

Eight importers repoint: the three guards, auth/files/plugins-proxy services,
platform.routes and the global MFA policy.

Two deliberate departures from the plan for this package.

JWT_SECRET does not move to app-config. It is not an env value: it is generated
into data/.jwt_secret, and the admin panel rotates it at runtime through
updateJwtSecret. `export let` is precisely what makes a rotation take effect
in-process — a registerAs token would freeze the boot value and silently break
rotation.

The verify also keeps reading the db singleton rather than DatabaseService. That
change is not a one-line swap: it gives the three guards a constructor
dependency, which means registering them as providers everywhere they are used.
Measured: 38 directories use those guards and 21 of them do not import
AuthModule, on top of 34 e2e tests that each build a container around a single
domain module. That is its own PR, not a rider on one that deletes dead files.

tests/unit/middleware/auth.test.ts covered real behaviour, so it is not simply
deleted. Its extractToken cases and the five verify cases it drove through
`authenticate` come back as tests/unit/nest/auth/jwt-verify.test.ts, now hitting
verifyJwtAndLoadUser directly. Four cases are new: the password_version gate —
the whole reason this function exists — had no direct coverage at all. Its three
adminOnly cases are dropped because AdminGuard, their successor, is covered by
three e2e suites.
2026-08-07 18:01:24 +02:00
Maurice 939452f9d8 refactor(server): consolidate trip access and fold the query helpers into the container (#1824)
* refactor(server): route every trip-access check through DatabaseService

Twelve Nest services carry the same verifyTripAccess/canEdit pair. Six already
asked DatabaseService.canAccessTrip; the other six went through
services/tripAccess, a nine-line re-export of the identical function. Same
query, same result, two ways to reach it.

All twelve now take the injected route, so services/tripAccess is gone.

The bigger duplication was in the types. Thirteen files declared their own
`Trip` for the same return value, in three different shapes:

  NonNullable<ReturnType<typeof verifyTripAccess>>   6x
  { user_id: number }                                4x
  TripAccess                                         3x

The narrow one is why assignments, days, places and accommodations all wrote
`as Trip | null | undefined` on the call — the cast existed to shrink
TripAccess down to a local type that was never meant to differ from it. All
thirteen now say TripAccess, and the four casts are gone with no other change.

middleware/tripAccess.ts goes too: it was already dead in production — nothing
under src/ imported requireTripAccess or requireTripOwner, only comments in the
services that had absorbed them. Its test went with it.

* refactor(server): fold the query helpers and trip joins into the container

Two more legacy files whose only reason for staying outside the container was
that they predated it.

services/queryHelpers.ts held five exports doing two different jobs. Three are
the batch loaders that keep the list endpoints off N+1 queries — they hit the
database and are now QueryHelpersService over the injected connection, imported
by assignments, days, places and share. The other two, formatAssignmentWithPlace
and ratingAggregate, touch no database at all; they sat next to the loaders only
because the loaders were free functions too. Those stay free functions, in
common/rowShape.ts.

The README note on the assignments service claimed the loaders had to stay in
services/ because the day, share and place services were unmigrated. All three
have since moved, so the reason is gone.

services/tripMembership.ts becomes TripMembershipService in its own module. It
could have folded into auth or trip-invite, but both call it — as do oidc and
the plugin RPC host — and putting it in either would have made AuthModule and
TripInviteModule import each other.

Neither module is @Global, matching PermissionsModule: e2e TestingModules
resolve them transitively through each consumer's explicit import.

Test seams changed shape accordingly. Two path mocks of services/tripMembership
became injected stubs; the trip-invite e2e overrides the provider in the
container instead. The oidc test keeps the real implementation, which is what
OIDC-SVC-045 was always asserting.

ratingAggregate moved out of the ungated services/ tree into the coverage-gated
one, so it gets the direct tests it never had — as does the tags/participants
pass-through, which two long-dead fixture variables in the moved test file had
been waiting for.

Also drops four stale comments that outlived their subject: the requireTripAccess
middleware they pointed at was deleted in the previous commit.
2026-08-07 16:00:12 +02:00
Maurice 6dda05f672 refactor(server): move the stateless helpers into nest/common (#1823)
Eight files in services/ hold no state and touch no database: avatarUrl,
conflictResult, demo, passwordPolicy, timezoneService, cookie, and the two
crypto helpers. They are imported from 65 places, 62 of them already inside
src/nest, so having them live in the legacy layer was pure inertia.

They stay plain functions rather than becoming providers. Three importers sit
outside the container and would have no way to reach an injectable one:
db/migrations.ts runs before the app is built, and middleware/auth.ts and
middleware/mfaPolicy.ts run beside it. Wrapping these in DI would buy ceremony
and nothing else.

mfaCrypto and apiKeyCrypto move together into common/crypto — they read the
same ENCRYPTION_KEY, and the directory says so.

services/ drops from 44 files to 36, 9,870 lines to 8,493.

One trap worth recording: src/app-config and src/nest/app-config both exist.
cookie.ts imported '../app-config' from services/, which resolved to the former;
after the move the same string silently pointed at the latter, which has no
readEnv. It is now explicit about which one it wants.

Nothing else changes — no signatures, no behaviour, no tests rewritten beyond
their import paths.
2026-08-07 15:14:14 +02:00
Maurice cf62567c48 fix(map): stop untrusted values reaching the marker HTML (#1820)
* fix(map): drop the casing layer before its source on teardown

Tearing down the reservation overlay logged an error on every map unmount:

  Source "trek-reservations" cannot be removed while layer
  "trek-reservations-lines-transit-casing" is using it.

The casing layer arrived with the transit paths in 3.2.0 but was never added to
destroy(), which only removed the main line layer. The try/catch around it gave
no protection either — the engine reports this by firing an error event rather
than throwing — so the source was silently left behind on the style.

The layer id is a named constant now, so the place that creates it and the place
that removes it can no longer drift apart.

The lifecycle test grew the assertion that would have caught it: the fake map
now refuses to remove a source that a layer still references, the same way the
real one does. Verified against the unfixed code — two tests fail without the
change.

* fix(map): stop untrusted values reaching the marker HTML

Both map renderers build their marker HTML as a string and hand it to
L.divIcon / innerHTML. Two of the interpolations carry values a user controls,
and neither was escaped.

place.image_url is the worse of the two, because it is not admin-only: any trip
member with edit rights can set it. placeUpdateRequestSchema is a plain record
of unknown values, and the length guard in places.controller covers name,
description, address and notes but not image_url. The renderers do check the
value, but only with startsWith('/uploads/') — a payload like

  /uploads/x" onerror="…" y="

satisfies that and then breaks out of the src="…" attribute.

category.color is the same shape one line up, reachable by an admin. It also
reaches SharedTripPage, which builds its own divIcon and answers without a
guard, so that payload lands in front of people who have no account here.

Colours are allow-listed rather than escaped. Escaping stops the attribute
breakout but leaves a working CSS value behind, and `url(https://…)` in a
background is a tracking pixel. Anything that is not #rgb/#rrggbb falls back.
image_url is a URL, not an enum, so that one is escaped; the prefix check stays.

Server side, the category contract now demands a hex colour, and the endpoints
validate through DTO classes instead of @Body('color') — a property read carries
no metatype, so the global pipe never saw it. That retires two entries from the
body-contract allow list. It replaces the hand-rolled 400 for a missing name
with the pipe's envelope, the same trade the places and trips migrations made.

The response schema deliberately still accepts any string: the write path was
open for a long time, so tightening it would make an instance fail on its own
stored rows. The client allow-list is what covers those.

Verified against the unpatched renderer — the three new cases fail without it.

* perf(client): serialize the map icons without react-dom/server

Six modules imported react-dom/server statically to turn a lucide icon into an
SVG string. That put Fizz — roughly 190 kB raw, 57 kB gzip — into a chunk three
lazy routes share, the Leaflet renderer among them, which is the default. For
output that is always a single <svg> with a flat list of shape children.

  mapViewport chunk   189,532 -> 3,710 B
  Tooltip chunk       194,333 -> 1,970 + 4,860 B
  precache total      17,856 -> 17,675 KiB

utils/iconMarkup.ts serializes React elements directly. It is generic over
elements rather than over lucide's icon tables on purpose: iconNode is closed
over inside createLucideIcon and never exported, so calling the forwardRef
render function is what survives a lucide upgrade.

Byte-compatibility is checked, not asserted. iconMarkup.parity.test.ts diffs the
output against real Fizz for every exported icon in three prop shapes — the test
file is the one place still allowed to import react-dom/server, and it never
reaches a bundle. Two traps it covers: viewBox must not be kebab-cased into
view-box, and Fizz emits <path …></path> rather than a self-closing tag.

TripPDF loses its lazy renderer entirely — ensureRenderer, the module-level
handle and the two "not ready yet" guards that returned an empty string. There
is nothing left to load.

An eslint rule keeps the import from creeping back. Without it the regression is
invisible: the build stays green, the app keeps working, and the chunk quietly
grows by 190 kB again.
2026-08-07 13:43:38 +02:00
Maurice e606ebd9b9 chore(server): pin the WebSocket envelope, dedupe the rate limiter (#1803)
* chore(nest): register LlmParseModule in AppModule

The module was only in the container because BookingImportModule and
PluginsModule happen to import it, so /api/admin/llm/local depended on those two
keeping that import. Register it where every other module is registered.

* refactor(nest): give the rate limiter one home and one instance

RateLimitService keeps its counters in a Map on the instance, but it was listed
in the providers of auth, oauth, transit and trip-invite, so each of them got its
own. Nothing is broken today — the bucket names in use (login/mfa/forgot/reset,
oauth_*, transit_*, trip_invite) don't overlap — but two modules picking the same
name would have quietly allowed twice the configured attempts.

Move it to nest/common, where it belongs now that four domains use it, and hand
it out through a small module they import. Not @Global: the e2e suites build a
container around one domain module, so a global AppModule never pulled in isn't
there at all.

transit.mcp.ts keeps its own instance on purpose — it keys by userId under
mcp_transit_* while the REST path keys by IP.

* test(ws): pin the wire envelope before touching the transport

Every frame on the socket is flat: {type, tripId, ...payload} out, {type, tripId}
in. Nest's WsAdapter dispatches on message.event and answers {event, data}, so
moving this onto a gateway without a custom serialiser would leave join silently
unhandled — no welcome.socketId to send back as X-Socket-Id, and every client
would then receive its own writes.

Nothing catches that today: the 100+ tests that touch realtime mock src/websocket
and never look at the bytes. WS-032 sends the exact shape the stock adapter would
and asserts the server ignores it.

* docs(nest): drop the parts of the README that no longer exist

strangler.ts, DEFAULT_NEST_PREFIXES/NEST_PREFIXES and tests/parity/ are all gone —
every request goes through Nest now — but the README still walks a new contributor
through adding a prefix and writing a parity test against the Express route.

Also spell out that the 80% gate is an average over src/nest/**, not a floor per
file, so a large untested module can sit behind the well-covered small ones.
2026-08-06 22:44:59 +02:00
jubnl 2ac6c76086 chore: move admin backend to proper DI
The 851-line services/adminService folds into the wrapper AdminService over
DatabaseService plus injected Settings/Addons/Passkey/Packing/Auth/Permissions/
Notifications services; the auth, notifications and permissions bridge imports
all become injections, PERMISSION_ACTIONS stays a plain const, and the
mcp/sessionManager deep import keeps its anti-cycle comment. Ahead of the fold
the 11 packing-template functions move to PackingService, which already owned all
three template tables — resolving the admin-2 residual with no bridge. The pure
and module-scoped half moves to admin.helpers.ts, the version cache staying
module-scoped so admin.bridge's instance and the container singleton share one
GitHub fetch. A 1-export admin.bridge.ts serves scheduler.ts's cron, the only
out-of-container consumer. Recipe steps 3 and 4 are no-ops: no src/mcp/tools/
admin.ts has ever existed and the plugin host never imported the domain.

Four lines are non-verbatim, all path re-anchoring one directory deeper, both
resolved paths verified against the emitted dist layout.

admin.dto.ts clears all twenty AdminController allow-list entries, retiring the
three 'enabled must be a boolean' checks plus 'permissions object required' and
'Object body required' for the pipe envelope. The module e2e goes DI-native: the
3-method whole-module mock dies, 6 cases become 15 over real SQL.

Quirk fixes on top, each pinned: the three places toggles go fail-closed with an
append-only backfill migration; the template item routes honour :templateId;
updateUser rejects an empty username/email; createInvite 404s an unresolvable
trip_id; listOAuthSessions guards its scopes parse; GitHub fetches gain a timeout
and size cap and failures cache briefly; updateOidcSettings/updateAddon/updateUser
become transactional; admin.oauth_session.revoke is renamed to match its
siblings; two mutations gain their missing audit rows; getAuditLog drops the
_parse_error sentinel.

Tests move with IDs preserved (ADMIN-SVC-001..069 incl. the pre-existing 029/030
gap and duplicated 069, VNOTIF-001..007), plus ADMIN-BR-001 and ADMIN-SVC-070..076.
2026-08-02 15:07:55 +02:00
jubnl 00447c9043 feat(server): validate notification request bodies
notifications.dto.ts wraps the @trek/shared request schemas via createZodDto
and types all five @Body() params; the five NotificationsController entries
leave the body-contract allow-list (ratchet gate enforced). Contract catch:
the client sends server/token: null on test-ntfy to mean 'use the saved
value', so testNtfyRequestSchema gains .nullable() there (spec pinned). The
inline response-enum and url-type checks die — malformed bodies now get the
pipe's standard envelope; valid bodies behave byte-identically. Two
integration cases pinning pre-ratchet tolerances (flat notify_* preferences
body, body-less test-smtp POST) update to the current wire contract.
2026-08-02 11:36:36 +02:00
jubnl 5d4b2abd26 feat(server): validate passkey request bodies 2026-08-02 10:16:10 +02:00
jubnl fb87202683 feat(server): validate auth request bodies
DTO ratchet for the auth domain (collections sibling-commit precedent):

- shared/src/auth/auth.schema.ts gains the six missing request schemas —
  mapsKey (nullable: the client clears with an explicit null), apiKeys and
  settings (per-key partials, nullable key values), appSettings (open
  record — the server's ADMIN_SETTINGS_KEYS allow-list and its bespoke
  lockout/self-MFA 400s stay service-owned), mfaDisable, resourceToken
  (the 'Invalid purpose' 400 stays service-owned) — plus spec cases.
- New nest/auth/auth.dto.ts createZodDto wrappers; all 14 raw @Body()
  params typed (9 AuthController + 5 AuthPublicController), validated by
  the global ZodValidationPipe.
- All 14 auth entries removed from body-contract-allow-list.ts
  (PasskeyController's 4 stay until passkeyService migrates).
- auth.e2e mirrors the production pipe and pins the { error: 'field: ...' }
  envelope on a shapeless login body; auth.controller.ts drops its
  duplicated avatarDir const in favour of the auth.helpers export.
2026-08-01 17:40:16 +02:00
jubnl ae3ba18b9e chore: move atlas backend to proper DI
Fold the 1612-line services/atlasService.ts into the DI-native AtlasService:
the stats aggregation, visited countries/regions with the #1490 tombstone and
cascade logic, and the bucket-list CRUD become methods over the injected
DatabaseService. The ~750-line pure-geo half — the bundled admin0/admin1
boundary stores with their #1576 streaming builders, the point-in-polygon
indexes, Nominatim geocoding with its shared throttle, and every module-scoped
cache including the import-time unref'd cleanup interval — moves verbatim to
the plain module atlas-geo.ts so the caches stay process-global across the
container instance, the bridge instance and test helpers (assetPath
re-anchored one directory deeper, the only non-verbatim line).

The 10-tool mcp/tools/atlas.ts registrar plus all four atlas resources in
mcp/resources.ts move onto the decorator registry as atlas.mcp.ts — the when:
atlas-addon gate is parity here (legacy tools and resources both gated; the
REST controller deliberately does not). resources.test.ts retires with its
last two cases, which move to tools-atlas-expanded.test.ts. The plugin RPC
host swaps its 9 atlas imports for the injected AtlasService (23rd constructor
dep); a minimal 2-export atlas.bridge.ts serves the one legacy consumer,
authService.getTravelStats, and dies when authService migrates.

Adopts atlas.dto.ts over the existing shared schemas and clears all three
AtlasController entries from body-contract-allow-list.ts: the hand-rolled
'name and country_code are required' 400 becomes the ZodValidationPipe
envelope (todo/places trade) while the whitespace-only bucket name keeps its
legacy 'Name is required' trim guard.

Fixes the verified defects the relocation had faithfully carried, each with a
regression test (ATLAS-SVC-031..036 plus two MCP casing cases): the four
multi-statement mark/unmark writes now run in db.transaction() (the region
cascade nests as a savepoint), the trip-less countryPlaces early return
honours manually_marked, the '|| null' bindings that dropped lat/lng 0 and
empty-string notes on bucket update become '?? null', the mutating bucket SQL
is user-scoped, and the MCP region/country-places tools uppercase their codes
to match REST.

Quirks deliberately preserved: the detached GET-path place_regions backfill
writes, getStats' two divergent return shapes, the import-time cache-cleanup
interval, getCountryFromAddress's unvalidated bare 2-letter codes, and the
whitespace-only-name silent no-op on bucket update.

Tests move with their case IDs: atlasService.test.ts becomes
unit/nest/atlas.service.test.ts (pure-geo imports from atlas-geo, DB calls
through the constructed service, plus bridge-delegation cases), the atlas e2e
converts to the DI-native pattern (real SQL against the full temp-db schema),
and the plugin-host path mock becomes a constructor stub.
2026-08-01 15:50:36 +02:00
jubnl d11b7bb054 feat(server): validate collections reorder and delete-many bodies 2026-08-01 13:54:36 +02:00
jubnl 4f0f80ad69 chore: move place backend to proper DI
Fold the 1029-line services/placeService.ts into the DI-native PlacesService:
the CRUD + ratings SQL, the GPX/KML/KMZ importers and the Google/Naver list
importers become methods over the injected DatabaseService, byte-identical
statements, COALESCE semantics and the If-Match conflict protocol (#1135).
The pure half — frozen XML parsers, the KMZ unpacker, the dedup predicates,
the Google hex-id parsers, reclaimPhotoCache — moves to places.helpers.ts
(maps.helpers precedent).

The 10-tool mcp/tools/places.ts registrar plus the trek://trips/{tripId}/places
resource move onto the decorator registry as places.mcp.ts. search_place travels
with them because its gate is places:read, not the read-only geo group, and now
injects MapsService. TripsService, DaysMcp, BookingImportService and the plugin
RPC host (PluginHostDepsFactory's 21st constructor dep) all inject PlacesService,
so the domain needs no bridge at all. The two assignments.bridge imports that
places.mcp.ts keeps are a deliberate cycle break: AssignmentsModule imports
DaysModule and DaysModule now imports PlacesModule for days.mcp.ts's place
creation, so injecting AssignmentsService there would close
DaysModule -> PlacesModule -> AssignmentsModule -> DaysModule. Same seam
reservations.mcp.ts uses, same trade trips.bridge.ts documents.

Also folds services/placeEnrichment.ts in — its DB/websocket/Maps half becomes
PlacesService methods over the injected DatabaseService/RealtimeService/
MapsService, its pure match selector joins places.helpers.ts — which retires
nest/maps/maps.bridge.ts with its last consumer.

Adopts places.dto.ts for the seven grandfathered body contracts and clears every
PlacesController entry from body-contract-allow-list.ts. Two consequences worth
knowing: the bespoke 400s 'Place name is required', 'ids must be an array of
numbers' and 'URL is required' are now the ZodValidationPipe envelope, and
because a pipe runs before the handler a malformed body 400s ahead of the
trip-access 404 it used to follow — the same trade the todo and trips migrations
took. placeBulkUpdateRequestSchema.ids drops .min(1) so the endpoint's
empty-list short-circuit stays reachable.

Fixes four defects the relocation had faithfully carried, each with a
failing-first regression test:

- create dropped lat/lng of exactly 0, so a place on the equator or the prime
  meridian lost its coordinates (`x || null` cannot tell absent from zero).
- duration_minutes: 0 was unsettable — replaced by the 60-minute default on
  create, and read as "absent" by COALESCE on update.
- the journey delete hook fired on unscoped ids: DELETE /:id, POST /bulk-delete
  and the delete_place MCP tool could detach another trip's journey entries for
  an id they then refused. All three scope first via the new
  PlacesService.scopedIds, mirroring the guard PluginHostDepsFactory already had.
  bulk_delete_places also moves its hook ahead of the DELETE —
  journey_entries.source_place_id is ON DELETE SET NULL, so running it afterwards
  left the entries as orphans.
- the place search treated % and _ as LIKE wildcards; a bare % returned the whole
  trip. The three clauses now carry ESCAPE with the term escaped.

Plus hardening on the Google list fetch: the response is capped at 8 MB
(declared content-length and post-read length, the transit.service precedent)
and its JSON.parse is guarded, so a malformed provider payload produces the
existing 'Invalid list data received from Google Maps' 400 instead of throwing.

Quirks deliberately preserved: the non-COALESCE route_color (#776),
currency/transport_mode staying unclearable, importGpx returning null for a file
that parses but yields no usable geometry, update skipping the If-Match check
when the stored updated_at is null, and every string-valued `x || null`.

Tests move with their case IDs: placeService.test.ts and the wrapper suite merge
into unit/nest/places.service.test.ts, kmzUnpack into places.helpers.test.ts,
placeEnrichment into places.enrichment.test.ts. The places e2e converts to the
DI-native pattern (real SQL against temp-db DDL), tools-places.test.ts is kept
and extended so it exercises the decorator path through the test registry, and
the plugin-host and booking-import path mocks become constructor stubs.
2026-07-30 17:07:12 +02:00