Commit Graph

127 Commits

Author SHA1 Message Date
Maurice ac57321840 merge dev and reconcile roadtrip planning 2026-09-12 15:45:34 +02:00
Maurice b10b02c685 finish overnight stops and cross-day range warnings 2026-09-12 15:37:09 +02:00
Konstantinos Thermos 359551653f Merge pull request #2319 from liketrek/feat/trip-route-overview
feat(maps): draw every travel day's route at once, with the trip total
2026-09-12 14:49:44 +03:00
Maurice f56e7a161e add open charging data and polish roadtrip warnings 2026-09-12 13:25:15 +02:00
Maurice 228e6a13a9 fix(admin): say what the local model actually gets 2026-09-12 13:07:08 +02:00
RemEx b58771fd4c fix(admin): recommend the local model that also reads images
The one model the addon offers to pull was Qwen3 8B, which is text-only, so a
scanned page or a photographed document reached it as no text at all and came
back empty. Qwen3.5 4B is multimodal at a third of the size: one download
covers every kind of document instead of leaving the operator to work out that
the recommended model cannot read half of them.

The list stays one entry long. A self-hosted host needs a model, not a menu,
and any other id can still be typed in or picked from what Ollama already has.
2026-09-12 13:07:08 +02:00
Maurice 41b18f970c feat(vacay): add manual school holiday calendars 2026-09-12 12:51:27 +02:00
Maurice ff5fa55fad add Google Maps route import and plugin corridor search 2026-09-12 12:24:33 +02:00
Maurice af45b0ca74 add roadtrip warning areas and tidy driving settings 2026-09-12 11:23:09 +02:00
Maurice b773faff70 feat(roadtrip): share trip preferences and improve service stops 2026-09-12 00:24:45 +02:00
Maurice fccc55732b feat(roadtrip): plan and edit trips through MCP without a browser 2026-09-11 23:25:06 +02:00
Maurice 491cfccc56 feat(roadtrip): add daily scheduling and configurable day endings 2026-09-11 22:28:03 +02:00
Konstantinos Thermos 3c735c80a4 Merge remote-tracking branch 'origin/dev' into feat/trip-route-overview
# Conflicts:
#	client/src/components/Map/MapView.tsx
#	client/src/components/Map/MapViewGL.tsx
#	client/src/pages/TripPlannerPage.tsx
#	client/src/pages/tripPlanner/useTripPlanner.ts
2026-09-11 16:22:21 +03:00
Maurice 2c06c9904b feat(collections): a list as a file, so it can leave the instance (#2198)
Sharing a list works between people on the same TREK. This is the other
case: giving one to somebody who runs their own, or keeping a copy. Export
sits beside Edit and Share in the list header and downloads the open list;
Import sits beside New list in the rail, reads a file, shows what is in it
and makes a new list of it. Desktop only, which the route already decides.

What travels is what somebody wrote down: the list's name, colour and
description, its labels, and each place with its address, coordinates,
notes, status, contact details, category name and labels. A file is dull
JSON on purpose, so it can be read and edited by a person.

What does not travel is everything that only means something on the instance
it came from. Ids of every kind, since the receiver issues its own and a
category crosses over by name. Star ratings and members, because a rating is
an opinion somebody gave the people on that list. Provenance into trips,
which points at rows the receiver cannot see. And an image path under
/uploads, which resolves on the sender's server: only an absolute https URL
survives.

A file is a stranger's JSON and is read as one. It is parsed in the browser
that chose it and never uploaded; the parsed list goes through the same
contract on both sides; and a link that is not http(s) arrives as nothing
rather than as an anchor. Strict where it must refuse, forgiving where it
can drop: a file that does not say what it is never reaches the importer,
while one unreadable place inside it is skipped and counted, and one bad
field on a good place costs that field alone.

An import is always a new list. Importing into an existing one would mean
deciding what happens to a place that is already there, and every answer to
that is a merge nobody asked for.
2026-09-11 14:58:34 +02:00
Maurice 87c73e50fb feat(studio): give a book its first and its last page (#2317)
A new book was a cover, a spread and a back cover, so it opened straight
onto pages 2 and 3. A bound book does not: the inside of the cover is
blank, and the first thing facing the reader is a single right-hand page,
page 1. The same at the end, where a single left-hand page sits before the
back cover. Both are laid down now, empty, in a new book and in a laid-out
one, and they are fixed the way the covers are: an inner spread only ever
goes between them.

The page numbers follow. They were counted from a spread's position in the
array, which assumed the spread after the cover was the first, and three
places did that sum on their own. One function now counts along the book,
one page for a single leaf and two for a spread, so the sheet on screen and
the label in the print view and the folio on the page all say the same
number. A book with a first page starts at 1; a book from before the first
page existed has none and keeps the numbers it always had.
2026-09-11 11:01:15 +02:00
Maurice a9907e2f10 feat(plugins): let a plugin answer place searches from its own index
TREK searches its own place index and OpenStreetMap. Neither carries a rating,
so "the best rated hotel around here" has never been answerable, and an operator
with a better index for their region had no way to reach the search box at all.
The searchProvider hook is that way in: a plugin is handed the query, a limit,
the language and the coordinate the search is biased toward, and answers with
places in the same shape the core search returns, plus a rating.

It sits beside the core search rather than inside it. The plugin runtime already
imports PlacesModule, which imports MapsModule, so asking the providers from the
maps service would close a module cycle — the same reason the other fourteen
contribution controllers stand next to what they contribute to. The client asks
both at once and merges, at the single point every caller already goes through,
so all nine search surfaces get plugin hits without knowing the hook exists.

A plugin index may never hold up the list: the client stops waiting at two and a
half seconds and the host gives a provider two, which is also why the budget is
the shortest of any hook here. With no such plugin installed the route answers
without reaching the runtime at all.

Every hit is normalized host-side and shared by the REST route and the MCP tool,
so a cap that holds for the browser holds for the assistant: coordinates are
range-checked, strings capped, the rating clamped to 0..5, a website must be
http or https, and ids are namespaced so they can never collide with an OSM one.

search_places_via_plugins is the MCP half, because what the user can see the
assistant has to be able to see; search_place now points at it.
2026-09-10 21:40:51 +02:00
Konstantinos Thermos bf1900c3a0 feat(maps): draw every travel day's route at once, with the trip total
Extract the day route builder out of useRouteCalculation into a pure module and
aggregate it over the whole trip: one map with every day's route, each day in its
own colour, the mode per stage and the trip's total distance. Straight lines first,
roads as they answer; a refused leg keeps its line and adds no distance.

Give the floating map controls one shared drop shadow while adding the toggle, so
the new pill and the base-layer switcher stand off the tiles the way the locate
button always has. Document the control in wiki/Map-Features.md.

Addresses #1736
2026-09-10 15:57:48 +03:00
Maurice 8e2840b29d docs(helm): document the probes block, and survive probes being nulled
A probe map is exactly the value someone reaches for when their cluster is
unhappy, so the two ways of reaching for it should not break the render.
`probes: null` hit a nil pointer on `.Values.probes.liveness`; guarding with
`with (default dict ...)` also makes dropping a single probe work, which is what
you want on a pod whose liveness check is the thing killing it.

The rest is documentation the chart already gives every other value: a wiki
section next to Resource Limits, a Notes bullet, and a values comment naming the
one trap here, that a handler switch has to clear the shipped httpGet in the same
override or the API refuses the pod.

Verified with helm template: defaults render byte for byte what the chart
produced before, probes=null renders no probes, probes.liveness=null leaves one,
and helm lint passes.
2026-09-06 17:26:51 +02:00
Maurice 94914295d1 fix(llm): let a model refuse temperature, and say which extractor is installed
Two reports, both about an import that fails without saying why.

#2262 — temperature: 0 sat in the shared request body, so it went out on every
attempt including both retries. Reasoning models (the gpt-5 family) reject any
explicit temperature, and the json_object retry fires on any 400, so the import
spent two doomed requests and could not succeed at all. The parameter is now
dropped once a server has explicitly said it will not take it.

Not as a third link in the chain, though. A gpt-5 rejects max_tokens AND
temperature, the API names one parameter per response, and it may name either
first — a chain of one-shot ifs converges on only one of the two orders. The
retries are a loop over what the server actually said instead: each remedy
applies at most once and they compose. Detection is a phrase, not the bare word,
because dropping temperature costs the deterministic sampling that small local
models depend on, and an unfamiliar wording still falls through to the
json_object retry exactly as before. NuExtract and Ollama are untouched: the
first attempt always carries temperature, and it is the only one they ever see.

#2261 — the bundled extractor is Debian trixie's 24.12.3, roughly eighteen
months behind upstream, and the reporter is right that this is invisible: a
stale extractor, a mistyped KITINERARY_EXTRACTOR_PATH and a genuinely
unsupported provider all look the same from the UI, which is nothing imported
and no reason given. The version now appears in the startup log and in a new
admin-only GET /api/admin/system-info, and LOG_LEVEL=debug passes the
extractor's raw stderr through — the JS ERROR and Invalid-result-type lines the
filter drops are the only evidence that a vendor script is missing or failing.
Capped at 200 lines, and opt-in because those lines can carry document
fragments.

Not bumping the package: the filter the reporter needs first ships in KDE Gear
26.08 and no Debian suite carries it, and sid's 26.04.3 wants KF6 6.22 and Qt6
6.10.2 against trixie's 6.6 and 6.8.2 — a partial upgrade of Qt6, KF6 and libc6
from unstable into the production image, on two architectures, that still would
not fix the reported carrier. It comes for free with the base-image move after
Debian 14.
2026-09-06 15:03:13 +02:00
Maurice 53c246e8e5 docs(wiki): the AI parsing addon now reads one environment variable
Three shipped sentences became false with LLM_TIMEOUT_MS: two claiming the
addon needs no environment variables (one of them in the very file the
variable was added to), and one naming the old five-minute ceiling that the
deleted constant used to set.
2026-09-04 18:56:22 +02:00
RemEx dba4434ef6 fix(llm-parse): one configurable ceiling for a model call, and undici honours it
The timeout for a model call was spelled three times and adjustable
nowhere: 120s in the Anthropic client, 300s in the OpenAI-compatible one,
300s in the Ollama one. None of them was the ceiling that actually
applied, either — every one of those calls goes through the pinned undici
dispatcher, and an undici Agent stops waiting for response headers after
five minutes by default, so a client constant set above that never got
the chance to fire.

- LLM_TIMEOUT_MS (default 900000) replaces the three constants and is
  passed to the dispatcher as headersTimeout/bodyTimeout, so the abort
  signal and undici are governed by the same setting. The default is
  deliberately generous: heavier parsing work should fit without a code
  change, and an admin who would rather fail fast can lower it.
- The ceiling stays on the model lane. safeFetchAdminConfigured is also
  the OIDC lane (discovery, token, userinfo, JWKS) and the plugin OAuth
  token exchange, and that one sends no AbortSignal of its own, so it
  keeps undici's five minutes rather than inheriting a model setting.
- The derived value is floored: the schema accepts any positive number
  and undici rejects a fractional headersTimeout.
2026-09-04 18:56:22 +02:00
Maurice 9ed88adf34 fix(budget): an expense with no payer creates no debt
An expense saved with "No one paid yet" was still charging its split
members in calculateSettlement while crediting nobody, so the ledger
drifted off zero by the unpaid total. The greedy simplifier that turns
balances into payments has no memory of which expense produced which
debt, so it handed that phantom amount to whoever happened to be a
creditor from an unrelated expense. In the report, two people each owed
about 80 to someone who never paid that bill. Booking every payment it
offered then left one member permanently short, next to an "everyone
square" line.

Both cost panels already promise the opposite: they label such a row
"Unfinished, total only, not settled yet" and count it into the
Outstanding amount card instead. The server now agrees with them.

The guard deliberately does not also require a non-zero total, unlike the
panels' label predicate. PUT /budget/:id/payers with an empty array zeroes
total_price without re-applying it, which would otherwise leave a custom
split debiting its members against no credit at all.

The #1543 fixture and BUDGET-SVC-DB-004/005 were re-baselined: they pinned
the old arithmetic through a deliberately payer-less 9 000 RUB item. The
wiki said the old behaviour was intended, so it is updated too.

Closes #2225
2026-09-04 12:15:38 +02:00
jubnl bc10fba389 feat(plugins): TREK_PLUGINS_IGNORE_TREK_RANGE turns the version gate into a warning (#2226)
* feat(plugins): TREK_PLUGINS_IGNORE_TREK_RANGE turns the version gate into a warning

A plugin whose author has not updated its `trek` range for the running TREK can
now be installed, updated and activated when the operator sets
TREK_PLUGINS_IGNORE_TREK_RANGE to a truthy value. Every gate (registry picker,
post-extraction re-check, sideload, dev-link, activation) warns instead of
refusing; a missing range is tolerated the same way; "install latest" takes the
newest published version. The plugin-API version gate is not lifted and the
Discover compatibility verdict stays truthful.

The admin is warned at every step: each bypass is logged, install/upload/link/
update responses carry a `trekRangeBypassed` marker, and both admin panels show
a "Version checks off" pill, an "Install anyway" confirm dialog before a
registry install, the same notice after a path that could not ask first, and a
persistent chip on the row while the plugin runs outside its range.

Nine new i18n keys in every locale; wiki pages updated (Environment-Variables,
Admin-Plugins, Plugin-Development, Security-Hardening).

* refactor(plugins): one useRangeBypass hook behind both admin shells

SonarQube flagged PR 2226 at 10.8% duplicated new code: the range-bypass
logic had been added to the desktop and phone plugin panels as identical
copies. The marker/warning types, the row chip, the "Install anyway" offer,
the dialog copy and the confirm-or-notice state now live in
useRangeBypass (next to useInstanceSettings, the existing shared-logic
precedent); each panel keeps only its own dialog/sheet markup.

* docs(claude): record the SonarQube gate rules and the no-English-placeholder i18n rule

The CI SonarCloud gate (Sonar way, on new code: >=80% coverage, <=3%
duplication, A ratings, hotspots reviewed) was only discoverable by waiting
20-25 minutes for the workflow. The root CLAUDE.md now states the conditions,
the desktop/mobile shell-mirror duplication trap and its shared-hook fix, and
how to check a PR with the sonarqube MCP first; client and server CLAUDE.md
point at it. shared/CLAUDE.md says every locale needs a real translation —
parity cannot tell an English placeholder from a translation.

* docs(claude): keep CLAUDE.md to commands, invariants, gates and philosophy

Verified every concrete claim in the five CLAUDE.md files against the code
and fixed the stale ones: the websocket server is the Nest realtime gateway,
MCP tools live in <domain>.mcp.ts files, the client "native shell" section
described code that was never written, offline maps also prefetch vector
basemaps, plus several renamed files and wrong table names.

Then trimmed everything that drifts on its own: dates, migration history,
counts, and inventories of stores, slices, domains, sync files and CLI
commands. What stays is the commands, the invariants with the file that
enforces each one, what CI actually runs, the reference implementations to
copy, and the rules for new code. Each file now opens with that rule.
2026-09-04 00:43:06 +02:00
Maurice 74c47a6158 docs(wiki): add the feature-requests banner used by the release notes
Release notes link the banner from the repo rather than from a comment
attachment, so it keeps working when the notes are edited later.
2026-09-03 16:18:21 +02:00
jubnl b82962f95b Feat/plugin instance actions (#2209)
* feat(shared): scope plugin action descriptors

* feat(plugin-sdk): instance-scoped settings actions

* feat(plugins): run instance-scoped actions from the admin settings dialog

* feat(admin): instance-scoped plugin actions in the settings dialog

* fix(plugins): surface instance actions without instance fields and harden the action gate

* docs(plugin-sdk): note that hosts before 4.2.0 ignore action scope

* fix(admin): open the dialog for instance actions, explain inactive 404s, guard the save window

* docs(wiki): plugin pages for instance actions and the admin panel changes
2026-09-02 23:06:03 +02:00
Maurice b6a86275f6 fix(notifications): the smtp test names its failure and the log carries it (#2196)
testSmtp logged nothing and the transport ran on nodemailer's two-minute defaults while the client gave up after eight seconds, so every failure was a generic toast over an empty log. The transport gets real timeouts, every outcome is logged without credentials, and a classified, scrubbed cause (auth, DNS, refused, filtered port, TLS mismatch, certificate) goes into the response the admin sees; the channel test calls wait long enough to receive it. The password field also rendered its mask as value, so replacing a stored password silently saved the bullets as part of the new one; it is a placeholder now, on desktop and the phone admin.
2026-09-02 21:06:42 +02:00
jubnl 1e6ab069cd Fix/sdk default (#2199)
* feat(plugin-sdk): settings-field default + unknown-attribute warning

* feat(shared): default on the plugin settings-field contract

* feat(server): persist and serve plugin settings-field defaults

* feat(client): pre-fill plugin settings defaults in the admin and user forms

* feat(server): enforce required plugin settings on both save paths

* feat(client): block plugin-settings save on an empty required field

* feat(server): resolve plugin settings defaults at runtime

A settings-field `default` was stored and served to the form but never
reached the plugin: the child's init envelope, `ctx.settings.get()`, the
channel dispatch's readAll and the OAuth provider config all read the
stored blob only. Fold `stored ?? default` in on every path that hands
config to a plugin (settings-defaults.ts), and let a default satisfy
`required` on both save gates — refusing a save the runtime would have
accepted was a contradiction.

Also: one `isFilled` rule shared by the save gate and `hasRequired`
(checkbox exempt, whitespace empty); the install parser drops a
non-boolean checkbox default or a select default outside its options,
matching what the SDK already refuses; export SETTING_FIELD_KEYS so the
SDK's mirror can be parity-tested.

* feat(client): share the plugin settings form rules across all three forms

The mobile user settings screen was a third consumer of the fields/config
contract that never got `default` pre-fill or the required check, and
the seed/patch loops lived in five hand copies. Extract
seedSettingsValues / findMissingRequired / settingsPatch into one module
and use it from the admin instance form, the desktop user tab and the
mobile screen. The user forms now surface a 4xx `{ error }` from the
server (the required-field refusal) instead of the generic toast.

* feat(plugin-sdk): seed dev fixtures from manifest defaults

`trek-plugin dev` handed the plugin only the fixture values, so a plugin
relying on its manifest defaults read undefined in dev and only in dev.
Seed ctx.config / ctx.settings from the declared defaults (a fixture
value still wins) via a new exported settingDefaults(manifest, scope).
SETTING_FIELD_KEYS is now parity-tested against the host's list.

* docs(plugins): document settings defaults as runtime fallbacks

A `default` is the field's effective value wherever nobody set one, not
just a form pre-fill; `required` is enforced on both ends; unknown
attributes are dropped at install and warned on by validate. Add a
Settings section with the attribute table to the SDK README and update
the ctx type comments, the shared schema, and the wiki guide.

* add missing exports
2026-09-02 18:42:16 +02:00
jubnl 54b0e4eed4 Fix/sdk screenshot gate parity (#2164)
* fix(sdk): grade docs/screenshot.png exactly, like the registry gate

* docs(wiki): document plugin MCP tools and the exact screenshot gate

* test(sdk): cover the preflight screenshot probe
2026-08-31 14:32:53 +02:00
Maurice 3c865c9605 docs(wiki): document plugins:use
The scope that turns on third-party tool execution was the one missing from the
scope reference, and the count beside it was stale. It says what the scope does
not do, since the boundary is the admin's mcp:tools grant rather than anything on
the token.
2026-08-29 12:22:59 +02:00
Maurice b401da2264 fix(maps): default the Leaflet basemap to OpenFreeMap
CARTO started stamping "API KEY REQUIRED" across every keyless basemap tile
on 26.08.2026. #2060 added a key field as the immediate answer, but it leaves
a self-hosted instance needing an email exchange with CARTO before its maps
look right, and every default in the client still pointed at CARTO. Default
to OpenFreeMap Positron instead: no key, no account, no request limit,
commercial use allowed, attribution required.

OpenFreeMap serves vector tiles only, so the Leaflet basemap is a MapLibre
style document rather than an XYZ template. maplibre-gl-leaflet hangs a GL
canvas into Leaflet's own tile pane, which leaves everything above it alone -
markers, GeoJSON, clusters, plugin layers, the custom panes, and the click
handling, since Leaflet's CSS already puts pointer-events:none on that layer.
maplibre-gl still arrives only through a dynamic import, so no chunk that
draws a map grows by a megabyte.

Switched: the trip planner, the MapView fallback every preview goes through,
the journey map, collections, the atlas, the shared trip page and the offline
tile prefetch. OpenFreeMap Positron and Bright join the tile presets in
settings, the admin defaults and both mobile screens.

A CARTO template that is already saved keeps working when a key backs it -
that stays a supported choice. Without a key it is not drawn at all:
resolveTileUrl falls back to the default rather than to a watermark, so an
instance that upgrades with one in its settings sees the new basemap
immediately and needs no migration. The stored value is left as it is, so the
Map settings tab still shows what was chosen and the warning underneath it
explains why - one sentence appended, in all 23 locales.

OpenFreeMap asks for a credit of its own; OpenStreetMap alone under its tiles
is wrong and a licence problem. attributionForTile() derives it from the URL
rather than from a flag at each call site, and the shared trip map, which
carried no attribution at all, now gets one.

Offline maps keep working and get cheaper. A vector style is four things
instead of one, so sync/glPrefetcher.ts walks the style document, its sprite,
the Latin glyph ranges of each font stack it uses, the TileJSON - the planet
path is versioned, so guessing it answers 200 with an empty body - and then
the .pbf tiles. Over the Berlin bounding box TREK computes, raster z10-16 is
2237 requests and 34.6 MB and gives up above z16; vector z10-14 is 177
requests and 32.1 MB and stays sharp above that, because MapLibre scales the
z14 tile rather than asking for another. The Workbox rule for
tiles.openfreemap.org moves to CacheFirst and shares the prefetcher's cache,
so browsing tops a downloaded region up instead of filling a second copy
beside it - a versioned tile path can never be stale. Clearing offline maps
clears it too.

The atlas loses its second raster layer. It existed to warm the HTTP cache of
neighbouring zoom levels, which a vector basemap does not need; a second one
would only cost a second WebGL context. OpenFreeMap has no label-free style,
so its labels are switched off on the live map instead - they are all symbol
layers, while borders and coastlines are line and fill - re-armed on
style.load so a theme switch keeps them off.

Theme changes call setStyle in place rather than rebuilding the layer: a swap
per toggle would leak a WebGL context, and browsers cap those in the low
teens.
2026-08-29 00:51:43 +02:00
Maurice 661777f81f feat(plugins): let a plugin attach a photo to a journal entry
A journal importer can create the journey, create its entries and write their
stories, and then it stops: there was no way to give an entry a picture. The
seven journal.* methods cover text and nothing else, so an importer reading an
export archive could only upload its photos as trip files, which live on a trip
and never reach the journey's gallery.

journal.addEntryPhoto takes the bytes. Deliberately not a link to an existing
gallery photo and not a provider asset: an importer holding an archive has
neither, it has a file it just unpacked.

The access decision stays where it already was. addPhoto answers null unless the
acting user may edit the journey the entry belongs to, and null becomes
RESOURCE_FORBIDDEN here, so a plugin reaches exactly the entries its acting user
does. Everything the REST upload refuses, this refuses too: images only and no
SVG, the operator's allowed-file-types setting, 10MB decoded, demo mode. The
stored filename is the host's, never the plugin's, because the gallery dedupes
on file_path. The object is written before any row points at it and removed
again when the link is refused.

JournalRpc moves into its own module for this. It now needs StorageService and
the EXIF backfill out of MemoriesModule, and JourneyDomainModule exists
precisely so places, assignments and the plugin host can reach a few journey
functions without dragging both photo providers into their graphs.

Discussion #1365
2026-08-28 18:04:25 +02:00
Maurice 5a7b8d75aa feat(api): add GET /api/v1/stats for dashboard widgets
Homepage's customapi widget and everything shaped like it want a handful
of numbers from one request; /api/v1/trips hands out the rows, but a
widget maps fields, it cannot aggregate a list.

The figures come from AtlasService.getTravelStats, the same call behind
the dashboard's passport card, rather than from a second count. Visited
countries carries years of rules — countries reached only by a flight,
layovers that do not count, countries hidden by hand — and a second
implementation would disagree with the card next to it within a release.

That is also why the controller sits in atlas/ and not in public-api/:
PublicApiModule stays a leaf, so the route comes to the data instead.
TravelStatsController next door already does this for
/api/auth/travel-stats. ApiTokenGuard is provided locally rather than
imported with PublicApiModule — @UseGuards instantiates a guard in the
declaring controller's module, so an export would still leave its
TokenService unresolvable here.

last_trip means the most recently started trip, not the most recently
created one, and is null while every trip is still ahead.

Closes #1367
2026-08-28 15:23:05 +02:00
Maurice 1e1a80c199 feat(api): expose shortlisted places, detached bookings and the bucket list
Reading the surface back against real data turned up three things it dropped
without saying so.

Places with no day were invisible, because the payload only reached places
through day_assignments. On a live instance that is half of them, and they
are the ones carrying a coordinate and no claim about when — exactly what an
integration can act on. They come back under `unplanned_places`, minus the
hotel, which is already reported as an accommodation.

Bookings with no day were dropped by the day filter. reservations.day_id is
nullable and deleting a day detaches its bookings rather than removing them,
so this is not a hypothetical.

The bucket list had no endpoint at all. It hangs off the user rather than a
trip, so it gets its own, and it answers whether or not the Atlas addon is
switched on: the addon decides what TREK shows, not what a key can read.

Also fixes ?include=notes returning the bare trip: places, notes and
reservations are reported on days, so asking for one now implies days
instead of quietly answering with nothing.

The module stays a leaf. Reaching for AtlasService pulled AuthModule and the
storage registry in behind it and broke the surface's own boot, so the one
SELECT lives here like every other query in the file.
2026-08-28 15:23:05 +02:00
Maurice 4b120d39c8 feat(api): separate API keys from MCP tokens, add travellers
An MCP token drives every assistant tool; an API key reads trips over HTTP.
Handing the same credential to a third-party integration gives it the whole
MCP surface, so the two are now separate kinds on `mcp_tokens` and the kind
sits in the WHERE clause — a token of the wrong kind is indistinguishable
from one that does not exist.

Also adds `travellers` to the trip payload: display name and an owner flag,
no ids and no email addresses. An integration key reads its owner's
itinerary; it is not a way to enumerate the people around them.

Documents the surface in the wiki, since integrators need something to read
that is not the source.
2026-08-28 15:23:05 +02:00
Maurice 8641788de0 docs(wiki): the new MCP surfaces
Files, settings, calendar feeds, invite links, the AirTrail import, provider
photos, help and the addon list. Also says why accepting an invite and
reading photo bytes have no tools.
2026-08-28 14:20:02 +02:00
Maurice ab3751761c feat(mcp): files and settings scope groups
Two domains an assistant cannot reach at all today need somewhere to hang,
so this is the credential half on its own, ahead of the tools.

files:content is a third mode rather than a second group. Listing the
documents on a trip and reading the bytes inside a booking PDF are different
privileges, and the plugin host already draws that line between
db:read:files and db:read:files:content. It is not implied by files:write,
the same way journey:share is not implied by journey:write; the policy's
fallback branch covers both.

settings:write is scoped to display preferences. Settings also holds the
webhook URL, the ntfy token and the Mapbox, Carto and LLM keys, so the tools
that ride this scope work off an allow-list rather than off the settings
row.

34 scopes across 16 groups, translated into all 23 languages.
2026-08-28 14:20:02 +02:00
Maurice 2ba10a9126 docs(wiki): the tools and fields added across the MCP surface
Also drops the em dashes from the earlier entries in this branch.
2026-08-28 14:20:02 +02:00
Maurice f1bbd94f2b feat(mcp): booking link and end time on a reservation
url is persisted on create and update and rendered as a link by the
planner, and reservation_end_time is what makes a dinner run 19:00-21:30
rather than just starting. Neither was in a tool schema, so a booking
written by an assistant lost both - which for an imported itinerary is
every confirmation link it came with.

url validates through reservationUrlSchema from @trek/shared, the same
refinement the REST route applies, rather than a second hand-written one.
2026-08-28 14:20:02 +02:00
Maurice aa5002b2ed fix(mcp): keep transit out of the transport create enum
The transport form's picker has nine entries, not ten: a transit booking
carries a provider itinerary in metadata.transit and create_transit_journey
is what writes one, so letting a caller hand-make a bare transit row
produces a shape the transit UI does not expect.

Splits the one list in two - what may be asked for, and what counts as a
transport for the update gate - so a stored transit booking stays editable
through update_transport, which is where it was already reachable.

Also reports the ids set_reservation_travelers dropped instead of
returning a plain success with a shorter list.
2026-08-28 14:20:02 +02:00
Maurice 86c0c197db feat(mcp): set who is travelling on a booking
Reads have hydrated a booking's travellers since #1517; there was no way
to write them, so an imported booking could not record who was on it -
and creating a guest is only half an answer if the guest cannot then be
attached to the flight they were on.

Mirrors PUT /api/trips/:tripId/reservations/:id/travelers. The roster
filter stays in the service, so an off-trip id is dropped rather than
attached and the tool cannot be used to enumerate users.
2026-08-28 14:20:02 +02:00
Maurice 4b21a70ab2 feat(mcp): add, rename and delete trip guests
add_trip_member resolves an existing account by username or email, so a
travelling companion who never signed up could be put on a trip in the
planner and not through an assistant. A guest is assignable to budget
splits, packing, to-dos and day participants like any member, so leaving
them out of an import loses who was actually there.

Mirrors the owner-only /api/trips/:id/guests routes, including the
trip-scoped guard that keeps one trip's owner away from another trip's
guest and off a real member's user row.
2026-08-28 14:20:02 +02:00
jubnl d8fd2deb12 feat(sdk): allow new signing key (#2088) 2026-08-28 00:56:44 +02:00
Maurice 1f66ef1d32 fix(maps): let users and admins enter a CARTO API key (#2060)
CARTO stamps an "API KEY REQUIRED" watermark onto every basemap tile
fetched without a key since 26.08.2026, and both CartoDB presets plus the
built-in fallback are CARTO. The key is free and needs no account, so add
it as a setting rather than dropping the provider.

carto_api_key sits next to mapbox_access_token: per user in Settings/Map,
instance-wide under Admin/User Defaults, encrypted at rest, injected from
CARTO_API_KEY on managed instances. Unlike Mapbox it never switches the
map provider, since a raster key says nothing about the renderer.

The key is appended as ?key= at render time instead of being stored in the
template, so a saved URL stays portable and survives a rotation. That is
also what reaches the four maps that pinned CARTO and ignored the user's
tile setting: atlas, collections, the public trip share and the public
journey. The two share pages read it from their token-gated payload, the
same way baseCurrency already travels.

Also lifts the release notice above page chrome. Its overlay sat at
z-index 50, under the Vacay mode toolbar at 61, so the dialog was neither
on top nor modal. Adds a six-step layer scale to index.css and moves the
three notice shells onto it.

Closes #2054
Closes #2052
2026-08-27 12:06:55 +02:00
Maurice 521e52c853 fix(collab): harden the link preview fetcher (#2058)
The preview endpoint takes a URL from whoever is typing in a trip chat and
fetches it from the server. Trip access was added to it earlier; this is the
rest of what that route needed.

The SSRF guard had a hole under it. `isAlwaysBlocked` knew `::1` and `0.0.0.0`
but not the IPv6 unspecified address: `::` starts with a colon, so the `0.`
check never saw it, and connecting to it lands on loopback. `http://[::]:9200/`
therefore passed the guard and came back with the target's title and og: tags
scraped out of the body — an exfiltrating read of anything listening locally,
from any account. `[::0]`, `[0:0:0:0:0:0:0:0]` and `[::ffff:0:0]` are the same
address in other spellings and behaved the same way. The two `::ffff:`-prefix
regexes had the same shape of gap: they matched the dotted spelling of loopback
and link-local, so `::ffff:7f00:1` walked past them. Both checks now decide on
the expanded hextets, which covers every spelling at once, and fe80::/10 is
read as ten bits rather than as the four characters 'fe80'. This is in
ssrfGuard, so every caller gets it, not just collab.

The scrape could be turned into a denial of service. Each og: regex had two
unbounded `[^>]*` runs separated by a literal, so a document with no `>` in it
made the engine rescan the remainder from every `<meta` it passed — quadratic,
and the byte cap admits half a megabyte. Measured on this build: 240KB of
'<meta ' took 58 seconds, and Node has one thread, so that is the entire
server, health check included, on one request. Bounding the runs at 512
characters makes it linear (540KB now takes 1.4s); no real attribute list comes
close to that.

The five-second timeout only covered the headers — `clearTimeout` ran before
the body was read — so a server that answered fast and then dripped one byte
every few seconds held the handler, the socket and the per-request undici agent
open indefinitely. The byte cap counts bytes and at that rate would never reach
one. `AbortSignal.timeout` covers both halves, the agent is closed in a
`finally`, and the three bail-outs that used to return without touching the
body now discard it, which is what cappedFetch's helper is there for.

The rest is smaller. Nothing metered the fetches, so an account with a trip of
its own could drive arbitrary outbound requests in a loop; there is a budget of
sixty new fetches a minute per user now, charged per outbound fetch rather than
per request. Previews are cached for ten minutes and simultaneous askers for
one URL share a single fetch, so the client's habit of requesting a preview per
rendered message — up to a hundred at once, with no debounce — costs one fetch
per new link rather than one per render. Refusals answer with one constant
reason instead of the guard's own: it distinguishes "could not resolve" from
"private address", which together map out the server's internal DNS for anyone
willing to guess hostnames. A body whose declared type is not markup is dropped
rather than combed for og: tags, and an og:image that is not an http(s) URL is
not passed to the client, which puts it straight into an <img src>.

Deliberately no `@RequirePermission` on the route: the client asks for a
preview while rendering a message or a note, never while composing one, so it
belongs with the GET siblings. Requiring collab_edit would blank every preview
for members of a trip whose owner narrowed that right while leaving them able
to read the chat, and it would not stop the abuse it looks like it addresses —
the permission check passes for a trip's owner at every level, and anyone can
create a trip.
2026-08-27 12:03:04 +02:00
Maurice f052d9d26b docs: the storage seed example lives in the wiki now
Admin-Storage pointed at the README's Storage section for the seed file
example and the compose mount line. That section is gone, so the link went
nowhere and the example existed in no copy at all - the wiki page described
storage-config.json without ever showing one. Both are on the page now.

The commented mount in docker-compose.yml pointed at the same removed
section.
2026-08-25 22:54:54 +02:00
Maurice ad2119efd6 docs(wiki): correct what reading each page whole turned up (#2036)
The last pass reviewed diffs, so it never saw a new paragraph contradicting an
older one on the same page, or a long-standing claim nobody had checked.
Plugin authors were told ctx.config carries their per-user secrets and that
definePlugin takes a permissions key — neither is true, and both fail silently.
FORCE_HTTPS was said to need TRUST_PROXY, which it does not.
2026-08-25 15:22:46 +02:00
Maurice 7da0080557 docs(wiki): bring the wiki in line with 4.0.0 (#2034)
* docs(wiki): correct the install and operations pages for 4.0.0

Quick Start passed docker an empty ENCRYPTION_KEY: an assignment prefix is not
expanded on the same command line, so the server silently generated its own key
instead of taking the one the page told you to set. The tag examples pinned to
the 3.x line, the documented tmpfs size did not match the shipped compose file,
and several .env values were described as reaching the container when compose
never interpolates them.

* docs(wiki): match the MCP reference to the registered surface

The scope table was missing the whole Collections group and the totals said 27
scopes across 13 groups where the code registers 29 across 14. The Cloudflare
link on the setup page pointed at an anchor on its own page instead of the
heading on Troubleshooting.

* docs(wiki): correct the plugin and admin pages

Three links to Plugin-Development carried a .md suffix, which does not resolve on
the wiki. The admin pages described tabs, permissions and storage behaviour that
had moved on.

* docs(wiki): bring the feature pages back in line with the app

Costs still documented the old panel, where categories could be added, renamed,
reordered and deleted; they are fixed now and the panel is a day-grouped ledger
with settlements. Trip sharing, journeys, maps, packing, reservations and the
account pages had all drifted in the same way.

* docs(wiki): cover the pages the first pass did not reach

The MCP catalog listed 168 of the 201 registered tools; the settlement tools and
rate_place were missing here and the Collections family from the addon page.
Several resource rows claimed a trips:* scope where the handler asks for
trips:read.

* docs(wiki): tighten the corrections a review picked apart

The audit-log resource is the row id, not the credential id; an admin password
reset does not bump password_version the way the page implied; and a few of the
new paragraphs described behaviour more confidently than the code supports.

* docs(wiki): fix what a last read-through caught

Costs shows every figure in your display currency, not the trip's, and an
expense with no payer still counts against the members it is split between.
Studio's cutoff is a 1024px viewport rather than phones alone, and the
Integrations tab is gated on the providers an admin has configured.

* docs(wiki): the last few the review caught

A passkey satisfies the MFA policy everywhere except the WebSocket handshake,
which still asks for TOTP, so a passkey-only account under that policy gets no
live updates. Mobile optimizes a day from two places, not three. The ctx.db row
carried an unescaped pipe that split the table, and the Backups table linked
with a wiki link whose pipe did the same.
2026-08-25 12:36:52 +02:00
jubnl d110b3bb6d Fix/plugin audit findings (#2031)
* fix(plugins): enforce manifest apiVersion at install and activation

* test(plugins): exercise the absent-apiVersion default

* feat(plugins): generate the core-event catalog into host-facts

* docs(plugins): explain the packing.listBags write-permission binding

* test(plugins): cover apiVersion upload refusal; tidy docblocks and generator header

* fix(plugin-sdk): validate settings options and oauth like the host

* fix(plugin-sdk): model the full manifest shape and stop dropping fields

* fix(plugin-sdk): drop non-string routeProfiles icons like the host

* docs(plugin-sdk): document host limits, oauth contract, and ignored asUserId

* feat(plugin-sdk): enforce daily budgets and meta quotas in the mock host

* feat(plugin-sdk): expose the generated core-event catalog

* feat(plugin-sdk): scaffold trek range >=3.4.0 <5.0.0 ahead of TREK 4.0

* feat(plugin-sdk): scaffold the oauth broker settings for oauth:client plugins

* chore(plugin-sdk): fix stale mirror docs and the colliding permissions export

* docs(plugin-sdk): add the rate-limit string; restore trek docblock; match host meta sizing

* feat(plugin-sdk): raise the scaffold trek floor to 4.0.0

* docs(wiki): bring the plugin pages in line with the current runtime

* docs: bring the workspace CLAUDE.md guides up to date and make them committable

* feat(plugin-sdk): roll back failed publishes, add unrelease, sync the fork, retro-sign updates

* docs(wiki): document the publish rollback, unrelease, fork sync, and retro-signing

* chore(plugin-sdk): 1.6.0
2026-08-24 23:48:36 +02:00
Maurice 38f3cb3101 fix(server): the second pass over this branch's own fixes
An adversarial review of what had already landed here turned up three that
were wrong, and they are the reason for this commit.

The CSP change removed 'unsafe-eval', which is not leftover: heic-to's
libheif build initialises embind through new Function(), and without it every
iPhone .heic upload fails in the browser. 'wasm-unsafe-eval' alone was tried
once already and was not enough (93b51a0b). Put back, with the reason written
down this time and the test pinning it rather than its absence.

The idempotency in-flight guard never fired. finalize() runs when the
handler's observable completes, but Nest writes the response - and therefore
stores the row - several microtasks later, so the waiting request always woke
up before the row existed and ran the handler a second time. It releases from
the point the row is written now; finalize is the backstop for handlers that
answer another way. The test drove the two in the opposite order from Nest,
which is why it passed either way.

And the trip-scoping check treated an id that resolves to nothing like one
belonging to someone else. accommodation_id has no foreign key, so shortening
a trip's dates leaves a booking pointing at a gap - which update() has always
healed by clearing the link, and which would have become a permanent 400.

The rest: a reply to a soft-deleted chat message is refused and the create
path blanks the quote like the list path does; the ws-token buckets are keyed
by account rather than by IP, where every user behind one NAT shared a
ceiling; the weather and vacay caps read through the stream, since a chunked
response has no content-length to check; the Google host gate takes country
domains again; abandoned response bodies are cancelled instead of left to GC;
the boot failure prints its errno; and a share link that is not yours to read
now says so instead of reporting that the journey has none.

TICKETJSON gets a real repair: the in-place GLOB fix only helps installs that
have not run it yet, so an appended migration puts back the note text the
case-insensitive LIKE cut off, for rows where what it stored is not valid
JSON. A genuine receipt always parses, so it cannot be touched.

Wiki: the audit log records the IP Express resolves after TRUST_PROXY, not
the leftmost X-Forwarded-For entry a caller can write. That is the right way
round, but it means the hop count has to be accurate - said plainly in both
places now.
2026-08-23 11:59:12 +02:00
jubnl 305b113d5f Feat/storage driver (#1949)
* feat(storage): storage key validation, driver contract suite, LocalDriver

* feat(storage): MirrorDriver composite backend

* feat(server): add TREK_UPLOADS_DIR env var for the uploads root

* feat(storage): settings-backed registry, StorageService facade, module wiring

* test(storage): pin src/nest/storage coverage thresholds

* feat(storage): multer upload options factory

* refactor(files): spool trip-file uploads through storage

* refactor(collab): spool note attachments through storage; normalize files/ prefix

* refactor(journey): spool journey media uploads through storage

* refactor(trips): spool trip cover upload through storage

* refactor(places): spool place-image and collection uploads through storage

* refactor(auth): spool avatar upload through storage

* fix(server): consolidate allowed_file_types onto AllowedFileTypesService

* fix(storage): default defParamCharset to utf8 in the upload factory

* test(db): re-point the trailing-migration guard at the trip_files prefix strip

* fix(storage): age-gate the boot spool sweep

* test(files): key FILE-P02 residue checks on upload size

* test(storage): raise src/nest/storage coverage pin

* refactor(platform): serve /uploads routes through storage

* refactor(files): serve trip-file downloads through storage

* refactor(memories): serve photo bytes through storage

* refactor(maps): stream place-photo bytes through storage

* refactor(share): stream shared place-photo bytes through storage

* refactor(journey): serve public journey assets through storage

* fix(share): answer place-photo cache miss with 204 like the maps proxy

* test(server): close serving-swap gaps surfaced by the full tier

* test(storage): raise src/nest/storage branches pin

* refactor(place-photos): route place-photo cache through storage

* refactor(places): reclaim place images and collection covers through storage

* refactor(trips): trip covers through storage

* refactor(auth): avatar reclaim through storage

* refactor(collab): note-file deletion through storage

* refactor(files): trash deletion through storage

* refactor(memories): trek photo cache internals through storage

* refactor(memories): thumbnail service category-addressed

* refactor(memories): immich upload reads through storage; drop uploads-root

* refactor(journey): photo deletion through storage

* fix(memories): reclaim orphaned trek photo cache objects in the ttl sweep

* feat(journey): journey-thumbs orphan sweep and delete-site derived-artifact reclaim

* test(server): close slice-4 gaps surfaced by the full tier

* test(coverage): raise slice-4 domain pins

* refactor(backup): list, existence and delete through storage

* refactor(backup): download through storage sendToResponse

* refactor(backup): create backup through per-category storage enumeration

* refactor(backup): restore and upload-restore through storage

* refactor(backup): auto-backup retention through storage

* refactor(server): boot dir-creation to driver init

* test(server): close slice-5 gaps surfaced by the coverage tier

* test(coverage): raise slice-5 backup pin

* test(server): resolve async backupFileExists mocks for the tests typecheck

* test(server): pin ranged-read stat and zero-byte roundtrip in the storage driver contract

* feat(server): TREK_S3_* env wiring — schema, s3 namespace, all-or-nothing boot rule

* feat(server): S3 driver core — aws-lite client seam, stat/delete, driver-owned deadlines

* refactor(server): type the aws-lite S3 client via s3-types reference

* feat(server): S3 driver getStream and list — ranged full-object stat, iterator pagination

* feat(server): S3 driver put — bounded-peek routing, Upload inactivity deadline

* fix(server): S3 driver Upload monitoring — Transform interposition, stream cleanup

* feat(server): env-declared s3-main storage backend; settings-declared s3 rejected

* test(server): env-gated S3 driver contract suite against MinIO

* fix(server): S3 driver zero-byte put — work around aws4 duplicate Content-Length signing defect

* fix(server): scope the zero-byte put workaround to buffered bodies; bind test MinIO to localhost

* ci(server): MinIO-backed S3 driver contract job

* ci(server): align s3-contract shim comment and MinIO bind with references

* docs: S3 storage backend configuration and off-box backup mirroring

* docs: restore README formatting, keep S3 additions only

* ci(server): make live-MinIO contract runs manual-only (AIStor licensing)

MinIO's community line is discontinued entirely; the maintained successor, AIStor, requires a license key that can't safely live in a shared CI runner. The s3-contract CI job is removed; live-provider contract runs against MinIO/AIStor are now a manual, license-gated procedure documented in docker-compose.minio-test.yml's header.

* chore(server): final-review polish — stale CI refs, dockerignore key, boundary pins, zero-byte tempfile contract case

* expose admin ui

* feat(shared): storage backend type registry, wire schemas, category relocation

* feat(server): accept settings-declared s3 backends — shared-schema options, decrypt-at-build, registry preview()

* feat(server): remove TREK_S3_*/TREK_UPLOADS_DIR — tombstone boot rule, built-in override as the relocation path

* feat(server): deriveSecurity.encryptionKeySet and storage secret plumbing (gate, encrypt, unmask, mask, redact)

* chore(server): drop storage-secrets import that only Task 8's probe helper will use

* feat(server): seed-once storage-config.json boot import — fail closed on any invalid seed

* feat(server): registry snapshot() — effective storage config with provenance

* feat(server): api/admin/storage GET/PUT — masked secrets, preview-gated saves, wired reload, audits

* feat(server): api/admin/storage/test — per-target ephemeral probes, mirror targets probed individually

* test(server): e2e coverage for api/admin/storage — guards, managed refusal, envelope, probes

* test(server): give e2e harness DBs the app_settings table the storage registry reads at boot

* test(coverage): raise pins after the storage admin-config slice

* fix(server): final-review polish — encryption-gate wording, all-field sentinel guard, README/e2e notes

* test(server): storage slice-1 review follow-ups — mirror s3-replica probe decrypt, all-encrypted seed without key, test-route 401

* feat(i18n): storage admin domain + tab label in every locale

* feat(client): adminApi storage endpoints — getStorage, updateStorage, testStorageBackend

* feat(client): storage admin draft-document helpers + relative-time util

* feat(client): schema-driven storage BackendForm — fields by kind, mask echo, encryption banner

* feat(client): admin storage panel — backends list, category map, health strip, one-PUT save

* feat(client): storage tab on the admin page

* feat(client): storage admin panel and section on the phone shell

* fix(client): block renaming a storage backend onto an existing name

* feat(server): allow mirrors on any storage category — lift the backups-only rule

* feat(i18n): replicas-on-primary mirror strings in every locale

* feat(client): mirror fold/synthesize model helpers for replicas-on-primary

* feat(client): BackendForm mirror-targets composer; raw mirror type hidden

* feat(client): desktop storage panel folds mirrors into their primaries

* feat(client): phone storage panel folds mirrors into their primaries

* fix(client): strip removed backends from remaining mirror replicas — no phantom refs

* feat(storage): retire photos as a configurable category — served-only via the ServedCategory seam

* feat(i18n): descriptive storage category names and descriptions in every locale

* feat(client): descriptive category names, descriptions and id badges in the storage panel

* docs(storage): fix stale nine-category comments after the eight-category shrink

* docs: storage configuration via the admin UI and seed file — README + compose

* docs(wiki): Admin Storage page and storage cross-links

* docs(server): storage domain blueprint entry; TREK_PLACE_PHOTO_DIR tombstone clarifier

* refactor(server): drop the storage env-var tombstones — the variables never shipped

* docs: storage config is simply new in v4 — drop the removed-variables story

* feat(shared): backfill-status and usage wire schemas on the storage admin state

* feat(server): MirrorDriver.backfill — prefix-scoped, size-compared, cancellable replica catch-up

* feat(server): one-at-a-time storage backfill jobs + the storage events bridge

* feat(server): storage usage scan — nightly cron, on-demand service, persisted with computedAt

* feat(server): replica_failure admin notifications — events bridge, hourly debounce, channel registration

* feat(i18n): replica-failure notification text in every locale

* feat(server): storage backfill and stats-refresh admin endpoints

* feat(client): adminApi backfill and stats-refresh endpoints

* feat(i18n): storage sync and usage strings in every locale

* feat(client): desktop storage panel — sync now, usage display, backfill polling

* feat(client): phone storage panel — sync now, usage display, backfill polling

* fix(client): per-category usage line on the phone storage panel

* docs: sync now, usage stats and replica-failure notifications

* test(server): admin scope now includes replica_failure, not just version_available

Commit 6db135da added replica_failure to ADMIN_SCOPED_EVENTS but left
NPREF-005b asserting the admin scope contained only version_available.
Update the expectation to match.

* test(server): cover storage backfill/stats/jobs branches to hold the coverage pin

* fix(client): guard storage admin polling against stale responses racing a save

A poll GET issued before a save's PUT resolves could land after the save
applied its fresher response, overwriting it with the pre-save world and
re-deriving the draft from that stale snapshot. useStorageAdmin now tracks
a state-application sequence number and drops any refreshState() response
whose sequence has moved on, or that resolves while a save is in flight.
Also guard the fire-and-forget refreshState() calls in startBackfill/
cancelBackfill against unhandled rejections, matching the poll's style.

* fix(server): report 404 before 409 when a backfill start names an unknown mirror

startBackfill checked the global busy flag before validating that :name is
a mirror routed by some category, so POST .../backends/garbage/backfill
answered 409 instead of 404 whenever a sync was already running. Resolve
and validate the target first (BackfillTargetError -> 404), then check
busy (BackfillBusyError -> 409).

* feat(client): two-column category grid on the storage panel

* feat(server): plugin files.getContent reads through the storage layer

* feat(server): plugin files.create writes through the storage layer

* refactor(server): drop FilesService.resolveFilePath, last direct-fs seam in the plugin surface

* fix(server): map StorageInvalidKeyError to the plugin file accessibility envelope

* feat(storage): allow credentialed backends without an explicit ENCRYPTION_KEY

Secrets are still always encrypted at rest; without an explicit key the
implicit data-directory key is used. Removes the plaintext-secret refusal
from the admin PUT and the seed import, the encryptionReady GET field, the
save-blocking banner in both admin shells, and the now-consumerless
security.encryptionKeySet env derivation. Docs now recommend (not require)
an explicit key, since the implicit key rides inside backup archives.

* test(client): pin the stale-poll release to the pre-save GET in FE-ADMIN-STOR-024

The stub deferred EVERY poll GET and rebound the release handle to the
newest one, so a tick landing after the save was released with the
pre-save payload — a response no real backend can produce, which the seq
guard rightly applied. Only the first (genuinely stale) GET is held now;
later GETs answer with the world the mock server currently holds. Also
drops the encryptionReady fixture field removed from the schema in the
previous commit.

* fix(storage): backfill puts carry content-type, stat failures report op stat, progress clamps

* feat(storage): category migration job — copy, flip, delta sweep, reclaimable tally

* feat(storage): migration start/cancel endpoints and client API

* test(client): storage admin fixtures carry the new migrations field

* fix(storage): migration POST answers 200 and validates its body via the shared schema

* fix(storage): cancel running storage jobs when a config save removes their backend

* fix(notifications): in-app replica-failure row carries the suppressed count

* feat(client): migration prompt, queue, and progress on the storage panel

* fix(client): migration queue holds a synchronous single-flight lock

* feat(client): migration prompt and progress on the mobile storage panel

* fix(server): register category-migration routes in the managed-route allow list

StorageAdminController.migrationStart/migrationCancel were added under the
controller-level @ManagedForbidden() decorator but never added to
MANAGED_ROUTE_ALLOW_LIST, so validateManagedRoutes() threw "Managed surface
changed" on every buildApp() call, failing 46 test files (any test booting
the real app).

* test(server): cover category-migration copy/sweep branches to hold the storage coverage pin

Adds MIG-005 (a target object already matching size/key is skipped in the
copy phase itself, not just the delta sweep) and MIG-006 (a copy failure
during the delta sweep is counted without undoing the flip that already
happened). src/nest/storage/**'s branch coverage was 83.91%, just under the
84% pin.

* docs(storage): category migration — move existing objects on reassign

* fix(storage): close the pre-flip cancel window and cancel re-routed migrations

A cancel landing between the last copy iteration and the category flip was
silently ignored, and a config save that re-routed the migrating category to
a third, still-defined backend never stopped the in-flight migration. Also
guards assignCategory against writing an unknown backend name, and fixes a
missing `suppressed` notification param being treated as truthy suppression.

* fix(client): migrations keep mirrors, queue waits on backfills, sweep failures surface

Reassigning a category onto a mirrored primary now posts the migration to the
mirror's wire name instead of the bare primary, so replication survives the
move. The queued-migration effect also waits out a running backfill (the
server's one-job rule spans both), and a done migration with sweep failures
now surfaces a doneFailures line in both shells, translated into every locale.

* ci(server): run the S3 contract suite against MinIO

Resurrects the s3-contract job removed in 6fbff835, adapted for AIStor:
writes the MINIO_LICENSE_KEY secret to a license file, boots the licensed
container, and runs the contract suite hermetically (no --coverage) so the
storage coverage pin keeps holding from unit tests alone. Gated to same-repo
PRs since fork PRs have no access to the license secret.

* fix(server): backup archives uploads from a remote primary

createBackup() enumerated uploads through withLocalFile, which downloads a
remote object to a temp file, hands it to the callback, then deletes it
before archiver ever reads it — archiver reads entries lazily during
finalize(), so a remote (S3) primary silently backed up zero upload bytes.

Add StorageService.getLocalPathOrNull(category, name): a narrow locality
probe that returns a real path only when one exists on disk right now (the
fs-existence check is the fail-safe half of the contract — a driver-reported
path that vanished between list() and here falls back too, not just a
path-less remote driver). The enumeration loop uses it directly when
available (zero-copy, unchanged for a local primary) and otherwise streams
the object into a per-backup staging dir under the backups spool, cleaned up
in the existing finally alongside zipSpool/dbSnap.

Also wire archive.on('warning', reject): archiver treats a missing/unreadable
entry as a warning, not an error, and skips it by default — which would
silently drop the entry from an otherwise "successful" backup.

* fix(server): client aborts stop 500ing and leaking streams

storage.service.ts sendToResponse: the local res.sendFile branch now treats
ECONNABORTED/ECONNRESET/ERR_STREAM_PREMATURE_CLOSE/syscall=write as success,
mirroring res.sendFile's own default callback (the contract
storageStaticHandler already implements caller-side). The remote branch
replaces the hand-rolled pipe+Promise with pipeline(stream, res), which
destroys the source stream on any failure — fixing the leaked S3 keep-alive
socket on a client abort — while swallowing the same client-abort error set;
a real source error still rejects so callers keep their miss contracts.

This centralizes the fix for every caller that streams through
sendToResponse: files-download, journey-public, photo-resolver,
trek-photo-cache, backup downloads, and /uploads/photos.

trek-exception.filter.ts now no-ops (destroying the socket) when headers are
already sent, so no filter path ever writes onto a response in flight.

maps.controller.ts and share.controller.ts's place-photo bytes proxies
switch their direct getStream+pipe to pipeline() with the same abort-tolerant
catch, keeping the existing 204-empty fallback for pre-header failures.

* fix(storage): mirror read fallback consults replicas on primary miss

MirrorDriver.getStream/stat now fall back to replicas on a primary miss
(StorageNotFoundError / null stat), not just an infra error — binding
decision: a read succeeds if ANY member holds the object. Only a
deterministic StorageInvalidKeyError skips fallback.

stat() gets a dedicated fallback loop instead of reusing fromReplicas:
a replica returning null means "keep looking," which the generic
catch-and-continue helper would have short-circuited on.

StorageService.sendToResponse/withLocalFile stop treating composite
driver.stat() as proof the object is at localPath; both now fs-check
the path itself and fall through to the stream branch when it's
missing, which is what actually reaches a MirrorDriver's replicas.

* feat(storage): remote serving parity — content-type, validators, range

Local backends get content type, ETag/Last-Modified, conditional GET, Range
and HEAD for free from res.sendFile; the proxying branch every remote
backend (and a local primary whose file vanished) falls through to had none
of it — every request re-streamed the whole object with no cache validator
and no way to seek.

sendStreamed() reproduces the contract from ONE pre-flight driver.stat():
a miss throws before any header is set (the caller's 404/204/next()
contract), and that same stat then answers 304s, 416s, HEADs and suffix
ranges without ever opening a body. The decisions live in a new pure module,
http-serving.ts:

- validatorsFor: the driver's ETag, else express/send's weak
  W/"<size hex>-<mtime hex>". mtimeMs 0 (S3 can report it) means no
  Last-Modified and no derived tag — serving uncached beats a 1970 header
  that reads as fresh forever.
- isNotModified: If-None-Match (weak compare, *, list) wins over
  If-Modified-Since, per RFC 9110 §13.2.2.
- decideRange: single range only, inclusive clamp, suffix computed from the
  pre-stat size, unsatisfiable → 416, If-Range strong-exact or full entity.
  A multi-range or malformed spec is answered with the full object rather
  than multipart/byteranges. If-Match/If-Unmodified-Since stay out of scope.

Range math uses the pre-stat size as the single source of truth, the ranged
getStream stat is never re-consulted, and the pipeline abort semantics are
unchanged.

S3Driver now surfaces the bucket's ETag on stat()/getStream() (on a ranged
GET it is still the full object's tag). aws-lite strips the tag's
surrounding quotes — verified live against AIStor RELEASE.2026-08-07 — and
an unquoted entity-tag is not a valid HTTP ETag, so the driver restores
exactly the quotes the SDK removed and nothing else.

* fix(server): key rotation covers storage.backends

The S3 storage backend's secretAccessKey is encrypted at rest inside the
app_settings row keyed storage.backends, but scripts/migrate-encryption.ts
didn't know about it — after a key rotation that secret stayed encrypted
under the old key and silently failed to decrypt on the next read.

The script now re-encrypts secretAccessKey for every s3 backend inside the
storage.backends JSON array, using a pinned copy of the secret-field list
(it deliberately can't import @trek/shared). The parity test pins that copy
against storageSecretFields('s3') so a new secret field can't be missed
silently.

* fix(server): restore reloads the storage registry

restoreFromZip closed and reopened the DB but never reloaded the
storage registry, so uploads rehydration ran against the STALE
pre-restore driver/category map instead of the config the restored
archive's own travel.db carries (audit #4, major).

StorageService gains reloadConfig(), a narrow passthrough to the
(unexported) registry's reload() — same shape as health(). backup.impl's
restoreFromZip now calls it right after reinitialize() and right before
the uploads wipe/rehydrateUploads/plugin staging, so rehydrated bytes
land where the restored config says. When reopen fails (reinitFailed),
reload and rehydration are skipped entirely — already reported as
"restart required", and there is no live DB handle for the registry to
read anyway.

* fix(server): category migration rewrites keys across prefix changes

Category migration copied objects to the destination backend using the
SOURCE key verbatim. A mode-A ('' prefix) source migrating onto a
prefixed backend (or vice versa) landed objects at keys the registry
could never resolve post-flip (audit #8).

Extracts StorageRegistryService.keyPrefixFor(category, backendName) from
build()'s inline conditional so startMigration can compute the
destination prefix before the flip. runMigration now maps every
destination key (destPrefix + key.slice(sourcePrefix.length)) in the
copy skip-check, the put, and the delta sweep; the reclaimable tally
stays source-keyed. Equal-prefix migrations are unaffected (identity
mapping).

* fix(plugins): await the now-async place/collection services in RPCs

PlacesService.update/remove and CollectionsService.deletePlace became
async when they started deleting storage objects, but their plugin RPC
handlers still called them synchronously: the null/false refusal check
compared against an always-truthy Promise, and places.update broadcast
a Promise instead of the resolved place. Both handlers now await the
service call; mapCollectionError is async-aware (accepts a sync-or-
async thunk) so the sync callers (create/update/savePlace/copyToTrip)
keep working through the same path. Verified the RPC host already
awaits handler results, so no change was needed there.

* fix(storage): backfill Sync now gains a deletion sweep

The Sync now backfill copied primary objects to replicas but never removed
replica objects the primary no longer has, leaving mirror read-fallback able
to serve stale data indefinitely after a failed replica delete. backfill()
now sweeps each replica post-copy: any replica key absent from the primary
key-set is re-checked against primary.stat (a race gate against the primary
regaining the key) before being deleted. Adds a deleted counter end-to-end
(driver, jobs service, shared schema, both admin shells, all 23 locales) and
a new ReplicaFailure 'list' op for a replica whose listing itself fails
during the sweep.

* feat(storage): optimistic concurrency for admin config saves

A stale admin PUT of storage settings could silently undo a category
migration's just-flipped assignment. Adds a version counter
(app_settings row storage.config_version) bumped inside the same
transaction by both writers — applyConfig and the migration flip
(assignCategory). applyConfig compares the submitted version first
and throws StorageConflictError, mapped to 409 ahead of the blanket
400. The client carries the draft's loaded version through every
local edit and maps a 409 to a distinct "settings changed underneath
you" message, refreshing state while preserving the dirty draft for
review.

* fix(storage): surface a failed config load + redact secretAccessKey

Registry load() now records the exact parse/validate failure as
lastLoadError() instead of silently falling back to last-good config or
built-in defaults; StorageAdminService exposes it as configError on the
admin state (storageAdminStateSchema extended additively), and
AdminStoragePanel renders it as a warning banner naming the error. Save
stays enabled — it's the recovery path — but an admin can no longer save
over a broken stored config without knowing.

Also adds 'secretaccesskey' to globalMiddleware's SENSITIVE_KEYS request-log
redaction list, so storage admin PUT bodies never leak the S3 secret into
debug logs.

* fix(client): admin storage panel state races

Migration prompt now recomputes computeMigrationCandidates(draft, state)
on render and again at confirm time, so a stale open-time snapshot never
drives stripCategories or the migration queue; adds a Cancel button to
the dialog. A failed startMigration POST explicitly clears the remaining
queue with a toast naming the dropped categories and refreshes state,
instead of leaving it invisibly stranded; a "Queued: X, Y" line renders
while the queue is non-empty. Testing a mirrored row now expands the
draft primary + replicas client-side and probes each individually,
instead of sending the mirror stub to the server (which resolved
targets from its own live snapshot, silently testing saved options over
unsaved draft edits). A completed/cancelled/errored backfill now renders
its terminal status alongside the Sync now button instead of hiding it,
so it can be re-run without a page reload.

* fix(storage): refuse shared replica backends so the sync sweep cannot cross categories

A mirror's "Sync now" sweep makes each replica MATCH the primary: it lists the
replica under every prefix the mirror's categories occupy and deletes the keys
the primary doesn't hold. The backups category's prefix is '' — the replica's
entire root — so making a category-serving backend (say uploads-local) a
replica of a backups mirror meant the first sync deleted every user upload.
Nothing forbade the config: validateConfig only checked that mirror targets
existed and weren't nested, and the admin picker offered every backend.

The sweep is correct, so the fix is at the config level:

- validateConfig refuses a backend that is both a mirror replica and a
  category target (directly, or as the primary of a mirror a category routes
  to), and refuses a backend replicating two mirrors whose swept prefixes
  overlap — '' overlaps everything. Disjoint prefixes stay legal.
- assignCategory (the migration flip, the other writer of the settings rows)
  refuses routing a category onto a backend that is currently a replica.
- The admin mirror-target picker (both shells) only offers backends that serve
  no category; already-selected targets stay listed so they can be unchecked.
  A category assigned to a replica still surfaces the server's 400 verbatim.

Existing shared-replica fixtures were rewritten onto dedicated replicas.

* fix(client): a 409 conflict save is recoverable without a page reload

After a 409 the admin storage panel was a dead end. refreshState leaves a
dirty draft's version alone on purpose (audit #7: the background poll must
never float a mid-edit draft past a migration flip it never carried), setDraft
re-attaches that same version to every later edit, and save() resubmits it —
so every retry 409s again and only a full page reload got the operator out,
while the banner told them to "review and save again".

The version block stays. Instead the conflict banner now offers the one action
that can clear it: discard the local edits and reload the saved settings —
a fresh GET applied through applyState, re-deriving the draft (and its
version) and clearing dirty/saveError. Bumping the version in refreshState,
or only in the 409 branch, was rejected: either resurrects audit #7, since a
full-document save would then revert a flip the draft never carried.

storage.saveConflict now points at that action instead of implying a plain
re-save works (all 23 locales). The escape lives in the shared hook, so the
phone panel renders it too.

* fix(client): deflake the queued-migration-waits-out-backfill panel test

FE-ADMIN-STOR-037 seeds a running backfill, so the panel polls GET from
first render — but the stub served the post-save world (files: off-box)
before the PUT had landed, which no real server would do. A poll
resolving while the confirm dialog was open made the confirm-time
candidate recompute see from === to, so nothing was queued and the
migration never POSTed.

The GET stub now stays faithful (pre-save world until the PUT lands,
usage included so candidates keep their object counts), and the test
deterministically forces the once-flaky interleaving: it waits for
polls to land while the dialog is open and asserts the candidate
survives them.

* fix(storage): surface the underlying cause in probe target errors

The admin storage Test probe reported only the driver wrapper's outer
message ("put failed for '…' on 'probe-s3'"), dropping the cause that
says WHY — connection refused vs missing bucket. Flatten the error's
cause chain into the probe result, appending an S3-style code when it
isn't already in the message.

* fix(storage): surface the underlying cause in replica-failure reports

Same defect as the probe fix, one layer over: MirrorDriver's
reportReplicaFailure captured only the wrapper's message, so
replica-failure notifications said "put failed" without the why.
Move the probe's cause-chain flattener into storage.types.ts as
describeError and use it in both call sites.

* fix(storage): strip ContentType from multipart Upload calls (MalformedXML)

Every multipart S3 put carrying a contentType — every backup zip and
photo over the 10MB threshold — failed with 400 MalformedXML. Upstream
@aws-lite/s3's Upload leaks the ContentType param into its final
CompleteMultipartUpload headers ({ ...xml, ...paramHeaders }), clobbering
application/xml, which flips @aws-lite/client's payload serializer to
JSON — so the server receives a JSON parts list where XML belongs.
Confirmed live against MinIO AIStor; any S3 refuses the same.

uploadMonitored now drops ContentType before s3.Upload. Multipart
objects are stored without a content-type, which TREK never reads back
(serving derives Content-Type from the key/caller, never object
metadata); single-request PutObject paths keep sending it.

Also closes the coverage gap that let this ship: the live contract
suite never crossed MULTIPART_THRESHOLD, so s3.Upload had never run
against a real server — it now round-trips a >10MB put carrying a
contentType.

---------

Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-08-22 11:33:55 +02:00