Commit Graph

12 Commits

Author SHA1 Message Date
Maurice d1dc8542ac fix(costs): leave out an expense no rate is known for instead of counting it 1:1
A foreign expense without a frozen rate is converted with today's rate.
When the server could not fetch rates itself, it counted the amount 1:1
in the trip currency, although the browser had the rates all along. A
VND bill on an AUD trip was settled as millions of AUD.

Such a row is now left out of the balances, flows, final budgets, trip
totals and per-person summaries. The settlement reports it under
`unconverted` and says which currency its figures are in; the trip
summary lists the ids and the budget prompt says how many are missing.

The browser lends the rates it already holds (`fallback_fx`) when it
saves an expense or a settle-up transfer, and to heal existing rows
through the new POST .../budget/freeze-rates route, which an editor's
Costs view calls for the rows the settlement left out. The server still
prefers its own rate, uses a lent one only while it is quoted against
the trip currency, and never replaces a frozen or explicit rate. The
settlement also takes the browser's display rate (`base_rate`) in place
of a display quote the server cannot fetch; the browser leaves out a
figure the query would refuse. The browser's own fetch gives up after
10 s.

Changing a row's currency while no rate is available no longer keeps
the old currency's rate; the row goes back to unfrozen.

MCP gets the same through the new freeze_budget_rates tool, which only
ever pins the server's own rate.
2026-09-26 13:19:27 +02:00
Maurice c7c9ba24a3 fix(journey): select all takes every photo in the range and imports keep their order
The photo picker's Select all took only the pages already loaded, 50 at a
time, and replaced whatever had been picked on another tab. It now loads
the rest of a day, trip or date-range search before selecting (cancelled
by closing the picker or switching tab), adds to earlier picks, and
Deselect all only clears what is in view. Add stays disabled until that
loading is done. The whole-library tab keeps selecting what is loaded and
shows the count as "N+".

A date-filtered Immich search stopped at 1,000 photos without a word,
because its page budget was absolute; it is now relative to the page
asked for, with an absolute ceiling of 270 raw pages sized so the
picker's deepest page still gets a full answer. Adds go out in batches of
500 ids, under the JSON body limit, and each batch is sent oldest first,
so photos land in an entry in the order they were taken. When a later
batch fails, the photos already saved are counted and the journey
reloads while the error still shows, through one hook for desktop and
phone.

After a provider add, the capture-time lookup runs at most four at a time
per user, shared by all of that user's batches. When it fills a row, a
journey:photos:updated event reloads open journeys once per batch, from
the REST routes and the MCP add tool alike. Device uploads, which arrive
one file per request, get the same lookup without the event.

Addresses discussion #1587.
2026-09-24 23:48:49 +00:00
Maurice b56f335978 feat(dayplan): a day can be added with the next date, which extends the trip by one
On a trip with dates the reorder dialog and the days sheet on a phone now offer two ways to add a day: a day without a date at the end as before, or the calendar day after the last date, named on its button, which extends the trip by one day. The new day goes right behind the last dated day, so the days without a date move one place back with their plans, and no existing day or booking changes its date.

The server does it in one transaction behind a new dated flag on the day create contract, used by REST, the create_day tool and the plugin RPC alike. It moves the road trip boundaries of the days behind the new one, and announces the day with day:reordered and the grown trip with trip:updated; an insert at a position announces its trip now as well. The next date and the day limit come from one shared rule, so the button never promises a date the server would not give.

On the client the add is online only and runs one at a time, so a double click cannot grow the trip by two days. A wiring test that picked the stay question by its position now finds it by its handler, since the delete question sits in front of it.
2026-09-23 22:40:46 +02:00
Maurice ed18c37903 fix: the findings of the final 4.3.0 audit
Every file the release touches was read once more, and what survived an
independent check against the code is fixed here, with a test for each.

Data and credentials
- A document sync connection edited to a new address no longer carries the
  stored token along: a blank form under another origin is refused, the same
  rule the probe route already had. Dawarich and AirTrail testConnection send
  the stored key only to the host it was stored for.
- The key rotation script now covers document_connections.secrets and
  trip_document_links.webhook_secret, which it had skipped.
- Turning a booked night into a pause in the road trip popup asks first and
  says that the booking and its expense go with it. Deleting a place says the
  same when a night is booked there, on the desktop and on the phone.
- A moved version marker over unchanged bytes no longer replaces the TREK file
  and its attachments; a real revision carries the attachments over.

Planner
- A drop in the day plan names the row it lands ahead of in the full day, so a
  hidden hotel stop no longer shifts it one slot early; the store sorts before
  it splices. Via re-anchoring, the booked-night guard and the sidebar's day
  membership read the stored day, the index space the server uses.
- A hotel stop placed mid-day re-pins the road trip vias of that day, and the
  booking session receives the new stop order too. The Public API and the iCal
  feed list a booked hotel once. A plugin can no longer write a stay against
  another trip's day.

Road trip and map
- The route overview paces its requests to the public routers and retries a
  rate limit, and it says which legs it could not route instead of quietly
  reporting a shorter trip. A click on a via handle no longer drops a second
  via; handles hide when zooming out. Read-only members cannot edit the driving
  limits. The phone shows a way back when no day is picked and prints the night
  drive departure in the chosen clock format.

Integrations and search
- Google transit keeps conventional trains when the train chip is on.
  TREK_PLACES_ENABLED=false stops the details and enrichment calls too, and
  accepts the whole boolean family. A caller-supplied description no longer
  lands in the instance-wide enrichment cache. Nominatim searches carry a
  timeout. An OSM place id is checked for shape before it reaches Overpass.
- Dawarich puts a stay on the trip whose dates hold it, the Atlas hand-off
  knows what the Atlas already shows, and a host change with a new key clears
  the old instance's state.
- Document sync: a webhook address pasted by hand works on the token alone,
  deleting a connection unregisters its subscriptions, Nextcloud under a
  sub-path connects, Synology answers a name collision before uploading and
  verifies a move into the bin, Paperless titles are capped at 128 characters,
  a DNS failure reads as unreachable rather than as a policy verdict, videos
  from the store are accepted like uploads, and remote_missing has a way out.
  Conflict buttons only for the owner, a refused choice is reported, a refused
  credential gets a Reconnect button, disconnecting asks first on the desktop
  as it does on the phone, and the folder picker no longer spins forever.

Release
- The server holds the release notice back from a bundle that cannot draw it,
  so the previous shell served from the service worker cache after an update
  no longer shows bare keys and no longer lets a click use the notice up.
- Import into an existing list gets the same body limit as a new list. Chat
  images say when the limit is reached and the new controls are translated.
  The transit backend switch reports a failed save. The Helm chart carries the
  Amap credentials in its Secret; the shipped templates document the Places
  API and Amap variables and the Overpass default. The wiki follows.
2026-09-20 13:54:31 +02:00
Maurice b88b92add7 fix(docsync): a paused binding no longer resets its own settings
PATCH links validated its body with a schema derived from the create
contract through .partial(). In Zod 4 a .default() still fires inside an
optional field, so a patch naming one setting arrived with every other
one filled in: pausing a pull-only binding switched it to two-way, reset
both policies and blanked its label. The update contract is now written
out without defaults, and an e2e test drives a patch through the real
pipe, which the controller unit tests never see.

Also, the cosmetic leftovers from the review:
- POST connections/:connectionId/scopes no longer wants the connection
  in the body as well. It was required, never read, and let one request
  name two connections; a client that still sends it has the key
  stripped rather than refused.
- createLink loses the scope argument every caller passed as null.
- The controller's owner check is now assertCanManage, because it lets
  instance admins through just as canManageDocSync does on the client.
  Who may manage a binding is unchanged.
- document_provider_fields drops settings_key and payload_key. Nothing
  read them: the connect form takes its fields from /docsync/providers
  and the admin shelf sends none for document providers.
- The e2e test imports the provider classes under their new names.
2026-09-19 12:07:28 +02:00
Maurice a9907e2f10 feat(plugins): let a plugin answer place searches from its own index
TREK searches its own place index and OpenStreetMap. Neither carries a rating,
so "the best rated hotel around here" has never been answerable, and an operator
with a better index for their region had no way to reach the search box at all.
The searchProvider hook is that way in: a plugin is handed the query, a limit,
the language and the coordinate the search is biased toward, and answers with
places in the same shape the core search returns, plus a rating.

It sits beside the core search rather than inside it. The plugin runtime already
imports PlacesModule, which imports MapsModule, so asking the providers from the
maps service would close a module cycle — the same reason the other fourteen
contribution controllers stand next to what they contribute to. The client asks
both at once and merges, at the single point every caller already goes through,
so all nine search surfaces get plugin hits without knowing the hook exists.

A plugin index may never hold up the list: the client stops waiting at two and a
half seconds and the host gives a provider two, which is also why the budget is
the shortest of any hook here. With no such plugin installed the route answers
without reaching the runtime at all.

Every hit is normalized host-side and shared by the REST route and the MCP tool,
so a cap that holds for the browser holds for the assistant: coordinates are
range-checked, strings capped, the rating clamped to 0..5, a website must be
http or https, and ids are namespaced so they can never collide with an OSM one.

search_places_via_plugins is the MCP half, because what the user can see the
assistant has to be able to see; search_place now points at it.
2026-09-10 21:40:51 +02:00
jubnl b82962f95b Feat/plugin instance actions (#2209)
* feat(shared): scope plugin action descriptors

* feat(plugin-sdk): instance-scoped settings actions

* feat(plugins): run instance-scoped actions from the admin settings dialog

* feat(admin): instance-scoped plugin actions in the settings dialog

* fix(plugins): surface instance actions without instance fields and harden the action gate

* docs(plugin-sdk): note that hosts before 4.2.0 ignore action scope

* fix(admin): open the dialog for instance actions, explain inactive 404s, guard the save window

* docs(wiki): plugin pages for instance actions and the admin panel changes
2026-09-02 23:06:03 +02:00
Maurice b6a86275f6 fix(notifications): the smtp test names its failure and the log carries it (#2196)
testSmtp logged nothing and the transport ran on nodemailer's two-minute defaults while the client gave up after eight seconds, so every failure was a generic toast over an empty log. The transport gets real timeouts, every outcome is logged without credentials, and a classified, scrubbed cause (auth, DNS, refused, filtered port, TLS mismatch, certificate) goes into the response the admin sees; the channel test calls wait long enough to receive it. The password field also rendered its mask as value, so replacing a stored password silently saved the bullets as part of the new one; it is a placeholder now, on desktop and the phone admin.
2026-09-02 21:06:42 +02:00
jubnl 394201cd32 feat(plugins): instance-wide settings editor in the admin UI (#2170)
* chore: bump version to 4.1.0 [skip ci]

* fix(collections): stop findDuplicateCollectionPlace merging places by coordinates alone

Three places answered "is this a place the trip already has?", not two:
PlacesService.findDuplicatePlace and isPlaceDuplicate got the shared strategy
list in #2130; CollectionsService.findDuplicateCollectionPlace was missed and
kept its own hand-rolled copy of the pre-#2130 order.

Unlike findDuplicatePlace, this one had no isPlaceDuplicate() guard in front of
it — savePlace calls it directly — so the coordinate fallback for a NAMED
candidate was live, not latent: two distinct places sharing an address (a
restaurant and the bar upstairs) would report each other as duplicates the
moment their coordinates round to the same ~11 m box, regardless of name.

It also never read google_place_id/google_ftid/osm_id at all, even though
every collection_places row stores them, so a place renamed since it was
saved could be saved again instead of being recognised by its provider id.

Now walks placeMatchStrategies() from @trek/shared, same as
findMatchingPlaceId: provider id, then name, then coordinates only when there
is no name. savePlace passes the provider id fields through since they are
already on the request body; the two bulk call sites (importablePlaces,
saveFromTripPlaces) are unchanged, since their source rows don't carry
provider ids to pass.

Closes #2137

* fix(collections): ask the same question at all three call sites

savePlace was not the only caller of findDuplicateCollectionPlace. The bulk copy
already carries google_place_id/google_ftid/osm_id into the row it writes but
asked without them, and the import picker never selected them at all, so a
renamed place was recognised on one path and not on the other two. The picker
answering differently from the import is the drift the method exists to prevent:
a row shown as new would come back refused.

COLLECTIONS-SVC-102 and -103 cover the bulk copy and the picker.

* fix(map): give the Leaflet map its own zoom ceiling, so a vector basemap cannot take the planner down

Opening a trip on 4.1.0 could land on the error boundary with "Map has no maxZoom
specified", and so could the map settings tab, which left no screen from which to
pick a different basemap.

Leaflet answers getMaxZoom() from the map options first and otherwise from a
layer that brought one, and only a GridLayer ever contributes via its beforeAdd
hook. A vector basemap is a GL canvas on a plain L.Layer, so it contributes
nothing, and MapContainer set no ceiling of its own. MarkerClusterGroup.onAdd
refuses an infinite one by throwing, before any clustering, so an empty trip hit
it too.

That path opened when the default basemap became a MapLibre style. It reaches
further than an operator choosing OpenFreeMap: resolveTileUrl sends a keyless
CARTO template to the app default, which is now a vector style, so every
instance still carrying the old CARTO basemap fell into it on upgrade.

The ceiling belongs on the map rather than on the base layer, because it has to
hold whichever of the three branches renders. 19 matches what the raster and
satellite layers already carried, so nothing changes for the maps that worked.
SharedTripPage gets the same treatment: it has no cluster to throw, but it draws
the same vector basemap and its fitBounds asks for a ceiling.

mapZoomCeiling.test.ts pins Leaflet's own semantics against the real library
rather than a mock, since a mocked react-leaflet would have kept passing through
all of this.

* fix(map): put the CARTO key back on the template the settings preview draws

The map settings fields hold what the user is editing rather than what
useTileUrl already resolved, so the preview received a bare template with no key
on it. resolveTileUrl then read it as a keyless CARTO url and did what it is
meant to do with one: fall back to the app default. The preview drew
OpenFreeMap while the fields said CARTO, whether or not a key was saved.

Same reason the tab crashed for anyone on a CARTO basemap even after entering a
key, which is the loop that made this hard to escape: the key is entered on the
page the fallback had taken down.

FE-COMP-MAP-033b pins the template the preview is handed, with and without a key.

* fix(admin): put the CARTO key back on the default-settings map preview too

Same shape as the user-facing map tab, and reported alongside it: the field holds
what the admin is editing, so the preview resolved a keyless CARTO template and
drew the app default instead of the basemap being set.

* fix(mobile): put the CARTO key back on the phone map settings preview too

The third preview with the same shape, and the one that matters most while the
crash is live: the phone layout is where someone would go to change the basemap
when the desktop tab will not open.

* fix(calendar): a fully timed stay is its two hand-overs, not a week-long bar

Closes #2136

A stay was always exported as an all-day block across its whole range, and the
check-in and check-out markers came on top of it. When the stay records both
ends of its clock the markers already say everything the block does, and the
block is the half nobody can act on: it takes a week of calendar to repeat what
two one-hour events state precisely. Same shape as the parking and rental
windows in #2068.

Only where the markers actually stand in for the booking, though. They are
emitted once per stay and titled from its lowest-id reservation, so a second
room on the same stay would otherwise be left with nothing at all. A stay that
knows only one end keeps its block too, since nothing else carries the other
end's date.

CAL-025b covers the drop, -025c the one-sided stay, -025d the second room.

* fix(mobile): the phone admin map preview needs the CARTO key too

The fourth preview with the same shape, missed when the other three were fixed.
Sonar pointed at it: the desktop and phone admin pages are near-identical, so the
duplication report named the file the change had not reached.

* chore: bump version to 4.1.1 [skip ci]

* feat(plugins): admin editor for instance-wide plugin settings

The scope:'instance' settings a plugin declares had API endpoints but no UI.

- shared: plugins.schema.ts — the settings-field descriptor and the admin
  instance-config GET/PUT wire contracts, imported by both sides
- server: GET /api/admin/plugins/:id/config now returns the declared fields
  alongside the (masked) values; PUT re-spawns an ACTIVE plugin (its child
  reads config once, at init) and reports it via 'restarted'; the admin list
  carries instanceSettingsCount so the UI gates its menu item without a fetch
- client: 'Instance settings' row action opening a form modal (desktop) /
  MSheet (phone) in both admin shells — checkbox/select/secret rendering as
  in the user settings form; an untouched secret mask is never sent back
- i18n: three new admin.plugins.* keys in every locale

* refactor(plugins): move the admin body contracts to @trek/shared

The remaining plugin DTO schemas (install, link, activate, uninstall,
retrust, update, egress-hosts, user-settings update, route) now live in
shared/src/plugins/plugins.schema.ts with their inferred types, alongside
the instance-config contracts. The journey-contract doctrine notes (#1842)
moved with them — each body stays deliberately loose so the handlers keep
the rejections and fallbacks they already own, and new spec tests pin that
looseness so a well-meant tightening breaks a test that explains itself.

plugins.dto.ts is now thin createZodDto wrappers over the shared schemas
(same class names, no controller changes), and the client's pluginInstall
types its options and body against PluginInstallRequest.

* fix(admin): offer Allowed hosts only to plugins that declared operatorEgress

The row's egress chip was already gated on the declaration, but the kebab
menu (desktop) and row action sheet (phone) offered Allowed hosts on every
installed plugin — inviting the admin to widen egress for plugins that
never asked for it. Both entries now render only when the plugin declares
operatorEgress; the dialog's unsupported-notice fallback stays as the
truth-teller for stale list data.

* test(admin): close the Sonar gaps on the instance-settings form

The PR gate flagged new-code coverage (76.4%) and duplication (10.9%).

- The two identical 39-line handler blocks were logic, not markup — they
  move into one shared useInstanceSettings hook; each shell keeps its own
  markup (the deliberate desktop/phone divergence stays)
- New tests per shell cover the untested branches: every field type
  renders and round-trips into the save payload (select, checkbox/MToggle,
  number, hint, no-label key fallback), a failed fetch is a toast, a
  rejected save shows the server reason and keeps the edits, and the
  dialog/sheet closes without saving
- INS-009 covers the controller's no-body '|| {}' fallback

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: michael-bohr <263459669+michael-bohr@users.noreply.github.com>
Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-09-01 06:45:24 +02:00
Maurice 3e0bfbbb29 fix(client): give the desktop halves the same fixes their mobile twins got
Several of the fixes in this branch landed on a mobile screen while the
desktop component with the identical bug sat in another package and was left
alone, which leaves the two disagreeing about the same handler.

So: the appearance tab flushes its debounced write on unmount, the journey
share copy guards navigator.clipboard (undefined on a plain-HTTP self-hosted
install), the admin collab toggle rolls back from a scoped snapshot instead
of a stale one, the reservation modal parses metadata through the guarded
helper rather than a bare JSON.parse that takes the route down, and the
notification toggle surfaces its failures and stops racing concurrent
toggles.

The Journey components reached the memories provider-status endpoint with a
raw fetch(), past every apiClient interceptor; they go through a small
memoriesApi now. And the server half of the OSM menu_url finding: the client
renders it through safeHttpUrl, so place-enrichment should not be handing a
javascript: value out in the first place.

The share section hides itself when the server says the link is not this
user's to manage, rather than offering a create button that 403s.
2026-08-23 11:59:12 +02:00
jubnl 305b113d5f Feat/storage driver (#1949)
* feat(storage): storage key validation, driver contract suite, LocalDriver

* feat(storage): MirrorDriver composite backend

* feat(server): add TREK_UPLOADS_DIR env var for the uploads root

* feat(storage): settings-backed registry, StorageService facade, module wiring

* test(storage): pin src/nest/storage coverage thresholds

* feat(storage): multer upload options factory

* refactor(files): spool trip-file uploads through storage

* refactor(collab): spool note attachments through storage; normalize files/ prefix

* refactor(journey): spool journey media uploads through storage

* refactor(trips): spool trip cover upload through storage

* refactor(places): spool place-image and collection uploads through storage

* refactor(auth): spool avatar upload through storage

* fix(server): consolidate allowed_file_types onto AllowedFileTypesService

* fix(storage): default defParamCharset to utf8 in the upload factory

* test(db): re-point the trailing-migration guard at the trip_files prefix strip

* fix(storage): age-gate the boot spool sweep

* test(files): key FILE-P02 residue checks on upload size

* test(storage): raise src/nest/storage coverage pin

* refactor(platform): serve /uploads routes through storage

* refactor(files): serve trip-file downloads through storage

* refactor(memories): serve photo bytes through storage

* refactor(maps): stream place-photo bytes through storage

* refactor(share): stream shared place-photo bytes through storage

* refactor(journey): serve public journey assets through storage

* fix(share): answer place-photo cache miss with 204 like the maps proxy

* test(server): close serving-swap gaps surfaced by the full tier

* test(storage): raise src/nest/storage branches pin

* refactor(place-photos): route place-photo cache through storage

* refactor(places): reclaim place images and collection covers through storage

* refactor(trips): trip covers through storage

* refactor(auth): avatar reclaim through storage

* refactor(collab): note-file deletion through storage

* refactor(files): trash deletion through storage

* refactor(memories): trek photo cache internals through storage

* refactor(memories): thumbnail service category-addressed

* refactor(memories): immich upload reads through storage; drop uploads-root

* refactor(journey): photo deletion through storage

* fix(memories): reclaim orphaned trek photo cache objects in the ttl sweep

* feat(journey): journey-thumbs orphan sweep and delete-site derived-artifact reclaim

* test(server): close slice-4 gaps surfaced by the full tier

* test(coverage): raise slice-4 domain pins

* refactor(backup): list, existence and delete through storage

* refactor(backup): download through storage sendToResponse

* refactor(backup): create backup through per-category storage enumeration

* refactor(backup): restore and upload-restore through storage

* refactor(backup): auto-backup retention through storage

* refactor(server): boot dir-creation to driver init

* test(server): close slice-5 gaps surfaced by the coverage tier

* test(coverage): raise slice-5 backup pin

* test(server): resolve async backupFileExists mocks for the tests typecheck

* test(server): pin ranged-read stat and zero-byte roundtrip in the storage driver contract

* feat(server): TREK_S3_* env wiring — schema, s3 namespace, all-or-nothing boot rule

* feat(server): S3 driver core — aws-lite client seam, stat/delete, driver-owned deadlines

* refactor(server): type the aws-lite S3 client via s3-types reference

* feat(server): S3 driver getStream and list — ranged full-object stat, iterator pagination

* feat(server): S3 driver put — bounded-peek routing, Upload inactivity deadline

* fix(server): S3 driver Upload monitoring — Transform interposition, stream cleanup

* feat(server): env-declared s3-main storage backend; settings-declared s3 rejected

* test(server): env-gated S3 driver contract suite against MinIO

* fix(server): S3 driver zero-byte put — work around aws4 duplicate Content-Length signing defect

* fix(server): scope the zero-byte put workaround to buffered bodies; bind test MinIO to localhost

* ci(server): MinIO-backed S3 driver contract job

* ci(server): align s3-contract shim comment and MinIO bind with references

* docs: S3 storage backend configuration and off-box backup mirroring

* docs: restore README formatting, keep S3 additions only

* ci(server): make live-MinIO contract runs manual-only (AIStor licensing)

MinIO's community line is discontinued entirely; the maintained successor, AIStor, requires a license key that can't safely live in a shared CI runner. The s3-contract CI job is removed; live-provider contract runs against MinIO/AIStor are now a manual, license-gated procedure documented in docker-compose.minio-test.yml's header.

* chore(server): final-review polish — stale CI refs, dockerignore key, boundary pins, zero-byte tempfile contract case

* expose admin ui

* feat(shared): storage backend type registry, wire schemas, category relocation

* feat(server): accept settings-declared s3 backends — shared-schema options, decrypt-at-build, registry preview()

* feat(server): remove TREK_S3_*/TREK_UPLOADS_DIR — tombstone boot rule, built-in override as the relocation path

* feat(server): deriveSecurity.encryptionKeySet and storage secret plumbing (gate, encrypt, unmask, mask, redact)

* chore(server): drop storage-secrets import that only Task 8's probe helper will use

* feat(server): seed-once storage-config.json boot import — fail closed on any invalid seed

* feat(server): registry snapshot() — effective storage config with provenance

* feat(server): api/admin/storage GET/PUT — masked secrets, preview-gated saves, wired reload, audits

* feat(server): api/admin/storage/test — per-target ephemeral probes, mirror targets probed individually

* test(server): e2e coverage for api/admin/storage — guards, managed refusal, envelope, probes

* test(server): give e2e harness DBs the app_settings table the storage registry reads at boot

* test(coverage): raise pins after the storage admin-config slice

* fix(server): final-review polish — encryption-gate wording, all-field sentinel guard, README/e2e notes

* test(server): storage slice-1 review follow-ups — mirror s3-replica probe decrypt, all-encrypted seed without key, test-route 401

* feat(i18n): storage admin domain + tab label in every locale

* feat(client): adminApi storage endpoints — getStorage, updateStorage, testStorageBackend

* feat(client): storage admin draft-document helpers + relative-time util

* feat(client): schema-driven storage BackendForm — fields by kind, mask echo, encryption banner

* feat(client): admin storage panel — backends list, category map, health strip, one-PUT save

* feat(client): storage tab on the admin page

* feat(client): storage admin panel and section on the phone shell

* fix(client): block renaming a storage backend onto an existing name

* feat(server): allow mirrors on any storage category — lift the backups-only rule

* feat(i18n): replicas-on-primary mirror strings in every locale

* feat(client): mirror fold/synthesize model helpers for replicas-on-primary

* feat(client): BackendForm mirror-targets composer; raw mirror type hidden

* feat(client): desktop storage panel folds mirrors into their primaries

* feat(client): phone storage panel folds mirrors into their primaries

* fix(client): strip removed backends from remaining mirror replicas — no phantom refs

* feat(storage): retire photos as a configurable category — served-only via the ServedCategory seam

* feat(i18n): descriptive storage category names and descriptions in every locale

* feat(client): descriptive category names, descriptions and id badges in the storage panel

* docs(storage): fix stale nine-category comments after the eight-category shrink

* docs: storage configuration via the admin UI and seed file — README + compose

* docs(wiki): Admin Storage page and storage cross-links

* docs(server): storage domain blueprint entry; TREK_PLACE_PHOTO_DIR tombstone clarifier

* refactor(server): drop the storage env-var tombstones — the variables never shipped

* docs: storage config is simply new in v4 — drop the removed-variables story

* feat(shared): backfill-status and usage wire schemas on the storage admin state

* feat(server): MirrorDriver.backfill — prefix-scoped, size-compared, cancellable replica catch-up

* feat(server): one-at-a-time storage backfill jobs + the storage events bridge

* feat(server): storage usage scan — nightly cron, on-demand service, persisted with computedAt

* feat(server): replica_failure admin notifications — events bridge, hourly debounce, channel registration

* feat(i18n): replica-failure notification text in every locale

* feat(server): storage backfill and stats-refresh admin endpoints

* feat(client): adminApi backfill and stats-refresh endpoints

* feat(i18n): storage sync and usage strings in every locale

* feat(client): desktop storage panel — sync now, usage display, backfill polling

* feat(client): phone storage panel — sync now, usage display, backfill polling

* fix(client): per-category usage line on the phone storage panel

* docs: sync now, usage stats and replica-failure notifications

* test(server): admin scope now includes replica_failure, not just version_available

Commit 6db135da added replica_failure to ADMIN_SCOPED_EVENTS but left
NPREF-005b asserting the admin scope contained only version_available.
Update the expectation to match.

* test(server): cover storage backfill/stats/jobs branches to hold the coverage pin

* fix(client): guard storage admin polling against stale responses racing a save

A poll GET issued before a save's PUT resolves could land after the save
applied its fresher response, overwriting it with the pre-save world and
re-deriving the draft from that stale snapshot. useStorageAdmin now tracks
a state-application sequence number and drops any refreshState() response
whose sequence has moved on, or that resolves while a save is in flight.
Also guard the fire-and-forget refreshState() calls in startBackfill/
cancelBackfill against unhandled rejections, matching the poll's style.

* fix(server): report 404 before 409 when a backfill start names an unknown mirror

startBackfill checked the global busy flag before validating that :name is
a mirror routed by some category, so POST .../backends/garbage/backfill
answered 409 instead of 404 whenever a sync was already running. Resolve
and validate the target first (BackfillTargetError -> 404), then check
busy (BackfillBusyError -> 409).

* feat(client): two-column category grid on the storage panel

* feat(server): plugin files.getContent reads through the storage layer

* feat(server): plugin files.create writes through the storage layer

* refactor(server): drop FilesService.resolveFilePath, last direct-fs seam in the plugin surface

* fix(server): map StorageInvalidKeyError to the plugin file accessibility envelope

* feat(storage): allow credentialed backends without an explicit ENCRYPTION_KEY

Secrets are still always encrypted at rest; without an explicit key the
implicit data-directory key is used. Removes the plaintext-secret refusal
from the admin PUT and the seed import, the encryptionReady GET field, the
save-blocking banner in both admin shells, and the now-consumerless
security.encryptionKeySet env derivation. Docs now recommend (not require)
an explicit key, since the implicit key rides inside backup archives.

* test(client): pin the stale-poll release to the pre-save GET in FE-ADMIN-STOR-024

The stub deferred EVERY poll GET and rebound the release handle to the
newest one, so a tick landing after the save was released with the
pre-save payload — a response no real backend can produce, which the seq
guard rightly applied. Only the first (genuinely stale) GET is held now;
later GETs answer with the world the mock server currently holds. Also
drops the encryptionReady fixture field removed from the schema in the
previous commit.

* fix(storage): backfill puts carry content-type, stat failures report op stat, progress clamps

* feat(storage): category migration job — copy, flip, delta sweep, reclaimable tally

* feat(storage): migration start/cancel endpoints and client API

* test(client): storage admin fixtures carry the new migrations field

* fix(storage): migration POST answers 200 and validates its body via the shared schema

* fix(storage): cancel running storage jobs when a config save removes their backend

* fix(notifications): in-app replica-failure row carries the suppressed count

* feat(client): migration prompt, queue, and progress on the storage panel

* fix(client): migration queue holds a synchronous single-flight lock

* feat(client): migration prompt and progress on the mobile storage panel

* fix(server): register category-migration routes in the managed-route allow list

StorageAdminController.migrationStart/migrationCancel were added under the
controller-level @ManagedForbidden() decorator but never added to
MANAGED_ROUTE_ALLOW_LIST, so validateManagedRoutes() threw "Managed surface
changed" on every buildApp() call, failing 46 test files (any test booting
the real app).

* test(server): cover category-migration copy/sweep branches to hold the storage coverage pin

Adds MIG-005 (a target object already matching size/key is skipped in the
copy phase itself, not just the delta sweep) and MIG-006 (a copy failure
during the delta sweep is counted without undoing the flip that already
happened). src/nest/storage/**'s branch coverage was 83.91%, just under the
84% pin.

* docs(storage): category migration — move existing objects on reassign

* fix(storage): close the pre-flip cancel window and cancel re-routed migrations

A cancel landing between the last copy iteration and the category flip was
silently ignored, and a config save that re-routed the migrating category to
a third, still-defined backend never stopped the in-flight migration. Also
guards assignCategory against writing an unknown backend name, and fixes a
missing `suppressed` notification param being treated as truthy suppression.

* fix(client): migrations keep mirrors, queue waits on backfills, sweep failures surface

Reassigning a category onto a mirrored primary now posts the migration to the
mirror's wire name instead of the bare primary, so replication survives the
move. The queued-migration effect also waits out a running backfill (the
server's one-job rule spans both), and a done migration with sweep failures
now surfaces a doneFailures line in both shells, translated into every locale.

* ci(server): run the S3 contract suite against MinIO

Resurrects the s3-contract job removed in 6fbff835, adapted for AIStor:
writes the MINIO_LICENSE_KEY secret to a license file, boots the licensed
container, and runs the contract suite hermetically (no --coverage) so the
storage coverage pin keeps holding from unit tests alone. Gated to same-repo
PRs since fork PRs have no access to the license secret.

* fix(server): backup archives uploads from a remote primary

createBackup() enumerated uploads through withLocalFile, which downloads a
remote object to a temp file, hands it to the callback, then deletes it
before archiver ever reads it — archiver reads entries lazily during
finalize(), so a remote (S3) primary silently backed up zero upload bytes.

Add StorageService.getLocalPathOrNull(category, name): a narrow locality
probe that returns a real path only when one exists on disk right now (the
fs-existence check is the fail-safe half of the contract — a driver-reported
path that vanished between list() and here falls back too, not just a
path-less remote driver). The enumeration loop uses it directly when
available (zero-copy, unchanged for a local primary) and otherwise streams
the object into a per-backup staging dir under the backups spool, cleaned up
in the existing finally alongside zipSpool/dbSnap.

Also wire archive.on('warning', reject): archiver treats a missing/unreadable
entry as a warning, not an error, and skips it by default — which would
silently drop the entry from an otherwise "successful" backup.

* fix(server): client aborts stop 500ing and leaking streams

storage.service.ts sendToResponse: the local res.sendFile branch now treats
ECONNABORTED/ECONNRESET/ERR_STREAM_PREMATURE_CLOSE/syscall=write as success,
mirroring res.sendFile's own default callback (the contract
storageStaticHandler already implements caller-side). The remote branch
replaces the hand-rolled pipe+Promise with pipeline(stream, res), which
destroys the source stream on any failure — fixing the leaked S3 keep-alive
socket on a client abort — while swallowing the same client-abort error set;
a real source error still rejects so callers keep their miss contracts.

This centralizes the fix for every caller that streams through
sendToResponse: files-download, journey-public, photo-resolver,
trek-photo-cache, backup downloads, and /uploads/photos.

trek-exception.filter.ts now no-ops (destroying the socket) when headers are
already sent, so no filter path ever writes onto a response in flight.

maps.controller.ts and share.controller.ts's place-photo bytes proxies
switch their direct getStream+pipe to pipeline() with the same abort-tolerant
catch, keeping the existing 204-empty fallback for pre-header failures.

* fix(storage): mirror read fallback consults replicas on primary miss

MirrorDriver.getStream/stat now fall back to replicas on a primary miss
(StorageNotFoundError / null stat), not just an infra error — binding
decision: a read succeeds if ANY member holds the object. Only a
deterministic StorageInvalidKeyError skips fallback.

stat() gets a dedicated fallback loop instead of reusing fromReplicas:
a replica returning null means "keep looking," which the generic
catch-and-continue helper would have short-circuited on.

StorageService.sendToResponse/withLocalFile stop treating composite
driver.stat() as proof the object is at localPath; both now fs-check
the path itself and fall through to the stream branch when it's
missing, which is what actually reaches a MirrorDriver's replicas.

* feat(storage): remote serving parity — content-type, validators, range

Local backends get content type, ETag/Last-Modified, conditional GET, Range
and HEAD for free from res.sendFile; the proxying branch every remote
backend (and a local primary whose file vanished) falls through to had none
of it — every request re-streamed the whole object with no cache validator
and no way to seek.

sendStreamed() reproduces the contract from ONE pre-flight driver.stat():
a miss throws before any header is set (the caller's 404/204/next()
contract), and that same stat then answers 304s, 416s, HEADs and suffix
ranges without ever opening a body. The decisions live in a new pure module,
http-serving.ts:

- validatorsFor: the driver's ETag, else express/send's weak
  W/"<size hex>-<mtime hex>". mtimeMs 0 (S3 can report it) means no
  Last-Modified and no derived tag — serving uncached beats a 1970 header
  that reads as fresh forever.
- isNotModified: If-None-Match (weak compare, *, list) wins over
  If-Modified-Since, per RFC 9110 §13.2.2.
- decideRange: single range only, inclusive clamp, suffix computed from the
  pre-stat size, unsatisfiable → 416, If-Range strong-exact or full entity.
  A multi-range or malformed spec is answered with the full object rather
  than multipart/byteranges. If-Match/If-Unmodified-Since stay out of scope.

Range math uses the pre-stat size as the single source of truth, the ranged
getStream stat is never re-consulted, and the pipeline abort semantics are
unchanged.

S3Driver now surfaces the bucket's ETag on stat()/getStream() (on a ranged
GET it is still the full object's tag). aws-lite strips the tag's
surrounding quotes — verified live against AIStor RELEASE.2026-08-07 — and
an unquoted entity-tag is not a valid HTTP ETag, so the driver restores
exactly the quotes the SDK removed and nothing else.

* fix(server): key rotation covers storage.backends

The S3 storage backend's secretAccessKey is encrypted at rest inside the
app_settings row keyed storage.backends, but scripts/migrate-encryption.ts
didn't know about it — after a key rotation that secret stayed encrypted
under the old key and silently failed to decrypt on the next read.

The script now re-encrypts secretAccessKey for every s3 backend inside the
storage.backends JSON array, using a pinned copy of the secret-field list
(it deliberately can't import @trek/shared). The parity test pins that copy
against storageSecretFields('s3') so a new secret field can't be missed
silently.

* fix(server): restore reloads the storage registry

restoreFromZip closed and reopened the DB but never reloaded the
storage registry, so uploads rehydration ran against the STALE
pre-restore driver/category map instead of the config the restored
archive's own travel.db carries (audit #4, major).

StorageService gains reloadConfig(), a narrow passthrough to the
(unexported) registry's reload() — same shape as health(). backup.impl's
restoreFromZip now calls it right after reinitialize() and right before
the uploads wipe/rehydrateUploads/plugin staging, so rehydrated bytes
land where the restored config says. When reopen fails (reinitFailed),
reload and rehydration are skipped entirely — already reported as
"restart required", and there is no live DB handle for the registry to
read anyway.

* fix(server): category migration rewrites keys across prefix changes

Category migration copied objects to the destination backend using the
SOURCE key verbatim. A mode-A ('' prefix) source migrating onto a
prefixed backend (or vice versa) landed objects at keys the registry
could never resolve post-flip (audit #8).

Extracts StorageRegistryService.keyPrefixFor(category, backendName) from
build()'s inline conditional so startMigration can compute the
destination prefix before the flip. runMigration now maps every
destination key (destPrefix + key.slice(sourcePrefix.length)) in the
copy skip-check, the put, and the delta sweep; the reclaimable tally
stays source-keyed. Equal-prefix migrations are unaffected (identity
mapping).

* fix(plugins): await the now-async place/collection services in RPCs

PlacesService.update/remove and CollectionsService.deletePlace became
async when they started deleting storage objects, but their plugin RPC
handlers still called them synchronously: the null/false refusal check
compared against an always-truthy Promise, and places.update broadcast
a Promise instead of the resolved place. Both handlers now await the
service call; mapCollectionError is async-aware (accepts a sync-or-
async thunk) so the sync callers (create/update/savePlace/copyToTrip)
keep working through the same path. Verified the RPC host already
awaits handler results, so no change was needed there.

* fix(storage): backfill Sync now gains a deletion sweep

The Sync now backfill copied primary objects to replicas but never removed
replica objects the primary no longer has, leaving mirror read-fallback able
to serve stale data indefinitely after a failed replica delete. backfill()
now sweeps each replica post-copy: any replica key absent from the primary
key-set is re-checked against primary.stat (a race gate against the primary
regaining the key) before being deleted. Adds a deleted counter end-to-end
(driver, jobs service, shared schema, both admin shells, all 23 locales) and
a new ReplicaFailure 'list' op for a replica whose listing itself fails
during the sweep.

* feat(storage): optimistic concurrency for admin config saves

A stale admin PUT of storage settings could silently undo a category
migration's just-flipped assignment. Adds a version counter
(app_settings row storage.config_version) bumped inside the same
transaction by both writers — applyConfig and the migration flip
(assignCategory). applyConfig compares the submitted version first
and throws StorageConflictError, mapped to 409 ahead of the blanket
400. The client carries the draft's loaded version through every
local edit and maps a 409 to a distinct "settings changed underneath
you" message, refreshing state while preserving the dirty draft for
review.

* fix(storage): surface a failed config load + redact secretAccessKey

Registry load() now records the exact parse/validate failure as
lastLoadError() instead of silently falling back to last-good config or
built-in defaults; StorageAdminService exposes it as configError on the
admin state (storageAdminStateSchema extended additively), and
AdminStoragePanel renders it as a warning banner naming the error. Save
stays enabled — it's the recovery path — but an admin can no longer save
over a broken stored config without knowing.

Also adds 'secretaccesskey' to globalMiddleware's SENSITIVE_KEYS request-log
redaction list, so storage admin PUT bodies never leak the S3 secret into
debug logs.

* fix(client): admin storage panel state races

Migration prompt now recomputes computeMigrationCandidates(draft, state)
on render and again at confirm time, so a stale open-time snapshot never
drives stripCategories or the migration queue; adds a Cancel button to
the dialog. A failed startMigration POST explicitly clears the remaining
queue with a toast naming the dropped categories and refreshes state,
instead of leaving it invisibly stranded; a "Queued: X, Y" line renders
while the queue is non-empty. Testing a mirrored row now expands the
draft primary + replicas client-side and probes each individually,
instead of sending the mirror stub to the server (which resolved
targets from its own live snapshot, silently testing saved options over
unsaved draft edits). A completed/cancelled/errored backfill now renders
its terminal status alongside the Sync now button instead of hiding it,
so it can be re-run without a page reload.

* fix(storage): refuse shared replica backends so the sync sweep cannot cross categories

A mirror's "Sync now" sweep makes each replica MATCH the primary: it lists the
replica under every prefix the mirror's categories occupy and deletes the keys
the primary doesn't hold. The backups category's prefix is '' — the replica's
entire root — so making a category-serving backend (say uploads-local) a
replica of a backups mirror meant the first sync deleted every user upload.
Nothing forbade the config: validateConfig only checked that mirror targets
existed and weren't nested, and the admin picker offered every backend.

The sweep is correct, so the fix is at the config level:

- validateConfig refuses a backend that is both a mirror replica and a
  category target (directly, or as the primary of a mirror a category routes
  to), and refuses a backend replicating two mirrors whose swept prefixes
  overlap — '' overlaps everything. Disjoint prefixes stay legal.
- assignCategory (the migration flip, the other writer of the settings rows)
  refuses routing a category onto a backend that is currently a replica.
- The admin mirror-target picker (both shells) only offers backends that serve
  no category; already-selected targets stay listed so they can be unchecked.
  A category assigned to a replica still surfaces the server's 400 verbatim.

Existing shared-replica fixtures were rewritten onto dedicated replicas.

* fix(client): a 409 conflict save is recoverable without a page reload

After a 409 the admin storage panel was a dead end. refreshState leaves a
dirty draft's version alone on purpose (audit #7: the background poll must
never float a mid-edit draft past a migration flip it never carried), setDraft
re-attaches that same version to every later edit, and save() resubmits it —
so every retry 409s again and only a full page reload got the operator out,
while the banner told them to "review and save again".

The version block stays. Instead the conflict banner now offers the one action
that can clear it: discard the local edits and reload the saved settings —
a fresh GET applied through applyState, re-deriving the draft (and its
version) and clearing dirty/saveError. Bumping the version in refreshState,
or only in the 409 branch, was rejected: either resurrects audit #7, since a
full-document save would then revert a flip the draft never carried.

storage.saveConflict now points at that action instead of implying a plain
re-save works (all 23 locales). The escape lives in the shared hook, so the
phone panel renders it too.

* fix(client): deflake the queued-migration-waits-out-backfill panel test

FE-ADMIN-STOR-037 seeds a running backfill, so the panel polls GET from
first render — but the stub served the post-save world (files: off-box)
before the PUT had landed, which no real server would do. A poll
resolving while the confirm dialog was open made the confirm-time
candidate recompute see from === to, so nothing was queued and the
migration never POSTed.

The GET stub now stays faithful (pre-save world until the PUT lands,
usage included so candidates keep their object counts), and the test
deterministically forces the once-flaky interleaving: it waits for
polls to land while the dialog is open and asserts the candidate
survives them.

* fix(storage): surface the underlying cause in probe target errors

The admin storage Test probe reported only the driver wrapper's outer
message ("put failed for '…' on 'probe-s3'"), dropping the cause that
says WHY — connection refused vs missing bucket. Flatten the error's
cause chain into the probe result, appending an S3-style code when it
isn't already in the message.

* fix(storage): surface the underlying cause in replica-failure reports

Same defect as the probe fix, one layer over: MirrorDriver's
reportReplicaFailure captured only the wrapper's message, so
replica-failure notifications said "put failed" without the why.
Move the probe's cause-chain flattener into storage.types.ts as
describeError and use it in both call sites.

* fix(storage): strip ContentType from multipart Upload calls (MalformedXML)

Every multipart S3 put carrying a contentType — every backup zip and
photo over the 10MB threshold — failed with 400 MalformedXML. Upstream
@aws-lite/s3's Upload leaks the ContentType param into its final
CompleteMultipartUpload headers ({ ...xml, ...paramHeaders }), clobbering
application/xml, which flips @aws-lite/client's payload serializer to
JSON — so the server receives a JSON parts list where XML belongs.
Confirmed live against MinIO AIStor; any S3 refuses the same.

uploadMonitored now drops ContentType before s3.Upload. Multipart
objects are stored without a content-type, which TREK never reads back
(serving derives Content-Type from the key/caller, never object
metadata); single-request PutObject paths keep sending it.

Also closes the coverage gap that let this ship: the live contract
suite never crossed MULTIPART_THRESHOLD, so s3.Upload had never run
against a real server — it now round-trips a >10MB put carrying a
contentType.

---------

Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-08-22 11:33:55 +02:00
Maurice 1e66bcc701 test(pages): cover the page hooks, stores and the api client 2026-07-29 11:55:31 +02:00