Commit Graph

213 Commits

Author SHA1 Message Date
Maurice ae73527362 fix(atlas): a place that moves no longer keeps its old country (#2527)
place_regions caches the country and region Atlas resolved for a place,
and a row that was already there was never re-derived. Correcting a
place's location in the trip left the old row in charge, so Atlas, the
dashboard stats and the journey stats kept counting the country the place
had just left, with the trip's dates on it.

A trigger on places now drops the cached row whenever lat, lng or address
actually change, so every writer is covered: the place editor, the MCP
update tools, the plugin RPC and the import enrichment and address
backfill. An edit that writes the same location back keeps the row.

The background lookup that fills the cache also only writes while the
place still sits where it was resolved from. A place moved while its
lookup was running would otherwise get the old answer written back.

Places corrected before this fix would have kept their old country until
they were moved again, because the trigger only sees new edits.
PlaceRegionsRepairJob goes over place_regions once, on the first start
after the update, and remembers that in app_settings. It re-derives each
row from the bundled borders only, never Nominatim, and replaces a row
whose country or region no longer matches where its place is now. A row
the bundle cannot answer for is left alone unless both the coordinates
and the address put the place in another country; then it is dropped and
the next Atlas load looks the place up again. A place without
coordinates keeps no row. Its writes are guarded the same way as the
background lookup, so a place edited during the pass is not overwritten.

The in-memory region cache is now keyed by the country the address names
as well as the coordinates, so correcting only the address of a border
point no longer returns the old answer until a restart.

On dev the trigger takes schema version 244, the slot it has on main, so a
4.3.3 install upgrading to 4.4.0 does not skip it. Web Push moves up to
245 and is idempotent, so a 4.3.3 install gets its table there. An
instance that already ran Web Push at 244 never runs 244 again, so 246
creates the trigger once more; everywhere else that is a no-op.
2026-09-26 10:36:03 +02:00
Maurice 83d591dc6c fix(db): keep SQLite temp files in memory so trips can be deleted (#2518)
Deleting a trip failed with SQLITE_IOERR_GETTEMPPATH when the container
runs with a read-only root filesystem and no tmpfs on /tmp. The cascade
behind DELETE FROM trips keeps a statement journal, SQLite moves it to a
temp file once it passes 64 KiB, and it looks for a directory for that
file in SQLITE_TMPDIR, TMPDIR, /var/tmp, /usr/tmp, /tmp and the working
directory. None of them is writable there. Installs on the shipped
compose file have a tmpfs on /tmp and never hit it.

Every connection the server opens now comes from openDatabase() in
src/db/connection.ts, which sets temp_store = MEMORY: the main database
(also when it is reopened after a restore), each plugin's own database
and the uploaded backup checked before a restore. reset-admin.js and the
key rotation script cannot import src/ in the image, so they set the
pragma themselves. A test pins that nothing else in src/ constructs a
handle, and one deletes a full trip on the real schema and watches for
temp files.
2026-09-26 10:36:02 +02:00
Maurice 650451bdf7 feat(notifications): Web Push delivers notifications to every device, even with TREK closed
A fourth built-in channel next to email, webhook and ntfy. Browsers
register per device, the server encrypts per RFC 8291 and signs per RFC
8292 with node:crypto and sends only to known push services through the
SSRF guard. The event matrix decides per category what arrives by push;
admins switch the channel on under Notifications.

VAPID keys come from the environment or are generated once, only while
neither stored row exists, and kept encrypted. A pair that does not
decrypt or match is never overwritten and never swapped for another one:
push stays off with an error in the log and every device stays
registered until the key is fixed. The subject is VAPID_SUBJECT, then an
https APP_URL, then the project URL.

Only chat messages collapse into one notification per trip, and a
replacement still alerts. A 403 removes a device only after three
failures in a row; 404 and 410 remove it at once. Password change, reset
and an admin setting a password drop the user's devices with the other
credentials. The request log redacts subscriptions and endpoints, boot
errors print *** for every secret variable, and logout forgets the
device with a plain fetch so an expired session cannot redirect midway.

Addresses discussion #894.
2026-09-24 23:17:03 +00:00
Maurice ae80ad266d fix(immich): a server with a self-signed certificate can be connected, as the other providers can
Synology Photos, AirTrail and Dawarich each have a switch to trust a
self-signed certificate; Immich had none, so every request to such a
server failed with a bare "fetch failed" (#2475).

Immich gets the same switch, per user and off by default:

- users.immich_allow_insecure_tls (migration 243, column guarded) and a
  settings row labelled with the existing skipSSLVerification string, so
  no new translations. The settings card on desktop and phone draws it
  from that row.
- Every request to the user's Immich (status, browse, search, asset info
  and proxy, albums, upload) passes rejectUnauthorized from the switch.
  Only 1 counts as on. A shared photo follows its owner's switch, since
  it goes to the owner's server with the owner's key.
- Settings PUT and the connection test take allow_insecure_tls as a
  boolean. Absent on a save keeps the stored choice, so an older client
  cannot clear it; disconnecting turns it off again.
- A failed request names its cause, e.g. "fetch failed (self-signed
  certificate)", instead of only the undici message.
- The upload mirror gets a timeout: the journey upload waits on it.

Fresh installs also never showed "Mirror journey photos to Immich on
upload": migration 112 added that row only where the Immich provider
already existed, and on a new database the seeds create it after the
migrations. The row is now in the seeds, and migration 243 adds it where
the provider exists without it.

Wiki: Photo-Providers.md documents the switch.
2026-09-23 17:41:25 +02:00
Maurice 81292d4d2b fix(server): the booked night settles in one pass, and the sibling paths seat it too
The audit over the 4.3.1 release found the seating rule reading a night by one
value and its neighbours by another, so two nights on one day never agreed:
the migration reported moves that changed nothing, flipped the pair on the
next run and dropped the road drawn between them, and in the service every
notes edit swapped them. One rule now lives in accommodations/night-seat.ts,
measures every night by its own check-in, breaks a tie by the booking made
first, and carries the vias; the service, the reseat migration and a trip
date change all seat through it, the last of which used to append the night
at the end of its day again. An account created by a first SSO login writes
the user.register row the password path writes. Tests cover the equal and
absent check-ins, the date change, the booking form path and a day that is
nothing but its night.
2026-09-22 13:30:12 +02:00
Maurice be9a38e3b0 fix(accommodations): a second night on the day counts by its check-in
Two bookings on one day used to settle by whichever was seated last, because
the other night's stop carries no hour of its own. Its check-in counts now,
in the service and in the migration alike.
2026-09-20 18:48:26 +02:00
Maurice 417d0445b9 fix(roadtrip): seat the nights booked before this release where their check-in says
A booked night leads its day now, but the stops the bookings put on their
check-in day before this release sit last, and the backfill for older
bookings appended them too. Every trip planned before the change kept that
order until somebody edited the check-in.

A migration seats every night a booking owns by the same rule the service
applies: first, behind only a stop whose own clock is at or before the
check-in. The drawn roads follow the stops they were drawn after, by the
rules the service uses when it moves a night. Stops the traveller placed
themselves are not touched. Re-runnable.
2026-09-20 18:45:37 +02:00
Maurice 62cccf32f3 fix(db): give clipped trips their days back and let road-trip day numbers follow the trip limit
Two boot migrations. The first appends the missing dated days to a trip
the old 365-day cap clipped, only where the dated days still run unbroken
from the start date, so a trip that was re-dated by hand or lost a day in
the middle is left alone; dateless days that hold content stay behind the
dated ones as generateDays keeps them. The second rebuilds
roadtrip_day_boundaries without the CHECK that held day_number to 366,
which would have refused a boundary on a later day of a longer trip.

The road-trip contracts follow the same limit: the day boundary and
corridor schemas, the set_day_boundary tool and the day window's runaway
guard read MAX_TRIP_DAYS instead of their own 365 or 366, and the
tooLong message carries the number as a placeholder in all 23 locales.
2026-09-20 12:34:34 +02:00
Maurice 9de8490176 style(docsync): drop dash punctuation from the added comments and test titles
The lines this branch adds used em and en dashes as sentence
punctuation across comments, JSDoc, CSS comments and test titles. Each
one is rewritten with a comma, a colon, parentheses or a second
sentence, keeping what the sentence said.

Only lines this branch adds are touched; everything that already
exists on dev stays as it is. No logic, error code or logged message
changes. The only string literals affected are describe and it titles,
which now read "X: topic" like the rest of the provider tests.
2026-09-19 13:36:51 +02:00
Maurice b88b92add7 fix(docsync): a paused binding no longer resets its own settings
PATCH links validated its body with a schema derived from the create
contract through .partial(). In Zod 4 a .default() still fires inside an
optional field, so a patch naming one setting arrived with every other
one filled in: pausing a pull-only binding switched it to two-way, reset
both policies and blanked its label. The update contract is now written
out without defaults, and an e2e test drives a patch through the real
pipe, which the controller unit tests never see.

Also, the cosmetic leftovers from the review:
- POST connections/:connectionId/scopes no longer wants the connection
  in the body as well. It was required, never read, and let one request
  name two connections; a client that still sends it has the key
  stripped rather than refused.
- createLink loses the scope argument every caller passed as null.
- The controller's owner check is now assertCanManage, because it lets
  instance admins through just as canManageDocSync does on the client.
  Who may manage a binding is unchanged.
- document_provider_fields drops settings_key and payload_key. Nothing
  read them: the connect form takes its fields from /docsync/providers
  and the admin shelf sends none for document providers.
- The e2e test imports the provider classes under their new names.
2026-09-19 12:07:28 +02:00
Maurice dd3d97589b fix(docsync): act on a deletion once, and know what to do when it is undone
Both planner branches planned local_deleted for as long as a file sat in
TREK's trash. Under unlink nothing showed, until the binding was switched
to trash: the next run then binned every document ever deleted in TREK,
months back included. A binding on trash from the start asked the provider
to bin the same copy on every run. A local_deleted row is now left alone,
so the policy is applied once, under the policy in force at that moment.

That guard also meant a file taken back out of TREK's trash stayed on
local_deleted for good, which it already did before: no transfer, no
upload, no flag, and a second deletion of the same file would never have
propagated. The row now leaves that state by what became of the provider
copy:

- still there: the pairing counts again and the row is judged like any
  synced one, so an upstream edit made meanwhile comes down;
- binned by TREK: the pairing is dropped and the file goes up again as a
  new document through the ordinary push, keeping its trek_doc_uid;
- deleted by somebody else: flagged remote_missing, never uploaded over
  their deletion.

Telling the last two apart needs a record of what TREK did, so the new
remote_trashed_at column (migration 239) is written when TREK bins a copy
and cleared when the pairing is back, or when the binned copy is listed
again because somebody restored it at the provider.

A file purged from the trash came straight back as well: file_id is ON
DELETE SET NULL, and the row read as a download that never landed. A
tombstone now stays a tombstone, and a purge no run had seen yet is
treated as the deletion it is.

Also drops snapshotHash, which nothing read.
2026-09-19 12:07:28 +02:00
Maurice 2d7cb124b3 feat(documents): two-way sync with Paperless, Papra, Nextcloud, OpenCloud and Synology
A trip can be bound to a folder, tag or space in a self-hosted document store,
and documents then move in both directions: what someone uploads in TREK appears
there, and what someone files there appears in TREK.

Three decisions shape the rest.

The connection belongs to the TRIP, not to a person. Every other integration
here is per user, and photos are shared only through an opt-in flag; neither can
deliver "everyone on the trip sees the same documents", because visibility would
depend on whose credentials fetched a file, and a member without their own
Paperless account would see nothing. So the trip owner binds the trip once, the
server talks to the provider under that one identity, and TREK's own membership
decides who sees what. The provider never learns that TREK has members. When the
credential holder leaves the trip the binding goes to `orphaned` rather than
quietly carrying on with an ex-member's token.

TREK keeps the bytes. A synced document is an ordinary trip_files row, so
download, trash, offline cache, PDF export and backup keep working untouched,
and a provider outage does not blank out a trip that starts tomorrow.

Deletion never crosses the boundary on its own. A document that disappears
upstream is recorded as missing and waits for a human, and a run that sees more
than 40 percent of its known documents vanish at once refuses the whole listing
rather than acting on it: an unmounted share answers exactly like a mass delete.

The core reconciles by enumerating both sides, because the weakest provider sets
the model. A changed-since query cannot see a deletion, Papra does not move
updatedAt when a tag changes, a Paperless bulk edit leaves `modified` alone, and
Synology FileStation has no change feed at all. A webhook is therefore only ever
"look now", never a source of truth. Echo is caught by comparing content, not
timestamps: pushed_sha256 records what TREK itself wrote, so a provider
reporting TREK's own upload is not mistaken for a stranger's edit.

Provider differences are answered by capability data rather than by branching on
an id: stable ids where they exist (Paperless document id, oc:fileid, driveId)
and a normalised path where they do not, replace-in-place where the API offers
it, and a content hash from the listing only where the provider really supplies
one.

Admins switch providers on under Admin, Addons, Documents, mirroring how photo
providers hang off Journey, including the cascade and the 409 when the parent
addon is off. The per-trip setup lives in the file manager, where the person has
the context that makes the folder choice obvious.
2026-09-18 19:23:07 +02:00
Konstantinos Thermos 5f57c63426 fix(journey): make the source_assignment_id migration replay-safe
An unreleased migration has to survive being replayed on a schema where it
already applied — installs skip releases, and a test rewinds a fully migrated
database to the last released version to prove it. The bare ADD COLUMN threw
"duplicate column name" on that replay, which took every migration test down
with it.

Guard the ALTER on pragma_table_info, and add `source_assignment_id IS NULL`
to the backfill so a replay cannot re-resolve a row that has since followed
its stop to another day.

The trips e2e harness hand-builds a minimal journey_entries table rather than
running migrations, so it needs the column spelled out too.
2026-09-16 16:32:02 +03:00
Konstantinos Thermos 4eaf90fd1b fix(journey): a place kept across two days is two days of the journal
The sync engine keyed a skeleton by source_place_id alone, so a stop standing
on two days — the city you land in at dusk and walk through the next morning —
got one entry on the first of them and nothing on the rest.

Key on the day assignment instead. The assignment row rather than the day,
because moving a stop updates day_assignments.day_id in place: keyed this way
an entry follows the move instead of reading as one stop leaving the plan and
another arriving. Existing rows are backfilled to the place's earliest
assignment, and reconciliation adopts an unlinked row rather than annotating
it out from under its author.

Fixes #2329
2026-09-15 15:31:59 +03:00
Maurice 54e5525568 fix(accommodations): a booking that moves carries its stop instead of rebuilding it
Correcting a booking's date deleted the day stop it had mirrored and inserted
a fresh one. A day stop is not just a day and a place: its participants and any
road-trip day boundary hang off its id by ON DELETE CASCADE, and its note, its
hour and its end-of-day flag live in its own columns. All of it went, silently,
for an edit that only meant to move the night one day on. The row is carried
across now, seated by its check-in the same way an insert would be, and the
delete-and-insert is kept only for the case where the target day already holds
that place under a stop of the traveller's own.

Three more holes in the same mirror, found alongside it:

Dropping a night while keeping the place cleared accommodation_id server side
and told nobody. Days hides a stop that carries a booking id, so on every other
screen the place stayed invisible on a day it was standing on.

A night is seated by its check-in, which renumbers the stops around it, and
nothing said so. The day that changed now sends its order along.

Deleting a place cancelled the nights booked there, which takes each night's
partner booking and that booking's expense with it, and announced neither. The
Bookings list kept the reservation and the Costs total kept the money until a
reload, and an expense linked by reservation_id is not one the place-linked
sweep finds.

Also: day_assignments.accommodation_id went in as a bare INTEGER, so shortening
a trip past a booking's last night cascaded the booking away and left the stop
pointing at nothing, hidden from Days for good while the route still drove to
it. Migration 234 clears the ones already out there and adds a trigger, which
SQLite fires even when the booking went down with a cascade. And a duplicated
trip now carries the stop's booking id, the booking's check-in window and a
place's fill_percent, none of which the copy took along.
2026-09-13 23:48:59 +02:00
Maurice c2707b7eff feat(journey): the enhancements from discussion #2299
roel-de-vries pitched seventeen changes to Journey. Four were already built or
rest on a misunderstanding, one cannot work as described, and three would take
away more than they give; the rest are here, plus two things the list did not
ask for.

**The entry you write on the move now has a name.** Quick capture hid the title
field outright and offered a note instead, so every entry caught on a phone
arrived nameless and a day's timeline read as a column of identical
placeholders. One line of input; it is the single change in this branch that
users will feel most.

**The map stops yanking.** Stepping through the timeline called flyTo with a
zoom floor of 12, so reading a journey from a country view snapped to street
level on the first scroll and stayed there — every marker filling the screen,
and no sense of where anything was. Both renderers pan now and leave the frame
the reader chose. The initial fitBounds still decides how close a journey opens.

**The phone timeline is photos.** A 64px thumbnail beside three lines of text
spent the card on a story nobody reads at that size and truncated the title
after a few characters. The picture is the card now, with the name and place on
it, the country's flag and the date in the corners, and the day carried as a
colour rather than as a number nobody was reading. Above the carousel, a
segment per day: it says how long the journey is, how far in you are, and takes
you to day nine without swiping past days three to eight.

Everything else, briefly:

- A plus in each day header, starting a draft on that day instead of on today.
- A journey opens on today when today is part of it, and at the beginning when
  it is not — a journey that ended last spring is something you read.
- Suggestions can be waved away one at a time. The row survives so the trip sync
  does not offer the same place again; journey settings brings them all back.
- Mood, weather and the pros/cons list can be switched off per journey. Nothing
  written is erased — the form stops asking, the values stay.
- The weather fills itself in at the desk the way it already did on the phone.
  The date picks the source, so writing up last Tuesday gets last Tuesday's.
- Enter opens the next pro or con, below the one you are in.
- The verdict moved behind the story's fold, and the fold itself came in from
  nine lines to four.
- The external picker lost the count it was printing twice and some padding it
  could not afford inside the editor.
- The phone can reorder entries within a day at last; the desktop arrows finally
  have translated labels.
- Marker labels are off where a carousel already names the entry, and on where
  they are the only name a marker has.
- Search over a journey's titles, stories, places and tags, accent-blind.
- An entry knows which country it happened in, resolved once from its
  coordinates against the bundled boundaries.

Not built, and why: the photos-for-markers idea needs clustering first or thirty
markers become soup, and the active entry's photo is already on screen in the
card below. Removing mood, weather and the verdict outright would take them from
everyone who uses them, so they became switches. `capture` on the file picker
takes the photo library away on a phone, which is why the camera is a second
input; it is hidden on desktop now instead. The suggestion strip in the new-entry
flow would be a third surface over the same data — tapping a suggestion already
opens the editor filled in.

The one gap the list missed and this branch does not close: Journey is the only
domain with no offline path. That belongs in its own change.
2026-09-13 18:45:57 +02:00
Maurice 62d4cdcc18 fix(accommodations): catch the nights the upgrade itself could not reach
A migration runs while the old container is still answering. A booking written in
those seconds comes from code that knows nothing about the day stop and lands
behind the sweep that would have given it one: on the test instance exactly one
of ten, and invisible to the drive ever after, with no way back short of saving
it again. The sweep runs a second time, which is safe because the same NOT EXISTS
decides it both times.

The other way into that state is closed too. "Remove from day" in the place panel
took the stop and left the booking, which is the very state this whole change
exists to remove — and it was the only control that could produce it after the
upgrade. A stop a booking wrote is no longer offered there, in either direction:
removing it strands the booking, adding one puts the same hotel on the day twice.
The night is removed where it is made, in the day's overnight block or by turning
it back into a pause in road trip mode.
2026-09-13 15:19:33 +02:00
Maurice edf0538260 fix(roadtrip): keep a booked night out of the day list, and off the traveller's stops
Three things the first pass got wrong.

A stop a booking puts on the day is for road trip mode, which needs the
hotel in the driving chain. The day planner already shows that booking as
its own overnight block, so the row there would be the same hotel twice.
It is hidden in Days whatever the service-stop switch says, and the switch
goes back to meaning what it says: the petrol stations the traveller added.

Cancelling a night from the road trip popup took the stop with it. That
control means "this is a pause now, not a night", and the stop sits
mid-drive where re-adding it would land it at the end of the day. The
booking now hands the stop over instead of deleting it.

Moving a booking whose stop belongs to the traveller put a second one on
the new day and left the first behind, which is the duplicate this change
exists to remove. Only a booking's own stop moves now. Stays booked before
this shipped get theirs from the migration, not from the next edit they
happen to receive.
2026-09-13 01:42:52 +02:00
Maurice f0e1da2d81 feat(accommodations): put a booked night on its check-in day
A road-trip stop is a day assignment: the rail, the server-side plan and the
map all build their stops from day_assignments and only look the stay up
afterwards, to hang check-in and check-out on one. A stay written without an
assignment was therefore invisible to every routing surface, so a hotel entered
in the day planner never reached the road-trip view and had to be entered a
second time as an ordinary place. The road-trip side has always written both
halves in one go; this is the day planner catching up, in the domain that
writes the stay.

Booking a night now also assigns its place to the check-in day, inside the
transaction that already writes the stay and its partner hotel reservation, and
types the place as a hotel stop when it carries no type of its own, so it reads
exactly like a night booked in the road trip. Only the check-in day gets a
stop; the later nights ride on the stay row. A place already planned for that
day gets no second stop and the booking claims none, so cancelling it cannot
delete a stop somebody placed by hand. Moving the booking moves its own stop,
deleting it takes its own stop back, and a trip re-date carries it along with
the stay. All four write surfaces (REST, plugin RPC and both MCP tools) run the
same write and announce the same events; the answers carry the stop too,
because the broadcast skips the socket that sent the request.

New nullable day_assignments.accommodation_id marks a stop as a booking's own.
It stays NULL for every existing row on purpose: everything planned before this
was planned by hand, and no backfill rewrites a finished trip. A stay booked
earlier picks up its stop the next time somebody saves it.

The booking form keeps writing stays on its own: that surface is gated on
reservation_edit and the day stop is day_edit work.
2026-09-13 01:13:25 +02:00
Maurice 218d8baf0a feat(server): read visits and recorded routes from Dawarich
A new `integrations/dawarich` domain behind its own addon. Each reader
connects their own instance; the API key is encrypted at rest and never
returned. Nothing is written back to Dawarich, and no raw position is
kept in TREK: the recorded route is fetched per request for the window
being looked at and thrown away again.

What it produces are suggestions, never facts. A recorded stay becomes a
proposed place or journal entry, a country or city becomes a proposed
Atlas mark, and a stay near a wish becomes a proposed tick-off; each one
waits for the reader to confirm it. Identity is (user, Dawarich visit),
so a second sync neither duplicates nor overwrites what was accepted,
and deleting the result puts the suggestion back on the list.

The sync runs as a job on the central scheduler, one user at a time,
guarded against overlapping runs.
2026-09-12 23:36:49 +02:00
Maurice 11b69ae815 test(places): give the hand-built places tables the Amap id column 2026-09-12 18:07:39 +02:00
Maurice 1944bdf0a5 feat(maps): let Amap take the keyed provider slot beside the TREK index
The rebase kept maps.service.ts as dev has it: the TREK index and
OpenStreetMap answer every search first, and a keyed provider only fills
what they leave empty. Amap now sits in that slot next to Google instead
of behind an extracted Google provider, which is dropped.

places_provider (auto | google | amap | openstreetmap) decides which keyed
provider holds the slot; auto keeps whatever the install already used.
An amap: id opens against the Amap key regardless of the setting, takes
the plain details lookup on expand, never reaches Google's photo call,
and a pasted Amap link resolves through parseAmapUrl. Reverse geocoding
asks Amap at the instance key before Nominatim.

The migration carries users.amap_api_key, which the rebase had left at a
stale array position, and places.amap_poi_id for the next commit.
2026-09-12 18:03:10 +02:00
A1phaN 57eac32f9f feat(map): amap as map api provider and display template 2026-09-12 18:02:57 +02:00
RemEx bd46089450 feat(budget): make a settle-up payment's date editable
The settle-up form only exposed from/to/amount and silently dated the
payment by created_at, unlike a regular expense which has an editable
date. Add a settled_at column (independent of created_at, same split as
expense_date) through the schema, migration, service, REST and MCP
surfaces, and the desktop + mobile forms; the ledger groups a payment by
settled_at, falling back to created_at for rows recorded before the
column existed.
2026-09-12 17:33:03 +02:00
Maurice ac57321840 merge dev and reconcile roadtrip planning 2026-09-12 15:45:34 +02:00
Maurice 41b18f970c feat(vacay): add manual school holiday calendars 2026-09-12 12:51:27 +02:00
Maurice b773faff70 feat(roadtrip): share trip preferences and improve service stops 2026-09-12 00:24:45 +02:00
Maurice 491cfccc56 feat(roadtrip): add daily scheduling and configurable day endings 2026-09-11 22:28:03 +02:00
Maurice 96ab748f34 feat(roadtrip): let one stop say how full it fills
The fill level was one figure for the whole trip, and a trip does not have one.
A motorway rapid charger is worth about 80 % because the last fifth costs as
long again; the charger at the hotel is worth all of it because the car stands
there all night. Between those two the difference is a whole leg, and a single
setting has to be wrong about one of them.

So the badge on a refuelling stop is now the way in: it shows what that stop
will actually fill to, its own figure or the traveller's, and opens a panel of
the six answers people actually give. Under them it says what the choice buys in
kilometres, because 80 % is an abstraction and "about 290 km from here" is the
thing being decided. Picking the figure a stop already has hands it back to the
setting, the same gesture the kind picker uses to un-pick a kind.

The value lives in places.fill_percent, beside stop_type, because it is a
property of the stop rather than of the person. Null means "follow the setting",
which is every stop that predates the column and every stop nobody has an
opinion about, so nothing changes for a trip that never touches it.

Two pieces of shared machinery came out of it rather than being copied. The
popover chrome — measure before showing, flip when the window is short, close on
an outside click, Escape or a scroll — is one component now, used by the kind
picker and this one. So is the assembly of the range from seven settings, which
the routes hook and the rail both need and which would have drifted the first
time a field was added.
2026-09-10 21:40:51 +02:00
Maurice d30b916f9c feat(maps): count how much routing an instance actually does
Every route is calculated in the browser against the public FOSSGIS hosts, so the
server never sees a routing request and nobody has a number: not how many a real trip
costs, not how big they are, not how much already goes to a self-hosted engine.
Deciding whether TREK could run its own router without that number is guessing at the
bill, and the counters have to be collecting before the question comes up.

Counters, not a log. A row is a day, a profile, a request kind and an engine kind with
totals on it: how many requests, how many waypoints, roughly how far, how many were
refused. No query, no coordinate, no route, no user, no trip. Nothing leaves the
instance: its own database holds them and its own admin reads them at
GET /api/route-usage/summary.

The client tallies in memory and flushes in batches — on a timer, past a threshold and
when the page is hidden — because a counter that cost a request of its own would
measure the load by adding to it. Routing all funnels through one fetch in
RouteCalculator, so that is the only place instrumented.

On by default and switchable off with route_usage_enabled, unlike the shadow log next
door: that one stores what people typed, this one stores four integers about a day.
2026-09-09 14:52:50 +02:00
Godric Hood 9543fdc3f2 feat(collab): add shared links, chat images, and plugin fullscreen
Adds a Collab Links panel (add/pin/delete) with translations in every locale so tab and button labels are not raw i18n keys, chat image attachments (up to 4), and native fullscreen on plugin iframes. Includes Zod request DTOs and CollabPanel coverage for the Links tab.
2026-09-09 00:11:00 +02:00
Maurice 6a3d4b1281 fix(budget): stop receipts deleting files, and share the upload path
Attaching a receipt no longer lets the budget domain delete one. Trashing the
file was reachable from the REST route, the MCP tool and the plugin RPC, none
of which check file_delete, and a receipt id is any file on the trip, so anyone
who could edit an expense could trash a document they were not allowed to
touch. A concurrent edit could do it by accident: the client sends the full set
of receipt ids from a snapshot taken when the form opened, so saving an expense
someone else had just attached a file to moved that file to the trash. Removing
a receipt now removes the link and nothing else.

The link row is treated as shared, because it is. One row can tie a file to a
place and to an expense at once, so the sync clears only the receipt half and
leaves the rest, the foreign key is SET NULL rather than CASCADE, and a receipt
already in the trash keeps its link so restoring it comes back attached.

The upload and its rollback moved into one module both shells call. It was
copied into six places, and all six rolled back with the permanent-delete
route, which only accepts a file that is already in the trash and therefore
answered 404 every time: a failed save left every uploaded file on the trip,
attached to the expense, and the next attempt uploaded another copy. It now
trashes first, keeps the pending files for the retry, and says so when a file
could not be removed instead of swallowing it.

Escape in the receipt viewer no longer closes the expense form behind it, the
download fallback uses a real accent class rather than a utility that does not
exist, and the upload indicator is actually switched on.

Tests: the receipt link behaviour on both service paths, the four new branches
in the files service, the upload module, and the viewer.
2026-09-08 23:15:26 +02:00
dersimoezdag 06ecc5135c feat(budget): attach receipts and invoices to expenses 2026-09-08 23:15:26 +02:00
jelly 8be594d34c fix(atlas): correct Guangdong Province name in region bundle 2026-09-08 16:36:47 +02:00
Maurice 676c82a308 feat(roadtrip): remember which track a day follows
The vias already made the day follow the road; nothing said which road. After a
reload the badge could only say "shaped by hand", and re-fitting after the stops
changed meant finding the track again by eye.

One row per day, written in the same transaction as the chain it describes: a
day claiming to follow a road whose vias never landed would be worse than a day
claiming nothing. Three states rather than two, which is why the field is an
object and not an id — absent leaves whatever the day followed alone, null says
it follows nothing, an object records the road it now takes. A bare nullable id
could not tell "do not touch this" apart from "forget it", and a hand-dragged
via on the same day must not wipe the label.

The foreign key is the point: delete the imported track and the label cascades
with it, so no day is left claiming to follow a line nobody can see. The vias
stay, because they are the drive.

Both write paths check that the place is a track of this trip before it becomes
a label — permission alone would let another trip's place id, or an ordinary
place, become a name that can never be drawn.

Also clears three reliability findings the scanner raised on this branch: two
default sorts given an explicit comparator, and a trailing-slash regex the
scanner is right to call backtracking, walked instead.
2026-09-07 18:36:54 +02:00
Maurice e11ae6f056 feat(roadtrip): shape the drive — vias, alternatives and stops along the way
The road-trip addon's second half: the parts that let a planned drive be
argued with rather than only read.

Reshaping. A via records which stop it follows by position, and that position
is resolved fresh every time the day is routed — so inserting, reordering or
removing a stop moved the ground under every via after it, and the drive
silently snapped back to the road it had been steered away from while the
handle still sat on the map where it was left. The correction is computed
against the day as it stands and written with the same change, in one
transaction: a half-applied re-anchoring mixes two numbering schemes in one
day. The one judgement that needs geometry — which half of a split leg a via
belongs to when a stop lands in the middle of it — is measured on the road
actually driven, so a fuel stop added to a bent leg keeps the bend on the side
it belongs to.

Alternatives. OSRM answers most long legs with exactly one route, which read as
"there is no other way" when a perfectly good slower road existed. When it
offers nothing, the same drive is asked for again without the motorway, and
failing that without the tolls — one question at a time, stopping as soon as
one lands. The offer says what it is rather than how much slower.

Stops along the way. Everything the corridor search can find is a pause, not a
destination: fuel, charging, rest areas, campsites, food and sights all sit
inside the leg they were found on, carry their own colour and icon, and are
left out of every stop count. That colour is defined once and read by the rail,
the corridor list, the map pin and the picker — and it replaced the chains'
logos, which turned a corridor full of petrol stations into a row of
advertisements and left the brands with no logo on file looking like something
else entirely.

How long a stop takes was readable in the rail and writable nowhere; new places
were given an invented hour that nobody had entered. It is now set from the
rail, cleared as a zero because the update statement folds a null into "leave
it alone", and new places start with nothing.

Both columns are rebuilt on one type scale, anchored to the day plan's own
sizes so switching tabs does not change the size of the text beside it. The
rail reads as trip totals, day cards and a chain of stops with the drives
between them; the corridor column as what to look for, how the search is
getting on, and what it found. Six category pills over three lines became one
dropdown, which is the space the results wanted.

Also: the offered routes are framed on the map when the picker opens (`fitKey`
carries no coordinates and each renderer reads it as "the selected day"), road
trip mode does not cluster at all so the whole drive stays readable zoomed out,
and the drive-time labels finally render in the app's own typeface instead of
Leaflet's Helvetica. All of it in both renderers.
2026-09-07 18:36:54 +02:00
Maurice 3902e77991 feat(roadtrip): add the road trip addon with a drive-first plan view
The trip planner could show one day's route at a time and nothing across days,
so a road trip had to be pieced together in Google Maps (#1797, #435).

The addon adds a mode to the plan view rather than a tab of its own: the map
stays the workspace and gains the route over the whole trip, while the left rail
becomes one chain of stops with the driving distance and time between them and
arrival times that follow from them. A time pinned on a stop is kept and restarts
the chain, so a ferry or a museum ticket stays where the user put it.

Car bookings can finally carry the stops the schema has always allowed: the
endpoints table has held role/sequence for every transport type since #384, but
only flights and trains had an editor for them. The MCP tools already accepted
them, so this closes the gap the other way round.

Legs are fetched one at a time and spaced out — the public routing hosts answer a
whole trip's worth of requests with 429 — and the distance cap now looks at the
single leg instead of the sum, which is what dropped long drives entirely.
2026-09-07 18:36:54 +02:00
Maurice b3d69aa760 feat(places): record which search result was picked, on this instance only
The corpus behind "is our own index good enough to replace Nominatim".
When somebody searches and then picks a result, the client posts the query,
the rank and the pick. Nothing else: no user, no trip, no session, and
coordinates only to three decimals.

Deliberately decoupled from the search endpoint. Correlating a response
with a later pick would mean handing out a ticket on POST /api/maps/search
and threading it through all fifteen call sites; the client already holds
everything the row needs at the moment of the pick.

Evaluation happens offline afterwards: export the corpus, run it against a
candidate index, count how often the pick came back in the top five. That
is why collection can start before such an index exists, which is the whole
point, because the corpus needs weeks of real use and the index needs a
server, and those two waits should not be serial.

Off unless an admin switches it on, and it reads === 'true' rather than
!== 'false'. The three older places switches fail open because a migration
backfilled a row for installs already using them; nothing uses this one, so
an instance that upgrades and never visits the admin panel logs nothing.

Reading is admin only, retention is 180 days, and the switch state is on the
unauthenticated app-config endpoint on purpose: whether an instance records
what its users search for is something a visitor may know before logging in.
2026-09-07 14:05:33 +02:00
Maurice 6cbefd23e8 feat(studio): filter photos by entry, upload from Studio, and switch stops off
The content browser gets a filter over the journey's pictures: all of them,
the ones no entry holds, or one entry at a time, with the day and the place
under each name. An entry card in the Entries tab reaches its own pictures
the same way. Pictures can now come in without leaving Studio, into the
entry the filter names or into the gallery, from the panel or dropped
straight onto the page, and a photo element can fill its page or the whole
spread in one press.

Stops can be taken out of the arithmetic. A journal entry carries a
stats_excluded flag; an entry that has it is off the route, out of the
distance, out of the countries and not a step, while staying in the
journal. Studio lists every stop in the travel panel with a switch, the
entry editor carries the same switch on desktop and phone, and the entry
card says when a day is off the route.
2026-09-03 10:47:08 +02:00
jubnl b82962f95b Feat/plugin instance actions (#2209)
* feat(shared): scope plugin action descriptors

* feat(plugin-sdk): instance-scoped settings actions

* feat(plugins): run instance-scoped actions from the admin settings dialog

* feat(admin): instance-scoped plugin actions in the settings dialog

* fix(plugins): surface instance actions without instance fields and harden the action gate

* docs(plugin-sdk): note that hosts before 4.2.0 ignore action scope

* fix(admin): open the dialog for instance actions, explain inactive 404s, guard the save window

* docs(wiki): plugin pages for instance actions and the admin panel changes
2026-09-02 23:06:03 +02:00
jubnl 1e6ab069cd Fix/sdk default (#2199)
* feat(plugin-sdk): settings-field default + unknown-attribute warning

* feat(shared): default on the plugin settings-field contract

* feat(server): persist and serve plugin settings-field defaults

* feat(client): pre-fill plugin settings defaults in the admin and user forms

* feat(server): enforce required plugin settings on both save paths

* feat(client): block plugin-settings save on an empty required field

* feat(server): resolve plugin settings defaults at runtime

A settings-field `default` was stored and served to the form but never
reached the plugin: the child's init envelope, `ctx.settings.get()`, the
channel dispatch's readAll and the OAuth provider config all read the
stored blob only. Fold `stored ?? default` in on every path that hands
config to a plugin (settings-defaults.ts), and let a default satisfy
`required` on both save gates — refusing a save the runtime would have
accepted was a contradiction.

Also: one `isFilled` rule shared by the save gate and `hasRequired`
(checkbox exempt, whitespace empty); the install parser drops a
non-boolean checkbox default or a select default outside its options,
matching what the SDK already refuses; export SETTING_FIELD_KEYS so the
SDK's mirror can be parity-tested.

* feat(client): share the plugin settings form rules across all three forms

The mobile user settings screen was a third consumer of the fields/config
contract that never got `default` pre-fill or the required check, and
the seed/patch loops lived in five hand copies. Extract
seedSettingsValues / findMissingRequired / settingsPatch into one module
and use it from the admin instance form, the desktop user tab and the
mobile screen. The user forms now surface a 4xx `{ error }` from the
server (the required-field refusal) instead of the generic toast.

* feat(plugin-sdk): seed dev fixtures from manifest defaults

`trek-plugin dev` handed the plugin only the fixture values, so a plugin
relying on its manifest defaults read undefined in dev and only in dev.
Seed ctx.config / ctx.settings from the declared defaults (a fixture
value still wins) via a new exported settingDefaults(manifest, scope).
SETTING_FIELD_KEYS is now parity-tested against the host's list.

* docs(plugins): document settings defaults as runtime fallbacks

A `default` is the field's effective value wherever nobody set one, not
just a form pre-fill; `required` is enforced on both ends; unknown
attributes are dropped at install and warned on by validate. Add a
Settings section with the attribute table to the SDK README and update
the ctx type comments, the shared schema, and the wiki guide.

* add missing exports
2026-09-02 18:42:16 +02:00
Claude b7f9c1e8ae feat(journey): make the linked trips' GPX tracks a switch, off by default
#1260 drew the GPX tracks of every trip behind a journey onto its map, with
nothing to switch off — the reasoning being that a route imported into a
linked trip is part of the journey's story. For a trip carrying a season of
recorded drives it is instead a map nobody asked for, drawn from places that
never became an entry.

So it becomes a journey-level setting, owner-only like the other Journey
Settings, and it starts off: a journal should show what its author put in it,
and #1260's set is everything the linked trips happen to contain. The switch
gates the request, not just the rendering — the endpoint ships full unthinned
geometry, which is megabytes fetched on every journey open for a map nobody
wanted.

`journeyTracks()` itself is deliberately untouched, so TREK Studio keeps the
set its saved books were laid out with.

Also fixes what the reporter's screenshot actually shows: JourneyMap fed track
coordinates into `fitBounds`, so one drive across a country zoomed the journey
map out until the entries were specks. Tracks now only frame the map when
there is nothing else to frame, which is what JourneyMapGL already did — the
same journey used to open at two different zooms depending on the renderer.

Closes #2194

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S5VwmiBQ9sPCJwfmpv64r3
2026-09-02 16:21:50 +02:00
Maurice 2834967919 fix(i18n): drop the self-hosted wording from the airtrail description 2026-09-01 16:19:21 +02:00
Maurice 9bfdc19e68 fix(admin): naver_list_import is an integration, not a trip addon
It has no trip tab and no page of its own — it pulls places from an
external service, which is what the integration group is for. The type
only drives grouping in the admin UI (and the global nav check), so
existing installs just see the tile move; migration appended for them,
seed fixed for fresh ones.
2026-09-01 16:19:21 +02:00
Maurice 4b120d39c8 feat(api): separate API keys from MCP tokens, add travellers
An MCP token drives every assistant tool; an API key reads trips over HTTP.
Handing the same credential to a third-party integration gives it the whole
MCP surface, so the two are now separate kinds on `mcp_tokens` and the kind
sits in the WHERE clause — a token of the wrong kind is indistinguishable
from one that does not exist.

Also adds `travellers` to the trip payload: display name and an owner flag,
no ids and no email addresses. An integration key reads its owner's
itinerary; it is not a way to enumerate the people around them.

Documents the surface in the wiki, since integrators need something to read
that is not the source.
2026-08-28 15:23:05 +02:00
Maurice 15588d20c9 fix(feeds): keep a staged booking out of the public feed and the shared trip
The ICS feed is public with a token that never expires and prints the
confirmation number into the DESCRIPTION, and the shared-trip payload does a
SELECT * over reservations with share_bookings on by default. Neither filters
anything, so whatever lands on a trip is published the moment it is written.
That is fine while a person types every booking in, and it stops being fine as
soon as something writes on their behalf.

reservations.ingest_state is that gate. It defaults to live, SQLite backfills
every existing row on the ALTER, and no current writer names the column, so
every booking visible today stays visible. Nothing writes staged yet.

The predicate takes its table alias because share.service.ts queried
reservations without one. A stay is withheld only when bookings point at it and
all of them are staged: a hand-added hotel has no booking at all and has to keep
showing up.
2026-08-28 14:25:08 +02:00
jubnl 4ba407aead feat(plugins): install an older version of a plugin (#2035)
Adds a version picker to the plugin admin panel, on desktop and mobile: every
published version with its own server-computed compatibility verdict, so the
client never re-derives semver range logic.

A deliberate non-latest pick sets an update hold on the row, which keeps the
update banner and "Update all" from immediately nagging the rollback away. The
hold releases when an install lands back on the newest compatible version by any
path, or when the admin resumes updates explicitly. An unresolvable registry
never sets one: refusing to pin on missing information beats silently muting
future updates.
2026-08-25 17:30:27 +02:00
Maurice 38f3cb3101 fix(server): the second pass over this branch's own fixes
An adversarial review of what had already landed here turned up three that
were wrong, and they are the reason for this commit.

The CSP change removed 'unsafe-eval', which is not leftover: heic-to's
libheif build initialises embind through new Function(), and without it every
iPhone .heic upload fails in the browser. 'wasm-unsafe-eval' alone was tried
once already and was not enough (93b51a0b). Put back, with the reason written
down this time and the test pinning it rather than its absence.

The idempotency in-flight guard never fired. finalize() runs when the
handler's observable completes, but Nest writes the response - and therefore
stores the row - several microtasks later, so the waiting request always woke
up before the row existed and ran the handler a second time. It releases from
the point the row is written now; finalize is the backstop for handlers that
answer another way. The test drove the two in the opposite order from Nest,
which is why it passed either way.

And the trip-scoping check treated an id that resolves to nothing like one
belonging to someone else. accommodation_id has no foreign key, so shortening
a trip's dates leaves a booking pointing at a gap - which update() has always
healed by clearing the link, and which would have become a permanent 400.

The rest: a reply to a soft-deleted chat message is refused and the create
path blanks the quote like the list path does; the ws-token buckets are keyed
by account rather than by IP, where every user behind one NAT shared a
ceiling; the weather and vacay caps read through the stream, since a chunked
response has no content-length to check; the Google host gate takes country
domains again; abandoned response bodies are cancelled instead of left to GC;
the boot failure prints its errno; and a share link that is not yours to read
now says so instead of reporting that the journey has none.

TICKETJSON gets a real repair: the in-place GLOB fix only helps installs that
have not run it yet, so an appended migration puts back the note text the
case-insensitive LIKE cut off, for rows where what it stored is not valid
JSON. A genuine receipt always parses, so it cannot be touched.

Wiki: the audit log records the IP Express resolves after TRUST_PROXY, not
the leftmost X-Forwarded-For entry a caller can write. That is the right way
round, but it means the hop count has to be accurate - said plainly in both
places now.
2026-08-23 11:59:12 +02:00
Maurice a7aaa29c16 fix(db): stop losing secrets and notes to the boot and migration paths
An unreadable or empty encryption-key file was treated the same as a missing
one: regenerate and overwrite. That is the one case where regenerating is
the worst possible move - every secret already at rest becomes undecryptable,
permanently. Both now refuse to start and say which file to fix.

The key-rotation script advertised re-encrypting all stored secrets while its
hardcoded column lists had drifted: the AirTrail key, the per-user LLM key
and the plugin OAuth tokens were never migrated, so they silently stopped
decrypting after a rotation. The parity test now pins the full set against
what the code actually encrypts, rather than two tables of it.

The schema_version bump ran outside the migration's transaction, so a crash
in between left the schema advanced and the version stale - the next boot
replayed a non-idempotent step and exited, turning one crash into a boot
loop. And the TICKETJSON receipt migration matched with LIKE, which SQLite
treats case-insensitively, so a note somebody had typed starting 'ticketjson:'
was cut up and its text dropped.

Also: APP_VERSION was interpolated into migration SQL instead of bound, the
capture-metadata backfill passed (photoId, userId) to a (userId, photoId)
signature and its mock keyed on the wrong argument, TRUST_PROXY=0 was
accepted and silently coerced to 1, demo-reset hardcoded the db path instead
of honouring TREK_DB_FILE, and a demo instance with no DEMO_ADMIN_PASS set
now says out loud that its admin password is the published default.
2026-08-23 11:59:12 +02:00
jubnl 305b113d5f Feat/storage driver (#1949)
* feat(storage): storage key validation, driver contract suite, LocalDriver

* feat(storage): MirrorDriver composite backend

* feat(server): add TREK_UPLOADS_DIR env var for the uploads root

* feat(storage): settings-backed registry, StorageService facade, module wiring

* test(storage): pin src/nest/storage coverage thresholds

* feat(storage): multer upload options factory

* refactor(files): spool trip-file uploads through storage

* refactor(collab): spool note attachments through storage; normalize files/ prefix

* refactor(journey): spool journey media uploads through storage

* refactor(trips): spool trip cover upload through storage

* refactor(places): spool place-image and collection uploads through storage

* refactor(auth): spool avatar upload through storage

* fix(server): consolidate allowed_file_types onto AllowedFileTypesService

* fix(storage): default defParamCharset to utf8 in the upload factory

* test(db): re-point the trailing-migration guard at the trip_files prefix strip

* fix(storage): age-gate the boot spool sweep

* test(files): key FILE-P02 residue checks on upload size

* test(storage): raise src/nest/storage coverage pin

* refactor(platform): serve /uploads routes through storage

* refactor(files): serve trip-file downloads through storage

* refactor(memories): serve photo bytes through storage

* refactor(maps): stream place-photo bytes through storage

* refactor(share): stream shared place-photo bytes through storage

* refactor(journey): serve public journey assets through storage

* fix(share): answer place-photo cache miss with 204 like the maps proxy

* test(server): close serving-swap gaps surfaced by the full tier

* test(storage): raise src/nest/storage branches pin

* refactor(place-photos): route place-photo cache through storage

* refactor(places): reclaim place images and collection covers through storage

* refactor(trips): trip covers through storage

* refactor(auth): avatar reclaim through storage

* refactor(collab): note-file deletion through storage

* refactor(files): trash deletion through storage

* refactor(memories): trek photo cache internals through storage

* refactor(memories): thumbnail service category-addressed

* refactor(memories): immich upload reads through storage; drop uploads-root

* refactor(journey): photo deletion through storage

* fix(memories): reclaim orphaned trek photo cache objects in the ttl sweep

* feat(journey): journey-thumbs orphan sweep and delete-site derived-artifact reclaim

* test(server): close slice-4 gaps surfaced by the full tier

* test(coverage): raise slice-4 domain pins

* refactor(backup): list, existence and delete through storage

* refactor(backup): download through storage sendToResponse

* refactor(backup): create backup through per-category storage enumeration

* refactor(backup): restore and upload-restore through storage

* refactor(backup): auto-backup retention through storage

* refactor(server): boot dir-creation to driver init

* test(server): close slice-5 gaps surfaced by the coverage tier

* test(coverage): raise slice-5 backup pin

* test(server): resolve async backupFileExists mocks for the tests typecheck

* test(server): pin ranged-read stat and zero-byte roundtrip in the storage driver contract

* feat(server): TREK_S3_* env wiring — schema, s3 namespace, all-or-nothing boot rule

* feat(server): S3 driver core — aws-lite client seam, stat/delete, driver-owned deadlines

* refactor(server): type the aws-lite S3 client via s3-types reference

* feat(server): S3 driver getStream and list — ranged full-object stat, iterator pagination

* feat(server): S3 driver put — bounded-peek routing, Upload inactivity deadline

* fix(server): S3 driver Upload monitoring — Transform interposition, stream cleanup

* feat(server): env-declared s3-main storage backend; settings-declared s3 rejected

* test(server): env-gated S3 driver contract suite against MinIO

* fix(server): S3 driver zero-byte put — work around aws4 duplicate Content-Length signing defect

* fix(server): scope the zero-byte put workaround to buffered bodies; bind test MinIO to localhost

* ci(server): MinIO-backed S3 driver contract job

* ci(server): align s3-contract shim comment and MinIO bind with references

* docs: S3 storage backend configuration and off-box backup mirroring

* docs: restore README formatting, keep S3 additions only

* ci(server): make live-MinIO contract runs manual-only (AIStor licensing)

MinIO's community line is discontinued entirely; the maintained successor, AIStor, requires a license key that can't safely live in a shared CI runner. The s3-contract CI job is removed; live-provider contract runs against MinIO/AIStor are now a manual, license-gated procedure documented in docker-compose.minio-test.yml's header.

* chore(server): final-review polish — stale CI refs, dockerignore key, boundary pins, zero-byte tempfile contract case

* expose admin ui

* feat(shared): storage backend type registry, wire schemas, category relocation

* feat(server): accept settings-declared s3 backends — shared-schema options, decrypt-at-build, registry preview()

* feat(server): remove TREK_S3_*/TREK_UPLOADS_DIR — tombstone boot rule, built-in override as the relocation path

* feat(server): deriveSecurity.encryptionKeySet and storage secret plumbing (gate, encrypt, unmask, mask, redact)

* chore(server): drop storage-secrets import that only Task 8's probe helper will use

* feat(server): seed-once storage-config.json boot import — fail closed on any invalid seed

* feat(server): registry snapshot() — effective storage config with provenance

* feat(server): api/admin/storage GET/PUT — masked secrets, preview-gated saves, wired reload, audits

* feat(server): api/admin/storage/test — per-target ephemeral probes, mirror targets probed individually

* test(server): e2e coverage for api/admin/storage — guards, managed refusal, envelope, probes

* test(server): give e2e harness DBs the app_settings table the storage registry reads at boot

* test(coverage): raise pins after the storage admin-config slice

* fix(server): final-review polish — encryption-gate wording, all-field sentinel guard, README/e2e notes

* test(server): storage slice-1 review follow-ups — mirror s3-replica probe decrypt, all-encrypted seed without key, test-route 401

* feat(i18n): storage admin domain + tab label in every locale

* feat(client): adminApi storage endpoints — getStorage, updateStorage, testStorageBackend

* feat(client): storage admin draft-document helpers + relative-time util

* feat(client): schema-driven storage BackendForm — fields by kind, mask echo, encryption banner

* feat(client): admin storage panel — backends list, category map, health strip, one-PUT save

* feat(client): storage tab on the admin page

* feat(client): storage admin panel and section on the phone shell

* fix(client): block renaming a storage backend onto an existing name

* feat(server): allow mirrors on any storage category — lift the backups-only rule

* feat(i18n): replicas-on-primary mirror strings in every locale

* feat(client): mirror fold/synthesize model helpers for replicas-on-primary

* feat(client): BackendForm mirror-targets composer; raw mirror type hidden

* feat(client): desktop storage panel folds mirrors into their primaries

* feat(client): phone storage panel folds mirrors into their primaries

* fix(client): strip removed backends from remaining mirror replicas — no phantom refs

* feat(storage): retire photos as a configurable category — served-only via the ServedCategory seam

* feat(i18n): descriptive storage category names and descriptions in every locale

* feat(client): descriptive category names, descriptions and id badges in the storage panel

* docs(storage): fix stale nine-category comments after the eight-category shrink

* docs: storage configuration via the admin UI and seed file — README + compose

* docs(wiki): Admin Storage page and storage cross-links

* docs(server): storage domain blueprint entry; TREK_PLACE_PHOTO_DIR tombstone clarifier

* refactor(server): drop the storage env-var tombstones — the variables never shipped

* docs: storage config is simply new in v4 — drop the removed-variables story

* feat(shared): backfill-status and usage wire schemas on the storage admin state

* feat(server): MirrorDriver.backfill — prefix-scoped, size-compared, cancellable replica catch-up

* feat(server): one-at-a-time storage backfill jobs + the storage events bridge

* feat(server): storage usage scan — nightly cron, on-demand service, persisted with computedAt

* feat(server): replica_failure admin notifications — events bridge, hourly debounce, channel registration

* feat(i18n): replica-failure notification text in every locale

* feat(server): storage backfill and stats-refresh admin endpoints

* feat(client): adminApi backfill and stats-refresh endpoints

* feat(i18n): storage sync and usage strings in every locale

* feat(client): desktop storage panel — sync now, usage display, backfill polling

* feat(client): phone storage panel — sync now, usage display, backfill polling

* fix(client): per-category usage line on the phone storage panel

* docs: sync now, usage stats and replica-failure notifications

* test(server): admin scope now includes replica_failure, not just version_available

Commit 6db135da added replica_failure to ADMIN_SCOPED_EVENTS but left
NPREF-005b asserting the admin scope contained only version_available.
Update the expectation to match.

* test(server): cover storage backfill/stats/jobs branches to hold the coverage pin

* fix(client): guard storage admin polling against stale responses racing a save

A poll GET issued before a save's PUT resolves could land after the save
applied its fresher response, overwriting it with the pre-save world and
re-deriving the draft from that stale snapshot. useStorageAdmin now tracks
a state-application sequence number and drops any refreshState() response
whose sequence has moved on, or that resolves while a save is in flight.
Also guard the fire-and-forget refreshState() calls in startBackfill/
cancelBackfill against unhandled rejections, matching the poll's style.

* fix(server): report 404 before 409 when a backfill start names an unknown mirror

startBackfill checked the global busy flag before validating that :name is
a mirror routed by some category, so POST .../backends/garbage/backfill
answered 409 instead of 404 whenever a sync was already running. Resolve
and validate the target first (BackfillTargetError -> 404), then check
busy (BackfillBusyError -> 409).

* feat(client): two-column category grid on the storage panel

* feat(server): plugin files.getContent reads through the storage layer

* feat(server): plugin files.create writes through the storage layer

* refactor(server): drop FilesService.resolveFilePath, last direct-fs seam in the plugin surface

* fix(server): map StorageInvalidKeyError to the plugin file accessibility envelope

* feat(storage): allow credentialed backends without an explicit ENCRYPTION_KEY

Secrets are still always encrypted at rest; without an explicit key the
implicit data-directory key is used. Removes the plaintext-secret refusal
from the admin PUT and the seed import, the encryptionReady GET field, the
save-blocking banner in both admin shells, and the now-consumerless
security.encryptionKeySet env derivation. Docs now recommend (not require)
an explicit key, since the implicit key rides inside backup archives.

* test(client): pin the stale-poll release to the pre-save GET in FE-ADMIN-STOR-024

The stub deferred EVERY poll GET and rebound the release handle to the
newest one, so a tick landing after the save was released with the
pre-save payload — a response no real backend can produce, which the seq
guard rightly applied. Only the first (genuinely stale) GET is held now;
later GETs answer with the world the mock server currently holds. Also
drops the encryptionReady fixture field removed from the schema in the
previous commit.

* fix(storage): backfill puts carry content-type, stat failures report op stat, progress clamps

* feat(storage): category migration job — copy, flip, delta sweep, reclaimable tally

* feat(storage): migration start/cancel endpoints and client API

* test(client): storage admin fixtures carry the new migrations field

* fix(storage): migration POST answers 200 and validates its body via the shared schema

* fix(storage): cancel running storage jobs when a config save removes their backend

* fix(notifications): in-app replica-failure row carries the suppressed count

* feat(client): migration prompt, queue, and progress on the storage panel

* fix(client): migration queue holds a synchronous single-flight lock

* feat(client): migration prompt and progress on the mobile storage panel

* fix(server): register category-migration routes in the managed-route allow list

StorageAdminController.migrationStart/migrationCancel were added under the
controller-level @ManagedForbidden() decorator but never added to
MANAGED_ROUTE_ALLOW_LIST, so validateManagedRoutes() threw "Managed surface
changed" on every buildApp() call, failing 46 test files (any test booting
the real app).

* test(server): cover category-migration copy/sweep branches to hold the storage coverage pin

Adds MIG-005 (a target object already matching size/key is skipped in the
copy phase itself, not just the delta sweep) and MIG-006 (a copy failure
during the delta sweep is counted without undoing the flip that already
happened). src/nest/storage/**'s branch coverage was 83.91%, just under the
84% pin.

* docs(storage): category migration — move existing objects on reassign

* fix(storage): close the pre-flip cancel window and cancel re-routed migrations

A cancel landing between the last copy iteration and the category flip was
silently ignored, and a config save that re-routed the migrating category to
a third, still-defined backend never stopped the in-flight migration. Also
guards assignCategory against writing an unknown backend name, and fixes a
missing `suppressed` notification param being treated as truthy suppression.

* fix(client): migrations keep mirrors, queue waits on backfills, sweep failures surface

Reassigning a category onto a mirrored primary now posts the migration to the
mirror's wire name instead of the bare primary, so replication survives the
move. The queued-migration effect also waits out a running backfill (the
server's one-job rule spans both), and a done migration with sweep failures
now surfaces a doneFailures line in both shells, translated into every locale.

* ci(server): run the S3 contract suite against MinIO

Resurrects the s3-contract job removed in 6fbff835, adapted for AIStor:
writes the MINIO_LICENSE_KEY secret to a license file, boots the licensed
container, and runs the contract suite hermetically (no --coverage) so the
storage coverage pin keeps holding from unit tests alone. Gated to same-repo
PRs since fork PRs have no access to the license secret.

* fix(server): backup archives uploads from a remote primary

createBackup() enumerated uploads through withLocalFile, which downloads a
remote object to a temp file, hands it to the callback, then deletes it
before archiver ever reads it — archiver reads entries lazily during
finalize(), so a remote (S3) primary silently backed up zero upload bytes.

Add StorageService.getLocalPathOrNull(category, name): a narrow locality
probe that returns a real path only when one exists on disk right now (the
fs-existence check is the fail-safe half of the contract — a driver-reported
path that vanished between list() and here falls back too, not just a
path-less remote driver). The enumeration loop uses it directly when
available (zero-copy, unchanged for a local primary) and otherwise streams
the object into a per-backup staging dir under the backups spool, cleaned up
in the existing finally alongside zipSpool/dbSnap.

Also wire archive.on('warning', reject): archiver treats a missing/unreadable
entry as a warning, not an error, and skips it by default — which would
silently drop the entry from an otherwise "successful" backup.

* fix(server): client aborts stop 500ing and leaking streams

storage.service.ts sendToResponse: the local res.sendFile branch now treats
ECONNABORTED/ECONNRESET/ERR_STREAM_PREMATURE_CLOSE/syscall=write as success,
mirroring res.sendFile's own default callback (the contract
storageStaticHandler already implements caller-side). The remote branch
replaces the hand-rolled pipe+Promise with pipeline(stream, res), which
destroys the source stream on any failure — fixing the leaked S3 keep-alive
socket on a client abort — while swallowing the same client-abort error set;
a real source error still rejects so callers keep their miss contracts.

This centralizes the fix for every caller that streams through
sendToResponse: files-download, journey-public, photo-resolver,
trek-photo-cache, backup downloads, and /uploads/photos.

trek-exception.filter.ts now no-ops (destroying the socket) when headers are
already sent, so no filter path ever writes onto a response in flight.

maps.controller.ts and share.controller.ts's place-photo bytes proxies
switch their direct getStream+pipe to pipeline() with the same abort-tolerant
catch, keeping the existing 204-empty fallback for pre-header failures.

* fix(storage): mirror read fallback consults replicas on primary miss

MirrorDriver.getStream/stat now fall back to replicas on a primary miss
(StorageNotFoundError / null stat), not just an infra error — binding
decision: a read succeeds if ANY member holds the object. Only a
deterministic StorageInvalidKeyError skips fallback.

stat() gets a dedicated fallback loop instead of reusing fromReplicas:
a replica returning null means "keep looking," which the generic
catch-and-continue helper would have short-circuited on.

StorageService.sendToResponse/withLocalFile stop treating composite
driver.stat() as proof the object is at localPath; both now fs-check
the path itself and fall through to the stream branch when it's
missing, which is what actually reaches a MirrorDriver's replicas.

* feat(storage): remote serving parity — content-type, validators, range

Local backends get content type, ETag/Last-Modified, conditional GET, Range
and HEAD for free from res.sendFile; the proxying branch every remote
backend (and a local primary whose file vanished) falls through to had none
of it — every request re-streamed the whole object with no cache validator
and no way to seek.

sendStreamed() reproduces the contract from ONE pre-flight driver.stat():
a miss throws before any header is set (the caller's 404/204/next()
contract), and that same stat then answers 304s, 416s, HEADs and suffix
ranges without ever opening a body. The decisions live in a new pure module,
http-serving.ts:

- validatorsFor: the driver's ETag, else express/send's weak
  W/"<size hex>-<mtime hex>". mtimeMs 0 (S3 can report it) means no
  Last-Modified and no derived tag — serving uncached beats a 1970 header
  that reads as fresh forever.
- isNotModified: If-None-Match (weak compare, *, list) wins over
  If-Modified-Since, per RFC 9110 §13.2.2.
- decideRange: single range only, inclusive clamp, suffix computed from the
  pre-stat size, unsatisfiable → 416, If-Range strong-exact or full entity.
  A multi-range or malformed spec is answered with the full object rather
  than multipart/byteranges. If-Match/If-Unmodified-Since stay out of scope.

Range math uses the pre-stat size as the single source of truth, the ranged
getStream stat is never re-consulted, and the pipeline abort semantics are
unchanged.

S3Driver now surfaces the bucket's ETag on stat()/getStream() (on a ranged
GET it is still the full object's tag). aws-lite strips the tag's
surrounding quotes — verified live against AIStor RELEASE.2026-08-07 — and
an unquoted entity-tag is not a valid HTTP ETag, so the driver restores
exactly the quotes the SDK removed and nothing else.

* fix(server): key rotation covers storage.backends

The S3 storage backend's secretAccessKey is encrypted at rest inside the
app_settings row keyed storage.backends, but scripts/migrate-encryption.ts
didn't know about it — after a key rotation that secret stayed encrypted
under the old key and silently failed to decrypt on the next read.

The script now re-encrypts secretAccessKey for every s3 backend inside the
storage.backends JSON array, using a pinned copy of the secret-field list
(it deliberately can't import @trek/shared). The parity test pins that copy
against storageSecretFields('s3') so a new secret field can't be missed
silently.

* fix(server): restore reloads the storage registry

restoreFromZip closed and reopened the DB but never reloaded the
storage registry, so uploads rehydration ran against the STALE
pre-restore driver/category map instead of the config the restored
archive's own travel.db carries (audit #4, major).

StorageService gains reloadConfig(), a narrow passthrough to the
(unexported) registry's reload() — same shape as health(). backup.impl's
restoreFromZip now calls it right after reinitialize() and right before
the uploads wipe/rehydrateUploads/plugin staging, so rehydrated bytes
land where the restored config says. When reopen fails (reinitFailed),
reload and rehydration are skipped entirely — already reported as
"restart required", and there is no live DB handle for the registry to
read anyway.

* fix(server): category migration rewrites keys across prefix changes

Category migration copied objects to the destination backend using the
SOURCE key verbatim. A mode-A ('' prefix) source migrating onto a
prefixed backend (or vice versa) landed objects at keys the registry
could never resolve post-flip (audit #8).

Extracts StorageRegistryService.keyPrefixFor(category, backendName) from
build()'s inline conditional so startMigration can compute the
destination prefix before the flip. runMigration now maps every
destination key (destPrefix + key.slice(sourcePrefix.length)) in the
copy skip-check, the put, and the delta sweep; the reclaimable tally
stays source-keyed. Equal-prefix migrations are unaffected (identity
mapping).

* fix(plugins): await the now-async place/collection services in RPCs

PlacesService.update/remove and CollectionsService.deletePlace became
async when they started deleting storage objects, but their plugin RPC
handlers still called them synchronously: the null/false refusal check
compared against an always-truthy Promise, and places.update broadcast
a Promise instead of the resolved place. Both handlers now await the
service call; mapCollectionError is async-aware (accepts a sync-or-
async thunk) so the sync callers (create/update/savePlace/copyToTrip)
keep working through the same path. Verified the RPC host already
awaits handler results, so no change was needed there.

* fix(storage): backfill Sync now gains a deletion sweep

The Sync now backfill copied primary objects to replicas but never removed
replica objects the primary no longer has, leaving mirror read-fallback able
to serve stale data indefinitely after a failed replica delete. backfill()
now sweeps each replica post-copy: any replica key absent from the primary
key-set is re-checked against primary.stat (a race gate against the primary
regaining the key) before being deleted. Adds a deleted counter end-to-end
(driver, jobs service, shared schema, both admin shells, all 23 locales) and
a new ReplicaFailure 'list' op for a replica whose listing itself fails
during the sweep.

* feat(storage): optimistic concurrency for admin config saves

A stale admin PUT of storage settings could silently undo a category
migration's just-flipped assignment. Adds a version counter
(app_settings row storage.config_version) bumped inside the same
transaction by both writers — applyConfig and the migration flip
(assignCategory). applyConfig compares the submitted version first
and throws StorageConflictError, mapped to 409 ahead of the blanket
400. The client carries the draft's loaded version through every
local edit and maps a 409 to a distinct "settings changed underneath
you" message, refreshing state while preserving the dirty draft for
review.

* fix(storage): surface a failed config load + redact secretAccessKey

Registry load() now records the exact parse/validate failure as
lastLoadError() instead of silently falling back to last-good config or
built-in defaults; StorageAdminService exposes it as configError on the
admin state (storageAdminStateSchema extended additively), and
AdminStoragePanel renders it as a warning banner naming the error. Save
stays enabled — it's the recovery path — but an admin can no longer save
over a broken stored config without knowing.

Also adds 'secretaccesskey' to globalMiddleware's SENSITIVE_KEYS request-log
redaction list, so storage admin PUT bodies never leak the S3 secret into
debug logs.

* fix(client): admin storage panel state races

Migration prompt now recomputes computeMigrationCandidates(draft, state)
on render and again at confirm time, so a stale open-time snapshot never
drives stripCategories or the migration queue; adds a Cancel button to
the dialog. A failed startMigration POST explicitly clears the remaining
queue with a toast naming the dropped categories and refreshes state,
instead of leaving it invisibly stranded; a "Queued: X, Y" line renders
while the queue is non-empty. Testing a mirrored row now expands the
draft primary + replicas client-side and probes each individually,
instead of sending the mirror stub to the server (which resolved
targets from its own live snapshot, silently testing saved options over
unsaved draft edits). A completed/cancelled/errored backfill now renders
its terminal status alongside the Sync now button instead of hiding it,
so it can be re-run without a page reload.

* fix(storage): refuse shared replica backends so the sync sweep cannot cross categories

A mirror's "Sync now" sweep makes each replica MATCH the primary: it lists the
replica under every prefix the mirror's categories occupy and deletes the keys
the primary doesn't hold. The backups category's prefix is '' — the replica's
entire root — so making a category-serving backend (say uploads-local) a
replica of a backups mirror meant the first sync deleted every user upload.
Nothing forbade the config: validateConfig only checked that mirror targets
existed and weren't nested, and the admin picker offered every backend.

The sweep is correct, so the fix is at the config level:

- validateConfig refuses a backend that is both a mirror replica and a
  category target (directly, or as the primary of a mirror a category routes
  to), and refuses a backend replicating two mirrors whose swept prefixes
  overlap — '' overlaps everything. Disjoint prefixes stay legal.
- assignCategory (the migration flip, the other writer of the settings rows)
  refuses routing a category onto a backend that is currently a replica.
- The admin mirror-target picker (both shells) only offers backends that serve
  no category; already-selected targets stay listed so they can be unchecked.
  A category assigned to a replica still surfaces the server's 400 verbatim.

Existing shared-replica fixtures were rewritten onto dedicated replicas.

* fix(client): a 409 conflict save is recoverable without a page reload

After a 409 the admin storage panel was a dead end. refreshState leaves a
dirty draft's version alone on purpose (audit #7: the background poll must
never float a mid-edit draft past a migration flip it never carried), setDraft
re-attaches that same version to every later edit, and save() resubmits it —
so every retry 409s again and only a full page reload got the operator out,
while the banner told them to "review and save again".

The version block stays. Instead the conflict banner now offers the one action
that can clear it: discard the local edits and reload the saved settings —
a fresh GET applied through applyState, re-deriving the draft (and its
version) and clearing dirty/saveError. Bumping the version in refreshState,
or only in the 409 branch, was rejected: either resurrects audit #7, since a
full-document save would then revert a flip the draft never carried.

storage.saveConflict now points at that action instead of implying a plain
re-save works (all 23 locales). The escape lives in the shared hook, so the
phone panel renders it too.

* fix(client): deflake the queued-migration-waits-out-backfill panel test

FE-ADMIN-STOR-037 seeds a running backfill, so the panel polls GET from
first render — but the stub served the post-save world (files: off-box)
before the PUT had landed, which no real server would do. A poll
resolving while the confirm dialog was open made the confirm-time
candidate recompute see from === to, so nothing was queued and the
migration never POSTed.

The GET stub now stays faithful (pre-save world until the PUT lands,
usage included so candidates keep their object counts), and the test
deterministically forces the once-flaky interleaving: it waits for
polls to land while the dialog is open and asserts the candidate
survives them.

* fix(storage): surface the underlying cause in probe target errors

The admin storage Test probe reported only the driver wrapper's outer
message ("put failed for '…' on 'probe-s3'"), dropping the cause that
says WHY — connection refused vs missing bucket. Flatten the error's
cause chain into the probe result, appending an S3-style code when it
isn't already in the message.

* fix(storage): surface the underlying cause in replica-failure reports

Same defect as the probe fix, one layer over: MirrorDriver's
reportReplicaFailure captured only the wrapper's message, so
replica-failure notifications said "put failed" without the why.
Move the probe's cause-chain flattener into storage.types.ts as
describeError and use it in both call sites.

* fix(storage): strip ContentType from multipart Upload calls (MalformedXML)

Every multipart S3 put carrying a contentType — every backup zip and
photo over the 10MB threshold — failed with 400 MalformedXML. Upstream
@aws-lite/s3's Upload leaks the ContentType param into its final
CompleteMultipartUpload headers ({ ...xml, ...paramHeaders }), clobbering
application/xml, which flips @aws-lite/client's payload serializer to
JSON — so the server receives a JSON parts list where XML belongs.
Confirmed live against MinIO AIStor; any S3 refuses the same.

uploadMonitored now drops ContentType before s3.Upload. Multipart
objects are stored without a content-type, which TREK never reads back
(serving derives Content-Type from the key/caller, never object
metadata); single-request PutObject paths keep sending it.

Also closes the coverage gap that let this ship: the live contract
suite never crossed MULTIPART_THRESHOLD, so s3.Upload had never run
against a real server — it now round-trips a >10MB put carrying a
contentType.

---------

Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-08-22 11:33:55 +02:00