Commit Graph

4720 Commits

Author SHA1 Message Date
Maurice 0b598680ec test(storage): spell the pending-reset case title without a stray quote 2026-11-01 18:32:43 +01:00
Maurice 13332cf06b fix(storage): a pass on a database this installation does not recognise only reports
While a restore's reset is still pending, or the database names another
installation, the nightly pass judges and reports but deletes nothing,
whatever the restored switch says.
2026-11-01 18:32:33 +01:00
Maurice e6d52cb061 docs(wiki): say every restore resets the cleanup state and what the release queue gives up
A restore of this installation's own backup now resets it too, through the .trek-reclaim-reset file. Files dropped from the release queue after 30 days, or refused by a full queue, are only removed by the nightly cleanup once deletion is on.
2026-11-01 18:31:45 +01:00
Maurice bef8fc20f1 fix(storage): reset reclaim state after any restore, re-check owners often, share release ticks
- A restore (admin panel or RESTORE_FROM_BACKUP) now leaves
  data/.trek-reclaim-reset before it swaps the database in. Boot reconcile
  resets the reclaim state when it finds the file, whatever install id the
  database names, and removes it afterwards; a successful reopen removes it
  too. A restart after a failed reopen, or a boot restore of this
  installation's own backup, no longer keeps the old switch, marks and
  confirmations. The id mismatch reset stays.
- The release job skips a tick (install_mismatch) while the database names
  another installation or a reset is still pending.
- An allowed owner verdict is cached for 15 minutes, a refusal for 26 hours.
  The local driver reads a list prefix as a folder, so the category root is
  still listed whole, at most once per category per window.
- Each tick reads every allowed category's due rows separately, an equal
  share of the 200 each, so a category whose deletes keep failing cannot
  starve the others.
- Rows dropped after 30 days overdue and releases refused by a full queue
  are logged at warn with their counts.
2026-11-01 18:31:41 +01:00
Maurice 9d2ffc91af feat(storage): confirm the count shown, and say a backup or a missing install id held the cleanup
Confirm sends the count the admin saw and waits for its answer; a stale
count reloads the state. A running backup is named as a skip reason, a
missing installation id and a stale count as errors, and the switch's
hint no longer promises removal right away: deleted files go a day
later, once no undo can bring them back. Strings in all 28 languages.
2026-11-01 18:14:49 +01:00
Maurice 10ad64d170 docs(wiki): describe the day-later release, the install id file and the confirmation margin
Covers what a clone must do (its own data dir, or delete .trek-install-id),
that an older TREK writes no owner marker and cannot be detected, and the
new backup_running reason.
2026-11-01 18:11:47 +01:00
Maurice 1c6a175ebc fix(storage): queue what a delete left unnamed past every undo, and tell a copied database apart
A delete or a replace no longer deletes inside the request. The owner's call
queues the keys in storage_release_queue, due after RELEASE_DELAY_MS (26 h:
the assistant's 24 h approval window plus a margin), and StorageReleaseJob
takes the due rows every ten minutes, at most 200 a tick and 50 keys a
question, asks every reference source again and deletes what none names.
It refuses what the nightly pass refuses (StorageReclaimGuardService: the
ledger, the required sources, a shared location, another installation's
owner marker, cached per category), and waits while a restore or a backup
runs or another storage job holds the slot. An assistant undo or a late
write that names the old value again keeps its object.

The install id lives in data/.trek-install-id, outside any backup and never
written by a restore. At boot and after every reopen the database is
reconciled with it: a database naming another installation (a boot restore,
a restart after a failed restore, a copied travel.db) loses its switch,
confirmations, reports, marks and queue before anything reads them.

A confirmation now stores the count the admin was shown and when; it covers
up to one and a half times that count (at least 100 more) for 14 days, is
used up once the share is back under a fifth, and a count the last report
no longer holds is refused as reclaim_confirm_stale. The admin state folds
a live confirmation in, so the prompt goes at once.

A pass reads each source once for every category and drops each category's
keys once judged; right before a batch it reads the category's sources
again, whole and in memory, instead of a LIKE per key. A narrowed read holds
at most 50 LIKE terms. A pass held off by a backup reports backup_running,
and a category the ledger never reclaims reports not_reclaimed even after a
stop.
2026-11-01 18:11:43 +01:00
Maurice 0a25de2c6d feat(storage): confirm the reclaim count the admin was shown, name a running backup as a skip, bound a trip cover
The confirm request now carries the unreferenced count the admin saw, so the
server can refuse a count its last report no longer holds. A pass that found
objects due while a backup read the uploads reports backup_running for them.
A trip cover is bounded to 2048 characters on REST, RPC and the MCP tool, as a
place image already is, since a replaced cover is queued for release by every
upload path it spells out.
2026-11-01 18:11:25 +01:00
Maurice 59a6d92cea fix(storage): type the key search run, split the reclaim view per category
The longest-run helper typed its reduce as never, reclaimView grew past
the complexity limit (one function per category line now), and a test
read a promise as a boolean.
2026-11-01 16:37:00 +01:00
Maurice e0982e708f feat(storage): show running checks, confirmations and the nightly result in Admin > Storage
The panels follow the reworked reclaim state: Check now starts a check
in the background and the panel polls until it is done, a category with
an unusually large share of unused files offers Confirm, Freed comes
from the last run that deleted, a stopped check says why and no longer
reads as nothing found, the switch waits for its answer and a poll that
was in flight cannot flip it back, and refusals name their reason in
the operator's language. Strings updated in all 28 languages.
2026-11-01 16:25:38 +01:00
Maurice d17457de56 fix(storage): delete what a delete or a replace left unnamed right away again
Leaving every replaced cover, place image and deleted journey photo to the
nightly pass kept them publicly fetchable by URL for a week at least, and
for good while deletion was off. Owners now hand the values a delete or a
replace left behind to StorageReleaseService once their write committed.
It asks every registered reference source, with the same parser as the
nightly pass, and deletes at once what no row names any more: an object a
trip copy, a journey or a saved place still shows stays, which the old
inline deletes got wrong.

The release holds the pass's guards (reclaimable categories at their root
only, never an owner marker, never while a restore runs, never with a
required source missing or an overlapping location), never fails the
request and logs at debug. It runs for trip covers (update, upload, trip
delete), collection covers and saved-place images, place images (update,
bulk update, delete), journey covers and journey or gallery photos
(their thumbnail goes with a file that went), avatars (change, delete,
account deletion), trip files (permanent delete, empty trash) and collab
note and chat attachments. A trip file row now goes before its bytes, so a
failing backend no longer fails the delete. The objects of a deleted trip's
places and files are left to the nightly pass: their rows are other
domains'.

PlacesService drops the collection places repository nothing read any
more. Size, warning and boundary baselines follow the shrunk files.
2026-11-01 16:18:21 +01:00
Maurice 790fccb92c fix(storage): judge unreferenced files without a brake anyone can trip
The 20 % brake skipped a category and wiped its marks every night once a
fifth of it looked unreferenced. Any user could push a small install over
that line, a backlog from #2924 started braked, and a second install on the
same bucket below the line was not caught at all. It is gone:

- Each pass leaves a trek-owner-<install id> marker at the root of every
  category it judges. A category holding another install's marker is
  skipped as foreign_install, one where the marker cannot be written as
  owner_unknown. Backups leave the markers out.
- The install id lives in app_settings outside the storage.* prefix. The
  switch is stored as that id, so a database from another install reads as
  off, and every reopen (a restore) writes this install's id back, turns
  deletion off and drops the marks, confirmations and reports.
- More than a fifth unreferenced still marks and reports, but deletes only
  after an admin confirmed the count shown (POST .../reclaim/confirm). A
  larger count later waits again.

A skip no longer touches the marks, so a failed listing or lookup does not
restart the week. Check now runs in the background and answers 202; the
admin state says running and keeps the last deleting report apart, so a
report-only check never resets what was freed. A pass that deletes writes a
system audit entry, a pass that stops or fails keeps its partial counts and
marks the rest interrupted, a backend error stops deleting in its category,
and deleting stops for the rest of the pass once the switch is off or a
backup is reading the uploads. The usage scan waits for a pass still
running.

Sources read each category once per pass and answer the pre-delete
re-check with a query narrowed to the keys' plain runs. Bare names keep
their raw spelling (a '#', '?' or trailing blank is part of an upload's
name), and local roots are resolved through their symlinks before the
overlap check. A pass lists only a category's root.
2026-11-01 16:18:08 +01:00
Maurice e59992d56d chore(ci): lower the knip baseline the reclaim branch shrank
storage-registry.service.ts no longer exports a type nothing uses.
2026-11-01 14:55:54 +01:00
Maurice 382797febc fix(storage): let the app boot with the reclaim routes and keep the ratchets where they were
The two admin routes go on the managed allow list, without which buildApp
refused to start. The reclaim calls move to their own api module so
client.ts stays at its size, the phone card sizes its text through the
type tiers, and the tests drop a hand-built service, a non-null assertion
and a promise read as a boolean.
2026-11-01 14:44:53 +01:00
Maurice a1d9cdc8f2 feat(storage): show unreferenced files in Admin > Storage, with the switch and a check
Desktop and phone show what the last check found (can be freed, unused for
less than a week, freed), why a category was skipped, the switch that lets
the nightly run delete and a report-only check on demand, through the
shared storage panel hook. All 28 languages, and the Admin-Storage wiki
page explains the check, the week's wait and when to leave it off.
2026-11-01 14:30:01 +01:00
Maurice b31666d1c9 fix(storage): re-check a batch right before its delete, and drop reads nothing calls
The last question to the sources now comes after the object's own check,
so nothing slower than the delete sits between the answer and the delete,
and it is asked once per batch instead of once per object: a pass read
every source's columns for each object it was about to remove. The
collection reads the removed inline reclaims used are gone with them.
2026-11-01 14:23:04 +01:00
Maurice 19e4f0dadf fix(storage): keep the reclaim pass and a backup restore apart
A restore wipes and rehydrates the uploads under a swapped database, but it
never took the storage job slot, so a reclaim pass could start or carry on
beside it, and a delete already past the generation check could land on an
object the restore had just put back. BackupService now marks a restore as
running on StorageJobsService and waits for a pass in flight to let go; a
pass refuses to start while a restore runs and stops at its next check when
one begins. Backfills and migrations are not affected. The service test also
reads its fake source sets through a helper instead of non-null assertions.
2026-11-01 14:20:11 +01:00
Maurice 8fde643347 fix(storage): make the reclaim state on the admin storage state nullable
The shared contract made reclaim a required field, so every client fixture
typed StorageAdminState stopped compiling. It is nullable now, as planned,
and the five base fixtures carry reclaim: null.
2026-11-01 14:20:11 +01:00
Maurice f44dd92ee0 test(storage): run the reclaim e2e and source check on the whole app
The reclaim e2e and the source-registration check mocked src/db/database,
which lint:test-mocks holds at zero for new files; both now boot buildApp
against the worker's snapshot db. The key parser drops its non-null
assertions (the indexed reads are typed string already), and the e2e sends
its refused bodies at once instead of awaiting inside a loop, so the eslint
ratchet stays where it was.
2026-11-01 14:20:10 +01:00
Maurice db0fbc1ccd fix(storage): stop deleting covers, place images and journey photos another row may still show
Replacing a trip or list cover, changing or deleting a place or saved-place
image and deleting a journey photo removed the bytes at once, without asking
whether a trip copy, a journey, a saved place or another photo row still
named them (#2924). Those paths now leave the object in place and the reclaim
job removes it once no row names it. A journey photo the user deletes takes
its local photo row with it when nothing else links it, so its bytes become
reclaimable instead of being kept by a row nobody sees.

A place image copied from a trip file is still dropped at once when the
update that would name it fails: that copy was never anyone else's.

Size and warning baselines follow the shrunk files.
2026-11-01 13:59:28 +01:00
Maurice 095bed2db8 feat(storage): let every owner of an upload column tell the reclaim pass what it still names
Trips, collections, places, the contribution queue, journeys, photos, trip
files and users each register a reference source from their own domain. A
source reads its own columns and parses every spelling they hold through the
storage helpers, and the free-form columns claim every reclaimable category,
since a cover or an image can point into any of them. The contribution queue
joins the ledger as its own source, because place-contrib owns that table.
2026-11-01 13:59:19 +01:00
Maurice fb2429ebb2 feat(storage): find and reclaim upload objects no row names any more
Domains whose rows name stored objects register a reference source with
StorageReclaimRegistryService; storage never imports them. A ledger in
storage decides per category whether it is reclaimed at all and which
sources must have answered first, and a category whose resolved location
overlaps another one is left alone.

A nightly pass at 04:05 lists the root level of files, journey, covers,
avatars and places, skips objects younger than a day or without a
modification time, asks every source in pages of 500 and marks the rest.
Only with deletion switched on, and only for marks a week old, it asks every
source about the single key again and deletes through StorageService. A
category is skipped and its marks start over when a source is missing or
fails, the listing fails, or more than a fifth of its objects look
unreferenced. A pass shares the one-job slot with backfills and migrations,
never runs on a demo instance and stops when the database is reopened.

Admins switch deletion and run a pass on demand under
/api/admin/storage/reclaim; both are audited, and GET /api/admin/storage
carries the switch and the last report. Deletion is off after install and
upgrade.
2026-11-01 13:37:53 +01:00
Maurice 29298dcfe5 feat(storage): keep a mark per stored object no row names
storage_reclaim_marks records since when the reclaim pass has seen an object
unreferenced, keyed by category and key. A mark is never moved forward, so
its age is how long the object has been unreferenced without a break; it goes
once the object is named again or gone. Owned by the storage domain.
2026-11-01 13:37:44 +01:00
Maurice 630f15b5ea feat(storage): describe the reclaim report, its switch and the run request in the shared contract
The admin state now carries the reclaim switch and the last pass's report.
A report covers every served category, the legacy photos folder included,
with the objects a pass looked at, the unreferenced, eligible and deleted
counts and bytes, and the reason a category was skipped.
2026-11-01 13:37:37 +01:00
Maurice 04debce79e fix(planner): keep the lint and size ratchets where they were
The zone helpers of the day order take the merged row's own type instead
of an explicit any, and the day list's size baseline follows it down.
2026-11-01 11:41:56 +01:00
Maurice 4003769a5c fix(planner): keep the date whole and rename the title where it stands with the date first
With "date first in day headings" a long day title squeezed the date in
the day list to "Th…" (#2920); the name now gives way and the date stays
whole. The day panel and the mobile day sheet kept the rename pencil on
the heading, which then is the date, and the input replaced it (#2921);
pencil and input now go with the title line under the date.
2026-11-01 11:41:56 +01:00
Maurice 7faf771f9c fix(docker): look at the volume roots only, and say when chown is refused
The ownership check still read every inode of the volumes on each start.
It now looks at the dirs and what sits directly in them, hands a volume
over once when it is not node's, and logs a warning instead of walking a
volume that refuses chown (NFS with root_squash).
2026-11-01 11:41:56 +01:00
Maurice 51b991f851 fix(docker): the container starts on large volumes and under runAsUser 1000
The entrypoint no longer runs chown -R over data and uploads on every start;
it only hands over files that are not node's yet, and logs when it does.
Started as a non-root user, it skips chown and gosu and runs Node directly.
The chart exposes podSecurityContext and securityContext for that setup.
2026-11-01 11:41:56 +01:00
Maurice 8692b8790c fix(planner): judge a move by the order a day is drawn in across time zones
The checks behind a drag and the up/down buttons read each booking's raw
clock, so on a day whose flights were put in the order they happen every
move of a timed stop asked to change its time (desktop) or was refused
(mobile). They now read the same key the day is ordered by.
2026-11-01 11:41:56 +01:00
Maurice 1b6a156969 fix(planner): order flights across time zones by when they actually leave
A booking's time is local to its own airport, so a flight leaving
Auckland at 20:20 and landing in Los Angeles at 12:25 the same date
was drawn below the 17:00 flight on from Los Angeles. Bookings whose
dated time has a known zone now take their clock slots in real order.
2026-11-01 11:41:55 +01:00
Maurice f076cb82da docs(readme): link dataforest cloud wherever the sponsor section names it 2026-11-01 10:20:21 +01:00
Maurice 073db83421 fix(assistant): answer the calls of a stopped step and ask for the words behind wordless follow-ups
A step stopped while its calls ran (the user left the page) kept the
provider's whole record of it but answers only for the calls that ran, so
Anthropic and the Responses API refused that thread with HTTP 400 from then
on. Replaying the thread now answers every recorded call that never ran,
which also mends threads already stuck that way.

Claude Sonnet 5.5 at times ends a turn on suggest_follow_ups alone, without
writing its answer, and the turn failed as empty_answer. That step now goes
back to the model whole, its follow-ups answered that the user saw nothing
yet, and their chips come under the answer it then writes. The prompts no
longer say the call goes alone in its step.
2026-11-01 10:18:27 +01:00
Maurice cafc8dd46e fix: link dataforest cloud with /en again
The 5.1.0 popup, the sponsor panel in Settings > About and the README had
switched to https://cloud.dataforest.net/?mtm_campaign=trek. The link stays
https://cloud.dataforest.net/en?mtm_campaign=trek everywhere.
2026-10-31 12:26:36 +01:00
github-actions[bot] dc36c21c08 chore: bump version to 5.1.0 [skip ci] v5.1.0 2026-10-31 11:12:52 +00:00
Maurice 175413b3c1 ci(release): take ci-ok from an earlier commit when main only changed workflows [skip ci]
A fix to a release workflow changes nothing that the tests cover, yet the
gate wanted a full new test run on it. With ci_from the release takes ci-ok
from the commit main's code comes from, provided that commit is an ancestor
of main, everything outside .github/ is identical, and one of its ci-ok runs
is green.
2026-10-31 12:10:35 +01:00
Maurice 030918cfa9 ci(release): check the dialect with node's types
TypeScript 6 includes no @types unless a file imports from them, and the
issued dialect imports nothing from node. Type-checked on its own it did
not know Buffer, so the 5.1.0 release threw away a valid dialect as one
that does not type-check. The check config names node now.
2026-10-31 12:09:32 +01:00
Maurice 75d65ccb1b test(release-notes): build the new cases without non-null assertions
The four cases added with the 5.1.0 popup patched the fixture through
n.release! and read the support card through card!, six warnings over the
tests baseline of the eslint ratchet. They go through a withRelease()
helper and optional chaining instead.
2026-10-31 11:31:13 +01:00
Maurice def8b8a24d release notes: the 5.1.0 popup, and dataforest cloud in Settings > About
The release popup now shows 5.1.0: place search, shared place details
and faster routing as three cards side by side, each with its own
drawing. The support band is set as a card like the promise above it
(optional supportLabelKey; a release without it keeps the plain line),
and the two support buttons get the same glass look in their colours.

Settings > About gets a full-width panel for dataforest cloud, who
sponsor the servers behind place search, address search and routing.
Both logos ship in client/public/brands, so opening Settings reaches no
third party. Wherever a line names dataforest cloud it links to
https://cloud.dataforest.net/?mtm_campaign=trek, the link dataforest
gave us; the README moves to it too.

All 28 locales carry the new copy, and the old 5.0 card keys are gone.
2026-10-31 11:04:19 +01:00
Maurice f4e524f3c3 place-enrichment: the details lookup is a standalone cache fill
The service reads a community place's details before the result cache and
every other place's after it, so the lookup has two call sites of which one
runs. It only fills the details row cache and turns a failure into no
details, so it is tagged @txStandalone and the enrich baseline drops to the
four writes it holds without it.
2026-10-31 06:45:13 +00:00
Maurice 682ae19143 test(place-enrichment): a cached OSM place still reads its details for the community version
An OSM element can hold community values, so the service reads its details
before the cache to learn the version it is served under. The second request
reads them once and then answers from the row cache without another lookup.
2026-10-31 06:27:31 +00:00
Maurice 255607b750 place-contrib: format the settle filters of the outbox repository
The id and cid lists the settle writes match on come from two small helpers,
which keeps each filter on one line in Prettier's layout.
2026-10-31 06:27:31 +00:00
Maurice ce7437a46f i18n: translate the place sharing, services card and refresh pill strings into Hebrew
The Hebrew locale arrived on dev while this branch was open, so its 80 new
admin.contribute, admin.servicesCard and places.refresh keys were missing
there.
2026-10-31 03:25:56 +00:00
Maurice 36d8cc2e7b wiki: the New place details pill offers the description too 2026-10-31 03:25:56 +00:00
Maurice dc76efc418 admin: a withdrawal that went through is reported even when the state re-read fails 2026-10-31 03:25:56 +00:00
Maurice 314b087f98 planner: the refresh pill compares with the place picked anew in the search box
After another place was picked while editing, the pill still offered the
saved place's details and would have written them onto the new one.
2026-10-31 03:25:56 +00:00
Maurice 2f689ea9ca maps: stop reading a community picture at its size cap
An answer without a content-length was buffered whole before its size was
checked. The read now stops at the cap.
2026-10-31 03:25:56 +00:00
Maurice 41edbcf67a maps: validate the community values and OSM elements the TREK Places API answers
Both answers were taken as typed. A value outside the published shape, such
as hours that are no text, reached the hours parser and failed the details
request. A place outside the shape is now left out of /v1/community, and an
element outside it sends the lookup to Overpass.
2026-10-31 03:25:56 +00:00
Maurice 6945b53c10 place-enrichment: a community summary the request carried stays out of the shared cache
Like an OpenStreetMap summary, a summary credited to the community is read
straight off the request's details, so any caller could have it served to
every user of the instance for a week under the community's name.
2026-10-31 03:25:56 +00:00
Maurice 7ade99c8ce place-contrib: audit a switch-off whose withdrawal the service did not confirm
Switching sharing off commits before the withdrawal asks the service. When
the service did not confirm, the decision had taken effect but left no audit
row. It is now audited with withdrawFailed, and the admin still gets the
error.
2026-10-31 03:25:56 +00:00
Maurice 593df53993 places: a picture route records the origin of the picture only
The upload and the trip-file routes vouched for every field that changed
between their read and their answer. A website the assistant or a
collaborator changed while a large picture uploaded was then recorded as
typed by the user and queued for the TREK Places API. Those writers now name
the one field they write, and other changes keep the origin their own writer
recorded.
2026-10-31 03:25:56 +00:00