Leaving every replaced cover, place image and deleted journey photo to the
nightly pass kept them publicly fetchable by URL for a week at least, and
for good while deletion was off. Owners now hand the values a delete or a
replace left behind to StorageReleaseService once their write committed.
It asks every registered reference source, with the same parser as the
nightly pass, and deletes at once what no row names any more: an object a
trip copy, a journey or a saved place still shows stays, which the old
inline deletes got wrong.
The release holds the pass's guards (reclaimable categories at their root
only, never an owner marker, never while a restore runs, never with a
required source missing or an overlapping location), never fails the
request and logs at debug. It runs for trip covers (update, upload, trip
delete), collection covers and saved-place images, place images (update,
bulk update, delete), journey covers and journey or gallery photos
(their thumbnail goes with a file that went), avatars (change, delete,
account deletion), trip files (permanent delete, empty trash) and collab
note and chat attachments. A trip file row now goes before its bytes, so a
failing backend no longer fails the delete. The objects of a deleted trip's
places and files are left to the nightly pass: their rows are other
domains'.
PlacesService drops the collection places repository nothing read any
more. Size, warning and boundary baselines follow the shrunk files.
The 20 % brake skipped a category and wiped its marks every night once a
fifth of it looked unreferenced. Any user could push a small install over
that line, a backlog from #2924 started braked, and a second install on the
same bucket below the line was not caught at all. It is gone:
- Each pass leaves a trek-owner-<install id> marker at the root of every
category it judges. A category holding another install's marker is
skipped as foreign_install, one where the marker cannot be written as
owner_unknown. Backups leave the markers out.
- The install id lives in app_settings outside the storage.* prefix. The
switch is stored as that id, so a database from another install reads as
off, and every reopen (a restore) writes this install's id back, turns
deletion off and drops the marks, confirmations and reports.
- More than a fifth unreferenced still marks and reports, but deletes only
after an admin confirmed the count shown (POST .../reclaim/confirm). A
larger count later waits again.
A skip no longer touches the marks, so a failed listing or lookup does not
restart the week. Check now runs in the background and answers 202; the
admin state says running and keeps the last deleting report apart, so a
report-only check never resets what was freed. A pass that deletes writes a
system audit entry, a pass that stops or fails keeps its partial counts and
marks the rest interrupted, a backend error stops deleting in its category,
and deleting stops for the rest of the pass once the switch is off or a
backup is reading the uploads. The usage scan waits for a pass still
running.
Sources read each category once per pass and answer the pre-delete
re-check with a query narrowed to the keys' plain runs. Bare names keep
their raw spelling (a '#', '?' or trailing blank is part of an upload's
name), and local roots are resolved through their symlinks before the
overlap check. A pass lists only a category's root.
The two admin routes go on the managed allow list, without which buildApp
refused to start. The reclaim calls move to their own api module so
client.ts stays at its size, the phone card sizes its text through the
type tiers, and the tests drop a hand-built service, a non-null assertion
and a promise read as a boolean.
The last question to the sources now comes after the object's own check,
so nothing slower than the delete sits between the answer and the delete,
and it is asked once per batch instead of once per object: a pass read
every source's columns for each object it was about to remove. The
collection reads the removed inline reclaims used are gone with them.
A restore wipes and rehydrates the uploads under a swapped database, but it
never took the storage job slot, so a reclaim pass could start or carry on
beside it, and a delete already past the generation check could land on an
object the restore had just put back. BackupService now marks a restore as
running on StorageJobsService and waits for a pass in flight to let go; a
pass refuses to start while a restore runs and stops at its next check when
one begins. Backfills and migrations are not affected. The service test also
reads its fake source sets through a helper instead of non-null assertions.
The reclaim e2e and the source-registration check mocked src/db/database,
which lint:test-mocks holds at zero for new files; both now boot buildApp
against the worker's snapshot db. The key parser drops its non-null
assertions (the indexed reads are typed string already), and the e2e sends
its refused bodies at once instead of awaiting inside a loop, so the eslint
ratchet stays where it was.
Replacing a trip or list cover, changing or deleting a place or saved-place
image and deleting a journey photo removed the bytes at once, without asking
whether a trip copy, a journey, a saved place or another photo row still
named them (#2924). Those paths now leave the object in place and the reclaim
job removes it once no row names it. A journey photo the user deletes takes
its local photo row with it when nothing else links it, so its bytes become
reclaimable instead of being kept by a row nobody sees.
A place image copied from a trip file is still dropped at once when the
update that would name it fails: that copy was never anyone else's.
Size and warning baselines follow the shrunk files.
Trips, collections, places, the contribution queue, journeys, photos, trip
files and users each register a reference source from their own domain. A
source reads its own columns and parses every spelling they hold through the
storage helpers, and the free-form columns claim every reclaimable category,
since a cover or an image can point into any of them. The contribution queue
joins the ledger as its own source, because place-contrib owns that table.
Domains whose rows name stored objects register a reference source with
StorageReclaimRegistryService; storage never imports them. A ledger in
storage decides per category whether it is reclaimed at all and which
sources must have answered first, and a category whose resolved location
overlaps another one is left alone.
A nightly pass at 04:05 lists the root level of files, journey, covers,
avatars and places, skips objects younger than a day or without a
modification time, asks every source in pages of 500 and marks the rest.
Only with deletion switched on, and only for marks a week old, it asks every
source about the single key again and deletes through StorageService. A
category is skipped and its marks start over when a source is missing or
fails, the listing fails, or more than a fifth of its objects look
unreferenced. A pass shares the one-job slot with backfills and migrations,
never runs on a demo instance and stops when the database is reopened.
Admins switch deletion and run a pass on demand under
/api/admin/storage/reclaim; both are audited, and GET /api/admin/storage
carries the switch and the last report. Deletion is off after install and
upgrade.
storage_reclaim_marks records since when the reclaim pass has seen an object
unreferenced, keyed by category and key. A mark is never moved forward, so
its age is how long the object has been unreferenced without a break; it goes
once the object is named again or gone. Owned by the storage domain.
The ownership check still read every inode of the volumes on each start.
It now looks at the dirs and what sits directly in them, hands a volume
over once when it is not node's, and logs a warning instead of walking a
volume that refuses chown (NFS with root_squash).
The entrypoint no longer runs chown -R over data and uploads on every start;
it only hands over files that are not node's yet, and logs when it does.
Started as a non-root user, it skips chown and gosu and runs Node directly.
The chart exposes podSecurityContext and securityContext for that setup.
A step stopped while its calls ran (the user left the page) kept the
provider's whole record of it but answers only for the calls that ran, so
Anthropic and the Responses API refused that thread with HTTP 400 from then
on. Replaying the thread now answers every recorded call that never ran,
which also mends threads already stuck that way.
Claude Sonnet 5.5 at times ends a turn on suggest_follow_ups alone, without
writing its answer, and the turn failed as empty_answer. That step now goes
back to the model whole, its follow-ups answered that the user saw nothing
yet, and their chips come under the answer it then writes. The prompts no
longer say the call goes alone in its step.
The release popup now shows 5.1.0: place search, shared place details
and faster routing as three cards side by side, each with its own
drawing. The support band is set as a card like the promise above it
(optional supportLabelKey; a release without it keeps the plain line),
and the two support buttons get the same glass look in their colours.
Settings > About gets a full-width panel for dataforest cloud, who
sponsor the servers behind place search, address search and routing.
Both logos ship in client/public/brands, so opening Settings reaches no
third party. Wherever a line names dataforest cloud it links to
https://cloud.dataforest.net/?mtm_campaign=trek, the link dataforest
gave us; the README moves to it too.
All 28 locales carry the new copy, and the old 5.0 card keys are gone.
The service reads a community place's details before the result cache and
every other place's after it, so the lookup has two call sites of which one
runs. It only fills the details row cache and turns a failure into no
details, so it is tagged @txStandalone and the enrich baseline drops to the
four writes it holds without it.
An OSM element can hold community values, so the service reads its details
before the cache to learn the version it is served under. The second request
reads them once and then answers from the row cache without another lookup.
Both answers were taken as typed. A value outside the published shape, such
as hours that are no text, reached the hours parser and failed the details
request. A place outside the shape is now left out of /v1/community, and an
element outside it sends the lookup to Overpass.
Like an OpenStreetMap summary, a summary credited to the community is read
straight off the request's details, so any caller could have it served to
every user of the instance for a week under the community's name.
Switching sharing off commits before the withdrawal asks the service. When
the service did not confirm, the decision had taken effect but left no audit
row. It is now audited with withdrawFailed, and the admin still gets the
error.
The upload and the trip-file routes vouched for every field that changed
between their read and their answer. A website the assistant or a
collaborator changed while a large picture uploaded was then recorded as
typed by the user and queued for the TREK Places API. Those writers now name
the one field they write, and other changes keep the origin their own writer
recorded.
A value saved while the request for its predecessor was out rewrote the same
outbox row with a new cid; the answer for the old value then deleted the row,
or charged it an attempt, and the newer value was never sent. The settle
writes now match the cid that was sent as well as the id, so such a row stays
queued for the next run. A replaced value also counts its age from the
replacement, so a long pause cannot expire a value queued days ago.
Two gaps kept it out. The service writes absolute picture URLs on its own
host, and the community service only took relative paths, so every picture
was dropped; it now takes the path of a URL on the configured service and
nothing from another host. And a saved place opened for editing sends no
details, so no community version, and the cached answer kept for a week; for
an index or OSM place the details are now read before the cache, so a new
version is a new answer.
The service delivers the pictures review approved for a place, but TREK never
read them. A new TrekCommunityService in the maps domain asks the TREK Places
API for them (the index row for a gers place, /v1/community for an OSM one),
accepts only paths of the service's picture store, and fetches the bytes with
the installation's key, capped at 4 MB. Enrichment puts them ahead of Google
and Commons, through the same photo cache, credited as TREK Community under
CC BY 4.0. The enrichment cache now follows the community version for OSM
places too, so a newly approved picture shows on the next open.
Also lowers the client's no-non-null-assertion baseline to 380: moving Use
this text into the description card removed two.
A community description laid over OSM details or an index row showed up as
OpenStreetMap under ODbL, or as the place's own website. It now has a source of
its own, community, credited as TREK Community under CC BY 4.0, like the
service publishes it.
The fresh lookup behind New place details goes through placeDetailsRepo
(online-only, like the service identity) instead of api/ directly. The form
loses the Translate type and the comment that moved with the head band's
pills, which takes it under its line limit and out of the size baseline; the
maps service suite's test-size entry follows its shorter count.
Community opening hours laid over OSM details now go through the same parser
as OSM's own, so they arrive as one line per day with open-now and the periods,
and a community description lands in the details' summary. A place copied from
a collection no longer records its values as an import: without an origin row a
value is never sent anyway, and a later edit marks only the fields it changes.
Opening an OSM place asked the public Overpass servers for its tags, which
took five seconds and more and answered nothing while they were down. Where
the TREK services are in use, the details now come from the Places API's
details layer (/v1/osm), with the community values in the same answer, and
Overpass is asked only for an element the layer does not hold.
A place found through Photon or Overpass, a relation or a way, is no row of the
TREK index, so the community values the index lays over its own rows never
reached it. Opening its details now asks /v1/community for the element and
lays what is published over the record, with the version, where the TREK index
is in use. No answer leaves the details as they were.
A value is due the moment it is queued, and the save starts a send run right
away; the two-minute tick stays for the retries. One run goes at a time, and a
run asked for meanwhile follows right after it. Only where the schedules run, so
tests and replicas send nothing.
A place copied from a collection and a venue a booking import creates now
record their details as an import, so a later edit compares against what was
really written. Dawarich and the background enrichment write none of the
shareable fields, so they need nothing.
The admin panel asks again for a few seconds while a fresh installation is
still fetching its key, so the prompt appears without a reload.
- The plugin RPC fallback tests build PlacesRpc with its origin bookkeeping.
- The e2e guard and contract cases send their requests one after another; the
test server reset connections opened all at once.
- The job's two cron registrations are tagged @txIndependent: neither writes.
- The panel draws its fold head itself instead of importing it from pages/.
- The pause setting's type comes from its schema, so nullable fields read the
same under the server's compiler settings.
- The place form passes the origin of a field through to the hook.
- Serial steps run through inOrder (src/nest/common/in-order.ts) and
independent ones through Promise.all, instead of an await inside a loop. The
bulk update in places.mcp.ts goes the same way, so its baseline drops by one.
- The status a call without a key gets is a named constant, not an envelope
literal.
- A picture test no longer inherits the previous test's mock answer.
- The entity counts include the three new tables.
- The log keeps its failure per query instead of resetting it in the effect,
and usePlaceForm sets its picture through one helper.
Every shareable field of a place now records where its value came from: typed
by a user, filled by a provider, written by the assistant, a plugin or an
import. Only values a user typed are ever sent, and only after an admin agreed
to the current terms for the whole installation, field by field.
The new place-contrib domain keeps the decision in app settings, queues values
in an outbox a cron job sends from, logs what the service made of each one and
can withdraw everything this installation sent. Development, test and demo
instances, an own TREK_PLACES_URL, TREK_SERVICES=off and
TREK_PLACES_CONTRIBUTE=off never send anything. Index results carry the
community version, so the details caches refresh when it changes.
The Atlas server no longer keeps the admin-1 regions of every country in memory. Each country's regions are now a pre-gzipped file of their own, sent to the map as they are.
**Why:** `GET /addons/atlas/regions/geo?countries=…` built `admin1Store` (every country's region text, kept until restart), then joined the requested countries, `JSON.parse`d them, and Nest stringified them again. On the 512 MB – 1 GB boxes TREK runs on, the first z≥5 view could OOM the server (same shape as #1576).
**What changes:**
- `server/assets/atlas/admin1/<ISO_A2>.geojson.gz`: 198 files, 14.85 MB, the same 3,447 features as `admin1.geojson.gz`.
- New `GET /addons/atlas/regions/geo/:code` with a Zod contract (`regionGeoParamsSchema` in `@trek/shared`: two letters, any case). It sends the country's file as it is, with `Content-Encoding: gzip` and the current 1-day `Cache-Control`, as `countries/geo` does. A country without regions gets an empty FeatureCollection. Anything that is not two letters gets a 400.
- `useAtlas.ts` fetches one URL per uncached country and still rebuilds the layer once. Each country has a stable URL, so a country evicted by `REGION_CACHE_MAX` comes back from the browser cache.
- `getRegionFeatures` reads only the country it needs. `admin1Store` is removed.
- `?countries=` keeps working for cached PWA clients. It joins the per-country files as bytes, without a parse.
- The migrations are untouched. `admin1.geojson.gz` still ships, only for `Migration20200101021500` and `Migration20200101031500`. This costs ~15 MB of disk in the image and no RAM. `build-atlas-geo.mjs` writes both, and ATLAS-BUNDLE-008 keeps them equal.
**Memory** (real `atlas-geo` module, first z≥5 view over Europe = 49 country codes, JS heap + Buffers):
| | peak | kept after |
|---|---|---|
| before, first view (builds the store) | +372 MB | +61 MB |
| before, later views | +115 MB | 0 |
| after, `/regions/geo/:code` | +4.7 MB | 0 |
| after, legacy `?countries=` | +59 MB | 0 |
**Tests:**
- New: `atlas-region-files.test.ts` (moved from `atlas.service.test.ts`, plus join and dedupe cases), the `regionGeoParamsSchema` spec, ATLAS-BUNDLE-007 (the file format the byte join depends on), ATLAS-BUNDLE-008 (bundle and per-country files are equal), ATLAS-016/017/018 (the new route over HTTP, including the 400).
- The `getVisitedRegions` tests (ATLAS-UNIT-019..024) used to pass only when `getRegionGeo`'s `beforeAll` had built the store first. They now also pass on their own.
- The shared, server and client gates pass. Baselines go down only where this change removed code (`no-console`, `no-explicit-any`, `no-swallowed-catch`, file sizes).
**Trade-offs:**
- A first zoom over Europe sends ~49 small parallel requests instead of 1. Over HTTP/1.1 the browser queues them 6 at a time. The total bytes are the same (4.4 MB gz).
- The per-country URLs are unversioned and cached for a day, like `countries/geo`. After an upgrade that changes a country's regions, a browser can show the old shapes for up to 24 h.
Approved on Discord in #github-pr: <paste the thread link here>
Related: #1576, #2559
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] I have read the [Contributing Guidelines](https://github.com/liketrek/TREK/wiki/Contributing)
- [ ] My branch is [up to date with `dev`](https://github.com/liketrek/TREK/wiki/Development-environment#3-keep-your-fork-up-to-date)
- [x] This PR targets the `dev` branch, not `main` *(wiki-only PRs are exempt)*
- [x] I have tested my changes locally
- [x] I have added/updated tests that prove my fix is effective or that my feature works
- [x] I have updated documentation if needed
transit is not one of the fixed Costs categories, so typeToCostCategory
would put an uncategorised fare under other. Default to transport, the
category every rail and bus booking maps to.
- create_transit_journey takes price / budget_category, as create_transport
does: a fare > 0 files the linked expense in the booking's own transaction
and rides along in metadata.price. The tool now writes through
ReservationWritesService.createBooking, so its events and booking notice are
the ones every other booking write sends
- create_budget_item and create_budget_item_with_members take reservation_id,
the field POST /budget already accepts; create_budget_item now runs
BudgetWritesService.createItem like its sibling, so a linked booking's
mirrored price is worked out again before budget:created goes out
- bookingPriceSchema / bookingBudgetCategorySchema in @trek/shared back the
price fields of all three booking tools (mcp-inline-zod baseline lowered)
- wiki: MCP-Tools-and-Resources describes the new params
The empty-result fallback can call Google after Transitous, so lint:tx now sees two googleQuota.record() calls in TransitService.plan. Only one of them runs per request, and each is a call counter written after its own upstream request, so it cannot share a transaction with it.
Admin settings, transport, environment and MCP pages still said Transitous
is the only fallback and that Google is billed only when picked. Document
that an empty plan asks the other backend, that the Google fallback uses
only an instance key within the daily limit, and mention it in the
search_transit_routes tool description.
Maurice