Every shareable field of a place now records where its value came from: typed
by a user, filled by a provider, written by the assistant, a plugin or an
import. Only values a user typed are ever sent, and only after an admin agreed
to the current terms for the whole installation, field by field.
The new place-contrib domain keeps the decision in app settings, queues values
in an outbox a cron job sends from, logs what the service made of each one and
can withdraw everything this installation sent. Development, test and demo
instances, an own TREK_PLACES_URL, TREK_SERVICES=off and
TREK_PLACES_CONTRIBUTE=off never send anything. Index results carry the
community version, so the details caches refresh when it changes.
The Atlas server no longer keeps the admin-1 regions of every country in memory. Each country's regions are now a pre-gzipped file of their own, sent to the map as they are.
**Why:** `GET /addons/atlas/regions/geo?countries=…` built `admin1Store` (every country's region text, kept until restart), then joined the requested countries, `JSON.parse`d them, and Nest stringified them again. On the 512 MB – 1 GB boxes TREK runs on, the first z≥5 view could OOM the server (same shape as #1576).
**What changes:**
- `server/assets/atlas/admin1/<ISO_A2>.geojson.gz`: 198 files, 14.85 MB, the same 3,447 features as `admin1.geojson.gz`.
- New `GET /addons/atlas/regions/geo/:code` with a Zod contract (`regionGeoParamsSchema` in `@trek/shared`: two letters, any case). It sends the country's file as it is, with `Content-Encoding: gzip` and the current 1-day `Cache-Control`, as `countries/geo` does. A country without regions gets an empty FeatureCollection. Anything that is not two letters gets a 400.
- `useAtlas.ts` fetches one URL per uncached country and still rebuilds the layer once. Each country has a stable URL, so a country evicted by `REGION_CACHE_MAX` comes back from the browser cache.
- `getRegionFeatures` reads only the country it needs. `admin1Store` is removed.
- `?countries=` keeps working for cached PWA clients. It joins the per-country files as bytes, without a parse.
- The migrations are untouched. `admin1.geojson.gz` still ships, only for `Migration20200101021500` and `Migration20200101031500`. This costs ~15 MB of disk in the image and no RAM. `build-atlas-geo.mjs` writes both, and ATLAS-BUNDLE-008 keeps them equal.
**Memory** (real `atlas-geo` module, first z≥5 view over Europe = 49 country codes, JS heap + Buffers):
| | peak | kept after |
|---|---|---|
| before, first view (builds the store) | +372 MB | +61 MB |
| before, later views | +115 MB | 0 |
| after, `/regions/geo/:code` | +4.7 MB | 0 |
| after, legacy `?countries=` | +59 MB | 0 |
**Tests:**
- New: `atlas-region-files.test.ts` (moved from `atlas.service.test.ts`, plus join and dedupe cases), the `regionGeoParamsSchema` spec, ATLAS-BUNDLE-007 (the file format the byte join depends on), ATLAS-BUNDLE-008 (bundle and per-country files are equal), ATLAS-016/017/018 (the new route over HTTP, including the 400).
- The `getVisitedRegions` tests (ATLAS-UNIT-019..024) used to pass only when `getRegionGeo`'s `beforeAll` had built the store first. They now also pass on their own.
- The shared, server and client gates pass. Baselines go down only where this change removed code (`no-console`, `no-explicit-any`, `no-swallowed-catch`, file sizes).
**Trade-offs:**
- A first zoom over Europe sends ~49 small parallel requests instead of 1. Over HTTP/1.1 the browser queues them 6 at a time. The total bytes are the same (4.4 MB gz).
- The per-country URLs are unversioned and cached for a day, like `countries/geo`. After an upgrade that changes a country's regions, a browser can show the old shapes for up to 24 h.
Approved on Discord in #github-pr: <paste the thread link here>
Related: #1576, #2559
- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] I have read the [Contributing Guidelines](https://github.com/liketrek/TREK/wiki/Contributing)
- [ ] My branch is [up to date with `dev`](https://github.com/liketrek/TREK/wiki/Development-environment#3-keep-your-fork-up-to-date)
- [x] This PR targets the `dev` branch, not `main` *(wiki-only PRs are exempt)*
- [x] I have tested my changes locally
- [x] I have added/updated tests that prove my fix is effective or that my feature works
- [x] I have updated documentation if needed
transit is not one of the fixed Costs categories, so typeToCostCategory
would put an uncategorised fare under other. Default to transport, the
category every rail and bus booking maps to.
- create_transit_journey takes price / budget_category, as create_transport
does: a fare > 0 files the linked expense in the booking's own transaction
and rides along in metadata.price. The tool now writes through
ReservationWritesService.createBooking, so its events and booking notice are
the ones every other booking write sends
- create_budget_item and create_budget_item_with_members take reservation_id,
the field POST /budget already accepts; create_budget_item now runs
BudgetWritesService.createItem like its sibling, so a linked booking's
mirrored price is worked out again before budget:created goes out
- bookingPriceSchema / bookingBudgetCategorySchema in @trek/shared back the
price fields of all three booking tools (mcp-inline-zod baseline lowered)
- wiki: MCP-Tools-and-Resources describes the new params
Logroño-Agoncillo (RJL) and fifteen other airports gained their IATA code in
OurAirports after the bundled list was built, so the airport search and the
code lookup of create_transport could not find them. The list is rebuilt from
the current data. The build now keeps every code the previous list carried
that upstream no longer lists (a scheduled-service flag flipped back, PBI moved
to a new record), since bookings and MCP calls still name them.
Refs #2897
A car or other transport booking carried its route only in the description, so
a calendar app had no place to navigate to. Without a location of its own, a
transport event now takes the endpoint the traveller has to get to: the
destination of a drive of their own (car, bicycle) and the departure of a ride
they board, the side the per-leg events already use. A car's pickup and
drop-off events are located at their own endpoints. A location the booking
states keeps winning.
Closes#2909
Claude Sonnet 5.5, Opus 5.5 and Fable 5.1 reject tool_choice type "tool" and
"any" with a 400, so every booking import through the Anthropic provider failed
on them. The request now sends tool_choice auto with parallel tool use off,
which every Anthropic model and Messages-compatible gateway accepts, and the
system prompt tells the model to answer through the emit tool exactly once. A
model that still answers in text is reported as before.
Closes#2902
maps.helpers.ts and osm.client.ts had grown past the 1000-line limit. The
merge behind the search and suggestion lists, with its lead rules, moves to
maps-merge.helpers.ts; the search records and Photon's half of the
OpenStreetMap geocoding (search, suggestions, airport codes, identity,
reverse, element details) move to osm-geocoding.ts. OsmClient keeps the
same methods and hands them on, so no caller changes.
- The lead is Photon's own row first; a row of its name inside the lead
only stands in where Photon's row went as the same place. Rows without a
line below the name are never folded as reading the same.
- A miss of Photon where it is the geocoder no longer asks it a second time
on the way to an empty answer.
- The transport words of the identity match use a Unicode word boundary
(straße, estación) and match 駅 anywhere.
- Opening an OSM place reads the coordinate it was placed at explicitly.
- The ferry spans of a Valhalla leg move to valhallaFerries.ts, and a day
cut into pieces is asked with one options object.
- "Limit" on the allowance tiles is "Daily limit", which no locale left
as an English copy.
- Tests follow the new behaviour: no Nominatim behind Photon, zoom 13,
and the expectations of the merge and edit-distance cases.
A request to the TREK Valhalla that runs out its deadline (a long bike
route) is answered 504 and no longer rests the relay for every user or a
tab for ten minutes; that one day goes to OSRM. Only an unreachable host
or a server error rests it.
A day goes to the relay with up to twenty stops per request and no pause
between pieces; a request that ends up at the public FOSSGIS Valhalla is
cut to ten and paced, as before. The 1.1 s spacing of the server applies
to the public hosts only, not to an admin's own OSRM or Valhalla. The
relay keeps its connections open and asks for maneuvers only, which the
relay contract now accepts.
The suggestions and the full search now share one merge. Photon's hits are
ranked with places that have a Wikidata link before their unlinked
namesakes, and its context bias is narrower (zoom 13). The list opens on a
lead where Photon's answer has one: a country or region always; a town,
island or sight when it is called what was typed (or a near typo of it),
lies within 1000 km of where the traveller is looking and no place of that
name ranks above it. Rows of the lead's name inside it, the same element
under two ids and rows that read the same are shown once. A three-letter
code in capitals is asked for as the airport of the bundled list.
Where Photon is the geocoder it answers alone: an empty answer and a miss
are the answer for search and imports, the reverse lookup asks again
within 50 km, a Google place is matched inside a box through Photon, and
an opened OpenStreetMap place is placed by Overpass and named by Photon.
Nominatim stays the search of installations without the TREK services or
with an operator's own Nominatim.
Measured on 5000 places from Wikidata: the right place first while typing
48 % before, 65 % now; full search 53 % to 68 %; no repeated rows.
A rail, bus or ferry ticket with changes arrived as one booking per
segment, each with a linked cost for the full fare, so the trip total
was the fare times the number of segments.
Import: the KItinerary mapper now groups consecutive train segments
under one booking reference that connect (same station by name or
within 500 m, no return, within a day) into one train booking with
stops and metadata.legs, the way connecting flights already were. The
connection rule is shared between flights and ground transport, and a
stop carries the onward departure for flights as for trains, as the
planner form and the MCP tools write it. The three copied ground
mappers are folded into one.
Within one such booking a fare repeated on every segment is booked
once, and fares that differ per segment are added in whole cents. That
summing now applies to connecting flights too, which used to keep the
first leg's price only.
Segments of one order that do not join into one booking (an outward
and a return trip, a gap, a change of station named two ways without
coordinates, bus and boat tickets, Rail&Fly) are compared across the
document: when every priced transport booking of a reference carries
the same amount in the same currency, the first to depart keeps it and
the others lose it. Amounts that differ stay. This runs on what
KItinerary extracted only.
AI import: the repeated-total safety net from #2651 now covers train,
bus and ferry tickets and any mix of them with flights, still only
when the document prints that amount on a total line.
MCP and assistant: create_transport says that a ticket with several
segments is one booking with endpoints[] and legs[], and the price
fields say they hold the whole booking's cost. When a new booking
repeats the price an earlier booking with the same confirmation number
already carries in the trip's currency (its own currency, or the
trip's when it names none), create_transport and create_reservation
still save it and return a warning naming the earlier booking. REST is
unchanged.
Existing duplicated costs are left alone: they cannot be told apart
from two tickets that really cost the same, and deleting expenses
automatically would change settled balances. They can be removed in
the Costs tab.
Closes#2895
A stable release without a dialect of its own now ships the one on main only while that one has 90 days left; dialect 0 or less stops the run before anything is committed, unless allow_stale_dialect is ticked. Fetching and checking moved into registration-dialect.yml, which never fails its caller and also type-checks the module under the strict config and lints it with --max-warnings 0, since the bump commit skips CI. The bump and everything after it run whatever became of the dialect job. Prereleases ask the issuer for a dialect of their -pre.N version and build it into the image without committing it. The release check reads the module token by token against an allowlist of keywords, globals and member names, refuses any assignment to a property, computed access and regular expressions, and names Object, JSON and defineProperty among the forbidden. The issuer's kitchen-sink sample is gone; the tests use a hand-written dialect instead.
Registration now goes through a registration dialect module: the paths of
the challenge and the registration, the bytes scrypt runs over and the body
and headers of the request. Each stable release will carry the dialect the
TREK services issuer writes for it; until the first such release the module
speaks the original registration, so nothing changes on the wire.
An issuer that no longer takes the release's dialect answers 410. TREK then
says so once per process, waits a day (or what Retry-After asks) and keeps
searching through the shared pool. The issuer client only lets a dialect
register at the issuer's own paths, with X-TREK-* headers that are not the
gate's and a body of registration size.
Renewal needs no dialect, and an expired key may now be renewed for a year
instead of 30 days, the same period the issuer allows, so an installation
that was offline for long comes back without an update. The key format
ignores the new optional dialect claim; the shared vectors gain one key that
carries it.
The share link has counted its map stops through the whole journey since
#1962, while the owner's journal still restarted the count every day, so
the same stop read 66 on the link and 6 in the journal.
Both views now take their stops from one helper, journeyMapStops: located,
non-skeleton entries in the server's reading order (date, sort_order, id),
numbered across the journey. A draft keeps its pin in the journal but gets
no number, so the published stops after it carry exactly the numbers the
share link shows. The public entry query gains the id tiebreak the owner
query already had, so entries sharing a sort_order come back in the same
order on both sides.
Closes#2886
The editor follow-up always hands the planner an afterSave, and its type now
says so, so the hook test calls it under strictNullChecks. The travel stats
test reads a country without a status as visited, as the standing helper
does for the hand-marked list. The client baseline holds the one non-null
assertion the count missed.
The booking save helper answers a Reservation the way the dialogs' onSave
does, and takes the payload the sheets read back. The export tests resolve
their routing promise without non-null assertions, the autocomplete test
reads its own console spy, and the Photon tally test asks its five dropped
questions together instead of awaiting in a loop. The Photon address line
falls back to empty when a feature has none.
The repair test now expects the dashboard to place the cleared beach in
Germany through the borders, as the Atlas map does since #2875. The client
lint baseline follows the counts this branch lowered.
The resolver's comment claimed the places of a city trip share their
coordinates, which they do not. It now says what an uncached place costs
per count and why the dashboard does not queue the Atlas region lookup:
that lookup falls back to Nominatim, and a dashboard load must not send a
user's places to a third party.
The #2875 cases continue the file's AUTH-DB numbering.
The suggestions waited for Photon beside the index for up to its full
2.5 s deadline on every keystroke, and their misses and spent allowance
paused Photon for the search and the reverse lookup as well.
Once the index has answered, Photon now gets 300 ms more and is then
called off; with nothing from the index it is still waited for. The
keystroke question has a shorter deadline and its own tally of misses and
pauses, a question called off by the list counts as no miss, and a pause
of the search still holds the suggestions back.
Refs #2877
The passport card and the last/next trip countries read place countries
from the place_regions cache alone, while the Atlas map falls back to the
bundled borders for a place without a cached region. A place in Vatican
City never gets a region row (the region bundle has no VA features and
Nominatim names no region there), so the map counted VA and the card did
not. The same gap hid any country whose place was still waiting for its
background lookup, or whose lookup had failed.
Both views now resolve each place through one helper: the cached country
first, the borders otherwise, once per location. The dashboard reads every
place of a started trip with its cached country instead of the distinct
cached codes, and the trip countries count resolved places the same way.
The MCP travel stats tool and the public stats API use the same service
methods and are fixed with it.
Closes#2875
The suggestion list asked only the TREK index and its OpenStreetMap layer,
because Nominatim's usage policy rules out autocomplete. The layer is a slice
of OpenStreetMap, so many places showed up only once the full search was sent
(#2877). Photon is built for this keystroke: the suggestions now ask it in
parallel with the index and interleave its rows, dropping any the list
already holds, by OSM id or by the search list's same-place rule.
Without Photon (services off, an operator's own Nominatim, a paused or failed
answer) the list reads exactly as before. A failing index no longer throws
away what Photon found, and the Google-only switch skips Photon as it skips
the index.
Refs #2877
MCP-SCHEMA-005 finds tuple keywords, it does not validate against the 2020-12 metaschema, so the name now says that. A restored map point lost its range and order to the tuple fix; the description keeps the order visible to a model that builds a page itself.
With the Journey addon on, ChatGPT refused every TREK tool: restore_book_page
took a map's path and roads as [lat, lng] tuples, which the SDK writes as the
draft-07 array form of items, and JSON Schema 2020-12 refuses that form.
The restore now takes a map point as two numbers and parses each page as a
spread before it is put back, so lat and lng stay checked and a point out of
range is an invalid_op instead of a thrown parse. MCP-SCHEMA-005 holds every
tool's input schema free of tuple keywords.
Closes#2879
A road trip day OSRM drew while the relay could not answer was cached for five
minutes, and a key added or removed kept serving the old engine's answer. The
stand-in is no longer cached and the key's availability is part of the cache
key. A walk kept off the boats may still go to FOSSGIS, which is now said and
tested.
The per-user 429 carried no Retry-After, so the planner left the relay alone for
its full ten minutes and routed every day with OSRM meanwhile. It now answers
like the installation limit does, with the minute the window takes.
Where the TREK services or an own Valhalla route the days, the MCP road trip
router asks Valhalla for a day that avoids nothing, through the relay alone,
and OSRM routes the whole day when it cannot answer. Requests carry twenty
locations unless they may end up at FOSSGIS, which keeps ten.
An empty Photon search, or one without a hit that has an OpenStreetMap id, is asked again at Nominatim as the reverse lookup already was. The background lane goes to Nominatim alone, a reverse lookup leaves Nominatim half of the caller's deadline, and a 429 with Retry-After pauses Photon for that long.
A hung TREK Valhalla cost every leg, height chunk and trace the full 30 s before FOSSGIS answered. The relay now waits 10 s for a route or heights, rests a minute after an error or timeout and answers 503 at once meanwhile, follows a 429's Retry-After (an hour at most), passes that header on to the browser, and holds the installation to 100 questions a minute below the gate's 120 (60 per user).
The status asked a fixed Photon origin and compared NOMINATIM_URL with the public address, so a moved Photon or an explicitly set public Nominatim showed the wrong service as in use. It now reads the same photonUrl and nominatimOwn the Photon client decides by. TREK_VALHALLA_URL joins the commented compose variables.
GET /api/admin/service-identity lists places, photon and valhalla, each with
today's allowance read fresh from /v1/quota of its own host (cached for 20
seconds per host) and whether this instance asks it right now: Photon only
beside the public Nominatim, the TREK Valhalla only where the routing choice
comes to it. Photon and Valhalla are asked only with a key and only while
TREK_SERVICES_HOSTS lists them. The last allowance an answer stated is kept
per host for when a host does not answer.
The card lists the three services under each other in the look it had,
a service this instance does not ask dimmed and named as such.
routing_valhalla_provider (fossgis, trek or custom) sits beside the routing
URLs under Admin, Default user settings, admin only and locked on a managed
install. Without a choice an own Valhalla is custom, an own OSRM alone asks
no Valhalla, and otherwise the TREK services answer.
The planner asks them through the same-origin relay, so no connect-src
changes. A relay that answers 401 (a shared page without a sign-in), 403,
413, 429 or 5xx sends the question to FOSSGIS without a word and is left
alone for ten minutes. A long line to describe goes as a polyline.
POST /api/routing/valhalla/route, /trace_attributes and /height ask the
Valhalla at TREK_VALHALLA_URL (default https://valhalla.liketrek.com) for a
signed-in user. The body is held to the shared contract (64 KB, at most 20
locations), every question counts against 120 a minute per user, and the
installation key goes along only to a host TREK_SERVICES_HOSTS lists.
Valhalla's answer is passed on as it came, refusals included.
The road trip router asks the same relay when the routing choice lands on
the TREK services, so the MCP road trip, city walk and hike tools share its
limits, and hands a question it cannot answer to FOSSGIS.
An installation that holds a key for the TREK services and has no
NOMINATIM_URL of its own now sends the OpenStreetMap search and the
reverse lookups to photon.liketrek.com first. The answers are read into
Nominatim's shape, so the place records stay the same, and every miss
(refused key, allowance spent, timeout, error, foreign body) falls back
to Nominatim, with a one minute pause after four misses in a row.
TREK_PHOTON_URL points at another copy, and the default of
TREK_SERVICES_HOSTS now lists photon.liketrek.com and
valhalla.liketrek.com beside places.liketrek.com.
The proof of work and the bounded read of a 403 body recurse instead of looping, the background-work checks compare against null instead of testing a promise, and the tests repeat their requests through oneByOne. The renewal test loads the stored key before its first request, and the panel test picks the last confirm button without Array.prototype.at.
An expired key is no longer sent; the request goes through the shared pool
while a due key, or an expired one within its grace, is renewed in the
background. Refusals in one burst count once, a refused renewal counts like
a refused search, and a Places request whose key is refused is sent once
more without it.
docker-compose.yml and the Helm chart name the switch and the key (the key in the Secret, never the ConfigMap), and the wiki explains the installation key, what it carries, where it goes and the admin view of the allowance.
A new service-identity domain registers the installation with places.liketrek.com on its first Places request, in the background and behind a light scrypt proof of work, keeps the key encrypted in app_settings, renews it from day 60 through a daily job, drops it after three 401s in a row with up to six hours of jitter, and freezes it on a revocation until an admin starts over. TREK_SERVICES_KEY wins and turns registration off; TREK_SERVICES=off sends no key; a mirror set with TREK_PLACES_URL never registers and never sees the key, which goes only to the hosts in TREK_SERVICES_HOSTS over https.
The Places client takes its headers from the identity port, sends a TREK/<version> User-Agent and keeps the instance token. A 401 or 403 does not open its breaker. GET /api/admin/service-identity reports the state, tier and today's allowance, asking the issuer at most every 20 seconds; POST .../regenerate starts a new identity.
Without an Access Key the image search goes through Unsplash's web endpoint,
which refuses datacenter addresses with a 401 or 403 or answers with a
challenge page. The search then said only "Unsplash search unavailable",
although an admin can already enter a free key in the settings. Such a
refusal now answers 502 with that hint and the code unsplash_key_needed, and
the cover search shows it in the user's language.
Closes#1449
Maurice