serve each country's regions from its own file, so the server no longer holds them all

The Atlas server no longer keeps the admin-1 regions of every country in memory. Each country's regions are now a pre-gzipped file of their own, sent to the map as they are.

**Why:** `GET /addons/atlas/regions/geo?countries=…` built `admin1Store` (every country's region text, kept until restart), then joined the requested countries, `JSON.parse`d them, and Nest stringified them again. On the 512 MB – 1 GB boxes TREK runs on, the first z≥5 view could OOM the server (same shape as #1576).

**What changes:**
- `server/assets/atlas/admin1/<ISO_A2>.geojson.gz`: 198 files, 14.85 MB, the same 3,447 features as `admin1.geojson.gz`.
- New `GET /addons/atlas/regions/geo/:code` with a Zod contract (`regionGeoParamsSchema` in `@trek/shared`: two letters, any case). It sends the country's file as it is, with `Content-Encoding: gzip` and the current 1-day `Cache-Control`, as `countries/geo` does. A country without regions gets an empty FeatureCollection. Anything that is not two letters gets a 400.
- `useAtlas.ts` fetches one URL per uncached country and still rebuilds the layer once. Each country has a stable URL, so a country evicted by `REGION_CACHE_MAX` comes back from the browser cache.
- `getRegionFeatures` reads only the country it needs. `admin1Store` is removed.
- `?countries=` keeps working for cached PWA clients. It joins the per-country files as bytes, without a parse.
- The migrations are untouched. `admin1.geojson.gz` still ships, only for `Migration20200101021500` and `Migration20200101031500`. This costs ~15 MB of disk in the image and no RAM. `build-atlas-geo.mjs` writes both, and ATLAS-BUNDLE-008 keeps them equal.

**Memory** (real `atlas-geo` module, first z≥5 view over Europe = 49 country codes, JS heap + Buffers):

| | peak | kept after |
|---|---|---|
| before, first view (builds the store) | +372 MB | +61 MB |
| before, later views | +115 MB | 0 |
| after, `/regions/geo/:code` | +4.7 MB | 0 |
| after, legacy `?countries=` | +59 MB | 0 |

**Tests:**
- New: `atlas-region-files.test.ts` (moved from `atlas.service.test.ts`, plus join and dedupe cases), the `regionGeoParamsSchema` spec, ATLAS-BUNDLE-007 (the file format the byte join depends on), ATLAS-BUNDLE-008 (bundle and per-country files are equal), ATLAS-016/017/018 (the new route over HTTP, including the 400).
- The `getVisitedRegions` tests (ATLAS-UNIT-019..024) used to pass only when `getRegionGeo`'s `beforeAll` had built the store first. They now also pass on their own.
- The shared, server and client gates pass. Baselines go down only where this change removed code (`no-console`, `no-explicit-any`, `no-swallowed-catch`, file sizes).

**Trade-offs:**
- A first zoom over Europe sends ~49 small parallel requests instead of 1. Over HTTP/1.1 the browser queues them 6 at a time. The total bytes are the same (4.4 MB gz).
- The per-country URLs are unversioned and cached for a day, like `countries/geo`. After an upgrade that changes a country's regions, a browser can show the old shapes for up to 24 h.

Approved on Discord in #github-pr: <paste the thread link here>
Related: #1576, #2559

- [x] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

- [ ] I have read the [Contributing Guidelines](https://github.com/liketrek/TREK/wiki/Contributing)
- [ ] My branch is [up to date with `dev`](https://github.com/liketrek/TREK/wiki/Development-environment#3-keep-your-fork-up-to-date)
- [x] This PR targets the `dev` branch, not `main` *(wiki-only PRs are exempt)*
- [x] I have tested my changes locally
- [x] I have added/updated tests that prove my fix is effective or that my feature works
- [x] I have updated documentation if needed
This commit is contained in:
Vini
2026-10-26 20:08:57 +01:00
committed by Maurice
parent 70f27085b9
commit 7277e35a96
220 changed files with 360 additions and 185 deletions
+23 -4
View File
@@ -1,5 +1,6 @@
#!/usr/bin/env node
// Build server/assets/atlas/{admin0,admin1}.geojson.gz from geoBoundaries (gbOpen).
// Build server/assets/atlas/{admin0,admin1}.geojson.gz and server/assets/atlas/admin1/<ISO_A2>.geojson.gz
// from geoBoundaries (gbOpen).
//
// Why: Atlas previously fetched country + sub-national boundaries from Natural Earth's
// GitHub `master` at runtime. Natural Earth is stale (e.g. it still shows Norway's
@@ -9,8 +10,9 @@
//
// This downloads the *simplified* per-country gbOpen ADM0 (countries) and ADM1
// (regions) layers from a pinned geoBoundaries revision, normalizes each feature to
// the property names the Atlas client/server already read, and writes two gzipped
// FeatureCollections that the server serves at runtime (no network at boot).
// the property names the Atlas client/server already read, and writes gzipped
// FeatureCollections that the server serves at runtime (no network at boot): one for
// every country's borders, and one per country for its regions.
//
// geoBoundaries: CC BY 4.0 — https://www.geoboundaries.org/ (attribution required).
import fs from 'node:fs';
@@ -461,7 +463,7 @@ function normalizeAdm1(geo, a3, countryName) {
}
return {
type: 'Feature',
// Property names the Atlas region layer + server getRegionGeo already read.
// Property names the Atlas region layer + server region lookup already read.
properties: {
iso_a2: a2,
iso_3166_2: code,
@@ -516,8 +518,25 @@ async function main() {
};
write('admin0', adm0Features);
// Read only by two shipped migrations (Migration20200101021500/…031500), whose bodies
// never change; the server itself reads the per-country files below.
write('admin1', adm1Features);
// admin1 also goes out as one file per country, named by ISO_A2, so the server can hand
// a country's regions to the map straight from disk and read only the country it needs
// for a lookup — the single bundle had to be held whole in memory to do either. A
// feature with no alpha-2 code has no file to go in, and nothing could look it up.
const byCountry = new Map();
for (const f of adm1Features) {
const a2 = f.properties.iso_a2;
if (!a2) continue;
if (!byCountry.has(a2)) byCountry.set(a2, []);
byCountry.get(a2).push(f);
}
fs.rmSync(path.join(OUT_DIR, 'admin1'), { recursive: true, force: true });
fs.mkdirSync(path.join(OUT_DIR, 'admin1'));
for (const [a2, features] of byCountry) write(`admin1/${a2}`, features);
const missing1 = COUNTRIES.filter((a3, i) => !normalizeAdm1(adm1Raw[i], a3, '').length);
console.log(`[atlas-geo] ADM0 country features: ${adm0Features.length}`);
console.log(`[atlas-geo] ADM1 countries without regions (skipped/404): ${missing1.length}`);
+2 -2
View File
@@ -6,7 +6,7 @@
"@typescript-eslint/no-unsafe-function-type": 1,
"@typescript-eslint/no-unused-vars": 36,
"no-await-in-loop": 562,
"no-console": 235,
"no-console": 234,
"no-empty": 6
},
"suppressed": {
@@ -16,7 +16,7 @@
"no-misleading-character-class": 1
},
"tests": {
"@typescript-eslint/no-explicit-any": 1562,
"@typescript-eslint/no-explicit-any": 1559,
"@typescript-eslint/no-non-null-assertion": 1755,
"@typescript-eslint/no-require-imports": 25,
"@typescript-eslint/no-unsafe-function-type": 7,
+1 -1
View File
@@ -5,7 +5,7 @@
"src/db/repositories/Reservations.repository.ts": 1604,
"src/db/repositories/Trips.repository.ts": 1121,
"src/db/repositories/Users.repository.ts": 1445,
"src/nest/atlas/atlas.service.ts": 1182,
"src/nest/atlas/atlas.service.ts": 1177,
"src/nest/auth/auth.service.ts": 1143,
"src/nest/budget/budget.service.ts": 1617,
"src/nest/collections/collections.service.ts": 1478,
+1 -1
View File
@@ -1,5 +1,5 @@
{
"tests/unit/nest/atlas.service.test.ts": 2193,
"tests/unit/nest/atlas.service.test.ts": 2154,
"tests/unit/nest/budget.service.db.test.ts": 2220,
"tests/unit/nest/doc-sync/doc-sync.service.test.ts": 2511,
"tests/unit/nest/journey-domain.service.test.ts": 3876,