mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-11-21 19:40:34 +00:00
serve each country's regions from its own file, so the server no longer holds them all
The Atlas server no longer keeps the admin-1 regions of every country in memory. Each country's regions are now a pre-gzipped file of their own, sent to the map as they are. **Why:** `GET /addons/atlas/regions/geo?countries=…` built `admin1Store` (every country's region text, kept until restart), then joined the requested countries, `JSON.parse`d them, and Nest stringified them again. On the 512 MB – 1 GB boxes TREK runs on, the first z≥5 view could OOM the server (same shape as #1576). **What changes:** - `server/assets/atlas/admin1/<ISO_A2>.geojson.gz`: 198 files, 14.85 MB, the same 3,447 features as `admin1.geojson.gz`. - New `GET /addons/atlas/regions/geo/:code` with a Zod contract (`regionGeoParamsSchema` in `@trek/shared`: two letters, any case). It sends the country's file as it is, with `Content-Encoding: gzip` and the current 1-day `Cache-Control`, as `countries/geo` does. A country without regions gets an empty FeatureCollection. Anything that is not two letters gets a 400. - `useAtlas.ts` fetches one URL per uncached country and still rebuilds the layer once. Each country has a stable URL, so a country evicted by `REGION_CACHE_MAX` comes back from the browser cache. - `getRegionFeatures` reads only the country it needs. `admin1Store` is removed. - `?countries=` keeps working for cached PWA clients. It joins the per-country files as bytes, without a parse. - The migrations are untouched. `admin1.geojson.gz` still ships, only for `Migration20200101021500` and `Migration20200101031500`. This costs ~15 MB of disk in the image and no RAM. `build-atlas-geo.mjs` writes both, and ATLAS-BUNDLE-008 keeps them equal. **Memory** (real `atlas-geo` module, first z≥5 view over Europe = 49 country codes, JS heap + Buffers): | | peak | kept after | |---|---|---| | before, first view (builds the store) | +372 MB | +61 MB | | before, later views | +115 MB | 0 | | after, `/regions/geo/:code` | +4.7 MB | 0 | | after, legacy `?countries=` | +59 MB | 0 | **Tests:** - New: `atlas-region-files.test.ts` (moved from `atlas.service.test.ts`, plus join and dedupe cases), the `regionGeoParamsSchema` spec, ATLAS-BUNDLE-007 (the file format the byte join depends on), ATLAS-BUNDLE-008 (bundle and per-country files are equal), ATLAS-016/017/018 (the new route over HTTP, including the 400). - The `getVisitedRegions` tests (ATLAS-UNIT-019..024) used to pass only when `getRegionGeo`'s `beforeAll` had built the store first. They now also pass on their own. - The shared, server and client gates pass. Baselines go down only where this change removed code (`no-console`, `no-explicit-any`, `no-swallowed-catch`, file sizes). **Trade-offs:** - A first zoom over Europe sends ~49 small parallel requests instead of 1. Over HTTP/1.1 the browser queues them 6 at a time. The total bytes are the same (4.4 MB gz). - The per-country URLs are unversioned and cached for a day, like `countries/geo`. After an upgrade that changes a country's regions, a browser can show the old shapes for up to 24 h. Approved on Discord in #github-pr: <paste the thread link here> Related: #1576, #2559 - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] I have read the [Contributing Guidelines](https://github.com/liketrek/TREK/wiki/Contributing) - [ ] My branch is [up to date with `dev`](https://github.com/liketrek/TREK/wiki/Development-environment#3-keep-your-fork-up-to-date) - [x] This PR targets the `dev` branch, not `main` *(wiki-only PRs are exempt)* - [x] I have tested my changes locally - [x] I have added/updated tests that prove my fix is effective or that my feature works - [x] I have updated documentation if needed
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
#!/usr/bin/env node
|
||||
// Build server/assets/atlas/{admin0,admin1}.geojson.gz from geoBoundaries (gbOpen).
|
||||
// Build server/assets/atlas/{admin0,admin1}.geojson.gz and server/assets/atlas/admin1/<ISO_A2>.geojson.gz
|
||||
// from geoBoundaries (gbOpen).
|
||||
//
|
||||
// Why: Atlas previously fetched country + sub-national boundaries from Natural Earth's
|
||||
// GitHub `master` at runtime. Natural Earth is stale (e.g. it still shows Norway's
|
||||
@@ -9,8 +10,9 @@
|
||||
//
|
||||
// This downloads the *simplified* per-country gbOpen ADM0 (countries) and ADM1
|
||||
// (regions) layers from a pinned geoBoundaries revision, normalizes each feature to
|
||||
// the property names the Atlas client/server already read, and writes two gzipped
|
||||
// FeatureCollections that the server serves at runtime (no network at boot).
|
||||
// the property names the Atlas client/server already read, and writes gzipped
|
||||
// FeatureCollections that the server serves at runtime (no network at boot): one for
|
||||
// every country's borders, and one per country for its regions.
|
||||
//
|
||||
// geoBoundaries: CC BY 4.0 — https://www.geoboundaries.org/ (attribution required).
|
||||
import fs from 'node:fs';
|
||||
@@ -461,7 +463,7 @@ function normalizeAdm1(geo, a3, countryName) {
|
||||
}
|
||||
return {
|
||||
type: 'Feature',
|
||||
// Property names the Atlas region layer + server getRegionGeo already read.
|
||||
// Property names the Atlas region layer + server region lookup already read.
|
||||
properties: {
|
||||
iso_a2: a2,
|
||||
iso_3166_2: code,
|
||||
@@ -516,8 +518,25 @@ async function main() {
|
||||
};
|
||||
|
||||
write('admin0', adm0Features);
|
||||
// Read only by two shipped migrations (Migration20200101021500/…031500), whose bodies
|
||||
// never change; the server itself reads the per-country files below.
|
||||
write('admin1', adm1Features);
|
||||
|
||||
// admin1 also goes out as one file per country, named by ISO_A2, so the server can hand
|
||||
// a country's regions to the map straight from disk and read only the country it needs
|
||||
// for a lookup — the single bundle had to be held whole in memory to do either. A
|
||||
// feature with no alpha-2 code has no file to go in, and nothing could look it up.
|
||||
const byCountry = new Map();
|
||||
for (const f of adm1Features) {
|
||||
const a2 = f.properties.iso_a2;
|
||||
if (!a2) continue;
|
||||
if (!byCountry.has(a2)) byCountry.set(a2, []);
|
||||
byCountry.get(a2).push(f);
|
||||
}
|
||||
fs.rmSync(path.join(OUT_DIR, 'admin1'), { recursive: true, force: true });
|
||||
fs.mkdirSync(path.join(OUT_DIR, 'admin1'));
|
||||
for (const [a2, features] of byCountry) write(`admin1/${a2}`, features);
|
||||
|
||||
const missing1 = COUNTRIES.filter((a3, i) => !normalizeAdm1(adm1Raw[i], a3, '').length);
|
||||
console.log(`[atlas-geo] ADM0 country features: ${adm0Features.length}`);
|
||||
console.log(`[atlas-geo] ADM1 countries without regions (skipped/404): ${missing1.length}`);
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
"@typescript-eslint/no-unsafe-function-type": 1,
|
||||
"@typescript-eslint/no-unused-vars": 36,
|
||||
"no-await-in-loop": 562,
|
||||
"no-console": 235,
|
||||
"no-console": 234,
|
||||
"no-empty": 6
|
||||
},
|
||||
"suppressed": {
|
||||
@@ -16,7 +16,7 @@
|
||||
"no-misleading-character-class": 1
|
||||
},
|
||||
"tests": {
|
||||
"@typescript-eslint/no-explicit-any": 1562,
|
||||
"@typescript-eslint/no-explicit-any": 1559,
|
||||
"@typescript-eslint/no-non-null-assertion": 1755,
|
||||
"@typescript-eslint/no-require-imports": 25,
|
||||
"@typescript-eslint/no-unsafe-function-type": 7,
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
"src/db/repositories/Reservations.repository.ts": 1604,
|
||||
"src/db/repositories/Trips.repository.ts": 1121,
|
||||
"src/db/repositories/Users.repository.ts": 1445,
|
||||
"src/nest/atlas/atlas.service.ts": 1182,
|
||||
"src/nest/atlas/atlas.service.ts": 1177,
|
||||
"src/nest/auth/auth.service.ts": 1143,
|
||||
"src/nest/budget/budget.service.ts": 1617,
|
||||
"src/nest/collections/collections.service.ts": 1478,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"tests/unit/nest/atlas.service.test.ts": 2193,
|
||||
"tests/unit/nest/atlas.service.test.ts": 2154,
|
||||
"tests/unit/nest/budget.service.db.test.ts": 2220,
|
||||
"tests/unit/nest/doc-sync/doc-sync.service.test.ts": 2511,
|
||||
"tests/unit/nest/journey-domain.service.test.ts": 3876,
|
||||
|
||||
Reference in New Issue
Block a user