Commit Graph

12 Commits

Author SHA1 Message Date
Maurice 4a88dbcbcf fix(places): say where a result came from, and stop asking questions in the form
Six changes to the place picker and the card behind it, all of them about a
panel saying less and meaning more.

The TREK place index has no switch in the admin panel any more. It decides
whether a search leaves the instance at all, which is a property of the
deployment rather than a taste: an operator pins TREK_PLACES_ENABLED=false in
their compose file, and it cannot be turned off from a browser by whoever holds
an admin account that day. The switch never shipped, so nothing to migrate.

The standing "Using OpenStreetMap" notice above the search box is gone. It was
a sentence about configuration in the middle of a form about a place, and it
answered a question nobody asked while leaving the real one open. Each result
now carries a small mark instead: TREK, OpenStreetMap or Google. It sits on the
row rather than above the list, because a list can be two indexes interleaved
and a heading could only name one of them.

The saved-places filters open across the full width of the row. Two of them sit
side by side, only one can be open at a time, and list names were being cut at
about ten characters for no reason.

Three empty states use the mascot the rest of TREK uses. The place details panel
no longer tells the reader to go and ask an administrator for a Google key, the
saved-places list and the road trip rail no longer explain themselves in two
lines of grey text. That needed one new mood, `sad`: `sleepy` reads as falling
asleep, which is a different thing from finding nothing.

The Kind of stop field is out of the place form. It is a road trip concept, and
the road trip view already sets it from the stop's own number, where the choice
is next to what it changes.
2026-09-08 20:41:57 +02:00
Maurice 7829043a00 test(app-config): pin the overpass budget against its own constant
The client-side budget moved to twenty-five seconds when the query timeout it
has to outlast was written down as a constant, and this assertion stayed on the
old twelve. Pointing it at OVERPASS_TIMEOUT_DEFAULT_MS is what the test was
after in the first place: the budget must be longer than the timeout the query
carries, and a literal here is exactly how the two came apart.
2026-09-07 18:36:54 +02:00
Maurice c5208967ac fix(app-config): refuse an LLM ceiling that would become a zero timeout
LLM_TIMEOUT_MS was validated as "a positive number", which lets two shapes
through that then degrade into an immediate abort rather than a long one.
A fractional value passes the check and floors to 0 in derive, which makes
the dispatcher fall back to undici's default and setTimeout(abort, 0) kill
every model call on the next tick. Anything past 2^31-1 does the same,
because Node clamps the delay to 1 ms. Both now refuse at boot, which is
what the wiki row already promised.

Also pins the seam the three deleted constants used to occupy: the clients'
abort deadline has to come from the configured ceiling. Nothing else in the
suite would have noticed a literal coming back.
2026-09-04 18:56:22 +02:00
RemEx dba4434ef6 fix(llm-parse): one configurable ceiling for a model call, and undici honours it
The timeout for a model call was spelled three times and adjustable
nowhere: 120s in the Anthropic client, 300s in the OpenAI-compatible one,
300s in the Ollama one. None of them was the ceiling that actually
applied, either — every one of those calls goes through the pinned undici
dispatcher, and an undici Agent stops waiting for response headers after
five minutes by default, so a client constant set above that never got
the chance to fire.

- LLM_TIMEOUT_MS (default 900000) replaces the three constants and is
  passed to the dispatcher as headersTimeout/bodyTimeout, so the abort
  signal and undici are governed by the same setting. The default is
  deliberately generous: heavier parsing work should fit without a code
  change, and an admin who would rather fail fast can lower it.
- The ceiling stays on the model lane. safeFetchAdminConfigured is also
  the OIDC lane (discovery, token, userinfo, JWKS) and the plugin OAuth
  token exchange, and that one sends no AbortSignal of its own, so it
  keeps undici's five minutes rather than inheriting a model setting.
- The derived value is floored: the schema accepts any positive number
  and undici rejects a fractional headersTimeout.
2026-09-04 18:56:22 +02:00
jubnl bc10fba389 feat(plugins): TREK_PLUGINS_IGNORE_TREK_RANGE turns the version gate into a warning (#2226)
* feat(plugins): TREK_PLUGINS_IGNORE_TREK_RANGE turns the version gate into a warning

A plugin whose author has not updated its `trek` range for the running TREK can
now be installed, updated and activated when the operator sets
TREK_PLUGINS_IGNORE_TREK_RANGE to a truthy value. Every gate (registry picker,
post-extraction re-check, sideload, dev-link, activation) warns instead of
refusing; a missing range is tolerated the same way; "install latest" takes the
newest published version. The plugin-API version gate is not lifted and the
Discover compatibility verdict stays truthful.

The admin is warned at every step: each bypass is logged, install/upload/link/
update responses carry a `trekRangeBypassed` marker, and both admin panels show
a "Version checks off" pill, an "Install anyway" confirm dialog before a
registry install, the same notice after a path that could not ask first, and a
persistent chip on the row while the plugin runs outside its range.

Nine new i18n keys in every locale; wiki pages updated (Environment-Variables,
Admin-Plugins, Plugin-Development, Security-Hardening).

* refactor(plugins): one useRangeBypass hook behind both admin shells

SonarQube flagged PR 2226 at 10.8% duplicated new code: the range-bypass
logic had been added to the desktop and phone plugin panels as identical
copies. The marker/warning types, the row chip, the "Install anyway" offer,
the dialog copy and the confirm-or-notice state now live in
useRangeBypass (next to useInstanceSettings, the existing shared-logic
precedent); each panel keeps only its own dialog/sheet markup.

* docs(claude): record the SonarQube gate rules and the no-English-placeholder i18n rule

The CI SonarCloud gate (Sonar way, on new code: >=80% coverage, <=3%
duplication, A ratings, hotspots reviewed) was only discoverable by waiting
20-25 minutes for the workflow. The root CLAUDE.md now states the conditions,
the desktop/mobile shell-mirror duplication trap and its shared-hook fix, and
how to check a PR with the sonarqube MCP first; client and server CLAUDE.md
point at it. shared/CLAUDE.md says every locale needs a real translation —
parity cannot tell an English placeholder from a translation.

* docs(claude): keep CLAUDE.md to commands, invariants, gates and philosophy

Verified every concrete claim in the five CLAUDE.md files against the code
and fixed the stale ones: the websocket server is the Nest realtime gateway,
MCP tools live in <domain>.mcp.ts files, the client "native shell" section
described code that was never written, offline maps also prefetch vector
basemaps, plus several renamed files and wrong table names.

Then trimmed everything that drifts on its own: dates, migration history,
counts, and inventories of stores, slices, domains, sync files and CLI
commands. What stays is the commands, the invariants with the file that
enforces each one, what CI actually runs, the reference implementations to
copy, and the rules for new code. Each file now opens with that rule.
2026-09-04 00:43:06 +02:00
Maurice a7aaa29c16 fix(db): stop losing secrets and notes to the boot and migration paths
An unreadable or empty encryption-key file was treated the same as a missing
one: regenerate and overwrite. That is the one case where regenerating is
the worst possible move - every secret already at rest becomes undecryptable,
permanently. Both now refuse to start and say which file to fix.

The key-rotation script advertised re-encrypting all stored secrets while its
hardcoded column lists had drifted: the AirTrail key, the per-user LLM key
and the plugin OAuth tokens were never migrated, so they silently stopped
decrypting after a rotation. The parity test now pins the full set against
what the code actually encrypts, rather than two tables of it.

The schema_version bump ran outside the migration's transaction, so a crash
in between left the schema advanced and the version stale - the next boot
replayed a non-idempotent step and exited, turning one crash into a boot
loop. And the TICKETJSON receipt migration matched with LIKE, which SQLite
treats case-insensitively, so a note somebody had typed starting 'ticketjson:'
was cut up and its text dropped.

Also: APP_VERSION was interpolated into migration SQL instead of bound, the
capture-metadata backfill passed (photoId, userId) to a (userId, photoId)
signature and its mock keyed on the wrong argument, TRUST_PROXY=0 was
accepted and silently coerced to 1, demo-reset hardcoded the db path instead
of honouring TREK_DB_FILE, and a demo instance with no DEMO_ADMIN_PASS set
now says out loud that its admin password is the published default.
2026-08-23 11:59:12 +02:00
jubnl 305b113d5f Feat/storage driver (#1949)
* feat(storage): storage key validation, driver contract suite, LocalDriver

* feat(storage): MirrorDriver composite backend

* feat(server): add TREK_UPLOADS_DIR env var for the uploads root

* feat(storage): settings-backed registry, StorageService facade, module wiring

* test(storage): pin src/nest/storage coverage thresholds

* feat(storage): multer upload options factory

* refactor(files): spool trip-file uploads through storage

* refactor(collab): spool note attachments through storage; normalize files/ prefix

* refactor(journey): spool journey media uploads through storage

* refactor(trips): spool trip cover upload through storage

* refactor(places): spool place-image and collection uploads through storage

* refactor(auth): spool avatar upload through storage

* fix(server): consolidate allowed_file_types onto AllowedFileTypesService

* fix(storage): default defParamCharset to utf8 in the upload factory

* test(db): re-point the trailing-migration guard at the trip_files prefix strip

* fix(storage): age-gate the boot spool sweep

* test(files): key FILE-P02 residue checks on upload size

* test(storage): raise src/nest/storage coverage pin

* refactor(platform): serve /uploads routes through storage

* refactor(files): serve trip-file downloads through storage

* refactor(memories): serve photo bytes through storage

* refactor(maps): stream place-photo bytes through storage

* refactor(share): stream shared place-photo bytes through storage

* refactor(journey): serve public journey assets through storage

* fix(share): answer place-photo cache miss with 204 like the maps proxy

* test(server): close serving-swap gaps surfaced by the full tier

* test(storage): raise src/nest/storage branches pin

* refactor(place-photos): route place-photo cache through storage

* refactor(places): reclaim place images and collection covers through storage

* refactor(trips): trip covers through storage

* refactor(auth): avatar reclaim through storage

* refactor(collab): note-file deletion through storage

* refactor(files): trash deletion through storage

* refactor(memories): trek photo cache internals through storage

* refactor(memories): thumbnail service category-addressed

* refactor(memories): immich upload reads through storage; drop uploads-root

* refactor(journey): photo deletion through storage

* fix(memories): reclaim orphaned trek photo cache objects in the ttl sweep

* feat(journey): journey-thumbs orphan sweep and delete-site derived-artifact reclaim

* test(server): close slice-4 gaps surfaced by the full tier

* test(coverage): raise slice-4 domain pins

* refactor(backup): list, existence and delete through storage

* refactor(backup): download through storage sendToResponse

* refactor(backup): create backup through per-category storage enumeration

* refactor(backup): restore and upload-restore through storage

* refactor(backup): auto-backup retention through storage

* refactor(server): boot dir-creation to driver init

* test(server): close slice-5 gaps surfaced by the coverage tier

* test(coverage): raise slice-5 backup pin

* test(server): resolve async backupFileExists mocks for the tests typecheck

* test(server): pin ranged-read stat and zero-byte roundtrip in the storage driver contract

* feat(server): TREK_S3_* env wiring — schema, s3 namespace, all-or-nothing boot rule

* feat(server): S3 driver core — aws-lite client seam, stat/delete, driver-owned deadlines

* refactor(server): type the aws-lite S3 client via s3-types reference

* feat(server): S3 driver getStream and list — ranged full-object stat, iterator pagination

* feat(server): S3 driver put — bounded-peek routing, Upload inactivity deadline

* fix(server): S3 driver Upload monitoring — Transform interposition, stream cleanup

* feat(server): env-declared s3-main storage backend; settings-declared s3 rejected

* test(server): env-gated S3 driver contract suite against MinIO

* fix(server): S3 driver zero-byte put — work around aws4 duplicate Content-Length signing defect

* fix(server): scope the zero-byte put workaround to buffered bodies; bind test MinIO to localhost

* ci(server): MinIO-backed S3 driver contract job

* ci(server): align s3-contract shim comment and MinIO bind with references

* docs: S3 storage backend configuration and off-box backup mirroring

* docs: restore README formatting, keep S3 additions only

* ci(server): make live-MinIO contract runs manual-only (AIStor licensing)

MinIO's community line is discontinued entirely; the maintained successor, AIStor, requires a license key that can't safely live in a shared CI runner. The s3-contract CI job is removed; live-provider contract runs against MinIO/AIStor are now a manual, license-gated procedure documented in docker-compose.minio-test.yml's header.

* chore(server): final-review polish — stale CI refs, dockerignore key, boundary pins, zero-byte tempfile contract case

* expose admin ui

* feat(shared): storage backend type registry, wire schemas, category relocation

* feat(server): accept settings-declared s3 backends — shared-schema options, decrypt-at-build, registry preview()

* feat(server): remove TREK_S3_*/TREK_UPLOADS_DIR — tombstone boot rule, built-in override as the relocation path

* feat(server): deriveSecurity.encryptionKeySet and storage secret plumbing (gate, encrypt, unmask, mask, redact)

* chore(server): drop storage-secrets import that only Task 8's probe helper will use

* feat(server): seed-once storage-config.json boot import — fail closed on any invalid seed

* feat(server): registry snapshot() — effective storage config with provenance

* feat(server): api/admin/storage GET/PUT — masked secrets, preview-gated saves, wired reload, audits

* feat(server): api/admin/storage/test — per-target ephemeral probes, mirror targets probed individually

* test(server): e2e coverage for api/admin/storage — guards, managed refusal, envelope, probes

* test(server): give e2e harness DBs the app_settings table the storage registry reads at boot

* test(coverage): raise pins after the storage admin-config slice

* fix(server): final-review polish — encryption-gate wording, all-field sentinel guard, README/e2e notes

* test(server): storage slice-1 review follow-ups — mirror s3-replica probe decrypt, all-encrypted seed without key, test-route 401

* feat(i18n): storage admin domain + tab label in every locale

* feat(client): adminApi storage endpoints — getStorage, updateStorage, testStorageBackend

* feat(client): storage admin draft-document helpers + relative-time util

* feat(client): schema-driven storage BackendForm — fields by kind, mask echo, encryption banner

* feat(client): admin storage panel — backends list, category map, health strip, one-PUT save

* feat(client): storage tab on the admin page

* feat(client): storage admin panel and section on the phone shell

* fix(client): block renaming a storage backend onto an existing name

* feat(server): allow mirrors on any storage category — lift the backups-only rule

* feat(i18n): replicas-on-primary mirror strings in every locale

* feat(client): mirror fold/synthesize model helpers for replicas-on-primary

* feat(client): BackendForm mirror-targets composer; raw mirror type hidden

* feat(client): desktop storage panel folds mirrors into their primaries

* feat(client): phone storage panel folds mirrors into their primaries

* fix(client): strip removed backends from remaining mirror replicas — no phantom refs

* feat(storage): retire photos as a configurable category — served-only via the ServedCategory seam

* feat(i18n): descriptive storage category names and descriptions in every locale

* feat(client): descriptive category names, descriptions and id badges in the storage panel

* docs(storage): fix stale nine-category comments after the eight-category shrink

* docs: storage configuration via the admin UI and seed file — README + compose

* docs(wiki): Admin Storage page and storage cross-links

* docs(server): storage domain blueprint entry; TREK_PLACE_PHOTO_DIR tombstone clarifier

* refactor(server): drop the storage env-var tombstones — the variables never shipped

* docs: storage config is simply new in v4 — drop the removed-variables story

* feat(shared): backfill-status and usage wire schemas on the storage admin state

* feat(server): MirrorDriver.backfill — prefix-scoped, size-compared, cancellable replica catch-up

* feat(server): one-at-a-time storage backfill jobs + the storage events bridge

* feat(server): storage usage scan — nightly cron, on-demand service, persisted with computedAt

* feat(server): replica_failure admin notifications — events bridge, hourly debounce, channel registration

* feat(i18n): replica-failure notification text in every locale

* feat(server): storage backfill and stats-refresh admin endpoints

* feat(client): adminApi backfill and stats-refresh endpoints

* feat(i18n): storage sync and usage strings in every locale

* feat(client): desktop storage panel — sync now, usage display, backfill polling

* feat(client): phone storage panel — sync now, usage display, backfill polling

* fix(client): per-category usage line on the phone storage panel

* docs: sync now, usage stats and replica-failure notifications

* test(server): admin scope now includes replica_failure, not just version_available

Commit 6db135da added replica_failure to ADMIN_SCOPED_EVENTS but left
NPREF-005b asserting the admin scope contained only version_available.
Update the expectation to match.

* test(server): cover storage backfill/stats/jobs branches to hold the coverage pin

* fix(client): guard storage admin polling against stale responses racing a save

A poll GET issued before a save's PUT resolves could land after the save
applied its fresher response, overwriting it with the pre-save world and
re-deriving the draft from that stale snapshot. useStorageAdmin now tracks
a state-application sequence number and drops any refreshState() response
whose sequence has moved on, or that resolves while a save is in flight.
Also guard the fire-and-forget refreshState() calls in startBackfill/
cancelBackfill against unhandled rejections, matching the poll's style.

* fix(server): report 404 before 409 when a backfill start names an unknown mirror

startBackfill checked the global busy flag before validating that :name is
a mirror routed by some category, so POST .../backends/garbage/backfill
answered 409 instead of 404 whenever a sync was already running. Resolve
and validate the target first (BackfillTargetError -> 404), then check
busy (BackfillBusyError -> 409).

* feat(client): two-column category grid on the storage panel

* feat(server): plugin files.getContent reads through the storage layer

* feat(server): plugin files.create writes through the storage layer

* refactor(server): drop FilesService.resolveFilePath, last direct-fs seam in the plugin surface

* fix(server): map StorageInvalidKeyError to the plugin file accessibility envelope

* feat(storage): allow credentialed backends without an explicit ENCRYPTION_KEY

Secrets are still always encrypted at rest; without an explicit key the
implicit data-directory key is used. Removes the plaintext-secret refusal
from the admin PUT and the seed import, the encryptionReady GET field, the
save-blocking banner in both admin shells, and the now-consumerless
security.encryptionKeySet env derivation. Docs now recommend (not require)
an explicit key, since the implicit key rides inside backup archives.

* test(client): pin the stale-poll release to the pre-save GET in FE-ADMIN-STOR-024

The stub deferred EVERY poll GET and rebound the release handle to the
newest one, so a tick landing after the save was released with the
pre-save payload — a response no real backend can produce, which the seq
guard rightly applied. Only the first (genuinely stale) GET is held now;
later GETs answer with the world the mock server currently holds. Also
drops the encryptionReady fixture field removed from the schema in the
previous commit.

* fix(storage): backfill puts carry content-type, stat failures report op stat, progress clamps

* feat(storage): category migration job — copy, flip, delta sweep, reclaimable tally

* feat(storage): migration start/cancel endpoints and client API

* test(client): storage admin fixtures carry the new migrations field

* fix(storage): migration POST answers 200 and validates its body via the shared schema

* fix(storage): cancel running storage jobs when a config save removes their backend

* fix(notifications): in-app replica-failure row carries the suppressed count

* feat(client): migration prompt, queue, and progress on the storage panel

* fix(client): migration queue holds a synchronous single-flight lock

* feat(client): migration prompt and progress on the mobile storage panel

* fix(server): register category-migration routes in the managed-route allow list

StorageAdminController.migrationStart/migrationCancel were added under the
controller-level @ManagedForbidden() decorator but never added to
MANAGED_ROUTE_ALLOW_LIST, so validateManagedRoutes() threw "Managed surface
changed" on every buildApp() call, failing 46 test files (any test booting
the real app).

* test(server): cover category-migration copy/sweep branches to hold the storage coverage pin

Adds MIG-005 (a target object already matching size/key is skipped in the
copy phase itself, not just the delta sweep) and MIG-006 (a copy failure
during the delta sweep is counted without undoing the flip that already
happened). src/nest/storage/**'s branch coverage was 83.91%, just under the
84% pin.

* docs(storage): category migration — move existing objects on reassign

* fix(storage): close the pre-flip cancel window and cancel re-routed migrations

A cancel landing between the last copy iteration and the category flip was
silently ignored, and a config save that re-routed the migrating category to
a third, still-defined backend never stopped the in-flight migration. Also
guards assignCategory against writing an unknown backend name, and fixes a
missing `suppressed` notification param being treated as truthy suppression.

* fix(client): migrations keep mirrors, queue waits on backfills, sweep failures surface

Reassigning a category onto a mirrored primary now posts the migration to the
mirror's wire name instead of the bare primary, so replication survives the
move. The queued-migration effect also waits out a running backfill (the
server's one-job rule spans both), and a done migration with sweep failures
now surfaces a doneFailures line in both shells, translated into every locale.

* ci(server): run the S3 contract suite against MinIO

Resurrects the s3-contract job removed in 6fbff835, adapted for AIStor:
writes the MINIO_LICENSE_KEY secret to a license file, boots the licensed
container, and runs the contract suite hermetically (no --coverage) so the
storage coverage pin keeps holding from unit tests alone. Gated to same-repo
PRs since fork PRs have no access to the license secret.

* fix(server): backup archives uploads from a remote primary

createBackup() enumerated uploads through withLocalFile, which downloads a
remote object to a temp file, hands it to the callback, then deletes it
before archiver ever reads it — archiver reads entries lazily during
finalize(), so a remote (S3) primary silently backed up zero upload bytes.

Add StorageService.getLocalPathOrNull(category, name): a narrow locality
probe that returns a real path only when one exists on disk right now (the
fs-existence check is the fail-safe half of the contract — a driver-reported
path that vanished between list() and here falls back too, not just a
path-less remote driver). The enumeration loop uses it directly when
available (zero-copy, unchanged for a local primary) and otherwise streams
the object into a per-backup staging dir under the backups spool, cleaned up
in the existing finally alongside zipSpool/dbSnap.

Also wire archive.on('warning', reject): archiver treats a missing/unreadable
entry as a warning, not an error, and skips it by default — which would
silently drop the entry from an otherwise "successful" backup.

* fix(server): client aborts stop 500ing and leaking streams

storage.service.ts sendToResponse: the local res.sendFile branch now treats
ECONNABORTED/ECONNRESET/ERR_STREAM_PREMATURE_CLOSE/syscall=write as success,
mirroring res.sendFile's own default callback (the contract
storageStaticHandler already implements caller-side). The remote branch
replaces the hand-rolled pipe+Promise with pipeline(stream, res), which
destroys the source stream on any failure — fixing the leaked S3 keep-alive
socket on a client abort — while swallowing the same client-abort error set;
a real source error still rejects so callers keep their miss contracts.

This centralizes the fix for every caller that streams through
sendToResponse: files-download, journey-public, photo-resolver,
trek-photo-cache, backup downloads, and /uploads/photos.

trek-exception.filter.ts now no-ops (destroying the socket) when headers are
already sent, so no filter path ever writes onto a response in flight.

maps.controller.ts and share.controller.ts's place-photo bytes proxies
switch their direct getStream+pipe to pipeline() with the same abort-tolerant
catch, keeping the existing 204-empty fallback for pre-header failures.

* fix(storage): mirror read fallback consults replicas on primary miss

MirrorDriver.getStream/stat now fall back to replicas on a primary miss
(StorageNotFoundError / null stat), not just an infra error — binding
decision: a read succeeds if ANY member holds the object. Only a
deterministic StorageInvalidKeyError skips fallback.

stat() gets a dedicated fallback loop instead of reusing fromReplicas:
a replica returning null means "keep looking," which the generic
catch-and-continue helper would have short-circuited on.

StorageService.sendToResponse/withLocalFile stop treating composite
driver.stat() as proof the object is at localPath; both now fs-check
the path itself and fall through to the stream branch when it's
missing, which is what actually reaches a MirrorDriver's replicas.

* feat(storage): remote serving parity — content-type, validators, range

Local backends get content type, ETag/Last-Modified, conditional GET, Range
and HEAD for free from res.sendFile; the proxying branch every remote
backend (and a local primary whose file vanished) falls through to had none
of it — every request re-streamed the whole object with no cache validator
and no way to seek.

sendStreamed() reproduces the contract from ONE pre-flight driver.stat():
a miss throws before any header is set (the caller's 404/204/next()
contract), and that same stat then answers 304s, 416s, HEADs and suffix
ranges without ever opening a body. The decisions live in a new pure module,
http-serving.ts:

- validatorsFor: the driver's ETag, else express/send's weak
  W/"<size hex>-<mtime hex>". mtimeMs 0 (S3 can report it) means no
  Last-Modified and no derived tag — serving uncached beats a 1970 header
  that reads as fresh forever.
- isNotModified: If-None-Match (weak compare, *, list) wins over
  If-Modified-Since, per RFC 9110 §13.2.2.
- decideRange: single range only, inclusive clamp, suffix computed from the
  pre-stat size, unsatisfiable → 416, If-Range strong-exact or full entity.
  A multi-range or malformed spec is answered with the full object rather
  than multipart/byteranges. If-Match/If-Unmodified-Since stay out of scope.

Range math uses the pre-stat size as the single source of truth, the ranged
getStream stat is never re-consulted, and the pipeline abort semantics are
unchanged.

S3Driver now surfaces the bucket's ETag on stat()/getStream() (on a ranged
GET it is still the full object's tag). aws-lite strips the tag's
surrounding quotes — verified live against AIStor RELEASE.2026-08-07 — and
an unquoted entity-tag is not a valid HTTP ETag, so the driver restores
exactly the quotes the SDK removed and nothing else.

* fix(server): key rotation covers storage.backends

The S3 storage backend's secretAccessKey is encrypted at rest inside the
app_settings row keyed storage.backends, but scripts/migrate-encryption.ts
didn't know about it — after a key rotation that secret stayed encrypted
under the old key and silently failed to decrypt on the next read.

The script now re-encrypts secretAccessKey for every s3 backend inside the
storage.backends JSON array, using a pinned copy of the secret-field list
(it deliberately can't import @trek/shared). The parity test pins that copy
against storageSecretFields('s3') so a new secret field can't be missed
silently.

* fix(server): restore reloads the storage registry

restoreFromZip closed and reopened the DB but never reloaded the
storage registry, so uploads rehydration ran against the STALE
pre-restore driver/category map instead of the config the restored
archive's own travel.db carries (audit #4, major).

StorageService gains reloadConfig(), a narrow passthrough to the
(unexported) registry's reload() — same shape as health(). backup.impl's
restoreFromZip now calls it right after reinitialize() and right before
the uploads wipe/rehydrateUploads/plugin staging, so rehydrated bytes
land where the restored config says. When reopen fails (reinitFailed),
reload and rehydration are skipped entirely — already reported as
"restart required", and there is no live DB handle for the registry to
read anyway.

* fix(server): category migration rewrites keys across prefix changes

Category migration copied objects to the destination backend using the
SOURCE key verbatim. A mode-A ('' prefix) source migrating onto a
prefixed backend (or vice versa) landed objects at keys the registry
could never resolve post-flip (audit #8).

Extracts StorageRegistryService.keyPrefixFor(category, backendName) from
build()'s inline conditional so startMigration can compute the
destination prefix before the flip. runMigration now maps every
destination key (destPrefix + key.slice(sourcePrefix.length)) in the
copy skip-check, the put, and the delta sweep; the reclaimable tally
stays source-keyed. Equal-prefix migrations are unaffected (identity
mapping).

* fix(plugins): await the now-async place/collection services in RPCs

PlacesService.update/remove and CollectionsService.deletePlace became
async when they started deleting storage objects, but their plugin RPC
handlers still called them synchronously: the null/false refusal check
compared against an always-truthy Promise, and places.update broadcast
a Promise instead of the resolved place. Both handlers now await the
service call; mapCollectionError is async-aware (accepts a sync-or-
async thunk) so the sync callers (create/update/savePlace/copyToTrip)
keep working through the same path. Verified the RPC host already
awaits handler results, so no change was needed there.

* fix(storage): backfill Sync now gains a deletion sweep

The Sync now backfill copied primary objects to replicas but never removed
replica objects the primary no longer has, leaving mirror read-fallback able
to serve stale data indefinitely after a failed replica delete. backfill()
now sweeps each replica post-copy: any replica key absent from the primary
key-set is re-checked against primary.stat (a race gate against the primary
regaining the key) before being deleted. Adds a deleted counter end-to-end
(driver, jobs service, shared schema, both admin shells, all 23 locales) and
a new ReplicaFailure 'list' op for a replica whose listing itself fails
during the sweep.

* feat(storage): optimistic concurrency for admin config saves

A stale admin PUT of storage settings could silently undo a category
migration's just-flipped assignment. Adds a version counter
(app_settings row storage.config_version) bumped inside the same
transaction by both writers — applyConfig and the migration flip
(assignCategory). applyConfig compares the submitted version first
and throws StorageConflictError, mapped to 409 ahead of the blanket
400. The client carries the draft's loaded version through every
local edit and maps a 409 to a distinct "settings changed underneath
you" message, refreshing state while preserving the dirty draft for
review.

* fix(storage): surface a failed config load + redact secretAccessKey

Registry load() now records the exact parse/validate failure as
lastLoadError() instead of silently falling back to last-good config or
built-in defaults; StorageAdminService exposes it as configError on the
admin state (storageAdminStateSchema extended additively), and
AdminStoragePanel renders it as a warning banner naming the error. Save
stays enabled — it's the recovery path — but an admin can no longer save
over a broken stored config without knowing.

Also adds 'secretaccesskey' to globalMiddleware's SENSITIVE_KEYS request-log
redaction list, so storage admin PUT bodies never leak the S3 secret into
debug logs.

* fix(client): admin storage panel state races

Migration prompt now recomputes computeMigrationCandidates(draft, state)
on render and again at confirm time, so a stale open-time snapshot never
drives stripCategories or the migration queue; adds a Cancel button to
the dialog. A failed startMigration POST explicitly clears the remaining
queue with a toast naming the dropped categories and refreshes state,
instead of leaving it invisibly stranded; a "Queued: X, Y" line renders
while the queue is non-empty. Testing a mirrored row now expands the
draft primary + replicas client-side and probes each individually,
instead of sending the mirror stub to the server (which resolved
targets from its own live snapshot, silently testing saved options over
unsaved draft edits). A completed/cancelled/errored backfill now renders
its terminal status alongside the Sync now button instead of hiding it,
so it can be re-run without a page reload.

* fix(storage): refuse shared replica backends so the sync sweep cannot cross categories

A mirror's "Sync now" sweep makes each replica MATCH the primary: it lists the
replica under every prefix the mirror's categories occupy and deletes the keys
the primary doesn't hold. The backups category's prefix is '' — the replica's
entire root — so making a category-serving backend (say uploads-local) a
replica of a backups mirror meant the first sync deleted every user upload.
Nothing forbade the config: validateConfig only checked that mirror targets
existed and weren't nested, and the admin picker offered every backend.

The sweep is correct, so the fix is at the config level:

- validateConfig refuses a backend that is both a mirror replica and a
  category target (directly, or as the primary of a mirror a category routes
  to), and refuses a backend replicating two mirrors whose swept prefixes
  overlap — '' overlaps everything. Disjoint prefixes stay legal.
- assignCategory (the migration flip, the other writer of the settings rows)
  refuses routing a category onto a backend that is currently a replica.
- The admin mirror-target picker (both shells) only offers backends that serve
  no category; already-selected targets stay listed so they can be unchecked.
  A category assigned to a replica still surfaces the server's 400 verbatim.

Existing shared-replica fixtures were rewritten onto dedicated replicas.

* fix(client): a 409 conflict save is recoverable without a page reload

After a 409 the admin storage panel was a dead end. refreshState leaves a
dirty draft's version alone on purpose (audit #7: the background poll must
never float a mid-edit draft past a migration flip it never carried), setDraft
re-attaches that same version to every later edit, and save() resubmits it —
so every retry 409s again and only a full page reload got the operator out,
while the banner told them to "review and save again".

The version block stays. Instead the conflict banner now offers the one action
that can clear it: discard the local edits and reload the saved settings —
a fresh GET applied through applyState, re-deriving the draft (and its
version) and clearing dirty/saveError. Bumping the version in refreshState,
or only in the 409 branch, was rejected: either resurrects audit #7, since a
full-document save would then revert a flip the draft never carried.

storage.saveConflict now points at that action instead of implying a plain
re-save works (all 23 locales). The escape lives in the shared hook, so the
phone panel renders it too.

* fix(client): deflake the queued-migration-waits-out-backfill panel test

FE-ADMIN-STOR-037 seeds a running backfill, so the panel polls GET from
first render — but the stub served the post-save world (files: off-box)
before the PUT had landed, which no real server would do. A poll
resolving while the confirm dialog was open made the confirm-time
candidate recompute see from === to, so nothing was queued and the
migration never POSTed.

The GET stub now stays faithful (pre-save world until the PUT lands,
usage included so candidates keep their object counts), and the test
deterministically forces the once-flaky interleaving: it waits for
polls to land while the dialog is open and asserts the candidate
survives them.

* fix(storage): surface the underlying cause in probe target errors

The admin storage Test probe reported only the driver wrapper's outer
message ("put failed for '…' on 'probe-s3'"), dropping the cause that
says WHY — connection refused vs missing bucket. Flatten the error's
cause chain into the probe result, appending an S3-style code when it
isn't already in the message.

* fix(storage): surface the underlying cause in replica-failure reports

Same defect as the probe fix, one layer over: MirrorDriver's
reportReplicaFailure captured only the wrapper's message, so
replica-failure notifications said "put failed" without the why.
Move the probe's cause-chain flattener into storage.types.ts as
describeError and use it in both call sites.

* fix(storage): strip ContentType from multipart Upload calls (MalformedXML)

Every multipart S3 put carrying a contentType — every backup zip and
photo over the 10MB threshold — failed with 400 MalformedXML. Upstream
@aws-lite/s3's Upload leaks the ContentType param into its final
CompleteMultipartUpload headers ({ ...xml, ...paramHeaders }), clobbering
application/xml, which flips @aws-lite/client's payload serializer to
JSON — so the server receives a JSON parts list where XML belongs.
Confirmed live against MinIO AIStor; any S3 refuses the same.

uploadMonitored now drops ContentType before s3.Upload. Multipart
objects are stored without a content-type, which TREK never reads back
(serving derives Content-Type from the key/caller, never object
metadata); single-request PutObject paths keep sending it.

Also closes the coverage gap that let this ship: the live contract
suite never crossed MULTIPART_THRESHOLD, so s3.Upload had never run
against a real server — it now round-trips a >10MB put carrying a
contentType.

---------

Co-authored-by: Maurice <mauriceboe@icloud.com>
2026-08-22 11:33:55 +02:00
Maurice 3a0604b3f5 feat(config): put the operator-owned surfaces behind TREK_MANAGED (#1946)
* feat(config): put the operator-owned surfaces behind TREK_MANAGED

The switch landed last release with nothing behind it. This is what it turns
off, plus the four fixes that came out of walking every one of these surfaces.

Nothing here changes anything for an install that does not set the flag. That
is not a hope, it is what managed-inert.test.ts asserts: the mode stays off for
any unparseable value, no other variable can turn it on, every settings key
stays writable, and splitManagedKeys hands back the same object rather than a
copy so no write path can quietly diverge on self-hosted installs only.

WHAT THE MODE WITHHOLDS

Fourteen routes carry @ManagedForbidden: key validation, static MCP tokens, JWT
rotation, the OIDC admin pair, restore and auto-backup, plugin sideload and
reload, the Ollama model list and pull, and the SMTP test send. Each names the
reason it is not the instance admin's, and the reason is mandatory.

Eighteen settings keys are filtered instead of refused. SMTP, the OIDC toggles
and the WebAuthn pair share one handler with keys the admin does own, and the
llm_* family is reachable through per-user settings, the instance defaults and
the addon config alike, so a route-level refusal would be both too coarse and
incomplete. A locked key is skipped and reported as managed_keys; the rest of
the save goes through. Refusing the request would take the registration toggles
down with the SMTP block.

GET /api/auth/me/settings stops returning the three API keys in the clear. It
is the only endpoint that decrypts a stored credential back to the caller, which
is fine when the admin pasted it in and wrong when the operator supplied it.

Two multipart routes check in the handler rather than the guard: a guard throws
before the parser drains the body and the client gets an ECONNRESET instead of
the 403 (PROFILE-015). They still carry the marker so the boot gate sees them.

WHAT KEEPS IT FROM DRIFTING

validate-managed-routes.ts refuses to boot on a marking that is not on the
reviewed list, a list entry whose route lost its marking, or a marking with no
reason. The third case has no equivalent in the guard gate next door and is the
one that matters most: a blank reason destroys the only record of why a control
was taken away, and nothing recovers that later. It found two wrong entries in
my own list while I was writing this.

managed-keys.test.ts asserts that every key in ADMIN_SETTINGS_KEYS and
DEFAULTABLE_USER_SETTING_KEYS belongs to exactly one of two sets. A key added to
either list belongs to neither until somebody decides, and that is the failure
this prevents: the lists grow, the mode does not.

PLACES_API_BASE AND PLACES_API_KEY

The nine Places endpoints spell out googleapis.com, so an install that wants
those calls to leave through something of its own has no way to say so. One
variable, substituted where every call already funnels through. A second one
supplies the key, and when it is set the users table is not consulted at all —
a per-user key would route around whatever the endpoint counts, and those
columns are readable by any admin of the instance.

FOUR FIXES THAT ARE NOT ABOUT THE MODE

An admin setting somebody's password now bumps password_version and clears
their MCP tokens. An admin sets a password for one reason, and until now
verifyJwtAndLoadUser kept accepting every cookie the intruder already had: the
one action taken to lock them out was the one action that did not.

The four OIDC provider calls go through the SSRF guard. Three are worse than
they look — the token, userinfo and JWKS URLs come out of the discovery
response, so they are chosen by the other end, not by an admin. safeFetchLlm is
renamed safeFetchAdminConfigured and keeps its old name as an alias, because it
is the variant that allows loopback and LAN: a self-hoster with Authentik on
localhost:9000 must keep working. The token exchange follows no redirect at all,
since one would hand client_secret to a second host.

DELETE /api/admin/users/:id/mfa clears another account's TOTP. The passkey half
has existed since passkeys landed; the TOTP half never did, which left
"somebody on the trip lost their phone" with no answer short of an operator in
the database. Never for the caller themselves.

TREK_PLUGIN_PERMISSIONS=off refuses to boot when the install is centrally
administered. A warning is right for a machine its owner chose to trust; where
the person who set the flag and the people whose data is in the database are
not the same person, fail closed.

TEXTS

Four strings lose the phrase "self-hosted" because they describe somebody
else's software — an Ollama endpoint, an AirTrail instance, a service a plugin
talks to — and read better without it in both modes. The one string that
describes TREK itself keeps it, and the mode picks a second key instead, so
self-hosters lose nothing.

A grep gate in shared/ keeps the canonical locale honest, because the parity
check compares the file set and the top-level keys and never looks at a value:
re-wording a string is invisible to CI today, and so is putting it back.

* feat(maps): supply the map credential with the instance when centrally administered

MAPBOX_ACCESS_TOKEN joins PLACES_API_KEY as an operator-supplied value.
SettingsService injects it when the settings are read, and picks GL as the
provider if the reader is still on Leaflet, since a token is always present.

The key moves from the customer set to the locked set. It is the one locked key
whose value is public by design — a pk.* token has to reach the browser to be
useful — so this is not about secrecy: a per-user save landing on top of the
instance token would only break the map for that user. Both settings fields for
it go away with it.

* fix(auth): count the operator key when reporting whether Google features work

has_maps_key only looked at the users table, but getMapsKey prefers the
PLACES_API_KEY env over anything stored there. On an install that supplies the
key that way and has an empty column, every lookup worked while the client was
told there was no key and hid the search, the autocomplete and the place photos.

Found by walking the running instance rather than the code: the map came up,
the search box did not.

* feat(admin): hide the settings a centrally administered install cannot act on

A control that answers 403 is worse than no control: the AI section rendered the
refusal as "Request failed with status code 403", which tells an admin they broke
something rather than that the decision is not theirs.

So the surfaces go rather than being disabled in place:

- AI parsing and AirTrail leave the addon list, and the AI addon route now
  refuses outright. The endpoint, the model and what a document costs come with
  the instance, so there is no instance decision left — not even whether it runs.
- Immich and Synology Photos leave it too. Both connect to a server the reader
  runs at home, and there is no route from here to one.
- The SMTP block, the admin webhook and ntfy targets, and the passkey RP ID and
  origins follow the same rule: they configure something the operator owns, and
  a wrong RP ID on a shared parent domain reaches past this instance entirely.
- The About tab goes. It is about the project — what TREK is, where to file a
  bug, where to support it — and is aimed at the person running the software.
  The version and the link to the source move to the sidebar footer, on both the
  desktop page and the mobile twin, because AGPL 13 wants that offer wherever
  people reach the software over a network.
- The update check stops asking GitHub and reports nothing available.

Separately, the version footers lose the "self-hosted" suffix everywhere, not
just here. It read as a label on the software rather than on the install, and
the version alone says what it needs to.

* feat(config): refuse to boot centrally administered without an at-rest key

Measured on a running instance rather than assumed: with ENCRYPTION_KEY unset,
the at-rest key falls back to a file in the data volume, and backupService
bundles that file into every archive it builds. The archive is downloadable by
any instance admin, so the encryption ships with its own key in the same zip.

The skip in backupService already handles this — it only omits the file when the
key came from the env — but nothing made sure the env actually carried one.
Provisioning sets it; forgetting it was silent.

validateEnvAtBoot now carries a second kind of check next to the schema: cross-
field rules that only apply with the flag set, for combinations where every value
is individually valid and the result is wrong. Empty without the flag, so an
install that never asked to be centrally administered cannot meet it.

* feat(client): add an attachment point for install-specific screens

Two empty exports on the client, an empty module on the server, and the four
places that read them. Nothing here adds a route, a nav entry or an endpoint —
the arrays are empty and the module contributes nothing, which managed-ext.test
pins so it stays that way.

Read unconditionally, never behind a flag. On the server a conditional import
would tie the module graph to an environment variable, and the graph is built
once per app while the flag is read live; on the client an empty array costs
nothing to spread, so every build walks the same path instead of one build
walking a path nobody tests.

Anything attached here follows the same rules as the rest of src/nest and
src/components: a real module with a controller and a provider, bodies through
the Zod pipe, routes under the global guards, tests.

* test(shared): move the wording gate out of src so shared typechecks again

CI installs shared with its own dependencies only (npm ci --workspace shared),
so there is no @types/node and the `node:fs` import inside src/ failed to
compile. The tests passed the whole time — vitest supplies the module — which is
why this only showed up in the typecheck step.

The walk moves to scripts/, next to i18n-parity.mjs, and the spec imports it the
same way that one does. It also runs on its own now
(`node scripts/i18n-selfhost-wording.mjs`), which is what you want when CI names
a line you have to go find.

* test(server): repair the call sites typecheck:tests caught

tests/ sits outside the build tsconfig, so none of this showed up in a green
test run — which is exactly the gap `npm run typecheck:tests` exists to close.
Three kinds, all mine:

- Seven `import type` paths for RuntimeEnvService carried one `../` too many.
  Type-only imports are erased, so vitest never resolved them and nothing failed.
- VersionCheckJob and PluginsController grew a constructor parameter and eleven
  hand-wired `new` calls kept passing the old count. The argument arrived as
  undefined and no assertion happened to read it.
- ConditionContext grew `managed`, and the shared context object in the
  conditions suite did not.

The condition also had no test of its own, so it has one now: both directions,
since the registry uses `is: false` to keep a notice away and a notice that only
belongs on a centrally administered install would use `is: true`.
2026-08-15 16:42:27 +02:00
jubnl cafff003a3 refactor(server): move instance-URL resolution to app-config 2026-07-28 20:21:09 +02:00
jubnl e162da6dbe fix(server): resolve zh-TW default language, close lint gap on bracket env access 2026-07-25 16:19:15 +02:00
jubnl 3b31f4256b refactor(server): move language/session parsing into app-config, config.ts becomes keys module 2026-07-25 16:01:42 +02:00
jubnl 48ba9092c3 feat(server): add app-config core with fail-fast env validation 2026-07-25 15:03:27 +02:00