TREK/wiki/Admin-Storage.md

# Admin: Storage

The **Storage** tab (Admin Panel > Configuration > Storage) controls where TREK stores each
kind of content: which storage backends exist, which content category writes
to which backend, and whether writes are replicated to additional backends.

![Admin Storage tab: the Health strip with no replica failures, the two built-in backends with their usage and Test and edit buttons, and the category cards with their backend](assets/AdminStorage.png)

## Backends

A backend is a named place bytes can live. Two exist out of the box:

| Backend | What it is |
|---------|------------|
| `uploads-local` | The local `uploads/` directory (the `/app/uploads` volume in Docker) |
| `backups-local` | The local `data/backups` directory |

You can **edit** a built-in to relocate its root directory, and **add**
backends of type:

- **Local**: a directory on a disk or mount reachable by the server.
- **S3**: any S3-compatible service (AWS S3, Cloudflare R2, Backblaze B2,
  Garage, MinIO/AIStor). Endpoint URL, bucket, credentials, and optionally
  region, key prefix, retries, and timeout. For self-hosted endpoints, use an
  IP address or `localhost` in the URL unless your server is configured for
  virtual-hosted bucket addressing.

If `TREK_PLACE_PHOTO_DIR` is set, a read-only `place-photos-local` backend
appears too; it is defined by that environment variable, not by this panel.

### Replication (Mirror targets)

Edit any backend and tick one or more **Mirror targets**: every write to that
backend is then also copied to each selected target. Removing all targets
turns replication off again.

- A backend can serve as a **mirror target or as a category target, not
  both**: only backends no category is stored on are offered in the list, and
  saving a configuration that mixes the two is refused. A sync makes a target
  match its backend (see below), so anything else stored there would be
  deleted.
- Writes go to the backend itself first (it stays the source of truth), then
  to each target in order. A slow or unreachable target slows every upload of
  every category on that backend, ideal for backups, worth weighing before
  replicating hot categories like trip documents.
- Replica failures **never fail the original request**. They are recorded and
  shown in the **Health** strip at the top of the tab, with the backend, the
  object, the operation, and how long ago it happened. An empty strip means
  every replicated write landed. Replica failures also notify admins, in-app
  and over the configured webhook/ntfy/email channels (first failure
  immediately, then at most one summary per backend per hour). Preferences
  live in the admin Notifications tab.
- Replication starts with the next write. To copy objects that existed before
  you added a target, press **Sync now** on the backend's row; one sync runs
  at a time, with progress and a cancel button; objects already present with
  matching sizes are skipped, and per-object failures land in the Health
  strip without stopping the run. A sync makes the target **match** the
  backend, not just catch up: it also deletes target objects that no longer
  exist on the backend; it's not a trash can, so run it only when you want
  the target pruned to match, and stray deletions land in the Health strip
  like any other replica failure.

### Test

Every backend row has a **Test** button: the server writes, checks, and
deletes a small probe object (`trek-probe/…`) on the backend (for a
replicated backend, on the backend itself and each target individually) and
reports per-target results. "Connection OK" means credentials, bucket, and
reachability all check out. Targets must be **saved** before Test can probe
them; a just-added, unsaved backend used as a target reports an error until
you save.

## Categories

Every kind of content TREK stores belongs to one of eight categories, each
assigned to exactly one backend:

| Category | Id | What it stores |
|----------|----|----------------|
| Trip documents | `files` | File attachments uploaded to trips, tickets, PDFs, booking confirmations, and files shared in trip chat |
| Journey photos | `journey` | Photos and thumbnails attached to journey entries |
| Cover images | `covers` | Trip and collection cover images, including covers fetched from Unsplash |
| Profile pictures | `avatars` | User account profile pictures |
| Place images | `places` | Images attached to places and collection places, uploaded or imported |
| Google photo cache | `photos-google` | Cached copies of Google Places photos, re-fetchable, safe to lose |
| TREK photo cache | `photos-trek` | Cached photos from the TREK photo service used by Memories, re-fetchable, safe to lose |
| Backups | `backups` | Server backup archives created by the Backup panel or schedule |

Reassigning a category changes where **new** objects go. If the category
already holds objects, the panel asks first: **Move existing objects** copies
everything to the new backend, switches the category, then sweeps any writes
that raced the copy; **Just route new writes** keeps the previous behavior:
new objects go to the new backend, existing ones stay put and keep being
served from wherever they already are. Replicating the Google photo cache or
Place images is flagged as not recommended; both hold content that is
re-fetchable or provider-derived.

The legacy `/uploads/photos` directory written by older TREK versions is not
a category: its files are still served and included in backups, but nothing
writes there anymore and it cannot be reassigned.

### Moving a category's objects

A category migration shares its one-at-a-time slot with mirror syncs (see
*Replication* above); starting a move while a sync (or another move) is
running is refused, and vice versa. Progress (done/total, copied/skipped/
failed) shows inline with a cancel button.

Cancelling before the category switches is completely safe: nothing has
changed yet, the copy just stops. A failed copy behaves the same way; the
category is never switched if any object failed to copy. Once the switch has
happened, the delta sweep (catching writes that landed on the old backend
while the copy was running) always runs to completion, even if you cancel.

The old objects are **not** deleted when a move finishes; that's what the
reclaimable line means (`N objects (size) remain on <backend>, reclaim
manually`). It's left behind on purpose so you can confirm the new backend
looks right before removing it yourself.

## Usage

The Backends section shows per-backend and per-category object counts and
sizes, scanned nightly (04:15 server time) and on demand via **Refresh**. The
stamp shows when the numbers were computed; reassigning categories does not
retroactively move them until the next scan. The uploads backend's total
includes the legacy photo library.

## Unreferenced files

Uploaded files can outlive what used them. When you delete or replace
something that shows a file (a trip or list cover, a place image, a journal
photo, an avatar, a trip file), TREK deletes that file right away, unless
something else still shows it: a copied trip, a journey that took over the
trip cover, a saved place in a list. Whatever is left over after that (files of
deleted trips and users, files from before this check existed) is found by a
check every night at 04:05 server time. It looks at the covers, place images,
avatars, trip files and journal photos for files that nothing in TREK names
any more, and the **Unreferenced files** section shows what it found:

- **Can be freed**: unused on every check for at least a week, so the nightly
  check deletes them once deletion is on.
- **Unused for less than a week**: these get their week first. A file that is
  used again in the meantime (an undo, a copied trip) starts over.
- **Freed by the last nightly cleanup**: what the last nightly check with deletion on
  deleted. A check you start by hand never deletes, so it never changes this
  figure.

**Delete unreferenced files** is off after install and upgrade, so the check
only reports. Switched on, the nightly check deletes a file once it has been
unused for a week and one more look right before the delete still finds
nothing that uses it. **Check now** starts a check in the background that only
reports, whatever the switch says; the section shows that it is running and
updates when it is done. Switching deletion on or off, starting a check,
confirming a category (below) and every nightly check that deleted files are
recorded in the [Audit-Log](Audit-Log); the nightly entry names how many files
went per category.

When more than a fifth of a category's files look unused, the check marks it
and deletes nothing there until you confirm it once with **Confirm** next to
the category. The confirmation covers the number of files shown at that
moment: if the number later grows past it, while still more than a fifth, the
category waits for you again. A long-running instance with many old leftovers
usually needs this once.

A category is skipped for that night, with the reason shown next to it, when:

- not every part of TREK could be asked about it;
- its files share a location with another category's;
- another TREK installation keeps its files in the same place (see below), or
  TREK could not leave its own mark there;
- listing it or asking about it failed. What the earlier nights found stays,
  so one failure does not restart anyone's week;
- deleting failed on the storage backend: the check then stops deleting in
  that category for the night.

**Several installations on one bucket or folder.** Each check leaves a small
file named `trek-owner-<id>` at the top of every category it looks at, naming
this installation. When it finds such a file of another installation there,
it leaves that category alone, and so does the other installation once it
sees this one's file. Backups leave these files out. If you moved an
installation and a stale `trek-owner-<id>` of its old self remains, delete
that one file to let the check work there again.

**Restoring a backup** switches deletion off and forgets what earlier checks
found and what you confirmed, because the restored data may come from another
installation. Switch deletion on again once you have looked at the next
report.

Caches (place photos), backups, assistant attachments and journal thumbnails
are cleaned up by their own routines and are never touched here. No check runs
on a demo instance, none while a sync, a move or a restore is running, and a
running check deletes nothing while a backup is being written or once you
switch deletion off.

## Secrets and encryption

Backend credentials (the S3 secret access key) are stored encrypted. Setting
the `ENCRYPTION_KEY` environment variable explicitly is recommended; without
it the implicit key persisted in the data directory is used, which works but
rides inside backup archives (someone holding a backup can decrypt the
secrets it contains). If you later move from the implicit key to an explicit
one, follow [[Encryption Key Rotation|Encryption-Key-Rotation]]; setting a
new key directly would orphan already-stored secrets. Saved secrets display
as a masked placeholder; leaving the mask untouched when editing keeps the
stored value.

## Provisioning at first boot (seed file)

For scripted deployments, place a `storage-config.json` in the data directory
(in Docker: mount it at `/app/data/storage-config.json`). It is imported
exactly once, on the first boot that has no stored storage configuration,
then ignored (a log line says so). An invalid file aborts boot with the exact
validation error.

```jsonc
// storage-config.json: secrets may be plaintext (encrypted on import)
// or already-encrypted enc:v1: values.
{
  "backends": [
    { "name": "off-site", "type": "s3", "options": {
      "endpoint": "https://s3.example.com", "bucket": "trek",
      "accessKeyId": "...", "secretAccessKey": "..." } },
    { "name": "backups-mirror", "type": "mirror", "options": {
      "primary": "backups-local", "replicas": ["off-site"] } }
  ],
  "categories": { "backups": "backups-mirror" }
}
```

The panel presents this exact setup as **Mirror targets** on `backups-local`.

```yaml
# docker-compose: add under the trek service's volumes
      - ./storage-config.json:/app/data/storage-config.json:ro
```

To reset storage configuration (or re-import a seed file): stop the server,
`sqlite3 data/travel.db "DELETE FROM app_settings WHERE key LIKE 'storage.%';"`,
start it again; it boots on the built-in defaults, or re-imports the seed
file if present.

**Restore chicken-and-egg:** backups include the storage configuration. If
your only backup lives on S3 and the credentials are inside it: start a fresh
instance, enter the S3 credentials here (or mount a seed file), then restore
from the Backup panel.

The phone admin has the same functions in its **Storage** section.

## Related pages

- [[Backups|Backups]]: what a backup contains, restore procedure
- [[Environment Variables|Environment-Variables]]
- [[Encryption Key Rotation|Encryption-Key-Rotation]]
- [[Admin Panel Overview|Admin-Panel-Overview]]
- [[Troubleshooting|Troubleshooting]]