mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-11-20 02:56:10 +00:00
docs(db): moving an existing install to Postgres, and the refusal of an empty database beside one
This commit is contained in:
@@ -323,7 +323,8 @@ for the full walkthrough.
|
||||
|
||||
TREK stores its data in SQLite by default. Set `TREK_DB_URL` to run it on a PostgreSQL 14 to 18 database instead. The
|
||||
[`docker-compose.postgres.yml`](docker-compose.postgres.yml) example starts one beside TREK, and the
|
||||
[Postgres guide](https://github.com/liketrek/TREK/wiki/Database-Postgres) covers the requirements and the backups.
|
||||
[Postgres guide](https://github.com/liketrek/TREK/wiki/Database-Postgres) covers the requirements, the backups and
|
||||
how to move an existing SQLite install to Postgres with the command TREK ships for it.
|
||||
|
||||
<br />
|
||||
|
||||
|
||||
+2
-1
File diff suppressed because one or more lines are too long
@@ -106,9 +106,55 @@ A database migrated by a newer TREK is refused, and the log says to run that ver
|
||||
|
||||
**Start one TREK process for the first boot and for every upgrade.** Two processes that start at the same time on the same new database are not coordinated. The second one fails its boot, because the first one's schema transaction wins. Once the database is up to date and running, a single container is the supported setup. See [Limitations](#limitations).
|
||||
|
||||
### Moving an existing SQLite install
|
||||
### An existing SQLite install beside an empty database
|
||||
|
||||
This first Postgres release has no tool that copies a SQLite install into Postgres. A SQLite backup cannot be restored into a Postgres database, and a Postgres backup cannot be restored into SQLite. Moving an existing install to Postgres means starting a new install on Postgres and recreating the data there, or keeping the SQLite install running.
|
||||
When `TREK_DB_URL` points at a database that holds no TREK data while the data directory still holds a `travel.db` with accounts, TREK refuses to start. Starting would bring up a new, empty TREK next to your data, which looks as if everything was lost. The log names the file and the move command. Move the install as described in [Moving an existing install to Postgres](#moving-an-existing-install-to-postgres), or unset `TREK_DB_URL` to keep running on SQLite. To start on the empty database on purpose, set `TREK_DB_IGNORE_SQLITE_FILE=true`; TREK then logs a warning and starts.
|
||||
|
||||
A SQLite backup cannot be restored into a Postgres database, and a Postgres backup cannot be restored into SQLite. The move below is the way from one to the other.
|
||||
|
||||
## Moving an existing install to Postgres
|
||||
|
||||
TREK ships a command that copies a SQLite install into an empty Postgres database: every table, the data of plugins that keep their own database, and the id sequences. It checks the copy before it keeps it and never changes the SQLite file, so the way back is to start TREK on SQLite again.
|
||||
|
||||
### What the command does
|
||||
|
||||
1. It checks that the SQLite file is at the migration level of the TREK version that runs the command. An older file is refused with the advice to start this version on it once, which migrates it; a file a newer TREK migrated is refused too.
|
||||
2. It checks that every row's references point at a row that exists, since Postgres enforces them, and lists the rows that do not.
|
||||
3. It refuses a Postgres database that holds any table or plugin schema, so it never mixes two installs.
|
||||
4. It creates the schema exactly as a first start on an empty database does.
|
||||
5. In one transaction it copies every table in the order of their references, converts each value to its Postgres column type, moves the id sequences past the copied ids, and reads every table back to compare its row count and a checksum over all its rows with the SQLite file. A value Postgres cannot store exactly (text in a number column, a NUL character in text) stops the copy and is listed by table, row and column.
|
||||
6. Only a copy that matches is committed. If anything fails, the transaction is rolled back and the schema the command created is dropped again, so the database is empty as before.
|
||||
|
||||
It prints a summary: the tables and rows it copied, the plugin schemas it filled, and anything it could not carry over.
|
||||
|
||||
**Uploads are not moved and need not be.** Files, covers, photos and avatars stay where the storage driver keeps them (the `uploads` directory, or your S3 bucket), and the rows the command copies point at them as before. The key files in the data directory (`.encryption_key`, `.jwt_secret`) stay where they are too. Keep the same `ENCRYPTION_KEY`, or the same data directory, for the Postgres install: the stored secrets are copied as they are, encrypted with that key.
|
||||
|
||||
**Plugin data.** A plugin's `plugin.db` in `data/plugins-data/<plugin id>/` goes into the plugin's schema `trek_plugin_<plugin id>` (see [Plugin data](#plugin-data)). Each table is created from the SQLite one: the column types follow SQLite's type rules, with NOT NULL, the primary key, an INTEGER PRIMARY KEY as an identity column, defaults that have a Postgres spelling, and indexes over plain columns. The plugin's record of the migrations it ran is copied, so the plugin does not run them again. Foreign keys between a plugin's tables, its views and triggers, and partial or expression indexes are not created; the summary lists them. The plugin's own SQL has to be written for Postgres for it to keep working, as for any plugin on Postgres.
|
||||
|
||||
### Steps with Docker Compose
|
||||
|
||||
1. **Update first.** Run the TREK version you want to keep, on SQLite, and let it start once, so the SQLite file is at that version's level. The command must come from the same image.
|
||||
2. **Take a backup.** Create one under **Admin > Backup** and download it. Then stop TREK and copy the data directory somewhere safe as well.
|
||||
3. **Stop TREK**: `docker compose down`. The SQLite file must not change while it is copied.
|
||||
4. **Prepare the database.** Switch to [`docker-compose.postgres.yml`](https://github.com/liketrek/TREK/blob/main/docker-compose.postgres.yml), set `POSTGRES_ADMIN_PASSWORD` and `POSTGRES_PASSWORD` in `.env` as described in [Compose example](#compose-example), keep the same `./data` and `./uploads` mounts and the same `ENCRYPTION_KEY`, and start only the database: `docker compose -f docker-compose.postgres.yml up -d db`.
|
||||
5. **Run the move**:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.postgres.yml run --rm --user node -w /app/server app \
|
||||
node --require tsconfig-paths/register dist/move-to-postgres.js
|
||||
```
|
||||
|
||||
It reads `data/travel.db` and `data/plugins-data`, and writes into the database `TREK_DB_URL` names. For a SQLite file elsewhere in the container, add `--from=/app/data/<file>`. A refusal or an error says what to fix and leaves the Postgres database empty, so you can run the command again.
|
||||
6. **Start TREK**: `docker compose -f docker-compose.postgres.yml up -d`.
|
||||
7. **Check**: the log reports the Postgres engine and no schema creation. Sign in, open a few trips, open a file and a photo, and look at your plugins.
|
||||
|
||||
Outside Docker, run `npm run db:move-to-postgres` in the server directory, with `TREK_DB_URL` set in the environment or in the server's `.env` file, and `--from=<file>` when the SQLite file is not the server's `data/travel.db`.
|
||||
|
||||
### Going back
|
||||
|
||||
The command never changes `travel.db`. To return to SQLite, stop TREK and start it again with the SQLite compose file, without `TREK_DB_URL`: it runs on the file as it was before the move. What was written on Postgres in the meantime is not in it. To try the move again, drop the Postgres database and create it empty, or use a new one.
|
||||
|
||||
The `travel.db` that stays in the data directory does not stop a Postgres install that holds your data. Keep it as a copy, or move it out of the data directory once you are sure.
|
||||
|
||||
## Upgrades
|
||||
|
||||
@@ -154,7 +200,7 @@ A plugin with the `db:own` permission keeps its own database. On SQLite that is
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No migration from SQLite to Postgres** in this first release. Start a new install on Postgres, or keep the SQLite install.
|
||||
- **Moving from SQLite goes one way.** The [move command](#moving-an-existing-install-to-postgres) copies a SQLite install into an empty Postgres database. The way back is to start TREK on the untouched SQLite file again; nothing copies Postgres data back into SQLite.
|
||||
- **No more than one TREK replica.** TREK keeps state in memory: rate limits, pending one-time codes, the permission cache and the WebSocket connections. Each process has its own copy, so two replicas would disagree with each other. Keep `replicas: 1` in the Helm chart and one `trek` container. Postgres does not change that.
|
||||
- **Two processes booting on one database at once are not coordinated.** Start one process for the first boot and for each upgrade, then continue with one.
|
||||
- **PostgreSQL 14 to 18 only.** A newer server is refused until a TREK release supports it, because the image's client tools would not be able to back it up.
|
||||
|
||||
@@ -513,6 +513,7 @@ through environment variables. `TREK_PLACE_PHOTO_DIR` below is unaffected.
|
||||
| `TREK_DB_FILE` | Explicit path of the SQLite database file. Unset, TREK uses `travel.db` in the data directory. Not used with `TREK_DB_URL`. | unset |
|
||||
| `TREK_DB_JOURNAL_MODE` | SQLite [journal mode](https://sqlite.org/pragma.html#pragma_journal_mode): `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, `WAL` or `OFF`. Set `DELETE` when the data directory lives on network storage — see below. Values SQLite doesn't know log a warning and fall back. | `WAL` |
|
||||
| `TREK_DB_PRE_MIGRATE_SNAPSHOT` | Copy the database to `pre-migrate-<schema>-<time>.db` next to it before a startup applies pending migrations, so the previous image can be started on that copy again. On PostgreSQL the copy is a `pg_dump` file, `pre-migrate-<schema>-<time>.pgdump`, and needs the client tools (see `TREK_DB_PG_BIN_DIR`). If the copy cannot be written the startup refuses rather than migrate without one. Set `false` only to get past that when you cannot free space and have a backup of your own. See [Updating](Updating#database-migrations). | `true` |
|
||||
| `TREK_DB_IGNORE_SQLITE_FILE` | With `TREK_DB_URL` set, TREK refuses to start on a Postgres database that holds no TREK data while the data directory holds a `travel.db` with accounts, because it would come up as a new, empty instance beside it. `true` starts anyway, with a warning. Move the install instead with the command in [Database: Postgres](Database-Postgres#moving-an-existing-install-to-postgres). Unused on SQLite. | `false` |
|
||||
| `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` | How many of those copies stay in the data directory, newest first (1 to 100). | `3` |
|
||||
| `TREK_DB_SYNCHRONOUS` | SQLite [synchronous](https://sqlite.org/pragma.html#pragma_synchronous) level: `OFF`, `NORMAL`, `FULL` or `EXTRA`. The default follows the journal mode — `NORMAL` under WAL (what SQLite itself uses there), `FULL` otherwise, because a rollback journal at `NORMAL` can lose committed transactions on a power cut. | `NORMAL` / `FULL` |
|
||||
|
||||
|
||||
@@ -194,6 +194,10 @@ Two things change with 4.3.1. Neither needs a manual step on Docker, but they ar
|
||||
- **Remove any `APP_VERSION` override.** The version now comes from the image itself. A leftover `APP_VERSION=` line in your compose file, `.env` or Portainer stack only gets in the way.
|
||||
- **Tours asks the TREK Places API first.** The grades, surfaces and kinds of way under a planned tour and the horse routes around the map are read from `https://places.liketrek.com`, and the Overpass mirrors are asked only when it has no answer. An `OVERPASS_URL` pointed at an internal Overpass therefore no longer keeps these lookups inside your network: set `TREK_PLACES_ENABLED=false`, or point `TREK_PLACES_URL` at a copy you run yourself. [TREK-Places-API](TREK-Places-API#what-leaves-your-instance) lists what is sent.
|
||||
|
||||
## Moving to PostgreSQL
|
||||
|
||||
An update never switches the database. To move an existing SQLite install to PostgreSQL, update first and let TREK start once on SQLite, then follow [Database: Postgres: Moving an existing install to Postgres](Database-Postgres#moving-an-existing-install-to-postgres). The move copies the database only; the uploads stay where they are. Setting `TREK_DB_URL` alone, without the move, is refused while `data/travel.db` holds your data, so an update cannot leave you on an empty instance by accident.
|
||||
|
||||
## Encryption Key Note
|
||||
|
||||
If you are upgrading from a version that predates the dedicated `ENCRYPTION_KEY` (i.e. you have no `ENCRYPTION_KEY` environment variable set), TREK automatically falls back to `./data/.jwt_secret` on startup and immediately promotes it to `./data/.encryption_key`. No manual steps are required — the transition is handled at first boot after the upgrade.
|
||||
|
||||
Reference in New Issue
Block a user