mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-18 21:16:47 +00:00
feat(db): copy the database before a boot migrates it
Migrations only go forward, and since #2610 an older image refuses a database a newer one migrated. Going back after a bad update therefore needs a copy from before it, and the only copies were backups the admin takes by hand or auto-backup, which is off by default. The first boot on 4.4 migrates every 4.3.x install through the legacy baseline and the new migrations without either. Before applying pending migrations to a file database that holds anything, the boot now writes a VACUUM INTO copy next to it, named pre-migrate-<schema>-<time>.db (legacy-<schema_version> for an install the positional runner left, otherwise the last recorded migration), and keeps the newest TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP (3). The copy is written under a temporary name and renamed, so a crash never leaves a truncated file that looks good. If it cannot be written the boot refuses before anything is migrated; TREK_DB_PRE_MIGRATE_SNAPSHOT=false is the documented way past that. The newer-database refusal names the copy taken right before the newer release migrated past this one. A fresh install, a restart with nothing pending, an in-memory database and a restore's re-bootstrap (the archive is the copy) take none, and NODE_ENV=test leaves it off unless a suite sets the switch.
This commit is contained in:
Maurice
@@ -48,6 +48,8 @@ DEMO_MODE=false # Demo mode - resets data hourly
|
||||
# BACKUP_UPLOAD_LIMIT_MB=500 # Max size (MB) of a backup archive you can upload when restoring. Raise it if your backup exceeds 500 MB. If you sit behind a reverse proxy, raise its upload limit too (e.g. nginx client_max_body_size).
|
||||
|
||||
# TREK_DB_JOURNAL_MODE=WAL # SQLite journal mode (DELETE, TRUNCATE, PERSIST, MEMORY, WAL, OFF). WAL is fastest and the right choice on local disk. Set DELETE if the data directory lives on network storage (Azure App Service, an SMB/NFS volume on a NAS) — WAL coordinates writers through a shared-memory file that those filesystems do not implement reliably. The mode is stored inside the database file, so it survives a restart; changing this applies on the next boot.
|
||||
# TREK_DB_PRE_MIGRATE_SNAPSHOT=true # Copy the database to data/pre-migrate-<schema>-<time>.db before a startup migrates it, so the previous image can run on the copy again. A copy that cannot be written stops the startup; false migrates without one.
|
||||
# TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP=3 # How many pre-migrate copies to keep, newest first (1 to 100). (default: 3)
|
||||
# TREK_DB_SYNCHRONOUS=NORMAL # How hard SQLite flushes to disk (OFF, NORMAL, FULL, EXTRA). Defaults to NORMAL under WAL and FULL under any other journal mode — leave it alone unless you know why you want otherwise. Both variables fall back to their default (with a warning in the log) if the value is not a mode SQLite knows.
|
||||
|
||||
# MCP_RATE_LIMIT=300 # Max MCP API requests per user per minute (default: 300)
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
|
||||
- **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator** — it bypasses that gate.
|
||||
- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest** — it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/` — run `gen:plugin-facts` after touching them and commit the output. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together.
|
||||
- **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly.
|
||||
- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only opens the connection and applies pragmas — it does not build a schema. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. `db/orm-driver.ts` binds the ORM to that same connection instead of opening a second one — with `:memory:` under test, a second connection is a second, empty database. Every read and write goes through a repository in `db/repositories/` (each extends the shared `TrekRepository` base in `db/repositories/_shared/`, which validates the request context before delegating to MikroORM) and, for multi-statement writes, `UnitOfWork.transactional(...)` (`nest/database/unit-of-work.ts`, provided by the global `OrmModule`) — there is no `DatabaseService`/`DatabaseModule` to inject any more. Entrypoints outside an HTTP request — an MCP tool call, a plugin RPC dispatch, a WebSocket message, a cron job, boot-time seeding, a backup restore — open one explicitly with `withRequestContext()` (`nest/database/request-context.ts`), since only the pathless `mikroOrmRequestContext` middleware `buildApp()` mounts (with `@mikro-orm/nestjs`'s own `registerRequestContext` turned off in `app.module.ts`, UPLOADS-P16) forks an `EntityManager` per HTTP request. `NODE_ENV=test` ⇒ isolated `:memory:` DB per vitest worker, opened from a snapshot `tests/global-setup.ts` migrates once per run; `TREK_DB_FILE` ⇒ explicit file DB (Playwright harness).
|
||||
- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only opens the connection and applies pragmas — it does not build a schema. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate-<schema>-<time>.db`, `<schema>` being `legacy-<schema_version>` or the last recorded migration's timestamp; the newest `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` stay) and the newer-database refusal names the matching copy; a copy that cannot be written refuses the boot unless `TREK_DB_PRE_MIGRATE_SNAPSHOT=false`. It is off under `NODE_ENV=test` unless a suite sets that variable to true, and a restore's re-bootstrap skips it (the archive is the copy). `db/orm-driver.ts` binds the ORM to that same connection instead of opening a second one — with `:memory:` under test, a second connection is a second, empty database. Every read and write goes through a repository in `db/repositories/` (each extends the shared `TrekRepository` base in `db/repositories/_shared/`, which validates the request context before delegating to MikroORM) and, for multi-statement writes, `UnitOfWork.transactional(...)` (`nest/database/unit-of-work.ts`, provided by the global `OrmModule`) — there is no `DatabaseService`/`DatabaseModule` to inject any more. Entrypoints outside an HTTP request — an MCP tool call, a plugin RPC dispatch, a WebSocket message, a cron job, boot-time seeding, a backup restore — open one explicitly with `withRequestContext()` (`nest/database/request-context.ts`), since only the pathless `mikroOrmRequestContext` middleware `buildApp()` mounts (with `@mikro-orm/nestjs`'s own `registerRequestContext` turned off in `app.module.ts`, UPLOADS-P16) forks an `EntityManager` per HTTP request. `NODE_ENV=test` ⇒ isolated `:memory:` DB per vitest worker, opened from a snapshot `tests/global-setup.ts` migrates once per run; `TREK_DB_FILE` ⇒ explicit file DB (Playwright harness).
|
||||
|
||||
## Rules for new code
|
||||
|
||||
|
||||
@@ -85,6 +85,8 @@ TREK_MANAGED, PLACES_API_BASE, PLACES_API_KEY, AMAP_API_BASE, AMAP_API_KEY, AMAP
|
||||
TREK_PLUGINS_ENABLED / _DEV_LINK / _IGNORE_TREK_RANGE / _DIR / _DATA_DIR / TREK_PLUGIN_PERMISSIONS,
|
||||
OIDC_*, SMTP_*, FORCE_HTTPS, COOKIE_SECURE, HSTS_INCLUDE_SUBDOMAINS,
|
||||
ALLOWED_ORIGINS, UNSPLASH_ACCESS_KEY, WEBAUTHN_*, TZ, ADMIN_EMAIL,
|
||||
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) (read when a migration run starts; the
|
||||
legacy-upgrade suites set it per file),
|
||||
ADMIN_PASSWORD, IDEMPOTENCY_TTL_SECONDS, MCP_RATE_LIMIT (request-path check),
|
||||
LLM_TIMEOUT_MS, NOMINATIM_URL, VAPID_* (resolved per use by the Web Push key
|
||||
service, which the notification suites build without the container).
|
||||
|
||||
@@ -335,6 +335,14 @@ export function deriveDb(raw: RawEnv) {
|
||||
synchronous: durability.synchronous,
|
||||
/** Complaints about unusable values; derivation stays side-effect free, so db/durability.ts logs them when it opens the file. */
|
||||
durabilityWarnings: durability.warnings,
|
||||
/**
|
||||
* Snapshot the database before a boot migrates it. Undefined when unset: the
|
||||
* caller then decides (on, except under NODE_ENV=test). false is the escape
|
||||
* hatch for a data dir that cannot hold a second copy.
|
||||
*/
|
||||
preMigrateSnapshot: parseBool(raw.TREK_DB_PRE_MIGRATE_SNAPSHOT),
|
||||
/** How many pre-migration snapshots stay in the data dir, newest first. */
|
||||
preMigrateSnapshotKeep: Math.min(positiveIntOr(raw.TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP, 3), 100),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -158,6 +158,12 @@ export const envSchema = z.object({
|
||||
// this schema at all. Resolution lives in parsers.resolveDurability().
|
||||
TREK_DB_JOURNAL_MODE: anyString,
|
||||
TREK_DB_SYNCHRONOUS: anyString,
|
||||
// The copy of the database taken before a boot applies pending migrations.
|
||||
// Unlike the two pragmas a typo here refuses the boot: the switch exists to
|
||||
// turn a safety net off, and a misspelt "off" must not leave it on silently
|
||||
// or a misspelt "on" turn it off.
|
||||
TREK_DB_PRE_MIGRATE_SNAPSHOT: boolStr,
|
||||
TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: integer(1, 100),
|
||||
TREK_WIKI_DIR: anyString,
|
||||
TREK_PLACE_PHOTO_DIR: anyString,
|
||||
BACKUP_UPLOAD_LIMIT_MB: positiveNumber,
|
||||
|
||||
@@ -38,7 +38,39 @@ import { knownMigrationNames, unknownMigrations } from './known-migrations';
|
||||
*/
|
||||
const STEP_MARKER = /Legacy migration step (\d+)\b/g;
|
||||
|
||||
const REFUSAL = '[DB] Refusing to boot:';
|
||||
export const REFUSAL = '[DB] Refusing to boot:';
|
||||
|
||||
/**
|
||||
* What the boot does around a migration run (`pre-migrate-snapshot.ts` in
|
||||
* production, nothing in the unit tests and on a restore).
|
||||
*/
|
||||
export interface MigrationSafetyNet {
|
||||
/**
|
||||
* Runs after the plan is known and before anything is written, only when
|
||||
* something is pending. `from` labels the schema the database holds: the
|
||||
* last recorded migration's timestamp, `legacy-<schema_version>`, or null
|
||||
* when nothing is recorded at all. Throwing refuses the boot.
|
||||
*/
|
||||
beforeMigrate(from: string | null): Promise<void>;
|
||||
/** A sentence pointing at a copy this release can run, for the newer-database refusal. */
|
||||
restoreHint(thisRelease: string): Promise<string | null>;
|
||||
}
|
||||
|
||||
export const NO_SAFETY_NET: MigrationSafetyNet = {
|
||||
beforeMigrate: async () => {},
|
||||
restoreHint: async () => null,
|
||||
};
|
||||
|
||||
/** The part of a migration name that orders it: what a schema state is labelled by. */
|
||||
export function migrationLabel(name: string): string {
|
||||
return /^Migration(\d+)_/.exec(name)?.[1] ?? name;
|
||||
}
|
||||
|
||||
function latest(names: Iterable<string>): string | null {
|
||||
let last: string | null = null;
|
||||
for (const name of names) if (last === null || name > last) last = name;
|
||||
return last;
|
||||
}
|
||||
|
||||
export interface LegacyStepMap {
|
||||
/** The migrations standing for `createTables()`: everything that sorts ahead of step 1. */
|
||||
@@ -184,12 +216,23 @@ export async function migrateToHead(
|
||||
migrator: Migrator,
|
||||
readSource: ReadSource = readFromDisk,
|
||||
known: Set<string> = knownMigrationNames(),
|
||||
safetyNet: MigrationSafetyNet = NO_SAFETY_NET,
|
||||
): Promise<void> {
|
||||
const baselined = await planLegacyBaseline(connection, migrator, readSource);
|
||||
refuseNewerDatabase((await migrator.getExecuted()).map((row) => row.name), known);
|
||||
const executed = (await migrator.getExecuted()).map((row) => row.name);
|
||||
if (unknownMigrations(executed, known).length > 0) {
|
||||
const thisRelease = latest(known);
|
||||
refuseNewerDatabase(executed, known, thisRelease ? await safetyNet.restoreHint(migrationLabel(thisRelease)) : null);
|
||||
}
|
||||
const already = new Set(baselined);
|
||||
const pending = (await migrator.getPending()).filter((migration) => !already.has(migration.name));
|
||||
if (pending.length === 0 && baselined.length === 0) return;
|
||||
|
||||
const lastRecorded = latest(executed);
|
||||
const legacy = lastRecorded === null ? await legacyVersion(connection, migrator) : null;
|
||||
await safetyNet.beforeMigrate(
|
||||
lastRecorded !== null ? migrationLabel(lastRecorded) : legacy !== null ? `legacy-${legacy}` : null,
|
||||
);
|
||||
console.log(`[DB] Applying ${pending.length} pending migration(s)`);
|
||||
if (baselined.length === 0) {
|
||||
await migrator.up();
|
||||
@@ -210,11 +253,12 @@ export async function migrateToHead(
|
||||
* older code against a schema it does not know and fail later, far from the
|
||||
* cause. Refused before anything is written.
|
||||
*/
|
||||
export function refuseNewerDatabase(executed: string[], known: Set<string>): void {
|
||||
export function refuseNewerDatabase(executed: string[], known: Set<string>, restoreHint: string | null = null): void {
|
||||
const unknown = unknownMigrations(executed, known);
|
||||
if (unknown.length === 0) return;
|
||||
throw new Error(
|
||||
`${REFUSAL} the database was migrated by a newer TREK (${unknown.length} unknown migration(s), latest ${unknown[unknown.length - 1]}). ` +
|
||||
'Run that version again, or restore a backup taken with this one.',
|
||||
'Run that version again, or restore a backup taken with this one.' +
|
||||
(restoreHint ? ` ${restoreHint}` : ''),
|
||||
);
|
||||
}
|
||||
|
||||
+13
-5
@@ -1,6 +1,7 @@
|
||||
import mikroOrmConfig from '../mikro-orm.config';
|
||||
import { registerReinitializeHook, runDemoSeed } from './database';
|
||||
import { migrateToHead } from './legacy-baseline';
|
||||
import { migrateToHead, NO_SAFETY_NET } from './legacy-baseline';
|
||||
import { preMigrateSnapshot } from './pre-migrate-snapshot';
|
||||
import { withRequestContext } from '../nest/database/request-context';
|
||||
import type { AnyEntity, EntityClass, EntityManager, EntitySchema, IDatabaseDriver, MikroORM } from '@mikro-orm/core';
|
||||
import type { Migrator } from '@mikro-orm/migrations';
|
||||
@@ -29,14 +30,21 @@ type AnyOrm = MikroORM<
|
||||
readonly (string | EntityClass<AnyEntity> | EntitySchema)[]
|
||||
>;
|
||||
|
||||
/** Migrate to head, then seed. Safe to call again — both halves are idempotent. */
|
||||
export async function runSchemaBootstrap(orm: AnyOrm): Promise<void> {
|
||||
/**
|
||||
* Migrate to head, then seed. Safe to call again, both halves are idempotent.
|
||||
*
|
||||
* `snapshot: false` skips the pre-migration copy (`pre-migrate-snapshot.ts`).
|
||||
* Only the restore hook passes it: the file it migrates was just unpacked from
|
||||
* a backup archive, which is still there to go back to.
|
||||
*/
|
||||
export async function runSchemaBootstrap(orm: AnyOrm, { snapshot = true }: { snapshot?: boolean } = {}): Promise<void> {
|
||||
const migrator = orm.config.getExtension('@mikro-orm/migrator') as Migrator;
|
||||
const connection = orm.em.getConnection();
|
||||
// A database the retired positional runner migrated has no migrator rows yet;
|
||||
// what it already has is recorded in the same transaction as the run, or the
|
||||
// run would replay the whole history over it (see legacy-baseline.ts). Every
|
||||
// other database just gets its pending migrations.
|
||||
await migrateToHead(orm.em.getConnection(), migrator);
|
||||
await migrateToHead(connection, migrator, undefined, undefined, snapshot ? preMigrateSnapshot(connection) : NO_SAFETY_NET);
|
||||
|
||||
const defaultSeeder = orm.config.get('seeder').defaultSeeder;
|
||||
if (defaultSeeder) await orm.seeder.seedString(defaultSeeder);
|
||||
@@ -77,7 +85,7 @@ export function attachOrm(orm: AnyOrm): void {
|
||||
// `destroy()` is a no-op (see `orm-driver.ts`), so this close never reaches it.
|
||||
await connection.close(true);
|
||||
await connection.connect();
|
||||
await runSchemaBootstrap(orm);
|
||||
await runSchemaBootstrap(orm, { snapshot: false });
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
import { readEnv, type AppEnv } from '../app-config';
|
||||
import { REFUSAL, type MigrationSafetyNet } from './legacy-baseline';
|
||||
import type { Connection } from '@mikro-orm/core';
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
/**
|
||||
* A copy of the database taken right before a boot migrates it.
|
||||
*
|
||||
* Migrations only go forward, and a database a newer TREK migrated is refused
|
||||
* by the older one (`refuseNewerDatabase`). Without a copy from before the
|
||||
* upgrade, going back to the previous image means restoring a backup, and
|
||||
* automatic backups are off by default. So the boot takes one itself: a
|
||||
* `VACUUM INTO` snapshot next to the database file, named after the schema it
|
||||
* holds, e.g. `pre-migrate-legacy-244-20261008T091500Z.db` for a v4.3.3
|
||||
* install or `pre-migrate-20200101042300-….db` for one the migrator already
|
||||
* owns. The newest `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` stay.
|
||||
*
|
||||
* Only when something is pending and the database holds anything: a restart
|
||||
* with nothing to apply, a fresh install and an in-memory database take none.
|
||||
* A snapshot that cannot be written refuses the boot before anything is
|
||||
* migrated; `TREK_DB_PRE_MIGRATE_SNAPSHOT=false` is the way past that.
|
||||
*/
|
||||
|
||||
const PREFIX = 'pre-migrate-';
|
||||
const SNAPSHOT_FILE = /^pre-migrate-.+\.db$/;
|
||||
|
||||
/** `20261008T091500Z`: sortable, and free of the colons Windows refuses in a file name. */
|
||||
function stamp(now: Date): string {
|
||||
return now
|
||||
.toISOString()
|
||||
.replace(/\.\d+Z$/, 'Z')
|
||||
.replace(/[-:]/g, '');
|
||||
}
|
||||
|
||||
/** The file behind the connection, or null for `:memory:` and temp databases. */
|
||||
async function databaseFile(connection: Connection): Promise<string | null> {
|
||||
const rows: Array<{ name: string; file: string }> = await connection.execute('PRAGMA database_list');
|
||||
const main = rows.find((row) => row.name === 'main');
|
||||
return main?.file ? main.file : null;
|
||||
}
|
||||
|
||||
/** A fresh install: nothing but the migrator's own bookkeeping table yet. */
|
||||
async function holdsAnything(connection: Connection): Promise<boolean> {
|
||||
const rows: unknown[] = await connection.execute(
|
||||
`SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' AND name <> 'mikro_orm_migrations' LIMIT 1`,
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
/** The snapshots in `dir`, newest first. */
|
||||
export function listSnapshots(dir: string): string[] {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = fs.readdirSync(dir).filter((entry) => SNAPSHOT_FILE.test(entry));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
return entries
|
||||
.map((entry) => ({ file: path.join(dir, entry), mtime: fs.statSync(path.join(dir, entry)).mtimeMs }))
|
||||
.sort((a, b) => b.mtime - a.mtime || b.file.localeCompare(a.file))
|
||||
.map((entry) => entry.file);
|
||||
}
|
||||
|
||||
/** Deletes all but the newest `keep`. A failure only warns: the new snapshot is already safe. */
|
||||
export function pruneSnapshots(dir: string, keep: number): void {
|
||||
for (const file of listSnapshots(dir).slice(keep)) {
|
||||
try {
|
||||
fs.rmSync(file, { force: true });
|
||||
} catch (err) {
|
||||
console.warn(
|
||||
`[DB] Could not delete the old pre-migration copy ${file}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes the copy under a temporary name and renames it into place, so a crash
|
||||
* halfway never leaves a truncated file that looks like a good snapshot.
|
||||
*/
|
||||
async function writeSnapshot(connection: Connection, file: string, from: string, now: Date): Promise<string> {
|
||||
const dir = path.dirname(file);
|
||||
const target = path.join(dir, `${PREFIX}${from}-${stamp(now)}.db`);
|
||||
const partial = `${target}.partial`;
|
||||
try {
|
||||
fs.rmSync(partial, { force: true });
|
||||
await connection.execute(`VACUUM INTO '${partial.replaceAll("'", "''")}'`);
|
||||
fs.renameSync(partial, target);
|
||||
} catch (err) {
|
||||
try {
|
||||
fs.rmSync(partial, { force: true });
|
||||
} catch {
|
||||
// The refusal below is what matters; a stray .partial is ignored by listSnapshots.
|
||||
}
|
||||
throw new Error(
|
||||
`${REFUSAL} could not write a copy of the database to ${target} before migrating it ` +
|
||||
`(${err instanceof Error ? err.message : String(err)}). Nothing was migrated. Free some space in ${dir} ` +
|
||||
'or fix its permissions, or set TREK_DB_PRE_MIGRATE_SNAPSHOT=false to migrate without a copy.',
|
||||
{ cause: err },
|
||||
);
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
/** The boot's safety net: snapshot before migrating, and point at the right copy when refusing. */
|
||||
export function preMigrateSnapshot(
|
||||
connection: Connection,
|
||||
env: () => AppEnv = readEnv,
|
||||
now: () => Date = () => new Date(),
|
||||
): MigrationSafetyNet {
|
||||
return {
|
||||
async beforeMigrate(from) {
|
||||
const { db, app } = env();
|
||||
// Unset means on, except under NODE_ENV=test, where a suite asks for it explicitly.
|
||||
if (db.preMigrateSnapshot === undefined && app.isTest) return;
|
||||
const file = await databaseFile(connection);
|
||||
if (!file) return;
|
||||
if (from === null && !(await holdsAnything(connection))) return;
|
||||
if (db.preMigrateSnapshot === false) {
|
||||
console.warn(`[DB] TREK_DB_PRE_MIGRATE_SNAPSHOT is off: migrating ${file} without a copy to go back to`);
|
||||
return;
|
||||
}
|
||||
const target = await writeSnapshot(connection, file, from ?? 'unversioned', now());
|
||||
console.log(`[DB] Copy of the database before migrating: ${target}`);
|
||||
pruneSnapshots(path.dirname(file), db.preMigrateSnapshotKeep);
|
||||
},
|
||||
|
||||
async restoreHint(thisRelease) {
|
||||
const file = await databaseFile(connection);
|
||||
if (!file) return null;
|
||||
const snapshots = listSnapshots(path.dirname(file));
|
||||
const match = snapshots.find((snapshot) => path.basename(snapshot).startsWith(`${PREFIX}${thisRelease}-`));
|
||||
if (match) {
|
||||
return (
|
||||
`The copy taken right before the newer TREK migrated it is ${match}: stop TREK, delete ${file}-wal and ` +
|
||||
`${file}-shm, and copy it over ${file}.`
|
||||
);
|
||||
}
|
||||
if (snapshots.length > 0) {
|
||||
return `Copies taken before earlier migrations are in ${path.dirname(file)} (${PREFIX}*.db), none of them from this release.`;
|
||||
}
|
||||
return null;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
derivePlugins,
|
||||
deriveIntegrations,
|
||||
deriveBackup,
|
||||
deriveDb,
|
||||
deriveFiles,
|
||||
deriveNet,
|
||||
derivePaths,
|
||||
@@ -269,6 +270,21 @@ describe('deriveBackup', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('deriveDb: the pre-migration snapshot', () => {
|
||||
it('leaves the switch undefined when unset, so the caller can default it per environment', () => {
|
||||
expect(deriveDb({}).preMigrateSnapshot).toBeUndefined();
|
||||
expect(deriveDb({ TREK_DB_PRE_MIGRATE_SNAPSHOT: 'off' }).preMigrateSnapshot).toBe(false);
|
||||
expect(deriveDb({ TREK_DB_PRE_MIGRATE_SNAPSHOT: 'TRUE' }).preMigrateSnapshot).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps three copies unless told otherwise, at most a hundred', () => {
|
||||
expect(deriveDb({}).preMigrateSnapshotKeep).toBe(3);
|
||||
expect(deriveDb({ TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '5' }).preMigrateSnapshotKeep).toBe(5);
|
||||
expect(deriveDb({ TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '0' }).preMigrateSnapshotKeep).toBe(3);
|
||||
expect(deriveDb({ TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '500' }).preMigrateSnapshotKeep).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('deriveNet', () => {
|
||||
it('ALLOW_INTERNAL_NETWORK coerces the boolean-like family (ssrfGuard)', () => {
|
||||
expect(deriveNet({ ALLOW_INTERNAL_NETWORK: 'True' }).allowInternalNetwork).toBe(true);
|
||||
|
||||
@@ -313,3 +313,21 @@ describe('readEnv', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateEnvAtBoot: the pre-migration snapshot switches', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('accepts a boolean switch and a keep count from 1 to 100', () => {
|
||||
expect(() =>
|
||||
validateEnvAtBoot({ TREK_DB_PRE_MIGRATE_SNAPSHOT: 'false', TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '100' }),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it('refuses a misspelt switch instead of guessing which way the safety net should go', () => {
|
||||
expect(bootReport({ TREK_DB_PRE_MIGRATE_SNAPSHOT: 'of' }).thrown).toMatch(/Invalid environment configuration/);
|
||||
expect(bootReport({ TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '0' }).printed).toContain('TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP');
|
||||
expect(bootReport({ TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP: '2.5' }).thrown).toMatch(/Invalid environment configuration/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* The copy of the database a boot takes before it migrates
|
||||
* (`src/db/pre-migrate-snapshot.ts`), driven through `migrateToHead` against a
|
||||
* temp FILE database, because an in-memory one has nothing to copy.
|
||||
*
|
||||
* The end-to-end case (a v4.3.3 install booting through `buildApp()`) is
|
||||
* `tests/integration/legacy-upgrade-v244.test.ts`.
|
||||
*/
|
||||
import { readEnv, type AppEnv } from '../../../src/app-config';
|
||||
import { knownMigrationNames } from '../../../src/db/known-migrations';
|
||||
import { migrateToHead, migrationLabel } from '../../../src/db/legacy-baseline';
|
||||
import { listSnapshots, preMigrateSnapshot, pruneSnapshots } from '../../../src/db/pre-migrate-snapshot';
|
||||
import { Migrator } from '@mikro-orm/migrations';
|
||||
import { MikroORM } from '@mikro-orm/sqlite';
|
||||
|
||||
import Database from 'better-sqlite3';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const MIGRATIONS = path.join(__dirname, '../../../src/db/migrations');
|
||||
const MIDPOINT = 'Migration20200101040200_trek_photo_cache_meta_cache_key_not_null';
|
||||
const NOW = new Date('2026-10-08T09:15:00.123Z');
|
||||
const STAMP = '20261008T091500Z';
|
||||
|
||||
let dir: string;
|
||||
let dbFile: string;
|
||||
let orm: MikroORM;
|
||||
|
||||
function ormFor(dbName: string): Promise<MikroORM> {
|
||||
return MikroORM.init({
|
||||
entities: [],
|
||||
discovery: { warnWhenNoEntities: false },
|
||||
extensions: [Migrator],
|
||||
migrations: { path: MIGRATIONS, pathTs: MIGRATIONS, snapshot: false, silent: true },
|
||||
dbName,
|
||||
});
|
||||
}
|
||||
|
||||
const migrator = () => orm.config.getExtension('@mikro-orm/migrator') as Migrator;
|
||||
|
||||
/** The live env with the snapshot switches set as given. */
|
||||
function envWith(db: Partial<AppEnv['db']>, isTest = false): () => AppEnv {
|
||||
return () => {
|
||||
const env = readEnv();
|
||||
return { ...env, app: { ...env.app, isTest }, db: { ...env.db, ...db } };
|
||||
};
|
||||
}
|
||||
|
||||
function migrate(env: () => AppEnv): Promise<void> {
|
||||
const connection = orm.em.getConnection();
|
||||
return migrateToHead(
|
||||
connection,
|
||||
migrator(),
|
||||
undefined,
|
||||
undefined,
|
||||
preMigrateSnapshot(connection, env, () => NOW),
|
||||
);
|
||||
}
|
||||
|
||||
async function recorded(): Promise<string[]> {
|
||||
const rows: Array<{ name: string }> = await orm.em
|
||||
.getConnection()
|
||||
.execute('SELECT name FROM mikro_orm_migrations ORDER BY id');
|
||||
return rows.map((row) => row.name);
|
||||
}
|
||||
|
||||
const snapshotsIn = () =>
|
||||
fs
|
||||
.readdirSync(dir)
|
||||
.filter((entry) => entry.startsWith('pre-migrate-'))
|
||||
.sort();
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'trek-premigrate-'));
|
||||
dbFile = path.join(dir, 'travel.db');
|
||||
orm = await ormFor(dbFile);
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await orm.close(true);
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('the pre-migration snapshot', () => {
|
||||
it('PREMIG-001: copies a database that is behind before migrating it, named after the schema it holds', async () => {
|
||||
await migrator().up({ to: MIDPOINT });
|
||||
await orm.em
|
||||
.getConnection()
|
||||
.execute(`INSERT INTO users (username, email, password_hash) VALUES ('kept', 'kept@example.test', 'x')`);
|
||||
const before = await recorded();
|
||||
|
||||
await migrate(envWith({ preMigrateSnapshot: true }));
|
||||
|
||||
const name = `pre-migrate-${migrationLabel(MIDPOINT)}-${STAMP}.db`;
|
||||
expect(snapshotsIn()).toEqual([name]);
|
||||
expect(await recorded()).toHaveLength(knownMigrationNames().size);
|
||||
const copy = new Database(path.join(dir, name), { readonly: true });
|
||||
try {
|
||||
// The copy holds the state from before the run: its rows and its migrations.
|
||||
expect(copy.prepare('SELECT email FROM users').all()).toEqual([{ email: 'kept@example.test' }]);
|
||||
expect(
|
||||
(copy.prepare('SELECT name FROM mikro_orm_migrations ORDER BY id').all() as Array<{ name: string }>).map(
|
||||
(r) => r.name,
|
||||
),
|
||||
).toEqual(before);
|
||||
} finally {
|
||||
copy.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('PREMIG-002: a legacy install is labelled by its schema_version', async () => {
|
||||
await orm.em.getConnection().execute('CREATE TABLE schema_version (version INTEGER NOT NULL)');
|
||||
await orm.em.getConnection().execute('INSERT INTO schema_version (version) VALUES (10)');
|
||||
await migrate(envWith({ preMigrateSnapshot: true }));
|
||||
expect(snapshotsIn()).toEqual([`pre-migrate-legacy-10-${STAMP}.db`]);
|
||||
});
|
||||
|
||||
it('PREMIG-003: a fresh install, a restart with nothing pending and the test default take none', async () => {
|
||||
await migrate(envWith({ preMigrateSnapshot: true }));
|
||||
expect(snapshotsIn()).toEqual([]);
|
||||
|
||||
await migrate(envWith({ preMigrateSnapshot: true }));
|
||||
expect(snapshotsIn()).toEqual([]);
|
||||
|
||||
await orm.close(true);
|
||||
fs.rmSync(dbFile);
|
||||
orm = await ormFor(dbFile);
|
||||
await migrator().up({ to: MIDPOINT });
|
||||
await migrate(envWith({ preMigrateSnapshot: undefined }, true));
|
||||
expect(snapshotsIn()).toEqual([]);
|
||||
expect(await recorded()).toHaveLength(knownMigrationNames().size);
|
||||
});
|
||||
|
||||
it('PREMIG-004: a copy that cannot be written refuses the boot and migrates nothing', async () => {
|
||||
await migrator().up({ to: MIDPOINT });
|
||||
const before = await recorded();
|
||||
// A directory where the copy has to land makes the final rename fail.
|
||||
fs.mkdirSync(path.join(dir, `pre-migrate-${migrationLabel(MIDPOINT)}-${STAMP}.db`));
|
||||
|
||||
await expect(migrate(envWith({ preMigrateSnapshot: true }))).rejects.toThrow(
|
||||
/\[DB\] Refusing to boot: could not write a copy of the database to .* Nothing was migrated\..*TREK_DB_PRE_MIGRATE_SNAPSHOT=false/,
|
||||
);
|
||||
expect(await recorded()).toEqual(before);
|
||||
expect(fs.readdirSync(dir).filter((entry) => entry.endsWith('.partial'))).toEqual([]);
|
||||
});
|
||||
|
||||
it('PREMIG-005: TREK_DB_PRE_MIGRATE_SNAPSHOT=false migrates without a copy, and says so', async () => {
|
||||
await migrator().up({ to: MIDPOINT });
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
await migrate(envWith({ preMigrateSnapshot: false }));
|
||||
expect(snapshotsIn()).toEqual([]);
|
||||
expect(await recorded()).toHaveLength(knownMigrationNames().size);
|
||||
expect(String(warn.mock.calls[0]?.[0])).toContain('TREK_DB_PRE_MIGRATE_SNAPSHOT is off');
|
||||
});
|
||||
|
||||
it('PREMIG-006: keeps only the newest copies', async () => {
|
||||
for (const [i, name] of ['a', 'b', 'c', 'd'].entries()) {
|
||||
const file = path.join(dir, `pre-migrate-${name}.db`);
|
||||
fs.writeFileSync(file, '');
|
||||
fs.utimesSync(file, 1_000 + i, 1_000 + i);
|
||||
}
|
||||
fs.writeFileSync(path.join(dir, 'pre-restore-1.db'), '');
|
||||
fs.writeFileSync(path.join(dir, 'pre-migrate-e.db.partial'), '');
|
||||
|
||||
pruneSnapshots(dir, 2);
|
||||
|
||||
expect(listSnapshots(dir).map((file) => path.basename(file))).toEqual(['pre-migrate-d.db', 'pre-migrate-c.db']);
|
||||
// Nothing else in the data dir is touched.
|
||||
expect(fs.existsSync(path.join(dir, 'pre-restore-1.db'))).toBe(true);
|
||||
expect(fs.existsSync(path.join(dir, 'pre-migrate-e.db.partial'))).toBe(true);
|
||||
});
|
||||
|
||||
it('PREMIG-007: the boot prunes to TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP after a new copy', async () => {
|
||||
for (const name of ['old-1', 'old-2']) {
|
||||
const file = path.join(dir, `pre-migrate-${name}.db`);
|
||||
fs.writeFileSync(file, '');
|
||||
fs.utimesSync(file, 1_000, 1_000);
|
||||
}
|
||||
await migrator().up({ to: MIDPOINT });
|
||||
await migrate(envWith({ preMigrateSnapshot: true, preMigrateSnapshotKeep: 1 }));
|
||||
expect(snapshotsIn()).toEqual([`pre-migrate-${migrationLabel(MIDPOINT)}-${STAMP}.db`]);
|
||||
});
|
||||
|
||||
it('PREMIG-008: refusing a database a newer TREK migrated names the copy taken from this release', async () => {
|
||||
await migrate(envWith({ preMigrateSnapshot: true }));
|
||||
const head = migrationLabel([...knownMigrationNames()].sort().at(-1) ?? '');
|
||||
const match = path.join(dir, `pre-migrate-${head}-20261001T000000Z.db`);
|
||||
fs.writeFileSync(match, '');
|
||||
await orm.em
|
||||
.getConnection()
|
||||
.execute(
|
||||
"INSERT INTO mikro_orm_migrations (name, executed_at) VALUES ('Migration20990101000000_from_a_newer_release', CURRENT_TIMESTAMP)",
|
||||
);
|
||||
|
||||
const refusal = migrate(envWith({ preMigrateSnapshot: true }));
|
||||
await expect(refusal).rejects.toThrow(/migrated by a newer TREK/);
|
||||
await expect(refusal).rejects.toThrow(`The copy taken right before the newer TREK migrated it is ${match}`);
|
||||
|
||||
fs.rmSync(match);
|
||||
fs.writeFileSync(path.join(dir, 'pre-migrate-legacy-200-20250101T000000Z.db'), '');
|
||||
await expect(migrate(envWith({ preMigrateSnapshot: true }))).rejects.toThrow(/none of them from this release/);
|
||||
});
|
||||
});
|
||||
@@ -92,6 +92,8 @@ After each auto-backup run, **auto-backup files** older than `keep_days` are pru
|
||||
|
||||
Always create a manual backup before updating. See [Updating](Updating).
|
||||
|
||||
TREK also copies the database by itself right before a startup migrates it, to `data/pre-migrate-<schema>-<time>.db`, and keeps the newest three. That copy is what an older image can be started on again if you have to roll an update back. It holds the database only, never uploads, so it does not replace a backup. See [Updating](Updating#database-migrations).
|
||||
|
||||
## Audit log
|
||||
|
||||
The following actions are recorded in the [Audit-Log](Audit-Log):
|
||||
|
||||
@@ -460,6 +460,8 @@ through environment variables. `TREK_PLACE_PHOTO_DIR` below is unaffected.
|
||||
| `RESTORE_FROM_BACKUP` | Path to a backup ZIP to restore on the **first** start, before any setup. Only acts while no database exists; afterwards it is ignored with a warning. A backup that fails the restore checks stops the start. See [Backups](Backups#restoring-on-a-new-install-before-setup). | none |
|
||||
| `BACKUP_MAX_DECOMPRESSED_MB` | Maximum **decompressed** size (in MB) of a restore-backup archive — the zip-bomb guard. Independent of `BACKUP_UPLOAD_LIMIT_MB` and enforced on both restore paths, so a restore that fits the upload cap can still be refused with `Backup exceeds the maximum decompressed size.` Raise both when restoring a very large instance. | `5120` (5 GB) |
|
||||
| `TREK_DB_JOURNAL_MODE` | SQLite [journal mode](https://sqlite.org/pragma.html#pragma_journal_mode): `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, `WAL` or `OFF`. Set `DELETE` when the data directory lives on network storage — see below. Values SQLite doesn't know log a warning and fall back. | `WAL` |
|
||||
| `TREK_DB_PRE_MIGRATE_SNAPSHOT` | Copy the database to `pre-migrate-<schema>-<time>.db` next to it before a startup applies pending migrations, so the previous image can be started on that copy again. If the copy cannot be written the startup refuses rather than migrate without one. Set `false` only to get past that when you cannot free space and have a backup of your own. See [Updating](Updating#database-migrations). | `true` |
|
||||
| `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` | How many of those copies stay in the data directory, newest first (1 to 100). | `3` |
|
||||
| `TREK_DB_SYNCHRONOUS` | SQLite [synchronous](https://sqlite.org/pragma.html#pragma_synchronous) level: `OFF`, `NORMAL`, `FULL` or `EXTRA`. The default follows the journal mode — `NORMAL` under WAL (what SQLite itself uses there), `FULL` otherwise, because a rollback journal at `NORMAL` can lose committed transactions on a power cut. | `NORMAL` / `FULL` |
|
||||
|
||||
### Running the database on network storage
|
||||
|
||||
@@ -80,6 +80,20 @@ See [Install-Helm](Install-Helm) for the full installation walkthrough and value
|
||||
|
||||
TREK runs any pending database migrations automatically at startup. No manual migration steps are required after pulling a new image.
|
||||
|
||||
Migrations only go forward. Once a newer TREK has migrated the database, an older image refuses to start on it, so going back means going back to a copy of the database from before the update.
|
||||
|
||||
TREK takes that copy itself. Right before a startup applies pending migrations, it writes the database to `data/pre-migrate-<schema>-<time>.db` (next to `travel.db`, or next to the file `TREK_DB_FILE` names) and logs the path. `<schema>` names what the copy holds: `legacy-244` for a 4.3.3 install, or the last migration's number for a newer one. The newest three are kept (`TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP`). A restart with nothing to migrate and a brand-new install take no copy.
|
||||
|
||||
If the copy cannot be written (the disk is full, or the data directory is read-only), TREK does not migrate and refuses to start, with a message naming the path. Free some space and start again. If you cannot, and you have a backup of your own, `TREK_DB_PRE_MIGRATE_SNAPSHOT=false` starts without the copy.
|
||||
|
||||
To go back to the previous image after an update:
|
||||
|
||||
1. Stop TREK.
|
||||
2. Copy the `pre-migrate-…db` file over `data/travel.db` and delete `data/travel.db-wal` and `data/travel.db-shm`.
|
||||
3. Start the previous image.
|
||||
|
||||
An older image that refuses a newer database names the copy it can run, if one is in the data directory. Uploads are not part of this copy; they are not changed by migrations. The copy is a safety net for the update, not a backup: keep taking backups as described in [Backups](Backups).
|
||||
|
||||
## After the Update
|
||||
|
||||
The first time a user opens TREK on the desktop after an update, a notice headed **Update installed** opens on top of the app: three cards with the headline features of that release, a note from the maintainer, a **Release notes** link to the full notes on GitHub, and links for supporting the project. Every user sees it once per version. Closing it keeps it closed on that version, and the next update brings it back, a patch release included. The phone does not show it.
|
||||
|
||||
Reference in New Issue
Block a user