fix(backup): report a lost connection when the swap and its reopen both fail

SqliteDatabaseBackup.replace reopens the connection in a finally block, so
it runs even when the swap throws. A reopen that failed there was stored
and then dropped: the swap's exception propagated past the return that
would have carried it. The demo reset caught that swap error, logged
"Failed to restore baseline" and returned normally, leaving the instance
without a database connection and nothing in the log to say so. Before
the port, the same branch awaited reinitialize() and its rejection reached
the tick's log line.

replace now throws a DatabaseConnectionLostError when both steps fail. It
carries the swap error and has the reopen failure as its cause, and its
message tells the operator to restart. The demo reset lets that error
through to DemoResetJob's tick, which logs it, and still logs and skips a
failed swap whose reopen succeeded. A restore that hits it answers the
same 500 as any other thrown restore error, with both failures in the
server log.
This commit is contained in:
Maurice
2026-10-08 15:40:00 +02:00
parent fd28ba7fd3
commit 175f5d96da
5 changed files with 100 additions and 14 deletions
@@ -58,7 +58,8 @@ export interface DatabaseBackupStrategy {
* Puts `source` (a database this strategy wrote or verified) in place of the
* live one, then reopens the connection, migrating it forward. The reopen
* runs even when the swap throws, so the process is never left without a
* connection; the swap's error is rethrown after it.
* connection; the swap's error is rethrown after it. When that reopen fails
* as well, it throws a `DatabaseConnectionLostError` instead, carrying both.
*
* Resolves with `reopenError` set when the new database landed but the
* connection could not be reopened: the caller reports "restart required".
@@ -74,3 +75,25 @@ export interface DatabaseBackupRefusal {
/** Injection token for the active `DatabaseBackupStrategy`. */
export const DATABASE_BACKUP = Symbol('DATABASE_BACKUP');
const describeError = (err: unknown): string => (err instanceof Error ? err.message : String(err));
/**
* Thrown by `replace` when the swap failed and the reopen that always follows
* failed too. The process is left without a database connection until a
* restart, so this must reach a log line rather than read as an ordinary failed
* swap. `swapError` is why the swap failed; `cause` is the reopen failure.
*/
export class DatabaseConnectionLostError extends Error {
constructor(
readonly swapError: unknown,
reopenError: unknown,
) {
super(
`The database could not be replaced (${describeError(swapError)}) and the connection could not be reopened ` +
`(${describeError(reopenError)}). Restart the server.`,
{ cause: reopenError },
);
this.name = 'DatabaseConnectionLostError';
}
}