fix(db): trace a broken key in the move without blaming the wrong row

The move read unique indexes by joining indkey to pg_attribute. An expression key such as lower(email) has attnum 0 there, so idx_users_email_lower was missing from the plan. The catalog query now reads each index's key as Postgres prints it, whether any part of it is an expression, and the columns it reads: the plain key columns and the Var nodes of indexprs, without INCLUDE columns or the predicate.

A broken unique key no longer names --drop-nul as its cause. The refusal names the key and the shared value from Postgres's detail. It explains that the SQLite file held the two rows apart, and the hint about SQLite's lower() appears only for an expression key. Values --drop-nul changed in the key's columns are listed as rows to check first, and only when the duplicate value holds their text. An unrelated value that lost a NUL is no longer named. A plain key over a quoted column is named by its column, not as an expression.

A row that breaks a constraint (SQLSTATE class 23) is now a MoveRefusal that says what to correct, in the SQLite file or in a plugin's plugin.db. Before, it was a bare error with a stack. A value Postgres refuses by its type (class 22) stays a failure: it names no row, and values.ts should have refused it first, so the message asks for a bug report. MoveRefusal and the remedy text move to refusal.ts so copy.ts can use them without an import cycle. The usage text of --drop-nul and the wiki's move section describe NUL in JSON, the key listing and these refusals.
This commit is contained in:
Maurice
2026-11-18 13:20:08 +01:00
parent e66330e226
commit 69e791206e
10 changed files with 560 additions and 68 deletions
@@ -1,4 +1,4 @@
import { quoteIdentifier, run, type CopiedSequence, type TableSpec, type Trx } from './copy';
import { quoteIdentifier, run, type CopiedSequence, type TableSpec, type Trx, type UniqueKey } from './copy';
import { planCopyOrder, type ForeignKey } from './order';
import { columnKind } from './values';
@@ -23,8 +23,8 @@ interface TargetTable {
name: string;
columns: TargetColumn[];
primaryKey: string[];
/** Every unique index (the primary key's among them) over plain columns, by name. */
uniqueKeys: Array<{ name: string; columns: string[] }>;
/** Every unique index (the primary key's among them), by name, expression indexes included. */
uniqueKeys: UniqueKey[];
}
export interface TargetSchema {
@@ -59,14 +59,30 @@ export async function readTargetSchema(trx: Trx): Promise<TargetSchema> {
"and pg_get_serial_sequence(format('%I', table_name), column_name) is not null order by table_name, column_name",
);
// A unique constraint is backed by an index of its name, so the index names what a refusal reports.
const uniqueIndexes = await run<{ table_name: string; index_name: string; columns: string[] }>(
// A unique constraint is backed by an index of its name, so the index names what a refusal reports. Its key
// is read as Postgres spells it. `indkey` holds 0 for a key that is an expression (`lower(email)`), which
// marks the index as an expression index; the columns an expression reads are the Var nodes of `indexprs`,
// listed after the plain key columns. The partial index's predicate and the INCLUDE columns are left out,
// since neither decides whether two rows collide.
const uniqueIndexes = await run<{
table_name: string;
index_name: string;
keys: string[];
columns: string[];
expression: boolean;
}>(
trx,
'select t.relname as table_name, i.relname as index_name, array_agg(a.attname::text order by k.ord) as columns ' +
'select t.relname as table_name, i.relname as index_name, ' +
'0 = any ((x.indkey::int2[])[0:x.indnkeyatts - 1]) as expression, ' +
'array(select pg_get_indexdef(x.indexrelid, k, true) from generate_series(1, x.indnkeyatts::int) as k ' +
'order by k) as keys, ' +
'array(select a.attname::text from pg_attribute a where a.attrelid = x.indrelid and a.attnum > 0 and (' +
'a.attnum = any ((x.indkey::int2[])[0:x.indnkeyatts - 1]) or a.attnum::text in (select m[1] from ' +
"regexp_matches(coalesce(x.indexprs::text, ''), ':varattno ([0-9]+)', 'g') as m)) " +
'order by coalesce(array_position((x.indkey::int2[])[0:x.indnkeyatts - 1], a.attnum), ' +
'x.indnkeyatts + a.attnum)) as columns ' +
'from pg_index x join pg_class i on i.oid = x.indexrelid join pg_class t on t.oid = x.indrelid ' +
'cross join lateral unnest(x.indkey) with ordinality as k(attnum, ord) ' +
'join pg_attribute a on a.attrelid = x.indrelid and a.attnum = k.attnum ' +
'where x.indisunique and t.relnamespace = current_schema()::regnamespace group by t.relname, i.relname',
'where x.indisunique and t.relnamespace = current_schema()::regnamespace order by t.relname, i.relname',
);
const tables = new Map<string, TargetTable>();
@@ -85,7 +101,12 @@ export async function readTargetSchema(trx: Trx): Promise<TargetSchema> {
tables.set(column.table_name, table);
}
for (const index of uniqueIndexes) {
tables.get(index.table_name)?.uniqueKeys.push({ name: index.index_name, columns: index.columns });
tables.get(index.table_name)?.uniqueKeys.push({
name: index.index_name,
key: index.keys.join(', '),
columns: index.columns,
expression: index.expression,
});
}
const foreignKeys: ForeignKey[] = [];
for (const constraint of constraints) {
@@ -5,7 +5,8 @@ import type { AnyOrm } from '../../../orm';
import { buildOrmConfig, resolveEngine } from '../../index';
import { redactUrl } from '../../redact-url';
import { readPostgresBaseline } from '../baseline';
import { MoveRefusal, moveSqliteToPostgres, renderSummary, type MoveOptions } from './move';
import { moveSqliteToPostgres, renderSummary, type MoveOptions } from './move';
import { MoveRefusal } from './refusal';
import { MikroORM } from '@mikro-orm/core';
import path from 'node:path';
@@ -23,8 +24,9 @@ export const USAGE =
'Usage: move-to-postgres [--from=<SQLite file>] [--drop-nul]\n' +
'Copies a TREK SQLite database (data/travel.db by default) into the empty Postgres database TREK_DB_URL names.\n' +
' --from=<file> the SQLite file to copy\n' +
" --drop-nul drop NUL characters (U+0000) from TREK's own text, which Postgres cannot hold and TREK drops on\n" +
' its next write anyway, instead of refusing the move; the summary lists every value changed.\n' +
" --drop-nul drop NUL characters (U+0000) from TREK's own text and JSON (raw or escaped as \\u0000), which\n" +
' Postgres cannot hold and TREK drops on its next write anyway, instead of refusing the move;\n' +
' the summary lists every value changed.\n' +
" A plugin's own data is never changed: a NUL there refuses the move either way.";
export interface MoveCommandIo {
+122 -20
View File
@@ -1,4 +1,5 @@
import type { SqliteSource } from '../../sqlite/sqlite-source';
import { FIX_IN_PLUGIN_DB, FIX_IN_SQLITE_FILE, MoveRefusal } from './refusal';
import {
beyondSafeInteger,
canonicalFromPostgres,
@@ -35,6 +36,20 @@ export const MAX_LISTED_PROBLEMS = 20;
/** Postgres's SQLSTATE for a row that would break a unique key. */
const UNIQUE_VIOLATION = '23505';
/**
* A unique index of a Postgres table. `key` is its key as Postgres spells it
* (`username`, or `lower(email)` for an expression), `columns` every column
* the key reads, so a broken key can be traced to the values that changed.
* `expression` is whether any part of the key is an expression rather than a
* plain column.
*/
export interface UniqueKey {
name: string;
key: string;
columns: string[];
expression: boolean;
}
/** One column of a table the move copies. */
interface CopyColumn {
name: string;
@@ -57,7 +72,7 @@ export interface TableSpec {
/** A plugin's own table, whose values must arrive exactly as stored (`PLUGIN_RULES` in `values.ts`). */
exact: boolean;
/** The unique keys of the Postgres table (the primary key among them), by constraint or index name. */
uniqueKeys: Array<{ name: string; columns: string[] }>;
uniqueKeys: UniqueKey[];
}
/** A value from which `--drop-nul` removed NUL characters, for the summary. */
@@ -65,6 +80,12 @@ interface DroppedNul {
/** The table, row and column, named as a refusal names a value. */
where: string;
column: string;
/**
* The text without its NUL characters, as Postgres spells it in a broken
* key's detail, or null where it spells it otherwise (JSON, which jsonb
* reformats).
*/
text: string | null;
}
/** A row's deferred values, kept until every table is in. */
@@ -147,29 +168,106 @@ export function listLines(lines: readonly string[], total = lines.length): strin
return shown.join('\n');
}
/** The SQLSTATE of a Postgres error, when it carries one. */
function sqlState(err: unknown): string {
const { code } = (err ?? {}) as { code?: unknown };
return typeof code === 'string' ? code : '';
}
/**
* Why Postgres refused rows of a table. A unique key broken in a table whose
* values lost NUL characters to `--drop-nul` is named as that: two values
* that differed only by a NUL are equal once it is gone, which the SQLite
* file's own key never saw.
* Whether Postgres refused rows over a constraint (SQLSTATE class 23: a
* unique, check, not-null or foreign key violation). Its detail names the
* key or the failing row, which the operator corrects in the SQLite file.
*/
function brokeConstraint(err: unknown): boolean {
return sqlState(err).startsWith('23');
}
/**
* Whether Postgres refused a value its column cannot hold (SQLSTATE class
* 22). `values.ts` refuses every such value by table, row and column before
* anything is written, so one that reaches Postgres is a gap in the move, not
* something the operator can find in the file: Postgres does not say which
* row of a batch it was.
*/
function valueRefused(err: unknown): boolean {
return sqlState(err).startsWith('22');
}
/**
* The key value of a unique violation's detail (`Key (lower(email))=(ab@x.test)
* already exists.`), lower-cased, or null when the detail has another shape
* (none, or the server's messages are in another language).
*/
function duplicateKeyValue(err: unknown): string | null {
const { detail } = (err ?? {}) as { detail?: unknown };
if (typeof detail !== 'string') return null;
const match = /^Key \(.*?\)=\((.*)\) already exists\.$/s.exec(detail);
return match ? match[1].toLowerCase() : null;
}
/** A unique key as a refusal names it: its columns, and for an expression also the key as Postgres spells it. */
function describeKey(key: UniqueKey): string {
const columns = `column(s) ${key.columns.join(', ')}`;
return key.expression ? `${columns} (read by its unique key ${key.key})` : columns;
}
/**
* Why Postgres refused rows of a table, and what to do. A broken unique key
* is two rows the SQLite file held apart: it may lack that index, compare
* differently, or have held two values that differed only by a NUL until
* `--drop-nul` removed it. Only Postgres's detail tells which, so the refusal
* never asserts a cause: it names the key and the value, and lists the values
* `--drop-nul` changed in the columns the key reads (for an expression index,
* the columns its expression reads) whose text the duplicate value holds.
* That test ignores case, so a key over lower() still finds them; it may keep
* a value that is not one of the two rows, but never leaves out one that is.
* Any other broken constraint is a row Postgres names in its detail. A value
* its column refuses is a gap in the move, reported as one.
*/
export function refusedRowsMessage(spec: TableSpec, err: unknown, dropped: readonly DroppedNul[]): string {
const message = `Postgres refused rows of ${spec.label}: ${describePgError(err)}`;
const { code, constraint } = (err ?? {}) as { code?: unknown; constraint?: unknown };
if (code !== UNIQUE_VIOLATION || dropped.length === 0) return message;
if (valueRefused(err)) {
return (
`${message}\nThe move let through a value of ${spec.label} that Postgres refuses. It should have refused ` +
'that value by table, row and column before writing anything, so this is a bug in TREK: please report it ' +
'with this message.'
);
}
if (!brokeConstraint(err)) return message;
const fix = spec.exact ? FIX_IN_PLUGIN_DB : FIX_IN_SQLITE_FILE;
if (sqlState(err) !== UNIQUE_VIOLATION) {
return (
`${message}\nA row of ${spec.label} breaks a rule of its Postgres table that the SQLite file did not hold ` +
`it to; the detail above names the failing row or the value it refers to. Correct or remove that row ${fix}`
);
}
const { constraint } = (err ?? {}) as { constraint?: unknown };
const key = spec.uniqueKeys.find((candidate) => candidate.name === constraint);
const suspects = key ? dropped.filter((value) => key.columns.includes(value.column)) : dropped;
if (suspects.length === 0) return message;
const where = key ? `its column(s) ${key.columns.join(', ')}` : spec.label;
return (
`${message}\nThe key is broken because --drop-nul removed NUL characters (U+0000) from ${suspects.length} ` +
`value(s) of ${where}, and a value that differed from another only by a NUL equals it once the NUL is gone:\n` +
`${listLines(
suspects.slice(0, MAX_LISTED_PROBLEMS).map((value) => value.where),
suspects.length,
)}\n` +
'Change or remove one of the two rows in the SQLite file (with TREK stopped), then run the move again.'
const keyValue = duplicateKeyValue(err);
const suspects = (key ? dropped.filter((value) => key.columns.includes(value.column)) : dropped).filter(
(value) => keyValue === null || value.text === null || keyValue.includes(value.text.toLowerCase()),
);
const differently = key?.expression ? ", or compare differently: SQLite's lower() changes only A to Z" : '';
const lines = [
message,
`Two rows of ${spec.label} share ${key ? `a value of its ${describeKey(key)}` : 'a unique key'}, which ` +
`Postgres holds unique and the SQLite file did not (it may lack that index${differently}). The detail ` +
'above names the value.',
];
if (suspects.length > 0) {
lines.push(
`--drop-nul removed NUL characters (U+0000) from ${suspects.length} value(s) of ` +
`${key ? `its ${describeKey(key)}` : spec.label} that may hold it; a value that differed from another ` +
'only by a NUL equals it once the NUL is gone, so check these first:',
listLines(
suspects.slice(0, MAX_LISTED_PROBLEMS).map((value) => value.where),
suspects.length,
),
);
}
lines.push(`Change or remove one of those rows ${fix}`);
return lines.join('\n');
}
async function insertBatch(
@@ -182,7 +280,8 @@ async function insertBatch(
try {
await run(trx, insertSql(spec, batch.length), batch.flat());
} catch (err) {
throw new Error(refusedRowsMessage(spec, err, dropped), { cause: err });
const message = refusedRowsMessage(spec, err, dropped);
throw brokeConstraint(err) ? new MoveRefusal(message, { cause: err }) : new Error(message, { cause: err });
}
}
@@ -238,7 +337,10 @@ export async function copyTable(
if (!('reason' in converted)) {
values.push(converted.value);
if (column.kind === 'integer' && beyondSafeInteger(converted.value)) result.unsafeIntegers++;
if (converted.droppedNul) result.nulDropped.push({ where, column: column.name });
if (converted.droppedNul) {
const text = column.kind === 'text' ? String(converted.value) : null;
result.nulDropped.push({ where, column: column.name, text });
}
return;
}
values.push(null);
@@ -26,6 +26,7 @@ import {
resolveIdentitySequences,
type PluginPlan,
} from './plugin-data';
import { FIX_IN_SQLITE_FILE, MoveRefusal } from './refusal';
import type { AbstractSqlConnection } from '@mikro-orm/sql';
import type { Kysely } from 'kysely';
@@ -63,9 +64,6 @@ import type { Kysely } from 'kysely';
* the database's: they stay where the storage driver keeps them.
*/
/** A refusal the operator can act on: printed without a stack. */
export class MoveRefusal extends Error {}
export interface MoveOptions {
/** The SQLite file to move: `data/travel.db` by default. */
sqliteFile: string;
@@ -247,10 +245,11 @@ async function copyAll(
if (refused > 0) {
const problems = copied.flatMap((table) => table.problems).slice(0, MAX_LISTED_PROBLEMS);
const nul = nulRefused > 0 ? `${nulRefused} of them hold a NUL character (U+0000). ${TREK_NUL_REMEDY}\n` : '';
const others =
refused > nulRefused ? `${nulRefused > 0 ? 'Correct the others' : 'Correct them'} ${FIX_IN_SQLITE_FILE}` : '';
throw new MoveRefusal(
`${refused} value(s) in the SQLite file cannot be stored in Postgres exactly; nothing was written:\n` +
`${listLines(problems, refused)}\n${nul}` +
'Correct those rows (or ask for help with them), then run the move again.',
`${listLines(problems, refused)}\n${nul}${others}`.trimEnd(),
);
}
@@ -0,0 +1,17 @@
/**
* A refusal the operator can act on: the move stops because of what the
* SQLite file holds, and the message says what to change before running it
* again. The command prints it without a stack, where any other error is a
* failure of the move itself.
*/
export class MoveRefusal extends Error {}
/** Where and how the operator corrects a row of TREK's own data, after "Correct them" or "Change or remove ...". */
export const FIX_IN_SQLITE_FILE =
'in the SQLite file (through TREK while it still runs on SQLite, or with the sqlite3 tool while TREK is ' +
'stopped), then run the move again.';
/** The same for a row of a plugin's own data, which lives in the plugin's own SQLite file. */
export const FIX_IN_PLUGIN_DB =
"in the plugin's plugin.db (through the plugin, or with the sqlite3 tool while TREK is stopped), then run the " +
'move again.';