The refusal listed the differences and nothing else. A table or column
only one side has now comes with what that means: the file is at this
version's level, the start that brought it there added every column it
could, the start log names the rest, and the operator keeps running on
SQLite and reports it. A column of TREK's own schema the move cannot copy
is listed apart and named a bug in TREK, since the file is not at fault.
The server guide describes the gap, the repair, the check and the refusal.
The move read unique indexes by joining indkey to pg_attribute. An expression key such as lower(email) has attnum 0 there, so idx_users_email_lower was missing from the plan. The catalog query now reads each index's key as Postgres prints it, whether any part of it is an expression, and the columns it reads: the plain key columns and the Var nodes of indexprs, without INCLUDE columns or the predicate.
A broken unique key no longer names --drop-nul as its cause. The refusal names the key and the shared value from Postgres's detail. It explains that the SQLite file held the two rows apart, and the hint about SQLite's lower() appears only for an expression key. Values --drop-nul changed in the key's columns are listed as rows to check first, and only when the duplicate value holds their text. An unrelated value that lost a NUL is no longer named. A plain key over a quoted column is named by its column, not as an expression.
A row that breaks a constraint (SQLSTATE class 23) is now a MoveRefusal that says what to correct, in the SQLite file or in a plugin's plugin.db. Before, it was a bare error with a stack. A value Postgres refuses by its type (class 22) stays a failure: it names no row, and values.ts should have refused it first, so the message asks for a bug report. MoveRefusal and the remedy text move to refusal.ts so copy.ts can use them without an import cycle. The usage text of --drop-nul and the wiki's move section describe NUL in JSON, the key listing and these refusals.
A raw U+0000 in TREK's JSON text, which an old or hand-edited travel.db can hold, was refused as "not valid JSON" without naming the NUL or the remedy. It is now refused with the NUL reason, so the refusal names --drop-nul. With --drop-nul, raw and escaped NULs are removed and nothing else, the way TREK's own writes drop them, and each change is listed. Text that is not valid JSON even without its NULs is refused as that, and the checksum's source side follows the same rule.
A \u escape of half a surrogate pair parses in JavaScript but jsonb refuses it. Before, that failed the insert without naming a row. It is now refused by table, row and column.
An attack on the move to Postgres with real SQLite files found five values
that changed without a refusal or a listing, or a valid file that failed:
- Negative zero in a plugin column became 0: pg sends the number -0 as "0",
and the checksum printed both as "0". It is sent as text and kept apart in
the checksum on both sides.
- Text whose bytes are not valid UTF-8 was read with each bad sequence
replaced by U+FFFD, and the checksum was built from the replaced text.
Text is now read as its bytes and decoded strictly in the file's encoding
(UTF-8, or UTF-16 for a file created so); such a value is refused in
TREK's and in plugin data, with --drop-nul too, and listed with its bytes.
- --drop-nul serialized a JSON value again, so a large number lost digits
and 1.0 became 1. Only the \u0000 escapes are removed now.
- A plugin default Postgres refuses for the column (DEFAULT 0 on a BLOB
column) failed the move, and DEFAULT 1.5 on an integer column would have
been rounded. Such a default is left off and listed as not carried over.
- A plugin index or primary key SQLite compared with COLLATE NOCASE or RTRIM
became one that compares exactly. It is still created, and the summary
lists the collation it lost.
Each case has a regression test that fails without its fix, on SQLite and
on the Postgres lane.
The move stripped NUL from a plugin's own text, merging keys that were
distinct, and turned a plugin's values into another type (text in a BLOB
column into bytea, numbers in an untyped column into text). Its checksum
was built from the converted values, so both passed as verified.
A plugin's data now moves exactly as the Postgres plugin store would
hand it back, or not at all: a NUL, a value of another storage class
than its column, or an integer past 2^53 refuses the move before
anything is written, listing plugin, table, row and column. A column
left open by SQLite's affinity takes the kind its values need, so text
in a BLOB column stays text.
TREK's own text holding a NUL from an older install is refused by
default too. --drop-nul drops it, lists every changed value in the
summary, and names a unique key that dropping a NUL broke.
Every table's checksum is now built from the values read from SQLite,
so a conversion that changes a value fails the verification.
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.