Commit Graph

7 Commits

Author SHA1 Message Date
Maurice 69e791206e fix(db): trace a broken key in the move without blaming the wrong row
The move read unique indexes by joining indkey to pg_attribute. An expression key such as lower(email) has attnum 0 there, so idx_users_email_lower was missing from the plan. The catalog query now reads each index's key as Postgres prints it, whether any part of it is an expression, and the columns it reads: the plain key columns and the Var nodes of indexprs, without INCLUDE columns or the predicate.

A broken unique key no longer names --drop-nul as its cause. The refusal names the key and the shared value from Postgres's detail. It explains that the SQLite file held the two rows apart, and the hint about SQLite's lower() appears only for an expression key. Values --drop-nul changed in the key's columns are listed as rows to check first, and only when the duplicate value holds their text. An unrelated value that lost a NUL is no longer named. A plain key over a quoted column is named by its column, not as an expression.

A row that breaks a constraint (SQLSTATE class 23) is now a MoveRefusal that says what to correct, in the SQLite file or in a plugin's plugin.db. Before, it was a bare error with a stack. A value Postgres refuses by its type (class 22) stays a failure: it names no row, and values.ts should have refused it first, so the message asks for a bug report. MoveRefusal and the remedy text move to refusal.ts so copy.ts can use them without an import cycle. The usage text of --drop-nul and the wiki's move section describe NUL in JSON, the key listing and these refusals.
2026-11-18 13:20:08 +01:00
Maurice e66330e226 fix(db): treat a raw NUL in JSON like any NUL in the move
A raw U+0000 in TREK's JSON text, which an old or hand-edited travel.db can hold, was refused as "not valid JSON" without naming the NUL or the remedy. It is now refused with the NUL reason, so the refusal names --drop-nul. With --drop-nul, raw and escaped NULs are removed and nothing else, the way TREK's own writes drop them, and each change is listed. Text that is not valid JSON even without its NULs is refused as that, and the checksum's source side follows the same rule.

A \u escape of half a surrogate pair parses in JavaScript but jsonb refuses it. Before, that failed the insert without naming a row. It is now refused by table, row and column.
2026-11-18 13:20:07 +01:00
Maurice f2cfbc4041 fix(db): close the move's remaining silent changes
An attack on the move to Postgres with real SQLite files found five values
that changed without a refusal or a listing, or a valid file that failed:

- Negative zero in a plugin column became 0: pg sends the number -0 as "0",
  and the checksum printed both as "0". It is sent as text and kept apart in
  the checksum on both sides.
- Text whose bytes are not valid UTF-8 was read with each bad sequence
  replaced by U+FFFD, and the checksum was built from the replaced text.
  Text is now read as its bytes and decoded strictly in the file's encoding
  (UTF-8, or UTF-16 for a file created so); such a value is refused in
  TREK's and in plugin data, with --drop-nul too, and listed with its bytes.
- --drop-nul serialized a JSON value again, so a large number lost digits
  and 1.0 became 1. Only the \u0000 escapes are removed now.
- A plugin default Postgres refuses for the column (DEFAULT 0 on a BLOB
  column) failed the move, and DEFAULT 1.5 on an integer column would have
  been rounded. Such a default is left off and listed as not carried over.
- A plugin index or primary key SQLite compared with COLLATE NOCASE or RTRIM
  became one that compares exactly. It is still created, and the summary
  lists the collation it lost.

Each case has a regression test that fails without its fix, on SQLite and
on the Postgres lane.
2026-11-17 23:30:35 +00:00
Maurice bc6728678b fix(db): let the move to Postgres change nothing silently
The move stripped NUL from a plugin's own text, merging keys that were
distinct, and turned a plugin's values into another type (text in a BLOB
column into bytea, numbers in an untyped column into text). Its checksum
was built from the converted values, so both passed as verified.

A plugin's data now moves exactly as the Postgres plugin store would
hand it back, or not at all: a NUL, a value of another storage class
than its column, or an integer past 2^53 refuses the move before
anything is written, listing plugin, table, row and column. A column
left open by SQLite's affinity takes the kind its values need, so text
in a BLOB column stays text.

TREK's own text holding a NUL from an older install is refused by
default too. --drop-nul drops it, lists every changed value in the
summary, and names a unique key that dropping a NUL broke.

Every table's checksum is now built from the values read from SQLite,
so a conversion that changes a value fails the verification.
2026-11-17 23:28:41 +01:00
Maurice f42ef5246c fix(db): store no NUL character on either engine
Postgres text cannot hold U+0000, so any write or filter carrying one (a
request body, a query string, an import, a feed, a forwarded mail, a
plugin's own SQL) answered a 500 on Postgres while SQLite stored it. Both
engines now drop the character where every value leaves for the database
(src/db/nul-free.ts): the MikroORM platforms when they inline a string or
serialize a JSON value, the Kysely dialects for every parameter, a
plugin's own SQLite file for its arguments, and the move from SQLite,
which used to refuse such text. The same request now answers the same
status and leaves the same row on both engines, so the places NUL suite
runs on Postgres as well and leaves SQLITE_ONLY.
2026-11-17 23:04:08 +01:00
Maurice 3a4c523509 test(db): the move command's own output and target ORM 2026-11-17 23:03:58 +01:00
Maurice 55607ff2f9 feat(db): move a SQLite install into an empty Postgres database
npm run db:move-to-postgres (dist/move-to-postgres.js in the image) copies
a SQLite file at this build's migration level into an empty Postgres
database. It refuses a file at another level, rows whose references point
at nothing and a target holding any table or plugin schema, creates the
schema with the engine's own bootstrap, then in one transaction copies every
table in foreign key order (setting a self-reference once its rows are in),
converts each value to its column's type and refuses one Postgres cannot
hold exactly, copies each plugin's plugin.db into its schema, moves the
sequences past the copied ids and compares every table's row count and an
order-free checksum over all rows before it commits. A failure drops the
schema again. The SQLite files and the uploads are never written.
2026-11-17 23:03:54 +01:00