mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-11-20 11:06:06 +00:00
fix(db): close the move's remaining silent changes
An attack on the move to Postgres with real SQLite files found five values that changed without a refusal or a listing, or a valid file that failed: - Negative zero in a plugin column became 0: pg sends the number -0 as "0", and the checksum printed both as "0". It is sent as text and kept apart in the checksum on both sides. - Text whose bytes are not valid UTF-8 was read with each bad sequence replaced by U+FFFD, and the checksum was built from the replaced text. Text is now read as its bytes and decoded strictly in the file's encoding (UTF-8, or UTF-16 for a file created so); such a value is refused in TREK's and in plugin data, with --drop-nul too, and listed with its bytes. - --drop-nul serialized a JSON value again, so a large number lost digits and 1.0 became 1. Only the \u0000 escapes are removed now. - A plugin default Postgres refuses for the column (DEFAULT 0 on a BLOB column) failed the move, and DEFAULT 1.5 on an integer column would have been rounded. Such a default is left off and listed as not carried over. - A plugin index or primary key SQLite compared with COLLATE NOCASE or RTRIM became one that compares exactly. It is still created, and the summary lists the collation it lost. Each case has a regression test that fails without its fix, on SQLite and on the Postgres lane.
This commit is contained in:
@@ -128,15 +128,17 @@ TREK ships a command that copies a SQLite install into an empty Postgres databas
|
||||
|
||||
It prints a summary: the tables and rows it copied, the plugin schemas it filled, every value it changed with `--drop-nul`, and anything it could not carry over.
|
||||
|
||||
**NUL characters in TREK's data.** Postgres text cannot hold the character U+0000 (NUL). TREK no longer stores it on either engine and drops it on its next write, but a `travel.db` from an older version may still hold one. The command changes nothing silently, so by default it refuses such a value and lists it by table, row and column. To drop the character during the move, the way TREK would on its next write, run the command again with `--drop-nul`. The summary then lists every value it changed. If two values in a unique column differed only by a NUL, dropping it makes them equal. The command then stops, names the key and lists the values it changed in that column. Change or remove one of the two rows in the SQLite file, with TREK stopped, and run the command again.
|
||||
**NUL characters in TREK's data.** Postgres text cannot hold the character U+0000 (NUL). TREK no longer stores it on either engine and drops it on its next write, but a `travel.db` from an older version may still hold one. The command changes nothing silently, so by default it refuses such a value and lists it by table, row and column. To drop the character during the move, the way TREK would on its next write, run the command again with `--drop-nul`. The summary then lists every value it changed. If two values in a unique column differed only by a NUL, dropping it makes them equal. The command then stops, names the key and lists the values it changed in that column. Change or remove one of the two rows in the SQLite file, with TREK stopped, and run the command again. In JSON, `--drop-nul` removes only the escaped character (`\u0000`); everything else in the value stays as it was written.
|
||||
|
||||
**Text that is not UTF-8.** SQLite stores the bytes of a text value as it is given them, so a value written by hand with the `sqlite3` tool can hold bytes that are not valid UTF-8. Postgres text cannot hold such bytes, and reading them as text would replace them. The command refuses such a value, in TREK's data and in plugin data, with or without `--drop-nul`, and lists it by table, row and column with its bytes in hex. Correct the value in the SQLite file, with TREK stopped, and run the command again.
|
||||
|
||||
**Uploads are not moved and need not be.** Files, covers, photos and avatars stay where the storage driver keeps them (the `uploads` directory, or your S3 bucket), and the rows the command copies point at them as before. The key files in the data directory (`.encryption_key`, `.jwt_secret`) stay where they are too. Keep the same `ENCRYPTION_KEY`, or the same data directory, for the Postgres install: the stored secrets are copied as they are, encrypted with that key.
|
||||
|
||||
**Plugin data.** A plugin's `plugin.db` in `data/plugins-data/<plugin id>/` goes into the plugin's schema `trek_plugin_<plugin id>` (see [Plugin data](#plugin-data)). Each table is created from the SQLite one: the column types follow SQLite's type rules, with NOT NULL, the primary key, an INTEGER PRIMARY KEY as an identity column, defaults that have a Postgres spelling, and indexes over plain columns. The plugin's record of the migrations it ran is copied, so the plugin does not run them again. Foreign keys between a plugin's tables, its views and triggers, and partial or expression indexes are not created; the summary lists them. The plugin's own SQL has to be written for Postgres for it to keep working, as for any plugin on Postgres.
|
||||
**Plugin data.** A plugin's `plugin.db` in `data/plugins-data/<plugin id>/` goes into the plugin's schema `trek_plugin_<plugin id>` (see [Plugin data](#plugin-data)). Each table is created from the SQLite one: the column types follow SQLite's type rules, with NOT NULL, the primary key, an INTEGER PRIMARY KEY as an identity column, defaults that have a Postgres spelling for the column's type, and indexes over plain columns. The plugin's record of the migrations it ran is copied, so the plugin does not run them again. Foreign keys between a plugin's tables, its views and triggers, partial or expression indexes, and defaults the Postgres column would refuse or store as another value (such as `DEFAULT 0` on a `BLOB` column, or `DEFAULT 1.5` on an `INTEGER` one) are not created; the summary lists them. An index or primary key whose values SQLite compared ignoring case or trailing spaces (`COLLATE NOCASE`, `RTRIM`) is created, but Postgres compares them exactly, so the summary lists it too. The plugin's own SQL has to be written for Postgres for it to keep working, as for any plugin on Postgres.
|
||||
|
||||
A plugin's data is the plugin's, so the command never changes it: every value has to come back to the plugin from Postgres exactly as it does from its SQLite file. An integer or a decimal number comes back as a number, text as text and a BLOB as bytes. Where SQLite's type rules leave a column's type open (a column declared `BLOB` or without a type), the values it holds decide. The command refuses the move, before it writes anything, for each value that would come back changed, and lists the plugin, table, row and column:
|
||||
|
||||
- text holding a NUL character, which the Postgres plugin store cannot hold (`--drop-nul` never applies to plugin data);
|
||||
- text holding a NUL character, which the Postgres plugin store cannot hold (`--drop-nul` never applies to plugin data), or bytes that are not valid UTF-8;
|
||||
- a value of another type than its column, such as a number in a column that also holds text, which Postgres would hand back as text;
|
||||
- an integer beyond 2^53, which the Postgres plugin store refuses to read.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user