fix(db): close the move's remaining silent changes

An attack on the move to Postgres with real SQLite files found five values
that changed without a refusal or a listing, or a valid file that failed:

- Negative zero in a plugin column became 0: pg sends the number -0 as "0",
  and the checksum printed both as "0". It is sent as text and kept apart in
  the checksum on both sides.
- Text whose bytes are not valid UTF-8 was read with each bad sequence
  replaced by U+FFFD, and the checksum was built from the replaced text.
  Text is now read as its bytes and decoded strictly in the file's encoding
  (UTF-8, or UTF-16 for a file created so); such a value is refused in
  TREK's and in plugin data, with --drop-nul too, and listed with its bytes.
- --drop-nul serialized a JSON value again, so a large number lost digits
  and 1.0 became 1. Only the \u0000 escapes are removed now.
- A plugin default Postgres refuses for the column (DEFAULT 0 on a BLOB
  column) failed the move, and DEFAULT 1.5 on an integer column would have
  been rounded. Such a default is left off and listed as not carried over.
- A plugin index or primary key SQLite compared with COLLATE NOCASE or RTRIM
  became one that compares exactly. It is still created, and the summary
  lists the collation it lost.

Each case has a regression test that fails without its fix, on SQLite and
on the Postgres lane.
This commit is contained in:
Maurice
2026-11-17 23:30:35 +00:00
parent 636fc7d0a6
commit f2cfbc4041
9 changed files with 315 additions and 46 deletions
+5 -3
View File
@@ -127,9 +127,10 @@ function seedPluginData(root: string): void {
'create unique index notes_body on notes (body);' +
"insert into notes (body, score, raw) values ('first', 1.5, x'00ff'), ('second', null, null), ('gone', 2, null);" +
"delete from notes where body = 'gone';" +
// Text in a column declared BLOB, and numbers in a column without a type: both come back as the plugin stored them.
// Text in a column declared BLOB, and numbers in a column without a type: both come back as the plugin stored them,
// negative zero (which a column without a type keeps) with its sign.
'create table docs (id integer primary key, payload blob, n);' +
`insert into docs values (1, '{"a":1}', 5), (2, 'plain', 2.5);`,
`insert into docs values (1, '{"a":1}', 5), (2, 'plain', 2.5), (3, 'signed', -0.0);`,
);
plugin.close();
}
@@ -212,7 +213,7 @@ describe('moving a SQLite install to Postgres', () => {
schema: 'trek_plugin_move_notes',
tables: [
expect.objectContaining({ label: '_plugin_migrations', rows: 1 }),
expect.objectContaining({ label: 'docs', rows: 2 }),
expect.objectContaining({ label: 'docs', rows: 3 }),
expect.objectContaining({ label: 'notes', rows: 2 }),
],
}),
@@ -289,6 +290,7 @@ describe('moving a SQLite install to Postgres', () => {
expect(await store.query('select id, payload, n from docs order by id')).toEqual([
{ id: 1, payload: '{"a":1}', n: 5 },
{ id: 2, payload: 'plain', n: 2.5 },
{ id: 3, payload: 'signed', n: -0 },
]);
// The migration the plugin ran on SQLite is not run again, and the next id skips the one SQLite handed out.
expect(await store.migrate('001-notes', 'create table notes (id bigint)')).toEqual({ applied: false });