The move read unique indexes by joining indkey to pg_attribute. An expression key such as lower(email) has attnum 0 there, so idx_users_email_lower was missing from the plan. The catalog query now reads each index's key as Postgres prints it, whether any part of it is an expression, and the columns it reads: the plain key columns and the Var nodes of indexprs, without INCLUDE columns or the predicate.
A broken unique key no longer names --drop-nul as its cause. The refusal names the key and the shared value from Postgres's detail. It explains that the SQLite file held the two rows apart, and the hint about SQLite's lower() appears only for an expression key. Values --drop-nul changed in the key's columns are listed as rows to check first, and only when the duplicate value holds their text. An unrelated value that lost a NUL is no longer named. A plain key over a quoted column is named by its column, not as an expression.
A row that breaks a constraint (SQLSTATE class 23) is now a MoveRefusal that says what to correct, in the SQLite file or in a plugin's plugin.db. Before, it was a bare error with a stack. A value Postgres refuses by its type (class 22) stays a failure: it names no row, and values.ts should have refused it first, so the message asks for a bug report. MoveRefusal and the remedy text move to refusal.ts so copy.ts can use them without an import cycle. The usage text of --drop-nul and the wiki's move section describe NUL in JSON, the key listing and these refusals.
An attack on the move to Postgres with real SQLite files found five values
that changed without a refusal or a listing, or a valid file that failed:
- Negative zero in a plugin column became 0: pg sends the number -0 as "0",
and the checksum printed both as "0". It is sent as text and kept apart in
the checksum on both sides.
- Text whose bytes are not valid UTF-8 was read with each bad sequence
replaced by U+FFFD, and the checksum was built from the replaced text.
Text is now read as its bytes and decoded strictly in the file's encoding
(UTF-8, or UTF-16 for a file created so); such a value is refused in
TREK's and in plugin data, with --drop-nul too, and listed with its bytes.
- --drop-nul serialized a JSON value again, so a large number lost digits
and 1.0 became 1. Only the \u0000 escapes are removed now.
- A plugin default Postgres refuses for the column (DEFAULT 0 on a BLOB
column) failed the move, and DEFAULT 1.5 on an integer column would have
been rounded. Such a default is left off and listed as not carried over.
- A plugin index or primary key SQLite compared with COLLATE NOCASE or RTRIM
became one that compares exactly. It is still created, and the summary
lists the collation it lost.
Each case has a regression test that fails without its fix, on SQLite and
on the Postgres lane.
The plugin in the move suite gains a BLOB column holding text and an
untyped column holding numbers, and the plugin store must read both back
as they were stored. A new suite moves a file whose plugin and own rows
hold NUL: the plugin's refuses before anything is written, TREK's
refuses until --drop-nul, a username that dropping the NUL makes
duplicate is named with users_username_unique, and a move without a
collision lists the value it changed and stores it without the NUL.
The token guard kept leaking: on a superuser connection an unqualified
`alter type int4 rename to int4x` renamed pg_catalog's type for the whole
server, sequences and catalog-only objects grew the database past the quota,
regtype/regconfig/regdictionary looked TREK's objects up by name, and a
column alias named insert or merge in front of INTO made a table. Instead
of a longer word list, the store now asks the database:
- No plugin SQL while TREK's role is privileged in the sense of the restore
check (PRIVILEGED_ROLE_SQL, imported from server-checks.ts): every db:own
call fails with PRIVILEGED_ROLE_PLUGIN_REFUSAL, read once per pool
connection, and the boot log says why.
- The objects an ALTER, DROP, REFRESH, TRUNCATE or CREATE INDEX ... ON
changes, and the tables a CREATE or ALTER TABLE ties to its own, are
resolved with to_regclass/to_regtype in the transaction right before the
statement, and it is refused unless each one found lies in the plugin's
schema. An unqualified int4, text or box is pg_catalog's and refused even
with the role check off.
- The quota counts every relation of the schema with its TOAST table, and a
schema holds at most PLUGIN_SCHEMA_MAX_OBJECTS catalog objects (relations,
types, constraints, enum labels).
- The guard refuses every reg* type and its array in every spelling, the
has_*_privilege functions, currtid2 and the BRIN/GIN maintenance
functions; INTO only after the INSERT or MERGE that is the statement's
verb or a WITH body's; TEMP only where it makes a temporary object
(CREATE TEMP/TEMPORARY, SELECT ... INTO TEMP), so a column named temp
works again; a base type; and a NUL anywhere in the SQL text.
- A bound value or a migration id holding NUL is refused with the reason
instead of being stripped, so the plugin's keys never change meaning.
Every bypass from the re-review is a test on the real database, on the
role the plugin data is meant to run as and on a superuser connection with
the role check off.
Plugin statements on Postgres went over the extended query protocol through a
marker parameter and a hook that replaced query() on every client of TREK's
pool. That kept a plugin to one statement per round trip, but it changed
TREK's own hot path for a concern only plugin data has.
The plugin data store now checks a pg client out of the ORM's pool for each
transaction, through MikroORM's getNativeClient, and runs the transaction on
it directly. Every plugin write goes as a query with queryMode 'extended' and
every read through a pg-cursor on that client, so Postgres parses exactly one
statement either way. The settings the transaction makes are local, and
before the client goes back to the pool the store drops temporary tables,
cursors, listeners, session advisory locks and sequence state, or closes a
client it cannot bring back to that state. The row cap, quota, time budget,
untimed migrate, per-plugin queue, pool share and script commits are as
before, and plugin args are still sent without NUL.
The marker, the hook and its install in the Postgres driver are gone.