From d3be6fb2b8067b06eca4f5d0d08c1d1a2b81430a Mon Sep 17 00:00:00 2001 From: Maurice Date: Thu, 8 Oct 2026 18:02:49 +0200 Subject: [PATCH] fix(plugins): hold exec scripts to the time budget between statements The time budget covered query() and tx() only. exec() without bound args handed the plugin's script to Database#exec, which runs any number of statements in one call with nothing checked between them, so a long script held the host loop as long as it liked. An exec script now runs one statement at a time. sql-script.ts cuts it after every semicolon outside a string, a quoted identifier or a comment; a piece SQLite reports as incomplete input (a trigger body cut at its inner semicolon) is joined with the next until it prepares. The clock is read after every statement, and past the budget the call throws. As with Database#exec, the statements before that point stay applied, and an error is SQLite's own; a transaction the script opened itself is rolled back so the connection is not left inside it. exec() with bound args is one statement and stays as it was. A single statement cannot be bounded at all on the host thread, a write such as INSERT ... SELECT over a cross join included, and the SDK README and the wiki now say so next to the cross-join read. migrate() keeps running unbudgeted on purpose: a migration stopped part of the way is rolled back and fails the same way on every later start, leaving the plugin unable to update its schema. The ctx.db rows of the SDK README and the plugin wiki pages describe the budget for scripts, and the lines touched there and in plugin-sdk's db comment lose their em dashes. --- plugin-sdk/README.md | 2 +- plugin-sdk/src/index.ts | 8 +- server/CLAUDE.md | 2 +- .../nest/plugins/host/plugin-data.service.ts | 79 ++++++++++++++++--- server/src/nest/plugins/host/sql-script.ts | 66 ++++++++++++++++ server/tests/unit/plugins/plugin-data.test.ts | 53 ++++++++++++- server/tests/unit/plugins/sql-script.test.ts | 41 ++++++++++ wiki/Plugin-Development.md | 4 +- wiki/Plugin-Permissions.md | 2 +- 9 files changed, 235 insertions(+), 22 deletions(-) create mode 100644 server/src/nest/plugins/host/sql-script.ts create mode 100644 server/tests/unit/plugins/sql-script.test.ts diff --git a/plugin-sdk/README.md b/plugin-sdk/README.md index f5a715ef3..9b40e6fd0 100644 --- a/plugin-sdk/README.md +++ b/plugin-sdk/README.md @@ -252,7 +252,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against | `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` | | `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` | | RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` | -| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count) and has 2 s of wall-clock time, checked between the rows it reads and the statements of a batch; past either it throws (`query exceeded its 2000 ms time budget`) and a `tx` rolls back. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement that produces no row until it is finished (an aggregate over a cross join of large tables), because SQLite runs it on TREK's main thread in one step: keep your queries indexed and bounded | +| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops after the statement that ran out of time (the statements before it stay applied, as they always did, and a transaction the script opened itself is rolled back). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded | | `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) | | Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes | | Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes | diff --git a/plugin-sdk/src/index.ts b/plugin-sdk/src/index.ts index f98a6a701..99588e00d 100644 --- a/plugin-sdk/src/index.ts +++ b/plugin-sdk/src/index.ts @@ -70,10 +70,12 @@ export interface PluginContext { settings: { get(key: string): Promise; }; - /** Your OWN sqlite database (`db:own`) — a separate file the plugin never gets a + /** Your OWN sqlite database (`db:own`): a separate file the plugin never gets a * path or connection to directly. Quota: 256 MB per plugin (writes past it fail); - * result sets are capped at 100,000 rows, and a `query` or `tx` gets 2 s of wall-clock - * time, checked between the rows it reads (past it, it throws and a `tx` rolls back). + * result sets are capped at 100,000 rows, and a `query`, a `tx` or an `exec` script + * without args gets 2 s of wall-clock time, checked between the rows it reads and + * the statements it runs (past it, it throws, a `tx` rolls back and a script stops). + * A single statement and `migrate` are not timed. * See `tx()` below for the atomic-batch cap and README § Runtime limits. */ db: { query(sql: string, ...args: unknown[]): Promise; diff --git a/server/CLAUDE.md b/server/CLAUDE.md index 1e6316650..3b7017159 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -48,7 +48,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; `src/websocket.ts` is a re-export stub for old call sites. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change. - **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change. - **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire` in one `UnitOfWork` transaction, where a conditional UPDATE decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send. -- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap and a time budget checked between rows. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. +- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--