From 0c9e1cd639f962bbda43f1454dd8b0a468ff490c Mon Sep 17 00:00:00 2001 From: Maurice Date: Thu, 8 Oct 2026 17:42:07 +0200 Subject: [PATCH] fix(plugins): hold plugin SQL to a time budget between rows A plugin's own database runs on the host's event loop, and the only bounds were a regex denylist, the 256 MB quota and a 100,000 row cap. A select whose every row is expensive, or a long scan under the cap, could hold the loop for as long as it liked. query() and tx() now get a wall-clock budget of 2 s, read between the rows a statement yields and between the statements of a batch. Past it the call throws, and a tx rolls back whole like any other failure inside it. The row cap and the budget are overridable per PluginDataDb for tests; the defaults are what production runs. better-sqlite3 has no interrupt or progress handler, so a statement that yields nothing until it is done, such as an aggregate over a cross join, still runs to its first row. Only moving plugin SQL off the host thread can bound that; the SDK README and the plugin wiki state the budget and this limit. --- plugin-sdk/README.md | 2 +- plugin-sdk/src/index.ts | 4 +- server/CLAUDE.md | 2 +- .../nest/plugins/host/plugin-data.service.ts | 52 ++++++++++++++++--- server/tests/unit/plugins/plugin-data.test.ts | 48 +++++++++++++++++ wiki/Plugin-Development.md | 4 +- wiki/Plugin-Permissions.md | 2 +- 7 files changed, 102 insertions(+), 12 deletions(-) diff --git a/plugin-sdk/README.md b/plugin-sdk/README.md index 3ce979647..d0160430c 100644 --- a/plugin-sdk/README.md +++ b/plugin-sdk/README.md @@ -238,7 +238,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against | `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` | | `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` | | RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` | -| `ctx.db` (your own sqlite) | 256 MB per plugin | +| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count) and has 2 s of wall-clock time, checked between the rows it reads and the statements of a batch; past either it throws (`query exceeded its 2000 ms time budget`) and a `tx` rolls back. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement that produces no row until it is finished (an aggregate over a cross join of large tables), because SQLite runs it on TREK's main thread in one step: keep your queries indexed and bounded | | `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) | | Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes | | Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes | diff --git a/plugin-sdk/src/index.ts b/plugin-sdk/src/index.ts index c8ac0002f..1ca2191b5 100644 --- a/plugin-sdk/src/index.ts +++ b/plugin-sdk/src/index.ts @@ -72,7 +72,9 @@ export interface PluginContext { }; /** Your OWN sqlite database (`db:own`) — a separate file the plugin never gets a * path or connection to directly. Quota: 256 MB per plugin (writes past it fail); - * result sets are capped at 100,000 rows. See `tx()` below for the atomic-batch cap. */ + * result sets are capped at 100,000 rows, and a `query` or `tx` gets 2 s of wall-clock + * time, checked between the rows it reads (past it, it throws and a `tx` rolls back). + * See `tx()` below for the atomic-batch cap and README § Runtime limits. */ db: { query(sql: string, ...args: unknown[]): Promise; exec(sql: string, ...args: unknown[]): Promise<{ changes: number }>; diff --git a/server/CLAUDE.md b/server/CLAUDE.md index d6e1c517e..13e231800 100644 --- a/server/CLAUDE.md +++ b/server/CLAUDE.md @@ -48,7 +48,7 @@ Nest owns everything. Every domain is a DI module under `src/nest//` (`c - **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; `src/websocket.ts` is a re-export stub for old call sites. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change. - **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change. - **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire` in one `UnitOfWork` transaction, where a conditional UPDATE decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send. -- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest** — it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/` — run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity read (`trips.getById`, `costs.listMine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, so a new column on `trips`, `places`, `reservations` and the other whole-row tables fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` is generated from it). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. +- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest** — it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/` — run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity read (`trips.getById`, `costs.listMine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, so a new column on `trips`, `places`, `reservations` and the other whole-row tables fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` is generated from it). A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap and a time budget checked between rows. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together. - **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`. - **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate--