mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-18 21:16:47 +00:00
fix(plugins): hold exec scripts to the time budget between statements
The time budget covered query() and tx() only. exec() without bound args handed the plugin's script to Database#exec, which runs any number of statements in one call with nothing checked between them, so a long script held the host loop as long as it liked. An exec script now runs one statement at a time. sql-script.ts cuts it after every semicolon outside a string, a quoted identifier or a comment; a piece SQLite reports as incomplete input (a trigger body cut at its inner semicolon) is joined with the next until it prepares. The clock is read after every statement, and past the budget the call throws. As with Database#exec, the statements before that point stay applied, and an error is SQLite's own; a transaction the script opened itself is rolled back so the connection is not left inside it. exec() with bound args is one statement and stays as it was. A single statement cannot be bounded at all on the host thread, a write such as INSERT ... SELECT over a cross join included, and the SDK README and the wiki now say so next to the cross-join read. migrate() keeps running unbudgeted on purpose: a migration stopped part of the way is rolled back and fails the same way on every later start, leaving the plugin unable to update its schema. The ctx.db rows of the SDK README and the plugin wiki pages describe the budget for scripts, and the lines touched there and in plugin-sdk's db comment lose their em dashes.
This commit is contained in:
Maurice
@@ -252,7 +252,7 @@ for a legitimate plugin and only bite a runaway or abusive one — build against
|
||||
| `ctx.ai` | 200 calls/day per plugin (UTC midnight rollover); past it throws `"daily AI budget exhausted (resets at UTC midnight)"` |
|
||||
| `ctx.notify` | 100 calls/day per plugin (UTC midnight rollover); past it throws `"daily notification budget exhausted (resets at UTC midnight)"` |
|
||||
| RPC (every `ctx.*` call) | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls` |
|
||||
| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count) and has 2 s of wall-clock time, checked between the rows it reads and the statements of a batch; past either it throws (`query exceeded its 2000 ms time budget`) and a `tx` rolls back. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement that produces no row until it is finished (an aggregate over a cross join of large tables), because SQLite runs it on TREK's main thread in one step: keep your queries indexed and bounded |
|
||||
| `ctx.db` (your own sqlite) | 256 MB per plugin; a `query` or `tx` returns at most 100,000 rows (a whole `tx` batch shares one count). A `query`, a whole `tx` batch and an `exec` script without bound args each have 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script; past either limit the call throws (`query exceeded its 2000 ms time budget`), a `tx` rolls back, and an `exec` script stops after the statement that ran out of time (the statements before it stay applied, as they always did, and a transaction the script opened itself is rolled back). Split a longer script over several calls, or batch it with `tx`. `migrate` has no time budget, since a migration stopped part of the way would fail again on every start. `ATTACH`, `DETACH`, `VACUUM`, `PRAGMA`, `WITH RECURSIVE` and `load_extension` are refused. The budget cannot stop a single statement, because SQLite runs it on TREK's main thread in one step: a write runs to its end (an `INSERT ... SELECT` over a cross join of large tables), and a read runs until its first row (an aggregate over one). Keep your statements indexed and bounded |
|
||||
| `ctx.meta` | 64 KB per value, 256 chars per key, 100 keys per (plugin, entity) |
|
||||
| Plugin process | 300 MB RSS ceiling; auto-disabled after 5 crashes in 5 minutes |
|
||||
| Event redelivery buffer | 200 events held per plugin, dropped unreplayed after 15 minutes |
|
||||
|
||||
@@ -70,10 +70,12 @@ export interface PluginContext {
|
||||
settings: {
|
||||
get(key: string): Promise<unknown>;
|
||||
};
|
||||
/** Your OWN sqlite database (`db:own`) — a separate file the plugin never gets a
|
||||
/** Your OWN sqlite database (`db:own`): a separate file the plugin never gets a
|
||||
* path or connection to directly. Quota: 256 MB per plugin (writes past it fail);
|
||||
* result sets are capped at 100,000 rows, and a `query` or `tx` gets 2 s of wall-clock
|
||||
* time, checked between the rows it reads (past it, it throws and a `tx` rolls back).
|
||||
* result sets are capped at 100,000 rows, and a `query`, a `tx` or an `exec` script
|
||||
* without args gets 2 s of wall-clock time, checked between the rows it reads and
|
||||
* the statements it runs (past it, it throws, a `tx` rolls back and a script stops).
|
||||
* A single statement and `migrate` are not timed.
|
||||
* See `tx()` below for the atomic-batch cap and README § Runtime limits. */
|
||||
db: {
|
||||
query<T = unknown>(sql: string, ...args: unknown[]): Promise<T[]>;
|
||||
|
||||
+1
-1
@@ -48,7 +48,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
|
||||
- **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; `src/websocket.ts` is a re-export stub for old call sites. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change.
|
||||
- **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `<domain>.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change.
|
||||
- **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire` in one `UnitOfWork` transaction, where a conditional UPDATE decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send.
|
||||
- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap and a time budget checked between rows. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together.
|
||||
- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest**; it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/`: run `gen:plugin-facts` after touching them and commit the output. What a plugin gets back is `protocol/output-contract.ts`: every wire method has an entry, and an entity result (`trips.getById`, `todos.list`, `vacay.mine`, ...) is cut down to its published fields in `rpc-host.ts` on the way out, together with every child row it carries whose field is listed in `PLUGIN_ENTITY_NESTED` (a day's `notes_items`, a reservation's `endpoints`). A new column on any whole-row table behind those entities, child tables included, fails `tests/unit/plugins/output-contract.schema.test.ts` until it is listed as published or withheld (the SDK's `PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` are generated from it); a method left as `readModel` must return a value built field by field from named columns. A plugin's own SQL (`host/plugin-data.service.ts`) is held to a row cap, and `query`, `tx` and an `exec` script to a time budget checked between rows and statements (`migrate` is deliberately not timed). `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together.
|
||||
- **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`.
|
||||
- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate-<schema>-<time>.db`, `<schema>` being `legacy-<schema_version>` or the last recorded migration's timestamp; the newest `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` stay) and the newer-database refusal names the matching copy; a copy that cannot be written refuses the boot unless `TREK_DB_PRE_MIGRATE_SNAPSHOT=false`. It is off under `NODE_ENV=test` unless a suite sets that variable to true, and a restore's re-bootstrap skips it (the archive is the copy). After the migrator, `db/email-case-index.ts` creates the case-insensitive unique index on `users.email` that its migration had to defer on an install holding two accounts whose emails differ only in case (flag `email_case_index_pending`). `db/orm-driver.ts` binds the ORM to that same connection instead of opening a second one — with `:memory:` under test, a second connection is a second, empty database. Kysely statements (`this.kysely<...>()`) type against `DB` from `db/kysely/db.ts`, narrowed with `Pick<DB, 'trips' | 'days'>`; those files are generated from the migrated schema by `scripts/generate-db-types.ts`, so a migration that adds or changes a column is followed by `npm run gen:db-types`, never by a hand-written table interface. Every read and write goes through a repository in `db/repositories/` (each extends the shared `TrekRepository` base in `db/repositories/_shared/`, which validates the request context before delegating to MikroORM) and, for multi-statement writes, `UnitOfWork.transactional(...)` (`nest/database/unit-of-work.ts`, provided by the global `OrmModule`) — there is no `DatabaseService`/`DatabaseModule` to inject any more. Entrypoints outside an HTTP request — an MCP tool call, a plugin RPC dispatch, a WebSocket message, a cron job, boot-time seeding, a backup restore — open one explicitly with `withRequestContext()` (`nest/database/request-context.ts`), since only the pathless `mikroOrmRequestContext` middleware `buildApp()` mounts (with `@mikro-orm/nestjs`'s own `registerRequestContext` turned off in `app.module.ts`, UPLOADS-P16) forks an `EntityManager` per HTTP request. `NODE_ENV=test` ⇒ isolated `:memory:` DB per vitest worker, opened from a snapshot `tests/global-setup.ts` migrates once per run; `TREK_DB_FILE` ⇒ explicit file DB (Playwright harness).
|
||||
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import type { Database as Db } from 'better-sqlite3';
|
||||
import type { Database as Db, Statement } from 'better-sqlite3';
|
||||
import { openDatabase } from '../../../db/connection';
|
||||
import { pluginDataDir, pluginDbFile, pluginsDataRoot } from '../paths';
|
||||
import { splitAfterSemicolons } from './sql-script';
|
||||
|
||||
/**
|
||||
* A plugin's own sqlite database (#plugins, db:own). The HOST owns the handle;
|
||||
@@ -13,7 +14,8 @@ import { pluginDataDir, pluginDbFile, pluginsDataRoot } from '../paths';
|
||||
*
|
||||
* A thin guard still rejects statements that would let a plugin escape its file
|
||||
* (ATTACH another db, VACUUM INTO elsewhere, PRAGMA fiddling) or DoS via
|
||||
* oversize SQL, and every read is held to a row cap and a time budget (below).
|
||||
* oversize SQL, every read is held to a row cap, and query(), tx() and an exec()
|
||||
* script are held to a time budget (below).
|
||||
*/
|
||||
|
||||
const MAX_SQL_LENGTH = 100_000;
|
||||
@@ -41,15 +43,23 @@ const MAX_ROWS = 100_000;
|
||||
// Cap statements per atomic batch so a single tx() can't monopolise the synchronous
|
||||
// host — generous for real write batches, far below anything abusive.
|
||||
const MAX_TX_OPS = 100;
|
||||
// Wall-clock budget for one query() or one whole tx() batch. better-sqlite3 has no
|
||||
// interrupt or progress handler, so the budget is checked between the rows a statement
|
||||
// yields (and between the statements of a batch): a scan that keeps producing rows, or
|
||||
// a select whose every row is expensive, stops at the first row past the budget
|
||||
// instead of holding the host loop until the row cap. A statement that yields nothing
|
||||
// until it is done (an aggregate over a cartesian product) still runs to its first
|
||||
// row; only moving plugin SQL off the host thread can bound that, and the plugin-sdk
|
||||
// README says so.
|
||||
// Wall-clock budget for one query(), one whole tx() batch or one exec() script.
|
||||
// better-sqlite3 has no interrupt or progress handler, so the budget is checked
|
||||
// between the rows a statement yields and between the statements of a batch or a
|
||||
// script: a scan that keeps producing rows, a select whose every row is expensive, or
|
||||
// a script of many statements stops at the first row or statement past the budget
|
||||
// instead of holding the host loop. One statement on its own still runs to the end
|
||||
// (a write such as `INSERT … SELECT` over a cartesian product) or to its first row
|
||||
// (an aggregate over one); only moving plugin SQL off the host thread can bound that,
|
||||
// and the plugin-sdk README says so. migrate() is left unbudgeted on purpose: a
|
||||
// migration is stopped part of the way only to fail the same way on every later
|
||||
// start, which leaves a plugin that cannot update its schema at all.
|
||||
const TIME_BUDGET_MS = 2_000;
|
||||
// What better-sqlite3 throws from prepare() for a statement cut off before its end:
|
||||
// in an exec() script, a trigger body whose semicolon split it.
|
||||
const INCOMPLETE_INPUT = 'incomplete input';
|
||||
// And for a piece that holds no statement at all, only whitespace or comments.
|
||||
const NO_STATEMENT = 'The supplied SQL string contains no statements';
|
||||
|
||||
/** The caps a PluginDataDb enforces. Overridable for tests; production uses the defaults. */
|
||||
export interface PluginDataLimits {
|
||||
@@ -129,17 +139,62 @@ export class PluginDataDb {
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** Write statement(s). exec() allows multiple statements (e.g. a small setup script). */
|
||||
/**
|
||||
* Write statement(s). With bound args, exec() runs one statement. Without, it runs a
|
||||
* script of any number (e.g. a small setup script), one statement at a time and
|
||||
* within the time budget, checked after each statement.
|
||||
*/
|
||||
exec(sql: string, args: unknown[] = []): { changes: number } {
|
||||
this.guard(sql);
|
||||
if (args.length > 0) {
|
||||
const info = this.db.prepare(sql).run(...(args as never[]));
|
||||
return { changes: info.changes };
|
||||
}
|
||||
this.db.exec(sql);
|
||||
this.runScript(sql, this.startBudget('exec'));
|
||||
return { changes: 0 };
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a script the way `Database#exec` did (statement after statement, each in
|
||||
* autocommit unless the script opens a transaction), with `checkTime` read after
|
||||
* every statement. A trigger body is gathered until SQLite stops calling it
|
||||
* incomplete. A script that runs out of time inside a transaction it opened itself
|
||||
* has that transaction rolled back, so the connection is not left holding it.
|
||||
*/
|
||||
private runScript(sql: string, checkTime: () => void): void {
|
||||
const openedNoTransaction = !this.db.inTransaction;
|
||||
let pending = '';
|
||||
for (const piece of splitAfterSemicolons(sql)) {
|
||||
pending += piece;
|
||||
const stmt = this.prepareComplete(pending);
|
||||
if (stmt === 'incomplete') continue;
|
||||
pending = '';
|
||||
if (stmt === 'empty') continue;
|
||||
stmt.run();
|
||||
try {
|
||||
checkTime();
|
||||
} catch (e) {
|
||||
if (openedNoTransaction && this.db.inTransaction) this.db.prepare('ROLLBACK').run();
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
// A statement still open at the end of the script: prepare it once more for
|
||||
// SQLite's own error, as Database#exec reports it.
|
||||
if (pending !== '') this.db.prepare(pending);
|
||||
}
|
||||
|
||||
/** `sql` prepared, or why it cannot be yet: cut off before its end, or no statement at all. */
|
||||
private prepareComplete(sql: string): Statement | 'incomplete' | 'empty' {
|
||||
try {
|
||||
return this.db.prepare(sql);
|
||||
} catch (e) {
|
||||
const message = e instanceof Error ? e.message : '';
|
||||
if (message === INCOMPLETE_INPUT) return 'incomplete';
|
||||
if (message === NO_STATEMENT) return 'empty';
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomic batch on the plugin's OWN db: every op runs in a single transaction, so
|
||||
* they all commit or all roll back — the primitive a plugin needs for a consistent
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* Cuts a SQL script after each semicolon that ends a statement: one outside a quoted
|
||||
* string or identifier ('…', "…", `…`, […]) and outside a comment. The pieces keep
|
||||
* their text, so joining them gives the script back.
|
||||
*
|
||||
* A semicolon inside a trigger body (`CREATE TRIGGER … BEGIN …; END;`) ends a piece
|
||||
* here as well. Telling it apart takes SQLite's own parser, so the caller does that:
|
||||
* it prepares the pieces it has gathered so far and keeps gathering while SQLite
|
||||
* reports them as incomplete input.
|
||||
*
|
||||
* Pure, no imports: plugin-data.service.ts runs a plugin's exec() script through it,
|
||||
* one statement at a time, so the time budget can be checked between statements.
|
||||
*/
|
||||
export function splitAfterSemicolons(sql: string): string[] {
|
||||
const pieces: string[] = [];
|
||||
let start = 0;
|
||||
let i = 0;
|
||||
while (i < sql.length) {
|
||||
const skipTo = endOfQuotedOrComment(sql, i);
|
||||
if (skipTo !== i) {
|
||||
i = skipTo;
|
||||
continue;
|
||||
}
|
||||
i += 1;
|
||||
if (sql[i - 1] === ';') {
|
||||
pieces.push(sql.slice(start, i));
|
||||
start = i;
|
||||
}
|
||||
}
|
||||
if (start < sql.length) pieces.push(sql.slice(start));
|
||||
return pieces;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where the quoted string, quoted identifier or comment starting at `i` ends (the index
|
||||
* after it), or `i` itself when none starts there. An unterminated one runs to the end.
|
||||
*/
|
||||
function endOfQuotedOrComment(sql: string, i: number): number {
|
||||
const c = sql[i];
|
||||
if (c === "'" || c === '"' || c === '`') return endOfQuoted(sql, i, c);
|
||||
if (c === '[') return endAfter(sql, ']', i + 1);
|
||||
if (c === '-' && sql[i + 1] === '-') return endAfter(sql, '\n', i + 2);
|
||||
if (c === '/' && sql[i + 1] === '*') return endAfter(sql, '*/', i + 2);
|
||||
return i;
|
||||
}
|
||||
|
||||
/** The index after `token`'s first occurrence from `from`, or the end of `sql`. */
|
||||
function endAfter(sql: string, token: string, from: number): number {
|
||||
const at = sql.indexOf(token, from);
|
||||
return at < 0 ? sql.length : at + token.length;
|
||||
}
|
||||
|
||||
/** A quote doubled inside the quoted text stands for itself and does not close it. */
|
||||
function endOfQuoted(sql: string, i: number, quote: string): number {
|
||||
let j = i + 1;
|
||||
while (j < sql.length) {
|
||||
if (sql[j] !== quote) {
|
||||
j += 1;
|
||||
} else if (sql[j + 1] === quote) {
|
||||
j += 2;
|
||||
} else {
|
||||
return j + 1;
|
||||
}
|
||||
}
|
||||
return sql.length;
|
||||
}
|
||||
@@ -110,8 +110,8 @@ describe('PluginDataDb', () => {
|
||||
const db = new PluginDataDb('budget-query', { timeBudgetMs: 100, now: steppingClock(30) });
|
||||
db.exec('CREATE TABLE seq (n INTEGER)');
|
||||
db.exec('INSERT INTO seq (n) VALUES ' + Array.from({ length: 50 }, (_, i) => `(${i})`).join(','));
|
||||
// The start reads 30 and each row reads again (60, 90, 120, 150): at the fourth
|
||||
// row the call is 120 ms in, past the budget.
|
||||
// The start reads the clock and each row reads it again, 30 ms later every time:
|
||||
// at the fourth row the call is 120 ms in, past the budget.
|
||||
expect(() => db.query('SELECT n FROM seq')).toThrow('query exceeded its 100 ms time budget');
|
||||
// A query that finishes inside the budget is untouched.
|
||||
expect(db.query('SELECT n FROM seq WHERE n < 2 ORDER BY n')).toEqual([{ n: 0 }, { n: 1 }]);
|
||||
@@ -138,6 +138,55 @@ describe('PluginDataDb', () => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('holds an exec script to the time budget between its statements', () => {
|
||||
const db = new PluginDataDb('budget-exec', { timeBudgetMs: 80, now: steppingClock(30) });
|
||||
db.exec('CREATE TABLE log (n INTEGER)');
|
||||
// Start, then one read after each statement: 30, 60, then 90 ms after the third
|
||||
// insert. The script stops there and the fourth insert never runs. Each statement
|
||||
// committed on its own, as Database#exec ran them, so the three stay.
|
||||
expect(() =>
|
||||
db.exec('INSERT INTO log VALUES (1); INSERT INTO log VALUES (2); INSERT INTO log VALUES (3); INSERT INTO log VALUES (4);'),
|
||||
).toThrow('exec exceeded its 80 ms time budget');
|
||||
// One row back, so the check itself reads the clock once and stays in budget.
|
||||
expect(db.query('SELECT group_concat(n) AS ns FROM (SELECT n FROM log ORDER BY n)')).toEqual([{ ns: '1,2,3' }]);
|
||||
// A script that opened its own transaction and ran out of time inside it is rolled
|
||||
// back, so the connection is not left in that transaction.
|
||||
expect(() =>
|
||||
db.exec('BEGIN; INSERT INTO log VALUES (5); INSERT INTO log VALUES (6); INSERT INTO log VALUES (7); COMMIT;'),
|
||||
).toThrow(/time budget/);
|
||||
expect(db.query('SELECT count(*) AS c FROM log WHERE n > 4')).toEqual([{ c: 0 }]);
|
||||
db.exec('INSERT INTO log VALUES (8)');
|
||||
expect(db.query('SELECT count(*) AS c FROM log WHERE n > 4')).toEqual([{ c: 1 }]);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('runs an exec script statement by statement with the semantics Database#exec had', () => {
|
||||
const db = new PluginDataDb('exec-script');
|
||||
db.exec(`
|
||||
-- a setup script: comments, a semicolon in a string, and a trigger body
|
||||
CREATE TABLE notes (id INTEGER PRIMARY KEY, body TEXT); /* one; two */
|
||||
CREATE TABLE audit (note_id INTEGER, what TEXT);
|
||||
CREATE TRIGGER notes_ai AFTER INSERT ON notes BEGIN
|
||||
INSERT INTO audit (note_id, what) VALUES (new.id, 'added;');
|
||||
INSERT INTO audit (note_id, what) VALUES (new.id, 'twice');
|
||||
END;
|
||||
INSERT INTO notes (body) VALUES ('it''s; fine');
|
||||
`);
|
||||
expect(db.query('SELECT body FROM notes')).toEqual([{ body: "it's; fine" }]);
|
||||
expect(db.query('SELECT what FROM audit ORDER BY rowid')).toEqual([{ what: 'added;' }, { what: 'twice' }]);
|
||||
// A script without a trailing semicolon, an empty one, and one of comments only.
|
||||
db.exec("INSERT INTO notes (body) VALUES ('last')");
|
||||
db.exec('');
|
||||
db.exec(' -- nothing to run\n/* nor here */');
|
||||
expect(db.query('SELECT count(*) AS n FROM notes')).toEqual([{ n: 2 }]);
|
||||
// The statements before a failing one have run, and the failure is SQLite's own.
|
||||
expect(() => db.exec("INSERT INTO notes (body) VALUES ('x'); INSERT INTO missing VALUES (1);")).toThrow(/no such table/);
|
||||
expect(db.query('SELECT count(*) AS n FROM notes')).toEqual([{ n: 3 }]);
|
||||
// A statement left open at the end reports what Database#exec reported.
|
||||
expect(() => db.exec('CREATE TRIGGER t AFTER INSERT ON notes BEGIN SELECT 1;')).toThrow('incomplete input');
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('applies an overridden row cap to query and tx alike', () => {
|
||||
const db = new PluginDataDb('budget-rows', { maxRows: 3 });
|
||||
db.exec('CREATE TABLE seq (n INTEGER)');
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
/**
|
||||
* splitAfterSemicolons: where a plugin's exec() script is cut into statements, so
|
||||
* the time budget can be read between them (plugin-data.service.ts).
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { splitAfterSemicolons } from '../../../src/nest/plugins/host/sql-script';
|
||||
|
||||
describe('splitAfterSemicolons', () => {
|
||||
it('SQLSCRIPT-001 cuts after each statement and keeps every character', () => {
|
||||
const sql = 'CREATE TABLE a (x); INSERT INTO a VALUES (1);\nSELECT 1';
|
||||
const pieces = splitAfterSemicolons(sql);
|
||||
expect(pieces).toEqual(['CREATE TABLE a (x);', ' INSERT INTO a VALUES (1);', '\nSELECT 1']);
|
||||
expect(pieces.join('')).toBe(sql);
|
||||
});
|
||||
|
||||
it('SQLSCRIPT-002 leaves the semicolons in strings, quoted identifiers and comments alone', () => {
|
||||
expect(splitAfterSemicolons("INSERT INTO t VALUES ('a;b', 'it''s;');")).toEqual(["INSERT INTO t VALUES ('a;b', 'it''s;');"]);
|
||||
expect(splitAfterSemicolons('SELECT "x;y", `p;q`, [r;s] FROM t;')).toEqual(['SELECT "x;y", `p;q`, [r;s] FROM t;']);
|
||||
expect(splitAfterSemicolons('SELECT 1; -- one; two\nSELECT 2;')).toEqual(['SELECT 1;', ' -- one; two\nSELECT 2;']);
|
||||
expect(splitAfterSemicolons('SELECT 1 /* a; b */;')).toEqual(['SELECT 1 /* a; b */;']);
|
||||
});
|
||||
|
||||
it('SQLSCRIPT-003 cuts inside a trigger body, which the caller joins back up', () => {
|
||||
expect(splitAfterSemicolons('CREATE TRIGGER t AFTER INSERT ON a BEGIN SELECT 1; END;')).toEqual([
|
||||
'CREATE TRIGGER t AFTER INSERT ON a BEGIN SELECT 1;',
|
||||
' END;',
|
||||
]);
|
||||
});
|
||||
|
||||
it('SQLSCRIPT-004 runs an unterminated string, identifier or comment to the end', () => {
|
||||
expect(splitAfterSemicolons("SELECT 'open; still")).toEqual(["SELECT 'open; still"]);
|
||||
expect(splitAfterSemicolons('SELECT [open; still')).toEqual(['SELECT [open; still']);
|
||||
expect(splitAfterSemicolons('SELECT 1 -- trailing; comment')).toEqual(['SELECT 1 -- trailing; comment']);
|
||||
expect(splitAfterSemicolons('SELECT 1 /* open; comment')).toEqual(['SELECT 1 /* open; comment']);
|
||||
});
|
||||
|
||||
it('SQLSCRIPT-005 gives nothing for an empty script and a piece per stray semicolon', () => {
|
||||
expect(splitAfterSemicolons('')).toEqual([]);
|
||||
expect(splitAfterSemicolons(';;')).toEqual([';', ';']);
|
||||
});
|
||||
});
|
||||
@@ -280,7 +280,7 @@ exports the list per method as `PLUGIN_METHOD_RESULT`:
|
||||
|
||||
| Area | Methods | Requires |
|
||||
|---|---|---|
|
||||
| `ctx.db` | `query(sql, …args)` / `exec(sql, …args)` / `migrate(id, sql)` / `tx(ops)` against your **own** SQLite file. `tx([{sql, args?}, …])` runs up to 100 statements in one transaction (all commit or all roll back; reads see the batch's own earlier writes) → `{ results: [{changes?}\|{rows?}, …] }`. Your file is capped at **256 MB** (a write past it fails `SQLITE_FULL`, contained to your plugin), a single result set at **100,000 rows** and a `query` or `tx` at **2 s** (see [Runtime limits](#runtime-limits)) — page your reads instead of materialising a cartesian product | `db:own` |
|
||||
| `ctx.db` | `query(sql, …args)` / `exec(sql, …args)` / `migrate(id, sql)` / `tx(ops)` against your **own** SQLite file. `tx([{sql, args?}, …])` runs up to 100 statements in one transaction (all commit or all roll back; reads see the batch's own earlier writes) → `{ results: [{changes?}\|{rows?}, …] }`. Your file is capped at **256 MB** (a write past it fails `SQLITE_FULL`, contained to your plugin), a single result set at **100,000 rows** and a `query`, a `tx` or an `exec` script at **2 s** (see [Runtime limits](#runtime-limits)): page your reads instead of materialising a cartesian product, and split long scripts | `db:own` |
|
||||
| `ctx.trips` | `getById` / `getPlaces` / `getReservations` / `getDays` / `getAccommodations` / `listMine()` — enumerate every trip the acting user can access (membership-checked). `getDays` includes each day's `assignments` + `notes_items`; `getReservations` includes `endpoints` + `day_positions` | `db:read:trips` |
|
||||
| `ctx.trips.update(tripId, fields)` | update trip fields (title/dates/currency/reminder_days/…) | `db:write:trips` |
|
||||
| `ctx.trips.create(input)` | create a **new trip owned by the acting user** (importers) — `title` required, plus `description?`/`start_date?`/`end_date?`/`currency?`/`reminder_days?`/`day_count?`; without `currency` the trip takes the acting user's display currency, then the instance default, then EUR | `db:create:trips` (+ `trip_create`) |
|
||||
@@ -412,7 +412,7 @@ stores blobs in `ctx.db` runs into, so build against them.
|
||||
| Area | Limit |
|
||||
|---|---|
|
||||
| every `ctx.*` call | burst 60, sustained 20/s, 16 in-flight per plugin; a throttled call is refused with `HOST_ERROR: rate limit exceeded — slow down ctx.* calls`. Tunable with `TREK_PLUGIN_RPC_BURST` / `TREK_PLUGIN_RPC_PER_SEC` / `TREK_PLUGIN_RPC_INFLIGHT` (see [Environment-Variables](Environment-Variables#plugins)) |
|
||||
| `ctx.db` | 256 MB per plugin; a `query` or a whole `tx` batch returns at most 100,000 rows and gets 2 s of wall-clock time, checked between the rows it reads and the statements of a batch (past it the call throws `query exceeded its 2000 ms time budget`, and a `tx` rolls back). A single statement that yields nothing until it is finished, such as an aggregate over a cross join of large tables, cannot be stopped midway, because SQLite runs it on TREK's main thread: keep your queries indexed and bounded |
|
||||
| `ctx.db` | 256 MB per plugin; a `query` or a whole `tx` batch returns at most 100,000 rows. A `query`, a whole `tx` batch and an `exec` script without bound args each get 2 s of wall-clock time, checked between the rows a statement reads and between the statements of a batch or script (past it the call throws `query exceeded its 2000 ms time budget`, a `tx` rolls back, and an `exec` script stops after the statement that ran out of time, keeping the ones before it and rolling back a transaction the script opened itself). `migrate` is not timed, since a migration stopped part of the way would fail again on every start. A single statement cannot be stopped midway, because SQLite runs it on TREK's main thread: a write such as `INSERT ... SELECT` over a cross join of large tables runs to its end, and a read that yields nothing until it is finished (an aggregate over one) runs to its first row. Keep your statements indexed and bounded |
|
||||
| event subscriptions | 200 events buffered per plugin while it restarts, dropped unreplayed after 15 minutes |
|
||||
| the plugin process | 300 MB RSS (`TREK_PLUGIN_MAX_RSS_MB`) — the child is killed past it; auto-disabled with status `error` after 5 crashes inside a 5-minute window |
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ ungranted capability is physically unreachable**, not just disallowed. See
|
||||
|
||||
| Permission | Grants | Notes |
|
||||
|---|---|---|
|
||||
| `db:own` | Read/write the plugin's **own** SQLite file via `ctx.db` — `db.query`, `db.exec`, **and `db.migrate`** | A separate `plugin.db` per plugin under the plugin's data dir — never TREK's own `travel.db`. `db.migrate` runs a keyed, idempotent migration (schema/table creation, e.g. `CREATE TABLE`) once per id. `ATTACH`/`DETACH`/`VACUUM`/`PRAGMA`/`WITH RECURSIVE` are refused, and a `query` or `tx` stops at 100,000 rows or 2 s. |
|
||||
| `db:own` | Read/write the plugin's **own** SQLite file via `ctx.db`: `db.query`, `db.exec`, `db.tx` **and `db.migrate`** | A separate `plugin.db` per plugin under the plugin's data dir, never TREK's own `travel.db`. `db.migrate` runs a keyed, idempotent migration (schema/table creation, e.g. `CREATE TABLE`) once per id. `ATTACH`/`DETACH`/`VACUUM`/`PRAGMA`/`WITH RECURSIVE` are refused, a `query` or `tx` stops at 100,000 rows, and a `query`, `tx` or `exec` script stops at 2 s. |
|
||||
| `db:read:trips` | Read-only trip data via `ctx.trips` (`getById`, `getPlaces`, `getReservations`, `getDays`, `getAccommodations`, `listMine`, `members`) | Every call is **membership-checked** against the acting user: a plugin can't read a trip that user can't see. `getDays` returns each day with its `assignments` + `notes_items`; `getReservations` rows carry `endpoints` + `day_positions` like the REST list. `members` returns the roster (id + display fields only). Each row, and each row nested in it such as a day's notes or a reservation's endpoints, carries the fields TREK publishes for it (`PLUGIN_ENTITY_FIELDS` and `PLUGIN_ENTITY_NESTED` in the SDK), never the trip's calendar `feed_token`, and a column added in a later release is not delivered until it is published. |
|
||||
| `db:read:users` | Read-only public profile via `ctx.users.getById` | Returns id, username, display name, avatar only — **never** password hashes, tokens, or secrets. |
|
||||
| `db:read:packing` | Read-only packing items of a trip via `ctx.packing.list(tripId)` | Membership-checked, and scoped to the acting user's visibility — a plugin never sees another member's private packing items. |
|
||||
|
||||
Reference in New Issue
Block a user