feat(server): take a database lease before a cron tick runs

waitForCompletion only kept a tick from overlapping itself inside one
process. Two processes on the same database (a rolling update's overlap,
a second replica, a dev container pointed at the same file) both ran
every job: two auto-backups, double syncs and retention sweeps, and
reminder runs racing each other.

A new migration adds scheduler_leases, and CronRegistrarService now runs
every registered tick through SchedulerLeasesRepository.acquire: one
conditional UPDATE moves the job's row to this process when it is free
(no holder, a lapsed holder, or this process already), so exactly one
process wins. A running tick renews its lease every 20 seconds, a crashed
holder frees the job after a minute, and a finished tick keeps the lease
for 30 seconds so a peer whose timer fires late does not run the same
tick again. The per-row reminder claims stay as they are; jobs need no
change of their own.
This commit is contained in:
Maurice
2026-10-08 15:39:24 +02:00
parent 8632f93c37
commit c5bc6798a9
11 changed files with 319 additions and 11 deletions
+1 -1
View File
@@ -47,7 +47,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
- **`src/bootstrap.ts` — `buildApp()`** is the single builder for production and the test harness. Everything on the raw Express instance (global middleware, uploads, discovery metadata, statics, the named `jsonParser`/`urlencodedParser` wrappers) is registered **before `app.init()`**, because Nest throws `NotFoundException` on unmatched routes and nothing after init is reachable. The wrappers exempt `/mcp` so the MCP SDK reads the raw stream — **never `@Body()` a `/mcp` route**. Don't add routes to that pre-init shell; it bypasses every global guard, interceptor and filter.
- **Realtime**: `/ws` is a Nest gateway in `src/nest/realtime/`. Inject `RealtimeService`; `src/websocket.ts` is a re-export stub for old call sites. Trip mutations forward `X-Socket-Id` so the originating client doesn't echo its own change.
- **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `<domain>.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change.
- **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator** — it bypasses that gate.
- **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator** — it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire`, one conditional UPDATE), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send.
- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest** — it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/` — run `gen:plugin-facts` after touching them and commit the output. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together.
- **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; Nest classes inject `DataPathsService`, and nothing anchors a data path on its own `__dirname` or on `process.cwd()`.
- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate-<schema>-<time>.db`, `<schema>` being `legacy-<schema_version>` or the last recorded migration's timestamp; the newest `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` stay) and the newer-database refusal names the matching copy; a copy that cannot be written refuses the boot unless `TREK_DB_PRE_MIGRATE_SNAPSHOT=false`. It is off under `NODE_ENV=test` unless a suite sets that variable to true, and a restore's re-bootstrap skips it (the archive is the copy). After the migrator, `db/email-case-index.ts` creates the case-insensitive unique index on `users.email` that its migration had to defer on an install holding two accounts whose emails differ only in case (flag `email_case_index_pending`). `db/orm-driver.ts` binds the ORM to that same connection instead of opening a second one — with `:memory:` under test, a second connection is a second, empty database. Kysely statements (`this.kysely<...>()`) type against `DB` from `db/kysely/db.ts`, narrowed with `Pick<DB, 'trips' | 'days'>`; those files are generated from the migrated schema by `scripts/generate-db-types.ts`, so a migration that adds or changes a column is followed by `npm run gen:db-types`, never by a hand-written table interface. Every read and write goes through a repository in `db/repositories/` (each extends the shared `TrekRepository` base in `db/repositories/_shared/`, which validates the request context before delegating to MikroORM) and, for multi-statement writes, `UnitOfWork.transactional(...)` (`nest/database/unit-of-work.ts`, provided by the global `OrmModule`) — there is no `DatabaseService`/`DatabaseModule` to inject any more. Entrypoints outside an HTTP request — an MCP tool call, a plugin RPC dispatch, a WebSocket message, a cron job, boot-time seeding, a backup restore — open one explicitly with `withRequestContext()` (`nest/database/request-context.ts`), since only the pathless `mikroOrmRequestContext` middleware `buildApp()` mounts (with `@mikro-orm/nestjs`'s own `registerRequestContext` turned off in `app.module.ts`, UPLOADS-P16) forks an `EntityManager` per HTTP request. `NODE_ENV=test` ⇒ isolated `:memory:` DB per vitest worker, opened from a snapshot `tests/global-setup.ts` migrates once per run; `TREK_DB_FILE` ⇒ explicit file DB (Playwright harness).
@@ -0,0 +1,20 @@
import { EntityRepositoryType, PrimaryKeyProp, defineEntity, p } from '@mikro-orm/core';
import { SchedulerLeasesRepository } from '../repositories/SchedulerLeases.repository';
export class SchedulerLeases {
[EntityRepositoryType]?: SchedulerLeasesRepository;
[PrimaryKeyProp]?: 'name';
name!: string;
owner!: string;
expires_at!: number;
}
export const SchedulerLeasesSchema = defineEntity({
class: SchedulerLeases,
repository: () => SchedulerLeasesRepository,
properties: {
name: p.text().primary(),
owner: p.text(),
expires_at: p.integer(),
},
});
+2
View File
@@ -92,6 +92,7 @@ import { RoadtripDayTracksSchema } from './RoadtripDayTracks.entity';
import { RoadtripPreferencesSchema } from './RoadtripPreferences.entity';
import { RoadtripViasSchema } from './RoadtripVias.entity';
import { RouteUsageDailySchema } from './RouteUsageDaily.entity';
import { SchedulerLeasesSchema } from './SchedulerLeases.entity';
import { SchemaVersionSchema } from './SchemaVersion.entity';
import { SchoolHolidayCountriesSchema } from './SchoolHolidayCountries.entity';
import { SchoolHolidayPeriodsSchema } from './SchoolHolidayPeriods.entity';
@@ -224,6 +225,7 @@ export const ALL_ENTITIES: readonly EntitySchema[] = [
RoadtripPreferencesSchema,
RoadtripViasSchema,
RouteUsageDailySchema,
SchedulerLeasesSchema,
SchemaVersionSchema,
SchoolHolidayCountriesSchema,
SchoolHolidayPeriodsSchema,
@@ -0,0 +1,27 @@
import { Migration } from '@mikro-orm/migrations';
/**
* A cron tick runs in one process. Every job registered through
* CronRegistrarService takes a row here before its tick body runs: the row
* names the process holding the job and until when, and a process only runs
* the tick when one conditional UPDATE moved the row to itself. Two processes
* on one database (a rolling update's overlap, a second replica, a stray dev
* container) therefore no longer both send the reminders, take the backup or
* run the sync.
*
* `expires_at` is in milliseconds since the epoch. A row is only ever written
* by the scheduler, so there is nothing to backfill.
*/
export class Migration20200101042500_a_cron_tick_runs_in_one_process extends Migration {
override name = 'Migration20200101042500_a_cron_tick_runs_in_one_process';
override async up(): Promise<void> {
this.addSql(`
CREATE TABLE IF NOT EXISTS scheduler_leases (
name TEXT NOT NULL PRIMARY KEY,
owner TEXT NOT NULL,
expires_at INTEGER NOT NULL
)
`);
}
}
@@ -0,0 +1,40 @@
import type { SchedulerLeases } from '../entities/SchedulerLeases.entity';
import { TrekRepository } from './_shared/trek-repository';
/**
* `scheduler_leases`: which process may run a cron job's tick, and until when
* (`CronRegistrarService`). A lease is free when no row exists yet, when its
* `expires_at` has passed, or when the asking process already holds it.
*/
export class SchedulerLeasesRepository extends TrekRepository<SchedulerLeases> {
/**
* Take (or renew) the lease on `name` for `owner` until `until`, if it is
* free at `now`. The decision is one conditional UPDATE, so of two
* processes asking at the same moment exactly one gets a changed row; the
* insert before it only makes sure there is a row to race on. True when
* `owner` holds the lease afterwards.
*/
async acquire(name: string, owner: string, now: number, until: number): Promise<boolean> {
await this.upsert(
{ name, owner, expires_at: 0 },
{ onConflictFields: ['name'], onConflictAction: 'ignore' },
);
const changed = await this.nativeUpdate(
{ name, $or: [{ expires_at: { $lte: now } }, { owner }] },
{ owner, expires_at: until },
);
return changed > 0;
}
/** Move the expiry of a lease `owner` still holds; false when somebody else has it now. */
async extend(name: string, owner: string, until: number): Promise<boolean> {
const changed = await this.nativeUpdate({ name, owner }, { expires_at: until });
return changed > 0;
}
/** The current holder and expiry, for diagnostics and tests. */
async holder(name: string): Promise<{ owner: string; expires_at: number } | null> {
const row = await this.findOne({ name }, { fields: ['owner', 'expires_at'] });
return row ? { owner: row.owner, expires_at: row.expires_at } : null;
}
}
@@ -1,3 +1,5 @@
import { hostname } from 'node:os';
import { randomUUID } from 'node:crypto';
import { Injectable, Optional, type OnApplicationShutdown } from '@nestjs/common';
import { SchedulerRegistry } from '@nestjs/schedule';
import { CronJob } from 'cron';
@@ -5,8 +7,28 @@ import { MikroORM } from '@mikro-orm/core';
import { readEnv } from '../../app-config';
import { RuntimeEnvService } from '../app-config/runtime-env.service';
import { withRequestContext } from '../database/request-context';
import { logError } from '../audit/audit-log.logger';
import { logDebug, logError } from '../audit/audit-log.logger';
import { traceEntry, wasTraced } from '../audit/entry-trace.logger';
import { SchedulerLeases } from '../../db/entities/SchedulerLeases.entity';
/**
* Who holds a lease: this process. Every registrar in one process shares it,
* so a job re-registered (or a second app built in the same process, as the
* test harness does) keeps the lease it already had, while another process on
* the same database is a different owner.
*/
export const LEASE_OWNER = `${hostname()}:${process.pid}:${randomUUID()}`;
/** How long a lease lasts without a heartbeat; a crashed holder frees its jobs after this. */
export const LEASE_TTL_MS = 60_000;
/** How often a running tick renews its lease. */
export const LEASE_HEARTBEAT_MS = 20_000;
/**
* How long a finished tick keeps its lease. Long enough that another process
* whose timer fires a little late for the same tick finds it taken, short
* enough that the next tick of a minutely job is free for whoever fires it.
*/
export const LEASE_SETTLE_MS = 30_000;
/**
* The one way TREK code schedules a cron. Job providers register here from
@@ -93,7 +115,7 @@ export class CronRegistrarService implements OnApplicationShutdown {
}
// Each tick is its own unit of work: one correlation id for every line
// the job writes, and one line for the tick itself.
return traceEntry('cron', name, () => withRequestContext(orm, () => onTick()), {
return traceEntry('cron', name, () => withRequestContext(orm, () => this.runLeased(orm, name, onTick)), {
failureLevel: 'error',
failureMessage: failed,
});
@@ -118,6 +140,38 @@ export class CronRegistrarService implements OnApplicationShutdown {
return true;
}
/**
* Run one tick only if this process holds the job's lease.
*
* waitForCompletion keeps a tick from overlapping itself inside one process;
* the lease does the same across processes sharing the database (a rolling
* update's overlap, a second replica, a dev container pointed at the same
* file). The lease is renewed while the tick runs, so a long backup keeps it,
* and held for LEASE_SETTLE_MS afterwards, so a peer whose timer fires late
* for the same tick does not run it again. A process that dies holding it
* frees the job after LEASE_TTL_MS.
*/
private async runLeased(orm: Pick<MikroORM, 'em'>, name: string, onTick: () => void | Promise<void>): Promise<void> {
const leases = orm.em.getRepository(SchedulerLeases);
if (!(await leases.acquire(name, LEASE_OWNER, Date.now(), Date.now() + LEASE_TTL_MS))) {
logDebug(`Cron job "${name}" skipped: another process holds this tick`);
return;
}
const leaseFailed = (err: unknown) =>
logError(`Cron job "${name}": lease update failed: ${err instanceof Error ? err.message : String(err)}`);
const heartbeat = setInterval(() => {
withRequestContext(orm, () => leases.extend(name, LEASE_OWNER, Date.now() + LEASE_TTL_MS)).catch(leaseFailed);
}, LEASE_HEARTBEAT_MS);
heartbeat.unref();
try {
await onTick();
} finally {
clearInterval(heartbeat);
// Never let the bookkeeping replace the tick's own error.
await leases.extend(name, LEASE_OWNER, Date.now() + LEASE_SETTLE_MS).catch(leaseFailed);
}
}
/**
* Runs a boot-time (`onApplicationBootstrap`) one-off sweep inside the same
* request context a registered tick gets from `register()` above — the one
+5 -4
View File
@@ -26,17 +26,18 @@ describe('ALL_ENTITIES', () => {
});
/**
* Plan 2 Task 3: every one of the 130 entities the migrated schema
* Plan 2 Task 3: every one of the 131 entities the migrated schema
* describes (120 rewritten + the 5 document-sync tables added this task,
* + push_subscriptions and google_api_usage from legacy steps 245 and 254,
* + tour_types, tours and tour_waypoints from the Tours migration)
* + tour_types, tours and tour_waypoints from the Tours migration,
* + scheduler_leases from the cron lease migration)
* is registered, and no two share a class name (a silent duplicate would
* shadow one of them in `orm.getMetadata()`, exactly the kind of drift
* `entity-schema-parity.test.ts` can't see — it iterates the metadata map,
* which de-duplicates by class name).
*/
it('ENT-003: exactly 130 entities, every class name unique', () => {
expect(ALL_ENTITIES.length).toBe(130);
it('ENT-003: exactly 131 entities, every class name unique', () => {
expect(ALL_ENTITIES.length).toBe(131);
const names = ALL_ENTITIES.map((s) => s.meta.className);
expect(new Set(names).size).toBe(names.length);
});
@@ -14,7 +14,7 @@ import { createTestOrm, type TestOrm } from '../../helpers/test-orm';
* `scripts/generate-entities.ts`'s `RULE1d_fixNaNNumericDefaults` and
* `tests/unit/db/generate-entities.test.ts`'s `RULE1D-*` fixture tests for
* the mechanism). This file pins the OUTPUT of that rule against the real,
* committed entities — every one of the 130, not a sample — and reproduces
* committed entities — every one of the 131, not a sample — and reproduces
* the reviewer's own insert probe so a future regenerate/hand-edit that
* reintroduces a `NaN` initialiser fails loudly here, not silently at a
* caller's first `em.create()`.
@@ -22,10 +22,11 @@ import { createTestOrm, type TestOrm } from '../../helpers/test-orm';
/** NOT NULL scalars with neither a column default nor a class-field initialiser (legitimate: the caller must supply them). */
// 424 + push_subscriptions (legacy step 245): id, user_id, endpoint, p256dh, auth, vapid_public_key.
// + Tours: tour_types label_key, icon, color; tours.tour_type; tour_waypoints id, place_id, lat, lng, role, sequence.
const KNOWN_UNINITIALISED_NOT_NULL_SCALARS = 440;
// + scheduler_leases: name, owner, expires_at (the scheduler always writes all three).
const KNOWN_UNINITIALISED_NOT_NULL_SCALARS = 443;
describe('entity class fields never carry a NaN initialiser (task-4-review-shape.md, Important 1)', () => {
it('CLASSFIELD-001: new X() has no own property whose value is NaN, across all 130 entities', () => {
it('CLASSFIELD-001: new X() has no own property whose value is NaN, across all 131 entities', () => {
const failures: string[] = [];
for (const schema of ALL_ENTITIES) {
const Ctor = schema.class as new () => Record<string, unknown>;
@@ -36,6 +36,7 @@ const UNNUMBERED = [
'Migration20200101042200_an_index_on_every_foreign_key',
'Migration20200101042300_every_tour_type_the_contract_names',
'Migration20200101042400_one_account_per_email_whatever_its_case',
'Migration20200101042500_a_cron_tick_runs_in_one_process',
];
// The last step the positional runner ever had; planLegacyBaseline refuses a
// schema_version past it.
@@ -0,0 +1,67 @@
/**
* SchedulerLeasesRepository: the lease a cron tick takes so that two
* processes on one database never run the same tick. LEASE-001 through
* LEASE-006.
*/
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
import { createSnapshotTestDb } from '../../../helpers/db-mock';
import { createTestOrm, type TestOrm } from '../../../helpers/test-orm';
import { SchedulerLeases } from '../../../../src/db/entities/SchedulerLeases.entity';
import type { SchedulerLeasesRepository } from '../../../../src/db/repositories/SchedulerLeases.repository';
const testDb = createSnapshotTestDb();
let t: TestOrm;
let leases: SchedulerLeasesRepository;
beforeAll(async () => {
t = await createTestOrm(testDb);
leases = t.repo(SchedulerLeases);
});
beforeEach(() => {
testDb.prepare('DELETE FROM scheduler_leases').run();
t.clear();
});
afterAll(async () => {
await t.close();
testDb.close();
});
describe('SchedulerLeasesRepository.acquire', () => {
it('LEASE-001: the first process to ask takes a lease nobody holds', async () => {
expect(await leases.acquire('docsync', 'a', 1_000, 61_000)).toBe(true);
expect(await leases.holder('docsync')).toEqual({ owner: 'a', expires_at: 61_000 });
});
it('LEASE-002: a second process is refused while the lease runs, and the holder is untouched', async () => {
await leases.acquire('docsync', 'a', 1_000, 61_000);
expect(await leases.acquire('docsync', 'b', 2_000, 62_000)).toBe(false);
expect(await leases.holder('docsync')).toEqual({ owner: 'a', expires_at: 61_000 });
});
it('LEASE-003: the holder renews its own lease before it runs out', async () => {
await leases.acquire('docsync', 'a', 1_000, 61_000);
expect(await leases.acquire('docsync', 'a', 2_000, 62_000)).toBe(true);
expect(await leases.holder('docsync')).toEqual({ owner: 'a', expires_at: 62_000 });
});
it('LEASE-004: an expired lease goes to whoever asks next', async () => {
await leases.acquire('docsync', 'a', 1_000, 61_000);
expect(await leases.acquire('docsync', 'b', 61_000, 121_000)).toBe(true);
expect(await leases.holder('docsync')).toEqual({ owner: 'b', expires_at: 121_000 });
});
it('LEASE-005: leases are per job', async () => {
await leases.acquire('docsync', 'a', 1_000, 61_000);
expect(await leases.acquire('auto-backup', 'b', 1_000, 61_000)).toBe(true);
});
});
describe('SchedulerLeasesRepository.extend', () => {
it('LEASE-006: only the holder moves the expiry', async () => {
await leases.acquire('docsync', 'a', 1_000, 61_000);
expect(await leases.extend('docsync', 'b', 99_000)).toBe(false);
expect(await leases.extend('docsync', 'a', 91_000)).toBe(true);
expect(await leases.holder('docsync')).toEqual({ owner: 'a', expires_at: 91_000 });
expect(await leases.holder('never-ran')).toBeNull();
});
});
@@ -51,7 +51,13 @@ vi.mock('../../../src/nest/audit/audit-log.logger', () => ({
}));
import { SchedulerRegistry } from '@nestjs/schedule';
import { CronRegistrarService } from '../../../src/nest/scheduling/cron-registrar.service';
import {
CronRegistrarService,
LEASE_HEARTBEAT_MS,
LEASE_OWNER,
LEASE_SETTLE_MS,
LEASE_TTL_MS,
} from '../../../src/nest/scheduling/cron-registrar.service';
import type { RuntimeEnvService } from '../../../src/nest/app-config/runtime-env.service';
import { createSnapshotTestDb } from '../../helpers/db-mock';
import { createTestOrm, type TestOrm } from '../../helpers/test-orm';
@@ -278,6 +284,95 @@ describe('CronRegistrarService', () => {
});
});
describe('the lease (one process per tick)', () => {
const testDb = createSnapshotTestDb();
let t: TestOrm;
beforeEach(async () => {
t = await createTestOrm(testDb, { allowGlobalContext: false });
});
afterEach(async () => {
vi.useRealTimers();
await t.close();
});
const holder = (name: string) =>
testDb.prepare('SELECT owner, expires_at FROM scheduler_leases WHERE name = ?').get(name) as
| { owner: string; expires_at: number }
| undefined;
it('CRONREG-017: a tick another process holds does not run here', async () => {
const name = 'lease-held-elsewhere';
testDb
.prepare('INSERT INTO scheduler_leases (name, owner, expires_at) VALUES (?, ?, ?)')
.run(name, 'other-host:1:peer', Date.now() + 60_000);
const registrar = new CronRegistrarService(new SchedulerRegistry(), { isTest: () => false } as RuntimeEnvService, t.orm);
let ran = false;
registrar.register(name, '* * * * *', () => {
ran = true;
});
await h.jobs[0].onTick();
expect(ran).toBe(false);
expect(holder(name)!.owner).toBe('other-host:1:peer');
});
it('CRONREG-018: a lapsed lease is taken over, and held for the settle window after the tick', async () => {
const name = 'lease-lapsed';
testDb
.prepare('INSERT INTO scheduler_leases (name, owner, expires_at) VALUES (?, ?, ?)')
.run(name, 'crashed-host:9:gone', Date.now() - 1);
const registrar = new CronRegistrarService(new SchedulerRegistry(), { isTest: () => false } as RuntimeEnvService, t.orm);
let ran = false;
registrar.register(name, '* * * * *', () => {
ran = true;
});
const before = Date.now();
await h.jobs[0].onTick();
expect(ran).toBe(true);
const row = holder(name)!;
expect(row.owner).toBe(LEASE_OWNER);
expect(row.expires_at).toBeGreaterThanOrEqual(before + LEASE_SETTLE_MS);
expect(row.expires_at).toBeLessThan(before + LEASE_TTL_MS);
});
it('CRONREG-019: this process runs its own next tick even while it still holds the lease', async () => {
const name = 'lease-own-next-tick';
const registrar = new CronRegistrarService(new SchedulerRegistry(), { isTest: () => false } as RuntimeEnvService, t.orm);
let runs = 0;
registrar.register(name, '* * * * *', () => {
runs++;
});
await h.jobs[0].onTick();
await h.jobs[0].onTick();
expect(runs).toBe(2);
});
it('CRONREG-020: a long tick renews its lease while it runs', async () => {
vi.useFakeTimers({ toFake: ['setInterval', 'clearInterval'] });
const name = 'lease-long-tick';
const registrar = new CronRegistrarService(new SchedulerRegistry(), { isTest: () => false } as RuntimeEnvService, t.orm);
let finish!: () => void;
let started!: () => void;
const running = new Promise<void>((resolve) => {
started = resolve;
});
registrar.register(name, '* * * * *', () => {
started();
return new Promise<void>((resolve) => {
finish = resolve;
});
});
const tick = h.jobs[0].onTick();
await running;
testDb.prepare('UPDATE scheduler_leases SET expires_at = 1 WHERE name = ?').run(name);
vi.advanceTimersByTime(LEASE_HEARTBEAT_MS);
await vi.waitFor(() => expect(holder(name)!.expires_at).toBeGreaterThan(Date.now()));
finish();
await tick;
});
});
describe('runOnBoot (task-6-fix-brief.md item 7 — the boot-sweep choke point)', () => {
const testDb = createSnapshotTestDb();
let t: TestOrm;