mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-19 05:26:47 +00:00
refactor(server): take the storage roots from DataPathsService
The data-directory change said Nest classes inject DataPathsService, but only AdminService did. StorageRegistryService still read the built-in uploads and backups roots and the scratch dir from module constants frozen at import, and StorageJobsService read the scratch dir the same way. The registry now takes DataPathsService through its constructor (a hand-built one resolves the same layout), and the jobs service asks the registry for its temp dir. The plain helper modules keep calling resolveDataPaths() because they run without a container or are shared with code that does: config.ts, the database path, index.ts, the logger, the backup functions and the plugin trees, which the out-of-process supervisor reads too. The app-config README now lists both sides, and the nest README, server/CLAUDE.md and the provider's own comment say the same.
This commit is contained in:
+1
-1
@@ -49,7 +49,7 @@ Nest owns everything. Every domain is a DI module under `src/nest/<domain>/` (`c
|
||||
- **MCP**: `src/nest-mcp/` is the decorator/registry layer (`@McpController`/`@Tool`/`@Resource`/`@Prompt`; see its `README.md`); tools live in `<domain>.mcp.ts` files beside each module; the HTTP transport is `src/nest/mcp-transport/`; `src/mcp/` holds process-wide state only (scopes, session manager, boot policy). Every tool invocation writes an audit row through the registry's `onInvoke` seam. MCP tools are adapters over the same services and permission checks as REST — a change on one side lands on the other in the same change.
|
||||
- **Scheduling**: `src/nest/scheduling/` (`CronRegistrarService`) is the one way to schedule a cron. Jobs are `*.job.ts` providers in their owning domain. The registrar refuses to schedule under `NODE_ENV=test` (an integration test pins it). **Never use a `@Cron` decorator**: it bypasses that gate. Every tick first takes the job's row in `scheduler_leases` (`SchedulerLeasesRepository.acquire` in one `UnitOfWork` transaction, where a conditional UPDATE decides the race), renews it while it runs and keeps it for a short settle window afterwards, so two processes on one database never run the same tick; a job needs nothing of its own for this. Claims on individual rows (the reminder jobs' `claimReminder`) stay where they are: the lease decides who runs a tick, the claim what a tick may send.
|
||||
- **Plugins** (`src/nest/plugins/`): the child-process runtime (`supervisor/`, `runtime/plugin-host-entry.ts`, `protocol/envelope.ts`) is **intentionally not Nest** — it is the security boundary. `host/rpc-host.ts` registers an RPC method at spawn only if the plugin holds the unlocking permission (registration = authorization). Install-time gates live in `install/`; the manifest's `trek` version range fails closed, and `TREK_PLUGINS_IGNORE_TREK_RANGE` downgrades that to a warning. The protocol tables in `protocol/envelope.ts` (+ `src/plugin-event-sink.ts`) are **generated** into `plugin-sdk/` and `shared/` — run `gen:plugin-facts` after touching them and commit the output. `runtime/egress-policy.ts` is a hand-kept mirror guarded by a parity test in `plugin-sdk/`; change both sides together.
|
||||
- **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; Nest classes inject `DataPathsService`, and nothing anchors a data path on its own `__dirname` or on `process.cwd()`.
|
||||
- **Storage**: `src/nest/storage/` is the upload driver abstraction (local / S3 / mirror behind one interface, with an admin surface and a usage job). Domain services never touch the filesystem or the S3 SDK directly. Where the data lives (`data/`, `uploads/`, backups, logs, key files, the DB file) is resolved in one place, `src/app-config/data-paths.ts`; a provider that reads one injects `DataPathsService`, plain helper modules shared with code outside the container call `resolveDataPaths()` (the list is in `src/app-config/README.md`), and nothing anchors a data path on its own `__dirname` or on `process.cwd()`.
|
||||
- **DB** (`src/db/`): `better-sqlite3`, WAL, FK on. `database.ts` only holds the connection and applies pragmas. It does not build a schema, and importing it opens nothing (the first reader does). The connection's lifecycle belongs to the `DatabaseLifecycle` provider (`nest/database/database-lifecycle.service.ts`): `buildApp()` opens it through `open()` (which binds the ORM to later swaps and runs the schema bootstrap below), the shutdown closes it through `close()`, and a restore or the demo reset swaps the file only through the database backup port, `DATABASE_BACKUP` (`nest/database/database-backup.interface.ts`; the SQLite implementation is `nest/backup/sqlite-database-backup.ts`: `VACUUM INTO` snapshot, close, atomic file swap, `reopen()`). The port works on `DatabaseLifecycle.file`, the one resolved path (`TREK_DB_FILE` or `data/travel.db`), and a snapshot that fails fails the backup rather than archiving the live file. `MaintenanceRepository` is provided only by `MaintenanceModule`'s factory and injected everywhere else; no consumer constructs it. A provider that has to follow every reopen (the demo reset job saves the first demo baseline after a restore's re-bootstrap seeded the example trips) registers with `DatabaseLifecycle.onReopened()`, which runs after the reopen's schema bootstrap; a listener contains its own failures, since one that throws fails the reopen. MikroORM's migrations (`db/migrations/`) are the only schema source, **including for tests**: the hand-kept `schema.ts`/`migrations.ts`/`seeds.ts` builders and the parity test that guarded them are gone. Seeding is the MikroORM seeders in `db/seeders/`, applied by `db/orm.ts` from `buildApp()` before anything reads. An install the retired positional runner migrated (a `schema_version` row, no `mikro_orm_migrations` rows) is baselined by `db/legacy-baseline.ts`: the migrations whose `Legacy migration step N` docstring is ≤ its version are recorded as executed (never replayed) in the same transaction as the migrator run, while the baseline migration itself still runs first, as the legacy runner's `createTables()` did on every boot; anything it cannot map refuses the boot. Before a boot applies pending migrations to a file database that holds anything, `db/pre-migrate-snapshot.ts` writes a `VACUUM INTO` copy next to it (`pre-migrate-<schema>-<time>.db`, `<schema>` being `legacy-<schema_version>` or the last recorded migration's timestamp; the newest `TREK_DB_PRE_MIGRATE_SNAPSHOT_KEEP` stay) and the newer-database refusal names the matching copy; a copy that cannot be written refuses the boot unless `TREK_DB_PRE_MIGRATE_SNAPSHOT=false`. It is off under `NODE_ENV=test` unless a suite sets that variable to true, and a restore's re-bootstrap skips it (the archive is the copy). After the migrator, `db/email-case-index.ts` creates the case-insensitive unique index on `users.email` that its migration had to defer on an install holding two accounts whose emails differ only in case (flag `email_case_index_pending`). `db/orm-driver.ts` binds the ORM to that same connection instead of opening a second one — with `:memory:` under test, a second connection is a second, empty database. Kysely statements (`this.kysely<...>()`) type against `DB` from `db/kysely/db.ts`, narrowed with `Pick<DB, 'trips' | 'days'>`; those files are generated from the migrated schema by `scripts/generate-db-types.ts`, so a migration that adds or changes a column is followed by `npm run gen:db-types`, never by a hand-written table interface. Every read and write goes through a repository in `db/repositories/` (each extends the shared `TrekRepository` base in `db/repositories/_shared/`, which validates the request context before delegating to MikroORM) and, for multi-statement writes, `UnitOfWork.transactional(...)` (`nest/database/unit-of-work.ts`, provided by the global `OrmModule`) — there is no `DatabaseService`/`DatabaseModule` to inject any more. Entrypoints outside an HTTP request — an MCP tool call, a plugin RPC dispatch, a WebSocket message, a cron job, boot-time seeding, a backup restore — open one explicitly with `withRequestContext()` (`nest/database/request-context.ts`), since only the pathless `mikroOrmRequestContext` middleware `buildApp()` mounts (with `@mikro-orm/nestjs`'s own `registerRequestContext` turned off in `app.module.ts`, UPLOADS-P16) forks an `EntityManager` per HTTP request. `NODE_ENV=test` ⇒ isolated `:memory:` DB per vitest worker, opened from a snapshot `tests/global-setup.ts` migrates once per run; `TREK_DB_FILE` ⇒ explicit file DB (Playwright harness).
|
||||
|
||||
## Rules for new code
|
||||
|
||||
@@ -60,10 +60,27 @@ tokens, `RuntimeEnvService`) and consumes the SAME derive functions.
|
||||
`uploads/` (on `SERVER_ROOT`, the package directory, never on `process.cwd()`)
|
||||
and names what lives in them: backups, the scratch dir, the logs, the JWT
|
||||
secret and encryption key files, and the database file (TREK_DB_FILE when set,
|
||||
else `data/travel.db`). Code that runs before the container exists imports it
|
||||
directly; Nest classes inject `DataPathsService` (`src/nest/app-config/`),
|
||||
which resolves the same layout once per built app. Never derive one of these
|
||||
paths from `__dirname` in a new file: a moved file would quietly move the data.
|
||||
else `data/travel.db`). Never derive one of these paths from `__dirname` in a
|
||||
new file: a moved file would quietly move the data.
|
||||
|
||||
Two ways in, by where the code runs:
|
||||
|
||||
- **Nest providers inject `DataPathsService`** (`src/nest/app-config/`), which
|
||||
resolves the same layout once per built app: `AdminService` (the JWT secret
|
||||
file), `StorageRegistryService` (the built-in uploads and backups roots and
|
||||
the scratch dir) and, through `StorageRegistryService.tempDir()`,
|
||||
`StorageJobsService`. A new provider that needs a data path takes it the
|
||||
same way.
|
||||
- **Plain modules call `resolveDataPaths()`** because they run without a
|
||||
container or are shared with code that does: `config.ts` key resolution,
|
||||
`db/db-path.ts`, `index.ts`, the file logger, `demo/demo-reset.ts`, the
|
||||
backup restore and archive functions (`nest/backup/backup.impl.ts`,
|
||||
`auto-backup.settings.ts`, file I/O with no container state), the plugin
|
||||
trees (`nest/plugins/paths.ts`, also read by the out-of-process supervisor),
|
||||
and `nest/storage/storage-paths.ts`, which keeps the seed-config test seam
|
||||
that `StorageAdminService` and the registry share. `files.constants.ts`
|
||||
still exports `filesDir` for the tests that pin the layout; no production
|
||||
code reads it.
|
||||
|
||||
## app-url.ts — instance base-URL resolution (moved 2026-07-28)
|
||||
|
||||
|
||||
@@ -378,7 +378,10 @@ sharing).
|
||||
or read runtime-toggled values live through `RuntimeEnvService` / `readEnv()`
|
||||
from `src/app-config`. Every variable has exactly one owner, a token or
|
||||
`readEnv()`, and `config-ownership.test.ts` fails on a variable read through
|
||||
both. Data paths come from `DataPathsService`. The classification and
|
||||
both. A provider that reads a data path takes `DataPathsService`
|
||||
(`AdminService`, `StorageRegistryService`, and through it the storage
|
||||
jobs); plain helper modules call `resolveDataPaths()` (see
|
||||
`src/app-config/README.md` for which and why). The classification and
|
||||
invariants live in `src/app-config/README.md`.
|
||||
|
||||
## Parity gotchas worth remembering
|
||||
|
||||
@@ -5,11 +5,14 @@ import { resolveDataPaths, type DataPaths } from '../../app-config/data-paths';
|
||||
* The data layout for Nest classes: the data directory, uploads, backups,
|
||||
* scratch space, logs, the key files and the database file.
|
||||
*
|
||||
* Code that runs before the container exists (config.ts key resolution, the
|
||||
* database connection, index.ts, the file logger) calls `resolveDataPaths()`
|
||||
* directly; this provider hands the same answer to everything that can take a
|
||||
* constructor argument, so neither side counts `..` hops from its own file.
|
||||
* Resolved once per built app: none of these move while a process runs.
|
||||
* Providers that read a data path take this one (AdminService,
|
||||
* StorageRegistryService). Code that runs before the container exists or is
|
||||
* shared with code outside it (config.ts key resolution, the database
|
||||
* connection, index.ts, the file logger, the backup functions, the plugin
|
||||
* trees) calls `resolveDataPaths()` directly; src/app-config/README.md keeps
|
||||
* the list. Both get the same answer, and neither counts `..` hops from its
|
||||
* own file. Resolved once per built app: none of these move while a process
|
||||
* runs.
|
||||
*/
|
||||
@Injectable()
|
||||
export class DataPathsService implements DataPaths {
|
||||
|
||||
@@ -11,7 +11,6 @@ import {
|
||||
} from '@trek/shared';
|
||||
import { contentTypeFor } from './content-type';
|
||||
import { MirrorDriver } from './drivers/mirror.driver';
|
||||
import { GLOBAL_TEMP_DIR } from './storage-paths';
|
||||
import { StorageRegistryService } from './storage-registry.service';
|
||||
import type { StorageDriver } from './storage.types';
|
||||
|
||||
@@ -252,7 +251,7 @@ export class StorageJobsService {
|
||||
sourceKey: string,
|
||||
destKey: string,
|
||||
): Promise<boolean> {
|
||||
const file = path.join(GLOBAL_TEMP_DIR, randomUUID());
|
||||
const file = path.join(this.registry.tempDir(), randomUUID());
|
||||
try {
|
||||
const { stream } = await source.getStream(sourceKey);
|
||||
await pipeline(stream, fs.createWriteStream(file)); // source errors RETHROW (abort → failed)
|
||||
|
||||
@@ -7,6 +7,7 @@ import { STORAGE_BACKEND_TYPES, storageConfigSchema } from '@trek/shared';
|
||||
import { UnitOfWork } from '../database/unit-of-work';
|
||||
import { withRequestContext } from '../database/request-context';
|
||||
import { storageConfig } from '../app-config/tokens';
|
||||
import { DataPathsService } from '../app-config/data-paths.service';
|
||||
import { decrypt_api_key } from '../common/crypto/apiKeyCrypto';
|
||||
import { AppSettings } from '../../db/entities/AppSettings.entity';
|
||||
import type { AppSettingsRepository } from '../../db/repositories/AppSettings.repository';
|
||||
@@ -14,7 +15,7 @@ import { LocalDriver } from './drivers/local.driver';
|
||||
import { MirrorDriver, type ReplicaFailure } from './drivers/mirror.driver';
|
||||
import { S3Driver } from './drivers/s3.driver';
|
||||
import { StorageEventsService } from './storage-events.service';
|
||||
import { DEFAULT_BACKUPS_ROOT, DEFAULT_UPLOADS_ROOT, GLOBAL_TEMP_DIR, getSeedConfigPath } from './storage-paths';
|
||||
import { getSeedConfigPath } from './storage-paths';
|
||||
import { assertNoMaskSentinels, encryptStorageSecrets } from './storage-secrets';
|
||||
import {
|
||||
SERVED_CATEGORIES,
|
||||
@@ -141,6 +142,9 @@ export class StorageRegistryService implements OnModuleInit {
|
||||
private readonly events: StorageEventsService,
|
||||
private readonly uow: UnitOfWork,
|
||||
private readonly orm: MikroORM,
|
||||
// The built-in roots and the scratch dir. A hand-built registry (the unit
|
||||
// suites) resolves the same layout the container's provider does.
|
||||
private readonly dataPaths: DataPathsService = new DataPathsService(),
|
||||
) {}
|
||||
|
||||
/**
|
||||
@@ -199,7 +203,7 @@ export class StorageRegistryService implements OnModuleInit {
|
||||
|
||||
/** Driver-agnostic global scratch space (data/tmp). */
|
||||
tempDir(): string {
|
||||
return GLOBAL_TEMP_DIR;
|
||||
return this.dataPaths.tmpDir;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -409,8 +413,8 @@ export class StorageRegistryService implements OnModuleInit {
|
||||
// uploads-local's root is the computed default; relocation is a settings
|
||||
// override row bearing the built-in's name.
|
||||
const backends = new Map<string, BackendConfig>();
|
||||
backends.set('uploads-local', { name: 'uploads-local', type: 'local', options: { root: DEFAULT_UPLOADS_ROOT } });
|
||||
backends.set('backups-local', { name: 'backups-local', type: 'local', options: { root: DEFAULT_BACKUPS_ROOT } });
|
||||
backends.set('uploads-local', { name: 'uploads-local', type: 'local', options: { root: this.dataPaths.uploadsDir } });
|
||||
backends.set('backups-local', { name: 'backups-local', type: 'local', options: { root: this.dataPaths.backupsDir } });
|
||||
const backendSources = new Map<string, BackendSource>([
|
||||
['uploads-local', 'built-in'],
|
||||
['backups-local', 'built-in'],
|
||||
@@ -482,7 +486,7 @@ export class StorageRegistryService implements OnModuleInit {
|
||||
}),
|
||||
);
|
||||
}
|
||||
fs.mkdirSync(GLOBAL_TEMP_DIR, { recursive: true });
|
||||
fs.mkdirSync(this.dataPaths.tmpDir, { recursive: true });
|
||||
|
||||
// 6. The effective world with provenance — assembled from the same merged
|
||||
// maps, so it always matches what drivers/categories actually resolve to.
|
||||
|
||||
@@ -22,6 +22,7 @@ import { Readable } from 'node:stream';
|
||||
import { encrypt_api_key } from '../../../../src/nest/common/crypto/apiKeyCrypto';
|
||||
import { StorageEventsService } from '../../../../src/nest/storage/storage-events.service';
|
||||
import { StorageRegistryService } from '../../../../src/nest/storage/storage-registry.service';
|
||||
import { DataPathsService } from '../../../../src/nest/app-config/data-paths.service';
|
||||
import { LocalDriver } from '../../../../src/nest/storage/drivers/local.driver';
|
||||
import { MirrorDriver, type ReplicaFailure } from '../../../../src/nest/storage/drivers/mirror.driver';
|
||||
import { S3Driver } from '../../../../src/nest/storage/drivers/s3.driver';
|
||||
@@ -173,6 +174,34 @@ describe('StorageRegistryService defaults', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it('takes the built-in roots and the scratch dir from the injected DataPathsService', async () => {
|
||||
const root = makeTmpDir();
|
||||
const paths = {
|
||||
...new DataPathsService(),
|
||||
uploadsDir: path.join(root, 'uploads'),
|
||||
backupsDir: path.join(root, 'backups'),
|
||||
tmpDir: path.join(root, 'tmp'),
|
||||
} as DataPathsService;
|
||||
const registry = new StorageRegistryService(
|
||||
await createTestAppSettingsRepo(testDb),
|
||||
makeEnvStub({}).env,
|
||||
new StorageEventsService(),
|
||||
await createTestUnitOfWork(testDb),
|
||||
(await sharedTestOrm(testDb)).orm,
|
||||
paths,
|
||||
);
|
||||
await registry.onModuleInit();
|
||||
|
||||
expect(registry.tempDir()).toBe(paths.tmpDir);
|
||||
expect(fs.statSync(paths.tmpDir).isDirectory()).toBe(true);
|
||||
expect(registry.resolve('files').driver.getLocalPath!('files/x.pdf')).toBe(
|
||||
path.join(fs.realpathSync(paths.uploadsDir), 'files/x.pdf'),
|
||||
);
|
||||
expect(registry.resolve('backups').driver.getLocalPath!('backup-1.zip')).toBe(
|
||||
path.join(fs.realpathSync(paths.backupsDir), 'backup-1.zip'),
|
||||
);
|
||||
});
|
||||
|
||||
it('a settings row bearing a built-in name replaces it (first-class override)', async () => {
|
||||
const relocated = makeTmpDir();
|
||||
const { registry } = await makeRegistry({ uploadsRoot: relocated });
|
||||
|
||||
Reference in New Issue
Block a user