mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-11-21 19:40:34 +00:00
fix(storage): judge unreferenced files without a brake anyone can trip
The 20 % brake skipped a category and wiped its marks every night once a fifth of it looked unreferenced. Any user could push a small install over that line, a backlog from #2924 started braked, and a second install on the same bucket below the line was not caught at all. It is gone: - Each pass leaves a trek-owner-<install id> marker at the root of every category it judges. A category holding another install's marker is skipped as foreign_install, one where the marker cannot be written as owner_unknown. Backups leave the markers out. - The install id lives in app_settings outside the storage.* prefix. The switch is stored as that id, so a database from another install reads as off, and every reopen (a restore) writes this install's id back, turns deletion off and drops the marks, confirmations and reports. - More than a fifth unreferenced still marks and reports, but deletes only after an admin confirmed the count shown (POST .../reclaim/confirm). A larger count later waits again. A skip no longer touches the marks, so a failed listing or lookup does not restart the week. Check now runs in the background and answers 202; the admin state says running and keeps the last deleting report apart, so a report-only check never resets what was freed. A pass that deletes writes a system audit entry, a pass that stops or fails keeps its partial counts and marks the rest interrupted, a backend error stops deleting in its category, and deleting stops for the rest of the pass once the switch is off or a backup is reading the uploads. The usage scan waits for a pass still running. Sources read each category once per pass and answer the pre-delete re-check with a query narrowed to the keys' plain runs. Bare names keep their raw spelling (a '#', '?' or trailing blank is part of an upload's name), and local roots are resolved through their symlinks before the overlap check. A pass lists only a category's root.
This commit is contained in:
@@ -56,8 +56,8 @@ export class StorageReclaimMarksRepository extends TrekRepository<StorageReclaim
|
||||
return Number(result.numDeletedRows);
|
||||
}
|
||||
|
||||
/** Drops every mark of a category, for a pass that could not judge it: the week starts over. */
|
||||
async deleteCategory(category: string): Promise<number> {
|
||||
return await this.nativeDelete({ category });
|
||||
/** Drops every mark, for a database a restore brought back: its marks are another history's. */
|
||||
async deleteAll(): Promise<number> {
|
||||
return await this.nativeDelete({});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -32,7 +32,9 @@ const ACTION_LABELS: Record<string, string> = {
|
||||
'admin.storage_backfill_cancel': 'cancelled a storage sync',
|
||||
'admin.storage_stats_refresh': 'refreshed storage usage stats',
|
||||
'admin.storage_reclaim_update': 'switched storage reclaim deletion',
|
||||
'admin.storage_reclaim_run': 'ran a storage reclaim pass',
|
||||
'admin.storage_reclaim_run': 'started a storage reclaim check',
|
||||
'admin.storage_reclaim_confirm': 'confirmed deleting a large share of unreferenced storage objects',
|
||||
'storage.reclaim_deleted': 'deleted unreferenced storage objects',
|
||||
'admin.places_contribute_decision': 'decided on sharing place details with TREK Places',
|
||||
'admin.places_contribute_withdraw': 'withdrew the place details this installation shared',
|
||||
'immich.private_ip_configured': 'configured Immich with private IP',
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Users } from '../../db/entities/Users.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
/**
|
||||
@@ -14,8 +14,10 @@ export class AuthStorageRefsService extends StorageReclaimSource {
|
||||
readonly claims = [{ category: 'avatars', prefix: '' }] as const;
|
||||
protected override readonly keyOptions = { bare: 'avatars' } as const;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em.getRepository(Users).find({ avatar: { $ne: null } }, { fields: ['avatar'] });
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(Users)
|
||||
.find(likeAny<Users>(['avatar'], containing), { fields: ['avatar'] });
|
||||
return rows.map((row) => row.avatar);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import { invalidatePermissionsCache } from '../permissions/permissions-cache';
|
||||
import { snapshotAllPluginDataDbs } from '../plugins/host/plugin-data.service';
|
||||
import { pluginsCodeRoot, pluginsDataRoot } from '../plugins/paths';
|
||||
import { stageExtractedPluginTrees, applyStagedRestoreNow } from '../plugins/plugin-backup';
|
||||
import { storageOwnerMarkerInstall } from '../storage/storage-reclaim.types';
|
||||
import type { StorageService } from '../storage/storage.service';
|
||||
import { StorageInvalidKeyError } from '../storage/storage.types';
|
||||
import { VALID_INTERVALS } from './auto-backup.settings';
|
||||
@@ -217,6 +218,10 @@ export async function createBackup(
|
||||
// In mode A the google/trek caches nest under the photos/ prefix — the
|
||||
// category walk would sweep them back in without this skip.
|
||||
if (category === 'photos' && (obj.key.startsWith('google/') || obj.key.startsWith('trek/'))) continue;
|
||||
// The storage reclaim pass's owner marker names the install that wrote
|
||||
// it; restored elsewhere, it would make that install's own uploads look
|
||||
// shared with a foreign one.
|
||||
if (storageOwnerMarkerInstall(obj.key) !== null) continue;
|
||||
// Local path available (exists on disk right now — the fail-safe half
|
||||
// of getLocalPathOrNull's contract) → push it directly: zero-copy, the
|
||||
// default-install path. Otherwise (a remote/S3 primary, or a local
|
||||
|
||||
@@ -32,8 +32,14 @@ export class BackupService {
|
||||
listBackups() {
|
||||
return svc.listBackups(this.storage);
|
||||
}
|
||||
createBackup(prefix?: 'backup' | 'auto-backup') {
|
||||
return svc.createBackup(this.deps, prefix);
|
||||
/** Marked running while it reads the uploads, so the storage reclaim pass deletes nothing it still archives. */
|
||||
async createBackup(prefix?: 'backup' | 'auto-backup') {
|
||||
this.jobs.beginBackup();
|
||||
try {
|
||||
return await svc.createBackup(this.deps, prefix);
|
||||
} finally {
|
||||
this.jobs.endBackup();
|
||||
}
|
||||
}
|
||||
restoreFromZip(zipPath: string) {
|
||||
return this.whileRestoring(() => svc.restoreFromZip(this.deps, zipPath));
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { CollectionPlaces } from '../../db/entities/CollectionPlaces.entity';
|
||||
import { Collections } from '../../db/entities/Collections.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -16,13 +16,13 @@ export class CollectionsStorageRefsService extends StorageReclaimSource {
|
||||
readonly id = 'collections';
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const covers = await this.em
|
||||
.getRepository(Collections)
|
||||
.find({ cover_image: { $ne: null } }, { fields: ['cover_image'] });
|
||||
.find(likeAny<Collections>(['cover_image'], containing), { fields: ['cover_image'] });
|
||||
const images = await this.em
|
||||
.getRepository(CollectionPlaces)
|
||||
.find({ image_url: { $ne: null } }, { fields: ['image_url'] });
|
||||
.find(likeAny<CollectionPlaces>(['image_url'], containing), { fields: ['image_url'] });
|
||||
return [...covers.map((row) => row.cover_image), ...images.map((row) => row.image_url)];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,6 +134,7 @@ export const MANAGED_ROUTE_ALLOW_LIST: string[] = [
|
||||
'StorageAdminController.statsRefresh',
|
||||
'StorageAdminController.test',
|
||||
'StorageAdminController.update',
|
||||
'StorageReclaimController.confirm',
|
||||
'StorageReclaimController.run',
|
||||
'StorageReclaimController.update',
|
||||
];
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { TripFiles } from '../../db/entities/TripFiles.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
/**
|
||||
@@ -19,8 +19,10 @@ export class FilesStorageRefsService extends StorageReclaimSource {
|
||||
readonly claims = [{ category: 'files', prefix: '' }] as const;
|
||||
protected override readonly keyOptions = { bare: 'files', basename: true } as const;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em.getRepository(TripFiles).find({}, { fields: ['filename'] });
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(TripFiles)
|
||||
.find(likeAny<TripFiles>(['filename'], containing), { fields: ['filename'] });
|
||||
return rows.map((row) => row.filename);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Journeys } from '../../db/entities/Journeys.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -15,10 +15,10 @@ export class JourneyStorageRefsService extends StorageReclaimSource {
|
||||
readonly id = 'journey';
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(Journeys)
|
||||
.find({ cover_image: { $ne: null } }, { fields: ['cover_image'] });
|
||||
.find(likeAny<Journeys>(['cover_image'], containing), { fields: ['cover_image'] });
|
||||
return rows.map((row) => row.cover_image);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { TrekPhotos } from '../../db/entities/TrekPhotos.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -20,13 +20,12 @@ export class PhotosStorageRefsService extends StorageReclaimSource {
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
protected override readonly keyOptions = { bare: 'journey', basename: true } as const;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(TrekPhotos)
|
||||
.find(
|
||||
{ $or: [{ file_path: { $ne: null } }, { thumbnail_path: { $ne: null } }] },
|
||||
{ fields: ['file_path', 'thumbnail_path'] },
|
||||
);
|
||||
.find(likeAny<TrekPhotos>(['file_path', 'thumbnail_path'], containing), {
|
||||
fields: ['file_path', 'thumbnail_path'],
|
||||
});
|
||||
return rows.flatMap((row) => [row.file_path, row.thumbnail_path]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { PlaceContribOutbox } from '../../db/entities/PlaceContribOutbox.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -15,8 +15,10 @@ export class PlaceContribStorageRefsService extends StorageReclaimSource {
|
||||
readonly id = 'place-contrib';
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em.getRepository(PlaceContribOutbox).find({}, { fields: ['value'] });
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(PlaceContribOutbox)
|
||||
.find(likeAny<PlaceContribOutbox>(['value'], containing), { fields: ['value'] });
|
||||
return rows.map((row) => row.value);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Places } from '../../db/entities/Places.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -16,8 +16,10 @@ export class PlacesStorageRefsService extends StorageReclaimSource {
|
||||
readonly id = 'places';
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em.getRepository(Places).find({ image_url: { $ne: null } }, { fields: ['image_url'] });
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(Places)
|
||||
.find(likeAny<Places>(['image_url'], containing), { fields: ['image_url'] });
|
||||
return rows.map((row) => row.image_url);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
StorageInvalidKeyError,
|
||||
StorageNotFoundError,
|
||||
type ByteRange,
|
||||
type ListOptions,
|
||||
type LocalTempFile,
|
||||
type ObjectStat,
|
||||
type StorageDriver,
|
||||
@@ -175,13 +176,13 @@ export class LocalDriver implements StorageDriver {
|
||||
}
|
||||
}
|
||||
|
||||
async *list(prefix: string): AsyncIterable<ObjectStat> {
|
||||
async *list(prefix: string, opts: ListOptions = {}): AsyncIterable<ObjectStat> {
|
||||
assertValidPrefix(prefix);
|
||||
const root = this.root();
|
||||
yield* this.walk(prefix ? path.resolve(root, prefix) : root, root);
|
||||
yield* this.walk(prefix ? path.resolve(root, prefix) : root, root, opts.shallow === true);
|
||||
}
|
||||
|
||||
private async *walk(dir: string, root: string): AsyncIterable<ObjectStat> {
|
||||
private async *walk(dir: string, root: string, shallow: boolean): AsyncIterable<ObjectStat> {
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = await fs.promises.readdir(dir, { withFileTypes: true });
|
||||
@@ -195,7 +196,7 @@ export class LocalDriver implements StorageDriver {
|
||||
if (entry.name.startsWith('.')) continue;
|
||||
const entryPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
yield* this.walk(entryPath, root);
|
||||
if (!shallow) yield* this.walk(entryPath, root, false);
|
||||
} else if (entry.isFile()) {
|
||||
const st = await fs.promises.stat(entryPath);
|
||||
yield {
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
StorageInvalidKeyError,
|
||||
StorageNotFoundError,
|
||||
type ByteRange,
|
||||
type ListOptions,
|
||||
type LocalTempFile,
|
||||
type ObjectStat,
|
||||
type PutOptions,
|
||||
@@ -195,11 +196,11 @@ export class MirrorDriver implements StorageDriver {
|
||||
}
|
||||
}
|
||||
|
||||
async *list(prefix: string): AsyncIterable<ObjectStat> {
|
||||
async *list(prefix: string, opts: ListOptions = {}): AsyncIterable<ObjectStat> {
|
||||
assertValidPrefix(prefix);
|
||||
let yieldedAny = false;
|
||||
try {
|
||||
for await (const stat of this.primary.list(prefix)) {
|
||||
for await (const stat of this.primary.list(prefix, opts)) {
|
||||
yieldedAny = true;
|
||||
yield stat;
|
||||
}
|
||||
@@ -210,7 +211,7 @@ export class MirrorDriver implements StorageDriver {
|
||||
if (yieldedAny || !shouldFallback(err)) throw err;
|
||||
for (const replica of this.replicas) {
|
||||
try {
|
||||
for await (const stat of replica.list(prefix)) yield stat;
|
||||
for await (const stat of replica.list(prefix, opts)) yield stat;
|
||||
return;
|
||||
} catch {
|
||||
/* try next replica */
|
||||
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
StorageNotFoundError,
|
||||
isLocalTempFile,
|
||||
type ByteRange,
|
||||
type ListOptions,
|
||||
type LocalTempFile,
|
||||
type ObjectStat,
|
||||
type PutOptions,
|
||||
@@ -403,13 +404,15 @@ export class S3Driver implements StorageDriver {
|
||||
}
|
||||
}
|
||||
|
||||
async *list(prefix: string): AsyncIterable<ObjectStat> {
|
||||
async *list(prefix: string, opts: ListOptions = {}): AsyncIterable<ObjectStat> {
|
||||
assertValidPrefix(prefix);
|
||||
const s3 = await this.client();
|
||||
// A delimiter makes S3 fold every deeper key into CommonPrefixes, which are not read.
|
||||
const shallow = opts.shallow === true ? { Delimiter: '/' } : {};
|
||||
let pages: AsyncIterable<{ Contents?: Array<{ Key?: string; Size?: number; LastModified?: Date }> }>;
|
||||
try {
|
||||
pages = await this.deadlined(
|
||||
s3.ListObjectsV2({ Bucket: this.bucket, Prefix: this.keyPrefix + prefix, paginate: 'iterator' }),
|
||||
s3.ListObjectsV2({ Bucket: this.bucket, Prefix: this.keyPrefix + prefix, paginate: 'iterator', ...shallow }),
|
||||
`list '${prefix}'`,
|
||||
);
|
||||
} catch (err) {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import {
|
||||
storageConfigPutSchema,
|
||||
storageMigrationRequestSchema,
|
||||
storageReclaimRunRequestSchema,
|
||||
storageReclaimConfirmRequestSchema,
|
||||
storageReclaimSettingsRequestSchema,
|
||||
storageTestRequestSchema,
|
||||
} from '@trek/shared';
|
||||
@@ -19,4 +19,4 @@ export class StorageConfigDto extends createZodDto(storageConfigPutSchema) {}
|
||||
export class StorageTestRequestDto extends createZodDto(storageTestRequestSchema) {}
|
||||
export class StorageMigrationRequestDto extends createZodDto(storageMigrationRequestSchema) {}
|
||||
export class StorageReclaimSettingsDto extends createZodDto(storageReclaimSettingsRequestSchema) {}
|
||||
export class StorageReclaimRunDto extends createZodDto(storageReclaimRunRequestSchema) {}
|
||||
export class StorageReclaimConfirmDto extends createZodDto(storageReclaimConfirmRequestSchema) {}
|
||||
|
||||
@@ -4,7 +4,7 @@ import { UnitOfWork } from '../database/unit-of-work';
|
||||
import { StorageJobsService } from './storage-jobs.service';
|
||||
import { getSeedConfigPath } from './storage-paths';
|
||||
import { ephemeralDriverFor, probeDriver, type ProbeTargetResult } from './storage-probe';
|
||||
import { readReclaimState } from './storage-reclaim.settings';
|
||||
import { StorageReclaimService } from './storage-reclaim.service';
|
||||
import { BACKENDS_KEY, CATEGORIES_KEY, VERSION_KEY, StorageRegistryService } from './storage-registry.service';
|
||||
import {
|
||||
assertNoMaskSentinels,
|
||||
@@ -44,6 +44,7 @@ export class StorageAdminService {
|
||||
private readonly jobs: StorageJobsService,
|
||||
private readonly stats: StorageStatsService,
|
||||
private readonly uow: UnitOfWork,
|
||||
private readonly reclaim: StorageReclaimService,
|
||||
) {}
|
||||
|
||||
/** The effective world — secrets masked, categories cross-referenced per backend. */
|
||||
@@ -64,7 +65,7 @@ export class StorageAdminService {
|
||||
health: { replicaFailures: this.storage.health().replicaFailures.map((f) => ({ ...f })) },
|
||||
seedFilePresent: fs.existsSync(getSeedConfigPath()),
|
||||
usage: await this.stats.readUsage(),
|
||||
reclaim: await readReclaimState(this.appSettings),
|
||||
reclaim: await this.reclaim.state(),
|
||||
backfills: this.jobs.statuses(),
|
||||
migrations: this.jobs.migrationStatuses(),
|
||||
version: await this.registry.currentConfigVersion(),
|
||||
|
||||
@@ -45,6 +45,8 @@ export class StorageJobsService {
|
||||
private exclusive: string | null = null;
|
||||
/** Restores in progress: no exclusive pass starts while one runs, and one in flight stops. */
|
||||
private restores = 0;
|
||||
/** Backups in progress: the reclaim pass deletes nothing while one reads the uploads. */
|
||||
private backups = 0;
|
||||
/** Settles once the exclusive pass holding the slot releases it; settled while none does. */
|
||||
private exclusiveReleased: Promise<void> = Promise.resolve();
|
||||
private onExclusiveReleased: (() => void) | null = null;
|
||||
@@ -90,6 +92,33 @@ export class StorageJobsService {
|
||||
return this.restores > 0;
|
||||
}
|
||||
|
||||
/** True while a backup reads the uploads: the reclaim pass deletes nothing then, the archive may still want them. */
|
||||
get backupRunning(): boolean {
|
||||
return this.backups > 0;
|
||||
}
|
||||
|
||||
/** The exclusive pass holding the one-job slot now, by name; null while none does. */
|
||||
get exclusiveHolder(): string | null {
|
||||
return this.exclusive;
|
||||
}
|
||||
|
||||
/**
|
||||
* Settles once no exclusive pass holds the slot (at once when none does): the
|
||||
* nightly usage scan waits here, so its figures follow what a reclaim pass freed.
|
||||
*/
|
||||
async exclusiveIdle(): Promise<void> {
|
||||
await this.exclusiveReleased;
|
||||
}
|
||||
|
||||
/** A backup started reading the uploads; the caller calls `endBackup` in a finally. */
|
||||
beginBackup(): void {
|
||||
this.backups += 1;
|
||||
}
|
||||
|
||||
endBackup(): void {
|
||||
if (this.backups > 0) this.backups -= 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks a restore as running and waits until an exclusive pass in flight
|
||||
* (the reclaim pass, which stops at its next check of `restoreRunning`) has
|
||||
|
||||
@@ -1,27 +1,32 @@
|
||||
import { StorageReclaimRegistryService } from './storage-reclaim-registry.service';
|
||||
import {
|
||||
referencedAmong,
|
||||
keySearchCore,
|
||||
keysNamedIn,
|
||||
type StorageKeyOptions,
|
||||
type StorageReferenceClaim,
|
||||
type StorageReferenceSource,
|
||||
type StorageReferenceSourceId,
|
||||
} from './storage-reclaim.types';
|
||||
import type { ServedCategory } from './storage.types';
|
||||
import { EntityManager } from '@mikro-orm/core';
|
||||
import { EntityManager, type FilterQuery } from '@mikro-orm/core';
|
||||
import { Injectable, type OnApplicationBootstrap } from '@nestjs/common';
|
||||
|
||||
/**
|
||||
* The shape every owner's reference source shares: it registers itself with
|
||||
* the reclaim registry once the app is up, and answers a page of keys by
|
||||
* reading the values its own columns hold and parsing them with
|
||||
* `referencedAmong`, so no source carries a parser of its own.
|
||||
* the reclaim registry once the app is up, and answers by reading the values
|
||||
* its own columns hold and parsing them with `keysNamedIn`, so no source
|
||||
* carries a parser of its own.
|
||||
*
|
||||
* An owner extends this in its own domain, names its id and claims, and reads
|
||||
* its columns through `em.getRepository(<its own entity>)`; storage never
|
||||
* learns which tables those are. The values are read whole on every call:
|
||||
* a column that takes any string (an absolute URL of this instance, a path
|
||||
* with a query, another category's object) cannot be matched with `IN`, and
|
||||
* a re-check right before a delete must see the rows as they are now.
|
||||
* learns which tables those are. A column that takes any string (an absolute
|
||||
* URL of this instance, a path with a query, another category's object)
|
||||
* cannot be matched with `IN`, so the values are read and parsed here:
|
||||
* - `namedKeys` reads every value once, for a pass that judges a whole category;
|
||||
* - `referenced` reads only the rows whose value holds one of the keys' plain
|
||||
* runs (`keySearchCore`, a `LIKE '%run%'` each), fresh, for the question a
|
||||
* delete asks last and for a release right after a user's change. A key with
|
||||
* no such run reads everything, as `namedKeys` does.
|
||||
*/
|
||||
@Injectable()
|
||||
export abstract class StorageReclaimSource implements StorageReferenceSource, OnApplicationBootstrap {
|
||||
@@ -39,11 +44,45 @@ export abstract class StorageReclaimSource implements StorageReferenceSource, On
|
||||
this.registry.register(this);
|
||||
}
|
||||
|
||||
async referenced(category: ServedCategory, keys: readonly string[]): Promise<Set<string>> {
|
||||
if (keys.length === 0 || !this.claims.some((claim) => claim.category === category)) return new Set();
|
||||
return referencedAmong(category, keys, await this.storedValues(), this.keyOptions);
|
||||
async namedKeys(category: ServedCategory): Promise<Set<string>> {
|
||||
if (!this.claimsCategory(category)) return new Set();
|
||||
return keysNamedIn(category, await this.storedValues(null), this.keyOptions);
|
||||
}
|
||||
|
||||
/** Every value the owner's columns hold that may name a stored object; null and empty ones may be among them. */
|
||||
protected abstract storedValues(): Promise<Iterable<string | null | undefined>>;
|
||||
async referenced(category: ServedCategory, keys: readonly string[]): Promise<Set<string>> {
|
||||
if (keys.length === 0 || !this.claimsCategory(category)) return new Set();
|
||||
const cores = keys.map(keySearchCore);
|
||||
const narrow = cores.every((core): core is string => core !== null) ? [...new Set(cores)] : null;
|
||||
const named = keysNamedIn(category, await this.storedValues(narrow), this.keyOptions);
|
||||
return new Set(keys.filter((key) => named.has(key.toLowerCase())));
|
||||
}
|
||||
|
||||
/**
|
||||
* The values the owner's columns hold that may name a stored object; null
|
||||
* and empty ones may be among them. With `containing`, only the rows whose
|
||||
* value holds one of these lowercase runs are needed (`likeAny` builds the
|
||||
* filter); more rows are fine, fewer would lose an object.
|
||||
*/
|
||||
protected abstract storedValues(containing: readonly string[] | null): Promise<Iterable<string | null | undefined>>;
|
||||
|
||||
private claimsCategory(category: ServedCategory): boolean {
|
||||
return this.claims.some((claim) => claim.category === category);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The filter a source narrows its columns with: rows where one of `fields`
|
||||
* holds one of the runs, or every row where one of them is set when there are
|
||||
* none to narrow by. The runs are `[a-z0-9-]` only, so they need no LIKE
|
||||
* escaping; SQLite's LIKE ignores ASCII case, as the parser does.
|
||||
*/
|
||||
export function likeAny<T>(
|
||||
fields: ReadonlyArray<keyof T & string>,
|
||||
containing: readonly string[] | null,
|
||||
): FilterQuery<T> {
|
||||
const terms =
|
||||
containing === null
|
||||
? fields.map((field) => ({ [field]: { $ne: null } }))
|
||||
: fields.flatMap((field) => containing.map((run) => ({ [field]: { $like: `%${run}%` } })));
|
||||
return { $or: terms } as unknown as FilterQuery<T>;
|
||||
}
|
||||
|
||||
@@ -6,23 +6,19 @@ import { CurrentUser } from '../auth-core/current-user.decorator';
|
||||
import { JwtAuthGuard } from '../auth-core/jwt-auth.guard';
|
||||
import { ManagedForbidden } from '../common/managed';
|
||||
import { ResponseContract } from '../common/response-contract';
|
||||
import { StorageReclaimRunDto, StorageReclaimSettingsDto } from './storage-admin.dto';
|
||||
import { StorageReclaimConfirmDto, StorageReclaimSettingsDto } from './storage-admin.dto';
|
||||
import { StorageReclaimService } from './storage-reclaim.service';
|
||||
import { Body, Controller, HttpCode, Post, Put, Req, UseGuards } from '@nestjs/common';
|
||||
import {
|
||||
storageReclaimReportSchema,
|
||||
storageReclaimStateSchema,
|
||||
type StorageReclaimReport,
|
||||
type StorageReclaimState,
|
||||
} from '@trek/shared';
|
||||
import { storageReclaimRunStartedSchema, storageReclaimStateSchema, type StorageReclaimState } from '@trek/shared';
|
||||
|
||||
import type { Request } from 'express';
|
||||
|
||||
/**
|
||||
* /api/admin/storage/reclaim: the admin's switch for the reclaim pass and a
|
||||
* pass on demand. The state itself rides on GET /api/admin/storage. Deleting
|
||||
* stays off until an admin turns it on here; a pass then still deletes only
|
||||
* what has been unreferenced for a week (StorageReclaimService).
|
||||
* /api/admin/storage/reclaim: the admin's switch for the reclaim pass, a check
|
||||
* on demand and the confirmation a category with a large unreferenced share
|
||||
* waits for. The state itself rides on GET /api/admin/storage. Deleting stays
|
||||
* off until an admin turns it on here; a pass then still deletes only what has
|
||||
* been unreferenced for a week (StorageReclaimService).
|
||||
*/
|
||||
@Controller('api/admin/storage/reclaim')
|
||||
@UseGuards(JwtAuthGuard, AdminGuard)
|
||||
@@ -50,27 +46,47 @@ export class StorageReclaimController {
|
||||
return state;
|
||||
}
|
||||
|
||||
/** One pass now, honouring the switch unless `reportOnly`. 409 on a demo or while another storage job runs. */
|
||||
/**
|
||||
* Starts a report-only pass in the background and answers 202 at once; the
|
||||
* admin state says `running` until its report lands. 409 on a demo or while
|
||||
* another storage job, a reclaim pass included, runs.
|
||||
*/
|
||||
@Post('run')
|
||||
@HttpCode(200)
|
||||
@ResponseContract(storageReclaimReportSchema)
|
||||
async run(
|
||||
@CurrentUser() user: User,
|
||||
@Body() body: StorageReclaimRunDto,
|
||||
@Req() req: Request,
|
||||
): Promise<StorageReclaimReport> {
|
||||
const report = await this.reclaim.run({ reportOnly: body.reportOnly === true });
|
||||
const rows = Object.values(report.categories);
|
||||
@HttpCode(202)
|
||||
@ResponseContract(storageReclaimRunStartedSchema)
|
||||
async run(@CurrentUser() user: User, @Req() req: Request): Promise<{ started: true }> {
|
||||
this.reclaim.start();
|
||||
await this.audit.writeAudit({
|
||||
userId: user.id,
|
||||
action: 'admin.storage_reclaim_run',
|
||||
ip: getClientIp(req),
|
||||
details: { mode: 'report' },
|
||||
});
|
||||
return { started: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* Lets the pass delete in a category whose unreferenced share is over a
|
||||
* fifth, up to the count the last report showed. 409 when nothing waits.
|
||||
*/
|
||||
@Post('confirm')
|
||||
@HttpCode(200)
|
||||
@ResponseContract(storageReclaimStateSchema)
|
||||
async confirm(
|
||||
@CurrentUser() user: User,
|
||||
@Body() body: StorageReclaimConfirmDto,
|
||||
@Req() req: Request,
|
||||
): Promise<StorageReclaimState> {
|
||||
const state = await this.reclaim.confirm(body.category);
|
||||
await this.audit.writeAudit({
|
||||
userId: user.id,
|
||||
action: 'admin.storage_reclaim_confirm',
|
||||
ip: getClientIp(req),
|
||||
details: {
|
||||
mode: report.mode,
|
||||
deleted: rows.reduce((sum, row) => sum + row.deleted.objects, 0),
|
||||
bytes: rows.reduce((sum, row) => sum + row.deleted.bytes, 0),
|
||||
category: body.category,
|
||||
unreferenced: state.lastRun?.categories[body.category].unreferenced.objects ?? 0,
|
||||
},
|
||||
});
|
||||
return report;
|
||||
return state;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,14 +2,15 @@ import { RuntimeEnvService } from '../app-config/runtime-env.service';
|
||||
import { logDebug, logError, logInfo } from '../audit/audit-log.logger';
|
||||
import { DomainError } from '../common/domain-error';
|
||||
import { CronRegistrarService } from '../scheduling/cron-registrar.service';
|
||||
import { reportSummary } from './storage-reclaim.report';
|
||||
import { StorageReclaimService } from './storage-reclaim.service';
|
||||
import { Injectable, type OnApplicationBootstrap } from '@nestjs/common';
|
||||
import type { StorageReclaimReport } from '@trek/shared';
|
||||
|
||||
/**
|
||||
* The nightly reclaim pass, 04:05: after the thumbnail sweep at 04:00 and
|
||||
* before the usage scan at 04:15, so the next usage figures already show what
|
||||
* it freed. No pass at boot: a pass that may delete waits for its hour.
|
||||
* The nightly reclaim pass, 04:05: after the thumbnail sweep at 04:00. The
|
||||
* usage scan at 04:15 waits for a pass still running (StorageUsageScanJob), so
|
||||
* its figures show what the pass freed. No pass at boot: a pass that may
|
||||
* delete waits for its hour. A pass that deletes audits itself.
|
||||
*/
|
||||
@Injectable()
|
||||
export class StorageReclaimJob implements OnApplicationBootstrap {
|
||||
@@ -32,7 +33,7 @@ export class StorageReclaimJob implements OnApplicationBootstrap {
|
||||
if (this.runtimeEnv.isDemoMode()) return;
|
||||
try {
|
||||
const report = await this.reclaim.run({ now });
|
||||
const line = summary(report);
|
||||
const line = reportSummary(report);
|
||||
if (line) logInfo(`Storage reclaim: ${line}`);
|
||||
} catch (err: unknown) {
|
||||
if (err instanceof DomainError && err.getStatus() === 409) {
|
||||
@@ -43,20 +44,3 @@ export class StorageReclaimJob implements OnApplicationBootstrap {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** One line when the pass found, deleted or refused something; null when there was nothing to say. */
|
||||
function summary(report: StorageReclaimReport): string | null {
|
||||
const rows = Object.values(report.categories);
|
||||
const unreferenced = rows.reduce((sum, row) => sum + row.unreferenced.objects, 0);
|
||||
const deleted = rows.reduce((sum, row) => sum + row.deleted.objects, 0);
|
||||
const bytes = rows.reduce((sum, row) => sum + row.deleted.bytes, 0);
|
||||
const braked = Object.entries(report.categories)
|
||||
.filter(([, row]) => row.skipped !== null && row.skipped !== 'not_reclaimed')
|
||||
.map(([category, row]) => `${category} (${row.skipped})`);
|
||||
const parts: string[] = [];
|
||||
if (unreferenced > 0) parts.push(`${unreferenced} unreferenced object(s)`);
|
||||
if (deleted > 0) parts.push(`deleted ${deleted} (${bytes} bytes)`);
|
||||
if (braked.length > 0) parts.push(`skipped ${braked.join(', ')}`);
|
||||
if (report.stopped) parts.push(`stopped: ${report.stopped}`);
|
||||
return parts.length > 0 ? parts.join('; ') : null;
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import type { StorageReclaimableCategory, StorageReferenceSourceId } from './storage-reclaim.types';
|
||||
import type { BackendSnapshot } from './storage-registry.service';
|
||||
import type { ServedCategory } from './storage.types';
|
||||
import type { BackendSnapshot, StorageRegistryService } from './storage-registry.service';
|
||||
import { SERVED_CATEGORIES, type ServedCategory } from './storage.types';
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
export type ReclaimRule =
|
||||
@@ -113,6 +114,17 @@ export function sharedLocationCategories(
|
||||
return shared;
|
||||
}
|
||||
|
||||
/** The categories whose resolved location overlaps another's, from the live registry. */
|
||||
export function sharedLocationsOf(registry: StorageRegistryService): Set<ServedCategory> {
|
||||
const assignments = new Map(
|
||||
SERVED_CATEGORIES.map((category) => {
|
||||
const { backendName, keyPrefix } = registry.resolve(category);
|
||||
return [category, { backendName, keyPrefix }] as const;
|
||||
}),
|
||||
);
|
||||
return sharedLocationCategories(assignments, registry.snapshot().backends);
|
||||
}
|
||||
|
||||
function locationsOf(assignment: CategoryAssignment, backends: ReadonlyMap<string, BackendSnapshot>): string[] | null {
|
||||
const backend = backends.get(assignment.backendName);
|
||||
if (!backend) return null;
|
||||
@@ -120,34 +132,54 @@ function locationsOf(assignment: CategoryAssignment, backends: ReadonlyMap<strin
|
||||
backend.type === 'mirror'
|
||||
? [backend.options.primary, ...asList(backend.options.replicas)].map((name) => backends.get(String(name)))
|
||||
: [backend];
|
||||
const bases: string[] = [];
|
||||
const locations: string[] = [];
|
||||
for (const member of members) {
|
||||
const base = member ? baseOf(member) : null;
|
||||
if (base === null) return null;
|
||||
bases.push(base);
|
||||
const location = member ? locationOf(member, assignment.keyPrefix) : null;
|
||||
if (location === null) return null;
|
||||
locations.push(location);
|
||||
}
|
||||
return bases.map((base) => base + assignment.keyPrefix.toLowerCase());
|
||||
return locations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a backend keeps its objects, lowercased: comparing more paths as equal
|
||||
* than the volume does only excludes more, never less.
|
||||
* Where a backend keeps a key prefix's objects, lowercased: comparing more
|
||||
* paths as equal than the volume does only excludes more, never less. A local
|
||||
* root is resolved through its symlinks as LocalDriver resolves it (the image
|
||||
* links /app/server/uploads to /app/uploads), and the prefix's own folder too
|
||||
* when it exists; a root that cannot be resolved reads as unknown, so shared.
|
||||
*/
|
||||
function baseOf(backend: BackendSnapshot): string | null {
|
||||
function locationOf(backend: BackendSnapshot, keyPrefix: string): string | null {
|
||||
const { options } = backend;
|
||||
if (backend.type === 'local') {
|
||||
if (typeof options.root !== 'string' || options.root === '') return null;
|
||||
return `local:${trimSlashes(path.resolve(options.root).replace(/\\/g, '/'))}/`.toLowerCase();
|
||||
const root = realOrNull(path.resolve(options.root));
|
||||
if (root === null) return null;
|
||||
const prefixed = keyPrefix === '' ? null : realOrNull(path.join(root, keyPrefix));
|
||||
const base = prefixed === null ? `${slashed(root)}${keyPrefix}` : slashed(prefixed);
|
||||
return `local:/${trimSlashes(base)}${base.endsWith('/') ? '/' : ''}`.toLowerCase();
|
||||
}
|
||||
if (backend.type === 's3') {
|
||||
if (typeof options.endpoint !== 'string' || typeof options.bucket !== 'string') return null;
|
||||
const keyPrefix = typeof options.keyPrefix === 'string' ? trimSlashes(options.keyPrefix) : '';
|
||||
const bucketPrefix = typeof options.keyPrefix === 'string' ? trimSlashes(options.keyPrefix) : '';
|
||||
const base = `s3:${trimSlashes(options.endpoint)}/${options.bucket}/`;
|
||||
return (keyPrefix === '' ? base : `${base}${keyPrefix}/`).toLowerCase();
|
||||
return `${bucketPrefix === '' ? base : `${base}${bucketPrefix}/`}${keyPrefix}`.toLowerCase();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function realOrNull(target: string): string | null {
|
||||
try {
|
||||
return fs.realpathSync(target);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** A resolved folder with forward slashes and one trailing slash. */
|
||||
function slashed(folder: string): string {
|
||||
return `${folder.replace(/\\/g, '/').replace(/\/+$/, '')}/`;
|
||||
}
|
||||
|
||||
function asList(value: string | number | string[] | undefined): string[] {
|
||||
return Array.isArray(value) ? value : [];
|
||||
}
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
import type { StorageReclaimCategoryReport, StorageReclaimReport } from '@trek/shared';
|
||||
|
||||
/** One object a pass looks at. */
|
||||
export interface ReclaimCandidate {
|
||||
key: string;
|
||||
size: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* What a pass has found and done in one category so far. It is filled in as
|
||||
* the pass goes, so a pass that stops part way still reports what it deleted.
|
||||
*/
|
||||
export interface CategoryProgress {
|
||||
objects: number;
|
||||
unreferenced: ReclaimCandidate[];
|
||||
eligible: ReclaimCandidate[];
|
||||
deleted: ReclaimCandidate[];
|
||||
confirmationRequired: boolean;
|
||||
skipped: StorageReclaimCategoryReport['skipped'];
|
||||
}
|
||||
|
||||
export function newProgress(): CategoryProgress {
|
||||
return { objects: 0, unreferenced: [], eligible: [], deleted: [], confirmationRequired: false, skipped: null };
|
||||
}
|
||||
|
||||
export function reportOf(progress: CategoryProgress): StorageReclaimCategoryReport {
|
||||
return {
|
||||
objects: progress.objects,
|
||||
unreferenced: countOf(progress.unreferenced),
|
||||
eligible: countOf(progress.eligible),
|
||||
deleted: countOf(progress.deleted),
|
||||
confirmationRequired: progress.confirmationRequired,
|
||||
skipped: progress.skipped,
|
||||
};
|
||||
}
|
||||
|
||||
function countOf(candidates: readonly ReclaimCandidate[]): { objects: number; bytes: number } {
|
||||
return { objects: candidates.length, bytes: candidates.reduce((sum, candidate) => sum + candidate.size, 0) };
|
||||
}
|
||||
|
||||
/** What a report says the pass deleted, over every category. */
|
||||
export function deletedTotal(report: StorageReclaimReport): { objects: number; bytes: number } {
|
||||
const rows = Object.values(report.categories);
|
||||
return {
|
||||
objects: rows.reduce((sum, row) => sum + row.deleted.objects, 0),
|
||||
bytes: rows.reduce((sum, row) => sum + row.deleted.bytes, 0),
|
||||
};
|
||||
}
|
||||
|
||||
/** One line when the pass found, deleted or refused something; null when there was nothing to say. */
|
||||
export function reportSummary(report: StorageReclaimReport): string | null {
|
||||
const rows = Object.values(report.categories);
|
||||
const unreferenced = rows.reduce((sum, row) => sum + row.unreferenced.objects, 0);
|
||||
const deleted = deletedTotal(report);
|
||||
const skipped = Object.entries(report.categories)
|
||||
.filter(([, row]) => row.skipped !== null && row.skipped !== 'not_reclaimed')
|
||||
.map(([category, row]) => `${category} (${row.skipped})`);
|
||||
const waiting = Object.entries(report.categories)
|
||||
.filter(([, row]) => row.confirmationRequired)
|
||||
.map(([category]) => category);
|
||||
const parts: string[] = [];
|
||||
if (unreferenced > 0) parts.push(`${unreferenced} unreferenced object(s)`);
|
||||
if (deleted.objects > 0) parts.push(`deleted ${deleted.objects} (${deleted.bytes} bytes)`);
|
||||
if (waiting.length > 0) parts.push(`waiting for an admin to confirm ${waiting.join(', ')}`);
|
||||
if (skipped.length > 0) parts.push(`skipped ${skipped.join(', ')}`);
|
||||
if (report.stopped) parts.push(`stopped: ${report.stopped}`);
|
||||
return parts.length > 0 ? parts.join('; ') : null;
|
||||
}
|
||||
@@ -7,51 +7,71 @@ import type {
|
||||
} from '../../db/repositories/StorageReclaimMarks.repository';
|
||||
import { dbNow, parseDbTimestamp } from '../../db/types';
|
||||
import { RuntimeEnvService } from '../app-config/runtime-env.service';
|
||||
import { logError } from '../audit/audit-log.logger';
|
||||
import { logError, logInfo, logWarn } from '../audit/audit-log.logger';
|
||||
import { AuditService } from '../audit/audit.service';
|
||||
import { DomainError } from '../common/domain-error';
|
||||
import { DatabaseLifecycle } from '../database/database-lifecycle.service';
|
||||
import { withRequestContext } from '../database/request-context';
|
||||
import { UnitOfWork } from '../database/unit-of-work';
|
||||
import { StorageJobsService } from './storage-jobs.service';
|
||||
import { StorageReclaimRegistryService } from './storage-reclaim-registry.service';
|
||||
import { inReclaimScope, RECLAIM_LEDGER, sharedLocationCategories } from './storage-reclaim.ledger';
|
||||
import { inReclaimScope, RECLAIM_LEDGER, sharedLocationsOf } from './storage-reclaim.ledger';
|
||||
import {
|
||||
deletedTotal,
|
||||
newProgress,
|
||||
reportOf,
|
||||
type CategoryProgress,
|
||||
type ReclaimCandidate,
|
||||
} from './storage-reclaim.report';
|
||||
import {
|
||||
INSTALL_ID_KEY,
|
||||
RECLAIM_DELETE_KEY,
|
||||
RECLAIM_DELETE_REPORT_KEY,
|
||||
RECLAIM_REPORT_KEY,
|
||||
RECLAIM_RESET_KEYS,
|
||||
readReclaimConfirmations,
|
||||
readReclaimDeleteEnabled,
|
||||
readReclaimState,
|
||||
readReclaimReport,
|
||||
writeReclaimConfirmations,
|
||||
} from './storage-reclaim.settings';
|
||||
import type { StorageReferenceSource } from './storage-reclaim.types';
|
||||
import { storageOwnerMarkerInstall, storageOwnerMarkerKey, type StorageReferenceSource } from './storage-reclaim.types';
|
||||
import { StorageRegistryService } from './storage-registry.service';
|
||||
import { StorageService } from './storage.service';
|
||||
import { SERVED_CATEGORIES, type ServedCategory } from './storage.types';
|
||||
import { MikroORM } from '@mikro-orm/core';
|
||||
import { InjectRepository } from '@mikro-orm/nestjs';
|
||||
import { Injectable, type OnModuleInit } from '@nestjs/common';
|
||||
import { Injectable, type OnApplicationBootstrap, type OnModuleInit } from '@nestjs/common';
|
||||
import type { StorageReclaimCategoryReport, StorageReclaimReport, StorageReclaimState } from '@trek/shared';
|
||||
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { Readable } from 'node:stream';
|
||||
|
||||
const HOUR_MS = 60 * 60 * 1000;
|
||||
/** An object younger than this is never judged: every writer puts the bytes before it inserts the row. */
|
||||
const RECLAIM_GRACE_MS = 24 * HOUR_MS;
|
||||
/** How long an object must stay unreferenced, on every pass, before a pass may delete it. */
|
||||
const RECLAIM_DELAY_MS = 7 * 24 * HOUR_MS;
|
||||
/** More unreferenced than this share of a category points at a bucket another install shares: hands off. */
|
||||
const RECLAIM_BRAKE_SHARE = 0.2;
|
||||
/** Keys per reference question, the page the sources are promised. */
|
||||
const SOURCE_PAGE = 500;
|
||||
/** More unreferenced than this share of a category waits for an admin's confirmation before anything goes. */
|
||||
const CONFIRM_SHARE = 0.2;
|
||||
/** Keys per re-check right before a delete: one fresh, narrowed question to every source. */
|
||||
const RECHECK_BATCH = 100;
|
||||
/** Deletes in flight at once. */
|
||||
const DELETE_CONCURRENCY = 20;
|
||||
/** Marks per statement: four bound values each, well under SQLite's limit. */
|
||||
const MARK_BATCH = 100;
|
||||
/** Deletes in flight at once. */
|
||||
const DELETE_BATCH = 20;
|
||||
/** Listed objects between two looks at whether a restore began, so a restore waits seconds, not a whole listing. */
|
||||
const LIST_CHECK_EVERY = 500;
|
||||
const JOB_NAME = 'a storage reclaim';
|
||||
|
||||
interface Candidate {
|
||||
key: string;
|
||||
size: number;
|
||||
}
|
||||
const INSTALL_ID = /^[a-z0-9-]{8,64}$/;
|
||||
|
||||
interface PassContext {
|
||||
now: number;
|
||||
installId: string;
|
||||
/** Deletes may go now; turned off for the rest of the pass once the switch reads off or a backup starts. */
|
||||
deleting: boolean;
|
||||
sources: readonly StorageReferenceSource[];
|
||||
shared: ReadonlySet<ServedCategory>;
|
||||
confirmed: Partial<Record<ServedCategory, number>>;
|
||||
generation: number;
|
||||
}
|
||||
|
||||
@@ -61,33 +81,44 @@ interface ReclaimRunOptions {
|
||||
now?: Date;
|
||||
}
|
||||
|
||||
/**
|
||||
* A restore began, or the database was swapped under the pass: nothing it
|
||||
* learnt before holds any more. Reported as `database_reopened` either way.
|
||||
*/
|
||||
/** A restore began, or the database was swapped under the pass: nothing it learnt before holds any more. */
|
||||
class ReclaimStopped extends Error {}
|
||||
|
||||
/**
|
||||
* The reclaim pass (#2924): objects in upload storage that no row names any
|
||||
* more, found in two phases and, only when the admin switched deletion on,
|
||||
* deleted. A pass lists each category the ledger allows, skips objects younger
|
||||
* than a day (the bytes of an upload land before its row), asks every
|
||||
* registered reference source which keys are still named, and marks the rest
|
||||
* with the time it first saw them. An object goes only once its mark is a week
|
||||
* old, after one more question to every source about that single key, through
|
||||
* `StorageService.delete` (so replicas follow and a demo keeps its bytes).
|
||||
* deleted. A pass lists each category the ledger allows at its root, skips
|
||||
* objects younger than a day (the bytes of an upload land before its row),
|
||||
* reads once per category what every registered reference source names, and
|
||||
* marks the rest with the time it first saw them. An object goes only once its
|
||||
* mark is a week old, after one more fresh question to every source about its
|
||||
* batch, through `StorageService.delete` (so replicas follow and a demo keeps
|
||||
* its bytes). What a user deletes or replaces is released at once instead
|
||||
* (StorageReleaseService); the pass is the net for whatever that left behind.
|
||||
*
|
||||
* Brakes, each failing closed: a category whose sources are not all
|
||||
* registered, whose objects share a location with another category's, whose
|
||||
* listing or lookups fail, or where more than a fifth of the objects look
|
||||
* unreferenced is skipped and its marks start over. A pass never runs beside
|
||||
* a backfill, a category migration or a restore (the one-job slot of
|
||||
* StorageJobsService, which a restore waits on), never on a demo instance,
|
||||
* and stops when a restore begins or the database is reopened under it.
|
||||
* Safety, each failing closed:
|
||||
* - a category whose sources are not all registered, whose location overlaps
|
||||
* another category's, or where another installation's owner marker lies
|
||||
* (or this one's cannot be written) is not judged;
|
||||
* - a failed listing or lookup skips the category for the night; no skip
|
||||
* touches the marks, so one bad night does not restart the week;
|
||||
* - more than a fifth unreferenced deletes nothing until an admin confirmed
|
||||
* that count (`confirm`), which no user action can do;
|
||||
* - a delete that fails on the backend stops deleting in that category;
|
||||
* - nothing is deleted while a backup reads the uploads, once the switch is
|
||||
* turned off mid-pass, or on a demo instance;
|
||||
* - a pass never runs beside a backfill, a category migration or a restore
|
||||
* (the one-job slot of StorageJobsService, which a restore waits on), and
|
||||
* stops when a restore begins or the database is reopened under it.
|
||||
*
|
||||
* Every restore turns deletion off and drops the marks, the confirmations and
|
||||
* the reports: the restored rows may come from another installation, and this
|
||||
* one keeps its own install id (written back over the restored one).
|
||||
*/
|
||||
@Injectable()
|
||||
export class StorageReclaimService implements OnModuleInit {
|
||||
export class StorageReclaimService implements OnModuleInit, OnApplicationBootstrap {
|
||||
private generation = 0;
|
||||
private knownInstallId: string | null = null;
|
||||
|
||||
constructor(
|
||||
private readonly storage: StorageService,
|
||||
@@ -96,38 +127,66 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
private readonly jobs: StorageJobsService,
|
||||
private readonly lifecycle: DatabaseLifecycle,
|
||||
private readonly runtimeEnv: RuntimeEnvService,
|
||||
private readonly audit: AuditService,
|
||||
private readonly uow: UnitOfWork,
|
||||
private readonly orm: MikroORM,
|
||||
@InjectRepository(StorageReclaimMarks) private readonly marks: StorageReclaimMarksRepository,
|
||||
@InjectRepository(AppSettings) private readonly appSettings: AppSettingsRepository,
|
||||
) {}
|
||||
|
||||
onModuleInit(): void {
|
||||
this.lifecycle.onReopened(async () => {
|
||||
this.generation += 1;
|
||||
});
|
||||
this.lifecycle.onReopened(() => this.afterReopen());
|
||||
}
|
||||
|
||||
/** Makes the install id on the first boot, so a restore later has one of this install's own to keep. */
|
||||
async onApplicationBootstrap(): Promise<void> {
|
||||
try {
|
||||
await withRequestContext(this.orm, () => this.installId());
|
||||
} catch (err) {
|
||||
logWarn(`[storage] could not read or make the install id: ${messageOf(err)}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Read only, so the admin state's GET never writes: an install id not made yet reads as deletion off. */
|
||||
async state(): Promise<StorageReclaimState> {
|
||||
return await readReclaimState(this.appSettings);
|
||||
const installId = this.knownInstallId ?? (await this.appSettings.getValue(INSTALL_ID_KEY));
|
||||
return {
|
||||
deleteEnabled: installId !== null && (await readReclaimDeleteEnabled(this.appSettings, installId)),
|
||||
running: this.jobs.exclusiveHolder === JOB_NAME,
|
||||
lastRun: await readReclaimReport(this.appSettings, RECLAIM_REPORT_KEY),
|
||||
lastDeleteRun: await readReclaimReport(this.appSettings, RECLAIM_DELETE_REPORT_KEY),
|
||||
};
|
||||
}
|
||||
|
||||
async setDeleteEnabled(enabled: boolean): Promise<StorageReclaimState> {
|
||||
await this.appSettings.setValue(RECLAIM_DELETE_KEY, enabled ? 'true' : 'false');
|
||||
if (enabled) await this.appSettings.upsertOrReplace(RECLAIM_DELETE_KEY, await this.installId());
|
||||
else await this.appSettings.deleteValue(RECLAIM_DELETE_KEY);
|
||||
return await this.state();
|
||||
}
|
||||
|
||||
/**
|
||||
* One pass over every category, its report persisted and returned. Refuses
|
||||
* with a 409 on a demo instance and while another storage job holds the slot.
|
||||
* The admin lets the pass delete in a category whose unreferenced share is
|
||||
* over a fifth, up to the count the last report showed. 409 when the last
|
||||
* report holds nothing that waits for it.
|
||||
*/
|
||||
async confirm(category: ServedCategory): Promise<StorageReclaimState> {
|
||||
const row = (await readReclaimReport(this.appSettings, RECLAIM_REPORT_KEY))?.categories[category];
|
||||
if (!row?.confirmationRequired) {
|
||||
throw refusal('reclaim_nothing_to_confirm', 'nothing in this category waits for a confirmation');
|
||||
}
|
||||
const installId = await this.installId();
|
||||
const counts = await readReclaimConfirmations(this.appSettings, installId);
|
||||
await writeReclaimConfirmations(this.appSettings, installId, { ...counts, [category]: row.unreferenced.objects });
|
||||
return await this.state();
|
||||
}
|
||||
|
||||
/**
|
||||
* One pass over every category in the caller, its report persisted and
|
||||
* returned: the nightly job's. Refuses with a 409 on a demo instance and
|
||||
* while another storage job holds the slot.
|
||||
*/
|
||||
async run(options: ReclaimRunOptions = {}): Promise<StorageReclaimReport> {
|
||||
if (this.runtimeEnv.isDemoMode()) {
|
||||
throw new DomainError(409, 'storage reclaim does not run on a demo instance', { code: 'reclaim_demo' });
|
||||
}
|
||||
if (!this.jobs.claimExclusive(JOB_NAME)) {
|
||||
throw new DomainError(409, 'another storage job is running, one storage job at a time', {
|
||||
code: 'reclaim_busy',
|
||||
});
|
||||
}
|
||||
this.claimSlot();
|
||||
try {
|
||||
return await this.pass(options.reportOnly === true, options.now ?? new Date());
|
||||
} finally {
|
||||
@@ -135,152 +194,269 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A report-only pass in the background, for the admin's check: it can take
|
||||
* longer than a proxy waits for an answer. The same refusals as `run`, at
|
||||
* once; the report lands on the admin state, which says `running` meanwhile.
|
||||
*/
|
||||
start(): void {
|
||||
this.claimSlot();
|
||||
void withRequestContext(this.orm, () => this.pass(true, new Date()))
|
||||
.catch((err: unknown) => {
|
||||
logError(`[storage] the reclaim check failed: ${messageOf(err)}`);
|
||||
})
|
||||
.finally(() => {
|
||||
this.jobs.releaseExclusive(JOB_NAME);
|
||||
});
|
||||
}
|
||||
|
||||
private claimSlot(): void {
|
||||
if (this.runtimeEnv.isDemoMode()) throw refusal('reclaim_demo', 'storage reclaim does not run on a demo instance');
|
||||
if (!this.jobs.claimExclusive(JOB_NAME)) {
|
||||
throw refusal('reclaim_busy', 'another storage job is running, one storage job at a time');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This install's id, made and stored on first use.
|
||||
*
|
||||
* @txStandalone an idempotent lazy provision: the row is made once and only
|
||||
* read after, whatever the caller writes besides.
|
||||
*/
|
||||
private async installId(): Promise<string> {
|
||||
if (this.knownInstallId) return this.knownInstallId;
|
||||
let id = await this.appSettings.getValue(INSTALL_ID_KEY);
|
||||
if (id === null || !INSTALL_ID.test(id)) {
|
||||
id = randomUUID();
|
||||
await this.appSettings.upsertOrReplace(INSTALL_ID_KEY, id);
|
||||
}
|
||||
this.knownInstallId = id;
|
||||
return id;
|
||||
}
|
||||
|
||||
/**
|
||||
* A restore or a reopen swapped the database: a pass in flight stops, and
|
||||
* the restored rows get this install's id back, deletion off and no marks,
|
||||
* confirmations or reports. Contained: a listener that throws fails the reopen.
|
||||
*/
|
||||
private async afterReopen(): Promise<void> {
|
||||
this.generation += 1;
|
||||
try {
|
||||
await withRequestContext(this.orm, () =>
|
||||
this.uow.transactional(async () => {
|
||||
const wasOn = this.knownInstallId !== null && (await this.appSettings.getValue(RECLAIM_DELETE_KEY)) !== null;
|
||||
if (this.knownInstallId) await this.appSettings.upsertOrReplace(INSTALL_ID_KEY, this.knownInstallId);
|
||||
await RECLAIM_RESET_KEYS.reduce<Promise<void>>(async (previous, key) => {
|
||||
await previous;
|
||||
await this.appSettings.deleteValue(key);
|
||||
}, Promise.resolve());
|
||||
await this.marks.deleteAll();
|
||||
if (wasOn)
|
||||
logInfo('[storage] the restored database had reclaim deletion on: it is off until an admin turns it on');
|
||||
}),
|
||||
);
|
||||
} catch (err) {
|
||||
logError(`[storage] could not reset the reclaim state after the database was reopened: ${messageOf(err)}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @txIndependent a sweep in passes: each category's marks and each delete
|
||||
* stand on their own, and a pass cut short leaves the work done so far valid.
|
||||
*/
|
||||
private async pass(reportOnly: boolean, now: Date): Promise<StorageReclaimReport> {
|
||||
const wallStart = Date.now();
|
||||
const installId = await this.installId();
|
||||
const deleting = !reportOnly && (await readReclaimDeleteEnabled(this.appSettings, installId));
|
||||
const ctx: PassContext = {
|
||||
now: now.getTime(),
|
||||
deleting: !reportOnly && (await readReclaimDeleteEnabled(this.appSettings)),
|
||||
installId,
|
||||
deleting,
|
||||
sources: [...this.references.sources()],
|
||||
shared: this.sharedLocations(),
|
||||
shared: sharedLocationsOf(this.registry),
|
||||
confirmed: await readReclaimConfirmations(this.appSettings, installId),
|
||||
generation: this.generation,
|
||||
};
|
||||
const categories = {} as Record<ServedCategory, StorageReclaimCategoryReport>;
|
||||
let stopped: StorageReclaimReport['stopped'] = null;
|
||||
await SERVED_CATEGORIES.reduce<Promise<void>>(async (previous, category) => {
|
||||
await previous;
|
||||
const progress = newProgress();
|
||||
if (stopped === null) {
|
||||
try {
|
||||
categories[category] = await this.reclaimCategory(category, ctx);
|
||||
return;
|
||||
await this.reclaimCategory(category, ctx, progress);
|
||||
} catch (err) {
|
||||
if (!(err instanceof ReclaimStopped)) throw err;
|
||||
stopped = 'database_reopened';
|
||||
stopped = err instanceof ReclaimStopped ? 'database_reopened' : 'failed';
|
||||
if (stopped === 'failed') logError(`[storage] the reclaim pass failed in '${category}': ${messageOf(err)}`);
|
||||
progress.skipped = 'interrupted';
|
||||
}
|
||||
} else {
|
||||
progress.skipped = 'interrupted';
|
||||
}
|
||||
categories[category] = emptyReport(null, 0);
|
||||
categories[category] = reportOf(progress);
|
||||
}, Promise.resolve());
|
||||
const report: StorageReclaimReport = {
|
||||
startedAt: ctx.now,
|
||||
finishedAt: ctx.now + (Date.now() - wallStart),
|
||||
mode: ctx.deleting ? 'delete' : 'report',
|
||||
mode: deleting ? 'delete' : 'report',
|
||||
stopped,
|
||||
categories,
|
||||
};
|
||||
await this.appSettings.upsertOrReplace(RECLAIM_REPORT_KEY, JSON.stringify(report));
|
||||
await this.persist(report);
|
||||
return report;
|
||||
}
|
||||
|
||||
/** The categories whose resolved location overlaps another's, from the live registry. */
|
||||
private sharedLocations(): Set<ServedCategory> {
|
||||
const assignments = new Map(
|
||||
SERVED_CATEGORIES.map((category) => {
|
||||
const { backendName, keyPrefix } = this.registry.resolve(category);
|
||||
return [category, { backendName, keyPrefix }] as const;
|
||||
}),
|
||||
);
|
||||
return sharedLocationCategories(assignments, this.registry.snapshot().backends);
|
||||
/**
|
||||
* The report as the last pass's, and as the last deleting pass's when it ran
|
||||
* with deletion on; an audit entry (the system as actor) when it deleted.
|
||||
*
|
||||
* @txIndependent each row stands on its own: the report, the delete report
|
||||
* and the audit entry describe one finished pass, whichever of them lands.
|
||||
*/
|
||||
private async persist(report: StorageReclaimReport): Promise<void> {
|
||||
const text = JSON.stringify(report);
|
||||
await this.appSettings.upsertOrReplace(RECLAIM_REPORT_KEY, text);
|
||||
if (report.mode !== 'delete') return;
|
||||
await this.appSettings.upsertOrReplace(RECLAIM_DELETE_REPORT_KEY, text);
|
||||
const deleted = deletedTotal(report);
|
||||
if (deleted.objects === 0) return;
|
||||
await this.audit.writeAudit({
|
||||
userId: null,
|
||||
action: 'storage.reclaim_deleted',
|
||||
details: {
|
||||
deleted: deleted.objects,
|
||||
bytes: deleted.bytes,
|
||||
...(report.stopped ? { stopped: report.stopped } : {}),
|
||||
categories: Object.fromEntries(
|
||||
Object.entries(report.categories)
|
||||
.filter(([, row]) => row.deleted.objects > 0)
|
||||
.map(([category, row]) => [category, row.deleted.objects]),
|
||||
),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Judges one category into `progress`, which the caller reports even when
|
||||
* this throws part way.
|
||||
*
|
||||
* @txIndependent a sweep in passes: marks are written per batch and every
|
||||
* delete stands alone, so a failure part way keeps what came before.
|
||||
*/
|
||||
private async reclaimCategory(category: ServedCategory, ctx: PassContext): Promise<StorageReclaimCategoryReport> {
|
||||
private async reclaimCategory(category: ServedCategory, ctx: PassContext, progress: CategoryProgress): Promise<void> {
|
||||
const rule = RECLAIM_LEDGER[category];
|
||||
if (!rule.reclaim) return emptyReport('not_reclaimed', 0);
|
||||
if (!rule.reclaim) {
|
||||
progress.skipped = 'not_reclaimed';
|
||||
return;
|
||||
}
|
||||
const sources = ctx.sources.filter((source) => claimsAny(source, category));
|
||||
const answering = new Set(sources.filter((source) => answersFor(source, category)).map((source) => source.id));
|
||||
if (rule.requiredSources.some((id) => !answering.has(id))) return this.skip(category, 'sources_missing', 0);
|
||||
if (ctx.shared.has(category)) return this.skip(category, 'shared_location', 0);
|
||||
|
||||
let listed: { objects: number; old: Candidate[] };
|
||||
try {
|
||||
listed = await this.listOld(category, ctx.now);
|
||||
} catch (err) {
|
||||
logError(`[storage] reclaim could not list '${category}': ${messageOf(err)}`);
|
||||
return this.skip(category, 'list_failed', 0);
|
||||
if (rule.requiredSources.some((id) => !answering.has(id))) {
|
||||
progress.skipped = 'sources_missing';
|
||||
return;
|
||||
}
|
||||
let unreferenced: Candidate[];
|
||||
if (ctx.shared.has(category)) {
|
||||
progress.skipped = 'shared_location';
|
||||
return;
|
||||
}
|
||||
|
||||
let listed: { objects: number; old: ReclaimCandidate[]; owners: Set<string> };
|
||||
try {
|
||||
unreferenced = await this.unreferencedOf(category, listed.old, sources);
|
||||
listed = await this.listOld(category, ctx);
|
||||
} catch (err) {
|
||||
if (err instanceof ReclaimStopped) throw err;
|
||||
logError(`[storage] reclaim could not list '${category}': ${messageOf(err)}`);
|
||||
progress.skipped = 'list_failed';
|
||||
return;
|
||||
}
|
||||
progress.objects = listed.objects;
|
||||
const owner = await this.ownerVerdict(category, listed.owners, ctx.installId);
|
||||
if (owner !== null) {
|
||||
progress.skipped = owner;
|
||||
return;
|
||||
}
|
||||
|
||||
let named: Set<string>;
|
||||
try {
|
||||
named = await this.namedIn(category, sources);
|
||||
} catch (err) {
|
||||
logError(`[storage] reclaim could not check the references of '${category}': ${messageOf(err)}`);
|
||||
return this.skip(category, 'source_failed', listed.objects);
|
||||
progress.skipped = 'source_failed';
|
||||
return;
|
||||
}
|
||||
this.assertSameDatabase(ctx);
|
||||
const unreferencedCount = countOf(unreferenced);
|
||||
if (unreferenced.length > listed.objects * RECLAIM_BRAKE_SHARE) {
|
||||
return {
|
||||
...(await this.skip(category, 'too_many_unreferenced', listed.objects)),
|
||||
unreferenced: unreferencedCount,
|
||||
};
|
||||
}
|
||||
const unreferenced = listed.old.filter((candidate) => !named.has(candidate.key.toLowerCase()));
|
||||
progress.unreferenced = unreferenced;
|
||||
|
||||
const marks = new Map((await this.marks.marksOf(category)).map((mark) => [mark.key, mark]));
|
||||
await this.writeMarks(category, unreferenced, marks, ctx.now);
|
||||
const cutoff = ctx.now - RECLAIM_DELAY_MS;
|
||||
const eligible = unreferenced.filter((candidate) => markedBefore(marks.get(candidate.key), cutoff));
|
||||
const deleted = ctx.deleting ? await this.deleteEligible(category, eligible, sources, ctx) : [];
|
||||
return {
|
||||
objects: listed.objects,
|
||||
unreferenced: unreferencedCount,
|
||||
eligible: countOf(eligible),
|
||||
deleted: countOf(deleted),
|
||||
skipped: null,
|
||||
};
|
||||
progress.eligible = unreferenced.filter((candidate) => markedBefore(marks.get(candidate.key), cutoff));
|
||||
progress.confirmationRequired =
|
||||
unreferenced.length > listed.objects * CONFIRM_SHARE && unreferenced.length > (ctx.confirmed[category] ?? 0);
|
||||
if (ctx.deleting && !progress.confirmationRequired) await this.deleteEligible(category, sources, ctx, progress);
|
||||
}
|
||||
|
||||
/** A category this pass cannot judge: nothing is deleted and its marks start over. */
|
||||
private async skip(
|
||||
/** The root-level objects of a category, of them the ones past the grace period, and the owner markers there. */
|
||||
private async listOld(
|
||||
category: ServedCategory,
|
||||
reason: NonNullable<StorageReclaimCategoryReport['skipped']>,
|
||||
objects: number,
|
||||
): Promise<StorageReclaimCategoryReport> {
|
||||
await this.marks.deleteCategory(category);
|
||||
return emptyReport(reason, objects);
|
||||
}
|
||||
|
||||
/** The root-level objects of a category, and of them the ones past the grace period. */
|
||||
private async listOld(category: ServedCategory, now: number): Promise<{ objects: number; old: Candidate[] }> {
|
||||
ctx: PassContext,
|
||||
): Promise<{ objects: number; old: ReclaimCandidate[]; owners: Set<string> }> {
|
||||
let objects = 0;
|
||||
const old: Candidate[] = [];
|
||||
for await (const stat of this.storage.list(category)) {
|
||||
let seen = 0;
|
||||
const old: ReclaimCandidate[] = [];
|
||||
const owners = new Set<string>();
|
||||
for await (const stat of this.storage.list(category, '', { shallow: true })) {
|
||||
seen += 1;
|
||||
if (seen % LIST_CHECK_EVERY === 0) this.assertSameDatabase(ctx);
|
||||
if (!inReclaimScope(stat.key)) continue;
|
||||
const owner = storageOwnerMarkerInstall(stat.key);
|
||||
if (owner !== null) {
|
||||
owners.add(owner);
|
||||
continue;
|
||||
}
|
||||
objects += 1;
|
||||
// A missing or zero modification time reads as infinitely old: such an object is kept.
|
||||
if (!(stat.mtimeMs > 0) || now - stat.mtimeMs < RECLAIM_GRACE_MS) continue;
|
||||
if (!(stat.mtimeMs > 0) || ctx.now - stat.mtimeMs < RECLAIM_GRACE_MS) continue;
|
||||
old.push({ key: stat.key, size: stat.size });
|
||||
}
|
||||
return { objects, old };
|
||||
}
|
||||
|
||||
/** The candidates no source names, asked a page at a time. */
|
||||
private async unreferencedOf(
|
||||
category: ServedCategory,
|
||||
candidates: readonly Candidate[],
|
||||
sources: readonly StorageReferenceSource[],
|
||||
): Promise<Candidate[]> {
|
||||
return chunk(candidates, SOURCE_PAGE).reduce<Promise<Candidate[]>>(async (previous, page) => {
|
||||
const done = await previous;
|
||||
const referenced = await this.referencedIn(
|
||||
category,
|
||||
page.map((candidate) => candidate.key),
|
||||
sources,
|
||||
);
|
||||
done.push(...page.filter((candidate) => !referenced.has(candidate.key.toLowerCase())));
|
||||
return done;
|
||||
}, Promise.resolve([]));
|
||||
return { objects, old, owners };
|
||||
}
|
||||
|
||||
/**
|
||||
* The keys some source still names, lowercased. Every key is claimed: the
|
||||
* ledger's required sources answer for the whole category, and a category
|
||||
* missing one never gets this far.
|
||||
* Whether this installation may judge the place a category lives in: not
|
||||
* where another one's owner marker lies, and not where its own could not be
|
||||
* written, since then a second installation cannot see it either.
|
||||
*/
|
||||
private async ownerVerdict(
|
||||
category: ServedCategory,
|
||||
owners: ReadonlySet<string>,
|
||||
installId: string,
|
||||
): Promise<'foreign_install' | 'owner_unknown' | null> {
|
||||
const foreign = [...owners].some((owner) => owner !== installId);
|
||||
if (!owners.has(installId)) {
|
||||
try {
|
||||
await this.storage.put(category, storageOwnerMarkerKey(installId), Readable.from([installId]), {
|
||||
contentType: 'text/plain',
|
||||
});
|
||||
} catch (err) {
|
||||
logWarn(`[storage] reclaim could not write its owner marker into '${category}': ${messageOf(err)}`);
|
||||
return foreign ? 'foreign_install' : 'owner_unknown';
|
||||
}
|
||||
}
|
||||
if (foreign) logWarn(`[storage] another TREK installation keeps its uploads where '${category}' does: not judged`);
|
||||
return foreign ? 'foreign_install' : null;
|
||||
}
|
||||
|
||||
/** Every key of the category some source names, lowercased: one read per source for the whole category. */
|
||||
private async namedIn(category: ServedCategory, sources: readonly StorageReferenceSource[]): Promise<Set<string>> {
|
||||
const answers = await Promise.all(sources.map((source) => source.namedKeys(category)));
|
||||
const named = new Set<string>();
|
||||
for (const answer of answers) for (const key of answer) named.add(key);
|
||||
return named;
|
||||
}
|
||||
|
||||
/** The keys some source names right now, lowercased: the fresh question before a delete. */
|
||||
private async referencedIn(
|
||||
category: ServedCategory,
|
||||
keys: readonly string[],
|
||||
@@ -303,7 +479,7 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
*/
|
||||
private async writeMarks(
|
||||
category: ServedCategory,
|
||||
unreferenced: readonly Candidate[],
|
||||
unreferenced: readonly ReclaimCandidate[],
|
||||
marks: ReadonlyMap<string, ReclaimMarkRow>,
|
||||
now: number,
|
||||
): Promise<void> {
|
||||
@@ -322,48 +498,68 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
}
|
||||
|
||||
/**
|
||||
* The eligible objects a batch at a time, while deleting may still go; a
|
||||
* batch whose backend failed stops the category with `delete_failed`.
|
||||
*
|
||||
* @txIndependent a batch at a time, each delete standing alone, so a failure
|
||||
* leaves the others' work standing.
|
||||
*/
|
||||
private async deleteEligible(
|
||||
category: ServedCategory,
|
||||
eligible: readonly Candidate[],
|
||||
sources: readonly StorageReferenceSource[],
|
||||
ctx: PassContext,
|
||||
): Promise<Candidate[]> {
|
||||
return chunk(eligible, DELETE_BATCH).reduce<Promise<Candidate[]>>(async (previous, batch) => {
|
||||
const done = await previous;
|
||||
done.push(...(await this.deleteBatch(category, batch, sources, ctx)));
|
||||
return done;
|
||||
}, Promise.resolve([]));
|
||||
progress: CategoryProgress,
|
||||
): Promise<void> {
|
||||
await chunk(progress.eligible, RECHECK_BATCH).reduce<Promise<void>>(async (previous, batch) => {
|
||||
await previous;
|
||||
if (progress.skipped !== null || !(await this.mayDelete(ctx))) return;
|
||||
this.assertSameDatabase(ctx);
|
||||
const failed = await this.deleteBatch(category, batch, sources, ctx, progress);
|
||||
if (failed) progress.skipped = 'delete_failed';
|
||||
}, Promise.resolve());
|
||||
}
|
||||
|
||||
/** Deleting goes on while the switch stays on and no backup reads the uploads; once off, off for the pass. */
|
||||
private async mayDelete(ctx: PassContext): Promise<boolean> {
|
||||
if (!ctx.deleting) return false;
|
||||
if (this.jobs.backupRunning || !(await readReclaimDeleteEnabled(this.appSettings, ctx.installId))) {
|
||||
ctx.deleting = false;
|
||||
}
|
||||
return ctx.deleting;
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks each object is still the old one the pass judged, then asks every
|
||||
* source about the batch again and deletes, at once, what none of them names.
|
||||
* The question goes last so nothing slower than the delete itself sits
|
||||
* between the answer and the delete, and once per batch so a pass reads each
|
||||
* source's columns once per batch rather than once per object.
|
||||
* Checks each object is still the old one the pass judged, asks every source
|
||||
* about the batch again, fresh, and deletes what none of them names. The
|
||||
* question goes last so nothing slower than the deletes sits between the
|
||||
* answer and them. True when the backend failed, so the category stops.
|
||||
*
|
||||
* @txIndependent every object goes on its own, see deleteOne.
|
||||
* @txIndependent every object goes on its own; its mark goes after it.
|
||||
*/
|
||||
private async deleteBatch(
|
||||
category: ServedCategory,
|
||||
batch: readonly Candidate[],
|
||||
batch: readonly ReclaimCandidate[],
|
||||
sources: readonly StorageReferenceSource[],
|
||||
ctx: PassContext,
|
||||
): Promise<Candidate[]> {
|
||||
const stats = await Promise.all(batch.map((candidate) => this.storage.stat(category, candidate.key)));
|
||||
progress: CategoryProgress,
|
||||
): Promise<boolean> {
|
||||
let stats: Awaited<ReturnType<StorageService['stat']>>[];
|
||||
try {
|
||||
stats = await Promise.all(batch.map((candidate) => this.storage.stat(category, candidate.key)));
|
||||
} catch (err) {
|
||||
logError(`[storage] reclaim stopped deleting in '${category}', the backend failed: ${messageOf(err)}`);
|
||||
return true;
|
||||
}
|
||||
// Gone, or written again since it was judged: the next pass looks at it afresh.
|
||||
const changed = batch.filter((_candidate, i) => {
|
||||
const stat = stats[i];
|
||||
return stat === null || !(stat.mtimeMs > 0) || ctx.now - stat.mtimeMs < RECLAIM_GRACE_MS;
|
||||
return !stat || !(stat.mtimeMs > 0) || ctx.now - stat.mtimeMs < RECLAIM_GRACE_MS;
|
||||
});
|
||||
if (changed.length > 0) await this.marks.deleteKeys(category, keysOf(changed));
|
||||
const due = batch.filter(
|
||||
(candidate) => !changed.includes(candidate) && !this.storage.keepsOnDelete(category, candidate.key),
|
||||
);
|
||||
if (due.length === 0) return [];
|
||||
if (due.length === 0) return false;
|
||||
let referenced: Set<string>;
|
||||
try {
|
||||
referenced = await this.referencedIn(category, keysOf(due), sources);
|
||||
@@ -371,30 +567,27 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
logError(
|
||||
`[storage] reclaim kept ${due.length} '${category}' object(s), their re-check failed: ${messageOf(err)}`,
|
||||
);
|
||||
return [];
|
||||
return false;
|
||||
}
|
||||
const named = due.filter((candidate) => referenced.has(candidate.key.toLowerCase()));
|
||||
if (named.length > 0) await this.marks.deleteKeys(category, keysOf(named));
|
||||
const unnamed = due.filter((candidate) => !referenced.has(candidate.key.toLowerCase()));
|
||||
this.assertSameDatabase(ctx);
|
||||
const results = await Promise.all(unnamed.map((candidate) => this.deleteOne(category, candidate)));
|
||||
return unnamed.filter((_candidate, i) => results[i]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes one object the batch cleared, then its mark. False keeps it.
|
||||
*
|
||||
* @txIndependent the object goes first and the mark after, so a mark never
|
||||
* outlives a delete that failed.
|
||||
*/
|
||||
private async deleteOne(category: ServedCategory, candidate: Candidate): Promise<boolean> {
|
||||
try {
|
||||
await this.storage.delete(category, candidate.key);
|
||||
} catch (err) {
|
||||
logError(`[storage] reclaim could not delete a '${category}' object: ${messageOf(err)}`);
|
||||
return false;
|
||||
}
|
||||
await this.marks.deleteKeys(category, [candidate.key]);
|
||||
const deleted: ReclaimCandidate[] = [];
|
||||
let failure: unknown = null;
|
||||
await chunk(unnamed, DELETE_CONCURRENCY).reduce<Promise<void>>(async (previous, group) => {
|
||||
await previous;
|
||||
if (failure !== null) return;
|
||||
const results = await Promise.allSettled(group.map((candidate) => this.storage.delete(category, candidate.key)));
|
||||
results.forEach((result, i) => {
|
||||
if (result.status === 'fulfilled') deleted.push(group[i] as ReclaimCandidate);
|
||||
else failure ??= result.reason;
|
||||
});
|
||||
}, Promise.resolve());
|
||||
progress.deleted.push(...deleted);
|
||||
if (deleted.length > 0) await this.marks.deleteKeys(category, keysOf(deleted));
|
||||
if (failure === null) return false;
|
||||
logError(`[storage] reclaim stopped deleting in '${category}' after a failed delete: ${messageOf(failure)}`);
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -405,13 +598,9 @@ export class StorageReclaimService implements OnModuleInit {
|
||||
}
|
||||
}
|
||||
|
||||
function emptyReport(skipped: StorageReclaimCategoryReport['skipped'], objects: number): StorageReclaimCategoryReport {
|
||||
const none = { objects: 0, bytes: 0 };
|
||||
return { objects, unreferenced: none, eligible: none, deleted: none, skipped };
|
||||
}
|
||||
|
||||
function countOf(candidates: readonly Candidate[]): { objects: number; bytes: number } {
|
||||
return { objects: candidates.length, bytes: candidates.reduce((sum, candidate) => sum + candidate.size, 0) };
|
||||
/** A 409 whose body names the refusal in `code` next to the text, so the client can say it in the user's language. */
|
||||
function refusal(code: 'reclaim_busy' | 'reclaim_demo' | 'reclaim_nothing_to_confirm', message: string): DomainError {
|
||||
return new DomainError(409, message, { code, details: { code } });
|
||||
}
|
||||
|
||||
function markedBefore(mark: ReclaimMarkRow | undefined, cutoff: number): boolean {
|
||||
@@ -433,7 +622,7 @@ function claimsKey(source: StorageReferenceSource, category: ServedCategory, key
|
||||
return source.claims.some((claim) => claim.category === category && key.startsWith(claim.prefix));
|
||||
}
|
||||
|
||||
function keysOf(candidates: readonly Candidate[]): string[] {
|
||||
function keysOf(candidates: readonly ReclaimCandidate[]): string[] {
|
||||
return candidates.map((candidate) => candidate.key);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,32 +1,91 @@
|
||||
import type { AppSettingsRepository } from '../../db/repositories/AppSettings.repository';
|
||||
import { logWarn } from '../audit/audit-log.logger';
|
||||
import { storageReclaimReportSchema, type StorageReclaimReport, type StorageReclaimState } from '@trek/shared';
|
||||
import type { ServedCategory } from './storage.types';
|
||||
import { storageReclaimReportSchema, type StorageReclaimReport } from '@trek/shared';
|
||||
|
||||
/** The admin's switch: 'true' lets a pass delete. Absent, or anything else, means report only. */
|
||||
import { z } from 'zod';
|
||||
|
||||
/**
|
||||
* This installation's random id, made on its first boot. A backup restore
|
||||
* brings back the app_settings of whatever install made the backup, so the
|
||||
* reclaim pass keeps the id it booted with and writes it back over the
|
||||
* restored one (StorageReclaimService, on every database reopen). Outside the
|
||||
* `storage.` prefix on purpose: the documented reset of the storage config
|
||||
* (deleting every `storage.%` row) must not make this install a stranger to
|
||||
* the owner markers it wrote.
|
||||
*/
|
||||
export const INSTALL_ID_KEY = 'install_id';
|
||||
/**
|
||||
* The admin's switch, stored as the install id it was turned on for: any other
|
||||
* value, a restored one from another install included, reads as off.
|
||||
*/
|
||||
export const RECLAIM_DELETE_KEY = 'storage.reclaim_delete';
|
||||
/** The last pass's report, one JSON row (the `storage.usage` pattern). */
|
||||
export const RECLAIM_REPORT_KEY = 'storage.reclaim_report';
|
||||
/** The last report of a pass that ran with deletion on, which a check by hand never replaces. */
|
||||
export const RECLAIM_DELETE_REPORT_KEY = 'storage.reclaim_delete_report';
|
||||
/**
|
||||
* Per category, the unreferenced count an admin confirmed when that was more
|
||||
* than a fifth of the category, for the install it was confirmed on.
|
||||
*/
|
||||
const RECLAIM_CONFIRMED_KEY = 'storage.reclaim_confirmed';
|
||||
|
||||
/** Fail closed: only the exact stored 'true' switches deletion on. */
|
||||
export async function readReclaimDeleteEnabled(appSettings: AppSettingsRepository): Promise<boolean> {
|
||||
return (await appSettings.getValue(RECLAIM_DELETE_KEY)) === 'true';
|
||||
/** Every row a restore or a foreign database must not carry over into this install's reclaim. */
|
||||
export const RECLAIM_RESET_KEYS = [
|
||||
RECLAIM_DELETE_KEY,
|
||||
RECLAIM_REPORT_KEY,
|
||||
RECLAIM_DELETE_REPORT_KEY,
|
||||
RECLAIM_CONFIRMED_KEY,
|
||||
] as const;
|
||||
|
||||
const confirmationsSchema = z.object({
|
||||
installId: z.string(),
|
||||
counts: z.record(z.string(), z.number().int().nonnegative()),
|
||||
});
|
||||
|
||||
/** Fail closed: only this install's own id switches deletion on. */
|
||||
export async function readReclaimDeleteEnabled(
|
||||
appSettings: AppSettingsRepository,
|
||||
installId: string,
|
||||
): Promise<boolean> {
|
||||
return (await appSettings.getValue(RECLAIM_DELETE_KEY)) === installId;
|
||||
}
|
||||
|
||||
/** The stored report, parsed; null when there is none or it no longer parses (logged, never a 500). */
|
||||
async function readReclaimReport(appSettings: AppSettingsRepository): Promise<StorageReclaimReport | null> {
|
||||
const value = await appSettings.getValue(RECLAIM_REPORT_KEY);
|
||||
/** A stored report, parsed; null when there is none or it no longer parses (logged, never a 500). */
|
||||
export async function readReclaimReport(
|
||||
appSettings: AppSettingsRepository,
|
||||
key: typeof RECLAIM_REPORT_KEY | typeof RECLAIM_DELETE_REPORT_KEY,
|
||||
): Promise<StorageReclaimReport | null> {
|
||||
const value = await appSettings.getValue(key);
|
||||
if (value == null) return null;
|
||||
try {
|
||||
return storageReclaimReportSchema.parse(JSON.parse(value));
|
||||
} catch {
|
||||
logWarn('[storage] the stored reclaim report is unparseable, treating it as never run');
|
||||
logWarn(`[storage] the stored '${key}' row is unparseable, treating it as never run`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function readReclaimState(appSettings: AppSettingsRepository): Promise<StorageReclaimState> {
|
||||
return {
|
||||
deleteEnabled: await readReclaimDeleteEnabled(appSettings),
|
||||
lastRun: await readReclaimReport(appSettings),
|
||||
};
|
||||
/** The confirmed counts of this install; none for a row another install wrote or that no longer parses. */
|
||||
export async function readReclaimConfirmations(
|
||||
appSettings: AppSettingsRepository,
|
||||
installId: string,
|
||||
): Promise<Partial<Record<ServedCategory, number>>> {
|
||||
const value = await appSettings.getValue(RECLAIM_CONFIRMED_KEY);
|
||||
if (value == null) return {};
|
||||
try {
|
||||
const parsed = confirmationsSchema.parse(JSON.parse(value));
|
||||
return parsed.installId === installId ? (parsed.counts as Partial<Record<ServedCategory, number>>) : {};
|
||||
} catch {
|
||||
logWarn('[storage] the stored reclaim confirmations are unparseable, treating them as none');
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
export async function writeReclaimConfirmations(
|
||||
appSettings: AppSettingsRepository,
|
||||
installId: string,
|
||||
counts: Partial<Record<ServedCategory, number>>,
|
||||
): Promise<void> {
|
||||
await appSettings.upsertOrReplace(RECLAIM_CONFIRMED_KEY, JSON.stringify({ installId, counts }));
|
||||
}
|
||||
|
||||
@@ -5,8 +5,7 @@ import { SERVED_CATEGORIES, type ServedCategory } from './storage.types';
|
||||
* name stored objects (#2924). Storage cannot read those rows itself: it may
|
||||
* import no domain that imports it, and it may not inject another domain's
|
||||
* repositories. So each owner registers a source with
|
||||
* `StorageReclaimRegistryService` and answers, for a page of keys, which of
|
||||
* them its rows still name.
|
||||
* `StorageReclaimRegistryService` and answers which keys its rows still name.
|
||||
*
|
||||
* The ids are a closed list: the ledger in storage names, per category, which
|
||||
* of them must have answered before a single object of that category may be
|
||||
@@ -53,11 +52,17 @@ export interface StorageReferenceSource {
|
||||
*/
|
||||
readonly claims: ReadonlyArray<StorageReferenceClaim>;
|
||||
/**
|
||||
* Of these category-relative keys (as `StorageService.list` yields them, at
|
||||
* most a page of 500), the ones a row still names. A row that is trashed,
|
||||
* Every key of `category` the rows name, lowercased: one read of the whole
|
||||
* column set, which a pass makes once per category. A row that is trashed,
|
||||
* pending or otherwise not gone counts. Throwing stops the pass for the
|
||||
* category; it never deletes on a failed answer.
|
||||
*/
|
||||
namedKeys(category: ServedCategory): Promise<Set<string>>;
|
||||
/**
|
||||
* Of these category-relative keys (as `StorageService.list` yields them, a
|
||||
* batch at a time), the ones a row names right now: the question a delete
|
||||
* asks last, read fresh and narrowed to the rows that can name them.
|
||||
*/
|
||||
referenced(category: ServedCategory, keys: readonly string[]): Promise<Set<string>>;
|
||||
}
|
||||
|
||||
@@ -76,16 +81,18 @@ export interface StorageKeyOptions {
|
||||
}
|
||||
|
||||
const UPLOADS_PATH = /(?:^|\/)uploads\/([a-z0-9][a-z0-9-]*)\//gi;
|
||||
const KEY_END = /[?#"'<>\r\n]/;
|
||||
/** Where a key inside a longer text ends at the latest: a quote, markup or a line break. */
|
||||
const TEXT_END = /["'<>\r\n]/;
|
||||
const SUFFIX = /[?#]/;
|
||||
const SERVED = new Set<string>(SERVED_CATEGORIES);
|
||||
|
||||
/**
|
||||
* Every `/uploads/<category>/<key>` in a value, wherever it stands: a path, an
|
||||
* absolute URL of this instance (`https://host/uploads/covers/x.jpg`), a leading
|
||||
* `uploads/` without the slash, several in one text. A query or a fragment is
|
||||
* dropped, the key is percent-decoded and trimmed. A key with a blank in it is
|
||||
* reported both whole and cut at the blank, since a value may be a sentence:
|
||||
* reporting one key too many only keeps an object, one too few would lose it.
|
||||
* `uploads/` without the slash, several in one text. Each key is reported as it
|
||||
* stands and in its other readings: cut at a query or fragment, percent-decoded,
|
||||
* trimmed, cut at a blank. Reporting one key too many only keeps an object; one
|
||||
* too few would lose it, so the raw spelling is always among them.
|
||||
*/
|
||||
export function uploadKeysIn(value: string | null | undefined): StorageKeyRef[] {
|
||||
if (typeof value !== 'string' || value === '') return [];
|
||||
@@ -94,7 +101,7 @@ export function uploadKeysIn(value: string | null | undefined): StorageKeyRef[]
|
||||
const start = match.index + match[0].length;
|
||||
const next = matches[i + 1];
|
||||
const tail = value.slice(start, next ? next.index : value.length);
|
||||
const end = tail.search(KEY_END);
|
||||
const end = tail.search(TEXT_END);
|
||||
const category = match[1].toLowerCase();
|
||||
return keyVariants(end === -1 ? tail : tail.slice(0, end)).map((key) => ({ category, key }));
|
||||
});
|
||||
@@ -103,26 +110,28 @@ export function uploadKeysIn(value: string | null | undefined): StorageKeyRef[]
|
||||
/**
|
||||
* Every object a stored value can name: the upload paths above, the relative
|
||||
* `<category>/<key>` spelling (`journeys.cover_image` holds `covers/x`), and
|
||||
* with `options` the bare or legacy-prefixed name of one category. A value
|
||||
* that names nothing this way (an https URL elsewhere, a data: URL, garbage)
|
||||
* answers an empty list: it references nothing a pass can find.
|
||||
* with `options` the bare or legacy-prefixed name of one category. A bare name
|
||||
* is the object's exact name (`uuid. #4711`, `uuid.pdf `), so it is reported
|
||||
* untouched first and in its other readings after. A value that names nothing
|
||||
* this way (an https URL elsewhere, a data: URL, garbage) answers an empty list:
|
||||
* it references nothing a pass can find.
|
||||
*/
|
||||
export function storageKeysIn(value: string | null | undefined, options: StorageKeyOptions = {}): StorageKeyRef[] {
|
||||
if (typeof value !== 'string') return [];
|
||||
const trimmed = stripSuffix(value).trim();
|
||||
if (trimmed === '') return [];
|
||||
if (typeof value !== 'string' || value === '') return [];
|
||||
const refs = uploadKeysIn(value);
|
||||
const relative = /^\.?\/?([a-z0-9][a-z0-9-]*)\/(.+)$/i.exec(trimmed);
|
||||
if (relative && SERVED.has(relative[1].toLowerCase())) {
|
||||
const category = relative[1].toLowerCase();
|
||||
for (const key of keyVariants(relative[2])) refs.push({ category, key });
|
||||
for (const form of new Set([value, value.trim(), stripSuffix(value).trim()])) {
|
||||
const relative = /^\.?\/?([a-z0-9][a-z0-9-]*)\/(.+)$/is.exec(form);
|
||||
if (relative && SERVED.has(relative[1].toLowerCase())) {
|
||||
const category = relative[1].toLowerCase();
|
||||
for (const key of keyVariants(relative[2])) refs.push({ category, key });
|
||||
}
|
||||
}
|
||||
if (options.bare) {
|
||||
const bare = options.bare;
|
||||
if (!/[/\\:]/.test(trimmed)) {
|
||||
for (const key of keyVariants(trimmed)) refs.push({ category: bare, key });
|
||||
if (!/[/\\:]/.test(value)) {
|
||||
for (const key of keyVariants(value)) refs.push({ category: bare, key });
|
||||
} else if (options.basename) {
|
||||
const last = trimmed.slice(Math.max(trimmed.lastIndexOf('/'), trimmed.lastIndexOf('\\')) + 1);
|
||||
const last = value.slice(Math.max(value.lastIndexOf('/'), value.lastIndexOf('\\')) + 1);
|
||||
for (const key of keyVariants(last)) refs.push({ category: bare, key });
|
||||
}
|
||||
}
|
||||
@@ -130,14 +139,13 @@ export function storageKeysIn(value: string | null | undefined, options: Storage
|
||||
}
|
||||
|
||||
/**
|
||||
* The candidates of `category` that any of `values` names, compared without
|
||||
* regard to case: a local volume on Windows or macOS serves `A.jpg` for `a.jpg`,
|
||||
* so an object is kept when a value names it in either spelling. The helper a
|
||||
* source builds `referenced()` from, so no source parses with a regex of its own.
|
||||
* The keys of `category` any of `values` names, lowercased: a local volume on
|
||||
* Windows or macOS serves `A.jpg` for `a.jpg`, so an object is kept when a value
|
||||
* names it in either spelling. The helper a source builds its answers from, so
|
||||
* no source parses with a regex of its own.
|
||||
*/
|
||||
export function referencedAmong(
|
||||
export function keysNamedIn(
|
||||
category: ServedCategory,
|
||||
candidates: readonly string[],
|
||||
values: Iterable<string | null | undefined>,
|
||||
options: StorageKeyOptions = {},
|
||||
): Set<string> {
|
||||
@@ -147,25 +155,57 @@ export function referencedAmong(
|
||||
if (ref.category === category) named.add(ref.key.toLowerCase());
|
||||
}
|
||||
}
|
||||
return new Set(candidates.filter((key) => named.has(key.toLowerCase())));
|
||||
return named;
|
||||
}
|
||||
|
||||
/**
|
||||
* The longest run of plain characters in a key (`[a-z0-9-]`, the random part
|
||||
* of every name TREK writes): a row that names the key in any spelling holds
|
||||
* this run verbatim, so a `LIKE '%run%'` finds every row worth parsing. Null
|
||||
* when the key has no run long enough to narrow anything.
|
||||
*/
|
||||
export function keySearchCore(key: string): string | null {
|
||||
const runs = key.toLowerCase().match(/[a-z0-9-]+/g) ?? [];
|
||||
const longest = runs.reduce((best, run) => (run.length > best.length ? run : best), '');
|
||||
return longest.length >= 8 ? longest : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The marker object a pass keeps at the root of each category it judges,
|
||||
* naming this installation (`trek-owner-<install id>`). A second TREK
|
||||
* installation that keeps its uploads in the same place writes its own, and
|
||||
* from then on neither deletes there.
|
||||
*/
|
||||
const OWNER_MARKER_PREFIX = 'trek-owner-';
|
||||
const OWNER_MARKER = /^trek-owner-([a-z0-9-]{8,64})$/;
|
||||
|
||||
export function storageOwnerMarkerKey(installId: string): string {
|
||||
return `${OWNER_MARKER_PREFIX}${installId}`;
|
||||
}
|
||||
|
||||
/** The install id an owner marker names, or null for any other key. Backups leave markers out. */
|
||||
export function storageOwnerMarkerInstall(key: string): string | null {
|
||||
return OWNER_MARKER.exec(key)?.[1] ?? null;
|
||||
}
|
||||
|
||||
function stripSuffix(value: string): string {
|
||||
const cut = value.search(/[?#]/);
|
||||
const cut = value.search(SUFFIX);
|
||||
return cut === -1 ? value : value.slice(0, cut);
|
||||
}
|
||||
|
||||
/** The raw key, decoded, and cut at a blank; never an empty key or one that climbs out of its category. */
|
||||
/** The raw key and its other readings; never an empty one. */
|
||||
function keyVariants(raw: string): string[] {
|
||||
const forms = new Set<string>();
|
||||
const add = (key: string) => {
|
||||
const clean = key.trim().replace(/^\/+/, '');
|
||||
if (clean === '' || clean.split('/').some((segment) => segment === '..' || segment === '.')) return;
|
||||
forms.add(clean);
|
||||
if (key.trim() === '') return;
|
||||
forms.add(key);
|
||||
const unslashed = key.replace(/^\/+/, '');
|
||||
if (unslashed.trim() !== '') forms.add(unslashed);
|
||||
};
|
||||
const whole = stripSuffix(raw);
|
||||
for (const form of [whole, decoded(whole)]) {
|
||||
const stripped = stripSuffix(raw);
|
||||
for (const form of [raw, stripped, decoded(raw), decoded(stripped)]) {
|
||||
add(form);
|
||||
add(form.trim());
|
||||
add(form.trim().split(/\s/)[0] ?? '');
|
||||
}
|
||||
return [...forms];
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
import { CronRegistrarService } from '../scheduling/cron-registrar.service';
|
||||
import { StorageJobsService } from './storage-jobs.service';
|
||||
import { StorageStatsService } from './storage-stats.service';
|
||||
import { Injectable, Logger, OnApplicationBootstrap } from '@nestjs/common';
|
||||
|
||||
/** Nightly usage scan — 04:15, off the backup and thumb-sweep hours. */
|
||||
/**
|
||||
* Nightly usage scan — 04:15, off the backup and thumb-sweep hours. A reclaim
|
||||
* pass still running from 04:05 is waited for first: the two would list the
|
||||
* same backends at once, and the figures should show what the pass freed.
|
||||
*/
|
||||
@Injectable()
|
||||
export class StorageUsageScanJob implements OnApplicationBootstrap {
|
||||
private readonly logger = new Logger(StorageUsageScanJob.name);
|
||||
@@ -10,6 +15,7 @@ export class StorageUsageScanJob implements OnApplicationBootstrap {
|
||||
constructor(
|
||||
private readonly registrar: CronRegistrarService,
|
||||
private readonly stats: StorageStatsService,
|
||||
private readonly jobs: StorageJobsService,
|
||||
) {}
|
||||
|
||||
onApplicationBootstrap(): void {
|
||||
@@ -21,6 +27,7 @@ export class StorageUsageScanJob implements OnApplicationBootstrap {
|
||||
|
||||
async tick(): Promise<void> {
|
||||
try {
|
||||
await this.jobs.exclusiveIdle();
|
||||
await this.stats.scan();
|
||||
} catch (err) {
|
||||
this.logger.error(`usage scan failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
isClientAbortError,
|
||||
StorageNotFoundError,
|
||||
type ByteRange,
|
||||
type ListOptions,
|
||||
type LocalTempFile,
|
||||
type ObjectStat,
|
||||
type PutOptions,
|
||||
@@ -168,9 +169,9 @@ export class StorageService {
|
||||
}
|
||||
|
||||
/** Names in and out are category-relative — the key prefix stays a registry detail. */
|
||||
async *list(category: ServedCategory, subPrefix = ''): AsyncIterable<ObjectStat> {
|
||||
async *list(category: ServedCategory, subPrefix = '', opts?: ListOptions): AsyncIterable<ObjectStat> {
|
||||
const { driver, keyPrefix } = this.registry.resolve(category);
|
||||
for await (const stat of driver.list(keyPrefix + subPrefix)) {
|
||||
for await (const stat of driver.list(keyPrefix + subPrefix, opts)) {
|
||||
yield { ...stat, key: stat.key.slice(keyPrefix.length) };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -98,6 +98,11 @@ export function isClientAbortError(err: unknown): boolean {
|
||||
);
|
||||
}
|
||||
|
||||
export interface ListOptions {
|
||||
/** Only the objects directly under the prefix, none in a folder below it (the reclaim pass judges a category's root). */
|
||||
shallow?: boolean;
|
||||
}
|
||||
|
||||
export interface StorageDriver {
|
||||
/** Backend instance name, e.g. 'uploads-local'. */
|
||||
readonly id: string;
|
||||
@@ -107,7 +112,7 @@ export interface StorageDriver {
|
||||
stat(key: string): Promise<ObjectStat | null>;
|
||||
/** Idempotent; resolving on a missing key is success. */
|
||||
delete(key: string): Promise<void>;
|
||||
list(prefix: string): AsyncIterable<ObjectStat>;
|
||||
list(prefix: string, opts?: ListOptions): AsyncIterable<ObjectStat>;
|
||||
/** Fast-path for local drivers; `null`/absent is exactly the remote-driver branch every caller must handle. */
|
||||
getLocalPath?(key: string): string | null;
|
||||
/** Same-filesystem spool dir (LocalDriver: `<root>/.tmp`), or null for drivers with no local filesystem. */
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { Trips } from '../../db/entities/Trips.entity';
|
||||
import { StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { likeAny, StorageReclaimSource } from '../storage/storage-reclaim-source.service';
|
||||
import { EVERY_RECLAIMABLE_CATEGORY } from '../storage/storage-reclaim.types';
|
||||
import { Injectable } from '@nestjs/common';
|
||||
|
||||
@@ -15,8 +15,10 @@ export class TripsStorageRefsService extends StorageReclaimSource {
|
||||
readonly id = 'trips';
|
||||
readonly claims = EVERY_RECLAIMABLE_CATEGORY;
|
||||
|
||||
protected async storedValues(): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em.getRepository(Trips).find({ cover_image: { $ne: null } }, { fields: ['cover_image'] });
|
||||
protected async storedValues(containing: readonly string[] | null): Promise<Array<string | null | undefined>> {
|
||||
const rows = await this.em
|
||||
.getRepository(Trips)
|
||||
.find(likeAny<Trips>(['cover_image'], containing), { fields: ['cover_image'] });
|
||||
return rows.map((row) => row.cover_image);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,13 +4,15 @@
|
||||
* AdminGuard and ManagedGuard, the Zod pipe on both bodies and the response
|
||||
* contracts the handlers declare all stand in front of it. Every owner
|
||||
* domain registers its reference source here, so a pass looks at every
|
||||
* reclaimable category; the passes below only report. RCLE-001 to RCLE-007.
|
||||
* reclaimable category; the checks below run in the background and only
|
||||
* report. RCLE-001 to RCLE-007.
|
||||
*/
|
||||
import { buildApp } from '../../src/bootstrap';
|
||||
import { AppSettings } from '../../src/db/entities/AppSettings.entity';
|
||||
import { AuditLog } from '../../src/db/entities/AuditLog.entity';
|
||||
import { StorageJobsService } from '../../src/nest/storage/storage-jobs.service';
|
||||
import { STORAGE_RECLAIMABLE_CATEGORIES } from '../../src/nest/storage/storage-reclaim.types';
|
||||
import { STORAGE_RECLAIMABLE_CATEGORIES, storageOwnerMarkerKey } from '../../src/nest/storage/storage-reclaim.types';
|
||||
import { StorageService } from '../../src/nest/storage/storage.service';
|
||||
import { countRows, deleteRows, findRow } from '../helpers/factories/rows';
|
||||
import { makeAdmin, makeUser } from '../helpers/factories/users';
|
||||
import { sessionCookie } from './harness';
|
||||
@@ -50,12 +52,33 @@ describe('Storage reclaim e2e (whole app: real auth + admin guard + managed guar
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
// The uploads folder is the test run's shared one: take back the owner markers this app's checks wrote.
|
||||
const installId = (await findRow(orm, AppSettings, { key: 'install_id' }))?.value;
|
||||
if (installId) {
|
||||
const storage = app.get(StorageService);
|
||||
await Promise.all(
|
||||
STORAGE_RECLAIMABLE_CATEGORIES.map((category) =>
|
||||
storage.delete(category, storageOwnerMarkerKey(installId)).catch(() => undefined),
|
||||
),
|
||||
);
|
||||
}
|
||||
await app?.close();
|
||||
});
|
||||
|
||||
it('RCLE-001: 401 without a session on both routes', async () => {
|
||||
/** The admin state's reclaim part once a check started by hand has finished. */
|
||||
async function settledState() {
|
||||
await app.get(StorageJobsService).exclusiveIdle();
|
||||
const res = await request(server).get('/api/admin/storage').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(200);
|
||||
return res.body.reclaim;
|
||||
}
|
||||
|
||||
it('RCLE-001: 401 without a session on every route', async () => {
|
||||
expect((await request(server).put('/api/admin/storage/reclaim').send({ deleteEnabled: true })).status).toBe(401);
|
||||
expect((await request(server).post('/api/admin/storage/reclaim/run').send({})).status).toBe(401);
|
||||
expect((await request(server).post('/api/admin/storage/reclaim/confirm').send({ category: 'covers' })).status).toBe(
|
||||
401,
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLE-002: 403 for a non-admin, and nothing is stored', async () => {
|
||||
@@ -68,63 +91,79 @@ describe('Storage reclaim e2e (whole app: real auth + admin guard + managed guar
|
||||
expect(await findRow(orm, AppSettings, { key: 'storage.reclaim_delete' })).toBeFalsy();
|
||||
});
|
||||
|
||||
it('RCLE-003: PUT stores the switch, answers the state, audits, and GET /api/admin/storage carries it', async () => {
|
||||
it('RCLE-003: PUT stores the switch for this install, answers the state, audits, and the admin state carries it', async () => {
|
||||
const res = await request(server)
|
||||
.put('/api/admin/storage/reclaim')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ deleteEnabled: true });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ deleteEnabled: true, lastRun: null });
|
||||
expect(res.body).toEqual({ deleteEnabled: true, running: false, lastRun: null, lastDeleteRun: null });
|
||||
expect(await countRows(orm, AuditLog, { action: 'admin.storage_reclaim_update' })).toBe(1);
|
||||
const state = await request(server).get('/api/admin/storage').set('Cookie', adminCookie);
|
||||
expect(state.status).toBe(200);
|
||||
expect(state.body.reclaim).toEqual({ deleteEnabled: true, lastRun: null });
|
||||
// Stored as the install id it was turned on for, so a database restored elsewhere reads as off.
|
||||
const installId = (await findRow(orm, AppSettings, { key: 'install_id' }))?.value;
|
||||
expect(installId).toMatch(/^[0-9a-f-]{36}$/);
|
||||
expect((await findRow(orm, AppSettings, { key: 'storage.reclaim_delete' }))?.value).toBe(installId);
|
||||
expect((await settledState()).deleteEnabled).toBe(true);
|
||||
const off = await request(server)
|
||||
.put('/api/admin/storage/reclaim')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ deleteEnabled: false });
|
||||
expect(off.body.deleteEnabled).toBe(false);
|
||||
});
|
||||
|
||||
it('RCLE-004: PUT refuses a body outside the contract', async () => {
|
||||
it('RCLE-004: PUT and confirm refuse a body outside the contract', async () => {
|
||||
const refused = await Promise.all(
|
||||
[{}, { deleteEnabled: 'yes' }, { deleteEnabled: true, extra: 1 }].map((body) =>
|
||||
request(server).put('/api/admin/storage/reclaim').set('Cookie', adminCookie).send(body),
|
||||
),
|
||||
);
|
||||
expect(refused.map((res) => res.status)).toEqual([400, 400, 400]);
|
||||
const run = await request(server)
|
||||
.post('/api/admin/storage/reclaim/run')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ reportOnly: 'no' });
|
||||
expect(run.status).toBe(400);
|
||||
const confirm = await Promise.all(
|
||||
[{}, { category: 'nowhere' }, { category: 'covers', count: 3 }].map((body) =>
|
||||
request(server).post('/api/admin/storage/reclaim/confirm').set('Cookie', adminCookie).send(body),
|
||||
),
|
||||
);
|
||||
expect(confirm.map((res) => res.status)).toEqual([400, 400, 400]);
|
||||
expect(await countRows(orm, AuditLog)).toBe(0);
|
||||
});
|
||||
|
||||
it('RCLE-005: a report-only pass asks every registered source, deletes nothing, persists and audits', async () => {
|
||||
const res = await request(server)
|
||||
.post('/api/admin/storage/reclaim/run')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ reportOnly: true });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.mode).toBe('report');
|
||||
for (const category of STORAGE_RECLAIMABLE_CATEGORIES) {
|
||||
expect(res.body.categories[category].skipped).not.toBe('sources_missing');
|
||||
expect(res.body.categories[category].deleted).toEqual({ objects: 0, bytes: 0 });
|
||||
}
|
||||
expect(res.body.categories.backups.skipped).toBe('not_reclaimed');
|
||||
it('RCLE-005: POST run answers 202 at once, the check asks every source, deletes nothing and lands on the state', async () => {
|
||||
const res = await request(server).post('/api/admin/storage/reclaim/run').set('Cookie', adminCookie).send({});
|
||||
expect(res.status).toBe(202);
|
||||
expect(res.body).toEqual({ started: true });
|
||||
const audit = await findRow(orm, AuditLog, { action: 'admin.storage_reclaim_run' });
|
||||
expect(JSON.parse(String(audit?.details))).toEqual({ mode: 'report', deleted: 0, bytes: 0 });
|
||||
const state = await request(server).get('/api/admin/storage').set('Cookie', adminCookie);
|
||||
expect(state.body.reclaim.lastRun).toEqual(res.body);
|
||||
expect(JSON.parse(String(audit?.details))).toEqual({ mode: 'report' });
|
||||
const reclaim = await settledState();
|
||||
expect(reclaim.running).toBe(false);
|
||||
expect(reclaim.lastRun.mode).toBe('report');
|
||||
expect(reclaim.lastDeleteRun).toBeNull();
|
||||
for (const category of STORAGE_RECLAIMABLE_CATEGORIES) {
|
||||
expect(reclaim.lastRun.categories[category].skipped).not.toBe('sources_missing');
|
||||
expect(reclaim.lastRun.categories[category].deleted).toEqual({ objects: 0, bytes: 0 });
|
||||
}
|
||||
expect(reclaim.lastRun.categories.backups.skipped).toBe('not_reclaimed');
|
||||
});
|
||||
|
||||
it('RCLE-006: 409 while another storage job holds the slot', async () => {
|
||||
it('RCLE-006: 409 with a code while another storage job holds the slot, and while nothing waits for a confirmation', async () => {
|
||||
const jobs = app.get(StorageJobsService);
|
||||
expect(jobs.claimExclusive('a test job')).toBe(true);
|
||||
try {
|
||||
const res = await request(server).post('/api/admin/storage/reclaim/run').set('Cookie', adminCookie).send({});
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body).toEqual({ error: 'another storage job is running, one storage job at a time' });
|
||||
expect(res.body).toEqual({
|
||||
error: 'another storage job is running, one storage job at a time',
|
||||
code: 'reclaim_busy',
|
||||
});
|
||||
expect(await countRows(orm, AuditLog)).toBe(0);
|
||||
} finally {
|
||||
jobs.releaseExclusive('a test job');
|
||||
}
|
||||
const confirm = await request(server)
|
||||
.post('/api/admin/storage/reclaim/confirm')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ category: 'covers' });
|
||||
expect(confirm.status).toBe(409);
|
||||
expect(confirm.body.code).toBe('reclaim_nothing_to_confirm');
|
||||
});
|
||||
|
||||
it('RCLE-007: managed mode refuses the surface with the standard body', async () => {
|
||||
|
||||
@@ -69,11 +69,11 @@ describe('StorageReclaimMarksRepository', () => {
|
||||
expect((await marks.marksOf('places')).map((m) => m.key)).toEqual(['a.jpg']);
|
||||
});
|
||||
|
||||
it('RCLM-004: drops every mark of one category', async () => {
|
||||
it('RCLM-004: drops every mark of every category, for a database a restore brought back', async () => {
|
||||
await marks.insertMissing('covers', [{ key: 'a.jpg', size: 1 }], '2026-01-01 00:00:00');
|
||||
await marks.insertMissing('places', [{ key: 'a.jpg', size: 1 }], '2026-01-01 00:00:00');
|
||||
expect(await marks.deleteCategory('covers')).toBe(1);
|
||||
expect(await marks.deleteAll()).toBe(2);
|
||||
expect(await marks.marksOf('covers')).toEqual([]);
|
||||
expect(await marks.marksOf('places')).toHaveLength(1);
|
||||
expect(await marks.marksOf('places')).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -448,6 +448,20 @@ describe('BackupService (wrapper)', () => {
|
||||
expect(jobs.claimExclusive('a storage reclaim')).toBe(true);
|
||||
jobs.releaseExclusive('a storage reclaim');
|
||||
});
|
||||
|
||||
it('BKS-BACKUP-001: a backup runs marked as reading the uploads, a failed one too, and the mark ends with it', async () => {
|
||||
let runningInside = false;
|
||||
vi.mocked(backupSvc.createBackup).mockImplementationOnce(async () => {
|
||||
runningInside = jobs.backupRunning;
|
||||
return { filename: 'svc.zip', size: 5 };
|
||||
});
|
||||
await wrapper.createBackup();
|
||||
expect(runningInside).toBe(true);
|
||||
expect(jobs.backupRunning).toBe(false);
|
||||
vi.mocked(backupSvc.createBackup).mockRejectedValueOnce(new Error('disk full'));
|
||||
await expect(wrapper.createBackup()).rejects.toThrow('disk full');
|
||||
expect(jobs.backupRunning).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BackupModule', () => {
|
||||
|
||||
@@ -714,6 +714,23 @@ describe('BACKUP-036 createBackup', () => {
|
||||
expect(storage.getLocalPathOrNull).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('BACKUP-036i — leaves the storage reclaim owner marker out, so a restore elsewhere does not read as shared', async () => {
|
||||
fsMock.existsSync.mockReturnValue(false);
|
||||
setupArchiveSuccess();
|
||||
const storage = stubStorage({
|
||||
stat: statOf(1024),
|
||||
list: listByCategory({
|
||||
covers: [{ key: 'trek-owner-0f8e2b7c-1d2a-4c3b-9e8f-7a6b5c4d3e2f' }, { key: 'cover.jpg' }],
|
||||
}),
|
||||
});
|
||||
|
||||
await createBackup(deps(storage));
|
||||
|
||||
const names = archiverInstanceMock.file.mock.calls.map((c) => c[1]?.name as string);
|
||||
expect(names).toContain('uploads/covers/cover.jpg');
|
||||
expect(names.some((n) => n?.includes('trek-owner-'))).toBe(false);
|
||||
});
|
||||
|
||||
it('BACKUP-036h — only category prefixes are ever archived; backups/ and restore-* are structurally out (issue #1358)', async () => {
|
||||
// The uploads/** glob is gone: even when data and uploads map to the same
|
||||
// directory, enumeration only ever walks the six archived categories, so
|
||||
|
||||
@@ -3,7 +3,11 @@
|
||||
* (#2924), each against the migrated schema: which of a page of keys its rows
|
||||
* still name, in every spelling those columns hold. One object kept too many
|
||||
* costs a few bytes; one called unreferenced while a row names it is lost, so
|
||||
* every case here is a spelling a real row can carry. RSRC-001 to RSRC-011.
|
||||
* every case here is a spelling a real row can carry. A source answers twice:
|
||||
* `namedKeys`, every key of a category its rows name (one read, for a pass),
|
||||
* and `referenced`, a few keys asked fresh and narrowed to the rows holding
|
||||
* their plain runs (the re-check before a delete, and a release).
|
||||
* RSRC-001 to RSRC-014.
|
||||
*/
|
||||
import { PlaceContribOutbox } from '../../../src/db/entities/PlaceContribOutbox.entity';
|
||||
import { TrekPhotos } from '../../../src/db/entities/TrekPhotos.entity';
|
||||
@@ -197,4 +201,36 @@ describe('reference sources', () => {
|
||||
);
|
||||
expect(answers).toEqual(SOURCES.map(() => []));
|
||||
});
|
||||
|
||||
it('RSRC-012: a narrowed question still finds a uuid-named object in every spelling a row holds', async () => {
|
||||
const id = '7b1f0c2e-5a4d-4e3f-8a9b-0c1d2e3f4a5b';
|
||||
await makeTrip(orm, userId, {
|
||||
cover_image: `https://trip.example.test/uploads/covers/${id.toUpperCase()}.JPG?v=3`,
|
||||
});
|
||||
await makePlace(orm, tripId, { image_url: `/uploads/places/${id}-p.jpg#frag` });
|
||||
const trips = t.get(TripsStorageRefsService);
|
||||
expect(await asked(trips, 'covers', [`${id}.jpg`, `${id}-other.jpg`])).toEqual([`${id}.jpg`]);
|
||||
expect(await asked(t.get(PlacesStorageRefsService), 'places', [`${id}-p.jpg`])).toEqual([`${id}-p.jpg`]);
|
||||
});
|
||||
|
||||
it('RSRC-013: namedKeys answers every key of the category its rows name, lowercased, in one read', async () => {
|
||||
await makeTrip(orm, userId, { cover_image: '/uploads/covers/RSRC-Named-A.jpg' });
|
||||
await makeTrip(orm, userId, { cover_image: '/uploads/places/rsrc-named-b.jpg' });
|
||||
const trips = t.get(TripsStorageRefsService);
|
||||
const covers = await trips.namedKeys('covers');
|
||||
expect(covers.has('rsrc-named-a.jpg')).toBe(true);
|
||||
expect(covers.has('rsrc-named-b.jpg')).toBe(false);
|
||||
expect((await trips.namedKeys('places')).has('rsrc-named-b.jpg')).toBe(true);
|
||||
expect(await t.get(FilesStorageRefsService).namedKeys('covers')).toEqual(new Set());
|
||||
});
|
||||
|
||||
it('RSRC-014: a trip file whose name holds a #, a ? or a trailing blank keeps its object both ways', async () => {
|
||||
const id = '3c9d8e7f-6a5b-4c4d-9e3f-2a1b0c9d8e7f';
|
||||
const names = [`${id}. #4711`, `${id}.c#`, `${id}.v2?final`, `${id}.pdf `];
|
||||
await Promise.all(names.map((filename) => makeTripFile(orm, tripId, { filename })));
|
||||
const files = t.get(FilesStorageRefsService);
|
||||
expect(await asked(files, 'files', names)).toEqual([...names].sort());
|
||||
const named = await files.namedKeys('files');
|
||||
for (const name of names) expect(named.has(name.toLowerCase())).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -136,4 +136,17 @@ describe('LocalDriver specifics', () => {
|
||||
for await (const stat of driver.list('nope/')) seen.push(stat);
|
||||
expect(seen).toEqual([]);
|
||||
});
|
||||
|
||||
it('a shallow list yields the objects at the prefix only, none of a folder below it', async () => {
|
||||
const driver = new LocalDriver({ id: 'shallow', root: tmp() });
|
||||
driver.init();
|
||||
await driver.put('journey/top.jpg', Readable.from('a'));
|
||||
await driver.put('journey/thumbs/deep.jpg', Readable.from('b'));
|
||||
const shallow: string[] = [];
|
||||
for await (const stat of driver.list('journey/', { shallow: true })) shallow.push(stat.key);
|
||||
expect(shallow).toEqual(['journey/top.jpg']);
|
||||
const deep: string[] = [];
|
||||
for await (const stat of driver.list('journey/')) deep.push(stat.key);
|
||||
expect(deep.sort()).toEqual(['journey/thumbs/deep.jpg', 'journey/top.jpg']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -174,6 +174,20 @@ describe('MirrorDriver specifics', () => {
|
||||
expect(keys).toEqual(['backup-5.zip']);
|
||||
});
|
||||
|
||||
it('hands a shallow list on to the primary, and to the replica it falls back to', async () => {
|
||||
const fx = track(makeMirror());
|
||||
await fx.mirror.put('top.zip', Readable.from('1'));
|
||||
const primaryList = vi.spyOn(fx.primary, 'list');
|
||||
for await (const stat of fx.mirror.list('', { shallow: true })) void stat;
|
||||
expect(primaryList).toHaveBeenCalledWith('', { shallow: true });
|
||||
primaryList.mockImplementationOnce(() => {
|
||||
throw new StorageBackendError('io');
|
||||
});
|
||||
const replicaList = vi.spyOn(fx.replica, 'list');
|
||||
for await (const stat of fx.mirror.list('', { shallow: true })) void stat;
|
||||
expect(replicaList).toHaveBeenCalledWith('', { shallow: true });
|
||||
});
|
||||
|
||||
it('fans delete out to all targets, tolerating a miss on either side', async () => {
|
||||
const fx = track(makeMirror());
|
||||
await fx.mirror.put('backup-6.zip', Readable.from('bye'));
|
||||
|
||||
@@ -287,6 +287,18 @@ describe('S3Driver list', () => {
|
||||
for await (const stat of makeDriver(api, { keyPrefix: 'trek/prod' }).list('')) out.push(stat);
|
||||
expect(out.map((s) => s.key)).toEqual(['a/ok.bin']);
|
||||
});
|
||||
it('a shallow list asks S3 to fold every deeper key away with a delimiter', async () => {
|
||||
const api = makeMockApi({
|
||||
ListObjectsV2: pages([{ Key: 'trek/prod/a/one.bin', Size: 2, LastModified: new Date(1000) }]),
|
||||
});
|
||||
const out = [];
|
||||
for await (const stat of makeDriver(api, { keyPrefix: 'trek/prod' }).list('a/', { shallow: true })) out.push(stat);
|
||||
expect(out.map((s) => s.key)).toEqual(['a/one.bin']);
|
||||
expect(api.ListObjectsV2).toHaveBeenCalledWith(expect.objectContaining({ Prefix: 'trek/prod/a/', Delimiter: '/' }));
|
||||
const deep = makeMockApi({ ListObjectsV2: pages([]) });
|
||||
for await (const stat of makeDriver(deep).list('a/')) void stat;
|
||||
expect(deep.ListObjectsV2).toHaveBeenCalledWith(expect.not.objectContaining({ Delimiter: '/' }));
|
||||
});
|
||||
it('yields nothing for an empty result and validates the prefix first', async () => {
|
||||
const api = makeMockApi({ ListObjectsV2: pages([]) });
|
||||
const driver = makeDriver(api);
|
||||
|
||||
@@ -4,6 +4,7 @@ import { encrypt_api_key } from '../../../../src/nest/common/crypto/apiKeyCrypto
|
||||
import { StorageAdminService } from '../../../../src/nest/storage/storage-admin.service';
|
||||
import { StorageEventsService } from '../../../../src/nest/storage/storage-events.service';
|
||||
import { StorageJobsService } from '../../../../src/nest/storage/storage-jobs.service';
|
||||
import type { StorageReclaimService } from '../../../../src/nest/storage/storage-reclaim.service';
|
||||
import {
|
||||
StorageRegistryService,
|
||||
BACKENDS_KEY,
|
||||
@@ -74,7 +75,11 @@ async function makeService(opts: { uploadsRoot?: string } = {}) {
|
||||
const storage = new StorageService(registry);
|
||||
const jobs = new StorageJobsService(registry);
|
||||
const stats = new StorageStatsService(storage, appSettings);
|
||||
const service = new StorageAdminService(appSettings, registry, storage, jobs, stats, uow);
|
||||
// The reclaim state has its own suite (storage-reclaim.service.test.ts); here it only rides along.
|
||||
const reclaim = {
|
||||
state: async () => ({ deleteEnabled: false, running: false, lastRun: null, lastDeleteRun: null }),
|
||||
} as unknown as StorageReclaimService;
|
||||
const service = new StorageAdminService(appSettings, registry, storage, jobs, stats, uow, reclaim);
|
||||
return { service, registry, uploadsRoot, stats, jobs, uow };
|
||||
}
|
||||
|
||||
|
||||
@@ -1,17 +1,20 @@
|
||||
/**
|
||||
* StorageReclaimController over a stub service: what it forwards, what it
|
||||
* audits, and that a refusal of the service reaches the client unchanged.
|
||||
* RCLC-001 to RCLC-004.
|
||||
* RCLC-001 to RCLC-006.
|
||||
*/
|
||||
import type { AuditService } from '../../../../src/nest/audit/audit.service';
|
||||
import { DomainError } from '../../../../src/nest/common/domain-error';
|
||||
import type { StorageReclaimRunDto, StorageReclaimSettingsDto } from '../../../../src/nest/storage/storage-admin.dto';
|
||||
import type {
|
||||
StorageReclaimConfirmDto,
|
||||
StorageReclaimSettingsDto,
|
||||
} from '../../../../src/nest/storage/storage-admin.dto';
|
||||
import { StorageReclaimController } from '../../../../src/nest/storage/storage-reclaim.controller';
|
||||
import type { StorageReclaimService } from '../../../../src/nest/storage/storage-reclaim.service';
|
||||
import { StorageModule } from '../../../../src/nest/storage/storage.module';
|
||||
import type { User } from '../../../../src/types';
|
||||
import { expectRegisteredController } from '../../../helpers/module-providers';
|
||||
import { STORAGE_RECLAIM_CATEGORIES, type StorageReclaimReport } from '@trek/shared';
|
||||
import { STORAGE_RECLAIM_CATEGORIES, type StorageReclaimReport, type StorageReclaimState } from '@trek/shared';
|
||||
|
||||
import type { Request } from 'express';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
@@ -21,25 +24,36 @@ const req = { headers: {}, socket: { remoteAddress: '127.0.0.1' } } as unknown a
|
||||
|
||||
function report(): StorageReclaimReport {
|
||||
const none = { objects: 0, bytes: 0 };
|
||||
const row = { objects: 0, unreferenced: none, eligible: none, deleted: none, skipped: null };
|
||||
const row = {
|
||||
objects: 0,
|
||||
unreferenced: none,
|
||||
eligible: none,
|
||||
deleted: none,
|
||||
confirmationRequired: false,
|
||||
skipped: null,
|
||||
};
|
||||
return {
|
||||
startedAt: 1,
|
||||
finishedAt: 2,
|
||||
mode: 'delete',
|
||||
mode: 'report',
|
||||
stopped: null,
|
||||
categories: {
|
||||
...(Object.fromEntries(STORAGE_RECLAIM_CATEGORIES.map((c) => [c, row])) as StorageReclaimReport['categories']),
|
||||
covers: { ...row, objects: 9, deleted: { objects: 2, bytes: 40 } },
|
||||
places: { ...row, objects: 3, deleted: { objects: 1, bytes: 2 } },
|
||||
covers: { ...row, objects: 9, unreferenced: { objects: 4, bytes: 40 }, confirmationRequired: true },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function state(over: Partial<StorageReclaimState> = {}): StorageReclaimState {
|
||||
return { deleteEnabled: false, running: false, lastRun: null, lastDeleteRun: null, ...over };
|
||||
}
|
||||
|
||||
function makeController(over: Partial<Record<keyof StorageReclaimService, unknown>> = {}) {
|
||||
const writeAudit = vi.fn();
|
||||
const service = {
|
||||
setDeleteEnabled: vi.fn(async (enabled: boolean) => ({ deleteEnabled: enabled, lastRun: null })),
|
||||
run: vi.fn(async () => report()),
|
||||
setDeleteEnabled: vi.fn(async (enabled: boolean) => state({ deleteEnabled: enabled })),
|
||||
start: vi.fn(),
|
||||
confirm: vi.fn(async () => state({ lastRun: report() })),
|
||||
...over,
|
||||
} as unknown as StorageReclaimService;
|
||||
const controller = new StorageReclaimController(service, { writeAudit } as unknown as AuditService);
|
||||
@@ -54,32 +68,56 @@ describe('StorageReclaimController', () => {
|
||||
it('RCLC-002: PUT stores the switch, answers the fresh state and audits the value', async () => {
|
||||
const { controller, service, writeAudit } = makeController();
|
||||
const body = { deleteEnabled: true } as StorageReclaimSettingsDto;
|
||||
expect(await controller.update(user, body, req)).toEqual({ deleteEnabled: true, lastRun: null });
|
||||
expect(await controller.update(user, body, req)).toEqual(state({ deleteEnabled: true }));
|
||||
expect(service.setDeleteEnabled).toHaveBeenCalledWith(true);
|
||||
expect(writeAudit).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ userId: 7, action: 'admin.storage_reclaim_update', details: { deleteEnabled: true } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLC-003: POST run forwards reportOnly, answers the report and audits the totals', async () => {
|
||||
it('RCLC-003: POST run starts the check in the background, answers at once and audits the start', async () => {
|
||||
const { controller, service, writeAudit } = makeController();
|
||||
const answer = await controller.run(user, { reportOnly: true } as StorageReclaimRunDto, req);
|
||||
expect(answer.categories.covers.deleted.objects).toBe(2);
|
||||
expect(service.run).toHaveBeenCalledWith({ reportOnly: true });
|
||||
await controller.run(user, {} as StorageReclaimRunDto, req);
|
||||
expect(service.run).toHaveBeenLastCalledWith({ reportOnly: false });
|
||||
expect(await controller.run(user, req)).toEqual({ started: true });
|
||||
expect(service.start).toHaveBeenCalledTimes(1);
|
||||
expect(writeAudit).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ userId: 7, action: 'admin.storage_reclaim_run', details: { mode: 'report' } }),
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLC-004: a refused start propagates as the DomainError and writes no audit row', async () => {
|
||||
const refusal = new DomainError(409, 'another storage job is running, one storage job at a time', {
|
||||
code: 'reclaim_busy',
|
||||
});
|
||||
const { controller, writeAudit } = makeController({
|
||||
start: vi.fn(() => {
|
||||
throw refusal;
|
||||
}),
|
||||
});
|
||||
await expect(controller.run(user, req)).rejects.toBe(refusal);
|
||||
expect(writeAudit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('RCLC-005: POST confirm forwards the category, answers the state and audits the confirmed count', async () => {
|
||||
const { controller, service, writeAudit } = makeController();
|
||||
const answer = await controller.confirm(user, { category: 'covers' } as StorageReclaimConfirmDto, req);
|
||||
expect(answer.lastRun?.categories.covers.confirmationRequired).toBe(true);
|
||||
expect(service.confirm).toHaveBeenCalledWith('covers');
|
||||
expect(writeAudit).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
action: 'admin.storage_reclaim_run',
|
||||
details: { mode: 'delete', deleted: 3, bytes: 42 },
|
||||
action: 'admin.storage_reclaim_confirm',
|
||||
details: { category: 'covers', unreferenced: 4 },
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLC-004: a refused pass propagates as the DomainError and writes no audit row', async () => {
|
||||
const refusal = new DomainError(409, 'another storage job is running, one storage job at a time');
|
||||
const { controller, writeAudit } = makeController({ run: vi.fn().mockRejectedValue(refusal) });
|
||||
await expect(controller.run(user, {} as StorageReclaimRunDto, req)).rejects.toBe(refusal);
|
||||
it('RCLC-006: a confirmation with nothing waiting propagates its 409 and writes no audit row', async () => {
|
||||
const refusal = new DomainError(409, 'nothing in this category waits for a confirmation', {
|
||||
code: 'reclaim_nothing_to_confirm',
|
||||
});
|
||||
const { controller, writeAudit } = makeController({ confirm: vi.fn().mockRejectedValue(refusal) });
|
||||
await expect(controller.confirm(user, { category: 'files' } as StorageReclaimConfirmDto, req)).rejects.toBe(
|
||||
refusal,
|
||||
);
|
||||
expect(writeAudit).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -35,7 +35,14 @@ function report(
|
||||
stopped: StorageReclaimReport['stopped'] = null,
|
||||
) {
|
||||
const none = { objects: 0, bytes: 0 };
|
||||
const row = { objects: 0, unreferenced: none, eligible: none, deleted: none, skipped: null };
|
||||
const row = {
|
||||
objects: 0,
|
||||
unreferenced: none,
|
||||
eligible: none,
|
||||
deleted: none,
|
||||
confirmationRequired: false,
|
||||
skipped: null,
|
||||
};
|
||||
return {
|
||||
startedAt: 1,
|
||||
finishedAt: 2,
|
||||
@@ -104,15 +111,18 @@ describe('StorageReclaimJob', () => {
|
||||
unreferenced: { objects: 2, bytes: 30 },
|
||||
eligible: { objects: 1, bytes: 10 },
|
||||
deleted: { objects: 1, bytes: 10 },
|
||||
confirmationRequired: false,
|
||||
skipped: null,
|
||||
},
|
||||
avatars: { ...report().categories.avatars, skipped: 'sources_missing' },
|
||||
files: { ...report().categories.files, confirmationRequired: true },
|
||||
avatars: { ...report().categories.avatars, skipped: 'interrupted' },
|
||||
},
|
||||
'database_reopened',
|
||||
);
|
||||
await makeJob(async () => busy).job.tick();
|
||||
expect(logInfo).toHaveBeenCalledWith(
|
||||
'Storage reclaim: 2 unreferenced object(s); deleted 1 (10 bytes); skipped avatars (sources_missing); stopped: database_reopened',
|
||||
'Storage reclaim: 2 unreferenced object(s); deleted 1 (10 bytes); waiting for an admin to confirm files; ' +
|
||||
'skipped avatars (interrupted); stopped: database_reopened',
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
/**
|
||||
* The reclaim ledger and the location check (#2924), plus the registry the
|
||||
* reference sources land in. RCLL-001 to RCLL-011.
|
||||
* reference sources land in. Local roots are resolved through their symlinks,
|
||||
* so the layouts live in a temp folder. RCLL-001 to RCLL-013.
|
||||
*/
|
||||
import { StorageReclaimRegistryService } from '../../../../src/nest/storage/storage-reclaim-registry.service';
|
||||
import {
|
||||
@@ -21,8 +22,10 @@ import type { BackendSnapshot } from '../../../../src/nest/storage/storage-regis
|
||||
import { CATEGORY_PREFIXES } from '../../../../src/nest/storage/storage-registry.service';
|
||||
import { SERVED_CATEGORIES, type ServedCategory } from '../../../../src/nest/storage/storage.types';
|
||||
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { afterAll, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
vi.mock('../../../../src/nest/audit/audit-log.logger', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('../../../../src/nest/audit/audit-log.logger')>()),
|
||||
@@ -32,8 +35,20 @@ vi.mock('../../../../src/nest/audit/audit-log.logger', async (importOriginal) =>
|
||||
logWarn: vi.fn(),
|
||||
}));
|
||||
|
||||
const UPLOADS = path.resolve('/srv/trek/uploads');
|
||||
const BACKUPS = path.resolve('/srv/trek/backups');
|
||||
const ROOT = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'trek-reclaim-ledger-')));
|
||||
/** A folder under the temp root, made if missing: a location is only resolved when its root exists. */
|
||||
function dir(...parts: string[]): string {
|
||||
const target = path.join(ROOT, ...parts);
|
||||
fs.mkdirSync(target, { recursive: true });
|
||||
return target;
|
||||
}
|
||||
const UPLOADS = dir('uploads');
|
||||
const BACKUPS = dir('backups');
|
||||
for (const category of SERVED_CATEGORIES) dir('uploads', CATEGORY_PREFIXES[category]);
|
||||
|
||||
afterAll(() => {
|
||||
fs.rmSync(ROOT, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function local(name: string, root: string): BackendSnapshot {
|
||||
return { name, type: 'local', source: 'settings', options: { root } };
|
||||
@@ -90,7 +105,7 @@ describe('sharedLocationCategories', () => {
|
||||
});
|
||||
|
||||
it('RCLL-005: TREK_PLACE_PHOTO_DIR inside the uploads folder overlaps the category it lands in', () => {
|
||||
const backends = [...BASE_BACKENDS, local('place-photos-local', path.join(UPLOADS, 'places'))];
|
||||
const backends = [...BASE_BACKENDS, local('place-photos-local', dir('uploads', 'places'))];
|
||||
const shared = sharedLocationCategories(
|
||||
defaults({ 'photos-google': { backendName: 'place-photos-local', keyPrefix: '' } }),
|
||||
backends,
|
||||
@@ -108,8 +123,8 @@ describe('sharedLocationCategories', () => {
|
||||
expect(shared.has('covers')).toBe(false);
|
||||
});
|
||||
|
||||
it('RCLL-007: two backends of other names on one folder overlap, compared without regard to case', () => {
|
||||
const backends = [...BASE_BACKENDS, local('again', UPLOADS.toUpperCase() + path.sep)];
|
||||
it('RCLL-007: two backends of other names on one folder overlap, however the root is spelled', () => {
|
||||
const backends = [...BASE_BACKENDS, local('again', `${path.join(UPLOADS, 'files', '..')}${path.sep}`)];
|
||||
const shared = sharedLocationCategories(
|
||||
defaults({ avatars: { backendName: 'again', keyPrefix: 'covers/' } }),
|
||||
backends,
|
||||
@@ -145,8 +160,8 @@ describe('sharedLocationCategories', () => {
|
||||
it('RCLL-009: a mirror counts its primary and every replica; a replica inside another category excludes it', () => {
|
||||
const backends: BackendSnapshot[] = [
|
||||
...BASE_BACKENDS,
|
||||
local('primary', path.resolve('/mnt/primary')),
|
||||
local('replica', path.join(UPLOADS, 'journey')),
|
||||
local('primary', dir('primary')),
|
||||
local('replica', dir('uploads', 'journey')),
|
||||
{ name: 'm', type: 'mirror', source: 'settings', options: { primary: 'primary', replicas: ['replica'] } },
|
||||
];
|
||||
const shared = sharedLocationCategories(defaults({ covers: { backendName: 'm', keyPrefix: 'covers/' } }), backends);
|
||||
@@ -155,6 +170,29 @@ describe('sharedLocationCategories', () => {
|
||||
expect(shared.has('covers')).toBe(false);
|
||||
});
|
||||
|
||||
it('RCLL-012: a place-photo folder reached through a symlink still overlaps the category it really is', () => {
|
||||
// The image links /app/server/uploads to /app/uploads; an operator names the latter.
|
||||
const linked = path.join(ROOT, 'linked-uploads');
|
||||
if (!fs.existsSync(linked)) fs.symlinkSync(UPLOADS, linked, 'junction');
|
||||
const backends = [...BASE_BACKENDS, local('place-photos-local', path.join(linked, 'places'))];
|
||||
const shared = sharedLocationCategories(
|
||||
defaults({ 'photos-google': { backendName: 'place-photos-local', keyPrefix: '' } }),
|
||||
backends,
|
||||
);
|
||||
expect(shared.has('places')).toBe(true);
|
||||
expect(shared.has('covers')).toBe(false);
|
||||
});
|
||||
|
||||
it('RCLL-013: a local root that does not resolve reads as unknown, so shared', () => {
|
||||
const backends = [...BASE_BACKENDS, local('gone', path.join(ROOT, 'never-made'))];
|
||||
const shared = sharedLocationCategories(
|
||||
defaults({ covers: { backendName: 'gone', keyPrefix: 'covers/' } }),
|
||||
backends,
|
||||
);
|
||||
expect(shared.has('covers')).toBe(true);
|
||||
expect(shared.has('files')).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLL-010: a location that cannot be resolved fails closed, for itself and for every other category', () => {
|
||||
const backends: BackendSnapshot[] = [
|
||||
...BASE_BACKENDS,
|
||||
@@ -178,6 +216,7 @@ describe('StorageReclaimRegistryService', () => {
|
||||
const source = (id: StorageReferenceSource['id']): StorageReferenceSource => ({
|
||||
id,
|
||||
claims: [{ category: 'covers', prefix: '' }],
|
||||
namedKeys: async () => new Set<string>(),
|
||||
referenced: async () => new Set<string>(),
|
||||
});
|
||||
|
||||
|
||||
@@ -3,11 +3,15 @@
|
||||
* and LocalDriver on temp folders, the marks table in the snapshot db, and
|
||||
* fake reference sources standing in for the owner domains. The pass marks
|
||||
* first, deletes a week later only with the switch on and only after asking
|
||||
* every source about the single key again; every brake skips a category and
|
||||
* starts its marks over. RCLS-001 to RCLS-024.
|
||||
* every source about its batch again. A skip keeps the marks; a share over a
|
||||
* fifth waits for an admin's confirmation; an owner marker of another install
|
||||
* keeps its hands off; a restore resets the switch, the marks and the reports
|
||||
* but keeps this install's id. RCLS-001 to RCLS-036.
|
||||
*/
|
||||
import { AppSettings } from '../../../../src/db/entities/AppSettings.entity';
|
||||
import { AuditLog } from '../../../../src/db/entities/AuditLog.entity';
|
||||
import { StorageReclaimMarks } from '../../../../src/db/entities/StorageReclaimMarks.entity';
|
||||
import { StorageReclaimMarksRepository } from '../../../../src/db/repositories/StorageReclaimMarks.repository';
|
||||
import { dbNow } from '../../../../src/db/types';
|
||||
import { RuntimeEnvService } from '../../../../src/nest/app-config/runtime-env.service';
|
||||
import { logError } from '../../../../src/nest/audit/audit-log.logger';
|
||||
@@ -20,6 +24,7 @@ import { StorageReclaimService } from '../../../../src/nest/storage/storage-recl
|
||||
import {
|
||||
STORAGE_RECLAIMABLE_CATEGORIES,
|
||||
STORAGE_REFERENCE_SOURCE_IDS,
|
||||
storageOwnerMarkerKey,
|
||||
type StorageReferenceClaim,
|
||||
type StorageReferenceSource,
|
||||
type StorageReferenceSourceId,
|
||||
@@ -28,7 +33,7 @@ import { StorageModule } from '../../../../src/nest/storage/storage.module';
|
||||
import { StorageService } from '../../../../src/nest/storage/storage.service';
|
||||
import type { ServedCategory } from '../../../../src/nest/storage/storage.types';
|
||||
import { createSnapshotTestDb } from '../../../helpers/db-mock';
|
||||
import { deleteRows } from '../../../helpers/factories/rows';
|
||||
import { countRows, deleteRows, findRow } from '../../../helpers/factories/rows';
|
||||
import { readAppSetting, setAppSetting } from '../../../helpers/factories/settings';
|
||||
import { createTestModule, type TestModule } from '../../../helpers/test-module';
|
||||
import { createTestOrm, type TestOrm } from '../../../helpers/test-orm';
|
||||
@@ -55,12 +60,14 @@ let uploadsRoot: string;
|
||||
let backupsRoot: string;
|
||||
let reclaim: StorageReclaimService;
|
||||
let storage: StorageService;
|
||||
let jobs: StorageJobsService;
|
||||
let references: StorageReclaimRegistryService;
|
||||
let reopen: () => Promise<void>;
|
||||
let installId: string;
|
||||
/** What the fake sources still name, per category. */
|
||||
let named: Map<ServedCategory, Set<string>>;
|
||||
/** Called on every question a source is asked, before it answers. */
|
||||
let onAsk: ((category: ServedCategory, keys: readonly string[]) => void | Promise<void>) | null;
|
||||
/** Called on every question a source is asked, before it answers: `keys` is null for a whole-category read. */
|
||||
let onAsk: ((category: ServedCategory, keys: readonly string[] | null) => void | Promise<void>) | null;
|
||||
|
||||
/** The keys the fake sources name in `category`, created empty on first use. */
|
||||
function namedIn(category: ServedCategory): Set<string> {
|
||||
@@ -78,6 +85,10 @@ function fakeSource(id: StorageReferenceSourceId): StorageReferenceSource {
|
||||
return {
|
||||
id,
|
||||
claims: categories.map((category): StorageReferenceClaim => ({ category, prefix: '' })),
|
||||
async namedKeys(category) {
|
||||
await onAsk?.(category, null);
|
||||
return new Set([...(named.get(category) ?? [])].map((key) => key.toLowerCase()));
|
||||
},
|
||||
async referenced(category, keys) {
|
||||
await onAsk?.(category, keys);
|
||||
const set = named.get(category) ?? new Set<string>();
|
||||
@@ -102,7 +113,7 @@ function objectPath(category: string, key: string): string {
|
||||
return path.join(uploadsRoot, category, key);
|
||||
}
|
||||
|
||||
/** The orphans plus `kept` named covers, so the orphans stay under the brake's fifth. */
|
||||
/** The orphans plus `kept` named covers, so the orphans stay under a fifth. */
|
||||
async function seedCovers(orphans: string[], kept = 4): Promise<void> {
|
||||
const keptKeys = Array.from({ length: kept }, (_unused, i) => `kept-${i}.jpg`);
|
||||
await Promise.all([...orphans, ...keptKeys].map((key) => put('covers', key)));
|
||||
@@ -116,6 +127,24 @@ async function marksOf(category: string): Promise<Array<{ key: string; first_see
|
||||
.sort((a, b) => a.key.localeCompare(b.key));
|
||||
}
|
||||
|
||||
/** What a synchronous call threw, or null. */
|
||||
function thrownBy(call: () => unknown): unknown {
|
||||
try {
|
||||
call();
|
||||
} catch (err) {
|
||||
return err;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function markWeekOld(category: ServedCategory, keys: string[]): Promise<void> {
|
||||
await orm.repo(StorageReclaimMarks).insertMissing(
|
||||
category,
|
||||
keys.map((key) => ({ key, size: 5 })),
|
||||
dbNow(new Date(0)),
|
||||
);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
uploadsRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'trek-reclaim-uploads-'));
|
||||
backupsRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'trek-reclaim-backups-'));
|
||||
@@ -138,8 +167,10 @@ beforeAll(async () => {
|
||||
});
|
||||
reclaim = t.get(StorageReclaimService);
|
||||
storage = t.get(StorageService);
|
||||
jobs = t.get(StorageJobsService);
|
||||
references = t.get(StorageReclaimRegistryService);
|
||||
for (const id of STORAGE_REFERENCE_SOURCE_IDS) references.register(fakeSource(id));
|
||||
installId = (await readAppSetting(orm, 'install_id')) ?? '';
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
@@ -150,6 +181,7 @@ beforeEach(async () => {
|
||||
}
|
||||
await deleteRows(orm, StorageReclaimMarks);
|
||||
await deleteRows(orm, AppSettings, { key: { $like: 'storage.reclaim%' } });
|
||||
await deleteRows(orm, AuditLog);
|
||||
named = new Map();
|
||||
onAsk = null;
|
||||
});
|
||||
@@ -165,8 +197,15 @@ afterAll(async () => {
|
||||
const T0 = new Date(Date.now() + 2 * DAY);
|
||||
const WEEK_LATER = new Date(T0.getTime() + 8 * DAY);
|
||||
|
||||
async function markedWeekOld(orphans: string[], kept = 4): Promise<void> {
|
||||
await seedCovers(orphans, kept);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
}
|
||||
|
||||
describe('StorageReclaimService: mark, then sweep', () => {
|
||||
it('RCLS-001: a first pass only reports and marks; the report is persisted and the state carries it', async () => {
|
||||
it('RCLS-001: a first pass reports and marks; the report is persisted and both states carry it', async () => {
|
||||
expect(installId).toMatch(/^[0-9a-f-]{36}$/);
|
||||
await seedCovers(['orphan.jpg']);
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.mode).toBe('report');
|
||||
@@ -176,6 +215,7 @@ describe('StorageReclaimService: mark, then sweep', () => {
|
||||
unreferenced: { objects: 1, bytes: 5 },
|
||||
eligible: { objects: 0, bytes: 0 },
|
||||
deleted: { objects: 0, bytes: 0 },
|
||||
confirmationRequired: false,
|
||||
skipped: null,
|
||||
});
|
||||
expect(report.categories.files).toMatchObject({ objects: 0, skipped: null });
|
||||
@@ -183,41 +223,44 @@ describe('StorageReclaimService: mark, then sweep', () => {
|
||||
expect(report.categories[category].skipped).toBe('not_reclaimed');
|
||||
}
|
||||
expect(await marksOf('covers')).toEqual([{ key: 'orphan.jpg', first_seen_at: dbNow(T0) }]);
|
||||
expect(await reclaim.state()).toEqual({ deleteEnabled: false, lastRun: report });
|
||||
expect((await t.get(StorageAdminService).state()).reclaim).toEqual({ deleteEnabled: false, lastRun: report });
|
||||
const expected = { deleteEnabled: false, running: false, lastRun: report, lastDeleteRun: null };
|
||||
expect(await reclaim.state()).toEqual(expected);
|
||||
expect((await t.get(StorageAdminService).state()).reclaim).toEqual(expected);
|
||||
});
|
||||
|
||||
it('RCLS-002: with deletion on, a mark a week old goes through the facade and takes its mark along', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
const first = await reclaim.run({ now: T0 });
|
||||
expect(first.mode).toBe('delete');
|
||||
expect(first.categories.covers.deleted.objects).toBe(0);
|
||||
it('RCLS-002: with deletion on, a mark a week old goes through the facade, takes its mark and is audited', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
|
||||
const second = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(second.mode).toBe('delete');
|
||||
expect(second.categories.covers.eligible).toEqual({ objects: 1, bytes: 5 });
|
||||
expect(second.categories.covers.deleted).toEqual({ objects: 1, bytes: 5 });
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(false);
|
||||
expect(fs.existsSync(objectPath('covers', 'kept-0.jpg'))).toBe(true);
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect((await reclaim.state()).lastDeleteRun).toEqual(second);
|
||||
const audit = await findRow(orm, AuditLog, { action: 'storage.reclaim_deleted' });
|
||||
expect(audit?.user_id ?? null).toBeNull();
|
||||
expect(JSON.parse(String(audit?.details))).toEqual({ deleted: 1, bytes: 5, categories: { covers: 1 } });
|
||||
});
|
||||
|
||||
it('RCLS-003: a report-only pass deletes nothing even with deletion on', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
it('RCLS-003: a report-only pass deletes nothing even with deletion on, and leaves the last deleting report', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
const deleting = (await reclaim.state()).lastDeleteRun;
|
||||
const report = await reclaim.run({ now: WEEK_LATER, reportOnly: true });
|
||||
expect(report.mode).toBe('report');
|
||||
expect(report.categories.covers.eligible.objects).toBe(1);
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
const state = await reclaim.state();
|
||||
expect(state.lastRun).toEqual(report);
|
||||
expect(state.lastDeleteRun).toEqual(deleting);
|
||||
expect(await countRows(orm, AuditLog, { action: 'storage.reclaim_deleted' })).toBe(0);
|
||||
});
|
||||
|
||||
it('RCLS-004: an object named again loses its mark, and its week starts over once it is unnamed', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
namedIn('covers').add('orphan.jpg');
|
||||
const renamed = await reclaim.run({ now: new Date(T0.getTime() + DAY) });
|
||||
expect(renamed.categories.covers.unreferenced.objects).toBe(0);
|
||||
@@ -230,13 +273,10 @@ describe('StorageReclaimService: mark, then sweep', () => {
|
||||
expect(await marksOf('covers')).toEqual([{ key: 'orphan.jpg', first_seen_at: dbNow(WEEK_LATER) }]);
|
||||
});
|
||||
|
||||
it('RCLS-005: the single-key re-check right before the delete keeps an object named in between', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
it('RCLS-005: the fresh re-check right before the delete keeps an object named in between', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
onAsk = (category, keys) => {
|
||||
const [only] = keys;
|
||||
if (category === 'covers' && keys.length === 1 && only !== undefined) namedIn('covers').add(only);
|
||||
if (category === 'covers' && keys !== null) namedIn('covers').add('orphan.jpg');
|
||||
};
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.eligible.objects).toBe(1);
|
||||
@@ -250,43 +290,113 @@ describe('StorageReclaimService: mark, then sweep', () => {
|
||||
await put('covers', 'zero.jpg');
|
||||
await put('files', 'assistant/attached.pdf');
|
||||
await put('journey', 'thumbs/abcdef.jpg');
|
||||
const now = T0;
|
||||
fs.utimesSync(
|
||||
objectPath('covers', 'young.jpg'),
|
||||
new Date(now.getTime() - 60_000),
|
||||
new Date(now.getTime() - 60_000),
|
||||
);
|
||||
fs.utimesSync(objectPath('covers', 'young.jpg'), new Date(T0.getTime() - 60_000), new Date(T0.getTime() - 60_000));
|
||||
fs.utimesSync(objectPath('covers', 'zero.jpg'), 0, 0);
|
||||
const report = await reclaim.run({ now });
|
||||
const listed = vi.spyOn(storage, 'list');
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.covers).toMatchObject({ objects: 2, unreferenced: { objects: 0, bytes: 0 } });
|
||||
expect(report.categories.files.objects).toBe(0);
|
||||
expect(report.categories.journey.objects).toBe(0);
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
// Only a category's root is listed: the subtrees with sweeps of their own are never walked.
|
||||
expect(listed).toHaveBeenCalledWith('journey', '', { shallow: true });
|
||||
});
|
||||
|
||||
it('RCLS-007: the owner marker is written into each category judged, never judged itself, and never counted', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.run({ now: T0 });
|
||||
const marker = storageOwnerMarkerKey(installId);
|
||||
for (const category of STORAGE_RECLAIMABLE_CATEGORIES) {
|
||||
expect(fs.readFileSync(objectPath(category, marker), 'utf8')).toBe(installId);
|
||||
}
|
||||
fs.utimesSync(objectPath('covers', marker), new Date(0), new Date(0));
|
||||
const again = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(again.categories.covers.objects).toBe(5);
|
||||
expect((await marksOf('covers')).map((mark) => mark.key)).toEqual(['orphan.jpg']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('StorageReclaimService: brakes', () => {
|
||||
it('RCLS-007: more than a fifth unreferenced skips the category, deletes nothing and clears its marks', async () => {
|
||||
describe('StorageReclaimService: a large unreferenced share waits for the admin', () => {
|
||||
it('RCLS-008: more than a fifth unreferenced marks and reports, deletes nothing, and keeps the marks', async () => {
|
||||
await seedCovers(['o1.jpg', 'o2.jpg'], 1);
|
||||
await orm.repo(StorageReclaimMarks).insertMissing('covers', [{ key: 'o1.jpg', size: 5 }], dbNow(new Date(0)));
|
||||
await markWeekOld('covers', ['o1.jpg', 'o2.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers).toEqual({
|
||||
objects: 3,
|
||||
unreferenced: { objects: 2, bytes: 10 },
|
||||
eligible: { objects: 0, bytes: 0 },
|
||||
eligible: { objects: 2, bytes: 10 },
|
||||
deleted: { objects: 0, bytes: 0 },
|
||||
skipped: 'too_many_unreferenced',
|
||||
confirmationRequired: true,
|
||||
skipped: null,
|
||||
});
|
||||
expect(fs.existsSync(objectPath('covers', 'o1.jpg'))).toBe(true);
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect(await marksOf('covers')).toEqual([
|
||||
{ key: 'o1.jpg', first_seen_at: dbNow(new Date(0)) },
|
||||
{ key: 'o2.jpg', first_seen_at: dbNow(new Date(0)) },
|
||||
]);
|
||||
});
|
||||
|
||||
it('RCLS-008: a category whose required source is missing, or answers for a part only, is skipped', async () => {
|
||||
it('RCLS-009: an admin confirmation of the shown count lets the next pass delete; a larger share waits again', async () => {
|
||||
await seedCovers(['o1.jpg', 'o2.jpg'], 1);
|
||||
await markWeekOld('covers', ['o1.jpg', 'o2.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: WEEK_LATER });
|
||||
const confirmed = await reclaim.confirm('covers');
|
||||
expect(confirmed.lastRun?.categories.covers.confirmationRequired).toBe(true);
|
||||
expect(JSON.parse(String(await readAppSetting(orm, 'storage.reclaim_confirmed')))).toEqual({
|
||||
installId,
|
||||
counts: { covers: 2 },
|
||||
});
|
||||
|
||||
await put('covers', 'o3.jpg');
|
||||
await put('covers', 'o4.jpg');
|
||||
await put('covers', 'o5.jpg');
|
||||
await markWeekOld('covers', ['o3.jpg', 'o4.jpg', 'o5.jpg']);
|
||||
const bigger = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(bigger.categories.covers.confirmationRequired).toBe(true);
|
||||
expect(bigger.categories.covers.deleted.objects).toBe(0);
|
||||
|
||||
await reclaim.confirm('covers');
|
||||
const after = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(after.categories.covers).toMatchObject({ confirmationRequired: false, deleted: { objects: 5, bytes: 25 } });
|
||||
expect(fs.existsSync(objectPath('covers', 'kept-0.jpg'))).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLS-010: a confirmation is refused with a code when the last report holds nothing waiting', async () => {
|
||||
await expect(reclaim.confirm('covers')).rejects.toMatchObject({ code: 'reclaim_nothing_to_confirm' });
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.run({ now: T0 });
|
||||
const refusal = await reclaim.confirm('covers').catch((err: unknown) => err);
|
||||
expect(refusal).toBeInstanceOf(DomainError);
|
||||
expect((refusal as DomainError).getStatus()).toBe(409);
|
||||
expect((refusal as DomainError).getResponse()).toEqual({
|
||||
error: 'nothing in this category waits for a confirmation',
|
||||
code: 'reclaim_nothing_to_confirm',
|
||||
});
|
||||
});
|
||||
|
||||
it('RCLS-011: a confirmation another install wrote counts for nothing here', async () => {
|
||||
await seedCovers(['o1.jpg', 'o2.jpg'], 1);
|
||||
await markWeekOld('covers', ['o1.jpg', 'o2.jpg']);
|
||||
await setAppSetting(
|
||||
orm,
|
||||
'storage.reclaim_confirmed',
|
||||
JSON.stringify({ installId: '11111111-2222-4333-8444-555555555555', counts: { covers: 99 } }),
|
||||
);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.confirmationRequired).toBe(true);
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('StorageReclaimService: categories it may not judge', () => {
|
||||
it('RCLS-012: a category whose required source is missing, or answers for a part only, is skipped with its marks', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await markWeekOld('covers', ['orphan.jpg']);
|
||||
const all = [...references.sources()];
|
||||
const withoutAuth = all.filter((source) => source.id !== 'auth');
|
||||
vi.spyOn(references, 'sources').mockReturnValue(withoutAuth);
|
||||
vi.spyOn(references, 'sources').mockReturnValue(all.filter((source) => source.id !== 'auth'));
|
||||
let report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.avatars.skipped).toBe('sources_missing');
|
||||
expect(report.categories.covers.skipped).toBeNull();
|
||||
@@ -294,28 +404,35 @@ describe('StorageReclaimService: brakes', () => {
|
||||
const partialTrips: StorageReferenceSource = {
|
||||
id: 'trips',
|
||||
claims: [{ category: 'covers', prefix: 'sub/' }],
|
||||
namedKeys: async () => new Set<string>(),
|
||||
referenced: async () => new Set<string>(),
|
||||
};
|
||||
vi.spyOn(references, 'sources').mockReturnValue([...all.filter((source) => source.id !== 'trips'), partialTrips]);
|
||||
report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.covers.skipped).toBe('sources_missing');
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect((await marksOf('covers')).map((mark) => mark.key)).toEqual(['orphan.jpg']);
|
||||
});
|
||||
|
||||
it('RCLS-009: a source answering for a subtree only is not asked about root keys', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
it('RCLS-013: a source answering for a subtree only is not asked about root keys before a delete', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
const subtree = vi.fn(async () => new Set<string>());
|
||||
vi.spyOn(references, 'sources').mockReturnValue([
|
||||
...references.sources(),
|
||||
{ id: 'trips', claims: [{ category: 'covers', prefix: 'sub/' }], referenced: subtree },
|
||||
{
|
||||
id: 'trips',
|
||||
claims: [{ category: 'covers', prefix: 'sub/' }],
|
||||
namedKeys: async () => new Set(),
|
||||
referenced: subtree,
|
||||
},
|
||||
]);
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.covers.unreferenced.objects).toBe(1);
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(1);
|
||||
expect(subtree).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('RCLS-010: a category whose location holds another category is skipped', async () => {
|
||||
it('RCLS-014: a category whose location holds another category is skipped with its marks', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await markWeekOld('covers', ['orphan.jpg']);
|
||||
const nestedRoot = path.join(uploadsRoot, 'covers');
|
||||
await setAppSetting(
|
||||
orm,
|
||||
@@ -328,6 +445,7 @@ describe('StorageReclaimService: brakes', () => {
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.covers.skipped).toBe('shared_location');
|
||||
expect(report.categories.files.skipped).toBeNull();
|
||||
expect(await marksOf('covers')).toHaveLength(1);
|
||||
} finally {
|
||||
await setAppSetting(orm, 'storage.backends', backends());
|
||||
await deleteRows(orm, AppSettings, { key: 'storage.categories' });
|
||||
@@ -335,8 +453,31 @@ describe('StorageReclaimService: brakes', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('RCLS-011: a listing that fails skips the category and clears its marks', async () => {
|
||||
await orm.repo(StorageReclaimMarks).insertMissing('covers', [{ key: 'x.jpg', size: 1 }], dbNow(new Date(0)));
|
||||
it('RCLS-015: another install marker in a category keeps this one off it, with its marks', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
await put('covers', storageOwnerMarkerKey('99999999-8888-4777-8666-555555555555'), 'someone else');
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers).toMatchObject({ objects: 5, skipped: 'foreign_install' });
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
expect(report.categories.places.skipped).toBeNull();
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
expect(await marksOf('covers')).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('RCLS-016: an owner marker that cannot be written leaves the category unjudged as owner_unknown', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
const realPut = storage.put.bind(storage);
|
||||
vi.spyOn(storage, 'put').mockImplementation(async (category, name, source, opts) => {
|
||||
if (name.startsWith('trek-owner-')) throw new Error('read-only volume');
|
||||
return realPut(category, name, source, opts);
|
||||
});
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.categories.covers).toMatchObject({ objects: 5, skipped: 'owner_unknown' });
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
});
|
||||
|
||||
it('RCLS-017: a listing that fails skips the category for the night and keeps its marks', async () => {
|
||||
await markWeekOld('covers', ['x.jpg']);
|
||||
vi.spyOn(storage, 'list').mockImplementation(() => ({
|
||||
[Symbol.asyncIterator]: () => ({ next: () => Promise.reject(new Error('backend down')) }),
|
||||
}));
|
||||
@@ -344,11 +485,12 @@ describe('StorageReclaimService: brakes', () => {
|
||||
expect(report.categories.covers.skipped).toBe('list_failed');
|
||||
expect(report.categories.files.skipped).toBe('list_failed');
|
||||
expect(logError).toHaveBeenCalledWith(expect.stringContaining('backend down'));
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect(await marksOf('covers')).toEqual([{ key: 'x.jpg', first_seen_at: dbNow(new Date(0)) }]);
|
||||
});
|
||||
|
||||
it('RCLS-012: a source that fails skips the category, never reads it as unreferenced', async () => {
|
||||
it('RCLS-018: a source that fails skips the category, never reads it as unreferenced, and keeps its marks', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await markWeekOld('covers', ['orphan.jpg']);
|
||||
onAsk = (category) => {
|
||||
if (category === 'covers') throw new Error('trips table locked');
|
||||
};
|
||||
@@ -356,27 +498,32 @@ describe('StorageReclaimService: brakes', () => {
|
||||
expect(report.categories.covers).toMatchObject({ objects: 5, skipped: 'source_failed' });
|
||||
expect(report.categories.places.skipped).toBeNull();
|
||||
expect(logError).toHaveBeenCalledWith(expect.stringContaining('trips table locked'));
|
||||
expect(await marksOf('covers')).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
it('RCLS-013: a pass refuses on a demo instance and while another storage job holds the slot', async () => {
|
||||
describe('StorageReclaimService: when a pass runs at all', () => {
|
||||
it('RCLS-019: a pass refuses with a coded 409 on a demo instance and while another storage job holds the slot', async () => {
|
||||
const demo = vi.spyOn(t.get(RuntimeEnvService), 'isDemoMode').mockReturnValue(true);
|
||||
await expect(reclaim.run()).rejects.toMatchObject({ code: 'reclaim_demo' });
|
||||
expect(thrownBy(() => reclaim.start())).toMatchObject({ code: 'reclaim_demo' });
|
||||
demo.mockRestore();
|
||||
|
||||
const jobs = t.get(StorageJobsService);
|
||||
expect(jobs.claimExclusive('a test')).toBe(true);
|
||||
try {
|
||||
const refusal = await reclaim.run().catch((err: unknown) => err);
|
||||
expect(refusal).toBeInstanceOf(DomainError);
|
||||
expect(refusal).toMatchObject({ code: 'reclaim_busy' });
|
||||
expect((refusal as DomainError).getStatus()).toBe(409);
|
||||
expect((refusal as DomainError).getResponse()).toEqual({
|
||||
error: 'another storage job is running, one storage job at a time',
|
||||
code: 'reclaim_busy',
|
||||
});
|
||||
} finally {
|
||||
jobs.releaseExclusive('a test');
|
||||
}
|
||||
});
|
||||
|
||||
it('RCLS-014: while a pass holds the slot, a migration is refused with its name, and the slot is free after', async () => {
|
||||
const jobs = t.get(StorageJobsService);
|
||||
it('RCLS-020: while a pass holds the slot, a migration is refused with its name, and the slot is free after', async () => {
|
||||
let refusal: unknown = null;
|
||||
onAsk = async (category) => {
|
||||
if (category !== 'covers' || refusal) return;
|
||||
@@ -389,36 +536,41 @@ describe('StorageReclaimService: brakes', () => {
|
||||
jobs.releaseExclusive('after');
|
||||
});
|
||||
|
||||
it('RCLS-015: a database reopened mid-pass stops it before anything else is judged', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await put('places', 'p.jpg');
|
||||
let reopened = false;
|
||||
onAsk = async (category) => {
|
||||
if (category === 'covers' && !reopened) {
|
||||
reopened = true;
|
||||
await reopen();
|
||||
}
|
||||
};
|
||||
const report = await reclaim.run({ now: T0 });
|
||||
expect(report.stopped).toBe('database_reopened');
|
||||
expect(report.categories.covers.unreferenced.objects).toBe(0);
|
||||
expect(report.categories.places.objects).toBe(0);
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect((await reclaim.state()).lastRun?.stopped).toBe('database_reopened');
|
||||
it('RCLS-021: start() runs a report-only pass in the background; the state says running until it lands', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
let release: () => void = () => undefined;
|
||||
const gate = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
onAsk = async () => gate;
|
||||
reclaim.start();
|
||||
expect((await reclaim.state()).running).toBe(true);
|
||||
expect(thrownBy(() => reclaim.start())).toMatchObject({ code: 'reclaim_busy' });
|
||||
release();
|
||||
await jobs.exclusiveIdle();
|
||||
const state = await reclaim.state();
|
||||
expect(state.running).toBe(false);
|
||||
expect(state.lastRun?.mode).toBe('report');
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLS-022: a pass refuses to start while a restore runs, and starts again once it ended', async () => {
|
||||
await jobs.beginRestore();
|
||||
try {
|
||||
await expect(reclaim.run({ now: T0 })).rejects.toMatchObject({ code: 'reclaim_busy' });
|
||||
} finally {
|
||||
jobs.endRestore();
|
||||
}
|
||||
expect(jobs.restoreRunning).toBe(false);
|
||||
expect((await reclaim.run({ now: T0 })).stopped).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('StorageReclaimService: the delete itself', () => {
|
||||
async function markedWeekOld(orphans: string[], kept = 4): Promise<void> {
|
||||
await seedCovers(orphans, kept);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
}
|
||||
|
||||
it('RCLS-016: a re-check that fails keeps the object', async () => {
|
||||
it('RCLS-023: a re-check that fails keeps the object', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
onAsk = (_category, keys) => {
|
||||
if (keys.length === 1) throw new Error('lookup timed out');
|
||||
if (keys !== null) throw new Error('lookup timed out');
|
||||
};
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
@@ -426,12 +578,11 @@ describe('StorageReclaimService: the delete itself', () => {
|
||||
expect(logError).toHaveBeenCalledWith(expect.stringContaining('lookup timed out'));
|
||||
});
|
||||
|
||||
it('RCLS-017: an object gone or written again since it was judged is left and its mark dropped', async () => {
|
||||
it('RCLS-024: an object gone or written again since it was judged is left and its mark dropped', async () => {
|
||||
await markedWeekOld(['gone.jpg', 'fresh.jpg'], 8);
|
||||
// Changed between the listing and the delete: the batch looks at each object again first.
|
||||
const listed = storage.list.bind(storage);
|
||||
vi.spyOn(storage, 'list').mockImplementation(async function* (category, prefix) {
|
||||
yield* listed(category, prefix);
|
||||
vi.spyOn(storage, 'list').mockImplementation(async function* (category, prefix, opts) {
|
||||
yield* listed(category, prefix, opts);
|
||||
if (category !== 'covers') return;
|
||||
fs.rmSync(objectPath('covers', 'gone.jpg'), { force: true });
|
||||
fs.utimesSync(objectPath('covers', 'fresh.jpg'), WEEK_LATER, WEEK_LATER);
|
||||
@@ -443,65 +594,123 @@ describe('StorageReclaimService: the delete itself', () => {
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
});
|
||||
|
||||
it('RCLS-024: the re-check before the delete asks about the whole batch at once, not once per object', async () => {
|
||||
it('RCLS-025: the re-check before the delete asks about the whole batch at once, and the listing reads each source once', async () => {
|
||||
await markedWeekOld(['a.jpg', 'b.jpg', 'c.jpg'], 12);
|
||||
const asked: string[][] = [];
|
||||
const asked: Array<string[] | null> = [];
|
||||
onAsk = (category, keys) => {
|
||||
if (category === 'covers') asked.push([...keys].sort());
|
||||
if (category === 'covers') asked.push(keys === null ? null : [...keys].sort());
|
||||
};
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(3);
|
||||
// Per source: the listing's question, then one re-check for the three.
|
||||
const rechecks = asked.filter((keys) => keys.length === 3 && keys.every((key) => !key.startsWith('kept-')));
|
||||
expect(rechecks.length).toBe(STORAGE_REFERENCE_SOURCE_IDS.filter((id) => id !== 'auth' && id !== 'files').length);
|
||||
expect(asked.some((keys) => keys.length === 1)).toBe(false);
|
||||
const freeForm = STORAGE_REFERENCE_SOURCE_IDS.filter((id) => id !== 'auth' && id !== 'files').length;
|
||||
expect(asked.filter((keys) => keys === null)).toHaveLength(freeForm);
|
||||
expect(asked.filter((keys) => keys !== null)).toEqual(
|
||||
Array.from({ length: freeForm }, () => ['a.jpg', 'b.jpg', 'c.jpg']),
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLS-018: a delete that fails keeps the mark for the next pass', async () => {
|
||||
it('RCLS-026: a delete that fails stops deleting in that category, keeps the mark and says why once', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
vi.spyOn(storage, 'delete').mockRejectedValue(new Error('permission denied'));
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
expect(report.categories.covers).toMatchObject({ skipped: 'delete_failed', deleted: { objects: 0, bytes: 0 } });
|
||||
expect(report.categories.places.skipped).toBeNull();
|
||||
expect(await marksOf('covers')).toEqual([{ key: 'orphan.jpg', first_seen_at: dbNow(T0) }]);
|
||||
expect(logError).toHaveBeenCalledWith(expect.stringContaining('permission denied'));
|
||||
expect(vi.mocked(logError).mock.calls.filter(([line]) => String(line).includes('permission denied'))).toHaveLength(
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLS-019: an object the facade keeps on delete is not counted as reclaimed', async () => {
|
||||
it('RCLS-027: a stat that fails on the backend stops the category the same way', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
vi.spyOn(storage, 'stat').mockRejectedValue(new Error('503 SlowDown'));
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.skipped).toBe('delete_failed');
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLS-028: an object the facade keeps on delete is not counted as reclaimed', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
vi.spyOn(storage, 'keepsOnDelete').mockReturnValue(true);
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(0);
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLS-029: deletion switched off mid-pass stops every later delete', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
await put('places', 'place-orphan.jpg');
|
||||
namedIn('places').add('p1.jpg');
|
||||
await Promise.all(['p1.jpg', 'p2.jpg', 'p3.jpg', 'p4.jpg', 'p5.jpg'].map((key) => put('places', key)));
|
||||
['p2.jpg', 'p3.jpg', 'p4.jpg', 'p5.jpg'].forEach((key) => namedIn('places').add(key));
|
||||
await markWeekOld('places', ['place-orphan.jpg']);
|
||||
const realDelete = storage.delete.bind(storage);
|
||||
vi.spyOn(storage, 'delete').mockImplementation(async (category, name) => {
|
||||
await realDelete(category, name);
|
||||
if (category === 'covers') await reclaim.setDeleteEnabled(false);
|
||||
});
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.categories.covers.deleted.objects).toBe(1);
|
||||
expect(report.categories.places).toMatchObject({ eligible: { objects: 1 }, deleted: { objects: 0 } });
|
||||
expect(fs.existsSync(objectPath('places', 'place-orphan.jpg'))).toBe(true);
|
||||
});
|
||||
|
||||
it('RCLS-030: nothing is deleted while a backup reads the uploads', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
jobs.beginBackup();
|
||||
try {
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.mode).toBe('delete');
|
||||
expect(report.categories.covers).toMatchObject({ eligible: { objects: 1 }, deleted: { objects: 0 } });
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
} finally {
|
||||
jobs.endBackup();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('StorageReclaimService: the switch and the stored report', () => {
|
||||
it('RCLS-020: the switch is stored as true or false and only the exact true deletes', async () => {
|
||||
expect((await reclaim.setDeleteEnabled(true)).deleteEnabled).toBe(true);
|
||||
expect(await readAppSetting(orm, 'storage.reclaim_delete')).toBe('true');
|
||||
expect((await reclaim.setDeleteEnabled(false)).deleteEnabled).toBe(false);
|
||||
await setAppSetting(orm, 'storage.reclaim_delete', 'yes');
|
||||
expect((await reclaim.state()).deleteEnabled).toBe(false);
|
||||
describe('StorageReclaimService: a pass that stops, and a restore', () => {
|
||||
it('RCLS-031: a pass stopped by a reopen keeps what it deleted and marks every unfinished category interrupted', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
await put('places', 'p.jpg');
|
||||
const realDelete = storage.delete.bind(storage);
|
||||
vi.spyOn(storage, 'delete').mockImplementation(async (category, name) => {
|
||||
await realDelete(category, name);
|
||||
if (category === 'covers') await reopen();
|
||||
});
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.stopped).toBe('database_reopened');
|
||||
expect(report.categories.covers).toMatchObject({ skipped: null, deleted: { objects: 1, bytes: 5 } });
|
||||
for (const category of ['avatars', 'places'] as const) {
|
||||
expect(report.categories[category]).toMatchObject({ objects: 0, skipped: 'interrupted' });
|
||||
}
|
||||
expect((await reclaim.state()).lastRun?.stopped).toBe('database_reopened');
|
||||
expect(await countRows(orm, AuditLog, { action: 'storage.reclaim_deleted' })).toBe(1);
|
||||
});
|
||||
|
||||
it('RCLS-021: a stored report that no longer parses reads as never run', async () => {
|
||||
await setAppSetting(orm, 'storage.reclaim_report', '{not json');
|
||||
expect((await reclaim.state()).lastRun).toBeNull();
|
||||
await setAppSetting(orm, 'storage.reclaim_report', JSON.stringify({ startedAt: 'yesterday' }));
|
||||
expect((await reclaim.state()).lastRun).toBeNull();
|
||||
it('RCLS-032: an unexpected failure part way still persists the report with what was deleted', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
const spy = vi
|
||||
.spyOn(StorageReclaimMarksRepository.prototype, 'deleteKeys')
|
||||
.mockRejectedValueOnce(new Error('disk I/O error'));
|
||||
try {
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.stopped).toBe('failed');
|
||||
expect(report.categories.covers).toMatchObject({ skipped: 'interrupted', deleted: { objects: 1, bytes: 5 } });
|
||||
expect(report.categories.places.skipped).toBe('interrupted');
|
||||
expect((await reclaim.state()).lastDeleteRun?.stopped).toBe('failed');
|
||||
expect(logError).toHaveBeenCalledWith(expect.stringContaining('disk I/O error'));
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('RCLS-022: a restore that begins mid-pass stops it before the delete, and waits for it to let go', async () => {
|
||||
await seedCovers(['orphan.jpg']);
|
||||
await reclaim.setDeleteEnabled(true);
|
||||
await reclaim.run({ now: T0 });
|
||||
const jobs = t.get(StorageJobsService);
|
||||
it('RCLS-033: a restore that begins mid-pass stops it before the delete, and waits for it to let go', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
let restoreStarted: Promise<void> | null = null;
|
||||
let restoreAsked = false;
|
||||
let restoreWent = false;
|
||||
onAsk = (_category, keys) => {
|
||||
if (keys.length !== 1 || restoreAsked) return;
|
||||
restoreAsked = true;
|
||||
if (keys === null || restoreStarted) return;
|
||||
restoreStarted = jobs.beginRestore().then(() => {
|
||||
restoreWent = true;
|
||||
});
|
||||
@@ -509,8 +718,8 @@ describe('StorageReclaimService: the switch and the stored report', () => {
|
||||
try {
|
||||
const report = await reclaim.run({ now: WEEK_LATER });
|
||||
expect(report.stopped).toBe('database_reopened');
|
||||
expect(report.categories.covers.skipped).toBe('interrupted');
|
||||
expect(fs.existsSync(objectPath('covers', 'orphan.jpg'))).toBe(true);
|
||||
expect(restoreAsked).toBe(true);
|
||||
await restoreStarted;
|
||||
expect(restoreWent).toBe(true);
|
||||
} finally {
|
||||
@@ -518,15 +727,40 @@ describe('StorageReclaimService: the switch and the stored report', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('RCLS-023: a pass refuses to start while a restore runs, and starts again once it ended', async () => {
|
||||
const jobs = t.get(StorageJobsService);
|
||||
await jobs.beginRestore();
|
||||
try {
|
||||
await expect(reclaim.run({ now: T0 })).rejects.toMatchObject({ code: 'reclaim_busy' });
|
||||
} finally {
|
||||
jobs.endRestore();
|
||||
}
|
||||
expect(jobs.restoreRunning).toBe(false);
|
||||
expect((await reclaim.run({ now: T0 })).stopped).toBeNull();
|
||||
it('RCLS-034: a reopen turns deletion off and drops marks, confirmations and reports, but keeps the install id', async () => {
|
||||
await markedWeekOld(['orphan.jpg']);
|
||||
await setAppSetting(orm, 'storage.reclaim_confirmed', JSON.stringify({ installId, counts: { covers: 1 } }));
|
||||
// A restored database carries another install's id and switch.
|
||||
await setAppSetting(orm, 'install_id', '12345678-aaaa-4bbb-8ccc-1234567890ab');
|
||||
await setAppSetting(orm, 'storage.reclaim_delete', '12345678-aaaa-4bbb-8ccc-1234567890ab');
|
||||
await reopen();
|
||||
expect(await readAppSetting(orm, 'install_id')).toBe(installId);
|
||||
const reset = ['storage.reclaim_delete', 'storage.reclaim_confirmed', 'storage.reclaim_report'];
|
||||
expect(await Promise.all(reset.map((key) => readAppSetting(orm, key)))).toEqual([null, null, null]);
|
||||
expect(await marksOf('covers')).toEqual([]);
|
||||
expect(await reclaim.state()).toEqual({ deleteEnabled: false, running: false, lastRun: null, lastDeleteRun: null });
|
||||
});
|
||||
});
|
||||
|
||||
describe('StorageReclaimService: the switch and the stored reports', () => {
|
||||
it('RCLS-035: the switch is stored as this install id; any other value, another install id included, reads off', async () => {
|
||||
expect((await reclaim.setDeleteEnabled(true)).deleteEnabled).toBe(true);
|
||||
expect(await readAppSetting(orm, 'storage.reclaim_delete')).toBe(installId);
|
||||
expect((await reclaim.setDeleteEnabled(false)).deleteEnabled).toBe(false);
|
||||
expect(await readAppSetting(orm, 'storage.reclaim_delete')).toBeNull();
|
||||
const readsOff = async (value: string): Promise<boolean> => {
|
||||
await setAppSetting(orm, 'storage.reclaim_delete', value);
|
||||
return (await reclaim.state()).deleteEnabled;
|
||||
};
|
||||
expect(await readsOff('true')).toBe(false);
|
||||
expect(await readsOff('yes')).toBe(false);
|
||||
expect(await readsOff('12345678-aaaa-4bbb-8ccc-1234567890ab')).toBe(false);
|
||||
});
|
||||
|
||||
it('RCLS-036: a stored report that no longer parses reads as never run', async () => {
|
||||
await setAppSetting(orm, 'storage.reclaim_report', '{not json');
|
||||
expect((await reclaim.state()).lastRun).toBeNull();
|
||||
await setAppSetting(orm, 'storage.reclaim_delete_report', JSON.stringify({ startedAt: 'yesterday' }));
|
||||
expect((await reclaim.state()).lastDeleteRun).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -3,11 +3,15 @@
|
||||
* stored value names, in every spelling the columns hold. A key reported too
|
||||
* many only keeps an object; one missed would delete it, so the cases lean on
|
||||
* the spellings the validators let through (absolute self-URLs, queries,
|
||||
* relative journey covers, legacy file names). RCLP-001 to RCLP-012.
|
||||
* relative journey covers, legacy file names) and on object names that hold
|
||||
* the characters a URL treats as special. RCLP-001 to RCLP-016.
|
||||
*/
|
||||
import {
|
||||
referencedAmong,
|
||||
keySearchCore,
|
||||
keysNamedIn,
|
||||
storageKeysIn,
|
||||
storageOwnerMarkerInstall,
|
||||
storageOwnerMarkerKey,
|
||||
uploadKeysIn,
|
||||
type StorageKeyOptions,
|
||||
type StorageKeyRef,
|
||||
@@ -15,44 +19,42 @@ import {
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
|
||||
const ref = (category: string, key: string): StorageKeyRef => ({ category, key });
|
||||
/** The keys of one category among the refs, as a set: the order and the repeats do not matter to a source. */
|
||||
function keysIn(refs: StorageKeyRef[], category: string): Set<string> {
|
||||
return new Set(refs.filter((ref) => ref.category === category).map((ref) => ref.key));
|
||||
}
|
||||
|
||||
describe('uploadKeysIn', () => {
|
||||
it('RCLP-001: finds the path in its plain, slash-less and absolute self-URL spellings', () => {
|
||||
expect(uploadKeysIn('/uploads/covers/a.jpg')).toEqual([ref('covers', 'a.jpg')]);
|
||||
expect(uploadKeysIn('uploads/files/b.pdf')).toEqual([ref('files', 'b.pdf')]);
|
||||
expect(uploadKeysIn('https://trip.example.com/uploads/places/c.png')).toEqual([ref('places', 'c.png')]);
|
||||
expect(uploadKeysIn('HTTPS://HOST/Uploads/Covers/D.JPG')).toEqual([ref('covers', 'D.JPG')]);
|
||||
expect(keysIn(uploadKeysIn('/uploads/covers/a.jpg'), 'covers')).toEqual(new Set(['a.jpg']));
|
||||
expect(keysIn(uploadKeysIn('uploads/files/b.pdf'), 'files')).toEqual(new Set(['b.pdf']));
|
||||
expect(keysIn(uploadKeysIn('https://trip.example.com/uploads/places/c.png'), 'places')).toEqual(new Set(['c.png']));
|
||||
expect(keysIn(uploadKeysIn('HTTPS://HOST/Uploads/Covers/D.JPG'), 'covers')).toEqual(new Set(['D.JPG']));
|
||||
});
|
||||
|
||||
it('RCLP-002: drops a query or a fragment, decodes and trims the key', () => {
|
||||
expect(uploadKeysIn('/uploads/places/x.jpg?v=2')).toEqual([ref('places', 'x.jpg')]);
|
||||
expect(uploadKeysIn('/uploads/places/x.jpg#top')).toEqual([ref('places', 'x.jpg')]);
|
||||
expect(uploadKeysIn(' /uploads/covers/y.jpg ')).toEqual([ref('covers', 'y.jpg')]);
|
||||
expect(uploadKeysIn('/uploads/photos/my%20photo.jpg')).toEqual([
|
||||
ref('photos', 'my%20photo.jpg'),
|
||||
ref('photos', 'my photo.jpg'),
|
||||
ref('photos', 'my'),
|
||||
]);
|
||||
it('RCLP-002: reports the key as it stands and cut at a query or fragment, decoded and trimmed', () => {
|
||||
expect(keysIn(uploadKeysIn('/uploads/places/x.jpg?v=2'), 'places')).toEqual(new Set(['x.jpg?v=2', 'x.jpg']));
|
||||
expect(keysIn(uploadKeysIn('/uploads/places/x.jpg#top'), 'places')).toEqual(new Set(['x.jpg#top', 'x.jpg']));
|
||||
expect([...keysIn(uploadKeysIn(' /uploads/covers/y.jpg '), 'covers')]).toContain('y.jpg');
|
||||
expect(keysIn(uploadKeysIn('/uploads/photos/my%20photo.jpg'), 'photos')).toEqual(
|
||||
new Set(['my%20photo.jpg', 'my photo.jpg', 'my']),
|
||||
);
|
||||
});
|
||||
|
||||
it('RCLP-003: finds every path in a text, each cut where the next begins or the text ends', () => {
|
||||
expect(uploadKeysIn('see /uploads/covers/a.jpg and "/uploads/places/b.jpg" too')).toEqual([
|
||||
ref('covers', 'a.jpg and'),
|
||||
ref('covers', 'a.jpg'),
|
||||
ref('places', 'b.jpg'),
|
||||
]);
|
||||
expect(uploadKeysIn('/uploads/covers/a.jpg/uploads/places/b.jpg')).toEqual([
|
||||
ref('covers', 'a.jpg'),
|
||||
ref('places', 'b.jpg'),
|
||||
]);
|
||||
const refs = uploadKeysIn('see /uploads/covers/a.jpg and "/uploads/places/b.jpg" too');
|
||||
expect(keysIn(refs, 'covers')).toEqual(new Set(['a.jpg and ', 'a.jpg and', 'a.jpg']));
|
||||
expect(keysIn(refs, 'places')).toEqual(new Set(['b.jpg']));
|
||||
const joined = uploadKeysIn('/uploads/covers/a.jpg/uploads/places/b.jpg');
|
||||
expect(keysIn(joined, 'covers')).toEqual(new Set(['a.jpg']));
|
||||
expect(keysIn(joined, 'places')).toEqual(new Set(['b.jpg']));
|
||||
});
|
||||
|
||||
it('RCLP-004: keeps a nested key whole and refuses empty or climbing keys', () => {
|
||||
expect(uploadKeysIn('/uploads/journey/thumbs/abc.jpg')).toEqual([ref('journey', 'thumbs/abc.jpg')]);
|
||||
it('RCLP-004: keeps a nested key whole and reports nothing for an empty one', () => {
|
||||
expect(keysIn(uploadKeysIn('/uploads/journey/thumbs/abc.jpg'), 'journey')).toEqual(new Set(['thumbs/abc.jpg']));
|
||||
expect(uploadKeysIn('/uploads/covers/')).toEqual([]);
|
||||
expect(uploadKeysIn('/uploads/covers/../avatars/a.jpg')).toEqual([]);
|
||||
expect(uploadKeysIn('/uploads/covers/%2E%2E/x')).toEqual([ref('covers', '%2E%2E/x')]);
|
||||
// A climbing key names no object a listing yields; reporting it only keeps.
|
||||
expect([...keysIn(uploadKeysIn('/uploads/covers/%2E%2E/x'), 'covers')]).toContain('%2E%2E/x');
|
||||
});
|
||||
|
||||
it('RCLP-005: names nothing for empty values, other hosts and other paths', () => {
|
||||
@@ -65,61 +67,87 @@ describe('uploadKeysIn', () => {
|
||||
});
|
||||
|
||||
it('RCLP-006: a broken percent sequence keeps the raw key', () => {
|
||||
expect(uploadKeysIn('/uploads/covers/100%.jpg')).toEqual([ref('covers', '100%.jpg')]);
|
||||
expect(keysIn(uploadKeysIn('/uploads/covers/100%.jpg'), 'covers')).toEqual(new Set(['100%.jpg']));
|
||||
});
|
||||
});
|
||||
|
||||
describe('storageKeysIn', () => {
|
||||
it('RCLP-007: reads the relative <category>/<key> spelling of journey covers', () => {
|
||||
expect(storageKeysIn('covers/a.jpg')).toEqual([ref('covers', 'a.jpg')]);
|
||||
expect(storageKeysIn('journey/b.jpg?x=1')).toEqual([ref('journey', 'b.jpg')]);
|
||||
expect(storageKeysIn('./journey/thumbs/c.jpg')).toEqual([ref('journey', 'thumbs/c.jpg')]);
|
||||
expect(keysIn(storageKeysIn('covers/a.jpg'), 'covers')).toEqual(new Set(['a.jpg']));
|
||||
expect(keysIn(storageKeysIn('journey/b.jpg?x=1'), 'journey')).toEqual(new Set(['b.jpg?x=1', 'b.jpg']));
|
||||
expect(keysIn(storageKeysIn('./journey/thumbs/c.jpg'), 'journey')).toEqual(new Set(['thumbs/c.jpg']));
|
||||
expect(storageKeysIn('notacategory/a.jpg')).toEqual([]);
|
||||
});
|
||||
|
||||
it('RCLP-008: a bare name belongs to the column category, a URL or a path does not', () => {
|
||||
expect(storageKeysIn('a1b2.jpg', { bare: 'avatars' })).toEqual([ref('avatars', 'a1b2.jpg')]);
|
||||
expect(keysIn(storageKeysIn('a1b2.jpg', { bare: 'avatars' }), 'avatars')).toEqual(new Set(['a1b2.jpg']));
|
||||
expect(storageKeysIn('https://idp.example.com/pic.jpg', { bare: 'avatars' })).toEqual([]);
|
||||
expect(storageKeysIn('some/dir/x.pdf', { bare: 'files' })).toEqual([]);
|
||||
expect(keysIn(storageKeysIn('some/dir/x.pdf', { bare: 'files' }), 'files')).toEqual(new Set());
|
||||
expect(storageKeysIn('a1b2.jpg')).toEqual([]);
|
||||
});
|
||||
|
||||
it('RCLP-009: with basename, a legacy path still names its last segment', () => {
|
||||
expect(storageKeysIn('files/x.pdf', { bare: 'files', basename: true })).toEqual([
|
||||
ref('files', 'x.pdf'),
|
||||
ref('files', 'x.pdf'),
|
||||
]);
|
||||
expect(storageKeysIn('/uploads/files/y.pdf', { bare: 'files', basename: true })).toEqual([
|
||||
ref('files', 'y.pdf'),
|
||||
ref('files', 'y.pdf'),
|
||||
]);
|
||||
expect(storageKeysIn('C:\\old\\uploads\\z.pdf', { bare: 'files', basename: true })).toEqual([
|
||||
ref('files', 'z.pdf'),
|
||||
]);
|
||||
const options: StorageKeyOptions = { bare: 'files', basename: true };
|
||||
expect(keysIn(storageKeysIn('files/x.pdf', options), 'files')).toEqual(new Set(['x.pdf']));
|
||||
expect(keysIn(storageKeysIn('/uploads/files/y.pdf', options), 'files')).toEqual(new Set(['y.pdf']));
|
||||
expect(keysIn(storageKeysIn('C:\\old\\uploads\\z.pdf', options), 'files')).toEqual(new Set(['z.pdf']));
|
||||
});
|
||||
|
||||
it('RCLP-010: names nothing for a blank or a non-string', () => {
|
||||
it('RCLP-010: names nothing for a blank or a non-string, and only the raw text for a bare query', () => {
|
||||
expect(storageKeysIn(' ', { bare: 'files' })).toEqual([]);
|
||||
expect(storageKeysIn('?only=query', { bare: 'files' })).toEqual([]);
|
||||
expect(keysIn(storageKeysIn('?only=query', { bare: 'files' }), 'files')).toEqual(new Set(['?only=query']));
|
||||
expect(storageKeysIn(null)).toEqual([]);
|
||||
expect(storageKeysIn(42 as unknown as string)).toEqual([]);
|
||||
});
|
||||
|
||||
it('RCLP-013: a bare name holding a #, a ? or a trailing blank is reported untouched first', () => {
|
||||
// An upload's key is <uuid><extname>, and extname takes whatever follows the last dot.
|
||||
const options: StorageKeyOptions = { bare: 'files', basename: true };
|
||||
expect([...keysIn(storageKeysIn('0f8e2b7c. #4711', options), 'files')]).toContain('0f8e2b7c. #4711');
|
||||
expect([...keysIn(storageKeysIn('0f8e2b7c.c#', options), 'files')]).toContain('0f8e2b7c.c#');
|
||||
expect([...keysIn(storageKeysIn('0f8e2b7c.v2?final', options), 'files')]).toContain('0f8e2b7c.v2?final');
|
||||
expect([...keysIn(storageKeysIn('0f8e2b7c.pdf ', options), 'files')]).toContain('0f8e2b7c.pdf ');
|
||||
expect([...keysIn(storageKeysIn('legacy/0f8e2b7c. #4711', options), 'files')]).toContain('0f8e2b7c. #4711');
|
||||
// The URL spellings keep the raw tail too, so a path naming such an object keeps it.
|
||||
expect([...keysIn(storageKeysIn('/uploads/files/0f8e2b7c.c#'), 'files')]).toContain('0f8e2b7c.c#');
|
||||
});
|
||||
});
|
||||
|
||||
describe('referencedAmong', () => {
|
||||
it('RCLP-011: answers the candidates of one category the values name, without regard to case', () => {
|
||||
describe('keysNamedIn', () => {
|
||||
it('RCLP-011: answers the keys of one category the values name, lowercased', () => {
|
||||
const values = ['/uploads/covers/A.JPG', 'covers/b.jpg', '/uploads/places/c.jpg', null, 'https://elsewhere/x'];
|
||||
expect(referencedAmong('covers', ['a.jpg', 'b.jpg', 'c.jpg', 'd.jpg'], values)).toEqual(
|
||||
new Set(['a.jpg', 'b.jpg']),
|
||||
);
|
||||
expect(referencedAmong('places', ['c.jpg'], values)).toEqual(new Set(['c.jpg']));
|
||||
expect(keysNamedIn('covers', values)).toEqual(new Set(['a.jpg', 'b.jpg']));
|
||||
expect(keysNamedIn('places', values)).toEqual(new Set(['c.jpg']));
|
||||
});
|
||||
|
||||
it('RCLP-012: passes the column options on, so a bare avatar name keeps its object', () => {
|
||||
const avatarColumn: StorageKeyOptions = { bare: 'avatars' };
|
||||
expect(referencedAmong('avatars', ['u1.png'], ['u1.png'], avatarColumn)).toEqual(new Set(['u1.png']));
|
||||
expect(
|
||||
referencedAmong('avatars', ['u1.png', 'u2.png'], ['u1.png', 'https://idp/u2.png'], { bare: 'avatars' }),
|
||||
).toEqual(new Set(['u1.png']));
|
||||
expect(keysNamedIn('avatars', ['U1.png', 'https://idp/u2.png'], avatarColumn)).toEqual(new Set(['u1.png']));
|
||||
});
|
||||
});
|
||||
|
||||
describe('keySearchCore', () => {
|
||||
it('RCLP-014: the longest plain run a row naming the key in any spelling holds verbatim', () => {
|
||||
expect(keySearchCore('0F8E2B7C-1D2A-4C3B-9E8F-7A6B5C4D3E2F.jpg')).toBe('0f8e2b7c-1d2a-4c3b-9e8f-7a6b5c4d3e2f');
|
||||
expect(keySearchCore('0f8e2b7c-1d2a. #4711')).toBe('0f8e2b7c-1d2a');
|
||||
expect(keySearchCore('my photo.jpg')).toBeNull();
|
||||
expect(keySearchCore('a.jpg')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('owner markers', () => {
|
||||
it('RCLP-015: the marker key names the install, and only a marker key reads as one', () => {
|
||||
const id = '0f8e2b7c-1d2a-4c3b-9e8f-7a6b5c4d3e2f';
|
||||
expect(storageOwnerMarkerKey(id)).toBe(`trek-owner-${id}`);
|
||||
expect(storageOwnerMarkerInstall(storageOwnerMarkerKey(id))).toBe(id);
|
||||
expect(storageOwnerMarkerInstall('0f8e2b7c.jpg')).toBeNull();
|
||||
expect(storageOwnerMarkerInstall('trek-owner-')).toBeNull();
|
||||
expect(storageOwnerMarkerInstall(`sub/trek-owner-${id}`)).toBeNull();
|
||||
});
|
||||
|
||||
it('RCLP-016: a marker key is a valid root-level key no upload is ever named', () => {
|
||||
const key = storageOwnerMarkerKey('0f8e2b7c-1d2a-4c3b-9e8f-7a6b5c4d3e2f');
|
||||
expect(key.includes('/')).toBe(false);
|
||||
expect(key.startsWith('.')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
* the tick error-containment mirrors VersionCheckJob/DemoResetJob there.
|
||||
*/
|
||||
import type { CronRegistrarService } from '../../../../src/nest/scheduling/cron-registrar.service';
|
||||
import type { StorageJobsService } from '../../../../src/nest/storage/storage-jobs.service';
|
||||
import type { StorageStatsService } from '../../../../src/nest/storage/storage-stats.service';
|
||||
import { StorageUsageScanJob } from '../../../../src/nest/storage/storage-usage-scan.job';
|
||||
import { Logger } from '@nestjs/common';
|
||||
@@ -30,19 +31,24 @@ function statsStub(): StorageStatsService {
|
||||
return { scan: vi.fn().mockResolvedValue(undefined) } as unknown as StorageStatsService;
|
||||
}
|
||||
|
||||
/** No exclusive pass holds the slot: the scan goes at once. */
|
||||
function jobsStub(idle: Promise<void> = Promise.resolve()): StorageJobsService {
|
||||
return { exclusiveIdle: vi.fn(() => idle) } as unknown as StorageJobsService;
|
||||
}
|
||||
|
||||
beforeEach(() => vi.restoreAllMocks());
|
||||
|
||||
describe('StorageUsageScanJob', () => {
|
||||
it('SCAN-001 registers the nightly 04:15 cron when the registrar gate is open', async () => {
|
||||
const registrar = registrarStub();
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, statsStub());
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, statsStub(), jobsStub());
|
||||
await job.onApplicationBootstrap();
|
||||
expect(registrar.register).toHaveBeenCalledWith('storage-usage-scan', '15 4 * * *', expect.any(Function));
|
||||
});
|
||||
|
||||
it('SCAN-002 does not register under the test gate (registrar.isEnabled() false)', async () => {
|
||||
const registrar = registrarStub(false);
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, statsStub());
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, statsStub(), jobsStub());
|
||||
await job.onApplicationBootstrap();
|
||||
expect(registrar.register).not.toHaveBeenCalled();
|
||||
});
|
||||
@@ -50,7 +56,7 @@ describe('StorageUsageScanJob', () => {
|
||||
it('SCAN-003 the registered callback drives tick(), which calls stats.scan()', async () => {
|
||||
const registrar = registrarStub();
|
||||
const stats = statsStub();
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, stats);
|
||||
const job = new StorageUsageScanJob(registrar as unknown as CronRegistrarService, stats, jobsStub());
|
||||
await job.onApplicationBootstrap();
|
||||
const onTick = registrar.register.mock.calls[0]![2] as () => void;
|
||||
onTick();
|
||||
@@ -60,7 +66,7 @@ describe('StorageUsageScanJob', () => {
|
||||
it('SCAN-004 tick() resolves and contains an Error rejection to the usage scan log line', async () => {
|
||||
const errorSpy = vi.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
|
||||
const stats = { scan: vi.fn().mockRejectedValue(new Error('disk full')) } as unknown as StorageStatsService;
|
||||
const job = new StorageUsageScanJob(registrarStub() as unknown as CronRegistrarService, stats);
|
||||
const job = new StorageUsageScanJob(registrarStub() as unknown as CronRegistrarService, stats, jobsStub());
|
||||
await expect(job.tick()).resolves.toBeUndefined();
|
||||
expect(errorSpy).toHaveBeenCalledWith('usage scan failed: disk full');
|
||||
});
|
||||
@@ -68,8 +74,23 @@ describe('StorageUsageScanJob', () => {
|
||||
it('SCAN-005 tick() stringifies a non-Error rejection rather than crashing', async () => {
|
||||
const errorSpy = vi.spyOn(Logger.prototype, 'error').mockImplementation(() => undefined);
|
||||
const stats = { scan: vi.fn().mockRejectedValue('backend unreachable') } as unknown as StorageStatsService;
|
||||
const job = new StorageUsageScanJob(registrarStub() as unknown as CronRegistrarService, stats);
|
||||
const job = new StorageUsageScanJob(registrarStub() as unknown as CronRegistrarService, stats, jobsStub());
|
||||
await expect(job.tick()).resolves.toBeUndefined();
|
||||
expect(errorSpy).toHaveBeenCalledWith('usage scan failed: backend unreachable');
|
||||
});
|
||||
|
||||
it('SCAN-006 tick() waits for a reclaim pass that still holds the slot before it scans', async () => {
|
||||
let release: () => void = () => undefined;
|
||||
const idle = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const stats = statsStub();
|
||||
const job = new StorageUsageScanJob(registrarStub() as unknown as CronRegistrarService, stats, jobsStub(idle));
|
||||
const ticked = job.tick();
|
||||
await Promise.resolve();
|
||||
expect(stats.scan).not.toHaveBeenCalled();
|
||||
release();
|
||||
await ticked;
|
||||
expect(stats.scan).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,8 +6,10 @@ import {
|
||||
storageAdminStateSchema,
|
||||
storageBackfillStatusSchema,
|
||||
storageConfigSchema,
|
||||
STORAGE_RECLAIM_SKIP_REASONS,
|
||||
storageReclaimConfirmRequestSchema,
|
||||
storageReclaimReportSchema,
|
||||
storageReclaimRunRequestSchema,
|
||||
storageReclaimRunStartedSchema,
|
||||
storageReclaimSettingsRequestSchema,
|
||||
storageReclaimStateSchema,
|
||||
storageSecretFields,
|
||||
@@ -191,7 +193,14 @@ describe('backfill + usage wire shapes', () => {
|
||||
|
||||
describe('the reclaim contract', () => {
|
||||
const counts = { objects: 0, bytes: 0 };
|
||||
const perCategory = { objects: 3, unreferenced: counts, eligible: counts, deleted: counts, skipped: null };
|
||||
const perCategory = {
|
||||
objects: 3,
|
||||
unreferenced: counts,
|
||||
eligible: counts,
|
||||
deleted: counts,
|
||||
confirmationRequired: false,
|
||||
skipped: null,
|
||||
};
|
||||
const report = {
|
||||
startedAt: 1_700_000_000_000,
|
||||
finishedAt: 1_700_000_001_000,
|
||||
@@ -205,25 +214,60 @@ describe('the reclaim contract', () => {
|
||||
expect(storageReclaimReportSchema.parse(report)).toEqual(report);
|
||||
});
|
||||
|
||||
it('a report missing a category, or naming an unknown skip reason, is refused', () => {
|
||||
it('a report missing a category, a confirmation flag, or naming an unknown skip reason, is refused', () => {
|
||||
const { photos: _photos, ...withoutPhotos } = report.categories;
|
||||
expect(storageReclaimReportSchema.safeParse({ ...report, categories: withoutPhotos }).success).toBe(false);
|
||||
const odd = { ...report, categories: { ...report.categories, covers: { ...perCategory, skipped: 'tired' } } };
|
||||
expect(storageReclaimReportSchema.safeParse(odd).success).toBe(false);
|
||||
const { confirmationRequired: _flag, ...unflagged } = perCategory;
|
||||
const old = { ...report, categories: { ...report.categories, covers: unflagged } };
|
||||
expect(storageReclaimReportSchema.safeParse(old).success).toBe(false);
|
||||
});
|
||||
|
||||
it('the admin state carries the switch and a nullable last run', () => {
|
||||
it('names every skip reason a pass reports, the brake that used to skip a category gone', () => {
|
||||
expect([...STORAGE_RECLAIM_SKIP_REASONS]).toEqual([
|
||||
'not_reclaimed',
|
||||
'sources_missing',
|
||||
'shared_location',
|
||||
'foreign_install',
|
||||
'owner_unknown',
|
||||
'list_failed',
|
||||
'source_failed',
|
||||
'delete_failed',
|
||||
'interrupted',
|
||||
]);
|
||||
for (const skipped of STORAGE_RECLAIM_SKIP_REASONS) {
|
||||
const row = { ...report, categories: { ...report.categories, covers: { ...perCategory, skipped } } };
|
||||
expect(storageReclaimReportSchema.safeParse(row).success).toBe(true);
|
||||
}
|
||||
expect(storageReclaimReportSchema.safeParse({ ...report, stopped: 'failed' }).success).toBe(true);
|
||||
expect(storageReclaimReportSchema.safeParse({ ...report, stopped: 'tired' }).success).toBe(false);
|
||||
});
|
||||
|
||||
it('the admin state carries the switch, the running flag and both nullable reports', () => {
|
||||
const shape = storageAdminStateSchema.pick({ reclaim: true });
|
||||
expect(shape.parse({ reclaim: { deleteEnabled: false, lastRun: null } })).toEqual({
|
||||
reclaim: { deleteEnabled: false, lastRun: null },
|
||||
});
|
||||
expect(storageReclaimStateSchema.safeParse({ deleteEnabled: 'yes', lastRun: null }).success).toBe(false);
|
||||
const idle = { deleteEnabled: false, running: false, lastRun: null, lastDeleteRun: null };
|
||||
expect(shape.parse({ reclaim: idle })).toEqual({ reclaim: idle });
|
||||
expect(shape.parse({ reclaim: null })).toEqual({ reclaim: null });
|
||||
expect(storageReclaimStateSchema.parse({ ...idle, running: true, lastRun: report, lastDeleteRun: report })).toEqual(
|
||||
{
|
||||
...idle,
|
||||
running: true,
|
||||
lastRun: report,
|
||||
lastDeleteRun: report,
|
||||
},
|
||||
);
|
||||
expect(storageReclaimStateSchema.safeParse({ ...idle, deleteEnabled: 'yes' }).success).toBe(false);
|
||||
expect(storageReclaimStateSchema.safeParse({ deleteEnabled: false, lastRun: null }).success).toBe(false);
|
||||
});
|
||||
|
||||
it('the requests are closed objects', () => {
|
||||
it('the requests are closed objects, and a started check answers started: true', () => {
|
||||
expect(storageReclaimSettingsRequestSchema.parse({ deleteEnabled: true })).toEqual({ deleteEnabled: true });
|
||||
expect(storageReclaimSettingsRequestSchema.safeParse({ deleteEnabled: true, extra: 1 }).success).toBe(false);
|
||||
expect(storageReclaimRunRequestSchema.parse({})).toEqual({});
|
||||
expect(storageReclaimRunRequestSchema.safeParse({ reportOnly: 'no' }).success).toBe(false);
|
||||
expect(storageReclaimConfirmRequestSchema.parse({ category: 'covers' })).toEqual({ category: 'covers' });
|
||||
expect(storageReclaimConfirmRequestSchema.safeParse({ category: 'nowhere' }).success).toBe(false);
|
||||
expect(storageReclaimConfirmRequestSchema.safeParse({ category: 'covers', count: 3 }).success).toBe(false);
|
||||
expect(storageReclaimRunStartedSchema.parse({ started: true })).toEqual({ started: true });
|
||||
expect(storageReclaimRunStartedSchema.safeParse({ started: false }).success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -283,20 +283,27 @@ export const STORAGE_RECLAIM_CATEGORIES = [...STORAGE_CATEGORIES, 'photos'] as c
|
||||
export const storageReclaimCategorySchema = z.enum(STORAGE_RECLAIM_CATEGORIES);
|
||||
|
||||
/**
|
||||
* Why a category deleted nothing on a pass, whatever the switch says:
|
||||
* Why a category was not judged on a pass, or stopped short, whatever the switch says:
|
||||
* - `not_reclaimed`: the category is never reclaimed (a cache, the backups, the legacy photos folder);
|
||||
* - `sources_missing`: a domain whose rows can name its objects did not answer, so nothing can be proven unreferenced;
|
||||
* - `shared_location`: its objects sit where another category's do, or where they cannot be told apart;
|
||||
* - `too_many_unreferenced`: more than a fifth of its objects look unreferenced, which points at a shared bucket;
|
||||
* - `list_failed` / `source_failed`: the listing or a reference lookup failed, so the pass stopped there.
|
||||
* - `foreign_install`: another TREK installation keeps its uploads in the same place (its owner marker is there);
|
||||
* - `owner_unknown`: this installation's owner marker could not be written there, so a shared place cannot be ruled out;
|
||||
* - `list_failed` / `source_failed`: the listing or a reference lookup failed, so the pass stopped there;
|
||||
* - `delete_failed`: deleting failed on the backend, so the pass stopped deleting in this category;
|
||||
* - `interrupted`: the pass stopped (`stopped` says why) before it finished this category; its figures are partial.
|
||||
* A skip keeps the marks it found: only a pass that judges the category moves them.
|
||||
*/
|
||||
export const STORAGE_RECLAIM_SKIP_REASONS = [
|
||||
'not_reclaimed',
|
||||
'sources_missing',
|
||||
'shared_location',
|
||||
'too_many_unreferenced',
|
||||
'foreign_install',
|
||||
'owner_unknown',
|
||||
'list_failed',
|
||||
'source_failed',
|
||||
'delete_failed',
|
||||
'interrupted',
|
||||
] as const;
|
||||
|
||||
const storageObjectCountSchema = z.object({ objects: z.number().int().nonnegative(), bytes: z.number().nonnegative() });
|
||||
@@ -309,6 +316,11 @@ export const storageReclaimCategoryReportSchema = z.object({
|
||||
/** Unreferenced on every pass for at least a week: what a pass with deletion on removes. */
|
||||
eligible: storageObjectCountSchema,
|
||||
deleted: storageObjectCountSchema,
|
||||
/**
|
||||
* More than a fifth of the category is unreferenced, and more than an admin
|
||||
* last confirmed: deletion waits for a confirmation (POST .../reclaim/confirm).
|
||||
*/
|
||||
confirmationRequired: z.boolean(),
|
||||
skipped: z.enum(STORAGE_RECLAIM_SKIP_REASONS).nullable(),
|
||||
});
|
||||
export type StorageReclaimCategoryReport = z.infer<typeof storageReclaimCategoryReportSchema>;
|
||||
@@ -316,27 +328,41 @@ export type StorageReclaimCategoryReport = z.infer<typeof storageReclaimCategory
|
||||
export const storageReclaimReportSchema = z.object({
|
||||
startedAt: z.number(),
|
||||
finishedAt: z.number(),
|
||||
/** `delete` only when the admin switched deletion on and the pass was not asked to only report. */
|
||||
/** `delete` when the pass started with deletion switched on; a check started by hand always only reports. */
|
||||
mode: z.enum(['report', 'delete']),
|
||||
/** Why the pass stopped before the end: the database was swapped under it (a restore). Null when it ran through. */
|
||||
stopped: z.enum(['database_reopened']).nullable(),
|
||||
/**
|
||||
* Why the pass stopped before the end: the database was swapped under it (a
|
||||
* restore), or an unexpected error. Null when it ran through.
|
||||
*/
|
||||
stopped: z.enum(['database_reopened', 'failed']).nullable(),
|
||||
categories: z.record(storageReclaimCategorySchema, storageReclaimCategoryReportSchema),
|
||||
});
|
||||
export type StorageReclaimReport = z.infer<typeof storageReclaimReportSchema>;
|
||||
|
||||
export const storageReclaimStateSchema = z.object({
|
||||
/** Off after install and upgrade: a pass then only reports. */
|
||||
/** Off after install, upgrade and every backup restore: a pass then only reports. */
|
||||
deleteEnabled: z.boolean(),
|
||||
/** A pass is running now (the nightly one or a check started by hand). */
|
||||
running: z.boolean(),
|
||||
/** The last pass of any kind; null before the first. */
|
||||
lastRun: storageReclaimReportSchema.nullable(),
|
||||
/** The last pass that ran with deletion on, which a check started by hand never replaces; null before one. */
|
||||
lastDeleteRun: storageReclaimReportSchema.nullable(),
|
||||
});
|
||||
export type StorageReclaimState = z.infer<typeof storageReclaimStateSchema>;
|
||||
|
||||
export const storageReclaimSettingsRequestSchema = z.strictObject({ deleteEnabled: z.boolean() });
|
||||
export type StorageReclaimSettingsRequest = z.infer<typeof storageReclaimSettingsRequestSchema>;
|
||||
|
||||
/** `reportOnly` runs a pass that deletes nothing even with deletion switched on. */
|
||||
export const storageReclaimRunRequestSchema = z.strictObject({ reportOnly: z.boolean().optional() });
|
||||
export type StorageReclaimRunRequest = z.infer<typeof storageReclaimRunRequestSchema>;
|
||||
/** POST .../reclaim/run answers 202 at once; the pass reports through the admin state. */
|
||||
export const storageReclaimRunStartedSchema = z.object({ started: z.literal(true) });
|
||||
|
||||
/**
|
||||
* Lets the pass delete in a category whose unreferenced share is over a
|
||||
* fifth, up to the count the last report showed for it.
|
||||
*/
|
||||
export const storageReclaimConfirmRequestSchema = z.strictObject({ category: storageReclaimCategorySchema });
|
||||
export type StorageReclaimConfirmRequest = z.infer<typeof storageReclaimConfirmRequestSchema>;
|
||||
|
||||
export const storageAdminStateSchema = z.object({
|
||||
backends: z.array(
|
||||
|
||||
+58
-21
@@ -127,33 +127,70 @@ includes the legacy photo library.
|
||||
|
||||
## Unreferenced files
|
||||
|
||||
Uploaded files can outlive what used them: deleting a trip, a user or a place
|
||||
image, or replacing a cover, leaves the old file in storage. Every night
|
||||
(04:05 server time) TREK checks the covers, place images, avatars, trip files
|
||||
and journal photos for files that no row names any more, and the
|
||||
**Unreferenced files** section shows what it found:
|
||||
Uploaded files can outlive what used them. When you delete or replace
|
||||
something that shows a file (a trip or list cover, a place image, a journal
|
||||
photo, an avatar, a trip file), TREK deletes that file right away, unless
|
||||
something else still shows it: a copied trip, a journey that took over the
|
||||
trip cover, a saved place in a list. Whatever is left over after that (files of
|
||||
deleted trips and users, files from before this check existed) is found by a
|
||||
check every night at 04:05 server time. It looks at the covers, place images,
|
||||
avatars, trip files and journal photos for files that nothing in TREK names
|
||||
any more, and the **Unreferenced files** section shows what it found:
|
||||
|
||||
- **Can be freed**: unused for at least a week, so the nightly run deletes them
|
||||
once deletion is on.
|
||||
- **Can be freed**: unused on every check for at least a week, so the nightly
|
||||
check deletes them once deletion is on.
|
||||
- **Unused for less than a week**: these get their week first. A file that is
|
||||
used again in the meantime (an undo, a copied trip) starts over.
|
||||
- **Freed on the last check**: what the nightly run deleted.
|
||||
- **Freed on the last check**: what the last nightly check with deletion on
|
||||
deleted. A check you start by hand never deletes, so it never changes this
|
||||
figure.
|
||||
|
||||
**Delete unreferenced files** is off after install and upgrade, so the check
|
||||
only reports. Switched on, the nightly run deletes a file once it has been
|
||||
unused for a week and a last check right before the delete still finds
|
||||
nothing that uses it. **Check now** runs a check that only reports, whatever
|
||||
the switch says. Each run is recorded in the [Audit-Log](Audit-Log).
|
||||
only reports. Switched on, the nightly check deletes a file once it has been
|
||||
unused for a week and one more look right before the delete still finds
|
||||
nothing that uses it. **Check now** starts a check in the background that only
|
||||
reports, whatever the switch says; the section shows that it is running and
|
||||
updates when it is done. Switching deletion on or off, starting a check,
|
||||
confirming a category (below) and every nightly check that deleted files are
|
||||
recorded in the [Audit-Log](Audit-Log); the nightly entry names how many files
|
||||
went per category.
|
||||
|
||||
A category is skipped, with the reason shown next to it, when not every part
|
||||
of TREK could be asked about it, when its files share a location with another
|
||||
category, when a listing or a lookup fails, or when more than a fifth of its
|
||||
files look unused. The last one usually means two installations share one
|
||||
bucket: each would see the other's files as unused, so leave deletion off in
|
||||
that setup. Caches (place photos), backups, assistant attachments and journal
|
||||
thumbnails are cleaned up by their own routines and are never touched here.
|
||||
No check runs on a demo instance, and none while a sync, a move or a restore
|
||||
is running.
|
||||
When more than a fifth of a category's files look unused, the check marks it
|
||||
and deletes nothing there until you confirm it once with **Confirm** next to
|
||||
the category. The confirmation covers the number of files shown at that
|
||||
moment: if the number later grows past it, while still more than a fifth, the
|
||||
category waits for you again. A long-running instance with many old leftovers
|
||||
usually needs this once.
|
||||
|
||||
A category is skipped for that night, with the reason shown next to it, when:
|
||||
|
||||
- not every part of TREK could be asked about it;
|
||||
- its files share a location with another category's;
|
||||
- another TREK installation keeps its files in the same place (see below), or
|
||||
TREK could not leave its own mark there;
|
||||
- listing it or asking about it failed. What the earlier nights found stays,
|
||||
so one failure does not restart anyone's week;
|
||||
- deleting failed on the storage backend: the check then stops deleting in
|
||||
that category for the night.
|
||||
|
||||
**Several installations on one bucket or folder.** Each check leaves a small
|
||||
file named `trek-owner-<id>` at the top of every category it looks at, naming
|
||||
this installation. When it finds such a file of another installation there,
|
||||
it leaves that category alone, and so does the other installation once it
|
||||
sees this one's file. Backups leave these files out. If you moved an
|
||||
installation and a stale `trek-owner-<id>` of its old self remains, delete
|
||||
that one file to let the check work there again.
|
||||
|
||||
**Restoring a backup** switches deletion off and forgets what earlier checks
|
||||
found and what you confirmed, because the restored data may come from another
|
||||
installation. Switch deletion on again once you have looked at the next
|
||||
report.
|
||||
|
||||
Caches (place photos), backups, assistant attachments and journal thumbnails
|
||||
are cleaned up by their own routines and are never touched here. No check runs
|
||||
on a demo instance, none while a sync, a move or a restore is running, and a
|
||||
running check deletes nothing while a backup is being written or once you
|
||||
switch deletion off.
|
||||
|
||||
## Secrets and encryption
|
||||
|
||||
|
||||
Reference in New Issue
Block a user