The rule that a multi-statement write runs in uow.transactional was
written down but nothing held it. scripts/tx-writes.mjs reads every class
under src/ with the TypeScript parser and counts, per method, the writes
it issues outside a transactional callback: repository calls that build
an INSERT, UPDATE or DELETE, calls to methods that write (on the class
itself or on a constructor-injected one), and a transaction holding
writes as one. A write in a loop counts twice, of two branches only the
larger counts, local helpers count where they are called, and a helper
whose every call sits inside a transaction is not reported. --explain
lists the line of every call that counted.
Today's 102 reported methods sit in scripts/tx-writes-baseline.json and
may only shrink; most are syncs with network I/O between their writes,
plugin lifecycle steps and adapters calling two service writes. A new
method fails CI, and an entry above what its method holds fails until
--update lowers it, the same ratchet as lint:size and lint:boundaries.
Two JSDoc tags mark the deliberate cases with their reason in the tag:
@txStandalone for a write that stands on its own whatever the caller does
(the audit row, notification sends, the booking price mirror, the
assignment journey refresh, the quota retention purge), @txIndependent
for writes complete one by one on purpose (the OAuth token retention
sweep, the notification claim and its release). Runs in CI after
lint:boundaries; documented in server/CLAUDE.md, the root CLAUDE.md and
the wiki's dev setup page.
Days, todo items, packing items and bags, packing template categories and
items, collections, collection labels and road-trip vias took their next
position as MAX(...) + 1 in one statement and inserted in the next, with
any other request free to run in between: two adds at once read the same
MAX and landed on the same position (the road-trip test pinned exactly
that duplicate). The read now sits in the transaction that inserts, and
the transaction holds the connection, so the second add waits and takes
the next position. A collection label's limit and name check move into
the same transaction, so two labels of one name added at once end in one
label and a 409.
Each site gets a test that adds several rows at once and reads back one
position each.
A sweep of the services for write sequences that committed step by step,
each one leaving half a change behind when the next statement failed:
- accommodations: the linked bookings' times moved after the stay's own
transaction had committed
- airports: the boot backfill wrote a flight's two endpoints one at a
time, and a flight left with only the first was never picked up again;
each flight is its own transaction now
- auth: the admin settings form and a password reset (retire the old
link, issue the new one) each land whole
- oauth: a refresh revoked the old pair before issuing the new one, so a
failure in between logged the client out; rotating a client secret and
revoking its tokens go together too
- oidc: the identity link, the role from the claims and the avatar of a
returning user are one write
- addons, notification preferences: a settings save lands whole
- places: create with its tags, update, and the GPX/KML imports with
their track colours
- trips: the trip row, the shifted leave entries and the rebuilt day grid
- vacay: a first visit provisions the plan in one transaction with a
second look against a concurrent first visit; updatePlan fetches the
public holidays between two transactions instead of inside its writes
- memories: a photo and its trip link, and the deletes with the orphan
sweep behind them
AddonsService, AirportsService and OauthService take the UnitOfWork for
this. Network I/O stays outside every transaction. Comments that only
narrated the migration plan make room under the lint:size entries, which
drop with them.
Uninstall deleted the plugin row and then up to fourteen tables of rows
about it one by one. A failure halfway left grants, scheduled tasks or
per-user settings behind for a later plugin that reuses the id to
inherit. The rows now go in one transaction, and the data directory is
removed after the commit instead of before the rows that describe it.
A registry install records the discovered row, its provenance, the
pinned author key and the cleared update block together, so a failure
never leaves a registered plugin without its pinned key. Starting an
OAuth connect drops the stale state and stores the fresh one together,
and a disconnect drops tokens and state together; PluginOAuthService
takes the UnitOfWork for that. The scheduler's per-task claims stay
separate on purpose and say so for the transaction gate.
createJourney inserted the journey, then its owner row, then linked each
trip with its skeleton entries and set the cover, every step committed on
its own: a failure halfway left a journey nobody owned. It now runs as
one transaction and broadcasts the linked trips after the commit.
addTripToJourney links inside a transaction the same way.
The photo paths wrapped only the gallery insert and left the entry link
and the skeleton promotion outside; the photo row, the gallery row, the
link and the promotion are one write now. removeTripFromJourney,
updateEntry, updatePhoto, setPhotoProvider, the photo deletes and the
place hooks write in one transaction each.
createEntry and uploadGalleryPhotos read MAX(sort_order) inside the
transaction that inserts, so two entries or uploads at once take one
position each. reconcileTripSkeletons and the photo capture helpers are
tagged as deliberately split for the transaction gate that follows.
A booking's create and update ran their own transaction, and the linked
cost sync ran afterwards from the REST controller and the plugin RPC
handler, so a failing cost write left the booking edited and its expense
on the old category or price. MCP linked the cost in a second call of its
own the same way.
ReservationsService.createWithCost and updateWithCost now run the
booking write and the cost sync in one transaction and hand the cost
events back, so REST, MCP and the plugin RPC call the same method and
broadcast only after the commit, in the order they did before. The cost
sync keeps swallowing its own failures the way the REST route always
has, so a cost that cannot be written still leaves the booking; MCP now
behaves like REST there instead of failing the tool after the booking
was already saved.
The service sheds its inventory markers (RS1..RS54, plan history) to
stay under its lint:size entry.
The time budget for an exec script was read after every statement, the
last one included. A single statement without args, or a script whose
closing COMMIT crossed the budget, had already run and committed, yet the
call threw a time budget error. A plugin that retries on error then wrote
the same rows a second time.
The script now reads the clock before each statement after the first, so
it still stops before the next statement once the time is up and rolls
back a transaction it opened, but a script whose statements have all run
is reported as done. query() likewise reads the clock only once another
row has come back, so a result read to its end is returned.
The ctx.db doc in the SDK said a single statement was not timed, which
contradicted the query budget. It now names what is really untimed: an
exec with bound args and migrate. README and wiki describe the script
stopping before its next statement.
The time budget covered query() and tx() only. exec() without bound args
handed the plugin's script to Database#exec, which runs any number of
statements in one call with nothing checked between them, so a long
script held the host loop as long as it liked.
An exec script now runs one statement at a time. sql-script.ts cuts it
after every semicolon outside a string, a quoted identifier or a
comment; a piece SQLite reports as incomplete input (a trigger body cut
at its inner semicolon) is joined with the next until it prepares. The
clock is read after every statement, and past the budget the call
throws. As with Database#exec, the statements before that point stay
applied, and an error is SQLite's own; a transaction the script opened
itself is rolled back so the connection is not left inside it.
exec() with bound args is one statement and stays as it was. A single
statement cannot be bounded at all on the host thread, a write such as
INSERT ... SELECT over a cross join included, and the SDK README and the
wiki now say so next to the cross-join read. migrate() keeps running
unbudgeted on purpose: a migration stopped part of the way is rolled back
and fails the same way on every later start, leaving the plugin unable to
update its schema.
The ctx.db rows of the SDK README and the plugin wiki pages describe the
budget for scripts, and the lines touched there and in plugin-sdk's db
comment lose their em dashes.
The output contract cut only the top level of an entity result. A derived
key's value went out untouched, so the children that are whole rows still
carried every column: a day's notes_items are full day_notes entities and
a reservation's endpoints are SELECT e.* on reservation_endpoints. About
fifty methods were also left as readModel and passed through whole, many
of them backed by SELECT * reads (day_accommodations, day_notes, todos,
tags, packing bags, journal entries, collab, collections, vacay plans).
A new column on any of those tables reached every plugin by default,
which is what the contract was meant to stop and what the SDK README and
the wiki already claimed it did.
PLUGIN_ENTITY_NESTED now names, per entity, the derived keys that hold
rows of another entity, and the shaper cuts those rows down to their own
field list, recursively. dayNote and reservationEndpoint are the child
tables that were whole rows; day.assignments and place.tags are held to
the assignment and tag lists too, although their reads already name
their columns. The remaining derived children (travelers, participants,
ratings, budget members, payers and receipts, packing recipients,
attachments, voters, bag members) are built field by field, which the
contract comment now states as the rule for a derived key without a
nested entity.
The whole-row read models get entity contracts: accommodation, dayNote,
todo, tag, category, packingBag, fileLink, collabNote, collabPoll,
collabMessage, journey, journalEntry, bucketItem and collection, and
vacayPlan with its holiday calendars. collections.listMine,
collections.get and vacay.mine return envelopes around rows; an entity
with a null table describes those, so their rows are reached through the
same nesting. Every list holds exactly the keys each read returned
before, so no plugin loses a field. What stays readModel is built field
by field from named columns, and each such entry says what it returns:
setBagMembers, atlas.visited, collections.savePlace and copyToTrip, the
two vacay toggles and journal.addEntryPhoto.
The schema test now covers every child and new table, and calls each
newly shaped method on the real services, comparing nested rows key for
key so the only field allowed to disappear is a withheld one. A further
case proves the nested rows it compares are not empty. gen:plugin-facts
publishes PLUGIN_ENTITY_NESTED to the SDK, and the SDK README, the plugin
wiki pages and server/CLAUDE.md describe the nested rows and list the new
entities.
A plugin's own database runs on the host's event loop, and the only bounds
were a regex denylist, the 256 MB quota and a 100,000 row cap. A select
whose every row is expensive, or a long scan under the cap, could hold the
loop for as long as it liked.
query() and tx() now get a wall-clock budget of 2 s, read between the rows
a statement yields and between the statements of a batch. Past it the call
throws, and a tx rolls back whole like any other failure inside it. The
row cap and the budget are overridable per PluginDataDb for tests; the
defaults are what production runs.
better-sqlite3 has no interrupt or progress handler, so a statement that
yields nothing until it is done, such as an aggregate over a cross join,
still runs to its first row. Only moving plugin SQL off the host thread can
bound that; the SDK README and the plugin wiki state the budget and this
limit.
The entity reads of the plugin RPC surface handed plugins whatever their
query produced, mostly SELECT t.*, with credentials stripped by a denylist
(withoutFeedToken). Every column a migration added reached every plugin
holding the read grant, and renaming one was an unversioned break of a
published API.
protocol/output-contract.ts now states, for every wire method, what its
result is: rows of one of the nine entities the SDK types (trip, place,
day, reservation, packingItem, tripFile, budgetItem, assignment, user), a
value the host builds itself, or a read model passed on as the REST route
returns it. The router applies it to each result on the way out, so an
entity row keeps exactly the fields plugins received when the list was
drawn, in the same order and with the same values, minus the trip's
feed_token (listMine, create and update used to carry it as null). The
table is keyed by every method, so a new one does not compile without an
entry.
output-contract.schema.test.ts holds the lists to the migrated schema: a
new column on a whole-row table fails until it is published or withheld,
and each entity method is called on the real services to prove the
contract drops nothing but the withheld fields.
gen:plugin-facts publishes the lists to the SDK as PLUGIN_ENTITY_FIELDS
and PLUGIN_METHOD_RESULT, and the SDK's entity interfaces are type-checked
against them. Place.day_id, which no read ever delivered, is deprecated
rather than removed. KNOWN_ADDONS is generated from ADDON_IDS as well,
which adds the dawarich, roadtrip and tours ids the hand-kept copy lacked.
The SDK goes to 1.9.0, and its README and the plugin wiki pages describe
what a ctx call returns.
A handler behind express.json() is reached from the request stream's 'end'
event, long after the correlation middleware returned, so it looked as if
every write lost its id. It does not: raw-body and multer bind their
callbacks with AsyncResource at the point they are called, which is inside
the request's correlation. Nothing tested that, though, because every
REQID case was a GET. REQID-007 posts a JSON body and REQID-008 one that
arrives in several chunks, and both expect the handler to see the id the
response carries. The middleware comment now says why this holds.
The ownership rule says a boot-stable value that only Nest providers read
belongs to a registerAs token, yet TransitService froze TRANSIT_API_URL in a
module-top readEnv() const and KitineraryExtractorService read
KITINERARY_EXTRACTOR_PATH and PATH through readEnv() in its probe. Both now
come from tokens: transitConfig and kitineraryConfig, derived in
boot-derive.ts and dropped from deriveIntegrations, so readEnv() no longer
reads those variables at all. TransitModule and KitineraryExtractorModule
import AppConfigModule, which keeps them buildable on their own in tests.
Behaviour is unchanged: the same defaults, the same trailing-slash strip and
PATH split, now frozen per built app instead of at import. The app-config
README names each boot-stable variable that stays on readEnv() together with
the reader the container cannot inject into.
The env reference parity test forced TREK_MANAGED, MAPBOX_ACCESS_TOKEN,
CARTO_API_KEY and PLACES_API_BASE into server/.env.example with how-to text.
They are managed hosting only and stay out of the public docs, the same way
TREK_DB_FILE and APP_VERSION do, so they move to NOT_OPERATOR_SETTINGS with
that reason and their lines leave the self-hoster reference again. The
variables themselves and their behaviour are unchanged.
The 65 s default sat above nginx's 60 s but below the 90 s Traefik keeps an
idle upstream connection, so a default Traefik deployment could still reuse a
socket Node had just closed and answer that request with a 502. 95 s clears
both, and the headers timeout one second above it stays well under Node's
300 s request timeout and the 290000 cap the schema accepts.
Also moves the stray RawEnv import in boot-derive.ts up to the other import.
The process-state ports were meant to let a store shared between
processes replace the in-memory one, but they could not. The readers
outside the container (the broadcast functions, the backup restore's
permissions-cache flush, the pending-code sweep, hand-built consumers)
went straight to the module-level in-memory instances, so an overridden
provider was used by the gateway and the services while the broadcasts
and the restore flush kept using the old store. And every port was
synchronous, while a store in the database or on the network is async,
so plugging one in would have changed every signature anyway.
RateLimitStore, PendingCodeStore, OidcFlowStore and
PermissionsCacheStore now return promises, and RateLimitService,
OauthService, OidcService, PermissionsService and their callers await
them. The in-memory stores still decide synchronously, so who gets a
429, a code or a cached level is unchanged. RoomRegistry stays
synchronous: membership holds live sockets and broadcast() is a
synchronous call every domain makes, so several processes need a
fan-out in front of broadcast(), not a different place for the sets.
The process-wide ports get a ProcessStoreSlot. PermissionsModule,
OauthModule and RealtimeGatewayModule install the store the container
resolved and release it on shutdown, and the free functions, the sweep
and hand-built consumers read the slot, so overriding the provider
changes every reader. Tests provide fake stores and check that the
broadcasts, the restore flush and the sweep use them.
Two callers needed more than an await. The transit MCP tools counted in
a RateLimitService of their own that no provider could reach; they now
take the one RateLimitModule provides (their mcp_transit_* buckets never
overlap the REST ones). The link preview claimed its in-flight slot only
after the budget check, which now yields, so simultaneous askers would
each have been charged; it now claims the slot first, and an asker that
joined someone out of budget asks again on its own budget.
The unused putPendingCode/takePendingCode helpers are gone.
The data-directory change said Nest classes inject DataPathsService, but
only AdminService did. StorageRegistryService still read the built-in
uploads and backups roots and the scratch dir from module constants
frozen at import, and StorageJobsService read the scratch dir the same
way.
The registry now takes DataPathsService through its constructor (a
hand-built one resolves the same layout), and the jobs service asks the
registry for its temp dir. The plain helper modules keep calling
resolveDataPaths() because they run without a container or are shared
with code that does: config.ts, the database path, index.ts, the logger,
the backup functions and the plugin trees, which the out-of-process
supervisor reads too. The app-config README now lists both sides, and
the nest README, server/CLAUDE.md and the provider's own comment say the
same.
The keep-alive change moved HTTP_KEEP_ALIVE_TIMEOUT_MS onto the httpConfig
token, but the app-config README still listed only TRUST_PROXY and
HSTS_INCLUDE_SUBDOMAINS for it and left the variable out of the
boot-stable list. Also drops the dashes from two lines the recent
scheduling and entity changes rewrote.
A tick whose lease another process held returned from runLeased without
running, and traceEntry still wrote `cron <name> ok`, so both processes
logged the tick as run and the per-tick line could not tell which
replica did the work. runLeased now says whether the tick ran, and
traceEntry takes a `skipped` option that logs such a call at debug as
`cron <name> skipped <ms>: lease held by another process`.
Taking the lease is two statements (make sure the row exists, then the
conditional UPDATE that decides the race), so it now runs inside
UnitOfWork.transactional like every other multi-statement write.
Rate-limit counters, OAuth pending codes, OIDC login states and codes,
the permissions cache and the WebSocket rooms were Maps on module or
instance scope. That is right while TREK runs as one process, but nothing
could be swapped without editing every consumer.
Each now sits behind a port with an in-memory implementation that keeps
the old behaviour exactly: RateLimitStore, PendingCodeStore,
OidcFlowStore, PermissionsCacheStore and RoomRegistry. The owning module
provides the implementation with the same lifetime the state had
(process-wide for the pending codes, the permissions cache and the rooms,
per instance for the rate limits and the OIDC flows), the consumers take
it through their constructor, and a hand-built consumer falls back to the
same in-memory store, so tests and the free-function paths (the backup
restore's cache flush, the broadcasts) see what they saw before. A store
shared between processes can replace a provider later without touching
the services.
Three small gaps in how the process behaves at its edges:
- An unhandled rejection or uncaught exception crashed the process with
the stack on stderr only. index.ts now logs it to trek.log and runs the
same orderly shutdown a SIGTERM gets, exiting with 1; a second fatal
error during that shutdown exits at once.
- /api/health/ready kept answering 200 while a SIGTERM was draining the
process, so an orchestrator sent traffic for another probe period. A
shutdown now marks the new ReadinessService first (and Nest's own
shutdown hook marks it too), and the probe answers 503 from then on.
- Node closes an idle keep-alive connection after 5 seconds while reverse
proxies keep upstream connections for about a minute, which shows up as
intermittent 502s. The server now keeps them for 65 seconds, configurable
through HTTP_KEEP_ALIVE_TIMEOUT_MS (owned by the httpConfig token), with
the headers timeout one second above.
waitForCompletion only kept a tick from overlapping itself inside one
process. Two processes on the same database (a rolling update's overlap,
a second replica, a dev container pointed at the same file) both ran
every job: two auto-backups, double syncs and retention sweeps, and
reminder runs racing each other.
A new migration adds scheduler_leases, and CronRegistrarService now runs
every registered tick through SchedulerLeasesRepository.acquire: one
conditional UPDATE moves the job's row to this process when it is free
(no holder, a lapsed holder, or this process already), so exactly one
process wins. A running tick renews its lease every 20 seconds, a crashed
holder frees the job after a minute, and a finished tick keeps the lease
for 30 seconds so a peer whose timer fires late does not run the same
tick again. The per-row reminder claims stay as they are; jobs need no
change of their own.
A 5xx stack and its access-log line were two unrelated lines, the
exception filter printed the stack with console.error and no request
context, and the MCP, WebSocket, plugin RPC and cron entry points logged
nothing per call. A self-hosted bug report could not be traced from a
failure back to the call that caused it.
Every HTTP request now runs under an id held in AsyncLocalStorage: a
well-formed incoming X-Request-Id is kept, anything else is replaced by a
UUID, and the response carries it in X-Request-Id. The logger tags every
line written inside a unit of work with `[http <id>]`. The exception
filter leaves a server-side failure on res.locals and the access log
writes the request line and the stack as one entry under that id; without
an access log watching the response it still prints the error itself.
MCP tool, resource and prompt calls (through a new `around` seam on the
registry), WebSocket messages, plugin RPC dispatches and cron ticks each
run through traceEntry(), which gives the call its own id (remembering the
outer one as its parent) and logs one line for it: debug on success, like
the HTTP access log, and warn or error with the reason on failure. The
call's result and timing are unchanged, a failed cron tick is logged once
rather than twice, and a WebSocket handler rejection no longer reaches
rxjs's asynchronous rethrow.
Every log line ran existsSync, statSync and appendFileSync on the event
loop before it returned. On a slow data volume (an SMB or NFS bind mount
on a NAS) that stalled every request that logged, and rotation renamed
files without checking whether another process sharing data/logs had
rotated already.
The file half of the logger is now BufferedLogFile: a line is queued in
memory and the queue goes out with one appendFile per batch, through a
single serial chain, so no append runs while files are being renamed. The
files and the rotation are the same as before (trek.log rotates to .1 at
10 MB, five files in total), but the size is re-read before a rotation
and a file another process already rotated is left alone. A stalled disk
costs dropped lines with a count instead of unbounded memory, a failed
write is reported instead of thrown, and index.ts writes whatever is
still queued synchronously when the process exits.
Eight registerAs tokens existed while every consumer read the same values
through readEnv() instead, so the boot-stable layer the repo rules describe
was unused and nothing said which path owned a variable. FORCE_HTTPS, for
instance, was frozen by the middleware through the http token and read live
by the cookie code at the same time.
A variable is now owned by exactly one side. TRUST_PROXY and
HSTS_INCLUDE_SUBDOMAINS belong to httpConfig, which buildApp() already
injects into the pre-init middleware, and TREK_PLACE_PHOTO_DIR to the new
storageConfig, which the storage registry now injects instead of reading it
through RuntimeEnvService. Their derive functions moved to boot-derive.ts,
which deriveAll() never calls. The six tokens nothing injected are gone:
their values are needed before the container exists (config.ts, the
database, src/mcp, the SSRF guard) and stay on readEnv().
config-ownership.test.ts records the keys each side reads through a Proxy
and fails on a variable read through both, on a token nothing injects and
on a variable the schema does not validate. env-reference.test.ts holds
server/.env.example to the schema, which had drifted by 34 variables (SMTP,
WebAuthn, plugins, MCP session tuning, TREK_MANAGED and others); they are
documented now. jwt-verify.ts no longer cites a middleware file that is
gone.
The data and uploads directories were derived independently in about ten
files, each counting `..` hops from its own `__dirname`, and the file
logger used `process.cwd()` instead. Moving a file or starting the server
from another directory quietly moved where secrets, backups or logs went,
and the backup code named `data/travel.db` directly, so with TREK_DB_FILE
set a backup archived and a restore replaced a file the connection never
opened.
`resolveDataPaths()` in app-config now anchors data/, uploads/, backups,
tmp, logs, the JWT secret and encryption key files and the database file
(TREK_DB_FILE when set, else data/travel.db) on the server package root.
Code that runs before the container calls it directly; Nest classes inject
the new `DataPathsService`. Every previous anchor uses it, and the resolved
paths are unchanged for the image and for `npm run dev`.
The Postgres probe only ratcheted the methods it managed to call. A method
it could not build arguments for, one that threw before its first
statement and one that timed out were counted and logged but held to
nothing, so a new repository method with a callback parameter could land
SQLite-only SQL unseen, and the part of the code the probe reaches could
shrink without anyone noticing.
pg-probe-baseline.json gains an `uncovered` list of those methods. A
method outside it that goes unmeasured fails the run, and so does an
entry whose method is measured now or gone, so the list only shrinks.
It is seeded from the same first measurement as `failing`; a baseline
without the key reads as unmeasured.
The failing count also counted every refusal, including data errors the
sample values cause (a "probe" string in a date column, SQLSTATE 22007),
so renaming a parameter could fail the gate on a change that never
touched the SQL. Only classes 42 and 0A count now; the rest is reported
in the log and the step summary without being held.
The docs said the probe measures every repository statement. They now
say it measures what each method reaches with sample arguments against
empty tables, and that the methods it cannot reach are held to the list.
The probe shipped with scripts/pg-probe-baseline.json at `failing: null`,
and an unmeasured baseline only reported. Every repository statement
Postgres refused, new SQLite-only SQL included, still passed the job;
only the dialect helper cases could fail it, so the ratchet was off until
some later commit seeded the file.
An unmeasured baseline now fails the run outside --update, and the log,
step summary and annotation say which artifact file to commit. The job
stays red until the first measurement lands, so the gate cannot reach dev
holding nothing. A run that called no repository method (an unreadable
tsconfig, a moved repositories directory) fails as well, also under
--update, so a broken plan can neither pass nor seed an empty baseline.
Nothing in CI ran a statement against anything but SQLite, so the claim
that the dialect layer keeps the repositories portable was never measured
and new SQLite-only SQL could land unseen. The new postgres-probe job
(server/scripts/pg-probe.ts, npm run probe:pg) does that against a
postgres:17 service container:
- it derives a Postgres schema from the entities with MikroORM's schema
generator, keeping TEXT timestamps, mapping the SQLite-only defaults and
the NOCASE collation, and leaving foreign keys out;
- it runs every dialect helper on an in-memory SQLite and on Postgres
against the same expectation, and every case has to pass;
- it calls every public repository method once with arguments the
TypeScript checker builds from the parameter types, through a Postgres
driver that runs each statement alone in a rolled-back transaction with
a statement timeout and records whether Postgres accepted it.
The statements Postgres refuses are held per method to
scripts/pg-probe-baseline.json, which may only shrink. The baseline starts
unmeasured (failing: null): the job reports, passes and uploads the first
measurement as the pg-probe-baseline artifact, together with a JSON report,
so the commit that seeds it and every later commit that lowers it needs no
local Postgres. The probe refuses a database holding tables it did not
create.
@mikro-orm/postgresql joins the server's devDependencies at the 7.2.1 the
other MikroORM packages resolve to; the image's production install leaves
it out. The job needs the preflight gates like every test job, runs only
for the code area, stops after ten minutes and is part of ci-ok.
The dialect layer promised that a second engine would be a branch in
src/db/dialect/ and no sweep through the repositories, but every helper
only knew SQLite and threw for anything else. Each one now has a Postgres
branch, and the helpers whose text both engines read alike (COALESCE,
LOWER, TRIM, substr, ||, the aggregates, the CASE shapes) share one
spelling behind isKnownPlatform.
The Postgres branches keep the storage model SQLite has: the clock helpers
render UTC as the same YYYY-MM-DD HH:MM:SS text (to_char over CURRENT_TIMESTAMP
AT TIME ZONE 'UTC'), GLOB becomes an anchored regular expression, the integer
cast truncates through numeric so '14.0' still reads as 14, JULIANDAY
distance comes from the epoch difference, SQLite's null-safe IS becomes IS
NOT DISTINCT FROM, and NOCASE names a nondeterministic ICU collation a
Postgres schema creates. Bound values on the Kysely side are cast to text
where pg would otherwise send an untyped parameter the planner cannot
resolve.
The SQLite branches are untouched, so nothing changes for a running
install. The Postgres text is pinned in sql-functions.postgres.test.ts.
sql-functions.ts sat at 1007 lines, held there by the size baseline, so it
had no room for the Postgres branch every helper is meant to grow. The
Kysely-expression twins are a separate family already (a Kysely Expression
and a MikroORM RawQueryFragment do not convert into each other), so they
move to src/db/dialect/kysely-functions.ts, and the platform check plus the
fail-closed unsupported() move to platform.ts, which both files share.
No helper changes behaviour or spelling. lint:dialect now skips the whole
src/db/dialect/ directory instead of the one file, and the size baseline
drops its sql-functions.ts entry.
The datetime rule only matched a literal holding a call such as
datetime('now'). The query builder spells the same function as
eb.fn('datetime', [...]), where the literal is just the name, so that form
slipped through while strftime, json_extract and glob are caught bare. A
literal that is exactly datetime, julianday or unixepoch now counts as well,
the same way the glob rule treats a bare glob. The only such call lives in the
dialect layer, which is not scanned, so the baseline does not change.
seedDocumentProviders() had no caller left: the migration that creates the
tables and DocumentProviderSeeder both import only the DOCUMENT_PROVIDERS and
DOCUMENT_PROVIDER_FIELDS constants. Its two INSERT OR IGNORE statements were
the only SQLite-only spellings in the file and had been baselined rather than
removed. Deleting it lets the file leave the dialect baseline (13 hits down to
11) and the better-sqlite3 import allowlist in eslint.config.mjs, since nothing
in it touches the driver any more. DOCUMENT_PROVIDER_SEED_IDS goes with it, it
had no importer either.
The repositories already lean on the query builder, the generated table
types and sql-functions.ts, but the rules behind that were only implied.
server/CLAUDE.md now states them: which builder call replaces each SQLite
spelling, which helper covers time, strings and casts, where PRAGMA may live,
how a new migration is treated, and that the generated types describe the
SQLite storage model.
It also says plainly that MySQL and MariaDB are out of scope. The
repositories rely on INSERT ... ON CONFLICT and RETURNING, which MySQL lacks,
so Postgres is the second engine the code is prepared for.
ESLint keeps raw SQL out of the repositories, but nothing looked at the rest
of src/: the entities, the domain services, the src/db lifecycle files or a
migration added from now on. scripts/dialect-lint.mjs (npm run lint:dialect)
reads every string and template literal under src/, skipping comments, and
counts INSERT OR IGNORE/REPLACE, datetime(), strftime, json_extract, GLOB,
last_insert_rowid, AUTOINCREMENT, PRAGMA outside the lifecycle files, ||
concatenation in SQL and .insertId reads. scripts/dialect-baseline.json
holds today's thirteen hits; the counts may only go down, like lint:size.
The dialect layer is not scanned, and neither are the migrations that have
shipped.
The four seeders that still spelled INSERT OR IGNORE (addons, photo and
document providers, the schema_version marker) are Kysely inserts that do
nothing on conflict now, so a re-run still leaves an operator's rows alone.
They leave the SQL-string exemption list in eslint.config.mjs and their pins
in the selector test.
Kysely fills InsertResult.insertId from SQLite's last_insert_rowid. On
Postgres it is undefined, so the sixteen repository inserts that returned
Number(result.insertId) would hand NaN ids to their services without an
error. They now add .returning('id') and read the id from the row, which
SQLite (3.35 and later, bundled by better-sqlite3) and Postgres both answer
with the same value.
DemoRepository ran its example-trip seed as nine SQL strings through
connection.execute, including INSERT OR IGNORE and a bare CURRENT_TIMESTAMP.
Those are Kysely inserts on the generated table types now: ON CONFLICT
(trip_id, user_id) DO NOTHING for the membership row, the dialect helper for
the timestamp, RETURNING id where the caller needs it. With no SQL text left
it drops off the ESLint exemption list and its pin.
ESLint now refuses any read of insertId in src/db/repositories and
src/db/seeders, so the pattern cannot come back.
The repositories typed their Kysely statements against 159 table interfaces
written by hand, one or more per repository. Nothing compared them with the
schema: a migration that renamed a column or changed its type left them
stale while tsc stayed green, and several had already drifted
(reservations.accommodation_id is TEXT, not INTEGER).
scripts/generate-db-types.ts migrates a throwaway database with the real
migrations, reads every table back and writes one interface per table into
src/db/kysely/ plus the DB map in db.ts. Every hand-written interface is now
Pick<DB, ...> of that type, and npm run check:db-types fails when the
committed files no longer match the migrations.
A column an insert may leave out (rowid, DEFAULT, nullable) is
InsertOptional<T>, declared as ColumnType<T, T | void, T>. The server
compiles without strictNullChecks, where Kysely's own Generated<T> still
demands the column; that is why the old code kept separate insert-only
interfaces. The ids callers pass as number | string are cast to the column
type at the bind, so every statement binds the same value as before.
Version 6 only adds the driving settings store; dropping the area cache
is version 7. The case name claimed both, so it now names only the new
store and a comment explains where the empty area cache it still checks
comes from.
The skip of newer-format queued writes only exists from this release on,
so a rollback to an older release still replays them. On a rolled-back
install the skipped rows also stay pending with no retry or discard path
and hold back later writes to their entity, leaving Clear cache as the
only way out. Say both in the wiki and in client/CLAUDE.md instead of
promising that a rollback is always safe.
Each version() of the offline database runs once, on a user's device, when a
new bundle opens what an older one wrote. None of them ran in CI: the v3 blob
cache backfill and the v8 area-fingerprint reset had never been exercised
against an older database.
The new test builds the database as each previous version left it in
fake-indexeddb, seeds a row into every table it had, opens it with the current
schema and checks that the rows survived and that the version did what it is
for. A guard reads the declared versions from offlineDb.ts and fails on a
duplicate, on a version without an upgrade case, or on stores the history
fixture does not reproduce. TrekOfflineDb is exported so the test can open a
named database with it.
A queued write outlives the bundle that wrote it. After a deploy a tab can
still run the previous bundle, and an install can be rolled back, so a row
written by a newer build may be replayed by an older one that does not know
its shape.
Each row enqueue() writes now carries schemaVersion (MUTATION_SCHEMA_VERSION)
and the UI build that queued it. Both fields are optional, so rows queued
before this read as format 1 and replay as before. A flush leaves a row from a
newer format in the queue without sending, marking or dropping it, and holds
back the later writes to the same entity as it does behind a parked write, so
they keep their order until a build that knows the format replays them.
The demo seed runs in the schema bootstrap, at boot and again after every
reopen. Since the baseline save moved out of the seed into DemoResetJob,
only the boot read the seed's mark. A restore onto a demo instance with no
baseline, whose archive held no admin trips, seeded the example trips
during the reopen and then saved nothing, so the hourly reset kept logging
"No baseline found"; the mark also stayed set for the rest of the process.
Before the move, that path saved the baseline from inside the seed.
DatabaseLifecycle now runs listeners after a reopen's schema bootstrap,
and DemoResetJob registers one in demo mode that does the same check as at
boot: save through the backup port when the seed just ran and no baseline
exists. Each check clears the mark. A failed save is logged, never turned
into a failed reopen. AdminModule imports DatabaseLifecycleModule from the
shared database kernel for it.
The DB paragraph said MaintenanceRepository is never built with `new` in
src/, but MaintenanceModule's factory does exactly that. Say what is
actually true: the module's factory is the one place that builds it, and
every consumer injects it.
DatabaseBackupModule stays @Global, but its comment gave a vague reason.
The real one is concrete: the backup domain already reaches admin through
auth and settings, so importing the module from AdminModule would add a
domain cycle that lint:boundaries rejects. The module comment now says
that and points at the e2e harness that imports it by hand, and
AdminModule notes where its DATABASE_BACKUP comes from.
SqliteDatabaseBackup.replace reopens the connection in a finally block, so
it runs even when the swap throws. A reopen that failed there was stored
and then dropped: the swap's exception propagated past the return that
would have carried it. The demo reset caught that swap error, logged
"Failed to restore baseline" and returned normally, leaving the instance
without a database connection and nothing in the log to say so. Before
the port, the same branch awaited reinitialize() and its rejection reached
the tick's log line.
replace now throws a DatabaseConnectionLostError when both steps fail. It
carries the swap error and has the reopen failure as its cause, and its
message tells the operator to restart. The demo reset lets that error
through to DemoResetJob's tick, which logs it, and still logs and skips a
failed swap whose reopen succeeded. A restore that hits it answers the
same 500 as any other thrown restore error, with both failures in the
server log.
Moving the first baseline save out of the boot-time seed into DemoResetJob
also changed when it happened. The seed saved it only on the boot that put
the example trips in; a later boot returned early and saved nothing, so an
instance with data but no baseline logged "No baseline found, skipping" on
every hourly reset. The job instead saved one on any demo boot that found
no travel-baseline.db, freezing whatever the database held at that moment,
user edits included, as the target the hourly reset rolls back to.
The seed now leaves a mark when it seeds the example trips, and the job
saves the first baseline only when that mark is set and no baseline
exists, which is the old trigger. As before, the save does not depend on
the cron registrar being enabled. The demo mode wiki page says so.
Backup, restore and the demo reset each handled the SQLite file
themselves: backup.impl closed and reinitialized the connection, built
MaintenanceRepository with new, and assumed the database was
data/travel.db. That tied the whole chain to one local file and hid two
bugs.
They now go through DATABASE_BACKUP, a port declared in the shared
database kernel (snapshot, verify, keep a copy before a restore, replace).
SqliteDatabaseBackup implements it with what the code did before:
VACUUM INTO for a consistent copy, and a close, copy, sidecar drop,
rename and reopen through DatabaseLifecycle for a restore. BackupService
injects it and hands it to backup.impl. The demo reset and the baseline
save use the same port; the first baseline is saved by DemoResetJob once
the app is up, because the boot-time seed runs before anything can be
injected. Another engine can plug in a logical dump behind the port.
Fixes on the way:
- The backup and the restore ignored TREK_DB_FILE. The backup looked for
data/travel.db and, with the database elsewhere, silently left it out
of the archive; the restore wrote data/travel.db and never replaced
the live database. Both now use DatabaseLifecycle.file, the path the
connection runs on, and the pre-restore copy sits next to it.
- A failed VACUUM INTO fell back to archiving the live WAL-mode file
without a word, the torn copy the snapshot exists to prevent. The
backup now fails: nothing is archived or committed, the existing
backups are untouched, a partial snapshot is removed, and the log says
why. A missing database file and a failed checkpoint are logged too.
MaintenanceRepository was built with new over an injected EntityManager
in the readiness probe and in UserCleanupService, outside the container.
MaintenanceModule now provides it, as a factory over the same
context-resolving EntityManager Nest injects everywhere, and both inject
it. The class itself stays free of Nest decorators, like the rest of
src/db, so tests keep building one over their own EntityManager.
No behaviour changes: the probe still pings and the erasure still runs
inside the account deletion's transaction.
db/database.ts opened the connection as a side effect of being imported,
so the moment anything reached it the database was open. index.ts had to
defer loading the app until a first-boot restore was done, and nothing
in the container decided when the connection opened or closed.
The module now opens the connection on first use and opens nothing at
import. A connection closed on purpose (a restore, the shutdown) stays
closed until it is reopened, as before. The new DatabaseLifecycle
provider owns the decisions: buildApp() opens through it, which also
binds the ORM to later swaps (the hook attachOrm used to register) and
runs the schema bootstrap in the same order as before, and the shutdown
closes through it. Its file property is the one resolved database path.
The module keeps every export, so the test suites that replace it with
vi.mock keep working; buildDbMock gains the new openDb.
_flushRun holds the running pass or null. Testing it as a boolean is
correct but reads like a forgotten await, and SonarCloud counts it as a
reliability bug (S6544). The explicit comparison says what is meant.
Maurice