Commit Graph

1353 Commits

Author SHA1 Message Date
Maurice 5983bead7c fix(extract): don't let the day-clamp fallback break reservation resync (#1288)
This branch added a clamp-to-nearest-day fallback to resolveDayIdFromTime so an
imported booking whose exact date has no day row still lands on a day. After
rebasing onto dev, that collided with #1288's resyncReservationDays, which
relies on the original "null when no exact day" semantics to leave a booking
whose date now falls outside the range untouched — instead it snapped to an edge
day (TRIP-SVC-019 failed: expected day_id kept, got the clamped one).

Make clampToNearest an opt-in parameter (default true, preserving the import
behaviour for create/update) and have resyncReservationDays pass false, so
out-of-range bookings keep their day_id. Full server suite green (4082).
2026-06-28 11:36:26 +02:00
Maurice 0d104d8a09 fix(i18n): add Swedish translations for the AI booking-import settings
The Swedish (sv) locale landed on dev (#1325) after this branch added the
AI-parsing settings/reservation keys to the other locales, so sv was missing
them — strict i18n key parity failed after rebasing onto dev. Adds the 3
reservations.import.* and 17 settings.aiParsing/aiAlwaysRetry keys in sv.
2026-06-28 11:28:09 +02:00
Maurice 97b96e3c3b test(setup): stub websocket addListener/removeListener in the global mock
BackgroundTasksWidget (mounted globally in App) subscribes via addListener/removeListener from api/websocket, but the global test mock didn't export them, so every test that renders <App/> threw on mount. Add the two stubs. (Surfaced now that the page-pattern check passes and the client test step actually runs.)
2026-06-28 11:19:45 +02:00
Maurice 0be1fbeed5 test(llm-parse): cover the extraction router, client factory and import jobs
The new LLM extraction router shipped with little branch coverage, dropping src/nest below the 80% gate. Add unit tests for routeExtraction (flights/single/union/error paths, deterministic booking-wide fill), the native Ollama format client, the provider factory, the local-router service path with its type-aware text cap, the flat->schema.org mapper's remaining reservation types, and the background import-jobs runner. Also remove the now-unused validate.ts (only its FlatLike type was still referenced; moved to flat-schemas).
2026-06-28 11:19:44 +02:00
Maurice b5ac967c01 refactor(planner): move the import-review bridge effect into the page hook
TripPlannerPage held a useEffect (the background-import → review bridge), which trips the page-pattern check (pages must stay wiring containers). Move the effect and its store/IndexedDB wiring into useTripPlanner where the rest of the import-review state already lives.
2026-06-28 11:19:44 +02:00
Maurice 539af62180 feat(settings): use the shared custom dropdown for the AI parsing provider
Swap the native select for CustomSelect so the provider picker matches the rest of the app's styling (dark mode, portal dropdown).
2026-06-28 11:19:44 +02:00
Maurice bcc1fd5a9e fix(settings): show the Integrations tab when only AI parsing is enabled
hasIntegrations gated the tab on memories/mcp/airtrail only, so a user with just the llm_parsing addon enabled saw no Integrations tab and could not reach the AI parsing config. Include llmEnabled in the gate.
2026-06-28 11:19:43 +02:00
Maurice 124193d56c feat(settings): let users set their own AI parsing model
Adds an "AI parsing" section under Settings -> Integrations where a user can choose the LLM provider, model, base URL, API key and multimodal option used for booking extraction. This per-user config applies when an admin has not configured an instance-wide model. Reuses the existing encrypted user settings: the API key is stored encrypted, never prefilled, and a blank field keeps the stored one. Adds settings.aiParsing.* across all 20 locales.
2026-06-28 11:19:43 +02:00
Maurice aff0e74519 chore(extract): recommend only Qwen3-8B (drop Qwen2.5 from the curated list)
Qwen3-8B is the identified default; the prior Qwen2.5 entries are no longer needed in the pull list.
2026-06-28 11:19:42 +02:00
Maurice cf1c7aea71 fix(import): persist source files in IndexedDB so attach survives a reload
The source document was only kept in memory on the background task, so a page reload during the (now always-LLM ~25s) parse lost it and the booking saved without its file. Store the uploaded files in IndexedDB keyed by job id; the review loads them from there when the in-memory copy is gone, and a 1h TTL prunes abandoned imports.
2026-06-28 11:19:42 +02:00
Maurice 5dee42bef7 fix(import): preview the parsed cost as linked in the review modal
During the per-item import review the booking isn't saved yet, so the Costs section showed an empty 'Create expense' even though a linked cost will be created on save. Show the parsed price (amount + category) as the pending linked expense so the user can verify it up front. Reuses existing i18n keys.
2026-06-28 11:19:42 +02:00
Maurice da9a87b59b perf(extract): cap single-booking text tighter; require rental company
A long single-booking PDF (e.g. an 11-page rental voucher) spent ~200s on CPU prompt-eval at the 16k cap, though its data sits in the first ~2k. Cap non-flight docs at 6k (flights keep 16k for all legs). Also make the rental operator a required field so the car gets a real title.
2026-06-28 11:19:41 +02:00
Maurice e0dfb6e4e3 fix(import): refresh costs immediately after an imported booking is saved
The saving client gets no budget:created echo (X-Socket-Id) and the create response omits the linked budget item, so the booking's Costs section and the Costs tab stayed stale until a manual reload. Reload the budget items right after a create that carried a budget entry.
2026-06-28 11:19:41 +02:00
Maurice 34525d57ee fix(extract): require the hotel address and ask for the rental company
After dropping the vendor templates, the model skipped the (often unlabeled) Expedia-style hotel address — making address a required schema field forces it to emit the street-address line, restoring the booking's location/place. Also hint the rental company so a car booking gets a real title instead of the generic fallback.
2026-06-28 11:19:41 +02:00
Maurice 5de57d846b refactor(extract): drop vendor templates, let the model drive with deterministic backfill
Now that a capable instruct model (Qwen3-8B, thinking off) reads name/address/dates/legs reliably across formats, the per-vendor template short-circuit distorted more than it fixed: brittle on layout variations and overriding the better model output. Remove the template layer; the model extracts the structure and Schicht 2 backfills the confirmation/total and takes the currency from the document's own symbol (correcting model misreads like ¥→$). Per-type prompts now also ask for address and price/currency.
2026-06-28 11:19:40 +02:00
Maurice ead71b5e3a feat(extract): recommend Qwen3-8B as the local extraction model
A/B against the prior default (qwen2.5:7b) on CPU showed Qwen3-8B is both faster and more accurate on tricky/multilingual booking docs (correct Airbnb year+price, correct DisneySea admission date), once thinking is disabled — which the router now does. Feature it as the recommended pull, keep qwen2.5:7b as the fallback.
2026-06-28 11:19:40 +02:00
Maurice 462400229c fix(extract): disable model thinking for grammar-constrained extraction
Hybrid/reasoning models (Qwen3 and similar) default to emitting reasoning tokens, which collide with Ollama's format-grammar constraint — on CPU this produced null/unparseable output and blew the latency budget (qwen3:8b: null or 300s timeouts vs ~20s with thinking off). Send think:false on the /api/chat call; Ollama ignores it for non-thinking models (verified on qwen2.5:7b), so it's safe and unlocks the stronger Qwen3 family.
2026-06-28 11:19:40 +02:00
Maurice 45f3e46a24 feat(import): attach the parsed source document to each booking
Keep the uploaded files on the background task and hand them to the review flow, so each reviewed booking pre-fills its Files with the document it was parsed from (uploaded with the booking on save). The two modals also adopt the shared resolveDayId helper.
2026-06-28 11:19:39 +02:00
Maurice 3acc8c9b40 refactor(extract): dedupe currency/day helpers, drop redundant casts, support JPY vouchers
Code-audit clean-ups: share one normCurrency between the router and the templates, lift the duplicated nearest-day resolver into formatters.resolveDayId, drop two needless as-unknown-as casts at the fillBookingWideFields call sites, restore routeExtraction's doc comment, and give the broker template readable names. Plus recognise ¥/JPY and fall back to a standalone symbol amount, so a Klook-style voucher whose price sits far from any label still yields a cost.
2026-06-28 11:19:39 +02:00
Maurice 11e52a7d05 fix(import): refresh costs after a booking review so imported expenses appear without a reload
Imported bookings auto-create their linked budget items server-side, but the saving client suppresses its own budget:created echo, so the Costs list stayed stale until a manual reload. Reload the budget items when the review session ends.
2026-06-28 11:19:39 +02:00
Maurice ef4cf5595b fix(import): resolve an imported transport's day from its parsed dates
A reviewed transport (e.g. a rental car) arrived with only its parsed pick-up/return dates and no day_id, so the modal kept just the time and saved a bare "HH:MM" with no date. Resolve start/end day from the parsed dates (exact match, else nearest trip day) so the booking lands on the right days.
2026-06-28 11:19:38 +02:00
Maurice ffb7e0e46a fix(reservations): skip un-geocoded endpoints instead of failing the save
reservation_endpoints.lat/lng are NOT NULL, so saving a reviewed transport whose pick-up/return couldn't be geocoded threw a 500 and lost the whole booking (dates, linked cost). Skip those rows; the dates still persist on reservation_time/reservation_end_time.
2026-06-28 11:19:38 +02:00
Maurice 7ff6900ff1 fix(import): keep the parse-progress widget across a reload
Persist the background-import tasks (id/trip/status only) and re-fetch each job's status on mount, so a parse still running when the page reloads keeps its widget instead of vanishing; expired jobs (404) are dropped and a restored 'done' task re-fetches its items.
2026-06-28 11:19:38 +02:00
Maurice dbcbd71b4c fix(extract): backfill booking code/total and harden the reference match
Apply the deterministic confirmation-code and total fill to vendor-template results too (not just model output), and require the captured reference to contain a digit so a bare 'Confirmation'/'Reference' label no longer grabs the next prose word.
2026-06-28 11:19:37 +02:00
Maurice 40b6aac2a9 feat(extract): add Expedia and rental-broker booking templates
Pull the hotel/rental fields these vendors print in a stable text layout (name, address, stay/pickup dates, price, reference) deterministically, so the import stops depending on the local model for them. Handles German long/abbreviated months and English dates incl. 12-hour and comma forms.
2026-06-28 11:19:37 +02:00
Maurice 8f061b0f4a fix(import): create linked costs and accommodations from reviewed bookings
Reviewing an imported booking saves it through the normal reservation
form, which dropped the parsed price (so no linked cost was created) and
only created the accommodation when both nights matched a trip day.
Carry the parsed price into a linked cost on save, and create the
accommodation from whichever day the check-in/out dates resolve to.
2026-06-28 11:19:37 +02:00
Maurice cfa53d4f3b feat(import): parse bookings in the background with a progress widget
Parsing a booking can take a while on a CPU host, so don't hold the
upload modal open for it. The async import endpoint returns a job id
right away; the parse runs server-side (one at a time per user) and
pushes progress over the user's WebSocket, and a small widget in the
bottom corner tracks it while the user keeps navigating and editing.
A finished job opens the per-item review from the widget.
2026-06-28 11:19:36 +02:00
Maurice 7fe68cc3c0 feat(extract): drive local parsing through a layered extraction router
The single-shot prompt was unreliable on multi-leg flights and longer
documents, and slow on a CPU host. For the local provider, run a small
router instead:

- deterministic vendor templates first, with no model call at all
- exactly one grammar-enforced call per document via Ollama's native
  `format` (flights as a flat array of legs, everything else as one flat
  reservation, the type picked from keywords or a union schema)
- booking-wide fields (booking reference, total price, the overnight
  arrival day) filled deterministically from the text afterwards, and
  dates coerced to ISO so a natural-language date can't slip through

Recommend qwen2.5 in the AI-parsing settings instead of NuExtract.
2026-06-28 11:19:36 +02:00
Maurice bc4bc6a3ed feat(import): review each parsed booking before it's saved
Instead of writing parsed items straight to the trip, the import opens the
normal edit modal pre-filled for each one, so you can check and fix it before
saving — useful when a model guesses a wrong date or address. Hotels gained an
editable address field; on save an existing place is matched by name, otherwise
the reviewed address is geocoded and a new place is created.
2026-06-28 11:19:35 +02:00
Maurice a355f4a226 fix(admin): tidy the AI parsing settings and recommend the 2B model
The provider picker is the shared CustomSelect now and the form is split into
clear sections rather than a flat stack of inputs. NuExtract 2.0 2B is the
recommended default — fastest on a CPU-only host and MIT licensed; the 4B
carries a non-commercial licence, so it's no longer flagged as recommended.
2026-06-28 11:19:35 +02:00
Maurice e567d9baaf feat(extract): drive NuExtract with its native template
NuExtract isn't an instruct model — fed a plain chat prompt it just echoes the
schema back. Detect a NuExtract model by id and talk to it the way the model
cards document: the JSON template inlined in a single user message, no system
prompt, no json_schema, temperature 0. Its flat result is mapped back to the
same KiReservation shape the rest of the pipeline already uses, so nothing
downstream changes; every other model keeps the generic prompt.

Money is taken as a verbatim string and parsed locally (German "1.580,22 €"
otherwise comes back as 1.49772), a rental car's pickup/return ride the from/to
fields so a stray form label doesn't become the location, and a lodging with no
name falls back to its address instead of being dropped.
2026-06-28 11:19:35 +02:00
Maurice 3e72108df8 fix(extract): refresh accommodations after a booking import
A freshly imported hotel links to an accommodation that lives outside the
trip store, so loadTrip alone left the reservation edit modal with blank
place/date fields. Reload the accommodations list once the import finishes.
2026-06-28 11:19:34 +02:00
Maurice 8d49872ce4 feat(extract): create a linked cost from the booking price on import
When a confirmation carries a total price, record it as a real expense
linked to the reservation (in the matching Costs category) instead of
leaving the amount in metadata only. Gated on the Costs addon.
2026-06-28 11:19:34 +02:00
Maurice 154c2a23dc feat(extract): capture seat, class, platform, price + event venue contact
Request and map root-level seat/class/platform and a total price/currency into reservation metadata (shown on the card; price reuses the existing label). Read both the root and reservationFor and tolerate common field-name aliases (priceAmount, priceCurrencyISO4217Code, fareClass, ...) since models name these inconsistently. Also capture event/attraction venue telephone + url onto the auto-created place, matching lodging/restaurant.
2026-06-28 11:19:33 +02:00
Maurice 87bc079525 perf(extract): cap LLM input at 4000 chars for CPU-only speed
On a GPU-less host the model's prompt-eval time scales with input length and dominates total latency. Booking details sit at the top of a confirmation, so capping the extracted text at 4000 chars (was 8000) roughly halves extraction time (~50s warm for a capable local 7B model) with no loss of fields on real hotel/rental confirmations. Tunable if a long multi-segment itinerary needs more.
2026-06-28 11:19:33 +02:00
Maurice 93b4051df4 feat(extract): fill transport/booking fields, geocode endpoints, assign days
- rental car: request+map dropoffLocation, emit pickup->return from/to endpoints, set a location string (G1/G2/G3). - geocode endpoints (stations/stops/terminals/rental desks) on confirm via Nominatim; mapper now emits coordless named endpoints and confirm persists only the geocoded ones (G6). - assign every dated booking to the nearest trip day so it still shows when slightly out of range, and keep hotel accommodation from vanishing when a check date misses (G5/G10). - fix bus mislabelled as train + add bus_number metadata (G7/G8), flag malformed boats (G9), accept root start/end time for events (G11). - raise the local-LLM timeout to 300s for CPU-only Ollama.
2026-06-28 11:19:33 +02:00
Maurice 874dbcbb32 fix(extract): make AI imports reliable and fast on local models
client: the import call inherited the global 8s axios timeout and aborted long LLM extractions even though the server finished it; remove the timeout. server: raise the OpenAI-compatible LLM timeout 60s->180s (a cold Ollama model can take ~45s to first token). server: cap extracted text to 8000 chars before the LLM - multi-page T&C tails (30k+ chars) overflowed the context window, truncating the relevant head and making CPU inference crawl; booking details sit at the top.
2026-06-28 11:19:32 +02:00
Maurice 534e7a798e fix(extract): auto-run the AI fallback when the addon is enabled
Booking import only fell back to the LLM when each user flipped an 'always retry with AI' toggle, so by default files kitinerary returned nothing for just failed. Run the fallback automatically whenever the AI Parsing addon is on (fallback-on-empty); drop the now-redundant per-user toggle and its setting.
2026-06-28 11:19:32 +02:00
jubnl 5d1c72a58a feat(extract): extract data using LLM 2026-06-28 11:19:32 +02:00
Maurice 41c541828f fix(setup): warn when ADMIN_EMAIL/ADMIN_PASSWORD are ignored, ship reset-admin
The first-run seeder only applies ADMIN_EMAIL/ADMIN_PASSWORD on an empty
database and then silently ignores them. People add the vars after the first
boot, or pull a fresh image without clearing ./data, restart, and cannot log
in with no hint why (#1339). The default is a generated password (not the
.env.example placeholder), printed once in the first-run box. Now: warn loudly
when the vars are set but a user already exists, and warn on a partial
(one-of-two) config instead of quietly falling back.

Also ship the reset-admin recovery script in the image -- it was never COPYed in
despite the wiki referencing it. node server/reset-admin.js resets/creates
admin@trek.local with a generated password (RESET_ADMIN_EMAIL/RESET_ADMIN_PASSWORD
overridable), picks a free username so it cannot trip UNIQUE(username), and sets
must_change_password.
2026-06-28 11:10:40 +02:00
Maurice 37f1fff367 Merge main into dev after the v3.1.3 release 2026-06-28 10:59:53 +02:00
Maurice 0c1c534435 docs(wiki): document the snap Docker + no-new-privileges startup failure 2026-06-28 10:30:37 +02:00
github-actions[bot] 0631e34a79 chore: bump version to 3.1.3 [skip ci] v3.1.3 2026-06-27 19:10:04 +00:00
Maurice 8a013f6fa9 fix(build): bump the client to vite 8.1.0 so the amd64 image builds
The 3.1.3 docker build failed on linux/amd64 while bundling the client
(arm64 happened to pass): vite 8.0.16 pins rolldown 1.0.3, but tsdown
pulls rolldown 1.1.2, and on amd64 npm hoists the 1.1.2 native binding so
vite's 1.0.3 rolldown loaded a mismatched one ("builtin:vite-wasm-fallback
does not match any variant of BindingBuiltinPluginName"). Moving the client
to vite 8.1.0, which expects rolldown 1.1.2, lines the bundler up with the
hoisted binding. Verified by building the client in a clean linux/amd64
node:22 container.
2026-06-27 21:09:05 +02:00
Maurice 7c3440f139 Revert "chore: bump version to 3.1.3 [skip ci]"
This reverts commit 4ceea09e31.
2026-06-27 20:49:58 +02:00
github-actions[bot] 4ceea09e31 chore: bump version to 3.1.3 [skip ci] 2026-06-27 18:15:27 +00:00
Maurice 03cdb4d276 fix(files): reject cross-trip reservation/place/assignment links
A member of one trip could point a file at a reservation, place or
day-assignment belonging to another, private trip — on upload, on a
metadata update, or through the file-link endpoint. The reservation join
in the file list and the links list then returned that trip's reservation
title, disclosing it across the trip boundary and letting an attacker
enumerate foreign reservation titles by their id.

The file already had to belong to the caller's trip; now the linked
reservation/place/assignment must too. findForeignLinkTarget checks each
supplied id against the trip (assignments via day -> trip) and the upload,
update and link handlers reject a cross-trip reference with 400 before it
is stored. Same-trip links and clearing a link are unchanged.
2026-06-27 20:14:52 +02:00
Maurice f0877a2e7d Replace the 3.0 upgrade notices with a thank-you / support modal
The 3.0 "what's new" notices have served their purpose, so swap them for a single thank-you notice that comes back once on every fresh install and version bump. It carries Buy Me a Coffee and Ko-fi buttons and only shows on desktop. Adds a per-version recurring mode (new dismissed_app_version column) plus external-link CTAs to support it; the 3.0.14 whitespace-collision admin notice stays active.
2026-06-27 20:14:52 +02:00
Maurice aa91f009ad fix(costs): freeze the FX rate so settled expenses don't reopen when rates drift (#1335)
Settle-up transfers are stored as fixed amounts, but a foreign-currency expense was re-converted with live rates on every settlement calc. When the rate drifted, the fixed transfer no longer cancelled the re-valued expense and a few-cent residual re-opened the settled position. Foreign-currency expenses now freeze the live rate at entry time into the existing budget_items.exchange_rate column, and the settlement converts with that frozen rate when working in the trip currency. Legacy rows (exchange_rate = 1) keep using live rates, so historical data is unchanged until re-edited; rate fetch failures fall back to live rates.
2026-06-27 20:14:52 +02:00
Maurice 2277f28a57 fix(airtrail): import the airline name, not the ICAO code (#1334)
AirTrail returns each airline as {icao, iata, name}, but the import reduced it to the ICAO/IATA code, so an imported flight showed e.g. 'EWG' instead of 'Eurowings'. The picker and the stored reservation now use the airline name (falling back to the code when AirTrail has none). The raw code is kept in metadata.airline_code so the writeback to AirTrail still sends a code, not a name (#1240), and the change-detection snapshot hash stays on the code so existing flights don't spuriously re-sync.
2026-06-27 20:14:52 +02:00