feat(llm-parse): read a photo, as a booking or as a receipt in Costs

Whether the AI Parsing model reads images is an admin setting now, and it
lands with the two features that send a photo, so it has something to
control.

The setting is `vision: 'auto' | 'on' | 'off'` on the addon config. On
Automatic a local Ollama is asked what the model can do (`/api/show`), and a
cloud provider means no; Yes and No are applied as set. The old `multimodal`
flag is not read: the admin screens wrote it false on every save, so a
stored false meant nothing. The per-user toggle keeps its on/off meaning and
is read by the extraction now. The lookup is server-side only: capped read,
validated body, never throws, cached per server and model.
GET /api/llm/capabilities answers it for the signed-in caller, and both
photo pickers below ask it.

The booking import takes JPG, PNG and WEBP when the model reads images.
The photo goes as an image to every provider (image_url, an Anthropic image
block, Ollama's native chat with `images`), capped at 1600px, and skips KDE
Itinerary. A scanned PDF, which used to reach the model as empty text, has
its first two pages drawn by pdf-parse and sent the same way. The rest is
the import as it was: background job, review, the file attached to the
booking, the linked cost. On a real qwen3.5:4b a train ticket came back as
bare clock times until the router gave the model every type's field
meanings, and its price as an object inside a string, which read as 4900;
both are handled.

Costs gets Scan receipt: a dialog laid out like the booking import's takes
one photo, read on the import job queue, then reviewed in the existing
expense editor pre-filled with merchant, total, currency, day and lines,
the photo waiting in Receipts & Invoices to be uploaded on save. No table,
no migration, no category guess, no document-type detection.
This commit is contained in:
RemEx
2026-09-24 17:12:54 +00:00
committed by Maurice
parent 706c6c3959
commit b1c05bca83
183 changed files with 3098 additions and 218 deletions
+9
View File
@@ -1,4 +1,6 @@
import {
asLlmVision,
LLM_VISION_MODES,
adminUserCreateRequestSchema,
adminUserUpdateRequestSchema,
adminPermissionsRequestSchema,
@@ -106,6 +108,13 @@ describe('adminAddonUpdateRequestSchema', () => {
});
});
describe('asLlmVision', () => {
it('keeps the three modes and reads anything else as auto', () => {
for (const mode of LLM_VISION_MODES) expect(asLlmVision(mode)).toBe(mode);
for (const other of [undefined, null, '', 'ON', true, false, 1]) expect(asLlmVision(other)).toBe('auto');
});
});
describe('adminCollabFeaturesRequestSchema', () => {
it('accepts a single computed key', () => {
expect(adminCollabFeaturesRequestSchema.safeParse({ chat: false }).success).toBe(true);
+24
View File
@@ -80,6 +80,30 @@ export const adminAddonUpdateRequestSchema = z.object({
});
export type AdminAddonUpdateRequest = z.infer<typeof adminAddonUpdateRequestSchema>;
/**
* Whether the AI Parsing model is handed a photo, stored as `vision` on that
* addon's config. `auto` asks a local Ollama server what the model can do and
* means no for a cloud provider; `on` and `off` are the admin's word.
*/
export const LLM_VISION_MODES = ['auto', 'on', 'off'] as const;
export type LlmVision = (typeof LLM_VISION_MODES)[number];
/**
* The photos handed to a model that reads images. HEIC is not here: no provider
* TREK talks to reads it, and a browser picking from the camera roll hands over
* a JPEG anyway.
*/
export const LLM_PHOTO_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.webp'] as const;
/** GET /api/llm/capabilities: what the caller's AI model can be handed. */
export const llmCapabilitiesResponseSchema = z.object({ images: z.boolean() });
export type LlmCapabilitiesResponse = z.infer<typeof llmCapabilitiesResponseSchema>;
/** A stored or submitted value as a mode; anything else, a missing value included, is `auto`. */
export function asLlmVision(value: unknown): LlmVision {
return (LLM_VISION_MODES as readonly unknown[]).includes(value) ? (value as LlmVision) : 'auto';
}
// Fully partial — the client PUTs one computed key at a time.
export const adminCollabFeaturesRequestSchema = z.object({
chat: z.boolean().optional(),