mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-21 06:26:26 +00:00
feat(llm-parse): read a photo, as a booking or as a receipt in Costs
Whether the AI Parsing model reads images is an admin setting now, and it lands with the two features that send a photo, so it has something to control. The setting is `vision: 'auto' | 'on' | 'off'` on the addon config. On Automatic a local Ollama is asked what the model can do (`/api/show`), and a cloud provider means no; Yes and No are applied as set. The old `multimodal` flag is not read: the admin screens wrote it false on every save, so a stored false meant nothing. The per-user toggle keeps its on/off meaning and is read by the extraction now. The lookup is server-side only: capped read, validated body, never throws, cached per server and model. GET /api/llm/capabilities answers it for the signed-in caller, and both photo pickers below ask it. The booking import takes JPG, PNG and WEBP when the model reads images. The photo goes as an image to every provider (image_url, an Anthropic image block, Ollama's native chat with `images`), capped at 1600px, and skips KDE Itinerary. A scanned PDF, which used to reach the model as empty text, has its first two pages drawn by pdf-parse and sent the same way. The rest is the import as it was: background job, review, the file attached to the booking, the linked cost. On a real qwen3.5:4b a train ticket came back as bare clock times until the router gave the model every type's field meanings, and its price as an object inside a string, which read as 4900; both are handled. Costs gets Scan receipt: a dialog laid out like the booking import's takes one photo, read on the import job queue, then reviewed in the existing expense editor pre-filled with merchant, total, currency, day and lines, the photo waiting in Receipts & Invoices to be uploaded on save. No table, no migration, no category guess, no document-type detection.
This commit is contained in:
@@ -1,4 +1,6 @@
|
||||
import {
|
||||
asLlmVision,
|
||||
LLM_VISION_MODES,
|
||||
adminUserCreateRequestSchema,
|
||||
adminUserUpdateRequestSchema,
|
||||
adminPermissionsRequestSchema,
|
||||
@@ -106,6 +108,13 @@ describe('adminAddonUpdateRequestSchema', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('asLlmVision', () => {
|
||||
it('keeps the three modes and reads anything else as auto', () => {
|
||||
for (const mode of LLM_VISION_MODES) expect(asLlmVision(mode)).toBe(mode);
|
||||
for (const other of [undefined, null, '', 'ON', true, false, 1]) expect(asLlmVision(other)).toBe('auto');
|
||||
});
|
||||
});
|
||||
|
||||
describe('adminCollabFeaturesRequestSchema', () => {
|
||||
it('accepts a single computed key', () => {
|
||||
expect(adminCollabFeaturesRequestSchema.safeParse({ chat: false }).success).toBe(true);
|
||||
|
||||
@@ -80,6 +80,30 @@ export const adminAddonUpdateRequestSchema = z.object({
|
||||
});
|
||||
export type AdminAddonUpdateRequest = z.infer<typeof adminAddonUpdateRequestSchema>;
|
||||
|
||||
/**
|
||||
* Whether the AI Parsing model is handed a photo, stored as `vision` on that
|
||||
* addon's config. `auto` asks a local Ollama server what the model can do and
|
||||
* means no for a cloud provider; `on` and `off` are the admin's word.
|
||||
*/
|
||||
export const LLM_VISION_MODES = ['auto', 'on', 'off'] as const;
|
||||
export type LlmVision = (typeof LLM_VISION_MODES)[number];
|
||||
|
||||
/**
|
||||
* The photos handed to a model that reads images. HEIC is not here: no provider
|
||||
* TREK talks to reads it, and a browser picking from the camera roll hands over
|
||||
* a JPEG anyway.
|
||||
*/
|
||||
export const LLM_PHOTO_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.webp'] as const;
|
||||
|
||||
/** GET /api/llm/capabilities: what the caller's AI model can be handed. */
|
||||
export const llmCapabilitiesResponseSchema = z.object({ images: z.boolean() });
|
||||
export type LlmCapabilitiesResponse = z.infer<typeof llmCapabilitiesResponseSchema>;
|
||||
|
||||
/** A stored or submitted value as a mode; anything else, a missing value included, is `auto`. */
|
||||
export function asLlmVision(value: unknown): LlmVision {
|
||||
return (LLM_VISION_MODES as readonly unknown[]).includes(value) ? (value as LlmVision) : 'auto';
|
||||
}
|
||||
|
||||
// Fully partial — the client PUTs one computed key at a time.
|
||||
export const adminCollabFeaturesRequestSchema = z.object({
|
||||
chat: z.boolean().optional(),
|
||||
|
||||
Reference in New Issue
Block a user