mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-22 06:56:35 +00:00
RemEx
b1c05bca83
Whether the AI Parsing model reads images is an admin setting now, and it lands with the two features that send a photo, so it has something to control. The setting is `vision: 'auto' | 'on' | 'off'` on the addon config. On Automatic a local Ollama is asked what the model can do (`/api/show`), and a cloud provider means no; Yes and No are applied as set. The old `multimodal` flag is not read: the admin screens wrote it false on every save, so a stored false meant nothing. The per-user toggle keeps its on/off meaning and is read by the extraction now. The lookup is server-side only: capped read, validated body, never throws, cached per server and model. GET /api/llm/capabilities answers it for the signed-in caller, and both photo pickers below ask it. The booking import takes JPG, PNG and WEBP when the model reads images. The photo goes as an image to every provider (image_url, an Anthropic image block, Ollama's native chat with `images`), capped at 1600px, and skips KDE Itinerary. A scanned PDF, which used to reach the model as empty text, has its first two pages drawn by pdf-parse and sent the same way. The rest is the import as it was: background job, review, the file attached to the booking, the linked cost. On a real qwen3.5:4b a train ticket came back as bare clock times until the router gave the model every type's field meanings, and its price as an object inside a string, which read as 4900; both are handled. Costs gets Scan receipt: a dialog laid out like the booking import's takes one photo, read on the import job queue, then reviewed in the existing expense editor pre-filled with merchant, total, currency, day and lines, the photo waiting in Receipts & Invoices to be uploaded on save. No table, no migration, no category guess, no document-type detection.
40 lines
1.5 KiB
TypeScript
40 lines
1.5 KiB
TypeScript
/** A single binary file (a PDF, or a photo) sent natively to a provider that reads it. */
|
|
export interface LlmExtractionFile {
|
|
mimeType: string;
|
|
data: Buffer;
|
|
}
|
|
|
|
/** Everything a provider client needs to extract reservations from one document. */
|
|
export interface LlmExtractionInput {
|
|
/** System instructions enumerating the schema.org shape (see llm-prompt.ts). */
|
|
prompt: string;
|
|
/** JSON Schema describing `{ reservations: KiReservation[] }`. */
|
|
jsonSchema: object;
|
|
model: string;
|
|
baseUrl?: string;
|
|
apiKey?: string;
|
|
/** Pre-extracted text (text-like files, or text-only-model mode). */
|
|
text?: string;
|
|
/** Native binary: a PDF for Anthropic, a photo for a model that reads images. */
|
|
file?: LlmExtractionFile;
|
|
/** The following pages of a scanned PDF, as images after `file`, which holds the first. */
|
|
pageImages?: LlmExtractionFile[];
|
|
/**
|
|
* The key the answer's array sits under, `jsonSchema` being `{ [rootKey]: [...] }`.
|
|
* Defaults to `reservations`; the receipt read asks for `receipts`.
|
|
*/
|
|
rootKey?: string;
|
|
/** The user-turn instruction. Defaults to the booking import's. */
|
|
userText?: string;
|
|
}
|
|
|
|
/**
|
|
* A provider client turns one document into raw schema.org reservation objects.
|
|
* It returns the parsed `reservations` array (best-effort: `[]` on a malformed or
|
|
* empty response, never throwing for content reasons). The caller validates and
|
|
* maps via the shared kitinerary mapper.
|
|
*/
|
|
export interface LlmExtractionClient {
|
|
extract(input: LlmExtractionInput): Promise<Record<string, unknown>[]>;
|
|
}
|