mirror of
https://github.com/mauriceboe/TREK.git
synced 2026-10-18 21:16:47 +00:00
fix(llm-parse): LLM_MAX_OUTPUT_TOKENS sets the answer cap, a cut-off answer says so
The OpenAI-compatible client always asked for 4096 tokens, and a document whose bookings needed more came back as "no reservations" with nothing that pointed at the cap. LLM_MAX_OUTPUT_TOKENS now sets the cap for every provider; unset keeps 4096 and 8192 so a small local model's context still fits. A finish_reason of "length" is reported like Anthropic's max_tokens stop, naming the limit and the variable that raises it. Closes #2677
This commit is contained in:
Maurice
@@ -52,6 +52,7 @@ services:
|
||||
# - MCP_MAX_SESSION_PER_USER=20 # Max concurrent MCP sessions per user (default: 20)
|
||||
# - KITINERARY_EXTRACTOR_PATH= # Optional. Full path to kitinerary-extractor binary. Auto-detected from PATH and /usr/lib/*/libexec/kf6/ when unset.
|
||||
# - LLM_TIMEOUT_MS=900000 # Optional. Ceiling (ms) for one AI-parsing call before it is abandoned, applied to every provider and to the HTTP client underneath. Generous by default so heavier documents fit; lower it on a cloud provider to fail fast. Must be a whole number of milliseconds between 1 and 2147483647 — anything else stops the server at startup with the reason. (default: 900000)
|
||||
# - LLM_MAX_OUTPUT_TOKENS=8192 # Optional. Longest answer (tokens) one AI-parsing call may ask for. Raise it when a document with many bookings comes back cut off; keep prompt plus cap inside your model's context. Whole number from 256 to 1000000. (default: 4096 for OpenAI-compatible, 8192 for Anthropic)
|
||||
# - TREK_PLUGINS_IGNORE_TREK_RANGE=true # Optional. Lets plugins whose declared TREK version range predates this build, and plugins that declare none, be installed and activated anyway. Every gate warns instead of refusing; the plugin-API version gate and the Discover compatibility verdict stay hard. Off unless set. (default: off)
|
||||
# - UNSPLASH_ACCESS_KEY= # Optional. Unsplash Access Key for trip-cover / place-image search. Without one, TREK uses Unsplash's unauthenticated endpoint, which some datacenter/VPS IPs are blocked from (#1449). Free key at unsplash.com/developers. Can also be set per-admin in Admin > Settings; the env var overrides that.
|
||||
# - TREK_PLACES_ENABLED=false # Optional. Place search (suggestions, full search, category buttons, Road trip search, import geocoding, offline downloads) asks the TREK Places API at https://places.liketrek.com by default, so the server makes outbound requests to that host. Set false to search the way it did before 4.3.0 (OpenStreetMap, or your Google/Amap key) and send nothing there. No switch in the admin panel on purpose: whether searches leave the instance is decided here. (default: true)
|
||||
|
||||
@@ -84,6 +84,7 @@ DEMO_MODE=false # Demo mode: resets data hourly
|
||||
# KITINERARY_EXTRACTOR_PATH= # Full path to the kitinerary-extractor binary. Unset, TREK searches /usr/lib/*/libexec/kf6/ and then PATH; the official image pins it. Set it to run a newer extractor than your distribution packages — the version in use is in the startup log and in GET /api/admin/system-info. (default: auto-detected)
|
||||
|
||||
# LLM_TIMEOUT_MS=900000 # Ceiling (ms) for one AI-parsing call before it is abandoned. One value for every provider, and the underlying HTTP client is held to it too. Generous by default so heavier parsing work fits without a code change; lower it on a cloud provider to fail fast. (default: 900000)
|
||||
# LLM_MAX_OUTPUT_TOKENS=8192 # Longest answer (tokens) one AI-parsing call may ask for. Raise it when a document with many bookings comes back cut off; keep prompt plus cap inside your model's context. (default: 4096 for OpenAI-compatible, 8192 for Anthropic)
|
||||
|
||||
# VAPID_PUBLIC_KEY= # Web Push: the public half of the key pair browsers subscribe with (base64url, 65 bytes). Leave both halves unset and TREK generates a pair on first start and keeps it in the database, so it travels with every backup. Set both to use a pair of your own, e.g. printed by `npx web-push generate-vapid-keys`. Changing the pair later means every device has to turn push on again.
|
||||
# VAPID_PRIVATE_KEY= # Web Push: the private half (base64url, 32 bytes). Only used together with VAPID_PUBLIC_KEY; a pair whose halves do not belong together, or only one half, turns push off with an error in the log until the pair is fixed. A pair stored in the database is not used in its place, and every device stays registered.
|
||||
|
||||
@@ -165,7 +165,7 @@ ALLOWED_ORIGINS, UNSPLASH_ACCESS_KEY, WEBAUTHN_*, TZ, ADMIN_EMAIL,
|
||||
TREK_DB_PRE_MIGRATE_SNAPSHOT(_KEEP) (read when a migration run starts; the
|
||||
legacy-upgrade suites set it per file),
|
||||
ADMIN_PASSWORD, IDEMPOTENCY_TTL_SECONDS, MCP_RATE_LIMIT (request-path check),
|
||||
LLM_TIMEOUT_MS, NOMINATIM_URL, VAPID_* (resolved per use by the Web Push key
|
||||
LLM_TIMEOUT_MS, LLM_MAX_OUTPUT_TOKENS, NOMINATIM_URL, VAPID_* (resolved per use by the Web Push key
|
||||
service, which the notification suites build without the container).
|
||||
|
||||
## Exemptions — raw `process.env` stays
|
||||
|
||||
@@ -295,6 +295,13 @@ export function deriveIntegrations(raw: RawEnv) {
|
||||
* rejects a fractional value.
|
||||
*/
|
||||
llmTimeoutMs: Math.floor(positiveNumberOr(raw.LLM_TIMEOUT_MS, 900_000)),
|
||||
/**
|
||||
* How long one AI-parsing answer may get. Unset leaves each client the cap it
|
||||
* has always sent (4096 to OpenAI-compatible servers, 8192 to Anthropic): a
|
||||
* bigger default would make the server of a small local model refuse every
|
||||
* request whose prompt plus cap overruns its context (#2677).
|
||||
*/
|
||||
llmMaxOutputTokens: positiveIntOr(raw.LLM_MAX_OUTPUT_TOKENS, 0) || undefined,
|
||||
// TRANSIT_API_URL, KITINERARY_EXTRACTOR_PATH and PATH are boot-stable and
|
||||
// read only by Nest providers, so the transitConfig and kitineraryConfig
|
||||
// tokens own them (boot-derive.ts).
|
||||
|
||||
@@ -144,6 +144,8 @@ export const envSchema = z.object({
|
||||
// 2^31-1 makes Node clamp the delay to 1 ms and do the same. Both refuse at
|
||||
// boot rather than degrading into a timeout of zero.
|
||||
LLM_TIMEOUT_MS: integer(1, 2_147_483_647, 'must be a whole number of milliseconds between 1 and 2147483647'),
|
||||
// Below 256 not even one reservation fits, so a value that small is a typo, not a choice.
|
||||
LLM_MAX_OUTPUT_TOKENS: integer(256, 1_000_000, 'must be a whole number of tokens between 256 and 1000000'),
|
||||
KITINERARY_EXTRACTOR_PATH: anyString,
|
||||
// The OS search path. Not configuration anybody sets for TREK — it is here so
|
||||
// the kitinerary probe can resolve its binary to an absolute path itself
|
||||
|
||||
@@ -5,7 +5,8 @@ import type { LlmExtractionClient, LlmExtractionInput } from '../llm-provider.in
|
||||
import { isContextOverflow, LlmContextOverflowError } from './context-overflow';
|
||||
import { UnreadableLlmResponse } from './openai-compatible.client';
|
||||
|
||||
const MAX_TOKENS = 8192;
|
||||
/** The answer cap when LLM_MAX_OUTPUT_TOKENS is unset. */
|
||||
const DEFAULT_MAX_TOKENS = 8192;
|
||||
const ANTHROPIC_VERSION = '2023-06-01';
|
||||
|
||||
/**
|
||||
@@ -43,9 +44,10 @@ export class AnthropicClient implements LlmExtractionClient {
|
||||
text: input.text ? `${userText}\n\n${input.text}` : userText,
|
||||
});
|
||||
|
||||
const maxTokens = readEnv().integrations.llmMaxOutputTokens ?? DEFAULT_MAX_TOKENS;
|
||||
const body = {
|
||||
model: input.model,
|
||||
max_tokens: MAX_TOKENS,
|
||||
max_tokens: maxTokens,
|
||||
system: input.prompt,
|
||||
tools: [
|
||||
{
|
||||
@@ -101,7 +103,7 @@ export class AnthropicClient implements LlmExtractionClient {
|
||||
// log: the #2375 symptom, on the provider the dropdown offers first.
|
||||
if (data.stop_reason === 'max_tokens') {
|
||||
throw new UnreadableLlmResponse(
|
||||
`the answer was cut off at the ${MAX_TOKENS}-token limit — the document is too long to extract in one pass`,
|
||||
`the answer was cut off at the ${maxTokens}-token limit — raise LLM_MAX_OUTPUT_TOKENS or import fewer bookings per document`,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -5,7 +5,8 @@ import type { LlmExtractionClient, LlmExtractionInput } from '../llm-provider.in
|
||||
import { isContextOverflow, LlmContextOverflowError } from './context-overflow';
|
||||
import { isNuExtractModel, buildNuExtractUserText, nuExtractToKiReservations } from './nuextract';
|
||||
|
||||
const MAX_TOKENS = 4096;
|
||||
/** The answer cap when LLM_MAX_OUTPUT_TOKENS is unset. */
|
||||
const DEFAULT_MAX_TOKENS = 4096;
|
||||
|
||||
/**
|
||||
* A response the client could not read at all: prose where JSON was asked for,
|
||||
@@ -106,10 +107,11 @@ export class OpenAiCompatibleClient implements LlmExtractionClient {
|
||||
// local server (Ollama/vLLM/llama.cpp), but newer OpenAI models reject it
|
||||
// with a 400 and demand `max_completion_tokens`. Start with the broadly
|
||||
// supported spelling and swap on that specific rejection (#1760).
|
||||
const maxTokens = readEnv().integrations.llmMaxOutputTokens ?? DEFAULT_MAX_TOKENS;
|
||||
const buildBody = (shape: RequestShape) => {
|
||||
const baseBody = {
|
||||
model: input.model,
|
||||
[shape.tokenParam]: MAX_TOKENS,
|
||||
[shape.tokenParam]: maxTokens,
|
||||
// Extraction is a deterministic task — Ollama defaults to 0.7, which makes
|
||||
// small models (NuExtract) drop fields or return empty. Pin to 0, and only
|
||||
// leave it out once a server has explicitly rejected the parameter (#2262).
|
||||
@@ -193,8 +195,16 @@ export class OpenAiCompatibleClient implements LlmExtractionClient {
|
||||
console.debug(`[DEBUG] LLM answered with response_format=${sentFormat(shape, nuextract)}`);
|
||||
|
||||
const data = (await res.json()) as {
|
||||
choices?: { message?: { content?: string } }[];
|
||||
choices?: { message?: { content?: string }; finish_reason?: string }[];
|
||||
};
|
||||
// A run that hits the cap stops mid-list, and the fragment either reads as
|
||||
// nothing or not at all: "no reservations" for a document full of them, with
|
||||
// no hint that a setting would fix it (#2677). Same rule as the Anthropic client.
|
||||
if (data.choices?.[0]?.finish_reason === 'length') {
|
||||
throw new UnreadableLlmResponse(
|
||||
`the answer was cut off at the ${maxTokens}-token limit — raise LLM_MAX_OUTPUT_TOKENS or import fewer bookings per document`,
|
||||
);
|
||||
}
|
||||
const content = data.choices?.[0]?.message?.content;
|
||||
return nuextract ? parseNuExtract(content) : parseReservations(content, rootKey);
|
||||
}
|
||||
|
||||
@@ -248,6 +248,12 @@ describe('deriveIntegrations', () => {
|
||||
expect(deriveIntegrations({ LLM_TIMEOUT_MS: '-1' }).llmTimeoutMs).toBe(900_000);
|
||||
expect(deriveIntegrations({ LLM_TIMEOUT_MS: '60000.5' }).llmTimeoutMs).toBe(60_000);
|
||||
});
|
||||
|
||||
it('leaves the LLM answer cap to each client unless one is set (#2677)', () => {
|
||||
expect(deriveIntegrations({}).llmMaxOutputTokens).toBeUndefined();
|
||||
expect(deriveIntegrations({ LLM_MAX_OUTPUT_TOKENS: '' }).llmMaxOutputTokens).toBeUndefined();
|
||||
expect(deriveIntegrations({ LLM_MAX_OUTPUT_TOKENS: '16384' }).llmMaxOutputTokens).toBe(16384);
|
||||
});
|
||||
});
|
||||
|
||||
describe('deriveTransit (transitConfig token)', () => {
|
||||
|
||||
@@ -74,6 +74,14 @@ describe('validateEnvAtBoot', () => {
|
||||
expect(() => validateEnvAtBoot({ LLM_TIMEOUT_MS: '' })).not.toThrow();
|
||||
});
|
||||
|
||||
it('refuses an LLM answer cap that is no whole token count or too small to hold a booking (#2677)', () => {
|
||||
expect(() => validateEnvAtBoot({ LLM_MAX_OUTPUT_TOKENS: '8192.5' })).toThrow();
|
||||
expect(() => validateEnvAtBoot({ LLM_MAX_OUTPUT_TOKENS: '100' })).toThrow();
|
||||
expect(() => validateEnvAtBoot({ LLM_MAX_OUTPUT_TOKENS: 'lots' })).toThrow();
|
||||
expect(() => validateEnvAtBoot({ LLM_MAX_OUTPUT_TOKENS: '16384' })).not.toThrow();
|
||||
expect(() => validateEnvAtBoot({ LLM_MAX_OUTPUT_TOKENS: '' })).not.toThrow();
|
||||
});
|
||||
|
||||
it('refuses an ALLOW_LINK_LOCAL_IPS entry that is not a listable link-local address', () => {
|
||||
// A metadata address would be dropped silently at runtime; saying so at boot
|
||||
// is the difference between an OIDC login that works and a puzzling one.
|
||||
|
||||
@@ -129,6 +129,29 @@ describe('OpenAiCompatibleClient', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('caps the answer at 4096 tokens unless LLM_MAX_OUTPUT_TOKENS says otherwise (#2677)', async () => {
|
||||
const fetchFn = mockFetch(() => jsonResponse({ choices: [{ message: { content: '{"reservations":[]}' } }] }));
|
||||
await new OpenAiCompatibleClient().extract(baseInput);
|
||||
expect(JSON.parse(fetchFn.mock.calls[0][1].body as string).max_tokens).toBe(4096);
|
||||
|
||||
vi.stubEnv('LLM_MAX_OUTPUT_TOKENS', '16384');
|
||||
try {
|
||||
await new OpenAiCompatibleClient().extract(baseInput);
|
||||
expect(JSON.parse(fetchFn.mock.calls[1][1].body as string).max_tokens).toBe(16384);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
|
||||
it('throws when the answer was cut off at the token cap instead of reading it as empty (#2677)', async () => {
|
||||
mockFetch(() =>
|
||||
jsonResponse({ choices: [{ message: { content: '{"reservations":[{"@type":"Flig' }, finish_reason: 'length' }] }),
|
||||
);
|
||||
await expect(new OpenAiCompatibleClient().extract(baseInput)).rejects.toThrow(
|
||||
/cut off at the 4096-token limit — raise LLM_MAX_OUTPUT_TOKENS/,
|
||||
);
|
||||
});
|
||||
|
||||
it('reads a single reservation the model answered on its own', async () => {
|
||||
mockFetch(() =>
|
||||
jsonResponse({ choices: [{ message: { content: '{"@type":"LodgingReservation","reservationNumber":"733"}' } }] }),
|
||||
@@ -695,6 +718,22 @@ describe('AnthropicClient', () => {
|
||||
await expect(new AnthropicClient().extract(baseInput)).rejects.toThrow(/cut off at the 8192-token limit/);
|
||||
});
|
||||
|
||||
it('sends the cap LLM_MAX_OUTPUT_TOKENS sets and names it when the answer hits it (#2677)', async () => {
|
||||
const fetchFn = mockFetch(() =>
|
||||
jsonResponse({
|
||||
stop_reason: 'max_tokens',
|
||||
content: [{ type: 'tool_use', name: 'emit_reservations', input: {} }],
|
||||
}),
|
||||
);
|
||||
vi.stubEnv('LLM_MAX_OUTPUT_TOKENS', '20000');
|
||||
try {
|
||||
await expect(new AnthropicClient().extract(baseInput)).rejects.toThrow(/cut off at the 20000-token limit/);
|
||||
expect(JSON.parse(fetchFn.mock.calls[0][1].body as string).max_tokens).toBe(20000);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
});
|
||||
|
||||
it('throws when the forced tool was never called (#2375)', async () => {
|
||||
mockFetch(() =>
|
||||
jsonResponse({
|
||||
|
||||
@@ -114,8 +114,9 @@ Photographing a receipt for the Costs tab is covered in [Budget-Tracking](Budget
|
||||
|
||||
## Good to know
|
||||
|
||||
- **Nothing to migrate**, and everything is set up in the UI. The only environment variable the addon reads is `LLM_TIMEOUT_MS` (see [Environment-Variables](Environment-Variables)).
|
||||
- **Nothing to migrate**, and everything is set up in the UI. The only environment variables the addon reads are `LLM_TIMEOUT_MS` and `LLM_MAX_OUTPUT_TOKENS` (see [Environment-Variables](Environment-Variables)).
|
||||
- **Local models can be slow.** On a CPU-only host one booking can take from tens of seconds to a couple of minutes. TREK gives a model 15 minutes per document by default; `LLM_TIMEOUT_MS` raises or lowers that. Uploads are parsed **one at a time** per user, so several files wait in line rather than run in parallel.
|
||||
- **A long booking list can outgrow the answer.** A model may write at most 4096 tokens per document (8192 on Anthropic). When a document holds more than that, the import says the answer was cut off; `LLM_MAX_OUTPUT_TOKENS` raises the cap, as far as your model's context allows.
|
||||
- **Parse jobs are kept for about 10 minutes** after they finish. Start the review within that time.
|
||||
- **Privacy**: with the Local provider nothing leaves your network. With OpenAI or Anthropic, the document's text (for Anthropic the PDF itself, and for either a photo) is sent to that provider.
|
||||
- **API keys are never sent back in plain text.** They are encrypted at rest and only ever shown masked.
|
||||
|
||||
@@ -340,7 +340,7 @@ both come back empty. `LOG_LEVEL=debug` additionally passes the extractor's raw
|
||||
the Reservations panel is hidden only when **neither** extractor is available — an instance running the AI Parsing
|
||||
addon still offers it with `bookingImport: false`.
|
||||
|
||||
Booking import can also fall back to an AI model for documents KDE Itinerary can't read. That feature (the **AI Parsing** addon) is configured in the UI; the only environment variable it reads is `LLM_TIMEOUT_MS` below. See [AI-Booking-Import](AI-Booking-Import).
|
||||
Booking import can also fall back to an AI model for documents KDE Itinerary can't read. That feature (the **AI Parsing** addon) is configured in the UI; the only environment variables it reads are `LLM_TIMEOUT_MS` and `LLM_MAX_OUTPUT_TOKENS` below. See [AI-Booking-Import](AI-Booking-Import).
|
||||
|
||||
---
|
||||
|
||||
@@ -499,6 +499,7 @@ next key rotation or admin reset would quietly switch the file back to WAL.
|
||||
| `OVERPASS_URL` | Custom [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API) endpoint(s), comma-separated, for the map's category buttons, the Road trip search along the drive and the `search_pois` MCP tool whenever the TREK Places API has no answer or is switched off. When set it **replaces** the bundled public mirrors: point it at an internal or self-hosted Overpass instance when the public mirrors are unreachable from your network (e.g. firewalled/locked-down egress in a Kubernetes cluster). Entries that aren't valid `http(s)` URLs are ignored. If you don't run your own Overpass but the public mirrors throttle TREK, first make sure `APP_URL` (or `ALLOWED_ORIGINS`) is set: that alone gives outbound Overpass/Nominatim requests a unique User-Agent, which the public mirrors rate-limit far less. One request is not covered: the tag lookup behind an OpenStreetMap place's details (website, phone, Wikidata) still goes to `overpass-api.de` whatever this is set to, and where that host is unreachable those details fall back to the tags Nominatim returns. | bundled public mirrors |
|
||||
| `OVERPASS_TIMEOUT_MS` | Per-endpoint timeout (in milliseconds) for Overpass POI requests. Endpoints race in parallel and one that hasn't answered within this window is abandoned so a faster mirror can win. The query itself gives Overpass 20 seconds of work, so a lower value hangs up on answers that were still coming; that is why the default rose from `12000` in 4.3.0. Drop any lower value you set by hand, and raise it only for a slow self-hosted Overpass instance. Invalid values abort startup. | `25000` |
|
||||
| `LLM_TIMEOUT_MS` | How long (in milliseconds) one AI-parsing call may take before it is abandoned. One ceiling for every provider, applied to the abort signal and to the underlying HTTP client alike. The default is generous so heavier parsing work fits without a code change; lower it if you use a cloud provider and would rather fail fast. Invalid values abort startup. | `900000` (15 min) |
|
||||
| `LLM_MAX_OUTPUT_TOKENS` | The longest answer (in tokens) one AI-parsing call may ask the model for, sent to every provider. A document with many bookings needs a long answer; when the model hits the cap the import says so and names this variable. Raise it for such documents, but keep prompt plus cap inside the context your model has, or the server refuses the request. Unset keeps each provider's own cap. A whole number from 256 to 1000000, anything else aborts startup. | `4096` (OpenAI-compatible), `8192` (Anthropic) |
|
||||
|
||||
> **Routing is not set here.** The routing engines behind the planner and the [Road-Trip](Road-Trip) addon (OSRM for routes, Valhalla for avoiding toll roads, motorways and ferries) have no environment variable. An admin sets them under **Admin → User Defaults** as **Own routing engine** and **Own Valhalla instance**, and they take effect after a restart. `OVERPASS_URL` and `OVERPASS_TIMEOUT_MS` above also apply to the road trip's search along the route.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user