fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)

Squash of PR #658. Discovery records each llama.cpp model's parallel slot
count (from --parallel/-np, or /props total_slots when the launch command
states none), and a Claude launch warns when the context split across
those slots leaves less than 40K tokens per request. An explicit
--parallel beside a /props context records a slot count of 1, so a later
round overwrites an earlier count, and a non-2xx /props is remembered per
launch command while a network error is retried.

Merge-time fix by the maintainer: the architecture-invariants sentence on
the explicit --parallel case now says it records a slot count of 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
DevvynandClaude Opus 5.5 authored and Codeman maintainer committed 2026-10-22 19:32:24 +02:00
1 parent 34cf798b10
commit 33d697afe3
17 files changed
+710 -34

No files matched your search

+16
View File
@@ -61,6 +61,22 @@ export interface CustomModelHost {
* has no entry for simply gets no context-length env override applied — never a guess.
*/
modelContextLengths?: Record<string, number>;
/**
* Discovered number of parallel server slots per model id, keyed like `modelContextLengths`
* and populated on the same loaded-models-only basis. Read from the launch command's
* `--parallel`/`-np` when it states one, otherwise from llama.cpp's own `/props`
* `total_slots` (a llama-server started without `--parallel` picks the count itself, so
* the command cannot say). llama-server splits its context across these slots (by default
* they share one unified KV pool) and Claude Code sends several requests at once (the
* main session plus each parallel subagent), so with N slots each request can only rely
* on roughly `contextLength / N` tokens. Used only by the context warning; a model this
* has no entry for is treated as a single slot, never a guess. The division always uses
* `modelContextLengths` (the launch command's figure, or `/props` only when the command states
* none): `/props` never widens the pool. An explicit `--parallel N` whose context came from
* `/props` is recorded as 1 slot, because that figure is already per slot, and that also
* overwrites any count an earlier round stored.
*/
modelSlotCounts?: Record<string, number>;
/**
* Discovered file size (GB) per model id, keyed by the same strings as `models`.
* Populated during discovery by parsing llama-swap's own `description` field for an