mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-17 19:59:33 +01:00
fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)
Squash of PR #658. Discovery records each llama.cpp model's parallel slot count (from --parallel/-np, or /props total_slots when the launch command states none), and a Claude launch warns when the context split across those slots leaves less than 40K tokens per request. An explicit --parallel beside a /props context records a slot count of 1, so a later round overwrites an earlier count, and a non-2xx /props is remembered per launch command while a network error is retried. Merge-time fix by the maintainer: the architecture-invariants sentence on the explicit --parallel case now says it records a slot count of 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
34cf798b10
commit
33d697afe3
17 files changed
+710
-34
No files matched your search
@@ -61,6 +61,22 @@ export interface CustomModelHost {
|
||||
* has no entry for simply gets no context-length env override applied — never a guess.
|
||||
*/
|
||||
modelContextLengths?: Record<string, number>;
|
||||
/**
|
||||
* Discovered number of parallel server slots per model id, keyed like `modelContextLengths`
|
||||
* and populated on the same loaded-models-only basis. Read from the launch command's
|
||||
* `--parallel`/`-np` when it states one, otherwise from llama.cpp's own `/props`
|
||||
* `total_slots` (a llama-server started without `--parallel` picks the count itself, so
|
||||
* the command cannot say). llama-server splits its context across these slots (by default
|
||||
* they share one unified KV pool) and Claude Code sends several requests at once (the
|
||||
* main session plus each parallel subagent), so with N slots each request can only rely
|
||||
* on roughly `contextLength / N` tokens. Used only by the context warning; a model this
|
||||
* has no entry for is treated as a single slot, never a guess. The division always uses
|
||||
* `modelContextLengths` (the launch command's figure, or `/props` only when the command states
|
||||
* none): `/props` never widens the pool. An explicit `--parallel N` whose context came from
|
||||
* `/props` is recorded as 1 slot, because that figure is already per slot, and that also
|
||||
* overwrites any count an earlier round stored.
|
||||
*/
|
||||
modelSlotCounts?: Record<string, number>;
|
||||
/**
|
||||
* Discovered file size (GB) per model id, keyed by the same strings as `models`.
|
||||
* Populated during discovery by parsing llama-swap's own `description` field for an
|
||||
|
||||
Reference in new issue
Block a user