mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-21 21:39:33 +01:00
fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)
Squash of PR #658. Discovery records each llama.cpp model's parallel slot count (from --parallel/-np, or /props total_slots when the launch command states none), and a Claude launch warns when the context split across those slots leaves less than 40K tokens per request. An explicit --parallel beside a /props context records a slot count of 1, so a later round overwrites an earlier count, and a non-2xx /props is remembered per launch command while a network error is retried. Merge-time fix by the maintainer: the architecture-invariants sentence on the explicit --parallel case now says it records a slot count of 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
34cf798b10
commit
33d697afe3
17 files changed
+710
-34
No files matched your search
@@ -161,6 +161,20 @@ CODEMAN_INSTANCE=beta CODEMAN_PORT=5000 codeman web
|
||||
`codeman web -d` and `codeman service install` both refuse to start a second server on one
|
||||
data directory for exactly this reason.
|
||||
|
||||
### `Context size has been exceeded` from a local llama.cpp model
|
||||
|
||||
Sessions pointed at a llama.cpp or llama-swap endpoint (see [Custom Model
|
||||
Endpoints](Custom-Model-Endpoints)) fail with `500 Context size has been exceeded`, usually
|
||||
when Claude Code runs subagents, or an agent fails with `Prompt is too long` while the main
|
||||
session carries on. The server runs several parallel slots that share one pool of context,
|
||||
and every Claude Code request carries about 40K tokens of its own system prompt and tools, so
|
||||
a few concurrent requests use the pool up. Start the model with a single slot by adding
|
||||
`--parallel 1` to its launch command, or give it a context large enough that each slot holds
|
||||
40K tokens. Check the current count with `GET /props` on the server (on llama-swap,
|
||||
`GET /props?model=<id>` for a loaded model): `total_slots` is the number of slots. `n_ctx` is the
|
||||
shared context only when the slot count is automatic; with an explicit `--parallel N` it is the
|
||||
size of each slot.
|
||||
|
||||
## The terminal
|
||||
|
||||
### I cannot scroll back through history
|
||||
|
||||
Reference in new issue
Block a user