fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)

Squash of PR #658. Discovery records each llama.cpp model's parallel slot
count (from --parallel/-np, or /props total_slots when the launch command
states none), and a Claude launch warns when the context split across
those slots leaves less than 40K tokens per request. An explicit
--parallel beside a /props context records a slot count of 1, so a later
round overwrites an earlier count, and a non-2xx /props is remembered per
launch command while a network error is retried.

Merge-time fix by the maintainer: the architecture-invariants sentence on
the explicit --parallel case now says it records a slot count of 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
DevvynandClaude Opus 5.5 authored and Codeman maintainer committed 2026-10-22 19:32:24 +02:00
1 parent 34cf798b10
commit 33d697afe3
17 files changed
+710 -34

No files matched your search

+14
View File
@@ -161,6 +161,20 @@ CODEMAN_INSTANCE=beta CODEMAN_PORT=5000 codeman web
`codeman web -d` and `codeman service install` both refuse to start a second server on one
data directory for exactly this reason.
### `Context size has been exceeded` from a local llama.cpp model
Sessions pointed at a llama.cpp or llama-swap endpoint (see [Custom Model
Endpoints](Custom-Model-Endpoints)) fail with `500 Context size has been exceeded`, usually
when Claude Code runs subagents, or an agent fails with `Prompt is too long` while the main
session carries on. The server runs several parallel slots that share one pool of context,
and every Claude Code request carries about 40K tokens of its own system prompt and tools, so
a few concurrent requests use the pool up. Start the model with a single slot by adding
`--parallel 1` to its launch command, or give it a context large enough that each slot holds
40K tokens. Check the current count with `GET /props` on the server (on llama-swap,
`GET /props?model=<id>` for a loaded model): `total_slots` is the number of slots. `n_ctx` is the
shared context only when the slot count is automatic; with an explicit `--parallel N` it is the
size of each slot.
## The terminal
### I cannot scroll back through history