fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)

Squash of PR #658. Discovery records each llama.cpp model's parallel slot
count (from --parallel/-np, or /props total_slots when the launch command
states none), and a Claude launch warns when the context split across
those slots leaves less than 40K tokens per request. An explicit
--parallel beside a /props context records a slot count of 1, so a later
round overwrites an earlier count, and a non-2xx /props is remembered per
launch command while a network error is retried.

Merge-time fix by the maintainer: the architecture-invariants sentence on
the explicit --parallel case now says it records a slot count of 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/api-reference.md

@@ -996,7 +996,10 @@ confirmed? } | { clear: true }` applies (or clears) the session's
     just because a swap is needed at all — only when it would disrupt
     someone else.
   - `{requiresContextWarning: true, modelId, contextLength,
-minSafeContextTokens}` — Claude Code's own fixed per-turn overhead
+minSafeContextTokens, slotCount?}` — `slotCount` is present whenever the model runs more than
+    one parallel llama.cpp slot, and the warning also fires when the context
+    divided across those slots is below the floor even though the whole context
+    is not. Claude Code's own fixed per-turn overhead
     (system prompt + tool schemas) can exceed a small model's entire
     discovered context on its own, before any conversation history exists
     to compact, guaranteeing the very first message fails regardless of

docs/architecture-invariants.md

@@ -647,9 +647,9 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse
 
 ⚠️ **That launch-time check cannot catch a swap caused by a DIFFERENT session's LATER, ordinary use** — confirmed live: a second Codex session picking a different model launched with no warning at all (nothing conflicted at that exact instant), yet it silently evicted the first session's model regardless, since llama-swap has no push notification of its own. `detectCustomModelSwapDisplacements()` (`custom-model-routes.ts`) is a separate periodic sweep (`server.ts`, `CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS` = 20s) that compares each live custom-model session's own `modelId` against what `/running` actually reports loaded, broadcasting a `custom-model:swapped-out` SSE event — shown as a global toast, never tied to the displaced session's own tab, since the whole point is telling the user before they type into it — the first time a mismatch appears, via a caller-owned de-dupe `Set` cleared once that session's own model is loaded and ready again so a later, genuinely new displacement notifies again rather than staying silently un-notified forever after the first one.
 
-⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx <N>` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all.
+⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx <N>` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all. The same rule covers the slot check: `/props` supplies `total_slots` only when `cmd` has no `--parallel`, and it NEVER widens the pool the slots divide (an earlier cut let a larger `/props` `n_ctx` become the pool and skip the warning; the 154112-vs-16384 measurement is exactly why that is wrong). `/props` refreshes llama-swap's last-use time for the model, so `discoverModels` probes it once per launch command (`propsByCmd`, in memory, keyed by host and model id and compared with `/running`'s `cmd`); probing on every 5-minute sweep would keep a model with a `ttl` loaded forever. With an explicit `--parallel N` llama.cpp reports `n_ctx` per slot, so a `/props` context beside it is stored with a slot count of 1: it is not divided a second time, and the 1 overwrites a count an earlier discovery round stored (`applyDiscoveredModels` merges onto what was learned before, so recording no count would keep the stale one).
 
-⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT.
+⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, or when `contextLength / slotCount` (`perSlotContext()`, from `modelSlotCounts`; an unknown count is one slot) is below it, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` (plus `slotCount` when it is above 1) instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT.
 
 ⚠️ **A fresh, isolated `CLAUDE_CONFIG_DIR` looks like a brand-new Claude Code profile and replays its ENTIRE first-run sequence on every launch** — the theme picker, the security-notes screen, the per-project "trust this folder?" dialog, and (running with a bypass-permissions flag) a one-time warning about it, confirmed live, none of which a real, already-onboarded profile shows again. `skipFirstRunPrompts` (claude's entry only, requires `apiKeyTrustFile` since it reuses the same file) pre-seeds that same "already been through this" state: `hasCompletedOnboarding` and this session's own `projects[workingDir].hasTrustDialogAccepted` merge into the same `.claude.json` the API-key trust file already writes to, and `skipDangerousModePermissionPrompt` merges into `settings.json` (a different file, same corrupt-tolerant merge).
 

docs/custom-model-endpoints.md

@@ -97,6 +97,13 @@ llama-swap's own launch command instead (`GET /running`'s `cmd` field —
 hand-written command might use), and only falls back to the `/props` probe
 when `cmd` states no recognizable flag at all.
 
+**Parallel slots are read the same way.** `--parallel N` / `-np N` from `cmd` gives
+the slot count (`modelSlotCounts`); only when `cmd` has none does discovery read
+`total_slots` from `/props`, because a server started without the flag picks the
+number itself. `/props` never widens the context the slot check divides, and it is
+probed once per launch command rather than on every sweep, because llama-swap
+treats the call as use of the model and a `ttl` would otherwise never expire.
+
 **File size is discovered too, when the server states one.** llama-swap
 writes a GB figure into an auto-discovered model's own `description`
 (`"Auto-discovered 16.35 GB - parameters auto-fitted by llama.cpp"`), parsed
@@ -438,11 +445,11 @@ construction, never a hardcoded mode check). If the model's discovered
 context (`modelContextLengths`, from discovery above) is below
 `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
 ~36.4K overhead), the response is `{requiresContextWarning: true, modelId,
-contextLength, minSafeContextTokens}` instead of applying — nothing is
+contextLength, minSafeContextTokens}` (plus `slotCount` when the model runs more than one parallel slot) instead of applying — nothing is
 restarted or created yet. A context length that was never discovered at
 all skips the check entirely (nothing to compare, so it fails open rather
 than warning on every model an endpoint hasn't reported a size for).
-Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
+The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
 
 The Run-menu picker shows this as an in-app modal
 (`#customModelContextWarningModal`, matching the llama-swap conflict

docs/wiki/Custom-Model-Endpoints.md

@@ -150,6 +150,34 @@ relying on auto-fit (`--fit-ctx`), which sizes the context around fitting the bi
 rather than the biggest context — for example adding `-c 65536` to that model's llama-swap
 entry. "Launch anyway" is still there if you want to try regardless.
 
+**A context that is big enough overall can still be too small once it is split across
+slots.** `llama-server` runs several parallel slots by default (four on the build this was measured on) and,
+unless you say otherwise, they all share one pool of context. Claude Code sends more than one
+request at a time: the main session plus every subagent it runs in parallel, and each of those
+requests carries the roughly 40K tokens of system prompt and tools on its own. Four requests
+of 40K tokens do not fit in a shared 131K pool, so the server refuses them with HTTP 500
+`Context size has been exceeded`, even though any single request would have been fine.
+Discovery therefore also records how many slots each loaded model has. It reads an explicit
+`--parallel` or `-np` from the launch command, and otherwise asks the server for
+`total_slots`, because a server started without the flag chooses the number itself. When the
+context divided by the slot count drops below what Claude Code needs, picking that model shows
+the same in-app dialog, this time explaining the shared slots. The division uses the context
+Codeman read from the launch command (`--fit-ctx` or `-c`), never a larger figure from the
+server's `/props`, which has been measured reporting far more than the server really runs
+with. So a default llama-swap model with auto slots shows the dialog, which names the fix and
+lets you launch anyway. The fix is a single slot:
+
+```yaml
+cmd: >-
+  llama-server -m /models/your-model.gguf --parallel 1 ...
+```
+
+With one slot the active conversation gets the whole context, requests from parallel
+subagents queue behind it instead of failing, and Claude Code's repeated prompt prefix stays
+in the cache instead of being processed again in a different slot each time. The alternative
+is a context large enough that every slot can hold 40K tokens. A slot count Codeman could not
+read is treated as one slot, never guessed.
+
 ## Which harnesses actually work
 
 | Harness                                  | Status                                                                                                                                                                                                               |

docs/wiki/Troubleshooting.md

@@ -161,6 +161,20 @@ CODEMAN_INSTANCE=beta CODEMAN_PORT=5000 codeman web
 `codeman web -d` and `codeman service install` both refuse to start a second server on one
 data directory for exactly this reason.
 
+### `Context size has been exceeded` from a local llama.cpp model
+
+Sessions pointed at a llama.cpp or llama-swap endpoint (see [Custom Model
+Endpoints](Custom-Model-Endpoints)) fail with `500 Context size has been exceeded`, usually
+when Claude Code runs subagents, or an agent fails with `Prompt is too long` while the main
+session carries on. The server runs several parallel slots that share one pool of context,
+and every Claude Code request carries about 40K tokens of its own system prompt and tools, so
+a few concurrent requests use the pool up. Start the model with a single slot by adding
+`--parallel 1` to its launch command, or give it a context large enough that each slot holds
+40K tokens. Check the current count with `GET /props` on the server (on llama-swap,
+`GET /props?model=<id>` for a loaded model): `total_slots` is the number of slots. `n_ctx` is the
+shared context only when the slot count is automatic; with an explicit `--parallel N` it is the
+size of each slot.
+
 ## The terminal
 
 ### I cannot scroll back through history