mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-18 20:29:33 +01:00
fix(custom-model): warn when parallel llama.cpp slots leave too little context per request (#658)
Squash of PR #658. Discovery records each llama.cpp model's parallel slot count (from --parallel/-np, or /props total_slots when the launch command states none), and a Claude launch warns when the context split across those slots leaves less than 40K tokens per request. An explicit --parallel beside a /props context records a slot count of 1, so a later round overwrites an earlier count, and a non-2xx /props is remembered per launch command while a network error is retried. Merge-time fix by the maintainer: the architecture-invariants sentence on the explicit --parallel case now says it records a slot count of 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
34cf798b10
commit
33d697afe3
17 files changed
+710
-34
No files matched your search
@@ -0,0 +1,5 @@
|
||||
---
|
||||
'aicodeman': patch
|
||||
---
|
||||
|
||||
A custom llama.cpp model whose parallel slots share a context pool too small for concurrent Claude Code requests now gets an in-app warning. Four simultaneous 40K-token requests against a 4-slot ~131K pool all failed with "Context size has been exceeded" while the old total-context check stayed silent. Discovery records each model's slot count (from `--parallel`/`-np`, or `/props` `total_slots` when the launch command states none) and the warning names the fix: `--parallel 1`, or a context large enough for every slot.
|
||||
@@ -181,7 +181,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
|
||||
|
||||
**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`): points a session at a user-configured OpenAI-compatible endpoint (store `custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`, 0600). `authStyle` is `bearer` or `api-key`, never both headers (hangs the server). The per-CLI redirect is registry data, `capabilities.customModelInjection` (`env` / `configContentEnv` / `configDir` / `unsupported`), computed by the pure `custom-model-injection.ts`; ⚠️ `configDir` writes an isolated per-session config, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config. ⚠️ Claude applies via `Session.restartCli()` (a relaunch in the existing pane, so the conversation is pinned as `resumeSessionId`), the rest one-shot at launch; retired env keys must also be `setenv -u`'d (`_pendingEnvUnsets`), since tmux env survives `respawn-pane`. ⚠️ Remote/Docker sessions are refused (400). ⚠️ Persist only KEYS (`__customModel`), never values (they carry the API key). ⚠️ Every redirectable var must be in that CLI's `privilegedEnvKeys`, and `ANTHROPIC_*` stays out of claude's `allowedPrefixes`. ⚠️ The Run-menu picker builds entries from `window.__codemanCustomModelClis` (escaped via `escapeScriptJson()`), never a hardcoded CLI id list, and launches through `run()` via a temporary `_runMode` swap, never `setRunMode()`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
**MCP server sync** (opt-in, `mcpSyncEnabled`, SYNCED, default OFF; `src/mcp-sync.ts`, `GET`/`POST /api/mcp-sync`): copies each installed, enabled CLI's user-level MCP servers into the others. It is the ONE subsystem that writes another CLI's REAL user config (`~/.claude.json`, `~/.codex/config.toml`, `~/.gemini/*`, opencode's), which is why it is opt-in and admin-only in multi-user mode (both verbs 403 for a non-admin, and the Settings group is hidden for them). Where each CLI keeps the file is registry data, `capabilities.mcpConfig` (`{ path, format, relocation? }`), never a branch on the id. ⚠️ Tools that sync but are not run modes (GitHub Copilot CLI, the first) are plain data in `MCP_SYNC_ONLY_TOOLS` (`src/mcp-sync-targets.ts`), take part only when installed or already configured, and their companion `settings.json` (`disabledMcpServers`) is read, never written; a test must clear their relocation vars too. ⚠️ ADDITIVE only: a name already defined, in any shape, is never edited or removed (a different same-name definition is a reported conflict), and a server switched off in its own CLI is never copied. ⚠️ Never write a file that did not parse; re-parse the NEW text and require every added server to read back before the tmp+rename (written through a symlink, previous file kept as `<file>.codeman-bak`, one apply at a time, else 409). ⚠️ A file that receives copied `env`/`headers` (secrets) is left `0600`, and so is the backup. ⚠️ Responses carry server NAMES only, never env values, headers or file text: a parse failure is reported by line and column (`describeMcpSyncError`), never the parser's own message (smol-toml and V8 both quote source). ⚠️ `mcpConfig.relocation` names the env var the CLI reads to move its file (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `XDG_CONFIG_HOME`, `GEMINI_CLI_HOME`, and `COPILOT_HOME` for Copilot), resolved from the SERVER env at call time; a relative value reports the target `skipped`, never a guessed write, and a per-session `envOverrides` relocation is not followed. Tests must pass `home` (which drops the `process.env` default) or clear those vars first. → `docs/cli-registry.md` (MCP server sync), `docs/api-reference.md`, `docs/wiki/Settings-Reference.md`
|
||||
|
||||
|
||||
@@ -996,7 +996,10 @@ confirmed? } | { clear: true }` applies (or clears) the session's
|
||||
just because a swap is needed at all — only when it would disrupt
|
||||
someone else.
|
||||
- `{requiresContextWarning: true, modelId, contextLength,
|
||||
minSafeContextTokens}` — Claude Code's own fixed per-turn overhead
|
||||
minSafeContextTokens, slotCount?}` — `slotCount` is present whenever the model runs more than
|
||||
one parallel llama.cpp slot, and the warning also fires when the context
|
||||
divided across those slots is below the floor even though the whole context
|
||||
is not. Claude Code's own fixed per-turn overhead
|
||||
(system prompt + tool schemas) can exceed a small model's entire
|
||||
discovered context on its own, before any conversation history exists
|
||||
to compact, guaranteeing the very first message fails regardless of
|
||||
|
||||
@@ -647,9 +647,9 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse
|
||||
|
||||
⚠️ **That launch-time check cannot catch a swap caused by a DIFFERENT session's LATER, ordinary use** — confirmed live: a second Codex session picking a different model launched with no warning at all (nothing conflicted at that exact instant), yet it silently evicted the first session's model regardless, since llama-swap has no push notification of its own. `detectCustomModelSwapDisplacements()` (`custom-model-routes.ts`) is a separate periodic sweep (`server.ts`, `CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS` = 20s) that compares each live custom-model session's own `modelId` against what `/running` actually reports loaded, broadcasting a `custom-model:swapped-out` SSE event — shown as a global toast, never tied to the displaced session's own tab, since the whole point is telling the user before they type into it — the first time a mismatch appears, via a caller-owned de-dupe `Set` cleared once that session's own model is loaded and ready again so a later, genuinely new displacement notifies again rather than staying silently un-notified forever after the first one.
|
||||
|
||||
⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx <N>` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all.
|
||||
⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx <N>` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all. The same rule covers the slot check: `/props` supplies `total_slots` only when `cmd` has no `--parallel`, and it NEVER widens the pool the slots divide (an earlier cut let a larger `/props` `n_ctx` become the pool and skip the warning; the 154112-vs-16384 measurement is exactly why that is wrong). `/props` refreshes llama-swap's last-use time for the model, so `discoverModels` probes it once per launch command (`propsByCmd`, in memory, keyed by host and model id and compared with `/running`'s `cmd`); probing on every 5-minute sweep would keep a model with a `ttl` loaded forever. With an explicit `--parallel N` llama.cpp reports `n_ctx` per slot, so a `/props` context beside it is stored with a slot count of 1: it is not divided a second time, and the 1 overwrites a count an earlier discovery round stored (`applyDiscoveredModels` merges onto what was learned before, so recording no count would keep the stale one).
|
||||
|
||||
⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT.
|
||||
⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, or when `contextLength / slotCount` (`perSlotContext()`, from `modelSlotCounts`; an unknown count is one slot) is below it, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` (plus `slotCount` when it is above 1) instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT.
|
||||
|
||||
⚠️ **A fresh, isolated `CLAUDE_CONFIG_DIR` looks like a brand-new Claude Code profile and replays its ENTIRE first-run sequence on every launch** — the theme picker, the security-notes screen, the per-project "trust this folder?" dialog, and (running with a bypass-permissions flag) a one-time warning about it, confirmed live, none of which a real, already-onboarded profile shows again. `skipFirstRunPrompts` (claude's entry only, requires `apiKeyTrustFile` since it reuses the same file) pre-seeds that same "already been through this" state: `hasCompletedOnboarding` and this session's own `projects[workingDir].hasTrustDialogAccepted` merge into the same `.claude.json` the API-key trust file already writes to, and `skipDangerousModePermissionPrompt` merges into `settings.json` (a different file, same corrupt-tolerant merge).
|
||||
|
||||
|
||||
@@ -97,6 +97,13 @@ llama-swap's own launch command instead (`GET /running`'s `cmd` field —
|
||||
hand-written command might use), and only falls back to the `/props` probe
|
||||
when `cmd` states no recognizable flag at all.
|
||||
|
||||
**Parallel slots are read the same way.** `--parallel N` / `-np N` from `cmd` gives
|
||||
the slot count (`modelSlotCounts`); only when `cmd` has none does discovery read
|
||||
`total_slots` from `/props`, because a server started without the flag picks the
|
||||
number itself. `/props` never widens the context the slot check divides, and it is
|
||||
probed once per launch command rather than on every sweep, because llama-swap
|
||||
treats the call as use of the model and a `ttl` would otherwise never expire.
|
||||
|
||||
**File size is discovered too, when the server states one.** llama-swap
|
||||
writes a GB figure into an auto-discovered model's own `description`
|
||||
(`"Auto-discovered 16.35 GB - parameters auto-fitted by llama.cpp"`), parsed
|
||||
@@ -438,11 +445,11 @@ construction, never a hardcoded mode check). If the model's discovered
|
||||
context (`modelContextLengths`, from discovery above) is below
|
||||
`CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
|
||||
~36.4K overhead), the response is `{requiresContextWarning: true, modelId,
|
||||
contextLength, minSafeContextTokens}` instead of applying — nothing is
|
||||
contextLength, minSafeContextTokens}` (plus `slotCount` when the model runs more than one parallel slot) instead of applying — nothing is
|
||||
restarted or created yet. A context length that was never discovered at
|
||||
all skips the check entirely (nothing to compare, so it fails open rather
|
||||
than warning on every model an endpoint hasn't reported a size for).
|
||||
Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
|
||||
The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
|
||||
|
||||
The Run-menu picker shows this as an in-app modal
|
||||
(`#customModelContextWarningModal`, matching the llama-swap conflict
|
||||
|
||||
@@ -150,6 +150,34 @@ relying on auto-fit (`--fit-ctx`), which sizes the context around fitting the bi
|
||||
rather than the biggest context — for example adding `-c 65536` to that model's llama-swap
|
||||
entry. "Launch anyway" is still there if you want to try regardless.
|
||||
|
||||
**A context that is big enough overall can still be too small once it is split across
|
||||
slots.** `llama-server` runs several parallel slots by default (four on the build this was measured on) and,
|
||||
unless you say otherwise, they all share one pool of context. Claude Code sends more than one
|
||||
request at a time: the main session plus every subagent it runs in parallel, and each of those
|
||||
requests carries the roughly 40K tokens of system prompt and tools on its own. Four requests
|
||||
of 40K tokens do not fit in a shared 131K pool, so the server refuses them with HTTP 500
|
||||
`Context size has been exceeded`, even though any single request would have been fine.
|
||||
Discovery therefore also records how many slots each loaded model has. It reads an explicit
|
||||
`--parallel` or `-np` from the launch command, and otherwise asks the server for
|
||||
`total_slots`, because a server started without the flag chooses the number itself. When the
|
||||
context divided by the slot count drops below what Claude Code needs, picking that model shows
|
||||
the same in-app dialog, this time explaining the shared slots. The division uses the context
|
||||
Codeman read from the launch command (`--fit-ctx` or `-c`), never a larger figure from the
|
||||
server's `/props`, which has been measured reporting far more than the server really runs
|
||||
with. So a default llama-swap model with auto slots shows the dialog, which names the fix and
|
||||
lets you launch anyway. The fix is a single slot:
|
||||
|
||||
```yaml
|
||||
cmd: >-
|
||||
llama-server -m /models/your-model.gguf --parallel 1 ...
|
||||
```
|
||||
|
||||
With one slot the active conversation gets the whole context, requests from parallel
|
||||
subagents queue behind it instead of failing, and Claude Code's repeated prompt prefix stays
|
||||
in the cache instead of being processed again in a different slot each time. The alternative
|
||||
is a context large enough that every slot can hold 40K tokens. A slot count Codeman could not
|
||||
read is treated as one slot, never guessed.
|
||||
|
||||
## Which harnesses actually work
|
||||
|
||||
| Harness | Status |
|
||||
|
||||
@@ -161,6 +161,20 @@ CODEMAN_INSTANCE=beta CODEMAN_PORT=5000 codeman web
|
||||
`codeman web -d` and `codeman service install` both refuse to start a second server on one
|
||||
data directory for exactly this reason.
|
||||
|
||||
### `Context size has been exceeded` from a local llama.cpp model
|
||||
|
||||
Sessions pointed at a llama.cpp or llama-swap endpoint (see [Custom Model
|
||||
Endpoints](Custom-Model-Endpoints)) fail with `500 Context size has been exceeded`, usually
|
||||
when Claude Code runs subagents, or an agent fails with `Prompt is too long` while the main
|
||||
session carries on. The server runs several parallel slots that share one pool of context,
|
||||
and every Claude Code request carries about 40K tokens of its own system prompt and tools, so
|
||||
a few concurrent requests use the pool up. Start the model with a single slot by adding
|
||||
`--parallel 1` to its launch command, or give it a context large enough that each slot holds
|
||||
40K tokens. Check the current count with `GET /props` on the server (on llama-swap,
|
||||
`GET /props?model=<id>` for a loaded model): `total_slots` is the number of slots. `n_ctx` is the
|
||||
shared context only when the slot count is automatic; with an explicit `--parallel N` it is the
|
||||
size of each slot.
|
||||
|
||||
## The terminal
|
||||
|
||||
### I cannot scroll back through history
|
||||
|
||||
@@ -61,6 +61,22 @@ export interface CustomModelHost {
|
||||
* has no entry for simply gets no context-length env override applied — never a guess.
|
||||
*/
|
||||
modelContextLengths?: Record<string, number>;
|
||||
/**
|
||||
* Discovered number of parallel server slots per model id, keyed like `modelContextLengths`
|
||||
* and populated on the same loaded-models-only basis. Read from the launch command's
|
||||
* `--parallel`/`-np` when it states one, otherwise from llama.cpp's own `/props`
|
||||
* `total_slots` (a llama-server started without `--parallel` picks the count itself, so
|
||||
* the command cannot say). llama-server splits its context across these slots (by default
|
||||
* they share one unified KV pool) and Claude Code sends several requests at once (the
|
||||
* main session plus each parallel subagent), so with N slots each request can only rely
|
||||
* on roughly `contextLength / N` tokens. Used only by the context warning; a model this
|
||||
* has no entry for is treated as a single slot, never a guess. The division always uses
|
||||
* `modelContextLengths` (the launch command's figure, or `/props` only when the command states
|
||||
* none): `/props` never widens the pool. An explicit `--parallel N` whose context came from
|
||||
* `/props` is recorded as 1 slot, because that figure is already per slot, and that also
|
||||
* overwrites any count an earlier round stored.
|
||||
*/
|
||||
modelSlotCounts?: Record<string, number>;
|
||||
/**
|
||||
* Discovered file size (GB) per model id, keyed by the same strings as `models`.
|
||||
* Populated during discovery by parsing llama-swap's own `description` field for an
|
||||
|
||||
@@ -1114,11 +1114,33 @@ Object.assign(CodemanApp.prototype, {
|
||||
* history yet for compaction to trim. Same promise-based pattern as
|
||||
* `_confirmModelSwap`; `_resolveContextWarningConfirm` settles it.
|
||||
*/
|
||||
_confirmContextWarning(modelId, contextLength, minSafeContextTokens) {
|
||||
_confirmContextWarning(modelId, contextLength, minSafeContextTokens, slotCount) {
|
||||
const modal = document.getElementById('customModelContextWarningModal');
|
||||
const messageEl = document.getElementById('customModelContextWarningMessage');
|
||||
if (messageEl) {
|
||||
const known = typeof contextLength === 'number';
|
||||
const slots = Number.isFinite(slotCount) && slotCount > 1 ? Math.floor(slotCount) : 1;
|
||||
// Enough total context, but split across parallel server slots: llama-server shares its
|
||||
// context between --parallel slots, and this CLI sends concurrent requests (the main
|
||||
// session plus each parallel subagent), so several 40k-token requests spend the shared
|
||||
// pool and the server answers HTTP 500 "Context size has been exceeded".
|
||||
if (known && slots > 1 && contextLength >= minSafeContextTokens) {
|
||||
messageEl.textContent =
|
||||
`${modelId}'s ${contextLength.toLocaleString()} tokens of context are split across ${slots} parallel ` +
|
||||
`server slots, about ${Math.floor(contextLength / slots).toLocaleString()} each. This CLI ` +
|
||||
`needs roughly ${minSafeContextTokens.toLocaleString()}+ tokens just for its own system prompt and tools on ` +
|
||||
`every request, and it sends several requests at once (the main session plus each parallel subagent). ` +
|
||||
`Concurrent requests can use up the shared context, and the server then fails them with ` +
|
||||
`"Context size has been exceeded".\n\n` +
|
||||
`To fix this, start the model with a single slot: add "--parallel 1" to its llama-swap cmd. The active ` +
|
||||
`conversation then gets the whole context, other requests queue instead of failing, and the CLI's ` +
|
||||
`repeated prompt prefix stays cached. Alternatively give the model a larger context (raise --fit-ctx or ` +
|
||||
`--ctx-size) so every slot has at least ${minSafeContextTokens.toLocaleString()} tokens.`;
|
||||
modal?.classList.add('active');
|
||||
return new Promise((resolve) => {
|
||||
this._resolveContextWarningConfirmPromise = resolve;
|
||||
});
|
||||
}
|
||||
messageEl.textContent =
|
||||
`${modelId} is configured with ` +
|
||||
(known ? `only ${contextLength.toLocaleString()} tokens of` : 'an unknown (too small)') +
|
||||
@@ -1129,7 +1151,11 @@ Object.assign(CodemanApp.prototype, {
|
||||
`instead of relying on auto-fit (--fit-ctx), which optimizes for the biggest MODEL that fits, not the ` +
|
||||
`biggest CONTEXT — e.g. add "-c 65536" (or as large a --ctx-size as your hardware holds) to its llama-swap ` +
|
||||
`config entry. A smaller model at a much larger explicit context often fits in the same VRAM a bigger ` +
|
||||
`model's auto-fit context gets shrunk to make room for.`;
|
||||
`model's auto-fit context gets shrunk to make room for.` +
|
||||
(slots > 1
|
||||
? `\n\nThis server also runs ${slots} parallel slots that share that context; add "--parallel 1" to ` +
|
||||
`the same cmd so a single conversation gets all of it.`
|
||||
: '');
|
||||
}
|
||||
modal?.classList.add('active');
|
||||
return new Promise((resolve) => {
|
||||
@@ -1254,8 +1280,13 @@ Object.assign(CodemanApp.prototype, {
|
||||
let answered = {};
|
||||
let data = await post(bodyObj);
|
||||
if (data?.data?.requiresContextWarning) {
|
||||
const { modelId, contextLength, minSafeContextTokens } = data.data;
|
||||
const proceed = await this._confirmContextWarning(modelId, contextLength, minSafeContextTokens);
|
||||
const { modelId, contextLength, minSafeContextTokens, slotCount } = data.data;
|
||||
const proceed = await this._confirmContextWarning(
|
||||
modelId,
|
||||
contextLength,
|
||||
minSafeContextTokens,
|
||||
slotCount
|
||||
);
|
||||
if (!proceed) {
|
||||
this._lastCustomModelLaunchResult = undefined;
|
||||
return { success: false, error: 'Launch cancelled — context window too small' };
|
||||
@@ -1371,7 +1402,8 @@ Object.assign(CodemanApp.prototype, {
|
||||
const proceed = await this._confirmContextWarning(
|
||||
payload.modelId,
|
||||
payload.contextLength,
|
||||
payload.minSafeContextTokens
|
||||
payload.minSafeContextTokens,
|
||||
payload.slotCount
|
||||
);
|
||||
if (!proceed) {
|
||||
switchingToast?.dismiss();
|
||||
|
||||
@@ -56,11 +56,31 @@ export const CLAUDE_MIN_SAFE_CONTEXT_TOKENS = 40000;
|
||||
*/
|
||||
export function exceedsSafeContextFloor(
|
||||
entry: Pick<CliEntry, 'capabilities'>,
|
||||
contextLength: number | undefined
|
||||
contextLength: number | undefined,
|
||||
slotCount?: number
|
||||
): boolean {
|
||||
const cap = entry.capabilities.customModelInjection;
|
||||
if (cap.kind !== 'env' || !cap.contextLengthVar) return false;
|
||||
return typeof contextLength === 'number' && contextLength < CLAUDE_MIN_SAFE_CONTEXT_TOKENS;
|
||||
if (typeof contextLength !== 'number') return false;
|
||||
if (contextLength < CLAUDE_MIN_SAFE_CONTEXT_TOKENS) return true;
|
||||
return perSlotContext(contextLength, slotCount) < CLAUDE_MIN_SAFE_CONTEXT_TOKENS;
|
||||
}
|
||||
|
||||
/**
|
||||
* The context one request can rely on when the server runs `slotCount` parallel slots.
|
||||
* llama-server divides its context across `--parallel` slots (by default they share ONE
|
||||
* unified KV pool), and Claude Code sends concurrent requests: the main session plus every
|
||||
* parallel subagent, each carrying the full `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` of its own
|
||||
* system prompt and tools. With N busy slots each request therefore gets roughly
|
||||
* `contextLength / N`, and once the pool is spent llama.cpp answers HTTP 500 "Context size
|
||||
* has been exceeded" (measured: 4 slots, ~131k pool, four simultaneous 40k-token requests
|
||||
* all failed; the same requests against one slot all succeeded). An unknown or single slot
|
||||
* count leaves the context whole.
|
||||
*/
|
||||
export function perSlotContext(contextLength: number, slotCount: number | undefined): number {
|
||||
const slots =
|
||||
typeof slotCount === 'number' && Number.isFinite(slotCount) && slotCount > 1 ? Math.floor(slotCount) : 1;
|
||||
return Math.floor(contextLength / slots);
|
||||
}
|
||||
|
||||
function adminOnly(req: FastifyRequest, reply: { code: (n: number) => unknown }): ApiResponse<never> | null {
|
||||
@@ -109,7 +129,7 @@ function applyStoredApiKey(incoming: CustomModelHost, existing: CustomModelHost)
|
||||
}
|
||||
|
||||
/**
|
||||
* `modelContextLengths`/`modelSizesGB` are server-populated by discovery, never
|
||||
* `modelContextLengths`/`modelSlotCounts`/`modelSizesGB` are server-populated by discovery, never
|
||||
* user-entered, and PUT replaces the whole record — so merge them back in from the
|
||||
* stored host rather than trust whatever the editor's body carried (or omitted).
|
||||
* The editor only ever sends `models`/`lastDiscoveredAt` verbatim from its cached
|
||||
@@ -117,7 +137,12 @@ function applyStoredApiKey(incoming: CustomModelHost, existing: CustomModelHost)
|
||||
* future caller forgets, same class of bug `applyStoredApiKey` exists to prevent.
|
||||
*/
|
||||
function applyDiscoveredFields(incoming: CustomModelHost, existing: CustomModelHost): CustomModelHost {
|
||||
return { ...incoming, modelContextLengths: existing.modelContextLengths, modelSizesGB: existing.modelSizesGB };
|
||||
return {
|
||||
...incoming,
|
||||
modelContextLengths: existing.modelContextLengths,
|
||||
modelSlotCounts: existing.modelSlotCounts,
|
||||
modelSizesGB: existing.modelSizesGB,
|
||||
};
|
||||
}
|
||||
|
||||
function authHeaders(host: Pick<CustomModelHost, 'apiKey' | 'authStyle'>): Record<string, string> {
|
||||
@@ -135,6 +160,8 @@ export interface DiscoveryResult {
|
||||
models: string[];
|
||||
/** See `CustomModelHost.modelContextLengths` — only ever populated for models already loaded. */
|
||||
contextLengths: Record<string, number>;
|
||||
/** See `CustomModelHost.modelSlotCounts`; populated on the same loaded-models-only basis as `contextLengths`. */
|
||||
slotCounts: Record<string, number>;
|
||||
/** See `CustomModelHost.modelSizesGB` — populated for every model whose own listing states one. */
|
||||
sizesGB: Record<string, number>;
|
||||
}
|
||||
@@ -157,7 +184,7 @@ function parseSizeGB(description: unknown): number | undefined {
|
||||
|
||||
/**
|
||||
* Best-effort: fetches `GET /props?model=<id>` (llama.cpp-native, llama-swap-proxied) for
|
||||
* ONE already-loaded model and pulls its real `n_ctx` out. Never called for a model that
|
||||
* ONE already-loaded model and pulls its `n_ctx` and `total_slots` out. Never called for a model that
|
||||
* isn't already loaded — see the caller and `CustomModelHost.modelContextLengths` for why
|
||||
* that's a hard safety requirement, not just a nicety: llama-swap treats this endpoint's
|
||||
* `?model=` as a routing hint, and asking it about an unloaded model risks triggering an
|
||||
@@ -173,20 +200,39 @@ function parseSizeGB(description: unknown): number | undefined {
|
||||
* (below), which reads the actual launch flag `/running` reports, is the primary source;
|
||||
* this is only used when that parse comes up empty (no recognized flag in `cmd`, or `cmd`
|
||||
* itself unavailable).
|
||||
*
|
||||
* ⚠️ `/props` only ever RAISES the slot warning, never skips it: its `n_ctx` is the context
|
||||
* fallback when `cmd` states no context flag at all, and `total_slots` is the slot count when
|
||||
* `cmd` has no `--parallel`. It never widens the pool the slot check divides. And it is not
|
||||
* free: llama-swap routes it through the model's process handler, which refreshes the model's
|
||||
* last-use time, so probing a model with a `ttl` on every 5-minute sweep would keep it loaded
|
||||
* forever. `discoverModels` therefore probes once per launch command (`propsByCmd`).
|
||||
*
|
||||
* Returns `undefined` when the request itself failed (network error, timeout, unreadable body),
|
||||
* which is retried next sweep; a non-2xx status, or a 2xx without the fields, gives an object with
|
||||
* them unset, which `propsForLaunch` remembers.
|
||||
*/
|
||||
async function fetchContextLength(
|
||||
async function fetchServerProps(
|
||||
host: Pick<CustomModelHost, 'baseUrl'>,
|
||||
modelId: string,
|
||||
headers: Record<string, string>
|
||||
): Promise<number | undefined> {
|
||||
): Promise<{ nCtx?: number; totalSlots?: number } | undefined> {
|
||||
try {
|
||||
const url = new URL(`${host.baseUrl.replace(/\/+$/, '')}/props`);
|
||||
url.searchParams.set('model', modelId);
|
||||
const res = await webviewFetch(url, { headers, signal: AbortSignal.timeout(PROPS_TIMEOUT_MS) });
|
||||
if (!res.ok) return undefined;
|
||||
const body = (await res.json()) as { n_ctx?: unknown; default_generation_settings?: { n_ctx?: unknown } };
|
||||
// A non-2xx answer (an upstream that does not serve /props) is a real answer: empty, so it is
|
||||
// remembered per launch command instead of being re-probed every sweep.
|
||||
if (!res.ok) return {};
|
||||
const body = (await res.json()) as {
|
||||
n_ctx?: unknown;
|
||||
total_slots?: unknown;
|
||||
default_generation_settings?: { n_ctx?: unknown };
|
||||
};
|
||||
const nCtx = body.n_ctx ?? body.default_generation_settings?.n_ctx;
|
||||
return typeof nCtx === 'number' && Number.isFinite(nCtx) && nCtx > 0 ? nCtx : undefined;
|
||||
const positive = (v: unknown): number | undefined =>
|
||||
typeof v === 'number' && Number.isFinite(v) && v > 0 ? v : undefined;
|
||||
return { nCtx: positive(nCtx), totalSlots: positive(body.total_slots) };
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
@@ -196,7 +242,7 @@ async function fetchContextLength(
|
||||
* Parses the REAL configured context size out of llama-swap's own launch command for a
|
||||
* model (`/running`'s `cmd` field, e.g. `"llama-server -m ... --fit-ctx 16384 ..."`) —
|
||||
* the primary source for `modelContextLengths`, preferred over `/props`'s `n_ctx` (see
|
||||
* `fetchContextLength`'s own doc comment for why that field is unreliable here). Checks
|
||||
* `fetchServerProps`'s own doc comment for why that field is unreliable here). Checks
|
||||
* `--fit-ctx` first (llama-swap's own auto-fit flag), then the plain llama.cpp
|
||||
* `-c`/`--ctx-size`/`--ctx_size` flags a hand-written launch command might use instead.
|
||||
* Returns `undefined` when `cmd` has none of these — not every launch command needs to
|
||||
@@ -211,6 +257,51 @@ function parseCtxFromCmd(cmd: unknown): number | undefined {
|
||||
return Number.isFinite(value) && value > 0 ? value : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses an explicit `--parallel N` / `-np N` (or `--parallel=N`) out of a launch command.
|
||||
* Returns `undefined` when the command states none, including `-np -1` (llama.cpp's "pick
|
||||
* automatically"), because a llama-server started without it chooses the slot count itself
|
||||
* (4 on the build this was measured on) and only `/props`'s `total_slots` can say what it chose.
|
||||
*/
|
||||
function parseSlotsFromCmd(cmd: unknown): number | undefined {
|
||||
if (typeof cmd !== 'string') return undefined;
|
||||
const match = /(?:^|\s)(?:-np|--parallel)(?:\s+|=)(\d+)(?=\s|$)/.exec(cmd);
|
||||
if (!match) return undefined;
|
||||
const value = Number(match[1]);
|
||||
return Number.isFinite(value) && value > 0 ? value : undefined;
|
||||
}
|
||||
|
||||
/** The `/props` answer for the launch command a model was last probed under, by host and model id. */
|
||||
const propsByCmd = new Map<string, { cmd: string; props: { nCtx?: number; totalSlots?: number } }>();
|
||||
|
||||
/** Forget what was probed (tests only: a changed launch command already probes again). */
|
||||
export function clearPropsProbeCache(): void {
|
||||
propsByCmd.clear();
|
||||
}
|
||||
|
||||
/**
|
||||
* `/props` for one loaded model, but only once per launch command: the probe refreshes the model's
|
||||
* last-use time in llama-swap, so repeating it on every rediscovery sweep would stop a model with a
|
||||
* `ttl` from ever unloading. A changed `cmd` (a reloaded or reconfigured model) probes again; a
|
||||
* `cmd` that is not a string cannot be told apart, so it is probed each time; a probe that did not
|
||||
* answer (network error, timeout) is not remembered; a non-2xx status is. Held in memory only, so a restart probes once more.
|
||||
*/
|
||||
async function propsForLaunch(
|
||||
host: Pick<CustomModelHost, 'baseUrl'>,
|
||||
modelId: string,
|
||||
cmd: unknown,
|
||||
headers: Record<string, string>
|
||||
): Promise<{ nCtx?: number; totalSlots?: number } | undefined> {
|
||||
const key = `${host.baseUrl}\0${modelId}`;
|
||||
if (typeof cmd === 'string') {
|
||||
const cached = propsByCmd.get(key);
|
||||
if (cached?.cmd === cmd) return cached.props;
|
||||
}
|
||||
const props = await fetchServerProps(host, modelId, headers);
|
||||
if (props && typeof cmd === 'string') propsByCmd.set(key, { cmd, props });
|
||||
return props;
|
||||
}
|
||||
|
||||
async function discoverModels(
|
||||
host: Pick<CustomModelHost, 'baseUrl' | 'apiKey' | 'authStyle'>
|
||||
): Promise<DiscoveryResult> {
|
||||
@@ -239,6 +330,7 @@ async function discoverModels(
|
||||
// fetchContextLength must only ever run against a model this server itself calls loaded.
|
||||
const hasStatusField = entries.some((m) => m && typeof m === 'object' && 'status' in m);
|
||||
const contextLengths: Record<string, number> = {};
|
||||
const slotCounts: Record<string, number> = {};
|
||||
if (hasStatusField) {
|
||||
const loadedIds = entries
|
||||
.filter((m) => m.status && typeof m.status === 'object' && (m.status as { value?: unknown }).value === 'loaded')
|
||||
@@ -252,13 +344,32 @@ async function discoverModels(
|
||||
const swapStatus = await getLlamaSwapStatus(host);
|
||||
const cmdById = new Map(swapStatus.running.map((r) => [r.model, r.cmd]));
|
||||
for (const id of loadedIds) {
|
||||
const fromCmd = parseCtxFromCmd(cmdById.get(id));
|
||||
const ctx = fromCmd ?? (await fetchContextLength(host, id, headers));
|
||||
const cmd = cmdById.get(id);
|
||||
let ctx = parseCtxFromCmd(cmd);
|
||||
const cmdSlots = parseSlotsFromCmd(cmd);
|
||||
let slots = cmdSlots;
|
||||
// /props is only needed for whatever the launch command did not answer. The context
|
||||
// still comes from the command whenever it states one (see fetchServerProps's doc
|
||||
// comment for why /props's n_ctx is only a fallback); the slot count usually cannot
|
||||
// come from the command at all, since an unset --parallel is chosen by the server.
|
||||
if (ctx === undefined || slots === undefined) {
|
||||
const props = await propsForLaunch(host, id, cmd, headers);
|
||||
const ctxFromProps = ctx === undefined && props?.nCtx !== undefined;
|
||||
ctx ??= props?.nCtx;
|
||||
slots ??= props?.totalSlots;
|
||||
// With an explicit --parallel N, llama.cpp reports /props's n_ctx PER SLOT (the unified
|
||||
// KV pool is only the default when the count is auto), so it is already the figure one
|
||||
// request can use, so the count is 1: recording N would divide it a second time, and
|
||||
// recording nothing would leave an earlier round's count in place (applyDiscoveredModels
|
||||
// merges onto what was learned before).
|
||||
if (cmdSlots !== undefined && ctxFromProps) slots = 1;
|
||||
}
|
||||
if (ctx !== undefined) contextLengths[id] = ctx;
|
||||
if (slots !== undefined) slotCounts[id] = slots;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { models, contextLengths, sizesGB };
|
||||
return { models, contextLengths, slotCounts, sizesGB };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -573,7 +684,7 @@ export function triggerLlamaSwapLoad(
|
||||
}
|
||||
|
||||
function applyDiscoveredModels(host: CustomModelHost, result: DiscoveryResult): CustomModelHost {
|
||||
const { models, contextLengths, sizesGB } = result;
|
||||
const { models, contextLengths, slotCounts, sizesGB } = result;
|
||||
const defaultModelId = host.defaultModelId && models.includes(host.defaultModelId) ? host.defaultModelId : undefined;
|
||||
// Merge onto what's already known rather than replacing: a model not probed this round
|
||||
// (not currently loaded) keeps whatever context length an earlier round already learned
|
||||
@@ -581,6 +692,10 @@ function applyDiscoveredModels(host: CustomModelHost, result: DiscoveryResult):
|
||||
const merged = { ...host.modelContextLengths, ...contextLengths };
|
||||
const kept = Object.fromEntries(Object.entries(merged).filter(([id]) => models.includes(id)));
|
||||
const modelContextLengths = Object.keys(kept).length > 0 ? kept : undefined;
|
||||
// Slot counts follow the same loaded-only, keep-what-an-earlier-round-learned rule.
|
||||
const mergedSlots = { ...host.modelSlotCounts, ...slotCounts };
|
||||
const keptSlots = Object.fromEntries(Object.entries(mergedSlots).filter(([id]) => models.includes(id)));
|
||||
const modelSlotCounts = Object.keys(keptSlots).length > 0 ? keptSlots : undefined;
|
||||
// sizesGB, unlike contextLengths, is populated for every model in the SAME pass (no
|
||||
// loaded-only restriction — see parseSizeGB), so this is closer to a plain replace, but
|
||||
// still merges onto the previous round rather than dropping a size for a model whose
|
||||
@@ -593,6 +708,7 @@ function applyDiscoveredModels(host: CustomModelHost, result: DiscoveryResult):
|
||||
models,
|
||||
defaultModelId,
|
||||
modelContextLengths,
|
||||
modelSlotCounts,
|
||||
modelSizesGB,
|
||||
lastDiscoveredAt: new Date().toISOString(),
|
||||
};
|
||||
|
||||
@@ -1535,6 +1535,7 @@ export function registerSessionRoutes(
|
||||
return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
|
||||
}
|
||||
const contextLength = endpoint.modelContextLengths?.[body.modelId];
|
||||
const slotCount = endpoint.modelSlotCounts?.[body.modelId];
|
||||
|
||||
// Some CLIs (today: only claude) carry enough of their own fixed system-prompt/tool-
|
||||
// schema overhead that a small enough real context guarantees a first-message failure
|
||||
@@ -1545,12 +1546,13 @@ export function registerSessionRoutes(
|
||||
// `confirmed`, which still means both) — NOT by `confirmedSwap`: this warning is
|
||||
// about the caller's own session, and the swap warning below is about someone
|
||||
// else's, so an answer to one is not consent to the other.
|
||||
if (!(body.confirmed || body.confirmedContext) && exceedsSafeContextFloor(entry, contextLength)) {
|
||||
if (!(body.confirmed || body.confirmedContext) && exceedsSafeContextFloor(entry, contextLength, slotCount)) {
|
||||
return {
|
||||
requiresContextWarning: true,
|
||||
modelId: body.modelId,
|
||||
contextLength,
|
||||
minSafeContextTokens: CLAUDE_MIN_SAFE_CONTEXT_TOKENS,
|
||||
...(slotCount && slotCount > 1 ? { slotCount } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -4332,6 +4334,7 @@ export function registerSessionRoutes(
|
||||
const cmEndpoint = cmHosts.find((h) => h.id === customModel.endpointId);
|
||||
if (!cmEndpoint) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
|
||||
const cmContextLength = cmEndpoint.modelContextLengths?.[customModel.modelId];
|
||||
const cmSlotCount = cmEndpoint.modelSlotCounts?.[customModel.modelId];
|
||||
|
||||
// See the dedicated route's own comment for the full reasoning: some CLIs' own fixed
|
||||
// overhead can exceed a small enough real context on the very first message,
|
||||
@@ -4341,13 +4344,14 @@ export function registerSessionRoutes(
|
||||
// `confirmedSwap`.
|
||||
if (
|
||||
!(customModel.confirmed || customModel.confirmedContext) &&
|
||||
exceedsSafeContextFloor(cmEntry, cmContextLength)
|
||||
exceedsSafeContextFloor(cmEntry, cmContextLength, cmSlotCount)
|
||||
) {
|
||||
return {
|
||||
requiresContextWarning: true,
|
||||
modelId: customModel.modelId,
|
||||
contextLength: cmContextLength,
|
||||
minSafeContextTokens: CLAUDE_MIN_SAFE_CONTEXT_TOKENS,
|
||||
...(cmSlotCount && cmSlotCount > 1 ? { slotCount: cmSlotCount } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -2274,6 +2274,8 @@ export const CustomModelHostSchema = z.object({
|
||||
// round-tripping the GET response back through PUT (edit-save) doesn't drop it.
|
||||
modelContextLengths: z.record(z.string().max(200), z.number().int().positive().max(100_000_000)).optional(),
|
||||
// Same reasoning as modelContextLengths above.
|
||||
modelSlotCounts: z.record(z.string().max(200), z.number().int().positive().max(4096)).optional(),
|
||||
// Same reasoning as modelContextLengths above.
|
||||
modelSizesGB: z.record(z.string().max(200), z.number().positive().max(100_000)).optional(),
|
||||
});
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
import { getDataDir } from '../src/config/instance.js';
|
||||
import { readCustomModelHosts, writeCustomModelHosts, type CustomModelHost } from '../src/custom-model-hosts.js';
|
||||
import { refreshAllCustomModelHosts } from '../src/web/routes/custom-model-routes.js';
|
||||
import { clearPropsProbeCache, refreshAllCustomModelHosts } from '../src/web/routes/custom-model-routes.js';
|
||||
import { webviewFetch } from '../src/web/webview-egress.js';
|
||||
|
||||
vi.mock('../src/web/webview-egress.js', async () => {
|
||||
@@ -31,6 +31,7 @@ function host(overrides: Partial<CustomModelHost> & Pick<CustomModelHost, 'id' |
|
||||
|
||||
beforeEach(() => {
|
||||
fetchMock.mockReset();
|
||||
clearPropsProbeCache();
|
||||
});
|
||||
|
||||
describe('refreshAllCustomModelHosts (the periodic re-discovery sweep)', () => {
|
||||
@@ -211,8 +212,9 @@ describe('refreshAllCustomModelHosts: context-length enrichment (llama.cpp/llama
|
||||
it('prefers the REAL configured context size parsed from /running’s launch command over /props’s unreliable n_ctx', async () => {
|
||||
// Confirmed live: llama-swap launched a model with --fit-ctx 16384 (the real, working
|
||||
// limit — the actual server then refused a request over it), but /props reported
|
||||
// n_ctx: 154112 for the same model, well over what it would really accept. /props must
|
||||
// never be reached at all once the /running command parse already answered it.
|
||||
// n_ctx: 154112 for the same model, well over what it would really accept. The context
|
||||
// must therefore keep coming from the /running command; /props is now also probed, but
|
||||
// only for `total_slots`, which the command does not state.
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
fetchMock.mockImplementation(async (url: URL) => {
|
||||
@@ -235,7 +237,9 @@ describe('refreshAllCustomModelHosts: context-length enrichment (llama.cpp/llama
|
||||
{ status: 200 }
|
||||
);
|
||||
}
|
||||
if (url.pathname === '/props') throw new Error('must never be reached — the cmd parse already answered it');
|
||||
if (url.pathname === '/props') {
|
||||
return new Response(JSON.stringify({ n_ctx: 154112, total_slots: 4 }), { status: 200 });
|
||||
}
|
||||
throw new Error(`unexpected request: ${url.href}`);
|
||||
});
|
||||
|
||||
@@ -243,6 +247,8 @@ describe('refreshAllCustomModelHosts: context-length enrichment (llama.cpp/llama
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelContextLengths).toEqual({ 'qwen3.8-27b': 16384 });
|
||||
expect(updated.modelSlotCounts).toEqual({ 'qwen3.8-27b': 4 });
|
||||
// /props's 154112 never becomes the context, and nothing else stores it
|
||||
});
|
||||
|
||||
it('falls back to /props when /running has no cmd, or the cmd states no recognizable context flag', async () => {
|
||||
@@ -283,13 +289,197 @@ describe('refreshAllCustomModelHosts: context-length enrichment (llama.cpp/llama
|
||||
{ status: 200 }
|
||||
);
|
||||
}
|
||||
throw new Error(`unexpected request: ${url.href}`); // /props must never be reached
|
||||
if (url.pathname === '/props') return new Response('not found', { status: 404 }); // slot probe only
|
||||
throw new Error(`unexpected request: ${url.href}`);
|
||||
});
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelContextLengths).toEqual({ a: 8192 });
|
||||
expect(updated.modelSlotCounts).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('refreshAllCustomModelHosts: parallel-slot enrichment (llama.cpp splits its context across --parallel slots)', () => {
|
||||
/** One loaded model whose launch command is `cmd`; `/props` behaviour is supplied by the test. */
|
||||
function serve(cmd: string, props: () => Response) {
|
||||
fetchMock.mockImplementation(async (url: URL) => {
|
||||
if (url.pathname === '/v1/models') {
|
||||
return new Response(JSON.stringify({ data: [{ id: 'm', status: { value: 'loaded' } }] }), { status: 200 });
|
||||
}
|
||||
if (url.pathname === '/running') {
|
||||
return new Response(JSON.stringify({ running: [{ model: 'm', state: 'ready', cmd }] }), { status: 200 });
|
||||
}
|
||||
if (url.pathname === '/props') return props();
|
||||
throw new Error(`unexpected request: ${url.href}`);
|
||||
});
|
||||
}
|
||||
const propsMustNotBeReached = () => {
|
||||
throw new Error('/props must not be probed: the launch command already answered both questions');
|
||||
};
|
||||
|
||||
it.each([
|
||||
['--parallel 1', 1],
|
||||
['-np 2', 2],
|
||||
['--parallel=3', 3],
|
||||
['--fit-ctx 65536 --parallel 1 --host 0.0.0.0', 1],
|
||||
])('reads an explicit %s from the launch command without probing /props', async (flags, slots) => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
serve(`llama-server -m /models/m.gguf --ctx-size 65536 ${flags}`, propsMustNotBeReached);
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ m: slots });
|
||||
expect(updated.modelContextLengths).toEqual({ m: 65536 });
|
||||
});
|
||||
|
||||
it('asks /props for total_slots when the command states no --parallel (llama-server picks the count itself)', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --fit-ctx 65536',
|
||||
() => new Response(JSON.stringify({ n_ctx: 133888, total_slots: 4 }), { status: 200 })
|
||||
);
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ m: 4 });
|
||||
// the context still comes from the command's floor, never from /props's n_ctx
|
||||
expect(updated.modelContextLengths).toEqual({ m: 65536 });
|
||||
});
|
||||
|
||||
it('probes /props once per launch command: a repeat sweep reuses the answer, a changed command probes again', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
let probes = 0;
|
||||
const answer = () => (probes++, new Response(JSON.stringify({ n_ctx: 133888, total_slots: 4 }), { status: 200 }));
|
||||
serve('llama-server -m /models/m.gguf --fit-ctx 65536', answer);
|
||||
await refreshAllCustomModelHosts();
|
||||
await refreshAllCustomModelHosts();
|
||||
expect(probes).toBe(1); // a /props call refreshes the model's last-use time in llama-swap, so not every sweep
|
||||
expect((await readCustomModelHosts(dir))[0].modelSlotCounts).toEqual({ m: 4 });
|
||||
|
||||
serve('llama-server -m /models/m.gguf --fit-ctx 65536 --flash-attn on', answer);
|
||||
await refreshAllCustomModelHosts();
|
||||
expect(probes).toBe(2);
|
||||
});
|
||||
|
||||
it('does not divide a /props context a second time when --parallel is explicit and the command states no context', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
// with an explicit count llama.cpp reports n_ctx per slot: 65,536 each is already what one request gets
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --parallel 4',
|
||||
() => new Response(JSON.stringify({ n_ctx: 65536, total_slots: 4 }), { status: 200 })
|
||||
);
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelContextLengths).toEqual({ m: 65536 });
|
||||
expect(updated.modelSlotCounts).toEqual({ m: 1 }); // already per slot, so one request gets all of it
|
||||
});
|
||||
|
||||
it('a later round with an explicit --parallel replaces the slot count an earlier round learned', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
// round 1: auto slots, the server chose 4 sharing 131,072 tokens
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf',
|
||||
() => new Response(JSON.stringify({ n_ctx: 131072, total_slots: 4 }), { status: 200 })
|
||||
);
|
||||
await refreshAllCustomModelHosts();
|
||||
expect((await readCustomModelHosts(dir))[0].modelSlotCounts).toEqual({ m: 4 });
|
||||
|
||||
// round 2: the user followed the dialog and added --parallel 1
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --parallel 1',
|
||||
() => new Response(JSON.stringify({ n_ctx: 131072, total_slots: 1 }), { status: 200 })
|
||||
);
|
||||
await refreshAllCustomModelHosts();
|
||||
expect((await readCustomModelHosts(dir))[0].modelSlotCounts).toEqual({ m: 1 });
|
||||
|
||||
// round 3: --parallel 2 with a per-slot /props context must not bring a pool back
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --parallel 2',
|
||||
() => new Response(JSON.stringify({ n_ctx: 65536, total_slots: 2 }), { status: 200 })
|
||||
);
|
||||
await refreshAllCustomModelHosts();
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ m: 1 });
|
||||
expect(updated.modelContextLengths).toEqual({ m: 65536 });
|
||||
});
|
||||
|
||||
it('remembers a /props that answers non-2xx per launch command, but retries a network failure', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
let probes = 0;
|
||||
serve('llama-server -m /models/m.gguf --fit-ctx 65536', () => (probes++, new Response('nope', { status: 404 })));
|
||||
await refreshAllCustomModelHosts();
|
||||
await refreshAllCustomModelHosts();
|
||||
expect(probes).toBe(1); // no probe every sweep: a /props call keeps a model with a ttl loaded
|
||||
|
||||
clearPropsProbeCache();
|
||||
probes = 0;
|
||||
serve('llama-server -m /models/m.gguf --fit-ctx 65536', () => {
|
||||
probes++;
|
||||
throw new TypeError('fetch failed');
|
||||
});
|
||||
await refreshAllCustomModelHosts();
|
||||
await refreshAllCustomModelHosts();
|
||||
expect(probes).toBe(2); // a network error says nothing about the server, so it is tried again
|
||||
});
|
||||
|
||||
it('treats -np -1 (llama.cpp auto) as unstated and falls back to /props', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --fit-ctx 65536 -np -1',
|
||||
() => new Response(JSON.stringify({ total_slots: 4 }), { status: 200 })
|
||||
);
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ m: 4 });
|
||||
});
|
||||
|
||||
it('a failed or slot-less /props leaves no slot count rather than failing the sweep or guessing one', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
serve(
|
||||
'llama-server -m /models/m.gguf --fit-ctx 65536',
|
||||
() => new Response(JSON.stringify({ n_ctx: 8192 }), { status: 200 })
|
||||
);
|
||||
|
||||
await expect(refreshAllCustomModelHosts()).resolves.toBeUndefined();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toBeUndefined();
|
||||
expect(updated.modelContextLengths).toEqual({ m: 65536 });
|
||||
});
|
||||
|
||||
it('keeps a previously-learned slot count for a model that is no longer loaded, and drops it once the model disappears', async () => {
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [
|
||||
host({
|
||||
id: 'ep',
|
||||
baseUrl: 'http://localhost:8080',
|
||||
models: ['a', 'b'],
|
||||
modelContextLengths: { a: 65536, b: 65536 },
|
||||
modelSlotCounts: { a: 4, b: 2 },
|
||||
}),
|
||||
]);
|
||||
fetchMock.mockResolvedValue(new Response(JSON.stringify({ data: [{ id: 'a' }] }), { status: 200 }));
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ a: 4 });
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -1281,7 +1281,7 @@ describe("Custom Model Endpoint Profiles: requiresContextWarning (this CLI's own
|
||||
|
||||
await app.runCustomModelEntry('claude', 'llama-box', 'qwen3');
|
||||
|
||||
expect(confirmArgs).toEqual(['qwen3', 16384, 40000]);
|
||||
expect(confirmArgs).toEqual(['qwen3', 16384, 40000, undefined]);
|
||||
expect(applyBodies).toEqual([
|
||||
{ endpointId: 'llama-box', modelId: 'qwen3' },
|
||||
{ endpointId: 'llama-box', modelId: 'qwen3', confirmedContext: true },
|
||||
@@ -1289,6 +1289,29 @@ describe("Custom Model Endpoint Profiles: requiresContextWarning (this CLI's own
|
||||
expect(win.localStorage.getItem('codeman:customModelLastUsed:claude:llama-box')).toBe('qwen3');
|
||||
});
|
||||
|
||||
it('forwards the slot count from the apply response into the modal', async () => {
|
||||
const { app, applyBodies } = launchHarness([
|
||||
{
|
||||
requiresContextWarning: true,
|
||||
modelId: 'qwen3',
|
||||
contextLength: 131072,
|
||||
minSafeContextTokens: 40000,
|
||||
slotCount: 4,
|
||||
},
|
||||
{ customModel: { endpointId: 'llama-box' }, restarted: true, modelSwapInProgress: false },
|
||||
]);
|
||||
let confirmArgs: unknown[] | undefined;
|
||||
app._confirmContextWarning = async (...args: unknown[]) => {
|
||||
confirmArgs = args;
|
||||
return true;
|
||||
};
|
||||
|
||||
await app.runCustomModelEntry('claude', 'llama-box', 'qwen3');
|
||||
|
||||
expect(confirmArgs).toEqual(['qwen3', 131072, 40000, 4]);
|
||||
expect(applyBodies).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('declining the in-app context-warning modal keeps the native backend, never re-sends the apply, and must NEVER record this model as "last used" — it cannot work with this CLI at all', async () => {
|
||||
const { win, app, applyBodies } = launchHarness([
|
||||
{ requiresContextWarning: true, modelId: 'qwen3', contextLength: 16384, minSafeContextTokens: 40000 },
|
||||
@@ -1361,6 +1384,49 @@ describe('Custom Model Endpoint Profiles: _confirmContextWarning (in-app modal,
|
||||
expect(win.document.getElementById('customModelContextWarningModal')!.classList.contains('active')).toBe(false);
|
||||
});
|
||||
|
||||
it('explains shared parallel slots, and the --parallel 1 fix, when the context is enough only as a whole', async () => {
|
||||
const { win, app } = bootApp({});
|
||||
void app._confirmContextWarning('qwen3.8-27b-ud-q4_k_xl', 131072, 40000, 4);
|
||||
const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!;
|
||||
expect(message).toContain('qwen3.8-27b-ud-q4_k_xl');
|
||||
expect(message).toContain('4 parallel');
|
||||
expect(message).toContain('131,072');
|
||||
expect(message).toContain('32,768'); // per slot
|
||||
expect(message).toContain('--parallel 1');
|
||||
expect(message).toContain('Context size has been exceeded');
|
||||
// this is not the "first message fails outright" case, so it must not claim that
|
||||
expect(message).not.toMatch(/first message will fail/i);
|
||||
expect(win.document.getElementById('customModelContextWarningModal')!.classList.contains('active')).toBe(true);
|
||||
app._resolveContextWarningConfirm(false);
|
||||
});
|
||||
|
||||
it('says the context is split across the slots, about N each, from the discovered context alone', async () => {
|
||||
const { win, app } = bootApp({});
|
||||
void app._confirmContextWarning('qwen3', 65536, 40000, 4);
|
||||
const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!;
|
||||
expect(message).toContain('65,536 tokens of context are split across 4 parallel server slots, about 16,384 each');
|
||||
app._resolveContextWarningConfirm(false);
|
||||
});
|
||||
|
||||
it('keeps the too-small-context message for a small model and adds the slot advice when slots are known', async () => {
|
||||
const { win, app } = bootApp({});
|
||||
void app._confirmContextWarning('qwen3', 16384, 40000, 4);
|
||||
const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!;
|
||||
expect(message).toContain('16,384');
|
||||
expect(message).toMatch(/first message will fail/i);
|
||||
expect(message).toContain('4 parallel slots');
|
||||
expect(message).toContain('--parallel 1');
|
||||
app._resolveContextWarningConfirm(false);
|
||||
});
|
||||
|
||||
it('does not mention slots when there is one (or none known)', async () => {
|
||||
const { win, app } = bootApp({});
|
||||
void app._confirmContextWarning('qwen3', 16384, 40000, 1);
|
||||
const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!;
|
||||
expect(message).not.toMatch(/parallel/i);
|
||||
app._resolveContextWarningConfirm(false);
|
||||
});
|
||||
|
||||
it('describes an unknown context length without printing a bogus number', async () => {
|
||||
const { win, app } = bootApp({});
|
||||
void app._confirmContextWarning('qwen3', undefined, 40000);
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* @fileoverview Unit tests for the parallel-slot awareness of the custom-model context floor
|
||||
* (`perSlotContext` / `exceedsSafeContextFloor`, src/web/routes/custom-model-routes.ts).
|
||||
*
|
||||
* llama-server splits its context across `--parallel` slots (by default they share one
|
||||
* unified KV pool) and Claude Code sends concurrent requests, each carrying its own ~40k
|
||||
* tokens of system prompt and tools. Measured against a real llama-swap server: with 4 slots
|
||||
* sharing ~131k tokens, four simultaneous 40k-token requests all failed with HTTP 500
|
||||
* "Context size has been exceeded"; the same requests against one slot all succeeded.
|
||||
*
|
||||
* Port: N/A (pure functions, no server).
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
CLAUDE_MIN_SAFE_CONTEXT_TOKENS,
|
||||
exceedsSafeContextFloor,
|
||||
perSlotContext,
|
||||
} from '../src/web/routes/custom-model-routes.js';
|
||||
import type { CliEntry } from '../src/config/cli-registry/types.js';
|
||||
|
||||
const withContextVar = {
|
||||
capabilities: { customModelInjection: { kind: 'env', contextLengthVar: 'CLAUDE_CODE_MAX_CONTEXT_TOKENS' } },
|
||||
} as unknown as Pick<CliEntry, 'capabilities'>;
|
||||
const withoutContextVar = {
|
||||
capabilities: { customModelInjection: { kind: 'env' } },
|
||||
} as unknown as Pick<CliEntry, 'capabilities'>;
|
||||
|
||||
describe('perSlotContext', () => {
|
||||
it('divides the context across the slots', () => {
|
||||
expect(perSlotContext(131072, 4)).toBe(32768);
|
||||
expect(perSlotContext(262144, 4)).toBe(65536);
|
||||
});
|
||||
|
||||
it('rounds down, since a fraction of a token is not usable', () => {
|
||||
expect(perSlotContext(100000, 3)).toBe(33333);
|
||||
});
|
||||
|
||||
it.each([undefined, 0, 1, -2, Number.NaN, Number.POSITIVE_INFINITY])(
|
||||
'leaves the context whole for an unusable or single slot count (%s)',
|
||||
(slots) => {
|
||||
expect(perSlotContext(65536, slots as number | undefined)).toBe(65536);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
describe('exceedsSafeContextFloor with a slot count', () => {
|
||||
it('divides only the context it is given, so a larger /props figure can never skip the warning', () => {
|
||||
// a --fit-ctx floor of 65,536 over 4 slots is 16,384 each, whatever /props might say about the server
|
||||
expect(exceedsSafeContextFloor(withContextVar, 65536, 4)).toBe(true);
|
||||
expect(exceedsSafeContextFloor(withContextVar, 65536, 1)).toBe(false);
|
||||
// the measured failure: ~131k over 4 slots is ~33k each
|
||||
expect(exceedsSafeContextFloor(withContextVar, 131072, 4)).toBe(true);
|
||||
expect(exceedsSafeContextFloor(withContextVar, 262144, 4)).toBe(false);
|
||||
});
|
||||
|
||||
it('still warns when the context itself is below the minimum', () => {
|
||||
expect(exceedsSafeContextFloor(withContextVar, 16384, 4)).toBe(true);
|
||||
expect(exceedsSafeContextFloor(withContextVar, 16384, 1)).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps its existing behaviour when no slot count is given', () => {
|
||||
expect(exceedsSafeContextFloor(withContextVar, 16384)).toBe(true);
|
||||
expect(exceedsSafeContextFloor(withContextVar, CLAUDE_MIN_SAFE_CONTEXT_TOKENS)).toBe(false);
|
||||
expect(exceedsSafeContextFloor(withContextVar, undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it('flags a context that only clears the floor as a whole, not per slot', () => {
|
||||
expect(exceedsSafeContextFloor(withContextVar, 131072, 4)).toBe(true);
|
||||
expect(exceedsSafeContextFloor(withContextVar, 131072, 1)).toBe(false);
|
||||
});
|
||||
|
||||
it('accepts exactly the floor per slot', () => {
|
||||
expect(exceedsSafeContextFloor(withContextVar, CLAUDE_MIN_SAFE_CONTEXT_TOKENS * 4, 4)).toBe(false);
|
||||
expect(exceedsSafeContextFloor(withContextVar, CLAUDE_MIN_SAFE_CONTEXT_TOKENS * 4 - 4, 4)).toBe(true);
|
||||
});
|
||||
|
||||
it('never applies to a CLI that declares no context variable, whatever the slots', () => {
|
||||
expect(exceedsSafeContextFloor(withoutContextVar, 4096, 8)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -316,6 +316,47 @@ describe('POST /api/quick-start: customModel (one-shot custom-model launch)', ()
|
||||
expect(ctx.sessions.size).toBe(2);
|
||||
});
|
||||
|
||||
describe('parallel slots share the context (the quick-start path has its own slot branch)', () => {
|
||||
const slotEndpoint = (contextLength: number, slots: number): CustomModelHost => ({
|
||||
id: 'ep-slots',
|
||||
label: 'multi-slot box',
|
||||
baseUrl: 'http://192.168.1.52:8080',
|
||||
apiKey: 'k',
|
||||
modelContextLengths: { m: contextLength },
|
||||
modelSlotCounts: { m: slots },
|
||||
});
|
||||
|
||||
it('warns when the context divided across the slots is below the floor, and reports the slot count', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), [ENDPOINT, slotEndpoint(131072, 4)]); // ~33k per slot
|
||||
|
||||
const res = await quickStart({
|
||||
caseName: 'cm-slots-warn',
|
||||
mode: 'claude',
|
||||
customModel: { endpointId: 'ep-slots', modelId: 'm' },
|
||||
});
|
||||
|
||||
const body = res.json();
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.contextLength).toBe(131072);
|
||||
expect(body.slotCount).toBe(4);
|
||||
expect(ctx.sessions.size).toBe(1); // nothing was created
|
||||
});
|
||||
|
||||
it('launches when every slot can hold the floor', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), [ENDPOINT, slotEndpoint(262144, 4)]); // 65,536 per slot
|
||||
|
||||
const res = await quickStart({
|
||||
caseName: 'cm-slots-ok',
|
||||
mode: 'claude',
|
||||
customModel: { endpointId: 'ep-slots', modelId: 'm' },
|
||||
});
|
||||
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(res.json().requiresContextWarning).toBeUndefined();
|
||||
expect(ctx.sessions.size).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
it('does not warn when nothing about context was discovered', async () => {
|
||||
const res = await quickStart({
|
||||
caseName: 'cm-no-ctx-data',
|
||||
|
||||
@@ -449,6 +449,78 @@ describe('POST /api/sessions/:id/custom-model', () => {
|
||||
expect(session.restartCli).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// llama-server splits its context across --parallel slots, and Claude Code sends several
|
||||
// requests at once, so a context that clears the floor as a whole can still fail per slot.
|
||||
describe('parallel slots share the context', () => {
|
||||
const slotEndpoint = (contextLength: number, slots?: number): CustomModelHost => ({
|
||||
id: 'ep-slots',
|
||||
label: 'multi-slot box',
|
||||
baseUrl: 'http://192.168.1.52:8080',
|
||||
apiKey: 'k',
|
||||
modelContextLengths: { m: contextLength },
|
||||
...(slots === undefined ? {} : { modelSlotCounts: { m: slots } }),
|
||||
});
|
||||
|
||||
async function apply(endpoint: CustomModelHost) {
|
||||
const { app, ctx } = await setup();
|
||||
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, endpoint]);
|
||||
const session = ctx.sessions.get('test-session-1')!;
|
||||
session.mode = 'claude';
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/sessions/test-session-1/custom-model',
|
||||
payload: { endpointId: endpoint.id, modelId: 'm' },
|
||||
});
|
||||
return { body: res.json(), session };
|
||||
}
|
||||
|
||||
it('warns when the context divided across the slots is below the floor, and reports the slot count', async () => {
|
||||
// the measured case: ~131k shared by 4 slots is ~33k each, under the 40k a request needs
|
||||
const { body, session } = await apply(slotEndpoint(131072, 4));
|
||||
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.contextLength).toBe(131072);
|
||||
expect(body.slotCount).toBe(4);
|
||||
expect(session.setCustomModel).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('divides the discovered context across the slots even when it clears the floor as a whole', async () => {
|
||||
// 65,536 over 4 slots is 16,384 each: nothing the server reports can widen this pool
|
||||
const { body } = await apply(slotEndpoint(65536, 4));
|
||||
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.contextLength).toBe(65536);
|
||||
expect(body.slotCount).toBe(4);
|
||||
});
|
||||
|
||||
it('does not warn once every slot can hold the floor', async () => {
|
||||
const { body, session } = await apply(slotEndpoint(262144, 4)); // 65,536 each
|
||||
|
||||
expect(body.requiresContextWarning).toBeUndefined();
|
||||
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('does not warn for a single slot with the same context that failed across four', async () => {
|
||||
const { body, session } = await apply(slotEndpoint(131072, 1));
|
||||
|
||||
expect(body.requiresContextWarning).toBeUndefined();
|
||||
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('an unknown slot count is treated as one slot, never guessed', async () => {
|
||||
const { body } = await apply(slotEndpoint(131072));
|
||||
|
||||
expect(body.requiresContextWarning).toBeUndefined();
|
||||
});
|
||||
|
||||
it('omits slotCount from the warning when the context is too small on its own with a single slot', async () => {
|
||||
const { body } = await apply(slotEndpoint(16384, 1));
|
||||
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.slotCount).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
it('applies once confirmed, skipping the context check the second time', async () => {
|
||||
const { app, ctx } = await setup();
|
||||
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, SMALL_CTX_ENDPOINT]);
|
||||
|
||||
Reference in new issue
Block a user