Root cause of the context-overflow regression reported live: "API Error: 400
request (36437 tokens) exceeds the available context size (16384 tokens)".
Discovery had stored modelContextLengths.qwen3.8-27b-ud-q4_k_xl = 154112,
so CLAUDE_CODE_MAX_CONTEXT_TOKENS told Claude Code it had a huge window and
it never compacted - but the real llama-swap server was launched with
--fit-ctx 16384 (confirmed against /running's own cmd field) and refused
the request right at that real limit.
/props?model=<id>'s n_ctx (the field discovery read) is confirmed live to
be unreliable for a --fit-ctx-launched backend: it reported 154112 for the
same model /running says was launched with --fit-ctx 16384 - appears to
report the model's theoretical/trained maximum context, not the runtime-
configured one.
discoverModels() now parses the REAL configured size straight out of
llama-swap's own launch command instead (parseCtxFromCmd(), reading
/running's cmd field - --fit-ctx first, then the plain llama.cpp -c/
--ctx-size a hand-written command might use), and only falls back to the
old /props probe when cmd states no recognizable flag at all. One /running
call now covers every loaded model's context length in a single request,
same as it already did for the swap-conflict check and the load trigger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
Discovery now also parses a GB figure out of an auto-discovered model's own
description (llama-swap writes "Auto-discovered 16.35 GB - parameters
auto-fitted by llama.cpp"), stored per model as modelSizesGB - unlike
context length this needs no /props probe (the figure is right there in
/v1/models) so it is populated for every model regardless of loaded state.
A hand-configured profile's own description has no such figure and
correctly gets no entry.
The loading banner (_watchLlamaSwapLoading) now looks this up and, when
known, shows it plus a rough estimate from a small size->time matrix
(_estimateModelLoad/_MODEL_LOAD_TIME_MATRIX, session-ui.js) -
"Loading qwen3.8-27b-ud-q4_k_xl (16.4 GB, typically ~1-3 min) on
llama-swap... this can take a while" - and uses that same estimate's own
bracket to scale the banner's default give-up timeout for a very large
model, instead of a flat 5 minutes for everything. Explicitly labelled as
an UNMEASURED, typical-hardware estimate in every relevant comment - this
is not benchmarked against any real endpoint's actual storage/GPU, just a
reasonable expectation-setter. A model with no discoverable size (a
hand-configured profile) gets no size/estimate shown at all, matching the
"never a guess" convention modelContextLengths already established.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
Addresses two live-validation findings on the Run-menu custom-model picker:
1. Both claude.ai and ANTHROPIC_API_KEY set warning. Claude Code still
coexists an OAuth login with an injected ANTHROPIC_API_KEY in the same
config directory and warns about it (confirmed cosmetic - the API key
wins for actual requests, verified via a real session's own API Usage
Billing line). A custom-model claude session now gets an isolated
CLAUDE_CONFIG_DIR (registry-declared via a new configDirVar field, empty,
no files written into it) so there is nothing to conflict with. projects
is symlinked (junction on Windows) back into the real config dir so the
response viewer, subagent windows and Read My Mind keep working for that
session, best-effort.
2. Context-window overflow. Claude Code assumes a large default context
window for a model id it doesn't recognise and never compacts, so a
custom endpoint's real, much smaller context (verified live: a 400
exceeding a 16384-token llama-swap model with a stock ~33.7K-token system
prompt) silently overflows. Discovery now also learns each model's real
context length from llama.cpp/llama-swap's GET /props?model=<id> (n_ctx),
but ONLY for a model llama-swap's own /v1/models response already marks
status.value === 'loaded' - never an unloaded one, since llama-swap
treats ?model= as a routing hint and probing an unloaded model risks
triggering an actual, slow, GPU-swapping load as a side effect of
read-only discovery. A server with no status field at all gets no
enrichment rather than a guess; a model not probed this round keeps its
previously-learned value until it disappears from the list entirely.
Stored per model (CustomModelHost.modelContextLengths) and applied via a
new contextLengthVar registry field, set to
CLAUDE_CODE_MAX_CONTEXT_TOKENS for claude.
Both new fields live on the existing env-kind customModelInjection
capability shape, declared only on claude's registry entry - every other
CLI's injection is unaffected (pinned by test).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
Two enhancements requested after live-validating PR #430 against a real
llama.cpp server:
1. Model picker dialog. Picking a Run-menu Custom Endpoints entry used to
apply the endpoint's defaultModelId (or the first discovered model)
silently. Now, via the new selectCustomModelEntry() (session-ui.js):
- exactly one discovered model launches straight away, same as before
- two or more open a new #customModelPickModal listing every discovered
model; defaultModelId (if set) is marked but never auto-chosen, since
the point of asking is letting ONE launch deliberately differ from
the saved default, not just confirming it
The endpoint is re-fetched at click time rather than trusting anything
cached from the dropdown's own render, since the model list can have
changed (the sweep below, or a settings-panel edit) since it opened.
runCustomModelEntry() itself — the actual launch, routed through run()
for the in-flight lock, snapshot-guarded against applying to the wrong
session — is unchanged; it now just always receives an explicit model
id from one of these two paths instead of computing one itself.
2. Periodic re-discovery. Every saved endpoint's models now refresh
automatically every 5 minutes in the background
(CUSTOM_MODEL_REDISCOVER_INTERVAL_MS, server.ts, registered the same way
as the Codex plan-usage poll it sits beside — this.cleanup.setInterval,
off under testMode), so a model the server starts or stops serving shows
up without another manual "Discover" click. The manual POST
.../discover-models route and the new refreshAllCustomModelHosts()
sweep (custom-model-routes.ts) now share one pure merge step
(applyDiscoveredModels: stamps lastDiscoveredAt, drops a defaultModelId
that no longer appears) rather than two copies that could drift. The
sweep is best-effort per host — one endpoint being unreachable on a
cycle never blocks the others — and re-reads the store before each
host's write, keyed by id, so a concurrent edit or delete from the
settings panel always wins over a sweep that started before it.
Tests: test/custom-model-endpoint-rediscovery.test.ts is a new, dedicated
file for the sweep (kept separate from custom-model-routes.test.ts because
that file's data dir is shared across every test in it — one temp HOME per
FILE, not per test — which would make a sweep-touches-every-host assertion
meaningless there). test/custom-model-run-menu-ui.test.ts gained a new
describe block driving the real picker modal through JSDOM: single-model
bypass, multi-model dialog with the default marked-not-chosen, picking a
row closes the modal and launches with that exact model, the endpoint
re-fetch, and the two "vanished by click time" toast paths.
Docs: docs/custom-model-endpoints.md, docs/wiki/Custom-Model-Endpoints.md,
docs/api-reference.md and CLAUDE.md's dense feature paragraph all updated
— the last of these also caught up two sentences that had gone stale after
the draft-review fixes landed (the picker routes through run() now, not a
raw run*() call).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
