Merge pull request #752: fix(custom-model): context limits, Codex model override, context discovery and picker filter

This commit is contained in:
Codeman maintainer committed 2026-11-04 16:51:56 +01:00
commit 3bcb55ce96
29 files changed
+1125 -102

No files matched your search

@@ -0,0 +1,5 @@
---
'aicodeman': patch
---
Custom endpoints now hand their context window to the CLIs that ask for it. OpenCode and Codex receive the discovered context length on injection (#749), and Codex launches on the selected custom model instead of its default (#750). Endpoint discovery reads each model's context size from `/v1/models` (with a sanity ceiling), with a per-endpoint "Default context length" field for gateways that answer nothing, and boot recovery keeps the saved figure (#754). The "Choose a model" picker also grew a search box that filters the rows as you type.
+30 -7
View File
@@ -1205,24 +1205,47 @@ data}` envelope on the wire — unwrap it the same way). Answers `[]` for a
boolean` reports whether one is stored, so a client can render "unchanged
if left blank" without ever holding the real value.
- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?,
authStyle?, defaultModelId? }` creates one. `id` must match
authStyle?, defaultModelId?, defaultContextLength? }` creates one. `id`
must match
`^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never
both (a real server hung indefinitely when sent both headers on one
request); `baseUrl` must be `http(s)`, carry no embedded credentials, and
request); `defaultContextLength` is a user-set fallback used when
assigning a model that discovery has no context length for (discovery
itself never writes it); `baseUrl` must be `http(s)`, carry no embedded
credentials, and
is refused if it points at (or resolves to) a link-local or
cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id.
- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey`
- `PUT /api/v1/model-endpoints/:id` updates one, also accepting
`defaultContextLength`. An **absent** `apiKey`
keeps the stored one rather than clearing it — the client never receives
the real value to resend deliberately unchanged, so omission is the only
way to say "leave it alone"; there is no way to clear a key back to unset
this way. `defaultModelId`, when set, must be one of that endpoint's own
this way. `defaultContextLength` behaves the opposite way: a PUT without
it clears the stored value, so send the field back to keep it.
`defaultModelId`, when set, must be one of that endpoint's own
`models` (`400 INVALID_INPUT` otherwise).
- `DELETE /api/v1/model-endpoints/:id` removes one.
- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's
own `GET /v1/models` and stores the result as `models`, updating
`lastDiscoveredAt`, plus (best-effort, only for a model llama-swap's own
response already reports loaded) `modelContextLengths` and `modelSizesGB`.
A `defaultModelId` that no longer appears in the fresh list is dropped
`lastDiscoveredAt`, plus (best-effort) `modelContextLengths`,
`modelSlotCounts` and `modelSizesGB`: every listing entry is read for a
stated context figure (`context_length`, then `context_window`) and a
size, BEFORE any loaded check; only the launch-command/`/props`
enrichment needs a model llama-swap's own response reports loaded. A
loaded model's context figure comes from llama-swap's own launch command
(`--fit-ctx` / `-c` / `--ctx-size`), falling back to `/props` (`n_ctx`,
`total_slots`) for whatever the command did not answer, and OVERWRITES the
listed figure for that model; the model's own listing
(`context_length`, then `context_window`) only ever fills the gap when
neither probe does, and is recorded as a single-slot figure —
llama-swap lists the PER-SLOT window, so whenever the listed figure is
the one taken the stored slot count is 1. A server with no `status`
field on any entry (plain llama.cpp, a non-llama-swap gateway) gets no
launch-command/`/props` enrichment at all, only whatever its listings
state. `modelContextLengths` falls back to the
endpoint's `defaultContextLength` when a session is assigned a model
discovery never learned a window for. A `defaultModelId` that no longer
appears in the fresh list is dropped
rather than carried forward invalid. Failures answer `422 OPERATION_FAILED`
with the underlying connection error, or a named egress refusal if the
resolved address turned out to be blocked. The same refresh also runs
+2 -2
View File
@@ -630,9 +630,9 @@ Adding the header changed the preamble, so `CODEMAN_PREAMBLE` was bumped (1.22.0
**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET <baseUrl>/v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×.
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/copilot/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
⚠️ **Two launch paths, chosen by mechanism, not preference**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id <id>` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume <id> || --session-id <id>` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. The dead-pane respawn shares this pin, see [Dead-pane respawn: the resume pin](#dead-pane-respawn-the-resume-pin). ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/<id>` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600.
⚠️ **Two launch paths, chosen by mechanism, not preference**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id <id>` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume <id> || --session-id <id>` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. The dead-pane respawn shares this pin, see [Dead-pane respawn: the resume pin](#dead-pane-respawn-the-resume-pin). ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/<id>` for pi/omp, grok's `[model.codeman-custom]` block name), and codex gets one too (`--model <bare modelId>`, its pattern widened to `model-plain-colon` for qwen3-style ids): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600.
⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`.
+43 -17
View File
@@ -67,15 +67,23 @@ configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one.
Endpoint management is admin-only in multi-user mode, same as remote/docker
hosts — these are machine-level infra, not per-user settings.
**Context length is discovered too, opportunistically and safely.** The plain
`GET /v1/models` response has no context-window field. Discovery only ever
looks for one for a model llama-swap's own response already reports
**Context length is discovered too, opportunistically and safely.** Older
servers state no context-window field in the plain `GET /v1/models`
response at all; current llama-swap states one per model (both
`context_length` and `context_window`) and gateways like Bifrost pass
`context_window` through — that listed figure is the *lowest*-precedence
source, used only when neither probe below answers, and because llama-swap
documents it as the PER-SLOT window, a model whose figure comes from the
listing is recorded with a slot count of 1 so the slot check never divides a
per-slot figure a second time. The probes are only ever looked for on a model
llama-swap's own response already reports
`status.value === "loaded"` for — never for an unloaded one, because
llama-swap treats `?model=` as a routing hint and asking about a model that
isn't loaded risks triggering an actual (slow, GPU-swapping) load as a side
effect of what should be read-only discovery. A server with no `status` field
on any entry at all (not llama-swap) gets no context-length enrichment,
rather than guessing. A model's previously-learned context length survives a
on any entry at all (not llama-swap) gets no launch-command/`/props`
enrichment, only whatever its listings state — rather than guessing. A
model's previously-learned context length survives a
later cycle where it wasn't the loaded one; it's dropped only once the model
disappears from the endpoint's list entirely. Stored per model in
`modelContextLengths` and applied automatically (see "Applying a model to a
@@ -286,6 +294,15 @@ hot-swap. A Claude session is relaunched with `--resume <conversation> ||
grok are relaunched with the `--model` value that selects the injected
provider (`custom/<modelId>` for pi and omp, `codeman-custom` for grok),
since for those three the config file alone does not switch the model.
**Codex is relaunched with `--model <modelId>` too**, the bare selected id
(not a provider selector): the launch param forces the session onto the
model its generated `config.toml` already selected, so a launch-time model
carried over from App Settings can never override the injected selection.
That launch arg must satisfy the codex entry's `model-plain-colon` token
pattern (a leading letter or digit, then letters, digits and `._-:/`, per
`stock.ts`), so an id outside that shape, e.g. LM Studio's
`org/model@q4_k_m`, is refused with a 400 at apply time now, where a
`config.toml` alone used to carry it.
**Remote (SSH) and Docker sessions are refused** (400) for now: their restart
reattaches the durable remote/in-container tmux rather than relaunching the
agent, so the selection would report success and change nothing.
@@ -293,14 +310,16 @@ agent, so the selection would report success and change nothing.
**Claude gets two more env vars when known/applicable, both declared on its
registry entry (`contextLengthVar`/`configDirVar`), not hardcoded here:**
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s discovered context
length (see the discovery section above) whenever one is known. Without
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s context length —
the discovered figure (see the discovery section above), falling back to
the endpoint's user-set `defaultContextLength` — whenever one is known.
Without
it, Claude Code assumes a large (200k) window for any unrecognized custom
model id and never compacts, which reliably overflows a much smaller real
local context — confirmed live: a stock ~33.7K-token system prompt against
a 16384-token llama-swap model failed with `exceeds the available context
size`. No entry for the model in `modelContextLengths` means the var is
simply omitted, never a guess. ⚠️ **This var only affects when Claude
size`. Neither a `modelContextLengths` entry nor a `defaultContextLength`
for the model means the var is simply omitted, never a guess. ⚠️ **This var only affects when Claude
Code compacts conversation _history_ — it cannot fix a model whose real
context is smaller than Claude Code's own fixed per-turn overhead**
(system prompt + tool schemas, empirically ~36.4K tokens, confirmed live
@@ -441,15 +460,17 @@ Both apply routes (the restart route and the one-shot `POST
/api/quick-start`) now check for this **before** launching or restarting
anything, gated on the CLI's registry entry declaring a `contextLengthVar`
(currently only claude — the check is a no-op for every other CLI by
construction, never a hardcoded mode check). If the model's discovered
context (`modelContextLengths`, from discovery above) is below
`CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
construction, never a hardcoded mode check). If the model's context (the
discovered figure from `modelContextLengths`, or the endpoint's
`defaultContextLength` when discovery never learned one for that model) is
below `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
~36.4K overhead), the response is `{requiresContextWarning: true, modelId,
contextLength, minSafeContextTokens}` (plus `slotCount` when the model runs more than one parallel slot) instead of applying — nothing is
restarted or created yet. A context length that was never discovered at
all skips the check entirely (nothing to compare, so it fails open rather
than warning on every model an endpoint hasn't reported a size for).
The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
restarted or created yet. A model with neither a discovered figure nor a
`defaultContextLength` skips the check entirely (nothing to compare, so it
fails open rather than warning on every model an endpoint hasn't reported a
size for).
The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. ⚠️ `defaultContextLength` is the model's whole window, never a per-slot figure, so a default small enough to split under the slot count trips that slot division exactly like a discovered figure does. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
The Run-menu picker shows this as an in-app modal
(`#customModelContextWarningModal`, matching the llama-swap conflict
@@ -526,7 +547,12 @@ for '<id>' not found. Defaulting to fallback metadata...` on launch —
and fabricating a fake catalog entry would mean copying the _shape_ of
OpenAI's own proprietary schema (their per-model system-prompt content
included) for a warning that doesn't otherwise affect behavior — not
something to build into discovery.
something to build into discovery. ⚠️ The fallback-metadata warning is
NOT about the context window going un-overridden. Codex does now get a
real context-window override: when a figure is known the generated
`config.toml` sets the top-level `model_context_window` to it and
`model_auto_compact_token_limit` to 75% of it (floored), so Codex
compacts at that point instead of trusting its fallback guess.
- **Gemini** — fails with `Invalid auth method selected`, traced to an
undocumented `GATEWAY` auth path gemini-cli selects once
`GOOGLE_GEMINI_BASE_URL` is set. Unresolved after real investigation
+17 -9
View File
@@ -27,14 +27,20 @@ hosts — these are machine-level infra, not a per-user setting.
shows up without another manual click of **Discover**. One endpoint being unreachable on a
given cycle (powered off, wrong network) never blocks the others from refreshing.
**Context length is picked up automatically where it can be, safely.** Against a
llama.cpp/llama-swap server, discovery also learns each _currently loaded_ model's real
context window and applies it to the launched session (Claude Code today — see below), so
the harness stops assuming a large default window for a model name it doesn't recognise and
overflowing a much smaller real one. It's deliberately never probed for a model that isn't
already loaded, since asking a llama-swap server about an unloaded model can trigger an
actual, slow model swap as a side effect — a model just not currently loaded keeps whatever
context length an earlier cycle already learned for it instead.
**Context length is picked up automatically where it can be, safely.** Every model's own
listing entry is read for a stated context figure (`context_length`, then `context_window`,
which current llama.cpp, llama-swap and gateway-style servers state to varying degrees);
against a llama-swap server the stronger figures from its launch command and from llama.cpp's
`/props` probe win where they exist, and those probes only ever run for a model the server
already reports loaded, since asking about an unloaded model can trigger an actual, slow model
swap as a side effect. A model that states no figure keeps whatever an earlier cycle learned:
the stored figure always beats the endpoint's **Default context length** field
(`modelContextLengths[id] ?? defaultContextLength`), which only applies where no figure was
ever learned (that number is the model's whole window, never a per-slot figure). A model
neither covers simply gets no override, never a guess. The window is applied to the launched session by the harnesses
that can take it: Claude Code today (see below), OpenCode as the model's context limit, and
Codex as a `model_context_window` entry in its generated config file, which also sets Codex's
own auto-compact point via `model_auto_compact_token_limit` at 75% of the window.
## Running a session against one
@@ -48,7 +54,9 @@ pointed at.
**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
the session launches straight away on that model — nothing to choose. With two or more, a
small dialog asks which one to use for this launch before starting the session; the
small dialog asks which one to use for this launch before starting the session; a search box
at the top narrows a long list with a case-insensitive substring match, so a server hosting
hundreds of models stays usable. The
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
use a different one without changing the saved default. The list is not raw discovery order
either: the model llama-swap reports loaded and ready is moved to the top and tagged
+5
View File
@@ -21,6 +21,7 @@ export type TokenPattern =
| 'model'
| 'model-claude'
| 'model-plain'
| 'model-plain-colon'
| 'model-pi'
| 'id'
| 'id-dotted'
@@ -47,6 +48,10 @@ const PATTERNS: Record<TokenPattern, RegExp> = {
// copilot --model. Same characters as `model`, but a leading alphanumeric: the value follows a
// flag, so `--yolo` must not pass as a model id and become its own flag (the allowAll clamp bypass).
'model-plain': /^[a-zA-Z0-9][a-zA-Z0-9._\-/]*$/,
// codex --model, and the value a custom-model `launchModel` is forced onto it. Same
// shape as `model-plain` (leading alphanumeric, so a flag cannot ride the flag) plus
// `:` — Ollama-style ids like `qwen3:32b` must survive onto the command line.
'model-plain-colon': /^[a-zA-Z0-9][a-zA-Z0-9._\-/:]*$/,
// buildPiCommand — `:` for a thinking suffix (`sonnet:high`), `/` for `provider/id`
'model-pi': /^[a-zA-Z0-9._\-/:]+$/,
// opencode --session, codex resume
+7 -3
View File
@@ -628,7 +628,10 @@ const CODEX: CliEntry = {
params: {
bypassApprovals: { type: 'bool' },
animations: { type: 'bool' },
model: { type: 'token', pattern: 'model' },
// `model-plain-colon`, not `model`: the custom-model feature forces the selected id
// onto this param (`launchModel: '{modelId}'` below), and Ollama-style ids carry a
// `:` (`qwen3:32b`). A leading alphanumeric still keeps `--yolo` from riding a flag.
model: { type: 'token', pattern: 'model-plain-colon' },
reasoningEffort: { type: 'enum', values: [...CODEX_REASONING_EFFORTS] },
resumeId: { type: 'token', pattern: 'id' },
},
@@ -841,6 +844,7 @@ const CODEX: CliEntry = {
dirEnvVar: 'CODEX_HOME',
fileName: 'config.toml',
template: 'codex-toml',
launchModel: '{modelId}',
},
// CODEX_HOME already matches the CODEX_ allowedPrefix above, so it was ALREADY
// reachable via plain envOverrides before this feature existed. It is arguably
@@ -1921,8 +1925,8 @@ const COPILOT: CliEntry = {
// custom provider and drops the GitHub sign-in requirement; COPILOT_PROVIDER_TYPE defaults to
// "openai", which is what a Codeman custom endpoint is (llama.cpp, vLLM, Ollama).
// COPILOT_MODEL names the model and COPILOT_PROVIDER_MODEL_ID / _WIRE_MODEL default to it.
// The OpenAI-style route is `<base>/v1/chat/completions`, so the base gets the `/v1` suffix
// the same way deepseek's does. All four already match the COPILOT_ prefix allowlist, so
// The OpenAI-style route is `<base>/v1/chat/completions`, so the base gets the `/v1`
// suffix via appendV1Suffix below. All four already match the COPILOT_ prefix allowlist, so
// they are privileged env keys below: a non-granted multi-user owner must not be able to
// redirect the session's endpoint or credentials through plain envOverrides.
customModelInjection: {
+12 -16
View File
@@ -812,9 +812,11 @@ export interface CliCapabilities {
* for pi and omp (`--model custom/<id>`, or the CLI stays on its own default
* provider and reports "No API key found for the selected model") or for
* grok (`--model codeman-custom`, the `[model.<name>]` block the config
* declares). Absent = the config alone selects the model (claude's env vars,
* opencode's blob, codex's top-level `model` key). Applied by the session's
* respawn options through the entry's `legacyConfigField`, never by id.
* declares), and codex carries its bare selected id (`{modelId}`, an id the
* entry's `model` token pattern must be able to carry). Absent = the config
* alone selects the model (claude's env vars, opencode's blob). Applied by
* the session's respawn options through the entry's `legacyConfigField`,
* never by id.
*
* `contextLengthVar` (env kind only): the env var a discovered per-model context-window
* size is written to when known (claude's `CLAUDE_CODE_MAX_CONTEXT_TOKENS`) — without it,
@@ -856,19 +858,13 @@ export interface CliCapabilities {
* reuses that file.
*
* `appendV1Suffix` (env kind only): the raw `endpoint.baseUrl` gets `withV1Suffix()`
* applied before being written to `baseUrlVar`, instead of being used verbatim.
* DeepSeek needs this and claude/gemini must NOT get it — a per-CLI asymmetry confirmed
* by reading each SDK's own request-building source, not assumed: DeepSeek Harness's
* bundled `@deepseek-ai/dsh-llm-deepseek` concatenates `${connection.baseURL}/chat/
* completions` with no `/v1` insertion of its own (its real public API base,
* `https://api.deepseek.com`, expects the caller's base URL to already carry any
* needed prefix), while llama-swap/llama.cpp only ever serves the OpenAI-conventional
* `/v1/chat/completions` — confirmed live: a bare `POST <baseUrl>/chat/completions`
* 404s, `POST <baseUrl>/v1/chat/completions` succeeds, and the harness's own error
* message template (`DeepSeek API error (HTTP ${status})`) reproduces the exact
* `HTTP_404` this feature originally shipped with unexplained. Claude Code's own SDK,
* by contrast, was already confirmed working end-to-end against the RAW `baseUrl` with
* no suffix — appending one there would be wrong, not just redundant.
* applied before being written to `baseUrlVar`, instead of being used verbatim. Only
* deepseek and copilot declare it — both speak to `<base>/v1/chat/completions` and a
* gateway saves the bare root. claude/gemini must NOT get it: Claude Code's own SDK was
* confirmed working end-to-end against the RAW `baseUrl`, so appending a suffix there
* would be wrong, not just redundant. The per-session config-file recipes do not need
* this flag at all — `renderConfigFile()` applies the suffix inside every template, so
* no configDir entry carries a raw base URL.
*/
customModelInjection:
| {
+35 -8
View File
@@ -52,18 +52,22 @@ export interface CustomModelHost {
defaultModelId?: string;
/**
* Discovered context-window size (tokens) per model id, keyed by the same strings as
* `models`. Populated opportunistically during discovery (`custom-model-routes.ts`) from
* llama.cpp/llama-swap's `GET /props?model=<id>` — the plain OpenAI-shaped `/v1/models`
* response has no such field. Only ever probed for a model the server already reports as
* loaded (llama-swap's `status.value === 'loaded'`); an unloaded one is deliberately never
* probed, since llama-swap treats `/props?model=` as a routing hint that can trigger an
* actual (slow, GPU-swapping) model load as a side effect of merely asking. A model this
* has no entry for simply gets no context-length env override applied — never a guess.
* `models`. Populated only by discovery (`custom-model-routes.ts`), from the strongest
* source the server offers: the llama-swap launch command reported by `GET /running`
* (the source of truth for a loaded model), then llama.cpp's `GET /props?model=<id>`
* `n_ctx` (only ever probed for a model the server already reports as loaded, since
* llama-swap treats `/props?model=` as a routing hint that can trigger an actual
* (slow, GPU-swapping) model load as a side effect of merely asking), then the model's
* own `GET /v1/models` entry (`context_length`, else `context_window`, which current
* llama-swap, Bifrost and OpenRouter-style servers each state to varying degrees). A
* model with no entry here falls back to the endpoint's `defaultContextLength`; with
* neither, it simply gets no context-length env override applied, never a guess.
*/
modelContextLengths?: Record<string, number>;
/**
* Discovered number of parallel server slots per model id, keyed like `modelContextLengths`
* and populated on the same loaded-models-only basis. Read from the launch command's
* but populated only for models the server reports loaded (a listing never states a slot
* count, unlike `contextLengths`, which every entry can supply). Read from the launch command's
* `--parallel`/`-np` when it states one, otherwise from llama.cpp's own `/props`
* `total_slots` (a llama-server started without `--parallel` picks the count itself, so
* the command cannot say). llama-server splits its context across these slots (by default
@@ -88,6 +92,29 @@ export interface CustomModelHost {
* server-side check relies on.
*/
modelSizesGB?: Record<string, number>;
/**
* User-set fallback context length (tokens), applied via
* `resolveCustomModelContextLength()` at every read site (the swap route, the
* quick-start route, boot recovery) only when `modelContextLengths` has no entry for
* the selected model (a gateway that states no per-model figure, a model never loaded
* yet). It is the model's WHOLE window, never a per-slot figure: the Claude floor
* check divides it by any stored slot count exactly like a discovered figure. Never
* written by discovery, so a value set in the endpoint editor survives every
* re-discovery sweep; unset keeps the long-standing behaviour of letting each CLI
* assume its own default.
*/
defaultContextLength?: number;
}
/**
* The context window to apply for `modelId` on `host`: the discovered per-model figure when
* discovery has one, else the endpoint's user-set `defaultContextLength`, else undefined
* (no override applied, never a guess). ONE helper for every read site — the swap route,
* the quickstart route and boot recovery each re-write the CLI's config files, and a site
* that forgot the fallback silently loses the endpoint default on the next respawn.
*/
export function resolveCustomModelContextLength(host: CustomModelHost, modelId: string): number | undefined {
return host.modelContextLengths?.[modelId] ?? host.defaultContextLength;
}
export function customModelHostsPath(configDir: string): string {
+37 -6
View File
@@ -136,12 +136,12 @@ export function buildCustomModelInjection(
}
case 'configContentEnv': {
const content = renderConfigContent(cap.template, endpoint, modelId, apiKey);
const content = renderConfigContent(cap.template, endpoint, modelId, apiKey, contextLength);
return withLaunchModel({ kind: 'env', envOverrides: { [cap.envVar]: content } }, cap.launchModel, modelId);
}
case 'configDir': {
const { content, extraEnv } = renderConfigFile(cap.template, endpoint, modelId, apiKey);
const { content, extraEnv } = renderConfigFile(cap.template, endpoint, modelId, apiKey, contextLength);
return withLaunchModel(
{ kind: 'configDir', dirEnvVar: cap.dirEnvVar, files: [{ relPath: cap.fileName, content }], extraEnv },
cap.launchModel,
@@ -164,24 +164,48 @@ function withLaunchModel<T extends EnvInjection | ConfigDirInjection>(
return { ...result, launchModel: template.split('{modelId}').join(modelId) };
}
/**
* One quarter of the window is held back for the reply, capped at 8192 tokens (a small local
* model's window must not lose a disproportionate slice to the reservation). Returns undefined
* for an unknown or non-finite size — opencode then keeps its own default guess, which is the
* pre-fix behaviour; never invent a limit here.
*/
function opencodeModelLimit(contextLength: number | undefined): { context: number; output: number } | undefined {
if (contextLength === undefined || !Number.isFinite(contextLength)) return undefined;
const total = Math.trunc(contextLength);
if (total <= 0) return undefined;
const output = Math.max(1, Math.min(8192, Math.floor(total / 4)));
// `limit.context` is the FULL window as far as opencode is concerned — it reserves for
// the reply itself — so the discovered total is passed through, never a shrunk figure.
return { context: total, output };
}
function renderConfigContent(
template: 'opencode-json',
endpoint: CustomModelEndpoint,
modelId: string,
apiKey: string
apiKey: string,
contextLength?: number
): string {
switch (template) {
case 'opencode-json':
case 'opencode-json': {
// `limit` carries the discovered context window (issue #749): without it opencode assumes
// a large default window for an unrecognized model id and eventually overflows a much
// smaller local server. `context` is the FULL window and opencode reserves for the reply
// inside it itself, so `opencodeModelLimit` passes the real total through and only
// `output` gets the capped reservation.
const limit = opencodeModelLimit(contextLength);
return JSON.stringify({
$schema: 'https://opencode.ai/config.json',
provider: {
custom: {
options: { baseURL: withV1Suffix(endpoint.baseUrl), apiKey },
models: { [modelId]: {} },
models: { [modelId]: limit ? { limit } : {} },
},
},
model: `custom/${modelId}`,
});
}
}
}
@@ -195,7 +219,8 @@ function renderConfigFile(
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' | 'grok-toml',
endpoint: CustomModelEndpoint,
modelId: string,
apiKey: string
apiKey: string,
contextLength?: number
): { content: string; extraEnv?: Record<string, string> } {
const baseUrl = withV1Suffix(endpoint.baseUrl);
switch (template) {
@@ -217,6 +242,12 @@ function renderConfigFile(
const content = [
`model = ${quoted(modelId)}`,
`model_provider = "custom"`,
...(contextLength !== undefined && Number.isFinite(contextLength) && contextLength > 0
? [
`model_context_window = ${Math.trunc(contextLength)}`,
`model_auto_compact_token_limit = ${Math.max(1, Math.floor(contextLength * 0.75))}`,
]
: []),
'',
'[model_providers.custom]',
`name = "Custom Endpoint"`,
+2 -1
View File
@@ -2724,7 +2724,8 @@ export class Session extends EventEmitter {
/**
* Force the custom-model selection's `launchModel` (pi/omp `custom/<id>`, grok's
* `[model.<name>]` block name) onto the CLI's `model` launch param. Where that param
* `[model.<name>]` block name, codex's bare selected id) onto the CLI's `model`
* launch param. Where that param
* lives is registry DATA — the entry's `legacyConfigField` (`piConfig`, `grokConfig`,
* ...) or the top-level `model` for an entry that declares none — so this stays a
* generic reader rather than a branch per CLI. Applied on the OPTIONS only: the stored
+6
View File
@@ -615,6 +615,9 @@
'API key': 'API 密钥',
'Optional. Left blank on edit keeps the existing key.': '可选。编辑时留空将保留现有密钥。',
'Auth header': '认证请求头',
'Default context length (tokens)': '默认上下文长度(token 数)',
"Used for models whose size cannot be discovered. The model's whole window; the Claude floor check divides it by any stored slot count. Leave empty to let each CLI assume its own default.":
'用于无法自动发现上下文大小的模型。这是模型的完整窗口;Claude 下限检查会将其除以已存储的槽位数。留空则让各个 CLI 使用其自身的默认值。',
'Never send both — some servers hang indefinitely.': '切勿同时发送两者 — 部分服务器会因此无限期挂起。',
'Authorization: Bearer (default)': 'Authorization: Bearer(默认)',
'api-key header (Azure)': 'api-key 请求头(Azure)',
@@ -627,6 +630,9 @@
'Last used': '上次使用',
'That endpoint no longer exists': '该端点已不存在',
'No models discovered for this endpoint yet': '此端点尚未发现任何模型',
// Model picker search box (index.html placeholder/aria-label and the no-matches line)
'Search models': '搜索模型',
'No models match': '没有匹配的模型',
'Subagent Options': '子智能体选项',
'Enable Tracking': '启用跟踪',
'Active Tab Only': '仅活动标签页',
+12
View File
@@ -1033,6 +1033,14 @@
</div>
<div class="modal-body">
<p class="form-hint" id="customModelPickHint"></p>
<!-- Case-insensitive filter over the rendered rows (app.filterCustomModelPick):
a display layer only, the rows themselves are never re-rendered. Reset and
focused on every open, same in-memory rule as the rail's search box. -->
<input type="search" id="customModelPickSearch" class="session-sidebar-filter-input custom-model-pick-search"
placeholder="Search models" aria-label="Search models"
autocomplete="off" autocapitalize="off" spellcheck="false" enterkeyhint="search"
oninput="app.filterCustomModelPick(this.value)">
<p class="form-hint custom-model-pick-no-matches" id="customModelPickNoMatches" role="status" hidden>No models match</p>
<div id="customModelPickList" class="run-mode-custom-models"></div>
</div>
</div>
@@ -2791,6 +2799,10 @@
<div class="set-row-text"><span class="set-row-label">Base URL</span></div>
<input type="text" id="customModelHostBaseUrl" class="set-input" placeholder="http://192.168.1.50:8080">
</div>
<div class="set-row has-field">
<div class="set-row-text"><span class="set-row-label">Default context length (tokens)</span><span class="set-row-desc">Used for models whose size cannot be discovered. The model's whole window; the Claude floor check divides it by any stored slot count. Leave empty to let each CLI assume its own default.</span></div>
<input type="number" id="customModelHostDefaultContextLength" class="set-input" min="1" max="100000000" step="1" placeholder="131072">
</div>
<div class="set-row has-field">
<div class="set-row-text"><span class="set-row-label">API key</span><span class="set-row-desc">Optional. Left blank on edit keeps the existing key.</span></div>
<input type="password" id="customModelHostApiKey" class="set-input" autocomplete="new-password">
+46 -1
View File
@@ -1088,13 +1088,58 @@ Object.assign(CodemanApp.prototype, {
const promotion = m === currentlyLoaded ? 'Currently loaded' : m === lastUsed ? 'Last used' : null;
const tags = [promotion, m === host.defaultModelId ? 'Default' : null].filter(Boolean);
const arg = escapeHtml(JSON.stringify(m));
// data-model-id is the filter's haystack (escapeHtml covers `"` as well as
// `<>&`, so a hostile model id cannot break out of the attribute); the
// filter reads it back through dataset, which decodes it.
return `
<button class="run-mode-option" onclick="app.chooseCustomModelAndRun(${arg})">
<button class="run-mode-option" data-model-id="${escapeHtml(m)}" onclick="app.chooseCustomModelAndRun(${arg})">
<span class="run-mode-dot ${escapeHtml(mode)}"></span>${escapeHtml(m)}${tags.map((t) => ` <span class="set-scope">${escapeHtml(t)}</span>`).join('')}
</button>`;
})
.join('');
modal.classList.add('active');
// A fresh open always starts unfiltered with the caret in the search box — the
// list was just rebuilt, so clearing the input is the whole reset; focus is the
// user's next keystroke away. Inside the generation guard above, so a
// superseded probe can neither focus nor clear a modal that now shows another
// endpoint's models.
const search = document.getElementById('customModelPickSearch');
if (search) {
search.value = '';
// readmymind-ui.js's rule: on touch devices never focus an input unprompted, or
// the OS keyboard pops over the very list the picker just opened to show. Desktop
// keeps the caret in the box, one keystroke from filtering.
if (!(typeof MobileDetection !== 'undefined' && MobileDetection.isTouchDevice())) {
search.focus();
}
}
this.filterCustomModelPick('');
},
/**
* View filter for the open "Choose a model" picker: case-insensitive substring
* over each row's model id, driven from the search input's own `oninput`.
* Deliberately toggles `hidden` on the already-rendered buttons rather than
* re-rendering from a second copy of the model list: the loaded/last-used
* promotion, the Default pill, the escaped inline click argument and the async
* generation guard all then stay exactly where they are, and clearing the
* query restores the list byte for byte. ⚠️ `.run-mode-option` carries its own
* `display: flex` (styles.css, restated in mobile.css), which outranks the UA
* `[hidden]` rule, so styles.css re-asserts `display: none !important` for
* these rows — without it the filter would silently do nothing.
*/
filterCustomModelPick(value) {
const list = document.getElementById('customModelPickList');
if (!list) return;
const query = String(value || '').trim().toLowerCase();
let visible = 0;
for (const btn of list.querySelectorAll('button[data-model-id]')) {
const match = !query || btn.dataset.modelId.toLowerCase().includes(query);
btn.hidden = !match;
if (match) visible += 1;
}
const noMatches = document.getElementById('customModelPickNoMatches');
if (noMatches) noMatches.hidden = visible > 0;
},
closeCustomModelPickModal() {
+16
View File
@@ -3875,6 +3875,9 @@ Object.assign(CodemanApp.prototype, {
document.getElementById('customModelHostApiKey').value = ''; // the server never returns the real value (apiKeySet is a bool)
document.getElementById('customModelHostApiKey').placeholder = host?.apiKeySet ? '•••••••• (unchanged if left blank)' : '';
document.getElementById('customModelHostAuthStyle').value = host?.authStyle || 'bearer';
// User-set fallback, round-tripped verbatim (discovery never touches it, so the
// cached copy is authoritative); blank means "no default", cleared on save.
document.getElementById('customModelHostDefaultContextLength').value = host?.defaultContextLength ?? '';
this._populateCustomModelDefaultSelect(host);
document.getElementById('customModelHostEditor').style.display = '';
},
@@ -3900,6 +3903,18 @@ Object.assign(CodemanApp.prototype, {
const baseUrl = document.getElementById('customModelHostBaseUrl').value.trim();
const apiKeyInput = document.getElementById('customModelHostApiKey').value;
const authStyle = document.getElementById('customModelHostAuthStyle').value;
// Blank means "no default" (key omitted, so PUT clears it); a value must satisfy
// the same bound the server schema enforces, caught here for a friendlier message.
const defaultContextLengthInput = document.getElementById('customModelHostDefaultContextLength').value.trim();
let defaultContextLength;
if (defaultContextLengthInput !== '') {
const parsed = Number(defaultContextLengthInput);
if (!Number.isInteger(parsed) || parsed < 1 || parsed > 100000000) {
this.showToast('Default context length must be a whole number of tokens between 1 and 100,000,000', 'warning');
return;
}
defaultContextLength = parsed;
}
const defaultModelId = document.getElementById('customModelHostDefaultModel').value || undefined;
if (!id || !label || !baseUrl) {
this.showToast('Id, label and base URL are all required', 'warning');
@@ -3920,6 +3935,7 @@ Object.assign(CodemanApp.prototype, {
baseUrl,
authStyle,
defaultModelId,
defaultContextLength,
apiKey: apiKeyInput || undefined,
models: existing?.models,
lastDiscoveredAt: existing?.lastDiscoveredAt,
+35
View File
@@ -7814,6 +7814,41 @@ html[data-skin="og"] .welcome-primary:hover {
opacity: 0.8;
}
/* The picker's search box reuses .session-sidebar-filter-input's theme-token
styling (same reasoning as the .set-scope pill above) — only its own spacing
and the no-matches line are added here. 0.35rem is the standard gap the
modal's neighbouring form fields already use (`.form-hint`'s own margin-top):
the global `* { margin: 0 }` reset left the box sitting flush under the CLI
hint line above it, and the old 0.5rem read as too big a gap above the list
(compounded by the always-visible no-matches line the rule below fixes). */
#customModelPickModal .custom-model-pick-search {
margin-top: 0.35rem;
margin-bottom: 0.35rem;
}
#customModelPickModal .custom-model-pick-no-matches {
text-align: center;
padding: 1rem 0;
}
/* Same class of bug as the row rule below: the no-matches line carries
`.form-hint`, whose `display: block` outranks the UA `[hidden]` rule, so
`filterCustomModelPick()` toggling `hidden` changed nothing on screen and
"No models match" showed while every model was listed with an empty search
box. The `[hidden]` re-assertion makes the attribute hide the line again. */
#customModelPickModal .custom-model-pick-no-matches[hidden] {
display: none !important;
}
/* A filtered-out row's `hidden` must actually hide it: `.run-mode-option` sets
`display: flex` (and mobile.css restates the class in a media query), which
outranks the UA `[hidden] { display: none }` rule, so the filter would
silently do nothing without this re-assertion. Scoped to this modal's rows;
the Run menu's own .run-mode-option entries never take `hidden`. */
#customModelPickModal .run-mode-option[hidden] {
display: none !important;
}
/* Custom Model Endpoint Profiles: llama-swap model-swap confirmation — replaces a native
confirm() popup (docs/custom-model-endpoints-plan.md) so it looks and feels like the
rest of the app instead of a browser chrome dialog. Shares the context-window-too-small
+64 -9
View File
@@ -21,7 +21,7 @@ import { isMultiUserMode } from '../../config/multiuser.js';
import { getDataDir } from '../../config/instance.js';
import { isBlockedWebviewUrl } from '../webview-egress-policy.js';
import { egressBlockedReason, webviewFetch } from '../webview-egress.js';
import { CustomModelHostSchema } from '../schemas.js';
import { CustomModelHostSchema, MAX_DISCOVERED_CONTEXT_TOKENS } from '../schemas.js';
import { readCustomModelHosts, writeCustomModelHosts, type CustomModelHost } from '../../custom-model-hosts.js';
import type { CliEntry } from '../../config/cli-registry/types.js';
@@ -158,9 +158,14 @@ function authHeaders(host: Pick<CustomModelHost, 'apiKey' | 'authStyle'>): Recor
export interface DiscoveryResult {
models: string[];
/** See `CustomModelHost.modelContextLengths` — only ever populated for models already loaded. */
/**
* See `CustomModelHost.modelContextLengths`: populated from a model's own `/v1/models`
* entry (`context_length`, else `context_window`) whenever the server states one, and
* from the launch command or `/props` for loaded models, which overwrite (and so win
* over) any listed figure for the same model.
*/
contextLengths: Record<string, number>;
/** See `CustomModelHost.modelSlotCounts`; populated on the same loaded-models-only basis as `contextLengths`. */
/** See `CustomModelHost.modelSlotCounts`; populated only for models the server reports loaded (launch command or `/props`). No listing states a slot count, unlike `contextLengths`. */
slotCounts: Record<string, number>;
/** See `CustomModelHost.modelSizesGB` — populated for every model whose own listing states one. */
sizesGB: Record<string, number>;
@@ -182,6 +187,20 @@ function parseSizeGB(description: unknown): number | undefined {
return Number.isFinite(size) && size > 0 ? size : undefined;
}
/**
* Best-effort: reads a context length (tokens) stated by a model's OWN entry in
* `GET /v1/models`. OpenRouter-style servers call the field `context_length`; current
* llama-swap sends both `context_length` and `context_window` per model, and current
* Bifrost (a gateway in front of llama-swap) passes `context_window` through. Older
* servers send neither, and a server that states nothing gets no entry, never a guess.
* Non-numeric, zero, negative, fractional or over-the-ceiling values are ignored, the
* same rule the stored field's schema enforces: a wrong figure here is worse than none.
*/
function parseListedContextLength(value: unknown): number | undefined {
if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) return undefined;
return value <= MAX_DISCOVERED_CONTEXT_TOKENS ? value : undefined;
}
/**
* Best-effort: fetches `GET /props?model=<id>` (llama.cpp-native, llama-swap-proxied) for
* ONE already-loaded model and pulls its `n_ctx` and `total_slots` out. Never called for a model that
@@ -312,25 +331,42 @@ async function discoverModels(
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const body = (await res.json()) as {
data?: Array<{ id?: unknown; status?: { value?: unknown }; description?: unknown }>;
data?: Array<{
id?: unknown;
status?: { value?: unknown };
description?: unknown;
context_length?: unknown;
context_window?: unknown;
}>;
};
const entries = body.data ?? [];
const models = entries.map((m) => m.id).filter((id): id is string => typeof id === 'string' && id.length > 0);
const sizesGB: Record<string, number> = {};
// The lowest-precedence context source, and the only one a gateway without /running or
// /props of its own (e.g. Bifrost) can offer: the figure a model's own listing states
// (`context_length`, else `context_window`). The loaded-model pass below OVERWRITES
// these for whatever it reads from the launch command or /props, so precedence per
// model stays: `/running` cmd, then `/props`, then this listed figure.
const contextLengths: Record<string, number> = {};
for (const entry of entries) {
if (typeof entry.id !== 'string' || !entry.id) continue;
const size = parseSizeGB(entry.description);
if (size !== undefined) sizesGB[entry.id] = size;
const listed = parseListedContextLength(entry.context_length) ?? parseListedContextLength(entry.context_window);
if (listed !== undefined) contextLengths[entry.id] = listed;
}
// llama-swap-specific, feature-detected: a server that never mentions `status` on ANY
// entry gets no context-length enrichment at all, rather than treating "no status field"
// as "assume unloaded" — either reading is a guess, and skipping is the safe one, since
// fetchContextLength must only ever run against a model this server itself calls loaded.
// entry gets no launch-command/props context enrichment at all, rather than treating
// "no status field" as "assume unloaded"; either reading is a guess, and skipping is
// the safe one, since fetchContextLength must only ever run against a model this server
// itself calls loaded. Listed figures above are unaffected: reading them triggers nothing.
const hasStatusField = entries.some((m) => m && typeof m === 'object' && 'status' in m);
const contextLengths: Record<string, number> = {};
const slotCounts: Record<string, number> = {};
// Ids whose figure THIS round's loaded-model probe answered — the listed figure remains
// the source only for the ids NOT in this set (see the per-slot correction below).
const ctxFromProbe = new Set<string>();
if (hasStatusField) {
const loadedIds = entries
.filter((m) => m.status && typeof m.status === 'object' && (m.status as { value?: unknown }).value === 'loaded')
@@ -364,11 +400,30 @@ async function discoverModels(
// merges onto what was learned before).
if (cmdSlots !== undefined && ctxFromProps) slots = 1;
}
if (ctx !== undefined) contextLengths[id] = ctx;
// Assigning only when a probe answered keeps the listed figure (if any) when
// neither the launch command nor /props stated one, and replaces it with the
// stronger source whenever they did: cmd > /props > listed, never the reverse.
if (ctx !== undefined) {
contextLengths[id] = ctx;
ctxFromProbe.add(id);
}
if (slots !== undefined) slotCounts[id] = slots;
}
}
}
// llama-swap documents the LISTED `context_length` as the PER-SLOT window, so whenever
// the listed figure is the one taken this round — the model was not loaded, or the
// probe answered nothing — its slot count is 1 by definition: the floor check must NOT
// divide a per-slot figure a second time. Without recording 1 here, the count a loaded
// round stored survives the merge (applyDiscoveredModels keeps what a later round does
// not overwrite): 262144 @ 4 slots stored, model unloaded, listed 65536 → 65536 with 4
// slots → a false 16384-per-slot warning. Even a count this round's launch command
// parsed must yield — pairing a per-slot listed figure with the cmd's --parallel is the
// same double division. Same rule the /props branch above applies. Ids whose figure the
// probe DID answer keep their probe-derived count untouched.
for (const id of Object.keys(contextLengths)) {
if (!ctxFromProbe.has(id)) slotCounts[id] = 1;
}
return { models, contextLengths, slotCounts, sizesGB };
}
+28 -15
View File
@@ -70,8 +70,9 @@ import {
SessionWaitOutputQuerySchema,
CustomModelSelectionSchema,
} from '../schemas.js';
import { readCustomModelHosts } from '../../custom-model-hosts.js';
import { readCustomModelHosts, resolveCustomModelContextLength } from '../../custom-model-hosts.js';
import { applyCustomModelInjection, removeConfigDir } from '../../custom-model-injection-apply.js';
import { buildCustomModelInjection } from '../../custom-model-injection.js';
import {
getLlamaSwapStatus,
triggerLlamaSwapLoad,
@@ -1541,7 +1542,10 @@ export function registerSessionRoutes(
if (!endpoint) {
return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
}
const contextLength = endpoint.modelContextLengths?.[body.modelId];
// Discovery's per-model figure wins; the endpoint's user-set default covers models
// whose size discovery could not determine (a gateway that states none, say). With
// neither, the existing "no override, CLI assumes its own default" behaviour stands.
const contextLength = resolveCustomModelContextLength(endpoint, body.modelId);
const slotCount = endpoint.modelSlotCounts?.[body.modelId];
// Some CLIs (today: only claude) carry enough of their own fixed system-prompt/tool-
@@ -1608,10 +1612,28 @@ export function registerSessionRoutes(
}
// A CLI whose config alone cannot select the model also gets its `model` launch param
// forced (pi/omp `custom/<id>`, grok's block name). The argv engine DROPS a token that
// forced (pi/omp `custom/<id>`, grok's block name, and codex now carries its bare
// selected id too). The argv engine DROPS a token that
// fails its pattern rather than quoting it, which would silently launch the CLI on its
// own default provider again, so refuse an id the pattern cannot carry up front.
// The check runs on the PURE build result, before applyCustomModelInjection below:
// the forced launch id only comes out of that build, and writing first would mean a
// refused id already replaced the config dir a live codex session uses as CODEX_HOME.
const modelSpec = entry.launch.params.model;
const injection = buildCustomModelInjection(entry, endpoint, body.modelId, contextLength);
if (injection.kind === 'unsupported') {
return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`);
}
if (
injection.launchModel !== undefined &&
modelSpec?.type === 'token' &&
!matchesPattern(modelSpec.pattern, injection.launchModel)
) {
return createErrorResponse(
ApiErrorCode.INVALID_INPUT,
`Model id ${JSON.stringify(body.modelId)} cannot be passed to ${session.mode} on its command line`
);
}
const applied = applyCustomModelInjection(
entry,
endpoint,
@@ -1623,17 +1645,6 @@ export function registerSessionRoutes(
if (!applied) {
return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`);
}
if (
applied.launchModel !== undefined &&
modelSpec?.type === 'token' &&
!matchesPattern(modelSpec.pattern, applied.launchModel)
) {
removeConfigDir(applied.configDir);
return createErrorResponse(
ApiErrorCode.INVALID_INPUT,
`Model id ${JSON.stringify(body.modelId)} cannot be passed to ${session.mode} on its command line`
);
}
const { previousConfigDir } = session.setCustomModel(
{
@@ -4447,7 +4458,9 @@ export function registerSessionRoutes(
const cmHosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR);
const cmEndpoint = cmHosts.find((h) => h.id === customModel.endpointId);
if (!cmEndpoint) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
const cmContextLength = cmEndpoint.modelContextLengths?.[customModel.modelId];
// Per-model discovery figure first, user-set endpoint default second, nothing third
// (see the dedicated route's own comment above).
const cmContextLength = resolveCustomModelContextLength(cmEndpoint, customModel.modelId);
const cmSlotCount = cmEndpoint.modelSlotCounts?.[customModel.modelId];
// See the dedicated route's own comment for the full reasoning: some CLIs' own fixed
+14 -1
View File
@@ -2387,6 +2387,14 @@ export const WebviewUpdateSchema = WebviewBaseSchema.partial();
/** POST /api/webviews/probe: reachability + framing check for the editor's Test button. */
export const WebviewProbeSchema = z.object({ url: webviewUrlSchema });
/**
* Ceiling on a stated context length (tokens), shared between the bounds below and
* discovery (`custom-model-routes.ts`): anything above it reads as nonsense and is
* ignored, because a bogus figure here overflows the smaller server, which is the exact
* failure the discovery field exists to prevent.
*/
export const MAX_DISCOVERED_CONTEXT_TOKENS = 100_000_000;
// Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md) — a
// user-configured custom OpenAI-compatible endpoint, local (llama.cpp) or cloud
// (Azure AI Foundry, etc.). Lives below `webviewUrlSchema` because `baseUrl` IS that
@@ -2410,11 +2418,16 @@ export const CustomModelHostSchema = z.object({
defaultModelId: z.string().max(200).optional(),
// Server-populated by discovery (custom-model-routes.ts); accepted here only so a client
// round-tripping the GET response back through PUT (edit-save) doesn't drop it.
modelContextLengths: z.record(z.string().max(200), z.number().int().positive().max(100_000_000)).optional(),
modelContextLengths: z
.record(z.string().max(200), z.number().int().positive().max(MAX_DISCOVERED_CONTEXT_TOKENS))
.optional(),
// Same reasoning as modelContextLengths above.
modelSlotCounts: z.record(z.string().max(200), z.number().int().positive().max(4096)).optional(),
// Same reasoning as modelContextLengths above.
modelSizesGB: z.record(z.string().max(200), z.number().positive().max(100_000)).optional(),
// User-set fallback for models discovery has no context length for (session-routes.ts);
// discovery never writes it. Same bound as the modelContextLengths values above.
defaultContextLength: z.number().int().positive().max(MAX_DISCOVERED_CONTEXT_TOKENS).optional(),
});
/**
+2 -2
View File
@@ -80,7 +80,7 @@ import { attachmentRegistry, buildFileThumbnailRoute, registerExternalAttachment
import { folderGrants } from '../folder-listing.js';
import { getCli, enabledClis, listClis } from '../config/cli-registry/registry.js';
import { isCliEntryInstalled, probeStockCliAvailability } from '../utils/cli-installed-probes.js';
import { readCustomModelHosts } from '../custom-model-hosts.js';
import { readCustomModelHosts, resolveCustomModelContextLength } from '../custom-model-hosts.js';
import { applyCustomModelInjection, customModelConfigDir, removeConfigDir } from '../custom-model-injection-apply.js';
import type { CustomModelBookkeeping } from '../types/session.js';
import { registerGeneratedArtifactAttachment } from '../generated-artifact-attachments.js';
@@ -1470,7 +1470,7 @@ export class WebServer extends EventEmitter {
endpoint,
saved.modelId,
session.id,
endpoint.modelContextLengths?.[saved.modelId]
resolveCustomModelContextLength(endpoint, saved.modelId)
)?.envOverrides;
} catch (err) {
console.warn('[WebServer] Failed to rebuild custom-model env on recovery:', err);
+9
View File
@@ -171,6 +171,15 @@ describe('codex', () => {
expect(cx({ model: 'gpt-5', resumeSessionId: 'roll_42' })).toBe('codex --model gpt-5 resume roll_42');
});
it('carries an Ollama-style `qwen3:32b` model id, and still drops a leading-flag shape', () => {
// `model-plain-colon`: the custom-model feature forces the selected id onto this param,
// and ids like `qwen3:32b` carry a colon. The leading-alphanumeric rule is what keeps a
// flag from riding the flag, so `:x` and `-x` must still drop out.
expect(cx({ model: 'qwen3:32b' })).toBe('codex --model qwen3:32b');
expect(cx({ model: ':leading' })).toBe('codex');
expect(cx({ model: '-yolo' })).toBe('codex');
});
it('sends reasoning effort as one model_reasoning_effort config value, for every level', () => {
for (const level of CODEX_REASONING_EFFORTS) {
expect(cx({ reasoningEffort: level })).toBe(`codex --config model_reasoning_effort=${level}`);
@@ -481,6 +481,52 @@ describe('refreshAllCustomModelHosts: parallel-slot enrichment (llama.cpp splits
const [updated] = await readCustomModelHosts(dir);
expect(updated.modelSlotCounts).toEqual({ a: 4 });
});
it('a round where only the listed figure answers resets the slot count a loaded round learned', async () => {
// The false-warning scenario, measured on a real llama-swap: round 1 has the model
// loaded under `--ctx-size 262144 --parallel 4` — a TOTAL window shared across slots,
// correctly stored with its count. Round 2 finds it unloaded and takes the LISTED
// `context_length`, which llama-swap documents as the PER-SLOT window. If the stored 4
// survived, the floor check would divide the per-slot figure a second time
// (65536/4 = 16384) and warn about a model that serves 65536 to every request.
const dir = getDataDir();
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
fetchMock.mockImplementation(async (url: URL) => {
if (url.pathname === '/v1/models') {
return new Response(JSON.stringify({ data: [{ id: 'm', status: { value: 'loaded' } }] }), { status: 200 });
}
if (url.pathname === '/running') {
return new Response(
JSON.stringify({
running: [{ model: 'm', state: 'ready', cmd: 'llama-server -m /models/m.gguf --ctx-size 262144 --parallel 4' }],
}),
{ status: 200 }
);
}
throw new Error(`unexpected request: ${url.href}`); // the command answers both questions
});
await refreshAllCustomModelHosts();
let [round1] = await readCustomModelHosts(dir);
expect(round1.modelContextLengths).toEqual({ m: 262144 });
expect(round1.modelSlotCounts).toEqual({ m: 4 });
// Round 2: unloaded, no /running or /props probe in order, listing its per-slot figure.
fetchMock.mockResolvedValue(
new Response(JSON.stringify({ data: [{ id: 'm', status: { value: 'unloaded' }, context_length: 65536 }] }), {
status: 200,
})
);
await refreshAllCustomModelHosts();
const [round2] = await readCustomModelHosts(dir);
expect(round2.modelContextLengths).toEqual({ m: 65536 });
// 1 slot, not the stale 4: 65536 per slot with 1 recorded slot is the honest pair, so
// the floor check reads 65536/request — exactly what the server gives a single request.
expect(round2.modelSlotCounts).toEqual({ m: 1 });
});
});
describe('refreshAllCustomModelHosts: model-size enrichment (parsed from /v1/models description)', () => {
+53 -1
View File
@@ -8,6 +8,7 @@
*/
import { describe, it, expect } from 'vitest';
import { parse as parseToml } from 'smol-toml';
import { getCli } from '../src/config/cli-registry/index.js';
import {
buildCustomModelInjection,
@@ -112,6 +113,31 @@ describe('buildCustomModelInjection', () => {
expect(parsed.provider.custom.models.qwen3).toEqual({});
});
it('opencode: threads a discovered context length into the model limit as the full window (issue #749)', () => {
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', 16384);
if (result.kind !== 'env') throw new Error('unreachable');
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
// `limit.context` IS the full window to opencode — it reserves for the reply itself —
// so the discovered total passes through; `output` only caps the reply at a quarter.
expect(parsed.provider.custom.models.qwen3).toEqual({ limit: { context: 16384, output: 4096 } });
});
it('opencode: caps the response headroom reservation at 8192 for big windows', () => {
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', 262144);
if (result.kind !== 'env') throw new Error('unreachable');
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
expect(parsed.provider.custom.models.qwen3).toEqual({ limit: { context: 262144, output: 8192 } });
});
it('opencode: never invents a limit for an unknown or non-finite context length (issue #749 regression)', () => {
for (const contextLength of [undefined, NaN]) {
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', contextLength);
if (result.kind !== 'env') throw new Error('unreachable');
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
expect(parsed.provider.custom.models.qwen3).toEqual({});
}
});
it('codex: configDir writes an isolated config.toml with model/base_url, and the key rides as extraEnv (never a literal TOML field)', () => {
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3');
expect(result.kind).toBe('configDir');
@@ -133,6 +159,30 @@ describe('buildCustomModelInjection', () => {
expect(result.files[0].content).toContain('model = "weird\\"model"');
});
it('codex: writes the discovered window as TOP-LEVEL model_context_window with a 75% compact limit (issue #750)', () => {
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3', 16384);
if (result.kind !== 'configDir') throw new Error('unreachable');
const doc = parseToml(result.files[0].content) as Record<string, unknown>;
// Top-level is the shape codex reads: parked under [model_providers.custom] (its
// provider table) the same keys would be silently ignored by codex itself.
expect(doc.model_context_window).toBe(16384);
expect(doc.model_auto_compact_token_limit).toBe(12288); // 75% of the window, floored
const provider = (doc.model_providers as Record<string, Record<string, unknown>>).custom;
expect(provider.model_context_window).toBeUndefined();
expect(provider.model_auto_compact_token_limit).toBeUndefined();
});
it('codex: writes neither context line for an unknown, non-finite or zero context length', () => {
for (const contextLength of [undefined, NaN, 0]) {
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3', contextLength);
if (result.kind !== 'configDir') throw new Error('unreachable');
const doc = parseToml(result.files[0].content) as Record<string, unknown>;
expect(doc.model_context_window, String(contextLength)).toBeUndefined();
expect(doc.model_auto_compact_token_limit, String(contextLength)).toBeUndefined();
expect(doc.model).toBe('qwen3'); // the rest of the file is still written
}
});
it('pi: configDir writes .pi/agent/models.json, redirected via HOME (verified live — PI_CONFIG_DIR does nothing for pi)', () => {
const result = buildCustomModelInjection(entryOrThrow('pi'), endpoint, 'qwen3');
if (result.kind !== 'configDir') throw new Error('unreachable');
@@ -232,7 +282,9 @@ describe('launchModel (the model launch param that selects the injected provider
});
it('CLIs whose config selects the model on its own declare none', () => {
for (const id of ['claude', 'opencode', 'codex', 'gemini', 'deepseek']) {
// codex now declares launchModel (selected-model launch override) and is covered by
// its own restart coverage, not here.
for (const id of ['claude', 'opencode', 'gemini', 'deepseek']) {
const result = buildCustomModelInjection(entryOrThrow(id), endpoint, 'qwen3');
if (result.kind === 'unsupported') throw new Error('unreachable');
expect(result.launchModel, id).toBeUndefined();
+91
View File
@@ -0,0 +1,91 @@
/**
* @fileoverview Boot recovery rebuilds a saved custom-model selection through the SAME
* context-length resolution the routes use: the model's discovered figure, falling back to
* the endpoint's `defaultContextLength` (`resolveCustomModelContextLength`). Before the
* shared helper, `_rebuildCustomModelEnv` read `modelContextLengths` alone — so a session
* on a model that only the endpoint default covers came back from a Codeman restart with
* `CLAUDE_CODE_MAX_CONTEXT_TOKENS` (or a config file's context figure) silently gone,
* re-arming exactly the window overflow the var exists to prevent.
*
* `_rebuildCustomModelEnv` reads no `this` state (it re-derives everything from the endpoint
* store and the Session it is handed), so it is driven straight off the prototype with an
* empty receiver — `restoreMuxSessions()` itself cannot be reached under vitest, the same
* constraint recorded in test/session-model-recovery.test.ts.
*/
import { afterEach, describe, expect, it } from 'vitest';
import { getDataDir } from '../src/config/instance.js';
import { writeCustomModelHosts, type CustomModelHost } from '../src/custom-model-hosts.js';
import { customModelConfigDir, removeConfigDir } from '../src/custom-model-injection-apply.js';
import { Session } from '../src/session.js';
import { WebServer } from '../src/web/server.js';
import type { CustomModelBookkeeping } from '../src/types/session.js';
type RebuildFn = (session: Session, saved: CustomModelBookkeeping) => Promise<Record<string, string> | undefined>;
/** The real method, with an empty receiver — see the file header for why that is sound. */
function rebuildCustomModelEnv(session: Session, saved: CustomModelBookkeeping) {
const fn = (WebServer.prototype as unknown as { _rebuildCustomModelEnv: RebuildFn })._rebuildCustomModelEnv;
expect(typeof fn, '_rebuildCustomModelEnv vanished from WebServer.prototype').toBe('function');
return fn.call({}, session, saved);
}
function endpoint(overrides: Partial<CustomModelHost> = {}): CustomModelHost {
return {
id: 'ep',
label: 'Ep',
baseUrl: 'http://localhost:8080',
apiKey: 'recovery-probe-key',
models: ['qwen3'],
...overrides,
};
}
describe('_rebuildCustomModelEnv: context-length fallback on boot recovery', () => {
const configs: string[] = [];
afterEach(() => {
for (const id of configs.splice(0)) removeConfigDir(customModelConfigDir(id));
});
it('falls back to the endpoint defaultContextLength when discovery recorded no per-model figure', async () => {
await writeCustomModelHosts(getDataDir(), [endpoint({ defaultContextLength: 32768 })]);
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
configs.push(session.id);
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('32768');
});
it('a discovered per-model figure still wins over the endpoint default', async () => {
await writeCustomModelHosts(getDataDir(), [
endpoint({ modelContextLengths: { qwen3: 131072 }, defaultContextLength: 32768 }),
]);
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
configs.push(session.id);
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('131072');
});
it('states nothing rather than guessing when neither source knows a window', async () => {
await writeCustomModelHosts(getDataDir(), [endpoint()]);
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
configs.push(session.id);
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBeUndefined();
});
it('drops the selection to undefined (kept for clearing) when the endpoint no longer exists', async () => {
await writeCustomModelHosts(getDataDir(), []);
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
configs.push(session.id);
await expect(rebuildCustomModelEnv(session, { endpointId: 'gone', modelId: 'qwen3', envKeys: [] })).resolves.toBe(
undefined
);
});
});
+173
View File
@@ -20,6 +20,10 @@ import { describe, expect, it, vi } from 'vitest';
const CONSTANTS_JS = readFileSync(new URL('../src/web/public/constants.js', import.meta.url), 'utf-8');
const SESSION_UI_JS = readFileSync(new URL('../src/web/public/session-ui.js', import.meta.url), 'utf-8');
// Only for the [hidden] CSS guard below — the filter toggles the `hidden` property, and
// `.run-mode-option`'s own `display: flex` would silently defeat it without the
// `!important` re-assertion in styles.css.
const STYLES_CSS = readFileSync(new URL('../src/web/public/styles.css', import.meta.url), 'utf-8');
// Only for the real _showCenterStatus DOM tests below (`bootAppWithRealCenterStatus`) —
// every other test in this file stubs _showCenterStatus itself and has no need of it.
const PANELS_UI_JS = readFileSync(new URL('../src/web/public/panels-ui.js', import.meta.url), 'utf-8');
@@ -55,6 +59,9 @@ function bootApp(
<div class="modal" id="customModelPickModal">
<h3 id="customModelPickTitle"></h3>
<p id="customModelPickHint"></p>
<input type="search" id="customModelPickSearch" aria-label="Search models"
oninput="app.filterCustomModelPick(this.value)">
<p id="customModelPickNoMatches" role="status" hidden>No models match</p>
<div id="customModelPickList"></div>
</div>
<div class="modal" id="customModelSwapConfirmModal">
@@ -532,6 +539,172 @@ describe('Custom Model Endpoint Profiles: the "which model" picker', () => {
await app.selectCustomModelEntry('claude', 'llama-box');
expect(toastMessage).toMatch(/no models discovered/i);
});
it('resets and focuses the search box every time the picker opens', async () => {
const { win, app } = bootApp({
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
});
win.app = app;
await app.selectCustomModelEntry('claude', 'llama-box');
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
// The caret lands in the box, so the first keystroke filters with no click.
expect(win.document.activeElement).toBe(search);
// A stale query from a previous open must not survive into the next one.
search.value = 'zzz';
search.dispatchEvent(new win.Event('input'));
app.closeCustomModelPickModal();
await app.selectCustomModelEntry('claude', 'llama-box');
expect(search.value).toBe('');
expect(win.document.activeElement).toBe(search);
expect(win.document.getElementById('customModelPickNoMatches')!.hasAttribute('hidden')).toBe(true);
const visible = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')].filter(
(b) => !b.hasAttribute('hidden')
);
expect(visible).toHaveLength(2);
});
it('leaves the search box unfocused on touch devices so the OS keyboard stays off the list', async () => {
const { win, app } = bootApp({
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
});
win.app = app;
// The guard reads the global the browser defines (mobile-detection.js).
(win as unknown as { MobileDetection: { isTouchDevice: () => boolean } }).MobileDetection = {
isTouchDevice: () => true,
};
await app.selectCustomModelEntry('claude', 'llama-box');
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
expect(win.document.activeElement).not.toBe(search);
// Only the focus is skipped: the rows still render, unfiltered and functional.
const visible = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')].filter(
(b) => !b.hasAttribute('hidden')
);
expect(visible).toHaveLength(2);
});
it('filters rows case-insensitively on the model id, keeping the surviving row fully functional', async () => {
const { win, app } = bootApp({
hosts: [
{
id: 'llama-box',
label: 'llama.cpp',
baseUrl: 'http://x',
models: ['qwen3', 'llama3'],
defaultModelId: 'qwen3',
},
],
});
let launched: unknown[] | null = null;
app.runCustomModelEntry = async (...args: unknown[]) => {
launched = args;
};
win.app = app;
await app.selectCustomModelEntry('claude', 'llama-box');
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
search.value = 'QWEN';
search.dispatchEvent(new win.Event('input'));
const buttons = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')];
expect(buttons.filter((b) => b.hasAttribute('hidden')).map((b) => b.textContent!.trim())).toEqual(['llama3']);
const [visible] = buttons.filter((b) => !b.hasAttribute('hidden'));
// The surviving row keeps its Default badge and its real, callable click handler.
expect(visible.textContent).toContain('qwen3');
expect(visible.textContent).toContain('Default');
(visible as unknown as HTMLButtonElement & { onclick: (e: unknown) => void }).onclick(
new (win as any).Event('click')
);
expect(launched).toEqual(['claude', 'llama-box', 'qwen3']);
});
it('shows the no-matches line only while a query matches nothing, and clears it with the query', async () => {
const { win, app } = bootApp({
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
});
win.app = app;
await app.selectCustomModelEntry('claude', 'llama-box');
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
const noMatches = win.document.getElementById('customModelPickNoMatches')!;
const isHidden = () =>
win.document.getElementById('customModelPickList')!.querySelectorAll('button[hidden]').length;
// Whitespace is not a query: everything stays visible, no message.
search.value = ' ';
search.dispatchEvent(new win.Event('input'));
expect(noMatches.hasAttribute('hidden')).toBe(true);
expect(isHidden()).toBe(0);
search.value = 'ghost';
search.dispatchEvent(new win.Event('input'));
expect(noMatches.hasAttribute('hidden')).toBe(false);
expect(isHidden()).toBe(2);
search.value = '';
search.dispatchEvent(new win.Event('input'));
expect(noMatches.hasAttribute('hidden')).toBe(true);
expect(isHidden()).toBe(0);
});
it('keeps an HTML-significant model id inert in the picker rows too, and filterable by its raw text', async () => {
// Same vector as the Run-menu row above, now via data-model-id: escapeHtml covers
// the double quote, and the filter reads the decoded raw id back off dataset.
const dangerousModel = '"><img src=x onerror=alert(1)>';
const { win, app } = bootApp({
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', dangerousModel] }],
});
win.app = app;
await app.selectCustomModelEntry('claude', 'llama-box');
const list = win.document.getElementById('customModelPickList')!;
expect(list.querySelector('img')).toBeNull();
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
search.value = 'src=x';
search.dispatchEvent(new win.Event('input'));
const visible = list.querySelectorAll('button:not([hidden])');
expect(visible).toHaveLength(1);
expect(visible[0].getAttribute('data-model-id')).toBe(dangerousModel);
});
it('re-asserts display:none !important for hidden picker rows so the filter cannot be silently defeated', () => {
// `.run-mode-option` carries its own `display: flex` (styles.css, restated in
// mobile.css), which outranks the UA `[hidden]` rule — without this scoped
// !important re-assertion, toggling `hidden` would change nothing on screen.
const rule = STYLES_CSS.match(/#customModelPickModal \.run-mode-option\[hidden\]\s*\{[^}]*\}/);
expect(rule, 'picker rows need a [hidden] rule scoped to #customModelPickModal').toBeTruthy();
expect(rule![0]).toMatch(/display:\s*none\s*!important/);
});
it('pins "No models match" hidden with an empty query and shown only when a query hides every row, in CSS too', async () => {
// The visible symptom: the line carries `.form-hint`, whose `display: block`
// outranks the UA `[hidden]` rule, so `hidden` alone showed "No models match"
// while every model was listed with an empty search box. The attribute half
// was always right (session-ui.js's `filterCustomModelPick`); the CSS half
// needed the same `[hidden] { display: none !important }` re-assertion the
// rows above carry. This test pins both states AND the CSS that makes them
// actually render.
const rule = STYLES_CSS.match(/#customModelPickModal \.custom-model-pick-no-matches\[hidden\]\s*\{[^}]*\}/);
expect(rule, 'the no-matches line needs a [hidden] rule scoped to #customModelPickModal').toBeTruthy();
expect(rule![0]).toMatch(/display:\s*none\s*!important/);
const { win, app } = bootApp({
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
});
win.app = app;
await app.selectCustomModelEntry('claude', 'llama-box');
const noMatches = win.document.getElementById('customModelPickNoMatches')!;
// Empty query, models listed: the attribute must be set, and the CSS rule
// above is what makes that attribute actually hide the line.
expect(noMatches.hasAttribute('hidden')).toBe(true);
// A non-empty query that hides every row: shown again.
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
search.value = 'ghost';
search.dispatchEvent(new win.Event('input'));
expect(noMatches.hasAttribute('hidden')).toBe(false);
});
});
describe('Custom Model Endpoint Profiles: _getCustomModelCurrentlyLoaded is client-side bounded', () => {
+164 -1
View File
@@ -11,7 +11,7 @@
import { describe, it, expect, vi, afterEach } from 'vitest';
import { registerCustomModelRoutes } from '../../src/web/routes/custom-model-routes.js';
import { webviewFetch } from '../../src/web/webview-egress.js';
import { createRouteTestHarness } from './_route-test-utils.js';
import { createRouteTestHarness, type RouteTestHarness } from './_route-test-utils.js';
vi.mock('../../src/web/webview-egress.js', async () => {
const actual = await vi.importActual<typeof import('../../src/web/webview-egress.js')>(
@@ -417,3 +417,166 @@ describe('apiKey is never handed back to the browser', () => {
expect(fetchMock).toHaveBeenCalledTimes(1);
});
});
describe('context-length discovery from the /v1/models listing', () => {
// The gateway path (a Bifrost in front of llama-swap, an OpenRouter-style server):
// no /running, no /props, just a per-model `context_length` or `context_window` in the
// model listing. Distinct ports per test keep the file's shared data dir unambiguous.
afterEach(() => {
fetchMock.mockReset();
});
type StoredHost = { id: string; modelContextLengths?: Record<string, number>; defaultContextLength?: number };
type HarnessApp = RouteTestHarness['app'];
async function endpointWith(id: string, port: number, extra: Record<string, unknown> = {}): Promise<HarnessApp> {
const { app } = await setup();
const res = await app.inject({
method: 'POST',
url: '/api/model-endpoints',
payload: { id, label: 'A', baseUrl: `http://localhost:${port}`, ...extra },
});
expect(res.statusCode).toBe(200);
return app;
}
async function storedHost(app: HarnessApp, id: string): Promise<StoredHost | undefined> {
const list = await app.inject({ method: 'GET', url: '/api/model-endpoints' });
return (list.json() as StoredHost[]).find((h) => h.id === id);
}
it('reads context_length (preferred) then context_window from each entry into modelContextLengths', async () => {
const app = await endpointWith('ep-ctx-listed', 8101);
fetchMock.mockResolvedValue(
new Response(
JSON.stringify({
data: [
{ id: 'both', context_length: 131072, context_window: 32768 },
{ id: 'window-only', context_window: 8192 },
{ id: 'no-figure' },
],
}),
{ status: 200 }
)
);
const discover = await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-listed/discover-models' });
expect(discover.json().success).toBe(true);
const stored = await storedHost(app, 'ep-ctx-listed');
expect(stored?.modelContextLengths).toEqual({ both: 131072, 'window-only': 8192 });
});
it('ignores non-numeric, zero, negative and absurd listed values; the schema bound itself is still accepted', async () => {
const app = await endpointWith('ep-ctx-bad', 8102);
fetchMock.mockResolvedValue(
new Response(
JSON.stringify({
data: [
{ id: 'string', context_length: '131072' },
{ id: 'zero', context_window: 0 },
{ id: 'negative', context_length: -10 },
{ id: 'over-bound', context_length: 100_000_001 },
{ id: 'fractional', context_window: 16384.5 },
{ id: 'at-bound', context_length: 100_000_000 },
],
}),
{ status: 200 }
)
);
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-bad/discover-models' });
const stored = await storedHost(app, 'ep-ctx-bad');
expect(stored?.modelContextLengths).toEqual({ 'at-bound': 100_000_000 });
});
it('prefers the /running launch command over the listed figure for a loaded model', async () => {
const app = await endpointWith('ep-ctx-cmd', 8103);
fetchMock.mockImplementation(async (url: URL) => {
if (url.pathname === '/v1/models') {
return new Response(
JSON.stringify({ data: [{ id: 'cmd-wins', status: { value: 'loaded' }, context_length: 131072 }] }),
{ status: 200 }
);
}
if (url.pathname === '/running') {
return new Response(
JSON.stringify({ running: [{ model: 'cmd-wins', state: 'ready', cmd: 'llama-server --ctx-size 16384' }] }),
{ status: 200 }
);
}
if (url.pathname === '/props') return new Response('nope', { status: 404 });
throw new Error(`unexpected request in this test: ${url.href}`);
});
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-cmd/discover-models' });
const stored = await storedHost(app, 'ep-ctx-cmd');
expect(stored?.modelContextLengths).toEqual({ 'cmd-wins': 16384 });
});
it('keeps a previously learned value the fresh listing says nothing about, and discovery never touches defaultContextLength', async () => {
const app = await endpointWith('ep-ctx-keep', 8104, {
models: ['re-listed', 'untouched'],
modelContextLengths: { 're-listed': 16384, untouched: 4096 },
defaultContextLength: 65536,
});
fetchMock.mockResolvedValue(
new Response(
JSON.stringify({ data: [{ id: 're-listed', context_length: 32768 }, { id: 'untouched' }] }),
{ status: 200 }
)
);
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-keep/discover-models' });
const stored = await storedHost(app, 'ep-ctx-keep');
expect(stored?.modelContextLengths).toEqual({ 're-listed': 32768, untouched: 4096 });
expect(stored?.defaultContextLength).toBe(65536);
});
});
describe('defaultContextLength (the user-set fallback discovery never overwrites)', () => {
afterEach(() => {
fetchMock.mockReset();
});
it('stores it on create, updates it on PUT, and a PUT without it clears it', async () => {
const { app } = await setup();
const create = await app.inject({
method: 'POST',
url: '/api/model-endpoints',
payload: { id: 'ep-defctx', label: 'A', baseUrl: 'http://localhost:8105', defaultContextLength: 131072 },
});
expect(create.json().data.host.defaultContextLength).toBe(131072);
const update = await app.inject({
method: 'PUT',
url: '/api/model-endpoints/ep-defctx',
payload: { label: 'A', baseUrl: 'http://localhost:8105', defaultContextLength: 32768 },
});
expect(update.json().data.host.defaultContextLength).toBe(32768);
const cleared = await app.inject({
method: 'PUT',
url: '/api/model-endpoints/ep-defctx',
payload: { label: 'A', baseUrl: 'http://localhost:8105' },
});
expect(cleared.json().data.host.defaultContextLength).toBeUndefined();
});
it('rejects zero, negative, fractional, over-the-bound and non-numeric values', async () => {
const { app } = await setup();
for (const defaultContextLength of [0, -1, 1.5, 100_000_001, '131072']) {
const res = await app.inject({
method: 'POST',
url: '/api/model-endpoints',
payload: { id: 'ep-defctx-bad', label: 'A', baseUrl: 'http://localhost:8106', defaultContextLength },
});
expect(res.json().success, String(defaultContextLength)).toBe(false);
expect(res.json().errorCode, String(defaultContextLength)).toBe('INVALID_INPUT');
}
});
});
@@ -122,6 +122,9 @@ describe('POST /api/quick-start: customModel (one-shot custom-model launch)', ()
const { sessionId } = res.json();
const session = ctx.sessions.get(sessionId) as unknown as Session;
expect(session.customModel?.endpointId).toBe('ep1');
// #750, pinned the same way as the pi/grok/omp one-shot paths below: the FIRST
// launch already carries the selected id, not just the file on disk.
expect(session.getCustomModelForPersist()?.launchModel).toBe('qwen3');
expect(restartSpy).not.toHaveBeenCalled();
const configDir = customModelConfigDir(sessionId);
@@ -367,6 +370,50 @@ describe('POST /api/quick-start: customModel (one-shot custom-model launch)', ()
expect(res.statusCode).toBe(200);
expect(res.json().requiresContextWarning).toBeUndefined();
});
it('warns using the endpoint defaultContextLength when discovery recorded no per-model figure', async () => {
const endpoint: CustomModelHost = {
id: 'ep-default',
label: 'silent gateway',
baseUrl: 'http://192.168.1.53:8080',
apiKey: 'k',
defaultContextLength: 16384,
};
await writeCustomModelHosts(getDataDir(), [ENDPOINT, endpoint]);
const res = await quickStart({
caseName: 'cm-default-ctx',
mode: 'claude',
customModel: { endpointId: 'ep-default', modelId: 'some-unknown-model' },
});
const body = res.json();
expect(body.requiresContextWarning).toBe(true);
expect(body.contextLength).toBe(16384);
expect(ctx.sessions.size).toBe(1); // nothing was created
});
it('a discovered per-model figure wins over the endpoint defaultContextLength', async () => {
const endpoint: CustomModelHost = {
id: 'ep-precedence',
label: 'mixed box',
baseUrl: 'http://192.168.1.54:8080',
apiKey: 'k',
modelContextLengths: { m: 131072 },
defaultContextLength: 16384,
};
await writeCustomModelHosts(getDataDir(), [ENDPOINT, endpoint]);
const res = await quickStart({
caseName: 'cm-precedence',
mode: 'claude',
customModel: { endpointId: 'ep-precedence', modelId: 'm' },
});
expect(res.statusCode).toBe(200);
expect(res.json().requiresContextWarning).toBeUndefined();
expect(ctx.sessions.size).toBe(2);
});
});
describe('triggering the actual llama-swap load (not just watching for it)', () => {
+96 -3
View File
@@ -9,7 +9,7 @@ import { createRouteTestHarness } from './_route-test-utils.js';
import { createMockSession } from '../mocks/index.js';
import { getDataDir } from '../../src/config/instance.js';
import { writeCustomModelHosts, type CustomModelHost } from '../../src/custom-model-hosts.js';
import { existsSync, readFileSync, statSync } from 'node:fs';
import { existsSync, readFileSync, rmSync, statSync } from 'node:fs';
import { join } from 'node:path';
import { webviewFetch } from '../../src/web/webview-egress.js';
@@ -203,6 +203,10 @@ describe('POST /api/sessions/:id/custom-model', () => {
const { app, ctx } = await setup();
const session = ctx.sessions.get('test-session-1')!;
session.mode = 'pi';
// Start from no dir at all (a sibling test's successful pi apply may have left one),
// so the existsSync below says something about THIS request.
const dir = join(getDataDir(), 'custom-model-configs', 'test-session-1');
rmSync(dir, { recursive: true, force: true });
const res = await app.inject({
method: 'POST',
@@ -214,8 +218,45 @@ describe('POST /api/sessions/:id/custom-model', () => {
expect(res.json().errorCode).toBe('INVALID_INPUT');
expect(session.setCustomModel).not.toHaveBeenCalled();
expect(session.restartCli).not.toHaveBeenCalled();
// The config dir written before the check is cleaned up again.
expect(existsSync(join(getDataDir(), 'custom-model-configs', 'test-session-1'))).toBe(false);
// The pattern check runs on the pure build BEFORE any write, so a refused id leaves
// no directory behind at all, nothing to clean up.
expect(existsSync(dir)).toBe(false);
});
it('codex: a refused forced launch id never touches the live config dir the session runs on as CODEX_HOME', async () => {
const { app, ctx } = await setup();
const session = ctx.sessions.get('test-session-1')!;
session.mode = 'codex';
// First apply succeeds: the session now runs codex on qwen3:32b, whose config.toml
// lives in the per-session dir the running CLI itself uses as CODEX_HOME.
const ok = await app.inject({
method: 'POST',
url: '/api/sessions/test-session-1/custom-model',
payload: { endpointId: 'ep1', modelId: 'qwen3:32b' },
});
expect(ok.statusCode).toBe(200);
const dir = join(getDataDir(), 'custom-model-configs', 'test-session-1');
const written = join(dir, 'config.toml');
expect(existsSync(written)).toBe(true);
// `@` is outside codex's `model-plain-colon` pattern. The pattern check now runs on
// the pure build BEFORE any write, so the refusal must leave the dir — and the file
// the live CLI reads — exactly as they were, rather than replacing and deleting the
// CODEX_HOME of a session that stays on qwen3:32b.
const before = readFileSync(written, 'utf8');
const res = await app.inject({
method: 'POST',
url: '/api/sessions/test-session-1/custom-model',
payload: { endpointId: 'ep1', modelId: 'org/model@q4_k_m' },
});
expect(res.json().success).toBe(false);
expect(res.json().errorCode).toBe('INVALID_INPUT');
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
expect(session.restartCli).toHaveBeenCalledTimes(1);
expect(existsSync(dir)).toBe(true);
expect(readFileSync(written, 'utf8')).toBe(before);
});
it('clear removes the previous config dir the session reports', async () => {
@@ -449,6 +490,58 @@ describe('POST /api/sessions/:id/custom-model', () => {
expect(session.restartCli).not.toHaveBeenCalled();
});
it('the endpoint defaultContextLength stands in when discovery recorded no per-model figure', async () => {
const { app, ctx } = await setup();
// A gateway that states nothing per model: only the user's default is known.
const endpoint: CustomModelHost = {
id: 'ep-default',
label: 'silent gateway',
baseUrl: 'http://192.168.1.53:8080',
apiKey: 'k',
defaultContextLength: 16384,
};
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, endpoint]);
const session = ctx.sessions.get('test-session-1')!;
session.mode = 'claude';
const res = await app.inject({
method: 'POST',
url: '/api/sessions/test-session-1/custom-model',
payload: { endpointId: 'ep-default', modelId: 'some-unknown-model' },
});
const body = res.json();
expect(body.requiresContextWarning).toBe(true);
expect(body.contextLength).toBe(16384);
expect(session.setCustomModel).not.toHaveBeenCalled();
});
it('a discovered per-model figure wins over the endpoint defaultContextLength', async () => {
const { app, ctx } = await setup();
// The default is small enough to warn on; the discovered figure for THIS model is
// not, so the fallback must not leak into the per-model path.
const endpoint: CustomModelHost = {
id: 'ep-precedence',
label: 'mixed box',
baseUrl: 'http://192.168.1.54:8080',
apiKey: 'k',
modelContextLengths: { 'qwen3.8-27b-ud-q4_k_xl': 131072 },
defaultContextLength: 16384,
};
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, endpoint]);
const session = ctx.sessions.get('test-session-1')!;
session.mode = 'claude';
const res = await app.inject({
method: 'POST',
url: '/api/sessions/test-session-1/custom-model',
payload: { endpointId: 'ep-precedence', modelId: 'qwen3.8-27b-ud-q4_k_xl' },
});
expect(res.json().requiresContextWarning).toBeUndefined();
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
});
// llama-server splits its context across --parallel slots, and Claude Code sends several
// requests at once, so a context that clears the floor as a whole can still fail per slot.
describe('parallel slots share the context', () => {
+28
View File
@@ -27,6 +27,8 @@ import { join } from 'node:path';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { Session } from '../src/session.js';
import { getCli } from '../src/config/cli-registry/index.js';
import { buildCustomModelInjection } from '../src/custom-model-injection.js';
import { TmuxManager } from '../src/tmux-manager.js';
import type { MuxSession, SessionMode } from '../src/types.js';
@@ -225,3 +227,29 @@ describe('launchModel reaches the CLI through its own launch param', () => {
expect(respawn.mock.calls[0][0].model).toBeUndefined();
});
});
describe('Codex custom endpoint model overrides', () => {
it('selects the local model over a native launch default and restores the default on clear', async () => {
const entry = getCli('codex');
if (!entry) throw new Error('Missing Codex entry');
const injection = buildCustomModelInjection(
entry,
{ id: 'ep', label: 'Local', baseUrl: 'http://box:8080' },
'qwen3'
);
if (injection.kind !== 'configDir') throw new Error('Expected isolated Codex config');
const { session, respawn } = liveSession('codex', {
codexConfig: { model: 'native-model', reasoningEffort: 'high' },
});
session.setCustomModel(
{ endpointId: 'ep', modelId: 'qwen3', envKeys: ['CODEX_HOME'], launchModel: injection.launchModel },
{ CODEX_HOME: '/isolated/codex' }
);
await session.restartCli();
expect(respawn.mock.calls[0][0].codexConfig).toEqual({ model: 'qwen3', reasoningEffort: 'high' });
session.setCustomModel(undefined);
respawn.mockClear();
await session.restartCli();
expect(respawn.mock.calls[0][0].codexConfig).toEqual({ model: 'native-model', reasoningEffort: 'high' });
});
});