mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-15 02:49:33 +01:00
Merge pull request #752: fix(custom-model): context limits, Codex model override, context discovery and picker filter
This commit is contained in:
commit
3bcb55ce96
29 files changed
+1125
-102
No files matched your search
@@ -0,0 +1,5 @@
|
||||
---
|
||||
'aicodeman': patch
|
||||
---
|
||||
|
||||
Custom endpoints now hand their context window to the CLIs that ask for it. OpenCode and Codex receive the discovered context length on injection (#749), and Codex launches on the selected custom model instead of its default (#750). Endpoint discovery reads each model's context size from `/v1/models` (with a sanity ceiling), with a per-endpoint "Default context length" field for gateways that answer nothing, and boot recovery keeps the saved figure (#754). The "Choose a model" picker also grew a search box that filters the rows as you type.
|
||||
+30
-7
@@ -1205,24 +1205,47 @@ data}` envelope on the wire — unwrap it the same way). Answers `[]` for a
|
||||
boolean` reports whether one is stored, so a client can render "unchanged
|
||||
if left blank" without ever holding the real value.
|
||||
- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?,
|
||||
authStyle?, defaultModelId? }` creates one. `id` must match
|
||||
authStyle?, defaultModelId?, defaultContextLength? }` creates one. `id`
|
||||
must match
|
||||
`^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never
|
||||
both (a real server hung indefinitely when sent both headers on one
|
||||
request); `baseUrl` must be `http(s)`, carry no embedded credentials, and
|
||||
request); `defaultContextLength` is a user-set fallback used when
|
||||
assigning a model that discovery has no context length for (discovery
|
||||
itself never writes it); `baseUrl` must be `http(s)`, carry no embedded
|
||||
credentials, and
|
||||
is refused if it points at (or resolves to) a link-local or
|
||||
cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id.
|
||||
- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey`
|
||||
- `PUT /api/v1/model-endpoints/:id` updates one, also accepting
|
||||
`defaultContextLength`. An **absent** `apiKey`
|
||||
keeps the stored one rather than clearing it — the client never receives
|
||||
the real value to resend deliberately unchanged, so omission is the only
|
||||
way to say "leave it alone"; there is no way to clear a key back to unset
|
||||
this way. `defaultModelId`, when set, must be one of that endpoint's own
|
||||
this way. `defaultContextLength` behaves the opposite way: a PUT without
|
||||
it clears the stored value, so send the field back to keep it.
|
||||
`defaultModelId`, when set, must be one of that endpoint's own
|
||||
`models` (`400 INVALID_INPUT` otherwise).
|
||||
- `DELETE /api/v1/model-endpoints/:id` removes one.
|
||||
- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's
|
||||
own `GET /v1/models` and stores the result as `models`, updating
|
||||
`lastDiscoveredAt`, plus (best-effort, only for a model llama-swap's own
|
||||
response already reports loaded) `modelContextLengths` and `modelSizesGB`.
|
||||
A `defaultModelId` that no longer appears in the fresh list is dropped
|
||||
`lastDiscoveredAt`, plus (best-effort) `modelContextLengths`,
|
||||
`modelSlotCounts` and `modelSizesGB`: every listing entry is read for a
|
||||
stated context figure (`context_length`, then `context_window`) and a
|
||||
size, BEFORE any loaded check; only the launch-command/`/props`
|
||||
enrichment needs a model llama-swap's own response reports loaded. A
|
||||
loaded model's context figure comes from llama-swap's own launch command
|
||||
(`--fit-ctx` / `-c` / `--ctx-size`), falling back to `/props` (`n_ctx`,
|
||||
`total_slots`) for whatever the command did not answer, and OVERWRITES the
|
||||
listed figure for that model; the model's own listing
|
||||
(`context_length`, then `context_window`) only ever fills the gap when
|
||||
neither probe does, and is recorded as a single-slot figure —
|
||||
llama-swap lists the PER-SLOT window, so whenever the listed figure is
|
||||
the one taken the stored slot count is 1. A server with no `status`
|
||||
field on any entry (plain llama.cpp, a non-llama-swap gateway) gets no
|
||||
launch-command/`/props` enrichment at all, only whatever its listings
|
||||
state. `modelContextLengths` falls back to the
|
||||
endpoint's `defaultContextLength` when a session is assigned a model
|
||||
discovery never learned a window for. A `defaultModelId` that no longer
|
||||
appears in the fresh list is dropped
|
||||
rather than carried forward invalid. Failures answer `422 OPERATION_FAILED`
|
||||
with the underlying connection error, or a named egress refusal if the
|
||||
resolved address turned out to be blocked. The same refresh also runs
|
||||
|
||||
@@ -630,9 +630,9 @@ Adding the header changed the preamble, so `CODEMAN_PREAMBLE` was bumped (1.22.0
|
||||
|
||||
**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET <baseUrl>/v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×.
|
||||
|
||||
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
|
||||
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/copilot/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
|
||||
|
||||
⚠️ **Two launch paths, chosen by mechanism, not preference**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id <id>` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume <id> || --session-id <id>` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. The dead-pane respawn shares this pin, see [Dead-pane respawn: the resume pin](#dead-pane-respawn-the-resume-pin). ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/<id>` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600.
|
||||
⚠️ **Two launch paths, chosen by mechanism, not preference**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id <id>` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume <id> || --session-id <id>` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. The dead-pane respawn shares this pin, see [Dead-pane respawn: the resume pin](#dead-pane-respawn-the-resume-pin). ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/<id>` for pi/omp, grok's `[model.codeman-custom]` block name), and codex gets one too (`--model <bare modelId>`, its pattern widened to `model-plain-colon` for qwen3-style ids): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600.
|
||||
|
||||
⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`.
|
||||
|
||||
|
||||
@@ -67,15 +67,23 @@ configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one.
|
||||
Endpoint management is admin-only in multi-user mode, same as remote/docker
|
||||
hosts — these are machine-level infra, not per-user settings.
|
||||
|
||||
**Context length is discovered too, opportunistically and safely.** The plain
|
||||
`GET /v1/models` response has no context-window field. Discovery only ever
|
||||
looks for one for a model llama-swap's own response already reports
|
||||
**Context length is discovered too, opportunistically and safely.** Older
|
||||
servers state no context-window field in the plain `GET /v1/models`
|
||||
response at all; current llama-swap states one per model (both
|
||||
`context_length` and `context_window`) and gateways like Bifrost pass
|
||||
`context_window` through — that listed figure is the *lowest*-precedence
|
||||
source, used only when neither probe below answers, and because llama-swap
|
||||
documents it as the PER-SLOT window, a model whose figure comes from the
|
||||
listing is recorded with a slot count of 1 so the slot check never divides a
|
||||
per-slot figure a second time. The probes are only ever looked for on a model
|
||||
llama-swap's own response already reports
|
||||
`status.value === "loaded"` for — never for an unloaded one, because
|
||||
llama-swap treats `?model=` as a routing hint and asking about a model that
|
||||
isn't loaded risks triggering an actual (slow, GPU-swapping) load as a side
|
||||
effect of what should be read-only discovery. A server with no `status` field
|
||||
on any entry at all (not llama-swap) gets no context-length enrichment,
|
||||
rather than guessing. A model's previously-learned context length survives a
|
||||
on any entry at all (not llama-swap) gets no launch-command/`/props`
|
||||
enrichment, only whatever its listings state — rather than guessing. A
|
||||
model's previously-learned context length survives a
|
||||
later cycle where it wasn't the loaded one; it's dropped only once the model
|
||||
disappears from the endpoint's list entirely. Stored per model in
|
||||
`modelContextLengths` and applied automatically (see "Applying a model to a
|
||||
@@ -286,6 +294,15 @@ hot-swap. A Claude session is relaunched with `--resume <conversation> ||
|
||||
grok are relaunched with the `--model` value that selects the injected
|
||||
provider (`custom/<modelId>` for pi and omp, `codeman-custom` for grok),
|
||||
since for those three the config file alone does not switch the model.
|
||||
**Codex is relaunched with `--model <modelId>` too**, the bare selected id
|
||||
(not a provider selector): the launch param forces the session onto the
|
||||
model its generated `config.toml` already selected, so a launch-time model
|
||||
carried over from App Settings can never override the injected selection.
|
||||
That launch arg must satisfy the codex entry's `model-plain-colon` token
|
||||
pattern (a leading letter or digit, then letters, digits and `._-:/`, per
|
||||
`stock.ts`), so an id outside that shape, e.g. LM Studio's
|
||||
`org/model@q4_k_m`, is refused with a 400 at apply time now, where a
|
||||
`config.toml` alone used to carry it.
|
||||
**Remote (SSH) and Docker sessions are refused** (400) for now: their restart
|
||||
reattaches the durable remote/in-container tmux rather than relaunching the
|
||||
agent, so the selection would report success and change nothing.
|
||||
@@ -293,14 +310,16 @@ agent, so the selection would report success and change nothing.
|
||||
**Claude gets two more env vars when known/applicable, both declared on its
|
||||
registry entry (`contextLengthVar`/`configDirVar`), not hardcoded here:**
|
||||
|
||||
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s discovered context
|
||||
length (see the discovery section above) whenever one is known. Without
|
||||
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s context length —
|
||||
the discovered figure (see the discovery section above), falling back to
|
||||
the endpoint's user-set `defaultContextLength` — whenever one is known.
|
||||
Without
|
||||
it, Claude Code assumes a large (200k) window for any unrecognized custom
|
||||
model id and never compacts, which reliably overflows a much smaller real
|
||||
local context — confirmed live: a stock ~33.7K-token system prompt against
|
||||
a 16384-token llama-swap model failed with `exceeds the available context
|
||||
size`. No entry for the model in `modelContextLengths` means the var is
|
||||
simply omitted, never a guess. ⚠️ **This var only affects when Claude
|
||||
size`. Neither a `modelContextLengths` entry nor a `defaultContextLength`
|
||||
for the model means the var is simply omitted, never a guess. ⚠️ **This var only affects when Claude
|
||||
Code compacts conversation _history_ — it cannot fix a model whose real
|
||||
context is smaller than Claude Code's own fixed per-turn overhead**
|
||||
(system prompt + tool schemas, empirically ~36.4K tokens, confirmed live
|
||||
@@ -441,15 +460,17 @@ Both apply routes (the restart route and the one-shot `POST
|
||||
/api/quick-start`) now check for this **before** launching or restarting
|
||||
anything, gated on the CLI's registry entry declaring a `contextLengthVar`
|
||||
(currently only claude — the check is a no-op for every other CLI by
|
||||
construction, never a hardcoded mode check). If the model's discovered
|
||||
context (`modelContextLengths`, from discovery above) is below
|
||||
`CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
|
||||
construction, never a hardcoded mode check). If the model's context (the
|
||||
discovered figure from `modelContextLengths`, or the endpoint's
|
||||
`defaultContextLength` when discovery never learned one for that model) is
|
||||
below `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured
|
||||
~36.4K overhead), the response is `{requiresContextWarning: true, modelId,
|
||||
contextLength, minSafeContextTokens}` (plus `slotCount` when the model runs more than one parallel slot) instead of applying — nothing is
|
||||
restarted or created yet. A context length that was never discovered at
|
||||
all skips the check entirely (nothing to compare, so it fails open rather
|
||||
than warning on every model an endpoint hasn't reported a size for).
|
||||
The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
|
||||
restarted or created yet. A model with neither a discovered figure nor a
|
||||
`defaultContextLength` skips the check entirely (nothing to compare, so it
|
||||
fails open rather than warning on every model an endpoint hasn't reported a
|
||||
size for).
|
||||
The same warning fires when the context divided across the model's parallel slots (`contextLength / slotCount`) is below the floor, because Claude Code sends several 40K-token requests at once and the slots share one pool. ⚠️ `defaultContextLength` is the model's whole window, never a per-slot figure, so a default small enough to split under the slot count trips that slot division exactly like a discovered figure does. Retrying with `confirmedContext: true` launches anyway (the legacy `confirmed: true` still means both questions).
|
||||
|
||||
The Run-menu picker shows this as an in-app modal
|
||||
(`#customModelContextWarningModal`, matching the llama-swap conflict
|
||||
@@ -526,7 +547,12 @@ for '<id>' not found. Defaulting to fallback metadata...` on launch —
|
||||
and fabricating a fake catalog entry would mean copying the _shape_ of
|
||||
OpenAI's own proprietary schema (their per-model system-prompt content
|
||||
included) for a warning that doesn't otherwise affect behavior — not
|
||||
something to build into discovery.
|
||||
something to build into discovery. ⚠️ The fallback-metadata warning is
|
||||
NOT about the context window going un-overridden. Codex does now get a
|
||||
real context-window override: when a figure is known the generated
|
||||
`config.toml` sets the top-level `model_context_window` to it and
|
||||
`model_auto_compact_token_limit` to 75% of it (floored), so Codex
|
||||
compacts at that point instead of trusting its fallback guess.
|
||||
- **Gemini** — fails with `Invalid auth method selected`, traced to an
|
||||
undocumented `GATEWAY` auth path gemini-cli selects once
|
||||
`GOOGLE_GEMINI_BASE_URL` is set. Unresolved after real investigation
|
||||
|
||||
@@ -27,14 +27,20 @@ hosts — these are machine-level infra, not a per-user setting.
|
||||
shows up without another manual click of **Discover**. One endpoint being unreachable on a
|
||||
given cycle (powered off, wrong network) never blocks the others from refreshing.
|
||||
|
||||
**Context length is picked up automatically where it can be, safely.** Against a
|
||||
llama.cpp/llama-swap server, discovery also learns each _currently loaded_ model's real
|
||||
context window and applies it to the launched session (Claude Code today — see below), so
|
||||
the harness stops assuming a large default window for a model name it doesn't recognise and
|
||||
overflowing a much smaller real one. It's deliberately never probed for a model that isn't
|
||||
already loaded, since asking a llama-swap server about an unloaded model can trigger an
|
||||
actual, slow model swap as a side effect — a model just not currently loaded keeps whatever
|
||||
context length an earlier cycle already learned for it instead.
|
||||
**Context length is picked up automatically where it can be, safely.** Every model's own
|
||||
listing entry is read for a stated context figure (`context_length`, then `context_window`,
|
||||
which current llama.cpp, llama-swap and gateway-style servers state to varying degrees);
|
||||
against a llama-swap server the stronger figures from its launch command and from llama.cpp's
|
||||
`/props` probe win where they exist, and those probes only ever run for a model the server
|
||||
already reports loaded, since asking about an unloaded model can trigger an actual, slow model
|
||||
swap as a side effect. A model that states no figure keeps whatever an earlier cycle learned:
|
||||
the stored figure always beats the endpoint's **Default context length** field
|
||||
(`modelContextLengths[id] ?? defaultContextLength`), which only applies where no figure was
|
||||
ever learned (that number is the model's whole window, never a per-slot figure). A model
|
||||
neither covers simply gets no override, never a guess. The window is applied to the launched session by the harnesses
|
||||
that can take it: Claude Code today (see below), OpenCode as the model's context limit, and
|
||||
Codex as a `model_context_window` entry in its generated config file, which also sets Codex's
|
||||
own auto-compact point via `model_auto_compact_token_limit` at 75% of the window.
|
||||
|
||||
## Running a session against one
|
||||
|
||||
@@ -48,7 +54,9 @@ pointed at.
|
||||
|
||||
**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
|
||||
the session launches straight away on that model — nothing to choose. With two or more, a
|
||||
small dialog asks which one to use for this launch before starting the session; the
|
||||
small dialog asks which one to use for this launch before starting the session; a search box
|
||||
at the top narrows a long list with a case-insensitive substring match, so a server hosting
|
||||
hundreds of models stays usable. The
|
||||
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
|
||||
use a different one without changing the saved default. The list is not raw discovery order
|
||||
either: the model llama-swap reports loaded and ready is moved to the top and tagged
|
||||
|
||||
@@ -21,6 +21,7 @@ export type TokenPattern =
|
||||
| 'model'
|
||||
| 'model-claude'
|
||||
| 'model-plain'
|
||||
| 'model-plain-colon'
|
||||
| 'model-pi'
|
||||
| 'id'
|
||||
| 'id-dotted'
|
||||
@@ -47,6 +48,10 @@ const PATTERNS: Record<TokenPattern, RegExp> = {
|
||||
// copilot --model. Same characters as `model`, but a leading alphanumeric: the value follows a
|
||||
// flag, so `--yolo` must not pass as a model id and become its own flag (the allowAll clamp bypass).
|
||||
'model-plain': /^[a-zA-Z0-9][a-zA-Z0-9._\-/]*$/,
|
||||
// codex --model, and the value a custom-model `launchModel` is forced onto it. Same
|
||||
// shape as `model-plain` (leading alphanumeric, so a flag cannot ride the flag) plus
|
||||
// `:` — Ollama-style ids like `qwen3:32b` must survive onto the command line.
|
||||
'model-plain-colon': /^[a-zA-Z0-9][a-zA-Z0-9._\-/:]*$/,
|
||||
// buildPiCommand — `:` for a thinking suffix (`sonnet:high`), `/` for `provider/id`
|
||||
'model-pi': /^[a-zA-Z0-9._\-/:]+$/,
|
||||
// opencode --session, codex resume
|
||||
|
||||
@@ -628,7 +628,10 @@ const CODEX: CliEntry = {
|
||||
params: {
|
||||
bypassApprovals: { type: 'bool' },
|
||||
animations: { type: 'bool' },
|
||||
model: { type: 'token', pattern: 'model' },
|
||||
// `model-plain-colon`, not `model`: the custom-model feature forces the selected id
|
||||
// onto this param (`launchModel: '{modelId}'` below), and Ollama-style ids carry a
|
||||
// `:` (`qwen3:32b`). A leading alphanumeric still keeps `--yolo` from riding a flag.
|
||||
model: { type: 'token', pattern: 'model-plain-colon' },
|
||||
reasoningEffort: { type: 'enum', values: [...CODEX_REASONING_EFFORTS] },
|
||||
resumeId: { type: 'token', pattern: 'id' },
|
||||
},
|
||||
@@ -841,6 +844,7 @@ const CODEX: CliEntry = {
|
||||
dirEnvVar: 'CODEX_HOME',
|
||||
fileName: 'config.toml',
|
||||
template: 'codex-toml',
|
||||
launchModel: '{modelId}',
|
||||
},
|
||||
// CODEX_HOME already matches the CODEX_ allowedPrefix above, so it was ALREADY
|
||||
// reachable via plain envOverrides before this feature existed. It is arguably
|
||||
@@ -1921,8 +1925,8 @@ const COPILOT: CliEntry = {
|
||||
// custom provider and drops the GitHub sign-in requirement; COPILOT_PROVIDER_TYPE defaults to
|
||||
// "openai", which is what a Codeman custom endpoint is (llama.cpp, vLLM, Ollama).
|
||||
// COPILOT_MODEL names the model and COPILOT_PROVIDER_MODEL_ID / _WIRE_MODEL default to it.
|
||||
// The OpenAI-style route is `<base>/v1/chat/completions`, so the base gets the `/v1` suffix
|
||||
// the same way deepseek's does. All four already match the COPILOT_ prefix allowlist, so
|
||||
// The OpenAI-style route is `<base>/v1/chat/completions`, so the base gets the `/v1`
|
||||
// suffix via appendV1Suffix below. All four already match the COPILOT_ prefix allowlist, so
|
||||
// they are privileged env keys below: a non-granted multi-user owner must not be able to
|
||||
// redirect the session's endpoint or credentials through plain envOverrides.
|
||||
customModelInjection: {
|
||||
|
||||
@@ -812,9 +812,11 @@ export interface CliCapabilities {
|
||||
* for pi and omp (`--model custom/<id>`, or the CLI stays on its own default
|
||||
* provider and reports "No API key found for the selected model") or for
|
||||
* grok (`--model codeman-custom`, the `[model.<name>]` block the config
|
||||
* declares). Absent = the config alone selects the model (claude's env vars,
|
||||
* opencode's blob, codex's top-level `model` key). Applied by the session's
|
||||
* respawn options through the entry's `legacyConfigField`, never by id.
|
||||
* declares), and codex carries its bare selected id (`{modelId}`, an id the
|
||||
* entry's `model` token pattern must be able to carry). Absent = the config
|
||||
* alone selects the model (claude's env vars, opencode's blob). Applied by
|
||||
* the session's respawn options through the entry's `legacyConfigField`,
|
||||
* never by id.
|
||||
*
|
||||
* `contextLengthVar` (env kind only): the env var a discovered per-model context-window
|
||||
* size is written to when known (claude's `CLAUDE_CODE_MAX_CONTEXT_TOKENS`) — without it,
|
||||
@@ -856,19 +858,13 @@ export interface CliCapabilities {
|
||||
* reuses that file.
|
||||
*
|
||||
* `appendV1Suffix` (env kind only): the raw `endpoint.baseUrl` gets `withV1Suffix()`
|
||||
* applied before being written to `baseUrlVar`, instead of being used verbatim.
|
||||
* DeepSeek needs this and claude/gemini must NOT get it — a per-CLI asymmetry confirmed
|
||||
* by reading each SDK's own request-building source, not assumed: DeepSeek Harness's
|
||||
* bundled `@deepseek-ai/dsh-llm-deepseek` concatenates `${connection.baseURL}/chat/
|
||||
* completions` with no `/v1` insertion of its own (its real public API base,
|
||||
* `https://api.deepseek.com`, expects the caller's base URL to already carry any
|
||||
* needed prefix), while llama-swap/llama.cpp only ever serves the OpenAI-conventional
|
||||
* `/v1/chat/completions` — confirmed live: a bare `POST <baseUrl>/chat/completions`
|
||||
* 404s, `POST <baseUrl>/v1/chat/completions` succeeds, and the harness's own error
|
||||
* message template (`DeepSeek API error (HTTP ${status})`) reproduces the exact
|
||||
* `HTTP_404` this feature originally shipped with unexplained. Claude Code's own SDK,
|
||||
* by contrast, was already confirmed working end-to-end against the RAW `baseUrl` with
|
||||
* no suffix — appending one there would be wrong, not just redundant.
|
||||
* applied before being written to `baseUrlVar`, instead of being used verbatim. Only
|
||||
* deepseek and copilot declare it — both speak to `<base>/v1/chat/completions` and a
|
||||
* gateway saves the bare root. claude/gemini must NOT get it: Claude Code's own SDK was
|
||||
* confirmed working end-to-end against the RAW `baseUrl`, so appending a suffix there
|
||||
* would be wrong, not just redundant. The per-session config-file recipes do not need
|
||||
* this flag at all — `renderConfigFile()` applies the suffix inside every template, so
|
||||
* no configDir entry carries a raw base URL.
|
||||
*/
|
||||
customModelInjection:
|
||||
| {
|
||||
|
||||
@@ -52,18 +52,22 @@ export interface CustomModelHost {
|
||||
defaultModelId?: string;
|
||||
/**
|
||||
* Discovered context-window size (tokens) per model id, keyed by the same strings as
|
||||
* `models`. Populated opportunistically during discovery (`custom-model-routes.ts`) from
|
||||
* llama.cpp/llama-swap's `GET /props?model=<id>` — the plain OpenAI-shaped `/v1/models`
|
||||
* response has no such field. Only ever probed for a model the server already reports as
|
||||
* loaded (llama-swap's `status.value === 'loaded'`); an unloaded one is deliberately never
|
||||
* probed, since llama-swap treats `/props?model=` as a routing hint that can trigger an
|
||||
* actual (slow, GPU-swapping) model load as a side effect of merely asking. A model this
|
||||
* has no entry for simply gets no context-length env override applied — never a guess.
|
||||
* `models`. Populated only by discovery (`custom-model-routes.ts`), from the strongest
|
||||
* source the server offers: the llama-swap launch command reported by `GET /running`
|
||||
* (the source of truth for a loaded model), then llama.cpp's `GET /props?model=<id>`
|
||||
* `n_ctx` (only ever probed for a model the server already reports as loaded, since
|
||||
* llama-swap treats `/props?model=` as a routing hint that can trigger an actual
|
||||
* (slow, GPU-swapping) model load as a side effect of merely asking), then the model's
|
||||
* own `GET /v1/models` entry (`context_length`, else `context_window`, which current
|
||||
* llama-swap, Bifrost and OpenRouter-style servers each state to varying degrees). A
|
||||
* model with no entry here falls back to the endpoint's `defaultContextLength`; with
|
||||
* neither, it simply gets no context-length env override applied, never a guess.
|
||||
*/
|
||||
modelContextLengths?: Record<string, number>;
|
||||
/**
|
||||
* Discovered number of parallel server slots per model id, keyed like `modelContextLengths`
|
||||
* and populated on the same loaded-models-only basis. Read from the launch command's
|
||||
* but populated only for models the server reports loaded (a listing never states a slot
|
||||
* count, unlike `contextLengths`, which every entry can supply). Read from the launch command's
|
||||
* `--parallel`/`-np` when it states one, otherwise from llama.cpp's own `/props`
|
||||
* `total_slots` (a llama-server started without `--parallel` picks the count itself, so
|
||||
* the command cannot say). llama-server splits its context across these slots (by default
|
||||
@@ -88,6 +92,29 @@ export interface CustomModelHost {
|
||||
* server-side check relies on.
|
||||
*/
|
||||
modelSizesGB?: Record<string, number>;
|
||||
/**
|
||||
* User-set fallback context length (tokens), applied via
|
||||
* `resolveCustomModelContextLength()` at every read site (the swap route, the
|
||||
* quick-start route, boot recovery) only when `modelContextLengths` has no entry for
|
||||
* the selected model (a gateway that states no per-model figure, a model never loaded
|
||||
* yet). It is the model's WHOLE window, never a per-slot figure: the Claude floor
|
||||
* check divides it by any stored slot count exactly like a discovered figure. Never
|
||||
* written by discovery, so a value set in the endpoint editor survives every
|
||||
* re-discovery sweep; unset keeps the long-standing behaviour of letting each CLI
|
||||
* assume its own default.
|
||||
*/
|
||||
defaultContextLength?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The context window to apply for `modelId` on `host`: the discovered per-model figure when
|
||||
* discovery has one, else the endpoint's user-set `defaultContextLength`, else undefined
|
||||
* (no override applied, never a guess). ONE helper for every read site — the swap route,
|
||||
* the quickstart route and boot recovery each re-write the CLI's config files, and a site
|
||||
* that forgot the fallback silently loses the endpoint default on the next respawn.
|
||||
*/
|
||||
export function resolveCustomModelContextLength(host: CustomModelHost, modelId: string): number | undefined {
|
||||
return host.modelContextLengths?.[modelId] ?? host.defaultContextLength;
|
||||
}
|
||||
|
||||
export function customModelHostsPath(configDir: string): string {
|
||||
|
||||
@@ -136,12 +136,12 @@ export function buildCustomModelInjection(
|
||||
}
|
||||
|
||||
case 'configContentEnv': {
|
||||
const content = renderConfigContent(cap.template, endpoint, modelId, apiKey);
|
||||
const content = renderConfigContent(cap.template, endpoint, modelId, apiKey, contextLength);
|
||||
return withLaunchModel({ kind: 'env', envOverrides: { [cap.envVar]: content } }, cap.launchModel, modelId);
|
||||
}
|
||||
|
||||
case 'configDir': {
|
||||
const { content, extraEnv } = renderConfigFile(cap.template, endpoint, modelId, apiKey);
|
||||
const { content, extraEnv } = renderConfigFile(cap.template, endpoint, modelId, apiKey, contextLength);
|
||||
return withLaunchModel(
|
||||
{ kind: 'configDir', dirEnvVar: cap.dirEnvVar, files: [{ relPath: cap.fileName, content }], extraEnv },
|
||||
cap.launchModel,
|
||||
@@ -164,24 +164,48 @@ function withLaunchModel<T extends EnvInjection | ConfigDirInjection>(
|
||||
return { ...result, launchModel: template.split('{modelId}').join(modelId) };
|
||||
}
|
||||
|
||||
/**
|
||||
* One quarter of the window is held back for the reply, capped at 8192 tokens (a small local
|
||||
* model's window must not lose a disproportionate slice to the reservation). Returns undefined
|
||||
* for an unknown or non-finite size — opencode then keeps its own default guess, which is the
|
||||
* pre-fix behaviour; never invent a limit here.
|
||||
*/
|
||||
function opencodeModelLimit(contextLength: number | undefined): { context: number; output: number } | undefined {
|
||||
if (contextLength === undefined || !Number.isFinite(contextLength)) return undefined;
|
||||
const total = Math.trunc(contextLength);
|
||||
if (total <= 0) return undefined;
|
||||
const output = Math.max(1, Math.min(8192, Math.floor(total / 4)));
|
||||
// `limit.context` is the FULL window as far as opencode is concerned — it reserves for
|
||||
// the reply itself — so the discovered total is passed through, never a shrunk figure.
|
||||
return { context: total, output };
|
||||
}
|
||||
|
||||
function renderConfigContent(
|
||||
template: 'opencode-json',
|
||||
endpoint: CustomModelEndpoint,
|
||||
modelId: string,
|
||||
apiKey: string
|
||||
apiKey: string,
|
||||
contextLength?: number
|
||||
): string {
|
||||
switch (template) {
|
||||
case 'opencode-json':
|
||||
case 'opencode-json': {
|
||||
// `limit` carries the discovered context window (issue #749): without it opencode assumes
|
||||
// a large default window for an unrecognized model id and eventually overflows a much
|
||||
// smaller local server. `context` is the FULL window and opencode reserves for the reply
|
||||
// inside it itself, so `opencodeModelLimit` passes the real total through and only
|
||||
// `output` gets the capped reservation.
|
||||
const limit = opencodeModelLimit(contextLength);
|
||||
return JSON.stringify({
|
||||
$schema: 'https://opencode.ai/config.json',
|
||||
provider: {
|
||||
custom: {
|
||||
options: { baseURL: withV1Suffix(endpoint.baseUrl), apiKey },
|
||||
models: { [modelId]: {} },
|
||||
models: { [modelId]: limit ? { limit } : {} },
|
||||
},
|
||||
},
|
||||
model: `custom/${modelId}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -195,7 +219,8 @@ function renderConfigFile(
|
||||
template: 'codex-toml' | 'pi-models-json' | 'omp-models-yml' | 'grok-toml',
|
||||
endpoint: CustomModelEndpoint,
|
||||
modelId: string,
|
||||
apiKey: string
|
||||
apiKey: string,
|
||||
contextLength?: number
|
||||
): { content: string; extraEnv?: Record<string, string> } {
|
||||
const baseUrl = withV1Suffix(endpoint.baseUrl);
|
||||
switch (template) {
|
||||
@@ -217,6 +242,12 @@ function renderConfigFile(
|
||||
const content = [
|
||||
`model = ${quoted(modelId)}`,
|
||||
`model_provider = "custom"`,
|
||||
...(contextLength !== undefined && Number.isFinite(contextLength) && contextLength > 0
|
||||
? [
|
||||
`model_context_window = ${Math.trunc(contextLength)}`,
|
||||
`model_auto_compact_token_limit = ${Math.max(1, Math.floor(contextLength * 0.75))}`,
|
||||
]
|
||||
: []),
|
||||
'',
|
||||
'[model_providers.custom]',
|
||||
`name = "Custom Endpoint"`,
|
||||
|
||||
+2
-1
@@ -2724,7 +2724,8 @@ export class Session extends EventEmitter {
|
||||
|
||||
/**
|
||||
* Force the custom-model selection's `launchModel` (pi/omp `custom/<id>`, grok's
|
||||
* `[model.<name>]` block name) onto the CLI's `model` launch param. Where that param
|
||||
* `[model.<name>]` block name, codex's bare selected id) onto the CLI's `model`
|
||||
* launch param. Where that param
|
||||
* lives is registry DATA — the entry's `legacyConfigField` (`piConfig`, `grokConfig`,
|
||||
* ...) or the top-level `model` for an entry that declares none — so this stays a
|
||||
* generic reader rather than a branch per CLI. Applied on the OPTIONS only: the stored
|
||||
|
||||
@@ -615,6 +615,9 @@
|
||||
'API key': 'API 密钥',
|
||||
'Optional. Left blank on edit keeps the existing key.': '可选。编辑时留空将保留现有密钥。',
|
||||
'Auth header': '认证请求头',
|
||||
'Default context length (tokens)': '默认上下文长度(token 数)',
|
||||
"Used for models whose size cannot be discovered. The model's whole window; the Claude floor check divides it by any stored slot count. Leave empty to let each CLI assume its own default.":
|
||||
'用于无法自动发现上下文大小的模型。这是模型的完整窗口;Claude 下限检查会将其除以已存储的槽位数。留空则让各个 CLI 使用其自身的默认值。',
|
||||
'Never send both — some servers hang indefinitely.': '切勿同时发送两者 — 部分服务器会因此无限期挂起。',
|
||||
'Authorization: Bearer (default)': 'Authorization: Bearer(默认)',
|
||||
'api-key header (Azure)': 'api-key 请求头(Azure)',
|
||||
@@ -627,6 +630,9 @@
|
||||
'Last used': '上次使用',
|
||||
'That endpoint no longer exists': '该端点已不存在',
|
||||
'No models discovered for this endpoint yet': '此端点尚未发现任何模型',
|
||||
// Model picker search box (index.html placeholder/aria-label and the no-matches line)
|
||||
'Search models': '搜索模型',
|
||||
'No models match': '没有匹配的模型',
|
||||
'Subagent Options': '子智能体选项',
|
||||
'Enable Tracking': '启用跟踪',
|
||||
'Active Tab Only': '仅活动标签页',
|
||||
|
||||
@@ -1033,6 +1033,14 @@
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<p class="form-hint" id="customModelPickHint"></p>
|
||||
<!-- Case-insensitive filter over the rendered rows (app.filterCustomModelPick):
|
||||
a display layer only, the rows themselves are never re-rendered. Reset and
|
||||
focused on every open, same in-memory rule as the rail's search box. -->
|
||||
<input type="search" id="customModelPickSearch" class="session-sidebar-filter-input custom-model-pick-search"
|
||||
placeholder="Search models" aria-label="Search models"
|
||||
autocomplete="off" autocapitalize="off" spellcheck="false" enterkeyhint="search"
|
||||
oninput="app.filterCustomModelPick(this.value)">
|
||||
<p class="form-hint custom-model-pick-no-matches" id="customModelPickNoMatches" role="status" hidden>No models match</p>
|
||||
<div id="customModelPickList" class="run-mode-custom-models"></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2791,6 +2799,10 @@
|
||||
<div class="set-row-text"><span class="set-row-label">Base URL</span></div>
|
||||
<input type="text" id="customModelHostBaseUrl" class="set-input" placeholder="http://192.168.1.50:8080">
|
||||
</div>
|
||||
<div class="set-row has-field">
|
||||
<div class="set-row-text"><span class="set-row-label">Default context length (tokens)</span><span class="set-row-desc">Used for models whose size cannot be discovered. The model's whole window; the Claude floor check divides it by any stored slot count. Leave empty to let each CLI assume its own default.</span></div>
|
||||
<input type="number" id="customModelHostDefaultContextLength" class="set-input" min="1" max="100000000" step="1" placeholder="131072">
|
||||
</div>
|
||||
<div class="set-row has-field">
|
||||
<div class="set-row-text"><span class="set-row-label">API key</span><span class="set-row-desc">Optional. Left blank on edit keeps the existing key.</span></div>
|
||||
<input type="password" id="customModelHostApiKey" class="set-input" autocomplete="new-password">
|
||||
|
||||
@@ -1088,13 +1088,58 @@ Object.assign(CodemanApp.prototype, {
|
||||
const promotion = m === currentlyLoaded ? 'Currently loaded' : m === lastUsed ? 'Last used' : null;
|
||||
const tags = [promotion, m === host.defaultModelId ? 'Default' : null].filter(Boolean);
|
||||
const arg = escapeHtml(JSON.stringify(m));
|
||||
// data-model-id is the filter's haystack (escapeHtml covers `"` as well as
|
||||
// `<>&`, so a hostile model id cannot break out of the attribute); the
|
||||
// filter reads it back through dataset, which decodes it.
|
||||
return `
|
||||
<button class="run-mode-option" onclick="app.chooseCustomModelAndRun(${arg})">
|
||||
<button class="run-mode-option" data-model-id="${escapeHtml(m)}" onclick="app.chooseCustomModelAndRun(${arg})">
|
||||
<span class="run-mode-dot ${escapeHtml(mode)}"></span>${escapeHtml(m)}${tags.map((t) => ` <span class="set-scope">${escapeHtml(t)}</span>`).join('')}
|
||||
</button>`;
|
||||
})
|
||||
.join('');
|
||||
modal.classList.add('active');
|
||||
// A fresh open always starts unfiltered with the caret in the search box — the
|
||||
// list was just rebuilt, so clearing the input is the whole reset; focus is the
|
||||
// user's next keystroke away. Inside the generation guard above, so a
|
||||
// superseded probe can neither focus nor clear a modal that now shows another
|
||||
// endpoint's models.
|
||||
const search = document.getElementById('customModelPickSearch');
|
||||
if (search) {
|
||||
search.value = '';
|
||||
// readmymind-ui.js's rule: on touch devices never focus an input unprompted, or
|
||||
// the OS keyboard pops over the very list the picker just opened to show. Desktop
|
||||
// keeps the caret in the box, one keystroke from filtering.
|
||||
if (!(typeof MobileDetection !== 'undefined' && MobileDetection.isTouchDevice())) {
|
||||
search.focus();
|
||||
}
|
||||
}
|
||||
this.filterCustomModelPick('');
|
||||
},
|
||||
|
||||
/**
|
||||
* View filter for the open "Choose a model" picker: case-insensitive substring
|
||||
* over each row's model id, driven from the search input's own `oninput`.
|
||||
* Deliberately toggles `hidden` on the already-rendered buttons rather than
|
||||
* re-rendering from a second copy of the model list: the loaded/last-used
|
||||
* promotion, the Default pill, the escaped inline click argument and the async
|
||||
* generation guard all then stay exactly where they are, and clearing the
|
||||
* query restores the list byte for byte. ⚠️ `.run-mode-option` carries its own
|
||||
* `display: flex` (styles.css, restated in mobile.css), which outranks the UA
|
||||
* `[hidden]` rule, so styles.css re-asserts `display: none !important` for
|
||||
* these rows — without it the filter would silently do nothing.
|
||||
*/
|
||||
filterCustomModelPick(value) {
|
||||
const list = document.getElementById('customModelPickList');
|
||||
if (!list) return;
|
||||
const query = String(value || '').trim().toLowerCase();
|
||||
let visible = 0;
|
||||
for (const btn of list.querySelectorAll('button[data-model-id]')) {
|
||||
const match = !query || btn.dataset.modelId.toLowerCase().includes(query);
|
||||
btn.hidden = !match;
|
||||
if (match) visible += 1;
|
||||
}
|
||||
const noMatches = document.getElementById('customModelPickNoMatches');
|
||||
if (noMatches) noMatches.hidden = visible > 0;
|
||||
},
|
||||
|
||||
closeCustomModelPickModal() {
|
||||
|
||||
@@ -3875,6 +3875,9 @@ Object.assign(CodemanApp.prototype, {
|
||||
document.getElementById('customModelHostApiKey').value = ''; // the server never returns the real value (apiKeySet is a bool)
|
||||
document.getElementById('customModelHostApiKey').placeholder = host?.apiKeySet ? '•••••••• (unchanged if left blank)' : '';
|
||||
document.getElementById('customModelHostAuthStyle').value = host?.authStyle || 'bearer';
|
||||
// User-set fallback, round-tripped verbatim (discovery never touches it, so the
|
||||
// cached copy is authoritative); blank means "no default", cleared on save.
|
||||
document.getElementById('customModelHostDefaultContextLength').value = host?.defaultContextLength ?? '';
|
||||
this._populateCustomModelDefaultSelect(host);
|
||||
document.getElementById('customModelHostEditor').style.display = '';
|
||||
},
|
||||
@@ -3900,6 +3903,18 @@ Object.assign(CodemanApp.prototype, {
|
||||
const baseUrl = document.getElementById('customModelHostBaseUrl').value.trim();
|
||||
const apiKeyInput = document.getElementById('customModelHostApiKey').value;
|
||||
const authStyle = document.getElementById('customModelHostAuthStyle').value;
|
||||
// Blank means "no default" (key omitted, so PUT clears it); a value must satisfy
|
||||
// the same bound the server schema enforces, caught here for a friendlier message.
|
||||
const defaultContextLengthInput = document.getElementById('customModelHostDefaultContextLength').value.trim();
|
||||
let defaultContextLength;
|
||||
if (defaultContextLengthInput !== '') {
|
||||
const parsed = Number(defaultContextLengthInput);
|
||||
if (!Number.isInteger(parsed) || parsed < 1 || parsed > 100000000) {
|
||||
this.showToast('Default context length must be a whole number of tokens between 1 and 100,000,000', 'warning');
|
||||
return;
|
||||
}
|
||||
defaultContextLength = parsed;
|
||||
}
|
||||
const defaultModelId = document.getElementById('customModelHostDefaultModel').value || undefined;
|
||||
if (!id || !label || !baseUrl) {
|
||||
this.showToast('Id, label and base URL are all required', 'warning');
|
||||
@@ -3920,6 +3935,7 @@ Object.assign(CodemanApp.prototype, {
|
||||
baseUrl,
|
||||
authStyle,
|
||||
defaultModelId,
|
||||
defaultContextLength,
|
||||
apiKey: apiKeyInput || undefined,
|
||||
models: existing?.models,
|
||||
lastDiscoveredAt: existing?.lastDiscoveredAt,
|
||||
|
||||
@@ -7814,6 +7814,41 @@ html[data-skin="og"] .welcome-primary:hover {
|
||||
opacity: 0.8;
|
||||
}
|
||||
|
||||
/* The picker's search box reuses .session-sidebar-filter-input's theme-token
|
||||
styling (same reasoning as the .set-scope pill above) — only its own spacing
|
||||
and the no-matches line are added here. 0.35rem is the standard gap the
|
||||
modal's neighbouring form fields already use (`.form-hint`'s own margin-top):
|
||||
the global `* { margin: 0 }` reset left the box sitting flush under the CLI
|
||||
hint line above it, and the old 0.5rem read as too big a gap above the list
|
||||
(compounded by the always-visible no-matches line the rule below fixes). */
|
||||
#customModelPickModal .custom-model-pick-search {
|
||||
margin-top: 0.35rem;
|
||||
margin-bottom: 0.35rem;
|
||||
}
|
||||
|
||||
#customModelPickModal .custom-model-pick-no-matches {
|
||||
text-align: center;
|
||||
padding: 1rem 0;
|
||||
}
|
||||
|
||||
/* Same class of bug as the row rule below: the no-matches line carries
|
||||
`.form-hint`, whose `display: block` outranks the UA `[hidden]` rule, so
|
||||
`filterCustomModelPick()` toggling `hidden` changed nothing on screen and
|
||||
"No models match" showed while every model was listed with an empty search
|
||||
box. The `[hidden]` re-assertion makes the attribute hide the line again. */
|
||||
#customModelPickModal .custom-model-pick-no-matches[hidden] {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* A filtered-out row's `hidden` must actually hide it: `.run-mode-option` sets
|
||||
`display: flex` (and mobile.css restates the class in a media query), which
|
||||
outranks the UA `[hidden] { display: none }` rule, so the filter would
|
||||
silently do nothing without this re-assertion. Scoped to this modal's rows;
|
||||
the Run menu's own .run-mode-option entries never take `hidden`. */
|
||||
#customModelPickModal .run-mode-option[hidden] {
|
||||
display: none !important;
|
||||
}
|
||||
|
||||
/* Custom Model Endpoint Profiles: llama-swap model-swap confirmation — replaces a native
|
||||
confirm() popup (docs/custom-model-endpoints-plan.md) so it looks and feels like the
|
||||
rest of the app instead of a browser chrome dialog. Shares the context-window-too-small
|
||||
|
||||
@@ -21,7 +21,7 @@ import { isMultiUserMode } from '../../config/multiuser.js';
|
||||
import { getDataDir } from '../../config/instance.js';
|
||||
import { isBlockedWebviewUrl } from '../webview-egress-policy.js';
|
||||
import { egressBlockedReason, webviewFetch } from '../webview-egress.js';
|
||||
import { CustomModelHostSchema } from '../schemas.js';
|
||||
import { CustomModelHostSchema, MAX_DISCOVERED_CONTEXT_TOKENS } from '../schemas.js';
|
||||
import { readCustomModelHosts, writeCustomModelHosts, type CustomModelHost } from '../../custom-model-hosts.js';
|
||||
import type { CliEntry } from '../../config/cli-registry/types.js';
|
||||
|
||||
@@ -158,9 +158,14 @@ function authHeaders(host: Pick<CustomModelHost, 'apiKey' | 'authStyle'>): Recor
|
||||
|
||||
export interface DiscoveryResult {
|
||||
models: string[];
|
||||
/** See `CustomModelHost.modelContextLengths` — only ever populated for models already loaded. */
|
||||
/**
|
||||
* See `CustomModelHost.modelContextLengths`: populated from a model's own `/v1/models`
|
||||
* entry (`context_length`, else `context_window`) whenever the server states one, and
|
||||
* from the launch command or `/props` for loaded models, which overwrite (and so win
|
||||
* over) any listed figure for the same model.
|
||||
*/
|
||||
contextLengths: Record<string, number>;
|
||||
/** See `CustomModelHost.modelSlotCounts`; populated on the same loaded-models-only basis as `contextLengths`. */
|
||||
/** See `CustomModelHost.modelSlotCounts`; populated only for models the server reports loaded (launch command or `/props`). No listing states a slot count, unlike `contextLengths`. */
|
||||
slotCounts: Record<string, number>;
|
||||
/** See `CustomModelHost.modelSizesGB` — populated for every model whose own listing states one. */
|
||||
sizesGB: Record<string, number>;
|
||||
@@ -182,6 +187,20 @@ function parseSizeGB(description: unknown): number | undefined {
|
||||
return Number.isFinite(size) && size > 0 ? size : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort: reads a context length (tokens) stated by a model's OWN entry in
|
||||
* `GET /v1/models`. OpenRouter-style servers call the field `context_length`; current
|
||||
* llama-swap sends both `context_length` and `context_window` per model, and current
|
||||
* Bifrost (a gateway in front of llama-swap) passes `context_window` through. Older
|
||||
* servers send neither, and a server that states nothing gets no entry, never a guess.
|
||||
* Non-numeric, zero, negative, fractional or over-the-ceiling values are ignored, the
|
||||
* same rule the stored field's schema enforces: a wrong figure here is worse than none.
|
||||
*/
|
||||
function parseListedContextLength(value: unknown): number | undefined {
|
||||
if (typeof value !== 'number' || !Number.isInteger(value) || value <= 0) return undefined;
|
||||
return value <= MAX_DISCOVERED_CONTEXT_TOKENS ? value : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort: fetches `GET /props?model=<id>` (llama.cpp-native, llama-swap-proxied) for
|
||||
* ONE already-loaded model and pulls its `n_ctx` and `total_slots` out. Never called for a model that
|
||||
@@ -312,25 +331,42 @@ async function discoverModels(
|
||||
});
|
||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||
const body = (await res.json()) as {
|
||||
data?: Array<{ id?: unknown; status?: { value?: unknown }; description?: unknown }>;
|
||||
data?: Array<{
|
||||
id?: unknown;
|
||||
status?: { value?: unknown };
|
||||
description?: unknown;
|
||||
context_length?: unknown;
|
||||
context_window?: unknown;
|
||||
}>;
|
||||
};
|
||||
const entries = body.data ?? [];
|
||||
const models = entries.map((m) => m.id).filter((id): id is string => typeof id === 'string' && id.length > 0);
|
||||
|
||||
const sizesGB: Record<string, number> = {};
|
||||
// The lowest-precedence context source, and the only one a gateway without /running or
|
||||
// /props of its own (e.g. Bifrost) can offer: the figure a model's own listing states
|
||||
// (`context_length`, else `context_window`). The loaded-model pass below OVERWRITES
|
||||
// these for whatever it reads from the launch command or /props, so precedence per
|
||||
// model stays: `/running` cmd, then `/props`, then this listed figure.
|
||||
const contextLengths: Record<string, number> = {};
|
||||
for (const entry of entries) {
|
||||
if (typeof entry.id !== 'string' || !entry.id) continue;
|
||||
const size = parseSizeGB(entry.description);
|
||||
if (size !== undefined) sizesGB[entry.id] = size;
|
||||
const listed = parseListedContextLength(entry.context_length) ?? parseListedContextLength(entry.context_window);
|
||||
if (listed !== undefined) contextLengths[entry.id] = listed;
|
||||
}
|
||||
|
||||
// llama-swap-specific, feature-detected: a server that never mentions `status` on ANY
|
||||
// entry gets no context-length enrichment at all, rather than treating "no status field"
|
||||
// as "assume unloaded" — either reading is a guess, and skipping is the safe one, since
|
||||
// fetchContextLength must only ever run against a model this server itself calls loaded.
|
||||
// entry gets no launch-command/props context enrichment at all, rather than treating
|
||||
// "no status field" as "assume unloaded"; either reading is a guess, and skipping is
|
||||
// the safe one, since fetchContextLength must only ever run against a model this server
|
||||
// itself calls loaded. Listed figures above are unaffected: reading them triggers nothing.
|
||||
const hasStatusField = entries.some((m) => m && typeof m === 'object' && 'status' in m);
|
||||
const contextLengths: Record<string, number> = {};
|
||||
const slotCounts: Record<string, number> = {};
|
||||
// Ids whose figure THIS round's loaded-model probe answered — the listed figure remains
|
||||
// the source only for the ids NOT in this set (see the per-slot correction below).
|
||||
const ctxFromProbe = new Set<string>();
|
||||
if (hasStatusField) {
|
||||
const loadedIds = entries
|
||||
.filter((m) => m.status && typeof m.status === 'object' && (m.status as { value?: unknown }).value === 'loaded')
|
||||
@@ -364,11 +400,30 @@ async function discoverModels(
|
||||
// merges onto what was learned before).
|
||||
if (cmdSlots !== undefined && ctxFromProps) slots = 1;
|
||||
}
|
||||
if (ctx !== undefined) contextLengths[id] = ctx;
|
||||
// Assigning only when a probe answered keeps the listed figure (if any) when
|
||||
// neither the launch command nor /props stated one, and replaces it with the
|
||||
// stronger source whenever they did: cmd > /props > listed, never the reverse.
|
||||
if (ctx !== undefined) {
|
||||
contextLengths[id] = ctx;
|
||||
ctxFromProbe.add(id);
|
||||
}
|
||||
if (slots !== undefined) slotCounts[id] = slots;
|
||||
}
|
||||
}
|
||||
}
|
||||
// llama-swap documents the LISTED `context_length` as the PER-SLOT window, so whenever
|
||||
// the listed figure is the one taken this round — the model was not loaded, or the
|
||||
// probe answered nothing — its slot count is 1 by definition: the floor check must NOT
|
||||
// divide a per-slot figure a second time. Without recording 1 here, the count a loaded
|
||||
// round stored survives the merge (applyDiscoveredModels keeps what a later round does
|
||||
// not overwrite): 262144 @ 4 slots stored, model unloaded, listed 65536 → 65536 with 4
|
||||
// slots → a false 16384-per-slot warning. Even a count this round's launch command
|
||||
// parsed must yield — pairing a per-slot listed figure with the cmd's --parallel is the
|
||||
// same double division. Same rule the /props branch above applies. Ids whose figure the
|
||||
// probe DID answer keep their probe-derived count untouched.
|
||||
for (const id of Object.keys(contextLengths)) {
|
||||
if (!ctxFromProbe.has(id)) slotCounts[id] = 1;
|
||||
}
|
||||
return { models, contextLengths, slotCounts, sizesGB };
|
||||
}
|
||||
|
||||
|
||||
@@ -70,8 +70,9 @@ import {
|
||||
SessionWaitOutputQuerySchema,
|
||||
CustomModelSelectionSchema,
|
||||
} from '../schemas.js';
|
||||
import { readCustomModelHosts } from '../../custom-model-hosts.js';
|
||||
import { readCustomModelHosts, resolveCustomModelContextLength } from '../../custom-model-hosts.js';
|
||||
import { applyCustomModelInjection, removeConfigDir } from '../../custom-model-injection-apply.js';
|
||||
import { buildCustomModelInjection } from '../../custom-model-injection.js';
|
||||
import {
|
||||
getLlamaSwapStatus,
|
||||
triggerLlamaSwapLoad,
|
||||
@@ -1541,7 +1542,10 @@ export function registerSessionRoutes(
|
||||
if (!endpoint) {
|
||||
return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
|
||||
}
|
||||
const contextLength = endpoint.modelContextLengths?.[body.modelId];
|
||||
// Discovery's per-model figure wins; the endpoint's user-set default covers models
|
||||
// whose size discovery could not determine (a gateway that states none, say). With
|
||||
// neither, the existing "no override, CLI assumes its own default" behaviour stands.
|
||||
const contextLength = resolveCustomModelContextLength(endpoint, body.modelId);
|
||||
const slotCount = endpoint.modelSlotCounts?.[body.modelId];
|
||||
|
||||
// Some CLIs (today: only claude) carry enough of their own fixed system-prompt/tool-
|
||||
@@ -1608,10 +1612,28 @@ export function registerSessionRoutes(
|
||||
}
|
||||
|
||||
// A CLI whose config alone cannot select the model also gets its `model` launch param
|
||||
// forced (pi/omp `custom/<id>`, grok's block name). The argv engine DROPS a token that
|
||||
// forced (pi/omp `custom/<id>`, grok's block name, and codex now carries its bare
|
||||
// selected id too). The argv engine DROPS a token that
|
||||
// fails its pattern rather than quoting it, which would silently launch the CLI on its
|
||||
// own default provider again, so refuse an id the pattern cannot carry up front.
|
||||
// The check runs on the PURE build result, before applyCustomModelInjection below:
|
||||
// the forced launch id only comes out of that build, and writing first would mean a
|
||||
// refused id already replaced the config dir a live codex session uses as CODEX_HOME.
|
||||
const modelSpec = entry.launch.params.model;
|
||||
const injection = buildCustomModelInjection(entry, endpoint, body.modelId, contextLength);
|
||||
if (injection.kind === 'unsupported') {
|
||||
return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`);
|
||||
}
|
||||
if (
|
||||
injection.launchModel !== undefined &&
|
||||
modelSpec?.type === 'token' &&
|
||||
!matchesPattern(modelSpec.pattern, injection.launchModel)
|
||||
) {
|
||||
return createErrorResponse(
|
||||
ApiErrorCode.INVALID_INPUT,
|
||||
`Model id ${JSON.stringify(body.modelId)} cannot be passed to ${session.mode} on its command line`
|
||||
);
|
||||
}
|
||||
const applied = applyCustomModelInjection(
|
||||
entry,
|
||||
endpoint,
|
||||
@@ -1623,17 +1645,6 @@ export function registerSessionRoutes(
|
||||
if (!applied) {
|
||||
return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`);
|
||||
}
|
||||
if (
|
||||
applied.launchModel !== undefined &&
|
||||
modelSpec?.type === 'token' &&
|
||||
!matchesPattern(modelSpec.pattern, applied.launchModel)
|
||||
) {
|
||||
removeConfigDir(applied.configDir);
|
||||
return createErrorResponse(
|
||||
ApiErrorCode.INVALID_INPUT,
|
||||
`Model id ${JSON.stringify(body.modelId)} cannot be passed to ${session.mode} on its command line`
|
||||
);
|
||||
}
|
||||
|
||||
const { previousConfigDir } = session.setCustomModel(
|
||||
{
|
||||
@@ -4447,7 +4458,9 @@ export function registerSessionRoutes(
|
||||
const cmHosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR);
|
||||
const cmEndpoint = cmHosts.find((h) => h.id === customModel.endpointId);
|
||||
if (!cmEndpoint) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found');
|
||||
const cmContextLength = cmEndpoint.modelContextLengths?.[customModel.modelId];
|
||||
// Per-model discovery figure first, user-set endpoint default second, nothing third
|
||||
// (see the dedicated route's own comment above).
|
||||
const cmContextLength = resolveCustomModelContextLength(cmEndpoint, customModel.modelId);
|
||||
const cmSlotCount = cmEndpoint.modelSlotCounts?.[customModel.modelId];
|
||||
|
||||
// See the dedicated route's own comment for the full reasoning: some CLIs' own fixed
|
||||
|
||||
+14
-1
@@ -2387,6 +2387,14 @@ export const WebviewUpdateSchema = WebviewBaseSchema.partial();
|
||||
/** POST /api/webviews/probe: reachability + framing check for the editor's Test button. */
|
||||
export const WebviewProbeSchema = z.object({ url: webviewUrlSchema });
|
||||
|
||||
/**
|
||||
* Ceiling on a stated context length (tokens), shared between the bounds below and
|
||||
* discovery (`custom-model-routes.ts`): anything above it reads as nonsense and is
|
||||
* ignored, because a bogus figure here overflows the smaller server, which is the exact
|
||||
* failure the discovery field exists to prevent.
|
||||
*/
|
||||
export const MAX_DISCOVERED_CONTEXT_TOKENS = 100_000_000;
|
||||
|
||||
// Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md) — a
|
||||
// user-configured custom OpenAI-compatible endpoint, local (llama.cpp) or cloud
|
||||
// (Azure AI Foundry, etc.). Lives below `webviewUrlSchema` because `baseUrl` IS that
|
||||
@@ -2410,11 +2418,16 @@ export const CustomModelHostSchema = z.object({
|
||||
defaultModelId: z.string().max(200).optional(),
|
||||
// Server-populated by discovery (custom-model-routes.ts); accepted here only so a client
|
||||
// round-tripping the GET response back through PUT (edit-save) doesn't drop it.
|
||||
modelContextLengths: z.record(z.string().max(200), z.number().int().positive().max(100_000_000)).optional(),
|
||||
modelContextLengths: z
|
||||
.record(z.string().max(200), z.number().int().positive().max(MAX_DISCOVERED_CONTEXT_TOKENS))
|
||||
.optional(),
|
||||
// Same reasoning as modelContextLengths above.
|
||||
modelSlotCounts: z.record(z.string().max(200), z.number().int().positive().max(4096)).optional(),
|
||||
// Same reasoning as modelContextLengths above.
|
||||
modelSizesGB: z.record(z.string().max(200), z.number().positive().max(100_000)).optional(),
|
||||
// User-set fallback for models discovery has no context length for (session-routes.ts);
|
||||
// discovery never writes it. Same bound as the modelContextLengths values above.
|
||||
defaultContextLength: z.number().int().positive().max(MAX_DISCOVERED_CONTEXT_TOKENS).optional(),
|
||||
});
|
||||
|
||||
/**
|
||||
|
||||
+2
-2
@@ -80,7 +80,7 @@ import { attachmentRegistry, buildFileThumbnailRoute, registerExternalAttachment
|
||||
import { folderGrants } from '../folder-listing.js';
|
||||
import { getCli, enabledClis, listClis } from '../config/cli-registry/registry.js';
|
||||
import { isCliEntryInstalled, probeStockCliAvailability } from '../utils/cli-installed-probes.js';
|
||||
import { readCustomModelHosts } from '../custom-model-hosts.js';
|
||||
import { readCustomModelHosts, resolveCustomModelContextLength } from '../custom-model-hosts.js';
|
||||
import { applyCustomModelInjection, customModelConfigDir, removeConfigDir } from '../custom-model-injection-apply.js';
|
||||
import type { CustomModelBookkeeping } from '../types/session.js';
|
||||
import { registerGeneratedArtifactAttachment } from '../generated-artifact-attachments.js';
|
||||
@@ -1470,7 +1470,7 @@ export class WebServer extends EventEmitter {
|
||||
endpoint,
|
||||
saved.modelId,
|
||||
session.id,
|
||||
endpoint.modelContextLengths?.[saved.modelId]
|
||||
resolveCustomModelContextLength(endpoint, saved.modelId)
|
||||
)?.envOverrides;
|
||||
} catch (err) {
|
||||
console.warn('[WebServer] Failed to rebuild custom-model env on recovery:', err);
|
||||
|
||||
@@ -171,6 +171,15 @@ describe('codex', () => {
|
||||
expect(cx({ model: 'gpt-5', resumeSessionId: 'roll_42' })).toBe('codex --model gpt-5 resume roll_42');
|
||||
});
|
||||
|
||||
it('carries an Ollama-style `qwen3:32b` model id, and still drops a leading-flag shape', () => {
|
||||
// `model-plain-colon`: the custom-model feature forces the selected id onto this param,
|
||||
// and ids like `qwen3:32b` carry a colon. The leading-alphanumeric rule is what keeps a
|
||||
// flag from riding the flag, so `:x` and `-x` must still drop out.
|
||||
expect(cx({ model: 'qwen3:32b' })).toBe('codex --model qwen3:32b');
|
||||
expect(cx({ model: ':leading' })).toBe('codex');
|
||||
expect(cx({ model: '-yolo' })).toBe('codex');
|
||||
});
|
||||
|
||||
it('sends reasoning effort as one model_reasoning_effort config value, for every level', () => {
|
||||
for (const level of CODEX_REASONING_EFFORTS) {
|
||||
expect(cx({ reasoningEffort: level })).toBe(`codex --config model_reasoning_effort=${level}`);
|
||||
|
||||
@@ -481,6 +481,52 @@ describe('refreshAllCustomModelHosts: parallel-slot enrichment (llama.cpp splits
|
||||
const [updated] = await readCustomModelHosts(dir);
|
||||
expect(updated.modelSlotCounts).toEqual({ a: 4 });
|
||||
});
|
||||
|
||||
it('a round where only the listed figure answers resets the slot count a loaded round learned', async () => {
|
||||
// The false-warning scenario, measured on a real llama-swap: round 1 has the model
|
||||
// loaded under `--ctx-size 262144 --parallel 4` — a TOTAL window shared across slots,
|
||||
// correctly stored with its count. Round 2 finds it unloaded and takes the LISTED
|
||||
// `context_length`, which llama-swap documents as the PER-SLOT window. If the stored 4
|
||||
// survived, the floor check would divide the per-slot figure a second time
|
||||
// (65536/4 = 16384) and warn about a model that serves 65536 to every request.
|
||||
const dir = getDataDir();
|
||||
await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]);
|
||||
fetchMock.mockImplementation(async (url: URL) => {
|
||||
if (url.pathname === '/v1/models') {
|
||||
return new Response(JSON.stringify({ data: [{ id: 'm', status: { value: 'loaded' } }] }), { status: 200 });
|
||||
}
|
||||
if (url.pathname === '/running') {
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
running: [{ model: 'm', state: 'ready', cmd: 'llama-server -m /models/m.gguf --ctx-size 262144 --parallel 4' }],
|
||||
}),
|
||||
{ status: 200 }
|
||||
);
|
||||
}
|
||||
throw new Error(`unexpected request: ${url.href}`); // the command answers both questions
|
||||
});
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
let [round1] = await readCustomModelHosts(dir);
|
||||
expect(round1.modelContextLengths).toEqual({ m: 262144 });
|
||||
expect(round1.modelSlotCounts).toEqual({ m: 4 });
|
||||
|
||||
// Round 2: unloaded, no /running or /props probe in order, listing its per-slot figure.
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(JSON.stringify({ data: [{ id: 'm', status: { value: 'unloaded' }, context_length: 65536 }] }), {
|
||||
status: 200,
|
||||
})
|
||||
);
|
||||
|
||||
await refreshAllCustomModelHosts();
|
||||
|
||||
const [round2] = await readCustomModelHosts(dir);
|
||||
expect(round2.modelContextLengths).toEqual({ m: 65536 });
|
||||
// 1 slot, not the stale 4: 65536 per slot with 1 recorded slot is the honest pair, so
|
||||
// the floor check reads 65536/request — exactly what the server gives a single request.
|
||||
expect(round2.modelSlotCounts).toEqual({ m: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('refreshAllCustomModelHosts: model-size enrichment (parsed from /v1/models description)', () => {
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { parse as parseToml } from 'smol-toml';
|
||||
import { getCli } from '../src/config/cli-registry/index.js';
|
||||
import {
|
||||
buildCustomModelInjection,
|
||||
@@ -112,6 +113,31 @@ describe('buildCustomModelInjection', () => {
|
||||
expect(parsed.provider.custom.models.qwen3).toEqual({});
|
||||
});
|
||||
|
||||
it('opencode: threads a discovered context length into the model limit as the full window (issue #749)', () => {
|
||||
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', 16384);
|
||||
if (result.kind !== 'env') throw new Error('unreachable');
|
||||
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
|
||||
// `limit.context` IS the full window to opencode — it reserves for the reply itself —
|
||||
// so the discovered total passes through; `output` only caps the reply at a quarter.
|
||||
expect(parsed.provider.custom.models.qwen3).toEqual({ limit: { context: 16384, output: 4096 } });
|
||||
});
|
||||
|
||||
it('opencode: caps the response headroom reservation at 8192 for big windows', () => {
|
||||
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', 262144);
|
||||
if (result.kind !== 'env') throw new Error('unreachable');
|
||||
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
|
||||
expect(parsed.provider.custom.models.qwen3).toEqual({ limit: { context: 262144, output: 8192 } });
|
||||
});
|
||||
|
||||
it('opencode: never invents a limit for an unknown or non-finite context length (issue #749 regression)', () => {
|
||||
for (const contextLength of [undefined, NaN]) {
|
||||
const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', contextLength);
|
||||
if (result.kind !== 'env') throw new Error('unreachable');
|
||||
const parsed = JSON.parse(result.envOverrides.OPENCODE_CONFIG_CONTENT);
|
||||
expect(parsed.provider.custom.models.qwen3).toEqual({});
|
||||
}
|
||||
});
|
||||
|
||||
it('codex: configDir writes an isolated config.toml with model/base_url, and the key rides as extraEnv (never a literal TOML field)', () => {
|
||||
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3');
|
||||
expect(result.kind).toBe('configDir');
|
||||
@@ -133,6 +159,30 @@ describe('buildCustomModelInjection', () => {
|
||||
expect(result.files[0].content).toContain('model = "weird\\"model"');
|
||||
});
|
||||
|
||||
it('codex: writes the discovered window as TOP-LEVEL model_context_window with a 75% compact limit (issue #750)', () => {
|
||||
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3', 16384);
|
||||
if (result.kind !== 'configDir') throw new Error('unreachable');
|
||||
const doc = parseToml(result.files[0].content) as Record<string, unknown>;
|
||||
// Top-level is the shape codex reads: parked under [model_providers.custom] (its
|
||||
// provider table) the same keys would be silently ignored by codex itself.
|
||||
expect(doc.model_context_window).toBe(16384);
|
||||
expect(doc.model_auto_compact_token_limit).toBe(12288); // 75% of the window, floored
|
||||
const provider = (doc.model_providers as Record<string, Record<string, unknown>>).custom;
|
||||
expect(provider.model_context_window).toBeUndefined();
|
||||
expect(provider.model_auto_compact_token_limit).toBeUndefined();
|
||||
});
|
||||
|
||||
it('codex: writes neither context line for an unknown, non-finite or zero context length', () => {
|
||||
for (const contextLength of [undefined, NaN, 0]) {
|
||||
const result = buildCustomModelInjection(entryOrThrow('codex'), endpoint, 'qwen3', contextLength);
|
||||
if (result.kind !== 'configDir') throw new Error('unreachable');
|
||||
const doc = parseToml(result.files[0].content) as Record<string, unknown>;
|
||||
expect(doc.model_context_window, String(contextLength)).toBeUndefined();
|
||||
expect(doc.model_auto_compact_token_limit, String(contextLength)).toBeUndefined();
|
||||
expect(doc.model).toBe('qwen3'); // the rest of the file is still written
|
||||
}
|
||||
});
|
||||
|
||||
it('pi: configDir writes .pi/agent/models.json, redirected via HOME (verified live — PI_CONFIG_DIR does nothing for pi)', () => {
|
||||
const result = buildCustomModelInjection(entryOrThrow('pi'), endpoint, 'qwen3');
|
||||
if (result.kind !== 'configDir') throw new Error('unreachable');
|
||||
@@ -232,7 +282,9 @@ describe('launchModel (the model launch param that selects the injected provider
|
||||
});
|
||||
|
||||
it('CLIs whose config selects the model on its own declare none', () => {
|
||||
for (const id of ['claude', 'opencode', 'codex', 'gemini', 'deepseek']) {
|
||||
// codex now declares launchModel (selected-model launch override) and is covered by
|
||||
// its own restart coverage, not here.
|
||||
for (const id of ['claude', 'opencode', 'gemini', 'deepseek']) {
|
||||
const result = buildCustomModelInjection(entryOrThrow(id), endpoint, 'qwen3');
|
||||
if (result.kind === 'unsupported') throw new Error('unreachable');
|
||||
expect(result.launchModel, id).toBeUndefined();
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* @fileoverview Boot recovery rebuilds a saved custom-model selection through the SAME
|
||||
* context-length resolution the routes use: the model's discovered figure, falling back to
|
||||
* the endpoint's `defaultContextLength` (`resolveCustomModelContextLength`). Before the
|
||||
* shared helper, `_rebuildCustomModelEnv` read `modelContextLengths` alone — so a session
|
||||
* on a model that only the endpoint default covers came back from a Codeman restart with
|
||||
* `CLAUDE_CODE_MAX_CONTEXT_TOKENS` (or a config file's context figure) silently gone,
|
||||
* re-arming exactly the window overflow the var exists to prevent.
|
||||
*
|
||||
* `_rebuildCustomModelEnv` reads no `this` state (it re-derives everything from the endpoint
|
||||
* store and the Session it is handed), so it is driven straight off the prototype with an
|
||||
* empty receiver — `restoreMuxSessions()` itself cannot be reached under vitest, the same
|
||||
* constraint recorded in test/session-model-recovery.test.ts.
|
||||
*/
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
import { getDataDir } from '../src/config/instance.js';
|
||||
import { writeCustomModelHosts, type CustomModelHost } from '../src/custom-model-hosts.js';
|
||||
import { customModelConfigDir, removeConfigDir } from '../src/custom-model-injection-apply.js';
|
||||
import { Session } from '../src/session.js';
|
||||
import { WebServer } from '../src/web/server.js';
|
||||
import type { CustomModelBookkeeping } from '../src/types/session.js';
|
||||
|
||||
type RebuildFn = (session: Session, saved: CustomModelBookkeeping) => Promise<Record<string, string> | undefined>;
|
||||
|
||||
/** The real method, with an empty receiver — see the file header for why that is sound. */
|
||||
function rebuildCustomModelEnv(session: Session, saved: CustomModelBookkeeping) {
|
||||
const fn = (WebServer.prototype as unknown as { _rebuildCustomModelEnv: RebuildFn })._rebuildCustomModelEnv;
|
||||
expect(typeof fn, '_rebuildCustomModelEnv vanished from WebServer.prototype').toBe('function');
|
||||
return fn.call({}, session, saved);
|
||||
}
|
||||
|
||||
function endpoint(overrides: Partial<CustomModelHost> = {}): CustomModelHost {
|
||||
return {
|
||||
id: 'ep',
|
||||
label: 'Ep',
|
||||
baseUrl: 'http://localhost:8080',
|
||||
apiKey: 'recovery-probe-key',
|
||||
models: ['qwen3'],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('_rebuildCustomModelEnv: context-length fallback on boot recovery', () => {
|
||||
const configs: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const id of configs.splice(0)) removeConfigDir(customModelConfigDir(id));
|
||||
});
|
||||
|
||||
it('falls back to the endpoint defaultContextLength when discovery recorded no per-model figure', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), [endpoint({ defaultContextLength: 32768 })]);
|
||||
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
|
||||
configs.push(session.id);
|
||||
|
||||
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
|
||||
|
||||
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('32768');
|
||||
});
|
||||
|
||||
it('a discovered per-model figure still wins over the endpoint default', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), [
|
||||
endpoint({ modelContextLengths: { qwen3: 131072 }, defaultContextLength: 32768 }),
|
||||
]);
|
||||
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
|
||||
configs.push(session.id);
|
||||
|
||||
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
|
||||
|
||||
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('131072');
|
||||
});
|
||||
|
||||
it('states nothing rather than guessing when neither source knows a window', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), [endpoint()]);
|
||||
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
|
||||
configs.push(session.id);
|
||||
|
||||
const env = await rebuildCustomModelEnv(session, { endpointId: 'ep', modelId: 'qwen3', envKeys: [] });
|
||||
|
||||
expect(env?.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBeUndefined();
|
||||
});
|
||||
|
||||
it('drops the selection to undefined (kept for clearing) when the endpoint no longer exists', async () => {
|
||||
await writeCustomModelHosts(getDataDir(), []);
|
||||
const session = new Session({ workingDir: '/tmp', mode: 'claude' });
|
||||
configs.push(session.id);
|
||||
|
||||
await expect(rebuildCustomModelEnv(session, { endpointId: 'gone', modelId: 'qwen3', envKeys: [] })).resolves.toBe(
|
||||
undefined
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -20,6 +20,10 @@ import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const CONSTANTS_JS = readFileSync(new URL('../src/web/public/constants.js', import.meta.url), 'utf-8');
|
||||
const SESSION_UI_JS = readFileSync(new URL('../src/web/public/session-ui.js', import.meta.url), 'utf-8');
|
||||
// Only for the [hidden] CSS guard below — the filter toggles the `hidden` property, and
|
||||
// `.run-mode-option`'s own `display: flex` would silently defeat it without the
|
||||
// `!important` re-assertion in styles.css.
|
||||
const STYLES_CSS = readFileSync(new URL('../src/web/public/styles.css', import.meta.url), 'utf-8');
|
||||
// Only for the real _showCenterStatus DOM tests below (`bootAppWithRealCenterStatus`) —
|
||||
// every other test in this file stubs _showCenterStatus itself and has no need of it.
|
||||
const PANELS_UI_JS = readFileSync(new URL('../src/web/public/panels-ui.js', import.meta.url), 'utf-8');
|
||||
@@ -55,6 +59,9 @@ function bootApp(
|
||||
<div class="modal" id="customModelPickModal">
|
||||
<h3 id="customModelPickTitle"></h3>
|
||||
<p id="customModelPickHint"></p>
|
||||
<input type="search" id="customModelPickSearch" aria-label="Search models"
|
||||
oninput="app.filterCustomModelPick(this.value)">
|
||||
<p id="customModelPickNoMatches" role="status" hidden>No models match</p>
|
||||
<div id="customModelPickList"></div>
|
||||
</div>
|
||||
<div class="modal" id="customModelSwapConfirmModal">
|
||||
@@ -532,6 +539,172 @@ describe('Custom Model Endpoint Profiles: the "which model" picker', () => {
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
expect(toastMessage).toMatch(/no models discovered/i);
|
||||
});
|
||||
|
||||
it('resets and focuses the search box every time the picker opens', async () => {
|
||||
const { win, app } = bootApp({
|
||||
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
|
||||
});
|
||||
win.app = app;
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
// The caret lands in the box, so the first keystroke filters with no click.
|
||||
expect(win.document.activeElement).toBe(search);
|
||||
// A stale query from a previous open must not survive into the next one.
|
||||
search.value = 'zzz';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
app.closeCustomModelPickModal();
|
||||
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
expect(search.value).toBe('');
|
||||
expect(win.document.activeElement).toBe(search);
|
||||
expect(win.document.getElementById('customModelPickNoMatches')!.hasAttribute('hidden')).toBe(true);
|
||||
const visible = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')].filter(
|
||||
(b) => !b.hasAttribute('hidden')
|
||||
);
|
||||
expect(visible).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('leaves the search box unfocused on touch devices so the OS keyboard stays off the list', async () => {
|
||||
const { win, app } = bootApp({
|
||||
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
|
||||
});
|
||||
win.app = app;
|
||||
// The guard reads the global the browser defines (mobile-detection.js).
|
||||
(win as unknown as { MobileDetection: { isTouchDevice: () => boolean } }).MobileDetection = {
|
||||
isTouchDevice: () => true,
|
||||
};
|
||||
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
expect(win.document.activeElement).not.toBe(search);
|
||||
// Only the focus is skipped: the rows still render, unfiltered and functional.
|
||||
const visible = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')].filter(
|
||||
(b) => !b.hasAttribute('hidden')
|
||||
);
|
||||
expect(visible).toHaveLength(2);
|
||||
});
|
||||
|
||||
it('filters rows case-insensitively on the model id, keeping the surviving row fully functional', async () => {
|
||||
const { win, app } = bootApp({
|
||||
hosts: [
|
||||
{
|
||||
id: 'llama-box',
|
||||
label: 'llama.cpp',
|
||||
baseUrl: 'http://x',
|
||||
models: ['qwen3', 'llama3'],
|
||||
defaultModelId: 'qwen3',
|
||||
},
|
||||
],
|
||||
});
|
||||
let launched: unknown[] | null = null;
|
||||
app.runCustomModelEntry = async (...args: unknown[]) => {
|
||||
launched = args;
|
||||
};
|
||||
win.app = app;
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
search.value = 'QWEN';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
|
||||
const buttons = [...win.document.getElementById('customModelPickList')!.querySelectorAll('button')];
|
||||
expect(buttons.filter((b) => b.hasAttribute('hidden')).map((b) => b.textContent!.trim())).toEqual(['llama3']);
|
||||
const [visible] = buttons.filter((b) => !b.hasAttribute('hidden'));
|
||||
// The surviving row keeps its Default badge and its real, callable click handler.
|
||||
expect(visible.textContent).toContain('qwen3');
|
||||
expect(visible.textContent).toContain('Default');
|
||||
(visible as unknown as HTMLButtonElement & { onclick: (e: unknown) => void }).onclick(
|
||||
new (win as any).Event('click')
|
||||
);
|
||||
expect(launched).toEqual(['claude', 'llama-box', 'qwen3']);
|
||||
});
|
||||
|
||||
it('shows the no-matches line only while a query matches nothing, and clears it with the query', async () => {
|
||||
const { win, app } = bootApp({
|
||||
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
|
||||
});
|
||||
win.app = app;
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
const noMatches = win.document.getElementById('customModelPickNoMatches')!;
|
||||
const isHidden = () =>
|
||||
win.document.getElementById('customModelPickList')!.querySelectorAll('button[hidden]').length;
|
||||
|
||||
// Whitespace is not a query: everything stays visible, no message.
|
||||
search.value = ' ';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
expect(noMatches.hasAttribute('hidden')).toBe(true);
|
||||
expect(isHidden()).toBe(0);
|
||||
|
||||
search.value = 'ghost';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
expect(noMatches.hasAttribute('hidden')).toBe(false);
|
||||
expect(isHidden()).toBe(2);
|
||||
|
||||
search.value = '';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
expect(noMatches.hasAttribute('hidden')).toBe(true);
|
||||
expect(isHidden()).toBe(0);
|
||||
});
|
||||
|
||||
it('keeps an HTML-significant model id inert in the picker rows too, and filterable by its raw text', async () => {
|
||||
// Same vector as the Run-menu row above, now via data-model-id: escapeHtml covers
|
||||
// the double quote, and the filter reads the decoded raw id back off dataset.
|
||||
const dangerousModel = '"><img src=x onerror=alert(1)>';
|
||||
const { win, app } = bootApp({
|
||||
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', dangerousModel] }],
|
||||
});
|
||||
win.app = app;
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
const list = win.document.getElementById('customModelPickList')!;
|
||||
expect(list.querySelector('img')).toBeNull();
|
||||
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
search.value = 'src=x';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
const visible = list.querySelectorAll('button:not([hidden])');
|
||||
expect(visible).toHaveLength(1);
|
||||
expect(visible[0].getAttribute('data-model-id')).toBe(dangerousModel);
|
||||
});
|
||||
|
||||
it('re-asserts display:none !important for hidden picker rows so the filter cannot be silently defeated', () => {
|
||||
// `.run-mode-option` carries its own `display: flex` (styles.css, restated in
|
||||
// mobile.css), which outranks the UA `[hidden]` rule — without this scoped
|
||||
// !important re-assertion, toggling `hidden` would change nothing on screen.
|
||||
const rule = STYLES_CSS.match(/#customModelPickModal \.run-mode-option\[hidden\]\s*\{[^}]*\}/);
|
||||
expect(rule, 'picker rows need a [hidden] rule scoped to #customModelPickModal').toBeTruthy();
|
||||
expect(rule![0]).toMatch(/display:\s*none\s*!important/);
|
||||
});
|
||||
|
||||
it('pins "No models match" hidden with an empty query and shown only when a query hides every row, in CSS too', async () => {
|
||||
// The visible symptom: the line carries `.form-hint`, whose `display: block`
|
||||
// outranks the UA `[hidden]` rule, so `hidden` alone showed "No models match"
|
||||
// while every model was listed with an empty search box. The attribute half
|
||||
// was always right (session-ui.js's `filterCustomModelPick`); the CSS half
|
||||
// needed the same `[hidden] { display: none !important }` re-assertion the
|
||||
// rows above carry. This test pins both states AND the CSS that makes them
|
||||
// actually render.
|
||||
const rule = STYLES_CSS.match(/#customModelPickModal \.custom-model-pick-no-matches\[hidden\]\s*\{[^}]*\}/);
|
||||
expect(rule, 'the no-matches line needs a [hidden] rule scoped to #customModelPickModal').toBeTruthy();
|
||||
expect(rule![0]).toMatch(/display:\s*none\s*!important/);
|
||||
|
||||
const { win, app } = bootApp({
|
||||
hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3'] }],
|
||||
});
|
||||
win.app = app;
|
||||
await app.selectCustomModelEntry('claude', 'llama-box');
|
||||
const noMatches = win.document.getElementById('customModelPickNoMatches')!;
|
||||
|
||||
// Empty query, models listed: the attribute must be set, and the CSS rule
|
||||
// above is what makes that attribute actually hide the line.
|
||||
expect(noMatches.hasAttribute('hidden')).toBe(true);
|
||||
|
||||
// A non-empty query that hides every row: shown again.
|
||||
const search = win.document.getElementById('customModelPickSearch') as HTMLInputElement;
|
||||
search.value = 'ghost';
|
||||
search.dispatchEvent(new win.Event('input'));
|
||||
expect(noMatches.hasAttribute('hidden')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Custom Model Endpoint Profiles: _getCustomModelCurrentlyLoaded is client-side bounded', () => {
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
import { describe, it, expect, vi, afterEach } from 'vitest';
|
||||
import { registerCustomModelRoutes } from '../../src/web/routes/custom-model-routes.js';
|
||||
import { webviewFetch } from '../../src/web/webview-egress.js';
|
||||
import { createRouteTestHarness } from './_route-test-utils.js';
|
||||
import { createRouteTestHarness, type RouteTestHarness } from './_route-test-utils.js';
|
||||
|
||||
vi.mock('../../src/web/webview-egress.js', async () => {
|
||||
const actual = await vi.importActual<typeof import('../../src/web/webview-egress.js')>(
|
||||
@@ -417,3 +417,166 @@ describe('apiKey is never handed back to the browser', () => {
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('context-length discovery from the /v1/models listing', () => {
|
||||
// The gateway path (a Bifrost in front of llama-swap, an OpenRouter-style server):
|
||||
// no /running, no /props, just a per-model `context_length` or `context_window` in the
|
||||
// model listing. Distinct ports per test keep the file's shared data dir unambiguous.
|
||||
afterEach(() => {
|
||||
fetchMock.mockReset();
|
||||
});
|
||||
|
||||
type StoredHost = { id: string; modelContextLengths?: Record<string, number>; defaultContextLength?: number };
|
||||
|
||||
type HarnessApp = RouteTestHarness['app'];
|
||||
|
||||
async function endpointWith(id: string, port: number, extra: Record<string, unknown> = {}): Promise<HarnessApp> {
|
||||
const { app } = await setup();
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/model-endpoints',
|
||||
payload: { id, label: 'A', baseUrl: `http://localhost:${port}`, ...extra },
|
||||
});
|
||||
expect(res.statusCode).toBe(200);
|
||||
return app;
|
||||
}
|
||||
|
||||
async function storedHost(app: HarnessApp, id: string): Promise<StoredHost | undefined> {
|
||||
const list = await app.inject({ method: 'GET', url: '/api/model-endpoints' });
|
||||
return (list.json() as StoredHost[]).find((h) => h.id === id);
|
||||
}
|
||||
|
||||
it('reads context_length (preferred) then context_window from each entry into modelContextLengths', async () => {
|
||||
const app = await endpointWith('ep-ctx-listed', 8101);
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
data: [
|
||||
{ id: 'both', context_length: 131072, context_window: 32768 },
|
||||
{ id: 'window-only', context_window: 8192 },
|
||||
{ id: 'no-figure' },
|
||||
],
|
||||
}),
|
||||
{ status: 200 }
|
||||
)
|
||||
);
|
||||
|
||||
const discover = await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-listed/discover-models' });
|
||||
expect(discover.json().success).toBe(true);
|
||||
|
||||
const stored = await storedHost(app, 'ep-ctx-listed');
|
||||
expect(stored?.modelContextLengths).toEqual({ both: 131072, 'window-only': 8192 });
|
||||
});
|
||||
|
||||
it('ignores non-numeric, zero, negative and absurd listed values; the schema bound itself is still accepted', async () => {
|
||||
const app = await endpointWith('ep-ctx-bad', 8102);
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
data: [
|
||||
{ id: 'string', context_length: '131072' },
|
||||
{ id: 'zero', context_window: 0 },
|
||||
{ id: 'negative', context_length: -10 },
|
||||
{ id: 'over-bound', context_length: 100_000_001 },
|
||||
{ id: 'fractional', context_window: 16384.5 },
|
||||
{ id: 'at-bound', context_length: 100_000_000 },
|
||||
],
|
||||
}),
|
||||
{ status: 200 }
|
||||
)
|
||||
);
|
||||
|
||||
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-bad/discover-models' });
|
||||
|
||||
const stored = await storedHost(app, 'ep-ctx-bad');
|
||||
expect(stored?.modelContextLengths).toEqual({ 'at-bound': 100_000_000 });
|
||||
});
|
||||
|
||||
it('prefers the /running launch command over the listed figure for a loaded model', async () => {
|
||||
const app = await endpointWith('ep-ctx-cmd', 8103);
|
||||
fetchMock.mockImplementation(async (url: URL) => {
|
||||
if (url.pathname === '/v1/models') {
|
||||
return new Response(
|
||||
JSON.stringify({ data: [{ id: 'cmd-wins', status: { value: 'loaded' }, context_length: 131072 }] }),
|
||||
{ status: 200 }
|
||||
);
|
||||
}
|
||||
if (url.pathname === '/running') {
|
||||
return new Response(
|
||||
JSON.stringify({ running: [{ model: 'cmd-wins', state: 'ready', cmd: 'llama-server --ctx-size 16384' }] }),
|
||||
{ status: 200 }
|
||||
);
|
||||
}
|
||||
if (url.pathname === '/props') return new Response('nope', { status: 404 });
|
||||
throw new Error(`unexpected request in this test: ${url.href}`);
|
||||
});
|
||||
|
||||
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-cmd/discover-models' });
|
||||
|
||||
const stored = await storedHost(app, 'ep-ctx-cmd');
|
||||
expect(stored?.modelContextLengths).toEqual({ 'cmd-wins': 16384 });
|
||||
});
|
||||
|
||||
it('keeps a previously learned value the fresh listing says nothing about, and discovery never touches defaultContextLength', async () => {
|
||||
const app = await endpointWith('ep-ctx-keep', 8104, {
|
||||
models: ['re-listed', 'untouched'],
|
||||
modelContextLengths: { 're-listed': 16384, untouched: 4096 },
|
||||
defaultContextLength: 65536,
|
||||
});
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({ data: [{ id: 're-listed', context_length: 32768 }, { id: 'untouched' }] }),
|
||||
{ status: 200 }
|
||||
)
|
||||
);
|
||||
|
||||
await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-ctx-keep/discover-models' });
|
||||
|
||||
const stored = await storedHost(app, 'ep-ctx-keep');
|
||||
expect(stored?.modelContextLengths).toEqual({ 're-listed': 32768, untouched: 4096 });
|
||||
expect(stored?.defaultContextLength).toBe(65536);
|
||||
});
|
||||
});
|
||||
|
||||
describe('defaultContextLength (the user-set fallback discovery never overwrites)', () => {
|
||||
afterEach(() => {
|
||||
fetchMock.mockReset();
|
||||
});
|
||||
|
||||
it('stores it on create, updates it on PUT, and a PUT without it clears it', async () => {
|
||||
const { app } = await setup();
|
||||
const create = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/model-endpoints',
|
||||
payload: { id: 'ep-defctx', label: 'A', baseUrl: 'http://localhost:8105', defaultContextLength: 131072 },
|
||||
});
|
||||
expect(create.json().data.host.defaultContextLength).toBe(131072);
|
||||
|
||||
const update = await app.inject({
|
||||
method: 'PUT',
|
||||
url: '/api/model-endpoints/ep-defctx',
|
||||
payload: { label: 'A', baseUrl: 'http://localhost:8105', defaultContextLength: 32768 },
|
||||
});
|
||||
expect(update.json().data.host.defaultContextLength).toBe(32768);
|
||||
|
||||
const cleared = await app.inject({
|
||||
method: 'PUT',
|
||||
url: '/api/model-endpoints/ep-defctx',
|
||||
payload: { label: 'A', baseUrl: 'http://localhost:8105' },
|
||||
});
|
||||
expect(cleared.json().data.host.defaultContextLength).toBeUndefined();
|
||||
});
|
||||
|
||||
it('rejects zero, negative, fractional, over-the-bound and non-numeric values', async () => {
|
||||
const { app } = await setup();
|
||||
for (const defaultContextLength of [0, -1, 1.5, 100_000_001, '131072']) {
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/model-endpoints',
|
||||
payload: { id: 'ep-defctx-bad', label: 'A', baseUrl: 'http://localhost:8106', defaultContextLength },
|
||||
});
|
||||
expect(res.json().success, String(defaultContextLength)).toBe(false);
|
||||
expect(res.json().errorCode, String(defaultContextLength)).toBe('INVALID_INPUT');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -122,6 +122,9 @@ describe('POST /api/quick-start: customModel (one-shot custom-model launch)', ()
|
||||
const { sessionId } = res.json();
|
||||
const session = ctx.sessions.get(sessionId) as unknown as Session;
|
||||
expect(session.customModel?.endpointId).toBe('ep1');
|
||||
// #750, pinned the same way as the pi/grok/omp one-shot paths below: the FIRST
|
||||
// launch already carries the selected id, not just the file on disk.
|
||||
expect(session.getCustomModelForPersist()?.launchModel).toBe('qwen3');
|
||||
expect(restartSpy).not.toHaveBeenCalled();
|
||||
|
||||
const configDir = customModelConfigDir(sessionId);
|
||||
@@ -367,6 +370,50 @@ describe('POST /api/quick-start: customModel (one-shot custom-model launch)', ()
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(res.json().requiresContextWarning).toBeUndefined();
|
||||
});
|
||||
|
||||
it('warns using the endpoint defaultContextLength when discovery recorded no per-model figure', async () => {
|
||||
const endpoint: CustomModelHost = {
|
||||
id: 'ep-default',
|
||||
label: 'silent gateway',
|
||||
baseUrl: 'http://192.168.1.53:8080',
|
||||
apiKey: 'k',
|
||||
defaultContextLength: 16384,
|
||||
};
|
||||
await writeCustomModelHosts(getDataDir(), [ENDPOINT, endpoint]);
|
||||
|
||||
const res = await quickStart({
|
||||
caseName: 'cm-default-ctx',
|
||||
mode: 'claude',
|
||||
customModel: { endpointId: 'ep-default', modelId: 'some-unknown-model' },
|
||||
});
|
||||
|
||||
const body = res.json();
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.contextLength).toBe(16384);
|
||||
expect(ctx.sessions.size).toBe(1); // nothing was created
|
||||
});
|
||||
|
||||
it('a discovered per-model figure wins over the endpoint defaultContextLength', async () => {
|
||||
const endpoint: CustomModelHost = {
|
||||
id: 'ep-precedence',
|
||||
label: 'mixed box',
|
||||
baseUrl: 'http://192.168.1.54:8080',
|
||||
apiKey: 'k',
|
||||
modelContextLengths: { m: 131072 },
|
||||
defaultContextLength: 16384,
|
||||
};
|
||||
await writeCustomModelHosts(getDataDir(), [ENDPOINT, endpoint]);
|
||||
|
||||
const res = await quickStart({
|
||||
caseName: 'cm-precedence',
|
||||
mode: 'claude',
|
||||
customModel: { endpointId: 'ep-precedence', modelId: 'm' },
|
||||
});
|
||||
|
||||
expect(res.statusCode).toBe(200);
|
||||
expect(res.json().requiresContextWarning).toBeUndefined();
|
||||
expect(ctx.sessions.size).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('triggering the actual llama-swap load (not just watching for it)', () => {
|
||||
|
||||
@@ -9,7 +9,7 @@ import { createRouteTestHarness } from './_route-test-utils.js';
|
||||
import { createMockSession } from '../mocks/index.js';
|
||||
import { getDataDir } from '../../src/config/instance.js';
|
||||
import { writeCustomModelHosts, type CustomModelHost } from '../../src/custom-model-hosts.js';
|
||||
import { existsSync, readFileSync, statSync } from 'node:fs';
|
||||
import { existsSync, readFileSync, rmSync, statSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { webviewFetch } from '../../src/web/webview-egress.js';
|
||||
|
||||
@@ -203,6 +203,10 @@ describe('POST /api/sessions/:id/custom-model', () => {
|
||||
const { app, ctx } = await setup();
|
||||
const session = ctx.sessions.get('test-session-1')!;
|
||||
session.mode = 'pi';
|
||||
// Start from no dir at all (a sibling test's successful pi apply may have left one),
|
||||
// so the existsSync below says something about THIS request.
|
||||
const dir = join(getDataDir(), 'custom-model-configs', 'test-session-1');
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
@@ -214,8 +218,45 @@ describe('POST /api/sessions/:id/custom-model', () => {
|
||||
expect(res.json().errorCode).toBe('INVALID_INPUT');
|
||||
expect(session.setCustomModel).not.toHaveBeenCalled();
|
||||
expect(session.restartCli).not.toHaveBeenCalled();
|
||||
// The config dir written before the check is cleaned up again.
|
||||
expect(existsSync(join(getDataDir(), 'custom-model-configs', 'test-session-1'))).toBe(false);
|
||||
// The pattern check runs on the pure build BEFORE any write, so a refused id leaves
|
||||
// no directory behind at all, nothing to clean up.
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
});
|
||||
|
||||
it('codex: a refused forced launch id never touches the live config dir the session runs on as CODEX_HOME', async () => {
|
||||
const { app, ctx } = await setup();
|
||||
const session = ctx.sessions.get('test-session-1')!;
|
||||
session.mode = 'codex';
|
||||
|
||||
// First apply succeeds: the session now runs codex on qwen3:32b, whose config.toml
|
||||
// lives in the per-session dir the running CLI itself uses as CODEX_HOME.
|
||||
const ok = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/sessions/test-session-1/custom-model',
|
||||
payload: { endpointId: 'ep1', modelId: 'qwen3:32b' },
|
||||
});
|
||||
expect(ok.statusCode).toBe(200);
|
||||
const dir = join(getDataDir(), 'custom-model-configs', 'test-session-1');
|
||||
const written = join(dir, 'config.toml');
|
||||
expect(existsSync(written)).toBe(true);
|
||||
|
||||
// `@` is outside codex's `model-plain-colon` pattern. The pattern check now runs on
|
||||
// the pure build BEFORE any write, so the refusal must leave the dir — and the file
|
||||
// the live CLI reads — exactly as they were, rather than replacing and deleting the
|
||||
// CODEX_HOME of a session that stays on qwen3:32b.
|
||||
const before = readFileSync(written, 'utf8');
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/sessions/test-session-1/custom-model',
|
||||
payload: { endpointId: 'ep1', modelId: 'org/model@q4_k_m' },
|
||||
});
|
||||
|
||||
expect(res.json().success).toBe(false);
|
||||
expect(res.json().errorCode).toBe('INVALID_INPUT');
|
||||
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
|
||||
expect(session.restartCli).toHaveBeenCalledTimes(1);
|
||||
expect(existsSync(dir)).toBe(true);
|
||||
expect(readFileSync(written, 'utf8')).toBe(before);
|
||||
});
|
||||
|
||||
it('clear removes the previous config dir the session reports', async () => {
|
||||
@@ -449,6 +490,58 @@ describe('POST /api/sessions/:id/custom-model', () => {
|
||||
expect(session.restartCli).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('the endpoint defaultContextLength stands in when discovery recorded no per-model figure', async () => {
|
||||
const { app, ctx } = await setup();
|
||||
// A gateway that states nothing per model: only the user's default is known.
|
||||
const endpoint: CustomModelHost = {
|
||||
id: 'ep-default',
|
||||
label: 'silent gateway',
|
||||
baseUrl: 'http://192.168.1.53:8080',
|
||||
apiKey: 'k',
|
||||
defaultContextLength: 16384,
|
||||
};
|
||||
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, endpoint]);
|
||||
const session = ctx.sessions.get('test-session-1')!;
|
||||
session.mode = 'claude';
|
||||
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/sessions/test-session-1/custom-model',
|
||||
payload: { endpointId: 'ep-default', modelId: 'some-unknown-model' },
|
||||
});
|
||||
|
||||
const body = res.json();
|
||||
expect(body.requiresContextWarning).toBe(true);
|
||||
expect(body.contextLength).toBe(16384);
|
||||
expect(session.setCustomModel).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('a discovered per-model figure wins over the endpoint defaultContextLength', async () => {
|
||||
const { app, ctx } = await setup();
|
||||
// The default is small enough to warn on; the discovered figure for THIS model is
|
||||
// not, so the fallback must not leak into the per-model path.
|
||||
const endpoint: CustomModelHost = {
|
||||
id: 'ep-precedence',
|
||||
label: 'mixed box',
|
||||
baseUrl: 'http://192.168.1.54:8080',
|
||||
apiKey: 'k',
|
||||
modelContextLengths: { 'qwen3.8-27b-ud-q4_k_xl': 131072 },
|
||||
defaultContextLength: 16384,
|
||||
};
|
||||
await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, endpoint]);
|
||||
const session = ctx.sessions.get('test-session-1')!;
|
||||
session.mode = 'claude';
|
||||
|
||||
const res = await app.inject({
|
||||
method: 'POST',
|
||||
url: '/api/sessions/test-session-1/custom-model',
|
||||
payload: { endpointId: 'ep-precedence', modelId: 'qwen3.8-27b-ud-q4_k_xl' },
|
||||
});
|
||||
|
||||
expect(res.json().requiresContextWarning).toBeUndefined();
|
||||
expect(session.setCustomModel).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
// llama-server splits its context across --parallel slots, and Claude Code sends several
|
||||
// requests at once, so a context that clears the floor as a whole can still fail per slot.
|
||||
describe('parallel slots share the context', () => {
|
||||
|
||||
@@ -27,6 +27,8 @@ import { join } from 'node:path';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { Session } from '../src/session.js';
|
||||
import { getCli } from '../src/config/cli-registry/index.js';
|
||||
import { buildCustomModelInjection } from '../src/custom-model-injection.js';
|
||||
import { TmuxManager } from '../src/tmux-manager.js';
|
||||
import type { MuxSession, SessionMode } from '../src/types.js';
|
||||
|
||||
@@ -225,3 +227,29 @@ describe('launchModel reaches the CLI through its own launch param', () => {
|
||||
expect(respawn.mock.calls[0][0].model).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Codex custom endpoint model overrides', () => {
|
||||
it('selects the local model over a native launch default and restores the default on clear', async () => {
|
||||
const entry = getCli('codex');
|
||||
if (!entry) throw new Error('Missing Codex entry');
|
||||
const injection = buildCustomModelInjection(
|
||||
entry,
|
||||
{ id: 'ep', label: 'Local', baseUrl: 'http://box:8080' },
|
||||
'qwen3'
|
||||
);
|
||||
if (injection.kind !== 'configDir') throw new Error('Expected isolated Codex config');
|
||||
const { session, respawn } = liveSession('codex', {
|
||||
codexConfig: { model: 'native-model', reasoningEffort: 'high' },
|
||||
});
|
||||
session.setCustomModel(
|
||||
{ endpointId: 'ep', modelId: 'qwen3', envKeys: ['CODEX_HOME'], launchModel: injection.launchModel },
|
||||
{ CODEX_HOME: '/isolated/codex' }
|
||||
);
|
||||
await session.restartCli();
|
||||
expect(respawn.mock.calls[0][0].codexConfig).toEqual({ model: 'qwen3', reasoningEffort: 'high' });
|
||||
session.setCustomModel(undefined);
|
||||
respawn.mockClear();
|
||||
await session.restartCli();
|
||||
expect(respawn.mock.calls[0][0].codexConfig).toEqual({ model: 'native-model', reasoningEffort: 'high' });
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user