mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 02:29:33 +01:00
fix(custom-model): core review fixes for context-length discovery and injection
- share one context-length resolution (discovered figure, endpoint default fallback) across the routes and boot recovery, with a recovery test - record listed per-slot context figures with a slot count of 1 and force the slot-count fallback once per rediscovery - open up the codex model launch token for qwen3-style ids (model-plain-colon), pinned by a spawn-golden case - pass the full discovered window to opencode as limit.context - translate the picker search strings and document the discovery precedence
This commit is contained in:
1 parent
dbd1b6e587
commit
1b8ac40baf
16 files changed
+256
-47
No files matched your search
+22
-5
@@ -1205,13 +1205,18 @@ data}` envelope on the wire — unwrap it the same way). Answers `[]` for a
|
||||
boolean` reports whether one is stored, so a client can render "unchanged
|
||||
if left blank" without ever holding the real value.
|
||||
- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?,
|
||||
authStyle?, defaultModelId? }` creates one. `id` must match
|
||||
authStyle?, defaultModelId?, defaultContextLength? }` creates one. `id`
|
||||
must match
|
||||
`^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never
|
||||
both (a real server hung indefinitely when sent both headers on one
|
||||
request); `baseUrl` must be `http(s)`, carry no embedded credentials, and
|
||||
request); `defaultContextLength` is a user-set fallback used when
|
||||
assigning a model that discovery has no context length for (discovery
|
||||
itself never writes it); `baseUrl` must be `http(s)`, carry no embedded
|
||||
credentials, and
|
||||
is refused if it points at (or resolves to) a link-local or
|
||||
cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id.
|
||||
- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey`
|
||||
- `PUT /api/v1/model-endpoints/:id` updates one, also accepting
|
||||
`defaultContextLength`. An **absent** `apiKey`
|
||||
keeps the stored one rather than clearing it — the client never receives
|
||||
the real value to resend deliberately unchanged, so omission is the only
|
||||
way to say "leave it alone"; there is no way to clear a key back to unset
|
||||
@@ -1221,8 +1226,20 @@ authStyle?, defaultModelId? }` creates one. `id` must match
|
||||
- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's
|
||||
own `GET /v1/models` and stores the result as `models`, updating
|
||||
`lastDiscoveredAt`, plus (best-effort, only for a model llama-swap's own
|
||||
response already reports loaded) `modelContextLengths` and `modelSizesGB`.
|
||||
A `defaultModelId` that no longer appears in the fresh list is dropped
|
||||
response already reports loaded) `modelContextLengths`,
|
||||
`modelSlotCounts` and `modelSizesGB`. A loaded model's context figure
|
||||
comes from llama-swap's own launch command (`--fit-ctx` / `-c` /
|
||||
`--ctx-size`), falling back to `/props` (`n_ctx`, `total_slots`) for
|
||||
whatever the command did not answer; the model's own listing
|
||||
(`context_length`, then `context_window`) only ever fills the gap when
|
||||
neither probe does, and is recorded as a single-slot figure —
|
||||
llama-swap lists the PER-SLOT window, so whenever the listed figure is
|
||||
the one taken the stored slot count is 1. A server with no `status`
|
||||
field on any entry (plain llama.cpp, a non-llama-swap gateway) gets no
|
||||
context/size enrichment at all. `modelContextLengths` falls back to the
|
||||
endpoint's `defaultContextLength` when a session is assigned a model
|
||||
discovery never learned a window for. A `defaultModelId` that no longer
|
||||
appears in the fresh list is dropped
|
||||
rather than carried forward invalid. Failures answer `422 OPERATION_FAILED`
|
||||
with the underlying connection error, or a named egress refusal if the
|
||||
resolved address turned out to be blocked. The same refresh also runs
|
||||
|
||||
@@ -628,7 +628,7 @@ Adding the header changed the preamble, so `CODEMAN_PREAMBLE` was bumped (1.22.0
|
||||
|
||||
**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET <baseUrl>/v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×.
|
||||
|
||||
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
|
||||
⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/copilot/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.<name>]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified.
|
||||
|
||||
⚠️ **Two launch paths, chosen by mechanism, not preference**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` relaunches a CLI in an existing pane, so a CLI whose launch declares a `fallback` chain (claude) gets a conversation pinned as `resumeSessionId` for that one respawn: `--session-id <id>` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume <id> || --session-id <id>` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. The dead-pane respawn shares this pin, see [Dead-pane respawn: the resume pin](#dead-pane-respawn-the-resume-pin). ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/<id>` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600.
|
||||
|
||||
|
||||
@@ -67,15 +67,23 @@ configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one.
|
||||
Endpoint management is admin-only in multi-user mode, same as remote/docker
|
||||
hosts — these are machine-level infra, not per-user settings.
|
||||
|
||||
**Context length is discovered too, opportunistically and safely.** The plain
|
||||
`GET /v1/models` response has no context-window field. Discovery only ever
|
||||
looks for one for a model llama-swap's own response already reports
|
||||
**Context length is discovered too, opportunistically and safely.** Older
|
||||
servers state no context-window field in the plain `GET /v1/models`
|
||||
response at all; current llama-swap states one per model (both
|
||||
`context_length` and `context_window`) and gateways like Bifrost pass
|
||||
`context_window` through — that listed figure is the *lowest*-precedence
|
||||
source, used only when neither probe below answers, and because llama-swap
|
||||
documents it as the PER-SLOT window, a model whose figure comes from the
|
||||
listing is recorded with a slot count of 1 so the slot check never divides a
|
||||
per-slot figure a second time. The probes are only ever looked for on a model
|
||||
llama-swap's own response already reports
|
||||
`status.value === "loaded"` for — never for an unloaded one, because
|
||||
llama-swap treats `?model=` as a routing hint and asking about a model that
|
||||
isn't loaded risks triggering an actual (slow, GPU-swapping) load as a side
|
||||
effect of what should be read-only discovery. A server with no `status` field
|
||||
on any entry at all (not llama-swap) gets no context-length enrichment,
|
||||
rather than guessing. A model's previously-learned context length survives a
|
||||
on any entry at all (not llama-swap) gets no launch-command/`/props`
|
||||
enrichment, only whatever its listings state — rather than guessing. A
|
||||
model's previously-learned context length survives a
|
||||
later cycle where it wasn't the loaded one; it's dropped only once the model
|
||||
disappears from the endpoint's list entirely. Stored per model in
|
||||
`modelContextLengths` and applied automatically (see "Applying a model to a
|
||||
@@ -293,14 +301,16 @@ agent, so the selection would report success and change nothing.
|
||||
**Claude gets two more env vars when known/applicable, both declared on its
|
||||
registry entry (`contextLengthVar`/`configDirVar`), not hardcoded here:**
|
||||
|
||||
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s discovered context
|
||||
length (see the discovery section above) whenever one is known. Without
|
||||
- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s context length —
|
||||
the discovered figure (see the discovery section above), falling back to
|
||||
the endpoint's user-set `defaultContextLength` — whenever one is known.
|
||||
Without
|
||||
it, Claude Code assumes a large (200k) window for any unrecognized custom
|
||||
model id and never compacts, which reliably overflows a much smaller real
|
||||
local context — confirmed live: a stock ~33.7K-token system prompt against
|
||||
a 16384-token llama-swap model failed with `exceeds the available context
|
||||
size`. No entry for the model in `modelContextLengths` means the var is
|
||||
simply omitted, never a guess. ⚠️ **This var only affects when Claude
|
||||
size`. Neither a `modelContextLengths` entry nor a `defaultContextLength`
|
||||
for the model means the var is simply omitted, never a guess. ⚠️ **This var only affects when Claude
|
||||
Code compacts conversation _history_ — it cannot fix a model whose real
|
||||
context is smaller than Claude Code's own fixed per-turn overhead**
|
||||
(system prompt + tool schemas, empirically ~36.4K tokens, confirmed live
|
||||
|
||||
Reference in new issue
Block a user