2026-09-15 12:58:23 +08:00
# Custom Model Endpoints
Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for
one session at a time. "Custom endpoint" covers **local** hardware (llama.cpp, Ollama, vLLM,
a home GPU rig, DGX Spark, Strix Halo) and **cloud** services (Azure AI Foundry's
OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering
`GET /v1/models` and `POST /v1/chat/completions` in the standard shape.
**Off by default.** Turn it on in App Settings → Models → **Custom model endpoints** .
## Adding an endpoint
Still in App Settings → Models → Custom model endpoints:
1. ** + Add endpoint** — give it an id, a label, and the base URL (`http://192.168.1.50:8080` ,
say). An API key is optional; most local servers don't check one.
2. **Discover** — fetches the endpoint's own model list over `GET /v1/models` and stores it.
3. Pick a **default model** from what was discovered. This is the model the Run-menu entry
2026-09-16 08:50:18 +08:00
applies directly when only one model is discovered; with two or more, it's just the one
pre-marked in the picker dialog described below, not a silent default.
2026-09-15 12:58:23 +08:00
Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker
hosts — these are machine-level infra, not a per-user setting.
2026-09-16 08:50:18 +08:00
**Model lists refresh themselves.** Every saved endpoint is re-discovered automatically every
5 minutes in the background, so a model the server starts serving later — or stops serving —
shows up without another manual click of **Discover** . One endpoint being unreachable on a
given cycle (powered off, wrong network) never blocks the others from refreshing.
2026-11-02 15:14:12 +08:00
**Context length is picked up automatically where it can be, safely.** Every model's own
listing entry is read for a stated context figure (`context_length` , then `context_window` ,
which current llama.cpp, llama-swap and gateway-style servers state to varying degrees);
against a llama-swap server the stronger figures from its launch command and from llama.cpp's
`/props` probe win where they exist, and those probes only ever run for a model the server
already reports loaded, since asking about an unloaded model can trigger an actual, slow model
swap as a side effect. A model that states no figure and was never loaded falls back to the
endpoint's **Default context length** field (that number is the model's whole window, never a
per-slot figure), and a model neither covers keeps whatever an earlier cycle learned or simply
gets no override, never a guess. The window is applied to the launched session by the harnesses
that can take it: Claude Code today (see below), OpenCode as the model's context limit, and
Codex as a `model_context_window` entry in its generated config file, which also sets Codex's
own auto-compact point via `model_auto_compact_token_limit` at 75% of the window.
2026-09-16 12:08:26 +08:00
2026-09-15 12:58:23 +08:00
## Running a session against one
2026-09-16 08:50:18 +08:00
With the setting on and at least one endpoint carrying a discovered model, the **Run**
2026-09-15 12:58:23 +08:00
dropdown grows a **Custom Endpoints** section: one entry per harness that can redirect to a
custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a
2026-09-16 08:50:18 +08:00
session on that harness exactly the way its own entry would. It is a one-off "try this
endpoint" action, not a sticky mode — the plain **Run** button still means "this harness,
native cloud" afterward, and a fresh session never inherits whatever the last one was
pointed at.
**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
the session launches straight away on that model — nothing to choose. With two or more, a
2026-11-02 15:14:12 +08:00
small dialog asks which one to use for this launch before starting the session; a search box
at the top narrows a long list with a case-insensitive substring match, so a server hosting
hundreds of models stays usable. The
2026-09-16 08:50:18 +08:00
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
2026-09-21 04:29:29 +02:00
use a different one without changing the saved default. The list is not raw discovery order
either: the model llama-swap reports loaded and ready is moved to the top and tagged
**Currently loaded** , and when nothing is loaded, the model you last launched on this harness
and endpoint pair is moved up instead and tagged **Last used** (a per-device browser value, so
another device starts from its own history). The default model keeps its own **Default** pill
in both cases, and nothing is ever auto-chosen: the promoted row is simply the one under your
thumb.
2026-09-15 12:58:23 +08:00
2026-10-11 12:50:14 +08:00
**For opencode, Codex, Gemini, Pi, Grok, DeepSeek, OMP and GitHub Copilot, picking an entry launches
2026-09-16 15:04:10 +08:00
straight onto the endpoint** — no restart, because the endpoint is applied before the
session's process ever starts. **Claude still restarts the harness's process in place** —
same tab, same conversation (`--resume` ) — after a normal native launch, since that restart
2026-10-11 12:50:14 +08:00
is far less jarring for Claude than for the others, whose own TUI can fully
2026-09-16 15:04:10 +08:00
reinitialize on a restart. Either way, every supported harness reads its endpoint config at
process start, never per turn, so there is no live hot-swap while a turn is running.
2026-09-15 12:58:23 +08:00
2026-09-16 15:04:10 +08:00
Picking an entry that launches a **brand-new** Claude session waits (up to 20 seconds) for it to
2026-09-16 10:38:37 +08:00
finish its own startup before applying — a freshly started CLI reports itself as busy for its
boot sequence, and applying to a genuinely busy session is refused so a real, in-progress
turn is never interrupted out from under you. A session that is still busy after that wait
(a very slow-starting CLI, or one you started typing into right away) surfaces that refusal
as an ordinary error, which now stays on screen with a close button instead of vanishing
after a few seconds — read it, it names the actual reason rather than a generic failure.
2026-09-15 12:58:23 +08:00
Entries are hidden entirely for a session in a **remote (SSH) or Docker case** — support for
2026-09-16 07:00:55 +08:00
redirecting those hasn't landed yet, see below. The picker also only appears in the desktop
**Run** dropdown; the phone home screen builds its own run picker separately and does not
currently offer these entries.
2026-09-15 12:58:23 +08:00
2026-09-16 15:53:34 +08:00
**Against llama-swap, applying a selection also starts the actual model load, rather than
waiting on your first prompt to do it.** llama-swap has no "switch model" button of its own
— the only thing that starts a swap is a real request naming the model, and confirmed live:
just applying a selection never reached llama-swap's own logs at all until something asked
it to load. Picking an entry now also sends the smallest real request that will trigger
2026-09-16 20:22:02 +08:00
that load, in the background, the moment the target model isn't already loaded and ready.
2026-09-17 13:00:50 +08:00
**The centred loading banner has no countdown and no automatic timeout — it waits as long as
it takes, and tells you so.** When it knows the model's discovered file size (its GB figure,
when llama-swap states one) it's shown too, e.g. "Loading qwen3.8-27b (16.4 GB) on
llama-swap — this can take a while depending on your hardware and the model size." An
earlier version tried to estimate and enforce a time limit, but real load time depends on
hardware this feature has no way to know, so a fixed number was always a guess — worse, one
that could kill a genuinely slow load partway through. If it really is taking too long, a
**Cancel** button right on the banner ends the wait and **closes the session that load was
for**, on your own call rather than a guessed deadline.
2026-09-16 15:53:34 +08:00
2026-09-17 12:22:45 +08:00
**The banner also shows a real, live second line of what llama.cpp itself is doing** — not
a made-up progress phase, the actual next line the `llama-server` process printed, e.g.
"llama.cpp: load_model: loading model '/models/.../Qwen3.8-27B.gguf'" then later
"llama.cpp: llama_server: model loaded". It comes straight from llama-swap's own event
feed, filtered down to just the backend process's own output (not llama-swap's own request
logging), and stays on whatever it last said once the load goes quiet, rather than
clearing back to nothing.
2026-09-17 11:09:24 +08:00
**You'll also be told if a session's model gets swapped out from under it later, not just
at launch.** The conflict warning above only fires at the moment you launch or apply a
model — llama.cpp only runs one model at a time, so if a DIFFERENT session using the same
endpoint later triggers its own load, whatever was loaded before (including a session you
already had running) gets silently evicted, with no warning at that instant since nothing
conflicted when it was first set up. A background check (every 20 seconds) catches this
after the fact and shows a toast naming which session lost its model and what's loaded now
— so you know before typing into that session that it's about to reload (and, in turn,
evict whatever displaced it).
2026-09-17 09:21:36 +08:00
**Claude Code specifically gets three extra fixes applied automatically:**
2026-09-16 12:08:26 +08:00
- Its discovered context length (see above) is passed through as
`CLAUDE_CODE_MAX_CONTEXT_TOKENS` , so it doesn't send a full-size prompt against a much
smaller real local context and overflow it.
- Its session runs with an isolated `CLAUDE_CONFIG_DIR` , so the injected API key never sits
in the same directory as a stored claude.ai login — that combination is harmless for actual
requests (the API key wins) but the CLI still prints a "both claude.ai and
ANTHROPIC_API_KEY set" warning about it, which this avoids entirely. The isolated directory
keeps a link back to your real session history so the response viewer and similar features
2026-09-16 13:08:07 +08:00
still work for that session. That isolated directory starts with no prior approvals of its
own, so Codeman also pre-approves the injected key the same way answering Claude Code's own
"Detected a custom API key" prompt once would — without it, that prompt would otherwise
reappear on every single launch with nobody there to answer it.
2026-09-17 09:21:36 +08:00
- **That same fresh isolated directory also looks like a brand-new Claude Code profile**, so
without this fix it replayed the WHOLE first-run sequence every single launch: the theme
picker, the security-notes screen, the "trust this folder?" dialog, and a one-time warning
about running with permissions bypassed — none of which a real, already-used profile shows
again. Codeman now pre-seeds that same "already been through this once" state (onboarding
completed, this session's own project marked trusted, the bypass-permissions warning
acknowledged) so a custom-model launch reaches the actual conversation exactly as fast as a
native cloud one does, instead of stopping at a wizard with nobody there to click through it.
2026-09-16 12:08:26 +08:00
2026-09-17 07:36:57 +08:00
**If a model's real context is too small for Claude Code to even get started, you get a
warning instead of a confusing failure.** Claude Code's own system prompt and tools take up
roughly 40K tokens on their own, before you've typed anything — a small local model with a
smaller real context than that fails outright on the very first message, no matter what
context size Codeman tells it to expect (raising the declared context only changes when
Claude Code trims _conversation history_ , and there is none yet on message one). Picking
such a model now shows an in-app dialog naming the model, its discovered context and what's
needed, before anything launches or restarts, with the fix spelled out: reconfigure
llama-swap to give that model (or a smaller one) an explicit larger context instead of
relying on auto-fit (`--fit-ctx` ), which sizes the context around fitting the biggest model
rather than the biggest context — for example adding `-c 65536` to that model's llama-swap
entry. "Launch anyway" is still there if you want to try regardless.
2026-10-22 19:32:24 +02:00
**A context that is big enough overall can still be too small once it is split across
slots.** `llama-server` runs several parallel slots by default (four on the build this was measured on) and,
unless you say otherwise, they all share one pool of context. Claude Code sends more than one
request at a time: the main session plus every subagent it runs in parallel, and each of those
requests carries the roughly 40K tokens of system prompt and tools on its own. Four requests
of 40K tokens do not fit in a shared 131K pool, so the server refuses them with HTTP 500
`Context size has been exceeded` , even though any single request would have been fine.
Discovery therefore also records how many slots each loaded model has. It reads an explicit
`--parallel` or `-np` from the launch command, and otherwise asks the server for
`total_slots` , because a server started without the flag chooses the number itself. When the
context divided by the slot count drops below what Claude Code needs, picking that model shows
the same in-app dialog, this time explaining the shared slots. The division uses the context
Codeman read from the launch command (`--fit-ctx` or `-c` ), never a larger figure from the
server's `/props` , which has been measured reporting far more than the server really runs
with. So a default llama-swap model with auto slots shows the dialog, which names the fix and
lets you launch anyway. The fix is a single slot:
```yaml
cmd : >-
llama-server -m /models/your-model.gguf --parallel 1 ...
```
With one slot the active conversation gets the whole context, requests from parallel
subagents queue behind it instead of failing, and Claude Code's repeated prompt prefix stays
in the cache instead of being processed again in a different slot each time. The alternative
is a context large enough that every slot can hold 40K tokens. A slot count Codeman could not
read is treated as one slot, never guessed.
2026-09-15 12:58:23 +08:00
## Which harnesses actually work
2026-09-17 13:41:19 +08:00
| Harness | Status |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
2026-10-11 12:50:14 +08:00
| **GitHub Copilot CLI** | Verified at the CLI level (1.0.95) against a local OpenAI-compatible endpoint: it posts to `/v1/chat/completions` with your key and model name and needs no GitHub sign-in. Tool calls through a custom endpoint were not exercised. |
2026-09-17 13:41:19 +08:00
| **Claude Code, opencode, Pi, Grok, OMP** | Verified end-to-end against a real local server. |
| **Codex** | Config is correct, and plain chat can work against a server that speaks the Responses API — but a real tool-call attempt comes back as inert text instead of running, so it's still not usable for real coding work. |
| **Gemini** | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet. |
| **DeepSeek** | The original 404 is root-caused and fixed (DeepSeek Harness's own code was missing a `/v1` most local servers require) — not yet re-run against a real `dsh` install to confirm end-to-end. |
| **Antigravity** | No known custom-endpoint mechanism at all. Not offered. |
2026-09-15 12:58:23 +08:00
Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry,
not a fixed list here, so this table can go stale before this page does — a greyed-out or
missing entry is the more current answer.
## What it does not do
- **No remote or Docker sessions yet.** Both restart their agent differently under the hood
(reattaching a durable tmux session rather than relaunching the process), so redirecting
them needs its own plumbing that hasn't been built.
- **No live hot-swap mid-conversation.** Applying a selection always restarts the process.
2026-09-16 07:00:55 +08:00
- **No button to un-point a session from the UI yet.** Clearing back to native cloud is an
HTTP call (`POST .../custom-model {"clear": true}` ) or deleting the session; the settings
panel manages saved endpoints, not what a running session is currently pointed at.
2026-09-15 12:58:23 +08:00
- **Nothing is shared with your real cloud credentials.** The endpoint's own key, if any,
never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate,
explicit choice per session.
## Security
An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save
time and against the address it actually resolves to), the same guard Web Tabs uses for
saved dashboards. Endpoint records and any per-session config files a harness needs are
written with owner-only permissions. See
[custom-model-endpoints-plan.md ](https://github.com/Ark0N/Codeman/blob/master/docs/custom-model-endpoints-plan.md )
in the repository for the full design reasoning, including why this feature closed a
pre-existing gap in how session environment overrides were guarded rather than opening a new
one.
