Codeman/docs/wiki/Custom-Model-Endpoints.md

# Custom Model Endpoints

Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for
one session at a time. "Custom endpoint" covers **local** hardware (llama.cpp, Ollama, vLLM,
a home GPU rig, DGX Spark, Strix Halo) and **cloud** services (Azure AI Foundry's
OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering
`GET /v1/models` and `POST /v1/chat/completions` in the standard shape.

**Off by default.** Turn it on in App Settings → Models → **Custom model endpoints**.

## Adding an endpoint

Still in App Settings → Models → Custom model endpoints:

1. **+ Add endpoint** — give it an id, a label, and the base URL (`http://192.168.1.50:8080`,
   say). An API key is optional; most local servers don't check one.
2. **Discover** — fetches the endpoint's own model list over `GET /v1/models` and stores it.
3. Pick a **default model** from what was discovered. This is the model the Run-menu entry
   applies directly when only one model is discovered; with two or more, it's just the one
   pre-marked in the picker dialog described below, not a silent default.

Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker
hosts — these are machine-level infra, not a per-user setting.

**Model lists refresh themselves.** Every saved endpoint is re-discovered automatically every
5 minutes in the background, so a model the server starts serving later — or stops serving —
shows up without another manual click of **Discover**. One endpoint being unreachable on a
given cycle (powered off, wrong network) never blocks the others from refreshing.

**Context length is picked up automatically where it can be, safely.** Every model's own
listing entry is read for a stated context figure (`context_length`, then `context_window`,
which current llama.cpp, llama-swap and gateway-style servers state to varying degrees);
against a llama-swap server the stronger figures from its launch command and from llama.cpp's
`/props` probe win where they exist, and those probes only ever run for a model the server
already reports loaded, since asking about an unloaded model can trigger an actual, slow model
swap as a side effect. A model that states no figure keeps whatever an earlier cycle learned:
the stored figure always beats the endpoint's **Default context length** field
(`modelContextLengths[id] ?? defaultContextLength`), which only applies where no figure was
ever learned (that number is the model's whole window, never a per-slot figure). A model
neither covers simply gets no override, never a guess. The window is applied to the launched session by the harnesses
that can take it: Claude Code today (see below), OpenCode as the model's context limit, and
Codex as a `model_context_window` entry in its generated config file, which also sets Codex's
own auto-compact point via `model_auto_compact_token_limit` at 75% of the window.

## Running a session against one

With the setting on and at least one endpoint carrying a discovered model, the **Run**
dropdown grows a **Custom Endpoints** section: one entry per harness that can redirect to a
custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a
session on that harness exactly the way its own entry would. It is a one-off "try this
endpoint" action, not a sticky mode — the plain **Run** button still means "this harness,
native cloud" afterward, and a fresh session never inherits whatever the last one was
pointed at.

**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
the session launches straight away on that model — nothing to choose. With two or more, a
small dialog asks which one to use for this launch before starting the session; a search box
at the top narrows a long list with a case-insensitive substring match, so a server hosting
hundreds of models stays usable. The
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
use a different one without changing the saved default. The list is not raw discovery order
either: the model llama-swap reports loaded and ready is moved to the top and tagged
**Currently loaded**. When nothing is reported loaded, the model one of your own running
sessions is using on that endpoint is moved up instead and tagged **In use by a session** (it
beats **Last used**, never **Currently loaded**), and failing that the model you last launched
on this harness and endpoint pair is moved up and tagged **Last used** (a per-device browser
value, so another device starts from its own history). With none of those, a line under the
search box says nothing is marked, so the plain list is not mistaken for an answer. The default
model keeps its own **Default** pill in every case, and nothing is ever auto-chosen: the
promoted row is simply the one under your thumb.

**For opencode, Codex, Gemini, Pi, Grok, DeepSeek, OMP and GitHub Copilot, picking an entry launches
straight onto the endpoint** — no restart, because the endpoint is applied before the
session's process ever starts. **Claude still restarts the harness's process in place** —
same tab, same conversation (`--resume`) — after a normal native launch, since that restart
is far less jarring for Claude than for the others, whose own TUI can fully
reinitialize on a restart. Either way, every supported harness reads its endpoint config at
process start, never per turn, so there is no live hot-swap while a turn is running.

Picking an entry that launches a **brand-new** Claude session waits (up to 20 seconds) for it to
finish its own startup before applying — a freshly started CLI reports itself as busy for its
boot sequence, and applying to a genuinely busy session is refused so a real, in-progress
turn is never interrupted out from under you. A session that is still busy after that wait
(a very slow-starting CLI, or one you started typing into right away) surfaces that refusal
as an ordinary error, which now stays on screen with a close button instead of vanishing
after a few seconds — read it, it names the actual reason rather than a generic failure.

Entries are hidden entirely for a session in a **remote (SSH) or Docker case** — support for
redirecting those hasn't landed yet, see below. The picker also only appears in the desktop
**Run** dropdown; the phone home screen builds its own run picker separately and does not
currently offer these entries.

**Against llama-swap, applying a selection also starts the actual model load, rather than
waiting on your first prompt to do it.** llama-swap has no "switch model" button of its own
— the only thing that starts a swap is a real request naming the model, and confirmed live:
just applying a selection never reached llama-swap's own logs at all until something asked
it to load. Picking an entry now also sends the smallest real request that will trigger
that load, in the background, the moment the target model isn't already loaded and ready.

**The centred loading banner has no countdown and no automatic timeout — it waits as long as
it takes, and tells you so.** When it knows the model's discovered file size (its GB figure,
when llama-swap states one) it's shown too, e.g. "Loading qwen3.8-27b (16.4 GB) on
llama-swap — this can take a while depending on your hardware and the model size." An
earlier version tried to estimate and enforce a time limit, but real load time depends on
hardware this feature has no way to know, so a fixed number was always a guess — worse, one
that could kill a genuinely slow load partway through. If it really is taking too long, a
**Cancel** button right on the banner ends the wait and **closes the session that load was
for**, on your own call rather than a guessed deadline.

**The banner also shows a real, live second line of what llama.cpp itself is doing** — not
a made-up progress phase, the actual next line the `llama-server` process printed, e.g.
"llama.cpp: load_model: loading model '/models/.../Qwen3.8-27B.gguf'" then later
"llama.cpp: llama_server: model loaded". It comes straight from llama-swap's own event
feed, filtered down to just the backend process's own output (not llama-swap's own request
logging), and stays on whatever it last said once the load goes quiet, rather than
clearing back to nothing.

**You'll also be told if a session's model gets swapped out from under it later, not just
at launch.** The conflict warning above only fires at the moment you launch or apply a
model — llama.cpp only runs one model at a time, so if a DIFFERENT session using the same
endpoint later triggers its own load, whatever was loaded before (including a session you
already had running) gets silently evicted, with no warning at that instant since nothing
conflicted when it was first set up. A background check (every 20 seconds) catches this
after the fact and shows a toast naming which session lost its model and what's loaded now
— so you know before typing into that session that it's about to reload (and, in turn,
evict whatever displaced it).

**Claude Code specifically gets three extra fixes applied automatically:**

- Its discovered context length (see above) is passed through as
  `CLAUDE_CODE_MAX_CONTEXT_TOKENS`, so it doesn't send a full-size prompt against a much
  smaller real local context and overflow it.
- Its session runs with an isolated `CLAUDE_CONFIG_DIR`, so the injected API key never sits
  in the same directory as a stored claude.ai login — that combination is harmless for actual
  requests (the API key wins) but the CLI still prints a "both claude.ai and
  ANTHROPIC_API_KEY set" warning about it, which this avoids entirely. The isolated directory
  keeps a link back to your real session history so the response viewer and similar features
  still work for that session. That isolated directory starts with no prior approvals of its
  own, so Codeman also pre-approves the injected key the same way answering Claude Code's own
  "Detected a custom API key" prompt once would — without it, that prompt would otherwise
  reappear on every single launch with nobody there to answer it.
- **That same fresh isolated directory also looks like a brand-new Claude Code profile**, so
  without this fix it replayed the WHOLE first-run sequence every single launch: the theme
  picker, the security-notes screen, the "trust this folder?" dialog, and a one-time warning
  about running with permissions bypassed — none of which a real, already-used profile shows
  again. Codeman now pre-seeds that same "already been through this once" state (onboarding
  completed, this session's own project marked trusted, the bypass-permissions warning
  acknowledged) so a custom-model launch reaches the actual conversation exactly as fast as a
  native cloud one does, instead of stopping at a wizard with nobody there to click through it.

**If a model's real context is too small for Claude Code to even get started, you get a
warning instead of a confusing failure.** Claude Code's own system prompt and tools take up
roughly 40K tokens on their own, before you've typed anything — a small local model with a
smaller real context than that fails outright on the very first message, no matter what
context size Codeman tells it to expect (raising the declared context only changes when
Claude Code trims _conversation history_, and there is none yet on message one). Picking
such a model now shows an in-app dialog naming the model, its discovered context and what's
needed, before anything launches or restarts, with the fix spelled out: reconfigure
llama-swap to give that model (or a smaller one) an explicit larger context instead of
relying on auto-fit (`--fit-ctx`), which sizes the context around fitting the biggest model
rather than the biggest context — for example adding `-c 65536` to that model's llama-swap
entry. "Launch anyway" is still there if you want to try regardless.

**A context that is big enough overall can still be too small once it is split across
slots.** `llama-server` runs several parallel slots by default (four on the build this was measured on) and,
unless you say otherwise, they all share one pool of context. Claude Code sends more than one
request at a time: the main session plus every subagent it runs in parallel, and each of those
requests carries the roughly 40K tokens of system prompt and tools on its own. Four requests
of 40K tokens do not fit in a shared 131K pool, so the server refuses them with HTTP 500
`Context size has been exceeded`, even though any single request would have been fine.
Discovery therefore also records how many slots each loaded model has. It reads an explicit
`--parallel` or `-np` from the launch command, and otherwise asks the server for
`total_slots`, because a server started without the flag chooses the number itself. When the
context divided by the slot count drops below what Claude Code needs, picking that model shows
the same in-app dialog, this time explaining the shared slots. The division uses the context
Codeman read from the launch command (`--fit-ctx` or `-c`), never a larger figure from the
server's `/props`, which has been measured reporting far more than the server really runs
with. So a default llama-swap model with auto slots shows the dialog, which names the fix and
lets you launch anyway. The fix is a single slot:

```yaml
cmd: >-
  llama-server -m /models/your-model.gguf --parallel 1 ...
```

With one slot the active conversation gets the whole context, requests from parallel
subagents queue behind it instead of failing, and Claude Code's repeated prompt prefix stays
in the cache instead of being processed again in a different slot each time. The alternative
is a context large enough that every slot can hold 40K tokens. A slot count Codeman could not
read is treated as one slot, never guessed.

## Which harnesses actually work

| Harness                                  | Status                                                                                                                                                                                                               |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **GitHub Copilot CLI**                   | Verified at the CLI level (1.0.95) against a local OpenAI-compatible endpoint: it posts to `/v1/chat/completions` with your key and model name and needs no GitHub sign-in. Tool calls through a custom endpoint were not exercised. |
| **Claude Code, opencode, Pi, Grok, OMP** | Verified end-to-end against a real local server.                                                                                                                                                                     |
| **Codex**                                | Config is correct, and plain chat can work against a server that speaks the Responses API — but a real tool-call attempt comes back as inert text instead of running, so it's still not usable for real coding work. |
| **Gemini**                               | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet.                                                                                                                        |
| **DeepSeek**                             | Verified end-to-end against a real `dsh` install (dsh 0.1.1-rc.2, dsh-TUI 0.10.0-beta.1): the original 404 is root-caused and fixed, and the redirect is now a generated per-session patch file whose prompt reaches the endpoint as `POST /v1/chat/completions`. Such a session does not carry over your home-level `cordis.patch.yml`, your own `llm-pi-ai` providers, or a profile-level `effort`/`preset`. |
| **Antigravity**                          | No known custom-endpoint mechanism at all. Not offered.                                                                                                                                                              |

Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry,
not a fixed list here, so this table can go stale before this page does — a greyed-out or
missing entry is the more current answer.

## A model that runs on this Mac

If the Codeman server runs on a Mac with Apple silicon, you do not need a server of your
own: Codeman can set one model up for you on that Mac. This is experimental.

1. Press **Run a local model** on the welcome screen (or open App Settings > Models > Model
   on this Mac).
2. The window shows what fits this Mac. A bar shows how much of the memory the selected
   model takes at its full context; a nearly full bar means a tight fit. Pick a model and
   the coding agent to run it with.
3. Press **Install on this Mac**, check the sizes and where they go, and confirm. Nothing is
   downloaded before this click.
4. Codeman downloads a model runtime (Ollama, about 186 MB) and the model into
   `~/.codeman/local-model`, loads the model, checks that all of it runs on the GPU and that
   it can call a tool, installs the coding agent if it is missing, and saves an endpoint
   named **This Mac**. No administrator password is asked for.
5. Press **Start a session**.

Good to know:

- You can close the window: the download continues while Codeman is running. **Cancel**
  pauses it and **Continue** picks it up where it stopped. After a Codeman restart an
  unfinished install waits for your click.
- The memory figure is an estimate until the runtime has started once. From then on it is
  what the runtime itself measured, and the model is judged against that before it is
  downloaded.
- The endpoint "This Mac" belongs to this install. It is in the list and in the Run menu,
  but it is changed and removed in the local model window, not here.
- Codeman starts the runtime with the server and stops it with it. **Stop** keeps it from
  starting next time.
- More than one model can be on the Mac, and one of them runs at a time. Once a model is
  installed the window lists the models with what each takes on disk: **Add a model**
  downloads another one, **Use** switches to a model that is already there (nothing is
  downloaded; close the sessions on the model first), and **Delete** removes one, after one
  question. The model in use cannot be deleted: switch first, or use **Remove**.
- **Remove** deletes the runtime and the endpoint, and the model too if you tick that. The
  coding agent stays installed.
- If the model on disk is no longer the one that was installed, Codeman refuses to start a
  session on it and tells you to remove and install it again.
- Any program on this Mac, and any web page served from `localhost`, can reach the model
  runtime: its port has no login. It listens on this Mac only.
- On a Mac without Apple's developer tools, macOS will offer to install them the first time
  a coding agent runs `git` or Python. The window tells you when that applies.

## What it does not do

- **No remote or Docker sessions yet.** Both restart their agent differently under the hood
  (reattaching a durable tmux session rather than relaunching the process), so redirecting
  them needs its own plumbing that hasn't been built.
- **No live hot-swap mid-conversation.** Applying a selection always restarts the process.
- **No button to un-point a session from the UI yet.** Clearing back to native cloud is an
  HTTP call (`POST .../custom-model {"clear": true}`) or deleting the session; the settings
  panel manages saved endpoints, not what a running session is currently pointed at.
  A clear, or a switch to another endpoint, gives the session back what it had set itself under
  a key the endpoint replaced, such as its own `DSH_HOME` or `CLAUDE_CONFIG_DIR`. If Codeman
  restarted in between, only the home directory comes back, and an own `DSH_HOME`, `CODEX_HOME`
  or `GROK_HOME` only when the endpoint was picked before that restart.
- **Nothing is shared with your real cloud credentials.** The endpoint's own key, if any,
  never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate,
  explicit choice per session.

## Security

An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save
time and against the address it actually resolves to), the same guard Web Tabs uses for
saved dashboards. Endpoint records and any per-session config files a harness needs are
written with owner-only permissions. See
[custom-model-endpoints-plan.md](https://github.com/Ark0N/Codeman/blob/master/docs/custom-model-endpoints-plan.md)
in the repository for the full design reasoning, including why this feature closed a
pre-existing gap in how session environment overrides were guarded rather than opening a new
one.