BitSquare/Codeman
2
0 Fork 0
mirror of https://github.com/Ark0N/Codeman.git synced 2026-11-14 10:39:32 +01:00
Code Issues Packages Projects Releases Wiki Activity
Files
dd83d34b8e42bf23b278ec1d0bc134dcd47a20fb
Codeman/docs/local-models.md
T
Codeman maintainer 46fa954b2e docs(local-model): what each Mac gets, and where every memory figure comes from 
docs/local-models.md gains "Which model fits which Mac": the answer of the
stock catalogue for every memory size from 8 to 512 GB, why the chip
changes it at 36 to 96 GB, and why nothing is offered below 65,536 tokens.
"How much memory a model needs" explains the two runners (llama.cpp
allocates its K/V cache at load, MLX grows it and keeps up to 8 GiB of
prefix snapshots), the three runtime settings the llama.cpp rows assume,
and per row where its memory and speed figures come from. The check after
the download now says that for an MLX build it proves only the weights,
and "What to know" says most figures are calculated, not measured.

CLAUDE.md: every row names its runner and carries its own figures, and
the llama.cpp rows and runtimeEnv() change together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:35:28 +01:00

359 lines
26 KiB
Markdown
Raw Blame History

# A model that runs on this Mac
Codeman can set up a coding model that runs on the Mac the server runs on: no account and
no API key, and the session's prompts and files go to a model on this Mac rather than a
cloud provider (the coding agent itself may still check for updates). One button on the welcome screen opens a window that
shows what fits this Mac, installs the choice and ends on a coding agent session that
runs on it.
**Experimental.** The routes and their payloads may change in any release
([versioning-policy.md](versioning-policy.md), "Experimental local model routes"). Apple
silicon macOS only: anywhere else the window says why nothing is offered.
This page is the reference for how it works. The HTTP contract is in
[api-reference.md](api-reference.md) ("Local model"). The rules that must not be broken are
in [architecture-invariants.md](architecture-invariants.md#local-model-install).
## Which model fits which Mac
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
fits this Mac's memory at a context a coding agent can work with, and up to four more. What
the catalogue answers today, before the runtime measured anything (the budget is then an
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
| Memory | Budget | What it gets |
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.8 GB at 65,536 tokens |
| 16 GB | 10.4 GB | A small model (`limited`): MiMo V2.6 9B at 65,536 tokens, 9.8 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
| 18 GB | 11.8 GB | The same at 131,072 tokens, 11.0 GB, tight |
| 24 GB | 16.1 GB | The same at 131,072 tokens, with room |
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 21.8 GB and misses the estimate by a hair; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it, or Qwen3.8 27B on a Max chip |
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max; on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build |
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens |
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is
offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own
prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default
file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table.
### How much memory a model needs
The need of a model is its peak: the weights, what stays held per token of context, what the
runner holds beside both, and what it takes for a moment while a long prompt is processed. The
runtime runs a model with one of two runners, which spend memory differently, so every row
names its runner (`runner`) and its figures follow that runner's rule:
- **llama.cpp** (GGUF builds, on Metal) allocates the K/V cache for the whole context when the
model loads and does not grow after that. The runtime is started with
`OLLAMA_KV_CACHE_TYPE=q8_0` (the cache in 8 bits, 17/32 of the default f16; the runtime's
own documentation calls the loss very small), `OLLAMA_FLASH_ATTENTION=1` (an 8-bit cache
needs it) and `LLAMA_ARG_CACHE_RAM=1024` (llama-server keeps copies of earlier prompts, and
this bounds them to 1 GiB). A row's figures are computed from its model's `config.json`: the
K/V cache of its attention layers, its recurrent or sliding-window state, 1.0 GB for the
runner and the 1 GiB prompt cache. No peak: the buffers are allocated at load.
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A
Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit
beside a useful model, so the catalogue lists GGUF builds there.
Where every figure comes from:
| Model | Runner | Memory figures | Speed (tokens per second per 100 GB/s) |
| ---------------------------- | --------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config | 13.6: a published run on an M1 Max |
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config | 1.7: the row below scaled by file size |
| `qwen3.8:27b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.8-27B config | 2.95: a published run without prediction (a floor) |
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
calculation. For a llama.cpp build the check after the download catches one that was judged
too kindly; for an MLX build it cannot (see "The check" below). The measured row's figures come from single long prompts, not a session of many requests, in
which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
## What an install does, in order
The person picks a model, presses "Install on this Mac" and confirms the sizes once. The
server then runs one job of five steps. The window polls `GET /api/local-model` once a
second; the job runs in the server, so the window can be closed.
| Step | What it does |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime` | Downloads the pinned runtime, compares its hash, unpacks it, chooses a port once, starts it, reads the measured memory budget and judges the choice again |
| `model` | Downloads the model through the runtime |
| `verify` | Loads the model and checks it: one tiny chat, the memory and context the runtime reports, one tool call |
| `agent` | Installs the coding agent with its stock install command. Left out when the agent is installed |
| `connect` | Saves one model endpoint and switches the custom model endpoints setting on |
Every step can be repeated: Resume and "Try again" enter the step the job stopped in.
### The runtime
- **One pinned build.** Ollama 0.40.15 for macOS, a compiled constant
(`OLLAMA_RUNTIME_PIN` in `src/local-model/runtime.ts`): the URL, the size (185,971,671
bytes) and the sha256. No file, setting, environment variable or request can change which
binary is downloaded.
- **Download.** Streamed to `*.partial` while it is hashed. A cut stream is continued with a
Range request, at most three times. The hash is compared before anything is unpacked, and
a mismatch deletes the file. Unpacking is `/usr/bin/tar` into a temporary folder, then one
rename.
- **No administrator rights.** Everything lives in the data directory.
- **Supervised by Codeman.** The runtime is a non-detached child of the server: started at
boot once a model is installed and not stopped, restarted at most three times after a
crash, sent SIGTERM at shutdown. It is not a login item: it stops with Codeman, and a
leftover from a server that was killed is taken back at the next start (see below).
- **Its environment is an allowlist.** `PATH`, `TMPDIR`, `LANG` and the proxy variables are
copied from the server. `HOME` is a private folder. Exactly these are set:
`OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
`OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
`OLLAMA_MAX_LOADED_MODELS=1`, and three memory settings of the llama.cpp runner:
`OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1` and `LLAMA_ARG_CACHE_RAM=1024`
(see "How much memory a model needs" above). No `CODEMAN_*` variable and none of the user's
own `OLLAMA_*` settings reach it.
- **The port is chosen once**, by binding, from 11435 across 20 ports, and stored. 11434 is
left alone: it is the default of a runtime the user may run themselves, which is never
touched.
- **An orphan is recognized.** The pid and the port are stored. At boot a listener on the
port that answers `/api/version` and whose pid runs a binary under Codeman's runtime folder
is Codeman's own leftover from a server that was killed: it is adopted or replaced. Any
other listener is "Port 11435 is in use by another program", with "Use another port" as
the way out.
### The memory budget is measured
Before the runtime exists the budget is an estimate from the installed memory. Right after
it starts, the runtime prints the device total in its log line "inference compute"
(`total="37.4 GiB"` on a 48 GB MacBook). Codeman reads that line, stores the figure and
judges the chosen model against it before a byte of the model is downloaded. A model that no
longer fits ends the job there with "does not fit" and the next model to try. From then on
the window shows the measured budget and the word "estimated" is gone.
The need of a model is its peak (see "How much memory a model needs" above).
### The model download
`POST /api/pull` with `stream: true`. A 23 GB pull sends about 1.4 million progress lines,
so the stream is parsed piece by piece and never kept. Cancel aborts the request, which
stops the download. Resume pulls again and the runtime continues its partial files. An error
line or a cut stream is retried up to five times. Once the pull reported its total, every
size shown for the model is that total.
### The check
After the pull, in this order:
1. One tiny chat with `reasoning_effort: "none"`. It loads the model.
2. `GET /api/ps` must show the whole model on the GPU (`size_vram` equal to `size`) at the
chosen context length. For a llama.cpp build that is the runtime's reckoning of the whole
load, K/V cache included. For an MLX build it proves only the weights: the runtime books
an MLX model at its weights (runtime issue 19417), so for those rows the catalogue's
figures are all there is.
3. One tool probe on `/v1/chat/completions` (a `tools` array, `reasoning_effort: "none"`,
`max_tokens` 1024). It passes on a tool call whose arguments parse.
A failure is a state with a code and one next step, never a bare "failed": `does-not-fit`
and `no-tool-call` offer the next model, `probe-unfinished` (the token limit ended the
probe, so it proves nothing) and `load-failed` offer "Try again". The digest the runtime
reports for the model is stored when the check passed.
### The coding agent
When the chosen coding agent is not installed, the job installs it with the CLI registry's
own stock install command, through the same runner as App Settings > CLIs
(`src/cli-install-runner.ts`). The rules:
- Only a CLI that is a harness row of the local model catalogue.
- Only the **compiled** stock command. A `~/.codeman/clis.json` override cannot change what
runs here.
- Only after the person's consent click, on a screen that showed the literal command.
- Admin only in multi-user mode, and written to the admin audit log (`cli_install`, with
`via: "local-model"`).
- One installer per CLI at a time, shared with App Settings: while one runs, the other
answers that it is busy.
- Afterwards the cached lookups of the CLI's binary are forgotten and the binary is looked
for again. Still missing is the failure `harness-failed`, with the installer's last lines.
This step does not need the `cliManagementEnabled` setting. That setting gates the general
CLI management surface; this path can only ever run one vetted command for one catalogue
row.
### The connection
Two writes outside the local model folder, both undone by Remove:
1. One record in `custom-model-hosts.json`:
```json
{
"id": "local-ollama",
"label": "This Mac",
"baseUrl": "http://127.0.0.1:11435",
"authStyle": "bearer",
"models": ["qwen3.6:35b-a3b"],
"defaultModelId": "qwen3.6:35b-a3b",
"defaultContextLength": 131072
}
```
`defaultContextLength` is the chosen context: the runtime's model listing states none, so
this field is what reaches the coding agent as its context limit.
2. `customModelEndpointsEnabled: true` in `settings.json`. The value it had before is kept
in the state file.
The id `local-ollama` is reserved. A new endpoint cannot take it, and a record with that id
that this feature did not write is never overwritten: the install says so before anything is
downloaded. While the record is the managed one it cannot be edited or deleted in App
Settings and the five-minute discovery sweep leaves it alone, so its model list stays the
one model the install checked.
Switching the setting on also starts that five-minute sweep for every other saved endpoint.
## Starting a session
"Start a session" starts the coding agent on the endpoint `local-ollama` through the
existing custom model launch. Before a coding agent is STARTED on that endpoint the server
checks, in this order:
- the runtime child Codeman started is alive (whatever else answers on the port is not
ours);
- the digest the runtime reports for the installed model is the one stored at install time;
- the model asked for is the one that was installed and checked.
That check runs wherever a new agent process comes to be on the endpoint: a session created
with the model, a model applied to a running session (which restarts its agent), the attach
that starts an agent in a session that has none or whose agent exited, and the rebuild of a
session after a reboot (that entry is skipped with "the local model is not ready" and stays
on offer). An agent that is already running when Codeman restarts is not started by
Codeman, so it is not asked: it keeps running on its port, see below.
A different digest refuses the launch: "The model on this Mac changed since it was
installed. Remove it and install it again." The digest is also read again at boot and when
the runtime is started.
### From a Claude Code session: cloud plus local
The codeman skill ([`skills/codeman`](../skills/codeman/SKILL.md#claude-in-the-cloud-a-local-model-on-this-mac))
lets an agent in a Codeman session hand work to this model, so Claude in the cloud plans and
reviews while the model on this Mac does the bulk: summaries, classification, first drafts, and
files that should not leave the machine. `local_model` reads `GET /api/local-model` and says what
is installed or why nothing can run; `spawn_worker <name> local` starts the runtime when it was
stopped (as Start does), then starts the installed coding agent on `local-ollama` through the same
quick-start launch and the same checks as above; `sendmark` gives the worker a task and reads its
answer back from a file it writes in its own folder. The skill never installs a model and never
answers a confirmation, the runtime still answers one request at a time, and while a worker it
opened is live, Stop and Remove are refused until it is closed.
## Stop, Start, Remove
- **Stop** stops the runtime and keeps it from starting at the next boot. **Start** undoes
that.
- **Remove** stops the runtime, removes the endpoint record, switches the setting off again
only when this feature switched it on and no other endpoint is saved, and deletes the
runtime and its private home. The downloaded model is deleted only when asked. The coding
agent stays. While a live session uses the model, Remove answers with the session names
instead.
## Where everything is
All under the data directory (`~/.codeman`, or the instance's own):
```
local-model.json the state file (mode 0600)
local-model/
runtime/ollama-0.40.15.tgz.partial the runtime archive while it downloads
runtime/ollama-0.40.15/ the unpacked runtime (about 579 MB)
models/ the model store
home/ the runtime's private HOME (its key pair is here)
```
Nothing is written or deleted outside `local-model/` except the state file beside it and the
two writes of the connection. The state file holds no path, so it can never point a delete
somewhere else. The user's own `~/.ollama` is never created, read or removed.
`local-model.json` is not part of a config backup. `custom-model-hosts.json` and
`settings.json` are, so restoring an older backup can take the endpoint record away while
the model stays installed. The record is put back the next time Codeman starts (the setting
is left as the backup had it). A restore can also bring a record back after a Remove: it
then points at a port nothing listens on, is an ordinary endpoint again, and can be deleted
in App Settings.
## The job's states
| State | A server restart | Cancel | What can be done |
| ------------ | ------------------------------------------------------------- | ---------------- | -------------------------------------- |
| `idle` | stays idle | refused | install |
| `installing` | becomes `paused` at the same step | becomes `paused` | cancel |
| `paused` | stays paused | refused | resume, install, remove |
| `failed` | stays failed, same error | refused | resume, install, remove |
| `installed` | stays installed; the runtime is started unless it was stopped | refused | start a session, stop or start, remove |
One job at a time. A job a server restart interrupted comes back paused and never continues
without a click. Cancel interrupts the step in flight (the download, the pull, the request,
the installer's process group); in the last step it is honoured before the first write, and
once the two writes began they finish.
## On a Mac without developer tools
Coding agents call `git` and `python3`. On a Mac without Xcode or the Command Line Tools
those are stubs that open macOS's "install the command line developer tools" window. So the
window says one sentence wherever the person commits or finishes, and only when
`/usr/bin/xcode-select -p` fails:
> Coding agents use git and Python. This Mac has no developer tools yet, so macOS will offer
> to install them the first time an agent needs one.
Codeman itself only ever runs `/usr/bin/xcode-select -p` for this, bounded, and never `git`
or `python3`.
## What to know before relying on it
- **The runtime's port has no login.** Any program on this Mac, and any web page served from
`localhost`, can use the model server and change its models. That is why the endpoint
record lists one model, why the sweep leaves it alone, and why the digest is compared
before a session starts.
- **A Codeman restart stops the runtime.** A session that is mid-answer on the local model
loses that answer. The runtime is started again with the server, and the first request
loads the model again (seconds).
- **An open session keeps calling the runtime's port.** The coding agent was given the
address when it started. While the runtime is stopped, or for the seconds of a Codeman
restart, its requests go to a port nothing of ours holds, and any program on this Mac could
bind that port and answer them. That is why Stop and "Use another port" are refused while
a session uses the model: close those sessions first.
- **The runtime does not enforce the context length.** A prompt longer than the context was
accepted in testing. Do not rely on the runtime to stop one.
- **Most memory figures are calculated, not measured.** One model was run on one Mac; the
others are computed from their architecture, and the runtime settings they assume (an 8-bit
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
- **The model stays in memory for ten minutes** after the last request, then the runtime
unloads it.
## Source
| What | Where |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Hardware probe, catalogue, recommendation | `src/local-model/hardware.ts`, `src/config/local-model-catalog.ts`, `src/local-model/recommend.ts` |
| Runtime pin, download, environment, supervisor | `src/local-model/runtime.ts` |
| Runtime HTTP client, pull parser, checks | `src/local-model/ollama-client.ts` |
| Developer tools check | `src/local-model/developer-tools.ts` |
| Job (state table, reducer, view, service) | `src/local-model/install-job.ts` |
| State file | `src/local-model/state-store.ts` |
| Connection and its undoing | `src/local-model/connect.ts` |
| Paths | `src/local-model/paths.ts` |
| Routes | `src/web/routes/local-model-routes.ts` |
| Shared CLI install runner | `src/cli-install-runner.ts` |
| Serialized writer of the endpoint store | `src/custom-model-hosts-writer.ts` |
Reference in new issue View Git Blame Copy Permalink