BitSquare/Codeman
2
0 Fork 0
mirror of https://github.com/Ark0N/Codeman.git synced 2026-11-15 02:49:33 +01:00
Code Issues Packages Projects Releases Wiki Activity

docs(local-model): what each Mac gets, and where every memory figure comes from

Browse Source 
docs/local-models.md gains "Which model fits which Mac": the answer of the
stock catalogue for every memory size from 8 to 512 GB, why the chip
changes it at 36 to 96 GB, and why nothing is offered below 65,536 tokens.
"How much memory a model needs" explains the two runners (llama.cpp
allocates its K/V cache at load, MLX grows it and keeps up to 8 GiB of
prefix snapshots), the three runtime settings the llama.cpp rows assume,
and per row where its memory and speed figures come from. The check after
the download now says that for an MLX build it proves only the weights,
and "What to know" says most figures are calculated, not measured.

CLAUDE.md: every row names its runner and carries its own figures, and
the llama.cpp rows and runtimeEnv() change together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 09:35:28 +01:00
1 parent 989fcd778a
commit 46fa954b2e
2 files changed
+83 -6
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/local-models.md
+82 -5
View File
@@ -14,6 +14,75 @@ This page is the reference for how it works. The HTTP contract is in
[api-reference.md](api-reference.md) ("Local model"). The rules that must not be broken are
in [architecture-invariants.md](architecture-invariants.md#local-model-install).
## Which model fits which Mac
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
fits this Mac's memory at a context a coding agent can work with, and up to four more. What
the catalogue answers today, before the runtime measured anything (the budget is then an
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
| Memory | Budget | What it gets |
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.8 GB at 65,536 tokens |
| 16 GB | 10.4 GB | A small model (`limited`): MiMo V2.6 9B at 65,536 tokens, 9.8 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
| 18 GB | 11.8 GB | The same at 131,072 tokens, 11.0 GB, tight |
| 24 GB | 16.1 GB | The same at 131,072 tokens, with room |
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 21.8 GB and misses the estimate by a hair; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it, or Qwen3.8 27B on a Max chip |
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max; on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build |
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens |
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is
offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own
prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default
file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table.
### How much memory a model needs
The need of a model is its peak: the weights, what stays held per token of context, what the
runner holds beside both, and what it takes for a moment while a long prompt is processed. The
runtime runs a model with one of two runners, which spend memory differently, so every row
names its runner (`runner`) and its figures follow that runner's rule:
- **llama.cpp** (GGUF builds, on Metal) allocates the K/V cache for the whole context when the
model loads and does not grow after that. The runtime is started with
`OLLAMA_KV_CACHE_TYPE=q8_0` (the cache in 8 bits, 17/32 of the default f16; the runtime's
own documentation calls the loss very small), `OLLAMA_FLASH_ATTENTION=1` (an 8-bit cache
needs it) and `LLAMA_ARG_CACHE_RAM=1024` (llama-server keeps copies of earlier prompts, and
this bounds them to 1 GiB). A row's figures are computed from its model's `config.json`: the
K/V cache of its attention layers, its recurrent or sliding-window state, 1.0 GB for the
runner and the 1 GiB prompt cache. No peak: the buffers are allocated at load.
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A
Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit
beside a useful model, so the catalogue lists GGUF builds there.
Where every figure comes from:
| Model | Runner | Memory figures | Speed (tokens per second per 100 GB/s) |
| ---------------------------- | --------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config | 13.6: a published run on an M1 Max |
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config | 1.7: the row below scaled by file size |
| `qwen3.8:27b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.8-27B config | 2.95: a published run without prediction (a floor) |
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
calculation. For a llama.cpp build the check after the download catches one that was judged
too kindly; for an MLX build it cannot (see "The check" below). The measured row's figures come from single long prompts, not a session of many requests, in
which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
## What an install does, in order
The person picks a model, presses "Install on this Mac" and confirms the sizes once. The
@@ -49,8 +118,10 @@ Every step can be repeated: Resume and "Try again" enter the step the job stoppe
copied from the server. `HOME` is a private folder. Exactly these are set:
`OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
`OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
`OLLAMA_MAX_LOADED_MODELS=1`. No `CODEMAN_*` variable and none of the user's own
`OLLAMA_*` settings reach it.
`OLLAMA_MAX_LOADED_MODELS=1`, and three memory settings of the llama.cpp runner:
`OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1` and `LLAMA_ARG_CACHE_RAM=1024`
(see "How much memory a model needs" above). No `CODEMAN_*` variable and none of the user's
own `OLLAMA_*` settings reach it.
- **The port is chosen once**, by binding, from 11435 across 20 ports, and stored. 11434 is
left alone: it is the default of a runtime the user may run themselves, which is never
touched.
@@ -69,8 +140,7 @@ judges the chosen model against it before a byte of the model is downloaded. A m
longer fits ends the job there with "does not fit" and the next model to try. From then on
the window shows the measured budget and the word "estimated" is gone.
The need of a model is its peak: the weights, what stays held per token of context, what the
runner holds beside both, and what it takes for a moment while a long prompt is processed.
The need of a model is its peak (see "How much memory a model needs" above).
### The model download
@@ -86,7 +156,10 @@ After the pull, in this order:
1. One tiny chat with `reasoning_effort: "none"`. It loads the model.
2. `GET /api/ps` must show the whole model on the GPU (`size_vram` equal to `size`) at the
chosen context length.
chosen context length. For a llama.cpp build that is the runtime's reckoning of the whole
load, K/V cache included. For an MLX build it proves only the weights: the runtime books
an MLX model at its weights (runtime issue 19417), so for those rows the catalogue's
figures are all there is.
3. One tool probe on `/v1/chat/completions` (a `tools` array, `reasoning_effort: "none"`,
`max_tokens` 1024). It passes on a tool call whose arguments parse.
@@ -261,6 +334,10 @@ or `python3`.
a session uses the model: close those sessions first.
- **The runtime does not enforce the context length.** A prompt longer than the context was
accepted in testing. Do not rely on the runtime to stop one.
- **Most memory figures are calculated, not measured.** One model was run on one Mac; the
others are computed from their architecture, and the runtime settings they assume (an 8-bit
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
- **The model stays in memory for ten minutes** after the last request, then the runtime
unloads it.