mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-15 02:49:33 +01:00
fix(local-model): count what llama-server keeps beside the K/V cache, and bound its checkpoints
The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row: - Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies of that state, two or three per request, up to 32 per slot by default (tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB, none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default before it went to 32), and every llamacpp row counts 8 copies of its state. - Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer (nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4, so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(), llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache type, f16 by default. The rows counted one state and that layer at q8_0. What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned): - 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered, 0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the estimate: this size sits on the edge, and only a run on such a Mac settles it. - 18 GB: the same pick, 0.964 instead of 0.928. - 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3). - 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good. - 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the 27B at 65,536 on a Max chip (was 131,072). - Every MLX row, and everything from 64 GB up, is unchanged. test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types, docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source, not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed
1 parent
bc9c3c1639
commit
2cd3cbd6a0
11 files changed
+286
-168
No files matched your search
@@ -702,7 +702,7 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse
|
||||
|
||||
**The pin** (`OLLAMA_RUNTIME_PIN`, `runtime.ts`) is a COMPILED constant: version, URL, size, sha256. ⚠️ No file, setting, env var or request may supply any part of it: a hash that comes from the same writable place as the URL pins nothing. The archive is streamed to `*.partial` while it is hashed, continued with a Range request (3 bounded retries), refused when it grows past the pinned size, and the hash is compared BEFORE `/usr/bin/tar` unpacks anything into a temporary folder that is then renamed.
|
||||
|
||||
**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`, plus the llama.cpp memory settings `OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1`, `LLAMA_ARG_CACHE_RAM=1024`, which the catalogue's `llamacpp` rows are computed for, so the two change together): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it.
|
||||
**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`, plus the llama.cpp memory settings `OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1`, `LLAMA_ARG_CACHE_RAM=1024`, `LLAMA_ARG_CTX_CHECKPOINTS=8`, which the catalogue's `llamacpp` rows are computed for, so the two change together): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it.
|
||||
|
||||
**The budget is measured.** The runtime prints the device total in its log line "inference compute" (`total="37.4 GiB"`); `parseInferenceComputeTotal()` reads it, the job stores it and judges the chosen model against it (`modelFitsBudget()`) BEFORE the model is downloaded; from then on the GET recommends with `budgetSource: 'measured'`. One consent covers the whole install: the sizes that were confirmed do not change, so a model that no longer fits is a failure with the next model, not a second question.
|
||||
|
||||
|
||||
+37
-27
@@ -22,19 +22,19 @@ the catalogue answers today, before the runtime measured anything (the budget is
|
||||
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
|
||||
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.8 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | A small model (`limited`): MiMo V2.6 9B at 65,536 tokens, 9.8 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 18 GB | 11.8 GB | The same at 131,072 tokens, 11.0 GB, tight |
|
||||
| 24 GB | 16.1 GB | The same at 131,072 tokens, with room |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 21.8 GB and misses the estimate by a hair; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it, or Qwen3.8 27B on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max; on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens. The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.2 GB, 98%). Both tight |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
|
||||
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
|
||||
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
|
||||
@@ -54,10 +54,18 @@ names its runner (`runner`) and its figures follow that runner's rule:
|
||||
model loads and does not grow after that. The runtime is started with
|
||||
`OLLAMA_KV_CACHE_TYPE=q8_0` (the cache in 8 bits, 17/32 of the default f16; the runtime's
|
||||
own documentation calls the loss very small), `OLLAMA_FLASH_ATTENTION=1` (an 8-bit cache
|
||||
needs it) and `LLAMA_ARG_CACHE_RAM=1024` (llama-server keeps copies of earlier prompts, and
|
||||
this bounds them to 1 GiB). A row's figures are computed from its model's `config.json`: the
|
||||
K/V cache of its attention layers, its recurrent or sliding-window state, 1.0 GB for the
|
||||
runner and the 1 GiB prompt cache. No peak: the buffers are allocated at load.
|
||||
needs it), `LLAMA_ARG_CACHE_RAM=1024` (llama-server keeps copies of earlier prompts, and
|
||||
this bounds them to 1 GiB) and `LLAMA_ARG_CTX_CHECKPOINTS=8` (llama-server keeps copies of
|
||||
the state it cannot roll back, two or three per request, up to 32 by default: 5 GB for the
|
||||
27B; 8 keeps the last few requests, which is what an agent's next request restores from). A
|
||||
row's figures are computed from its model's `config.json`: the K/V cache of its attention
|
||||
layers, its recurrent or sliding-window state, live and in each of the 8 checkpoints, 1.0
|
||||
GB for the runner and the 1 GiB prompt cache. A build with a multi-token-prediction layer
|
||||
(the two `-mtp` tags and `qwen3.8:27b-q8_0`) costs more: the runtime turns prediction on,
|
||||
so llama-server keeps that layer's cache in f16 and 4 more copies of the recurrent state
|
||||
for rollback. No peak: the buffers are allocated at load. The checkpoints and the
|
||||
prediction costs were read in the llama.cpp source the runtime bundles (b12111), not
|
||||
measured.
|
||||
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
|
||||
state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A
|
||||
Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit
|
||||
@@ -70,17 +78,19 @@ Where every figure comes from:
|
||||
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
|
||||
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
|
||||
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config, prediction layer included | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config | 1.7: the row below scaled by file size |
|
||||
| `qwen3.8:27b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.8-27B config | 2.95: a published run without prediction (a floor) |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 1.7: the row below scaled by file size |
|
||||
| `qwen3.8:27b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 2.95: a published run without prediction (a floor) |
|
||||
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
|
||||
|
||||
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
|
||||
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
|
||||
calculation. For a llama.cpp build the check after the download catches one that was judged
|
||||
too kindly; for an MLX build it cannot (see "The check" below). The measured row's figures come from single long prompts, not a session of many requests, in
|
||||
which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
|
||||
calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt
|
||||
cache, which are counted, not checked); for an MLX build it catches neither (see "The check"
|
||||
below). The measured row's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
|
||||
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
|
||||
|
||||
A new row needs its model's architecture (attention layers, K/V heads, head size, and its
|
||||
@@ -124,10 +134,10 @@ Every step can be repeated: Resume and "Try again" enter the step the job stoppe
|
||||
copied from the server. `HOME` is a private folder. Exactly these are set:
|
||||
`OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
|
||||
`OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
|
||||
`OLLAMA_MAX_LOADED_MODELS=1`, and three memory settings of the llama.cpp runner:
|
||||
`OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1` and `LLAMA_ARG_CACHE_RAM=1024`
|
||||
(see "How much memory a model needs" above). No `CODEMAN_*` variable and none of the user's
|
||||
own `OLLAMA_*` settings reach it.
|
||||
`OLLAMA_MAX_LOADED_MODELS=1`, and four memory settings of the llama.cpp runner:
|
||||
`OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1`, `LLAMA_ARG_CACHE_RAM=1024` and
|
||||
`LLAMA_ARG_CTX_CHECKPOINTS=8` (see "How much memory a model needs" above). No `CODEMAN_*`
|
||||
variable and none of the user's own `OLLAMA_*` or `LLAMA_ARG_*` settings reach it.
|
||||
- **The port is chosen once**, by binding, from 11435 across 20 ports, and stored. 11434 is
|
||||
left alone: it is the default of a runtime the user may run themselves, which is never
|
||||
touched.
|
||||
|
||||
Reference in new issue
Block a user