fix(local-model): count what llama-server keeps beside the K/V cache, and bound its checkpoints

The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:

- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
  the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
  of that state, two or three per request, up to 32 per slot by default
  (tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
  none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
  llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
  before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
  (nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
  Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
  so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
  llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
  type, f16 by default. The rows counted one state and that layer at q8_0.

What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):

- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
  estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
  0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
  estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
  27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.

test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 10:26:57 +01:00
1 parent bc9c3c1639
commit 2cd3cbd6a0
11 files changed
+286 -168

No files matched your search

+19 -3
View File
@@ -16,9 +16,9 @@
* either case) are copied from the server's; HOME is the private one; and exactly these are
* set: `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
* `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
* `OLLAMA_MAX_LOADED_MODELS=1`, and the three memory settings of the llama.cpp runner (see
* `OLLAMA_MAX_LOADED_MODELS=1`, and the four memory settings of the llama.cpp runner (see
* MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own
* `OLLAMA_*`.
* `OLLAMA_*` or `LLAMA_ARG_*`.
* - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both
* together: test/local-model-catalog.test.ts holds them to each other). Read in the
* runtime's v0.40.15 source, not yet measured on a Mac:
@@ -34,7 +34,20 @@
* for a coding model, and llama-server reads the variable from the environment it
* inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a
* session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`.
* The MLX runner reads none of the three (nothing under mlxrunner/ does): an `mlx` row's
* - `LLAMA_ARG_CTX_CHECKPOINTS=8` bounds the context checkpoints llama-server keeps per
* slot (`--ctx-checkpoints`, read from the environment the same way). The llama.cpp the
* runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes them for every model
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
* cannot be rolled back (the recurrent state of the linear-attention layers, or the
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by
* default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for
* nothing: an agent's next request restores the newest checkpoint of the previous one.
* 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in
* 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from
* the last checkpoints further back (a compaction, a cleared old tool result) is
* processed again from the newest checkpoint before that point, or from its start.
* 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
* figures do not depend on them.
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
* `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start.
@@ -105,6 +118,8 @@ export const RUNTIME_KEEP_ALIVE = '10m';
export const RUNTIME_KV_CACHE_TYPE = 'q8_0';
/** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */
export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024;
/** `LLAMA_ARG_CTX_CHECKPOINTS`: the context checkpoints llama-server keeps per slot. Counted in every `llamacpp` row's overhead. */
export const RUNTIME_CTX_CHECKPOINTS = 8;
/** The name of the runtime's binary inside its folder and inside the archive. */
const RUNTIME_BINARY_NAME = 'ollama';
@@ -279,6 +294,7 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv {
env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE;
env.OLLAMA_FLASH_ATTENTION = '1';
env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB);
env.LLAMA_ARG_CTX_CHECKPOINTS = String(RUNTIME_CTX_CHECKPOINTS);
return env;
}