fix(local-model): count what llama-server keeps beside the K/V cache, and bound its checkpoints

The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:

- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
  the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
  of that state, two or three per request, up to 32 per slot by default
  (tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
  none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
  llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
  before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
  (nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
  Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
  so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
  llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
  type, f16 by default. The rows counted one state and that layer at q8_0.

What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):

- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
  estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
  0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
  estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
  27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.

test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 10:26:57 +01:00
1 parent bc9c3c1639
commit 2cd3cbd6a0
11 files changed
+286 -168

No files matched your search

+6 -5
View File
@@ -120,10 +120,10 @@ export interface TestedOn {
tokensPerSecond?: number;
}
/** One model of the catalogue. */
/** The runner of the model runtime that runs a model on a Mac, by the registry's name for it. See `LocalModelEntry.runner`. */
export type LocalModelRunner = 'mlx' | 'llamacpp';
/** One model of the catalogue. */
export interface LocalModelEntry {
/** `name:tag` exactly as the model registry lists it. Lowercase, one colon, no slash (`LOCAL_MODEL_ID_PATTERN`). */
id: string;
@@ -156,7 +156,7 @@ export interface LocalModelEntry {
/**
* Bytes that stay HELD in memory per token of context. For a `llamacpp` row: the K/V cache
* of its attention layers in the runtime's cache type, which llama.cpp allocates for the
* whole context at load. For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
* whole context at load (a multi-token-prediction layer's in f16). For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
* recurrent state the runner keeps every 8,192 tokens, which it grows with use. Measured
* on one row (36,000 to 38,000 on the MLX runner, so about 37,000); computed from the
* model's config for the others, see `figures` and the catalogue header.
@@ -165,9 +165,10 @@ export interface LocalModelEntry {
/**
* What the runner holds beside the weights and the context, whatever the context: its own
* buffers, the live recurrent or sliding-window caches, and the caches it keeps of earlier
* prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows, for
* an estimated `mlx` row the 8 GiB of prefix snapshots the runner may keep). Measured on
* one row (about 3 GB); see the catalogue header for the others.
* prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows and
* the copies of the recurrent or sliding-window state in its `LLAMA_ARG_CTX_CHECKPOINTS`
* context checkpoints, for an estimated `mlx` row the 8 GiB of prefix snapshots the runner
* may keep). Measured on one row (about 3 GB); see the catalogue header for the others.
*/
overheadBytes: number;
/**