fix(local-model): count what llama-server keeps beside the K/V cache, and bound its checkpoints

The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:

- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
  the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
  of that state, two or three per request, up to 32 per slot by default
  (tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
  none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
  llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
  before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
  (nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
  Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
  so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
  llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
  type, f16 by default. The rows counted one state and that layer at q8_0.

What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):

- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
  estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
  0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
  estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
  27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.

test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/config/local-model-catalog.ts

@@ -68,23 +68,35 @@
  * (src/local-model/recommend.ts). The two runners spend memory differently, so a row's three
  * figures are filled by the rule of its runner:
  *
- *  - `llamacpp` (llama-server on Metal; runtime v0.40.15 source, llm/llama_server.go). It
- *    allocates the K/V cache for the WHOLE context when the model loads, in the cache type the
- *    runtime sets (`RUNTIME_KV_CACHE_TYPE` in src/local-model/runtime.ts, q8_0: 34 bytes per
- *    32 values), and does not grow after that.
- *     - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. The one
- *       multi-token-prediction layer of the `-mtp` tags is counted as one more attention
- *       layer, and so it is for `qwen3.8:27b-q8_0`, whose GGUF may carry it too: neither
- *       whether that GGUF has it nor whether llama.cpp gives it a cache was checked, so this
- *       errs on the careful side.
+ *  - `llamacpp` (llama-server on Metal; runtime v0.40.15 source, llm/llama_server.go, and the
+ *    llama.cpp it bundles, b12111). It allocates the K/V cache for the WHOLE context when the
+ *    model loads, in the cache type the runtime sets (`RUNTIME_KV_CACHE_TYPE` in
+ *    src/local-model/runtime.ts, q8_0: 34 bytes per 32 values), and does not grow after that.
+ *    "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
+ *    its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
+ *    tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
+ *    such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max 4`
+ *    (its default `draft_num_predict` is 4).
+ *     - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model
+ *       adds its prediction layer, whose cache llama-server keeps in a draft context of the
+ *       same length in the draft's own cache type, f16 by default (2 bytes per value).
  *     - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
- *       or the sliding-window cache of `n_swa + 512` cells), + 1.0 GB for the runner's own
- *       buffers (calibrated on a third-party table of a 27B model on llama.cpp: 8.4 GB at
- *       4,096 tokens with 7.15 GB of weights), + the 1024 MiB host prompt cache that
- *       `LLAMA_ARG_CACHE_RAM` allows (`RUNTIME_HOST_PROMPT_CACHE_MIB`).
+ *       or the sliding-window cache of `n_swa + 512` cells), held 1 + 4 times for an MTP
+ *       model (llama.cpp keeps one recurrent state per drafted token for rollback,
+ *       `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
+ *       (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
+ *       state, or the sliding-window cache), + 1.0 GB for the runner's own buffers (calibrated
+ *       on a third-party table of a 27B model on llama.cpp: 8.4 GB at 4,096 tokens with
+ *       7.15 GB of weights), + the 1024 MiB host prompt cache that `LLAMA_ARG_CACHE_RAM`
+ *       allows (`RUNTIME_HOST_PROMPT_CACHE_MIB`). The checkpoints and the prompt cache are
+ *       host memory, not Metal buffers; on a Mac both come out of the same memory, so they
+ *       are counted like the rest.
  *     - peak = 0: its compute buffers are allocated at load and are inside the 1.0 GB.
  *    The one outside check of this rule: a third party measured 65,600 bytes per token of
  *    growth at f16 for a model of the Qwen3.8-27B architecture, where its config gives 65,536.
+ *    The checkpoints and the rollback copies are read in the llama.cpp source
+ *    (tools/server/server-context.cpp `create_checkpoint`, src/llama-memory-recurrent.cpp),
+ *    not measured.
  *  - `mlx` (runtime v0.40.15 source, mlxrunner/). It grows the K/V cache in bf16 with the
  *    tokens used, keeps the recurrent state in f32 and a snapshot of it every 8,192 tokens,
  *    and keeps up to 8 GiB of paged-out prefix snapshots (prefix_cache.go `maxPagedOutBytes`,
@@ -106,9 +118,10 @@
  * After the download the install checks that the runtime holds the whole model on the GPU.
  * For a `llamacpp` row both of the runtime's figures are the buffers llama-server reports
  * allocating (weights, K/V cache, compute; `MemorySize()` in llm/llama_server.go), so the
- * check proves the whole load is on the GPU, whatever the cache type. For an `mlx` row it
- * proves only the weights (issue 19417): the figures above are all that stands between such
- * a row and swap.
+ * check proves the whole load is on the GPU, whatever the cache type; what grows after the
+ * load in host memory (the context checkpoints and the prompt cache) is counted above, not
+ * checked. For an `mlx` row it proves only the weights (issue 19417): the figures above are
+ * all that stands between such a row and swap.
  *
  * Models set aside on purpose (a weekly review should look at them again):
  *  - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B
@@ -308,6 +321,10 @@ const MEASURED_MEMORY = {
 const LLAMACPP_RUNNER_BYTES = 1_000_000_000;
 /** The host prompt cache llama-server may keep: `LLAMA_ARG_CACHE_RAM`, 1024 MiB (`RUNTIME_HOST_PROMPT_CACHE_MIB`). */
 const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024;
+/** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */
+const LLAMACPP_CTX_CHECKPOINTS = 8;
+/** The recurrent states llama.cpp keeps beside the live one for an MTP model: the runtime's draft length, 4. */
+const LLAMACPP_MTP_ROLLBACK_STATES = 4;
 /** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
 const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
 
@@ -334,8 +351,9 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       bytes: 6_550_825_550,
       // 8 attention layers x 4 K/V heads x 256 x 2 x 34/32.
       heldBytesPerToken: 17_408,
-      // 24 linear-attention layers hold 52,690,944 bytes of f32 state.
-      overheadBytes: 52_690_944 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      // 24 linear-attention layers hold 52,690,944 bytes of f32 state: once live, and once in
+      // each context checkpoint.
+      overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [131072, 65536],
@@ -357,7 +375,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       bytes: 6_550_811_033,
       // The architecture and the quantization of the row above, so its figures.
       heldBytesPerToken: 17_408,
-      overheadBytes: 52_690_944 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [131072, 65536],
@@ -378,8 +396,9 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       bytes: 18_593_967_008,
       // 13 full-attention layers x 4 K/V heads x 128 x 2 x 34/32.
       heldBytesPerToken: 14_144,
-      // 36 sliding-window layers: (4,096 + 512) cells x 4 x 128 x 2 x 34/32 = 180,486,144.
-      overheadBytes: 180_486_144 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      // 36 sliding-window layers: (4,096 + 512) cells x 4 x 128 x 2 x 34/32 = 180,486,144, live
+      // and in each context checkpoint.
+      overheadBytes: 180_486_144 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [262144, 131072, 65536],
@@ -398,10 +417,15 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       paramsB: 35,
       runner: 'llamacpp',
       bytes: 22_621_314_381,
-      // (10 attention layers + 1 prediction layer) x 2 K/V heads x 256 x 2 x 34/32.
-      heldBytesPerToken: 11_968,
-      // 30 linear-attention layers hold 65,863,680 bytes of f32 state.
-      overheadBytes: 65_863_680 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      // 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in
+      // f16: 2 x 256 x 2 x 2 = 2,048.
+      heldBytesPerToken: 12_928,
+      // 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 4 rollback
+      // copies of prediction, and in each context checkpoint.
+      overheadBytes:
+        65_863_680 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
+        LLAMACPP_RUNNER_BYTES +
+        LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [262144, 131072, 65536],
@@ -438,9 +462,13 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       paramsB: 27.8,
       runner: 'llamacpp',
       bytes: 29_978_242_050,
-      // The architecture of the row below, its prediction layer counted (file header).
-      heldBytesPerToken: 36_992,
-      overheadBytes: 156_893_184 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      // The architecture of the row below, prediction layer included (its GGUF has one), so its
+      // figures.
+      heldBytesPerToken: 38_912,
+      overheadBytes:
+        156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
+        LLAMACPP_RUNNER_BYTES +
+        LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [262144, 131072, 65536],
@@ -458,10 +486,15 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
       paramsB: 27.8,
       runner: 'llamacpp',
       bytes: 17_741_872_154,
-      // (16 attention layers + 1 prediction layer) x 4 K/V heads x 256 x 2 x 34/32.
-      heldBytesPerToken: 36_992,
-      // 48 linear-attention layers hold 156,893,184 bytes of f32 state.
-      overheadBytes: 156_893_184 + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
+      // 16 attention layers x 4 K/V heads x 256 x 2 x 34/32 = 34,816, + the prediction layer in
+      // f16: 4 x 256 x 2 x 2 = 4,096.
+      heldBytesPerToken: 38_912,
+      // 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback
+      // copies of prediction, and in each context checkpoint.
+      overheadBytes:
+        156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
+        LLAMACPP_RUNNER_BYTES +
+        LLAMACPP_PROMPT_CACHE_BYTES,
       peakExtraBytes: 0,
       figures: 'estimated',
       contexts: [262144, 131072, 65536],

src/local-model/runtime.ts

@@ -16,9 +16,9 @@
  *    either case) are copied from the server's; HOME is the private one; and exactly these are
  *    set: `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
  *    `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
- *    `OLLAMA_MAX_LOADED_MODELS=1`, and the three memory settings of the llama.cpp runner (see
+ *    `OLLAMA_MAX_LOADED_MODELS=1`, and the four memory settings of the llama.cpp runner (see
  *    MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own
- *    `OLLAMA_*`.
+ *    `OLLAMA_*` or `LLAMA_ARG_*`.
  *  - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both
  *    together: test/local-model-catalog.test.ts holds them to each other). Read in the
  *    runtime's v0.40.15 source, not yet measured on a Mac:
@@ -34,7 +34,20 @@
  *       for a coding model, and llama-server reads the variable from the environment it
  *       inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a
  *       session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`.
- *    The MLX runner reads none of the three (nothing under mlxrunner/ does): an `mlx` row's
+ *     - `LLAMA_ARG_CTX_CHECKPOINTS=8` bounds the context checkpoints llama-server keeps per
+ *       slot (`--ctx-checkpoints`, read from the environment the same way). The llama.cpp the
+ *       runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes them for every model
+ *       of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
+ *       cannot be rolled back (the recurrent state of the linear-attention layers, or the
+ *       sliding-window cache), two or three per request, and llama-server keeps up to 32 by
+ *       default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for
+ *       nothing: an agent's next request restores the newest checkpoint of the previous one.
+ *       8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in
+ *       8,192-token steps) keeps the last few requests. The cost: a prompt that differs from
+ *       the last checkpoints further back (a compaction, a cleared old tool result) is
+ *       processed again from the newest checkpoint before that point, or from its start.
+ *       8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
+ *    The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
  *    figures do not depend on them.
  *  - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
  *    `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start.
@@ -105,6 +118,8 @@ export const RUNTIME_KEEP_ALIVE = '10m';
 export const RUNTIME_KV_CACHE_TYPE = 'q8_0';
 /** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */
 export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024;
+/** `LLAMA_ARG_CTX_CHECKPOINTS`: the context checkpoints llama-server keeps per slot. Counted in every `llamacpp` row's overhead. */
+export const RUNTIME_CTX_CHECKPOINTS = 8;
 
 /** The name of the runtime's binary inside its folder and inside the archive. */
 const RUNTIME_BINARY_NAME = 'ollama';
@@ -279,6 +294,7 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv {
   env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE;
   env.OLLAMA_FLASH_ATTENTION = '1';
   env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB);
+  env.LLAMA_ARG_CTX_CHECKPOINTS = String(RUNTIME_CTX_CHECKPOINTS);
   return env;
 }
 

src/types/local-model.ts

@@ -120,10 +120,10 @@ export interface TestedOn {
   tokensPerSecond?: number;
 }
 
-/** One model of the catalogue. */
 /** The runner of the model runtime that runs a model on a Mac, by the registry's name for it. See `LocalModelEntry.runner`. */
 export type LocalModelRunner = 'mlx' | 'llamacpp';
 
+/** One model of the catalogue. */
 export interface LocalModelEntry {
   /** `name:tag` exactly as the model registry lists it. Lowercase, one colon, no slash (`LOCAL_MODEL_ID_PATTERN`). */
   id: string;
@@ -156,7 +156,7 @@ export interface LocalModelEntry {
   /**
    * Bytes that stay HELD in memory per token of context. For a `llamacpp` row: the K/V cache
    * of its attention layers in the runtime's cache type, which llama.cpp allocates for the
-   * whole context at load. For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
+   * whole context at load (a multi-token-prediction layer's in f16). For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
    * recurrent state the runner keeps every 8,192 tokens, which it grows with use. Measured
    * on one row (36,000 to 38,000 on the MLX runner, so about 37,000); computed from the
    * model's config for the others, see `figures` and the catalogue header.
@@ -165,9 +165,10 @@ export interface LocalModelEntry {
   /**
    * What the runner holds beside the weights and the context, whatever the context: its own
    * buffers, the live recurrent or sliding-window caches, and the caches it keeps of earlier
-   * prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows, for
-   * an estimated `mlx` row the 8 GiB of prefix snapshots the runner may keep). Measured on
-   * one row (about 3 GB); see the catalogue header for the others.
+   * prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows and
+   * the copies of the recurrent or sliding-window state in its `LLAMA_ARG_CTX_CHECKPOINTS`
+   * context checkpoints, for an estimated `mlx` row the 8 GiB of prefix snapshots the runner
+   * may keep). Measured on one row (about 3 GB); see the catalogue header for the others.
    */
   overheadBytes: number;
   /**