The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a
prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its
params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what
`need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08:
- qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4).
- qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4.
- qwen3.8:27b-q8_0: none in its params, so the runtime's 4.
The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or
M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row
instead of a yes/no prediction layer.
Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200,
32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those
tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context
of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at
or above n_layer()), and the main context leaves it out, so the row counts that layer once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:
- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
of that state, two or three per request, up to 32 per slot by default
(tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
(nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
type, f16 by default. The rows counted one state and that layer at q8_0.
What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):
- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.
test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 48 GB row now says the test was a short task and that MLX may keep up
to 8 GiB more in a long session, which brings the pair to the edge of the
budget. The check after the download says what it proves for each runner,
from the runtime's source: for a GGUF build both of its figures are the
buffers llama-server reports allocating, so the check covers the whole
load whatever the cache type; for an MLX build only the weights. And the
docs say where a new row's architecture goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/local-models.md gains "Which model fits which Mac": the answer of the
stock catalogue for every memory size from 8 to 512 GB, why the chip
changes it at 36 to 96 GB, and why nothing is offered below 65,536 tokens.
"How much memory a model needs" explains the two runners (llama.cpp
allocates its K/V cache at load, MLX grows it and keeps up to 8 GiB of
prefix snapshots), the three runtime settings the llama.cpp rows assume,
and per row where its memory and speed figures come from. The check after
the download now says that for an MLX build it proves only the weights,
and "What to know" says most figures are calculated, not measured.
CLAUDE.md: every row names its runner and carries its own figures, and
the llama.cpp rows and runtimeEnv() change together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill-on-by-default change ships in its own pull request with the self-ignoring copy, so this branch keeps the old default and its wording. The local models guide no longer says nothing leaves the machine: the prompts go to the local model, while the coding agent may still check for updates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/local-models.md says how a Claude Code session drives the model on
this Mac through the skill (local_model, spawn_worker local, sendmark)
and what stays true: the skill never installs a model or answers a
confirmation, one request at a time, and Stop and Remove wait for the
workers it opened.
The README gets the local model as a feature with that pairing, and an
example prompt in the skill's table; the wiki's skill page gets the same
row and a short paragraph.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner header says what each caller passes (the job: the compiled stock
command; the route: the merged entry, behind cliManagementEnabled).
docs/cli-registry.md names both executors in its trust boundary. The
security architecture gains a section on the pinned runtime and its hash,
the loopback port without a login and the digest check, and the install
path without cliManagementEnabled; SECURITY.md lists the port as a known
limitation. The guides and the rules name exactly the paths the launch
guard covers, the session that keeps calling a stopped runtime's port, the
claim before the connect writes, and the outcome kept in memory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/local-models.md is the guide: the steps, the layout on disk, what is
written outside the folder, the failures and their next steps. The API
reference documents the install object and the five verbs, and how the list
is ordered. The rules for the server side are in CLAUDE.md and
architecture-invariants.md, the user-facing pages in the wiki.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codeman maintainer