The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a
prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its
params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what
`need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08:
- qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4).
- qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4.
- qwen3.8:27b-q8_0: none in its params, so the runtime's 4.
The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or
M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row
instead of a yes/no prediction layer.
Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200,
32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those
tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context
of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at
or above n_layer()), and the main context leaves it out, so the row counts that layer once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:
- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
of that state, two or three per request, up to 32 per slot by default
(tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
(nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
type, f16 by default. The rows counted one state and that layer at q8_0.
What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):
- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.
test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalogue had seven Qwen-family rows that all carried the memory
figures measured on one model (Qwen3.6 35B-A3B on the MLX runner), so a
9B or a dense 27B was judged with figures that do not describe it, two
rows had the wrong Mac download size, and every Mac below 32 GB got
"none".
Now eight rows, each with the runner a Mac really uses for its tag (a new
`runner` field: `mlx` or `llamacpp`) and memory figures of its own:
- the 35B-A3B MLX row keeps its measured figures and the tested pair;
- every llama.cpp row is computed from its model's config.json: a q8_0
K/V cache for the whole context (what the runtime now sets), its
recurrent or sliding-window state, 1.0 GB of runner and the 1 GiB host
prompt cache;
- the one other MLX row (Ornith 1.5 122B) is computed the same way,
scaled by what the measured row showed, plus the runner's 8 GiB of
prefix snapshots.
Sizes are the registry's Mac pull totals (2026-11-08, lowercase ids
verified there). What each memory size gets: 8 GB none; 16 to 32 GB a 9B
(limited); 36 GB Qwen3.8 27B or the 35B-A3B as GGUF; 48 GB the tested
pair (or the 27B on a Max chip); 64 and 96 GB the 27B or the 35B-A3B;
128 to 512 GB Ornith 1.5 122B. Nothing but the one row was measured.
The catalogue test recomputes every computed figure from a table of
architectures and lets two rows share figures only when they share the
architecture and the runner. The rule tests of recommend() now run on a
frozen copy of the 2026-11-06 catalogue, so a weekly catalogue update
changes only the per-memory table and the real-Mac tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalogue rows carry the figures measured on a Mac (the MLX build of
qwen3.6:35b-a3b is 23,105,798,956 bytes, about 37 KB held per context token,
about 3 GB of runner overhead, about 5 GB more for a moment while a long
prompt is processed). A model's need is now its peak. Rows that were not
measured use the same per-token and overhead figures and say `estimated`.
qwen3.6:35b-a3b gets the tested entry for an Apple M5 Pro with 48 GB and
OpenCode, with its measured 67.7 tokens per second, so that row may say
"Tested on a Mac like this".
The ranking changes with it: a pair tested on a Mac like this one comes
first, then speed (ok, unknown, slow), then model rank, harness rank and the
larger context. The fit grade decides whether a row is listed and what it
says, it no longer sorts, so a model is offered at the largest context it
fits at. recommend() takes the budget the runtime measured
(`budgetSource: measured`), and modelFitsBudget() judges one model against it.
Candidate rows now always carry `needSource`, `harnesses`, `installBytes`,
`installGb` and, where one applies, `tag`. `kvBytesPerToken` and the prompt
cache tiers are gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The types, the module skeletons and the test fake that let the runtime, the
install job and the window be built at the same time.
Types:
- src/types/local-model-install.ts (new): the runtime pin, the install
states, steps and error codes with one sentence and one next step each,
the progress, the `install` object of GET /api/local-model, the bodies and
409 answers of the install verbs, the state file. `LocalModelStatus` moves
here; `install` is optional until the route emits it.
- src/types/local-model.ts: the catalogue row's Mac memory figures, the
measured speed of a tested entry, the `measured` budget source,
`RecommendOptions`, and the row tag, the coding agents and the install
total of a candidate. All optional until the catalogue and `recommend()`
carry them, so the read-only stage keeps compiling and its tests stay
green.
Modules with final signatures and bodies that throw:
runtime.ts (the compiled pin is final), ollama-client.ts, developer-tools.ts,
state-store.ts, install-job.ts (the state, step and transition tables are
final), connect.ts, custom-model-hosts-writer.ts, cli-install-runner.ts.
Nothing imports them yet, so the server is unchanged.
Implemented: paths.ts (every folder of the feature, from dataPath()), and
`modelFitsBudget()` in recommend.ts, the judgement the job makes with the
measured budget before it downloads a model. `recommend()` takes the options
parameter and does not read it yet.
Test fake: test/fixtures/fake-ollama.ts, a runtime on a port the system
picks, in the shapes measured on a Mac: version, tags, ps, the pull stream
(MLX and GGUF lines, an error line, a cut stream, a resumed pull), chat with
and without a tool call, the model list, a runtime archive with Range
support, and a fake child process that logs the device total.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codeman maintainer