The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the
runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row:
- Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of
the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies
of that state, two or three per request, up to 32 per slot by default
(tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB,
none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by
llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default
before it went to 32), and every llamacpp row counts 8 copies of its state.
- Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer
(nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini
Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4,
so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(),
llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache
type, f16 by default. The rows counted one state and that layer at q8_0.
What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned):
- 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB
estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered,
0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the
estimate: this size sits on the edge, and only a run on such a Mac settles it.
- 18 GB: the same pick, 0.964 instead of 0.928.
- 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3).
- 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good.
- 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the
27B at 65,536 on a Max chip (was 131,072).
- Every MLX row, and everything from 64 GB up, is unchanged.
test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds
the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types,
docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source,
not measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
runtimeEnv() now also sets OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_FLASH_ATTENTION=1
and LLAMA_ARG_CACHE_RAM=1024. Read in the v0.40.15 source of the pinned
runtime: the first becomes --cache-type-k/-v of llama-server (17/32 of the
f16 cache), the second --flash-attn on (a quantized V cache needs it, and
"auto" or "off" would fail such a load), and llama-server inherits the third
from ollama serve's environment, which bounds its host prompt cache to
1 GiB instead of letting it grow by gigabytes in a session. The MLX runner
reads none of them.
The catalogue's llama.cpp rows are computed for these values, so the two
change together. Not yet measured on a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The supervisor reports the child's pid the moment it is spawned, before it
answers, so a server killed during a start still leaves a pid that names the
runtime as ours. stop() takes that stored pid (previousPid) and ends a
runtime an earlier server left behind when its command line is a binary
under our runtime folder, so no caller deletes the folder under a live
process. start() takes a version: an installed model keeps the runtime
version it was installed with when a later build pins another one.
removeLocalModelFiles() refuses a local-model folder that is itself a
symbolic link (every check compared against the real path of the root, so
a link in its place passed whatever it pointed at). removePartialModelBlobs()
deletes what an interrupted download left in the model store, which the
runtime answers "not found" for and never deletes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
runtime.ts: the runtime is pinned in a compiled constant (version, URL, size,
sha256). The archive is streamed to a .partial file while it is hashed,
continued with a Range request, retried at most 3 times, and the hash is
compared before anything is unpacked; a mismatch deletes the file. It is
unpacked with /usr/bin/tar into a temporary folder and renamed.
The supervisor runs the runtime as a non-detached child with an environment
built from an allowlist by the pure runtimeEnv(), on a port chosen once by
binding. It restarts the child at most 3 times, stops it with SIGTERM, adopts
or replaces its own orphan on the port and answers port-in-use for any other
listener. parseInferenceComputeTotal() reads the device total the runtime
logs when it starts. Under vitest nothing spawns or fetches.
ollama-client.ts: the pull is parsed incrementally by a pure parser (a 23 GB
pull is about 1.4 million lines), with up to 5 retries on an error line or a
cut stream. verifyModel() loads the model with one tiny chat, checks what the
runtime reports (all on the GPU, the chosen context) and makes one tool
probe; each failure is a code.
developer-tools.ts: only /usr/bin/xcode-select -p, bounded, darwin only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codeman maintainer