docs(local-model): the long-session caveat of the tested pair, and what the check proves

The 48 GB row now says the test was a short task and that MLX may keep up
to 8 GiB more in a long session, which brings the pair to the edge of the
budget. The check after the download says what it proves for each runner,
from the runtime's source: for a GGUF build both of its figures are the
buffers llama-server reports allocating, so the check covers the whole
load whatever the cache type; for an MLX build only the weights. And the
docs say where a new row's architecture goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 09:39:27 +01:00
1 parent dd83d34b8e
commit 97f10a1033
2 files changed
+28 -20

No files matched your search

+5 -3
View File
@@ -103,9 +103,11 @@
* measured.
*
* After the download the install checks that the runtime holds the whole model on the GPU.
* For a `llamacpp` row that is the runtime's own reckoning of the load, K/V cache included.
* For an `mlx` row it proves only the weights (issue 19417): the figures above are all that
* stands between such a row and swap.
* For a `llamacpp` row both of the runtime's figures are the buffers llama-server reports
* allocating (weights, K/V cache, compute; `MemorySize()` in llm/llama_server.go), so the
* check proves the whole load is on the GPU, whatever the cache type. For an `mlx` row it
* proves only the weights (issue 19417): the figures above are all that stands between such
* a row and swap.
*
* Models set aside on purpose (a weekly review should look at them again):
* - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B