mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-15 02:49:33 +01:00
docs(local-model): the long-session caveat of the tested pair, and what the check proves
The 48 GB row now says the test was a short task and that MLX may keep up to 8 GiB more in a long session, which brings the pair to the edge of the budget. The check after the download says what it proves for each runner, from the runtime's source: for a GGUF build both of its figures are the buffers llama-server reports allocating, so the check covers the whole load whatever the cache type; for an MLX build only the weights. And the docs say where a new row's architecture goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
dd83d34b8e
commit
97f10a1033
2 files changed
+28
-20
No files matched your search
@@ -103,9 +103,11 @@
|
||||
* measured.
|
||||
*
|
||||
* After the download the install checks that the runtime holds the whole model on the GPU.
|
||||
* For a `llamacpp` row that is the runtime's own reckoning of the load, K/V cache included.
|
||||
* For an `mlx` row it proves only the weights (issue 19417): the figures above are all that
|
||||
* stands between such a row and swap.
|
||||
* For a `llamacpp` row both of the runtime's figures are the buffers llama-server reports
|
||||
* allocating (weights, K/V cache, compute; `MemorySize()` in llm/llama_server.go), so the
|
||||
* check proves the whole load is on the GPU, whatever the cache type. For an `mlx` row it
|
||||
* proves only the weights (issue 19417): the figures above are all that stands between such
|
||||
* a row and swap.
|
||||
*
|
||||
* Models set aside on purpose (a weekly review should look at them again):
|
||||
* - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B
|
||||
|
||||
Reference in new issue
Block a user