mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-15 10:59:32 +01:00
fix(local-model): count what llama-server keeps beside the K/V cache, and bound its checkpoints
The llama.cpp memory rule of the catalogue left out two allocations that the llama.cpp the runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes for every GGUF row: - Context checkpoints. For a model whose state cannot be rolled back (the recurrent state of the Qwen-family rows, the sliding window of North Mini Code), llama-server keeps host copies of that state, two or three per request, up to 32 per slot by default (tools/server/server-context.cpp create_checkpoint). For the 27B that is 32 x 157 MB = 5 GB, none of which was counted. The runtime now sets LLAMA_ARG_CTX_CHECKPOINTS=8 (read by llama-server from its environment like LLAMA_ARG_CACHE_RAM; 8 was llama.cpp's own default before it went to 32), and every llamacpp row counts 8 copies of its state. - Multi-token prediction. The two -mtp tags and qwen3.8:27b-q8_0 carry a prediction layer (nextn_predict_layers 1, read from the first bytes of each GGUF; the 9B tags and North Mini Code have none). The runtime then always passes --spec-type draft-mtp --spec-draft-n-max 4, so llama.cpp keeps 4 rollback copies of the recurrent state (need_n_rs_seq(), llama-memory-recurrent.cpp) and the prediction layer's cache in the draft context's cache type, f16 by default. The rows counted one state and that layer at q8_0. What moves (test/local-model-recommend.test.ts, "every memory size", re-pinned): - 16 GB: limited -> none. MiMo 9B at 65,536 tokens needs 10.2 GB, 0.987 of the 10.4 GB estimate. With the budget the Mac mini measured (10.7 GB after the reserve) it is offered, 0.954, tight; that test is kept. With 4 checkpoints instead of 8 it would be 0.966 on the estimate: this size sits on the edge, and only a run on such a Mac settles it. - 18 GB: the same pick, 0.964 instead of 0.928. - 36 GB: the same picks, 0.966 (27B on M4/M5 Max) and 0.976 (35B-A3B GGUF on M3). - 48 GB on a Max chip: the 27B at 262,144 tokens is tight (0.853) instead of good. - 32 GB with a measured budget of 0.74: North Mini Code at 131,072 tokens (was 262,144), the 27B at 65,536 on a Max chip (was 131,072). - Every MLX row, and everything from 64 GB up, is unchanged. test/local-model-catalog.test.ts recomputes the new rule from the architecture table and holds the catalogue to LLAMA_ARG_CTX_CHECKPOINTS. The catalogue header, runtime.ts, the types, docs/local-models.md, CLAUDE.md and architecture-invariants say the same. Read in the source, not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc9c3c1639
commit
2cd3cbd6a0
11 files changed
+286
-168
No files matched your search
@@ -120,10 +120,10 @@ export interface TestedOn {
|
||||
tokensPerSecond?: number;
|
||||
}
|
||||
|
||||
/** One model of the catalogue. */
|
||||
/** The runner of the model runtime that runs a model on a Mac, by the registry's name for it. See `LocalModelEntry.runner`. */
|
||||
export type LocalModelRunner = 'mlx' | 'llamacpp';
|
||||
|
||||
/** One model of the catalogue. */
|
||||
export interface LocalModelEntry {
|
||||
/** `name:tag` exactly as the model registry lists it. Lowercase, one colon, no slash (`LOCAL_MODEL_ID_PATTERN`). */
|
||||
id: string;
|
||||
@@ -156,7 +156,7 @@ export interface LocalModelEntry {
|
||||
/**
|
||||
* Bytes that stay HELD in memory per token of context. For a `llamacpp` row: the K/V cache
|
||||
* of its attention layers in the runtime's cache type, which llama.cpp allocates for the
|
||||
* whole context at load. For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
|
||||
* whole context at load (a multi-token-prediction layer's in f16). For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
|
||||
* recurrent state the runner keeps every 8,192 tokens, which it grows with use. Measured
|
||||
* on one row (36,000 to 38,000 on the MLX runner, so about 37,000); computed from the
|
||||
* model's config for the others, see `figures` and the catalogue header.
|
||||
@@ -165,9 +165,10 @@ export interface LocalModelEntry {
|
||||
/**
|
||||
* What the runner holds beside the weights and the context, whatever the context: its own
|
||||
* buffers, the live recurrent or sliding-window caches, and the caches it keeps of earlier
|
||||
* prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows, for
|
||||
* an estimated `mlx` row the 8 GiB of prefix snapshots the runner may keep). Measured on
|
||||
* one row (about 3 GB); see the catalogue header for the others.
|
||||
* prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows and
|
||||
* the copies of the recurrent or sliding-window state in its `LLAMA_ARG_CTX_CHECKPOINTS`
|
||||
* context checkpoints, for an estimated `mlx` row the 8 GiB of prefix snapshots the runner
|
||||
* may keep). Measured on one row (about 3 GB); see the catalogue header for the others.
|
||||
*/
|
||||
overheadBytes: number;
|
||||
/**
|
||||
|
||||
Reference in new issue
Block a user