mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 18:39:33 +01:00
fix(local-model): rollback copies follow each tag's own draft length
The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what `need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08: - qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4). - qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4. - qwen3.8:27b-q8_0: none in its params, so the runtime's 4. The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row instead of a yes/no prediction layer. Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200, 32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at or above n_layer()), and the main context leaves it out, so the row counts that layer once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2cd3cbd6a0
commit
83fb523979
7 files changed
+64
-64
No files matched your search
@@ -40,13 +40,13 @@
|
||||
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
|
||||
* cannot be rolled back (the recurrent state of the linear-attention layers, or the
|
||||
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by
|
||||
* default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for
|
||||
* nothing: an agent's next request restores the newest checkpoint of the previous one.
|
||||
* 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in
|
||||
* 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from
|
||||
* the last checkpoints further back (a compaction, a cleared old tool result) is
|
||||
* processed again from the newest checkpoint before that point, or from its start.
|
||||
* 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
|
||||
* default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never
|
||||
* uses: its next request restores the newest checkpoint of the previous one. 8
|
||||
* (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a
|
||||
* checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost:
|
||||
* a prompt that differs from the last checkpoints further back (a compaction, a cleared
|
||||
* old tool result) is processed again from the newest checkpoint before that point, or
|
||||
* from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
|
||||
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
|
||||
* figures do not depend on them.
|
||||
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
|
||||
|
||||
Reference in new issue
Block a user