fix(local-model): rollback copies follow each tag's own draft length

The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a
prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its
params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what
`need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08:

- qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4).
- qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4.
- qwen3.8:27b-q8_0: none in its params, so the runtime's 4.

The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or
M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row
instead of a yes/no prediction layer.

Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200,
32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those
tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context
of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at
or above n_layer()), and the main context leaves it out, so the row counts that layer once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 10:38:12 +01:00
1 parent 2cd3cbd6a0
commit 83fb523979
7 files changed
+64 -64

No files matched your search

+7 -7
View File
@@ -40,13 +40,13 @@
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
* cannot be rolled back (the recurrent state of the linear-attention layers, or the
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by
* default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for
* nothing: an agent's next request restores the newest checkpoint of the previous one.
* 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in
* 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from
* the last checkpoints further back (a compaction, a cleared old tool result) is
* processed again from the newest checkpoint before that point, or from its start.
* 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
* default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never
* uses: its next request restores the newest checkpoint of the previous one. 8
* (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a
* checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost:
* a prompt that differs from the last checkpoints further back (a compaction, a cleared
* old tool result) is processed again from the newest checkpoint before that point, or
* from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
* figures do not depend on them.
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across