fix(local-model): rollback copies follow each tag's own draft length

The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a
prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its
params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what
`need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08:

- qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4).
- qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4.
- qwen3.8:27b-q8_0: none in its params, so the runtime's 4.

The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or
M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row
instead of a yes/no prediction layer.

Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200,
32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those
tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context
of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at
or above n_layer()), and the main context leaves it out, so the row counts that layer once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 10:38:12 +01:00
1 parent 2cd3cbd6a0
commit 83fb523979
7 files changed
+64 -64

No files matched your search

+5 -5
View File
@@ -489,7 +489,7 @@ describe('needBytes', () => {
'qwen3.5:9b-q4_k_m': 11_380_487_246,
'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_940_281_677,
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
'qwen3.6:35b-a3b': 35_955_462_956,
'qwen3.8:27b-q8_0': 44_292_142_594,
'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698,
@@ -647,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
expect(
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
).toEqual([
// The same model as a GGUF build: its fixed footprint holds the full window, 28.9 GB.
// The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
@@ -809,7 +809,7 @@ describe('every memory size, with the stock catalogue', () => {
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
// at 131072 tokens, 0.966 and 0.976 of the budget.
// at 131072 tokens, 0.966 and 0.971 of the budget.
[36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'],
[36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'],
// The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick.
@@ -2077,8 +2077,8 @@ describe('the measured budget, end to end', () => {
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
// The same rows in the same order; only the share of each changes.
expect(order(measured)).toEqual(order(estimated));
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.77, 0.692, 0.303, 0.303]);
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.721, 0.647, 0.283, 0.283]);
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
for (const candidate of rows(measured)) {
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
}