mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-18 12:19:33 +01:00
fix(local-model): rollback copies follow each tag's own draft length
The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what `need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08: - qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4). - qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4. - qwen3.8:27b-q8_0: none in its params, so the runtime's 4. The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row instead of a yes/no prediction layer. Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200, 32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at or above n_layer()), and the main context leaves it out, so the row counts that layer once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2cd3cbd6a0
commit
83fb523979
7 files changed
+64
-64
No files matched your search
@@ -489,7 +489,7 @@ describe('needBytes', () => {
|
||||
'qwen3.5:9b-q4_k_m': 11_380_487_246,
|
||||
'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
|
||||
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_940_281_677,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
|
||||
'qwen3.6:35b-a3b': 35_955_462_956,
|
||||
'qwen3.8:27b-q8_0': 44_292_142_594,
|
||||
'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698,
|
||||
@@ -647,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
expect(
|
||||
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
|
||||
).toEqual([
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.9 GB.
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
@@ -809,7 +809,7 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
|
||||
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
|
||||
// at 131072 tokens, 0.966 and 0.976 of the budget.
|
||||
// at 131072 tokens, 0.966 and 0.971 of the budget.
|
||||
[36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'],
|
||||
[36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'],
|
||||
// The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick.
|
||||
@@ -2077,8 +2077,8 @@ describe('the measured budget, end to end', () => {
|
||||
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
|
||||
// The same rows in the same order; only the share of each changes.
|
||||
expect(order(measured)).toEqual(order(estimated));
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.77, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.721, 0.647, 0.283, 0.283]);
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
|
||||
for (const candidate of rows(measured)) {
|
||||
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user