From 4423949923d04f1cba883eefcebb8c05f7821182 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Thu, 12 Nov 2026 03:02:08 +0100 Subject: [PATCH] fix(local-model): the MLX row of Muse Glimmer takes a 5 GB peak, and says what was compared Three corrections to the row added in 5396e33b, none of which changes where it is offered: - Its peak. The run gave the process footprint's LIFETIME maximum, read at the end: 3.0 to 3.3 GB above what was held then. That is a floor, because a larger transient during the earlier, longer prompts would not show in it. The row now takes the first measured row's 5 GB: 36.1 GB at 65,536 tokens (0.960 of a 48 GB Mac's estimate) and 39.0 GB at 131,072 (0.972 of what that Mac's runtime measures). - What was compared. The two builds were not given the same prompt in OpenCode: the GGUF build got a one-line task through the skill's sendmark (which adds its answer-file sentence), the MLX build a plain one-line prompt. And the speeds to set side by side are the two from the same 300-token request, 15.2 and 30.2. - Runtime issue 18269: whether the memory fix its maintainer promised predates the pinned runtime is not known, and the header says so. Also a stale comment: the catalogue holds eleven models. Co-Authored-By: Claude Fable 5.2 --- docs/local-models.md | 4 ++-- src/config/local-model-catalog.ts | 29 ++++++++++++++++++++--------- src/local-model/recommend.ts | 2 +- test/local-model-catalog.test.ts | 11 ++++++----- test/local-model-recommend.test.ts | 18 +++++++++--------- 5 files changed, 38 insertions(+), 26 deletions(-) diff --git a/docs/local-models.md b/docs/local-models.md index 42f9fda8..22dbc3c1 100644 --- a/docs/local-models.md +++ b/docs/local-models.md @@ -29,9 +29,9 @@ never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and | 16 GB | 10.4 GB | A small model (`limited`): Ornith 1.0 9B at 65,536 tokens, 9.3 GB, tight (90%). Its build has no vision projector, which makes it 0.9 GB smaller than the other two 9B builds: MiMo V2.6 9B needs 10.2 GB there, 98.7% of the budget, just past the 98% a tight fit may reach. Once the runtime measures what a 16 GB Mac mini measured (10.7 GB), all three fit and MiMo is the pick. No 9B was run on such a Mac by us: only a run settles it | | 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Ornith 1.0 9B (10.5 GB) and Qwen3.5 9B are the alternatives | | 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow | -| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip. Run once on a Mac (an M5 Pro with 48 GB, 2026-11-12): this GGUF build wrote 15 tokens per second, and an OpenCode turn of one line showed no answer after 7 minutes. Its MLX build, which answered the same turn in 45 seconds, needs 34.6 GB and is offered from 48 GB | +| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip. Run once on a Mac (an M5 Pro with 48 GB, 2026-11-12): this GGUF build wrote 15 tokens per second, and an OpenCode worker given a one-line task showed no answer after 7 minutes. Its MLX build, which finished an OpenCode turn on a one-line prompt in 45 seconds, needs 36.1 GB and is offered from 48 GB | | 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare | -| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). A session of many prompts on that Mac (a 95,000-token prompt, then twelve separate ones of 15,000 tokens, measured on 2026-11-12) ended at 33.4 GB of the 40.2 GB its runtime measures, still growing by 0.5 GiB a prompt: MLX may keep up to 8 GiB of earlier prompts, which brings this pair to the edge of the budget. Muse Glimmer 30B as its MLX build is offered here too, at 65,536 tokens (34.6 GB, tight) | +| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). A session of many prompts on that Mac (a 95,000-token prompt, then twelve separate ones of 15,000 tokens, measured on 2026-11-12) ended at 33.4 GB of the 40.2 GB its runtime measures, still growing by 0.5 GiB a prompt: MLX may keep up to 8 GiB of earlier prompts, which brings this pair to the edge of the budget. Muse Glimmer 30B as its MLX build is offered here too, at 65,536 tokens (36.1 GB, tight) | | 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip | | 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) | | 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight | diff --git a/src/config/local-model-catalog.ts b/src/config/local-model-catalog.ts index 22826c40..580012e4 100644 --- a/src/config/local-model-catalog.ts +++ b/src/config/local-model-catalog.ts @@ -130,9 +130,12 @@ * figure, not read in its source). The row takes the upper end. It counts the whole * 8 GiB prefix budget, because the same run filled it: twelve separate prompts of * 15,000 tokens added 0.53 GiB each until the runner dropped the oldest. The process - * (`footprint`) stood 0.2 GB above the runner's own count at the end and had peaked - * 3.0 to 3.3 GB above it. The tag carries a draft model; its tensors are inside - * `bytes`, and what it holds is inside these figures. + * (`footprint`) stood 0.2 GB above the runner's own count at the end. Its lifetime + * peak stood 3.0 to 3.3 GB above that, which is a FLOOR for the peak of a long prompt: + * a lifetime maximum taken at the end of the run hides a transient of up to 9 GB + * during the earlier, longer prompts. So the row takes the first measured row's 5 GB. + * The tag carries a draft model; its tensors are inside `bytes`, and what it holds is + * inside these figures. * - every other `mlx` row: held = the K/V cache in bf16 + one recurrent snapshot per * 8,192 tokens, computed from the config, x 37,000 / 28,340: the ratio of what the * measured row really held to the same computation for it (same runner, same @@ -192,7 +195,8 @@ * a build the MLX engine cannot accelerate (on the Mac it ran on the runtime logged * "MLX engine initialized ... device=gpu" and it wrote twice as fast as the GGUF build), * and 18269 reports it stuck in "Stopping..." and a restart on a 32 GB Mac, a size the - * measured figures keep it away from. + * measured figures keep it away from (its maintainer promised a memory fix "in the next + * release" in September; whether that predates the pinned 0.40.15 is not known). * - The plain tags `qwen3.5:9b`, `mimo-v2.6:9b` and `qwen3.8:27b`: a Mac pulls their MLX * builds, whose 8 GiB of prefix snapshots do not fit the Macs a 9B or a 27B is for. Their * GGUF builds are listed instead. @@ -405,8 +409,12 @@ const MUSE_MLX_MEASURED_MEMORY = { * runner's own count (0.2 GB measured). */ overheadBytes: MLX_PREFIX_CACHE_BYTES + 500_000_000, - /** The process footprint had peaked 3.0 to 3.3 GB above what stayed held. */ - peakExtraBytes: 3_500_000_000, + /** + * The first measured row's 5 GB. This run only gives a floor: the footprint's lifetime peak + * stood 3.0 to 3.3 GB above what was held at the end, and a larger transient during the + * earlier, longer prompts would not show in that figure. + */ + peakExtraBytes: 5_000_000_000, } as const; /** @@ -542,7 +550,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { // The 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on llama.cpp // (16.8 GB each without the projector). Left there on purpose after the one run on a Mac // (M5 Pro, 2026-11-12): the runtime alone wrote 14.5 to 15.2 tokens per second, 4.8 per - // 100 GB/s, but an OpenCode turn of one line showed no answer after 7 minutes, twice. + // 100 GB/s, but an OpenCode worker given one line (through the skill's `sendmark`, which + // adds its answer-file sentence) showed no answer after 7 minutes, twice. speedPer100Gbps: 2.95, tested: [], }, @@ -550,8 +559,10 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { // The same model as the MLX build: about twice as fast, because the tag carries a draft // model (the runtime logs "Loaded draft model", acceptance 0.34 to 0.62) and its own // renderer and tool parser. On the MacBook Pro (Mac17,13, M5 Pro, 48 GB, 2026-11-12) it - // answered a tool call in under 3 seconds and finished a one-line OpenCode turn in 45 - // seconds, where the GGUF build above had shown nothing after 7 minutes. The price is + // wrote 30.2 tokens per second where the GGUF build wrote 15.2 (the same 300-token + // request), answered a tool call in under 3 seconds, and finished an OpenCode turn on + // a one-line prompt in 45 seconds. The GGUF build above had shown nothing after 7 + // minutes on a one-line task (not the same prompt: see that row). The price is // memory: what the MLX runner holds grows six times as fast per token (MEMORY in the // file header), so this build starts at 48 GB and the 32 and 36 GB Macs keep the row // above. Not installed through the window yet, and no agent session beyond that turn: diff --git a/src/local-model/recommend.ts b/src/local-model/recommend.ts index 45b291b5..ff1f849d 100644 --- a/src/local-model/recommend.ts +++ b/src/local-model/recommend.ts @@ -112,7 +112,7 @@ export const LIMITED_BELOW_PARAMS_B = 12; export const DISK_HEADROOM_BYTES = 10 * GIB; /** * How many other models are listed under the pick, at most. Above what the stock catalogue - * holds (10 models: 9 beside the pick), so a Mac is shown every model that fits it and the + * holds (11 models: 10 beside the pick), so a Mac is shown every model that fits it and the * install accepts every one of them. Still a bound: test/local-model-catalog.test.ts fails for * a catalogue that outgrows it, so nothing drops out of the list unseen. */ diff --git a/test/local-model-catalog.test.ts b/test/local-model-catalog.test.ts index b0fa3dc0..5f71dfc4 100644 --- a/test/local-model-catalog.test.ts +++ b/test/local-model-catalog.test.ts @@ -172,10 +172,10 @@ function mlxComputedPerToken(arch: Architecture): number { const MEASURED: [number, number, number] = [37_000, 3_000_000_000, 5_000_000_000]; /** * The figures measured for `muse-glimmer:30b-mlx` on the same MacBook Pro (2026-11-12): the upper - * end of what stayed held per token, the whole prefix budget plus 0.5 GB, and the peak of the - * process above what stayed held. + * end of what stayed held per token, the whole prefix budget plus 0.5 GB, and for the peak the + * first measured row's 5 GB (its own run gave only a floor of 3.0 to 3.3 GB). */ -const MEASURED_MUSE_MLX: [number, number, number] = [45_000, 8 * GIB + 500_000_000, 3_500_000_000]; +const MEASURED_MUSE_MLX: [number, number, number] = [45_000, 8 * GIB + 500_000_000, 5_000_000_000]; /** What an estimated `mlx` row's three figures must be: computed, scaled by what the measured row showed. */ function mlxFigures(arch: Architecture): [number, number, number] { @@ -427,8 +427,9 @@ describe('the memory figures of the stock rows: each row has its own', () => { 'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0], // The same rule: 39 layers x 2,560 cells x 544 bytes = 54,312,960 per copy. 'muse-glimmer:30b-q4_k_m': [7_072, 2_562_558_464, 0], - // Measured: the upper end per token, 8 GiB of prefix snapshots + 0.5 GB, the process peak. - 'muse-glimmer:30b-mlx': [45_000, 9_089_934_592, 3_500_000_000], + // Measured: the upper end per token, 8 GiB of prefix snapshots + 0.5 GB. The peak is the + // first measured row's: this run gave only a floor for it (3.0 to 3.3 GB). + 'muse-glimmer:30b-mlx': [45_000, 9_089_934_592, 5_000_000_000], // State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction // layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies. 'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0], diff --git a/test/local-model-recommend.test.ts b/test/local-model-recommend.test.ts index f89c8c63..c3ddb6e1 100644 --- a/test/local-model-recommend.test.ts +++ b/test/local-model-recommend.test.ts @@ -493,9 +493,9 @@ describe('needBytes', () => { 'north-mini-code-1.0:q4_k_m': 25_999_848_864, // At its own window of 131072 tokens: 18.2 GB + 0.9 GB of K/V cache + 2.6 GB. 'muse-glimmer:30b-q4_k_m': 21_646_509_900, - // The MLX build at the same window: 19.1 GB + 5.9 GB of cache (every layer at full length, - // measured) + the 8 GiB of prefix snapshots and 0.5 GB + 3.5 GB of peak. - 'muse-glimmer:30b-mlx': 37_541_796_584, + // The MLX build at the same window: 19.1 GB + 5.9 GB of cache (45,000 bytes per token, + // measured) + the 8 GiB of prefix snapshots and 0.5 GB + 5 GB of peak. + 'muse-glimmer:30b-mlx': 39_041_796_584, 'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317, 'qwen3.6:35b-a3b': 35_955_462_956, 'qwen3.8:27b-q8_0': 44_292_142_594, @@ -659,8 +659,8 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { ).toEqual([ // The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB. ['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined], - // Muse Glimmer as the MLX build: 34.6 GB at 65536 tokens (0.920 of the estimate), and - // 5.4 x 3.07 = 16.6 tokens per second. At 131072 it needs 37.5 GB, 0.998: out. + // Muse Glimmer as the MLX build: 36.1 GB at 65536 tokens (0.960 of the estimate), and + // 5.4 x 3.07 = 16.6 tokens per second. At 131072 it needs 39.0 GB, 1.04: out. ['muse-glimmer:30b-mlx', 65536, 'tight', 'ok', 16.6, 'tight'], ['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], @@ -911,7 +911,7 @@ describe('every memory size, with the stock catalogue', () => { it('offers Muse Glimmer as the MLX build from 48 GB only, and never as the pick of a Mac whose disk has room', () => { // Measured on a 48 GB Mac: 45,000 bytes held per token and up to 8 GiB of prefix snapshots, - // 34.6 GB at 65536 tokens. Runtime issue 18269 reports this build stuck, and a restart, on + // 36.1 GB at 65536 tokens. Runtime issue 18269 reports this build stuck, and a restart, on // a 32 GB Mac: the figures keep it away from there. for (const hw of SWEEP) { const result = recommend(hw, STOCK, ALL_INSTALLED); @@ -2192,17 +2192,17 @@ describe('the measured budget, end to end', () => { // The same models in the same order. The share of each changes, and two models step up a // context: the 27B in 8 bits fits 65536 tokens under the estimate (0.975) and 131072 under // what the runtime reported (0.976), and Muse Glimmer as the MLX build goes from 65536 - // (0.920) to its whole window of 131072 (0.935). + // (0.960) to its whole window of 131072 (0.972). expect(rows(measured).map((c) => c.modelId)).toEqual(rows(estimated).map((c) => c.modelId)); const stepped = order(estimated).filter((entry, index) => order(measured)[index] !== entry); expect(stepped).toEqual(['muse-glimmer:30b-mlx@65536', 'qwen3.8:27b-q8_0@65536']); expect(order(measured)).toContain('qwen3.8:27b-q8_0@131072'); expect(order(measured)).toContain('muse-glimmer:30b-mlx@131072'); expect(rows(estimated).map((c) => c.budgetShare)).toEqual([ - 0.957, 0.767, 0.92, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576, + 0.957, 0.767, 0.96, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576, ]); expect(rows(measured).map((c) => c.budgetShare)).toEqual([ - 0.895, 0.717, 0.935, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539, + 0.895, 0.717, 0.972, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539, ]); for (const candidate of rows(measured)) { expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);