mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 02:29:33 +01:00
feat(local-model): list every model that fits, and add Muse Glimmer 30B
The "Run a local model" chooser showed the pick and four more models. It now lists every catalogue model that fits the Mac, and the install accepts every listed row. - MAX_ALTERNATIVES goes from 4 to 11, above what the catalogue holds. It stays a bound: test/local-model-catalog.test.ts fails for a catalogue that outgrows it, so nothing drops off the list unseen. - New row `muse-glimmer:30b-q4_k_m` (Meta, Apache-2.0): SWE-bench Verified 76.0, SWE-bench Pro 51.2, Terminal-Bench 2.1 51.7 on Meta's card. Its K/V cache is the smallest of the 27B class, so it is the first model of that class a 32 GB Mac holds: 21.2 GB at 65,536 tokens, 0.970 of the estimated budget. Figures computed from its config.json and its GGUF header by the llamacpp rule (13 full-attention layers, 39 sliding-window layers of 2,048 tokens, no prediction layer); the sliding-window cache rule was read again in llama.cpp b12111 (src/llama-kv-cache-iswa.cpp). Never run on a Mac: `estimated`. - What changes for a Mac: 32 GB is no longer `limited`. On an M1 Max or M2 Max the pick is Muse Glimmer; on a base or Pro chip the dense model is slow and the pick stays MiMo V2.6 9B, now with the small-pick note. Every other pick is unchanged. - The "set aside" list was read again at its sources on 2026-11-10 and gained ripple-preview:65b (every build requires runtime 0.40.16, one above the pin), Ornith 1.0 (runtime issue 16992), nemotron-3.5-super:120b, qwen3.5:4b, qwen3-coder-next and the MLX and DFlash tags of Muse Glimmer. - The header and the guide now say that runtime issue 16992 has one diagnosed cause in the tool parser every Qwen-family row shares (a tool argument that quotes a whole tool call is cut off; the fix is runtime PR 17021, open). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed
1 parent
f62c1f7e2f
commit
9608a17d99
9 files changed
+271
-88
No files matched your search
@@ -1766,7 +1766,7 @@ A probe that fails is an answer, not an error: a server that is not on macOS, or
|
||||
- `verdict`: `ok`, `limited` or `none`. `limited` is a statement about the Mac: no model with 12 billion parameters or more fits its memory, so what it can run handles small edits and questions and does not replace a cloud model. It is judged on every model that fits, before the free disk space is looked at and whatever the pick is. With `ok` the pick can still be a small model (a larger one may be slow, have no speed figure or have no room on the disk); `notes` then says so.
|
||||
- `reason`, present only with `none`: `not-macos`, `intel`, `memory`, `disk`, `macos` (too old) or `no-harness` (no agent CLI that can be connected to a local model is installed or installable). `memory` has three sentences: nothing fits at all (slot `memory`, the size the Mac is sold as), a model fits but only below the context length an agent CLI needs (slots `budget` and `need`, so a Mac that misses by a little reads as that), and the memory size could not be read (no slot).
|
||||
- `headline` and `notes[]`: `{ text, vars? }`. `text` is an English sentence that may hold `{placeholder}` slots, and `vars` carries the slot values as text. Notes, in order: the pick is a small model although larger ones fit (verdict `ok` only); a model that fits the memory was left out because its download would not leave 10 GiB free on the disk (slots `model` and `missing`: the best-ranked such model and the GB to free up); nobody tested the pick on a Mac like this one.
|
||||
- `pick` (absent with `none`) and `alternatives[]` (at most four, best first; empty with `none`), each one model at one context length with one agent CLI. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `pick` (absent with `none`) and `alternatives[]` (every other model of the catalogue that fits, best first; empty with `none`), each one model at one context length with one agent CLI. An install is accepted only for a row of this list. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `modelId` (`name:tag`), `modelLabel`, `license`, `paramsB` (parameters in billions), `context` (tokens);
|
||||
- `cli`, `cliLabel`, `cliInstalled` (`false`: the CLI is not on this machine yet but can be installed without administrator rights), `slowFirstTurn?`;
|
||||
- `needBytes` and `needGb` (memory the model needs at that context, at its PEAK: the weights, what stays held per token of context, what the runner holds beside both, and what it takes for a moment while a long prompt is processed), `needSource` (`measured`: the figures were measured for this model on a Mac; `estimated`: they are another model's), `downloadBytes` and `downloadGb`, `budgetShare` (need divided by `budgetBytes`, rounded to three decimals);
|
||||
|
||||
+34
-22
@@ -17,28 +17,29 @@ in [architecture-invariants.md](architecture-invariants.md#local-model-install).
|
||||
## Which model fits which Mac
|
||||
|
||||
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
|
||||
fits this Mac's memory at a context a coding agent can work with, and up to four more. What
|
||||
the catalogue answers today, before the runtime measured anything (the budget is then an
|
||||
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
|
||||
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
fits this Mac's memory at a context a coding agent can work with, and under it every other
|
||||
model of the catalogue that fits (nine models today: an 18 GB Mac sees two of them, a 512 GB
|
||||
Mac all nine). What the catalogue answers today, before the runtime measured anything (the
|
||||
budget is then an estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and
|
||||
never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
|
||||
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
|
||||
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
|
||||
after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is
|
||||
after the fast ones, which is why the chip changes the pick at 32 to 96 GB. Every model is
|
||||
offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own
|
||||
prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default
|
||||
file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table.
|
||||
@@ -79,6 +80,7 @@ Where every figure comes from:
|
||||
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
|
||||
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
|
||||
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
|
||||
| `muse-glimmer:30b-q4_k_m` | llama.cpp | computed from its config: 13 full-attention and 39 sliding-window layers | 2.95: the dense 27B's, whose weights are the same size |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config, prediction layer included | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 1.7: the row below scaled by file size |
|
||||
@@ -86,13 +88,16 @@ Where every figure comes from:
|
||||
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
|
||||
|
||||
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
|
||||
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
|
||||
calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
2026-11-08 (the Muse Glimmer row on 2026-11-10). Only one row was ever run on
|
||||
a Mac, so every other answer in the table above is a calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt
|
||||
cache, which are counted, not checked); for an MLX build it catches neither (see "The check"
|
||||
below). The measured row's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
|
||||
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left
|
||||
out on purpose (no evidence of working tool calls, a restrictive licence, a known bug, a
|
||||
build that needs a newer runtime than the pinned one) and why. The strongest of those today
|
||||
is Ripple 65B Preview, published on 2026-11-09: every build of it requires Ollama 0.40.16,
|
||||
one above the pin.
|
||||
|
||||
A new row needs its model's architecture (attention layers, K/V heads, head size, and its
|
||||
linear-attention or sliding-window layers, from its `config.json`) in the table of
|
||||
@@ -355,6 +360,13 @@ or `python3`.
|
||||
others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
|
||||
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
|
||||
- **A tool call that quotes a tool call can be cut off.** The pinned runtime reads the tool
|
||||
calls of every Qwen-family model in the catalogue (the Ornith, MiMo and Qwen rows, the pair
|
||||
we ran included) with one parser, and that parser ends a call at the first closing tag it
|
||||
sees. An argument that itself contains a whole tool call, such as a prompt handed to a
|
||||
sub-agent that shows one, arrives cut off and the rest as text (runtime issue 16992, filed
|
||||
on `ornith:35b`; the fix, runtime pull request 17021, is open). The check after a download
|
||||
asks for one plain tool call and does not see this.
|
||||
- **The model stays in memory for ten minutes** after the last request, then the runtime
|
||||
unloads it.
|
||||
|
||||
|
||||
Reference in new issue
Block a user