mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-12 10:09:33 +01:00
feat(local-model): Muse Glimmer 30B as its MLX build, with figures measured on a Mac
Run on a MacBook Pro (M5 Pro, 48 GB, runtime 0.40.15) on 2026-11-12, after the GGUF build this branch lists turned out to be unusable in agent use there: an OpenCode turn of one line showed no answer after 7 minutes, twice, although the runtime alone wrote 15 tokens per second. The MLX build of the same model (`muse-glimmer:30b-mlx`, 19.1 GB) on that Mac: - 32.6 tokens per second for a 1,500-token answer. The tag carries a draft model, and the runtime logs speculative decoding with 0.34 to 0.62 of the drafts accepted. - A correct tool call in under 3 seconds, and the same OpenCode turn finished in 45 seconds. - Memory, read from the runner's own log line after every request: 37,900 to 45,100 bytes held per token (the GGUF build computes to 7,072), prefix snapshots that filled their 8 GiB during twelve separate prompts, and a process peak 3.0 to 3.3 GB above what stayed held. So it gets a row of its own, the second with measured figures. They put it at 34.6 GB for 65,536 tokens: offered from 48 GB up, never on the 32 and 36 GB Macs, which keep the GGUF row untouched (it is still the pick of a 32 GB Mac with a Max chip: whether it should be is the maintainer's decision). It is never the pick of a Mac whose disk has room for Qwen3.6 35B-A3B, and it has no `tested` entry: one turn is not a test. Also written down from the same session: the session of many prompts the Qwen3.6 35B-A3B row owed (5.6 GiB of prefix snapshots after twelve prompts, still growing; its figures are unchanged), and what the two open runtime issues on the MLX tag say against what ran. Not done: the row was not installed through the window, and the measurement used a second instance of the pinned runtime, not Codeman's. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed
1 parent
d25cc08606
commit
5396e33b48
5 files changed
+203
-45
No files matched your search
@@ -192,7 +192,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
|
||||
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ The answer lists EVERY model that fits, and the install accepts only a listed row: `MAX_ALTERNATIVES` is a bound above the catalogue's size (`test/local-model-catalog.test.ts` fails for a catalogue that outgrows it), never a way to shorten the list. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for two rows, the MLX builds of Qwen3.6 35B-A3B and Muse Glimmer 30B, and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ The answer lists EVERY model that fits, and the install accepts only a listed row: `MAX_ALTERNATIVES` is a bound above the catalogue's size (`test/local-model-catalog.test.ts` fails for a catalogue that outgrows it), never a way to shorten the list. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
|
||||
**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated
|
||||
|
||||
|
||||
+21
-11
@@ -18,8 +18,8 @@ in [architecture-invariants.md](architecture-invariants.md#local-model-install).
|
||||
|
||||
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
|
||||
fits this Mac's memory at a context a coding agent can work with, and under it every other
|
||||
model of the catalogue that fits (ten models today: a 16 GB Mac sees one of them, a 512 GB
|
||||
Mac all ten). What the catalogue answers today, before the runtime measured anything (the
|
||||
model of the catalogue that fits (eleven models today: a 16 GB Mac sees one of them, a 512 GB
|
||||
Mac all eleven). What the catalogue answers today, before the runtime measured anything (the
|
||||
budget is then an estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and
|
||||
never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
|
||||
@@ -29,9 +29,9 @@ never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and
|
||||
| 16 GB | 10.4 GB | A small model (`limited`): Ornith 1.0 9B at 65,536 tokens, 9.3 GB, tight (90%). Its build has no vision projector, which makes it 0.9 GB smaller than the other two 9B builds: MiMo V2.6 9B needs 10.2 GB there, 98.7% of the budget, just past the 98% a tight fit may reach. Once the runtime measures what a 16 GB Mac mini measured (10.7 GB), all three fit and MiMo is the pick. No 9B was run on such a Mac by us: only a run settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Ornith 1.0 9B (10.5 GB) and Qwen3.5 9B are the alternatives |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip. Run once on a Mac (an M5 Pro with 48 GB, 2026-11-12): this GGUF build wrote 15 tokens per second, and an OpenCode turn of one line showed no answer after 7 minutes. Its MLX build, which answered the same turn in 45 seconds, needs 34.6 GB and is offered from 48 GB |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). A session of many prompts on that Mac (a 95,000-token prompt, then twelve separate ones of 15,000 tokens, measured on 2026-11-12) ended at 33.4 GB of the 40.2 GB its runtime measures, still growing by 0.5 GiB a prompt: MLX may keep up to 8 GiB of earlier prompts, which brings this pair to the edge of the budget. Muse Glimmer 30B as its MLX build is offered here too, at 65,536 tokens (34.6 GB, tight) |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
@@ -71,7 +71,10 @@ names its runner (`runner`) and its figures follow that runner's rule:
|
||||
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
|
||||
state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A
|
||||
Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit
|
||||
beside a useful model, so the catalogue lists GGUF builds there.
|
||||
beside a useful model, so the catalogue lists GGUF builds there. For Muse Glimmer the MLX
|
||||
runner was measured holding 38,000 to 45,000 bytes per token, against the 7,072 its GGUF
|
||||
build computes to: close to every layer's cache at full length, so it evidently does not
|
||||
trim the sliding-window layers the way llama.cpp does.
|
||||
|
||||
Where every figure comes from:
|
||||
|
||||
@@ -82,6 +85,7 @@ Where every figure comes from:
|
||||
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
|
||||
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
|
||||
| `muse-glimmer:30b-q4_k_m` | llama.cpp | computed from its config: 13 full-attention and 39 sliding-window layers | 2.95: the dense 27B's, whose weights are the same size |
|
||||
| `muse-glimmer:30b-mlx` | MLX | **measured** on an Apple M5 Pro with 48 GB, the 8 GiB of prefix snapshots counted | 5.4: measured beside the 35B-A3B on MLX, 0.32 of its 16.9 |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config, prediction layer included | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 1.7: the row below scaled by file size |
|
||||
@@ -89,12 +93,18 @@ Where every figure comes from:
|
||||
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
|
||||
|
||||
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
|
||||
2026-11-08 (the Ornith 1.0 and Muse Glimmer rows on 2026-11-10). Only one row was ever run on
|
||||
a Mac, so every other answer in the table above is a calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
2026-11-08 (the Ornith 1.0 and Muse Glimmer rows on 2026-11-10, Muse Glimmer's MLX build on
|
||||
2026-11-12). Two rows carry memory figures measured on a Mac, both on an Apple M5 Pro with
|
||||
48 GB; every other answer in the table above is a calculation. Three more rows were installed
|
||||
and loaded on that Mac on 2026-11-12 (Qwen3.5 9B, Ornith 1.0 9B and Muse Glimmer as a GGUF
|
||||
build): that checked their download size and that they load on the GPU, not their memory
|
||||
figures. For a llama.cpp build the check after the download catches a load that was
|
||||
judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt
|
||||
cache, which are counted, not checked); for an MLX build it catches neither (see "The check"
|
||||
below). The measured row's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left
|
||||
below). The 35B-A3B's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more: such a session was measured on it later
|
||||
(the 48 GB row above) and its figures were left as they are. Muse Glimmer's MLX row counts
|
||||
the 8 GiB. The catalogue header lists the models left
|
||||
out on purpose (no evidence of working tool calls, a restrictive licence, a known bug, a
|
||||
build that needs a newer runtime than the pinned one) and why. The strongest of those today
|
||||
is Ripple 65B Preview, published on 2026-11-09: every build of it requires Ollama 0.40.16,
|
||||
@@ -357,8 +367,8 @@ or `python3`.
|
||||
a session uses the model: close those sessions first.
|
||||
- **The runtime does not enforce the context length.** A prompt longer than the context was
|
||||
accepted in testing. Do not rely on the runtime to stop one.
|
||||
- **Most memory figures are calculated, not measured.** One model was run on one Mac; the
|
||||
others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
- **Most memory figures are calculated, not measured.** Two models were measured on one Mac;
|
||||
the others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
|
||||
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
|
||||
- **A tool call that quotes a tool call can be cut off.** The pinned runtime reads the tool
|
||||
|
||||
@@ -25,7 +25,8 @@
|
||||
* so a bad row fails the gate and never reaches the route.
|
||||
*
|
||||
* Where each figure of the stock rows comes from, and what was checked (2026-11-08; the two rows
|
||||
* added on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m`, were read that day).
|
||||
* added on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m`, were read that day;
|
||||
* `muse-glimmer:30b-mlx` was read and measured on 2026-11-12).
|
||||
* [V] read at the source that day, [M] measured on a Mac, [C] computed from [V] inputs:
|
||||
*
|
||||
* | Figure | Source | Checked |
|
||||
@@ -115,6 +116,23 @@
|
||||
* of many requests, in which the prefix snapshots may add up to 8 GiB. Its figures stay
|
||||
* the measured ones (with 8 GiB on top it would not be offered on the Mac it ran on),
|
||||
* and a 30-request session on it is the first thing to measure next.
|
||||
* Measured again on 2026-11-12 on the same Mac, from the runner's own log line
|
||||
* (`msg=memory peak=... held=...` after every request): 39,600 bytes per token between
|
||||
* prompts of 30,000 and 94,500 tokens, and the session this row owed, twelve separate
|
||||
* prompts of 15,000 tokens, added 0.5 GiB of prefix snapshots each (5.6 GiB after
|
||||
* twelve, still growing): the whole run ended at 33.4 GB of the 40.2 GB that Mac's
|
||||
* runtime measures. The row's figures are unchanged by it.
|
||||
* - `muse-glimmer:30b-mlx`: MEASURED the same way on that Mac (2026-11-12, runtime
|
||||
* 0.40.15). What stayed held grew by 37,900 to 45,100 bytes per token between prompts
|
||||
* of 30,000, 60,000 and 94,600 tokens. The 13 full-attention layers alone compute to
|
||||
* 13,312 in bf16 and all 52 layers at full length to 53,248, so the runner evidently
|
||||
* does not trim the 39 sliding-window layers to their window (an inference from the
|
||||
* figure, not read in its source). The row takes the upper end. It counts the whole
|
||||
* 8 GiB prefix budget, because the same run filled it: twelve separate prompts of
|
||||
* 15,000 tokens added 0.53 GiB each until the runner dropped the oldest. The process
|
||||
* (`footprint`) stood 0.2 GB above the runner's own count at the end and had peaked
|
||||
* 3.0 to 3.3 GB above it. The tag carries a draft model; its tensors are inside
|
||||
* `bytes`, and what it holds is inside these figures.
|
||||
* - every other `mlx` row: held = the K/V cache in bf16 + one recurrent snapshot per
|
||||
* 8,192 tokens, computed from the config, x 37,000 / 28,340: the ratio of what the
|
||||
* measured row really held to the same computation for it (same runner, same
|
||||
@@ -167,9 +185,14 @@
|
||||
* open, has it answer with thinking only in a long conversation at 65,536 tokens.
|
||||
* - `qwen3-coder-next` (51.7 GB): the generation before Qwen3.6. Its card shows its results
|
||||
* as pictures only, so there is no figure to rank it by against the rows listed here.
|
||||
* - `muse-glimmer:30b-mlx` and the `-dflash` tags of that model: open runtime issues 17656
|
||||
* and 18269 on the MLX build, and no rule here covers the memory of a separate draft
|
||||
* model. Its GGUF build `muse-glimmer:30b-q4_k_m` is listed.
|
||||
* - The `-dflash` tags of Muse Glimmer (`30b-q4_k_m-dflash`, `30b-nvfp4-dflash`): a separate
|
||||
* draft model whose memory no rule here covers. `muse-glimmer:30b-mlx`, which is built
|
||||
* from the second one's layers, IS listed since 2026-11-12, with measured figures. Its
|
||||
* two open runtime issues were read again that day: 17656 takes its `nvfp4` weights for
|
||||
* a build the MLX engine cannot accelerate (on the Mac it ran on the runtime logged
|
||||
* "MLX engine initialized ... device=gpu" and it wrote twice as fast as the GGUF build),
|
||||
* and 18269 reports it stuck in "Stopping..." and a restart on a 32 GB Mac, a size the
|
||||
* measured figures keep it away from.
|
||||
* - The plain tags `qwen3.5:9b`, `mimo-v2.6:9b` and `qwen3.8:27b`: a Mac pulls their MLX
|
||||
* builds, whose 8 GiB of prefix snapshots do not fit the Macs a 9B or a 27B is for. Their
|
||||
* GGUF builds are listed instead.
|
||||
@@ -368,13 +391,31 @@ const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4;
|
||||
/** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
|
||||
const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* The memory figures measured for `muse-glimmer:30b-mlx` on the same Mac (2026-11-12, runtime
|
||||
* 0.40.15, the MLX runner), in decimal bytes: the runner's own `memory peak / held` log line
|
||||
* after every request, and the process footprint at the end. See the file header (MEMORY).
|
||||
*/
|
||||
const MUSE_MLX_MEASURED_MEMORY = {
|
||||
/** Held grew by 37,900 to 45,100 bytes per token between prompts of 30,000, 60,000 and 94,600 tokens: the upper end. */
|
||||
heldBytesPerToken: 45_000,
|
||||
/**
|
||||
* The prefix snapshots in full (the run filled them: 0.53 GiB per separate prompt of 15,000
|
||||
* tokens until the oldest was dropped), and 0.5 GB for what the process holds beside the
|
||||
* runner's own count (0.2 GB measured).
|
||||
*/
|
||||
overheadBytes: MLX_PREFIX_CACHE_BYTES + 500_000_000,
|
||||
/** The process footprint had peaked 3.0 to 3.3 GB above what stayed held. */
|
||||
peakExtraBytes: 3_500_000_000,
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* The catalogue that ships with Codeman. Never read directly outside this file and its test:
|
||||
* callers use `getLocalModelCatalog()`, which validates it first.
|
||||
*/
|
||||
export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
schema: 1,
|
||||
updated: '2026-11-10',
|
||||
updated: '2026-11-12',
|
||||
// The oldest macOS the model runtime supports.
|
||||
minMacosMajor: 14,
|
||||
// Ordered by rank. The higher rank wins among models that fit and whose speed is graded
|
||||
@@ -498,11 +539,40 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
figures: 'estimated',
|
||||
contexts: [131072, 65536],
|
||||
rank: 5,
|
||||
// Not measured: the 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on
|
||||
// llama.cpp (16.8 GB each without the projector).
|
||||
// The 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on llama.cpp
|
||||
// (16.8 GB each without the projector). Left there on purpose after the one run on a Mac
|
||||
// (M5 Pro, 2026-11-12): the runtime alone wrote 14.5 to 15.2 tokens per second, 4.8 per
|
||||
// 100 GB/s, but an OpenCode turn of one line showed no answer after 7 minutes, twice.
|
||||
speedPer100Gbps: 2.95,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// The same model as the MLX build: about twice as fast, because the tag carries a draft
|
||||
// model (the runtime logs "Loaded draft model", acceptance 0.34 to 0.62) and its own
|
||||
// renderer and tool parser. On the MacBook Pro (Mac17,13, M5 Pro, 48 GB, 2026-11-12) it
|
||||
// answered a tool call in under 3 seconds and finished a one-line OpenCode turn in 45
|
||||
// seconds, where the GGUF build above had shown nothing after 7 minutes. The price is
|
||||
// memory: what the MLX runner holds grows six times as fast per token (MEMORY in the
|
||||
// file header), so this build starts at 48 GB and the 32 and 36 GB Macs keep the row
|
||||
// above. Not installed through the window yet, and no agent session beyond that turn:
|
||||
// no `tested` entry.
|
||||
id: 'muse-glimmer:30b-mlx',
|
||||
label: 'Muse Glimmer 30B (MLX)',
|
||||
license: 'Apache-2.0',
|
||||
paramsB: 29.6,
|
||||
runner: 'mlx',
|
||||
// The layers of its manifest, the draft model's included.
|
||||
bytes: 19_053_621_992,
|
||||
...MUSE_MLX_MEASURED_MEMORY,
|
||||
figures: 'measured',
|
||||
contexts: [131072, 65536],
|
||||
rank: 6,
|
||||
// Measured beside Qwen3.6 35B-A3B on MLX, same Mac, same request (a 1,500-token answer):
|
||||
// 32.6 against 101.9 tokens per second, so 0.32 of that row's 16.9. Like that row's, on
|
||||
// the low side: on 307 GB/s it computes to 16.6.
|
||||
speedPer100Gbps: 5.4,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Macs of 36 GB on a slower chip: the model of the tested pair, as a GGUF build whose
|
||||
// memory does not grow after it loaded. With its multi-token-prediction layer.
|
||||
@@ -523,7 +593,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 6,
|
||||
rank: 7,
|
||||
// Measured: 54.5 tokens per second on 400 GB/s (llama.cpp, without prediction).
|
||||
speedPer100Gbps: 13.6,
|
||||
tested: [],
|
||||
@@ -541,7 +611,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
...MEASURED_MEMORY,
|
||||
figures: 'measured',
|
||||
contexts: [131072, 65536],
|
||||
rank: 7,
|
||||
rank: 8,
|
||||
// Measured: 67.8 tokens per second on 400 GB/s with MLX. The one Mac we tested ran it at
|
||||
// 67.7 on 307 GB/s, so on other chips the estimate is on the low side.
|
||||
speedPer100Gbps: 16.9,
|
||||
@@ -566,7 +636,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 8,
|
||||
rank: 9,
|
||||
// Not measured: the 2.95 of the row below scaled by file size.
|
||||
speedPer100Gbps: 1.7,
|
||||
tested: [],
|
||||
@@ -590,7 +660,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 9,
|
||||
rank: 10,
|
||||
// Measured: 11.8 tokens per second on 400 GB/s (llama.cpp, Q4_K_M, without prediction).
|
||||
// This build predicts several tokens at a time, so it is faster than this figure says.
|
||||
speedPer100Gbps: 2.95,
|
||||
@@ -614,7 +684,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: MEASURED_MEMORY.peakExtraBytes,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072],
|
||||
rank: 10,
|
||||
rank: 11,
|
||||
// Not measured: the 16.9 of Qwen3.6 35B-A3B on MLX x 3/10, its active parameters
|
||||
// against these, rounded down.
|
||||
speedPer100Gbps: 5,
|
||||
|
||||
@@ -106,6 +106,9 @@ const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; draftTo
|
||||
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 },
|
||||
// No prediction layer; the tags with a separate draft model (`-dflash`) are not listed.
|
||||
'muse-glimmer:30b-q4_k_m': { architecture: 'Muse-Glimmer-30B', draftTokens: 0 },
|
||||
// The MLX build. Its figures are measured, so nothing is recomputed from this entry. The tag
|
||||
// carries a draft MODEL (not a prediction layer of llama.cpp): what that holds was measured too.
|
||||
'muse-glimmer:30b-mlx': { architecture: 'Muse-Glimmer-30B', draftTokens: 0 },
|
||||
// Its params layer sets `draft_num_predict` 2.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 },
|
||||
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 },
|
||||
@@ -167,6 +170,12 @@ function mlxComputedPerToken(arch: Architecture): number {
|
||||
|
||||
/** The figures measured for `qwen3.6:35b-a3b` on the MacBook Pro. */
|
||||
const MEASURED: [number, number, number] = [37_000, 3_000_000_000, 5_000_000_000];
|
||||
/**
|
||||
* The figures measured for `muse-glimmer:30b-mlx` on the same MacBook Pro (2026-11-12): the upper
|
||||
* end of what stayed held per token, the whole prefix budget plus 0.5 GB, and the peak of the
|
||||
* process above what stayed held.
|
||||
*/
|
||||
const MEASURED_MUSE_MLX: [number, number, number] = [45_000, 8 * GIB + 500_000_000, 3_500_000_000];
|
||||
|
||||
/** What an estimated `mlx` row's three figures must be: computed, scaled by what the measured row showed. */
|
||||
function mlxFigures(arch: Architecture): [number, number, number] {
|
||||
@@ -193,22 +202,23 @@ describe('the stock catalogue', () => {
|
||||
expect(getLocalModelCatalog()).toBe(first);
|
||||
});
|
||||
|
||||
it('lists ten models, by rank, and none of those set aside in the header', () => {
|
||||
it('lists eleven models, by rank, and none of those set aside in the header', () => {
|
||||
expect(models.map((m) => m.id)).toEqual([
|
||||
'qwen3.5:9b-q4_k_m',
|
||||
'ornith:9b-q4_k_m',
|
||||
'mimo-v2.6:9b-q4_k_m',
|
||||
'north-mini-code-1.0:q4_k_m',
|
||||
'muse-glimmer:30b-q4_k_m',
|
||||
'muse-glimmer:30b-mlx',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m',
|
||||
'qwen3.6:35b-a3b',
|
||||
'qwen3.8:27b-q8_0',
|
||||
'qwen3.8:27b-mtp-q4_k_m',
|
||||
'ornith-1.5:122b',
|
||||
]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]);
|
||||
const setAside =
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:35b|^muse-glimmer:.*(mlx|dflash)/;
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:35b|^muse-glimmer:.*dflash/;
|
||||
expect(models.filter((m) => setAside.test(m.id)).map((m) => m.id)).toEqual([]);
|
||||
// The plain 9B and 27B tags are MLX builds on a Mac: their GGUF builds are listed instead.
|
||||
for (const plain of ['qwen3.5:9b', 'mimo-v2.6:9b', 'qwen3.8:27b']) {
|
||||
@@ -223,6 +233,8 @@ describe('the stock catalogue', () => {
|
||||
'mimo-v2.6:9b-q4_k_m': 'llamacpp',
|
||||
'north-mini-code-1.0:q4_k_m': 'llamacpp',
|
||||
'muse-glimmer:30b-q4_k_m': 'llamacpp',
|
||||
// Its manifest names no runner in a list; the runtime starts its MLX runner for it.
|
||||
'muse-glimmer:30b-mlx': 'mlx',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 'llamacpp',
|
||||
// Manifest lists: a Mac takes the MLX child.
|
||||
'qwen3.6:35b-a3b': 'mlx',
|
||||
@@ -242,6 +254,9 @@ describe('the stock catalogue', () => {
|
||||
'north-mini-code-1.0:q4_k_m': 18_593_967_008,
|
||||
// Fetched on 2026-11-10: 16.76 GB of weights and a 1.40 GB projector.
|
||||
'muse-glimmer:30b-q4_k_m': 18_157_010_252,
|
||||
// Fetched on 2026-11-12: 1,503 layers, the draft model's among them. A pull on a Mac
|
||||
// that day ended at 19.1 GB.
|
||||
'muse-glimmer:30b-mlx': 19_053_621_992,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 22_621_314_381,
|
||||
// The MLX child of the list, and the total of a real pull on a Mac, byte for byte.
|
||||
'qwen3.6:35b-a3b': 23_105_798_956,
|
||||
@@ -252,9 +267,29 @@ describe('the stock catalogue', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('says which row was measured on a Mac: one, and every other row is an estimate', () => {
|
||||
expect(models.filter((m) => m.figures === 'measured').map((m) => m.id)).toEqual(['qwen3.6:35b-a3b']);
|
||||
it('says which rows were measured on a Mac: two, and every other row is an estimate', () => {
|
||||
expect(models.filter((m) => m.figures === 'measured').map((m) => m.id)).toEqual([
|
||||
'muse-glimmer:30b-mlx',
|
||||
'qwen3.6:35b-a3b',
|
||||
]);
|
||||
expect(figuresOf(models.find((m) => m.id === 'qwen3.6:35b-a3b') as LocalModelEntry)).toEqual(MEASURED);
|
||||
expect(figuresOf(models.find((m) => m.id === 'muse-glimmer:30b-mlx') as LocalModelEntry)).toEqual(
|
||||
MEASURED_MUSE_MLX
|
||||
);
|
||||
});
|
||||
|
||||
it('counts the whole prefix budget of the MLX runner in the row that was measured filling it', () => {
|
||||
// Every estimated MLX row counts the 8 GiB too (mlxFigures). The first measured row does
|
||||
// not: the catalogue header says why, and what a session on it really held.
|
||||
expect(MEASURED_MUSE_MLX[1]).toBe(8 * GIB + 500_000_000);
|
||||
expect(MEASURED[1]).toBeLessThan(8 * GIB);
|
||||
// What the runner really keeps per token is far above the 13 full-attention layers alone
|
||||
// and under all 52 layers at full length: it evidently does not trim the sliding-window layers.
|
||||
const arch = ARCHITECTURES['Muse-Glimmer-30B'];
|
||||
expect(arch.attentionLayers * arch.kvHeads * arch.headDim * 2 * 2).toBe(13_312);
|
||||
expect((arch.attentionLayers + arch.sliding.layers) * arch.kvHeads * arch.headDim * 2 * 2).toBe(53_248);
|
||||
expect(MEASURED_MUSE_MLX[0]).toBeGreaterThan(13_312 * 2);
|
||||
expect(MEASURED_MUSE_MLX[0]).toBeLessThanOrEqual(53_248);
|
||||
});
|
||||
|
||||
it('has one tested entry: the 35B-A3B with OpenCode on the MacBook Pro, with its measured speed', () => {
|
||||
@@ -277,6 +312,7 @@ describe('the stock catalogue', () => {
|
||||
'north-mini-code-1.0:q4_k_m': [30, 'Apache-2.0'],
|
||||
// The card's total: a 27.9B language model and a vision encoder.
|
||||
'muse-glimmer:30b-q4_k_m': [29.6, 'Apache-2.0'],
|
||||
'muse-glimmer:30b-mlx': [29.6, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [35, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b': [35, 'Apache-2.0'],
|
||||
'qwen3.8:27b-q8_0': [27.8, 'Apache-2.0'],
|
||||
@@ -309,6 +345,8 @@ describe('the stock catalogue', () => {
|
||||
'north-mini-code-1.0:q4_k_m': 13.6,
|
||||
// The dense 27B's figure: the weights of the two are the same size.
|
||||
'muse-glimmer:30b-q4_k_m': 2.95,
|
||||
// Measured beside the 35B-A3B on MLX, same Mac and request: 32.6 against 101.9, x 16.9.
|
||||
'muse-glimmer:30b-mlx': 5.4,
|
||||
// 54.5 on 400 GB/s with llama.cpp.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 13.6,
|
||||
// 67.8 on 400 GB/s with MLX.
|
||||
@@ -389,6 +427,8 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0],
|
||||
// The same rule: 39 layers x 2,560 cells x 544 bytes = 54,312,960 per copy.
|
||||
'muse-glimmer:30b-q4_k_m': [7_072, 2_562_558_464, 0],
|
||||
// Measured: the upper end per token, 8 GiB of prefix snapshots + 0.5 GB, the process peak.
|
||||
'muse-glimmer:30b-mlx': [45_000, 9_089_934_592, 3_500_000_000],
|
||||
// State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
|
||||
// layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0],
|
||||
|
||||
@@ -493,6 +493,9 @@ describe('needBytes', () => {
|
||||
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
|
||||
// At its own window of 131072 tokens: 18.2 GB + 0.9 GB of K/V cache + 2.6 GB.
|
||||
'muse-glimmer:30b-q4_k_m': 21_646_509_900,
|
||||
// The MLX build at the same window: 19.1 GB + 5.9 GB of cache (every layer at full length,
|
||||
// measured) + the 8 GiB of prefix snapshots and 0.5 GB + 3.5 GB of peak.
|
||||
'muse-glimmer:30b-mlx': 37_541_796_584,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
|
||||
'qwen3.6:35b-a3b': 35_955_462_956,
|
||||
'qwen3.8:27b-q8_0': 44_292_142_594,
|
||||
@@ -656,6 +659,9 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
).toEqual([
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
// Muse Glimmer as the MLX build: 34.6 GB at 65536 tokens (0.920 of the estimate), and
|
||||
// 5.4 x 3.07 = 16.6 tokens per second. At 131072 it needs 37.5 GB, 0.998: out.
|
||||
['muse-glimmer:30b-mlx', 65536, 'tight', 'ok', 16.6, 'tight'],
|
||||
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
['ornith:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
@@ -666,11 +672,17 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
['qwen3.8:27b-q8_0', 65536, 'tight', 'slow', 5.2, 'tight'],
|
||||
['muse-glimmer:30b-q4_k_m', 131072, 'good', 'slow', 9.1, 'slow'],
|
||||
]);
|
||||
// Only the tested row rests on a measurement of that model.
|
||||
expect(new Set(result.alternatives.map((c) => c.needSource))).toEqual(new Set(['estimated']));
|
||||
// Beside the tested pick, one more row rests on a measurement of its own model's memory.
|
||||
expect(result.alternatives.filter((c) => c.needSource === 'measured').map((c) => c.modelId)).toEqual([
|
||||
'muse-glimmer:30b-mlx',
|
||||
]);
|
||||
// Every model of the catalogue that fits is on the list: nothing is cut off after four.
|
||||
const fitting = STOCK.models.filter((m) => m.contexts.some((ctx) => modelFitsBudget(hw48, STOCK, m.id, ctx)));
|
||||
expect(rows(result).map((c) => c.modelId).sort()).toEqual(fitting.map((m) => m.id).sort());
|
||||
expect(
|
||||
rows(result)
|
||||
.map((c) => c.modelId)
|
||||
.sort()
|
||||
).toEqual(fitting.map((m) => m.id).sort());
|
||||
// The 122B fits at no context.
|
||||
expect(order(result).some((entry) => entry.startsWith('ornith-1.5'))).toBe(false);
|
||||
});
|
||||
@@ -704,6 +716,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
['qwen3.8:27b-mtp-q4_k_m', 'unknown'],
|
||||
['qwen3.8:27b-q8_0', 'unknown'],
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 'unknown'],
|
||||
['muse-glimmer:30b-mlx', 'unknown'],
|
||||
['muse-glimmer:30b-q4_k_m', 'unknown'],
|
||||
['north-mini-code-1.0:q4_k_m', 'unknown'],
|
||||
['mimo-v2.6:9b-q4_k_m', 'unknown'],
|
||||
@@ -814,11 +827,7 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => {
|
||||
});
|
||||
// The other two 9B builds follow by rank. The Ornith build is small enough for 131072
|
||||
// tokens here (10.5 GB, 0.974), and a model is listed at the largest context it fits at.
|
||||
expect(order(result)).toEqual([
|
||||
'mimo-v2.6:9b-q4_k_m@65536',
|
||||
'ornith:9b-q4_k_m@131072',
|
||||
'qwen3.5:9b-q4_k_m@65536',
|
||||
]);
|
||||
expect(order(result)).toEqual(['mimo-v2.6:9b-q4_k_m@65536', 'ornith:9b-q4_k_m@131072', 'qwen3.5:9b-q4_k_m@65536']);
|
||||
});
|
||||
|
||||
it('gets the same with Claude Code as its only coding agent, which also needs 65536 tokens', async () => {
|
||||
@@ -900,6 +909,22 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
expect(smallest / result.budgetBytes).toBeGreaterThan(4);
|
||||
});
|
||||
|
||||
it('offers Muse Glimmer as the MLX build from 48 GB only, and never as the pick of a Mac whose disk has room', () => {
|
||||
// Measured on a 48 GB Mac: 45,000 bytes held per token and up to 8 GiB of prefix snapshots,
|
||||
// 34.6 GB at 65536 tokens. Runtime issue 18269 reports this build stuck, and a restart, on
|
||||
// a 32 GB Mac: the figures keep it away from there.
|
||||
for (const hw of SWEEP) {
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
const label = `${hw.memClassGb} GB ${hw.chip}`;
|
||||
const offered = rows(result).some((c) => c.modelId === 'muse-glimmer:30b-mlx');
|
||||
expect(offered, label).toBe(hw.memClassGb >= 48);
|
||||
expect(result.pick?.modelId, label).not.toBe('muse-glimmer:30b-mlx');
|
||||
}
|
||||
// The 32 GB Mac with a Max chip keeps the GGUF build as its pick.
|
||||
const m1max = recommend(mac(32, 'Apple M1 Max'), STOCK, ALL_INSTALLED);
|
||||
expect(m1max.pick?.modelId).toBe('muse-glimmer:30b-q4_k_m');
|
||||
});
|
||||
|
||||
it('gives no model to 8 GB, a small model to every Mac from 16 to 24 GB, and every Mac from 36 GB a model of 27 billion or more', () => {
|
||||
for (const hw of SWEEP) {
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
@@ -912,7 +937,10 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
// A model of 30 billion fits, so the Mac is not `limited`. It is dense: the pick only
|
||||
// where it is not slow (a chip this file knows is fast, or one it knows nothing about).
|
||||
expect(result.verdict, label).toBe('ok');
|
||||
expect(rows(result).map((c) => c.modelId), label).toContain('muse-glimmer:30b-q4_k_m');
|
||||
expect(
|
||||
rows(result).map((c) => c.modelId),
|
||||
label
|
||||
).toContain('muse-glimmer:30b-q4_k_m');
|
||||
expect(result.pick?.paramsB, label).toBe(hw.chip === 'Apple M4' ? 9 : 29.6);
|
||||
if (hw.chip === 'Apple M4') expect(result.notes[0], label).toEqual({ text: LOCAL_MODEL_LABELS.noteSmallPick });
|
||||
} else {
|
||||
@@ -1331,10 +1359,18 @@ describe('verdict none', () => {
|
||||
const oneShort = recommend(withFree(cut - 1), STOCK, ALL_INSTALLED);
|
||||
expect(oneShort.pick?.modelId).toBe('qwen3.6:35b-a3b-mtp-q4_k_m');
|
||||
expect(oneShort.headline.text).toBe(LOCAL_MODEL_LABELS.headlineOk);
|
||||
// At 33 GB neither 35B-A3B build has room: North Mini Code, and the tested pair is named.
|
||||
// At 33 GB neither 35B-A3B build has room: the next rank whose speed is `ok` and whose
|
||||
// download fits is Muse Glimmer as the MLX build (19.1 GB), and the tested pair is named.
|
||||
const short = recommend(withFree(33 * GB), STOCK, ALL_INSTALLED);
|
||||
expect(short.verdict).toBe('ok');
|
||||
expect(short.pick?.modelId).toBe('north-mini-code-1.0:q4_k_m');
|
||||
expect(short.pick).toMatchObject({
|
||||
modelId: 'muse-glimmer:30b-mlx',
|
||||
context: 65536,
|
||||
fit: 'tight',
|
||||
tested: false,
|
||||
});
|
||||
// Below 29.8 GB that build has no room either, and North Mini Code (18.6 GB) is the pick.
|
||||
expect(recommend(withFree(29.5 * GB), STOCK, ALL_INSTALLED).pick?.modelId).toBe('north-mini-code-1.0:q4_k_m');
|
||||
expect(short.notes).toEqual([
|
||||
{ text: LOCAL_MODEL_LABELS.noteDiskDropped, vars: { model: 'Qwen3.6 35B-A3B', missing: '0.8' } },
|
||||
{ text: LOCAL_MODEL_LABELS.noteUntested },
|
||||
@@ -2153,18 +2189,20 @@ describe('the measured budget, end to end', () => {
|
||||
const measured = recommend(hw, STOCK, ALL_INSTALLED, { measuredTotalBytes: MACBOOK_MEASURED_TOTAL });
|
||||
expect([estimated.budgetSource, estimated.budgetGb]).toEqual(['estimated', '37.6']);
|
||||
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
|
||||
// The same models in the same order. The share of each changes, and one model steps up a
|
||||
// The same models in the same order. The share of each changes, and two models step up a
|
||||
// context: the 27B in 8 bits fits 65536 tokens under the estimate (0.975) and 131072 under
|
||||
// what the runtime reported (0.976).
|
||||
// what the runtime reported (0.976), and Muse Glimmer as the MLX build goes from 65536
|
||||
// (0.920) to its whole window of 131072 (0.935).
|
||||
expect(rows(measured).map((c) => c.modelId)).toEqual(rows(estimated).map((c) => c.modelId));
|
||||
const stepped = order(estimated).filter((entry, index) => order(measured)[index] !== entry);
|
||||
expect(stepped).toEqual(['qwen3.8:27b-q8_0@65536']);
|
||||
expect(stepped).toEqual(['muse-glimmer:30b-mlx@65536', 'qwen3.8:27b-q8_0@65536']);
|
||||
expect(order(measured)).toContain('qwen3.8:27b-q8_0@131072');
|
||||
expect(order(measured)).toContain('muse-glimmer:30b-mlx@131072');
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([
|
||||
0.957, 0.767, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576,
|
||||
0.957, 0.767, 0.92, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576,
|
||||
]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([
|
||||
0.895, 0.717, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539,
|
||||
0.895, 0.717, 0.935, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539,
|
||||
]);
|
||||
for (const candidate of rows(measured)) {
|
||||
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
|
||||
|
||||
Reference in new issue
Block a user