mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 02:29:33 +01:00
feat(local-model): list every model that fits, and add Muse Glimmer 30B
The "Run a local model" chooser showed the pick and four more models. It now lists every catalogue model that fits the Mac, and the install accepts every listed row. - MAX_ALTERNATIVES goes from 4 to 11, above what the catalogue holds. It stays a bound: test/local-model-catalog.test.ts fails for a catalogue that outgrows it, so nothing drops off the list unseen. - New row `muse-glimmer:30b-q4_k_m` (Meta, Apache-2.0): SWE-bench Verified 76.0, SWE-bench Pro 51.2, Terminal-Bench 2.1 51.7 on Meta's card. Its K/V cache is the smallest of the 27B class, so it is the first model of that class a 32 GB Mac holds: 21.2 GB at 65,536 tokens, 0.970 of the estimated budget. Figures computed from its config.json and its GGUF header by the llamacpp rule (13 full-attention layers, 39 sliding-window layers of 2,048 tokens, no prediction layer); the sliding-window cache rule was read again in llama.cpp b12111 (src/llama-kv-cache-iswa.cpp). Never run on a Mac: `estimated`. - What changes for a Mac: 32 GB is no longer `limited`. On an M1 Max or M2 Max the pick is Muse Glimmer; on a base or Pro chip the dense model is slow and the pick stays MiMo V2.6 9B, now with the small-pick note. Every other pick is unchanged. - The "set aside" list was read again at its sources on 2026-11-10 and gained ripple-preview:65b (every build requires runtime 0.40.16, one above the pin), Ornith 1.0 (runtime issue 16992), nemotron-3.5-super:120b, qwen3.5:4b, qwen3-coder-next and the MLX and DFlash tags of Muse Glimmer. - The header and the guide now say that runtime issue 16992 has one diagnosed cause in the tool parser every Qwen-family row shares (a tool argument that quotes a whole tool call is cut off; the fix is runtime PR 17021, open). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed
1 parent
f62c1f7e2f
commit
9608a17d99
9 files changed
+271
-88
No files matched your search
@@ -192,7 +192,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
|
||||
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ The answer lists EVERY model that fits, and the install accepts only a listed row: `MAX_ALTERNATIVES` is a bound above the catalogue's size (`test/local-model-catalog.test.ts` fails for a catalogue that outgrows it), never a way to shorten the list. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
|
||||
**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated
|
||||
|
||||
|
||||
@@ -1766,7 +1766,7 @@ A probe that fails is an answer, not an error: a server that is not on macOS, or
|
||||
- `verdict`: `ok`, `limited` or `none`. `limited` is a statement about the Mac: no model with 12 billion parameters or more fits its memory, so what it can run handles small edits and questions and does not replace a cloud model. It is judged on every model that fits, before the free disk space is looked at and whatever the pick is. With `ok` the pick can still be a small model (a larger one may be slow, have no speed figure or have no room on the disk); `notes` then says so.
|
||||
- `reason`, present only with `none`: `not-macos`, `intel`, `memory`, `disk`, `macos` (too old) or `no-harness` (no agent CLI that can be connected to a local model is installed or installable). `memory` has three sentences: nothing fits at all (slot `memory`, the size the Mac is sold as), a model fits but only below the context length an agent CLI needs (slots `budget` and `need`, so a Mac that misses by a little reads as that), and the memory size could not be read (no slot).
|
||||
- `headline` and `notes[]`: `{ text, vars? }`. `text` is an English sentence that may hold `{placeholder}` slots, and `vars` carries the slot values as text. Notes, in order: the pick is a small model although larger ones fit (verdict `ok` only); a model that fits the memory was left out because its download would not leave 10 GiB free on the disk (slots `model` and `missing`: the best-ranked such model and the GB to free up); nobody tested the pick on a Mac like this one.
|
||||
- `pick` (absent with `none`) and `alternatives[]` (at most four, best first; empty with `none`), each one model at one context length with one agent CLI. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `pick` (absent with `none`) and `alternatives[]` (every other model of the catalogue that fits, best first; empty with `none`), each one model at one context length with one agent CLI. An install is accepted only for a row of this list. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `modelId` (`name:tag`), `modelLabel`, `license`, `paramsB` (parameters in billions), `context` (tokens);
|
||||
- `cli`, `cliLabel`, `cliInstalled` (`false`: the CLI is not on this machine yet but can be installed without administrator rights), `slowFirstTurn?`;
|
||||
- `needBytes` and `needGb` (memory the model needs at that context, at its PEAK: the weights, what stays held per token of context, what the runner holds beside both, and what it takes for a moment while a long prompt is processed), `needSource` (`measured`: the figures were measured for this model on a Mac; `estimated`: they are another model's), `downloadBytes` and `downloadGb`, `budgetShare` (need divided by `budgetBytes`, rounded to three decimals);
|
||||
|
||||
+34
-22
@@ -17,28 +17,29 @@ in [architecture-invariants.md](architecture-invariants.md#local-model-install).
|
||||
## Which model fits which Mac
|
||||
|
||||
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
|
||||
fits this Mac's memory at a context a coding agent can work with, and up to four more. What
|
||||
the catalogue answers today, before the runtime measured anything (the budget is then an
|
||||
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
|
||||
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
fits this Mac's memory at a context a coding agent can work with, and under it every other
|
||||
model of the catalogue that fits (nine models today: an 18 GB Mac sees two of them, a 512 GB
|
||||
Mac all nine). What the catalogue answers today, before the runtime measured anything (the
|
||||
budget is then an estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and
|
||||
never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
|
||||
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
|
||||
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
|
||||
after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is
|
||||
after the fast ones, which is why the chip changes the pick at 32 to 96 GB. Every model is
|
||||
offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own
|
||||
prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default
|
||||
file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table.
|
||||
@@ -79,6 +80,7 @@ Where every figure comes from:
|
||||
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
|
||||
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
|
||||
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
|
||||
| `muse-glimmer:30b-q4_k_m` | llama.cpp | computed from its config: 13 full-attention and 39 sliding-window layers | 2.95: the dense 27B's, whose weights are the same size |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config, prediction layer included | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 1.7: the row below scaled by file size |
|
||||
@@ -86,13 +88,16 @@ Where every figure comes from:
|
||||
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
|
||||
|
||||
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
|
||||
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
|
||||
calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
2026-11-08 (the Muse Glimmer row on 2026-11-10). Only one row was ever run on
|
||||
a Mac, so every other answer in the table above is a calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt
|
||||
cache, which are counted, not checked); for an MLX build it catches neither (see "The check"
|
||||
below). The measured row's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
|
||||
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left
|
||||
out on purpose (no evidence of working tool calls, a restrictive licence, a known bug, a
|
||||
build that needs a newer runtime than the pinned one) and why. The strongest of those today
|
||||
is Ripple 65B Preview, published on 2026-11-09: every build of it requires Ollama 0.40.16,
|
||||
one above the pin.
|
||||
|
||||
A new row needs its model's architecture (attention layers, K/V heads, head size, and its
|
||||
linear-attention or sliding-window layers, from its `config.json`) in the table of
|
||||
@@ -355,6 +360,13 @@ or `python3`.
|
||||
others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
|
||||
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
|
||||
- **A tool call that quotes a tool call can be cut off.** The pinned runtime reads the tool
|
||||
calls of every Qwen-family model in the catalogue (the Ornith, MiMo and Qwen rows, the pair
|
||||
we ran included) with one parser, and that parser ends a call at the first closing tag it
|
||||
sees. An argument that itself contains a whole tool call, such as a prompt handed to a
|
||||
sub-agent that shows one, arrives cut off and the rest as text (runtime issue 16992, filed
|
||||
on `ornith:35b`; the fix, runtime pull request 17021, is open). The check after a download
|
||||
asks for one plain tool call and does not see this.
|
||||
- **The model stays in memory for ten minutes** after the last request, then the runtime
|
||||
unloads it.
|
||||
|
||||
|
||||
@@ -24,8 +24,9 @@
|
||||
* throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses,
|
||||
* so a bad row fails the gate and never reaches the route.
|
||||
*
|
||||
* Where each figure of the stock rows comes from, and what was checked (2026-11-08). [V] read
|
||||
* at the source that day, [M] measured on a Mac, [C] computed from [V] inputs:
|
||||
* Where each figure of the stock rows comes from, and what was checked (2026-11-08; the row
|
||||
* added on 2026-11-10, `muse-glimmer:30b-q4_k_m`, was read that day). [V] read at the source
|
||||
* that day, [M] measured on a Mac, [C] computed from [V] inputs:
|
||||
*
|
||||
* | Figure | Source | Checked |
|
||||
* |----------------------|-----------------------------------------------------------|--------------------------------------|
|
||||
@@ -74,8 +75,10 @@
|
||||
* src/local-model/runtime.ts, q8_0: 34 bytes per 32 values), and does not grow after that.
|
||||
* "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
|
||||
* its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
|
||||
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
|
||||
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N
|
||||
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not; read
|
||||
* the same way on 2026-11-10, `muse-glimmer:30b-q4_k_m` does not either). For such a
|
||||
* model the runtime always passes
|
||||
* `--spec-type draft-mtp --spec-draft-n-max N`, N
|
||||
* being the tag's own `draft_num_predict` (its params layer: 4 for
|
||||
* `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default
|
||||
* of 4 (`qwen3.8:27b-q8_0` sets none).
|
||||
@@ -83,9 +86,11 @@
|
||||
* adds its prediction layer, whose cache llama-server keeps in a draft context of the
|
||||
* same length in the draft's own cache type, f16 by default (2 bytes per value).
|
||||
* - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
|
||||
* or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP
|
||||
* model (llama.cpp keeps one recurrent state per drafted token for rollback,
|
||||
* `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
|
||||
* or the sliding-window cache of `n_swa + 512` cells: `size_swa` in
|
||||
* src/llama-kv-cache-iswa.cpp, the cache every model with a sliding window gets), held
|
||||
* 1 + N times for an MTP model (llama.cpp keeps one recurrent state per drafted token
|
||||
* for rollback, `need_n_rs_seq()`), + the context checkpoints:
|
||||
* `LLAMA_ARG_CTX_CHECKPOINTS` host copies
|
||||
* (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
|
||||
* state, or the sliding-window cache), + 1.0 GB for the runner's own buffers (calibrated
|
||||
* on a third-party table of a 27B model on llama.cpp: 8.4 GB at 4,096 tokens with
|
||||
@@ -125,26 +130,59 @@
|
||||
* checked. For an `mlx` row it proves only the weights (issue 19417): the figures above are
|
||||
* all that stands between such a row and swap.
|
||||
*
|
||||
* Models set aside on purpose (a weekly review should look at them again):
|
||||
* Models set aside on purpose (a weekly review should look at them again). Every reason was
|
||||
* read again at its source on 2026-11-10:
|
||||
* - `ripple-preview:65b` (Ricursive, Apache-2.0, a dense 65B published on 2026-11-09;
|
||||
* Terminal-Bench 2.1 80.5 and SWE-bench Verified 76.5 on its card; 38.8 GB as a GGUF build,
|
||||
* 45.8 GB as the MLX build a Mac pulls): every build requires runtime 0.40.16, one above
|
||||
* the pin. Computed from its config.json by the llamacpp rule, its GGUF build would be the
|
||||
* first row between the 27B and the 122B (47 GB at 131,072 tokens: 64 and 96 GB Macs, slow
|
||||
* below an Ultra chip). Look at it the day the pin moves.
|
||||
* - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B
|
||||
* would make a 24 GB Mac `ok`): the vendor reports no SWE-bench and no agent loop, and
|
||||
* whether 2-bit weights keep tool calls reliable was never measured.
|
||||
* would make a 24 GB Mac `ok`): the vendor reports no SWE-bench and no agent loop (its card
|
||||
* of 2026-11-04 still lists BFCL v3 only), and whether 2-bit weights keep tool calls
|
||||
* reliable was never measured.
|
||||
* - `ornith-1.5:9b`, `ornith-1.5:35b`, `ornith-1.5:397b` (GGUF only): strong cards, but their
|
||||
* manifests name no renderer and no tool parser, so whether tool calls work through the
|
||||
* runtime is unknown (runtime pull request 17965 is open). `ornith-1.5:122b` has both.
|
||||
* runtime is unknown (runtime pull request 17965 is still open). `ornith-1.5:122b` has
|
||||
* both.
|
||||
* - `ornith:9b`, `ornith:35b` (Ornith 1.0, MIT, with a renderer and a tool parser; SWE-bench
|
||||
* Verified 69.4 and 75.6 on Ornith's card, against 73.4 for Qwen3.6 35B-A3B): runtime issue
|
||||
* 16992, open, reports the 35B's tool calls arriving as text in long agent chains, and the
|
||||
* two share the renderer and the parser. The 9B (5.6 GB, no vision projector) is the one
|
||||
* model that would fit a 16 GB Mac on the estimated budget: 9.3 GB at 65,536 tokens.
|
||||
* - `gpt-oss:20b`: weak in agentic coding (SWE-bench Verified 52.4, Terminal-Bench 2.1 15.2
|
||||
* in NVIDIA's table).
|
||||
* - `flint:40b`: runtime issue 19191 breaks integer tool arguments after the first call.
|
||||
* - `flint:40b`: runtime issue 19191 (open) breaks integer tool arguments after the first
|
||||
* call.
|
||||
* - `lfm2.5:24b-a2b`: its licence allows commercial use only below USD 10M of revenue.
|
||||
* - `laguna-xs-2.1`: the runtime's own page names an open problem with it on macOS.
|
||||
* `laguna-s-2.1`: open runtime issues 17602 and 17632.
|
||||
* - `qwen3.8-flash-next:125b`: the Qwen Community License, not an open licence.
|
||||
* - `qwen3.8-flash-next:125b`: the Qwen Community License, not an open licence, and its
|
||||
* builds require runtime 0.40.16 since 2026-11-09.
|
||||
* - `nemotron-3.5-super:120b` (79.6 GB as the MLX build a Mac pulls): Terminal-Bench 2.1
|
||||
* 60.7 and SWE-bench Verified 71.1 in NVIDIA's table, below `ornith-1.5:122b` at a larger
|
||||
* download, so it adds no choice a 128 GB Mac would take.
|
||||
* - `qwen3.5:4b-q4_k_m` (3.3 GB, it would fit a 16 GB Mac with room): runtime issue 18916,
|
||||
* open, has it answer with thinking only in a long conversation at 65,536 tokens.
|
||||
* - `qwen3-coder-next` (51.7 GB): the generation before Qwen3.6. Its card shows its results
|
||||
* as pictures only, so there is no figure to rank it by against the rows listed here.
|
||||
* - `muse-glimmer:30b-mlx` and the `-dflash` tags of that model: open runtime issues 17656
|
||||
* and 18269 on the MLX build, and no rule here covers the memory of a separate draft
|
||||
* model. Its GGUF build `muse-glimmer:30b-q4_k_m` is listed.
|
||||
* - The plain tags `qwen3.5:9b`, `mimo-v2.6:9b` and `qwen3.8:27b`: a Mac pulls their MLX
|
||||
* builds, whose 8 GiB of prefix snapshots do not fit the Macs a 9B or a 27B is for. Their
|
||||
* GGUF builds are listed instead.
|
||||
* - `qwen3.5:122b-a10b`: replaced by `ornith-1.5:122b`, a fine-tune of it (SWE-bench
|
||||
* Verified 84.5 against 72.0) whose MLX build carries a renderer and a tool parser.
|
||||
*
|
||||
* One open issue touches rows that ARE listed: runtime issue 16992 (above). Its one diagnosed
|
||||
* cause sits in the tool parser itself (it closes a call at the first closing tag, so a tool
|
||||
* argument that quotes a whole tool call is cut off; the fix, runtime pull request 17021, is
|
||||
* open). The names `ornith` and `qwen3.5` map to that one parser in the pinned runtime
|
||||
* (model/parsers/parsers.go), so every Qwen-family row here shares it, the pair we ran
|
||||
* included. The check after a download asks for one plain tool call and cannot see this.
|
||||
*
|
||||
* `tested` has one entry: `qwen3.6:35b-a3b` with OpenCode on an Apple M5 Pro with 48 GB,
|
||||
* 2026-11-06, 67.7 tokens per second. Only a Mac of exactly that chip and memory size is told
|
||||
* "Recommended" and shown that speed in place of the estimate. Every other row was never run
|
||||
@@ -336,7 +374,7 @@ const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
|
||||
*/
|
||||
export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
schema: 1,
|
||||
updated: '2026-11-08',
|
||||
updated: '2026-11-10',
|
||||
// The oldest macOS the model runtime supports.
|
||||
minMacosMajor: 14,
|
||||
// Ordered by rank. The higher rank wins among models that fit and whose speed is graded
|
||||
@@ -344,7 +382,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
// that are left out and why.
|
||||
models: [
|
||||
{
|
||||
// Small Macs (16 to 32 GB, `limited`): the base model of the row below.
|
||||
// Small Macs (18 to 32 GB): the base model of the row below.
|
||||
id: 'qwen3.5:9b-q4_k_m',
|
||||
label: 'Qwen3.5 9B',
|
||||
license: 'Apache-2.0',
|
||||
@@ -365,7 +403,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Small Macs (16 to 32 GB, `limited`). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of
|
||||
// Small Macs (18 to 32 GB). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of
|
||||
// Qwen3.5-9B for code and agents: SWE-bench Verified 61.1 and SWE-bench Pro 44.6,
|
||||
// against 60.0 and 32.0 for its base (Xiaomi's card). Requires exactly the pinned
|
||||
// runtime, 0.40.15.
|
||||
@@ -410,6 +448,33 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
speedPer100Gbps: 13.6,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Meta's 30B for local agents, a dense 27.9B language model with a vision encoder:
|
||||
// SWE-bench Verified 76.0, SWE-bench Pro 51.2, Terminal-Bench 2.1 51.7 (Meta's card). Its
|
||||
// K/V cache is the smallest of the 27B class (2 K/V heads of 128, and three layers in
|
||||
// four look at a window of 2,048 tokens), so it is the first model of that class a 32 GB
|
||||
// Mac can hold: 21.2 GB at 65,536 tokens, tight. Dense, so it is slow below a Max chip.
|
||||
// Its own window ends at 131,072 tokens.
|
||||
id: 'muse-glimmer:30b-q4_k_m',
|
||||
label: 'Muse Glimmer 30B',
|
||||
license: 'Apache-2.0',
|
||||
paramsB: 29.6,
|
||||
runner: 'llamacpp',
|
||||
bytes: 18_157_010_252,
|
||||
// 13 full-attention layers x 2 K/V heads x 128 x 2 x 34/32.
|
||||
heldBytesPerToken: 7_072,
|
||||
// 39 sliding-window layers: (2,048 + 512) cells x 2 x 128 x 2 x 34/32 = 54,312,960, live
|
||||
// and in each context checkpoint.
|
||||
overheadBytes: 54_312_960 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [131072, 65536],
|
||||
rank: 4,
|
||||
// Not measured: the 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on
|
||||
// llama.cpp (16.8 GB each without the projector).
|
||||
speedPer100Gbps: 2.95,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Macs of 36 GB on a slower chip: the model of the tested pair, as a GGUF build whose
|
||||
// memory does not grow after it loaded. With its multi-token-prediction layer.
|
||||
@@ -430,7 +495,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 4,
|
||||
rank: 5,
|
||||
// Measured: 54.5 tokens per second on 400 GB/s (llama.cpp, without prediction).
|
||||
speedPer100Gbps: 13.6,
|
||||
tested: [],
|
||||
@@ -448,7 +513,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
...MEASURED_MEMORY,
|
||||
figures: 'measured',
|
||||
contexts: [131072, 65536],
|
||||
rank: 5,
|
||||
rank: 6,
|
||||
// Measured: 67.8 tokens per second on 400 GB/s with MLX. The one Mac we tested ran it at
|
||||
// 67.7 on 307 GB/s, so on other chips the estimate is on the low side.
|
||||
speedPer100Gbps: 16.9,
|
||||
@@ -473,7 +538,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 6,
|
||||
rank: 7,
|
||||
// Not measured: the 2.95 of the row below scaled by file size.
|
||||
speedPer100Gbps: 1.7,
|
||||
tested: [],
|
||||
@@ -497,7 +562,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 7,
|
||||
rank: 8,
|
||||
// Measured: 11.8 tokens per second on 400 GB/s (llama.cpp, Q4_K_M, without prediction).
|
||||
// This build predicts several tokens at a time, so it is faster than this figure says.
|
||||
speedPer100Gbps: 2.95,
|
||||
@@ -521,7 +586,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: MEASURED_MEMORY.peakExtraBytes,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072],
|
||||
rank: 8,
|
||||
rank: 9,
|
||||
// Not measured: the 16.9 of Qwen3.6 35B-A3B on MLX x 3/10, its active parameters
|
||||
// against these, rounded down.
|
||||
speedPer100Gbps: 5,
|
||||
|
||||
@@ -28,9 +28,10 @@
|
||||
* gated, installed or installable, and accepting the model id.
|
||||
* - Ranking, lexicographic: tested on a Mac like this before not tested (a measurement beats
|
||||
* a formula); ok before unknown before slow; the higher model rank; the higher harness
|
||||
* rank; the larger context. The pick is the first candidate, the alternatives are the next
|
||||
* four MODELS, each at its own best context and harness, so a model is always offered at
|
||||
* the largest context it fits at.
|
||||
* rank; the larger context. The pick is the first candidate, the alternatives are EVERY
|
||||
* other model that fits (`MAX_ALTERNATIVES` is above what the catalogue holds), each at its
|
||||
* own best context and harness, so a model is always offered at the largest context it
|
||||
* fits at.
|
||||
* - Disk: a model whose download would leave less than 10 GiB free is not offered. It is not
|
||||
* dropped in silence: the best-ranked model left out that way is named in `noteDiskDropped`
|
||||
* with what is missing.
|
||||
@@ -109,8 +110,13 @@ export const SLOW_BELOW_TOKENS_PER_SECOND = 10;
|
||||
export const LIMITED_BELOW_PARAMS_B = 12;
|
||||
/** What must stay free on the disk after the model download. */
|
||||
export const DISK_HEADROOM_BYTES = 10 * GIB;
|
||||
/** How many other models are listed under the pick. */
|
||||
export const MAX_ALTERNATIVES = 4;
|
||||
/**
|
||||
* How many other models are listed under the pick, at most. Above what the stock catalogue
|
||||
* holds (9 models: 8 beside the pick), so a Mac is shown every model that fits it and the
|
||||
* install accepts every one of them. Still a bound: test/local-model-catalog.test.ts fails for
|
||||
* a catalogue that outgrows it, so nothing drops out of the list unseen.
|
||||
*/
|
||||
export const MAX_ALTERNATIVES = 11;
|
||||
|
||||
/** A memory size of at least this many GB (as sold) gets 3/4 of its memory as device limit, below it 2/3. */
|
||||
const LARGE_LIMIT_FROM_GB = 36;
|
||||
|
||||
@@ -577,10 +577,10 @@ export interface Recommendation {
|
||||
/** Present exactly when `verdict` is `ok` or `limited`. */
|
||||
pick?: Candidate;
|
||||
/**
|
||||
* The next models that fit and have room on the disk (at most four), best first: tested
|
||||
* before not tested, then ok before unknown before slow speed, then the higher model rank.
|
||||
* Each is at the largest context it fits at, with the best harness that can run it there.
|
||||
* Empty when `verdict` is `none`.
|
||||
* Every other model that fits and has room on the disk (`MAX_ALTERNATIVES` is above what the
|
||||
* catalogue holds), best first: tested before not tested, then ok before unknown before
|
||||
* slow speed, then the higher model rank. Each is at the largest context it fits at, with
|
||||
* the best harness that can run it there. Empty when `verdict` is `none`.
|
||||
*/
|
||||
alternatives: Candidate[];
|
||||
/** `DeviceBudget.budgetBytes`. 0 when no budget is computed: not macOS, or a Mac without Apple silicon. */
|
||||
|
||||
@@ -31,6 +31,7 @@ import {
|
||||
LocalModelCatalogSchema,
|
||||
STOCK_LOCAL_MODEL_CATALOG,
|
||||
} from '../src/config/local-model-catalog.js';
|
||||
import { MAX_ALTERNATIVES } from '../src/local-model/recommend.js';
|
||||
import {
|
||||
RUNTIME_CTX_CHECKPOINTS,
|
||||
RUNTIME_HOST_PROMPT_CACHE_MIB,
|
||||
@@ -63,9 +64,10 @@ const GIB = 1024 * MIB;
|
||||
|
||||
/**
|
||||
* The architecture of every model the stock catalogue lists, read from its config.json on
|
||||
* Hugging Face (2026-11-08): full-attention layers, their K/V heads and head size, and the
|
||||
* layers whose cache does not grow with the context (Gated DeltaNet layers with their value
|
||||
* and key heads, or sliding-window layers with their window). Fine-tunes share their base's.
|
||||
* Hugging Face (2026-11-08; Muse Glimmer on 2026-11-10): full-attention layers, their K/V heads
|
||||
* and head size, and the layers whose cache does not grow with the context (Gated DeltaNet
|
||||
* layers with their value and key heads, or sliding-window layers with their window).
|
||||
* Fine-tunes share their base's.
|
||||
*/
|
||||
const ARCHITECTURES = {
|
||||
'Qwen3.5-9B': { attentionLayers: 8, kvHeads: 4, headDim: 256, linear: { layers: 24, vHeads: 32, kHeads: 16 } },
|
||||
@@ -78,6 +80,8 @@ const ARCHITECTURES = {
|
||||
linear: { layers: 36, vHeads: 64, kHeads: 16 },
|
||||
},
|
||||
'North-Mini-Code-1.0': { attentionLayers: 13, kvHeads: 4, headDim: 128, sliding: { layers: 36, window: 4096 } },
|
||||
// 52 layers, three in four on a sliding window (the GGUF header's `sliding_window_pattern`).
|
||||
'Muse-Glimmer-30B': { attentionLayers: 13, kvHeads: 2, headDim: 128, sliding: { layers: 39, window: 2048 } },
|
||||
} as const;
|
||||
|
||||
type ArchitectureName = keyof typeof ARCHITECTURES;
|
||||
@@ -86,7 +90,8 @@ type Architecture = (typeof ARCHITECTURES)[ArchitectureName];
|
||||
/**
|
||||
* Which architecture each stock row runs, and how many tokens the runtime drafts per step with
|
||||
* its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF
|
||||
* header, read on 2026-11-08 from the first bytes of each model layer), else the tag's own
|
||||
* header, read on 2026-11-08 from the first bytes of each model layer; on 2026-11-10 for the
|
||||
* Muse Glimmer row), else the tag's own
|
||||
* `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none.
|
||||
* Prediction costs a cache for that layer and one rollback copy of the recurrent state per
|
||||
* drafted token (see the catalogue header). A row the catalogue gains must be added here, or the
|
||||
@@ -96,6 +101,8 @@ const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; draftTo
|
||||
'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 },
|
||||
// No prediction layer; the tags with a separate draft model (`-dflash`) are not listed.
|
||||
'muse-glimmer:30b-q4_k_m': { architecture: 'Muse-Glimmer-30B', draftTokens: 0 },
|
||||
// Its params layer sets `draft_num_predict` 2.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 },
|
||||
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 },
|
||||
@@ -183,20 +190,21 @@ describe('the stock catalogue', () => {
|
||||
expect(getLocalModelCatalog()).toBe(first);
|
||||
});
|
||||
|
||||
it('lists eight models, by rank, and none of those set aside in the header', () => {
|
||||
it('lists nine models, by rank, and none of those set aside in the header', () => {
|
||||
expect(models.map((m) => m.id)).toEqual([
|
||||
'qwen3.5:9b-q4_k_m',
|
||||
'mimo-v2.6:9b-q4_k_m',
|
||||
'north-mini-code-1.0:q4_k_m',
|
||||
'muse-glimmer:30b-q4_k_m',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m',
|
||||
'qwen3.6:35b-a3b',
|
||||
'qwen3.8:27b-q8_0',
|
||||
'qwen3.8:27b-mtp-q4_k_m',
|
||||
'ornith-1.5:122b',
|
||||
]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9]);
|
||||
const setAside =
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:122b)|^ornith-1\.5:(9b|35b|397b)/;
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:(9b|35b)|^muse-glimmer:.*(mlx|dflash)/;
|
||||
expect(models.filter((m) => setAside.test(m.id)).map((m) => m.id)).toEqual([]);
|
||||
// The plain 9B and 27B tags are MLX builds on a Mac: their GGUF builds are listed instead.
|
||||
for (const plain of ['qwen3.5:9b', 'mimo-v2.6:9b', 'qwen3.8:27b']) {
|
||||
@@ -209,6 +217,7 @@ describe('the stock catalogue', () => {
|
||||
'qwen3.5:9b-q4_k_m': 'llamacpp',
|
||||
'mimo-v2.6:9b-q4_k_m': 'llamacpp',
|
||||
'north-mini-code-1.0:q4_k_m': 'llamacpp',
|
||||
'muse-glimmer:30b-q4_k_m': 'llamacpp',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 'llamacpp',
|
||||
// Manifest lists: a Mac takes the MLX child.
|
||||
'qwen3.6:35b-a3b': 'mlx',
|
||||
@@ -224,6 +233,8 @@ describe('the stock catalogue', () => {
|
||||
'qwen3.5:9b-q4_k_m': 6_550_825_550,
|
||||
'mimo-v2.6:9b-q4_k_m': 6_550_811_033,
|
||||
'north-mini-code-1.0:q4_k_m': 18_593_967_008,
|
||||
// Fetched on 2026-11-10: 16.76 GB of weights and a 1.40 GB projector.
|
||||
'muse-glimmer:30b-q4_k_m': 18_157_010_252,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 22_621_314_381,
|
||||
// The MLX child of the list, and the total of a real pull on a Mac, byte for byte.
|
||||
'qwen3.6:35b-a3b': 23_105_798_956,
|
||||
@@ -256,6 +267,8 @@ describe('the stock catalogue', () => {
|
||||
'qwen3.5:9b-q4_k_m': [9, 'Apache-2.0'],
|
||||
'mimo-v2.6:9b-q4_k_m': [9, 'MIT'],
|
||||
'north-mini-code-1.0:q4_k_m': [30, 'Apache-2.0'],
|
||||
// The card's total: a 27.9B language model and a vision encoder.
|
||||
'muse-glimmer:30b-q4_k_m': [29.6, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [35, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b': [35, 'Apache-2.0'],
|
||||
'qwen3.8:27b-q8_0': [27.8, 'Apache-2.0'],
|
||||
@@ -281,6 +294,8 @@ describe('the stock catalogue', () => {
|
||||
'mimo-v2.6:9b-q4_k_m': 9.3,
|
||||
// The 35B-A3B's llama.cpp figure: both have 3 billion active parameters.
|
||||
'north-mini-code-1.0:q4_k_m': 13.6,
|
||||
// The dense 27B's figure: the weights of the two are the same size.
|
||||
'muse-glimmer:30b-q4_k_m': 2.95,
|
||||
// 54.5 on 400 GB/s with llama.cpp.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 13.6,
|
||||
// 67.8 on 400 GB/s with MLX.
|
||||
@@ -302,6 +317,13 @@ describe('the stock catalogue', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('holds no more models than one window lists: the pick and MAX_ALTERNATIVES others', () => {
|
||||
// recommend() cuts the list at MAX_ALTERNATIVES. While the catalogue stays under it, a Mac
|
||||
// is shown every model that fits and the install accepts every one of them. A catalogue
|
||||
// that outgrows the bound must fail here, so the bound is raised (or kept) on purpose.
|
||||
expect(models.length - 1).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
});
|
||||
|
||||
it('names the oldest macOS and its schema version', () => {
|
||||
expect(STOCK_LOCAL_MODEL_CATALOG.minMacosMajor).toBe(14);
|
||||
expect(STOCK_LOCAL_MODEL_CATALOG.schema).toBe(1);
|
||||
@@ -351,6 +373,8 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0],
|
||||
// The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB.
|
||||
'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0],
|
||||
// The same rule: 39 layers x 2,560 cells x 544 bytes = 54,312,960 per copy.
|
||||
'muse-glimmer:30b-q4_k_m': [7_072, 2_562_558_464, 0],
|
||||
// State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
|
||||
// layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0],
|
||||
|
||||
@@ -12,9 +12,9 @@
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { getLocalModelCatalog } from '../src/config/local-model-catalog.js';
|
||||
import { getLocalModelCatalog, LocalModelCatalogSchema } from '../src/config/local-model-catalog.js';
|
||||
import { formatGb, memClassGb } from '../src/local-model/format.js';
|
||||
import { modelFitsBudget, recommend } from '../src/local-model/recommend.js';
|
||||
import { MAX_ALTERNATIVES, modelFitsBudget, recommend } from '../src/local-model/recommend.js';
|
||||
import type { HardwareFacts, HarnessFact } from '../src/types.js';
|
||||
|
||||
/** A Mac with Apple silicon and `gb` GB of memory as sold. */
|
||||
@@ -72,18 +72,46 @@ describe('modelFitsBudget', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('does not depend on the ranking: a model that fits and is listed nowhere still fits', () => {
|
||||
// On a 128 GB Mac more models fit than the recommendation lists.
|
||||
const hw = mac(128, 'Apple M5 Max');
|
||||
it('does not depend on what the recommendation lists: a model left out for disk space still fits', () => {
|
||||
// The recommendation lists every model that fits the memory, unless its download would
|
||||
// leave under 10 GiB on the disk. With 40 GB free on a 128 GB Mac that leaves out the two
|
||||
// largest downloads, which fit the memory all the same.
|
||||
const hw = { ...mac(128, 'Apple M5 Max'), freeDiskBytes: 40_000_000_000 };
|
||||
const offered = recommend(hw, catalog, harnessFacts());
|
||||
const listed = new Set([offered.pick, ...offered.alternatives].map((row) => row?.modelId));
|
||||
const unlisted = catalog.models.filter((model) => !listed.has(model.id));
|
||||
expect(unlisted.length).toBeGreaterThan(0);
|
||||
expect(unlisted.map((model) => model.id)).toEqual(['qwen3.8:27b-q8_0', 'ornith-1.5:122b']);
|
||||
for (const model of unlisted) {
|
||||
expect(modelFitsBudget(hw, catalog, model.id, model.contexts.at(-1) as number), model.id).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('does not depend on the ranking: a model cut off by the length of the list still fits', () => {
|
||||
// A catalogue longer than the list: the stock rows plus six more copies of its smallest.
|
||||
const smallest = catalog.models[0];
|
||||
const long = LocalModelCatalogSchema.parse({
|
||||
...structuredClone(catalog),
|
||||
models: [
|
||||
...structuredClone(catalog.models),
|
||||
...Array.from({ length: 6 }, (_, index) => ({
|
||||
...structuredClone(smallest),
|
||||
id: `extra-${index + 1}:9b`,
|
||||
rank: -1 - index,
|
||||
})),
|
||||
],
|
||||
});
|
||||
expect(long.models.length).toBeGreaterThan(1 + MAX_ALTERNATIVES);
|
||||
const hw = mac(512, 'Apple M5 Max');
|
||||
const offered = recommend(hw, long, harnessFacts());
|
||||
const listed = new Set([offered.pick, ...offered.alternatives].map((row) => row?.modelId));
|
||||
expect(listed.size).toBe(1 + MAX_ALTERNATIVES);
|
||||
const unlisted = long.models.filter((model) => !listed.has(model.id));
|
||||
expect(unlisted.length).toBe(long.models.length - 1 - MAX_ALTERNATIVES);
|
||||
for (const model of unlisted) {
|
||||
expect(modelFitsBudget(hw, long, model.id, model.contexts.at(-1) as number), model.id).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('rules out a model that is too large, and one the catalogue does not have', () => {
|
||||
expect(modelFitsBudget(mac(48), catalog, 'ornith-1.5:122b', 131072)).toBe(false);
|
||||
expect(modelFitsBudget(mac(48), catalog, 'no-such:1b', 65536)).toBe(false);
|
||||
|
||||
@@ -489,6 +489,8 @@ describe('needBytes', () => {
|
||||
'qwen3.5:9b-q4_k_m': 11_380_487_246,
|
||||
'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
|
||||
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
|
||||
// At its own window of 131072 tokens: 18.2 GB + 0.9 GB of K/V cache + 2.6 GB.
|
||||
'muse-glimmer:30b-q4_k_m': 21_646_509_900,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
|
||||
'qwen3.6:35b-a3b': 35_955_462_956,
|
||||
'qwen3.8:27b-q8_0': 44_292_142_594,
|
||||
@@ -544,6 +546,9 @@ describe('the fit grade', () => {
|
||||
// ---------- 2. The two real Macs ----------
|
||||
|
||||
describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
/** The same Mac without the probe: 48 GiB and the chip are all the fit and the speed depend on. */
|
||||
const hw48 = mac(48, 'Apple M5 Pro');
|
||||
|
||||
/**
|
||||
* Expected: Qwen3.6 35B-A3B at 131,072 tokens with OpenCode, a TIGHT fit, 67.7 tokens per
|
||||
* second, "Recommended for this Mac".
|
||||
@@ -642,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
expect(result.notes).toEqual([]);
|
||||
});
|
||||
|
||||
it('lists four other models that fit, each at the largest context it fits at, the fast ones first', async () => {
|
||||
it('lists every other model that fits, each at the largest context it fits at, the fast ones first', async () => {
|
||||
const result = recommend(await macbook(), STOCK, ALL_INSTALLED);
|
||||
expect(
|
||||
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
|
||||
@@ -652,15 +657,19 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
['qwen3.5:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
// The dense models compute to 2.95 x 3.07 = 9.1 tokens per second on this chip (5.2 in
|
||||
// 8 bits): slow, so they come after every fast model, in the order of their rank.
|
||||
['qwen3.8:27b-mtp-q4_k_m', 262144, 'tight', 'slow', 9.1, 'tight'],
|
||||
['qwen3.8:27b-q8_0', 65536, 'tight', 'slow', 5.2, 'tight'],
|
||||
['muse-glimmer:30b-q4_k_m', 131072, 'good', 'slow', 9.1, 'slow'],
|
||||
]);
|
||||
// Only the tested row rests on a measurement of that model.
|
||||
expect(result.alternatives.map((c) => c.needSource)).toEqual(['estimated', 'estimated', 'estimated', 'estimated']);
|
||||
// The dense 27B fits too (32.1 GB at 262144) but computes to 2.95 x 3.07 = 9.1 tokens per
|
||||
// second here: slow, so it ranks after four fast models and is not listed.
|
||||
expect(modelFitsBudget(await macbook(), STOCK, 'qwen3.8:27b-mtp-q4_k_m', 262144)).toBe(true);
|
||||
expect(order(result).some((entry) => entry.startsWith('qwen3.8:27b'))).toBe(false);
|
||||
expect(new Set(result.alternatives.map((c) => c.needSource))).toEqual(new Set(['estimated']));
|
||||
// Every model of the catalogue that fits is on the list: nothing is cut off after four.
|
||||
const fitting = STOCK.models.filter((m) => m.contexts.some((ctx) => modelFitsBudget(hw48, STOCK, m.id, ctx)));
|
||||
expect(rows(result).map((c) => c.modelId).sort()).toEqual(fitting.map((m) => m.id).sort());
|
||||
// The 122B fits at no context.
|
||||
expect(order(result).some((entry) => entry.startsWith('ornith'))).toBe(false);
|
||||
expect(order(result).some((entry) => entry.startsWith('ornith-1.5'))).toBe(false);
|
||||
});
|
||||
|
||||
it('would get the same model as "Best fit", with the estimated speed, had nobody tested it', async () => {
|
||||
@@ -692,7 +701,10 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
['qwen3.8:27b-mtp-q4_k_m', 'unknown'],
|
||||
['qwen3.8:27b-q8_0', 'unknown'],
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 'unknown'],
|
||||
['muse-glimmer:30b-q4_k_m', 'unknown'],
|
||||
['north-mini-code-1.0:q4_k_m', 'unknown'],
|
||||
['mimo-v2.6:9b-q4_k_m', 'unknown'],
|
||||
['qwen3.5:9b-q4_k_m', 'unknown'],
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -803,10 +815,13 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
[24, 'Apple M3', 'limited mimo-v2.6:9b-q4_k_m@131072 good slow'],
|
||||
[24, 'Apple M4', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[24, 'Apple M4 Pro', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// North Mini Code needs 23.2 GB at 65536 tokens: 1.06 of the 21.8 GB estimate. A runtime
|
||||
// that measures about 3/4 of the memory admits it (see below).
|
||||
[32, 'Apple M4', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// Muse Glimmer 30B fits at 65536 tokens: 21.2 GB, 0.970 of the 21.8 GB estimate. So a 32 GB
|
||||
// Mac is no small-model Mac (`ok`). The model is dense: on a base chip it computes to 3.5
|
||||
// tokens per second and the fast 9B stays the pick, on a Max chip (11.8) it is the pick.
|
||||
// North Mini Code needs 23.2 GB at 65536 tokens, 1.06 of the estimate: a runtime that
|
||||
// measures about 3/4 of the memory admits it (see below).
|
||||
[32, 'Apple M4', 'ok mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[32, 'Apple M1 Max', 'ok muse-glimmer:30b-q4_k_m@65536 tight ok'],
|
||||
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
|
||||
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
|
||||
// at 131072 tokens, 0.966 and 0.971 of the budget.
|
||||
@@ -850,14 +865,21 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
expect(smallest / result.budgetBytes).toBeGreaterThan(4);
|
||||
});
|
||||
|
||||
it('gives no model to 8 and 16 GB, a small model to every Mac from 18 to 32 GB, and every larger one a model of 27 billion or more', () => {
|
||||
it('gives no model to 8 and 16 GB, a small model to every Mac of 18 and 24 GB, and every Mac from 36 GB a model of 27 billion or more', () => {
|
||||
for (const hw of SWEEP) {
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
const label = `${hw.memClassGb} GB ${hw.chip}`;
|
||||
if (hw.memClassGb <= 16) expect(result.reason, label).toBe('memory');
|
||||
else if (hw.memClassGb <= 32) {
|
||||
else if (hw.memClassGb <= 24) {
|
||||
expect(result.verdict, label).toBe('limited');
|
||||
expect(result.pick?.paramsB, label).toBe(9);
|
||||
} else if (hw.memClassGb === 32) {
|
||||
// A model of 30 billion fits, so the Mac is not `limited`. It is dense: the pick only
|
||||
// where it is not slow (a chip this file knows is fast, or one it knows nothing about).
|
||||
expect(result.verdict, label).toBe('ok');
|
||||
expect(rows(result).map((c) => c.modelId), label).toContain('muse-glimmer:30b-q4_k_m');
|
||||
expect(result.pick?.paramsB, label).toBe(hw.chip === 'Apple M4' ? 9 : 29.6);
|
||||
if (hw.chip === 'Apple M4') expect(result.notes[0], label).toEqual({ text: LOCAL_MODEL_LABELS.noteSmallPick });
|
||||
} else {
|
||||
expect(result.verdict, label).toBe('ok');
|
||||
expect(result.pick?.paramsB, label).toBeGreaterThanOrEqual(27);
|
||||
@@ -1620,11 +1642,13 @@ describe('the ranking', () => {
|
||||
expect(result.pick).toMatchObject({ context: 1024, fit: 'tight', budgetShare: 0.92 });
|
||||
});
|
||||
|
||||
it('lists each model once, never the pick again, and at most four alternatives', () => {
|
||||
expect(MAX_ALTERNATIVES).toBe(4);
|
||||
// Seven models fit a 128 GB Mac; the pick plus four are shown.
|
||||
it('lists each model once, never the pick again, and every model that fits', () => {
|
||||
// The bound is above what either catalogue holds, so nothing that fits is cut off.
|
||||
expect(MAX_ALTERNATIVES).toBe(11);
|
||||
for (const [, catalog] of BOTH) expect(catalog.models.length - 1).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
// Seven models fit a 128 GB Mac: the pick and all six others are shown.
|
||||
const result = recommend(mac(128, 'Apple M5 Ultra'), RULES, ALL_INSTALLED);
|
||||
expect(result.alternatives).toHaveLength(4);
|
||||
expect(result.alternatives).toHaveLength(6);
|
||||
const ids = rows(result).map((c) => c.modelId);
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
for (const [, catalog] of BOTH) {
|
||||
@@ -1633,9 +1657,26 @@ describe('the ranking', () => {
|
||||
const sweptIds = rows(swept).map((c) => c.modelId);
|
||||
expect(new Set(sweptIds).size).toBe(sweptIds.length);
|
||||
expect(swept.alternatives.length).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
// Every model that fits at a context a harness works at is on the list (the sweep's
|
||||
// Macs have 500 GB of free disk, so none is left out for room).
|
||||
const fitting = catalog.models.filter((m) =>
|
||||
m.contexts.some((ctx) => ctx >= 65536 && modelFitsBudget(hw, catalog, m.id, ctx))
|
||||
);
|
||||
expect([...sweptIds].sort(), `${hw.memClassGb} GB ${hw.chip}`).toEqual(fitting.map((m) => m.id).sort());
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('still cuts a list longer than the bound, and the install can only take what is listed', () => {
|
||||
// Fourteen models that all fit: the pick and MAX_ALTERNATIVES others, best rank first.
|
||||
const many = catalogOf(
|
||||
Array.from({ length: 14 }, (_, index) => modelNeeding(`m${index + 1}:x`, GIB, { rank: index + 1 }))
|
||||
);
|
||||
const result = recommend(HUNDRED_GIB_MAC, many, [fact('alpha')]);
|
||||
expect(rows(result)).toHaveLength(1 + MAX_ALTERNATIVES);
|
||||
expect(rows(result)[0].modelId).toBe('m14:x');
|
||||
expect(rows(result).at(-1)?.modelId).toBe('m3:x');
|
||||
});
|
||||
});
|
||||
|
||||
describe('harnesses', () => {
|
||||
@@ -2064,6 +2105,8 @@ describe('what each row of the chooser carries', () => {
|
||||
'qwen3.5:9b-q8_0': 'estimated',
|
||||
'qwen3.5:9b': 'estimated',
|
||||
'gpt-oss:20b': 'estimated',
|
||||
// Slow on this chip, so it was the sixth row: listed since the list holds every model.
|
||||
'qwen3.8:27b-q8_0': 'estimated',
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -2075,10 +2118,15 @@ describe('the measured budget, end to end', () => {
|
||||
const measured = recommend(hw, STOCK, ALL_INSTALLED, { measuredTotalBytes: MACBOOK_MEASURED_TOTAL });
|
||||
expect([estimated.budgetSource, estimated.budgetGb]).toEqual(['estimated', '37.6']);
|
||||
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
|
||||
// The same rows in the same order; only the share of each changes.
|
||||
expect(order(measured)).toEqual(order(estimated));
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
|
||||
// The same models in the same order. The share of each changes, and one model steps up a
|
||||
// context: the 27B in 8 bits fits 65536 tokens under the estimate (0.975) and 131072 under
|
||||
// what the runtime reported (0.976).
|
||||
expect(rows(measured).map((c) => c.modelId)).toEqual(rows(estimated).map((c) => c.modelId));
|
||||
const stepped = order(estimated).filter((entry, index) => order(measured)[index] !== entry);
|
||||
expect(stepped).toEqual(['qwen3.8:27b-q8_0@65536']);
|
||||
expect(order(measured)).toContain('qwen3.8:27b-q8_0@131072');
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303, 0.853, 0.975, 0.576]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283, 0.798, 0.976, 0.539]);
|
||||
for (const candidate of rows(measured)) {
|
||||
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user