mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-13 10:29:32 +01:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Codeman maintainer
|
4423949923 | ||
|
Codeman maintainer
|
5396e33b48 | ||
|
Codeman maintainer
|
d25cc08606 | ||
|
Codeman maintainer
|
9608a17d99 |
No files matched your search
@@ -192,7 +192,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
|
||||
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for two rows, the MLX builds of Qwen3.6 35B-A3B and Muse Glimmer 30B, and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ The answer lists EVERY model that fits, and the install accepts only a listed row: `MAX_ALTERNATIVES` is a bound above the catalogue's size (`test/local-model-catalog.test.ts` fails for a catalogue that outgrows it), never a way to shorten the list. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
|
||||
**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated
|
||||
|
||||
|
||||
@@ -1766,7 +1766,7 @@ A probe that fails is an answer, not an error: a server that is not on macOS, or
|
||||
- `verdict`: `ok`, `limited` or `none`. `limited` is a statement about the Mac: no model with 12 billion parameters or more fits its memory, so what it can run handles small edits and questions and does not replace a cloud model. It is judged on every model that fits, before the free disk space is looked at and whatever the pick is. With `ok` the pick can still be a small model (a larger one may be slow, have no speed figure or have no room on the disk); `notes` then says so.
|
||||
- `reason`, present only with `none`: `not-macos`, `intel`, `memory`, `disk`, `macos` (too old) or `no-harness` (no agent CLI that can be connected to a local model is installed or installable). `memory` has three sentences: nothing fits at all (slot `memory`, the size the Mac is sold as), a model fits but only below the context length an agent CLI needs (slots `budget` and `need`, so a Mac that misses by a little reads as that), and the memory size could not be read (no slot).
|
||||
- `headline` and `notes[]`: `{ text, vars? }`. `text` is an English sentence that may hold `{placeholder}` slots, and `vars` carries the slot values as text. Notes, in order: the pick is a small model although larger ones fit (verdict `ok` only); a model that fits the memory was left out because its download would not leave 10 GiB free on the disk (slots `model` and `missing`: the best-ranked such model and the GB to free up); nobody tested the pick on a Mac like this one.
|
||||
- `pick` (absent with `none`) and `alternatives[]` (at most four, best first; empty with `none`), each one model at one context length with one agent CLI. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `pick` (absent with `none`) and `alternatives[]` (every other model of the catalogue that fits, best first; empty with `none`), each one model at one context length with one agent CLI. An install is accepted only for a row of this list. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing):
|
||||
- `modelId` (`name:tag`), `modelLabel`, `license`, `paramsB` (parameters in billions), `context` (tokens);
|
||||
- `cli`, `cliLabel`, `cliInstalled` (`false`: the CLI is not on this machine yet but can be installed without administrator rights), `slowFirstTurn?`;
|
||||
- `needBytes` and `needGb` (memory the model needs at that context, at its PEAK: the weights, what stays held per token of context, what the runner holds beside both, and what it takes for a moment while a long prompt is processed), `needSource` (`measured`: the figures were measured for this model on a Mac; `estimated`: they are another model's), `downloadBytes` and `downloadGb`, `budgetShare` (need divided by `budgetBytes`, rounded to three decimals);
|
||||
|
||||
+49
-26
@@ -17,28 +17,29 @@ in [architecture-invariants.md](architecture-invariants.md#local-model-install).
|
||||
## Which model fits which Mac
|
||||
|
||||
The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that
|
||||
fits this Mac's memory at a context a coding agent can work with, and up to four more. What
|
||||
the catalogue answers today, before the runtime measured anything (the budget is then an
|
||||
estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the
|
||||
memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
fits this Mac's memory at a context a coding agent can work with, and under it every other
|
||||
model of the catalogue that fits (eleven models today: a 16 GB Mac sees one of them, a 512 GB
|
||||
Mac all eleven). What the catalogue answers today, before the runtime measured anything (the
|
||||
budget is then an estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and
|
||||
never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
| Memory | Budget | What it gets |
|
||||
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.3 GB at 65,536 tokens |
|
||||
| 16 GB | 10.4 GB | A small model (`limited`): Ornith 1.0 9B at 65,536 tokens, 9.3 GB, tight (90%). Its build has no vision projector, which makes it 0.9 GB smaller than the other two 9B builds: MiMo V2.6 9B needs 10.2 GB there, 98.7% of the budget, just past the 98% a tight fit may reach. Once the runtime measures what a 16 GB Mac mini measured (10.7 GB), all three fit and MiMo is the pick. No 9B was run on such a Mac by us: only a run settles it |
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Ornith 1.0 9B (10.5 GB) and Qwen3.5 9B are the alternatives |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip. Run once on a Mac (an M5 Pro with 48 GB, 2026-11-12): this GGUF build wrote 15 tokens per second, and an OpenCode worker given a one-line task showed no answer after 7 minutes. Its MLX build, which finished an OpenCode turn on a one-line prompt in 45 seconds, needs 36.1 GB and is offered from 48 GB |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). A session of many prompts on that Mac (a 95,000-token prompt, then twelve separate ones of 15,000 tokens, measured on 2026-11-12) ended at 33.4 GB of the 40.2 GB its runtime measures, still growing by 0.5 GiB a prompt: MLX may keep up to 8 GiB of earlier prompts, which brings this pair to the edge of the budget. Muse Glimmer 30B as its MLX build is offered here too, at 65,536 tokens (36.1 GB, tight) |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight |
|
||||
| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens |
|
||||
|
||||
"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not
|
||||
offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks
|
||||
after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is
|
||||
after the fast ones, which is why the chip changes the pick at 32 to 96 GB. Every model is
|
||||
offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own
|
||||
prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default
|
||||
file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table.
|
||||
@@ -70,15 +71,21 @@ names its runner (`runner`) and its figures follow that runner's rule:
|
||||
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
|
||||
state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A
|
||||
Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit
|
||||
beside a useful model, so the catalogue lists GGUF builds there.
|
||||
beside a useful model, so the catalogue lists GGUF builds there. For Muse Glimmer the MLX
|
||||
runner was measured holding 38,000 to 45,000 bytes per token, against the 7,072 its GGUF
|
||||
build computes to: close to every layer's cache at full length, so it evidently does not
|
||||
trim the sliding-window layers the way llama.cpp does.
|
||||
|
||||
Where every figure comes from:
|
||||
|
||||
| Model | Runner | Memory figures | Speed (tokens per second per 100 GB/s) |
|
||||
| ---------------------------- | --------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------- |
|
||||
| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max |
|
||||
| `ornith:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B; its config and its GGUF header say so) | 9.3: the same architecture and weights size |
|
||||
| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size |
|
||||
| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters |
|
||||
| `muse-glimmer:30b-q4_k_m` | llama.cpp | computed from its config: 13 full-attention and 39 sliding-window layers | 2.95: the dense 27B's, whose weights are the same size |
|
||||
| `muse-glimmer:30b-mlx` | MLX | **measured** on an Apple M5 Pro with 48 GB, the 8 GiB of prefix snapshots counted | 5.4: measured beside the 35B-A3B on MLX, 0.32 of its 16.9 |
|
||||
| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config, prediction layer included | 13.6: a published run on an M1 Max |
|
||||
| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac |
|
||||
| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config, prediction layer included | 1.7: the row below scaled by file size |
|
||||
@@ -86,13 +93,22 @@ Where every figure comes from:
|
||||
| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters |
|
||||
|
||||
Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on
|
||||
2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a
|
||||
calculation. For a llama.cpp build the check after the download catches a load that was
|
||||
2026-11-08 (the Ornith 1.0 and Muse Glimmer rows on 2026-11-10, Muse Glimmer's MLX build on
|
||||
2026-11-12). Two rows carry memory figures measured on a Mac, both on an Apple M5 Pro with
|
||||
48 GB; every other answer in the table above is a calculation. Three more rows were installed
|
||||
and loaded on that Mac on 2026-11-12 (Qwen3.5 9B, Ornith 1.0 9B and Muse Glimmer as a GGUF
|
||||
build): that checked their download size and that they load on the GPU, not their memory
|
||||
figures. For a llama.cpp build the check after the download catches a load that was
|
||||
judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt
|
||||
cache, which are counted, not checked); for an MLX build it catches neither (see "The check"
|
||||
below). The measured row's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on
|
||||
purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why.
|
||||
below). The 35B-A3B's figures come from single long prompts, not a session of many
|
||||
requests, in which MLX may keep up to 8 GiB more: such a session was measured on it later
|
||||
(the 48 GB row above) and its figures were left as they are. Muse Glimmer's MLX row counts
|
||||
the 8 GiB. The catalogue header lists the models left
|
||||
out on purpose (no evidence of working tool calls, a restrictive licence, a known bug, a
|
||||
build that needs a newer runtime than the pinned one) and why. The strongest of those today
|
||||
is Ripple 65B Preview, published on 2026-11-09: every build of it requires Ollama 0.40.16,
|
||||
one above the pin.
|
||||
|
||||
A new row needs its model's architecture (attention layers, K/V heads, head size, and its
|
||||
linear-attention or sliding-window layers, from its `config.json`) in the table of
|
||||
@@ -351,10 +367,17 @@ or `python3`.
|
||||
a session uses the model: close those sessions first.
|
||||
- **The runtime does not enforce the context length.** A prompt longer than the context was
|
||||
accepted in testing. Do not rely on the runtime to stop one.
|
||||
- **Most memory figures are calculated, not measured.** One model was run on one Mac; the
|
||||
others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
- **Most memory figures are calculated, not measured.** Two models were measured on one Mac;
|
||||
the others are computed from their architecture, and the runtime settings they assume (an 8-bit
|
||||
K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed
|
||||
on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on.
|
||||
- **A tool call that quotes a tool call can be cut off.** The pinned runtime reads the tool
|
||||
calls of every Qwen-family model in the catalogue (the Ornith, MiMo and Qwen rows, the pair
|
||||
we ran included) with one parser, and that parser ends a call at the first closing tag it
|
||||
sees. An argument that itself contains a whole tool call, such as a prompt handed to a
|
||||
sub-agent that shows one, arrives cut off and the rest as text (runtime issue 16992, filed
|
||||
on `ornith:35b`; the fix, runtime pull request 17021, is open). The check after a download
|
||||
asks for one plain tool call and does not see this.
|
||||
- **The model stays in memory for ten minutes** after the last request, then the runtime
|
||||
unloads it.
|
||||
|
||||
|
||||
@@ -24,8 +24,10 @@
|
||||
* throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses,
|
||||
* so a bad row fails the gate and never reaches the route.
|
||||
*
|
||||
* Where each figure of the stock rows comes from, and what was checked (2026-11-08). [V] read
|
||||
* at the source that day, [M] measured on a Mac, [C] computed from [V] inputs:
|
||||
* Where each figure of the stock rows comes from, and what was checked (2026-11-08; the two rows
|
||||
* added on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m`, were read that day;
|
||||
* `muse-glimmer:30b-mlx` was read and measured on 2026-11-12).
|
||||
* [V] read at the source that day, [M] measured on a Mac, [C] computed from [V] inputs:
|
||||
*
|
||||
* | Figure | Source | Checked |
|
||||
* |----------------------|-----------------------------------------------------------|--------------------------------------|
|
||||
@@ -74,8 +76,10 @@
|
||||
* src/local-model/runtime.ts, q8_0: 34 bytes per 32 values), and does not grow after that.
|
||||
* "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
|
||||
* its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
|
||||
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
|
||||
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N
|
||||
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not; read
|
||||
* the same way on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m` do not
|
||||
* either). For such a model the runtime always passes
|
||||
* `--spec-type draft-mtp --spec-draft-n-max N`, N
|
||||
* being the tag's own `draft_num_predict` (its params layer: 4 for
|
||||
* `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default
|
||||
* of 4 (`qwen3.8:27b-q8_0` sets none).
|
||||
@@ -83,9 +87,11 @@
|
||||
* adds its prediction layer, whose cache llama-server keeps in a draft context of the
|
||||
* same length in the draft's own cache type, f16 by default (2 bytes per value).
|
||||
* - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
|
||||
* or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP
|
||||
* model (llama.cpp keeps one recurrent state per drafted token for rollback,
|
||||
* `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
|
||||
* or the sliding-window cache of `n_swa + 512` cells: `size_swa` in
|
||||
* src/llama-kv-cache-iswa.cpp, the cache every model with a sliding window gets), held
|
||||
* 1 + N times for an MTP model (llama.cpp keeps one recurrent state per drafted token
|
||||
* for rollback, `need_n_rs_seq()`), + the context checkpoints:
|
||||
* `LLAMA_ARG_CTX_CHECKPOINTS` host copies
|
||||
* (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
|
||||
* state, or the sliding-window cache), + 1.0 GB for the runner's own buffers (calibrated
|
||||
* on a third-party table of a 27B model on llama.cpp: 8.4 GB at 4,096 tokens with
|
||||
@@ -110,6 +116,26 @@
|
||||
* of many requests, in which the prefix snapshots may add up to 8 GiB. Its figures stay
|
||||
* the measured ones (with 8 GiB on top it would not be offered on the Mac it ran on),
|
||||
* and a 30-request session on it is the first thing to measure next.
|
||||
* Measured again on 2026-11-12 on the same Mac, from the runner's own log line
|
||||
* (`msg=memory peak=... held=...` after every request): 39,600 bytes per token between
|
||||
* prompts of 30,000 and 94,500 tokens, and the session this row owed, twelve separate
|
||||
* prompts of 15,000 tokens, added 0.5 GiB of prefix snapshots each (5.6 GiB after
|
||||
* twelve, still growing): the whole run ended at 33.4 GB of the 40.2 GB that Mac's
|
||||
* runtime measures. The row's figures are unchanged by it.
|
||||
* - `muse-glimmer:30b-mlx`: MEASURED the same way on that Mac (2026-11-12, runtime
|
||||
* 0.40.15). What stayed held grew by 37,900 to 45,100 bytes per token between prompts
|
||||
* of 30,000, 60,000 and 94,600 tokens. The 13 full-attention layers alone compute to
|
||||
* 13,312 in bf16 and all 52 layers at full length to 53,248, so the runner evidently
|
||||
* does not trim the 39 sliding-window layers to their window (an inference from the
|
||||
* figure, not read in its source). The row takes the upper end. It counts the whole
|
||||
* 8 GiB prefix budget, because the same run filled it: twelve separate prompts of
|
||||
* 15,000 tokens added 0.53 GiB each until the runner dropped the oldest. The process
|
||||
* (`footprint`) stood 0.2 GB above the runner's own count at the end. Its lifetime
|
||||
* peak stood 3.0 to 3.3 GB above that, which is a FLOOR for the peak of a long prompt:
|
||||
* a lifetime maximum taken at the end of the run hides a transient of up to 9 GB
|
||||
* during the earlier, longer prompts. So the row takes the first measured row's 5 GB.
|
||||
* The tag carries a draft model; its tensors are inside `bytes`, and what it holds is
|
||||
* inside these figures.
|
||||
* - every other `mlx` row: held = the K/V cache in bf16 + one recurrent snapshot per
|
||||
* 8,192 tokens, computed from the config, x 37,000 / 28,340: the ratio of what the
|
||||
* measured row really held to the same computation for it (same runner, same
|
||||
@@ -125,26 +151,65 @@
|
||||
* checked. For an `mlx` row it proves only the weights (issue 19417): the figures above are
|
||||
* all that stands between such a row and swap.
|
||||
*
|
||||
* Models set aside on purpose (a weekly review should look at them again):
|
||||
* Models set aside on purpose (a weekly review should look at them again). Every reason was
|
||||
* read again at its source on 2026-11-10:
|
||||
* - `ripple-preview:65b` (Ricursive, Apache-2.0, a dense 65B published on 2026-11-09;
|
||||
* Terminal-Bench 2.1 80.5 and SWE-bench Verified 76.5 on its card; 38.8 GB as a GGUF build,
|
||||
* 45.8 GB as the MLX build a Mac pulls): every build requires runtime 0.40.16, one above
|
||||
* the pin. Computed from its config.json by the llamacpp rule, its GGUF build would be the
|
||||
* first row between the 27B and the 122B (47 GB at 131,072 tokens: 64 and 96 GB Macs, slow
|
||||
* below an Ultra chip). Look at it the day the pin moves.
|
||||
* - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B
|
||||
* would make a 24 GB Mac `ok`): the vendor reports no SWE-bench and no agent loop, and
|
||||
* whether 2-bit weights keep tool calls reliable was never measured.
|
||||
* would make a 24 GB Mac `ok`): the vendor reports no SWE-bench and no agent loop (its card
|
||||
* of 2026-11-04 still lists BFCL v3 only), and whether 2-bit weights keep tool calls
|
||||
* reliable was never measured.
|
||||
* - `ornith-1.5:9b`, `ornith-1.5:35b`, `ornith-1.5:397b` (GGUF only): strong cards, but their
|
||||
* manifests name no renderer and no tool parser, so whether tool calls work through the
|
||||
* runtime is unknown (runtime pull request 17965 is open). `ornith-1.5:122b` has both.
|
||||
* runtime is unknown (runtime pull request 17965 is still open). `ornith-1.5:122b` has
|
||||
* both.
|
||||
* - `ornith:35b` (Ornith 1.0 35B-A3B, MIT, with a renderer and a tool parser; SWE-bench
|
||||
* Verified 75.6 against 73.4 for Qwen3.6 35B-A3B on Ornith's card): runtime issue 16992,
|
||||
* open, reports its tool calls arriving as text in long agent chains, and its size is
|
||||
* served by the pair we ran. Its 9B sibling `ornith:9b-q4_k_m` is listed: see that row
|
||||
* for why.
|
||||
* - `gpt-oss:20b`: weak in agentic coding (SWE-bench Verified 52.4, Terminal-Bench 2.1 15.2
|
||||
* in NVIDIA's table).
|
||||
* - `flint:40b`: runtime issue 19191 breaks integer tool arguments after the first call.
|
||||
* - `flint:40b`: runtime issue 19191 (open) breaks integer tool arguments after the first
|
||||
* call.
|
||||
* - `lfm2.5:24b-a2b`: its licence allows commercial use only below USD 10M of revenue.
|
||||
* - `laguna-xs-2.1`: the runtime's own page names an open problem with it on macOS.
|
||||
* `laguna-s-2.1`: open runtime issues 17602 and 17632.
|
||||
* - `qwen3.8-flash-next:125b`: the Qwen Community License, not an open licence.
|
||||
* - `qwen3.8-flash-next:125b`: the Qwen Community License, not an open licence, and its
|
||||
* builds require runtime 0.40.16 since 2026-11-09.
|
||||
* - `nemotron-3.5-super:120b` (79.6 GB as the MLX build a Mac pulls): Terminal-Bench 2.1
|
||||
* 60.7 and SWE-bench Verified 71.1 in NVIDIA's table, below `ornith-1.5:122b` at a larger
|
||||
* download, so it adds no choice a 128 GB Mac would take.
|
||||
* - `qwen3.5:4b-q4_k_m` (3.3 GB, it would fit a 16 GB Mac with room): runtime issue 18916,
|
||||
* open, has it answer with thinking only in a long conversation at 65,536 tokens.
|
||||
* - `qwen3-coder-next` (51.7 GB): the generation before Qwen3.6. Its card shows its results
|
||||
* as pictures only, so there is no figure to rank it by against the rows listed here.
|
||||
* - The `-dflash` tags of Muse Glimmer (`30b-q4_k_m-dflash`, `30b-nvfp4-dflash`): a separate
|
||||
* draft model whose memory no rule here covers. `muse-glimmer:30b-mlx`, which is built
|
||||
* from the second one's layers, IS listed since 2026-11-12, with measured figures. Its
|
||||
* two open runtime issues were read again that day: 17656 takes its `nvfp4` weights for
|
||||
* a build the MLX engine cannot accelerate (on the Mac it ran on the runtime logged
|
||||
* "MLX engine initialized ... device=gpu" and it wrote twice as fast as the GGUF build),
|
||||
* and 18269 reports it stuck in "Stopping..." and a restart on a 32 GB Mac, a size the
|
||||
* measured figures keep it away from (its maintainer promised a memory fix "in the next
|
||||
* release" in September; whether that predates the pinned 0.40.15 is not known).
|
||||
* - The plain tags `qwen3.5:9b`, `mimo-v2.6:9b` and `qwen3.8:27b`: a Mac pulls their MLX
|
||||
* builds, whose 8 GiB of prefix snapshots do not fit the Macs a 9B or a 27B is for. Their
|
||||
* GGUF builds are listed instead.
|
||||
* - `qwen3.5:122b-a10b`: replaced by `ornith-1.5:122b`, a fine-tune of it (SWE-bench
|
||||
* Verified 84.5 against 72.0) whose MLX build carries a renderer and a tool parser.
|
||||
*
|
||||
* One open issue touches rows that ARE listed: runtime issue 16992 (above). Its one diagnosed
|
||||
* cause sits in the tool parser itself (it closes a call at the first closing tag, so a tool
|
||||
* argument that quotes a whole tool call is cut off; the fix, runtime pull request 17021, is
|
||||
* open). The names `ornith` and `qwen3.5` map to that one parser in the pinned runtime
|
||||
* (model/parsers/parsers.go), so every Qwen-family row here shares it, the pair we ran
|
||||
* included. The check after a download asks for one plain tool call and cannot see this.
|
||||
*
|
||||
* `tested` has one entry: `qwen3.6:35b-a3b` with OpenCode on an Apple M5 Pro with 48 GB,
|
||||
* 2026-11-06, 67.7 tokens per second. Only a Mac of exactly that chip and memory size is told
|
||||
* "Recommended" and shown that speed in place of the estimate. Every other row was never run
|
||||
@@ -330,13 +395,35 @@ const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4;
|
||||
/** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
|
||||
const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* The memory figures measured for `muse-glimmer:30b-mlx` on the same Mac (2026-11-12, runtime
|
||||
* 0.40.15, the MLX runner), in decimal bytes: the runner's own `memory peak / held` log line
|
||||
* after every request, and the process footprint at the end. See the file header (MEMORY).
|
||||
*/
|
||||
const MUSE_MLX_MEASURED_MEMORY = {
|
||||
/** Held grew by 37,900 to 45,100 bytes per token between prompts of 30,000, 60,000 and 94,600 tokens: the upper end. */
|
||||
heldBytesPerToken: 45_000,
|
||||
/**
|
||||
* The prefix snapshots in full (the run filled them: 0.53 GiB per separate prompt of 15,000
|
||||
* tokens until the oldest was dropped), and 0.5 GB for what the process holds beside the
|
||||
* runner's own count (0.2 GB measured).
|
||||
*/
|
||||
overheadBytes: MLX_PREFIX_CACHE_BYTES + 500_000_000,
|
||||
/**
|
||||
* The first measured row's 5 GB. This run only gives a floor: the footprint's lifetime peak
|
||||
* stood 3.0 to 3.3 GB above what was held at the end, and a larger transient during the
|
||||
* earlier, longer prompts would not show in that figure.
|
||||
*/
|
||||
peakExtraBytes: 5_000_000_000,
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* The catalogue that ships with Codeman. Never read directly outside this file and its test:
|
||||
* callers use `getLocalModelCatalog()`, which validates it first.
|
||||
*/
|
||||
export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
schema: 1,
|
||||
updated: '2026-11-08',
|
||||
updated: '2026-11-12',
|
||||
// The oldest macOS the model runtime supports.
|
||||
minMacosMajor: 14,
|
||||
// Ordered by rank. The higher rank wins among models that fit and whose speed is graded
|
||||
@@ -344,7 +431,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
// that are left out and why.
|
||||
models: [
|
||||
{
|
||||
// Small Macs (16 to 32 GB, `limited`): the base model of the row below.
|
||||
// Small Macs (18 to 32 GB): the base model of the two rows below.
|
||||
id: 'qwen3.5:9b-q4_k_m',
|
||||
label: 'Qwen3.5 9B',
|
||||
license: 'Apache-2.0',
|
||||
@@ -365,7 +452,35 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Small Macs (16 to 32 GB, `limited`). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of
|
||||
// The model a 16 GB Mac can hold: 9.3 GB at 65,536 tokens, tight. Ornith 1.0 9B, a
|
||||
// fine-tune of Qwen3.5-9B for coding agents, as a GGUF build without a vision projector,
|
||||
// so 0.9 GB smaller than the rows around it. SWE-bench Verified 69.4, SWE-bench Pro 42.9
|
||||
// and Terminal-Bench 2.1 43.1, against 53.2, 31.3 and 21.3 for its base (Ornith's card,
|
||||
// which puts that base 7 points under Xiaomi's figure for it: the two cards do not
|
||||
// compare). Ranked under MiMo for that, and because runtime issue 16992 was filed on its
|
||||
// 35B sibling, which has the same renderer and parser (see the header): it is the pick
|
||||
// only where nothing else fits. Text only.
|
||||
id: 'ornith:9b-q4_k_m',
|
||||
label: 'Ornith 1.0 9B',
|
||||
license: 'MIT',
|
||||
paramsB: 9,
|
||||
runner: 'llamacpp',
|
||||
bytes: 5_629_110_568,
|
||||
// The architecture of the row above (its config.json and its GGUF header say so), so
|
||||
// its figures.
|
||||
heldBytesPerToken: 17_408,
|
||||
overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [131072, 65536],
|
||||
rank: 2,
|
||||
// Not measured: the 9.3 of the row above, the same architecture and the same weights file
|
||||
// size (5.63 GB without the projector).
|
||||
speedPer100Gbps: 9.3,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Small Macs (18 to 32 GB). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of
|
||||
// Qwen3.5-9B for code and agents: SWE-bench Verified 61.1 and SWE-bench Pro 44.6,
|
||||
// against 60.0 and 32.0 for its base (Xiaomi's card). Requires exactly the pinned
|
||||
// runtime, 0.40.15.
|
||||
@@ -375,14 +490,14 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
paramsB: 9,
|
||||
runner: 'llamacpp',
|
||||
bytes: 6_550_811_033,
|
||||
// The architecture and the quantization of the row above, so its figures.
|
||||
// The architecture and the quantization of the base model's row, so its figures.
|
||||
heldBytesPerToken: 17_408,
|
||||
overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [131072, 65536],
|
||||
rank: 2,
|
||||
// Not measured: the 9.3 of the row above, the same architecture and file size.
|
||||
rank: 3,
|
||||
// Not measured: the 9.3 of the base model's row, the same architecture and file size.
|
||||
speedPer100Gbps: 9.3,
|
||||
tested: [],
|
||||
},
|
||||
@@ -404,12 +519,71 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 3,
|
||||
rank: 4,
|
||||
// Not measured: the 13.6 measured for Qwen3.6 35B-A3B on llama.cpp, which has the same
|
||||
// 3 billion active parameters.
|
||||
speedPer100Gbps: 13.6,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Meta's 30B for local agents, a dense 27.9B language model with a vision encoder:
|
||||
// SWE-bench Verified 76.0, SWE-bench Pro 51.2, Terminal-Bench 2.1 51.7 (Meta's card). Its
|
||||
// K/V cache is the smallest of the 27B class (2 K/V heads of 128, and three layers in
|
||||
// four look at a window of 2,048 tokens), so it is the first model of that class a 32 GB
|
||||
// Mac can hold: 21.2 GB at 65,536 tokens, tight. Dense, so it is slow below a Max chip.
|
||||
// Its own window ends at 131,072 tokens.
|
||||
id: 'muse-glimmer:30b-q4_k_m',
|
||||
label: 'Muse Glimmer 30B',
|
||||
license: 'Apache-2.0',
|
||||
paramsB: 29.6,
|
||||
runner: 'llamacpp',
|
||||
bytes: 18_157_010_252,
|
||||
// 13 full-attention layers x 2 K/V heads x 128 x 2 x 34/32.
|
||||
heldBytesPerToken: 7_072,
|
||||
// 39 sliding-window layers: (2,048 + 512) cells x 2 x 128 x 2 x 34/32 = 54,312,960, live
|
||||
// and in each context checkpoint.
|
||||
overheadBytes: 54_312_960 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [131072, 65536],
|
||||
rank: 5,
|
||||
// The 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on llama.cpp
|
||||
// (16.8 GB each without the projector). Left there on purpose after the one run on a Mac
|
||||
// (M5 Pro, 2026-11-12): the runtime alone wrote 14.5 to 15.2 tokens per second, 4.8 per
|
||||
// 100 GB/s, but an OpenCode worker given one line (through the skill's `sendmark`, which
|
||||
// adds its answer-file sentence) showed no answer after 7 minutes, twice.
|
||||
speedPer100Gbps: 2.95,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// The same model as the MLX build: about twice as fast, because the tag carries a draft
|
||||
// model (the runtime logs "Loaded draft model", acceptance 0.34 to 0.62) and its own
|
||||
// renderer and tool parser. On the MacBook Pro (Mac17,13, M5 Pro, 48 GB, 2026-11-12) it
|
||||
// wrote 30.2 tokens per second where the GGUF build wrote 15.2 (the same 300-token
|
||||
// request), answered a tool call in under 3 seconds, and finished an OpenCode turn on
|
||||
// a one-line prompt in 45 seconds. The GGUF build above had shown nothing after 7
|
||||
// minutes on a one-line task (not the same prompt: see that row). The price is
|
||||
// memory: what the MLX runner holds grows six times as fast per token (MEMORY in the
|
||||
// file header), so this build starts at 48 GB and the 32 and 36 GB Macs keep the row
|
||||
// above. Not installed through the window yet, and no agent session beyond that turn:
|
||||
// no `tested` entry.
|
||||
id: 'muse-glimmer:30b-mlx',
|
||||
label: 'Muse Glimmer 30B (MLX)',
|
||||
license: 'Apache-2.0',
|
||||
paramsB: 29.6,
|
||||
runner: 'mlx',
|
||||
// The layers of its manifest, the draft model's included.
|
||||
bytes: 19_053_621_992,
|
||||
...MUSE_MLX_MEASURED_MEMORY,
|
||||
figures: 'measured',
|
||||
contexts: [131072, 65536],
|
||||
rank: 6,
|
||||
// Measured beside Qwen3.6 35B-A3B on MLX, same Mac, same request (a 1,500-token answer):
|
||||
// 32.6 against 101.9 tokens per second, so 0.32 of that row's 16.9. Like that row's, on
|
||||
// the low side: on 307 GB/s it computes to 16.6.
|
||||
speedPer100Gbps: 5.4,
|
||||
tested: [],
|
||||
},
|
||||
{
|
||||
// Macs of 36 GB on a slower chip: the model of the tested pair, as a GGUF build whose
|
||||
// memory does not grow after it loaded. With its multi-token-prediction layer.
|
||||
@@ -430,7 +604,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 4,
|
||||
rank: 7,
|
||||
// Measured: 54.5 tokens per second on 400 GB/s (llama.cpp, without prediction).
|
||||
speedPer100Gbps: 13.6,
|
||||
tested: [],
|
||||
@@ -448,7 +622,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
...MEASURED_MEMORY,
|
||||
figures: 'measured',
|
||||
contexts: [131072, 65536],
|
||||
rank: 5,
|
||||
rank: 8,
|
||||
// Measured: 67.8 tokens per second on 400 GB/s with MLX. The one Mac we tested ran it at
|
||||
// 67.7 on 307 GB/s, so on other chips the estimate is on the low side.
|
||||
speedPer100Gbps: 16.9,
|
||||
@@ -473,7 +647,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 6,
|
||||
rank: 9,
|
||||
// Not measured: the 2.95 of the row below scaled by file size.
|
||||
speedPer100Gbps: 1.7,
|
||||
tested: [],
|
||||
@@ -497,7 +671,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
rank: 7,
|
||||
rank: 10,
|
||||
// Measured: 11.8 tokens per second on 400 GB/s (llama.cpp, Q4_K_M, without prediction).
|
||||
// This build predicts several tokens at a time, so it is faster than this figure says.
|
||||
speedPer100Gbps: 2.95,
|
||||
@@ -521,7 +695,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
peakExtraBytes: MEASURED_MEMORY.peakExtraBytes,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072],
|
||||
rank: 8,
|
||||
rank: 11,
|
||||
// Not measured: the 16.9 of Qwen3.6 35B-A3B on MLX x 3/10, its active parameters
|
||||
// against these, rounded down.
|
||||
speedPer100Gbps: 5,
|
||||
|
||||
@@ -28,9 +28,10 @@
|
||||
* gated, installed or installable, and accepting the model id.
|
||||
* - Ranking, lexicographic: tested on a Mac like this before not tested (a measurement beats
|
||||
* a formula); ok before unknown before slow; the higher model rank; the higher harness
|
||||
* rank; the larger context. The pick is the first candidate, the alternatives are the next
|
||||
* four MODELS, each at its own best context and harness, so a model is always offered at
|
||||
* the largest context it fits at.
|
||||
* rank; the larger context. The pick is the first candidate, the alternatives are EVERY
|
||||
* other model that fits (`MAX_ALTERNATIVES` is above what the catalogue holds), each at its
|
||||
* own best context and harness, so a model is always offered at the largest context it
|
||||
* fits at.
|
||||
* - Disk: a model whose download would leave less than 10 GiB free is not offered. It is not
|
||||
* dropped in silence: the best-ranked model left out that way is named in `noteDiskDropped`
|
||||
* with what is missing.
|
||||
@@ -109,8 +110,13 @@ export const SLOW_BELOW_TOKENS_PER_SECOND = 10;
|
||||
export const LIMITED_BELOW_PARAMS_B = 12;
|
||||
/** What must stay free on the disk after the model download. */
|
||||
export const DISK_HEADROOM_BYTES = 10 * GIB;
|
||||
/** How many other models are listed under the pick. */
|
||||
export const MAX_ALTERNATIVES = 4;
|
||||
/**
|
||||
* How many other models are listed under the pick, at most. Above what the stock catalogue
|
||||
* holds (11 models: 10 beside the pick), so a Mac is shown every model that fits it and the
|
||||
* install accepts every one of them. Still a bound: test/local-model-catalog.test.ts fails for
|
||||
* a catalogue that outgrows it, so nothing drops out of the list unseen.
|
||||
*/
|
||||
export const MAX_ALTERNATIVES = 11;
|
||||
|
||||
/** A memory size of at least this many GB (as sold) gets 3/4 of its memory as device limit, below it 2/3. */
|
||||
const LARGE_LIMIT_FROM_GB = 36;
|
||||
|
||||
@@ -577,10 +577,10 @@ export interface Recommendation {
|
||||
/** Present exactly when `verdict` is `ok` or `limited`. */
|
||||
pick?: Candidate;
|
||||
/**
|
||||
* The next models that fit and have room on the disk (at most four), best first: tested
|
||||
* before not tested, then ok before unknown before slow speed, then the higher model rank.
|
||||
* Each is at the largest context it fits at, with the best harness that can run it there.
|
||||
* Empty when `verdict` is `none`.
|
||||
* Every other model that fits and has room on the disk (`MAX_ALTERNATIVES` is above what the
|
||||
* catalogue holds), best first: tested before not tested, then ok before unknown before
|
||||
* slow speed, then the higher model rank. Each is at the largest context it fits at, with
|
||||
* the best harness that can run it there. Empty when `verdict` is `none`.
|
||||
*/
|
||||
alternatives: Candidate[];
|
||||
/** `DeviceBudget.budgetBytes`. 0 when no budget is computed: not macOS, or a Mac without Apple silicon. */
|
||||
|
||||
@@ -31,6 +31,7 @@ import {
|
||||
LocalModelCatalogSchema,
|
||||
STOCK_LOCAL_MODEL_CATALOG,
|
||||
} from '../src/config/local-model-catalog.js';
|
||||
import { MAX_ALTERNATIVES } from '../src/local-model/recommend.js';
|
||||
import {
|
||||
RUNTIME_CTX_CHECKPOINTS,
|
||||
RUNTIME_HOST_PROMPT_CACHE_MIB,
|
||||
@@ -63,9 +64,11 @@ const GIB = 1024 * MIB;
|
||||
|
||||
/**
|
||||
* The architecture of every model the stock catalogue lists, read from its config.json on
|
||||
* Hugging Face (2026-11-08): full-attention layers, their K/V heads and head size, and the
|
||||
* layers whose cache does not grow with the context (Gated DeltaNet layers with their value
|
||||
* and key heads, or sliding-window layers with their window). Fine-tunes share their base's.
|
||||
* Hugging Face (2026-11-08; Muse Glimmer on 2026-11-10): full-attention layers, their K/V heads
|
||||
* and head size, and the layers whose cache does not grow with the context (Gated DeltaNet
|
||||
* layers with their value and key heads, or sliding-window layers with their window).
|
||||
* Fine-tunes share their base's: the config.json of Ornith 1.0 9B (2026-11-10) is that of
|
||||
* Qwen3.5-9B in every field below, and the GGUF header of its build says the same.
|
||||
*/
|
||||
const ARCHITECTURES = {
|
||||
'Qwen3.5-9B': { attentionLayers: 8, kvHeads: 4, headDim: 256, linear: { layers: 24, vHeads: 32, kHeads: 16 } },
|
||||
@@ -78,6 +81,8 @@ const ARCHITECTURES = {
|
||||
linear: { layers: 36, vHeads: 64, kHeads: 16 },
|
||||
},
|
||||
'North-Mini-Code-1.0': { attentionLayers: 13, kvHeads: 4, headDim: 128, sliding: { layers: 36, window: 4096 } },
|
||||
// 52 layers, three in four on a sliding window (the GGUF header's `sliding_window_pattern`).
|
||||
'Muse-Glimmer-30B': { attentionLayers: 13, kvHeads: 2, headDim: 128, sliding: { layers: 39, window: 2048 } },
|
||||
} as const;
|
||||
|
||||
type ArchitectureName = keyof typeof ARCHITECTURES;
|
||||
@@ -86,7 +91,8 @@ type Architecture = (typeof ARCHITECTURES)[ArchitectureName];
|
||||
/**
|
||||
* Which architecture each stock row runs, and how many tokens the runtime drafts per step with
|
||||
* its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF
|
||||
* header, read on 2026-11-08 from the first bytes of each model layer), else the tag's own
|
||||
* header, read on 2026-11-08 from the first bytes of each model layer; on 2026-11-10 for the
|
||||
* Ornith 1.0 and Muse Glimmer rows), else the tag's own
|
||||
* `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none.
|
||||
* Prediction costs a cache for that layer and one rollback copy of the recurrent state per
|
||||
* drafted token (see the catalogue header). A row the catalogue gains must be added here, or the
|
||||
@@ -94,8 +100,15 @@ type Architecture = (typeof ARCHITECTURES)[ArchitectureName];
|
||||
*/
|
||||
const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; draftTokens: number }> = {
|
||||
'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
// A fine-tune of Qwen3.5-9B; its GGUF has 32 blocks and no prediction layer.
|
||||
'ornith:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 },
|
||||
// No prediction layer; the tags with a separate draft model (`-dflash`) are not listed.
|
||||
'muse-glimmer:30b-q4_k_m': { architecture: 'Muse-Glimmer-30B', draftTokens: 0 },
|
||||
// The MLX build. Its figures are measured, so nothing is recomputed from this entry. The tag
|
||||
// carries a draft MODEL (not a prediction layer of llama.cpp): what that holds was measured too.
|
||||
'muse-glimmer:30b-mlx': { architecture: 'Muse-Glimmer-30B', draftTokens: 0 },
|
||||
// Its params layer sets `draft_num_predict` 2.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 },
|
||||
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 },
|
||||
@@ -157,6 +170,12 @@ function mlxComputedPerToken(arch: Architecture): number {
|
||||
|
||||
/** The figures measured for `qwen3.6:35b-a3b` on the MacBook Pro. */
|
||||
const MEASURED: [number, number, number] = [37_000, 3_000_000_000, 5_000_000_000];
|
||||
/**
|
||||
* The figures measured for `muse-glimmer:30b-mlx` on the same MacBook Pro (2026-11-12): the upper
|
||||
* end of what stayed held per token, the whole prefix budget plus 0.5 GB, and for the peak the
|
||||
* first measured row's 5 GB (its own run gave only a floor of 3.0 to 3.3 GB).
|
||||
*/
|
||||
const MEASURED_MUSE_MLX: [number, number, number] = [45_000, 8 * GIB + 500_000_000, 5_000_000_000];
|
||||
|
||||
/** What an estimated `mlx` row's three figures must be: computed, scaled by what the measured row showed. */
|
||||
function mlxFigures(arch: Architecture): [number, number, number] {
|
||||
@@ -183,20 +202,23 @@ describe('the stock catalogue', () => {
|
||||
expect(getLocalModelCatalog()).toBe(first);
|
||||
});
|
||||
|
||||
it('lists eight models, by rank, and none of those set aside in the header', () => {
|
||||
it('lists eleven models, by rank, and none of those set aside in the header', () => {
|
||||
expect(models.map((m) => m.id)).toEqual([
|
||||
'qwen3.5:9b-q4_k_m',
|
||||
'ornith:9b-q4_k_m',
|
||||
'mimo-v2.6:9b-q4_k_m',
|
||||
'north-mini-code-1.0:q4_k_m',
|
||||
'muse-glimmer:30b-q4_k_m',
|
||||
'muse-glimmer:30b-mlx',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m',
|
||||
'qwen3.6:35b-a3b',
|
||||
'qwen3.8:27b-q8_0',
|
||||
'qwen3.8:27b-mtp-q4_k_m',
|
||||
'ornith-1.5:122b',
|
||||
]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8]);
|
||||
expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]);
|
||||
const setAside =
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:122b)|^ornith-1\.5:(9b|35b|397b)/;
|
||||
/^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:35b|^muse-glimmer:.*dflash/;
|
||||
expect(models.filter((m) => setAside.test(m.id)).map((m) => m.id)).toEqual([]);
|
||||
// The plain 9B and 27B tags are MLX builds on a Mac: their GGUF builds are listed instead.
|
||||
for (const plain of ['qwen3.5:9b', 'mimo-v2.6:9b', 'qwen3.8:27b']) {
|
||||
@@ -207,8 +229,12 @@ describe('the stock catalogue', () => {
|
||||
it('names the runner a Mac uses for each tag', () => {
|
||||
expect(Object.fromEntries(models.map((m) => [m.id, m.runner]))).toEqual({
|
||||
'qwen3.5:9b-q4_k_m': 'llamacpp',
|
||||
'ornith:9b-q4_k_m': 'llamacpp',
|
||||
'mimo-v2.6:9b-q4_k_m': 'llamacpp',
|
||||
'north-mini-code-1.0:q4_k_m': 'llamacpp',
|
||||
'muse-glimmer:30b-q4_k_m': 'llamacpp',
|
||||
// Its manifest names no runner in a list; the runtime starts its MLX runner for it.
|
||||
'muse-glimmer:30b-mlx': 'mlx',
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 'llamacpp',
|
||||
// Manifest lists: a Mac takes the MLX child.
|
||||
'qwen3.6:35b-a3b': 'mlx',
|
||||
@@ -222,8 +248,15 @@ describe('the stock catalogue', () => {
|
||||
expect(Object.fromEntries(models.map((m) => [m.id, m.bytes]))).toEqual({
|
||||
// The registry's manifests as fetched on 2026-11-08, in lowercase.
|
||||
'qwen3.5:9b-q4_k_m': 6_550_825_550,
|
||||
// Fetched on 2026-11-10. No vision projector: the weights, a system prompt, the licence.
|
||||
'ornith:9b-q4_k_m': 5_629_110_568,
|
||||
'mimo-v2.6:9b-q4_k_m': 6_550_811_033,
|
||||
'north-mini-code-1.0:q4_k_m': 18_593_967_008,
|
||||
// Fetched on 2026-11-10: 16.76 GB of weights and a 1.40 GB projector.
|
||||
'muse-glimmer:30b-q4_k_m': 18_157_010_252,
|
||||
// Fetched on 2026-11-12: 1,503 layers, the draft model's among them. A pull on a Mac
|
||||
// that day ended at 19.1 GB.
|
||||
'muse-glimmer:30b-mlx': 19_053_621_992,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 22_621_314_381,
|
||||
// The MLX child of the list, and the total of a real pull on a Mac, byte for byte.
|
||||
'qwen3.6:35b-a3b': 23_105_798_956,
|
||||
@@ -234,9 +267,29 @@ describe('the stock catalogue', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('says which row was measured on a Mac: one, and every other row is an estimate', () => {
|
||||
expect(models.filter((m) => m.figures === 'measured').map((m) => m.id)).toEqual(['qwen3.6:35b-a3b']);
|
||||
it('says which rows were measured on a Mac: two, and every other row is an estimate', () => {
|
||||
expect(models.filter((m) => m.figures === 'measured').map((m) => m.id)).toEqual([
|
||||
'muse-glimmer:30b-mlx',
|
||||
'qwen3.6:35b-a3b',
|
||||
]);
|
||||
expect(figuresOf(models.find((m) => m.id === 'qwen3.6:35b-a3b') as LocalModelEntry)).toEqual(MEASURED);
|
||||
expect(figuresOf(models.find((m) => m.id === 'muse-glimmer:30b-mlx') as LocalModelEntry)).toEqual(
|
||||
MEASURED_MUSE_MLX
|
||||
);
|
||||
});
|
||||
|
||||
it('counts the whole prefix budget of the MLX runner in the row that was measured filling it', () => {
|
||||
// Every estimated MLX row counts the 8 GiB too (mlxFigures). The first measured row does
|
||||
// not: the catalogue header says why, and what a session on it really held.
|
||||
expect(MEASURED_MUSE_MLX[1]).toBe(8 * GIB + 500_000_000);
|
||||
expect(MEASURED[1]).toBeLessThan(8 * GIB);
|
||||
// What the runner really keeps per token is far above the 13 full-attention layers alone
|
||||
// and under all 52 layers at full length: it evidently does not trim the sliding-window layers.
|
||||
const arch = ARCHITECTURES['Muse-Glimmer-30B'];
|
||||
expect(arch.attentionLayers * arch.kvHeads * arch.headDim * 2 * 2).toBe(13_312);
|
||||
expect((arch.attentionLayers + arch.sliding.layers) * arch.kvHeads * arch.headDim * 2 * 2).toBe(53_248);
|
||||
expect(MEASURED_MUSE_MLX[0]).toBeGreaterThan(13_312 * 2);
|
||||
expect(MEASURED_MUSE_MLX[0]).toBeLessThanOrEqual(53_248);
|
||||
});
|
||||
|
||||
it('has one tested entry: the 35B-A3B with OpenCode on the MacBook Pro, with its measured speed', () => {
|
||||
@@ -254,8 +307,12 @@ describe('the stock catalogue', () => {
|
||||
it('gives every model a parameter count, an id of the default registry and a license', () => {
|
||||
expect(Object.fromEntries(models.map((m) => [m.id, [m.paramsB, m.license]]))).toEqual({
|
||||
'qwen3.5:9b-q4_k_m': [9, 'Apache-2.0'],
|
||||
'ornith:9b-q4_k_m': [9, 'MIT'],
|
||||
'mimo-v2.6:9b-q4_k_m': [9, 'MIT'],
|
||||
'north-mini-code-1.0:q4_k_m': [30, 'Apache-2.0'],
|
||||
// The card's total: a 27.9B language model and a vision encoder.
|
||||
'muse-glimmer:30b-q4_k_m': [29.6, 'Apache-2.0'],
|
||||
'muse-glimmer:30b-mlx': [29.6, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [35, 'Apache-2.0'],
|
||||
'qwen3.6:35b-a3b': [35, 'Apache-2.0'],
|
||||
'qwen3.8:27b-q8_0': [27.8, 'Apache-2.0'],
|
||||
@@ -263,8 +320,12 @@ describe('the stock catalogue', () => {
|
||||
'ornith-1.5:122b': [122, 'MIT'],
|
||||
});
|
||||
for (const model of models) expect(model.id).toMatch(LOCAL_MODEL_ID_PATTERN);
|
||||
// The two 9B rows are the ones under the 12 billion line of the `limited` verdict.
|
||||
expect(models.filter((m) => m.paramsB < 12).map((m) => m.id)).toEqual(['qwen3.5:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m']);
|
||||
// The three 9B rows are the ones under the 12 billion line of the `limited` verdict.
|
||||
expect(models.filter((m) => m.paramsB < 12).map((m) => m.id)).toEqual([
|
||||
'qwen3.5:9b-q4_k_m',
|
||||
'ornith:9b-q4_k_m',
|
||||
'mimo-v2.6:9b-q4_k_m',
|
||||
]);
|
||||
});
|
||||
|
||||
it('offers no model below the 65536 tokens a harness needs, nor beyond its own window', () => {
|
||||
@@ -276,11 +337,16 @@ describe('the stock catalogue', () => {
|
||||
|
||||
it('has a speed figure only where one was measured or scaled from a measurement', () => {
|
||||
expect(Object.fromEntries(models.map((m) => [m.id, m.speedPer100Gbps ?? null]))).toEqual({
|
||||
// 37.1 tokens per second on 400 GB/s, and the same for the fine-tune of the same size.
|
||||
// 37.1 tokens per second on 400 GB/s, and the same for the two fine-tunes of the same size.
|
||||
'qwen3.5:9b-q4_k_m': 9.3,
|
||||
'ornith:9b-q4_k_m': 9.3,
|
||||
'mimo-v2.6:9b-q4_k_m': 9.3,
|
||||
// The 35B-A3B's llama.cpp figure: both have 3 billion active parameters.
|
||||
'north-mini-code-1.0:q4_k_m': 13.6,
|
||||
// The dense 27B's figure: the weights of the two are the same size.
|
||||
'muse-glimmer:30b-q4_k_m': 2.95,
|
||||
// Measured beside the 35B-A3B on MLX, same Mac and request: 32.6 against 101.9, x 16.9.
|
||||
'muse-glimmer:30b-mlx': 5.4,
|
||||
// 54.5 on 400 GB/s with llama.cpp.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 13.6,
|
||||
// 67.8 on 400 GB/s with MLX.
|
||||
@@ -302,6 +368,13 @@ describe('the stock catalogue', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('holds no more models than one window lists: the pick and MAX_ALTERNATIVES others', () => {
|
||||
// recommend() cuts the list at MAX_ALTERNATIVES. While the catalogue stays under it, a Mac
|
||||
// is shown every model that fits and the install accepts every one of them. A catalogue
|
||||
// that outgrows the bound must fail here, so the bound is raised (or kept) on purpose.
|
||||
expect(models.length - 1).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
});
|
||||
|
||||
it('names the oldest macOS and its schema version', () => {
|
||||
expect(STOCK_LOCAL_MODEL_CATALOG.minMacosMajor).toBe(14);
|
||||
expect(STOCK_LOCAL_MODEL_CATALOG.schema).toBe(1);
|
||||
@@ -348,9 +421,15 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
expect(Object.fromEntries(models.map((m) => [m.id, figuresOf(m)]))).toEqual({
|
||||
// State x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB.
|
||||
'qwen3.5:9b-q4_k_m': [17_408, 2_547_960_320, 0],
|
||||
'ornith:9b-q4_k_m': [17_408, 2_547_960_320, 0],
|
||||
'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0],
|
||||
// The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB.
|
||||
'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0],
|
||||
// The same rule: 39 layers x 2,560 cells x 544 bytes = 54,312,960 per copy.
|
||||
'muse-glimmer:30b-q4_k_m': [7_072, 2_562_558_464, 0],
|
||||
// Measured: the upper end per token, 8 GiB of prefix snapshots + 0.5 GB. The peak is the
|
||||
// first measured row's: this run gave only a floor for it (3.0 to 3.3 GB).
|
||||
'muse-glimmer:30b-mlx': [45_000, 9_089_934_592, 5_000_000_000],
|
||||
// State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
|
||||
// layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0],
|
||||
@@ -374,9 +453,9 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
byFigures.set(key, [...(byFigures.get(key) ?? []), model]);
|
||||
}
|
||||
const shared = [...byFigures.values()].filter((group) => group.length > 1);
|
||||
// The two 9B rows (a fine-tune and its base) and the two 27B rows (two quantizations of one model).
|
||||
// The three 9B rows (a base and two fine-tunes of it) and the two 27B rows (two quantizations of one model).
|
||||
expect(shared.map((group) => group.map((m) => m.id))).toEqual([
|
||||
['qwen3.5:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m'],
|
||||
['qwen3.5:9b-q4_k_m', 'ornith:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m'],
|
||||
['qwen3.8:27b-q8_0', 'qwen3.8:27b-mtp-q4_k_m'],
|
||||
]);
|
||||
for (const group of shared) {
|
||||
|
||||
@@ -12,9 +12,9 @@
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { getLocalModelCatalog } from '../src/config/local-model-catalog.js';
|
||||
import { getLocalModelCatalog, LocalModelCatalogSchema } from '../src/config/local-model-catalog.js';
|
||||
import { formatGb, memClassGb } from '../src/local-model/format.js';
|
||||
import { modelFitsBudget, recommend } from '../src/local-model/recommend.js';
|
||||
import { MAX_ALTERNATIVES, modelFitsBudget, recommend } from '../src/local-model/recommend.js';
|
||||
import type { HardwareFacts, HarnessFact } from '../src/types.js';
|
||||
|
||||
/** A Mac with Apple silicon and `gb` GB of memory as sold. */
|
||||
@@ -72,18 +72,46 @@ describe('modelFitsBudget', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('does not depend on the ranking: a model that fits and is listed nowhere still fits', () => {
|
||||
// On a 128 GB Mac more models fit than the recommendation lists.
|
||||
const hw = mac(128, 'Apple M5 Max');
|
||||
it('does not depend on what the recommendation lists: a model left out for disk space still fits', () => {
|
||||
// The recommendation lists every model that fits the memory, unless its download would
|
||||
// leave under 10 GiB on the disk. With 40 GB free on a 128 GB Mac that leaves out the two
|
||||
// largest downloads, which fit the memory all the same.
|
||||
const hw = { ...mac(128, 'Apple M5 Max'), freeDiskBytes: 40_000_000_000 };
|
||||
const offered = recommend(hw, catalog, harnessFacts());
|
||||
const listed = new Set([offered.pick, ...offered.alternatives].map((row) => row?.modelId));
|
||||
const unlisted = catalog.models.filter((model) => !listed.has(model.id));
|
||||
expect(unlisted.length).toBeGreaterThan(0);
|
||||
expect(unlisted.map((model) => model.id)).toEqual(['qwen3.8:27b-q8_0', 'ornith-1.5:122b']);
|
||||
for (const model of unlisted) {
|
||||
expect(modelFitsBudget(hw, catalog, model.id, model.contexts.at(-1) as number), model.id).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('does not depend on the ranking: a model cut off by the length of the list still fits', () => {
|
||||
// A catalogue longer than the list: the stock rows plus six more copies of its smallest.
|
||||
const smallest = catalog.models[0];
|
||||
const long = LocalModelCatalogSchema.parse({
|
||||
...structuredClone(catalog),
|
||||
models: [
|
||||
...structuredClone(catalog.models),
|
||||
...Array.from({ length: 6 }, (_, index) => ({
|
||||
...structuredClone(smallest),
|
||||
id: `extra-${index + 1}:9b`,
|
||||
rank: -1 - index,
|
||||
})),
|
||||
],
|
||||
});
|
||||
expect(long.models.length).toBeGreaterThan(1 + MAX_ALTERNATIVES);
|
||||
const hw = mac(512, 'Apple M5 Max');
|
||||
const offered = recommend(hw, long, harnessFacts());
|
||||
const listed = new Set([offered.pick, ...offered.alternatives].map((row) => row?.modelId));
|
||||
expect(listed.size).toBe(1 + MAX_ALTERNATIVES);
|
||||
const unlisted = long.models.filter((model) => !listed.has(model.id));
|
||||
expect(unlisted.length).toBe(long.models.length - 1 - MAX_ALTERNATIVES);
|
||||
for (const model of unlisted) {
|
||||
expect(modelFitsBudget(hw, long, model.id, model.contexts.at(-1) as number), model.id).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('rules out a model that is too large, and one the catalogue does not have', () => {
|
||||
expect(modelFitsBudget(mac(48), catalog, 'ornith-1.5:122b', 131072)).toBe(false);
|
||||
expect(modelFitsBudget(mac(48), catalog, 'no-such:1b', 65536)).toBe(false);
|
||||
|
||||
@@ -487,8 +487,15 @@ describe('needBytes', () => {
|
||||
it('states the need of every stock row at its largest context', () => {
|
||||
expect(Object.fromEntries(STOCK.models.map((m) => [m.id, needBytes(m, m.contexts[0])]))).toEqual({
|
||||
'qwen3.5:9b-q4_k_m': 11_380_487_246,
|
||||
// 0.92 GB under the other two 9B builds: it has no vision projector.
|
||||
'ornith:9b-q4_k_m': 10_458_772_264,
|
||||
'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
|
||||
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
|
||||
// At its own window of 131072 tokens: 18.2 GB + 0.9 GB of K/V cache + 2.6 GB.
|
||||
'muse-glimmer:30b-q4_k_m': 21_646_509_900,
|
||||
// The MLX build at the same window: 19.1 GB + 5.9 GB of cache (45,000 bytes per token,
|
||||
// measured) + the 8 GiB of prefix snapshots and 0.5 GB + 5 GB of peak.
|
||||
'muse-glimmer:30b-mlx': 39_041_796_584,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
|
||||
'qwen3.6:35b-a3b': 35_955_462_956,
|
||||
'qwen3.8:27b-q8_0': 44_292_142_594,
|
||||
@@ -544,6 +551,9 @@ describe('the fit grade', () => {
|
||||
// ---------- 2. The two real Macs ----------
|
||||
|
||||
describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
/** The same Mac without the probe: 48 GiB and the chip are all the fit and the speed depend on. */
|
||||
const hw48 = mac(48, 'Apple M5 Pro');
|
||||
|
||||
/**
|
||||
* Expected: Qwen3.6 35B-A3B at 131,072 tokens with OpenCode, a TIGHT fit, 67.7 tokens per
|
||||
* second, "Recommended for this Mac".
|
||||
@@ -642,25 +652,39 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
expect(result.notes).toEqual([]);
|
||||
});
|
||||
|
||||
it('lists four other models that fit, each at the largest context it fits at, the fast ones first', async () => {
|
||||
it('lists every other model that fits, each at the largest context it fits at, the fast ones first', async () => {
|
||||
const result = recommend(await macbook(), STOCK, ALL_INSTALLED);
|
||||
expect(
|
||||
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
|
||||
).toEqual([
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
// Muse Glimmer as the MLX build: 36.1 GB at 65536 tokens (0.960 of the estimate), and
|
||||
// 5.4 x 3.07 = 16.6 tokens per second. At 131072 it needs 39.0 GB, 1.04: out.
|
||||
['muse-glimmer:30b-mlx', 65536, 'tight', 'ok', 16.6, 'tight'],
|
||||
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
['ornith:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
['qwen3.5:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
// The dense models compute to 2.95 x 3.07 = 9.1 tokens per second on this chip (5.2 in
|
||||
// 8 bits): slow, so they come after every fast model, in the order of their rank.
|
||||
['qwen3.8:27b-mtp-q4_k_m', 262144, 'tight', 'slow', 9.1, 'tight'],
|
||||
['qwen3.8:27b-q8_0', 65536, 'tight', 'slow', 5.2, 'tight'],
|
||||
['muse-glimmer:30b-q4_k_m', 131072, 'good', 'slow', 9.1, 'slow'],
|
||||
]);
|
||||
// Only the tested row rests on a measurement of that model.
|
||||
expect(result.alternatives.map((c) => c.needSource)).toEqual(['estimated', 'estimated', 'estimated', 'estimated']);
|
||||
// The dense 27B fits too (32.1 GB at 262144) but computes to 2.95 x 3.07 = 9.1 tokens per
|
||||
// second here: slow, so it ranks after four fast models and is not listed.
|
||||
expect(modelFitsBudget(await macbook(), STOCK, 'qwen3.8:27b-mtp-q4_k_m', 262144)).toBe(true);
|
||||
expect(order(result).some((entry) => entry.startsWith('qwen3.8:27b'))).toBe(false);
|
||||
// Beside the tested pick, one more row rests on a measurement of its own model's memory.
|
||||
expect(result.alternatives.filter((c) => c.needSource === 'measured').map((c) => c.modelId)).toEqual([
|
||||
'muse-glimmer:30b-mlx',
|
||||
]);
|
||||
// Every model of the catalogue that fits is on the list: nothing is cut off after four.
|
||||
const fitting = STOCK.models.filter((m) => m.contexts.some((ctx) => modelFitsBudget(hw48, STOCK, m.id, ctx)));
|
||||
expect(
|
||||
rows(result)
|
||||
.map((c) => c.modelId)
|
||||
.sort()
|
||||
).toEqual(fitting.map((m) => m.id).sort());
|
||||
// The 122B fits at no context.
|
||||
expect(order(result).some((entry) => entry.startsWith('ornith'))).toBe(false);
|
||||
expect(order(result).some((entry) => entry.startsWith('ornith-1.5'))).toBe(false);
|
||||
});
|
||||
|
||||
it('would get the same model as "Best fit", with the estimated speed, had nobody tested it', async () => {
|
||||
@@ -692,41 +716,69 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
['qwen3.8:27b-mtp-q4_k_m', 'unknown'],
|
||||
['qwen3.8:27b-q8_0', 'unknown'],
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 'unknown'],
|
||||
['muse-glimmer:30b-mlx', 'unknown'],
|
||||
['muse-glimmer:30b-q4_k_m', 'unknown'],
|
||||
['north-mini-code-1.0:q4_k_m', 'unknown'],
|
||||
['mimo-v2.6:9b-q4_k_m', 'unknown'],
|
||||
['ornith:9b-q4_k_m', 'unknown'],
|
||||
['qwen3.5:9b-q4_k_m', 'unknown'],
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => {
|
||||
/**
|
||||
* Expected: verdict `none` (reason `memory`) on the estimated budget, and `limited` with
|
||||
* MiMo V2.6 9B (a GGUF build) at 65,536 tokens, a TIGHT fit, once the runtime measured its
|
||||
* budget.
|
||||
* Expected: `limited` with Ornith 1.0 9B at 65,536 tokens, a TIGHT fit, on the estimated
|
||||
* budget, and MiMo V2.6 9B at 65,536 tokens, also tight, once the runtime measured its budget.
|
||||
*
|
||||
* Why. The estimated budget is 9.67 GiB (10.4 GB: 2/3 of 16 GiB minus 1). The 9B on
|
||||
* llama.cpp needs 6.55 GB of weights + 1.14 GB for a q8_0 K/V cache of 65,536 tokens +
|
||||
* 2.55 GB beside them (its recurrent state live and in 8 context checkpoints, the runner, and
|
||||
* the 1 GiB prompt cache the runtime allows) = 10.2 GB: 0.987 of the estimate, just past the
|
||||
* 0.98 a tight fit may reach. The runtime measured 11.8 GiB on this Mac; the 6 GiB reserve
|
||||
* leaves 10 GiB (10.7 GB) of it, and the same 10.2 GB is 0.954 of that.
|
||||
* Why. The estimated budget is 9.67 GiB (10.4 GB: 2/3 of 16 GiB minus 1). A 9B on llama.cpp
|
||||
* needs its weights + 1.14 GB for a q8_0 K/V cache of 65,536 tokens + 2.55 GB beside them (its
|
||||
* recurrent state live and in 8 context checkpoints, the runner, and the 1 GiB prompt cache
|
||||
* the runtime allows). MiMo and its base are 6.55 GB (a vision projector of 0.92 GB
|
||||
* included): 10.2 GB, 0.987 of the estimate, just past the 0.98 a tight fit may reach. The
|
||||
* Ornith build has no projector, 5.63 GB: 9.3 GB, 0.898 of the estimate. The runtime measured
|
||||
* 11.8 GiB on this Mac; the 6 GiB reserve leaves 10 GiB (10.7 GB) of it, which admits all
|
||||
* three, and the rank puts MiMo first.
|
||||
*
|
||||
* Every figure of this row is computed from the model's architecture, none was measured
|
||||
* (`needSource: 'estimated'`), so this Mac is on the edge either way. Only a run of the 9B on
|
||||
* it settles which side.
|
||||
* Every figure of these rows is computed from the model's architecture, none was measured
|
||||
* (`needSource: 'estimated'`), and no 9B was ever run on such a Mac by us.
|
||||
*/
|
||||
it('gets no model on the estimated budget: the 9B needs 0.987 of it', async () => {
|
||||
it('gets Ornith 1.0 9B at 65536 tokens on the estimated budget, tight: the other 9B builds need 0.987 of it', async () => {
|
||||
const hw = await macMini();
|
||||
expect(hw.chip).toBe('Apple M4');
|
||||
expect(hw.memClassGb).toBe(16);
|
||||
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
|
||||
expect(result).toMatchObject({ verdict: 'none', reason: 'memory', budgetSource: 'estimated' });
|
||||
expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.noneMemory, vars: { memory: '16' } });
|
||||
expect(result.pick).toBeUndefined();
|
||||
expect(result).toMatchObject({ verdict: 'limited', budgetSource: 'estimated' });
|
||||
expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.headlineLimited });
|
||||
expect(result.notes).toEqual([{ text: LOCAL_MODEL_LABELS.noteUntested }]);
|
||||
// floor(17,179,869,184 x 2/3) - 1 GiB.
|
||||
expect(result.budgetBytes).toBe(10_379_504_298);
|
||||
expect(result.budgetGb).toBe('10.4');
|
||||
expect(result.pick).toMatchObject({
|
||||
modelId: 'ornith:9b-q4_k_m',
|
||||
modelLabel: 'Ornith 1.0 9B',
|
||||
license: 'MIT',
|
||||
context: 65536,
|
||||
cli: 'opencode',
|
||||
// 5,629,110,568 + 17,408 x 65,536 + 2,547,960,320.
|
||||
needBytes: 9_317_921_576,
|
||||
needGb: '9.3',
|
||||
memory: { modelBytes: 8_177_070_888, contextBytes: 1_140_850_688, peakBytes: 0 },
|
||||
downloadGb: '5.6',
|
||||
budgetShare: 0.898,
|
||||
fit: 'tight',
|
||||
// 9.3 x 120 / 100.
|
||||
speed: 'ok',
|
||||
tokensPerSecond: 11.2,
|
||||
tested: false,
|
||||
needSource: 'estimated',
|
||||
tag: { kind: 'best-tight', label: 'Best fit, tight' },
|
||||
});
|
||||
// It is the only model that fits, and only at this context: at 131072 it needs 10.5 GB.
|
||||
expect(order(result)).toEqual(['ornith:9b-q4_k_m@65536']);
|
||||
expect(modelFitsBudget(hw, STOCK, 'ornith:9b-q4_k_m', 131072)).toBe(false);
|
||||
const nineB = stockModel('mimo-v2.6:9b-q4_k_m');
|
||||
// 6,550,811,033 + 17,408 x 65,536 + 2,547,960,320.
|
||||
expect(needBytes(nineB, 65536)).toBe(10_239_622_041);
|
||||
@@ -734,7 +786,7 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => {
|
||||
expect(needBytes(nineB, 65536) / result.budgetBytes).toBeLessThan(0.99);
|
||||
});
|
||||
|
||||
it('would need 11.4 GB at 131072 tokens, more than the whole budget', async () => {
|
||||
it('would need 11.4 GB for MiMo at 131072 tokens, more than the whole budget', async () => {
|
||||
const hw = await macMini();
|
||||
const nineB = stockModel('mimo-v2.6:9b-q4_k_m');
|
||||
const budget = deviceBudget(hw).budgetBytes;
|
||||
@@ -744,7 +796,7 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => {
|
||||
expect(modelFitsBudget(hw, STOCK, 'mimo-v2.6:9b-q4_k_m', 65536)).toBe(false);
|
||||
});
|
||||
|
||||
it('gets a small model with the budget its runtime measured: the 9B at 65536 tokens, tight', async () => {
|
||||
it('gets MiMo with the budget its runtime measured: at 65536 tokens, tight', async () => {
|
||||
// The runtime reported 11.8 GiB here; the 6 GiB reserve leaves 10 GiB (10.7 GB) of it.
|
||||
const result = recommend(await macMini(), STOCK, ALL_INSTALLED, { measuredTotalBytes: MACMINI_MEASURED_TOTAL });
|
||||
expect(result).toMatchObject({
|
||||
@@ -773,13 +825,16 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => {
|
||||
needSource: 'estimated',
|
||||
tag: { kind: 'best-tight', label: 'Best fit, tight' },
|
||||
});
|
||||
// Its base model, the same figures, is the one alternative.
|
||||
expect(order(result)).toEqual(['mimo-v2.6:9b-q4_k_m@65536', 'qwen3.5:9b-q4_k_m@65536']);
|
||||
// The other two 9B builds follow by rank. The Ornith build is small enough for 131072
|
||||
// tokens here (10.5 GB, 0.974), and a model is listed at the largest context it fits at.
|
||||
expect(order(result)).toEqual(['mimo-v2.6:9b-q4_k_m@65536', 'ornith:9b-q4_k_m@131072', 'qwen3.5:9b-q4_k_m@65536']);
|
||||
});
|
||||
|
||||
it('gets the same with Claude Code as its only coding agent, which also needs 65536 tokens', async () => {
|
||||
const hw = await macMini();
|
||||
expect(recommend(hw, STOCK, [fact('claude')])).toMatchObject({ verdict: 'none', reason: 'memory' });
|
||||
const estimated = recommend(hw, STOCK, [fact('claude')]);
|
||||
expect(estimated.verdict).toBe('limited');
|
||||
expect(estimated.pick).toMatchObject({ modelId: 'ornith:9b-q4_k_m', context: 65536, cli: 'claude' });
|
||||
const measured = recommend(hw, STOCK, [fact('claude')], { measuredTotalBytes: MACMINI_MEASURED_TOTAL });
|
||||
expect(measured.verdict).toBe('limited');
|
||||
expect(measured.pick).toMatchObject({ modelId: 'mimo-v2.6:9b-q4_k_m', context: 65536, cli: 'claude' });
|
||||
@@ -791,22 +846,26 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
// size. A change of a catalogue row or of a threshold that moves one of these answers has to
|
||||
// change this table on purpose. docs/local-models.md has the same table for people.
|
||||
const EXPECTED: [soldGb: number, chip: string, answer: string][] = [
|
||||
// The smallest need at 65536 tokens is 10.2 GB; the budget is 2.1 GB.
|
||||
// The smallest need at 65536 tokens is 9.3 GB; the budget is 2.1 GB.
|
||||
[8, 'Apple M2', 'none memory'],
|
||||
// The 9B needs 10.2 GB at 65536 tokens, 0.987 of the 10.4 GB estimate: just out. A runtime
|
||||
// that measured the budget a real 16 GB Mac measured admits it (see the Mac mini above).
|
||||
[16, 'Apple M4', 'none memory'],
|
||||
[16, 'Apple M1', 'none memory'],
|
||||
// The 9B build without a vision projector needs 9.3 GB at 65536 tokens, 0.898 of the 10.4 GB
|
||||
// estimate. The other two need 10.2 GB, 0.987: just out, until a runtime measures what a
|
||||
// real 16 GB Mac measured (see the Mac mini above). On an M1 (68 GB/s) a 9B is slow: 6.3.
|
||||
[16, 'Apple M4', 'limited ornith:9b-q4_k_m@65536 tight ok'],
|
||||
[16, 'Apple M1', 'limited ornith:9b-q4_k_m@65536 tight slow'],
|
||||
// 11.4 GB at 131072 tokens, 0.964 of 11.8 GB, and nothing of 12 billion fits: `limited`.
|
||||
[18, 'Apple M3 Pro', 'limited mimo-v2.6:9b-q4_k_m@131072 tight ok'],
|
||||
// On a base chip with 100 GB/s the 9B computes to 9.3 tokens per second: slow.
|
||||
[24, 'Apple M3', 'limited mimo-v2.6:9b-q4_k_m@131072 good slow'],
|
||||
[24, 'Apple M4', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[24, 'Apple M4 Pro', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// North Mini Code needs 23.2 GB at 65536 tokens: 1.06 of the 21.8 GB estimate. A runtime
|
||||
// that measures about 3/4 of the memory admits it (see below).
|
||||
[32, 'Apple M4', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// Muse Glimmer 30B fits at 65536 tokens: 21.2 GB, 0.970 of the 21.8 GB estimate. So a 32 GB
|
||||
// Mac is no small-model Mac (`ok`). The model is dense: on a base chip it computes to 3.5
|
||||
// tokens per second and the fast 9B stays the pick, on a Max chip (11.8) it is the pick.
|
||||
// North Mini Code needs 23.2 GB at 65536 tokens, 1.06 of the estimate: a runtime that
|
||||
// measures about 3/4 of the memory admits it (see below).
|
||||
[32, 'Apple M4', 'ok mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
[32, 'Apple M1 Max', 'ok muse-glimmer:30b-q4_k_m@65536 tight ok'],
|
||||
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
|
||||
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
|
||||
// at 131072 tokens, 0.966 and 0.971 of the budget.
|
||||
@@ -846,18 +905,44 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.noneMemory, vars: { memory: '8' } });
|
||||
// The smallest model a harness can work with needs more than four times the budget.
|
||||
const smallest = Math.min(...STOCK.models.map((m) => needBytes(m, 65536)));
|
||||
expect(formatGb(smallest)).toBe('10.2');
|
||||
expect(formatGb(smallest)).toBe('9.3');
|
||||
expect(smallest / result.budgetBytes).toBeGreaterThan(4);
|
||||
});
|
||||
|
||||
it('gives no model to 8 and 16 GB, a small model to every Mac from 18 to 32 GB, and every larger one a model of 27 billion or more', () => {
|
||||
it('offers Muse Glimmer as the MLX build from 48 GB only, and never as the pick of a Mac whose disk has room', () => {
|
||||
// Measured on a 48 GB Mac: 45,000 bytes held per token and up to 8 GiB of prefix snapshots,
|
||||
// 36.1 GB at 65536 tokens. Runtime issue 18269 reports this build stuck, and a restart, on
|
||||
// a 32 GB Mac: the figures keep it away from there.
|
||||
for (const hw of SWEEP) {
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
const label = `${hw.memClassGb} GB ${hw.chip}`;
|
||||
if (hw.memClassGb <= 16) expect(result.reason, label).toBe('memory');
|
||||
else if (hw.memClassGb <= 32) {
|
||||
const offered = rows(result).some((c) => c.modelId === 'muse-glimmer:30b-mlx');
|
||||
expect(offered, label).toBe(hw.memClassGb >= 48);
|
||||
expect(result.pick?.modelId, label).not.toBe('muse-glimmer:30b-mlx');
|
||||
}
|
||||
// The 32 GB Mac with a Max chip keeps the GGUF build as its pick.
|
||||
const m1max = recommend(mac(32, 'Apple M1 Max'), STOCK, ALL_INSTALLED);
|
||||
expect(m1max.pick?.modelId).toBe('muse-glimmer:30b-q4_k_m');
|
||||
});
|
||||
|
||||
it('gives no model to 8 GB, a small model to every Mac from 16 to 24 GB, and every Mac from 36 GB a model of 27 billion or more', () => {
|
||||
for (const hw of SWEEP) {
|
||||
const result = recommend(hw, STOCK, ALL_INSTALLED);
|
||||
const label = `${hw.memClassGb} GB ${hw.chip}`;
|
||||
if (hw.memClassGb <= 8) expect(result.reason, label).toBe('memory');
|
||||
else if (hw.memClassGb <= 24) {
|
||||
expect(result.verdict, label).toBe('limited');
|
||||
expect(result.pick?.paramsB, label).toBe(9);
|
||||
} else if (hw.memClassGb === 32) {
|
||||
// A model of 30 billion fits, so the Mac is not `limited`. It is dense: the pick only
|
||||
// where it is not slow (a chip this file knows is fast, or one it knows nothing about).
|
||||
expect(result.verdict, label).toBe('ok');
|
||||
expect(
|
||||
rows(result).map((c) => c.modelId),
|
||||
label
|
||||
).toContain('muse-glimmer:30b-q4_k_m');
|
||||
expect(result.pick?.paramsB, label).toBe(hw.chip === 'Apple M4' ? 9 : 29.6);
|
||||
if (hw.chip === 'Apple M4') expect(result.notes[0], label).toEqual({ text: LOCAL_MODEL_LABELS.noteSmallPick });
|
||||
} else {
|
||||
expect(result.verdict, label).toBe('ok');
|
||||
expect(result.pick?.paramsB, label).toBeGreaterThanOrEqual(27);
|
||||
@@ -1274,10 +1359,18 @@ describe('verdict none', () => {
|
||||
const oneShort = recommend(withFree(cut - 1), STOCK, ALL_INSTALLED);
|
||||
expect(oneShort.pick?.modelId).toBe('qwen3.6:35b-a3b-mtp-q4_k_m');
|
||||
expect(oneShort.headline.text).toBe(LOCAL_MODEL_LABELS.headlineOk);
|
||||
// At 33 GB neither 35B-A3B build has room: North Mini Code, and the tested pair is named.
|
||||
// At 33 GB neither 35B-A3B build has room: the next rank whose speed is `ok` and whose
|
||||
// download fits is Muse Glimmer as the MLX build (19.1 GB), and the tested pair is named.
|
||||
const short = recommend(withFree(33 * GB), STOCK, ALL_INSTALLED);
|
||||
expect(short.verdict).toBe('ok');
|
||||
expect(short.pick?.modelId).toBe('north-mini-code-1.0:q4_k_m');
|
||||
expect(short.pick).toMatchObject({
|
||||
modelId: 'muse-glimmer:30b-mlx',
|
||||
context: 65536,
|
||||
fit: 'tight',
|
||||
tested: false,
|
||||
});
|
||||
// Below 29.8 GB that build has no room either, and North Mini Code (18.6 GB) is the pick.
|
||||
expect(recommend(withFree(29.5 * GB), STOCK, ALL_INSTALLED).pick?.modelId).toBe('north-mini-code-1.0:q4_k_m');
|
||||
expect(short.notes).toEqual([
|
||||
{ text: LOCAL_MODEL_LABELS.noteDiskDropped, vars: { model: 'Qwen3.6 35B-A3B', missing: '0.8' } },
|
||||
{ text: LOCAL_MODEL_LABELS.noteUntested },
|
||||
@@ -1620,11 +1713,13 @@ describe('the ranking', () => {
|
||||
expect(result.pick).toMatchObject({ context: 1024, fit: 'tight', budgetShare: 0.92 });
|
||||
});
|
||||
|
||||
it('lists each model once, never the pick again, and at most four alternatives', () => {
|
||||
expect(MAX_ALTERNATIVES).toBe(4);
|
||||
// Seven models fit a 128 GB Mac; the pick plus four are shown.
|
||||
it('lists each model once, never the pick again, and every model that fits', () => {
|
||||
// The bound is above what either catalogue holds, so nothing that fits is cut off.
|
||||
expect(MAX_ALTERNATIVES).toBe(11);
|
||||
for (const [, catalog] of BOTH) expect(catalog.models.length - 1).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
// Seven models fit a 128 GB Mac: the pick and all six others are shown.
|
||||
const result = recommend(mac(128, 'Apple M5 Ultra'), RULES, ALL_INSTALLED);
|
||||
expect(result.alternatives).toHaveLength(4);
|
||||
expect(result.alternatives).toHaveLength(6);
|
||||
const ids = rows(result).map((c) => c.modelId);
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
for (const [, catalog] of BOTH) {
|
||||
@@ -1633,9 +1728,26 @@ describe('the ranking', () => {
|
||||
const sweptIds = rows(swept).map((c) => c.modelId);
|
||||
expect(new Set(sweptIds).size).toBe(sweptIds.length);
|
||||
expect(swept.alternatives.length).toBeLessThanOrEqual(MAX_ALTERNATIVES);
|
||||
// Every model that fits at a context a harness works at is on the list (the sweep's
|
||||
// Macs have 500 GB of free disk, so none is left out for room).
|
||||
const fitting = catalog.models.filter((m) =>
|
||||
m.contexts.some((ctx) => ctx >= 65536 && modelFitsBudget(hw, catalog, m.id, ctx))
|
||||
);
|
||||
expect([...sweptIds].sort(), `${hw.memClassGb} GB ${hw.chip}`).toEqual(fitting.map((m) => m.id).sort());
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('still cuts a list longer than the bound, and the install can only take what is listed', () => {
|
||||
// Fourteen models that all fit: the pick and MAX_ALTERNATIVES others, best rank first.
|
||||
const many = catalogOf(
|
||||
Array.from({ length: 14 }, (_, index) => modelNeeding(`m${index + 1}:x`, GIB, { rank: index + 1 }))
|
||||
);
|
||||
const result = recommend(HUNDRED_GIB_MAC, many, [fact('alpha')]);
|
||||
expect(rows(result)).toHaveLength(1 + MAX_ALTERNATIVES);
|
||||
expect(rows(result)[0].modelId).toBe('m14:x');
|
||||
expect(rows(result).at(-1)?.modelId).toBe('m3:x');
|
||||
});
|
||||
});
|
||||
|
||||
describe('harnesses', () => {
|
||||
@@ -2064,6 +2176,8 @@ describe('what each row of the chooser carries', () => {
|
||||
'qwen3.5:9b-q8_0': 'estimated',
|
||||
'qwen3.5:9b': 'estimated',
|
||||
'gpt-oss:20b': 'estimated',
|
||||
// Slow on this chip, so it was the sixth row: listed since the list holds every model.
|
||||
'qwen3.8:27b-q8_0': 'estimated',
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -2075,10 +2189,21 @@ describe('the measured budget, end to end', () => {
|
||||
const measured = recommend(hw, STOCK, ALL_INSTALLED, { measuredTotalBytes: MACBOOK_MEASURED_TOTAL });
|
||||
expect([estimated.budgetSource, estimated.budgetGb]).toEqual(['estimated', '37.6']);
|
||||
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
|
||||
// The same rows in the same order; only the share of each changes.
|
||||
expect(order(measured)).toEqual(order(estimated));
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
|
||||
// The same models in the same order. The share of each changes, and two models step up a
|
||||
// context: the 27B in 8 bits fits 65536 tokens under the estimate (0.975) and 131072 under
|
||||
// what the runtime reported (0.976), and Muse Glimmer as the MLX build goes from 65536
|
||||
// (0.960) to its whole window of 131072 (0.972).
|
||||
expect(rows(measured).map((c) => c.modelId)).toEqual(rows(estimated).map((c) => c.modelId));
|
||||
const stepped = order(estimated).filter((entry, index) => order(measured)[index] !== entry);
|
||||
expect(stepped).toEqual(['muse-glimmer:30b-mlx@65536', 'qwen3.8:27b-q8_0@65536']);
|
||||
expect(order(measured)).toContain('qwen3.8:27b-q8_0@131072');
|
||||
expect(order(measured)).toContain('muse-glimmer:30b-mlx@131072');
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([
|
||||
0.957, 0.767, 0.96, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576,
|
||||
]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([
|
||||
0.895, 0.717, 0.972, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539,
|
||||
]);
|
||||
for (const candidate of rows(measured)) {
|
||||
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user