diff --git a/docs/local-models.md b/docs/local-models.md index 4770e952..db99a1ad 100644 --- a/docs/local-models.md +++ b/docs/local-models.md @@ -18,16 +18,16 @@ in [architecture-invariants.md](architecture-invariants.md#local-model-install). The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that fits this Mac's memory at a context a coding agent can work with, and under it every other -model of the catalogue that fits (nine models today: an 18 GB Mac sees two of them, a 512 GB -Mac all nine). What the catalogue answers today, before the runtime measured anything (the +model of the catalogue that fits (ten models today: a 16 GB Mac sees one of them, a 512 GB +Mac all ten). What the catalogue answers today, before the runtime measured anything (the budget is then an estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent): | Memory | Budget | What it gets | | ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 10.2 GB at 65,536 tokens | -| 16 GB | 10.4 GB | Nothing. MiMo V2.6 9B at 65,536 tokens needs 10.2 GB, 98.7% of the budget, just past the 98% a tight fit may reach (it is 95% of the 10.7 GB a 16 GB Mac mini measured). On the edge either way: only a run on such a Mac settles it | -| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative | +| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.3 GB at 65,536 tokens | +| 16 GB | 10.4 GB | A small model (`limited`): Ornith 1.0 9B at 65,536 tokens, 9.3 GB, tight (90%). Its build has no vision projector, which makes it 0.9 GB smaller than the other two 9B builds: MiMo V2.6 9B needs 10.2 GB there, 98.7% of the budget, just past the 98% a tight fit may reach. Once the runtime measures what a 16 GB Mac mini measured (10.7 GB), all three fit and MiMo is the pick. No 9B was run on such a Mac by us: only a run settles it | +| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Ornith 1.0 9B (10.5 GB) and Qwen3.5 9B are the alternatives | | 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow | | 32 GB | 21.8 GB | No longer a small-model Mac: Muse Glimmer 30B fits at 65,536 tokens (21.2 GB, 97%, tight). It is a dense model. On an M1 Max or M2 Max it computes to 11.8 tokens per second and is the pick; on a base or Pro chip it computes to under 6 and is listed as slow, and the pick stays MiMo V2.6 9B at 131,072 tokens with the note that it is a small model. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip | | 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight. Muse Glimmer 30B at 131,072 tokens (21.6 GB, 78%) is the alternative with room to spare | @@ -78,6 +78,7 @@ Where every figure comes from: | Model | Runner | Memory figures | Speed (tokens per second per 100 GB/s) | | ---------------------------- | --------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------- | | `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max | +| `ornith:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B; its config and its GGUF header say so) | 9.3: the same architecture and weights size | | `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size | | `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters | | `muse-glimmer:30b-q4_k_m` | llama.cpp | computed from its config: 13 full-attention and 39 sliding-window layers | 2.95: the dense 27B's, whose weights are the same size | @@ -88,7 +89,7 @@ Where every figure comes from: | `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters | Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on -2026-11-08 (the Muse Glimmer row on 2026-11-10). Only one row was ever run on +2026-11-08 (the Ornith 1.0 and Muse Glimmer rows on 2026-11-10). Only one row was ever run on a Mac, so every other answer in the table above is a calculation. For a llama.cpp build the check after the download catches a load that was judged too kindly, but not what grows after it in host memory (the checkpoints and the prompt cache, which are counted, not checked); for an MLX build it catches neither (see "The check" diff --git a/src/config/local-model-catalog.ts b/src/config/local-model-catalog.ts index bf4e982a..406d12de 100644 --- a/src/config/local-model-catalog.ts +++ b/src/config/local-model-catalog.ts @@ -24,9 +24,9 @@ * throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses, * so a bad row fails the gate and never reaches the route. * - * Where each figure of the stock rows comes from, and what was checked (2026-11-08; the row - * added on 2026-11-10, `muse-glimmer:30b-q4_k_m`, was read that day). [V] read at the source - * that day, [M] measured on a Mac, [C] computed from [V] inputs: + * Where each figure of the stock rows comes from, and what was checked (2026-11-08; the two rows + * added on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m`, were read that day). + * [V] read at the source that day, [M] measured on a Mac, [C] computed from [V] inputs: * * | Figure | Source | Checked | * |----------------------|-----------------------------------------------------------|--------------------------------------| @@ -76,8 +76,8 @@ * "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in * its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp` * tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not; read - * the same way on 2026-11-10, `muse-glimmer:30b-q4_k_m` does not either). For such a - * model the runtime always passes + * the same way on 2026-11-10, `ornith:9b-q4_k_m` and `muse-glimmer:30b-q4_k_m` do not + * either). For such a model the runtime always passes * `--spec-type draft-mtp --spec-draft-n-max N`, N * being the tag's own `draft_num_predict` (its params layer: 4 for * `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default @@ -146,11 +146,11 @@ * manifests name no renderer and no tool parser, so whether tool calls work through the * runtime is unknown (runtime pull request 17965 is still open). `ornith-1.5:122b` has * both. - * - `ornith:9b`, `ornith:35b` (Ornith 1.0, MIT, with a renderer and a tool parser; SWE-bench - * Verified 69.4 and 75.6 on Ornith's card, against 73.4 for Qwen3.6 35B-A3B): runtime issue - * 16992, open, reports the 35B's tool calls arriving as text in long agent chains, and the - * two share the renderer and the parser. The 9B (5.6 GB, no vision projector) is the one - * model that would fit a 16 GB Mac on the estimated budget: 9.3 GB at 65,536 tokens. + * - `ornith:35b` (Ornith 1.0 35B-A3B, MIT, with a renderer and a tool parser; SWE-bench + * Verified 75.6 against 73.4 for Qwen3.6 35B-A3B on Ornith's card): runtime issue 16992, + * open, reports its tool calls arriving as text in long agent chains, and its size is + * served by the pair we ran. Its 9B sibling `ornith:9b-q4_k_m` is listed: see that row + * for why. * - `gpt-oss:20b`: weak in agentic coding (SWE-bench Verified 52.4, Terminal-Bench 2.1 15.2 * in NVIDIA's table). * - `flint:40b`: runtime issue 19191 (open) breaks integer tool arguments after the first @@ -382,7 +382,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { // that are left out and why. models: [ { - // Small Macs (18 to 32 GB): the base model of the row below. + // Small Macs (18 to 32 GB): the base model of the two rows below. id: 'qwen3.5:9b-q4_k_m', label: 'Qwen3.5 9B', license: 'Apache-2.0', @@ -402,6 +402,34 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { speedPer100Gbps: 9.3, tested: [], }, + { + // The model a 16 GB Mac can hold: 9.3 GB at 65,536 tokens, tight. Ornith 1.0 9B, a + // fine-tune of Qwen3.5-9B for coding agents, as a GGUF build without a vision projector, + // so 0.9 GB smaller than the rows around it. SWE-bench Verified 69.4, SWE-bench Pro 42.9 + // and Terminal-Bench 2.1 43.1, against 53.2, 31.3 and 21.3 for its base (Ornith's card, + // which puts that base 7 points under Xiaomi's figure for it: the two cards do not + // compare). Ranked under MiMo for that, and because runtime issue 16992 was filed on its + // 35B sibling, which has the same renderer and parser (see the header): it is the pick + // only where nothing else fits. Text only. + id: 'ornith:9b-q4_k_m', + label: 'Ornith 1.0 9B', + license: 'MIT', + paramsB: 9, + runner: 'llamacpp', + bytes: 5_629_110_568, + // The architecture of the row above (its config.json and its GGUF header say so), so + // its figures. + heldBytesPerToken: 17_408, + overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES, + peakExtraBytes: 0, + figures: 'estimated', + contexts: [131072, 65536], + rank: 2, + // Not measured: the 9.3 of the row above, the same architecture and the same weights file + // size (5.63 GB without the projector). + speedPer100Gbps: 9.3, + tested: [], + }, { // Small Macs (18 to 32 GB). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of // Qwen3.5-9B for code and agents: SWE-bench Verified 61.1 and SWE-bench Pro 44.6, @@ -413,14 +441,14 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { paramsB: 9, runner: 'llamacpp', bytes: 6_550_811_033, - // The architecture and the quantization of the row above, so its figures. + // The architecture and the quantization of the base model's row, so its figures. heldBytesPerToken: 17_408, overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES, peakExtraBytes: 0, figures: 'estimated', contexts: [131072, 65536], - rank: 2, - // Not measured: the 9.3 of the row above, the same architecture and file size. + rank: 3, + // Not measured: the 9.3 of the base model's row, the same architecture and file size. speedPer100Gbps: 9.3, tested: [], }, @@ -442,7 +470,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], - rank: 3, + rank: 4, // Not measured: the 13.6 measured for Qwen3.6 35B-A3B on llama.cpp, which has the same // 3 billion active parameters. speedPer100Gbps: 13.6, @@ -469,7 +497,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: 0, figures: 'estimated', contexts: [131072, 65536], - rank: 4, + rank: 5, // Not measured: the 2.95 of Qwen3.8 27B, a dense model whose weights are the same size on // llama.cpp (16.8 GB each without the projector). speedPer100Gbps: 2.95, @@ -495,7 +523,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], - rank: 5, + rank: 6, // Measured: 54.5 tokens per second on 400 GB/s (llama.cpp, without prediction). speedPer100Gbps: 13.6, tested: [], @@ -513,7 +541,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { ...MEASURED_MEMORY, figures: 'measured', contexts: [131072, 65536], - rank: 6, + rank: 7, // Measured: 67.8 tokens per second on 400 GB/s with MLX. The one Mac we tested ran it at // 67.7 on 307 GB/s, so on other chips the estimate is on the low side. speedPer100Gbps: 16.9, @@ -538,7 +566,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], - rank: 7, + rank: 8, // Not measured: the 2.95 of the row below scaled by file size. speedPer100Gbps: 1.7, tested: [], @@ -562,7 +590,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], - rank: 8, + rank: 9, // Measured: 11.8 tokens per second on 400 GB/s (llama.cpp, Q4_K_M, without prediction). // This build predicts several tokens at a time, so it is faster than this figure says. speedPer100Gbps: 2.95, @@ -586,7 +614,7 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { peakExtraBytes: MEASURED_MEMORY.peakExtraBytes, figures: 'estimated', contexts: [262144, 131072], - rank: 9, + rank: 10, // Not measured: the 16.9 of Qwen3.6 35B-A3B on MLX x 3/10, its active parameters // against these, rounded down. speedPer100Gbps: 5, diff --git a/src/local-model/recommend.ts b/src/local-model/recommend.ts index 6b6b90b5..45b291b5 100644 --- a/src/local-model/recommend.ts +++ b/src/local-model/recommend.ts @@ -112,7 +112,7 @@ export const LIMITED_BELOW_PARAMS_B = 12; export const DISK_HEADROOM_BYTES = 10 * GIB; /** * How many other models are listed under the pick, at most. Above what the stock catalogue - * holds (9 models: 8 beside the pick), so a Mac is shown every model that fits it and the + * holds (10 models: 9 beside the pick), so a Mac is shown every model that fits it and the * install accepts every one of them. Still a bound: test/local-model-catalog.test.ts fails for * a catalogue that outgrows it, so nothing drops out of the list unseen. */ diff --git a/test/local-model-catalog.test.ts b/test/local-model-catalog.test.ts index 9554c52e..5881c187 100644 --- a/test/local-model-catalog.test.ts +++ b/test/local-model-catalog.test.ts @@ -67,7 +67,8 @@ const GIB = 1024 * MIB; * Hugging Face (2026-11-08; Muse Glimmer on 2026-11-10): full-attention layers, their K/V heads * and head size, and the layers whose cache does not grow with the context (Gated DeltaNet * layers with their value and key heads, or sliding-window layers with their window). - * Fine-tunes share their base's. + * Fine-tunes share their base's: the config.json of Ornith 1.0 9B (2026-11-10) is that of + * Qwen3.5-9B in every field below, and the GGUF header of its build says the same. */ const ARCHITECTURES = { 'Qwen3.5-9B': { attentionLayers: 8, kvHeads: 4, headDim: 256, linear: { layers: 24, vHeads: 32, kHeads: 16 } }, @@ -91,7 +92,7 @@ type Architecture = (typeof ARCHITECTURES)[ArchitectureName]; * Which architecture each stock row runs, and how many tokens the runtime drafts per step with * its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF * header, read on 2026-11-08 from the first bytes of each model layer; on 2026-11-10 for the - * Muse Glimmer row), else the tag's own + * Ornith 1.0 and Muse Glimmer rows), else the tag's own * `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none. * Prediction costs a cache for that layer and one rollback copy of the recurrent state per * drafted token (see the catalogue header). A row the catalogue gains must be added here, or the @@ -99,6 +100,8 @@ type Architecture = (typeof ARCHITECTURES)[ArchitectureName]; */ const ROW_ARCHITECTURE: Record = { 'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 }, + // A fine-tune of Qwen3.5-9B; its GGUF has 32 blocks and no prediction layer. + 'ornith:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 }, 'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 }, 'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 }, // No prediction layer; the tags with a separate draft model (`-dflash`) are not listed. @@ -190,9 +193,10 @@ describe('the stock catalogue', () => { expect(getLocalModelCatalog()).toBe(first); }); - it('lists nine models, by rank, and none of those set aside in the header', () => { + it('lists ten models, by rank, and none of those set aside in the header', () => { expect(models.map((m) => m.id)).toEqual([ 'qwen3.5:9b-q4_k_m', + 'ornith:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m', 'north-mini-code-1.0:q4_k_m', 'muse-glimmer:30b-q4_k_m', @@ -202,9 +206,9 @@ describe('the stock catalogue', () => { 'qwen3.8:27b-mtp-q4_k_m', 'ornith-1.5:122b', ]); - expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9]); + expect(models.map((m) => m.rank)).toEqual([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]); const setAside = - /^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:(9b|35b)|^muse-glimmer:.*(mlx|dflash)/; + /^(bonsai-2|gpt-oss|flint|lfm2\.5|laguna|qwen3\.8-flash-next|qwen3\.5:(122b|4b)|ripple-preview|nemotron-3\.5-super|qwen3-coder-next)|^ornith-1\.5:(9b|35b|397b)|^ornith:35b|^muse-glimmer:.*(mlx|dflash)/; expect(models.filter((m) => setAside.test(m.id)).map((m) => m.id)).toEqual([]); // The plain 9B and 27B tags are MLX builds on a Mac: their GGUF builds are listed instead. for (const plain of ['qwen3.5:9b', 'mimo-v2.6:9b', 'qwen3.8:27b']) { @@ -215,6 +219,7 @@ describe('the stock catalogue', () => { it('names the runner a Mac uses for each tag', () => { expect(Object.fromEntries(models.map((m) => [m.id, m.runner]))).toEqual({ 'qwen3.5:9b-q4_k_m': 'llamacpp', + 'ornith:9b-q4_k_m': 'llamacpp', 'mimo-v2.6:9b-q4_k_m': 'llamacpp', 'north-mini-code-1.0:q4_k_m': 'llamacpp', 'muse-glimmer:30b-q4_k_m': 'llamacpp', @@ -231,6 +236,8 @@ describe('the stock catalogue', () => { expect(Object.fromEntries(models.map((m) => [m.id, m.bytes]))).toEqual({ // The registry's manifests as fetched on 2026-11-08, in lowercase. 'qwen3.5:9b-q4_k_m': 6_550_825_550, + // Fetched on 2026-11-10. No vision projector: the weights, a system prompt, the licence. + 'ornith:9b-q4_k_m': 5_629_110_568, 'mimo-v2.6:9b-q4_k_m': 6_550_811_033, 'north-mini-code-1.0:q4_k_m': 18_593_967_008, // Fetched on 2026-11-10: 16.76 GB of weights and a 1.40 GB projector. @@ -265,6 +272,7 @@ describe('the stock catalogue', () => { it('gives every model a parameter count, an id of the default registry and a license', () => { expect(Object.fromEntries(models.map((m) => [m.id, [m.paramsB, m.license]]))).toEqual({ 'qwen3.5:9b-q4_k_m': [9, 'Apache-2.0'], + 'ornith:9b-q4_k_m': [9, 'MIT'], 'mimo-v2.6:9b-q4_k_m': [9, 'MIT'], 'north-mini-code-1.0:q4_k_m': [30, 'Apache-2.0'], // The card's total: a 27.9B language model and a vision encoder. @@ -276,8 +284,12 @@ describe('the stock catalogue', () => { 'ornith-1.5:122b': [122, 'MIT'], }); for (const model of models) expect(model.id).toMatch(LOCAL_MODEL_ID_PATTERN); - // The two 9B rows are the ones under the 12 billion line of the `limited` verdict. - expect(models.filter((m) => m.paramsB < 12).map((m) => m.id)).toEqual(['qwen3.5:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m']); + // The three 9B rows are the ones under the 12 billion line of the `limited` verdict. + expect(models.filter((m) => m.paramsB < 12).map((m) => m.id)).toEqual([ + 'qwen3.5:9b-q4_k_m', + 'ornith:9b-q4_k_m', + 'mimo-v2.6:9b-q4_k_m', + ]); }); it('offers no model below the 65536 tokens a harness needs, nor beyond its own window', () => { @@ -289,8 +301,9 @@ describe('the stock catalogue', () => { it('has a speed figure only where one was measured or scaled from a measurement', () => { expect(Object.fromEntries(models.map((m) => [m.id, m.speedPer100Gbps ?? null]))).toEqual({ - // 37.1 tokens per second on 400 GB/s, and the same for the fine-tune of the same size. + // 37.1 tokens per second on 400 GB/s, and the same for the two fine-tunes of the same size. 'qwen3.5:9b-q4_k_m': 9.3, + 'ornith:9b-q4_k_m': 9.3, 'mimo-v2.6:9b-q4_k_m': 9.3, // The 35B-A3B's llama.cpp figure: both have 3 billion active parameters. 'north-mini-code-1.0:q4_k_m': 13.6, @@ -370,6 +383,7 @@ describe('the memory figures of the stock rows: each row has its own', () => { expect(Object.fromEntries(models.map((m) => [m.id, figuresOf(m)]))).toEqual({ // State x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB. 'qwen3.5:9b-q4_k_m': [17_408, 2_547_960_320, 0], + 'ornith:9b-q4_k_m': [17_408, 2_547_960_320, 0], 'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0], // The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB. 'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0], @@ -398,9 +412,9 @@ describe('the memory figures of the stock rows: each row has its own', () => { byFigures.set(key, [...(byFigures.get(key) ?? []), model]); } const shared = [...byFigures.values()].filter((group) => group.length > 1); - // The two 9B rows (a fine-tune and its base) and the two 27B rows (two quantizations of one model). + // The three 9B rows (a base and two fine-tunes of it) and the two 27B rows (two quantizations of one model). expect(shared.map((group) => group.map((m) => m.id))).toEqual([ - ['qwen3.5:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m'], + ['qwen3.5:9b-q4_k_m', 'ornith:9b-q4_k_m', 'mimo-v2.6:9b-q4_k_m'], ['qwen3.8:27b-q8_0', 'qwen3.8:27b-mtp-q4_k_m'], ]); for (const group of shared) { diff --git a/test/local-model-recommend.test.ts b/test/local-model-recommend.test.ts index d80594d4..cbdc07fa 100644 --- a/test/local-model-recommend.test.ts +++ b/test/local-model-recommend.test.ts @@ -487,6 +487,8 @@ describe('needBytes', () => { it('states the need of every stock row at its largest context', () => { expect(Object.fromEntries(STOCK.models.map((m) => [m.id, needBytes(m, m.contexts[0])]))).toEqual({ 'qwen3.5:9b-q4_k_m': 11_380_487_246, + // 0.92 GB under the other two 9B builds: it has no vision projector. + 'ornith:9b-q4_k_m': 10_458_772_264, 'mimo-v2.6:9b-q4_k_m': 11_380_472_729, 'north-mini-code-1.0:q4_k_m': 25_999_848_864, // At its own window of 131072 tokens: 18.2 GB + 0.9 GB of K/V cache + 2.6 GB. @@ -656,6 +658,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { ['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], + ['ornith:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], ['qwen3.5:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], // The dense models compute to 2.95 x 3.07 = 9.1 tokens per second on this chip (5.2 in // 8 bits): slow, so they come after every fast model, in the order of their rank. @@ -704,6 +707,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { ['muse-glimmer:30b-q4_k_m', 'unknown'], ['north-mini-code-1.0:q4_k_m', 'unknown'], ['mimo-v2.6:9b-q4_k_m', 'unknown'], + ['ornith:9b-q4_k_m', 'unknown'], ['qwen3.5:9b-q4_k_m', 'unknown'], ]); }); @@ -711,34 +715,57 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { /** - * Expected: verdict `none` (reason `memory`) on the estimated budget, and `limited` with - * MiMo V2.6 9B (a GGUF build) at 65,536 tokens, a TIGHT fit, once the runtime measured its - * budget. + * Expected: `limited` with Ornith 1.0 9B at 65,536 tokens, a TIGHT fit, on the estimated + * budget, and MiMo V2.6 9B at 65,536 tokens, also tight, once the runtime measured its budget. * - * Why. The estimated budget is 9.67 GiB (10.4 GB: 2/3 of 16 GiB minus 1). The 9B on - * llama.cpp needs 6.55 GB of weights + 1.14 GB for a q8_0 K/V cache of 65,536 tokens + - * 2.55 GB beside them (its recurrent state live and in 8 context checkpoints, the runner, and - * the 1 GiB prompt cache the runtime allows) = 10.2 GB: 0.987 of the estimate, just past the - * 0.98 a tight fit may reach. The runtime measured 11.8 GiB on this Mac; the 6 GiB reserve - * leaves 10 GiB (10.7 GB) of it, and the same 10.2 GB is 0.954 of that. + * Why. The estimated budget is 9.67 GiB (10.4 GB: 2/3 of 16 GiB minus 1). A 9B on llama.cpp + * needs its weights + 1.14 GB for a q8_0 K/V cache of 65,536 tokens + 2.55 GB beside them (its + * recurrent state live and in 8 context checkpoints, the runner, and the 1 GiB prompt cache + * the runtime allows). MiMo and its base are 6.55 GB (a vision projector of 0.92 GB + * included): 10.2 GB, 0.987 of the estimate, just past the 0.98 a tight fit may reach. The + * Ornith build has no projector, 5.63 GB: 9.3 GB, 0.898 of the estimate. The runtime measured + * 11.8 GiB on this Mac; the 6 GiB reserve leaves 10 GiB (10.7 GB) of it, which admits all + * three, and the rank puts MiMo first. * - * Every figure of this row is computed from the model's architecture, none was measured - * (`needSource: 'estimated'`), so this Mac is on the edge either way. Only a run of the 9B on - * it settles which side. + * Every figure of these rows is computed from the model's architecture, none was measured + * (`needSource: 'estimated'`), and no 9B was ever run on such a Mac by us. */ - it('gets no model on the estimated budget: the 9B needs 0.987 of it', async () => { + it('gets Ornith 1.0 9B at 65536 tokens on the estimated budget, tight: the other 9B builds need 0.987 of it', async () => { const hw = await macMini(); expect(hw.chip).toBe('Apple M4'); expect(hw.memClassGb).toBe(16); const result = recommend(hw, STOCK, ALL_INSTALLED); - expect(result).toMatchObject({ verdict: 'none', reason: 'memory', budgetSource: 'estimated' }); - expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.noneMemory, vars: { memory: '16' } }); - expect(result.pick).toBeUndefined(); + expect(result).toMatchObject({ verdict: 'limited', budgetSource: 'estimated' }); + expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.headlineLimited }); + expect(result.notes).toEqual([{ text: LOCAL_MODEL_LABELS.noteUntested }]); // floor(17,179,869,184 x 2/3) - 1 GiB. expect(result.budgetBytes).toBe(10_379_504_298); expect(result.budgetGb).toBe('10.4'); + expect(result.pick).toMatchObject({ + modelId: 'ornith:9b-q4_k_m', + modelLabel: 'Ornith 1.0 9B', + license: 'MIT', + context: 65536, + cli: 'opencode', + // 5,629,110,568 + 17,408 x 65,536 + 2,547,960,320. + needBytes: 9_317_921_576, + needGb: '9.3', + memory: { modelBytes: 8_177_070_888, contextBytes: 1_140_850_688, peakBytes: 0 }, + downloadGb: '5.6', + budgetShare: 0.898, + fit: 'tight', + // 9.3 x 120 / 100. + speed: 'ok', + tokensPerSecond: 11.2, + tested: false, + needSource: 'estimated', + tag: { kind: 'best-tight', label: 'Best fit, tight' }, + }); + // It is the only model that fits, and only at this context: at 131072 it needs 10.5 GB. + expect(order(result)).toEqual(['ornith:9b-q4_k_m@65536']); + expect(modelFitsBudget(hw, STOCK, 'ornith:9b-q4_k_m', 131072)).toBe(false); const nineB = stockModel('mimo-v2.6:9b-q4_k_m'); // 6,550,811,033 + 17,408 x 65,536 + 2,547,960,320. expect(needBytes(nineB, 65536)).toBe(10_239_622_041); @@ -746,7 +773,7 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { expect(needBytes(nineB, 65536) / result.budgetBytes).toBeLessThan(0.99); }); - it('would need 11.4 GB at 131072 tokens, more than the whole budget', async () => { + it('would need 11.4 GB for MiMo at 131072 tokens, more than the whole budget', async () => { const hw = await macMini(); const nineB = stockModel('mimo-v2.6:9b-q4_k_m'); const budget = deviceBudget(hw).budgetBytes; @@ -756,7 +783,7 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { expect(modelFitsBudget(hw, STOCK, 'mimo-v2.6:9b-q4_k_m', 65536)).toBe(false); }); - it('gets a small model with the budget its runtime measured: the 9B at 65536 tokens, tight', async () => { + it('gets MiMo with the budget its runtime measured: at 65536 tokens, tight', async () => { // The runtime reported 11.8 GiB here; the 6 GiB reserve leaves 10 GiB (10.7 GB) of it. const result = recommend(await macMini(), STOCK, ALL_INSTALLED, { measuredTotalBytes: MACMINI_MEASURED_TOTAL }); expect(result).toMatchObject({ @@ -785,13 +812,20 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { needSource: 'estimated', tag: { kind: 'best-tight', label: 'Best fit, tight' }, }); - // Its base model, the same figures, is the one alternative. - expect(order(result)).toEqual(['mimo-v2.6:9b-q4_k_m@65536', 'qwen3.5:9b-q4_k_m@65536']); + // The other two 9B builds follow by rank. The Ornith build is small enough for 131072 + // tokens here (10.5 GB, 0.974), and a model is listed at the largest context it fits at. + expect(order(result)).toEqual([ + 'mimo-v2.6:9b-q4_k_m@65536', + 'ornith:9b-q4_k_m@131072', + 'qwen3.5:9b-q4_k_m@65536', + ]); }); it('gets the same with Claude Code as its only coding agent, which also needs 65536 tokens', async () => { const hw = await macMini(); - expect(recommend(hw, STOCK, [fact('claude')])).toMatchObject({ verdict: 'none', reason: 'memory' }); + const estimated = recommend(hw, STOCK, [fact('claude')]); + expect(estimated.verdict).toBe('limited'); + expect(estimated.pick).toMatchObject({ modelId: 'ornith:9b-q4_k_m', context: 65536, cli: 'claude' }); const measured = recommend(hw, STOCK, [fact('claude')], { measuredTotalBytes: MACMINI_MEASURED_TOTAL }); expect(measured.verdict).toBe('limited'); expect(measured.pick).toMatchObject({ modelId: 'mimo-v2.6:9b-q4_k_m', context: 65536, cli: 'claude' }); @@ -803,12 +837,13 @@ describe('every memory size, with the stock catalogue', () => { // size. A change of a catalogue row or of a threshold that moves one of these answers has to // change this table on purpose. docs/local-models.md has the same table for people. const EXPECTED: [soldGb: number, chip: string, answer: string][] = [ - // The smallest need at 65536 tokens is 10.2 GB; the budget is 2.1 GB. + // The smallest need at 65536 tokens is 9.3 GB; the budget is 2.1 GB. [8, 'Apple M2', 'none memory'], - // The 9B needs 10.2 GB at 65536 tokens, 0.987 of the 10.4 GB estimate: just out. A runtime - // that measured the budget a real 16 GB Mac measured admits it (see the Mac mini above). - [16, 'Apple M4', 'none memory'], - [16, 'Apple M1', 'none memory'], + // The 9B build without a vision projector needs 9.3 GB at 65536 tokens, 0.898 of the 10.4 GB + // estimate. The other two need 10.2 GB, 0.987: just out, until a runtime measures what a + // real 16 GB Mac measured (see the Mac mini above). On an M1 (68 GB/s) a 9B is slow: 6.3. + [16, 'Apple M4', 'limited ornith:9b-q4_k_m@65536 tight ok'], + [16, 'Apple M1', 'limited ornith:9b-q4_k_m@65536 tight slow'], // 11.4 GB at 131072 tokens, 0.964 of 11.8 GB, and nothing of 12 billion fits: `limited`. [18, 'Apple M3 Pro', 'limited mimo-v2.6:9b-q4_k_m@131072 tight ok'], // On a base chip with 100 GB/s the 9B computes to 9.3 tokens per second: slow. @@ -861,15 +896,15 @@ describe('every memory size, with the stock catalogue', () => { expect(result.headline).toEqual({ text: LOCAL_MODEL_LABELS.noneMemory, vars: { memory: '8' } }); // The smallest model a harness can work with needs more than four times the budget. const smallest = Math.min(...STOCK.models.map((m) => needBytes(m, 65536))); - expect(formatGb(smallest)).toBe('10.2'); + expect(formatGb(smallest)).toBe('9.3'); expect(smallest / result.budgetBytes).toBeGreaterThan(4); }); - it('gives no model to 8 and 16 GB, a small model to every Mac of 18 and 24 GB, and every Mac from 36 GB a model of 27 billion or more', () => { + it('gives no model to 8 GB, a small model to every Mac from 16 to 24 GB, and every Mac from 36 GB a model of 27 billion or more', () => { for (const hw of SWEEP) { const result = recommend(hw, STOCK, ALL_INSTALLED); const label = `${hw.memClassGb} GB ${hw.chip}`; - if (hw.memClassGb <= 16) expect(result.reason, label).toBe('memory'); + if (hw.memClassGb <= 8) expect(result.reason, label).toBe('memory'); else if (hw.memClassGb <= 24) { expect(result.verdict, label).toBe('limited'); expect(result.pick?.paramsB, label).toBe(9); @@ -2125,8 +2160,12 @@ describe('the measured budget, end to end', () => { const stepped = order(estimated).filter((entry, index) => order(measured)[index] !== entry); expect(stepped).toEqual(['qwen3.8:27b-q8_0@65536']); expect(order(measured)).toContain('qwen3.8:27b-q8_0@131072'); - expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303, 0.853, 0.975, 0.576]); - expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283, 0.798, 0.976, 0.539]); + expect(rows(estimated).map((c) => c.budgetShare)).toEqual([ + 0.957, 0.767, 0.692, 0.303, 0.278, 0.303, 0.853, 0.975, 0.576, + ]); + expect(rows(measured).map((c) => c.budgetShare)).toEqual([ + 0.895, 0.717, 0.647, 0.283, 0.26, 0.283, 0.798, 0.976, 0.539, + ]); for (const candidate of rows(measured)) { expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3); }