From 492df776d047774bc598d7911d2e67375308c949 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Fri, 6 Nov 2026 22:16:37 +0100 Subject: [PATCH] feat(local-model): judge the fit on the measured peak and rank a tested Mac first The catalogue rows carry the figures measured on a Mac (the MLX build of qwen3.6:35b-a3b is 23,105,798,956 bytes, about 37 KB held per context token, about 3 GB of runner overhead, about 5 GB more for a moment while a long prompt is processed). A model's need is now its peak. Rows that were not measured use the same per-token and overhead figures and say `estimated`. qwen3.6:35b-a3b gets the tested entry for an Apple M5 Pro with 48 GB and OpenCode, with its measured 67.7 tokens per second, so that row may say "Tested on a Mac like this". The ranking changes with it: a pair tested on a Mac like this one comes first, then speed (ok, unknown, slow), then model rank, harness rank and the larger context. The fit grade decides whether a row is listed and what it says, it no longer sorts, so a model is offered at the largest context it fits at. recommend() takes the budget the runtime measured (`budgetSource: measured`), and modelFitsBudget() judges one model against it. Candidate rows now always carry `needSource`, `harnesses`, `installBytes`, `installGb` and, where one applies, `tag`. `kvBytesPerToken` and the prompt cache tiers are gone. Co-Authored-By: Claude Fable 5.1 --- src/config/local-model-catalog.ts | 116 +-- src/local-model/format.ts | 12 +- src/local-model/recommend.ts | 224 ++++-- src/types/local-model.ts | 121 +-- test/local-model-catalog.test.ts | 112 ++- test/local-model-fits-budget.test.ts | 48 +- test/local-model-recommend.test.ts | 1110 +++++++++++++++++++------- 7 files changed, 1241 insertions(+), 502 deletions(-) diff --git a/src/config/local-model-catalog.ts b/src/config/local-model-catalog.ts index a66a111e..a1e578b9 100644 --- a/src/config/local-model-catalog.ts +++ b/src/config/local-model-catalog.ts @@ -3,8 +3,9 @@ * knows, as a compiled constant validated by a Zod schema. * * Rules: - * - There is NO override file and no runtime download data in this stage: nothing here reads - * the data dir, and a data file can never decide what gets recommended. + * - There is NO override file and no runtime download data here: nothing in this file reads + * the data dir, and a data file can never decide what gets recommended. Which runtime build + * is downloaded is a compiled constant of its own (src/local-model/runtime.ts). * - A model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`), so it can * only ever name a model of the default registry and namespace. * - A harness is a ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`). The `cli` is a CLI @@ -13,7 +14,7 @@ * src/local-model/harness-facts.ts, never here. * - The schema checks more than fields. Across rows it requires: unique model ids and ranks, * unique harness `cli`s and ranks, `contexts` largest first, a `tested` entry naming a harness - * row, `chips` most specific first, and `promptCacheTiers` ascending from 0. + * row, and `chips` most specific first. * - `getLocalModelCatalog()` parses the constant once, on first use and never at import, and * throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses, * so a bad row fails the gate and never reaches the route. @@ -22,26 +23,44 @@ * * | Figure | Source | Checked | * |----------------------|-----------------------------------------------------------|--------------------------------------| - * | `id`, `bytes` | registry manifest of the tag, sum of its layer sizes | yes, to the byte, by the facts review | + * | `id` | the registry's tag | yes, by the facts review | + * | `bytes` | `qwen3.6:35b-a3b`: the total of a real pull on a Mac | that row: measured. It is an MLX | + * | | (23,105,798,956). Every other row: the sum of the layer | build on a Mac, 2% above its | + * | | sizes of the tag's registry manifest | manifest. The others: not on a Mac | + * | `heldBytesPerToken`, | measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with | that row: measured. The others carry | + * | `overheadBytes`, | 48 GB: 36,000 to 38,000 bytes held per token of context, | the SAME three figures, not their | + * | `peakExtraBytes` | 2.5 to 3 GB beside the weights, and about 5 GB more for a | own: nobody ran them. `figures` says | + * | | moment while a long prompt is processed | which row is which | * | `license` | the model card (all Apache-2.0) | read by the research, not re-checked | * | `paramsB` | 27.8 from the model card; the others are the count in the | no. Only the 12 billion line reads | * | | model's name | it, and no row is near that line | - * | `kvBytesPerToken` | COMPUTED from each model's config: 2 x full-attention | no. Not measured in a running | - * | | layers x KV heads x head size x 2 bytes | runtime. The review called the | - * | | | gpt-oss figure plausible, no more | * | `contexts`, `rank` | a judgement of the design | no | * | `speedPer100Gbps` | one published run on an M1 Max (400 GB/s): generation | the run was read. It stands for less | * | | only, short prompt. The two q8_0 rows are SCALED by file | than a session: no long prompt, no | * | | size, not measured | thinking tokens | + * | `tested` | our own run of the model with a harness on a real Mac | one entry, with its measured speed | * | `minMacosMajor` | the runtime's documentation (macOS 14) | yes, by the facts review | - * | `promptCacheTiers` | a decision of the design, not a measurement | no | * | harness `minContext` | 65536: the runtime's documentation for both harnesses. | 65536 yes. 32768 no, which is why | * | | 32768: recalled | that row is gated | * | `chips` | the bandwidth column of | yes, read at the source for this | * | | https://github.com/ggml-org/llama.cpp/discussions/4167 | file | * - * `tested` is empty on every row: nobody has run any of these models with a harness on a Mac - * yet. Until a row has an entry, the recommendation never says "Recommended". + * The memory figures decide a lot and rest on ONE measured model, so for every other row they + * can be wrong in either direction: + * - For the 9B rows they may be too HIGH (the 5 GB for a moment was seen with a prompt of + * 89,000 tokens on the 35B model). That errs on the careful side: a small Mac is told + * "none" before it is told to download something that may not fit. + * - For the two dense `qwen3.8:27b` rows the per-token figure may be too LOW: computed from + * its config alone that model holds 64 KiB per token, more than the 37,000 bytes measured + * on the 35B (whose own computed figure was 20 KiB). Their need at a large context is + * therefore the least certain figure of the catalogue. + * A row gets its own figures when somebody measures it. Until then the check after the + * download (the runtime must report the whole model on the GPU) is what catches a row that + * was judged too kindly. + * + * `tested` has one entry: `qwen3.6:35b-a3b` with OpenCode on an Apple M5 Pro with 48 GB, + * 2026-11-06, 67.7 tokens per second. Only a Mac of exactly that chip and memory size is told + * "Recommended" and shown that speed in place of the estimate. */ import { z } from 'zod'; @@ -85,6 +104,7 @@ const TestedOnSchema = z memGb: positiveInt, cli: z.string().min(1).max(24), date: isoDate, + tokensPerSecond: z.number().positive().optional(), }) .strict(); @@ -95,7 +115,10 @@ const LocalModelEntrySchema = z license: z.string().min(1).max(40), paramsB: z.number().positive(), bytes: positiveInt, - kvBytesPerToken: positiveInt, + heldBytesPerToken: positiveInt, + overheadBytes: z.number().int().nonnegative(), + peakExtraBytes: z.number().int().nonnegative(), + figures: z.enum(['measured', 'estimated']), contexts: z.array(positiveInt).min(1), rank: z.number().int(), speedPer100Gbps: z.number().positive().optional(), @@ -120,13 +143,6 @@ const ChipEntrySchema = z }) .strict(); -const PromptCacheTierSchema = z - .object({ - minMemGb: z.number().int().nonnegative(), - bytes: z.number().int().nonnegative(), - }) - .strict(); - /** The index of the first value that already occurred earlier in the list, or -1. */ function firstRepeat(values: readonly T[]): number { const seen = new Set(); @@ -146,7 +162,6 @@ export const LocalModelCatalogSchema: z.ZodType = z schema: z.literal(1), updated: isoDate, minMacosMajor: positiveInt, - promptCacheTiers: z.array(PromptCacheTierSchema).min(1), models: z.array(LocalModelEntrySchema), harnesses: z.array(LocalHarnessEntrySchema), chips: z.array(ChipEntrySchema), @@ -199,23 +214,21 @@ export const LocalModelCatalogSchema: z.ZodType = z break; } } - - // The tiers are read as "the last one whose minimum is not above this Mac's memory", so they - // must start at 0 (every Mac has a tier) and ascend. - if (catalog.promptCacheTiers[0].minMemGb !== 0) { - issue(['promptCacheTiers', 0, 'minMemGb'], 'the first prompt cache tier must start at 0'); - } - for (let i = 1; i < catalog.promptCacheTiers.length; i++) { - if (catalog.promptCacheTiers[i].minMemGb <= catalog.promptCacheTiers[i - 1].minMemGb) { - issue(['promptCacheTiers', i, 'minMemGb'], 'prompt cache tiers must ascend by minMemGb'); - break; - } - } }); -const KIB = 1024; -const MIB = 1024 ** 2; -const GIB = 1024 ** 3; +/** + * The memory figures measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with 48 GB (2026-11-06, + * the MLX runner), in decimal bytes. That row carries them as `measured`; every other row + * carries the same three as `estimated`, see the file header. + */ +const MEASURED_MEMORY = { + /** 36,000 to 38,000 bytes stayed held per token of context after a prompt of 89,000 and one of 167,000 tokens. */ + heldBytesPerToken: 37_000, + /** The runner held 2.5 to 3 GB beside the weights. */ + overheadBytes: 3_000_000_000, + /** While a long prompt was processed the peak stood about 5 GB above what stayed held. */ + peakExtraBytes: 5_000_000_000, +} as const; /** * The catalogue that ships with Codeman. Never read directly outside this file and its test: @@ -226,12 +239,6 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { updated: '2026-11-06', // The oldest macOS the model runtime supports. minMacosMajor: 14, - // Host memory the runtime's prompt cache may take beside the model: 512 MiB on a Mac below - // 32 GB, 2 GiB from 32 GB. Part of the need of every candidate. - promptCacheTiers: [ - { minMemGb: 0, bytes: 512 * MIB }, - { minMemGb: 32, bytes: 2 * GIB }, - ], // Ordered by rank. The higher rank wins among models that fit equally well and are equally // fast. `flint:40b` is left out on purpose: its plain tag is the slow build on a Mac and it // has an open bug with tool arguments. @@ -242,7 +249,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 9, bytes: 6_550_825_373, - kvBytesPerToken: 32 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [65536, 32768], rank: 1, // 37.1 tokens per second measured on 400 GB/s. @@ -255,7 +263,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 9, bytes: 10_449_217_949, - kvBytesPerToken: 32 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [65536, 32768], rank: 2, // Not measured: the 9.3 of the row above scaled by file size. @@ -268,7 +277,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 20, bytes: 13_793_441_427, - kvBytesPerToken: 24 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072, 65536], rank: 3, // No speed figure: its speed is unknown on every chip. @@ -279,13 +289,18 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { label: 'Qwen3.6 35B-A3B', license: 'Apache-2.0', paramsB: 35, - bytes: 22_621_314_161, - kvBytesPerToken: 20 * KIB, + // What a Mac really downloads: an MLX build, 2% above the 22,621,314,161 of the manifest. + bytes: 23_105_798_956, + ...MEASURED_MEMORY, + figures: 'measured', contexts: [131072, 65536], rank: 4, // 54.5 tokens per second measured on 400 GB/s (3 billion parameters are active per token). + // The MLX build ran faster than this figure predicts on the one Mac we tested (67.7 + // against 41.8), so on other chips the estimate is on the low side. speedPer100Gbps: 13.6, - tested: [], + // Run by us with OpenCode on the MacBook Pro (Mac17,13): a four-step task with tool calls. + tested: [{ chip: 'Apple M5 Pro', memGb: 48, cli: 'opencode', date: '2026-11-06', tokensPerSecond: 67.7 }], }, { id: 'qwen3.8:27b', @@ -293,7 +308,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 27.8, bytes: 17_741_871_939, - kvBytesPerToken: 64 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072, 65536, 32768], rank: 5, // 11.8 tokens per second measured on 400 GB/s. @@ -306,7 +322,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 27.8, bytes: 29_978_241_837, - kvBytesPerToken: 64 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [262144, 131072], rank: 6, // Not measured: the 2.95 of the row above scaled by file size. @@ -319,7 +336,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 122, bytes: 81_370_035_877, - kvBytesPerToken: 24 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072], rank: 7, // No speed figure: its speed is unknown on every chip. diff --git a/src/local-model/format.ts b/src/local-model/format.ts index 4f8db32d..1fb6508b 100644 --- a/src/local-model/format.ts +++ b/src/local-model/format.ts @@ -7,12 +7,12 @@ * `budgetGb`, and the size slots of a `LocalModelMessage`). Nothing else may turn bytes into a * size for display, the browser included: it prints the text it was sent. * - * GiB and MiB exist only for the budget math (reserves, the wired limit, the prompt cache) and - * never reach a person, with one exception: `memClassGb()`, the memory size a Mac is sold as, - * is how the card names the installed memory ("48 GB memory", what About This Mac shows). It - * is a name of the machine, not a computed size: `formatGb()` of the same bytes reads 51.5, - * which no owner recognises. Every computed size (need, budget, download, free disk) is - * `formatGb()`. + * GiB and MiB exist only for the budget math (reserves, the wired limit, the device total the + * runtime reports) and never reach a person, with one exception: `memClassGb()`, the memory + * size a Mac is sold as, is how the card names the installed memory ("48 GB memory", what + * About This Mac shows). It is a name of the machine, not a computed size: `formatGb()` of the + * same bytes reads 51.5, which no owner recognises. Every computed size (need, budget, + * download, free disk) is `formatGb()`. * * Pure: no IO, no state. */ diff --git a/src/local-model/recommend.ts b/src/local-model/recommend.ts index f9845285..752a32df 100644 --- a/src/local-model/recommend.ts +++ b/src/local-model/recommend.ts @@ -3,21 +3,33 @@ * fits this Mac. Pure: no IO, no clock, no CLI registry, no data dir. * * The math, in the order it runs: - * - `deviceBudget()`: L is the wired limit when it is above 0, else 2/3 of memory below 36 GB - * and 3/4 from 36 GB. B = min(L - 1 GiB, memory - reserve), with a reserve of 6 GiB up to - * 18 GB, 8 GiB up to 48 GB and 12 GiB above. - * - `needBytes()`: F = weights + kvBytesPerToken * ctx + 1 GiB + the prompt cache tier of the - * catalogue for this Mac's memory. - * - Fit: F/B up to 0.85 is good, up to 0.98 is tight, above is out. - * - Speed: tokens per second = the model's `speedPer100Gbps` x the chip's bandwidth / 100. - * Below 10 is slow, decided on the unrounded product (9.96 is slow although it prints as - * 10). A chip that is in no chips row, or a model without a speed figure, is UNKNOWN: a + * - `deviceBudget()`. ESTIMATED (all there is before the runtime ran): L is the wired limit + * when it is above 0, else 2/3 of memory below 36 GB and 3/4 from 36 GB, and + * B = min(L - 1 GiB, memory - reserve), with a reserve of 6 GiB up to 18 GB, 8 GiB up to + * 48 GB and 12 GiB above. MEASURED (`RecommendOptions.measuredTotalBytes`, the device total + * the runtime printed when it started): L is that total and B = min(L, memory - reserve), + * without the 1 GiB, because the figure is already what the runtime will really use. + * - `needBytes()`: the PEAK, `bytes + heldBytesPerToken * ctx + overheadBytes + + * peakExtraBytes` of the catalogue row: the weights, what stays held per token of context, + * what the runner holds beside both, and what it takes for a moment while a long prompt is + * processed. Measured for one model on one Mac (see the catalogue header); every other row + * carries the same figures and says `estimated`. + * - Fit: need over budget up to 0.85 is good, up to 0.98 is tight, above is out. The grade + * decides whether a model is offered at all and what its row says. It does NOT sort: the + * need is the worst moment at the full context, so a tight fit is a model that runs, and a + * Mac whose best model is tight should be offered that model, not a smaller one. + * - Speed: a candidate that matches a `tested` entry with a measured speed reports that + * speed. Every other one is estimated: the model's `speedPer100Gbps` x the chip's bandwidth + * / 100. Below 10 is slow, decided on the unrounded figure (9.96 is slow although it prints + * as 10). A chip that is in no chips row, or a model without a speed figure, is UNKNOWN: a * bandwidth is never guessed. * - Candidates: (model, context, harness) with `ctx >= harness.minContext`, the harness not * gated, installed or installable, and accepting the model id. - * - Ranking, lexicographic: good before tight; ok before unknown before slow; the higher model - * rank; the higher harness rank; the larger context. The pick is the first candidate, the - * alternatives are the next four MODELS, each at its own best context and harness. + * - Ranking, lexicographic: tested on a Mac like this before not tested (a measurement beats + * a formula); ok before unknown before slow; the higher model rank; the higher harness + * rank; the larger context. The pick is the first candidate, the alternatives are the next + * four MODELS, each at its own best context and harness, so a model is always offered at + * the largest context it fits at. * - Disk: a model whose download would leave less than 10 GiB free is not offered. It is not * dropped in silence: the best-ranked model left out that way is named in `noteDiskDropped` * with what is missing. @@ -32,8 +44,8 @@ * installable). * 6. `memory`: models fit, but none at a context a usable harness needs. Its own sentence * (`noneMemoryContext`) with the budget and the smallest need at such a context, because - * this Mac can miss by a hair (the 16 GB Mac mini: 10.3 of 10.4 GB) and "too little - * memory" would hide that the answer rests on an estimate. + * "too little memory" would hide that a model does fit at a smaller context and that the + * answer rests on estimates. * 7. `no-harness`: models fit at such a context, but no usable harness accepts their id. * 8. `disk`: for every remaining model, free disk minus the download is below 10 GiB (an * unknown free disk does not block). @@ -42,16 +54,20 @@ * - Verdict `limited` when NO model with 12 billion parameters or more fits this Mac's memory * at a context a usable harness works at. It is a statement about the Mac ("This Mac can run * a small model"), so it is judged on everything that fits, BEFORE the disk filter and - * whatever the pick is: a 36 GB Mac whose pick is a 9B model while a 20B, a 27B and a 35B - * are listed under it is not a small-model Mac, and a 48 GB Mac does not become one because - * its disk is full. A small pick under verdict `ok` carries `noteSmallPick` instead. + * whatever the pick is: a 36 GB Mac whose pick is a 9B model while a 20B is listed under it + * is not a small-model Mac, and a 48 GB Mac does not become one because its disk is full. A + * small pick under verdict `ok` carries `noteSmallPick` instead. * - The word "Recommended" (`LOCAL_MODEL_LABELS.headlineTested`) only for verdict `ok` with a * pick whose model has a `tested` entry for exactly this chip, this memory size and this * harness. Otherwise `headlineOk` (or `headlineLimited`) plus `noteUntested`. - * - Every label in the result comes from `LOCAL_MODEL_LABELS`, every computed size text from - * `formatGb()`. The one size that is not decimal GB is the installed memory in `noneMemory`: - * `HardwareFacts.memClassGb`, the size the Mac is sold as. No number is spliced into a - * sentence here. + * - Every row carries what the chooser needs and nothing the browser must work out: its one + * tag (`tagFor()`), every harness it can run with (`Candidate.harnesses`, the flat harness + * fields first), and the whole download of an install (`installBytes`: the model plus + * `RecommendOptions.runtimeDownloadBytes`). + * - Every label in the result comes from `LOCAL_MODEL_LABELS` or `LOCAL_MODEL_TAG_LABELS`, + * every computed size text from `formatGb()`. The one size that is not decimal GB is the + * installed memory in `noneMemory`: `HardwareFacts.memClassGb`, the size the Mac is sold + * as. No number is spliced into a sentence here. * - No CLI id is ever compared with a literal: a harness is a catalogue row plus a * `HarnessFact`. */ @@ -59,7 +75,11 @@ import { LOCAL_MODEL_LABELS, LOCAL_MODEL_NONE_LABELS, + LOCAL_MODEL_TAG_LABELS, type Candidate, + type CandidateHarness, + type CandidateTag, + type CandidateTagKind, type DeviceBudget, type FitGrade, type HardwareFacts, @@ -73,6 +93,7 @@ import { type Recommendation, type RecommendOptions, type SpeedGrade, + type TestedOn, } from '../types.js'; import { chipBandwidthGbps } from '../config/local-model-catalog.js'; import { formatGb, GIB, MIB } from './format.js'; @@ -81,14 +102,12 @@ import { formatGb, GIB, MIB } from './format.js'; export const FIT_GOOD_MAX_SHARE = 0.85; /** Need over budget up to this share is a `tight` fit; above it the model is out. */ export const FIT_TIGHT_MAX_SHARE = 0.98; -/** An estimated speed below this many tokens per second is `slow`. */ +/** A speed below this many tokens per second is `slow`. */ export const SLOW_BELOW_TOKENS_PER_SECOND = 10; /** When no model that fits has at least this many billion parameters, the verdict is `limited`. A pick below it is a "small model". */ export const LIMITED_BELOW_PARAMS_B = 12; /** What must stay free on the disk after the model download. */ export const DISK_HEADROOM_BYTES = 10 * GIB; -/** What the runtime needs beside the weights, the KV cache and the prompt cache (compute buffers). */ -export const RUNTIME_OVERHEAD_BYTES = GIB; /** How many other models are listed under the pick. */ export const MAX_ALTERNATIVES = 4; @@ -102,30 +121,39 @@ function reserveBytes(memClassGb: number): number { return 12 * GIB; } -/** The memory a model may use on this Mac: the device limit L, the budget B and where L came from. */ -export function deviceBudget(hw: HardwareFacts): DeviceBudget { +/** A byte count a caller supplied, when it is one: finite and above 0. */ +function positiveBytes(value: number | undefined): number | undefined { + return value !== undefined && Number.isFinite(value) && value > 0 ? Math.floor(value) : undefined; +} + +/** + * The memory a model may use on this Mac: the device limit L, the budget B and where L came + * from. With `options.measuredTotalBytes` the limit is what the runtime itself reported and + * the 1 GiB the estimate keeps back is not subtracted; the reserve bounds both. + */ +export function deviceBudget(hw: HardwareFacts, options: RecommendOptions = {}): DeviceBudget { const memBytes = Number.isFinite(hw.memBytes) && hw.memBytes > 0 ? hw.memBytes : 0; + const afterReserve = memBytes - reserveBytes(hw.memClassGb); + const measured = positiveBytes(options.measuredTotalBytes); + if (measured !== undefined) { + return { limitBytes: measured, budgetBytes: Math.max(0, Math.min(measured, afterReserve)), source: 'measured' }; + } // 0 means "not set" (both test Macs report 0): only a value above 0 is a limit. const wired = hw.wiredLimitMb !== undefined && Number.isFinite(hw.wiredLimitMb) && hw.wiredLimitMb > 0; const limitBytes = wired ? Math.floor((hw.wiredLimitMb as number) * MIB) : Math.floor(memBytes * (hw.memClassGb < LARGE_LIMIT_FROM_GB ? 2 / 3 : 3 / 4)); - const budgetBytes = Math.max(0, Math.min(limitBytes - GIB, memBytes - reserveBytes(hw.memClassGb))); + const budgetBytes = Math.max(0, Math.min(limitBytes - GIB, afterReserve)); return { limitBytes, budgetBytes, source: wired ? 'wired-limit' : 'estimated' }; } -/** The prompt cache of the last tier whose minimum is not above this Mac's memory size. */ -function promptCacheBytes(catalog: LocalModelCatalog, memClassGb: number): number { - let bytes = 0; - for (const tier of catalog.promptCacheTiers) { - if (tier.minMemGb <= memClassGb) bytes = tier.bytes; - } - return bytes; -} - -/** F: the bytes `model` needs in memory at a context of `ctx` tokens on this Mac. */ -export function needBytes(model: LocalModelEntry, ctx: number, catalog: LocalModelCatalog, hw: HardwareFacts): number { - return model.bytes + model.kvBytesPerToken * ctx + RUNTIME_OVERHEAD_BYTES + promptCacheBytes(catalog, hw.memClassGb); +/** + * The bytes `model` needs in memory at a context of `ctx` tokens, at its PEAK: the weights, + * what stays held for the context, what the runner holds beside them, and what it takes for a + * moment while a long prompt is processed. It does not depend on the Mac. + */ +export function needBytes(model: LocalModelEntry, ctx: number): number { + return model.bytes + model.heldBytesPerToken * ctx + model.overheadBytes + model.peakExtraBytes; } function fitGrade(need: number, budgetBytes: number): FitGrade { @@ -139,12 +167,10 @@ function fitGrade(need: number, budgetBytes: number): FitGrade { /** * Whether the catalogue model `modelId` at `context` tokens fits this Mac's budget (a `good` * or a `tight` fit). The install job asks this right after the runtime started, with the - * budget the runtime measured, BEFORE it downloads the model: a `false` ends the job with - * `does-not-fit`. It is the same judgement `recommend()` makes for a candidate, without the - * ranking, so a model that fits is never ruled out for being listed sixth. - * - * `options` is the stage 2 input, as for `recommend()`; nothing reads it yet, so the budget - * is still the estimate. False for a model the catalogue does not have. + * budget the runtime measured (`options.measuredTotalBytes`), BEFORE it downloads the model: a + * `false` ends the job with `does-not-fit`. It is the same judgement `recommend()` makes for a + * candidate, without the ranking, so a model that fits is never ruled out for being listed + * sixth. False for a model the catalogue does not have. */ export function modelFitsBudget( hw: HardwareFacts, @@ -153,10 +179,9 @@ export function modelFitsBudget( context: number, options: RecommendOptions = {} ): boolean { - void options; const model = catalog.models.find((entry) => entry.id === modelId); - if (!model) return false; - return fitGrade(needBytes(model, context, catalog, hw), deviceBudget(hw).budgetBytes) !== 'out'; + if (!model || !Number.isFinite(context) || context <= 0) return false; + return fitGrade(needBytes(model, context), deviceBudget(hw, options).budgetBytes) !== 'out'; } /** @@ -189,13 +214,12 @@ interface UsableHarness { fact: HarnessFact; } -const FIT_ORDER: Record = { good: 0, tight: 1 }; const SPEED_ORDER: Record = { ok: 0, unknown: 1, slow: 2 }; -/** Best first. */ +/** Best first. The fit grade is not a key: every candidate here fits, see the file header. */ function compareRanked(a: Ranked, b: Ranked): number { return ( - FIT_ORDER[a.candidate.fit] - FIT_ORDER[b.candidate.fit] || + Number(b.candidate.tested) - Number(a.candidate.tested) || SPEED_ORDER[a.candidate.speed] - SPEED_ORDER[b.candidate.speed] || b.modelRank - a.modelRank || b.harnessRank - a.harnessRank || @@ -212,27 +236,53 @@ function accepts(fact: HarnessFact, modelId: string): boolean { } } +/** What the chooser's "Runs with" select needs to know about one harness. */ +function candidateHarness(harness: UsableHarness): CandidateHarness { + const { row, fact } = harness; + return { + cli: row.cli, + cliLabel: fact.label, + installed: fact.installed, + // Only a harness that still has to be installed has a command to show. + ...(!fact.installed && fact.installCommand ? { installCommand: fact.installCommand } : {}), + ...(row.slowFirstTurn ? { slowFirstTurn: true } : {}), + }; +} + +/** + * The entry that makes this model "tested on a Mac like this": exactly the same chip string, + * the same memory size as sold, the same harness. + */ +function testedEntry(model: LocalModelEntry, hw: HardwareFacts, cli: string): TestedOn | undefined { + return model.tested.find((entry) => entry.chip === hw.chip && entry.memGb === hw.memClassGb && entry.cli === cli); +} + function buildCandidate( model: LocalModelEntry, ctx: number, harness: UsableHarness, + /** Every usable harness that can run this model at this context, `harness` among them. */ + runsWith: readonly UsableHarness[], need: number, fit: Candidate['fit'], hw: HardwareFacts, catalog: LocalModelCatalog, - budgetBytes: number + budgetBytes: number, + runtimeDownloadBytes: number ): Candidate { + const testedOn = testedEntry(model, hw, harness.row.cli); const gbps = chipBandwidthGbps(catalog.chips, hw.chip); + // A speed measured on a Mac like this one beats the estimate from the chip's bandwidth. const rawSpeed = - gbps !== undefined && model.speedPer100Gbps !== undefined ? (model.speedPer100Gbps * gbps) / 100 : undefined; + testedOn?.tokensPerSecond ?? + (gbps !== undefined && model.speedPer100Gbps !== undefined ? (model.speedPer100Gbps * gbps) / 100 : undefined); // Graded on the unrounded figure: 9.96 rounds to 10.0 for the payload and is still slow. const speed: SpeedGrade = rawSpeed === undefined ? 'unknown' : rawSpeed < SLOW_BELOW_TOKENS_PER_SECOND ? 'slow' : 'ok'; const tokensPerSecond = rawSpeed === undefined ? undefined : round(rawSpeed, 1); - // "A Mac like this" is exact: the same chip string, the same memory size as sold, the same harness. - const tested = model.tested.some( - (entry) => entry.chip === hw.chip && entry.memGb === hw.memClassGb && entry.cli === harness.row.cli - ); + // The harness of the flat fields first, then the others, best first. + const others = runsWith.filter((other) => other !== harness).sort((a, b) => b.row.rank - a.row.rank); + const installBytes = model.bytes + runtimeDownloadBytes; return { modelId: model.id, modelLabel: model.label, @@ -254,10 +304,39 @@ function buildCandidate( ...(speed === 'slow' ? { speedLabel: LOCAL_MODEL_LABELS.speedSlow } : {}), ...(speed === 'unknown' ? { speedLabel: LOCAL_MODEL_LABELS.speedUnknown } : {}), ...(tokensPerSecond !== undefined ? { tokensPerSecond } : {}), - tested, + tested: testedOn !== undefined, + needSource: model.figures, + harnesses: [harness, ...others].map(candidateHarness), + installBytes, + installGb: formatGb(installBytes), }; } +/** + * The one tag of a row: the first kind that applies, in the order tested, best (the pick), + * tight, slow, small. `undefined` when none does. + */ +function tagFor(candidate: Candidate, isPick: boolean): CandidateTag | undefined { + const kind: CandidateTagKind | undefined = candidate.tested + ? 'tested' + : isPick + ? 'best' + : candidate.fit === 'tight' + ? 'tight' + : candidate.speed === 'slow' + ? 'slow' + : candidate.paramsB < LIMITED_BELOW_PARAMS_B + ? 'small' + : undefined; + return kind === undefined ? undefined : { kind, label: LOCAL_MODEL_TAG_LABELS[kind] }; +} + +/** `candidate` with its tag, when it has one. */ +function withTag(candidate: Candidate, isPick: boolean): Candidate { + const tag = tagFor(candidate, isPick); + return tag ? { ...candidate, tag } : candidate; +} + /** A `none` answer. `text` replaces the reason's default sentence (the `memory` reason has three). */ function none( reason: NoneReason, @@ -281,9 +360,9 @@ function none( * The recommendation for this Mac. `harnessFacts` holds one fact per catalogue harness the * caller could resolve; a catalogue harness without a fact is not offered. * - * `options` is the stage 2 input (`RecommendOptions`: the budget the runtime measured, and - * the runtime download an install still needs). The parameter is in place so callers compile - * against the final signature; nothing reads it yet. + * `options` is what the caller knows beyond the probe: the budget the runtime measured (the + * answer then says `budgetSource: 'measured'`), and the runtime download an install still + * needs (it is added to every row's `installBytes`). */ export function recommend( hw: HardwareFacts, @@ -291,13 +370,13 @@ export function recommend( harnessFacts: readonly HarnessFact[], options: RecommendOptions = {} ): Recommendation { - void options; // Off macOS and on an Intel Mac there is no budget to compute: a model has no memory to run in. const noBudget: DeviceBudget = { limitBytes: 0, budgetBytes: 0, source: 'estimated' }; if (hw.platform !== 'darwin') return none('not-macos', noBudget); if (!isAppleSilicon(hw)) return none('intel', noBudget); - const budget = deviceBudget(hw); + const budget = deviceBudget(hw, options); + const runtimeDownloadBytes = positiveBytes(options.runtimeDownloadBytes) ?? 0; if (hw.macosMajor !== undefined && hw.macosMajor < catalog.minMacosMajor) { return none('macos', budget, { version: hw.macosVersion ?? String(hw.macosMajor), @@ -309,7 +388,7 @@ export function recommend( const fitting: { model: LocalModelEntry; ctx: number; need: number; fit: Candidate['fit'] }[] = []; for (const model of catalog.models) { for (const ctx of model.contexts) { - const need = needBytes(model, ctx, catalog, hw); + const need = needBytes(model, ctx); const fit = fitGrade(need, budget.budgetBytes); if (fit !== 'out') fitting.push({ model, ctx, need, fit }); } @@ -335,14 +414,14 @@ export function recommend( usable.filter((harness) => entry.ctx >= harness.row.minContext).map((harness) => ({ ...entry, harness })) ); if (atHarnessContext.length === 0) { - // Models fit, but only below the context a harness needs. Say both numbers: the smallest - // need at such a context can sit a hair over the line, and the budget is an estimate. + // Models fit, but only below the context a harness needs. Say both numbers: the reader + // sees how far apart they are, and that both are estimates. const smallestContext = Math.min(...usable.map((harness) => harness.row.minContext)); let smallestNeed: number | undefined; for (const model of catalog.models) { for (const ctx of model.contexts) { if (ctx < smallestContext) continue; - const need = needBytes(model, ctx, catalog, hw); + const need = needBytes(model, ctx); if (smallestNeed === undefined || need < smallestNeed) smallestNeed = need; } } @@ -379,11 +458,15 @@ export function recommend( entry.model, entry.ctx, entry.harness, + accepted + .filter((other) => other.model === entry.model && other.ctx === entry.ctx) + .map((other) => other.harness), entry.need, entry.fit, hw, catalog, - budget.budgetBytes + budget.budgetBytes, + runtimeDownloadBytes ), model: entry.model, modelRank: entry.model.rank, @@ -398,7 +481,8 @@ export function recommend( } const offered = bestPerModel.filter((entry) => hasRoom(entry.model)).map((entry) => entry.candidate); const leftOutForDisk = bestPerModel.find((entry) => !hasRoom(entry.model)); - const [pick, ...others] = offered; + const pick = withTag(offered[0], true); + const alternatives = offered.slice(1, 1 + MAX_ALTERNATIVES).map((candidate) => withTag(candidate, false)); // About the Mac, not the pick, and judged before the disk filter (see the file header). const limited = !accepted.some((entry) => entry.model.paramsB >= LIMITED_BELOW_PARAMS_B); @@ -427,7 +511,7 @@ export function recommend( headline: { text: headline }, notes, pick, - alternatives: others.slice(0, MAX_ALTERNATIVES), + alternatives, budgetBytes: budget.budgetBytes, budgetGb: formatGb(budget.budgetBytes), budgetSource: budget.source, diff --git a/src/types/local-model.ts b/src/types/local-model.ts index 3db7709e..4e146941 100644 --- a/src/types/local-model.ts +++ b/src/types/local-model.ts @@ -15,10 +15,10 @@ * call time; it holds a function, so it never goes over HTTP). * - `Recommendation`, `Candidate`, `DeviceBudget`: src/local-model/recommend.ts (pure). * - * Fields marked "Stage 2" were added for the install. They are OPTIONAL in the type only so - * that the stage 1 catalogue and `recommend()` keep compiling until their owner fills them: - * once `recommend()` and the catalogue rows carry them, they become required, and a consumer - * (the chooser window, the install job) may rely on every one of them. + * A field that is optional here is optional by meaning, and its comment says when it is + * absent (a tested entry without a measured speed, a harness that needs no install command, a + * row without a tag). Everything else is always there, so the chooser window and the install + * job may rely on it. * * Three rules every producer and consumer follows: * @@ -97,7 +97,11 @@ export interface HardwareFacts { // ---------- Catalogue ---------- -/** One Mac a model and harness pair was really run on by us. A model row with a matching entry may be called "Recommended". */ +/** + * One Mac a model and harness pair was really run on by us. A candidate that matches an entry + * (the same chip string, the same memory size as sold, the same harness) is `tested`: it ranks + * before every candidate that is not, and as the pick it may be called "Recommended". + */ export interface TestedOn { /** The chip, as `machdep.cpu.brand_string` reports it ('Apple M4'). */ chip: string; @@ -108,7 +112,7 @@ export interface TestedOn { /** ISO date of the test ('2026-11-06'). */ date: string; /** - * Stage 2. The generation speed measured in that test, in tokens per second (67.7 for the + * The generation speed measured in that test, in tokens per second (67.7 for the * 2026-11-06 run on an Apple M5 Pro with 48 GB). A candidate that matches this entry reports * this figure as `Candidate.tokensPerSecond` in place of the estimate from the chip's memory * bandwidth: a measurement on the same Mac beats a formula. @@ -129,46 +133,41 @@ export interface LocalModelEntry { /** * Size of the weights in bytes: both the download and what they take in memory. * - * Stage 2: the size of the build the runtime really downloads ON A MAC. That is not always + * It is the size of the build the runtime really downloads ON A MAC. That is not always * the sum of the registry manifest: `qwen3.6:35b-a3b` is an MLX build of 23,105,798,956 bytes * on Apple silicon, 2% more than the manifest's 22,621,314,161. A row whose Mac size nobody * measured keeps the manifest sum and says so in `figures`. */ bytes: number; /** - * Stage 1 figure: the bytes of KV cache one token of context costs, COMPUTED from the model's - * config. Measured on a Mac it was too low by almost half (20 KiB against about 37 KB held), - * so the need is computed from `heldBytesPerToken` once that is filled. Kept only until - * `recommend()` stops reading it; it is deleted together with `promptCacheTiers` then. + * Bytes that stay HELD in memory per token of context once the token was processed (measured + * on the MLX runner: 36,000 to 38,000, so about 37,000). A figure computed from a model's + * config alone came out too low by almost half, so a row that was not measured carries the + * measured row's figure and says so in `figures`. */ - kvBytesPerToken: number; + heldBytesPerToken: number; + /** What the runner holds beside the weights and the context (measured: about 3 GB). */ + overheadBytes: number; /** - * Stage 2. Bytes that stay HELD in memory per token of context once the token was processed - * (measured on the MLX runner: 36,000 to 38,000, so about 37,000). - */ - heldBytesPerToken?: number; - /** Stage 2. What the runner holds beside the weights and the context (measured: about 3 GB). */ - overheadBytes?: number; - /** - * Stage 2. The TRANSIENT extra the runner takes while it processes a long prompt, on top of + * The TRANSIENT extra the runner takes while it processes a long prompt, on top of * what stays held (measured: about 5 GB). The fit is judged on the peak: * `bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`. For * `qwen3.6:35b-a3b` at 131072 tokens that is about 36 GB, which is tight on a 48 GB Mac. */ - peakExtraBytes?: number; + peakExtraBytes: number; /** - * Stage 2. Where `bytes` and the three memory figures above come from: `measured` on a Mac by + * Where `bytes` and the three memory figures above come from: `measured` on a Mac by * us, or `estimated` (the manifest size, and the per-token, overhead and peak figures of the * measured row applied to this one). */ - figures?: 'measured' | 'estimated'; + figures: 'measured' | 'estimated'; /** Context lengths (tokens) the model may be run at, largest first. */ contexts: number[]; /** Preference among models that fit: the HIGHER rank wins. A judgement, not a measurement. */ rank: number; /** Measured tokens per second per 100 GB/s of memory bandwidth. Absent: the speed of this model is unknown. */ speedPer100Gbps?: number; - /** Macs this model was really run on. Empty today for every row. */ + /** Macs this model was really run on. Empty for every row but one. */ tested: TestedOn[]; } @@ -201,16 +200,6 @@ export interface ChipEntry { gbps: number; } -/** - * Host memory the runtime's prompt cache takes, by the size of the Mac. Tiers are ordered by - * `minMemGb` ascending and the last tier with `minMemGb <= HardwareFacts.memClassGb` applies. - */ -export interface PromptCacheTier { - /** Smallest memory size (as sold, binary gigabytes) this tier applies to. The first tier has 0. */ - minMemGb: number; - bytes: number; -} - /** The catalogue: a compiled constant, validated by `LocalModelCatalogSchema`. No override file, no download data. */ export interface LocalModelCatalog { schema: 1; @@ -218,7 +207,6 @@ export interface LocalModelCatalog { updated: string; /** Oldest macOS major version a local model runs on. Below it: verdict `none`, reason `macos`. */ minMacosMajor: number; - promptCacheTiers: PromptCacheTier[]; models: LocalModelEntry[]; harnesses: LocalHarnessEntry[]; chips: ChipEntry[]; @@ -241,7 +229,7 @@ export interface HarnessFact { /** Not installed, but the registry has an install command for this platform that needs no administrator rights. */ installable: boolean; /** - * Stage 2. The literal install command the install job would run for this CLI: the STOCK + * The literal install command the install job would run for this CLI: the STOCK * command of the CLI registry for this platform, never one a `clis.json` override supplies. * Present exactly when `installable` is true. The consent screen prints it * (`CandidateHarness.installCommand`). @@ -253,7 +241,12 @@ export interface HarnessFact { // ---------- Recommendation ---------- -/** Need over budget: up to 0.85 is `good`, up to 0.98 is `tight`, above is `out`. */ +/** + * Need over budget: up to 0.85 is `good`, up to 0.98 is `tight`, above is `out`. The need is + * the PEAK, so the grade says how much room is left at the worst moment. It decides whether a + * model is offered at all (`out` never is) and what its row says (`tight`). It does not sort + * the list. + */ export type FitGrade = 'good' | 'tight' | 'out'; /** @@ -280,7 +273,7 @@ export type NoneReason = 'not-macos' | 'intel' | 'memory' | 'disk' | 'macos' | ' * Where the device budget came from. * - `estimated`: a formula over the installed memory. All there is before the runtime ran. * - `wired-limit`: `iogpu.wired_limit_mb` above 0. - * - `measured` (stage 2): the device total the runtime itself printed when it started + * - `measured`: the device total the runtime itself printed when it started * (`RecommendOptions.measuredTotalBytes`). The chooser says "estimated" next to the memory * bar for the first value only. */ @@ -304,8 +297,8 @@ export interface DeviceBudget { } /** - * Stage 2. What the caller of `recommend()` knows beyond the hardware probe. Both members are - * optional and an empty object is the stage 1 behaviour. + * What the caller of `recommend()` knows beyond the hardware probe. Both members are optional: + * an empty object means the budget is the estimate and an install downloads the model alone. */ export interface RecommendOptions { /** @@ -346,7 +339,8 @@ export interface RecommendOptions { * | `noneMacos` | `version`: `HardwareFacts.macosVersion`; `minimum`: the catalogue minimum | * * The word "Recommended" exists only in `headlineTested`, which may be used only when the - * pick's `tested` is true. No catalogue row has a tested entry today, so it is never sent. + * pick's `tested` is true: one catalogue row has a tested entry (`qwen3.6:35b-a3b` on an Apple + * M5 Pro with 48 GB), so one kind of Mac is told so. */ export const LOCAL_MODEL_LABELS = { /** Verdict `ok`, pick not tested by us on a Mac like this one. */ @@ -405,7 +399,7 @@ export interface LocalModelMessage { } /** - * Stage 2. The one tag a model row of the chooser carries, by what it says. + * The one tag a model row of the chooser carries, by what it says. * - `tested`: `Candidate.tested` is true. Replaces every other tag. * - `best`: the pick. The one tag drawn in the accent colour. * - `tight`: `Candidate.fit` is `tight`. @@ -431,7 +425,7 @@ export const LOCAL_MODEL_TAG_LABELS = { /** One of the texts in `LOCAL_MODEL_TAG_LABELS`. */ export type LocalModelTagLabel = (typeof LOCAL_MODEL_TAG_LABELS)[CandidateTagKind]; -/** Stage 2. The tag of a model row: what kind it is (for the styling) and the text to print. */ +/** The tag of a model row: what kind it is (for the styling) and the text to print. */ export interface CandidateTag { kind: CandidateTagKind; /** `LOCAL_MODEL_TAG_LABELS[kind]`. */ @@ -439,7 +433,7 @@ export interface CandidateTag { } /** - * Stage 2. One coding agent a candidate's model can be run with on this Mac, for the "Runs + * One coding agent a candidate's model can be run with on this Mac, for the "Runs * with" select of the chooser. Everything a browser needs is here, so it compares no CLI id. */ export interface CandidateHarness { @@ -477,10 +471,9 @@ export interface Candidate { /** The harness row's `slowFirstTurn`. */ slowFirstTurn?: boolean; /** - * What the model needs in memory at this context. Stage 1: F = weights + kvBytesPerToken * - * context + 1 GiB + prompt cache. Stage 2: the PEAK, `bytes + heldBytesPerToken * context + - * overheadBytes + peakExtraBytes` of the catalogue row. The memory bar of the chooser is - * filled to `budgetShare`, which is this over the budget. + * What the model needs in memory at this context: the PEAK, `bytes + heldBytesPerToken * + * context + overheadBytes + peakExtraBytes` of the catalogue row. The memory bar of the + * chooser is filled to `budgetShare`, which is this over the budget. */ needBytes: number; /** `needBytes` as decimal GB text, no unit. */ @@ -501,30 +494,33 @@ export interface Candidate { /** * Generation speed in tokens per second, rounded to one decimal. Present exactly when `speed` * is not `unknown`; `speed` is decided on the unrounded figure. An estimate from the chip's - * memory bandwidth, except (stage 2) for a candidate that is `tested`, which reports the + * memory bandwidth, except for a candidate that is `tested`, which reports the * speed measured in that test (`TestedOn.tokensPerSecond`). */ tokensPerSecond?: number; - /** A `TestedOn` entry of the model matches this Mac (same chip, same `memClassGb`) and this harness. */ + /** + * A `TestedOn` entry of the model matches this Mac (same chip, same `memClassGb`) and this + * harness. Such a candidate ranks first, and its `tag` says so in place of any other tag. + */ tested: boolean; - /** Stage 2. `LocalModelEntry.figures` of the model: whether `needBytes` rests on a measurement of this model. */ - needSource?: 'measured' | 'estimated'; - /** Stage 2. The row's one tag, see `CandidateTagKind`. Absent when no kind applies. */ + /** `LocalModelEntry.figures` of the model: whether `needBytes` rests on a measurement of this model. */ + needSource: 'measured' | 'estimated'; + /** The row's one tag, see `CandidateTagKind`. Absent when no kind applies. */ tag?: CandidateTag; /** - * Stage 2. Every coding agent this model can be run with at `context` on this Mac, best + * Every coding agent this model can be run with at `context` on this Mac, best * first. The first entry is the one the flat fields above name (`cli`, `cliLabel`, * `cliInstalled`, `slowFirstTurn`). Never empty. */ - harnesses?: CandidateHarness[]; + harnesses: CandidateHarness[]; /** - * Stage 2. The whole download a click on Install starts for this row: `downloadBytes` plus + * The whole download a click on Install starts for this row: `downloadBytes` plus * `RecommendOptions.runtimeDownloadBytes`. It is the server's own total, and exactly the * number `POST /api/local-model/install` must be sent as `acknowledgedBytes`. */ - installBytes?: number; - /** Stage 2. `installBytes` as decimal GB text, no unit. */ - installGb?: string; + installBytes: number; + /** `installBytes` as decimal GB text, no unit. */ + installGb: string; } /** @@ -547,7 +543,12 @@ export interface Recommendation { notes: LocalModelMessage[]; /** Present exactly when `verdict` is `ok` or `limited`. */ pick?: Candidate; - /** The next models that fit and have room on the disk (at most four, each at its best context and harness), best first. Empty when `verdict` is `none`. */ + /** + * The next models that fit and have room on the disk (at most four), best first: tested + * before not tested, then ok before unknown before slow speed, then the higher model rank. + * Each is at the largest context it fits at, with the best harness that can run it there. + * Empty when `verdict` is `none`. + */ alternatives: Candidate[]; /** `DeviceBudget.budgetBytes`. 0 when no budget is computed: not macOS, or a Mac without Apple silicon. */ budgetBytes: number; diff --git a/test/local-model-catalog.test.ts b/test/local-model-catalog.test.ts index 5f72b764..4c185e0b 100644 --- a/test/local-model-catalog.test.ts +++ b/test/local-model-catalog.test.ts @@ -2,7 +2,7 @@ * @fileoverview Tests for src/config/local-model-catalog.ts: the compiled catalogue of models, * harnesses and chips, its schema, and the chip lookup. * - * Three things are pinned: + * Four things are pinned: * * 1. The stock constant PARSES. `getLocalModelCatalog()` validates it on first use and throws on * a bad row, so this test is what keeps a bad row from ever reaching the route. @@ -12,6 +12,9 @@ * hides a more specific one). * 3. The chip lookup never guesses: a chip that no row names has no bandwidth, including one * whose name merely contains a known name ('Apple M10', 'Apple M6 Pro'). + * 4. Which figures were MEASURED on a Mac and which are carried over from the measured row: + * one row says `measured` (its size is what a Mac really downloads, and it alone has a + * tested entry), every other row says `estimated`. * * Port: N/A (pure data and functions). */ @@ -39,8 +42,8 @@ function issuesOf(catalog: unknown): string[] { } /** The stock catalogue with its first model changed. */ -/** `url` is no field of a model row: one test passes it to prove the schema refuses an unknown key. */ -function withFirstModel(change: Partial>): unknown { +/** `url` and `kvBytesPerToken` are no fields of a model row: tests pass them to prove the schema refuses an unknown key. */ +function withFirstModel(change: Partial>): unknown { const catalog = copy(); return { ...catalog, models: [{ ...catalog.models[0], ...change }, ...catalog.models.slice(1)] }; } @@ -70,36 +73,62 @@ describe('the stock catalogue', () => { expect(STOCK_LOCAL_MODEL_CATALOG.models.some((m) => m.id.startsWith('flint'))).toBe(false); }); - it('carries the byte sizes of the registry manifests', () => { - // The sum of the layer sizes of each tag's manifest, as fetched on 2026-11-06. + it('carries the size a Mac really downloads where that was measured, else the registry manifest', () => { expect(Object.fromEntries(STOCK_LOCAL_MODEL_CATALOG.models.map((m) => [m.id, m.bytes]))).toEqual({ + // The sum of the layer sizes of each tag's manifest, as fetched on 2026-11-06. 'qwen3.5:9b': 6_550_825_373, 'qwen3.5:9b-q8_0': 10_449_217_949, 'gpt-oss:20b': 13_793_441_427, - 'qwen3.6:35b-a3b': 22_621_314_161, + // The total of the real pull on a Mac (an MLX build). Its manifest sums to 22,621,314,161. + 'qwen3.6:35b-a3b': 23_105_798_956, 'qwen3.8:27b': 17_741_871_939, 'qwen3.8:27b-q8_0': 29_978_241_837, 'qwen3.5:122b-a10b': 81_370_035_877, }); }); - it('carries the KV cost per token of each model', () => { - expect(Object.fromEntries(STOCK_LOCAL_MODEL_CATALOG.models.map((m) => [m.id, m.kvBytesPerToken]))).toEqual({ - 'qwen3.5:9b': 32_768, - 'qwen3.5:9b-q8_0': 32_768, - 'gpt-oss:20b': 24_576, - 'qwen3.6:35b-a3b': 20_480, - 'qwen3.8:27b': 65_536, - 'qwen3.8:27b-q8_0': 65_536, - 'qwen3.5:122b-a10b': 24_576, + it('says which row was measured on a Mac: one, and every other row is an estimate', () => { + expect(Object.fromEntries(STOCK_LOCAL_MODEL_CATALOG.models.map((m) => [m.id, m.figures]))).toEqual({ + 'qwen3.5:9b': 'estimated', + 'qwen3.5:9b-q8_0': 'estimated', + 'gpt-oss:20b': 'estimated', + 'qwen3.6:35b-a3b': 'measured', + 'qwen3.8:27b': 'estimated', + 'qwen3.8:27b-q8_0': 'estimated', + 'qwen3.5:122b-a10b': 'estimated', }); }); - it('gives every model a parameter count and an empty tested list', () => { + it('gives every row the memory figures measured on the one row: 37 KB per token, 3 GB, 5 GB', () => { + for (const model of STOCK_LOCAL_MODEL_CATALOG.models) { + expect( + [model.heldBytesPerToken, model.overheadBytes, model.peakExtraBytes], + `${model.id}: an unmeasured row carries the measured row's figures, not invented ones` + ).toEqual([37_000, 3_000_000_000, 5_000_000_000]); + } + // No row keeps the figure that was computed from the model's config. + expect(JSON.stringify(STOCK_LOCAL_MODEL_CATALOG)).not.toContain('kvBytesPerToken'); + }); + + it('has one tested entry: the 35B-A3B with OpenCode on the MacBook Pro, with its measured speed', () => { + expect( + Object.fromEntries( + STOCK_LOCAL_MODEL_CATALOG.models.filter((m) => m.tested.length > 0).map((m) => [m.id, m.tested]) + ) + ).toEqual({ + 'qwen3.6:35b-a3b': [ + { chip: 'Apple M5 Pro', memGb: 48, cli: 'opencode', date: '2026-11-06', tokensPerSecond: 67.7 }, + ], + }); + // A row may claim a test only when its figures were measured. + for (const model of STOCK_LOCAL_MODEL_CATALOG.models) { + if (model.tested.length > 0) expect(model.figures, model.id).toBe('measured'); + } + }); + + it('gives every model a parameter count, an id of the default registry and a license', () => { for (const model of STOCK_LOCAL_MODEL_CATALOG.models) { expect(model.paramsB, model.id).toBeGreaterThan(0); - // Nobody has run any of these with a harness on a Mac yet. - expect(model.tested, model.id).toEqual([]); expect(model.id).toMatch(LOCAL_MODEL_ID_PATTERN); expect(model.license).toBe('Apache-2.0'); } @@ -130,11 +159,7 @@ describe('the stock catalogue', () => { ]); }); - it('counts the prompt cache as 512 MiB below 32 GB and 2 GiB from 32 GB', () => { - expect(STOCK_LOCAL_MODEL_CATALOG.promptCacheTiers).toEqual([ - { minMemGb: 0, bytes: 512 * 1024 ** 2 }, - { minMemGb: 32, bytes: 2 * 1024 ** 3 }, - ]); + it('names the oldest macOS and its schema version', () => { expect(STOCK_LOCAL_MODEL_CATALOG.minMacosMajor).toBe(14); expect(STOCK_LOCAL_MODEL_CATALOG.schema).toBe(1); }); @@ -144,7 +169,7 @@ describe('the stock catalogue', () => { expect(text).not.toMatch(/https?:/); expect(text).not.toMatch(/sha256/i); expect(Object.keys(STOCK_LOCAL_MODEL_CATALOG).sort()).toEqual( - ['chips', 'harnesses', 'minMacosMajor', 'models', 'promptCacheTiers', 'schema', 'updated'].sort() + ['chips', 'harnesses', 'minMacosMajor', 'models', 'schema', 'updated'].sort() ); }); }); @@ -193,6 +218,26 @@ describe('the schema, field by field', () => { const catalog = copy(); expect(issuesOf({ ...catalog, harnesses: [{ ...catalog.harnesses[0], dialect: 'openai' }] })).not.toEqual([]); expect(issuesOf({ ...catalog, chips: [{ match: 'M4', gbps: 120, cores: 10 }] })).not.toEqual([]); + // The two figures the need was once computed from are gone, and a row that still has one is refused. + expect(issuesOf(withFirstModel({ kvBytesPerToken: 32768 }))).not.toEqual([]); + expect(issuesOf({ ...catalog, promptCacheTiers: [{ minMemGb: 0, bytes: 1 }] })).not.toEqual([]); + }); + + it('requires the memory figures and where they come from on every model row', () => { + for (const field of ['heldBytesPerToken', 'overheadBytes', 'peakExtraBytes', 'figures'] as const) { + const row: Record = { ...copy().models[0] }; + delete row[field]; + const issues = issuesOf({ ...copy(), models: [row] }); + expect(issues, field).toHaveLength(1); + expect(issues[0]).toContain(`models.0.${field}`); + } + expect(issuesOf(withFirstModel({ figures: 'guessed' }))).not.toEqual([]); + // What stays held per token must be a positive whole number of bytes; the other two may be 0. + expect(issuesOf(withFirstModel({ heldBytesPerToken: 0 }))).not.toEqual([]); + expect(issuesOf(withFirstModel({ heldBytesPerToken: 36.5 }))).not.toEqual([]); + expect(issuesOf(withFirstModel({ overheadBytes: 0, peakExtraBytes: 0 }))).toEqual([]); + expect(issuesOf(withFirstModel({ overheadBytes: -1 }))).not.toEqual([]); + expect(issuesOf(withFirstModel({ peakExtraBytes: 1.5 }))).not.toEqual([]); }); it('requires paramsB and tested on every model row', () => { @@ -208,7 +253,6 @@ describe('the schema, field by field', () => { it('refuses sizes and counts that are not positive whole numbers', () => { expect(issuesOf(withFirstModel({ bytes: 0 }))).not.toEqual([]); expect(issuesOf(withFirstModel({ bytes: 1.5 }))).not.toEqual([]); - expect(issuesOf(withFirstModel({ kvBytesPerToken: -1 }))).not.toEqual([]); expect(issuesOf(withFirstModel({ contexts: [] }))).not.toEqual([]); expect(issuesOf(withFirstModel({ paramsB: 0 }))).not.toEqual([]); expect(issuesOf(withFirstModel({ speedPer100Gbps: 0 }))).not.toEqual([]); @@ -220,6 +264,10 @@ describe('the schema, field by field', () => { expect(issuesOf(withFirstModel({ tested: [{ ...good, date: '6 November 2026' }] }))).not.toEqual([]); expect(issuesOf(withFirstModel({ tested: [{ ...good, memGb: 0 }] }))).not.toEqual([]); expect(issuesOf(withFirstModel({ tested: [{ ...good, result: 'pass' }] }))).not.toEqual([]); + // The speed measured in that test is optional, and a speed is above 0. + expect(issuesOf(withFirstModel({ tested: [{ ...good, tokensPerSecond: 67.7 }] }))).toEqual([]); + expect(issuesOf(withFirstModel({ tested: [{ ...good, tokensPerSecond: 0 }] }))).not.toEqual([]); + expect(issuesOf(withFirstModel({ tested: [{ ...good, tokensPerSecond: '67.7' }] }))).not.toEqual([]); }); it('refuses another schema version and a date that is not ISO', () => { @@ -284,20 +332,6 @@ describe('the schema, across rows', () => { ]; expect(issuesOf(repeated)).toHaveLength(1); }); - - it('refuses prompt cache tiers that do not start at 0 or do not ascend', () => { - const late = copy(); - late.promptCacheTiers = [{ minMemGb: 16, bytes: 1 }]; - expect(issuesOf(late)).toEqual(['promptCacheTiers.0.minMemGb: the first prompt cache tier must start at 0']); - - const unordered = copy(); - unordered.promptCacheTiers = [ - { minMemGb: 0, bytes: 1 }, - { minMemGb: 32, bytes: 2 }, - { minMemGb: 24, bytes: 3 }, - ]; - expect(issuesOf(unordered)).toEqual(['promptCacheTiers.2.minMemGb: prompt cache tiers must ascend by minMemGb']); - }); }); describe('the chips table', () => { diff --git a/test/local-model-fits-budget.test.ts b/test/local-model-fits-budget.test.ts index 5ae2ff98..f66707d4 100644 --- a/test/local-model-fits-budget.test.ts +++ b/test/local-model-fits-budget.test.ts @@ -4,7 +4,9 @@ * * Pinned: it agrees with what `recommend()` offers (a model that is the pick or listed fits, * a model that is in neither and is not left out for its rank does not), it does not depend - * on the ranking, and a model the catalogue does not have never fits. + * on the ranking, a model the catalogue does not have never fits, and it judges against the + * budget the runtime measured when the caller passes one (the whole point of asking again + * after the runtime started). * * Port: N/A (pure functions). */ @@ -87,4 +89,48 @@ describe('modelFitsBudget', () => { expect(modelFitsBudget(mac(48), catalog, 'no-such:1b', 65536)).toBe(false); expect(modelFitsBudget(mac(48), catalog, 'qwen3.6:35b-a3b', 131072)).toBe(true); }); + + it('judges against the budget the runtime measured, when it is given one', () => { + const id = 'qwen3.6:35b-a3b'; + // The MacBook: the runtime reported 37.4 GiB, and the model needs 36.0 of those 40.2 GB. + expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 40_157_944_217 })).toBe(true); + // A runtime that reports less than the estimate rules out what the estimate allowed. + expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 33 * 1024 ** 3 })).toBe(false); + // And one that reports more admits what the estimate ruled out: the 27B on a 36 GB Mac. + expect(modelFitsBudget(mac(36), catalog, 'qwen3.8:27b', 65536)).toBe(false); + expect(modelFitsBudget(mac(36), catalog, 'qwen3.8:27b', 65536, { measuredTotalBytes: 27 * 1024 ** 3 })).toBe(true); + // The reserve still bounds it: 100 GiB "measured" on a 48 GB Mac is 40 GiB (42.9 GB) of + // budget, so the Q8 27B with its 42.8 GB stays out (0.997) although 100 GiB would hold it. + const huge = { measuredTotalBytes: 100 * 1024 ** 3 }; + expect(modelFitsBudget(mac(48), catalog, 'qwen3.8:27b-q8_0', 131072, huge)).toBe(false); + expect(modelFitsBudget(mac(128), catalog, 'qwen3.8:27b-q8_0', 131072, huge)).toBe(true); + }); + + it('agrees with the recommendation under a measured budget too', () => { + for (const [gb, measuredGib] of [ + [16, 11.8], + [32, 20], + [48, 37.4], + [48, 30], + [64, 50], + ]) { + const hw = mac(gb); + const options = { measuredTotalBytes: Math.floor(measuredGib * 1024 ** 3) }; + const offered = recommend(hw, catalog, harnessFacts(), options); + expect(offered.verdict === 'none' || offered.budgetSource === 'measured').toBe(true); + for (const row of [offered.pick, ...offered.alternatives].filter((entry) => entry !== undefined)) { + expect(modelFitsBudget(hw, catalog, row.modelId, row.context, options), `${gb} GB ${row.modelId}`).toBe(true); + // The next larger context of the same model does not fit, or the row would be at it. + const model = catalog.models.find((entry) => entry.id === row.modelId); + const larger = model?.contexts.filter((ctx) => ctx > row.context) ?? []; + for (const ctx of larger) expect(modelFitsBudget(hw, catalog, row.modelId, ctx, options)).toBe(false); + } + } + }); + + it('is false for a context that is no token count', () => { + for (const context of [0, -1, Number.NaN, Number.POSITIVE_INFINITY]) { + expect(modelFitsBudget(mac(128), catalog, 'qwen3.5:9b', context), String(context)).toBe(false); + } + }); }); diff --git a/test/local-model-recommend.test.ts b/test/local-model-recommend.test.ts index bf22dc8b..4a6ee5f8 100644 --- a/test/local-model-recommend.test.ts +++ b/test/local-model-recommend.test.ts @@ -4,17 +4,18 @@ * * Laid out like the module: * - * 1. The budget (`deviceBudget`) for every memory size Apple sells from 8 to 128 GB, and the - * need of a model at a context (`needBytes`). + * 1. The budget (`deviceBudget`) for every memory size Apple sells from 8 to 128 GB, estimated + * and measured, and the need of a model at a context (`needBytes`: the peak). * 2. The two REAL Macs of the project, from their captured `sysctl` output through the real * parser: the MacBook Pro (M5 Pro, 48 GB) and the Mac mini (M4, 16 GB). Each test states the - * expected answer and why. + * expected answer and why. The MacBook's answer is checked against what was MEASURED on it. * 3. Every reason for verdict `none`, one fixture per reason, and their order. * 4. The `limited` verdict (a statement about the Mac, not about the pick), the notes that say - * what the headline cannot (a small pick, a model left out for disk space), the three-state - * speed in the ranking, harness rows (a gated one is never chosen), the `tested` flag that - * alone may turn on the word "Recommended", and the shape of the answer (labels from the - * fixed list, sizes in one unit). + * what the headline cannot (a small pick, a model left out for disk space), the ranking + * (tested, speed, model rank, harness rank, context: the fit grade filters and does not + * sort), harness rows (a gated one is never chosen), the `tested` flag that alone may turn + * on the word "Recommended", what each row of the chooser carries (its tag, its harnesses, + * the whole download of an install), and the shape of the answer. * * The stock catalogue is used wherever the answer for a real Mac is at stake. Small synthetic * catalogues (validated by the real schema) isolate one rule of the ranking at a time. @@ -38,12 +39,14 @@ import { FIT_TIGHT_MAX_SHARE, LIMITED_BELOW_PARAMS_B, MAX_ALTERNATIVES, + modelFitsBudget, needBytes, recommend, SLOW_BELOW_TOKENS_PER_SECOND, } from '../src/local-model/recommend.js'; import { LOCAL_MODEL_LABELS, + LOCAL_MODEL_TAG_LABELS, type Candidate, type HardwareFacts, type HarnessFact, @@ -51,6 +54,7 @@ import { type LocalModelCatalog, type LocalModelEntry, type Recommendation, + type TestedOn, } from '../src/types.js'; // ---------- Fixtures ---------- @@ -83,6 +87,16 @@ exit=0 /dev/disk3s5 239362496 172954124 41752848 81% 2353838 417528480 1% /System/Volumes/Data `; +/** + * What the runtime printed as the device total on each real Mac when it started: 37.4 GiB on + * the MacBook, 11.8 GiB on the Mac mini (`parseInferenceComputeTotal()` of those two log lines). + */ +const MACBOOK_MEASURED_TOTAL = 40_157_944_217; +const MACMINI_MEASURED_TOTAL = 12_670_153_523; + +/** The size of the pinned runtime archive: what an install adds to the download while the runtime is not installed. */ +const RUNTIME_ARCHIVE_BYTES = 185_971_671; + /** The hardware facts of a real Mac: its captured output through the real parser, no IO. */ async function realMac(sysctlOutput: string, freeDiskKb: number): Promise { resetHardwareCache(); @@ -97,6 +111,9 @@ async function realMac(sysctlOutput: string, freeDiskKb: number): Promise => realMac(MACBOOK_SYSCTL, 890_926_084); +const macMini = (): Promise => realMac(MACMINI_SYSCTL, 41_752_848); + /** An Apple silicon Mac with `soldGb` of memory, healthy in every other respect. */ function mac(soldGb: number, chip: string, overrides: Partial = {}): HardwareFacts { const memBytes = soldGb * GIB; @@ -147,6 +164,11 @@ const STOCK = getLocalModelCatalog(); /** Every stock harness installed, the gated one included. */ const ALL_INSTALLED: HarnessFact[] = STOCK.harnesses.map((row) => fact(row.cli)); +/** A stock model row by id. */ +function stockModel(id: string): LocalModelEntry { + return STOCK.models.find((m) => m.id === id) as LocalModelEntry; +} + /** The stock catalogue with a change applied, validated by the real schema. */ function stockWith(change: (catalog: LocalModelCatalog) => void): LocalModelCatalog { const catalog = structuredClone(STOCK_LOCAL_MODEL_CATALOG); @@ -154,7 +176,15 @@ function stockWith(change: (catalog: LocalModelCatalog) => void): LocalModelCata return LocalModelCatalogSchema.parse(catalog); } -/** A model row for a synthetic catalogue: 20 billion parameters, 10 GiB, one small context. */ +/** The stock catalogue as it was before anybody tested a row: no `tested` entry anywhere. */ +const UNTESTED = stockWith((catalog) => { + for (const row of catalog.models) row.tested = []; +}); + +/** + * A model row for a synthetic catalogue: 20 billion parameters, 10 GiB, one small context, one + * byte held per token and nothing else on top, so its need is its weights plus its context. + */ function model(id: string, overrides: Partial = {}): LocalModelEntry { return { id, @@ -162,7 +192,10 @@ function model(id: string, overrides: Partial = {}): LocalModel license: 'Apache-2.0', paramsB: 20, bytes: 10 * GIB, - kvBytesPerToken: 1, + heldBytesPerToken: 1, + overheadBytes: 0, + peakExtraBytes: 0, + figures: 'estimated', contexts: [1024], rank: 1, tested: [], @@ -171,15 +204,14 @@ function model(id: string, overrides: Partial = {}): LocalModel } /** - * A synthetic catalogue: no prompt cache, one harness `alpha` that works from 1024 tokens, and - * one chip row ('M4 Max' at 400 GB/s, so tokens per second = `speedPer100Gbps` x 4). + * A synthetic catalogue: one harness `alpha` that works from 1024 tokens, and one chip row + * ('M4 Max' at 400 GB/s, so tokens per second = `speedPer100Gbps` x 4). */ function catalogOf(models: LocalModelEntry[], harnesses?: LocalHarnessEntry[]): LocalModelCatalog { return LocalModelCatalogSchema.parse({ schema: 1, updated: '2026-11-06', minMacosMajor: 14, - promptCacheTiers: [{ minMemGb: 0, bytes: 0 }], models, harnesses: harnesses ?? [{ cli: 'alpha', minContext: 1024, rank: 1 }], chips: [{ match: 'M4 Max', gbps: 400 }], @@ -190,19 +222,22 @@ function catalogOf(models: LocalModelEntry[], harnesses?: LocalHarnessEntry[]): const HUNDRED_GIB_MAC = mac(128, 'Apple M4 Max', { wiredLimitMb: 101 * 1024 }); const HUNDRED_GIB = 100 * GIB; -/** What a synthetic model needs beside its weights at its 1024-token context: 1 GiB plus the KV cache. */ -const SYNTHETIC_OVERHEAD = GIB + 1024; +/** What a synthetic model needs beside its weights at its 1024-token context: one byte per token. */ +const SYNTHETIC_OVERHEAD = 1024; /** A model whose need is exactly `need` bytes in a synthetic catalogue. */ function modelNeeding(id: string, need: number, overrides: Partial = {}): LocalModelEntry { return model(id, { bytes: need - SYNTHETIC_OVERHEAD, ...overrides }); } +/** The pick and the alternatives, in order. */ +function rows(result: Recommendation): Candidate[] { + return [result.pick, ...result.alternatives].filter((candidate): candidate is Candidate => candidate !== undefined); +} + /** `id@context` of the pick and of each alternative, in order. */ function order(result: Recommendation): string[] { - return [result.pick, ...result.alternatives] - .filter((candidate): candidate is Candidate => candidate !== undefined) - .map((candidate) => `${candidate.modelId}@${candidate.context}`); + return rows(result).map((candidate) => `${candidate.modelId}@${candidate.context}`); } /** Every string anywhere in a value. */ @@ -232,7 +267,7 @@ beforeEach(() => { // ---------- 1. Budget and need ---------- -describe('deviceBudget', () => { +describe('deviceBudget, estimated', () => { // Sold size, device limit L and budget B in GiB. L is 2/3 of memory below 36 GB and 3/4 from // 36 GB; B = min(L - 1, memory - reserve). Only at 8 GB is the reserve the smaller bound. const TIERS: [soldGb: number, limitGib: number, budgetGib: number][] = [ @@ -321,34 +356,94 @@ describe('deviceBudget', () => { }); }); -describe('needBytes', () => { - const nineB = STOCK.models.find((m) => m.id === 'qwen3.5:9b') as LocalModelEntry; - - it('adds weights, the KV cache of the context, 1 GiB and the prompt cache', () => { - // 6,550,825,373 + 32768 x 65536 + 1 GiB + 512 MiB on a Mac below 32 GB. - expect(needBytes(nineB, 65536, STOCK, mac(16, 'Apple M4'))).toBe(6_550_825_373 + 2 * GIB + GIB + 512 * MIB); - expect(needBytes(nineB, 65536, STOCK, mac(16, 'Apple M4'))).toBe(10_308_921_757); - // Half the context, half the KV cache. - expect(needBytes(nineB, 32768, STOCK, mac(16, 'Apple M4'))).toBe(6_550_825_373 + GIB + GIB + 512 * MIB); +describe('deviceBudget, measured', () => { + it('takes what the runtime reported as the limit, without the 1 GiB of the estimate', () => { + // The MacBook: 37.4 GiB reported, memory - reserve is 40 GiB, so the report is the budget. + expect(deviceBudget(mac(48, 'Apple M5 Pro'), { measuredTotalBytes: MACBOOK_MEASURED_TOTAL })).toEqual({ + limitBytes: 40_157_944_217, + budgetBytes: 40_157_944_217, + source: 'measured', + }); + // 1.4 GiB more than the estimate of 35 GiB, which was on the careful side. + expect(40_157_944_217 - deviceBudget(mac(48, 'Apple M5 Pro')).budgetBytes).toBe(2_576_980_377); }); - it('counts a prompt cache of 512 MiB below 32 GB of memory and 2 GiB from 32 GB', () => { - const base = 6_550_825_373 + 2 * GIB + GIB; - for (const soldGb of [8, 16, 18, 24]) { - expect(needBytes(nineB, 65536, STOCK, mac(soldGb, 'Apple M4')) - base, `${soldGb} GB`).toBe(512 * MIB); - } - for (const soldGb of [32, 36, 48, 64, 96, 128]) { - expect(needBytes(nineB, 65536, STOCK, mac(soldGb, 'Apple M4')) - base, `${soldGb} GB`).toBe(2 * GIB); - } + it('is still bounded by the reserve', () => { + // The Mac mini: 11.8 GiB reported, but 16 GiB minus the 6 GiB reserve is 10 GiB. + expect(deviceBudget(mac(16, 'Apple M4'), { measuredTotalBytes: MACMINI_MEASURED_TOTAL })).toEqual({ + limitBytes: 12_670_153_523, + budgetBytes: 10 * GIB, + source: 'measured', + }); }); - it('does not depend on the chip or the harness', () => { - expect(needBytes(nineB, 65536, STOCK, mac(48, 'Apple M1'))).toBe( - needBytes(nineB, 65536, STOCK, mac(48, 'Apple M5 Ultra')) + it('beats a wired limit: the runtime read what it will really use', () => { + const hw = mac(48, 'Apple M5 Pro', { wiredLimitMb: 20_000 }); + expect(deviceBudget(hw).source).toBe('wired-limit'); + expect(deviceBudget(hw, { measuredTotalBytes: MACBOOK_MEASURED_TOTAL })).toMatchObject({ + budgetBytes: MACBOOK_MEASURED_TOTAL, + source: 'measured', + }); + }); + + it('ignores a figure that is no byte count and falls back to the estimate', () => { + const estimated = deviceBudget(mac(48, 'Apple M5 Pro')); + for (const measuredTotalBytes of [0, -1, Number.NaN, Number.POSITIVE_INFINITY, undefined]) { + expect(deviceBudget(mac(48, 'Apple M5 Pro'), { measuredTotalBytes }), String(measuredTotalBytes)).toEqual( + estimated + ); + } + // A fraction of a byte is rounded down. + expect(deviceBudget(mac(48, 'Apple M5 Pro'), { measuredTotalBytes: 40_157_944_217.6 }).limitBytes).toBe( + 40_157_944_217 ); }); }); +describe('needBytes', () => { + it('is the peak: weights, what stays held for the context, the runner, and the moment of a long prompt', () => { + // The measured row: 23,105,798,956 + 37,000 x 131,072 + 3 GB + 5 GB. + const moe = stockModel('qwen3.6:35b-a3b'); + expect(needBytes(moe, 131072)).toBe(23_105_798_956 + 37_000 * 131_072 + 3_000_000_000 + 5_000_000_000); + expect(needBytes(moe, 131072)).toBe(35_955_462_956); + // What was measured on the MacBook: about 30.5 GB held and 35 to 36 GB at the peak. + expect(formatGb(needBytes(moe, 131072))).toBe('36.0'); + expect(formatGb(needBytes(moe, 131072) - moe.peakExtraBytes)).toBe('31.0'); + }); + + it('grows with the context by what one token holds', () => { + const nineB = stockModel('qwen3.5:9b'); + expect(needBytes(nineB, 65536)).toBe(16_975_657_373); + expect(needBytes(nineB, 32768)).toBe(15_763_241_373); + // Half the context, half of what the context holds: 37,000 x 32,768. + expect(needBytes(nineB, 65536) - needBytes(nineB, 32768)).toBe(1_212_416_000); + }); + + it('reads every figure from the row it is given', () => { + const row = model('probe:1', { + bytes: 1000, + heldBytesPerToken: 7, + overheadBytes: 30, + peakExtraBytes: 500, + contexts: [10], + }); + expect(needBytes(row, 10)).toBe(1000 + 70 + 30 + 500); + expect(needBytes({ ...row, peakExtraBytes: 0 }, 10)).toBe(1100); + }); + + it('states the need of every stock row at its largest context', () => { + expect(Object.fromEntries(STOCK.models.map((m) => [m.id, formatGb(needBytes(m, m.contexts[0]))]))).toEqual({ + 'qwen3.5:9b': '17.0', + 'qwen3.5:9b-q8_0': '20.9', + 'gpt-oss:20b': '26.6', + 'qwen3.6:35b-a3b': '36.0', + 'qwen3.8:27b': '30.6', + 'qwen3.8:27b-q8_0': '47.7', + 'qwen3.5:122b-a10b': '94.2', + }); + }); +}); + describe('the fit grade', () => { const pickOf = (need: number): Candidate | undefined => recommend(HUNDRED_GIB_MAC, catalogOf([modelNeeding('probe:1', need)]), [fact('alpha')]).pick; @@ -384,18 +479,21 @@ describe('the fit grade', () => { describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { /** - * Expected: Qwen3.6 35B-A3B at 131,072 tokens with OpenCode, Good, about 42 tokens per second. + * Expected: Qwen3.6 35B-A3B at 131,072 tokens with OpenCode, a TIGHT fit, 67.7 tokens per + * second, "Recommended for this Mac". * - * Why. The budget is 35 GiB (3/4 of 48 GiB minus 1). Five models fit it as Good. The M5 Pro is - * in the chips table (307 GB/s), so each model with a speed figure gets an estimate: - * the dense 27B computes to 2.95 x 3.07 = 9.1 tokens per second, under the 10 that separate - * slow from ok, while the 35B-A3B (3 billion parameters active per token) computes to 41.8. - * Speed ranks before model rank, so the 35B-A3B wins although the 27B has the higher rank. The - * 27B stays in the list, last, marked Slow. Both harnesses can run it; OpenCode has the - * higher rank. + * Why. This is the Mac the model was run on (2026-11-06), so its row has a `tested` entry for + * exactly this chip, memory size and harness. A tested row ranks first, and it reports the + * speed that was measured (67.7) in place of the estimate from the chip (13.6 x 3.07 = 41.8). + * + * The need is the peak at the full context: 23.1 GB of weights + 4.8 GB held for 131,072 + * tokens + 3 GB of runner + 5 GB for a moment while a long prompt is processed = 36.0 GB. + * Before the runtime ran, the budget is the estimate, 35 GiB (37.6 GB): 0.957 of it, tight. + * It is not stepped down to 65,536 tokens, where it would be 0.892: a model is offered at + * the largest context it fits at. */ - it('gets Qwen3.6 35B-A3B at 131072 tokens with OpenCode', async () => { - const hw = await realMac(MACBOOK_SYSCTL, 890_926_084); + it('gets Qwen3.6 35B-A3B at 131072 tokens with OpenCode, as a tight fit', async () => { + const hw = await macbook(); expect(hw.chip).toBe('Apple M5 Pro'); expect(hw.memClassGb).toBe(48); @@ -415,70 +513,135 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { cli: 'opencode', cliLabel: 'OpenCode', cliInstalled: true, - // 22,621,314,161 + 20480 x 131072 + 1 GiB + 2 GiB. - needBytes: 28_526_894_193, - needGb: '28.5', - downloadBytes: 22_621_314_161, - downloadGb: '22.6', - budgetShare: 0.759, - fit: 'good', - fitLabel: 'Good', + // 23,105,798,956 + 37,000 x 131,072 + 3,000,000,000 + 5,000,000,000. + needBytes: 35_955_462_956, + needGb: '36.0', + // What a Mac really downloads (the MLX build), not the 22.6 GB of the registry manifest. + downloadBytes: 23_105_798_956, + downloadGb: '23.1', + budgetShare: 0.957, + fit: 'tight', + fitLabel: 'Tight', speed: 'ok', - tokensPerSecond: 41.8, - tested: false, + // Measured on this Mac, not the 41.8 the chip's bandwidth predicts. + tokensPerSecond: 67.7, + tested: true, + needSource: 'measured', + tag: { kind: 'tested', label: 'Tested on a Mac like this' }, + harnesses: [ + { cli: 'opencode', cliLabel: 'OpenCode', installed: true }, + { cli: 'claude', cliLabel: 'Claude Code', installed: true, slowFirstTurn: true }, + ], + // No runtime download was named, so an install downloads the model alone. + installBytes: 23_105_798_956, + installGb: '23.1', }); }); - it('says "Best fit", not "Recommended", and that nobody tested it on such a Mac', async () => { - const result = recommend(await realMac(MACBOOK_SYSCTL, 890_926_084), STOCK, ALL_INSTALLED); + /** + * Expected with the budget the runtime MEASURED on this Mac (37.4 GiB, 40.2 GB): the same + * pick at the same context, 0.895 of the budget, still tight. That is what the measurement + * notes say of it: "it FITS the measured 40.2 GB budget, and it is TIGHT, not Good". + */ + it('keeps the pick under the measured budget: 36.0 of 40.2 GB, tight', async () => { + const result = recommend(await macbook(), STOCK, ALL_INSTALLED, { + measuredTotalBytes: MACBOOK_MEASURED_TOTAL, + runtimeDownloadBytes: RUNTIME_ARCHIVE_BYTES, + }); + expect(result.budgetBytes).toBe(40_157_944_217); + expect(result.budgetGb).toBe('40.2'); + expect(result.budgetSource).toBe('measured'); + expect(result.pick).toMatchObject({ + modelId: 'qwen3.6:35b-a3b', + context: 131072, + needGb: '36.0', + budgetShare: 0.895, + fit: 'tight', + tested: true, + // The model plus the runtime archive: 23,105,798,956 + 185,971,671. + installBytes: 23_291_770_627, + installGb: '23.3', + }); + // At 65536 tokens it would be a good fit (0.835), and it is still offered at the full context. + const moe = stockModel('qwen3.6:35b-a3b'); + expect(needBytes(moe, 65536) / result.budgetBytes).toBeCloseTo(0.835, 3); + expect(needBytes(moe, 65536) / result.budgetBytes).toBeLessThan(FIT_GOOD_MAX_SHARE); + }); + + it('says "Recommended", because the pick was run on exactly such a Mac', async () => { + const result = recommend(await macbook(), STOCK, ALL_INSTALLED); + expect(result.headline).toEqual({ text: 'Recommended for this Mac' }); + // The "not yet tested" note would be false. + expect(result.notes).toEqual([]); + }); + + it('lists the other four models that fit: the fast ones, the one of unknown speed, the slow one', async () => { + const result = recommend(await macbook(), STOCK, ALL_INSTALLED); + expect( + result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind]) + ).toEqual([ + ['qwen3.5:9b-q8_0', 65536, 'good', 'ok', 17.8, 'small'], + ['qwen3.5:9b', 65536, 'good', 'ok', 28.6, 'small'], + ['gpt-oss:20b', 131072, 'good', 'unknown', null, undefined], + // The dense 27B computes to 2.95 x 3.07 = 9.1 tokens per second on this chip. + ['qwen3.8:27b', 131072, 'good', 'slow', 9.1, 'slow'], + ]); + expect(result.alternatives[2].speedLabel).toBe('Speed unknown'); + expect(result.alternatives[3].speedLabel).toBe('Slow'); + // Only the tested row rests on a measurement of that model. + expect(result.alternatives.map((c) => c.needSource)).toEqual(['estimated', 'estimated', 'estimated', 'estimated']); + // The two that do not fit at any context are not listed. + expect(order(result).some((entry) => entry.startsWith('qwen3.8:27b-q8_0'))).toBe(false); + expect(order(result).some((entry) => entry.startsWith('qwen3.5:122b'))).toBe(false); + }); + + it('would get the same model as "Best fit", with the estimated speed, had nobody tested it', async () => { + const result = recommend(await macbook(), UNTESTED, ALL_INSTALLED); + expect(result.pick).toMatchObject({ + modelId: 'qwen3.6:35b-a3b', + context: 131072, + fit: 'tight', + tested: false, + // 13.6 x 307 / 100. + tokensPerSecond: 41.8, + tag: { kind: 'best', label: 'Best fit' }, + }); expect(result.headline).toEqual({ text: 'Best fit for this Mac' }); expect(result.notes).toEqual([{ text: 'Not yet tested by us on a Mac like this.' }]); expect(stringsIn(result).join(' ')).not.toMatch(/recommended/i); }); - it('lists the other four models that fit: the fast ones, the one of unknown speed, the slow one', async () => { - const result = recommend(await realMac(MACBOOK_SYSCTL, 890_926_084), STOCK, ALL_INSTALLED); - expect(result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null])).toEqual([ - ['qwen3.5:9b-q8_0', 65536, 'good', 'ok', 17.8], - ['qwen3.5:9b', 65536, 'good', 'ok', 28.6], - ['gpt-oss:20b', 131072, 'good', 'unknown', null], - ['qwen3.8:27b', 131072, 'good', 'slow', 9.1], - ]); - expect(result.alternatives[2].speedLabel).toBe('Speed unknown'); - expect(result.alternatives[3].speedLabel).toBe('Slow'); - // The two that do not fit 35 GiB at any context are not listed. - expect(order(result)).not.toContain('qwen3.8:27b-q8_0@131072'); - expect(order(result).some((entry) => entry.startsWith('qwen3.5:122b'))).toBe(false); - }); - - it('would get the 27B instead if its chip were in no chips row', async () => { - // The same Mac with the M5 Pro row removed: every speed is unknown, so model rank decides. - const hw = await realMac(MACBOOK_SYSCTL, 890_926_084); + it('keeps its measured speed when its chip is in no chips row', async () => { + // The same Mac with the M5 Pro row removed: every ESTIMATE is gone, the measurement is not. const withoutChip = stockWith((catalog) => { catalog.chips = catalog.chips.filter((row) => row.match !== 'M5 Pro'); }); - const result = recommend(hw, withoutChip, ALL_INSTALLED); - expect(result.pick).toMatchObject({ modelId: 'qwen3.8:27b', context: 131072, speed: 'unknown' }); - expect(result.pick).not.toHaveProperty('tokensPerSecond'); + const result = recommend(await macbook(), withoutChip, ALL_INSTALLED); + expect(result.pick).toMatchObject({ modelId: 'qwen3.6:35b-a3b', speed: 'ok', tokensPerSecond: 67.7, tested: true }); + // The others have no speed, so the model rank orders them. + expect(result.alternatives.map((c) => [c.modelId, c.speed])).toEqual([ + ['qwen3.8:27b', 'unknown'], + ['gpt-oss:20b', 'unknown'], + ['qwen3.5:9b-q8_0', 'unknown'], + ['qwen3.5:9b', 'unknown'], + ]); }); }); describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { /** - * Expected: no recommendation, reason `memory`, in the sentence that names both numbers. + * Expected: no recommendation, reason `memory`, in the plain sentence. * - * Why. The budget is 9.67 GiB (2/3 of 16 GiB minus 1). The only model near it is the 9B. Both - * harnesses that may be offered need 65,536 tokens of context, and there the 9B needs - * 6.10 GiB of weights + 2 GiB of KV cache + 1 GiB + 0.5 GiB of prompt cache = 9.60 GiB, which - * is 0.993 of the budget: above the 0.98 where "tight" ends. At 32,768 tokens it would fit - * (0.890, tight), but the one harness that works at that context is gated. + * Why. The budget is 9.67 GiB (10.4 GB: 2/3 of 16 GiB minus 1). The smallest model, the 9B, + * needs 6.6 GB of weights + 1.2 GB held for its smallest context of 32,768 tokens + 3 GB of + * runner + 5 GB at the peak = 15.8 GB: one and a half times the budget. Nothing fits at any + * context, so this is no longer a miss by a hair that deserves the sentence with both numbers. * - * The miss is 1.3 points on an ESTIMATED budget, so the sentence is not "too little memory" - * (what an 8 GB Mac is told): it says the budget (10.4 GB), the need (10.3 GB) and that - * Codeman estimates. + * The 3 GB and the 5 GB were measured on a 35B model and are applied to the 9B unmeasured + * (`needSource: 'estimated'`), so this answer is careful, not proven. */ - it('gets no recommendation: nothing fits at the context a harness needs', async () => { - const hw = await realMac(MACMINI_SYSCTL, 41_752_848); + it('gets no recommendation: no model fits at any context', async () => { + const hw = await macMini(); expect(hw.chip).toBe('Apple M4'); expect(hw.memClassGb).toBe(16); @@ -488,9 +651,8 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { verdict: 'none', reason: 'memory', headline: { - text: 'Codeman estimates that this Mac can give a model about {budget} GB of memory. The smallest model a coding agent can work with needs about {need} GB. That leaves no safe margin, so Codeman recommends none here.', - // floor(17,179,869,184 x 2/3) - 1 GiB, and the 9B at 65536 tokens: 10,308,921,757 bytes. - vars: { budget: '10.4', need: '10.3' }, + text: 'This Mac has {memory} GB of memory. That is too little to run a model that can drive a coding agent, so Codeman recommends none here.', + vars: { memory: '16' }, }, notes: [], alternatives: [], @@ -501,25 +663,38 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { }); }); - it('misses by a hair at 65536 tokens and would fit at 32768', async () => { - const hw = await realMac(MACMINI_SYSCTL, 41_752_848); - const nineB = STOCK.models.find((m) => m.id === 'qwen3.5:9b') as LocalModelEntry; + it('is far from fitting: the smallest model at its smallest context needs 1.5 times the budget', async () => { + const hw = await macMini(); + const nineB = stockModel('qwen3.5:9b'); const budget = deviceBudget(hw).budgetBytes; - const shareAt64k = needBytes(nineB, 65536, STOCK, hw) / budget; - const shareAt32k = needBytes(nineB, 32768, STOCK, hw) / budget; - expect(shareAt64k).toBeCloseTo(0.9932, 4); - expect(shareAt64k).toBeGreaterThan(FIT_TIGHT_MAX_SHARE); - expect(shareAt32k).toBeCloseTo(0.8898, 4); - expect(shareAt32k).toBeLessThan(FIT_TIGHT_MAX_SHARE); - // Every other model needs more than the whole budget at its smallest context. - for (const other of STOCK.models.filter((m) => m.id !== 'qwen3.5:9b')) { - const smallest = other.contexts[other.contexts.length - 1]; - expect(needBytes(other, smallest, STOCK, hw), other.id).toBeGreaterThan(budget); + expect(formatGb(needBytes(nineB, 32768))).toBe('15.8'); + expect(needBytes(nineB, 32768) / budget).toBeCloseTo(1.5187, 4); + expect(needBytes(nineB, 65536) / budget).toBeCloseTo(1.6355, 4); + // Every model needs more than the whole budget at every context. + for (const row of STOCK.models) { + for (const ctx of row.contexts) expect(needBytes(row, ctx), `${row.id}@${ctx}`).toBeGreaterThan(budget); } }); - it('would get the 9B at 32768 tokens, as a limited answer, once the gated harness is opened', async () => { - const hw = await realMac(MACMINI_SYSCTL, 41_752_848); + it('gets none with the budget its runtime measured, too', async () => { + // The runtime reported 11.8 GiB here; the 6 GiB reserve leaves 10 GiB (10.7 GB) of it. + const result = recommend(await macMini(), STOCK, ALL_INSTALLED, { measuredTotalBytes: MACMINI_MEASURED_TOTAL }); + expect(result).toMatchObject({ + verdict: 'none', + reason: 'memory', + headline: { text: LOCAL_MODEL_LABELS.noneMemory, vars: { memory: '16' } }, + budgetBytes: 10 * GIB, + budgetGb: '10.7', + budgetSource: 'measured', + }); + }); + + it('gets none with a wired limit or with the gated harness opened: memory is what is missing', async () => { + // iogpu.wired_limit_mb = 14000 lifts the budget to 10 GiB (memory minus the 6 GiB reserve). + const wired = recommend({ ...(await macMini()), wiredLimitMb: 14000 }, STOCK, ALL_INSTALLED); + expect(wired).toMatchObject({ verdict: 'none', reason: 'memory', budgetSource: 'wired-limit' }); + expect(wired.budgetBytes).toBe(10 * GIB); + const ungated = stockWith((catalog) => { catalog.harnesses = catalog.harnesses.map((row) => ({ cli: row.cli, @@ -527,28 +702,9 @@ describe('the Mac mini (Mac16,10: Apple M4, 16 GB)', () => { rank: row.rank, })); }); - const result = recommend(hw, ungated, ALL_INSTALLED); - expect(result.verdict).toBe('limited'); - expect(result.pick).toMatchObject({ - modelId: 'qwen3.5:9b', - context: 32768, - cli: 'pi', - fit: 'tight', - budgetShare: 0.89, - speed: 'ok', - // 9.3 x 1.2. - tokensPerSecond: 11.2, - }); - }); - - it('would get the 9B at 65536 tokens if a wired limit had been set on it', async () => { - // iogpu.wired_limit_mb = 14000 lifts the budget to 10 GiB (memory minus the 6 GiB reserve). - const hw = { ...(await realMac(MACMINI_SYSCTL, 41_752_848)), wiredLimitMb: 14000 }; - const result = recommend(hw, STOCK, ALL_INSTALLED); - expect(result.verdict).toBe('limited'); - expect(result.budgetSource).toBe('wired-limit'); - expect(result.budgetBytes).toBe(10 * GIB); - expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b', context: 65536, cli: 'opencode', fit: 'tight' }); + const opened = recommend(await macMini(), ungated, ALL_INSTALLED); + expect(opened.reason).toBe('memory'); + expect(opened.headline.text).toBe(LOCAL_MODEL_LABELS.noneMemory); }); }); @@ -558,25 +714,27 @@ describe('every memory size, with the stock catalogue', () => { const EXPECTED: [soldGb: number, chip: string, answer: string][] = [ [8, 'Apple M2', 'none memory'], [16, 'Apple M4', 'none memory'], - // The 9B at 65536 tokens is 0.87 of 11 GiB. - [18, 'Apple M3 Pro', 'limited qwen3.5:9b@65536 tight ok'], - // gpt-oss:20b at 65536 tokens needs 15.8 GiB of the 15 GiB budget: out. - [24, 'Apple M4', 'limited qwen3.5:9b@65536 good ok'], - // gpt-oss:20b fits only as Tight here (0.853 at 65536 tokens, 0.927 at 131072), and Good - // ranks before Tight. The Q8 9B is Good but slow on a base M4 (7 tokens per second). The - // pick is a 9B, the verdict is `ok` all the same: a 20B fits this Mac. - [32, 'Apple M4', 'ok qwen3.5:9b@65536 good ok'], - [32, 'Apple M1 Max', 'ok qwen3.5:9b-q8_0@65536 good ok'], - // The 27B (0.905) and the 35B-A3B (0.974) are Tight; gpt-oss:20b is Good but has no speed - // figure, and a known speed ranks before an unknown one. + // The 9B at its smallest context needs 15.8 GB of the 11.8 GB budget. + [18, 'Apple M3 Pro', 'none memory'], + // The 9B fits only at 32768 tokens (0.979), and the one harness that works there is gated. + [24, 'Apple M4', 'none memory'], + // Only the two 9B rows fit (0.778, and 0.956 for the Q8, which is slow on a base M4), so + // this is a small-model Mac: gpt-oss:20b needs 24.2 GB of the 21.8 GB budget. + [32, 'Apple M4', 'limited qwen3.5:9b@65536 good ok'], + // Both 9B rows are fast enough here, and the higher rank wins although it is the tight one. + [32, 'Apple M1 Max', 'limited qwen3.5:9b-q8_0@65536 tight ok'], + // gpt-oss:20b fits as Tight (0.954), so the verdict is `ok`; it has no speed figure, and a + // known speed ranks before an unknown one. [36, 'Apple M4 Max', 'ok qwen3.5:9b-q8_0@65536 good ok'], // On a Pro chip the dense 27B is slow (8.1 tokens per second), the 35B-A3B is not. - [48, 'Apple M4 Pro', 'ok qwen3.6:35b-a3b@131072 good ok'], + [48, 'Apple M4 Pro', 'ok qwen3.6:35b-a3b@131072 tight ok'], + [48, 'Apple M5 Pro', 'ok qwen3.6:35b-a3b@131072 tight ok'], + // On a Max chip the 27B is fast enough (12.1) and has the higher rank. [48, 'Apple M4 Max', 'ok qwen3.8:27b@131072 good ok'], // The Q8 27B fits 47 GiB but computes to 7 tokens per second on this chip. [64, 'Apple M4 Max', 'ok qwen3.8:27b@131072 good ok'], [96, 'Apple M3 Ultra', 'ok qwen3.8:27b-q8_0@262144 good ok'], - // The 122B fits 95 GiB only as Tight (0.861) and has no speed figure. + // The 122B fits 95 GiB only as Tight (0.924) and has no speed figure. [128, 'Apple M5 Ultra', 'ok qwen3.8:27b-q8_0@262144 good ok'], ]; @@ -588,6 +746,23 @@ describe('every memory size, with the stock catalogue', () => { expect(actual).toBe(answer); }); + it('tells the four sizes without a model which kind of "too little" it is', () => { + const headlineOf = (soldGb: number): Recommendation['headline'] => + recommend(mac(soldGb, 'Apple M4'), STOCK, ALL_INSTALLED).headline; + // Nothing fits at any context: the plain sentence with the size the Mac is sold as. + for (const soldGb of [8, 16, 18]) { + expect(headlineOf(soldGb), `${soldGb} GB`).toEqual({ + text: LOCAL_MODEL_LABELS.noneMemory, + vars: { memory: String(soldGb) }, + }); + } + // A model fits, but only below the context a harness needs: both numbers. + expect(headlineOf(24)).toEqual({ + text: LOCAL_MODEL_LABELS.noneMemoryContext, + vars: { budget: '16.1', need: '17.0' }, + }); + }); + it('always pairs the pick with the top-ranked harness, OpenCode', () => { for (const hw of SWEEP) { const result = recommend(hw, STOCK, ALL_INSTALLED); @@ -745,27 +920,56 @@ describe('verdict none', () => { }); it('says both numbers when models fit, but only below the context a harness needs', () => { - // The 16 GB case of the Mac mini, with harnesses that are all present. - const result = recommend(mac(16, 'Apple M4'), STOCK, ALL_INSTALLED); + // The 24 GB case, with harnesses that are all present: the 9B fits at 32768 tokens. + const hw = mac(24, 'Apple M4'); + const nineB = stockModel('qwen3.5:9b'); + expect(needBytes(nineB, 32768) / deviceBudget(hw).budgetBytes).toBeCloseTo(0.9787, 4); + expect(needBytes(nineB, 65536) / deviceBudget(hw).budgetBytes).toBeCloseTo(1.054, 4); + + const result = recommend(hw, STOCK, ALL_INSTALLED); expectNone(result, 'memory', LOCAL_MODEL_LABELS.noneMemoryContext); // The need is the smallest one at a context a usable harness works at (the 9B at 65536), // not the 9B at 32768, which fits and which only the gated harness could use. - expect(result.headline.vars).toEqual({ budget: '10.4', need: '10.3' }); + expect(result.headline.vars).toEqual({ budget: '16.1', need: '17.0' }); expect(result.headline.text).toMatch(/^Codeman estimates/); expect(result.headline.text).not.toContain('{memory}'); }); + it('would give that Mac the 9B at 32768 tokens, as a limited answer, once the gated harness is opened', () => { + const ungated = stockWith((catalog) => { + catalog.harnesses = catalog.harnesses.map((row) => ({ + cli: row.cli, + minContext: row.minContext, + rank: row.rank, + })); + }); + const result = recommend(mac(24, 'Apple M4'), ungated, ALL_INSTALLED); + expect(result.verdict).toBe('limited'); + expect(result.pick).toMatchObject({ + modelId: 'qwen3.5:9b', + context: 32768, + cli: 'pi', + fit: 'tight', + budgetShare: 0.979, + speed: 'ok', + // 9.3 x 1.2. + tokensPerSecond: 11.2, + }); + // Only the harness that works at 32768 tokens can run it there. + expect(result.pick?.harnesses.map((h) => h.cli)).toEqual(['pi']); + }); + it('says the same two numbers when the need is far above the budget', () => { - // One model: 3 GiB at 1024 tokens (fits 10 GiB), 30 GiB at the context the harness needs. + // One model: 2 GiB at 1024 tokens (fits 10 GiB), 30 GiB at the context the harness needs. const catalog = catalogOf( - [model('probe:1', { bytes: 2 * GIB, kvBytesPerToken: 1024, contexts: [28 * 1024 * 1024, 1024] })], + [model('probe:1', { bytes: 2 * GIB, heldBytesPerToken: 1024, contexts: [28 * 1024 * 1024, 1024] })], [{ cli: 'alpha', minContext: 4096, rank: 1 }] ); const hw = mac(16, 'Apple M4', { wiredLimitMb: 14000 }); const result = recommend(hw, catalog, [fact('alpha')]); expectNone(result, 'memory', LOCAL_MODEL_LABELS.noneMemoryContext); - // 2 GiB + 28 GiB of KV cache + 1 GiB = 31 GiB; the budget is 10 GiB. - expect(result.headline.vars).toEqual({ budget: formatGb(10 * GIB), need: formatGb(31 * GIB) }); + // 2 GiB + 28 GiB held for the context = 30 GiB; the budget is 10 GiB. + expect(result.headline.vars).toEqual({ budget: formatGb(10 * GIB), need: formatGb(30 * GIB) }); expect(result.budgetSource).toBe('wired-limit'); }); @@ -853,7 +1057,7 @@ describe('verdict none', () => { it('gives the reader every number of the sum: download + reserve - free = missing', () => { // 12 GB free is more than the 6.6 GB download, so without the reserve "not enough" reads false. - const hw = mac(24, 'Apple M4', { freeDiskBytes: 12 * GB, freeDiskGb: formatGb(12 * GB) }); + const hw = mac(32, 'Apple M4', { freeDiskBytes: 12 * GB, freeDiskGb: formatGb(12 * GB) }); const vars = recommend(hw, STOCK, ALL_INSTALLED).headline.vars as Record; expect(vars).toEqual({ download: '6.6', reserve: '10.7', free: '12.0', missing: '5.3' }); expect(vars.reserve).toBe(formatGb(DISK_HEADROOM_BYTES)); @@ -895,11 +1099,11 @@ describe('verdict none', () => { ]); }); - it('turns on 0.4 GB of disk on the real MacBook without calling it a small-model Mac', async () => { - // The cut for the 35B-A3B: 22,621,314,161 + 10 GiB = 33,358,732,401 bytes free. - const macbook = await realMac(MACBOOK_SYSCTL, 890_926_084); + it('turns on 0.8 GB of disk on the real MacBook without calling it a small-model Mac', async () => { + // The cut for the 35B-A3B: its real download of 23,105,798,956 + 10 GiB = 33,843,217,196 bytes free. + const hw = await macbook(); const withFree = (bytes: number): HardwareFacts => ({ - ...macbook, + ...hw, freeDiskBytes: bytes, freeDiskGb: formatGb(bytes), }); @@ -907,20 +1111,22 @@ describe('verdict none', () => { const enough = recommend(withFree(34 * GB), STOCK, ALL_INSTALLED); expect(enough.verdict).toBe('ok'); expect(enough.pick?.modelId).toBe('qwen3.6:35b-a3b'); - expect(enough.notes).toEqual([{ text: LOCAL_MODEL_LABELS.noteUntested }]); + expect(enough.notes).toEqual([]); const short = recommend(withFree(33 * GB), STOCK, ALL_INSTALLED); expect(short.verdict).toBe('ok'); + // The tested model has no room, so nothing here is "Recommended". expect(short.headline.text).toBe(LOCAL_MODEL_LABELS.headlineOk); expect(short.pick?.modelId).toBe('qwen3.5:9b-q8_0'); expect(order(short)).not.toContain('qwen3.6:35b-a3b@131072'); expect(short.notes).toEqual([ { text: LOCAL_MODEL_LABELS.noteSmallPick }, - { text: LOCAL_MODEL_LABELS.noteDiskDropped, vars: { model: 'Qwen3.6 35B-A3B', missing: '0.4' } }, + { text: LOCAL_MODEL_LABELS.noteDiskDropped, vars: { model: 'Qwen3.6 35B-A3B', missing: '0.8' } }, { text: LOCAL_MODEL_LABELS.noteUntested }, ]); // The exact cut: one byte decides, and the note appears with the first missing byte. - const cut = 22_621_314_161 + DISK_HEADROOM_BYTES; + const cut = 23_105_798_956 + DISK_HEADROOM_BYTES; + expect(cut).toBe(33_843_217_196); expect(recommend(withFree(cut), STOCK, ALL_INSTALLED).pick?.modelId).toBe('qwen3.6:35b-a3b'); expect(recommend(withFree(cut - 1), STOCK, ALL_INSTALLED).pick?.modelId).toBe('qwen3.5:9b-q8_0'); }); @@ -948,9 +1154,9 @@ describe('verdict none', () => { it('allows a download that leaves exactly 10 GiB', () => { const exact = STOCK.models[0].bytes + DISK_HEADROOM_BYTES; expect(DISK_HEADROOM_BYTES).toBe(10 * GIB); - const fits = recommend(mac(24, 'Apple M4', { freeDiskBytes: exact }), STOCK, ALL_INSTALLED); + const fits = recommend(mac(32, 'Apple M4', { freeDiskBytes: exact }), STOCK, ALL_INSTALLED); expect(fits.pick?.modelId).toBe('qwen3.5:9b'); - const short = recommend(mac(24, 'Apple M4', { freeDiskBytes: exact - 1 }), STOCK, ALL_INSTALLED); + const short = recommend(mac(32, 'Apple M4', { freeDiskBytes: exact - 1 }), STOCK, ALL_INSTALLED); expect(short.reason).toBe('disk'); }); @@ -973,12 +1179,12 @@ describe('verdict none', () => { }); }); -// ---------- 4. Verdicts, ranking, harnesses, tested ---------- +// ---------- 4. Verdicts, ranking, harnesses, tested, rows ---------- describe('verdict limited', () => { it('is the answer when no model that fits has 12 billion parameters or more', () => { expect(LIMITED_BELOW_PARAMS_B).toBe(12); - const result = recommend(mac(24, 'Apple M4'), STOCK, ALL_INSTALLED); + const result = recommend(mac(32, 'Apple M4'), STOCK, ALL_INSTALLED); expect(result.verdict).toBe('limited'); expect(result.reason).toBeUndefined(); expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b', paramsB: 9, context: 65536, fit: 'good' }); @@ -988,11 +1194,16 @@ describe('verdict limited', () => { expect(result.notes).toEqual([{ text: 'Not yet tested by us on a Mac like this.' }]); }); - it('holds for a tight fit too (18 GB)', () => { - const result = recommend(mac(18, 'Apple M3 Pro'), STOCK, ALL_INSTALLED); + it('holds when the pick is a tight fit too (32 GB with a fast chip)', () => { + const result = recommend(mac(32, 'Apple M1 Max'), STOCK, ALL_INSTALLED); expect(result.verdict).toBe('limited'); - expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b', fit: 'tight', fitLabel: 'Tight', budgetShare: 0.873 }); - expect(result.alternatives).toEqual([]); + expect(result.pick).toMatchObject({ + modelId: 'qwen3.5:9b-q8_0', + fit: 'tight', + fitLabel: 'Tight', + budgetShare: 0.956, + }); + expect(result.alternatives.map((c) => c.modelId)).toEqual(['qwen3.5:9b']); }); it('draws the line at 12 billion: 11.9 is limited, 12 is ok', () => { @@ -1005,11 +1216,14 @@ describe('verdict limited', () => { }); it('is about the MAC, not the pick: a larger model further down the list lifts it', () => { - // 32 GB: the 20B fits only as Tight, so a 9B is the pick. The Mac can still run a 20B, so - // "This Mac can run a small model" would be false of it. - const result = recommend(mac(32, 'Apple M4'), STOCK, ALL_INSTALLED); - expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b', paramsB: 9 }); - expect(result.alternatives.map((c) => [c.modelId, c.fit])).toContainEqual(['gpt-oss:20b', 'tight']); + // 36 GB: the 20B fits as Tight and has no speed figure, so a 9B is the pick. The Mac can + // still run a 20B, so "This Mac can run a small model" would be false of it. + const result = recommend(mac(36, 'Apple M4 Max'), STOCK, ALL_INSTALLED); + expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b-q8_0', paramsB: 9, fit: 'good', speed: 'ok' }); + expect(result.alternatives.map((c) => [c.modelId, c.fit, c.speed])).toEqual([ + ['qwen3.5:9b', 'good', 'ok'], + ['gpt-oss:20b', 'tight', 'unknown'], + ]); expect(result.verdict).toBe('ok'); expect(result.headline).toEqual({ text: 'Best fit for this Mac' }); // The pick is still called what it is. @@ -1021,30 +1235,16 @@ describe('verdict limited', () => { ]); }); - it('does not tell a 36 GB Mac it is a small-model Mac above a list of 20B, 27B and 35B models', () => { - const result = recommend(mac(36, 'Apple M4 Max'), STOCK, ALL_INSTALLED); - expect(result.pick).toMatchObject({ modelId: 'qwen3.5:9b-q8_0', fit: 'good', speed: 'ok' }); - expect(result.alternatives.map((c) => [c.modelId, c.fit, c.speed])).toEqual([ - ['qwen3.5:9b', 'good', 'ok'], - ['gpt-oss:20b', 'good', 'unknown'], - // Both Tight and fast enough: the higher model rank first. - ['qwen3.8:27b', 'tight', 'ok'], - ['qwen3.6:35b-a3b', 'tight', 'ok'], - ]); - expect(result.verdict).toBe('ok'); - expect(result.headline.text).not.toBe(LOCAL_MODEL_LABELS.headlineLimited); - expect(result.notes[0]).toEqual({ text: LOCAL_MODEL_LABELS.noteSmallPick }); - }); - it('does not depend on whether the chip is in the chips table', () => { - // Knowing the chip must not lower the verdict: before, a known chip made the 9B win on - // speed and the answer limited, while an unknown chip got `ok` with the 20B. - for (const soldGb of [32, 36]) { - const known = recommend(mac(soldGb, 'Apple M4 Max'), STOCK, ALL_INSTALLED); - const unknown = recommend(mac(soldGb, 'Apple M6'), STOCK, ALL_INSTALLED); - expect([soldGb, known.verdict]).toEqual([soldGb, unknown.verdict]); - expect(known.verdict).toBe('ok'); - } + // Knowing the chip must not change the verdict: it is about what fits, not about speed. + const verdicts = [32, 36].map((soldGb) => [ + recommend(mac(soldGb, 'Apple M4 Max'), STOCK, ALL_INSTALLED).verdict, + recommend(mac(soldGb, 'Apple M6'), STOCK, ALL_INSTALLED).verdict, + ]); + expect(verdicts).toEqual([ + ['limited', 'limited'], + ['ok', 'ok'], + ]); }); it('is limited exactly when nothing with 12 billion parameters fits, on every Mac of the sweep', () => { @@ -1055,7 +1255,7 @@ describe('verdict limited', () => { const largeFits = STOCK.models.some( (m) => m.paramsB >= LIMITED_BELOW_PARAMS_B && - m.contexts.some((ctx) => ctx >= 65536 && needBytes(m, ctx, STOCK, hw) / budget <= FIT_TIGHT_MAX_SHARE) + m.contexts.some((ctx) => ctx >= 65536 && needBytes(m, ctx) / budget <= FIT_TIGHT_MAX_SHARE) ); expect([hw.memClassGb, hw.chip, result.verdict]).toEqual([hw.memClassGb, hw.chip, largeFits ? 'ok' : 'limited']); // The small-pick note: under `ok` only, and exactly when the pick is small. @@ -1121,7 +1321,7 @@ describe('speed', () => { // 48 GB: five models fit, so all of them are listed and gpt-oss:20b is among them. for (const chip of ['Apple M1', 'Apple M4 Max', 'Apple M5 Ultra']) { const result = recommend(mac(48, chip), STOCK, ALL_INSTALLED); - const gptOss = [result.pick, ...result.alternatives].find((c) => c?.modelId === 'gpt-oss:20b'); + const gptOss = rows(result).find((c) => c.modelId === 'gpt-oss:20b'); expect(gptOss, chip).toMatchObject({ speed: 'unknown', speedLabel: 'Speed unknown' }); expect(gptOss).not.toHaveProperty('tokensPerSecond'); } @@ -1130,7 +1330,7 @@ describe('speed', () => { it('is unknown for every model on a chip that is in no chips row: no bandwidth is guessed', () => { for (const chip of ['Apple M6', 'Apple M6 Pro', 'Apple M10', 'Some Future Chip']) { const result = recommend(mac(64, chip), STOCK, ALL_INSTALLED); - for (const candidate of [result.pick, ...result.alternatives]) { + for (const candidate of rows(result)) { expect(candidate, chip).toMatchObject({ speed: 'unknown', speedLabel: 'Speed unknown' }); expect(candidate).not.toHaveProperty('tokensPerSecond'); } @@ -1139,11 +1339,32 @@ describe('speed', () => { const result = recommend(without(mac(64, 'x'), 'chip'), STOCK, ALL_INSTALLED); expect(result.pick?.speed).toBe('unknown'); }); + + it('is the speed that was measured, for a row tested on a Mac like this', () => { + const entry = (tokensPerSecond?: number): TestedOn => ({ + chip: 'Apple M4 Max', + memGb: 128, + cli: 'alpha', + date: '2026-11-06', + ...(tokensPerSecond !== undefined ? { tokensPerSecond } : {}), + }); + const pickOf = (tested: TestedOn[]): Candidate | undefined => + recommend(HUNDRED_GIB_MAC, catalogOf([model('probe:1', { speedPer100Gbps: 5, tested })]), [fact('alpha')]).pick; + // The estimate: 5 x 4. + expect(pickOf([])).toMatchObject({ tested: false, tokensPerSecond: 20 }); + // The measurement replaces it, and is graded like any speed. + expect(pickOf([entry(33.33)])).toMatchObject({ tested: true, speed: 'ok', tokensPerSecond: 33.3 }); + expect(pickOf([entry(4)])).toMatchObject({ tested: true, speed: 'slow', speedLabel: 'Slow', tokensPerSecond: 4 }); + // A tested entry without a speed keeps the estimate. + expect(pickOf([entry()])).toMatchObject({ tested: true, tokensPerSecond: 20 }); + // An entry for another Mac changes nothing. + expect(pickOf([{ ...entry(99), memGb: 64 }])).toMatchObject({ tested: false, tokensPerSecond: 20 }); + }); }); describe('the ranking', () => { it('puts ok before unknown before slow, ahead of the model rank', () => { - // Three models that fit equally well. The ranks run AGAINST the speed. + // Three models that fit. The ranks run AGAINST the speed. const catalog = catalogOf([ model('slow-top:1', { rank: 3, speedPer100Gbps: 1 }), model('unknown-mid:1', { rank: 2 }), @@ -1151,7 +1372,7 @@ describe('the ranking', () => { ]); const result = recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]); expect(order(result)).toEqual(['ok-low:1@1024', 'unknown-mid:1@1024', 'slow-top:1@1024']); - expect([result.pick, ...result.alternatives].map((c) => c?.speed)).toEqual(['ok', 'unknown', 'slow']); + expect(rows(result).map((c) => c.speed)).toEqual(['ok', 'unknown', 'slow']); }); it('ranks by model rank alone when every speed is unknown', () => { @@ -1169,7 +1390,7 @@ describe('the ranking', () => { it('ranks an unknown speed below a known ok one and above a slow one, on a real tier', () => { // 48 GB, M4 Pro: three ok models, then gpt-oss:20b (unknown), then the 27B (slow). const result = recommend(mac(48, 'Apple M4 Pro'), STOCK, ALL_INSTALLED); - expect([result.pick, ...result.alternatives].map((c) => [c?.modelId, c?.speed])).toEqual([ + expect(rows(result).map((c) => [c.modelId, c.speed])).toEqual([ ['qwen3.6:35b-a3b', 'ok'], ['qwen3.5:9b-q8_0', 'ok'], ['qwen3.5:9b', 'ok'], @@ -1178,13 +1399,27 @@ describe('the ranking', () => { ]); }); - it('puts a good fit before a tight one, ahead of speed and rank', () => { + it('does not sort by the fit grade: a tight model of the higher rank stays above a good one', () => { + // The need is the peak at the full context, so a tight fit is a model that runs. Both are + // equally fast here; the rank decides, and the row says Tight. const catalog = catalogOf([ - modelNeeding('tight-fast-top:1', 0.9 * HUNDRED_GIB, { rank: 9, speedPer100Gbps: 20 }), - modelNeeding('good-slow-low:1', 0.5 * HUNDRED_GIB, { rank: 1, speedPer100Gbps: 1 }), + modelNeeding('tight-top:1', 0.9 * HUNDRED_GIB, { rank: 9, speedPer100Gbps: 5 }), + modelNeeding('good-low:1', 0.5 * HUNDRED_GIB, { rank: 1, speedPer100Gbps: 5 }), ]); const result = recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]); - expect(order(result)).toEqual(['good-slow-low:1@1024', 'tight-fast-top:1@1024']); + expect(order(result)).toEqual(['tight-top:1@1024', 'good-low:1@1024']); + expect(rows(result).map((c) => c.fit)).toEqual(['tight', 'good']); + }); + + it('still puts the speed first: a good and fast model beats a tight and slow one of the higher rank', () => { + const catalog = catalogOf([ + modelNeeding('tight-slow-top:1', 0.9 * HUNDRED_GIB, { rank: 9, speedPer100Gbps: 1 }), + modelNeeding('good-fast-low:1', 0.5 * HUNDRED_GIB, { rank: 1, speedPer100Gbps: 5 }), + ]); + expect(order(recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]))).toEqual([ + 'good-fast-low:1@1024', + 'tight-slow-top:1@1024', + ]); }); it('takes the higher model rank among equals', () => { @@ -1196,33 +1431,42 @@ describe('the ranking', () => { ]); }); - it('takes the larger context of one model when both fit equally well', () => { - // 1 KiB per token: 4 GiB of KV cache at 4,194,304 tokens, still a good fit. - const catalog = catalogOf([model('probe:1', { kvBytesPerToken: 1024, contexts: [4_194_304, 1024] })]); + it('takes the larger context of one model when both fit', () => { + // 1 KiB held per token: 4 GiB at 4,194,304 tokens, still a good fit. + const catalog = catalogOf([model('probe:1', { heldBytesPerToken: 1024, contexts: [4_194_304, 1024] })]); const result = recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]); expect(result.pick?.context).toBe(4_194_304); expect(result.alternatives).toEqual([]); }); - it('steps a model down to the context at which it fits well', () => { - // Good at the small context, tight at the large one: the good one is that model's entry. + it('offers a model at the largest context it fits at, also when that fit is tight', () => { + // Good at the small context (0.8), tight at the large one (80 + 8 GiB = 0.88): the large one. const catalog = catalogOf([ - model('probe:1', { bytes: 80 * GIB, kvBytesPerToken: 1024, contexts: [8_388_608, 1024] }), + model('probe:1', { bytes: 80 * GIB, heldBytesPerToken: 1024, contexts: [8_388_608, 1024] }), ]); const result = recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]); - expect(result.pick).toMatchObject({ context: 1024, fit: 'good' }); + expect(result.pick).toMatchObject({ context: 8_388_608, fit: 'tight', budgetShare: 0.88 }); + }); + + it('steps a model down only when the larger context does not fit at all', () => { + // 92 + 8 GiB is the whole budget (out); at 1024 tokens it is 0.92. + const catalog = catalogOf([ + model('probe:1', { bytes: 92 * GIB, heldBytesPerToken: 1024, contexts: [8_388_608, 1024] }), + ]); + const result = recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]); + expect(result.pick).toMatchObject({ context: 1024, fit: 'tight', budgetShare: 0.92 }); }); it('lists each model once, never the pick again, and at most four alternatives', () => { expect(MAX_ALTERNATIVES).toBe(4); - // Six models fit a 128 GB Mac; the pick plus four are shown. + // Seven models fit a 128 GB Mac; the pick plus four are shown. const result = recommend(mac(128, 'Apple M5 Ultra'), STOCK, ALL_INSTALLED); expect(result.alternatives).toHaveLength(4); - const ids = [result.pick, ...result.alternatives].map((c) => c?.modelId); + const ids = rows(result).map((c) => c.modelId); expect(new Set(ids).size).toBe(ids.length); for (const hw of SWEEP) { const swept = recommend(hw, STOCK, ALL_INSTALLED); - const sweptIds = [swept.pick, ...swept.alternatives].filter(Boolean).map((c) => c?.modelId); + const sweptIds = rows(swept).map((c) => c.modelId); expect(new Set(sweptIds).size).toBe(sweptIds.length); expect(swept.alternatives.length).toBeLessThanOrEqual(MAX_ALTERNATIVES); } @@ -1237,21 +1481,23 @@ describe('harnesses', () => { expect(gated).toEqual(['pi']); for (const swept of SWEEP) { const result = recommend(swept, STOCK, ALL_INSTALLED); - for (const candidate of [result.pick, ...result.alternatives]) { - if (candidate) expect(gated).not.toContain(candidate.cli); + for (const candidate of rows(result)) { + expect(gated).not.toContain(candidate.cli); + expect(candidate.harnesses.map((h) => h.cli).some((cli) => gated.includes(cli))).toBe(false); } } - // 16 GB: the gated harness alone works at 32768 tokens, where the 9B would fit. Still none. - expect(recommend(mac(16, 'Apple M4'), STOCK, ALL_INSTALLED).verdict).toBe('none'); + // 24 GB: the gated harness alone works at 32768 tokens, where the 9B would fit. Still none. + expect(recommend(mac(24, 'Apple M4'), STOCK, ALL_INSTALLED).verdict).toBe('none'); }); it('never offers a context below the minimum of the harness', () => { for (const swept of SWEEP) { const result = recommend(swept, STOCK, ALL_INSTALLED); - for (const candidate of [result.pick, ...result.alternatives]) { - if (!candidate) continue; - const row = STOCK.harnesses.find((harness) => harness.cli === candidate.cli); - expect(candidate.context).toBeGreaterThanOrEqual(row?.minContext ?? Number.POSITIVE_INFINITY); + for (const candidate of rows(result)) { + for (const harness of candidate.harnesses) { + const row = STOCK.harnesses.find((entry) => entry.cli === harness.cli); + expect(candidate.context).toBeGreaterThanOrEqual(row?.minContext ?? Number.POSITIVE_INFINITY); + } } } }); @@ -1287,6 +1533,7 @@ describe('harnesses', () => { // OpenCode refuses the 27B: it goes with the next harness. const one = recommend(hw, STOCK, [fact('opencode', { acceptsModelId: refusing27b }), fact('claude')]); expect(one.pick).toMatchObject({ modelId: 'qwen3.8:27b', cli: 'claude' }); + expect(one.pick?.harnesses.map((h) => h.cli)).toEqual(['claude']); expect(one.alternatives[0]).toMatchObject({ modelId: 'qwen3.6:35b-a3b', cli: 'opencode' }); // Both refuse it: the model is not offered. const both = recommend(hw, STOCK, [ @@ -1309,48 +1556,77 @@ describe('harnesses', () => { expect(recommend(hw, STOCK, [throwing]).reason).toBe('no-harness'); }); - it('puts the fit and the model before the harness rank', () => { + it('puts the model before the harness rank, and pairs it with the best harness that can run it', () => { // `big` needs a large context and is the higher-ranked harness; `small` works from 1024. const harnesses: LocalHarnessEntry[] = [ { cli: 'big', minContext: 8_388_608, rank: 2 }, { cli: 'small', minContext: 1024, rank: 1 }, ]; - const tightAtLarge = catalogOf( - [model('probe:1', { bytes: 80 * GIB, kvBytesPerToken: 1024, contexts: [8_388_608, 1024] })], + // Fits at both contexts (tight at the large one): the higher-ranked harness, at the + // context it needs. Both harnesses can run it there. + const fitsAtLarge = catalogOf( + [model('probe:1', { bytes: 80 * GIB, heldBytesPerToken: 1024, contexts: [8_388_608, 1024] })], harnesses ); - // Tight at the context `big` needs, good at the small one: the good fit with `small` wins. - expect(recommend(HUNDRED_GIB_MAC, tightAtLarge, [fact('big'), fact('small')]).pick).toMatchObject({ - cli: 'small', - context: 1024, - fit: 'good', - }); - // Good at both: the higher-ranked harness wins, at the context it needs. - const goodAtBoth = catalogOf( - [model('probe:1', { bytes: 10 * GIB, kvBytesPerToken: 1024, contexts: [8_388_608, 1024] })], + const large = recommend(HUNDRED_GIB_MAC, fitsAtLarge, [fact('big'), fact('small')]).pick; + expect(large).toMatchObject({ cli: 'big', context: 8_388_608, fit: 'tight' }); + expect(large?.harnesses.map((h) => h.cli)).toEqual(['big', 'small']); + // Does not fit at the large context: only `small` can run it, at the small one. + const onlyAtSmall = catalogOf( + [model('probe:1', { bytes: 92 * GIB, heldBytesPerToken: 1024, contexts: [8_388_608, 1024] })], harnesses ); - expect(recommend(HUNDRED_GIB_MAC, goodAtBoth, [fact('big'), fact('small')]).pick).toMatchObject({ - cli: 'big', - context: 8_388_608, - }); + const small = recommend(HUNDRED_GIB_MAC, onlyAtSmall, [fact('big'), fact('small')]).pick; + expect(small).toMatchObject({ cli: 'small', context: 1024 }); + expect(small?.harnesses.map((h) => h.cli)).toEqual(['small']); + // A higher-ranked MODEL with the lower-ranked harness still comes before a lower-ranked + // model with the higher-ranked harness. + const twoModels = catalogOf( + [ + model('top:1', { rank: 2, contexts: [1024] }), + model('low:1', { rank: 1, heldBytesPerToken: 1024, contexts: [8_388_608] }), + ], + harnesses + ); + expect( + rows(recommend(HUNDRED_GIB_MAC, twoModels, [fact('big'), fact('small')])).map((c) => [c.modelId, c.cli]) + ).toEqual([ + ['top:1', 'small'], + ['low:1', 'big'], + ]); }); }); describe('the tested flag and the word "Recommended"', () => { - const testedOn = (chip: string, memGb: number, cli: string) => ({ chip, memGb, cli, date: '2026-11-06' }); + const testedOn = (chip: string, memGb: number, cli: string): TestedOn => ({ chip, memGb, cli, date: '2026-11-06' }); + /** The untested catalogue with one tested entry on the dense 27B. */ const with27bTested = (chip: string, memGb: number, cli: string): LocalModelCatalog => stockWith((catalog) => { - const row = catalog.models.find((m) => m.id === 'qwen3.8:27b') as LocalModelEntry; - row.tested = [testedOn(chip, memGb, cli)]; + for (const row of catalog.models) row.tested = []; + (catalog.models.find((m) => m.id === 'qwen3.8:27b') as LocalModelEntry).tested = [testedOn(chip, memGb, cli)]; }); - it('is off for every model of the stock catalogue, so "Recommended" is never said today', () => { - for (const hw of SWEEP) { + it('is on for exactly one kind of Mac in the stock catalogue: an Apple M5 Pro with 48 GB', () => { + expect(STOCK.models.flatMap((m) => m.tested.map((entry) => [m.id, entry.chip, entry.memGb, entry.cli]))).toEqual([ + ['qwen3.6:35b-a3b', 'Apple M5 Pro', 48, 'opencode'], + ]); + + const tested = recommend(mac(48, 'Apple M5 Pro'), STOCK, ALL_INSTALLED); + expect(tested.pick).toMatchObject({ modelId: 'qwen3.6:35b-a3b', cli: 'opencode', tested: true }); + expect(tested.headline.text).toBe('Recommended for this Mac'); + + // The same chip with another memory size, and the same memory size with another chip. + const others = [ + ...SWEEP, + mac(24, 'Apple M5 Pro'), + mac(64, 'Apple M5 Pro'), + mac(48, 'Apple M4 Pro'), + mac(48, 'Apple M5 Max'), + mac(48, 'Apple M5'), + ]; + for (const hw of others) { const result = recommend(hw, STOCK, ALL_INSTALLED); - for (const candidate of [result.pick, ...result.alternatives]) { - if (candidate) expect(candidate.tested).toBe(false); - } + for (const candidate of rows(result)) expect(candidate.tested, `${hw.memClassGb} ${hw.chip}`).toBe(false); expect(stringsIn(result).join(' ')).not.toMatch(/recommended/i); if (result.verdict !== 'none') { // Always the last note; a small pick under `ok` has one more before it. @@ -1359,6 +1635,16 @@ describe('the tested flag and the word "Recommended"', () => { } }); + it('is off for the tested Mac when its harness is not the tested one', () => { + // The MacBook without OpenCode: the model is the same, the pair was not run by us. + const result = recommend(mac(48, 'Apple M5 Pro'), STOCK, [fact('claude')]); + expect(result.pick).toMatchObject({ modelId: 'qwen3.6:35b-a3b', cli: 'claude', tested: false }); + // So the speed is the estimate again, and nothing says "Recommended". + expect(result.pick?.tokensPerSecond).toBe(41.8); + expect(result.headline.text).toBe('Best fit for this Mac'); + expect(result.notes).toEqual([{ text: LOCAL_MODEL_LABELS.noteUntested }]); + }); + it('turns on "Recommended for this Mac" when the pick was tested on exactly such a Mac', () => { const result = recommend(mac(48, 'Apple M4 Max'), with27bTested('Apple M4 Max', 48, 'opencode'), ALL_INSTALLED); expect(result.verdict).toBe('ok'); @@ -1368,13 +1654,12 @@ describe('the tested flag and the word "Recommended"', () => { expect(result.notes).toEqual([]); }); - it('stays off when the chip, the memory size or the harness differs', () => { + it('stays off when the chip or the memory size differs', () => { const hw = mac(48, 'Apple M4 Max'); const cases: LocalModelCatalog[] = [ with27bTested('Apple M4 Pro', 48, 'opencode'), with27bTested('Apple M4', 48, 'opencode'), with27bTested('Apple M4 Max', 64, 'opencode'), - with27bTested('Apple M4 Max', 48, 'claude'), ]; for (const catalog of cases) { const result = recommend(hw, catalog, ALL_INSTALLED); @@ -1384,47 +1669,283 @@ describe('the tested flag and the word "Recommended"', () => { } }); - it('follows the harness: tested with the second harness counts when that one is the pick', () => { + it('follows the tested harness: the pair that was run comes first, also against a higher-ranked harness', () => { + // Tested with the second harness. Both are installed: the tested pair is the pick, and + // the other harness stays selectable. const catalog = with27bTested('Apple M4 Max', 48, 'claude'); - const result = recommend(mac(48, 'Apple M4 Max'), catalog, [fact('claude')]); - expect(result.pick).toMatchObject({ cli: 'claude', tested: true }); + const result = recommend(mac(48, 'Apple M4 Max'), catalog, ALL_INSTALLED); + expect(result.pick).toMatchObject({ modelId: 'qwen3.8:27b', cli: 'claude', tested: true }); + expect(result.pick?.harnesses.map((h) => h.cli)).toEqual(['claude', 'opencode']); expect(result.headline.text).toBe('Recommended for this Mac'); + // Without the tested harness on the machine the pair is another one, and untested. + expect(recommend(mac(48, 'Apple M4 Max'), catalog, [fact('opencode')]).pick).toMatchObject({ + cli: 'opencode', + tested: false, + }); }); - it('does not change the ranking: a tested model further down stays an alternative', () => { - // On the M4 Pro the pick is the 35B-A3B. A tested 27B is flagged in the list, no more. - const result = recommend(mac(48, 'Apple M4 Pro'), with27bTested('Apple M4 Pro', 48, 'opencode'), ALL_INSTALLED); - expect(result.pick).toMatchObject({ modelId: 'qwen3.6:35b-a3b', tested: false }); - expect(result.headline.text).toBe('Best fit for this Mac'); - expect(result.notes).toEqual([{ text: LOCAL_MODEL_LABELS.noteUntested }]); - expect(result.alternatives.find((c) => c.modelId === 'qwen3.8:27b')?.tested).toBe(true); + it('ranks a tested model first: a measurement on this Mac beats the estimates of the others', () => { + // On the M4 Pro the untested pick is the 35B-A3B, because the 27B is estimated as slow. + const hw = mac(48, 'Apple M4 Pro'); + expect(recommend(hw, UNTESTED, ALL_INSTALLED).pick?.modelId).toBe('qwen3.6:35b-a3b'); + // Once the 27B was run on such a Mac, it is the pick, and the 35B-A3B is the first alternative. + const result = recommend(hw, with27bTested('Apple M4 Pro', 48, 'opencode'), ALL_INSTALLED); + expect(result.pick).toMatchObject({ modelId: 'qwen3.8:27b', tested: true, speed: 'slow' }); + expect(result.headline.text).toBe('Recommended for this Mac'); + expect(result.notes).toEqual([]); + expect(result.alternatives[0]).toMatchObject({ modelId: 'qwen3.6:35b-a3b', tested: false }); }); it('never says "Recommended" for a limited answer, tested or not', () => { const catalog = stockWith((stock) => { const row = stock.models.find((m) => m.id === 'qwen3.5:9b') as LocalModelEntry; - row.tested = [testedOn('Apple M4', 24, 'opencode')]; + row.tested = [testedOn('Apple M4', 32, 'opencode')]; }); - const result = recommend(mac(24, 'Apple M4'), catalog, ALL_INSTALLED); + const result = recommend(mac(32, 'Apple M4'), catalog, ALL_INSTALLED); expect(result.verdict).toBe('limited'); expect(result.pick?.tested).toBe(true); expect(result.headline.text).toBe(LOCAL_MODEL_LABELS.headlineLimited); - expect(stringsIn(result).join(' ')).not.toMatch(/recommended/i); + expect(stringsIn(result.headline).join(' ')).not.toMatch(/recommended/i); // It WAS tested, so the "not yet tested" note would be false. expect(result.notes).toEqual([]); }); }); +describe('what each row of the chooser carries', () => { + describe('the tag', () => { + it('is "Best fit" on an untested pick and the first kind that applies on the others', () => { + // 48 GB, M4 Max: the 27B is the pick; the 35B-A3B is tight; the two 9B rows are small; + // gpt-oss:20b is none of these and has no tag. + const result = recommend(mac(48, 'Apple M4 Max'), STOCK, ALL_INSTALLED); + expect(rows(result).map((c) => [c.modelId, c.tag ?? null])).toEqual([ + ['qwen3.8:27b', { kind: 'best', label: 'Best fit' }], + ['qwen3.6:35b-a3b', { kind: 'tight', label: 'Tight' }], + ['qwen3.5:9b-q8_0', { kind: 'small', label: 'Small model' }], + ['qwen3.5:9b', { kind: 'small', label: 'Small model' }], + ['gpt-oss:20b', null], + ]); + expect(result.alternatives[3]).not.toHaveProperty('tag'); + }); + + it('says "Best fit" on a pick that is tight: the memory bar shows how full it is', () => { + const result = recommend(mac(48, 'Apple M4 Pro'), STOCK, ALL_INSTALLED); + expect(result.pick).toMatchObject({ fit: 'tight', tag: { kind: 'best', label: 'Best fit' } }); + }); + + it('says "Tested on a Mac like this" in place of every other tag', () => { + // The MacBook's pick is tight and the pick, and says neither. + expect(recommend(mac(48, 'Apple M5 Pro'), STOCK, ALL_INSTALLED).pick?.tag).toEqual({ + kind: 'tested', + label: 'Tested on a Mac like this', + }); + // Two tested rows: the second is an alternative and says so too. + const entry: TestedOn = { chip: 'Apple M4 Max', memGb: 128, cli: 'alpha', date: '2026-11-06' }; + const catalog = catalogOf([ + model('first:1', { rank: 2, tested: [entry] }), + modelNeeding('second:1', 0.9 * HUNDRED_GIB, { rank: 1, paramsB: 9, speedPer100Gbps: 1, tested: [entry] }), + ]); + expect(rows(recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')])).map((c) => c.tag?.kind)).toEqual([ + 'tested', + 'tested', + ]); + }); + + it('orders the kinds: tight before slow before small', () => { + const tagOfSecond = (overrides: Partial, need = 0.5 * HUNDRED_GIB): string | undefined => { + const catalog = catalogOf([ + model('pick:1', { rank: 2, speedPer100Gbps: 5 }), + modelNeeding('second:1', need, { rank: 1, speedPer100Gbps: 5, ...overrides }), + ]); + return recommend(HUNDRED_GIB_MAC, catalog, [fact('alpha')]).alternatives[0].tag?.kind; + }; + // Tight, slow and small at once. + expect(tagOfSecond({ speedPer100Gbps: 1, paramsB: 9 }, 0.9 * HUNDRED_GIB)).toBe('tight'); + // Slow and small. + expect(tagOfSecond({ speedPer100Gbps: 1, paramsB: 9 })).toBe('slow'); + // Small only. + expect(tagOfSecond({ paramsB: 9 })).toBe('small'); + expect(tagOfSecond({ paramsB: 11.9 })).toBe('small'); + // None of them: no tag. An unknown speed is not a tag either. + expect(tagOfSecond({})).toBeUndefined(); + expect(tagOfSecond({ speedPer100Gbps: undefined })).toBeUndefined(); + }); + + it('takes every tag text from the fixed list', () => { + const texts = new Set(Object.values(LOCAL_MODEL_TAG_LABELS)); + for (const hw of [...SWEEP, mac(48, 'Apple M5 Pro')]) { + for (const candidate of rows(recommend(hw, STOCK, ALL_INSTALLED))) { + if (!candidate.tag) continue; + expect(texts.has(candidate.tag.label)).toBe(true); + expect(candidate.tag.label).toBe(LOCAL_MODEL_TAG_LABELS[candidate.tag.kind]); + } + } + }); + + it('gives exactly one row the pick tag, and it is the first', () => { + for (const hw of SWEEP) { + const result = recommend(hw, STOCK, ALL_INSTALLED); + if (!result.pick) continue; + expect(result.pick.tag?.kind).toBe('best'); + expect(result.alternatives.some((c) => c.tag?.kind === 'best')).toBe(false); + } + }); + }); + + describe('the harnesses', () => { + const hw = mac(48, 'Apple M4 Max'); + + it('lists every harness that can run the model at its context, best first, the flat one first', () => { + const result = recommend(hw, STOCK, ALL_INSTALLED); + for (const candidate of rows(result)) { + expect(candidate.harnesses).toEqual([ + { cli: 'opencode', cliLabel: 'OpenCode', installed: true }, + { cli: 'claude', cliLabel: 'Claude Code', installed: true, slowFirstTurn: true }, + ]); + expect(candidate.harnesses[0]).toMatchObject({ + cli: candidate.cli, + cliLabel: candidate.cliLabel, + installed: candidate.cliInstalled, + }); + } + }); + + it('prints the install command of a harness that is not installed, and only of such a one', () => { + const command = 'curl -fsSL https://example.com/install | bash'; + const facts = [ + fact('opencode', { installed: false, installable: true, installCommand: command }), + // Installed: a command the caller knows is not shown. + fact('claude', { installCommand: 'npm install -g something' }), + ]; + expect(recommend(hw, STOCK, facts).pick?.harnesses).toEqual([ + { cli: 'opencode', cliLabel: 'OpenCode', installed: false, installCommand: command }, + { cli: 'claude', cliLabel: 'Claude Code', installed: true, slowFirstTurn: true }, + ]); + // Installable without a command the caller could name: listed, without the field. + const bare = recommend(hw, STOCK, [fact('opencode', { installed: false, installable: true })]); + expect(bare.pick?.harnesses).toEqual([{ cli: 'opencode', cliLabel: 'OpenCode', installed: false }]); + }); + + it('leaves out a harness that refuses the model id or is not on the machine', () => { + const facts = [ + fact('opencode', { acceptsModelId: (id: string) => id !== 'qwen3.8:27b' }), + fact('claude', { installed: false, installable: false }), + ]; + const result = recommend(hw, STOCK, facts); + for (const candidate of rows(result)) expect(candidate.harnesses.map((h) => h.cli)).toEqual(['opencode']); + expect(order(result)).not.toContain('qwen3.8:27b@131072'); + }); + + it('is never empty', () => { + for (const swept of [...SWEEP, mac(48, 'Apple M5 Pro')]) { + for (const candidate of rows(recommend(swept, STOCK, ALL_INSTALLED))) { + expect(candidate.harnesses.length).toBeGreaterThan(0); + expect(new Set(candidate.harnesses.map((h) => h.cli)).size).toBe(candidate.harnesses.length); + } + } + }); + }); + + describe('the download of an install', () => { + const hw = mac(48, 'Apple M4 Max'); + + it('is the model alone once the runtime is installed', () => { + for (const candidate of rows(recommend(hw, STOCK, ALL_INSTALLED))) { + expect(candidate.installBytes).toBe(candidate.downloadBytes); + expect(candidate.installGb).toBe(candidate.downloadGb); + } + }); + + it('adds the runtime archive to every row while the runtime is not installed', () => { + const result = recommend(hw, STOCK, ALL_INSTALLED, { runtimeDownloadBytes: RUNTIME_ARCHIVE_BYTES }); + for (const candidate of rows(result)) { + expect(candidate.installBytes).toBe(candidate.downloadBytes + 185_971_671); + expect(candidate.installGb).toBe(formatGb(candidate.installBytes)); + } + // 17,741,871,939 + 185,971,671: reads 17.9 where the model alone reads 17.7. + expect(result.pick).toMatchObject({ downloadGb: '17.7', installBytes: 17_927_843_610, installGb: '17.9' }); + }); + + it('does not let it change which model is the pick, what fits, or the need', () => { + const plain = recommend(hw, STOCK, ALL_INSTALLED); + const withRuntime = recommend(hw, STOCK, ALL_INSTALLED, { runtimeDownloadBytes: RUNTIME_ARCHIVE_BYTES }); + const strip = (candidate: Candidate): Omit => { + const { installBytes: _bytes, installGb: _gb, ...rest } = candidate; + return rest; + }; + expect(rows(withRuntime).map(strip)).toEqual(rows(plain).map(strip)); + }); + + it('ignores a runtime size that is no byte count', () => { + for (const runtimeDownloadBytes of [0, -5, Number.NaN, Number.POSITIVE_INFINITY]) { + const pick = recommend(hw, STOCK, ALL_INSTALLED, { runtimeDownloadBytes }).pick as Candidate; + expect(pick.installBytes, String(runtimeDownloadBytes)).toBe(pick.downloadBytes); + } + }); + }); + + it('says whether the need of a row rests on a measurement of that model', () => { + const result = recommend(mac(64, 'Apple M4 Max'), STOCK, ALL_INSTALLED); + expect(Object.fromEntries(rows(result).map((c) => [c.modelId, c.needSource]))).toEqual({ + 'qwen3.8:27b': 'estimated', + 'qwen3.6:35b-a3b': 'measured', + 'qwen3.5:9b-q8_0': 'estimated', + 'qwen3.5:9b': 'estimated', + 'gpt-oss:20b': 'estimated', + }); + }); +}); + +describe('the measured budget, end to end', () => { + it('re-runs the recommendation against what the runtime reported', async () => { + const hw = await macbook(); + const estimated = recommend(hw, STOCK, ALL_INSTALLED); + const measured = recommend(hw, STOCK, ALL_INSTALLED, { measuredTotalBytes: MACBOOK_MEASURED_TOTAL }); + expect([estimated.budgetSource, estimated.budgetGb]).toEqual(['estimated', '37.6']); + expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']); + // The same rows in the same order; only the share of each changes. + expect(order(measured)).toEqual(order(estimated)); + expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.555, 0.452, 0.709, 0.814]); + expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.52, 0.423, 0.663, 0.762]); + for (const candidate of rows(measured)) { + expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3); + } + }); + + it('can rule a model out that the estimate offered, and offer one the estimate ruled out', () => { + const hw = mac(48, 'Apple M4 Pro'); + // The estimate (35 GiB) offers the 35B-A3B at 131072 tokens (0.957). + expect(order(recommend(hw, STOCK, ALL_INSTALLED))[0]).toBe('qwen3.6:35b-a3b@131072'); + expect(modelFitsBudget(hw, STOCK, 'qwen3.6:35b-a3b', 131072)).toBe(true); + // A runtime that reports only 33 GiB rules that context out (1.015), so the install job + // stops before the download, and the recommendation steps the model down to 65536 (0.946). + const lower = { measuredTotalBytes: 33 * GIB }; + expect(modelFitsBudget(hw, STOCK, 'qwen3.6:35b-a3b', 131072, lower)).toBe(false); + expect(modelFitsBudget(hw, STOCK, 'qwen3.6:35b-a3b', 65536, lower)).toBe(true); + expect(recommend(hw, STOCK, ALL_INSTALLED, lower).pick).toMatchObject({ + modelId: 'qwen3.6:35b-a3b', + context: 65536, + budgetShare: 0.946, + }); + // At 31 GiB it fits at no context and is not listed at all. + const lowest = { measuredTotalBytes: 31 * GIB }; + expect(modelFitsBudget(hw, STOCK, 'qwen3.6:35b-a3b', 65536, lowest)).toBe(false); + expect(order(recommend(hw, STOCK, ALL_INSTALLED, lowest)).some((entry) => entry.startsWith('qwen3.6'))).toBe(false); + // On a 36 GB Mac the estimate (26 GiB) rules the 27B out at 65536 tokens; 27 GiB admits it. + const small = mac(36, 'Apple M4 Max'); + expect(modelFitsBudget(small, STOCK, 'qwen3.8:27b', 65536)).toBe(false); + expect(modelFitsBudget(small, STOCK, 'qwen3.8:27b', 65536, { measuredTotalBytes: 27 * GIB })).toBe(true); + }); +}); + describe('the shape of the answer', () => { const labels = new Set(Object.values(LOCAL_MODEL_LABELS)); it('takes every sentence and label from the fixed list', () => { - for (const hw of SWEEP) { + for (const hw of [...SWEEP, mac(48, 'Apple M5 Pro')]) { const result = recommend(hw, STOCK, ALL_INSTALLED); expect(labels.has(result.headline.text)).toBe(true); for (const note of result.notes) expect(labels.has(note.text)).toBe(true); - for (const candidate of [result.pick, ...result.alternatives]) { - if (!candidate) continue; + for (const candidate of rows(result)) { expect(labels.has(candidate.fitLabel)).toBe(true); if (candidate.speedLabel !== undefined) expect(labels.has(candidate.speedLabel)).toBe(true); expect(candidate.speedLabel === undefined).toBe(candidate.speed === 'ok'); @@ -1450,7 +1971,7 @@ describe('the shape of the answer', () => { ]); // The two other sentences of the `memory` reason, and an answer with every kind of note. answers.push( - recommend(mac(16, 'Apple M4'), STOCK, ALL_INSTALLED), + recommend(mac(24, 'Apple M4'), STOCK, ALL_INSTALLED), recommend({ ...mac(48, 'Apple M4'), memBytes: 0, memGb: '0.0', memClassGb: 0 }, STOCK, ALL_INSTALLED), recommend(mac(48, 'Apple M4', { freeDiskBytes: 20 * GIB, freeDiskGb: '21.5' }), STOCK, ALL_INSTALLED) ); @@ -1476,26 +1997,26 @@ describe('the shape of the answer', () => { }); it('prints every size in one unit: decimal GB from the one formatter, never GiB', () => { - for (const hw of SWEEP) { - const result = recommend(hw, STOCK, ALL_INSTALLED); + for (const hw of [...SWEEP, mac(48, 'Apple M5 Pro')]) { + const result = recommend(hw, STOCK, ALL_INSTALLED, { runtimeDownloadBytes: RUNTIME_ARCHIVE_BYTES }); expect(result.budgetGb).toBe(formatGb(result.budgetBytes)); - for (const candidate of [result.pick, ...result.alternatives]) { - if (!candidate) continue; + for (const candidate of rows(result)) { expect(candidate.needGb).toBe(formatGb(candidate.needBytes)); expect(candidate.downloadGb).toBe(formatGb(candidate.downloadBytes)); + expect(candidate.installGb).toBe(formatGb(candidate.installBytes)); expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / result.budgetBytes, 3); } expect(JSON.stringify(result)).not.toMatch(/GiB|MiB/); } }); - it('adds up for a person: download, plus what the context and the runtime take, is the need', () => { + it('adds up for a person: download, plus what the context and the runner take, is the need', () => { const result = recommend(mac(48, 'Apple M4 Max'), STOCK, ALL_INSTALLED); const pick = result.pick as Candidate; - // 17.7 GB of weights + 8 GiB of KV cache + 1 GiB + 2 GiB of prompt cache, all in decimal GB. + // 17.7 GB of weights + 4.8 GB held for 131,072 tokens + 3 GB + 5 GB, all in decimal GB. expect(pick.downloadGb).toBe('17.7'); - expect(pick.needGb).toBe('29.6'); - expect(formatGb(pick.needBytes - pick.downloadBytes)).toBe('11.8'); + expect(pick.needGb).toBe('30.6'); + expect(formatGb(pick.needBytes - pick.downloadBytes)).toBe('12.8'); expect(result.budgetGb).toBe('37.6'); expect(pick.needBytes).toBeLessThan(result.budgetBytes); }); @@ -1513,12 +2034,47 @@ describe('the shape of the answer', () => { ); }); + it('gives every row the same fields, plus the ones that say something only sometimes', () => { + const always = [ + 'budgetShare', + 'cli', + 'cliInstalled', + 'cliLabel', + 'context', + 'downloadBytes', + 'downloadGb', + 'fit', + 'fitLabel', + 'harnesses', + 'installBytes', + 'installGb', + 'license', + 'modelId', + 'modelLabel', + 'needBytes', + 'needGb', + 'needSource', + 'paramsB', + 'speed', + 'tested', + ]; + const sometimes = ['slowFirstTurn', 'speedLabel', 'tag', 'tokensPerSecond']; + for (const hw of [...SWEEP, mac(48, 'Apple M5 Pro')]) { + for (const candidate of rows(recommend(hw, STOCK, ALL_INSTALLED))) { + const keys = Object.keys(candidate); + expect(keys.filter((key) => !sometimes.includes(key)).sort()).toEqual(always); + for (const key of keys) expect((candidate as unknown as Record)[key]).not.toBeUndefined(); + } + } + }); + it('is pure: frozen inputs are enough, and the same inputs give the same answer', () => { const hw = deepFreeze(mac(48, 'Apple M4 Pro')); const catalog = deepFreeze(structuredClone(STOCK_LOCAL_MODEL_CATALOG)); const facts = deepFreeze(ALL_INSTALLED.map((entry) => ({ ...entry }))); - const first = recommend(hw, catalog, facts); - const second = recommend(hw, catalog, facts); + const options = deepFreeze({ measuredTotalBytes: MACBOOK_MEASURED_TOTAL, runtimeDownloadBytes: 1 }); + const first = recommend(hw, catalog, facts, options); + const second = recommend(hw, catalog, facts, options); expect(second).toEqual(first); expect(first.verdict).toBe('ok'); });