BitSquare/Codeman
2
0 Fork 0
mirror of https://github.com/Ark0N/Codeman.git synced 2026-11-14 10:39:32 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge #784: a local model catalogue for every Mac memory size, and its weekly check

Browse Source
This commit is contained in:
Codeman maintainer committed 2026-11-08 11:10:13 +01:00
2 parents 8fa2884f48 04c4e43913
commit 8a54ded78b
21 files changed
+3199 -476
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/config/local-model-catalog.ts
+278 -94
View File
@@ -8,6 +8,11 @@
* is downloaded is a compiled constant of its own (src/local-model/runtime.ts).
* - A model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`), so it can
* only ever name a model of the default registry and namespace.
* - A model row names the `runner` a Mac uses for its tag, and its memory figures follow that
* runner's rule (MEMORY below). A row's figures are its OWN: measured on that model, or
* computed from that model's architecture. test/local-model-catalog.test.ts recomputes every
* computed figure from a table of architectures and refuses a row that is not in it, so a
* new row needs its model's config.json read, not a figure borrowed from a neighbour.
* - A harness is a ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`). The `cli` is a CLI
* registry id as data: no code may compare it with a literal. Whether that CLI exists and can
* take a custom model is checked against the registry at call time, in
@@ -19,48 +24,131 @@
* throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses,
* so a bad row fails the gate and never reaches the route.
*
* Where each figure of the stock rows comes from, and what was checked (2026-11-06):
* Where each figure of the stock rows comes from, and what was checked (2026-11-08). [V] read
* at the source that day, [M] measured on a Mac, [C] computed from [V] inputs:
*
* | Figure | Source | Checked |
* |----------------------|-----------------------------------------------------------|--------------------------------------|
* | `id` | the registry's tag | yes, by the facts review |
* | `bytes` | `qwen3.6:35b-a3b`: the total of a real pull on a Mac | that row: measured. It is an MLX |
* | | (23,105,798,956). Every other row: the sum of the layer | build on a Mac, 2% above its |
* | | sizes of the tag's registry manifest | manifest. The others: not on a Mac |
* | `heldBytesPerToken`, | measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with | that row: measured. The others carry |
* | `overheadBytes`, | 48 GB: 36,000 to 38,000 bytes held per token of context, | the SAME three figures, not their |
* | `peakExtraBytes` | 2.5 to 3 GB beside the weights, and about 5 GB more for a | own: nobody ran them. `figures` says |
* | | moment while a long prompt is processed | which row is which |
* | `license` | the model card (all Apache-2.0) | read by the research, not re-checked |
* | `paramsB` | 27.8 from the model card; the others are the count in the | no. Only the 12 billion line reads |
* | | model's name | it, and no row is near that line |
* | `contexts`, `rank` | a judgement of the design | no |
* | `speedPer100Gbps` | one published run on an M1 Max (400 GB/s): generation | the run was read. It stands for less |
* | | only, short prompt. The two q8_0 rows are SCALED by file | than a session: no long prompt, no |
* | | size, not measured | thinking tokens |
* | `id` | the registry's tag. The GGUF tags in lowercase: the | [V] every row: the registry answers |
* | | registry spells them `-q4_K_M`, and an id here is | the lowercase id with the same |
* | | lowercase (`LOCAL_MODEL_ID_PATTERN`) | manifest. NOT checked: that the |
* | | | runtime lists the pulled model under |
* | | | that name (Codeman compares exactly) |
* | `runner` | the build the runtime 0.40.15 pulls on Apple silicon: the | [V] every row |
* | | MLX child of a manifest list, else the one manifest | |
* | | (`runnerPreferences()`: mlx, then llamacpp) | |
* | `bytes` | every layer plus the config of that build: the `total` | [V] every row. [M] for |
* | | a pull reports | `qwen3.6:35b-a3b`, byte for byte |
* | `heldBytesPerToken`, | `qwen3.6:35b-a3b`: [M] on an Apple M5 Pro with 48 GB. | the others: [C], never run on a Mac. |
* | `overheadBytes`, | Every other row: [C] from its model's config.json for its | `figures` says which row is which. |
* | `peakExtraBytes` | runner, see MEMORY below. No row borrows another's | test/local-model-catalog.test.ts |
* | | figures: two rows share figures only when they share the | recomputes every one |
* | | architecture and the runner | |
* | `license`, `paramsB` | the model card (`paramsB` is the TOTAL, not the active | [V] |
* | | parameters) | |
* | `rank` | a judgement on the agentic coding results of the vendors' | no: vendor tables disagree with each |
* | | cards (SWE-bench Verified, SWE-bench Pro, Terminal-Bench) | other by several points |
* | `contexts` | up to the model's own window, and at least the 65536 a | no |
* | | harness needs | |
* | `speedPer100Gbps` | one published run per model family on an M1 Max | the run was read. It stands for less |
* | | (400 GB/s): generation only, short prompt. A row that is | than a session: no long prompt, no |
* | | not that run is SCALED from it, and says how | thinking tokens. An MTP build is |
* | | | faster than the run without it |
* | `tested` | our own run of the model with a harness on a real Mac | one entry, with its measured speed |
* | `minMacosMajor` | the runtime's documentation (macOS 14) | yes, by the facts review |
* | harness `minContext` | 65536: the runtime's documentation for both harnesses. | 65536 yes. 32768 no, which is why |
* | | 32768: recalled | that row is gated |
* | harness `minContext` | 65536: the runtime's documentation for OpenCode ("64k or | 65536 yes. 32768 no, which is why |
* | | higher"), and OpenCode v1.19.5's own fixed prompt (about | that row is gated |
* | | 8,000 to 10,000 tokens) and compaction rule: at 32768 it | |
* | | leaves 14,000 to 16,000 tokens, less than one default | |
* | | 2,000-line file read | |
* | `chips` | the bandwidth column of | yes, read at the source for this |
* | | https://github.com/ggml-org/llama.cpp/discussions/4167 | file |
*
* The memory figures decide a lot and rest on ONE measured model, so for every other row they
* can be wrong in either direction:
* - For the 9B rows they may be too HIGH (the 5 GB for a moment was seen with a prompt of
* 89,000 tokens on the 35B model). That errs on the careful side: a small Mac is told
* "none" before it is told to download something that may not fit.
* - For the two dense `qwen3.8:27b` rows the per-token figure may be too LOW: computed from
* its config alone that model holds 64 KiB per token, more than the 37,000 bytes measured
* on the 35B (whose own computed figure was 20 KiB). Their need at a large context is
* therefore the least certain figure of the catalogue.
* A row gets its own figures when somebody measures it. Until then the check after the
* download (the runtime must report the whole model on the GPU) is what catches a row that
* was judged too kindly.
* MEMORY. The need is `bytes + heldBytesPerToken x context + overheadBytes + peakExtraBytes`
* (src/local-model/recommend.ts). The two runners spend memory differently, so a row's three
* figures are filled by the rule of its runner:
*
* - `llamacpp` (llama-server on Metal; runtime v0.40.15 source, llm/llama_server.go, and the
* llama.cpp it bundles, b12111). It allocates the K/V cache for the WHOLE context when the
* model loads, in the cache type the runtime sets (`RUNTIME_KV_CACHE_TYPE` in
* src/local-model/runtime.ts, q8_0: 34 bytes per 32 values), and does not grow after that.
* "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
* its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N
* being the tag's own `draft_num_predict` (its params layer: 4 for
* `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default
* of 4 (`qwen3.8:27b-q8_0` sets none).
* - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model
* adds its prediction layer, whose cache llama-server keeps in a draft context of the
* same length in the draft's own cache type, f16 by default (2 bytes per value).
* - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
* or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP
* model (llama.cpp keeps one recurrent state per drafted token for rollback,
* `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
* (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
* state, or the sliding-window cache), + 1.0 GB for the runner's own buffers (calibrated
* on a third-party table of a 27B model on llama.cpp: 8.4 GB at 4,096 tokens with
* 7.15 GB of weights), + the 1024 MiB host prompt cache that `LLAMA_ARG_CACHE_RAM`
* allows (`RUNTIME_HOST_PROMPT_CACHE_MIB`). The checkpoints and the prompt cache are
* host memory, not Metal buffers; on a Mac both come out of the same memory, so they
* are counted like the rest.
* - peak = 0: its compute buffers are allocated at load and are inside the 1.0 GB.
* The one outside check of this rule: a third party measured 65,600 bytes per token of
* growth at f16 for a model of the Qwen3.8-27B architecture, where its config gives 65,536.
* The checkpoints and the rollback copies are read in the llama.cpp source
* (tools/server/server-context.cpp `create_checkpoint`, src/llama-memory-recurrent.cpp),
* not measured.
* - `mlx` (runtime v0.40.15 source, mlxrunner/). It grows the K/V cache in bf16 with the
* tokens used, keeps the recurrent state in f32 and a snapshot of it every 8,192 tokens,
* and keeps up to 8 GiB of paged-out prefix snapshots (prefix_cache.go `maxPagedOutBytes`,
* not configurable). Third-party OpenCode sessions show that growth (runtime issues 17924
* and 18131), and the runtime books an MLX model at its weights alone (issue 19417).
* - `qwen3.6:35b-a3b`: MEASURED. 36,000 to 38,000 bytes held per token after prompts of
* 89,000 and 167,000 tokens, 2.5 to 3 GB beside the weights, about 5 GB more for a
* moment while a long prompt was processed. NOT covered by that measurement: a session
* of many requests, in which the prefix snapshots may add up to 8 GiB. Its figures stay
* the measured ones (with 8 GiB on top it would not be offered on the Mac it ran on),
* and a 30-request session on it is the first thing to measure next.
* - every other `mlx` row: held = the K/V cache in bf16 + one recurrent snapshot per
* 8,192 tokens, computed from the config, x 37,000 / 28,340: the ratio of what the
* measured row really held to the same computation for it (same runner, same
* architecture family). Overhead = the measured row's 3 GB + the whole 8 GiB prefix
* budget. Peak = the measured row's 5 GB, NOT scaled: what makes it up was not
* measured.
*
* After the download the install checks that the runtime holds the whole model on the GPU.
* For a `llamacpp` row both of the runtime's figures are the buffers llama-server reports
* allocating (weights, K/V cache, compute; `MemorySize()` in llm/llama_server.go), so the
* check proves the whole load is on the GPU, whatever the cache type; what grows after the
* load in host memory (the context checkpoints and the prompt cache) is counted above, not
* checked. For an `mlx` row it proves only the weights (issue 19417): the figures above are
* all that stands between such a row and swap.
*
* Models set aside on purpose (a weekly review should look at them again):
* - `bonsai-2:27b`, `bonsai-2:9b` (ternary builds of Qwen3.8-27B and Qwen3.5-9B; the 27B
* would make a 24 GB Mac `ok`): the vendor reports no SWE-bench and no agent loop, and
* whether 2-bit weights keep tool calls reliable was never measured.
* - `ornith-1.5:9b`, `ornith-1.5:35b`, `ornith-1.5:397b` (GGUF only): strong cards, but their
* manifests name no renderer and no tool parser, so whether tool calls work through the
* runtime is unknown (runtime pull request 17965 is open). `ornith-1.5:122b` has both.
* - `gpt-oss:20b`: weak in agentic coding (SWE-bench Verified 52.4, Terminal-Bench 2.1 15.2
* in NVIDIA's table).
* - `flint:40b`: runtime issue 19191 breaks integer tool arguments after the first call.
* - `lfm2.5:24b-a2b`: its licence allows commercial use only below USD 10M of revenue.
* - `laguna-xs-2.1`: the runtime's own page names an open problem with it on macOS.
* `laguna-s-2.1`: open runtime issues 17602 and 17632.
* - `qwen3.8-flash-next:125b`: the Qwen Community License, not an open licence.
* - The plain tags `qwen3.5:9b`, `mimo-v2.6:9b` and `qwen3.8:27b`: a Mac pulls their MLX
* builds, whose 8 GiB of prefix snapshots do not fit the Macs a 9B or a 27B is for. Their
* GGUF builds are listed instead.
* - `qwen3.5:122b-a10b`: replaced by `ornith-1.5:122b`, a fine-tune of it (SWE-bench
* Verified 84.5 against 72.0) whose MLX build carries a renderer and a tool parser.
*
* `tested` has one entry: `qwen3.6:35b-a3b` with OpenCode on an Apple M5 Pro with 48 GB,
* 2026-11-06, 67.7 tokens per second. Only a Mac of exactly that chip and memory size is told
* "Recommended" and shown that speed in place of the estimate.
* "Recommended" and shown that speed in place of the estimate. Every other row was never run
* on a Mac by us, and its row says `estimated` and "not run on a Mac like this yet".
*/
import { z } from 'zod';
@@ -114,6 +202,7 @@ const LocalModelEntrySchema = z
label: z.string().min(1).max(80),
license: z.string().min(1).max(40),
paramsB: z.number().positive(),
runner: z.enum(['mlx', 'llamacpp']),
bytes: positiveInt,
heldBytesPerToken: positiveInt,
overheadBytes: z.number().int().nonnegative(),
@@ -218,8 +307,8 @@ export const LocalModelCatalogSchema: z.ZodType<LocalModelCatalog> = z
/**
* The memory figures measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with 48 GB (2026-11-06,
* the MLX runner), in decimal bytes. That row carries them as `measured`; every other row
* carries the same three as `estimated`, see the file header.
* the MLX runner), in decimal bytes. Only that row carries them; see the file header (MEMORY)
* for what they do not cover.
*/
const MEASURED_MEMORY = {
/** 36,000 to 38,000 bytes stayed held per token of context after a prompt of 89,000 and one of 167,000 tokens. */
@@ -230,117 +319,212 @@ const MEASURED_MEMORY = {
peakExtraBytes: 5_000_000_000,
} as const;
/** The llama.cpp runner's own buffers, compute buffers included (file header, MEMORY). */
const LLAMACPP_RUNNER_BYTES = 1_000_000_000;
/** The host prompt cache llama-server may keep: `LLAMA_ARG_CACHE_RAM`, 1024 MiB (`RUNTIME_HOST_PROMPT_CACHE_MIB`). */
const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024;
/** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */
const LLAMACPP_CTX_CHECKPOINTS = 8;
/** The tokens the runtime drafts per step for an MTP model whose tag sets no `draft_num_predict`, 4: so many rollback states. */
const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4;
/** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
/**
* The catalogue that ships with Codeman. Never read directly outside this file and its test:
* callers use `getLocalModelCatalog()`, which validates it first.
*/
export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
schema: 1,
updated: '2026-11-06',
updated: '2026-11-08',
// The oldest macOS the model runtime supports.
minMacosMajor: 14,
// Ordered by rank. The higher rank wins among models that fit equally well and are equally
// fast. `flint:40b` is left out on purpose: its plain tag is the slow build on a Mac and it
// has an open bug with tool arguments.
// Ordered by rank. The higher rank wins among models that fit and whose speed is graded
// alike (recommend.ts ranks the speed grade first). See the file header for the models
// that are left out and why.
models: [
{
id: 'qwen3.5:9b',
// Small Macs (16 to 32 GB, `limited`): the base model of the row below.
id: 'qwen3.5:9b-q4_k_m',
label: 'Qwen3.5 9B',
license: 'Apache-2.0',
paramsB: 9,
bytes: 6_550_825_373,
...MEASURED_MEMORY,
runner: 'llamacpp',
bytes: 6_550_825_550,
// 8 attention layers x 4 K/V heads x 256 x 2 x 34/32.
heldBytesPerToken: 17_408,
// 24 linear-attention layers hold 52,690,944 bytes of f32 state: once live, and once in
// each context checkpoint.
overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [65536, 32768],
contexts: [131072, 65536],
rank: 1,
// 37.1 tokens per second measured on 400 GB/s.
// Measured: 37.1 tokens per second on 400 GB/s (this Q4_K_M build, llama.cpp).
speedPer100Gbps: 9.3,
tested: [],
},
{
id: 'qwen3.5:9b-q8_0',
label: 'Qwen3.5 9B (Q8)',
license: 'Apache-2.0',
// Small Macs (16 to 32 GB, `limited`). MiMo-V2.6-Distill-Qwen-9B, a fine-tune of
// Qwen3.5-9B for code and agents: SWE-bench Verified 61.1 and SWE-bench Pro 44.6,
// against 60.0 and 32.0 for its base (Xiaomi's card). Requires exactly the pinned
// runtime, 0.40.15.
id: 'mimo-v2.6:9b-q4_k_m',
label: 'MiMo V2.6 9B',
license: 'MIT',
paramsB: 9,
bytes: 10_449_217_949,
...MEASURED_MEMORY,
figures: 'estimated',
contexts: [65536, 32768],
rank: 2,
// Not measured: the 9.3 of the row above scaled by file size.
speedPer100Gbps: 5.8,
tested: [],
},
{
id: 'gpt-oss:20b',
label: 'GPT-OSS 20B',
license: 'Apache-2.0',
paramsB: 20,
bytes: 13_793_441_427,
...MEASURED_MEMORY,
runner: 'llamacpp',
bytes: 6_550_811_033,
// The architecture and the quantization of the row above, so its figures.
heldBytesPerToken: 17_408,
overheadBytes: 52_690_944 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [131072, 65536],
rank: 3,
// No speed figure: its speed is unknown on every chip.
rank: 2,
// Not measured: the 9.3 of the row above, the same architecture and file size.
speedPer100Gbps: 9.3,
tested: [],
},
{
// Cohere's model for agentic software engineering, 3 billion parameters active:
// SWE-bench Verified 67.6 (vendor). Its card asks OpenCode for interleaved reasoning,
// which Codeman's OpenCode settings do not set.
id: 'north-mini-code-1.0:q4_k_m',
label: 'North Mini Code 1.0',
license: 'Apache-2.0',
paramsB: 30,
runner: 'llamacpp',
bytes: 18_593_967_008,
// 13 full-attention layers x 4 K/V heads x 128 x 2 x 34/32.
heldBytesPerToken: 14_144,
// 36 sliding-window layers: (4,096 + 512) cells x 4 x 128 x 2 x 34/32 = 180,486,144, live
// and in each context checkpoint.
overheadBytes: 180_486_144 * (1 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [262144, 131072, 65536],
rank: 3,
// Not measured: the 13.6 measured for Qwen3.6 35B-A3B on llama.cpp, which has the same
// 3 billion active parameters.
speedPer100Gbps: 13.6,
tested: [],
},
{
// Macs of 36 GB on a slower chip: the model of the tested pair, as a GGUF build whose
// memory does not grow after it loaded. With its multi-token-prediction layer.
id: 'qwen3.6:35b-a3b-mtp-q4_k_m',
label: 'Qwen3.6 35B-A3B (GGUF)',
license: 'Apache-2.0',
paramsB: 35,
runner: 'llamacpp',
bytes: 22_621_314_381,
// 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in
// f16: 2 x 256 x 2 x 2 = 2,048.
heldBytesPerToken: 12_928,
// 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 2 rollback
// copies of prediction (this tag drafts 2 tokens, `draft_num_predict` 2), and in each
// context checkpoint.
overheadBytes:
65_863_680 * (1 + 2 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [262144, 131072, 65536],
rank: 4,
// Measured: 54.5 tokens per second on 400 GB/s (llama.cpp, without prediction).
speedPer100Gbps: 13.6,
tested: [],
},
{
// 48 GB: the tested pair. SWE-bench Verified 73.4, SWE-bench Pro 49.5 (Qwen's card).
id: 'qwen3.6:35b-a3b',
label: 'Qwen3.6 35B-A3B',
license: 'Apache-2.0',
paramsB: 35,
// What a Mac really downloads: an MLX build, 2% above the 22,621,314,161 of the manifest.
// The plain tag is a manifest list; a Mac takes its MLX child.
runner: 'mlx',
// What a Mac really downloads: the MLX build, 2% above the GGUF build of the row above.
bytes: 23_105_798_956,
...MEASURED_MEMORY,
figures: 'measured',
contexts: [131072, 65536],
rank: 4,
// 54.5 tokens per second measured on 400 GB/s (3 billion parameters are active per token).
// The MLX build ran faster than this figure predicts on the one Mac we tested (67.7
// against 41.8), so on other chips the estimate is on the low side.
speedPer100Gbps: 13.6,
rank: 5,
// Measured: 67.8 tokens per second on 400 GB/s with MLX. The one Mac we tested ran it at
// 67.7 on 307 GB/s, so on other chips the estimate is on the low side.
speedPer100Gbps: 16.9,
// Run by us with OpenCode on the MacBook Pro (Mac17,13): a four-step task with tool calls.
tested: [{ chip: 'Apple M5 Pro', memGb: 48, cli: 'opencode', date: '2026-11-06', tokensPerSecond: 67.7 }],
},
{
id: 'qwen3.8:27b',
label: 'Qwen3.8 27B',
license: 'Apache-2.0',
paramsB: 27.8,
bytes: 17_741_871_939,
...MEASURED_MEMORY,
figures: 'estimated',
contexts: [131072, 65536, 32768],
rank: 5,
// 11.8 tokens per second measured on 400 GB/s.
speedPer100Gbps: 2.95,
tested: [],
},
{
// Large Macs that want 8-bit weights: the row below in Q8_0, half as fast.
id: 'qwen3.8:27b-q8_0',
label: 'Qwen3.8 27B (Q8)',
license: 'Apache-2.0',
paramsB: 27.8,
bytes: 29_978_241_837,
...MEASURED_MEMORY,
runner: 'llamacpp',
bytes: 29_978_242_050,
// The architecture of the row below, prediction layer included (its GGUF has one), and the
// runtime's 4 drafted tokens (the tag sets none), so its figures.
heldBytesPerToken: 38_912,
overheadBytes:
156_893_184 * (1 + LLAMACPP_DEFAULT_DRAFT_TOKENS + LLAMACPP_CTX_CHECKPOINTS) +
LLAMACPP_RUNNER_BYTES +
LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [262144, 131072],
contexts: [262144, 131072, 65536],
rank: 6,
// Not measured: the 2.95 of the row above scaled by file size.
// Not measured: the 2.95 of the row below scaled by file size.
speedPer100Gbps: 1.7,
tested: [],
},
{
id: 'qwen3.5:122b-a10b',
label: 'Qwen3.5 122B-A10B',
// 36 to 96 GB on a fast chip: the strongest model of this size. Terminal-Bench 2.1 73.0,
// SWE-bench Pro 61.7 (Qwen's card). Dense, so it is slow on a Pro chip.
id: 'qwen3.8:27b-mtp-q4_k_m',
label: 'Qwen3.8 27B',
license: 'Apache-2.0',
paramsB: 122,
bytes: 81_370_035_877,
...MEASURED_MEMORY,
paramsB: 27.8,
runner: 'llamacpp',
bytes: 17_741_872_154,
// 16 attention layers x 4 K/V heads x 256 x 2 x 34/32 = 34,816, + the prediction layer in
// f16: 4 x 256 x 2 x 2 = 4,096.
heldBytesPerToken: 38_912,
// 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback
// copies of prediction (`draft_num_predict` 4), and in each context checkpoint.
overheadBytes:
156_893_184 * (1 + 4 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0,
figures: 'estimated',
contexts: [131072],
contexts: [262144, 131072, 65536],
rank: 7,
// No speed figure: its speed is unknown on every chip.
// Measured: 11.8 tokens per second on 400 GB/s (llama.cpp, Q4_K_M, without prediction).
// This build predicts several tokens at a time, so it is faster than this figure says.
speedPer100Gbps: 2.95,
tested: [],
},
{
// 128 GB and more. A fine-tune of Qwen3.5-122B-A10B, 10 billion parameters active:
// Terminal-Bench 2.1 81.2, SWE-bench Verified 84.5, SWE-bench Pro 76.9 (Ornith's card).
id: 'ornith-1.5:122b',
label: 'Ornith 1.5 122B-A10B',
license: 'MIT',
paramsB: 122,
// The plain tag is a manifest list; a Mac takes its MLX child (with a renderer and a
// tool parser, which the GGUF builds of this family lack).
runner: 'mlx',
bytes: 73_388_069_552,
// 12 attention layers x 2 K/V heads x 256 x 2 x 2 bytes = 24,576, + a 153,649,152-byte
// recurrent snapshot per 8,192 tokens = 18,756: 43,332, x 37,000 / 28,340.
heldBytesPerToken: 56_574,
overheadBytes: MEASURED_MEMORY.overheadBytes + MLX_PREFIX_CACHE_BYTES,
peakExtraBytes: MEASURED_MEMORY.peakExtraBytes,
figures: 'estimated',
contexts: [262144, 131072],
rank: 8,
// Not measured: the 16.9 of Qwen3.6 35B-A3B on MLX x 3/10, its active parameters
// against these, rounded down.
speedPer100Gbps: 5,
tested: [],
},
],
src/local-model/recommend.ts
+3 -2
View File
@@ -12,8 +12,9 @@
* - `needBytes()`: the PEAK, `bytes + heldBytesPerToken * ctx + overheadBytes +
* peakExtraBytes` of the catalogue row: the weights, what stays held per token of context,
* what the runner holds beside both, and what it takes for a moment while a long prompt is
* processed. Measured for one model on one Mac (see the catalogue header); every other row
* carries the same figures and says `estimated`.
* processed. Measured for one model on one Mac; every other row's figures are computed from
* its own model's architecture for its runner and say `estimated` (see the catalogue
* header, MEMORY).
* - Fit: need over budget up to 0.85 is good, up to 0.98 is tight, above is out. The grade
* decides whether a model is offered at all and what its row says. It does NOT sort: the
* need is the worst moment at the full context, so a tight fit is a model that runs, and a
src/local-model/runtime.ts
+44 -2
View File
@@ -16,8 +16,39 @@
* either case) are copied from the server's; HOME is the private one; and exactly these are
* set: `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
* `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
* `OLLAMA_MAX_LOADED_MODELS=1`. Nothing else gets through: no `CODEMAN_*` and none of the
* user's own `OLLAMA_*`.
* `OLLAMA_MAX_LOADED_MODELS=1`, and the four memory settings of the llama.cpp runner (see
* MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own
* `OLLAMA_*` or `LLAMA_ARG_*`.
* - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both
* together: test/local-model-catalog.test.ts holds them to each other). Read in the
* runtime's v0.40.15 source, not yet measured on a Mac:
* - `OLLAMA_KV_CACHE_TYPE=q8_0` becomes `--cache-type-k q8_0 --cache-type-v q8_0` of
* llama-server (llm/llama_server.go): the K/V cache takes 34 bytes per 32 values, 17/32
* of f16. The runtime's FAQ calls the quality loss "very small". Never q4_0: its loss
* grows with the context, and a coding agent works at long contexts.
* - `OLLAMA_FLASH_ATTENTION=1` becomes `--flash-attn on`. A quantized V cache needs flash
* attention, and left unset the runtime passes `auto`, or `off` where it thinks the GPU
* cannot, which would make every load with a q8_0 cache fail.
* - `LLAMA_ARG_CACHE_RAM=1024` bounds llama-server's host prompt cache (the copies of
* earlier prompts it keeps in memory) to 1024 MiB. The runtime passes no `--cache-ram`
* for a coding model, and llama-server reads the variable from the environment it
* inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a
* session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`.
* - `LLAMA_ARG_CTX_CHECKPOINTS=8` bounds the context checkpoints llama-server keeps per
* slot (`--ctx-checkpoints`, read from the environment the same way). The llama.cpp the
* runtime bundles (b12111, the runtime's LLAMA_CPP_VERSION) makes them for every model
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
* cannot be rolled back (the recurrent state of the linear-attention layers, or the
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by
* default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never
* uses: its next request restores the newest checkpoint of the previous one. 8
* (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a
* checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost:
* a prompt that differs from the last checkpoints further back (a compaction, a cleared
* old tool result) is processed again from the newest checkpoint before that point, or
* from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
* figures do not depend on them.
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
* `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start.
* - SUPERVISOR (`createRuntimeSupervisor()`). The runtime is a NON-detached child of the
@@ -83,6 +114,12 @@ export const RUNTIME_MAX_RESTARTS = 3;
export const RUNTIME_DOWNLOAD_RETRIES = 3;
/** `OLLAMA_KEEP_ALIVE`: how long the model stays loaded after the last request. */
export const RUNTIME_KEEP_ALIVE = '10m';
/** `OLLAMA_KV_CACHE_TYPE`: the K/V cache type of the llama.cpp runner. The catalogue's `llamacpp` rows are computed for it. */
export const RUNTIME_KV_CACHE_TYPE = 'q8_0';
/** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */
export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024;
/** `LLAMA_ARG_CTX_CHECKPOINTS`: the context checkpoints llama-server keeps per slot. Counted in every `llamacpp` row's overhead. */
export const RUNTIME_CTX_CHECKPOINTS = 8;
/** The name of the runtime's binary inside its folder and inside the archive. */
const RUNTIME_BINARY_NAME = 'ollama';
@@ -253,6 +290,11 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv {
env.OLLAMA_KEEP_ALIVE = RUNTIME_KEEP_ALIVE;
env.OLLAMA_NUM_PARALLEL = '1';
env.OLLAMA_MAX_LOADED_MODELS = '1';
// The memory settings of the llama.cpp runner (file header, MEMORY). Inert for an MLX model.
env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE;
env.OLLAMA_FLASH_ATTENTION = '1';
env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB);
env.LLAMA_ARG_CTX_CHECKPOINTS = String(RUNTIME_CTX_CHECKPOINTS);
return env;
}
src/types/local-model.ts
+37 -13
View File
@@ -120,6 +120,9 @@ export interface TestedOn {
tokensPerSecond?: number;
}
/** The runner of the model runtime that runs a model on a Mac, by the registry's name for it. See `LocalModelEntry.runner`. */
export type LocalModelRunner = 'mlx' | 'llamacpp';
/** One model of the catalogue. */
export interface LocalModelEntry {
/** `name:tag` exactly as the model registry lists it. Lowercase, one colon, no slash (`LOCAL_MODEL_ID_PATTERN`). */
@@ -130,35 +133,56 @@ export interface LocalModelEntry {
license: string;
/** Parameter count in billions. When no model that fits has 12 or more, the verdict is `limited`, never `ok`. */
paramsB: number;
/**
* The runner the runtime uses for this tag ON A MAC, as the registry names it: `mlx` (a
* safetensors build, or a manifest list with an MLX child, which a Mac always takes) or
* `llamacpp` (a GGUF build, run by llama-server on Metal). The two spend memory in different
* ways, so a row's memory figures are only meaningful together with its runner: llama.cpp
* allocates the whole K/V cache for the context when the model loads (in the runtime's
* `OLLAMA_KV_CACHE_TYPE`), MLX grows it with the tokens actually used and keeps snapshots of
* it besides.
*/
runner: LocalModelRunner;
/**
* Size of the weights in bytes: both the download and what they take in memory.
*
* It is the size of the build the runtime really downloads ON A MAC. That is not always
* the sum of the registry manifest: `qwen3.6:35b-a3b` is an MLX build of 23,105,798,956 bytes
* on Apple silicon, 2% more than the manifest's 22,621,314,161. A row whose Mac size nobody
* measured keeps the manifest sum and says so in `figures`.
* It is the size of the build the runtime really downloads ON A MAC: the build of `runner`,
* every layer of its manifest plus its config, which is the `total` a real pull reports.
* That is not always the first manifest the registry answers with: `qwen3.6:35b-a3b` is an
* MLX build of 23,105,798,956 bytes on Apple silicon, 2% more than its GGUF build of
* 22,621,314,381.
*/
bytes: number;
/**
* Bytes that stay HELD in memory per token of context once the token was processed (measured
* on the MLX runner: 36,000 to 38,000, so about 37,000). A figure computed from a model's
* config alone came out too low by almost half, so a row that was not measured carries the
* measured row's figure and says so in `figures`.
* Bytes that stay HELD in memory per token of context. For a `llamacpp` row: the K/V cache
* of its attention layers in the runtime's cache type, which llama.cpp allocates for the
* whole context at load (a multi-token-prediction layer's in f16). For an `mlx` row: the K/V cache in bf16 plus the snapshots of the
* recurrent state the runner keeps every 8,192 tokens, which it grows with use. Measured
* on one row (36,000 to 38,000 on the MLX runner, so about 37,000); computed from the
* model's config for the others, see `figures` and the catalogue header.
*/
heldBytesPerToken: number;
/** What the runner holds beside the weights and the context (measured: about 3 GB). */
/**
* What the runner holds beside the weights and the context, whatever the context: its own
* buffers, the live recurrent or sliding-window caches, and the caches it keeps of earlier
* prompts (for `llamacpp` the 1024 MiB host prompt cache `LLAMA_ARG_CACHE_RAM` allows and
* the copies of the recurrent or sliding-window state in its `LLAMA_ARG_CTX_CHECKPOINTS`
* context checkpoints, for an estimated `mlx` row the 8 GiB of prefix snapshots the runner
* may keep). Measured on one row (about 3 GB); see the catalogue header for the others.
*/
overheadBytes: number;
/**
* The TRANSIENT extra the runner takes while it processes a long prompt, on top of
* what stays held (measured: about 5 GB). The fit is judged on the peak:
* what stays held (measured on the MLX runner: about 5 GB; 0 for `llamacpp`, which
* allocates its buffers at load). The fit is judged on the peak:
* `bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`. For
* `qwen3.6:35b-a3b` at 131072 tokens that is about 36 GB, which is tight on a 48 GB Mac.
*/
peakExtraBytes: number;
/**
* Where `bytes` and the three memory figures above come from: `measured` on a Mac by
* us, or `estimated` (the manifest size, and the per-token, overhead and peak figures of the
* measured row applied to this one).
* Where the three memory figures above come from: `measured` on a Mac by us, or
* `estimated` (computed from the model's architecture for its runner, see the catalogue
* header). `bytes` is the registry's size of the build a Mac pulls in both cases.
*/
figures: 'measured' | 'estimated';
/** Context lengths (tokens) the model may be run at, largest first. */