fix(local-model): rollback copies follow each tag's own draft length

The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a
prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its
params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what
`need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08:

- qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4).
- qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4.
- qwen3.8:27b-q8_0: none in its params, so the runtime's 4.

The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or
M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row
instead of a yes/no prediction layer.

Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200,
32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those
tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context
of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at
or above n_layer()), and the main context leaves it out, so the row counts that layer once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-11-08 10:38:12 +01:00
1 parent 2cd3cbd6a0
commit 83fb523979
7 files changed
+64 -64

No files matched your search

+1 -1
View File
@@ -191,7 +191,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) ⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer 4 rollback copies of the recurrent state and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` **Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated **Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated
+4 -3
View File
@@ -29,7 +29,7 @@ memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative | | 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow | | 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip | | 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.2 GB, 98%). Both tight | | 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight |
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed | | 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip | | 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) | | 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
@@ -62,8 +62,9 @@ names its runner (`runner`) and its figures follow that runner's rule:
layers, its recurrent or sliding-window state, live and in each of the 8 checkpoints, 1.0 layers, its recurrent or sliding-window state, live and in each of the 8 checkpoints, 1.0
GB for the runner and the 1 GiB prompt cache. A build with a multi-token-prediction layer GB for the runner and the 1 GiB prompt cache. A build with a multi-token-prediction layer
(the two `-mtp` tags and `qwen3.8:27b-q8_0`) costs more: the runtime turns prediction on, (the two `-mtp` tags and `qwen3.8:27b-q8_0`) costs more: the runtime turns prediction on,
so llama-server keeps that layer's cache in f16 and 4 more copies of the recurrent state so llama-server keeps that layer's cache in f16 and one more copy of the recurrent state for
for rollback. No peak: the buffers are allocated at load. The checkpoints and the rollback per drafted token (the tag's `draft_num_predict`: 2 for the 35B-A3B, 4 for the
27B). No peak: the buffers are allocated at load. The checkpoints and the
prediction costs were read in the llama.cpp source the runtime bundles (b12111), not prediction costs were read in the llama.cpp source the runtime bundles (b12111), not
measured. measured.
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent - **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
+16 -17
View File
@@ -75,13 +75,15 @@
* "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in * "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
* its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp` * its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For * tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max 4` * such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N
* (its default `draft_num_predict` is 4). * being the tag's own `draft_num_predict` (its params layer: 4 for
* `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default
* of 4 (`qwen3.8:27b-q8_0` sets none).
* - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model * - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model
* adds its prediction layer, whose cache llama-server keeps in a draft context of the * adds its prediction layer, whose cache llama-server keeps in a draft context of the
* same length in the draft's own cache type, f16 by default (2 bytes per value). * same length in the draft's own cache type, f16 by default (2 bytes per value).
* - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers, * - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
* or the sliding-window cache of `n_swa + 512` cells), held 1 + 4 times for an MTP * or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP
* model (llama.cpp keeps one recurrent state per drafted token for rollback, * model (llama.cpp keeps one recurrent state per drafted token for rollback,
* `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies * `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
* (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent * (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
@@ -323,8 +325,8 @@ const LLAMACPP_RUNNER_BYTES = 1_000_000_000;
const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024; const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024;
/** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */ /** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */
const LLAMACPP_CTX_CHECKPOINTS = 8; const LLAMACPP_CTX_CHECKPOINTS = 8;
/** The recurrent states llama.cpp keeps beside the live one for an MTP model: the runtime's draft length, 4. */ /** The tokens the runtime drafts per step for an MTP model whose tag sets no `draft_num_predict`, 4: so many rollback states. */
const LLAMACPP_MTP_ROLLBACK_STATES = 4; const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4;
/** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */ /** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024; const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
@@ -420,12 +422,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
// 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in // 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in
// f16: 2 x 256 x 2 x 2 = 2,048. // f16: 2 x 256 x 2 x 2 = 2,048.
heldBytesPerToken: 12_928, heldBytesPerToken: 12_928,
// 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 4 rollback // 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 2 rollback
// copies of prediction, and in each context checkpoint. // copies of prediction (this tag drafts 2 tokens, `draft_num_predict` 2), and in each
// context checkpoint.
overheadBytes: overheadBytes:
65_863_680 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + 65_863_680 * (1 + 2 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
LLAMACPP_RUNNER_BYTES +
LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0, peakExtraBytes: 0,
figures: 'estimated', figures: 'estimated',
contexts: [262144, 131072, 65536], contexts: [262144, 131072, 65536],
@@ -462,11 +463,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
paramsB: 27.8, paramsB: 27.8,
runner: 'llamacpp', runner: 'llamacpp',
bytes: 29_978_242_050, bytes: 29_978_242_050,
// The architecture of the row below, prediction layer included (its GGUF has one), so its // The architecture of the row below, prediction layer included (its GGUF has one), and the
// figures. // runtime's 4 drafted tokens (the tag sets none), so its figures.
heldBytesPerToken: 38_912, heldBytesPerToken: 38_912,
overheadBytes: overheadBytes:
156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + 156_893_184 * (1 + LLAMACPP_DEFAULT_DRAFT_TOKENS + LLAMACPP_CTX_CHECKPOINTS) +
LLAMACPP_RUNNER_BYTES + LLAMACPP_RUNNER_BYTES +
LLAMACPP_PROMPT_CACHE_BYTES, LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0, peakExtraBytes: 0,
@@ -490,11 +491,9 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
// f16: 4 x 256 x 2 x 2 = 4,096. // f16: 4 x 256 x 2 x 2 = 4,096.
heldBytesPerToken: 38_912, heldBytesPerToken: 38_912,
// 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback // 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback
// copies of prediction, and in each context checkpoint. // copies of prediction (`draft_num_predict` 4), and in each context checkpoint.
overheadBytes: overheadBytes:
156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + 156_893_184 * (1 + 4 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
LLAMACPP_RUNNER_BYTES +
LLAMACPP_PROMPT_CACHE_BYTES,
peakExtraBytes: 0, peakExtraBytes: 0,
figures: 'estimated', figures: 'estimated',
contexts: [262144, 131072, 65536], contexts: [262144, 131072, 65536],
+7 -7
View File
@@ -40,13 +40,13 @@
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that * of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
* cannot be rolled back (the recurrent state of the linear-attention layers, or the * cannot be rolled back (the recurrent state of the linear-attention layers, or the
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by * sliding-window cache), two or three per request, and llama-server keeps up to 32 by
* default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for * default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never
* nothing: an agent's next request restores the newest checkpoint of the previous one. * uses: its next request restores the newest checkpoint of the previous one. 8
* 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in * (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a
* 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from * checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost:
* the last checkpoints further back (a compaction, a cleared old tool result) is * a prompt that differs from the last checkpoints further back (a compaction, a cleared
* processed again from the newest checkpoint before that point, or from its start. * old tool result) is processed again from the newest checkpoint before that point, or
* 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`. * from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's * The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
* figures do not depend on them. * figures do not depend on them.
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across * - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
+30 -30
View File
@@ -84,22 +84,26 @@ type ArchitectureName = keyof typeof ARCHITECTURES;
type Architecture = (typeof ARCHITECTURES)[ArchitectureName]; type Architecture = (typeof ARCHITECTURES)[ArchitectureName];
/** /**
* Which architecture each stock row runs, and whether its GGUF carries a * Which architecture each stock row runs, and how many tokens the runtime drafts per step with
* multi-token-prediction layer (`nextn_predict_layers` 1 in the GGUF header, read on * its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF
* 2026-11-08 from the first bytes of each model layer). For such a model the runtime turns on * header, read on 2026-11-08 from the first bytes of each model layer), else the tag's own
* prediction, which costs a cache for that layer and rollback copies of the recurrent state * `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none.
* (see the catalogue header). A row the catalogue gains must be added here, or the test below * Prediction costs a cache for that layer and one rollback copy of the recurrent state per
* fails: its figures cannot be checked without it. * drafted token (see the catalogue header). A row the catalogue gains must be added here, or the
* test below fails: its figures cannot be checked without it.
*/ */
const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; predictionLayer: boolean }> = { const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; draftTokens: number }> = {
'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false }, 'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false }, 'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', predictionLayer: false }, 'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 },
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: true }, // Its params layer sets `draft_num_predict` 2.
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: false }, 'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 },
'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', predictionLayer: true }, 'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 },
'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', predictionLayer: true }, // A prediction layer, and no `draft_num_predict`: the runtime's 4.
'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', predictionLayer: false }, 'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', draftTokens: 4 },
// Its params layer sets `draft_num_predict` 4.
'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', draftTokens: 4 },
'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', draftTokens: 0 },
}; };
/** Bytes per cached value in each K/V cache type of llama.cpp: q8_0 keeps 32 values in 34 bytes. */ /** Bytes per cached value in each K/V cache type of llama.cpp: q8_0 keeps 32 values in 34 bytes. */
@@ -109,12 +113,6 @@ const KV_BYTES_PER_VALUE: Record<string, number> = { f16: 2, q8_0: 34 / 32 };
* default, because the runtime sets only the main context's (`--cache-type-k`). * default, because the runtime sets only the main context's (`--cache-type-k`).
*/ */
const DRAFT_KV_CACHE_TYPE = 'f16'; const DRAFT_KV_CACHE_TYPE = 'f16';
/**
* The recurrent states llama.cpp keeps beside the live one for a model with prediction: the
* runtime's draft length (`draft_num_predict` 4, passed as `--spec-draft-n-max`), which
* `need_n_rs_seq()` turns into that many rollback copies.
*/
const MTP_ROLLBACK_STATES = 4;
/** The sliding-window cache holds the window plus one batch of 512 tokens (`n_swa + n_ubatch`). */ /** The sliding-window cache holds the window plus one batch of 512 tokens (`n_swa + n_ubatch`). */
const SLIDING_EXTRA_CELLS = 512; const SLIDING_EXTRA_CELLS = 512;
/** The head size of a Gated DeltaNet layer's keys and values, and its convolution kernel, in every model above. */ /** The head size of a Gated DeltaNet layer's keys and values, and its convolution kernel, in every model above. */
@@ -132,7 +130,8 @@ function recurrentStateBytes(arch: Architecture, convBytes: number): number {
} }
/** What a `llamacpp` row's three figures must be, from its architecture and the runtime's settings. */ /** What a `llamacpp` row's three figures must be, from its architecture and the runtime's settings. */
function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number, number, number] { function llamaCppFigures(arch: Architecture, draftTokens: number): [number, number, number] {
const predictionLayer = draftTokens > 0;
const perValue = KV_BYTES_PER_VALUE[RUNTIME_KV_CACHE_TYPE]; const perValue = KV_BYTES_PER_VALUE[RUNTIME_KV_CACHE_TYPE];
const perLayer = arch.kvHeads * arch.headDim * 2; const perLayer = arch.kvHeads * arch.headDim * 2;
// The prediction layer's cache sits in the draft context, in the draft's cache type. // The prediction layer's cache sits in the draft context, in the draft's cache type.
@@ -143,8 +142,9 @@ function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number,
const recurrent = recurrentStateBytes(arch, 4); const recurrent = recurrentStateBytes(arch, 4);
const sliding = const sliding =
'sliding' in arch ? arch.sliding.layers * (arch.sliding.window + SLIDING_EXTRA_CELLS) * perLayer * perValue : 0; 'sliding' in arch ? arch.sliding.layers * (arch.sliding.window + SLIDING_EXTRA_CELLS) * perLayer * perValue : 0;
// Live: the sliding window, and the recurrent state with its rollback copies under prediction. // Live: the sliding window, and the recurrent state with one rollback copy per drafted token
const live = sliding + recurrent * (1 + (predictionLayer ? MTP_ROLLBACK_STATES : 0)); // (`--spec-draft-n-max`, which `need_n_rs_seq()` turns into that many copies).
const live = sliding + recurrent * (1 + draftTokens);
// A context checkpoint is a copy of what cannot be rolled back: the recurrent state or the window. // A context checkpoint is a copy of what cannot be rolled back: the recurrent state or the window.
const checkpoints = RUNTIME_CTX_CHECKPOINTS * (recurrent + sliding); const checkpoints = RUNTIME_CTX_CHECKPOINTS * (recurrent + sliding);
return [held, live + checkpoints + 1_000_000_000 + RUNTIME_HOST_PROMPT_CACHE_MIB * MIB, 0]; return [held, live + checkpoints + 1_000_000_000 + RUNTIME_HOST_PROMPT_CACHE_MIB * MIB, 0];
@@ -337,9 +337,9 @@ describe('the memory figures of the stock rows: each row has its own', () => {
it("recomputes every estimated row's figures from its own model's architecture and runner", () => { it("recomputes every estimated row's figures from its own model's architecture and runner", () => {
for (const model of models.filter((m) => m.figures === 'estimated')) { for (const model of models.filter((m) => m.figures === 'estimated')) {
const { architecture, predictionLayer } = ROW_ARCHITECTURE[model.id]; const { architecture, draftTokens } = ROW_ARCHITECTURE[model.id];
const arch = ARCHITECTURES[architecture]; const arch = ARCHITECTURES[architecture];
const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, predictionLayer) : mlxFigures(arch); const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, draftTokens) : mlxFigures(arch);
expect(figuresOf(model), `${model.id} (${architecture}, ${model.runner})`).toEqual(expected); expect(figuresOf(model), `${model.id} (${architecture}, ${model.runner})`).toEqual(expected);
} }
}); });
@@ -351,9 +351,9 @@ describe('the memory figures of the stock rows: each row has its own', () => {
'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0], 'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0],
// The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB. // The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB.
'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0], 'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0],
// State x (1 live + 4 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction // State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
// layer's cache in f16 per token. // layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies.
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_929_969_664, 0], 'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0],
'qwen3.6:35b-a3b': [37_000, 3_000_000_000, 5_000_000_000], 'qwen3.6:35b-a3b': [37_000, 3_000_000_000, 5_000_000_000],
'qwen3.8:27b-q8_0': [38_912, 4_113_353_216, 0], 'qwen3.8:27b-q8_0': [38_912, 4_113_353_216, 0],
'qwen3.8:27b-mtp-q4_k_m': [38_912, 4_113_353_216, 0], 'qwen3.8:27b-mtp-q4_k_m': [38_912, 4_113_353_216, 0],
@@ -367,7 +367,7 @@ describe('the memory figures of the stock rows: each row has its own', () => {
expect(mlxComputedPerToken(ARCHITECTURES['Qwen3.5-122B-A10B'])).toBe(43_332); expect(mlxComputedPerToken(ARCHITECTURES['Qwen3.5-122B-A10B'])).toBe(43_332);
}); });
it('lets two rows share figures only when they share the architecture, the prediction layer and the runner', () => { it('lets two rows share figures only when they share the architecture, the drafted tokens and the runner', () => {
const byFigures = new Map<string, LocalModelEntry[]>(); const byFigures = new Map<string, LocalModelEntry[]>();
for (const model of models) { for (const model of models) {
const key = figuresOf(model).join('/'); const key = figuresOf(model).join('/');
+1 -1
View File
@@ -97,7 +97,7 @@ describe('modelFitsBudget', () => {
// A runtime that reports less than the estimate rules out what the estimate allowed. // A runtime that reports less than the estimate rules out what the estimate allowed.
expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 33 * 1024 ** 3 })).toBe(false); expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 33 * 1024 ** 3 })).toBe(false);
// And one that reports more admits what the estimate ruled out: on a 36 GB Mac the GGUF // And one that reports more admits what the estimate ruled out: on a 36 GB Mac the GGUF
// 35B-A3B at its full window needs 28.9 GB, 1.037 of the 27.9 GB estimate and 0.963 of 28 GiB. // 35B-A3B at its full window needs 28.8 GB, 1.032 of the 27.9 GB estimate and 0.958 of 28 GiB.
const moeGguf = 'qwen3.6:35b-a3b-mtp-q4_k_m'; const moeGguf = 'qwen3.6:35b-a3b-mtp-q4_k_m';
expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144)).toBe(false); expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144)).toBe(false);
expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144, { measuredTotalBytes: 28 * 1024 ** 3 })).toBe(true); expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144, { measuredTotalBytes: 28 * 1024 ** 3 })).toBe(true);
+5 -5
View File
@@ -489,7 +489,7 @@ describe('needBytes', () => {
'qwen3.5:9b-q4_k_m': 11_380_487_246, 'qwen3.5:9b-q4_k_m': 11_380_487_246,
'mimo-v2.6:9b-q4_k_m': 11_380_472_729, 'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
'north-mini-code-1.0:q4_k_m': 25_999_848_864, 'north-mini-code-1.0:q4_k_m': 25_999_848_864,
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_940_281_677, 'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
'qwen3.6:35b-a3b': 35_955_462_956, 'qwen3.6:35b-a3b': 35_955_462_956,
'qwen3.8:27b-q8_0': 44_292_142_594, 'qwen3.8:27b-q8_0': 44_292_142_594,
'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698, 'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698,
@@ -647,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
expect( expect(
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind]) result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
).toEqual([ ).toEqual([
// The same model as a GGUF build: its fixed footprint holds the full window, 28.9 GB. // The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], ['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
@@ -809,7 +809,7 @@ describe('every memory size, with the stock catalogue', () => {
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'], [32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9), // A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both // and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
// at 131072 tokens, 0.966 and 0.976 of the budget. // at 131072 tokens, 0.966 and 0.971 of the budget.
[36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'], [36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'],
[36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'], [36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'],
// The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick. // The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick.
@@ -2077,8 +2077,8 @@ describe('the measured budget, end to end', () => {
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']); expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
// The same rows in the same order; only the share of each changes. // The same rows in the same order; only the share of each changes.
expect(order(measured)).toEqual(order(estimated)); expect(order(measured)).toEqual(order(estimated));
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.77, 0.692, 0.303, 0.303]); expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.721, 0.647, 0.283, 0.283]); expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
for (const candidate of rows(measured)) { for (const candidate of rows(measured)) {
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3); expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
} }