From 83fb523979a9d9aaacb0aabff596d581ce5d0654 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Sun, 8 Nov 2026 10:38:12 +0100 Subject: [PATCH] fix(local-model): rollback copies follow each tag's own draft length The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what `need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08: - qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4). - qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4. - qwen3.8:27b-q8_0: none in its params, so the runtime's 4. The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row instead of a yes/no prediction layer. Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200, 32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at or above n_layer()), and the main context leaves it out, so the row counts that layer once. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- docs/local-models.md | 7 ++-- src/config/local-model-catalog.ts | 33 ++++++++------- src/local-model/runtime.ts | 14 +++---- test/local-model-catalog.test.ts | 60 ++++++++++++++-------------- test/local-model-fits-budget.test.ts | 2 +- test/local-model-recommend.test.ts | 10 ++--- 7 files changed, 64 insertions(+), 64 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d6faacea..b6d6f2c7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -191,7 +191,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph ⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) -**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer 4 rollback copies of the recurrent state and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` +**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` **Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaves a record the state file does not know; a record under the reserved id that says exactly what ours would say is ours (`isSameLocalEndpoint()`). ⚠️ A job whose end could not be written (a full disk) is kept in memory (`unrecorded`): the view shows it, the next verb writes it, and `installing` in the file while this process runs no job always reads as `paused` or that failure, never as a running job. ⚠️ The runtime's pid is stored the moment it is spawned, and Stop, Remove and the boot of anything that is not a running installation end a runtime an earlier server left behind (`supervisor.stop({ previousPid })`, ours by its command line) before any file is deleted; an installed model keeps the runtime VERSION it was installed with while that folder is unpacked, and a start that fails shows as `runtime-start-failed` with Start as the next step. Stop and "Use another port" answer 409 `sessions-open` while a live session uses the model. ⚠️ Remove (409 `sessions-open` with the names while a live session uses the endpoint) undoes the connection FIRST, restores the flag only when this feature switched it on AND no other endpoint is saved, deletes only inside `local-model/` (the models only when asked), and leaves the coding agent. ⚠️ Developer tools: only `/usr/bin/xcode-select -p`, bounded; never `git`/`python3`/`xcrun` from the server (each opens a system window on such a Mac). Tests: `test/local-model-install-job.test.ts` (every state with every event), `-install-service`, `-install-e2e` (real modules over `test/fixtures/fake-ollama.ts`), `-state-store`, `-connect`, `test/cli-install-runner.test.ts`, `test/custom-model-hosts-writer.test.ts`, `test/routes/local-model-routes.test.ts`, `test/routes/local-model-launch-guard.test.ts`. → [architecture-invariants#local-model-install](docs/architecture-invariants.md#local-model-install), `docs/local-models.md`, `docs/api-reference.md` (Local model) diff --git a/docs/local-models.md b/docs/local-models.md index a3c6d068..d3d3e378 100644 --- a/docs/local-models.md +++ b/docs/local-models.md @@ -29,7 +29,7 @@ memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent): | 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative | | 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow | | 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip | -| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.2 GB, 98%). Both tight | +| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight | | 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed | | 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip | | 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) | @@ -62,8 +62,9 @@ names its runner (`runner`) and its figures follow that runner's rule: layers, its recurrent or sliding-window state, live and in each of the 8 checkpoints, 1.0 GB for the runner and the 1 GiB prompt cache. A build with a multi-token-prediction layer (the two `-mtp` tags and `qwen3.8:27b-q8_0`) costs more: the runtime turns prediction on, - so llama-server keeps that layer's cache in f16 and 4 more copies of the recurrent state - for rollback. No peak: the buffers are allocated at load. The checkpoints and the + so llama-server keeps that layer's cache in f16 and one more copy of the recurrent state for + rollback per drafted token (the tag's `draft_num_predict`: 2 for the 35B-A3B, 4 for the + 27B). No peak: the buffers are allocated at load. The checkpoints and the prediction costs were read in the llama.cpp source the runtime bundles (b12111), not measured. - **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent diff --git a/src/config/local-model-catalog.ts b/src/config/local-model-catalog.ts index 9003d5bf..a6ed1668 100644 --- a/src/config/local-model-catalog.ts +++ b/src/config/local-model-catalog.ts @@ -75,13 +75,15 @@ * "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in * its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp` * tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For - * such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max 4` - * (its default `draft_num_predict` is 4). + * such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N + * being the tag's own `draft_num_predict` (its params layer: 4 for + * `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default + * of 4 (`qwen3.8:27b-q8_0` sets none). * - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model * adds its prediction layer, whose cache llama-server keeps in a draft context of the * same length in the draft's own cache type, f16 by default (2 bytes per value). * - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers, - * or the sliding-window cache of `n_swa + 512` cells), held 1 + 4 times for an MTP + * or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP * model (llama.cpp keeps one recurrent state per drafted token for rollback, * `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies * (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent @@ -323,8 +325,8 @@ const LLAMACPP_RUNNER_BYTES = 1_000_000_000; const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024; /** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */ const LLAMACPP_CTX_CHECKPOINTS = 8; -/** The recurrent states llama.cpp keeps beside the live one for an MTP model: the runtime's draft length, 4. */ -const LLAMACPP_MTP_ROLLBACK_STATES = 4; +/** The tokens the runtime drafts per step for an MTP model whose tag sets no `draft_num_predict`, 4: so many rollback states. */ +const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4; /** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */ const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024; @@ -420,12 +422,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { // 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in // f16: 2 x 256 x 2 x 2 = 2,048. heldBytesPerToken: 12_928, - // 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 4 rollback - // copies of prediction, and in each context checkpoint. + // 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 2 rollback + // copies of prediction (this tag drafts 2 tokens, `draft_num_predict` 2), and in each + // context checkpoint. overheadBytes: - 65_863_680 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + - LLAMACPP_RUNNER_BYTES + - LLAMACPP_PROMPT_CACHE_BYTES, + 65_863_680 * (1 + 2 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES, peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], @@ -462,11 +463,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { paramsB: 27.8, runner: 'llamacpp', bytes: 29_978_242_050, - // The architecture of the row below, prediction layer included (its GGUF has one), so its - // figures. + // The architecture of the row below, prediction layer included (its GGUF has one), and the + // runtime's 4 drafted tokens (the tag sets none), so its figures. heldBytesPerToken: 38_912, overheadBytes: - 156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + + 156_893_184 * (1 + LLAMACPP_DEFAULT_DRAFT_TOKENS + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES, peakExtraBytes: 0, @@ -490,11 +491,9 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { // f16: 4 x 256 x 2 x 2 = 4,096. heldBytesPerToken: 38_912, // 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback - // copies of prediction, and in each context checkpoint. + // copies of prediction (`draft_num_predict` 4), and in each context checkpoint. overheadBytes: - 156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) + - LLAMACPP_RUNNER_BYTES + - LLAMACPP_PROMPT_CACHE_BYTES, + 156_893_184 * (1 + 4 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES, peakExtraBytes: 0, figures: 'estimated', contexts: [262144, 131072, 65536], diff --git a/src/local-model/runtime.ts b/src/local-model/runtime.ts index a67ce5b1..eaf281d5 100644 --- a/src/local-model/runtime.ts +++ b/src/local-model/runtime.ts @@ -40,13 +40,13 @@ * of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that * cannot be rolled back (the recurrent state of the linear-attention layers, or the * sliding-window cache), two or three per request, and llama-server keeps up to 32 by - * default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for - * nothing: an agent's next request restores the newest checkpoint of the previous one. - * 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in - * 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from - * the last checkpoints further back (a compaction, a cleared old tool result) is - * processed again from the newest checkpoint before that point, or from its start. - * 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`. + * default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never + * uses: its next request restores the newest checkpoint of the previous one. 8 + * (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a + * checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost: + * a prompt that differs from the last checkpoints further back (a compaction, a cleared + * old tool result) is processed again from the newest checkpoint before that point, or + * from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`. * The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's * figures do not depend on them. * - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across diff --git a/test/local-model-catalog.test.ts b/test/local-model-catalog.test.ts index a3760f68..d3468f3a 100644 --- a/test/local-model-catalog.test.ts +++ b/test/local-model-catalog.test.ts @@ -84,22 +84,26 @@ type ArchitectureName = keyof typeof ARCHITECTURES; type Architecture = (typeof ARCHITECTURES)[ArchitectureName]; /** - * Which architecture each stock row runs, and whether its GGUF carries a - * multi-token-prediction layer (`nextn_predict_layers` 1 in the GGUF header, read on - * 2026-11-08 from the first bytes of each model layer). For such a model the runtime turns on - * prediction, which costs a cache for that layer and rollback copies of the recurrent state - * (see the catalogue header). A row the catalogue gains must be added here, or the test below - * fails: its figures cannot be checked without it. + * Which architecture each stock row runs, and how many tokens the runtime drafts per step with + * its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF + * header, read on 2026-11-08 from the first bytes of each model layer), else the tag's own + * `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none. + * Prediction costs a cache for that layer and one rollback copy of the recurrent state per + * drafted token (see the catalogue header). A row the catalogue gains must be added here, or the + * test below fails: its figures cannot be checked without it. */ -const ROW_ARCHITECTURE: Record = { - 'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false }, - 'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false }, - 'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', predictionLayer: false }, - 'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: true }, - 'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: false }, - 'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', predictionLayer: true }, - 'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', predictionLayer: true }, - 'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', predictionLayer: false }, +const ROW_ARCHITECTURE: Record = { + 'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 }, + 'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 }, + 'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 }, + // Its params layer sets `draft_num_predict` 2. + 'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 }, + 'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 }, + // A prediction layer, and no `draft_num_predict`: the runtime's 4. + 'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', draftTokens: 4 }, + // Its params layer sets `draft_num_predict` 4. + 'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', draftTokens: 4 }, + 'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', draftTokens: 0 }, }; /** Bytes per cached value in each K/V cache type of llama.cpp: q8_0 keeps 32 values in 34 bytes. */ @@ -109,12 +113,6 @@ const KV_BYTES_PER_VALUE: Record = { f16: 2, q8_0: 34 / 32 }; * default, because the runtime sets only the main context's (`--cache-type-k`). */ const DRAFT_KV_CACHE_TYPE = 'f16'; -/** - * The recurrent states llama.cpp keeps beside the live one for a model with prediction: the - * runtime's draft length (`draft_num_predict` 4, passed as `--spec-draft-n-max`), which - * `need_n_rs_seq()` turns into that many rollback copies. - */ -const MTP_ROLLBACK_STATES = 4; /** The sliding-window cache holds the window plus one batch of 512 tokens (`n_swa + n_ubatch`). */ const SLIDING_EXTRA_CELLS = 512; /** The head size of a Gated DeltaNet layer's keys and values, and its convolution kernel, in every model above. */ @@ -132,7 +130,8 @@ function recurrentStateBytes(arch: Architecture, convBytes: number): number { } /** What a `llamacpp` row's three figures must be, from its architecture and the runtime's settings. */ -function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number, number, number] { +function llamaCppFigures(arch: Architecture, draftTokens: number): [number, number, number] { + const predictionLayer = draftTokens > 0; const perValue = KV_BYTES_PER_VALUE[RUNTIME_KV_CACHE_TYPE]; const perLayer = arch.kvHeads * arch.headDim * 2; // The prediction layer's cache sits in the draft context, in the draft's cache type. @@ -143,8 +142,9 @@ function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number, const recurrent = recurrentStateBytes(arch, 4); const sliding = 'sliding' in arch ? arch.sliding.layers * (arch.sliding.window + SLIDING_EXTRA_CELLS) * perLayer * perValue : 0; - // Live: the sliding window, and the recurrent state with its rollback copies under prediction. - const live = sliding + recurrent * (1 + (predictionLayer ? MTP_ROLLBACK_STATES : 0)); + // Live: the sliding window, and the recurrent state with one rollback copy per drafted token + // (`--spec-draft-n-max`, which `need_n_rs_seq()` turns into that many copies). + const live = sliding + recurrent * (1 + draftTokens); // A context checkpoint is a copy of what cannot be rolled back: the recurrent state or the window. const checkpoints = RUNTIME_CTX_CHECKPOINTS * (recurrent + sliding); return [held, live + checkpoints + 1_000_000_000 + RUNTIME_HOST_PROMPT_CACHE_MIB * MIB, 0]; @@ -337,9 +337,9 @@ describe('the memory figures of the stock rows: each row has its own', () => { it("recomputes every estimated row's figures from its own model's architecture and runner", () => { for (const model of models.filter((m) => m.figures === 'estimated')) { - const { architecture, predictionLayer } = ROW_ARCHITECTURE[model.id]; + const { architecture, draftTokens } = ROW_ARCHITECTURE[model.id]; const arch = ARCHITECTURES[architecture]; - const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, predictionLayer) : mlxFigures(arch); + const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, draftTokens) : mlxFigures(arch); expect(figuresOf(model), `${model.id} (${architecture}, ${model.runner})`).toEqual(expected); } }); @@ -351,9 +351,9 @@ describe('the memory figures of the stock rows: each row has its own', () => { 'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0], // The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB. 'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0], - // State x (1 live + 4 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction - // layer's cache in f16 per token. - 'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_929_969_664, 0], + // State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction + // layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies. + 'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0], 'qwen3.6:35b-a3b': [37_000, 3_000_000_000, 5_000_000_000], 'qwen3.8:27b-q8_0': [38_912, 4_113_353_216, 0], 'qwen3.8:27b-mtp-q4_k_m': [38_912, 4_113_353_216, 0], @@ -367,7 +367,7 @@ describe('the memory figures of the stock rows: each row has its own', () => { expect(mlxComputedPerToken(ARCHITECTURES['Qwen3.5-122B-A10B'])).toBe(43_332); }); - it('lets two rows share figures only when they share the architecture, the prediction layer and the runner', () => { + it('lets two rows share figures only when they share the architecture, the drafted tokens and the runner', () => { const byFigures = new Map(); for (const model of models) { const key = figuresOf(model).join('/'); diff --git a/test/local-model-fits-budget.test.ts b/test/local-model-fits-budget.test.ts index 3b112150..63a1c873 100644 --- a/test/local-model-fits-budget.test.ts +++ b/test/local-model-fits-budget.test.ts @@ -97,7 +97,7 @@ describe('modelFitsBudget', () => { // A runtime that reports less than the estimate rules out what the estimate allowed. expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 33 * 1024 ** 3 })).toBe(false); // And one that reports more admits what the estimate ruled out: on a 36 GB Mac the GGUF - // 35B-A3B at its full window needs 28.9 GB, 1.037 of the 27.9 GB estimate and 0.963 of 28 GiB. + // 35B-A3B at its full window needs 28.8 GB, 1.032 of the 27.9 GB estimate and 0.958 of 28 GiB. const moeGguf = 'qwen3.6:35b-a3b-mtp-q4_k_m'; expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144)).toBe(false); expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144, { measuredTotalBytes: 28 * 1024 ** 3 })).toBe(true); diff --git a/test/local-model-recommend.test.ts b/test/local-model-recommend.test.ts index 59139d03..8114bbc4 100644 --- a/test/local-model-recommend.test.ts +++ b/test/local-model-recommend.test.ts @@ -489,7 +489,7 @@ describe('needBytes', () => { 'qwen3.5:9b-q4_k_m': 11_380_487_246, 'mimo-v2.6:9b-q4_k_m': 11_380_472_729, 'north-mini-code-1.0:q4_k_m': 25_999_848_864, - 'qwen3.6:35b-a3b-mtp-q4_k_m': 28_940_281_677, + 'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317, 'qwen3.6:35b-a3b': 35_955_462_956, 'qwen3.8:27b-q8_0': 44_292_142_594, 'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698, @@ -647,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => { expect( result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind]) ).toEqual([ - // The same model as a GGUF build: its fixed footprint holds the full window, 28.9 GB. + // The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB. ['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined], ['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'], @@ -809,7 +809,7 @@ describe('every memory size, with the stock catalogue', () => { [32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'], // A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9), // and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both - // at 131072 tokens, 0.966 and 0.976 of the budget. + // at 131072 tokens, 0.966 and 0.971 of the budget. [36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'], [36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'], // The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick. @@ -2077,8 +2077,8 @@ describe('the measured budget, end to end', () => { expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']); // The same rows in the same order; only the share of each changes. expect(order(measured)).toEqual(order(estimated)); - expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.77, 0.692, 0.303, 0.303]); - expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.721, 0.647, 0.283, 0.283]); + expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]); + expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]); for (const candidate of rows(measured)) { expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3); }