mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 10:39:32 +01:00
fix(local-model): rollback copies follow each tag's own draft length
The previous commit counted 4 rollback copies of the recurrent state for every GGUF with a prediction layer, the runtime's default `draft_num_predict`. A tag can set its own in its params layer, and the runtime then passes that as `--spec-draft-n-max`, which is what `need_n_rs_seq()` turns into rollback copies. Read on 2026-11-08: - qwen3.6:35b-a3b-mtp-q4_k_m: draft_num_predict 2, so 2 copies (was counted as 4). - qwen3.8:27b-mtp-q4_k_m: draft_num_predict 4. - qwen3.8:27b-q8_0: none in its params, so the runtime's 4. The 35B-A3B GGUF row's overhead drops by 132 MB: 0.971 instead of 0.976 on a 36 GB M3 Pro or M3 Max, the same pick. test/local-model-catalog.test.ts now carries the drafted tokens per row instead of a yes/no prediction layer. Also states what was verified about llama.cpp's checkpoint default: 8 from b6800 to b8200, 32 from b8250 together with a checkpoint every 8,192 prompt tokens (common/common.h at those tags), in place of an inferred reason. Checked in the b12111 source as well: the draft context of a Qwen-family MTP model caches the prediction block alone (`mtp_on_hybrid_qwen`, layers at or above n_layer()), and the main context leaves it out, so the row counts that layer once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed
1 parent
2cd3cbd6a0
commit
83fb523979
7 files changed
+64
-64
No files matched your search
@@ -191,7 +191,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
|
||||
|
||||
⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles)
|
||||
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer 4 rollback copies of the recurrent state and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0`, `LLAMA_ARG_CACHE_RAM=1024` and `LLAMA_ARG_CTX_CHECKPOINTS=8` (`runtimeEnv()`) and count what llama-server keeps beside the cache (8 context checkpoints of the recurrent or sliding-window state, and for a GGUF with a prediction layer one rollback copy of the recurrent state per drafted token and that layer's cache in f16), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md`
|
||||
|
||||
**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-<version>/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaveLine truncated
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@ memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent):
|
||||
| 18 GB | 11.8 GB | A small model (`limited`): MiMo V2.6 9B at 131,072 tokens, 11.4 GB, tight. Its base, Qwen3.5 9B, is the one alternative |
|
||||
| 24 GB | 16.1 GB | The same, with room. On a base M2 or M3 (100 GB/s) it computes to 9.3 tokens per second: slow |
|
||||
| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 23.2 GB at 65,536 tokens; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it at 131,072 tokens, or Qwen3.8 27B at 65,536 tokens on a Max chip |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.2 GB, 98%). Both tight |
|
||||
| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max (27.0 GB, 97%); on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build (27.1 GB, 97%). Both tight |
|
||||
| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens (32.1 GB, tight). The test was a short task: in a long session MLX may keep up to 8 GiB more, which brings this pair to the edge of the budget, and past it while a long prompt is processed |
|
||||
| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip |
|
||||
| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) |
|
||||
@@ -62,8 +62,9 @@ names its runner (`runner`) and its figures follow that runner's rule:
|
||||
layers, its recurrent or sliding-window state, live and in each of the 8 checkpoints, 1.0
|
||||
GB for the runner and the 1 GiB prompt cache. A build with a multi-token-prediction layer
|
||||
(the two `-mtp` tags and `qwen3.8:27b-q8_0`) costs more: the runtime turns prediction on,
|
||||
so llama-server keeps that layer's cache in f16 and 4 more copies of the recurrent state
|
||||
for rollback. No peak: the buffers are allocated at load. The checkpoints and the
|
||||
so llama-server keeps that layer's cache in f16 and one more copy of the recurrent state for
|
||||
rollback per drafted token (the tag's `draft_num_predict`: 2 for the 35B-A3B, 4 for the
|
||||
27B). No peak: the buffers are allocated at load. The checkpoints and the
|
||||
prediction costs were read in the llama.cpp source the runtime bundles (b12111), not
|
||||
measured.
|
||||
- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent
|
||||
|
||||
@@ -75,13 +75,15 @@
|
||||
* "MTP" below is a GGUF with a multi-token-prediction layer (`nextn_predict_layers` 1 in
|
||||
* its header, read on 2026-11-08 from the first bytes of each model layer: the two `-mtp`
|
||||
* tags and `qwen3.8:27b-q8_0` have it, the two 9B tags and North Mini Code do not). For
|
||||
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max 4`
|
||||
* (its default `draft_num_predict` is 4).
|
||||
* such a model the runtime always passes `--spec-type draft-mtp --spec-draft-n-max N`, N
|
||||
* being the tag's own `draft_num_predict` (its params layer: 4 for
|
||||
* `qwen3.8:27b-mtp-q4_k_m`, 2 for `qwen3.6:35b-a3b-mtp-q4_k_m`) or the runtime's default
|
||||
* of 4 (`qwen3.8:27b-q8_0` sets none).
|
||||
* - held = attention layers x K/V heads x head dim x 2 (K and V) x 34/32. An MTP model
|
||||
* adds its prediction layer, whose cache llama-server keeps in a draft context of the
|
||||
* same length in the draft's own cache type, f16 by default (2 bytes per value).
|
||||
* - overhead = the fixed caches (the f32 recurrent state of the linear-attention layers,
|
||||
* or the sliding-window cache of `n_swa + 512` cells), held 1 + 4 times for an MTP
|
||||
* or the sliding-window cache of `n_swa + 512` cells), held 1 + N times for an MTP
|
||||
* model (llama.cpp keeps one recurrent state per drafted token for rollback,
|
||||
* `need_n_rs_seq()`), + the context checkpoints: `LLAMA_ARG_CTX_CHECKPOINTS` host copies
|
||||
* (`RUNTIME_CTX_CHECKPOINTS`, 8) of the state that cannot be rolled back (the recurrent
|
||||
@@ -323,8 +325,8 @@ const LLAMACPP_RUNNER_BYTES = 1_000_000_000;
|
||||
const LLAMACPP_PROMPT_CACHE_BYTES = 1024 * 1024 * 1024;
|
||||
/** The context checkpoints llama-server may keep: `LLAMA_ARG_CTX_CHECKPOINTS`, 8 (`RUNTIME_CTX_CHECKPOINTS`). */
|
||||
const LLAMACPP_CTX_CHECKPOINTS = 8;
|
||||
/** The recurrent states llama.cpp keeps beside the live one for an MTP model: the runtime's draft length, 4. */
|
||||
const LLAMACPP_MTP_ROLLBACK_STATES = 4;
|
||||
/** The tokens the runtime drafts per step for an MTP model whose tag sets no `draft_num_predict`, 4: so many rollback states. */
|
||||
const LLAMACPP_DEFAULT_DRAFT_TOKENS = 4;
|
||||
/** The prefix snapshots the MLX runner may keep: `maxPagedOutBytes`, 8 GiB. */
|
||||
const MLX_PREFIX_CACHE_BYTES = 8 * 1024 * 1024 * 1024;
|
||||
|
||||
@@ -420,12 +422,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
// 10 attention layers x 2 K/V heads x 256 x 2 x 34/32 = 10,880, + the prediction layer in
|
||||
// f16: 2 x 256 x 2 x 2 = 2,048.
|
||||
heldBytesPerToken: 12_928,
|
||||
// 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 4 rollback
|
||||
// copies of prediction, and in each context checkpoint.
|
||||
// 30 linear-attention layers hold 65,863,680 bytes of f32 state: live, in the 2 rollback
|
||||
// copies of prediction (this tag drafts 2 tokens, `draft_num_predict` 2), and in each
|
||||
// context checkpoint.
|
||||
overheadBytes:
|
||||
65_863_680 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
|
||||
LLAMACPP_RUNNER_BYTES +
|
||||
LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
65_863_680 * (1 + 2 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
@@ -462,11 +463,11 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
paramsB: 27.8,
|
||||
runner: 'llamacpp',
|
||||
bytes: 29_978_242_050,
|
||||
// The architecture of the row below, prediction layer included (its GGUF has one), so its
|
||||
// figures.
|
||||
// The architecture of the row below, prediction layer included (its GGUF has one), and the
|
||||
// runtime's 4 drafted tokens (the tag sets none), so its figures.
|
||||
heldBytesPerToken: 38_912,
|
||||
overheadBytes:
|
||||
156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
|
||||
156_893_184 * (1 + LLAMACPP_DEFAULT_DRAFT_TOKENS + LLAMACPP_CTX_CHECKPOINTS) +
|
||||
LLAMACPP_RUNNER_BYTES +
|
||||
LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
@@ -490,11 +491,9 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = {
|
||||
// f16: 4 x 256 x 2 x 2 = 4,096.
|
||||
heldBytesPerToken: 38_912,
|
||||
// 48 linear-attention layers hold 156,893,184 bytes of f32 state: live, in the 4 rollback
|
||||
// copies of prediction, and in each context checkpoint.
|
||||
// copies of prediction (`draft_num_predict` 4), and in each context checkpoint.
|
||||
overheadBytes:
|
||||
156_893_184 * (1 + LLAMACPP_MTP_ROLLBACK_STATES + LLAMACPP_CTX_CHECKPOINTS) +
|
||||
LLAMACPP_RUNNER_BYTES +
|
||||
LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
156_893_184 * (1 + 4 + LLAMACPP_CTX_CHECKPOINTS) + LLAMACPP_RUNNER_BYTES + LLAMACPP_PROMPT_CACHE_BYTES,
|
||||
peakExtraBytes: 0,
|
||||
figures: 'estimated',
|
||||
contexts: [262144, 131072, 65536],
|
||||
|
||||
@@ -40,13 +40,13 @@
|
||||
* of the catalogue's `llamacpp` rows: a checkpoint is a host copy of the state that
|
||||
* cannot be rolled back (the recurrent state of the linear-attention layers, or the
|
||||
* sliding-window cache), two or three per request, and llama-server keeps up to 32 by
|
||||
* default. 32 copies of the 27B's 157 MB state are 5 GB that the session holds for
|
||||
* nothing: an agent's next request restores the newest checkpoint of the previous one.
|
||||
* 8 (llama.cpp's own default before it went to 32 to cover a 262,144-token window in
|
||||
* 8,192-token steps) keeps the last few requests. The cost: a prompt that differs from
|
||||
* the last checkpoints further back (a compaction, a cleared old tool result) is
|
||||
* processed again from the newest checkpoint before that point, or from its start.
|
||||
* 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
|
||||
* default. 32 copies of the 27B's 157 MB state are 5 GB, most of which an agent never
|
||||
* uses: its next request restores the newest checkpoint of the previous one. 8
|
||||
* (llama.cpp's own default from b6800 to b8200; it became 32 at b8250, together with a
|
||||
* checkpoint every 8,192 tokens of a long prompt) keeps the last few requests. The cost:
|
||||
* a prompt that differs from the last checkpoints further back (a compaction, a cleared
|
||||
* old tool result) is processed again from the newest checkpoint before that point, or
|
||||
* from its start. 8 checkpoints are counted in every `llamacpp` row's `overheadBytes`.
|
||||
* The MLX runner reads none of the four (nothing under mlxrunner/ does): an `mlx` row's
|
||||
* figures do not depend on them.
|
||||
* - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
|
||||
|
||||
@@ -84,22 +84,26 @@ type ArchitectureName = keyof typeof ARCHITECTURES;
|
||||
type Architecture = (typeof ARCHITECTURES)[ArchitectureName];
|
||||
|
||||
/**
|
||||
* Which architecture each stock row runs, and whether its GGUF carries a
|
||||
* multi-token-prediction layer (`nextn_predict_layers` 1 in the GGUF header, read on
|
||||
* 2026-11-08 from the first bytes of each model layer). For such a model the runtime turns on
|
||||
* prediction, which costs a cache for that layer and rollback copies of the recurrent state
|
||||
* (see the catalogue header). A row the catalogue gains must be added here, or the test below
|
||||
* fails: its figures cannot be checked without it.
|
||||
* Which architecture each stock row runs, and how many tokens the runtime drafts per step with
|
||||
* its multi-token-prediction layer: 0 when the GGUF has none (`nextn_predict_layers` in the GGUF
|
||||
* header, read on 2026-11-08 from the first bytes of each model layer), else the tag's own
|
||||
* `draft_num_predict` (its params layer) or the runtime's default of 4 when the tag sets none.
|
||||
* Prediction costs a cache for that layer and one rollback copy of the recurrent state per
|
||||
* drafted token (see the catalogue header). A row the catalogue gains must be added here, or the
|
||||
* test below fails: its figures cannot be checked without it.
|
||||
*/
|
||||
const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; predictionLayer: boolean }> = {
|
||||
'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false },
|
||||
'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', predictionLayer: false },
|
||||
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', predictionLayer: false },
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: true },
|
||||
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', predictionLayer: false },
|
||||
'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', predictionLayer: true },
|
||||
'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', predictionLayer: true },
|
||||
'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', predictionLayer: false },
|
||||
const ROW_ARCHITECTURE: Record<string, { architecture: ArchitectureName; draftTokens: number }> = {
|
||||
'qwen3.5:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'mimo-v2.6:9b-q4_k_m': { architecture: 'Qwen3.5-9B', draftTokens: 0 },
|
||||
'north-mini-code-1.0:q4_k_m': { architecture: 'North-Mini-Code-1.0', draftTokens: 0 },
|
||||
// Its params layer sets `draft_num_predict` 2.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 2 },
|
||||
'qwen3.6:35b-a3b': { architecture: 'Qwen3.6-35B-A3B', draftTokens: 0 },
|
||||
// A prediction layer, and no `draft_num_predict`: the runtime's 4.
|
||||
'qwen3.8:27b-q8_0': { architecture: 'Qwen3.8-27B', draftTokens: 4 },
|
||||
// Its params layer sets `draft_num_predict` 4.
|
||||
'qwen3.8:27b-mtp-q4_k_m': { architecture: 'Qwen3.8-27B', draftTokens: 4 },
|
||||
'ornith-1.5:122b': { architecture: 'Qwen3.5-122B-A10B', draftTokens: 0 },
|
||||
};
|
||||
|
||||
/** Bytes per cached value in each K/V cache type of llama.cpp: q8_0 keeps 32 values in 34 bytes. */
|
||||
@@ -109,12 +113,6 @@ const KV_BYTES_PER_VALUE: Record<string, number> = { f16: 2, q8_0: 34 / 32 };
|
||||
* default, because the runtime sets only the main context's (`--cache-type-k`).
|
||||
*/
|
||||
const DRAFT_KV_CACHE_TYPE = 'f16';
|
||||
/**
|
||||
* The recurrent states llama.cpp keeps beside the live one for a model with prediction: the
|
||||
* runtime's draft length (`draft_num_predict` 4, passed as `--spec-draft-n-max`), which
|
||||
* `need_n_rs_seq()` turns into that many rollback copies.
|
||||
*/
|
||||
const MTP_ROLLBACK_STATES = 4;
|
||||
/** The sliding-window cache holds the window plus one batch of 512 tokens (`n_swa + n_ubatch`). */
|
||||
const SLIDING_EXTRA_CELLS = 512;
|
||||
/** The head size of a Gated DeltaNet layer's keys and values, and its convolution kernel, in every model above. */
|
||||
@@ -132,7 +130,8 @@ function recurrentStateBytes(arch: Architecture, convBytes: number): number {
|
||||
}
|
||||
|
||||
/** What a `llamacpp` row's three figures must be, from its architecture and the runtime's settings. */
|
||||
function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number, number, number] {
|
||||
function llamaCppFigures(arch: Architecture, draftTokens: number): [number, number, number] {
|
||||
const predictionLayer = draftTokens > 0;
|
||||
const perValue = KV_BYTES_PER_VALUE[RUNTIME_KV_CACHE_TYPE];
|
||||
const perLayer = arch.kvHeads * arch.headDim * 2;
|
||||
// The prediction layer's cache sits in the draft context, in the draft's cache type.
|
||||
@@ -143,8 +142,9 @@ function llamaCppFigures(arch: Architecture, predictionLayer: boolean): [number,
|
||||
const recurrent = recurrentStateBytes(arch, 4);
|
||||
const sliding =
|
||||
'sliding' in arch ? arch.sliding.layers * (arch.sliding.window + SLIDING_EXTRA_CELLS) * perLayer * perValue : 0;
|
||||
// Live: the sliding window, and the recurrent state with its rollback copies under prediction.
|
||||
const live = sliding + recurrent * (1 + (predictionLayer ? MTP_ROLLBACK_STATES : 0));
|
||||
// Live: the sliding window, and the recurrent state with one rollback copy per drafted token
|
||||
// (`--spec-draft-n-max`, which `need_n_rs_seq()` turns into that many copies).
|
||||
const live = sliding + recurrent * (1 + draftTokens);
|
||||
// A context checkpoint is a copy of what cannot be rolled back: the recurrent state or the window.
|
||||
const checkpoints = RUNTIME_CTX_CHECKPOINTS * (recurrent + sliding);
|
||||
return [held, live + checkpoints + 1_000_000_000 + RUNTIME_HOST_PROMPT_CACHE_MIB * MIB, 0];
|
||||
@@ -337,9 +337,9 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
|
||||
it("recomputes every estimated row's figures from its own model's architecture and runner", () => {
|
||||
for (const model of models.filter((m) => m.figures === 'estimated')) {
|
||||
const { architecture, predictionLayer } = ROW_ARCHITECTURE[model.id];
|
||||
const { architecture, draftTokens } = ROW_ARCHITECTURE[model.id];
|
||||
const arch = ARCHITECTURES[architecture];
|
||||
const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, predictionLayer) : mlxFigures(arch);
|
||||
const expected = model.runner === 'llamacpp' ? llamaCppFigures(arch, draftTokens) : mlxFigures(arch);
|
||||
expect(figuresOf(model), `${model.id} (${architecture}, ${model.runner})`).toEqual(expected);
|
||||
}
|
||||
});
|
||||
@@ -351,9 +351,9 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
'mimo-v2.6:9b-q4_k_m': [17_408, 2_547_960_320, 0],
|
||||
// The sliding window x (1 live + 8 checkpoints) + 1.0 GB + 1 GiB.
|
||||
'north-mini-code-1.0:q4_k_m': [14_144, 3_698_117_120, 0],
|
||||
// State x (1 live + 4 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
|
||||
// layer's cache in f16 per token.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_929_969_664, 0],
|
||||
// State x (1 live + 2 rollback copies + 8 checkpoints) + 1.0 GB + 1 GiB; the prediction
|
||||
// layer's cache in f16 per token. The 27B rows below draft 4 tokens: 4 rollback copies.
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': [12_928, 2_798_242_304, 0],
|
||||
'qwen3.6:35b-a3b': [37_000, 3_000_000_000, 5_000_000_000],
|
||||
'qwen3.8:27b-q8_0': [38_912, 4_113_353_216, 0],
|
||||
'qwen3.8:27b-mtp-q4_k_m': [38_912, 4_113_353_216, 0],
|
||||
@@ -367,7 +367,7 @@ describe('the memory figures of the stock rows: each row has its own', () => {
|
||||
expect(mlxComputedPerToken(ARCHITECTURES['Qwen3.5-122B-A10B'])).toBe(43_332);
|
||||
});
|
||||
|
||||
it('lets two rows share figures only when they share the architecture, the prediction layer and the runner', () => {
|
||||
it('lets two rows share figures only when they share the architecture, the drafted tokens and the runner', () => {
|
||||
const byFigures = new Map<string, LocalModelEntry[]>();
|
||||
for (const model of models) {
|
||||
const key = figuresOf(model).join('/');
|
||||
|
||||
@@ -97,7 +97,7 @@ describe('modelFitsBudget', () => {
|
||||
// A runtime that reports less than the estimate rules out what the estimate allowed.
|
||||
expect(modelFitsBudget(mac(48), catalog, id, 131072, { measuredTotalBytes: 33 * 1024 ** 3 })).toBe(false);
|
||||
// And one that reports more admits what the estimate ruled out: on a 36 GB Mac the GGUF
|
||||
// 35B-A3B at its full window needs 28.9 GB, 1.037 of the 27.9 GB estimate and 0.963 of 28 GiB.
|
||||
// 35B-A3B at its full window needs 28.8 GB, 1.032 of the 27.9 GB estimate and 0.958 of 28 GiB.
|
||||
const moeGguf = 'qwen3.6:35b-a3b-mtp-q4_k_m';
|
||||
expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144)).toBe(false);
|
||||
expect(modelFitsBudget(mac(36), catalog, moeGguf, 262144, { measuredTotalBytes: 28 * 1024 ** 3 })).toBe(true);
|
||||
|
||||
@@ -489,7 +489,7 @@ describe('needBytes', () => {
|
||||
'qwen3.5:9b-q4_k_m': 11_380_487_246,
|
||||
'mimo-v2.6:9b-q4_k_m': 11_380_472_729,
|
||||
'north-mini-code-1.0:q4_k_m': 25_999_848_864,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_940_281_677,
|
||||
'qwen3.6:35b-a3b-mtp-q4_k_m': 28_808_554_317,
|
||||
'qwen3.6:35b-a3b': 35_955_462_956,
|
||||
'qwen3.8:27b-q8_0': 44_292_142_594,
|
||||
'qwen3.8:27b-mtp-q4_k_m': 32_055_772_698,
|
||||
@@ -647,7 +647,7 @@ describe('the MacBook Pro (Mac17,13: Apple M5 Pro, 48 GB)', () => {
|
||||
expect(
|
||||
result.alternatives.map((c) => [c.modelId, c.context, c.fit, c.speed, c.tokensPerSecond ?? null, c.tag?.kind])
|
||||
).toEqual([
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.9 GB.
|
||||
// The same model as a GGUF build: its fixed footprint holds the full window, 28.8 GB.
|
||||
['qwen3.6:35b-a3b-mtp-q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['north-mini-code-1.0:q4_k_m', 262144, 'good', 'ok', 41.8, undefined],
|
||||
['mimo-v2.6:9b-q4_k_m', 131072, 'good', 'ok', 28.6, 'small'],
|
||||
@@ -809,7 +809,7 @@ describe('every memory size, with the stock catalogue', () => {
|
||||
[32, 'Apple M1 Max', 'limited mimo-v2.6:9b-q4_k_m@131072 good ok'],
|
||||
// A Max chip runs the dense 27B fast enough (12.1); an M3 Max (300 GB/s) does not (8.9),
|
||||
// and takes the 35B-A3B as a GGUF build, whose memory does not grow after it loaded. Both
|
||||
// at 131072 tokens, 0.966 and 0.976 of the budget.
|
||||
// at 131072 tokens, 0.966 and 0.971 of the budget.
|
||||
[36, 'Apple M4 Max', 'ok qwen3.8:27b-mtp-q4_k_m@131072 tight ok'],
|
||||
[36, 'Apple M3 Max', 'ok qwen3.6:35b-a3b-mtp-q4_k_m@131072 tight ok'],
|
||||
// The tested pair on the tested Mac; on a Pro chip the 27B is slow (8.1), so the same pick.
|
||||
@@ -2077,8 +2077,8 @@ describe('the measured budget, end to end', () => {
|
||||
expect([measured.budgetSource, measured.budgetGb]).toEqual(['measured', '40.2']);
|
||||
// The same rows in the same order; only the share of each changes.
|
||||
expect(order(measured)).toEqual(order(estimated));
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.77, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.721, 0.647, 0.283, 0.283]);
|
||||
expect(rows(estimated).map((c) => c.budgetShare)).toEqual([0.957, 0.767, 0.692, 0.303, 0.303]);
|
||||
expect(rows(measured).map((c) => c.budgetShare)).toEqual([0.895, 0.717, 0.647, 0.283, 0.283]);
|
||||
for (const candidate of rows(measured)) {
|
||||
expect(candidate.budgetShare).toBeCloseTo(candidate.needBytes / MACBOOK_MEASURED_TOTAL, 3);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user