From 46fa954b2ebd1a8ba9bfe78c74d5f795cb632f51 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Sun, 8 Nov 2026 09:35:28 +0100 Subject: [PATCH] docs(local-model): what each Mac gets, and where every memory figure comes from docs/local-models.md gains "Which model fits which Mac": the answer of the stock catalogue for every memory size from 8 to 512 GB, why the chip changes it at 36 to 96 GB, and why nothing is offered below 65,536 tokens. "How much memory a model needs" explains the two runners (llama.cpp allocates its K/V cache at load, MLX grows it and keeps up to 8 GiB of prefix snapshots), the three runtime settings the llama.cpp rows assume, and per row where its memory and speed figures come from. The check after the download now says that for an MLX build it proves only the weights, and "What to know" says most figures are calculated, not measured. CLAUDE.md: every row names its runner and carries its own figures, and the llama.cpp rows and runtimeEnv() change together. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- docs/local-models.md | 87 +++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 83 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 655cffa8..4500c408 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -191,7 +191,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph ⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) -**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` +**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), and every row names its `runner` (`mlx` or `llamacpp`, the build a Mac really pulls) and carries figures of its OWN: computed from its model's config.json by that runner's rule (catalogue header, MEMORY), never borrowed from another row; `test/local-model-catalog.test.ts` recomputes them, and the `llamacpp` rows assume the runtime's `OLLAMA_KV_CACHE_TYPE=q8_0` and `LLAMA_ARG_CACHE_RAM=1024` (`runtimeEnv()`), so change the two together. The need is judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` **Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaves a record the state file does not know; a record under the reserved id that says exactly what ours would say is ours (`isSameLocalEndpoint()`). ⚠️ A job whose end could not be written (a full disk) is kept in memory (`unrecorded`): the view shows it, the next verb writes it, and `installing` in the file while this process runs no job always reads as `paused` or that failure, never as a running job. ⚠️ The runtime's pid is stored the moment it is spawned, and Stop, Remove and the boot of anything that is not a running installation end a runtime an earlier server left behind (`supervisor.stop({ previousPid })`, ours by its command line) before any file is deleted; an installed model keeps the runtime VERSION it was installed with while that folder is unpacked, and a start that fails shows as `runtime-start-failed` with Start as the next step. Stop and "Use another port" answer 409 `sessions-open` while a live session uses the model. ⚠️ Remove (409 `sessions-open` with the names while a live session uses the endpoint) undoes the connection FIRST, restores the flag only when this feature switched it on AND no other endpoint is saved, deletes only inside `local-model/` (the models only when asked), and leaves the coding agent. ⚠️ Developer tools: only `/usr/bin/xcode-select -p`, bounded; never `git`/`python3`/`xcrun` from the server (each opens a system window on such a Mac). Tests: `test/local-model-install-job.test.ts` (every state with every event), `-install-service`, `-install-e2e` (real modules over `test/fixtures/fake-ollama.ts`), `-state-store`, `-connect`, `test/cli-install-runner.test.ts`, `test/custom-model-hosts-writer.test.ts`, `test/routes/local-model-routes.test.ts`, `test/routes/local-model-launch-guard.test.ts`. → [architecture-invariants#local-model-install](docs/architecture-invariants.md#local-model-install), `docs/local-models.md`, `docs/api-reference.md` (Local model) diff --git a/docs/local-models.md b/docs/local-models.md index 6400634e..47fdfac3 100644 --- a/docs/local-models.md +++ b/docs/local-models.md @@ -14,6 +14,75 @@ This page is the reference for how it works. The HTTP contract is in [api-reference.md](api-reference.md) ("Local model"). The rules that must not be broken are in [architecture-invariants.md](architecture-invariants.md#local-model-install). +## Which model fits which Mac + +The window offers the best model of the catalogue (`src/config/local-model-catalog.ts`) that +fits this Mac's memory at a context a coding agent can work with, and up to four more. What +the catalogue answers today, before the runtime measured anything (the budget is then an +estimate: 2/3 of the memory below 36 GB, 3/4 from 36 GB, less 1 GiB, and never more than the +memory less 6, 8 or 12 GiB for macOS, Codeman, a browser and the agent): + +| Memory | Budget | What it gets | +| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 8 GB | 2.1 GB | Nothing. The smallest model a coding agent can work with needs 9.8 GB at 65,536 tokens | +| 16 GB | 10.4 GB | A small model (`limited`): MiMo V2.6 9B at 65,536 tokens, 9.8 GB, tight. Its base, Qwen3.5 9B, is the one alternative | +| 18 GB | 11.8 GB | The same at 131,072 tokens, 11.0 GB, tight | +| 24 GB | 16.1 GB | The same at 131,072 tokens, with room | +| 32 GB | 21.8 GB | The same. North Mini Code 1.0 needs 21.8 GB and misses the estimate by a hair; once the runtime measures about 3/4 of the memory, as it did on a 16 and a 48 GB Mac, the Mac gets it, or Qwen3.8 27B on a Max chip | +| 36 GB | 27.9 GB | Qwen3.8 27B at 131,072 tokens on an M4 or M5 Max; on a slower chip, where the dense 27B computes to under 10 tokens per second, Qwen3.6 35B-A3B as a GGUF build | +| 48 GB | 37.6 GB | Qwen3.6 35B-A3B (MLX) at 131,072 tokens: the pair we tested, "Recommended" on an Apple M5 Pro with 48 GB. On a Max chip Qwen3.8 27B at 262,144 tokens | +| 64 GB | 50.5 GB | Qwen3.8 27B at 262,144 tokens on a Max chip; Qwen3.6 35B-A3B (MLX) on a Pro chip | +| 96 GB | 76.2 GB | Qwen3.8 27B at 262,144 tokens; on an M3 Max, Qwen3.6 35B-A3B (MLX) | +| 128 GB | 102 GB | Ornith 1.5 122B-A10B (MLX) at 131,072 tokens, 97.4 GB, tight | +| 192, 256, 512 GB | 154 GB+ | Ornith 1.5 122B-A10B at 262,144 tokens | + +"Tight" means the need is between 85% and 98% of the budget; above 98% a model is not +offered. A model that computes to under 10 tokens per second on the chip is "slow" and ranks +after the fast ones, which is why the chip changes the pick at 36 to 96 GB. Every model is +offered at the largest context it fits at, and never below 65,536 tokens: OpenCode's own +prompt is 8,000 to 10,000 tokens, and at 32,768 it would compact after less than one default +file read. `test/local-model-recommend.test.ts` ("every memory size") pins this table. + +### How much memory a model needs + +The need of a model is its peak: the weights, what stays held per token of context, what the +runner holds beside both, and what it takes for a moment while a long prompt is processed. The +runtime runs a model with one of two runners, which spend memory differently, so every row +names its runner (`runner`) and its figures follow that runner's rule: + +- **llama.cpp** (GGUF builds, on Metal) allocates the K/V cache for the whole context when the + model loads and does not grow after that. The runtime is started with + `OLLAMA_KV_CACHE_TYPE=q8_0` (the cache in 8 bits, 17/32 of the default f16; the runtime's + own documentation calls the loss very small), `OLLAMA_FLASH_ATTENTION=1` (an 8-bit cache + needs it) and `LLAMA_ARG_CACHE_RAM=1024` (llama-server keeps copies of earlier prompts, and + this bounds them to 1 GiB). A row's figures are computed from its model's `config.json`: the + K/V cache of its attention layers, its recurrent or sliding-window state, 1.0 GB for the + runner and the 1 GiB prompt cache. No peak: the buffers are allocated at load. +- **MLX** grows its cache with the tokens actually used, keeps snapshots of the recurrent + state, and may keep up to 8 GiB of snapshots of earlier prompts, which no setting bounds. A + Mac always takes the MLX build of a tag that has one. Below 48 GB that 8 GiB does not fit + beside a useful model, so the catalogue lists GGUF builds there. + +Where every figure comes from: + +| Model | Runner | Memory figures | Speed (tokens per second per 100 GB/s) | +| ---------------------------- | --------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------- | +| `qwen3.5:9b-q4_k_m` | llama.cpp | computed from the Qwen3.5-9B config | 9.3: a published run on an M1 Max | +| `mimo-v2.6:9b-q4_k_m` | llama.cpp | the same (a fine-tune of Qwen3.5-9B) | 9.3: the same architecture and size | +| `north-mini-code-1.0:q4_k_m` | llama.cpp | computed from its config | 13.6: the 35B-A3B's, the same 3 billion active parameters | +| `qwen3.6:35b-a3b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.6-35B-A3B config | 13.6: a published run on an M1 Max | +| `qwen3.6:35b-a3b` | MLX | **measured** on an Apple M5 Pro with 48 GB | 16.9: a published run; 67.7 measured on the tested Mac | +| `qwen3.8:27b-q8_0` | llama.cpp | computed from the Qwen3.8-27B config | 1.7: the row below scaled by file size | +| `qwen3.8:27b-mtp-q4_k_m` | llama.cpp | computed from the Qwen3.8-27B config | 2.95: a published run without prediction (a floor) | +| `ornith-1.5:122b` | MLX | computed from the Qwen3.5-122B-A10B config, scaled by the measured MLX row, + 8 GiB | 5.0: the 35B-A3B's, x 3/10 active parameters | + +Sizes are what the registry serves a Mac for each tag (every layer plus the config), read on +2026-11-08. Only one row was ever run on a Mac, so every other answer in the table above is a +calculation. For a llama.cpp build the check after the download catches one that was judged +too kindly; for an MLX build it cannot (see "The check" below). The measured row's figures come from single long prompts, not a session of many requests, in +which MLX may keep up to 8 GiB more. The catalogue header lists the models left out on +purpose (no evidence of working tool calls, a restrictive licence, a known bug) and why. + ## What an install does, in order The person picks a model, presses "Install on this Mac" and confirms the sizes once. The @@ -49,8 +118,10 @@ Every step can be repeated: Resume and "Try again" enter the step the job stoppe copied from the server. `HOME` is a private folder. Exactly these are set: `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, - `OLLAMA_MAX_LOADED_MODELS=1`. No `CODEMAN_*` variable and none of the user's own - `OLLAMA_*` settings reach it. + `OLLAMA_MAX_LOADED_MODELS=1`, and three memory settings of the llama.cpp runner: + `OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1` and `LLAMA_ARG_CACHE_RAM=1024` + (see "How much memory a model needs" above). No `CODEMAN_*` variable and none of the user's + own `OLLAMA_*` settings reach it. - **The port is chosen once**, by binding, from 11435 across 20 ports, and stored. 11434 is left alone: it is the default of a runtime the user may run themselves, which is never touched. @@ -69,8 +140,7 @@ judges the chosen model against it before a byte of the model is downloaded. A m longer fits ends the job there with "does not fit" and the next model to try. From then on the window shows the measured budget and the word "estimated" is gone. -The need of a model is its peak: the weights, what stays held per token of context, what the -runner holds beside both, and what it takes for a moment while a long prompt is processed. +The need of a model is its peak (see "How much memory a model needs" above). ### The model download @@ -86,7 +156,10 @@ After the pull, in this order: 1. One tiny chat with `reasoning_effort: "none"`. It loads the model. 2. `GET /api/ps` must show the whole model on the GPU (`size_vram` equal to `size`) at the - chosen context length. + chosen context length. For a llama.cpp build that is the runtime's reckoning of the whole + load, K/V cache included. For an MLX build it proves only the weights: the runtime books + an MLX model at its weights (runtime issue 19417), so for those rows the catalogue's + figures are all there is. 3. One tool probe on `/v1/chat/completions` (a `tools` array, `reasoning_effort: "none"`, `max_tokens` 1024). It passes on a tool call whose arguments parse. @@ -261,6 +334,10 @@ or `python3`. a session uses the model: close those sessions first. - **The runtime does not enforce the context length.** A prompt longer than the context was accepted in testing. Do not rely on the runtime to stop one. +- **Most memory figures are calculated, not measured.** One model was run on one Mac; the + others are computed from their architecture, and the runtime settings they assume (an 8-bit + K/V cache, flash attention, a 1 GiB prompt cache) were read in its source, not yet observed + on a Mac. "Recommended" appears only for the pair we ran, on the Mac we ran it on. - **The model stays in memory for ten minutes** after the last request, then the runtime unloads it.