feat(local-model): q8_0 K/V cache, flash attention and a bounded prompt cache for llama.cpp

runtimeEnv() now also sets OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_FLASH_ATTENTION=1
and LLAMA_ARG_CACHE_RAM=1024. Read in the v0.40.15 source of the pinned
runtime: the first becomes --cache-type-k/-v of llama-server (17/32 of the
f16 cache), the second --flash-attn on (a quantized V cache needs it, and
"auto" or "off" would fail such a load), and llama-server inherits the third
from ollama serve's environment, which bounds its host prompt cache to
1 GiB instead of letting it grow by gigabytes in a session. The MLX runner
reads none of them.

The catalogue's llama.cpp rows are computed for these values, so the two
change together. Not yet measured on a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/architecture-invariants.md

@@ -702,7 +702,7 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse
 
 **The pin** (`OLLAMA_RUNTIME_PIN`, `runtime.ts`) is a COMPILED constant: version, URL, size, sha256. ⚠️ No file, setting, env var or request may supply any part of it: a hash that comes from the same writable place as the URL pins nothing. The archive is streamed to `*.partial` while it is hashed, continued with a Range request (3 bounded retries), refused when it grows past the pinned size, and the hash is compared BEFORE `/usr/bin/tar` unpacks anything into a temporary folder that is then renamed.
 
-**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it.
+**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`, plus the llama.cpp memory settings `OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1`, `LLAMA_ARG_CACHE_RAM=1024`, which the catalogue's `llamacpp` rows are computed for, so the two change together): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it.
 
 **The budget is measured.** The runtime prints the device total in its log line "inference compute" (`total="37.4 GiB"`); `parseInferenceComputeTotal()` reads it, the job stores it and judges the chosen model against it (`modelFitsBudget()`) BEFORE the model is downloaded; from then on the GET recommends with `budgetSource: 'measured'`. One consent covers the whole install: the sizes that were confirmed do not change, so a model that no longer fits is a failure with the next model, not a second question.
 

src/local-model/runtime.ts

@@ -16,8 +16,26 @@
  *    either case) are copied from the server's; HOME is the private one; and exactly these are
  *    set: `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
  *    `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
- *    `OLLAMA_MAX_LOADED_MODELS=1`. Nothing else gets through: no `CODEMAN_*` and none of the
- *    user's own `OLLAMA_*`.
+ *    `OLLAMA_MAX_LOADED_MODELS=1`, and the three memory settings of the llama.cpp runner (see
+ *    MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own
+ *    `OLLAMA_*`.
+ *  - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both
+ *    together: test/local-model-catalog.test.ts holds them to each other). Read in the
+ *    runtime's v0.40.15 source, not yet measured on a Mac:
+ *     - `OLLAMA_KV_CACHE_TYPE=q8_0` becomes `--cache-type-k q8_0 --cache-type-v q8_0` of
+ *       llama-server (llm/llama_server.go): the K/V cache takes 34 bytes per 32 values, 17/32
+ *       of f16. The runtime's FAQ calls the quality loss "very small". Never q4_0: its loss
+ *       grows with the context, and a coding agent works at long contexts.
+ *     - `OLLAMA_FLASH_ATTENTION=1` becomes `--flash-attn on`. A quantized V cache needs flash
+ *       attention, and left unset the runtime passes `auto`, or `off` where it thinks the GPU
+ *       cannot, which would make every load with a q8_0 cache fail.
+ *     - `LLAMA_ARG_CACHE_RAM=1024` bounds llama-server's host prompt cache (the copies of
+ *       earlier prompts it keeps in memory) to 1024 MiB. The runtime passes no `--cache-ram`
+ *       for a coding model, and llama-server reads the variable from the environment it
+ *       inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a
+ *       session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`.
+ *    The MLX runner reads none of the three (nothing under mlxrunner/ does): an `mlx` row's
+ *    figures do not depend on them.
  *  - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
  *    `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start.
  *  - SUPERVISOR (`createRuntimeSupervisor()`). The runtime is a NON-detached child of the
@@ -83,6 +101,10 @@ export const RUNTIME_MAX_RESTARTS = 3;
 export const RUNTIME_DOWNLOAD_RETRIES = 3;
 /** `OLLAMA_KEEP_ALIVE`: how long the model stays loaded after the last request. */
 export const RUNTIME_KEEP_ALIVE = '10m';
+/** `OLLAMA_KV_CACHE_TYPE`: the K/V cache type of the llama.cpp runner. The catalogue's `llamacpp` rows are computed for it. */
+export const RUNTIME_KV_CACHE_TYPE = 'q8_0';
+/** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */
+export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024;
 
 /** The name of the runtime's binary inside its folder and inside the archive. */
 const RUNTIME_BINARY_NAME = 'ollama';
@@ -253,6 +275,10 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv {
   env.OLLAMA_KEEP_ALIVE = RUNTIME_KEEP_ALIVE;
   env.OLLAMA_NUM_PARALLEL = '1';
   env.OLLAMA_MAX_LOADED_MODELS = '1';
+  // The memory settings of the llama.cpp runner (file header, MEMORY). Inert for an MLX model.
+  env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE;
+  env.OLLAMA_FLASH_ATTENTION = '1';
+  env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB);
   return env;
 }
 

test/local-model-runtime.test.ts

@@ -39,7 +39,9 @@ import {
   removeLocalModelFiles,
   removePartialModelBlobs,
   RUNTIME_DOWNLOAD_RETRIES,
+  RUNTIME_HOST_PROMPT_CACHE_MIB,
   RUNTIME_KEEP_ALIVE,
+  RUNTIME_KV_CACHE_TYPE,
   RUNTIME_MAX_RESTARTS,
   RUNTIME_PORT_FIRST,
   RUNTIME_PORT_SPAN,
@@ -273,7 +275,7 @@ describe('runtimeEnv', () => {
     DYLD_INSERT_LIBRARIES: '/tmp/evil.dylib',
   };
 
-  it('holds exactly the allowlist: what is copied, the private HOME, and seven settings', () => {
+  it('holds exactly the allowlist: what is copied, the private HOME, and ten settings', () => {
     expect(runtimeEnv(input(SERVER_ENV))).toEqual({
       PATH: '/usr/local/bin:/usr/bin:/bin',
       TMPDIR: '/var/folders/xy/T/',
@@ -292,9 +294,28 @@ describe('runtimeEnv', () => {
       OLLAMA_KEEP_ALIVE: '10m',
       OLLAMA_NUM_PARALLEL: '1',
       OLLAMA_MAX_LOADED_MODELS: '1',
+      // The memory settings the catalogue's llamacpp rows are computed for.
+      OLLAMA_KV_CACHE_TYPE: 'q8_0',
+      OLLAMA_FLASH_ATTENTION: '1',
+      LLAMA_ARG_CACHE_RAM: '1024',
     });
   });
 
+  it('sets the llama.cpp memory settings from the exported constants, never from the server', () => {
+    expect(RUNTIME_KV_CACHE_TYPE).toBe('q8_0');
+    expect(RUNTIME_HOST_PROMPT_CACHE_MIB).toBe(1024);
+    const env = runtimeEnv(input(SERVER_ENV));
+    expect(env.OLLAMA_KV_CACHE_TYPE).toBe(RUNTIME_KV_CACHE_TYPE);
+    expect(env.LLAMA_ARG_CACHE_RAM).toBe(String(RUNTIME_HOST_PROMPT_CACHE_MIB));
+    // The server's own `LLAMA_ARG_CACHE_RAM=0` and a user's q4_0 cache lose against ours.
+    const env2 = runtimeEnv(input({ ...SERVER_ENV, OLLAMA_KV_CACHE_TYPE: 'q4_0', OLLAMA_FLASH_ATTENTION: '0' }));
+    expect([env2.OLLAMA_KV_CACHE_TYPE, env2.OLLAMA_FLASH_ATTENTION, env2.LLAMA_ARG_CACHE_RAM]).toEqual([
+      'q8_0',
+      '1',
+      '1024',
+    ]);
+  });
+
   it('lets no CODEMAN_ variable through, the password least of all', () => {
     const env = runtimeEnv(input(SERVER_ENV));
     expect(Object.keys(env).filter((name) => name.startsWith('CODEMAN_'))).toEqual([]);
@@ -309,8 +330,10 @@ describe('runtimeEnv', () => {
         .sort()
     ).toEqual([
       'OLLAMA_CONTEXT_LENGTH',
+      'OLLAMA_FLASH_ATTENTION',
       'OLLAMA_HOST',
       'OLLAMA_KEEP_ALIVE',
+      'OLLAMA_KV_CACHE_TYPE',
       'OLLAMA_MAX_LOADED_MODELS',
       'OLLAMA_MODELS',
       'OLLAMA_NO_CLOUD',
@@ -329,7 +352,8 @@ describe('runtimeEnv', () => {
     ]) {
       expect(env, name).not.toHaveProperty(name);
     }
-    expect(env).not.toHaveProperty('LLAMA_ARG_CACHE_RAM');
+    // The server's `LLAMA_ARG_CACHE_RAM=0` is not copied: the value is ours.
+    expect(env.LLAMA_ARG_CACHE_RAM).toBe('1024');
   });
 
   it('carries no secret and no value of the server that is not on the list', () => {
@@ -365,9 +389,12 @@ describe('runtimeEnv', () => {
   it('leaves out what the server does not have, and sets the rest all the same', () => {
     expect(Object.keys(runtimeEnv(input({}))).sort()).toEqual([
       'HOME',
+      'LLAMA_ARG_CACHE_RAM',
       'OLLAMA_CONTEXT_LENGTH',
+      'OLLAMA_FLASH_ATTENTION',
       'OLLAMA_HOST',
       'OLLAMA_KEEP_ALIVE',
+      'OLLAMA_KV_CACHE_TYPE',
       'OLLAMA_MAX_LOADED_MODELS',
       'OLLAMA_MODELS',
       'OLLAMA_NO_CLOUD',

test/local-model-supervisor.test.ts

@@ -206,6 +206,9 @@ describe('start', () => {
       OLLAMA_KEEP_ALIVE: '10m',
       OLLAMA_NUM_PARALLEL: '1',
       OLLAMA_MAX_LOADED_MODELS: '1',
+      OLLAMA_KV_CACHE_TYPE: 'q8_0',
+      OLLAMA_FLASH_ATTENTION: '1',
+      LLAMA_ARG_CACHE_RAM: '1024',
     });
   });