feat(local-model): q8_0 K/V cache, flash attention and a bounded prompt cache for llama.cpp

runtimeEnv() now also sets OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_FLASH_ATTENTION=1
and LLAMA_ARG_CACHE_RAM=1024. Read in the v0.40.15 source of the pinned
runtime: the first becomes --cache-type-k/-v of llama-server (17/32 of the
f16 cache), the second --flash-attn on (a quantized V cache needs it, and
"auto" or "off" would fail such a load), and llama-server inherits the third
from ollama serve's environment, which bounds its host prompt cache to
1 GiB instead of letting it grow by gigabytes in a session. The MLX runner
reads none of them.

The catalogue's llama.cpp rows are computed for these values, so the two
change together. Not yet measured on a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/local-model/runtime.ts

@@ -16,8 +16,26 @@
  *    either case) are copied from the server's; HOME is the private one; and exactly these are
  *    set: `OLLAMA_HOST=127.0.0.1:<port>`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`,
  *    `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`,
- *    `OLLAMA_MAX_LOADED_MODELS=1`. Nothing else gets through: no `CODEMAN_*` and none of the
- *    user's own `OLLAMA_*`.
+ *    `OLLAMA_MAX_LOADED_MODELS=1`, and the three memory settings of the llama.cpp runner (see
+ *    MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own
+ *    `OLLAMA_*`.
+ *  - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both
+ *    together: test/local-model-catalog.test.ts holds them to each other). Read in the
+ *    runtime's v0.40.15 source, not yet measured on a Mac:
+ *     - `OLLAMA_KV_CACHE_TYPE=q8_0` becomes `--cache-type-k q8_0 --cache-type-v q8_0` of
+ *       llama-server (llm/llama_server.go): the K/V cache takes 34 bytes per 32 values, 17/32
+ *       of f16. The runtime's FAQ calls the quality loss "very small". Never q4_0: its loss
+ *       grows with the context, and a coding agent works at long contexts.
+ *     - `OLLAMA_FLASH_ATTENTION=1` becomes `--flash-attn on`. A quantized V cache needs flash
+ *       attention, and left unset the runtime passes `auto`, or `off` where it thinks the GPU
+ *       cannot, which would make every load with a q8_0 cache fail.
+ *     - `LLAMA_ARG_CACHE_RAM=1024` bounds llama-server's host prompt cache (the copies of
+ *       earlier prompts it keeps in memory) to 1024 MiB. The runtime passes no `--cache-ram`
+ *       for a coding model, and llama-server reads the variable from the environment it
+ *       inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a
+ *       session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`.
+ *    The MLX runner reads none of the three (nothing under mlxrunner/ does): an `mlx` row's
+ *    figures do not depend on them.
  *  - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across
  *    `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start.
  *  - SUPERVISOR (`createRuntimeSupervisor()`). The runtime is a NON-detached child of the
@@ -83,6 +101,10 @@ export const RUNTIME_MAX_RESTARTS = 3;
 export const RUNTIME_DOWNLOAD_RETRIES = 3;
 /** `OLLAMA_KEEP_ALIVE`: how long the model stays loaded after the last request. */
 export const RUNTIME_KEEP_ALIVE = '10m';
+/** `OLLAMA_KV_CACHE_TYPE`: the K/V cache type of the llama.cpp runner. The catalogue's `llamacpp` rows are computed for it. */
+export const RUNTIME_KV_CACHE_TYPE = 'q8_0';
+/** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */
+export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024;
 
 /** The name of the runtime's binary inside its folder and inside the archive. */
 const RUNTIME_BINARY_NAME = 'ollama';
@@ -253,6 +275,10 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv {
   env.OLLAMA_KEEP_ALIVE = RUNTIME_KEEP_ALIVE;
   env.OLLAMA_NUM_PARALLEL = '1';
   env.OLLAMA_MAX_LOADED_MODELS = '1';
+  // The memory settings of the llama.cpp runner (file header, MEMORY). Inert for an MLX model.
+  env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE;
+  env.OLLAMA_FLASH_ATTENTION = '1';
+  env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB);
   return env;
 }