Files
Codeman/src
Codeman maintainer f60b63c005 feat(local-model): q8_0 K/V cache, flash attention and a bounded prompt cache for llama.cpp
runtimeEnv() now also sets OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_FLASH_ATTENTION=1
and LLAMA_ARG_CACHE_RAM=1024. Read in the v0.40.15 source of the pinned
runtime: the first becomes --cache-type-k/-v of llama-server (17/32 of the
f16 cache), the second --flash-attn on (a quantized V cache needs it, and
"auto" or "off" would fail such a load), and llama-server inherits the third
from ollama serve's environment, which bounds its host prompt cache to
1 GiB instead of letting it grow by gigabytes in a session. The MLX runner
reads none of them.

The catalogue's llama.cpp rows are computed for these values, so the two
change together. Not yet measured on a Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:18:51 +01:00
..
2026-05-31 05:17:07 +02:00
2026-02-18 12:51:35 +01:00