runtimeEnv() now also sets OLLAMA_KV_CACHE_TYPE=q8_0, OLLAMA_FLASH_ATTENTION=1
and LLAMA_ARG_CACHE_RAM=1024. Read in the v0.40.15 source of the pinned
runtime: the first becomes --cache-type-k/-v of llama-server (17/32 of the
f16 cache), the second --flash-attn on (a quantized V cache needs it, and
"auto" or "off" would fail such a load), and llama-server inherits the third
from ollama serve's environment, which bounds its host prompt cache to
1 GiB instead of letting it grow by gigabytes in a session. The MLX runner
reads none of them.
The catalogue's llama.cpp rows are computed for these values, so the two
change together. Not yet measured on a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The supervisor reports the child's pid the moment it is spawned, before it
answers, so a server killed during a start still leaves a pid that names the
runtime as ours. stop() takes that stored pid (previousPid) and ends a
runtime an earlier server left behind when its command line is a binary
under our runtime folder, so no caller deletes the folder under a live
process. start() takes a version: an installed model keeps the runtime
version it was installed with when a later build pins another one.
removeLocalModelFiles() refuses a local-model folder that is itself a
symbolic link (every check compared against the real path of the root, so
a link in its place passed whatever it pointed at). removePartialModelBlobs()
deletes what an interrupted download left in the model store, which the
runtime answers "not found" for and never deletes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
runtime.ts: the runtime is pinned in a compiled constant (version, URL, size,
sha256). The archive is streamed to a .partial file while it is hashed,
continued with a Range request, retried at most 3 times, and the hash is
compared before anything is unpacked; a mismatch deletes the file. It is
unpacked with /usr/bin/tar into a temporary folder and renamed.
The supervisor runs the runtime as a non-detached child with an environment
built from an allowlist by the pure runtimeEnv(), on a port chosen once by
binding. It restarts the child at most 3 times, stops it with SIGTERM, adopts
or replaces its own orphan on the port and answers port-in-use for any other
listener. parseInferenceComputeTotal() reads the device total the runtime
logs when it starts. Under vitest nothing spawns or fetches.
ollama-client.ts: the pull is parsed incrementally by a pure parser (a 23 GB
pull is about 1.4 million lines), with up to 5 retries on an error line or a
cut stream. verifyModel() loads the model with one tiny chat, checks what the
runtime reports (all on the GPU, the chosen context) and makes one tool
probe; each failure is a code.
developer-tools.ts: only /usr/bin/xcode-select -p, bounded, darwin only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codeman maintainer