diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index 53b53b9c..7aa02a57 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -702,7 +702,7 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse **The pin** (`OLLAMA_RUNTIME_PIN`, `runtime.ts`) is a COMPILED constant: version, URL, size, sha256. ⚠️ No file, setting, env var or request may supply any part of it: a hash that comes from the same writable place as the URL pins nothing. The archive is streamed to `*.partial` while it is hashed, continued with a Range request (3 bounded retries), refused when it grows past the pinned size, and the hash is compared BEFORE `/usr/bin/tar` unpacks anything into a temporary folder that is then renamed. -**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it. +**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`, plus the llama.cpp memory settings `OLLAMA_KV_CACHE_TYPE=q8_0`, `OLLAMA_FLASH_ATTENTION=1`, `LLAMA_ARG_CACHE_RAM=1024`, which the catalogue's `llamacpp` rows are computed for, so the two change together): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it. **The budget is measured.** The runtime prints the device total in its log line "inference compute" (`total="37.4 GiB"`); `parseInferenceComputeTotal()` reads it, the job stores it and judges the chosen model against it (`modelFitsBudget()`) BEFORE the model is downloaded; from then on the GET recommends with `budgetSource: 'measured'`. One consent covers the whole install: the sizes that were confirmed do not change, so a model that no longer fits is a failure with the next model, not a second question. diff --git a/src/local-model/runtime.ts b/src/local-model/runtime.ts index 0ff00614..08f15d1a 100644 --- a/src/local-model/runtime.ts +++ b/src/local-model/runtime.ts @@ -16,8 +16,26 @@ * either case) are copied from the server's; HOME is the private one; and exactly these are * set: `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, * `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, - * `OLLAMA_MAX_LOADED_MODELS=1`. Nothing else gets through: no `CODEMAN_*` and none of the - * user's own `OLLAMA_*`. + * `OLLAMA_MAX_LOADED_MODELS=1`, and the three memory settings of the llama.cpp runner (see + * MEMORY below). Nothing else gets through: no `CODEMAN_*` and none of the user's own + * `OLLAMA_*`. + * - MEMORY (what the catalogue's figures for its `llamacpp` rows assume, so change both + * together: test/local-model-catalog.test.ts holds them to each other). Read in the + * runtime's v0.40.15 source, not yet measured on a Mac: + * - `OLLAMA_KV_CACHE_TYPE=q8_0` becomes `--cache-type-k q8_0 --cache-type-v q8_0` of + * llama-server (llm/llama_server.go): the K/V cache takes 34 bytes per 32 values, 17/32 + * of f16. The runtime's FAQ calls the quality loss "very small". Never q4_0: its loss + * grows with the context, and a coding agent works at long contexts. + * - `OLLAMA_FLASH_ATTENTION=1` becomes `--flash-attn on`. A quantized V cache needs flash + * attention, and left unset the runtime passes `auto`, or `off` where it thinks the GPU + * cannot, which would make every load with a q8_0 cache fail. + * - `LLAMA_ARG_CACHE_RAM=1024` bounds llama-server's host prompt cache (the copies of + * earlier prompts it keeps in memory) to 1024 MiB. The runtime passes no `--cache-ram` + * for a coding model, and llama-server reads the variable from the environment it + * inherits (`cmd.Env = os.Environ()`). Unbounded, that cache can grow by gigabytes in a + * session; 1024 MiB is counted in every `llamacpp` row's `overheadBytes`. + * The MLX runner reads none of the three (nothing under mlxrunner/ does): an `mlx` row's + * figures do not depend on them. * - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across * `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start. * - SUPERVISOR (`createRuntimeSupervisor()`). The runtime is a NON-detached child of the @@ -83,6 +101,10 @@ export const RUNTIME_MAX_RESTARTS = 3; export const RUNTIME_DOWNLOAD_RETRIES = 3; /** `OLLAMA_KEEP_ALIVE`: how long the model stays loaded after the last request. */ export const RUNTIME_KEEP_ALIVE = '10m'; +/** `OLLAMA_KV_CACHE_TYPE`: the K/V cache type of the llama.cpp runner. The catalogue's `llamacpp` rows are computed for it. */ +export const RUNTIME_KV_CACHE_TYPE = 'q8_0'; +/** `LLAMA_ARG_CACHE_RAM`, in MiB: the bound of llama-server's host prompt cache. Counted in every `llamacpp` row's overhead. */ +export const RUNTIME_HOST_PROMPT_CACHE_MIB = 1024; /** The name of the runtime's binary inside its folder and inside the archive. */ const RUNTIME_BINARY_NAME = 'ollama'; @@ -253,6 +275,10 @@ export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv { env.OLLAMA_KEEP_ALIVE = RUNTIME_KEEP_ALIVE; env.OLLAMA_NUM_PARALLEL = '1'; env.OLLAMA_MAX_LOADED_MODELS = '1'; + // The memory settings of the llama.cpp runner (file header, MEMORY). Inert for an MLX model. + env.OLLAMA_KV_CACHE_TYPE = RUNTIME_KV_CACHE_TYPE; + env.OLLAMA_FLASH_ATTENTION = '1'; + env.LLAMA_ARG_CACHE_RAM = String(RUNTIME_HOST_PROMPT_CACHE_MIB); return env; } diff --git a/test/local-model-runtime.test.ts b/test/local-model-runtime.test.ts index e6ecbc76..e61c7434 100644 --- a/test/local-model-runtime.test.ts +++ b/test/local-model-runtime.test.ts @@ -39,7 +39,9 @@ import { removeLocalModelFiles, removePartialModelBlobs, RUNTIME_DOWNLOAD_RETRIES, + RUNTIME_HOST_PROMPT_CACHE_MIB, RUNTIME_KEEP_ALIVE, + RUNTIME_KV_CACHE_TYPE, RUNTIME_MAX_RESTARTS, RUNTIME_PORT_FIRST, RUNTIME_PORT_SPAN, @@ -273,7 +275,7 @@ describe('runtimeEnv', () => { DYLD_INSERT_LIBRARIES: '/tmp/evil.dylib', }; - it('holds exactly the allowlist: what is copied, the private HOME, and seven settings', () => { + it('holds exactly the allowlist: what is copied, the private HOME, and ten settings', () => { expect(runtimeEnv(input(SERVER_ENV))).toEqual({ PATH: '/usr/local/bin:/usr/bin:/bin', TMPDIR: '/var/folders/xy/T/', @@ -292,9 +294,28 @@ describe('runtimeEnv', () => { OLLAMA_KEEP_ALIVE: '10m', OLLAMA_NUM_PARALLEL: '1', OLLAMA_MAX_LOADED_MODELS: '1', + // The memory settings the catalogue's llamacpp rows are computed for. + OLLAMA_KV_CACHE_TYPE: 'q8_0', + OLLAMA_FLASH_ATTENTION: '1', + LLAMA_ARG_CACHE_RAM: '1024', }); }); + it('sets the llama.cpp memory settings from the exported constants, never from the server', () => { + expect(RUNTIME_KV_CACHE_TYPE).toBe('q8_0'); + expect(RUNTIME_HOST_PROMPT_CACHE_MIB).toBe(1024); + const env = runtimeEnv(input(SERVER_ENV)); + expect(env.OLLAMA_KV_CACHE_TYPE).toBe(RUNTIME_KV_CACHE_TYPE); + expect(env.LLAMA_ARG_CACHE_RAM).toBe(String(RUNTIME_HOST_PROMPT_CACHE_MIB)); + // The server's own `LLAMA_ARG_CACHE_RAM=0` and a user's q4_0 cache lose against ours. + const env2 = runtimeEnv(input({ ...SERVER_ENV, OLLAMA_KV_CACHE_TYPE: 'q4_0', OLLAMA_FLASH_ATTENTION: '0' })); + expect([env2.OLLAMA_KV_CACHE_TYPE, env2.OLLAMA_FLASH_ATTENTION, env2.LLAMA_ARG_CACHE_RAM]).toEqual([ + 'q8_0', + '1', + '1024', + ]); + }); + it('lets no CODEMAN_ variable through, the password least of all', () => { const env = runtimeEnv(input(SERVER_ENV)); expect(Object.keys(env).filter((name) => name.startsWith('CODEMAN_'))).toEqual([]); @@ -309,8 +330,10 @@ describe('runtimeEnv', () => { .sort() ).toEqual([ 'OLLAMA_CONTEXT_LENGTH', + 'OLLAMA_FLASH_ATTENTION', 'OLLAMA_HOST', 'OLLAMA_KEEP_ALIVE', + 'OLLAMA_KV_CACHE_TYPE', 'OLLAMA_MAX_LOADED_MODELS', 'OLLAMA_MODELS', 'OLLAMA_NO_CLOUD', @@ -329,7 +352,8 @@ describe('runtimeEnv', () => { ]) { expect(env, name).not.toHaveProperty(name); } - expect(env).not.toHaveProperty('LLAMA_ARG_CACHE_RAM'); + // The server's `LLAMA_ARG_CACHE_RAM=0` is not copied: the value is ours. + expect(env.LLAMA_ARG_CACHE_RAM).toBe('1024'); }); it('carries no secret and no value of the server that is not on the list', () => { @@ -365,9 +389,12 @@ describe('runtimeEnv', () => { it('leaves out what the server does not have, and sets the rest all the same', () => { expect(Object.keys(runtimeEnv(input({}))).sort()).toEqual([ 'HOME', + 'LLAMA_ARG_CACHE_RAM', 'OLLAMA_CONTEXT_LENGTH', + 'OLLAMA_FLASH_ATTENTION', 'OLLAMA_HOST', 'OLLAMA_KEEP_ALIVE', + 'OLLAMA_KV_CACHE_TYPE', 'OLLAMA_MAX_LOADED_MODELS', 'OLLAMA_MODELS', 'OLLAMA_NO_CLOUD', diff --git a/test/local-model-supervisor.test.ts b/test/local-model-supervisor.test.ts index b3e883ad..6924b065 100644 --- a/test/local-model-supervisor.test.ts +++ b/test/local-model-supervisor.test.ts @@ -206,6 +206,9 @@ describe('start', () => { OLLAMA_KEEP_ALIVE: '10m', OLLAMA_NUM_PARALLEL: '1', OLLAMA_MAX_LOADED_MODELS: '1', + OLLAMA_KV_CACHE_TYPE: 'q8_0', + OLLAMA_FLASH_ATTENTION: '1', + LLAMA_ARG_CACHE_RAM: '1024', }); });