diff --git a/.github/SECURITY.md b/.github/SECURITY.md index db062166..f97d2c13 100644 --- a/.github/SECURITY.md +++ b/.github/SECURITY.md @@ -69,6 +69,7 @@ shared-host, multi-user, or tunneled deployments. - **Multi-instance tmux socket is process-wide.** Two Codeman instances on the same `CODEMAN_INSTANCE` share a tmux socket and can attach each other's live sessions — isolate with distinct `CODEMAN_INSTANCE` values. - **The live log-tail route reads `/var/log` and `~/logs`** in addition to the session working directory (read-only) — a deliberate choice for tailing system/app logs. On a password-protected remote deployment an authenticated user can therefore read those roots outside their session. See `docs/security-architecture.md` §5. +- **The local model runtime listens on a loopback port with no login.** The model server the local model install starts (macOS only, opt-in by a click) accepts requests from any process of any account on the Mac, and from any page served from `localhost`, like every model server of its kind. Codeman limits what that buys (one model in the endpoint record, a digest check before an agent is started on it) but does not authenticate the port. See `docs/security-architecture.md` §10d. - **The web-tab proxy fetches from the server's network position.** Any authenticated user can save a dashboard URL on loopback or a private range and have Codeman relay to it; that is the feature. Link-local and cloud-metadata addresses are the only refused targets (see below). On a shared host, restrict who holds an account. Recent hardening (2026-09-04): the web-tab proxy, its "Test" probe and its diff --git a/CLAUDE.md b/CLAUDE.md index fdaf6491..dbc1a783 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -188,11 +188,13 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Case rename** (`POST /api/cases/:name/rename`, `src/case-rename.ts`; `docs/wiki/Core-Concepts.md#case`): a case under the cases dir is its folder name and always moves; a linked case renames its alias and moves its real folder only with `renameFolder`; remote and Docker cases are refused. ⚠️ A folder that sessions have open is the hazard: live sessions at or under it make the route answer `409 requiresStop` with their names, and only `stopSessions: true` ends them (via `ctx.cleanupSession`: closed like a tab close, uploads included, nothing is restarted); the free-target pre-check runs before that and refuses a dangling symlink too (`lstat`, as `moveCaseFolder` does), so no session is ended for a move that cannot happen. `withCaseLock` holds the old and new paths for the whole operation (`isCasePathLocked`; the `Session` constructor throws for a working directory inside a held path, so nothing is constructed mid-move). ⚠️ A create awaits the owner's layout lock and a settings read between `new Session` and the server's map, where the rename looks for open sessions, so the constructor also records the session (`noteCaseSessionStarting`) until `registerSessionWithLayout()` or `stop()` settles it (30 s TTL for one that never registers), and a folder move refuses `409` with `renameCode: "BUSY"` while one is recorded at or under the folder (`isCaseSessionStarting`, checked in the same tick that takes the lock); an alias-only rename is not refused. A create whose path was validated before the move but whose `Session` is constructed after it is not covered (the plain check-then-use race of any create path), `moveCaseFolder` refuses an existing target itself (POSIX `rename` would silently replace an EMPTY directory) and retries `EBUSY`/`EPERM`/`EACCES` before answering `423 LOCKED`. If the registries (`linked-cases.json`, `caseOrder`, `lastUsedCase`) fail after the move, `settings.json` and the folder are put back (a failed move-back says the folder is now at the new path), and an error after `stopSessions` carries `stoppedSessions`. Claude Code keys history by path (`claudeProjectDirName`: every non-alphanumeric becomes `-`), so `/projects/` is renamed too, or merged entry by entry, never overwriting, into one already at the new key (case delete never removes history, so an earlier case of the new name leaves one); other CLIs' history, Docker bind mounts, `git worktree` links, cron jobs' `workingDir`, Claude history of subfolder sessions and Claude Code's per-project entry in `~/.claude.json` (local-scope `mcpServers`, `allowedTools`, trust; writing another CLI's config is the opt-in MCP sync's job) are NOT migrated. The history scanner prefers the decoded project key over a transcript's `cwd` that no longer exists, so Resume follows a rename. Only an admin's rename touches the server-wide `settings.json` (`caseOrder`, `lastUsedCase`); the owner's default session group is carried across with `TabLayoutService`. `settings.json` is written before `linked-cases.json`, so a failed or refused registry write puts `settings.json` back and leaves the registry as it was. ⚠️ Every write to `linked-cases.json` (link, unlink, rename, and a create or clone into a custom folder) goes through `mutateLinkedCases()` (`src/web/linked-cases-writer.ts`: serialized, temp file + rename, and a file that does not parse is refused, never read as `{}`; the one direct writer is config-backup restore, `restoreConfigBackup()`, which must work on such a file), and a check that must still hold when the entry is written (the name is free; for a rename, the old entry still names the folder) runs INSIDE its callback: two clones that claimed one name used to both pass a re-read made outside it and both answer success. Pinned by `test/linked-cases-writer.test.ts`, and for a rename by `test/routes/case-rename-routes.test.ts`. -**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`): points a session at a user-configured OpenAI-compatible endpoint (store `custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`, 0600). `authStyle` is `bearer` or `api-key`, never both headers (hangs the server). The per-CLI redirect is registry data, `capabilities.customModelInjection` (`env` / `configContentEnv` / `configDir` / `unsupported`), computed by the pure `custom-model-injection.ts`; ⚠️ `configDir` writes an isolated per-session config, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config; the one exception is a recipe's `linkFromRealHome` (deepseek's `profiles`, `sessions`, `settings.yaml`), which symlinks those named entries back out of the real home so they survive the redirect. ⚠️ Claude applies via `Session.restartCli()` (a relaunch in the existing pane, so the conversation is pinned as `resumeSessionId`), the rest one-shot at launch; retired env keys must also be `setenv -u`'d (`_pendingEnvUnsets`), since tmux env survives `respawn-pane`. ⚠️ Remote/Docker sessions are refused (400). ⚠️ Persist only KEYS (`__customModel`), never values (they carry the API key); the one exception is `ownEnv`'s `homeEnvVar` entry, the session's OWN home path (absolute only: a `clis.json` override can name any variable there), which `setCustomModel()` puts back on clear and recovery clamps, since the selection overwrote the tmux `setenv` that held it. ⚠️ Every redirectable var must be in that CLI's `privilegedEnvKeys`, and `ANTHROPIC_*` stays out of claude's `allowedPrefixes`. ⚠️ The Run-menu picker builds entries from `window.__codemanCustomModelClis` (escaped via `escapeScriptJson()`), never a hardcoded CLI id list, and launches through `run()` via a temporary `_runMode` swap, never `setRunMode()`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) +**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`): points a session at a user-configured OpenAI-compatible endpoint (store `custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`, 0600). `authStyle` is `bearer` or `api-key`, never both headers (hangs the server). The per-CLI redirect is registry data, `capabilities.customModelInjection` (`env` / `configContentEnv` / `configDir` / `unsupported`), computed by the pure `custom-model-injection.ts`; ⚠️ `configDir` writes an isolated per-session config, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config; the one exception is a recipe's `linkFromRealHome` (deepseek's `profiles`, `sessions`, `settings.yaml`), which symlinks those named entries back out of the real home so they survive the redirect. ⚠️ Claude applies via `Session.restartCli()` (a relaunch in the existing pane, so the conversation is pinned as `resumeSessionId`), the rest one-shot at launch; retired env keys must also be `setenv -u`'d (`_pendingEnvUnsets`), since tmux env survives `respawn-pane`. ⚠️ Remote/Docker sessions are refused (400). ⚠️ Persist only KEYS (`__customModel`), never values (they carry the API key); the one exception is `ownEnv`'s `homeEnvVar` entry, the session's OWN home path (absolute only: a `clis.json` override can name any variable there), which `setCustomModel()` puts back on clear and recovery clamps, since the selection overwrote the tmux `setenv` that held it. ⚠️ Every redirectable var must be in that CLI's `privilegedEnvKeys`, and `ANTHROPIC_*` stays out of claude's `allowedPrefixes`. ⚠️ The Run-menu picker builds entries from `window.__codemanCustomModelClis` (escaped via `escapeScriptJson()`), never a hardcoded CLI id list, and launches through `run()` via a temporary `_runMode` swap, never `setRunMode()`. ⚠️ Every write of `custom-model-hosts.json` goes through `mutateCustomModelHosts()` (`src/custom-model-hosts-writer.ts`: one at a time, temp file + rename, the route's check inside the callback, and a file that does not parse is refused with 409, never read as `[]` and replaced); the one direct writer is config-backup restore. The endpoint id `local-ollama` is reserved for the local model install (see Local model install). Pinned by `test/custom-model-hosts-writer.test.ts`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) ⚠️ **llama-swap endpoints** (one model at a time): the apply routes check `GET /running` and return `requiresConfirmation` before evicting a model another live session uses; `confirmedSwap` and `confirmedContext` are SEPARATE flags and must stay so. Claude alone gets a context floor (`CLAUDE_MIN_SAFE_CONTEXT_TOKENS`); context is parsed from `/running`'s `cmd`, never trusted from `/props`. The floor also divides that context by the model's parallel slot count (`modelSlotCounts`: `--parallel`/`-np` from `cmd`, else `/props` `total_slots`; `perSlotContext()`), and `/props` only ever RAISES that warning: it supplies the context only when `cmd` states none and the count only when `cmd` has no `--parallel`, never a larger pool. `/props` is probed once per launch command (it refreshes llama-swap's last-use time, so a `ttl` would never expire), and a `/props` context beside an explicit `--parallel N` is already per slot, so it is not divided again. Backend log lines come from llama-swap's `/api/events` `upstream` source, never `/logs`. → [architecture-invariants#custom-model-endpoint-profiles](docs/architecture-invariants.md#custom-model-endpoint-profiles) -**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL, read-only stage): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ **Read-only**: it downloads nothing, installs nothing, writes no file and adds no settings key; the install flow is a later stage and nothing here may grow a write before it. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested` and is sent only when the pick's model has a `tested` entry for exactly this chip, memory size and harness; no row has one today. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Card rules: `src/web/CLAUDE.md` (Model on this Mac). → `docs/api-reference.md` (Local model recommendation), `docs/versioning-policy.md` +**Local model recommendation** (`GET /api/local-model`, `src/local-model/`, `src/config/local-model-catalog.ts`, `src/types/local-model.ts`; EXPERIMENTAL): the server looks at the Mac it runs on and says which local coding model fits it and with which agent CLI. ⚠️ The READ side stays read-only: the probe, the catalogue and `recommend()` download nothing, install nothing and write nothing; everything that changes the machine is the install below, behind an explicit request. Apple silicon macOS only: anywhere else the answer is a `200` with verdict `none` (reasons `not-macos`, `intel`, `memory`, `disk`, `macos`, `no-harness`), and a failed probe is an answer, never a 500. The probe (`hardware.ts`) is ONE bounded async spawn of `/usr/sbin/sysctl` WITHOUT `-n`, parsed as `key: value` lines by the pure `parseSysctl()`, plus `statfs` of the data dir (probed bounded first, #516) and `node:os` fallbacks, cached 60 s; ⚠️ never `system_profiler`, never a second spawn, and `iogpu.wired_limit_mb` of 0 means not set. The catalogue is a compiled constant validated by a Zod schema: ⚠️ no override file and no runtime data, and a model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`, so an id can never name another registry host). ⚠️ A harness is a catalogue ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`) plus a `HarnessFact` built from the CLI registry at call time (`harness-facts.ts`); no code compares a CLI id (both id-branching guards scan these files). `recommend()` (`recommend.ts`) is pure. ⚠️ A model's need is its PEAK (`bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`, measured on a Mac for one row and marked `estimated` on the others), judged against a budget that is `estimated` until the runtime ran and `measured` from then on (`RecommendOptions.measuredTotalBytes`). ⚠️ Ranking, lexicographic: tested on a Mac like this first, then speed `ok` before `unknown` before `slow`, then model rank, harness rank, larger context; the fit grade decides whether a row is listed and what it says, it does NOT sort, so a model is offered at the largest context it fits at. ⚠️ `limited` is a statement about the MAC (no model with 12 billion parameters or more fits), judged before the disk filter and whatever the pick is; a small pick under `ok` carries `noteSmallPick`, and a model left out for disk space is named in `noteDiskDropped`, never dropped in silence. ⚠️ The word "Recommended" exists only in `headlineTested`, and the tag "Tested on a Mac like this" only on a row whose model has a `tested` entry for exactly this chip, memory size and harness. ⚠️ Speed is `ok`/`slow`/`unknown`, graded on the unrounded figure, and a chip in no chips row is `unknown`: never guess a bandwidth. ⚠️ ONE unit for every computed size a person reads: decimal GB text from `formatGb()` (`format.ts`), sent beside the bytes, and the browser never divides bytes; the one exception is the installed memory, shown as `memClassGb` (the size the Mac is sold as, 48 and not 51.5). ⚠️ Every sentence the server chooses is a `LOCAL_MODEL_LABELS`, `LOCAL_MODEL_TAG_LABELS` or `LOCAL_MODEL_INSTALL_LABELS` text with `{placeholder}` slots filled from `vars` (no apostrophe in a text: the zh-CN key is single-quoted), and each needs a zh-CN entry. Admin only in multi-user mode (403 before anything is probed). Tests: `test/local-model-*.test.ts`, `test/routes/local-model-routes.test.ts`. Window rules: `src/web/CLAUDE.md` (Run a model on this Mac). → `docs/api-reference.md` (Local model), `docs/versioning-policy.md` + +**Local model install** (`POST /api/local-model/install|cancel|resume|runtime`, `DELETE /api/local-model`; `src/local-model/{install-job,runtime,ollama-client,connect,state-store,paths,developer-tools}.ts`; EXPERIMENTAL, admin only in multi-user mode on EVERY verb; guide `docs/local-models.md`): installs the chosen model on the Mac the server runs on and ends on a session of the coding agent running on it. ONE job of five idempotent steps (`runtime`, `model`, `verify`, `agent`, `connect`; `agent` only when the coding agent is missing), a pure reducer (`reduceInstallJob()`) over three tables in `install-job.ts` driven by one service per process (`getLocalModelService()`); the window polls the GET once a second, no SSE event. ⚠️ Layout only through `localModelPaths()` (all `dataPath()`): `local-model/runtime/ollama-/`, `local-model/models/`, `local-model/home/` (the runtime's PRIVATE HOME, so the user's `~/.ollama` is never touched) and the state file `local-model.json` (0600, ONE serialized writer, total reader, holds NO path); nothing is written or deleted outside `local-model/` except that file and the two writes of the connect step. ⚠️ The runtime pin (`OLLAMA_RUNTIME_PIN`: version, URL, size, sha256) is a COMPILED constant no file, setting or request may supply; the hash is compared BEFORE anything is unpacked. ⚠️ The runtime is a NON-detached child whose environment comes from the `runtimeEnv()` ALLOWLIST, never `process.env` (no `CODEMAN_*`, none of the user's `OLLAMA_*`); the port is chosen ONCE by binding (from 11435, never 11434) and stored; a listener on it is adopted or replaced only when its stored pid runs a binary under OUR runtime folder, anything else is `port-in-use`; started at boot when installed and enabled, max 3 restarts, SIGTERM at shutdown beside `stopDeepSeekWeb()`; under vitest nothing spawns or fetches. ⚠️ Consent is bound to the size: `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`) or the answer is 409 `size-changed`; the model, context and CLI must be a row the current recommendation offers (400). ⚠️ ONE job at a time (409 `job-running`); a job a restart interrupted comes back `paused` and NEVER continues without a click (so `shutdown()` writes nothing), and an interrupted step is never recorded as failed. ⚠️ An event the transition table does not list is refused (same object back, 409 `wrong-state`): add a row to the table, never a special case. ⚠️ The budget is MEASURED from the runtime's "inference compute" log line and the model judged against it BEFORE it is downloaded. ⚠️ The pull stream is parsed incrementally (about 1.4 million lines for 23 GB): never buffer it, never write state per line. ⚠️ A failure is a code with ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed"; a probe that ended on its token limit is `probe-unfinished`, not `no-tool-call`. ⚠️ The coding agent is installed through the shared runner `src/cli-install-runner.ts` (extracted from `cli-registry-routes.ts`, ONE install slot per CLI for both callers) with `stockInstallCommand()`, the COMPILED stock command, never the `clis.json`-merged one (this path has no `cliManagementEnabled` gate); only for a catalogue harness row, after the consent click that showed the literal command, audited (`cli_install`, `via: 'local-model'`); afterwards `forgetResolvedCli()` and a fresh lookup, and still missing is `harness-failed`. ⚠️ Connect writes ONE endpoint record under the RESERVED id `local-ollama` through `mutateCustomModelHosts()` and `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, remembering the flag's first value; a record with that id that is not ours is never overwritten, and while it is ours it is MANAGED: the endpoint routes refuse to edit or delete it and the discovery sweep skips it (the runtime's port has no login, so its `models` must stay the one model that was checked). ⚠️ The model digest is stored and read again at boot and before every START of a coding agent on that endpoint (`launchRefusalMessage()`): the two custom-model launch routes in `session-routes.ts` ask it before anything is written, the reboot-restore rebuild asks it before it builds (reason `local-model-not-ready`, left on offer), and `Session`'s launch guard (`setLaunchGuard()`, set in `setupSessionListeners()`) asks it right before a NEW agent process starts (a fresh pane, the respawn of a dead one, `restartCli()`), never for an attach to an agent that runs; a refusal is a `LaunchRefusedError` that never falls back to a direct PTY. A changed or missing model, a runtime Codeman did not start or that did not start, or another model id refuses the launch with one sentence. ⚠️ The connection is CLAIMED in the state file (`connection`, with the flag's value read by `inspectLocalEndpoint()`) BEFORE the endpoint record and the settings flag are written, so a state write that fails or a kill between the two never leaves a record the state file does not know; a record under the reserved id that says exactly what ours would say is ours (`isSameLocalEndpoint()`). ⚠️ A job whose end could not be written (a full disk) is kept in memory (`unrecorded`): the view shows it, the next verb writes it, and `installing` in the file while this process runs no job always reads as `paused` or that failure, never as a running job. ⚠️ The runtime's pid is stored the moment it is spawned, and Stop, Remove and the boot of anything that is not a running installation end a runtime an earlier server left behind (`supervisor.stop({ previousPid })`, ours by its command line) before any file is deleted; an installed model keeps the runtime VERSION it was installed with while that folder is unpacked, and a start that fails shows as `runtime-start-failed` with Start as the next step. Stop and "Use another port" answer 409 `sessions-open` while a live session uses the model. ⚠️ Remove (409 `sessions-open` with the names while a live session uses the endpoint) undoes the connection FIRST, restores the flag only when this feature switched it on AND no other endpoint is saved, deletes only inside `local-model/` (the models only when asked), and leaves the coding agent. ⚠️ Developer tools: only `/usr/bin/xcode-select -p`, bounded; never `git`/`python3`/`xcrun` from the server (each opens a system window on such a Mac). Tests: `test/local-model-install-job.test.ts` (every state with every event), `-install-service`, `-install-e2e` (real modules over `test/fixtures/fake-ollama.ts`), `-state-store`, `-connect`, `test/cli-install-runner.test.ts`, `test/custom-model-hosts-writer.test.ts`, `test/routes/local-model-routes.test.ts`, `test/routes/local-model-launch-guard.test.ts`. → [architecture-invariants#local-model-install](docs/architecture-invariants.md#local-model-install), `docs/local-models.md`, `docs/api-reference.md` (Local model) **MCP server sync** (opt-in, `mcpSyncEnabled`, SYNCED, default OFF; `src/mcp-sync.ts`, `GET`/`POST /api/mcp-sync`): copies each installed, enabled CLI's user-level MCP servers into the others. It is the ONE subsystem that writes another CLI's REAL user config (`~/.claude.json`, `~/.codex/config.toml`, `~/.gemini/*`, opencode's), which is why it is opt-in and admin-only in multi-user mode (both verbs 403 for a non-admin, and the Settings group is hidden for them). Where each CLI keeps the file is registry data, `capabilities.mcpConfig` (`{ path, format, relocation? }`), never a branch on the id. ⚠️ Tools that sync but are not run modes (GitHub Copilot CLI, the first) are plain data in `MCP_SYNC_ONLY_TOOLS` (`src/mcp-sync-targets.ts`), take part only when installed or already configured, and their companion `settings.json` (`disabledMcpServers`) is read, never written; a test must clear their relocation vars too. ⚠️ ADDITIVE only: a name already defined, in any shape, is never edited or removed (a different same-name definition is a reported conflict), and a server switched off in its own CLI is never copied. ⚠️ Never write a file that did not parse; re-parse the NEW text and require every added server to read back before the tmp+rename (written through a symlink, previous file kept as `.codeman-bak`, one apply at a time, else 409). ⚠️ A file that receives copied `env`/`headers` (secrets) is left `0600`, and so is the backup. ⚠️ Responses carry server NAMES only, never env values, headers or file text: a parse failure is reported by line and column (`describeMcpSyncError`), never the parser's own message (smol-toml and V8 both quote source). ⚠️ `mcpConfig.relocation` names the env var the CLI reads to move its file (`CLAUDE_CONFIG_DIR`, `CODEX_HOME`, `XDG_CONFIG_HOME`, `GEMINI_CLI_HOME`, and `COPILOT_HOME` for Copilot), resolved from the SERVER env at call time; a relative value reports the target `skipped`, never a guessed write, and a per-session `envOverrides` relocation is not followed. Tests must pass `home` (which drops the `process.env` default) or clear those vars first. → `docs/cli-registry.md` (MCP server sync), `docs/api-reference.md`, `docs/wiki/Settings-Reference.md` @@ -326,7 +328,7 @@ One module per domain in `src/web/routes/` (plus a barrel; `ls src/web/routes/` ## State Files -All in `~/.codeman/`: `state.json` (sessions, settings, respawn, orchestrator, cron jobs/runs, owner tab layouts, case default groups), `mux-sessions.json` (tmux recovery), `settings.json` (user prefs), `push-keys.json` + `push-subscriptions.json`, `session-lifecycle.jsonl` (audit log), `update-status.json` (self-updater progress, polled across the service restart), `docker-env-applied.json` (Compose deployment only: sha256 of the Dockerfile + compose file the running container was built from, written by `Start-Codeman.sh`, read by the self-updater's environment gate), `docker-build-source.json` (Compose deployment only: the checkout's HEAD commit and `package-lock.json` hash the `codeman-node-modules`/`codeman-dist` volumes currently reflect, written by both `Start-Codeman.sh` and a successful in-place self-update, compared to detect and refresh a volume left stale by an externally-triggered rebuild), `linked-cases.json`, `webviews.json` (saved web-tab dashboard URLs), `remote-hosts.json` + `remote-cases.json`, `docker-hosts.json` + `docker-cases.json` + `docker-exports/`, `subagent-window-states.json` + `subagent-parents.json` (subagent window layout, GET/PUT `/api/subagent-window-states`/`-parents`), `hook-secret` (per-instance), `users.json` (multi-user, mode 0600) + `admin-audit.jsonl`, `intents.json` (Read My Mind intent profiles, mode 0600), `agent-inbox.json` (agent inbox snapshot, only while `agentInboxPersistEnabled` is on, mode 0600), `edit-history/.json` (Edit history panel, one store per session, every diff an agent wrote, mode 0600), `webhook.json` (webhook notifications: enabled/service/scope plus the ntfy/Slack/Discord URL, a bearer secret, so mode 0600, kept out of `settings.json` and never returned by `/api/webhook`, which is admin-only in multi-user mode), `discord.json` (Discord bridge: bot token, a bearer secret, so mode 0600, never returned by `/api/discord`, admin-only in multi-user mode) + `discord-bridge.json` (sessionId to Discord thread id), `config-backup-id` (the random id config backups record as each manifest's `origin`, so instances sharing a backup folder keep to their own snapshots; mode 0600, never tracked or restored, and deleting it unlists this instance's earlier snapshots), `backups/config/` (the default config-backup folder, moved by `configBackupDir`: `cfg-YYYYMMDD-HHMMSS/` snapshots, 0700 with 0600 files, each with a `manifest.json` of SHA-256s, the origin and the retention in force (`keepCount`/`keepDays`), which a pass falls back to when `settings.json` lacks a retention key), `certs/` (self-signed TLS for `--https`), `.env` (CODEMAN_USERNAME/PASSWORD fallback for the `codeman attach` CLI), `install.log` (installer step output, written by `install.sh`'s `run_step`) `tailscale-rename` (the node name before `install.sh` renamed it, so uninstall can offer it back) and `runtime/` (the Node.js and tmux `install.sh` downloaded on its no-admin route, linked from `~/.local/bin`; all three installer-route only). Transient: `self-update-runner.sh`. Multi-user case spaces live OUTSIDE the data dir at `~/codeman-users//cases` (shared across instances like `~/codeman-cases`, override `CODEMAN_USER_SPACES_DIR`). +All in `~/.codeman/`: `state.json` (sessions, settings, respawn, orchestrator, cron jobs/runs, owner tab layouts, case default groups), `mux-sessions.json` (tmux recovery), `settings.json` (user prefs), `push-keys.json` + `push-subscriptions.json`, `session-lifecycle.jsonl` (audit log), `update-status.json` (self-updater progress, polled across the service restart), `docker-env-applied.json` (Compose deployment only: sha256 of the Dockerfile + compose file the running container was built from, written by `Start-Codeman.sh`, read by the self-updater's environment gate), `docker-build-source.json` (Compose deployment only: the checkout's HEAD commit and `package-lock.json` hash the `codeman-node-modules`/`codeman-dist` volumes currently reflect, written by both `Start-Codeman.sh` and a successful in-place self-update, compared to detect and refresh a volume left stale by an externally-triggered rebuild), `linked-cases.json`, `webviews.json` (saved web-tab dashboard URLs), `remote-hosts.json` + `remote-cases.json`, `docker-hosts.json` + `docker-cases.json` + `docker-exports/`, `subagent-window-states.json` + `subagent-parents.json` (subagent window layout, GET/PUT `/api/subagent-window-states`/`-parents`), `hook-secret` (per-instance), `users.json` (multi-user, mode 0600) + `admin-audit.jsonl`, `intents.json` (Read My Mind intent profiles, mode 0600), `agent-inbox.json` (agent inbox snapshot, only while `agentInboxPersistEnabled` is on, mode 0600), `edit-history/.json` (Edit history panel, one store per session, every diff an agent wrote, mode 0600), `webhook.json` (webhook notifications: enabled/service/scope plus the ntfy/Slack/Discord URL, a bearer secret, so mode 0600, kept out of `settings.json` and never returned by `/api/webhook`, which is admin-only in multi-user mode), `discord.json` (Discord bridge: bot token, a bearer secret, so mode 0600, never returned by `/api/discord`, admin-only in multi-user mode) + `discord-bridge.json` (sessionId to Discord thread id), `local-model.json` (local model install: state, step, model, port, pid, digest, the previous `customModelEndpointsEnabled` value; mode 0600, holds no path, not part of a config backup) + `local-model/` (the pinned runtime, the model store and the runtime's private HOME, removed by `DELETE /api/local-model`), `config-backup-id` (the random id config backups record as each manifest's `origin`, so instances sharing a backup folder keep to their own snapshots; mode 0600, never tracked or restored, and deleting it unlists this instance's earlier snapshots), `backups/config/` (the default config-backup folder, moved by `configBackupDir`: `cfg-YYYYMMDD-HHMMSS/` snapshots, 0700 with 0600 files, each with a `manifest.json` of SHA-256s, the origin and the retention in force (`keepCount`/`keepDays`), which a pass falls back to when `settings.json` lacks a retention key), `certs/` (self-signed TLS for `--https`), `.env` (CODEMAN_USERNAME/PASSWORD fallback for the `codeman attach` CLI), `install.log` (installer step output, written by `install.sh`'s `run_step`) `tailscale-rename` (the node name before `install.sh` renamed it, so uninstall can offer it back) and `runtime/` (the Node.js and tmux `install.sh` downloaded on its no-admin route, linked from `~/.local/bin`; all three installer-route only). Transient: `self-update-runner.sh`. Multi-user case spaces live OUTSIDE the data dir at `~/codeman-users//cases` (shared across instances like `~/codeman-cases`, override `CODEMAN_USER_SPACES_DIR`). **Generated top-level dirs** (all gitignored — don't edit or commit): `dist/` (esbuild output), `out/`, `coverage/`, `test-results/`, `tmp/`, `screenshots-echo-diag/`. The committed gesture bundle (`src/web/public/gesture/gesture-codeman.js`) IS tracked, but its runtime wasm/model assets (`src/web/public/gesture/wasm/`, `*.task`) are fetched and gitignored. diff --git a/config/test-suites.ts b/config/test-suites.ts index e6f95e04..ffcfb2f8 100644 --- a/config/test-suites.ts +++ b/config/test-suites.ts @@ -61,6 +61,7 @@ export const BROWSER_TEST_GLOBS = [ 'test/terminal-touch-swipe-axis.browser.test.ts', 'test/phone-press-feedback.browser.test.ts', 'test/desktop-connect-page.browser.test.ts', // no server: the page is served by request interception + 'test/local-model-ui.browser.test.ts', ]; /** diff --git a/docs/api-reference.md b/docs/api-reference.md index b0464ac8..0e0a53dd 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -1676,15 +1676,44 @@ Only replies from `allowedUserIds`, in `guildId`, inside a thread the bridge cre `GET /api/doctor[?category=core|office|other]` returns the `codeman doctor --json` report (`platform`, `summary`, `tools[]` with `status` `ok` \| `missing` \| `outdated` \| `skipped` \| `error`, `version`, `path`, `installHint`). The probe engine is synchronous, so it runs in a child process of the same entry script, never on the server's event loop (30 s timeout). It names install paths and versions, so it is admin only in multi-user mode (`403`). `400` for an unknown category, `500` if the child produces no report. -## Local model recommendation +## Local model -`GET /api/local-model` looks at the machine the server runs on and says which local coding model fits it, and with which agent CLI. It covers Apple silicon macOS only; anywhere else the answer is still a `200`, with verdict `none`. **Read-only:** the route downloads nothing, installs nothing, writes no file and reads or changes no setting (setting a model up from Codeman is a later step and is not built yet). It names the host's hardware, so it is admin only in multi-user mode (`403`, answered before anything is probed). The models, their sizes and the agent CLIs it considers are a catalogue compiled into the release; there is no override file. +`GET /api/local-model` looks at the machine the server runs on and says which local coding model fits it, with which agent CLI, and where an install of one stands. Five verbs install a model, pause and continue the install, stop and start the model runtime, and remove everything again. It covers Apple silicon macOS only; anywhere else the GET is still a `200`, with verdict `none`, and nothing can be installed. How it works: [local-models.md](local-models.md). -**Experimental.** This endpoint and its payload are labeled experimental until the step that sets a model up ships: that step will add to and may change the fields below, the verdicts and the sentences, in a MINOR or PATCH release (see [versioning-policy.md](versioning-policy.md), "What SemVer does NOT cover", item 3). Do not build on its shape yet. +**Experimental.** These routes and their payloads are labeled experimental: the fields below, the states, the error codes and the sentences may change or go in a MINOR or PATCH release (see [versioning-policy.md](versioning-policy.md), "What SemVer does NOT cover", item 3). Do not build on their shape yet. -| Method | Path | Body | Notes | -| ------ | ------------------ | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `GET` | `/api/local-model` | none | `200` with `{ hardware, recommendation, catalogUpdated }`. `403 FORBIDDEN` for a non-admin in multi-user mode. `500 INTERNAL_ERROR` only when the recommendation itself fails, never for a failed probe. | +Every route is admin only in multi-user mode (`403 FORBIDDEN`, answered before anything is probed, parsed or started): the GET names the host's hardware, and the verbs download and run a program, install an agent CLI and write the server-wide settings. There is no SSE event: the window polls the GET once a second while it is open. + +| Method | Path | Body | Notes | +| -------- | -------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `GET` | `/api/local-model` | none | `200` with `{ hardware, recommendation, catalogUpdated, install }`. `500 INTERNAL_ERROR` only when the recommendation itself fails, never for a failed probe. | +| `POST` | `/api/local-model/install` | `{ modelId, cli, context, acknowledgedBytes }` | Starts the one install job and answers at once; the job runs on in the server. `400 INVALID_INPUT` when the three choices are not a row the server offers right now. `409`: `job-running`, `already-installed`, `size-changed`. | +| `POST` | `/api/local-model/cancel` | none | Interrupts the step in flight; the job is `paused` at that step. `409 wrong-state` when no job is running. | +| `POST` | `/api/local-model/resume` | `{ newPort? }` (may be absent) | Continues a `paused` or `failed` job at the same step. `newPort: true` chooses another port first. `409`: `job-running`, `wrong-state`, `sessions-open` (`newPort` while live sessions use the model). | +| `POST` | `/api/local-model/runtime` | `{ action: "start" \| "stop", newPort? }` | `stop` stops the runtime and keeps it from starting at the next boot; `start` undoes that. `newPort: true` with `start` chooses another port and moves the saved endpoint to it. `409`: `job-running`, `wrong-state` (nothing is installed), `sessions-open` (`stop` or `newPort` while live sessions use the model: their agents keep calling the address they were started with). | +| `DELETE` | `/api/local-model` | `{ removeModel? }` (may be absent) | Stops the runtime, removes the saved endpoint, restores the settings flag when this feature switched it on and no other endpoint is saved, deletes the runtime; the model only with `removeModel: true`. `409`: `job-running`, `sessions-open`. | + +Every verb answers `{ success: true, data: }` with the `install` object (below) as it is after the verb. A refusal the feature names is a `409` whose envelope also carries the conflict: + +```json +{ + "success": false, + "error": "The download is 23.3 GB now. Check the sizes and confirm again.", + "errorCode": "CONFLICT", + "data": { + "reason": "size-changed", + "message": { "text": "The download is {size} GB now. Check the sizes and confirm again.", "vars": { "size": "23.3" } }, + "expectedBytes": 23291770627, + "expectedGb": "23.3" + } +} +``` + +`data.reason` is `job-running`, `already-installed` (remove the installed model first), `size-changed` (with `expectedBytes` and `expectedGb`), `sessions-open` (with `sessions`, the names of the live sessions that use the model) or `wrong-state`. A `409 CONFLICT` without `data` means a config file that must not be overwritten (`settings.json` or `custom-model-hosts.json` does not parse); `error` names the fix. + +`POST /api/local-model/install` also answers `400` when a saved model endpoint already uses the id `local-ollama` and was not created by this feature: rename or delete that endpoint first. The check is made before anything is downloaded. Every `400` the install names, and the `507 OPERATION_FAILED` of a first install on a full disk, carries its sentence as `data.message` (`{ text, vars }`, like a `409`), so a client can print it in its own language. + +**Consent is bound to the size shown.** `acknowledgedBytes` must be the `installBytes` of the chosen row: the model download plus the runtime download while the runtime is not unpacked yet. The server compares it with its own total before anything starts and answers `409 size-changed` when they differ, so a click confirms exactly the size the person saw. A probe that fails is an answer, not an error: a server that is not on macOS, or a Mac whose `sysctl` call failed or timed out, still gets a `200`, and `hardware.source` says what was read. @@ -1696,18 +1725,46 @@ A probe that fails is an answer, not an error: a server that is not on macOS, or - `chip`, `model`, `macosVersion`, `macosMajor`, `wiredLimitMb` (`0` means not set), `freeDiskBytes` and `freeDiskGb` (free space on the filesystem of Codeman's data directory). Each is absent when it could not be read. - `source`: `sysctl` (the one bounded `sysctl` call answered), `os-fallback` (macOS, but that call failed: memory, architecture, the chip name and the free disk space come from Node, and nothing only `sysctl` knows, such as `arm64`, `translated`, `model`, `macosVersion`, `macosMajor` or `wiredLimitMb`, is present) or `not-probed` (not macOS, nothing was run). - `recommendation`: - - `verdict`: `ok`, `limited` or `none`. `limited` is a statement about the Mac: no model with 12 billion parameters or more fits its memory, so what it can run handles small edits and questions and does not replace a cloud model. It is judged on every model that fits, before the free disk space is looked at and whatever the pick is. With `ok` the pick can still be a small model (a larger one may fit only tightly, have no speed figure or have no room on the disk); `notes` then says so. + - `verdict`: `ok`, `limited` or `none`. `limited` is a statement about the Mac: no model with 12 billion parameters or more fits its memory, so what it can run handles small edits and questions and does not replace a cloud model. It is judged on every model that fits, before the free disk space is looked at and whatever the pick is. With `ok` the pick can still be a small model (a larger one may be slow, have no speed figure or have no room on the disk); `notes` then says so. - `reason`, present only with `none`: `not-macos`, `intel`, `memory`, `disk`, `macos` (too old) or `no-harness` (no agent CLI that can be connected to a local model is installed or installable). `memory` has three sentences: nothing fits at all (slot `memory`, the size the Mac is sold as), a model fits but only below the context length an agent CLI needs (slots `budget` and `need`, so a Mac that misses by a little reads as that), and the memory size could not be read (no slot). - `headline` and `notes[]`: `{ text, vars? }`. `text` is an English sentence that may hold `{placeholder}` slots, and `vars` carries the slot values as text. Notes, in order: the pick is a small model although larger ones fit (verdict `ok` only); a model that fits the memory was left out because its download would not leave 10 GiB free on the disk (slots `model` and `missing`: the best-ranked such model and the GB to free up); nobody tested the pick on a Mac like this one. - - `pick` (absent with `none`) and `alternatives[]` (at most four, best first; empty with `none`), each one model at one context length with one agent CLI. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing): + - `pick` (absent with `none`) and `alternatives[]` (at most four, best first; empty with `none`), each one model at one context length with one agent CLI. The order: a pair tested on a Mac like this one first, then speed (`ok` before `unknown` before `slow`), then the model's rank, the agent CLI's rank and the larger context. The memory fit decides whether a row is listed and what it says (`fit`, `tag`), it does not sort, so a model is offered at the largest context it fits at. A model whose download would leave less than 10 GiB free on the disk is not listed (an unknown free disk space excludes nothing): - `modelId` (`name:tag`), `modelLabel`, `license`, `paramsB` (parameters in billions), `context` (tokens); - `cli`, `cliLabel`, `cliInstalled` (`false`: the CLI is not on this machine yet but can be installed without administrator rights), `slowFirstTurn?`; - - `needBytes` and `needGb` (memory the model needs at that context), `downloadBytes` and `downloadGb`, `budgetShare` (need divided by `budgetBytes`, rounded to three decimals); + - `needBytes` and `needGb` (memory the model needs at that context, at its PEAK: the weights, what stays held per token of context, what the runner holds beside both, and what it takes for a moment while a long prompt is processed), `needSource` (`measured`: the figures were measured for this model on a Mac; `estimated`: they are another model's), `downloadBytes` and `downloadGb`, `budgetShare` (need divided by `budgetBytes`, rounded to three decimals); + - `installBytes` and `installGb`: the whole download a click on Install starts for this row, `downloadBytes` plus the runtime archive while the runtime is not unpacked. It is the number to send as `acknowledgedBytes`; + - `harnesses[]`: every agent CLI the model can be run with at this context, best first: `{ cli, cliLabel, installed, installCommand?, slowFirstTurn? }`. `installCommand` is the literal command the install would run for a CLI that is not installed (the release's own stock command), for the consent screen. The first entry is the one the flat `cli` fields name; + - `tag?`: `{ kind, label }`, the one tag of the row. `kind` is `tested`, `best`, `tight`, `slow` or `small`; - `fit`: `good` (up to 0.85 of the budget) or `tight` (up to 0.98), with its label `fitLabel`; - `speed`: `ok`, `slow` (under 10 tokens per second) or `unknown`, with `speedLabel` (absent for `ok`) and `tokensPerSecond` (rounded to one decimal, absent for `unknown`; `speed` is decided on the unrounded figure, so `10` can be `slow`). `unknown` means the chip is in no row of the catalogue's chip table, or the catalogue has no speed figure for the model: a speed is never guessed; - - `tested`: we ran this model with this CLI on a Mac like this one. No catalogue row has such an entry yet, so it is `false` today. - - `budgetBytes` and `budgetGb`: the memory a model may use on this Mac (`0` for `not-macos` and `intel`, where none is computed). `budgetSource` is `wired-limit` (taken from `iogpu.wired_limit_mb`) or `estimated` (computed from the installed memory). + - `tested`: we ran this model with this CLI on a Mac like this one (same chip, same memory size). Its `tokensPerSecond` is then the speed measured in that run. + - `budgetBytes` and `budgetGb`: the memory a model may use on this Mac (`0` for `not-macos` and `intel`, where none is computed). `budgetSource` is `estimated` (computed from the installed memory: all there is before the runtime ran), `wired-limit` (taken from `iogpu.wired_limit_mb`) or `measured` (the device total the runtime itself printed when it started; the GET uses it from the first install on). - `catalogUpdated`: ISO date on which the catalogue's rows were last checked. +- `install`: where the install stands and what can be done about it. + - `state`: `idle` (nothing installed, no job), `installing`, `paused` (Cancel or a server restart stopped the job; it never continues without a click), `failed` or `installed`. + - `step` and `steps[]` (present with `installing`, `paused` and `failed`): the step the job is in or stopped in, and the steps of this job in order, out of `runtime`, `model`, `verify`, `agent`, `connect`. `agent` is absent from a job whose agent CLI was already installed. + - `progress` (while a download runs, and as it stood on a paused or failed job): `completedBytes`, `totalBytes`, `completedGb`, `totalGb`, and while it moves `bytesPerSecond`, `mbPerSecond` (whole decimal megabytes per second as text) and `secondsLeft`. During the `model` step both byte figures come from the download's own lines, not from the catalogue. + - `error` (with `failed`; with `installed` only when something blocks a session): `{ code, message: { text, vars? }, next, nextModel?, detail? }`. `next` is the ONE action that helps: `retry` (`POST .../resume`), `other-port` (`POST .../resume` or `.../runtime` with `newPort: true`), `next-model` (choose again; `nextModel` is `{ modelId, modelLabel }` when the server knows one), `start` (`POST .../runtime`) or `remove` (`DELETE`). `detail` is raw text for the curious (an installer's last lines), not translated. + - `model` (`{ id, label }`), `cli`, `cliLabel`, `context`: the choice of the job or of the installation. `port`: the runtime's loopback port, once chosen. + - `running`: the runtime Codeman started is alive right now. A runtime the person stopped is `running: false` with no `error`. + - `loaded` (with `installed`): what the runtime reported when the job checked the loaded model: `context`, `memoryBytes`, `memoryGb`, `onGpu`. + - `endpointId` (with `installed`): the saved model endpoint a session is started on, with `cli`. + - `developerTools`: `present`, `missing` (the Mac has no Xcode or Command Line Tools, so macOS will offer to install them the first time an agent runs `git` or `python3`) or `unknown`. + - `runtime`: `{ version, installed, downloadBytes, downloadGb, unpackedGb }`. `downloadBytes` is `0` once the runtime is unpacked. + - `folder`: the absolute folder everything of this feature is kept in. + - `actions[]`: exactly what the server accepts right now, out of `install`, `cancel`, `resume`, `start`, `stop`, `remove` and `start-session` (not a route here: start a session on `endpointId`; offered only while the runtime runs and nothing blocks a session). + +Error codes (`install.error.code`), each with one next step: + +| Next step | Codes | +| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `retry` | `runtime-download-failed`, `runtime-damaged`, `runtime-unpack-failed`, `runtime-start-failed`, `disk-full`, `pull-failed`, `context-not-applied`, `load-failed`, `probe-unfinished`, `harness-failed`, `endpoint-id-taken`, `connect-failed`, `internal` | +| `start` | `runtime-stopped` (the runtime crashed three times in a row and is no longer restarted) | +| `other-port` | `port-in-use` | +| `next-model` | `model-not-found`, `does-not-fit`, `no-tool-call` | +| `remove` | `model-changed` (the model's digest is no longer the one stored at install time; a session on it is refused) | + +Starting a session on the saved endpoint goes through the existing custom model launch (`POST /api/quick-start` with `customModel`, or `POST /api/sessions/:id/custom-model`). For the endpoint this feature manages both answer `OPERATION_FAILED` with one sentence while the runtime is not running, when the model's digest changed, or for a model other than the installed one. Every computed size a person reads travels twice: as bytes, and as decimal GB text without the unit (`"needGb": "29.6"`). Show the text and add the unit; do not divide the bytes yourself, so one request never shows two different roundings. The installed memory is the exception named above: show `memClassGb`. diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index f3fb427d..fef64c53 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -694,6 +694,38 @@ Either way the actual launch (`runCustomModelEntry`) routes through `run()` itse ⚠️ **The loading banner shows the REAL backend log line, not a guess, and has no countdown or auto-timeout at all.** `getLatestLlamaSwapLogLine()` holds one `GET /api/events` SSE connection open per endpoint (confirmed live to stay open indefinitely — read past 220KB over 8s with no `done`; idle-closed after 30s via `pruneIdleLlamaSwapLogTails`, same 20s sweep as the swap-displacement check above), parsing `logData` frames and keeping only `source: "upstream"` (the real `llama-server` process's own stdout) lines, never `source: "proxy"` (llama-swap's own request-access log). ⚠️ `GET /logs` — the endpoint this feature's own first cut targeted, since the name suggested it — was confirmed live to carry ONLY the proxy log and never a single backend line, even seconds after a real, verified model swap; caught and corrected by a live check before merge, not after. The banner itself dropped its size-scaled expected-time estimate and matching auto-timeout (a guess dressed up as a fact that could kill a genuinely slow load partway through on slower hardware) for a generic hardware/model-size disclaimer plus a user-driven **Cancel** button (`_showCenterStatus`'s `onCancel` option, a real button distinct from the plain "×" close glyph an `'error'`-type banner gets) that ends the wait and closes the session on the user's own call rather than a guessed deadline. +### Local model install + +**A model that runs on this Mac** (`src/local-model/`, `src/config/local-model-catalog.ts`, routes `src/web/routes/local-model-routes.ts`; EXPERIMENTAL; guide `docs/local-models.md`): the server looks at the Mac it runs on, says which local coding model fits it, and installs one: a pinned runtime it supervises, the model, the checks, the coding agent, and one saved model endpoint. The read side (`hardware.ts`, the catalogue, the pure `recommend()`) is described in the root `CLAUDE.md`; this section is the install. + +**The layout** is one pure function (`localModelPaths()`, `paths.ts`): `/local-model/` holds `runtime/ollama-/` (plus its `.tgz.partial` and `.unpack` while it is fetched), `models/` and `home/`; `/local-model.json` is the state file. ⚠️ Every path comes from `dataPath()` through that function and NONE is ever read back from the state file (which holds no path), so a state file cannot point a delete anywhere. ⚠️ Nothing is written or deleted outside `local-model/` except the state file and the two writes of the connect step. The runtime's `HOME` is the private `home/` (measured to work), so its key pair lands there and the user's own `~/.ollama` is never created, read or removed; `home/` and a `.ollama/id_ed25519` anywhere are on the sensitive-path list. + +**The pin** (`OLLAMA_RUNTIME_PIN`, `runtime.ts`) is a COMPILED constant: version, URL, size, sha256. ⚠️ No file, setting, env var or request may supply any part of it: a hash that comes from the same writable place as the URL pins nothing. The archive is streamed to `*.partial` while it is hashed, continued with a Range request (3 bounded retries), refused when it grows past the pinned size, and the hash is compared BEFORE `/usr/bin/tar` unpacks anything into a temporary folder that is then renamed. + +**Supervision** (`createRuntimeSupervisor()`): the runtime is a NON-detached child, started at boot once a model is installed and enabled, restarted at most 3 times in a row, SIGTERMed at shutdown beside `stopDeepSeekWeb()`. ⚠️ Its environment comes from the pure `runtimeEnv()` ALLOWLIST (PATH, the private HOME, TMPDIR, LANG, the proxy variables, and exactly `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, `OLLAMA_MAX_LOADED_MODELS=1`): never `process.env` whole (it carries `CODEMAN_PASSWORD`, and a user's own `OLLAMA_ORIGINS=*` would open the port to every web page). ⚠️ The port is chosen ONCE by binding (11435 across 20; 11434 belongs to a runtime the user may run) and stored; it is never searched again on a later start, because it is part of the saved endpoint's URL. ⚠️ Pid and port are in the state file: a listener on the port that answers `/api/version` and whose pid runs a binary under OUR runtime folder is our own orphan (a server that was killed leaves one: a child is not ended by its parent's death) and is adopted or replaced; ANY other listener is `port-in-use`, never something to talk to. ⚠️ The supervisor never writes the state file: it reports through `onChange`, and the job stores what it reports. ⚠️ Under vitest nothing spawns and nothing is fetched unless a test injects it. + +**The budget is measured.** The runtime prints the device total in its log line "inference compute" (`total="37.4 GiB"`); `parseInferenceComputeTotal()` reads it, the job stores it and judges the chosen model against it (`modelFitsBudget()`) BEFORE the model is downloaded; from then on the GET recommends with `budgetSource: 'measured'`. One consent covers the whole install: the sizes that were confirmed do not change, so a model that no longer fits is a failure with the next model, not a second question. + +**The job** (`install-job.ts`) is a pure reducer (`reduceInstallJob()`) over three tables in that file (`INSTALL_STATE_TABLE`, `INSTALL_STEP_TABLE`, `INSTALL_TRANSITIONS`), driven by one service per process (`getLocalModelService()`). States: `idle`, `installing`, `paused`, `failed`, `installed`; steps in order: `runtime`, `model`, `verify`, `agent`, `connect` (`agent` is left out when the coding agent is installed). ⚠️ An event the transition table does not list is REFUSED by returning the same object, which the service answers with 409 `wrong-state`; add a transition to the table, never a special case to the service. ⚠️ ONE job at a time: verbs run on one queue, so two installs sent together cannot both find the state free. ⚠️ A job the state file still calls `installing` when the process has none was interrupted by a restart: it becomes `paused` the first time anything looks (`boot()`, a GET or a verb), and NOTHING continues without a click. `shutdown()` therefore writes nothing: it aborts the step and leaves `installing` in the file. ⚠️ An interrupted step is never recorded as failed, whatever its last call answered (an aborted fetch looks like a network error): the job checks its own signal before it writes `failed`. ⚠️ Every step is idempotent, because Resume and "Try again" re-enter the step the job stopped in. ⚠️ Cancel in the `connect` step is honoured only before the first write: once the two writes began they finish, so the pair is never left half done. ⚠️ The state file (`state-store.ts`) has ONE writer, serialized, temp file + rename, mode 0600, and reading is TOTAL (a file that does not parse, another schema, or a state that lacks a member it needs reads as idle): it holds nothing of the user, so it is replaced rather than refused, unlike `settings.json`. ⚠️ Progress is kept in memory and written at most every few seconds and when the job pauses or fails; the window reads the live figure. A 23 GB pull emits about 1.4 million lines, so neither the stream nor one state write per line is affordable. + +**Install is checked before anything is downloaded** (`LocalModelService.install()`): the model, the context and the coding agent must be a row the CURRENT recommendation offers (a model id is looked up, never passed on: without a slash it cannot name another registry host, and the request schema repeats that); the coding agent must be a harness row of the catalogue; `acknowledgedBytes` must equal the server's own total (`Candidate.installBytes`: the model plus the runtime archive while the runtime is not unpacked), else 409 `size-changed` with that total, so a consent is bound to the size that was shown; and a saved endpoint that already has the reserved id and is not ours is said now, not after a 23 GB download. + +**The check after the pull** (`verifyModel()`): one tiny chat with `reasoning_effort: "none"`, then `/api/ps` must show `size_vram == size` at the chosen context, then one tool probe (`tools`, `reasoning_effort: "none"`, `max_tokens` 1024) that passes on a tool call with parseable arguments. ⚠️ Thinking is switched off and the cap is generous on purpose: every catalogue model thinks first, and a probe that ends on its token limit proves nothing (`probe-unfinished`, never `no-tool-call`). ⚠️ A failure is a state with a code and ONE next step (`LOCAL_MODEL_ERROR_NEXT`), never a bare "failed". ⚠️ The job fails when the runtime reports no digest for the model: there would be nothing to compare later. + +**The coding agent** is installed by `runInstallCommand()` of `src/cli-install-runner.ts`, the runner extracted from `cli-registry-routes.ts` (whose route keeps its behaviour and re-exports `installEnv`/`npmGlobalPrefixWritable` for its tests). ⚠️ The job runs only `stockInstallCommand(id)`, read from the COMPILED `STOCK_CLIS`, never the registry merged with `clis.json` (the route's own command still comes from the merged registry, behind its `cliManagementEnabled` gate): this path has no such gate, so an override file must not be able to change what it runs. `HarnessFact.installCommand` is that same string, which is what the consent screen prints. ⚠️ Only for a harness row of the catalogue, only after the consent click, admin only, audited (`cli_install` with `via: 'local-model'`). ⚠️ ONE install slot per CLI id (`tryBeginCliInstall()`), shared by the route and the job. ⚠️ After the installer, successful or not, the resolver caches are forgotten (`forgetResolvedCli()`) and the binary is looked for again: a pre-install miss is negative-cached for up to five minutes, and "Start a session" would otherwise answer "not installed". Still missing is `harness-failed`; found is success whatever the exit code. The runner takes an abort signal that kills the installer's process GROUP. + +**The connection** (`connect.ts`): one record `{ id: 'local-ollama', label: 'This Mac', baseUrl, authStyle: 'bearer', models: [], defaultModelId, defaultContextLength: }`, validated by `CustomModelHostSchema` and written through `mutateCustomModelHosts()`, then `customModelEndpointsEnabled: true` through `mutateSettingsFile()`, with the flag's previous value kept in the state file (a second connect must hand the FIRST value back, or Remove could never switch the flag off again). ⚠️ `defaultContextLength` is load-bearing: the runtime's model listing states no context, so this field is what reaches the coding agent as its limit. ⚠️ The id is reserved: `POST /api/model-endpoints` refuses it, and a record with it that this feature did not write is never overwritten (`endpoint-id-taken`). ⚠️ While the state file records the connection, the record is MANAGED (`isManagedLocalEndpoint()`): `PUT`/`DELETE /api/model-endpoints/local-ollama` are refused, discover answers the stored list without a request, and the sweep skips it, so a model somebody pulled through the port never becomes launchable and the URL never leaves the supervised port. A hand-made record with that id (no state) stays an ordinary endpoint. `local-model.json` is not a config-backup file while the endpoint store and the settings are, so a restore can take the record away under an installed model: `boot()` puts the RECORD back when it is gone (`ensureLocalEndpoint()`, never replacing one that is there) and leaves the settings flag as it finds it. ⚠️ The runtime's port has NO authentication, and its CORS allowlist always admits `localhost` origins: any local process or localhost page can pull, create or delete models there. That is the reason for the three rules above and for the digest. + +**The digest** `/api/tags` reported after the check is stored. ⚠️ Before a session starts on the managed endpoint (`LocalModelService.launchRefusalMessage()`, asked by BOTH launch paths in `session-routes.ts`: `POST /api/sessions/:id/custom-model` and the quick-start `customModel` branch, before anything is written or created) and at boot it is read again: a different digest, a model that is gone, a runtime Codeman did not start, or a model id other than the installed one refuses the launch with one sentence. The view shows what blocks a session as `error` while the state stays `installed`; it is found live and never stored. Boot recovery of an already running session (`_rebuildCustomModelEnv`) is not guarded: the pane exists and keeps its environment. + +**Remove** stops the child, undoes the connection FIRST (a refused write of the endpoint store or of the settings rejects before any file is deleted), restores the flag only when this feature switched it on AND no other endpoint is saved, then deletes the runtime and the private home and, when asked, the models (`removeLocalModelFiles()`, which refuses any path outside the root). ⚠️ While a live session uses the endpoint the answer is 409 `sessions-open` with the session names (`sessionNamesUsingEndpoint()`, read at the moment of the request). ⚠️ The coding agent stays: what its installer wrote is not ours to remove. + +**Developer tools**: `checkDeveloperTools()` runs ONLY `/usr/bin/xcode-select -p`, bounded, on darwin. ⚠️ Never `git`, `python3` or `xcrun` from the server: on a Mac without the tools each opens a system window. The view carries `developerTools`, and the window prints the one sentence when it is `missing`. + +**The endpoint store's writer** (`src/custom-model-hosts-writer.ts`): `mutateCustomModelHosts(fn)` is the shape of `mutateSettingsFile()`: one promise chain, a random temp name, a refusal (409) to write over a file that does not parse. ⚠️ EVERY writer of `custom-model-hosts.json` goes through it: the four endpoint routes, the sweep and the connect step (`test/custom-model-hosts-writer.test.ts` fails a `writeCustomModelHosts()` call anywhere else in `src/`). The routes used to read, change and write on their own through one shared temp name, so two creates sent together lost a record or failed, and a file that did not parse was read as `[]` and replaced by one record, with every other endpoint's API key gone. ⚠️ A route's check runs INSIDE the callback, and discover splices onto the record as it is after the probe, not the copy read before it. + +Tests: `test/local-model-install-job.test.ts` (the tables and the reducer, every state with every event), `test/local-model-install-service.test.ts`, `test/local-model-install-e2e.test.ts` (the real modules over the fake runtime), `test/local-model-state-store.test.ts`, `test/local-model-connect.test.ts`, `test/custom-model-hosts-writer.test.ts`, `test/cli-install-runner.test.ts`, `test/routes/local-model-routes.test.ts`, `test/routes/local-model-launch-guard.test.ts`, and the runtime's own `test/local-model-runtime.test.ts`, `test/local-model-ollama-client.test.ts`. The fake runtime is `test/fixtures/fake-ollama.ts`. + ### Approvals Inbox **Approvals Inbox** (cross-session queue of prompts waiting on a human; `approvalsInboxEnabled`, SYNCED, default OFF: every surface is opt-in; only the store and answer endpoints run regardless, so flipping it ON shows anything already pending): `web/approval-inbox.ts` is a `sessionWaits`-style singleton fed by `/api/hook-event`, holding at most ONE item per session (a new prompt supersedes), in-memory; answerable items are claude-mode only, and the DeepSeek status bridge and codex's screen-read dialogs (`web/screen-blocked.ts`) open ALERT-ONLY items (`answerInTerminal`: the answer route refuses them BEFORE its hooks gate, push carries no `approvalId`, the card and overview strip say "answer in the terminal"). Cards are answered via `POST /api/approvals/:id/answer`, which sends a digit / Esc / idle-prompt text through `writeViaMux` (menu answers never carry `\r`). diff --git a/docs/cli-registry.md b/docs/cli-registry.md index 47fef4ec..c6911114 100644 --- a/docs/cli-registry.md +++ b/docs/cli-registry.md @@ -25,7 +25,7 @@ Every run mode Codeman can launch — Claude Code, Terminal/Shell, OpenCode, Cod App Settings → Agents & CLIs → **CLI management** (`cliManagementEnabled`, default OFF; admin-only in multi-user mode) lists every entry with an installed/not-installed badge and: - toggles any entry on or off. A `kind: 'shell'` entry cannot be disabled, and the row shows no switch for it. A disabled CLI disappears from the Run menu, the welcome screen and the phone overview, and new session requests for it are rejected. -- installs a missing **stock** CLI by running its shipped install command, after a confirm that names the exact command. Only one install per CLI runs at a time, and the command runs without any `CODEMAN_*` variable in its environment. An `npm install -g` command is pointed at `~/.local` (which every resolver searches) when the npm global prefix is not writable by the server user, for example a system node under `/usr`; a writable prefix, a prefix that does not exist yet but could be created, and an explicit `NPM_CONFIG_PREFIX` are left alone. A custom entry's install command is never executed. +- installs a missing **stock** CLI by running its shipped install command (the entry as merged with `clis.json`, so an override of a stock entry's install command is what runs here), after a confirm that names the exact command. Only one install per CLI runs at a time, and the command runs without any `CODEMAN_*` variable in its environment. An `npm install -g` command is pointed at `~/.local` (which every resolver searches) when the npm global prefix is not writable by the server user, for example a system node under `/usr`; a writable prefix, a prefix that does not exist yet but could be created, and an explicit `NPM_CONFIG_PREFIX` are left alone. A custom entry's install command is never executed. - adds, edits and deletes **custom** entries (id, label, badge, binaries, launch argv). The server re-validates the whole assembled entry through `CliEntrySchema`, so the form cannot bypass the load-time rules. These are the only writes to `clis.json`. They are serialized, and a file that does not parse or has unsafe permissions is refused rather than overwritten; fix it (or `chmod 600` it) and retry. The HTTP routes are listed in `docs/api-reference.md` under *CLI management*. @@ -332,7 +332,7 @@ already have. Three rules, and the middle one is why the embed matters: -1. **The server never executes an entry's `install.command`.** Unchanged, and still enforced by nothing executing it: the field is display text (`CliDiscovery.install.command`). +1. **The server executes an entry's `install.command` in exactly two places, both behind a consent click that shows the literal text.** CLI management (above, `cliManagementEnabled`, the merged entry's command) and the local model install (`docs/local-models.md`: only the COMPILED stock command of a catalogue harness row, through `stockInstallCommand()`, so no `clis.json` override reaches it; no setting gates that path, the consent screen does). Both run through the one runner in `src/cli-install-runner.ts` (one install per CLI at a time, no `CODEMAN_*` in the environment, the whole process group killed on a timeout or a cancel) and both are written to the admin audit log in multi-user mode. Everywhere else the field is display text (`CliDiscovery.install.command`). 2. **`install.sh` executes only commands embedded in itself.** Those arrive in the same file, over the same TLS fetch, in the same commit as the `curl \| bash` line that fetched the script — identical trust to the hardcoded vendor one-liners it replaces. 3. **Nothing fetched at install time is ever executed.** There is no second code path that fetches anything after the script itself has been fetched. diff --git a/docs/custom-model-endpoints.md b/docs/custom-model-endpoints.md index 78001ed4..31f5de00 100644 --- a/docs/custom-model-endpoints.md +++ b/docs/custom-model-endpoints.md @@ -663,6 +663,45 @@ each result. `scripts/test-local-llm-harnesses.ts` is the standalone script used to check a harness against a real endpoint outside the web UI entirely; see its own `--help` for usage. +## The endpoint store has one writer + +`~/.codeman/custom-model-hosts.json` is written only through `mutateCustomModelHosts()` +(`src/custom-model-hosts-writer.ts`): the create, update, delete and discover routes, the +five-minute re-discovery sweep and the local model install all go through it. + +- **One at a time.** Changes run on one queue, and the check a route makes ("this id is + free", "this endpoint still exists") runs in the same step as its write. Before, each + writer read the file, changed its copy and wrote it back on its own, all through one temp + file name: a route and the sweep, or two requests sent together, could lose each other's + change or fail outright. +- **Atomic.** A temp file with a random name beside the target, then a rename. Mode 0600. +- **A file that does not parse is refused, not replaced.** The writing routes answer + `409 CONFLICT` with the fix (repair or delete the file, or restore it from a config + backup) and leave it byte for byte. Before, such a file read as an empty list, and the next + save replaced every stored endpoint and its API key with one record. The message never + quotes the file, which holds keys. + +The one direct writer left is config-backup restore, which must work on a file that does +not parse. + +## The endpoint of the model on this Mac + +Installing a local model from Codeman ([local-models.md](local-models.md)) saves one endpoint +with the reserved id `local-ollama` and the label "This Mac". It shows up in the list and in +the Run menu like any other endpoint. Three things are different: + +- A new endpoint cannot be given the id `local-ollama`. +- While it is the endpoint the install manages, it cannot be edited or deleted here, and + "Discover" answers its stored model list without asking the runtime. Its model list must + stay the one model the install checked, and its address the port of the runtime Codeman + supervises. Change or remove it in the local model window. +- A session is started on it only while Codeman's own runtime is running and the model is + still the one that was installed (its digest is compared). Otherwise the launch answers + with one sentence. + +An endpoint with that id that somebody made by hand, with no model installed by Codeman, is +an ordinary endpoint: it can be edited and deleted as before. + ## Security note Every env var this feature can set that redirects a session's traffic diff --git a/docs/local-models.md b/docs/local-models.md new file mode 100644 index 00000000..86c9f98a --- /dev/null +++ b/docs/local-models.md @@ -0,0 +1,267 @@ +# A model that runs on this Mac + +Codeman can set up a coding model that runs on the Mac the server runs on: no account, no +API key, nothing leaves the machine. One button on the welcome screen opens a window that +shows what fits this Mac, installs the choice and ends on a coding agent session that +runs on it. + +**Experimental.** The routes and their payloads may change in any release +([versioning-policy.md](versioning-policy.md), "Experimental local model routes"). Apple +silicon macOS only: anywhere else the window says why nothing is offered. + +This page is the reference for how it works. The HTTP contract is in +[api-reference.md](api-reference.md) ("Local model"). The rules that must not be broken are +in [architecture-invariants.md](architecture-invariants.md#local-model-install). + +## What an install does, in order + +The person picks a model, presses "Install on this Mac" and confirms the sizes once. The +server then runs one job of five steps. The window polls `GET /api/local-model` once a +second; the job runs in the server, so the window can be closed. + +| Step | What it does | +| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `runtime` | Downloads the pinned runtime, compares its hash, unpacks it, chooses a port once, starts it, reads the measured memory budget and judges the choice again | +| `model` | Downloads the model through the runtime | +| `verify` | Loads the model and checks it: one tiny chat, the memory and context the runtime reports, one tool call | +| `agent` | Installs the coding agent with its stock install command. Left out when the agent is installed | +| `connect` | Saves one model endpoint and switches the custom model endpoints setting on | + +Every step can be repeated: Resume and "Try again" enter the step the job stopped in. + +### The runtime + +- **One pinned build.** Ollama 0.40.15 for macOS, a compiled constant + (`OLLAMA_RUNTIME_PIN` in `src/local-model/runtime.ts`): the URL, the size (185,971,671 + bytes) and the sha256. No file, setting, environment variable or request can change which + binary is downloaded. +- **Download.** Streamed to `*.partial` while it is hashed. A cut stream is continued with a + Range request, at most three times. The hash is compared before anything is unpacked, and + a mismatch deletes the file. Unpacking is `/usr/bin/tar` into a temporary folder, then one + rename. +- **No administrator rights.** Everything lives in the data directory. +- **Supervised by Codeman.** The runtime is a non-detached child of the server: started at + boot once a model is installed and not stopped, restarted at most three times after a + crash, sent SIGTERM at shutdown. It is not a login item: it stops with Codeman, and a + leftover from a server that was killed is taken back at the next start (see below). +- **Its environment is an allowlist.** `PATH`, `TMPDIR`, `LANG` and the proxy variables are + copied from the server. `HOME` is a private folder. Exactly these are set: + `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, + `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, + `OLLAMA_MAX_LOADED_MODELS=1`. No `CODEMAN_*` variable and none of the user's own + `OLLAMA_*` settings reach it. +- **The port is chosen once**, by binding, from 11435 across 20 ports, and stored. 11434 is + left alone: it is the default of a runtime the user may run themselves, which is never + touched. +- **An orphan is recognized.** The pid and the port are stored. At boot a listener on the + port that answers `/api/version` and whose pid runs a binary under Codeman's runtime folder + is Codeman's own leftover from a server that was killed: it is adopted or replaced. Any + other listener is "Port 11435 is in use by another program", with "Use another port" as + the way out. + +### The memory budget is measured + +Before the runtime exists the budget is an estimate from the installed memory. Right after +it starts, the runtime prints the device total in its log line "inference compute" +(`total="37.4 GiB"` on a 48 GB MacBook). Codeman reads that line, stores the figure and +judges the chosen model against it before a byte of the model is downloaded. A model that no +longer fits ends the job there with "does not fit" and the next model to try. From then on +the window shows the measured budget and the word "estimated" is gone. + +The need of a model is its peak: the weights, what stays held per token of context, what the +runner holds beside both, and what it takes for a moment while a long prompt is processed. + +### The model download + +`POST /api/pull` with `stream: true`. A 23 GB pull sends about 1.4 million progress lines, +so the stream is parsed piece by piece and never kept. Cancel aborts the request, which +stops the download. Resume pulls again and the runtime continues its partial files. An error +line or a cut stream is retried up to five times. Once the pull reported its total, every +size shown for the model is that total. + +### The check + +After the pull, in this order: + +1. One tiny chat with `reasoning_effort: "none"`. It loads the model. +2. `GET /api/ps` must show the whole model on the GPU (`size_vram` equal to `size`) at the + chosen context length. +3. One tool probe on `/v1/chat/completions` (a `tools` array, `reasoning_effort: "none"`, + `max_tokens` 1024). It passes on a tool call whose arguments parse. + +A failure is a state with a code and one next step, never a bare "failed": `does-not-fit` +and `no-tool-call` offer the next model, `probe-unfinished` (the token limit ended the +probe, so it proves nothing) and `load-failed` offer "Try again". The digest the runtime +reports for the model is stored when the check passed. + +### The coding agent + +When the chosen coding agent is not installed, the job installs it with the CLI registry's +own stock install command, through the same runner as App Settings > CLIs +(`src/cli-install-runner.ts`). The rules: + +- Only a CLI that is a harness row of the local model catalogue. +- Only the **compiled** stock command. A `~/.codeman/clis.json` override cannot change what + runs here. +- Only after the person's consent click, on a screen that showed the literal command. +- Admin only in multi-user mode, and written to the admin audit log (`cli_install`, with + `via: "local-model"`). +- One installer per CLI at a time, shared with App Settings: while one runs, the other + answers that it is busy. +- Afterwards the cached lookups of the CLI's binary are forgotten and the binary is looked + for again. Still missing is the failure `harness-failed`, with the installer's last lines. + +This step does not need the `cliManagementEnabled` setting. That setting gates the general +CLI management surface; this path can only ever run one vetted command for one catalogue +row. + +### The connection + +Two writes outside the local model folder, both undone by Remove: + +1. One record in `custom-model-hosts.json`: + + ```json + { + "id": "local-ollama", + "label": "This Mac", + "baseUrl": "http://127.0.0.1:11435", + "authStyle": "bearer", + "models": ["qwen3.6:35b-a3b"], + "defaultModelId": "qwen3.6:35b-a3b", + "defaultContextLength": 131072 + } + ``` + + `defaultContextLength` is the chosen context: the runtime's model listing states none, so + this field is what reaches the coding agent as its context limit. + +2. `customModelEndpointsEnabled: true` in `settings.json`. The value it had before is kept + in the state file. + +The id `local-ollama` is reserved. A new endpoint cannot take it, and a record with that id +that this feature did not write is never overwritten: the install says so before anything is +downloaded. While the record is the managed one it cannot be edited or deleted in App +Settings and the five-minute discovery sweep leaves it alone, so its model list stays the +one model the install checked. + +Switching the setting on also starts that five-minute sweep for every other saved endpoint. + +## Starting a session + +"Start a session" starts the coding agent on the endpoint `local-ollama` through the +existing custom model launch. Before a coding agent is STARTED on that endpoint the server +checks, in this order: + +- the runtime child Codeman started is alive (whatever else answers on the port is not + ours); +- the digest the runtime reports for the installed model is the one stored at install time; +- the model asked for is the one that was installed and checked. + +That check runs wherever a new agent process comes to be on the endpoint: a session created +with the model, a model applied to a running session (which restarts its agent), the attach +that starts an agent in a session that has none or whose agent exited, and the rebuild of a +session after a reboot (that entry is skipped with "the local model is not ready" and stays +on offer). An agent that is already running when Codeman restarts is not started by +Codeman, so it is not asked: it keeps running on its port, see below. + +A different digest refuses the launch: "The model on this Mac changed since it was +installed. Remove it and install it again." The digest is also read again at boot and when +the runtime is started. + +## Stop, Start, Remove + +- **Stop** stops the runtime and keeps it from starting at the next boot. **Start** undoes + that. +- **Remove** stops the runtime, removes the endpoint record, switches the setting off again + only when this feature switched it on and no other endpoint is saved, and deletes the + runtime and its private home. The downloaded model is deleted only when asked. The coding + agent stays. While a live session uses the model, Remove answers with the session names + instead. + +## Where everything is + +All under the data directory (`~/.codeman`, or the instance's own): + +``` +local-model.json the state file (mode 0600) +local-model/ + runtime/ollama-0.40.15.tgz.partial the runtime archive while it downloads + runtime/ollama-0.40.15/ the unpacked runtime (about 579 MB) + models/ the model store + home/ the runtime's private HOME (its key pair is here) +``` + +Nothing is written or deleted outside `local-model/` except the state file beside it and the +two writes of the connection. The state file holds no path, so it can never point a delete +somewhere else. The user's own `~/.ollama` is never created, read or removed. + +`local-model.json` is not part of a config backup. `custom-model-hosts.json` and +`settings.json` are, so restoring an older backup can take the endpoint record away while +the model stays installed. The record is put back the next time Codeman starts (the setting +is left as the backup had it). A restore can also bring a record back after a Remove: it +then points at a port nothing listens on, is an ordinary endpoint again, and can be deleted +in App Settings. + +## The job's states + +| State | A server restart | Cancel | What can be done | +| ------------ | ------------------------------------------------------------- | ---------------- | -------------------------------------- | +| `idle` | stays idle | refused | install | +| `installing` | becomes `paused` at the same step | becomes `paused` | cancel | +| `paused` | stays paused | refused | resume, install, remove | +| `failed` | stays failed, same error | refused | resume, install, remove | +| `installed` | stays installed; the runtime is started unless it was stopped | refused | start a session, stop or start, remove | + +One job at a time. A job a server restart interrupted comes back paused and never continues +without a click. Cancel interrupts the step in flight (the download, the pull, the request, +the installer's process group); in the last step it is honoured before the first write, and +once the two writes began they finish. + +## On a Mac without developer tools + +Coding agents call `git` and `python3`. On a Mac without Xcode or the Command Line Tools +those are stubs that open macOS's "install the command line developer tools" window. So the +window says one sentence wherever the person commits or finishes, and only when +`/usr/bin/xcode-select -p` fails: + +> Coding agents use git and Python. This Mac has no developer tools yet, so macOS will offer +> to install them the first time an agent needs one. + +Codeman itself only ever runs `/usr/bin/xcode-select -p` for this, bounded, and never `git` +or `python3`. + +## What to know before relying on it + +- **The runtime's port has no login.** Any program on this Mac, and any web page served from + `localhost`, can use the model server and change its models. That is why the endpoint + record lists one model, why the sweep leaves it alone, and why the digest is compared + before a session starts. +- **A Codeman restart stops the runtime.** A session that is mid-answer on the local model + loses that answer. The runtime is started again with the server, and the first request + loads the model again (seconds). +- **An open session keeps calling the runtime's port.** The coding agent was given the + address when it started. While the runtime is stopped, or for the seconds of a Codeman + restart, its requests go to a port nothing of ours holds, and any program on this Mac could + bind that port and answer them. That is why Stop and "Use another port" are refused while + a session uses the model: close those sessions first. +- **The runtime does not enforce the context length.** A prompt longer than the context was + accepted in testing. Do not rely on the runtime to stop one. +- **The model stays in memory for ten minutes** after the last request, then the runtime + unloads it. + +## Source + +| What | Where | +| ---------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| Hardware probe, catalogue, recommendation | `src/local-model/hardware.ts`, `src/config/local-model-catalog.ts`, `src/local-model/recommend.ts` | +| Runtime pin, download, environment, supervisor | `src/local-model/runtime.ts` | +| Runtime HTTP client, pull parser, checks | `src/local-model/ollama-client.ts` | +| Developer tools check | `src/local-model/developer-tools.ts` | +| Job (state table, reducer, view, service) | `src/local-model/install-job.ts` | +| State file | `src/local-model/state-store.ts` | +| Connection and its undoing | `src/local-model/connect.ts` | +| Paths | `src/local-model/paths.ts` | +| Routes | `src/web/routes/local-model-routes.ts` | +| Shared CLI install runner | `src/cli-install-runner.ts` | +| Serialized writer of the endpoint store | `src/custom-model-hosts-writer.ts` | diff --git a/docs/security-architecture.md b/docs/security-architecture.md index 8fb4a284..a3e42ea1 100644 --- a/docs/security-architecture.md +++ b/docs/security-architecture.md @@ -594,6 +594,15 @@ Opt-in and off by default: the server POSTs the Web Push events (permission prom - **Discord bridge** (`src/discord-bridge.ts`, [setup](discord-bridge.md)): the bot token lives in `~/.codeman/discord.json` (0600, tmp+rename) and is never returned by the Discord routes, logged or put in an error, and the attachment, folder and download routes refuse the file. A session whose folder is the home directory can still read it through its own file viewer, which serves that workspace without the blocklist. Inbound replies are typed into a session only from `allowedUserIds`, in the configured guild, inside a thread the bridge created; anything else is dropped unread. A reply becomes one printable line plus Enter, capped at 1000 characters and one per second per tab. Posts send `allowed_mentions` limited to the allowlisted ids, so agent output cannot ping a server. **Files:** an attachment is fetched only from Discord's own HTTPS hosts (no redirects followed), at most 5 per reply and 8 MB each, saved 0600 under `/.codeman-discord/` with the name reduced to `[A-Za-z0-9._-]`; a file a CLI sends must be a regular file whose real path (symlinks followed) lies inside its own tab's folder and is not on the attachment blocklist (`isBlockedAttachmentPath`, since a tab's folder can be the home directory), which stops a sensitive file being uploaded by mistake, and remote sessions send and receive none. A received file is untrusted content to whatever opens it. **What the checks do not stop:** the agent runs as the same OS user, so the path check prevents accidental uploads, not a determined agent that pastes a file's contents into the message. **What leaves the machine:** a permission or question alert sends the rendered screen (whatever the agent printed), and a finished turn or an error sends the CLI's whole last message, read from its transcript (up to 8 posts of 1,900 characters). `idle_prompt` is a warning, so with the default scope every Claude turn left unanswered for a minute is posted, and with **Which events** set to Everything every turn end is; treat the Discord server as holding that output. **Multi-user:** the bridge is one allowlist for the whole server, so an allowlisted Discord account can type into every user's sessions; only an admin can configure it. **Startup hint:** a fixed text (no user input) is added to Claude's system prompt; no other CLI is sent it. The bridge only dials out (REST and the gateway WebSocket), so it needs no inbound exposure. A Discord account on the allowlist can drive agents that run shell commands: it needs 2FA. - **It carries session data to a third party.** Titles and bodies include session names, tool names and error text, all agent- or user-controlled, so Discord gets `allowed_mentions: { parse: [] }` and Slack's `& < >` are escaped: agent output cannot ping a channel. In multi-user mode all three routes are admin-only and the channel is instance-wide: it receives every user's session events, the same reach an admin's own Web Push has, which means non-admins' session details leave the box at the admin's choice. +## 10d. The model on this Mac (local model install) + +Opt-in by a click, macOS only, admin-only in multi-user mode on every verb. The server downloads one pinned runtime build, runs it as its own child, pulls one model into it, installs the coding agent when it is missing, and saves one endpoint record. Source: `src/local-model/`, routes in `src/web/routes/local-model-routes.ts`. Guide: [`local-models.md`](local-models.md). + +- **Supply chain: a compiled pin.** The runtime's URL, size and sha256 are a constant in `src/local-model/runtime.ts` that no file, setting or request can change; the archive is hashed while it streams and compared BEFORE anything is unpacked, a mismatch deletes it, and only `/usr/bin/tar` unpacks it into a folder under the data dir. The model comes from the runtime's own registry under the name the catalogue pins; the digest the runtime reports after the pull is stored, and a model whose digest changed under that name is refused at the next start. +- **A loopback port without a login.** The runtime listens on `127.0.0.1:` (chosen once from 11435) with no authentication, like every model server of its kind: any program of any account on the Mac, and any web page served from `localhost`, can use it and change its models. What limits that: the endpoint record lists ONE model and the discovery sweep leaves it alone, the digest is compared before an agent is started on it (every start path, see the guide), and a listener on the port that is not our own child (judged by the stored pid's command line, never by the port alone) is reported as `port-in-use` and never signalled. An open session keeps calling the port while the runtime is stopped, which is why Stop and a port change are refused while one uses the model. +- **A second install path for a coding agent, without `cliManagementEnabled`.** The job runs the COMPILED stock install command of a catalogue harness row (never a `clis.json` override, never a custom entry) through the same runner as CLI management, only after the consent screen showed the literal command, and writes the admin audit entry (`cli_install`, `via: 'local-model'`). The installer runs as the server's user with the same reach as every other spawn. +- **The runtime's own files stay inside `~/.codeman/local-model/`**: its private HOME (so no `~/.ollama` is created), its model store and its unpacked folder; its environment is an allowlist (no `CODEMAN_*`). Remove deletes only inside that folder, refuses a folder that is a symbolic link, and ends the runtime (ours by its command line) before any file goes. + --- ## 11. Quick reference diff --git a/docs/versioning-policy.md b/docs/versioning-policy.md index ed6dd809..ffd7b0cf 100644 --- a/docs/versioning-policy.md +++ b/docs/versioning-policy.md @@ -86,18 +86,25 @@ covered like any other stable behavior. Other agent commands and API routes keep their existing guarantees. Promotion to stable requires an explicit documented decision. -### Experimental local model recommendation +### Experimental local model routes -`GET /api/v1/local-model` (and its `/api` alias) is labeled experimental, so item 3 -above applies to it: its path, its payload (the hardware facts, the verdicts and -reasons, the candidate fields and the English sentences in `headline` and -`notes`) and its behavior may change or be removed in a MINOR or PATCH release. -It is read-only today: it looks at the Mac the server runs on and says which -local model fits. The step that sets a model up from Codeman is not built yet -and will change what the route reports, which is why the shape is not promised. -It is not behind a setting (it writes nothing and adds no settings key); it is -admin only in multi-user mode. Promotion to stable requires an explicit -documented decision. +`GET /api/v1/local-model`, `POST /api/v1/local-model/install`, `/cancel`, `/resume` +and `/runtime`, and `DELETE /api/v1/local-model` (and their `/api` aliases) are +labeled experimental, so item 3 above applies to them: their paths, their payloads +(the hardware facts, the verdicts and reasons, the candidate fields, the `install` +object with its states, steps, error codes and actions, the 409 conflict shape and +every English sentence) and their behavior may change or be removed in a MINOR or +PATCH release. They look at the Mac the server runs on, say which local model fits, +and install, start, stop and remove one ([local-models.md](local-models.md)). + +They are not behind a setting. Reading changes nothing; an install starts only on an +explicit request that carries the size the person confirmed. All of them are admin +only in multi-user mode. What an install leaves behind is ordinary, stable state: one +saved model endpoint with the id `local-ollama` and the `customModelEndpointsEnabled` +setting switched on, both of which the stable custom model endpoint routes read like +any other endpoint. On-disk state (`local-model.json`, the `local-model/` folder) is +internal, as for every other file in the data directory. Promotion to stable requires +an explicit documented decision. ## Deprecation policy diff --git a/docs/wiki/Custom-Model-Endpoints.md b/docs/wiki/Custom-Model-Endpoints.md index a7e4f0d6..4ee5aa59 100644 --- a/docs/wiki/Custom-Model-Endpoints.md +++ b/docs/wiki/Custom-Model-Endpoints.md @@ -204,6 +204,45 @@ Which harnesses show up in the Run-menu picker is read live off Codeman's own CL not a fixed list here, so this table can go stale before this page does — a greyed-out or missing entry is the more current answer. +## A model that runs on this Mac + +If the Codeman server runs on a Mac with Apple silicon, you do not need a server of your +own: Codeman can set one model up for you on that Mac. This is experimental. + +1. Press **Run a local model** on the welcome screen (or open App Settings > Models > Model + on this Mac). +2. The window shows what fits this Mac. A bar shows how much of the memory the selected + model takes at its full context; a nearly full bar means a tight fit. Pick a model and + the coding agent to run it with. +3. Press **Install on this Mac**, check the sizes and where they go, and confirm. Nothing is + downloaded before this click. +4. Codeman downloads a model runtime (Ollama, about 186 MB) and the model into + `~/.codeman/local-model`, loads the model, checks that all of it runs on the GPU and that + it can call a tool, installs the coding agent if it is missing, and saves an endpoint + named **This Mac**. No administrator password is asked for. +5. Press **Start a session**. + +Good to know: + +- You can close the window: the download continues while Codeman is running. **Cancel** + pauses it and **Continue** picks it up where it stopped. After a Codeman restart an + unfinished install waits for your click. +- The memory figure is an estimate until the runtime has started once. From then on it is + what the runtime itself measured, and the model is judged against that before it is + downloaded. +- The endpoint "This Mac" belongs to this install. It is in the list and in the Run menu, + but it is changed and removed in the local model window, not here. +- Codeman starts the runtime with the server and stops it with it. **Stop** keeps it from + starting next time. +- **Remove** deletes the runtime and the endpoint, and the model too if you tick that. The + coding agent stays installed. +- If the model on disk is no longer the one that was installed, Codeman refuses to start a + session on it and tells you to remove and install it again. +- Any program on this Mac, and any web page served from `localhost`, can reach the model + runtime: its port has no login. It listens on this Mac only. +- On a Mac without Apple's developer tools, macOS will offer to install them the first time + a coding agent runs `git` or Python. The window tells you when that applies. + ## What it does not do - **No remote or Docker sessions yet.** Both restart their agent differently under the hood diff --git a/docs/wiki/Settings-Reference.md b/docs/wiki/Settings-Reference.md index 0ffcdf52..a88b55ad 100644 --- a/docs/wiki/Settings-Reference.md +++ b/docs/wiki/Settings-Reference.md @@ -193,19 +193,25 @@ attached. section to the Run dropdown, for pointing a harness at your own OpenAI-compatible server instead of its native cloud backend. See [Custom Model Endpoints](Custom-Model-Endpoints). -**Model on this Mac** is not a setting: it is a read-only answer. When the Codeman server -runs on a Mac with Apple silicon, it shows that Mac's chip, memory and free disk space, the -local coding model that fits it best with the agent CLI to run it, what the model needs of -the memory the Mac can give one, its download size and an estimated speed, and the other -models that fit (Good or Tight for the memory, Slow or Speed unknown for the speed). A Mac -that can only run a model under 12 billion parameters is told so in plain words, and a Mac -with too little memory, too little free disk space or an Intel processor is told why nothing -is offered. The numbers are estimates from a catalogue that ships with each release; nothing -has been tested on your Mac. **Nothing is downloaded or installed from here yet**: setting -the model up from Codeman is the next step and is not built. The same answer appears as a -card on the welcome screen, which you can hide per device (it stays here). The group is not -shown when the server runs on Linux or Windows, and in multi-user mode only an admin sees -it. The "Mac" is always the machine the server runs on, not the device you are reading on. +**Model on this Mac** is not a setting: it is the state of a model Codeman can run for you +on the Mac the server runs on, with a button that opens the local model window. The same +window opens from the **Run a local model** button on the welcome screen. It shows that +Mac's chip, memory and free disk space, a bar for how much of the memory the selected model +takes, and the models that fit, each with one tag (Best fit, Tight, Slow, Small model, or +"Tested on a Mac like this"), a speed and a download size. **Install on this Mac** asks you +to confirm the sizes once, then downloads a model runtime and the model into +`~/.codeman/local-model`, checks that the model loads and can call a tool, installs the +coding agent if it is missing, and ends on **Start a session**. No account, no API key and +no administrator password. You can close the window while it downloads, cancel and continue +later; after a Codeman restart an unfinished install waits for you to press Continue. Once a +model is installed the window offers Start a session, Stop or Start, and Remove. Installing +switches **Custom model endpoints** on and saves one endpoint named "This Mac"; Remove takes +the endpoint away again and switches the setting back off if nothing else uses it. A Mac +that can only run a small model is told so in plain words, and a Mac with too little memory, +too little free disk space or an Intel processor is told why nothing is offered. The group +is not shown when the server runs on Linux or Windows, and in multi-user mode only an admin +sees it. The "Mac" is always the machine the server runs on, not the device you are reading +on. Details: [Custom Model Endpoints](Custom-Model-Endpoints#a-model-that-runs-on-this-mac). ### Agents & CLIs diff --git a/src/cli-install-runner.ts b/src/cli-install-runner.ts new file mode 100644 index 00000000..b58ba2a4 --- /dev/null +++ b/src/cli-install-runner.ts @@ -0,0 +1,311 @@ +/** + * @fileoverview The shared runner of a STOCK CLI's install command: the one place Codeman + * executes an install one-liner, used by `POST /api/clis/:id/install` + * (src/web/routes/cli-registry-routes.ts) and by the local model install job. + * + * The code was moved here from cli-registry-routes.ts, and that route keeps its behaviour + * byte for byte (same answers, same audit entry, same 409 for a second install of one id, + * same environment). + * + * What lives here: + * - `installEnv()` and `npmGlobalPrefixWritable()`: the installer's environment is the + * server's minus every `CODEMAN_*` variable, with npm's global prefix pointed at a place + * the server user can write. test/routes/cli-registry-routes.test.ts imports both from the + * route module, which re-exports them from here. + * - `runInstallCommand()`: `shell: true`, its own process group, output capped, the whole + * group killed on timeout. New: an abort signal that kills the group the same way, so the + * install job's Cancel has something to stop. + * - `forgetResolvedCli()`: after an install, successful or not, every cached lookup of the + * CLI's binaries is dropped, or a pre-install miss is replayed for up to five minutes. + * - The set of ids with an install running. ONE set for both callers + * (`tryBeginCliInstall()`), so the route and the job can never run two installers of one + * CLI over the same prefix. + * + * Rules: + * - Which command runs is the CALLER's choice, and the two callers choose differently. The + * local model job passes `stockInstallCommand()`, which reads the compiled `STOCK_CLIS` + * and never the registry merged with `clis.json`, so an override file cannot change what + * that job runs. The CLI management route passes the merged entry's command + * (`resolveInstallCommandForPlatform(listClis()...)`), as it did before this module + * existed: a `clis.json` override of a stock entry's install command IS what that route + * runs (behind `cliManagementEnabled`, after a confirm that shows the exact text). A custom + * entry's install text is never executed by either. Switching the route to + * `stockInstallCommand()` would close that gap at the cost of a behaviour change the route + * tests pin; it is an owner decision, not made here. + * - Who may call it is the caller's rule, not this module's: the route keeps its + * `cliManagementEnabled` gate; the install job runs it only for a harness row of the local + * model catalogue, after the person's consent click on a screen that showed the literal + * command, admin only in multi-user mode, and writes the admin audit entry itself. + */ + +import { execFile, spawn } from 'node:child_process'; +import { constants as fsConstants } from 'node:fs'; +import { access } from 'node:fs/promises'; +import { dirname, join } from 'node:path'; +import { promisify } from 'node:util'; +import { resolveInstallCommandForPlatform } from './config/cli-registry/registry.js'; +import { STOCK_CLIS } from './config/cli-registry/stock.js'; +import { getErrorMessage } from './types.js'; +import { invalidateCliExecutableResolvers } from './utils/cli-executable-resolver.js'; +import { invalidateCliResolverCache } from './utils/cli-resolver.js'; + +/** How long an install command may run before its process group is killed. */ +export const CLI_INSTALL_TIMEOUT_MS = 300_000; + +/** How an install command ended. */ +export interface CliInstallResult { + /** The exit code, `null` when the command was killed or could not be started. */ + code: number | null; + /** stdout and stderr together, capped. */ + output: string; + /** The timeout killed it. */ + timedOut: boolean; + /** The abort signal killed it. */ + aborted: boolean; +} + +export interface RunInstallCommandOptions { + /** Kills the installer's whole process group (SIGTERM, then SIGKILL). */ + signal?: AbortSignal; +} + +/** + * Ids with an install running right now. A second request for the same id gets 409 rather + * than a second `curl | bash` or `npm install -g` racing the first over the same prefix. + */ +const installsInFlight = new Set(); + +const execFileAsync = promisify(execFile); + +/** + * True when `npm install -g` can write to this process's npm global prefix, or when that cannot + * be determined (then nothing is redirected: a wrong guess would move installs somewhere the + * user did not choose). `npm config get prefix` is asked rather than guessed from `process.execPath` + * because a user `.npmrc` / `NPM_CONFIG_PREFIX` can point it anywhere. + * + * Async so the server keeps serving while npm boots (130 to 240 ms), and killed with SIGKILL on + * timeout because `SIGTERM` alone leaves the wait running. A prefix that does not exist yet is + * judged by the nearest ancestor that does: npm creates the missing directories, so a user + * `.npmrc` pointing at `~/.npm-global` before it was made is not moved. + */ +export async function npmGlobalPrefixWritable(source: NodeJS.ProcessEnv): Promise { + try { + const { stdout } = await execFileAsync('npm', ['config', 'get', 'prefix'], { + env: source, + encoding: 'utf8', + timeout: 5_000, + killSignal: 'SIGKILL', + }); + const prefix = stdout.trim(); + if (!prefix) return true; + // npm creates lib/node_modules under the prefix; walk up to the first directory that exists. + let dir = join(prefix, 'lib', 'node_modules'); + for (;;) { + try { + await access(dir, fsConstants.W_OK); + return true; + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') return false; + const parent = dirname(dir); + if (parent === dir) return false; + dir = parent; + } + } + } catch { + return true; + } +} + +/** + * The server's environment minus every `CODEMAN_*` variable. An install script is third-party + * code, and those variables carry Codeman's own secrets and wiring (`CODEMAN_PASSWORD`, the + * data dir, the tmux socket), none of which an installer needs. + * + * It also points `NPM_CONFIG_PREFIX` at `$HOME/.local` so an `npm install -g` lands somewhere the + * server user can write and Codeman's resolvers already search (`~/.local/bin`): + * - inside the Docker Compose container (`CODEMAN_IN_CONTAINER=1`), so installs survive an image + * update (the image's own prefix is image content); + * - on a native install whose npm global prefix is not writable by the server user (a system node + * under `/usr`, installed by root). Without this `npm install -g` died with EACCES (exit 243), + * e.g. DeepSeek's `npm install -g @deepseek-ai/dsh`. An explicit `NPM_CONFIG_PREFIX` the + * operator set is respected, and so is a prefix that is writable (nvm, `~/.npm-global`, ...). + */ +export function installEnv( + source: NodeJS.ProcessEnv = process.env, + prefixWritable: (env: NodeJS.ProcessEnv) => boolean = () => true +): NodeJS.ProcessEnv { + const env: NodeJS.ProcessEnv = {}; + for (const [key, value] of Object.entries(source)) { + if (!key.startsWith('CODEMAN_')) env[key] = value; + } + // ⚠️ In the Docker Compose deployment the image sets NPM_CONFIG_PREFIX=/opt/codeman-cli, which is IMAGE + // content: `Update-Codeman.sh` recreates the container and every CLI installed there (dsh, pi, ...) + // vanishes. HOME is the persistent bind mount and `~/.local/bin` is already on every resolver's search + // list, so npm-based installs are redirected there. curl|bash installers already target HOME. + if (source.CODEMAN_IN_CONTAINER === '1' && source.HOME) { + redirectNpmPrefix(env, source.HOME); + } else if (process.platform !== 'win32' && source.HOME && !source.NPM_CONFIG_PREFIX && !prefixWritable(env)) { + redirectNpmPrefix(env, source.HOME); + } + return env; +} + +/** + * Point npm at `$HOME/.local`, dropping every spelling of the prefix key first. `npm run` exports a + * lowercase `npm_config_prefix`, npm reads `npm_config_*` case-insensitively, and when both spellings + * are present a `/bin/sh` that sorts its environment (bash) lets the older value win. The explicit + * operator guard in `installEnv` stays on the uppercase key only: npm always injects the lowercase one. + */ +function redirectNpmPrefix(env: NodeJS.ProcessEnv, home: string): void { + for (const key of Object.keys(env)) if (/^npm_config_prefix$/i.test(key)) delete env[key]; + env.NPM_CONFIG_PREFIX = `${home}/.local`; +} + +/** + * `installEnv` for this process, with the (async) npm prefix probe done first and only when the + * command runs npm at all: a `curl | bash` installer never pays for it. + */ +async function installEnvFor(command: string, source: NodeJS.ProcessEnv = process.env): Promise { + const usesNpm = /\bnpm\b/.test(command); + const probeNeeded = + usesNpm && + source.CODEMAN_IN_CONTAINER !== '1' && + process.platform !== 'win32' && + !!source.HOME && + !source.NPM_CONFIG_PREFIX; + const writable = probeNeeded ? await npmGlobalPrefixWritable(installEnv(source)) : true; + return installEnv(source, () => writable); +} + +/** + * Runs a STOCK entry's already-vetted install command. `shell: true` is unavoidable here: + * the shipped commands are genuinely `curl | bash` / `npm install -g` one-liners. This is NOT + * a reopening of the config-shell-text concern the registry's `shellToken` pattern exists to + * prevent: the string executed here is NEVER user input, only ever what is already hardcoded + * and reviewed in `stock.ts`, and a CUSTOM entry can never reach this function at all (see the + * guard of each caller). + * + * Never throws: a spawn that fails is a result with `code: null`. + */ +export async function runInstallCommand( + command: string, + options: RunInstallCommandOptions = {} +): Promise { + const { signal } = options; + if (signal?.aborted) return { code: null, output: '', timedOut: false, aborted: true }; + const env = await installEnvFor(command); + // The environment probe above can take a moment (npm boots): an abort during it starts nothing. + if (signal?.aborted) return { code: null, output: '', timedOut: false, aborted: true }; + return new Promise((resolve) => { + let child: ReturnType; + try { + child = spawn(command, { + shell: true, + stdio: ['ignore', 'pipe', 'pipe'], + // Own process group; the timeout below kills the whole tree by hand, mirroring + // the DeepSeek profile-install endpoint's own reasoning: an install command fans + // out into package-manager children, and spawn's own `timeout` option signals + // only the direct child, leaving survivors holding the pipes open forever. + detached: true, + env, + }); + } catch (err) { + resolve({ code: null, output: `spawn failed: ${getErrorMessage(err)}`, timedOut: false, aborted: false }); + return; + } + + let output = ''; + let timedOut = false; + let aborted = false; + let settled = false; + let killTimer: NodeJS.Timeout | undefined; + let reapTimer: NodeJS.Timeout | undefined; + + const capture = (chunk: Buffer) => { + if (output.length < 16_384) output += chunk.toString('utf-8'); + }; + child.stdout?.on('data', capture); + child.stderr?.on('data', capture); + + const killTree = (killSignal: NodeJS.Signals) => { + try { + if (child.pid) process.kill(-child.pid, killSignal); + } catch { + /* already gone */ + } + }; + + /** SIGTERM to the group, SIGKILL three seconds later, and give up waiting after eight. */ + const stopTree = () => { + if (killTimer) return; + killTree('SIGTERM'); + killTimer = setTimeout(() => killTree('SIGKILL'), 3_000); + reapTimer = setTimeout(() => finish(null), 8_000); + }; + + const onAbort = () => { + aborted = true; + stopTree(); + }; + + const finish = (code: number | null) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (killTimer) clearTimeout(killTimer); + if (reapTimer) clearTimeout(reapTimer); + signal?.removeEventListener('abort', onAbort); + resolve({ code, output, timedOut, aborted }); + }; + + const timer = setTimeout(() => { + timedOut = true; + stopTree(); + }, CLI_INSTALL_TIMEOUT_MS); + + signal?.addEventListener('abort', onAbort, { once: true }); + + child.on('error', (err) => { + output = `${output}\n${err.message}`; + finish(null); + }); + child.on('close', (code) => finish(code)); + }); +} + +/** + * The install command of the STOCK CLI `id` for this platform, read from the compiled + * `STOCK_CLIS`. `undefined` for an id that is not a stock CLI or has no command here. + */ +export function stockInstallCommand(id: string): string | undefined { + const entry = STOCK_CLIS.find((candidate) => (candidate.id as string) === id); + if (!entry) return undefined; + const command = resolveInstallCommandForPlatform(entry); + return command !== undefined && command.trim() !== '' ? command : undefined; +} + +/** + * Claims the one install slot of `id`. Returns the function that releases it, or `null` when + * an install of that CLI is already running (the caller answers 409 or waits). + */ +export function tryBeginCliInstall(id: string): (() => void) | null { + if (installsInFlight.has(id)) return null; + installsInFlight.add(id); + let released = false; + return () => { + if (released) return; + released = true; + installsInFlight.delete(id); + }; +} + +/** + * Forgets every cached binary lookup of a CLI: the generic per-id resolver (which captures + * the entry's binaries when first built) and every per-binary cache, hits and negative-cache + * backoff alike. Called after anything that changes what is on disk or what the CLI's binary + * IS; see `invalidateCliExecutableResolvers`. + */ +export function forgetResolvedCli(id: string, binaries: readonly string[]): void { + invalidateCliExecutableResolvers(binaries); + invalidateCliResolverCache(id); +} diff --git a/src/config/local-model-catalog.ts b/src/config/local-model-catalog.ts index a66a111e..a1e578b9 100644 --- a/src/config/local-model-catalog.ts +++ b/src/config/local-model-catalog.ts @@ -3,8 +3,9 @@ * knows, as a compiled constant validated by a Zod schema. * * Rules: - * - There is NO override file and no runtime download data in this stage: nothing here reads - * the data dir, and a data file can never decide what gets recommended. + * - There is NO override file and no runtime download data here: nothing in this file reads + * the data dir, and a data file can never decide what gets recommended. Which runtime build + * is downloaded is a compiled constant of its own (src/local-model/runtime.ts). * - A model id is `name:tag` in lowercase with no slash (`LOCAL_MODEL_ID_PATTERN`), so it can * only ever name a model of the default registry and namespace. * - A harness is a ROW (`{ cli, minContext, rank, gated?, slowFirstTurn? }`). The `cli` is a CLI @@ -13,7 +14,7 @@ * src/local-model/harness-facts.ts, never here. * - The schema checks more than fields. Across rows it requires: unique model ids and ranks, * unique harness `cli`s and ranks, `contexts` largest first, a `tested` entry naming a harness - * row, `chips` most specific first, and `promptCacheTiers` ascending from 0. + * row, and `chips` most specific first. * - `getLocalModelCatalog()` parses the constant once, on first use and never at import, and * throws on a bad row. test/local-model-catalog.test.ts pins that the stock constant parses, * so a bad row fails the gate and never reaches the route. @@ -22,26 +23,44 @@ * * | Figure | Source | Checked | * |----------------------|-----------------------------------------------------------|--------------------------------------| - * | `id`, `bytes` | registry manifest of the tag, sum of its layer sizes | yes, to the byte, by the facts review | + * | `id` | the registry's tag | yes, by the facts review | + * | `bytes` | `qwen3.6:35b-a3b`: the total of a real pull on a Mac | that row: measured. It is an MLX | + * | | (23,105,798,956). Every other row: the sum of the layer | build on a Mac, 2% above its | + * | | sizes of the tag's registry manifest | manifest. The others: not on a Mac | + * | `heldBytesPerToken`, | measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with | that row: measured. The others carry | + * | `overheadBytes`, | 48 GB: 36,000 to 38,000 bytes held per token of context, | the SAME three figures, not their | + * | `peakExtraBytes` | 2.5 to 3 GB beside the weights, and about 5 GB more for a | own: nobody ran them. `figures` says | + * | | moment while a long prompt is processed | which row is which | * | `license` | the model card (all Apache-2.0) | read by the research, not re-checked | * | `paramsB` | 27.8 from the model card; the others are the count in the | no. Only the 12 billion line reads | * | | model's name | it, and no row is near that line | - * | `kvBytesPerToken` | COMPUTED from each model's config: 2 x full-attention | no. Not measured in a running | - * | | layers x KV heads x head size x 2 bytes | runtime. The review called the | - * | | | gpt-oss figure plausible, no more | * | `contexts`, `rank` | a judgement of the design | no | * | `speedPer100Gbps` | one published run on an M1 Max (400 GB/s): generation | the run was read. It stands for less | * | | only, short prompt. The two q8_0 rows are SCALED by file | than a session: no long prompt, no | * | | size, not measured | thinking tokens | + * | `tested` | our own run of the model with a harness on a real Mac | one entry, with its measured speed | * | `minMacosMajor` | the runtime's documentation (macOS 14) | yes, by the facts review | - * | `promptCacheTiers` | a decision of the design, not a measurement | no | * | harness `minContext` | 65536: the runtime's documentation for both harnesses. | 65536 yes. 32768 no, which is why | * | | 32768: recalled | that row is gated | * | `chips` | the bandwidth column of | yes, read at the source for this | * | | https://github.com/ggml-org/llama.cpp/discussions/4167 | file | * - * `tested` is empty on every row: nobody has run any of these models with a harness on a Mac - * yet. Until a row has an entry, the recommendation never says "Recommended". + * The memory figures decide a lot and rest on ONE measured model, so for every other row they + * can be wrong in either direction: + * - For the 9B rows they may be too HIGH (the 5 GB for a moment was seen with a prompt of + * 89,000 tokens on the 35B model). That errs on the careful side: a small Mac is told + * "none" before it is told to download something that may not fit. + * - For the two dense `qwen3.8:27b` rows the per-token figure may be too LOW: computed from + * its config alone that model holds 64 KiB per token, more than the 37,000 bytes measured + * on the 35B (whose own computed figure was 20 KiB). Their need at a large context is + * therefore the least certain figure of the catalogue. + * A row gets its own figures when somebody measures it. Until then the check after the + * download (the runtime must report the whole model on the GPU) is what catches a row that + * was judged too kindly. + * + * `tested` has one entry: `qwen3.6:35b-a3b` with OpenCode on an Apple M5 Pro with 48 GB, + * 2026-11-06, 67.7 tokens per second. Only a Mac of exactly that chip and memory size is told + * "Recommended" and shown that speed in place of the estimate. */ import { z } from 'zod'; @@ -85,6 +104,7 @@ const TestedOnSchema = z memGb: positiveInt, cli: z.string().min(1).max(24), date: isoDate, + tokensPerSecond: z.number().positive().optional(), }) .strict(); @@ -95,7 +115,10 @@ const LocalModelEntrySchema = z license: z.string().min(1).max(40), paramsB: z.number().positive(), bytes: positiveInt, - kvBytesPerToken: positiveInt, + heldBytesPerToken: positiveInt, + overheadBytes: z.number().int().nonnegative(), + peakExtraBytes: z.number().int().nonnegative(), + figures: z.enum(['measured', 'estimated']), contexts: z.array(positiveInt).min(1), rank: z.number().int(), speedPer100Gbps: z.number().positive().optional(), @@ -120,13 +143,6 @@ const ChipEntrySchema = z }) .strict(); -const PromptCacheTierSchema = z - .object({ - minMemGb: z.number().int().nonnegative(), - bytes: z.number().int().nonnegative(), - }) - .strict(); - /** The index of the first value that already occurred earlier in the list, or -1. */ function firstRepeat(values: readonly T[]): number { const seen = new Set(); @@ -146,7 +162,6 @@ export const LocalModelCatalogSchema: z.ZodType = z schema: z.literal(1), updated: isoDate, minMacosMajor: positiveInt, - promptCacheTiers: z.array(PromptCacheTierSchema).min(1), models: z.array(LocalModelEntrySchema), harnesses: z.array(LocalHarnessEntrySchema), chips: z.array(ChipEntrySchema), @@ -199,23 +214,21 @@ export const LocalModelCatalogSchema: z.ZodType = z break; } } - - // The tiers are read as "the last one whose minimum is not above this Mac's memory", so they - // must start at 0 (every Mac has a tier) and ascend. - if (catalog.promptCacheTiers[0].minMemGb !== 0) { - issue(['promptCacheTiers', 0, 'minMemGb'], 'the first prompt cache tier must start at 0'); - } - for (let i = 1; i < catalog.promptCacheTiers.length; i++) { - if (catalog.promptCacheTiers[i].minMemGb <= catalog.promptCacheTiers[i - 1].minMemGb) { - issue(['promptCacheTiers', i, 'minMemGb'], 'prompt cache tiers must ascend by minMemGb'); - break; - } - } }); -const KIB = 1024; -const MIB = 1024 ** 2; -const GIB = 1024 ** 3; +/** + * The memory figures measured for `qwen3.6:35b-a3b` on an Apple M5 Pro with 48 GB (2026-11-06, + * the MLX runner), in decimal bytes. That row carries them as `measured`; every other row + * carries the same three as `estimated`, see the file header. + */ +const MEASURED_MEMORY = { + /** 36,000 to 38,000 bytes stayed held per token of context after a prompt of 89,000 and one of 167,000 tokens. */ + heldBytesPerToken: 37_000, + /** The runner held 2.5 to 3 GB beside the weights. */ + overheadBytes: 3_000_000_000, + /** While a long prompt was processed the peak stood about 5 GB above what stayed held. */ + peakExtraBytes: 5_000_000_000, +} as const; /** * The catalogue that ships with Codeman. Never read directly outside this file and its test: @@ -226,12 +239,6 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { updated: '2026-11-06', // The oldest macOS the model runtime supports. minMacosMajor: 14, - // Host memory the runtime's prompt cache may take beside the model: 512 MiB on a Mac below - // 32 GB, 2 GiB from 32 GB. Part of the need of every candidate. - promptCacheTiers: [ - { minMemGb: 0, bytes: 512 * MIB }, - { minMemGb: 32, bytes: 2 * GIB }, - ], // Ordered by rank. The higher rank wins among models that fit equally well and are equally // fast. `flint:40b` is left out on purpose: its plain tag is the slow build on a Mac and it // has an open bug with tool arguments. @@ -242,7 +249,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 9, bytes: 6_550_825_373, - kvBytesPerToken: 32 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [65536, 32768], rank: 1, // 37.1 tokens per second measured on 400 GB/s. @@ -255,7 +263,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 9, bytes: 10_449_217_949, - kvBytesPerToken: 32 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [65536, 32768], rank: 2, // Not measured: the 9.3 of the row above scaled by file size. @@ -268,7 +277,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 20, bytes: 13_793_441_427, - kvBytesPerToken: 24 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072, 65536], rank: 3, // No speed figure: its speed is unknown on every chip. @@ -279,13 +289,18 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { label: 'Qwen3.6 35B-A3B', license: 'Apache-2.0', paramsB: 35, - bytes: 22_621_314_161, - kvBytesPerToken: 20 * KIB, + // What a Mac really downloads: an MLX build, 2% above the 22,621,314,161 of the manifest. + bytes: 23_105_798_956, + ...MEASURED_MEMORY, + figures: 'measured', contexts: [131072, 65536], rank: 4, // 54.5 tokens per second measured on 400 GB/s (3 billion parameters are active per token). + // The MLX build ran faster than this figure predicts on the one Mac we tested (67.7 + // against 41.8), so on other chips the estimate is on the low side. speedPer100Gbps: 13.6, - tested: [], + // Run by us with OpenCode on the MacBook Pro (Mac17,13): a four-step task with tool calls. + tested: [{ chip: 'Apple M5 Pro', memGb: 48, cli: 'opencode', date: '2026-11-06', tokensPerSecond: 67.7 }], }, { id: 'qwen3.8:27b', @@ -293,7 +308,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 27.8, bytes: 17_741_871_939, - kvBytesPerToken: 64 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072, 65536, 32768], rank: 5, // 11.8 tokens per second measured on 400 GB/s. @@ -306,7 +322,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 27.8, bytes: 29_978_241_837, - kvBytesPerToken: 64 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [262144, 131072], rank: 6, // Not measured: the 2.95 of the row above scaled by file size. @@ -319,7 +336,8 @@ export const STOCK_LOCAL_MODEL_CATALOG: LocalModelCatalog = { license: 'Apache-2.0', paramsB: 122, bytes: 81_370_035_877, - kvBytesPerToken: 24 * KIB, + ...MEASURED_MEMORY, + figures: 'estimated', contexts: [131072], rank: 7, // No speed figure: its speed is unknown on every chip. diff --git a/src/custom-model-hosts-writer.ts b/src/custom-model-hosts-writer.ts new file mode 100644 index 00000000..5db70aed --- /dev/null +++ b/src/custom-model-hosts-writer.ts @@ -0,0 +1,124 @@ +/** + * @fileoverview The serialized writer of `custom-model-hosts.json`, the store of the saved + * custom model endpoints (src/custom-model-hosts.ts). + * + * Why it exists: `writeCustomModelHosts()` writes through ONE temp file name per process and + * has no queue, and `readCustomModelHosts()` reads a file that does not parse as an empty + * list. Two overlapping writers (an endpoint route and the 5-minute discovery sweep) can + * leave one body over the tail of a longer one, and the next writer then saves a file that + * holds only its own record: every other endpoint and its API key is gone. The same fault + * `mutateSettingsFile()` ended for `settings.json`. + * + * The shape of `mutateSettingsFile()` (src/web/settings-writer.ts): + * - Serialized: mutations run one at a time on one promise chain. The check a route makes + * (the id is free, the record still exists) runs INSIDE `fn`, in the same step as the write. + * - Atomic: each write goes to a temp file with a random name beside the target and is + * renamed over it. The file is mode 0600 (it holds API keys). + * - Refuses a file it must not trust: only a missing or empty file starts from an empty + * list. A file that cannot be read, does not parse or is not an array is left alone and + * the write rejects with `CustomModelHostsWriteRefusedError`. The message never repeats + * the parser's own text, which can quote an API key. + * + * EVERY writer goes through it: the four endpoint routes of + * src/web/routes/custom-model-routes.ts (create, update, delete, discover-models), the + * discovery sweep `refreshAllCustomModelHosts()`, and the local model's connect step. The one + * direct writer left is config-backup restore, which must work on a file that does not parse. + */ + +import { randomUUID } from 'node:crypto'; +import fs from 'node:fs/promises'; +import { dirname } from 'node:path'; +import { getDataDir } from './config/instance.js'; +import { customModelHostsPath, type CustomModelHost } from './custom-model-hosts.js'; +import { ApiErrorCode, createErrorResponse, getErrorMessage, type ApiResponse } from './types.js'; + +/** A write refused because the existing `custom-model-hosts.json` must not be overwritten. The message is user-facing. */ +export class CustomModelHostsWriteRefusedError extends Error { + constructor(message: string) { + super(message); + this.name = 'CustomModelHostsWriteRefusedError'; + } +} + +/** The list a write starts from. Only a missing or empty file is an empty list; anything else that is not an array is refused. */ +async function readHostsForWrite(path: string): Promise { + let raw: string; + try { + raw = await fs.readFile(path, 'utf-8'); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return []; + throw new CustomModelHostsWriteRefusedError( + `Cannot read ${path} (${(err as Error).message}), so this change was not saved.` + ); + } + // An empty file holds no endpoint to lose. + if (raw.trim() === '') return []; + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + parsed = undefined; + } + if (!Array.isArray(parsed)) { + throw new CustomModelHostsWriteRefusedError( + `${path} is not a valid JSON list, so this change was not saved (saving would have replaced every other model endpoint in it, with its API key). Fix or delete the file, or restore it from a config backup, then try again.` + ); + } + return parsed as CustomModelHost[]; +} + +async function writeHostsFile(path: string, hosts: CustomModelHost[]): Promise { + // Through a symlink: the rename lands on the real file. + const target = await fs.realpath(path).catch(() => path); + await fs.mkdir(dirname(target), { recursive: true }); + const tmp = `${target}.${process.pid}.${randomUUID()}.tmp`; + try { + await fs.writeFile(tmp, JSON.stringify(hosts, null, 2), { mode: 0o600 }); + await fs.rename(tmp, target); + } catch (err) { + await fs.rm(tmp, { force: true }).catch(() => {}); + throw err; + } +} + +let mutationChain: Promise = Promise.resolve(); + +/** What a mutation returns: the list to write (absent: write nothing) and what the caller gets back. */ +export interface CustomModelHostsMutation { + /** The whole list to store. Absent: the file is left as it is. */ + hosts?: CustomModelHost[]; + /** Handed back to the caller of `mutateCustomModelHosts()`. */ + result: T; +} + +/** + * Runs one change to the endpoint store. `fn` receives the current list (a copy it may change) + * and returns what to write and what to hand back. Resolves with `result`. A refused or failed + * mutation rejects its own caller only; the chain keeps going. + * + * `configDir` is the folder of the file, as for `readCustomModelHosts()`. Default: the data dir. + */ +export function mutateCustomModelHosts( + fn: (hosts: CustomModelHost[]) => CustomModelHostsMutation, + configDir?: string +): Promise { + const run = mutationChain.then(async () => { + // Resolved inside the chain: the data dir is read when the mutation runs, like the file. + const path = customModelHostsPath(configDir ?? getDataDir()); + const current = await readHostsForWrite(path).catch((err: unknown) => { + console.warn(`[custom-model] write refused: ${getErrorMessage(err)}`); + throw err; + }); + const mutation = fn(current); + if (mutation.hosts) await writeHostsFile(path, mutation.hosts); + return mutation.result; + }); + mutationChain = run.catch(() => {}); + return run; +} + +/** A refused write answers 409 CONFLICT with the fix; any other failure stays OPERATION_FAILED. Like `settingsWriteErrorResponse()`. */ +export function customModelHostsWriteErrorResponse(err: unknown): ApiResponse { + if (err instanceof CustomModelHostsWriteRefusedError) return createErrorResponse(ApiErrorCode.CONFLICT, err.message); + return createErrorResponse(ApiErrorCode.OPERATION_FAILED, getErrorMessage(err)); +} diff --git a/src/local-model/connect.ts b/src/local-model/connect.ts new file mode 100644 index 00000000..0af1fb12 --- /dev/null +++ b/src/local-model/connect.ts @@ -0,0 +1,305 @@ +/** + * @fileoverview Connecting the installed local model to Codeman, and undoing it: the endpoint + * record, the settings flag, and the checks around them. + * + * These are the ONLY two writes of the local model outside its own folder: + * 1. One record in `custom-model-hosts.json`, through `mutateCustomModelHosts()`: + * `{ id: 'local-ollama', label: 'This Mac', baseUrl: 'http://127.0.0.1:', + * authStyle: 'bearer', models: [], defaultModelId, + * defaultContextLength: }`, validated by `CustomModelHostSchema` + * before it is written. No `apiKey`. `defaultContextLength` is what reaches the coding + * agent as its context limit, because the runtime's model listing states none. + * 2. `customModelEndpointsEnabled: true` in `settings.json`, through `mutateSettingsFile()`. + * The value it had before is returned and kept in the state file. + * + * Rules: + * - The id is reserved. A record with it that this feature did not write is never + * overwritten: the connect step fails with `endpoint-id-taken`. + * - The connection is CLAIMED before it is written: the install job records `connection` in + * the state file first (`inspectLocalEndpoint()` tells it what is there and what the flag + * is), then calls `connectLocalModel()` with `ours: true`. A state write that fails, or a + * server that is killed between the two, therefore never leaves a record of ours that the + * state file does not know: the worst case is a claim with nothing behind it, which a + * resumed job fills in and Remove undoes. + * - Second net: a record with the reserved id that already says exactly what ours would say + * (`isSameLocalEndpoint()`) is ours whatever `ours` says. Nobody else can have made it: the + * endpoint routes refuse the reserved id. + * - Both writes are idempotent, so a resumed job writes them again. + * - Undoing removes our record, and switches the flag off again only when this feature + * switched it on AND no other endpoint is saved. + * - The model's digest is compared again before a session starts and at boot + * (`compareModelDigest()`): a model that changed under the same name is not launched. + * - The record is MANAGED while the state file records it as ours (`isManagedLocalEndpoint()`): + * the endpoint routes then refuse to edit or delete it and the discovery sweep leaves it + * alone, so its `models` stays the one model the install job checked. + */ + +import { mutateCustomModelHosts } from '../custom-model-hosts-writer.js'; +import type { CustomModelHost } from '../custom-model-hosts.js'; +import { getDataDir } from '../config/instance.js'; +import { readCustomModelHosts } from '../custom-model-hosts.js'; +import { getErrorMessage, type LocalModelErrorCode } from '../types.js'; +import { readJsonConfig, SETTINGS_PATH } from '../web/route-helpers.js'; +import { CustomModelHostSchema } from '../web/schemas.js'; +import { mutateSettingsFile } from '../web/settings-writer.js'; +import { readLocalModelState } from './state-store.js'; + +/** The key of the settings flag that shows saved endpoints in the Run menu and runs the discovery sweep. */ +const ENDPOINTS_FLAG = 'customModelEndpointsEnabled'; + +/** The id of the endpoint record this feature writes. Reserved: nobody else may use it. */ +export const LOCAL_MODEL_ENDPOINT_ID = 'local-ollama'; +/** The label of that record, what the Run menu shows in brackets. */ +export const LOCAL_MODEL_ENDPOINT_LABEL = 'This Mac'; + +/** What the endpoint record is built from. */ +export interface LocalEndpointInput { + /** The runtime's loopback port. */ + port: number; + /** The model id exactly as the runtime's `/v1/models` lists it. */ + modelId: string; + /** The chosen context length in tokens. */ + context: number; +} + +/** The endpoint record of the file header. Pure; the caller validates it with `CustomModelHostSchema`. */ +export function buildLocalEndpointRecord(input: LocalEndpointInput): CustomModelHost { + return { + id: LOCAL_MODEL_ENDPOINT_ID, + label: LOCAL_MODEL_ENDPOINT_LABEL, + baseUrl: `http://127.0.0.1:${input.port}`, + authStyle: 'bearer', + models: [input.modelId], + defaultModelId: input.modelId, + defaultContextLength: input.context, + }; +} + +/** + * Whether a saved record says exactly what our record for `input` says: the same address, the + * one model, its context, no key. What discovery may have added to it (context lengths, a + * time stamp) does not count. Pure. + */ +export function isSameLocalEndpoint(host: CustomModelHost, input: LocalEndpointInput): boolean { + const ours = buildLocalEndpointRecord(input); + return ( + host.id === ours.id && + host.label === ours.label && + host.baseUrl === ours.baseUrl && + host.authStyle === ours.authStyle && + host.apiKey === undefined && + Array.isArray(host.models) && + host.models.length === 1 && + host.models[0] === input.modelId && + host.defaultModelId === ours.defaultModelId && + host.defaultContextLength === ours.defaultContextLength + ); +} + +/** What the connect step finds before it claims the connection. */ +export interface LocalEndpointInspection { + /** + * The saved record with the reserved id: there is none, it says what ours would say, or it + * says something else (somebody else's: `endpoint-id-taken`). + */ + record: 'absent' | 'same' | 'other'; + /** `customModelEndpointsEnabled` right now: what the claim keeps as the value before this feature. */ + flagEnabled: boolean; +} + +/** + * Looks at the endpoint store and the settings flag WITHOUT writing either. The install job + * calls it before it claims the connection in the state file. Rejects when the endpoint store + * cannot be read; a settings file that cannot be read counts as "the flag is off". + */ +export async function inspectLocalEndpoint(input: LocalEndpointInput): Promise { + const hosts = await readCustomModelHosts(getDataDir()); + const existing = hosts.find((host) => host.id === LOCAL_MODEL_ENDPOINT_ID); + const settings = await readJsonConfig>(SETTINGS_PATH, 'settings.json', {}); + return { + record: !existing ? 'absent' : isSameLocalEndpoint(existing, input) ? 'same' : 'other', + flagEnabled: settings[ENDPOINTS_FLAG] === true, + }; +} + +export interface ConnectLocalModelInput extends LocalEndpointInput { + /** + * True when the state file already records a connection of ours: an existing record with + * our id is then ours and is replaced. False: such a record is somebody else's, unless it + * says exactly what ours would say (`isSameLocalEndpoint()`). + */ + ours: boolean; + /** + * The flag's value before this feature first switched it on, when the state file already + * holds one. It is handed back unchanged, so a second connect cannot overwrite it with `true`. + */ + flagWasEnabled?: boolean; +} + +/** How the connect step ended. */ +export type ConnectResult = + | { + ok: true; + /** The value `customModelEndpointsEnabled` had before this feature switched it on. */ + flagWasEnabled: boolean; + } + | { ok: false; code: Extract; detail?: string }; + +/** + * Writes the record and the flag. Idempotent. Never throws: a refused write is `connect-failed` + * with its message as `detail`. + * + * The record goes first, with the "is the id free" check in the same serialized step as the + * write, so a record that is somebody else's is found before anything is changed. Should the + * flag then fail to save, a record this call ADDED is taken out again: the pair is never left + * half done, and a later attempt starts from the same place. + */ +export async function connectLocalModel(input: ConnectLocalModelInput): Promise { + const parsed = CustomModelHostSchema.safeParse(buildLocalEndpointRecord(input)); + if (!parsed.success) { + return { ok: false, code: 'connect-failed', detail: `The endpoint record is not valid: ${parsed.error.message}` }; + } + const record: CustomModelHost = parsed.data; + + let outcome: 'added' | 'replaced' | 'taken'; + try { + outcome = await mutateCustomModelHosts<'added' | 'replaced' | 'taken'>((hosts) => { + const index = hosts.findIndex((host) => host.id === record.id); + if (index === -1) return { hosts: [...hosts, record], result: 'added' }; + if (!input.ours && !isSameLocalEndpoint(hosts[index], input)) return { result: 'taken' }; + const next = [...hosts]; + next[index] = record; + return { hosts: next, result: 'replaced' }; + }); + } catch (err) { + return { ok: false, code: 'connect-failed', detail: getErrorMessage(err) }; + } + if (outcome === 'taken') return { ok: false, code: 'endpoint-id-taken' }; + + let flagBefore: boolean; + try { + const { before } = await mutateSettingsFile((current) => + current[ENDPOINTS_FLAG] === true ? undefined : { ...current, [ENDPOINTS_FLAG]: true } + ); + flagBefore = before[ENDPOINTS_FLAG] === true; + } catch (err) { + if (outcome === 'added') await removeLocalEndpoint().catch(() => {}); + return { ok: false, code: 'connect-failed', detail: getErrorMessage(err) }; + } + return { ok: true, flagWasEnabled: input.flagWasEnabled ?? flagBefore }; +} + +/** + * Puts our record back when it is gone, and does nothing when a record with our id exists + * (it is never replaced here). For the start of the server while a model is installed: a + * config-backup restore can bring back an older endpoint store without our record while the + * model, the runtime and the state file stay. The settings flag is NOT touched: a person who + * switched it off meant to. Resolves true when the record was added. Never throws. + */ +export async function ensureLocalEndpoint(input: LocalEndpointInput): Promise { + const parsed = CustomModelHostSchema.safeParse(buildLocalEndpointRecord(input)); + if (!parsed.success) return false; + const record: CustomModelHost = parsed.data; + try { + return await mutateCustomModelHosts((hosts) => + hosts.some((host) => host.id === record.id) ? { result: false } : { hosts: [...hosts, record], result: true } + ); + } catch { + // A store that must not be overwritten stays as it is; a session start then says what is wrong. + return false; + } +} + +/** Takes our record out of the store. Resolves with how many OTHER endpoints are saved. */ +async function removeLocalEndpoint(): Promise { + return mutateCustomModelHosts((hosts) => { + const others = hosts.filter((host) => host.id !== LOCAL_MODEL_ENDPOINT_ID); + return others.length === hosts.length ? { result: others.length } : { hosts: others, result: others.length }; + }); +} + +/** + * Moves our record to another port (the answer to "Use another port" once connected). A + * missing record is not an error, and a record that is already there is not written again. + */ +export async function moveLocalEndpoint(port: number): Promise { + const baseUrl = `http://127.0.0.1:${port}`; + await mutateCustomModelHosts((hosts) => { + const index = hosts.findIndex((host) => host.id === LOCAL_MODEL_ENDPOINT_ID); + if (index === -1 || hosts[index].baseUrl === baseUrl) return { result: undefined }; + const next = [...hosts]; + next[index] = { ...next[index], baseUrl }; + return { hosts: next, result: undefined }; + }); +} + +/** + * Removes our record, and restores the settings flag when this feature switched it on + * (`flagWasEnabled` is false) and no other endpoint is left. Idempotent. + */ +export async function disconnectLocalModel(input: { flagWasEnabled: boolean }): Promise { + const othersLeft = await removeLocalEndpoint(); + if (input.flagWasEnabled || othersLeft > 0) return; + await mutateSettingsFile((current) => + current[ENDPOINTS_FLAG] === true ? { ...current, [ENDPOINTS_FLAG]: false } : undefined + ); +} + +/** Whether `id` is the reserved endpoint id. For the endpoint routes and the discovery sweep, which leave that record alone. */ +export function isLocalModelEndpointId(id: string): boolean { + return id === LOCAL_MODEL_ENDPOINT_ID; +} + +/** + * Whether the saved endpoint `id` is the record THIS feature wrote and still owns: the id is + * the reserved one and the state file records a connection under it. A record with the + * reserved id that somebody made by hand (before this feature, or while nothing is installed) + * is not managed, so its owner can still edit and delete it. Never throws. + */ +export async function isManagedLocalEndpoint(id: string): Promise { + if (!isLocalModelEndpointId(id)) return false; + const state = await readLocalModelState(); + return state.connection?.endpointId === id; +} + +/** + * The names of the live sessions that use the endpoint `endpointId`, in the order given. + * Pure. What `DELETE /api/local-model` answers its 409 with. + */ +export function sessionNamesUsingEndpoint( + sessions: Iterable<{ name: string; customModel?: { endpointId: string } }>, + endpointId: string = LOCAL_MODEL_ENDPOINT_ID +): string[] { + const names: string[] = []; + for (const session of sessions) { + if (session.customModel?.endpointId === endpointId) names.push(session.name); + } + return names; +} + +/** + * Compares the digest stored at install time with the one the runtime reports now. Pure. + * - `same`: launch. + * - `changed`: refuse (`model-changed`): the name now stands for other weights or another template. + * - `unknown`: one of the two is absent (the runtime does not answer, the model is gone). + * The caller refuses as well, with the reason it knows. + */ +export function compareModelDigest( + stored: string | undefined, + current: string | undefined +): 'same' | 'changed' | 'unknown' { + const a = normalizeDigest(stored); + const b = normalizeDigest(current); + if (!a || !b) return 'unknown'; + return a === b ? 'same' : 'changed'; +} + +/** A digest as it is compared: lowercase hex, without the `sha256:` some listings put in front. */ +function normalizeDigest(digest: string | undefined): string | undefined { + if (typeof digest !== 'string') return undefined; + const hex = digest + .trim() + .toLowerCase() + .replace(/^sha256:/, ''); + return hex === '' ? undefined : hex; +} diff --git a/src/local-model/developer-tools.ts b/src/local-model/developer-tools.ts new file mode 100644 index 00000000..c7f7446b --- /dev/null +++ b/src/local-model/developer-tools.ts @@ -0,0 +1,117 @@ +/** + * @fileoverview Whether this Mac has the developer tools a coding agent calls (git, Python). + * + * Why it exists: on a Mac without Xcode or the Command Line Tools, `git` and `python3` are + * stubs that open macOS's "install the command line developer tools" window. A coding agent + * runs both, so the install window says one honest sentence when the tools are missing + * (`LOCAL_MODEL_INSTALL_LABELS.developerToolsMissing`). + * + * Rules: + * - The ONLY command ever run is `/usr/bin/xcode-select -p`, bounded by a timeout. Never + * `git`, `python3`, `xcrun` or anything else that would open that window itself. + * - macOS only. Anywhere else, and when the command does not answer in time, the answer is + * `unknown`, and nothing is said. + * - Never throws. The answer is cached for a short while, so a window that polls once a + * second does not spawn once a second. Only `present` and `missing` are cached: an + * `unknown` is asked again, because it says the check itself did not work. + * - Under vitest the default `run` spawns nothing and answers "did not start", so a test + * that wants an answer injects its own. + */ + +import { execFile } from 'node:child_process'; +import type { LocalModelDeveloperTools } from '../types.js'; + +/** How long `/usr/bin/xcode-select -p` may take. */ +export const DEVELOPER_TOOLS_TIMEOUT_MS = 2_000; + +/** How long an answer is reused. Installing the tools takes minutes, so a minute of staleness is not seen. */ +const DEVELOPER_TOOLS_CACHE_MS = 60_000; + +/** The one command this module runs. It prints the active developer directory and exits 0, or exits with an error when there is none. */ +const XCODE_SELECT = '/usr/bin/xcode-select'; + +/** What the check calls. Each member defaults to the real thing; a test injects its own. */ +export interface DeveloperToolsDeps { + /** Default `process.platform`. */ + platform: string; + /** Runs `file` with `args` and resolves with its exit code (`null` when it timed out or could not be started). */ + run: (file: string, args: readonly string[], timeoutMs: number) => Promise; +} + +/** The default `run`: one bounded `execFile`, no shell. Spawns nothing under vitest. */ +function runBounded(file: string, args: readonly string[], timeoutMs: number): Promise { + return new Promise((resolve) => { + if (process.env.VITEST) return resolve(null); + try { + execFile(file, [...args], { timeout: timeoutMs }, (err) => { + if (!err) return resolve(0); + // A number is the exit code of a command that ran. Anything else (ENOENT, a timeout + // that killed it) is "it did not answer". + const code = (err as { code?: unknown }).code; + resolve(typeof code === 'number' && !(err as { killed?: boolean }).killed ? code : null); + }); + } catch { + resolve(null); + } + }); +} + +async function check(deps: DeveloperToolsDeps): Promise { + if (deps.platform !== 'darwin') return 'unknown'; + let code: number | null; + try { + code = await deps.run(XCODE_SELECT, ['-p'], DEVELOPER_TOOLS_TIMEOUT_MS); + } catch { + code = null; + } + if (code === null || !Number.isInteger(code)) return 'unknown'; + return code === 0 ? 'present' : 'missing'; +} + +/** + * A check that remembers its answer for a minute and shares one run among callers that ask at + * the same time. `now` is the clock of the cache; a test injects its own. + */ +export function createDeveloperToolsCheck( + deps: Partial & { now?: () => number } = {} +): () => Promise { + const resolved: DeveloperToolsDeps = { + platform: deps.platform ?? process.platform, + run: deps.run ?? runBounded, + }; + const now = deps.now ?? Date.now; + let cached: { answer: LocalModelDeveloperTools; at: number } | undefined; + let inFlight: Promise | undefined; + return () => { + if (cached && now() - cached.at < DEVELOPER_TOOLS_CACHE_MS) return Promise.resolve(cached.answer); + inFlight ??= check(resolved) + .then((answer) => { + if (answer !== 'unknown') cached = { answer, at: now() }; + return answer; + }) + .finally(() => { + inFlight = undefined; + }); + return inFlight; + }; +} + +/** The check of this server's own Mac. */ +let shared = createDeveloperToolsCheck(); + +/** Forgets the cached answer of this server's own Mac. For tests. */ +export function resetDeveloperToolsCache(): void { + shared = createDeveloperToolsCheck(); +} + +/** + * `present` when `/usr/bin/xcode-select -p` exits 0, `missing` when it exits with anything + * else, `unknown` otherwise. Without `deps` it is the cached check of this server's Mac; with + * a platform or a runner of the caller's it asks once and remembers nothing. + */ +export async function checkDeveloperTools(deps: Partial = {}): Promise { + if (deps.platform !== undefined || deps.run !== undefined) { + return check({ platform: deps.platform ?? process.platform, run: deps.run ?? runBounded }); + } + return shared(); +} diff --git a/src/local-model/format.ts b/src/local-model/format.ts index 4f8db32d..1fb6508b 100644 --- a/src/local-model/format.ts +++ b/src/local-model/format.ts @@ -7,12 +7,12 @@ * `budgetGb`, and the size slots of a `LocalModelMessage`). Nothing else may turn bytes into a * size for display, the browser included: it prints the text it was sent. * - * GiB and MiB exist only for the budget math (reserves, the wired limit, the prompt cache) and - * never reach a person, with one exception: `memClassGb()`, the memory size a Mac is sold as, - * is how the card names the installed memory ("48 GB memory", what About This Mac shows). It - * is a name of the machine, not a computed size: `formatGb()` of the same bytes reads 51.5, - * which no owner recognises. Every computed size (need, budget, download, free disk) is - * `formatGb()`. + * GiB and MiB exist only for the budget math (reserves, the wired limit, the device total the + * runtime reports) and never reach a person, with one exception: `memClassGb()`, the memory + * size a Mac is sold as, is how the card names the installed memory ("48 GB memory", what + * About This Mac shows). It is a name of the machine, not a computed size: `formatGb()` of the + * same bytes reads 51.5, which no owner recognises. Every computed size (need, budget, + * download, free disk) is `formatGb()`. * * Pure: no IO, no state. */ diff --git a/src/local-model/harness-facts.ts b/src/local-model/harness-facts.ts index b86403e4..143826f1 100644 --- a/src/local-model/harness-facts.ts +++ b/src/local-model/harness-facts.ts @@ -10,8 +10,11 @@ * - reports `label` from the registry entry and `installed` from the same probe the page * render and `GET /api/clis` use (`utils/cli-installed-probes.ts`), asked once per build; * - reports `installable` only for a STOCK CLI that is not installed and whose install - * command for this platform needs no administrator rights (`installCommandNeedsAdmin()`). - * Stock only, because the install runner never executes a custom entry's command; + * command for this platform needs no administrator rights (`installCommandNeedsAdmin()`), + * and then carries that command as `installCommand`, so the consent screen can print the + * literal text. The command is the COMPILED stock one (`stockInstallCommand()`), never the + * one a `clis.json` override supplies: it is exactly what the install job would run, and + * an override file must not be able to change that; * - supplies `acceptsModelId` (`cliAcceptsCustomModelId()`), the test a custom model launch * applies to the model id before anything is written. * @@ -33,7 +36,8 @@ import type { CliEntry } from '../config/cli-registry/types.js'; import { matchesPattern } from '../config/cli-registry/patterns.js'; -import { getCli, resolveInstallCommandForPlatform } from '../config/cli-registry/registry.js'; +import { stockInstallCommand } from '../cli-install-runner.js'; +import { getCli } from '../config/cli-registry/registry.js'; import { buildCustomModelInjection, type CustomModelEndpoint } from '../custom-model-injection.js'; import { isCliEntryInstalled, probeStockCliAvailability } from '../utils/cli-installed-probes.js'; import { getErrorMessage, type HarnessFact, type LocalModelCatalog } from '../types.js'; @@ -48,7 +52,11 @@ export interface HarnessFactsDeps { * `installed` passes its own. */ probeInstalled: () => Promise<(entry: CliEntry) => boolean>; - /** The install command the registry holds for this platform. Read as text, never run. */ + /** + * The install command the install job would run for this CLI on this platform: the compiled + * stock command, `undefined` for a CLI that is not stock or has none here. Read as text, + * never run. + */ installCommandFor: (entry: CliEntry) => string | undefined; } @@ -58,7 +66,7 @@ export const defaultHarnessFactsDeps: HarnessFactsDeps = { const stockAvailability = await probeStockCliAvailability(); return (entry) => isCliEntryInstalled(entry, stockAvailability); }, - installCommandFor: resolveInstallCommandForPlatform, + installCommandFor: (entry) => stockInstallCommand(entry.id as string), }; /** @@ -147,17 +155,20 @@ export async function buildHarnessFacts( console.warn(`[local-model] Probing ${entry.label} failed, reading it as not installed: ${getErrorMessage(err)}`); } - let installable = false; + let installCommand: string | undefined; if (!installed && entry.stock) { const command = resolved.installCommandFor(entry); - installable = command !== undefined && command.trim() !== '' && !installCommandNeedsAdmin(command); + if (command !== undefined && command.trim() !== '' && !installCommandNeedsAdmin(command)) { + installCommand = command; + } } facts.push({ cli: row.cli, label: entry.label, installed, - installable, + installable: installCommand !== undefined, + ...(installCommand !== undefined ? { installCommand } : {}), acceptsModelId: (modelId: string) => cliAcceptsCustomModelId(entry, modelId), }); } diff --git a/src/local-model/install-job.ts b/src/local-model/install-job.ts new file mode 100644 index 00000000..cc2ae0d8 --- /dev/null +++ b/src/local-model/install-job.ts @@ -0,0 +1,2235 @@ +/** + * @fileoverview The local model install job: its state table, the pure reducer over it, the + * pure view of it, and the service that drives it. + * + * ONE job at a time. The job is five steps in a fixed order (`LOCAL_MODEL_INSTALL_STEPS`: + * runtime, model, verify, agent, connect; `agent` is left out when the coding agent is + * installed). Its state is one of five (`LocalModelInstallState`), stored in the state file. + * + * The state table. "On disk" names what lies under the local model folder; the two writes + * outside it belong to the `connect` step. + * + * | State | On disk | A server restart | Cancel | Allowed next actions | + * |--------------|-------------------------------------------|-----------------------------|------------------------|---------------------------------| + * | `idle` | nothing, or leftovers a Remove could not | stays `idle` | refused (409) | install | + * | | delete (never read back) | | | | + * | `installing` | what the finished steps wrote, plus what | becomes `paused` at the | becomes `paused` at | cancel | + * | | the current step has written so far | same step; nothing resumes | the same step | | + * | | (see the step table) | without a click | | | + * | `paused` | as `installing` at that step, nothing in | stays `paused` | refused (409) | resume, install (another | + * | | flight | | | choice), remove | + * | `failed` | as `installing` at that step, nothing in | stays `failed`, same error | refused (409) | resume (try again), install | + * | | flight | | | (another choice), remove | + * | `installed` | the runtime, the private home, the model | stays `installed`; the | refused (409) | start a session, start or stop | + * | | | runtime is started again | | the runtime, remove | + * | | | when it was not stopped | | | + * + * The step table. Every step is IDEMPOTENT, so Resume and "Try again" re-enter the step the + * job stopped in; they first make sure the runtime runs when the step needs it. + * + * | Step | Adds to disk | Cancel | Resume | + * |-----------|---------------------------------------|------------------------------------------|------------------------------------------------| + * | `runtime` | the archive's partial file, then the | aborts the download, the partial file | continues the partial file with a Range | + * | | unpacked runtime, then the private | stays | request, or skips what is already unpacked | + * | | home once the runtime started | | | + * | `model` | the model store: partial blobs, then | aborts the pull request, which stops the | pulls again; the runtime continues its partial | + * | | the model | download; the partial blobs stay | blobs | + * | `verify` | nothing (the model is loaded into | aborts the request in flight | runs the three checks again from the first | + * | | memory) | | | + * | `agent` | whatever the coding agent's own | kills the installer's process group | looks for the binary first and runs the | + * | | installer writes, outside our folder; | | installer only when it is still missing | + * | | never removed by us | | | + * | `connect` | our endpoint record and the settings | honoured before the first write; once | writes both again | + * | | flag: the two writes outside our | the writes began they finish and the job | | + * | | folder. The state file claims them | ends `installed` | | + * | | first | | | + * + * How the memory budget gets measured before a model is downloaded (ONE consent, bound to the + * size shown): the person confirms `Candidate.installBytes` once. The `runtime` step then + * gets and starts the runtime, reads the device total from its log and judges the choice + * again with it (`modelFitsBudget()`; `recommend()` with the same options names the next + * model). When the chosen model at the chosen context no longer fits, the job fails right + * there with `does-not-fit` and the next model that fits, BEFORE the model download starts. When it fits, the job goes on without a second question: the sizes that were + * confirmed did not change. From then on the GET reports `budgetSource: 'measured'`. + * + * Rules: + * - `reduceInstallJob()` is PURE and total: an event the transition table does not list for + * the current state is refused by returning the SAME object, which the service answers + * with 409 `wrong-state`. + * - The reducer moves the state, the step, the error and the progress. The FACTS a step + * learns (the port and pid, the measured total, the pull's total, the digest, what the + * runtime loaded, the previous settings flag) are written by the service through + * `mutateLocalModelState()` and never change the state. + * - A job read as `installing` at boot was interrupted by the restart: the service applies + * the event `restart` before anything else, so it is `paused` and waits for a click. + * - A new install from `paused` or `failed` REPLACES the job. When it names another model, + * what the old job left of its model is deleted from the store first: the model through the + * runtime when it was downloaded whole, and the partial blobs of a download that was + * interrupted (the runtime knows no model for those, so they are deleted as files). + * - Install is checked before anything is downloaded: admin only in multi-user mode; the + * model, the context and the coding agent must be a row the current recommendation offers; + * the coding agent must be a harness row of the catalogue; `acknowledgedBytes` must be the + * server's own total. + * - No CLI id is compared with a literal here: the coding agent is a catalogue row plus a + * `HarnessFact`. + * + * The service (`createLocalModelService()`), beyond the reducer: + * - Verbs run one at a time on one queue, so two installs sent together cannot both find the + * state free. The view is not on that queue: the window polls it every second. + * - An interrupted step is never recorded as failed, whatever its last call answered (an + * aborted request looks like a network error): the job looks at its own signal before it + * writes `failed`. Cancel writes `paused` itself; a server that is stopping writes nothing, + * so the file still says `installing` and the next start reads that as a paused job. + * - A download's progress is kept in memory, written to the state file every few seconds and + * when the job pauses or fails. The view shows the live figure and its speed. + * - What blocks a session on an installed model (another program on the port, a runtime that + * keeps stopping or did not start, a model whose digest changed) is found live and shown as + * `error` on the view. It is never stored, and the state stays `installed`. + * - `launchRefusalMessage()` is what every path that STARTS a coding agent on the managed + * endpoint asks first: a new session with the model, a model applied to a session, the + * attach that starts an agent in a session that has none, and the rebuild after a reboot. + * The digest is read again for every one of them. Two things are not launches and are not + * asked: a pane that survived a server restart (its agent is already running), and a + * respawn cycle (it types into the agent that runs). + * - With a model installed, `boot()` puts the saved endpoint record back when it is gone (a + * config-backup restore can take it away), starts the runtime unless it was stopped, and + * reads the digest again. + * - A runtime an earlier server left behind (a crash, a forced quit) is never left running + * under a deleted folder or beside a second one: its pid is stored the moment it is + * spawned, a start adopts or replaces it, and Stop, Remove and the boot of anything that + * is not a running installation end it (`RuntimeSupervisor.stop({ previousPid })`). + * - An installed model keeps the runtime VERSION it was installed with: when a later build + * pins another version, the stored one is started for as long as its folder is unpacked. + * - A job whose end could not be written to the state file (a full disk) is not lost: the + * outcome is kept in memory, the view shows it, Resume and a new install start from it, and + * the next verb writes it. The same holds for any `installing` in the file while this + * process runs no job, which is what a restart leaves too. + * - Stop and "Use another port" are refused (409 `sessions-open`) while live sessions use + * the model: their coding agents were given the address when they started. + */ + +import { appendAdminAudit } from '../web/admin-audit.js'; +import { + CLI_INSTALL_TIMEOUT_MS, + forgetResolvedCli, + runInstallCommand, + stockInstallCommand, + tryBeginCliInstall, + type CliInstallResult, + type RunInstallCommandOptions, +} from '../cli-install-runner.js'; +import { getCli } from '../config/cli-registry/registry.js'; +import { getDataDir } from '../config/instance.js'; +import { getLocalModelCatalog } from '../config/local-model-catalog.js'; +import { readCustomModelHosts } from '../custom-model-hosts.js'; +import { + getErrorMessage, + LOCAL_MODEL_ERROR_LABELS, + LOCAL_MODEL_ERROR_NEXT, + LOCAL_MODEL_INSTALL_LABELS, + LOCAL_MODEL_INSTALL_STEPS, + type Candidate, + type CandidateHarness, + type HardwareFacts, + type HarnessFact, + type LocalModelAction, + type LocalModelCatalog, + type LocalModelConflict, + type LocalModelDeveloperTools, + type LocalModelErrorCode, + type LocalModelInstallError, + type LocalModelInstallMessage, + type LocalModelInstallRequest, + type LocalModelInstallState, + type LocalModelInstallStep, + type LocalModelInstallView, + type LocalModelNextStep, + type LocalModelProgress, + type LocalModelRemoveRequest, + type LocalModelResumeRequest, + type LocalModelRuntimePin, + type LocalModelRuntimeRequest, + type LocalModelStateFile, + type Recommendation, + type RecommendOptions, +} from '../types.js'; +import { isCliEntryInstalled, probeStockCliAvailability } from '../utils/cli-installed-probes.js'; +import { + compareModelDigest, + connectLocalModel, + disconnectLocalModel, + ensureLocalEndpoint, + inspectLocalEndpoint, + isLocalModelEndpointId, + LOCAL_MODEL_ENDPOINT_ID, + moveLocalEndpoint, + type ConnectLocalModelInput, + type ConnectResult, + type LocalEndpointInput, + type LocalEndpointInspection, +} from './connect.js'; +import { checkDeveloperTools } from './developer-tools.js'; +import { formatGb } from './format.js'; +import { readHardware } from './hardware.js'; +import { buildHarnessFacts } from './harness-facts.js'; +import { + deleteModel, + getRuntimeVersion, + listOpenAiModels, + listRunning, + pullModel, + readModelDigest, + verifyModel, + type OllamaPsModel, + type OllamaTarget, + type PullOptions, + type PullResult, + type VerifyResult, +} from './ollama-client.js'; +import { localModelPaths, runtimeBinaryForVersion, type LocalModelPaths } from './paths.js'; +import { modelFitsBudget, needBytes, recommend } from './recommend.js'; +import { + chooseRuntimePort, + createRuntimeSupervisor, + ensureRuntimeInstalled, + isRuntimeInstalled, + OLLAMA_RUNTIME_PIN, + removeLocalModelFiles, + removePartialModelBlobs, + RUNTIME_PORT_FIRST, + type ChooseRuntimePortOptions, + type RemoveLocalModelFilesOptions, + type RuntimeInstallOptions, + type RuntimeInstallResult, + type RuntimeStartResult, + type RuntimeStatus, + type RuntimeSupervisor, +} from './runtime.js'; +import { IDLE_LOCAL_MODEL_STATE, mutateLocalModelState, readLocalModelState } from './state-store.js'; + +// ---------- The state table ---------- + +/** One row of the state table. */ +export interface InstallStateRow { + /** What lies on disk in this state, in words. */ + onDisk: string; + /** The state a server restart turns this one into. */ + afterRestart: LocalModelInstallState; + /** What the restart does, in words. */ + restart: string; + /** The state Cancel turns this one into, or `null` when Cancel is refused (409 `wrong-state`). */ + afterCancel: LocalModelInstallState | null; + /** What Cancel does, in words. */ + cancel: string; + /** + * Every action a client may take in this state. The view's `actions` is this list narrowed + * by what is live (`allowedActions()`): `start` only while the runtime is down, `stop` and + * `start-session` only while it runs, `start-session` only when nothing blocks a session. + */ + actions: readonly LocalModelAction[]; +} + +/** The state table of the file header, as data. */ +export const INSTALL_STATE_TABLE: Readonly> = { + idle: { + onDisk: + 'Nothing of ours. The local model folder is absent, or holds leftovers an earlier Remove could not delete; they are never read back.', + afterRestart: 'idle', + restart: 'Nothing happens.', + afterCancel: null, + cancel: 'Refused: no job is running.', + actions: ['install'], + }, + installing: { + onDisk: + 'What the finished steps wrote, plus what the current step has written so far (INSTALL_STEP_TABLE). The state file names the model, the coding agent, the context and the step.', + afterRestart: 'paused', + restart: + 'The job comes back paused at the same step. Nothing is downloaded, started or installed until the person presses Resume.', + afterCancel: 'paused', + cancel: 'The step in flight is interrupted the way its step row says, and the job is paused at that step.', + actions: ['cancel'], + }, + paused: { + onDisk: 'As installing at that step, with nothing in flight.', + afterRestart: 'paused', + restart: 'Stays paused.', + afterCancel: null, + cancel: 'Refused: no job is running.', + actions: ['resume', 'install', 'remove'], + }, + failed: { + onDisk: 'As installing at that step, with nothing in flight.', + afterRestart: 'failed', + restart: 'Stays failed, with the same error.', + afterCancel: null, + cancel: 'Refused: no job is running.', + actions: ['resume', 'install', 'remove'], + }, + installed: { + onDisk: + 'The runtime, its private home and the model. The state file holds the port, the pid, the digest, what the runtime loaded and the previous settings flag. Outside the folder: our endpoint record and the settings flag.', + afterRestart: 'installed', + restart: + 'Stays installed. The runtime is started again unless the person had stopped it, and the digest is read again: a changed digest or a port taken by another program shows as an error on the view and blocks a session, the state stays installed.', + afterCancel: null, + cancel: 'Refused: no job is running.', + actions: ['start-session', 'start', 'stop', 'remove'], + }, +}; + +// ---------- The step table ---------- + +/** One row of the step table. */ +export interface InstallStepRow { + /** What the step does, in words. */ + does: string; + /** What it adds to disk (possibly partly, when it is interrupted). */ + onDisk: string; + /** How Cancel interrupts it. */ + cancel: string; + /** What Resume, and "Try again" after a failure, does: every step is idempotent and is entered again. */ + resume: string; + /** Whether the step reports byte progress (`LocalModelProgress`). */ + progress: boolean; + /** Whether the runtime must run before the step is entered (the service starts it first, idempotently). */ + needsRuntime: boolean; + /** Every error code the step can fail with. `internal` and `disk-full` can come from any step that writes. */ + errors: readonly LocalModelErrorCode[]; +} + +/** The step table of the file header, as data. */ +export const INSTALL_STEP_TABLE: Readonly> = { + runtime: { + does: 'Downloads the pinned runtime archive unless it is unpacked already, compares its sha256, unpacks it, chooses the port once, starts the runtime, reads the measured memory budget from its log and judges the chosen model against it.', + onDisk: + 'runtime/ollama-.tgz.partial while it downloads, runtime/ollama-/ once unpacked, home/ once the runtime started.', + cancel: 'Aborts the download. The partial file stays. A runtime that already runs keeps running.', + resume: + 'Continues the partial file with a Range request, or skips the download when the folder is unpacked. Starting the runtime again is a no-op when it runs.', + progress: true, + needsRuntime: false, + errors: [ + 'runtime-download-failed', + 'runtime-damaged', + 'runtime-unpack-failed', + 'runtime-start-failed', + 'port-in-use', + 'does-not-fit', + 'disk-full', + 'internal', + ], + }, + model: { + does: 'Pulls the model through the runtime, reading the progress stream line by line.', + onDisk: 'models/ with partial blobs, then the finished model.', + cancel: 'Aborts the pull request, which stops the download. The partial blobs stay.', + resume: 'Pulls again. The runtime continues its partial blobs.', + progress: true, + needsRuntime: true, + errors: [ + 'pull-failed', + 'model-not-found', + 'runtime-start-failed', + 'runtime-stopped', + 'port-in-use', + 'disk-full', + 'internal', + ], + }, + verify: { + does: 'Loads the model with one tiny chat, checks that the runtime put all of it on the GPU at the chosen context, runs one tool probe, and stores the digest the runtime reports.', + onDisk: 'Nothing new. The model is loaded into memory.', + cancel: 'Aborts the request in flight.', + resume: 'Runs the three checks again from the first.', + progress: false, + needsRuntime: true, + errors: [ + 'load-failed', + 'does-not-fit', + 'context-not-applied', + 'no-tool-call', + 'probe-unfinished', + 'runtime-start-failed', + 'port-in-use', + 'internal', + ], + }, + agent: { + does: 'Installs the coding agent with its stock install command through the shared CLI install runner, writes the admin audit entry, forgets the cached lookups of its binary and looks for it again.', + onDisk: + 'Whatever the coding agent installer writes, outside the local model folder (its own folder in the home directory). Never removed by us.', + cancel: 'Kills the whole process group of the installer.', + resume: 'Looks for the binary first and runs the installer only when it is still missing.', + progress: false, + needsRuntime: false, + errors: ['harness-failed', 'internal'], + }, + connect: { + does: 'Claims the connection in the state file (with the value the custom model endpoints setting has now), then writes the endpoint record and switches that setting on.', + onDisk: + 'Our record in custom-model-hosts.json and the flag in settings.json: the two writes outside the local model folder. The state file records the claim before either is written, so neither can exist without it.', + cancel: + 'Honoured before the first write. Once the writes began they finish and the job ends installed. A job interrupted between the claim and the writes has a claim with nothing behind it, which Resume fills in and Remove undoes.', + resume: 'Writes both again. Both are idempotent, and the claim keeps the value the setting had before.', + progress: false, + needsRuntime: true, + errors: ['endpoint-id-taken', 'connect-failed', 'internal'], + }, +}; + +// ---------- Events and transitions ---------- + +/** What moves the job. */ +export type InstallJobEvent = + /** A new job: `POST /api/local-model/install` passed its checks. From `idle`, `paused` or `failed`. */ + | { + type: 'install'; + model: { id: string; label: string; context: number }; + cli: { id: string; label: string }; + /** The steps of this job in order (`jobSteps()`). */ + steps: LocalModelInstallStep[]; + } + /** A download moved. Changes `progress` only. */ + | { type: 'progress'; completedBytes: number; totalBytes: number } + /** The current step finished: the next step of `steps`, or `installed` after the last. Clears `progress`. */ + | { type: 'step-done' } + /** The current step failed. */ + | { + type: 'fail'; + code: LocalModelErrorCode; + vars?: Record; + detail?: string; + nextModelId?: string; + } + /** `POST /api/local-model/cancel`. */ + | { type: 'cancel' } + /** The server started and found this state in the file. */ + | { type: 'restart' } + /** `POST /api/local-model/resume`: the same step again. Clears `error`. */ + | { type: 'resume' } + /** `DELETE /api/local-model` finished: back to `idle`, every other member dropped. */ + | { type: 'remove' }; + +export type InstallJobEventType = InstallJobEvent['type']; + +/** + * Which state an event leads to, by current state. A pair that is not listed is REFUSED: the + * reducer returns the object it was given. One entry needs a word: `step-done` in + * `installing` leads to `installing` at the next step, and to `installed` when the step that + * finished was the last of the job's `steps`. + */ +export const INSTALL_TRANSITIONS: Readonly< + Record>>> +> = { + idle: { install: 'installing', restart: 'idle', remove: 'idle' }, + installing: { + progress: 'installing', + 'step-done': 'installing', + fail: 'failed', + cancel: 'paused', + restart: 'paused', + }, + paused: { resume: 'installing', install: 'installing', remove: 'idle', restart: 'paused' }, + failed: { resume: 'installing', install: 'installing', remove: 'idle', restart: 'failed' }, + installed: { remove: 'idle', restart: 'installed' }, +}; + +/** The steps a job runs, in order: all of `LOCAL_MODEL_INSTALL_STEPS`, without `agent` when the coding agent is installed. Pure. */ +export function jobSteps(input: { agentInstalled: boolean }): LocalModelInstallStep[] { + return LOCAL_MODEL_INSTALL_STEPS.filter((step) => step !== 'agent' || !input.agentInstalled); +} + +/** A byte count as a download reports it: a whole number that is not negative. */ +const isByteCount = (value: number): boolean => Number.isSafeInteger(value) && value >= 0; + +/** + * The state file after `event`. Pure: no IO, no clock (`updatedAt` is the state store's + * business). Refuses an event the transition table does not list for the current state by + * returning `current` itself. Keeps `runtime` and `connection` across a new install, because + * what they record is still on disk; `remove` drops everything. + * + * An event that is listed but cannot be applied is refused the same way: an `install` whose + * steps are not a job's, a `progress` that is not two byte counts, a `step-done` on a state + * without a step, and a last `step-done` on a job that has no runtime recorded. + */ +export function reduceInstallJob(current: LocalModelStateFile, event: InstallJobEvent): LocalModelStateFile { + if (INSTALL_TRANSITIONS[current.state][event.type] === undefined) return current; + switch (event.type) { + case 'install': { + const steps = event.steps.filter((step, index) => event.steps.indexOf(step) === index); + if ( + steps.length === 0 || + steps.length !== event.steps.length || + steps.some((step) => !LOCAL_MODEL_INSTALL_STEPS.includes(step)) + ) { + return current; + } + // What the replaced job left of ANOTHER model is deleted before the new one is pulled. + const pending = current.model && current.model.id !== event.model.id ? current.model.id : current.replacedModelId; + const replacedModelId = pending !== undefined && pending !== event.model.id ? pending : undefined; + return { + schema: 1, + state: 'installing', + step: steps[0], + steps, + model: { id: event.model.id, label: event.model.label, context: event.model.context }, + ...(replacedModelId ? { replacedModelId } : {}), + cli: { id: event.cli.id, label: event.cli.label }, + ...(current.runtime ? { runtime: current.runtime } : {}), + ...(current.connection ? { connection: current.connection } : {}), + }; + } + case 'progress': { + if (!isByteCount(event.completedBytes) || !isByteCount(event.totalBytes)) return current; + return { ...current, progress: { completedBytes: event.completedBytes, totalBytes: event.totalBytes } }; + } + case 'step-done': { + const steps = current.steps ?? []; + const index = current.step ? steps.indexOf(current.step) : -1; + if (index === -1) return current; + if (index < steps.length - 1) { + const { progress: _progress, ...rest } = current; + return { ...rest, step: steps[index + 1] }; + } + // The last step: the model is connected. + if (!current.model || !current.cli || !current.runtime) return current; + return { + schema: 1, + state: 'installed', + model: current.model, + cli: current.cli, + runtime: current.runtime, + ...(current.loaded ? { loaded: current.loaded } : {}), + ...(current.connection ? { connection: current.connection } : {}), + }; + } + case 'fail': + return { + ...current, + state: 'failed', + error: { + code: event.code, + ...(event.vars && Object.keys(event.vars).length > 0 ? { vars: event.vars } : {}), + ...(event.detail ? { detail: event.detail } : {}), + ...(event.nextModelId ? { nextModelId: event.nextModelId } : {}), + }, + }; + case 'cancel': + return { ...current, state: 'paused' }; + case 'restart': + // Only a running job changes: the restart interrupted it, and it waits for a click. + return current.state === 'installing' ? { ...current, state: 'paused' } : current; + case 'resume': { + const { error: _error, ...rest } = current; + return { ...rest, state: 'installing' }; + } + case 'remove': + return { schema: 1, state: 'idle' }; + } +} + +// ---------- The view ---------- + +/** What is live and not in the state file, for the view. Gathered by the service on every GET. */ +export interface InstallLiveFacts { + /** The runtime child is alive (`RuntimeStatus.running`). */ + running: boolean; + /** The supervisor stopped restarting the runtime (`RuntimeStatus.gaveUp`): the error `runtime-stopped` in state `installed`. */ + gaveUp: boolean; + /** Another program holds the stored port: the error `port-in-use` in state `installed`. */ + portInUse: boolean; + /** The last start of the runtime failed for another reason: the error `runtime-start-failed` in state `installed`, with what the runtime said. */ + startFailed?: { detail?: string }; + /** The stored digest against the runtime's current one (`compareModelDigest()`). `changed` is the error `model-changed`. Absent while not installed. */ + digest?: 'same' | 'changed' | 'unknown'; + developerTools: LocalModelDeveloperTools; + /** The pinned runtime is unpacked on disk. */ + runtimeInstalled: boolean; + /** The current download speed. Not stored, so absent after a restart. */ + bytesPerSecond?: number; + /** For an error whose next step is `next-model`: the model to offer, from the current recommendation. */ + nextModel?: { modelId: string; modelLabel: string }; + pin: LocalModelRuntimePin; + /** `LocalModelPaths.root`, for the "where it goes" line. */ + folder: string; +} + +/** An error code with its sentence, its slots and its one next step. Pure. */ +export function buildInstallError( + code: LocalModelErrorCode, + extra: { + vars?: Record; + detail?: string; + nextModel?: { modelId: string; modelLabel: string }; + /** The next step in the state the error is shown in, when it is not the code's default. */ + next?: LocalModelNextStep; + } = {} +): LocalModelInstallError { + const next = extra.next ?? LOCAL_MODEL_ERROR_NEXT[code]; + return { + code, + message: { + text: LOCAL_MODEL_ERROR_LABELS[code], + ...(extra.vars && Object.keys(extra.vars).length > 0 ? { vars: extra.vars } : {}), + }, + next, + ...(next === 'next-model' && extra.nextModel ? { nextModel: extra.nextModel } : {}), + ...(extra.detail ? { detail: extra.detail } : {}), + }; +} + +/** + * What blocks a session in state `installed`, found live and never stored: another program + * on the port, a runtime that keeps stopping, a runtime that did not start, a model that + * changed. `undefined` when nothing does, which includes a runtime the person stopped. Pure. + */ +export function installedBlocker( + state: LocalModelStateFile, + live: InstallLiveFacts +): LocalModelInstallError | undefined { + if (state.state !== 'installed') return undefined; + if (!live.running && live.portInUse) { + return buildInstallError('port-in-use', { vars: { port: String(state.runtime?.port ?? '') } }); + } + if (!live.running && live.gaveUp) return buildInstallError('runtime-stopped'); + if (!live.running && live.startFailed) { + // Start is the verb an installed model has: it tries the start again. + return buildInstallError('runtime-start-failed', { + next: 'start', + ...(live.startFailed.detail ? { detail: live.startFailed.detail } : {}), + }); + } + if (live.digest === 'changed') return buildInstallError('model-changed'); + return undefined; +} + +/** The actions the server accepts right now: the state row's list narrowed by what is live. Pure. */ +export function allowedActions(state: LocalModelStateFile, live: InstallLiveFacts): LocalModelAction[] { + const actions = INSTALL_STATE_TABLE[state.state].actions; + if (state.state !== 'installed') return [...actions]; + const blocked = installedBlocker(state, live) !== undefined; + return actions.filter((action) => { + if (action === 'start') return !live.running; + if (action === 'stop') return live.running; + if (action === 'start-session') return live.running && !blocked; + return true; + }); +} + +/** How far a download is, as the window prints it. Pure. */ +function buildProgress( + progress: { completedBytes: number; totalBytes: number }, + bytesPerSecond: number | undefined +): LocalModelProgress { + const view: LocalModelProgress = { + completedBytes: progress.completedBytes, + totalBytes: progress.totalBytes, + completedGb: formatGb(progress.completedBytes), + totalGb: formatGb(progress.totalBytes), + }; + if (bytesPerSecond !== undefined && Number.isFinite(bytesPerSecond) && bytesPerSecond >= 0) { + view.bytesPerSecond = Math.round(bytesPerSecond); + view.mbPerSecond = String(Math.round(bytesPerSecond / 1_000_000)); + const left = progress.totalBytes - progress.completedBytes; + if (bytesPerSecond > 0 && left >= 0) view.secondsLeft = Math.ceil(left / bytesPerSecond); + } + return view; +} + +/** The `install` object of the GET. Pure: every size text comes from `formatGb()`, every sentence from `LOCAL_MODEL_INSTALL_LABELS`. */ +export function buildInstallView(state: LocalModelStateFile, live: InstallLiveFacts): LocalModelInstallView { + const runtimeDownload = live.runtimeInstalled ? 0 : live.pin.bytes; + const view: LocalModelInstallView = { + state: state.state, + running: live.running, + developerTools: live.developerTools, + runtime: { + version: live.pin.version, + installed: live.runtimeInstalled, + downloadBytes: runtimeDownload, + downloadGb: formatGb(runtimeDownload), + unpackedGb: formatGb(live.pin.unpackedBytes), + }, + folder: live.folder, + actions: allowedActions(state, live), + }; + if (state.state === 'idle') return view; + + if (state.model) { + view.model = { id: state.model.id, label: state.model.label }; + view.context = state.model.context; + } + if (state.cli) { + view.cli = state.cli.id; + view.cliLabel = state.cli.label; + } + if (state.runtime) view.port = state.runtime.port; + + if (state.state === 'installed') { + if (state.loaded) { + view.loaded = { + context: state.loaded.context, + memoryBytes: state.loaded.sizeBytes, + memoryGb: formatGb(state.loaded.sizeBytes), + onGpu: state.loaded.sizeVramBytes === state.loaded.sizeBytes, + }; + } + if (state.connection) view.endpointId = state.connection.endpointId; + const blocker = installedBlocker(state, live); + if (blocker) view.error = blocker; + return view; + } + + if (state.step) view.step = state.step; + if (state.steps) view.steps = [...state.steps]; + // A download whose total is not known yet has nothing to show. + if (state.progress && state.progress.totalBytes > 0) { + view.progress = buildProgress(state.progress, state.state === 'installing' ? live.bytesPerSecond : undefined); + } + if (state.state === 'failed' && state.error) { + view.error = buildInstallError(state.error.code, { + vars: state.error.vars, + detail: state.error.detail, + nextModel: live.nextModel, + // A job has no Start verb: what helps there is to run the step again. + ...(LOCAL_MODEL_ERROR_NEXT[state.error.code] === 'start' ? { next: 'retry' as const } : {}), + }); + } + return view; +} + +/** A sentence of the install as plain English text, its slots filled. For an API error message and the log. Pure. */ +export function formatInstallMessage(message: LocalModelInstallMessage): string { + const vars = message.vars ?? {}; + return message.text.replace(/\{([a-zA-Z]+)\}/g, (slot, key: string) => + Object.prototype.hasOwnProperty.call(vars, key) ? vars[key] : slot + ); +} + +// ---------- The service ---------- + +/** Who asked, for the admin audit entry of the coding agent install. */ +export interface LocalModelActor { + username: string; + ip?: string; +} + +/** How a verb ended: the view after it, a refusal with its sentence, or a 409 with its conflict. */ +export type LocalModelVerbResult = + | { ok: true; view: LocalModelInstallView } + /** 400: the request is not something the server offers. 507: nothing could be started because the disk is full. */ + | { ok: false; status: 400 | 507; message: LocalModelInstallMessage } + | { ok: false; status: 409; conflict: LocalModelConflict }; + +/** + * Everything the service calls, so a test spawns, fetches and downloads nothing. Each member + * defaults to the real module. The members named like a function of another module have + * exactly that function's signature. + */ +export interface LocalModelServiceDeps { + readHardware: () => Promise; + getCatalog: () => LocalModelCatalog; + buildHarnessFacts: (catalog: LocalModelCatalog) => Promise; + recommend: ( + hw: HardwareFacts, + catalog: LocalModelCatalog, + harnessFacts: readonly HarnessFact[], + options?: RecommendOptions + ) => Recommendation; + /** `modelFitsBudget()` of src/local-model/recommend.ts: the judgement of the `runtime` step, with the measured budget. */ + modelFitsBudget: ( + hw: HardwareFacts, + catalog: LocalModelCatalog, + modelId: string, + context: number, + options?: RecommendOptions + ) => boolean; + // src/local-model/runtime.ts + isRuntimeInstalled: (pin?: LocalModelRuntimePin, paths?: LocalModelPaths) => Promise; + ensureRuntimeInstalled: (options?: RuntimeInstallOptions) => Promise; + chooseRuntimePort: (options?: ChooseRuntimePortOptions) => Promise; + removeLocalModelFiles: (options: RemoveLocalModelFilesOptions) => Promise; + supervisor: RuntimeSupervisor; + // src/local-model/ollama-client.ts + pullModel: (target: OllamaTarget, modelId: string, options?: PullOptions) => Promise; + verifyModel: (target: OllamaTarget, modelId: string, context: number, signal?: AbortSignal) => Promise; + readModelDigest: (target: OllamaTarget, modelId: string, signal?: AbortSignal) => Promise; + listOpenAiModels: (target: OllamaTarget, signal?: AbortSignal) => Promise; + listRunning: (target: OllamaTarget, signal?: AbortSignal) => Promise; + getRuntimeVersion: (target: OllamaTarget, signal?: AbortSignal) => Promise; + deleteModel: (target: OllamaTarget, modelId: string, signal?: AbortSignal) => Promise; + /** `removePartialModelBlobs()` of src/local-model/runtime.ts: what an interrupted download left in the model store. Resolves with the bytes freed. */ + removePartialModelBlobs: (paths?: LocalModelPaths) => Promise; + // src/local-model/developer-tools.ts + checkDeveloperTools: () => Promise; + // src/cli-install-runner.ts + stockInstallCommand: (id: string) => string | undefined; + tryBeginCliInstall: (id: string) => (() => void) | null; + runInstallCommand: (command: string, options?: RunInstallCommandOptions) => Promise; + /** Forgets the cached lookups of the CLI's binaries, then says whether its binary is found now. */ + recheckCliInstalled: (cli: string) => Promise; + /** Writes one admin audit line (`appendAdminAudit()`). */ + audit: (entry: { admin: string; action: string; target: string; ip?: string; detail?: unknown }) => Promise; + // src/local-model/connect.ts + /** What is saved under the reserved id and what the settings flag is, read before the connection is claimed. Writes nothing. */ + inspectLocalEndpoint: (input: LocalEndpointInput) => Promise; + connectLocalModel: (input: ConnectLocalModelInput) => Promise; + moveLocalEndpoint: (port: number) => Promise; + /** Puts our endpoint record back when it is gone (after a config-backup restore). True when it was added. */ + ensureLocalEndpoint: (input: LocalEndpointInput) => Promise; + disconnectLocalModel: (input: { flagWasEnabled: boolean }) => Promise; + /** Whether a saved endpoint already has the reserved id (asked before a first install, so the answer comes before the download). */ + isEndpointIdTaken: () => Promise; + // src/local-model/state-store.ts + readState: () => Promise; + mutateState: ( + fn: (current: LocalModelStateFile) => LocalModelStateFile | undefined + ) => Promise<{ before: LocalModelStateFile; after: LocalModelStateFile | undefined }>; + pin: LocalModelRuntimePin; + /** Default `localModelPaths(pin.version)`, built when it is first needed. */ + paths?: LocalModelPaths; + /** The clock of the progress throttle and the digest re-read, in milliseconds. */ + now: () => number; +} + +/** The install as the routes and the server see it. One per process. */ +export interface LocalModelService { + /** The `install` object of the GET. Never throws. */ + view(): Promise; + /** + * Whether a session may be started on the saved endpoint `endpointId` right now. `undefined`: + * yes, which is also the answer for every endpoint that is not this feature's. Otherwise the + * error that refuses the launch. While the runtime child Codeman started is not alive + * (whatever answers on the port then is not ours), the reason is the one that is known: + * `port-in-use`, `runtime-stopped`, `runtime-start-failed`, else `runtime-not-running`. + * While it runs, the digest is read again: `model-changed` when it is not the stored one or + * the model is gone. Every path that starts a coding agent on the managed endpoint asks + * this first (see the file header for the list). Never throws. + */ + launchRefusal(endpointId: string): Promise; + /** + * `launchRefusal()` as one plain English sentence, plus the one check it does not make: on + * the managed endpoint only the model that was installed and checked may be started, so a + * model somebody pulled through the runtime's port is not launched. `undefined`: start it. + * What the session launch paths call. Never throws. + */ + launchRefusalMessage(endpointId: string, modelId: string): Promise; + /** What the GET passes to `recommend()`: the stored measured total and the runtime download that is still needed. */ + recommendOptions(): Promise; + /** `POST /api/local-model/install`. Answers once the job is STARTED; the job runs on in the server. */ + install(request: LocalModelInstallRequest, actor: LocalModelActor): Promise; + /** `POST /api/local-model/cancel`. Answers once the step in flight was interrupted. */ + cancel(): Promise; + /** + * `POST /api/local-model/resume`. `liveSessionNames` as for `remove()`: with `newPort` and a + * connection of ours, not empty answers 409 `sessions-open`. + */ + resume( + request: LocalModelResumeRequest, + actor: LocalModelActor, + liveSessionNames?: readonly string[] + ): Promise; + /** + * `POST /api/local-model/runtime`. `liveSessionNames` as for `remove()`: Stop and a start on + * another port answer 409 `sessions-open` while one of them uses the model, because their + * coding agents keep calling the address they were started with. + */ + runtime(request: LocalModelRuntimeRequest, liveSessionNames?: readonly string[]): Promise; + /** + * `DELETE /api/local-model`. `liveSessionNames` is what `sessionNamesUsingEndpoint()` found: + * not empty answers 409 `sessions-open` while a connection of ours exists. Else: stop the + * runtime, remove our endpoint record, restore the settings flag when this feature switched + * it on and no other endpoint is left, delete the runtime and the private home and (when + * asked) the models. The coding agent stays. A refused write of the endpoint store or of the + * settings REJECTS, before any file is deleted. + */ + remove( + request: LocalModelRemoveRequest, + liveSessionNames: readonly string[], + actor?: LocalModelActor + ): Promise; + /** + * Called once when the server starts. An interrupted job becomes `paused`. With a model + * installed: the saved endpoint record is put back when it is gone, a runtime that was not + * stopped is started (an own orphan on the port is adopted or replaced, another listener is + * `port-in-use`) and the digest is read again. In every other case a runtime an earlier + * server left behind is ended: nothing would use it, and it holds the model's memory. + * Never throws, and never keeps the server from starting. + */ + boot(): Promise; + /** Called when the server stops, beside `stopDeepSeekWeb()`: aborts a job in flight (it reads `paused` at the next boot) and stops the runtime. */ + shutdown(): Promise; + /** What the runtime supervisor reports through `onChange`: the pid of a child the moment it is spawned, a new pid after a restart, the budget line. Stored, never thrown. */ + onRuntimeChange(status: RuntimeStatus): void; + /** Resolves once no job is in flight. For `shutdown()` and for tests. */ + idle(): Promise; +} + +/** How often a download's progress is written to the state file. The window reads the live figure, not the file. */ +const PROGRESS_FLUSH_MS = 5_000; +/** How long a digest read stays good for the view. A session launch always reads it again. */ +const DIGEST_RECHECK_MS = 30_000; +/** How long a read of the runtime's listings may take. */ +const RUNTIME_READ_TIMEOUT_MS = 5_000; +/** The window a download speed is measured over when the source reports none. */ +const SPEED_WINDOW_MS = 5_000; +/** How much of an installer's output is kept as the error detail. */ +const INSTALLER_DETAIL_CHARS = 1_000; + +/** How a step ended. */ +type StepOutcome = + | { kind: 'done' } + | { kind: 'cancelled' } + | { kind: 'fail'; code: LocalModelErrorCode; vars?: Record; detail?: string; nextModelId?: string }; + +const DONE: StepOutcome = { kind: 'done' }; +const CANCELLED: StepOutcome = { kind: 'cancelled' }; + +const baseUrlOf = (port: number): OllamaTarget => ({ baseUrl: `http://127.0.0.1:${port}` }); + +function conflict( + reason: LocalModelConflict['reason'], + extra: Partial = {}, + vars?: Record +): LocalModelVerbResult { + const text = { + 'job-running': LOCAL_MODEL_INSTALL_LABELS.conflictJobRunning, + 'already-installed': LOCAL_MODEL_INSTALL_LABELS.conflictAlreadyInstalled, + 'size-changed': LOCAL_MODEL_INSTALL_LABELS.conflictSizeChanged, + 'sessions-open': LOCAL_MODEL_INSTALL_LABELS.conflictSessionsOpen, + 'wrong-state': LOCAL_MODEL_INSTALL_LABELS.conflictWrongState, + }[reason]; + return { ok: false, status: 409, conflict: { reason, message: { text, ...(vars ? { vars } : {}) }, ...extra } }; +} + +/** A supervisor that is built when it is first started, so a server that never installs a model builds none. */ +function lazySupervisor(build: () => RuntimeSupervisor): RuntimeSupervisor { + let instance: RuntimeSupervisor | undefined; + return { + start: (options) => (instance ??= build()).start(options), + stop: async (options) => { + // Built for a stored pid too: a runtime an earlier server left behind has no instance here. + if (instance || options?.previousPid !== undefined) await (instance ??= build()).stop(options); + }, + status: () => (instance ? instance.status() : { running: false, restarts: 0, gaveUp: false }), + }; +} + +/** Whether a write failed for lack of space. */ +function isDiskFullError(err: unknown): boolean { + const code = typeof err === 'object' && err !== null ? (err as NodeJS.ErrnoException).code : undefined; + return code === 'ENOSPC' || code === 'EDQUOT' || /no space left on device/i.test(getErrorMessage(err)); +} + +/** How a job ended, as the `fail` event carries it. */ +type JobFailure = Omit, 'type'>; + +/** Forgets the cached lookups of a CLI's binaries the way the CLI registry routes do, then probes it again. */ +async function recheckCliInstalledNow(cli: string): Promise { + const entry = getCli(cli); + if (!entry) return false; + forgetResolvedCli(cli, entry.discovery.binaries); + const stockAvailability = await probeStockCliAvailability(); + return isCliEntryInstalled(entry, stockAvailability); +} + +class LocalModelServiceImpl implements LocalModelService { + private readonly deps: LocalModelServiceDeps; + /** The one job of this process, while it runs. */ + private job: { controller: AbortController; promise: Promise } | undefined; + /** Verbs run one at a time, so two installs cannot both find the state free. */ + private verbChain: Promise = Promise.resolve(); + private recovery: Promise | undefined; + private stopping = false; + /** A verb wrote `installing` and is about to start the job: the view must not take that for a job that is gone. */ + private jobStarting = false; + /** + * How the last job ended when the state file could not be told (a full disk): the failure, + * or nothing for a pause, and the download as it stood. Shown from memory and written by + * the next verb. + */ + private unrecorded: { failure?: JobFailure; progress?: { completedBytes: number; totalBytes: number } } | undefined; + /** The download in flight, as last reported. The state file gets it every few seconds. */ + private liveProgress: { completedBytes: number; totalBytes: number; bytesPerSecond?: number } | undefined; + private lastProgressFlush = 0; + private speedSamples: Array<{ at: number; bytes: number }> = []; + /** The last start found another program on the stored port. */ + private portInUse = false; + /** The last start failed for another reason, with what the runtime said. Cleared by a start that works and by Stop. */ + private startFailed: { detail?: string } | undefined; + private digestStatus: 'same' | 'changed' | 'unknown' | undefined; + private digestCheckedAt = 0; + + constructor(deps: Partial) { + const pin = deps.pin ?? OLLAMA_RUNTIME_PIN; + this.deps = { + readHardware: () => readHardware(), + getCatalog: getLocalModelCatalog, + buildHarnessFacts: (catalog) => buildHarnessFacts(catalog), + recommend, + modelFitsBudget, + isRuntimeInstalled, + ensureRuntimeInstalled, + chooseRuntimePort, + removeLocalModelFiles, + supervisor: lazySupervisor(() => + createRuntimeSupervisor({ pin, paths: this.paths(), onChange: (status) => this.onRuntimeChange(status) }) + ), + pullModel, + verifyModel, + readModelDigest, + listOpenAiModels, + listRunning, + getRuntimeVersion, + deleteModel, + removePartialModelBlobs, + checkDeveloperTools: () => checkDeveloperTools(), + stockInstallCommand, + tryBeginCliInstall, + runInstallCommand, + recheckCliInstalled: recheckCliInstalledNow, + audit: (entry) => appendAdminAudit({ ...entry, detail: entry.detail as Record | undefined }), + inspectLocalEndpoint, + connectLocalModel, + moveLocalEndpoint, + ensureLocalEndpoint, + disconnectLocalModel, + isEndpointIdTaken: async () => + (await readCustomModelHosts(getDataDir())).some((host) => host.id === LOCAL_MODEL_ENDPOINT_ID), + readState: readLocalModelState, + mutateState: mutateLocalModelState, + now: () => Date.now(), + ...deps, + pin, + }; + } + + // ---------- Small helpers ---------- + + private paths(): LocalModelPaths { + return this.deps.paths ?? localModelPaths(this.deps.pin.version); + } + + /** Runs `fn` after every verb that was asked before it. */ + private exclusive(fn: () => Promise): Promise { + const run = this.verbChain.then(fn); + this.verbChain = run.catch(() => {}); + return run; + } + + /** Applies one event of the running job to the stored state. False when the reducer refused it. */ + private async dispatch(event: InstallJobEvent): Promise { + const { after } = await this.deps.mutateState((current) => { + const next = reduceInstallJob(current, event); + return next === current ? undefined : next; + }); + return after !== undefined; + } + + /** + * `state` as it stands once a job that is over is counted as over: `installing` in the file + * while this process runs no job is a job that ended without the file knowing. It is + * `failed` with the outcome kept in memory, or `paused` when there is none (a restart, a + * Cancel whose write was refused). A paused or failed job with an outcome in memory is a + * Resume that could not be written. Pure apart from reading `this.job` and `this.unrecorded`. + */ + private settled(state: LocalModelStateFile): LocalModelStateFile { + if (this.job) return state; + const failure = this.unrecorded?.failure; + let next: LocalModelStateFile; + if (state.state === 'installing') { + next = failure + ? reduceInstallJob(state, { type: 'fail', ...failure }) + : reduceInstallJob(state, { type: 'restart' }); + } else if (failure && (state.state === 'paused' || state.state === 'failed')) { + // A Resume that could not even be written (the disk is full): the job did not start, + // and it says why instead of looking as if the click did nothing. + next = reduceInstallJob({ ...state, state: 'installing' }, { type: 'fail', ...failure }); + } else { + return state; + } + const progress = this.unrecorded?.progress; + return progress ? { ...next, progress } : next; + } + + /** The stored state as a view or a verb must see it: settled, unless a verb is just starting a job. */ + private async currentState(): Promise { + const state = await this.deps.readState(); + return this.jobStarting ? state : this.settled(state); + } + + /** + * Applies one event of a VERB to the stored state, after settling it (`settled()`), in one + * write. False when the reducer refused the event; the settlement is written all the same. + */ + private async verbDispatch(event: InstallJobEvent): Promise { + let applied = false; + let settledNow = false; + await this.deps.mutateState((current) => { + const base = this.settled(current); + settledNow = base !== current; + const next = reduceInstallJob(base, event); + applied = next !== base; + if (applied) return next; + return settledNow ? base : undefined; + }); + if (settledNow || applied) this.unrecorded = undefined; + return applied; + } + + /** + * Writes the settlement alone, so the file says what the view already shows. Best effort: a + * write that is refused (the disk is still full) changes nothing, the outcome stays in + * memory and the next verb tries again. + */ + private async settle(): Promise { + try { + let settledNow = false; + await this.deps.mutateState((current) => { + const base = this.settled(current); + settledNow = base !== current; + return settledNow ? base : undefined; + }); + if (settledNow) this.unrecorded = undefined; + } catch (err) { + console.warn(`[local-model] Could not write the state of the install: ${getErrorMessage(err)}`); + } + } + + /** Writes facts a step learned. Never moves the state, and writes nothing once the job's state is gone. */ + private async record(change: (current: LocalModelStateFile) => LocalModelStateFile | undefined): Promise { + await this.deps.mutateState((current) => (current.state === 'idle' ? undefined : change(current))); + } + + /** + * A job the state file still calls `installing` when this process has none is one a restart + * interrupted: it becomes `paused`. Written once per process, before the first view or verb, + * whether or not `boot()` ran yet. This is the one write a GET makes; a job that ends later + * without the file knowing is shown from memory (`settled()`) and written by the next verb. + */ + private recovered(): Promise { + this.recovery ??= this.settle(); + return this.recovery; + } + + private async isRuntimeOnDisk(): Promise { + try { + return await this.deps.isRuntimeInstalled(this.deps.pin, this.paths()); + } catch { + return false; + } + } + + /** + * The runtime version to start: the pinned one when it is unpacked, else the version the + * state file records when THAT is still unpacked (a model installed by an earlier build + * with another pin keeps running), else the pinned one, whose start then says what is + * missing. + */ + private async runtimeVersionToRun(state: LocalModelStateFile): Promise { + const pinned = this.deps.pin.version; + const stored = state.runtime?.version; + if (!stored || stored === pinned || (await this.isRuntimeOnDisk())) return pinned; + try { + const paths = this.paths(); + const binary = runtimeBinaryForVersion(paths, stored); + if (await this.deps.isRuntimeInstalled({ ...this.deps.pin, version: stored }, { ...paths, binary })) { + return stored; + } + } catch { + /* not a version this build can name: the pinned one is tried */ + } + return pinned; + } + + private errorVars(code: LocalModelErrorCode, state: LocalModelStateFile): Record | undefined { + const model = state.model?.label ?? ''; + switch (code) { + case 'port-in-use': + return { port: String(state.runtime?.port ?? RUNTIME_PORT_FIRST) }; + case 'model-not-found': + case 'does-not-fit': + case 'load-failed': + case 'no-tool-call': + case 'probe-unfinished': + return { model }; + case 'harness-failed': + return { cli: state.cli?.label ?? '' }; + case 'endpoint-id-taken': + return { id: LOCAL_MODEL_ENDPOINT_ID }; + default: + return undefined; + } + } + + /** + * The model to offer after `failed` did not work: the best one the current recommendation + * offers that is another model. With `smallerThan` (the memory the failed model needs, for + * `does-not-fit`) only a model that needs LESS is offered: one that needs more would fail + * the same way. `undefined` when there is none. + */ + private async nextModelId( + failed: { id: string; context: number }, + options?: RecommendOptions, + rule: { needsLessMemory?: boolean } = {} + ): Promise { + try { + const hw = await this.deps.readHardware(); + const catalog = this.deps.getCatalog(); + const facts = await this.deps.buildHarnessFacts(catalog); + const rec = this.deps.recommend(hw, catalog, facts, options ?? (await this.recommendOptions())); + const rows = [rec.pick, ...rec.alternatives].filter( + (row): row is Candidate => row !== undefined && row.modelId !== failed.id + ); + if (!rule.needsLessMemory) return rows[0]?.modelId; + // What the failed model needs: its row when the recommendation still has it, else the + // catalogue's figures for it at the context that was chosen. + const failedRow = [rec.pick, ...rec.alternatives].find((row) => row?.modelId === failed.id); + const entry = catalog.models.find((model) => model.id === failed.id); + const limit = + failedRow && failedRow.context === failed.context + ? failedRow.needBytes + : entry + ? needBytes(entry, failed.context) + : undefined; + if (limit === undefined || !Number.isFinite(limit)) return undefined; + return rows.find((row) => row.needBytes < limit)?.modelId; + } catch { + return undefined; + } + } + + // ---------- Progress ---------- + + private onProgress(progress: { completedBytes: number; totalBytes: number; bytesPerSecond?: number }): void { + const now = this.deps.now(); + let bytesPerSecond = progress.bytesPerSecond; + if (bytesPerSecond === undefined) { + // The archive download reports bytes only: the speed is what arrived over the last seconds. + this.speedSamples.push({ at: now, bytes: progress.completedBytes }); + while (this.speedSamples.length > 2 && now - this.speedSamples[1].at >= SPEED_WINDOW_MS) + this.speedSamples.shift(); + const first = this.speedSamples[0]; + const elapsed = now - first.at; + if (elapsed >= 1_000 && progress.completedBytes >= first.bytes) { + bytesPerSecond = ((progress.completedBytes - first.bytes) * 1_000) / elapsed; + } + } + this.liveProgress = { + completedBytes: progress.completedBytes, + totalBytes: progress.totalBytes, + ...(bytesPerSecond !== undefined ? { bytesPerSecond } : {}), + }; + if (now - this.lastProgressFlush >= PROGRESS_FLUSH_MS) { + this.lastProgressFlush = now; + void this.flushProgress().catch(() => {}); + } + } + + private async flushProgress(): Promise { + const progress = this.liveProgress; + if (!progress) return; + await this.dispatch({ + type: 'progress', + completedBytes: progress.completedBytes, + totalBytes: progress.totalBytes, + }); + } + + /** The download as it stands, for a job that ends: what a paused or failed job keeps. */ + private progressNow(): { completedBytes: number; totalBytes: number } | undefined { + const progress = this.liveProgress; + return progress ? { completedBytes: progress.completedBytes, totalBytes: progress.totalBytes } : undefined; + } + + private clearProgress(): void { + this.liveProgress = undefined; + this.speedSamples = []; + this.lastProgressFlush = 0; + } + + // ---------- The runtime ---------- + + /** Starts the runtime on the stored port (a no-op when it runs with this context) and stores what the start learned. */ + private async startRuntime(state: LocalModelStateFile): Promise { + if (!state.runtime || !state.model) { + return { ok: false, code: 'runtime-start-failed', detail: 'No port was chosen for the runtime yet.' }; + } + const { port } = state.runtime; + const version = await this.runtimeVersionToRun(state); + let result: RuntimeStartResult; + try { + result = await this.deps.supervisor.start({ + port, + contextLength: state.model.context, + ...(state.runtime.pid !== undefined ? { previousPid: state.runtime.pid } : {}), + ...(version !== this.deps.pin.version ? { version } : {}), + }); + } catch (err) { + result = { ok: false, code: 'runtime-start-failed', detail: getErrorMessage(err) }; + } + this.portInUse = !result.ok && result.code === 'port-in-use'; + // Kept, so an installed model whose runtime does not start says so instead of only "stopped". + this.startFailed = + !result.ok && result.code === 'runtime-start-failed' + ? { ...(result.detail ? { detail: result.detail } : {}) } + : undefined; + if (result.ok) { + const { pid, measuredTotalBytes } = result; + await this.record((current) => + current.runtime + ? { + ...current, + runtime: { + ...current.runtime, + version, + pid, + ...(measuredTotalBytes !== undefined ? { measuredTotalBytes } : {}), + }, + } + : undefined + ); + } + return result; + } + + private startFailure(result: Extract, state: LocalModelStateFile): StepOutcome { + return { + kind: 'fail', + code: result.code, + vars: this.errorVars(result.code, state), + ...(result.detail ? { detail: result.detail } : {}), + }; + } + + /** Reads the model's digest from the running runtime and compares it with the stored one. */ + private async refreshDigest(state: LocalModelStateFile): Promise<'same' | 'changed' | 'unknown'> { + let current: string | undefined; + if (state.runtime && state.model && this.deps.supervisor.status().running) { + try { + current = await this.deps.readModelDigest( + baseUrlOf(state.runtime.port), + state.model.id, + AbortSignal.timeout(RUNTIME_READ_TIMEOUT_MS) + ); + } catch { + current = undefined; + } + } + this.digestStatus = compareModelDigest(state.model?.digest, current); + this.digestCheckedAt = this.deps.now(); + return this.digestStatus; + } + + onRuntimeChange(status: RuntimeStatus): void { + // The pid is stored from the moment the child is spawned, not only once it answers: a + // server that is killed in between must still find its own runtime at the next start. + if (status.pid === undefined) return; + const { pid, measuredTotalBytes } = status; + void this.record((current) => { + if (!current.runtime) return undefined; + const samePid = current.runtime.pid === pid; + const sameTotal = measuredTotalBytes === undefined || current.runtime.measuredTotalBytes === measuredTotalBytes; + if (samePid && sameTotal) return undefined; + return { + ...current, + runtime: { ...current.runtime, pid, ...(measuredTotalBytes !== undefined ? { measuredTotalBytes } : {}) }, + }; + }).catch(() => {}); + } + + // ---------- The view ---------- + + private async liveFacts(state: LocalModelStateFile): Promise { + const status = this.deps.supervisor.status(); + let developerTools: LocalModelDeveloperTools = 'unknown'; + try { + developerTools = await this.deps.checkDeveloperTools(); + } catch { + /* said as unknown: the sentence is not shown */ + } + let nextModel: InstallLiveFacts['nextModel']; + const nextModelId = state.error?.nextModelId; + if (nextModelId) { + try { + const entry = this.deps.getCatalog().models.find((model) => model.id === nextModelId); + if (entry) nextModel = { modelId: entry.id, modelLabel: entry.label }; + } catch { + /* no model is offered then */ + } + } + return { + running: status.running, + gaveUp: status.gaveUp, + portInUse: this.portInUse, + ...(this.startFailed ? { startFailed: this.startFailed } : {}), + ...(state.state === 'installed' && this.digestStatus ? { digest: this.digestStatus } : {}), + developerTools, + runtimeInstalled: await this.isRuntimeOnDisk(), + ...(this.liveProgress?.bytesPerSecond !== undefined ? { bytesPerSecond: this.liveProgress.bytesPerSecond } : {}), + ...(nextModel ? { nextModel } : {}), + pin: this.deps.pin, + folder: this.paths().root, + }; + } + + async view(): Promise { + try { + await this.recovered(); + let state = await this.currentState(); + if (state.state === 'installing' && this.liveProgress) { + state = { + ...state, + progress: { + completedBytes: this.liveProgress.completedBytes, + totalBytes: this.liveProgress.totalBytes, + }, + }; + } + if ( + state.state === 'installed' && + this.deps.supervisor.status().running && + this.deps.now() - this.digestCheckedAt >= DIGEST_RECHECK_MS + ) { + // In the background: the window polls every second and must not wait for the runtime. + this.digestCheckedAt = this.deps.now(); + void this.refreshDigest(state).catch(() => {}); + } + return buildInstallView(state, await this.liveFacts(state)); + } catch (err) { + console.warn(`[local-model] Could not build the install view: ${getErrorMessage(err)}`); + return buildInstallView(IDLE_LOCAL_MODEL_STATE, { + running: false, + gaveUp: false, + portInUse: false, + developerTools: 'unknown', + runtimeInstalled: false, + pin: this.deps.pin, + folder: '', + }); + } + } + + async recommendOptions(): Promise { + let measuredTotalBytes: number | undefined; + try { + await this.recovered(); + measuredTotalBytes = (await this.deps.readState()).runtime?.measuredTotalBytes; + } catch { + measuredTotalBytes = undefined; + } + return { + ...(measuredTotalBytes !== undefined ? { measuredTotalBytes } : {}), + runtimeDownloadBytes: (await this.isRuntimeOnDisk()) ? 0 : this.deps.pin.bytes, + }; + } + + async launchRefusal(endpointId: string): Promise { + if (!isLocalModelEndpointId(endpointId)) return undefined; + try { + await this.recovered(); + const state = await this.currentState(); + // A record with the reserved id that this feature did not write is not ours to judge. + if (state.connection?.endpointId !== endpointId) return undefined; + const status = this.deps.supervisor.status(); + if (state.state !== 'installed' || !state.runtime || !state.model || !status.running) { + if (this.portInUse) return buildInstallError('port-in-use', { vars: this.errorVars('port-in-use', state) }); + if (status.gaveUp) return buildInstallError('runtime-stopped'); + if (this.startFailed) return buildInstallError('runtime-start-failed', { next: 'start' }); + return buildInstallError('runtime-not-running'); + } + const digest = await this.refreshDigest(state); + if (digest === 'same') return undefined; + if (digest === 'unknown') { + // No digest to compare: either the runtime stopped answering, or the model is gone. + let version: string | undefined; + try { + version = await this.deps.getRuntimeVersion( + baseUrlOf(state.runtime.port), + AbortSignal.timeout(RUNTIME_READ_TIMEOUT_MS) + ); + } catch { + version = undefined; + } + if (version === undefined) return buildInstallError('runtime-not-running'); + this.digestStatus = 'changed'; + } + return buildInstallError('model-changed'); + } catch (err) { + console.warn(`[local-model] The launch check failed: ${getErrorMessage(err)}`); + return buildInstallError('runtime-not-running'); + } + } + + async launchRefusalMessage(endpointId: string, modelId: string): Promise { + const refusal = await this.launchRefusal(endpointId); + if (refusal) return formatInstallMessage(refusal.message); + if (!isLocalModelEndpointId(endpointId)) return undefined; + try { + const state = await this.deps.readState(); + if (state.connection?.endpointId === endpointId && state.model && state.model.id !== modelId) { + return formatInstallMessage({ + text: LOCAL_MODEL_INSTALL_LABELS.refusedOtherModel, + vars: { model: state.model.label }, + }); + } + } catch { + /* an unreadable state is "nothing of ours": nothing to refuse */ + } + return undefined; + } + + // ---------- Verbs ---------- + + private async answer(): Promise { + return { ok: true, view: await this.view() }; + } + + install(request: LocalModelInstallRequest, actor: LocalModelActor): Promise { + return this.exclusive(async () => { + await this.settle(); + const state = await this.currentState(); + if (state.state === 'installing') return conflict('job-running'); + if (state.state === 'installed') return conflict('already-installed'); + + // Everything below is checked against what the server itself offers right now, before + // anything is downloaded. + const hw = await this.deps.readHardware(); + const catalog = this.deps.getCatalog(); + const facts = await this.deps.buildHarnessFacts(catalog); + const options = await this.recommendOptions(); + const rec = this.deps.recommend(hw, catalog, facts, options); + const rows = [rec.pick, ...rec.alternatives].filter((row): row is Candidate => row !== undefined); + const candidate = rows.find((row) => row.modelId === request.modelId); + if (!candidate || candidate.context !== request.context) { + return { ok: false, status: 400, message: { text: LOCAL_MODEL_INSTALL_LABELS.refusedModelNotOffered } }; + } + // A row without the list (an older shape) still names its one coding agent in the flat fields. + const harnesses: CandidateHarness[] = candidate.harnesses ?? [ + { cli: candidate.cli, cliLabel: candidate.cliLabel, installed: candidate.cliInstalled }, + ]; + const harness = harnesses.find((row) => row.cli === request.cli); + if (!harness || !catalog.harnesses.some((row) => row.cli === request.cli)) { + return { ok: false, status: 400, message: { text: LOCAL_MODEL_INSTALL_LABELS.refusedAgentNotOffered } }; + } + if (!harness.installed && !this.deps.stockInstallCommand(harness.cli)) { + return { + ok: false, + status: 400, + message: { text: LOCAL_MODEL_INSTALL_LABELS.refusedAgentNotInstallable, vars: { cli: harness.cliLabel } }, + }; + } + const expectedBytes = candidate.installBytes ?? candidate.downloadBytes + (options.runtimeDownloadBytes ?? 0); + if (request.acknowledgedBytes !== expectedBytes) { + const expectedGb = formatGb(expectedBytes); + return conflict('size-changed', { expectedBytes, expectedGb }, { size: expectedGb }); + } + // Said now, not after the download: a saved endpoint that already has our id and is not ours. + if (!state.connection && (await this.deps.isEndpointIdTaken())) { + return { + ok: false, + status: 400, + message: buildInstallError('endpoint-id-taken', { vars: { id: LOCAL_MODEL_ENDPOINT_ID } }).message, + }; + } + + const refused = await this.beginJob(actor, { + type: 'install', + model: { id: candidate.modelId, label: candidate.modelLabel, context: candidate.context }, + cli: { id: harness.cli, label: harness.cliLabel }, + steps: jobSteps({ agentInstalled: harness.installed }), + }); + if (refused) return refused; + void this.deps + .audit({ + admin: actor.username, + action: 'local_model_install', + target: candidate.modelId, + ...(actor.ip ? { ip: actor.ip } : {}), + detail: { cli: harness.cli, context: candidate.context, bytes: expectedBytes }, + }) + .catch(() => {}); + return this.answer(); + }); + } + + /** + * Writes the event that starts a job (`install` or `resume`) and starts the job. `undefined`: + * started. A state file that cannot be written for lack of space is not a 500: the job did + * not start, and the view says `disk-full` with its "Try again" (a first install, which has + * no job to show it on, is answered with that sentence). + */ + private async beginJob( + actor: LocalModelActor, + event: Extract + ): Promise { + this.jobStarting = true; + try { + const started = await this.verbDispatch(event); + if (!started) return conflict('wrong-state'); + this.startJob(actor); + return undefined; + } catch (err) { + if (!isDiskFullError(err)) throw err; + // A first install has no job to show the failure on: the sentence goes back as it is. + if (event.type === 'install' && (await this.currentState().catch(() => undefined))?.state === 'idle') { + return { ok: false, status: 507, message: { text: LOCAL_MODEL_INSTALL_LABELS.errDiskFull } }; + } + const kept = this.unrecorded?.progress; + this.unrecorded = { + failure: { code: 'disk-full', detail: getErrorMessage(err) }, + ...(kept ? { progress: kept } : {}), + }; + return this.answer(); + } finally { + this.jobStarting = false; + } + } + + cancel(): Promise { + return this.exclusive(async () => { + // Taken before anything is awaited: a Cancel that arrives while a job runs is answered + // with the view after it, also when the job finishes in the same moment. + const job = this.job; + if (!job) { + // No job in this process: a job a restart interrupted is paused, never running. + await this.settle(); + return conflict('wrong-state'); + } + job.controller.abort(); + await job.promise.catch(() => {}); + // The step may have finished the job while it was being interrupted (the connect step + // completes its two writes). Then there is nothing left to pause, and the view says so. + // Otherwise the job is paused with its download as it stood, in ONE write. + const progress = this.progressNow(); + try { + await this.deps.mutateState((current) => { + if (current.state !== 'installing') return undefined; + const kept = progress ? reduceInstallJob(current, { type: 'progress', ...progress }) : current; + return reduceInstallJob(kept, { type: 'cancel' }); + }); + } catch (err) { + // The job IS stopped. The file could not be told: the view shows the pause from memory. + console.warn(`[local-model] Could not write the paused install: ${getErrorMessage(err)}`); + this.unrecorded = { ...(progress ? { progress } : {}) }; + } + this.clearProgress(); + return this.answer(); + }); + } + + resume( + request: LocalModelResumeRequest, + actor: LocalModelActor, + liveSessionNames: readonly string[] = [] + ): Promise { + return this.exclusive(async () => { + await this.settle(); + const state = await this.currentState(); + if (state.state === 'installing') return conflict('job-running'); + if (state.state !== 'paused' && state.state !== 'failed') return conflict('wrong-state'); + if (request.newPort === true && state.runtime) { + const open = this.sessionsOpen(state, liveSessionNames); + if (open) return open; + await this.changePort(state); + } + return (await this.beginJob(actor, { type: 'resume' })) ?? this.answer(); + }); + } + + /** 409 `sessions-open` while live sessions use a connection of ours, else `undefined`. */ + private sessionsOpen( + state: LocalModelStateFile, + liveSessionNames: readonly string[] + ): LocalModelVerbResult | undefined { + if (!state.connection || liveSessionNames.length === 0) return undefined; + const sessions = [...liveSessionNames]; + return conflict('sessions-open', { sessions }, { sessions: sessions.join(', ') }); + } + + /** + * Chooses another port than the stored one and moves the saved endpoint with it. False when + * no other port is free. The state file first, then the saved endpoint: a state write that + * fails changes nothing, and a refused endpoint write puts the old port back, so the two + * never disagree. The stored pid stays: when it is still our own runtime on the OLD port, + * the next start ends it instead of leaving a second runtime behind. + */ + private async changePort(state: LocalModelStateFile): Promise { + if (!state.runtime) return false; + const previous = state.runtime.port; + const port = await this.deps.chooseRuntimePort({ avoid: [previous] }); + if (port === undefined) return false; + const setPort = (value: number): Promise => + this.record((current) => + current.runtime ? { ...current, runtime: { ...current.runtime, port: value } } : undefined + ); + await setPort(port); + if (state.connection) { + try { + await this.deps.moveLocalEndpoint(port); + } catch (err) { + await setPort(previous).catch(() => {}); + throw err; + } + } + this.portInUse = false; + return true; + } + + runtime(request: LocalModelRuntimeRequest, liveSessionNames: readonly string[] = []): Promise { + return this.exclusive(async () => { + await this.settle(); + let state = await this.currentState(); + if (state.state === 'installing') return conflict('job-running'); + if (state.state !== 'installed' || !state.runtime) return conflict('wrong-state'); + + if (request.action === 'stop') { + // Their coding agents keep calling the address they were started with. + const open = this.sessionsOpen(state, liveSessionNames); + if (open) return open; + await this.deps.supervisor.stop({ + ...(state.runtime.pid !== undefined ? { previousPid: state.runtime.pid } : {}), + }); + this.portInUse = false; + this.startFailed = undefined; + await this.record((current) => + current.runtime ? { ...current, runtime: { ...current.runtime, enabled: false } } : undefined + ); + return this.answer(); + } + + if (request.newPort === true) { + const open = this.sessionsOpen(state, liveSessionNames); + if (open) return open; + // The old child, when one is still alive, holds the old port: it must go first. + await this.deps.supervisor.stop({ + ...(state.runtime.pid !== undefined ? { previousPid: state.runtime.pid } : {}), + }); + await this.changePort(state); + } + await this.record((current) => + current.runtime ? { ...current, runtime: { ...current.runtime, enabled: true } } : undefined + ); + state = await this.deps.readState(); + const started = await this.startRuntime(state); + if (started.ok) await this.refreshDigest(await this.deps.readState()); + return this.answer(); + }); + } + + remove( + request: LocalModelRemoveRequest, + liveSessionNames: readonly string[], + actor?: LocalModelActor + ): Promise { + return this.exclusive(async () => { + await this.settle(); + const state = await this.currentState(); + if (state.state === 'installing') return conflict('job-running'); + const open = this.sessionsOpen(state, liveSessionNames); + if (open) return open; + + // Our child, and a runtime an earlier server left behind (known by the stored pid): no + // file is deleted under a process that still runs from it. + await this.deps.supervisor.stop({ + ...(state.runtime?.pid !== undefined ? { previousPid: state.runtime.pid } : {}), + }); + // The two writes outside our folder are undone first: when one is refused, nothing was + // deleted yet and the person can try again. + if (state.connection) { + await this.deps.disconnectLocalModel({ flagWasEnabled: state.connection.flagWasEnabled }); + } + const removeModel = request.removeModel === true; + await this.deps.removeLocalModelFiles({ models: removeModel, paths: this.paths() }); + await this.verbDispatch({ type: 'remove' }); + this.clearProgress(); + this.unrecorded = undefined; + this.portInUse = false; + this.startFailed = undefined; + this.digestStatus = undefined; + if (actor) { + void this.deps + .audit({ + admin: actor.username, + action: 'local_model_remove', + target: state.model?.id ?? '', + ...(actor.ip ? { ip: actor.ip } : {}), + detail: { removeModel }, + }) + .catch(() => {}); + } + return this.answer(); + }); + } + + async boot(): Promise { + try { + await this.recovered(); + const state = await this.currentState(); + const installed = state.state === 'installed' && state.runtime !== undefined && state.model !== undefined; + // The record of the connection can be gone while the model is still installed (a + // config-backup restore of an older endpoint store): it is put back, never replaced. + if (installed && state.connection && state.runtime && state.model) { + const restored = await this.deps.ensureLocalEndpoint({ + port: state.runtime.port, + modelId: state.model.id, + context: state.model.context, + }); + if (restored) + console.warn('[local-model] The saved endpoint of the model on this Mac was missing and was put back.'); + } + if (!installed || !state.runtime?.enabled) { + // Nothing starts the runtime here (a job waits for a click, the person stopped the + // model). One an earlier server left behind is ended: it holds the model's memory, + // and a later Remove must not delete its folder under it. + if (state.runtime?.pid !== undefined) { + await this.deps.supervisor.stop({ previousPid: state.runtime.pid }); + } + return; + } + const started = await this.startRuntime(state); + if (!started.ok) { + console.warn( + `[local-model] The runtime did not start: ${started.code}${started.detail ? ` (${started.detail})` : ''}` + ); + return; + } + const digest = await this.refreshDigest(await this.deps.readState()); + if (digest === 'changed') { + console.warn('[local-model] The model changed since it was installed. Sessions on it are refused.'); + } + } catch (err) { + console.warn(`[local-model] Start-up check failed: ${getErrorMessage(err)}`); + } + } + + async shutdown(): Promise { + this.stopping = true; + const job = this.job; + // The state file keeps saying `installing`: the next start reads that as a paused job. + job?.controller.abort(); + // The runtime is signalled at once, not after the job let go: the server may exit soon. + const stopped = Promise.resolve() + .then(() => this.deps.supervisor.stop()) + .catch((err: unknown) => { + console.warn(`[local-model] Stopping the runtime failed: ${getErrorMessage(err)}`); + }); + if (job) { + await Promise.race([job.promise.catch(() => {}), new Promise((resolve) => setTimeout(resolve, 3_000).unref())]); + } + await stopped; + } + + async idle(): Promise { + while (this.job) await this.job.promise.catch(() => {}); + } + + // ---------- The job ---------- + + private startJob(actor: LocalModelActor): void { + const controller = new AbortController(); + this.clearProgress(); + this.unrecorded = undefined; + const job = { + controller, + promise: this.runJob(actor, controller.signal).finally(() => { + if (this.job === job) this.job = undefined; + }), + }; + this.job = job; + } + + /** Runs the steps from the stored one on, until the job is installed, failed or interrupted. Never rejects. */ + private async runJob(actor: LocalModelActor, signal: AbortSignal): Promise { + try { + for (;;) { + const state = await this.deps.readState(); + if (state.state !== 'installing' || !state.step || signal.aborted) return; + let outcome: StepOutcome; + try { + outcome = await this.runStep(state.step, state, actor, signal); + } catch (err) { + // A write of the step itself that found no space is a full disk, not a mystery. + outcome = { + kind: 'fail', + code: isDiskFullError(err) ? 'disk-full' : 'internal', + detail: getErrorMessage(err), + }; + } + if (outcome.kind === 'done') { + // A step that finished stays finished, also when Cancel arrived while it ran. Its + // download figures go first, so the next step is never shown with them. + this.clearProgress(); + await this.dispatch({ type: 'step-done' }); + continue; + } + // An interrupted step is not a failed one, whatever its last call answered. Cancel + // writes `paused` itself; a server that is stopping writes nothing. + if (outcome.kind === 'cancelled' || signal.aborted || this.stopping) return; + const failure: JobFailure = { + code: outcome.code, + ...(outcome.vars ? { vars: outcome.vars } : {}), + ...(outcome.detail ? { detail: outcome.detail } : {}), + ...(outcome.nextModelId ? { nextModelId: outcome.nextModelId } : {}), + }; + const progress = this.progressNow(); + try { + // The failure and the download as it stood, in ONE write. + await this.deps.mutateState((current) => { + const kept = progress ? reduceInstallJob(current, { type: 'progress', ...progress }) : current; + const next = reduceInstallJob(kept, { type: 'fail', ...failure }); + return next === current ? undefined : next; + }); + } catch (err) { + // The disk that failed the step also refuses the state file. The outcome is not + // lost: the view shows it from memory and the next verb writes it. + console.warn(`[local-model] Could not write why the install stopped: ${getErrorMessage(err)}`); + this.unrecorded = { failure, ...(progress ? { progress } : {}) }; + } + this.clearProgress(); + return; + } + } catch (err) { + // A write of the job itself was refused (the step that finished could not be recorded). + console.warn(`[local-model] The install job stopped: ${getErrorMessage(err)}`); + const progress = this.progressNow(); + this.unrecorded = { + failure: { code: isDiskFullError(err) ? 'disk-full' : 'internal', detail: getErrorMessage(err) }, + ...(progress ? { progress } : {}), + }; + this.clearProgress(); + } + } + + private runStep( + step: LocalModelInstallStep, + state: LocalModelStateFile, + actor: LocalModelActor, + signal: AbortSignal + ): Promise { + switch (step) { + case 'runtime': + return this.stepRuntime(state, signal); + case 'model': + return this.stepModel(state, signal); + case 'verify': + return this.stepVerify(state, signal); + case 'agent': + return this.stepAgent(state, actor, signal); + case 'connect': + return this.stepConnect(state, signal); + } + } + + /** Makes sure the runtime runs before a step that needs it. `undefined`: it runs. */ + private async needRuntime(signal: AbortSignal): Promise { + const state = await this.deps.readState(); + const started = await this.startRuntime(state); + if (signal.aborted) return CANCELLED; + return started.ok ? undefined : this.startFailure(started, state); + } + + private async stepRuntime(state: LocalModelStateFile, signal: AbortSignal): Promise { + if (!state.model) return { kind: 'fail', code: 'internal', detail: 'The job has no model.' }; + const installed = await this.deps.ensureRuntimeInstalled({ + pin: this.deps.pin, + paths: this.paths(), + signal, + onProgress: (progress) => this.onProgress(progress), + }); + if (!installed.ok) { + if (installed.code === 'cancelled') return CANCELLED; + return { kind: 'fail', code: installed.code, ...(installed.detail ? { detail: installed.detail } : {}) }; + } + if (signal.aborted) return CANCELLED; + + // The port is chosen ONCE. A job that already has one keeps it. + if (!state.runtime) { + const port = await this.deps.chooseRuntimePort(); + if (port === undefined) return { kind: 'fail', code: 'port-in-use', vars: this.errorVars('port-in-use', state) }; + await this.record((current) => + current.runtime ? undefined : { ...current, runtime: { version: this.deps.pin.version, port, enabled: true } } + ); + } + const withPort = await this.deps.readState(); + const started = await this.startRuntime(withPort); + if (signal.aborted) return CANCELLED; + if (!started.ok) return this.startFailure(started, withPort); + + // The budget is MEASURED now: the runtime printed the device total when it started. The + // model is judged against it before a byte of it is downloaded. + const measuredTotalBytes = started.measuredTotalBytes ?? withPort.runtime?.measuredTotalBytes; + if (measuredTotalBytes !== undefined) { + const hw = await this.deps.readHardware(); + const catalog = this.deps.getCatalog(); + const options: RecommendOptions = { measuredTotalBytes, runtimeDownloadBytes: 0 }; + if (!this.deps.modelFitsBudget(hw, catalog, state.model.id, state.model.context, options)) { + const nextModelId = await this.nextModelId(state.model, options, { needsLessMemory: true }); + return { + kind: 'fail', + code: 'does-not-fit', + vars: this.errorVars('does-not-fit', state), + ...(nextModelId ? { nextModelId } : {}), + }; + } + } + return DONE; + } + + private async stepModel(state: LocalModelStateFile, signal: AbortSignal): Promise { + if (!state.model) return { kind: 'fail', code: 'internal', detail: 'The job has no model.' }; + const blocked = await this.needRuntime(signal); + if (blocked) return blocked; + const running = await this.deps.readState(); + if (!running.runtime) return { kind: 'fail', code: 'internal', detail: 'The runtime has no port.' }; + const target = baseUrlOf(running.runtime.port); + + // What a replaced job left of another model goes first, so two models never share the store. + if (running.replacedModelId) { + try { + await this.deps.deleteModel(target, running.replacedModelId, signal); + } catch { + /* a leftover is not a reason to stop: Remove with the models deletes it */ + } + if (signal.aborted) return CANCELLED; + // A download that was interrupted left partial blobs and no model: the runtime answers + // "not found" for it and deletes nothing. No download is in flight here, so every + // partial file in the store is a dead one. + try { + const freed = await this.deps.removePartialModelBlobs(this.paths()); + if (freed > 0) console.log(`[local-model] Deleted ${formatGb(freed)} GB a replaced download had left.`); + } catch { + /* as above */ + } + await this.record((current) => { + const { replacedModelId: _replaced, ...rest } = current; + return rest; + }); + } + + const pulled = await this.deps.pullModel(target, state.model.id, { + signal, + onProgress: (progress) => this.onProgress(progress), + }); + if (!pulled.ok) { + if (pulled.code === 'cancelled') return CANCELLED; + // A runtime that is gone for good looks like a broken network to the pull. It is not. + const status = this.deps.supervisor.status(); + if (pulled.code === 'pull-failed' && (status.gaveUp || !status.running)) { + return { + kind: 'fail', + code: 'runtime-stopped', + ...(pulled.detail ? { detail: pulled.detail } : {}), + }; + } + const nextModelId = pulled.code === 'model-not-found' ? await this.nextModelId(state.model) : undefined; + return { + kind: 'fail', + code: pulled.code, + vars: this.errorVars(pulled.code, state), + ...(pulled.detail ? { detail: pulled.detail } : {}), + ...(nextModelId ? { nextModelId } : {}), + }; + } + // From here on every size shown for the model is the pull's own total. + const pullBytes = pulled.totalBytes; + await this.record((current) => + current.model ? { ...current, model: { ...current.model, pullBytes } } : undefined + ); + return DONE; + } + + private async stepVerify(state: LocalModelStateFile, signal: AbortSignal): Promise { + if (!state.model) return { kind: 'fail', code: 'internal', detail: 'The job has no model.' }; + const blocked = await this.needRuntime(signal); + if (blocked) return blocked; + const running = await this.deps.readState(); + if (!running.runtime) return { kind: 'fail', code: 'internal', detail: 'The runtime has no port.' }; + const target = baseUrlOf(running.runtime.port); + const { id, context } = state.model; + + const verified = await this.deps.verifyModel(target, id, context, signal); + if (!verified.ok) { + if (verified.code === 'cancelled') return CANCELLED; + let vars = this.errorVars(verified.code, state); + if (verified.code === 'context-not-applied') { + let actual = '?'; + try { + const loaded = (await this.deps.listRunning(target, AbortSignal.timeout(RUNTIME_READ_TIMEOUT_MS))).find( + (model) => model.name === id + ); + if (loaded) actual = String(loaded.contextLength); + } catch { + /* the sentence then says "?" */ + } + vars = { model: state.model.label, actual, wanted: String(context) }; + } + const nextModelId = + verified.code === 'does-not-fit' + ? await this.nextModelId(state.model, undefined, { needsLessMemory: true }) + : verified.code === 'no-tool-call' + ? await this.nextModelId(state.model) + : undefined; + return { + kind: 'fail', + code: verified.code, + ...(vars ? { vars } : {}), + ...(verified.detail ? { detail: verified.detail } : {}), + ...(nextModelId ? { nextModelId } : {}), + }; + } + + // The digest of what was just checked: what a session launch compares with from now on. + const digest = await this.deps.readModelDigest(target, id, signal); + if (signal.aborted) return CANCELLED; + if (!digest) { + return { + kind: 'fail', + code: 'load-failed', + vars: this.errorVars('load-failed', state), + detail: 'The runtime reports no digest for the model.', + }; + } + const loaded = verified.loaded; + await this.record((current) => + current.model ? { ...current, model: { ...current.model, digest }, loaded: { ...loaded } } : undefined + ); + this.digestStatus = 'same'; + this.digestCheckedAt = this.deps.now(); + return DONE; + } + + private async stepAgent( + state: LocalModelStateFile, + actor: LocalModelActor, + signal: AbortSignal + ): Promise { + if (!state.cli) return { kind: 'fail', code: 'internal', detail: 'The job has no coding agent.' }; + const cli = state.cli.id; + const vars = this.errorVars('harness-failed', state); + // A resumed job, or an installer that finished before a restart: the binary may be there. + if (await this.deps.recheckCliInstalled(cli)) return DONE; + if (signal.aborted) return CANCELLED; + + // Only a coding agent of the catalogue, and only its STOCK install command. + if (!this.deps.getCatalog().harnesses.some((row) => row.cli === cli)) { + return { kind: 'fail', code: 'harness-failed', vars, detail: 'Not a coding agent of the local model catalogue.' }; + } + const command = this.deps.stockInstallCommand(cli); + if (!command) { + return { kind: 'fail', code: 'harness-failed', vars, detail: 'No install command for this platform.' }; + } + const release = this.deps.tryBeginCliInstall(cli); + if (!release) { + return { kind: 'fail', code: 'harness-failed', vars, detail: 'Another install of this coding agent is running.' }; + } + let result: CliInstallResult; + try { + result = await this.deps.runInstallCommand(command, { signal }); + } finally { + release(); + } + void this.deps + .audit({ + admin: actor.username, + action: 'cli_install', + target: cli, + ...(actor.ip ? { ip: actor.ip } : {}), + detail: { + command, + exitCode: result.code, + timedOut: result.timedOut, + aborted: result.aborted, + via: 'local-model', + }, + }) + .catch(() => {}); + // Even a failed or interrupted install may have left a binary behind: the cached lookups + // are forgotten either way, and what counts is whether the binary is found now. + const installed = await this.deps.recheckCliInstalled(cli); + if (result.aborted || signal.aborted) return CANCELLED; + if (installed) return DONE; + const detail = result.timedOut + ? `The installer was stopped after ${Math.round(CLI_INSTALL_TIMEOUT_MS / 1000)} seconds.` + : result.output.slice(-INSTALLER_DETAIL_CHARS).trim() || + (result.code === 0 ? 'The installer finished, but the program was not found afterwards.' : 'No output.'); + return { kind: 'fail', code: 'harness-failed', vars, detail }; + } + + private async stepConnect(state: LocalModelStateFile, signal: AbortSignal): Promise { + if (!state.model) return { kind: 'fail', code: 'internal', detail: 'The job has no model.' }; + const blocked = await this.needRuntime(signal); + if (blocked) return blocked; + const running = await this.deps.readState(); + if (!running.runtime) return { kind: 'fail', code: 'internal', detail: 'The runtime has no port.' }; + const { port } = running.runtime; + const { id, context } = state.model; + + // The record names the model exactly as the runtime lists it for a coding agent. + const listed = await this.deps.listOpenAiModels(baseUrlOf(port), signal); + if (signal.aborted) return CANCELLED; + if (!listed.includes(id)) { + return { kind: 'fail', code: 'connect-failed', detail: `The runtime does not list ${id}.` }; + } + + // The connection is CLAIMED in the state file before either write, so no record of ours + // can exist that the state file does not know: a state write that fails, or a server that + // is killed between the two, leaves at worst a claim with nothing behind it. + let claim = running.connection; + if (!claim) { + let found: LocalEndpointInspection; + try { + found = await this.deps.inspectLocalEndpoint({ port, modelId: id, context }); + } catch (err) { + return { kind: 'fail', code: 'connect-failed', detail: getErrorMessage(err) }; + } + if (signal.aborted) return CANCELLED; + // A record with our id that says something else is somebody else's: never overwritten. + if (found.record === 'other') { + return { kind: 'fail', code: 'endpoint-id-taken', vars: this.errorVars('endpoint-id-taken', state) }; + } + // A record that already says what ours would say can only be ours (the routes refuse + // the id): the flag was then switched on by us, and its earlier value is not known, so + // it is never switched off again. + claim = { + endpointId: LOCAL_MODEL_ENDPOINT_ID, + flagWasEnabled: found.record === 'same' ? true : found.flagEnabled, + }; + const claimed = claim; + await this.record((current) => ({ ...current, connection: claimed })); + } + if (signal.aborted) return CANCELLED; + + // From here on the two writes run to their end: Cancel no longer interrupts. + const connected = await this.deps.connectLocalModel({ + port, + modelId: id, + context, + ours: true, + flagWasEnabled: claim.flagWasEnabled, + }); + if (!connected.ok) { + return { + kind: 'fail', + code: connected.code, + vars: this.errorVars(connected.code, state), + ...(connected.detail ? { detail: connected.detail } : {}), + }; + } + return DONE; + } +} + +/** A service over `deps`. A test builds its own with fakes. */ +export function createLocalModelService(deps: Partial = {}): LocalModelService { + return new LocalModelServiceImpl(deps); +} + +let processService: LocalModelService | undefined; + +/** The one service of this process, built on first use with the real modules. What the routes and server.ts call. */ +export function getLocalModelService(): LocalModelService { + processService ??= createLocalModelService(); + return processService; +} diff --git a/src/local-model/ollama-client.ts b/src/local-model/ollama-client.ts new file mode 100644 index 00000000..90fe3bb7 --- /dev/null +++ b/src/local-model/ollama-client.ts @@ -0,0 +1,810 @@ +/** + * @fileoverview The HTTP client of the model runtime: what Codeman asks its own runtime on + * the loopback port, and the pure functions that read the answers. + * + * What it asks: + * - `GET /api/version`, `GET /api/tags`, `GET /api/ps`, `GET /v1/models`, `DELETE /api/delete`. + * - `POST /api/pull` with `stream: true`. The answer is one JSON object per line, and a + * 23 GB pull sends about 1.4 million of them, so the stream is parsed INCREMENTALLY + * (`createPullParser()`, pure): only the unfinished last line and the running totals are + * kept, never the body. Aborting the request stops the download; pulling again continues + * it. An error line or a stream that ends without `success` is retried, at most + * `PULL_RETRIES` times. + * - The check after the pull (`verifyModel()`), in this order: one tiny chat with + * `reasoning_effort: "none"` (it loads the model), then `GET /api/ps` must show the whole + * model on the GPU (`size_vram` equal to `size`) at the chosen context + * (`judgeLoaded()`, pure), then one tool probe on `/v1/chat/completions` with a `tools` + * array, `reasoning_effort: "none"` and `max_tokens` of `TOOL_PROBE_MAX_TOKENS`, which + * passes on a tool call whose arguments parse as JSON (`judgeToolProbe()`, pure). + * + * Rules: + * - Nothing here throws for something a runtime can answer: every outcome is a result with + * the job's error code. An abort signal is the result `cancelled`. + * - Every request is bounded: a listing by seconds, the two probes by minutes (the first + * chat loads the model). Only the pull has no deadline of its own, because its length is + * the download; it ends with the stream, and the caller's signal cancels it. + * - The runtime does not enforce the context length (a prompt of 167,012 tokens was accepted + * at 131072), so nothing here relies on it to. + * - Every function takes its `fetch` from the target, so a test passes the fake runtime's + * address and spawns nothing. The target is always a loopback address the caller built from + * the stored port: no function here takes a URL from a file or a request. + */ + +import type { LocalModelErrorCode } from '../types.js'; +import { formatGb } from './format.js'; + +/** How often a pull that ended on an error line or a cut stream is started again. */ +export const PULL_RETRIES = 5; +/** `max_tokens` of the tool probe. */ +export const TOOL_PROBE_MAX_TOKENS = 1024; + +/** How long `/api/version` may take: it is asked while a runtime starts, many times. */ +const VERSION_TIMEOUT_MS = 2_000; +/** How long a listing or a delete may take. */ +const LIST_TIMEOUT_MS = 10_000; +/** How long the chat that loads the model may take. A 23 GB model loaded in 5.9 s on the test Mac; a cold disk is slower. */ +const CHAT_PROBE_TIMEOUT_MS = 180_000; +/** How long the tool probe may take. The model is loaded by then. */ +const TOOL_PROBE_TIMEOUT_MS = 120_000; +/** `max_tokens` of the chat that loads the model: its answer is not read. */ +const CHAT_PROBE_MAX_TOKENS = 16; +/** The least time between two progress reports of a pull. */ +const PULL_PROGRESS_INTERVAL_MS = 250; +/** The window the download speed is averaged over. */ +const PULL_SPEED_WINDOW_MS = 5_000; +/** The first wait before a pull is started again; it doubles with every retry. */ +const PULL_RETRY_DELAY_MS = 1_000; +/** A pull line longer than this is not a pull line. It is dropped, so a stream without line breaks cannot grow the buffer. */ +const MAX_PULL_LINE_CHARS = 1024 * 1024; +/** More layers than any model has. Lines for further ones are counted and not added up, so the layer table cannot grow without bound. */ +const MAX_PULL_LAYERS = 1024; +/** How much of a runtime's error text is kept as `detail`. */ +const MAX_DETAIL_CHARS = 400; + +/** Which runtime to ask. */ +export interface OllamaTarget { + /** `http://127.0.0.1:`, without a trailing slash. */ + baseUrl: string; + /** Default: the global `fetch`. */ + fetch?: typeof fetch; +} + +// ---------- Plumbing ---------- + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function asString(value: unknown): string | undefined { + return typeof value === 'string' && value !== '' ? value : undefined; +} + +/** A byte count or a token count as the runtime reports it: a finite number that is not negative. */ +function asCount(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : undefined; +} + +function clip(text: string): string { + const trimmed = text.trim(); + return trimmed.length > MAX_DETAIL_CHARS ? `${trimmed.slice(0, MAX_DETAIL_CHARS)}...` : trimmed; +} + +function errorText(err: unknown): string { + if (err instanceof Error) { + // undici wraps the reason ("fetch failed") and keeps the useful part in `cause`. + const cause = (err as { cause?: unknown }).cause; + return clip(cause instanceof Error && cause.message ? `${err.message}: ${cause.message}` : err.message); + } + return clip(String(err)); +} + +/** The error text of a runtime answer: `{"error":"..."}` (its own routes) or `{"error":{"message":"..."}}` (the OpenAI routes). */ +function errorOfBody(body: unknown): string | undefined { + if (!isRecord(body)) return undefined; + const { error } = body; + if (typeof error === 'string') return asString(error); + if (isRecord(error)) return asString(error.message); + return undefined; +} + +/** The caller's signal joined with a deadline. */ +function bounded(signal: AbortSignal | undefined, timeoutMs: number): AbortSignal { + const deadline = AbortSignal.timeout(timeoutMs); + return signal ? AbortSignal.any([signal, deadline]) : deadline; +} + +/** An answer of the runtime: its status and its body as JSON (`undefined` when it is not JSON). */ +interface Answer { + status: number; + body: unknown; + text: string; +} + +/** + * One bounded request. `undefined` when nothing answered (refused, timed out, aborted): the + * caller looks at its own signal to tell an abort from the rest. + */ +async function ask( + target: OllamaTarget, + path: string, + init: { method?: string; json?: unknown }, + timeoutMs: number, + signal?: AbortSignal +): Promise { + const fetchFn = target.fetch ?? fetch; + try { + const res = await fetchFn(`${target.baseUrl}${path}`, { + method: init.method ?? 'GET', + ...(init.json !== undefined + ? { headers: { 'content-type': 'application/json' }, body: JSON.stringify(init.json) } + : {}), + signal: bounded(signal, timeoutMs), + }); + const text = await res.text(); + let body: unknown; + try { + body = text === '' ? undefined : JSON.parse(text); + } catch { + body = undefined; + } + return { status: res.status, body, text }; + } catch { + return undefined; + } +} + +/** A digest as it is stored and compared: lowercase hex without the `sha256:` prefix. */ +function normalizeDigest(value: unknown): string { + return typeof value === 'string' + ? value + .trim() + .toLowerCase() + .replace(/^sha256:/, '') + : ''; +} + +// ---------- Listings ---------- + +/** One model of `GET /api/tags`: what is on disk. */ +export interface OllamaTagModel { + /** `name:tag`. */ + name: string; + /** Size on disk in bytes. */ + size: number; + /** The model's digest, lowercase hex without a prefix. What is stored and compared again before a session starts. */ + digest: string; + /** `details.format` ('safetensors' for an MLX build, 'gguf'). */ + format?: string; + /** `details.runner` ('mlx', 'ggml'). */ + runner?: string; + /** `capabilities` ('completion', 'tools', 'thinking', 'vision'). */ + capabilities?: string[]; +} + +/** One model of `GET /api/ps`: what is loaded. */ +export interface OllamaPsModel { + name: string; + /** Memory the loaded model takes, in bytes. */ + size: number; + /** The part of `size` that is on the GPU. Equal to `size` when all of it is. */ + sizeVram: number; + /** The context length the model was loaded with. */ + contextLength: number; + digest?: string; +} + +/** The `models` array of a listing, or an empty one. */ +function modelsOf(answer: Answer | undefined): Record[] { + if (!answer || answer.status !== 200 || !isRecord(answer.body)) return []; + const { models } = answer.body; + return Array.isArray(models) ? models.filter(isRecord) : []; +} + +/** The runtime's version ('0.40.15'), or `undefined` when nothing answers there or the answer is not a runtime's. */ +export async function getRuntimeVersion(target: OllamaTarget, signal?: AbortSignal): Promise { + const answer = await ask(target, '/api/version', {}, VERSION_TIMEOUT_MS, signal); + if (!answer || answer.status !== 200 || !isRecord(answer.body)) return undefined; + return asString(answer.body.version); +} + +/** The models on disk. An empty list when the runtime has none or does not answer. */ +export async function listTags(target: OllamaTarget, signal?: AbortSignal): Promise { + const listed: OllamaTagModel[] = []; + for (const row of modelsOf(await ask(target, '/api/tags', {}, LIST_TIMEOUT_MS, signal))) { + const name = asString(row.name) ?? asString(row.model); + if (!name) continue; + const details = isRecord(row.details) ? row.details : {}; + const format = asString(details.format); + const runner = asString(details.runner) ?? asString(row.runner); + const capabilities = Array.isArray(row.capabilities) + ? row.capabilities.filter((entry): entry is string => typeof entry === 'string') + : undefined; + listed.push({ + name, + size: asCount(row.size) ?? 0, + digest: normalizeDigest(row.digest), + ...(format ? { format } : {}), + ...(runner ? { runner } : {}), + ...(capabilities ? { capabilities } : {}), + }); + } + return listed; +} + +/** The models loaded right now. An empty list when none is or the runtime does not answer. */ +export async function listRunning(target: OllamaTarget, signal?: AbortSignal): Promise { + const listed: OllamaPsModel[] = []; + for (const row of modelsOf(await ask(target, '/api/ps', {}, LIST_TIMEOUT_MS, signal))) { + const name = asString(row.name) ?? asString(row.model); + if (!name) continue; + const digest = normalizeDigest(row.digest); + listed.push({ + name, + size: asCount(row.size) ?? 0, + sizeVram: asCount(row.size_vram) ?? 0, + contextLength: asCount(row.context_length) ?? 0, + ...(digest ? { digest } : {}), + }); + } + return listed; +} + +/** The model ids `GET /v1/models` lists, exactly as listed (the runtime answers `data: null` for none). */ +export async function listOpenAiModels(target: OllamaTarget, signal?: AbortSignal): Promise { + const answer = await ask(target, '/v1/models', {}, LIST_TIMEOUT_MS, signal); + if (!answer || answer.status !== 200 || !isRecord(answer.body)) return []; + const { data } = answer.body; + if (!Array.isArray(data)) return []; + return data.flatMap((row) => (isRecord(row) && typeof row.id === 'string' && row.id !== '' ? [row.id] : [])); +} + +/** The digest `GET /api/tags` reports for `modelId`, or `undefined` when the model is not on disk. */ +export async function readModelDigest( + target: OllamaTarget, + modelId: string, + signal?: AbortSignal +): Promise { + const found = (await listTags(target, signal)).find((model) => model.name === modelId); + return found && found.digest !== '' ? found.digest : undefined; +} + +/** Deletes a model from the runtime's store. True when it is gone afterwards (also when it never was there). */ +export async function deleteModel(target: OllamaTarget, modelId: string, signal?: AbortSignal): Promise { + const answer = await ask( + target, + '/api/delete', + { method: 'DELETE', json: { model: modelId } }, + LIST_TIMEOUT_MS, + signal + ); + // 404 is the runtime's answer for a model it does not have. + return answer !== undefined && (answer.status === 200 || answer.status === 404); +} + +// ---------- Pull ---------- + +/** What the pull's lines add up to so far. */ +export interface PullSnapshot { + /** Sum of the last `completed` of every layer seen. */ + completedBytes: number; + /** Sum of the `total` of every layer seen. Grows while new layers appear; 0 before the first. */ + totalBytes: number; + /** The `status` text of the last line that had one ('pulling manifest', 'writing manifest'). */ + status?: string; + /** A line `{"status":"success"}` was seen. */ + succeeded: boolean; + /** The text of the first `{"error":"..."}` line. */ + error?: string; + /** How many complete lines were parsed. */ + lines: number; +} + +/** + * Reads a pull stream piece by piece. It keeps the unfinished last line and the totals per + * layer, never the lines themselves, so its memory does not grow with the stream. + */ +export interface PullParser { + /** Feed the next piece of the body. A line may be split anywhere across pieces, a multi-byte character too. */ + push(chunk: string | Uint8Array): void; + /** The body ended: reads a last line that had no line break. */ + end(): void; + snapshot(): PullSnapshot; +} + +/** + * A parser for one pull stream. Pure: no IO, no clock. Lines it understands (both real): + * + * - an MLX build: `{"status":"pulling model","digest":"sha256:model","total":23105798956,"completed":N}` + * - a GGUF build, one layer after the other: + * `{"status":"pulling c5396e06af29","digest":"sha256:c539...","total":397807936,"completed":N}` + * (`completed` is absent until the layer's first bytes arrived) + * - `{"status":"pulling manifest"}`, `{"status":"verifying sha256 digest"}`, + * `{"status":"writing manifest"}`, `{"status":"success"}` + * - `{"error":"max retries exceeded: local error: tls: bad record MAC"}` + * + * A line that is not JSON is skipped and still counted. A blank line is not a line. + */ +export function createPullParser(): PullParser { + const decoder = new TextDecoder(); + // One entry per layer: what the layer's last line said. A model has a handful of layers. + const layers = new Map(); + let pending = ''; + // True while a line longer than the limit is being skipped up to its line break. + let skipping = false; + let status: string | undefined; + let succeeded = false; + let error: string | undefined; + let lines = 0; + + function line(text: string): void { + const trimmed = text.trim(); + if (trimmed === '') return; + lines += 1; + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch { + return; + } + if (!isRecord(parsed)) return; + const failure = errorOfBody(parsed); + if (failure !== undefined) { + error ??= clip(failure); + return; + } + const lineStatus = asString(parsed.status); + if (lineStatus !== undefined) { + status = lineStatus; + if (lineStatus === 'success') succeeded = true; + } + const total = asCount(parsed.total); + if (total === undefined) return; + // A layer is named by its digest; a line without one by its status text. + const key = asString(parsed.digest) ?? lineStatus; + if (key === undefined) return; + const known = layers.get(key); + if (!known && layers.size >= MAX_PULL_LAYERS) return; + // `completed` is absent until the layer's first bytes arrived: what was seen before stays. + const completed = asCount(parsed.completed) ?? known?.completed ?? 0; + layers.set(key, { total, completed: Math.min(completed, total) }); + } + + function feed(text: string): void { + let rest = text; + for (;;) { + const breakAt = rest.indexOf('\n'); + if (breakAt < 0) break; + const head = rest.slice(0, breakAt); + rest = rest.slice(breakAt + 1); + if (skipping) { + // The end of the line that was too long: it was counted when it was dropped. + skipping = false; + } else { + line(pending + head); + } + pending = ''; + } + if (skipping) return; + pending += rest; + if (pending.length > MAX_PULL_LINE_CHARS) { + pending = ''; + skipping = true; + lines += 1; + } + } + + return { + push(chunk) { + feed(typeof chunk === 'string' ? chunk : decoder.decode(chunk, { stream: true })); + }, + end() { + feed(decoder.decode()); + if (!skipping && pending !== '') line(pending); + pending = ''; + skipping = false; + }, + snapshot() { + let completedBytes = 0; + let totalBytes = 0; + for (const layer of layers.values()) { + completedBytes += layer.completed; + totalBytes += layer.total; + } + return { + completedBytes, + totalBytes, + ...(status !== undefined ? { status } : {}), + succeeded, + ...(error !== undefined ? { error } : {}), + lines, + }; + }, + }; +} + +/** How far a pull is, for the caller's progress display. */ +export interface PullProgress { + completedBytes: number; + totalBytes: number; + /** Over the last few seconds. Absent until it can be told. */ + bytesPerSecond?: number; +} + +export interface PullOptions { + /** Aborts the request, which stops the download. The result is `cancelled`. */ + signal?: AbortSignal; + /** Called at most a few times a second, and once more when the pull ends. */ + onProgress?: (progress: PullProgress) => void; + /** Default `PULL_RETRIES`. */ + retries?: number; + /** Waits between retries. A test injects its own. */ + sleep?: (ms: number) => Promise; + /** The clock of the speed figure, in milliseconds. A test injects its own. */ + now?: () => number; +} + +/** How a pull ended. */ +export type PullResult = + | { + ok: true; + /** The pull's own total: what every size shown for the model comes from once it is known. */ + totalBytes: number; + } + | { + ok: false; + code: Extract; + /** The runtime's last error text. */ + detail?: string; + } + | { ok: false; code: 'cancelled' }; + +/** How one request of a pull ended. */ +type PullAttempt = + | { kind: 'success'; totalBytes: number } + | { kind: 'cancelled' } + | { kind: 'fatal'; code: 'model-not-found' | 'disk-full'; detail: string } + | { kind: 'retry'; detail: string }; + +/** + * What an error text of the runtime means for the pull. A model the registry does not have and + * a full disk do not get better by asking again; everything else (a damaged download, a + * dropped connection) may. + */ +function classifyPullError(text: string): PullAttempt { + const detail = clip(text); + if (/file does not exist|manifest unknown|manifest\b.*\bnot found|model\b.*\bnot found/i.test(text)) { + return { kind: 'fatal', code: 'model-not-found', detail }; + } + if (/no space left on device|disk (is )?full|not enough (free )?(disk )?space/i.test(text)) { + return { kind: 'fatal', code: 'disk-full', detail }; + } + return { kind: 'retry', detail }; +} + +/** A wait that ends early when the signal aborts. */ +function waitFor(ms: number, signal?: AbortSignal): Promise { + return new Promise((resolve) => { + if (signal?.aborted) return resolve(); + const done = (): void => { + clearTimeout(timer); + signal?.removeEventListener('abort', done); + resolve(); + }; + const timer = setTimeout(done, ms); + signal?.addEventListener('abort', done, { once: true }); + }); +} + +/** One `POST /api/pull`, read to its end. */ +async function pullOnce(target: OllamaTarget, modelId: string, options: PullOptions): Promise { + const { signal, onProgress } = options; + const now = options.now ?? Date.now; + const fetchFn = target.fetch ?? fetch; + let res: Response; + try { + res = await fetchFn(`${target.baseUrl}/api/pull`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ model: modelId, stream: true }), + ...(signal ? { signal } : {}), + }); + } catch (err) { + return signal?.aborted ? { kind: 'cancelled' } : { kind: 'retry', detail: errorText(err) }; + } + + const parser = createPullParser(); + // The speed is the bytes of the last few seconds: (when, how far) pairs, the oldest first. + const samples: { at: number; completed: number }[] = []; + let lastReport = Number.NEGATIVE_INFINITY; + const report = (force: boolean): void => { + if (!onProgress) return; + const at = now(); + if (!force && at - lastReport < PULL_PROGRESS_INTERVAL_MS) return; + const snap = parser.snapshot(); + // Before the first layer line there is no size to show. + if (snap.totalBytes <= 0) return; + lastReport = at; + samples.push({ at, completed: snap.completedBytes }); + while (samples.length > 2 && at - samples[1].at >= PULL_SPEED_WINDOW_MS) samples.shift(); + const first = samples[0]; + const seconds = (at - first.at) / 1000; + const moved = snap.completedBytes - first.completed; + try { + onProgress({ + completedBytes: snap.completedBytes, + totalBytes: snap.totalBytes, + ...(seconds > 0 && moved >= 0 ? { bytesPerSecond: Math.round(moved / seconds) } : {}), + }); + } catch { + // A display that fails must not fail the download. + } + }; + + let streamError: string | undefined; + try { + if (res.body) { + for await (const chunk of res.body as unknown as AsyncIterable) { + parser.push(chunk); + report(false); + } + } + } catch (err) { + streamError = errorText(err); + } + // An abort ends the stream with an error too: it is a cancel, not a cut. + if (signal?.aborted) return { kind: 'cancelled' }; + parser.end(); + const snap = parser.snapshot(); + + if (snap.error !== undefined) return classifyPullError(snap.error); + if (res.status !== 200) return classifyPullError(`The runtime answered ${res.status} to the pull`); + if (!snap.succeeded) { + return { kind: 'retry', detail: streamError ?? 'The download ended before the runtime reported success' }; + } + report(true); + if (snap.totalBytes > 0) return { kind: 'success', totalBytes: snap.totalBytes }; + // A pull of a model that was already complete may name no layer. The size is then what the + // runtime lists for the model, which is the same figure a full pull reports as its total. + const listed = (await listTags(target, signal)).find((model) => model.name === modelId); + return { kind: 'success', totalBytes: listed?.size ?? 0 }; +} + +/** Pulls `modelId` and waits for the end. Resuming is calling it again. Never throws. */ +export async function pullModel(target: OllamaTarget, modelId: string, options: PullOptions = {}): Promise { + const { signal } = options; + const retries = + options.retries !== undefined && Number.isInteger(options.retries) && options.retries >= 0 + ? options.retries + : PULL_RETRIES; + const sleep = options.sleep ?? ((ms: number) => waitFor(ms, signal)); + let lastDetail: string | undefined; + for (let attempt = 0; attempt <= retries; attempt++) { + if (signal?.aborted) return { ok: false, code: 'cancelled' }; + if (attempt > 0) { + await sleep(PULL_RETRY_DELAY_MS * 2 ** (attempt - 1)); + if (signal?.aborted) return { ok: false, code: 'cancelled' }; + } + let outcome: PullAttempt; + try { + outcome = await pullOnce(target, modelId, options); + } catch (err) { + // Only an injected `fetch` or clock can throw here. It must not end the pull as a crash. + outcome = { kind: 'retry', detail: errorText(err) }; + } + if (outcome.kind === 'success') return { ok: true, totalBytes: outcome.totalBytes }; + if (outcome.kind === 'cancelled') return { ok: false, code: 'cancelled' }; + if (outcome.kind === 'fatal') return { ok: false, code: outcome.code, detail: outcome.detail }; + lastDetail = outcome.detail; + } + return { ok: false, code: 'pull-failed', ...(lastDetail !== undefined ? { detail: lastDetail } : {}) }; +} + +// ---------- The check after the pull ---------- + +/** The codes the check can end on. */ +export type VerifyFailureCode = Extract< + LocalModelErrorCode, + 'load-failed' | 'does-not-fit' | 'context-not-applied' | 'no-tool-call' | 'probe-unfinished' +>; + +/** What the runtime reported about the loaded model. */ +export interface LoadedModel { + context: number; + sizeBytes: number; + sizeVramBytes: number; +} + +/** The body of the tiny chat that loads the model: one short user message, `reasoning_effort: "none"`, a small `max_tokens`. Pure. */ +export function buildChatProbeRequest(modelId: string): Record { + return { + model: modelId, + messages: [{ role: 'user', content: 'Reply with the single word: ready' }], + reasoning_effort: 'none', + max_tokens: CHAT_PROBE_MAX_TOKENS, + temperature: 0, + stream: false, + }; +} + +/** A function tool of the tool probe, in the shape `/v1/chat/completions` takes. */ +function probeTool(name: string, description: string, properties: Record): Record { + return { + type: 'function', + function: { + name, + description, + parameters: { + type: 'object', + properties: Object.fromEntries( + Object.entries(properties).map(([key, about]) => [key, { type: 'string', description: about }]) + ), + required: Object.keys(properties), + }, + }, + }; +} + +/** The body of the tool probe: a `tools` array, `reasoning_effort: "none"`, `max_tokens: TOOL_PROBE_MAX_TOKENS`. Pure. */ +export function buildToolProbeRequest(modelId: string): Record { + return { + model: modelId, + messages: [ + { role: 'system', content: 'You are a coding agent. Use the tools you are given to do what the user asks.' }, + { role: 'user', content: 'Read the file README.md and tell me what it says.' }, + ], + // The three tools a coding agent always has. The task needs the first. + tools: [ + probeTool('read_file', 'Read a file and return its content.', { path: 'The path of the file.' }), + probeTool('write_file', 'Write a file.', { path: 'The path of the file.', content: 'The new content.' }), + probeTool('run_command', 'Run a shell command and return its output.', { command: 'The command line.' }), + ], + reasoning_effort: 'none', + max_tokens: TOOL_PROBE_MAX_TOKENS, + temperature: 0, + stream: false, + }; +} + +/** + * Judges the process list after the chat. Pure. + * - the model is not listed: `load-failed`; + * - `sizeVram` is not `size`: `does-not-fit` (part of the model is not on the GPU); + * - `contextLength` is not `context`: `context-not-applied`. + */ +export function judgeLoaded( + models: readonly OllamaPsModel[], + modelId: string, + context: number +): { ok: true; loaded: LoadedModel } | { ok: false; code: VerifyFailureCode; detail?: string } { + const model = models.find((entry) => entry.name === modelId); + if (!model) return { ok: false, code: 'load-failed', detail: 'The runtime does not list the model as loaded' }; + if (model.size <= 0) + return { ok: false, code: 'load-failed', detail: 'The runtime reports no size for the loaded model' }; + if (model.sizeVram !== model.size) { + return { + ok: false, + code: 'does-not-fit', + // A size a person reads: decimal GB, like every other size of this feature. + detail: `${formatGb(model.sizeVram)} of ${formatGb(model.size)} GB of the model are on the GPU`, + }; + } + if (model.contextLength !== context) { + return { + ok: false, + code: 'context-not-applied', + detail: `The runtime reports a context of ${model.contextLength} tokens, not ${context}`, + }; + } + return { ok: true, loaded: { context: model.contextLength, sizeBytes: model.size, sizeVramBytes: model.sizeVram } }; +} + +/** Whether the arguments of a tool call are a JSON object: as text that parses to one, or already as one. */ +function argumentsAreObject(value: unknown): boolean { + if (isRecord(value)) return true; + if (typeof value !== 'string') return false; + try { + return isRecord(JSON.parse(value)); + } catch { + return false; + } +} + +/** + * Judges the answer of the tool probe. Pure. `status` is the HTTP status, `body` the parsed + * JSON (or `undefined` when it did not parse). + * - a tool call whose `arguments` parse as a JSON object: passed (when the answer holds + * several calls, every one must have a name and such arguments); + * - `finish_reason` is `length`: `probe-unfinished` (the token limit ended it, so it proves nothing); + * - any other answer of status 200: `no-tool-call`; + * - another status: `load-failed`. + */ +export function judgeToolProbe( + status: number, + body: unknown +): { ok: true } | { ok: false; code: VerifyFailureCode; detail?: string } { + if (status !== 200) { + const said = errorOfBody(body); + return { ok: false, code: 'load-failed', detail: clip(said ?? `The runtime answered ${status}`) }; + } + const choice = isRecord(body) && Array.isArray(body.choices) ? body.choices[0] : undefined; + const message = isRecord(choice) && isRecord(choice.message) ? choice.message : undefined; + const calls = message && Array.isArray(message.tool_calls) ? message.tool_calls : []; + if (calls.length > 0) { + const wellFormed = calls.every( + (call) => + isRecord(call) && + isRecord(call.function) && + asString(call.function.name) !== undefined && + argumentsAreObject(call.function.arguments) + ); + if (wellFormed) return { ok: true }; + return { ok: false, code: 'no-tool-call', detail: 'The model called a tool with arguments that are not JSON' }; + } + if (isRecord(choice) && choice.finish_reason === 'length') { + return { ok: false, code: 'probe-unfinished', detail: `The answer ended at ${TOOL_PROBE_MAX_TOKENS} tokens` }; + } + const said = message ? asString(message.content) : undefined; + return { + ok: false, + code: 'no-tool-call', + detail: said ? clip(`The model answered in text: ${said}`) : 'The answer holds no tool call', + }; +} + +/** How the check ended. */ +export type VerifyResult = + | { ok: true; loaded: LoadedModel } + | { ok: false; code: VerifyFailureCode; detail?: string } + | { ok: false; code: 'cancelled' }; + +/** + * The whole check, in order: the chat, the process list, the tool probe. Stops at the first + * failure. Never throws. + */ +export async function verifyModel( + target: OllamaTarget, + modelId: string, + context: number, + signal?: AbortSignal +): Promise { + const cancelled: VerifyResult = { ok: false, code: 'cancelled' }; + if (signal?.aborted) return cancelled; + + // 1. The chat that loads the model. Its text is not read: only that it was answered. + const chat = await ask( + target, + '/v1/chat/completions', + { method: 'POST', json: buildChatProbeRequest(modelId) }, + CHAT_PROBE_TIMEOUT_MS, + signal + ); + if (signal?.aborted) return cancelled; + if (!chat) return { ok: false, code: 'load-failed', detail: 'The runtime did not answer the first request' }; + if (chat.status !== 200) { + return { + ok: false, + code: 'load-failed', + detail: clip(errorOfBody(chat.body) ?? `The runtime answered ${chat.status}`), + }; + } + + // 2. What the runtime says it loaded. + const judged = judgeLoaded(await listRunning(target, signal), modelId, context); + if (signal?.aborted) return cancelled; + if (!judged.ok) return judged; + + // 3. One tool call. + const probe = await ask( + target, + '/v1/chat/completions', + { method: 'POST', json: buildToolProbeRequest(modelId) }, + TOOL_PROBE_TIMEOUT_MS, + signal + ); + if (signal?.aborted) return cancelled; + if (!probe) + return { ok: false, code: 'probe-unfinished', detail: 'The runtime did not answer the tool request in time' }; + const tool = judgeToolProbe(probe.status, probe.body); + if (!tool.ok) return tool; + return { ok: true, loaded: judged.loaded }; +} diff --git a/src/local-model/paths.ts b/src/local-model/paths.ts new file mode 100644 index 00000000..749ece9a --- /dev/null +++ b/src/local-model/paths.ts @@ -0,0 +1,100 @@ +/** + * @fileoverview Where everything of the local model lives on disk. One pure function, so the + * runtime, the install job and uninstall all name the same folders. + * + * Layout, all under the data dir (`dataPath()`, so a second Codeman instance has its own): + * + * ``` + * local-model.json the state file (mode 0600) + * local-model/ + * runtime/ollama-.tgz.partial the runtime archive while it downloads + * runtime/ollama-.unpack/ where the archive is unpacked, before the rename + * runtime/ollama-/ the unpacked runtime; its binary is `ollama` + * models/ the runtime's model store + * home/ the runtime's private HOME (its key pair lands here) + * ``` + * + * Rules: + * - `root` is the ONE folder this feature owns. Nothing is written or deleted outside it, + * except the state file beside it and the two writes of the connect step (the endpoint + * record and the settings flag), which have their own writers. + * - Every path comes from here. None is ever read back from the state file, so a state file + * cannot point a delete anywhere else. `isInsideLocalModelRoot()` is the check a delete + * makes before it runs. + * - Pure apart from `dataPath()`, which resolves (and creates) the data dir. + */ + +import { join, resolve, sep } from 'node:path'; +import { dataPath } from '../config/instance.js'; + +/** A runtime version as it may appear in a folder name: digits and dots only ('0.40.15'). */ +const VERSION_PATTERN = /^\d+(?:\.\d+){1,3}$/; + +export interface LocalModelPaths { + /** `/local-model`: the one folder this feature owns. */ + root: string; + /** `/runtime`: the archive and every unpacked version. */ + runtimeRoot: string; + /** `/runtime/ollama-`: the unpacked runtime. */ + runtimeDir: string; + /** The runtime's binary: `/ollama`. */ + binary: string; + /** `/runtime/ollama-.tgz.partial`: the archive while it downloads. */ + archivePartial: string; + /** `/runtime/ollama-.unpack`: the temporary folder the archive is unpacked into. */ + unpackDir: string; + /** `/models`: the runtime's model store. */ + modelsDir: string; + /** `/home`: the runtime's private HOME. */ + homeDir: string; + /** `/local-model.json`: the state file. */ + stateFile: string; +} + +/** + * The paths of the local model for one runtime version. Throws on a version that is not + * digits and dots, so a version string can never add a path segment. + */ +export function localModelPaths(version: string): LocalModelPaths { + if (!VERSION_PATTERN.test(version)) { + throw new Error(`Not a runtime version: ${JSON.stringify(version)}`); + } + const root = dataPath('local-model'); + const runtimeRoot = join(root, 'runtime'); + const runtimeDir = join(runtimeRoot, `ollama-${version}`); + return { + root, + runtimeRoot, + runtimeDir, + binary: join(runtimeDir, 'ollama'), + archivePartial: join(runtimeRoot, `ollama-${version}.tgz.partial`), + unpackDir: join(runtimeRoot, `ollama-${version}.unpack`), + modelsDir: join(root, 'models'), + homeDir: join(root, 'home'), + stateFile: dataPath('local-model.json'), + }; +} + +/** + * The binary of another unpacked runtime version beside the pinned one: `/ + * ollama-/ollama`. For an installed model whose runtime was unpacked by an earlier + * build with another pin: it keeps running the version it was installed with. Throws on a + * version that is not digits and dots, like `localModelPaths()`. Pure. + */ +export function runtimeBinaryForVersion(paths: LocalModelPaths, version: string): string { + if (!VERSION_PATTERN.test(version)) { + throw new Error(`Not a runtime version: ${JSON.stringify(version)}`); + } + return join(paths.runtimeRoot, `ollama-${version}`, 'ollama'); +} + +/** + * Whether `path` is the local model folder itself or lies inside it, judged on the resolved + * text (no filesystem access, so a symlink is not followed: a caller that deletes checks the + * real path first). `..` segments are resolved before the comparison. + */ +export function isInsideLocalModelRoot(path: string, root: string = dataPath('local-model')): boolean { + const base = resolve(root); + const target = resolve(path); + return target === base || target.startsWith(base + sep); +} diff --git a/src/local-model/recommend.ts b/src/local-model/recommend.ts index 884cc71f..54a747f4 100644 --- a/src/local-model/recommend.ts +++ b/src/local-model/recommend.ts @@ -3,21 +3,33 @@ * fits this Mac. Pure: no IO, no clock, no CLI registry, no data dir. * * The math, in the order it runs: - * - `deviceBudget()`: L is the wired limit when it is above 0, else 2/3 of memory below 36 GB - * and 3/4 from 36 GB. B = min(L - 1 GiB, memory - reserve), with a reserve of 6 GiB up to - * 18 GB, 8 GiB up to 48 GB and 12 GiB above. - * - `needBytes()`: F = weights + kvBytesPerToken * ctx + 1 GiB + the prompt cache tier of the - * catalogue for this Mac's memory. - * - Fit: F/B up to 0.85 is good, up to 0.98 is tight, above is out. - * - Speed: tokens per second = the model's `speedPer100Gbps` x the chip's bandwidth / 100. - * Below 10 is slow, decided on the unrounded product (9.96 is slow although it prints as - * 10). A chip that is in no chips row, or a model without a speed figure, is UNKNOWN: a + * - `deviceBudget()`. ESTIMATED (all there is before the runtime ran): L is the wired limit + * when it is above 0, else 2/3 of memory below 36 GB and 3/4 from 36 GB, and + * B = min(L - 1 GiB, memory - reserve), with a reserve of 6 GiB up to 18 GB, 8 GiB up to + * 48 GB and 12 GiB above. MEASURED (`RecommendOptions.measuredTotalBytes`, the device total + * the runtime printed when it started): L is that total and B = min(L, memory - reserve), + * without the 1 GiB, because the figure is already what the runtime will really use. + * - `needBytes()`: the PEAK, `bytes + heldBytesPerToken * ctx + overheadBytes + + * peakExtraBytes` of the catalogue row: the weights, what stays held per token of context, + * what the runner holds beside both, and what it takes for a moment while a long prompt is + * processed. Measured for one model on one Mac (see the catalogue header); every other row + * carries the same figures and says `estimated`. + * - Fit: need over budget up to 0.85 is good, up to 0.98 is tight, above is out. The grade + * decides whether a model is offered at all and what its row says. It does NOT sort: the + * need is the worst moment at the full context, so a tight fit is a model that runs, and a + * Mac whose best model is tight should be offered that model, not a smaller one. + * - Speed: a candidate that matches a `tested` entry with a measured speed reports that + * speed. Every other one is estimated: the model's `speedPer100Gbps` x the chip's bandwidth + * / 100. Below 10 is slow, decided on the unrounded figure (9.96 is slow although it prints + * as 10). A chip that is in no chips row, or a model without a speed figure, is UNKNOWN: a * bandwidth is never guessed. * - Candidates: (model, context, harness) with `ctx >= harness.minContext`, the harness not * gated, installed or installable, and accepting the model id. - * - Ranking, lexicographic: good before tight; ok before unknown before slow; the higher model - * rank; the higher harness rank; the larger context. The pick is the first candidate, the - * alternatives are the next four MODELS, each at its own best context and harness. + * - Ranking, lexicographic: tested on a Mac like this before not tested (a measurement beats + * a formula); ok before unknown before slow; the higher model rank; the higher harness + * rank; the larger context. The pick is the first candidate, the alternatives are the next + * four MODELS, each at its own best context and harness, so a model is always offered at + * the largest context it fits at. * - Disk: a model whose download would leave less than 10 GiB free is not offered. It is not * dropped in silence: the best-ranked model left out that way is named in `noteDiskDropped` * with what is missing. @@ -32,8 +44,8 @@ * installable). * 6. `memory`: models fit, but none at a context a usable harness needs. Its own sentence * (`noneMemoryContext`) with the budget and the smallest need at such a context, because - * this Mac can miss by a hair (the 16 GB Mac mini: 10.3 of 10.4 GB) and "too little - * memory" would hide that the answer rests on an estimate. + * "too little memory" would hide that a model does fit at a smaller context and that the + * answer rests on estimates. * 7. `no-harness`: models fit at such a context, but no usable harness accepts their id. * 8. `disk`: for every remaining model, free disk minus the download is below 10 GiB (an * unknown free disk does not block). @@ -42,16 +54,20 @@ * - Verdict `limited` when NO model with 12 billion parameters or more fits this Mac's memory * at a context a usable harness works at. It is a statement about the Mac ("This Mac can run * a small model"), so it is judged on everything that fits, BEFORE the disk filter and - * whatever the pick is: a 36 GB Mac whose pick is a 9B model while a 20B, a 27B and a 35B - * are listed under it is not a small-model Mac, and a 48 GB Mac does not become one because - * its disk is full. A small pick under verdict `ok` carries `noteSmallPick` instead. + * whatever the pick is: a 36 GB Mac whose pick is a 9B model while a 20B is listed under it + * is not a small-model Mac, and a 48 GB Mac does not become one because its disk is full. A + * small pick under verdict `ok` carries `noteSmallPick` instead. * - The word "Recommended" (`LOCAL_MODEL_LABELS.headlineTested`) only for verdict `ok` with a * pick whose model has a `tested` entry for exactly this chip, this memory size and this * harness. Otherwise `headlineOk` (or `headlineLimited`) plus `noteUntested`. - * - Every label in the result comes from `LOCAL_MODEL_LABELS`, every computed size text from - * `formatGb()`. The one size that is not decimal GB is the installed memory in `noneMemory`: - * `HardwareFacts.memClassGb`, the size the Mac is sold as. No number is spliced into a - * sentence here. + * - Every row carries what the chooser needs and nothing the browser must work out: its one + * tag (`tagFor()`), every harness it can run with (`Candidate.harnesses`, the flat harness + * fields first), and the whole download of an install (`installBytes`: the model plus + * `RecommendOptions.runtimeDownloadBytes`). + * - Every label in the result comes from `LOCAL_MODEL_LABELS` or `LOCAL_MODEL_TAG_LABELS`, + * every computed size text from `formatGb()`. The one size that is not decimal GB is the + * installed memory in `noneMemory`: `HardwareFacts.memClassGb`, the size the Mac is sold + * as. No number is spliced into a sentence here. * - No CLI id is ever compared with a literal: a harness is a catalogue row plus a * `HarnessFact`. */ @@ -59,7 +75,11 @@ import { LOCAL_MODEL_LABELS, LOCAL_MODEL_NONE_LABELS, + LOCAL_MODEL_TAG_LABELS, type Candidate, + type CandidateHarness, + type CandidateTag, + type CandidateTagKind, type DeviceBudget, type FitGrade, type HardwareFacts, @@ -71,7 +91,9 @@ import { type LocalModelMessage, type NoneReason, type Recommendation, + type RecommendOptions, type SpeedGrade, + type TestedOn, } from '../types.js'; import { chipBandwidthGbps } from '../config/local-model-catalog.js'; import { formatGb, GIB, MIB } from './format.js'; @@ -80,14 +102,12 @@ import { formatGb, GIB, MIB } from './format.js'; export const FIT_GOOD_MAX_SHARE = 0.85; /** Need over budget up to this share is a `tight` fit; above it the model is out. */ export const FIT_TIGHT_MAX_SHARE = 0.98; -/** An estimated speed below this many tokens per second is `slow`. */ +/** A speed below this many tokens per second is `slow`. */ export const SLOW_BELOW_TOKENS_PER_SECOND = 10; /** When no model that fits has at least this many billion parameters, the verdict is `limited`. A pick below it is a "small model". */ export const LIMITED_BELOW_PARAMS_B = 12; /** What must stay free on the disk after the model download. */ export const DISK_HEADROOM_BYTES = 10 * GIB; -/** What the runtime needs beside the weights, the KV cache and the prompt cache (compute buffers). */ -export const RUNTIME_OVERHEAD_BYTES = GIB; /** How many other models are listed under the pick. */ export const MAX_ALTERNATIVES = 4; @@ -101,30 +121,39 @@ function reserveBytes(memClassGb: number): number { return 12 * GIB; } -/** The memory a model may use on this Mac: the device limit L, the budget B and where L came from. */ -export function deviceBudget(hw: HardwareFacts): DeviceBudget { +/** A byte count a caller supplied, when it is one: finite and above 0. */ +function positiveBytes(value: number | undefined): number | undefined { + return value !== undefined && Number.isFinite(value) && value > 0 ? Math.floor(value) : undefined; +} + +/** + * The memory a model may use on this Mac: the device limit L, the budget B and where L came + * from. With `options.measuredTotalBytes` the limit is what the runtime itself reported and + * the 1 GiB the estimate keeps back is not subtracted; the reserve bounds both. + */ +export function deviceBudget(hw: HardwareFacts, options: RecommendOptions = {}): DeviceBudget { const memBytes = Number.isFinite(hw.memBytes) && hw.memBytes > 0 ? hw.memBytes : 0; + const afterReserve = memBytes - reserveBytes(hw.memClassGb); + const measured = positiveBytes(options.measuredTotalBytes); + if (measured !== undefined) { + return { limitBytes: measured, budgetBytes: Math.max(0, Math.min(measured, afterReserve)), source: 'measured' }; + } // 0 means "not set" (both test Macs report 0): only a value above 0 is a limit. const wired = hw.wiredLimitMb !== undefined && Number.isFinite(hw.wiredLimitMb) && hw.wiredLimitMb > 0; const limitBytes = wired ? Math.floor((hw.wiredLimitMb as number) * MIB) : Math.floor(memBytes * (hw.memClassGb < LARGE_LIMIT_FROM_GB ? 2 / 3 : 3 / 4)); - const budgetBytes = Math.max(0, Math.min(limitBytes - GIB, memBytes - reserveBytes(hw.memClassGb))); + const budgetBytes = Math.max(0, Math.min(limitBytes - GIB, afterReserve)); return { limitBytes, budgetBytes, source: wired ? 'wired-limit' : 'estimated' }; } -/** The prompt cache of the last tier whose minimum is not above this Mac's memory size. */ -function promptCacheBytes(catalog: LocalModelCatalog, memClassGb: number): number { - let bytes = 0; - for (const tier of catalog.promptCacheTiers) { - if (tier.minMemGb <= memClassGb) bytes = tier.bytes; - } - return bytes; -} - -/** F: the bytes `model` needs in memory at a context of `ctx` tokens on this Mac. */ -export function needBytes(model: LocalModelEntry, ctx: number, catalog: LocalModelCatalog, hw: HardwareFacts): number { - return model.bytes + model.kvBytesPerToken * ctx + RUNTIME_OVERHEAD_BYTES + promptCacheBytes(catalog, hw.memClassGb); +/** + * The bytes `model` needs in memory at a context of `ctx` tokens, at its PEAK: the weights, + * what stays held for the context, what the runner holds beside them, and what it takes for a + * moment while a long prompt is processed. It does not depend on the Mac. + */ +export function needBytes(model: LocalModelEntry, ctx: number): number { + return model.bytes + model.heldBytesPerToken * ctx + model.overheadBytes + model.peakExtraBytes; } function fitGrade(need: number, budgetBytes: number): FitGrade { @@ -135,6 +164,26 @@ function fitGrade(need: number, budgetBytes: number): FitGrade { return 'out'; } +/** + * Whether the catalogue model `modelId` at `context` tokens fits this Mac's budget (a `good` + * or a `tight` fit). The install job asks this right after the runtime started, with the + * budget the runtime measured (`options.measuredTotalBytes`), BEFORE it downloads the model: a + * `false` ends the job with `does-not-fit`. It is the same judgement `recommend()` makes for a + * candidate, without the ranking, so a model that fits is never ruled out for being listed + * sixth. False for a model the catalogue does not have. + */ +export function modelFitsBudget( + hw: HardwareFacts, + catalog: LocalModelCatalog, + modelId: string, + context: number, + options: RecommendOptions = {} +): boolean { + const model = catalog.models.find((entry) => entry.id === modelId); + if (!model || !Number.isFinite(context) || context <= 0) return false; + return fitGrade(needBytes(model, context), deviceBudget(hw, options).budgetBytes) !== 'out'; +} + /** * Whether the MACHINE is Apple silicon. `hw.optional.arm64` decides when it was read. Without * it: a process under Rosetta, an arm64 process or an Apple chip name each prove Apple silicon @@ -165,13 +214,12 @@ interface UsableHarness { fact: HarnessFact; } -const FIT_ORDER: Record = { good: 0, tight: 1 }; const SPEED_ORDER: Record = { ok: 0, unknown: 1, slow: 2 }; -/** Best first. */ +/** Best first. The fit grade is not a key: every candidate here fits, see the file header. */ function compareRanked(a: Ranked, b: Ranked): number { return ( - FIT_ORDER[a.candidate.fit] - FIT_ORDER[b.candidate.fit] || + Number(b.candidate.tested) - Number(a.candidate.tested) || SPEED_ORDER[a.candidate.speed] - SPEED_ORDER[b.candidate.speed] || b.modelRank - a.modelRank || b.harnessRank - a.harnessRank || @@ -188,27 +236,53 @@ function accepts(fact: HarnessFact, modelId: string): boolean { } } +/** What the chooser's "Runs with" select needs to know about one harness. */ +function candidateHarness(harness: UsableHarness): CandidateHarness { + const { row, fact } = harness; + return { + cli: row.cli, + cliLabel: fact.label, + installed: fact.installed, + // Only a harness that still has to be installed has a command to show. + ...(!fact.installed && fact.installCommand ? { installCommand: fact.installCommand } : {}), + ...(row.slowFirstTurn ? { slowFirstTurn: true } : {}), + }; +} + +/** + * The entry that makes this model "tested on a Mac like this": exactly the same chip string, + * the same memory size as sold, the same harness. + */ +function testedEntry(model: LocalModelEntry, hw: HardwareFacts, cli: string): TestedOn | undefined { + return model.tested.find((entry) => entry.chip === hw.chip && entry.memGb === hw.memClassGb && entry.cli === cli); +} + function buildCandidate( model: LocalModelEntry, ctx: number, harness: UsableHarness, + /** Every usable harness that can run this model at this context, `harness` among them. */ + runsWith: readonly UsableHarness[], need: number, fit: Candidate['fit'], hw: HardwareFacts, catalog: LocalModelCatalog, - budgetBytes: number + budgetBytes: number, + runtimeDownloadBytes: number ): Candidate { + const testedOn = testedEntry(model, hw, harness.row.cli); const gbps = chipBandwidthGbps(catalog.chips, hw.chip); + // A speed measured on a Mac like this one beats the estimate from the chip's bandwidth. const rawSpeed = - gbps !== undefined && model.speedPer100Gbps !== undefined ? (model.speedPer100Gbps * gbps) / 100 : undefined; + testedOn?.tokensPerSecond ?? + (gbps !== undefined && model.speedPer100Gbps !== undefined ? (model.speedPer100Gbps * gbps) / 100 : undefined); // Graded on the unrounded figure: 9.96 rounds to 10.0 for the payload and is still slow. const speed: SpeedGrade = rawSpeed === undefined ? 'unknown' : rawSpeed < SLOW_BELOW_TOKENS_PER_SECOND ? 'slow' : 'ok'; const tokensPerSecond = rawSpeed === undefined ? undefined : round(rawSpeed, 1); - // "A Mac like this" is exact: the same chip string, the same memory size as sold, the same harness. - const tested = model.tested.some( - (entry) => entry.chip === hw.chip && entry.memGb === hw.memClassGb && entry.cli === harness.row.cli - ); + // The harness of the flat fields first, then the others, best first. + const others = runsWith.filter((other) => other !== harness).sort((a, b) => b.row.rank - a.row.rank); + const installBytes = model.bytes + runtimeDownloadBytes; return { modelId: model.id, modelLabel: model.label, @@ -230,10 +304,41 @@ function buildCandidate( ...(speed === 'slow' ? { speedLabel: LOCAL_MODEL_LABELS.speedSlow } : {}), ...(speed === 'unknown' ? { speedLabel: LOCAL_MODEL_LABELS.speedUnknown } : {}), ...(tokensPerSecond !== undefined ? { tokensPerSecond } : {}), - tested, + tested: testedOn !== undefined, + needSource: model.figures, + harnesses: [harness, ...others].map(candidateHarness), + installBytes, + installGb: formatGb(installBytes), }; } +/** + * The one tag of a row: the first kind that applies, in the order tested, the pick (which + * says so when it only just fits), tight, slow, small. `undefined` when none does. + */ +function tagFor(candidate: Candidate, isPick: boolean): CandidateTag | undefined { + const kind: CandidateTagKind | undefined = candidate.tested + ? 'tested' + : isPick + ? candidate.fit === 'tight' + ? 'best-tight' + : 'best' + : candidate.fit === 'tight' + ? 'tight' + : candidate.speed === 'slow' + ? 'slow' + : candidate.paramsB < LIMITED_BELOW_PARAMS_B + ? 'small' + : undefined; + return kind === undefined ? undefined : { kind, label: LOCAL_MODEL_TAG_LABELS[kind] }; +} + +/** `candidate` with its tag, when it has one. */ +function withTag(candidate: Candidate, isPick: boolean): Candidate { + const tag = tagFor(candidate, isPick); + return tag ? { ...candidate, tag } : candidate; +} + /** A `none` answer. `text` replaces the reason's default sentence (the `memory` reason has three). */ function none( reason: NoneReason, @@ -256,18 +361,24 @@ function none( /** * The recommendation for this Mac. `harnessFacts` holds one fact per catalogue harness the * caller could resolve; a catalogue harness without a fact is not offered. + * + * `options` is what the caller knows beyond the probe: the budget the runtime measured (the + * answer then says `budgetSource: 'measured'`), and the runtime download an install still + * needs (it is added to every row's `installBytes`). */ export function recommend( hw: HardwareFacts, catalog: LocalModelCatalog, - harnessFacts: readonly HarnessFact[] + harnessFacts: readonly HarnessFact[], + options: RecommendOptions = {} ): Recommendation { // Off macOS and on an Intel Mac there is no budget to compute: a model has no memory to run in. const noBudget: DeviceBudget = { limitBytes: 0, budgetBytes: 0, source: 'estimated' }; if (hw.platform !== 'darwin') return none('not-macos', noBudget); if (!isAppleSilicon(hw)) return none('intel', noBudget); - const budget = deviceBudget(hw); + const budget = deviceBudget(hw, options); + const runtimeDownloadBytes = positiveBytes(options.runtimeDownloadBytes) ?? 0; if (hw.macosMajor !== undefined && hw.macosMajor < catalog.minMacosMajor) { return none('macos', budget, { version: hw.macosVersion ?? String(hw.macosMajor), @@ -279,7 +390,7 @@ export function recommend( const fitting: { model: LocalModelEntry; ctx: number; need: number; fit: Candidate['fit'] }[] = []; for (const model of catalog.models) { for (const ctx of model.contexts) { - const need = needBytes(model, ctx, catalog, hw); + const need = needBytes(model, ctx); const fit = fitGrade(need, budget.budgetBytes); if (fit !== 'out') fitting.push({ model, ctx, need, fit }); } @@ -305,14 +416,14 @@ export function recommend( usable.filter((harness) => entry.ctx >= harness.row.minContext).map((harness) => ({ ...entry, harness })) ); if (atHarnessContext.length === 0) { - // Models fit, but only below the context a harness needs. Say both numbers: the smallest - // need at such a context can sit a hair over the line, and the budget is an estimate. + // Models fit, but only below the context a harness needs. Say both numbers: the reader + // sees how far apart they are, and that both are estimates. const smallestContext = Math.min(...usable.map((harness) => harness.row.minContext)); let smallestNeed: number | undefined; for (const model of catalog.models) { for (const ctx of model.contexts) { if (ctx < smallestContext) continue; - const need = needBytes(model, ctx, catalog, hw); + const need = needBytes(model, ctx); if (smallestNeed === undefined || need < smallestNeed) smallestNeed = need; } } @@ -333,12 +444,8 @@ export function recommend( const hasRoom = (model: LocalModelEntry): boolean => free === undefined || free - model.bytes >= DISK_HEADROOM_BYTES; if (free !== undefined && !accepted.some((entry) => hasRoom(entry.model))) { const smallest = Math.min(...accepted.map((entry) => entry.model.bytes)); - return none('disk', budget, { - download: formatGb(smallest), - reserve: formatGb(DISK_HEADROOM_BYTES), - free: hw.freeDiskGb ?? formatGb(free), - missing: formatGb(smallest + DISK_HEADROOM_BYTES - free), - }); + // One number, the one that says what to do: what must be freed for the smallest model. + return none('disk', budget, { missing: formatGb(smallest + DISK_HEADROOM_BYTES - free) }); } // Everything that fits is ranked, the models without room on the disk included: the best of @@ -349,11 +456,15 @@ export function recommend( entry.model, entry.ctx, entry.harness, + accepted + .filter((other) => other.model === entry.model && other.ctx === entry.ctx) + .map((other) => other.harness), entry.need, entry.fit, hw, catalog, - budget.budgetBytes + budget.budgetBytes, + runtimeDownloadBytes ), model: entry.model, modelRank: entry.model.rank, @@ -368,7 +479,8 @@ export function recommend( } const offered = bestPerModel.filter((entry) => hasRoom(entry.model)).map((entry) => entry.candidate); const leftOutForDisk = bestPerModel.find((entry) => !hasRoom(entry.model)); - const [pick, ...others] = offered; + const pick = withTag(offered[0], true); + const alternatives = offered.slice(1, 1 + MAX_ALTERNATIVES).map((candidate) => withTag(candidate, false)); // About the Mac, not the pick, and judged before the disk filter (see the file header). const limited = !accepted.some((entry) => entry.model.paramsB >= LIMITED_BELOW_PARAMS_B); @@ -397,7 +509,7 @@ export function recommend( headline: { text: headline }, notes, pick, - alternatives: others.slice(0, MAX_ALTERNATIVES), + alternatives, budgetBytes: budget.budgetBytes, budgetGb: formatGb(budget.budgetBytes), budgetSource: budget.source, diff --git a/src/local-model/runtime.ts b/src/local-model/runtime.ts new file mode 100644 index 00000000..0ff00614 --- /dev/null +++ b/src/local-model/runtime.ts @@ -0,0 +1,1407 @@ +/** + * @fileoverview The model runtime: which build Codeman runs, how it gets onto the disk, and + * how it is kept running. + * + * What this module does: + * - THE PIN. `OLLAMA_RUNTIME_PIN` is a compiled constant. No file, setting, environment + * variable or request can change the URL, the size or the hash. + * - DOWNLOAD (`ensureRuntimeInstalled()`). The archive is streamed to + * `LocalModelPaths.archivePartial` while it is hashed. A cut stream is continued with a + * Range request, at most `RUNTIME_DOWNLOAD_RETRIES` times. A body longer than `pin.bytes` + * is refused. The sha256 is compared BEFORE anything is unpacked, and a mismatch deletes the + * file. Unpacking is `/usr/bin/tar` into `LocalModelPaths.unpackDir`, then one rename to + * `LocalModelPaths.runtimeDir`. + * - ENVIRONMENT (`runtimeEnv()`, pure). The child's environment is built from an ALLOWLIST: + * PATH, TMPDIR, LANG and the proxy variables (`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`, in + * either case) are copied from the server's; HOME is the private one; and exactly these are + * set: `OLLAMA_HOST=127.0.0.1:`, `OLLAMA_MODELS`, `OLLAMA_CONTEXT_LENGTH`, + * `OLLAMA_NO_CLOUD=1`, `OLLAMA_KEEP_ALIVE=10m`, `OLLAMA_NUM_PARALLEL=1`, + * `OLLAMA_MAX_LOADED_MODELS=1`. Nothing else gets through: no `CODEMAN_*` and none of the + * user's own `OLLAMA_*`. + * - PORT (`chooseRuntimePort()`). Chosen ONCE by binding, from `RUNTIME_PORT_FIRST` across + * `RUNTIME_PORT_SPAN`, then stored by the caller. Never searched again on a later start. + * - SUPERVISOR (`createRuntimeSupervisor()`). The runtime is a NON-detached child of the + * server. `start()` first looks at the port: a listener that answers `/api/version` and + * whose pid (`previousPid`) runs a binary under our runtime folder is our own orphan and is + * adopted or replaced; any other listener is the answer `port-in-use`. An orphan is ADOPTED + * only when keeping it keeps something: it answers the pinned version and has a model loaded + * at the wanted context. Otherwise it is REPLACED (SIGTERM, then a child of our own), which + * loses nothing and gives back what an orphan cannot: its log, so the measured budget, and + * its exit. A child that exits by itself is restarted, at most `RUNTIME_MAX_RESTARTS` times + * since the last `start()`; `stop()` sends SIGTERM. The supervisor does not write the state + * file: it reports through `onChange`, and its caller stores the pid and the port. The pid + * is reported the moment the child is SPAWNED, before it answers, so a server that is + * killed during a start still leaves a pid that names the runtime as ours. `stop()` takes + * that stored pid too (`previousPid`): a runtime an earlier server left behind is ended the + * same way, when its command line is a binary under our runtime folder. `start()` runs the + * pinned version, or with `version` another one that is unpacked beside it (the version an + * installed model was installed with). + * - BUDGET (`parseInferenceComputeTotal()`, pure). Right after it starts, the runtime prints + * the device total in its log line "inference compute" (`total="37.4 GiB"`). The supervisor + * reads the child's output for it and reports it as `measuredTotalBytes`. + * - REMOVAL (`removeLocalModelFiles()`). Deletes the runtime, the private home and, when + * asked, the models. Only inside `LocalModelPaths.root`, and never through a `root` that + * is itself a symbolic link. `removePartialModelBlobs()` deletes what an interrupted + * download left in the model store. + * + * Under vitest nothing spawns and nothing is fetched unless a test injects it: the default + * `spawn` and `fetch` of the dependency objects refuse there, the default `commandOfPid` + * knows no pid and the default `kill` signals nothing. + */ + +import { execFile, spawn as nodeSpawn, type ChildProcess, type SpawnOptions } from 'node:child_process'; +import { createHash, type Hash } from 'node:crypto'; +import { createReadStream } from 'node:fs'; +import { lstat, mkdir, open, readdir, realpath, rename, rm, rmdir, stat, unlink } from 'node:fs/promises'; +import { createServer } from 'node:net'; +import { join, resolve as resolvePath } from 'node:path'; +import type { LocalModelErrorCode, LocalModelRuntimePin } from '../types.js'; +import { getRuntimeVersion, listRunning } from './ollama-client.js'; +import { isInsideLocalModelRoot, localModelPaths, runtimeBinaryForVersion, type LocalModelPaths } from './paths.js'; + +// ---------- The pin and the constants ---------- + +/** The one runtime build Codeman downloads and runs. See the file header. */ +export const OLLAMA_RUNTIME_PIN: LocalModelRuntimePin = Object.freeze({ + id: 'ollama', + version: '0.40.15', + url: 'https://github.com/ollama/ollama/releases/download/v0.40.15/ollama-darwin.tgz', + bytes: 185_971_671, + sha256: '607c79d5d6315ef9d896db83adde736a4c0057196329843544e79b95afb260a8', + // Measured on a MacBook on 2026-11-06: the unpacked folder takes 579 MB. + unpackedBytes: 579_000_000, + minMacosMajor: 14, +}); + +/** The first loopback port tried for the runtime. Not 11434: that is the default of a runtime the user may run. */ +export const RUNTIME_PORT_FIRST = 11435; +/** How many ports are tried from `RUNTIME_PORT_FIRST`. */ +export const RUNTIME_PORT_SPAN = 20; +/** How often in a row a runtime that exited by itself is started again. */ +export const RUNTIME_MAX_RESTARTS = 3; +/** How often a cut archive download is continued. */ +export const RUNTIME_DOWNLOAD_RETRIES = 3; +/** `OLLAMA_KEEP_ALIVE`: how long the model stays loaded after the last request. */ +export const RUNTIME_KEEP_ALIVE = '10m'; + +/** The name of the runtime's binary inside its folder and inside the archive. */ +const RUNTIME_BINARY_NAME = 'ollama'; +/** A download that delivers no byte for this long is treated as cut. */ +const DOWNLOAD_IDLE_TIMEOUT_MS = 60_000; +/** The first wait before a cut download is continued; it doubles with every retry. */ +const DOWNLOAD_RETRY_DELAY_MS = 500; +/** The least time between two progress reports of the download. */ +const DOWNLOAD_PROGRESS_INTERVAL_MS = 200; +/** How long `/usr/bin/tar` may take for the 579 MB of the runtime. */ +const UNTAR_TIMEOUT_MS = 180_000; +/** How long a started runtime has to answer `/api/version`. The first start on the test Mac took a few seconds. */ +const START_TIMEOUT_MS = 30_000; +/** The wait between two `/api/version` requests while the runtime starts. */ +const START_POLL_MS = 150; +/** How long `start()` waits for the budget line once the runtime answers. It is printed within a second. */ +const BUDGET_WAIT_MS = 3_000; +/** How long a runtime gets to exit after SIGTERM before it is sent SIGKILL. */ +const STOP_GRACE_MS = 5_000; +/** How long the exit is awaited after SIGKILL. */ +const STOP_KILL_WAIT_MS = 2_000; +/** The wait before a runtime that exited by itself is started again. */ +const RESTART_DELAY_MS = 1_000; +/** How often an adopted orphan is looked at: it is not our child, so its exit is not an event. */ +const ORPHAN_POLL_MS = 5_000; +/** How long `ps` may take. */ +const PS_TIMEOUT_MS = 2_000; +/** How many of the runtime's last log lines are kept for the detail of a failed start. */ +const LOG_TAIL_LINES = 12; +/** A log line longer than this is cut: the runtime logs request bodies in some modes. */ +const LOG_LINE_MAX_CHARS = 2_000; +/** How much of a failure text is kept as `detail`. */ +const MAX_DETAIL_CHARS = 600; + +const UNDER_VITEST = Boolean(process.env.VITEST); + +/** A default dependency that refuses under vitest: a test must inject its own. */ +class NotInjectedError extends Error { + constructor(what: string) { + super(`${what} is not available under vitest: the test must inject its own`); + this.name = 'NotInjectedError'; + } +} + +// ---------- Small helpers ---------- + +function clip(text: string): string { + const trimmed = text.trim(); + return trimmed.length > MAX_DETAIL_CHARS ? `${trimmed.slice(0, MAX_DETAIL_CHARS)}...` : trimmed; +} + +function errorText(err: unknown): string { + if (err instanceof Error) { + // undici wraps the reason ("fetch failed") and keeps the useful part in `cause`. + const cause = (err as { cause?: unknown }).cause; + return clip(cause instanceof Error && cause.message ? `${err.message}: ${cause.message}` : err.message); + } + return clip(String(err)); +} + +function errnoOf(err: unknown): string | undefined { + const code = (err as { code?: unknown } | null | undefined)?.code; + return typeof code === 'string' ? code : undefined; +} + +function isDiskFull(err: unknown): boolean { + const code = errnoOf(err); + return code === 'ENOSPC' || code === 'EDQUOT' || /no space left on device/i.test(errorText(err)); +} + +/** A wait on the real clock that does not keep the process alive. For grace periods, which an injected `sleep` must not shorten. */ +function pause(ms: number): Promise { + return new Promise((resolve) => { + setTimeout(resolve, ms).unref(); + }); +} + +function realSleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** The size of a file, 0 when it is not there. */ +async function fileSize(path: string): Promise { + try { + const info = await stat(path); + return info.isFile() ? info.size : 0; + } catch { + return 0; + } +} + +async function isFile(path: string): Promise { + try { + return (await stat(path)).isFile(); + } catch { + return false; + } +} + +// ---------- Pure ---------- + +const BINARY_UNIT_BYTES: Record = { + B: 1, + KiB: 1024, + MiB: 1024 ** 2, + GiB: 1024 ** 3, + TiB: 1024 ** 4, +}; + +/** + * The device total of the runtime's log line "inference compute", in bytes, or `undefined` + * for any other line. The real line (a MacBook, 2026-11-06): + * + * `time=... level=INFO source=types.go:32 msg="inference compute" id=0 ... type=iGPU total="37.4 GiB" available="37.4 GiB"` + * + * reads 40,157,944,217 (37.4 x 1024^3, rounded down). The value is the quoted text after + * ` total=`: a number and a binary unit (`B`, `KiB`, `MiB`, `GiB`, `TiB`). `total_vram=` of the + * line "vram-based default context" is not read. + */ +export function parseInferenceComputeTotal(line: string): number | undefined { + if (!line.includes('msg="inference compute"')) return undefined; + // The key must stand alone: `total_vram=` and `subtotal=` are other keys. + const match = /(?:^|\s)total="(\d+(?:\.\d+)?) ?(B|KiB|MiB|GiB|TiB)"/.exec(line); + if (!match) return undefined; + const bytes = Math.floor(Number(match[1]) * BINARY_UNIT_BYTES[match[2]]); + return Number.isFinite(bytes) && bytes > 0 ? bytes : undefined; +} + +/** What `runtimeEnv()` is built from. */ +export interface RuntimeEnvInput { + /** The server's environment. Only the allowlisted names are read from it. */ + source: NodeJS.ProcessEnv; + /** The private HOME (`LocalModelPaths.homeDir`). */ + home: string; + /** The model store (`LocalModelPaths.modelsDir`). */ + modelsDir: string; + /** The loopback port. */ + port: number; + /** The context length in tokens. */ + contextLength: number; +} + +/** The names copied from the server's environment when they are set. Everything else stays behind. */ +const RUNTIME_ENV_COPIED = [ + 'PATH', + 'TMPDIR', + 'LANG', + 'HTTP_PROXY', + 'HTTPS_PROXY', + 'NO_PROXY', + 'http_proxy', + 'https_proxy', + 'no_proxy', +] as const; + +/** The environment of the runtime child: the allowlist of the file header, nothing more. Pure. */ +export function runtimeEnv(input: RuntimeEnvInput): NodeJS.ProcessEnv { + const env: NodeJS.ProcessEnv = {}; + for (const name of RUNTIME_ENV_COPIED) { + const value = input.source[name]; + if (typeof value === 'string' && value !== '') env[name] = value; + } + env.HOME = input.home; + env.OLLAMA_HOST = `127.0.0.1:${input.port}`; + env.OLLAMA_MODELS = input.modelsDir; + env.OLLAMA_CONTEXT_LENGTH = String(input.contextLength); + env.OLLAMA_NO_CLOUD = '1'; + env.OLLAMA_KEEP_ALIVE = RUNTIME_KEEP_ALIVE; + env.OLLAMA_NUM_PARALLEL = '1'; + env.OLLAMA_MAX_LOADED_MODELS = '1'; + return env; +} + +/** A path without the `/private` macOS puts in front of `/var`, `/tmp` and `/etc`. */ +function withoutPrivatePrefix(path: string): string { + return path.startsWith('/private/') ? path.slice('/private'.length) : path; +} + +/** + * Whether `command` (one line of `ps -o command= -p `) is a process whose binary lies + * under `runtimeRoot` (`LocalModelPaths.runtimeRoot`): our own runtime, so an orphan of an + * earlier server. Pure. `/private` in front of a path is the same path on macOS. + * + * It tells our orphan from a runtime the user started, so that only ours is ever signalled. + * It is not a defence against another program of the same account, which can name itself + * anything. + */ +export function isOwnRuntimeCommand(command: string, runtimeRoot: string): boolean { + const root = withoutPrivatePrefix(runtimeRoot.trim()).replace(/\/+$/, ''); + if (!root.startsWith('/') || root === '') return false; + const text = withoutPrivatePrefix(command.trimStart()); + if (!text.startsWith(`${root}/`)) return false; + // A `..` segment could lead out of the folder again. + return !/(?:^|\/)\.\.(?:\/|\s|$)/.test(text); +} + +// ---------- Download and unpack ---------- + +/** How far the archive download is. */ +export interface RuntimeDownloadProgress { + completedBytes: number; + /** `pin.bytes`. */ + totalBytes: number; +} + +/** What `ensureRuntimeInstalled()` calls. Each member defaults to the real thing; a test injects its own. */ +export interface RuntimeInstallDeps { + /** Fetches the archive. Must honour `signal` and a `Range` header. */ + fetch: typeof fetch; + /** Unpacks `archive` into the existing empty folder `intoDir` (`/usr/bin/tar -xzf`). Rejects on a non-zero exit. */ + untar: (archive: string, intoDir: string, signal?: AbortSignal) => Promise; + /** Waits between retries. */ + sleep: (ms: number) => Promise; +} + +export interface RuntimeInstallOptions { + /** Default `OLLAMA_RUNTIME_PIN`. A test passes a pin that points at its fake server. */ + pin?: LocalModelRuntimePin; + /** Default `localModelPaths(pin.version)`. */ + paths?: LocalModelPaths; + /** Aborts the download or the unpack. The partial archive stays for a later Range request. */ + signal?: AbortSignal; + /** Called while the archive downloads, at most a few times a second. */ + onProgress?: (progress: RuntimeDownloadProgress) => void; + deps?: Partial; + /** How long the download may deliver nothing before it counts as cut. Default one minute. */ + idleTimeoutMs?: number; +} + +/** + * How getting the runtime ended. A failure carries the job's error code; `cancelled` is the + * abort signal and is not an error. + */ +export type RuntimeInstallResult = + | { + ok: true; + /** `LocalModelPaths.binary`. */ + binaryPath: string; + /** True: the folder was already unpacked and nothing was downloaded. */ + alreadyInstalled: boolean; + } + | { + ok: false; + code: Extract< + LocalModelErrorCode, + 'runtime-download-failed' | 'runtime-damaged' | 'runtime-unpack-failed' | 'disk-full' + >; + detail?: string; + } + | { ok: false; code: 'cancelled' }; + +/** A failure of `ensureRuntimeInstalled()`, without the success arm. */ +type RuntimeInstallFailure = Exclude; + +const CANCELLED: RuntimeInstallFailure = { ok: false, code: 'cancelled' }; + +/** The default `fetch` of the download: the global one, refused under vitest. */ +const defaultFetch: typeof fetch = (input, init) => { + if (UNDER_VITEST) return Promise.reject(new NotInjectedError('fetch')); + return fetch(input, init); +}; + +/** The default `untar`: `/usr/bin/tar -xzf -C `, bounded. */ +function untarWithSystemTar(archive: string, intoDir: string, signal?: AbortSignal): Promise { + return new Promise((resolve, reject) => { + execFile( + '/usr/bin/tar', + ['-xzf', archive, '-C', intoDir], + { timeout: UNTAR_TIMEOUT_MS, ...(signal ? { signal } : {}) }, + (err, _stdout, stderr) => { + if (!err) return resolve(); + const said = typeof stderr === 'string' && stderr.trim() !== '' ? stderr : err.message; + reject(new Error(clip(said))); + } + ); + }); +} + +/** Whether the pinned runtime is unpacked on disk (its binary exists). Never throws. */ +export async function isRuntimeInstalled(pin?: LocalModelRuntimePin, paths?: LocalModelPaths): Promise { + try { + const resolved = paths ?? localModelPaths((pin ?? OLLAMA_RUNTIME_PIN).version); + return await isFile(resolved.binary); + } catch { + return false; + } +} + +/** Feeds a file on disk into `hash`: the part of the archive an earlier run left behind. */ +async function hashFile(path: string, hash: Hash, signal?: AbortSignal): Promise { + for await (const chunk of createReadStream(path, signal ? { signal } : {})) { + hash.update(chunk as Buffer); + } +} + +/** How one request for the archive ended. `detail` says why it must be asked again. */ +type DownloadAttempt = + | { kind: 'ended'; detail?: string } + | { kind: 'cancelled' } + | { kind: 'failed'; result: RuntimeInstallFailure }; + +/** + * Downloads the pinned archive to `paths.archivePartial` and proves its hash. On success the + * file is complete and verified; nothing was unpacked. Never throws. + */ +async function downloadArchive( + pin: LocalModelRuntimePin, + paths: LocalModelPaths, + deps: RuntimeInstallDeps, + options: RuntimeInstallOptions +): Promise<{ ok: true } | RuntimeInstallFailure> { + const { signal, onProgress } = options; + const idleTimeoutMs = options.idleTimeoutMs ?? DOWNLOAD_IDLE_TIMEOUT_MS; + const partial = paths.archivePartial; + + // What an earlier run left: continue after it. More than the whole archive cannot be it. + let offset = await fileSize(partial); + if (offset > pin.bytes) { + await rm(partial, { force: true }); + offset = 0; + } + let hash = createHash('sha256'); + if (offset > 0) await hashFile(partial, hash, signal); + + let lastReport = Number.NEGATIVE_INFINITY; + const report = (force: boolean): void => { + if (!onProgress) return; + const at = Date.now(); + if (!force && at - lastReport < DOWNLOAD_PROGRESS_INTERVAL_MS) return; + lastReport = at; + try { + onProgress({ completedBytes: offset, totalBytes: pin.bytes }); + } catch { + // A display that fails must not fail the download. + } + }; + + /** One request, written to the file as far as it gets. */ + const attempt = async (): Promise => { + const controller = new AbortController(); + const onAbort = (): void => controller.abort(); + signal?.addEventListener('abort', onAbort, { once: true }); + let idle: NodeJS.Timeout | undefined; + let wentIdle = false; + const armIdle = (): void => { + clearTimeout(idle); + idle = setTimeout(() => { + wentIdle = true; + controller.abort(); + }, idleTimeoutMs); + }; + // Opened BEFORE the request, in append mode: once the answer is there, nothing is awaited + // until its body is being read. A body that is cut while nobody reads it loses what had + // arrived, and the continuation would start further back than it has to. + const file = await open(partial, 'a', 0o600); + // The file is deleted after it is closed: a download that cannot be the pinned archive. + let unlinkAfter = false; + const startOver = async (): Promise => { + await file.truncate(0); + offset = 0; + hash = createHash('sha256'); + }; + const damaged = (detail: string): DownloadAttempt => { + unlinkAfter = true; + return { kind: 'failed', result: { ok: false, code: 'runtime-damaged', detail } }; + }; + try { + armIdle(); + let res: Response; + try { + res = await deps.fetch(pin.url, { + ...(offset > 0 ? { headers: { range: `bytes=${offset}-` } } : {}), + signal: controller.signal, + redirect: 'follow', + }); + } catch (err) { + if (signal?.aborted) return { kind: 'cancelled' }; + if (err instanceof NotInjectedError) { + return { kind: 'failed', result: { ok: false, code: 'runtime-download-failed', detail: err.message } }; + } + return { kind: 'ended', detail: wentIdle ? 'The download server did not answer' : errorText(err) }; + } + + const discard = async (): Promise => { + await res.body?.cancel().catch(() => undefined); + }; + // The size the server says the whole archive has, when it says one. + let declaredTotal: number | undefined; + // The whole archive from its first byte although a range was asked for: the file starts over. + let restart = false; + if (res.status === 206) { + const range = /^bytes (\d+)-\d+\/(\d+)$/.exec(res.headers.get('content-range') ?? ''); + // A 206 that does not start where the file ends cannot be appended to it. + if (!range || Number(range[1]) !== offset) { + await discard(); + await startOver(); + return { kind: 'ended', detail: 'The server answered another range than the one asked for' }; + } + declaredTotal = Number(range[2]); + } else if (res.status === 200) { + restart = offset > 0; + const length = res.headers.get('content-length'); + if (length !== null && !res.headers.get('content-encoding') && /^\d+$/.test(length)) { + declaredTotal = Number(length); + } + } else if (res.status === 416) { + // The file on disk is not a prefix of what the server has. + await discard(); + await startOver(); + return { kind: 'ended', detail: 'The server refused the range (HTTP 416)' }; + } else { + await discard(); + return { kind: 'ended', detail: `The download server answered HTTP ${res.status}` }; + } + if (declaredTotal !== undefined && declaredTotal !== pin.bytes) { + await discard(); + return damaged(`The download is ${declaredTotal} bytes, the pinned archive has ${pin.bytes}`); + } + if (!res.body) return { kind: 'ended', detail: 'The download server sent no body' }; + + try { + if (restart) await startOver(); + for await (const chunk of res.body as unknown as AsyncIterable) { + armIdle(); + // Refused before a byte of it is written: a longer body can never have the pinned hash. + if (offset + chunk.length > pin.bytes) { + return damaged(`The download is longer than the ${pin.bytes} bytes of the pinned archive`); + } + await file.appendFile(chunk); + hash.update(chunk); + offset += chunk.length; + report(false); + } + } catch (err) { + // Whatever was written past the last hashed byte is cut off, so file and hash agree. + await file.truncate(offset).catch(() => undefined); + if (signal?.aborted) return { kind: 'cancelled' }; + if (isDiskFull(err)) { + return { kind: 'failed', result: { ok: false, code: 'disk-full', detail: errorText(err) } }; + } + return { + kind: 'ended', + detail: wentIdle + ? `The download delivered nothing for ${Math.max(1, Math.round(idleTimeoutMs / 1000))} s` + : errorText(err), + }; + } + return { kind: 'ended' }; + } finally { + clearTimeout(idle); + signal?.removeEventListener('abort', onAbort); + await file.close().catch(() => undefined); + // An empty file is no start of a download either. + if (unlinkAfter || offset === 0) await rm(partial, { force: true }).catch(() => undefined); + } + }; + + // A download that continues says at once how far it already is. + if (offset > 0) report(true); + let failures = 0; + while (offset < pin.bytes) { + if (signal?.aborted) return CANCELLED; + const outcome = await attempt(); + report(true); + if (outcome.kind === 'cancelled') return CANCELLED; + if (outcome.kind === 'failed') return outcome.result; + if (offset >= pin.bytes) break; + // Cut, or ended early: continue from where the file ends, a bounded number of times. + failures += 1; + if (failures > RUNTIME_DOWNLOAD_RETRIES) { + return { + ok: false, + code: 'runtime-download-failed', + detail: outcome.detail ?? `The download ended at ${offset} of ${pin.bytes} bytes`, + }; + } + await deps.sleep(DOWNLOAD_RETRY_DELAY_MS * 2 ** (failures - 1)); + } + if (signal?.aborted) return CANCELLED; + + // The proof, BEFORE anything is unpacked. + const digest = hash.digest('hex'); + if (digest !== pin.sha256.toLowerCase()) { + await rm(partial, { force: true }); + return { + ok: false, + code: 'runtime-damaged', + detail: `The sha256 of the download is ${digest}, the pinned archive has ${pin.sha256}`, + }; + } + return { ok: true }; +} + +/** + * Unpacks the verified archive into the temporary folder and renames it to the runtime folder. + * Never throws. + */ +async function unpackArchive( + paths: LocalModelPaths, + deps: RuntimeInstallDeps, + signal?: AbortSignal +): Promise<{ ok: true } | RuntimeInstallFailure> { + const fail = async (err: unknown): Promise => { + await rm(paths.unpackDir, { recursive: true, force: true }).catch(() => undefined); + if (signal?.aborted) return CANCELLED; + return { ok: false, code: isDiskFull(err) ? 'disk-full' : 'runtime-unpack-failed', detail: errorText(err) }; + }; + try { + await rm(paths.unpackDir, { recursive: true, force: true }); + await mkdir(paths.unpackDir, { recursive: true, mode: 0o700 }); + await deps.untar(paths.archivePartial, paths.unpackDir, signal); + if (signal?.aborted) return await fail(new Error('aborted')); + if (!(await isFile(join(paths.unpackDir, RUNTIME_BINARY_NAME)))) { + return await fail(new Error(`The archive holds no ${RUNTIME_BINARY_NAME} binary`)); + } + // A folder without its binary is a leftover of an unpack that was interrupted. + await rm(paths.runtimeDir, { recursive: true, force: true }); + await rename(paths.unpackDir, paths.runtimeDir); + await rm(paths.archivePartial, { force: true }); + return { ok: true }; + } catch (err) { + return fail(err); + } +} + +/** + * Makes sure the pinned runtime is unpacked: nothing to do when it is, else download (with + * Range resume), compare the hash, unpack, rename. Idempotent, so a resumed job calls it + * again. Never throws: every outcome is a `RuntimeInstallResult`. + */ +export async function ensureRuntimeInstalled(options: RuntimeInstallOptions = {}): Promise { + let phase: 'download' | 'unpack' = 'download'; + try { + const pin = options.pin ?? OLLAMA_RUNTIME_PIN; + const paths = options.paths ?? localModelPaths(pin.version); + const deps: RuntimeInstallDeps = { + fetch: options.deps?.fetch ?? defaultFetch, + untar: options.deps?.untar ?? untarWithSystemTar, + sleep: options.deps?.sleep ?? realSleep, + }; + if (await isFile(paths.binary)) return { ok: true, binaryPath: paths.binary, alreadyInstalled: true }; + if (options.signal?.aborted) return CANCELLED; + + await mkdir(paths.runtimeRoot, { recursive: true, mode: 0o700 }); + const downloaded = await downloadArchive(pin, paths, deps, options); + if (!downloaded.ok) return downloaded; + + phase = 'unpack'; + const unpacked = await unpackArchive(paths, deps, options.signal); + if (!unpacked.ok) return unpacked; + return { ok: true, binaryPath: paths.binary, alreadyInstalled: false }; + } catch (err) { + if (options.signal?.aborted) return CANCELLED; + if (isDiskFull(err)) return { ok: false, code: 'disk-full', detail: errorText(err) }; + return { + ok: false, + code: phase === 'download' ? 'runtime-download-failed' : 'runtime-unpack-failed', + detail: errorText(err), + }; + } +} + +/** What `removeLocalModelFiles()` deletes. */ +export interface RemoveLocalModelFilesOptions { + /** True: delete the model store too. False: the runtime, its archive and the private home only. */ + models: boolean; + /** Default `localModelPaths(OLLAMA_RUNTIME_PIN.version)`. */ + paths?: LocalModelPaths; +} + +/** What is to be done with one folder of the local model: nothing, unlink the link in its place, or delete its tree. */ +type RemovalStep = { target: string; how: 'absent' | 'unlink' | 'delete' }; + +/** + * Decides what removing `target` means, and refuses what must not be removed. A symlink in the + * place of a folder is unlinked and never followed; a real folder is deleted only when its + * real path is still inside the real root. Nothing is deleted here. + */ +async function planRemoval(target: string, root: string): Promise { + if (!isInsideLocalModelRoot(target, root) || resolvePath(target) === resolvePath(root)) { + throw new Error(`Refusing to delete ${target}: it is not a folder inside ${root}`); + } + let info; + try { + info = await lstat(target); + } catch (err) { + if (errnoOf(err) === 'ENOENT') return { target, how: 'absent' }; + throw err; + } + // The link goes, what it points at stays: that is somebody else's folder. + if (info.isSymbolicLink()) return { target, how: 'unlink' }; + const [realTarget, realRoot] = await Promise.all([realpath(target), realpath(root)]); + if (!isInsideLocalModelRoot(realTarget, realRoot) || realTarget === realRoot) { + throw new Error(`Refusing to delete ${target}: it resolves to ${realTarget}, outside ${realRoot}`); + } + return { target, how: 'delete' }; +} + +/** + * Whether `root` may be deleted in: it is absent (nothing to do) or a real folder. A `root` + * that is itself a symbolic link is REFUSED: every check below compares against the real + * path of the root, so a link in its place would make whatever it points at pass as ours. + */ +async function rootIsDeletable(root: string): Promise<'absent' | 'folder'> { + let info; + try { + info = await lstat(root); + } catch (err) { + if (errnoOf(err) === 'ENOENT') return 'absent'; + throw err; + } + if (info.isSymbolicLink()) { + throw new Error(`Refusing to delete in ${root}: it is a symbolic link, not the folder Codeman made`); + } + if (!info.isDirectory()) throw new Error(`Refusing to delete in ${root}: it is not a folder`); + return 'folder'; +} + +/** + * Deletes what this feature put on disk: every runtime folder and archive, the private home + * and, when asked, the models. Refuses any path that is not inside `LocalModelPaths.root`, + * refuses a `root` that is a symbolic link, follows no symlink out of it, and removes `root` + * itself only when it is empty afterwards. Every folder is judged BEFORE the first one is + * deleted, so a refusal deletes nothing. The state file and the coding agent are not its + * business. + */ +export async function removeLocalModelFiles(options: RemoveLocalModelFilesOptions): Promise { + const paths = options.paths ?? localModelPaths(OLLAMA_RUNTIME_PIN.version); + if ((await rootIsDeletable(paths.root)) === 'absent') return; + const targets = [paths.runtimeRoot, paths.homeDir, ...(options.models ? [paths.modelsDir] : [])]; + const plan: RemovalStep[] = []; + for (const target of targets) plan.push(await planRemoval(target, paths.root)); + for (const step of plan) { + if (step.how === 'unlink') await unlink(step.target); + // `rm` removes a symlink it meets inside the tree, never the tree behind it. + if (step.how === 'delete') await rm(step.target, { recursive: true, force: true }); + } + // Empty now (the models went too, or there never were any): the folder itself goes. + await rmdir(paths.root).catch(() => undefined); +} + +/** The marker the runtime puts into the name of a blob it has not finished downloading. */ +const PARTIAL_BLOB_MARK = '-partial'; + +/** + * Deletes what an interrupted download left in the model store: the files directly in + * `/blobs` whose name carries the runtime's partial marker (`sha256--partial` + * and its numbered parts). A model the runtime has no manifest for cannot be deleted through + * the runtime (it answers "not found" and touches no blob), so its partial blobs are removed + * here. Call it only while NO download is in flight: every partial file is then a dead one. + * + * Only regular files, only that one folder, only inside `LocalModelPaths.root`; a `root`, a + * model store or a blob folder that is a symbolic link is left alone. Resolves with the bytes + * that were freed. Never throws: a file that cannot be deleted stays for Remove. + */ +export async function removePartialModelBlobs(paths?: LocalModelPaths): Promise { + let freed = 0; + try { + const resolved = paths ?? localModelPaths(OLLAMA_RUNTIME_PIN.version); + const blobs = join(resolved.modelsDir, 'blobs'); + if (!isInsideLocalModelRoot(blobs, resolved.root)) return 0; + for (const folder of [resolved.root, resolved.modelsDir, blobs]) { + const info = await lstat(folder); + if (info.isSymbolicLink() || !info.isDirectory()) return 0; + } + for (const name of await readdir(blobs)) { + if (!name.includes(PARTIAL_BLOB_MARK)) continue; + const file = join(blobs, name); + try { + const info = await lstat(file); + if (!info.isFile()) continue; + await unlink(file); + freed += info.size; + } catch { + // Gone already, or not ours to delete: Remove with the models takes the rest. + } + } + } catch { + // No model store yet: nothing was left behind. + } + return freed; +} + +// ---------- Port ---------- + +/** Whether nothing holds `port` on 127.0.0.1, decided by binding it and letting go again. */ +export function isLoopbackPortFree(port: number): Promise { + return new Promise((resolve) => { + if (!Number.isInteger(port) || port < 1 || port > 65535) return resolve(false); + const probe = createServer(); + probe.unref(); + probe.once('error', () => resolve(false)); + probe.listen(port, '127.0.0.1', () => { + probe.close(() => resolve(true)); + }); + }); +} + +export interface ChooseRuntimePortOptions { + /** Default `RUNTIME_PORT_FIRST`. */ + first?: number; + /** Default `RUNTIME_PORT_SPAN`. */ + span?: number; + /** Ports to skip: the one that is in use when the person asked for another. */ + avoid?: readonly number[]; + /** Whether nothing holds `port` on 127.0.0.1, decided by BINDING it. A test injects its own. */ + isFree?: (port: number) => Promise; +} + +/** The first free loopback port of the span, or `undefined` when none is free. Chosen once, then stored by the caller. */ +export async function chooseRuntimePort(options: ChooseRuntimePortOptions = {}): Promise { + const first = options.first ?? RUNTIME_PORT_FIRST; + const span = options.span ?? RUNTIME_PORT_SPAN; + const isFree = options.isFree ?? isLoopbackPortFree; + const avoid = new Set(options.avoid ?? []); + for (let port = first; port < first + span; port++) { + if (avoid.has(port)) continue; + try { + if (await isFree(port)) return port; + } catch { + // A probe that fails says nothing about the port: it is not taken as free. + } + } + return undefined; +} + +// ---------- Supervisor ---------- + +/** What the supervisor knows about the runtime right now. Plain data. */ +export interface RuntimeStatus { + /** The child Codeman started (or adopted) is alive and answered `/api/version`. */ + running: boolean; + /** + * The pid of the child. Present from the moment it is SPAWNED, so also while `running` is + * still false: the caller stores it at once, and a server killed during a start leaves a + * pid that names the runtime as ours. + */ + pid?: number; + port?: number; + /** The context length the running child was started with. */ + contextLength?: number; + /** The device total parsed from the child's log, in bytes. Absent for an adopted orphan (its log is not ours to read). */ + measuredTotalBytes?: number; + /** How often the child was restarted since the last `start()`. */ + restarts: number; + /** True: the child exited `RUNTIME_MAX_RESTARTS` times in a row and is no longer restarted (`runtime-stopped`). */ + gaveUp: boolean; +} + +export interface RuntimeStartOptions { + /** The stored port. */ + port: number; + /** The context length in tokens (`OLLAMA_CONTEXT_LENGTH`). A running child with another value is replaced. */ + contextLength: number; + /** The pid the state file holds from an earlier server: decides whether a listener on `port` is our own orphan. */ + previousPid?: number; + /** + * Another unpacked runtime version than the pinned one: the version an installed model was + * installed with. Its binary is `runtimeBinaryForVersion()`, and it must answer this + * version. Absent: the pinned version. + */ + version?: string; +} + +/** What `stop()` may be told. */ +export interface RuntimeStopOptions { + /** + * The pid the state file holds. When no child of this process runs under it and its command + * line is a binary under our runtime folder, it is a runtime an earlier server left behind, + * and it is ended like our own child (SIGTERM, then SIGKILL). Any other pid is left alone. + */ + previousPid?: number; +} + +/** How a `start()` ended. */ +export type RuntimeStartResult = + | { + ok: true; + pid: number; + port: number; + /** What happened to a listener that was already on the port: there was none, or it was ours. */ + orphan: 'none' | 'adopted' | 'replaced'; + /** The device total, when the log line was read before `start()` returned. */ + measuredTotalBytes?: number; + } + | { + ok: false; + /** `port-in-use`: a listener that is not ours. `runtime-start-failed`: no answer in time, or the child exited. */ + code: Extract; + detail?: string; + }; + +/** What the supervisor calls. Each member defaults to the real thing; a test injects its own. */ +export interface RuntimeSupervisorDeps { + /** Starts the child. The default refuses under vitest. */ + spawn: (command: string, args: readonly string[], options: SpawnOptions) => ChildProcess; + /** Asks the runtime for `/api/version`. */ + fetch: typeof fetch; + /** The server's environment, the `source` of `runtimeEnv()`. */ + env: NodeJS.ProcessEnv; + /** `ps -o command= -p `, bounded. `undefined` when the pid is gone. */ + commandOfPid: (pid: number) => Promise; + /** Signals a pid that is not our child (an orphan). */ + kill: (pid: number, signal: NodeJS.Signals) => void; + /** Whether nothing holds `port` on 127.0.0.1, decided by binding it. */ + isPortFree: (port: number) => Promise; + sleep: (ms: number) => Promise; + /** Default `OLLAMA_RUNTIME_PIN`. */ + pin: LocalModelRuntimePin; + /** Default `localModelPaths(pin.version)`. */ + paths: LocalModelPaths; + /** Called after every change of `status()`: a start, an exit, a restart, the budget line. The caller stores pid and port. */ + onChange: (status: RuntimeStatus) => void; + /** How long a started runtime has to answer `/api/version`. Default 30 seconds. */ + startTimeoutMs?: number; + /** How often an adopted orphan is looked at. Default 5 seconds. */ + orphanPollMs?: number; + /** How long `start()` waits for the budget line once the runtime answers. Default 3 seconds. */ + budgetWaitMs?: number; +} + +/** The one supervisor of the runtime child. */ +export interface RuntimeSupervisor { + /** + * Makes sure the runtime runs on `port` with `contextLength`. Idempotent: a child that + * already runs with both is kept. Resolves once `/api/version` answers the pinned version + * and the budget line was read (or a few seconds passed without it). Never throws. + */ + start(options: RuntimeStartOptions): Promise; + /** + * SIGTERM to the child, SIGKILL after a grace period. Resolves once it is gone. Safe to call + * when nothing runs. With `previousPid` it also ends a runtime an earlier server left behind + * (see `RuntimeStopOptions`), so no caller deletes the runtime folder under a live process. + */ + stop(options?: RuntimeStopOptions): Promise; + status(): RuntimeStatus; +} + +const defaultSpawn: RuntimeSupervisorDeps['spawn'] = (command, args, options) => { + if (UNDER_VITEST) throw new NotInjectedError('spawn'); + return nodeSpawn(command, [...args], options); +}; + +/** `ps -o command= -p `: the command line of a pid, `undefined` when it is gone. Spawns nothing under vitest. */ +function defaultCommandOfPid(pid: number): Promise { + return new Promise((resolve) => { + if (UNDER_VITEST || !Number.isInteger(pid) || pid <= 1) return resolve(undefined); + execFile('/bin/ps', ['-o', 'command=', '-p', String(pid)], { timeout: PS_TIMEOUT_MS }, (err, stdout) => { + const line = typeof stdout === 'string' ? stdout.trim() : ''; + resolve(err || line === '' ? undefined : line); + }); + }); +} + +const defaultKill: RuntimeSupervisorDeps['kill'] = (pid, signal) => { + if (UNDER_VITEST || !Number.isInteger(pid) || pid <= 1) return; + try { + process.kill(pid, signal); + } catch { + // Gone already. + } +}; + +function hasExited(child: ChildProcess): boolean { + return child.exitCode !== null || child.signalCode !== null; +} + +/** How one launch of the child ended. */ +type LaunchResult = { ok: true; pid: number; measuredTotalBytes?: number } | { ok: false; detail: string }; + +/** A supervisor. The server keeps ONE (see `getLocalModelService()` in install-job.ts). */ +export function createRuntimeSupervisor(overrides: Partial = {}): RuntimeSupervisor { + const pin = overrides.pin ?? OLLAMA_RUNTIME_PIN; + const deps = { + spawn: overrides.spawn ?? defaultSpawn, + fetch: overrides.fetch ?? defaultFetch, + env: overrides.env ?? process.env, + commandOfPid: overrides.commandOfPid ?? defaultCommandOfPid, + kill: overrides.kill ?? defaultKill, + isPortFree: overrides.isPortFree ?? isLoopbackPortFree, + sleep: overrides.sleep ?? realSleep, + onChange: overrides.onChange ?? ((): void => undefined), + startTimeoutMs: overrides.startTimeoutMs ?? START_TIMEOUT_MS, + orphanPollMs: overrides.orphanPollMs ?? ORPHAN_POLL_MS, + budgetWaitMs: overrides.budgetWaitMs ?? BUDGET_WAIT_MS, + }; + // Resolved on first use: `localModelPaths()` creates the data dir, which making a supervisor must not. + let resolvedPaths = overrides.paths; + const paths = (): LocalModelPaths => (resolvedPaths ??= localModelPaths(pin.version)); + + /** Our child, once it answered. */ + let child: ChildProcess | undefined; + /** A child that was spawned and has not answered yet. */ + let starting: ChildProcess | undefined; + /** An orphan of an earlier server that was adopted: not our child, so it is signalled by pid. */ + let adoptedPid: number | undefined; + let orphanTimer: NodeJS.Timeout | undefined; + /** What the last `start()` asked for: what a restart starts again. */ + let wanted: { port: number; contextLength: number; version: string } | undefined; + /** The version the running child was started as. */ + let runningVersion: string | undefined; + /** Bumped by every `start()` and `stop()`: work of an older generation stops when it notices. */ + let generation = 0; + /** The restart loop that is running, if one is. */ + let restarting: Promise | undefined; + /** `start()` and `stop()` run one after the other. */ + let queue: Promise = Promise.resolve(); + let state: RuntimeStatus = { running: false, restarts: 0, gaveUp: false }; + + function setState(patch: Partial): void { + const next: RuntimeStatus = { ...state, ...patch }; + for (const key of Object.keys(next) as (keyof RuntimeStatus)[]) { + if (next[key] === undefined) delete next[key]; + } + const changed = + Object.keys(next).length !== Object.keys(state).length || + (Object.keys(next) as (keyof RuntimeStatus)[]).some((key) => next[key] !== state[key]); + state = next; + if (!changed) return; + try { + deps.onChange({ ...state }); + } catch { + // A caller that throws must not stop the supervisor. + } + } + + function serialize(work: () => Promise): Promise { + const run = queue.then(work, work); + queue = run.catch(() => undefined); + return run; + } + + const isOwn = (command: string | undefined): boolean => + command !== undefined && isOwnRuntimeCommand(command, paths().runtimeRoot); + + /** SIGTERM, then SIGKILL after the grace period. Resolves once the child is gone. */ + async function stopChild(proc: ChildProcess): Promise { + // A child that never got a pid was never started: there is no exit to wait for. + if (hasExited(proc) || proc.pid === undefined) return; + const gone = new Promise((resolve) => proc.once('exit', () => resolve())); + try { + if (!proc.kill('SIGTERM')) return; + } catch { + return; + } + const force = setTimeout(() => { + try { + proc.kill('SIGKILL'); + } catch { + // Gone already. + } + }, STOP_GRACE_MS); + force.unref(); + await Promise.race([gone, pause(STOP_GRACE_MS + STOP_KILL_WAIT_MS)]); + clearTimeout(force); + } + + /** The same for a pid that is not our child: signalled through `deps.kill`, watched through `ps`. */ + async function stopOrphan(pid: number): Promise { + const goneWithin = async (ms: number): Promise => { + const until = Date.now() + ms; + for (;;) { + if (!isOwn(await deps.commandOfPid(pid))) return true; + if (Date.now() >= until) return false; + await pause(100); + } + }; + deps.kill(pid, 'SIGTERM'); + if (await goneWithin(STOP_GRACE_MS)) return; + deps.kill(pid, 'SIGKILL'); + await goneWithin(STOP_KILL_WAIT_MS); + } + + /** Stops whatever runs now: our child, one that is starting, an adopted orphan. */ + async function stopCurrent(): Promise { + clearInterval(orphanTimer); + orphanTimer = undefined; + const procs = [child, starting].filter((proc): proc is ChildProcess => proc !== undefined); + // Forgotten first, so that its exit is not taken for a crash. + child = undefined; + starting = undefined; + const orphan = adoptedPid; + adoptedPid = undefined; + await Promise.all(procs.map((proc) => stopChild(proc))); + if (orphan !== undefined) await stopOrphan(orphan); + } + + /** The runtime exited by itself: start it again, a bounded number of times. */ + function restart(): void { + if (restarting) return; + const gen = generation; + const loop = async (): Promise => { + while (gen === generation && wanted) { + if (state.restarts >= RUNTIME_MAX_RESTARTS) { + setState({ gaveUp: true }); + return; + } + setState({ restarts: state.restarts + 1 }); + await deps.sleep(RESTART_DELAY_MS); + if (gen !== generation || !wanted) return; + const launched = await launch(wanted.port, wanted.contextLength, wanted.version, gen); + // Running again: a later exit comes back here through the child's exit handler. + if (launched.ok) return; + } + }; + restarting = loop() + .catch(() => undefined) + .finally(() => { + restarting = undefined; + }); + } + + /** Spawns the runtime and waits until it answers. The child becomes `child` only then. */ + async function launch(port: number, contextLength: number, version: string, gen: number): Promise { + const folders = paths(); + let spawned: ChildProcess; + try { + // The pinned build, or the one an installed model was installed with. + const binary = version === pin.version ? folders.binary : runtimeBinaryForVersion(folders, version); + await mkdir(folders.homeDir, { recursive: true, mode: 0o700 }); + await mkdir(folders.modelsDir, { recursive: true, mode: 0o700 }); + // Asked to stop in the meantime: nothing is started only to be stopped again. + if (gen !== generation) return { ok: false, detail: 'The start was replaced by a newer request' }; + spawned = deps.spawn(binary, ['serve'], { + env: runtimeEnv({ + source: deps.env, + home: folders.homeDir, + modelsDir: folders.modelsDir, + port, + contextLength, + }), + cwd: folders.homeDir, + stdio: ['ignore', 'pipe', 'pipe'], + // A child of the server, in its process group: it is not left behind by a normal exit. + detached: false, + }); + } catch (err) { + return { ok: false, detail: errorText(err) }; + } + starting = spawned; + // Said at once, before the child answers: the caller stores the pid, so a server that is + // killed during this start still leaves a pid that names the runtime as ours. + if (spawned.pid !== undefined) setState({ pid: spawned.pid }); + + // The child's output: the budget line, and the last lines for the detail of a failure. + // It is read for as long as the child lives, so a full pipe never blocks it. + const tail: string[] = []; + let measured: number | undefined; + let budgetSeen: () => void = () => undefined; + const budget = new Promise((resolve) => { + budgetSeen = resolve; + }); + const onLine = (text: string): void => { + const line = text.length > LOG_LINE_MAX_CHARS ? text.slice(0, LOG_LINE_MAX_CHARS) : text; + if (line.trim() === '') return; + tail.push(line); + if (tail.length > LOG_TAIL_LINES) tail.shift(); + const total = parseInferenceComputeTotal(line); + if (total === undefined) return; + measured = total; + if (child === spawned) setState({ measuredTotalBytes: total }); + budgetSeen(); + }; + for (const stream of [spawned.stdout, spawned.stderr]) { + if (!stream) continue; + let pending = ''; + stream.setEncoding('utf8'); + stream.on('data', (chunk: string) => { + pending += chunk; + let breakAt = pending.indexOf('\n'); + while (breakAt >= 0) { + onLine(pending.slice(0, breakAt)); + pending = pending.slice(breakAt + 1); + breakAt = pending.indexOf('\n'); + } + // A line without an end is cut, so the buffer cannot grow without bound. + if (pending.length > LOG_LINE_MAX_CHARS) pending = pending.slice(0, LOG_LINE_MAX_CHARS); + }); + stream.on('error', () => undefined); + } + + let ended: string | undefined; + let goneNow: () => void = () => undefined; + const gone = new Promise((resolve) => { + goneNow = resolve; + }); + // The last resort when the server exits without its shutdown: the child must not outlive it. + const onProcessExit = (): void => { + try { + spawned.kill('SIGTERM'); + } catch { + // Gone already. + } + }; + process.once('exit', onProcessExit); + spawned.once('error', (err) => { + // A spawn that failed emits no exit. + if (spawned.pid === undefined) process.off('exit', onProcessExit); + ended ??= `The runtime could not be started: ${errorText(err)}`; + goneNow(); + }); + spawned.once('exit', (code, signal) => { + process.off('exit', onProcessExit); + ended ??= signal ? `The runtime was ended by ${signal}` : `The runtime exited with code ${String(code)}`; + goneNow(); + if (starting === spawned) starting = undefined; + if (child !== spawned) return; + // Our running child exited by itself. + child = undefined; + setState({ running: false, pid: undefined }); + restart(); + }); + + const fail = async (reason: string): Promise => { + if (starting === spawned) starting = undefined; + await stopChild(spawned); + // The pid that was announced at the spawn is no runtime any more. + if (state.pid === spawned.pid && child !== spawned) setState({ pid: undefined }); + const log = tail.join('\n'); + return { ok: false, detail: clip(log === '' ? reason : `${reason}\n${log}`) }; + }; + + const target = { baseUrl: `http://127.0.0.1:${port}`, fetch: deps.fetch }; + const deadline = Date.now() + deps.startTimeoutMs; + for (;;) { + if (ended !== undefined) return fail(ended); + if (gen !== generation) return fail('The start was replaced by a newer request'); + const answered = await getRuntimeVersion(target); + if (ended !== undefined) return fail(ended); + if (answered !== undefined) { + if (answered !== version) return fail(`The runtime answers as version ${answered}, not ${version}`); + break; + } + if (Date.now() >= deadline) { + return fail(`The runtime did not answer within ${Math.round(deps.startTimeoutMs / 1000)} seconds`); + } + await deps.sleep(START_POLL_MS); + } + + // The budget line follows the first answer within a moment. An older runtime may not print it. + if (measured === undefined) await Promise.race([budget, gone, pause(deps.budgetWaitMs)]); + if (ended !== undefined) return fail(ended); + if (gen !== generation) return fail('The start was replaced by a newer request'); + + starting = undefined; + child = spawned; + runningVersion = version; + const pid = spawned.pid ?? 0; + setState({ + running: true, + pid, + port, + contextLength, + measuredTotalBytes: measured, + gaveUp: false, + }); + return { ok: true, pid, ...(measured !== undefined ? { measuredTotalBytes: measured } : {}) }; + } + + /** Keeps an orphan of an earlier server: it is watched by pid, because its exit is no event of ours. */ + function adopt(pid: number, port: number, contextLength: number, version: string): void { + adoptedPid = pid; + runningVersion = version; + setState({ running: true, pid, port, contextLength, measuredTotalBytes: undefined, gaveUp: false }); + let looking = false; + orphanTimer = setInterval(() => { + if (looking || adoptedPid !== pid) return; + looking = true; + void deps + .commandOfPid(pid) + .catch(() => undefined) + .then((command) => { + looking = false; + if (adoptedPid !== pid || isOwn(command)) return; + // The orphan is gone: from here on the runtime is a child of our own. + clearInterval(orphanTimer); + orphanTimer = undefined; + adoptedPid = undefined; + setState({ running: false, pid: undefined }); + restart(); + }); + }, deps.orphanPollMs); + orphanTimer.unref(); + } + + /** Whether keeping this orphan keeps something: a model is loaded, and at the wanted context. */ + async function worthAdopting( + target: { baseUrl: string; fetch: typeof fetch }, + contextLength: number + ): Promise { + const loaded = await listRunning(target); + return loaded.length > 0 && loaded.every((model) => model.contextLength === contextLength); + } + + /** Waits for the port to come free after an orphan was told to go. */ + async function portFreeWithin(port: number, ms: number): Promise { + const until = Date.now() + ms; + for (;;) { + if (await deps.isPortFree(port).catch(() => false)) return true; + if (Date.now() >= until) return false; + await pause(100); + } + } + + async function doStart(options: RuntimeStartOptions): Promise { + const { port, contextLength, previousPid } = options; + const version = options.version ?? pin.version; + if (!Number.isInteger(port) || port < 1 || port > 65535) { + return { ok: false, code: 'runtime-start-failed', detail: `Not a port: ${String(port)}` }; + } + if (!Number.isInteger(contextLength) || contextLength < 1) { + return { ok: false, code: 'runtime-start-failed', detail: `Not a context length: ${String(contextLength)}` }; + } + // Already running as asked: nothing to do. + const alive = (child !== undefined && !hasExited(child)) || adoptedPid !== undefined; + if ( + alive && + state.running && + state.port === port && + state.contextLength === contextLength && + runningVersion === version + ) { + return { + ok: true, + pid: state.pid ?? 0, + port, + orphan: 'none', + ...(state.measuredTotalBytes !== undefined ? { measuredTotalBytes: state.measuredTotalBytes } : {}), + }; + } + + generation += 1; + const gen = generation; + await restarting; + await stopCurrent(); + wanted = { port, contextLength, version }; + runningVersion = undefined; + setState({ + running: false, + pid: undefined, + port: undefined, + contextLength: undefined, + measuredTotalBytes: undefined, + restarts: 0, + gaveUp: false, + }); + + // Who holds the port, if anybody does. + const target = { baseUrl: `http://127.0.0.1:${port}`, fetch: deps.fetch }; + const ownOrphan = + previousPid !== undefined && + Number.isInteger(previousPid) && + isOwn(await deps.commandOfPid(previousPid).catch(() => undefined)) + ? previousPid + : undefined; + const answering = await getRuntimeVersion(target); + let orphan: 'none' | 'adopted' | 'replaced' = 'none'; + if (ownOrphan !== undefined) { + if (answering === version && (await worthAdopting(target, contextLength))) { + adopt(ownOrphan, port, contextLength, version); + return { ok: true, pid: ownOrphan, port, orphan: 'adopted' }; + } + await stopOrphan(ownOrphan); + orphan = 'replaced'; + } else if (answering !== undefined) { + return { + ok: false, + code: 'port-in-use', + detail: `A model runtime that Codeman did not start answers on port ${port} (version ${answering})`, + }; + } + if (!(await portFreeWithin(port, orphan === 'replaced' ? STOP_GRACE_MS : 0))) { + return { ok: false, code: 'port-in-use', detail: `Another program holds port ${port}` }; + } + + const launched = await launch(port, contextLength, version, gen); + if (!launched.ok) return { ok: false, code: 'runtime-start-failed', detail: launched.detail }; + return { + ok: true, + pid: launched.pid, + port, + orphan, + ...(launched.measuredTotalBytes !== undefined ? { measuredTotalBytes: launched.measuredTotalBytes } : {}), + }; + } + + async function doStop(options: RuntimeStopOptions = {}): Promise { + generation += 1; + wanted = undefined; + await restarting; + await stopCurrent(); + runningVersion = undefined; + // A runtime an earlier server left behind: not a child of this process, known only by the + // pid the state file kept. Ended when (and only when) its command line is our own binary. + const { previousPid } = options; + if (previousPid !== undefined && Number.isInteger(previousPid) && previousPid > 1) { + if (isOwn(await deps.commandOfPid(previousPid).catch(() => undefined))) await stopOrphan(previousPid); + } + setState({ + running: false, + pid: undefined, + port: undefined, + contextLength: undefined, + measuredTotalBytes: undefined, + restarts: 0, + gaveUp: false, + }); + } + + return { + start: (options) => + serialize(() => doStart(options)).catch( + (err): RuntimeStartResult => ({ ok: false, code: 'runtime-start-failed', detail: errorText(err) }) + ), + stop: (options) => { + // At once, not when its turn comes: a start that is still waiting for the runtime gives up. + generation += 1; + return serialize(() => doStop(options)).catch(() => undefined); + }, + status: () => ({ ...state }), + }; +} diff --git a/src/local-model/state-store.ts b/src/local-model/state-store.ts new file mode 100644 index 00000000..434a686e --- /dev/null +++ b/src/local-model/state-store.ts @@ -0,0 +1,276 @@ +/** + * @fileoverview The ONE reader and writer of `local-model.json`, the state file of the local + * model install (`LocalModelStateFile`). + * + * Why one module: the install job writes the state, the step and the digest, and the caller + * of the runtime supervisor writes the pid and the port. Two writers with their own temp file + * and rename would lose each other's change, which is the fault `mutateCustomModelHosts()` + * exists to end for the endpoint store. So every write goes through `mutateLocalModelState()`. + * + * The shape of `mutateSettingsFile()` (src/web/settings-writer.ts): + * - Serialized: mutations run one at a time on one promise chain. + * - Atomic: each write goes to a temp file with a random name beside the target and is + * renamed over it. The file is mode 0600. + * - Reading is TOTAL, unlike the settings writer: the file holds no setting of the user, only + * what this feature recorded, so a missing file, one that does not parse or one with + * another `schema` reads as `IDLE_LOCAL_MODEL_STATE` and the next write replaces it. + * - The path is `localModelPaths().stateFile`. The file itself holds no path. + */ + +import { randomUUID } from 'node:crypto'; +import fs from 'node:fs/promises'; +import { dirname } from 'node:path'; +import { + LOCAL_MODEL_ERROR_LABELS, + LOCAL_MODEL_INSTALL_STEPS, + type LocalModelErrorCode, + type LocalModelInstallState, + type LocalModelInstallStep, + type LocalModelStateFile, +} from '../types.js'; +import { OLLAMA_RUNTIME_PIN } from './runtime.js'; +import { localModelPaths } from './paths.js'; + +/** The state when nothing was ever installed. What an unreadable file reads as. */ +export const IDLE_LOCAL_MODEL_STATE: LocalModelStateFile = Object.freeze({ schema: 1, state: 'idle' }); + +const STATES: readonly LocalModelInstallState[] = ['idle', 'installing', 'paused', 'failed', 'installed']; + +// ---------- Reading one member at a time ---------- + +const isRecord = (value: unknown): value is Record => + typeof value === 'object' && value !== null && !Array.isArray(value); + +const text = (value: unknown, max = 4_096): string | undefined => + typeof value === 'string' && value !== '' && value.length <= max ? value : undefined; + +/** A whole number of bytes, tokens, a port or a pid: finite, not negative, and an integer. */ +const count = (value: unknown): number | undefined => + typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 ? value : undefined; + +const step = (value: unknown): LocalModelInstallStep | undefined => + LOCAL_MODEL_INSTALL_STEPS.find((candidate) => candidate === value); + +function parseSteps(value: unknown): LocalModelInstallStep[] | undefined { + if (!Array.isArray(value)) return undefined; + const steps: LocalModelInstallStep[] = []; + for (const item of value) { + const parsed = step(item); + // A list with an unknown or a repeated step is not a list this build wrote. + if (!parsed || steps.includes(parsed)) return undefined; + steps.push(parsed); + } + return steps.length > 0 ? steps : undefined; +} + +function parseVars(value: unknown): Record | undefined { + if (!isRecord(value)) return undefined; + const vars: Record = {}; + for (const [key, item] of Object.entries(value)) { + if (typeof item === 'string' && item.length <= 4_096) vars[key] = item; + } + return Object.keys(vars).length > 0 ? vars : undefined; +} + +function parseError(value: unknown): LocalModelStateFile['error'] { + if (!isRecord(value)) return undefined; + const code = text(value.code); + if (!code || !Object.prototype.hasOwnProperty.call(LOCAL_MODEL_ERROR_LABELS, code)) return undefined; + const vars = parseVars(value.vars); + const detail = text(value.detail, 16_384); + const nextModelId = text(value.nextModelId, 200); + return { + code: code as LocalModelErrorCode, + ...(vars ? { vars } : {}), + ...(detail ? { detail } : {}), + ...(nextModelId ? { nextModelId } : {}), + }; +} + +function parseModel(value: unknown): LocalModelStateFile['model'] { + if (!isRecord(value)) return undefined; + const id = text(value.id, 200); + const label = text(value.label, 200); + const context = count(value.context); + if (!id || !label || !context) return undefined; + const pullBytes = count(value.pullBytes); + const digest = text(value.digest, 200); + return { + id, + label, + context, + ...(pullBytes !== undefined ? { pullBytes } : {}), + ...(digest ? { digest } : {}), + }; +} + +function parseCli(value: unknown): LocalModelStateFile['cli'] { + if (!isRecord(value)) return undefined; + const id = text(value.id, 200); + const label = text(value.label, 200); + return id && label ? { id, label } : undefined; +} + +function parseRuntime(value: unknown): LocalModelStateFile['runtime'] { + if (!isRecord(value)) return undefined; + const version = text(value.version, 64); + const port = count(value.port); + if (!version || !port || port > 65_535) return undefined; + const pid = count(value.pid); + const measuredTotalBytes = count(value.measuredTotalBytes); + return { + version, + port, + ...(pid ? { pid } : {}), + // Anything but an explicit `false` reads as enabled: only Stop switches the runtime off. + enabled: value.enabled !== false, + ...(measuredTotalBytes ? { measuredTotalBytes } : {}), + }; +} + +function parseProgress(value: unknown): LocalModelStateFile['progress'] { + if (!isRecord(value)) return undefined; + const completedBytes = count(value.completedBytes); + const totalBytes = count(value.totalBytes); + return completedBytes !== undefined && totalBytes !== undefined ? { completedBytes, totalBytes } : undefined; +} + +function parseLoaded(value: unknown): LocalModelStateFile['loaded'] { + if (!isRecord(value)) return undefined; + const context = count(value.context); + const sizeBytes = count(value.sizeBytes); + const sizeVramBytes = count(value.sizeVramBytes); + return context !== undefined && sizeBytes !== undefined && sizeVramBytes !== undefined + ? { context, sizeBytes, sizeVramBytes } + : undefined; +} + +function parseConnection(value: unknown): LocalModelStateFile['connection'] { + if (!isRecord(value)) return undefined; + const endpointId = text(value.endpointId, 200); + return endpointId && typeof value.flagWasEnabled === 'boolean' + ? { endpointId, flagWasEnabled: value.flagWasEnabled } + : undefined; +} + +/** + * The state a file's text stands for. Pure and total: text that is not a JSON object of + * `schema: 1` with a known `state` is `IDLE_LOCAL_MODEL_STATE`. Members it does not know are + * dropped, and a member of the wrong type is dropped rather than trusted. + * + * A state that needs a member it does not have is not trusted either: a job (`installing`, + * `paused`, `failed`) without its model, its coding agent or its step, a `failed` job without + * its error, and an `installed` state without its model or its runtime all read as idle. + * Nothing is deleted by that: what lies in the folder stays until a Remove or a new install. + */ +export function parseLocalModelState(raw: string): LocalModelStateFile { + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + return IDLE_LOCAL_MODEL_STATE; + } + if (!isRecord(parsed) || parsed.schema !== 1) return IDLE_LOCAL_MODEL_STATE; + const state = STATES.find((candidate) => candidate === parsed.state); + if (!state || state === 'idle') return IDLE_LOCAL_MODEL_STATE; + + const model = parseModel(parsed.model); + const cli = parseCli(parsed.cli); + const runtime = parseRuntime(parsed.runtime); + const connection = parseConnection(parsed.connection); + const updatedAt = text(parsed.updatedAt, 64); + const replacedModelId = text(parsed.replacedModelId, 200); + if (!model || !cli) return IDLE_LOCAL_MODEL_STATE; + + if (state === 'installed') { + if (!runtime) return IDLE_LOCAL_MODEL_STATE; + const loaded = parseLoaded(parsed.loaded); + return { + schema: 1, + state, + model, + cli, + runtime, + ...(loaded ? { loaded } : {}), + ...(connection ? { connection } : {}), + ...(updatedAt ? { updatedAt } : {}), + }; + } + + const steps = parseSteps(parsed.steps); + const current = step(parsed.step); + if (!steps || !current || !steps.includes(current)) return IDLE_LOCAL_MODEL_STATE; + const error = parseError(parsed.error); + if (state === 'failed' && !error) return IDLE_LOCAL_MODEL_STATE; + const progress = parseProgress(parsed.progress); + const loaded = parseLoaded(parsed.loaded); + return { + schema: 1, + state, + step: current, + steps, + ...(state === 'failed' && error ? { error } : {}), + model, + cli, + ...(runtime ? { runtime } : {}), + ...(progress ? { progress } : {}), + ...(loaded ? { loaded } : {}), + ...(connection ? { connection } : {}), + ...(replacedModelId ? { replacedModelId } : {}), + ...(updatedAt ? { updatedAt } : {}), + }; +} + +/** Where the state file is. Built from `dataPath()` on every call, so a test's own data dir is honoured. */ +function stateFilePath(): string { + return localModelPaths(OLLAMA_RUNTIME_PIN.version).stateFile; +} + +async function readStateFrom(path: string): Promise { + try { + return parseLocalModelState(await fs.readFile(path, 'utf-8')); + } catch { + return IDLE_LOCAL_MODEL_STATE; + } +} + +/** The stored state. Never throws: a file that cannot be read is `IDLE_LOCAL_MODEL_STATE`. */ +export async function readLocalModelState(): Promise { + return readStateFrom(stateFilePath()); +} + +async function writeStateFile(path: string, state: LocalModelStateFile): Promise { + await fs.mkdir(dirname(path), { recursive: true }); + const tmp = `${path}.${process.pid}.${randomUUID()}.tmp`; + try { + await fs.writeFile(tmp, JSON.stringify(state, null, 2), { mode: 0o600 }); + await fs.rename(tmp, path); + } catch (err) { + await fs.rm(tmp, { force: true }).catch(() => {}); + throw err; + } +} + +let mutationChain: Promise = Promise.resolve(); + +/** + * Runs one change to the state file. `fn` receives the current state and returns the state to + * write, or `undefined` to write nothing. Resolves with the state before and after the change + * (`after` is `undefined` when nothing was written). `updatedAt` is stamped here. A failed + * write rejects its own caller only; the chain keeps going. + */ +export function mutateLocalModelState( + fn: (current: LocalModelStateFile) => LocalModelStateFile | undefined +): Promise<{ before: LocalModelStateFile; after: LocalModelStateFile | undefined }> { + const run = mutationChain.then(async () => { + const path = stateFilePath(); + const before = await readStateFrom(path); + const next = fn(before); + if (!next) return { before, after: undefined }; + const after: LocalModelStateFile = { ...next, updatedAt: new Date().toISOString() }; + await writeStateFile(path, after); + return { before, after }; + }); + mutationChain = run.catch(() => {}); + return run; +} diff --git a/src/reboot-restore.ts b/src/reboot-restore.ts index fe7bff40..b1aec5f4 100644 --- a/src/reboot-restore.ts +++ b/src/reboot-restore.ts @@ -134,6 +134,8 @@ export interface RebootRestoreRejection { | 'workspace-forbidden' | 'already-live' | 'capacity-reached' + /** The session ran on the model Codeman runs on this Mac, and that model cannot be started on right now. */ + | 'local-model-not-ready' | 'rebuild-failed'; } diff --git a/src/session.ts b/src/session.ts index 4b84fb5d..9d17f767 100644 --- a/src/session.ts +++ b/src/session.ts @@ -510,6 +510,18 @@ export class GuardedInputBusyError extends Error { } } +/** + * The session's launch guard refused the start of a new agent process + * (`Session.setLaunchGuard()`). Its message is the sentence for the user. A typed error, so a + * start that was refused is told apart from a tmux failure and never falls back to a direct PTY. + */ +export class LaunchRefusedError extends Error { + constructor(message: string) { + super(message); + this.name = 'LaunchRefusedError'; + } +} + /** * Whether a programmatic write leaves the keep-alive turn running: the nudge itself, or a bare * Enter (which submits nothing new). Anything else carrying an Enter starts a turn of its own. @@ -880,6 +892,12 @@ export class Session extends EventEmitter { // config dir) without guessing what it once wrote. Persisted disk-only (`__customModel`). private _customModel: CustomModelBookkeeping | undefined; + // Asked right before a NEW agent process is started for this session (a fresh pane, the + // respawn of a dead one, a restart in place), never for an attach to an agent that runs. + // Resolves with the sentence that refuses the start, or undefined. Set by the web layer + // (`WebServer.setupSessionListeners()`); this file does not know who asks or why. + private _launchGuard: (() => Promise) | null = null; + // Env keys a retired custom-model selection injected that the NEXT respawn must // `tmux setenv -u`. Deleting a key from `_envOverrides` alone does nothing to the // tmux session, which keeps every `setenv` and hands it to `respawn-pane`, so the @@ -2063,6 +2081,22 @@ export class Session extends EventEmitter { } } + /** + * Sets what is asked before a new agent process is started for this session (see + * `_launchGuard`). The first user is the model Codeman runs on this Mac: an agent is not + * started on it while its runtime is down or its model changed. `null` removes it. + */ + setLaunchGuard(guard: (() => Promise) | null): void { + this._launchGuard = guard; + } + + /** Throws the guard's sentence (`LaunchRefusedError`) when a new agent process must not be started now. */ + private async _assertLaunchAllowed(): Promise { + if (!this._launchGuard) return; + const refusal = await this._launchGuard(); + if (refusal) throw new LaunchRefusedError(refusal); + } + // Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md) — public-safe // subset only (never envKeys/configDir/launchModel, the bookkeeping for setCustomModel). get customModel(): CustomModelSelection | undefined { @@ -2635,6 +2669,8 @@ export class Session extends EventEmitter { let respawnedResumeId: string | undefined; if (this._muxSession && mux.isPaneDead(this._muxSession.muxName)) { console.log('[Session] Dead pane detected, respawning:', this._muxSession.muxName); + // A new agent process: asked first (see `_launchGuard`). A refusal leaves the pane as it is. + await this._assertLaunchAllowed(); // Confirmed dead — safe to resolve/pin now (see `_pinOmpRespawnId()`). // `options.respawnPaneOptions` was built eagerly before this dead-pane // check ran, so it still carries the pre-pin ompConfig; rebuild it. @@ -2689,6 +2725,8 @@ export class Session extends EventEmitter { this._resumeSessionId = pinned; } } + // A new agent process: asked first (see `_launchGuard`). + await this._assertLaunchAllowed(); this._muxSession = await mux.createSession(options.createSessionOptions); console.log('[Session] Created mux session:', this._muxSession.muxName); // No extra sleep — createSession() already waits for tmux readiness @@ -2808,6 +2846,9 @@ export class Session extends EventEmitter { return false; } + // A new agent process: asked first (see `_launchGuard`). A refusal rejects, and the pane + // keeps what runs in it. + await this._assertLaunchAllowed(); this._pinOmpRespawnId(); const newPid = await mux.respawnPane(await this._buildRespawnPaneOptionsWithResumePin()); if (!newPid) { @@ -3461,6 +3502,9 @@ export class Session extends EventEmitter { this._armPaneSettle(true); } } catch (err) { + // A refused launch is an answer, not a tmux failure: it must not fall through to + // the direct PTY below, which would start the very process that was refused. + if (err instanceof LaunchRefusedError) throw err; if (this._adopt) this._failAdoptedStart(err); console.error('[Session] Failed to create mux session, falling back to direct PTY:', err); this._useMux = false; @@ -3482,6 +3526,8 @@ export class Session extends EventEmitter { throw new Error(`${getModeLabel(this.mode)} sessions require tmux. Direct PTY fallback is not supported.`); } if (this._adopt) this._failAdoptedStart(new Error('tmux is not available')); + // A new agent process on this path too: asked first (see `_launchGuard`). + await this._assertLaunchAllowed(); try { // Pass --session-id to use the SAME ID as the Codeman session // This ensures subagents can be directly matched to the correct tab. @@ -4485,6 +4531,8 @@ export class Session extends EventEmitter { }, 100); } } catch (err) { + // As in startInteractive(): a refused launch never falls back to a direct PTY. + if (err instanceof LaunchRefusedError) throw err; if (this._adopt) this._failAdoptedStart(err); console.error('[Session] Failed to create mux session, falling back to direct PTY:', err); this._useMux = false; diff --git a/src/types/index.ts b/src/types/index.ts index 592b28e2..06dda8aa 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -29,7 +29,8 @@ * | plan | PlanItem, PlanTaskStatus, TddPhase | In-memory → `GET /api/sessions/:id/plan/tasks` | * | orchestrator | OrchestratorState, OrchestratorPlan, OrchestratorConfig, OrchestratorPersistState | `~/.codeman/state.json` → `GET /api/orchestrator/status` | * | mcp-sync | McpSyncResult, McpSyncTargetResult | Other CLIs' own config files → `GET`/`POST /api/mcp-sync` | - * | local-model | HardwareFacts, LocalModelCatalog, Candidate, Recommendation, LocalModelStatus, LOCAL_MODEL_LABELS | In-memory only (nothing persisted) → `GET /api/local-model` | + * | local-model | HardwareFacts, LocalModelCatalog, Candidate, Recommendation, RecommendOptions, LOCAL_MODEL_LABELS | In-memory only (nothing persisted) → `GET /api/local-model` | + * | local-model-install | LocalModelStatus, LocalModelInstallView, LocalModelStateFile, LocalModelRuntimePin, LOCAL_MODEL_INSTALL_LABELS | `~/.codeman/local-model.json` → `GET /api/local-model` and its install verbs | * * ## Cross-domain relationship map * @@ -79,3 +80,4 @@ export * from './intent.js'; export * from './foreign-tmux.js'; export * from './mcp-sync.js'; export * from './local-model.js'; +export * from './local-model-install.js'; diff --git a/src/types/local-model-install.ts b/src/types/local-model-install.ts new file mode 100644 index 00000000..2fdd8f2b --- /dev/null +++ b/src/types/local-model-install.ts @@ -0,0 +1,638 @@ +/** + * @fileoverview Types of the local model INSTALL: the runtime pin, the job's states, steps and + * error codes, the progress, the `install` object of `GET /api/local-model`, the bodies and + * answers of the install verbs, and the shape of the state file. + * + * The read side (hardware, catalogue, recommendation) is in src/types/local-model.ts. This + * file holds `LocalModelStatus`, the whole `data` of the GET, because it joins the two. + * + * Who produces what: + * - `LocalModelRuntimePin`: a compiled constant in src/local-model/runtime.ts. No file, no + * setting and no request can change which binary is downloaded. + * - `LocalModelStateFile`: src/local-model/state-store.ts, the ONE writer of + * `local-model.json` (mode 0600, temp file + rename). + * - `LocalModelInstallView`: src/local-model/install-job.ts (`buildInstallView()`, pure), from + * the state file plus what is live (the runtime child, the developer tools check). + * - `LocalModelStatus` and the verb answers: src/web/routes/local-model-routes.ts. + * + * The HTTP contract. Every route is admin only in multi-user mode (403 `FORBIDDEN`) and + * EXPERIMENTAL like the GET (docs/versioning-policy.md). Every verb answers + * `{ success: true, data: LocalModelInstallView }` with the view AFTER the verb. + * + * | Route | Body | Refusals | + * |----------------------------------|-----------------------------|-------------------------------------------------------| + * | `GET /api/local-model` | none | none (a failed probe is an answer) | + * | `POST /api/local-model/install` | `LocalModelInstallRequest` | 400 not a row the server offers, or a saved endpoint | + * | | | already has the reserved id; 409 `job-running`, | + * | | | `already-installed`, `size-changed` | + * | `POST /api/local-model/cancel` | none | 409 `wrong-state` (no job is running) | + * | `POST /api/local-model/resume` | `LocalModelResumeRequest` | 409 `job-running`, `wrong-state` (not paused or | + * | | | failed), `sessions-open` (another port while live | + * | | | sessions use the model) | + * | `POST /api/local-model/runtime` | `LocalModelRuntimeRequest` | 409 `wrong-state` (not installed), `job-running`, | + * | | | `sessions-open` (stop, or another port, while live | + * | | | sessions use the model) | + * | `DELETE /api/local-model` | `LocalModelRemoveRequest` | 409 `job-running`, `sessions-open` | + * + * A 409 is the error envelope plus `data: LocalModelConflict`: + * `{ success: false, error, errorCode: 'CONFLICT', data: { reason, message, ... } }`. A 409 + * WITHOUT `data` is a write of `settings.json` or of the endpoint store that was refused + * because the file does not parse; `error` names the fix. A 400 the service names carries its + * sentence the same way: `{ success: false, error, errorCode: 'INVALID_INPUT', data: { + * message } }` (`LocalModelRefusal`), so the window can print it in its own language. + * + * The rules of src/types/local-model.ts hold here too: + * 1. A size a person reads travels twice: as bytes and as decimal GB text from `formatGb()` + * (`completedGb`, `totalGb`, `downloadGb`). The browser prints the text and adds or + * divides nothing for display. `mbPerSecond` is the one other unit, decimal MB per second. + * 2. Every sentence the server chooses comes from `LOCAL_MODEL_INSTALL_LABELS`, with its + * numbers and names in `vars`. `{name}` is taken by the translator, so no slot has that name. + * 3. A CLI id is data (`LocalModelInstallView.cli`): the browser passes it on and compares it + * with nothing. What a person reads is `cliLabel`. + */ + +import type { HardwareFacts, Recommendation } from './local-model.js'; + +// ---------- The runtime pin ---------- + +/** + * The one runtime build Codeman downloads and runs. A compiled constant + * (`OLLAMA_RUNTIME_PIN` in src/local-model/runtime.ts): the hash is compared before anything + * is unpacked, so the pin and the hash must come from the same reviewed source file. + */ +export interface LocalModelRuntimePin { + /** The runtime's family. One value today. */ + id: 'ollama'; + /** The release ('0.40.15'). `GET /api/version` of the running runtime must answer it. */ + version: string; + /** Where the archive is downloaded from. https, a fixed release asset. */ + url: string; + /** Size of the archive in bytes. A body that grows past it is refused before the hash is finished. */ + bytes: number; + /** Lowercase hex sha256 of the archive. */ + sha256: string; + /** What the unpacked folder takes on disk, in bytes (measured: about 579 MB). */ + unpackedBytes: number; + /** The oldest macOS major version the runtime supports. */ + minMacosMajor: number; +} + +// ---------- States, steps, codes ---------- + +/** + * Where the install stands. The verb of the flow is the same throughout: install, installing, + * installed. + * - `idle`: nothing is installed and no job exists. + * - `installing`: the one job is running; `step` says where. + * - `paused`: the job stopped at `step` because of Cancel or a server restart. It never + * continues without a click. + * - `failed`: the job stopped at `step` with `error`. + * - `installed`: the model is connected. Whether the runtime runs is `running`, not a state. + * The state table (what is on disk, what a restart and Cancel do, the allowed actions) is + * `INSTALL_STATE_TABLE` in src/local-model/install-job.ts. + */ +export type LocalModelInstallState = 'idle' | 'installing' | 'paused' | 'failed' | 'installed'; + +/** + * The steps of a job, in the order they run. + * - `runtime`: get the pinned runtime (download, compare the hash, unpack), start it, read + * the measured memory budget and judge the chosen model against it. + * - `model`: download the model (the runtime's pull). + * - `verify`: load the model and check it: one tiny chat, the memory and context the runtime + * reports, one tool call. + * - `agent`: install the coding agent. Left out of a job whose coding agent is installed. + * - `connect`: save the connection (the endpoint record and the settings flag). + */ +export type LocalModelInstallStep = 'runtime' | 'model' | 'verify' | 'agent' | 'connect'; + +/** Every step, in order. A job runs these, without `agent` when the coding agent is installed. */ +export const LOCAL_MODEL_INSTALL_STEPS: readonly LocalModelInstallStep[] = [ + 'runtime', + 'model', + 'verify', + 'agent', + 'connect', +]; + +/** + * Why a job failed, or (in state `installed`) what blocks a session. Each code has one + * sentence (`LOCAL_MODEL_ERROR_LABELS`) and one helpful next step (`LOCAL_MODEL_ERROR_NEXT`). + * + * | Code | Raised when | + * |---------------------------|--------------------------------------------------------------------------| + * | `runtime-download-failed` | the archive download still fails after its retries | + * | `runtime-damaged` | the archive's sha256 is not the pinned one (the file was deleted) | + * | `runtime-unpack-failed` | `/usr/bin/tar` failed | + * | `runtime-start-failed` | the runtime was started and never answered `/api/version`. Installed: | + * | | the last start failed (the detail says why), and Start tries again | + * | `runtime-stopped` | the runtime crashed and was restarted 3 times, then given up: on an | + * | | installed model, or during a job's download | + * | `runtime-not-running` | installed: the runtime is not running (the person stopped it). Only a | + * | | refused session start says it; the view shows no error for a stopped runtime | + * | `port-in-use` | another program listens on the stored port, or no port of the span is free | + * | `disk-full` | a write failed for lack of space | + * | `pull-failed` | the pull still ends on an error line or a cut stream after its retries | + * | `model-not-found` | the registry no longer has this model under this name | + * | `does-not-fit` | the measured budget rules the model out, or the runtime did not put all of it on the GPU | + * | `context-not-applied` | the runtime loaded the model with another context than the chosen one | + * | `load-failed` | the tiny chat that loads the model failed | + * | `no-tool-call` | the tool probe was answered without a tool call with parseable arguments | + * | `probe-unfinished` | the tool probe ended on its token limit, so it proves nothing | + * | `harness-failed` | the coding agent's install command failed, or its binary is still missing | + * | `endpoint-id-taken` | a saved endpoint with our id exists that this feature did not write | + * | `connect-failed` | the endpoint record or the settings flag could not be written | + * | `model-changed` | installed: the model's digest is not the one stored at install time | + * | `internal` | anything unexpected (the exception text is in `detail`) | + */ +export type LocalModelErrorCode = + | 'runtime-download-failed' + | 'runtime-damaged' + | 'runtime-unpack-failed' + | 'runtime-start-failed' + | 'runtime-stopped' + | 'runtime-not-running' + | 'port-in-use' + | 'disk-full' + | 'pull-failed' + | 'model-not-found' + | 'does-not-fit' + | 'context-not-applied' + | 'load-failed' + | 'no-tool-call' + | 'probe-unfinished' + | 'harness-failed' + | 'endpoint-id-taken' + | 'connect-failed' + | 'model-changed' + | 'internal'; + +/** + * The ONE action that helps after an error. The window draws one button for it. + * - `retry`: "Try again". `POST /api/local-model/resume`. + * - `other-port`: "Use another port". `POST /api/local-model/resume` with `newPort: true` + * (state `failed`), or `POST /api/local-model/runtime` with `action: 'start'` and + * `newPort: true` (state `installed`). + * - `next-model`: "Try ". Back to the chooser with `error.nextModel` preselected; the + * person confirms its sizes and a new `POST /api/local-model/install` replaces the job. + * Without a `nextModel` the window offers the chooser and Remove. + * - `start`: "Start". `POST /api/local-model/runtime` with `action: 'start'`. + * - `remove`: "Remove". `DELETE /api/local-model`. + * + * `LOCAL_MODEL_ERROR_NEXT` is the default of a code. The view adapts it to the state the + * error is shown in, so the button is always one the server accepts there: a code whose + * default is `start` says `retry` on a failed job (there is no Start verb for a job), and + * `runtime-start-failed` says `start` on an installed model (there is nothing to resume). + */ +export type LocalModelNextStep = 'retry' | 'other-port' | 'next-model' | 'start' | 'remove'; + +/** + * What a client may do in the current state. The view's `actions` lists exactly the ones the + * server would accept right now, so the window shows a button only for those. + * - `install`: `POST /api/local-model/install`. + * - `cancel`: `POST /api/local-model/cancel`. + * - `resume`: `POST /api/local-model/resume`. + * - `start` and `stop`: `POST /api/local-model/runtime`. + * - `remove`: `DELETE /api/local-model`. + * - `start-session`: not a route of this feature: the window starts a session on the endpoint + * `endpointId` with the coding agent `cli`. Offered only while the runtime runs and nothing + * blocks a session. + */ +export type LocalModelAction = 'install' | 'cancel' | 'resume' | 'start' | 'stop' | 'remove' | 'start-session'; + +/** Whether the Mac has the developer tools (git, Python) a coding agent calls. */ +export type LocalModelDeveloperTools = + /** `/usr/bin/xcode-select -p` succeeded. */ + | 'present' + /** It failed: macOS will offer to install the tools the first time an agent needs one. */ + | 'missing' + /** Not macOS, or the check did not answer in time. Nothing is said then. */ + | 'unknown'; + +// ---------- Sentences ---------- + +/** + * Every sentence the server may put into the `install` object or a 409, in English (the i18n + * source language). Like `LOCAL_MODEL_LABELS`: the zh-CN dictionary in src/web/public/i18n.js + * has one entry per value, keyed by this exact text, and the i18n test lists them from here. + * A constant of its own, so the sentences of the recommendation and of the install can be + * listed apart. + * + * | Key | Slots (all text) | + * |------------------------|----------------------------------------------------------------------| + * | `errPortInUse` | `port`: the port number | + * | `errModelNotFound`, `errDoesNotFit`, `errLoadFailed`, `errNoToolCall`, `errProbeUnfinished` | `model`: the model's label | + * | `errContextNotApplied` | `model`; `actual`: the context the runtime reports; `wanted`: the chosen one | + * | `errHarnessFailed` | `cli`: the coding agent's label | + * | `errEndpointIdTaken` | `id`: the endpoint id this feature uses | + * | `refusedAgentNotInstallable` | `cli`: the coding agent's label | + * | `refusedOtherModel` | `model`: the label of the model that is installed | + * | `conflictSizeChanged` | `size`: the server's own total as decimal GB | + * | `conflictSessionsOpen` | `sessions`: the session names, joined with ', ' | + * + * `developerToolsMissing` is not sent: the view carries `developerTools` and the window prints + * this sentence when it is `missing`. It is here so that its wording has one source. + * + * `endpointManaged` and `endpointIdReserved` are what the endpoint routes + * (src/web/routes/custom-model-routes.ts) answer for the record this feature manages and for + * its reserved id. They have no slot, so the App Settings toast that prints them translates + * them as it translates every other whole sentence. + */ +export const LOCAL_MODEL_INSTALL_LABELS = { + errRuntimeDownloadFailed: + 'The download of the runtime stopped and could not be continued. Check the network connection, then try again.', + errRuntimeDamaged: 'The runtime download was damaged and has been deleted. Try again.', + errRuntimeUnpackFailed: 'The runtime could not be unpacked on this Mac.', + errRuntimeStartFailed: 'The runtime did not start on this Mac.', + errRuntimeStopped: 'The runtime keeps stopping, so Codeman no longer restarts it.', + errRuntimeNotRunning: 'The model on this Mac is not running. Start it first.', + errPortInUse: 'Port {port} is in use by another program.', + errDiskFull: 'The disk is full. Free up space, then try again.', + errPullFailed: 'The download keeps stopping. The network may be damaging data. Try again or use another network.', + errModelNotFound: '{model} is no longer offered under this name. Update Codeman or pick another model.', + errDoesNotFit: '{model} does not fit in the memory of this Mac after all.', + errContextNotApplied: '{model} loaded with a context of {actual} tokens instead of {wanted}.', + errLoadFailed: '{model} did not load on this Mac.', + errNoToolCall: '{model} loaded but did not call a tool, so a coding agent cannot work with it.', + errProbeUnfinished: 'The check of {model} did not finish.', + errHarnessFailed: '{cli} could not be installed.', + errEndpointIdTaken: + 'A saved model connection named {id} already exists and was not created here. Rename or delete it in App Settings, then try again.', + errConnectFailed: 'Codeman could not save the connection to the model.', + errModelChanged: 'The model on this Mac changed since it was installed. Remove it and install it again.', + errInternal: 'The install stopped on an unexpected error.', + conflictJobRunning: 'An install is already running.', + conflictAlreadyInstalled: 'A model is already installed on this Mac. Remove it first.', + conflictSizeChanged: 'The download is {size} GB now. Check the sizes and confirm again.', + conflictSessionsOpen: 'Sessions are still using this model: {sessions}. Close them first.', + conflictWrongState: 'That is not possible in the current state.', + refusedModelNotOffered: 'This model at this context is not offered for this Mac right now.', + refusedAgentNotOffered: 'This coding agent is not offered for this model.', + refusedAgentNotInstallable: '{cli} is not installed and cannot be installed from here.', + refusedOtherModel: 'Only {model} is installed on this Mac. Remove it and install another model first.', + endpointManaged: + 'This endpoint belongs to the model Codeman runs on this Mac. Change or remove it in the local model window.', + endpointIdReserved: 'This endpoint id is reserved for the model Codeman runs on this Mac. Pick another id.', + developerToolsMissing: + 'Coding agents use git and Python. This Mac has no developer tools yet, so macOS will offer to install them the first time an agent needs one.', +} as const; + +/** One of the texts in `LOCAL_MODEL_INSTALL_LABELS`. */ +export type LocalModelInstallLabel = (typeof LOCAL_MODEL_INSTALL_LABELS)[keyof typeof LOCAL_MODEL_INSTALL_LABELS]; + +/** A sentence of the install: a text from `LOCAL_MODEL_INSTALL_LABELS` plus the values of its `{placeholder}` slots. */ +export interface LocalModelInstallMessage { + text: LocalModelInstallLabel; + /** Slot values, already formatted as text. Absent when the text has no slot. */ + vars?: Record; +} + +/** The sentence of each error code. */ +export const LOCAL_MODEL_ERROR_LABELS: Record = { + 'runtime-download-failed': LOCAL_MODEL_INSTALL_LABELS.errRuntimeDownloadFailed, + 'runtime-damaged': LOCAL_MODEL_INSTALL_LABELS.errRuntimeDamaged, + 'runtime-unpack-failed': LOCAL_MODEL_INSTALL_LABELS.errRuntimeUnpackFailed, + 'runtime-start-failed': LOCAL_MODEL_INSTALL_LABELS.errRuntimeStartFailed, + 'runtime-stopped': LOCAL_MODEL_INSTALL_LABELS.errRuntimeStopped, + 'runtime-not-running': LOCAL_MODEL_INSTALL_LABELS.errRuntimeNotRunning, + 'port-in-use': LOCAL_MODEL_INSTALL_LABELS.errPortInUse, + 'disk-full': LOCAL_MODEL_INSTALL_LABELS.errDiskFull, + 'pull-failed': LOCAL_MODEL_INSTALL_LABELS.errPullFailed, + 'model-not-found': LOCAL_MODEL_INSTALL_LABELS.errModelNotFound, + 'does-not-fit': LOCAL_MODEL_INSTALL_LABELS.errDoesNotFit, + 'context-not-applied': LOCAL_MODEL_INSTALL_LABELS.errContextNotApplied, + 'load-failed': LOCAL_MODEL_INSTALL_LABELS.errLoadFailed, + 'no-tool-call': LOCAL_MODEL_INSTALL_LABELS.errNoToolCall, + 'probe-unfinished': LOCAL_MODEL_INSTALL_LABELS.errProbeUnfinished, + 'harness-failed': LOCAL_MODEL_INSTALL_LABELS.errHarnessFailed, + 'endpoint-id-taken': LOCAL_MODEL_INSTALL_LABELS.errEndpointIdTaken, + 'connect-failed': LOCAL_MODEL_INSTALL_LABELS.errConnectFailed, + 'model-changed': LOCAL_MODEL_INSTALL_LABELS.errModelChanged, + internal: LOCAL_MODEL_INSTALL_LABELS.errInternal, +}; + +/** The one helpful next step of each error code. */ +export const LOCAL_MODEL_ERROR_NEXT: Record = { + 'runtime-download-failed': 'retry', + 'runtime-damaged': 'retry', + 'runtime-unpack-failed': 'retry', + 'runtime-start-failed': 'retry', + 'runtime-stopped': 'start', + 'runtime-not-running': 'start', + 'port-in-use': 'other-port', + 'disk-full': 'retry', + 'pull-failed': 'retry', + 'model-not-found': 'next-model', + 'does-not-fit': 'next-model', + 'context-not-applied': 'retry', + 'load-failed': 'retry', + 'no-tool-call': 'next-model', + 'probe-unfinished': 'retry', + 'harness-failed': 'retry', + 'endpoint-id-taken': 'retry', + 'connect-failed': 'retry', + 'model-changed': 'remove', + internal: 'retry', +}; + +// ---------- The `install` object of the GET ---------- + +/** + * How far a download is. Present while the `runtime` step downloads the archive and while the + * `model` step pulls, once the total is known; kept on a paused or failed job as it stood. + * During the `model` step both byte figures come from the pull's own lines, never from the + * catalogue. + */ +export interface LocalModelProgress { + completedBytes: number; + totalBytes: number; + /** `completedBytes` as decimal GB text, no unit ('6.2'). */ + completedGb: string; + /** `totalBytes` as decimal GB text, no unit ('23.1'). */ + totalGb: string; + /** Download speed. Absent until it can be told (the first moments, a paused job). */ + bytesPerSecond?: number; + /** `bytesPerSecond` as whole decimal megabytes per second, no unit ('58'). Present exactly when `bytesPerSecond` is. */ + mbPerSecond?: string; + /** Seconds left at the current speed, rounded up. Absent when the speed is unknown or 0. */ + secondsLeft?: number; +} + +/** An error as the window shows it: the sentence, the one action that helps, and the raw detail. */ +export interface LocalModelInstallError { + code: LocalModelErrorCode; + /** `LOCAL_MODEL_ERROR_LABELS[code]` with its slots filled. */ + message: LocalModelInstallMessage; + /** `LOCAL_MODEL_ERROR_NEXT[code]`, adapted to the state the error is shown in (see `LocalModelNextStep`). */ + next: LocalModelNextStep; + /** + * For `next-model`: the model to try instead. After `does-not-fit` it is one that needs LESS + * memory than the one that failed; after the other codes the best other model that fits. + * Absent when there is none. + */ + nextModel?: { modelId: string; modelLabel: string }; + /** + * Raw text for the person who wants to know more: the last lines of an installer, the + * message of a refused write, an exception. Not translated; printed with `textContent`. + */ + detail?: string; +} + +/** The model of the job or of the installation. */ +export interface LocalModelInstalledModel { + /** `LocalModelEntry.id`. */ + id: string; + /** `LocalModelEntry.label`. */ + label: string; +} + +/** The runtime, as the consent screen needs it. */ +export interface LocalModelRuntimeView { + /** `LocalModelRuntimePin.version`. */ + version: string; + /** The pinned runtime is unpacked on disk. Then an install downloads only the model. */ + installed: boolean; + /** What an install would download for the runtime: the archive size, 0 when `installed`. */ + downloadBytes: number; + /** `downloadBytes` as decimal GB text, no unit ('0.2'). */ + downloadGb: string; + /** What the unpacked runtime takes on disk, as decimal GB text, no unit ('0.6'). */ + unpackedGb: string; +} + +/** What the runtime reported about the loaded model when the job checked it. The facts of the Done line. */ +export interface LocalModelLoadedFacts { + /** The context length in tokens the runtime loaded the model with. */ + context: number; + /** Memory the loaded model took, in bytes (`size` of the runtime's process list). */ + memoryBytes: number; + /** `memoryBytes` as decimal GB text, no unit. */ + memoryGb: string; + /** All of it is on the GPU (`size_vram` equals `size`). Always true after a job that passed. */ + onGpu: boolean; +} + +/** + * The `install` object of `GET /api/local-model`: where the install stands and what can be + * done about it. The window polls it once a second while it is open; there is no SSE event. + * + * Which members are present, by state: + * + * | Member | idle | installing | paused | failed | installed | + * |-----------------------------------|------|------------|--------|--------|-----------| + * | `step`, `steps` | | yes | yes | yes | | + * | `progress` | | during a download | as it stood, when known | as it stood, when known | | + * | `error` | | | | yes | only when something blocks a session | + * | `model`, `cli`, `cliLabel`, `context` | | yes | yes | yes | yes | + * | `port` | | once chosen | once chosen | once chosen | yes | + * | `loaded`, `endpointId` | | | | | yes | + * + * `running`, `developerTools`, `runtime`, `folder` and `actions` are always present. + */ +export interface LocalModelInstallView { + state: LocalModelInstallState; + /** The step the job is in, or stopped in. */ + step?: LocalModelInstallStep; + /** The steps of THIS job in order, for the step list (`agent` is absent when the coding agent was already installed). */ + steps?: LocalModelInstallStep[]; + progress?: LocalModelProgress; + error?: LocalModelInstallError; + model?: LocalModelInstalledModel; + /** The coding agent: a CLI registry id, as data. */ + cli?: string; + /** The coding agent's display label ('OpenCode'). */ + cliLabel?: string; + /** The context length in tokens the runtime is started with. */ + context?: number; + /** The loopback port of the runtime. */ + port?: number; + /** The runtime child Codeman started is alive right now. */ + running: boolean; + developerTools: LocalModelDeveloperTools; + runtime: LocalModelRuntimeView; + /** The absolute folder everything of this feature is kept in, for the "where it goes" line. */ + folder: string; + loaded?: LocalModelLoadedFacts; + /** The id of the saved endpoint a session is started on. */ + endpointId?: string; + /** What the server accepts right now, see `LocalModelAction`. */ + actions: LocalModelAction[]; +} + +/** The `data` of `GET /api/local-model`. Admin only in multi-user mode. */ +export interface LocalModelStatus { + hardware: HardwareFacts; + recommendation: Recommendation; + /** `LocalModelCatalog.updated`. */ + catalogUpdated: string; + /** Where the install stands. Always present. */ + install: LocalModelInstallView; +} + +// ---------- Request bodies and answers ---------- + +/** + * Body of `POST /api/local-model/install`. The three choices must name a row the server offers + * right now: a model of `recommendation.pick` or `recommendation.alternatives`, at that + * candidate's `context`, with one of its `harnesses`. Anything else is a 400 before any + * download. + */ +export interface LocalModelInstallRequest { + /** `Candidate.modelId`. */ + modelId: string; + /** `CandidateHarness.cli`. */ + cli: string; + /** `Candidate.context`. */ + context: number; + /** + * The total the person saw and confirmed: `Candidate.installBytes`. The server compares it + * with its own total and answers 409 `size-changed` when they differ, so a consent is bound + * to the size that was shown. + */ + acknowledgedBytes: number; +} + +/** Body of `POST /api/local-model/resume`. May be absent or empty. */ +export interface LocalModelResumeRequest { + /** True: choose another port first (the answer to `port-in-use`). */ + newPort?: boolean; +} + +/** Body of `POST /api/local-model/runtime`. */ +export interface LocalModelRuntimeRequest { + /** `stop` also keeps the runtime from being started at the next boot; `start` undoes that. */ + action: 'start' | 'stop'; + /** With `start`: choose another port first (the answer to `port-in-use`), and move the saved endpoint to it. */ + newPort?: boolean; +} + +/** Body of `DELETE /api/local-model`. May be absent or empty. */ +export interface LocalModelRemoveRequest { + /** True: delete the downloaded model too. Default false: the model stays for a later install. */ + removeModel?: boolean; +} + +/** The `data` of a 400 the service names: its sentence, for the window to print in its own language. */ +export interface LocalModelRefusal { + message: LocalModelInstallMessage; +} + +/** What every verb answers on success: the view after the verb. */ +export type LocalModelVerbAnswer = LocalModelInstallView; + +/** + * Why a verb was refused with 409. + * - `job-running`: a job is running (a second install, a resume, a remove or a runtime action). + * - `already-installed`: install while a model is installed. Remove it first. + * - `size-changed`: `acknowledgedBytes` is not the server's total; `expectedBytes` and + * `expectedGb` carry that total, so the window can show it and ask again. + * - `sessions-open`: remove, stop or a change of port while live sessions use the endpoint; + * `sessions` names them. Their coding agents were given the runtime's address when they + * started, so they would go on calling a port nothing of ours holds. + * - `wrong-state`: the verb makes no sense in the current state (see the route table above). + */ +export type LocalModelConflictReason = + | 'job-running' + | 'already-installed' + | 'size-changed' + | 'sessions-open' + | 'wrong-state'; + +/** The `data` of a 409 of an install verb. */ +export interface LocalModelConflict { + reason: LocalModelConflictReason; + /** The sentence for the window (`conflict...` of `LOCAL_MODEL_INSTALL_LABELS`). */ + message: LocalModelInstallMessage; + /** `size-changed`: the server's own total in bytes. */ + expectedBytes?: number; + /** `size-changed`: `expectedBytes` as decimal GB text, no unit. */ + expectedGb?: string; + /** `sessions-open`: the names of the live sessions that use the endpoint. */ + sessions?: string[]; +} + +// ---------- The state file ---------- + +/** + * `local-model.json` in the data dir: everything that must survive a restart. Mode 0600, + * written through a temp file and a rename, by src/local-model/state-store.ts alone. + * + * Rules: + * - It holds NO path. Every path of this feature is built from `dataPath()` constants + * (`localModelPaths()`), so a state file can never point a delete somewhere else. + * - Reading is total: a missing file, a file that does not parse or one with another `schema` + * reads as `{ schema: 1, state: 'idle' }`. + * - A file read at boot with state `installing` is a job a restart interrupted: it becomes + * `paused` before anything else looks at it. + */ +export interface LocalModelStateFile { + schema: 1; + state: LocalModelInstallState; + /** The step the job is in or stopped in. Absent in `idle` and `installed`. */ + step?: LocalModelInstallStep; + /** The steps of the job, in order. Absent in `idle` and `installed`. */ + steps?: LocalModelInstallStep[]; + /** State `failed`: why. State `installed`: absent (a blocking problem is found live, never stored). */ + error?: { + code: LocalModelErrorCode; + /** The slot values of the code's sentence. */ + vars?: Record; + detail?: string; + /** For a `next-model` code: the model to offer, when the job knew one. */ + nextModelId?: string; + }; + /** The model of the job or of the installation. */ + model?: { + id: string; + label: string; + /** The context length in tokens the person chose. */ + context: number; + /** The total the pull reported, once known. */ + pullBytes?: number; + /** The digest `/api/tags` reported after the pull. Compared again at boot and before a session starts. */ + digest?: string; + }; + /** + * The model of the job this one replaced, when it was another model: what that job left in + * the model store is deleted once the runtime runs, before the new model is downloaded. + * Absent once that was done, and in `idle` and `installed`. + */ + replacedModelId?: string; + /** The coding agent of the job or of the installation. */ + cli?: { id: string; label: string }; + /** The runtime, once its port was chosen. */ + runtime?: { + /** + * The runtime version this installation runs. A later build with another pin keeps + * starting this version for as long as its folder is unpacked. + */ + version: string; + /** The loopback port, chosen once by binding. */ + port: number; + /** + * The pid of the child Codeman last started, stored the moment it is spawned. Used to + * tell our own runtime from another program: at boot, and before the folder is deleted. + */ + pid?: number; + /** False after Stop: the runtime is not started at boot. */ + enabled: boolean; + /** The device total the runtime printed when it started, in bytes. */ + measuredTotalBytes?: number; + }; + /** How far the last download was when the job paused or failed. A restart may leave it absent. */ + progress?: { completedBytes: number; totalBytes: number }; + /** What the runtime reported when the job checked the loaded model. */ + loaded?: { context: number; sizeBytes: number; sizeVramBytes: number }; + /** + * Present once the `connect` step CLAIMED the connection, which it does before its two + * writes: what uninstall must undo. A claim with nothing behind it (the step was interrupted + * right after it) is harmless: a resumed job writes both, and uninstall removes nothing. + */ + connection?: { + /** The id of the endpoint record this feature wrote. */ + endpointId: string; + /** The value `customModelEndpointsEnabled` had before this feature switched it on. */ + flagWasEnabled: boolean; + }; + /** ISO time of the last write. */ + updatedAt?: string; +} diff --git a/src/types/local-model.ts b/src/types/local-model.ts index 8b8cfc75..5baa6903 100644 --- a/src/types/local-model.ts +++ b/src/types/local-model.ts @@ -1,9 +1,12 @@ /** * @fileoverview Types of the hardware-aware local model recommendation: what the server sees of - * this Mac, the catalogue of models and harnesses, and the answer of `GET /api/local-model`. + * this Mac, the catalogue of models and harnesses, and the recommendation part of the answer of + * `GET /api/local-model`. * - * This stage is read-only: Codeman looks at the Mac and recommends. It downloads nothing, - * installs nothing and writes no file. + * Everything here is the READ side: Codeman looks at the Mac and recommends. The types of the + * install (the job states, the progress, the `install` object of the GET, the request bodies, + * the state file, the runtime pin) and `LocalModelStatus`, the whole `data` of the GET, are in + * src/types/local-model-install.ts. Both files go out through the barrel (`../types.js`). * * Who produces what: * - `HardwareFacts`: src/local-model/hardware.ts (one bounded `sysctl` spawn, never throws). @@ -11,7 +14,11 @@ * - `HarnessFact`: src/local-model/harness-facts.ts (server only, read from the CLI registry at * call time; it holds a function, so it never goes over HTTP). * - `Recommendation`, `Candidate`, `DeviceBudget`: src/local-model/recommend.ts (pure). - * - `LocalModelStatus`: src/web/routes/local-model-routes.ts (the `data` of the GET). + * + * A field that is optional here is optional by meaning, and its comment says when it is + * absent (a tested entry without a measured speed, a harness that needs no install command, a + * row without a tag). Everything else is always there, so the chooser window and the install + * job may rely on it. * * Three rules every producer and consumer follows: * @@ -90,7 +97,11 @@ export interface HardwareFacts { // ---------- Catalogue ---------- -/** One Mac a model and harness pair was really run on by us. A model row with a matching entry may be called "Recommended". */ +/** + * One Mac a model and harness pair was really run on by us. A candidate that matches an entry + * (the same chip string, the same memory size as sold, the same harness) is `tested`: it ranks + * before every candidate that is not, and as the pick it may be called "Recommended". + */ export interface TestedOn { /** The chip, as `machdep.cpu.brand_string` reports it ('Apple M4'). */ chip: string; @@ -100,6 +111,13 @@ export interface TestedOn { cli: string; /** ISO date of the test ('2026-11-06'). */ date: string; + /** + * The generation speed measured in that test, in tokens per second (67.7 for the + * 2026-11-06 run on an Apple M5 Pro with 48 GB). A candidate that matches this entry reports + * this figure as `Candidate.tokensPerSecond` in place of the estimate from the chip's memory + * bandwidth: a measurement on the same Mac beats a formula. + */ + tokensPerSecond?: number; } /** One model of the catalogue. */ @@ -112,17 +130,44 @@ export interface LocalModelEntry { license: string; /** Parameter count in billions. When no model that fits has 12 or more, the verdict is `limited`, never `ok`. */ paramsB: number; - /** Size of the weights in bytes: both the download and what they take in memory. */ + /** + * Size of the weights in bytes: both the download and what they take in memory. + * + * It is the size of the build the runtime really downloads ON A MAC. That is not always + * the sum of the registry manifest: `qwen3.6:35b-a3b` is an MLX build of 23,105,798,956 bytes + * on Apple silicon, 2% more than the manifest's 22,621,314,161. A row whose Mac size nobody + * measured keeps the manifest sum and says so in `figures`. + */ bytes: number; - /** Bytes of KV cache one token of context costs. */ - kvBytesPerToken: number; + /** + * Bytes that stay HELD in memory per token of context once the token was processed (measured + * on the MLX runner: 36,000 to 38,000, so about 37,000). A figure computed from a model's + * config alone came out too low by almost half, so a row that was not measured carries the + * measured row's figure and says so in `figures`. + */ + heldBytesPerToken: number; + /** What the runner holds beside the weights and the context (measured: about 3 GB). */ + overheadBytes: number; + /** + * The TRANSIENT extra the runner takes while it processes a long prompt, on top of + * what stays held (measured: about 5 GB). The fit is judged on the peak: + * `bytes + heldBytesPerToken * context + overheadBytes + peakExtraBytes`. For + * `qwen3.6:35b-a3b` at 131072 tokens that is about 36 GB, which is tight on a 48 GB Mac. + */ + peakExtraBytes: number; + /** + * Where `bytes` and the three memory figures above come from: `measured` on a Mac by + * us, or `estimated` (the manifest size, and the per-token, overhead and peak figures of the + * measured row applied to this one). + */ + figures: 'measured' | 'estimated'; /** Context lengths (tokens) the model may be run at, largest first. */ contexts: number[]; /** Preference among models that fit: the HIGHER rank wins. A judgement, not a measurement. */ rank: number; /** Measured tokens per second per 100 GB/s of memory bandwidth. Absent: the speed of this model is unknown. */ speedPer100Gbps?: number; - /** Macs this model was really run on. Empty today for every row. */ + /** Macs this model was really run on. Empty for every row but one. */ tested: TestedOn[]; } @@ -155,16 +200,6 @@ export interface ChipEntry { gbps: number; } -/** - * Host memory the runtime's prompt cache takes, by the size of the Mac. Tiers are ordered by - * `minMemGb` ascending and the last tier with `minMemGb <= HardwareFacts.memClassGb` applies. - */ -export interface PromptCacheTier { - /** Smallest memory size (as sold, binary gigabytes) this tier applies to. The first tier has 0. */ - minMemGb: number; - bytes: number; -} - /** The catalogue: a compiled constant, validated by `LocalModelCatalogSchema`. No override file, no download data. */ export interface LocalModelCatalog { schema: 1; @@ -172,7 +207,6 @@ export interface LocalModelCatalog { updated: string; /** Oldest macOS major version a local model runs on. Below it: verdict `none`, reason `macos`. */ minMacosMajor: number; - promptCacheTiers: PromptCacheTier[]; models: LocalModelEntry[]; harnesses: LocalHarnessEntry[]; chips: ChipEntry[]; @@ -194,13 +228,25 @@ export interface HarnessFact { installed: boolean; /** Not installed, but the registry has an install command for this platform that needs no administrator rights. */ installable: boolean; + /** + * The literal install command the install job would run for this CLI: the STOCK + * command of the CLI registry for this platform, never one a `clis.json` override supplies. + * Present exactly when `installable` is true. The consent screen prints it + * (`CandidateHarness.installCommand`). + */ + installCommand?: string; /** Whether this CLI can be launched with that model id (its command-line pattern accepts it). */ acceptsModelId: (modelId: string) => boolean; } // ---------- Recommendation ---------- -/** Need over budget: up to 0.85 is `good`, up to 0.98 is `tight`, above is `out`. */ +/** + * Need over budget: up to 0.85 is `good`, up to 0.98 is `tight`, above is `out`. The need is + * the PEAK, so the grade says how much room is left at the worst moment. It decides whether a + * model is offered at all (`out` never is) and what its row says (`tight`). It does not sort + * the list. + */ export type FitGrade = 'good' | 'tight' | 'out'; /** @@ -223,18 +269,54 @@ export type SpeedGrade = 'ok' | 'slow' | 'unknown'; */ export type NoneReason = 'not-macos' | 'intel' | 'memory' | 'disk' | 'macos' | 'no-harness'; -/** Where the device budget came from: a formula (`estimated`) or `iogpu.wired_limit_mb` above 0 (`wired-limit`). */ -export type BudgetSource = 'estimated' | 'wired-limit'; +/** + * Where the device budget came from. + * - `estimated`: a formula over the installed memory. All there is before the runtime ran. + * - `wired-limit`: `iogpu.wired_limit_mb` above 0. + * - `measured`: the device total the runtime itself printed when it started + * (`RecommendOptions.measuredTotalBytes`). The chooser says "estimated" next to the memory + * bar for the first value only. + */ +export type BudgetSource = 'estimated' | 'wired-limit' | 'measured'; /** The memory a model may use on this Mac. */ export interface DeviceBudget { - /** L: the wired limit when above 0, else 2/3 of memory below 36 GB and 3/4 from 36 GB. */ + /** + * L: the measured device total when there is one, else the wired limit when above 0, else + * 2/3 of memory below 36 GB and 3/4 from 36 GB. + */ limitBytes: number; - /** B = min(L - 1 GiB, memory - reserve). What `Candidate.budgetShare` is measured against. */ + /** + * What `Candidate.budgetShare` is measured against. Estimated or from the wired limit: + * B = min(L - 1 GiB, memory - reserve). Measured: B = min(L, memory - reserve), without the + * 1 GiB, because the runtime's figure is already what it will really use (37.4 GiB on the + * 48 GB test Mac, which reads '40.2'). + */ budgetBytes: number; source: BudgetSource; } +/** + * What the caller of `recommend()` knows beyond the hardware probe. Both members are optional: + * an empty object means the budget is the estimate and an install downloads the model alone. + */ +export interface RecommendOptions { + /** + * The device total the runtime reported in its log line "inference compute" + * (`total="37.4 GiB"`), in bytes: `parseInferenceComputeTotal()` in + * src/local-model/runtime.ts. Present: the budget is `measured`. The install job passes it + * right after the runtime started, before the model is downloaded, and the GET passes the + * stored figure from then on. + */ + measuredTotalBytes?: number; + /** + * What an install adds to every candidate's download beside the model: the size of the + * pinned runtime archive while the runtime is not installed, 0 once it is. It is the + * difference between `Candidate.installBytes` and `Candidate.downloadBytes`. Default 0. + */ + runtimeDownloadBytes?: number; +} + /** * Every sentence and label the server may put into the payload, in English (the i18n source * language). The zh-CN dictionary in src/web/public/i18n.js has one entry per value, keyed by @@ -257,7 +339,8 @@ export interface DeviceBudget { * | `noneMacos` | `version`: `HardwareFacts.macosVersion`; `minimum`: the catalogue minimum | * * The word "Recommended" exists only in `headlineTested`, which may be used only when the - * pick's `tested` is true. No catalogue row has a tested entry today, so it is never sent. + * pick's `tested` is true: one catalogue row has a tested entry (`qwen3.6:35b-a3b` on an Apple + * M5 Pro with 48 GB), so one kind of Mac is told so. */ export const LOCAL_MODEL_LABELS = { /** Verdict `ok`, pick not tested by us on a Mac like this one. */ @@ -268,7 +351,7 @@ export const LOCAL_MODEL_LABELS = { headlineLimited: 'This Mac can run a small model. It handles small edits and questions. It does not replace a cloud model.', /** A note under an `ok` or `limited` pick whose `tested` is false. */ - noteUntested: 'Not yet tested by us on a Mac like this.', + noteUntested: 'Codeman has not run this on a Mac like this yet.', /** A note under an `ok` pick that is itself a small model: larger ones fit this Mac, but rank lower or have no room on the disk. */ noteSmallPick: 'The best fit is a small model. It handles small edits and questions. It does not replace a cloud model.', @@ -280,18 +363,17 @@ export const LOCAL_MODEL_LABELS = { speedSlow: 'Slow', speedUnknown: 'Speed unknown', noneNotMacos: 'Codeman recommends a local model only on a Mac with Apple silicon.', - noneIntel: 'This Mac has an Intel processor. Local coding models need Apple silicon.', + noneIntel: 'Local coding models need Apple silicon, and this Mac has an Intel processor.', /** Reason `memory`: nothing fits at any context. */ - noneMemory: - 'This Mac has {memory} GB of memory. That is too little to run a model that can drive a coding agent, so Codeman recommends none here.', - /** Reason `memory`: a model fits, but only below the context a harness needs. Says both numbers, and that they are an estimate. */ + noneMemory: 'With {memory} GB of memory, this Mac cannot run a model that can drive a coding agent.', + /** Reason `memory`: a model fits, but only below the context a harness needs. Says both numbers, and ("about") that they are estimates. */ noneMemoryContext: - 'Codeman estimates that this Mac can give a model about {budget} GB of memory. The smallest model a coding agent can work with needs about {need} GB. That leaves no safe margin, so Codeman recommends none here.', + 'This Mac can give a model about {budget} GB of memory, and the smallest model a coding agent can work with needs about {need} GB.', /** Reason `memory`: the memory size could not be read, so no size is stated. */ noneMemoryUnknown: 'Codeman could not read how much memory this Mac has, so it recommends no model here.', - noneDisk: - 'Not enough free disk space. The smallest model that fits this Mac is a {download} GB download, and {reserve} GB must stay free after it. {free} GB is free now. Free up about {missing} GB.', - noneMacos: 'This Mac runs macOS {version}. Local models need macOS {minimum} or newer.', + /** Reason `disk`. One number: what to do. The other sizes travel in `vars` for a client that wants them. */ + noneDisk: 'Free up about {missing} GB of disk space to install the smallest model that fits this Mac.', + noneMacos: 'Local models need macOS {minimum} or newer, and this Mac runs macOS {version}.', noneNoHarness: 'No coding agent that Codeman can connect to a local model is available on this Mac.', } as const; @@ -315,6 +397,64 @@ export interface LocalModelMessage { vars?: Record; } +/** + * The one tag a model row of the chooser carries, by what it says. + * - `tested`: `Candidate.tested` is true. Replaces every other tag. + * - `best-tight`: the pick, when its `fit` is `tight`. The words must not say less than the + * memory bar beside them, which is nearly full for such a row. + * - `best`: the pick. + * - `tight`: `Candidate.fit` is `tight`. + * - `slow`: `Candidate.speed` is `slow`. + * - `small`: the model has fewer than 12 billion parameters. + * A row gets the FIRST kind of this list that applies and no tag when none does. `tested`, + * `best` and `best-tight` are the tags of the row the window opens on, drawn in the accent + * colour. + */ +export type CandidateTagKind = 'tested' | 'best-tight' | 'best' | 'tight' | 'slow' | 'small'; + +/** + * The text of each tag, in English (the i18n source language), like `LOCAL_MODEL_LABELS`. A + * constant of its own so that a consumer can list the tag texts apart from the sentences. + */ +export const LOCAL_MODEL_TAG_LABELS = { + tested: 'Tested on a Mac like this', + 'best-tight': 'Best fit, tight', + best: 'Best fit', + tight: 'Tight', + slow: 'Slow', + small: 'Small model', +} as const satisfies Record; + +/** One of the texts in `LOCAL_MODEL_TAG_LABELS`. */ +export type LocalModelTagLabel = (typeof LOCAL_MODEL_TAG_LABELS)[CandidateTagKind]; + +/** The tag of a model row: what kind it is (for the styling) and the text to print. */ +export interface CandidateTag { + kind: CandidateTagKind; + /** `LOCAL_MODEL_TAG_LABELS[kind]`. */ + label: LocalModelTagLabel; +} + +/** + * One coding agent a candidate's model can be run with on this Mac, for the "Runs + * with" select of the chooser. Everything a browser needs is here, so it compares no CLI id. + */ +export interface CandidateHarness { + /** The CLI registry id, as data: what `POST /api/local-model/install` is sent as `cli`. */ + cli: string; + /** The registry's display label ('OpenCode'). */ + cliLabel: string; + /** False: the install job installs it first (it is installable, or it would not be listed). */ + installed: boolean; + /** + * The literal command the install job would run to install it (`HarnessFact.installCommand`). + * Present exactly when `installed` is false. The consent screen prints it as it is. + */ + installCommand?: string; + /** The harness row's `slowFirstTurn`. */ + slowFirstTurn?: boolean; +} + /** One model at one context with one harness, measured against this Mac. Plain JSON. */ export interface Candidate { /** `LocalModelEntry.id`. */ @@ -333,7 +473,11 @@ export interface Candidate { cliInstalled: boolean; /** The harness row's `slowFirstTurn`. */ slowFirstTurn?: boolean; - /** F = weights + kvBytesPerToken * context + 1 GiB + prompt cache. */ + /** + * What the model needs in memory at this context: the PEAK, `bytes + heldBytesPerToken * + * context + overheadBytes + peakExtraBytes` of the catalogue row. The memory bar of the + * chooser is filled to `budgetShare`, which is this over the budget. + */ needBytes: number; /** `needBytes` as decimal GB text, no unit. */ needGb: string; @@ -350,10 +494,36 @@ export interface Candidate { speed: SpeedGrade; /** `speedSlow` or `speedUnknown`. Absent when `speed` is `ok`. */ speedLabel?: LocalModelLabel; - /** Estimated generation speed, rounded to one decimal. Present exactly when `speed` is not `unknown`. `speed` is decided on the unrounded figure. */ + /** + * Generation speed in tokens per second, rounded to one decimal. Present exactly when `speed` + * is not `unknown`; `speed` is decided on the unrounded figure. An estimate from the chip's + * memory bandwidth, except for a candidate that is `tested`, which reports the + * speed measured in that test (`TestedOn.tokensPerSecond`). + */ tokensPerSecond?: number; - /** A `TestedOn` entry of the model matches this Mac (same chip, same `memClassGb`) and this harness. */ + /** + * A `TestedOn` entry of the model matches this Mac (same chip, same `memClassGb`) and this + * harness. Such a candidate ranks first, and its `tag` says so in place of any other tag. + */ tested: boolean; + /** `LocalModelEntry.figures` of the model: whether `needBytes` rests on a measurement of this model. */ + needSource: 'measured' | 'estimated'; + /** The row's one tag, see `CandidateTagKind`. Absent when no kind applies. */ + tag?: CandidateTag; + /** + * Every coding agent this model can be run with at `context` on this Mac, best + * first. The first entry is the one the flat fields above name (`cli`, `cliLabel`, + * `cliInstalled`, `slowFirstTurn`). Never empty. + */ + harnesses: CandidateHarness[]; + /** + * The whole download a click on Install starts for this row: `downloadBytes` plus + * `RecommendOptions.runtimeDownloadBytes`. It is the server's own total, and exactly the + * number `POST /api/local-model/install` must be sent as `acknowledgedBytes`. + */ + installBytes: number; + /** `installBytes` as decimal GB text, no unit. */ + installGb: string; } /** @@ -376,7 +546,12 @@ export interface Recommendation { notes: LocalModelMessage[]; /** Present exactly when `verdict` is `ok` or `limited`. */ pick?: Candidate; - /** The next models that fit and have room on the disk (at most four, each at its best context and harness), best first. Empty when `verdict` is `none`. */ + /** + * The next models that fit and have room on the disk (at most four), best first: tested + * before not tested, then ok before unknown before slow speed, then the higher model rank. + * Each is at the largest context it fits at, with the best harness that can run it there. + * Empty when `verdict` is `none`. + */ alternatives: Candidate[]; /** `DeviceBudget.budgetBytes`. 0 when no budget is computed: not macOS, or a Mac without Apple silicon. */ budgetBytes: number; @@ -385,12 +560,4 @@ export interface Recommendation { budgetSource: BudgetSource; } -// ---------- API ---------- - -/** The `data` of `GET /api/local-model`. Read-only; admin only in multi-user mode. */ -export interface LocalModelStatus { - hardware: HardwareFacts; - recommendation: Recommendation; - /** `LocalModelCatalog.updated`. */ - catalogUpdated: string; -} +// `LocalModelStatus`, the `data` of `GET /api/local-model`, is in src/types/local-model-install.ts. diff --git a/src/web/CLAUDE.md b/src/web/CLAUDE.md index 2f29a211..f1b58419 100644 --- a/src/web/CLAUDE.md +++ b/src/web/CLAUDE.md @@ -48,7 +48,7 @@ Frontend JS modules have `@fileoverview` with `@dependency`/`@loadorder` tags. L **Opened outside Codeman** (`foreign-sessions.js`, both home screens): the tmux sessions a human started, listed from `GET /api/mux/foreign` (root CLAUDE.md, Adopted tmux sessions). ONE renderer, `renderForeignSessions(container)`, fills the welcome block and the phone overview's, and every row goes through `textContent`, since a session name and a command line come from someone else's process. ⚠️ It polls only while a home screen is up (`startForeignPolling`/`stopForeignPolling` from `showWelcome()`/`hideWelcome()`) and the tab is visible (`_foreignPollTick()` is the one gate, and `visibilitychange` catches up; the cache TTL sits under the poll interval, so one tab's every tick is a full socket sweep), and the poll is local only: the docker and ssh scan costs one connection per target, so it runs once on the toggle and again on Rescan, and its rows and notes are kept apart (`_foreignWideSessions`, `_foreignWideNotes`) or the next local poll would wipe them. ⚠️ A 403 is a non-admin in multi-user mode, an answer and not a failed poll: `_foreignRefused` stops the polling and shows no block. ⚠️ `closeSession()` re-reads the list after an adopted tab's DELETE, since the home screen's own load races it and the row would say `Go to tab` until the next poll. ⚠️ An adopted tab has no kill outcome to offer, so the close dialog hides the red button and always tears the wrapper down. ⚠️ `session.remote` and `session.docker` are undefined on an adopted session, so a check that hides a row for those two usually needs `session.adopt` as well. Tests: `test/foreign-sessions-ui.test.ts`, `test/foreign-sessions-i18n.test.ts` (harvests every string from the real module). -**Model on this Mac** (`local-model-ui.js`; welcome card `#localModelCard` + App Settings > Models group `#localModelGroup`; server rules in the root CLAUDE.md, Local model recommendation): a READ-ONLY view of `GET /api/local-model`. ONE builder (`localModelBody`) paints both containers, so the card and the Settings row can never name a different pick; the card folds the alternatives, Settings lists them. ⚠️ Every dynamic string goes in through `textContent` (model, harness and chip names are someone else's text), both containers carry `data-i18n-skip`, and the module translates its own sentences at render time, so a language or display-name change must repaint from the module (the `` observer). ⚠️ Labels and sentences come from the payload (`{ text, vars }` through `localModelT`); the module holds no list of models or harnesses, never compares a CLI id, and never contains the word "Recommended" (that headline is the server's to send). ⚠️ `test/local-model-i18n.test.ts` harvests every `localModelT('...')` literal: a new string is a single-quoted literal with no apostrophe and gets a zh-CN entry, and the only non-literal calls are the two payload ones. ⚠️ Sizes are printed as sent (decimal GB text), bytes are never divided here; the installed memory alone is `memClassGb`, a number or nothing (an unknown size is left out, never `0 GB`). ⚠️ The card ships `hidden` and shows only once the server answered for a Mac; the card AND the Settings group stay hidden for `not-macos` (the server is usually Linux, and "this Mac" always means the machine the server runs on), for a refusal and while the first request is in flight; a failed first request shows one sentence in Settings only. ⚠️ Refused means `403` with `errorCode: FORBIDDEN` (a non-admin in multi-user mode: stop asking); the auth layer's `403 PASSWORD_CHANGE_REQUIRED` is a failed request that is asked again on `codeman:me`. ⚠️ Each fit and speed pill carries what it measures as `title` + `aria-label` ("Good" is the memory fit, not a rating), and the line saying Codeman cannot download or install yet sits right under the pick and its notes, in `--text-dim` like every other line. The dismissal is a per-device localStorage key named for this stage (`codeman:local-model-dismissed:readonly`), never a server setting: the stage that can set a model up takes a new key so its offer shows once more. At most one request a minute (the server caches its probe for 60 s), shared while in flight, and a failed refresh keeps the answer on screen. Tests: `test/local-model-ui.test.ts`, `test/local-model-i18n.test.ts`. +**Run a model on this Mac** (`local-model-ui.js`; welcome button `#welcomeLocalModelBtn`, chooser window `#localModelModal`, App Settings > Models row `#localModelSettings`; server rules in the root CLAUDE.md, Local model): ONE window does everything (choose, confirm, progress, paused, failed, installed, no recommendation); the button and the Settings row only open it. Which screen shows follows `install.state` of `GET /api/local-model`; the page itself holds only the selection and whether the person is on the chooser or the confirm screen. ⚠️ The window is ONE frame for the whole flow: a fixed height on desktop (`.modal-content.local-model-window`), App Settings' boxed close button, the body scrolling between the header and a foot of its own (`#localModelWindowFoot`): every screen's last part (`.lm-foot`, or the `.lm-remove` question) is moved there by `_renderLocalModelWindow()`, so a button never moves away from under the pointer and is never clipped; a test that looks for a button looks in the frame, not the body. The short screens are centred in that frame on desktop and sit above the buttons on a phone (`data-screen` on the body). ⚠️ The memory bar (`.lm-bar`, `role="meter"`) is the one strong element: filled to the selected row's `budgetShare`, the model at its PEAK (the legend says so, and the installed line uses the same words beside what the model takes now, so the two figures are never read as a contradiction), a flat `--accent` fill on a `--text-muted` track (about 3:1 against the window, so the free part can be seen; a border token is 1.5:1 and the inputs' `--bg-input` is the window's own surface on the dark skins), 10px, no gradient or glow, and its width transition is the only motion (off under `prefers-reduced-motion`); a repaint starts each bar at the last width of its kind, so keep `_settleLocalModelBars()` after every body write. The figure beside it is an estimate and says "about" in whole GB (`localModelWholeGb()` on the server's text); only the download figure keeps tabular figures. ⚠️ The rows are the server's pick first, then the other models by download size, largest first (`localModelCandidates()`): the server's alternatives are its ranking, which no column shows. ⚠️ The download bar means ONE thing, the model download: at the start while the runtime is fetched, gone once the model is in (the step list carries verify, agent and connect); the first paint after an install finished shows "Start a session" alone (`justInstalled`), an error on an installed model is the headline with a grey dot, and Remove is quiet everywhere but red in its own question, never the accent. ⚠️ The window polls the GET once a second WHILE OPEN and never otherwise (no SSE event): `closeLocalModelWindow()` clears the timer, a background tab asks nothing, an unchanged answer repaints nothing, and a tick that moved only the download replaces `.lm-meter` alone, so a button is never swapped out under a click. A read that was already under way when a verb answered is dropped (`_localModelVerbs`), or the old state flashes back. ⚠️ The consent is bound to the size shown: the install request carries the row's `installBytes` as `acknowledgedBytes`, and a 409 `size-changed` shows `expectedGb` and sends `expectedBytes` next; never compute a total here. ⚠️ A button exists only for an action in `install.actions`, and after an error exactly one helpful action (`error.next`, already adapted by the server to the state: `start` on an installed model, `retry` on a job) plus a quiet Remove; an error's text is split after its first sentence (`localModelSentences()`, which knows the ideographic full stop) and only that sentence is the headline. A refusal the server names (a 400, a full disk) carries `data.message` like a 409 and is printed from it, never from the English `error` text. ⚠️ Every dynamic string goes in through `textContent` (model, coding agent and chip names, an error's detail, session names, the folder and the install command are someone else's text); the window body, the button and the row carry `data-i18n-skip` and the module translates its own sentences at render time, so a language or display-name change repaints from the module (the `` observer). ⚠️ Labels and sentences come from the payload (`{ text, vars }` through `localModelT`); the module holds no list of models or coding agents, compares no CLI id (the agent select's values are positions in the server's list, the id travels only as data into the install request and `selectCustomModelEntry`) and never contains the word "Recommended". ⚠️ `test/local-model-i18n.test.ts` harvests every `localModelT('...')` literal: a new string is a single-quoted literal with no apostrophe and gets a zh-CN entry, the only non-literal calls are the two payload ones, and a unit alone (`23.1 GB`) is not a translator call. ⚠️ Sizes are printed as sent (decimal GB text), bytes are never turned into a size; the installed memory alone is `memClassGb`. ⚠️ The model list is a real radio group (`role="radiogroup"`, roving `tabindex`, arrows, Home, End, the focus follows through the repaint's own focus restore, keyed by `data-lm-key`); the columns line up through `subgrid` with a fixed-track `@supports` fallback, and under 560px a row is two lines (mobile.css). ⚠️ The window has its OWN `FocusTrap` (`_localModelFocusTrap`, never `activeFocusTrap`: it opens on top of App Settings) and app.js claims its Escape before `closeAllPanels()`. ⚠️ Its buttons are the Git worktrees dialog's `.wt-btn` set, never `.btn-toolbar`: the skin block greys `.btn-toolbar.btn-primary` and the phone block pins every `.btn-toolbar` to 26px with `!important`. ⚠️ "Start a session" re-reads the settings and the CLI list first (`loadAppSettingsFromServer`, `loadCliListForSettings`: the server switched the endpoints flag on and may have installed the agent a moment ago), then calls the existing `selectCustomModelEntry(install.cli, install.endpointId)`. The button ships `hidden` and shows only once the server answered for a Mac ("Local model" once one is installed); button AND Settings group stay hidden for `not-macos`, for a refusal and while the first request is in flight; refused means `403` with `errorCode: FORBIDDEN` (the auth layer's `403 PASSWORD_CHANGE_REQUIRED` is a failed request, asked again on `codeman:me`). Remove asks first and deletes the downloaded model unless the box is cleared (nothing else in the UI can free that space later). Not on the phone overview. Tests: `test/local-model-ui.test.ts`, `test/local-model-i18n.test.ts`; the visual walk is a scratch Playwright script, not a test. **Home-screen session order** (`CodemanSessionOrder` in constants.js, pure): BOTH home screens (phone overview, desktop rail) must order rows through this ONE comparator. Rank `needs` → `error` → `waiting` → `working` → `idle` → `done`. ⚠️ The tiebreak flips: states a session is still IN sort oldest-first, states it has STOPPED sort newest-first. ⚠️ The running group keys off `lastSubmitAt`, never `lastActivityAt` (a working pane repaints constantly). ⚠️ A 0 stamp means unknown and sorts last within its state. Final tiebreak is `orderIndex`, so the list never shuffles. The tab strip itself is NOT sorted by this. Test: `test/session-overview-order.test.ts`. → [architecture-invariants#home-screen-session-order](../../docs/architecture-invariants.md#home-screen-session-order) diff --git a/src/web/public/app.js b/src/web/public/app.js index 69b46af8..3257792e 100644 --- a/src/web/public/app.js +++ b/src/web/public/app.js @@ -1468,6 +1468,12 @@ class CodemanApp { this.handleTabRailSearchKeydown(e); return; } + // The local model window can sit on top of App Settings (it opens from a + // row there): Escape closes the window alone and leaves Settings open. + if (this.isLocalModelWindowOpen?.()) { + this.closeLocalModelWindow(); + return; + } this.closeAllPanels(); this.closeHelp(); if (this.attachmentHistoryDrawerOpen) this.closeAttachmentHistory(); diff --git a/src/web/public/i18n.js b/src/web/public/i18n.js index 13d60637..f75a0ecc 100644 --- a/src/web/public/i18n.js +++ b/src/web/public/i18n.js @@ -1415,46 +1415,109 @@ '该 tmux 会话在此处打开时所在的位置。它是在 Codeman 之外启动的，关闭此标签页后仍会继续运行。', 'Git status is not available for a tmux session opened outside Codeman.': '在 Codeman 之外打开的 tmux 会话无法使用 Git 状态。', - // Model on this Mac (local-model-ui.js: the welcome card and its App Settings group). The first - // block is the module's own text; sizes and names fill the {placeholder} slots. + // Run a model on this Mac (local-model-ui.js: the welcome button, the chooser window and its + // App Settings group). The first block is the page's and the module's own text; sizes and + // names fill the {placeholder} slots. 'Model on this Mac': '此 Mac 上的模型', - 'What the Mac that Codeman runs on can run as a local coding model. Read-only: nothing is downloaded or installed from here.': - '运行 {name} 的这台 Mac 可以运行哪种本地编程模型。只读：此处不会下载或安装任何内容。', + 'Run a coding model on the Mac that Codeman runs on. No account and no API key.': + '在运行 {name} 的这台 Mac 上运行编程模型。无需账号，也无需 API 密钥。', + 'Run a model on this Mac': '在此 Mac 上运行模型', + 'Run a local model': '运行本地模型', + 'Local model': '本地模型', 'The Mac that Codeman runs on, which is not always the device you are reading on.': '指运行 {name} 的那台 Mac，不一定是你正在使用的这台设备。', - Hide: '隐藏', - 'Hide this card on this device. It stays in App Settings, under Models.': - '在此设备上隐藏此卡片。它仍保留在“应用设置”的“模型”中。', '{chip}, {memory} GB memory, {free} GB free': '{chip}，内存 {memory} GB，可用磁盘空间 {free} GB', '{chip}, {memory} GB memory': '{chip}，内存 {memory} GB', '{memory} GB memory, {free} GB free': '内存 {memory} GB，可用磁盘空间 {free} GB', '{memory} GB memory': '内存 {memory} GB', - '{model} ({id}) with {harness}': '{model}（{id}），搭配 {harness}', - 'Context {context} tokens': '上下文 {context} 个 Token', - '{need} GB of memory': '占用内存 {need} GB', - '{download} GB download': '下载 {download} GB', - 'Needs about {need} GB of the {budget} GB this Mac can give a model.': - '大约需要 {need} GB，此 Mac 可分配给模型的内存为 {budget} GB。', - 'Needs about {need} GB of the {budget} GB this Mac can give a model (estimated).': - '大约需要 {need} GB，此 Mac 可分配给模型的内存约为 {budget} GB（估算）。', - 'About {speed} tokens per second (estimated).': '每秒约 {speed} 个 Token（估算）。', - '{harness} is not installed on this Mac yet.': '此 Mac 上尚未安装 {harness}。', - 'With {harness}, the first answer can take minutes.': '使用 {harness} 时，第一次回答可能需要几分钟。', - 'about {speed} tokens per second': '每秒约 {speed} 个 Token', - // What the fit and speed labels measure (their tooltip and accessible name). - 'Memory fit: needs {share}% of what this Mac can give a model': - '内存占用：需要此 Mac 可分配给模型的内存的 {share}%', - 'No speed estimate for this model on this chip': '此芯片上没有该模型的速度估算', - 'Other models that fit': '其他装得下的模型', - 'Codeman cannot download or install this for you yet. That is the next step, and it is not built yet.': - '{name} 目前还不能为你下载或安装它。这是下一步的功能，尚未实现。', + // The App Settings row: where the install stands, in a few words. + 'No model installed': '尚未安装模型', + 'Installing {model}': '正在安装 {model}', + 'The install of {model} is paused': '{model} 的安装已暂停', + 'The install of {model} stopped': '{model} 的安装已中止', + '{model} is running': '{model} 正在运行', + '{model} is installed and stopped': '{model} 已安装，当前已停止', + 'Checking this Mac': '正在检查此 Mac', + // The chooser: the memory bar, the list, the coding agent. + 'About {need} of {budget} GB of memory': '约需 {need} GB 内存，共 {budget} GB 可用', + estimated: '估算', + 'The selected model needs {share}% of the memory this Mac can give a model': + '所选模型需要此 Mac 可分配给模型的内存的 {share}%', + 'The model at its peak, with all {context} tokens of context in use': + '模型的峰值占用（{context} 个 Token 的上下文全部用满时）', + 'What stays free': '剩余可用', + 'Models that fit this Mac': '此 Mac 装得下的模型', + Speed: '速度', + Download: '下载', + 'speed not known': '速度未知', + '{speed} tokens/s': '{speed} Token/秒', + 'about {speed} tokens/s': '约 {speed} Token/秒', + 'Runs with': '搭配', + 'Not on this Mac yet. Codeman installs it for you.': '此 Mac 上尚未安装，{name} 会为你一并安装。', + 'The first answer can take minutes.': '第一次回答可能需要几分钟。', + 'No account and no API key. The model runs only on this Mac.': '无需账号，也无需 API 密钥。模型只在此 Mac 上运行。', + 'Install on this Mac': '安装到此 Mac', + // The confirm screen: the download as a table, where it goes. + 'What will be downloaded': '将下载的内容', + 'At {context} tokens of context, with {cli}': '上下文 {context} 个 Token，搭配 {cli}', + 'Ollama {version}': 'Ollama {version}', + 'The program that runs the model': '用于运行模型的程序', + Total: '合计', + 'Not on this Mac yet. Its own installer fetches it with this command:': + '此 Mac 上尚未安装。它自带的安装程序会用以下命令获取它：', + 'Not on this Mac yet. Its own installer fetches it.': '此 Mac 上尚未安装。它自带的安装程序会获取它。', + 'Goes into': '存放位置：', + 'No administrator password.': '无需管理员密码。', + 'Download and install': '下载并安装', + // Progress, and the same picture paused. + '{done} of {total} GB': '已下载 {done} GB，共 {total} GB', + '{speed} MB per second, {left}': '每秒 {speed} MB，{left}', + '{speed} MB per second': '每秒 {speed} MB', + 'less than a minute left': '剩余不到 1 分钟', + 'about 1 minute left': '剩余约 1 分钟', + 'about {minutes} minutes left': '剩余约 {minutes} 分钟', + 'about 1 hour left': '剩余约 1 小时', + 'about {hours} hours left': '剩余约 {hours} 小时', + 'Getting the runtime': '获取运行时', + 'Downloading the model': '下载模型', + 'Loading and checking': '加载并检查', + 'Installing {cli}': '安装 {cli}', + Connecting: '正在连接', + 'Download of the model': '模型下载进度', + '0 of {total} GB': '已下载 0 GB，共 {total} GB', + 'You can close this window. The download continues while Codeman is running.': + '你可以关闭此窗口。只要 {name} 在运行，下载就会继续。', + 'You can close this window. The install continues while Codeman is running.': + '你可以关闭此窗口。只要 {name} 在运行，安装就会继续。', + 'The download waits until you continue.': '下载会等你点击继续后再进行。', + 'The install waits until you continue.': '安装会等你点击继续后再进行。', + Continue: '继续', + 'Choose another model': '选择其他模型', + // After an error: the one action that helps. + 'Use another port': '改用其他端口', + 'Try {model}': '试试 {model}', + Start: '启动', + 'The server did not accept that.': '服务器未接受该请求。', + // Installed, and the Remove question. + '{model} is ready on this Mac.': '{model} 已在此 Mac 上准备就绪。', + '{model} is installed on this Mac.': '{model} 已安装在此 Mac 上。', + 'Runs with {cli}.': '搭配 {cli} 运行。', + 'Context {context} tokens.': '上下文 {context} 个 Token。', + 'Takes {memory} GB of memory now, and up to about {peak} GB at its peak, with all of the context in use.': + '目前占用 {memory} GB 内存；上下文全部用满时，峰值约为 {peak} GB。', + 'Takes {memory} GB of memory now.': '目前占用 {memory} GB 内存。', + 'All of it is on the GPU.': '全部位于 GPU 上。', + 'Start a session': '开始会话', + 'Remove the local model from this Mac?': '要从此 Mac 移除本地模型吗？', + 'Also delete the downloaded model': '同时删除已下载的模型', + 'The coding agent stays installed.': '编程智能体会保留，不会被卸载。', // The sentences and labels the SERVER picks (LOCAL_MODEL_LABELS in src/types/local-model.ts), keyed // by their exact English text: changing a text there means changing its key here. 'Best fit for this Mac': '最适合此 Mac 的选择', 'Recommended for this Mac': '为此 Mac 推荐', 'This Mac can run a small model. It handles small edits and questions. It does not replace a cloud model.': '此 Mac 可以运行小型模型。它能处理小幅修改和问答，但不能替代云端模型。', - 'Not yet tested by us on a Mac like this.': '我们尚未在类似的 Mac 上测试过。', + 'Codeman has not run this on a Mac like this yet.': '{name} 尚未在同类 Mac 上运行过它。', 'The best fit is a small model. It handles small edits and questions. It does not replace a cloud model.': '最适合的是一个小型模型。它能处理小幅修改和问答，但不能替代云端模型。', '{model} fits the memory of this Mac too, but its download needs about {missing} GB more free disk space.': @@ -1465,20 +1528,75 @@ 'Speed unknown': '速度未知', 'Codeman recommends a local model only on a Mac with Apple silicon.': '{name} 仅在搭载 Apple 芯片的 Mac 上推荐本地模型。', - 'This Mac has an Intel processor. Local coding models need Apple silicon.': - '此 Mac 使用 Intel 处理器。本地编程模型需要 Apple 芯片。', - 'This Mac has {memory} GB of memory. That is too little to run a model that can drive a coding agent, so Codeman recommends none here.': - '此 Mac 有 {memory} GB 内存，不足以运行能够驱动编程智能体的模型，因此 {name} 在此不推荐任何模型。', - 'Codeman estimates that this Mac can give a model about {budget} GB of memory. The smallest model a coding agent can work with needs about {need} GB. That leaves no safe margin, so Codeman recommends none here.': - '据 {name} 估算，此 Mac 可分配给模型的内存约为 {budget} GB。编程智能体能够使用的最小模型约需 {need} GB。这样没有留出安全余量，因此 {name} 在此不推荐任何模型。', + 'Local coding models need Apple silicon, and this Mac has an Intel processor.': + '本地编程模型需要 Apple 芯片，而此 Mac 使用的是 Intel 处理器。', + 'With {memory} GB of memory, this Mac cannot run a model that can drive a coding agent.': + '此 Mac 只有 {memory} GB 内存，无法运行能够驱动编程智能体的模型。', + 'This Mac can give a model about {budget} GB of memory, and the smallest model a coding agent can work with needs about {need} GB.': + '此 Mac 可分配给模型的内存约为 {budget} GB，而编程智能体能够使用的最小模型约需 {need} GB。', 'Codeman could not read how much memory this Mac has, so it recommends no model here.': '{name} 无法读取此 Mac 的内存大小，因此在此不推荐任何模型。', - 'Not enough free disk space. The smallest model that fits this Mac is a {download} GB download, and {reserve} GB must stay free after it. {free} GB is free now. Free up about {missing} GB.': - '可用磁盘空间不足。适合此 Mac 的最小模型需要下载 {download} GB，下载后还必须保留 {reserve} GB 的可用空间。当前可用空间为 {free} GB。请腾出约 {missing} GB。', - 'This Mac runs macOS {version}. Local models need macOS {minimum} or newer.': - '此 Mac 运行的是 macOS {version}。本地模型需要 macOS {minimum} 或更高版本。', + 'Free up about {missing} GB of disk space to install the smallest model that fits this Mac.': + '请腾出约 {missing} GB 磁盘空间，以便安装适合此 Mac 的最小模型。', + 'Local models need macOS {minimum} or newer, and this Mac runs macOS {version}.': + '本地模型需要 macOS {minimum} 或更高版本，而此 Mac 运行的是 macOS {version}。', 'No coding agent that Codeman can connect to a local model is available on this Mac.': '此 Mac 上没有 {name} 可以连接到本地模型的编程智能体。', + // The tag of a model row (LOCAL_MODEL_TAG_LABELS in src/types/local-model.ts). Tight and Slow are + // the entries above. + 'Tested on a Mac like this': '已在同类 Mac 上测试', + 'Best fit': '最适合', + 'Best fit, tight': '最适合，内存偏紧', + 'Small model': '小型模型', + // The sentences of the install (LOCAL_MODEL_INSTALL_LABELS in src/types/local-model-install.ts): + // why a job stopped, why a request was refused, and the one sentence about the developer tools. + 'The download of the runtime stopped and could not be continued. Check the network connection, then try again.': + '运行时的下载已中断，且无法继续。请检查网络连接，然后重试。', + 'The runtime download was damaged and has been deleted. Try again.': '下载的运行时已损坏，已被删除。请重试。', + 'The runtime could not be unpacked on this Mac.': '无法在此 Mac 上解压运行时。', + 'The runtime did not start on this Mac.': '运行时未能在此 Mac 上启动。', + 'The runtime keeps stopping, so Codeman no longer restarts it.': '运行时反复停止，因此 {name} 不再重新启动它。', + 'The model on this Mac is not running. Start it first.': '此 Mac 上的模型没有在运行。请先启动它。', + 'Port {port} is in use by another program.': '端口 {port} 已被其他程序占用。', + 'The disk is full. Free up space, then try again.': '磁盘已满。请腾出空间，然后重试。', + 'The download keeps stopping. The network may be damaging data. Try again or use another network.': + '下载反复中断。网络可能正在损坏数据。请重试，或换一个网络。', + '{model} is no longer offered under this name. Update Codeman or pick another model.': + '{model} 已不再以此名称提供。请更新 {name}，或选择其他模型。', + '{model} does not fit in the memory of this Mac after all.': '{model} 终究装不进此 Mac 的内存。', + '{model} loaded with a context of {actual} tokens instead of {wanted}.': + '{model} 加载时的上下文为 {actual} 个 Token，而不是 {wanted} 个。', + '{model} did not load on this Mac.': '{model} 未能在此 Mac 上加载。', + '{model} loaded but did not call a tool, so a coding agent cannot work with it.': + '{model} 已加载，但没有调用工具，因此编程智能体无法使用它。', + 'The check of {model} did not finish.': '对 {model} 的检查没有完成。', + '{cli} could not be installed.': '无法安装 {cli}。', + 'A saved model connection named {id} already exists and was not created here. Rename or delete it in App Settings, then try again.': + '已存在名为 {id} 的已保存模型连接，而它不是在这里创建的。请在“应用设置”中重命名或删除它，然后重试。', + 'Codeman could not save the connection to the model.': '{name} 无法保存与模型的连接。', + 'The model on this Mac changed since it was installed. Remove it and install it again.': + '此 Mac 上的模型在安装后发生了变化。请移除后重新安装。', + 'The install stopped on an unexpected error.': '安装因意外错误而中止。', + 'An install is already running.': '已有一个安装正在进行。', + 'A model is already installed on this Mac. Remove it first.': '此 Mac 上已安装了一个模型。请先移除它。', + 'The download is {size} GB now. Check the sizes and confirm again.': + '现在的下载大小为 {size} GB。请核对大小后再次确认。', + 'Sessions are still using this model: {sessions}. Close them first.': + '仍有会话在使用此模型：{sessions}。请先关闭它们。', + 'That is not possible in the current state.': '当前状态下无法执行此操作。', + 'This model at this context is not offered for this Mac right now.': + '目前不为此 Mac 提供这一上下文大小下的该模型。', + 'This coding agent is not offered for this model.': '该模型不提供这个编程智能体。', + '{cli} is not installed and cannot be installed from here.': '{cli} 尚未安装，也无法从这里安装。', + 'Only {model} is installed on this Mac. Remove it and install another model first.': + '此 Mac 上只安装了 {model}。请先移除它，再安装其他模型。', + // What the endpoint routes answer for the record of the model on this Mac (shown as a toast). + 'This endpoint belongs to the model Codeman runs on this Mac. Change or remove it in the local model window.': + '此端点属于 {name} 在此 Mac 上运行的模型。请在本地模型窗口中更改或移除它。', + 'This endpoint id is reserved for the model Codeman runs on this Mac. Pick another id.': + '此端点标识已保留给 {name} 在此 Mac 上运行的模型。请换一个标识。', + 'Coding agents use git and Python. This Mac has no developer tools yet, so macOS will offer to install them the first time an agent needs one.': + '编程智能体会用到 git 和 Python。此 Mac 尚未安装开发者工具，因此在智能体首次需要时，macOS 会提示安装。', }); const ZH_CN_LOWER = new Map(Object.entries(ZH_CN).map(([key, value]) => [key.toLocaleLowerCase('en'), value])); diff --git a/src/web/public/index.html b/src/web/public/index.html index 8026b7ce..c3c502c0 100644 --- a/src/web/public/index.html +++ b/src/web/public/index.html @@ -511,6 +511,17 @@ contents on every state change and owns its inline display. -->
+ +
- - @@ -2876,20 +2879,24 @@ - + for a non-admin in multi-user mode (the routes are admin-only). Not gated on + the endpoints toggle above: an install switches that on itself. -->

Model on this Mac

-

What the Mac that Codeman runs on can run as a local coding model. Read-only: nothing is downloaded or installed from here.

+

Run a coding model on the Mac that Codeman runs on. No account and no API key.

-
+
+
+ +
@@ -3671,6 +3678,27 @@
+ + +