mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-15 02:49:33 +01:00
ci(local-model): check the model catalogue and the runtime pin every week
scripts/check-local-model-catalog.mjs reads every catalogue tag the way Ollama 0.40.15 on Apple silicon reads it: it asks for the manifest list, takes the mlx build when the tag offers one (else llamacpp, else ggml), and compares that build's pull total (layers plus config) with the row's `bytes`. It also reports a tag that is gone, a build whose `requires` is newer than the pinned runtime, a pinned digest that moved, a newer Ollama release, and a pinned ollama-darwin.tgz that is gone or changed. It never requests a layer, bounds every request, retries nothing and validates everything that reaches the report. Exit 0 clean, 1 drift, 2 failed. The plain Docker manifest of a list tag is the llamacpp build, not what a Mac pulls. Read the right way, the first run finds two rows sized from the wrong build: qwen3.5:9b (a Mac downloads 7,639,229,431 bytes, +16.6%) and qwen3.8:27b (18,174,721,847, +2.4%). .github/workflows/local-model-catalog.yml runs it Mondays at 06:00 UTC and on dispatch. The check job installs with --ignore-scripts and holds a read-only token; a separate job with issues: write and no checkout opens or updates the one "Local model catalogue: weekly check" issue on drift, comments on it on a clean week when it is open, and is skipped when the check failed. Actions are pinned by commit. docs/local-models.md gains "Keeping the catalogue current". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
be8fc2980a
commit
d12bef0833
8 files changed
+1577
No files matched your search
@@ -250,6 +250,62 @@ or `python3`.
|
||||
- **The model stays in memory for ten minutes** after the last request, then the runtime
|
||||
unloads it.
|
||||
|
||||
## Keeping the catalogue current
|
||||
|
||||
The catalogue (`src/config/local-model-catalog.ts`) and the runtime pin are compiled into
|
||||
Codeman on purpose: no file or download decides what is recommended or run. What the
|
||||
registry serves can still move away from them: a tag is pushed again or removed, the build
|
||||
a Mac downloads changes size, Ollama ships a new version. The workflow
|
||||
`.github/workflows/local-model-catalog.yml` looks for that every Monday at 06:00 UTC. It
|
||||
runs `scripts/check-local-model-catalog.mjs`, which also runs locally:
|
||||
`npm run check:local-model-catalog` prints the report and exits 0 (clean), 1 (drift) or 2
|
||||
(the check itself failed).
|
||||
|
||||
For every model row it checks:
|
||||
|
||||
- **The tag still exists** in the registry.
|
||||
- **The size of what a Mac downloads** is within 0.1% of the row's `bytes`. A tag can be a
|
||||
manifest list with one build per runner, and Ollama on Apple silicon takes the `mlx` build
|
||||
when there is one, else `llamacpp`, else `ggml`. So the check asks for the tag the way
|
||||
Ollama asks, picks the build a Mac picks and sizes it the way a pull counts it (every layer
|
||||
plus the config). The plain manifest other tools get for the same tag is the `llamacpp`
|
||||
build, which is not what a Mac downloads.
|
||||
- **The build a Mac pulls runs on the pinned runtime**: its config declares no newer
|
||||
Ollama (`requires`) than `OLLAMA_RUNTIME_PIN`.
|
||||
- **The pinned digest**, once a row carries one: a Mac would still pull that manifest. The
|
||||
catalogue pins none yet, so the report lists today's digests, the one to pin being what
|
||||
`/api/tags` reports after the pull.
|
||||
|
||||
For the runtime pin it checks whether a newer Ollama release exists, and whether the pinned
|
||||
`ollama-darwin.tgz` is still on GitHub with the pinned size and sha256.
|
||||
|
||||
It reads manifests and configs only, never a model layer. It does not check the memory
|
||||
figures, the speeds or whether a coding agent works with a model: no manifest says anything
|
||||
about those, and they change only when somebody measures on a Mac.
|
||||
|
||||
**The issue.** Drift opens one issue, "Local model catalogue: weekly check", or, while that
|
||||
issue is open, replaces its description with the new report and adds a comment. A clean week
|
||||
comments on the open issue and does nothing when there is none. A check that failed (the
|
||||
network, a rate limit, an answer that did not parse) fails the run and leaves the issue
|
||||
alone; what it found is in the run's summary. The schedule runs the copy of the workflow on
|
||||
`master`, and so does `gh workflow run local-model-catalog.yml`.
|
||||
|
||||
**What a maintainer does with it**, in a pull request each time:
|
||||
|
||||
- **Size drift or a changed digest**: correct the row. When the build itself changed (another
|
||||
runner or quantization), its memory figures and speed no longer describe it: mark the row
|
||||
`estimated` until somebody measures it again.
|
||||
- **A tag that is gone, or a build that needs a newer runtime**: replace or drop the row, or
|
||||
move the runtime pin first.
|
||||
- **A newer Ollama**: read its release notes, then move `OLLAMA_RUNTIME_PIN` (version, URL,
|
||||
size and sha256, checked against the release's `sha256sum.txt`) and run an install on a
|
||||
Mac before merging.
|
||||
- **A pinned archive that changed**: a security question first. Every install refuses it at
|
||||
the hash compare, so nobody gets it, but find out why before moving the pin.
|
||||
|
||||
Close the issue once the fix is merged. A catalogue or pin change reaches users only in a
|
||||
release: an installed Codeman keeps the catalogue it was built with until it is updated.
|
||||
|
||||
## Source
|
||||
|
||||
| What | Where |
|
||||
@@ -265,3 +321,4 @@ or `python3`.
|
||||
| Routes | `src/web/routes/local-model-routes.ts` |
|
||||
| Shared CLI install runner | `src/cli-install-runner.ts` |
|
||||
| Serialized writer of the endpoint store | `src/custom-model-hosts-writer.ts` |
|
||||
| Weekly catalogue check | `scripts/check-local-model-catalog.mjs`, `.github/workflows/local-model-catalog.yml` |
|
||||
Reference in new issue
Block a user