|
|
|
@@ -0,0 +1,115 @@
|
|
|
|
|
# Local Models
|
|
|
|
|
|
|
|
|
|
Run a coding model on your own Mac, next to the cloud models you already use. Codeman
|
|
|
|
|
installs the model, keeps it running, and lets you start sessions on it or hand it work from
|
|
|
|
|
a cloud session. No account and no API key: the model runs only on that Mac.
|
|
|
|
|
|
|
|
|
|
This is **experimental**, and it needs a Mac with Apple silicon as the machine the Codeman
|
|
|
|
|
server runs on. On a machine that is not a Mac the button stays hidden.
|
|
|
|
|
|
|
|
|
|
## Install a model
|
|
|
|
|
|
|
|
|
|
1. Press **Run a local model** on the welcome screen (or open App Settings > Models > Model
|
|
|
|
|
on this Mac).
|
|
|
|
|
2. The window shows which models fit this Mac. The bar at the top is the memory the Mac can
|
|
|
|
|
give a model, filled to what the selected model needs at its peak. A nearly full bar is a
|
|
|
|
|
tight fit.
|
|
|
|
|
3. Pick a model, and under **Runs with** the coding agent to start it with.
|
|
|
|
|
4. Press **Install on this Mac**. The next screen lists every download with its size and
|
|
|
|
|
where it goes. Nothing is downloaded before you confirm there.
|
|
|
|
|
5. Codeman downloads a model runtime and the model into `~/.codeman/local-model`, loads the
|
|
|
|
|
model, checks that all of it runs on the GPU and that it can call a tool, and installs
|
|
|
|
|
the coding agent if it is missing. No administrator password is asked for.
|
|
|
|
|
6. Press **Start a session**.
|
|
|
|
|
|
|
|
|
|
You can close the window while it downloads. **Cancel** pauses the install and **Continue**
|
|
|
|
|
picks it up where it stopped. After a Codeman restart an unfinished install waits for your
|
|
|
|
|
click.
|
|
|
|
|
|
|
|
|
|
## Pick the coding agent
|
|
|
|
|
|
|
|
|
|
A model is driven by a coding agent: the program that reads your prompt, calls tools and
|
|
|
|
|
edits files. You choose it **per session**. Open the window again and look at **Start a
|
|
|
|
|
session with**: one chip per agent the model can be run with.
|
|
|
|
|
|
|
|
|
|
| Agent | Pick it when |
|
|
|
|
|
| ---------------- | ---------------------------------------------------------------------------------------------- |
|
|
|
|
|
| OpenCode | You have no reason to choose. It is the pair Codeman was tested with on a Mac |
|
|
|
|
|
| Pi | You want the first answer soon. Its fixed prompt is the smallest by far, which also leaves the most room for your work |
|
|
|
|
|
| Claude Code | You want Claude Code's own tools and habits. Its first answer on a local model takes minutes |
|
|
|
|
|
| DeepSeek Harness | You already use it and have it set up |
|
|
|
|
|
|
|
|
|
|
Select a chip and press **Start a session**. The choice is remembered on that device.
|
|
|
|
|
|
|
|
|
|
**A chip that says "Not installed"** is an agent Codeman can install for you. Select it: the
|
|
|
|
|
window shows the exact command it would run, and the main button becomes **Install** with
|
|
|
|
|
the agent's name. The model stays ready while the installer runs, and you can start a
|
|
|
|
|
session with another agent meanwhile. If the install fails, the window says so under the
|
|
|
|
|
chip and nothing else changes.
|
|
|
|
|
|
|
|
|
|
**A chip that says "Needs setup"** is an agent that needs more than one command. DeepSeek
|
|
|
|
|
Harness is the one today: set it up from the **Run** menu first, then it can be started
|
|
|
|
|
here. A new DeepSeek Harness also opens its own setup guide in every session until you have
|
|
|
|
|
finished it once.
|
|
|
|
|
|
|
|
|
|
## More than one model
|
|
|
|
|
|
|
|
|
|
Once a model is installed, the window lists **Models on this Mac** with what each takes on
|
|
|
|
|
disk, and the total under the list.
|
|
|
|
|
|
|
|
|
|
- **Add a model** opens the list of models again and downloads another one. When it is
|
|
|
|
|
ready it becomes the model in use; the earlier one stays on the Mac.
|
|
|
|
|
- **Use** switches to a model that is already there. Nothing is downloaded: Codeman loads it
|
|
|
|
|
and checks it again.
|
|
|
|
|
- **Delete** removes one model, after one question that says how much space it frees.
|
|
|
|
|
|
|
|
|
|
One model runs at a time. Adding and switching are refused while sessions still use the
|
|
|
|
|
current model: close those sessions first. The model in use cannot be deleted: switch first,
|
|
|
|
|
or use **Remove**, which takes the runtime and the connection away (and the downloaded
|
|
|
|
|
models too, if you tick that).
|
|
|
|
|
|
|
|
|
|
## Cloud and local together
|
|
|
|
|
|
|
|
|
|
The reason to keep a local model next to a cloud one: the cloud model plans and reviews, the
|
|
|
|
|
Mac does the bulk. From a Claude Code session in Codeman, with the
|
|
|
|
|
[Codeman skill](Codeman-Skill) (on by default), you can just say it:
|
|
|
|
|
|
|
|
|
|
| You say | What happens |
|
|
|
|
|
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
|
|
|
|
|
| "Is there a local model on this Mac?" | It reads what is installed, which agents can run it and which models are on the Mac. Read-only |
|
|
|
|
|
| "Use the local model to summarize every file in `docs/`, then check its summaries." | A worker on the local model does the reading; the cloud session reads its answer back and reviews it |
|
|
|
|
|
| "Spawn a local worker with Pi to draft tests for `src/parser.ts`." | The worker runs the local model through Pi |
|
|
|
|
|
| "Start two cloud workers for the API and the UI, and a local one for the changelog." | One mixed fleet: cloud workers side by side, the Mac as a queue |
|
|
|
|
|
|
|
|
|
|
The worker sessions appear as tabs in the dashboard and are closed when the work is done.
|
|
|
|
|
|
|
|
|
|
Good to know before you rely on it:
|
|
|
|
|
|
|
|
|
|
- **One request at a time.** The local model answers one request at a time, so two local
|
|
|
|
|
workers take turns. Fan out to the cloud, and give the Mac a queue.
|
|
|
|
|
- **Slower than a cloud model**, and the first request after a start loads the model first.
|
|
|
|
|
- **Review what comes back.** A local model is good at bulk work and weaker at judgment.
|
|
|
|
|
Treat its answer as a first draft.
|
|
|
|
|
- **The skill never installs anything.** A model or an agent that is missing is yours to
|
|
|
|
|
install in the window; the skill tells you what is missing and stops.
|
|
|
|
|
- **While a local worker is open**, the window refuses Stop, Remove and a switch of models,
|
|
|
|
|
and names the sessions that are in the way.
|
|
|
|
|
|
|
|
|
|
## Good to know
|
|
|
|
|
|
|
|
|
|
- **What was tested.** One pair was run on a real Mac: OpenCode with Qwen3.6 35B-A3B on an
|
|
|
|
|
M5 Pro with 48 GB. The window marks that pair on a Mac like it. Everything else in the
|
|
|
|
|
list is calculated from each model's own figures, and the install checks the model on your
|
|
|
|
|
Mac before it lets you start a session.
|
|
|
|
|
- **Memory first.** A model that does not fit in memory is not offered. An estimate becomes
|
|
|
|
|
a measurement the first time the runtime starts, and the model is judged against that
|
|
|
|
|
before it is downloaded.
|
|
|
|
|
- **The developer tools.** Coding agents use git and Python. On a Mac without them, macOS
|
|
|
|
|
offers to install them the first time an agent needs one.
|
|
|
|
|
- **Stop** keeps the runtime from starting with Codeman next time. **Start** undoes that.
|
|
|
|
|
- **In multi-user mode** only an admin sees and uses this.
|
|
|
|
|
|
|
|
|
|
More detail: [A model that runs on this Mac](Custom-Model-Endpoints#a-model-that-runs-on-this-mac)
|
|
|
|
|
for how it relates to your own model servers, and
|
|
|
|
|
[`docs/local-models.md`](https://github.com/Ark0N/Codeman/blob/master/docs/local-models.md)
|
|
|
|
|
for how it works inside.
|
Codeman maintainer