diff --git a/CLAUDE.md b/CLAUDE.md index f43a20cf..93e73e78 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -156,7 +156,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Auto-resume on usage limit** (opt-in per session, top of the Respawn tab): when Claude halts on a subscription limit, `usage-limit-patterns.ts` (pure, unit-tested) parses the reset time and `SessionAutoOps` arms a timer for reset+2min, then sends Esc + `continue`. ⚠️ Respawn cycles are blocked while paused (`isLimitPaused` guard in `onIdleDetected`), which is what prevents `/clear` from wiping the paused conversation. Claude-mode only. → [architecture-invariants#auto-resume-on-usage-limit](docs/architecture-invariants.md#auto-resume-on-usage-limit) -**Config backups** (`src/config-backup.ts`, `routes/config-backup-routes.ts`; `configBackupEnabled` absent = ON, `configBackupDir`, `configBackupKeepCount` 1-500 default 20, `configBackupKeepDays` 0-3650 default 30; `docs/wiki/Config-Backups.md`): timestamped snapshots `cfg-YYYYMMDD-HHMMSS/` (+ `manifest.json` with SHA-256s, `reason`, `appVersion`) of a FIXED allowlist (`CONFIG_BACKUP_FILES`: settings, clis, custom-model-hosts, webhook, intents, linked-cases, remote/docker hosts+cases (incl. `remote-cases.json`), webviews, push-keys, users, `.env`) into `/backups/config` or a custom absolute folder. ⚠️ `state.json` is deliberately NOT tracked (rewritten per session event; it also holds cron jobs and respawn presets). One snapshot per real content change (digest of the file set vs the latest manifest), at startup + a 5-minute timer, skipped in test mode. ⚠️ Retention (`pruneConfigBackups`) only deletes folders matching the name pattern AND holding a parseable manifest, and never the newest; a custom folder equal to the data dir falls back to the default. ⚠️ Restore verifies every file's SHA-256 BEFORE writing, takes a `pre-restore` snapshot, writes atomically at 0600 and only allowlisted names; the id is validated against the name pattern (no traversal). Snapshots are 0700/0600 because `.env`/`users.json`/`custom-model-hosts.json` carry credentials; all routes are admin-only in multi-user mode. ⚠️ A custom folder is probed with `probePathKind(dir, { pastCap: true })` before every pass and in each route (#516: it may be a network mount; `unknown` skips the pass / answers 503 `OPERATION_FAILED`), and a missing custom folder is only created when its parent is present. ⚠️ Each manifest records an `origin` (hostname + data dir); change check, prune, list, restore and diff are scoped to it so instances sharing one folder do not snapshot or prune each other. Restoring `intents.json` calls `intentStore.reload()`; `settings.json` applies its watcher toggles after a restart. → [docs/wiki/Config-Backups.md](docs/wiki/Config-Backups.md) +**Config backups** (`src/config-backup.ts`, `routes/config-backup-routes.ts`; `configBackupEnabled` absent = ON, `configBackupDir`, `configBackupKeepCount` 1-500 default 20, `configBackupKeepDays` 0-3650 default 30; `docs/wiki/Config-Backups.md`): timestamped snapshots `cfg-YYYYMMDD-HHMMSS/` (+ `manifest.json` with SHA-256s, `reason`, `appVersion`) of a FIXED allowlist (`CONFIG_BACKUP_FILES`: settings, clis, custom-model-hosts, webhook, intents, linked-cases, remote/docker hosts+cases (incl. `remote-cases.json`), webviews, push-keys, users, `.env`) into `/backups/config` or a custom absolute folder. ⚠️ `state.json` is deliberately NOT tracked (rewritten per session event; it also holds cron jobs and respawn presets). One snapshot per real content change (digest of the file set vs the latest manifest), at startup + a 5-minute timer, skipped in test mode. ⚠️ Retention (`pruneConfigBackups`) only deletes folders matching the name pattern AND holding a parseable manifest, and never the newest; a custom folder equal to the data dir falls back to the default. ⚠️ Restore verifies every file's SHA-256 BEFORE writing, takes a `pre-restore` snapshot, writes atomically at 0600 and only allowlisted names; the id is validated against the name pattern (no traversal). Snapshots are 0700/0600 because `.env`/`users.json`/`custom-model-hosts.json` carry credentials; all routes are admin-only in multi-user mode. ⚠️ A custom folder is probed with `probePathKind(dir, { pastCap: true })` before every pass and in each route (#516: it may be a network mount; `unknown` skips the pass / answers 503 `OPERATION_FAILED`), and a missing custom folder is only created when its parent is present. ⚠️ Each manifest records an `origin`, a random id kept in `/config-backup-id` (NOT the hostname: a Docker container gets a new one per recreate, which orphaned every snapshot); change check, prune, list, restore and diff are scoped to it so instances sharing one folder do not snapshot or prune each other. Restoring `intents.json` calls `intentStore.reload()`; `settings.json` applies its watcher toggles after a restart; Settings re-reads `/api/settings` into the form after a restore of it, since Save would otherwise PUT the old values back. ⚠️ Restoring `clis.json` writes it directly (the one writer besides `mutateRegistryFile()`) and then calls `reloadCliRegistry()`. → [docs/wiki/Config-Backups.md](docs/wiki/Config-Backups.md) **Plan-usage chip** (`showPlanUsageLimits`, per-device: desktop default **ON**, handhelds OFF): resolve DISPLAY only through `planUsageChipEnabled()` (settings-ui.js). The same setting is the server-side COLLECTION switch, read fresh by `readPlanUsageTelemetryEnabled()` (hooks-config.ts) at every claude create/respawn. ⚠️ An ABSENT key reads as ON in the reader; `GET /api/settings` must never write. ⚠️ A save sends `showPlanUsageLimits` ONLY when it flips the chip on that device (`planUsageCollectionFlip()`), or a phone switches collection off for every desktop. Claude data comes from the statusLine exporter, injected as an EPHEMERAL `claude --settings` flag (`resolveStatusLineCliCommand`), never written to disk, WRAPPING a user's own statusLine, posting to `POST /api/status-telemetry`. Codex comes from a read-only `account/rateLimits/read` poll (main bucket only). The statusline's `prompt_cache` object (Claude Code v2.1.251+) is kept by `StatusTelemetrySchema` (every documented scalar field, null-tolerant; `parsePromptCache()` in usage-telemetry.ts normalizes only the stored subset) and shown in the footer as `cache:until HH:MM` / `cache:cold` (only when the user has no statusline of their own) and in Session Options → Respawn (`#promptCacheStatus`, fed by `Session.promptCache` in `toState()` plus the change-only `session:promptCache` SSE event, worded by the pure `CodemanPromptCache.format`): warm-until/cold, the re-cache stake, the hit rate, the last miss's causes, and a `Context compacted at HH:MM` note after a compaction or a tool-result clearing (stamped on the first warm report whose `recache_tokens_if_cold` is null after one that carried it, never from `expected_rebuilds`, which Claude Code bumps only at the next request; NEVER an alert). ⚠️ `Session.setPromptCache()` stamps `compactedAt` onto a NEW object (never mutating the parsed report) and the route broadcasts the STORED object; the last report is restored on restart (constructor `promptCache`, like `displayModel`) and the readout is time-aware (a passed `expiresAt` reads cold, not stale-warm). It is the source of truth for the cache TTL, never the plan tier. → [architecture-invariants#plan-usage-chip-statusline-telemetry](docs/architecture-invariants.md#plan-usage-chip-statusline-telemetry), `docs/usage-limits-display-plan.md` @@ -172,7 +172,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Docker Compose deployment** (`docker/`): Codeman runs in a container and spawns Docker cases as **SIBLING** containers via the host socket, never nested. `resolveDockerDaemonMountSource()` maps HOME bind sources into the daemon's namespace (`CODEMAN_DOCKER_HOST_HOME`); `CODEMAN_CASES_PATH` makes workspaces resolve to the same absolute path on both sides. ⚠️ `CODEMAN_CASES_PATH` must move every consumer: resolve it only via `config/cases-dir.ts`. ⚠️ `.dockerignore` matches whole paths: keep `**/.env` or `docker/.env` secrets ship in the image. ⚠️ Long-form binds create missing sources ROOT-OWNED: `Start-Codeman.sh` pre-creates them, and `docker/entrypoint.sh` (root, `cap_add: [CHOWN, DAC_OVERRIDE, KILL, SETGID, SETUID]` against `cap_drop: ALL`, `KILL` for tini; pinned by the test) fixes ownership then drops to `PUID:PGID` via `setpriv`, never re-owning foreign dirs. ⚠️ Append `/opt/codeman-cli` and `~/.local/bin` to `PATH`, never prepend. ⚠️ `server.Dockerfile`, the compose file and `.env.example` feed the self-updater's environment gate (`docs/docker-self-update.md`). → [architecture-invariants#docker-compose-deployment](docs/architecture-invariants.md#docker-compose-deployment), `docs/docker-compose.md` -**CLI registry** (`src/config/cli-registry/`): every run mode is a `CliEntry` (discovery, launch argv template, env handling, `capabilities`, and the `overlays` behind remote/docker pane commands). **No code outside `stock.ts` may branch on a CLI id**: use a capability field or a NAMED PROFILE (`profiles.ts`); `test/cli-registry-no-id-branching.test.ts` and `test/frontend-cli-no-id-branching.test.ts` enforce it. ⚠️ Config holds typed argv tokens, never shell text; literals are validated at LOAD time and a bad one rejects the whole entry. ⚠️ Keep `external`, `hooks` and `altScreen` independent; never derive one from another. ⚠️ Config regexes (`discovery.version.regex`, `capabilities.workDetect.workingLine`, `capabilities.workDetect.watchingLine`, `capabilities.workDetect.awaitingLine`, `capabilities.workDetect.blockedLine`, `capabilities.modelDetect.screenLine`) must compile through `compileVersionRegex()`. ⚠️ `privilegedParams[].param` names a LAUNCH PARAM, not the legacy `Config` field (bridged only by `launch.legacyConfigAliases`); a wrong name silently clamps nothing. ⚠️ Resolve the registry AT CALL TIME, never in a module-level const. ⚠️ Remote claude/omp arms of `buildRemoteLaunchCommand` are not covered by the pane-command golden. `~/.codeman/clis.json` overrides entries. It is WRITTEN only by the opt-in CLI management routes (`cliManagementEnabled`, default OFF; `/api/clis`, `cli-registry-routes.ts`), and only through `mutateRegistryFile()` in `registry-writer.ts`, which serializes mutations and refuses (409) a file that does not parse or has group/world permission bits rather than overwriting it. Importing the registry still writes nothing. → [architecture-invariants#cli-registry](docs/architecture-invariants.md#cli-registry), `docs/cli-registry.md` +**CLI registry** (`src/config/cli-registry/`): every run mode is a `CliEntry` (discovery, launch argv template, env handling, `capabilities`, and the `overlays` behind remote/docker pane commands). **No code outside `stock.ts` may branch on a CLI id**: use a capability field or a NAMED PROFILE (`profiles.ts`); `test/cli-registry-no-id-branching.test.ts` and `test/frontend-cli-no-id-branching.test.ts` enforce it. ⚠️ Config holds typed argv tokens, never shell text; literals are validated at LOAD time and a bad one rejects the whole entry. ⚠️ Keep `external`, `hooks` and `altScreen` independent; never derive one from another. ⚠️ Config regexes (`discovery.version.regex`, `capabilities.workDetect.workingLine`, `capabilities.workDetect.watchingLine`, `capabilities.workDetect.awaitingLine`, `capabilities.workDetect.blockedLine`, `capabilities.modelDetect.screenLine`) must compile through `compileVersionRegex()`. ⚠️ `privilegedParams[].param` names a LAUNCH PARAM, not the legacy `Config` field (bridged only by `launch.legacyConfigAliases`); a wrong name silently clamps nothing. ⚠️ Resolve the registry AT CALL TIME, never in a module-level const. ⚠️ Remote claude/omp arms of `buildRemoteLaunchCommand` are not covered by the pane-command golden. `~/.codeman/clis.json` overrides entries. It is WRITTEN only by the opt-in CLI management routes (`cliManagementEnabled`, default OFF; `/api/clis`, `cli-registry-routes.ts`), and only through `mutateRegistryFile()` in `registry-writer.ts` (the one exception: a config-backup restore of `clis.json`, which then calls `reloadCliRegistry()`), which serializes mutations and refuses (409) a file that does not parse or has group/world permission bits rather than overwriting it. Importing the registry still writes nothing. → [architecture-invariants#cli-registry](docs/architecture-invariants.md#cli-registry), `docs/cli-registry.md` **External CLI modes (OpenCode, Codex, Gemini, Antigravity, Pi, Grok, DeepSeek, OMP, Copilot)**: `isExternalCliMode()` in `session.ts` gates Claude-specific behavior off (Ralph tracker, BashToolParser, token parsing, ❯ readiness; readiness is output stabilization); work detection is per-CLI `capabilities.workDetect` data, not this gate. All nine **require tmux, no direct PTY fallback** (secrets go via socket-scoped `tmux setenv`, never the command line). ⚠️ `run*()` in `session-ui.js` MUST unwrap the `{success,data}` envelope. ⚠️ **Codex uses predictive write-through echo, never the buffer overlay**: `_predictHookOnData` must never `return` (wire path stays byte-identical), and flushed text and a bracketed paste must go out as separate delayed writes. ⚠️ **Pi**: no bypass flag, never invent one; `approveProjectTrust` executes repo code, so it is in the clamp's **materialize** branch; never wire `--api-key`. ⚠️ **Grok**: `alwaysApprove` is stripped for non-granted owners (only-if-sent). ⚠️ **DeepSeek**: the agent is a PROFILE (Run gates on `isDeepSeekRunnable()`); the permission switch is the `DSH_PERMISSION_MODE` env var, so `clampEnvOverridesForOwner()` must DROP `DSH_PERMISSION_MODE`, `DSH_HOME` and `DEEPSEEK_BASE_URL` for non-granted owners; `hooksAvailableForMode()` is per-SESSION for it (pass `sessionHookOptions(session)`) and is never a stand-in for `mode === 'claude'`; answers come from `deepseek-transcript.ts`, paired by header `cwd` + boot window, never newest-mtime. ⚠️ **OMP**: `OMP_AUTH_BROKER_URL`/`_TOKEN` are clamped the same way. ⚠️ **Copilot**: `allowAll` (`--yolo`) is a `privilegedParam` and `COPILOT_HOME`/`COPILOT_ALLOW_ALL`/`COPILOT_PROVIDER_*`/`COPILOT_MODEL` are privileged env keys, and so is every other `COPILOT_*` variable the CLI reads that switches approvals, runs code with no approval row or sends its sign-in token elsewhere (the 1.0.98 list is in `stock.ts`; the prefix admits whatever a later release adds, so re-read the bundle whenever the measured version moves); the resume id must start with a letter or digit (`path-segment`), since `--resume`'s value is optional and `--resume --yolo` would otherwise make `--yolo` its own flag; `--name` is dropped next to `--resume`/`--continue` (Copilot refuses the pair); its mouse DECSETs are stripped (`strip-mux-and-mouse`) so a drag selects and the wheel is forwarded by `_shouldForwardWheelToApp` while `cliMouseTracking` is on; `composerReadyMark` is the footer hint `/ commands · ? help`, NOT `❯` (the folder-trust dialog draws `❯ 1. Yes`); `last-response` reads the pane's visible screen (`transcriptNeedsScreenCapture`) through a per-CLI `PANE_DIALECTS` row, because its stream is cursor moves, not lines (a prompt row scrolled off by a tall answer reads as the answer's tail unless a banner/trust dialog is on screen, and typed composer text is not a prompt); Docker seeds no Copilot credential (`config.json` holds a plain-text token without a keyring). → [architecture-invariants#external-cli-modes-opencode-codex-gemini-antigravity-pi-grok-deepseek-omp](docs/architecture-invariants.md#external-cli-modes-opencode-codex-gemini-antigravity-pi-grok-deepseek-omp) diff --git a/docs/api-reference.md b/docs/api-reference.md index 4e5b5e4b..dc2fa42a 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -1154,18 +1154,18 @@ The result carries server **names** only, never `env` values, `headers` or file ## Config backups -Timestamped snapshots of the files that hold what a user has customized (`src/config-backup.ts`). On by default; a snapshot is taken at startup and whenever the tracked files' content changes (checked every 5 minutes), so there is one entry per real change. All three routes are admin only in multi-user mode (`403`), because the files include credentials. Responses carry file names and sizes, never content. +Timestamped snapshots of the files that hold what a user has customized (`src/config-backup.ts`). On by default; a snapshot is taken at startup and whenever the tracked files' content changes (checked every 5 minutes), so there is one entry per real change. All four routes are admin only in multi-user mode (`403`), because the files include credentials. Responses carry file names and sizes; only the diff returns values, and never a credential's. -Tracked files (a fixed list, relative to the data dir): `settings.json`, `clis.json`, `custom-model-hosts.json`, `webhook.json`, `intents.json`, `linked-cases.json`, `remote-hosts.json`, `docker-hosts.json`, `docker-cases.json`, `webviews.json`, `push-keys.json`, `users.json`, `.env`. Session state (`state.json`, which also holds cron jobs and respawn presets), logs, the hook secret and caches are not. +Tracked files (a fixed list, relative to the data dir): `settings.json`, `clis.json`, `custom-model-hosts.json`, `webhook.json`, `intents.json`, `linked-cases.json`, `remote-hosts.json`, `remote-cases.json`, `docker-hosts.json`, `docker-cases.json`, `webviews.json`, `push-keys.json`, `users.json`, `.env`. Session state (`state.json`, which also holds cron jobs and respawn presets), logs, the hook secret and caches are not. -Settings (synced, in `settings.json`): `configBackupEnabled` (absent = on), `configBackupDir` (absolute path, `~` expanded; empty = `/backups/config`), `configBackupKeepCount` (1–500, default 20), `configBackupKeepDays` (0–3650, default 30; `0` = no age limit). Retention only ever deletes snapshot folders Codeman made there, and never the newest one. Each snapshot is a `0700` folder named `cfg-YYYYMMDD-HHMMSS` (server local time; `-2`, `-3`… on a same-second collision) holding the tracked files under their own names plus a `manifest.json` (`id`, `createdAt`, `reason` of `startup`/`auto`/`manual`/`pre-restore`, `appVersion`, and a SHA-256 and size per file), all `0600`. Only folders with that name and a valid manifest are listed, restored or pruned. Full guide: `docs/wiki/Config-Backups.md`. +Settings (synced, in `settings.json`): `configBackupEnabled` (absent = on), `configBackupDir` (absolute path, `~` expanded; empty = `/backups/config`), `configBackupKeepCount` (1–500, default 20), `configBackupKeepDays` (0–3650, default 30; `0` = no age limit). Retention only ever deletes snapshot folders Codeman made there, and never the newest one. Each snapshot is a `0700` folder named `cfg-YYYYMMDD-HHMMSS` (server local time; `-2`, `-3`… on a same-second collision) holding the tracked files under their own names plus a `manifest.json` (`id`, `createdAt`, `reason` of `startup`/`auto`/`manual`/`pre-restore`, `appVersion`, an `origin` id, and a SHA-256 and size per file), all `0600`. Only folders with that name and a valid manifest are listed, restored or pruned. Full guide: `docs/wiki/Config-Backups.md`. | Method | Path | Body | Notes | | ------ | ------------------------------------ | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/api/config-backups` | none | `settings` in effect, `dir`, `defaultDir`, `tracked[]` and `backups[]` (`id`, `createdAt`, `reason`, `appVersion`, `files[]`, `bytes`), newest first. | -| `POST` | `/api/config-backups` | none | Snapshot now, even if nothing changed (`reason: manual`), then apply retention. `status` is `created` or `empty` (nothing tracked exists yet). | +| `POST` | `/api/config-backups` | none | Snapshot now, even if nothing changed (`reason: manual`), then apply retention. `status` is `created` or `empty` (nothing tracked exists yet); `removed[]` lists the ids retention deleted. `503` when a custom folder does not answer. | | `GET` | `/api/config-backups/diff?from=&to=` | none | What differs between two states: `from`/`to` are a backup id or `current` (the running config; the default for `to`). `files[]` (`name`, `status` `added`/`removed`/`changed`/`unchanged`, `changes[]` of `path`, `kind`, `before`/`after`, `truncated`, `detail`) and `changed`. Credential values are never returned (`redacted: true`). `400` without `from`; `404` unknown id; `409` if a file fails its checksum. | -| `POST` | `/api/config-backups/:id/restore` | none | Put a snapshot's files back, atomically, after a `pre-restore` snapshot of the current ones. `404` unknown id; `409 CONFLICT` if a file fails its checksum (nothing changed). `restartRequired[]` names files the running server only reloads on restart. | +| `POST` | `/api/config-backups/:id/restore` | none | Put a snapshot's files back, atomically, after a `pre-restore` snapshot of the current ones. `404` unknown id; `409 CONFLICT` if a file fails its checksum (nothing changed). Returns `restored[]`, `untouched[]` (tracked files present now that the backup did not hold; left as they are), `safetySnapshot` (the `pre-restore` id) and `restartRequired[]` (files the running server only reads at start: `.env`, `push-keys.json`, `settings.json`). | ## Webhook notifications diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index a50539c8..4c9bdbde 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -113,7 +113,7 @@ Further detail: ⚠️ An ADOPTED container may back SEVERAL cases at different Spawn commands are pinned as literal strings in `test/cli-registry-spawn-golden.test.ts`, remote/docker pane commands in `test/location-overlay-commands.test.ts`. ⚠️ That second golden no longer covers **remote claude or remote omp**: both now have their own arm in `buildRemoteLaunchCommand` (a `--session-id || --resume` pair, and `--continue`, so a respawn continues the same conversation) and never reach `defaultRemoteCommandForMode`, which is what that test asserts. Their real pins are `toContain` substrings in `test/tmux-manager.test.ts` and `test/remote-shared-sessions.test.ts`; changing either arm will NOT fail the golden. -⚠️ Anything reading the registry resolves it AT CALL TIME (`sessionModeSchema()`, `allowedEnvPrefixes()`, `dependencyRegistry()`, the resolvers' `searchDirs` thunks) — a module-level const freezes at first import, so a CLI enabled while the server ran moved the run menu but not that surface. `~/.codeman/clis.json` overrides any entry. Importing the registry still has no filesystem side effects: the only writer is `registry-writer.ts`, imported only by `cli-registry-routes.ts` (the opt-in CLI management routes, `cliManagementEnabled`, default OFF). ⚠️ Every write goes through `mutateRegistryFile()`: mutations run one at a time on a single promise chain, each with a unique temp file (three parallel toggles once lost two and 500'd on a shared temp name), and a file the READER would ignore (group/world permission bits) or quarantine (unparseable) is REFUSED with a 409 rather than rewritten, so one Settings click cannot replace a hand-edit or turn a refused file into trusted 0600 config. Only a missing file starts fresh. After a write the registry is reloaded, so changes apply without a restart. User guide: `docs/cli-registry.md`. +⚠️ Anything reading the registry resolves it AT CALL TIME (`sessionModeSchema()`, `allowedEnvPrefixes()`, `dependencyRegistry()`, the resolvers' `searchDirs` thunks) — a module-level const freezes at first import, so a CLI enabled while the server ran moved the run menu but not that surface. `~/.codeman/clis.json` overrides any entry. Importing the registry still has no filesystem side effects: the only writer is `registry-writer.ts`, imported only by `cli-registry-routes.ts` (the opt-in CLI management routes, `cliManagementEnabled`, default OFF). ⚠️ Every write goes through `mutateRegistryFile()` (the one exception is a config-backup restore of `clis.json`, which writes it atomically and calls `reloadCliRegistry()`): mutations run one at a time on a single promise chain, each with a unique temp file (three parallel toggles once lost two and 500'd on a shared temp name), and a file the READER would ignore (group/world permission bits) or quarantine (unparseable) is REFUSED with a 409 rather than rewritten, so one Settings click cannot replace a hand-edit or turn a refused file into trusted 0600 config. Only a missing file starts fresh. After a write the registry is reloaded, so changes apply without a restart. User guide: `docs/cli-registry.md`. ## Session data and lifecycle diff --git a/docs/wiki/Config-Backups.md b/docs/wiki/Config-Backups.md index 295a4b9c..0f819f4a 100644 --- a/docs/wiki/Config-Backups.md +++ b/docs/wiki/Config-Backups.md @@ -122,14 +122,17 @@ Only the tracked names are ever written, whatever a manifest lists. Settings are re-read on the next request, so reload the page. `clis.json`, `users.json`, `.env` and `push-keys.json` are read once per process: **restart Codeman** for those to take -effect (the response lists which). Restoring `settings.json` also does not re-run the switches +effect (the response lists which). `clis.json` is reloaded by the restore itself and `users.json` +is re-read within a second. Restoring `settings.json` also does not re-run the switches that saving in Settings applies at once (file watchers and the like), so those apply after a restart, and `settings.json` is listed with the restart-required files. A restore does not touch running sessions. When the backup folder is shared (a synced folder used by two machines, or prod and a beta), -each snapshot records the host and data directory that made it, and every instance only -snapshots, lists, prunes, compares and restores its own. +each snapshot records a random id that the instance made once and keeps in its data directory +(`config-backup-id`; not the hostname, which a Docker container changes on every recreate). Every +instance only snapshots, lists, prunes, compares and restores its own. Deleting that file makes +the instance a new owner, and its earlier snapshots stop being listed (they stay on disk). You can also restore by hand: the files in a backup folder are the originals, so copying one back over the file in the data directory is equivalent (stop Codeman first if you are replacing diff --git a/src/config-backup.ts b/src/config-backup.ts index befbfc7c..9ac01bad 100644 --- a/src/config-backup.ts +++ b/src/config-backup.ts @@ -21,7 +21,7 @@ * @module config-backup */ -import { createHash } from 'node:crypto'; +import { createHash, randomUUID } from 'node:crypto'; import { chmodSync, existsSync, @@ -34,7 +34,7 @@ import { statSync, writeFileSync, } from 'node:fs'; -import { homedir, hostname } from 'node:os'; +import { homedir } from 'node:os'; import { isAbsolute, join, resolve, sep } from 'node:path'; /** @@ -96,7 +96,7 @@ export interface ConfigBackupManifest { appVersion: string; /** Digest of the set of files, so an unchanged state is not snapshotted twice. */ hash: string; - /** Host and data dir that made it, so instances sharing one folder keep to their own snapshots. */ + /** Id of the data dir that made it (see {@link configBackupOrigin}), so instances sharing one folder keep to their own snapshots. */ origin?: string; files: ConfigBackupFile[]; } @@ -156,8 +156,33 @@ export function resolveConfigBackupDir(settings: ConfigBackupSettings, dataDir: return dir; } -/** Who owns a snapshot: this machine's name plus the data dir (prod and a beta differ by the latter). */ -export const configBackupOrigin = (dataDir: string): string => `${hostname()}:${resolve(dataDir)}`; +const ORIGIN_FILE = 'config-backup-id'; +const ORIGIN_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +/** + * Who owns a snapshot: a random id made once and kept in the data dir. A hostname would not do (a + * Docker container gets a new one on every recreate, which orphaned every earlier snapshot), nor + * would the path alone (two machines can share a `$HOME` path). It is a file of its own, not the + * hook secret, and is not tracked or restored. If it cannot be written the id is not persisted, so + * snapshots made meanwhile are not listed later; the data dir being unwritable breaks far more. + */ +export function configBackupOrigin(dataDir: string): string { + const path = join(dataDir, ORIGIN_FILE); + try { + const existing = readFileSync(path, 'utf8').trim(); + if (ORIGIN_RE.test(existing)) return existing; + } catch { + /* missing: made below */ + } + const id = randomUUID(); + try { + mkdirSync(dataDir, { recursive: true }); + writeFileSync(path, id + '\n', { mode: 0o600 }); + } catch { + /* see above */ + } + return id; +} const sha256 = (data: Buffer | string): string => createHash('sha256').update(data).digest('hex'); diff --git a/src/web/public/settings-ui.js b/src/web/public/settings-ui.js index 78f41506..a09119d3 100644 --- a/src/web/public/settings-ui.js +++ b/src/web/public/settings-ui.js @@ -1482,7 +1482,11 @@ Object.assign(CodemanApp.prototype, { }, async restoreConfigBackup(id) { - if (!confirm('Restore this backup? Its files replace the current ones (a copy of the current files is made first). Some files only take effect after a restart.')) return; + const names = ((this._configBackups || []).find((b) => b.id === id)?.files || []).map((f) => f.name); + const secret = names.filter((n) => ['.env', 'users.json', 'push-keys.json', 'custom-model-hosts.json', 'webhook.json'].includes(n)); + const list = names.length ? `\n\nFiles replaced: ${names.join(', ')}` : ''; + const warn = secret.length ? `\n\nThis rolls back credentials and accounts (${secret.join(', ')}): passwords, disabled accounts and keys return to how they were then.` : ''; + if (!confirm(`Restore this backup? Its files replace the current ones (a copy of the current files is made first). Some files only take effect after a restart.${list}${warn}`)) return; const res = await this._apiPost(`/api/config-backups/${encodeURIComponent(id)}/restore`, {}); let body = null; try { body = res ? await res.json() : null; } catch { /* fall through */ } @@ -1492,7 +1496,12 @@ Object.assign(CodemanApp.prototype, { } const d = body.data; const restart = d.restartRequired?.length ? ` Restart Codeman to apply: ${d.restartRequired.join(', ')}.` : ''; - this.showToast(`Restored ${d.restored.length} file${d.restored.length === 1 ? '' : 's'}. Reload this page to see settings.${restart}`, 'success'); + // The form still holds the pre-restore values, and Save would PUT them over the restored file. + if (d.restored.includes('settings.json')) { + await this.loadAppSettingsFromServer(); + this.openAppSettings(); + } + this.showToast(`Restored ${d.restored.length} file${d.restored.length === 1 ? '' : 's'}.${restart}`, 'success'); this.loadConfigBackups(); }, diff --git a/src/web/routes/config-backup-routes.ts b/src/web/routes/config-backup-routes.ts index 3494d602..1ecf9e03 100644 --- a/src/web/routes/config-backup-routes.ts +++ b/src/web/routes/config-backup-routes.ts @@ -7,9 +7,9 @@ * GET /api/config-backups/diff?from=&to= — what differs between two snapshots, or a snapshot and the * running config (`current`, the default for `to`). * - * The files include credentials (`.env`, `users.json`, endpoint keys), so in multi-user mode all three + * The files include credentials (`.env`, `users.json`, endpoint keys), so in multi-user mode all four * are admin only. Responses carry file NAMES and sizes, never file content, except the diff, which shows - * changed values but never a credential's (`.env`, `users.json`, `push-keys.json` and any key that looks + * changed values but never a credential's (`.env`, `users.json`, `push-keys.json`, `webhook.json`, `intents.json` and any key that looks * like a token/secret/password are reported as "changed" only). Restore replaces files the running server * may have cached: `restartRequired` says which ones only apply after a restart. */ @@ -18,6 +18,7 @@ import { createRequire } from 'node:module'; import type { FastifyInstance, FastifyReply, FastifyRequest } from 'fastify'; import { ApiErrorCode, createErrorResponse, getErrorMessage, type ApiResponse } from '../../types.js'; import { isAdmin, readJsonConfig, SETTINGS_PATH } from '../route-helpers.js'; +import { reloadCliRegistry } from '../../config/cli-registry/registry.js'; import { isMultiUserMode } from '../../config/multiuser.js'; import { getDataDir } from '../../config/instance.js'; import { intentStore } from '../../intent-store.js'; @@ -53,8 +54,9 @@ const APP_VERSION: string = (() => { } })(); -/** Files whose running-process copy only reloads on restart (the rest are read per request). */ -const RESTART_REQUIRED = new Set(['clis.json', 'users.json', '.env', 'push-keys.json', 'settings.json']); +/** Files whose running-process copy only reloads on restart (the rest are read per request; `users.json` + * is re-read within a second, `clis.json` is reloaded below, `intents.json` by `reload()`). */ +const RESTART_REQUIRED = new Set(['.env', 'push-keys.json', 'settings.json']); export interface ConfigBackupStatus { settings: ConfigBackupSettings; @@ -203,6 +205,8 @@ export function registerConfigBackupRoutes(app: FastifyInstance): void { const out = restoreConfigBackup({ dataDir: getDataDir(), backupDir: dir, id, appVersion: APP_VERSION }); // IntentStore caches the file and rewrites it whole on the next prompt, which would undo the restore. if (out.restored.includes('intents.json')) intentStore.reload(); + // restoreConfigBackup writes clis.json directly (the one writer besides mutateRegistryFile). + if (out.restored.includes('clis.json')) reloadCliRegistry(); return { success: true, data: { diff --git a/test/config-backup-origin.test.ts b/test/config-backup-origin.test.ts new file mode 100644 index 00000000..4db940e5 --- /dev/null +++ b/test/config-backup-origin.test.ts @@ -0,0 +1,58 @@ +/** + * @fileoverview The snapshot origin must survive a hostname change (a Docker container gets a new + * hostname on every recreate) and still differ between data dirs. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +const host = vi.hoisted(() => ({ name: 'container-a' })); +vi.mock('node:os', async (importOriginal) => ({ + ...(await importOriginal()), + hostname: () => host.name, +})); + +import { + configBackupOrigin, + createConfigBackup, + listConfigBackups, + pruneConfigBackups, + restoreConfigBackup, +} from '../src/config-backup.js'; + +let root: string; +let data: string; +let backups: string; +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'cfg-origin-')); + data = join(root, 'data'); + backups = join(root, 'backups'); + mkdirSync(data, { recursive: true }); + writeFileSync(join(data, 'settings.json'), '{"a":1}'); + host.name = 'container-a'; +}); +afterEach(() => rmSync(root, { recursive: true, force: true })); + +describe('configBackupOrigin', () => { + it('is made once, kept in the data dir, and differs between data dirs', () => { + const a = configBackupOrigin(data); + expect(configBackupOrigin(data)).toBe(a); + expect(readFileSync(join(data, 'config-backup-id'), 'utf8').trim()).toBe(a); + const other = join(root, 'other'); + expect(configBackupOrigin(other)).not.toBe(a); + }); + + it('keeps snapshots listed, restorable and prunable after the hostname changes', () => { + const made = createConfigBackup({ dataDir: data, backupDir: backups, reason: 'auto', now: 1_000_000 }) as { + id: string; + }; + createConfigBackup({ dataDir: data, backupDir: backups, reason: 'manual', force: true, now: 2_000_000 }); + host.name = 'container-b'; // the container was recreated + const origin = configBackupOrigin(data); + expect(listConfigBackups(backups, origin)).toHaveLength(2); + expect(() => restoreConfigBackup({ dataDir: data, backupDir: backups, id: made.id })).not.toThrow(); + const removed = pruneConfigBackups(backups, { keepCount: 1, keepDays: 0, origin }); + expect(removed).toHaveLength(2); // 3 now: the restore took a pre-restore copy + }); +}); diff --git a/test/config-backup-restore-ui.test.ts b/test/config-backup-restore-ui.test.ts new file mode 100644 index 00000000..3e79e972 --- /dev/null +++ b/test/config-backup-restore-ui.test.ts @@ -0,0 +1,64 @@ +/** + * @fileoverview Restoring settings.json from inside App Settings must refill the form, or Save + * would PUT the pre-restore values back over the restored file. Evaluates the real settings-ui.js + * against a stub app. + */ +import { describe, expect, it } from 'vitest'; +import { readFileSync } from 'node:fs'; +import vm from 'node:vm'; + +const SRC = readFileSync(new URL('../src/web/public/settings-ui.js', import.meta.url), 'utf-8'); + +function makeApp(restored: string[], confirmed = true) { + const calls: string[] = []; + const confirms: string[] = []; + function CodemanApp(this: unknown) {} + const ctx = vm.createContext({ + CodemanApp, + window: {}, + document: {}, + confirm: (m: string) => (confirms.push(m), confirmed), + escapeHtml: (s: string) => s, + }); + vm.runInContext(SRC, ctx); + const app = Object.create((CodemanApp as unknown as { prototype: object }).prototype) as Record; + Object.assign(app, { + _configBackups: [ + { + id: 'cfg-1', + files: [{ name: 'settings.json' }, { name: 'users.json' }, { name: '.env' }, { name: 'clis.json' }], + }, + ], + _apiPost: async () => ({ + ok: true, + json: async () => ({ success: true, data: { restored, restartRequired: [] } }), + }), + loadAppSettingsFromServer: async () => void calls.push('reload-settings'), + openAppSettings: () => void calls.push('refill-form'), + loadConfigBackups: () => void calls.push('reload-list'), + showToast: () => {}, + }); + return { app, calls, confirms }; +} + +describe('restoreConfigBackup (Settings UI)', () => { + it('re-reads settings into the form when settings.json was restored', async () => { + const { app, calls } = makeApp(['settings.json', 'clis.json']); + await (app.restoreConfigBackup as (id: string) => Promise)('cfg-1'); + expect(calls).toEqual(['reload-settings', 'refill-form', 'reload-list']); + }); + + it('leaves the form alone when settings.json was not part of the restore', async () => { + const { app, calls } = makeApp(['clis.json']); + await (app.restoreConfigBackup as (id: string) => Promise)('cfg-1'); + expect(calls).toEqual(['reload-list']); + }); + + it('names the files being replaced and calls out the credential ones', async () => { + const { app, confirms } = makeApp(['settings.json'], false); + await (app.restoreConfigBackup as (id: string) => Promise)('cfg-1'); + expect(confirms[0]).toContain('settings.json, users.json, .env, clis.json'); + expect(confirms[0]).toContain('credentials'); + expect(confirms[0]).toMatch(/users\.json, \.env\)/); + }); +}); diff --git a/test/routes/config-backup-routes.test.ts b/test/routes/config-backup-routes.test.ts index 3d2cceba..a0a2bc17 100644 --- a/test/routes/config-backup-routes.test.ts +++ b/test/routes/config-backup-routes.test.ts @@ -84,7 +84,7 @@ describe('/api/config-backups', () => { expect(res.statusCode).toBe(200); const d = res.json().data; expect(d.restored).toEqual(expect.arrayContaining(['settings.json', 'clis.json'])); - expect(d.restartRequired).toContain('clis.json'); + expect(d.restartRequired).not.toContain('clis.json'); // reloaded by the restore itself expect(d.restartRequired).toContain('settings.json'); // its watcher toggles are not re-run by a restore expect(d.safetySnapshot).toBeTruthy(); expect(JSON.parse(readFileSync(SETTINGS_PATH, 'utf8')).displayName).toBe('before');