From 6cf5ab05d6f47899a4ddec45e29caee6b26250c9 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Thu, 22 Oct 2026 19:21:02 +0200 Subject: [PATCH] docs: config backups section in architecture-invariants, CLAUDE.md pointer and State Files Merge-time fix for #661, as promised in the reviews. The long Config backups paragraph in CLAUDE.md moves into a new architecture-invariants section that gives each rule with the reason behind it (allowlist, volatile keys, retention by successor and the date floor, the origin id, retention after a bad or reset settings.json, the bounded probe, one walk per pass, verify before write, the undo copy and the partial-restore answer, the reload hooks, redaction, admin only). CLAUDE.md keeps the short rules and a pointer, and the State Files list names config-backup-id and backups/config/ with the retention each manifest records. Co-Authored-By: Claude Opus 5.5 (1M context) --- CLAUDE.md | 4 ++-- docs/architecture-invariants.md | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 357c1db4..3e2905cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -156,7 +156,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Auto-resume on usage limit** (opt-in per session, top of the Respawn tab): when Claude halts on a subscription limit, `usage-limit-patterns.ts` (pure, unit-tested) parses the reset time and `SessionAutoOps` arms a timer for reset+2min, then sends Esc + `continue`. ⚠️ Respawn cycles are blocked while paused (`isLimitPaused` guard in `onIdleDetected`), which is what prevents `/clear` from wiping the paused conversation. Claude-mode only. → [architecture-invariants#auto-resume-on-usage-limit](docs/architecture-invariants.md#auto-resume-on-usage-limit) -**Config backups** (`src/config-backup.ts`, `routes/config-backup-routes.ts`; `configBackupEnabled` absent = ON, `configBackupDir`, `configBackupKeepCount` 2-500 default 20, `configBackupKeepDays` 0-3650 default 30 (an ABSENT key means what the newest backup in the folder recorded, `withRecordedRetention`); `docs/wiki/Config-Backups.md`): timestamped snapshots `cfg-YYYYMMDD-HHMMSS/` (+ `manifest.json` with SHA-256s, `reason`, `appVersion`) of a FIXED allowlist (`CONFIG_BACKUP_FILES`: settings, clis, custom-model-hosts, webhook, intents, linked-cases, remote/docker hosts+cases (incl. `remote-cases.json`), webviews, push-keys, users, `.env`) into `/backups/config` or a custom absolute folder. ⚠️ `state.json` is deliberately NOT tracked (rewritten per session event; it also holds cron jobs and respawn presets). One snapshot per real content change (digest of the file set vs the latest manifest), at startup + a 5-minute timer, skipped in test mode. ⚠️ Retention (`pruneConfigBackups`) only deletes folders matching the name pattern AND holding a parseable manifest, and never the newest; a custom folder equal to the data dir falls back to the default. ⚠️ Restore verifies every file's SHA-256 BEFORE writing, takes a `pre-restore` snapshot, writes atomically at 0600 and only allowlisted names; the id is validated against the name pattern (no traversal). Snapshots are 0700/0600 because `.env`/`users.json`/`custom-model-hosts.json` carry credentials; all routes are admin-only in multi-user mode. ⚠️ A custom folder is probed with `probePathKind(dir, { pastCap: true })` before every pass and in each route (#516: it may be a network mount; `unknown` skips the pass / answers 503 `OPERATION_FAILED`), and a missing custom folder is only created when its parent is present. ⚠️ Each manifest records an `origin`, a random id kept in `/config-backup-id` (NOT the hostname: a Docker container gets a new one per recreate, which orphaned every snapshot); change check, prune, list, restore and diff are scoped to it so instances sharing one folder do not snapshot or prune each other. Restoring `intents.json` calls `intentStore.reload()`; `settings.json` applies its watcher toggles after a restart; Settings re-reads `/api/settings` into the form after a restore of it, since Save would otherwise PUT the old values back. ⚠️ Restoring `clis.json` writes it directly (the one writer besides `mutateRegistryFile()`) and then calls `reloadCliRegistry()`. → [docs/wiki/Config-Backups.md](docs/wiki/Config-Backups.md) +**Config backups** (`src/config-backup.ts`, `routes/config-backup-routes.ts`; `configBackupEnabled` absent = ON, SYNCED; `docs/wiki/Config-Backups.md`): timestamped `cfg-*` snapshots of a FIXED allowlist of config files (`CONFIG_BACKUP_FILES`; `state.json` deliberately not) at startup and whenever their content changes, into `/backups/config` or `configBackupDir`. ⚠️ A field a tracked file rewrites on its own goes in `VOLATILE_KEYS`, or every tick makes a snapshot and the history shrinks to about 95 minutes. ⚠️ Retention deletes only this instance's folders that carry a valid manifest, never the newest, and ages a snapshot from its SUCCESSOR; a retention key ABSENT from `settings.json` means what the newest manifest recorded (`withRecordedRetention`), never the defaults. ⚠️ The instance is identified by the random id in `config-backup-id`, never the hostname (a Docker recreate changes it). ⚠️ A custom folder is probed BOUNDED (#516) before ANY read, the recorded-retention fallback included, and a pass reads the folder once (`runConfigBackupPass`). ⚠️ Restore verifies every SHA-256 and refuses symlinks and group/world bits before it writes anything, takes the undo copy first, and reports a partial failure as 500 with `restored`/`failed`/`safetySnapshot` in `data`; every path that wrote files runs `afterRestore()` (intents reload, CLI registry reload, admin audit), and Settings re-reads the form after a `settings.json` restore. ⚠️ Compare never returns a credential (`REDACT_ALL_FILES`, `SECRET_KEY_RE`). Admin only in multi-user mode. → [architecture-invariants#config-backups](docs/architecture-invariants.md#config-backups) **Plan-usage chip** (`showPlanUsageLimits`, per-device: desktop default **ON**, handhelds OFF): resolve DISPLAY only through `planUsageChipEnabled()` (settings-ui.js). The same setting is the server-side COLLECTION switch, read fresh by `readPlanUsageTelemetryEnabled()` (hooks-config.ts) at every claude create/respawn. ⚠️ An ABSENT key reads as ON in the reader; `GET /api/settings` must never write. ⚠️ A save sends `showPlanUsageLimits` ONLY when it flips the chip on that device (`planUsageCollectionFlip()`), or a phone switches collection off for every desktop. Claude data comes from the statusLine exporter, injected as an EPHEMERAL `claude --settings` flag (`resolveStatusLineCliCommand`), never written to disk, WRAPPING a user's own statusLine, posting to `POST /api/status-telemetry`. Codex comes from a read-only `account/rateLimits/read` poll (main bucket only). The statusline's `prompt_cache` object (Claude Code v2.1.251+) is kept by `StatusTelemetrySchema` (every documented scalar field, null-tolerant; `parsePromptCache()` in usage-telemetry.ts normalizes only the stored subset) and shown in the footer as `cache:until HH:MM` / `cache:cold` (only when the user has no statusline of their own) and in Session Options → Respawn (`#promptCacheStatus`, fed by `Session.promptCache` in `toState()` plus the change-only `session:promptCache` SSE event, worded by the pure `CodemanPromptCache.format`): warm-until/cold, the re-cache stake, the hit rate, the last miss's causes, and a `Context compacted at HH:MM` note after a compaction or a tool-result clearing (stamped on the first warm report whose `recache_tokens_if_cold` is null after one that carried it, never from `expected_rebuilds`, which Claude Code bumps only at the next request; NEVER an alert). ⚠️ `Session.setPromptCache()` stamps `compactedAt` onto a NEW object (never mutating the parsed report) and the route broadcasts the STORED object; the last report is restored on restart (constructor `promptCache`, like `displayModel`) and the readout is time-aware (a passed `expiresAt` reads cold, not stale-warm). It is the source of truth for the cache TTL, never the plan tier. → [architecture-invariants#plan-usage-chip-statusline-telemetry](docs/architecture-invariants.md#plan-usage-chip-statusline-telemetry), `docs/usage-limits-display-plan.md` @@ -304,7 +304,7 @@ One module per domain in `src/web/routes/` (plus a barrel; `ls src/web/routes/` ## State Files -All in `~/.codeman/`: `state.json` (sessions, settings, respawn, orchestrator, cron jobs/runs, owner tab layouts), `mux-sessions.json` (tmux recovery), `settings.json` (user prefs), `push-keys.json` + `push-subscriptions.json`, `session-lifecycle.jsonl` (audit log), `update-status.json` (self-updater progress, polled across the service restart), `docker-env-applied.json` (Compose deployment only: sha256 of the Dockerfile + compose file the running container was built from, written by `Start-Codeman.sh`, read by the self-updater's environment gate), `docker-build-source.json` (Compose deployment only: the checkout's HEAD commit and `package-lock.json` hash the `codeman-node-modules`/`codeman-dist` volumes currently reflect, written by both `Start-Codeman.sh` and a successful in-place self-update, compared to detect and refresh a volume left stale by an externally-triggered rebuild), `linked-cases.json`, `webviews.json` (saved web-tab dashboard URLs), `remote-hosts.json` + `remote-cases.json`, `docker-hosts.json` + `docker-cases.json` + `docker-exports/`, `subagent-window-states.json` + `subagent-parents.json` (subagent window layout, GET/PUT `/api/subagent-window-states`/`-parents`), `hook-secret` (per-instance), `users.json` (multi-user, mode 0600) + `admin-audit.jsonl`, `intents.json` (Read My Mind intent profiles, mode 0600), `webhook.json` (webhook notifications: enabled/service/scope plus the ntfy/Slack/Discord URL, a bearer secret, so mode 0600, kept out of `settings.json` and never returned by `/api/webhook`, which is admin-only in multi-user mode), `certs/` (self-signed TLS for `--https`), `.env` (CODEMAN_USERNAME/PASSWORD fallback for the `codeman attach` CLI), `install.log` (installer step output, written by `install.sh`'s `run_step`) and `tailscale-rename` (the node name before `install.sh` renamed it, so uninstall can offer it back; both installer-route only). Transient: `self-update-runner.sh`. Multi-user case spaces live OUTSIDE the data dir at `~/codeman-users//cases` (shared across instances like `~/codeman-cases`, override `CODEMAN_USER_SPACES_DIR`). +All in `~/.codeman/`: `state.json` (sessions, settings, respawn, orchestrator, cron jobs/runs, owner tab layouts), `mux-sessions.json` (tmux recovery), `settings.json` (user prefs), `push-keys.json` + `push-subscriptions.json`, `session-lifecycle.jsonl` (audit log), `update-status.json` (self-updater progress, polled across the service restart), `docker-env-applied.json` (Compose deployment only: sha256 of the Dockerfile + compose file the running container was built from, written by `Start-Codeman.sh`, read by the self-updater's environment gate), `docker-build-source.json` (Compose deployment only: the checkout's HEAD commit and `package-lock.json` hash the `codeman-node-modules`/`codeman-dist` volumes currently reflect, written by both `Start-Codeman.sh` and a successful in-place self-update, compared to detect and refresh a volume left stale by an externally-triggered rebuild), `linked-cases.json`, `webviews.json` (saved web-tab dashboard URLs), `remote-hosts.json` + `remote-cases.json`, `docker-hosts.json` + `docker-cases.json` + `docker-exports/`, `subagent-window-states.json` + `subagent-parents.json` (subagent window layout, GET/PUT `/api/subagent-window-states`/`-parents`), `hook-secret` (per-instance), `users.json` (multi-user, mode 0600) + `admin-audit.jsonl`, `intents.json` (Read My Mind intent profiles, mode 0600), `webhook.json` (webhook notifications: enabled/service/scope plus the ntfy/Slack/Discord URL, a bearer secret, so mode 0600, kept out of `settings.json` and never returned by `/api/webhook`, which is admin-only in multi-user mode), `config-backup-id` (the random id config backups record as each manifest's `origin`, so instances sharing a backup folder keep to their own snapshots; mode 0600, never tracked or restored, and deleting it unlists this instance's earlier snapshots), `backups/config/` (the default config-backup folder, moved by `configBackupDir`: `cfg-YYYYMMDD-HHMMSS/` snapshots, 0700 with 0600 files, each with a `manifest.json` of SHA-256s, the origin and the retention in force (`keepCount`/`keepDays`), which a pass falls back to when `settings.json` lacks a retention key), `certs/` (self-signed TLS for `--https`), `.env` (CODEMAN_USERNAME/PASSWORD fallback for the `codeman attach` CLI), `install.log` (installer step output, written by `install.sh`'s `run_step`) and `tailscale-rename` (the node name before `install.sh` renamed it, so uninstall can offer it back; both installer-route only). Transient: `self-update-runner.sh`. Multi-user case spaces live OUTSIDE the data dir at `~/codeman-users//cases` (shared across instances like `~/codeman-cases`, override `CODEMAN_USER_SPACES_DIR`). **Generated top-level dirs** (all gitignored — don't edit or commit): `dist/` (esbuild output), `out/`, `coverage/`, `test-results/`, `tmp/`, `screenshots-echo-diag/`. The committed gesture bundle (`src/web/public/gesture/gesture-codeman.js`) IS tracked, but its runtime wasm/model assets (`src/web/public/gesture/wasm/`, `*.task`) are fetched and gitignored. diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index 4c9bdbde..6a7d4b72 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -707,6 +707,36 @@ Matching semantics: a query containing `/` matches the relative path, otherwise ⚠️ Closing the preview must **pause and unload** the media (`_stopFilePreviewMedia` in panels-ui.js): dropping the overlay's `visible` class is `display:none` and nothing else, and a DETACHED `HTMLMediaElement` keeps playing, which is how the X button used to leave a video audible with no player to pause. +### Config backups + +**Config backups** (`src/config-backup.ts`, pure over the directories it is handed; `routes/config-backup-routes.ts`; the Config backups group in App Settings → System, `settings-ui.js`; contributed in #661, user guide `docs/wiki/Config-Backups.md`, endpoints in `docs/api-reference.md`): timestamped snapshots of the files that hold what a user customized, so a bad write can be undone. Settings (SYNCED): `configBackupEnabled` (absent = ON, since it protects against an event nobody schedules), `configBackupDir` (absolute, `~` expanded; empty, relative or the data dir itself = `/backups/config`), `configBackupKeepCount` (2 to 500, default 20), `configBackupKeepDays` (0 to 3650, default 30, `0` = no age limit). A snapshot is a `cfg-YYYYMMDD-HHMMSS` folder (server local time, `-2`, `-3`… on a same-second collision; mode 0700) holding the tracked files under their own names plus a `manifest.json` (all 0600): `id`, `createdAt`, `reason` (`startup`/`auto`/`manual`/`pre-restore`), `appVersion`, the change-check `hash`, the `origin`, the `keepCount`/`keepDays` in force, and a SHA-256 and size per file. It is built in a `.tmp-cfg-*` folder and renamed into place; a pass sweeps temp folders older than an hour. Passes run at startup and every 5 minutes (`CONFIG_BACKUP_INTERVAL_MS` in `server.ts`, not in test mode), and on **Back up now** (forced). + +⚠️ **The tracked list is a fixed allowlist** (`CONFIG_BACKUP_FILES`), never "everything in the data dir", and restore writes only those names whatever a manifest lists. `state.json` is deliberately NOT tracked: it is rewritten on every session event, so it would make a snapshot per event and bury the real changes. It also holds cron jobs and respawn presets, which is the documented gap. + +⚠️ **One snapshot per real change.** The change check compares a digest of the file set with the newest manifest's `hash`, looking through `VOLATILE_KEYS`, fields a tracked file rewrites on its own: `custom-model-hosts.json` `lastDiscoveredAt` (the discovery poll), `users.json` `lastLoginAt` (every Basic login), `settings.json` `lastUsedCase` (a session started in another case), `intents.json` `recentPrompts`/`updatedAt` (every captured prompt), `docker-cases.json` `lastClaudeSessionId` (a `/clear` in a Docker case) and `webviews.json` `lastOpenedAt` (opening a web tab). Each was found the same way: one tick-driven rewrite made a snapshot per tick, and the 20 kept copies covered about 95 minutes. A new self-rewritten field in a tracked file needs an entry there and a row in the `it.each` table of `test/config-backup.test.ts`. The snapshot still stores the bytes and restore verifies the raw digest; only the comparison looks through these keys (so Compare calls `intents.json` unchanged when only prompts differ, yet a restore drops the prompts captured since). + +⚠️ **Retention deletes only what this module made, and never the newest snapshot.** A folder counts only with the name pattern AND a parseable manifest of this origin, so a custom folder that also holds other things is safe to point at. A snapshot is aged from its SUCCESSOR's `createdAt` (its state was live until the next one was made): aging it by its own date deleted the last good copy on the first change after a quiet month, which is exactly the bad write the feature exists for. The keep count floor is 2 (`CONFIG_BACKUP_MIN_KEEP_COUNT`), since 1 deletes the state before every change. A new snapshot is never dated earlier than the newest one (`createdAt + 1`): one written while the clock read ahead (dual boot, a VM resumed with a stale clock) stayed newest after the correction, differed from every later pass, and made each one write a copy until real time caught up. The flip side, a clock once set to a future year, is a Troubleshooting entry (delete the future-dated folder). + +⚠️ **The origin is a random id in `/config-backup-id`, never the hostname.** Instances that share a folder (two machines on one synced folder, prod and a beta) must keep to their own snapshots, so the change check, list, prune, Compare and restore are all scoped to it; without that, each instance snapshotted on every tick and Restore offered the other machine's `users.json` and `.env`. The hostname was tried first and was wrong: a Docker Compose container gets a new one on every recreate, which unlisted every earlier snapshot, and since prune is origin-scoped they were never deleted either. The id is not the hook secret and is never tracked or restored. Listing never creates it (`readConfigBackupOrigin()`; only a pass or a restore does). A fresh or reinstalled data dir is a new owner; copying an `origin` from an old manifest into the file makes those snapshots visible again. + +⚠️ **Retention survives a bad `settings.json`.** Each manifest records the retention in force. `runConfigBackup()` reads `settings.json` itself, because `readJsonConfig` answers `{}` for any failure, including a read that lands inside the non-atomic write of `PUT /api/settings`: a file that does not parse still gets its snapshot but prunes nothing, and a retention key ABSENT from the file (a reset) means what the newest manifest that recorded one holds (`withRecordedRetention()`, per key, in the folder the file names, so re-entering a custom folder after a reset finds that folder's own history). `GET /api/config-backups` reports the same values the pass uses, the form fills from them, and Save sends a retention key only when it was saved before or edited there (`_configBackupRetentionPayload()`), so merely displaying the recorded retention never writes it back. Restoring `settings.json` keeps the four `configBackup*` keys at their live values (an old `configBackupDir` would point the list at another folder and hide the undo copy just made), except a retention key the live file lacks, which keeps the backup's. While `settings.json` does not parse a pass reads it as `{}`, so its snapshot goes to the default folder; nothing is widened. + +⚠️ **A custom folder is probed BOUNDED before anything reads it** (#516: it may be a network mount that stopped answering, and the GET runs every time App Settings opens). `probeBackupDir()` runs `probePathKind(dir, { pastCap: true })` before every pass and in every route, and `readEffectiveBackupSettings()` probes BEFORE the recorded-retention fallback lists the folder (that fallback once did a `readdirSync` first whenever only `configBackupDir` was set). `unknown` answers 503 `OPERATION_FAILED` through `describeUnknownPath()` and skips the pass; a missing custom folder is created only when its parent is present. The default folder needs no probe. `test/routes/config-backup-unreachable.test.ts` pins all four routes with a recording `node:fs` mock. + +⚠️ **One walk of the folder per pass.** `runConfigBackupPass()` reads the folder once and hands the names to the temp sweep and the manifests to the retention fallback, the change check and the prune (with the snapshot it just made prepended: it is always the newest, by the date floor above). Each of the three used to list the folder itself, so every manifest was read three times per pass, every five minutes, on what may be a network folder; restore lists once too. `test/routes/config-backup-one-walk.test.ts` counts the reads. + +⚠️ **Restore verifies everything before it writes anything.** Every file's SHA-256 against the manifest, and the id must match the name pattern (no traversal); a mismatch is a 409 with nothing changed. It also refuses a snapshot folder or file that is a symlink or has a group/world bit: this module only ever writes 0700/0600 real files, so such a thing was planted or loosened later, and rewriting it as 0600 would turn it into trusted config (`clis.json` selects the binaries Codeman spawns; `isUnsafePermissions` refuses the same bits there). The mode test is skipped where the filesystem keeps no modes (vfat/exFAT, CIFS, WSL `/mnt/d` without `metadata` report 0755/0777 for everything, and Restore would refuse every backup there): `folderKeepsModes()` decides by creating a 0600 probe file with a random name and `wx`, because the earlier fixed `.mode-probe-` name let a planted symlink be truncated and chmodded by one Compare click. Compare applies the same checks to the snapshots it reads. + +⚠️ **The undo copy comes first, and a partial restore is reported, never hidden.** A `pre-restore` snapshot of the live files precedes the writes; it is not forced, so when nothing changed since the newest backup that backup is the undo copy and is the one reported (a forced copy spent a retention slot on a duplicate). It records the retention in force, so it cannot hide that retention from the next pass. Each file is replaced atomically (temp + rename, 0600). A file that cannot be written (a full disk, a directory in its place) does not stop the others: `ConfigBackupPartialRestoreError` carries `restored`, `failed`, `untouched` and `safetySnapshot`, and the route answers 500 with the error envelope plus those fields (and `restartRequired`) in `data`, the shape the tab-layout 409 uses. The bare 500 it replaced named neither the files already put back nor the undo copy. + +⚠️ **What a restore wrote must reach the running server**, on the full and the partial path alike (`afterRestore()` in the route): `intentStore.reload()` (the store caches `intents.json` and rewrote it whole on the next captured prompt, reverting the restore), `reloadCliRegistry()` (a restore of `clis.json` is the one write that bypasses `mutateRegistryFile()`, see CLI registry), and in multi-user mode a `config_backup_restore` admin-audit entry (`users.json` holds accounts, password hashes, roles and bypass grants). `users.json` itself is re-read within a second by the user store. `.env`, `push-keys.json` and `settings.json` (whose watcher toggles only `PUT /api/settings` runs) apply after a restart: `RESTART_REQUIRED`, returned as `restartRequired`. Settings re-reads `/api/settings` into the form after any restore that wrote `settings.json`, partial ones included, because the form still holds the old values and the next Save would PUT them over the restored file. + +⚠️ **Compare never returns a credential.** `REDACT_ALL_FILES` (`.env`, `users.json`, `push-keys.json`, `webhook.json` whose URL is a bearer secret, `intents.json` whose prompts carry secrets) report changed key names only; anywhere else a key matching `SECRET_KEY_RE` (token, secret, password, credential, auth, `*key`) is reported as changed without its value, and URL values lose userinfo, query and fragment. Listing returns file names and sizes only. + +⚠️ **Admin only in multi-user mode**: all four routes answer 403 for a non-admin and the Settings group is hidden for them (`_applyDoctorAdminGate()`), because the files include credentials. + +Tests: `test/config-backup.test.ts`, `test/config-backup-origin.test.ts`, `test/config-backup-restore-ui.test.ts` (vm harness over the real `settings-ui.js`), `test/routes/config-backup-routes.test.ts`, `test/routes/config-backup-unreachable.test.ts`, `test/routes/config-backup-one-walk.test.ts`. + ## Frontend ### Command palette and shortcut registry