From e408fa21bc962b0b9738818ed3ee332597b18dba Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Thu, 22 Oct 2026 19:10:11 +0200 Subject: [PATCH] fix(sessions): retry a failed keep-alive write before the expiry and close its window Merge-time fix for #666. A nudge only goes out inside the last five minutes before `expiresAt`, so the check armed a whole `KEEP_WARM_LEAD_MS` after a failed write landed at or past the expiry and stopped with "too close to expiry": the retry the status promised never happened. The window `beginKeepAliveTurn()` opened also stayed open for its ten-minute ceiling with no turn running. A failed write now ends the window (`endKeepAliveTurn()` joins `KeepWarmSessionView`; `Session` already has it) and re-arms at `min(now + KEEP_WARM_RETRY_MS, expiresAt - KEEP_WARM_LAST_CHANCE_MS)`. The test advances the clock and sees the second write, and a second one pins that the retry never lands past the last-chance line. Co-Authored-By: Claude Opus 5.5 (1M context) --- CLAUDE.md | 2 +- docs/architecture-invariants.md | 2 +- docs/wiki/Keeping-Agents-Running.md | 2 +- src/prompt-cache-keepwarm.ts | 14 +++++++++-- test/prompt-cache-keepwarm.test.ts | 38 ++++++++++++++++++++++++++++- 5 files changed, 52 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9b8f64f2..fc95bc37 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,7 +158,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Plan-usage chip** (`showPlanUsageLimits`, per-device: desktop default **ON**, handhelds OFF): resolve DISPLAY only through `planUsageChipEnabled()` (settings-ui.js). The same setting is the server-side COLLECTION switch, read fresh by `readPlanUsageTelemetryEnabled()` (hooks-config.ts) at every claude create/respawn. ⚠️ An ABSENT key reads as ON in the reader; `GET /api/settings` must never write. ⚠️ A save sends `showPlanUsageLimits` ONLY when it flips the chip on that device (`planUsageCollectionFlip()`), or a phone switches collection off for every desktop. Claude data comes from the statusLine exporter, injected as an EPHEMERAL `claude --settings` flag (`resolveStatusLineCliCommand`), never written to disk, WRAPPING a user's own statusLine, posting to `POST /api/status-telemetry`. Codex comes from a read-only `account/rateLimits/read` poll (main bucket only). The statusline's `prompt_cache` object (Claude Code v2.1.251+) is kept by `StatusTelemetrySchema` (every documented scalar field, null-tolerant; `parsePromptCache()` in usage-telemetry.ts normalizes only the stored subset) and shown in the footer as `cache:until HH:MM` / `cache:cold` (only when the user has no statusline of their own) and in Session Options → Respawn (`#promptCacheStatus`, fed by `Session.promptCache` in `toState()` plus the change-only `session:promptCache` SSE event, worded by the pure `CodemanPromptCache.format`): warm-until/cold, the re-cache stake, the hit rate, the last miss's causes, and a `Context compacted at HH:MM` note after a compaction or a tool-result clearing (stamped on the first warm report whose `recache_tokens_if_cold` is null after one that carried it, never from `expected_rebuilds`, which Claude Code bumps only at the next request; NEVER an alert). ⚠️ `Session.setPromptCache()` stamps `compactedAt` onto a NEW object (never mutating the parsed report) and the route broadcasts the STORED object; the last report is restored on restart (constructor `promptCache`, like `displayModel`) and the readout is time-aware (a passed `expiresAt` reads cold, not stale-warm). It is the source of truth for the cache TTL, never the plan tier. → [architecture-invariants#plan-usage-chip-statusline-telemetry](docs/architecture-invariants.md#plan-usage-chip-statusline-telemetry), `docs/usage-limits-display-plan.md` -**Prompt-cache keep-warm** (`src/prompt-cache-keepwarm.ts`; per-session opt-in on the Respawn tab, app-wide default `promptCacheKeepWarm` + cap `promptCacheKeepWarmMaxHours` in App Settings → System → Automation, SYNCED, default OFF / 2 h; step 2 of Discussion #554): one keep-alive turn (`KEEP_WARM_NUDGE`, under 100 chars) five minutes before an idle claude session's cache would expire, so the next prompt reads the conversation from cache instead of re-writing it at 2x. Codeman owns the idle policy for these sessions: a launch with keep-warm on also carries `"idleCompaction": false`, folded LAST into the ONE `--settings` JSON (`session-cli-registry-bridge.ts`), decided once per create/respawn by `decideIdleCompactionOptOut()` (tmux-manager.ts: keep-warm on via `keepWarmOn()`, exporter `injected`, claude's `idleCompactionSetting` gate at 2.1.290, fail-closed like `--name`) and recorded as `MuxSession.idleCompactionOff`; a keep-warm-off launch stays byte-identical (`test/cli-registry-spawn-golden.test.ts`). Switched on after launch it is best-effort and the readout says idle compaction may still run until the next launch. ⚠️ The deadline is `Session.promptCache.expiresAt` from the statusline, never idle detection or the plan tier; a `5m` TTL is a hard stop, and so is a launch whose `statusLineExporter` is not `injected`. ⚠️ Every guard lives in the pure `decideKeepWarm()` (effective opt-in, where an agent-spawned session (`parentSessionId`) does not inherit the app default; exporter injected; warm; no `compactedAt` this gap; tokens at stake; live pane (pid AND no `paneExit`); no limit pause (auto-resume's, or a limit message on screen); no respawn/Ralph/task/waiter; inside the cap since `lastRealPromptAt`; idle; no dialog; EMPTY composer via `composerIsEmpty()`, continuation lines included, read from a STYLED capture so Claude Code's faint prompt suggestion is not taken for a draft; no recent keystroke); add a guard there, never in the service. ⚠️ The nudge IS a submit and moves `lastSubmitAt` (the response viewer's conversation correlation needs it); the window `beginKeepAliveTurn()` opens keeps it off `lastRealPromptAt` (the cap's own persisted clock) and opens its `stop`/`idle_prompt` hooks pre-acknowledged (`preAcknowledged` on `notePrompt`), so nothing alerts, and leaves an idle item already pending exactly as found (only a human spends an idle alert; `session:working`/`session:idle` carry `keepAlive: true` so the page's stuck timer ignores the turn); the keep-alive's own `idle_prompt` ends that window (`endKeepAliveTurn()`; `KEEP_WARM_TURN_WINDOW_MS` is only a ceiling, since a slow turn's idle prompt lands a minute after it ends), and so does any other submit: a `fromUser` write, a programmatic write of anything but the nudge, or a `prompt_submitted` hook whose prompt is not the nudge. Read My Mind drops its exact text (`isCapturablePrompt`). → [architecture-invariants#prompt-cache-keep-warm](docs/architecture-invariants.md#prompt-cache-keep-warm) +**Prompt-cache keep-warm** (`src/prompt-cache-keepwarm.ts`; per-session opt-in on the Respawn tab, app-wide default `promptCacheKeepWarm` + cap `promptCacheKeepWarmMaxHours` in App Settings → System → Automation, SYNCED, default OFF / 2 h; step 2 of Discussion #554): one keep-alive turn (`KEEP_WARM_NUDGE`, under 100 chars) five minutes before an idle claude session's cache would expire, so the next prompt reads the conversation from cache instead of re-writing it at 2x. Codeman owns the idle policy for these sessions: a launch with keep-warm on also carries `"idleCompaction": false`, folded LAST into the ONE `--settings` JSON (`session-cli-registry-bridge.ts`), decided once per create/respawn by `decideIdleCompactionOptOut()` (tmux-manager.ts: keep-warm on via `keepWarmOn()`, exporter `injected`, claude's `idleCompactionSetting` gate at 2.1.290, fail-closed like `--name`) and recorded as `MuxSession.idleCompactionOff`; a keep-warm-off launch stays byte-identical (`test/cli-registry-spawn-golden.test.ts`). Switched on after launch it is best-effort and the readout says idle compaction may still run until the next launch. ⚠️ The deadline is `Session.promptCache.expiresAt` from the statusline, never idle detection or the plan tier; a `5m` TTL is a hard stop, and so is a launch whose `statusLineExporter` is not `injected`. ⚠️ Every guard lives in the pure `decideKeepWarm()` (effective opt-in, where an agent-spawned session (`parentSessionId`) does not inherit the app default; exporter injected; warm; no `compactedAt` this gap; tokens at stake; live pane (pid AND no `paneExit`); no limit pause (auto-resume's, or a limit message on screen); no respawn/Ralph/task/waiter; inside the cap since `lastRealPromptAt`; idle; no dialog; EMPTY composer via `composerIsEmpty()`, continuation lines included, read from a STYLED capture so Claude Code's faint prompt suggestion is not taken for a draft; no recent keystroke); add a guard there, never in the service. ⚠️ The nudge IS a submit and moves `lastSubmitAt` (the response viewer's conversation correlation needs it); the window `beginKeepAliveTurn()` opens keeps it off `lastRealPromptAt` (the cap's own persisted clock) and opens its `stop`/`idle_prompt` hooks pre-acknowledged (`preAcknowledged` on `notePrompt`), so nothing alerts, and leaves an idle item already pending exactly as found (only a human spends an idle alert; `session:working`/`session:idle` carry `keepAlive: true` so the page's stuck timer ignores the turn); the keep-alive's own `idle_prompt` ends that window (`endKeepAliveTurn()`; `KEEP_WARM_TURN_WINDOW_MS` is only a ceiling, since a slow turn's idle prompt lands a minute after it ends), so does a nudge whose write failed (retried a minute later, never past the last-chance line), and so does any other submit: a `fromUser` write, a programmatic write of anything but the nudge, or a `prompt_submitted` hook whose prompt is not the nudge. Read My Mind drops its exact text (`isCapturablePrompt`). → [architecture-invariants#prompt-cache-keep-warm](docs/architecture-invariants.md#prompt-cache-keep-warm) **Orchestrator**: State machine that turns a user goal into a phased plan and drives it to completion: `idle → planning → approval → executing → verifying → (replanning) → completed/failed`. `OrchestratorLoop` (engine) delegates plan generation to `orchestrator-planner` and per-phase verification gates to `orchestrator-verifier`, executing phases via team agents/`task-queue`. State persists under the `orchestrator` key in `state.json`. Distinct from Ralph (single-session autonomous loop) — orchestrator coordinates multi-phase, multi-agent execution. See `docs/orchestrator-loop-architecture.md`. diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index 835d5279..374348ea 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -161,7 +161,7 @@ Further detail: the handheld OFF default comes from the mobile block in `getDefa ### Prompt-cache keep-warm -**Prompt-cache keep-warm** (`src/prompt-cache-keepwarm.ts`; per-session opt-in `keepWarmEnabled` on the Respawn tab, app-wide default `promptCacheKeepWarm` and cap `promptCacheKeepWarmMaxHours` under App Settings → System → Automation, both SYNCED, default OFF and 2 h; step 2 of Discussion #554): shortly before an idle claude session's prompt cache would expire, Codeman sends ONE keep-alive turn (`KEEP_WARM_NUDGE`, under 100 characters so Claude Code never reads the burst as a paste) so the next real prompt still reads the conversation from cache. Measured on two sessions of ~110k tokens: the nudge read 109,831 tokens from cache and wrote 70, replied `ok` in three seconds with no tool use, and a real turn seven minutes past the original expiry stayed a hit while the un-nudged control re-wrote 89,882 tokens with `last_miss_cause: ttl_expired_1h`; the payoff on real usage is concentrated in gaps of one to four hours, which is why there is a cap after the last real prompt rather than an indefinite loop (2 h by default: the maintainer's simulation over 30 days of gaps came out slightly cheaper than 3 h). ⚠️ **Codeman owns the idle policy for a keep-warm session.** Claude Code 2.1.289+ can compact an idle 1h-cache session by itself at 90% of the TTL (a server-flagged experiment, `fireAtFraction` tunable down to 0.5), which on the default timing always lands a minute before the nudge and leaves only a summary to keep warm. So a launch with keep-warm on carries `"idleCompaction": false`, folded LAST into the ONE `--settings` JSON with ultracode, the advisor and the statusLine exporter (`buildSpawnCommandFromRegistry`; never a second `--settings` flag, Claude Code takes one), so a keep-warm-off launch is byte-identical to one from before keep-warm (`test/cli-registry-spawn-golden.test.ts`). The `--settings` flag outranks user, project and local settings, which is why the key is conditional: everywhere else it would take Claude Code's own behaviour away with no way back. `decideIdleCompactionOptOut()` (tmux-manager.ts, pure) decides it once per create and respawn: keep-warm on (`keepWarmOn()`: the session's choice, else the app default, read fresh by `readKeepWarmSettings()`), the exporter `injected` (without `prompt_cache` reports keep-warm cannot run, and the opt-out alone would only cost the user the idle compaction), and claude's `idleCompactionSetting` gate (`capabilities.gates`, 2.1.290, the first release whose settings schema has the key; 2.1.289 already idle-compacts with no opt-out), fail-closed through `launchGatePasses()` exactly like `--name`: an unknown version means no key, and a log line says so. The result is recorded on the mux record (`MuxSession.idleCompactionOff`, like `statusLine`, persisted in mux-sessions.json) and published in the keep-warm status as `idleCompactionOff`; a session switched on after launch is kept warm best-effort (a relaunch to install the opt-out would cost exactly the re-write being avoided) and its readout says Claude Code's idle compaction may still run until its next launch, which any relaunch (dead-pane recovery, a custom-model `restartCli()`) fixes for free since every relaunch re-decides. ⚠️ The deadline is the statusline's `prompt_cache.expires_at` (`Session.promptCache`), never idle detection or the plan tier, and a `5m` lifetime is a hard stop (a nudge every four minutes costs more than the miss it prevents; API-key users get the hour from `CLAUDE_CODE_PROMPT_CACHE_TTL=1h` in `envOverrides`). ⚠️ `decideKeepWarm()` is the ONE rulebook and every guard lives in it, tested as a table: the effective opt-in (`keepWarmOn()`; a session an agent spawned, `parentSessionId`, does not inherit the app default, since an idle worker that reported done looks exactly like one that will be prompted again and most never are; its own switch still works), the launch's `statusLineExporter` is `injected` (which also stops every CLI without cache telemetry; the toggle line says why in the readout's own `PROMPT_CACHE_NOT_COLLECTED` words), a warm cache with an expiry, no `compactedAt` this gap (a compaction already replaced the context, and the stamp is dropped at the next request, which re-opens the next gap), not a `5m` lifetime, at least `KEEP_WARM_MIN_TOKENS` at stake, a live pane (a pid AND no recorded `paneExit`: an agent that exited in a `remain-on-exit` pane keeps its pid), no usage-limit pause (auto-resume's, or, read off the same pane capture as the composer, a usage-limit message whose reset is still ahead, since auto-resume tracks pauses only while it is on), no automation (respawn running, a Ralph loop active, a task assigned, or an agent long-polling `wait`), inside the cap measured from `lastRealPromptAt` (else `createdAt`), due at `expires_at` minus `KEEP_WARM_LEAD_MS` but not within `KEEP_WARM_LAST_CHANCE_MS`, idle, no open permission or question dialog, an EMPTY composer (`composerIsEmpty()` in session-submit-verifier.ts reads the last glyph line AND its continuation lines down to the composer's bottom rule, since a draft that opens with a newline leaves the glyph line blank: the nudge's paste would join a draft, and a missing rule is never read as empty). ⚠️ The composer is read from a STYLED capture (`capturePaneText(…, { styles: true })`, `capture-pane -p -e`, still one line per row; never `capturePaneBuffer()`, whose visible frame is a cursor-addressed repaint with no line breaks): Claude Code's prompt suggestion (on by default) fills an empty composer with faint (SGR 2) text whose first letter sits in the reverse-video cursor cell, and only that styling tells it from a typed draft, so `composerIsEmpty()` drops faint runs and a reverse cell followed by faint text. The paste replaces a suggestion rather than joining it (measured on Claude Code 2.1.300), and no human keystroke within `KEEP_WARM_RECENT_INPUT_MS`. Transient guards (working, a dialog, automation, an unreadable or non-empty composer, a recent keystroke) defer by `KEEP_WARM_RETRY_MS` and never past the last-chance line, an evaluation that throws re-arms the same retry, and a trigger landing during an evaluation queues one follow-up after it instead of being dropped, and `remove()` cancels both the follow-up and the retry; the composer is read only once everything else says "due". ⚠️ **Two submit clocks.** The nudge goes out through a plain `writeViaMux()` after `beginKeepAliveTurn()`. It IS a submit, so it moves `lastSubmitAt` from the write and again from its `prompt_submitted` hook (`markPromptSubmitted()`): the response viewer's conversation correlation (`resolveActiveClaudeSessionIdFromHistory()`) matches that stamp against Claude's history, which records the nudge like any prompt. The cap needs the last REAL prompt instead, so `Session._stampSubmit()` also moves `lastRealPromptAt` for every submit outside the keep-alive window (a person in the browser, an agent over the API, a typist in an attached tmux client seen through the hook), persisted in `SessionState.lastRealPromptAt` and restored by both rebuild paths so a restart never reopens the window. The window closes on the keep-alive's own `idle_prompt` (the hook route calls `endKeepAliveTurn()`; Claude Code sends it about a minute after the turn ends, so a fixed short window would let a slow turn's idle prompt alert and push, and `KEEP_WARM_TURN_WINDOW_MS` is only a ceiling for a workspace whose hooks never report). Any other submit closes it at once: a `fromUser` write, a programmatic write carrying anything but the nudge or a bare Enter (an agent, respawn, auto-resume; `isKeepAliveWrite()` in session.ts), or a `prompt_submitted` hook whose `prompt` is not the exact nudge (the route hands the hook's prompt text to `markPromptSubmitted()`), so a person's prompt right after a nudge, typed in the browser or in an attached tmux client, counts and alerts normally. The home screens and the tile grid read `lastSubmitAt` only to anchor a WORKING row, so a nudge shows there only for the seconds its turn runs; idle rows key on `lastActivityAt`, which the nudge's reply moves like any output. ⚠️ The same window makes the turn's `stop` hook and `idle_prompt` open pre-acknowledged (`preAcknowledged: 'cache keep-alive'` through `notePrompt()`, the same path `watching` uses), so no tab alert, toast, push or webhook fires. ⚠️ Only a human spends an idle alert, so the keep-alive turn leaves an idle item already pending exactly as it found it: the `working` transition's idle resolve (session-listener-wiring.ts) and the `stop` resolve skip it, and the keep-alive's `idle_prompt` does not replace it (a replacement would forget that it was unseen, or that a person had seen it), while the page's `_onHookStop` returns on `acknowledgedReason` before clearing anything; `session:working` and `session:idle` carry `keepAlive: true` for the turn, so the page's stuck-detection timer is neither cleared nor re-armed by it. Read My Mind drops the exact nudge text in `isCapturablePrompt()`, the filter `captureIntentPrompt()` → `intentStore.recordPrompt()` runs every transcript prompt through. ⚠️ Re-evaluation is edge-triggered and cheap: every cache report (`status-telemetry-routes.ts`), the toggle route, a settings change (`rescheduleAll()`, which also drops the service's in-memory copy of the default and cap, so a cache report never reads settings.json), `setupSessionListeners()` and the session's `idle` event (`keepWarmSchedule` in session-listener-wiring.ts, which is also what arms a restored report once the recovered pane is attached); the service keeps one timer per session (clamped to the 2^31-1 ms `setTimeout` honours, so a far-off expiry cannot turn into a 1 ms loop), `remove()`s it with the session and stops with the server, and a `remove()` or `stop()` landing during an evaluation's await cancels it there (nothing sent, armed or published after), and the server logs one `[KeepWarm]` line per status change naming the guard that holds it. Status rides `Session.keepWarm` in `toState()` and the change-only `session:keepWarm` event for the readout; `keepWarmEnabled`, `lastRealPromptAt` and the last `promptCache` report are restored by the constructor on both rebuild paths (that is what re-arms the timer after a restart), the status never is. Tests: `test/prompt-cache-keepwarm.test.ts`, `test/keep-warm-launch.test.ts`, `test/keep-warm-i18n.test.ts`, `test/routes/session-routes-keep-warm.test.ts`, the keep-warm readout in `test/prompt-cache-status.test.ts`, the opt-out literals in `test/cli-registry-spawn-golden.test.ts`, the keep-alive cases in `test/routes/hook-event-routes.test.ts` and `test/session-listener-wiring.test.ts`, and the page handlers in `test/keep-alive-page-alerts.test.ts`. +**Prompt-cache keep-warm** (`src/prompt-cache-keepwarm.ts`; per-session opt-in `keepWarmEnabled` on the Respawn tab, app-wide default `promptCacheKeepWarm` and cap `promptCacheKeepWarmMaxHours` under App Settings → System → Automation, both SYNCED, default OFF and 2 h; step 2 of Discussion #554): shortly before an idle claude session's prompt cache would expire, Codeman sends ONE keep-alive turn (`KEEP_WARM_NUDGE`, under 100 characters so Claude Code never reads the burst as a paste) so the next real prompt still reads the conversation from cache. Measured on two sessions of ~110k tokens: the nudge read 109,831 tokens from cache and wrote 70, replied `ok` in three seconds with no tool use, and a real turn seven minutes past the original expiry stayed a hit while the un-nudged control re-wrote 89,882 tokens with `last_miss_cause: ttl_expired_1h`; the payoff on real usage is concentrated in gaps of one to four hours, which is why there is a cap after the last real prompt rather than an indefinite loop (2 h by default: the maintainer's simulation over 30 days of gaps came out slightly cheaper than 3 h). ⚠️ **Codeman owns the idle policy for a keep-warm session.** Claude Code 2.1.289+ can compact an idle 1h-cache session by itself at 90% of the TTL (a server-flagged experiment, `fireAtFraction` tunable down to 0.5), which on the default timing always lands a minute before the nudge and leaves only a summary to keep warm. So a launch with keep-warm on carries `"idleCompaction": false`, folded LAST into the ONE `--settings` JSON with ultracode, the advisor and the statusLine exporter (`buildSpawnCommandFromRegistry`; never a second `--settings` flag, Claude Code takes one), so a keep-warm-off launch is byte-identical to one from before keep-warm (`test/cli-registry-spawn-golden.test.ts`). The `--settings` flag outranks user, project and local settings, which is why the key is conditional: everywhere else it would take Claude Code's own behaviour away with no way back. `decideIdleCompactionOptOut()` (tmux-manager.ts, pure) decides it once per create and respawn: keep-warm on (`keepWarmOn()`: the session's choice, else the app default, read fresh by `readKeepWarmSettings()`), the exporter `injected` (without `prompt_cache` reports keep-warm cannot run, and the opt-out alone would only cost the user the idle compaction), and claude's `idleCompactionSetting` gate (`capabilities.gates`, 2.1.290, the first release whose settings schema has the key; 2.1.289 already idle-compacts with no opt-out), fail-closed through `launchGatePasses()` exactly like `--name`: an unknown version means no key, and a log line says so. The result is recorded on the mux record (`MuxSession.idleCompactionOff`, like `statusLine`, persisted in mux-sessions.json) and published in the keep-warm status as `idleCompactionOff`; a session switched on after launch is kept warm best-effort (a relaunch to install the opt-out would cost exactly the re-write being avoided) and its readout says Claude Code's idle compaction may still run until its next launch, which any relaunch (dead-pane recovery, a custom-model `restartCli()`) fixes for free since every relaunch re-decides. ⚠️ The deadline is the statusline's `prompt_cache.expires_at` (`Session.promptCache`), never idle detection or the plan tier, and a `5m` lifetime is a hard stop (a nudge every four minutes costs more than the miss it prevents; API-key users get the hour from `CLAUDE_CODE_PROMPT_CACHE_TTL=1h` in `envOverrides`). ⚠️ `decideKeepWarm()` is the ONE rulebook and every guard lives in it, tested as a table: the effective opt-in (`keepWarmOn()`; a session an agent spawned, `parentSessionId`, does not inherit the app default, since an idle worker that reported done looks exactly like one that will be prompted again and most never are; its own switch still works), the launch's `statusLineExporter` is `injected` (which also stops every CLI without cache telemetry; the toggle line says why in the readout's own `PROMPT_CACHE_NOT_COLLECTED` words), a warm cache with an expiry, no `compactedAt` this gap (a compaction already replaced the context, and the stamp is dropped at the next request, which re-opens the next gap), not a `5m` lifetime, at least `KEEP_WARM_MIN_TOKENS` at stake, a live pane (a pid AND no recorded `paneExit`: an agent that exited in a `remain-on-exit` pane keeps its pid), no usage-limit pause (auto-resume's, or, read off the same pane capture as the composer, a usage-limit message whose reset is still ahead, since auto-resume tracks pauses only while it is on), no automation (respawn running, a Ralph loop active, a task assigned, or an agent long-polling `wait`), inside the cap measured from `lastRealPromptAt` (else `createdAt`), due at `expires_at` minus `KEEP_WARM_LEAD_MS` but not within `KEEP_WARM_LAST_CHANCE_MS`, idle, no open permission or question dialog, an EMPTY composer (`composerIsEmpty()` in session-submit-verifier.ts reads the last glyph line AND its continuation lines down to the composer's bottom rule, since a draft that opens with a newline leaves the glyph line blank: the nudge's paste would join a draft, and a missing rule is never read as empty). ⚠️ The composer is read from a STYLED capture (`capturePaneText(…, { styles: true })`, `capture-pane -p -e`, still one line per row; never `capturePaneBuffer()`, whose visible frame is a cursor-addressed repaint with no line breaks): Claude Code's prompt suggestion (on by default) fills an empty composer with faint (SGR 2) text whose first letter sits in the reverse-video cursor cell, and only that styling tells it from a typed draft, so `composerIsEmpty()` drops faint runs and a reverse cell followed by faint text. The paste replaces a suggestion rather than joining it (measured on Claude Code 2.1.300), and no human keystroke within `KEEP_WARM_RECENT_INPUT_MS`. Transient guards (working, a dialog, automation, an unreadable or non-empty composer, a recent keystroke) defer by `KEEP_WARM_RETRY_MS` and never past the last-chance line, and so does a nudge whose write fails, which also closes the keep-alive window at once (`endKeepAliveTurn()`, since no turn is running; a check a whole `KEEP_WARM_LEAD_MS` later would land past the expiry and stop), an evaluation that throws re-arms the same retry, and a trigger landing during an evaluation queues one follow-up after it instead of being dropped, and `remove()` cancels both the follow-up and the retry; the composer is read only once everything else says "due". ⚠️ **Two submit clocks.** The nudge goes out through a plain `writeViaMux()` after `beginKeepAliveTurn()`. It IS a submit, so it moves `lastSubmitAt` from the write and again from its `prompt_submitted` hook (`markPromptSubmitted()`): the response viewer's conversation correlation (`resolveActiveClaudeSessionIdFromHistory()`) matches that stamp against Claude's history, which records the nudge like any prompt. The cap needs the last REAL prompt instead, so `Session._stampSubmit()` also moves `lastRealPromptAt` for every submit outside the keep-alive window (a person in the browser, an agent over the API, a typist in an attached tmux client seen through the hook), persisted in `SessionState.lastRealPromptAt` and restored by both rebuild paths so a restart never reopens the window. The window closes on the keep-alive's own `idle_prompt` (the hook route calls `endKeepAliveTurn()`; Claude Code sends it about a minute after the turn ends, so a fixed short window would let a slow turn's idle prompt alert and push, and `KEEP_WARM_TURN_WINDOW_MS` is only a ceiling for a workspace whose hooks never report). Any other submit closes it at once: a `fromUser` write, a programmatic write carrying anything but the nudge or a bare Enter (an agent, respawn, auto-resume; `isKeepAliveWrite()` in session.ts), or a `prompt_submitted` hook whose `prompt` is not the exact nudge (the route hands the hook's prompt text to `markPromptSubmitted()`), so a person's prompt right after a nudge, typed in the browser or in an attached tmux client, counts and alerts normally. The home screens and the tile grid read `lastSubmitAt` only to anchor a WORKING row, so a nudge shows there only for the seconds its turn runs; idle rows key on `lastActivityAt`, which the nudge's reply moves like any output. ⚠️ The same window makes the turn's `stop` hook and `idle_prompt` open pre-acknowledged (`preAcknowledged: 'cache keep-alive'` through `notePrompt()`, the same path `watching` uses), so no tab alert, toast, push or webhook fires. ⚠️ Only a human spends an idle alert, so the keep-alive turn leaves an idle item already pending exactly as it found it: the `working` transition's idle resolve (session-listener-wiring.ts) and the `stop` resolve skip it, and the keep-alive's `idle_prompt` does not replace it (a replacement would forget that it was unseen, or that a person had seen it), while the page's `_onHookStop` returns on `acknowledgedReason` before clearing anything; `session:working` and `session:idle` carry `keepAlive: true` for the turn, so the page's stuck-detection timer is neither cleared nor re-armed by it. Read My Mind drops the exact nudge text in `isCapturablePrompt()`, the filter `captureIntentPrompt()` → `intentStore.recordPrompt()` runs every transcript prompt through. ⚠️ Re-evaluation is edge-triggered and cheap: every cache report (`status-telemetry-routes.ts`), the toggle route, a settings change (`rescheduleAll()`, which also drops the service's in-memory copy of the default and cap, so a cache report never reads settings.json), `setupSessionListeners()` and the session's `idle` event (`keepWarmSchedule` in session-listener-wiring.ts, which is also what arms a restored report once the recovered pane is attached); the service keeps one timer per session (clamped to the 2^31-1 ms `setTimeout` honours, so a far-off expiry cannot turn into a 1 ms loop), `remove()`s it with the session and stops with the server, and a `remove()` or `stop()` landing during an evaluation's await cancels it there (nothing sent, armed or published after), and the server logs one `[KeepWarm]` line per status change naming the guard that holds it. Status rides `Session.keepWarm` in `toState()` and the change-only `session:keepWarm` event for the readout; `keepWarmEnabled`, `lastRealPromptAt` and the last `promptCache` report are restored by the constructor on both rebuild paths (that is what re-arms the timer after a restart), the status never is. Tests: `test/prompt-cache-keepwarm.test.ts`, `test/keep-warm-launch.test.ts`, `test/keep-warm-i18n.test.ts`, `test/routes/session-routes-keep-warm.test.ts`, the keep-warm readout in `test/prompt-cache-status.test.ts`, the opt-out literals in `test/cli-registry-spawn-golden.test.ts`, the keep-alive cases in `test/routes/hook-event-routes.test.ts` and `test/session-listener-wiring.test.ts`, and the page handlers in `test/keep-alive-page-alerts.test.ts`. ### Cron jobs diff --git a/docs/wiki/Keeping-Agents-Running.md b/docs/wiki/Keeping-Agents-Running.md index 927e1011..905f215f 100644 --- a/docs/wiki/Keeping-Agents-Running.md +++ b/docs/wiki/Keeping-Agents-Running.md @@ -111,7 +111,7 @@ Claude Code 2.1.251 and newer reports the cache state on the statusline after ev **Keep the cache warm while idle** is the switch under the readout. With it on, Codeman sends a one-line keep-alive turn (`Codeman cache keep-alive: no action needed, reply with just "ok".`) about five minutes before the cache would expire. Claude answers `ok`, the request refreshes the cache for another hour, and your next real prompt reads the conversation from cache instead of re-processing it. Measured on sessions of about 110k tokens, the keep-alive cost a cache read plus a one-word reply, a small fraction of the re-write it prevented. -It only fires when there is nothing else going on: the session is idle with an empty prompt (Claude Code's greyed-out prompt suggestion counts as empty), no permission or question dialog is open, nobody has typed in the last few minutes, no respawn loop, Ralph loop or waiting agent is driving the session, and the session is not paused on a usage limit. If any of those hold it waits a minute and looks again, and gives up a minute before the expiry rather than race it. It also stops by itself a few hours after your last prompt (**Keep warm for at most**, under App Settings → System → Automation, default 2 hours), because the saving is concentrated in gaps of one to four hours and an overnight run of hourly keep-alives would cost more than the single cold turn in the morning. It stands down for the rest of a gap once the context has been compacted. The readout's Status line gains a `Keep-alive:` line that says what it is doing: the next keep-alive and when the window closes, waiting and why, sent at a time with a count, or stopped and why. A session started without the cache readout (Plan Usage off at launch, a workspace with its own statusLine, a remote or Docker session) cannot be kept warm: the switch is greyed out and cannot be switched on (one that is already on can still be switched off), and the Status line says why. +It only fires when there is nothing else going on: the session is idle with an empty prompt (Claude Code's greyed-out prompt suggestion counts as empty), no permission or question dialog is open, nobody has typed in the last few minutes, no respawn loop, Ralph loop or waiting agent is driving the session, and the session is not paused on a usage limit. If any of those hold it waits a minute and looks again, and gives up a minute before the expiry rather than race it; a keep-alive that could not be sent is retried the same way. It also stops by itself a few hours after your last prompt (**Keep warm for at most**, under App Settings → System → Automation, default 2 hours), because the saving is concentrated in gaps of one to four hours and an overnight run of hourly keep-alives would cost more than the single cold turn in the morning. It stands down for the rest of a gap once the context has been compacted. The readout's Status line gains a `Keep-alive:` line that says what it is doing: the next keep-alive and when the window closes, waiting and why, sent at a time with a count, or stopped and why. A session started without the cache readout (Plan Usage off at launch, a workspace with its own statusLine, a remote or Docker session) cannot be kept warm: the switch is greyed out and cannot be switched on (one that is already on can still be switched off), and the Status line says why. Recent Claude Code builds can compact a large idle session on their own shortly before the cache expires, which would leave only a summary to keep warm. So a session **started** with keep-warm on also launches with Claude Code's idle compaction switched off (Claude Code 2.1.290 or newer); Codeman then owns the idle policy for it. A session you switch on after it started is still kept warm, but Claude Code may compact it first until its next launch, and the Status line says so; one you switch off keeps Claude Code's idle compaction off until its next launch, and the Status line says that too. Sessions with keep-warm off launch exactly as before. diff --git a/src/prompt-cache-keepwarm.ts b/src/prompt-cache-keepwarm.ts index f721e16a..7d83461b 100644 --- a/src/prompt-cache-keepwarm.ts +++ b/src/prompt-cache-keepwarm.ts @@ -173,6 +173,8 @@ export interface KeepWarmSessionView { readonly lastRealPromptAt: number; isIdle(): boolean; beginKeepAliveTurn(windowMs: number): void; + /** Close the window `beginKeepAliveTurn()` opened, when the nudge never went out. */ + endKeepAliveTurn(): void; writeViaMux(data: string): Promise; } @@ -368,8 +370,16 @@ export class PromptCacheKeepWarmService implements KeepWarmScheduler { this.ports.log?.( `[KeepWarm] ${ok ? 'nudged' : 'failed to nudge'} ${id.slice(0, 8)} with ${facts.cache?.recacheTokensIfCold ?? '?'} tokens at stake (nudge ${tally.nudges})` ); - // The nudge's own cache report re-schedules; this timer is the fallback if none arrives. - const nextCheckAt = Date.now() + KEEP_WARM_LEAD_MS; + // Sent: the nudge's own cache report re-schedules, and this timer is the fallback if none + // arrives. Not sent: no turn is running, so the window closes now rather than at its + // ceiling, and the retry lands inside the last minutes before the expiry, where a nudge + // can still go out (a lead time later would be past it). + let nextCheckAt = Date.now() + KEEP_WARM_LEAD_MS; + if (!ok) { + session.endKeepAliveTurn(); + const expiresAt = facts.cache?.expiresAt ?? 0; + nextCheckAt = Math.min(Date.now() + KEEP_WARM_RETRY_MS, expiresAt - KEEP_WARM_LAST_CHANCE_MS); + } this.arm(id, nextCheckAt); this.ports.publish(id, { ...base, diff --git a/test/prompt-cache-keepwarm.test.ts b/test/prompt-cache-keepwarm.test.ts index 4ad0f9e7..3dafa356 100644 --- a/test/prompt-cache-keepwarm.test.ts +++ b/test/prompt-cache-keepwarm.test.ts @@ -337,6 +337,10 @@ class FakeSession implements KeepWarmSessionView { beginKeepAliveTurn(windowMs: number): void { this.keepAliveWindows.push(windowMs); } + keepAliveWindowsEnded = 0; + endKeepAliveTurn(): void { + this.keepAliveWindowsEnded += 1; + } async writeViaMux(data: string): Promise { this.writes.push(data); return this.writeOk; @@ -474,7 +478,12 @@ describe('PromptCacheKeepWarmService', () => { service.stop(); }); - it('keeps a failed write out of the tally and retries after the lead time', async () => { + /** + * A nudge only goes out inside the last five minutes before the expiry, so a retry armed a + * whole lead time later would land past it and stop: the failed write ends the keep-alive + * window (no turn is running) and retries a minute later, never past the last-chance line. + */ + it('keeps a failed write out of the tally, ends the keep-alive window and retries a minute later', async () => { const session = new FakeSession(); session.keepWarmEnabled = true; session.writeOk = false; @@ -484,11 +493,38 @@ describe('PromptCacheKeepWarmService', () => { service.schedule(session.id); await settle(); expect(session.writes).toHaveLength(1); + expect(session.keepAliveWindowsEnded).toBe(1); expect(last()).toMatchObject({ state: 'waiting', nudges: 0, + nextCheckAt: NOW + KEEP_WARM_RETRY_MS, reason: expect.stringContaining('could not be written'), }); + + session.writeOk = true; + await vi.advanceTimersByTimeAsync(KEEP_WARM_RETRY_MS); + await settle(); + expect(session.writes).toEqual([`${KEEP_WARM_NUDGE}\r`, `${KEEP_WARM_NUDGE}\r`]); + expect(session.keepAliveWindowsEnded).toBe(1); + expect(last()).toMatchObject({ state: 'nudged', nudges: 1 }); + service.stop(); + }); + + it('never arms the retry after a failed write past the last-chance line', async () => { + const session = new FakeSession(); + session.keepWarmEnabled = true; + session.writeOk = false; + // Due, with only 30 s left before the last-chance line. + session.promptCache = warmCache(NOW + KEEP_WARM_LAST_CHANCE_MS + 30_000); + const { ports, last } = makePorts(session); + const service = new PromptCacheKeepWarmService(ports); + await service.schedule(session.id); + expect(last()).toMatchObject({ state: 'waiting', nextCheckAt: NOW + 30_000 }); + await vi.advanceTimersByTimeAsync(30_000); + await settle(); + expect(session.writes).toHaveLength(1); + expect(last()).toMatchObject({ state: 'stopped', reason: 'too close to expiry to land a turn' }); + expect(vi.getTimerCount()).toBe(0); service.stop(); });