fix(edit-history): review round 2 — whole lines around skipped ones, full-budget reads, orphan sweep

- Forward: after a line longer than the budget, the next line is read from
  its start instead of being skipped with it when it crosses the chunk end.
- Backward: the line before a skipped long line is read whole; a floor that
  turned out to be a line start no longer stalls.
- A chunk the shared forward budget cut short is not applied (it read as one
  over-long line) but read in a later pass, so a first open of a long main
  transcript reaches the subagents in the same sync.
- Load older reads a full budget block of up to three transcripts per click,
  taking turns, instead of splitting one budget (a normal Write longer than
  the share was skipped for good).
- A subagent transcript that vanished loses its cursor, so Load older does not
  stay on forever.
- POST /api/cleanup-state forgets the history of every record it drops; a
  startup sweep deletes edit-history files no session or record owns and
  stale tmp files of other processes.
- load() rebuilds entries from known fields (string diff lines, per-edit cap
  re-applied, ids counted against the byte cap), bounds pending Bash fields
  and agent ids, and does not parse a file over 32 MiB.
- No re-read after Codeman's own keep-alive turn.
- Tests: regression tests for each of the above (they fail on the previous
  commit); the remove-during-write test now meets the write under way; the
  sync and route tests remove their stores before the temp dir, so no late
  save recreates it in /tmp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
RandalixandClaude Opus 5.5 committed 2026-10-23 15:00:43 +02:00
1 parent f5ce80dce2
commit 321268754a
15 files changed
+540 -89

No files matched your search

+1 -1
View File
@@ -232,7 +232,7 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph
**Git status indicator** (`showGitStatus`, per-device, default OFF; `src/git-workspace-status.ts`, `routes/git-status-routes.ts`, `git-status-ui.js`): the bottom-bar indicator and its panel read `GET /api/sessions/:id/git-status` and `/git-diff`, read-only and offline (it never fetches and never writes). ⚠️ git never runs on a repository at or inside a Docker case workspace (walk-up, scan and diff alike, and a cached repository list is re-checked against the CURRENT Docker roots), since a container could plant a clean filter that runs on the host; remote and Docker sessions answer `unsupported`. ⚠️ `git-diff` takes `repo`/`path`/`kind` only as keys matched against the workspace's own repository list (`findWorkspaceRepo()`) and that repository's current status, never as paths. ⚠️ Keep `--no-optional-locks`, `core.fsmonitor=false` and `log.showSignature=false` on every call and `--no-ext-diff --no-textconv` on diffs; clean filters still run, which is why the Docker rule exists. ⚠️ An enclosing repository at `$HOME` or above is ignored (`isUnrelatedAncestor()`), and is identified by one cached `rev-parse` before any full status runs. Everything git supplies renders via `textContent`. Tests: `test/git-workspace-status.test.ts`, `test/routes/git-status-routes.test.ts`, and `test/git-status.browser.test.ts` (browser suite, not in the gate).
**Edit history panel** (`showEditHistory`, per-device, default OFF; `src/edit-history-parse.ts`, `edit-history-store.ts`, `edit-history-local-source.ts`, `web/edit-history-sync.ts`, `routes/edit-history-routes.ts`, `edit-history-ui.js`): which files a LOCAL claude session edited (main agent and subagents, labelled), each edit's diff, and images a Bash call produced. Read from the session's transcripts by byte offset, never from git. `GET /api/sessions/:id/edits[?sync=1]`, `GET …/edits/:entryId` (unified diff), `POST …/edits/older`. ⚠️ **Reads only on request:** the panel sends `?sync=1` when it opens and, while it is open on that session, after an edit-capable `transcript:tool_end` or the session's `session:idle` (debounced; `transcript:complete` never fires for interactive claude, its transcript has no `result` line). The server never syncs on its own: no hook, no timer, no second watcher; `server.ts` hands the reader to the routes and only calls `remove` (via `forgetEditHistory`) and `flushAll` itself (wiring-guard test). The main transcript is the path the session's `TranscriptWatcher` follows (reported by the hooks), else `<id>.jsonl` under the session's `CLAUDE_CONFIG_DIR` override or the server's (`resolveClaudeTranscript`); a config dir set outside Codeman is found with the next hook, until then `unavailable` / `no-transcript`. ⚠️ The store holds every diff, secrets included: `dataPath('edit-history')/<id>.json`, 0600 via tmp + rename, capped per session (500 entries, 4 MiB diff text; damaged files are re-bounded on load), deleted with the session's persisted record (`forgetEditHistory`: kill, persisted-only DELETE, stale cleanup; a pinned killed session keeps it), blocked in `isSensitivePath`, never fed to `/api/search`. Images use the existing file surfaces only: thumbnails via `file-raw` inside the workspace; an image outside it has no thumbnail and opens in the file preview (attachment route, `notify: false`); no route here serves a file. A Bash image counts only when the file's mtime lies in the call's window (±2 s) and no entry already covers that file version. Remote, Docker and non-claude sessions answer `unsupported`. Tests: `test/edit-history-*.test.ts`, `test/routes/edit-history-routes.test.ts`, `test/edit-history.browser.test.ts` (browser suite).
**Edit history panel** (`showEditHistory`, per-device, default OFF; `src/edit-history-parse.ts`, `edit-history-store.ts`, `edit-history-local-source.ts`, `web/edit-history-sync.ts`, `routes/edit-history-routes.ts`, `edit-history-ui.js`): which files a LOCAL claude session edited (main agent and subagents, labelled), each edit's diff, and images a Bash call produced. Read from the session's transcripts by byte offset, never from git. `GET /api/sessions/:id/edits[?sync=1]`, `GET …/edits/:entryId` (unified diff), `POST …/edits/older`. ⚠️ **Reads only on request:** the panel sends `?sync=1` when it opens and, while it is open on that session, after an edit-capable `transcript:tool_end` or the session's `session:idle` (debounced; `transcript:complete` never fires for interactive claude, its transcript has no `result` line). The server never syncs on its own: no hook, no timer, no second watcher; `server.ts` hands the reader to the routes and only calls `remove` (via `forgetEditHistory`), `sweep` (startup) and `flushAll` itself (wiring-guard test). The main transcript is the path the session's `TranscriptWatcher` follows (reported by the hooks), else `<id>.jsonl` under the session's `CLAUDE_CONFIG_DIR` override or the server's (`resolveClaudeTranscript`); a config dir set outside Codeman is found with the next hook, until then `unavailable` / `no-transcript`. ⚠️ The store holds every diff, secrets included: `dataPath('edit-history')/<id>.json`, 0600 via tmp + rename, capped per session (500 entries, 4 MiB diff text; damaged files are re-bounded on load), deleted with the session's persisted record (`forgetEditHistory`: kill, persisted-only DELETE, stale cleanup at startup and via `POST /api/cleanup-state`; a pinned killed session keeps it; the startup `sweep` removes files no session owns), blocked in `isSensitivePath`, never fed to `/api/search`. Images use the existing file surfaces only: thumbnails via `file-raw` inside the workspace; an image outside it has no thumbnail and opens in the file preview (attachment route, `notify: false`); no route here serves a file. A Bash image counts only when the file's mtime lies in the call's window (±2 s) and no entry already covers that file version. Remote, Docker and non-claude sessions answer `unsupported`. Tests: `test/edit-history-*.test.ts`, `test/routes/edit-history-routes.test.ts`, `test/edit-history.browser.test.ts` (browser suite).
**Raw file bodies are streamed and range-aware**: `file-raw`, the attachments `/raw` route, `GET /api/download` and the rendered-HTML route `GET /html-view/<cap>/...` share `sendFileBody()`, advertise `Accept-Ranges: bytes` and answer `Range` with `206` + `Content-Range` (single-range, parser in `src/web/http-range.ts`); without it `<video>` cannot seek. The size cap (`MAX_FILE_DOWNLOAD_BYTES`, default 2GB, env `CODEMAN_MAX_DOWNLOAD_BYTES`, `0` = unlimited) is a sanity bound, not memory protection; never reintroduce a whole-file buffer. ⚠️ Bodies go out via `reply.hijack()`, so `sendRawStream` must copy the status onto `reply.raw` by hand or a partial body ships as `200`. ⚠️ Closing the preview must pause and unload media (`_stopFilePreviewMedia`), since a detached `HTMLMediaElement` keeps playing. → [architecture-invariants#raw-file-bodies-streamed-and-range-aware](docs/architecture-invariants.md#raw-file-bodies-streamed-and-range-aware)
+3 -3
View File
@@ -1087,13 +1087,13 @@ Lists are capped (300 files and 50 commits per repository) while the counts stay
What the Edit history panel reads (Settings → Header & Panels → Header, per-device, default off): the edits a **local claude session** made, main agent and subagents, read from its transcripts. Read-only. Every route resolves the session through `findSessionOrFail` (another user's session is `404`). Remote, Docker and non-claude sessions answer `state: 'unsupported'` and are never read.
`GET /api/sessions/:id/edits[?sync=1]` lists the stored edits, newest first, without diffs: `{ state, reason?, backfillTruncated, agents, entries }`. With `sync=1` it first reads what the transcripts gained since the last read; without it the server reads nothing. `state` is `ok`, `unavailable` (stored entries only; `reason` is `no-transcript` before the session has a transcript Codeman can find, or the read error; the transcript is the one the session's hooks report, else the newest `<id>.jsonl` under the session's `CLAUDE_CONFIG_DIR` override or the server's config dir) or `unsupported` (`reason`: `remote`, `docker`, `mode`). Each entry is `{ id, ts, path, relPath, kind: 'code'|'image', tool, created, added, removed, truncated, agentId? }`, where `tool` is the edit tool, `bash` for an image a Bash call produced or `sent` for one the agent sent; `relPath` is relative to the workspace (a subagent's worktree for its edits). `agents` maps a subagent id to `{ description, agentType, worktreeBranch? }`. `backfillTruncated: true` means older edits can still be loaded.
`GET /api/sessions/:id/edits[?sync=1]` lists the stored edits, newest first, without diffs: `{ state, reason?, backfillTruncated, agents, entries }`. With `sync=1` it first reads what the transcripts gained since the last read; without it the server reads nothing. `state` is `ok`, `unavailable` (stored entries only; `reason` is `no-transcript` before the session has a transcript Codeman can find, `removed` once the session's history was deleted, or the read error; the transcript is the one the session's hooks report, else the newest `<id>.jsonl` under the session's `CLAUDE_CONFIG_DIR` override or the server's config dir) or `unsupported` (`reason`: `remote`, `docker`, `mode`). Each entry is `{ id, ts, path, relPath, kind: 'code'|'image', tool, created, added, removed, truncated, agentId? }`, where `tool` is the edit tool, `bash` for an image a Bash call produced or `sent` for one the agent sent; `relPath` is relative to the workspace (a subagent's worktree for its edits). `agents` maps a subagent id to `{ description, agentType, worktreeBranch? }`. `backfillTruncated: true` means older edits can still be loaded.
`GET /api/sessions/:id/edits/:entryId` returns one edit's diff as unified text: `{ diff, truncated }` (each edit's diff is capped at 64 KiB). `404` for an edit the session does not have.
`POST /api/sessions/:id/edits/older` reads the next older block of every transcript (main and subagents share the read budget) and answers like the list.
`POST /api/sessions/:id/edits/older` reads the next older block of up to three transcripts, a full read budget each (with more transcripts, main and subagents take turns), and answers like the list.
The edits are stored per session in `~/.codeman/edit-history/<id>.json` (mode 0600, at most 500 edits and 4 MiB of diff text, the oldest dropped), deleted when the session's record is removed (a killed pinned session keeps it) and never fed into `/api/v1/search`. No route here serves a file: the panel shows thumbnails through `file-raw` inside the workspace only; an image outside it opens in the file preview, which registers it through the attachment route.
The edits are stored per session in `~/.codeman/edit-history/<id>.json` (mode 0600, at most 500 edits and 4 MiB of diff text, the oldest dropped), deleted when the session's record is removed (a killed pinned session keeps it; at startup, a file whose session no longer exists is deleted too) and never fed into `/api/v1/search`. No route here serves a file: the panel shows thumbnails through `file-raw` inside the workspace only; an image outside it opens in the file preview, which registers it through the attachment route.
## Git worktrees of a case
+3 -3
View File
@@ -726,13 +726,13 @@ Matching semantics: a query containing `/` matches the relative path, otherwise
**Edit history panel** (issue #643; `showEditHistory`, per-device, default OFF): which files a local claude session edited (main agent and subagents), each edit's diff, and images a Bash call produced, read from the session's transcripts by byte offset, never from git. Parser `src/edit-history-parse.ts` (pure), store `src/edit-history-store.ts`, reader `src/edit-history-local-source.ts`, sync `src/web/edit-history-sync.ts`, routes `src/web/routes/edit-history-routes.ts` (`GET /api/sessions/:id/edits[?sync=1]`, `GET …/edits/:entryId`, `POST …/edits/older`, all through `findSessionOrFail` WITH `req`), panel `edit-history-ui.js`. User guide: `docs/wiki/Working-With-Files.md` § Edit history.
⚠️ **Reads only on request, opt-in all the way down.** `server.ts` builds the `EditHistorySync` and hands it to the routes; its only own calls are `remove` (through `forgetEditHistory`, when the session's persisted record goes) and `flushAll` (shutdown). No hook, no timer, no second watcher. The panel drives every read: `?sync=1` when it opens, and while it is open on that session a debounced (600 ms) re-read after a `transcript:tool_end` of an edit-capable tool (Edit, Write, MultiEdit, NotebookEdit, Bash, Agent, Task, SendUserFile), which the existing `TranscriptWatcher` already broadcasts, or the session's `session:idle` (the end of a turn, via `_onSessionIdle`; it also catches a background subagent that writes after the tool end that started it). Not `transcript:complete`: it fires on a `result` line, which claude's interactive transcript never has. A closed panel reads nothing and keeps no badge. Guarded by the "reads only on request" wiring test in `test/edit-history-sync.test.ts`.
⚠️ **Reads only on request, opt-in all the way down.** `server.ts` builds the `EditHistorySync` and hands it to the routes; its only own calls are `remove` (through `forgetEditHistory`, when the session's persisted record goes), `sweep` (startup) and `flushAll` (shutdown). No hook, no timer, no second watcher. The panel drives every read: `?sync=1` when it opens, and while it is open on that session a debounced (600 ms) re-read after a `transcript:tool_end` of an edit-capable tool (Edit, Write, MultiEdit, NotebookEdit, Bash, Agent, Task, SendUserFile), which the existing `TranscriptWatcher` already broadcasts, or the session's `session:idle` (the end of a turn, via `_onSessionIdle`; it also catches a background subagent that writes after the tool end that started it). Not `transcript:complete`: it fires on a `result` line, which claude's interactive transcript never has. A closed panel reads nothing and keeps no badge. Guarded by the "reads only on request" wiring test in `test/edit-history-sync.test.ts`.
⚠️ **Which transcript** (`resolveClaudeTranscript`). The main transcript is the path the session's `TranscriptWatcher` follows (the hook-reported `transcript_path`), so a session is read from where its claude really writes, whatever its `CLAUDE_CONFIG_DIR`. Without a watcher (server restarted, no hook since), `findClaudeTranscript()` looks for `<claudeSessionId|id>.jsonl` under `claudeProjectsDir(session.claudeConfigDir())`: the session's own override (#255, the same dir the relaunch pin checks), else the server's. A config dir set outside Codeman (a shell profile, a wrapper) is never guessed: until the next hook such a session answers `unavailable` / `no-transcript`, and the panel says so. Subagents are `<main path without .jsonl>/subagents/agent-<id>.jsonl`. A different main file (`/clear`, resume) is read from its end with fresh cursors; the stored entries stay.
⚠️ **Bounded reads.** A sync is at most `MAX_FORWARD_PASSES` (3) reads, each at most `READ_BUDGET_BYTES` (8 MiB) shared by the main transcript and all subagents (main first); a first read starts that far before the end, and "Load older" reads one older block of every transcript, the budget split between them. A line longer than a read's budget is skipped, going forward and going back. Parsing happens in the request, line by line; nothing runs on the PTY data path. One sync per session at a time, and requests during a run fold into one more run.
⚠️ **Bounded reads.** A sync is at most `MAX_FORWARD_PASSES` (3) reads, each at most `READ_BUDGET_BYTES` (8 MiB) shared by the main transcript and all subagents (main first); a first read starts that far before the end. A file whose chunk the shared budget cut short is not applied but read in a later pass (the store would take a chunk with no line end for one line longer than the budget). "Load older" reads a full-budget block of up to `MAX_OLDER_SOURCES` (3) transcripts per click; with more, they take turns. A line longer than the budget is skipped, going forward and going back, and the line after (before) it is read whole. A subagent transcript that vanished loses its cursor, not its entries. Parsing happens in the request, line by line; nothing runs on the PTY data path. One sync per session at a time, and requests during a run fold into one more run.
⚠️ **The store holds secrets.** Every diff an agent wrote ends up in `dataPath('edit-history')/<sessionId>.json`, API keys and `.env` contents included. It is written 0600 via tmp + rename (a fresh file each time, so an existing store never keeps wider bits; a new directory is created 0700) and bounded per session: `MAX_ENTRIES` (500) and `MAX_STORE_BYTES` (4 MiB of diff text and paths), oldest dropped, also on load. Each edit's diff is capped at `MAX_HUNK_BYTES` (64 KiB); the file also holds at most `MAX_PENDING_BASH` (200) unanswered Bash calls of at most 8 KiB each and `MAX_AGENTS` (501) subagent labels of at most 1 KiB per string, and `load()` drops whatever in a damaged file breaks those bounds. Writes are serialized. The store lives as long as the session's persisted record: `forgetEditHistory` runs when `cleanupSession` with `killMux` removes the record (a pinned session's record is only demoted and keeps its history), on the persisted-only `DELETE /api/sessions/:id`, and for every record `cleanupStaleSessions` drops at startup. After `remove()` nothing writes the file again, neither a sync in flight nor a later request (the session answers `unavailable` / `removed`). The store is blocked in `isSensitivePath` (`/.codeman*/edit-history/`, same reasoning as `intents.json`) and never fed to `/api/search`.
⚠️ **The store holds secrets.** Every diff an agent wrote ends up in `dataPath('edit-history')/<sessionId>.json`, API keys and `.env` contents included. It is written 0600 via tmp + rename (a fresh file each time, so an existing store never keeps wider bits; a new directory is created 0700) and bounded per session: `MAX_ENTRIES` (500) and `MAX_STORE_BYTES` (4 MiB of diff text, paths and ids), oldest dropped, also on load. Each edit's diff is capped at `MAX_HUNK_BYTES` (64 KiB); the file also holds at most `MAX_PENDING_BASH` (200) unanswered Bash calls of at most 8,192 characters of command text each and `MAX_AGENTS` (501) subagent labels of at most 1,024 characters per string. `load()` takes the file as untrusted: one over `MAX_FILE_BYTES` (32 MiB) is not parsed, and the rest is rebuilt from its known fields under the same bounds (unknown fields and non-string diff lines dropped, the per-edit cap applied again). Writes are serialized. The store lives as long as the session's persisted record: `forgetEditHistory` runs when `cleanupSession` with `killMux` removes the record (a pinned session's record is only demoted and keeps its history), on the persisted-only `DELETE /api/sessions/:id`, for every record `cleanupStaleSessions` drops at startup and every one `POST /api/cleanup-state` drops. After that startup cleanup, `sweep` deletes the `edit-history/*.json` files whose session is neither live nor recorded (left behind by a crash between removing the record and deleting the file, or by an older build) and tmp files of other processes. After `remove()` nothing writes the file again, neither a sync in flight nor a later request (the session answers `unavailable` / `removed`). The store is blocked in `isSensitivePath` (`/.codeman*/edit-history/`, same reasoning as `intents.json`) and never fed to `/api/search`.
⚠️ **Images only through existing file surfaces.** A Bash image candidate counts only when the file's mtime lies in the call's window (start − 2 s … end + 2 s) and no entry already covers that file version; `SendUserFile` images need no window. The panel shows a thumbnail only inside the workspace, through the confined `file-raw` route. An image outside it gets no thumbnail (each would mint an attachment id per render); a click opens the shared file preview, which registers it once through `POST /api/sessions/:id/attachments` with `notify: false` (its extension allowlist and `isSensitivePath` apply). SVGs get no thumbnail. No route of this feature serves a file, least of all a transcript path.
+6 -6
View File
@@ -269,12 +269,12 @@ Click it for a draggable window listing the active session's edits, newest first
its worktree branch. **Main agent only** hides them. **Group by file** puts the edits under
their file, with the totals on the file's row.
- Images an agent produced with a shell command (a render, a plot, a screenshot) appear as rows with
a thumbnail: a `.png`, `.jpg`, `.gif`, `.webp` or `.avif` path named in the command or its output
counts when the file's modification time falls within that command's run. Images the agent sent
you count as well. Images outside the workspace show no thumbnail. Click one to open it in the
preview.
a thumbnail: a `.png`, `.jpg`, `.gif`, `.webp`, `.avif` or `.svg` path named in the command or
its output counts when the file's modification time falls within that command's run. Images the
agent sent you count as well. Images outside the workspace and SVGs show no thumbnail. Click one
to open it in the preview.
- **Load older** reads further back in the conversation, in the main agent's transcript and the
subagents' alike.
subagents' alike (up to three transcripts per click; with more, they take turns).
It works for local Claude sessions only. Remote (SSH), Docker and other CLIs' sessions say so instead.
@@ -294,7 +294,7 @@ How it reads:
`~/.codeman/edit-history/<session-id>.json`, including API keys or `.env` contents it edited. The file is
readable by your user only, capped at 500 edits and 4 MiB of diff text per session (the oldest
are dropped), never part of search, blocked from every file route, and deleted when you delete
the session. A pinned session you kill keeps its record and its history until you unpin and
the session (a file left behind, by a crash for instance, goes at the next start). A pinned session you kill keeps its record and its history until you unpin and
delete it.
- **Images go through the existing file routes.** A thumbnail inside the workspace uses the File
Viewer's route. An image outside it opens like any clicked path outside the workspace (no
+23 -11
View File
@@ -149,22 +149,20 @@ function isRecord(v: unknown): v is Record<string, unknown> {
return typeof v === 'object' && v !== null && !Array.isArray(v);
}
function toHunks(patch: unknown[]): { hunks: EditHunk[]; added: number; removed: number; truncated: boolean } {
/**
* The hunks of `patch` with their string lines, cut once the lines reach {@link MAX_HUNK_BYTES}
* (`truncated`). Also applied to hunks loaded from a store file.
*/
export function capHunks(patch: readonly unknown[]): { hunks: EditHunk[]; truncated: boolean } {
const hunks: EditHunk[] = [];
let added = 0;
let removed = 0;
let budget = MAX_HUNK_BYTES;
let truncated = false;
for (const h of patch) {
if (!isRecord(h) || !Array.isArray(h.lines)) continue;
const lines = h.lines.filter((l): l is string => typeof l === 'string');
for (const l of lines) {
if (l.startsWith('+')) added++;
else if (l.startsWith('-')) removed++;
}
if (truncated) continue;
if (truncated) break;
const kept: string[] = [];
for (const l of lines) {
for (const l of h.lines) {
if (typeof l !== 'string') continue;
const cost = Buffer.byteLength(l) + 3;
if (cost > budget) {
truncated = true;
@@ -181,7 +179,21 @@ function toHunks(patch: unknown[]): { hunks: EditHunk[]; added: number; removed:
lines: kept,
});
}
return { hunks, added, removed, truncated };
return { hunks, truncated };
}
function toHunks(patch: unknown[]): { hunks: EditHunk[]; added: number; removed: number; truncated: boolean } {
let added = 0;
let removed = 0;
for (const h of patch) {
if (!isRecord(h) || !Array.isArray(h.lines)) continue;
for (const l of h.lines) {
if (typeof l !== 'string') continue;
if (l.startsWith('+')) added++;
else if (l.startsWith('-')) removed++;
}
}
return { ...capHunks(patch), added, removed };
}
/** A new file's content as one all-added hunk, in `structuredPatch` form. */
+144 -42
View File
@@ -15,7 +15,10 @@
* A first read of a file starts `budget` bytes before its end (transcripts reach 100 MB+),
* drops the cut first line and leaves `floor` there. A line longer than the budget is
* skipped instead of blocking the cursor, reading forward (`midLine`) and backward
* (`floorMidLine`).
* (`floorMidLine`); the line after it (forward) or before it (backward) is read whole.
* That needs every chunk to be a full budget unless it reaches the file's end (forward) or
* start (older): a chunk with no line end then means one line longer than the budget. The
* caller does not hand over a chunk that a budget shared by several files cut short.
*
* A new conversation (`/clear`, resume) arrives as a new `transcriptId`: cursors and
* pending Bash calls start over, entries stay. Entries are capped at {@link MAX_ENTRIES}
@@ -25,8 +28,10 @@
*
* ⚠️ The file holds secrets: a diff carries whatever the agent wrote, keys and `.env`
* contents included. So, like `intents.json`, it is written 0600 via tmp+rename, bounded,
* deleted with the session (`remove()`), never fed to `/api/search`, and blocked from the
* file routes by `isSensitivePath` (`web/sensitive-path.ts`).
* deleted with the session (`remove()`; `sweep()` at startup removes files no session owns),
* never fed to `/api/search`, and blocked from the file routes by `isSensitivePath`
* (`web/sensitive-path.ts`). `load()` takes the file as untrusted: past
* {@link MAX_FILE_BYTES} it is not parsed, and every part is rebuilt under the same bounds.
*
* Persisted as JSON under `dataPath('edit-history', '<sessionId>.json')`, written debounced.
*
@@ -35,10 +40,11 @@
* @module edit-history-store
*/
import { mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises';
import { mkdir, readdir, readFile, rename, stat, unlink, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { dataPath } from './config/instance.js';
import {
capHunks,
MAX_HUNK_BYTES,
MAX_PENDING_BASH,
MAX_PENDING_COMMAND_CHARS,
@@ -61,6 +67,13 @@ export const MAX_AGENT_META_CHARS = 1024;
export const MAX_AGENTS = MAX_ENTRIES + 1;
/** Bytes read per file and pass (first read, forward step, "older" step). */
export const READ_BUDGET_BYTES = 8 * 1024 * 1024;
/**
* A store file larger than this is not parsed (starts empty). One written within the caps
* above stays far below it; JSON escaping at most doubles ordinary text.
*/
export const MAX_FILE_BYTES = 32 * 1024 * 1024;
/** Longest id, cwd or agent id of a loaded pending Bash call; a longer one is no real one. */
const MAX_LOADED_FIELD_CHARS = 4096;
export const MAIN_SOURCE = 'main';
const SAVE_DEBOUNCE_MS = 1000;
const FILE_VERSION = 1;
@@ -137,10 +150,10 @@ function isObj(v: unknown): v is Record<string, unknown> {
return typeof v === 'object' && v !== null && !Array.isArray(v);
}
/** What an entry costs against {@link MAX_STORE_BYTES}: its path and diff lines, plus a fixed overhead. */
/** What an entry costs against {@link MAX_STORE_BYTES}: its strings and diff lines, plus a fixed overhead. */
export function entryBytes(e: EditHistoryEntry): number {
let n = 128 + Buffer.byteLength(e.path);
for (const h of e.hunks) if (Array.isArray(h?.lines)) for (const l of h.lines) n += Buffer.byteLength(String(l)) + 3;
let n = 128 + Buffer.byteLength(e.id) + Buffer.byteLength(e.path) + Buffer.byteLength(e.agentId ?? '');
for (const h of e.hunks) for (const l of h.lines) n += Buffer.byteLength(l) + 3;
return n;
}
@@ -185,26 +198,47 @@ function cursorFrom(raw: unknown): SourceCursor | null {
return cursor;
}
const loadedField = (v: unknown): v is string => typeof v === 'string' && v.length <= MAX_LOADED_FIELD_CHARS;
function pendingBashFrom(raw: unknown): PendingBash | null {
if (!isObj(raw)) return null;
const { id, ts, command, cwd, agentId } = raw;
if (typeof id !== 'string' || typeof ts !== 'number' || typeof command !== 'string' || typeof cwd !== 'string') {
return null;
}
const pending: PendingBash = { id, ts, command: command.slice(0, MAX_PENDING_COMMAND_CHARS), cwd };
if (typeof agentId === 'string') pending.agentId = agentId;
if (!loadedField(id) || !Number.isFinite(ts) || typeof command !== 'string' || !loadedField(cwd)) return null;
if (agentId !== undefined && !loadedField(agentId)) return null;
const pending: PendingBash = { id, ts: ts as number, command: command.slice(0, MAX_PENDING_COMMAND_CHARS), cwd };
if (agentId) pending.agentId = agentId;
return pending;
}
function isEntry(raw: unknown): raw is EditHistoryEntry {
return (
isObj(raw) &&
typeof raw.id === 'string' &&
typeof raw.path === 'string' &&
typeof raw.ts === 'number' &&
Array.isArray(raw.hunks) &&
raw.hunks.every((h) => isObj(h) && Array.isArray(h.lines))
);
const ENTRY_KINDS = new Set(['code', 'image']);
const ENTRY_TOOLS = new Set(['Edit', 'Write', 'edit', 'bash', 'sent']);
const countFrom = (v: unknown): number => (Number.isSafeInteger(v) && (v as number) >= 0 ? (v as number) : 0);
/**
* A loaded entry rebuilt from its known fields: unknown fields and non-string diff lines are
* dropped and the per-edit {@link MAX_HUNK_BYTES} is applied again; null when it is unusable.
*/
function entryFrom(raw: unknown): EditHistoryEntry | null {
if (!isObj(raw) || typeof raw.id !== 'string' || !raw.id || typeof raw.path !== 'string' || !raw.path) return null;
if (!Number.isFinite(raw.ts) || !ENTRY_KINDS.has(raw.kind as string) || !ENTRY_TOOLS.has(raw.tool as string)) {
return null;
}
if (!Array.isArray(raw.hunks)) return null;
const { hunks, truncated } = capHunks(raw.hunks);
const entry: EditHistoryEntry = {
id: raw.id,
ts: raw.ts as number,
path: raw.path,
kind: raw.kind as EditHistoryEntry['kind'],
tool: raw.tool as EditHistoryTool,
created: raw.created === true,
added: countFrom(raw.added),
removed: countFrom(raw.removed),
hunks,
truncated: truncated || raw.truncated === true,
};
if (typeof raw.agentId === 'string' && raw.agentId) entry.agentId = raw.agentId;
return entry;
}
export class EditHistoryStore {
@@ -229,11 +263,15 @@ export class EditHistoryStore {
this.debounceMs = opts.debounceMs ?? SAVE_DEBOUNCE_MS;
}
/** A store with the persisted state of `sessionId`; empty when the file is missing or unreadable. */
/**
* A store with the persisted state of `sessionId`; empty when the file is missing,
* unreadable or larger than {@link MAX_FILE_BYTES}.
*/
static async load(sessionId: string, opts: EditHistoryStoreOptions = {}): Promise<EditHistoryStore> {
const store = new EditHistoryStore(sessionId, opts);
let data: unknown;
try {
if ((await stat(store.file)).size > MAX_FILE_BYTES) return store;
data = JSON.parse(await readFile(store.file, 'utf-8'));
} catch {
return store;
@@ -252,14 +290,46 @@ export class EditHistoryStore {
.filter((p): p is PendingBash => p !== null)
.slice(-MAX_PENDING_BASH);
}
if (Array.isArray(f.entries)) store.addRecords(f.entries.filter(isEntry));
if (Array.isArray(f.entries)) {
store.addRecords(f.entries.map(entryFrom).filter((e): e is EditHistoryEntry => e !== null));
}
for (const [k, raw] of Object.entries(isObj(f.agents) ? f.agents : {})) {
const meta = agentMetaFrom(raw);
const meta = k.length <= MAX_AGENT_META_CHARS ? agentMetaFrom(raw) : null;
if (meta) store.putAgent(k, meta);
}
return store;
}
/**
* Delete the stored history of every session `keep` rejects, and tmp files other processes
* left behind (startup: a crash between removing a session's record and deleting its file,
* or mid-write; an older build).
* Returns how many sessions' files were deleted.
*/
static async sweep(
keep: (sessionId: string) => boolean,
opts: Pick<EditHistoryStoreOptions, 'dir'> = {}
): Promise<number> {
const dir = opts.dir ?? dataPath('edit-history');
let names: string[];
try {
names = await readdir(dir);
} catch {
return 0;
}
let removed = 0;
for (const name of names) {
const tmp = /^(.+)\.json\.(\d+)\.tmp$/.exec(name);
const stale = tmp
? Number(tmp[2]) !== process.pid
: name.endsWith('.json') && !keep(name.slice(0, -'.json'.length));
if (!stale) continue;
await unlink(join(dir, name)).catch(() => undefined);
if (!tmp) removed++;
}
return removed;
}
getTranscriptId(): string | null {
return this.transcriptId;
}
@@ -338,7 +408,19 @@ export class EditHistoryStore {
return [...this.cursors.entries()].filter(([, c]) => c.floor > 0).map(([s]) => s);
}
/** Apply a forward chunk read at `nextRange`. Returns new entries and Bash image candidates. */
/**
* A source whose file is gone (a deleted subagent transcript): its cursor goes, so it no
* longer offers older lines. Its entries stay.
*/
dropSource(source: string): void {
if (this.removed || !this.cursors.delete(source)) return;
this.scheduleSave();
}
/**
* Apply a forward chunk read at `nextRange`: a full budget unless it reaches the file's end.
* Returns new entries and Bash image candidates.
*/
apply(input: ApplyInput): ApplyResult {
if (this.removed) return { added: 0, candidates: [] };
const { transcriptId, source, startOffset, chunk, size } = input;
@@ -369,11 +451,14 @@ export class EditHistoryStore {
}
const lastNl = chunk.lastIndexOf(NL);
if (lastNl < from) {
if (!atEof && chunk.length > 0) {
if (from > 0) {
// The skipped (or cut first) line ended here; the next one is read from its start.
this.setCursor(source, cursor, startOffset + from, startOffset + from, false);
} else if (!atEof && chunk.length > 0) {
// One line longer than the budget: skip it instead of re-reading it forever.
this.setCursor(source, cursor, startOffset + chunk.length, startOffset + from, true);
this.setCursor(source, cursor, startOffset + chunk.length, startOffset, true);
} else if (!cursor) {
this.setCursor(source, undefined, startOffset + from, startOffset + from, false);
this.setCursor(source, undefined, startOffset, startOffset, false);
}
return { added: 0, candidates: [] };
}
@@ -387,8 +472,9 @@ export class EditHistoryStore {
}
/**
* Apply a chunk read at `olderRange`: it ends at the floor, which is a line start, or with
* `floorMidLine` lies inside a line longer than the budget (its part in this chunk is dropped).
* Apply a chunk read at `olderRange`: a full budget unless it reaches the file's start. It
* ends at the floor, which is a line start, or with `floorMidLine` lies inside a line longer
* than the budget (its part in this chunk is dropped).
*/
applyOlder(input: ApplyInput): ApplyResult {
const { transcriptId, source, startOffset, chunk } = input;
@@ -396,23 +482,39 @@ export class EditHistoryStore {
if (transcriptId !== this.transcriptId || !cursor || startOffset + chunk.length !== cursor.floor) {
return { added: 0, candidates: [] };
}
const end = cursor.floorMidLine ? chunk.lastIndexOf(NL) + 1 : chunk.length;
const none = { added: 0, candidates: [] };
let end = chunk.length;
if (cursor.floorMidLine) {
const lastNl = chunk.lastIndexOf(NL);
if (lastNl === -1 && startOffset > 0) {
// Still inside the long line.
cursor.floor = startOffset;
this.scheduleSave();
return none;
}
// The long line starts at `end`: drop its part.
end = lastNl + 1;
delete cursor.floorMidLine;
}
let from = 0;
let insideLine = false;
if (startOffset > 0) {
// The newline at `end - 1` closes a line that may have started before this chunk.
const nl = chunk.subarray(0, end).indexOf(NL);
insideLine = nl === -1 || nl === end - 1;
from = insideLine ? end : nl + 1;
}
if (insideLine) {
// No line starts in this chunk: it lies inside one line longer than the budget. Skip it.
cursor.floor = startOffset;
cursor.floorMidLine = true;
} else {
cursor.floor = startOffset + from;
delete cursor.floorMidLine;
if (nl === -1 || nl === end - 1) {
if (end < chunk.length) {
// It ends where the long line starts: the next block, ending there, holds it whole.
cursor.floor = startOffset + end;
} else {
// No line starts in this chunk: it lies inside one line longer than the budget. Skip it.
cursor.floor = startOffset;
cursor.floorMidLine = true;
}
this.scheduleSave();
return none;
}
from = nl + 1;
}
cursor.floor = startOffset + from;
// Calls whose results lie in already-read newer lines cannot be paired here.
const parsed = parseClaudeEditLines(chunk.subarray(from, end).toString('utf8'));
const added = this.addRecords(parsed.records);
+59 -12
View File
@@ -8,12 +8,19 @@
* looks at. One sync per session runs at a time; requests during a run are folded into one
* more forward read and/or one more older read afterwards.
*
* HOW MUCH. A forward read shares one budget over all files, main first. A file whose chunk
* the shared budget cut short is not applied (the store would take a chunk with no line end
* for a line longer than the budget) but read in the next pass; up to
* {@link MAX_FORWARD_PASSES} passes per sync, the rest with the next one.
*
* GONE. `remove()` (the session's record is deleted) marks the session gone before it deletes
* the file: a run in flight stops after its current read, its store no longer writes, and a
* later request reads nothing (`unavailable`, reason `removed`) instead of loading a new store.
* `sweep()` deletes, at startup, the stored history of sessions that no longer exist. A
* subagent transcript that vanished loses its cursor (`dropSource`), not its entries.
*
* OLDER. "Load older" splits the budget evenly over the sources with older lines and reads
* only those blocks, so every source moves back with each click.
* OLDER. "Load older" reads a full budget block of up to {@link MAX_OLDER_SOURCES} sources
* with older lines per click; with more sources they take turns.
*
* WHICH SESSIONS. Local claude sessions. Remote, Docker and non-claude sessions answer
* `unsupported` and are never read.
@@ -34,8 +41,10 @@ import { getCli } from '../config/cli-registry/registry.js';
/** Slack around a Bash call's window for the image mtime check. */
export const IMAGE_MTIME_SLACK_MS = 2000;
/** Forward passes per sync when a file grew by more than one budget. */
/** Forward passes per sync when files grew by more than one budget. */
export const MAX_FORWARD_PASSES = 3;
/** Sources read back per "Load older" click, a full budget each. */
export const MAX_OLDER_SOURCES = 3;
/** `reason` when the session has no transcript yet (no prompt yet, or not found). */
export const NO_TRANSCRIPT = 'no-transcript';
/** `reason` once the session's history was deleted. */
@@ -70,6 +79,8 @@ export interface EditHistorySyncDeps {
): Promise<EditSources | null>;
probe(paths: readonly string[]): Promise<Array<PathProbe | null>>;
loadStore(sessionId: string): Promise<EditHistoryStore>;
/** Delete the stored history of every session `keep` rejects; returns how many. */
sweepStores(keep: (sessionId: string) => boolean): Promise<number>;
budget?: number;
}
@@ -135,6 +146,8 @@ interface SessionSlot {
again?: SyncWant;
/** The history was deleted: nothing reads for this session any more. */
gone?: boolean;
/** The last source the previous "Load older" read; the next one starts after it. */
olderAfter?: string;
}
export class EditHistorySync {
@@ -212,6 +225,14 @@ export class EditHistorySync {
await store.remove();
}
/**
* Delete the stored history of sessions that no longer exist (startup: left behind by a
* crash during deletion, or by an older build). A session in use here is kept.
*/
sweep(exists: (sessionId: string) => boolean): Promise<number> {
return this.deps.sweepStores((id) => this.slots.has(id) || exists(id));
}
private slot(sessionId: string): SessionSlot {
let slot = this.slots.get(sessionId);
if (!slot) {
@@ -240,7 +261,7 @@ export class EditHistorySync {
if (!path) {
found = false;
} else if (older) {
found = await this.readOlder(path, store);
found = await this.readOlder(path, store, slot);
} else {
found = true;
for (let pass = 0; pass < MAX_FORWARD_PASSES && found && !slot.gone; pass++) {
@@ -266,20 +287,29 @@ export class EditHistorySync {
}
/**
* One forward read of all sources. `more` = some file has unread bytes after it.
* A transcript other than the stored one (`/clear`, resume) is read like a first read.
* One forward read of all sources. `more` = some file has unread bytes after it, or was cut
* short by the shared budget and left for the next pass. A transcript other than the stored
* one (`/clear`, resume) is read like a first read.
*/
private async readForward(path: string, store: EditHistoryStore): Promise<{ found: boolean; more: boolean }> {
const known = store.getTranscriptId();
const requests = known !== null && path.endsWith(`/${known}.jsonl`) ? this.forwardRequests(store) : [];
const res = await this.deps.readSources(path, requests, this.budget);
if (!res) return { found: false, more: false };
this.dropMissing(store, requests, res);
for (const [agentId, text] of Object.entries(res.metas)) {
if (!store.hasAgentMeta(agentId)) store.setAgentMeta(agentId, text);
}
const requested = new Set(requests.map((r) => r.relPath));
let more = false;
const candidates: ImageCandidate[] = [];
for (const f of res.files) {
// A requested file reads on from its cursor, a new one its tail.
const available = requested.has(f.relPath) ? f.size - f.startOffset : f.size;
if (f.chunk.length < Math.min(this.budget, available)) {
more = true;
continue;
}
const r = store.apply({
transcriptId: res.transcriptId,
source: f.relPath,
@@ -294,20 +324,31 @@ export class EditHistorySync {
return { found: true, more };
}
private async readOlder(path: string, store: EditHistoryStore): Promise<boolean> {
private async readOlder(path: string, store: EditHistoryStore, slot: SessionSlot): Promise<boolean> {
if (!store.backfillTruncated) return true;
const sources = store.sourcesWithOlder();
const sources = store.sourcesWithOlder().sort();
if (sources.length === 0) return true;
const share = Math.max(1, Math.floor(this.budget / sources.length));
// Take turns: start after the last source the previous click read.
const after = slot.olderAfter;
const first =
after === undefined
? 0
: Math.max(
0,
sources.findIndex((s) => s > after)
);
const picked = [...sources.slice(first), ...sources.slice(0, first)].slice(0, MAX_OLDER_SOURCES);
const requests: EditSourceRequest[] = [];
for (const relPath of sources) {
const range = store.olderRange(relPath, share);
for (const relPath of picked) {
const range = store.olderRange(relPath, this.budget);
if (range) requests.push({ relPath, offset: range.offset, len: range.len });
}
if (requests.length === 0) return true;
const res = await this.deps.readSources(path, requests, this.budget, { onlyRequested: true });
slot.olderAfter = picked[picked.length - 1];
const res = await this.deps.readSources(path, requests, this.budget * requests.length, { onlyRequested: true });
if (!res) return false;
if (res.transcriptId !== store.getTranscriptId()) return true;
this.dropMissing(store, requests, res);
const wanted = new Set(requests.map((r) => r.relPath));
const candidates: ImageCandidate[] = [];
for (const f of res.files) {
@@ -325,6 +366,12 @@ export class EditHistorySync {
return true;
}
/** Requested files the reader no longer lists are gone: their cursors go too. */
private dropMissing(store: EditHistoryStore, requests: readonly EditSourceRequest[], res: EditSources): void {
const listed = new Set(res.files.map((f) => f.relPath));
for (const r of requests) if (!listed.has(r.relPath)) store.dropSource(r.relPath);
}
private async addImages(store: EditHistoryStore, candidates: readonly ImageCandidate[]): Promise<void> {
const paths = [...new Set(candidates.flatMap((c) => c.paths))];
if (paths.length === 0) return;
+3 -2
View File
@@ -3518,8 +3518,9 @@ class CodemanApp {
this.sendPendingCtrlL(data.id);
if (data.id === this.activeSessionId) this._updateLocalEchoState();
}
// An open Edit history panel re-reads at the end of its session's turn (edit-history-ui.js).
this.onEditHistorySessionIdle?.(data.id);
// An open Edit history panel re-reads at the end of its session's turn (edit-history-ui.js),
// not after Codeman's own keep-alive turn, which edits nothing.
if (!data.keepAlive) this.onEditHistorySessionIdle?.(data.id);
// Start stuck detection timer (only if no respawn running). Not at the end of Codeman's
// own keep-alive turn (`keepAlive`): the session has been waiting since the last real
// turn, whose timer is still running or has already fired.
+1
View File
@@ -801,6 +801,7 @@ export function registerSystemRoutes(
const lifecycleLog = getLifecycleLog();
for (const s of result.cleaned) {
lifecycleLog.log({ event: 'stale_cleaned', sessionId: s.id, name: s.name });
ctx.forgetEditHistory(s.id);
}
return { cleanedSessions: result.count };
});
+8 -1
View File
@@ -348,6 +348,7 @@ export class WebServer extends EventEmitter {
readSources: readClaudeEditSources,
probe: probeLocalPaths,
loadStore: (sessionId) => EditHistoryStore.load(sessionId),
sweepStores: (keep) => EditHistoryStore.sweep(keep),
});
private respawnControllers: Map<string, RespawnController> = new Map();
private respawnTimers: Map<string, { timer: NodeJS.Timeout; endAt: number; startedAt: number }> = new Map();
@@ -2506,12 +2507,18 @@ export class WebServer extends EventEmitter {
lifecycleLog.log({ event: 'stale_cleaned', sessionId: s.id, name: s.name });
this.forgetEditHistory(s.id);
}
// History files no session owns (a crash between removing a record and deleting its file).
const exists = (id: string) => this.sessions.has(id) || !!this.store.getSession(id);
await this.editHistorySync.sweep(exists).catch((err) => {
console.warn('[EditHistory] Sweeping orphaned histories failed:', getErrorMessage(err));
});
return result.count;
}
/**
* Delete a session's edit history (diffs can hold secrets) once its record is gone: deleted
* live (`cleanupSession`), deleted as a record only (DELETE route), or cleaned as stale.
* live (`cleanupSession`), deleted as a record only (DELETE route), or cleaned as stale
* (startup, `POST /api/cleanup-state`).
*/
private forgetEditHistory(sessionId: string): void {
void this.editHistorySync.remove(sessionId).catch((err) => {
+142
View File
@@ -13,9 +13,11 @@ import {
MAX_AGENT_META_CHARS,
MAX_AGENTS,
MAX_ENTRIES,
MAX_FILE_BYTES,
MAX_STORE_BYTES,
parseAgentMeta,
} from '../src/edit-history-store.js';
import { MAX_HUNK_BYTES } from '../src/edit-history-parse.js';
let dir: string;
beforeEach(() => {
@@ -161,6 +163,23 @@ describe('EditHistoryStore: forward reads', () => {
expect(store.getCursor(MAIN_SOURCE)?.offset).toBe(file.length);
});
it('reads the line after a skipped one whole, wherever it falls against the chunk grid', async () => {
const budget = 1000;
const first = jsonl([editLine('e0', 0)]);
const lost: number[] = [];
for (let pad = 0; pad < budget; pad++) {
const store = new EditHistoryStore('s1', { dir, debounceMs: 60_000 });
syncForward(store, first);
const huge = JSON.stringify({ type: 'user', message: { content: 'h'.repeat(2400 + pad) } });
const next = editLine('next', 2, '/work/next.ts', { pad: 'p'.repeat(300) }); // < budget
const file = Buffer.concat([first, jsonl([huge, next, editLine('e3', 3)])]);
for (let i = 0; i < 50 && syncForward(store, file, budget); i++);
if (!store.get('next') || !store.get('e3')) lost.push(pad);
await store.remove();
}
expect(lost).toEqual([]);
});
it('keeps a multibyte character split by the chunk boundary intact', () => {
const store = new EditHistoryStore('s1', { dir });
const path = '/work/Größe – Ümlaut 😀.ts';
@@ -273,6 +292,37 @@ describe('EditHistoryStore: older blocks', () => {
expect(store.backfillTruncated).toBe(false);
});
it('going back, reads the line before a skipped one whole, wherever it falls against the block grid', async () => {
const budget = 1000;
const huge = JSON.stringify({ type: 'user', message: { content: 'h'.repeat(2500) } });
const lost: number[] = [];
for (let pad = 0; pad < budget; pad++) {
const store = new EditHistoryStore('s1', { dir, debounceMs: 60_000 });
const lines = [editLine('p0', 0), editLine('p1', 1), editLine('before', 2), huge];
for (let i = 0; i < 6; i++)
lines.push(editLine(`t${i}`, 10 + i, `/work/t${i}.ts`, i === 0 ? { pad: 'p'.repeat(pad) } : {}));
const file = jsonl(lines);
syncForward(store, file, budget);
for (let i = 0; i < 50 && syncOlder(store, file, budget); i++);
if (!store.get('before') || !store.get('p0')) lost.push(pad);
await store.remove();
}
expect(lost).toEqual([]);
});
it('a source whose file is gone (dropSource) offers no older lines and keeps its entries', () => {
const store = new EditHistoryStore('s1', { dir });
const sub = 'subagents/agent-a1.jsonl';
const file = jsonl(Array.from({ length: 10 }, (_, i) => editLine(`s${i}`, i, `/work/s${i}.ts`, { agentId: 'a1' })));
syncForward(store, file, 400, sub);
expect(store.sourcesWithOlder()).toEqual([sub]);
const kept = ids(store);
store.dropSource(sub);
expect(store.getCursor(sub)).toBeUndefined();
expect(store.backfillTruncated).toBe(false);
expect(ids(store)).toEqual(kept);
});
it('refuses an older chunk that does not end at the floor', () => {
const store = new EditHistoryStore('s1', { dir });
const file = jsonl(Array.from({ length: 10 }, (_, i) => editLine(`e${i}`, i)));
@@ -460,6 +510,96 @@ describe('EditHistoryStore: persistence', () => {
expect(syncForward(loaded, jsonl([bashResult('b1', 3)]))?.candidates[0]?.paths).toEqual(['/work/a.png']);
});
it('rebuilds loaded entries under the same bounds: known fields only, string diff lines, the per-edit cap', async () => {
const good = {
ts: 1,
path: '/a.ts',
kind: 'code',
tool: 'Edit',
created: false,
added: 1,
removed: 0,
truncated: false,
};
writeFileSync(
join(dir, 's1.json'),
JSON.stringify({
version: 1,
sessionId: 's1',
transcriptId: 'conv-1',
cursors: {},
pendingBash: [
{ id: 'b1', ts: 1, command: 'ls', cwd: '/work' },
{ id: 'b2', ts: 1, command: 'ls', cwd: '/' + 'c'.repeat(5000) },
{ id: 'p'.repeat(5000), ts: 1, command: 'ls', cwd: '/work' },
],
agents: { ['a'.repeat(MAX_AGENT_META_CHARS + 1)]: { description: 'd', agentType: 't' } },
entries: [
{ ...good, id: 'extra', hunks: [], blob: 'x'.repeat(1000) },
{ ...good, id: 'objlines', hunks: [{ oldStart: 1, lines: [{ k: 'y' }, '+kept'] }] },
{ ...good, id: 'bighunk', hunks: [{ lines: ['+' + 'q'.repeat(MAX_HUNK_BYTES)] }] },
{ ...good, id: 'badtool', tool: 'Rm', hunks: [] },
],
})
);
const loaded = await EditHistoryStore.load('s1', { dir, debounceMs: 60_000 });
expect(
loaded
.list()
.map((e) => e.id)
.sort()
).toEqual(['bighunk', 'extra', 'objlines']);
expect(loaded.get('extra')).not.toHaveProperty('blob');
expect(loaded.get('objlines')!.hunks[0].lines).toEqual(['+kept']);
expect(loaded.get('bighunk')).toMatchObject({ truncated: true, hunks: [{ lines: [] }] });
expect(loaded.getAgents()).toEqual({});
await loaded.flush();
expect(
JSON.parse(readFileSync(join(dir, 's1.json'), 'utf-8')).pendingBash.map((p: { id: string }) => p.id)
).toEqual(['b1']);
});
it("counts an entry's id against the byte cap", () => {
const e = {
id: 'i'.repeat(1000),
ts: 1,
path: '/a.ts',
kind: 'code',
tool: 'Edit',
created: false,
added: 0,
removed: 0,
hunks: [],
truncated: false,
} as const;
expect(entryBytes(e) - entryBytes({ ...e, id: 'i' })).toBe(999);
});
it(`does not parse a store file larger than ${MAX_FILE_BYTES} bytes`, async () => {
const store = new EditHistoryStore('s1', { dir, debounceMs: 60_000 });
syncForward(store, jsonl([editLine('e1', 1)]));
await store.flush();
const text = readFileSync(join(dir, 's1.json'), 'utf-8');
expect((await EditHistoryStore.load('s1', { dir })).get('e1')).toBeDefined();
writeFileSync(join(dir, 's1.json'), text + ' '.repeat(MAX_FILE_BYTES)); // still valid JSON
expect((await EditHistoryStore.load('s1', { dir })).list()).toEqual([]);
});
it('sweep() deletes the histories of sessions that are gone and stale tmp files, nothing else', async () => {
for (const name of [
'live.json',
'gone.json',
`gone.json.${process.pid + 1}.tmp`,
`live.json.${process.pid}.tmp`,
'notes.txt',
]) {
writeFileSync(join(dir, name), '{}');
}
expect(await EditHistoryStore.sweep((id) => id === 'live', { dir })).toBe(1);
expect(readdirSync(dir).sort()).toEqual([`live.json.${process.pid}.tmp`, 'live.json', 'notes.txt'].sort());
expect(await EditHistoryStore.sweep(() => false, { dir: join(dir, 'missing') })).toBe(0);
});
it('ignores a file of an unknown version', async () => {
writeFileSync(join(dir, 'v9.json'), JSON.stringify({ version: 9, entries: [{ id: 'x' }] }));
expect((await EditHistoryStore.load('v9', { dir })).list()).toEqual([]);
@@ -499,6 +639,8 @@ describe('EditHistoryStore: persistence', () => {
const store = new EditHistoryStore('s1', { dir, debounceMs: 60_000 });
syncForward(store, jsonl([editLine('e1', 1)]));
const writing = store.flush();
// Let the write get past its first check, so remove() meets it under way.
await new Promise((r) => setImmediate(r));
await Promise.all([writing, store.remove()]);
expect(readdirSync(dir)).toEqual([]);
});
+118 -2
View File
@@ -16,6 +16,7 @@ import type {
import {
confirmImages,
EditHistorySync,
MAX_OLDER_SOURCES,
NO_TRANSCRIPT,
REMOVED,
type EditHistorySession,
@@ -24,10 +25,13 @@ import {
} from '../src/web/edit-history-sync.js';
let dir: string;
/** Stores the tests opened; removed before the temp dir, so no late save recreates it. */
const opened: EditHistoryStore[] = [];
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'edit-sync-'));
});
afterEach(() => {
afterEach(async () => {
await Promise.all(opened.splice(0).map((s) => s.remove()));
rmSync(dir, { recursive: true, force: true });
});
@@ -130,8 +134,15 @@ function setup(over: Partial<EditHistorySyncDeps> = {}) {
readSources: (path, req, budget, opts) => host.read(path, req, budget, opts),
probe: async (paths) => paths.map((p) => probes.get(p) ?? null),
loadStore: (id) => EditHistoryStore.load(id, { dir, debounceMs: 0 }),
sweepStores: async () => 0,
...over,
};
const load = deps.loadStore;
deps.loadStore = async (id) => {
const store = await load(id);
opened.push(store);
return store;
};
return { host, probes, lookups, deps, sync: new EditHistorySync(deps) };
}
@@ -382,6 +393,111 @@ describe('EditHistorySync', () => {
for (let i = 0; i < 20 && store.backfillTruncated; i++) await sync.sync(local, { older: true });
expect(store.list()).toHaveLength(20);
});
it('a file the shared budget cut short is read in the next pass, not skipped as one long line', async () => {
const budget = 4000;
const { host, sync } = setup({ budget });
const sub = 'subagents/agent-a1.jsonl';
host.append('main', editLine('m0', T0, '/w/m0.ts'));
host.append(sub, editLine('s0', T0 + 1, '/w/s0.ts', { agentId: 'a1' }));
await sync.sync(local);
// main grows by almost a whole budget, leaving the subagent 50 bytes of this pass
const filler = JSON.stringify({ type: 'user', message: { content: '' } });
host.append(
'main',
JSON.stringify({ type: 'user', message: { content: 'z'.repeat(budget - 51 - filler.length) } }) + '\n'
);
host.append(sub, editLine('s1', T0 + 2, '/w/s1.ts', { agentId: 'a1', pad: 'p'.repeat(100) }));
host.append(sub, editLine('s2', T0 + 3, '/w/s2.ts', { agentId: 'a1' }));
await sync.sync(local);
const store = await sync.getStore('s1');
expect(
store
.list()
.map((e) => e.id)
.sort()
).toEqual(['m0', 's0', 's1', 's2']);
expect(store.getCursor(sub)?.midLine).toBeUndefined();
});
it('a first read reaches the subagents in a further pass when main fills the budget', async () => {
const { host, sync } = setup({ budget: 3000 });
for (let i = 0; i < 20; i++) host.append('main', editLine(`m${i}`, T0 + i, `/w/m${i}.ts`));
host.append('subagents/agent-a1.jsonl', editLine('s0', T0 + 100, '/w/s0.ts', { agentId: 'a1' }));
await sync.sync(local);
const store = await sync.getStore('s1');
expect(store.get('s0')?.agentId).toBe('a1');
expect(store.getCursor('subagents/agent-a1.jsonl')).toBeDefined();
});
it('load older reads a full budget per source: a line longer than the budget split over the sources is kept', async () => {
const budget = 2000;
const { host, sync } = setup({ budget });
host.append('main', editLine('m0', T0, '/w/m0.ts'));
host.append('main', editLine('big', T0 + 1, '/w/big.ts', { pad: 'p'.repeat(800) })); // > budget / 3
for (let i = 2; i < 12; i++) host.append('main', editLine(`m${i}`, T0 + i, `/w/m${i}.ts`));
for (const a of ['a1', 'a2']) {
for (let i = 0; i < 12; i++) {
host.append(
`subagents/agent-${a}.jsonl`,
editLine(`${a}-${i}`, T0 + 100 + i, `/w/${a}${i}.ts`, { agentId: a })
);
}
}
for (let i = 0; i < 3; i++) await sync.sync(local);
const store = await sync.getStore('s1');
expect(store.sourcesWithOlder()).toHaveLength(3);
for (let i = 0; i < 20 && store.backfillTruncated; i++) await sync.sync(local, { older: true });
expect(store.get('big')).toBeDefined();
expect(store.list()).toHaveLength(36);
});
it(`load older reads at most ${MAX_OLDER_SOURCES} sources per click, taking turns`, async () => {
const { host, sync } = setup({ budget: 600 });
const subs = ['a1', 'a2', 'a3', 'a4'].map((a) => `subagents/agent-${a}.jsonl`);
for (const relPath of ['main', ...subs]) {
for (let i = 0; i < 8; i++) host.append(relPath, editLine(`${relPath}-${i}`, T0 + i, `/w/${i}.ts`));
}
for (let i = 0; i < 3; i++) await sync.sync(local);
const store = await sync.getStore('s1');
expect(store.sourcesWithOlder()).toHaveLength(5);
const olderCall = () => host.calls.at(-1)!.map((r) => r.relPath);
await sync.sync(local, { older: true });
expect(olderCall()).toEqual(['main', subs[0], subs[1]]);
await sync.sync(local, { older: true });
expect(olderCall()).toEqual([subs[2], subs[3], 'main']);
for (let i = 0; i < 30 && store.backfillTruncated; i++) await sync.sync(local, { older: true });
expect(store.list()).toHaveLength(40);
});
it('a subagent transcript that vanished loses its cursor (no endless "Load older"), not its entries', async () => {
const { host, sync } = setup({ budget: 600 });
const sub = 'subagents/agent-a1.jsonl';
host.append('main', editLine('m0', T0, '/w/m0.ts'));
for (let i = 0; i < 8; i++) host.append(sub, editLine(`s${i}`, T0 + 10 + i, `/w/s${i}.ts`, { agentId: 'a1' }));
for (let i = 0; i < 2; i++) await sync.sync(local);
const store = await sync.getStore('s1');
expect(store.backfillTruncated).toBe(true);
const kept = store.list().length;
host.files.delete(sub);
await sync.sync(local, { older: true });
expect(store.getCursor(sub)).toBeUndefined();
expect(store.backfillTruncated).toBe(false);
expect(store.list()).toHaveLength(kept);
});
it('sweep keeps the sessions in use here and those that exist', async () => {
const asked: Record<string, boolean> = {};
const { sync } = setup({
sweepStores: async (keep) => {
for (const id of ['s1', 'kept', 'gone']) asked[id] = keep(id);
return 1;
},
});
await sync.getStore('s1');
expect(await sync.sweep((id) => id === 'kept')).toBe(1);
expect(asked).toEqual({ s1: true, kept: true, gone: false });
});
});
describe('confirmImages', () => {
@@ -442,7 +558,7 @@ describe('reads only on request (wiring guard)', () => {
it('the server never starts a sync itself: it only hands the reader to the routes', () => {
const server = read('src/web/server.ts');
const uses = [...server.matchAll(/this\.editHistorySync\b(\.\w+)?/g)].map((m) => m[1] ?? '');
expect(uses.sort()).toEqual(['', '.flushAll', '.remove']);
expect(uses.sort()).toEqual(['', '.flushAll', '.remove', '.sweep']);
expect(server).toMatch(/registerEditHistoryRoutes\(this\.app, ctx, this\.editHistorySync\)/);
});
+3 -1
View File
@@ -304,7 +304,7 @@ describe('Edit history panel in a real browser', () => {
expect(await page.locator('#editHistoryBody button.edit-history-older').count()).toBe(0);
});
it('while open it follows the session: an edit-capable tool end re-reads (debounced), other tools and sessions do not', async () => {
it('while open it follows the session: an edit-capable tool end or a turn end re-reads (debounced), other tools, sessions and keep-alive turns do not', async () => {
const before = syncReads(sessionA);
const s = state.get(sessionA)!;
state.set(sessionA, {
@@ -325,7 +325,9 @@ describe('Edit history panel in a real browser', () => {
expect(syncReads(sessionA)).toBe(before + 1);
// The end of a turn re-reads too (a parallel tool call reports one tool end only, a background
// subagent writes later): `session:idle`, since claude's transcript has no `result` line.
// Not another session's, and not the end of Codeman's own keep-alive turn (it edits nothing).
(server as any).broadcast('session:idle', { id: 'someone-else' });
(server as any).broadcast('session:idle', { id: sessionA, keepAlive: true });
await page.waitForTimeout(1200);
expect(syncReads(sessionA)).toBe(before + 1);
(server as any).broadcast('session:idle', { id: sessionA });
+13 -5
View File
@@ -63,7 +63,12 @@ async function setup(
transcriptPath: async () => '/p/conv1.jsonl',
readSources: read,
probe: async (p) => p.map(() => null),
loadStore: (id) => EditHistoryStore.load(id, { dir, debounceMs: 0 }),
loadStore: async (id) => {
const store = await EditHistoryStore.load(id, { dir, debounceMs: 0 });
opened.push(store);
return store;
},
sweepStores: async () => 0,
...deps,
});
const h = await createRouteTestHarness((app, ctx) => registerEditHistoryRoutes(app, ctx, sync), {
@@ -76,11 +81,14 @@ async function setup(
return { ...h, read, sync };
}
/** Stores the tests opened; removed before the temp dir, so no late save recreates it. */
const opened: EditHistoryStore[] = [];
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'edit-routes-'));
});
afterEach(() => {
afterEach(async () => {
delete process.env.CODEMAN_MULTIUSER;
await Promise.all(opened.splice(0).map((s) => s.remove()));
rmSync(dir, { recursive: true, force: true });
});
@@ -215,9 +223,9 @@ describe('POST /api/sessions/:id/edits/older', () => {
const lines = [0, 1, 2, 3].map((i) => editLine(`t${i}`, T0 + i, `/w/${i}.ts`)).join('');
const full = Buffer.from(lines);
const read = vi.fn(
async (_path: string, requests: readonly { relPath: string; offset: number; len?: number }[]) => {
async (_path: string, requests: readonly { relPath: string; offset: number; len?: number }[], budget: number) => {
const req = requests.find((r) => r.relPath === 'main');
const start = req ? req.offset : full.length - 400;
const start = req ? req.offset : full.length - budget;
const len = req?.len ?? full.length - start;
return {
transcriptId: 'conv1',
@@ -226,7 +234,7 @@ describe('POST /api/sessions/:id/edits/older', () => {
};
}
);
const { app } = await setup({ readSources: read });
const { app } = await setup({ readSources: read, budget: 400 });
const first = (await app.inject({ method: 'GET', url: `${URL}?sync=1` })).json().data;
expect(first.backfillTruncated).toBe(true);
const older = (await app.inject({ method: 'POST', url: `${URL}/older` })).json().data;
+13
View File
@@ -343,6 +343,19 @@ describe('system-routes', () => {
expect(harness.ctx.store.cleanupSessionsByIds).toHaveBeenCalledWith(new Set());
expect(runStaleSessionCleanup).toHaveBeenCalledOnce();
});
it('deletes the edit history of every session it cleaned (diffs can hold secrets)', async () => {
harness.ctx.tabLayouts.runStaleSessionCleanup = vi.fn(async () => ({
count: 2,
cleaned: [
{ id: 'stale-1', name: 'a' },
{ id: 'stale-2', name: 'b' },
],
})) as never;
const res = await harness.app.inject({ method: 'POST', url: '/api/cleanup-state' });
expect(JSON.parse(res.body).cleanedSessions).toBe(2);
expect(vi.mocked(harness.ctx.forgetEditHistory).mock.calls).toEqual([['stale-1'], ['stale-2']]);
});
});
// ========== GET /api/subagents ==========