From b5066744be621fa3abd750291ae2513f2b67e547 Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Fri, 6 Nov 2026 03:13:03 +0100 Subject: [PATCH] fix(edit-history): a remote read with many subagents spans several round trips The read script carries one case line per subagent cursor, and the whole script is one argument of the ssh line, which Linux caps at 128 KiB. From about 1000 subagents the command failed with E2BIG. The cursors have no cap and the stored conversation is read first, so the session stayed unavailable and not even a /clear got past it. A read with more than 300 subagent requests now takes several round trips. The cut is by name range, not only by count: the script takes a file no case line names for a new one, so plain groups would read each other's files as new. Each round trip walks only the names between its cuts, through one awk, so a new file lands in exactly one range. Budget and meta budget carry over, and a later round trip still lists its files, because a requested file missing from the result loses its cursor. A failed round trip fails the whole read and nothing of it reaches the store. A read of up to 300 runs no awk at all. Each round of the 60-seed fuzz is also cut at a random size and compared with the local reader. Run outside the suite too: bash --posix, BusyBox and mawk, and bash 3.2 in a container, which is macOS sh. That shell cannot parse a case inside $(...), so the names come from a shell function. Co-Authored-By: Claude Opus 5.5 --- docs/architecture-invariants.md | 4 +- docs/remote-sessions.md | 2 +- src/remote-files.ts | 231 +++++++++++++++++++------- src/web/edit-history-sync.ts | 5 +- test/remote-edit-sources.test.ts | 267 +++++++++++++++++++++++++++++++ 5 files changed, 451 insertions(+), 58 deletions(-) diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index cdf0d810..c3fa886a 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -768,7 +768,9 @@ Matching semantics: a query containing `/` matches the relative path, otherwise Paths, agent descriptions and diff lines render via `textContent` only (browser test with a hostile path). The header button uses the marker-class pattern (`btn-edit-history--hidden`) and stays off phones (`mobile.css`, locked in `test/mobile-header-buttons-policy.test.ts`). Docker and non-claude sessions answer `unsupported` and are never read. -⚠️ **Remote (SSH) sessions: read over ssh, never woken.** The sync stays path-based; its deps get the session, and `server.ts` routes them to the local reader or to `remote-files.ts`. `transcriptPath` is one ssh round trip (`buildRemoteClaudeTranscriptLocateCommand`): first the conversation the claude process in the session's remote pane writes now (its `/sessions/.json`, found through the pane's process tree, since a remote host sends Codeman no hooks), else the most recently written `/projects/*/.jsonl` of the stored history's conversation, `` and `` (`remoteTranscriptIds`; nothing moves `claudeSessionId` after a remote `/clear`, and no fixed order is right both after a `/clear` and after a respawn that resumed the launch conversation); the result is an absolute path on the host. A read pass is one more round trip (`buildRemoteClaudeEditSourcesCommand`) with the local reader's semantics: offset clamped to the size, a block with `len` ends at `offset + len`, a new file reads its tail, one budget shared main-first, `onlyRequested` for "Load older", at most 8 KiB of `meta.json` per subagent (within a 4 MiB meta budget). `test/remote-edit-sources.test.ts` runs both scripts on a real `sh` against a temp `$HOME` and compares the read byte for byte with `readClaudeEditSources` on seeded random transcripts, so `/clear` catch-up, "Load older", cursors and the store LRU behave remotely as locally. Every request value reaches the script only after validation (id `[A-Za-z0-9._-]`, relPath `main` or `subagents/agent-.jsonl`, integer offsets) and shellescaped; the scripts run under `sh -c` with `LC_ALL=C`. The config dir is an absolute `CLAUDE_CONFIG_DIR` of the ssh command's environment, else `$HOME/.claude` (claude's own rule); the remote claude runs under `$SHELL -i -l -c`, the lookup under a plain `sh`, so a value set only in a login or interactive profile (`.profile`, `.bash_profile`, `.zshrc`) is not seen. A read's timeout grows with its byte cap (`remoteEditSourcesTimeoutMs`: 30 s plus the transfer at 256 KiB/s), since a forward pass may carry 13 MiB and "Load older" 29 MiB. ⚠️ **Never wakes:** the reads go through `buildRemoteFileCommand` + `runWithRemoteSshLimit` (`ConnectTimeout=10`), so a sleeping host is a connect timeout and the panel shows `unavailable` with the stored entries; the edit-history read path reaches neither `remote-wake.ts` nor `session-routes.ts`, even transitively (guard in `test/remote-edit-sources.test.ts`). Without hooks there is no `transcript:tool_end`, so an open panel re-reads on `session:idle` and the refresh button only. +⚠️ **Remote (SSH) sessions: read over ssh, never woken.** The sync stays path-based; its deps get the session, and `server.ts` routes them to the local reader or to `remote-files.ts`. `transcriptPath` is one ssh round trip (`buildRemoteClaudeTranscriptLocateCommand`): first the conversation the claude process in the session's remote pane writes now (its `/sessions/.json`, found through the pane's process tree, since a remote host sends Codeman no hooks), else the most recently written `/projects/*/.jsonl` of the stored history's conversation, `` and `` (`remoteTranscriptIds`; nothing moves `claudeSessionId` after a remote `/clear`, and no fixed order is right both after a `/clear` and after a respawn that resumed the launch conversation); the result is an absolute path on the host. A read pass is one more round trip (`buildRemoteClaudeEditSourcesCommand`), or several with more than 300 subagent requests (below), with the local reader's semantics: offset clamped to the size, a block with `len` ends at `offset + len`, a new file reads its tail, one budget shared main-first, `onlyRequested` for "Load older", at most 8 KiB of `meta.json` per subagent (within a 4 MiB meta budget per read, which the local reader does not have: a meta comes only with a subagent's first read, so one past the budget is never sent and that subagent shows its id instead of its description). `test/remote-edit-sources.test.ts` runs both scripts on a real `sh` against a temp `$HOME` and compares the read byte for byte with `readClaudeEditSources` on seeded random transcripts, so `/clear` catch-up, "Load older", cursors and the store LRU behave remotely as locally. Every request value reaches the script only after validation (id `[A-Za-z0-9._-]`, relPath `main` or `subagents/agent-.jsonl`, integer offsets) and shellescaped; the scripts run under `sh -c` with `LC_ALL=C`. The config dir is an absolute `CLAUDE_CONFIG_DIR` of the ssh command's environment, else `$HOME/.claude` (claude's own rule); the remote claude runs under `$SHELL -i -l -c`, the lookup under a plain `sh`, so a value set only in a login or interactive profile (`.profile`, `.bash_profile`, `.zshrc`) is not seen. A round trip's timeout grows with its byte cap (`remoteEditSourcesTimeoutMs`: 30 s plus the transfer at 256 KiB/s), since a forward pass may carry 13 MiB and "Load older" 29 MiB. ⚠️ **Never wakes:** the reads go through `buildRemoteEditHistoryCommand` (`buildRemoteFileCommand` with `exec`, so a timeout reaches ssh and not only the local `sh -c`, which on dash stays its parent) + `runWithRemoteSshLimit` (`ConnectTimeout=10`), so a sleeping host is a connect timeout and the panel shows `unavailable` with the stored entries; the edit-history read path reaches neither `remote-wake.ts` nor `session-routes.ts`, even transitively (guard in `test/remote-edit-sources.test.ts`). Without hooks there is no `transcript:tool_end`, so an open panel re-reads on `session:idle` and the refresh button only. + +⚠️ **A read with more than 300 subagent requests spans several round trips** (`planRemoteEditSourceRounds`, `EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP`). The script is one argument of the ssh line, Linux caps a single argv string at 128 KiB (`E2BIG` at about 1000 subagents), the cursors have no cap and the stored conversation is read first, so one script for all of them left a long history `unavailable`, and not even a `/clear` got past it. The cut is by NAME RANGE, not only by count: the script takes a file no `case` line names for a new one (its tail, its meta), so plain groups would read each other's files as new. Each round trip walks only the names between its cuts (one `awk` per round trip, never a fork per file), so a new file lands in exactly one range. Main comes with the first round trip, budget and meta budget carry over, and a later round trip still lists its files, because a requested file missing from the result loses its cursor. A failed round trip fails the whole read and nothing of it reaches the store; each one takes its own limiter slot. The merged result equals the local reader's, byte for byte (the fuzz test cuts every read at a random size). A read of up to 300 runs no `awk` at all. ⚠️ No `case` inside a `$(…)`: bash 3.2, macOS `sh`, cannot parse it, which is why the names come from a shell function. ### Config backups diff --git a/docs/remote-sessions.md b/docs/remote-sessions.md index 70962d72..10f7d158 100644 --- a/docs/remote-sessions.md +++ b/docs/remote-sessions.md @@ -419,7 +419,7 @@ still has it. **Not available over ssh (by choice, not by accident):** editing a file (writes would need SFTP; `docs/file-viewer-edit-plan.md` §6), office-document previews and -generated thumbnails (both need the bytes on the server's disk — no remote file is ever +generated thumbnails (both need the bytes on the server's disk: no remote file is ever spilled onto the server, the Edit history store's diffs being the one exception), the file-tree/picker listings, and `tail-file`. Those routes are still local-only, so with an `sshfs` mount in place they read the mounted copy — diff --git a/src/remote-files.ts b/src/remote-files.ts index f50dd4bd..765b953e 100644 --- a/src/remote-files.ts +++ b/src/remote-files.ts @@ -439,6 +439,15 @@ const EDIT_SOURCE_META_BUDGET_BYTES = 4 * 1024 * 1024; /** Room for the record headers (one per transcript file) on top of budget and metas. */ const EDIT_SOURCE_HEADER_SLACK_BYTES = 1024 * 1024; +/** + * Subagent requests per ssh round trip of an edit-history read. Like + * {@link REMOTE_PROBE_CHUNK_SIZE}: the script is ONE argument of the ssh line, Linux caps a + * single argv string at 128 KiB, and each request adds a `case` line of 125 bytes once + * shellescaped twice (about 370 with the longest name a filesystem allows; the test pins that + * worst case). See {@link planRemoteEditSourceRounds}. + */ +const EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP = 300; + /** * The slowest link an edit-history read is given time for. Such a read can carry megabytes * (a forward pass its budget plus the metas, "Load older" up to three budgets), which a flat @@ -618,9 +627,79 @@ export async function remoteLocateClaudeTranscript( return path; } +/** The requests a read script may take: the first per relPath, valid ones only. */ +function validEditSourceRequests(requests: readonly EditSourceRequest[]): Map { + const byRelPath = new Map(); + for (const request of requests) { + if (!request || typeof request.relPath !== 'string') continue; + if (!EDIT_SOURCE_REL_PATH_PATTERN.test(request.relPath)) continue; + if (!isByteCount(request.offset)) continue; + if (request.len !== undefined && !isByteCount(request.len)) continue; + if (byRelPath.has(request.relPath)) continue; + byRelPath.set(request.relPath, request); + } + return byRelPath; +} + +/** One ssh round trip of an edit-history read ({@link planRemoteEditSourceRounds}). */ +export interface RemoteEditSourceRound { + requests: EditSourceRequest[]; + /** It reads the main transcript: the first round trip of a read does. */ + main: boolean; + /** It takes the subagent files after this name ('' = from the first), in byte order. */ + after: string; + /** It takes them up to and including this name ('' = to the last). */ + upTo: string; +} + /** - * Script that reads a remote claude session's main transcript at `mainPath` and its subagent - * transcripts (`/subagents/agent-.jsonl`) in ONE ssh round trip, + * The ssh round trips a read takes. Up to `perRound` subagent requests fit one; more are sorted + * and cut into groups of `perRound`, or the ssh line would hit `E2BIG` and the session would + * stay `unavailable`: the cursors have no cap and the stored conversation is read first, so + * not even a `/clear` would get past it. + * + * ⚠️ The cuts are NAME RANGES, not only groups of requests. The script takes a file that no + * `case` line names for a new one (its tail, its meta), so a plain group would read every + * file of the other groups as new. Each round trip walks only the names between its cuts; the + * ranges cover every possible name exactly once, so a new file lands in one of them and the + * merged result is the one a single script would give (the fuzz test compares it with the + * local reader). + */ +export function planRemoteEditSourceRounds( + requests: readonly EditSourceRequest[], + perRound: number = EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP +): RemoteEditSourceRound[] { + const valid = validEditSourceRequests(requests); + const subagents = [...valid.values()].filter((request) => request.relPath !== 'main'); + if (subagents.length <= perRound) return [{ requests: [...valid.values()], main: true, after: '', upTo: '' }]; + + const main = valid.get('main'); + const fileName = (request: EditSourceRequest): string => request.relPath.slice('subagents/'.length); + subagents.sort((a, b) => (a.relPath < b.relPath ? -1 : 1)); + const rounds: RemoteEditSourceRound[] = []; + for (let i = 0; i < subagents.length; i += perRound) { + const group = subagents.slice(i, i + perRound); + rounds.push({ + requests: i === 0 && main ? [main, ...group] : group, + main: i === 0, + after: i === 0 ? '' : fileName(subagents[i - 1]), + upTo: i + perRound < subagents.length ? fileName(group[group.length - 1]) : '', + }); + } + return rounds; +} + +/** What {@link buildRemoteClaudeEditSourcesCommand} takes beyond the local reader's options. */ +export interface RemoteEditSourceCommandOptions extends EditSourceReadOptions { + /** Which part of a split read this command is; without it, all of it. */ + round?: Pick; + /** Meta bytes this command may send; without it, {@link EDIT_SOURCE_META_BUDGET_BYTES}. */ + metaBudget?: number; +} + +/** + * Script for one round trip of a read of a remote claude session's main transcript at + * `mainPath` and its subagent transcripts (`/subagents/agent-.jsonl`), * with exactly the local reader's rules (`readClaudeEditSources`): * * - a request without `len` reads from `offset` (clamped to the size); @@ -631,7 +710,8 @@ export async function remoteLocateClaudeTranscript( * * `budgetBytes` is shared by all transcript files, main first, then the subagents in byte * order. A file that gets none of it still produces a zero-length record, because the caller - * needs its size. Exit {@link TRANSCRIPT_NOT_FOUND_EXIT} when `mainPath` is not a file. + * needs its size. Exit {@link TRANSCRIPT_NOT_FOUND_EXIT} when `mainPath` is not a file. A read + * with more requests than fit one script is several of these ({@link planRemoteEditSourceRounds}). * * Output: `\0\0`, then per record `\0\0\0\0` * followed by exactly `` raw bytes. Should a file shrink between `wc -c` and the read, @@ -646,30 +726,30 @@ export function buildRemoteClaudeEditSourcesCommand( mainPath: string, requests: readonly EditSourceRequest[], budgetBytes: number, - opts: EditSourceReadOptions = {} + opts: RemoteEditSourceCommandOptions = {} ): string { if (!isRemoteClaudeTranscriptPath(mainPath)) return `exit ${TRANSCRIPT_NOT_FOUND_EXIT}`; const transcriptId = mainPath.slice(mainPath.lastIndexOf('/') + 1, -'.jsonl'.length); - const byRelPath = new Map(); - for (const request of requests) { - if (!request || typeof request.relPath !== 'string') continue; - if (!EDIT_SOURCE_REL_PATH_PATTERN.test(request.relPath)) continue; - if (!isByteCount(request.offset)) continue; - if (request.len !== undefined && !isByteCount(request.len)) continue; - if (byRelPath.has(request.relPath)) continue; - byRelPath.set(request.relPath, { offset: request.offset, len: request.len ?? null }); - } + const byRelPath = validEditSourceRequests(requests); const budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; - const requestArgs = (request: { offset: number; len: number | null }): string => - `r ${request.offset} ${request.len === null ? "''" : String(request.len)}`; + const requestArgs = (request: EditSourceRequest): string => + `r ${request.offset} ${request.len === undefined ? "''" : String(request.len)}`; + const { main: readsMain, after, upTo } = opts.round ?? { main: true, after: '', upTo: '' }; const main = byRelPath.get('main'); - const mainLine = main - ? `emit main "$best" ${requestArgs(main)}` - : opts.onlyRequested - ? '' - : 'emit main "$best" t 0 \'\''; + const mainLine = !readsMain + ? '' + : main + ? `emit main "$best" ${requestArgs(main)}` + : opts.onlyRequested + ? '' + : 'emit main "$best" t 0 \'\''; + // The names come from validated relPaths, and `$0` stays a string: every name starts `agent-`. + const rangeFilter = + after || upTo + ? ` | awk -v after=${shellescape(after)} -v upto=${shellescape(upTo)} '$0 > after && (upto == "" || $0 <= upto)'` + : ''; const subagentCases = [...byRelPath.entries()] .filter(([relPath]) => relPath !== 'main') .map(([relPath, request]) => { @@ -699,7 +779,7 @@ export function buildRemoteClaudeEditSourcesCommand( `[ -f "$best" ] || exit ${TRANSCRIPT_NOT_FOUND_EXIT}`, `printf '\\0%s\\0' ${shellescape(transcriptId)}`, `budget=${budget}`, - `mbudget=${EDIT_SOURCE_META_BUDGET_BYTES}`, + `mbudget=${opts.metaBudget ?? EDIT_SOURCE_META_BUDGET_BYTES}`, // Size via `wc -c <` (POSIX; BSD pads it with spaces, hence the tr). 'fsize() {', ' s=$(wc -c < "$1" 2>/dev/null | tr -d \' \\t\\n\')', @@ -728,11 +808,20 @@ export function buildRemoteClaudeEditSourcesCommand( ' printf \'%s\\0%s\\0%s\\0%s\\0\' "$1" "$s" "$start" "$n"', ' [ "$n" -eq 0 ] || bytes "$2" "$start" "$n"', '}', + // The valid names, one per line in byte order (the glob's order under `C`). Nothing but + // `[A-Za-z0-9_.-]` is left, so the unquoted `$(…)` below splits on them and globs nothing. + // A function, because bash 3.2 (macOS `sh`) cannot parse a `case` inside `$(…)`. + 'subagent_names() {', + ' for f in "${best%.jsonl}"/subagents/agent-*.jsonl; do', + ' b=${f##*/}', + ' case $b in agent-.jsonl|agent-*[!A-Za-z0-9_-]*.jsonl) continue ;; esac', + ' printf \'%s\\n\' "$b"', + ' done', + '}', mainLine, - 'for f in "${best%.jsonl}"/subagents/agent-*.jsonl; do', + `for b in $(subagent_names${rangeFilter}); do`, + ' f=${best%.jsonl}/subagents/$b', ' [ -f "$f" ] || continue', - ' b=${f##*/}', - ' case $b in agent-.jsonl|agent-*[!A-Za-z0-9_-]*.jsonl) continue ;; esac', ' case $b in', ...subagentCases, ...newSubagent, @@ -795,9 +884,53 @@ export function parseRemoteClaudeEditSources(stdout: Buffer): EditSources | null } /** - * Read a remote claude session's transcripts (see {@link buildRemoteClaudeEditSourcesCommand}), - * the remote twin of `readClaudeEditSources`. `null` when `mainPath` does not exist; an - * unreachable host, an ssh failure or broken output is a `RemoteFileAccessError`. Never wakes. + * Runs one read command on the host: its stdout, at most `maxBytes`, or null for "no such + * transcript" ({@link TRANSCRIPT_NOT_FOUND_EXIT}). + */ +export type RemoteEditSourceRunner = (command: string, maxBytes: number) => Promise; + +/** + * Run the round trips of a read ({@link planRemoteEditSourceRounds}) one after the other and + * merge them into what one read returns. Budget and meta budget carry over: a later round trip + * gets what the earlier ones left, and still lists its files, because a requested file missing + * from the result counts as deleted and loses its cursor. So a read is all or nothing: a failed + * round trip fails it, and nothing of it reaches the store. + */ +export async function readRemoteEditSourceRounds( + mainPath: string, + requests: readonly EditSourceRequest[], + budgetBytes: number, + opts: EditSourceReadOptions, + run: RemoteEditSourceRunner, + perRound?: number +): Promise { + let budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; + let metaBudget = EDIT_SOURCE_META_BUDGET_BYTES; + const merged: EditSources = { transcriptId: '', files: [], metas: {} }; + for (const round of planRemoteEditSourceRounds(requests, perRound)) { + const stdout = await run( + buildRemoteClaudeEditSourcesCommand(mainPath, round.requests, budget, { ...opts, round, metaBudget }), + budget + metaBudget + EDIT_SOURCE_HEADER_SLACK_BYTES + ); + if (!stdout) return null; + const parsed = parseRemoteClaudeEditSources(stdout); + if (!parsed || !mainPath.endsWith(`/${parsed.transcriptId}.jsonl`)) { + throw new RemoteFileAccessError('unexpected output from the remote edit-history read'); + } + merged.transcriptId = parsed.transcriptId; + merged.files.push(...parsed.files); + Object.assign(merged.metas, parsed.metas); + for (const file of parsed.files) budget -= file.chunk.length; + for (const text of Object.values(parsed.metas)) metaBudget = Math.max(0, metaBudget - Buffer.byteLength(text)); + } + return merged; +} + +/** + * Read a remote claude session's transcripts (see {@link readRemoteEditSourceRounds}), the + * remote twin of `readClaudeEditSources`. `null` when `mainPath` does not exist; an unreachable + * host, an ssh failure or broken output is a `RemoteFileAccessError`. Each round trip takes its + * own slot of the ssh limiter. Never wakes. */ export async function remoteReadClaudeEditSources( remote: SessionRemote, @@ -807,33 +940,23 @@ export async function remoteReadClaudeEditSources( opts: EditSourceReadOptions = {} ): Promise { assertNotUnderTest(); - const budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; - const command = buildRemoteEditHistoryCommand( - remote, - buildRemoteClaudeEditSourcesCommand(mainPath, requests, budget, opts) - ); - const maxBytes = budget + EDIT_SOURCE_META_BUDGET_BYTES + EDIT_SOURCE_HEADER_SLACK_BYTES; - let stdout: Buffer; - try { - const result = await runWithRemoteSshLimit(() => - execAsync(command, { - timeout: remoteEditSourcesTimeoutMs(maxBytes), - maxBuffer: maxBytes, - encoding: 'buffer', - }) - ); - stdout = Buffer.isBuffer(result.stdout) ? result.stdout : Buffer.from(result.stdout); - } catch (err) { - if ((err as { code?: unknown }).code === TRANSCRIPT_NOT_FOUND_EXIT) return null; - throw new RemoteFileAccessError( - `remote host ${remote.label || remote.host} unreachable: ${describeExecError(err)}` - ); - } - const parsed = parseRemoteClaudeEditSources(stdout); - if (!parsed || !mainPath.endsWith(`/${parsed.transcriptId}.jsonl`)) { - throw new RemoteFileAccessError('unexpected output from the remote edit-history read'); - } - return parsed; + return readRemoteEditSourceRounds(mainPath, requests, budgetBytes, opts, async (command, maxBytes) => { + try { + const result = await runWithRemoteSshLimit(() => + execAsync(buildRemoteEditHistoryCommand(remote, command), { + timeout: remoteEditSourcesTimeoutMs(maxBytes), + maxBuffer: maxBytes, + encoding: 'buffer', + }) + ); + return Buffer.isBuffer(result.stdout) ? result.stdout : Buffer.from(result.stdout); + } catch (err) { + if ((err as { code?: unknown }).code === TRANSCRIPT_NOT_FOUND_EXIT) return null; + throw new RemoteFileAccessError( + `remote host ${remote.label || remote.host} unreachable: ${describeExecError(err)}` + ); + } + }); } /** diff --git a/src/web/edit-history-sync.ts b/src/web/edit-history-sync.ts index 1852734a..bd3377c7 100644 --- a/src/web/edit-history-sync.ts +++ b/src/web/edit-history-sync.ts @@ -33,8 +33,9 @@ * * WHICH SESSIONS. Claude sessions, local and remote. Docker and non-claude sessions answer * `unsupported` and are never read. The deps get the session and read a remote one over ssh - * (`remote-files.ts`): one round trip to find its transcript, one per read. A remote read never - * wakes its host; a sleeping one is a connect timeout (`unavailable`, stored entries kept). + * (`remote-files.ts`): one round trip to find its transcript, one per read (more for a + * conversation with hundreds of subagents). A remote read never wakes its host; a sleeping one + * is a connect timeout (`unavailable`, stored entries kept). * * IMAGES. A Bash candidate counts only when the file's mtime lies in the call's window * (start − 2 s … end + 2 s), so `ls *.png` lists nothing. Transcript times and mtimes come diff --git a/test/remote-edit-sources.test.ts b/test/remote-edit-sources.test.ts index 66639ddd..f0f37101 100644 --- a/test/remote-edit-sources.test.ts +++ b/test/remote-edit-sources.test.ts @@ -34,11 +34,15 @@ import { isRemoteClaudeTranscriptPath, parseRemoteClaudeEditSources, parseRemoteClaudeTranscriptLocate, + planRemoteEditSourceRounds, + readRemoteEditSourceRounds, remoteEditSourcesTimeoutMs, remoteLocateClaudeTranscript, remoteReadClaudeEditSources, } from '../src/remote-files.js'; import { readClaudeEditSources, type EditSourceRequest } from '../src/edit-history-local-source.js'; +import { EditHistoryStore } from '../src/edit-history-store.js'; +import { EditHistorySync, type EditHistorySession } from '../src/web/edit-history-sync.js'; import type { SessionRemote } from '../src/types/session.js'; let home: string; @@ -62,6 +66,12 @@ function runSh(command: string, env: Record = {}): { code: numbe } } +/** Runs a read command the way a {@link readRemoteEditSourceRounds} runner does, here on `sh`. */ +async function runRead(command: string): Promise { + const { code, stdout } = runSh(command); + return code === 0 ? stdout : null; +} + /** One edit-source record in the script's wire format. */ function editRecord(relPath: string, size: number, start: number, body: Buffer | string): Buffer { const bytes = Buffer.isBuffer(body) ? body : Buffer.from(body); @@ -259,6 +269,59 @@ describe('buildRemoteClaudeEditSourcesCommand', () => { }); }); +describe('planRemoteEditSourceRounds', () => { + const agents = (...ids: string[]): EditSourceRequest[] => + ids.map((id) => ({ relPath: `subagents/agent-${id}.jsonl`, offset: 1 })); + + it('keeps a read that fits one script in one round trip, with no name range', () => { + const requests = [{ relPath: 'main', offset: 5 }, ...agents('b', 'a')]; + expect(planRemoteEditSourceRounds(requests, 2)).toEqual([{ requests, main: true, after: '', upTo: '' }]); + expect(planRemoteEditSourceRounds([], 2)).toEqual([{ requests: [], main: true, after: '', upTo: '' }]); + }); + + it('cuts more requests by name: ranges that join up, main in the first, every request once', () => { + const main = { relPath: 'main', offset: 5 }; + const rounds = planRemoteEditSourceRounds([...agents('e', 'c'), main, ...agents('a', 'd', 'b')], 2); + expect(rounds.map((r) => [r.main, r.after, r.upTo, r.requests.map((q) => q.relPath)])).toEqual([ + [true, '', 'agent-b.jsonl', ['main', 'subagents/agent-a.jsonl', 'subagents/agent-b.jsonl']], + [false, 'agent-b.jsonl', 'agent-d.jsonl', ['subagents/agent-c.jsonl', 'subagents/agent-d.jsonl']], + [false, 'agent-d.jsonl', '', ['subagents/agent-e.jsonl']], + ]); + }); + + it('counts only valid requests, the first per relPath', () => { + const rounds = planRemoteEditSourceRounds( + [ + ...agents('a'), + { relPath: 'subagents/agent-a.jsonl', offset: 9 }, + { relPath: '../../etc/passwd', offset: 0 }, + { relPath: 'subagents/agent-neg.jsonl', offset: -1 }, + ], + 1 + ); + expect(rounds).toEqual([{ requests: agents('a'), main: true, after: '', upTo: '' }]); + }); + + it('keeps every ssh line under the 128 KiB argv limit, longest names and offsets included', () => { + // 255 bytes is the longest file name a filesystem allows; a request without `len` is the + // longer one, because its `''` costs 13 bytes per quote once shellescaped twice. + const requests: EditSourceRequest[] = Array.from({ length: 5000 }, (_, i) => ({ + relPath: `subagents/agent-${String(i).padStart(6, '0')}${'x'.repeat(237)}.jsonl`, + offset: Number.MAX_SAFE_INTEGER, + })); + const mainPath = `/${'p/'.repeat(2000)}conv-1.jsonl`; + const line = (reqs: EditSourceRequest[], round?: { main: boolean; after: string; upTo: string }) => + Buffer.byteLength( + buildRemoteEditHistoryCommand(REMOTE, buildRemoteClaudeEditSourcesCommand(mainPath, reqs, 1000, { round })) + ); + const ARGV_LIMIT = 128 * 1024; + expect(line(requests)).toBeGreaterThan(ARGV_LIMIT); + const rounds = planRemoteEditSourceRounds(requests); + expect(rounds.length).toBeGreaterThan(1); + for (const round of rounds) expect(line(round.requests, round)).toBeLessThan(ARGV_LIMIT); + }); +}); + describe('parseRemoteClaudeEditSources', () => { it('splits several records byte-exactly and keeps a UTF-8 sequence cut at a chunk edge intact as bytes', () => { const euro = Buffer.from('a€', 'utf8'); // 61 e2 82 ac @@ -416,6 +479,8 @@ describe('the transcript read on a real shell', () => { const local = await readClaudeEditSources(path, requests, budget, { onlyRequested }); const { code, stdout } = runSh(buildRemoteClaudeEditSourcesCommand(path, requests, budget, { onlyRequested })); const remote = code === 0 ? parseRemoteClaudeEditSources(stdout) : null; + // The same read cut into several round trips must come out the same. + const split = await readRemoteEditSourceRounds(path, requests, budget, { onlyRequested }, runRead, ri(1, 3)); const shape = (r: typeof local) => r && { transcriptId: r.transcriptId, @@ -424,6 +489,7 @@ describe('the transcript read on a real shell', () => { }; try { expect(shape(remote)).toEqual(shape(local)); + expect(shape(split)).toEqual(shape(local)); } catch { failures.push(`seed ${seed}: ${JSON.stringify({ requests, budget, onlyRequested })}`); } @@ -432,6 +498,207 @@ describe('the transcript read on a real shell', () => { }); }); +describe('a read too large for one ssh line', () => { + // `ssh` is a fake that runs the remote command on `sh`, so the whole ssh line really goes + // through the local `sh -c` argv: what must not hit E2BIG. + let bin: string; + beforeAll(() => { + bin = mkdtempSync(join(tmpdir(), 'codeman-fake-ssh-read-')); + writeFileSync(join(bin, 'ssh'), '#!/bin/sh\nfor arg; do :; done\nexec sh -c "$arg"\n'); + chmodSync(join(bin, 'ssh'), 0o755); + }); + afterAll(() => rmSync(bin, { recursive: true, force: true })); + + /** A runner that goes through the fake ssh, counting its round trips. */ + function overSsh(): { run: (command: string, maxBytes: number) => Promise; trips: () => number } { + let trips = 0; + return { + trips: () => trips, + run: async (command, maxBytes) => { + trips++; + try { + return execFileSync('sh', ['-c', buildRemoteEditHistoryCommand(REMOTE, command)], { + env: { ...process.env, HOME: home, PATH: `${bin}:${process.env.PATH}` }, + cwd: home, + maxBuffer: maxBytes, + }); + } catch (err) { + if ((err as { status?: number }).status === 3) return null; + throw err; + } + }, + }; + } + + const shape = (r: Awaited>) => + r && { + transcriptId: r.transcriptId, + metas: r.metas, + files: r.files.map((f) => [f.relPath, f.size, f.startOffset, f.chunk.toString('hex')]), + }; + + describe('as a read', () => { + const project = () => join(home, '.claude', 'projects', 'many'); + const main = () => join(project(), 'conv-many.jsonl'); + const sub = () => join(project(), 'conv-many', 'subagents'); + const id = (i: number) => `a${String(i).padStart(4, '0')}`; + /** A cursor for each of 1500 subagents; only every 100th still has a file. */ + const requests: EditSourceRequest[] = [ + { relPath: 'main', offset: 2 }, + ...Array.from({ length: 1500 }, (_, i) => ({ relPath: `subagents/agent-${id(i)}.jsonl`, offset: 1 })), + ]; + + beforeAll(() => { + mkdirSync(sub(), { recursive: true }); + writeFileSync(main(), 'main transcript\n'); + for (let i = 0; i < 1500; i += 100) writeFileSync(join(sub(), `agent-${id(i)}.jsonl`), `subagent ${i}\n`); + // New files in the first, a middle and the last name range: each read once, with its meta. + for (const name of ['agent-0first', 'agent-a0750new', 'agent-zlast']) { + writeFileSync(join(sub(), `${name}.jsonl`), `${name}\n`); + writeFileSync(join(sub(), `${name}.meta.json`), `{"description":"${name}"}`); + } + }); + + it.each([0, 40, 8 * 1024 * 1024])('reads what the local reader reads, budget %i', async (budget) => { + const ssh = overSsh(); + const read = await readRemoteEditSourceRounds(main(), requests, budget, {}, ssh.run); + expect(shape(read)).toEqual(shape(await readClaudeEditSources(main(), requests, budget))); + expect(ssh.trips()).toBe(5); + }); + + it('is null when the transcript goes away during the read', async () => { + let trips = 0; + const read = readRemoteEditSourceRounds(main(), requests, 100, {}, async (command) => { + return ++trips === 2 ? null : runRead(command); + }); + await expect(read).resolves.toBeNull(); + expect(trips).toBe(2); + }); + + it('fails as a whole on broken output from a later round trip', async () => { + let trips = 0; + const read = readRemoteEditSourceRounds(main(), requests, 100, {}, async (command) => { + return ++trips === 2 ? Buffer.from('\0other-conversation\0') : runRead(command); + }); + await expect(read).rejects.toBeInstanceOf(RemoteFileAccessError); + expect(trips).toBe(2); + }); + + it('gives a later round trip the meta budget the earlier ones left', async () => { + const commands: string[] = []; + await readRemoteEditSourceRounds( + main(), + requests, + 1000, + {}, + async (command) => { + commands.push(command); + return runRead(command); + }, + 900 + ); + const budgets = commands.map((c) => Number(/mbudget=(\d+)/.exec(c)?.[1])); + expect(budgets).toHaveLength(2); + // `agent-0first` sorts before every request, so its meta went with the first round trip. + expect(budgets[0] - budgets[1]).toBe( + Buffer.byteLength('{"description":"agent-0first"}') + Buffer.byteLength('{"description":"agent-a0750new"}') + ); + }); + }); + + describe('through the sync', () => { + const project = () => join(home, '.claude', 'projects', 'long-history'); + const sub = () => join(project(), 'conv-old', 'subagents'); + const edit = (id: string, path: string) => + JSON.stringify({ + type: 'user', + timestamp: '2026-10-15T20:00:00.000Z', + cwd: '/w', + message: { role: 'user', content: [{ tool_use_id: id, type: 'tool_result', content: 'ok' }] }, + toolUseResult: { + filePath: path, + oldString: 'a', + newString: 'b', + structuredPatch: [{ oldStart: 1, oldLines: 1, newStart: 1, newLines: 1, lines: ['-a', '+b'] }], + }, + }) + '\n'; + + it('keeps a conversation with 1200 subagents available, and a /clear gets past them', async () => { + const dir = mkdtempSync(join(tmpdir(), 'codeman-edit-store-')); + try { + mkdirSync(sub(), { recursive: true }); + const oldMain = join(project(), 'conv-old.jsonl'); + const newMain = join(project(), 'conv-new.jsonl'); + writeFileSync(oldMain, '{}\n'); + // Ids as long as claude's own: with these, 1200 requests do not fit one ssh line. + const name = (i: number) => `agent-a${String(i).padStart(16, '0')}`; + for (let i = 0; i < 1200; i++) writeFileSync(join(sub(), `${name(i)}.jsonl`), '{}\n'); + + const ssh = overSsh(); + let path = oldMain; + const sync = new EditHistorySync({ + transcriptPath: async () => path, + readSources: (_session, mainPath, requests, budget, opts) => + readRemoteEditSourceRounds(mainPath, requests, budget, opts ?? {}, ssh.run), + probe: async () => [], + loadStore: (id) => EditHistoryStore.load(id, { dir, debounceMs: 0 }), + deleteStore: async () => {}, + sweepStores: async () => 0, + }); + const session: EditHistorySession = { + id: 's1', + mode: 'claude', + remote: REMOTE, + claudeSessionId: 'conv-old', + }; + // The cursors of a history that was read before, one per file at its end. + const store = await sync.getStore('s1'); + const seed = (source: string) => + store.apply({ transcriptId: 'conv-old', source, startOffset: 0, chunk: Buffer.from('{}\n'), size: 3 }); + seed('main'); + for (let i = 0; i < 1200; i++) seed(`subagents/${name(i)}.jsonl`); + expect(store.knownSources()).toHaveLength(1201); + + // Past the argv limit: one script would not even start. + const all = store.knownSources().map((relPath) => ({ relPath, offset: 3 })); + const line = buildRemoteEditHistoryCommand(REMOTE, buildRemoteClaudeEditSourcesCommand(oldMain, all, 1000)); + expect(() => execFileSync('sh', ['-c', line])).toThrow(/E2BIG/); + + // A known file that grew in the last name range, a new one in the middle. + writeFileSync(join(sub(), `${name(1199)}.jsonl`), `{}\n${edit('late', '/w/late.ts')}`); + const fresh = `${name(600)}new`; + writeFileSync(join(sub(), `${fresh}.jsonl`), edit('fresh', '/w/fresh.ts')); + writeFileSync(join(sub(), `${fresh}.meta.json`), '{"description":"new one","agentType":"Explore"}'); + expect(await sync.sync(session)).toEqual({ state: 'ok' }); + expect(ssh.trips()).toBe(4); + expect( + store + .list() + .map((e) => e.path) + .sort() + ).toEqual(['/w/fresh.ts', '/w/late.ts']); + expect(store.getAgents()[fresh.slice('agent-'.length)]).toMatchObject({ description: 'new one' }); + expect(store.knownSources()).toHaveLength(1202); // no cursor lost + + // /clear: the stored conversation is read to its end first, then the new one. + writeFileSync(newMain, edit('after', '/w/after.ts')); + path = newMain; + expect(await sync.sync(session)).toEqual({ state: 'ok' }); + expect(store.getTranscriptId()).toBe('conv-new'); + expect( + store + .list() + .map((e) => e.path) + .sort() + ).toEqual(['/w/after.ts', '/w/fresh.ts', '/w/late.ts']); + await store.remove(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); + }); +}); + describe('a timeout ends ssh itself', () => { // Node signals the local `sh -c`. On dash that shell stays the parent of a plain command, so // only `exec` makes the signal reach ssh. The fake ssh records its pid and then just waits.