diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index cdf0d810..c3fa886a 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -768,7 +768,9 @@ Matching semantics: a query containing `/` matches the relative path, otherwise Paths, agent descriptions and diff lines render via `textContent` only (browser test with a hostile path). The header button uses the marker-class pattern (`btn-edit-history--hidden`) and stays off phones (`mobile.css`, locked in `test/mobile-header-buttons-policy.test.ts`). Docker and non-claude sessions answer `unsupported` and are never read. -⚠️ **Remote (SSH) sessions: read over ssh, never woken.** The sync stays path-based; its deps get the session, and `server.ts` routes them to the local reader or to `remote-files.ts`. `transcriptPath` is one ssh round trip (`buildRemoteClaudeTranscriptLocateCommand`): first the conversation the claude process in the session's remote pane writes now (its `/sessions/.json`, found through the pane's process tree, since a remote host sends Codeman no hooks), else the most recently written `/projects/*/.jsonl` of the stored history's conversation, `` and `` (`remoteTranscriptIds`; nothing moves `claudeSessionId` after a remote `/clear`, and no fixed order is right both after a `/clear` and after a respawn that resumed the launch conversation); the result is an absolute path on the host. A read pass is one more round trip (`buildRemoteClaudeEditSourcesCommand`) with the local reader's semantics: offset clamped to the size, a block with `len` ends at `offset + len`, a new file reads its tail, one budget shared main-first, `onlyRequested` for "Load older", at most 8 KiB of `meta.json` per subagent (within a 4 MiB meta budget). `test/remote-edit-sources.test.ts` runs both scripts on a real `sh` against a temp `$HOME` and compares the read byte for byte with `readClaudeEditSources` on seeded random transcripts, so `/clear` catch-up, "Load older", cursors and the store LRU behave remotely as locally. Every request value reaches the script only after validation (id `[A-Za-z0-9._-]`, relPath `main` or `subagents/agent-.jsonl`, integer offsets) and shellescaped; the scripts run under `sh -c` with `LC_ALL=C`. The config dir is an absolute `CLAUDE_CONFIG_DIR` of the ssh command's environment, else `$HOME/.claude` (claude's own rule); the remote claude runs under `$SHELL -i -l -c`, the lookup under a plain `sh`, so a value set only in a login or interactive profile (`.profile`, `.bash_profile`, `.zshrc`) is not seen. A read's timeout grows with its byte cap (`remoteEditSourcesTimeoutMs`: 30 s plus the transfer at 256 KiB/s), since a forward pass may carry 13 MiB and "Load older" 29 MiB. ⚠️ **Never wakes:** the reads go through `buildRemoteFileCommand` + `runWithRemoteSshLimit` (`ConnectTimeout=10`), so a sleeping host is a connect timeout and the panel shows `unavailable` with the stored entries; the edit-history read path reaches neither `remote-wake.ts` nor `session-routes.ts`, even transitively (guard in `test/remote-edit-sources.test.ts`). Without hooks there is no `transcript:tool_end`, so an open panel re-reads on `session:idle` and the refresh button only. +⚠️ **Remote (SSH) sessions: read over ssh, never woken.** The sync stays path-based; its deps get the session, and `server.ts` routes them to the local reader or to `remote-files.ts`. `transcriptPath` is one ssh round trip (`buildRemoteClaudeTranscriptLocateCommand`): first the conversation the claude process in the session's remote pane writes now (its `/sessions/.json`, found through the pane's process tree, since a remote host sends Codeman no hooks), else the most recently written `/projects/*/.jsonl` of the stored history's conversation, `` and `` (`remoteTranscriptIds`; nothing moves `claudeSessionId` after a remote `/clear`, and no fixed order is right both after a `/clear` and after a respawn that resumed the launch conversation); the result is an absolute path on the host. A read pass is one more round trip (`buildRemoteClaudeEditSourcesCommand`), or several with more than 300 subagent requests (below), with the local reader's semantics: offset clamped to the size, a block with `len` ends at `offset + len`, a new file reads its tail, one budget shared main-first, `onlyRequested` for "Load older", at most 8 KiB of `meta.json` per subagent (within a 4 MiB meta budget per read, which the local reader does not have: a meta comes only with a subagent's first read, so one past the budget is never sent and that subagent shows its id instead of its description). `test/remote-edit-sources.test.ts` runs both scripts on a real `sh` against a temp `$HOME` and compares the read byte for byte with `readClaudeEditSources` on seeded random transcripts, so `/clear` catch-up, "Load older", cursors and the store LRU behave remotely as locally. Every request value reaches the script only after validation (id `[A-Za-z0-9._-]`, relPath `main` or `subagents/agent-.jsonl`, integer offsets) and shellescaped; the scripts run under `sh -c` with `LC_ALL=C`. The config dir is an absolute `CLAUDE_CONFIG_DIR` of the ssh command's environment, else `$HOME/.claude` (claude's own rule); the remote claude runs under `$SHELL -i -l -c`, the lookup under a plain `sh`, so a value set only in a login or interactive profile (`.profile`, `.bash_profile`, `.zshrc`) is not seen. A round trip's timeout grows with its byte cap (`remoteEditSourcesTimeoutMs`: 30 s plus the transfer at 256 KiB/s), since a forward pass may carry 13 MiB and "Load older" 29 MiB. ⚠️ **Never wakes:** the reads go through `buildRemoteEditHistoryCommand` (`buildRemoteFileCommand` with `exec`, so a timeout reaches ssh and not only the local `sh -c`, which on dash stays its parent) + `runWithRemoteSshLimit` (`ConnectTimeout=10`), so a sleeping host is a connect timeout and the panel shows `unavailable` with the stored entries; the edit-history read path reaches neither `remote-wake.ts` nor `session-routes.ts`, even transitively (guard in `test/remote-edit-sources.test.ts`). Without hooks there is no `transcript:tool_end`, so an open panel re-reads on `session:idle` and the refresh button only. + +⚠️ **A read with more than 300 subagent requests spans several round trips** (`planRemoteEditSourceRounds`, `EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP`). The script is one argument of the ssh line, Linux caps a single argv string at 128 KiB (`E2BIG` at about 1000 subagents), the cursors have no cap and the stored conversation is read first, so one script for all of them left a long history `unavailable`, and not even a `/clear` got past it. The cut is by NAME RANGE, not only by count: the script takes a file no `case` line names for a new one (its tail, its meta), so plain groups would read each other's files as new. Each round trip walks only the names between its cuts (one `awk` per round trip, never a fork per file), so a new file lands in exactly one range. Main comes with the first round trip, budget and meta budget carry over, and a later round trip still lists its files, because a requested file missing from the result loses its cursor. A failed round trip fails the whole read and nothing of it reaches the store; each one takes its own limiter slot. The merged result equals the local reader's, byte for byte (the fuzz test cuts every read at a random size). A read of up to 300 runs no `awk` at all. ⚠️ No `case` inside a `$(…)`: bash 3.2, macOS `sh`, cannot parse it, which is why the names come from a shell function. ### Config backups diff --git a/docs/remote-sessions.md b/docs/remote-sessions.md index 70962d72..10f7d158 100644 --- a/docs/remote-sessions.md +++ b/docs/remote-sessions.md @@ -419,7 +419,7 @@ still has it. **Not available over ssh (by choice, not by accident):** editing a file (writes would need SFTP; `docs/file-viewer-edit-plan.md` §6), office-document previews and -generated thumbnails (both need the bytes on the server's disk — no remote file is ever +generated thumbnails (both need the bytes on the server's disk: no remote file is ever spilled onto the server, the Edit history store's diffs being the one exception), the file-tree/picker listings, and `tail-file`. Those routes are still local-only, so with an `sshfs` mount in place they read the mounted copy — diff --git a/src/remote-files.ts b/src/remote-files.ts index f50dd4bd..765b953e 100644 --- a/src/remote-files.ts +++ b/src/remote-files.ts @@ -439,6 +439,15 @@ const EDIT_SOURCE_META_BUDGET_BYTES = 4 * 1024 * 1024; /** Room for the record headers (one per transcript file) on top of budget and metas. */ const EDIT_SOURCE_HEADER_SLACK_BYTES = 1024 * 1024; +/** + * Subagent requests per ssh round trip of an edit-history read. Like + * {@link REMOTE_PROBE_CHUNK_SIZE}: the script is ONE argument of the ssh line, Linux caps a + * single argv string at 128 KiB, and each request adds a `case` line of 125 bytes once + * shellescaped twice (about 370 with the longest name a filesystem allows; the test pins that + * worst case). See {@link planRemoteEditSourceRounds}. + */ +const EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP = 300; + /** * The slowest link an edit-history read is given time for. Such a read can carry megabytes * (a forward pass its budget plus the metas, "Load older" up to three budgets), which a flat @@ -618,9 +627,79 @@ export async function remoteLocateClaudeTranscript( return path; } +/** The requests a read script may take: the first per relPath, valid ones only. */ +function validEditSourceRequests(requests: readonly EditSourceRequest[]): Map { + const byRelPath = new Map(); + for (const request of requests) { + if (!request || typeof request.relPath !== 'string') continue; + if (!EDIT_SOURCE_REL_PATH_PATTERN.test(request.relPath)) continue; + if (!isByteCount(request.offset)) continue; + if (request.len !== undefined && !isByteCount(request.len)) continue; + if (byRelPath.has(request.relPath)) continue; + byRelPath.set(request.relPath, request); + } + return byRelPath; +} + +/** One ssh round trip of an edit-history read ({@link planRemoteEditSourceRounds}). */ +export interface RemoteEditSourceRound { + requests: EditSourceRequest[]; + /** It reads the main transcript: the first round trip of a read does. */ + main: boolean; + /** It takes the subagent files after this name ('' = from the first), in byte order. */ + after: string; + /** It takes them up to and including this name ('' = to the last). */ + upTo: string; +} + /** - * Script that reads a remote claude session's main transcript at `mainPath` and its subagent - * transcripts (`/subagents/agent-.jsonl`) in ONE ssh round trip, + * The ssh round trips a read takes. Up to `perRound` subagent requests fit one; more are sorted + * and cut into groups of `perRound`, or the ssh line would hit `E2BIG` and the session would + * stay `unavailable`: the cursors have no cap and the stored conversation is read first, so + * not even a `/clear` would get past it. + * + * ⚠️ The cuts are NAME RANGES, not only groups of requests. The script takes a file that no + * `case` line names for a new one (its tail, its meta), so a plain group would read every + * file of the other groups as new. Each round trip walks only the names between its cuts; the + * ranges cover every possible name exactly once, so a new file lands in one of them and the + * merged result is the one a single script would give (the fuzz test compares it with the + * local reader). + */ +export function planRemoteEditSourceRounds( + requests: readonly EditSourceRequest[], + perRound: number = EDIT_SOURCE_REQUESTS_PER_ROUND_TRIP +): RemoteEditSourceRound[] { + const valid = validEditSourceRequests(requests); + const subagents = [...valid.values()].filter((request) => request.relPath !== 'main'); + if (subagents.length <= perRound) return [{ requests: [...valid.values()], main: true, after: '', upTo: '' }]; + + const main = valid.get('main'); + const fileName = (request: EditSourceRequest): string => request.relPath.slice('subagents/'.length); + subagents.sort((a, b) => (a.relPath < b.relPath ? -1 : 1)); + const rounds: RemoteEditSourceRound[] = []; + for (let i = 0; i < subagents.length; i += perRound) { + const group = subagents.slice(i, i + perRound); + rounds.push({ + requests: i === 0 && main ? [main, ...group] : group, + main: i === 0, + after: i === 0 ? '' : fileName(subagents[i - 1]), + upTo: i + perRound < subagents.length ? fileName(group[group.length - 1]) : '', + }); + } + return rounds; +} + +/** What {@link buildRemoteClaudeEditSourcesCommand} takes beyond the local reader's options. */ +export interface RemoteEditSourceCommandOptions extends EditSourceReadOptions { + /** Which part of a split read this command is; without it, all of it. */ + round?: Pick; + /** Meta bytes this command may send; without it, {@link EDIT_SOURCE_META_BUDGET_BYTES}. */ + metaBudget?: number; +} + +/** + * Script for one round trip of a read of a remote claude session's main transcript at + * `mainPath` and its subagent transcripts (`/subagents/agent-.jsonl`), * with exactly the local reader's rules (`readClaudeEditSources`): * * - a request without `len` reads from `offset` (clamped to the size); @@ -631,7 +710,8 @@ export async function remoteLocateClaudeTranscript( * * `budgetBytes` is shared by all transcript files, main first, then the subagents in byte * order. A file that gets none of it still produces a zero-length record, because the caller - * needs its size. Exit {@link TRANSCRIPT_NOT_FOUND_EXIT} when `mainPath` is not a file. + * needs its size. Exit {@link TRANSCRIPT_NOT_FOUND_EXIT} when `mainPath` is not a file. A read + * with more requests than fit one script is several of these ({@link planRemoteEditSourceRounds}). * * Output: `\0\0`, then per record `\0\0\0\0` * followed by exactly `` raw bytes. Should a file shrink between `wc -c` and the read, @@ -646,30 +726,30 @@ export function buildRemoteClaudeEditSourcesCommand( mainPath: string, requests: readonly EditSourceRequest[], budgetBytes: number, - opts: EditSourceReadOptions = {} + opts: RemoteEditSourceCommandOptions = {} ): string { if (!isRemoteClaudeTranscriptPath(mainPath)) return `exit ${TRANSCRIPT_NOT_FOUND_EXIT}`; const transcriptId = mainPath.slice(mainPath.lastIndexOf('/') + 1, -'.jsonl'.length); - const byRelPath = new Map(); - for (const request of requests) { - if (!request || typeof request.relPath !== 'string') continue; - if (!EDIT_SOURCE_REL_PATH_PATTERN.test(request.relPath)) continue; - if (!isByteCount(request.offset)) continue; - if (request.len !== undefined && !isByteCount(request.len)) continue; - if (byRelPath.has(request.relPath)) continue; - byRelPath.set(request.relPath, { offset: request.offset, len: request.len ?? null }); - } + const byRelPath = validEditSourceRequests(requests); const budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; - const requestArgs = (request: { offset: number; len: number | null }): string => - `r ${request.offset} ${request.len === null ? "''" : String(request.len)}`; + const requestArgs = (request: EditSourceRequest): string => + `r ${request.offset} ${request.len === undefined ? "''" : String(request.len)}`; + const { main: readsMain, after, upTo } = opts.round ?? { main: true, after: '', upTo: '' }; const main = byRelPath.get('main'); - const mainLine = main - ? `emit main "$best" ${requestArgs(main)}` - : opts.onlyRequested - ? '' - : 'emit main "$best" t 0 \'\''; + const mainLine = !readsMain + ? '' + : main + ? `emit main "$best" ${requestArgs(main)}` + : opts.onlyRequested + ? '' + : 'emit main "$best" t 0 \'\''; + // The names come from validated relPaths, and `$0` stays a string: every name starts `agent-`. + const rangeFilter = + after || upTo + ? ` | awk -v after=${shellescape(after)} -v upto=${shellescape(upTo)} '$0 > after && (upto == "" || $0 <= upto)'` + : ''; const subagentCases = [...byRelPath.entries()] .filter(([relPath]) => relPath !== 'main') .map(([relPath, request]) => { @@ -699,7 +779,7 @@ export function buildRemoteClaudeEditSourcesCommand( `[ -f "$best" ] || exit ${TRANSCRIPT_NOT_FOUND_EXIT}`, `printf '\\0%s\\0' ${shellescape(transcriptId)}`, `budget=${budget}`, - `mbudget=${EDIT_SOURCE_META_BUDGET_BYTES}`, + `mbudget=${opts.metaBudget ?? EDIT_SOURCE_META_BUDGET_BYTES}`, // Size via `wc -c <` (POSIX; BSD pads it with spaces, hence the tr). 'fsize() {', ' s=$(wc -c < "$1" 2>/dev/null | tr -d \' \\t\\n\')', @@ -728,11 +808,20 @@ export function buildRemoteClaudeEditSourcesCommand( ' printf \'%s\\0%s\\0%s\\0%s\\0\' "$1" "$s" "$start" "$n"', ' [ "$n" -eq 0 ] || bytes "$2" "$start" "$n"', '}', + // The valid names, one per line in byte order (the glob's order under `C`). Nothing but + // `[A-Za-z0-9_.-]` is left, so the unquoted `$(…)` below splits on them and globs nothing. + // A function, because bash 3.2 (macOS `sh`) cannot parse a `case` inside `$(…)`. + 'subagent_names() {', + ' for f in "${best%.jsonl}"/subagents/agent-*.jsonl; do', + ' b=${f##*/}', + ' case $b in agent-.jsonl|agent-*[!A-Za-z0-9_-]*.jsonl) continue ;; esac', + ' printf \'%s\\n\' "$b"', + ' done', + '}', mainLine, - 'for f in "${best%.jsonl}"/subagents/agent-*.jsonl; do', + `for b in $(subagent_names${rangeFilter}); do`, + ' f=${best%.jsonl}/subagents/$b', ' [ -f "$f" ] || continue', - ' b=${f##*/}', - ' case $b in agent-.jsonl|agent-*[!A-Za-z0-9_-]*.jsonl) continue ;; esac', ' case $b in', ...subagentCases, ...newSubagent, @@ -795,9 +884,53 @@ export function parseRemoteClaudeEditSources(stdout: Buffer): EditSources | null } /** - * Read a remote claude session's transcripts (see {@link buildRemoteClaudeEditSourcesCommand}), - * the remote twin of `readClaudeEditSources`. `null` when `mainPath` does not exist; an - * unreachable host, an ssh failure or broken output is a `RemoteFileAccessError`. Never wakes. + * Runs one read command on the host: its stdout, at most `maxBytes`, or null for "no such + * transcript" ({@link TRANSCRIPT_NOT_FOUND_EXIT}). + */ +export type RemoteEditSourceRunner = (command: string, maxBytes: number) => Promise; + +/** + * Run the round trips of a read ({@link planRemoteEditSourceRounds}) one after the other and + * merge them into what one read returns. Budget and meta budget carry over: a later round trip + * gets what the earlier ones left, and still lists its files, because a requested file missing + * from the result counts as deleted and loses its cursor. So a read is all or nothing: a failed + * round trip fails it, and nothing of it reaches the store. + */ +export async function readRemoteEditSourceRounds( + mainPath: string, + requests: readonly EditSourceRequest[], + budgetBytes: number, + opts: EditSourceReadOptions, + run: RemoteEditSourceRunner, + perRound?: number +): Promise { + let budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; + let metaBudget = EDIT_SOURCE_META_BUDGET_BYTES; + const merged: EditSources = { transcriptId: '', files: [], metas: {} }; + for (const round of planRemoteEditSourceRounds(requests, perRound)) { + const stdout = await run( + buildRemoteClaudeEditSourcesCommand(mainPath, round.requests, budget, { ...opts, round, metaBudget }), + budget + metaBudget + EDIT_SOURCE_HEADER_SLACK_BYTES + ); + if (!stdout) return null; + const parsed = parseRemoteClaudeEditSources(stdout); + if (!parsed || !mainPath.endsWith(`/${parsed.transcriptId}.jsonl`)) { + throw new RemoteFileAccessError('unexpected output from the remote edit-history read'); + } + merged.transcriptId = parsed.transcriptId; + merged.files.push(...parsed.files); + Object.assign(merged.metas, parsed.metas); + for (const file of parsed.files) budget -= file.chunk.length; + for (const text of Object.values(parsed.metas)) metaBudget = Math.max(0, metaBudget - Buffer.byteLength(text)); + } + return merged; +} + +/** + * Read a remote claude session's transcripts (see {@link readRemoteEditSourceRounds}), the + * remote twin of `readClaudeEditSources`. `null` when `mainPath` does not exist; an unreachable + * host, an ssh failure or broken output is a `RemoteFileAccessError`. Each round trip takes its + * own slot of the ssh limiter. Never wakes. */ export async function remoteReadClaudeEditSources( remote: SessionRemote, @@ -807,33 +940,23 @@ export async function remoteReadClaudeEditSources( opts: EditSourceReadOptions = {} ): Promise { assertNotUnderTest(); - const budget = Number.isFinite(budgetBytes) ? Math.max(0, Math.floor(budgetBytes)) : 0; - const command = buildRemoteEditHistoryCommand( - remote, - buildRemoteClaudeEditSourcesCommand(mainPath, requests, budget, opts) - ); - const maxBytes = budget + EDIT_SOURCE_META_BUDGET_BYTES + EDIT_SOURCE_HEADER_SLACK_BYTES; - let stdout: Buffer; - try { - const result = await runWithRemoteSshLimit(() => - execAsync(command, { - timeout: remoteEditSourcesTimeoutMs(maxBytes), - maxBuffer: maxBytes, - encoding: 'buffer', - }) - ); - stdout = Buffer.isBuffer(result.stdout) ? result.stdout : Buffer.from(result.stdout); - } catch (err) { - if ((err as { code?: unknown }).code === TRANSCRIPT_NOT_FOUND_EXIT) return null; - throw new RemoteFileAccessError( - `remote host ${remote.label || remote.host} unreachable: ${describeExecError(err)}` - ); - } - const parsed = parseRemoteClaudeEditSources(stdout); - if (!parsed || !mainPath.endsWith(`/${parsed.transcriptId}.jsonl`)) { - throw new RemoteFileAccessError('unexpected output from the remote edit-history read'); - } - return parsed; + return readRemoteEditSourceRounds(mainPath, requests, budgetBytes, opts, async (command, maxBytes) => { + try { + const result = await runWithRemoteSshLimit(() => + execAsync(buildRemoteEditHistoryCommand(remote, command), { + timeout: remoteEditSourcesTimeoutMs(maxBytes), + maxBuffer: maxBytes, + encoding: 'buffer', + }) + ); + return Buffer.isBuffer(result.stdout) ? result.stdout : Buffer.from(result.stdout); + } catch (err) { + if ((err as { code?: unknown }).code === TRANSCRIPT_NOT_FOUND_EXIT) return null; + throw new RemoteFileAccessError( + `remote host ${remote.label || remote.host} unreachable: ${describeExecError(err)}` + ); + } + }); } /** diff --git a/src/web/edit-history-sync.ts b/src/web/edit-history-sync.ts index 1852734a..bd3377c7 100644 --- a/src/web/edit-history-sync.ts +++ b/src/web/edit-history-sync.ts @@ -33,8 +33,9 @@ * * WHICH SESSIONS. Claude sessions, local and remote. Docker and non-claude sessions answer * `unsupported` and are never read. The deps get the session and read a remote one over ssh - * (`remote-files.ts`): one round trip to find its transcript, one per read. A remote read never - * wakes its host; a sleeping one is a connect timeout (`unavailable`, stored entries kept). + * (`remote-files.ts`): one round trip to find its transcript, one per read (more for a + * conversation with hundreds of subagents). A remote read never wakes its host; a sleeping one + * is a connect timeout (`unavailable`, stored entries kept). * * IMAGES. A Bash candidate counts only when the file's mtime lies in the call's window * (start − 2 s … end + 2 s), so `ls *.png` lists nothing. Transcript times and mtimes come diff --git a/test/remote-edit-sources.test.ts b/test/remote-edit-sources.test.ts index 66639ddd..f0f37101 100644 --- a/test/remote-edit-sources.test.ts +++ b/test/remote-edit-sources.test.ts @@ -34,11 +34,15 @@ import { isRemoteClaudeTranscriptPath, parseRemoteClaudeEditSources, parseRemoteClaudeTranscriptLocate, + planRemoteEditSourceRounds, + readRemoteEditSourceRounds, remoteEditSourcesTimeoutMs, remoteLocateClaudeTranscript, remoteReadClaudeEditSources, } from '../src/remote-files.js'; import { readClaudeEditSources, type EditSourceRequest } from '../src/edit-history-local-source.js'; +import { EditHistoryStore } from '../src/edit-history-store.js'; +import { EditHistorySync, type EditHistorySession } from '../src/web/edit-history-sync.js'; import type { SessionRemote } from '../src/types/session.js'; let home: string; @@ -62,6 +66,12 @@ function runSh(command: string, env: Record = {}): { code: numbe } } +/** Runs a read command the way a {@link readRemoteEditSourceRounds} runner does, here on `sh`. */ +async function runRead(command: string): Promise { + const { code, stdout } = runSh(command); + return code === 0 ? stdout : null; +} + /** One edit-source record in the script's wire format. */ function editRecord(relPath: string, size: number, start: number, body: Buffer | string): Buffer { const bytes = Buffer.isBuffer(body) ? body : Buffer.from(body); @@ -259,6 +269,59 @@ describe('buildRemoteClaudeEditSourcesCommand', () => { }); }); +describe('planRemoteEditSourceRounds', () => { + const agents = (...ids: string[]): EditSourceRequest[] => + ids.map((id) => ({ relPath: `subagents/agent-${id}.jsonl`, offset: 1 })); + + it('keeps a read that fits one script in one round trip, with no name range', () => { + const requests = [{ relPath: 'main', offset: 5 }, ...agents('b', 'a')]; + expect(planRemoteEditSourceRounds(requests, 2)).toEqual([{ requests, main: true, after: '', upTo: '' }]); + expect(planRemoteEditSourceRounds([], 2)).toEqual([{ requests: [], main: true, after: '', upTo: '' }]); + }); + + it('cuts more requests by name: ranges that join up, main in the first, every request once', () => { + const main = { relPath: 'main', offset: 5 }; + const rounds = planRemoteEditSourceRounds([...agents('e', 'c'), main, ...agents('a', 'd', 'b')], 2); + expect(rounds.map((r) => [r.main, r.after, r.upTo, r.requests.map((q) => q.relPath)])).toEqual([ + [true, '', 'agent-b.jsonl', ['main', 'subagents/agent-a.jsonl', 'subagents/agent-b.jsonl']], + [false, 'agent-b.jsonl', 'agent-d.jsonl', ['subagents/agent-c.jsonl', 'subagents/agent-d.jsonl']], + [false, 'agent-d.jsonl', '', ['subagents/agent-e.jsonl']], + ]); + }); + + it('counts only valid requests, the first per relPath', () => { + const rounds = planRemoteEditSourceRounds( + [ + ...agents('a'), + { relPath: 'subagents/agent-a.jsonl', offset: 9 }, + { relPath: '../../etc/passwd', offset: 0 }, + { relPath: 'subagents/agent-neg.jsonl', offset: -1 }, + ], + 1 + ); + expect(rounds).toEqual([{ requests: agents('a'), main: true, after: '', upTo: '' }]); + }); + + it('keeps every ssh line under the 128 KiB argv limit, longest names and offsets included', () => { + // 255 bytes is the longest file name a filesystem allows; a request without `len` is the + // longer one, because its `''` costs 13 bytes per quote once shellescaped twice. + const requests: EditSourceRequest[] = Array.from({ length: 5000 }, (_, i) => ({ + relPath: `subagents/agent-${String(i).padStart(6, '0')}${'x'.repeat(237)}.jsonl`, + offset: Number.MAX_SAFE_INTEGER, + })); + const mainPath = `/${'p/'.repeat(2000)}conv-1.jsonl`; + const line = (reqs: EditSourceRequest[], round?: { main: boolean; after: string; upTo: string }) => + Buffer.byteLength( + buildRemoteEditHistoryCommand(REMOTE, buildRemoteClaudeEditSourcesCommand(mainPath, reqs, 1000, { round })) + ); + const ARGV_LIMIT = 128 * 1024; + expect(line(requests)).toBeGreaterThan(ARGV_LIMIT); + const rounds = planRemoteEditSourceRounds(requests); + expect(rounds.length).toBeGreaterThan(1); + for (const round of rounds) expect(line(round.requests, round)).toBeLessThan(ARGV_LIMIT); + }); +}); + describe('parseRemoteClaudeEditSources', () => { it('splits several records byte-exactly and keeps a UTF-8 sequence cut at a chunk edge intact as bytes', () => { const euro = Buffer.from('a€', 'utf8'); // 61 e2 82 ac @@ -416,6 +479,8 @@ describe('the transcript read on a real shell', () => { const local = await readClaudeEditSources(path, requests, budget, { onlyRequested }); const { code, stdout } = runSh(buildRemoteClaudeEditSourcesCommand(path, requests, budget, { onlyRequested })); const remote = code === 0 ? parseRemoteClaudeEditSources(stdout) : null; + // The same read cut into several round trips must come out the same. + const split = await readRemoteEditSourceRounds(path, requests, budget, { onlyRequested }, runRead, ri(1, 3)); const shape = (r: typeof local) => r && { transcriptId: r.transcriptId, @@ -424,6 +489,7 @@ describe('the transcript read on a real shell', () => { }; try { expect(shape(remote)).toEqual(shape(local)); + expect(shape(split)).toEqual(shape(local)); } catch { failures.push(`seed ${seed}: ${JSON.stringify({ requests, budget, onlyRequested })}`); } @@ -432,6 +498,207 @@ describe('the transcript read on a real shell', () => { }); }); +describe('a read too large for one ssh line', () => { + // `ssh` is a fake that runs the remote command on `sh`, so the whole ssh line really goes + // through the local `sh -c` argv: what must not hit E2BIG. + let bin: string; + beforeAll(() => { + bin = mkdtempSync(join(tmpdir(), 'codeman-fake-ssh-read-')); + writeFileSync(join(bin, 'ssh'), '#!/bin/sh\nfor arg; do :; done\nexec sh -c "$arg"\n'); + chmodSync(join(bin, 'ssh'), 0o755); + }); + afterAll(() => rmSync(bin, { recursive: true, force: true })); + + /** A runner that goes through the fake ssh, counting its round trips. */ + function overSsh(): { run: (command: string, maxBytes: number) => Promise; trips: () => number } { + let trips = 0; + return { + trips: () => trips, + run: async (command, maxBytes) => { + trips++; + try { + return execFileSync('sh', ['-c', buildRemoteEditHistoryCommand(REMOTE, command)], { + env: { ...process.env, HOME: home, PATH: `${bin}:${process.env.PATH}` }, + cwd: home, + maxBuffer: maxBytes, + }); + } catch (err) { + if ((err as { status?: number }).status === 3) return null; + throw err; + } + }, + }; + } + + const shape = (r: Awaited>) => + r && { + transcriptId: r.transcriptId, + metas: r.metas, + files: r.files.map((f) => [f.relPath, f.size, f.startOffset, f.chunk.toString('hex')]), + }; + + describe('as a read', () => { + const project = () => join(home, '.claude', 'projects', 'many'); + const main = () => join(project(), 'conv-many.jsonl'); + const sub = () => join(project(), 'conv-many', 'subagents'); + const id = (i: number) => `a${String(i).padStart(4, '0')}`; + /** A cursor for each of 1500 subagents; only every 100th still has a file. */ + const requests: EditSourceRequest[] = [ + { relPath: 'main', offset: 2 }, + ...Array.from({ length: 1500 }, (_, i) => ({ relPath: `subagents/agent-${id(i)}.jsonl`, offset: 1 })), + ]; + + beforeAll(() => { + mkdirSync(sub(), { recursive: true }); + writeFileSync(main(), 'main transcript\n'); + for (let i = 0; i < 1500; i += 100) writeFileSync(join(sub(), `agent-${id(i)}.jsonl`), `subagent ${i}\n`); + // New files in the first, a middle and the last name range: each read once, with its meta. + for (const name of ['agent-0first', 'agent-a0750new', 'agent-zlast']) { + writeFileSync(join(sub(), `${name}.jsonl`), `${name}\n`); + writeFileSync(join(sub(), `${name}.meta.json`), `{"description":"${name}"}`); + } + }); + + it.each([0, 40, 8 * 1024 * 1024])('reads what the local reader reads, budget %i', async (budget) => { + const ssh = overSsh(); + const read = await readRemoteEditSourceRounds(main(), requests, budget, {}, ssh.run); + expect(shape(read)).toEqual(shape(await readClaudeEditSources(main(), requests, budget))); + expect(ssh.trips()).toBe(5); + }); + + it('is null when the transcript goes away during the read', async () => { + let trips = 0; + const read = readRemoteEditSourceRounds(main(), requests, 100, {}, async (command) => { + return ++trips === 2 ? null : runRead(command); + }); + await expect(read).resolves.toBeNull(); + expect(trips).toBe(2); + }); + + it('fails as a whole on broken output from a later round trip', async () => { + let trips = 0; + const read = readRemoteEditSourceRounds(main(), requests, 100, {}, async (command) => { + return ++trips === 2 ? Buffer.from('\0other-conversation\0') : runRead(command); + }); + await expect(read).rejects.toBeInstanceOf(RemoteFileAccessError); + expect(trips).toBe(2); + }); + + it('gives a later round trip the meta budget the earlier ones left', async () => { + const commands: string[] = []; + await readRemoteEditSourceRounds( + main(), + requests, + 1000, + {}, + async (command) => { + commands.push(command); + return runRead(command); + }, + 900 + ); + const budgets = commands.map((c) => Number(/mbudget=(\d+)/.exec(c)?.[1])); + expect(budgets).toHaveLength(2); + // `agent-0first` sorts before every request, so its meta went with the first round trip. + expect(budgets[0] - budgets[1]).toBe( + Buffer.byteLength('{"description":"agent-0first"}') + Buffer.byteLength('{"description":"agent-a0750new"}') + ); + }); + }); + + describe('through the sync', () => { + const project = () => join(home, '.claude', 'projects', 'long-history'); + const sub = () => join(project(), 'conv-old', 'subagents'); + const edit = (id: string, path: string) => + JSON.stringify({ + type: 'user', + timestamp: '2026-10-15T20:00:00.000Z', + cwd: '/w', + message: { role: 'user', content: [{ tool_use_id: id, type: 'tool_result', content: 'ok' }] }, + toolUseResult: { + filePath: path, + oldString: 'a', + newString: 'b', + structuredPatch: [{ oldStart: 1, oldLines: 1, newStart: 1, newLines: 1, lines: ['-a', '+b'] }], + }, + }) + '\n'; + + it('keeps a conversation with 1200 subagents available, and a /clear gets past them', async () => { + const dir = mkdtempSync(join(tmpdir(), 'codeman-edit-store-')); + try { + mkdirSync(sub(), { recursive: true }); + const oldMain = join(project(), 'conv-old.jsonl'); + const newMain = join(project(), 'conv-new.jsonl'); + writeFileSync(oldMain, '{}\n'); + // Ids as long as claude's own: with these, 1200 requests do not fit one ssh line. + const name = (i: number) => `agent-a${String(i).padStart(16, '0')}`; + for (let i = 0; i < 1200; i++) writeFileSync(join(sub(), `${name(i)}.jsonl`), '{}\n'); + + const ssh = overSsh(); + let path = oldMain; + const sync = new EditHistorySync({ + transcriptPath: async () => path, + readSources: (_session, mainPath, requests, budget, opts) => + readRemoteEditSourceRounds(mainPath, requests, budget, opts ?? {}, ssh.run), + probe: async () => [], + loadStore: (id) => EditHistoryStore.load(id, { dir, debounceMs: 0 }), + deleteStore: async () => {}, + sweepStores: async () => 0, + }); + const session: EditHistorySession = { + id: 's1', + mode: 'claude', + remote: REMOTE, + claudeSessionId: 'conv-old', + }; + // The cursors of a history that was read before, one per file at its end. + const store = await sync.getStore('s1'); + const seed = (source: string) => + store.apply({ transcriptId: 'conv-old', source, startOffset: 0, chunk: Buffer.from('{}\n'), size: 3 }); + seed('main'); + for (let i = 0; i < 1200; i++) seed(`subagents/${name(i)}.jsonl`); + expect(store.knownSources()).toHaveLength(1201); + + // Past the argv limit: one script would not even start. + const all = store.knownSources().map((relPath) => ({ relPath, offset: 3 })); + const line = buildRemoteEditHistoryCommand(REMOTE, buildRemoteClaudeEditSourcesCommand(oldMain, all, 1000)); + expect(() => execFileSync('sh', ['-c', line])).toThrow(/E2BIG/); + + // A known file that grew in the last name range, a new one in the middle. + writeFileSync(join(sub(), `${name(1199)}.jsonl`), `{}\n${edit('late', '/w/late.ts')}`); + const fresh = `${name(600)}new`; + writeFileSync(join(sub(), `${fresh}.jsonl`), edit('fresh', '/w/fresh.ts')); + writeFileSync(join(sub(), `${fresh}.meta.json`), '{"description":"new one","agentType":"Explore"}'); + expect(await sync.sync(session)).toEqual({ state: 'ok' }); + expect(ssh.trips()).toBe(4); + expect( + store + .list() + .map((e) => e.path) + .sort() + ).toEqual(['/w/fresh.ts', '/w/late.ts']); + expect(store.getAgents()[fresh.slice('agent-'.length)]).toMatchObject({ description: 'new one' }); + expect(store.knownSources()).toHaveLength(1202); // no cursor lost + + // /clear: the stored conversation is read to its end first, then the new one. + writeFileSync(newMain, edit('after', '/w/after.ts')); + path = newMain; + expect(await sync.sync(session)).toEqual({ state: 'ok' }); + expect(store.getTranscriptId()).toBe('conv-new'); + expect( + store + .list() + .map((e) => e.path) + .sort() + ).toEqual(['/w/after.ts', '/w/fresh.ts', '/w/late.ts']); + await store.remove(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); + }); +}); + describe('a timeout ends ssh itself', () => { // Node signals the local `sh -c`. On dash that shell stays the parent of a plain command, so // only `exec` makes the signal reach ssh. The fake ssh records its pid and then just waits.