diff --git a/.changeset/offline-codex-history.md b/.changeset/offline-codex-history.md index 9ed54d37..f0a2eb0c 100644 --- a/.changeset/offline-codex-history.md +++ b/.changeset/offline-codex-history.md @@ -2,4 +2,4 @@ 'aicodeman': minor --- -Add offline `codeman history read` for saved Codex transcripts by file, native session ID, or conversation family (spawned sub-agents and forks included), with whole-turn topic and UTC date filters. A forked or spawned thread's copy of its parent's history is dropped, so every record belongs to the thread that wrote it. Store attribution has its own budget (`--metadata-bytes`), separate from the transcript budget (`--scan-bytes`), and every bound is raisable (`--max-entries` too), so id lookups keep working on stores of thousands of rollouts. A store file that cannot be attributed, an unreadable directory, or a rollout a running Codex is still appending to no longer fails the lookup; each is reported through `complete: false`. Assistant records carry Codex's own phase and user records none. The Past Sessions scanner and the offline reader now share one Codex rollout format module, including the injected-context rule. +Add offline `codeman history read` for saved Codex transcripts by file, native session ID, or conversation family (spawned sub-agents and forks included), with whole-turn topic and UTC date filters. A forked or spawned thread's copy of its parent's history is dropped, so every record belongs to the thread that wrote it. Store attribution has its own budget (`--metadata-bytes`), separate from the transcript budget (`--scan-bytes`), and every bound is raisable (`--max-entries` too), so id lookups keep working on stores of thousands of rollouts. A store file that cannot be attributed, an unreadable directory, or a rollout a running Codex is still appending to no longer fails the lookup; each is reported through `complete: false`, as is a Codex rollout compressed to `.jsonl.zst` (counted in `compressedFiles` and named when a lookup misses). A parent file read to prove where a forked copy ends is charged to `--scan-bytes` once, and a parent that cannot be read leaves that boundary unproven instead of failing the child. Assistant records carry Codex's own phase and user records none. The Past Sessions scanner and the offline reader now share one Codex rollout format module, including the injected-context rule. diff --git a/README.md b/README.md index 3fc1a495..90412d5c 100644 --- a/README.md +++ b/README.md @@ -1006,7 +1006,7 @@ codeman history read --root-session-id --topic auth # a conversa codeman history read --file /path/to/rollout.jsonl --format text ``` -Filters keep whole user/assistant turns (`--topic` is a literal, case-insensitive match; `--after`/`--before` are inclusive UTC dates). A forked or spawned thread's copy of its parent's history is dropped, so every record belongs to the thread that wrote it. Output is JSON by default and carries `complete: false` whenever the evidence is partial (a file that could not be attributed, a record a running Codex is still writing, an unknown ancestor). Every bound fails closed and is raisable: `--metadata-bytes` (attributing the store, default 1 GiB), `--max-entries` (default 65536), `--scan-bytes` (the selected transcripts, default 16 MiB) and `--max-bytes` (output, default 8192). The full reference is [`skills/codeman/reference/history.md`](skills/codeman/reference/history.md). +Filters keep whole user/assistant turns (`--topic` is a literal, case-insensitive match; `--after`/`--before` are inclusive UTC dates). A forked or spawned thread's copy of its parent's history is dropped, so every record belongs to the thread that wrote it. Output is JSON by default and carries `complete: false` whenever the evidence is partial (a file that could not be attributed, a compressed `.jsonl.zst` rollout, a record a running Codex is still writing, an unknown ancestor). Every bound fails closed and is raisable: `--metadata-bytes` (attributing the store, default 1 GiB), `--max-entries` (default 65536), `--scan-bytes` (the selected transcripts, default 16 MiB) and `--max-bytes` (output, default 8192). The full reference is [`skills/codeman/reference/history.md`](skills/codeman/reference/history.md). ### MCP server (`codeman mcp`): drive Codeman from any MCP client diff --git a/docs/wiki/Driving-Codeman-From-An-Agent.md b/docs/wiki/Driving-Codeman-From-An-Agent.md index 6e7ce85c..1a594682 100644 --- a/docs/wiki/Driving-Codeman-From-An-Agent.md +++ b/docs/wiki/Driving-Codeman-From-An-Agent.md @@ -87,7 +87,8 @@ codeman history read --file /path/to/rollout.jsonl --format text - **Forks and sub-agents**: a child's copy of its parent's history is dropped, so every record belongs to the thread that wrote it. - **`complete: false`** in the JSON means the evidence is partial: a store file that could - not be attributed, a record a running Codex is still writing, an ancestor not found. + not be attributed, a compressed `.jsonl.zst` rollout (not read), a record a running + Codex is still writing, an ancestor not found. - **Budgets** fail closed and are raisable: `--metadata-bytes` (attributing the store, default 1 GiB), `--max-entries` (65536), `--scan-bytes` (the selected transcripts, 16 MiB) and `--max-bytes` (output, 8192). `--json` and `--format text` together are an diff --git a/plugins/codeman/skills/codeman/reference/history.md b/plugins/codeman/skills/codeman/reference/history.md index bcdfc92c..ae087fb6 100644 --- a/plugins/codeman/skills/codeman/reference/history.md +++ b/plugins/codeman/skills/codeman/reference/history.md @@ -43,27 +43,34 @@ plus scan counters and `complete`. `phase` is Codex's own (`commentary` or recorded. A root ID is `null` when ancestry cannot be proven. Report that uncertainty; never relabel the queried child as its own root. `complete: false` reports malformed or partial records (a running Codex may be -mid-write), skipped links, unattributed files, unreadable directories, missing -ancestors, or an inherited copy whose end could not be found or proven. It does not -prove the store includes every conversation ever created, and a filtered result is -only evidence for its selected scope. +mid-write), skipped links, unattributed files, compressed rollouts, unreadable +directories, missing ancestors, or an inherited copy whose end could not be found +or proven (including when the parent's file is present but cannot be read). It +does not prove the store includes every conversation ever created, and a filtered +result is only evidence for its selected scope. Reads require regular UTF8 files and reject final-component symlinks, conflicting metadata, ambiguous IDs, cyclic ancestry and invalid selectors. A store file that cannot be attributed (invalid UTF8 in its head, no metadata in its first 128 KiB, changing while its head is read) is counted in `unattributedFiles` instead of -failing the lookup. A selected rollout that a running Codex is appending to is read -as a snapshot up to its last complete line; one that was replaced or truncated -fails. Parent directories of an explicitly supplied path must be trusted; this is +failing the lookup. Codex's optional rollout compression rewrites cold rollouts to +`*.jsonl.zst`; those are not read. Each is counted in `compressedFiles` (one whose +plain `.jsonl` sibling exists is skipped, as Codex skips it), and a lookup that +misses names them. A compressed file passed to `--file` is refused; read a +decompressed copy instead (`zstd -dc > copy.jsonl`). A selected rollout +that a running Codex is appending to is read as a snapshot up to its last complete +line; one that was replaced or truncated fails. Parent directories of an explicitly supplied path must be trusted; this is not a filesystem sandbox. -Budgets, each failing with an error rather than a partial answer: +Budgets, each failing with an error rather than a partial answer. Every flag is +validated before the first file is read, and only plain decimal integers are +accepted: | Flag | Bounds | Default | Maximum | | ---- | ------ | ------- | ------- | | `--metadata-bytes` | head reads attributing every rollout in the store | 1 GiB | 16 GiB | | `--max-entries` | directory entries visited | 65536 | 1048576 | -| `--scan-bytes` | the selected transcripts | 16 MiB | 256 MiB | +| `--scan-bytes` | the selected transcripts, plus any parent file read to prove where a copy ends (each file charged once) | 16 MiB | 256 MiB | | `--max-bytes` | output, including the final newline | 8192 | 1048576 | Each rollout's `session_meta` carries Codex's base instructions (about 20 KiB), so diff --git a/skills/codeman/reference/history.md b/skills/codeman/reference/history.md index bcdfc92c..ae087fb6 100644 --- a/skills/codeman/reference/history.md +++ b/skills/codeman/reference/history.md @@ -43,27 +43,34 @@ plus scan counters and `complete`. `phase` is Codex's own (`commentary` or recorded. A root ID is `null` when ancestry cannot be proven. Report that uncertainty; never relabel the queried child as its own root. `complete: false` reports malformed or partial records (a running Codex may be -mid-write), skipped links, unattributed files, unreadable directories, missing -ancestors, or an inherited copy whose end could not be found or proven. It does not -prove the store includes every conversation ever created, and a filtered result is -only evidence for its selected scope. +mid-write), skipped links, unattributed files, compressed rollouts, unreadable +directories, missing ancestors, or an inherited copy whose end could not be found +or proven (including when the parent's file is present but cannot be read). It +does not prove the store includes every conversation ever created, and a filtered +result is only evidence for its selected scope. Reads require regular UTF8 files and reject final-component symlinks, conflicting metadata, ambiguous IDs, cyclic ancestry and invalid selectors. A store file that cannot be attributed (invalid UTF8 in its head, no metadata in its first 128 KiB, changing while its head is read) is counted in `unattributedFiles` instead of -failing the lookup. A selected rollout that a running Codex is appending to is read -as a snapshot up to its last complete line; one that was replaced or truncated -fails. Parent directories of an explicitly supplied path must be trusted; this is +failing the lookup. Codex's optional rollout compression rewrites cold rollouts to +`*.jsonl.zst`; those are not read. Each is counted in `compressedFiles` (one whose +plain `.jsonl` sibling exists is skipped, as Codex skips it), and a lookup that +misses names them. A compressed file passed to `--file` is refused; read a +decompressed copy instead (`zstd -dc > copy.jsonl`). A selected rollout +that a running Codex is appending to is read as a snapshot up to its last complete +line; one that was replaced or truncated fails. Parent directories of an explicitly supplied path must be trusted; this is not a filesystem sandbox. -Budgets, each failing with an error rather than a partial answer: +Budgets, each failing with an error rather than a partial answer. Every flag is +validated before the first file is read, and only plain decimal integers are +accepted: | Flag | Bounds | Default | Maximum | | ---- | ------ | ------- | ------- | | `--metadata-bytes` | head reads attributing every rollout in the store | 1 GiB | 16 GiB | | `--max-entries` | directory entries visited | 65536 | 1048576 | -| `--scan-bytes` | the selected transcripts | 16 MiB | 256 MiB | +| `--scan-bytes` | the selected transcripts, plus any parent file read to prove where a copy ends (each file charged once) | 16 MiB | 256 MiB | | `--max-bytes` | output, including the final newline | 8192 | 1048576 | Each rollout's `session_meta` carries Codex's base instructions (about 20 KiB), so diff --git a/src/cli-history.ts b/src/cli-history.ts index 11be6088..238224a2 100644 --- a/src/cli-history.ts +++ b/src/cli-history.ts @@ -27,8 +27,18 @@ import { HISTORY_MAX_OUTPUT_BYTES, HISTORY_MAX_SCAN_BYTES, readLocalHistory, + validateOutputBytes, } from './local-history.js'; +/** + * A budget flag's value as a number. Only plain decimal digits count, so `0x10`, + * `1e3` and ` 5 ` are refused rather than read as 16, 1000 and 5; anything else + * becomes NaN, which the reader's range check rejects with the flag's name. + */ +function budgetFlag(value: string | boolean | undefined): number { + return typeof value === 'string' && /^\d+$/.test(value) ? Number(value) : NaN; +} + export function registerHistoryCommands(program: Command): Command { const history = program .command('history') @@ -74,6 +84,10 @@ export function registerHistoryCommands(program: Command): Command { if (options.json && options.format === 'text' && command.getOptionValueSource('format') === 'cli') throw new Error('--json conflicts with --format text; pass one of them'); const format = options.format; + // Every flag is checked before the first file is read: --max-bytes would otherwise + // only be judged after the whole store had been scanned. + const maxBytes = budgetFlag(options.maxBytes); + validateOutputBytes(maxBytes); const result = await readLocalHistory({ file: options.file as string | undefined, sessionId: options.sessionId as string | undefined, @@ -82,11 +96,11 @@ export function registerHistoryCommands(program: Command): Command { topic: options.topic as string | undefined, after: options.after as string | undefined, before: options.before as string | undefined, - scanBytes: Number(options.scanBytes), - metadataBytes: Number(options.metadataBytes), - maxEntries: Number(options.maxEntries), + scanBytes: budgetFlag(options.scanBytes), + metadataBytes: budgetFlag(options.metadataBytes), + maxEntries: budgetFlag(options.maxEntries), }); - program.configureOutput().writeOut?.(`${formatLocalHistory(result, format, Number(options.maxBytes))}\n`); + program.configureOutput().writeOut?.(`${formatLocalHistory(result, format, maxBytes)}\n`); } catch (error) { program.error(error instanceof Error ? error.message : String(error), { exitCode: 1, code: 'history.read' }); } diff --git a/src/local-history.ts b/src/local-history.ts index e6162c0d..4ff244bf 100644 --- a/src/local-history.ts +++ b/src/local-history.ts @@ -46,16 +46,28 @@ * the copy. A legacy copy keeps the parent's own `thread_settings_applied` * events too, so when more than one candidate follows, the boundary is the * last one whose preceding messages are still a prefix of the parent's own - * messages, read from the parent's file. Without the parent's file that - * cannot be proven: the cut then falls at the LAST candidate, so parent text - * is never attributed to the child, and the result says so - * (`unverifiedPrefixes`, `complete: false`). + * messages, read from the parent's file (once: a family read that selects + * the parent reuses that same read, so no file is charged to `scanBytes` + * twice). Without a readable parent file that cannot be proven: the cut then + * falls at the LAST candidate, so parent text is never attributed to the + * child, and the result says so (`unverifiedPrefixes`, `complete: false`). + * Any failure of the parent's read other than a budget leads here, rather + * than failing the child whose own file is intact. * * A later `session_meta` must be the file's own id or a declared ancestor * (`parent_thread_id`, `forked_from_id`, or the shared root `session_id`, and * transitively the ancestors those declare); any other id still refuses the * whole file rather than attribute one conversation's answer to another. * + * ## Compressed rollouts + * + * Codex's `local_thread_store_compression` feature rewrites cold rollouts to + * `*.jsonl.zst` in place. This reader does not decompress them: a store scan + * counts each one in `compressedFiles` (so `complete` is false and a not-found + * error names them), skipping one whose plain `.jsonl` sibling exists, as + * Codex's own discovery does. A Zstandard file given to `file` is refused by + * its magic number, whatever its name. + * * ## Live rollouts * * A running Codex appends to its rollout. Reads take a snapshot: the file's @@ -73,7 +85,7 @@ import { constants } from 'node:fs'; import { isUtf8 } from 'node:buffer'; import { lstat, open, opendir, stat, type FileHandle } from 'node:fs/promises'; -import { join, resolve } from 'node:path'; +import { basename, join, resolve } from 'node:path'; import { codexMessageFromLine, codexSessionsRoot, isCodexInjectedContext } from './codex-rollout-format.js'; @@ -93,6 +105,13 @@ export const HISTORY_MAX_OUTPUT_BYTES = 1 * MIB; const METADATA_HEAD_BYTES = 128 * KIB; /** Read the head in steps, so a typical rollout costs one step past its session_meta line. */ const METADATA_CHUNK_BYTES = 32 * KIB; +/** + * Codex's compressed rollout suffix (codex-rs `rollout/src/compression.rs`: + * `is_compressed_rollout_path` matches a name ending in `.jsonl.zst`). + */ +const COMPRESSED_ROLLOUT_SUFFIX = '.jsonl.zst'; +/** The Zstandard frame magic number, little-endian on disk. */ +const ZSTD_MAGIC = Buffer.from([0x28, 0xb5, 0x2f, 0xfd]); /** Codex writes `sessions/YYYY/MM/DD`; this leaves five levels of headroom. */ const MAX_DIRECTORY_DEPTH = 8; @@ -152,6 +171,11 @@ export interface LocalHistoryResult { partialRecords: number; skippedSymlinks: number; unattributedFiles: number; + /** + * Zstandard-compressed rollouts (`*.jsonl.zst`, Codex's `local_thread_store_compression`) + * this reader cannot open. Their threads are missing from the result. + */ + compressedFiles: number; unreadableDirectories: number; /** Inherited fork or sub-agent prefixes dropped (informational). */ inheritedPrefixes: number; @@ -166,6 +190,8 @@ interface Budget { metadataRemaining: number; entriesRemaining: number; scan: LocalHistoryResult['scan']; + /** Paths behind `scan.compressedFiles`, for the not-found error. */ + compressedPaths: string[]; } /** A budget ran out: always fatal, never counted. */ @@ -318,6 +344,10 @@ async function readMetadata(path: string, budget: Budget): Promise= 0) { @@ -396,6 +426,13 @@ async function findSources(root: string, budget: Budget): Promise { } if (source) sources.push(source); else budget.scan.unattributedFiles++; + } else if (entry.isFile() && entry.name.endsWith(COMPRESSED_ROLLOUT_SUFFIX)) { + // Codex reads the plain sibling instead when both exist (`should_skip_compressed_sibling`), + // and that sibling is visited (or counted) on its own. + const plain = await lstat(path.slice(0, -'.zst'.length)).catch(() => undefined); + if (plain) continue; + budget.scan.compressedFiles++; + budget.compressedPaths.push(path); } } }; @@ -457,7 +494,10 @@ function isPrefixOf(prefix: Message[], whole: Message[]): boolean { } interface ReadContext { - /** The ancestor's own messages, uncut, or undefined when its file is not in the store. */ + /** + * The ancestor's own messages, uncut, or undefined when its file is not in the + * store or could not be read for any reason other than a budget. + */ ancestorMessages(id: string): Promise; } type Piece = Message | { boundary: true }; @@ -466,8 +506,16 @@ interface LegacyPrefix { /** Messages and `thread_settings_applied` boundaries after the inherited session_meta. */ pieces: Piece[]; } +/** One read of one rollout: every view later needed of it, so no file is read (or charged) twice. */ +interface ParsedSource { + /** Every message in file order, inherited copies included: what a child's copy of this thread holds. */ + raw: Message[]; + /** Messages before any legacy inherited prefix, paginated inherited records already dropped. */ + own: Message[]; + legacy?: LegacyPrefix; +} -async function readMessages(source: Source, budget: Budget, context: ReadContext, raw = false): Promise { +async function parseSource(source: Source, budget: Budget): Promise { const handle = await regularFile(source.path); let buffer: Buffer; try { @@ -487,6 +535,7 @@ async function readMessages(source: Source, budget: Budget, context: ReadContext if (!isUtf8(complete)) throw new Error('History source contains invalid UTF8'); const ancestors = new Set([source.parentId, source.forkedFromId, source.rootId].filter((id): id is string => !!id)); + const raw: Message[] = []; const own: Message[] = []; let legacy: LegacyPrefix | undefined; let lastOrdinal = -1; @@ -515,27 +564,34 @@ async function readMessages(source: Source, budget: Budget, context: ReadContext throw new Error('History source contains conflicting native metadata; attribution refused'); // An ancestor that was itself a fork carries its own ancestors' metadata in the copy. for (const id of [declared.parentId, declared.forkedFromId, declared.rootId]) if (id) ancestors.add(id); - if (raw || inherited || legacy) continue; + if (inherited || legacy) continue; budget.scan.inheritedPrefixes++; legacy = { ancestorId: declared.sessionId, pieces: [] }; continue; } - if (inherited && !raw) continue; if (legacy && entry.type === 'event_msg' && object(entry.payload).type === 'thread_settings_applied') { - legacy.pieces.push({ boundary: true }); + if (!inherited) legacy.pieces.push({ boundary: true }); continue; } const item = message(parsed); if (!item) continue; + raw.push(item); + if (inherited) continue; if (legacy) legacy.pieces.push(item); else own.push(item); } - if (!raw && source.historyStart !== undefined && source.historyStart > 0) { + if (source.historyStart !== undefined && source.historyStart > 0) { budget.scan.inheritedPrefixes++; // Initialization died while copying the prefix: Codex refuses to resume such a file. if (lastOrdinal < source.historyStart - 1) budget.scan.incompletePrefixes++; } - if (legacy) own.push(...(await afterLegacyPrefix(legacy, budget, context))); + return { raw: collapse(raw), own, legacy }; +} + +/** The thread's own messages: its inherited prefix dropped, proven against the parent where one is needed. */ +async function readMessages(parsed: ParsedSource, budget: Budget, context: ReadContext): Promise { + const own = [...parsed.own]; + if (parsed.legacy) own.push(...(await afterLegacyPrefix(parsed.legacy, budget, context))); return collapse(own); } @@ -603,6 +659,32 @@ function nativeRoot(source: Source, sources: Source[]): string | null { } } +/** + * The not-found error, naming what the store held but could not attribute. + * Codex names a rollout `rollout--.jsonl[.zst]`, so a + * compressed file whose name carries the wanted id is named outright; the name + * is only a hint here, never identity. + */ +function missingSessionMessage(wanted: string, budget: Budget): string { + const { scan } = budget; + const notes: string[] = []; + if (scan.compressedFiles) { + // `rollout-` (8) + `YYYY-MM-DDTHH-MM-SS` (19) + `-` (1), then the thread id. + const named = budget.compressedPaths.filter((path) => { + const name = basename(path).toLowerCase(); + return name.startsWith('rollout-') && name.slice(28).startsWith(wanted); + }); + const shown = named.slice(0, 3).join(', ') + (named.length > 3 ? ` and ${named.length - 3} more` : ''); + notes.push( + named.length + ? `it may be in compressed rollout ${shown}, which this reader cannot open; read a decompressed copy with --file (zstd -dc > copy.jsonl)` + : `${scan.compressedFiles} compressed .jsonl.zst rollout(s) could not be read` + ); + } + if (scan.unattributedFiles) notes.push(`${scan.unattributedFiles} file(s) could not be attributed`); + return `Native session missing from history store${notes.length ? ` (${notes.join('; ')})` : ''}`; +} + export async function readLocalHistory(options: LocalHistoryOptions): Promise { validate(options); const scan: LocalHistoryResult['scan'] = { @@ -613,6 +695,7 @@ export async function readLocalHistory(options: LocalHistoryOptions): Promise source.sessionId.startsWith(wanted)); - if (!matches.length) - throw new Error( - scan.unattributedFiles - ? `Native session missing from history store (${scan.unattributedFiles} file(s) could not be attributed)` - : 'Native session missing from history store' - ); + if (!matches.length) throw new Error(missingSessionMessage(wanted, budget)); if (matches.length !== 1) throw new Error('Native session id is ambiguous'); const target = matches[0]; attributionSources = all; @@ -676,23 +755,34 @@ export async function readLocalHistory(options: LocalHistoryOptions): Promise a.timestamp.localeCompare(b.timestamp) || a.sessionId.localeCompare(b.sessionId)); const roots = new Map(sources.map((source) => [source.sessionId, nativeRoot(source, attributionSources)])); - const ancestorCache = new Map>(); + // Each selected file is read once, whether the main loop or a child's boundary proof asks first. + const parsedCache = new Map>(); + const parse = (source: Source) => { + let parsed = parsedCache.get(source); + if (!parsed) { + parsed = parseSource(source, budget); + parsedCache.set(source, parsed); + } + return parsed; + }; const context: ReadContext = { - // A raw read never opens a legacy prefix, so this never recurses. - ancestorMessages(id) { - const cached = ancestorCache.get(id); - if (cached) return cached; + // Raw messages need no boundary proof of their own, so this never recurses. + async ancestorMessages(id) { const candidates = attributionSources.filter((source) => source.sessionId === id); - if (candidates.length !== 1) return Promise.resolve(undefined); - const read = readMessages(candidates[0], budget, context, true); - ancestorCache.set(id, read); - return read; + if (candidates.length !== 1) return undefined; + try { + return (await parse(candidates[0])).raw; + } catch (error) { + // A budget always fails the read; any other failure only leaves the boundary unproven. + if (error instanceof HistoryBudgetError) throw error; + return undefined; + } }, }; const records: HistoryRecord[] = []; for (const source of sources) { records.push( - ...selectedTurns(await readMessages(source, budget, context), options).map((item) => ({ + ...selectedTurns(await readMessages(await parse(source), budget, context), options).map((item) => ({ rootSessionId: roots.get(source.sessionId) ?? null, sessionId: source.sessionId, role: item.role, @@ -710,6 +800,7 @@ export async function readLocalHistory(options: LocalHistoryOptions): Promise HISTORY_MAX_OUTPUT_BYTES) - throw new Error(`Invalid --max-bytes (1-${HISTORY_MAX_OUTPUT_BYTES})`); + validateOutputBytes(maxBytes); const body = format === 'json' ? JSON.stringify(result) diff --git a/test/local-history-live.test.ts b/test/local-history-live.test.ts index aa0348a6..850afdab 100644 --- a/test/local-history-live.test.ts +++ b/test/local-history-live.test.ts @@ -30,7 +30,8 @@ vi.mock('node:fs/promises', async (importOriginal) => { }); const history = await import('../src/local-history.js'); -const { assistantTurn, realMeta, userTurn, writeRollout } = await import('./fixtures/codex-rollout.js'); +const { assistantTurn, realMeta, settingsApplied, userTurn, writeRollout } = + await import('./fixtures/codex-rollout.js'); const ROOT = 'aaaaaaaa-1111-4111-8111-111111111111'; const CHILD = 'bbbbbbbb-1111-4111-8111-111111111111'; @@ -126,4 +127,31 @@ describe('rollouts that change while they are read', () => { /changed after attribution/ ); }); + it('leaves a child readable, with an unproven cut, when its parent is replaced after attribution', async () => { + const parentSettings = [settingsApplied(), ...userTurn('parent second question')]; + const parent = await writeRollout(dir, 'rollout-root.jsonl', [ + realMeta({ id: ROOT }), + ...userTurn('parent question'), + ...parentSettings, + ]); + const child = await writeRollout(dir, 'rollout-child.jsonl', [ + realMeta({ id: CHILD, parent: ROOT }), + realMeta({ id: ROOT }), + ...userTurn('parent question'), + ...parentSettings, + settingsApplied(), + ...userTurn('child task'), + ]); + hooks.onOpen = async (opened, count) => { + // The child's transcript read comes before the parent's proof read: replace the parent in between. + if (opened !== child || count !== 2) return; + const replacement = join(dir, 'replacement.tmp'); + await writeFile(replacement, [realMeta({ id: ROOT }), ...userTurn('rewritten')].map(line).join('')); + await rename(replacement, parent); + }; + const result = await history.readLocalHistory({ sessionId: CHILD, sessionsRoot: dir }); + expect(result.records.map((r) => r.message)).toEqual(['child task']); + expect(result.scan.unverifiedPrefixes).toBe(1); + expect(result.complete).toBe(false); + }); }); diff --git a/test/local-history.test.ts b/test/local-history.test.ts index 0a7cf6dc..15989c6c 100644 --- a/test/local-history.test.ts +++ b/test/local-history.test.ts @@ -471,6 +471,120 @@ describe('forked and spawned threads', () => { }); }); +describe('one read per file, and a parent that cannot be read', () => { + const parentSettings = [ + settingsApplied(), + ...userTurn('parent second question'), + ...assistantTurn('parent second answer'), + ]; + const sizes = async () => { + const { stat } = await import('node:fs/promises'); + const parent = (await stat(join(dir, `rollout-${ROOT}.jsonl`))).size; + const child = (await stat(join(dir, `rollout-${CHILD}.jsonl`))).size; + return { parent, child }; + }; + + it('charges the parent of a legacy copy to --scan-bytes once, even when it proves the boundary', async () => { + await legacyFamily(dir, parentSettings, parentSettings); + const { parent, child } = await sizes(); + // A budget of exactly the two files: the family read selects the parent and proves the child's cut with it. + const family = await history.readLocalHistory({ + rootSessionId: ROOT, + sessionsRoot: dir, + scanBytes: parent + child, + }); + expect(family.scan.bytesRead).toBe(parent + child); + expect(family.complete).toBe(true); + expect(messages(family).filter((m) => m.startsWith('b:'))).toEqual([ + 'b:child task', + 'b:child commentary', + 'b:child answer', + ]); + // One byte less fails closed rather than returning less. + await expect( + history.readLocalHistory({ rootSessionId: ROOT, sessionsRoot: dir, scanBytes: parent + child - 1 }) + ).rejects.toThrow(/scan byte budget/); + // A child read pays for the parent it proves against, once. + const alone = await history.readLocalHistory({ sessionId: CHILD, sessionsRoot: dir, scanBytes: parent + child }); + expect(alone.scan.bytesRead).toBe(parent + child); + expect(alone.complete).toBe(true); + }); + + it('counts a parent record once when the parent is both selected and the proof of a child', async () => { + await legacyFamily(dir, parentSettings, parentSettings); + const { appendFile } = await import('node:fs/promises'); + await appendFile(join(dir, `rollout-${ROOT}.jsonl`), '{not json\n'); + const family = await history.readLocalHistory({ rootSessionId: ROOT, sessionsRoot: dir }); + expect(family.scan.malformedRecords).toBe(1); + expect(family.scan.inheritedPrefixes).toBe(1); + }); + + it('falls back to the unproven cut when the parent file cannot be read, instead of failing the child', async () => { + await legacyFamily(dir, parentSettings, parentSettings); + const { appendFile } = await import('node:fs/promises'); + await appendFile(join(dir, `rollout-${ROOT}.jsonl`), Buffer.from([0xff, 0x0a])); + const child = await history.readLocalHistory({ sessionId: CHILD, sessionsRoot: dir }); + expect(messages(child)).toEqual(['b:child task', 'b:child commentary', 'b:child answer']); + expect(child.scan.unverifiedPrefixes).toBe(1); + expect(child.complete).toBe(false); + // The parent's own read still fails: its evidence is what is broken. + await expect(history.readLocalHistory({ sessionId: ROOT, sessionsRoot: dir })).rejects.toThrow(/invalid UTF8/); + }); + + it('still fails when the parent read runs out of budget', async () => { + await legacyFamily(dir, parentSettings, parentSettings); + const { child } = await sizes(); + await expect( + history.readLocalHistory({ sessionId: CHILD, sessionsRoot: dir, scanBytes: child + 1 }) + ).rejects.toThrow(/scan byte budget/); + }); +}); + +describe('compressed rollouts', () => { + const ZSTD = Buffer.from([0x28, 0xb5, 0x2f, 0xfd, 0x04, 0x00, 0x00, 0x00]); + const compressedName = (id: string) => `rollout-2026-10-24T01-00-00-${id}.jsonl.zst`; + + it('counts a .jsonl.zst rollout, so a family with a compressed member is incomplete', async () => { + await legacyFamily(dir); + await mkdir(join(dir, '2026', '10', '24'), { recursive: true }); + await writeFile(join(dir, '2026', '10', '24', compressedName(GRANDCHILD)), ZSTD); + const family = await history.readLocalHistory({ rootSessionId: ROOT, sessionsRoot: dir }); + expect(family.scan.compressedFiles).toBe(1); + expect(family.scan.unattributedFiles).toBe(0); + expect(family.complete).toBe(false); + }); + + it('names the compressed rollout when the wanted thread is not otherwise found', async () => { + await fixture(ROOT, [row('assistant', 'root')]); + const path = join(dir, compressedName(CHILD)); + await writeFile(path, ZSTD); + const error = await history.readLocalHistory({ sessionId: CHILD.slice(0, 8), sessionsRoot: dir }).catch((e) => e); + expect(error.message).toContain(path); + expect(error.message).toMatch(/compressed rollout/); + // A compressed file whose name does not carry the id is still reported, by count. + const other = await history.readLocalHistory({ sessionId: GRANDCHILD, sessionsRoot: dir }).catch((e) => e); + expect(other.message).toMatch(/1 compressed \.jsonl\.zst rollout\(s\) could not be read/); + }); + + it('skips a compressed rollout whose plain sibling exists, as Codex does', async () => { + const plain = await fixture(ROOT, [row('assistant', 'root')]); + await writeFile(`${plain}.zst`, ZSTD); + const result = await history.readLocalHistory({ sessionId: ROOT, sessionsRoot: dir }); + expect(result.scan.compressedFiles).toBe(0); + expect(result.complete).toBe(true); + }); + + it('refuses a Zstandard file given to --file, and counts one in the store whatever its name', async () => { + const file = join(dir, 'renamed.jsonl'); + await writeFile(file, ZSTD); + await expect(history.readLocalHistory({ file })).rejects.toThrow(/Zstandard-compressed/); + await fixture(ROOT, [row('assistant', 'root')]); + const result = await history.readLocalHistory({ sessionId: ROOT, sessionsRoot: dir }); + expect(result.scan.unattributedFiles).toBe(1); + expect(result.complete).toBe(false); + }); +}); + describe('large and untidy stores', () => { it('finds one id among thousands of rollouts with real-sized session_meta at the default budgets', async () => { const count = 3500; @@ -692,6 +806,31 @@ describe('history CLI flags', () => { expect(await run(['--file', file, '--format', 'text'])).toMatch(/^source=codex-jsonl/); }); + it('validates --max-bytes before any file is read', async () => { + // A missing store would be the first thing a read touches; an invalid --max-bytes must be refused first. + const missing = join(dir, 'no-such-store'); + for (const value of ['0', String(history.HISTORY_MAX_OUTPUT_BYTES + 1), 'lots', '0x10', '1e3', ' 5']) + await expect(run(['--session-id', ROOT, '--sessions-root', missing, '--max-bytes', value])).rejects.toThrow( + /Invalid --max-bytes/ + ); + await expect(run(['--session-id', ROOT, '--sessions-root', missing, '--max-bytes', '65536'])).rejects.toThrow( + /Sessions root not found/ + ); + }); + + it('reads budget flags as plain decimal integers only', async () => { + await fixture(ROOT, [row('assistant', 'answer')]); + await expect(run(['--session-id', ROOT, '--sessions-root', dir, '--scan-bytes', '0x100000'])).rejects.toThrow( + /Invalid --scan-bytes/ + ); + await expect(run(['--session-id', ROOT, '--sessions-root', dir, '--max-entries', '1e3'])).rejects.toThrow( + /Invalid --max-entries/ + ); + expect( + JSON.parse(await run(['--session-id', ROOT, '--sessions-root', dir, '--scan-bytes', '1048576'])).complete + ).toBe(true); + }); + it('passes the store caps through and validates them', async () => { await fixture(ROOT, [row('assistant', 'answer')]); await fixture(CHILD, [row('assistant', 'child')], ROOT);