From 8cdd18fa64a1ab51b9f3230ed56e9b7860d67bd9 Mon Sep 17 00:00:00 2001 From: JD Date: Sat, 24 Oct 2026 18:56:37 -0400 Subject: [PATCH 1/2] test(deepseek): stop pinning Node's one-shot zstd decoder to the first frame The frame-walking test asserted that zlib.zstdDecompressSync returns only the first of 40 concatenated frames. That was true when the walker was written, but nodejs/node#65865 (shipped in 26.10.0 and in 24.22.0 LTS) makes Node decode every frame, so the assertion fails on current Node 24 LTS and 26 while the walker itself is still correct. The assertion tested Node, not Codeman: the walker's own 40-frame checks stay, and the comments now say which Node versions stop at the first frame, since the supported range (>=22) still includes them. --- docs/deepseek-integration.md | 7 ++++--- src/deepseek-transcript.ts | 9 +++++---- test/deepseek-transcript.test.ts | 13 +++++-------- 3 files changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/deepseek-integration.md b/docs/deepseek-integration.md index 874f6fd1..5f1072a7 100644 --- a/docs/deepseek-integration.md +++ b/docs/deepseek-integration.md @@ -235,9 +235,10 @@ worker's first answer reads as an answer. Three things about the file shaped the reader (`src/deepseek-transcript.ts`): - **It is one zstd FRAME per append, not one zstd stream.** `zstd -dc` decodes all - of them, Node's `zlib` zstd decoder stops at the first: a real 56-line - transcript came back as 1 line. The reader walks frame headers itself. On a Node - older than 22.15 (no zstd at all) the mode falls back to the pane, as before. + of them, Node's `zlib` zstd decoder before 24.22 and 26.10 stops at the first: + a real 56-line transcript came back as 1 line. The reader walks frame headers + itself, which stays correct on a Node that decodes every frame + (nodejs/node#65865). On a Node older than 22.15 (no zstd at all) the mode falls back to the pane, as before. - **Not every `user/message` is the user.** Each turn also records a plugin-sourced runtime-context snapshot; only `source.kind === 'user'` is a prompt. diff --git a/src/deepseek-transcript.ts b/src/deepseek-transcript.ts index dc374f10..7fbdcdd1 100644 --- a/src/deepseek-transcript.ts +++ b/src/deepseek-transcript.ts @@ -26,10 +26,11 @@ * **1. One zstd FRAME per append, not one zstd stream.** The file is * `session.jsonl.zstd`, and dsh appends by compressing each batch of lines into * its own frame and writing it at the end. `zstd -dc` handles that (frames - * concatenate by definition), but Node's `zlib.zstdDecompress()` and - * `createZstdDecompress()` both stop at the first frame end: measured on a real - * 56-line transcript, Node returned 158 bytes / 1 line where the CLI returned - * 43,747 bytes / 56 lines. That is a silent truncation to the session header — + * concatenate by definition), but before Node 24.22 and 26.10 + * (nodejs/node#65865) `zlib.zstdDecompress()` and `createZstdDecompress()` both + * stop at the first frame end, and the supported range still includes those + * versions: measured on a real 56-line transcript, Node returned 158 bytes / + * 1 line where the CLI returned 43,747 bytes / 56 lines. That is a silent truncation to the session header — * every call would have reported "no answer yet" forever. `decodeZstdFrames()` * below walks the frame headers itself and decompresses each frame, and * `test/deepseek-transcript.test.ts` pins it against multi-frame fixtures. diff --git a/test/deepseek-transcript.test.ts b/test/deepseek-transcript.test.ts index e0837070..292aac14 100644 --- a/test/deepseek-transcript.test.ts +++ b/test/deepseek-transcript.test.ts @@ -4,10 +4,11 @@ * Two of these assertions exist because the obvious implementation was measured * to be wrong against real files: * - * - dsh appends ONE ZSTD FRAME PER WRITE, and Node's `zlib` zstd decoder stops - * at the first frame end. A 56-line transcript decoded as 1 line / 158 bytes, - * which reads as "the worker never answered" rather than as an error. The - * multi-frame fixtures below are the guard. + * - dsh appends ONE ZSTD FRAME PER WRITE, and Node's `zlib` zstd decoder + * before 24.22 and 26.10 stops at the first frame end. A 56-line transcript + * decoded as 1 line / 158 bytes, which reads as "the worker never answered" + * rather than as an error. Newer Node decodes every frame (nodejs/node#65865), + * so the multi-frame fixtures below pin the walker itself, not Node's decoder. * - a fresh worker in a case directory that had been used before answered its * first `last-response` with the PREVIOUS session's reply. Session-to- * transcript pairing is therefore its own describe block. @@ -67,10 +68,6 @@ describe.skipIf(!zstdSupported())('zstd frame walking', () => { const lines = Array.from({ length: 40 }, (_, i) => JSON.stringify({ type: 'noise', seq: i })); const buf = framed(lines); - // The one-shot decoder is what this module had to replace. - const oneShot = (zlib as unknown as { zstdDecompressSync?: (b: Buffer) => Buffer }).zstdDecompressSync!(buf); - expect(oneShot.toString('utf8').trim().split('\n')).toHaveLength(1); - expect(zstdFrameRanges(buf)).toHaveLength(40); expect(decodeZstdFrames(buf).trim().split('\n')).toHaveLength(40); }); From e9f3aa20a5c6693de5ad9c80f74a0c3541c41b9e Mon Sep 17 00:00:00 2001 From: Codeman maintainer Date: Tue, 27 Oct 2026 01:21:56 +0100 Subject: [PATCH 2/2] fix: apply the review items for #710 at merge - docs/architecture-invariants.md DeepSeek mechanism (1): the decoder stops at the first frame only before Node 24.22 and 26.10 (nodejs/node#65865) - src/deepseek-transcript.ts:33: reflow the over-long comment line to the width of the lines around it - docs/deepseek-integration.md:241: same reflow Co-Authored-By: Claude Opus 5.5 --- docs/architecture-invariants.md | 2 +- docs/deepseek-integration.md | 3 ++- src/deepseek-transcript.ts | 9 +++++---- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/architecture-invariants.md b/docs/architecture-invariants.md index 7a743bb6..26141a7d 100644 --- a/docs/architecture-invariants.md +++ b/docs/architecture-invariants.md @@ -53,7 +53,7 @@ Pure helpers unit-tested in `test/base-path.test.ts` + `test/webview-proxy.test. ⚠️ **`agent_working` is a hook event with no Claude Code hook behind it** (157th SSE constant). It exists because a harness turn cannot run while one of its own modal approvals is on screen, so "the agent started working" proves a dialog was answered in the terminal. It joins `APPROVAL_RESOLVING_EVENTS`; without it a dsh session's red alert would survive until the next `stop`, the exact stuck-alert bug the claude path already had to fix once — and the pane-capture staleness sweep that fixed it there is Claude-dialog-shaped and cannot help here. -⚠️ **Answers are READ FROM DISK for this mode, not scraped off the pane** (`src/deepseek-transcript.ts`, behind `GET /api/sessions/:id/last-response`). dsh writes a structured JSONL transcript at `$DSH_HOME/sessions///session.jsonl.zstd`, so it belongs with claude and codex rather than with the pane-segmented modes — and for dsh specifically the segmenter was not merely coarse but WRONG: dsh-TUI paints a full-screen splash, so a `last-response` call on a fresh dsh session returned its ASCII-art logo, which anything polling for a worker's first answer reads as an answer. Four mechanisms in that file are load-bearing. **(1)** dsh appends **one zstd FRAME per write**, and Node's `zlib` zstd decoder — one-shot AND streaming — stops at the first frame end: a real 56-line transcript decoded as 1 line / 158 bytes, i.e. the session header alone, so every call would have reported "nothing said yet" forever. `zstdFrameRanges()` walks frame and block headers (no decompression) to find exact boundaries and decompresses each frame; splitting on the 4-byte magic instead would corrupt everything after a magic sequence that happens to occur inside compressed data. zstd itself is resolved at RUNTIME (`zstdSupported()`), because it landed in Node 22.15 while the project floor is 22.0 and `@types/node` still does not declare it; without it the mode falls back to the pane exactly as before, which is why `readDeepSeekLastResponse()` distinguishes `null` ("this reader cannot run here") from an empty result ("read fine, nothing said yet"). **(2)** Every turn also records a plugin-sourced `user/message` (dsh's runtime-context snapshot: sandbox policy, approval policy, cwd), so only `source.kind === 'user'` is a real prompt. **(3)** A turn that ends in `reason.kind: 'error'` is surfaced as `Turn error: …` (and a non-error early stop such as `max-tokens` as `Turn ended: …`) rather than as an empty string, which an agent reads as "still thinking" through fifteen polls. **(4)** Reply text is assembled per (turn, step): a finalized `assistant/message` wins, and the streamed `assistant/chunk` / `text-chunks` deltas are consulted ONLY for a step that never finalized (so a partial answer is readable mid-turn without ever being appended twice) — ⚠️ and "finalized" is tracked as a SET of steps, not as non-empty text, because a step whose whole reply was reasoning strips to `''` at the `` boundary and would otherwise resurrect the raw, unstripped deltas in its place (measured on a real conversation). ⚠️ Session→transcript pairing is by the transcript's own header `cwd` plus a ±60 s boot window against the Codeman session's `createdAt`, never by reproducing dsh's directory mangling (which already has two forms on disk, `` and `session-`) and never by newest-mtime alone: mtime alone handed a freshly spawned worker its PREDECESSOR's answer in the same case directory, which is worse than saying nothing because an agent cannot tell a stale answer from a fresh one. The `DSH_HOME` override reaches the reader through the narrow `Session.deepSeekHomeOverride` getter rather than an `envOverrides` accessor, since that map can hold provider credentials; it is ephemeral by design (never persisted), so a session that overrode it and outlived a server restart resolves the default tree and reads as "nothing said yet". Tests: `test/deepseek-transcript.test.ts`. +⚠️ **Answers are READ FROM DISK for this mode, not scraped off the pane** (`src/deepseek-transcript.ts`, behind `GET /api/sessions/:id/last-response`). dsh writes a structured JSONL transcript at `$DSH_HOME/sessions///session.jsonl.zstd`, so it belongs with claude and codex rather than with the pane-segmented modes — and for dsh specifically the segmenter was not merely coarse but WRONG: dsh-TUI paints a full-screen splash, so a `last-response` call on a fresh dsh session returned its ASCII-art logo, which anything polling for a worker's first answer reads as an answer. Four mechanisms in that file are load-bearing. **(1)** dsh appends **one zstd FRAME per write**, and Node's `zlib` zstd decoder (one-shot AND streaming) before 24.22 and 26.10 (nodejs/node#65865) stops at the first frame end: a real 56-line transcript decoded as 1 line / 158 bytes, i.e. the session header alone, so every call would have reported "nothing said yet" forever. `zstdFrameRanges()` walks frame and block headers (no decompression) to find exact boundaries and decompresses each frame; splitting on the 4-byte magic instead would corrupt everything after a magic sequence that happens to occur inside compressed data. zstd itself is resolved at RUNTIME (`zstdSupported()`), because it landed in Node 22.15 while the project floor is 22.0 and `@types/node` still does not declare it; without it the mode falls back to the pane exactly as before, which is why `readDeepSeekLastResponse()` distinguishes `null` ("this reader cannot run here") from an empty result ("read fine, nothing said yet"). **(2)** Every turn also records a plugin-sourced `user/message` (dsh's runtime-context snapshot: sandbox policy, approval policy, cwd), so only `source.kind === 'user'` is a real prompt. **(3)** A turn that ends in `reason.kind: 'error'` is surfaced as `Turn error: …` (and a non-error early stop such as `max-tokens` as `Turn ended: …`) rather than as an empty string, which an agent reads as "still thinking" through fifteen polls. **(4)** Reply text is assembled per (turn, step): a finalized `assistant/message` wins, and the streamed `assistant/chunk` / `text-chunks` deltas are consulted ONLY for a step that never finalized (so a partial answer is readable mid-turn without ever being appended twice) — ⚠️ and "finalized" is tracked as a SET of steps, not as non-empty text, because a step whose whole reply was reasoning strips to `''` at the `` boundary and would otherwise resurrect the raw, unstripped deltas in its place (measured on a real conversation). ⚠️ Session→transcript pairing is by the transcript's own header `cwd` plus a ±60 s boot window against the Codeman session's `createdAt`, never by reproducing dsh's directory mangling (which already has two forms on disk, `` and `session-`) and never by newest-mtime alone: mtime alone handed a freshly spawned worker its PREDECESSOR's answer in the same case directory, which is worse than saying nothing because an agent cannot tell a stale answer from a fresh one. The `DSH_HOME` override reaches the reader through the narrow `Session.deepSeekHomeOverride` getter rather than an `envOverrides` accessor, since that map can hold provider credentials; it is ephemeral by design (never persisted), so a session that overrode it and outlived a server restart resolves the default tree and reads as "nothing said yet". Tests: `test/deepseek-transcript.test.ts`. ⚠️ **This is also what makes dsh the one non-claude mode the `codeman` agent skill drives like claude** (`skills/codeman/preamble.sh`, preamble 1.20.0): with a real end-of-turn signal AND a real transcript, `spawn_workers alpha beta:deepseek` is a mixed fleet in one call and `sendwait`/`last_text` need no per-mode variant. Two traps are handled in the preamble rather than left to the agent. **Readiness is not the stop signal**: the harness reports `idle` at BOOT roughly 300 ms before its composer paints (measured 2.26 s vs 2.56 s after spawn, twice), so a send-and-wait fired straight after `quick-start` resolves on the boot edge, reports a turn that never ran, and strands the prompt in a pane that was not yet accepting input — `spawn_worker`'s dsh branch gates on the composer glyph (`❯`, overridable via `DSH_READY_MARK`) instead, after which the boot edge is spent and unobservable. And `sendwait` asks for `wait:"stop,exit"` rather than the `wait:true` default set, because that set also carries `idle`, which for any external CLI is inferred from output stabilization: on a dsh worker whose TUI repaints rarely, a re-wait resolved in 0 ms with `signal:"idle"` on a turn that had three minutes left to run. The skill also sends `deepSeekConfig.permissionMode: 'danger-full-access'` for its own workers, matching what the Run button sends, because the harness default still asks and a worker parked on an approval row cannot finish a fan-out (the multi-user clamp still applies). diff --git a/docs/deepseek-integration.md b/docs/deepseek-integration.md index 5f1072a7..02af1002 100644 --- a/docs/deepseek-integration.md +++ b/docs/deepseek-integration.md @@ -238,7 +238,8 @@ reader (`src/deepseek-transcript.ts`): of them, Node's `zlib` zstd decoder before 24.22 and 26.10 stops at the first: a real 56-line transcript came back as 1 line. The reader walks frame headers itself, which stays correct on a Node that decodes every frame - (nodejs/node#65865). On a Node older than 22.15 (no zstd at all) the mode falls back to the pane, as before. + (nodejs/node#65865). On a Node older than 22.15 (no zstd at all) the mode + falls back to the pane, as before. - **Not every `user/message` is the user.** Each turn also records a plugin-sourced runtime-context snapshot; only `source.kind === 'user'` is a prompt. diff --git a/src/deepseek-transcript.ts b/src/deepseek-transcript.ts index 7fbdcdd1..525188f7 100644 --- a/src/deepseek-transcript.ts +++ b/src/deepseek-transcript.ts @@ -30,10 +30,11 @@ * (nodejs/node#65865) `zlib.zstdDecompress()` and `createZstdDecompress()` both * stop at the first frame end, and the supported range still includes those * versions: measured on a real 56-line transcript, Node returned 158 bytes / - * 1 line where the CLI returned 43,747 bytes / 56 lines. That is a silent truncation to the session header — - * every call would have reported "no answer yet" forever. `decodeZstdFrames()` - * below walks the frame headers itself and decompresses each frame, and - * `test/deepseek-transcript.test.ts` pins it against multi-frame fixtures. + * 1 line where the CLI returned 43,747 bytes / 56 lines. That is a silent + * truncation to the session header — every call would have reported "no answer + * yet" forever. `decodeZstdFrames()` below walks the frame headers itself and + * decompresses each frame, and `test/deepseek-transcript.test.ts` pins it + * against multi-frame fixtures. * * **2. The user's prompts are mixed with injected context.** Every turn also * writes a `user/message` whose source is a plugin (the runtime-context