fix: bound what the Files panel walk reads and keep its subfolders in it (release review)

The round-robin walk capped what it listed, not what it read: every queued
folder was read whole, sorted and held until drained, so one GET could pull
millions of entries (100 folders of 50,000 files killed a 400 MB heap). A read
now keeps only the entries the listing can use, and MAX_WALK_ENTRIES_READ stops
opening folders at 250,000 entries, leaving the rest queued and flagged.

A folder also spent its whole share on its own files, leaving nothing for the
subfolders it had just found, alone in its round or beside a sibling, so search
hits inside them were lost. Its files now count as one more share beside those
subfolders. Math.max(directoriesInRound, 2) would have fixed only the sole-folder
case and cut a wide root's folder list short.

Folders below the walk root are probed before their readdir (#516), since the
walk reaches folders the depth-first one never did.

Filtering at the read also clears the false flag on a folder whose only unread
entries were hidden or excluded, and an exactly-full workspace now finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.changeset/quiet-folders-share.md

@@ -2,4 +2,4 @@
 'aicodeman': patch
 ---
 
-Files panel: one large folder no longer uses up the 5,000-entry scan limit and hides the folders after it. The tree and its search now walk folders in a per-directory round-robin (breadth-first), so search results come back shallow-first rather than depth-first. This is fair per directory, not per subtree: a folder made of thousands of small subfolders can still use most of the limit one level down. A folder the limit stopped short of no longer looks empty: `GET /api/sessions/:id/files` marks it `incomplete: true` (additive) and, when expanded, it shows a dim "Not fully scanned (file limit reached)" row.
+Files panel: one large folder no longer uses up the 5,000-entry listing limit before the folders after it, or the ones inside it, get a turn. The tree and its search now walk folders in a per-directory round-robin (breadth-first), so search results come back shallow-first rather than depth-first. This is fair per directory, not per subtree: a folder made of thousands of small subfolders can still use most of the limit one level down. One listing also stops opening folders after reading 250,000 entries, so folders that add up to that can leave later ones unread, and a folder below the workspace that does not answer is skipped rather than waited on. A folder a limit stopped short of no longer looks empty: `GET /api/sessions/:id/files` marks it `incomplete: true` (additive) and, when expanded, it shows a dim "Not fully scanned (file limit reached)" row.

CLAUDE.md

@@ -236,7 +236,7 @@ The terminal provider scans `terminalLinkLine()`, which extends `terminalPathLin
 
 **File Viewer edit mode** (issue #212): the file-preview overlay edits workspace text files in place: `GET .../file-content?edit=1` + `PUT /api/sessions/:id/file-content`, policy in `src/config/file-editing.ts`. This is a **third file surface and the only one that writes file contents** (the path picker's New folder creates folders only): read-path confinement (realpath + workspace + ownership) plus sensitive/blocked/`.git` denies and an extension **allowlist**; writes are `wx`-temp + rename (no `O_CREAT` anywhere = edit-in-place is structural); optimistic concurrency via sha256 `baseHash` → 409. ⚠️ `edit=1` never truncates and the client must never save a plain-preview buffer (the 500-line truncation would silently delete the rest). ⚠️ CRLF/UTF-8 guards: EOL re-applied server-side, non-UTF-8 refused via round-trip compare. → [architecture-invariants#file-viewer-edit-mode](docs/architecture-invariants.md#file-viewer-edit-mode), `docs/file-viewer-edit-plan.md`
 
-**Files panel search** (COD-236, the `q` param on `GET /api/sessions/:id/files`): `compileFileQuery()` (`utils/file-query.ts`, pure) compiles the query into a predicate the server-side walk prunes with; a query returns a FLAT match list and the walk recurses past non-matching directories. An empty, whitespace-only or overlong (`MAX_QUERY_LENGTH`, 256) query compiles to `null`, keeping the default tree response byte-identical. ⚠️ **Never compile a glob into a RegExp** (`*a*a*a…` backtracks and freezes the event loop for the whole server): `globMatch()` is a two-pointer wildcard walk. Tree and search share one walk shape: a per-directory round-robin (breadth-first, so fair per directory, not per subtree) over one 5,000-node budget. ⚠️ A directory has its node before it is read, so every cursor still queued when the budget ends must get `incomplete: true` or it reads as an empty folder. → [architecture-invariants#files-panel-search](docs/architecture-invariants.md#files-panel-search)
+**Files panel search** (COD-236, the `q` param on `GET /api/sessions/:id/files`): `compileFileQuery()` (`utils/file-query.ts`, pure) compiles the query into a predicate the server-side walk prunes with; a query returns a FLAT match list and the walk recurses past non-matching directories. An empty, whitespace-only or overlong (`MAX_QUERY_LENGTH`, 256) query compiles to `null`, keeping the default tree response byte-identical. ⚠️ **Never compile a glob into a RegExp** (`*a*a*a…` backtracks and freezes the event loop for the whole server): `globMatch()` is a two-pointer wildcard walk. Tree and search share one walk shape: a per-directory round-robin (breadth-first, so fair per directory, not per subtree) over one 5,000-node budget, where a folder's own files count as one share beside the subfolders it just found (`walkTurnEnd()`). ⚠️ That budget caps what is LISTED, never what is read: a read keeps only what the listing can use (`readWalkFolder()`) and `MAX_WALK_ENTRIES_READ` (250,000) stops opening folders, or one GET reads the whole workspace into memory. ⚠️ A folder below the walk's root is probed (#516) before its `readdir`, and anything but `directory` is skipped unflagged. ⚠️ A directory has its node before it is read, so every cursor still queued or unread when the walk ends must get `incomplete: true` or it reads as an empty folder. → [architecture-invariants#files-panel-search](docs/architecture-invariants.md#files-panel-search)
 
 **Files panel in a worktree** (#765, `src/live-worktree.ts`, `root=live` on `GET /api/sessions/:id/files`): Claude's EnterWorktree moves the process, and `paneCwd` follows it (tmux reads the process cwd, measured on 2.1.318, back out on ExitWorktree too), while `workingDir` stays the session's identity. `refreshLiveWorktree()` (server.ts) turns a changed `paneCwd` into `SessionState.liveWorktree` by reading the `.git` FILE (never spawning git), walking up and stopping at `workingDir`, a repository or submodule of its own, or anything unreadable. ⚠️ The security argument has two halves. The lookup only ever answers a folder strictly below `workingDir`, but that is a LEXICAL judgement made when the pane moves, and a symlink can replace the folder or a parent later without moving the pane, so `root=live` re-checks on every request (`canWalkLiveWorktree()`, file-routes.ts: walk the worktree only when it resolves strictly inside the resolved `workingDir`, else walk `workingDir`). The walk starts there, returned `path`s stay relative to `workingDir`, and the routes keep confining to `workingDir`, so never make `liveWorktree` a confinement root (a worktree outside the folder stays unfollowed). ⚠️ Only that check uses resolved paths: the walk keeps the session's spelling, or a workspace reached through a symlink stops producing paths relative to `workingDir`. ⚠️ A plain subfolder is deliberately no answer, or a `cd` in a shell would re-root the panel. ⚠️ A listing reports `worktree` whichever folder it walked, even when the worktree has gone or the check refused it: `_followFileBrowserWorktree()` compares that with the session's state, and an absent report would reload forever. Tests: `test/live-worktree.test.ts`, `test/routes/file-routes-live-worktree.test.ts`, `test/file-browser-worktree.test.ts`. → [architecture-invariants#files-panel-in-a-worktree](docs/architecture-invariants.md#files-panel-in-a-worktree)
 

docs/api-reference.md

@@ -1014,13 +1014,20 @@ while a `q` containing `/` is matched against paths relative to the folder walke
 also carries `worktree`, the same `{ name, path }`, whenever the session has a `liveWorktree`,
 whichever folder was walked and with or without `root=live`.
 
-The walk reads at most 5,000 files and folders and sets `truncated` when it stops early. It is
-a per-directory round-robin (breadth-first): each folder found so far gets an equal share of what
-is left before any of them gets more, so one huge folder cannot hide its later siblings, and the
-`matches` of a `q` search come back shallow-first. A folder made of thousands of small subfolders
-can still use most of the limit one level down. In the tree, a directory node the limit stopped
-short of also carries `incomplete: true`: its `children` hold what was read before the limit,
-which can be nothing. The key is absent on every other node.
+The walk lists at most 5,000 files and folders and opens no more folders once it has read 250,000
+directory entries (the folder that crosses that line is read whole). It sets `truncated` when it
+stops with anything left, which includes a folder it never opened, even if that folder would have
+turned out to be empty. It is a per-directory round-robin (breadth-first): each folder found so far
+gets an equal share of what is left before any of them gets more, and a folder's own files count
+as one more share beside the subfolders it has just found, so one huge folder cannot use up the
+listing limit before its later siblings or its own subfolders are read, and the `matches` of a `q`
+search come back shallow-first. A folder made of thousands of small subfolders can still use most
+of the limit one level down, and folders that add up to the read limit can leave later ones
+unread. In the tree, a directory node either limit stopped short of also carries
+`incomplete: true`: its `children` hold what was listed before the limit, which can be nothing.
+The key is absent on every other node. A folder below the start of the walk that does not answer a
+bounded probe (a mount that stopped responding) is skipped like an unreadable one: it is listed
+with empty `children` and no `incomplete`.
 
 ## Session model (`displayModel`)
 

docs/architecture-invariants.md

@@ -734,7 +734,13 @@ Frontend `readmymind-ui.js`: header 🧠 marker-hidden (`btn-readmymind--hidden`
 
 ⚠️ **A query turns that endpoint into a FLAT match list rather than a nested tree**, and the walk deliberately recurses past non-matching directories, since the whole point of searching is to reach a file whose ancestors do not match. An empty or whitespace-only query compiles to `null`, which is what keeps the default tree response byte-identical when no search is requested.
 
-⚠️ **Both walks are a per-directory round-robin (breadth-first) over ONE 5,000-node budget.** They used to recurse depth-first through name-sorted folders, so an early folder could spend the whole budget: 4 of 57 top-level folders were shown in one real workspace. Each round divides what is left equally among the directories queued so far. That is fair per directory, not per subtree: a folder of thousands of one-file subfolders still crowds out its neighbours one level down (`a/b/c/d/needle.ts` beside such a `z/` is found depth-first and missed here), and that trade is accepted, since a file browser wants the shallow levels first. Search results therefore arrive shallow-first. ⚠️ A directory gets its node, with `children: []`, when it is counted and is read in a later round, so every cursor still queued after the loop is a folder that is unread or half read: it must carry `incomplete: true` or it reads as an empty folder, the failure that went from 0 to 635 such folders before the flag existed. `renderFileBrowserTree` draws one dim, inert row under it while it is expanded, and the row is not a `.file-tree-item`, so the click wiring leaves it alone. The budget check at the top of each round runs BEFORE the `shift()`, so a directory the round skips stays queued and keeps its flag. The walk's root has no node, so `truncated` is its only signal, and a folder dropped at the depth cap is not flagged, since the row says "file limit" and it looks the same as it always has.
+⚠️ **Both walks are a per-directory round-robin (breadth-first) over ONE 5,000-node budget.** They used to recurse depth-first through name-sorted folders, so an early folder could spend the whole budget: 4 of 57 top-level folders were shown in one real workspace. Each round divides what is left equally among the directories queued so far, and a folder's own files count as one more share beside the subfolders it has just found (`walkTurnEnd()`). Those subfolders are read in a later round, so a folder that spent all of its share on its own files left them nothing, alone in its round or beside a sibling, and search hits inside them were lost. A turn that found no subfolder holds nothing back. That is fair per directory, not per subtree: a folder of thousands of one-file subfolders still crowds out its neighbours one level down (`a/b/c/d/needle.ts` beside such a `z/` is found depth-first and missed here), and that trade is accepted, since a file browser wants the shallow levels first. Search results therefore arrive shallow-first.
+
+⚠️ **The budget caps what a listing returns, never what it reads.** Taking turns means opening every folder found so far, where the depth-first walk stopped reading soon after its budget ran out, and a share of 16 entries costs the same read as a share of 5,000. So `readWalkFolder()` keeps only what the listing can use (the entries it lists, sorted, no more than the budget has left, with `more` recording that it dropped some: such a cursor stays queued and is flagged), and `MAX_WALK_ENTRIES_READ` (250,000) stops opening folders. The folder that crosses that line is read whole, as any folder always was, the ones not opened yet wait in `unread` and are flagged, and what was already read keeps its turns. Without the two, 100 folders of 50,000 files cost one GET 5 million entries and a 400 MB heap died (which ends the server, not the request). The price is that folders adding up to the ceiling leave later ones unread: flagged, not hidden, but not the fairness the budget gets, and a depth-first walk over the same tree showed less.
+
+⚠️ **A folder below the walk's root is probed before it is read** (rule #516), because the round-robin reaches folders the depth-first walk stopped short of, each with an unbounded `readdir`. It is a bulk scan, so default options and the stall cap apply. Anything but `directory` is skipped like an unreadable folder, with no flag, since the row says "file limit". With the cap engaged (two unrelated paths stalled, by default) that is every folder below the root, until one answers. The root is not probed: a refused probe there would blank a healthy workspace.
+
+⚠️ A directory gets its node, with `children: []`, when it is counted and is read in a later round, so every cursor still queued after the loop, and every one in `unread`, is a folder that is unread or half read: it must carry `incomplete: true` or it reads as an empty folder, the failure that went from 0 to 635 such folders before the flag existed. `renderFileBrowserTree` draws one dim, inert row under it while it is expanded, and the row is not a `.file-tree-item`, so the click wiring leaves it alone. The budget check inside the round runs BEFORE each `shift()`, so a directory the round skips stays queued and keeps its flag. Hidden and excluded entries are dropped at the read, so they never leave a folder looking unfinished, but a folder found as the budget ended is flagged unread even if it would have turned out to be empty: only reading it could tell. The walk's root has no node, so `truncated` is its only signal, and a folder dropped at the depth cap is not flagged, since the row says "file limit" and it looks the same as it always has.
 
 ⚠️ **Globs are never compiled into a RegExp**: `*a*a*a…` translated to `^.*a.*a.*a…$` is a classic backtracking blowup evaluated synchronously against every walked path, so one pathological query would freeze the event loop for the whole server (the same reason `search-service.ts` is regex-free). `globMatch()` is a two-pointer wildcard walk instead, O(text · pattern) with both operands short by construction, and an overlong query (`MAX_QUERY_LENGTH`, 256) also compiles to `null` rather than running.
 

docs/wiki/Working-With-Files.md

@@ -10,10 +10,14 @@ A panel that browses the active session's working directory, or the git worktree
 moved into ([Git Worktrees](Git-Worktrees#when-an-agent-moves-into-a-worktree)). Its header
 button is on by default; if it is missing, re-enable it in **App Settings → Header & Panels**.
 
-The panel and its search scan up to 5,000 files and folders per load, shallow levels first and
-taking turns between folders, so one huge folder cannot hide the ones after it. When a workspace
-is bigger than that the status line says `(truncated)`, and an expanded folder the scan did not
-finish shows **Not fully scanned (file limit reached)** rather than looking empty.
+The panel and its search list up to 5,000 files and folders per load, shallow levels first and
+taking turns between folders, so one huge folder cannot use up that limit before the ones after it,
+or the folders inside it, get a turn. They also stop opening folders after reading 250,000 entries,
+so a workspace with several enormous folders can still leave later ones unread. When a workspace is
+bigger than either limit the status line says `(truncated)`, and an expanded folder the scan did
+not finish shows **Not fully scanned (file limit reached)** rather than looking empty. A folder that
+does not answer, such as a network mount that went away, is skipped rather than waited on, and
+looks empty.
 
 It renders what it can:
 

src/web/routes/file-routes.ts

@@ -8,7 +8,7 @@
 
 import { FastifyInstance, type FastifyReply } from 'fastify';
 import { basename as pathBasename, dirname, extname, isAbsolute, join, relative, resolve, sep } from 'node:path';
-import { createReadStream, realpathSync } from 'node:fs';
+import { createReadStream, realpathSync, type Dirent } from 'node:fs';
 import type { Readable } from 'node:stream';
 import fs from 'node:fs/promises';
 import { createHash, randomBytes } from 'node:crypto';
@@ -782,6 +782,72 @@ export function isPathWithinRoot(root: string, candidate: string): boolean {
   return rel === '' || (!isAbsolute(rel) && rel !== '..' && !rel.startsWith(`..${sep}`));
 }
 
+/**
+ * Folder entries one Files panel listing may read, summed over the folders it opens. The budget of
+ * the walk caps what a listing RETURNS, but taking turns means opening every folder found so far,
+ * so this is what caps the reading: past it, folders not opened yet stay unread and are reported as
+ * unfinished. It also bounds what the walk holds, since a folder keeps only entries it has read. A
+ * folder is read whole, so the one that crosses this line overshoots by its own size.
+ */
+const MAX_WALK_ENTRIES_READ = 250_000;
+
+/** One folder of a Files panel walk, tree or search. */
+interface WalkCursor {
+  dirPath: string;
+  depth: number;
+  /** What the listing can use of this folder, folders first. Unset until its first turn. */
+  entries?: Dirent[];
+  index: number;
+  /** The read kept fewer entries than the folder lists, so running out of `entries` does not finish it. */
+  more?: boolean;
+}
+
+/**
+ * First turn of a folder: reads it and sets `cursor.entries` to the part the listing could use,
+ * which is the entries it lists, sorted, up to what is left of the budget. Leaves `entries` unset
+ * for a folder it could not read. Answers the number of entries read, for the walk's read ceiling.
+ */
+async function readWalkFolder(
+  cursor: WalkCursor,
+  budgetLeft: number,
+  isListed: (entry: Dirent) => boolean
+): Promise<number> {
+  // The walk's root is the session's own folder. Below it any folder can be a mount that stopped
+  // answering, and a bulk scan keeps the stall cap (#516): an unbounded readdir there would hold a
+  // threadpool worker for good. Skipped like an unreadable one: it is not the file limit's doing.
+  if (cursor.depth > 1 && (await probePathKind(cursor.dirPath)) !== 'directory') return 0;
+
+  let read: Dirent[];
+  try {
+    read = await fs.readdir(cursor.dirPath, { withFileTypes: true });
+  } catch {
+    // Can't read directory (permission denied, etc.).
+    return 0;
+  }
+  const listed = read.filter(isListed).sort((a, b) => {
+    if (a.isDirectory() && !b.isDirectory()) return -1;
+    if (!a.isDirectory() && b.isDirectory()) return 1;
+    return a.name.localeCompare(b.name);
+  });
+  cursor.more = listed.length > budgetLeft;
+  cursor.entries = cursor.more ? listed.slice(0, budgetLeft) : listed;
+  return read.length;
+}
+
+/**
+ * Where a folder's turn ends, given a `share` of the budget. Folders sort first, and each one found
+ * here waits for a later round, by which time a folder that spent the rest of its share on its own
+ * files has left nothing to read it with. So those files count as one more folder: they take an
+ * equal part of what the subfolders left and the remainder stays in the budget. A turn that found
+ * no subfolder has nothing to hold back for, and a turn always covers at least one entry.
+ */
+function walkTurnEnd(entries: Dirent[], start: number, share: number): number {
+  const end = Math.min(start + share, entries.length);
+  let firstFile = start;
+  while (firstFile < end && entries[firstFile].isDirectory()) firstFile++;
+  return firstFile + Math.floor((end - firstFile) / (firstFile - start + 1));
+}
+
 /**
  * Whether a `root=live` listing may start at a session's `liveWorktree` (#765). `findLiveWorktree()`
  * judged that path lexically when the pane moved, and a symlink can take its place, or a parent's,
@@ -1665,7 +1731,7 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
       size?: number;
       extension?: string;
       children?: FileTreeNode[];
-      /** Set only on a folder the file limit stopped the walk short of, so `children` is not all of it. */
+      /** Set only on a folder the walk's limits stopped it short of, so `children` is not all of it. */
       incomplete?: boolean;
     }
 
@@ -1673,6 +1739,9 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
     let totalDirectories = 0;
     let truncated = false;
     const maxFiles = 5000;
+    let entriesRead = 0;
+    const isListed = (entry: Dirent): boolean =>
+      (includeHidden || !entry.name.startsWith('.')) && !(entry.isDirectory() && excludeDirs.has(entry.name));
 
     // ===== Search mode =====
     // A query turns this endpoint into a FLAT match list rather than a nested
@@ -1684,19 +1753,15 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
       const matches: FileTreeNode[] = [];
       const maxMatches = 1000;
 
-      interface SearchDirectoryCursor {
-        dirPath: string;
-        depth: number;
-        entries?: import('node:fs').Dirent[];
-        index: number;
-      }
-
-      const queue: SearchDirectoryCursor[] = [{ dirPath: walkRoot, depth: 1, index: 0 }];
+      const queue: WalkCursor[] = [{ dirPath: walkRoot, depth: 1, index: 0 }];
+      // Folders the read ceiling kept shut. What was already read still gets its turns.
+      const unread: WalkCursor[] = [];
       while (queue.length > 0 && totalFiles + totalDirectories < maxFiles && matches.length < maxMatches) {
         // Per-directory round-robin (breadth-first): each queued directory gets an equal share of what
         // is left before any of them gets more, so results arrive shallow-first. Fair per directory,
         // not per subtree: a folder of thousands of small subfolders queues thousands of directories
-        // and can still take most of the budget one level down.
+        // and can still take most of the budget one level down. Reading a folder costs the same
+        // whatever its share, so this never bounds the reading: MAX_WALK_ENTRIES_READ does.
         const directoriesInRound = queue.length;
         const nodesPerDirectory = Math.max(
           1,
@@ -1713,25 +1778,17 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
             continue;
           }
           if (!cursor.entries) {
-            try {
-              cursor.entries = await fs.readdir(cursor.dirPath, { withFileTypes: true });
-              cursor.entries.sort((a, b) => {
-                if (a.isDirectory() && !b.isDirectory()) return -1;
-                if (!a.isDirectory() && b.isDirectory()) return 1;
-                return a.name.localeCompare(b.name);
-              });
-            } catch {
-              // Can't read directory (permission denied, etc.).
+            if (entriesRead >= MAX_WALK_ENTRIES_READ) {
+              unread.push(cursor);
               continue;
             }
+            entriesRead += await readWalkFolder(cursor, maxFiles - totalFiles - totalDirectories, isListed);
+            if (!cursor.entries) continue;
           }
 
-          const endIndex = Math.min(cursor.index + nodesPerDirectory, cursor.entries.length);
+          const endIndex = walkTurnEnd(cursor.entries, cursor.index, nodesPerDirectory);
           while (cursor.index < endIndex && totalFiles + totalDirectories < maxFiles && matches.length < maxMatches) {
             const entry = cursor.entries[cursor.index++];
-            if (!includeHidden && entry.name.startsWith('.')) continue;
-            if (entry.isDirectory() && excludeDirs.has(entry.name)) continue;
-
             const fullPath = join(cursor.dirPath, entry.name);
             const relativePath = relative(workingDir, fullPath);
             const listedPath = relative(walkRoot, fullPath);
@@ -1762,10 +1819,10 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
               }
             }
           }
-          if (cursor.index < cursor.entries.length) queue.push(cursor);
+          if (cursor.index < cursor.entries.length || cursor.more) queue.push(cursor);
         }
       }
-      if (queue.length > 0) truncated = true;
+      if (queue.length > 0 || unread.length > 0) truncated = true;
 
       return {
         success: true,
@@ -1784,18 +1841,15 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
       };
     }
 
-    interface TreeDirectoryCursor {
-      dirPath: string;
-      depth: number;
+    interface TreeDirectoryCursor extends WalkCursor {
       /** This directory's node. The root of the walk has none: `tree` is its list of children. */
       node?: FileTreeNode;
       nodes: FileTreeNode[];
-      entries?: import('node:fs').Dirent[];
-      index: number;
     }
 
     const tree: FileTreeNode[] = [];
     const queue: TreeDirectoryCursor[] = [{ dirPath: walkRoot, depth: 1, nodes: tree, index: 0 }];
+    const unread: TreeDirectoryCursor[] = [];
     while (queue.length > 0 && totalFiles + totalDirectories < maxFiles) {
       // The same per-directory round-robin as the search walk above, with the same limits.
       const directoriesInRound = queue.length;
@@ -1814,25 +1868,17 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
           continue;
         }
         if (!cursor.entries) {
-          try {
-            cursor.entries = await fs.readdir(cursor.dirPath, { withFileTypes: true });
-            cursor.entries.sort((a, b) => {
-              if (a.isDirectory() && !b.isDirectory()) return -1;
-              if (!a.isDirectory() && b.isDirectory()) return 1;
-              return a.name.localeCompare(b.name);
-            });
-          } catch {
-            // Can't read directory (permission denied, etc.).
+          if (entriesRead >= MAX_WALK_ENTRIES_READ) {
+            unread.push(cursor);
             continue;
           }
+          entriesRead += await readWalkFolder(cursor, maxFiles - totalFiles - totalDirectories, isListed);
+          if (!cursor.entries) continue;
         }
 
-        const endIndex = Math.min(cursor.index + nodesPerDirectory, cursor.entries.length);
+        const endIndex = walkTurnEnd(cursor.entries, cursor.index, nodesPerDirectory);
         while (cursor.index < endIndex && totalFiles + totalDirectories < maxFiles) {
           const entry = cursor.entries[cursor.index++];
-          if (!includeHidden && entry.name.startsWith('.')) continue;
-          if (entry.isDirectory() && excludeDirs.has(entry.name)) continue;
-
           const fullPath = join(cursor.dirPath, entry.name);
           const relativePath = fullPath.slice(workingDir.length + 1);
           if (entry.isDirectory()) {
@@ -1858,15 +1904,15 @@ export function registerFileRoutes(app: FastifyInstance, ctx: SessionPort & Even
             cursor.nodes.push({ name: entry.name, path: relativePath, type: 'file', size, extension: ext });
           }
         }
-        if (cursor.index < cursor.entries.length) queue.push(cursor);
+        if (cursor.index < cursor.entries.length || cursor.more) queue.push(cursor);
       }
     }
     // Every directory still queued is unread or half read, and an empty `children` would read as an
     // empty folder. The root has no node to flag: `truncated` says the same for it.
-    for (const cursor of queue) {
+    for (const cursor of [...queue, ...unread]) {
       if (cursor.node) cursor.node.incomplete = true;
     }
-    if (queue.length > 0) truncated = true;
+    if (queue.length > 0 || unread.length > 0) truncated = true;
 
     return {
       success: true,

test/routes/file-search-mode.test.ts

@@ -172,8 +172,8 @@ describe('files endpoint search mode', () => {
     }
   });
 
-  it('copes with the root itself being unfinished', async () => {
-    // The root has no node to flag. Its folders are listed before its files, and none is read.
+  it('copes with the root itself being unfinished, and its subfolders still get read', async () => {
+    // The root has no node to flag. Its files stop the walk, but only after the folder it found.
     const wide = mkdtempSync(join(tmpdir(), 'codeman-file-root-'));
     mkdirSync(join(wide, 'first'));
     writeFileSync(join(wide, 'first', 'inside.txt'), '');
@@ -185,9 +185,99 @@ describe('files endpoint search mode', () => {
     try {
       const { data } = await get('');
       expect(data.truncated).toBe(true);
-      expect(data.tree).toHaveLength(5000);
-      expect(data.tree[0]).toMatchObject({ name: 'first', incomplete: true, children: [] });
-      expect(data.tree.filter((node: { incomplete?: boolean }) => node.incomplete)).toHaveLength(1);
+      expect(data.totalFiles + data.totalDirectories).toBe(5000);
+      expect(data.tree[0]).toMatchObject({ name: 'first', children: [{ name: 'inside.txt' }] });
+      expect(data.tree[0]).not.toHaveProperty('incomplete');
+
+      const search = await get('?q=inside');
+      expect(search.data.matches.map((match: { path: string }) => match.path)).toEqual([join('first', 'inside.txt')]);
+      expect(search.data.truncated).toBe(true);
+    } finally {
+      rmSync(wide, { recursive: true, force: true });
+    }
+  });
+
+  it('does not let a large folder alone in its round starve the subfolders it holds', async () => {
+    // One level below the walk root, with a second level of subfolder to reach.
+    const wide = mkdtempSync(join(tmpdir(), 'codeman-file-sole-'));
+    mkdirSync(join(wide, 'repo', 'src', 'lib'), { recursive: true });
+    writeFileSync(join(wide, 'repo', 'src', 'lib', 'needle.ts'), '');
+    for (let index = 0; index < 5100; index++) {
+      writeFileSync(join(wide, 'repo', `item-${index}.txt`), '');
+    }
+    harness.ctx._session.workingDir = wide;
+
+    try {
+      const { data } = await get('?q=needle');
+      expect(data.matches.map((match: { path: string }) => match.path)).toEqual([
+        join('repo', 'src', 'lib', 'needle.ts'),
+      ]);
+      expect(data.truncated).toBe(true);
+    } finally {
+      rmSync(wide, { recursive: true, force: true });
+    }
+  });
+
+  it('does not let large sibling folders starve the subfolders they hold', async () => {
+    // Two in a round, so "alone in its round" is not what makes this go wrong: each folder's
+    // equal share used to be spent on its own files, leaving nothing for what it had just found.
+    const wide = mkdtempSync(join(tmpdir(), 'codeman-file-pair-'));
+    for (const name of ['a-big', 'b-big']) {
+      mkdirSync(join(wide, name, 'src'), { recursive: true });
+      writeFileSync(join(wide, name, 'src', 'needle.ts'), '');
+      for (let index = 0; index < 5100; index++) {
+        writeFileSync(join(wide, name, `item-${index}.txt`), '');
+      }
+    }
+    harness.ctx._session.workingDir = wide;
+
+    try {
+      const { data } = await get('?q=needle');
+      expect(data.matches.map((match: { path: string }) => match.path).sort()).toEqual([
+        join('a-big', 'src', 'needle.ts'),
+        join('b-big', 'src', 'needle.ts'),
+      ]);
+    } finally {
+      rmSync(wide, { recursive: true, force: true });
+    }
+  });
+
+  it('calls a workspace of exactly 5,000 nodes finished', async () => {
+    const exact = mkdtempSync(join(tmpdir(), 'codeman-file-exact-'));
+    mkdirSync(join(exact, 'empty'));
+    for (let index = 0; index < 4999; index++) {
+      writeFileSync(join(exact, `item-${index}.txt`), '');
+    }
+    harness.ctx._session.workingDir = exact;
+
+    try {
+      const { data } = await get('');
+      expect(data.totalFiles + data.totalDirectories).toBe(5000);
+      expect(data.truncated).toBe(false);
+      expect(JSON.stringify(data)).not.toContain('incomplete');
+    } finally {
+      rmSync(exact, { recursive: true, force: true });
+    }
+  });
+
+  it('does not flag a folder whose only unread entries are ones the listing leaves out', async () => {
+    // 3,000 folders leave 2,000 of the budget, which buys each of the first 2,000 its `src`. All
+    // that is left unread in them is `vendor` and `.hidden`, which would never be listed anyway.
+    const wide = mkdtempSync(join(tmpdir(), 'codeman-file-excluded-'));
+    for (let index = 0; index < 3000; index++) {
+      const dir = join(wide, `dir-${String(index).padStart(4, '0')}`);
+      mkdirSync(join(dir, 'src'), { recursive: true });
+      mkdirSync(join(dir, 'vendor'));
+      writeFileSync(join(dir, '.hidden'), '');
+    }
+    harness.ctx._session.workingDir = wide;
+
+    try {
+      const { data } = await get('');
+      expect(data.truncated).toBe(true);
+      expect(data.tree[0].children.map((node: { name: string }) => node.name)).toEqual(['src']);
+      expect(data.tree[0]).not.toHaveProperty('incomplete');
+      expect(data.tree[2999]).toMatchObject({ incomplete: true, children: [] });
     } finally {
       rmSync(wide, { recursive: true, force: true });
     }

test/routes/file-walk-bounds.test.ts

@@ -0,0 +1,160 @@
+/**
+ * GET /api/sessions/:id/files: what one listing costs the server.
+ *
+ * test/routes/file-search-mode.test.ts pins what the walk returns, on real folders. This file pins
+ * what it reads, which is where a real tree cannot go: 100 folders of 50,000 files is minutes of
+ * setup, and a dead network mount cannot be made in a test. So `fs.readdir` and `fs.stat` are
+ * stubbed here, on the one object the route and the bounded probe both call through.
+ *
+ * Port: none (app.inject).
+ */
+import { Dirent, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
+import fs from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+
+import { createRouteTestHarness, type RouteTestHarness } from './_route-test-utils.js';
+import { registerFileRoutes } from '../../src/web/routes/file-routes.js';
+
+const UV_DIRENT_FILE = 1;
+const UV_DIRENT_DIR = 2;
+const dirent = (name: string, type: number, parent: string): Dirent =>
+  new (Dirent as unknown as new (name: string, type: number, path: string) => Dirent)(name, type, parent);
+
+describe('files endpoint read bounds', () => {
+  let harness: RouteTestHarness;
+
+  beforeEach(async () => {
+    harness = await createRouteTestHarness(registerFileRoutes);
+  });
+
+  afterEach(() => {
+    vi.restoreAllMocks();
+  });
+
+  const get = async (query: string) => {
+    const res = await harness.app.inject({
+      method: 'GET',
+      url: `/api/sessions/${harness.ctx._sessionId}/files${query}`,
+    });
+    expect(res.statusCode).toBe(200);
+    return JSON.parse(res.body).data;
+  };
+
+  describe('over a workspace far bigger than the budget', () => {
+    const FOLDERS = 100;
+    const FILES_EACH = 50_000;
+    const WORKSPACE = '/virtual/ws';
+    let entriesServed = 0;
+
+    beforeEach(() => {
+      entriesServed = 0;
+      const folders = Array.from({ length: FOLDERS }, (_, index) =>
+        dirent(`class-${String(index).padStart(5, '0')}`, UV_DIRENT_DIR, WORKSPACE)
+      );
+      const files = Array.from({ length: FILES_EACH }, (_, index) =>
+        dirent(`image-${String(index).padStart(8, '0')}.jpg`, UV_DIRENT_FILE, 'x')
+      );
+      vi.spyOn(fs, 'readdir').mockImplementation((async (path: string) => {
+        const entries = path === WORKSPACE ? folders : files;
+        entriesServed += entries.length;
+        return [...entries];
+      }) as never);
+      vi.spyOn(fs, 'stat').mockImplementation((async (path: string) => ({
+        size: 1,
+        isDirectory: () => !String(path).endsWith('.jpg'),
+      })) as never);
+      harness.ctx._session.workingDir = WORKSPACE;
+    });
+
+    it('stops reading once it has read 250,000 entries, not at the end of the workspace', async () => {
+      const data = await get('');
+
+      // Past the ceiling by at most the folder that crossed it.
+      expect(entriesServed).toBeLessThanOrEqual(250_000 + FILES_EACH);
+      // What was read is still used, up to the budget, and nothing unread passes as empty.
+      expect(data.totalFiles + data.totalDirectories).toBe(5000);
+      expect(data.truncated).toBe(true);
+      expect(data.tree).toHaveLength(FOLDERS);
+      for (const folder of data.tree) expect(folder.incomplete).toBe(true);
+      expect(data.tree.filter((folder: { children: unknown[] }) => folder.children.length === 0)).toHaveLength(
+        FOLDERS - 5
+      );
+    });
+
+    it('stops a search the same way', async () => {
+      const data = await get('?q=image-00000000');
+
+      expect(entriesServed).toBeLessThanOrEqual(250_000 + FILES_EACH);
+      expect(data.matches.length).toBeGreaterThan(0);
+      expect(data.truncated).toBe(true);
+    });
+  });
+
+  describe('with a dead mount below the workspace', () => {
+    let workspace: string;
+    let deadFolder: string;
+    let deadReads: number;
+    let releaseStat: (() => void) | undefined;
+
+    beforeEach(() => {
+      workspace = mkdtempSync(join(tmpdir(), 'codeman-file-dead-'));
+      deadFolder = join(workspace, 'z-nas');
+      mkdirSync(deadFolder);
+      mkdirSync(join(workspace, 'a-fine'));
+      writeFileSync(join(workspace, 'a-fine', 'needle.ts'), '');
+      harness.ctx._session.workingDir = workspace;
+
+      deadReads = 0;
+      const never = new Promise<never>(() => {});
+      const realReaddir = fs.readdir.bind(fs);
+      const realStat = fs.stat.bind(fs);
+      vi.spyOn(fs, 'readdir').mockImplementation(((path: string, options: never) => {
+        if (path !== deadFolder) return realReaddir(path, options);
+        deadReads++;
+        return never;
+      }) as never);
+      vi.spyOn(fs, 'stat').mockImplementation(((path: string) => {
+        if (path !== deadFolder) return realStat(path);
+        // Settled in afterEach: the probe calls a path stalled for as long as its stat is pending.
+        return new Promise((_resolve, reject) => {
+          releaseStat = () => reject(Object.assign(new Error('gone'), { code: 'EIO' }));
+        });
+      }) as never);
+      vi.spyOn(console, 'warn').mockImplementation(() => {});
+    });
+
+    afterEach(async () => {
+      releaseStat?.();
+      releaseStat = undefined;
+      await new Promise((settled) => setTimeout(settled, 0));
+      rmSync(workspace, { recursive: true, force: true });
+    });
+
+    const deadline = <T>(work: Promise<T>): Promise<T> =>
+      Promise.race([
+        work,
+        new Promise<T>((_resolve, reject) => {
+          setTimeout(() => reject(new Error('the listing never answered')), 5000).unref();
+        }),
+      ]);
+
+    it('answers without it, and costs it one stuck stat however often the panel asks', async () => {
+      const first = await deadline(get(''));
+
+      const dead = first.tree.find((node: { name: string }) => node.name === 'z-nas');
+      expect(dead).toEqual({ name: 'z-nas', path: 'z-nas', type: 'directory', children: [] });
+      expect(first.truncated).toBe(false);
+      expect(first.tree[0].children.map((node: { name: string }) => node.name)).toEqual(['needle.ts']);
+      expect(deadReads).toBe(0);
+
+      const statCalls = () => vi.mocked(fs.stat).mock.calls.filter(([path]) => path === deadFolder).length;
+      const stats = statCalls();
+      const search = await deadline(get('?q=needle'));
+      expect(search.matches.map((match: { path: string }) => match.path)).toEqual([join('a-fine', 'needle.ts')]);
+      expect(statCalls()).toBe(stats);
+      expect(deadReads).toBe(0);
+    });
+  });
+});