mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-17 03:39:33 +01:00
fix: bound what the Files panel walk reads and keep its subfolders in it (release review)
The round-robin walk capped what it listed, not what it read: every queued folder was read whole, sorted and held until drained, so one GET could pull millions of entries (100 folders of 50,000 files killed a 400 MB heap). A read now keeps only the entries the listing can use, and MAX_WALK_ENTRIES_READ stops opening folders at 250,000 entries, leaving the rest queued and flagged. A folder also spent its whole share on its own files, leaving nothing for the subfolders it had just found, alone in its round or beside a sibling, so search hits inside them were lost. Its files now count as one more share beside those subfolders. Math.max(directoriesInRound, 2) would have fixed only the sole-folder case and cut a wide root's folder list short. Folders below the walk root are probed before their readdir (#516), since the walk reaches folders the depth-first one never did. Filtering at the read also clears the false flag on a folder whose only unread entries were hidden or excluded, and an exactly-full workspace now finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a908be05ac
commit
fafec35344
8 files changed
+379
-66
No files matched your search
@@ -734,7 +734,13 @@ Frontend `readmymind-ui.js`: header 🧠 marker-hidden (`btn-readmymind--hidden`
|
||||
|
||||
⚠️ **A query turns that endpoint into a FLAT match list rather than a nested tree**, and the walk deliberately recurses past non-matching directories, since the whole point of searching is to reach a file whose ancestors do not match. An empty or whitespace-only query compiles to `null`, which is what keeps the default tree response byte-identical when no search is requested.
|
||||
|
||||
⚠️ **Both walks are a per-directory round-robin (breadth-first) over ONE 5,000-node budget.** They used to recurse depth-first through name-sorted folders, so an early folder could spend the whole budget: 4 of 57 top-level folders were shown in one real workspace. Each round divides what is left equally among the directories queued so far. That is fair per directory, not per subtree: a folder of thousands of one-file subfolders still crowds out its neighbours one level down (`a/b/c/d/needle.ts` beside such a `z/` is found depth-first and missed here), and that trade is accepted, since a file browser wants the shallow levels first. Search results therefore arrive shallow-first. ⚠️ A directory gets its node, with `children: []`, when it is counted and is read in a later round, so every cursor still queued after the loop is a folder that is unread or half read: it must carry `incomplete: true` or it reads as an empty folder, the failure that went from 0 to 635 such folders before the flag existed. `renderFileBrowserTree` draws one dim, inert row under it while it is expanded, and the row is not a `.file-tree-item`, so the click wiring leaves it alone. The budget check at the top of each round runs BEFORE the `shift()`, so a directory the round skips stays queued and keeps its flag. The walk's root has no node, so `truncated` is its only signal, and a folder dropped at the depth cap is not flagged, since the row says "file limit" and it looks the same as it always has.
|
||||
⚠️ **Both walks are a per-directory round-robin (breadth-first) over ONE 5,000-node budget.** They used to recurse depth-first through name-sorted folders, so an early folder could spend the whole budget: 4 of 57 top-level folders were shown in one real workspace. Each round divides what is left equally among the directories queued so far, and a folder's own files count as one more share beside the subfolders it has just found (`walkTurnEnd()`). Those subfolders are read in a later round, so a folder that spent all of its share on its own files left them nothing, alone in its round or beside a sibling, and search hits inside them were lost. A turn that found no subfolder holds nothing back. That is fair per directory, not per subtree: a folder of thousands of one-file subfolders still crowds out its neighbours one level down (`a/b/c/d/needle.ts` beside such a `z/` is found depth-first and missed here), and that trade is accepted, since a file browser wants the shallow levels first. Search results therefore arrive shallow-first.
|
||||
|
||||
⚠️ **The budget caps what a listing returns, never what it reads.** Taking turns means opening every folder found so far, where the depth-first walk stopped reading soon after its budget ran out, and a share of 16 entries costs the same read as a share of 5,000. So `readWalkFolder()` keeps only what the listing can use (the entries it lists, sorted, no more than the budget has left, with `more` recording that it dropped some: such a cursor stays queued and is flagged), and `MAX_WALK_ENTRIES_READ` (250,000) stops opening folders. The folder that crosses that line is read whole, as any folder always was, the ones not opened yet wait in `unread` and are flagged, and what was already read keeps its turns. Without the two, 100 folders of 50,000 files cost one GET 5 million entries and a 400 MB heap died (which ends the server, not the request). The price is that folders adding up to the ceiling leave later ones unread: flagged, not hidden, but not the fairness the budget gets, and a depth-first walk over the same tree showed less.
|
||||
|
||||
⚠️ **A folder below the walk's root is probed before it is read** (rule #516), because the round-robin reaches folders the depth-first walk stopped short of, each with an unbounded `readdir`. It is a bulk scan, so default options and the stall cap apply. Anything but `directory` is skipped like an unreadable folder, with no flag, since the row says "file limit". With the cap engaged (two unrelated paths stalled, by default) that is every folder below the root, until one answers. The root is not probed: a refused probe there would blank a healthy workspace.
|
||||
|
||||
⚠️ A directory gets its node, with `children: []`, when it is counted and is read in a later round, so every cursor still queued after the loop, and every one in `unread`, is a folder that is unread or half read: it must carry `incomplete: true` or it reads as an empty folder, the failure that went from 0 to 635 such folders before the flag existed. `renderFileBrowserTree` draws one dim, inert row under it while it is expanded, and the row is not a `.file-tree-item`, so the click wiring leaves it alone. The budget check inside the round runs BEFORE each `shift()`, so a directory the round skips stays queued and keeps its flag. Hidden and excluded entries are dropped at the read, so they never leave a folder looking unfinished, but a folder found as the budget ended is flagged unread even if it would have turned out to be empty: only reading it could tell. The walk's root has no node, so `truncated` is its only signal, and a folder dropped at the depth cap is not flagged, since the row says "file limit" and it looks the same as it always has.
|
||||
|
||||
⚠️ **Globs are never compiled into a RegExp**: `*a*a*a…` translated to `^.*a.*a.*a…$` is a classic backtracking blowup evaluated synchronously against every walked path, so one pathological query would freeze the event loop for the whole server (the same reason `search-service.ts` is regex-free). `globMatch()` is a two-pointer wildcard walk instead, O(text · pattern) with both operands short by construction, and an overlong query (`MAX_QUERY_LENGTH`, 256) also compiles to `null` rather than running.
|
||||
|
||||
|
||||
Reference in new issue
Block a user