fix: give the host-sensor readers a hard deadline so a hung nvidia-smi cannot hold the stats route (release review)

`run()` relied on execFile's `timeout`, which only sends the child a
signal: execFile's callback, and so the promise, settles when the child
exits. An nvidia-smi stuck on a hung driver (uninterruptible sleep) never
does, so `readNativeGpu()` stayed pending, every later `?gpu=1` request
joined that promise through `inFlight`, and since the route awaits the
sensors before `getSystemStats()`, CPU and MEM froze too for every device
with GPU Stats on. The backoff (`nvidiaSkipUntil`) only armed once the
child finally exited.

`run()` now races its own unref'd timer and answers undefined at the
deadline, clearing the timer when execFile settles first. The existing
failure path then arms the 60 s backoff as for any failing reader. The
PowerShell RAM probe shares `run()` and gets the same bound.

Test: an nvidia-smi that never calls back answers within the deadline, the
next poll does not run it again, and its late exit changes nothing (the
test fails on the previous `run()`). Checked against a real TERM-ignoring
child on PATH: the read settles at 1.5 s instead of when the child exits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Codeman maintainer committed 2026-10-24 16:46:31 +02:00
1 parent fb83a5c8df
commit 40158ce1ef
2 files changed
+47 -3

No files matched your search

+12 -3
View File
@@ -166,11 +166,20 @@ export function parseTypeperf(output: string): WindowsCounters {
return result;
}
/**
* Run a reader, answering undefined on failure or once `timeout` has passed. execFile's own `timeout` only
* sends the child a signal, and its callback waits for the child to exit: an nvidia-smi stuck on a hung
* driver (uninterruptible sleep) never does, and every stats request would wait on it with CPU and MEM. So
* the deadline here answers by itself; the child is left to the signal.
*/
const run = (cmd: string, args: string[], timeout = EXEC_TIMEOUT_MS) =>
new Promise<string | undefined>((resolve) => {
execFile(cmd, args, { timeout, windowsHide: true, maxBuffer: 4 * MB }, (err, stdout) =>
resolve(err ? undefined : stdout)
);
const deadline = setTimeout(() => resolve(undefined), timeout);
deadline.unref();
execFile(cmd, args, { timeout, windowsHide: true, maxBuffer: 4 * MB }, (err, stdout) => {
clearTimeout(deadline);
resolve(err ? undefined : stdout);
});
});
const readText = (path: string) =>