mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-11-14 02:29:33 +01:00
fix: give the host-sensor readers a hard deadline so a hung nvidia-smi cannot hold the stats route (release review)
`run()` relied on execFile's `timeout`, which only sends the child a signal: execFile's callback, and so the promise, settles when the child exits. An nvidia-smi stuck on a hung driver (uninterruptible sleep) never does, so `readNativeGpu()` stayed pending, every later `?gpu=1` request joined that promise through `inFlight`, and since the route awaits the sensors before `getSystemStats()`, CPU and MEM froze too for every device with GPU Stats on. The backoff (`nvidiaSkipUntil`) only armed once the child finally exited. `run()` now races its own unref'd timer and answers undefined at the deadline, clearing the timer when execFile settles first. The existing failure path then arms the 60 s backoff as for any failing reader. The PowerShell RAM probe shares `run()` and gets the same bound. Test: an nvidia-smi that never calls back answers within the deadline, the next poll does not run it again, and its late exit changes nothing (the test fails on the previous `run()`). Checked against a real TERM-ignoring child on PATH: the read settles at 1.5 s instead of when the child exits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
fb83a5c8df
commit
40158ce1ef
2 files changed
+47
-3
No files matched your search
+12
-3
@@ -166,11 +166,20 @@ export function parseTypeperf(output: string): WindowsCounters {
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a reader, answering undefined on failure or once `timeout` has passed. execFile's own `timeout` only
|
||||
* sends the child a signal, and its callback waits for the child to exit: an nvidia-smi stuck on a hung
|
||||
* driver (uninterruptible sleep) never does, and every stats request would wait on it with CPU and MEM. So
|
||||
* the deadline here answers by itself; the child is left to the signal.
|
||||
*/
|
||||
const run = (cmd: string, args: string[], timeout = EXEC_TIMEOUT_MS) =>
|
||||
new Promise<string | undefined>((resolve) => {
|
||||
execFile(cmd, args, { timeout, windowsHide: true, maxBuffer: 4 * MB }, (err, stdout) =>
|
||||
resolve(err ? undefined : stdout)
|
||||
);
|
||||
const deadline = setTimeout(() => resolve(undefined), timeout);
|
||||
deadline.unref();
|
||||
execFile(cmd, args, { timeout, windowsHide: true, maxBuffer: 4 * MB }, (err, stdout) => {
|
||||
clearTimeout(deadline);
|
||||
resolve(err ? undefined : stdout);
|
||||
});
|
||||
});
|
||||
|
||||
const readText = (path: string) =>
|
||||
|
||||
Reference in new issue
Block a user