FN-6834: use OS-available memory for system metrics

Use shared OS-available memory readings so macOS cache/inactive pages are not counted as used.

- Add a core available-memory helper that prefers `process.availableMemory()` and flags `freemem` fallback reliability.
- Route core metrics, dashboard system stats, and the dashboard TUI through the shared helper.
- Move and expand memory tests across core, dashboard routes, and verification coverage while documenting the telemetry behavior.
- Add a patch changeset for the published CLI package.

Files changed:
 .changeset/fix-macos-memory-used.md                |  5 ++
 docs/architecture.md                               |  2 +-
 docs/dashboard-guide.md                            |  2 +-
 .../__tests__/available-memory.test.ts             | 54 --------------
 .../cli/src/commands/dashboard-tui/controller.ts   | 37 +---------
 .../core/src/__tests__/available-memory.test.ts    | 83 ++++++++++++++++++++++
 packages/core/src/__tests__/system-metrics.test.ts | 36 +++++++++-
 packages/core/src/available-memory.ts              | 32 +++++++++
 packages/core/src/index.ts                         |  1 +
 packages/core/src/system-metrics.ts                |  9 ++-
 .../dashboard/src/__tests__/routes-system.test.ts  | 27 +++++++
 packages/dashboard/src/routes.ts                   |  7 +-
 .../src/__tests__/run-verification-command.test.ts | 18 +++++
 13 files changed, 218 insertions(+), 95 deletions(-)

Fusion-Task-Id: FN-6834

Fusion-Task-Lineage: 2e651551-6c35-46d1-b517-535758c7ace2
This commit is contained in:
gsxdsm
2026-06-21 13:32:19 -07:00
parent 217fd74f56
commit d99246cb74
13 changed files with 218 additions and 95 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
Fix macOS system memory usage reporting by deriving host memory used from OS-available memory instead of raw `os.freemem()` pages.

View File

@@ -859,7 +859,7 @@ Operator setup + troubleshooting guide: **[Remote Access runbook](./remote-acces
Key server capabilities:
- REST APIs for tasks, git, GitHub, agents, missions, planning, automations/routines, settings
- System stats snapshot and vitest process controls APIs (`GET /api/system-stats`, `POST /api/kill-vitest`) exposing dashboard process/system telemetry (including app CPU percentage and host memory rendered as numeric values, radial gauges, and trend sparklines in the Command Center System area), task/agent aggregates, and manual vitest process termination
- System stats snapshot and vitest process controls APIs (`GET /api/system-stats`, `POST /api/kill-vitest`) exposing dashboard process/system telemetry (including app CPU percentage and host memory rendered as numeric values, radial gauges, and trend sparklines in the Command Center System area), task/agent aggregates, and manual vitest process termination. Host-memory usage is derived from shared OS-available memory (`process.availableMemory()` with an unreliable `freemem` fallback) rather than raw free pages so macOS inactive/cache memory is not counted as used.
- Command Center analytics APIs (`GET /api/command-center/tokens`, `/tools`, `/activity`, `/productivity`, `/team`, `/github`, `/signals`, `/plugin-activations`, `/live`) are project-scoped dashboard routes. `/productivity` reads Lines changed from nullable `task_commit_associations.additions`/`deletions` merge-time diff stats, derives estimated `hoursSaved` from that LOC via the exported `HUMAN_LINES_PER_HOUR` rate, and keeps the unavailable sentinel for both fields when no in-range association has stats. Its `taskDuration` payload aggregates done tasks whose `executionCompletedAt` falls in the selected range, using positive `tasks.cumulativeActiveMs` values for completed count, average, median, p90, and total active execution time; missing qualifying durations remain unavailable rather than zero. `/signals` aggregates real local `incidents` rows for total/open/resolved counts, MTTR, and source/severity/status breakdowns and returns honest empty/unavailable sentinels instead of synthetic signal volume. `/plugin-activations` aggregates persisted plugin/extension load events for the selected range and returns unavailable when no rows exist instead of treating missing history as zero activations.
- Remote access APIs (`/api/remote/*`) for provider config, activation, tunnel lifecycle, status, token issuance, authenticated URL generation, and QR payload generation
- Operational runbook (prereqs/security/troubleshooting): [`docs/remote-access.md`](./remote-access.md)

View File

@@ -694,7 +694,7 @@ Features:
<!-- FNXC:CommandCenter 2026-06-21-07:07: FN-6722 requires the GitHub area to expose a resolved-issue detail list from local task-store analytics only, with exact close timestamps flagged when reconciliation populated `sourceIssueClosedAt` and approximation called out otherwise. -->
- **GitHub** shows local GitHub issue flow for the selected range: **Filed by Fusion** counts tasks with a persisted `githubTracking.issue`, **Fixed by Fusion** counts tasks imported from GitHub source issues (`sourceIssueProvider = "github"`) that are currently in `done`, using the persisted `sourceIssueClosedAt` / `TaskSourceIssue.closedAt` close time when the reconciler has observed it. Rows that predate the field or have not been observed closed fall back to task `updatedAt` as the documented completion-time approximation; Fusion never fabricates a close timestamp and this analytics path never calls GitHub, the `gh` CLI, or any external network source. To make historical fixed dates exact, use **Backfill exact close times** in the Fixed by Fusion card; the dashboard calls the project-scoped manual `POST /api/git/github/backfill-source-issue-closed-at` endpoint in `{ offset, limit }` batches until `hasMore` is false, then surfaces the accumulated `scanned`, `filled`, `skipped`, and `errors` counts. The endpoint fetches real GitHub `closed_at` values once, fills only missing `sourceIssueClosedAt` values, and never runs automatically or from analytics-time rendering. The area shows filed/fixed/net stat cards, a filed-vs-fixed pie, a filed/fixed recharts trend line, existing daily sparklines, a by-repository bar breakdown, and a **Resolved issues** detail list. Resolved rows include the Fusion task, repository, source issue number, optional issue link, resolved timestamp, and whether that timestamp is exact (`sourceIssueClosedAt`) or the documented `updatedAt` approximation; missing issue URLs render as plain text rather than empty anchors or click targets. The same resolved rows are available from the GitHub analytics payload as `resolved` and from the CSV export.
- **Signals** is backed by the project-scoped `/api/command-center/signals` endpoint, which aggregates real rows from the local `incidents` table. It shows total/open/resolved counts, MTTR when resolved incidents have enough timestamps, and source/severity/status breakdowns; an empty incidents table renders honest zero counts with MTTR unavailable rather than fabricated signal volume. It adds an open-vs-resolved status pie from the same response. Signals has no per-day series today, so it intentionally does not render a line chart or fabricate a trend. External connectors that ingest third-party signals into incidents are tracked separately in FN-6706.
- **System** is the canonical system-telemetry destination. It reads local telemetry from `GET /api/system-stats` and, when multiple registered nodes exist, shows a node selector that can proxy the same system-stats payload through `GET /api/nodes/:id/system-stats` for remote nodes. It renders live radial gauges for app CPU, host memory, and heap usage, keeps a small client-side rolling buffer for CPU/memory/heap trend sparklines, adds a recharts CPU/memory/heap line from that same rolling buffer, and adds a task-by-column pie alongside the existing tasks-by-column and agents-by-state bars. The Vitest process count, manual kill confirmation, auto-kill toggle, threshold controls, and last-auto-kill timestamp moved here unchanged; the standalone System Stats modal and its desktop Header/mobile More affordances were removed.
- **System** is the canonical system-telemetry destination. It reads local telemetry from `GET /api/system-stats` and, when multiple registered nodes exist, shows a node selector that can proxy the same system-stats payload through `GET /api/nodes/:id/system-stats` for remote nodes. It renders live radial gauges for app CPU, host memory, and heap usage, keeps a small client-side rolling buffer for CPU/memory/heap trend sparklines, adds a recharts CPU/memory/heap line from that same rolling buffer, and adds a task-by-column pie alongside the existing tasks-by-column and agents-by-state bars. Host memory uses OS-available memory (Node `process.availableMemory()` when available, with a flagged `freemem` fallback) so macOS inactive/cache pages are not reported as used. The Vitest process count, manual kill confirmation, auto-kill toggle, threshold controls, and last-auto-kill timestamp moved here unchanged; the standalone System Stats modal and its desktop Header/mobile More affordances were removed.
- **Mission Control** shows live active sessions/runs/nodes, current sessions and nodes, an animated live activity snapshot, and a live SDLC funnel; when idle it reports that live updates resume when work starts. No additional pie or line chart is rendered because the live SDLC funnel already visualizes the panel's only quantitative distribution (`snapshot.columns`), while sessions/nodes are live control lists rather than categorical analytics. Motion-heavy accents respect reduced-motion preferences.
- CSV exports are available from the analytics endpoints with `?format=csv`. The Activity CSV includes daily `agentRuns` values plus summary rows for `(agentRuns.total)`, `(agentRuns.active)`, `(agentRuns.completed)`, and `(agentRuns.failed)`.

View File

@@ -1,54 +0,0 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import os from "node:os";
import { getAvailableMemoryInfo } from "../controller.js";
type ProcessWithAvailableMemory = NodeJS.Process & { availableMemory?: () => number };
describe("getAvailableMemoryInfo", () => {
afterEach(() => {
vi.restoreAllMocks();
});
it("reports a reliable reading from process.availableMemory when present", () => {
const proc = process as ProcessWithAvailableMemory;
if (typeof proc.availableMemory !== "function") {
// Older runtime without the API — covered by the fallback test below.
return;
}
const spy = vi.spyOn(proc, "availableMemory").mockReturnValue(123_456_789);
expect(getAvailableMemoryInfo()).toEqual({ bytes: 123_456_789, reliable: true });
expect(spy).toHaveBeenCalled();
});
it("falls back to os.freemem and flags the reading unreliable when the API is missing", () => {
const proc = process as ProcessWithAvailableMemory;
const original = proc.availableMemory;
// Simulate a runtime without process.availableMemory (Node < 22). The
// freemem fallback must be flagged unreliable: on macOS freemem reads
// ~99% used on an idle machine, and treating it as a pressure signal made
// the vitest auto-kill fire every 30s (2026-06-03 incident).
Reflect.deleteProperty(proc, "availableMemory");
const freememSpy = vi.spyOn(os, "freemem").mockReturnValue(42);
try {
expect(getAvailableMemoryInfo()).toEqual({ bytes: 42, reliable: false });
} finally {
if (original) proc.availableMemory = original;
freememSpy.mockRestore();
}
});
it("falls back unreliable when process.availableMemory throws", () => {
const proc = process as ProcessWithAvailableMemory;
if (typeof proc.availableMemory !== "function") return;
vi.spyOn(proc, "availableMemory").mockImplementation(() => {
throw new Error("not supported");
});
const freememSpy = vi.spyOn(os, "freemem").mockReturnValue(7);
try {
expect(getAvailableMemoryInfo()).toEqual({ bytes: 7, reliable: false });
} finally {
freememSpy.mockRestore();
}
});
});

View File

@@ -1,40 +1,7 @@
import os from "node:os";
import v8 from "node:v8";
import { appendFileSync } from "node:fs";
import { findVitestProcessIds } from "@fusion/core";
// `os.freemem()` on macOS only counts truly-free pages and excludes the large
// "inactive"/cached pool that the OS will reclaim on demand — so total-free
// reads ~95%+ used on an otherwise-idle machine. `process.availableMemory()`
// (Node 22+ — NOT `os.availableMemory`, which does not exist and silently
// fell through to the freemem trap this function was written to avoid)
// reports memory the OS considers available, matching Activity Monitor's
// notion of "used". The freemem fallback is flagged unreliable so pressure-
// triggered actions can refuse to fire on a garbage ratio: with freemem, an
// idle 256GB Mac reads ~99% used and the vitest auto-kill fired every 30s
// regardless of real pressure (2026-06-03 incident).
interface AvailableMemoryReading {
bytes: number;
/** False when only `os.freemem()` was available — unusable as a pressure signal. */
reliable: boolean;
}
export function getAvailableMemoryInfo(): AvailableMemoryReading {
const processFn = (process as unknown as { availableMemory?: () => number }).availableMemory;
if (typeof processFn === "function") {
try {
const v = processFn.call(process);
if (Number.isFinite(v) && v >= 0) return { bytes: v, reliable: true };
} catch {
// fall through
}
}
return { bytes: os.freemem(), reliable: false };
}
function getAvailableMemory(): number {
return getAvailableMemoryInfo().bytes;
}
import { findVitestProcessIds, getAvailableMemoryBytes, getAvailableMemoryInfo } from "@fusion/core";
const TUI_DEBUG_LOG = process.env.FUSION_TUI_DEBUG_LOG;
function tuiDebug(tag: string, data: Record<string, unknown>): void {
@@ -317,7 +284,7 @@ export class DashboardTUI {
loadAvg: [load[0] ?? 0, load[1] ?? 0, load[2] ?? 0],
cpuCount: os.cpus().length,
systemTotalMem: os.totalmem(),
systemFreeMem: getAvailableMemory(),
systemFreeMem: getAvailableMemoryBytes(),
pid: process.pid,
nodeVersion: process.version,
platform: `${process.platform}/${process.arch}`,

View File

@@ -0,0 +1,83 @@
import { afterEach, describe, expect, it, vi } from "vitest";
const { freememMock } = vi.hoisted(() => ({
freememMock: vi.fn(),
}));
vi.mock("node:os", () => ({
freemem: freememMock,
}));
import { getAvailableMemoryBytes, getAvailableMemoryInfo } from "../available-memory.js";
type ProcessWithAvailableMemory = NodeJS.Process & { availableMemory?: () => number };
function restoreAvailableMemory(original: ProcessWithAvailableMemory["availableMemory"]): void {
const proc = process as ProcessWithAvailableMemory;
if (original) {
proc.availableMemory = original;
} else {
Reflect.deleteProperty(proc, "availableMemory");
}
}
describe("getAvailableMemoryInfo", () => {
afterEach(() => {
vi.restoreAllMocks();
freememMock.mockReset();
});
it("reports a reliable reading from process.availableMemory when present", () => {
const proc = process as ProcessWithAvailableMemory;
const original = proc.availableMemory;
proc.availableMemory = vi.fn(() => 123_456_789);
try {
expect(getAvailableMemoryInfo()).toEqual({ bytes: 123_456_789, reliable: true });
expect(getAvailableMemoryBytes()).toBe(123_456_789);
expect(proc.availableMemory).toHaveBeenCalledTimes(2);
expect(freememMock).not.toHaveBeenCalled();
} finally {
restoreAvailableMemory(original);
}
});
it("falls back to os.freemem and flags the reading unreliable when the API is missing", () => {
const proc = process as ProcessWithAvailableMemory;
const original = proc.availableMemory;
Reflect.deleteProperty(proc, "availableMemory");
freememMock.mockReturnValue(42);
try {
expect(getAvailableMemoryInfo()).toEqual({ bytes: 42, reliable: false });
} finally {
restoreAvailableMemory(original);
}
});
it("falls back unreliable when process.availableMemory throws", () => {
const proc = process as ProcessWithAvailableMemory;
const original = proc.availableMemory;
proc.availableMemory = vi.fn(() => {
throw new Error("not supported");
});
freememMock.mockReturnValue(7);
try {
expect(getAvailableMemoryInfo()).toEqual({ bytes: 7, reliable: false });
} finally {
restoreAvailableMemory(original);
}
});
it("treats zero, NaN, and negative availableMemory readings as unavailable", () => {
const proc = process as ProcessWithAvailableMemory;
const original = proc.availableMemory;
freememMock.mockReturnValue(64);
try {
for (const invalid of [0, Number.NaN, -1]) {
proc.availableMemory = vi.fn(() => invalid);
expect(getAvailableMemoryInfo()).toEqual({ bytes: 64, reliable: false });
}
} finally {
restoreAvailableMemory(original);
}
});
});

View File

@@ -1,6 +1,9 @@
import { describe, it, expect, beforeEach, vi } from "vitest";
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { collectSystemMetrics } from "../system-metrics.js";
type ProcessWithAvailableMemory = NodeJS.Process & { availableMemory?: () => number };
const originalAvailableMemory = (process as ProcessWithAvailableMemory).availableMemory;
const { checkDiskSpaceMock, cpusMock, totalmemMock, freememMock, uptimeMock } = vi.hoisted(() => ({
checkDiskSpaceMock: vi.fn(),
cpusMock: vi.fn(),
@@ -23,6 +26,7 @@ vi.mock("node:os", () => ({
describe("collectSystemMetrics", () => {
beforeEach(() => {
vi.clearAllMocks();
(process as ProcessWithAvailableMemory).availableMemory = vi.fn(() => 6_000);
cpusMock.mockReturnValue([
{
times: {
@@ -44,6 +48,14 @@ describe("collectSystemMetrics", () => {
});
});
afterEach(() => {
if (originalAvailableMemory) {
(process as ProcessWithAvailableMemory).availableMemory = originalAvailableMemory;
} else {
Reflect.deleteProperty(process as ProcessWithAvailableMemory, "availableMemory");
}
});
it("returns a valid SystemMetrics object", async () => {
const metrics = await collectSystemMetrics();
@@ -86,6 +98,28 @@ describe("collectSystemMetrics", () => {
expect(new Date(metrics.reportedAt).toISOString()).toBe(metrics.reportedAt);
});
it("uses process.availableMemory instead of macOS-shaped freemem for memoryUsed", async () => {
totalmemMock.mockReturnValue(16_000_000_000);
freememMock.mockReturnValue(200_000_000);
(process as ProcessWithAvailableMemory).availableMemory = vi.fn(() => 10_000_000_000);
const metrics = await collectSystemMetrics();
expect(metrics.memoryTotal).toBe(16_000_000_000);
expect(metrics.memoryUsed).toBe(6_000_000_000);
expect(metrics.memoryUsed).not.toBe(15_800_000_000);
});
it("falls back to freemem for memoryUsed when process.availableMemory is absent", async () => {
totalmemMock.mockReturnValue(16_000);
freememMock.mockReturnValue(6_000);
Reflect.deleteProperty(process as ProcessWithAvailableMemory, "availableMemory");
const metrics = await collectSystemMetrics();
expect(metrics.memoryUsed).toBe(10_000);
});
it("passes dbPath through to check-disk-space", async () => {
const customPath = "/tmp/kb-metrics-db";

View File

@@ -0,0 +1,32 @@
import * as os from "node:os";
export interface AvailableMemoryReading {
bytes: number;
/** False when only `os.freemem()` was available — unusable as a pressure signal. */
reliable: boolean;
}
/**
* FNXC:SystemMetrics 2026-06-21-13:01:
* macOS `os.freemem()` only counts truly-free pages and excludes inactive/cached pages that the OS can reclaim on demand, so total-minus-freemem over-reports memory used and can make an idle Mac look ~95–99% full.
* Prefer Node's `process.availableMemory()` because it reports OS-available memory and matches user-facing tools such as Activity Monitor. Keep the `os.freemem()` fallback for runtimes without the API, but flag it unreliable so pressure-sensitive callers can refuse to act on a garbage ratio.
*/
export function getAvailableMemoryInfo(): AvailableMemoryReading {
const processFn = (process as unknown as { availableMemory?: () => number }).availableMemory;
if (typeof processFn === "function") {
try {
const value = processFn.call(process);
if (Number.isFinite(value) && value > 0) {
return { bytes: value, reliable: true };
}
} catch {
// Fall through to the compatibility path below.
}
}
return { bytes: os.freemem(), reliable: false };
}
export function getAvailableMemoryBytes(): number {
return getAvailableMemoryInfo().bytes;
}

View File

@@ -1326,6 +1326,7 @@ export type { CentralCoreEvents } from "./central-core.js";
export { CentralDatabase, createCentralDatabase, getDefaultCentralDbPath } from "./central-db.js";
export { NodeConnection } from "./node-connection.js";
export { NodeDiscovery } from "./node-discovery.js";
export { getAvailableMemoryBytes, getAvailableMemoryInfo, type AvailableMemoryReading } from "./available-memory.js";
export { collectSystemMetrics } from "./system-metrics.js";
export { getAppVersion, parseSemver } from "./app-version.js";
export { DockerClientService } from "./docker-client.js";

View File

@@ -1,5 +1,6 @@
import { cpus, totalmem, freemem, uptime as getUptime } from "node:os";
import { cpus, totalmem, uptime as getUptime } from "node:os";
import * as checkDiskSpaceModule from "check-disk-space";
import { getAvailableMemoryBytes } from "./available-memory.js";
import type { SystemMetrics } from "./types.js";
const checkDiskSpace = ((checkDiskSpaceModule as { default?: unknown }).default ??
@@ -40,7 +41,11 @@ export async function collectSystemMetrics(dbPath?: string): Promise<SystemMetri
const cpuUsage = totalTime > 0 ? (busyTime / totalTime) * 100 : 0;
const memoryTotal = toNonNegative(totalmem());
const rawMemoryUsed = memoryTotal - toNonNegative(freemem());
/*
FNXC:SystemMetrics 2026-06-21-13:01:
Mesh metrics must compute used memory from OS-available memory instead of raw `freemem()` so macOS inactive/cache pages are not incorrectly reported as used.
*/
const rawMemoryUsed = memoryTotal - toNonNegative(getAvailableMemoryBytes());
const memoryUsed = clamp(rawMemoryUsed, 0, memoryTotal);
const diskPath = dbPath ?? process.cwd();

View File

@@ -490,6 +490,33 @@ describe("GET /api/system-stats", () => {
mockExecFile.mockClear();
});
it("reports systemFreeMem from process.availableMemory instead of macOS-shaped freemem", async () => {
type ProcessWithAvailableMemory = NodeJS.Process & { availableMemory?: () => number };
const proc = process as ProcessWithAvailableMemory;
const originalAvailableMemory = proc.availableMemory;
proc.availableMemory = vi.fn(() => 10_000_000_000);
try {
vi.spyOn(AgentStore.prototype, "init").mockResolvedValue(undefined);
vi.spyOn(AgentStore.prototype, "listAgents").mockResolvedValue([]);
const store = createMockStore({
listTasks: vi.fn().mockResolvedValue([]),
getFusionDir: vi.fn().mockReturnValue("/fake/default"),
});
const res = await GET(buildApp(store), "/api/system-stats");
expect(res.status).toBe(200);
expect(res.body.systemStats.systemFreeMem).toBe(10_000_000_000);
} finally {
if (originalAvailableMemory) {
proc.availableMemory = originalAvailableMemory;
} else {
Reflect.deleteProperty(proc, "availableMemory");
}
}
});
it("includes last auto-kill timestamp when available in global settings", async () => {
const store = createMockStore({
listTasks: vi.fn().mockResolvedValue([]),

View File

@@ -23,6 +23,7 @@ import {
RoutineStore,
discoverPiExtensions,
findVitestProcessIds,
getAvailableMemoryBytes,
getFusionAgentDir,
getLegacyPiAgentDir,
isWebhookTrigger,
@@ -1586,7 +1587,11 @@ export function createApiRoutes(store: TaskStore, options?: ServerOptions): Rout
loadAvg: [load[0] ?? 0, load[1] ?? 0, load[2] ?? 0],
cpuCount: os.cpus().length,
systemTotalMem: os.totalmem(),
systemFreeMem: os.freemem(),
/*
FNXC:CommandCenter 2026-06-21-13:01:
The public `systemFreeMem` field carries OS-available memory so SystemStatsArea derives Memory Used from reclaimable-aware bytes and matches Activity Monitor on macOS.
*/
systemFreeMem: getAvailableMemoryBytes(),
pid: process.pid,
nodeVersion: process.version,
platform: `${process.platform}/${process.arch}`,

View File

@@ -20,6 +20,19 @@ import {
const onPosix = process.platform !== "win32";
const itPosix = onPosix ? it : it.skip;
function isProcessAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/**
* Tests for runVerificationCommand - the core verification execution logic.
* These tests validate basic command execution, output capture, and error handling.
@@ -393,6 +406,11 @@ describe("runVerificationCommand", { timeout: 30000 }, () => {
const leakedPid = Number.parseInt(result.stdout.trim(), 10);
expect(Number.isFinite(leakedPid)).toBe(true);
expect(result.timedOut).toBe(false);
for (let i = 0; i < 15 && isProcessAlive(leakedPid); i++) {
await sleep(100);
}
expect(isProcessAlive(leakedPid)).toBe(false);
});
it("escalates non-timeout process-group reaping with fake timers", () => {