fix: quiet per-poll scheduler hold-release and routing log spam
Both lines fired on every scheduler poll while nothing changed: a held card re-attempts release each sweep, and every dispatch candidate logged its resolved node. On a busy board that filled the operator log pane with "Hold release for FN-XXXX deferred" and "routed to node=local" within seconds, burying real scheduler events. Add a Logger.debug() level, off by default and opted into per subsystem via FUSION_DEBUG, and demote both lines to it. Routing to a remote node stays at info since it explains where work actually went; only the local default is demoted. Lines reporting a real transition (capacity rejection, racing sweep, release failure) are untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
7
.changeset/quiet-scheduler-steady-state-logs.md
Normal file
7
.changeset/quiet-scheduler-steady-state-logs.md
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
summary: Quiet repetitive scheduler hold-release and task-routing lines that flooded the engine log pane.
|
||||||
|
category: fix
|
||||||
|
dev: Adds `Logger.debug()` in `packages/engine/src/logger.ts`, gated per subsystem by `FUSION_DEBUG` (`1`/`true`/`all`/`*`, or a comma-separated prefix list). Demotes `Hold release for <id> deferred — no reservable slot` and local-only `Task <id> routed to node=local` to debug; remote routing stays at info. See `docs/diagnostics.md`.
|
||||||
@@ -1,5 +1,26 @@
|
|||||||
# Diagnostics
|
# Diagnostics
|
||||||
|
|
||||||
|
## Debug-level engine logs (`FUSION_DEBUG`)
|
||||||
|
|
||||||
|
Engine subsystem loggers (`createLogger` in `packages/engine/src/logger.ts`) expose a `debug()` level for steady-state per-poll chatter. It is **off by default** so the TUI log pane and engine stderr show state *changes* rather than the scheduler reprinting its resting state every poll.
|
||||||
|
|
||||||
|
Opt in per subsystem with the logger prefix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FUSION_DEBUG=scheduler # one subsystem
|
||||||
|
FUSION_DEBUG=scheduler,merger # several
|
||||||
|
FUSION_DEBUG=1 # everything (also: true, all, *)
|
||||||
|
```
|
||||||
|
|
||||||
|
The variable is re-read per call, so it can be toggled on a long-lived process without recreating loggers. Debug lines emit under the `info` severity marker and render like any other info line.
|
||||||
|
|
||||||
|
Currently debug-gated:
|
||||||
|
|
||||||
|
- `Task <id> routed to node=local (source=local)` — routing to a **remote** node stays at info; only the local default is demoted.
|
||||||
|
- `Hold release for <id> deferred — no reservable slot for <column>` — being at capacity is the expected steady state, not an event.
|
||||||
|
|
||||||
|
Guidance for new log sites: if a line repeats on every scheduler poll while nothing changed, it belongs at `debug()`. Anything reporting a transition, a rejection, or something needing operator action stays at `log()`/`warn()`/`error()`.
|
||||||
|
|
||||||
## Goal injection diagnostics (`[goal-injection]`)
|
## Goal injection diagnostics (`[goal-injection]`)
|
||||||
|
|
||||||
Executor, heartbeat, and planning runs emit one goal-injection diagnostic with outcome `applied`, `no-goals`, or `disabled-or-failed`.
|
Executor, heartbeat, and planning runs emit one goal-injection diagnostic with outcome `applied`, `no-goals`, or `disabled-or-failed`.
|
||||||
|
|||||||
63
packages/engine/src/__tests__/logger-debug-gating.test.ts
Normal file
63
packages/engine/src/__tests__/logger-debug-gating.test.ts
Normal file
@@ -0,0 +1,63 @@
|
|||||||
|
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||||
|
import { createLogger } from "../logger.js";
|
||||||
|
|
||||||
|
/*
|
||||||
|
FNXC:EngineDiagnostics 2026-07-15-12:55:
|
||||||
|
Guards the contract that keeps steady-state scheduler chatter out of the operator log pane: `debug()` is silent unless the subsystem opted in via `FUSION_DEBUG`, and `log()` is never gated.
|
||||||
|
A regression either way is invisible in normal use — it either re-floods the TUI or silently swallows real events — so it is asserted rather than eyeballed.
|
||||||
|
*/
|
||||||
|
describe("createLogger debug gating", () => {
|
||||||
|
afterEach(() => {
|
||||||
|
delete process.env.FUSION_DEBUG;
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("suppresses debug output when FUSION_DEBUG is unset", () => {
|
||||||
|
const spy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
createLogger("scheduler").debug("steady-state chatter");
|
||||||
|
expect(spy).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("emits debug output for a subsystem named in the FUSION_DEBUG list", () => {
|
||||||
|
process.env.FUSION_DEBUG = "scheduler,merger";
|
||||||
|
const spy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
createLogger("scheduler").debug("steady-state chatter");
|
||||||
|
expect(spy).toHaveBeenCalledOnce();
|
||||||
|
expect(spy.mock.calls[0]?.[0]).toContain("[scheduler] steady-state chatter");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("suppresses debug output for a subsystem absent from the FUSION_DEBUG list", () => {
|
||||||
|
process.env.FUSION_DEBUG = "merger";
|
||||||
|
const spy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
createLogger("scheduler").debug("steady-state chatter");
|
||||||
|
expect(spy).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each(["1", "true", "all", "*"])("emits debug output for every subsystem when FUSION_DEBUG=%s", (value) => {
|
||||||
|
process.env.FUSION_DEBUG = value;
|
||||||
|
const spy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
createLogger("scheduler").debug("steady-state chatter");
|
||||||
|
expect(spy).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("re-reads FUSION_DEBUG per call so toggling does not require a new logger", () => {
|
||||||
|
const log = createLogger("scheduler");
|
||||||
|
const spy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
log.debug("first");
|
||||||
|
expect(spy).not.toHaveBeenCalled();
|
||||||
|
process.env.FUSION_DEBUG = "scheduler";
|
||||||
|
log.debug("second");
|
||||||
|
expect(spy).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("never gates log/warn/error on FUSION_DEBUG", () => {
|
||||||
|
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||||
|
const warnSpy = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||||
|
const log = createLogger("scheduler");
|
||||||
|
log.log("real event");
|
||||||
|
log.warn("real warning");
|
||||||
|
log.error("real failure");
|
||||||
|
expect(errorSpy).toHaveBeenCalledTimes(2);
|
||||||
|
expect(warnSpy).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -486,8 +486,13 @@ async function issueRelease(
|
|||||||
if (targetIsProcessing && deps.reserveSlot) {
|
if (targetIsProcessing && deps.reserveSlot) {
|
||||||
reservation = await deps.reserveSlot(task, target);
|
reservation = await deps.reserveSlot(task, target);
|
||||||
if (!reservation) {
|
if (!reservation) {
|
||||||
// Semaphore/worktree exhausted — reservation-first means no move at all.
|
/*
|
||||||
schedulerLog.log(`Hold release for ${task.id} deferred — no reservable slot for ${target}`);
|
Semaphore/worktree exhausted — reservation-first means no move at all.
|
||||||
|
|
||||||
|
FNXC:WorkflowScheduling 2026-07-15-12:55:
|
||||||
|
A held card re-attempts release on every sweep, so a full board reprinted this line per task per poll and buried real scheduler events. Being at capacity is the expected steady state, not an event: debug-only (`FUSION_DEBUG=scheduler`).
|
||||||
|
*/
|
||||||
|
schedulerLog.debug(`Hold release for ${task.id} deferred — no reservable slot for ${target}`);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,6 +18,11 @@
|
|||||||
|
|
||||||
export interface Logger {
|
export interface Logger {
|
||||||
log(message: string, ...args: unknown[]): void;
|
log(message: string, ...args: unknown[]): void;
|
||||||
|
/**
|
||||||
|
* Steady-state per-poll chatter. Suppressed unless the subsystem is opted in
|
||||||
|
* via `FUSION_DEBUG` (see `isDebugEnabled`).
|
||||||
|
*/
|
||||||
|
debug(message: string, ...args: unknown[]): void;
|
||||||
warn(message: string, ...args: unknown[]): void;
|
warn(message: string, ...args: unknown[]): void;
|
||||||
error(message: string, ...args: unknown[]): void;
|
error(message: string, ...args: unknown[]): void;
|
||||||
}
|
}
|
||||||
@@ -29,6 +34,22 @@ function withSeverityMarker(level: "info" | "warn" | "error", payload: string):
|
|||||||
return `${LOG_LEVEL_MARKER_PREFIX}${level}${LOG_LEVEL_MARKER_SUFFIX}${payload}`;
|
return `${LOG_LEVEL_MARKER_PREFIX}${level}${LOG_LEVEL_MARKER_SUFFIX}${payload}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
FNXC:EngineDiagnostics 2026-07-15-12:55:
|
||||||
|
Operators read the TUI log pane to see what the engine is *doing*; per-poll steady-state chatter (routine node routing, capacity-deferred hold releases) drowned real events out — a 1000-line buffer held only a few seconds of scheduler polls.
|
||||||
|
Such lines move to `debug()`, off by default and opted in per subsystem via `FUSION_DEBUG` (`FUSION_DEBUG=1`/`all` for everything, or a comma-separated prefix list like `FUSION_DEBUG=scheduler,merger`).
|
||||||
|
Anything that reports a state *change* or needs operator action stays on `log()`/`warn()`/`error()`.
|
||||||
|
*/
|
||||||
|
function isDebugEnabled(prefix: string): boolean {
|
||||||
|
const raw = process.env.FUSION_DEBUG?.trim();
|
||||||
|
if (!raw) return false;
|
||||||
|
if (raw === "1" || raw === "true" || raw === "all" || raw === "*") return true;
|
||||||
|
return raw
|
||||||
|
.split(",")
|
||||||
|
.map((entry) => entry.trim())
|
||||||
|
.includes(prefix);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Create a structured logger that prefixes every message with `[prefix]`.
|
* Create a structured logger that prefixes every message with `[prefix]`.
|
||||||
*
|
*
|
||||||
@@ -40,6 +61,11 @@ function withSeverityMarker(level: "info" | "warn" | "error", payload: string):
|
|||||||
* The logger prepends an internal control-character severity marker
|
* The logger prepends an internal control-character severity marker
|
||||||
* so dashboard TUI console-capture can preserve info/warn/error
|
* so dashboard TUI console-capture can preserve info/warn/error
|
||||||
* semantics even when `log()` is transported via `console.error`.
|
* semantics even when `log()` is transported via `console.error`.
|
||||||
|
*
|
||||||
|
* `debug()` is gated on `FUSION_DEBUG` and re-reads the env var per
|
||||||
|
* call so tests and long-lived processes can toggle it without
|
||||||
|
* re-creating loggers. When enabled it emits under the `info` marker,
|
||||||
|
* so TUI console-capture needs no new severity to render it.
|
||||||
*/
|
*/
|
||||||
export function createLogger(prefix: string): Logger {
|
export function createLogger(prefix: string): Logger {
|
||||||
const tag = `[${prefix}]`;
|
const tag = `[${prefix}]`;
|
||||||
@@ -47,6 +73,10 @@ export function createLogger(prefix: string): Logger {
|
|||||||
log(message: string, ...args: unknown[]) {
|
log(message: string, ...args: unknown[]) {
|
||||||
console.error(withSeverityMarker("info", `${tag} ${message}`), ...args);
|
console.error(withSeverityMarker("info", `${tag} ${message}`), ...args);
|
||||||
},
|
},
|
||||||
|
debug(message: string, ...args: unknown[]) {
|
||||||
|
if (!isDebugEnabled(prefix)) return;
|
||||||
|
console.error(withSeverityMarker("info", `${tag} ${message}`), ...args);
|
||||||
|
},
|
||||||
warn(message: string, ...args: unknown[]) {
|
warn(message: string, ...args: unknown[]) {
|
||||||
console.warn(withSeverityMarker("warn", `${tag} ${message}`), ...args);
|
console.warn(withSeverityMarker("warn", `${tag} ${message}`), ...args);
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ import { schedulerLog } from "./logger.js";
|
|||||||
import { type PrMonitor, type PrComment } from "./pr-monitor.js";
|
import { type PrMonitor, type PrComment } from "./pr-monitor.js";
|
||||||
import { reconcileMissionFeatureState } from "./mission-feature-sync.js";
|
import { reconcileMissionFeatureState } from "./mission-feature-sync.js";
|
||||||
import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js";
|
import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js";
|
||||||
import { resolveEffectiveNode } from "./effective-node.js";
|
import { resolveEffectiveNode, type EffectiveNode } from "./effective-node.js";
|
||||||
import { applyUnavailableNodePolicy, decideOwningNodeHandoff } from "./node-routing-policy.js";
|
import { applyUnavailableNodePolicy, decideOwningNodeHandoff } from "./node-routing-policy.js";
|
||||||
import type { NodeDispatchValidationResult } from "./node-dispatch-validation.js";
|
import type { NodeDispatchValidationResult } from "./node-dispatch-validation.js";
|
||||||
import type { MeshLeaseManager } from "./mesh-lease-manager.js";
|
import type { MeshLeaseManager } from "./mesh-lease-manager.js";
|
||||||
@@ -47,6 +47,20 @@ import { runHoldReleaseSweep, isUnplannedForExecution, type SlotReservation } fr
|
|||||||
import { moveTaskToReplanColumn } from "./replan-target.js";
|
import { moveTaskToReplanColumn } from "./replan-target.js";
|
||||||
import { evaluateParkedAgentTaskLink } from "./task-agent-sync.js";
|
import { evaluateParkedAgentTaskLink } from "./task-agent-sync.js";
|
||||||
|
|
||||||
|
/*
|
||||||
|
FNXC:WorkflowScheduling 2026-07-15-12:55:
|
||||||
|
Every dispatch pass resolves a routing node, so logging each resolution at info level reprinted `routed to node=local (source=local)` for every task on every poll and buried real scheduler events in the TUI log pane.
|
||||||
|
Local routing is the default nobody needs told about: it is debug-only (`FUSION_DEBUG=scheduler`). Remote routing stays at info because it explains where work actually went and is what an operator reaches for when a task runs on the wrong host.
|
||||||
|
*/
|
||||||
|
function logTaskRouting(taskId: string, node: EffectiveNode): void {
|
||||||
|
const message = `Task ${taskId} routed to node=${node.nodeId ?? "local"} (source=${node.source})`;
|
||||||
|
if (node.nodeId === undefined) {
|
||||||
|
schedulerLog.debug(message);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
schedulerLog.log(message);
|
||||||
|
}
|
||||||
|
|
||||||
function shouldRunWorkflowColumnScheduler(_settings: Settings): boolean {
|
function shouldRunWorkflowColumnScheduler(_settings: Settings): boolean {
|
||||||
/*
|
/*
|
||||||
FNXC:WorkflowScheduling 2026-06-22-00:00:
|
FNXC:WorkflowScheduling 2026-06-22-00:00:
|
||||||
@@ -1902,7 +1916,7 @@ export class Scheduler {
|
|||||||
|
|
||||||
// Resolve effective node for routing
|
// Resolve effective node for routing
|
||||||
let effectiveNode = resolveEffectiveNode(freshTask, settings);
|
let effectiveNode = resolveEffectiveNode(freshTask, settings);
|
||||||
schedulerLog.log(`Task ${task.id} routed to node=${effectiveNode.nodeId ?? "local"} (source=${effectiveNode.source})`);
|
logTaskRouting(task.id, effectiveNode);
|
||||||
|
|
||||||
// Enforce dispatch configuration validation before node-health fallback logic.
|
// Enforce dispatch configuration validation before node-health fallback logic.
|
||||||
if (effectiveNode.nodeId !== undefined && this.options.validateNodeDispatch) {
|
if (effectiveNode.nodeId !== undefined && this.options.validateNodeDispatch) {
|
||||||
@@ -2426,7 +2440,7 @@ export class Scheduler {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let effectiveNode = resolveEffectiveNode(freshTask, settings);
|
let effectiveNode = resolveEffectiveNode(freshTask, settings);
|
||||||
schedulerLog.log(`Task ${task.id} routed to node=${effectiveNode.nodeId ?? "local"} (source=${effectiveNode.source})`);
|
logTaskRouting(task.id, effectiveNode);
|
||||||
|
|
||||||
if (effectiveNode.nodeId !== undefined && this.options.validateNodeDispatch) {
|
if (effectiveNode.nodeId !== undefined && this.options.validateNodeDispatch) {
|
||||||
const nodeValidation = await this.options.validateNodeDispatch(effectiveNode.nodeId);
|
const nodeValidation = await this.options.validateNodeDispatch(effectiveNode.nodeId);
|
||||||
|
|||||||
Reference in New Issue
Block a user