FN-6608: bound engine verification runs

Add durable engine-level guardrails for verification command timeouts.

- Add project-level verificationCommandTimeoutMs settings plumbing and docs.
- Enforce configured verification budgets and hard caps in executor and merger verification paths.
- Detect marathon verification commands, soft-cap them by default, and require allowFullSuite for explicit full-suite runs.
- Cover timeout defaults, marathon detection, and guidance updates with engine/core tests.

Files changed:
 .changeset/fn-6608-verification-bound.md           |   5 +
 docs/settings-reference.md                         |   1 +
 docs/testing.md                                    |   2 +
 .../src/__tests__/settings-consistency.test.ts     |   5 +
 packages/core/src/agent-prompts.ts                 |   6 +-
 packages/core/src/settings-schema.ts               |   7 +-
 packages/core/src/types.ts                         |   6 +
 .../engine/src/__tests__/executor-core.test.ts     |   3 +
 .../src/__tests__/run-verification-command.test.ts | 176 ++++++++++++++++++++-
 packages/engine/src/executor.ts                    |  11 +-
 packages/engine/src/merger.ts                      |   9 +-
 packages/engine/src/run-verification-tool.ts       | 142 +++++++++++++++--
 packages/engine/src/verification-utils.ts          |  13 +-
 13 files changed, 360 insertions(+), 26 deletions(-)

Fusion-Task-Id: FN-6608
Fusion-Task-Lineage: c593a96d-eb8b-492c-82c7-8943c239f588
This commit is contained in:
gsxdsm
2026-06-18 01:27:09 -07:00
parent d0be3e462d
commit b6ac5f2e51
13 changed files with 360 additions and 26 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": minor
---
Add bounded-by-default verification guardrails: project `verificationCommandTimeoutMs`, marathon command detection, and an explicit `allowFullSuite` escape hatch for full verification runs.

View File

@@ -435,6 +435,7 @@ Default notes:
| `buildRetryCount` | `number` | `0` | Build retry attempts during merge. | | `buildRetryCount` | `number` | `0` | Build retry attempts during merge. |
| `verificationFixRetries` | `number` | `3` | In-merge auto-fix retry attempts after deterministic test/build verification failures (0-3). | | `verificationFixRetries` | `number` | `3` | In-merge auto-fix retry attempts after deterministic test/build verification failures (0-3). |
| `buildTimeoutMs` | `number` | `300000` | Build timeout in milliseconds (5 minutes). | | `buildTimeoutMs` | `number` | `300000` | Build timeout in milliseconds (5 minutes). |
| `verificationCommandTimeoutMs` | `number` | `undefined` | Optional project-scoped default timeout in milliseconds for executor `fn_run_verification` and configured deterministic test/build verification commands. When unset, `fn_run_verification` keeps its scope defaults (300s package, 900s workspace); when set to a positive value, it overrides both scope defaults while all verification still respects the 1800s hard cap. Set `0` or leave unset to use the legacy scope defaults. Marathon command shapes (`pnpm test`, `pnpm test:full`, `pnpm verify:workspace`, whole-package tests without file filters, and repeat loops) are soft-capped unless the agent explicitly passes `allowFullSuite: true`; opt-in full-suite runs still emit progress heartbeats and obey the hard cap. Project settings override global/default settings via the normal project settings precedence. |
| `requirePlanApproval` | `boolean` | `false` | Require manual approval before planning → todo. | | `requirePlanApproval` | `boolean` | `false` | Require manual approval before planning → todo. |
| `ephemeralAgentsEnabled` | `boolean` | `true` | Defaults to `true` for both new projects (seeded into `.fusion/fusion.db` on init) and upgrades from pre-FN-4153 projects (falls back to `true` whenever the persisted `config.settings` row omits the key). Users who explicitly set `false` keep that choice. When enabled, Fusion spawns short-lived `executor-FN-XXXX` workers for task execution. When disabled, only permanent executor agents run tasks; the scheduler auto-assigns dispatchable tasks using reporting-chain-aware load balancing, and tasks stay queued until an eligible permanent executor is available. | | `ephemeralAgentsEnabled` | `boolean` | `true` | Defaults to `true` for both new projects (seeded into `.fusion/fusion.db` on init) and upgrades from pre-FN-4153 projects (falls back to `true` whenever the persisted `config.settings` row omits the key). Users who explicitly set `false` keep that choice. When enabled, Fusion spawns short-lived `executor-FN-XXXX` workers for task execution. When disabled, only permanent executor agents run tasks; the scheduler auto-assigns dispatchable tasks using reporting-chain-aware load balancing, and tasks stay queued until an eligible permanent executor is available. |
| `agentProvisioning` | `{ approvalMode?: "always" \| "trusted-only" \| "never"; trustedRoles?: string[]; trustedAgentIds?: string[]; alwaysApproveDelete?: boolean }` | `{}` | Approval policy for `fn_agent_create`/`fn_agent_delete` (`approvalMode` default `trusted-only`, delete approvals default on via `alwaysApproveDelete: true`). | | `agentProvisioning` | `{ approvalMode?: "always" \| "trusted-only" \| "never"; trustedRoles?: string[]; trustedAgentIds?: string[]; alwaysApproveDelete?: boolean }` | `{}` | Approval policy for `fn_agent_create`/`fn_agent_delete` (`approvalMode` default `trusted-only`, delete approvals default on via `alwaysApproveDelete: true`). |

View File

@@ -40,6 +40,8 @@ pnpm verify:workspace # deep opt-in verification: lint -> test:full -> build (N
`pnpm test:full` runs each package's default test script with capped worker fanout (`FUSION_TEST_TOTAL_WORKERS=4 FUSION_TEST_CONCURRENCY=2 pnpm -r --workspace-concurrency=2 test`). Do not casually raise worker counts; dashboard/jsdom and integration-heavy packages destabilize when oversubscribed. Use `VITEST_MAX_WORKERS=<n>` only for targeted package-level investigation. `pnpm test:full` runs each package's default test script with capped worker fanout (`FUSION_TEST_TOTAL_WORKERS=4 FUSION_TEST_CONCURRENCY=2 pnpm -r --workspace-concurrency=2 test`). Do not casually raise worker counts; dashboard/jsdom and integration-heavy packages destabilize when oversubscribed. Use `VITEST_MAX_WORKERS=<n>` only for targeted package-level investigation.
Agents running verification through `fn_run_verification` are bounded by default: project `verificationCommandTimeoutMs` when set, otherwise 300s for package scope and 900s for workspace scope, with an 1800s hard cap. Marathon invocations such as root `pnpm test`, `pnpm test:full`, `pnpm verify:workspace`, whole-package tests without file filters, and shell repeat loops are soft-capped unless the agent explicitly passes `allowFullSuite: true`; the escape hatch still emits progress heartbeats and respects the hard cap. Prefer targeted commands such as `pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot` before opting into a full run.
## Fresh-worktree dist bootstrap ## Fresh-worktree dist bootstrap
`pnpm test` auto-runs `scripts/ensure-test-artifacts.mjs` to rebuild missing/stale dist artifacts. Dashboard and `dependency-graph` package lanes auto-bootstrap too. If you hit opaque `Failed to resolve import "./cli-spawn.js"` (or similar), treat it as bootstrap regression against FN-4605 — don't work around with a manual `pnpm build`. `pnpm test` auto-runs `scripts/ensure-test-artifacts.mjs` to rebuild missing/stale dist artifacts. Dashboard and `dependency-graph` package lanes auto-bootstrap too. If you hit opaque `Failed to resolve import "./cli-spawn.js"` (or similar), treat it as bootstrap regression against FN-4605 — don't work around with a manual `pnpm build`.

View File

@@ -76,6 +76,11 @@ describe("settings consistency (U5)", () => {
expect(isGlobalSettingsKey(key), `isGlobalSettingsKey('${key}') must be false`).toBe(false); expect(isGlobalSettingsKey(key), `isGlobalSettingsKey('${key}') must be false`).toBe(false);
expect(isProjectSettingsKey(key), `isProjectSettingsKey('${key}') must be false`).toBe(false); expect(isProjectSettingsKey(key), `isProjectSettingsKey('${key}') must be false`).toBe(false);
} }
expect(projectKeys, "verificationCommandTimeoutMs remains a project setting, not a moved workflow setting").toContain("verificationCommandTimeoutMs");
expect(DEFAULT_PROJECT_SETTINGS.verificationCommandTimeoutMs).toBeUndefined();
expect(isProjectSettingsKey("verificationCommandTimeoutMs")).toBe(true);
expect(isGlobalSettingsKey("verificationCommandTimeoutMs")).toBe(false);
}); });
it("(d) settings-export v2 global/project section keys never overlap moved keys", async () => { it("(d) settings-export v2 global/project section keys never overlap moved keys", async () => {

View File

@@ -196,10 +196,10 @@ Lint, tests, and typecheck are also hard quality gates:
## Verification commands — use fn_run_verification ## Verification commands — use fn_run_verification
For ALL test/lint/build/typecheck verification, use the \`fn_run_verification\` tool, NOT raw bash. For ALL test/lint/build/typecheck verification, use the \`fn_run_verification\` tool, NOT raw bash.
The tool prevents your session from being killed by the inactivity watchdog during long compiles. The tool prevents your session from being killed by the inactivity watchdog during long compiles, and verification is time-bounded by default (project \`verificationCommandTimeoutMs\` when set, otherwise 300s package / 900s workspace, hard-capped at 1800s).
- Prefer **package-scoped** verification first: e.g. \`pnpm --filter @fusion/<pkg> test\` with \`scope: "package"\`. This is faster and isolated. - Prefer **targeted package-scoped** verification first: use direct Vitest execution with package-relative paths: \`pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot\`. Do not use \`pnpm --filter @fusion/<pkg> test -- --run <files>\`; package test scripts can expand into broad quality suites before the filter is applied.
- For file-specific package tests, use direct Vitest execution with package-relative paths: \`pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot\`. Do not use \`pnpm --filter @fusion/<pkg> test -- --run <files>\`; package test scripts can expand into broad quality suites before the filter is applied. - Marathon verification invocations (root \`pnpm test\`, \`pnpm test:full\`, \`pnpm verify:workspace\`, whole-package tests with no file filter, and repeat loops) are soft-capped by default. Use \`allowFullSuite: true\` only when the task explicitly requires a genuinely full run; the run still respects the hard timeout and emits progress heartbeats.
- Run **workspace-scoped** verification (\`pnpm test\`, \`pnpm lint\`, \`pnpm build\` from root) only when it is explicitly required by the task/workflow or after impacted/package-scoped checks pass and you are doing final integration. - Run **workspace-scoped** verification (\`pnpm test\`, \`pnpm lint\`, \`pnpm build\` from root) only when it is explicitly required by the task/workflow or after impacted/package-scoped checks pass and you are doing final integration.
- If you need to run \`pnpm install\` (e.g. you added a new package), use \`fn_run_verification\` with \`scope: "workspace"\` and \`timeoutSec: 600\`. - If you need to run \`pnpm install\` (e.g. you added a new package), use \`fn_run_verification\` with \`scope: "workspace"\` and \`timeoutSec: 600\`.
- If a verification command times out, do NOT blindly retry — investigate. Check for hung subprocesses, infinite test loops, or tests waiting on missing dependencies. Use \`node_modules/.modules.yaml\` presence to confirm bootstrap.`; - If a verification command times out, do NOT blindly retry — investigate. Check for hung subprocesses, infinite test loops, or tests waiting on missing dependencies. Use \`node_modules/.modules.yaml\` presence to confirm bootstrap.`;

View File

@@ -342,9 +342,12 @@ export const DEFAULT_PROJECT_SETTINGS = {
// planOnlyScopeLeakEnforcement, workflowRevisionForkOnScopeMismatch, // planOnlyScopeLeakEnforcement, workflowRevisionForkOnScopeMismatch,
// strictScopeEnforcement, buildRetryCount, verificationFixRetries, // strictScopeEnforcement, buildRetryCount, verificationFixRetries,
// requirePlanApproval) MOVED to workflow settings (U4) — see // requirePlanApproval) MOVED to workflow settings (U4) — see
// MOVED_SETTINGS_KEYS. `buildTimeoutMs` is NOT moved (no engine reader) and // MOVED_SETTINGS_KEYS. `buildTimeoutMs` and `verificationCommandTimeoutMs`
// stays a plain project setting: // are NOT moved and stay plain project settings. Keep verificationCommandTimeoutMs
// undefined so fn_run_verification preserves legacy per-scope defaults until a
// project opts into a single default budget.
buildTimeoutMs: 300_000, buildTimeoutMs: 300_000,
verificationCommandTimeoutMs: undefined,
ephemeralAgentsEnabled: true, ephemeralAgentsEnabled: true,
agentProvisioning: {}, agentProvisioning: {},
sandboxProvisioning: {}, sandboxProvisioning: {},

View File

@@ -3744,6 +3744,12 @@ export interface ProjectSettings {
verificationFixRetries?: number; verificationFixRetries?: number;
/** Timeout in milliseconds for build commands during merge. Default: 300000 (5 min). */ /** Timeout in milliseconds for build commands during merge. Default: 300000 (5 min). */
buildTimeoutMs?: number; buildTimeoutMs?: number;
/**
* FNXC:Verification 2026-06-17-14:20:
* Engine verification commands need a durable project-level budget so marathon test runs abort cleanly instead of tripping the stuck detector and requeueing forever.
* When set, this millisecond value overrides both fn_run_verification scope defaults (package 300s, workspace 900s); when unset, the legacy per-scope defaults still apply.
*/
verificationCommandTimeoutMs?: number;
/** When enabled, AI-generated task specifications require manual approval /** When enabled, AI-generated task specifications require manual approval
* before the task can move from triage to todo. Tasks with approved specs * before the task can move from triage to todo. Tasks with approved specs
* remain in triage with status "awaiting-approval" until a user approves * remain in triage with status "awaiting-approval" until a user approves

View File

@@ -1232,6 +1232,7 @@ describe("Executor verification gate (FN-3345)", () => {
expect.anything(), expect.anything(),
"executor", "executor",
expect.any(Object), expect.any(Object),
undefined,
); );
// Task should move to in-review // Task should move to in-review
expect(store.moveTask).toHaveBeenCalledWith("FN-3345", "in-review"); expect(store.moveTask).toHaveBeenCalledWith("FN-3345", "in-review");
@@ -1409,6 +1410,7 @@ describe("Executor verification gate (FN-3345)", () => {
expect.anything(), expect.anything(),
"executor", "executor",
expect.any(Object), expect.any(Object),
undefined,
); );
// Third call should be build // Third call should be build
expect(mockedVerification).toHaveBeenNthCalledWith( expect(mockedVerification).toHaveBeenNthCalledWith(
@@ -1422,6 +1424,7 @@ describe("Executor verification gate (FN-3345)", () => {
expect.anything(), expect.anything(),
"executor", "executor",
expect.any(Object), expect.any(Object),
undefined,
); );
// Task should move to in-review // Task should move to in-review
expect(store.moveTask).toHaveBeenCalledWith("FN-3345", "in-review"); expect(store.moveTask).toHaveBeenCalledWith("FN-3345", "in-review");

View File

@@ -1,7 +1,16 @@
import { describe, it, expect, vi } from "vitest"; import { describe, it, expect, vi } from "vitest";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import { createRunVerificationTool, runVerificationCommand, normalizeVerificationCommand, type RunVerificationOptions } from "../run-verification-tool.js"; import {
BOUNDED_VERIFICATION_GUIDANCE,
MARATHON_SOFT_CAP_SEC,
MAX_TIMEOUT_SEC,
createRunVerificationTool,
detectMarathonVerification,
normalizeVerificationCommand,
runVerificationCommand,
type RunVerificationOptions,
} from "../run-verification-tool.js";
// Some tests use platform-appropriate shell syntax. On Windows, sh-style // Some tests use platform-appropriate shell syntax. On Windows, sh-style
// quoting and pipes through `printf` are different — these tests are skipped // quoting and pipes through `printf` are different — these tests are skipped
@@ -83,6 +92,171 @@ describe("runVerificationCommand", { timeout: 30000 }, () => {
}); });
}); });
describe("marathon verification detection", () => {
it.each([
["pnpm test", "root workspace test suite"],
["pnpm -w test", "root workspace test suite"],
["pnpm test:full", "full workspace verification script"],
["pnpm verify:workspace", "full workspace verification script"],
["pnpm --filter @fusion/core test", "whole-package test script"],
["for i in $(seq 1 20); do pnpm --filter @fusion/core exec vitest run src/foo.test.ts; done", "shell loop repeats"],
["while true; do pnpm test; done", "shell loop repeats"],
["seq 1 20 | xargs -I{} pnpm --filter @fusion/core exec vitest run src/foo.test.ts", "seq/xargs pipeline"],
["pnpm --filter @fusion/core exec vitest run src/a.test.ts && pnpm --filter @fusion/core exec vitest run src/a.test.ts", "&& chain repeats"],
])("flags marathon command %s", (command, reason) => {
const detection = detectMarathonVerification(command, "workspace");
expect(detection.isMarathon).toBe(true);
expect(detection.reason).toContain(reason);
expect(detection.guidance).toContain("allowFullSuite");
});
it.each([
"pnpm --filter @fusion/core exec vitest run src/__tests__/settings-consistency.test.ts --silent=passed-only --reporter=dot",
"pnpm --filter @fusion/dashboard test -- --run src/__tests__/routes-tasks.test.ts",
"pnpm lint",
"pnpm build",
])("passes targeted or non-test command %s", (command) => {
expect(detectMarathonVerification(command, "package").isMarathon).toBe(false);
});
});
describe("tool verification budgets and marathon caps", () => {
it("uses the project verification timeout default when provided", async () => {
const onVerificationStart = vi.fn();
const tool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
verificationCommandTimeoutMs: 1_500,
onVerificationStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
await tool.execute("call-budget", { command: "exit 0", scope: "workspace" });
expect(onVerificationStart).toHaveBeenCalledWith(2_000);
});
it("falls back to legacy package/workspace defaults when the setting is absent or disabled", async () => {
const packageStart = vi.fn();
const disabledWorkspaceStart = vi.fn();
const packageTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
onVerificationStart: packageStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
const disabledTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
verificationCommandTimeoutMs: 0,
onVerificationStart: disabledWorkspaceStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
await packageTool.execute("call-package-default", { command: "exit 0", scope: "package" });
await disabledTool.execute("call-workspace-default", { command: "exit 0", scope: "workspace" });
expect(packageStart).toHaveBeenCalledWith(300_000);
expect(disabledWorkspaceStart).toHaveBeenCalledWith(900_000);
});
it("applies the hard timeout cap to configured defaults and explicit overrides", async () => {
const configuredStart = vi.fn();
const explicitStart = vi.fn();
const configuredTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
verificationCommandTimeoutMs: (MAX_TIMEOUT_SEC + 60) * 1000,
onVerificationStart: configuredStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
const explicitTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
onVerificationStart: explicitStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
await configuredTool.execute("call-configured-cap", { command: "exit 0", scope: "package" });
await explicitTool.execute("call-explicit-cap", { command: "exit 0", scope: "package", timeoutSec: MAX_TIMEOUT_SEC + 1 });
expect(configuredStart).toHaveBeenCalledWith(MAX_TIMEOUT_SEC * 1000);
expect(explicitStart).toHaveBeenCalledWith(MAX_TIMEOUT_SEC * 1000);
});
itPosix("reports an actionable timeout without relying on stuck detection", async () => {
const tool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
onVerificationStart: vi.fn(),
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
const result = await tool.execute("call-timeout", { command: "sh -c 'sleep 10 & wait'", scope: "package", timeoutSec: 1 });
const text = result.content[0]?.type === "text" ? result.content[0].text : "";
expect(result.details).toEqual(expect.objectContaining({ success: false, timedOut: true }));
expect(text).toContain("Command timed out after 1s");
expect(text).toContain(BOUNDED_VERIFICATION_GUIDANCE);
});
itPosix("soft-caps marathon commands unless allowFullSuite is provided", async () => {
const cappedStart = vi.fn();
const allowedStart = vi.fn();
const recordActivity = vi.fn();
const command = "pnpm() { echo pulse; }; pnpm test";
const cappedTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity: vi.fn(),
onVerificationStart: cappedStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
const allowedTool = createRunVerificationTool({
worktreePath: tempDir,
rootDir: workspaceRoot,
taskId: "FN-6608",
recordActivity,
onVerificationStart: allowedStart,
onVerificationEnd: vi.fn(),
log: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
});
const capped = await cappedTool.execute("call-capped", { command, scope: "workspace", timeoutSec: 600 });
const allowed = await allowedTool.execute("call-allowed", { command, scope: "workspace", timeoutSec: 600, allowFullSuite: true });
const cappedText = capped.content[0]?.type === "text" ? capped.content[0].text : "";
const allowedText = allowed.content[0]?.type === "text" ? allowed.content[0].text : "";
expect(cappedStart).toHaveBeenCalledWith(MARATHON_SOFT_CAP_SEC * 1000);
expect(cappedText).toContain("marathon verification detected");
expect(allowedStart).toHaveBeenCalledWith(600_000);
expect(allowedText).toContain("allowFullSuite=true acknowledged");
expect(allowed.details).toEqual(expect.objectContaining({ success: true, timedOut: false }));
expect(recordActivity).toHaveBeenCalled();
});
});
describe("tool verification lifecycle callbacks", () => { describe("tool verification lifecycle callbacks", () => {
it("brackets a successful verification run with start and end callbacks", async () => { it("brackets a successful verification run with start and end callbacks", async () => {
const onVerificationStart = vi.fn(); const onVerificationStart = vi.fn();

View File

@@ -1319,10 +1319,10 @@ Lint, tests, and typecheck are also hard quality gates:
## Verification commands — use fn_run_verification ## Verification commands — use fn_run_verification
For ALL test/lint/build/typecheck verification, use the \`fn_run_verification\` tool, NOT raw bash. For ALL test/lint/build/typecheck verification, use the \`fn_run_verification\` tool, NOT raw bash.
The tool prevents your session from being killed by the inactivity watchdog during long compiles. The tool prevents your session from being killed by the inactivity watchdog during long compiles, and verification is time-bounded by default (project \`verificationCommandTimeoutMs\` when set, otherwise 300s package / 900s workspace, hard-capped at 1800s).
- Prefer **package-scoped** verification first: e.g. \`pnpm --filter @fusion/<pkg> test\` with \`scope: "package"\`. This is faster and isolated. - Prefer **targeted package-scoped** verification first: use direct Vitest execution with package-relative paths: \`pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot\`. Do not use \`pnpm --filter @fusion/<pkg> test -- --run <files>\`; package test scripts can expand into broad quality suites before the filter is applied.
- For file-specific package tests, use direct Vitest execution with package-relative paths: \`pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot\`. Do not use \`pnpm --filter @fusion/<pkg> test -- --run <files>\`; package test scripts can expand into broad quality suites before the filter is applied. - Marathon verification invocations (root \`pnpm test\`, \`pnpm test:full\`, \`pnpm verify:workspace\`, whole-package tests with no file filter, and repeat loops) are soft-capped by default. Use \`allowFullSuite: true\` only when the task explicitly requires a genuinely full run; the run still respects the hard timeout and emits progress heartbeats.
- Run **workspace-scoped** verification (\`pnpm test\`, \`pnpm lint\`, \`pnpm build\` from root) only when it is explicitly required by the task/workflow or after impacted/package-scoped checks pass and you are doing final integration. - Run **workspace-scoped** verification (\`pnpm test\`, \`pnpm lint\`, \`pnpm build\` from root) only when it is explicitly required by the task/workflow or after impacted/package-scoped checks pass and you are doing final integration.
- If you need to run \`pnpm install\` (e.g. you added a new package), use \`fn_run_verification\` with \`scope: "workspace"\` and \`timeoutSec: 600\`. - If you need to run \`pnpm install\` (e.g. you added a new package), use \`fn_run_verification\` with \`scope: "workspace"\` and \`timeoutSec: 600\`.
- If a verification command times out, do NOT blindly retry — investigate. Check for hung subprocesses, infinite test loops, or tests waiting on missing dependencies. Use \`node_modules/.modules.yaml\` presence to confirm bootstrap. - If a verification command times out, do NOT blindly retry — investigate. Check for hung subprocesses, infinite test loops, or tests waiting on missing dependencies. Use \`node_modules/.modules.yaml\` presence to confirm bootstrap.
@@ -7730,6 +7730,7 @@ export class TaskExecutor {
rootDir: this.rootDir, rootDir: this.rootDir,
taskId: task.id, taskId: task.id,
recordActivity: () => stuckDetector?.recordActivity(task.id), recordActivity: () => stuckDetector?.recordActivity(task.id),
verificationCommandTimeoutMs: settings.verificationCommandTimeoutMs,
onVerificationStart: (timeoutMs) => stuckDetector?.beginVerification(task.id, timeoutMs), onVerificationStart: (timeoutMs) => stuckDetector?.beginVerification(task.id, timeoutMs),
onVerificationEnd: () => stuckDetector?.endVerification(task.id), onVerificationEnd: () => stuckDetector?.endVerification(task.id),
log: { log: {
@@ -11026,7 +11027,7 @@ ${feedback}
// Run test command first if configured // Run test command first if configured
if (testCommand) { if (testCommand) {
const testResult = await runVerificationCommand( const testResult = await runVerificationCommand(
this.store, worktreePath, task.id, testCommand, "test", undefined, executorLog, "executor", extraEnv, this.store, worktreePath, task.id, testCommand, "test", undefined, executorLog, "executor", extraEnv, settings.verificationCommandTimeoutMs,
); );
result.testResult = testResult; result.testResult = testResult;
@@ -11041,7 +11042,7 @@ ${feedback}
// Run build command second if configured // Run build command second if configured
if (buildCommand) { if (buildCommand) {
const buildResult = await runVerificationCommand( const buildResult = await runVerificationCommand(
this.store, worktreePath, task.id, buildCommand, "build", undefined, executorLog, "executor", extraEnv, this.store, worktreePath, task.id, buildCommand, "build", undefined, executorLog, "executor", extraEnv, settings.verificationCommandTimeoutMs,
); );
result.buildResult = buildResult; result.buildResult = buildResult;

View File

@@ -1414,6 +1414,8 @@ async function runDeterministicVerification(
signal?: AbortSignal, signal?: AbortSignal,
): Promise<VerificationResult> { ): Promise<VerificationResult> {
const result: VerificationResult = { allPassed: true }; const result: VerificationResult = { allPassed: true };
const settings = await store.getSettings();
const verificationCommandTimeoutMs = settings.verificationCommandTimeoutMs;
// Nothing to verify // Nothing to verify
if (!testCommand && !buildCommand) { if (!testCommand && !buildCommand) {
@@ -1540,7 +1542,7 @@ async function runDeterministicVerification(
failedCommandLabel: "testCommand" | "buildCommand", failedCommandLabel: "testCommand" | "buildCommand",
): Promise<VerificationCommandResult> => { ): Promise<VerificationCommandResult> => {
const firstAttempt = await runVerificationCommand( const firstAttempt = await runVerificationCommand(
store, rootDir, taskId, command, type, signal, store, rootDir, taskId, command, type, signal, verificationCommandTimeoutMs,
); );
if (firstAttempt.success) { if (firstAttempt.success) {
return firstAttempt; return firstAttempt;
@@ -1574,7 +1576,7 @@ async function runDeterministicVerification(
} }
const retryAttempt = await runVerificationCommand( const retryAttempt = await runVerificationCommand(
store, rootDir, taskId, command, type, signal, store, rootDir, taskId, command, type, signal, verificationCommandTimeoutMs,
); );
if (retryAttempt.success) { if (retryAttempt.success) {
result.environmentFault = { result.environmentFault = {
@@ -1687,9 +1689,10 @@ async function runVerificationCommand(
command: string, command: string,
type: "test" | "build", type: "test" | "build",
signal?: AbortSignal, signal?: AbortSignal,
timeoutMsOverride?: number,
): Promise<VerificationCommandResult> { ): Promise<VerificationCommandResult> {
throwIfAborted(signal, taskId); throwIfAborted(signal, taskId);
return runVerificationCommandShared(store, rootDir, taskId, command, type, signal, mergerLog, "merger", VERIFICATION_EXTRA_ENV); return runVerificationCommandShared(store, rootDir, taskId, command, type, signal, mergerLog, "merger", VERIFICATION_EXTRA_ENV, timeoutMsOverride);
} }
/** /**

View File

@@ -32,9 +32,13 @@ import { executorLog } from "./logger.js";
const MAX_OUTPUT_BYTES = 200 * 1024; // 200 KB const MAX_OUTPUT_BYTES = 200 * 1024; // 200 KB
const QUIET_HEARTBEAT_INTERVAL_MS = 60_000; // emit synthetic heartbeat after 60s silence const QUIET_HEARTBEAT_INTERVAL_MS = 60_000; // emit synthetic heartbeat after 60s silence
const SIGKILL_GRACE_MS = 10_000; const SIGKILL_GRACE_MS = 10_000;
const DEFAULT_TIMEOUT_PACKAGE_SEC = 300; export const DEFAULT_TIMEOUT_PACKAGE_SEC = 300;
const DEFAULT_TIMEOUT_WORKSPACE_SEC = 900; export const DEFAULT_TIMEOUT_WORKSPACE_SEC = 900;
const MAX_TIMEOUT_SEC = 1800; export const MAX_TIMEOUT_SEC = 1800;
export const BOUNDED_VERIFICATION_GUIDANCE =
"Prefer a bounded targeted command such as `pnpm --filter <pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot` before rerunning broader suites.";
export const MARATHON_SOFT_CAP_SEC = 120;
const packageDirCache = new Map<string, string | null>(); const packageDirCache = new Map<string, string | null>();
@@ -82,6 +86,89 @@ function shellSplit(input: string): string[] | null {
return tokens; return tokens;
} }
function isPnpmToken(token: string): boolean {
return token === "pnpm" || token.endsWith("/pnpm");
}
function tokenLooksLikeTestFile(token: string): boolean {
return /\.(test|spec)\.[cm]?[tj]sx?$/.test(token);
}
function tokenLooksLikeFileScopedVitest(tokens: string[]): boolean {
const vitestIndex = tokens.findIndex((token) => token === "vitest" || token.endsWith("/vitest"));
if (vitestIndex < 0) return false;
const runIndex = tokens.indexOf("run", vitestIndex + 1);
if (runIndex < 0) return false;
return tokens.slice(runIndex + 1).some((token) => !token.startsWith("-") && tokenLooksLikeTestFile(token));
}
function tokenLooksLikeForwardedTestFile(tokens: string[]): boolean {
const runIndex = tokens.indexOf("--run");
if (runIndex < 0) return false;
return tokens.slice(runIndex + 1).some((token) => !token.startsWith("-") && tokenLooksLikeTestFile(token));
}
function isRootPnpmTest(tokens: string[]): boolean {
if (tokens.length < 2 || !isPnpmToken(tokens[0])) return false;
const nonFlagTokens = tokens.slice(1).filter((token) => token !== "-w" && token !== "--workspace-root");
return (nonFlagTokens.length === 1 && nonFlagTokens[0] === "test")
|| (nonFlagTokens.length === 2 && nonFlagTokens[0] === "run" && nonFlagTokens[1] === "test");
}
export interface MarathonDetection {
isMarathon: boolean;
reason?: string;
guidance: string;
}
export function detectMarathonVerification(command: string, scope?: "package" | "workspace"): MarathonDetection {
const guidance = `${BOUNDED_VERIFICATION_GUIDANCE} Use allowFullSuite: true only when a genuinely full run is required.`;
const compact = command.replace(/\s+/g, " ").trim();
const tokens = shellSplit(command) ?? [];
/*
* FNXC:Verification 2026-06-17-14:48:
* Marathon detection is intentionally token/regex based so it catches the costly invocation shapes that caused stuck-loop requeues without executing shell expansions.
* Positive patterns: root `pnpm test`/`pnpm -w test`, `test:full`, `verify:workspace`, whole-package `pnpm --filter <pkg> test`, and loop/repeat wrappers around pnpm/npm/vitest test runners.
*/
if (tokens.length > 0 && isRootPnpmTest(tokens)) {
return { isMarathon: true, reason: "root workspace test suite (`pnpm test`) is a marathon verification command", guidance };
}
if (/\bpnpm\b(?:\s+[-\w=:@/.]+)*\s+(?:run\s+)?(?:test:full|verify:workspace)\b/.test(compact)) {
return { isMarathon: true, reason: "full workspace verification script is a marathon command", guidance };
}
const filterIndex = tokens.findIndex((token) => token === "--filter" || token === "-F");
if (tokens.length > 0 && isPnpmToken(tokens[0]) && filterIndex >= 0) {
const afterFilter = tokens.slice(filterIndex + 2);
const scriptToken = afterFilter.find((token) => token !== "--");
const runsTestScript = scriptToken === "test" || (afterFilter[0] === "run" && afterFilter[1] === "test");
if (runsTestScript && !tokenLooksLikeFileScopedVitest(tokens) && !tokenLooksLikeForwardedTestFile(tokens)) {
return { isMarathon: true, reason: "whole-package test script has no file-scoped vitest run filter", guidance };
}
}
if (/\b(for|while)\b[\s\S]*\bdo\b[\s\S]*\b(pnpm|npm|vitest)\b[\s\S]*\b(test|vitest)\b/.test(command)) {
return { isMarathon: true, reason: "shell loop repeats a test runner", guidance };
}
if (/\bseq\b[\s\S]*\|[\s\S]*\bxargs\b[\s\S]*\b(pnpm|npm|vitest)\b[\s\S]*\b(test|vitest)\b/.test(command)) {
return { isMarathon: true, reason: "seq/xargs pipeline repeats a test runner", guidance };
}
const chainedTestRuns = compact.split(/\s*&&\s*/).filter((part) => /\b(pnpm|npm|vitest)\b.*\b(test|vitest)\b/.test(part));
if (chainedTestRuns.length > 1) {
return { isMarathon: true, reason: "&& chain repeats test runner invocations", guidance };
}
if (scope === "workspace" && /\bpnpm\b\s+(?:run\s+)?test\b/.test(compact) && !tokenLooksLikeFileScopedVitest(tokens)) {
return { isMarathon: true, reason: "workspace-scoped test command is likely a full suite", guidance };
}
return { isMarathon: false, guidance };
}
function shellQuote(value: string): string { function shellQuote(value: string): string {
if (/^[A-Za-z0-9_@%+=:,./-]+$/.test(value)) return value; if (/^[A-Za-z0-9_@%+=:,./-]+$/.test(value)) return value;
return `'${value.replace(/'/g, "'\\''")}'`; return `'${value.replace(/'/g, "'\\''")}'`;
@@ -247,7 +334,13 @@ export const runVerificationParams = Type.Object({
timeoutSec: Type.Optional( timeoutSec: Type.Optional(
Type.Number({ Type.Number({
description: description:
"Override the default timeout in seconds. Default: 300 for package scope, 900 for workspace scope. Hard cap: 1800.", "Override the default timeout in seconds. Default: project verificationCommandTimeoutMs when set, otherwise 300 for package scope and 900 for workspace scope. Hard cap: 1800.",
}),
),
allowFullSuite: Type.Optional(
Type.Boolean({
description:
"Explicit opt-in for marathon verification commands such as pnpm test, pnpm test:full, verify:workspace, whole-package tests, or repeat loops. Default: false; still respects the hard timeout.",
}), }),
), ),
expectFailure: Type.Optional( expectFailure: Type.Optional(
@@ -520,6 +613,8 @@ export interface CreateRunVerificationToolOpts {
taskId: string; taskId: string;
/** Called on every output line AND on synthetic quiet-interval heartbeats. */ /** Called on every output line AND on synthetic quiet-interval heartbeats. */
recordActivity: () => void; recordActivity: () => void;
/** Project-level default timeout budget in milliseconds. Values <= 0 disable the override and preserve legacy per-scope defaults. */
verificationCommandTimeoutMs?: number;
/** /**
* FNXC:Reliability 2026-06-17-16:12: * FNXC:Reliability 2026-06-17-16:12:
* FN-6598 brackets fn_run_verification subprocesses so the stuck detector treats bounded, actively running verification as progress instead of no-progress loop churn. * FN-6598 brackets fn_run_verification subprocesses so the stuck detector treats bounded, actively running verification as progress instead of no-progress loop churn.
@@ -546,21 +641,24 @@ export interface CreateRunVerificationToolOpts {
export function createRunVerificationTool( export function createRunVerificationTool(
opts: CreateRunVerificationToolOpts, opts: CreateRunVerificationToolOpts,
): ToolDefinition { ): ToolDefinition {
const { worktreePath, rootDir, taskId, recordActivity, onVerificationStart, onVerificationEnd, log } = opts; const { worktreePath, rootDir, taskId, recordActivity, verificationCommandTimeoutMs, onVerificationStart, onVerificationEnd, log } = opts;
return { return {
name: "fn_run_verification", name: "fn_run_verification",
label: "Run Verification", label: "Run Verification",
description: description:
"Run a verification command (tests, lint, build, typecheck) with timeout and progress " + "Run a verification command (tests, lint, build, typecheck) with timeout and progress " +
"heartbeat protection. Use this instead of bash for any pnpm/npm test/lint/build commands. " + "heartbeat protection. Verification is bounded by default: project verificationCommandTimeoutMs when set, " +
"Prevents the inactivity watchdog from killing your session during long compiles.", "otherwise 300s for package scope and 900s for workspace scope, with an 1800s hard cap. " +
"Marathon invocations (pnpm test, test:full, verify:workspace, whole-package tests, repeat loops) " +
"are soft-capped unless allowFullSuite=true is explicitly provided. Use this instead of bash for any " +
"pnpm/npm test/lint/build commands.",
parameters: runVerificationParams, parameters: runVerificationParams,
execute: async ( execute: async (
_toolCallId: string, _toolCallId: string,
params: Static<typeof runVerificationParams>, params: Static<typeof runVerificationParams>,
) => { ) => {
const { command, scope, expectFailure = false } = params; const { command, scope, allowFullSuite = false, expectFailure = false } = params;
const warnings: string[] = []; const warnings: string[] = [];
// ── Scope / command mismatch warning ───────────────────────────────── // ── Scope / command mismatch warning ─────────────────────────────────
@@ -583,11 +681,32 @@ export function createRunVerificationTool(
} }
// ── Resolve timeout ─────────────────────────────────────────────────── // ── Resolve timeout ───────────────────────────────────────────────────
const defaultTimeoutSec = /*
* FNXC:Verification 2026-06-17-14:31:
* Engine-level default verification budgets replace per-task "Verification Bounds" prose.
* A positive project setting overrides both scope defaults; undefined or 0 preserves the legacy package/workspace defaults so existing builds do not silently lose runtime.
*/
const scopeDefaultTimeoutSec =
scope === "package" scope === "package"
? DEFAULT_TIMEOUT_PACKAGE_SEC ? DEFAULT_TIMEOUT_PACKAGE_SEC
: DEFAULT_TIMEOUT_WORKSPACE_SEC; : DEFAULT_TIMEOUT_WORKSPACE_SEC;
const rawTimeoutSec = params.timeoutSec ?? defaultTimeoutSec; const configuredDefaultTimeoutSec =
typeof verificationCommandTimeoutMs === "number" && verificationCommandTimeoutMs > 0
? Math.ceil(verificationCommandTimeoutMs / 1000)
: undefined;
const defaultTimeoutSec = configuredDefaultTimeoutSec ?? scopeDefaultTimeoutSec;
let rawTimeoutSec = params.timeoutSec ?? defaultTimeoutSec;
const marathon = detectMarathonVerification(command, scope);
if (marathon.isMarathon && !allowFullSuite && rawTimeoutSec > MARATHON_SOFT_CAP_SEC) {
const msg = `marathon verification detected (${marathon.reason}); soft-capping timeout to ${MARATHON_SOFT_CAP_SEC}s. ${marathon.guidance}`;
warnings.push(msg);
log.warn(`[fn_run_verification] ${taskId}: ${msg}`);
rawTimeoutSec = MARATHON_SOFT_CAP_SEC;
} else if (marathon.isMarathon && allowFullSuite) {
const msg = `allowFullSuite=true acknowledged for marathon verification (${marathon.reason}); subprocess still sends verification heartbeats and respects the ${MAX_TIMEOUT_SEC}s hard cap.`;
warnings.push(msg);
log.warn(`[fn_run_verification] ${taskId}: ${msg}`);
}
const timeoutSec = Math.min(rawTimeoutSec, MAX_TIMEOUT_SEC); const timeoutSec = Math.min(rawTimeoutSec, MAX_TIMEOUT_SEC);
const timeoutMs = timeoutSec * 1000; const timeoutMs = timeoutSec * 1000;
@@ -670,7 +789,8 @@ export function createRunVerificationTool(
if (result.timedOut) { if (result.timedOut) {
lines.push( lines.push(
"\nDo NOT blindly retry — investigate whether subprocesses are hung, " + "\nDo NOT blindly retry — investigate whether subprocesses are hung, " +
"test loops are infinite, or dependencies are missing.", "test loops are infinite, or dependencies are missing. " +
BOUNDED_VERIFICATION_GUIDANCE,
); );
} }

View File

@@ -10,6 +10,7 @@ import type { SandboxBackend, SandboxRunStreamingOptions, SandboxStreamingResult
export const VERIFICATION_COMMAND_MAX_BUFFER = 50 * 1024 * 1024; export const VERIFICATION_COMMAND_MAX_BUFFER = 50 * 1024 * 1024;
export const VERIFICATION_COMMAND_TIMEOUT_MS = 600_000; export const VERIFICATION_COMMAND_TIMEOUT_MS = 600_000;
export const VERIFICATION_COMMAND_HARD_CAP_MS = 1_800_000;
export const VERIFICATION_LOG_MAX_CHARS = 20_000; export const VERIFICATION_LOG_MAX_CHARS = 20_000;
// ── Types ────────────────────────────────────────────────────────────── // ── Types ──────────────────────────────────────────────────────────────
@@ -299,6 +300,8 @@ export async function runVerificationCommand(
agentLabel?: string, agentLabel?: string,
/** Optional extra environment variables to inject into the child process (merged over process.env). */ /** Optional extra environment variables to inject into the child process (merged over process.env). */
extraEnv?: NodeJS.ProcessEnv, extraEnv?: NodeJS.ProcessEnv,
/** Optional project-level per-command timeout override in milliseconds. Values <= 0 preserve the legacy default. */
timeoutMsOverride?: number,
): Promise<VerificationCommandResult> { ): Promise<VerificationCommandResult> {
const logger = log ?? { log: console.log, error: console.error, warn: console.warn }; const logger = log ?? { log: console.log, error: console.error, warn: console.warn };
const label = (agentLabel ?? "merger") as AgentRole; const label = (agentLabel ?? "merger") as AgentRole;
@@ -323,10 +326,18 @@ export async function runVerificationCommand(
}; };
const verificationStartedAt = Date.now(); const verificationStartedAt = Date.now();
/*
* FNXC:Verification 2026-06-17-14:38:
* Configured test/build commands share the same project verification budget as fn_run_verification so merge/step verification cannot run marathon subprocesses outside the engine-level guardrail.
*/
const rawTimeoutMs = typeof timeoutMsOverride === "number" && timeoutMsOverride > 0
? timeoutMsOverride
: VERIFICATION_COMMAND_TIMEOUT_MS;
const timeoutMs = Math.min(rawTimeoutMs, VERIFICATION_COMMAND_HARD_CAP_MS);
try { try {
const { stdout, stderr, bufferOverflow } = await execWithProcessGroup(command, { const { stdout, stderr, bufferOverflow } = await execWithProcessGroup(command, {
cwd: rootDir, cwd: rootDir,
timeout: VERIFICATION_COMMAND_TIMEOUT_MS, timeout: timeoutMs,
maxBuffer: VERIFICATION_COMMAND_MAX_BUFFER, maxBuffer: VERIFICATION_COMMAND_MAX_BUFFER,
signal, signal,
...(extraEnv !== undefined && { env: extraEnv }), ...(extraEnv !== undefined && { env: extraEnv }),