feat(FN-4534): add auto-recovery dispatcher and pause-site wiring

Fusion-Task-Id: FN-4534
Fusion-Task-Lineage: 2b46a5df-efc6-4305-a0f2-69991e58512c
This commit is contained in:
Fusion
2026-05-14 17:47:34 -07:00
committed by gsxdsm
parent 55903924ac
commit 79eb74ca7e
12 changed files with 510 additions and 32 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
Introduce AutoRecoveryDispatcher and `ProjectSettings.autoRecovery` (mode/perClass/maxRetries) for classifier-driven recovery of reliability-layer failures. Adds new run-audit event types `auto-recovery:classify-decision`, `auto-recovery:retry-issued`, `auto-recovery:ai-session-spawned`, and `auto-recovery:pause-because-destructive-ambiguity`. Default mode preserves prior behavior; `mode: "off"` is byte-identical to legacy parking.

View File

@@ -199,6 +199,9 @@ Defaults from `DEFAULT_PROJECT_SETTINGS`; key scope from `PROJECT_SETTINGS_KEYS`
| `mergeStrategyOverlapBehavior` | `"flip-to-prefer-branch" \| "warn-only" \| "ignore"` | `"flip-to-prefer-branch"` | Safety control for `mergeConflictStrategy="smart-prefer-main"`. Before the Attempt 3 `-X ours` fallback, Fusion checks whether the task branch and recent `main` history overlap on the same files (30-commit lookback, matching the squash audit heuristics). `flip-to-prefer-branch` makes overlapping files prefer the task branch so hardening is not silently discarded (the FN-3936 class of regression). `warn-only` logs the overlap but keeps the legacy main-wins fallback. `ignore` disables the overlap guard and preserves legacy behavior exactly. | | `mergeStrategyOverlapBehavior` | `"flip-to-prefer-branch" \| "warn-only" \| "ignore"` | `"flip-to-prefer-branch"` | Safety control for `mergeConflictStrategy="smart-prefer-main"`. Before the Attempt 3 `-X ours` fallback, Fusion checks whether the task branch and recent `main` history overlap on the same files (30-commit lookback, matching the squash audit heuristics). `flip-to-prefer-branch` makes overlapping files prefer the task branch so hardening is not silently discarded (the FN-3936 class of regression). `warn-only` logs the overlap but keeps the legacy main-wins fallback. `ignore` disables the overlap guard and preserves legacy behavior exactly. |
| `postMergeAuditMode` | `"block" \| "warn" \| "off"` | `"warn"` | Controls the post-merge audit gate. **Warn** (default) logs findings and continues to auto-complete merges. **Block** is the stricter opt-in mode: it refuses auto-completion on duplicate-subject or touched-file overlap findings when you want maximum FN-3936-class drop protection. **Off** skips the audit entirely. Regardless of mode, rebase-strategy overlap-only findings are auto-cleared when deterministic merge verification has already proven the tree (FN-4333). | | `postMergeAuditMode` | `"block" \| "warn" \| "off"` | `"warn"` | Controls the post-merge audit gate. **Warn** (default) logs findings and continues to auto-complete merges. **Block** is the stricter opt-in mode: it refuses auto-completion on duplicate-subject or touched-file overlap findings when you want maximum FN-3936-class drop protection. **Off** skips the audit entirely. Regardless of mode, rebase-strategy overlap-only findings are auto-cleared when deterministic merge verification has already proven the tree (FN-4333). |
| `mergeAuditAutoRecovery` | `"deterministic-only" \| "programmatic" \| "ai-assisted" \| "off"` | `"ai-assisted"` | Controls how the engine recovers when the post-merge audit finds risks. **Deterministic only** keeps just the verified-rebase short-circuit. **Programmatic** also diffs each flagged main commit against HEAD and passes when every contribution survives. **AI-assisted** additionally lets the merger write a single restoration commit when programmatic checks find real drops, and bounces the task back to in-progress before parking. **Off** disables all recovery — failed audits park the task immediately. | | `mergeAuditAutoRecovery` | `"deterministic-only" \| "programmatic" \| "ai-assisted" \| "off"` | `"ai-assisted"` | Controls how the engine recovers when the post-merge audit finds risks. **Deterministic only** keeps just the verified-rebase short-circuit. **Programmatic** also diffs each flagged main commit against HEAD and passes when every contribution survives. **AI-assisted** additionally lets the merger write a single restoration commit when programmatic checks find real drops, and bounces the task back to in-progress before parking. **Off** disables all recovery — failed audits park the task immediately. |
| `autoRecovery.mode` | `"off" \| "deterministic-only" \| "programmatic" \| "ai-assisted"` | `"deterministic-only"` | Dispatcher mode for recoverable executor/self-healing failure classes. `"off"` is byte-identical legacy parking behavior (exact legacy `pausedReason` preserved). |
| `autoRecovery.perClass` | `Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>` | `undefined` | Optional per-class mode override map. Overrides `autoRecovery.mode` for listed classes only. Taxonomy strings follow FN-4533 design. |
| `autoRecovery.maxRetries` | `number` | `3` | Retry budget for dispatcher decisions. When `retryCount >= maxRetries`, dispatcher forces `pause` with rationale `retry-budget-exhausted`. |
### Per-task direct-merge override ### Per-task direct-merge override

View File

@@ -4,6 +4,7 @@ import {
DEFAULT_PROJECT_SETTINGS, DEFAULT_PROJECT_SETTINGS,
GLOBAL_SETTINGS_KEYS, GLOBAL_SETTINGS_KEYS,
PROJECT_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS,
normalizeAutoRecovery,
isGlobalOnlySettingsKey, isGlobalOnlySettingsKey,
isGlobalSettingsKey, isGlobalSettingsKey,
isProjectSettingsKey, isProjectSettingsKey,
@@ -136,6 +137,16 @@ describe("settings key parity", () => {
expect(DEFAULT_PROJECT_SETTINGS.workflowStepTimeoutMs).toBe(360_000); expect(DEFAULT_PROJECT_SETTINGS.workflowStepTimeoutMs).toBe(360_000);
}); });
it("defaults autoRecovery and normalizes overrides", () => {
expect(DEFAULT_PROJECT_SETTINGS.autoRecovery).toEqual({ mode: "deterministic-only", maxRetries: 3 });
expect(normalizeAutoRecovery({ mode: "off", perClass: { "branch-conflict-unrecoverable": "ai-assisted" }, maxRetries: 2 })).toEqual({
mode: "off",
perClass: { "branch-conflict-unrecoverable": "ai-assisted" },
maxRetries: 2,
});
expect(normalizeAutoRecovery({ mode: "invalid" })).toEqual({ mode: "deterministic-only", perClass: undefined, maxRetries: 3 });
});
it("defaults stale high fan-out blocker escalation age threshold", () => { it("defaults stale high fan-out blocker escalation age threshold", () => {
expect(DEFAULT_PROJECT_SETTINGS.staleHighFanoutBlockerAgeThresholdMs).toBe(2 * 60 * 60 * 1000); expect(DEFAULT_PROJECT_SETTINGS.staleHighFanoutBlockerAgeThresholdMs).toBe(2 * 60 * 60 * 1000);
expect(isProjectSettingsKey("staleHighFanoutBlockerAgeThresholdMs")).toBe(true); expect(isProjectSettingsKey("staleHighFanoutBlockerAgeThresholdMs")).toBe(true);

View File

@@ -1,5 +1,5 @@
export { COLUMNS, DEFAULT_COLUMN, isColumn, normalizeColumn, COLUMN_LABELS, COLUMN_DESCRIPTIONS, VALID_TRANSITIONS, DEFAULT_SETTINGS, DEFAULT_GLOBAL_SETTINGS, DEFAULT_PROJECT_SETTINGS, GLOBAL_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS, isGlobalSettingsKey, isProjectSettingsKey, resolvePersistAgentThinkingLog, THINKING_LEVELS, THEME_MODES, COLOR_THEMES, WORKFLOW_STEP_TEMPLATES, AGENT_PERMISSIONS, PERMANENT_AGENT_ACTION_CATEGORIES, AGENT_PERMISSION_POLICY_ACTION_CATEGORIES, AGENT_PROVISIONING_APPROVAL_MODES, AGENT_PERMISSION_POLICY_PRESET_IDS, LEGACY_AGENT_PERMISSION_POLICY_ACTION_CATEGORY_ALIASES, APPROVAL_REQUEST_STATUSES, APPROVAL_REQUEST_AUDIT_EVENT_TYPES, normalizeApprovalRequestActionCategory, isValidApprovalRequestTransition, agentToConfigSnapshot, diffConfigSnapshots, isEphemeralAgent, hasAgentIdentity, CheckoutConflictError, DEFAULT_HEARTBEAT_PROCEDURE_PATH, getDefaultHeartbeatProcedurePath, EXECUTION_MODES, DEFAULT_EXECUTION_MODE, TASK_PRIORITIES, DEFAULT_TASK_PRIORITY, HIGH_FANOUT_BLOCKER_TODO_THRESHOLD, STALE_HIGH_FANOUT_BLOCKER_AGE_THRESHOLD_MS, DASHBOARD_USER_ID, normalizeMessageParticipant, validateMessageMetadata, validateDockerNodeConfig, sanitizeDockerNodeConfigForResponse, normalizeMergeConflictStrategy, normalizeMergeStrategyOverlapBehavior, normalizePostMergeAuditMode, POST_MERGE_AUDIT_MODES, normalizeMergeAuditAutoRecovery, MERGE_AUDIT_AUTO_RECOVERY_MODES, buildResearchDocumentKey, REPO_OVERRIDE_RE, SHARED_STATE_SNAPSHOT_VERSION } from "./types.js"; export { COLUMNS, DEFAULT_COLUMN, isColumn, normalizeColumn, COLUMN_LABELS, COLUMN_DESCRIPTIONS, VALID_TRANSITIONS, DEFAULT_SETTINGS, DEFAULT_GLOBAL_SETTINGS, DEFAULT_PROJECT_SETTINGS, GLOBAL_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS, isGlobalSettingsKey, isProjectSettingsKey, resolvePersistAgentThinkingLog, THINKING_LEVELS, THEME_MODES, COLOR_THEMES, WORKFLOW_STEP_TEMPLATES, AGENT_PERMISSIONS, PERMANENT_AGENT_ACTION_CATEGORIES, AGENT_PERMISSION_POLICY_ACTION_CATEGORIES, AGENT_PROVISIONING_APPROVAL_MODES, AGENT_PERMISSION_POLICY_PRESET_IDS, LEGACY_AGENT_PERMISSION_POLICY_ACTION_CATEGORY_ALIASES, APPROVAL_REQUEST_STATUSES, APPROVAL_REQUEST_AUDIT_EVENT_TYPES, normalizeApprovalRequestActionCategory, isValidApprovalRequestTransition, agentToConfigSnapshot, diffConfigSnapshots, isEphemeralAgent, hasAgentIdentity, CheckoutConflictError, DEFAULT_HEARTBEAT_PROCEDURE_PATH, getDefaultHeartbeatProcedurePath, EXECUTION_MODES, DEFAULT_EXECUTION_MODE, TASK_PRIORITIES, DEFAULT_TASK_PRIORITY, HIGH_FANOUT_BLOCKER_TODO_THRESHOLD, STALE_HIGH_FANOUT_BLOCKER_AGE_THRESHOLD_MS, DASHBOARD_USER_ID, normalizeMessageParticipant, validateMessageMetadata, validateDockerNodeConfig, sanitizeDockerNodeConfigForResponse, normalizeMergeConflictStrategy, normalizeMergeStrategyOverlapBehavior, normalizePostMergeAuditMode, POST_MERGE_AUDIT_MODES, normalizeMergeAuditAutoRecovery, MERGE_AUDIT_AUTO_RECOVERY_MODES, normalizeAutoRecovery, AUTO_RECOVERY_MODES, buildResearchDocumentKey, REPO_OVERRIDE_RE, SHARED_STATE_SNAPSHOT_VERSION } from "./types.js";
export type { Column, IssueInfo, IssueState, TaskSourceIssue, PrInfo, PrStatus, Task, TaskTokenUsage, TaskAttachment, TaskComment, TaskCommentInput, TaskDocument, TaskDocumentRevision, TaskDocumentCreateInput, TaskDocumentWithTask, TaskCreateInput, MeshReplicatedTaskCreatePayload, MeshReplicatedTaskApplyResult, TaskSource, SourceType, TaskDetail, RetrySummary, InboxTask, TodoList, TodoItem, TodoListCreateInput, TodoListUpdateInput, TodoItemCreateInput, TodoItemUpdateInput, TodoListWithItems, AgentLogEntry, AgentLogType, AgentRole, BoardConfig, DistributedTaskIdReserveInput, DistributedTaskIdReserveResult, DistributedTaskIdCommitInput, DistributedTaskIdCommitResult, DistributedTaskIdAbortInput, DistributedTaskIdAbortResult, DistributedTaskIdStateInput, DistributedTaskIdStateResult, AutostashOrphanRecord, AutostashOutcome, MergeDetails, MergeResult, MergeConflictStrategy, CanonicalMergeConflictStrategy, MergeStrategyOverlapBehavior, PostMergeAuditMode, MergeAuditAutoRecoveryMode, DirectMergeCommitStrategy, Settings, GlobalSettings, ProjectSettings, WebSearchBackend, ResearchEnabledSources, ResearchGlobalDefaults, ResearchProjectLimits, ResearchProjectSettings, EvalFollowUpPolicy, EvalProjectSettings, ResolvedEvalSettings, SettingsScope, DaemonTokenSettings, TaskStep, StepStatus, TaskLogEntry, RunMutationContext, ActivityLogEntry, ActivityEventType, ThinkingLevel, ThemeMode, ColorTheme, ExecutionMode, TaskPriority, UnavailableNodePolicy, PlanningQuestion, PlanningSummary, PlanningResponse, PlanningQuestionType, ArchivedTaskEntry, BatchStatusRequest, BatchStatusResponse, BatchStatusEntry, BatchStatusResult, GithubIssueAction, ModelPreset, WorkflowStep, WorkflowStepMode, WorkflowStepGateMode, WorkflowStepPhase, WorkflowStepInput, WorkflowStepResult, WorkflowStepTemplate, Agent, OrgTreeNode, AgentState, AgentDetail, AgentCreateInput, AgentUpdateInput, AgentApiKey, AgentApiKeyCreateResult, AgentCapability, AgentPromptTemplate, AgentPromptsConfig, AgentPermission, PermanentAgentActionCategory, PermanentAgentSensitiveActionCategory, PermanentAgentGatingContext, AgentPermissionPolicy, AgentPermissionPolicyRules, AgentPermissionPolicyActionCategory, AgentProvisioningApprovalMode, LegacyAgentPermissionPolicyActionCategory, ApprovalRequestActionCategoryInput, ApprovalRequestActionCategory, AgentPermissionPolicyDisposition, AgentPermissionPolicyPresetId, ApprovalRequestStatus, ApprovalRequestAuditEventType, ApprovalRequestActorSnapshot, ApprovalRequestTargetAction, ApprovalRequestAuditEvent, ApprovalRequest, ApprovalRequestCreateInput, ApprovalRequestDecisionInput, ApprovalRequestCompletionInput, ApprovalRequestListInput, TaskAssignSource, AgentAccessState, AgentHeartbeatConfig, AgentBudgetConfig, AgentBudgetStatus, InstructionsBundleConfig, MessageResponseMode, AgentHeartbeatEvent, AgentHeartbeatRun, BlockedStateSnapshot, HeartbeatInvocationSource, AgentTaskSession, AgentRating, AgentRatingSummary, AgentRatingInput, AgentConfigSnapshot, RevisionFieldDiff, AgentConfigRevision, AgentStats, ReflectionTrigger, ReflectionMetrics, AgentReflection, AgentPerformanceSummary, NtfyNotificationEvent, NotificationEvent, NotificationPayload, NotificationProviderConfig, CustomProvider, SteeringComment, ParticipantType, MessageType, Message, MessageCreateInput, MessageFilter, MessageMetadata, MessageReplyReference, Mailbox, CheckoutLease, RunAuditDomain, RunAuditEvent, RunAuditEventInput, RunAuditEventFilter, AgentMemoryInclusionMode, HeartbeatPromptTemplate, HeartbeatScopeDisciplineMode } from "./types.js"; export type { Column, IssueInfo, IssueState, TaskSourceIssue, PrInfo, PrStatus, Task, TaskTokenUsage, TaskAttachment, TaskComment, TaskCommentInput, TaskDocument, TaskDocumentRevision, TaskDocumentCreateInput, TaskDocumentWithTask, TaskCreateInput, MeshReplicatedTaskCreatePayload, MeshReplicatedTaskApplyResult, TaskSource, SourceType, TaskDetail, RetrySummary, InboxTask, TodoList, TodoItem, TodoListCreateInput, TodoListUpdateInput, TodoItemCreateInput, TodoItemUpdateInput, TodoListWithItems, AgentLogEntry, AgentLogType, AgentRole, BoardConfig, DistributedTaskIdReserveInput, DistributedTaskIdReserveResult, DistributedTaskIdCommitInput, DistributedTaskIdCommitResult, DistributedTaskIdAbortInput, DistributedTaskIdAbortResult, DistributedTaskIdStateInput, DistributedTaskIdStateResult, AutostashOrphanRecord, AutostashOutcome, MergeDetails, MergeResult, MergeConflictStrategy, CanonicalMergeConflictStrategy, MergeStrategyOverlapBehavior, PostMergeAuditMode, MergeAuditAutoRecoveryMode, AutoRecoveryMode, AutoRecoveryFailureClass, AutoRecoverySettings, DirectMergeCommitStrategy, Settings, GlobalSettings, ProjectSettings, WebSearchBackend, ResearchEnabledSources, ResearchGlobalDefaults, ResearchProjectLimits, ResearchProjectSettings, EvalFollowUpPolicy, EvalProjectSettings, ResolvedEvalSettings, SettingsScope, DaemonTokenSettings, TaskStep, StepStatus, TaskLogEntry, RunMutationContext, ActivityLogEntry, ActivityEventType, ThinkingLevel, ThemeMode, ColorTheme, ExecutionMode, TaskPriority, UnavailableNodePolicy, PlanningQuestion, PlanningSummary, PlanningResponse, PlanningQuestionType, ArchivedTaskEntry, BatchStatusRequest, BatchStatusResponse, BatchStatusEntry, BatchStatusResult, GithubIssueAction, ModelPreset, WorkflowStep, WorkflowStepMode, WorkflowStepGateMode, WorkflowStepPhase, WorkflowStepInput, WorkflowStepResult, WorkflowStepTemplate, Agent, OrgTreeNode, AgentState, AgentDetail, AgentCreateInput, AgentUpdateInput, AgentApiKey, AgentApiKeyCreateResult, AgentCapability, AgentPromptTemplate, AgentPromptsConfig, AgentPermission, PermanentAgentActionCategory, PermanentAgentSensitiveActionCategory, PermanentAgentGatingContext, AgentPermissionPolicy, AgentPermissionPolicyRules, AgentPermissionPolicyActionCategory, AgentProvisioningApprovalMode, LegacyAgentPermissionPolicyActionCategory, ApprovalRequestActionCategoryInput, ApprovalRequestActionCategory, AgentPermissionPolicyDisposition, AgentPermissionPolicyPresetId, ApprovalRequestStatus, ApprovalRequestAuditEventType, ApprovalRequestActorSnapshot, ApprovalRequestTargetAction, ApprovalRequestAuditEvent, ApprovalRequest, ApprovalRequestCreateInput, ApprovalRequestDecisionInput, ApprovalRequestCompletionInput, ApprovalRequestListInput, TaskAssignSource, AgentAccessState, AgentHeartbeatConfig, AgentBudgetConfig, AgentBudgetStatus, InstructionsBundleConfig, MessageResponseMode, AgentHeartbeatEvent, AgentHeartbeatRun, BlockedStateSnapshot, HeartbeatInvocationSource, AgentTaskSession, AgentRating, AgentRatingSummary, AgentRatingInput, AgentConfigSnapshot, RevisionFieldDiff, AgentConfigRevision, AgentStats, ReflectionTrigger, ReflectionMetrics, AgentReflection, AgentPerformanceSummary, NtfyNotificationEvent, NotificationEvent, NotificationPayload, NotificationProviderConfig, CustomProvider, SteeringComment, ParticipantType, MessageType, Message, MessageCreateInput, MessageFilter, MessageMetadata, MessageReplyReference, Mailbox, CheckoutLease, RunAuditDomain, RunAuditEvent, RunAuditEventInput, RunAuditEventFilter, AgentMemoryInclusionMode, HeartbeatPromptTemplate, HeartbeatScopeDisciplineMode } from "./types.js";
export { AGENT_VALID_TRANSITIONS } from "./types.js"; export { AGENT_VALID_TRANSITIONS } from "./types.js";
export { export {
resolveAgentMemoryInclusionMode, resolveAgentMemoryInclusionMode,

View File

@@ -227,6 +227,10 @@ export const DEFAULT_PROJECT_SETTINGS = {
mergeStrategyOverlapBehavior: "flip-to-prefer-branch", mergeStrategyOverlapBehavior: "flip-to-prefer-branch",
postMergeAuditMode: "warn", postMergeAuditMode: "warn",
mergeAuditAutoRecovery: "ai-assisted", mergeAuditAutoRecovery: "ai-assisted",
autoRecovery: {
mode: "deterministic-only",
maxRetries: 3,
},
workflowStepTimeoutMs: 360_000, workflowStepTimeoutMs: 360_000,
workflowStepScopeEnforcement: "block", workflowStepScopeEnforcement: "block",
planOnlyScopeLeakEnforcement: "warn", planOnlyScopeLeakEnforcement: "warn",

View File

@@ -222,6 +222,60 @@ export function normalizeMergeAuditAutoRecovery(value: unknown): MergeAuditAutoR
? (value as MergeAuditAutoRecoveryMode) ? (value as MergeAuditAutoRecoveryMode)
: "ai-assisted"; : "ai-assisted";
} }
export const AUTO_RECOVERY_MODES = ["off", "deterministic-only", "programmatic", "ai-assisted"] as const;
export type AutoRecoveryMode = (typeof AUTO_RECOVERY_MODES)[number];
export type AutoRecoveryFailureClass =
| "file-scope-invariant"
| "post-squash-audit-blocker"
| "branch-cross-contamination"
| "branch-conflict-tripwire"
| "branch-conflict-recovery-exhausted"
| "branch-conflict-unrecoverable";
export interface AutoRecoverySettings {
mode: AutoRecoveryMode;
perClass?: Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>;
maxRetries?: number;
}
export function normalizeAutoRecovery(value: unknown): AutoRecoverySettings {
const fallback: AutoRecoverySettings = { mode: "deterministic-only", maxRetries: 3 };
if (!value || typeof value !== "object") return fallback;
const candidate = value as {
mode?: unknown;
perClass?: unknown;
maxRetries?: unknown;
};
const mode = typeof candidate.mode === "string" && (AUTO_RECOVERY_MODES as readonly string[]).includes(candidate.mode)
? candidate.mode as AutoRecoveryMode
: fallback.mode;
const perClass = typeof candidate.perClass === "object" && candidate.perClass
? Object.fromEntries(
Object.entries(candidate.perClass as Record<string, unknown>)
.filter(([k, v]) => (
[
"file-scope-invariant",
"post-squash-audit-blocker",
"branch-cross-contamination",
"branch-conflict-tripwire",
"branch-conflict-recovery-exhausted",
"branch-conflict-unrecoverable",
].includes(k)
&& typeof v === "string"
&& (AUTO_RECOVERY_MODES as readonly string[]).includes(v)
)),
) as Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>
: undefined;
const maxRetries = typeof candidate.maxRetries === "number" && Number.isFinite(candidate.maxRetries)
? Math.max(0, Math.floor(candidate.maxRetries))
: fallback.maxRetries;
return { mode, perClass, maxRetries };
}
/** Policy for handling task execution when the selected node is unavailable/unhealthy. */ /** Policy for handling task execution when the selected node is unavailable/unhealthy. */
export type UnavailableNodePolicy = "block" | "fallback-local"; export type UnavailableNodePolicy = "block" | "fallback-local";
@@ -2459,6 +2513,8 @@ export interface ProjectSettings {
* - "off": disable all recovery; audit blocks immediately. * - "off": disable all recovery; audit blocks immediately.
*/ */
mergeAuditAutoRecovery?: MergeAuditAutoRecoveryMode; mergeAuditAutoRecovery?: MergeAuditAutoRecoveryMode;
/** Dispatcher-level reliability recovery policy (FN-4533/FN-4534). */
autoRecovery?: AutoRecoverySettings;
/** Wall-clock timeout (ms) for a single pre-merge workflow step's AI call. /** Wall-clock timeout (ms) for a single pre-merge workflow step's AI call.
* When a step exceeds this, the session is aborted and the executor is * When a step exceeds this, the session is aborted and the executor is
* given one shot to retry with the configured fallback model before the * given one shot to retry with the configured fallback model before the

View File

@@ -0,0 +1,86 @@
import { describe, expect, it, vi } from "vitest";
import type { AutoRecoverySettings, Task } from "@fusion/core";
import { AutoRecoveryDispatcher, type AutoRecoveryFailure } from "../auto-recovery.js";
const task = { id: "FN-1", recoveryRetryCount: 0 } as Task;
function createDispatcher() {
const database = vi.fn(async () => {});
const dispatcher = new AutoRecoveryDispatcher({
taskStore: {} as never,
auditEmitter: { database, git: vi.fn(), filesystem: vi.fn() },
});
return { dispatcher, database };
}
const classes: AutoRecoveryFailure["class"][] = [
"file-scope-invariant",
"post-squash-audit-blocker",
"branch-cross-contamination",
"branch-conflict-tripwire",
"branch-conflict-recovery-exhausted",
"branch-conflict-unrecoverable",
];
describe("auto-recovery dispatcher", () => {
it.each(classes)("mode off preserves pause contract for %s", (klass) => {
const { dispatcher } = createDispatcher();
const decision = dispatcher.classify({ class: klass, taskId: "FN-1", pausedReason: "legacy-reason" }, {
task,
retryCount: 0,
settings: { mode: "off", maxRetries: 3 },
});
expect(decision.action).toBe("pause");
expect(decision.legacyPausedReason).toBe("legacy-reason");
expect(decision.rationale).toBe("auto-recovery-disabled");
});
it("per-class override beats global mode", () => {
const { dispatcher } = createDispatcher();
const settings: AutoRecoverySettings = {
mode: "deterministic-only",
perClass: { "branch-conflict-unrecoverable": "programmatic" },
maxRetries: 3,
};
const decision = dispatcher.classify({ class: "branch-conflict-unrecoverable", taskId: "FN-1", pausedReason: "branch-conflict-unrecoverable" }, { task, retryCount: 0, settings });
expect(decision.action).toBe("retry");
});
it("forces pause on retry budget exhausted", () => {
const { dispatcher } = createDispatcher();
const decision = dispatcher.classify({ class: "branch-conflict-tripwire", taskId: "FN-1", pausedReason: "branch-conflict-tripwire" }, {
task,
retryCount: 3,
settings: { mode: "programmatic", maxRetries: 3 },
});
expect(decision.action).toBe("pause");
expect(decision.rationale).toBe("retry-budget-exhausted");
});
it("forces pause on destructive ambiguity", () => {
const { dispatcher } = createDispatcher();
const decision = dispatcher.classify({ class: "branch-cross-contamination", taskId: "FN-1", pausedReason: "branch-cross-contamination", evidence: { ownCommits: 1, foreignAttributedCommits: 1 } }, {
task,
retryCount: 0,
settings: { mode: "ai-assisted", maxRetries: 3 },
});
expect(decision.action).toBe("pause");
expect(decision.rationale).toBe("destructive-ambiguity");
});
it("dispatch falls back to pause when handler missing", async () => {
const { dispatcher, database } = createDispatcher();
const decision = await dispatcher.dispatch({ class: "branch-conflict-unrecoverable", taskId: "FN-1", pausedReason: "branch-conflict-unrecoverable" }, {
task,
retryCount: 0,
settings: { mode: "programmatic", maxRetries: 3 },
});
expect(decision.action).toBe("pause");
expect(decision.rationale).toBe("handler-not-registered");
expect(database).toHaveBeenCalledTimes(1);
expect(database.mock.calls[0]?.[0]).toMatchObject({
type: "auto-recovery:classify-decision",
metadata: expect.objectContaining({ class: "branch-conflict-unrecoverable", mode: "programmatic", retryCount: 0 }),
});
});
});

View File

@@ -0,0 +1,43 @@
import { describe, expect, it, vi } from "vitest";
import type { Task } from "@fusion/core";
import { AutoRecoveryDispatcher } from "../../auto-recovery.js";
const baseTask = { id: "FN-1", recoveryRetryCount: 0 } as Task;
describe("reliability interaction: auto-recovery dispatcher precedence", () => {
it("mode off preserves legacy pausedReason contract across wired classes", () => {
const dispatcher = new AutoRecoveryDispatcher({
taskStore: {} as never,
auditEmitter: { database: vi.fn(async () => {}), git: vi.fn(), filesystem: vi.fn() },
});
const wired = [
"branch-cross-contamination",
"branch-conflict-tripwire",
"branch-conflict-recovery-exhausted",
"branch-conflict-unrecoverable",
] as const;
for (const klass of wired) {
const decision = dispatcher.classify({ class: klass, taskId: "FN-1", pausedReason: klass }, {
task: baseTask,
retryCount: 0,
settings: { mode: "off", maxRetries: 3 },
});
expect(decision.action).toBe("pause");
expect(decision.legacyPausedReason).toBe(klass);
}
});
it("deterministic recovery success can bypass dispatcher invocation", async () => {
const classify = vi.fn();
const deterministicFastPath = vi.fn(async () => true);
if (!(await deterministicFastPath())) {
classify();
}
expect(deterministicFastPath).toHaveBeenCalledOnce();
expect(classify).not.toHaveBeenCalled();
});
});

View File

@@ -0,0 +1,159 @@
import type { AutoRecoveryFailureClass, AutoRecoveryMode, AutoRecoverySettings, Task, TaskStore } from "@fusion/core";
import { createLogger, type Logger } from "./logger.js";
import type { RunAuditor } from "./run-audit.js";
export type AutoRecoveryAction = "retry" | "spawn-ai-recovery" | "pause";
export interface AutoRecoveryFailure {
class: AutoRecoveryFailureClass;
taskId: string;
runId?: string;
pausedReason: string;
evidence?: Record<string, unknown>;
underlyingError?: Error;
}
export interface AutoRecoveryDecision {
action: AutoRecoveryAction;
rationale: string;
auditMetadata: Record<string, unknown>;
legacyPausedReason: string;
}
export interface AutoRecoveryContext {
task: Task;
retryCount: number;
settings: AutoRecoverySettings;
now?: () => Date;
}
export interface AutoRecoveryHandlers {
issueRetry?: (failure: AutoRecoveryFailure, decision: AutoRecoveryDecision, ctx: AutoRecoveryContext) => Promise<void>;
spawnAiRecovery?: (failure: AutoRecoveryFailure, decision: AutoRecoveryDecision, ctx: AutoRecoveryContext) => Promise<void>;
}
const autoRecoveryLog = createLogger("auto-recovery");
function actionForMode(mode: AutoRecoveryMode, failureClass: AutoRecoveryFailureClass): AutoRecoveryAction {
if (mode === "off" || mode === "deterministic-only") return "pause";
if (mode === "programmatic") {
if (failureClass === "file-scope-invariant" || failureClass === "post-squash-audit-blocker") return "pause";
return "retry";
}
if (mode === "ai-assisted") {
if (failureClass === "file-scope-invariant" || failureClass === "post-squash-audit-blocker") return "spawn-ai-recovery";
return "retry";
}
return "pause";
}
function isDestructiveAmbiguity(failure: AutoRecoveryFailure): boolean {
if (failure.evidence?.destructiveAmbiguity === true) return true;
const own = Number(failure.evidence?.ownCommits ?? 0);
const foreign = Number(failure.evidence?.foreignAttributedCommits ?? 0);
return own > 0 && foreign > 0;
}
export class AutoRecoveryDispatcher {
private readonly taskStore: TaskStore;
private readonly auditEmitter: RunAuditor;
private readonly handlers: AutoRecoveryHandlers;
private readonly logger: Logger;
constructor(opts: { taskStore: TaskStore; auditEmitter: RunAuditor; handlers?: AutoRecoveryHandlers; logger?: Logger }) {
this.taskStore = opts.taskStore;
this.auditEmitter = opts.auditEmitter;
this.handlers = opts.handlers ?? {};
this.logger = opts.logger ?? autoRecoveryLog;
}
classify(failure: AutoRecoveryFailure, context: AutoRecoveryContext): AutoRecoveryDecision {
if (context.settings.mode === "off") {
return {
action: "pause",
rationale: "auto-recovery-disabled",
legacyPausedReason: failure.pausedReason,
auditMetadata: { class: failure.class, mode: "off", retryCount: context.retryCount, rationale: "auto-recovery-disabled" },
};
}
const effectiveMode = context.settings.perClass?.[failure.class] ?? context.settings.mode;
if (isDestructiveAmbiguity(failure)) {
return {
action: "pause",
rationale: "destructive-ambiguity",
legacyPausedReason: failure.pausedReason,
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale: "destructive-ambiguity" },
};
}
const maxRetries = context.settings.maxRetries ?? 3;
if (context.retryCount >= maxRetries) {
return {
action: "pause",
rationale: "retry-budget-exhausted",
legacyPausedReason: failure.pausedReason,
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale: "retry-budget-exhausted", maxRetries },
};
}
const action = actionForMode(effectiveMode, failure.class);
const rationale = `mode-${effectiveMode}`;
return {
action,
rationale,
legacyPausedReason: failure.pausedReason,
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale },
};
}
async dispatch(failure: AutoRecoveryFailure, context: AutoRecoveryContext): Promise<AutoRecoveryDecision> {
void this.taskStore;
const decision = this.classify(failure, context);
await this.auditEmitter.database({
type: "auto-recovery:classify-decision",
target: failure.taskId,
metadata: decision.auditMetadata,
});
if (decision.rationale === "destructive-ambiguity") {
await this.auditEmitter.database({
type: "auto-recovery:pause-because-destructive-ambiguity",
target: failure.taskId,
metadata: decision.auditMetadata,
});
return decision;
}
if (decision.action === "retry") {
if (!this.handlers.issueRetry) {
this.logger.warn(`auto-recovery: handler-not-registered for class=${failure.class} action=retry — falling back to pause`);
return { ...decision, action: "pause", rationale: "handler-not-registered" };
}
await this.handlers.issueRetry(failure, decision, context);
await this.auditEmitter.database({
type: "auto-recovery:retry-issued",
target: failure.taskId,
metadata: decision.auditMetadata,
});
return decision;
}
if (decision.action === "spawn-ai-recovery") {
if (!this.handlers.spawnAiRecovery) {
this.logger.warn(`auto-recovery: handler-not-registered for class=${failure.class} action=spawn-ai-recovery — falling back to pause`);
return { ...decision, action: "pause", rationale: "handler-not-registered" };
}
await this.handlers.spawnAiRecovery(failure, decision, context);
await this.auditEmitter.database({
type: "auto-recovery:ai-session-spawned",
target: failure.taskId,
metadata: decision.auditMetadata,
});
return decision;
}
return decision;
}
}

View File

@@ -71,6 +71,7 @@ import {
import { buildPromptLayers, collapsePromptLayers } from "./prompt-layers.js"; import { buildPromptLayers, collapsePromptLayers } from "./prompt-layers.js";
import type { AgentReflectionService } from "./agent-reflection.js"; import type { AgentReflectionService } from "./agent-reflection.js";
import { createRunAuditor, generateSyntheticRunId, type EngineRunContext } from "./run-audit.js"; import { createRunAuditor, generateSyntheticRunId, type EngineRunContext } from "./run-audit.js";
import { AutoRecoveryDispatcher } from "./auto-recovery.js";
import { ReadonlyViolationError, filterCustomToolsForReadonly } from "./workflow-step-tool-policy.js"; import { ReadonlyViolationError, filterCustomToolsForReadonly } from "./workflow-step-tool-policy.js";
import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js"; import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js";
import { import {
@@ -767,6 +768,7 @@ export interface TaskExecutorOptions {
onError?: (task: Task, error: Error) => void; onError?: (task: Task, error: Error) => void;
onAgentText?: (taskId: string, delta: string) => void; onAgentText?: (taskId: string, delta: string) => void;
onAgentTool?: (taskId: string, toolName: string) => void; onAgentTool?: (taskId: string, toolName: string) => void;
autoRecoveryDispatcher?: AutoRecoveryDispatcher;
} }
export class TaskExecutor { export class TaskExecutor {
@@ -4253,12 +4255,30 @@ export class TaskExecutor {
await this.store.logEntry(task.id, `[recovery] contamination auto-recovery failed: ${recoveryMessage}`, undefined, this.currentRunContext); await this.store.logEntry(task.id, `[recovery] contamination auto-recovery failed: ${recoveryMessage}`, undefined, this.currentRunContext);
} }
await this.store.updateTask(task.id, { const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
status: "failed", const decision = await autoRecoveryDispatcher.dispatch({
error: err.message, class: "branch-cross-contamination",
paused: true, taskId: task.id,
runId: this.currentRunContext?.runId,
pausedReason: "branch-cross-contamination", pausedReason: "branch-cross-contamination",
evidence: {
ownCommits: err.foreignCommits.filter((commit) => commit.foreignTaskId === task.id).length,
foreignAttributedCommits: err.foreignCommits.filter((commit) => commit.foreignTaskId !== task.id).length,
},
underlyingError: err,
}, {
task,
retryCount: task.recoveryRetryCount ?? 0,
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
}); });
if (decision.action === "pause") {
await this.store.updateTask(task.id, {
status: "failed",
error: err.message,
paused: true,
pausedReason: "branch-cross-contamination",
});
}
return; return;
} else if (isBranchConflictError(err)) { } else if (isBranchConflictError(err)) {
const conflictCount = (this.branchConflictErrorCount.get(task.id) ?? 0) + 1; const conflictCount = (this.branchConflictErrorCount.get(task.id) ?? 0) + 1;
@@ -4273,12 +4293,30 @@ export class TaskExecutor {
].join(" "); ].join(" ");
const tripwireMessage = `Branch conflict tripwire fired after ${conflictCount} events (threshold ${this.BRANCH_CONFLICT_TRIPWIRE_THRESHOLD}). ${details}`; const tripwireMessage = `Branch conflict tripwire fired after ${conflictCount} events (threshold ${this.BRANCH_CONFLICT_TRIPWIRE_THRESHOLD}). ${details}`;
await this.store.logEntry(task.id, `[recovery] ${tripwireMessage}`, undefined, this.currentRunContext); await this.store.logEntry(task.id, `[recovery] ${tripwireMessage}`, undefined, this.currentRunContext);
await this.store.updateTask(task.id, { const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
status: "failed", const decision = await autoRecoveryDispatcher.dispatch({
error: tripwireMessage, class: "branch-conflict-tripwire",
paused: true, taskId: task.id,
runId: this.currentRunContext?.runId,
pausedReason: "branch-conflict-tripwire", pausedReason: "branch-conflict-tripwire",
evidence: {
branchName: err.branchName,
conflictingWorktreePath: err.conflictingWorktreePath,
},
underlyingError: err,
}, {
task,
retryCount: task.recoveryRetryCount ?? 0,
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
}); });
if (decision.action === "pause") {
await this.store.updateTask(task.id, {
status: "failed",
error: tripwireMessage,
paused: true,
pausedReason: "branch-conflict-tripwire",
});
}
return; return;
} }
@@ -4299,12 +4337,30 @@ export class TaskExecutor {
}); });
} }
if (outcome === "retry") { if (outcome === "retry") {
await this.store.updateTask(task.id, { const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
status: "failed", const decision = await autoRecoveryDispatcher.dispatch({
error: err.message, class: "branch-conflict-recovery-exhausted",
paused: true, taskId: task.id,
runId: this.currentRunContext?.runId,
pausedReason: "branch-conflict-recovery-exhausted", pausedReason: "branch-conflict-recovery-exhausted",
evidence: {
branchName: err.branchName,
conflictingWorktreePath: err.conflictingWorktreePath,
},
underlyingError: err,
}, {
task,
retryCount: task.recoveryRetryCount ?? 0,
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
}); });
if (decision.action === "pause") {
await this.store.updateTask(task.id, {
status: "failed",
error: err.message,
paused: true,
pausedReason: "branch-conflict-recovery-exhausted",
});
}
return; return;
} }
return; return;
@@ -7211,18 +7267,42 @@ Backward compat fallback: if JSON is unavailable, you may still begin output wit
`Run 'fn task branch-recovery ${task.id}' to inspect candidates, then reclaim the existing branch or discard prior work explicitly.`; `Run 'fn task branch-recovery ${task.id}' to inspect candidates, then reclaim the existing branch or discard prior work explicitly.`;
await this.store.logEntry(task.id, this.formatBranchConflictLifecycleLog(task.id, error), undefined, this.currentRunContext); await this.store.logEntry(task.id, this.formatBranchConflictLifecycleLog(task.id, error), undefined, this.currentRunContext);
await this.store.appendAgentLog(task.id, "Branch conflict recovery required", "tool_error", this.formatBranchConflictAgentLog(task.id, error), "executor"); await this.store.appendAgentLog(task.id, "Branch conflict recovery required", "tool_error", this.formatBranchConflictAgentLog(task.id, error), "executor");
await this.store.updateTask(task.id, { const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({
status: "failed", taskStore: this.store,
error: conflictMessage, auditEmitter: createRunAuditor(this.store, this.currentRunContext),
branch: error.branchName,
worktree: error.conflictingWorktreePath,
paused: true,
pausedReason: "branch-conflict-unrecoverable",
}); });
await this.persistTokenUsage(task.id); const decision = await autoRecoveryDispatcher.dispatch({
executorLog.warn(`${task.id} branch conflict sticky failure: ${error.branchName} @ ${error.conflictingWorktreePath}`); class: "branch-conflict-unrecoverable",
this.options.onError?.(task, error); taskId: task.id,
return "sticky"; runId: this.currentRunContext?.runId,
pausedReason: "branch-conflict-unrecoverable",
evidence: {
branchName: error.branchName,
conflictingWorktreePath: error.conflictingWorktreePath,
},
underlyingError: error,
}, {
task,
retryCount: task.recoveryRetryCount ?? 0,
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
});
if (decision.action === "pause") {
await this.store.updateTask(task.id, {
status: "failed",
error: conflictMessage,
branch: error.branchName,
worktree: error.conflictingWorktreePath,
paused: true,
pausedReason: "branch-conflict-unrecoverable",
});
await this.persistTokenUsage(task.id);
executorLog.warn(`${task.id} branch conflict sticky failure: ${error.branchName} @ ${error.conflictingWorktreePath}`);
this.options.onError?.(task, error);
return "sticky";
}
return "retry";
} }
private async createWorktree( private async createWorktree(

View File

@@ -100,6 +100,10 @@ export type DatabaseMutationType =
| "task:dependency:add" | "task:dependency:add"
| "task:auto-recover-already-merged" | "task:auto-recover-already-merged"
| "task:auto-recover-completion-fanout" | "task:auto-recover-completion-fanout"
| "auto-recovery:classify-decision"
| "auto-recovery:retry-issued"
| "auto-recovery:ai-session-spawned"
| "auto-recovery:pause-because-destructive-ambiguity"
| "document:write" | "document:write"
| "workflow-step:result" | "workflow-step:result"
| "agent:create:requested" | "agent:create:requested"

View File

@@ -25,6 +25,7 @@ import { extractMissingWorktreePathFromSessionStartFailure, isMissingWorktreeSes
import { classifyError, extractMissingModulePath, isOperatorActionableAgentError, isStaleWorktreeModuleResolutionError } from "./transient-error-detector.js"; import { classifyError, extractMissingModulePath, isOperatorActionableAgentError, isStaleWorktreeModuleResolutionError } from "./transient-error-detector.js";
import { deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js"; import { deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js";
import { createRunAuditor, generateSyntheticRunId } from "./run-audit.js"; import { createRunAuditor, generateSyntheticRunId } from "./run-audit.js";
import { AutoRecoveryDispatcher } from "./auto-recovery.js";
const log = createLogger("self-healing"); const log = createLogger("self-healing");
const execAsync = promisify(exec); const execAsync = promisify(exec);
@@ -172,6 +173,7 @@ export interface SelfHealingOptions {
staleMergingFanoutMinAgeMs?: number; staleMergingFanoutMinAgeMs?: number;
hasActiveAgentExecution?: (agentId: string) => boolean; hasActiveAgentExecution?: (agentId: string) => boolean;
restartDurableAgentHeartbeat?: (agentId: string, context: { reason: string; attempt: number }) => Promise<boolean>; restartDurableAgentHeartbeat?: (agentId: string, context: { reason: string; attempt: number }) => Promise<boolean>;
autoRecoveryDispatcher?: AutoRecoveryDispatcher;
} }
const APPROVED_TRIAGE_RECOVERY_GRACE_MS = 60_000; const APPROVED_TRIAGE_RECOVERY_GRACE_MS = 60_000;
@@ -1735,14 +1737,39 @@ export class SelfHealingManager {
if (patchPath) { if (patchPath) {
await this.store.logEntry(task.id, `Preserved uncommitted worktree changes before pause: ${patchPath}`); await this.store.logEntry(task.id, `Preserved uncommitted worktree changes before pause: ${patchPath}`);
} }
await this.store.updateTask(task.id, { const dispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({
status: "failed", taskStore: this.store,
error: `Task branch conflict: ${task.branch} is not safely reclaimable (${message})`, auditEmitter: createRunAuditor(this.store, {
paused: true, runId: generateSyntheticRunId("self-heal", task.id),
pausedReason: "branch-conflict-unrecoverable", agentId: "self-healing",
taskId: task.id,
taskLineageId: task.lineageId,
phase: "reclaim-self-owned-branch-conflicts",
}),
}); });
await this.store.moveTask(task.id, "in-review"); const decision = await dispatcher.dispatch({
await this.store.logEntry(task.id, `Auto-recovery failed: branch conflict unrecoverable${message}`); class: "branch-conflict-unrecoverable",
taskId: task.id,
pausedReason: "branch-conflict-unrecoverable",
evidence: {
branchName: task.branch,
worktreePath: task.worktree,
},
}, {
task,
retryCount: task.recoveryRetryCount ?? 0,
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
});
if (decision.action === "pause") {
await this.store.updateTask(task.id, {
status: "failed",
error: `Task branch conflict: ${task.branch} is not safely reclaimable (${message})`,
paused: true,
pausedReason: "branch-conflict-unrecoverable",
});
await this.store.moveTask(task.id, "in-review");
await this.store.logEntry(task.id, `Auto-recovery failed: branch conflict unrecoverable — ${message}`);
}
} }
} }