feat(FN-4534): add auto-recovery dispatcher and pause-site wiring
Fusion-Task-Id: FN-4534 Fusion-Task-Lineage: 2b46a5df-efc6-4305-a0f2-69991e58512c
This commit is contained in:
5
.changeset/fn-4534-auto-recovery-orchestrator.md
Normal file
5
.changeset/fn-4534-auto-recovery-orchestrator.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
Introduce AutoRecoveryDispatcher and `ProjectSettings.autoRecovery` (mode/perClass/maxRetries) for classifier-driven recovery of reliability-layer failures. Adds new run-audit event types `auto-recovery:classify-decision`, `auto-recovery:retry-issued`, `auto-recovery:ai-session-spawned`, and `auto-recovery:pause-because-destructive-ambiguity`. Default mode preserves prior behavior; `mode: "off"` is byte-identical to legacy parking.
|
||||||
@@ -199,6 +199,9 @@ Defaults from `DEFAULT_PROJECT_SETTINGS`; key scope from `PROJECT_SETTINGS_KEYS`
|
|||||||
| `mergeStrategyOverlapBehavior` | `"flip-to-prefer-branch" \| "warn-only" \| "ignore"` | `"flip-to-prefer-branch"` | Safety control for `mergeConflictStrategy="smart-prefer-main"`. Before the Attempt 3 `-X ours` fallback, Fusion checks whether the task branch and recent `main` history overlap on the same files (30-commit lookback, matching the squash audit heuristics). `flip-to-prefer-branch` makes overlapping files prefer the task branch so hardening is not silently discarded (the FN-3936 class of regression). `warn-only` logs the overlap but keeps the legacy main-wins fallback. `ignore` disables the overlap guard and preserves legacy behavior exactly. |
|
| `mergeStrategyOverlapBehavior` | `"flip-to-prefer-branch" \| "warn-only" \| "ignore"` | `"flip-to-prefer-branch"` | Safety control for `mergeConflictStrategy="smart-prefer-main"`. Before the Attempt 3 `-X ours` fallback, Fusion checks whether the task branch and recent `main` history overlap on the same files (30-commit lookback, matching the squash audit heuristics). `flip-to-prefer-branch` makes overlapping files prefer the task branch so hardening is not silently discarded (the FN-3936 class of regression). `warn-only` logs the overlap but keeps the legacy main-wins fallback. `ignore` disables the overlap guard and preserves legacy behavior exactly. |
|
||||||
| `postMergeAuditMode` | `"block" \| "warn" \| "off"` | `"warn"` | Controls the post-merge audit gate. **Warn** (default) logs findings and continues to auto-complete merges. **Block** is the stricter opt-in mode: it refuses auto-completion on duplicate-subject or touched-file overlap findings when you want maximum FN-3936-class drop protection. **Off** skips the audit entirely. Regardless of mode, rebase-strategy overlap-only findings are auto-cleared when deterministic merge verification has already proven the tree (FN-4333). |
|
| `postMergeAuditMode` | `"block" \| "warn" \| "off"` | `"warn"` | Controls the post-merge audit gate. **Warn** (default) logs findings and continues to auto-complete merges. **Block** is the stricter opt-in mode: it refuses auto-completion on duplicate-subject or touched-file overlap findings when you want maximum FN-3936-class drop protection. **Off** skips the audit entirely. Regardless of mode, rebase-strategy overlap-only findings are auto-cleared when deterministic merge verification has already proven the tree (FN-4333). |
|
||||||
| `mergeAuditAutoRecovery` | `"deterministic-only" \| "programmatic" \| "ai-assisted" \| "off"` | `"ai-assisted"` | Controls how the engine recovers when the post-merge audit finds risks. **Deterministic only** keeps just the verified-rebase short-circuit. **Programmatic** also diffs each flagged main commit against HEAD and passes when every contribution survives. **AI-assisted** additionally lets the merger write a single restoration commit when programmatic checks find real drops, and bounces the task back to in-progress before parking. **Off** disables all recovery — failed audits park the task immediately. |
|
| `mergeAuditAutoRecovery` | `"deterministic-only" \| "programmatic" \| "ai-assisted" \| "off"` | `"ai-assisted"` | Controls how the engine recovers when the post-merge audit finds risks. **Deterministic only** keeps just the verified-rebase short-circuit. **Programmatic** also diffs each flagged main commit against HEAD and passes when every contribution survives. **AI-assisted** additionally lets the merger write a single restoration commit when programmatic checks find real drops, and bounces the task back to in-progress before parking. **Off** disables all recovery — failed audits park the task immediately. |
|
||||||
|
| `autoRecovery.mode` | `"off" \| "deterministic-only" \| "programmatic" \| "ai-assisted"` | `"deterministic-only"` | Dispatcher mode for recoverable executor/self-healing failure classes. `"off"` is byte-identical legacy parking behavior (exact legacy `pausedReason` preserved). |
|
||||||
|
| `autoRecovery.perClass` | `Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>` | `undefined` | Optional per-class mode override map. Overrides `autoRecovery.mode` for listed classes only. Taxonomy strings follow FN-4533 design. |
|
||||||
|
| `autoRecovery.maxRetries` | `number` | `3` | Retry budget for dispatcher decisions. When `retryCount >= maxRetries`, dispatcher forces `pause` with rationale `retry-budget-exhausted`. |
|
||||||
|
|
||||||
### Per-task direct-merge override
|
### Per-task direct-merge override
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import {
|
|||||||
DEFAULT_PROJECT_SETTINGS,
|
DEFAULT_PROJECT_SETTINGS,
|
||||||
GLOBAL_SETTINGS_KEYS,
|
GLOBAL_SETTINGS_KEYS,
|
||||||
PROJECT_SETTINGS_KEYS,
|
PROJECT_SETTINGS_KEYS,
|
||||||
|
normalizeAutoRecovery,
|
||||||
isGlobalOnlySettingsKey,
|
isGlobalOnlySettingsKey,
|
||||||
isGlobalSettingsKey,
|
isGlobalSettingsKey,
|
||||||
isProjectSettingsKey,
|
isProjectSettingsKey,
|
||||||
@@ -136,6 +137,16 @@ describe("settings key parity", () => {
|
|||||||
expect(DEFAULT_PROJECT_SETTINGS.workflowStepTimeoutMs).toBe(360_000);
|
expect(DEFAULT_PROJECT_SETTINGS.workflowStepTimeoutMs).toBe(360_000);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("defaults autoRecovery and normalizes overrides", () => {
|
||||||
|
expect(DEFAULT_PROJECT_SETTINGS.autoRecovery).toEqual({ mode: "deterministic-only", maxRetries: 3 });
|
||||||
|
expect(normalizeAutoRecovery({ mode: "off", perClass: { "branch-conflict-unrecoverable": "ai-assisted" }, maxRetries: 2 })).toEqual({
|
||||||
|
mode: "off",
|
||||||
|
perClass: { "branch-conflict-unrecoverable": "ai-assisted" },
|
||||||
|
maxRetries: 2,
|
||||||
|
});
|
||||||
|
expect(normalizeAutoRecovery({ mode: "invalid" })).toEqual({ mode: "deterministic-only", perClass: undefined, maxRetries: 3 });
|
||||||
|
});
|
||||||
|
|
||||||
it("defaults stale high fan-out blocker escalation age threshold", () => {
|
it("defaults stale high fan-out blocker escalation age threshold", () => {
|
||||||
expect(DEFAULT_PROJECT_SETTINGS.staleHighFanoutBlockerAgeThresholdMs).toBe(2 * 60 * 60 * 1000);
|
expect(DEFAULT_PROJECT_SETTINGS.staleHighFanoutBlockerAgeThresholdMs).toBe(2 * 60 * 60 * 1000);
|
||||||
expect(isProjectSettingsKey("staleHighFanoutBlockerAgeThresholdMs")).toBe(true);
|
expect(isProjectSettingsKey("staleHighFanoutBlockerAgeThresholdMs")).toBe(true);
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
export { COLUMNS, DEFAULT_COLUMN, isColumn, normalizeColumn, COLUMN_LABELS, COLUMN_DESCRIPTIONS, VALID_TRANSITIONS, DEFAULT_SETTINGS, DEFAULT_GLOBAL_SETTINGS, DEFAULT_PROJECT_SETTINGS, GLOBAL_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS, isGlobalSettingsKey, isProjectSettingsKey, resolvePersistAgentThinkingLog, THINKING_LEVELS, THEME_MODES, COLOR_THEMES, WORKFLOW_STEP_TEMPLATES, AGENT_PERMISSIONS, PERMANENT_AGENT_ACTION_CATEGORIES, AGENT_PERMISSION_POLICY_ACTION_CATEGORIES, AGENT_PROVISIONING_APPROVAL_MODES, AGENT_PERMISSION_POLICY_PRESET_IDS, LEGACY_AGENT_PERMISSION_POLICY_ACTION_CATEGORY_ALIASES, APPROVAL_REQUEST_STATUSES, APPROVAL_REQUEST_AUDIT_EVENT_TYPES, normalizeApprovalRequestActionCategory, isValidApprovalRequestTransition, agentToConfigSnapshot, diffConfigSnapshots, isEphemeralAgent, hasAgentIdentity, CheckoutConflictError, DEFAULT_HEARTBEAT_PROCEDURE_PATH, getDefaultHeartbeatProcedurePath, EXECUTION_MODES, DEFAULT_EXECUTION_MODE, TASK_PRIORITIES, DEFAULT_TASK_PRIORITY, HIGH_FANOUT_BLOCKER_TODO_THRESHOLD, STALE_HIGH_FANOUT_BLOCKER_AGE_THRESHOLD_MS, DASHBOARD_USER_ID, normalizeMessageParticipant, validateMessageMetadata, validateDockerNodeConfig, sanitizeDockerNodeConfigForResponse, normalizeMergeConflictStrategy, normalizeMergeStrategyOverlapBehavior, normalizePostMergeAuditMode, POST_MERGE_AUDIT_MODES, normalizeMergeAuditAutoRecovery, MERGE_AUDIT_AUTO_RECOVERY_MODES, buildResearchDocumentKey, REPO_OVERRIDE_RE, SHARED_STATE_SNAPSHOT_VERSION } from "./types.js";
|
export { COLUMNS, DEFAULT_COLUMN, isColumn, normalizeColumn, COLUMN_LABELS, COLUMN_DESCRIPTIONS, VALID_TRANSITIONS, DEFAULT_SETTINGS, DEFAULT_GLOBAL_SETTINGS, DEFAULT_PROJECT_SETTINGS, GLOBAL_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS, isGlobalSettingsKey, isProjectSettingsKey, resolvePersistAgentThinkingLog, THINKING_LEVELS, THEME_MODES, COLOR_THEMES, WORKFLOW_STEP_TEMPLATES, AGENT_PERMISSIONS, PERMANENT_AGENT_ACTION_CATEGORIES, AGENT_PERMISSION_POLICY_ACTION_CATEGORIES, AGENT_PROVISIONING_APPROVAL_MODES, AGENT_PERMISSION_POLICY_PRESET_IDS, LEGACY_AGENT_PERMISSION_POLICY_ACTION_CATEGORY_ALIASES, APPROVAL_REQUEST_STATUSES, APPROVAL_REQUEST_AUDIT_EVENT_TYPES, normalizeApprovalRequestActionCategory, isValidApprovalRequestTransition, agentToConfigSnapshot, diffConfigSnapshots, isEphemeralAgent, hasAgentIdentity, CheckoutConflictError, DEFAULT_HEARTBEAT_PROCEDURE_PATH, getDefaultHeartbeatProcedurePath, EXECUTION_MODES, DEFAULT_EXECUTION_MODE, TASK_PRIORITIES, DEFAULT_TASK_PRIORITY, HIGH_FANOUT_BLOCKER_TODO_THRESHOLD, STALE_HIGH_FANOUT_BLOCKER_AGE_THRESHOLD_MS, DASHBOARD_USER_ID, normalizeMessageParticipant, validateMessageMetadata, validateDockerNodeConfig, sanitizeDockerNodeConfigForResponse, normalizeMergeConflictStrategy, normalizeMergeStrategyOverlapBehavior, normalizePostMergeAuditMode, POST_MERGE_AUDIT_MODES, normalizeMergeAuditAutoRecovery, MERGE_AUDIT_AUTO_RECOVERY_MODES, normalizeAutoRecovery, AUTO_RECOVERY_MODES, buildResearchDocumentKey, REPO_OVERRIDE_RE, SHARED_STATE_SNAPSHOT_VERSION } from "./types.js";
|
||||||
export type { Column, IssueInfo, IssueState, TaskSourceIssue, PrInfo, PrStatus, Task, TaskTokenUsage, TaskAttachment, TaskComment, TaskCommentInput, TaskDocument, TaskDocumentRevision, TaskDocumentCreateInput, TaskDocumentWithTask, TaskCreateInput, MeshReplicatedTaskCreatePayload, MeshReplicatedTaskApplyResult, TaskSource, SourceType, TaskDetail, RetrySummary, InboxTask, TodoList, TodoItem, TodoListCreateInput, TodoListUpdateInput, TodoItemCreateInput, TodoItemUpdateInput, TodoListWithItems, AgentLogEntry, AgentLogType, AgentRole, BoardConfig, DistributedTaskIdReserveInput, DistributedTaskIdReserveResult, DistributedTaskIdCommitInput, DistributedTaskIdCommitResult, DistributedTaskIdAbortInput, DistributedTaskIdAbortResult, DistributedTaskIdStateInput, DistributedTaskIdStateResult, AutostashOrphanRecord, AutostashOutcome, MergeDetails, MergeResult, MergeConflictStrategy, CanonicalMergeConflictStrategy, MergeStrategyOverlapBehavior, PostMergeAuditMode, MergeAuditAutoRecoveryMode, DirectMergeCommitStrategy, Settings, GlobalSettings, ProjectSettings, WebSearchBackend, ResearchEnabledSources, ResearchGlobalDefaults, ResearchProjectLimits, ResearchProjectSettings, EvalFollowUpPolicy, EvalProjectSettings, ResolvedEvalSettings, SettingsScope, DaemonTokenSettings, TaskStep, StepStatus, TaskLogEntry, RunMutationContext, ActivityLogEntry, ActivityEventType, ThinkingLevel, ThemeMode, ColorTheme, ExecutionMode, TaskPriority, UnavailableNodePolicy, PlanningQuestion, PlanningSummary, PlanningResponse, PlanningQuestionType, ArchivedTaskEntry, BatchStatusRequest, BatchStatusResponse, BatchStatusEntry, BatchStatusResult, GithubIssueAction, ModelPreset, WorkflowStep, WorkflowStepMode, WorkflowStepGateMode, WorkflowStepPhase, WorkflowStepInput, WorkflowStepResult, WorkflowStepTemplate, Agent, OrgTreeNode, AgentState, AgentDetail, AgentCreateInput, AgentUpdateInput, AgentApiKey, AgentApiKeyCreateResult, AgentCapability, AgentPromptTemplate, AgentPromptsConfig, AgentPermission, PermanentAgentActionCategory, PermanentAgentSensitiveActionCategory, PermanentAgentGatingContext, AgentPermissionPolicy, AgentPermissionPolicyRules, AgentPermissionPolicyActionCategory, AgentProvisioningApprovalMode, LegacyAgentPermissionPolicyActionCategory, ApprovalRequestActionCategoryInput, ApprovalRequestActionCategory, AgentPermissionPolicyDisposition, AgentPermissionPolicyPresetId, ApprovalRequestStatus, ApprovalRequestAuditEventType, ApprovalRequestActorSnapshot, ApprovalRequestTargetAction, ApprovalRequestAuditEvent, ApprovalRequest, ApprovalRequestCreateInput, ApprovalRequestDecisionInput, ApprovalRequestCompletionInput, ApprovalRequestListInput, TaskAssignSource, AgentAccessState, AgentHeartbeatConfig, AgentBudgetConfig, AgentBudgetStatus, InstructionsBundleConfig, MessageResponseMode, AgentHeartbeatEvent, AgentHeartbeatRun, BlockedStateSnapshot, HeartbeatInvocationSource, AgentTaskSession, AgentRating, AgentRatingSummary, AgentRatingInput, AgentConfigSnapshot, RevisionFieldDiff, AgentConfigRevision, AgentStats, ReflectionTrigger, ReflectionMetrics, AgentReflection, AgentPerformanceSummary, NtfyNotificationEvent, NotificationEvent, NotificationPayload, NotificationProviderConfig, CustomProvider, SteeringComment, ParticipantType, MessageType, Message, MessageCreateInput, MessageFilter, MessageMetadata, MessageReplyReference, Mailbox, CheckoutLease, RunAuditDomain, RunAuditEvent, RunAuditEventInput, RunAuditEventFilter, AgentMemoryInclusionMode, HeartbeatPromptTemplate, HeartbeatScopeDisciplineMode } from "./types.js";
|
export type { Column, IssueInfo, IssueState, TaskSourceIssue, PrInfo, PrStatus, Task, TaskTokenUsage, TaskAttachment, TaskComment, TaskCommentInput, TaskDocument, TaskDocumentRevision, TaskDocumentCreateInput, TaskDocumentWithTask, TaskCreateInput, MeshReplicatedTaskCreatePayload, MeshReplicatedTaskApplyResult, TaskSource, SourceType, TaskDetail, RetrySummary, InboxTask, TodoList, TodoItem, TodoListCreateInput, TodoListUpdateInput, TodoItemCreateInput, TodoItemUpdateInput, TodoListWithItems, AgentLogEntry, AgentLogType, AgentRole, BoardConfig, DistributedTaskIdReserveInput, DistributedTaskIdReserveResult, DistributedTaskIdCommitInput, DistributedTaskIdCommitResult, DistributedTaskIdAbortInput, DistributedTaskIdAbortResult, DistributedTaskIdStateInput, DistributedTaskIdStateResult, AutostashOrphanRecord, AutostashOutcome, MergeDetails, MergeResult, MergeConflictStrategy, CanonicalMergeConflictStrategy, MergeStrategyOverlapBehavior, PostMergeAuditMode, MergeAuditAutoRecoveryMode, AutoRecoveryMode, AutoRecoveryFailureClass, AutoRecoverySettings, DirectMergeCommitStrategy, Settings, GlobalSettings, ProjectSettings, WebSearchBackend, ResearchEnabledSources, ResearchGlobalDefaults, ResearchProjectLimits, ResearchProjectSettings, EvalFollowUpPolicy, EvalProjectSettings, ResolvedEvalSettings, SettingsScope, DaemonTokenSettings, TaskStep, StepStatus, TaskLogEntry, RunMutationContext, ActivityLogEntry, ActivityEventType, ThinkingLevel, ThemeMode, ColorTheme, ExecutionMode, TaskPriority, UnavailableNodePolicy, PlanningQuestion, PlanningSummary, PlanningResponse, PlanningQuestionType, ArchivedTaskEntry, BatchStatusRequest, BatchStatusResponse, BatchStatusEntry, BatchStatusResult, GithubIssueAction, ModelPreset, WorkflowStep, WorkflowStepMode, WorkflowStepGateMode, WorkflowStepPhase, WorkflowStepInput, WorkflowStepResult, WorkflowStepTemplate, Agent, OrgTreeNode, AgentState, AgentDetail, AgentCreateInput, AgentUpdateInput, AgentApiKey, AgentApiKeyCreateResult, AgentCapability, AgentPromptTemplate, AgentPromptsConfig, AgentPermission, PermanentAgentActionCategory, PermanentAgentSensitiveActionCategory, PermanentAgentGatingContext, AgentPermissionPolicy, AgentPermissionPolicyRules, AgentPermissionPolicyActionCategory, AgentProvisioningApprovalMode, LegacyAgentPermissionPolicyActionCategory, ApprovalRequestActionCategoryInput, ApprovalRequestActionCategory, AgentPermissionPolicyDisposition, AgentPermissionPolicyPresetId, ApprovalRequestStatus, ApprovalRequestAuditEventType, ApprovalRequestActorSnapshot, ApprovalRequestTargetAction, ApprovalRequestAuditEvent, ApprovalRequest, ApprovalRequestCreateInput, ApprovalRequestDecisionInput, ApprovalRequestCompletionInput, ApprovalRequestListInput, TaskAssignSource, AgentAccessState, AgentHeartbeatConfig, AgentBudgetConfig, AgentBudgetStatus, InstructionsBundleConfig, MessageResponseMode, AgentHeartbeatEvent, AgentHeartbeatRun, BlockedStateSnapshot, HeartbeatInvocationSource, AgentTaskSession, AgentRating, AgentRatingSummary, AgentRatingInput, AgentConfigSnapshot, RevisionFieldDiff, AgentConfigRevision, AgentStats, ReflectionTrigger, ReflectionMetrics, AgentReflection, AgentPerformanceSummary, NtfyNotificationEvent, NotificationEvent, NotificationPayload, NotificationProviderConfig, CustomProvider, SteeringComment, ParticipantType, MessageType, Message, MessageCreateInput, MessageFilter, MessageMetadata, MessageReplyReference, Mailbox, CheckoutLease, RunAuditDomain, RunAuditEvent, RunAuditEventInput, RunAuditEventFilter, AgentMemoryInclusionMode, HeartbeatPromptTemplate, HeartbeatScopeDisciplineMode } from "./types.js";
|
||||||
export { AGENT_VALID_TRANSITIONS } from "./types.js";
|
export { AGENT_VALID_TRANSITIONS } from "./types.js";
|
||||||
export {
|
export {
|
||||||
resolveAgentMemoryInclusionMode,
|
resolveAgentMemoryInclusionMode,
|
||||||
|
|||||||
@@ -227,6 +227,10 @@ export const DEFAULT_PROJECT_SETTINGS = {
|
|||||||
mergeStrategyOverlapBehavior: "flip-to-prefer-branch",
|
mergeStrategyOverlapBehavior: "flip-to-prefer-branch",
|
||||||
postMergeAuditMode: "warn",
|
postMergeAuditMode: "warn",
|
||||||
mergeAuditAutoRecovery: "ai-assisted",
|
mergeAuditAutoRecovery: "ai-assisted",
|
||||||
|
autoRecovery: {
|
||||||
|
mode: "deterministic-only",
|
||||||
|
maxRetries: 3,
|
||||||
|
},
|
||||||
workflowStepTimeoutMs: 360_000,
|
workflowStepTimeoutMs: 360_000,
|
||||||
workflowStepScopeEnforcement: "block",
|
workflowStepScopeEnforcement: "block",
|
||||||
planOnlyScopeLeakEnforcement: "warn",
|
planOnlyScopeLeakEnforcement: "warn",
|
||||||
|
|||||||
@@ -222,6 +222,60 @@ export function normalizeMergeAuditAutoRecovery(value: unknown): MergeAuditAutoR
|
|||||||
? (value as MergeAuditAutoRecoveryMode)
|
? (value as MergeAuditAutoRecoveryMode)
|
||||||
: "ai-assisted";
|
: "ai-assisted";
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export const AUTO_RECOVERY_MODES = ["off", "deterministic-only", "programmatic", "ai-assisted"] as const;
|
||||||
|
|
||||||
|
export type AutoRecoveryMode = (typeof AUTO_RECOVERY_MODES)[number];
|
||||||
|
|
||||||
|
export type AutoRecoveryFailureClass =
|
||||||
|
| "file-scope-invariant"
|
||||||
|
| "post-squash-audit-blocker"
|
||||||
|
| "branch-cross-contamination"
|
||||||
|
| "branch-conflict-tripwire"
|
||||||
|
| "branch-conflict-recovery-exhausted"
|
||||||
|
| "branch-conflict-unrecoverable";
|
||||||
|
|
||||||
|
export interface AutoRecoverySettings {
|
||||||
|
mode: AutoRecoveryMode;
|
||||||
|
perClass?: Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>;
|
||||||
|
maxRetries?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function normalizeAutoRecovery(value: unknown): AutoRecoverySettings {
|
||||||
|
const fallback: AutoRecoverySettings = { mode: "deterministic-only", maxRetries: 3 };
|
||||||
|
if (!value || typeof value !== "object") return fallback;
|
||||||
|
|
||||||
|
const candidate = value as {
|
||||||
|
mode?: unknown;
|
||||||
|
perClass?: unknown;
|
||||||
|
maxRetries?: unknown;
|
||||||
|
};
|
||||||
|
const mode = typeof candidate.mode === "string" && (AUTO_RECOVERY_MODES as readonly string[]).includes(candidate.mode)
|
||||||
|
? candidate.mode as AutoRecoveryMode
|
||||||
|
: fallback.mode;
|
||||||
|
const perClass = typeof candidate.perClass === "object" && candidate.perClass
|
||||||
|
? Object.fromEntries(
|
||||||
|
Object.entries(candidate.perClass as Record<string, unknown>)
|
||||||
|
.filter(([k, v]) => (
|
||||||
|
[
|
||||||
|
"file-scope-invariant",
|
||||||
|
"post-squash-audit-blocker",
|
||||||
|
"branch-cross-contamination",
|
||||||
|
"branch-conflict-tripwire",
|
||||||
|
"branch-conflict-recovery-exhausted",
|
||||||
|
"branch-conflict-unrecoverable",
|
||||||
|
].includes(k)
|
||||||
|
&& typeof v === "string"
|
||||||
|
&& (AUTO_RECOVERY_MODES as readonly string[]).includes(v)
|
||||||
|
)),
|
||||||
|
) as Partial<Record<AutoRecoveryFailureClass, AutoRecoveryMode>>
|
||||||
|
: undefined;
|
||||||
|
const maxRetries = typeof candidate.maxRetries === "number" && Number.isFinite(candidate.maxRetries)
|
||||||
|
? Math.max(0, Math.floor(candidate.maxRetries))
|
||||||
|
: fallback.maxRetries;
|
||||||
|
|
||||||
|
return { mode, perClass, maxRetries };
|
||||||
|
}
|
||||||
/** Policy for handling task execution when the selected node is unavailable/unhealthy. */
|
/** Policy for handling task execution when the selected node is unavailable/unhealthy. */
|
||||||
export type UnavailableNodePolicy = "block" | "fallback-local";
|
export type UnavailableNodePolicy = "block" | "fallback-local";
|
||||||
|
|
||||||
@@ -2459,6 +2513,8 @@ export interface ProjectSettings {
|
|||||||
* - "off": disable all recovery; audit blocks immediately.
|
* - "off": disable all recovery; audit blocks immediately.
|
||||||
*/
|
*/
|
||||||
mergeAuditAutoRecovery?: MergeAuditAutoRecoveryMode;
|
mergeAuditAutoRecovery?: MergeAuditAutoRecoveryMode;
|
||||||
|
/** Dispatcher-level reliability recovery policy (FN-4533/FN-4534). */
|
||||||
|
autoRecovery?: AutoRecoverySettings;
|
||||||
/** Wall-clock timeout (ms) for a single pre-merge workflow step's AI call.
|
/** Wall-clock timeout (ms) for a single pre-merge workflow step's AI call.
|
||||||
* When a step exceeds this, the session is aborted and the executor is
|
* When a step exceeds this, the session is aborted and the executor is
|
||||||
* given one shot to retry with the configured fallback model before the
|
* given one shot to retry with the configured fallback model before the
|
||||||
|
|||||||
86
packages/engine/src/__tests__/auto-recovery.test.ts
Normal file
86
packages/engine/src/__tests__/auto-recovery.test.ts
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
import type { AutoRecoverySettings, Task } from "@fusion/core";
|
||||||
|
import { AutoRecoveryDispatcher, type AutoRecoveryFailure } from "../auto-recovery.js";
|
||||||
|
|
||||||
|
const task = { id: "FN-1", recoveryRetryCount: 0 } as Task;
|
||||||
|
|
||||||
|
function createDispatcher() {
|
||||||
|
const database = vi.fn(async () => {});
|
||||||
|
const dispatcher = new AutoRecoveryDispatcher({
|
||||||
|
taskStore: {} as never,
|
||||||
|
auditEmitter: { database, git: vi.fn(), filesystem: vi.fn() },
|
||||||
|
});
|
||||||
|
return { dispatcher, database };
|
||||||
|
}
|
||||||
|
|
||||||
|
const classes: AutoRecoveryFailure["class"][] = [
|
||||||
|
"file-scope-invariant",
|
||||||
|
"post-squash-audit-blocker",
|
||||||
|
"branch-cross-contamination",
|
||||||
|
"branch-conflict-tripwire",
|
||||||
|
"branch-conflict-recovery-exhausted",
|
||||||
|
"branch-conflict-unrecoverable",
|
||||||
|
];
|
||||||
|
|
||||||
|
describe("auto-recovery dispatcher", () => {
|
||||||
|
it.each(classes)("mode off preserves pause contract for %s", (klass) => {
|
||||||
|
const { dispatcher } = createDispatcher();
|
||||||
|
const decision = dispatcher.classify({ class: klass, taskId: "FN-1", pausedReason: "legacy-reason" }, {
|
||||||
|
task,
|
||||||
|
retryCount: 0,
|
||||||
|
settings: { mode: "off", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
expect(decision.action).toBe("pause");
|
||||||
|
expect(decision.legacyPausedReason).toBe("legacy-reason");
|
||||||
|
expect(decision.rationale).toBe("auto-recovery-disabled");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("per-class override beats global mode", () => {
|
||||||
|
const { dispatcher } = createDispatcher();
|
||||||
|
const settings: AutoRecoverySettings = {
|
||||||
|
mode: "deterministic-only",
|
||||||
|
perClass: { "branch-conflict-unrecoverable": "programmatic" },
|
||||||
|
maxRetries: 3,
|
||||||
|
};
|
||||||
|
const decision = dispatcher.classify({ class: "branch-conflict-unrecoverable", taskId: "FN-1", pausedReason: "branch-conflict-unrecoverable" }, { task, retryCount: 0, settings });
|
||||||
|
expect(decision.action).toBe("retry");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("forces pause on retry budget exhausted", () => {
|
||||||
|
const { dispatcher } = createDispatcher();
|
||||||
|
const decision = dispatcher.classify({ class: "branch-conflict-tripwire", taskId: "FN-1", pausedReason: "branch-conflict-tripwire" }, {
|
||||||
|
task,
|
||||||
|
retryCount: 3,
|
||||||
|
settings: { mode: "programmatic", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
expect(decision.action).toBe("pause");
|
||||||
|
expect(decision.rationale).toBe("retry-budget-exhausted");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("forces pause on destructive ambiguity", () => {
|
||||||
|
const { dispatcher } = createDispatcher();
|
||||||
|
const decision = dispatcher.classify({ class: "branch-cross-contamination", taskId: "FN-1", pausedReason: "branch-cross-contamination", evidence: { ownCommits: 1, foreignAttributedCommits: 1 } }, {
|
||||||
|
task,
|
||||||
|
retryCount: 0,
|
||||||
|
settings: { mode: "ai-assisted", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
expect(decision.action).toBe("pause");
|
||||||
|
expect(decision.rationale).toBe("destructive-ambiguity");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("dispatch falls back to pause when handler missing", async () => {
|
||||||
|
const { dispatcher, database } = createDispatcher();
|
||||||
|
const decision = await dispatcher.dispatch({ class: "branch-conflict-unrecoverable", taskId: "FN-1", pausedReason: "branch-conflict-unrecoverable" }, {
|
||||||
|
task,
|
||||||
|
retryCount: 0,
|
||||||
|
settings: { mode: "programmatic", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
expect(decision.action).toBe("pause");
|
||||||
|
expect(decision.rationale).toBe("handler-not-registered");
|
||||||
|
expect(database).toHaveBeenCalledTimes(1);
|
||||||
|
expect(database.mock.calls[0]?.[0]).toMatchObject({
|
||||||
|
type: "auto-recovery:classify-decision",
|
||||||
|
metadata: expect.objectContaining({ class: "branch-conflict-unrecoverable", mode: "programmatic", retryCount: 0 }),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
import type { Task } from "@fusion/core";
|
||||||
|
import { AutoRecoveryDispatcher } from "../../auto-recovery.js";
|
||||||
|
|
||||||
|
const baseTask = { id: "FN-1", recoveryRetryCount: 0 } as Task;
|
||||||
|
|
||||||
|
describe("reliability interaction: auto-recovery dispatcher precedence", () => {
|
||||||
|
it("mode off preserves legacy pausedReason contract across wired classes", () => {
|
||||||
|
const dispatcher = new AutoRecoveryDispatcher({
|
||||||
|
taskStore: {} as never,
|
||||||
|
auditEmitter: { database: vi.fn(async () => {}), git: vi.fn(), filesystem: vi.fn() },
|
||||||
|
});
|
||||||
|
|
||||||
|
const wired = [
|
||||||
|
"branch-cross-contamination",
|
||||||
|
"branch-conflict-tripwire",
|
||||||
|
"branch-conflict-recovery-exhausted",
|
||||||
|
"branch-conflict-unrecoverable",
|
||||||
|
] as const;
|
||||||
|
|
||||||
|
for (const klass of wired) {
|
||||||
|
const decision = dispatcher.classify({ class: klass, taskId: "FN-1", pausedReason: klass }, {
|
||||||
|
task: baseTask,
|
||||||
|
retryCount: 0,
|
||||||
|
settings: { mode: "off", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
expect(decision.action).toBe("pause");
|
||||||
|
expect(decision.legacyPausedReason).toBe(klass);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("deterministic recovery success can bypass dispatcher invocation", async () => {
|
||||||
|
const classify = vi.fn();
|
||||||
|
const deterministicFastPath = vi.fn(async () => true);
|
||||||
|
|
||||||
|
if (!(await deterministicFastPath())) {
|
||||||
|
classify();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(deterministicFastPath).toHaveBeenCalledOnce();
|
||||||
|
expect(classify).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
});
|
||||||
159
packages/engine/src/auto-recovery.ts
Normal file
159
packages/engine/src/auto-recovery.ts
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
import type { AutoRecoveryFailureClass, AutoRecoveryMode, AutoRecoverySettings, Task, TaskStore } from "@fusion/core";
|
||||||
|
import { createLogger, type Logger } from "./logger.js";
|
||||||
|
import type { RunAuditor } from "./run-audit.js";
|
||||||
|
|
||||||
|
export type AutoRecoveryAction = "retry" | "spawn-ai-recovery" | "pause";
|
||||||
|
|
||||||
|
export interface AutoRecoveryFailure {
|
||||||
|
class: AutoRecoveryFailureClass;
|
||||||
|
taskId: string;
|
||||||
|
runId?: string;
|
||||||
|
pausedReason: string;
|
||||||
|
evidence?: Record<string, unknown>;
|
||||||
|
underlyingError?: Error;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AutoRecoveryDecision {
|
||||||
|
action: AutoRecoveryAction;
|
||||||
|
rationale: string;
|
||||||
|
auditMetadata: Record<string, unknown>;
|
||||||
|
legacyPausedReason: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AutoRecoveryContext {
|
||||||
|
task: Task;
|
||||||
|
retryCount: number;
|
||||||
|
settings: AutoRecoverySettings;
|
||||||
|
now?: () => Date;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AutoRecoveryHandlers {
|
||||||
|
issueRetry?: (failure: AutoRecoveryFailure, decision: AutoRecoveryDecision, ctx: AutoRecoveryContext) => Promise<void>;
|
||||||
|
spawnAiRecovery?: (failure: AutoRecoveryFailure, decision: AutoRecoveryDecision, ctx: AutoRecoveryContext) => Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
const autoRecoveryLog = createLogger("auto-recovery");
|
||||||
|
|
||||||
|
function actionForMode(mode: AutoRecoveryMode, failureClass: AutoRecoveryFailureClass): AutoRecoveryAction {
|
||||||
|
if (mode === "off" || mode === "deterministic-only") return "pause";
|
||||||
|
if (mode === "programmatic") {
|
||||||
|
if (failureClass === "file-scope-invariant" || failureClass === "post-squash-audit-blocker") return "pause";
|
||||||
|
return "retry";
|
||||||
|
}
|
||||||
|
if (mode === "ai-assisted") {
|
||||||
|
if (failureClass === "file-scope-invariant" || failureClass === "post-squash-audit-blocker") return "spawn-ai-recovery";
|
||||||
|
return "retry";
|
||||||
|
}
|
||||||
|
return "pause";
|
||||||
|
}
|
||||||
|
|
||||||
|
function isDestructiveAmbiguity(failure: AutoRecoveryFailure): boolean {
|
||||||
|
if (failure.evidence?.destructiveAmbiguity === true) return true;
|
||||||
|
const own = Number(failure.evidence?.ownCommits ?? 0);
|
||||||
|
const foreign = Number(failure.evidence?.foreignAttributedCommits ?? 0);
|
||||||
|
return own > 0 && foreign > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class AutoRecoveryDispatcher {
|
||||||
|
private readonly taskStore: TaskStore;
|
||||||
|
private readonly auditEmitter: RunAuditor;
|
||||||
|
private readonly handlers: AutoRecoveryHandlers;
|
||||||
|
private readonly logger: Logger;
|
||||||
|
|
||||||
|
constructor(opts: { taskStore: TaskStore; auditEmitter: RunAuditor; handlers?: AutoRecoveryHandlers; logger?: Logger }) {
|
||||||
|
this.taskStore = opts.taskStore;
|
||||||
|
this.auditEmitter = opts.auditEmitter;
|
||||||
|
this.handlers = opts.handlers ?? {};
|
||||||
|
this.logger = opts.logger ?? autoRecoveryLog;
|
||||||
|
}
|
||||||
|
|
||||||
|
classify(failure: AutoRecoveryFailure, context: AutoRecoveryContext): AutoRecoveryDecision {
|
||||||
|
if (context.settings.mode === "off") {
|
||||||
|
return {
|
||||||
|
action: "pause",
|
||||||
|
rationale: "auto-recovery-disabled",
|
||||||
|
legacyPausedReason: failure.pausedReason,
|
||||||
|
auditMetadata: { class: failure.class, mode: "off", retryCount: context.retryCount, rationale: "auto-recovery-disabled" },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const effectiveMode = context.settings.perClass?.[failure.class] ?? context.settings.mode;
|
||||||
|
|
||||||
|
if (isDestructiveAmbiguity(failure)) {
|
||||||
|
return {
|
||||||
|
action: "pause",
|
||||||
|
rationale: "destructive-ambiguity",
|
||||||
|
legacyPausedReason: failure.pausedReason,
|
||||||
|
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale: "destructive-ambiguity" },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const maxRetries = context.settings.maxRetries ?? 3;
|
||||||
|
if (context.retryCount >= maxRetries) {
|
||||||
|
return {
|
||||||
|
action: "pause",
|
||||||
|
rationale: "retry-budget-exhausted",
|
||||||
|
legacyPausedReason: failure.pausedReason,
|
||||||
|
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale: "retry-budget-exhausted", maxRetries },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const action = actionForMode(effectiveMode, failure.class);
|
||||||
|
const rationale = `mode-${effectiveMode}`;
|
||||||
|
return {
|
||||||
|
action,
|
||||||
|
rationale,
|
||||||
|
legacyPausedReason: failure.pausedReason,
|
||||||
|
auditMetadata: { class: failure.class, mode: effectiveMode, retryCount: context.retryCount, rationale },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async dispatch(failure: AutoRecoveryFailure, context: AutoRecoveryContext): Promise<AutoRecoveryDecision> {
|
||||||
|
void this.taskStore;
|
||||||
|
const decision = this.classify(failure, context);
|
||||||
|
await this.auditEmitter.database({
|
||||||
|
type: "auto-recovery:classify-decision",
|
||||||
|
target: failure.taskId,
|
||||||
|
metadata: decision.auditMetadata,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (decision.rationale === "destructive-ambiguity") {
|
||||||
|
await this.auditEmitter.database({
|
||||||
|
type: "auto-recovery:pause-because-destructive-ambiguity",
|
||||||
|
target: failure.taskId,
|
||||||
|
metadata: decision.auditMetadata,
|
||||||
|
});
|
||||||
|
return decision;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (decision.action === "retry") {
|
||||||
|
if (!this.handlers.issueRetry) {
|
||||||
|
this.logger.warn(`auto-recovery: handler-not-registered for class=${failure.class} action=retry — falling back to pause`);
|
||||||
|
return { ...decision, action: "pause", rationale: "handler-not-registered" };
|
||||||
|
}
|
||||||
|
await this.handlers.issueRetry(failure, decision, context);
|
||||||
|
await this.auditEmitter.database({
|
||||||
|
type: "auto-recovery:retry-issued",
|
||||||
|
target: failure.taskId,
|
||||||
|
metadata: decision.auditMetadata,
|
||||||
|
});
|
||||||
|
return decision;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (decision.action === "spawn-ai-recovery") {
|
||||||
|
if (!this.handlers.spawnAiRecovery) {
|
||||||
|
this.logger.warn(`auto-recovery: handler-not-registered for class=${failure.class} action=spawn-ai-recovery — falling back to pause`);
|
||||||
|
return { ...decision, action: "pause", rationale: "handler-not-registered" };
|
||||||
|
}
|
||||||
|
await this.handlers.spawnAiRecovery(failure, decision, context);
|
||||||
|
await this.auditEmitter.database({
|
||||||
|
type: "auto-recovery:ai-session-spawned",
|
||||||
|
target: failure.taskId,
|
||||||
|
metadata: decision.auditMetadata,
|
||||||
|
});
|
||||||
|
return decision;
|
||||||
|
}
|
||||||
|
|
||||||
|
return decision;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -71,6 +71,7 @@ import {
|
|||||||
import { buildPromptLayers, collapsePromptLayers } from "./prompt-layers.js";
|
import { buildPromptLayers, collapsePromptLayers } from "./prompt-layers.js";
|
||||||
import type { AgentReflectionService } from "./agent-reflection.js";
|
import type { AgentReflectionService } from "./agent-reflection.js";
|
||||||
import { createRunAuditor, generateSyntheticRunId, type EngineRunContext } from "./run-audit.js";
|
import { createRunAuditor, generateSyntheticRunId, type EngineRunContext } from "./run-audit.js";
|
||||||
|
import { AutoRecoveryDispatcher } from "./auto-recovery.js";
|
||||||
import { ReadonlyViolationError, filterCustomToolsForReadonly } from "./workflow-step-tool-policy.js";
|
import { ReadonlyViolationError, filterCustomToolsForReadonly } from "./workflow-step-tool-policy.js";
|
||||||
import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js";
|
import { evaluateSpecStaleness, getPromptPath } from "./spec-staleness.js";
|
||||||
import {
|
import {
|
||||||
@@ -767,6 +768,7 @@ export interface TaskExecutorOptions {
|
|||||||
onError?: (task: Task, error: Error) => void;
|
onError?: (task: Task, error: Error) => void;
|
||||||
onAgentText?: (taskId: string, delta: string) => void;
|
onAgentText?: (taskId: string, delta: string) => void;
|
||||||
onAgentTool?: (taskId: string, toolName: string) => void;
|
onAgentTool?: (taskId: string, toolName: string) => void;
|
||||||
|
autoRecoveryDispatcher?: AutoRecoveryDispatcher;
|
||||||
}
|
}
|
||||||
|
|
||||||
export class TaskExecutor {
|
export class TaskExecutor {
|
||||||
@@ -4253,12 +4255,30 @@ export class TaskExecutor {
|
|||||||
await this.store.logEntry(task.id, `[recovery] contamination auto-recovery failed: ${recoveryMessage}`, undefined, this.currentRunContext);
|
await this.store.logEntry(task.id, `[recovery] contamination auto-recovery failed: ${recoveryMessage}`, undefined, this.currentRunContext);
|
||||||
}
|
}
|
||||||
|
|
||||||
await this.store.updateTask(task.id, {
|
const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
|
||||||
status: "failed",
|
const decision = await autoRecoveryDispatcher.dispatch({
|
||||||
error: err.message,
|
class: "branch-cross-contamination",
|
||||||
paused: true,
|
taskId: task.id,
|
||||||
|
runId: this.currentRunContext?.runId,
|
||||||
pausedReason: "branch-cross-contamination",
|
pausedReason: "branch-cross-contamination",
|
||||||
|
evidence: {
|
||||||
|
ownCommits: err.foreignCommits.filter((commit) => commit.foreignTaskId === task.id).length,
|
||||||
|
foreignAttributedCommits: err.foreignCommits.filter((commit) => commit.foreignTaskId !== task.id).length,
|
||||||
|
},
|
||||||
|
underlyingError: err,
|
||||||
|
}, {
|
||||||
|
task,
|
||||||
|
retryCount: task.recoveryRetryCount ?? 0,
|
||||||
|
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
|
||||||
});
|
});
|
||||||
|
if (decision.action === "pause") {
|
||||||
|
await this.store.updateTask(task.id, {
|
||||||
|
status: "failed",
|
||||||
|
error: err.message,
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "branch-cross-contamination",
|
||||||
|
});
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
} else if (isBranchConflictError(err)) {
|
} else if (isBranchConflictError(err)) {
|
||||||
const conflictCount = (this.branchConflictErrorCount.get(task.id) ?? 0) + 1;
|
const conflictCount = (this.branchConflictErrorCount.get(task.id) ?? 0) + 1;
|
||||||
@@ -4273,12 +4293,30 @@ export class TaskExecutor {
|
|||||||
].join(" ");
|
].join(" ");
|
||||||
const tripwireMessage = `Branch conflict tripwire fired after ${conflictCount} events (threshold ${this.BRANCH_CONFLICT_TRIPWIRE_THRESHOLD}). ${details}`;
|
const tripwireMessage = `Branch conflict tripwire fired after ${conflictCount} events (threshold ${this.BRANCH_CONFLICT_TRIPWIRE_THRESHOLD}). ${details}`;
|
||||||
await this.store.logEntry(task.id, `[recovery] ${tripwireMessage}`, undefined, this.currentRunContext);
|
await this.store.logEntry(task.id, `[recovery] ${tripwireMessage}`, undefined, this.currentRunContext);
|
||||||
await this.store.updateTask(task.id, {
|
const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
|
||||||
status: "failed",
|
const decision = await autoRecoveryDispatcher.dispatch({
|
||||||
error: tripwireMessage,
|
class: "branch-conflict-tripwire",
|
||||||
paused: true,
|
taskId: task.id,
|
||||||
|
runId: this.currentRunContext?.runId,
|
||||||
pausedReason: "branch-conflict-tripwire",
|
pausedReason: "branch-conflict-tripwire",
|
||||||
|
evidence: {
|
||||||
|
branchName: err.branchName,
|
||||||
|
conflictingWorktreePath: err.conflictingWorktreePath,
|
||||||
|
},
|
||||||
|
underlyingError: err,
|
||||||
|
}, {
|
||||||
|
task,
|
||||||
|
retryCount: task.recoveryRetryCount ?? 0,
|
||||||
|
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
|
||||||
});
|
});
|
||||||
|
if (decision.action === "pause") {
|
||||||
|
await this.store.updateTask(task.id, {
|
||||||
|
status: "failed",
|
||||||
|
error: tripwireMessage,
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "branch-conflict-tripwire",
|
||||||
|
});
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -4299,12 +4337,30 @@ export class TaskExecutor {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
if (outcome === "retry") {
|
if (outcome === "retry") {
|
||||||
await this.store.updateTask(task.id, {
|
const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({ taskStore: this.store, auditEmitter: audit });
|
||||||
status: "failed",
|
const decision = await autoRecoveryDispatcher.dispatch({
|
||||||
error: err.message,
|
class: "branch-conflict-recovery-exhausted",
|
||||||
paused: true,
|
taskId: task.id,
|
||||||
|
runId: this.currentRunContext?.runId,
|
||||||
pausedReason: "branch-conflict-recovery-exhausted",
|
pausedReason: "branch-conflict-recovery-exhausted",
|
||||||
|
evidence: {
|
||||||
|
branchName: err.branchName,
|
||||||
|
conflictingWorktreePath: err.conflictingWorktreePath,
|
||||||
|
},
|
||||||
|
underlyingError: err,
|
||||||
|
}, {
|
||||||
|
task,
|
||||||
|
retryCount: task.recoveryRetryCount ?? 0,
|
||||||
|
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
|
||||||
});
|
});
|
||||||
|
if (decision.action === "pause") {
|
||||||
|
await this.store.updateTask(task.id, {
|
||||||
|
status: "failed",
|
||||||
|
error: err.message,
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "branch-conflict-recovery-exhausted",
|
||||||
|
});
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
@@ -7211,18 +7267,42 @@ Backward compat fallback: if JSON is unavailable, you may still begin output wit
|
|||||||
`Run 'fn task branch-recovery ${task.id}' to inspect candidates, then reclaim the existing branch or discard prior work explicitly.`;
|
`Run 'fn task branch-recovery ${task.id}' to inspect candidates, then reclaim the existing branch or discard prior work explicitly.`;
|
||||||
await this.store.logEntry(task.id, this.formatBranchConflictLifecycleLog(task.id, error), undefined, this.currentRunContext);
|
await this.store.logEntry(task.id, this.formatBranchConflictLifecycleLog(task.id, error), undefined, this.currentRunContext);
|
||||||
await this.store.appendAgentLog(task.id, "Branch conflict recovery required", "tool_error", this.formatBranchConflictAgentLog(task.id, error), "executor");
|
await this.store.appendAgentLog(task.id, "Branch conflict recovery required", "tool_error", this.formatBranchConflictAgentLog(task.id, error), "executor");
|
||||||
await this.store.updateTask(task.id, {
|
const autoRecoveryDispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({
|
||||||
status: "failed",
|
taskStore: this.store,
|
||||||
error: conflictMessage,
|
auditEmitter: createRunAuditor(this.store, this.currentRunContext),
|
||||||
branch: error.branchName,
|
|
||||||
worktree: error.conflictingWorktreePath,
|
|
||||||
paused: true,
|
|
||||||
pausedReason: "branch-conflict-unrecoverable",
|
|
||||||
});
|
});
|
||||||
await this.persistTokenUsage(task.id);
|
const decision = await autoRecoveryDispatcher.dispatch({
|
||||||
executorLog.warn(`✗ ${task.id} branch conflict sticky failure: ${error.branchName} @ ${error.conflictingWorktreePath}`);
|
class: "branch-conflict-unrecoverable",
|
||||||
this.options.onError?.(task, error);
|
taskId: task.id,
|
||||||
return "sticky";
|
runId: this.currentRunContext?.runId,
|
||||||
|
pausedReason: "branch-conflict-unrecoverable",
|
||||||
|
evidence: {
|
||||||
|
branchName: error.branchName,
|
||||||
|
conflictingWorktreePath: error.conflictingWorktreePath,
|
||||||
|
},
|
||||||
|
underlyingError: error,
|
||||||
|
}, {
|
||||||
|
task,
|
||||||
|
retryCount: task.recoveryRetryCount ?? 0,
|
||||||
|
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
|
||||||
|
if (decision.action === "pause") {
|
||||||
|
await this.store.updateTask(task.id, {
|
||||||
|
status: "failed",
|
||||||
|
error: conflictMessage,
|
||||||
|
branch: error.branchName,
|
||||||
|
worktree: error.conflictingWorktreePath,
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "branch-conflict-unrecoverable",
|
||||||
|
});
|
||||||
|
await this.persistTokenUsage(task.id);
|
||||||
|
executorLog.warn(`✗ ${task.id} branch conflict sticky failure: ${error.branchName} @ ${error.conflictingWorktreePath}`);
|
||||||
|
this.options.onError?.(task, error);
|
||||||
|
return "sticky";
|
||||||
|
}
|
||||||
|
|
||||||
|
return "retry";
|
||||||
}
|
}
|
||||||
|
|
||||||
private async createWorktree(
|
private async createWorktree(
|
||||||
|
|||||||
@@ -100,6 +100,10 @@ export type DatabaseMutationType =
|
|||||||
| "task:dependency:add"
|
| "task:dependency:add"
|
||||||
| "task:auto-recover-already-merged"
|
| "task:auto-recover-already-merged"
|
||||||
| "task:auto-recover-completion-fanout"
|
| "task:auto-recover-completion-fanout"
|
||||||
|
| "auto-recovery:classify-decision"
|
||||||
|
| "auto-recovery:retry-issued"
|
||||||
|
| "auto-recovery:ai-session-spawned"
|
||||||
|
| "auto-recovery:pause-because-destructive-ambiguity"
|
||||||
| "document:write"
|
| "document:write"
|
||||||
| "workflow-step:result"
|
| "workflow-step:result"
|
||||||
| "agent:create:requested"
|
| "agent:create:requested"
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ import { extractMissingWorktreePathFromSessionStartFailure, isMissingWorktreeSes
|
|||||||
import { classifyError, extractMissingModulePath, isOperatorActionableAgentError, isStaleWorktreeModuleResolutionError } from "./transient-error-detector.js";
|
import { classifyError, extractMissingModulePath, isOperatorActionableAgentError, isStaleWorktreeModuleResolutionError } from "./transient-error-detector.js";
|
||||||
import { deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js";
|
import { deriveTaskIdFromFusionBranch, inspectBranchConflict, listUniqueBranchCommits } from "./branch-conflicts.js";
|
||||||
import { createRunAuditor, generateSyntheticRunId } from "./run-audit.js";
|
import { createRunAuditor, generateSyntheticRunId } from "./run-audit.js";
|
||||||
|
import { AutoRecoveryDispatcher } from "./auto-recovery.js";
|
||||||
|
|
||||||
const log = createLogger("self-healing");
|
const log = createLogger("self-healing");
|
||||||
const execAsync = promisify(exec);
|
const execAsync = promisify(exec);
|
||||||
@@ -172,6 +173,7 @@ export interface SelfHealingOptions {
|
|||||||
staleMergingFanoutMinAgeMs?: number;
|
staleMergingFanoutMinAgeMs?: number;
|
||||||
hasActiveAgentExecution?: (agentId: string) => boolean;
|
hasActiveAgentExecution?: (agentId: string) => boolean;
|
||||||
restartDurableAgentHeartbeat?: (agentId: string, context: { reason: string; attempt: number }) => Promise<boolean>;
|
restartDurableAgentHeartbeat?: (agentId: string, context: { reason: string; attempt: number }) => Promise<boolean>;
|
||||||
|
autoRecoveryDispatcher?: AutoRecoveryDispatcher;
|
||||||
}
|
}
|
||||||
|
|
||||||
const APPROVED_TRIAGE_RECOVERY_GRACE_MS = 60_000;
|
const APPROVED_TRIAGE_RECOVERY_GRACE_MS = 60_000;
|
||||||
@@ -1735,14 +1737,39 @@ export class SelfHealingManager {
|
|||||||
if (patchPath) {
|
if (patchPath) {
|
||||||
await this.store.logEntry(task.id, `Preserved uncommitted worktree changes before pause: ${patchPath}`);
|
await this.store.logEntry(task.id, `Preserved uncommitted worktree changes before pause: ${patchPath}`);
|
||||||
}
|
}
|
||||||
await this.store.updateTask(task.id, {
|
const dispatcher = this.options.autoRecoveryDispatcher ?? new AutoRecoveryDispatcher({
|
||||||
status: "failed",
|
taskStore: this.store,
|
||||||
error: `Task branch conflict: ${task.branch} is not safely reclaimable (${message})`,
|
auditEmitter: createRunAuditor(this.store, {
|
||||||
paused: true,
|
runId: generateSyntheticRunId("self-heal", task.id),
|
||||||
pausedReason: "branch-conflict-unrecoverable",
|
agentId: "self-healing",
|
||||||
|
taskId: task.id,
|
||||||
|
taskLineageId: task.lineageId,
|
||||||
|
phase: "reclaim-self-owned-branch-conflicts",
|
||||||
|
}),
|
||||||
});
|
});
|
||||||
await this.store.moveTask(task.id, "in-review");
|
const decision = await dispatcher.dispatch({
|
||||||
await this.store.logEntry(task.id, `Auto-recovery failed: branch conflict unrecoverable — ${message}`);
|
class: "branch-conflict-unrecoverable",
|
||||||
|
taskId: task.id,
|
||||||
|
pausedReason: "branch-conflict-unrecoverable",
|
||||||
|
evidence: {
|
||||||
|
branchName: task.branch,
|
||||||
|
worktreePath: task.worktree,
|
||||||
|
},
|
||||||
|
}, {
|
||||||
|
task,
|
||||||
|
retryCount: task.recoveryRetryCount ?? 0,
|
||||||
|
settings: (await this.store.getSettings()).autoRecovery ?? { mode: "deterministic-only", maxRetries: 3 },
|
||||||
|
});
|
||||||
|
if (decision.action === "pause") {
|
||||||
|
await this.store.updateTask(task.id, {
|
||||||
|
status: "failed",
|
||||||
|
error: `Task branch conflict: ${task.branch} is not safely reclaimable (${message})`,
|
||||||
|
paused: true,
|
||||||
|
pausedReason: "branch-conflict-unrecoverable",
|
||||||
|
});
|
||||||
|
await this.store.moveTask(task.id, "in-review");
|
||||||
|
await this.store.logEntry(task.id, `Auto-recovery failed: branch conflict unrecoverable — ${message}`);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user