root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(FN-775): add recoverable-retry with bounded exponential backoff

Browse Source 
- Add  state to tasks: persisted in DB, gates scheduler pickup to prevent immediate retry of transient failures
- Introduce shared recovery-policy module with bounded exponential backoff (1s → 60s, max 5 attempts)
- Wire recovery policy into executor, scheduler, and triage so all agents respect the same retry cadence
- Persist retry state (attempt count, next eligible time) in task metadata via store and DB schema
- Add DB migration for new retry columns and update schema tests
- Update README with recovery policy documentation
- Refactor dashboard Header component and styles, consolidate header tests
- Fix session-files route tests to align with updated route signatures
This commit is contained in:
gsxdsm
2026-04-03 07:58:45 -07:00
parent ee8f19fc93
commit 51855f43d9
15 changed files with 961 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
packages/engine/src/executor.ts
View File

@@ -15,6 +15,7 @@ import { executorLog, reviewerLog } from "./logger.js";
import { isUsageLimitError, checkSessionError, type UsageLimitPauser } from "./usage-limit-detector.js";
import { isTransientError } from "./transient-error-detector.js";
import { withRateLimitRetry } from "./rate-limit-retry.js";
import { computeRecoveryDecision, formatDelay, MAX_RECOVERY_RETRIES } from "./recovery-policy.js";
import type { StuckTaskDetector } from "./stuck-task-detector.js";
// Re-export for backward compatibility (tests import from executor.ts)
@@ -723,10 +724,35 @@ export class TaskExecutor {
if (this.options.usageLimitPauser && isUsageLimitError(err.message)) {
await this.options.usageLimitPauser.onUsageLimitHit("executor", task.id, err.message);
} else if (isTransientError(err.message)) {
// Transient network/infrastructure error — retry instead of failing
executorLog.warn(`⚡ ${task.id} transient error — moving to todo for retry: ${err.message}`);
await this.store.logEntry(task.id, `Transient error (will retry): ${err.message}`);
await this.store.moveTask(task.id, "todo");
// Transient network/infrastructure error — use bounded recovery policy
const decision = computeRecoveryDecision({
recoveryRetryCount: task.recoveryRetryCount,
nextRecoveryAt: task.nextRecoveryAt,
});
if (decision.shouldRetry) {
const attempt = decision.nextState.recoveryRetryCount;
const delay = formatDelay(decision.delayMs);
executorLog.warn(`⚡ ${task.id} transient error — retry ${attempt}/${MAX_RECOVERY_RETRIES} in ${delay}: ${err.message}`);
await this.store.logEntry(task.id, `Transient error (retry ${attempt}/${MAX_RECOVERY_RETRIES} in ${delay}): ${err.message}`);
await this.store.updateTask(task.id, {
recoveryRetryCount: decision.nextState.recoveryRetryCount,
nextRecoveryAt: decision.nextState.nextRecoveryAt,
});
await this.store.moveTask(task.id, "todo");
return;
}
// Recovery budget exhausted — escalate to real failure
executorLog.error(`✗ ${task.id} transient error retries exhausted (${MAX_RECOVERY_RETRIES} attempts): ${err.message}`);
await this.store.logEntry(task.id, `Transient error retries exhausted after ${MAX_RECOVERY_RETRIES} attempts: ${err.message}`);
await this.store.updateTask(task.id, {
status: "failed",
error: err.message,
recoveryRetryCount: null,
nextRecoveryAt: null,
});
this.options.onError?.(task, err);
return;
}
executorLog.error(`✗ ${task.id} execution failed:`, err.message);