Files
fusion/packages/engine/src/engine-errors.ts
flexi767 c71a9545b0 fix(engine): isolate provider rate-limit pauses (#2339)
## What changed

- Construct one `UsageLimitPauser` per project runtime and wire it into
both executor and triage.
- Replace the project-wide emergency stop for 429/quota failures with
provider-scoped task parking.
- Resolve execution, planning, validator, and merger providers for
active tasks; park only tasks routed through the unavailable provider.
- Preserve the actual reviewer provider on `ReviewerProviderError`, so a
Claude Plan Review 429 does not stop Codex work.
- Record `provider-rate-limit:<provider>` pause provenance without
storing provider response bodies in pause metadata.
- Run one daemon-owned provider-health monitor that probes only
providers with persisted rate-limit parks.
- Resume exact matching provider parks across every project only after
the existing authenticated usage probe succeeds and all reported
capacity windows are usable.
- Probe at five-minute intervals for the first five checks, then back
off independently per provider to 10/20/40/60 minutes with a one-hour
cap.

## Root cause and impact

The runtime refactor left `usageLimitPauser` undefined for
`TriageProcessor`. In the observed FN-922 incident, Claude Plan Review
returned four explicit 429 responses; Fusion backed off for roughly
60/120/240 seconds and then failed the task, but never invoked its pause
coordinator. The older coordinator also used `globalPause`, which would
terminate healthy sessions on every other provider.

After this change, active tasks using the unavailable provider are
parked while work routed exclusively through healthy providers
continues. Recovery is a provider-health state transition: the daemon
checks Claude/Codex authentication and metered capacity independently of
task execution, including after restart, and clears only exact
`provider-rate-limit:<provider>` parks. Logged-out, errored, exhausted,
manually paused, user-paused, and other-provider tasks remain parked.
Explicit global/engine pause controls remain unchanged.

## Surface enumeration

- executor usage-limit catches
- triage planner and Plan Review catches
- reviewer provider-error propagation
- merger usage-limit catches
- per-project runtime construction and wiring
- task model overrides plus project/global execution, planning,
validator, and merger resolution
- daemon startup/listen and shutdown lifecycle
- multi-project provider-probe deduplication
- Claude and Codex authenticated usage/capacity probes
- done/archived/already-paused task exclusions
- manual, user, generic, and other-provider pause provenance

## Symptom verification

**Original symptom:** Anthropic/Claude 429s retried and failed FN-922
without pausing Claude-routed work; a functioning global pauser would
also have stopped Codex, and provider parks had no positive-health
recovery path.

**Exact reproduction:** Raise `ReviewerProviderError("429
overloaded_error", "usage-limit", { provider: "anthropic" })` during
Plan Review with Anthropic and Codex tasks present, then return
logged-out/error/exhausted and finally healthy Claude usage responses
from the daemon probe.

**Assertion it is gone:** Anthropic-routed active tasks receive
`provider-rate-limit:anthropic`; Codex-only tasks are not paused and
`globalPause` is never changed. Unhealthy probes leave the Anthropic
tasks parked; a positive authenticated response with remaining capacity
resumes only exact Anthropic provider parks without executing a model
call as a probe.

## Validation

- `packages/engine/src/__tests__/usage-limit-detector.test.ts`: 49
passed
- `packages/dashboard/src/__tests__/provider-health-monitor.test.ts`: 8
passed
- Engine TypeScript check passed
- Dashboard server and app TypeScript checks passed
- Scoped ESLint passed
- Changeset strict format check passed
- Reapply script passed `bash -n`, two consecutive fixture applications,
and `node --check`


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Tasks paused due to a provider’s rate limits can now automatically
resume when capacity returns.
- Provider health is monitored in the background, including retry
backoff for unavailable providers.

- **Bug Fixes**
- Rate-limit issues now pause only affected provider-routed tasks
instead of stopping unrelated work.
- Provider failures are handled separately from invalid review results,
improving recovery behavior.
- Healthy providers remain available while another provider is
rate-limited.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: v <v@v.speedport.ip>
2026-07-21 17:08:44 -07:00

270 lines
8.8 KiB
TypeScript

/**
* Structured Engine Error Types — domain-specific error classes for the Fusion engine.
*
* These error types replace generic `catch (err)` blocks with typed, classifiable
* errors that callers can match on for domain-specific handling (retry, fail-fast,
* alerting, etc.).
*
* ## Hierarchy
*
* ```
* EngineError (base)
* ├── TransientError — temporary, retryable (network blip, 5xx, timeout)
* │ ├── NetworkError — connection refused/reset, DNS failure, socket hang-up
* │ ├── ServiceUnavailableError — upstream 5xx, overloaded, maintenance mode
* │ └── TimeoutError — request/operation exceeded deadline
* ├── PermanentError — non-retryable, task-defect or config error
* │ ├── ConfigurationError — bad env, missing keys, invalid settings
* │ └── ValidationError — schema violations, invalid inputs
* └── RateLimitError — quota/rate-limit, needs global pause (not local retry)
* ```
*
* ## Usage
*
* ```ts
* catch (err) {
* if (err instanceof TransientError) {
* // Move task to todo for retry
* } else if (err instanceof RateLimitError) {
* // Trigger global usage-limit pause
* } else if (err instanceof PermanentError) {
* // Mark task as failed
* }
* }
* ```
*/
import { isUsageLimitError } from "./usage-limit-detector.js";
import { isTransientError } from "./transient-error-detector.js";
// ── Base Error ──────────────────────────────────────────────────────────
/**
* Base class for all structured engine errors.
*
* Adds a `code` (machine-readable string) and optional `cause` chain to the
* standard Error. Subclasses set `retryable` to indicate whether the operation
* should be retried by the caller.
*/
export abstract class EngineError extends Error {
/** Machine-readable error code for programmatic matching. */
public readonly code: string;
/** Whether the caller should retry the operation. */
public readonly retryable: boolean;
/** Optional structured metadata for logging/metrics. */
public readonly details?: Record<string, unknown>;
constructor(
message: string,
code: string,
retryable: boolean,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, { cause });
this.name = this.constructor.name;
this.code = code;
this.retryable = retryable;
this.details = details;
}
}
// ── Transient Errors (retryable) ────────────────────────────────────────
/**
* A transient error — the operation failed due to a temporary condition
* that is expected to resolve on its own (network blip, brief service
* unavailability, timeout). Callers should retry with backoff.
*/
export class TransientError extends EngineError {
constructor(
message: string,
code: string = "TRANSIENT",
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, code, true, details, cause);
}
}
/**
* Network-level error — connection refused, DNS resolution failure,
* socket hang-up, TLS handshake failure, etc.
*/
export class NetworkError extends TransientError {
constructor(
message: string,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "NETWORK", details, cause);
}
}
/**
* Upstream service returned a 5xx or is temporarily unavailable
* (overloaded, maintenance mode).
*/
export class ServiceUnavailableError extends TransientError {
/** HTTP status code if available. */
public readonly statusCode?: number;
constructor(
message: string,
statusCode?: number,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "SERVICE_UNAVAILABLE", { ...details, statusCode }, cause);
this.statusCode = statusCode;
}
}
/**
* Operation or request exceeded its deadline / timeout.
*/
export class TimeoutError extends TransientError {
/** Configured timeout in milliseconds. */
public readonly timeoutMs?: number;
constructor(
message: string,
timeoutMs?: number,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "TIMEOUT", { ...details, timeoutMs }, cause);
this.timeoutMs = timeoutMs;
}
}
// ── Permanent Errors (non-retryable) ────────────────────────────────────
/**
* A permanent error — the operation failed due to a defect in the task,
* configuration, or input. Retrying will not help.
*/
export class PermanentError extends EngineError {
constructor(
message: string,
code: string = "PERMANENT",
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, code, false, details, cause);
}
}
/**
* Configuration error — missing env vars, invalid settings, bad keys.
*/
export class ConfigurationError extends PermanentError {
constructor(
message: string,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "CONFIGURATION", details, cause);
}
}
/**
* Validation error — schema violations, invalid inputs, malformed data.
*/
export class ValidationError extends PermanentError {
constructor(
message: string,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "VALIDATION", details, cause);
}
}
// ── Rate Limit Error (special — provider-scoped park, not local retry) ──
/**
* Rate-limit / usage-limit error. Unlike transient errors, these should
* NOT be retried indefinitely — after bounded backoff they park only the
* affected provider-routed task via UsageLimitPauser.
*/
export class RateLimitError extends EngineError {
/** Suggested retry-after in milliseconds (from Retry-After header or heuristic). */
public readonly retryAfterMs?: number;
constructor(
message: string,
retryAfterMs?: number,
details?: Record<string, unknown>,
cause?: Error,
) {
super(message, "RATE_LIMIT", false, { ...details, retryAfterMs }, cause);
this.retryAfterMs = retryAfterMs;
}
}
// ── Classification helpers ──────────────────────────────────────────────
/**
* Classify a raw error into a structured EngineError subtype.
*
* This bridges the gap between legacy string-based error classification
* (transient-error-detector, usage-limit-detector) and the new typed system.
* New code should throw typed errors directly; this function upgrades
* untyped errors from external libraries.
*
* @param err - The raw thrown value
* @returns A structured EngineError instance
*/
export function classifyThrownError(err: unknown): EngineError {
// Already structured — return as-is
if (err instanceof EngineError) {
return err;
}
const message = err instanceof Error ? err.message : String(err ?? "");
// Rate-limit (triggers a provider-scoped task park)
if (isUsageLimitError(message)) {
return new RateLimitError(message, undefined, undefined, err instanceof Error ? err : undefined);
}
// Network / connection errors
if (/ECONNREFUSED|connection refused|connection reset|socket hang up|EHOSTUNREACH|ENETUNREACH/i.test(message)) {
return new NetworkError(message, undefined, err instanceof Error ? err : undefined);
}
// Timeout errors
if (/ETIMEDOUT|timeout.*connection|connection.*timeout|deadline exceeded|timed out after \d+ms/i.test(message)) {
return new TimeoutError(message, undefined, undefined, err instanceof Error ? err : undefined);
}
// 5xx / service unavailable
if (/upstream connect error|disconnect\/reset before headers|remote connection failure|transport failure/i.test(message)) {
return new ServiceUnavailableError(message, undefined, undefined, err instanceof Error ? err : undefined);
}
if (/"type":"server_error"|"code":"server_error"/i.test(message)) {
return new ServiceUnavailableError(message, 500, undefined, err instanceof Error ? err : undefined);
}
// Generic transient (WebSocket errors, provider aborts, etc.)
if (isTransientError(message)) {
return new TransientError(message, "TRANSIENT", undefined, err instanceof Error ? err : undefined);
}
// Default: permanent
return new PermanentError(message, "UNKNOWN", undefined, err instanceof Error ? err : undefined);
}
/**
* Type guard: is the error retryable (transient)?
*/
export function isRetryableError(err: unknown): err is TransientError {
if (err instanceof TransientError) return true;
if (err instanceof EngineError) return err.retryable;
// Fall back to string-based detection for untyped errors
const message = err instanceof Error ? err.message : String(err ?? "");
return isTransientError(message);
}