## What changed
- Construct one `UsageLimitPauser` per project runtime and wire it into
both executor and triage.
- Replace the project-wide emergency stop for 429/quota failures with
provider-scoped task parking.
- Resolve execution, planning, validator, and merger providers for
active tasks; park only tasks routed through the unavailable provider.
- Preserve the actual reviewer provider on `ReviewerProviderError`, so a
Claude Plan Review 429 does not stop Codex work.
- Record `provider-rate-limit:<provider>` pause provenance without
storing provider response bodies in pause metadata.
- Run one daemon-owned provider-health monitor that probes only
providers with persisted rate-limit parks.
- Resume exact matching provider parks across every project only after
the existing authenticated usage probe succeeds and all reported
capacity windows are usable.
- Probe at five-minute intervals for the first five checks, then back
off independently per provider to 10/20/40/60 minutes with a one-hour
cap.
## Root cause and impact
The runtime refactor left `usageLimitPauser` undefined for
`TriageProcessor`. In the observed FN-922 incident, Claude Plan Review
returned four explicit 429 responses; Fusion backed off for roughly
60/120/240 seconds and then failed the task, but never invoked its pause
coordinator. The older coordinator also used `globalPause`, which would
terminate healthy sessions on every other provider.
After this change, active tasks using the unavailable provider are
parked while work routed exclusively through healthy providers
continues. Recovery is a provider-health state transition: the daemon
checks Claude/Codex authentication and metered capacity independently of
task execution, including after restart, and clears only exact
`provider-rate-limit:<provider>` parks. Logged-out, errored, exhausted,
manually paused, user-paused, and other-provider tasks remain parked.
Explicit global/engine pause controls remain unchanged.
## Surface enumeration
- executor usage-limit catches
- triage planner and Plan Review catches
- reviewer provider-error propagation
- merger usage-limit catches
- per-project runtime construction and wiring
- task model overrides plus project/global execution, planning,
validator, and merger resolution
- daemon startup/listen and shutdown lifecycle
- multi-project provider-probe deduplication
- Claude and Codex authenticated usage/capacity probes
- done/archived/already-paused task exclusions
- manual, user, generic, and other-provider pause provenance
## Symptom verification
**Original symptom:** Anthropic/Claude 429s retried and failed FN-922
without pausing Claude-routed work; a functioning global pauser would
also have stopped Codex, and provider parks had no positive-health
recovery path.
**Exact reproduction:** Raise `ReviewerProviderError("429
overloaded_error", "usage-limit", { provider: "anthropic" })` during
Plan Review with Anthropic and Codex tasks present, then return
logged-out/error/exhausted and finally healthy Claude usage responses
from the daemon probe.
**Assertion it is gone:** Anthropic-routed active tasks receive
`provider-rate-limit:anthropic`; Codex-only tasks are not paused and
`globalPause` is never changed. Unhealthy probes leave the Anthropic
tasks parked; a positive authenticated response with remaining capacity
resumes only exact Anthropic provider parks without executing a model
call as a probe.
## Validation
- `packages/engine/src/__tests__/usage-limit-detector.test.ts`: 49
passed
- `packages/dashboard/src/__tests__/provider-health-monitor.test.ts`: 8
passed
- Engine TypeScript check passed
- Dashboard server and app TypeScript checks passed
- Scoped ESLint passed
- Changeset strict format check passed
- Reapply script passed `bash -n`, two consecutive fixture applications,
and `node --check`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Tasks paused due to a provider’s rate limits can now automatically
resume when capacity returns.
- Provider health is monitored in the background, including retry
backoff for unavailable providers.
- **Bug Fixes**
- Rate-limit issues now pause only affected provider-routed tasks
instead of stopping unrelated work.
- Provider failures are handled separately from invalid review results,
improving recovery behavior.
- Healthy providers remain available while another provider is
rate-limited.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: v <v@v.speedport.ip>
270 lines
8.8 KiB
TypeScript
270 lines
8.8 KiB
TypeScript
/**
|
|
* Structured Engine Error Types — domain-specific error classes for the Fusion engine.
|
|
*
|
|
* These error types replace generic `catch (err)` blocks with typed, classifiable
|
|
* errors that callers can match on for domain-specific handling (retry, fail-fast,
|
|
* alerting, etc.).
|
|
*
|
|
* ## Hierarchy
|
|
*
|
|
* ```
|
|
* EngineError (base)
|
|
* ├── TransientError — temporary, retryable (network blip, 5xx, timeout)
|
|
* │ ├── NetworkError — connection refused/reset, DNS failure, socket hang-up
|
|
* │ ├── ServiceUnavailableError — upstream 5xx, overloaded, maintenance mode
|
|
* │ └── TimeoutError — request/operation exceeded deadline
|
|
* ├── PermanentError — non-retryable, task-defect or config error
|
|
* │ ├── ConfigurationError — bad env, missing keys, invalid settings
|
|
* │ └── ValidationError — schema violations, invalid inputs
|
|
* └── RateLimitError — quota/rate-limit, needs global pause (not local retry)
|
|
* ```
|
|
*
|
|
* ## Usage
|
|
*
|
|
* ```ts
|
|
* catch (err) {
|
|
* if (err instanceof TransientError) {
|
|
* // Move task to todo for retry
|
|
* } else if (err instanceof RateLimitError) {
|
|
* // Trigger global usage-limit pause
|
|
* } else if (err instanceof PermanentError) {
|
|
* // Mark task as failed
|
|
* }
|
|
* }
|
|
* ```
|
|
*/
|
|
|
|
import { isUsageLimitError } from "./usage-limit-detector.js";
|
|
import { isTransientError } from "./transient-error-detector.js";
|
|
|
|
// ── Base Error ──────────────────────────────────────────────────────────
|
|
|
|
/**
|
|
* Base class for all structured engine errors.
|
|
*
|
|
* Adds a `code` (machine-readable string) and optional `cause` chain to the
|
|
* standard Error. Subclasses set `retryable` to indicate whether the operation
|
|
* should be retried by the caller.
|
|
*/
|
|
export abstract class EngineError extends Error {
|
|
/** Machine-readable error code for programmatic matching. */
|
|
public readonly code: string;
|
|
/** Whether the caller should retry the operation. */
|
|
public readonly retryable: boolean;
|
|
/** Optional structured metadata for logging/metrics. */
|
|
public readonly details?: Record<string, unknown>;
|
|
|
|
constructor(
|
|
message: string,
|
|
code: string,
|
|
retryable: boolean,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, { cause });
|
|
this.name = this.constructor.name;
|
|
this.code = code;
|
|
this.retryable = retryable;
|
|
this.details = details;
|
|
}
|
|
}
|
|
|
|
// ── Transient Errors (retryable) ────────────────────────────────────────
|
|
|
|
/**
|
|
* A transient error — the operation failed due to a temporary condition
|
|
* that is expected to resolve on its own (network blip, brief service
|
|
* unavailability, timeout). Callers should retry with backoff.
|
|
*/
|
|
export class TransientError extends EngineError {
|
|
constructor(
|
|
message: string,
|
|
code: string = "TRANSIENT",
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, code, true, details, cause);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Network-level error — connection refused, DNS resolution failure,
|
|
* socket hang-up, TLS handshake failure, etc.
|
|
*/
|
|
export class NetworkError extends TransientError {
|
|
constructor(
|
|
message: string,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "NETWORK", details, cause);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Upstream service returned a 5xx or is temporarily unavailable
|
|
* (overloaded, maintenance mode).
|
|
*/
|
|
export class ServiceUnavailableError extends TransientError {
|
|
/** HTTP status code if available. */
|
|
public readonly statusCode?: number;
|
|
|
|
constructor(
|
|
message: string,
|
|
statusCode?: number,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "SERVICE_UNAVAILABLE", { ...details, statusCode }, cause);
|
|
this.statusCode = statusCode;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Operation or request exceeded its deadline / timeout.
|
|
*/
|
|
export class TimeoutError extends TransientError {
|
|
/** Configured timeout in milliseconds. */
|
|
public readonly timeoutMs?: number;
|
|
|
|
constructor(
|
|
message: string,
|
|
timeoutMs?: number,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "TIMEOUT", { ...details, timeoutMs }, cause);
|
|
this.timeoutMs = timeoutMs;
|
|
}
|
|
}
|
|
|
|
// ── Permanent Errors (non-retryable) ────────────────────────────────────
|
|
|
|
/**
|
|
* A permanent error — the operation failed due to a defect in the task,
|
|
* configuration, or input. Retrying will not help.
|
|
*/
|
|
export class PermanentError extends EngineError {
|
|
constructor(
|
|
message: string,
|
|
code: string = "PERMANENT",
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, code, false, details, cause);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Configuration error — missing env vars, invalid settings, bad keys.
|
|
*/
|
|
export class ConfigurationError extends PermanentError {
|
|
constructor(
|
|
message: string,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "CONFIGURATION", details, cause);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Validation error — schema violations, invalid inputs, malformed data.
|
|
*/
|
|
export class ValidationError extends PermanentError {
|
|
constructor(
|
|
message: string,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "VALIDATION", details, cause);
|
|
}
|
|
}
|
|
|
|
// ── Rate Limit Error (special — provider-scoped park, not local retry) ──
|
|
|
|
/**
|
|
* Rate-limit / usage-limit error. Unlike transient errors, these should
|
|
* NOT be retried indefinitely — after bounded backoff they park only the
|
|
* affected provider-routed task via UsageLimitPauser.
|
|
*/
|
|
export class RateLimitError extends EngineError {
|
|
/** Suggested retry-after in milliseconds (from Retry-After header or heuristic). */
|
|
public readonly retryAfterMs?: number;
|
|
|
|
constructor(
|
|
message: string,
|
|
retryAfterMs?: number,
|
|
details?: Record<string, unknown>,
|
|
cause?: Error,
|
|
) {
|
|
super(message, "RATE_LIMIT", false, { ...details, retryAfterMs }, cause);
|
|
this.retryAfterMs = retryAfterMs;
|
|
}
|
|
}
|
|
|
|
// ── Classification helpers ──────────────────────────────────────────────
|
|
|
|
/**
|
|
* Classify a raw error into a structured EngineError subtype.
|
|
*
|
|
* This bridges the gap between legacy string-based error classification
|
|
* (transient-error-detector, usage-limit-detector) and the new typed system.
|
|
* New code should throw typed errors directly; this function upgrades
|
|
* untyped errors from external libraries.
|
|
*
|
|
* @param err - The raw thrown value
|
|
* @returns A structured EngineError instance
|
|
*/
|
|
export function classifyThrownError(err: unknown): EngineError {
|
|
// Already structured — return as-is
|
|
if (err instanceof EngineError) {
|
|
return err;
|
|
}
|
|
|
|
const message = err instanceof Error ? err.message : String(err ?? "");
|
|
|
|
// Rate-limit (triggers a provider-scoped task park)
|
|
if (isUsageLimitError(message)) {
|
|
return new RateLimitError(message, undefined, undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
// Network / connection errors
|
|
if (/ECONNREFUSED|connection refused|connection reset|socket hang up|EHOSTUNREACH|ENETUNREACH/i.test(message)) {
|
|
return new NetworkError(message, undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
// Timeout errors
|
|
if (/ETIMEDOUT|timeout.*connection|connection.*timeout|deadline exceeded|timed out after \d+ms/i.test(message)) {
|
|
return new TimeoutError(message, undefined, undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
// 5xx / service unavailable
|
|
if (/upstream connect error|disconnect\/reset before headers|remote connection failure|transport failure/i.test(message)) {
|
|
return new ServiceUnavailableError(message, undefined, undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
if (/"type":"server_error"|"code":"server_error"/i.test(message)) {
|
|
return new ServiceUnavailableError(message, 500, undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
// Generic transient (WebSocket errors, provider aborts, etc.)
|
|
if (isTransientError(message)) {
|
|
return new TransientError(message, "TRANSIENT", undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
// Default: permanent
|
|
return new PermanentError(message, "UNKNOWN", undefined, err instanceof Error ? err : undefined);
|
|
}
|
|
|
|
/**
|
|
* Type guard: is the error retryable (transient)?
|
|
*/
|
|
export function isRetryableError(err: unknown): err is TransientError {
|
|
if (err instanceof TransientError) return true;
|
|
if (err instanceof EngineError) return err.retryable;
|
|
// Fall back to string-based detection for untyped errors
|
|
const message = err instanceof Error ? err.message : String(err ?? "");
|
|
return isTransientError(message);
|
|
}
|