Commit Graph

3 Commits

Author SHA1 Message Date
flexi767
c71a9545b0 fix(engine): isolate provider rate-limit pauses (#2339)
## What changed

- Construct one `UsageLimitPauser` per project runtime and wire it into
both executor and triage.
- Replace the project-wide emergency stop for 429/quota failures with
provider-scoped task parking.
- Resolve execution, planning, validator, and merger providers for
active tasks; park only tasks routed through the unavailable provider.
- Preserve the actual reviewer provider on `ReviewerProviderError`, so a
Claude Plan Review 429 does not stop Codex work.
- Record `provider-rate-limit:<provider>` pause provenance without
storing provider response bodies in pause metadata.
- Run one daemon-owned provider-health monitor that probes only
providers with persisted rate-limit parks.
- Resume exact matching provider parks across every project only after
the existing authenticated usage probe succeeds and all reported
capacity windows are usable.
- Probe at five-minute intervals for the first five checks, then back
off independently per provider to 10/20/40/60 minutes with a one-hour
cap.

## Root cause and impact

The runtime refactor left `usageLimitPauser` undefined for
`TriageProcessor`. In the observed FN-922 incident, Claude Plan Review
returned four explicit 429 responses; Fusion backed off for roughly
60/120/240 seconds and then failed the task, but never invoked its pause
coordinator. The older coordinator also used `globalPause`, which would
terminate healthy sessions on every other provider.

After this change, active tasks using the unavailable provider are
parked while work routed exclusively through healthy providers
continues. Recovery is a provider-health state transition: the daemon
checks Claude/Codex authentication and metered capacity independently of
task execution, including after restart, and clears only exact
`provider-rate-limit:<provider>` parks. Logged-out, errored, exhausted,
manually paused, user-paused, and other-provider tasks remain parked.
Explicit global/engine pause controls remain unchanged.

## Surface enumeration

- executor usage-limit catches
- triage planner and Plan Review catches
- reviewer provider-error propagation
- merger usage-limit catches
- per-project runtime construction and wiring
- task model overrides plus project/global execution, planning,
validator, and merger resolution
- daemon startup/listen and shutdown lifecycle
- multi-project provider-probe deduplication
- Claude and Codex authenticated usage/capacity probes
- done/archived/already-paused task exclusions
- manual, user, generic, and other-provider pause provenance

## Symptom verification

**Original symptom:** Anthropic/Claude 429s retried and failed FN-922
without pausing Claude-routed work; a functioning global pauser would
also have stopped Codex, and provider parks had no positive-health
recovery path.

**Exact reproduction:** Raise `ReviewerProviderError("429
overloaded_error", "usage-limit", { provider: "anthropic" })` during
Plan Review with Anthropic and Codex tasks present, then return
logged-out/error/exhausted and finally healthy Claude usage responses
from the daemon probe.

**Assertion it is gone:** Anthropic-routed active tasks receive
`provider-rate-limit:anthropic`; Codex-only tasks are not paused and
`globalPause` is never changed. Unhealthy probes leave the Anthropic
tasks parked; a positive authenticated response with remaining capacity
resumes only exact Anthropic provider parks without executing a model
call as a probe.

## Validation

- `packages/engine/src/__tests__/usage-limit-detector.test.ts`: 49
passed
- `packages/dashboard/src/__tests__/provider-health-monitor.test.ts`: 8
passed
- Engine TypeScript check passed
- Dashboard server and app TypeScript checks passed
- Scoped ESLint passed
- Changeset strict format check passed
- Reapply script passed `bash -n`, two consecutive fixture applications,
and `node --check`


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Tasks paused due to a provider’s rate limits can now automatically
resume when capacity returns.
- Provider health is monitored in the background, including retry
backoff for unavailable providers.

- **Bug Fixes**
- Rate-limit issues now pause only affected provider-routed tasks
instead of stopping unrelated work.
- Provider failures are handled separately from invalid review results,
improving recovery behavior.
- Healthy providers remain available while another provider is
rate-limited.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: v <v@v.speedport.ip>
2026-07-21 17:08:44 -07:00
gsxdsm
a27921a276 fix: address PR review feedback (#1467)
- guard optional project selection callbacks and keep bookmarked search matches visible
- use structured timeout/rate-limit handling in retry backoff
- add focused regressions for selector and retry behavior

Note: pnpm test passed the merge gate but hit pre-existing non-blocking engine mock failures in changed-package tests.
2026-06-07 23:03:38 -07:00
Fusion Worker
1afdbbbb47 feat(engine): add structured error types and general-purpose retry with exponential backoff
Add domain-specific error classes (engine-errors.ts) replacing generic
catch blocks with typed, classifiable errors:
- EngineError base with code/retryable/details fields
- TransientError hierarchy: NetworkError, ServiceUnavailableError, TimeoutError
- PermanentError hierarchy: ConfigurationError, ValidationError
- RateLimitError (triggers global pause, not local retry)
- classifyThrownError() bridge from legacy string-based detection

Add general-purpose retry with exponential backoff (retry-with-backoff.ts):
- withRetry() wraps async ops with configurable retries for transient errors
- Exponential backoff: delay = min(baseMs * 2^attempt, maxMs)
- Three jitter strategies: full (default), equal, none
- Per-attempt timeout support via timeoutMs option
- AbortSignal integration for task pause/cancel/shutdown
- Custom isRetryable check for domain-specific retry logic
- withRetryResult() variant returning retry metadata
- Never retries rate-limit errors (delegates to global pause)

53 new tests covering error hierarchy, backoff math, cancellable sleep,
retry on transient errors, non-retryable fail-fast, rate-limit bypass,
abort cancellation, per-attempt timeout, custom predicates.
2026-06-07 00:15:48 +07:00