A rate-limited Plan Review re-ran every 30s for hours (~1,900 requests
per 5h window, reviewerFallbackRetryCount observed past 100), which is
the request volume that trips a provider's low-interactivity throttle —
so the retry storm prolonged the very outage it was retrying.
Root cause: runPlanReviewBeforeExecution catches every reviewStep throw
inline to keep triage alive, which converts them all to an UNAVAILABLE
verdict. That laundering had two consequences the earlier fixes missed:
FN-8006 terminalized RetryStormError and the reviewer started throwing
ReviewerProviderError for 429s, but a ReviewerProviderError still landed
in the UNAVAILABLE park — a FIXED 30s nextRecoveryAt with no attempt
counter and no cap. The reviewer's own escalation contract ("escalate so
UsageLimitPauser pauses every lane") held only on the executor path,
because the inline catch hid the error from triage's usage-limit handler
in specifyTask.
- triage: fire usageLimitPauser.onUsageLimitHit for usage-limit reviewer
failures, so a 429 pauses every lane instead of re-parking one task.
- triage: re-park via computeRecoveryDecision (60s/120s/240s, ±10%
jitter) and terminalize at MAX_RECOVERY_RETRIES. A reviewer that never
yields a verdict is a real failure and must surface, not spin.
- triage: clear the borrowed recoveryRetryCount budget on any real
verdict, so surviving an outage cannot shorten the executor's later
transient budget.
- core: RetryStormError takes an optional cause, surfaced as
underlyingError in serializeRetryStormError and folded into the
message, so a cap no longer masks the real error. recordRetry threads
it from the reviewer's error path.
Surface enumeration: the park is driven by a thrown provider error, a
thrown generic error, and a plain UNAVAILABLE verdict with no throw.
All three are covered — a repro pinned only to the reported 429 would
leave the other two spinning on the old fixed timer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
08a10bf486