fix(engine): classify provider request timeouts as transient

"Request timed out." is the literal default message of the Anthropic and OpenAI
SDKs' APIConnectionTimeoutError, surfaced to Fusion by checkSessionError after
pi-coding-agent exhausts its in-session retries. It matched none of the
connection-scoped timeout patterns, which deliberately excluded "general
timeouts", so it fell through to specifyTask's generic failure branch — the one
that restores status: null and writes no counter, no nextRecoveryAt, and no park.
Triage rediscovery then re-admitted the card on the very next poll, forever.

Measured before this change: 48 "Specification failed: Request timed out." events
across 10 tasks in 30 hours with zero backoff between attempts. FN-8950 alone
burned 8 consecutive attempts over ~8 hours and never reached implementation.
Across 2 days, 91 failed planning attempts averaged 33 minutes each — ~50 hours of
wall-clock producing nothing, 24% of all planning time.

Classifying these as transient routes them into the bounded recovery policy
(MAX_RECOVERY_RETRIES = 3, 60s/120s/300s jittered backoff) already used by the
connection-level patterns, so a provider blip costs three spaced retries instead
of an unbounded loop.

The pattern is anchored to "request timed out" rather than a bare timeout match:
agent log prose and verification output legitimately contain "timed out"
("BuildKit timed out", "stuck-kill unwind timeout"), and a broad pattern would
reclassify real permanent failures as retryable — the mistake the connection-only
rule was written to avoid. Regression tests pin both directions.

This does not affect model fallback, which pi decides internally and Fusion only
observes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-08-10 12:01:25 -07:00
parent 00ddafd5fe
commit 81139dbbd0
3 changed files with 69 additions and 1 deletions

View File

@@ -0,0 +1,7 @@
---
"@runfusion/fusion": patch
---
summary: Provider request timeouts now retry with backoff instead of looping planning forever.
category: fix
dev: Adds `/\brequest timed out\b/i` and `/\bAPIConnectionTimeoutError\b/i` to `TRANSIENT_ERROR_PATTERNS` (`packages/engine/src/errors/transient-error-patterns.ts`). The Anthropic/OpenAI SDK `APIConnectionTimeoutError` default message `"Request timed out."` matched no pattern, so it fell through to `specifyTask`'s generic failure branch which restores `status: null` with no counter and no `nextRecoveryAt` — triage rediscovery then re-admitted the card every poll indefinitely. Now routes into the bounded `MAX_RECOVERY_RETRIES` (3) + 60s/120s/300s backoff policy. Pattern is anchored to "request timed out" rather than a bare timeout match so agent prose and verification output ("BuildKit timed out", "Test suite timed out after 30000ms") stay permanent. Does not affect model fallback, which pi decides internally.

View File

@@ -179,6 +179,36 @@ describe("Transient Error Detector", () => {
expect(isTransientError("Aborted")).toBe(false);
expect(isTransientError("The operation was aborted by user")).toBe(false);
});
/*
FNXC:Reliability-ErrorClassification 2026-08-10-18:32:
`"Request timed out."` is the Anthropic/OpenAI SDK `APIConnectionTimeoutError` default message.
It previously matched no pattern, so it fell through to the generic planning-failure branch that
writes no counter and no backoff — triage re-admitted the card every poll, forever. Measured:
48 such events across 10 tasks in 30 hours with 0-minute gaps between attempts. Classifying it
transient routes it into the bounded MAX_RECOVERY_RETRIES=3 + backoff policy.
*/
it("classifies a provider request timeout as transient", () => {
expect(isTransientError("Request timed out.")).toBe(true);
expect(isTransientError("Specification failed: Request timed out.")).toBe(true);
expect(isTransientError("request timed out")).toBe(true);
expect(isTransientError("APIConnectionTimeoutError: Request timed out.")).toBe(true);
});
/*
The pattern is anchored to "request timed out" rather than a bare /timed? out/ because agent log
prose and verification output legitimately contain "timed out". A broad match would reclassify
real, permanent task failures as retryable — the mistake the connection-only rule was written to
avoid. These strings are all observed in production task logs.
*/
it("does NOT treat non-request timeout prose as transient", () => {
expect(isTransientError("BuildKit timed out while building the image")).toBe(false);
expect(isTransientError("force-requeue after stuck-kill unwind timeout")).toBe(false);
expect(isTransientError("Test suite timed out after 30000ms")).toBe(false);
expect(isTransientError("Verification command timed out")).toBe(false);
// Pre-existing negative cases that must stay negative: "timeout" is not "timed out".
expect(isTransientError("Request timeout")).toBe(false);
});
});
describe("isNonPlanDefectPlanReviewFailure", () => {

View File

@@ -47,10 +47,41 @@ export const TRANSIENT_ERROR_PATTERNS: RegExp[] = [
/ETIMEDOUT/i,
/socket hang up/i,
// Timeout patterns (only when related to connections, not general timeouts)
// Timeout patterns (connection-scoped — see the provider-request-timeout block below for the
// narrow exception covering SDK request timeouts)
/timeout.*connection/i,
/connection.*timeout/i,
/*
FNXC:Reliability-ErrorClassification 2026-08-10-18:32:
Provider REQUEST timeouts are transient. `"Request timed out."` is the literal default message of
the Anthropic and OpenAI SDKs' `APIConnectionTimeoutError`, surfaced to Fusion by
`checkSessionError` after pi-coding-agent exhausts its own in-session retries.
Previously the connection-scoped patterns above deliberately excluded "general timeouts", so this
string matched NOTHING and fell through to the generic failure branch in `specifyTask`
(triage.ts) — which restores `status: null` and writes no counter, no `nextRecoveryAt`, and no
park. Triage rediscovery then re-admitted the card on the very next poll, forever.
Measured impact before this change: 48 `Specification failed: Request timed out.` events across
10 tasks in 30 hours, with zero backoff between attempts (FN-8950 alone burned 8 consecutive
attempts over ~8 hours and never reached implementation). Failed planning attempts averaged 33
minutes each — 91 of them in 2 days, ~50 hours of wall-clock producing nothing, which was 24% of
all planning time. Classifying these as transient routes them into the BOUNDED recovery policy
(`MAX_RECOVERY_RETRIES` = 3 with 60s/120s/300s jittered backoff via `nextRecoveryAt`) that the
connection-level patterns already use, so a provider blip costs three spaced retries instead of an
unbounded loop.
ANCHORING IS LOAD-BEARING: match `request timed out`, never a bare /timed? out/. Agent log prose
and verification output legitimately contain "timed out" (observed: "BuildKit timed out",
"stuck-kill unwind timeout"), and a broad pattern would reclassify real, permanent task failures
as retryable — the exact mistake the original connection-only rule was written to avoid. This
does NOT affect model fallback, which pi decides internally and Fusion only observes
(`auth/fallback-model-observer.ts`).
*/
/\brequest timed out\b/i,
/\bAPIConnectionTimeoutError\b/i,
// AI provider abort errors — temporary request cancellations (e.g., Anthropic streaming aborts)
// These occur when the provider's infrastructure drops an in-flight request.
/request was aborted/i,