root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(engine): address PR #2027 review — tighten auth exclusions, accurate park accounting

Browse Source 
- Exclude revoked/suspended/disabled/deactivated keys, inactive subscriptions,
  and locked accounts from the transient-auth classifier: no retry fixes those,
  so they stay operator-actionable even inside an authentication_error envelope.
- Self-healing sweep logs unrecoverable-error parks separately from
  recovered-to-active agents (return value still counts actions taken).
- Document same-session retry continuation semantics at the heartbeat
  withRateLimitRetry call site (side-effect replay concern).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-07-12 13:41:39 -07:00
parent c4fad2d793
commit cbe07ee86b
4 changed files with 33 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
packages/engine/src/__tests__/transient-error-detector.test.ts
View File

@@ -461,6 +461,21 @@ describe("Transient Error Detector", () => {
expect(isOperatorActionableAgentError("missing OPENAI_API_KEY")).toBe(true);
});
it("keeps revoked/suspended/subscription credential states operator-actionable inside an authentication_error envelope (PR #2027 review)", () => {
const shapes = [
'401 {"type":"error","error":{"type":"authentication_error","message":"Access denied: API key revoked"}}',
'401 {"type":"error","error":{"type":"authentication_error","message":"account suspended"}}',
'401 {"type":"error","error":{"type":"authentication_error","message":"subscription inactive"}}',
'401 {"type":"error","error":{"type":"authentication_error","message":"this API key has been disabled"}}',
'401 {"type":"error","error":{"type":"authentication_error","message":"credentials deactivated"}}',
'401 {"type":"error","error":{"type":"authentication_error","message":"account is locked"}}',
];
for (const shape of shapes) {
expect(isTransientAuthCredentialError(shape)).toBe(false);
expect(classifyError(shape)).toBe("permanent");
}
});
it("does not match unrelated auth failures or empty input", () => {
expect(isTransientAuthCredentialError("Authentication failed for provider")).toBe(false);
expect(isOperatorActionableAgentError("Authentication failed for provider")).toBe(true);

3
packages/engine/src/agent-heartbeat.ts
View File

@@ -3425,6 +3425,9 @@ export class HeartbeatMonitor {
/*
FNXC:AgentHeartbeat 2026-07-12-20:10:
Heartbeat prompts must run under the same rate-limit + transient-auth retry wrapper as executor/triage/merger work. Claude Max OAuth tokens rotate mid-run (~8 h); the in-flight call 401s ("authentication_error: Invalid authentication credentials") even though refreshed credentials already exist, and the next attempt succeeds. Without this wrapper a routine token rotation failed the run, pushed every durable agent to `error`, and (via FN-7859 unrecoverable classification) parked them paused for operator action. Retrying in-run prevents the error state at the source; the durable-agent error-recovery budget stays the backstop for errors that escape.
FNXC:AgentHeartbeat 2026-07-12-21:05:
PR #2027 review (side-effect replay): the retry re-prompts the SAME session, whose transcript already contains any tool calls completed before the failure, so the model continues from its partial work rather than blindly re-executing it — the same continuation semantics executor/triage/merger rely on under this wrapper. A rotation 401 additionally fails on the turn's FIRST provider call (the stale token never reaches a tool call), so the dominant retry case has no partial work to duplicate.
*/
await withRateLimitRetry(() => promptWithFallback(session, executionPrompt), {
onRetry: (attempt, delayMs, retryError) => {

12
packages/engine/src/self-healing.ts
View File

@@ -10555,6 +10555,11 @@ export class SelfHealingManager {
}
let recovered = 0;
/*
FNXC:AgentHeartbeat 2026-07-12-21:05:
PR #2027 review: unrecoverable-error parks are a handled outcome of the sweep (the return value counts actions taken, preserving the existing caller contract), but they are NOT recoveries to active — the summary log must say "parked for operator action", never fold them into "→ active", or maintenance logs misreport agents that still need manual repair.
*/
let parkedUnrecoverable = 0;
for (const agent of orphaned) {
const updatedAt = Date.parse(agent.updatedAt ?? "");
const stuckForMs = Math.max(0, now - updatedAt);
@@ -10595,7 +10600,7 @@ export class SelfHealingManager {
source: "self-healing",
});
log.warn(`Suppressed durable-agent auto-restart for ${agent.id}: unrecoverable heartbeat error; paused for operator action`);
recovered++;
parkedUnrecoverable++;
continue;
}
if (isStaleMissingModule) {
@@ -10707,7 +10712,10 @@ export class SelfHealingManager {
if (recovered > 0) {
log.log(`Recovered ${recovered} orphaned agent(s) → active`);
}
return recovered;
if (parkedUnrecoverable > 0) {
log.warn(`Parked ${parkedUnrecoverable} durable agent(s) with unrecoverable errors for operator action (not recovered)`);
}
return recovered + parkedUnrecoverable;
} catch (err: unknown) {
const errorMessage = err instanceof Error ? err.message : String(err);
log.error(`Orphaned agent recovery failed: ${errorMessage}`);

6
packages/engine/src/transient-error-detector.ts
View File

@@ -111,8 +111,12 @@ Genuinely operator-actionable auth failures are excluded first: OAuth scope/perm
*/
const TRANSIENT_AUTH_CREDENTIAL_ROTATION_PATTERN =
/"type":\s*"authentication_error"|invalid authentication credentials|token[_\s]?expired/i;
/*
FNXC:Reliability-ErrorClassification 2026-07-12-21:05:
PR #2027 review: the `"type":"authentication_error"` envelope is intentionally broad (providers put rotation failures behind it with varying messages), so the exclusion list must carry the operator-actionable load. Beyond scope grants and invalid/missing API keys, exclude account/credential states no retry can fix: revoked/suspended/disabled/deactivated keys or accounts and inactive subscriptions. A message matching any of these stays permanent/operator-actionable even inside an authentication_error envelope; retries are pointless and would un-park agents a human must repair. Unmatched novel auth messages still classify transient, but the bounded heartbeat error-recovery budget re-parks them as `error-retry-exhausted` after a few attempts, so the failure mode is a handful of visible retries, not an unpark loop.
*/
const OPERATOR_ACTIONABLE_AUTH_EXCLUSION_PATTERN =
/oauth token does not meet scope|insufficient[_\s-]?scope|invalid[_\s-]?scope|invalid (?:api[_\s-]?key|x-api-key)|missing\s+(?:\S+\s+)?(?:api[_\s-]?)?key/i;
/oauth token does not meet scope|insufficient[_\s-]?scope|invalid[_\s-]?scope|invalid (?:api[_\s-]?key|x-api-key)|missing\s+(?:\S+\s+)?(?:api[_\s-]?)?key|revoked|suspend(?:ed)?|disabled|deactivated|subscription|account (?:is )?(?:locked|closed|inactive)|access denied/i;
/**
* Detect a transient authentication failure caused by credential rotation