FN-8654: rotate credential instances after provider limits
Retry provider-limit failures with eligible credential instances before falling back to existing pauses and backoff.
- Add a runtime-shared credential rotator with cooldown, exhaustion, and audit handling.
- Wire credential rotation into executor and heartbeat retry lanes while preserving user pause controls.
- Document the behavior and cover rotation, recovery, and retry paths.
Files changed:
.changeset/fn-8654-credential-instance-rotation.md | 7 +
AGENTS.md | 1 +
docs/architecture.md | 2 +-
docs/settings-reference.md | 4 +
.../__tests__/credential-instance-rotation.test.ts | 88 +++++++++++
.../__tests__/credential-rotation-lanes.test.ts | 20 +++
.../__tests__/credential-rotation-recovery.test.ts | 19 +++
.../__tests__/credential-rotation-wiring.test.ts | 15 ++
.../__tests__/rate-limit-retry-rotation.test.ts | 50 ++++++
.../src/__tests__/usage-limit-detector.test.ts | 14 ++
packages/engine/src/agent-heartbeat.ts | 102 +++++++++++-
.../engine/src/credential-instance-rotation.ts | 175 +++++++++++++++++++++
packages/engine/src/executor.ts | 141 +++++++++++++++--
packages/engine/src/index.ts | 7 +
packages/engine/src/project-engine.ts | 5 +
packages/engine/src/rate-limit-retry.ts | 32 +++-
packages/engine/src/runtimes/in-process-runtime.ts | 29 +++-
packages/engine/src/usage-limit-detector.ts | 18 ++-
18 files changed, 699 insertions(+), 30 deletions(-)
Fusion-Task-Id: FN-8654
Fusion-Task-Lineage: 44d63441-270c-4949-8c34-47ec4c9992e4
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
04c2bb4707