root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(FN-5627): self-heal transient merge failures stuck at mergeRetries=3

Browse Source 
After the FN-5627 merger fix (b2d547eae, 230f6f45b) landed, two in-review
tasks (FN-5628, FN-5632) remained stuck at mergeRetries=3 with
status=failed because the merger correctly identified transient failure
classes but had no auto-recovery path \u2014 the AUTO_MERGE_COOLDOWN_MS reset
takes hours and gives up too easily.

Failure classes covered:
- lease-handoff-failed: target-not-queued (FN-5353/FN-5363 race where the
  merge queue lease was cleared between enqueue and handoff acquisition).
- Legacy same-SHA spurious 'Integration branch X advanced concurrently
  (expected SHA, observed SHA)' errors from pre-FN-5627 code paths.

Implementation:
- New MergeDetails.transientRecoveryCount field tracks per-task recovery
  attempts, bounded by MAX_TRANSIENT_MERGE_RECOVERIES = 2.
- New classifyTransientMergeError() string matcher in self-healing.ts
  identifies recoverable classes by error pattern. Returns null for
  genuine merge failures (verification, conflicts, real concurrent
  advances with different SHAs).
- SelfHealingManager.recoverTransientMergeFailures() sweep finds
  matching in-review tasks, resets mergeRetries=0, clears status/error,
  increments recovery count, re-enqueues via requeueForAutoMerge.
- Wired into BOTH startup recovery and periodic Batch 2 maintenance loop.
- Emits merger:transient-failure-auto-recovered (recovered) and
  merger:transient-failure-budget-exhausted (terminal) audit events.

No-op when autoMerge=false, requeueForAutoMerge not wired, or pause
active. Repeat-suppression on budget-exhausted emit via error marker
[transient-recovery-budget-exhausted] to prevent log spam.

Tests (6 new):
- target-not-queued recovery path
- spurious-concurrent-advance-same-sha recovery path (legacy)
- genuine concurrent-advance (different SHAs) NOT recovered
- non-transient failures NOT recovered (verification, conflicts)
- budget exhaustion emits marker once, no further requeue
- autoMerge=false no-op

Engine suite: 6157 tests pass (6 new).

In-flight: FN-5628 and FN-5632 were manually reset via SQL so the
already-running engine (which has the FN-5627 merger fix) can re-attempt
their merges before this self-healing path lands and reloads. Future
occurrences self-recover.

Fusion-Task-Id: FN-5627
This commit is contained in:
gsxdsm
2026-05-28 13:53:15 -07:00
parent 5b5da2c24f
commit 5768d5ec45
5 changed files with 461 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
.changeset/FN-5627-self-heal-transient-merge-failures.md Normal file
View File

@@ -0,0 +1,31 @@
---
"@runfusion/fusion": patch
---
feat(FN-5627): self-heal transient merge failures stuck at mergeRetries=3
After the FN-5627 merger fix landed, two in-review tasks (FN-5628, FN-5632) remained stuck at `mergeRetries=3` with `status='failed'` due to transient merge errors that the merger correctly identified but had no auto-recovery for:
- `lease-handoff-failed: target-not-queued` — FN-5353 class race where the merge queue lease acquisition saw the task drop out of the queue between enqueue and handoff (typically due to a self-healing sweep cleaning stale `mergeQueue` rows mid-flight).
- Legacy same-SHA spurious concurrent-advance errors persisted before FN-5627's `merger-ref-update-advance.ts` classifier fix landed.
These tasks had no path forward except manual intervention. The `AUTO_MERGE_COOLDOWN_MS` cooldown reset takes hours and gives up too easily.
This change adds `SelfHealingManager.recoverTransientMergeFailures()`, wired into both startup recovery and the periodic Batch 2 maintenance loop. For each in-review task with `mergeRetries >= MAX_AUTO_MERGE_RETRIES`, `status='failed'`, and an `error` matching `classifyTransientMergeError()`:
1. Reset `mergeRetries=0`, clear `status`/`error`.
2. Increment `mergeDetails.transientRecoveryCount` (new field on `MergeDetails`).
3. Re-enqueue via `requeueForAutoMerge`.
4. Emit `merger:transient-failure-auto-recovered` run-audit event.
Bounded by `MAX_TRANSIENT_MERGE_RECOVERIES = 2` to avoid infinite loops on genuinely stuck tasks. Once exhausted, the task stays parked as failed and emits `merger:transient-failure-budget-exhausted` once with a `[transient-recovery-budget-exhausted]` marker on `error` for repeat-suppression.
Non-transient failure classes (verification, build, real conflicts, etc.) are not eligible — only the pattern-matched transient classes auto-recover. No-op when `autoMerge=false`, no `requeueForAutoMerge` callback wired, or pause is active.
Tests:
- Lease-handoff transient recovery path
- Same-SHA spurious-advance recovery (legacy pre-FN-5627)
- Genuine concurrent-advance (different SHAs) NOT recovered
- Non-transient failures (verification errors) NOT recovered
- Budget exhaustion behavior
- autoMerge=false no-op