root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(FN-5627): suppress ntfy notifications for transient merge failures the engine auto-recovers

Browse Source 
Even with FN-5627's merger TOCTOU fix + transient-failure self-healing
sweep + safety-fallback auto-prerebase landed, the merger can still hit
transient failure classes (lease handoff races, brief same-SHA non-FF
advances) for tasks whose branches are particularly out-of-sync. The
self-healing sweep auto-recovers them within bounded budget \u2014 but each
individual failure cycle was firing a ntfy alarm before the recovery
cleared the failed state, producing user-facing alarm spam for tasks
that were never actually stuck.

Two layers of fix:

1. NotificationService.handleTaskUpdated now classifies task.error via
   the new shared classifyTransientMergeError helper before scheduling
   the deferred failure notification. Transient classes
   (lease-handoff-target-not-queued, spurious-concurrent-advance-same-sha)
   get logged as suppressed and never schedule a ntfy timer.

2. Defense-in-depth: fireDeferredFailureNotification re-classifies the
   error at dispatch time, so a failure scheduled before the suppression
   landed on a newer cycle still suppresses if the error matches a
   transient class.

The classifier itself moved from self-healing.ts to a new logger-free
transient-merge-error-classifier.ts module so consumers in
NotificationService don't pull createLogger through the import chain and
break test mocks of ../logger.js (per project-memory rule about new
modules using createLogger). self-healing.ts re-exports the symbol for
backward compatibility.

Log prefix for the recovery actions also changed from
'[FN-5627] Auto-recovering...' to 'Auto-recovered:' so that
NotificationService.maybeSuppressTransientFailedNotification's existing
/^Auto-recovered:/ log-prefix check cancels any already-scheduled failure
notification when the sweep runs mid-grace-window.

Tests (3 new):
- transient lease-handoff-target-not-queued failure NOT notified
- transient spurious-concurrent-advance-same-sha failure NOT notified
- genuine different-SHAs concurrent-advance still notifies (control)

Engine suite: 6166 tests pass.

Fusion-Task-Id: FN-5627
This commit is contained in:
gsxdsm
2026-05-28 15:39:23 -07:00
parent 694970b2f1
commit e75c4dae28
6 changed files with 164 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
.changeset/FN-5627-suppress-transient-ntfy.md Normal file
View File

@@ -0,0 +1,21 @@
---
"@runfusion/fusion": patch
---
fix(FN-5627): suppress ntfy notifications for transient merge failures the engine auto-recovers
Even with the FN-5627 merger TOCTOU fix + transient-failure self-healing sweep + safety-fallback auto-prerebase landed, the merger can still hit transient failure classes (lease handoff races, brief same-SHA non-FF advances) for tasks whose branches are particularly out-of-sync. The self-healing sweep auto-recovers them within bounded budget — but each individual failure cycle was firing a ntfy alarm before the recovery cleared the failed state, producing user-facing alarm spam for tasks that were never actually stuck.
Two layers of fix:
1. `NotificationService.handleTaskUpdated` now classifies `task.error` via the new shared `classifyTransientMergeError` helper before scheduling the deferred failure notification. Transient classes (`lease-handoff-target-not-queued`, `spurious-concurrent-advance-same-sha`) get logged as suppressed and never schedule a ntfy timer.
2. Defense-in-depth: `fireDeferredFailureNotification` re-classifies the error at dispatch time, so a failure scheduled before the suppression landed on a newer cycle still suppresses if the error matches a transient class.
The classifier itself moved from `self-healing.ts` to a new logger-free `transient-merge-error-classifier.ts` module so consumers in `NotificationService` don't pull `createLogger` through the import chain and break test mocks of `../logger.js`. `self-healing.ts` re-exports the symbol for backward compatibility.
Log prefix for the recovery actions also changed from `[FN-5627] Auto-recovering...` to `Auto-recovered:` so that `NotificationService.maybeSuppressTransientFailedNotification`'s existing `/^Auto-recovered:/` log-prefix check cancels any already-scheduled failure notification when the sweep runs mid-grace-window.
Tests:
- 3 new notification-service tests covering transient suppression for both error classes plus a control case ensuring genuine non-transient failures still notify.
- Existing transient-recovery tests in self-healing.test.ts continue to pass against the relocated classifier.