FN-8953: defer terminal wedge alerts during recovery
Hold terminal wedge alerts until their recovery window has elapsed. - Persist and settle pending wedge notifications across restarts. - Clear pending alerts on task progress and reconcile expired holds during self-healing. - Expose the settle window in notification settings with coverage for store and notification flows. Files changed: .changeset/fn-8953-wedge-settle-window.md | 7 + AGENTS.md | 1 + docs/architecture.md | 3 +- docs/settings-reference.md | 1 + .../core/src/__tests__/store-wedge-pending.test.ts | 56 +++++ packages/core/src/config/settings-schema.ts | 1 + packages/core/src/store.ts | 46 ++++ packages/core/src/types/settings/settings-scope.ts | 2 + packages/core/src/types/task/task-core.ts | 15 ++ .../app/components/settings/save-split.ts | 1 + .../sections/NotificationsSection.search.ts | 9 + .../settings/sections/NotificationsSection.tsx | 15 ++ .../settings-default-descriptions.test.tsx | 1 + ...self-healing-pending-wedge-notification.test.ts | 148 ++++++++++++ .../__tests__/notification-service.test.ts | 7 +- .../__tests__/task-wedge-notification.test.ts | 258 ++++++++++++++++++++- .../src/notification/notification-service.ts | 260 +++++++++++++++++++++ packages/engine/src/self-healing.ts | 38 +++ packages/i18n/locales/en/app.json | 2 + 19 files changed, 850 insertions(+), 21 deletions(-) Fusion-Task-Id: FN-8953 Fusion-Task-Lineage: fd5b5827-c69d-409f-86d5-01ff23405ee3 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
@@ -16,7 +16,7 @@ Actionable terminal task updates are classified into bounded reasons such as a n
|
||||
|
||||
A failed snapshot is not actionable while persisted automatic-recovery ownership remains: a scheduled recovery has both its retry counter and deadline, while transient merge recovery has an in-budget persisted retry counter. Pause-derived wedge reasons additionally require real pause state (`paused: true` or `status: "paused"`); an actively progressing task is never wedged. `NotificationService` re-reads and reclassifies the live task immediately before a wedge claim and again when a generic failure grace timer fires, so recovery that begins after a failed event, including a resume that deliberately leaves a stale pause reason behind, cannot create a mailbox row or `task-wedged` provider event. Explicit operator-action parks and cleared/exhausted recovery markers remain terminal and claim exactly one episode.
|
||||
|
||||
Each task also stores `lastNotifiedAtByReason`, an independent timestamp map keyed by bounded reason. `WEDGE_RENOTIFY_COOLDOWN_MS` defaults to six hours: resolving an episode does not clear its reason's live stamp, so a scheduler/self-healing resolve→re-wedge flap sends neither a provider push nor a mailbox message until the window expires. A different reason notifies immediately, including X→Y→X while X remains within its own cooldown; expired or invalid entries are pruned during the atomic claim, and legacy rows without the map notify normally before initializing it. The no-durable-store fallback applies the same per-reason window in memory. Provider and mailbox delivery are independently best-effort after sharing this single claim decision, while run-audit metadata remains ids/counts/outcomes-only.
|
||||
Each task also stores `lastNotifiedAtByReason`, an independent timestamp map keyed by bounded reason. Deferred wedge delivery uses a durable pending marker and a bounded self-healing backstop: it revalidates the live task before dispatch, clears recovery evidence silently, and re-stamps holds older than its maintenance-derived horizon rather than alerting on stale process-era evidence. `WEDGE_RENOTIFY_COOLDOWN_MS` defaults to six hours: resolving an episode does not clear its reason's live stamp, so a scheduler/self-healing resolve→re-wedge flap sends neither a provider push nor a mailbox message until the window expires. A different reason notifies immediately, including X→Y→X while X remains within its own cooldown; expired or invalid entries are pruned during the atomic claim, and legacy rows without the map notify normally before initializing it. The no-durable-store fallback applies the same per-reason window in memory. Provider and mailbox delivery are independently best-effort after sharing this single claim decision, while run-audit metadata remains ids/counts/outcomes-only.
|
||||
|
||||
Generic `terminal-failed` parks are engine-owned before they become operator work. A durable `wedgeNotification.autoRecovery` budget supplies bounded attempts and backoff; its claim state and rotating apply fence ensure one observer is authorized to clear and requeue a park. The grace window detects an abandoned apply only—it is not a lock. An exhausted budget receives one reason-scoped escalation, confirmed durably at the shared dispatch seam; a write-once exhaustion marker prevents an earlier drain alert or cooldown suppression from satisfying that escalation. The budget resets on terminal success, archive, explicit operator Retry, soft-delete, or a sufficiently old foreign write. Disabling auto-recovery drains an owed alert without consuming the remaining retries; re-enabling resumes them.
|
||||
|
||||
@@ -1155,6 +1155,7 @@ The run-audit system records every mutation performed by the engine across four
|
||||
- **Database / `task:reattach-orphaned-execution`** — emitted by `reattachOrphanedAssignedExecutions` (FN-6336) when self-healing re-dispatches an idle assigned `in-progress` task forward via `executor.resumeTaskForAgent(agentId)` after proving the assigned agent has no active heartbeat run or active execution.
|
||||
- **Database / `task:reconcile-stale-agent-assignment`** — emitted when self-healing or heartbeat reconciliation clears stale durable `Agent.taskId`/`state` for a task parked in `todo`/`triage` without live execution proof. Metadata includes `{ agentId, taskId, taskColumn, agentState, status, blockedBy, overlapBlockedBy, hadFreshRun, hadActiveExecution, reason }`; task queue/lease fields are preserved.
|
||||
- **Database / `task:reconcile-stale-duplicate-decision`** — emitted when self-healing clears a triage-marker duplicate-decision pause whose canonical is missing, deleted, done, or archived. Metadata is ids/outcomes-only: `{ taskId, canonicalId, canonicalColumn, canonicalDeleted, priorPausedReason }`; active canonicals and user pauses are excluded.
|
||||
- **Database / `task:reconcile-pending-wedge-notification`** — a bounded startup and maintenance sweep completes a restart-durable pending wedge hold through `NotificationService`, which alone revalidates the live subject and owns delivery. Holds older than the derived stale horizon are re-stamped rather than delivered on old evidence; metadata is ids/counts/outcomes-only: `{ taskId, reasonKey, pendingAgeMs, outcome }` where outcome is `delivered`, `suppressed`, `cleared`, `rearmed`, `held`, `absent`, `unreadable`, `deferred`, or `failed`.
|
||||
- **Database / `task:soft-delete-column-reconciled`** — emitted by `reconcileSoftDeletedColumnDrift` (FN-5566, re-land FN-5446) when a soft-deleted row (`deletedAt IS NOT NULL`) is found with legacy `column != 'archived'`; rewrites only `column` (no resurrection), with metadata `{ previousColumn }`.
|
||||
- **Database / `session:runtime-resolved`** — emitted once per `createResolvedAgentSession` call with metadata `{ sessionPurpose, runtimeId, wasConfigured, provider, modelId, mockProviderActive, testModeActive, runtimeHint?, credentialInstanceId?, credentialInstanceMissing?, requestedCredentialInstanceId?, resolvedCredentialInstanceId? }` for per-lane runtime/provider attribution. Credential fields are ids/outcomes-only; a missing selected instance is visibly recorded when the canonical provider default is used.
|
||||
- **Database / `task:reconcile-dependency-blocking-lease`** — emitted by `reconcileDependencyBlockingLeases()` (FN-6292) when self-healing rebounds an `in-progress` holder to `todo` because an unmet dependency is blocked by the holder's stale file-scope lease. Metadata includes the dependency ID, blocked-by marker, and unmet dependency list.
|
||||
|
||||
@@ -95,6 +95,7 @@ Fallback thinking-level values are applied at runtime when Fusion swaps from the
|
||||
| `agentClarificationEnabled` | `boolean` | `false` | Legacy default for programmatic Planning Mode session notification eligibility. Dashboard Planning Mode always starts its infinite, user-validated interview with follow-up questions enabled; this setting no longer suppresses questions or creates a final summary. |
|
||||
| `failureNotificationMode` | `"sticky-only" \| "terminal-only" \| "all"` | `"sticky-only"` | Failure notification behavior. `sticky-only` defers failed-task notifications by `failureNotificationDelayMs` and suppresses transient self-recoveries. `terminal-only` suppresses while auto-retry is still active and only dispatches when `paused === true` or `column === "in-review"` with `status === "failed"`. `all` restores legacy immediate failure notifications. |
|
||||
| `failureNotificationDelayMs` | `number` | `30000` | Delay window (ms) before evaluating/sending a `failed` notification in `sticky-only` and `terminal-only` modes. Set `0` for immediate dispatch in legacy `all` mode. |
|
||||
| `wedgeNotificationSettleMs` | `number` | `300000` | Time a terminal wedge must persist before its operator alert. `0` restores immediate delivery and clears outstanding holds. Production stores persist holds across restart; lightweight stores retain them only in memory. A stale hold is re-stamped using a horizon derived from the 15-minute maintenance interval, so an outage can delay a still-wedged alert rather than dispatching on stale evidence. |
|
||||
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
|
||||
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
|
||||
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |
|
||||
|
||||
Reference in New Issue
Block a user