FN-8953: defer terminal wedge alerts during recovery

Hold terminal wedge alerts until their recovery window has elapsed.

- Persist and settle pending wedge notifications across restarts.
- Clear pending alerts on task progress and reconcile expired holds during self-healing.
- Expose the settle window in notification settings with coverage for store and notification flows.

Files changed:
 .changeset/fn-8953-wedge-settle-window.md          |   7 +
 AGENTS.md                                          |   1 +
 docs/architecture.md                               |   3 +-
 docs/settings-reference.md                         |   1 +
 .../core/src/__tests__/store-wedge-pending.test.ts |  56 +++++
 packages/core/src/config/settings-schema.ts        |   1 +
 packages/core/src/store.ts                         |  46 ++++
 packages/core/src/types/settings/settings-scope.ts |   2 +
 packages/core/src/types/task/task-core.ts          |  15 ++
 .../app/components/settings/save-split.ts          |   1 +
 .../sections/NotificationsSection.search.ts        |   9 +
 .../settings/sections/NotificationsSection.tsx     |  15 ++
 .../settings-default-descriptions.test.tsx         |   1 +
 ...self-healing-pending-wedge-notification.test.ts | 148 ++++++++++++
 .../__tests__/notification-service.test.ts         |   7 +-
 .../__tests__/task-wedge-notification.test.ts      | 258 ++++++++++++++++++++-
 .../src/notification/notification-service.ts       | 260 +++++++++++++++++++++
 packages/engine/src/self-healing.ts                |  38 +++
 packages/i18n/locales/en/app.json                  |   2 +
 19 files changed, 850 insertions(+), 21 deletions(-)

Fusion-Task-Id: FN-8953

Fusion-Task-Lineage: fd5b5827-c69d-409f-86d5-01ff23405ee3

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-08-11 12:34:47 -07:00
parent dfd88e7540
commit 6ae9299576
19 changed files with 849 additions and 20 deletions

View File

@@ -16,7 +16,7 @@ Actionable terminal task updates are classified into bounded reasons such as a n
A failed snapshot is not actionable while persisted automatic-recovery ownership remains: a scheduled recovery has both its retry counter and deadline, while transient merge recovery has an in-budget persisted retry counter. Pause-derived wedge reasons additionally require real pause state (`paused: true` or `status: "paused"`); an actively progressing task is never wedged. `NotificationService` re-reads and reclassifies the live task immediately before a wedge claim and again when a generic failure grace timer fires, so recovery that begins after a failed event, including a resume that deliberately leaves a stale pause reason behind, cannot create a mailbox row or `task-wedged` provider event. Explicit operator-action parks and cleared/exhausted recovery markers remain terminal and claim exactly one episode.
Each task also stores `lastNotifiedAtByReason`, an independent timestamp map keyed by bounded reason. `WEDGE_RENOTIFY_COOLDOWN_MS` defaults to six hours: resolving an episode does not clear its reason's live stamp, so a scheduler/self-healing resolve→re-wedge flap sends neither a provider push nor a mailbox message until the window expires. A different reason notifies immediately, including X→Y→X while X remains within its own cooldown; expired or invalid entries are pruned during the atomic claim, and legacy rows without the map notify normally before initializing it. The no-durable-store fallback applies the same per-reason window in memory. Provider and mailbox delivery are independently best-effort after sharing this single claim decision, while run-audit metadata remains ids/counts/outcomes-only.
Each task also stores `lastNotifiedAtByReason`, an independent timestamp map keyed by bounded reason. Deferred wedge delivery uses a durable pending marker and a bounded self-healing backstop: it revalidates the live task before dispatch, clears recovery evidence silently, and re-stamps holds older than its maintenance-derived horizon rather than alerting on stale process-era evidence. `WEDGE_RENOTIFY_COOLDOWN_MS` defaults to six hours: resolving an episode does not clear its reason's live stamp, so a scheduler/self-healing resolve→re-wedge flap sends neither a provider push nor a mailbox message until the window expires. A different reason notifies immediately, including X→Y→X while X remains within its own cooldown; expired or invalid entries are pruned during the atomic claim, and legacy rows without the map notify normally before initializing it. The no-durable-store fallback applies the same per-reason window in memory. Provider and mailbox delivery are independently best-effort after sharing this single claim decision, while run-audit metadata remains ids/counts/outcomes-only.
Generic `terminal-failed` parks are engine-owned before they become operator work. A durable `wedgeNotification.autoRecovery` budget supplies bounded attempts and backoff; its claim state and rotating apply fence ensure one observer is authorized to clear and requeue a park. The grace window detects an abandoned apply only—it is not a lock. An exhausted budget receives one reason-scoped escalation, confirmed durably at the shared dispatch seam; a write-once exhaustion marker prevents an earlier drain alert or cooldown suppression from satisfying that escalation. The budget resets on terminal success, archive, explicit operator Retry, soft-delete, or a sufficiently old foreign write. Disabling auto-recovery drains an owed alert without consuming the remaining retries; re-enabling resumes them.
@@ -1155,6 +1155,7 @@ The run-audit system records every mutation performed by the engine across four
- **Database / `task:reattach-orphaned-execution`** — emitted by `reattachOrphanedAssignedExecutions` (FN-6336) when self-healing re-dispatches an idle assigned `in-progress` task forward via `executor.resumeTaskForAgent(agentId)` after proving the assigned agent has no active heartbeat run or active execution.
- **Database / `task:reconcile-stale-agent-assignment`** — emitted when self-healing or heartbeat reconciliation clears stale durable `Agent.taskId`/`state` for a task parked in `todo`/`triage` without live execution proof. Metadata includes `{ agentId, taskId, taskColumn, agentState, status, blockedBy, overlapBlockedBy, hadFreshRun, hadActiveExecution, reason }`; task queue/lease fields are preserved.
- **Database / `task:reconcile-stale-duplicate-decision`** — emitted when self-healing clears a triage-marker duplicate-decision pause whose canonical is missing, deleted, done, or archived. Metadata is ids/outcomes-only: `{ taskId, canonicalId, canonicalColumn, canonicalDeleted, priorPausedReason }`; active canonicals and user pauses are excluded.
- **Database / `task:reconcile-pending-wedge-notification`** — a bounded startup and maintenance sweep completes a restart-durable pending wedge hold through `NotificationService`, which alone revalidates the live subject and owns delivery. Holds older than the derived stale horizon are re-stamped rather than delivered on old evidence; metadata is ids/counts/outcomes-only: `{ taskId, reasonKey, pendingAgeMs, outcome }` where outcome is `delivered`, `suppressed`, `cleared`, `rearmed`, `held`, `absent`, `unreadable`, `deferred`, or `failed`.
- **Database / `task:soft-delete-column-reconciled`** — emitted by `reconcileSoftDeletedColumnDrift` (FN-5566, re-land FN-5446) when a soft-deleted row (`deletedAt IS NOT NULL`) is found with legacy `column != 'archived'`; rewrites only `column` (no resurrection), with metadata `{ previousColumn }`.
- **Database / `session:runtime-resolved`** — emitted once per `createResolvedAgentSession` call with metadata `{ sessionPurpose, runtimeId, wasConfigured, provider, modelId, mockProviderActive, testModeActive, runtimeHint?, credentialInstanceId?, credentialInstanceMissing?, requestedCredentialInstanceId?, resolvedCredentialInstanceId? }` for per-lane runtime/provider attribution. Credential fields are ids/outcomes-only; a missing selected instance is visibly recorded when the canonical provider default is used.
- **Database / `task:reconcile-dependency-blocking-lease`** — emitted by `reconcileDependencyBlockingLeases()` (FN-6292) when self-healing rebounds an `in-progress` holder to `todo` because an unmet dependency is blocked by the holder's stale file-scope lease. Metadata includes the dependency ID, blocked-by marker, and unmet dependency list.

View File

@@ -95,6 +95,7 @@ Fallback thinking-level values are applied at runtime when Fusion swaps from the
| `agentClarificationEnabled` | `boolean` | `false` | Legacy default for programmatic Planning Mode session notification eligibility. Dashboard Planning Mode always starts its infinite, user-validated interview with follow-up questions enabled; this setting no longer suppresses questions or creates a final summary. |
| `failureNotificationMode` | `"sticky-only" \| "terminal-only" \| "all"` | `"sticky-only"` | Failure notification behavior. `sticky-only` defers failed-task notifications by `failureNotificationDelayMs` and suppresses transient self-recoveries. `terminal-only` suppresses while auto-retry is still active and only dispatches when `paused === true` or `column === "in-review"` with `status === "failed"`. `all` restores legacy immediate failure notifications. |
| `failureNotificationDelayMs` | `number` | `30000` | Delay window (ms) before evaluating/sending a `failed` notification in `sticky-only` and `terminal-only` modes. Set `0` for immediate dispatch in legacy `all` mode. |
| `wedgeNotificationSettleMs` | `number` | `300000` | Time a terminal wedge must persist before its operator alert. `0` restores immediate delivery and clears outstanding holds. Production stores persist holds across restart; lightweight stores retain them only in memory. A stale hold is re-stamped using a horizon derived from the 15-minute maintenance interval, so an outage can delay a still-wedged alert rather than dispatching on stale evidence. |
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |