FN-5901: reap stale mission validator runs
Add self-healing recovery for stale mission validator runs left behind after execution ownership disappears. - add mission-store support to find and reap stale running validator runs, preserving terminal error status and resetting eligible features to needs_fix - teach the mission execution loop and self-healing maintenance sweep to skip live validations, reap abandoned runs, record audit events, and avoid double-completing runs - extend regression coverage, mission docs, architecture notes, and add a published-package changeset for the new recovery behavior Files changed: .changeset/fn-5901-validator-run-reaper.md | 7 + AGENTS.md | 1 + docs/architecture.md | 2 + docs/missions.md | 26 ++- packages/core/src/__tests__/mission-store.test.ts | 99 +++++++++ packages/core/src/mission-store.ts | 91 ++++++++ packages/engine/src/__tests__/mission-execution-loop.test.ts | 232 +++++++++++++++++++++ packages/engine/src/__tests__/reliability-interactions/mission-validator-run-reaper.test.ts | 181 ++++++++++++++++ packages/engine/src/mission-execution-loop.ts | 102 +++++++-- packages/engine/src/runtimes/in-process-runtime.ts | 8 +- packages/engine/src/self-healing.ts | 22 ++ 11 files changed, 746 insertions(+), 25 deletions(-) Fusion-Task-Id: FN-5901 Fusion-Task-Lineage: 87eb2f3f-fc31-4e0a-b0fc-b771f6dc48a3
This commit is contained in:
gsxdsm
@@ -670,6 +670,7 @@ Runtime action-gate flow (v1):
|
||||
- `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
|
||||
- `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
|
||||
- `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`.
|
||||
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.
|
||||
|
||||
#### Stuck-loop exhaustion terminal contract
|
||||
When stuck-kill retries are exhausted, `checkStuckBudget()` marks the task `status: "failed"`, moves it to `in-review`, and writes an error that starts with `STUCK_LOOP_EXHAUSTED:`. The error and final task-log line both include the kill count/max and last stuck reason (`loop` or `inactivity`). `StuckTaskDetector` also untracks the task and refuses to re-track it while that failed terminal error remains, preventing further automatic kill/requeue churn. The final log line explicitly states that no further automatic retries will run and directs operators to manually retry, pause, or move the task back to triage to resume work.
|
||||
@@ -1777,6 +1778,7 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
|
||||
- FN-5788 backstop: `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` guards the promotion eligibility hook/audit seam so member landings emit `merge:branch-group-promotion-gated` with deterministic reason metadata (`eligible`, `group-automerge-disabled`, `settings-automerge-disabled`, `global-pause`, `engine-paused`) while group branches remain open and do not auto-promote to the default branch.
|
||||
- FN-5830 backstop: `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` guards branch-group completion-gate + promotion lifecycle so promotion happens exactly once after all members land, re-calls are idempotent, and gated paths emit `merge:branch-group-promotion-gated` without default-branch promotion.
|
||||
- FN-5819/FN-5846 backstop: `packages/engine/src/__tests__/reliability-interactions/shared-group-member-integration.test.ts` and `shared-branch-group-lifecycle.test.ts` guard the scoped autoMerge-off exception and deterministic finalize path so shared members integrate into the single group branch, produce `mergeTargetSource: "branch-group-integration"`/`mergeTargetBranch`, do not land on main, and are not moved backward by self-healing maintenance.
|
||||
- FN-5901 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validator-run-reaper.test.ts` guards stale mission-validator-run recovery across manual and automatic trigger types, verifies `mission:validator-run-reaped` audit metadata, ensures archived/complete parents keep their terminal feature state untouched, and proves reaped active features resume validation instead of staying wedged behind abandoned `running` rows.
|
||||
- FN-5738 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends coverage so zero-assertion auto-pass deterministically advances `loopState` to `passed`, sets `lastValidatorStatus="passed"`, emits `validation_auto_passed_no_assertions`, and does not re-fire on repeated recovery passes.
|
||||
- FN-5741 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-shadow-handoff.test.ts` guards Phase-1 merge-request contract shadow writes: flag OFF is a no-op, flag ON writes marker/record strictly after legacy handoff, and `autoMerge:false` remains `manual-required` without shadow running transitions.
|
||||
- FN-5742 backstop: `packages/engine/src/__tests__/reliability-interactions/dual-observe-merge-seam.test.ts` guards Phase-2 dual-observe invariants: legacy dependency satisfaction remains authoritative while parity diffs emit, and shadow dequeue selection never advances `manual-required` rows.
|
||||
|
||||
@@ -449,7 +449,7 @@ On task completion, the scheduler calls `MissionExecutionLoop.processTaskOutcome
|
||||
2. If assertions are linked, keep feature completion gated until validation passes
|
||||
3. Transition feature to `validating` state
|
||||
4. Fire AI validator agent against contract assertions
|
||||
5. Record `MissionValidatorRun` with per-assertion results
|
||||
5. Record `MissionValidatorRun` metadata for the validation attempt (per-assertion failures are stored separately in `MissionAssertionFailureRecord` rows)
|
||||
|
||||
Mission validation resolves its model from the validator lane before session creation: assigned agent runtime model (when the linked task has an assigned durable agent) → per-task `validatorModelProvider`/`validatorModelId` → project `validatorProvider`/`validatorModelId` → global `validatorGlobalProvider`/`validatorGlobalModelId` → project `defaultProviderOverride`/`defaultModelIdOverride` → global `defaultProvider`/`defaultModelId`. In `testMode`, validation is forced to `mock/scripted` instead of falling through to provider auto-detection.
|
||||
|
||||
@@ -459,22 +459,26 @@ Validation runs are internal mission-loop operations: Fusion does **not** create
|
||||
interface MissionValidatorRun {
|
||||
id: string;
|
||||
featureId: string;
|
||||
missionId: string;
|
||||
taskId: string;
|
||||
triggerType: "manual" | "automatic";
|
||||
milestoneId: string;
|
||||
sliceId: string;
|
||||
status: "running" | "passed" | "failed" | "blocked" | "error";
|
||||
triggerType?: string;
|
||||
implementationAttempt: number;
|
||||
validatorAttempt: number;
|
||||
status: "started" | "passed" | "failed" | "blocked" | "error";
|
||||
summary: string;
|
||||
results: AssertionResult[];
|
||||
taskId?: string;
|
||||
summary?: string;
|
||||
blockedReason?: string;
|
||||
startedAt: string;
|
||||
completedAt?: string;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
```
|
||||
|
||||
**Validation timeout:** 10 minutes (`VALIDATION_TIMEOUT_MS = 10 * 60 * 1000`). If session creation, auth/credit checks, prompting, or timeout fails, the run is marked `error` and emits a surfaced `validation_error` mission event instead of silently spawning a fix feature.
|
||||
|
||||
**Stale validator-run reaper:** startup recovery and periodic self-healing also sweep `MissionValidatorRun` rows stuck in `status="running"` longer than `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). Ownerless stale runs are reaped to terminal `status="error"`, their reap reason is stored in `summary`, and active mission features are moved to `loopState="needs_fix"` with `lastValidatorStatus="error"` so the loop can re-trigger. Runs whose parent mission is already `complete`/`archived` are still terminated, but their feature state is left untouched. Each successful reap emits a run-audit event with `mutationType: "mission:validator-run-reaped"`.
|
||||
|
||||
### Phase 5: Fix-Feature Retries
|
||||
|
||||
When validation fails, `MissionStore.createGeneratedFixFeature()` creates a fix feature with lineage tracking:
|
||||
@@ -504,7 +508,7 @@ A feature transitions to `blocked` when:
|
||||
- `MilestoneValidationRollup.state` reflects `blocked` assertions
|
||||
- The feature remains in `blocked` state until operator intervention
|
||||
|
||||
On engine restart, `recoverActiveMissions()` re-enqueues features in `validating` or `needs_fix` states from the `activeValidations` set, ensuring no validation work is lost. It also re-triggers `implementing` features whose linked task is already `done`/`archived` and whose assertion validation has not passed yet. The same recovery path is replayed during periodic self-heal maintenance, so historically stranded `implementing` features can self-heal without requiring an engine restart.
|
||||
On engine restart, `recoverActiveMissions()` re-enqueues features in `validating` or `needs_fix` states, ensuring no validation work is lost. It also re-triggers `implementing` features whose linked task is already `done`/`archived` and whose assertion validation has not passed yet. When the stale-run reaper has already converted an abandoned validator run into `needs_fix`, `processTaskOutcome()` promotes the feature back through `implementing` and re-validates instead of skipping it. The same recovery path is replayed during periodic self-heal maintenance, so historically stranded `implementing` features can self-heal without requiring an engine restart.
|
||||
|
||||
For features with zero linked assertions, the completion path is explicit: the loop marks the feature `done`, advances `loopState` to `passed`, emits `validation:passed` with summary `"No assertions linked"`, and records mission event code `validation_auto_passed_no_assertions`. Contract details (including canonical no-assertions behavior and FN-5696 assertion-authoring separation) are defined in [Mission Completion Gate Contract](./missions-completion-contract.md).
|
||||
|
||||
@@ -542,10 +546,10 @@ These are independent tracking mechanisms — autopilot monitors mission progres
|
||||
- `fix_feature:created`, `feature:blocked`
|
||||
|
||||
**Validator run telemetry:**
|
||||
- `triggerType` — manual vs automatic
|
||||
- `triggerType` — free-form trigger source (`manual`, `task_completion`, `auto`, etc.)
|
||||
- `implementationAttempt` — which retry attempt this was
|
||||
- `validatorAttempt` — how many validator runs for this implementation
|
||||
- `status` — started | passed | failed | blocked | error
|
||||
- `status` — running | passed | failed | blocked | error
|
||||
- `summary` — natural language summary of results
|
||||
|
||||
**Assertion failure records:**
|
||||
@@ -565,7 +569,7 @@ interface MissionAssertionFailureRecord {
|
||||
|
||||
| Symptom | Diagnosis | Resolution |
|
||||
|---------|-----------|------------|
|
||||
| Feature stuck in "validating" | `activeValidations` set may be stale; engine restart needed | Check logs for validator errors; restart engine to trigger `recoverActiveMissions()` |
|
||||
| Feature stuck in "validating" | Validator owner may have died, leaving a stale `MissionValidatorRun` in `status="running"` | Check mission-loop/self-healing logs; the startup or maintenance reaper should terminate runs older than `VALIDATOR_RUN_STALE_MAX_AGE_MS` (6h) and emit `mission:validator-run-reaped` |
|
||||
| Fix feature not auto-planning | `planFeature()` may have errored; check logs | Manual planning via `fn mission plan-feature <id>`; investigate `planFeature()` errors |
|
||||
| Budget exhaustion loop | `implementationAttemptCount >= maxRetryBudget` (default: 3) | Increase `maxRetryBudget` in mission settings or fix root cause |
|
||||
| Blocked mission not advancing | `MilestoneValidationRollup.state` shows `blocked` | Identify blocked assertions; operator must resolve root cause |
|
||||
|
||||
Reference in New Issue
Block a user