FN-5755: harden mission validation recovery flow
Ensure mission feature assertions recover and validate consistently across startup and periodic maintenance. - add FN-5755 changeset and AGENTS reliability backstop note - document canonical zero-assertion auto-pass and assertion validation lifecycle updates in missions docs - expand mission execution loop and reliability interaction tests for startup recovery, periodic replay, and idempotency - wire self-healing maintenance to replay active mission validation recovery via runtime callback Files changed: .../fn-5755-mission-validation-end-to-end.md | 5 ++ AGENTS.md | 1 + docs/missions-completion-contract.md | 15 ++-- docs/missions.md | 4 +- .../src/__tests__/mission-execution-loop.test.ts | 88 +++++++++++++++++++++- .../mission-validation-trigger-gap.test.ts | 42 ++++++++++- packages/engine/src/runtimes/in-process-runtime.ts | 6 ++ packages/engine/src/self-healing.ts | 11 +++ 8 files changed, 159 insertions(+), 13 deletions(-) Fusion-Task-Id: FN-5755 Fusion-Task-Lineage: e54ac4dc-7b8c-4fc9-8cd6-18702708546f
This commit is contained in:
gsxdsm
5
.changeset/fn-5755-mission-validation-end-to-end.md
Normal file
5
.changeset/fn-5755-mission-validation-end-to-end.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
Harden mission validation end-to-end by locking the canonical zero-assertion auto-pass path, strengthening assertion pass/fail regression coverage, and wiring bounded periodic mission recovery into existing self-healing maintenance so stranded implementing features recover without engine restart.
|
||||
@@ -163,6 +163,7 @@ When `settings.autoMerge: false`, `in-review` is terminal-until-merged by a huma
|
||||
- FN-5738 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends mission-loop coverage so zero-assertion auto-pass deterministically advances to `loopState="passed"` and emits `validation_auto_passed_no_assertions` without duplicate recovery re-fire.
|
||||
- FN-5741 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-shadow-handoff.test.ts` guards Phase-1 write-only-shadow merge-request record + handoff-accepted marker seam (flag OFF = no-op, ON = shadow-only non-authoritative).
|
||||
- FN-5754 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-stranded-feature-retriage.test.ts` guards startup/maintenance stranded-feature re-triage for active autopilot slices, including link-first dedupe, non-defined skip safety, non-autopilot no-op, idempotency, and `mission:stranded-feature-triaged` audit shape.
|
||||
- FN-5755 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends mission validation coverage so bounded periodic maintenance replays `recoverActiveMissions` for stranded `implementing` features and remains idempotent on repeated passes.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -57,17 +57,17 @@ Canonical authored source and enforcement path:
|
||||
|
||||
## Zero-Assertion Behavior and FN-5696 Failure Shape
|
||||
|
||||
### Zero-assertions runtime behavior (must align with FN-5715)
|
||||
### Zero-assertions runtime behavior (canonical FN-5738 path)
|
||||
|
||||
When a feature reaches completion trigger points and has **zero linked assertions**, mission execution must take an explicit auto-pass path (not a silent stall):
|
||||
When a feature reaches completion trigger points and has **zero linked assertions**, mission execution must take exactly one canonical auto-pass path (not a silent stall and not a competing behavior):
|
||||
|
||||
- mark feature as passed/done through the no-assertions path,
|
||||
- emit explicit observability/audit evidence for the auto-pass reason,
|
||||
- continue normal slice/mission advancement checks.
|
||||
- mark feature terminal as `status="done"`, `loopState="passed"`, `lastValidatorStatus="passed"`,
|
||||
- emit explicit observability/audit evidence with mission event code `validation_auto_passed_no_assertions`,
|
||||
- continue normal slice/mission advancement checks idempotently (no duplicate re-fire on repeated recovery).
|
||||
|
||||
### FN-5696 legacy shape clarification
|
||||
|
||||
A feature can show acceptance text while links are missing (legacy pre-repair data). This must be treated as a **linkage/data integrity problem**, not as proof that milestone text alone is enforced. The contract prevents ambiguity by separating:
|
||||
A feature can show acceptance text while links are missing (legacy pre-repair data). This must be treated as a **linkage/data integrity problem**, not as proof that milestone text alone is enforced. Assertion authoring/backfill (FN-5696) is outside the execution loop; the loop must not synthesize `mission_feature_assertions` rows. The contract prevents ambiguity by separating:
|
||||
|
||||
- authored/informational text surfaces, from
|
||||
- linked assertion enforcement surfaces.
|
||||
@@ -141,7 +141,8 @@ Target surface: `packages/dashboard/app/components/MissionManager.tsx`
|
||||
- done mission-linked tasks with linked assertions trigger validation,
|
||||
- completion-trigger starts loop if needed,
|
||||
- startup recovery replays done-implementing features with unpassed assertions,
|
||||
- zero-linked-assertions path remains explicit auto-pass.
|
||||
- periodic self-heal maintenance replays the same `recoverActiveMissions` path so historically stranded `implementing` features recover without restart,
|
||||
- zero-linked-assertions path remains explicit canonical auto-pass.
|
||||
|
||||
3. **UI behavior**
|
||||
- Implement the Step-2 label reconciliation and per-row indicator requirements.
|
||||
|
||||
@@ -395,9 +395,9 @@ A feature transitions to `blocked` when:
|
||||
- `MilestoneValidationRollup.state` reflects `blocked` assertions
|
||||
- The feature remains in `blocked` state until operator intervention
|
||||
|
||||
On engine restart, `recoverActiveMissions()` re-enqueues features in `validating` or `needs_fix` states from the `activeValidations` set, ensuring no validation work is lost. It also re-triggers `implementing` features whose linked task is already `done`/`archived` and whose assertion validation has not passed yet.
|
||||
On engine restart, `recoverActiveMissions()` re-enqueues features in `validating` or `needs_fix` states from the `activeValidations` set, ensuring no validation work is lost. It also re-triggers `implementing` features whose linked task is already `done`/`archived` and whose assertion validation has not passed yet. The same recovery path is replayed during periodic self-heal maintenance, so historically stranded `implementing` features can self-heal without requiring an engine restart.
|
||||
|
||||
For features with zero linked assertions, the completion path is explicit: the loop marks the feature `done`, advances `loopState` to `passed`, emits `validation:passed` with summary `"No assertions linked"`, and records mission event code `validation_auto_passed_no_assertions`.
|
||||
For features with zero linked assertions, the completion path is explicit: the loop marks the feature `done`, advances `loopState` to `passed`, emits `validation:passed` with summary `"No assertions linked"`, and records mission event code `validation_auto_passed_no_assertions`. Contract details (including canonical no-assertions behavior and FN-5696 assertion-authoring separation) are defined in [Mission Completion Gate Contract](./missions-completion-contract.md).
|
||||
|
||||
### Autopilot / Scheduler Interplay
|
||||
|
||||
|
||||
@@ -216,6 +216,40 @@ function createMockMissionStore() {
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
validatorRuns.set(id, updated);
|
||||
|
||||
const feature = features.get(run.featureId);
|
||||
if (feature) {
|
||||
if (status === "passed") {
|
||||
features.set(run.featureId, {
|
||||
...feature,
|
||||
loopState: "passed",
|
||||
lastValidatorStatus: "passed",
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
} else if (status === "failed") {
|
||||
features.set(run.featureId, {
|
||||
...feature,
|
||||
loopState: "needs_fix",
|
||||
lastValidatorStatus: "failed",
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
} else if (status === "blocked") {
|
||||
features.set(run.featureId, {
|
||||
...feature,
|
||||
loopState: "blocked",
|
||||
lastValidatorStatus: "blocked",
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
} else if (status === "error") {
|
||||
features.set(run.featureId, {
|
||||
...feature,
|
||||
loopState: "validating",
|
||||
lastValidatorStatus: "error",
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return updated;
|
||||
}),
|
||||
recordValidatorFailures: vi.fn(() => []),
|
||||
@@ -238,7 +272,7 @@ function createMockMissionStore() {
|
||||
const updatedSource = {
|
||||
...sourceFeature,
|
||||
implementationAttemptCount: (sourceFeature.implementationAttemptCount ?? 0) + 1,
|
||||
loopState: "needs_fix" as const,
|
||||
loopState: "implementing" as const,
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
features.set(sourceFeatureId, updatedSource);
|
||||
@@ -680,6 +714,58 @@ describe("MissionExecutionLoop", () => {
|
||||
"task_completion",
|
||||
);
|
||||
});
|
||||
|
||||
it("runs linked assertions and marks completion only when validation passes", async () => {
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-ASSERT-PASS", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore.getFeatureByTaskId = vi.fn().mockReturnValue(feature);
|
||||
missionStore.listAssertionsForFeature = vi.fn().mockReturnValue(makeAssertions(2));
|
||||
taskStore._setTask({ id: "FN-ASSERT-PASS", title: "Assertion pass", description: "Implementation", log: [] });
|
||||
mockSessionHolder.session.state.messages = [
|
||||
{ role: "assistant", content: JSON.stringify({ status: "pass", assertions: [{ assertionId: "CA-1", passed: true }, { assertionId: "CA-2", passed: true }], summary: "all good" }) },
|
||||
];
|
||||
|
||||
loop = new MissionExecutionLoop({
|
||||
taskStore: taskStore as any,
|
||||
missionStore: missionStore as any,
|
||||
rootDir: "/tmp",
|
||||
});
|
||||
loop.start();
|
||||
|
||||
await loop.processTaskOutcome("FN-ASSERT-PASS");
|
||||
|
||||
expect(missionStore.startValidatorRun).toHaveBeenCalledWith("F-001", "task_completion");
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "passed", expect.any(String));
|
||||
expect(missionStore.getFeature("F-001")?.loopState).toBe("passed");
|
||||
expect(missionStore.getFeature("F-001")?.lastValidatorStatus).toBe("passed");
|
||||
expect(missionStore.updateFeatureStatus).toHaveBeenCalledWith("F-001", "done");
|
||||
});
|
||||
|
||||
it("routes failed assertion validation to fix flow and does not pass feature", async () => {
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-ASSERT-FAIL", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore.getFeatureByTaskId = vi.fn().mockReturnValue(feature);
|
||||
missionStore.listAssertionsForFeature = vi.fn().mockReturnValue(makeAssertions(2));
|
||||
taskStore._setTask({ id: "FN-ASSERT-FAIL", title: "Assertion fail", description: "Implementation", log: [] });
|
||||
mockSessionHolder.session.state.messages = [
|
||||
{ role: "assistant", content: JSON.stringify({ status: "fail", assertions: [{ assertionId: "CA-1", passed: false, message: "miss" }], summary: "failed" }) },
|
||||
];
|
||||
|
||||
loop = new MissionExecutionLoop({
|
||||
taskStore: taskStore as any,
|
||||
missionStore: missionStore as any,
|
||||
rootDir: "/tmp",
|
||||
});
|
||||
loop.start();
|
||||
|
||||
await loop.processTaskOutcome("FN-ASSERT-FAIL");
|
||||
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "failed", expect.any(String));
|
||||
expect(missionStore.createGeneratedFixFeature).toHaveBeenCalled();
|
||||
expect(missionStore.getFeature("F-001")?.lastValidatorStatus).toBe("failed");
|
||||
expect(missionStore.getFeature("F-001")?.loopState).toBe("implementing");
|
||||
expect(missionStore.getFeature("F-001")?.status).not.toBe("done");
|
||||
});
|
||||
});
|
||||
|
||||
// ── recoverActiveMissions ────────────────────────────────────────────────
|
||||
|
||||
@@ -120,6 +120,41 @@ describe("FN-5715 reliability: mission validation trigger gap", () => {
|
||||
loop.stop();
|
||||
});
|
||||
|
||||
it("recovery trigger for done implementing feature is idempotent across subsequent passes", async () => {
|
||||
const feature = makeFeature({ status: "done", lastValidatorStatus: undefined, loopState: "implementing" });
|
||||
const missionStore = {
|
||||
listMissions: vi.fn(() => [{ id: "M-001", status: "active" }]),
|
||||
getMissionWithHierarchy: vi.fn(() => ({
|
||||
id: "M-001",
|
||||
status: "active",
|
||||
milestones: [{ status: "active", slices: [{ status: "active", features: [feature] }] }],
|
||||
})),
|
||||
listAssertionsForFeature: vi.fn(() => [{ id: "CA-1" }]),
|
||||
getFeature: vi.fn(() => feature),
|
||||
transitionLoopState: vi.fn(),
|
||||
};
|
||||
const taskStore = {
|
||||
getTask: vi.fn(async () => ({ id: "FN-001", column: "done" })),
|
||||
};
|
||||
|
||||
const loop = new MissionExecutionLoop({
|
||||
missionStore: missionStore as any,
|
||||
taskStore: taskStore as any,
|
||||
rootDir: process.cwd(),
|
||||
});
|
||||
const processSpy = vi.spyOn(loop, "processTaskOutcome").mockImplementation(async () => {
|
||||
feature.lastValidatorStatus = "passed";
|
||||
feature.loopState = "passed";
|
||||
});
|
||||
loop.start();
|
||||
|
||||
await loop.recoverActiveMissions();
|
||||
await loop.recoverActiveMissions();
|
||||
|
||||
expect(processSpy).toHaveBeenCalledTimes(1);
|
||||
loop.stop();
|
||||
});
|
||||
|
||||
it("is idempotent for already-passed implementing features", async () => {
|
||||
const feature = makeFeature({ status: "done", lastValidatorStatus: "passed" });
|
||||
const missionStore = {
|
||||
@@ -151,7 +186,7 @@ describe("FN-5715 reliability: mission validation trigger gap", () => {
|
||||
loop.stop();
|
||||
});
|
||||
|
||||
it("recovery replays implementing done tasks with zero assertions and advances loop state", async () => {
|
||||
it("periodic recovery pass replays implementing done tasks with zero assertions and advances loop state", async () => {
|
||||
const feature = makeFeature({ status: "done", lastValidatorStatus: undefined, loopState: "implementing" });
|
||||
const currentFeature = { ...feature };
|
||||
const missionStore = {
|
||||
@@ -187,8 +222,9 @@ describe("FN-5715 reliability: mission validation trigger gap", () => {
|
||||
});
|
||||
loop.start();
|
||||
|
||||
await loop.recoverActiveMissions();
|
||||
await loop.recoverActiveMissions();
|
||||
const periodicMaintenancePass = async () => loop.recoverActiveMissions();
|
||||
await periodicMaintenancePass();
|
||||
await periodicMaintenancePass();
|
||||
|
||||
expect(missionStore.updateFeature).toHaveBeenCalledTimes(1);
|
||||
expect(missionStore.updateFeature).toHaveBeenCalledWith(
|
||||
|
||||
@@ -732,6 +732,12 @@ export class InProcessRuntime
|
||||
getActiveMergeTaskId: () => this.activeMergeTaskIdProvider?.() ?? null,
|
||||
leaseManager: this.leaseManager,
|
||||
hasActiveAgentExecution: (agentId: string) => this.heartbeatMonitor?.getTrackedAgents().includes(agentId) ?? false,
|
||||
recoverActiveMissionValidations: async () => {
|
||||
if (!this.missionExecutionLoop) {
|
||||
return { recoveredCount: 0 };
|
||||
}
|
||||
return this.missionExecutionLoop.recoverActiveMissions();
|
||||
},
|
||||
reconcileAllMissionFeatures: async () => this.scheduler.reconcileAllMissionFeatures(),
|
||||
chatStore: this.chatStore,
|
||||
messageStore: this.messageStore,
|
||||
|
||||
@@ -276,6 +276,8 @@ export interface SelfHealingOptions {
|
||||
getProjectId?: () => string;
|
||||
/** Optional callback to reconcile active mission features during maintenance. */
|
||||
reconcileAllMissionFeatures?: () => Promise<number>;
|
||||
/** Optional callback to re-run mission validation recovery during maintenance. */
|
||||
recoverActiveMissionValidations?: () => Promise<{ recoveredCount: number }>;
|
||||
}
|
||||
|
||||
const APPROVED_TRIAGE_RECOVERY_GRACE_MS = 60_000;
|
||||
@@ -1470,6 +1472,15 @@ export class SelfHealingManager {
|
||||
} else {
|
||||
// Batch 2 — Task recovery (operations are independent of each other)
|
||||
const batch2Fns: Array<{ name: string; fn: () => Promise<unknown> }> = [
|
||||
{
|
||||
name: "recover-active-mission-validations",
|
||||
fn: async () => {
|
||||
if (!this.options.recoverActiveMissionValidations) {
|
||||
return;
|
||||
}
|
||||
await this.options.recoverActiveMissionValidations();
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "reconcile-mission-features",
|
||||
fn: async () => {
|
||||
|
||||
Reference in New Issue
Block a user