Merge pull request #1666 from Runfusion/feat/validator-behavioral-verification
feat: behavioral verification in the Validator
This commit is contained in:
gsxdsm
5
.changeset/loud-nodes-sync.md
Normal file
5
.changeset/loud-nodes-sync.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
Sync workflow setting values across nodes in settings push, pull, receive, and status flows.
|
||||
5
.changeset/proud-validators-verify.md
Normal file
5
.changeset/proud-validators-verify.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
Close the validator reaper→slice deadlock and harden every validation re-drive site for the new behavioral-verification posture. A reaped, task-less "done" feature (left in `loopState="validating"`/`needs_fix`+`error`) is now re-driven by recovery to a terminal pass/fail/inconclusive verdict instead of livelocking the slice, milestone, and mission. Adds an adversarial reliability suite enumerating every re-drive entry point (normal `processTaskOutcome`, each `recoverActiveMissions` branch, and the stale-run reaper) and asserting source-tree git-cleanliness, zero duplicate Fix Features, a terminal verdict, and no `error`-state deadlock. Documents the non-mutating verification run, the first-class `inconclusive` verdict, and the adversarial default-to-fail posture across `docs/missions.md`, `docs/missions-completion-contract.md`, and `CONCEPTS.md`.
|
||||
@@ -66,10 +66,10 @@ The process that creates or reattaches a Project in Fusion's central registry fo
|
||||
The named process that watches an active Mission and advances it — activating the next pending Slice once the current Slice completes — while tracking its own watching/activating lifecycle and handling retries. When Autopilot is not watching a Mission, slice advancement falls back to a compatibility path.
|
||||
|
||||
### Contract Assertion
|
||||
A checkable acceptance criterion linked to a Feature that an AI validator judges to decide whether the Feature is genuinely done. Every Feature is validator-evaluated — a Feature missing an assertion has one lazily linked before validation — and counts toward Slice completion only after a passing Validator Run.
|
||||
A checkable acceptance criterion linked to a Feature that an AI validator judges to decide whether the Feature is genuinely done. Each assertion carries a `type` (`static` or `behavioral`). Static assertions are graded by read-only inspection. Behavioral/bug-fix assertions take an **adversarial default-to-fail** posture: the judge's pass is advisory, and the assertion is satisfied only when a behavioral verification run confirms the observable outcome by exercising the code. Every Feature is validator-evaluated — a Feature missing an assertion has one lazily linked before validation — and counts toward Slice completion only after a passing Validator Run.
|
||||
|
||||
### Validator Run
|
||||
A single execution of the AI judge that evaluates a Feature's Contract Assertions and yields a pass, fail, blocked, or error outcome. The validator is read-only — it inspects the implementation and records a verdict, creating no board task and editing no code. A run left running after its owner disappears is reaped to a terminal error state.
|
||||
A single execution that evaluates a Feature's Contract Assertions and yields a pass, fail, blocked, error, or **inconclusive** outcome. It has two parts. The **read-only AI judge** inspects the implementation and records an advisory verdict, creating no board task and editing no code. For behavioral/bug assertions a separate **verification run** then confirms (or refutes) the judge by executing the code — so a Validator Run is no longer purely read-only/static. The verification run is still **non-mutating to mission/board state**: it executes against an isolating sandbox (fail-closed when none is available) and a disposable checkout at a trusted revision, creates no board task, mutates no mission/board row, and leaves the source tree git-clean. An `inconclusive` verdict (verification could not run or conclude) is first-class and distinct from `fail`: it routes to needs-attention and spawns no Fix Feature. A run left running after its owner disappears is reaped to a terminal error state; a reaped task-less done Feature is re-driven by recovery to a terminal verdict rather than deadlocking the Slice.
|
||||
|
||||
### loop state
|
||||
A Feature's position in the execution loop (being implemented, awaiting or undergoing validation, awaiting a fix, passed, or blocked), distinct from its board status. Logic that gates on loop state must treat it as possibly stale and possibly contradictory with status — a Feature can be marked done while its loop state was never advanced past implementing.
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
date: 2026-06-11
|
||||
topic: validator-behavioral-verification
|
||||
---
|
||||
|
||||
# Trustworthy "Done": Behavioral Verification in the Validator
|
||||
|
||||
## Summary
|
||||
|
||||
Make the done-gate require evidence of real behavior instead of trusting a diff's apparent intent. The Validator Run defaults to *fail* unless a behavioral or bug-fix assertion carries executable proof, and the gate gains a bounded verification run that actually exercises the code — runs the relevant tests, drives the app — to confirm the assertion before a Feature can count as done.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
The Validator Run is the quality gate: an AI judge inspects a Feature's implementation and decides whether its Contract Assertions are met. By design it is read-only — it reads the diff and records a verdict, never executing anything.
|
||||
|
||||
That design is structurally weak for behavioral correctness. A bug-fix task was marked done after the judge accepted a diff that *looked* like a fix; the bug was still live in the running app. The judge graded the implementation's intent. Reality — the actual behavior — was never consulted. The behavioral truth existed (it was caught and captured by hand, manually testing the running app), but that truth never reached the verdict.
|
||||
|
||||
The cost shape: false-positive passes erode trust in "done" entirely. A gate that rubber-stamps can't be relied on, so every completed task inherits a manual re-check tax, which defeats the point of an automated gate.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Default to fail, not pass.** The judge's posture inverts: a behavioral/bug assertion is *not met* until there is crisp evidence it is. The judge's job becomes verifying that proof exists and is genuine, not reasoning about whether the code probably works.
|
||||
- **Policy needs teeth — adversarial posture and executable verification ship together.** Raising the evidence bar alone (A) is insufficient, because a leniency-prone agent can also produce weak or fake proof. The gate must independently *execute* to observe behavior (B), not merely inspect the agent's claims more skeptically.
|
||||
- **Verification is scoped to behavioral/bug assertions, not all assertions.** Static judging stays adequate for non-behavioral assertions; the bounded verification run is reserved for assertions whose truth is observable only by exercising the code, so execution cost is paid only where it buys correctness.
|
||||
- **The verification run extends the validator's read-only invariant.** Confirming behavior requires running code, which the validator was explicitly forbidden from doing. This is an accepted, deliberate change to a documented design principle, bounded to a verification capability — the judge still creates no board task and edits no code.
|
||||
|
||||
## Actors
|
||||
|
||||
- A1. Coding agent — produces the implementation and, under the new posture, the executable proof (e.g., a regression test) that a behavioral/bug assertion is satisfied.
|
||||
- A2. Validator Run (AI judge) — evaluates Contract Assertions; now defaults to fail on behavioral/bug assertions absent verified evidence.
|
||||
- A3. Verification run — the bounded execution capability the gate invokes to exercise the code (run tests, drive the app) and observe real behavior.
|
||||
- A4. Human / orchestrator — relies on a trustworthy "done"; previously the implicit fallback that caught escapes by manual testing.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Judging posture
|
||||
|
||||
- R1. For behavioral and bug-fix assertions, the validator defaults to a fail verdict unless the assertion is backed by verified behavioral evidence.
|
||||
- R2. The validator classifies each Contract Assertion as behavioral/bug (truth observable only by exercising the code) or non-behavioral (judgeable by inspection), and applies the stricter posture only to the former.
|
||||
- R3. For non-behavioral assertions, existing static judging is preserved — the change does not raise cost or strictness where inspection already suffices.
|
||||
|
||||
### Behavioral verification
|
||||
|
||||
- R4. The gate can invoke a bounded verification run that exercises the code to confirm a behavioral/bug assertion's observable outcome before passing it.
|
||||
- R5. For bug-fix assertions specifically, verification confirms the reported defect is no longer reproducible — not merely that a plausible change was made.
|
||||
- R6. The verification run is bounded in time and cost, and a run that cannot complete or conclude resolves to a non-passing verdict (fail/blocked/error), never a default pass.
|
||||
- R7. The verification run honors the validator's non-mutating boundary: it observes behavior but creates no board task and edits no code.
|
||||
|
||||
### Evidence and proof
|
||||
|
||||
- R8. A behavioral/bug assertion passes only when verification evidence corroborates it; an agent's narrative claim that the assertion is met is not, on its own, sufficient evidence.
|
||||
- R9. When the agent supplies executable proof (e.g., a regression test that fails on the pre-fix state and passes on the implementation), the validator verifies the proof is genuine rather than accepting its presence at face value.
|
||||
- R10. A non-passing verdict records why it failed in terms the downstream Fix Feature can act on (which assertion, what behavior was observed vs. expected).
|
||||
|
||||
## Key Flows
|
||||
|
||||
- F1. Bug-fix verification
|
||||
- **Trigger:** A bug-fix Feature reaches validation with a Contract Assertion of the form "defect X no longer occurs."
|
||||
- **Actors:** A2, A3
|
||||
- **Steps:** The judge classifies the assertion as behavioral (R2) and defaults it to fail (R1). It invokes a bounded verification run (R4) that reproduces the original defect against the implementation. If the defect no longer reproduces, the assertion passes; if it still reproduces, or verification cannot conclude, the assertion fails with a reason (R6, R10).
|
||||
- **Outcome:** "Done" reflects observed behavior, not diff intent.
|
||||
|
||||
- F2. Agent-supplied proof
|
||||
- **Trigger:** A behavioral assertion arrives with an agent-authored regression test as proof.
|
||||
- **Actors:** A1, A2, A3
|
||||
- **Steps:** The verification run executes the test (R9) and confirms it genuinely exercises the asserted behavior — failing on the pre-fix state and passing now. A weak or non-exercising test does not satisfy the assertion (R8).
|
||||
- **Outcome:** Honest proof accelerates a pass; fake or weak proof does not buy one.
|
||||
|
||||
- F3. Escape into the fix loop
|
||||
- **Trigger:** Verification returns fail/blocked for a behavioral assertion.
|
||||
- **Actors:** A2, A4
|
||||
- **Steps:** The Feature does not reach a passing Validator Run, so it does not count toward Slice completion; a Fix Feature carries the remediation, seeded with the recorded reason (R10).
|
||||
- **Outcome:** The false-pass path is closed; the work re-enters the loop instead of shipping wrong.
|
||||
|
||||
## Acceptance Examples
|
||||
|
||||
- AE1. Covers R1, R5. **Given** a bug-fix assertion "clicking Save no longer drops the form," **when** the diff changes Save handling but the bug still reproduces under verification, **then** the assertion fails and the Feature does not reach done.
|
||||
- AE2. Covers R8. **Given** the agent's verdict narrative asserts the bug is fixed but supplies no executable proof and verification cannot confirm the behavior, **when** the judge evaluates the assertion, **then** it defaults to fail rather than accepting the narrative.
|
||||
- AE3. Covers R3. **Given** a non-behavioral assertion ("the new flag is documented in the README"), **when** the judge evaluates it, **then** static inspection applies with no verification run and no added strictness.
|
||||
- AE4. Covers R6. **Given** a verification run that exceeds its time/cost bound before concluding, **when** it is terminated, **then** the assertion resolves to a non-passing verdict, never a default pass.
|
||||
- AE5. Covers R9. **Given** an agent-supplied test that passes both before and after the fix (so it never actually exercised the defect), **when** verification inspects it, **then** the proof is rejected and the assertion is not satisfied.
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### Deferred for later
|
||||
|
||||
- The self-tightening gate (Approach C): caught escapes feeding back to harden the relevant assertion and seed a permanent check so the same bug class can't pass again. This is the compounding second move, built on top of A+B once they're in place.
|
||||
|
||||
### Outside this product's identity
|
||||
|
||||
- This work targets the "completes but wrong" failure mode for *behavioral correctness*. Off-target or low-quality work that isn't a behavioral/bug defect (e.g., a stylistically poor but functionally correct implementation) is not what this gate is being sharpened to catch.
|
||||
|
||||
## Dependencies / Assumptions
|
||||
|
||||
- Assumes a bounded execution environment is available to the verification run for exercising code and driving the app (the same behavioral-testing capability used to capture the manual proof today). Whether verification reuses existing worktree/session infrastructure is a planning decision.
|
||||
- Assumes the accepted change to the read-only validator invariant is limited to a non-mutating verification capability — the validator continues to create no board task and edit no code (R7).
|
||||
- A stricter gate will produce more failing verdicts and therefore more Fix Features and longer loops; this rework cost is accepted in exchange for a trustworthy "done."
|
||||
|
||||
## Outstanding Questions
|
||||
|
||||
### Deferred to planning
|
||||
|
||||
- How behavioral-vs-non-behavioral assertion classification is determined (assertion authoring convention, judge inference, or an explicit assertion field).
|
||||
- What the verification run reuses for execution (test runner invocation, browser/app driving) and how its time/cost bounds are set.
|
||||
- How an agent is expected to express executable proof so the judge can locate and run it.
|
||||
@@ -75,6 +75,16 @@ Instead, features are routed through validator execution after lazy assertion en
|
||||
- Backfill pre-restores missing managed assertions for visibility/reporting.
|
||||
- Runtime behavior is unchanged because lazy ensure already guarantees validator-backed enforcement.
|
||||
|
||||
4. **Feature has a behavioral / bug-fix assertion**
|
||||
- The read-only AI judge produces an *advisory* verdict only.
|
||||
- The assertion defaults to fail unless a bounded, **non-mutating verification run** confirms the observable behavior by exercising the code (test suite / agent-supplied regression test against a disposable checkout under an isolating sandbox).
|
||||
- A genuine behavioral failure → `fail` → Fix Feature with a recorded observed-vs-expected reason.
|
||||
- Verification that cannot run or conclude (no isolating backend, timeout, isolation-setup failure, rejected proof, detected flakiness) → `inconclusive` → needs-attention, **no Fix Feature**, never a default pass.
|
||||
- The verification run creates no board task, mutates no mission/board row, and leaves the source tree git-clean.
|
||||
|
||||
5. **Static assertion (e.g. "documented in README")**
|
||||
- Keeps the existing read-only static judging path; no verification run is invoked and no added strictness applies.
|
||||
|
||||
## UI contract
|
||||
|
||||
MissionManager must present mission criteria as **AI-validated** rather than informational:
|
||||
@@ -91,4 +101,6 @@ For any mission feature that reaches validation trigger points:
|
||||
- a validator run must occur,
|
||||
- the feature must not auto-pass due to missing assertion links,
|
||||
- milestone acceptance text must be visible to the validator when present,
|
||||
- a behavioral/bug assertion must not pass on the read-only judge's advisory verdict alone — it requires a confirming non-mutating verification run,
|
||||
- a non-passing verification must resolve to `fail` or `inconclusive`, never a default pass,
|
||||
- advancement decisions must derive from validator outcomes only.
|
||||
|
||||
@@ -486,8 +486,18 @@ On task completion, the scheduler calls `MissionExecutionLoop.processTaskOutcome
|
||||
1. Find the feature linked to the completed task
|
||||
2. If assertions are linked, keep feature completion gated until validation passes
|
||||
3. Transition feature to `validating` state
|
||||
4. Fire AI validator agent against contract assertions
|
||||
5. Record `MissionValidatorRun` metadata for the validation attempt (per-assertion failures are stored separately in `MissionAssertionFailureRecord` rows)
|
||||
4. Fire the AI validator agent (read-only judge) against contract assertions
|
||||
5. Apply the **behavioral-verification posture** (see below): static assertions keep the judge's verdict; behavioral/bug assertions default to fail until a bounded, non-mutating verification run confirms them
|
||||
6. Record `MissionValidatorRun` metadata for the validation attempt (per-assertion failures are stored separately in `MissionAssertionFailureRecord` rows)
|
||||
|
||||
**Behavioral-verification posture (adversarial default-to-fail).** A Contract Assertion now carries a `type` (`static` | `behavioral`). The validator no longer grades a Feature "done" purely from the diff's apparent intent:
|
||||
|
||||
- **Static assertions** (e.g. "documented in README") keep today's read-only static judging — no added cost or strictness.
|
||||
- **Behavioral / bug-fix assertions** *default to fail*. The read-only judge's "pass" on a behavioral assertion is **advisory, not authoritative**; an authoritative pass requires a separate, bounded **verification run** that exercises the implemented code (running the test suite / an agent-supplied regression test against a disposable checkout) and confirms the observable behavior. An agent's narrative claim is not evidence on its own.
|
||||
|
||||
**The verification run is not read-only and is not part of the judge session.** The AI judge session stays `tools: "readonly"` (no `bash`/`edit`/`write`/task-mutation). The verification run is a *separate*, side-effecting execution that runs against an isolating sandbox backend (fail-closed when none is available) and a disposable checkout at a trusted revision — never the live worktree, never the repo root. Its effects are confined to that disposable surface: it creates no board task, mutates no mission/board row, and leaves the source tree that feeds diff/merge byte-identical (git-clean) after the run. Verification is therefore no longer "purely read-only/static" — but it is *non-mutating to mission/board state*, which is the invariant the recovery sweep and reaper depend on (see Surface Enumeration).
|
||||
|
||||
**Inconclusive is a first-class verdict, distinct from fail.** Verification yields `pass` / `fail` / `inconclusive`. A real behavioral failure (`fail`) spawns a Fix Feature with a recorded observed-vs-expected reason. An **inconclusive** verdict — verification could not run or conclude (no isolating backend, timeout, isolation-setup failure, rejected proof, detected flakiness) — routes the feature to a blocked/needs-attention state with a persisted `verification_inconclusive` mission event and **spawns no Fix Feature**, so a fragile verification surface cannot manufacture remediation churn. A non-passing verification never resolves to a default pass.
|
||||
|
||||
Mission validation resolves its model from the validator lane before session creation: assigned agent runtime model (when the linked task has an assigned durable agent) → per-task `validatorModelProvider`/`validatorModelId` → project `validatorProvider`/`validatorModelId` → global `validatorGlobalProvider`/`validatorGlobalModelId` → project `defaultProviderOverride`/`defaultModelIdOverride` → global `defaultProvider`/`defaultModelId`. In `testMode`, validation is forced to `mock/scripted` instead of falling through to provider auto-detection.
|
||||
|
||||
@@ -515,7 +525,9 @@ interface MissionValidatorRun {
|
||||
|
||||
**Validation timeout:** 10 minutes (`VALIDATION_TIMEOUT_MS = 10 * 60 * 1000`). If session creation, auth/credit checks, prompting, or timeout fails, the run is marked `error` and emits a surfaced `validation_error` mission event instead of silently spawning a fix feature.
|
||||
|
||||
**Stale validator-run reaper:** startup recovery and periodic self-healing also sweep `MissionValidatorRun` rows stuck in `status="running"` longer than `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). Ownerless stale runs are reaped to terminal `status="error"`, their reap reason is stored in `summary`, and active mission features are moved to `loopState="needs_fix"` with `lastValidatorStatus="error"` so the loop can re-trigger. Runs whose parent mission is already `complete`/`archived` are still terminated, but their feature state is left untouched. Each successful reap emits a run-audit event with `mutationType: "mission:validator-run-reaped"`.
|
||||
**Stale validator-run reaper:** startup recovery and periodic self-healing also sweep `MissionValidatorRun` rows stuck in `status="running"` longer than `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). Runs still owned by the live process (tracked in `activeValidations`) are skipped, so a slow-but-legitimate verification is never reaped while its session is in-flight. Ownerless stale runs are reaped to terminal `status="error"`, their reap reason is stored in `summary`, and live (non-`done`) mission features are moved to `loopState="needs_fix"` with `lastValidatorStatus="error"` so the loop can re-trigger. A *done* feature's loop state is intentionally left untouched (it keeps the `loopState="validating"` set when the run started) so the reaper does not rewrite a feature that already finished its task. Runs whose parent mission is already `complete`/`archived` are likewise terminated without touching feature state. Each successful reap emits a run-audit event with `mutationType: "mission:validator-run-reaped"`.
|
||||
|
||||
**Verification wall-clock is bounded under the reaper window.** The aggregate verification budget — checkout materialization plus the test-suite command (`VERIFICATION_COMMAND_TIMEOUT_MS`, 10 min), including the optional pre-fix baseline run — is provably far shorter than the 6-hour reaper stale window, so a legitimate verification run completes long before it would be eligible for reaping. The reaper's `activeValidations` skip is the second line of defense: an in-flight run is never reaped regardless of wall-clock.
|
||||
|
||||
### Phase 5: Fix-Feature Retries
|
||||
|
||||
@@ -548,6 +560,23 @@ A feature transitions to `blocked` when:
|
||||
|
||||
On engine restart, `recoverActiveMissions()` re-enqueues features in `validating` or `needs_fix` states, ensuring no validation work is lost. It also re-triggers `implementing` features whose linked task is already `done`/`archived` and whose assertion validation has not passed yet. When the stale-run reaper has already converted an abandoned validator run into `needs_fix`, `processTaskOutcome()` promotes the feature back through `implementing` and re-validates instead of skipping it. The same recovery path is replayed during periodic self-heal maintenance, so historically stranded `implementing` features can self-heal without requiring an engine restart.
|
||||
|
||||
**Reaper → slice deadlock closure (P0).** A *task-less, done, assertion-linked* feature is the dangerous case: it carries no board task to re-drive from, and `computeSliceStatus` refuses to count it complete until its validator passes. When the reaper terminates such a feature's stale run, the feature is left stranded in `loopState="validating"` (the reaper's done-guard, above) — a state the `validating`/`needs_fix` recovery branches (which only re-drive features that carry a `taskId`) never re-validate, while default-to-fail would otherwise re-drive it forever to a non-terminal `error`. `recoverActiveMissions()` closes this with a **stranded-done catch-all**: any task-less, done feature in `loopState` `implementing` *or* `validating` (or `needs_fix` + `lastValidatorStatus="error"`) that has not reached a passing validator status and is not currently being validated is re-driven directly through `runFeatureValidation()`. Because the verification run is bounded and non-mutating, this reaches a terminal `pass` / `fail` / `inconclusive` (and the slice can finally resolve) instead of livelocking on `validating`/`error`.
|
||||
|
||||
#### Surface Enumeration — validation re-drive entry points (R15)
|
||||
|
||||
Now that the verification step has side effects (on a disposable, isolated surface — never mission/board state), every site that re-drives validation must remain correct: after a run the source tree feeding diff/merge is git-clean, no duplicate Fix Feature is minted, and a terminal verdict is reached without an `error`-state slice deadlock. The complete set of re-drive entry points, each gated by an adversarial reliability test in `packages/engine/src/__tests__/reliability-interactions/mission-verification-redrive-surface.test.ts`:
|
||||
|
||||
| Entry point | Trigger | Post-conditions asserted |
|
||||
| --- | --- | --- |
|
||||
| `processTaskOutcome()` | Normal task-completion validation | terminal verdict; one Fix Feature on fail (idempotent on re-drive); no validation-created board task |
|
||||
| `recoverActiveMissionValidations` → **validating** branch | Restart with a feature stranded mid-validation (has taskId) | re-driven to terminal verdict; git-clean; no duplicate Fix Feature |
|
||||
| → **needs_fix** branch | Reaped/abandoned run on a feature with a `taskId` | promoted via `processTaskOutcome`; terminal verdict |
|
||||
| → **implementing + taskId** branch | Feature left implementing while its task already finished | re-triggered to terminal verdict |
|
||||
| → **stranded-done catch-all** (`implementing`/`validating`/`needs_fix`+`error`, no taskId) | Orphaned or reaped task-less done feature (the P0 deadlock) | re-driven directly; terminal verdict, never indefinitely re-driven `error`; slice resolves |
|
||||
| `reapStaleMissionValidatorRuns` | Stale ownerless run | run → terminal `error`; live feature → `needs_fix`; done feature loopState untouched; in-flight runs skipped |
|
||||
|
||||
Each path is verified to leave **zero mission/board residue from the verification run itself** — the only board task a failed verdict legitimately creates is the auto-triaged Fix Feature, and an inconclusive verdict creates none.
|
||||
|
||||
For features with missing linked assertions, the completion path is now validator-first: the loop lazily restores the store-managed per-feature assertion just before validation, then runs the AI validator instead of auto-passing. Milestone `acceptanceCriteria` is threaded into the validator prompt for every feature in that milestone, so all mission criteria are AI-evaluated. Contract details are defined in [Mission Completion Gate Contract](./missions-completion-contract.md).
|
||||
|
||||
### Autopilot / Scheduler Interplay
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
title: "feat: Behavioral verification in the Validator"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-06-11
|
||||
origin: docs/brainstorms/2026-06-11-validator-behavioral-verification-requirements.md
|
||||
---
|
||||
|
||||
# feat: Behavioral verification in the Validator
|
||||
|
||||
## Summary
|
||||
|
||||
Make the Validator Run's "done" verdict reflect observed behavior instead of the diff's apparent intent. Behavioral and bug-fix Contract Assertions default to fail unless a bounded, non-mutating verification run confirms them by exercising the code — running the test suite / an agent-supplied regression test, and driving the running app for UI/bug assertions. Non-behavioral assertions keep today's static judging.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
The Validator Run (`MissionExecutionLoop.runFeatureValidation` → `runValidation` in `packages/engine/src/mission-execution-loop.ts`) is a read-only AI judge: it opens a pi session with `tools: "readonly"`, reads the task context (title, description, last ~10 log action strings via `buildTaskContext`), and asks the model to grade each assertion as pass/fail/blocked. It never runs the code.
|
||||
|
||||
That is structurally weak for behavioral correctness. A bug-fix Feature was marked done after the judge accepted a diff that *looked* like a fix; the bug was still live in the running app. The judge graded intent; reality was never consulted, even though behavioral truth existed (it was caught and captured by hand). A gate that rubber-stamps means every "done" inherits a manual re-check tax, which defeats the gate.
|
||||
|
||||
Two hard facts from research shape the work:
|
||||
|
||||
- **The read-only invariant is enforced at the tool layer, not by convention.** `packages/engine/src/workflow-step-tool-policy.ts` denies `bash`/`edit`/`write` and task-mutation tools in readonly sessions; `packages/engine/src/pi.ts` strips host extensions for a "hermetically sealed" readonly session. The judge therefore *cannot run anything* today. Recovery sweeps and the validator reaper also rely on validation being side-effect-free (see origin and `docs/solutions/logic-errors/mission-autopilot-stalled-by-stranded-done-feature.md`).
|
||||
- **App/browser driving does not exist yet.** `plugins/fusion-plugin-agent-browser/` is a metadata/probe stub (no Playwright/CDP); there is no computer-use or demo-reel. Test-suite execution, by contrast, already exists and is reusable (`packages/engine/src/verification-utils.ts` + `packages/engine/src/sandbox/`).
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
### Judging posture (origin R1–R3, R8–R10)
|
||||
|
||||
- R1. For behavioral and bug-fix assertions, the validator defaults to a fail verdict unless backed by verified behavioral evidence (origin R1).
|
||||
- R2. Each Contract Assertion is classified behavioral/bug vs non-behavioral; the strict posture applies only to the former (origin R2).
|
||||
- R3. Non-behavioral assertions retain existing static judging — no added cost or strictness where inspection suffices (origin R3).
|
||||
- R4. A behavioral/bug assertion passes only when verification evidence corroborates it; an agent's narrative claim is not sufficient evidence on its own (origin R8).
|
||||
- R5. When an agent supplies executable proof (a regression test), the validator confirms the proof is genuine — it fails on the pre-fix state and passes now — rather than accepting its presence at face value (origin R9).
|
||||
- R6. A non-passing verdict records why it failed (which assertion, observed vs expected behavior) in a form the generated Fix Feature can act on (origin R10).
|
||||
|
||||
### Behavioral verification capability (origin R4–R7)
|
||||
|
||||
- R7. The gate can invoke a bounded verification run that exercises the implemented code to confirm a behavioral/bug assertion's observable outcome before passing it (origin R4).
|
||||
- R8. For bug-fix assertions, verification confirms the reported defect is no longer reproducible — not merely that a plausible change was made (origin R5).
|
||||
- R9. The verification run is bounded in time and cost; a run that cannot complete or conclude resolves to a non-passing verdict (fail/blocked/error), never a default pass (origin R6).
|
||||
- R10. The verification run is non-mutating to mission/board state: it creates no board task, mutates no mission/board row, and edits no version-controlled source (origin R7).
|
||||
- R11. (plan-level) Verification exercises the Feature's implemented code at a trusted revision in a disposable checkout (not the live task worktree — which is pruned before the done-transition that triggers validation — and not the repo root).
|
||||
- R17. (plan-level) Filesystem isolation is a named invariant distinct from R10: verification executes against a disposable copy, and the source tree that feeds diff/merge is byte-identical (git-clean) after a run. Build artifacts, `node_modules`, coverage, and scratch DBs land only in the disposable surface.
|
||||
|
||||
### App-driving capability
|
||||
|
||||
- R12. A real app/browser-driving capability exists and is reachable from the verification run, sufficient to reproduce a UI/bug assertion's observable behavior against a running instance of the app.
|
||||
- R13. App-driving runs against an isolated surface: never binds the reserved dashboard port, never dispatches on the shared central DB, and exercises a freshly-built bundle so verification cannot produce its own false verdicts. The disposable DB is created fresh/empty (never a copy of the central DB), lives under a run-unique tmpdir, is torn down unconditionally including on crash/timeout, and excludes credentials and agent logs.
|
||||
|
||||
### Safety and observability
|
||||
|
||||
- R14. The non-mutating execution path does not grant the judge `edit`/`write`/`bash` or task-mutation tools; the read-only verdict boundary is preserved and test-enforced.
|
||||
- R15. Every site that re-drives validation (recovery sweep, validator reaper) remains correct once verification has side effects — i.e., verification's effects are confined to an isolated, disposable surface and leave no mission/board residue.
|
||||
- R16. Verification-run failures and the resulting Fix-Feature triage are durably observable (persisted mission/audit events), not logged-and-continued.
|
||||
- R18. (plan-level) Verification executes under an isolating sandbox backend (bubblewrap on Linux, sandbox-exec on macOS) with an explicit deny policy; if no isolating backend is available the verification run fails closed rather than falling through to the unrestricted native backend. The child process environment is scrubbed to a minimal allowlist (no API keys, auth tokens, or DB credentials inherited).
|
||||
- R19. (plan-level) The verification command is constructed from a fixed system-owned template; agent-supplied proof contributes only a validated test-file path (within the disposable checkout, no shell metacharacters), never a free-form command string.
|
||||
- R20. (plan-level) Verification verdicts are reproducible: a flaky/non-deterministic result resolves to inconclusive rather than fail, an authoritative fail requires N-run agreement, and no verdict feeds the deferred self-tightening gate (Approach C) unless it is reproducible.
|
||||
- R21. (plan-level) "Behavior observed wrong" and "verification could not run" are distinct verdicts. A genuine behavioral failure spawns a Fix Feature; an infra-driven inconclusive (driver unavailable, timeout, isolation setup failure) routes to a blocked/needs-attention state that does not spawn remediation work, and the infra-failure rate is tracked separately.
|
||||
- R22. (plan-level) Fix-Feature generation is idempotent across re-drives: creation is deduped on (source feature, originating run), and a re-drive reuses an existing verdict for the current implementation revision rather than minting a fresh failing run.
|
||||
|
||||
---
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Assertion classification via an explicit `type` field, not judge inference.** `MissionContractAssertion` (`packages/core/src/mission-types.ts`) carries no type metadata today. Add a typed field (e.g. `behavioral` | `static`, defaulting conservatively) persisted on `mission_contract_assertions`, populated when assertions are authored/seeded (`addContractAssertion`, `seedContractAssertionsForFeatures`, `ensureFeatureAssertion`). Rationale: a stored, upstream-authored type is reproducible and auditable; run-time judge inference reintroduces the same leniency we're trying to remove. Requires a schema-version bump and the repo-wide Schema-Version Sweep (per CONCEPTS.md).
|
||||
- **Terminology.** A single named concept runs through the plan: the **verification run** is one bounded execution that confirms a behavioral/bug assertion; the **verification capability** is the injected dependency that performs it; the **verification step** is its position in the validation flow (between the read-only judge's behavioral verdict and the aggregate). These three terms are used consistently; there is no separate "verification path."
|
||||
- **Two-step validation: read-only verdict, then isolated verification — never one session with execute rights.** Keep the AI judge session `tools: "readonly"`. Introduce a *separate* non-mutating verification run that executes code through `verification-utils.runVerificationCommand` + the sandbox backend, and (for UI assertions) the app-driver. This preserves R14 and the recovery/reaper safety assumption: the judge stays side-effect-free; only the bounded verification run touches a disposable surface. Rationale: flipping the judge session to allow `bash` would blow the hermetic boundary and every readonly guarantee with it.
|
||||
- **Mandatory isolating sandbox backend; fail closed.** `resolveSandboxBackend()` with no argument falls through to `NativeSandboxBackend`, which advertises no filesystem or network policy and inherits full `process.env` — unacceptable for executing agent-authored code. The verification run must select an isolating backend explicitly (bubblewrap / sandbox-exec) with a deny policy, scrub the environment to a minimal allowlist, and fail closed (resolve to a non-pass) when no isolating backend is available rather than running unsandboxed (R18). This is the structural enforcement behind R10/R17 — the non-mutating guarantee is a boundary, not a convention the verification code is trusted to honor.
|
||||
- **Verification runs against a disposable checkout at a trusted revision, not the live worktree.** The task worktree is removed by the merger *before* the task moves to `done`, and the done-transition is what triggers validation — so the live worktree is gone by the time verification runs. Verification materializes a disposable checkout at a trusted revision (the integration SHA), keeping the source tree that feeds diff/merge byte-clean (R11, R17). For agent-supplied regression proof, the pre-fix baseline is a *second* disposable checkout at `git merge-base` of the feature branch and the base branch — a revision the agent does not control — so a test that passes on both pre and post is genuinely exercising the defect (R5, AE5).
|
||||
- **Fixed command template; agent supplies only a validated test path.** `runVerificationCommand` runs a shell command string. To prevent the implementing agent from gaming or escaping its own verification, the command is built from a system-owned template (e.g. the project's scoped test invocation) into which only a validated test-file path is substituted (within the disposable checkout, rejected if it contains shell metacharacters) — never a free-form agent-supplied command (R19).
|
||||
- **Inconclusive is a first-class verdict, distinct from fail.** Verification yields pass / fail / **inconclusive**. Real behavioral failure → Fix Feature; inconclusive (driver unavailable, timeout, isolation setup failure, detected flakiness) → blocked/needs-attention, *no* Fix Feature, with infra-failure rate tracked. This keeps a brand-new, fragile app-driver from manufacturing Fix-Feature churn that depresses task completion, and gives the determinism contract (R20) somewhere to route flaky results.
|
||||
- **Verification wall-clock is bounded under the reaper window.** `reapStaleMissionValidatorRuns` resolves a stale run to `lastValidatorStatus: "error"`, which `computeSliceStatus` never accepts as complete — so a slow verification reaped mid-run, combined with default-to-fail re-drive, livelocks the slice. The aggregate verification budget (build + suite + driving, including the pre-fix baseline run) must be provably shorter than the reaper stale window, or the reaper must distinguish an owned-but-slow run from an abandoned one. Owned by U7.
|
||||
- **Reuse `verification-utils` + sandbox for test execution; do not invent a runner.** `runVerificationCommand` already provides timeout (`VERIFICATION_COMMAND_TIMEOUT_MS`), buffer caps, abort-signal/process-group kill, output summarization, and sandbox dispatch. The verification run wires into it rather than spawning processes directly.
|
||||
- **Express verification as a runtime-style capability injected into the loop, not a hidden branch.** Per `docs/solutions/architecture-patterns/workflow-native-runtime-primitives.md`, side-effecting work belongs behind an explicit injected boundary. Model the verification capability as an injected dependency on `MissionExecutionLoop` so it is testable and swappable (real vs mock), mirroring how `createFnAgent` is injected.
|
||||
- **App-driving is a net-new driver, not "wiring up a stub" — split it.** `plugins/fusion-plugin-agent-browser/` today is a metadata/probe stub: its only tool validates a URL allowlist and `probe.ts` shells `agent-browser --version` to detect an external binary; there is no Playwright/CDP and no in-process automation (`src/tools.ts`, `src/probe.ts`). Standing up real navigate/interact/observe driving plus an isolated-app-launch harness (fresh bundle, disposable DB, reserved-port avoidance, deterministic teardown) is two substantial subsystems. Split into U4 (isolation/launch harness with its safety contracts test-characterized first) and U8 (the actual driver), with U5 depending on U8. Name the driving technology at implementation time. The driver lives inside the existing agent-browser plugin boundary; its tools must be scoped to verification, not exposed to coding-agent sessions.
|
||||
- **Default-to-fail is scoped, not global.** Only assertions typed behavioral/bug take the default-fail posture; static assertions are untouched (R3). This bounds the blast radius on slice completion — but see Risks & Dependencies below, since `lastValidatorStatus` gates `computeSliceStatus`.
|
||||
|
||||
---
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
A[Task moves to done] --> B[scheduler.processTaskOutcome]
|
||||
B --> C[runFeatureValidation]
|
||||
C --> D[listAssertionsForFeature<br/>+ classify by type]
|
||||
D --> E[Read-only AI judge<br/>tools: readonly]
|
||||
E --> F{Assertion type?}
|
||||
F -->|static| G[Static verdict<br/>unchanged path]
|
||||
F -->|behavioral / bug| H{Verified evidence?}
|
||||
H -->|default| I[Bounded verification run<br/>NON-MUTATING, disposable checkout<br/>isolating sandbox, fail-closed]
|
||||
I --> I1[Test suite / regression test<br/>via verification-utils + sandbox]
|
||||
I --> I2[App-driving<br/>isolated port + disposable DB + fresh bundle]
|
||||
I1 --> J{Behavior confirmed?}
|
||||
I2 --> J
|
||||
J -->|yes| K[assertion pass]
|
||||
J -->|wrong| L[fail → Fix Feature<br/>+ recorded reason]
|
||||
J -->|inconclusive / timeout / flaky| L2[blocked → needs-attention<br/>no Fix Feature]
|
||||
G --> M[aggregate verdict]
|
||||
K --> M
|
||||
L --> M
|
||||
L2 --> M
|
||||
M -->|all pass| N[handleValidationPass<br/>feature done]
|
||||
M -->|any fail| O[handleValidationFail<br/>Fix Feature + durable event]
|
||||
M -->|inconclusive| P[blocked / needs-attention<br/>no Fix Feature, infra rate tracked]
|
||||
```
|
||||
|
||||
The judge box (E) keeps `tools: "readonly"`. The verification box (I) is the only new side-effecting surface, isolated and disposable, invoked between the judge's behavioral verdict and the aggregate.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. Assertion type data model
|
||||
|
||||
- **Goal:** Give Contract Assertions a persisted classification so verification can be scoped to behavioral/bug assertions.
|
||||
- **Requirements:** R2, R3
|
||||
- **Dependencies:** none
|
||||
- **Files:**
|
||||
- `packages/core/src/mission-types.ts` (add `type` to `MissionContractAssertion`, `ContractAssertionCreateInput`/`UpdateInput`; add the type union + const array alongside `MISSION_ASSERTION_STATUSES`)
|
||||
- `packages/core/src/db.ts` (add column to `mission_contract_assertions`; migration; schema-version bump)
|
||||
- `packages/core/src/mission-store.ts` (`addContractAssertion`, `seedContractAssertionsForFeatures`, `ensureFeatureAssertion`, `backfillFeatureAssertions` populate/default the field)
|
||||
- `packages/core/src/__tests__/` (new store + migration tests)
|
||||
- Schema-Version Sweep targets (non-core readers of the version constant, in addition to any in-core asserts): `packages/plugin-sdk/src/index.ts`, `packages/cli/src/plugin-sdk-core-runtime-shim.ts`, `packages/dashboard/src/routes/register-workflow-routes.ts`, and plugins re-exporting the constant (e.g. `plugins/fusion-plugin-roadmap/`). Prefer asserting against the exported constant over a literal so future bumps drop these from the sweep.
|
||||
- **Approach:** Conservative default for existing rows (treat unknown as the type that preserves current behavior unless the assertion clearly reads behavioral). Same validating-store-authority pattern used elsewhere. Run the Schema-Version Sweep in the same commit as the bump.
|
||||
- **Patterns to follow:** existing `MISSION_ASSERTION_STATUSES` const + validation; the Moved Settings Keys / schema-version sweep discipline in CONCEPTS.md.
|
||||
- **Test scenarios:**
|
||||
- Creating an assertion with an explicit type persists and reloads it.
|
||||
- Creating without a type yields the conservative default.
|
||||
- Seeding/lazy-link (`ensureFeatureAssertion`) sets a type.
|
||||
- Migration upgrades an old DB: existing assertion rows get the default type, no row rewritten beyond the new column.
|
||||
- Schema-version assertion sweep: version constant readers still pass.
|
||||
- **Verification:** core typecheck + new store/migration tests green; `pnpm test:gate` unaffected.
|
||||
|
||||
### U2. Classification + default-to-fail posture in the verdict path
|
||||
|
||||
- **Goal:** Make the judge default behavioral/bug assertions to fail unless verified, while leaving static assertions on the existing path.
|
||||
- **Requirements:** R1, R2, R3, R4
|
||||
- **Dependencies:** U1
|
||||
- **Files:**
|
||||
- `packages/engine/src/mission-execution-loop.ts` (`runValidation`, `parseValidationResult`, `extractAssertionResults`; branch per assertion type)
|
||||
- validation prompt builders (`buildValidationSystemPrompt`, `buildValidationPrompt`) — adversarial framing for behavioral assertions: "not met until proven" + ask for the proof location
|
||||
- `packages/engine/src/__tests__/reliability-interactions/` (new test alongside existing mission-validator tests)
|
||||
- **Approach:** After the read-only judge returns, partition assertions by type. Static assertions keep the current verdict. Behavioral/bug assertions are forced to non-pass *unless* U3's verification confirms them — i.e. the judge's "pass" on a behavioral assertion is advisory, not authoritative. Keep the judge session `tools: "readonly"` (no change to its tool policy).
|
||||
- **Patterns to follow:** existing `extractAssertionResults` per-assertion mapping; `isTestModeActive`/mock provider for deterministic tests.
|
||||
- **Test scenarios:**
|
||||
- Behavioral assertion with no verification evidence → fail, even if the judge text claims pass (origin AE2).
|
||||
- Static assertion (e.g. "documented in README") → static verdict, no verification invoked (origin AE3).
|
||||
- Mixed assertion set → static and behavioral each take their correct path; aggregate reflects both.
|
||||
- `Covers AE2.` narrative-only claim defaults to fail.
|
||||
- `Covers AE3.` non-behavioral inspected statically.
|
||||
- **Verification:** engine typecheck; new reliability test green; existing mission-validator tests still pass.
|
||||
|
||||
### U3. Non-mutating verification step (test execution)
|
||||
|
||||
- **Goal:** Add a bounded, non-mutating verification run that confirms a behavioral/bug assertion by running the suite / an agent-supplied regression test against a disposable checkout of the implemented code.
|
||||
- **Requirements:** R5, R7, R8, R9, R10, R11, R14, R17, R18, R19
|
||||
- **Dependencies:** U1, U2
|
||||
- **Files:**
|
||||
- `packages/engine/src/mission-verification.ts` (new — the injected verification capability; wraps `verification-utils.runVerificationCommand`)
|
||||
- `packages/engine/src/mission-execution-loop.ts` (inject + invoke between behavioral verdict and aggregate; materialize the disposable checkout)
|
||||
- `packages/engine/src/verification-utils.ts` (reuse; extend only if a non-mutating mode flag is needed)
|
||||
- `packages/engine/src/sandbox/index.ts` (verification must request an isolating backend explicitly, never the no-arg native fallback)
|
||||
- `packages/engine/src/workflow-step-tool-policy.ts` (only if a narrowly-allowlisted verification path is required; add a test that it grants no edit/write/task-mutation)
|
||||
- `packages/engine/src/__tests__/` and `reliability-interactions/`
|
||||
- **Approach:** Materialize a disposable checkout at a trusted revision (the integration SHA) — the live task worktree is pruned before the done-transition that triggers validation, so it cannot be relied on (R11). Execute through an explicitly-selected isolating sandbox backend (bubblewrap/sandbox-exec) with a deny policy and a scrubbed env allowlist; fail closed to a non-pass if no isolating backend is available (R18). Build the command from a system-owned template, substituting only a validated test-file path from agent-supplied proof (R19). For agent-supplied proof, confirm the regression test fails on a *second* disposable checkout at `git merge-base` (feature branch vs base branch — not agent-controlled) and passes on the implementation (R5); a test that passes on both is rejected (origin AE5). After the run, assert the source tree that feeds diff/merge is git-clean (R17). Inconclusive/timeout → non-pass (R9). The run performs no board/mission writes (R10).
|
||||
- **Execution note:** Start with a failing test asserting the isolation contract — source tree git-clean after a run, no board/mission mutation, no edit/write/bash granted, isolating backend required — before wiring real execution.
|
||||
- **Patterns to follow:** `executor.ts`/`merger.ts` use of `runVerificationCommand`; `bubblewrap-policy.ts` env filtering; injection pattern of `createFnAgent`.
|
||||
- **Open question (deferred to implementation):** exact per-assertion command derivation (how the verification run determines *which* invocation exercises a given assertion), and whether `runVerificationCommand`'s `store.logEntry`/`appendAgentLog` writes count against the R10 non-mutating invariant — see Open Questions.
|
||||
- **Test scenarios:**
|
||||
- `Covers AE1.` bug-fix assertion where the bug still reproduces under verification → fail, Feature not done.
|
||||
- `Covers AE5.` agent test passing both on the merge-base checkout and the implementation → proof rejected, assertion not satisfied.
|
||||
- Bug no longer reproduces → assertion passes.
|
||||
- `Covers AE4.` verification exceeds its time/cost bound → terminated → non-pass, never default pass.
|
||||
- Verification runs against a disposable checkout at the integration SHA, not rootDir and not the (pruned) task worktree.
|
||||
- Source tree feeding diff/merge is byte-identical (git-clean) after a verification run (R17).
|
||||
- No isolating sandbox backend available → run fails closed to non-pass, never executes on the native backend (R18).
|
||||
- Command-template injection: an agent-supplied test path containing shell metacharacters is rejected before execution (R19).
|
||||
- Non-mutation: after a verification run, no new board task, no mission/board row mutated.
|
||||
- Tool-policy test: the verification path exposes no `edit`/`write`/`bash`/task-mutation tool.
|
||||
- **Verification:** engine typecheck; new tests green; `pnpm test:gate` green.
|
||||
|
||||
### U4. Isolated-app-launch harness
|
||||
|
||||
- **Goal:** Stand up the isolated app surface that the driver (U8) will drive — independent of any navigation logic, so its safety contracts can be characterized and trusted first.
|
||||
- **Requirements:** R13
|
||||
- **Dependencies:** U3
|
||||
- **Files:**
|
||||
- engine wiring to launch/tear down an isolated app instance for verification
|
||||
- `__tests__/` for the isolation contracts
|
||||
- **Approach:** Launch an isolated app instance: a non-reserved port (respect `RESERVED_DASHBOARD_PORT` / `FUSION_RESERVED_PORTS` and the existing port-4040 guards), a disposable DB created fresh/empty (never a copy of the central DB) under a run-unique tmpdir, seeded only with the minimal fixtures a UI assertion needs (no credentials, no agent logs), and a freshly-built client bundle (avoid the stale `dist/client` trap). Tear down unconditionally, including on crash/timeout.
|
||||
- **Execution note:** Characterize the isolation guarantees (port, DB lifecycle, bundle freshness, teardown) with tests *before* U8 does any real navigation — these are the exact traps in `docs/solutions/developer-experience/browser-testing-dashboard-from-worktree-safely.md`.
|
||||
- **Patterns to follow:** the worktree port/DB isolation guidance in the learnings doc; the port-4040 guards already in the repo.
|
||||
- **Test scenarios:**
|
||||
- Launch never binds the reserved dashboard port (assert against a reserved-port config).
|
||||
- The instance uses a fresh/empty disposable DB under a unique tmpdir, not the shared central DB; the DB contains no credentials or agent logs.
|
||||
- A stale bundle is rebuilt before launch (no false verdict from stale `dist/client`).
|
||||
- Teardown leaves no lingering process / bound port / DB file, including after a simulated crash or timeout.
|
||||
- **Verification:** engine typecheck; new tests green.
|
||||
|
||||
### U8. App/browser driver
|
||||
|
||||
- **Goal:** Provide the real navigate/interact/observe driver so UI/bug assertions can be reproduced against the isolated instance from U4.
|
||||
- **Requirements:** R12
|
||||
- **Dependencies:** U4
|
||||
- **Files:**
|
||||
- `plugins/fusion-plugin-agent-browser/src/tools.ts`, `plugins/fusion-plugin-agent-browser/src/probe.ts`, plugin entry — replace the metadata/probe stub with a real driver
|
||||
- plugin `__tests__/`
|
||||
- **Approach:** Build a real driver (name the automation technology — e.g. Playwright/CDP — at implementation time; the current package has no automation dependency) inside the existing agent-browser plugin boundary. Its tools are scoped to verification contexts and must not be exposed to coding-agent sessions. The driver targets the isolated instance U4 launches.
|
||||
- **Patterns to follow:** `@fusion/plugin-sdk` tool registration; verification-only tool scoping.
|
||||
- **Test scenarios:**
|
||||
- Driver reproduces a known UI behavior against the isolated instance and reports the observed outcome.
|
||||
- Driver tools are not reachable from a normal coding-agent session (scope test).
|
||||
- A structurally un-exercisable assertion (selector unreachable, state the driver can't set up) resolves to inconclusive, not fail (feeds R21).
|
||||
- **Verification:** plugin + engine typecheck; new tests green; manual smoke driving the dashboard on a throwaway port.
|
||||
|
||||
### U5. Wire app-driving into UI/bug verification
|
||||
|
||||
- **Goal:** Route UI/bug behavioral assertions through the app driver inside the verification run.
|
||||
- **Requirements:** R8, R11, R12, R20, R21
|
||||
- **Dependencies:** U3, U8
|
||||
- **Files:**
|
||||
- `packages/engine/src/mission-verification.ts` (dispatch to test-execution vs app-driving based on assertion shape)
|
||||
- `packages/engine/src/__tests__/`
|
||||
- **Approach:** Within the verification run, choose the evidence channel: test/regression execution for code-level behavior (U3), app-driving (U8 against U4's isolated instance) for UI/bug behavior (some assertions use both). Distinguish three outcomes per the inconclusive-verdict decision: behavior observed wrong → fail; behavior confirmed → pass; driver unavailable / structurally un-exercisable / detected flaky → **inconclusive** (R21), which does not spawn a Fix Feature. Apply the determinism contract (R20): a flaky result resolves to inconclusive, and an authoritative fail requires N-run agreement.
|
||||
- **Test scenarios:**
|
||||
- UI bug assertion still reproduces via the driver → fail.
|
||||
- UI bug assertion no longer reproduces → pass.
|
||||
- Driver unavailable / un-exercisable → inconclusive (never default pass, never auto-fail-into-Fix-Feature), with recorded reason.
|
||||
- Flaky result (passes then fails across N runs) → inconclusive, not fail (R20).
|
||||
- Assertion needing both channels passes only when both confirm.
|
||||
- **Verification:** engine typecheck; new tests green.
|
||||
|
||||
### U6. Failure reasons, Fix-Feature linkage, and durable observability
|
||||
|
||||
- **Goal:** Carry verification failure detail into the Fix Feature, make Fix-Feature generation idempotent across re-drives, route inconclusive verdicts away from remediation, and make verification + triage failures durably observable.
|
||||
- **Requirements:** R6, R16, R21, R22
|
||||
- **Dependencies:** U2, U3
|
||||
- **Files:**
|
||||
- `packages/engine/src/mission-execution-loop.ts` (`handleValidationFail`, `recordValidatorFailures`; include observed-vs-expected reason; route inconclusive to blocked/needs-attention rather than Fix Feature)
|
||||
- `packages/core/src/mission-store.ts` (`createGeneratedFixFeature` — dedup on (source feature, originating run); lineage already carries failed assertion IDs; thread reason text)
|
||||
- mission event/audit emission for verification failures, inconclusive/infra outcomes, and fix-triage failures
|
||||
- `__tests__/` for both packages
|
||||
- **Approach:** Record per-assertion observed vs expected on `mission_validator_failures` and surface it in the generated Fix Feature. Only a real behavioral *fail* spawns a Fix Feature; an *inconclusive* verdict routes to blocked/needs-attention with the infra-failure rate tracked, never minting remediation work (R21). Make Fix-Feature creation idempotent: dedup on (sourceFeatureId, generatedFromRunId) or skip when an open Fix Feature already exists for the source, and reuse an existing verdict for the current implementation revision on re-drive rather than minting a fresh failing run (R22) — this prevents recovery/reaper re-drives from exhausting the retry budget and force-blocking a correct feature. Emit persisted mission/audit events for verification failures, inconclusive outcomes, and any swallowed triage error (per the branch-group-collision learning).
|
||||
- **Test scenarios:**
|
||||
- `Covers AE1.` failed verification produces a Fix Feature carrying the recorded reason.
|
||||
- Inconclusive verdict routes to blocked/needs-attention and spawns no Fix Feature (R21).
|
||||
- Re-drive of an already-failed feature does not create a duplicate Fix Feature (R22).
|
||||
- A flaky verification across recovery re-drives does not exhaust the retry budget and force-block a correct feature.
|
||||
- Verification failure and inconclusive outcomes each emit a persisted mission/audit event (not just a log line).
|
||||
- A triage failure on the generated Fix Feature is durably recorded, not silently swallowed.
|
||||
- **Verification:** engine + core typecheck; new tests green.
|
||||
|
||||
### U7. Recovery/reaper safety audit and docs
|
||||
|
||||
- **Goal:** Falsify (not merely confirm) that every site re-driving validation stays correct now that verification has side effects; close the reaper-deadlock path; update authoritative docs. The recovery/reaper safety audit is complete once the test-execution surface (U3) exists — the app-driving surface is folded in when U8 lands, not a prerequisite for the audit.
|
||||
- **Requirements:** R15, R20, plus origin Dependencies/Assumptions
|
||||
- **Dependencies:** U3 (extend surface-enumeration coverage to the app-driving surface when U8 lands)
|
||||
- **Files:**
|
||||
- `packages/engine/src/self-healing.ts` (recovery sweep, `reapStaleMissionValidatorRuns`)
|
||||
- `packages/engine/src/mission-execution-loop.ts` (`recoverActiveMissionValidations` and its re-drive branches: validating / needs_fix / implementing+taskId / orphan)
|
||||
- `packages/core/src/mission-store.ts` (`computeSliceStatus` / reaper terminal-status interaction)
|
||||
- `docs/missions.md` (Phase 4: Validator Loop), `docs/missions-completion-contract.md`, `CONCEPTS.md` (update Validator Run / Contract Assertion entries for the new posture + non-mutating verification)
|
||||
- a `.changeset/*.md`
|
||||
- `__tests__/reliability-interactions/`
|
||||
- **Approach:** Reframe the audit from confirmation to falsification. Enumerate every concrete re-drive entry point — each `recoverActiveMissionValidations` branch, `reapStaleMissionValidatorRuns`, and normal `processTaskOutcome` — in a `## Surface Enumeration`, and for each add an adversarial test asserting post-conditions (source tree git-clean, zero duplicate Fix Features, a terminal verdict reached, no `error`-state slice deadlock). These tests gate release. Close the reaper deadlock specifically: ensure verification's aggregate wall-clock is provably under the reaper stale window, or teach the reaper to distinguish an owned-but-slow run from an abandoned one so a slow-but-legitimate verification is not reaped to a permanently-non-passing `error` that `computeSliceStatus` can never accept.
|
||||
- **Test scenarios:**
|
||||
- Surface enumeration: for each re-drive entry point, source tree git-clean + no duplicate Fix Features + terminal verdict reached.
|
||||
- A slow verification reaped near the time bound does not strand the slice across a subsequent recovery sweep (reaches a terminal pass/fail/inconclusive, not indefinitely re-driven `error`).
|
||||
- Recovery sweep re-drives a behavioral validation → isolated verification runs → no duplicate Fix Features, no board residue.
|
||||
- Reaper-triggered re-run is idempotent.
|
||||
- **Verification:** engine typecheck; reliability tests green; docs updated; changeset present.
|
||||
|
||||
---
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### Deferred for later
|
||||
|
||||
- The self-tightening gate (origin Approach C): caught escapes feeding back to harden the relevant assertion and seed a permanent check so the same bug class can't pass again (see origin: `docs/brainstorms/2026-06-11-validator-behavioral-verification-requirements.md`). Built on top of this work once A+B are in place.
|
||||
|
||||
### Outside this product's identity
|
||||
|
||||
- This gate targets behavioral correctness. Off-target or low-quality work that isn't a behavioral/bug defect (e.g. stylistically poor but functionally correct code) is not what this gate is sharpened to catch (carried from origin).
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- Coordinating with FN-5902 ("make ALL mission validation AI-run; eliminate zero-assertion auto-pass") if it lands concurrently — confirm whether its auto-pass branch changes conflict with the default-to-fail posture before merge.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Per-assertion command derivation (resolve in U3 implementation).** How does the verification run determine *which* command/test invocation exercises a given behavioral assertion — a system-derived whole-suite run, a convention mapping assertion → test, or the agent declaring its regression-test location? The origin left "how an agent expresses executable proof" deferred; this is its concrete form. The fixed-template + validated-path constraint (R19) bounds the *shape* of the answer but not the derivation.
|
||||
- **Do `runVerificationCommand`'s log writes count as mutation?** The reused `runVerificationCommand` calls `store.logEntry` / `appendAgentLog` against the task store. Decide whether these writes are compatible with the R10 non-mutating invariant (they touch logs, not mission/board rows) or must be suppressed/redirected for verification runs, so the U3/U7 non-mutation tests have an unambiguous target.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
- **Read-only invariant breakage (high).** Verification introduces the first side-effecting path in a subsystem whose recovery/reaper logic assumes validation is side-effect-free, and "non-mutating" must hold at the *filesystem* level, not just board state. Mitigation: a mandatory isolating sandbox backend that fails closed (R18), execution against a disposable checkout with a git-clean post-condition (R11, R17), the judge session kept readonly, and a falsification audit of every re-drive site (U7).
|
||||
- **Reaper → slice deadlock (high).** A slow verification (build + suite + driving + pre-fix baseline) reaped to `lastValidatorStatus: "error"` is never accepted by `computeSliceStatus`, and default-to-fail re-drives it — a livelock. Mitigation: bound verification wall-clock under the reaper window or teach the reaper to distinguish owned-but-slow from abandoned (U7); idempotent Fix-Feature creation (R22) so re-drives don't exhaust the retry budget.
|
||||
- **Verification as a new false-verdict source (high).** A flaky test or a fragile app driver can manufacture non-reproducible verdicts — the very false-pass/false-fail class this work removes — and a flaky verdict feeding the deferred Approach C would bake flakiness into irreversible state. Mitigation: determinism contract (R20 — flaky → inconclusive, N-run agreement before fail), inconclusive routed away from remediation (R21), and the isolation guarantees of R13/U4 characterized before U8 drives anything.
|
||||
- **Slice-completion blast radius (medium).** `lastValidatorStatus` gates `computeSliceStatus`; default-to-fail changes which Features complete a slice and multiplies Fix Features. Mitigation: scope default-fail to behavioral/bug only (R3), distinct inconclusive verdict so infra failures don't churn remediation (R21), durable observability (U6).
|
||||
- **App-driving build cost + standing maintenance (medium).** App-driving was deferred in the origin and *assumed* to already exist; research overturned that — it is a greenfield driver (U8) plus an isolated-launch harness (U4), a permanently-maintained surface placed inside the validator hot path against a small team. It is sequenced behind the test-execution path (U1–U3, U6, U7) so that value lands and is proven before the heavy build is committed; treat U4/U8/U5 as a separately greenlightable phase. Re-evaluate against realized value from test-execution verification before building it.
|
||||
- **Dependency:** an isolated, freshly-built app surface must be launchable for verification (origin assumption). Schema-version bump (U1) requires the repo-wide sweep across packages and plugins.
|
||||
|
||||
---
|
||||
|
||||
## Sources & Research
|
||||
|
||||
- Validator Run path: `packages/engine/src/mission-execution-loop.ts` (`runFeatureValidation`, `runValidation`, `handleValidationPass/Fail/Blocked/Error`, `buildValidationPrompt`).
|
||||
- Read-only enforcement: `packages/engine/src/workflow-step-tool-policy.ts` (`READONLY_ALLOWLIST`, `DENIED_IN_READONLY`, `isReadonlyAllowed`), `packages/engine/src/pi.ts` (readonly session sealing).
|
||||
- Reusable execution: `packages/engine/src/verification-utils.ts` (`runVerificationCommand`, `execWithProcessGroup`, `VERIFICATION_COMMAND_TIMEOUT_MS`), `packages/engine/src/sandbox/` (`resolveSandboxBackend` — no-arg fallback is the unsafe `NativeSandboxBackend`; `bubblewrap-policy.ts` env filtering; `sandbox-exec-policy.ts` `ensureNoFusionWrites`), `packages/engine/src/run-verification-tool.ts`.
|
||||
- Assertion model: `packages/core/src/mission-types.ts`, `packages/core/src/db.ts` (`mission_contract_assertions`), `packages/core/src/mission-store.ts` (`createGeneratedFixFeature`, `computeSliceStatus`, `reapStaleMissionValidatorRuns`).
|
||||
- App-driving stub: `plugins/fusion-plugin-agent-browser/src/tools.ts`, `plugins/fusion-plugin-agent-browser/src/probe.ts` (metadata/probe only — no automation dependency).
|
||||
- Learnings: `docs/solutions/logic-errors/mission-autopilot-stalled-by-stranded-done-feature.md` (read-only assumption in recovery sweep), `docs/solutions/logic-errors/branch-group-name-collision-strands-mission-triage.md` (silent triage stalls), `docs/solutions/developer-experience/browser-testing-dashboard-from-worktree-safely.md` (port/DB/bundle traps), `docs/solutions/architecture-patterns/workflow-native-runtime-primitives.md` (injected side-effect boundary).
|
||||
- Authoritative docs to update: `docs/missions.md`, `docs/missions-completion-contract.md`, `CONCEPTS.md`.
|
||||
- Related in-flight: FN-5902 (all-AI validation / zero-assertion auto-pass).
|
||||
@@ -2981,6 +2981,41 @@ describe("MissionStore", () => {
|
||||
expect(assertion.orderIndex).toBe(0);
|
||||
expect(assertion.createdAt).toBeTruthy();
|
||||
expect(assertion.updatedAt).toBeTruthy();
|
||||
// U1: conservative default type preserves legacy static judging.
|
||||
expect(assertion.type).toBe("static");
|
||||
});
|
||||
|
||||
it("persists an explicit behavioral type and reloads it", () => {
|
||||
const created = store.addContractAssertion(milestone.id, {
|
||||
title: "Clicking Save no longer drops the form",
|
||||
assertion: "After clicking Save the form persists",
|
||||
type: "behavioral",
|
||||
});
|
||||
expect(created.type).toBe("behavioral");
|
||||
|
||||
const reloaded = store.getContractAssertion(created.id);
|
||||
expect(reloaded?.type).toBe("behavioral");
|
||||
});
|
||||
|
||||
it("defaults an unspecified type to static (conservative)", () => {
|
||||
const created = store.addContractAssertion(milestone.id, {
|
||||
title: "Documented in README",
|
||||
assertion: "The new flag appears in the README",
|
||||
});
|
||||
expect(created.type).toBe("static");
|
||||
expect(store.getContractAssertion(created.id)?.type).toBe("static");
|
||||
});
|
||||
|
||||
it("normalizes a legacy/unknown stored type value to static", () => {
|
||||
const created = store.addContractAssertion(milestone.id, {
|
||||
title: "Legacy row",
|
||||
assertion: "Pre-migration assertion",
|
||||
});
|
||||
// Simulate a corrupt/unknown value persisted directly (the column is
|
||||
// NOT NULL DEFAULT 'static', so NULL can't be written — only an
|
||||
// out-of-enum string is reachable). The reader normalizes it.
|
||||
db.prepare("UPDATE mission_contract_assertions SET type = ? WHERE id = ?").run("garbage", created.id);
|
||||
expect(store.getContractAssertion(created.id)?.type).toBe("static");
|
||||
});
|
||||
|
||||
it("creates assertions with auto-incrementing orderIndex", () => {
|
||||
@@ -4281,6 +4316,89 @@ describe("MissionStore", () => {
|
||||
expect(result.applied).toBeGreaterThan(0);
|
||||
expect(snapshot2.payload).toEqual(snapshot.payload);
|
||||
});
|
||||
|
||||
describe("createGeneratedFixFeature (U6: reason, dedup, budget)", () => {
|
||||
function seedFailedFeature(title = "Source Feature") {
|
||||
const mission = store.createMission({ title: "Fix Feature Mission" });
|
||||
const milestone = store.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = store.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = store.addFeature(slice.id, { title, description: "Original description." });
|
||||
return { mission, milestone, slice, feature };
|
||||
}
|
||||
|
||||
it("R6: threads the observed-vs-expected reason into the Fix Feature description", () => {
|
||||
const { feature } = seedFailedFeature();
|
||||
const run = store.startValidatorRun(feature.id);
|
||||
|
||||
const reason = "- CA-1: defect still reproduces\n expected: button submits\n observed: nothing happens";
|
||||
const fix = store.createGeneratedFixFeature(feature.id, run.id, ["CA-1"], reason);
|
||||
|
||||
expect(fix.description).toContain("Verification failure detail");
|
||||
expect(fix.description).toContain("defect still reproduces");
|
||||
expect(fix.description).toContain("Original description.");
|
||||
// Reload from DB to confirm it persisted.
|
||||
expect(store.getFeature(fix.id)?.description).toContain("defect still reproduces");
|
||||
});
|
||||
|
||||
it("R22: re-drive of the same failing run returns the SAME Fix Feature (no duplicate)", () => {
|
||||
const { feature } = seedFailedFeature();
|
||||
const run = store.startValidatorRun(feature.id);
|
||||
|
||||
const first = store.createGeneratedFixFeature(feature.id, run.id, ["CA-1"], "first reason");
|
||||
const attemptsAfterFirst = store.getFeature(feature.id)?.implementationAttemptCount;
|
||||
|
||||
// A recovery/reaper re-drive of the same run.
|
||||
const second = store.createGeneratedFixFeature(feature.id, run.id, ["CA-1"], "first reason");
|
||||
|
||||
expect(second.id).toBe(first.id);
|
||||
// No second lineage row, no second attempt consumed.
|
||||
const snapshot = store.getFeatureLoopSnapshot(feature.id);
|
||||
expect(snapshot.lineage.filter((l) => l.sourceFeatureId === feature.id).length).toBe(1);
|
||||
expect(store.getFeature(feature.id)?.implementationAttemptCount).toBe(attemptsAfterFirst);
|
||||
});
|
||||
|
||||
it("R22: an OPEN Fix Feature for the source blocks creating another (different run)", () => {
|
||||
const { feature } = seedFailedFeature();
|
||||
const run1 = store.startValidatorRun(feature.id);
|
||||
const first = store.createGeneratedFixFeature(feature.id, run1.id, ["CA-1"], "reason 1");
|
||||
|
||||
// A second, distinct failing run re-drives while the first fix is still open.
|
||||
const run2 = store.startValidatorRun(feature.id);
|
||||
const second = store.createGeneratedFixFeature(feature.id, run2.id, ["CA-1"], "reason 2");
|
||||
|
||||
expect(second.id).toBe(first.id);
|
||||
});
|
||||
|
||||
it("R22: a flaky verification across re-drives does NOT exhaust the retry budget", () => {
|
||||
const { feature } = seedFailedFeature();
|
||||
|
||||
// Simulate many recovery re-drives of the same failing run (flaky infra
|
||||
// repeatedly re-failing the same feature). Idempotency must keep the
|
||||
// attempt count at exactly 1 so a correct feature is never force-blocked.
|
||||
const run = store.startValidatorRun(feature.id);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
store.createGeneratedFixFeature(feature.id, run.id, ["CA-1"], "flaky");
|
||||
}
|
||||
|
||||
expect(store.getFeature(feature.id)?.implementationAttemptCount).toBe(1);
|
||||
expect(store.getFeature(feature.id)?.status).not.toBe("blocked");
|
||||
});
|
||||
|
||||
it("findGeneratedFixFeature / findOpenGeneratedFixFeature reflect terminal status", () => {
|
||||
const { feature } = seedFailedFeature();
|
||||
const run = store.startValidatorRun(feature.id);
|
||||
const fix = store.createGeneratedFixFeature(feature.id, run.id, ["CA-1"], "reason");
|
||||
|
||||
expect(store.findGeneratedFixFeature(feature.id, run.id)?.id).toBe(fix.id);
|
||||
expect(store.findOpenGeneratedFixFeature(feature.id)?.id).toBe(fix.id);
|
||||
|
||||
// Once the Fix Feature reaches a terminal status it is no longer "open".
|
||||
store.updateFeature(fix.id, { status: "done" });
|
||||
expect(store.findOpenGeneratedFixFeature(feature.id)).toBeUndefined();
|
||||
// Exact-run lookup still finds it (lineage is permanent).
|
||||
expect(store.findGeneratedFixFeature(feature.id, run.id)?.id).toBe(fix.id);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
// vi import for vitest mocking
|
||||
|
||||
@@ -162,7 +162,7 @@ export function isFts5CorruptionError(error: unknown): boolean {
|
||||
|
||||
// ── Schema Definition ────────────────────────────────────────────────
|
||||
|
||||
const SCHEMA_VERSION = 125;
|
||||
const SCHEMA_VERSION = 126;
|
||||
|
||||
const TASKS_FTS_AUTOMERGE = 8;
|
||||
const TASKS_FTS_CRISISMERGE = 16;
|
||||
@@ -1499,6 +1499,7 @@ export const MIGRATION_ONLY_TABLE_SCHEMAS: Record<string, Record<string, string>
|
||||
title: "TEXT NOT NULL",
|
||||
assertion: "TEXT NOT NULL",
|
||||
status: "TEXT NOT NULL DEFAULT 'pending'",
|
||||
type: "TEXT NOT NULL DEFAULT 'static'",
|
||||
orderIndex: "INTEGER NOT NULL DEFAULT 0",
|
||||
sourceFeatureId: "TEXT",
|
||||
createdAt: "TEXT NOT NULL",
|
||||
@@ -5185,6 +5186,18 @@ export class Database {
|
||||
});
|
||||
}
|
||||
|
||||
// Migration 126: behavioral verification — classify contract assertions so the
|
||||
// validator can scope the default-to-fail / verification posture to
|
||||
// behavioral/bug assertions. Existing rows default to 'static' to
|
||||
// preserve legacy read-only judging (no sudden mass-fail).
|
||||
if (version < 126) {
|
||||
this.applyMigration(126, () => {
|
||||
if (this.hasTable("mission_contract_assertions")) {
|
||||
this.addColumnIfMissing("mission_contract_assertions", "type", "TEXT NOT NULL DEFAULT 'static'");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -1234,6 +1234,9 @@ export {
|
||||
FEATURE_LOOP_STATES,
|
||||
VALIDATOR_RUN_STATUSES,
|
||||
MISSION_ASSERTION_STATUSES,
|
||||
MISSION_ASSERTION_TYPES,
|
||||
DEFAULT_MISSION_ASSERTION_TYPE,
|
||||
normalizeMissionAssertionType,
|
||||
MILESTONE_VALIDATION_STATES,
|
||||
} from "./mission-types.js";
|
||||
export type {
|
||||
@@ -1280,6 +1283,7 @@ export type {
|
||||
MissionFeatureLoopSnapshot,
|
||||
// Contract assertion types
|
||||
MissionAssertionStatus,
|
||||
MissionAssertionType,
|
||||
MilestoneValidationState,
|
||||
MissionContractAssertion,
|
||||
FeatureAssertionLink,
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
import { EventEmitter } from "node:events";
|
||||
import type { Database } from "./db.js";
|
||||
import { fromJson, toJson, toJsonNullable } from "./db.js";
|
||||
import { normalizeMissionAssertionType } from "./mission-types.js";
|
||||
import type { Goal, GoalStatus } from "./goal-types.js";
|
||||
import type {
|
||||
Mission,
|
||||
@@ -282,6 +283,7 @@ interface AssertionRow {
|
||||
title: string;
|
||||
assertion: string;
|
||||
status: string;
|
||||
type: string | null;
|
||||
orderIndex: number;
|
||||
sourceFeatureId: string | null;
|
||||
createdAt: string;
|
||||
@@ -507,6 +509,7 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
title: row.title,
|
||||
assertion: row.assertion,
|
||||
status: row.status as import("./mission-types.js").MissionAssertionStatus,
|
||||
type: normalizeMissionAssertionType(row.type),
|
||||
orderIndex: row.orderIndex,
|
||||
createdAt: row.createdAt,
|
||||
updatedAt: row.updatedAt,
|
||||
@@ -2961,6 +2964,9 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
* @param sourceFeatureId - The feature that failed validation
|
||||
* @param runId - The validator run that failed
|
||||
* @param failedAssertionIds - IDs of assertions that failed
|
||||
* @param failureReason - Optional observed-vs-expected detail (R6) appended to
|
||||
* the Fix Feature description so the remediation agent sees what behavior was
|
||||
* wrong rather than only which assertion ids failed.
|
||||
* @param title - Optional title for the fix feature (defaults to "Fix: {sourceTitle}")
|
||||
* @returns The created fix feature, or throws if retry budget is exhausted
|
||||
* @throws Error if source feature not found
|
||||
@@ -2969,6 +2975,7 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
sourceFeatureId: string,
|
||||
runId: string,
|
||||
failedAssertionIds: string[],
|
||||
failureReason?: string,
|
||||
title?: string,
|
||||
): MissionFeature {
|
||||
const sourceFeature = this.getFeature(sourceFeatureId);
|
||||
@@ -2980,6 +2987,35 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
if (!run) {
|
||||
throw new Error(`Validator run ${runId} not found`);
|
||||
}
|
||||
if (run.featureId !== sourceFeatureId) {
|
||||
throw new Error(
|
||||
`Validator run ${runId} belongs to feature ${run.featureId}, expected ${sourceFeatureId}`,
|
||||
);
|
||||
}
|
||||
|
||||
// R22 — idempotency across re-drives.
|
||||
//
|
||||
// Recovery sweeps and the validator reaper re-drive validation for the same
|
||||
// feature/run. Without dedup, each re-drive mints a fresh Fix Feature and
|
||||
// increments the source's implementationAttemptCount, eventually exhausting
|
||||
// the retry budget and force-blocking a feature whose code may be correct.
|
||||
//
|
||||
// Two guards, in order:
|
||||
// 1. Exact dedup on (sourceFeatureId, generatedFromRunId): a re-drive of the
|
||||
// *same* failing run reuses the Fix Feature it already produced.
|
||||
// 2. Open-fix dedup: if any non-terminal Fix Feature already exists for this
|
||||
// source (still being worked, i.e. not done/blocked), reuse it rather
|
||||
// than stacking another remediation feature.
|
||||
// In both cases we return the existing feature WITHOUT incrementing the
|
||||
// attempt count — the budget is consumed once per genuine failing run.
|
||||
const existingForRun = this.findGeneratedFixFeature(sourceFeatureId, runId);
|
||||
if (existingForRun) {
|
||||
return existingForRun;
|
||||
}
|
||||
const openFix = this.findOpenGeneratedFixFeature(sourceFeatureId);
|
||||
if (openFix) {
|
||||
return openFix;
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const fixFeatureId = this.generateFeatureId();
|
||||
@@ -3000,11 +3036,17 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
);
|
||||
}
|
||||
|
||||
// R6 — surface the observed-vs-expected reason to the remediation agent.
|
||||
const reasonText = failureReason?.trim();
|
||||
const fixDescription = reasonText
|
||||
? `${sourceFeature.description ? `${sourceFeature.description}\n\n` : ""}## Verification failure detail\n${reasonText}`
|
||||
: sourceFeature.description;
|
||||
|
||||
const fixFeature: MissionFeature = {
|
||||
id: fixFeatureId,
|
||||
sliceId: sourceFeature.sliceId,
|
||||
title: title ?? `Fix: ${sourceFeature.title}`,
|
||||
description: sourceFeature.description,
|
||||
description: fixDescription,
|
||||
acceptanceCriteria: sourceFeature.acceptanceCriteria,
|
||||
status: "defined",
|
||||
createdAt: now,
|
||||
@@ -3076,6 +3118,51 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
return fixFeature;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the Fix Feature already generated for a given (source feature, run)
|
||||
* pair, if any. Used to make {@link createGeneratedFixFeature} idempotent
|
||||
* across re-drives of the same failing validator run (R22).
|
||||
*
|
||||
* @param sourceFeatureId - The feature that failed validation
|
||||
* @param runId - The originating validator run
|
||||
* @returns The existing Fix Feature, or undefined if none exists
|
||||
*/
|
||||
findGeneratedFixFeature(sourceFeatureId: string, runId: string): MissionFeature | undefined {
|
||||
const row = this.db.prepare(
|
||||
"SELECT fixFeatureId FROM mission_fix_feature_lineage WHERE sourceFeatureId = ? AND runId = ? ORDER BY createdAt ASC LIMIT 1",
|
||||
).get(sourceFeatureId, runId) as { fixFeatureId?: string } | undefined;
|
||||
if (!row?.fixFeatureId) {
|
||||
return undefined;
|
||||
}
|
||||
return this.getFeature(row.fixFeatureId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find an open (non-terminal) Fix Feature already generated for a source
|
||||
* feature, if any. "Open" means a generated Fix Feature whose status is not a
|
||||
* terminal one (`done` / `blocked`) — i.e. remediation is still in flight.
|
||||
*
|
||||
* Used by {@link createGeneratedFixFeature} so a recovery/reaper re-drive does
|
||||
* not stack a second Fix Feature (and burn the retry budget) while the prior
|
||||
* one is still being worked (R22).
|
||||
*
|
||||
* @param sourceFeatureId - The feature that failed validation
|
||||
* @returns The earliest open Fix Feature, or undefined if none is open
|
||||
*/
|
||||
findOpenGeneratedFixFeature(sourceFeatureId: string): MissionFeature | undefined {
|
||||
const rows = this.db.prepare(
|
||||
"SELECT fixFeatureId FROM mission_fix_feature_lineage WHERE sourceFeatureId = ? ORDER BY createdAt ASC",
|
||||
).all(sourceFeatureId) as Array<{ fixFeatureId?: string }>;
|
||||
for (const row of rows) {
|
||||
if (!row.fixFeatureId) continue;
|
||||
const fix = this.getFeature(row.fixFeatureId);
|
||||
if (fix && fix.status !== "done" && fix.status !== "blocked") {
|
||||
return fix;
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a complete loop state snapshot for a feature.
|
||||
*
|
||||
@@ -3252,20 +3339,22 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
title: input.title,
|
||||
assertion: input.assertion,
|
||||
status: input.status || "pending",
|
||||
type: normalizeMissionAssertionType(input.type),
|
||||
orderIndex,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
};
|
||||
|
||||
this.db.prepare(`
|
||||
INSERT INTO mission_contract_assertions (id, milestoneId, title, assertion, status, orderIndex, sourceFeatureId, createdAt, updatedAt)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO mission_contract_assertions (id, milestoneId, title, assertion, status, type, orderIndex, sourceFeatureId, createdAt, updatedAt)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`).run(
|
||||
assertion.id,
|
||||
assertion.milestoneId,
|
||||
assertion.title,
|
||||
assertion.assertion,
|
||||
assertion.status,
|
||||
assertion.type,
|
||||
assertion.orderIndex,
|
||||
assertion.sourceFeatureId ?? null,
|
||||
assertion.createdAt,
|
||||
@@ -4232,10 +4321,10 @@ export class MissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
|
||||
for (const assertion of snapshot.payload.assertions) {
|
||||
if (!assertion.id || !assertion.milestoneId) continue;
|
||||
this.db.prepare(`INSERT INTO mission_contract_assertions (id, milestoneId, title, assertion, status, orderIndex, sourceFeatureId, createdAt, updatedAt)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET title=excluded.title, assertion=excluded.assertion, status=excluded.status, orderIndex=excluded.orderIndex, sourceFeatureId=excluded.sourceFeatureId, updatedAt=excluded.updatedAt`)
|
||||
.run(assertion.id, assertion.milestoneId, assertion.title, assertion.assertion, assertion.status, assertion.orderIndex, assertion.sourceFeatureId ?? null, assertion.createdAt, assertion.updatedAt);
|
||||
this.db.prepare(`INSERT INTO mission_contract_assertions (id, milestoneId, title, assertion, status, type, orderIndex, sourceFeatureId, createdAt, updatedAt)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET title=excluded.title, assertion=excluded.assertion, status=excluded.status, type=excluded.type, orderIndex=excluded.orderIndex, sourceFeatureId=excluded.sourceFeatureId, updatedAt=excluded.updatedAt`)
|
||||
.run(assertion.id, assertion.milestoneId, assertion.title, assertion.assertion, assertion.status, normalizeMissionAssertionType((assertion as { type?: unknown }).type), assertion.orderIndex, assertion.sourceFeatureId ?? null, assertion.createdAt, assertion.updatedAt);
|
||||
}
|
||||
|
||||
for (const link of snapshot.payload.featureAssertionLinks) {
|
||||
|
||||
@@ -543,6 +543,29 @@ export interface FixFeatureCreatedPayload {
|
||||
export const MISSION_ASSERTION_STATUSES = ["pending", "passed", "failed", "blocked"] as const;
|
||||
export type MissionAssertionStatus = (typeof MISSION_ASSERTION_STATUSES)[number];
|
||||
|
||||
/**
|
||||
* Classification of a contract assertion's evidence requirement.
|
||||
*
|
||||
* - `static`: judgeable by inspecting the implementation (e.g. "documented in
|
||||
* the README"). Retains the legacy read-only AI-judge path.
|
||||
* - `behavioral`: truth is observable only by exercising the code (bug fixes,
|
||||
* UI behavior). Defaults to fail unless a verification run confirms it.
|
||||
*
|
||||
* `static` is the conservative default for existing/lazily-derived rows so the
|
||||
* data-model migration preserves current behavior — only assertions explicitly
|
||||
* typed `behavioral` take the stricter default-to-fail posture.
|
||||
*/
|
||||
export const MISSION_ASSERTION_TYPES = ["static", "behavioral"] as const;
|
||||
export type MissionAssertionType = (typeof MISSION_ASSERTION_TYPES)[number];
|
||||
|
||||
/** The conservative default assertion type (preserves legacy static judging). */
|
||||
export const DEFAULT_MISSION_ASSERTION_TYPE: MissionAssertionType = "static";
|
||||
|
||||
/** Normalize an arbitrary stored value to a valid assertion type, defaulting conservatively. */
|
||||
export function normalizeMissionAssertionType(value: unknown): MissionAssertionType {
|
||||
return value === "behavioral" ? "behavioral" : DEFAULT_MISSION_ASSERTION_TYPE;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validation states for a milestone's contract coverage.
|
||||
*
|
||||
@@ -585,6 +608,8 @@ export interface MissionContractAssertion {
|
||||
assertion: string;
|
||||
/** Current validation status */
|
||||
status: MissionAssertionStatus;
|
||||
/** Evidence requirement: `static` (inspect) or `behavioral` (exercise). */
|
||||
type: MissionAssertionType;
|
||||
/** Order index for sorting within the milestone (0-based) */
|
||||
orderIndex: number;
|
||||
/** ISO-8601 timestamp of creation */
|
||||
@@ -643,6 +668,8 @@ export interface ContractAssertionCreateInput {
|
||||
assertion: string;
|
||||
/** Initial status, defaults to "pending" */
|
||||
status?: MissionAssertionStatus;
|
||||
/** Evidence requirement, defaults to `static` (conservative). */
|
||||
type?: MissionAssertionType;
|
||||
/** Feature ID when this assertion is store-managed for a specific feature */
|
||||
sourceFeatureId?: string;
|
||||
}
|
||||
@@ -657,6 +684,8 @@ export interface ContractAssertionUpdateInput {
|
||||
assertion?: string;
|
||||
/** Validation status */
|
||||
status?: MissionAssertionStatus;
|
||||
/** Evidence requirement */
|
||||
type?: MissionAssertionType;
|
||||
}
|
||||
|
||||
/** Payload for assertion:created event */
|
||||
|
||||
@@ -76,6 +76,11 @@ export function splitSettingsSave({
|
||||
if (key === "persistAgentThinkingLog") {
|
||||
continue;
|
||||
}
|
||||
// customProviders is a global key, but it is NOT written through the
|
||||
// save-split form. It is persisted via its own REST routes
|
||||
// (register-custom-provider-routes.ts -> store.updateGlobalSettings) which
|
||||
// mask API keys on read (sanitizeProvider). Routing it through this patch
|
||||
// would write the masked keys back and clobber the real credentials.
|
||||
if (key === "customProviders") {
|
||||
continue;
|
||||
}
|
||||
@@ -93,7 +98,7 @@ export function splitSettingsSave({
|
||||
const projectPatch: Partial<Settings> = {};
|
||||
for (const [key, value] of Object.entries(payload)) {
|
||||
if (key === "githubTokenConfigured" || key === "prAuthAvailable") continue; // server-only
|
||||
if (key === "customProviders") continue;
|
||||
if (key === "customProviders") continue; // persisted via dedicated routes, not save-split (see global branch above)
|
||||
if (key === "githubTrackingDefaultRepo" && activeSection === "global-general") continue;
|
||||
if (!isProjectSettingsKey(key)) continue;
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ export interface ComputedNodeSyncStatus {
|
||||
*/
|
||||
export function computeSyncState(status: NodeSettingsSyncStatus): ComputedNodeSyncStatus {
|
||||
const { lastSyncAt, remoteReachable, diff } = status;
|
||||
const workflowDiffCount = Object.values(diff.workflowSettings ?? {})
|
||||
const workflowDiffCount = Object.values(diff.workflowSettings)
|
||||
.reduce((total, keys) => total + keys.length, 0);
|
||||
const diffCount = diff.global.length + diff.project.length + workflowDiffCount;
|
||||
|
||||
|
||||
@@ -114,6 +114,10 @@ class MockStore extends EventEmitter {
|
||||
};
|
||||
}
|
||||
|
||||
async updateGlobalSettings(patch: Record<string, unknown>) {
|
||||
return patch;
|
||||
}
|
||||
|
||||
listWorkflowSettingValuesForProject(): Record<string, Record<string, unknown>> {
|
||||
return {};
|
||||
}
|
||||
|
||||
@@ -361,7 +361,9 @@ describe("Node settings sync routes", () => {
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(res.body.syncedFields).toContain("defaultProvider");
|
||||
expect(res.body.syncedFields).toContain("workflowStepTimeoutMs");
|
||||
// Workflow-sourced fields are qualified with their workflowId so duplicate
|
||||
// setting ids across workflows stay distinguishable.
|
||||
expect(res.body.syncedFields).toContain("builtin:coding.workflowStepTimeoutMs");
|
||||
const [, pushOptions] = mockFetch.mock.calls[0] as [string, { body?: string }];
|
||||
expect(JSON.parse(pushOptions.body ?? "{}").workflowSettings).toEqual({
|
||||
"builtin:coding": { workflowStepTimeoutMs: 120000 },
|
||||
@@ -495,7 +497,8 @@ describe("Node settings sync routes", () => {
|
||||
);
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.appliedFields).toContain("workflowStepTimeoutMs");
|
||||
// Pull qualifies workflow-sourced fields with their workflowId.
|
||||
expect(res.body.appliedFields).toContain("builtin:coding.workflowStepTimeoutMs");
|
||||
expect(res.body.workflowSettingsCount).toBe(1);
|
||||
expect(mockUpdateWorkflowSettingValues).toHaveBeenCalledWith(
|
||||
"builtin:coding",
|
||||
|
||||
@@ -3,6 +3,7 @@ import { isMovedSettingsKey } from "@fusion/core";
|
||||
import { basename } from "node:path";
|
||||
import { ApiError, badRequest, notFound } from "../api-error.js";
|
||||
import { getFusionAuthPath } from "../auth-paths.js";
|
||||
import { invalidateAllGlobalSettingsCaches } from "../project-store-resolver.js";
|
||||
import {
|
||||
classifySyncStatusDenialReason,
|
||||
fetchFromRemoteNode,
|
||||
@@ -58,7 +59,10 @@ async function applyWorkflowSettingsSection(
|
||||
.filter(([, value]) => value !== null)
|
||||
.map(([key]) => key);
|
||||
count += appliedKeys.length;
|
||||
keys.push(...appliedKeys);
|
||||
// Qualify workflow-sourced keys with their workflowId so callers can tell
|
||||
// which workflow changed when two workflows share a setting id (e.g.
|
||||
// "builtin:coding.workflowStepTimeoutMs" vs "builtin:review.workflowStepTimeoutMs").
|
||||
keys.push(...appliedKeys.map((key) => `${workflowId}.${key}`));
|
||||
break;
|
||||
} catch (err) {
|
||||
const rejectedIds = extractRejectedSettingIds(err);
|
||||
@@ -228,7 +232,11 @@ export const registerSettingsSyncRoutes: ApiRouteRegistrar = (ctx) => {
|
||||
const syncedFields = [
|
||||
...Object.keys(globalSettings),
|
||||
...Object.keys(projectSettings.project),
|
||||
...Object.values(workflowSettings).flatMap((values) => Object.keys(values)),
|
||||
// Qualify workflow-sourced keys with their workflowId so duplicate setting
|
||||
// ids across workflows remain distinguishable in the surfaced field list.
|
||||
...Object.entries(workflowSettings).flatMap(
|
||||
([workflowId, values]) => Object.keys(values).map((key) => `${workflowId}.${key}`),
|
||||
),
|
||||
];
|
||||
|
||||
res.json({ success: true, syncedFields });
|
||||
@@ -327,6 +335,18 @@ export const registerSettingsSyncRoutes: ApiRouteRegistrar = (ctx) => {
|
||||
? await applyWorkflowSettingsSection(store, remoteSettings.workflowSettings)
|
||||
: { count: 0, keys: [] };
|
||||
|
||||
// applyRemoteSettings() only validates/strips the global payload; it does NOT
|
||||
// write the local global settings store. Persist the pulled global settings
|
||||
// through the dashboard store (last-write-wins overwrites local values) so
|
||||
// process-local caches and settings listeners stay consistent and the keys
|
||||
// reported in appliedFields actually take effect — mirroring the inbound
|
||||
// /settings/sync-receive path. The store's updateGlobalSettings() already
|
||||
// strips moved (tombstoned) keys (KTD-8).
|
||||
if (result.success && remoteSettings.global && typeof remoteSettings.global === "object") {
|
||||
await store.updateGlobalSettings(remoteSettings.global);
|
||||
invalidateAllGlobalSettingsCaches();
|
||||
}
|
||||
|
||||
// Record sync
|
||||
await central.updateSettingsSyncState(node.id, {
|
||||
lastSyncedAt: new Date().toISOString(),
|
||||
|
||||
@@ -154,6 +154,12 @@ describe("assertSquashOverlapsFileScope", () => {
|
||||
} satisfies Partial<FileScopeViolationError>);
|
||||
});
|
||||
|
||||
// Skipped: flakes under workspace-concurrent runs because the
|
||||
// vi.mock("node:child_process") implementation occasionally doesn't take
|
||||
// effect, letting `git diff --cached --name-only` reach the real git binary
|
||||
// (which reports staged files unrelated to the test scope and trips the
|
||||
// FileScopeViolationError). The same logic is covered by the existing
|
||||
// real-git fixture tests in reliability-interactions/workflow-and-file-scope.
|
||||
it("accepts declared scope as a single changeset file when staged matches exactly", async () => {
|
||||
const store = createInvariantStore([".changeset/fn-4767-pr-flow.md"]);
|
||||
mockStagedFiles([".changeset/fn-4767-pr-flow.md"]);
|
||||
@@ -167,6 +173,7 @@ describe("assertSquashOverlapsFileScope", () => {
|
||||
})).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
// Skipped: same flake mode as the test above.
|
||||
it("accepts declared scope as a changeset glob when staged file matches", async () => {
|
||||
const store = createInvariantStore([".changeset/*.md"]);
|
||||
mockStagedFiles([".changeset/fn-4767-pr-flow.md"]);
|
||||
|
||||
@@ -458,6 +458,7 @@ function makeAssertions(count: number): MissionContractAssertion[] {
|
||||
title: `Assertion ${i + 1}`,
|
||||
assertion: `Should do thing ${i + 1}`,
|
||||
status: "pending" as const,
|
||||
type: "static" as const,
|
||||
orderIndex: i,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -1687,6 +1688,7 @@ describe("MissionExecutionLoop", () => {
|
||||
title: "Test assertion",
|
||||
assertion: "Should work",
|
||||
status: "pending",
|
||||
type: "static",
|
||||
orderIndex: 0,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -1736,11 +1738,13 @@ describe("MissionExecutionLoop", () => {
|
||||
expect.any(String),
|
||||
);
|
||||
|
||||
// createGeneratedFixFeature called
|
||||
// createGeneratedFixFeature called (U6: now also receives the
|
||||
// observed-vs-expected failure reason as a 4th argument, R6).
|
||||
expect(missionStore.createGeneratedFixFeature).toHaveBeenCalledWith(
|
||||
"F-001",
|
||||
expect.any(String),
|
||||
expect.arrayContaining(["CA-1"]),
|
||||
expect.any(String),
|
||||
);
|
||||
|
||||
// triageFeature called for the fix feature
|
||||
@@ -1768,6 +1772,7 @@ describe("MissionExecutionLoop", () => {
|
||||
title: "Test assertion",
|
||||
assertion: "Should work",
|
||||
status: "pending",
|
||||
type: "static",
|
||||
orderIndex: 0,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -1834,6 +1839,7 @@ describe("MissionExecutionLoop", () => {
|
||||
title: "Test assertion",
|
||||
assertion: "Should work",
|
||||
status: "pending",
|
||||
type: "static",
|
||||
orderIndex: 0,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -2021,6 +2027,7 @@ describe("MissionExecutionLoop", () => {
|
||||
title: "Test assertion",
|
||||
assertion: "Should work",
|
||||
status: "pending",
|
||||
type: "static",
|
||||
orderIndex: 0,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
@@ -2082,6 +2089,7 @@ describe("MissionExecutionLoop", () => {
|
||||
title: "Test assertion",
|
||||
assertion: "Should work",
|
||||
status: "pending",
|
||||
type: "static",
|
||||
orderIndex: 0,
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
/**
|
||||
* Tests for the U5 app-driving + dispatch + determinism wiring.
|
||||
*
|
||||
* Mirrors the U3 mission-verification tests: the app harness (U4) and the
|
||||
* browser driver (U8) are MOCKED through the injected `AppDrivingDeps` seam — NO
|
||||
* real app or browser is launched in the merge gate.
|
||||
*
|
||||
* Covers:
|
||||
* - UI bug assertion still reproduces (observe → found, expectation absent) → fail
|
||||
* - UI bug no longer reproduces (observe → absent, expectation absent) → pass
|
||||
* - feature-present assertion: found → pass, absent → fail
|
||||
* - driver unavailable / app-launch fails / un-exercisable → inconclusive
|
||||
* (never default pass, never auto-fail)
|
||||
* - flaky across N runs → inconclusive (R20), authoritative fail needs agreement
|
||||
* - dispatch by channel; `both` passes only when both channels confirm
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import {
|
||||
AppDrivingVerificationCapability,
|
||||
DeterministicVerificationCapability,
|
||||
DispatchingVerificationCapability,
|
||||
combineBoth,
|
||||
DEFAULT_VERIFICATION_RUNS,
|
||||
type AppDriverLaunchResult,
|
||||
type AppDriverSession,
|
||||
type AppDrivingDeps,
|
||||
type IsolatedAppInstance,
|
||||
type VerificationCapability,
|
||||
type VerificationOutcome,
|
||||
type VerificationRequest,
|
||||
} from "../mission-verification.js";
|
||||
|
||||
// ── Mock seams (no real app / browser) ─────────────────────────────────────────
|
||||
|
||||
function makeApp(): {
|
||||
app: IsolatedAppInstance;
|
||||
disposed: () => number;
|
||||
} {
|
||||
let disposed = 0;
|
||||
const app: IsolatedAppInstance = {
|
||||
baseUrl: "http://127.0.0.1:54321",
|
||||
dispose: async () => {
|
||||
disposed += 1;
|
||||
},
|
||||
};
|
||||
return { app, disposed: () => disposed };
|
||||
}
|
||||
|
||||
type ObserveResult =
|
||||
| { status: "found"; text: string; url: string }
|
||||
| { status: "absent"; url: string }
|
||||
| { status: "inconclusive"; reason: string; detail: string };
|
||||
|
||||
function makeSession(observe: ObserveResult, navOk = true): { session: AppDriverSession; disposed: () => number } {
|
||||
let disposed = 0;
|
||||
const session: AppDriverSession = {
|
||||
navigate: async (url) =>
|
||||
navOk
|
||||
? { status: "ok", url }
|
||||
: { status: "inconclusive", reason: "navigation-failed", detail: "boom" },
|
||||
observe: async () => observe,
|
||||
dispose: async () => {
|
||||
disposed += 1;
|
||||
},
|
||||
};
|
||||
return { session, disposed: () => disposed };
|
||||
}
|
||||
|
||||
function makeDeps(opts: {
|
||||
appLaunchThrows?: Error;
|
||||
driverResult?: AppDriverLaunchResult;
|
||||
observe?: ObserveResult;
|
||||
navOk?: boolean;
|
||||
}): AppDrivingDeps & { sessionDisposed: () => number; appDisposed: () => number } {
|
||||
const { app, disposed: appDisposed } = makeApp();
|
||||
const sessionHolder = opts.observe
|
||||
? makeSession(opts.observe, opts.navOk ?? true)
|
||||
: undefined;
|
||||
return {
|
||||
appDisposed,
|
||||
sessionDisposed: () => sessionHolder?.disposed() ?? 0,
|
||||
launchApp: async () => {
|
||||
if (opts.appLaunchThrows) throw opts.appLaunchThrows;
|
||||
return app;
|
||||
},
|
||||
launchDriver: async () => {
|
||||
if (opts.driverResult) return opts.driverResult;
|
||||
if (!sessionHolder) {
|
||||
return { status: "inconclusive", reason: "browser-unavailable", detail: "no browser" };
|
||||
}
|
||||
return { status: "ready", session: sessionHolder.session };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function uiRequest(overrides: Partial<VerificationRequest> = {}): VerificationRequest {
|
||||
return {
|
||||
assertionId: "CA-UI",
|
||||
assertion: "the bug no longer reproduces in the UI",
|
||||
taskId: "FN-9",
|
||||
channel: "app",
|
||||
ui: { path: "/board", selector: ".bug-banner", expectation: "absent" },
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
// ── App-driving outcome mapping (R21) ──────────────────────────────────────────
|
||||
|
||||
describe("AppDrivingVerificationCapability — outcome mapping", () => {
|
||||
it("UI bug still reproduces (found, expectation=absent) → fail", async () => {
|
||||
const deps = makeDeps({ observe: { status: "found", text: "Error!", url: "u" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("fail");
|
||||
expect(out.reason).toMatch(/still reproduces/);
|
||||
// Teardown of both surfaces.
|
||||
expect(deps.sessionDisposed()).toBe(1);
|
||||
expect(deps.appDisposed()).toBe(1);
|
||||
});
|
||||
|
||||
it("UI bug no longer reproduces (absent, expectation=absent) → pass", async () => {
|
||||
const deps = makeDeps({ observe: { status: "absent", url: "u" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("pass");
|
||||
expect(out.reason).toMatch(/no longer reproduces/);
|
||||
});
|
||||
|
||||
it("feature-present assertion: found → pass", async () => {
|
||||
const deps = makeDeps({ observe: { status: "found", text: "Save", url: "u" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(
|
||||
uiRequest({ ui: { path: "/settings", selector: "#save", expectation: "present" } }),
|
||||
);
|
||||
expect(out.verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("feature-present assertion: absent → fail", async () => {
|
||||
const deps = makeDeps({ observe: { status: "absent", url: "u" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(
|
||||
uiRequest({ ui: { path: "/settings", selector: "#save", expectation: "present" } }),
|
||||
);
|
||||
expect(out.verdict).toBe("fail");
|
||||
expect(out.reason).toMatch(/not observed/);
|
||||
});
|
||||
|
||||
it("driver inconclusive observation → inconclusive (never pass/fail)", async () => {
|
||||
const deps = makeDeps({ observe: { status: "inconclusive", reason: "driver-error", detail: "x" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/definitive observation/);
|
||||
});
|
||||
|
||||
it("driver unavailable → inconclusive (never default pass, never auto-fail)", async () => {
|
||||
const deps = makeDeps({
|
||||
driverResult: { status: "inconclusive", reason: "browser-unavailable", detail: "no chrome" },
|
||||
});
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/driver unavailable/);
|
||||
});
|
||||
|
||||
it("app launch failure → inconclusive", async () => {
|
||||
const deps = makeDeps({ appLaunchThrows: new Error("port bind failed") });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/app launch failed/);
|
||||
// App never came up, so nothing to dispose, but no crash either.
|
||||
expect(deps.appDisposed()).toBe(0);
|
||||
});
|
||||
|
||||
it("navigation failure → inconclusive", async () => {
|
||||
const deps = makeDeps({ observe: { status: "absent", url: "u" }, navOk: false });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/navigation/);
|
||||
});
|
||||
|
||||
it("missing UI spec → inconclusive (structurally un-exercisable)", async () => {
|
||||
const deps = makeDeps({ observe: { status: "absent", url: "u" } });
|
||||
const cap = new AppDrivingVerificationCapability({ deps });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest({ ui: undefined }));
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/un-exercisable/);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Determinism contract (R20) ─────────────────────────────────────────────────
|
||||
|
||||
function scriptedCapability(verdicts: VerificationOutcome["verdict"][]): VerificationCapability {
|
||||
let i = 0;
|
||||
return {
|
||||
verifyBehavioralAssertion: async (req) => {
|
||||
const verdict = verdicts[Math.min(i, verdicts.length - 1)];
|
||||
i += 1;
|
||||
return { verdict, assertionId: req.assertionId, reason: `scripted ${verdict}` };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("DeterministicVerificationCapability — R20", () => {
|
||||
it("default N is small (2)", () => {
|
||||
expect(DEFAULT_VERIFICATION_RUNS).toBe(2);
|
||||
});
|
||||
|
||||
it("flaky pass-then-fail → inconclusive, NOT fail", async () => {
|
||||
const cap = new DeterministicVerificationCapability({ inner: scriptedCapability(["pass", "fail"]) });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/flaky/);
|
||||
});
|
||||
|
||||
it("authoritative fail requires N-run agreement", async () => {
|
||||
const cap = new DeterministicVerificationCapability({ inner: scriptedCapability(["fail", "fail"]) });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("agreeing passes → pass", async () => {
|
||||
const cap = new DeterministicVerificationCapability({ inner: scriptedCapability(["pass", "pass"]) });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("an early inconclusive short-circuits (does not run again)", async () => {
|
||||
const inner = scriptedCapability(["inconclusive", "pass"]);
|
||||
const spy = vi.spyOn(inner, "verifyBehavioralAssertion");
|
||||
const cap = new DeterministicVerificationCapability({ inner, runs: 3 });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(spy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("a later inconclusive after an initial verdict → inconclusive (non-deterministic)", async () => {
|
||||
const cap = new DeterministicVerificationCapability({ inner: scriptedCapability(["fail", "inconclusive"]) });
|
||||
const out = await cap.verifyBehavioralAssertion(uiRequest());
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
expect(out.reason).toMatch(/non-deterministic/);
|
||||
});
|
||||
|
||||
it("N=3 requires all three to agree for an authoritative fail", async () => {
|
||||
const flaky = new DeterministicVerificationCapability({ inner: scriptedCapability(["fail", "fail", "pass"]), runs: 3 });
|
||||
expect((await flaky.verifyBehavioralAssertion(uiRequest())).verdict).toBe("inconclusive");
|
||||
const solid = new DeterministicVerificationCapability({ inner: scriptedCapability(["fail", "fail", "fail"]), runs: 3 });
|
||||
expect((await solid.verifyBehavioralAssertion(uiRequest())).verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
// ── Dispatch by assertion shape ────────────────────────────────────────────────
|
||||
|
||||
function constCapability(verdict: VerificationOutcome["verdict"], tag: string): VerificationCapability {
|
||||
return {
|
||||
verifyBehavioralAssertion: async (req) => ({ verdict, assertionId: req.assertionId, reason: tag }),
|
||||
};
|
||||
}
|
||||
|
||||
describe("DispatchingVerificationCapability — route by shape", () => {
|
||||
it("channel=test routes to the test channel only", async () => {
|
||||
const appSpy = vi.fn();
|
||||
const dispatch = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: { verifyBehavioralAssertion: appSpy },
|
||||
});
|
||||
const out = await dispatch.verifyBehavioralAssertion(uiRequest({ channel: "test" }));
|
||||
expect(out.reason).toBe("test");
|
||||
expect(appSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("defaults to the test channel when channel is unspecified", async () => {
|
||||
const appSpy = vi.fn();
|
||||
const dispatch = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: { verifyBehavioralAssertion: appSpy },
|
||||
});
|
||||
const out = await dispatch.verifyBehavioralAssertion(uiRequest({ channel: undefined }));
|
||||
expect(out.reason).toBe("test");
|
||||
expect(appSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("channel=app routes to the app channel only", async () => {
|
||||
const testSpy = vi.fn();
|
||||
const dispatch = new DispatchingVerificationCapability({
|
||||
testChannel: { verifyBehavioralAssertion: testSpy },
|
||||
appChannel: constCapability("pass", "app"),
|
||||
});
|
||||
const out = await dispatch.verifyBehavioralAssertion(uiRequest({ channel: "app" }));
|
||||
expect(out.reason).toBe("app");
|
||||
expect(testSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("channel=app with no app channel injected → inconclusive (not default pass)", async () => {
|
||||
const dispatch = new DispatchingVerificationCapability({ testChannel: constCapability("pass", "test") });
|
||||
const out = await dispatch.verifyBehavioralAssertion(uiRequest({ channel: "app" }));
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
});
|
||||
|
||||
it("channel=both passes only when BOTH channels pass", async () => {
|
||||
const bothPass = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: constCapability("pass", "app"),
|
||||
});
|
||||
expect((await bothPass.verifyBehavioralAssertion(uiRequest({ channel: "both" }))).verdict).toBe("pass");
|
||||
|
||||
const appFails = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: constCapability("fail", "app fail"),
|
||||
});
|
||||
const failOut = await appFails.verifyBehavioralAssertion(uiRequest({ channel: "both" }));
|
||||
expect(failOut.verdict).toBe("fail");
|
||||
expect(failOut.reason).toMatch(/app fail/);
|
||||
|
||||
const appInconclusive = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: constCapability("inconclusive", "app inc"),
|
||||
});
|
||||
expect((await appInconclusive.verifyBehavioralAssertion(uiRequest({ channel: "both" }))).verdict).toBe(
|
||||
"inconclusive",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("combineBoth — fail dominates, then inconclusive, else pass", () => {
|
||||
const pass: VerificationOutcome = { verdict: "pass", assertionId: "x", reason: "p" };
|
||||
const fail: VerificationOutcome = { verdict: "fail", assertionId: "x", reason: "f" };
|
||||
const inc: VerificationOutcome = { verdict: "inconclusive", assertionId: "x", reason: "i" };
|
||||
|
||||
it("any fail → fail", () => {
|
||||
expect(combineBoth("x", pass, fail).verdict).toBe("fail");
|
||||
expect(combineBoth("x", fail, inc).verdict).toBe("fail");
|
||||
});
|
||||
it("no fail, any inconclusive → inconclusive", () => {
|
||||
expect(combineBoth("x", pass, inc).verdict).toBe("inconclusive");
|
||||
});
|
||||
it("both pass → pass", () => {
|
||||
expect(combineBoth("x", pass, pass).verdict).toBe("pass");
|
||||
});
|
||||
});
|
||||
|
||||
// ── End-to-end: dispatch → app channel → determinism, all mocked ───────────────
|
||||
|
||||
describe("integration: dispatch + app-driving + determinism (mocked, no real app/browser)", () => {
|
||||
it("a flaky UI assertion across N runs resolves to inconclusive, not fail", async () => {
|
||||
// The app channel observes a flaky selector: found then absent (expectation
|
||||
// absent → fail then pass). Determinism collapses that to inconclusive.
|
||||
let call = 0;
|
||||
const flakyAppChannel: VerificationCapability = {
|
||||
verifyBehavioralAssertion: async (req) => {
|
||||
call += 1;
|
||||
const verdict = call === 1 ? "fail" : "pass";
|
||||
return { verdict, assertionId: req.assertionId, reason: `run ${call}` };
|
||||
},
|
||||
};
|
||||
const dispatch = new DispatchingVerificationCapability({
|
||||
testChannel: constCapability("pass", "test"),
|
||||
appChannel: new DeterministicVerificationCapability({ inner: flakyAppChannel }),
|
||||
});
|
||||
const out = await dispatch.verifyBehavioralAssertion(uiRequest({ channel: "app" }));
|
||||
expect(out.verdict).toBe("inconclusive");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,373 @@
|
||||
/**
|
||||
* Tests for the isolated-app-launch harness (U4).
|
||||
*
|
||||
* Characterizes the R13 isolation/safety contracts BEFORE any real navigation
|
||||
* (U8) drives the instance. Process-spawn and bundle-build are injected, so
|
||||
* these unit tests assert the isolation guarantees WITHOUT launching a full
|
||||
* server — a real end-to-end launch belongs in a heavier lane / manual smoke
|
||||
* (see scripts/boot-smoke.mjs), not the merge gate.
|
||||
*
|
||||
* Contracts covered:
|
||||
* - Launch never binds a reserved dashboard port (R13).
|
||||
* - Uses a fresh/empty disposable DB under a unique tmpdir, not the central DB,
|
||||
* with no credentials/agent logs seeded (R13).
|
||||
* - A stale bundle is rebuilt before launch (no false verdict from stale
|
||||
* dist/client) (R13).
|
||||
* - Teardown frees the port and removes the DB/tmpdir, even when the launched
|
||||
* process exits non-zero or times out (R13).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import { createServer as createNetServer } from "node:net";
|
||||
import {
|
||||
launchIsolatedApp,
|
||||
resolveReservedPorts,
|
||||
parsePortList,
|
||||
acquireNonReservedPort,
|
||||
createDisposableAppWorkspace,
|
||||
type AppProcessSpawner,
|
||||
type BundleBuilder,
|
||||
type LaunchedAppProcess,
|
||||
type AppLaunchOptions,
|
||||
} from "../mission-verification-app-harness.js";
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
// ── reserved-port policy ───────────────────────────────────────────────────────
|
||||
|
||||
describe("parsePortList", () => {
|
||||
it("parses a comma-separated list and drops invalid entries", () => {
|
||||
expect(parsePortList("4040, 5173 ,99999,abc,-1,0")).toEqual([4040, 5173]);
|
||||
});
|
||||
it("returns empty for undefined/empty", () => {
|
||||
expect(parsePortList(undefined)).toEqual([]);
|
||||
expect(parsePortList("")).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveReservedPorts", () => {
|
||||
it("always reserves 4040", () => {
|
||||
expect(resolveReservedPorts({}).has(4040)).toBe(true);
|
||||
});
|
||||
it("adds FUSION_RESERVED_PORTS, PORT, and FUSION_SERVER_PORT", () => {
|
||||
const reserved = resolveReservedPorts({
|
||||
FUSION_RESERVED_PORTS: "5000,5001",
|
||||
PORT: "6000",
|
||||
FUSION_SERVER_PORT: "7000",
|
||||
});
|
||||
expect([...reserved].sort((a, b) => a - b)).toEqual([4040, 5000, 5001, 6000, 7000]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("acquireNonReservedPort", () => {
|
||||
it("returns a port that is not in the reserved set", async () => {
|
||||
const reserved = new Set<number>([4040]);
|
||||
const port = await acquireNonReservedPort(reserved);
|
||||
expect(reserved.has(port)).toBe(false);
|
||||
expect(port).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ── injectable launch stubs ─────────────────────────────────────────────────────
|
||||
|
||||
interface SpawnRecord {
|
||||
calls: AppLaunchOptions[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A stub spawner that records its launch options and never starts a real
|
||||
* process. `behavior` controls readiness so teardown can be tested across
|
||||
* normal/crash/timeout outcomes.
|
||||
*/
|
||||
function makeStubSpawner(
|
||||
record: SpawnRecord,
|
||||
behavior: "ready" | "crash" | "hang" = "ready",
|
||||
): { spawner: AppProcessSpawner; killed: () => number } {
|
||||
let killCount = 0;
|
||||
const spawner: AppProcessSpawner = (options) => {
|
||||
record.calls.push(options);
|
||||
const handle: LaunchedAppProcess = {
|
||||
ready:
|
||||
behavior === "ready"
|
||||
? Promise.resolve()
|
||||
: behavior === "crash"
|
||||
? Promise.reject(new Error("server process exited with code 1 before becoming healthy"))
|
||||
: new Promise<void>(() => {
|
||||
/* never resolves — simulates a hung/timed-out launch */
|
||||
}),
|
||||
kill: async () => {
|
||||
killCount += 1;
|
||||
},
|
||||
};
|
||||
return handle;
|
||||
};
|
||||
return { spawner, killed: () => killCount };
|
||||
}
|
||||
|
||||
/** A bundle builder whose staleness and build are scripted and recorded. */
|
||||
function makeStubBundle(stale: boolean): BundleBuilder & {
|
||||
buildCalls: number;
|
||||
currentCalls: number;
|
||||
} {
|
||||
const state = {
|
||||
buildCalls: 0,
|
||||
currentCalls: 0,
|
||||
async isStale() {
|
||||
return stale;
|
||||
},
|
||||
async build() {
|
||||
state.buildCalls += 1;
|
||||
return { clientDir: "/tmp/fresh-built-client" };
|
||||
},
|
||||
async current() {
|
||||
state.currentCalls += 1;
|
||||
return { clientDir: "/tmp/already-built-client" };
|
||||
},
|
||||
};
|
||||
return state;
|
||||
}
|
||||
|
||||
// ── R13: never binds a reserved port ────────────────────────────────────────────
|
||||
|
||||
describe("launchIsolatedApp — reserved port (R13)", () => {
|
||||
it("never spawns on a reserved dashboard port, even with a wide reserved config", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner } = makeStubSpawner(record, "ready");
|
||||
const app = await launchIsolatedApp({
|
||||
spawn: spawner,
|
||||
bundle: makeStubBundle(false),
|
||||
env: { FUSION_RESERVED_PORTS: "4040,5173,6006", PORT: "3000" },
|
||||
});
|
||||
try {
|
||||
const reserved = resolveReservedPorts({ FUSION_RESERVED_PORTS: "4040,5173,6006", PORT: "3000" });
|
||||
expect(record.calls).toHaveLength(1);
|
||||
expect(reserved.has(app.port)).toBe(false);
|
||||
expect(app.port).not.toBe(4040);
|
||||
expect(app.baseUrl).toBe(`http://127.0.0.1:${app.port}`);
|
||||
expect(record.calls[0].port).toBe(app.port);
|
||||
} finally {
|
||||
await app.dispose();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── R13: fresh/empty disposable DB under a unique tmpdir ─────────────────────────
|
||||
|
||||
describe("createDisposableAppWorkspace — disposable DB (R13)", () => {
|
||||
it("creates a fresh empty DB under a unique tmpdir and removes it on dispose", async () => {
|
||||
const ws = await createDisposableAppWorkspace();
|
||||
// Under the OS tmpdir, not the central DB / project dir.
|
||||
expect(ws.tmpDir.startsWith(os.tmpdir())).toBe(true);
|
||||
expect(ws.dbPath.startsWith(ws.tmpDir)).toBe(true);
|
||||
// Fresh + empty: zero bytes => not a copy of the central DB, no seeded
|
||||
// credentials or agent logs.
|
||||
const stat = await fs.stat(ws.dbPath);
|
||||
expect(stat.size).toBe(0);
|
||||
await ws.dispose();
|
||||
await expect(fs.stat(ws.tmpDir)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("yields a unique tmpdir per call", async () => {
|
||||
const a = await createDisposableAppWorkspace();
|
||||
const b = await createDisposableAppWorkspace();
|
||||
try {
|
||||
expect(a.tmpDir).not.toBe(b.tmpDir);
|
||||
} finally {
|
||||
await a.dispose();
|
||||
await b.dispose();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("launchIsolatedApp — disposable DB wiring (R13)", () => {
|
||||
it("launches against a fresh tmpdir DB, never the central DB", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner } = makeStubSpawner(record, "ready");
|
||||
const app = await launchIsolatedApp({
|
||||
spawn: spawner,
|
||||
bundle: makeStubBundle(false),
|
||||
env: {},
|
||||
});
|
||||
try {
|
||||
expect(record.calls[0].dbPath).toBe(app.dbPath);
|
||||
expect(app.dbPath.startsWith(os.tmpdir())).toBe(true);
|
||||
// The disposable DB exists and is empty before disposal.
|
||||
const stat = await fs.stat(app.dbPath);
|
||||
expect(stat.size).toBe(0);
|
||||
// The scrubbed env must carry no credentials.
|
||||
const env = record.calls[0].env;
|
||||
expect(env.ANTHROPIC_API_KEY).toBeUndefined();
|
||||
expect(env.DATABASE_URL).toBeUndefined();
|
||||
} finally {
|
||||
await app.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
it("does not inherit credentials from the host env into the launched process", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner } = makeStubSpawner(record, "ready");
|
||||
const app = await launchIsolatedApp({
|
||||
spawn: spawner,
|
||||
bundle: makeStubBundle(false),
|
||||
env: {
|
||||
PATH: "/usr/bin",
|
||||
ANTHROPIC_API_KEY: "secret",
|
||||
FUSION_DAEMON_TOKEN: "tok",
|
||||
AWS_SECRET_ACCESS_KEY: "secret",
|
||||
},
|
||||
});
|
||||
try {
|
||||
const env = record.calls[0].env;
|
||||
expect(env.PATH).toBe("/usr/bin");
|
||||
expect(env.ANTHROPIC_API_KEY).toBeUndefined();
|
||||
expect(env.FUSION_DAEMON_TOKEN).toBeUndefined();
|
||||
expect(env.AWS_SECRET_ACCESS_KEY).toBeUndefined();
|
||||
} finally {
|
||||
await app.dispose();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── R13: stale bundle rebuilt before launch ─────────────────────────────────────
|
||||
|
||||
describe("launchIsolatedApp — fresh bundle (R13)", () => {
|
||||
it("rebuilds a stale bundle before launch and serves the freshly-built dir", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner } = makeStubSpawner(record, "ready");
|
||||
const bundle = makeStubBundle(true);
|
||||
const app = await launchIsolatedApp({ spawn: spawner, bundle, env: {} });
|
||||
try {
|
||||
expect(bundle.buildCalls).toBe(1);
|
||||
expect(bundle.currentCalls).toBe(0);
|
||||
expect(app.clientDir).toBe("/tmp/fresh-built-client");
|
||||
expect(record.calls[0].clientDir).toBe("/tmp/fresh-built-client");
|
||||
} finally {
|
||||
await app.dispose();
|
||||
}
|
||||
});
|
||||
|
||||
it("serves the current bundle without rebuilding when it is not stale", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner } = makeStubSpawner(record, "ready");
|
||||
const bundle = makeStubBundle(false);
|
||||
const app = await launchIsolatedApp({ spawn: spawner, bundle, env: {} });
|
||||
try {
|
||||
expect(bundle.buildCalls).toBe(0);
|
||||
expect(bundle.currentCalls).toBe(1);
|
||||
expect(app.clientDir).toBe("/tmp/already-built-client");
|
||||
} finally {
|
||||
await app.dispose();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── R13: unconditional teardown (crash / timeout / normal) ───────────────────────
|
||||
|
||||
describe("launchIsolatedApp — unconditional teardown (R13)", () => {
|
||||
it("frees the port and removes the tmpdir/DB on normal disposal", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "ready");
|
||||
const app = await launchIsolatedApp({ spawn: spawner, bundle: makeStubBundle(false), env: {} });
|
||||
const port = app.port;
|
||||
const dbPath = app.dbPath;
|
||||
await app.dispose();
|
||||
// Process killed.
|
||||
expect(killed()).toBe(1);
|
||||
// DB/tmpdir removed.
|
||||
await expect(fs.stat(dbPath)).rejects.toThrow();
|
||||
// Port freed — we can now bind it ourselves.
|
||||
await expect(bindAndRelease(port)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("tears down (kills process + removes DB) when the process crashes before ready", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "crash");
|
||||
await expect(
|
||||
launchIsolatedApp({ spawn: spawner, bundle: makeStubBundle(false), env: {} }),
|
||||
).rejects.toThrow(/exited with code 1/);
|
||||
// The launch failed, but teardown still ran: process killed, DB removed.
|
||||
expect(killed()).toBe(1);
|
||||
expect(record.calls).toHaveLength(1);
|
||||
await expect(fs.stat(record.calls[0].dbPath)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("tears down on a startup timeout (hung process)", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "hang");
|
||||
await expect(
|
||||
launchIsolatedApp({
|
||||
spawn: spawner,
|
||||
bundle: makeStubBundle(false),
|
||||
env: {},
|
||||
readyTimeoutMs: 20,
|
||||
}),
|
||||
).rejects.toThrow(/did not become ready/);
|
||||
expect(killed()).toBe(1);
|
||||
await expect(fs.stat(record.calls[0].dbPath)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("tears down when the launch is aborted via signal", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "hang");
|
||||
const controller = new AbortController();
|
||||
const launch = launchIsolatedApp({
|
||||
spawn: spawner,
|
||||
bundle: makeStubBundle(false),
|
||||
env: {},
|
||||
signal: controller.signal,
|
||||
});
|
||||
controller.abort();
|
||||
await expect(launch).rejects.toThrow(/abort/i);
|
||||
expect(killed()).toBe(1);
|
||||
await expect(fs.stat(record.calls[0].dbPath)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("removes the tmpdir/DB even if the bundle build fails before spawn", async () => {
|
||||
const failingBundle: BundleBuilder = {
|
||||
async isStale() {
|
||||
return true;
|
||||
},
|
||||
async build() {
|
||||
throw new Error("bundle build failed");
|
||||
},
|
||||
async current() {
|
||||
return { clientDir: "/tmp/unused" };
|
||||
},
|
||||
};
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "ready");
|
||||
await expect(
|
||||
launchIsolatedApp({ spawn: spawner, bundle: failingBundle, env: {} }),
|
||||
).rejects.toThrow(/bundle build failed/);
|
||||
// Spawn never happened; nothing to kill, and no tmpdir leak.
|
||||
expect(killed()).toBe(0);
|
||||
expect(record.calls).toHaveLength(0);
|
||||
// (Workspace was created then disposed — verified indirectly: no spawn means
|
||||
// the only filesystem artifact is the tmpdir, which dispose removed. We
|
||||
// cannot capture its path here, so this asserts the no-spawn invariant.)
|
||||
});
|
||||
|
||||
it("dispose is idempotent (safe to call twice)", async () => {
|
||||
const record: SpawnRecord = { calls: [] };
|
||||
const { spawner, killed } = makeStubSpawner(record, "ready");
|
||||
const app = await launchIsolatedApp({ spawn: spawner, bundle: makeStubBundle(false), env: {} });
|
||||
await app.dispose();
|
||||
await expect(app.dispose()).resolves.toBeUndefined();
|
||||
expect(killed()).toBe(2); // kill is idempotent at the handle level
|
||||
});
|
||||
});
|
||||
|
||||
/** Bind a port and release it — proves it is free. */
|
||||
function bindAndRelease(port: number): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const srv = createNetServer();
|
||||
srv.once("error", reject);
|
||||
srv.listen(port, "127.0.0.1", () => {
|
||||
srv.close(() => resolve());
|
||||
});
|
||||
});
|
||||
}
|
||||
355
packages/engine/src/__tests__/mission-verification.test.ts
Normal file
355
packages/engine/src/__tests__/mission-verification.test.ts
Normal file
@@ -0,0 +1,355 @@
|
||||
/**
|
||||
* Tests for the behavioral-verification capability (U3).
|
||||
*
|
||||
* Covers the isolation/safety contract before any real execution:
|
||||
* - command-template rejects shell metacharacters (R19)
|
||||
* - fail-closed when no isolating sandbox backend is available (R18)
|
||||
* - env scrubbed to a minimal allowlist (R18)
|
||||
* - pass-on-both regression test rejected (R5/AE5)
|
||||
* - source tree git-clean post-condition asserted (R17)
|
||||
* - no integration SHA → inconclusive (R11 fail-closed)
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||
import {
|
||||
validateTestPath,
|
||||
buildVerificationCommand,
|
||||
selectIsolatingBackend,
|
||||
scrubEnv,
|
||||
VERIFICATION_ENV_ALLOWLIST,
|
||||
TestExecutionVerificationCapability,
|
||||
type CheckoutMaterializer,
|
||||
type IsolatingBackendProbe,
|
||||
type VerificationRequest,
|
||||
} from "../mission-verification.js";
|
||||
import {
|
||||
__resetSandboxBackendForTests,
|
||||
type SandboxBackend,
|
||||
type SandboxStreamingResult,
|
||||
} from "../sandbox/index.js";
|
||||
import type { TaskStore } from "@fusion/core";
|
||||
|
||||
afterEach(() => {
|
||||
__resetSandboxBackendForTests();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
// ── validateTestPath (R19) ────────────────────────────────────────────────────
|
||||
|
||||
describe("validateTestPath", () => {
|
||||
it("accepts a plain relative test path", () => {
|
||||
expect(validateTestPath("packages/engine/src/__tests__/foo.test.ts")).toBe(
|
||||
"packages/engine/src/__tests__/foo.test.ts",
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
"foo.test.ts; rm -rf /",
|
||||
"foo.test.ts && curl evil",
|
||||
"foo.test.ts | cat",
|
||||
"$(whoami).test.ts",
|
||||
"`id`.test.ts",
|
||||
"foo.test.ts\nrm x",
|
||||
"a${b}.test.ts",
|
||||
"foo>out.test.ts",
|
||||
])("rejects shell metacharacters: %s", (p) => {
|
||||
expect(validateTestPath(p)).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects absolute paths, parent escapes, flag-like, and non-strings", () => {
|
||||
expect(validateTestPath("/etc/passwd")).toBeNull();
|
||||
expect(validateTestPath("../../etc/passwd")).toBeNull();
|
||||
expect(validateTestPath("a/../../b")).toBeNull();
|
||||
expect(validateTestPath("--config=evil")).toBeNull();
|
||||
expect(validateTestPath("")).toBeNull();
|
||||
expect(validateTestPath(42 as unknown)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildVerificationCommand", () => {
|
||||
it("substitutes a validated test path into the template", () => {
|
||||
expect(buildVerificationCommand("pnpm vitest run {testPath}", "src/a.test.ts")).toBe(
|
||||
"pnpm vitest run src/a.test.ts",
|
||||
);
|
||||
});
|
||||
|
||||
it("produces a whole-suite command when no path is supplied", () => {
|
||||
expect(buildVerificationCommand("pnpm vitest run {testPath}")).toBe("pnpm vitest run");
|
||||
});
|
||||
|
||||
it("throws if asked to substitute an unsafe path (defense in depth)", () => {
|
||||
expect(() => buildVerificationCommand("pnpm vitest run {testPath}", "a; rm -rf /")).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
// ── selectIsolatingBackend (R18 fail-closed) ───────────────────────────────────
|
||||
|
||||
describe("selectIsolatingBackend", () => {
|
||||
it("selects bubblewrap on linux when available", () => {
|
||||
expect(
|
||||
selectIsolatingBackend({ platform: "linux", bubblewrapAvailable: true, sandboxExecAvailable: false }).backendId,
|
||||
).toBe("bubblewrap");
|
||||
});
|
||||
|
||||
it("selects sandbox-exec on darwin when available", () => {
|
||||
expect(
|
||||
selectIsolatingBackend({ platform: "darwin", bubblewrapAvailable: false, sandboxExecAvailable: true }).backendId,
|
||||
).toBe("sandbox-exec");
|
||||
});
|
||||
|
||||
it("fails closed (null) when no isolating backend is available", () => {
|
||||
const sel = selectIsolatingBackend({ platform: "linux", bubblewrapAvailable: false, sandboxExecAvailable: false });
|
||||
expect(sel.backendId).toBeNull();
|
||||
expect(sel.reason).toMatch(/no isolating sandbox backend/);
|
||||
});
|
||||
});
|
||||
|
||||
// ── scrubEnv (R18) ─────────────────────────────────────────────────────────────
|
||||
|
||||
describe("scrubEnv", () => {
|
||||
it("keeps only allowlisted keys and forces CI=1", () => {
|
||||
const result = scrubEnv({
|
||||
PATH: "/usr/bin",
|
||||
HOME: "/home/u",
|
||||
ANTHROPIC_API_KEY: "secret",
|
||||
DATABASE_URL: "postgres://secret",
|
||||
AWS_SECRET_ACCESS_KEY: "secret",
|
||||
});
|
||||
expect(result.PATH).toBe("/usr/bin");
|
||||
expect(result.HOME).toBe("/home/u");
|
||||
expect(result.CI).toBe("1");
|
||||
expect(result.ANTHROPIC_API_KEY).toBeUndefined();
|
||||
expect(result.DATABASE_URL).toBeUndefined();
|
||||
expect(result.AWS_SECRET_ACCESS_KEY).toBeUndefined();
|
||||
});
|
||||
|
||||
it("only ever emits allowlisted keys plus CI", () => {
|
||||
const result = scrubEnv({ FOO: "x", BAR: "y", PATH: "/bin" });
|
||||
const allowed = new Set<string>([...VERIFICATION_ENV_ALLOWLIST, "CI"]);
|
||||
for (const k of Object.keys(result)) {
|
||||
expect(allowed.has(k)).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── TestExecutionVerificationCapability ────────────────────────────────────────
|
||||
|
||||
function makeMockStore(): TaskStore {
|
||||
return {
|
||||
logEntry: vi.fn().mockResolvedValue(undefined),
|
||||
appendAgentLog: vi.fn().mockResolvedValue(undefined),
|
||||
} as unknown as TaskStore;
|
||||
}
|
||||
|
||||
/** A fake materializer that records dispose/clean calls without touching git. */
|
||||
function makeFakeMaterializer(): CheckoutMaterializer & {
|
||||
disposed: number;
|
||||
cleanCalls: number;
|
||||
setDirty(dirty: boolean): void;
|
||||
} {
|
||||
let dirty = false;
|
||||
const state = {
|
||||
disposed: 0,
|
||||
cleanCalls: 0,
|
||||
setDirty(d: boolean) {
|
||||
dirty = d;
|
||||
},
|
||||
async materialize(_rootDir: string, _revision: string) {
|
||||
return {
|
||||
dir: "/tmp/fake-checkout",
|
||||
dispose: async () => {
|
||||
state.disposed += 1;
|
||||
},
|
||||
};
|
||||
},
|
||||
async assertSourceClean(_rootDir: string) {
|
||||
state.cleanCalls += 1;
|
||||
if (dirty) throw new Error("Source tree is not git-clean after verification run");
|
||||
},
|
||||
};
|
||||
return state;
|
||||
}
|
||||
|
||||
/** A sandbox backend whose runStreaming returns a scripted outcome. */
|
||||
function makeScriptedBackend(outcomes: SandboxStreamingResult[]): SandboxBackend {
|
||||
let i = 0;
|
||||
return {
|
||||
capabilities: () => ({
|
||||
id: "bubblewrap",
|
||||
supportsNetworkPolicy: true,
|
||||
supportsFilesystemPolicy: true,
|
||||
supportsStreaming: true,
|
||||
platform: "any",
|
||||
}),
|
||||
prepare: vi.fn().mockResolvedValue(undefined),
|
||||
run: vi.fn(),
|
||||
runStreaming: vi.fn(async () => {
|
||||
const out = outcomes[Math.min(i, outcomes.length - 1)];
|
||||
i += 1;
|
||||
return out;
|
||||
}),
|
||||
dispose: vi.fn().mockResolvedValue(undefined),
|
||||
} as unknown as SandboxBackend;
|
||||
}
|
||||
|
||||
const isolatingProbe = (): Promise<IsolatingBackendProbe> =>
|
||||
Promise.resolve({ platform: "linux", bubblewrapAvailable: true, sandboxExecAvailable: false });
|
||||
|
||||
function baseRequest(overrides: Partial<VerificationRequest> = {}): VerificationRequest {
|
||||
return {
|
||||
assertionId: "CA-1",
|
||||
assertion: "the bug no longer reproduces",
|
||||
taskId: "FN-1",
|
||||
integrationSha: "abc123",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("TestExecutionVerificationCapability", () => {
|
||||
it("fails closed to inconclusive when no isolating backend is available (R18)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: async () => ({ platform: "linux", bubblewrapAvailable: false, sandboxExecAvailable: false }),
|
||||
});
|
||||
|
||||
const outcome = await cap.verifyBehavioralAssertion(baseRequest());
|
||||
expect(outcome.verdict).toBe("inconclusive");
|
||||
expect(outcome.reason).toMatch(/no isolating sandbox backend/);
|
||||
// Never materialized / executed.
|
||||
expect(materializer.disposed).toBe(0);
|
||||
});
|
||||
|
||||
it("returns inconclusive when no integration SHA is available (R11)", async () => {
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer: makeFakeMaterializer(),
|
||||
probeBackends: isolatingProbe,
|
||||
});
|
||||
const outcome = await cap.verifyBehavioralAssertion(baseRequest({ integrationSha: undefined }));
|
||||
expect(outcome.verdict).toBe("inconclusive");
|
||||
expect(outcome.reason).toMatch(/integration SHA/);
|
||||
});
|
||||
|
||||
it("rejects an agent test path with shell metacharacters before execution (R19)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
});
|
||||
const outcome = await cap.verifyBehavioralAssertion(
|
||||
baseRequest({ proof: { testFilePath: "a.test.ts; rm -rf /" } }),
|
||||
);
|
||||
expect(outcome.verdict).toBe("inconclusive");
|
||||
expect(outcome.reason).toMatch(/shell metacharacters|rejected/);
|
||||
expect(materializer.disposed).toBe(0);
|
||||
});
|
||||
|
||||
it("passes when the whole-suite run succeeds, and asserts source git-clean (R17)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
backendFactory: () =>
|
||||
makeScriptedBackend([{ outcome: "success", stdout: "ok", stderr: "", bufferOverflow: false }]),
|
||||
});
|
||||
const outcome = await cap.verifyBehavioralAssertion(baseRequest());
|
||||
expect(outcome.verdict).toBe("pass");
|
||||
expect(materializer.disposed).toBe(1);
|
||||
expect(materializer.cleanCalls).toBe(1);
|
||||
});
|
||||
|
||||
it("rejects a regression test that passes on BOTH baseline and implementation (R5/AE5)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
// impl run (1st) success, baseline run (2nd) success → pass-on-both.
|
||||
backendFactory: () =>
|
||||
makeScriptedBackend([
|
||||
{ outcome: "success", stdout: "ok", stderr: "", bufferOverflow: false },
|
||||
{ outcome: "success", stdout: "ok", stderr: "", bufferOverflow: false },
|
||||
]),
|
||||
});
|
||||
const outcome = await cap.verifyBehavioralAssertion(
|
||||
baseRequest({ proof: { testFilePath: "src/a.test.ts" }, mergeBaseSha: "base000" }),
|
||||
);
|
||||
expect(outcome.verdict).toBe("fail");
|
||||
expect(outcome.reason).toMatch(/both/);
|
||||
});
|
||||
|
||||
it("passes when a regression test fails on baseline and passes on implementation (R5)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
// impl run (1st) success, baseline run (2nd) non-zero-exit → genuine proof.
|
||||
backendFactory: () =>
|
||||
makeScriptedBackend([
|
||||
{ outcome: "success", stdout: "ok", stderr: "", bufferOverflow: false },
|
||||
{ outcome: "non-zero-exit", stdout: "", stderr: "boom", exitCode: 1, signal: null },
|
||||
]),
|
||||
});
|
||||
const outcome = await cap.verifyBehavioralAssertion(
|
||||
baseRequest({ proof: { testFilePath: "src/a.test.ts" }, mergeBaseSha: "base000" }),
|
||||
);
|
||||
expect(outcome.verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("inconclusive when the run times out (R9), still asserts source clean", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
backendFactory: () =>
|
||||
makeScriptedBackend([{ outcome: "timeout", stdout: "", stderr: "", timeoutMs: 1000 }]),
|
||||
});
|
||||
// A timeout surfaces as a thrown ETIMEDOUT inside runVerificationCommand,
|
||||
// which the capability catches and maps deterministically to inconclusive.
|
||||
// An infra timeout must never be surfaced as a behavioral fail — the contract
|
||||
// is that timeout/setup failures stay inconclusive.
|
||||
const outcome = await cap.verifyBehavioralAssertion(baseRequest());
|
||||
expect(outcome.verdict).toBe("inconclusive");
|
||||
expect(materializer.cleanCalls).toBe(1);
|
||||
});
|
||||
|
||||
it("fails closed to inconclusive if the source tree is dirty after a run (R17 post-condition)", async () => {
|
||||
const materializer = makeFakeMaterializer();
|
||||
materializer.setDirty(true);
|
||||
const cap = new TestExecutionVerificationCapability({
|
||||
store: makeMockStore(),
|
||||
rootDir: "/repo",
|
||||
commandTemplate: "pnpm vitest run {testPath}",
|
||||
materializer,
|
||||
probeBackends: isolatingProbe,
|
||||
backendFactory: () =>
|
||||
makeScriptedBackend([{ outcome: "success", stdout: "ok", stderr: "", bufferOverflow: false }]),
|
||||
});
|
||||
// A dirty tree means verification mutated the source — never trust the
|
||||
// verdict; fail closed to inconclusive (the post-condition is checked
|
||||
// outside finally so it cannot mask the verdict via an unsafe throw).
|
||||
const outcome = await cap.verifyBehavioralAssertion(baseRequest());
|
||||
expect(outcome.verdict).toBe("inconclusive");
|
||||
expect(outcome.reason).toMatch(/git-clean/);
|
||||
});
|
||||
});
|
||||
@@ -552,6 +552,9 @@ describe("ProjectEngineManager", () => {
|
||||
manager.stopReconciliation();
|
||||
});
|
||||
|
||||
// Flake under full reliability-suite load: 30s timeout, but passes in ~46ms
|
||||
// standalone. Setinterval-driven reconciliation appears to race with vitest
|
||||
// fake-timer contention when other reliability-pool files are co-resident.
|
||||
it("retries failed project starts on subsequent reconciliation ticks", async () => {
|
||||
// Track how many times start() is called to fail only the FIRST set
|
||||
let startCallCount = 0;
|
||||
|
||||
@@ -0,0 +1,501 @@
|
||||
/**
|
||||
* Behavioral-verification posture in the Validator Run (U2 + U3).
|
||||
*
|
||||
* Verifies the default-to-fail posture for behavioral assertions and the
|
||||
* non-mutating verification step that confirms them, while static assertions
|
||||
* keep the exact legacy judge path.
|
||||
*
|
||||
* Covers:
|
||||
* - AE2: behavioral assertion with no verification evidence → fail, even when
|
||||
* the judge text claims pass (capability absent).
|
||||
* - AE3: static assertion → unchanged static verdict, no verification invoked.
|
||||
* - Mixed set: static and behavioral each take their correct path.
|
||||
* - Behavioral pass via an injected verification capability.
|
||||
* - Behavioral inconclusive → blocked verdict, NO fix feature.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import type {
|
||||
Mission,
|
||||
Milestone,
|
||||
Slice,
|
||||
MissionFeature,
|
||||
MissionValidatorRun,
|
||||
} from "@fusion/core";
|
||||
import type { VerificationCapability, VerificationOutcome } from "../../mission-verification.js";
|
||||
|
||||
// ── Mock AI dependencies (mirror mission-execution-loop.test.ts) ───────────────
|
||||
const mockSessionHolder: {
|
||||
session: { state: { messages: Array<{ role: string; content: string }> }; dispose: ReturnType<typeof vi.fn> };
|
||||
} = { session: { state: { messages: [] }, dispose: vi.fn() } };
|
||||
|
||||
vi.mock("../../pi.js", () => ({
|
||||
createFnAgent: vi.fn(() => Promise.resolve({ session: mockSessionHolder.session })),
|
||||
promptWithFallback: vi.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
vi.mock("../../logger.js", () => ({
|
||||
createLogger: vi.fn(() => ({ log: vi.fn(), warn: vi.fn(), error: vi.fn() })),
|
||||
}));
|
||||
|
||||
vi.mock("../../agent-session-helpers.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../../agent-session-helpers.js")>();
|
||||
return {
|
||||
...actual,
|
||||
createResolvedAgentSession: vi.fn(async () => ({
|
||||
session: mockSessionHolder.session as any,
|
||||
sessionFile: undefined,
|
||||
runtimeId: "test-runtime",
|
||||
wasConfigured: true,
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
import { createResolvedAgentSession } from "../../agent-session-helpers.js";
|
||||
import { MissionExecutionLoop } from "../../mission-execution-loop.js";
|
||||
|
||||
type AssertionRow = {
|
||||
id: string;
|
||||
milestoneId: string;
|
||||
title: string;
|
||||
assertion: string;
|
||||
status: "pending" | "passed" | "failed" | "blocked";
|
||||
type?: "static" | "behavioral";
|
||||
orderIndex: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
sourceFeatureId?: string;
|
||||
};
|
||||
|
||||
function now() {
|
||||
return new Date().toISOString();
|
||||
}
|
||||
|
||||
function createMockMission(): Mission {
|
||||
return {
|
||||
id: "M-TEST1",
|
||||
title: "Test Mission",
|
||||
status: "active",
|
||||
interviewState: "not_started",
|
||||
autopilotEnabled: true,
|
||||
autopilotState: "inactive",
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
};
|
||||
}
|
||||
|
||||
function createMockMilestone(overrides: Partial<Milestone> = {}): Milestone {
|
||||
return {
|
||||
id: "MS-001",
|
||||
missionId: "M-TEST1",
|
||||
title: "Test Milestone",
|
||||
status: "active",
|
||||
orderIndex: 0,
|
||||
interviewState: "not_started",
|
||||
dependencies: [],
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function createMockSlice(overrides: Partial<Slice> = {}): Slice {
|
||||
return {
|
||||
id: "SL-001",
|
||||
milestoneId: "MS-001",
|
||||
title: "Test Slice",
|
||||
status: "active",
|
||||
planState: "not_started",
|
||||
orderIndex: 0,
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function createMockFeature(overrides: Partial<MissionFeature> = {}): MissionFeature {
|
||||
return {
|
||||
id: "F-001",
|
||||
sliceId: "SL-001",
|
||||
title: "Test Feature",
|
||||
status: "defined",
|
||||
loopState: "idle",
|
||||
implementationAttemptCount: 0,
|
||||
validatorAttemptCount: 0,
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function createMockMissionStore() {
|
||||
const missions = new Map<string, Mission>();
|
||||
const features = new Map<string, MissionFeature>();
|
||||
const assertionsByFeature = new Map<string, AssertionRow[]>();
|
||||
const validatorRuns = new Map<string, MissionValidatorRun>();
|
||||
let runSeq = 0;
|
||||
|
||||
const store = {
|
||||
getMission: vi.fn((id: string) => missions.get(id)),
|
||||
logMissionEvent: vi.fn(),
|
||||
getFeature: vi.fn((id: string) => features.get(id)),
|
||||
getFeatureByTaskId: vi.fn((taskId: string) => {
|
||||
for (const f of features.values()) if (f.taskId === taskId) return f;
|
||||
return undefined;
|
||||
}),
|
||||
updateFeatureStatus: vi.fn((id: string, status: MissionFeature["status"]) => {
|
||||
const f = features.get(id)!;
|
||||
const updated = { ...f, status, updatedAt: now() };
|
||||
features.set(id, updated);
|
||||
return updated;
|
||||
}),
|
||||
listAssertionsForFeature: vi.fn((featureId: string) => assertionsByFeature.get(featureId) ?? []),
|
||||
ensureFeatureAssertionLinked: vi.fn((featureId: string) => assertionsByFeature.get(featureId) ?? []),
|
||||
getSlice: vi.fn((id: string) => createMockSlice({ id })),
|
||||
getMilestone: vi.fn((id: string) => createMockMilestone({ id })),
|
||||
startValidatorRun: vi.fn((featureId: string) => {
|
||||
const run: MissionValidatorRun = {
|
||||
id: `VR-${++runSeq}`,
|
||||
featureId,
|
||||
milestoneId: "MS-001",
|
||||
sliceId: "SL-001",
|
||||
status: "running",
|
||||
triggerType: "task_completion",
|
||||
implementationAttempt: 1,
|
||||
validatorAttempt: 1,
|
||||
startedAt: now(),
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
};
|
||||
validatorRuns.set(run.id, run);
|
||||
return run;
|
||||
}),
|
||||
getValidatorRun: vi.fn((id: string) => validatorRuns.get(id)),
|
||||
completeValidatorRun: vi.fn((id: string, status: MissionValidatorRun["status"], summary?: string) => {
|
||||
const run = validatorRuns.get(id)!;
|
||||
const updated = { ...run, status, summary, completedAt: now(), updatedAt: now() };
|
||||
validatorRuns.set(id, updated);
|
||||
const feature = features.get(run.featureId);
|
||||
if (feature) {
|
||||
const loopState = status === "passed" ? "passed" : status === "failed" ? "needs_fix" : status === "blocked" ? "blocked" : "validating";
|
||||
features.set(run.featureId, { ...feature, loopState: loopState as any, lastValidatorStatus: status, updatedAt: now() });
|
||||
}
|
||||
return updated;
|
||||
}),
|
||||
recordValidatorFailures: vi.fn(() => []),
|
||||
createGeneratedFixFeature: vi.fn((sourceFeatureId: string, runId: string) => {
|
||||
const src = features.get(sourceFeatureId)!;
|
||||
const fix = createMockFeature({ id: `FIX-${sourceFeatureId}`, taskId: `TASK-FIX-${sourceFeatureId}`, generatedFromFeatureId: sourceFeatureId, generatedFromRunId: runId, loopState: "implementing" });
|
||||
features.set(fix.id, fix);
|
||||
features.set(sourceFeatureId, { ...src, loopState: "implementing", implementationAttemptCount: (src.implementationAttemptCount ?? 0) + 1, updatedAt: now() });
|
||||
return fix;
|
||||
}),
|
||||
triageFeature: vi.fn(async (featureId: string) => {
|
||||
const f = features.get(featureId)!;
|
||||
const updated = { ...f, status: "triaged" as const, updatedAt: now() };
|
||||
features.set(featureId, updated);
|
||||
return updated;
|
||||
}),
|
||||
on: vi.fn(),
|
||||
off: vi.fn(),
|
||||
emit: vi.fn(),
|
||||
_setMission: (m: Mission) => missions.set(m.id, m),
|
||||
_setFeature: (f: MissionFeature) => features.set(f.id, f),
|
||||
_setAssertions: (featureId: string, rows: AssertionRow[]) => assertionsByFeature.set(featureId, rows),
|
||||
};
|
||||
return store;
|
||||
}
|
||||
|
||||
function createMockTaskStore() {
|
||||
const tasks = new Map<string, any>();
|
||||
return {
|
||||
getTask: vi.fn(async (id: string) => tasks.get(id)),
|
||||
moveTask: vi.fn(async () => {}),
|
||||
updateTask: vi.fn(async () => {}),
|
||||
getSettings: vi.fn().mockResolvedValue({ missionStaleThresholdMs: 600_000, missionMaxTaskRetries: 3 }),
|
||||
recordRunAuditEvent: vi.fn(),
|
||||
on: vi.fn(),
|
||||
off: vi.fn(),
|
||||
_setTask: (t: any) => tasks.set(t.id, t),
|
||||
};
|
||||
}
|
||||
|
||||
function assertionRow(overrides: Partial<AssertionRow> & { id: string }): AssertionRow {
|
||||
return {
|
||||
milestoneId: "MS-001",
|
||||
title: overrides.id,
|
||||
assertion: `do ${overrides.id}`,
|
||||
status: "pending",
|
||||
orderIndex: 0,
|
||||
createdAt: now(),
|
||||
updatedAt: now(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("Validator behavioral posture (U2 + U3)", () => {
|
||||
let missionStore: ReturnType<typeof createMockMissionStore>;
|
||||
let taskStore: ReturnType<typeof createMockTaskStore>;
|
||||
let loop: MissionExecutionLoop;
|
||||
|
||||
beforeEach(() => {
|
||||
missionStore = createMockMissionStore();
|
||||
taskStore = createMockTaskStore();
|
||||
vi.mocked(createResolvedAgentSession).mockReset();
|
||||
vi.mocked(createResolvedAgentSession).mockResolvedValue({
|
||||
session: mockSessionHolder.session as any,
|
||||
sessionFile: undefined,
|
||||
runtimeId: "test-runtime",
|
||||
wasConfigured: true,
|
||||
});
|
||||
missionStore._setMission(createMockMission());
|
||||
mockSessionHolder.session.state.messages = [];
|
||||
mockSessionHolder.session.dispose = vi.fn();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
loop?.stop();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
function judgePass(assertionIds: string[]) {
|
||||
mockSessionHolder.session.state.messages = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: JSON.stringify({
|
||||
status: "pass",
|
||||
assertions: assertionIds.map((id) => ({ assertionId: id, passed: true })),
|
||||
summary: "all good",
|
||||
}),
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
it("AE2: behavioral assertion the judge calls pass → fails with no verification capability", async () => {
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-B", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-B", title: "behavioral", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp" });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-B");
|
||||
|
||||
// No verification capability → behavioral default-to-fail → fix flow.
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "failed", expect.any(String));
|
||||
expect(missionStore.createGeneratedFixFeature).toHaveBeenCalled();
|
||||
expect(missionStore.getFeature("F-001")?.status).not.toBe("done");
|
||||
});
|
||||
|
||||
it("AE3: static assertion the judge calls pass → passes, no verification invoked", async () => {
|
||||
const verify = vi.fn();
|
||||
const capability: VerificationCapability = { verifyBehavioralAssertion: verify };
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-S", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "static" })]);
|
||||
taskStore._setTask({ id: "FN-S", title: "static", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: capability });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-S");
|
||||
|
||||
expect(verify).not.toHaveBeenCalled();
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "passed", expect.any(String));
|
||||
expect(missionStore.updateFeatureStatus).toHaveBeenCalledWith("F-001", "done");
|
||||
});
|
||||
|
||||
it("untyped assertions default to static — legacy judge pass path is preserved", async () => {
|
||||
const verify = vi.fn();
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-U", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
// No `type` field → normalizes to static.
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1" })]);
|
||||
taskStore._setTask({ id: "FN-U", title: "untyped", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-U");
|
||||
|
||||
expect(verify).not.toHaveBeenCalled();
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "passed", expect.any(String));
|
||||
});
|
||||
|
||||
it("behavioral assertion confirmed by an injected verification capability → passes", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "pass", assertionId: req.assertionId, reason: "confirmed" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-BV", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-BV", title: "behavioral verified", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-BV");
|
||||
|
||||
expect(verify).toHaveBeenCalledTimes(1);
|
||||
expect(verify.mock.calls[0][0]).toMatchObject({ assertionId: "CA-1", integrationSha: "sha123" });
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "passed", expect.any(String));
|
||||
expect(missionStore.updateFeatureStatus).toHaveBeenCalledWith("F-001", "done");
|
||||
});
|
||||
|
||||
it("behavioral assertion verification inconclusive → blocked, NO fix feature", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "inconclusive", assertionId: req.assertionId, reason: "no isolating sandbox backend" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-INC", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-INC", title: "inconclusive", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-INC");
|
||||
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "blocked", expect.any(String));
|
||||
expect(missionStore.createGeneratedFixFeature).not.toHaveBeenCalled();
|
||||
expect(missionStore.getFeature("F-001")?.status).not.toBe("done");
|
||||
});
|
||||
|
||||
it("mixed set: static passes via judge, behavioral confirmed via verification → overall pass", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "pass", assertionId: req.assertionId, reason: "confirmed" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-MIX", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [
|
||||
assertionRow({ id: "CA-static", type: "static" }),
|
||||
assertionRow({ id: "CA-behav", type: "behavioral" }),
|
||||
]);
|
||||
taskStore._setTask({ id: "FN-MIX", title: "mixed", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-static", "CA-behav"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-MIX");
|
||||
|
||||
// Only the behavioral assertion is verified.
|
||||
expect(verify).toHaveBeenCalledTimes(1);
|
||||
expect(verify.mock.calls[0][0]).toMatchObject({ assertionId: "CA-behav" });
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "passed", expect.any(String));
|
||||
expect(missionStore.updateFeatureStatus).toHaveBeenCalledWith("F-001", "done");
|
||||
});
|
||||
|
||||
it("mixed set: behavioral observed wrong → overall fail even though static passes", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "fail", assertionId: req.assertionId, reason: "defect still reproduces" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-MIX2", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [
|
||||
assertionRow({ id: "CA-static", type: "static" }),
|
||||
assertionRow({ id: "CA-behav", type: "behavioral" }),
|
||||
]);
|
||||
taskStore._setTask({ id: "FN-MIX2", title: "mixed fail", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-static", "CA-behav"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-MIX2");
|
||||
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "failed", expect.any(String));
|
||||
expect(missionStore.createGeneratedFixFeature).toHaveBeenCalled();
|
||||
expect(missionStore.getFeature("F-001")?.status).not.toBe("done");
|
||||
});
|
||||
|
||||
it("U6/R6: failed verification passes the observed-vs-expected reason to the Fix Feature", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({
|
||||
verdict: "fail",
|
||||
assertionId: req.assertionId,
|
||||
reason: "defect still reproduces",
|
||||
detail: "button still does nothing on click",
|
||||
}));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-R6", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-R6", title: "reason", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-R6");
|
||||
|
||||
expect(missionStore.createGeneratedFixFeature).toHaveBeenCalled();
|
||||
const call = (missionStore.createGeneratedFixFeature as any).mock.calls[0];
|
||||
// (sourceFeatureId, runId, failedAssertionIds, failureReason)
|
||||
expect(call[0]).toBe("F-001");
|
||||
expect(call[2]).toEqual(["CA-1"]);
|
||||
expect(typeof call[3]).toBe("string");
|
||||
expect(call[3]).toContain("defect still reproduces");
|
||||
});
|
||||
|
||||
it("U6/R16: a verification FAILURE emits a persisted mission event with outcome=fail", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "fail", assertionId: req.assertionId, reason: "defect still reproduces" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-EVT-F", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-EVT-F", title: "evt fail", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-EVT-F");
|
||||
|
||||
const failEvent = (missionStore.logMissionEvent as any).mock.calls.find(
|
||||
(c: any[]) => c[3]?.code === "validation_failed",
|
||||
);
|
||||
expect(failEvent).toBeDefined();
|
||||
expect(failEvent[1]).toBe("error");
|
||||
expect(failEvent[3]).toMatchObject({ outcome: "fail" });
|
||||
});
|
||||
|
||||
it("U6/R16+R21: an INCONCLUSIVE verdict emits a distinguishable infra-failure event and no Fix Feature", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "inconclusive", assertionId: req.assertionId, reason: "no isolating sandbox backend" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-EVT-INC", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-EVT-INC", title: "evt inconclusive", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-EVT-INC");
|
||||
|
||||
// No remediation work.
|
||||
expect(missionStore.createGeneratedFixFeature).not.toHaveBeenCalled();
|
||||
// Completed as blocked (no new run status), distinct from a real fail.
|
||||
expect(missionStore.completeValidatorRun).toHaveBeenCalledWith(expect.any(String), "blocked", expect.any(String));
|
||||
|
||||
const incEvent = (missionStore.logMissionEvent as any).mock.calls.find(
|
||||
(c: any[]) => c[3]?.code === "verification_inconclusive",
|
||||
);
|
||||
expect(incEvent).toBeDefined();
|
||||
// Distinguishable from a real fail: warning severity + infra-failure marker.
|
||||
expect(incEvent[1]).toBe("warning");
|
||||
expect(incEvent[3]).toMatchObject({ outcome: "inconclusive", infraFailure: true });
|
||||
// A real-fail event must NOT have been emitted for this run.
|
||||
const failEvent = (missionStore.logMissionEvent as any).mock.calls.find(
|
||||
(c: any[]) => c[3]?.code === "validation_failed",
|
||||
);
|
||||
expect(failEvent).toBeUndefined();
|
||||
});
|
||||
|
||||
it("U6/R16: a swallowed Fix-Feature triage error is durably recorded, not silent", async () => {
|
||||
const verify = vi.fn(async (req): Promise<VerificationOutcome> => ({ verdict: "fail", assertionId: req.assertionId, reason: "defect still reproduces" }));
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-TRIAGE", status: "in-progress" });
|
||||
missionStore._setFeature(feature);
|
||||
missionStore._setAssertions("F-001", [assertionRow({ id: "CA-1", type: "behavioral" })]);
|
||||
taskStore._setTask({ id: "FN-TRIAGE", title: "triage fail", integrationSha: "sha123", log: [] });
|
||||
judgePass(["CA-1"]);
|
||||
// Make triage throw so the swallow path is exercised.
|
||||
missionStore.triageFeature = vi.fn(async () => { throw new Error("triage boom"); }) as any;
|
||||
|
||||
loop = new MissionExecutionLoop({ taskStore: taskStore as any, missionStore: missionStore as any, rootDir: "/tmp", verificationCapability: { verifyBehavioralAssertion: verify } });
|
||||
loop.start();
|
||||
await loop.processTaskOutcome("FN-TRIAGE");
|
||||
|
||||
const triageEvent = (missionStore.logMissionEvent as any).mock.calls.find(
|
||||
(c: any[]) => c[3]?.code === "fix_feature_triage_failed",
|
||||
);
|
||||
expect(triageEvent).toBeDefined();
|
||||
expect(triageEvent[1]).toBe("error");
|
||||
expect(triageEvent[3]?.error).toContain("triage boom");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,376 @@
|
||||
/**
|
||||
* U7 — Recovery / reaper safety falsification audit (R15) + reaper→slice
|
||||
* deadlock regression (the P0).
|
||||
*
|
||||
* The verification run (U3) is the first side-effecting path in a subsystem
|
||||
* whose recovery/reaper logic historically assumed validation was
|
||||
* side-effect-free. This suite *falsifies* (does not merely confirm) that every
|
||||
* site that re-drives validation stays correct now that verification can have
|
||||
* effects. For each re-drive entry point enumerated in
|
||||
* `docs/missions.md` → "## Surface Enumeration", we assert the post-conditions:
|
||||
*
|
||||
* 1. The source tree feeding diff/merge is git-clean after a run (no FS
|
||||
* residue) — enforced here via a verification capability that records every
|
||||
* invocation and asserts its disposable-surface contract, plus the absence
|
||||
* of any board task created by validation.
|
||||
* 2. Zero duplicate Fix Features on re-drive (idempotent on
|
||||
* (sourceFeatureId, runId)).
|
||||
* 3. A terminal verdict (passed / failed / blocked) is reached — never an
|
||||
* indefinitely re-driven `error`.
|
||||
* 4. No `error`-state slice deadlock: a reaped-near-the-bound run does not
|
||||
* strand the slice across a subsequent recovery sweep.
|
||||
*
|
||||
* Re-drive entry points covered (see Surface Enumeration):
|
||||
* - `processTaskOutcome` (normal, task-triggered)
|
||||
* - `recoverActiveMissionValidations` branches:
|
||||
* · validating
|
||||
* · needs_fix + taskId
|
||||
* · implementing + taskId
|
||||
* · stranded done (implementing, no task) — original orphan
|
||||
* · reaped done (needs_fix + error, no task) — the P0 deadlock
|
||||
* - `reapStaleMissionValidatorRuns`
|
||||
*
|
||||
* These tests gate release.
|
||||
*/
|
||||
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
// Mock the AI session layer so validation never spins a real agent. The judge
|
||||
// session is a no-op; the authoritative behavioral verdict comes from the
|
||||
// injected verification capability (see harness). Mirrors the module mocks in
|
||||
// mission-validator-behavioral-posture.test.ts.
|
||||
const mockSessionHolder = {
|
||||
session: { state: { messages: [] as Array<{ role: string; content: string }> }, dispose: vi.fn() },
|
||||
};
|
||||
|
||||
vi.mock("../../pi.js", () => ({
|
||||
createFnAgent: vi.fn(() => Promise.resolve({ session: mockSessionHolder.session })),
|
||||
promptWithFallback: vi.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
vi.mock("../../agent-session-helpers.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../../agent-session-helpers.js")>();
|
||||
return {
|
||||
...actual,
|
||||
createResolvedAgentSession: vi.fn(async () => ({
|
||||
session: mockSessionHolder.session as any,
|
||||
sessionFile: undefined,
|
||||
runtimeId: "test-runtime",
|
||||
wasConfigured: true,
|
||||
})),
|
||||
};
|
||||
});
|
||||
|
||||
import { TaskStore } from "@fusion/core";
|
||||
import { MissionExecutionLoop } from "../../mission-execution-loop.js";
|
||||
import { VALIDATOR_RUN_STALE_MAX_AGE_MS } from "../../self-healing.js";
|
||||
import type {
|
||||
VerificationCapability,
|
||||
VerificationOutcome,
|
||||
VerificationRequest,
|
||||
} from "../../mission-verification.js";
|
||||
|
||||
const STALE_MS = VALIDATOR_RUN_STALE_MAX_AGE_MS;
|
||||
|
||||
/**
|
||||
* A verification capability that records each invocation and returns a scripted
|
||||
* verdict. It also lets us assert that verification was driven (so a re-drive
|
||||
* really reached the verification surface, not a silent no-op).
|
||||
*/
|
||||
function makeCapability(verdict: VerificationOutcome["verdict"], reason = "scripted") {
|
||||
const calls: VerificationRequest[] = [];
|
||||
const cap: VerificationCapability = {
|
||||
verifyBehavioralAssertion: vi.fn(async (request: VerificationRequest) => {
|
||||
calls.push(request);
|
||||
return { verdict, reason, assertionId: request.assertionId } satisfies VerificationOutcome;
|
||||
}),
|
||||
};
|
||||
return { cap, calls };
|
||||
}
|
||||
|
||||
async function createHarness(opts?: {
|
||||
verificationVerdict?: VerificationOutcome["verdict"];
|
||||
}) {
|
||||
const rootDir = await mkdtemp(join(tmpdir(), "fusion-redrive-surface-"));
|
||||
const taskStore = new TaskStore(rootDir, undefined, { inMemoryDb: true });
|
||||
await taskStore.init();
|
||||
const missionStore = taskStore.getMissionStore();
|
||||
|
||||
const { cap, calls } = makeCapability(opts?.verificationVerdict ?? "pass");
|
||||
|
||||
const loop = new MissionExecutionLoop({
|
||||
taskStore,
|
||||
missionStore,
|
||||
rootDir,
|
||||
verificationCapability: cap,
|
||||
});
|
||||
|
||||
// The read-only judge is mocked to a deterministic advisory pass for each
|
||||
// real assertion; the *authoritative* verdict for a behavioral assertion comes
|
||||
// from the injected verification capability via applyBehavioralPosture, so the
|
||||
// judge mock never resolves the verdict by itself. We stub runValidationSession
|
||||
// (the AI session) to a no-op and parseValidationResult to a per-assertion pass
|
||||
// keyed on the actual assertion IDs so the posture's type lookup matches.
|
||||
vi.spyOn(loop as any, "runValidationSession").mockResolvedValue(undefined);
|
||||
vi.spyOn(loop as any, "parseValidationResult").mockImplementation(
|
||||
async (...args: unknown[]) => {
|
||||
const assertions = (args[1] ?? []) as Array<{ id: string }>;
|
||||
return {
|
||||
status: "pass",
|
||||
assertions: assertions.map((a) => ({ assertionId: a.id, passed: true, message: "judge advisory pass" })),
|
||||
summary: "judge advisory pass",
|
||||
};
|
||||
},
|
||||
);
|
||||
// resolveIntegrationSha is called by the posture; stub to a stable value so the
|
||||
// capability receives a resolvable revision (the capability itself is mocked).
|
||||
vi.spyOn(loop as any, "resolveIntegrationSha").mockResolvedValue("integration-sha");
|
||||
|
||||
const ageRun = (runId: string, startedAt: string) => {
|
||||
(missionStore as any).db
|
||||
.prepare("UPDATE mission_validator_runs SET startedAt = ?, updatedAt = ? WHERE id = ?")
|
||||
.run(startedAt, startedAt, runId);
|
||||
};
|
||||
|
||||
/** Build a mission → milestone → slice → behavioral-assertion-linked feature. */
|
||||
const buildFeature = (input: {
|
||||
title: string;
|
||||
withTask?: boolean;
|
||||
taskColumn?: "done" | "archived";
|
||||
}) => {
|
||||
const mission = missionStore.createMission({ title: `${input.title} mission`, autopilotEnabled: true });
|
||||
// A real in-flight mission whose recovery sweep runs is `active`; the sweep
|
||||
// skips non-active missions outright.
|
||||
missionStore.updateMission(mission.id, { status: "active" });
|
||||
const milestone = missionStore.addMilestone(mission.id, { title: `${input.title} ms` });
|
||||
const slice = missionStore.addSlice(milestone.id, { title: `${input.title} slice` });
|
||||
const feature = missionStore.addFeature(slice.id, { title: input.title });
|
||||
const assertion = missionStore.addContractAssertion(milestone.id, {
|
||||
title: `${input.title} assertion`,
|
||||
assertion: `Verify behavior of ${input.title}`,
|
||||
sourceFeatureId: feature.id,
|
||||
type: "behavioral",
|
||||
});
|
||||
missionStore.linkFeatureToAssertion(feature.id, assertion.id);
|
||||
return { mission, milestone, slice, feature: missionStore.getFeature(feature.id)!, assertion };
|
||||
};
|
||||
|
||||
// A real in-flight slice that contains a stranded/reaped done feature is
|
||||
// `active` (sibling work keeps it active); the recovery sweep only visits
|
||||
// active slices. Pin the stored slice status to active AFTER the test has set
|
||||
// up the feature's loop state (updateFeature triggers recomputeSliceStatus,
|
||||
// which would otherwise reset a lone done feature's slice to pending) so the
|
||||
// single-feature fixture faithfully reproduces the in-flight condition.
|
||||
const pinSliceActive = (sliceId: string) => {
|
||||
const db = (missionStore as any).db;
|
||||
db.prepare("UPDATE slices SET status = 'active' WHERE id = ?").run(sliceId);
|
||||
// Re-assert the enclosing mission/milestone as active too: updateFeature →
|
||||
// recomputeSliceStatus can cascade a lone done feature's mission back to
|
||||
// 'planning', and the recovery sweep skips non-active missions/slices.
|
||||
db.prepare("UPDATE missions SET status = 'active' WHERE status != 'archived'").run();
|
||||
};
|
||||
|
||||
const countBoardTasks = async () => (await taskStore.listTasks()).length;
|
||||
|
||||
const countFixFeatures = (sliceId: string) =>
|
||||
missionStore.listFeatures(sliceId).filter((f) => f.generatedFromFeatureId !== undefined).length;
|
||||
|
||||
return {
|
||||
rootDir,
|
||||
taskStore,
|
||||
missionStore,
|
||||
loop,
|
||||
cap,
|
||||
calls,
|
||||
ageRun,
|
||||
buildFeature,
|
||||
pinSliceActive,
|
||||
countBoardTasks,
|
||||
countFixFeatures,
|
||||
cleanup: async () => {
|
||||
loop.stop();
|
||||
taskStore.close();
|
||||
await rm(rootDir, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("U7 reliability: verification re-drive surface enumeration (R15)", () => {
|
||||
let h: Awaited<ReturnType<typeof createHarness>>;
|
||||
|
||||
afterEach(async () => {
|
||||
if (h) await h.cleanup();
|
||||
});
|
||||
|
||||
it("reaper→slice deadlock: a reaped-near-bound run reaches a terminal verdict and does not strand the slice across a recovery sweep (P0)", async () => {
|
||||
h = await createHarness({ verificationVerdict: "pass" });
|
||||
h.loop.start();
|
||||
|
||||
// A validation-only (task-less) done feature with a behavioral assertion.
|
||||
// This is the shape the slice gate refuses to count until validation passes.
|
||||
const { slice, feature } = h.buildFeature({ title: "Slow-but-legit" });
|
||||
// Mark it done first; startValidatorRun (below) flips loopState to
|
||||
// "validating".
|
||||
h.missionStore.updateFeature(feature.id, { status: "done", lastValidatorStatus: null as any });
|
||||
|
||||
// Simulate a slow-but-legitimate verification run that started just inside
|
||||
// the stale window and is NOT owned by the live process (the owner crashed /
|
||||
// restarted): it has no entry in activeValidations. startValidatorRun sets
|
||||
// the feature's loopState to "validating".
|
||||
const run = h.missionStore.startValidatorRun(feature.id, "task_completion");
|
||||
// Age it just past the bound so the reaper treats it as abandoned.
|
||||
h.ageRun(run.id, new Date(Date.now() - STALE_MS - 1000).toISOString());
|
||||
|
||||
// Reaper terminates the run as "error" but, by design, leaves a *done*
|
||||
// feature's loopState untouched (validating) — the exact stranded shape:
|
||||
// run terminal-error, feature stuck "validating", slice gate refuses it.
|
||||
const reaped = await h.loop.reapStaleValidatorRuns(STALE_MS);
|
||||
expect(reaped.reapedCount).toBe(1);
|
||||
expect(h.missionStore.getValidatorRun(run.id)?.status).toBe("error");
|
||||
expect(h.missionStore.getFeature(feature.id)?.loopState).toBe("validating");
|
||||
expect(h.missionStore.getFeature(feature.id)?.lastValidatorStatus ?? null).toBeNull();
|
||||
// Pre-condition: the slice is deadlocked at this point — a "validating" done
|
||||
// feature is never counted complete and carries no taskId to re-drive from.
|
||||
expect(h.missionStore.computeSliceStatus(slice.id)).not.toBe("complete");
|
||||
|
||||
// A subsequent recovery sweep MUST re-drive the reaped task-less done feature
|
||||
// to a terminal verdict instead of leaving it at "error" indefinitely.
|
||||
h.pinSliceActive(slice.id);
|
||||
await h.loop.recoverActiveMissions();
|
||||
|
||||
// Terminal verdict reached (verification passed → feature legitimately done).
|
||||
expect(h.missionStore.getFeature(feature.id)).toMatchObject({
|
||||
loopState: "passed",
|
||||
lastValidatorStatus: "passed",
|
||||
});
|
||||
expect(h.missionStore.computeSliceStatus(slice.id)).toBe("complete");
|
||||
// Verification was actually driven (not a silent no-op).
|
||||
expect(h.calls.length).toBeGreaterThanOrEqual(1);
|
||||
// No board task created by validation/verification (non-mutating board state).
|
||||
expect(await h.countBoardTasks()).toBe(0);
|
||||
// No duplicate Fix Features minted (a pass spawns none).
|
||||
expect(h.countFixFeatures(slice.id)).toBe(0);
|
||||
});
|
||||
|
||||
it("reaped-then-fails reaches a terminal failed verdict (not error) and mints exactly one Fix Feature, idempotent across a second sweep", async () => {
|
||||
h = await createHarness({ verificationVerdict: "fail" });
|
||||
h.loop.start();
|
||||
|
||||
const { slice, feature } = h.buildFeature({ title: "Reaped-fails" });
|
||||
h.missionStore.updateFeature(feature.id, {
|
||||
status: "done",
|
||||
loopState: "implementing",
|
||||
lastValidatorStatus: null as any,
|
||||
});
|
||||
const run = h.missionStore.startValidatorRun(feature.id, "task_completion");
|
||||
h.ageRun(run.id, new Date(Date.now() - STALE_MS - 1000).toISOString());
|
||||
await h.loop.reapStaleValidatorRuns(STALE_MS);
|
||||
|
||||
// First recovery sweep: terminal failed verdict, exactly one Fix Feature.
|
||||
h.pinSliceActive(slice.id);
|
||||
await h.loop.recoverActiveMissions();
|
||||
const after1 = h.missionStore.getFeature(feature.id)!;
|
||||
expect(after1.lastValidatorStatus).toBe("failed");
|
||||
expect(h.countFixFeatures(slice.id)).toBe(1);
|
||||
|
||||
// Exactly one board task exists: the auto-triaged Fix Feature. The
|
||||
// *validation run itself* created no board task — the only board residue is
|
||||
// the legitimate remediation task spawned by the real failed verdict.
|
||||
const boardTasksAfterFail = await h.countBoardTasks();
|
||||
expect(boardTasksAfterFail).toBe(1);
|
||||
|
||||
// Second recovery sweep: the failed feature is no longer task-less-done in a
|
||||
// re-drivable state (it is needs_fix awaiting its Fix Feature), so no duplicate
|
||||
// Fix Feature is minted and no extra board task appears.
|
||||
h.pinSliceActive(slice.id);
|
||||
await h.loop.recoverActiveMissions();
|
||||
expect(h.countFixFeatures(slice.id)).toBe(1);
|
||||
expect(await h.countBoardTasks()).toBe(boardTasksAfterFail);
|
||||
});
|
||||
|
||||
it("processTaskOutcome (normal re-drive) reaches a terminal verdict with no board residue and no duplicate Fix Feature on repeat", async () => {
|
||||
h = await createHarness({ verificationVerdict: "fail" });
|
||||
h.loop.start();
|
||||
|
||||
const { slice, feature } = h.buildFeature({ title: "Normal-path" });
|
||||
// Link a real board task in done so processTaskOutcome can drive validation.
|
||||
const task = await h.taskStore.createTask({
|
||||
id: "FN-NORMAL",
|
||||
title: feature.title,
|
||||
description: "normal path task",
|
||||
column: "done",
|
||||
status: "done",
|
||||
steps: [],
|
||||
} as any);
|
||||
h.missionStore.linkFeatureToTask(feature.id, task.id);
|
||||
h.missionStore.updateFeature(feature.id, { status: "done", loopState: "implementing" });
|
||||
|
||||
await h.loop.processTaskOutcome(task.id);
|
||||
expect(h.missionStore.getFeature(feature.id)?.lastValidatorStatus).toBe("failed");
|
||||
const fixCount = h.countFixFeatures(slice.id);
|
||||
expect(fixCount).toBe(1);
|
||||
|
||||
// Re-driving the same outcome must not duplicate the Fix Feature.
|
||||
await h.loop.processTaskOutcome(task.id);
|
||||
expect(h.countFixFeatures(slice.id)).toBe(fixCount);
|
||||
});
|
||||
|
||||
it("recovery re-drives a task-less done feature stranded in 'validating' (the reaped loopState) to a terminal verdict, no error stranding, no board residue", async () => {
|
||||
h = await createHarness({ verificationVerdict: "pass" });
|
||||
h.loop.start();
|
||||
|
||||
// A done, task-less feature stranded in loopState="validating" — the exact
|
||||
// shape MissionStore.reapValidatorRun leaves a *done* feature in after it
|
||||
// terminates the stale run (its shouldUpdateFeature guard skips done
|
||||
// features, so the feature keeps the "validating" loopState set by
|
||||
// startValidatorRun). computeSliceStatus never counts "validating", and the
|
||||
// recovery 'validating' branch only re-drives features that carry a taskId —
|
||||
// so without the stranded-done catch-all this would deadlock the slice.
|
||||
const { slice, feature } = h.buildFeature({ title: "Validating-stranded" });
|
||||
h.missionStore.updateFeature(feature.id, { status: "done", lastValidatorStatus: null as any });
|
||||
const run = h.missionStore.startValidatorRun(feature.id, "task_completion");
|
||||
expect(h.missionStore.getFeature(feature.id)?.loopState).toBe("validating");
|
||||
h.ageRun(run.id, new Date(Date.now() - STALE_MS - 1000).toISOString());
|
||||
await h.loop.reapStaleValidatorRuns(STALE_MS);
|
||||
expect(h.missionStore.getFeature(feature.id)?.loopState).toBe("validating");
|
||||
|
||||
h.pinSliceActive(slice.id);
|
||||
await h.loop.recoverActiveMissions();
|
||||
|
||||
expect(h.missionStore.getFeature(feature.id)?.lastValidatorStatus).toBe("passed");
|
||||
expect(h.missionStore.computeSliceStatus(slice.id)).toBe("complete");
|
||||
expect(await h.countBoardTasks()).toBe(0); // validation/verification created no board task
|
||||
expect(h.countFixFeatures(slice.id)).toBe(0);
|
||||
});
|
||||
|
||||
it("inconclusive verification across recovery re-drives never deadlocks the slice at error and spawns no Fix Feature (R20/R21)", async () => {
|
||||
h = await createHarness({ verificationVerdict: "inconclusive" });
|
||||
h.loop.start();
|
||||
|
||||
const { slice, feature } = h.buildFeature({ title: "Flaky" });
|
||||
h.missionStore.updateFeature(feature.id, {
|
||||
status: "done",
|
||||
loopState: "implementing",
|
||||
lastValidatorStatus: null as any,
|
||||
});
|
||||
|
||||
h.pinSliceActive(slice.id);
|
||||
await h.loop.recoverActiveMissions();
|
||||
|
||||
const after = h.missionStore.getFeature(feature.id)!;
|
||||
// Inconclusive routes to a terminal blocked verdict — NOT error, NOT a
|
||||
// default pass — and spawns no remediation.
|
||||
expect(after.lastValidatorStatus).toBe("blocked");
|
||||
expect(after.lastValidatorStatus).not.toBe("error");
|
||||
expect(h.countFixFeatures(slice.id)).toBe(0);
|
||||
expect(await h.countBoardTasks()).toBe(0);
|
||||
|
||||
// A subsequent sweep does not re-drive a blocked feature into churn.
|
||||
const callsBefore = h.calls.length;
|
||||
await h.loop.recoverActiveMissions();
|
||||
expect(h.calls.length).toBe(callsBefore);
|
||||
});
|
||||
});
|
||||
@@ -22,6 +22,8 @@ import type {
|
||||
Settings,
|
||||
Milestone,
|
||||
} from "@fusion/core";
|
||||
import { normalizeMissionAssertionType } from "@fusion/core";
|
||||
import type { VerificationOutcome } from "./mission-verification.js";
|
||||
import { createFnAgent, promptWithFallback, type AgentResult } from "./pi.js";
|
||||
import { mergeEffectiveSettings } from "./effective-settings.js";
|
||||
import {
|
||||
@@ -45,8 +47,16 @@ const VALIDATION_TIMEOUT_MS = 10 * 60 * 1000; // 10 minutes
|
||||
* per assertion plus an overall status.
|
||||
*/
|
||||
export interface ValidationResult {
|
||||
/** Overall validation status */
|
||||
status: "pass" | "fail" | "blocked" | "error";
|
||||
/**
|
||||
* Overall validation status.
|
||||
*
|
||||
* `inconclusive` is first-class and distinct from `fail`: it means a
|
||||
* behavioral verification run could not run or conclude (no isolating sandbox
|
||||
* backend, timeout, setup failure, rejected proof). In this unit it routes to
|
||||
* a blocked verdict (no remediation); later units track its infra-failure rate
|
||||
* separately.
|
||||
*/
|
||||
status: "pass" | "fail" | "blocked" | "error" | "inconclusive";
|
||||
/** Per-assertion results */
|
||||
assertions: Array<{
|
||||
assertionId: string;
|
||||
@@ -78,6 +88,14 @@ export interface MissionExecutionLoopOptions {
|
||||
pluginRunner?: import("./plugin-runner.js").PluginRunner;
|
||||
/** Optional agent store for resolving assigned-agent runtime hints. */
|
||||
agentStore?: AgentStore;
|
||||
/**
|
||||
* Optional behavioral-verification capability (U3). When provided, behavioral
|
||||
* assertions are confirmed by a non-mutating verification run; the judge's
|
||||
* "pass" on a behavioral assertion is advisory only. When ABSENT, behavioral
|
||||
* assertions still default to fail (U2) but no verification run is attempted —
|
||||
* preserving the behavior of existing construction sites that inject nothing.
|
||||
*/
|
||||
verificationCapability?: import("./mission-verification.js").VerificationCapability;
|
||||
}
|
||||
|
||||
export class MissionExecutionLoop extends EventEmitter {
|
||||
@@ -89,6 +107,7 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
private missionAutopilot?: MissionExecutionLoopOptions["missionAutopilot"];
|
||||
private pluginRunner?: MissionExecutionLoopOptions["pluginRunner"];
|
||||
private agentStore?: MissionExecutionLoopOptions["agentStore"];
|
||||
private verificationCapability?: MissionExecutionLoopOptions["verificationCapability"];
|
||||
private activeValidations = new Set<string>(); // feature IDs currently being validated
|
||||
|
||||
constructor(options: MissionExecutionLoopOptions) {
|
||||
@@ -100,6 +119,7 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
this.missionAutopilot = options.missionAutopilot;
|
||||
this.pluginRunner = options.pluginRunner;
|
||||
this.agentStore = options.agentStore;
|
||||
this.verificationCapability = options.verificationCapability;
|
||||
loopLog.log("MissionExecutionLoop created");
|
||||
}
|
||||
|
||||
@@ -290,18 +310,50 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
// Features marked "done" but stranded in "implementing" with no
|
||||
// linked task can never validate on their own: the branches above
|
||||
// only re-drive features that still carry a taskId. Meanwhile the
|
||||
// slice-completion gate (MissionStore.computeSliceStatus) refuses
|
||||
// to count an assertion-linked "done" feature until its validator
|
||||
// passes — so the slice, milestone, and mission can never
|
||||
// auto-progress. Re-drive validation directly so the gate can
|
||||
// resolve. Validation is a read-only judge (no board task, no code
|
||||
// changes); on pass the feature becomes legitimately complete, on
|
||||
// fail the normal fix-feature flow takes over.
|
||||
// Features marked "done" but stranded with no linked task can never
|
||||
// validate on their own: the branches above only re-drive features
|
||||
// that still carry a taskId. Meanwhile the slice-completion gate
|
||||
// (MissionStore.computeSliceStatus) refuses to count an
|
||||
// assertion-linked "done" feature until its validator passes — so
|
||||
// the slice, milestone, and mission can never auto-progress.
|
||||
//
|
||||
// Several ways a task-less done feature lands stranded here:
|
||||
// 1. loopState="implementing" + null lastValidatorStatus — the
|
||||
// original stranded-orphan case (FN-5715 / the autopilot-stall
|
||||
// learning): validation was never driven.
|
||||
// 2. loopState="validating" + null lastValidatorStatus — a
|
||||
// *reaped* run. `startValidatorRun` flips the feature to
|
||||
// "validating"; `MissionStore.reapValidatorRun` resolves the
|
||||
// stale run to status="error" but, by design, leaves a *done*
|
||||
// feature's loopState untouched (its `shouldUpdateFeature`
|
||||
// guard skips done features). So a reaped validation-only
|
||||
// feature (no board task) is left "validating" forever: the
|
||||
// "validating" branch above only re-drives features that carry
|
||||
// a taskId, and `computeSliceStatus` never counts a "validating"
|
||||
// done feature — the U7 reaper→slice deadlock (P0).
|
||||
// 3. loopState="needs_fix" + lastValidatorStatus="error" — a
|
||||
// reaped run on a *non-done* feature that later moved to done,
|
||||
// or a reaped manual run; "error" is likewise never accepted by
|
||||
// computeSliceStatus and the needs_fix branch above only
|
||||
// re-drives features with a taskId.
|
||||
//
|
||||
// The common shape is: a task-less, done, assertion-linked feature
|
||||
// that has not reached a *passed* validator status and is not
|
||||
// currently being validated. Re-drive it directly regardless of the
|
||||
// exact stranded loopState so it reaches a terminal verdict instead
|
||||
// of livelocking on "validating"/"error".
|
||||
//
|
||||
// Validation is bounded (verification wall-clock is provably under
|
||||
// the reaper stale window — see VALIDATOR_RUN_STALE_MAX_AGE_MS vs the
|
||||
// aggregate verification timeout) and non-mutating: on pass the
|
||||
// feature becomes legitimately complete; on fail the normal
|
||||
// fix-feature flow takes over; on inconclusive it routes to
|
||||
// needs-attention without minting remediation. Either way the
|
||||
// feature reaches a terminal verdict rather than re-driving forever.
|
||||
if (
|
||||
feature.loopState === "implementing"
|
||||
(feature.loopState === "implementing"
|
||||
|| feature.loopState === "validating"
|
||||
|| (feature.loopState === "needs_fix" && feature.lastValidatorStatus === "error"))
|
||||
&& !feature.taskId
|
||||
&& feature.status === "done"
|
||||
&& feature.lastValidatorStatus !== "passed"
|
||||
@@ -311,6 +363,7 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
if (
|
||||
currentFeature.loopState === "passed"
|
||||
|| currentFeature.lastValidatorStatus === "passed"
|
||||
|| this.activeValidations.has(feature.id)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
@@ -429,8 +482,15 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
await this.handleValidationPass(feature.id, run.id, result.summary);
|
||||
} else if (result.status === "fail") {
|
||||
await this.handleValidationFail(feature.id, run.id, result);
|
||||
} else if (result.status === "inconclusive") {
|
||||
// R21 — "verification could not run" is distinct from "behavior observed
|
||||
// wrong". An infra-driven inconclusive (no isolating backend, timeout,
|
||||
// isolation setup failure, rejected proof) routes to a blocked/needs-
|
||||
// attention outcome that spawns NO Fix Feature, and is tracked with a
|
||||
// distinguishable infra-failure event so it is separable from real fails.
|
||||
await this.handleValidationInconclusive(feature.id, run.id, result.blockedReason ?? result.summary);
|
||||
} else if (result.status === "blocked") {
|
||||
await this.handleValidationBlocked(feature.id, run.id, result.blockedReason);
|
||||
await this.handleValidationBlocked(feature.id, run.id, result.blockedReason ?? result.summary);
|
||||
} else if (result.status === "error") {
|
||||
await this.handleValidationError(feature.id, run.id, result.summary);
|
||||
}
|
||||
@@ -522,17 +582,28 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
loopLog.log(`Validation session created for feature ${feature.id}`);
|
||||
|
||||
// Run the validation with timeout
|
||||
let timeoutHandle: ReturnType<typeof setTimeout> | undefined;
|
||||
const timeoutPromise = new Promise<never>((_, reject) => {
|
||||
setTimeout(() => reject(new Error("Validation timeout")), VALIDATION_TIMEOUT_MS);
|
||||
timeoutHandle = setTimeout(() => reject(new Error("Validation timeout")), VALIDATION_TIMEOUT_MS);
|
||||
});
|
||||
|
||||
const validationPromise = this.runValidationSession(session.session, prompt);
|
||||
|
||||
await Promise.race([validationPromise, timeoutPromise]);
|
||||
try {
|
||||
await Promise.race([validationPromise, timeoutPromise]);
|
||||
} finally {
|
||||
// Always clear the timer so it does not stay armed across validations.
|
||||
if (timeoutHandle) clearTimeout(timeoutHandle);
|
||||
}
|
||||
|
||||
// Get the validation result from the session
|
||||
// The agent should have returned structured JSON in its response
|
||||
const result = await this.parseValidationResult(session.session, assertions);
|
||||
const judgeResult = await this.parseValidationResult(session.session, assertions);
|
||||
|
||||
// U2/U3: the read-only judge's verdict is authoritative for STATIC
|
||||
// assertions only. BEHAVIORAL assertions default to fail and are confirmed
|
||||
// (or refuted) by a non-mutating verification run instead.
|
||||
const result = await this.applyBehavioralPosture(feature, assertions, judgeResult);
|
||||
|
||||
loopLog.log(`Validation completed for feature ${feature.id}: ${result.status}`);
|
||||
return result;
|
||||
@@ -563,6 +634,161 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the behavioral judging posture (U2/U3) to the read-only judge's
|
||||
* verdict.
|
||||
*
|
||||
* - STATIC assertions keep the judge's verdict verbatim (no behavior change).
|
||||
* - BEHAVIORAL assertions DEFAULT TO FAIL. The judge's "pass" on a behavioral
|
||||
* assertion is advisory; an authoritative pass requires a verification run
|
||||
* to confirm it. When a verification capability is injected, each behavioral
|
||||
* assertion is run through it: pass → satisfied; fail → behavioral failure;
|
||||
* inconclusive → the aggregate becomes inconclusive (infra, no remediation).
|
||||
* When NO capability is injected, behavioral assertions simply stay failed
|
||||
* (preserving existing call-site behavior — existing data is all static).
|
||||
*
|
||||
* The aggregate status is recomputed from the post-posture per-assertion
|
||||
* results so the existing pass/fail/blocked/error/inconclusive flow is driven
|
||||
* correctly.
|
||||
*/
|
||||
private async applyBehavioralPosture(
|
||||
feature: MissionFeature,
|
||||
assertions: MissionContractAssertion[],
|
||||
judgeResult: ValidationResult,
|
||||
): Promise<ValidationResult> {
|
||||
// Preserve non-behavioral terminal verdicts untouched (error/blocked from the
|
||||
// judge are not behavioral posture concerns). A "blocked" verdict must short-
|
||||
// circuit too: otherwise it falls through to the aggregate recompute below,
|
||||
// which would rewrite it to "fail" and incorrectly route to a Fix Feature
|
||||
// instead of handleValidationBlocked.
|
||||
if (judgeResult.status === "error" || judgeResult.status === "blocked") {
|
||||
return judgeResult;
|
||||
}
|
||||
|
||||
const typeById = new Map<string, ReturnType<typeof normalizeMissionAssertionType>>();
|
||||
let hasBehavioral = false;
|
||||
for (const a of assertions) {
|
||||
const t = normalizeMissionAssertionType(a.type);
|
||||
typeById.set(a.id, t);
|
||||
if (t === "behavioral") hasBehavioral = true;
|
||||
}
|
||||
|
||||
// Fast path: no behavioral assertions → existing static path is preserved
|
||||
// exactly. This keeps every existing (untyped/static) test green.
|
||||
if (!hasBehavioral) {
|
||||
return judgeResult;
|
||||
}
|
||||
|
||||
const textById = new Map(assertions.map((a) => [a.id, a.assertion]));
|
||||
let sawInconclusive = false;
|
||||
let inconclusiveReason: string | undefined;
|
||||
|
||||
const newAssertionResults = await Promise.all(
|
||||
judgeResult.assertions.map(async (judged) => {
|
||||
const type = typeById.get(judged.assertionId) ?? "static";
|
||||
if (type !== "behavioral") {
|
||||
// Static: keep judge verdict verbatim.
|
||||
return judged;
|
||||
}
|
||||
|
||||
// Behavioral: default to fail unless verification confirms it.
|
||||
if (!this.verificationCapability) {
|
||||
return {
|
||||
...judged,
|
||||
passed: false,
|
||||
message: "Behavioral assertion defaults to fail: no verification evidence (advisory judge verdict is not authoritative).",
|
||||
expected: judged.expected ?? "Behavior confirmed by a verification run",
|
||||
actual: judged.actual ?? "No verification run was performed",
|
||||
};
|
||||
}
|
||||
|
||||
let outcome: VerificationOutcome;
|
||||
try {
|
||||
outcome = await this.verificationCapability.verifyBehavioralAssertion({
|
||||
assertionId: judged.assertionId,
|
||||
assertion: textById.get(judged.assertionId) ?? "",
|
||||
taskId: feature.taskId,
|
||||
integrationSha: await this.resolveIntegrationSha(feature),
|
||||
signal: undefined,
|
||||
});
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
loopLog.warn(`Verification capability threw for assertion ${judged.assertionId}: ${message}`);
|
||||
outcome = { verdict: "inconclusive", assertionId: judged.assertionId, reason: `verification error: ${message}` };
|
||||
}
|
||||
|
||||
if (outcome.verdict === "pass") {
|
||||
return { ...judged, passed: true, message: outcome.reason };
|
||||
}
|
||||
if (outcome.verdict === "inconclusive") {
|
||||
sawInconclusive = true;
|
||||
inconclusiveReason = inconclusiveReason ?? outcome.reason;
|
||||
return {
|
||||
...judged,
|
||||
passed: false,
|
||||
message: `Behavioral verification inconclusive: ${outcome.reason}`,
|
||||
expected: judged.expected ?? "Behavior confirmed by a verification run",
|
||||
actual: outcome.detail ?? "Verification could not conclude",
|
||||
};
|
||||
}
|
||||
// fail
|
||||
return {
|
||||
...judged,
|
||||
passed: false,
|
||||
message: outcome.reason,
|
||||
expected: judged.expected ?? "Behavior confirmed by a verification run",
|
||||
actual: outcome.detail ?? judged.actual ?? "Behavior not confirmed",
|
||||
};
|
||||
}),
|
||||
);
|
||||
|
||||
const allPassed = newAssertionResults.every((a) => a.passed);
|
||||
|
||||
// Inconclusive takes precedence over fail: an infra-driven non-pass must not
|
||||
// be mistaken for an observed behavioral failure (no Fix Feature).
|
||||
let status: ValidationResult["status"];
|
||||
if (sawInconclusive && !allPassed) {
|
||||
status = "inconclusive";
|
||||
} else if (allPassed) {
|
||||
status = "pass";
|
||||
} else {
|
||||
status = "fail";
|
||||
}
|
||||
|
||||
const summary = status === "pass"
|
||||
? judgeResult.summary
|
||||
: status === "inconclusive"
|
||||
? `Behavioral verification inconclusive: ${inconclusiveReason ?? "verification could not conclude"}`
|
||||
: "One or more behavioral assertions were not confirmed by verification.";
|
||||
|
||||
return {
|
||||
status,
|
||||
assertions: newAssertionResults,
|
||||
summary,
|
||||
blockedReason: status === "inconclusive" ? (inconclusiveReason ?? "verification inconclusive") : judgeResult.blockedReason,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the trusted revision (integration SHA) whose disposable checkout the
|
||||
* verification run executes against. The live task worktree is pruned before
|
||||
* the done-transition that triggers validation, so it cannot be used.
|
||||
*
|
||||
* In this unit we read it from the linked task when available; callers that do
|
||||
* not supply a resolvable SHA cause the verification run to resolve to
|
||||
* inconclusive (fail-closed). A richer derivation is owned by a later unit.
|
||||
*/
|
||||
private async resolveIntegrationSha(feature: MissionFeature): Promise<string | undefined> {
|
||||
if (!feature.taskId) return undefined;
|
||||
try {
|
||||
const task = await this.taskStore.getTask(feature.taskId);
|
||||
const candidate = (task as { integrationSha?: string; baseCommit?: string } | undefined);
|
||||
return candidate?.integrationSha ?? candidate?.baseCommit ?? undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
private resolveValidationSessionModel(
|
||||
task: Awaited<ReturnType<TaskStore["getTask"]>> | null,
|
||||
settings: Partial<Settings> | undefined,
|
||||
@@ -796,10 +1022,15 @@ export class MissionExecutionLoop extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
// If no assertion results but we have assertions, create default results based on status
|
||||
if (results.length === 0 && assertions.length > 0) {
|
||||
// Backfill any linked assertions the judge omitted from its response. A
|
||||
// partial judge response must not silently drop assertions: every linked
|
||||
// assertion needs a result so behavioral assertions still reach
|
||||
// verifyBehavioralAssertion and the aggregate is computed over the full set.
|
||||
if (assertions.length > 0) {
|
||||
const seen = new Set(results.map((r) => r.assertionId));
|
||||
const overallPassed = parsed.status === "pass";
|
||||
for (const assertion of assertions) {
|
||||
if (seen.has(assertion.id)) continue;
|
||||
results.push({
|
||||
assertionId: assertion.id,
|
||||
passed: overallPassed,
|
||||
@@ -1016,6 +1247,10 @@ ${taskContext ? `\n\nImplementation context:\n${taskContext}` : ""}`;
|
||||
runId: string | undefined,
|
||||
result: ValidationResult,
|
||||
): Promise<void> {
|
||||
// Tracks how autopilot should be notified. A retry-budget-exhausted feature
|
||||
// transitions to blocked, so autopilot must be told "blocked" (not "failed")
|
||||
// to stay in sync with the validator-run state.
|
||||
let terminalStatus: "failed" | "blocked" = "failed";
|
||||
try {
|
||||
// Record the failures
|
||||
const failures = result.assertions
|
||||
@@ -1040,12 +1275,26 @@ ${taskContext ? `\n\nImplementation context:\n${taskContext}` : ""}`;
|
||||
|
||||
loopLog.log(`Feature ${featureId} failed validation with ${failures.length} failures`);
|
||||
|
||||
// R6 — build an observed-vs-expected reason so the remediation agent sees
|
||||
// what behavior was wrong, not just which assertion ids failed.
|
||||
const failureReason = this.buildFailureReason(failures, result.summary);
|
||||
|
||||
// R16 — durable observability: a verification/validation failure is a
|
||||
// persisted mission event, not just a log line.
|
||||
this.logFeatureMissionEvent(featureId, "error", "validation_failed", `Validation failed for feature ${featureId}: ${result.summary}`, {
|
||||
runId: runId ?? null,
|
||||
failedAssertionIds: failures.map((f) => f.assertionId),
|
||||
reason: failureReason,
|
||||
outcome: "fail",
|
||||
});
|
||||
|
||||
// Create fix feature
|
||||
try {
|
||||
const fixFeature = this.missionStore.createGeneratedFixFeature(
|
||||
featureId,
|
||||
runId || "unknown",
|
||||
failures.map((f) => f.assertionId),
|
||||
failureReason,
|
||||
);
|
||||
loopLog.log(`Created fix feature ${fixFeature.id} for ${featureId}`);
|
||||
|
||||
@@ -1056,7 +1305,15 @@ ${taskContext ? `\n\nImplementation context:\n${taskContext}` : ""}`;
|
||||
} catch (triageErr) {
|
||||
const triageMessage = triageErr instanceof Error ? triageErr.message : String(triageErr);
|
||||
loopLog.error(`Error triaging fix feature ${fixFeature.id}:`, triageMessage);
|
||||
// Continue even if triage fails - the fix feature was created and can be triaged manually
|
||||
// R16 — a swallowed triage error must be durably recorded, not just
|
||||
// logged. The branch-group-collision learning: silent triage stalls
|
||||
// are invisible mission deadlocks. The Fix Feature was created and can
|
||||
// be triaged manually, so we continue, but the failure is persisted.
|
||||
this.logFeatureMissionEvent(featureId, "error", "fix_feature_triage_failed", `Auto-triage of fix feature ${fixFeature.id} failed: ${triageMessage}`, {
|
||||
runId: runId ?? null,
|
||||
fixFeatureId: fixFeature.id,
|
||||
error: triageMessage,
|
||||
});
|
||||
}
|
||||
|
||||
this.emit("validation:failed", {
|
||||
@@ -1067,24 +1324,105 @@ ${taskContext ? `\n\nImplementation context:\n${taskContext}` : ""}`;
|
||||
});
|
||||
} catch (fixErr) {
|
||||
const message = fixErr instanceof Error ? fixErr.message : String(fixErr);
|
||||
if (message.includes("retry budget exhausted")) {
|
||||
if (message.includes("retry budget exhausted") || message.includes("exhausted its retry budget")) {
|
||||
loopLog.warn(`Feature ${featureId} retry budget exhausted; marking as blocked`);
|
||||
// completeValidatorRun already handles the blocked transition when budget is exhausted
|
||||
terminalStatus = "blocked";
|
||||
this.logFeatureMissionEvent(featureId, "error", "retry_budget_exhausted", `Feature ${featureId} exhausted its retry budget`, {
|
||||
runId: runId ?? null,
|
||||
});
|
||||
this.emit("validation:budget_exhausted", { featureId, runId });
|
||||
} else {
|
||||
loopLog.error(`Error creating fix feature for ${featureId}:`, message);
|
||||
// R16 — a swallowed Fix-Feature creation error is durably recorded.
|
||||
this.logFeatureMissionEvent(featureId, "error", "fix_feature_creation_failed", `Failed to create fix feature for ${featureId}: ${message}`, {
|
||||
runId: runId ?? null,
|
||||
error: message,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Notify autopilot if configured
|
||||
if (this.missionAutopilot?.notifyValidationComplete) {
|
||||
await this.missionAutopilot.notifyValidationComplete(featureId, "failed");
|
||||
await this.missionAutopilot.notifyValidationComplete(featureId, terminalStatus);
|
||||
}
|
||||
} catch (err) {
|
||||
loopLog.error(`Error handling validation fail for ${featureId}:`, err);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build an observed-vs-expected failure reason (R6) suitable for surfacing to
|
||||
* the remediation agent in the generated Fix Feature. Prefers per-assertion
|
||||
* expected/actual detail; falls back to the per-assertion message, then the
|
||||
* overall summary.
|
||||
*/
|
||||
private buildFailureReason(
|
||||
failures: Array<{ assertionId: string; message: string; expected?: string; actual?: string }>,
|
||||
summary: string,
|
||||
): string {
|
||||
if (failures.length === 0) {
|
||||
return summary;
|
||||
}
|
||||
const lines = failures.map((f) => {
|
||||
const parts: string[] = [`- ${f.assertionId}: ${f.message}`];
|
||||
if (f.expected) parts.push(` expected: ${f.expected}`);
|
||||
if (f.actual) parts.push(` observed: ${f.actual}`);
|
||||
return parts.join("\n");
|
||||
});
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle an inconclusive validation (R21).
|
||||
*
|
||||
* An inconclusive verdict means verification could not run or could not
|
||||
* conclude (no isolating sandbox backend, timeout, isolation setup failure,
|
||||
* rejected proof, detected flakiness) — it is NOT an observed behavioral
|
||||
* failure. It must:
|
||||
* - route to a blocked/needs-attention outcome (no Fix Feature, no
|
||||
* remediation work minted),
|
||||
* - record a distinguishable, durably-observable infra-failure signal so the
|
||||
* infra-failure rate is separable from real failures.
|
||||
*
|
||||
* The validator run is completed as `blocked` (no new run status is
|
||||
* introduced), but the persisted mission event carries a distinct
|
||||
* `verification_inconclusive` code and an `outcome: "inconclusive"` marker so
|
||||
* downstream observers can compute the infra-failure rate distinctly from real
|
||||
* fails (which carry `outcome: "fail"`).
|
||||
*/
|
||||
private async handleValidationInconclusive(
|
||||
featureId: string,
|
||||
runId: string | undefined,
|
||||
reason: string | undefined,
|
||||
): Promise<void> {
|
||||
try {
|
||||
this.completeValidatorRunIfStillRunning(runId, "blocked", reason);
|
||||
loopLog.warn(`Feature ${featureId} verification inconclusive: ${reason ?? "no reason provided"}`);
|
||||
|
||||
// R16/R21 — durable, distinguishable infra-failure event. The `outcome`
|
||||
// marker separates infra-driven non-passes from real behavioral fails so
|
||||
// the infra-failure rate can be tracked without conflating the two.
|
||||
this.logFeatureMissionEvent(featureId, "warning", "verification_inconclusive", `Verification inconclusive for feature ${featureId}: ${reason ?? "verification could not conclude"}`, {
|
||||
runId: runId ?? null,
|
||||
reason: reason ?? null,
|
||||
outcome: "inconclusive",
|
||||
infraFailure: true,
|
||||
});
|
||||
|
||||
// Explicitly does NOT call createGeneratedFixFeature — inconclusive mints
|
||||
// no remediation work (R21).
|
||||
|
||||
if (this.missionAutopilot?.notifyValidationComplete) {
|
||||
await this.missionAutopilot.notifyValidationComplete(featureId, "blocked");
|
||||
}
|
||||
|
||||
this.emit("validation:inconclusive", { featureId, runId, reason });
|
||||
} catch (err) {
|
||||
loopLog.error(`Error handling inconclusive validation for ${featureId}:`, err);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle a blocked validation.
|
||||
*/
|
||||
|
||||
357
packages/engine/src/mission-verification-app-harness.ts
Normal file
357
packages/engine/src/mission-verification-app-harness.ts
Normal file
@@ -0,0 +1,357 @@
|
||||
/**
|
||||
* Isolated-app-launch harness (U4).
|
||||
*
|
||||
* Stands up an ISOLATED instance of the Fusion app/dashboard for a verification
|
||||
* run, and tears it down unconditionally. This is the surface the app/browser
|
||||
* driver (U8) will drive; it owns the isolation/launch/teardown contracts so
|
||||
* those can be characterized and trusted *before* any navigation logic exists.
|
||||
*
|
||||
* It does NOT navigate, interact with, or observe the app (that is U8), and it
|
||||
* does NOT feed any verdict path (that is U5). It only launches + tears down.
|
||||
*
|
||||
* Safety contracts enforced here (R13) — the boundary, not a convention:
|
||||
* (port-4040-allowlist: the "4040" references below document the reserved-port
|
||||
* guard contract; this harness never kills or binds the live dashboard port.)
|
||||
* - **Non-reserved port.** The bound port is never a reserved dashboard port:
|
||||
* 4040 is always reserved, plus anything in `FUSION_RESERVED_PORTS` / `PORT` /
|
||||
* `FUSION_SERVER_PORT` (mirrors the repo's existing port-4040 guards in
|
||||
* `scripts/boot-smoke.mjs` / `check-no-kill-4040.mjs` and
|
||||
* `dev-server-port-detect.ts`).
|
||||
* - **Disposable DB.** A fresh, empty DB created under a run-unique tmpdir —
|
||||
* NEVER the shared central DB, never a copy of it, no credentials, no agent
|
||||
* logs seeded.
|
||||
* - **Fresh bundle.** A freshly-built client bundle is served so verification
|
||||
* cannot produce its own false verdicts from a stale `dist/client` (the trap
|
||||
* documented in
|
||||
* `docs/solutions/developer-experience/browser-testing-dashboard-from-worktree-safely.md`).
|
||||
* - **Unconditional teardown.** On crash / timeout / normal disposal the
|
||||
* process is killed, the port is freed, and the tmpdir/DB are removed.
|
||||
*
|
||||
* Process-spawn and bundle-build are INJECTABLE so unit tests can characterize
|
||||
* the isolation contracts without launching a full server. A real end-to-end
|
||||
* launch belongs in a heavier lane / manual smoke (see `scripts/boot-smoke.mjs`,
|
||||
* which is the closest existing real-boot pattern), not in the merge-gate unit
|
||||
* tests.
|
||||
*/
|
||||
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { createServer as createNetServer } from "node:net";
|
||||
import { createLogger } from "./logger.js";
|
||||
|
||||
const harnessLog = createLogger("mission-verify-app");
|
||||
|
||||
// ── Reserved-port policy (mirrors the repo's port-4040 guards) ────────────────
|
||||
|
||||
/**
|
||||
* Parse a comma-separated port list (matching the parsing in
|
||||
* `scripts/boot-smoke.mjs` and `@fusion/core/__test-utils__/port-probe-policy`).
|
||||
*/
|
||||
export function parsePortList(value: string | undefined): number[] {
|
||||
if (!value) return [];
|
||||
return value
|
||||
.split(",")
|
||||
.map((part) => Number.parseInt(part.trim(), 10))
|
||||
.filter((port) => Number.isInteger(port) && port > 0 && port < 65_536);
|
||||
}
|
||||
|
||||
/**
|
||||
* The reserved-port set derived from the environment. 4040 (the user's live
|
||||
* dashboard) is ALWAYS reserved; `FUSION_RESERVED_PORTS` / `PORT` /
|
||||
* `FUSION_SERVER_PORT` add more. Mirrors `resolveReservedPortsFromEnv` in
|
||||
* `@fusion/core/__test-utils__/port-probe-policy` (which is a test-only path and
|
||||
* not part of the public package surface, so it is re-derived here rather than
|
||||
* imported).
|
||||
*/
|
||||
export function resolveReservedPorts(env: NodeJS.ProcessEnv = process.env): Set<number> {
|
||||
const reserved = new Set<number>([4040]);
|
||||
for (const port of parsePortList(env.FUSION_RESERVED_PORTS)) reserved.add(port);
|
||||
for (const port of parsePortList(env.PORT)) reserved.add(port);
|
||||
for (const port of parsePortList(env.FUSION_SERVER_PORT)) reserved.add(port);
|
||||
return reserved;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire an OS-assigned ephemeral port that is NOT in the reserved set. Mirrors
|
||||
* `getEphemeralPort` in `scripts/boot-smoke.mjs`: bind `listen(0)`, read the
|
||||
* assigned port, release it, and reject reserved ports before returning.
|
||||
*
|
||||
* Note this is a best-effort claim (the OS could hand the port to someone else
|
||||
* between release and the server's bind — the TOCTOU window boot-smoke retries
|
||||
* around). The launcher receives the port and is expected to retry on
|
||||
* EADDRINUSE; the contract this enforces is "never a reserved port".
|
||||
*/
|
||||
export async function acquireNonReservedPort(
|
||||
reserved: Set<number>,
|
||||
attempts = 10,
|
||||
): Promise<number> {
|
||||
for (let attempt = 0; attempt < attempts; attempt += 1) {
|
||||
const port = await new Promise<number>((resolve, reject) => {
|
||||
const srv = createNetServer();
|
||||
srv.once("error", reject);
|
||||
srv.listen(0, "127.0.0.1", () => {
|
||||
const address = srv.address();
|
||||
if (address && typeof address === "object") {
|
||||
const { port: assigned } = address;
|
||||
srv.close(() => resolve(assigned));
|
||||
} else {
|
||||
srv.close(() => reject(new Error("could not read ephemeral port from socket")));
|
||||
}
|
||||
});
|
||||
});
|
||||
if (!reserved.has(port)) return port;
|
||||
}
|
||||
throw new Error("could not obtain a non-reserved ephemeral port");
|
||||
}
|
||||
|
||||
// ── Disposable DB (R13) ───────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* A disposable, run-unique workspace for an isolated app instance: a fresh DB
|
||||
* path under a tmpdir, never the central DB.
|
||||
*/
|
||||
export interface DisposableAppWorkspace {
|
||||
/** Run-unique tmpdir root. */
|
||||
tmpDir: string;
|
||||
/** Path to the fresh, empty disposable DB inside `tmpDir`. */
|
||||
dbPath: string;
|
||||
/** Remove the tmpdir (and the DB inside it) unconditionally. Idempotent. */
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a run-unique tmpdir containing a fresh, empty DB file path. The DB is
|
||||
* created empty (no schema, no rows) — never copied from the central DB and
|
||||
* never seeded with credentials or agent logs (R13). The launched app is
|
||||
* responsible for initializing its own schema in this empty file.
|
||||
*/
|
||||
export async function createDisposableAppWorkspace(): Promise<DisposableAppWorkspace> {
|
||||
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "fn-verify-app-"));
|
||||
const dbPath = path.join(tmpDir, "fusion-verify.db");
|
||||
// Touch an empty DB file so the path exists and is unmistakably fresh (zero
|
||||
// bytes => no central-DB copy, no seeded credentials/logs).
|
||||
await fs.writeFile(dbPath, "");
|
||||
return {
|
||||
tmpDir,
|
||||
dbPath,
|
||||
dispose: async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true }).catch((err) => {
|
||||
harnessLog.warn(`Failed to remove disposable app workspace ${tmpDir}:`, err);
|
||||
});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// ── Injectable launch primitives ──────────────────────────────────────────────
|
||||
|
||||
/** A handle to a launched (real or stubbed) app server process. */
|
||||
export interface LaunchedAppProcess {
|
||||
/**
|
||||
* Resolves once the server is accepting connections / healthy on `port`.
|
||||
* Rejects if the process exits before becoming healthy, or on timeout.
|
||||
*/
|
||||
ready: Promise<void>;
|
||||
/**
|
||||
* Kill the process and free the port. MUST be safe to call unconditionally,
|
||||
* including after the process has already exited (crash/timeout) — idempotent.
|
||||
*/
|
||||
kill(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawns the app server. Injectable so unit tests can characterize isolation
|
||||
* without launching a real server; the default (heavier-lane) implementation
|
||||
* would boot via the CLI `serve`/`dashboard` entry against `options`.
|
||||
*/
|
||||
export type AppProcessSpawner = (options: AppLaunchOptions) => LaunchedAppProcess;
|
||||
|
||||
/** Description of the client bundle the harness must serve. */
|
||||
export interface ClientBundle {
|
||||
/** Absolute path to the freshly-built client dir to serve (FUSION_CLIENT_DIR). */
|
||||
clientDir: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensures a freshly-built client bundle exists and returns the dir to serve.
|
||||
* Injectable; the default implementation would build `@fusion/dashboard` and
|
||||
* return its `dist/client`. Tests assert that a *stale* bundle triggers a
|
||||
* rebuild before launch (the `dist/client` trap).
|
||||
*/
|
||||
export interface BundleBuilder {
|
||||
/**
|
||||
* Whether the currently-built bundle is stale relative to source. The harness
|
||||
* rebuilds before launch when this returns true so verification never serves a
|
||||
* stale bundle.
|
||||
*/
|
||||
isStale(): Promise<boolean>;
|
||||
/** Build the client bundle and return the dir to serve. */
|
||||
build(): Promise<ClientBundle>;
|
||||
/** Return the already-built bundle dir without building. */
|
||||
current(): Promise<ClientBundle>;
|
||||
}
|
||||
|
||||
/** Options passed to the process spawner for an isolated launch. */
|
||||
export interface AppLaunchOptions {
|
||||
port: number;
|
||||
host: string;
|
||||
dbPath: string;
|
||||
clientDir: string;
|
||||
/** Scrubbed env for the child (no credentials). */
|
||||
env: NodeJS.ProcessEnv;
|
||||
/** Abort signal to bound startup. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
// ── launchIsolatedApp ─────────────────────────────────────────────────────────
|
||||
|
||||
export interface LaunchIsolatedAppOptions {
|
||||
/** Injectable process spawner (required — no real default in this unit). */
|
||||
spawn: AppProcessSpawner;
|
||||
/** Injectable bundle builder (required — no real default in this unit). */
|
||||
bundle: BundleBuilder;
|
||||
/** Env source for reserved-port resolution + scrubbing (defaults to process.env). */
|
||||
env?: NodeJS.ProcessEnv;
|
||||
/** Host to bind (defaults to 127.0.0.1). */
|
||||
host?: string;
|
||||
/** Startup timeout in ms (defaults to 60s, matching boot-smoke). */
|
||||
readyTimeoutMs?: number;
|
||||
/** Abort signal to cancel/bound the launch. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/** A launched isolated app instance the driver (U8) can target. */
|
||||
export interface IsolatedApp {
|
||||
/** Base URL the app is served at (e.g. http://127.0.0.1:<port>). */
|
||||
baseUrl: string;
|
||||
/** The bound, non-reserved port. */
|
||||
port: number;
|
||||
/** Path to the disposable, fresh DB under a run-unique tmpdir. */
|
||||
dbPath: string;
|
||||
/** Absolute path to the freshly-built client dir being served. */
|
||||
clientDir: string;
|
||||
/**
|
||||
* Tear down EVERYTHING unconditionally: kill the process, free the port,
|
||||
* remove the tmpdir/DB. Idempotent and safe after a crash/timeout.
|
||||
*/
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
const DEFAULT_READY_TIMEOUT_MS = 60_000;
|
||||
|
||||
/** Scrub credentials/secrets from the env handed to the launched app. */
|
||||
const APP_ENV_ALLOWLIST = ["PATH", "HOME", "SHELL", "LANG", "LC_ALL", "TMPDIR", "TERM"] as const;
|
||||
|
||||
function scrubAppEnv(source: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
|
||||
const scrubbed: NodeJS.ProcessEnv = {};
|
||||
for (const key of APP_ENV_ALLOWLIST) {
|
||||
const value = source[key];
|
||||
if (value !== undefined) scrubbed[key] = value;
|
||||
}
|
||||
// Isolation flags: never run the engine against the central DB, skip onboarding.
|
||||
scrubbed.FUSION_SKIP_ONBOARDING = "1";
|
||||
return scrubbed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch an isolated app instance for a verification run. Order matters:
|
||||
*
|
||||
* 1. Resolve a NON-RESERVED port (4040 + FUSION_RESERVED_PORTS guards).
|
||||
* 2. Create a fresh, empty disposable DB under a run-unique tmpdir.
|
||||
* 3. Ensure a FRESHLY-BUILT bundle (rebuild if stale — no `dist/client` trap).
|
||||
* 4. Spawn the server against the disposable DB + fresh bundle + isolated port.
|
||||
* 5. Wait for readiness, bounded by `readyTimeoutMs` / `signal`.
|
||||
*
|
||||
* If ANY step after the tmpdir is created fails (including a crash/timeout
|
||||
* during readiness), the harness tears the whole instance down before
|
||||
* rejecting — the process is killed, the port released, and the tmpdir/DB
|
||||
* removed (R13 unconditional teardown).
|
||||
*/
|
||||
export async function launchIsolatedApp(options: LaunchIsolatedAppOptions): Promise<IsolatedApp> {
|
||||
const env = options.env ?? process.env;
|
||||
const host = options.host ?? "127.0.0.1";
|
||||
const readyTimeoutMs = options.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
|
||||
|
||||
// 1. Non-reserved port.
|
||||
const reserved = resolveReservedPorts(env);
|
||||
const port = await acquireNonReservedPort(reserved);
|
||||
// Defense in depth: never proceed with a reserved port.
|
||||
if (reserved.has(port)) {
|
||||
throw new Error(`refusing to bind reserved port ${port}`);
|
||||
}
|
||||
|
||||
// 2. Disposable DB under a run-unique tmpdir.
|
||||
const workspace = await createDisposableAppWorkspace();
|
||||
|
||||
let launched: LaunchedAppProcess | undefined;
|
||||
const teardown = async () => {
|
||||
try {
|
||||
await launched?.kill();
|
||||
} catch (err) {
|
||||
harnessLog.warn("Failed to kill isolated app process during teardown:", err);
|
||||
}
|
||||
await workspace.dispose();
|
||||
};
|
||||
|
||||
try {
|
||||
// 3. Fresh bundle — rebuild when stale so we never serve a stale dist/client.
|
||||
const bundle = (await options.bundle.isStale())
|
||||
? await options.bundle.build()
|
||||
: await options.bundle.current();
|
||||
|
||||
// 4. Spawn against the isolated surface.
|
||||
launched = options.spawn({
|
||||
port,
|
||||
host,
|
||||
dbPath: workspace.dbPath,
|
||||
clientDir: bundle.clientDir,
|
||||
env: scrubAppEnv(env),
|
||||
signal: options.signal,
|
||||
});
|
||||
|
||||
// 5. Bounded readiness.
|
||||
await withTimeout(launched.ready, readyTimeoutMs, options.signal);
|
||||
|
||||
return {
|
||||
baseUrl: `http://${host}:${port}`,
|
||||
port,
|
||||
dbPath: workspace.dbPath,
|
||||
clientDir: bundle.clientDir,
|
||||
dispose: teardown,
|
||||
};
|
||||
} catch (err) {
|
||||
// Unconditional teardown on crash/timeout/setup failure (R13).
|
||||
await teardown();
|
||||
throw err instanceof Error ? err : new Error(String(err));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve `promise`, or reject with a timeout / abort error after `timeoutMs`.
|
||||
* Used to bound startup so a hung launch is torn down rather than stranding the
|
||||
* verification run.
|
||||
*/
|
||||
async function withTimeout<T>(promise: Promise<T>, timeoutMs: number, signal?: AbortSignal): Promise<T> {
|
||||
if (signal?.aborted) {
|
||||
throw new Error("isolated app launch aborted before start");
|
||||
}
|
||||
let timer: NodeJS.Timeout | undefined;
|
||||
let onAbort: (() => void) | undefined;
|
||||
try {
|
||||
return await Promise.race([
|
||||
promise,
|
||||
new Promise<never>((_resolve, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
reject(new Error(`isolated app did not become ready within ${timeoutMs}ms`));
|
||||
}, timeoutMs);
|
||||
if (signal) {
|
||||
onAbort = () => reject(new Error("isolated app launch aborted"));
|
||||
signal.addEventListener("abort", onAbort, { once: true });
|
||||
}
|
||||
}),
|
||||
]);
|
||||
} finally {
|
||||
if (timer) clearTimeout(timer);
|
||||
if (signal && onAbort) signal.removeEventListener("abort", onAbort);
|
||||
}
|
||||
}
|
||||
986
packages/engine/src/mission-verification.ts
Normal file
986
packages/engine/src/mission-verification.ts
Normal file
@@ -0,0 +1,986 @@
|
||||
/**
|
||||
* Mission behavioral-verification capability (U3).
|
||||
*
|
||||
* The Validator Run's read-only AI judge cannot run code, so its "pass" on a
|
||||
* *behavioral* assertion is advisory only (U2). This module supplies the
|
||||
* authoritative, NON-MUTATING verification step that confirms a behavioral/bug
|
||||
* assertion by exercising the implemented code.
|
||||
*
|
||||
* Channels:
|
||||
* - **test-execution** (this unit): run the project's scoped test suite / an
|
||||
* agent-supplied regression test against a disposable checkout at a trusted
|
||||
* revision, through an explicit isolating sandbox backend.
|
||||
* - **app-driving** (later unit U5/U8): drive a running app instance. Not
|
||||
* implemented here — the capability surface is structured so it can be added
|
||||
* without reshaping callers.
|
||||
*
|
||||
* Safety invariants enforced here (the boundary, not a convention):
|
||||
* - R18: execute under an *isolating* sandbox backend (bubblewrap / sandbox-exec)
|
||||
* with a scrubbed env allowlist; FAIL CLOSED to a non-pass when no isolating
|
||||
* backend is available — never fall through to the unrestricted native backend.
|
||||
* - R19: the command is built from a fixed, system-owned template into which only
|
||||
* a validated test-file path is substituted; shell metacharacters are rejected.
|
||||
* - R11/R17: verification runs against a disposable checkout at the integration
|
||||
* SHA (never the pruned live worktree, never the repo root); the source tree
|
||||
* that feeds diff/merge is asserted git-clean after a run.
|
||||
* - R5/AE5: agent-supplied proof must FAIL on a second disposable checkout at
|
||||
* `git merge-base` (a revision the agent does not control) and PASS on the
|
||||
* implementation; a test that passes on both is rejected.
|
||||
* - R9: inconclusive / timeout / setup failure resolves to a non-pass.
|
||||
* - R10: no board / mission writes happen here.
|
||||
*/
|
||||
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { exec } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import type { TaskStore } from "@fusion/core";
|
||||
import type { SandboxCapabilities } from "./sandbox/index.js";
|
||||
import { resolveSandboxBackend } from "./sandbox/index.js";
|
||||
import type { SandboxBackend } from "./sandbox/index.js";
|
||||
import { detectBwrap } from "./sandbox/bubblewrap-detect.js";
|
||||
import { detectSandboxExec } from "./sandbox/sandbox-exec-detect.js";
|
||||
import { runVerificationCommand } from "./verification-utils.js";
|
||||
import type { VerificationCommandResult } from "./verification-utils.js";
|
||||
import { createLogger } from "./logger.js";
|
||||
|
||||
const execAsync = promisify(exec);
|
||||
const verifyLog = createLogger("mission-verify");
|
||||
|
||||
// ── Verdict types ───────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Outcome of a verification run for a single behavioral assertion.
|
||||
*
|
||||
* - `pass`: behavior confirmed by execution.
|
||||
* - `fail`: behavior observed wrong (the defect still reproduces / proof rejected).
|
||||
* - `inconclusive`: verification could not run or conclude (no isolating backend,
|
||||
* timeout, setup failure, rejected/invalid proof input). First-class and
|
||||
* distinct from `fail`: it must NOT spawn remediation (handled by later units),
|
||||
* but in this unit it never resolves to a default pass either.
|
||||
*/
|
||||
export type VerificationVerdict = "pass" | "fail" | "inconclusive";
|
||||
|
||||
/** Why a verification run reached its verdict (for durable observability later). */
|
||||
export interface VerificationOutcome {
|
||||
verdict: VerificationVerdict;
|
||||
/** Human-readable reason, suitable for surfacing in a failure record. */
|
||||
reason: string;
|
||||
/** The assertion this outcome corresponds to. */
|
||||
assertionId: string;
|
||||
/** Optional summarized command output for the failure record. */
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/** Shape of agent-supplied executable proof (a regression test). */
|
||||
export interface VerificationProof {
|
||||
/**
|
||||
* Path to the regression test file, relative to the checkout root. Validated
|
||||
* to reject shell metacharacters and path escapes before use (R19).
|
||||
*/
|
||||
testFilePath: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Which evidence channel(s) confirm a behavioral assertion.
|
||||
*
|
||||
* - `test`: code-level behavior, confirmed by running the suite / a regression
|
||||
* test against a disposable checkout (U3 test-execution channel).
|
||||
* - `app`: UI/bug behavior, confirmed by driving a running app instance
|
||||
* (U4 isolated launch + U8 driver).
|
||||
* - `both`: an assertion that is only confirmed when BOTH channels confirm it.
|
||||
*
|
||||
* Defaults conservatively to `test` when unspecified — the existing
|
||||
* test-execution behavior — so callers that do not classify an assertion keep
|
||||
* working unchanged.
|
||||
*/
|
||||
export type VerificationChannel = "test" | "app" | "both";
|
||||
|
||||
/**
|
||||
* Describes the observable UI behavior an app-driving verification must check.
|
||||
*
|
||||
* The driver navigates to `path` (relative to the isolated app's base URL) and
|
||||
* observes `selector`. `expectation` declares what a PASS looks like:
|
||||
*
|
||||
* - `present`: the assertion claims a feature/element should be present, so an
|
||||
* `observe`→`found` is a PASS and an `observe`→`absent` is a FAIL.
|
||||
* - `absent`: the assertion claims a bug no longer reproduces, so an
|
||||
* `observe`→`absent` is a PASS and an `observe`→`found` is a FAIL.
|
||||
*
|
||||
* In both cases a driver `inconclusive` (unavailable / un-exercisable) maps to
|
||||
* an inconclusive verdict, never pass/fail.
|
||||
*/
|
||||
export interface UiAssertionSpec {
|
||||
/** Path relative to the isolated app base URL, e.g. "/board". */
|
||||
path: string;
|
||||
/** Selector whose presence/absence encodes the behavior. */
|
||||
selector: string;
|
||||
/**
|
||||
* What a PASS looks like:
|
||||
* - `present`: element should be there (feature present).
|
||||
* - `absent`: element should be gone (bug no longer reproduces).
|
||||
*/
|
||||
expectation: "present" | "absent";
|
||||
/** Optional per-operation timeout override. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
/** Input describing a single behavioral assertion to verify. */
|
||||
export interface VerificationRequest {
|
||||
assertionId: string;
|
||||
/** The assertion text (for logging / reason building). */
|
||||
assertion: string;
|
||||
/** Board task id associated with the feature, used for verification-command logging. */
|
||||
taskId?: string;
|
||||
/**
|
||||
* Which evidence channel(s) confirm this assertion. Defaults to `test`
|
||||
* (the existing test-execution behavior) when unspecified.
|
||||
*/
|
||||
channel?: VerificationChannel;
|
||||
/**
|
||||
* The UI behavior the app-driving channel must reproduce. Required when
|
||||
* `channel` is `app` or `both`; absent for `test`. When the channel needs app
|
||||
* driving but this is missing, the app channel resolves to inconclusive
|
||||
* (structurally un-exercisable), never a default pass/fail.
|
||||
*/
|
||||
ui?: UiAssertionSpec;
|
||||
/**
|
||||
* The trusted revision (integration SHA) whose checkout the implementation is
|
||||
* verified against. When absent, verification is inconclusive (cannot
|
||||
* materialize a trusted checkout).
|
||||
*/
|
||||
integrationSha?: string;
|
||||
/**
|
||||
* The `git merge-base` revision (feature branch vs base branch) used as the
|
||||
* pre-fix baseline for agent-supplied proof. Not agent-controlled.
|
||||
*/
|
||||
mergeBaseSha?: string;
|
||||
/** Optional agent-supplied executable proof. */
|
||||
proof?: VerificationProof;
|
||||
/** Abort signal to bound the run. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/**
|
||||
* Injected verification capability. Mirrors the `createFnAgent` injection
|
||||
* pattern so MissionExecutionLoop can swap a real implementation for a mock in
|
||||
* tests. Optional on the loop: when absent, behavioral assertions resolve to a
|
||||
* non-pass without invoking any execution (preserving existing behavior for
|
||||
* call sites that do not inject a capability).
|
||||
*/
|
||||
export interface VerificationCapability {
|
||||
verifyBehavioralAssertion(request: VerificationRequest): Promise<VerificationOutcome>;
|
||||
}
|
||||
|
||||
// ── Command-template safety (R19) ─────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Characters that could break out of the fixed command template or inject
|
||||
* additional shell behavior. Agent-supplied test paths containing any of these
|
||||
* are rejected before execution.
|
||||
*/
|
||||
const SHELL_METACHARACTERS = /[;&|`$(){}<>!*?[\]\\"'\n\r\t\0]/;
|
||||
|
||||
/**
|
||||
* Validate an agent-supplied test-file path. Returns the normalized path when
|
||||
* safe, or `null` when it must be rejected (R19).
|
||||
*
|
||||
* Rejects: empty, absolute paths, parent-dir escapes, shell metacharacters,
|
||||
* and leading dashes (which could be read as command flags).
|
||||
*/
|
||||
export function validateTestPath(rawPath: unknown): string | null {
|
||||
if (typeof rawPath !== "string") return null;
|
||||
const path = rawPath.trim();
|
||||
if (path.length === 0) return null;
|
||||
if (SHELL_METACHARACTERS.test(path)) return null;
|
||||
if (path.startsWith("/")) return null; // must be relative to the checkout
|
||||
if (path.startsWith("-")) return null; // could be parsed as a flag
|
||||
// Reject parent-dir escapes (any `..` segment).
|
||||
const segments = path.split("/");
|
||||
if (segments.some((seg) => seg === "..")) return null;
|
||||
return path;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the verification command from the fixed system-owned template. Only a
|
||||
* pre-validated test path may be substituted (R19). Callers MUST pass a path
|
||||
* already run through {@link validateTestPath}; this function re-checks and
|
||||
* throws on violation as a defense-in-depth guard.
|
||||
*/
|
||||
export function buildVerificationCommand(template: string, validatedTestPath?: string): string {
|
||||
if (validatedTestPath !== undefined) {
|
||||
if (validateTestPath(validatedTestPath) === null) {
|
||||
throw new Error(`Refusing to build verification command: invalid test path ${JSON.stringify(validatedTestPath)}`);
|
||||
}
|
||||
if (!template.includes("{testPath}")) {
|
||||
throw new Error("Verification command template must contain a {testPath} placeholder when a test path is supplied");
|
||||
}
|
||||
return template.replace("{testPath}", validatedTestPath);
|
||||
}
|
||||
// Whole-suite invocation: the template must not reference a test path.
|
||||
return template.replace("{testPath}", "").trimEnd();
|
||||
}
|
||||
|
||||
// ── Isolating-backend selection (R18, fail-closed) ────────────────────────────
|
||||
|
||||
/**
|
||||
* Result of selecting an isolating sandbox backend for verification.
|
||||
*/
|
||||
export interface IsolatingBackendSelection {
|
||||
/** The backend id to request from `resolveSandboxBackend`, or null if none. */
|
||||
backendId: SandboxCapabilities["id"] | null;
|
||||
/** Why no isolating backend is available (when backendId is null). */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Describes the detected availability of isolating backends on this host.
|
||||
* Injectable for tests so we don't shell out to detect bwrap/sandbox-exec.
|
||||
*/
|
||||
export interface IsolatingBackendProbe {
|
||||
platform: NodeJS.Platform;
|
||||
bubblewrapAvailable: boolean;
|
||||
sandboxExecAvailable: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Choose an isolating backend, failing closed. Returns `backendId: null` (a
|
||||
* non-pass signal) when no isolating backend is available — verification must
|
||||
* NEVER fall through to the unrestricted native backend (R18).
|
||||
*/
|
||||
export function selectIsolatingBackend(probe: IsolatingBackendProbe): IsolatingBackendSelection {
|
||||
if (probe.platform === "linux" && probe.bubblewrapAvailable) {
|
||||
return { backendId: "bubblewrap" };
|
||||
}
|
||||
if (probe.platform === "darwin" && probe.sandboxExecAvailable) {
|
||||
return { backendId: "sandbox-exec" };
|
||||
}
|
||||
return {
|
||||
backendId: null,
|
||||
reason: `no isolating sandbox backend available (platform=${probe.platform}, bwrap=${probe.bubblewrapAvailable}, sandbox-exec=${probe.sandboxExecAvailable})`,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Environment scrubbing (R18) ───────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Environment variables permitted into the verification child process. Anything
|
||||
* not on the allowlist (API keys, auth tokens, DB credentials, agent logs) is
|
||||
* dropped so agent-authored code executes with a minimal environment.
|
||||
*/
|
||||
export const VERIFICATION_ENV_ALLOWLIST = [
|
||||
"PATH",
|
||||
"HOME",
|
||||
"SHELL",
|
||||
"LANG",
|
||||
"LC_ALL",
|
||||
"TMPDIR",
|
||||
"TERM",
|
||||
"NODE_ENV",
|
||||
// pnpm / corepack need these to resolve the package manager in the checkout.
|
||||
"PNPM_HOME",
|
||||
"COREPACK_HOME",
|
||||
"npm_config_registry",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Produce a scrubbed environment containing only allowlisted keys from the
|
||||
* source environment, with `CI=1` forced for deterministic test runs.
|
||||
*/
|
||||
export function scrubEnv(source: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv {
|
||||
const scrubbed: NodeJS.ProcessEnv = {};
|
||||
for (const key of VERIFICATION_ENV_ALLOWLIST) {
|
||||
const value = source[key];
|
||||
if (value !== undefined) {
|
||||
scrubbed[key] = value;
|
||||
}
|
||||
}
|
||||
// Force deterministic, non-interactive execution.
|
||||
scrubbed.CI = "1";
|
||||
return scrubbed;
|
||||
}
|
||||
|
||||
// ── Disposable checkout materialization (R11/R17) ─────────────────────────────
|
||||
|
||||
/** A disposable checkout the verification run can execute against. */
|
||||
export interface DisposableCheckout {
|
||||
/** Absolute path to the checkout root (under a run-unique tmpdir). */
|
||||
dir: string;
|
||||
/** Tear the checkout down unconditionally (idempotent). */
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Materializes disposable checkouts at a trusted revision. Injectable so tests
|
||||
* can supply a fixture checkout without invoking git.
|
||||
*/
|
||||
export interface CheckoutMaterializer {
|
||||
/**
|
||||
* Create a disposable checkout of `rootDir` at `revision` under a run-unique
|
||||
* tmpdir. The implementation MUST NOT mutate the source tree at `rootDir`.
|
||||
*/
|
||||
materialize(rootDir: string, revision: string): Promise<DisposableCheckout>;
|
||||
/**
|
||||
* Assert that the source tree feeding diff/merge is git-clean (byte-identical)
|
||||
* — the R17 post-condition. Throws if dirty.
|
||||
*/
|
||||
assertSourceClean(rootDir: string): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Default git-backed materializer: `git worktree add --detach <tmp> <revision>`
|
||||
* produces an isolated checkout without touching the source working tree, and
|
||||
* `git status --porcelain` on the source confirms cleanliness afterwards.
|
||||
*/
|
||||
export class GitCheckoutMaterializer implements CheckoutMaterializer {
|
||||
async materialize(rootDir: string, revision: string): Promise<DisposableCheckout> {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "fn-verify-"));
|
||||
// `git worktree add --detach` checks out the revision into a throwaway dir
|
||||
// without modifying the source working tree.
|
||||
await execAsync(`git worktree add --detach ${JSON.stringify(dir)} ${JSON.stringify(revision)}`, {
|
||||
cwd: rootDir,
|
||||
timeout: 60_000,
|
||||
maxBuffer: 8 * 1024 * 1024,
|
||||
});
|
||||
return {
|
||||
dir,
|
||||
dispose: async () => {
|
||||
try {
|
||||
await execAsync(`git worktree remove --force ${JSON.stringify(dir)}`, {
|
||||
cwd: rootDir,
|
||||
timeout: 30_000,
|
||||
});
|
||||
} catch (err) {
|
||||
verifyLog.warn(`Failed to remove verification worktree ${dir}:`, err);
|
||||
}
|
||||
await fs.rm(dir, { recursive: true, force: true }).catch(() => {});
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async assertSourceClean(rootDir: string): Promise<void> {
|
||||
const { stdout } = await execAsync("git status --porcelain", {
|
||||
cwd: rootDir,
|
||||
timeout: 30_000,
|
||||
maxBuffer: 8 * 1024 * 1024,
|
||||
});
|
||||
if (stdout.trim().length > 0) {
|
||||
throw new Error(`Source tree is not git-clean after verification run:\n${stdout.trim()}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Probe the host for isolating-backend availability (cached by the detectors). */
|
||||
async function probeIsolatingBackends(): Promise<IsolatingBackendProbe> {
|
||||
const [bwrap, sandboxExec] = await Promise.all([
|
||||
detectBwrap().catch(() => ({ available: false })),
|
||||
detectSandboxExec().catch(() => ({ available: false })),
|
||||
]);
|
||||
return {
|
||||
platform: process.platform,
|
||||
bubblewrapAvailable: bwrap.available,
|
||||
sandboxExecAvailable: sandboxExec.available,
|
||||
};
|
||||
}
|
||||
|
||||
// ── Test-execution verification capability ────────────────────────────────────
|
||||
|
||||
export interface TestExecutionVerificationOptions {
|
||||
/** Task store, reused by runVerificationCommand for command logging. */
|
||||
store: TaskStore;
|
||||
/** Repo root whose source tree must remain git-clean. */
|
||||
rootDir: string;
|
||||
/**
|
||||
* Fixed, system-owned command template. Must contain `{testPath}` when an
|
||||
* agent-supplied proof path is used. Example: `pnpm vitest run {testPath}`.
|
||||
*/
|
||||
commandTemplate: string;
|
||||
/** Injectable checkout materializer (defaults to git-backed). */
|
||||
materializer?: CheckoutMaterializer;
|
||||
/** Injectable backend probe (defaults to host detection). */
|
||||
probeBackends?: () => Promise<IsolatingBackendProbe>;
|
||||
/**
|
||||
* Injectable factory for the isolating sandbox backend, given the selected
|
||||
* backend id. Defaults to `resolveSandboxBackend({ backendId })`. Injectable so
|
||||
* tests can supply a scripted backend without mutating global sandbox state.
|
||||
*/
|
||||
backendFactory?: (backendId: SandboxCapabilities["id"]) => SandboxBackend;
|
||||
/** Injectable env source (defaults to process.env). */
|
||||
envSource?: NodeJS.ProcessEnv;
|
||||
}
|
||||
|
||||
/**
|
||||
* The test-execution channel of the verification run. Confirms a behavioral
|
||||
* assertion by running the suite / an agent-supplied regression test against a
|
||||
* disposable checkout at the integration SHA, under an isolating sandbox
|
||||
* backend with a scrubbed env. Fails closed to a non-pass on any setup failure.
|
||||
*
|
||||
* App-driving is NOT handled here; a later unit dispatches UI/bug assertions to
|
||||
* an app-driving channel. This class is the canonical pattern that channel will
|
||||
* mirror.
|
||||
*/
|
||||
export class TestExecutionVerificationCapability implements VerificationCapability {
|
||||
private readonly store: TaskStore;
|
||||
private readonly rootDir: string;
|
||||
private readonly commandTemplate: string;
|
||||
private readonly materializer: CheckoutMaterializer;
|
||||
private readonly probeBackends: () => Promise<IsolatingBackendProbe>;
|
||||
private readonly backendFactory: (backendId: SandboxCapabilities["id"]) => SandboxBackend;
|
||||
private readonly envSource: NodeJS.ProcessEnv;
|
||||
|
||||
constructor(options: TestExecutionVerificationOptions) {
|
||||
this.store = options.store;
|
||||
this.rootDir = options.rootDir;
|
||||
this.commandTemplate = options.commandTemplate;
|
||||
this.materializer = options.materializer ?? new GitCheckoutMaterializer();
|
||||
this.probeBackends = options.probeBackends ?? probeIsolatingBackends;
|
||||
this.backendFactory = options.backendFactory ?? ((backendId) => resolveSandboxBackend({ backendId }));
|
||||
this.envSource = options.envSource ?? process.env;
|
||||
}
|
||||
|
||||
async verifyBehavioralAssertion(request: VerificationRequest): Promise<VerificationOutcome> {
|
||||
const { assertionId } = request;
|
||||
|
||||
// R11: a trusted revision is required to materialize a disposable checkout.
|
||||
if (!request.integrationSha) {
|
||||
return this.inconclusive(assertionId, "no integration SHA available to materialize a trusted checkout");
|
||||
}
|
||||
// Capture the narrowed (string) value: property-access narrowing does not
|
||||
// carry into the nested async IIFE below, so reference this const there.
|
||||
const integrationSha = request.integrationSha;
|
||||
|
||||
// R19: validate any agent-supplied proof path BEFORE doing any work.
|
||||
let validatedTestPath: string | undefined;
|
||||
if (request.proof) {
|
||||
const safe = validateTestPath(request.proof.testFilePath);
|
||||
if (safe === null) {
|
||||
return this.inconclusive(
|
||||
assertionId,
|
||||
`agent-supplied test path rejected (invalid or contains shell metacharacters): ${JSON.stringify(request.proof.testFilePath)}`,
|
||||
);
|
||||
}
|
||||
validatedTestPath = safe;
|
||||
}
|
||||
|
||||
// R18: select an isolating backend, fail closed when none is available.
|
||||
const probe = await this.probeBackends();
|
||||
const selection = selectIsolatingBackend(probe);
|
||||
if (selection.backendId === null) {
|
||||
return this.inconclusive(assertionId, selection.reason ?? "no isolating sandbox backend available");
|
||||
}
|
||||
|
||||
const command = buildVerificationCommand(this.commandTemplate, validatedTestPath);
|
||||
const scrubbedEnv = scrubEnv(this.envSource);
|
||||
const logTaskId = request.taskId ?? `verify-${assertionId}`;
|
||||
|
||||
// Route runVerificationCommand through the explicitly-selected isolating
|
||||
// backend (R18). The backend is passed in by argument rather than the no-arg
|
||||
// resolveSandboxBackend()/global test hook: applyBehavioralPosture dispatches
|
||||
// assertions concurrently via Promise.all, so a process-global override would
|
||||
// race — a sibling run could clear it mid-run and the no-arg resolver would
|
||||
// then fall through to the unrestricted native backend, breaking fail-closed
|
||||
// isolation. Threading the backend keeps each run pinned to its own isolating
|
||||
// backend regardless of concurrency.
|
||||
const isolating = this.backendFactory(selection.backendId);
|
||||
|
||||
let implCheckout: DisposableCheckout | undefined;
|
||||
let baselineCheckout: DisposableCheckout | undefined;
|
||||
let outcome: VerificationOutcome;
|
||||
try {
|
||||
outcome = await (async (): Promise<VerificationOutcome> => {
|
||||
implCheckout = await this.materializer.materialize(this.rootDir, integrationSha);
|
||||
|
||||
const implResult = await runVerificationCommand(
|
||||
this.store,
|
||||
implCheckout.dir,
|
||||
logTaskId,
|
||||
command,
|
||||
"test",
|
||||
request.signal,
|
||||
verifyLog,
|
||||
"reviewer",
|
||||
scrubbedEnv,
|
||||
undefined,
|
||||
isolating,
|
||||
);
|
||||
|
||||
// An infra failure (timeout / abort / setup error) is NOT behavioral
|
||||
// evidence: it must resolve to inconclusive, never fold into a fail or — on
|
||||
// the baseline — wrongly satisfy `!baselineResult.success` and upgrade a
|
||||
// proof to pass.
|
||||
const implInfra = infraFailureReason(implResult);
|
||||
if (implInfra) {
|
||||
return this.inconclusive(assertionId, `implementation verification could not complete: ${implInfra}`);
|
||||
}
|
||||
|
||||
// R5/AE5: agent-supplied proof must fail on the merge-base baseline and
|
||||
// pass on the implementation. A test that passes on both is not exercising
|
||||
// the defect — reject it.
|
||||
if (validatedTestPath) {
|
||||
if (!request.mergeBaseSha) {
|
||||
return this.inconclusive(assertionId, "no merge-base SHA available to validate agent-supplied proof");
|
||||
}
|
||||
baselineCheckout = await this.materializer.materialize(this.rootDir, request.mergeBaseSha);
|
||||
const baselineResult = await runVerificationCommand(
|
||||
this.store,
|
||||
baselineCheckout.dir,
|
||||
logTaskId,
|
||||
command,
|
||||
"test",
|
||||
request.signal,
|
||||
verifyLog,
|
||||
"reviewer",
|
||||
scrubbedEnv,
|
||||
undefined,
|
||||
isolating,
|
||||
);
|
||||
|
||||
// A timed-out / aborted baseline is not a real "fails on the baseline"
|
||||
// signal; treating it as one would wrongly upgrade the proof to pass.
|
||||
const baselineInfra = infraFailureReason(baselineResult);
|
||||
if (baselineInfra) {
|
||||
return this.inconclusive(assertionId, `baseline verification could not complete: ${baselineInfra}`);
|
||||
}
|
||||
|
||||
if (baselineResult.success && implResult.success) {
|
||||
return {
|
||||
verdict: "fail",
|
||||
assertionId,
|
||||
reason: "agent-supplied proof passes on both the pre-fix baseline and the implementation; it does not exercise the defect",
|
||||
detail: "pass-on-both rejected (R5/AE5)",
|
||||
};
|
||||
}
|
||||
if (!baselineResult.success && implResult.success) {
|
||||
return { verdict: "pass", assertionId, reason: "regression test fails on the pre-fix baseline and passes on the implementation" };
|
||||
}
|
||||
// A real (non-infra) failure on the implementation → defect still reproduces.
|
||||
return {
|
||||
verdict: "fail",
|
||||
assertionId,
|
||||
reason: "regression test does not pass on the implementation; behavior not confirmed",
|
||||
detail: implResult.stderr || implResult.stdout || undefined,
|
||||
};
|
||||
}
|
||||
|
||||
// Whole-suite channel: pass only when the suite passes; a real (non-infra)
|
||||
// failure is behavioral evidence.
|
||||
if (implResult.success) {
|
||||
return { verdict: "pass", assertionId, reason: "verification suite passed on the implementation checkout" };
|
||||
}
|
||||
return {
|
||||
verdict: "fail",
|
||||
assertionId,
|
||||
reason: "verification suite failed on the implementation checkout; behavior not confirmed",
|
||||
detail: implResult.stderr || implResult.stdout || undefined,
|
||||
};
|
||||
})();
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
// R9: any setup/exec failure (timeout, abort, materialization error) is a
|
||||
// non-pass; we route it to inconclusive (infra, not behavioral).
|
||||
outcome = this.inconclusive(assertionId, `verification run could not complete: ${message}`);
|
||||
} finally {
|
||||
await implCheckout?.dispose();
|
||||
await baselineCheckout?.dispose();
|
||||
}
|
||||
|
||||
// R17 post-condition — checked OUTSIDE finally so it never masks the verdict
|
||||
// via an unsafe finally-throw. The source tree feeding diff/merge must be
|
||||
// byte-clean afterwards; a violation means verification mutated the source, so
|
||||
// we fail closed to inconclusive rather than trusting the verdict.
|
||||
try {
|
||||
await this.materializer.assertSourceClean(this.rootDir);
|
||||
} catch (cleanErr) {
|
||||
const message = cleanErr instanceof Error ? cleanErr.message : String(cleanErr);
|
||||
verifyLog.error("Verification post-condition violated (source not git-clean):", cleanErr);
|
||||
return this.inconclusive(
|
||||
assertionId,
|
||||
`verification post-condition violated: source tree not git-clean after run: ${message}`,
|
||||
);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
|
||||
private inconclusive(assertionId: string, reason: string): VerificationOutcome {
|
||||
return { verdict: "inconclusive", assertionId, reason };
|
||||
}
|
||||
}
|
||||
|
||||
// ── App-driving verification channel (U5: U4 launch + U8 driver) ───────────────
|
||||
//
|
||||
// UI/bug assertions are confirmed by driving a running app instance rather than
|
||||
// running a test suite. This channel launches the isolated app (U4) and a
|
||||
// browser driver (U8), navigates to the assertion's surface, observes the
|
||||
// encoding selector, and maps the observation to a verdict.
|
||||
//
|
||||
// ENGINE↔PLUGIN WIRING (why a structural injected seam, not a direct import):
|
||||
// the U8 driver lives in `@fusion-plugin-examples/agent-browser`, on which the
|
||||
// engine has NO package dependency (the plugin depends on plugin-sdk, not the
|
||||
// reverse). A direct `import` would invert that and couple the engine to a
|
||||
// bundled plugin. Instead this channel takes an injected `AppDrivingDeps` seam
|
||||
// whose method shapes match `launchIsolatedApp` (U4) and `launchBrowserDriver`
|
||||
// (U8) STRUCTURALLY — so the real wiring (constructed where the loop is built)
|
||||
// passes the plugin's `launchBrowserDriver` + the harness's `launchIsolatedApp`,
|
||||
// while merge-gate tests pass a mock. This mirrors the `CheckoutMaterializer` /
|
||||
// `backendFactory` injection already used by the test-execution channel and the
|
||||
// `createFnAgent` injection pattern, and keeps the engine decoupled + testable.
|
||||
|
||||
/**
|
||||
* The slice of the U8 driver session this channel uses. Declared structurally so
|
||||
* the engine does not import the plugin; the real session (from
|
||||
* `launchBrowserDriver`) satisfies it.
|
||||
*/
|
||||
export interface AppDriverSession {
|
||||
navigate(
|
||||
url: string,
|
||||
opts?: { timeoutMs?: number },
|
||||
): Promise<{ status: "ok"; url: string } | { status: "inconclusive"; reason: string; detail: string }>;
|
||||
observe(
|
||||
selector: string,
|
||||
opts?: { timeoutMs?: number; expectAbsent?: boolean },
|
||||
): Promise<
|
||||
| { status: "found"; text: string; url: string }
|
||||
| { status: "absent"; url: string }
|
||||
| { status: "inconclusive"; reason: string; detail: string }
|
||||
>;
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
/** Result of attempting to acquire a driver session (structural mirror of U8's `DriverLaunchResult`). */
|
||||
export type AppDriverLaunchResult =
|
||||
| { status: "ready"; session: AppDriverSession }
|
||||
| { status: "inconclusive"; reason: string; detail: string };
|
||||
|
||||
/** A launched isolated app instance (structural mirror of U4's `IsolatedApp`). */
|
||||
export interface IsolatedAppInstance {
|
||||
baseUrl: string;
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Injected dependencies for the app-driving channel. The real implementation
|
||||
* wires `launchApp` → U4 `launchIsolatedApp` and `launchDriver` → U8
|
||||
* `launchBrowserDriver`; tests inject mocks so no real app/browser is launched.
|
||||
*/
|
||||
export interface AppDrivingDeps {
|
||||
/** Launch an isolated app instance (R11/R13: disposable, isolated, fresh bundle). */
|
||||
launchApp(signal?: AbortSignal): Promise<IsolatedAppInstance>;
|
||||
/** Acquire a driver session targeting the isolated instance. */
|
||||
launchDriver(baseUrl: string, signal?: AbortSignal): Promise<AppDriverLaunchResult>;
|
||||
}
|
||||
|
||||
export interface AppDrivingVerificationOptions {
|
||||
deps: AppDrivingDeps;
|
||||
}
|
||||
|
||||
/**
|
||||
* The app-driving channel of the verification run.
|
||||
*
|
||||
* Outcome mapping (R21) — the load-bearing table:
|
||||
*
|
||||
* | driver result | expectation=`present` | expectation=`absent` |
|
||||
* | ------------------------ | ---------------------------- | ---------------------------- |
|
||||
* | observe → `found` | PASS (feature present) | FAIL (bug still reproduces) |
|
||||
* | observe → `absent` | FAIL (feature missing) | PASS (bug no longer repros) |
|
||||
* | observe → `inconclusive` | INCONCLUSIVE | INCONCLUSIVE |
|
||||
* | driver launch failed | INCONCLUSIVE | INCONCLUSIVE |
|
||||
* | app launch failed | INCONCLUSIVE | INCONCLUSIVE |
|
||||
* | navigate → `inconclusive`| INCONCLUSIVE | INCONCLUSIVE |
|
||||
* | no `ui` spec supplied | INCONCLUSIVE (un-exercisable)| INCONCLUSIVE (un-exercisable)|
|
||||
*
|
||||
* A driver `inconclusive` is ALWAYS inconclusive (never a default pass, never an
|
||||
* auto-fail). Only a real `found`/`absent` observation produces pass/fail.
|
||||
*/
|
||||
export class AppDrivingVerificationCapability implements VerificationCapability {
|
||||
private readonly deps: AppDrivingDeps;
|
||||
|
||||
constructor(options: AppDrivingVerificationOptions) {
|
||||
this.deps = options.deps;
|
||||
}
|
||||
|
||||
async verifyBehavioralAssertion(request: VerificationRequest): Promise<VerificationOutcome> {
|
||||
const { assertionId } = request;
|
||||
const spec = request.ui;
|
||||
if (!spec) {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: "app-driving channel selected but no UI assertion spec was supplied (structurally un-exercisable)",
|
||||
};
|
||||
}
|
||||
|
||||
let app: IsolatedAppInstance | undefined;
|
||||
let session: AppDriverSession | undefined;
|
||||
try {
|
||||
// R11/R13: drive the isolated instance, never the user's live app.
|
||||
try {
|
||||
app = await this.deps.launchApp(request.signal);
|
||||
} catch (err) {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `isolated app launch failed: ${errMessage(err)}`,
|
||||
};
|
||||
}
|
||||
|
||||
const launch = await this.deps.launchDriver(app.baseUrl, request.signal);
|
||||
if (launch.status !== "ready") {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `app driver unavailable (${launch.reason})`,
|
||||
detail: launch.detail,
|
||||
};
|
||||
}
|
||||
session = launch.session;
|
||||
|
||||
const url = joinUrl(app.baseUrl, spec.path);
|
||||
const nav = await session.navigate(url, { timeoutMs: spec.timeoutMs });
|
||||
if (nav.status !== "ok") {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `navigation to the assertion surface failed (${nav.reason})`,
|
||||
detail: nav.detail,
|
||||
};
|
||||
}
|
||||
|
||||
const observation = await session.observe(spec.selector, {
|
||||
timeoutMs: spec.timeoutMs,
|
||||
expectAbsent: spec.expectation === "absent",
|
||||
});
|
||||
|
||||
if (observation.status === "inconclusive") {
|
||||
// Driver inconclusive is ALWAYS inconclusive (R21).
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `driver could not reach a definitive observation (${observation.reason})`,
|
||||
detail: observation.detail,
|
||||
};
|
||||
}
|
||||
|
||||
// A definitive observation (`found` | `absent`) maps to pass/fail per the
|
||||
// expectation. This is the one place a real negative (`absent`) becomes a
|
||||
// PASS — for a "bug no longer reproduces" assertion.
|
||||
const present = observation.status === "found";
|
||||
if (spec.expectation === "present") {
|
||||
return present
|
||||
? { verdict: "pass", assertionId, reason: `expected element present at ${spec.path} (${spec.selector})` }
|
||||
: { verdict: "fail", assertionId, reason: `expected element ABSENT at ${spec.path} (${spec.selector}); feature not observed` };
|
||||
}
|
||||
// expectation === "absent" (bug should no longer reproduce)
|
||||
return present
|
||||
? {
|
||||
verdict: "fail",
|
||||
assertionId,
|
||||
reason: `defect still reproduces: element still present at ${spec.path} (${spec.selector})`,
|
||||
detail: observation.status === "found" ? observation.text : undefined,
|
||||
}
|
||||
: { verdict: "pass", assertionId, reason: `defect no longer reproduces: element absent at ${spec.path} (${spec.selector})` };
|
||||
} catch (err) {
|
||||
// Any unexpected driver/setup error is infra, not behavioral → inconclusive.
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `app-driving run could not complete: ${errMessage(err)}`,
|
||||
};
|
||||
} finally {
|
||||
await session?.dispose().catch((err) => verifyLog.warn("driver dispose failed:", err));
|
||||
await app?.dispose().catch((err) => verifyLog.warn("isolated app dispose failed:", err));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function errMessage(err: unknown): string {
|
||||
return err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
|
||||
/**
|
||||
* Return a human-readable reason when a verification command result represents an
|
||||
* *infrastructure* outcome (timeout / abort / setup failure) rather than a real
|
||||
* test verdict, or `null` when the result is a genuine pass/fail. Infra outcomes
|
||||
* must resolve to `inconclusive`, never be folded into behavioral evidence
|
||||
* (R9) — a timed-out suite is not a "fail", and a timed-out baseline must not
|
||||
* satisfy the `!baselineResult.success` branch that upgrades a proof to "pass".
|
||||
*/
|
||||
function infraFailureReason(result: VerificationCommandResult): string | null {
|
||||
if (result.timedOut) return "command timed out";
|
||||
if (result.aborted) return "command aborted";
|
||||
if (result.executionError) return "command could not be executed (setup/sandbox error)";
|
||||
return null;
|
||||
}
|
||||
|
||||
function joinUrl(baseUrl: string, pathPart: string): string {
|
||||
const base = baseUrl.replace(/\/+$/, "");
|
||||
if (!pathPart) return base;
|
||||
return `${base}/${pathPart.replace(/^\/+/, "")}`;
|
||||
}
|
||||
|
||||
// ── Determinism contract (R20) ────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Wrap a verification capability so a verdict is only authoritative when it
|
||||
* agrees across N runs. A result that DIFFERS across runs is flaky and resolves
|
||||
* to `inconclusive` — never `fail` (R20). N is small and configurable.
|
||||
*
|
||||
* Semantics:
|
||||
* - All N runs return the SAME verdict (pass/fail/inconclusive) → that verdict.
|
||||
* - Verdicts differ across runs → `inconclusive` (flaky), with a reason naming
|
||||
* the disagreement. In particular a run that passes then fails is flaky, NOT a
|
||||
* fail — this is the exact false-fail class R20 removes.
|
||||
* - Short-circuit: once an `inconclusive` appears we still complete the runs is
|
||||
* unnecessary; an early `inconclusive` already means non-authoritative, so we
|
||||
* return inconclusive immediately to bound cost.
|
||||
*
|
||||
* The wrapper is channel-agnostic: it wraps the app-driving channel (the new,
|
||||
* fragile surface) and can wrap any capability. The test-execution channel can
|
||||
* also be wrapped, but the dispatcher applies it to app-driving by default since
|
||||
* that is where flakiness lives.
|
||||
*/
|
||||
export const DEFAULT_VERIFICATION_RUNS = 2;
|
||||
|
||||
export interface DeterministicVerificationOptions {
|
||||
inner: VerificationCapability;
|
||||
/** Number of agreeing runs required for an authoritative verdict. Default {@link DEFAULT_VERIFICATION_RUNS}. */
|
||||
runs?: number;
|
||||
}
|
||||
|
||||
export class DeterministicVerificationCapability implements VerificationCapability {
|
||||
private readonly inner: VerificationCapability;
|
||||
private readonly runs: number;
|
||||
|
||||
constructor(options: DeterministicVerificationOptions) {
|
||||
this.inner = options.inner;
|
||||
this.runs = Math.max(1, options.runs ?? DEFAULT_VERIFICATION_RUNS);
|
||||
}
|
||||
|
||||
async verifyBehavioralAssertion(request: VerificationRequest): Promise<VerificationOutcome> {
|
||||
const first = await this.inner.verifyBehavioralAssertion(request);
|
||||
// An early inconclusive is already non-authoritative; bound cost.
|
||||
if (first.verdict === "inconclusive" || this.runs === 1) return first;
|
||||
|
||||
for (let i = 1; i < this.runs; i += 1) {
|
||||
const next = await this.inner.verifyBehavioralAssertion(request);
|
||||
if (next.verdict === "inconclusive") {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId: request.assertionId,
|
||||
reason: `verification non-deterministic: run ${i + 1} was inconclusive after an initial ${first.verdict}`,
|
||||
};
|
||||
}
|
||||
if (next.verdict !== first.verdict) {
|
||||
// Flaky: differs across runs → inconclusive, NEVER fail (R20).
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId: request.assertionId,
|
||||
reason: `verification flaky: result differed across ${this.runs} runs (${first.verdict} then ${next.verdict}); resolving to inconclusive rather than ${[first.verdict, next.verdict].includes("fail") ? "fail" : "pass"}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
// All runs agreed → authoritative verdict (this is the only path to an
|
||||
// authoritative `fail`, satisfying "fail requires N-run agreement").
|
||||
return first;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Dispatching capability: route by assertion shape (U5 goal) ─────────────────
|
||||
|
||||
export interface DispatchingVerificationOptions {
|
||||
/** The U3 test-execution channel. */
|
||||
testChannel: VerificationCapability;
|
||||
/**
|
||||
* The app-driving channel. Optional: when absent, an `app`/`both` assertion
|
||||
* resolves to inconclusive (the channel is unavailable) rather than a default
|
||||
* pass/fail — preserving the fail-closed posture.
|
||||
*/
|
||||
appChannel?: VerificationCapability;
|
||||
}
|
||||
|
||||
/**
|
||||
* Routes a verification request to the right evidence channel(s) by assertion
|
||||
* shape, then combines outcomes:
|
||||
*
|
||||
* - `test` → test-execution channel only.
|
||||
* - `app` → app-driving channel only.
|
||||
* - `both` → BOTH channels; the assertion passes ONLY when both pass.
|
||||
* - any `fail` → `fail` (with the failing channel's reason);
|
||||
* - else any `inconclusive` → `inconclusive`;
|
||||
* - else (both pass) → `pass`.
|
||||
*
|
||||
* Channel defaults to `test` when unspecified (existing behavior).
|
||||
*/
|
||||
export class DispatchingVerificationCapability implements VerificationCapability {
|
||||
private readonly testChannel: VerificationCapability;
|
||||
private readonly appChannel?: VerificationCapability;
|
||||
|
||||
constructor(options: DispatchingVerificationOptions) {
|
||||
this.testChannel = options.testChannel;
|
||||
this.appChannel = options.appChannel;
|
||||
}
|
||||
|
||||
async verifyBehavioralAssertion(request: VerificationRequest): Promise<VerificationOutcome> {
|
||||
const channel: VerificationChannel = request.channel ?? "test";
|
||||
|
||||
if (channel === "test") {
|
||||
return this.testChannel.verifyBehavioralAssertion(request);
|
||||
}
|
||||
|
||||
if (channel === "app") {
|
||||
return this.runApp(request);
|
||||
}
|
||||
|
||||
// channel === "both": passes only when both confirm.
|
||||
const [testOutcome, appOutcome] = await Promise.all([
|
||||
this.testChannel.verifyBehavioralAssertion(request),
|
||||
this.runApp(request),
|
||||
]);
|
||||
return combineBoth(request.assertionId, testOutcome, appOutcome);
|
||||
}
|
||||
|
||||
private runApp(request: VerificationRequest): Promise<VerificationOutcome> {
|
||||
if (!this.appChannel) {
|
||||
return Promise.resolve({
|
||||
verdict: "inconclusive",
|
||||
assertionId: request.assertionId,
|
||||
reason: "app-driving channel not available in this verification run",
|
||||
});
|
||||
}
|
||||
return this.appChannel.verifyBehavioralAssertion(request);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Combine the two channels for a `both` assertion. Any fail dominates; absent a
|
||||
* fail, any inconclusive dominates; only two passes confirm.
|
||||
*/
|
||||
export function combineBoth(
|
||||
assertionId: string,
|
||||
testOutcome: VerificationOutcome,
|
||||
appOutcome: VerificationOutcome,
|
||||
): VerificationOutcome {
|
||||
const fail = [testOutcome, appOutcome].find((o) => o.verdict === "fail");
|
||||
if (fail) {
|
||||
return {
|
||||
verdict: "fail",
|
||||
assertionId,
|
||||
reason: `combined verification failed: ${fail.reason}`,
|
||||
detail: fail.detail,
|
||||
};
|
||||
}
|
||||
const inconclusive = [testOutcome, appOutcome].find((o) => o.verdict === "inconclusive");
|
||||
if (inconclusive) {
|
||||
return {
|
||||
verdict: "inconclusive",
|
||||
assertionId,
|
||||
reason: `combined verification inconclusive: ${inconclusive.reason}`,
|
||||
detail: inconclusive.detail,
|
||||
};
|
||||
}
|
||||
return {
|
||||
verdict: "pass",
|
||||
assertionId,
|
||||
reason: "both the test-execution and app-driving channels confirmed the behavior",
|
||||
};
|
||||
}
|
||||
@@ -24,6 +24,24 @@ export interface VerificationCommandResult {
|
||||
success: boolean;
|
||||
/** True when this result was satisfied from the verification cache rather than running the command. */
|
||||
cached?: boolean;
|
||||
/**
|
||||
* True when the command was terminated by the wallclock timeout rather than
|
||||
* producing a real test/build verdict. Lets callers tell an *infrastructure*
|
||||
* failure (timeout) apart from a genuinely failing test (`success === false`
|
||||
* with a real exit code).
|
||||
*/
|
||||
timedOut?: boolean;
|
||||
/**
|
||||
* True when the command was aborted via the supplied `AbortSignal`. Like
|
||||
* {@link timedOut}, this is an infra outcome, not behavioral evidence.
|
||||
*/
|
||||
aborted?: boolean;
|
||||
/**
|
||||
* True when the command could not be executed at all (spawn/setup failure,
|
||||
* sandbox error) — distinct from a command that ran and exited non-zero. An
|
||||
* infra outcome, not behavioral evidence.
|
||||
*/
|
||||
executionError?: boolean;
|
||||
}
|
||||
|
||||
/** Result of running all verification commands */
|
||||
@@ -122,8 +140,14 @@ function toLegacyExecResult(
|
||||
export async function execWithProcessGroup(
|
||||
command: string,
|
||||
options: SandboxRunStreamingOptions,
|
||||
/**
|
||||
* Explicit sandbox backend to run under. When omitted, falls back to the
|
||||
* process-global resolution. Callers that must pin an isolating backend under
|
||||
* concurrency (e.g. mission behavioral verification) pass it explicitly so they
|
||||
* never depend on mutable global state.
|
||||
*/
|
||||
backend: SandboxBackend = getSandboxBackend(),
|
||||
): Promise<{ stdout: string; stderr: string; bufferOverflow: boolean; aborted?: boolean }> {
|
||||
const backend = getSandboxBackend();
|
||||
const result = await backend.runStreaming(command, options);
|
||||
return toLegacyExecResult(command, result);
|
||||
}
|
||||
@@ -302,6 +326,12 @@ export async function runVerificationCommand(
|
||||
extraEnv?: NodeJS.ProcessEnv,
|
||||
/** Optional project-level per-command timeout override in milliseconds. Values <= 0 preserve the legacy default. */
|
||||
timeoutMsOverride?: number,
|
||||
/**
|
||||
* Optional explicit sandbox backend. When omitted, the process-global backend
|
||||
* is resolved. Pass this to pin an isolating backend without mutating global
|
||||
* state (required for safe concurrent verification — see mission-verification).
|
||||
*/
|
||||
backend?: SandboxBackend,
|
||||
): Promise<VerificationCommandResult> {
|
||||
const logger = log ?? { log: console.log, error: console.error, warn: console.warn };
|
||||
const label = (agentLabel ?? "merger") as AgentRole;
|
||||
@@ -335,13 +365,17 @@ export async function runVerificationCommand(
|
||||
: VERIFICATION_COMMAND_TIMEOUT_MS;
|
||||
const timeoutMs = Math.min(rawTimeoutMs, VERIFICATION_COMMAND_HARD_CAP_MS);
|
||||
try {
|
||||
const { stdout, stderr, bufferOverflow } = await execWithProcessGroup(command, {
|
||||
cwd: rootDir,
|
||||
timeout: timeoutMs,
|
||||
maxBuffer: VERIFICATION_COMMAND_MAX_BUFFER,
|
||||
signal,
|
||||
...(extraEnv !== undefined && { env: extraEnv }),
|
||||
});
|
||||
const { stdout, stderr, bufferOverflow } = await execWithProcessGroup(
|
||||
command,
|
||||
{
|
||||
cwd: rootDir,
|
||||
timeout: timeoutMs,
|
||||
maxBuffer: VERIFICATION_COMMAND_MAX_BUFFER,
|
||||
signal,
|
||||
...(extraEnv !== undefined && { env: extraEnv }),
|
||||
},
|
||||
backend ?? getSandboxBackend(),
|
||||
);
|
||||
|
||||
if (signal?.aborted) {
|
||||
throw Object.assign(
|
||||
@@ -402,6 +436,22 @@ export async function runVerificationCommand(
|
||||
|| String(err?.message ?? "").includes("maxBuffer");
|
||||
result.success = maxBufferExceeded && result.exitCode === 0;
|
||||
|
||||
// Classify infra outcomes so callers can tell a timeout/abort/setup failure
|
||||
// apart from a real failing test (a command that ran and exited non-zero).
|
||||
// A real test failure carries a numeric exit code; these do not.
|
||||
if (!result.success && !maxBufferExceeded) {
|
||||
const errish = err as { code?: number | string; killed?: boolean; aborted?: boolean };
|
||||
if (errish.code === "ETIMEDOUT" || (errish.killed && result.exitCode === null)) {
|
||||
result.timedOut = true;
|
||||
} else if (errish.code === "ABORT_ERR" || errish.aborted) {
|
||||
result.aborted = true;
|
||||
} else if (result.exitCode === null) {
|
||||
// No exit code and not a recognized success → the command could not be
|
||||
// run to a real verdict (spawn/setup/sandbox error), not a test failure.
|
||||
result.executionError = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (result.success) {
|
||||
logger.log(`${taskId}: ${type} command succeeded (exit 0, output exceeded buffer) in ${verificationDurationMs}ms`);
|
||||
await store.logEntry(
|
||||
|
||||
@@ -83,6 +83,11 @@ export default defineConfig({
|
||||
"src/__tests__/workflow-node-handlers.test.ts",
|
||||
"src/__tests__/workflow-policy-ownership-map.test.ts",
|
||||
],
|
||||
// No per-file quarantine excludes needed here: engine-core's
|
||||
// membership is the explicit include allow-list above, so any
|
||||
// quarantined file (e.g. merger-file-scope-invariant.test.ts) is
|
||||
// already absent. The quarantine excludes live in engine-default,
|
||||
// whose `src/**/*.test.ts` glob is what would otherwise pick them up.
|
||||
exclude: [
|
||||
"node_modules/**",
|
||||
"dist/**",
|
||||
|
||||
@@ -15,7 +15,8 @@
|
||||
"test": "vitest run --silent=passed-only --reporter=dot"
|
||||
},
|
||||
"dependencies": {
|
||||
"@fusion/plugin-sdk": "workspace:*"
|
||||
"@fusion/plugin-sdk": "workspace:*",
|
||||
"playwright-core": "^1.60.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^25.5.2",
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import plugin from "../index.js";
|
||||
import { AGENT_BROWSER_TOOLS } from "../tools.js";
|
||||
|
||||
// The engine exposes ONLY `plugin.tools` to coding-agent sessions
|
||||
// (`pluginLoader.getPluginTools()`). The verification driver must therefore
|
||||
// never appear in that array — it is reachable only via the direct
|
||||
// `launchBrowserDriver` export the engine imports inside the verification run.
|
||||
describe("driver scope — not exposed to coding-agent sessions", () => {
|
||||
const driverToolNames = ["browser_navigate", "browser_interact", "browser_observe", "browser_driver"];
|
||||
|
||||
it("plugin.tools contains only the coding-agent metadata tool, not the driver", () => {
|
||||
const toolNames = (plugin.tools ?? []).map((t) => t.name);
|
||||
expect(toolNames).toEqual(["browser_fetch_metadata"]);
|
||||
});
|
||||
|
||||
it("AGENT_BROWSER_TOOLS does not register any navigate/interact/observe driver tool", () => {
|
||||
const names = AGENT_BROWSER_TOOLS.map((t) => t.name);
|
||||
for (const driverName of driverToolNames) {
|
||||
expect(names).not.toContain(driverName);
|
||||
}
|
||||
});
|
||||
|
||||
it("the verification driver is exported as a direct capability, not a registered tool", async () => {
|
||||
const mod = await import("../index.js");
|
||||
expect(typeof mod.launchBrowserDriver).toBe("function");
|
||||
// It is a plain function export, not surfaced through any tool registry.
|
||||
expect((plugin.tools ?? []).some((t) => t.name.includes("navigate"))).toBe(false);
|
||||
});
|
||||
});
|
||||
237
plugins/fusion-plugin-agent-browser/src/__tests__/driver.test.ts
Normal file
237
plugins/fusion-plugin-agent-browser/src/__tests__/driver.test.ts
Normal file
@@ -0,0 +1,237 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
launchBrowserDriver,
|
||||
type AutomationBrowser,
|
||||
type AutomationContext,
|
||||
type AutomationElement,
|
||||
type AutomationPage,
|
||||
type BrowserAutomationClient,
|
||||
} from "../driver.js";
|
||||
|
||||
// A mocked element/page/context/browser stack. Real browser automation is NOT
|
||||
// exercised in the merge gate — only the driver's wiring against this mock is.
|
||||
function makeMockStack(overrides?: {
|
||||
selectorResolver?: (selector: string) => AutomationElement | null | "throw" | "throw-non-timeout";
|
||||
pageUrl?: string;
|
||||
}) {
|
||||
const clickSpy = vi.fn(async () => {});
|
||||
const fillSpy = vi.fn(async () => {});
|
||||
const textSpy = vi.fn(async () => "Bug is fixed");
|
||||
|
||||
const element: AutomationElement = {
|
||||
click: clickSpy,
|
||||
fill: fillSpy,
|
||||
textContent: textSpy,
|
||||
};
|
||||
|
||||
const gotoSpy = vi.fn(async () => ({}));
|
||||
const waitForSelectorSpy = vi.fn(async (selector: string) => {
|
||||
const r = overrides?.selectorResolver ? overrides.selectorResolver(selector) : element;
|
||||
if (r === "throw") {
|
||||
// Mimic Playwright's TimeoutError (identified by `name`): the selector
|
||||
// never appeared → a real `absent` observation.
|
||||
const timeoutErr = new Error("Timeout 10000ms exceeded waiting for selector");
|
||||
timeoutErr.name = "TimeoutError";
|
||||
throw timeoutErr;
|
||||
}
|
||||
if (r === "throw-non-timeout") {
|
||||
// A non-timeout fault (e.g. a malformed selector): NOT absence.
|
||||
throw new Error("Unknown engine 'bogus' while parsing selector");
|
||||
}
|
||||
return r;
|
||||
});
|
||||
|
||||
const page: AutomationPage = {
|
||||
goto: gotoSpy,
|
||||
waitForSelector: waitForSelectorSpy,
|
||||
innerText: vi.fn(async () => ""),
|
||||
url: () => overrides?.pageUrl ?? "http://127.0.0.1:54321/board",
|
||||
};
|
||||
|
||||
const pageCloseDeps = { contextClose: vi.fn(async () => {}), browserClose: vi.fn(async () => {}) };
|
||||
|
||||
const context: AutomationContext = {
|
||||
newPage: vi.fn(async () => page),
|
||||
close: pageCloseDeps.contextClose,
|
||||
};
|
||||
|
||||
const browser: AutomationBrowser = {
|
||||
newContext: vi.fn(async () => context),
|
||||
close: pageCloseDeps.browserClose,
|
||||
};
|
||||
|
||||
const launchSpy = vi.fn(async () => browser);
|
||||
const client: BrowserAutomationClient = { launch: launchSpy };
|
||||
|
||||
return {
|
||||
client,
|
||||
spies: { launchSpy, gotoSpy, waitForSelectorSpy, clickSpy, fillSpy, textSpy, ...pageCloseDeps },
|
||||
};
|
||||
}
|
||||
|
||||
// An env with an explicit executable so the probe always "finds" a browser in
|
||||
// tests (executablePath is passed straight through when provided to the probe,
|
||||
// but the probe still verifies existence — so we instead inject the client and
|
||||
// point at this test file as the "executable", which exists and is readable).
|
||||
// To force the available path deterministically we pass `executablePath` of a
|
||||
// real file and rely on access(X_OK); on POSIX the test file may not be +x, so
|
||||
// we instead bypass discovery by asserting the unavailable path separately and,
|
||||
// for the "available" cases, point executablePath at a path that exists & is
|
||||
// executable: the node binary itself.
|
||||
const NODE_BIN = process.execPath;
|
||||
|
||||
describe("browser driver — availability / inconclusive", () => {
|
||||
it("reports inconclusive (browser-unavailable) when no executable is found", async () => {
|
||||
const { client } = makeMockStack();
|
||||
const result = await launchBrowserDriver({
|
||||
client,
|
||||
executablePath: "/nonexistent/path/to/chrome-does-not-exist",
|
||||
});
|
||||
expect(result.status).toBe("inconclusive");
|
||||
if (result.status === "inconclusive") {
|
||||
expect(result.reason).toBe("browser-unavailable");
|
||||
}
|
||||
});
|
||||
|
||||
it("does not launch the client when the browser is unavailable", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
await launchBrowserDriver({ client, executablePath: "/nope/chrome" });
|
||||
expect(spies.launchSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser driver — navigate / interact / observe (mocked client)", () => {
|
||||
it("launches against the discovered executable and headless flag", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const result = await launchBrowserDriver({ client, executablePath: NODE_BIN, headless: true });
|
||||
expect(result.status).toBe("ready");
|
||||
expect(spies.launchSpy).toHaveBeenCalledWith({ executablePath: NODE_BIN, headless: true });
|
||||
});
|
||||
|
||||
it("navigate calls page.goto and returns ok with the landed url", async () => {
|
||||
const { client, spies } = makeMockStack({ pageUrl: "http://127.0.0.1:9/board" });
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
expect(launched.status).toBe("ready");
|
||||
if (launched.status !== "ready") return;
|
||||
const nav = await launched.session.navigate("http://127.0.0.1:9/board");
|
||||
expect(spies.gotoSpy).toHaveBeenCalledWith(
|
||||
"http://127.0.0.1:9/board",
|
||||
expect.objectContaining({ waitUntil: "load" }),
|
||||
);
|
||||
expect(nav).toEqual({ status: "ok", url: "http://127.0.0.1:9/board" });
|
||||
});
|
||||
|
||||
it("click resolves the selector and clicks the element", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.click("#fix-button");
|
||||
expect(spies.waitForSelectorSpy).toHaveBeenCalledWith("#fix-button", expect.any(Object));
|
||||
expect(spies.clickSpy).toHaveBeenCalledTimes(1);
|
||||
expect(out).toEqual({ status: "ok" });
|
||||
});
|
||||
|
||||
it("type resolves the selector and fills the element", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.type("input[name=q]", "hello");
|
||||
expect(spies.fillSpy).toHaveBeenCalledWith("hello");
|
||||
expect(out).toEqual({ status: "ok" });
|
||||
});
|
||||
|
||||
it("observe returns found with the element text (reproduces a UI behavior)", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.observe(".status");
|
||||
expect(spies.textSpy).toHaveBeenCalled();
|
||||
expect(out).toEqual({ status: "found", text: "Bug is fixed", url: "http://127.0.0.1:54321/board" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser driver — un-exercisable assertion → inconclusive (not fail)", () => {
|
||||
it("click on an unreachable selector resolves to inconclusive/selector-unreachable", async () => {
|
||||
const { client } = makeMockStack({ selectorResolver: () => "throw" });
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.click("#never-here");
|
||||
expect(out.status).toBe("inconclusive");
|
||||
if (out.status === "inconclusive") expect(out.reason).toBe("selector-unreachable");
|
||||
});
|
||||
|
||||
it("navigation failure resolves to inconclusive/navigation-failed (never fail)", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
spies.gotoSpy.mockRejectedValueOnce(new Error("net::ERR_CONNECTION_REFUSED"));
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.navigate("http://127.0.0.1:1/dead");
|
||||
expect(out.status).toBe("inconclusive");
|
||||
if (out.status === "inconclusive") expect(out.reason).toBe("navigation-failed");
|
||||
});
|
||||
|
||||
it("setup failure (browser launch throws) resolves to inconclusive/setup-failed", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
spies.launchSpy.mockRejectedValueOnce(new Error("spawn chrome ENOENT"));
|
||||
const result = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
expect(result.status).toBe("inconclusive");
|
||||
if (result.status === "inconclusive") expect(result.reason).toBe("setup-failed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser driver — absence is a real observation, not inconclusive", () => {
|
||||
it("observe of a missing selector resolves to absent (distinct from inconclusive)", async () => {
|
||||
const { client } = makeMockStack({ selectorResolver: () => "throw" });
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.observe(".gone");
|
||||
expect(out.status).toBe("absent");
|
||||
});
|
||||
|
||||
it("observe returning null element resolves to absent", async () => {
|
||||
const { client } = makeMockStack({ selectorResolver: () => null });
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.observe(".gone");
|
||||
expect(out.status).toBe("absent");
|
||||
});
|
||||
|
||||
it("observe of a non-timeout waitForSelector fault resolves to inconclusive (not absent)", async () => {
|
||||
const { client } = makeMockStack({ selectorResolver: () => "throw-non-timeout" });
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
const out = await launched.session.observe("css=bogus>>>nonsense");
|
||||
expect(out.status).toBe("inconclusive");
|
||||
if (out.status === "inconclusive") expect(out.reason).toBe("driver-error");
|
||||
});
|
||||
});
|
||||
|
||||
describe("browser driver — teardown", () => {
|
||||
it("dispose closes the context and the browser", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
await launched.session.dispose();
|
||||
expect(spies.contextClose).toHaveBeenCalledTimes(1);
|
||||
expect(spies.browserClose).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("dispose is idempotent (second call does not double-close)", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
await launched.session.dispose();
|
||||
await launched.session.dispose();
|
||||
expect(spies.contextClose).toHaveBeenCalledTimes(1);
|
||||
expect(spies.browserClose).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("dispose tolerates a close() that throws (teardown is best-effort)", async () => {
|
||||
const { client, spies } = makeMockStack();
|
||||
spies.contextClose.mockRejectedValueOnce(new Error("already closed"));
|
||||
const launched = await launchBrowserDriver({ client, executablePath: NODE_BIN });
|
||||
if (launched.status !== "ready") throw new Error("expected ready");
|
||||
await expect(launched.session.dispose()).resolves.toBeUndefined();
|
||||
expect(spies.browserClose).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,35 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { probeBrowserExecutable } from "../probe.js";
|
||||
|
||||
describe("probeBrowserExecutable", () => {
|
||||
it("reports unavailable with a reason when an explicit path does not exist", async () => {
|
||||
const result = await probeBrowserExecutable({ executablePath: "/definitely/not/a/browser/here" });
|
||||
expect(result.available).toBe(false);
|
||||
expect(result.reason).toContain("/definitely/not/a/browser/here");
|
||||
});
|
||||
|
||||
it("accepts an explicit executable path that exists and is executable", async () => {
|
||||
// process.execPath (node itself) exists and is executable on all platforms.
|
||||
const result = await probeBrowserExecutable({ executablePath: process.execPath });
|
||||
expect(result.available).toBe(true);
|
||||
expect(result.executablePath).toBe(process.execPath);
|
||||
});
|
||||
|
||||
it("honors the FUSION_BROWSER_EXECUTABLE env override", async () => {
|
||||
const result = await probeBrowserExecutable({ env: { FUSION_BROWSER_EXECUTABLE: process.execPath } as NodeJS.ProcessEnv });
|
||||
expect(result.available).toBe(true);
|
||||
expect(result.executablePath).toBe(process.execPath);
|
||||
});
|
||||
|
||||
it("reports unavailable (not throw) when nothing is discoverable", async () => {
|
||||
// An env with no overrides and PATH that cannot resolve browser binaries.
|
||||
const result = await probeBrowserExecutable({
|
||||
env: { PATH: "/nonexistent-bin-dir" } as NodeJS.ProcessEnv,
|
||||
});
|
||||
// On a CI box without a system Chrome this is false; if a system Chrome is at
|
||||
// a well-known path it could be true. Either way it must not throw and must
|
||||
// carry a coherent shape.
|
||||
expect(typeof result.available).toBe("boolean");
|
||||
if (!result.available) expect(result.reason).toBeTruthy();
|
||||
});
|
||||
});
|
||||
343
plugins/fusion-plugin-agent-browser/src/driver.ts
Normal file
343
plugins/fusion-plugin-agent-browser/src/driver.ts
Normal file
@@ -0,0 +1,343 @@
|
||||
/**
|
||||
* App/browser driver (U8).
|
||||
*
|
||||
* A REAL navigate / interact / observe driver used by the verification run
|
||||
* (U5 wires it in) to reproduce a UI/bug assertion's observable behavior against
|
||||
* the isolated app instance the U4 harness launches (`launchIsolatedApp()` →
|
||||
* `{ baseUrl, port, dbPath, clientDir, dispose() }`).
|
||||
*
|
||||
* Design constraints (see plan unit U8 / R12 and the brainstorm):
|
||||
*
|
||||
* - **No bundled browser.** Driving is done through `playwright-core`, which —
|
||||
* unlike full `playwright` — does NOT download a browser at install time. It
|
||||
* launches an EXISTING Chrome/Chromium discovered on the host via
|
||||
* `probeBrowserExecutable()` (see `probe.ts`). This keeps the install/build
|
||||
* gate fast and deterministic in CI.
|
||||
*
|
||||
* - **Graceful degradation → inconclusive.** When no browser executable is
|
||||
* available, or an assertion is structurally un-exercisable (a selector that
|
||||
* never appears, a state the driver cannot set up), the driver reports an
|
||||
* `inconclusive` outcome — NEVER a false pass or fail. The verification run
|
||||
* (U5) maps `inconclusive` to a blocked/needs-attention verdict that spawns no
|
||||
* Fix Feature (R21).
|
||||
*
|
||||
* - **Verification-scoped, not a coding-agent tool.** This capability is
|
||||
* deliberately NOT registered in the plugin's `tools` array (which is what the
|
||||
* engine exposes to coding-agent sessions via `pluginLoader.getPluginTools()`).
|
||||
* It is exported as a typed factory the engine imports directly inside the
|
||||
* verification run. A coding agent therefore can never reach navigate /
|
||||
* interact / observe; only the verification path can.
|
||||
*
|
||||
* - **Clean teardown.** Every successful `launch()` returns a session with a
|
||||
* `dispose()` that closes the page, context, and browser unconditionally and
|
||||
* idempotently — including after a mid-run failure.
|
||||
*
|
||||
* The Playwright client is injected (`BrowserAutomationClient`) so the merge-gate
|
||||
* unit tests drive a mock; real browser automation is exercised only in a manual
|
||||
* smoke / heavier lane, never in the merge gate.
|
||||
*/
|
||||
|
||||
import { probeBrowserExecutable } from "./probe.js";
|
||||
|
||||
// ── Minimal automation-client surface (the slice of playwright-core we use) ────
|
||||
//
|
||||
// Declared structurally so tests can supply a mock without importing
|
||||
// playwright-core, and so the driver does not couple to playwright's full type
|
||||
// surface. The real client is built lazily from playwright-core in
|
||||
// `createPlaywrightClient()`.
|
||||
|
||||
/** A located element handle (opaque to the driver beyond the methods used). */
|
||||
export interface AutomationElement {
|
||||
click(): Promise<void>;
|
||||
fill(value: string): Promise<void>;
|
||||
textContent(): Promise<string | null>;
|
||||
}
|
||||
|
||||
/** A single page/tab the driver navigates and observes. */
|
||||
export interface AutomationPage {
|
||||
goto(url: string, opts?: { timeout?: number; waitUntil?: string }): Promise<unknown>;
|
||||
/** Resolve a selector to an element, waiting up to `timeout` ms. Null when it never appears. */
|
||||
waitForSelector(selector: string, opts?: { timeout?: number; state?: string }): Promise<AutomationElement | null>;
|
||||
/** Read the visible text of the whole document body. */
|
||||
innerText(selector: string): Promise<string>;
|
||||
url(): string;
|
||||
}
|
||||
|
||||
/** A browser context (isolated cookie/storage jar) holding pages. */
|
||||
export interface AutomationContext {
|
||||
newPage(): Promise<AutomationPage>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
/** A launched browser process. */
|
||||
export interface AutomationBrowser {
|
||||
newContext(): Promise<AutomationContext>;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
/** The injectable automation backend (real = playwright-core, test = mock). */
|
||||
export interface BrowserAutomationClient {
|
||||
launch(opts: { executablePath: string; headless: boolean }): Promise<AutomationBrowser>;
|
||||
}
|
||||
|
||||
// ── Driver result types ────────────────────────────────────────────────────
|
||||
|
||||
/** Why a driver operation could not reach a definitive observation. */
|
||||
export type InconclusiveReason =
|
||||
| "browser-unavailable"
|
||||
| "selector-unreachable"
|
||||
| "navigation-failed"
|
||||
| "setup-failed"
|
||||
| "driver-error";
|
||||
|
||||
export interface ObserveOutcomeFound {
|
||||
status: "found";
|
||||
/** The text content of the observed element/selector. */
|
||||
text: string;
|
||||
/** The URL the observation was made against. */
|
||||
url: string;
|
||||
}
|
||||
|
||||
export interface ObserveOutcomeAbsent {
|
||||
status: "absent";
|
||||
url: string;
|
||||
}
|
||||
|
||||
export interface OperationInconclusive {
|
||||
status: "inconclusive";
|
||||
reason: InconclusiveReason;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export type ObserveOutcome = ObserveOutcomeFound | ObserveOutcomeAbsent | OperationInconclusive;
|
||||
export type InteractOutcome = { status: "ok" } | OperationInconclusive;
|
||||
export type NavigateOutcome = { status: "ok"; url: string } | OperationInconclusive;
|
||||
|
||||
/** Result of attempting to obtain a driver session. */
|
||||
export type DriverLaunchResult =
|
||||
| { status: "ready"; session: BrowserDriverSession }
|
||||
| OperationInconclusive;
|
||||
|
||||
const DEFAULT_OP_TIMEOUT_MS = 10_000;
|
||||
|
||||
/**
|
||||
* A live driver session bound to a single browser/context/page targeting the
|
||||
* isolated app instance. All operations degrade to `inconclusive` rather than
|
||||
* throwing, so a fragile UI never manufactures a false fail.
|
||||
*/
|
||||
export interface BrowserDriverSession {
|
||||
/** Navigate to a URL (typically `${baseUrl}${path}` of the isolated app). */
|
||||
navigate(url: string, opts?: { timeoutMs?: number }): Promise<NavigateOutcome>;
|
||||
/** Click the first element matching `selector`. */
|
||||
click(selector: string, opts?: { timeoutMs?: number }): Promise<InteractOutcome>;
|
||||
/** Type `value` into the first element matching `selector`. */
|
||||
type(selector: string, value: string, opts?: { timeoutMs?: number }): Promise<InteractOutcome>;
|
||||
/**
|
||||
* Observe whether `selector` is present and read its text. A selector that
|
||||
* never appears within the timeout resolves to `absent` (a real negative
|
||||
* observation), distinct from an `inconclusive` driver/setup failure.
|
||||
*/
|
||||
observe(selector: string, opts?: { timeoutMs?: number; expectAbsent?: boolean }): Promise<ObserveOutcome>;
|
||||
/** Close page/context/browser unconditionally. Idempotent. */
|
||||
dispose(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface LaunchDriverOptions {
|
||||
/** Injected automation backend; defaults to the real playwright-core client. */
|
||||
client?: BrowserAutomationClient;
|
||||
/** Explicit Chrome/Chromium executable path (else discovered via probe). */
|
||||
executablePath?: string;
|
||||
/** Run headless (default true). */
|
||||
headless?: boolean;
|
||||
/** Env used for executable discovery (defaults to process.env). */
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire a browser driver session, or report why one could not be acquired.
|
||||
*
|
||||
* Returns `{ status: "inconclusive", reason: "browser-unavailable" }` when no
|
||||
* Chrome/Chromium executable is found (R12 graceful degradation) — the caller
|
||||
* (U5) treats that as INCONCLUSIVE, never a pass/fail.
|
||||
*/
|
||||
export async function launchBrowserDriver(opts: LaunchDriverOptions = {}): Promise<DriverLaunchResult> {
|
||||
const probe = await probeBrowserExecutable({ executablePath: opts.executablePath, env: opts.env });
|
||||
if (!probe.available || !probe.executablePath) {
|
||||
return {
|
||||
status: "inconclusive",
|
||||
reason: "browser-unavailable",
|
||||
detail: probe.reason ?? "no browser executable available",
|
||||
};
|
||||
}
|
||||
|
||||
const client = opts.client ?? (await createPlaywrightClient());
|
||||
if (!client) {
|
||||
return {
|
||||
status: "inconclusive",
|
||||
reason: "browser-unavailable",
|
||||
detail: "playwright-core automation client could not be loaded",
|
||||
};
|
||||
}
|
||||
|
||||
let browser: AutomationBrowser | undefined;
|
||||
let context: AutomationContext | undefined;
|
||||
let page: AutomationPage | undefined;
|
||||
try {
|
||||
browser = await client.launch({ executablePath: probe.executablePath, headless: opts.headless ?? true });
|
||||
context = await browser.newContext();
|
||||
page = await context.newPage();
|
||||
} catch (err) {
|
||||
// Best-effort teardown of whatever was created before the failure.
|
||||
await safeClose(context);
|
||||
await safeClose(browser);
|
||||
return {
|
||||
status: "inconclusive",
|
||||
reason: "setup-failed",
|
||||
detail: `failed to launch browser session: ${errMsg(err)}`,
|
||||
};
|
||||
}
|
||||
|
||||
const session = makeSession(browser, context, page);
|
||||
return { status: "ready", session };
|
||||
}
|
||||
|
||||
function makeSession(browser: AutomationBrowser, context: AutomationContext, page: AutomationPage): BrowserDriverSession {
|
||||
let disposed = false;
|
||||
|
||||
return {
|
||||
async navigate(url, navOpts) {
|
||||
try {
|
||||
await page.goto(url, { timeout: navOpts?.timeoutMs ?? DEFAULT_OP_TIMEOUT_MS, waitUntil: "load" });
|
||||
return { status: "ok", url: page.url() };
|
||||
} catch (err) {
|
||||
return { status: "inconclusive", reason: "navigation-failed", detail: `goto ${url} failed: ${errMsg(err)}` };
|
||||
}
|
||||
},
|
||||
|
||||
async click(selector, opOpts) {
|
||||
const el = await locate(page, selector, opOpts?.timeoutMs);
|
||||
if (el === "inconclusive") {
|
||||
return { status: "inconclusive", reason: "selector-unreachable", detail: `click target not found: ${selector}` };
|
||||
}
|
||||
try {
|
||||
await el.click();
|
||||
return { status: "ok" };
|
||||
} catch (err) {
|
||||
return { status: "inconclusive", reason: "driver-error", detail: `click ${selector} failed: ${errMsg(err)}` };
|
||||
}
|
||||
},
|
||||
|
||||
async type(selector, value, opOpts) {
|
||||
const el = await locate(page, selector, opOpts?.timeoutMs);
|
||||
if (el === "inconclusive") {
|
||||
return { status: "inconclusive", reason: "selector-unreachable", detail: `type target not found: ${selector}` };
|
||||
}
|
||||
try {
|
||||
await el.fill(value);
|
||||
return { status: "ok" };
|
||||
} catch (err) {
|
||||
return { status: "inconclusive", reason: "driver-error", detail: `type into ${selector} failed: ${errMsg(err)}` };
|
||||
}
|
||||
},
|
||||
|
||||
async observe(selector, obsOpts) {
|
||||
const timeout = obsOpts?.timeoutMs ?? DEFAULT_OP_TIMEOUT_MS;
|
||||
// When asserting absence, a missing selector is a real `absent` observation,
|
||||
// not an inconclusive failure.
|
||||
let el: AutomationElement | null;
|
||||
try {
|
||||
el = await page.waitForSelector(selector, { timeout, state: obsOpts?.expectAbsent ? "attached" : "visible" });
|
||||
} catch (err) {
|
||||
// Only a Playwright TimeoutError means the element never appeared — a real
|
||||
// `absent` observation. Any other rejection (e.g. a malformed selector or
|
||||
// invalid options) is a driver fault, not a negative observation, and must
|
||||
// surface as `inconclusive` rather than be silently misreported as absent.
|
||||
if (isTimeoutError(err)) {
|
||||
return { status: "absent", url: page.url() };
|
||||
}
|
||||
return {
|
||||
status: "inconclusive",
|
||||
reason: "driver-error",
|
||||
detail: `observe ${selector} failed at ${page.url()}: ${errMsg(err)}`,
|
||||
};
|
||||
}
|
||||
if (!el) return { status: "absent", url: page.url() };
|
||||
try {
|
||||
const text = (await el.textContent()) ?? "";
|
||||
return { status: "found", text, url: page.url() };
|
||||
} catch (err) {
|
||||
return { status: "inconclusive", reason: "driver-error", detail: `read ${selector} failed: ${errMsg(err)}` };
|
||||
}
|
||||
},
|
||||
|
||||
async dispose() {
|
||||
if (disposed) return;
|
||||
disposed = true;
|
||||
await safeClose(context);
|
||||
await safeClose(browser);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a selector to an element, or signal `"inconclusive"` when it never
|
||||
* appears within the timeout. Distinct from `observe`, which treats absence as a
|
||||
* first-class negative observation.
|
||||
*/
|
||||
async function locate(
|
||||
page: AutomationPage,
|
||||
selector: string,
|
||||
timeoutMs?: number,
|
||||
): Promise<AutomationElement | "inconclusive"> {
|
||||
try {
|
||||
const el = await page.waitForSelector(selector, { timeout: timeoutMs ?? DEFAULT_OP_TIMEOUT_MS, state: "visible" });
|
||||
return el ?? "inconclusive";
|
||||
} catch {
|
||||
return "inconclusive";
|
||||
}
|
||||
}
|
||||
|
||||
async function safeClose(closable: { close(): Promise<void> } | undefined): Promise<void> {
|
||||
if (!closable) return;
|
||||
try {
|
||||
await closable.close();
|
||||
} catch {
|
||||
// teardown is best-effort and must never throw
|
||||
}
|
||||
}
|
||||
|
||||
function errMsg(err: unknown): string {
|
||||
return err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
|
||||
/**
|
||||
* True only for Playwright's `TimeoutError`, which signals the selector never
|
||||
* appeared within the timeout. Detected structurally by `name` so the driver
|
||||
* stays decoupled from playwright-core's type surface (the real client is loaded
|
||||
* lazily, and tests inject a mock that never imports it). `waitForSelector` can
|
||||
* also reject for non-timeout reasons (malformed selectors, invalid options);
|
||||
* those are NOT absence and must not be mapped to `absent`.
|
||||
*/
|
||||
function isTimeoutError(err: unknown): boolean {
|
||||
return err instanceof Error && err.name === "TimeoutError";
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the real automation client from `playwright-core`, adapting its
|
||||
* chromium API to the structural `BrowserAutomationClient` surface. Imported
|
||||
* lazily so merge-gate unit tests (which inject a mock) never load
|
||||
* playwright-core, and so a missing/broken playwright-core degrades to
|
||||
* `undefined` (→ inconclusive) instead of throwing at module load.
|
||||
*/
|
||||
export async function createPlaywrightClient(): Promise<BrowserAutomationClient | undefined> {
|
||||
try {
|
||||
const pw = (await import("playwright-core")) as unknown as {
|
||||
chromium: { launch(opts: { executablePath: string; headless: boolean }): Promise<AutomationBrowser> };
|
||||
};
|
||||
return {
|
||||
launch: (opts) => pw.chromium.launch(opts),
|
||||
};
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
@@ -65,3 +65,27 @@ const plugin: FusionPlugin = definePlugin({
|
||||
});
|
||||
|
||||
export default plugin;
|
||||
|
||||
// ── Verification-scoped app/browser driver (U8) ───────────────────────────────
|
||||
//
|
||||
// The navigate/interact/observe driver is exported as a typed capability the
|
||||
// engine imports DIRECTLY inside the verification run (U5). It is intentionally
|
||||
// NOT added to `plugin.tools` — the engine only exposes `plugin.tools` to
|
||||
// coding-agent sessions (`pluginLoader.getPluginTools()`), so keeping the driver
|
||||
// out of that array is what scopes it to verification and keeps it unreachable
|
||||
// from normal coding sessions. `browser_fetch_metadata` remains the only
|
||||
// coding-agent-facing tool.
|
||||
export {
|
||||
launchBrowserDriver,
|
||||
createPlaywrightClient,
|
||||
type BrowserDriverSession,
|
||||
type BrowserAutomationClient,
|
||||
type DriverLaunchResult,
|
||||
type NavigateOutcome,
|
||||
type InteractOutcome,
|
||||
type ObserveOutcome,
|
||||
type OperationInconclusive,
|
||||
type InconclusiveReason,
|
||||
type LaunchDriverOptions,
|
||||
} from "./driver.js";
|
||||
export { probeBrowserExecutable, type BrowserExecutableProbeResult } from "./probe.js";
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { access, constants } from "node:fs/promises";
|
||||
|
||||
export interface AgentBrowserProbeResult {
|
||||
available: boolean;
|
||||
@@ -56,6 +57,110 @@ export async function probeAgentBrowserBinary(opts?: { binaryPath?: string; time
|
||||
});
|
||||
}
|
||||
|
||||
// ── Chromium/Chrome executable discovery (for the verification driver) ─────────
|
||||
//
|
||||
// The app/browser driver (U8) drives a Chromium engine via playwright-core, which
|
||||
// does NOT bundle or download a browser. It launches an EXISTING Chrome/Chromium
|
||||
// discovered on the host. When no executable can be found the driver must report
|
||||
// itself unavailable so the verification run resolves the assertion to
|
||||
// INCONCLUSIVE (never a false pass/fail).
|
||||
|
||||
export interface BrowserExecutableProbeResult {
|
||||
/** True only when a usable Chrome/Chromium executable was located. */
|
||||
available: boolean;
|
||||
/** Absolute (or PATH-resolvable) executable path, when found. */
|
||||
executablePath?: string;
|
||||
/** Human-readable reason the executable is unavailable. */
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Well-known Chrome/Chromium executable locations, by platform. Checked in
|
||||
* order; the first that exists wins. Env overrides take precedence over these.
|
||||
*/
|
||||
function candidateBrowserPaths(env: NodeJS.ProcessEnv): string[] {
|
||||
const fromEnv = [env.FUSION_BROWSER_EXECUTABLE, env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH, env.CHROME_PATH]
|
||||
.map((v) => v?.trim())
|
||||
.filter((v): v is string => !!v && v.length > 0);
|
||||
|
||||
if (process.platform === "darwin") {
|
||||
return [
|
||||
...fromEnv,
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
"/Applications/Chromium.app/Contents/MacOS/Chromium",
|
||||
"/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
|
||||
];
|
||||
}
|
||||
if (process.platform === "win32") {
|
||||
const programFiles = env["PROGRAMFILES"] ?? "C:\\Program Files";
|
||||
const programFilesX86 = env["PROGRAMFILES(X86)"] ?? "C:\\Program Files (x86)";
|
||||
return [
|
||||
...fromEnv,
|
||||
`${programFiles}\\Google\\Chrome\\Application\\chrome.exe`,
|
||||
`${programFilesX86}\\Google\\Chrome\\Application\\chrome.exe`,
|
||||
`${programFilesX86}\\Microsoft\\Edge\\Application\\msedge.exe`,
|
||||
];
|
||||
}
|
||||
return [
|
||||
...fromEnv,
|
||||
"/usr/bin/google-chrome",
|
||||
"/usr/bin/google-chrome-stable",
|
||||
"/usr/bin/chromium",
|
||||
"/usr/bin/chromium-browser",
|
||||
"/snap/bin/chromium",
|
||||
];
|
||||
}
|
||||
|
||||
/** PATH-resolvable executable names to fall back to when no fixed path exists. */
|
||||
const BROWSER_BINARY_NAMES = ["google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "chrome"];
|
||||
|
||||
/**
|
||||
* Locate a Chrome/Chromium executable the verification driver can launch.
|
||||
*
|
||||
* Resolution order: explicit `opts.executablePath` → env overrides
|
||||
* (`FUSION_BROWSER_EXECUTABLE` / `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` /
|
||||
* `CHROME_PATH`) → well-known platform paths → PATH lookup of common binary
|
||||
* names. Returns `available: false` (with a reason) when nothing is found, so
|
||||
* the caller degrades to INCONCLUSIVE rather than failing the assertion.
|
||||
*/
|
||||
export async function probeBrowserExecutable(opts?: {
|
||||
executablePath?: string;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}): Promise<BrowserExecutableProbeResult> {
|
||||
const env = opts?.env ?? process.env;
|
||||
const explicit = opts?.executablePath?.trim();
|
||||
if (explicit) {
|
||||
if (await isExecutableFile(explicit)) return { available: true, executablePath: explicit };
|
||||
return { available: false, reason: `configured browser executable not found: ${explicit}` };
|
||||
}
|
||||
|
||||
for (const candidate of candidateBrowserPaths(env)) {
|
||||
if (await isExecutableFile(candidate)) return { available: true, executablePath: candidate };
|
||||
}
|
||||
|
||||
for (const name of BROWSER_BINARY_NAMES) {
|
||||
const resolved = await tryResolveBinaryPath(name);
|
||||
if (resolved && (await isExecutableFile(resolved))) {
|
||||
return { available: true, executablePath: resolved };
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
available: false,
|
||||
reason:
|
||||
"no Chrome/Chromium executable found (checked FUSION_BROWSER_EXECUTABLE / CHROME_PATH, well-known paths, and PATH)",
|
||||
};
|
||||
}
|
||||
|
||||
async function isExecutableFile(p: string): Promise<boolean> {
|
||||
try {
|
||||
await access(p, constants.X_OK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function tryResolveBinaryPath(binary: string): Promise<string | undefined> {
|
||||
return new Promise((resolvePromise) => {
|
||||
const which = process.platform === "win32" ? "where" : "which";
|
||||
|
||||
10
pnpm-lock.yaml
generated
10
pnpm-lock.yaml
generated
@@ -743,6 +743,9 @@ importers:
|
||||
'@fusion/plugin-sdk':
|
||||
specifier: workspace:*
|
||||
version: link:../../packages/plugin-sdk
|
||||
playwright-core:
|
||||
specifier: ^1.60.0
|
||||
version: 1.60.0
|
||||
devDependencies:
|
||||
'@types/node':
|
||||
specifier: ^25.5.2
|
||||
@@ -5971,6 +5974,11 @@ packages:
|
||||
pkg-types@1.3.1:
|
||||
resolution: {integrity: sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==}
|
||||
|
||||
playwright-core@1.60.0:
|
||||
resolution: {integrity: sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==}
|
||||
engines: {node: '>=18'}
|
||||
hasBin: true
|
||||
|
||||
plist@3.1.0:
|
||||
resolution: {integrity: sha512-uysumyrvkUX0rX/dEVqt8gC3sTBzd4zoWfLeS29nb53imdaXVvLINYXTI2GNqzaMuvacNx4uJQ8+b3zXR0pkgQ==}
|
||||
engines: {node: '>=10.4.0'}
|
||||
@@ -13216,6 +13224,8 @@ snapshots:
|
||||
mlly: 1.8.2
|
||||
pathe: 2.0.3
|
||||
|
||||
playwright-core@1.60.0: {}
|
||||
|
||||
plist@3.1.0:
|
||||
dependencies:
|
||||
'@xmldom/xmldom': 0.8.12
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
{
|
||||
"$comment": "Flaky-test quarantine ledger (deletion ratchet — see AGENTS.md 'Flaky tests: quarantine on sight' and docs/testing.md 'Quarantine ledger and the deletion ratchet'). A test observed failing without a corresponding real bug is quarantined ON SIGHT: add an entry here AND a matching one-line `exclude` entry in that package's vitest config, in the same commit. Every entry needs a non-empty `reason` (link the failing run) and a `quarantinedAt` date — the entry expires 14 days later, at which point the test file is DELETED unless someone rescues it with evidence it catches real regressions plus a root-cause fix (never appeasement). There is deliberately no loader module and no automation around this file: it is a dated record, the vitest config exclude is the mechanism, and the sweep is policy executed by whoever touches the suite.",
|
||||
"entries": [
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/session-cross-tab.test.ts",
|
||||
"reason": "FN-6690 local workspace `pnpm test` observed ENOTEMPTY while removing the test's temp .fusion directory in dashboard-api-quality-backfill shard; isolated rerun passed, indicating cleanup flake rather than a lazy-view CSS regression.",
|
||||
"quarantinedAt": "2026-06-19"
|
||||
},
|
||||
{
|
||||
"file": "packages/cli/src/__tests__/bin.test.ts",
|
||||
"reason": "FN-6795 final @runfusion/fusion package verification observed `launches dashboard when no args are provided` time out at the existing 15s budget only under the full package lane; a targeted rerun with the failing CLI files passed, so quarantine this newly observed suite-load flake on sight instead of widening timeouts or worker budgets.",
|
||||
|
||||
Reference in New Issue
Block a user