docs(verification): discourage allowFullSuite in fn_run_verification (#1772)

Agents were free to pass `allowFullSuite: true` to
`fn_run_verification`, which runs a marathon command (`pnpm test`, `pnpm
test:full`, whole-package tests) far beyond what a change needs — the
main way verification balloons past its budget.

This strengthens the guidance in three places agents read:
- **`run-verification-tool.ts`** — the `allowFullSuite` param
description now leads with "DO NOT SET THIS unless absolutely necessary"
and points to a file-scoped command.
- **`AGENTS.md`** — new standing rule: scope verification to changed
files; reserve `allowFullSuite` for cross-cutting changes with no
targetable test set; the thin merge gate is the safety net.
- **`docs/testing.md`** — same emphasis inline.

No functional change. Pairs with the file-scoped-verification work
(verification now runs only the tests affected by the diff).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- stage-review-badge-begin -->

---

<a href="https://stagereview.app/Runfusion/Fusion/pull/1772">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://stagereview.app/assets/gh-open-in-stage-dark.svg">
<img src="https://stagereview.app/assets/gh-open-in-stage-light.svg"
alt="Open in Stage">
  </picture>
</a>

<!-- stage-review-badge-end -->

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified verification guidance to strongly prefer targeted,
file-scoped test runs for changed files.
* Added clearer rules for when full-suite verification may be used,
including when to note the reason.
* Updated the `allowFullSuite` guidance to emphasize it as a last-resort
option while keeping timeout behavior unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
gsxdsm
2026-06-25 17:21:01 -07:00
committed by GitHub
3 changed files with 8 additions and 2 deletions

View File

@@ -131,6 +131,12 @@ pnpm verify:workspace # deep opt-in verification (lint -> test:full -> build);
- Do not add new real-network calls, real polling loops, or mock-the-world shells when a narrower seam exists.
- Use the testing taxonomy in `docs/testing.md` when deciding trim vs keep.
### Standing Rule: Scope Verification to Changed Files — Do Not Use `allowFullSuite`
- When verifying via `fn_run_verification`, **do not pass `allowFullSuite: true` unless absolutely necessary.** It is a last-resort escape hatch that runs a marathon command (root `pnpm test`, `pnpm test:full`, `verify:workspace`, whole-package tests, repeat loops) far in excess of what the change requires, and it is the main way verification balloons past its budget.
- Default to a **file-scoped** command targeting only the tests affected by the diff, e.g. `pnpm --filter @fusion/<pkg> exec vitest run src/path/to/changed.test.ts --silent=passed-only --reporter=dot`. The marathon soft-cap exists to push you toward this.
- `allowFullSuite: true` is justified only for a genuinely full run with no targetable test set (e.g. a cross-cutting infra change) — and then state the reason. The thin merge gate (`pnpm test:gate`) is the cross-cutting safety net, not per-task verification.
### Standing Rule: Fix the Invariant, Not the Repro (FN-5893)
- When fixing a bug, the regression test must assert the general invariant across ALL known surfaces — not only the single reported reproduction.

View File

@@ -46,7 +46,7 @@ Custom workflow reliability release signoff has a dedicated on-demand lane: `pnp
<!-- FNXC:iOSAcceptance 2026-06-18-17:25: Terminal acceptance gates that depend on real mobile Safari must use the credential-driven real-iOS surface runbook instead of treating desktop WebKit or jsdom as evidence. -->
Terminal acceptance tasks that require real mobile Safari should use [`docs/ios-acceptance.md`](./ios-acceptance.md) for the `--check` run-vs-NO-OP probe, credential wiring, and physical/cloud real-iOS evidence workflow.
Agents running verification through `fn_run_verification` are bounded by default: project `verificationCommandTimeoutMs` when set, otherwise 300s for package scope and 900s for workspace scope, with an 1800s hard cap. Marathon invocations such as root `pnpm test`, `pnpm test:full`, `pnpm verify:workspace`, whole-package tests without file filters, and shell repeat loops are soft-capped unless the agent explicitly passes `allowFullSuite: true`; the escape hatch still emits progress heartbeats and respects the hard cap. Prefer targeted commands such as `pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot` before opting into a full run.
Agents running verification through `fn_run_verification` are bounded by default: project `verificationCommandTimeoutMs` when set, otherwise 300s for package scope and 900s for workspace scope, with an 1800s hard cap. Marathon invocations such as root `pnpm test`, `pnpm test:full`, `pnpm verify:workspace`, whole-package tests without file filters, and shell repeat loops are soft-capped unless the agent explicitly passes `allowFullSuite: true`; the escape hatch still emits progress heartbeats and respects the hard cap. **Do not pass `allowFullSuite: true` unless absolutely necessary** — it is the main way verification balloons past its budget. Default to a targeted, file-scoped command such as `pnpm --filter @fusion/<pkg> exec vitest run src/path/to/test.ts --silent=passed-only --reporter=dot`; reserve `allowFullSuite` for a genuinely full run with no targetable test set (state the reason), with the thin merge gate (`pnpm test:gate`) as the cross-cutting safety net.
## Fresh-worktree dist bootstrap

View File

@@ -367,7 +367,7 @@ export const runVerificationParams = Type.Object({
allowFullSuite: Type.Optional(
Type.Boolean({
description:
"Explicit opt-in for marathon verification commands such as pnpm test, pnpm test:full, verify:workspace, whole-package tests, or repeat loops. Default: false; still respects the hard timeout.",
"DO NOT SET THIS unless absolutely necessary. It is a last-resort opt-in for marathon commands (`pnpm test`, `pnpm test:full`, `verify:workspace`, whole-package tests, repeat loops) that run far more than the change requires and make verification slow. Default false — keep it false. First scope verification to the changed files (e.g. `pnpm --filter <pkg> exec vitest run src/path/to/changed.test.ts --silent=passed-only --reporter=dot`); the soft cap exists to push you toward that. Only set true when a genuinely full run is unavoidable (e.g. a cross-cutting infra change with no targetable test set), and say why. Still respects the hard timeout.",
}),
),
expectFailure: Type.Optional(