diff --git a/docs/custom-workflow-reliability-acceptance-map.md b/docs/custom-workflow-reliability-acceptance-map.md index e2264e7263..150f09be71 100644 --- a/docs/custom-workflow-reliability-acceptance-map.md +++ b/docs/custom-workflow-reliability-acceptance-map.md @@ -116,10 +116,27 @@ The following journeys are intentionally out of scope for the MVP reliability ba ## Release-check checklist -Before claiming the custom workflow system is reliable for goal **G-MPW67VQR-0001-97S3**, QA or engineering should be able to demonstrate: + -- A custom workflow can be authored/imported, rejected on invalid IR, saved, discovered, selected, and reloaded. -- A task can execute the selected workflow through runtime primitives, and explicit missing custom workflow IDs fail closed. -- Gate/advisory/readonly/`REVISE`/required-artifact behavior is observable in task state, workflow results, task documents, and logs. -- `autoMerge:false`, hard-cancel, file-scope, and recovery invariants are preserved under custom workflow execution. -- Engine/scheduler restart preserves workflow selection and progress, and any recovery emits typed run-audit evidence instead of silent lifecycle mutation. +Before claiming the custom workflow system is reliable for goal **G-MPW67VQR-0001-97S3**, QA or engineering should run the executable release-check harness: + +```bash +pnpm test:workflow-release-check # run the targeted manifest-listed seams and emit text PASS/FAIL evidence +pnpm test:workflow-release-check --json # emit the same item/seam evidence as machine-readable JSON +pnpm test:workflow-release-check --dry-run # validate the manifest and print planned commands without running Vitest +``` + +The source of truth for the checklist-to-seam mapping is [`scripts/lib/workflow-reliability-release-check.json`](../scripts/lib/workflow-reliability-release-check.json). The runner validates that every referenced file exists, groups the seams into targeted package-scoped Vitest commands, and exits non-zero if the manifest is invalid or any required item fails. It is intentionally an on-demand QA/release lane, not a merge-gate expansion. + +| Release-check item | Manifest ID | Automated evidence seams | +|---|---|---| +| A custom workflow can be authored/imported, rejected on invalid IR, saved, discovered, selected, and reloaded. | `author-import-save-discover-reload` | `packages/core/src/__tests__/workflow-definition-store.test.ts`; `packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts`; `packages/dashboard/src/routes/__tests__/workflow-design-route.test.ts`; `packages/core/src/__tests__/workflow-selection-store.test.ts` | +| A task can execute the selected workflow through runtime primitives, and explicit missing custom workflow IDs fail closed. | `selected-workflow-execution-fail-closed` | `packages/core/src/__tests__/workflow-selection-store.test.ts`; `packages/engine/src/__tests__/workflow-task-runtime.test.ts` | +| Gate/advisory/readonly/`REVISE`/required-artifact behavior is observable in task state, workflow results, task documents, and logs. | `gate-advisory-readonly-revise-required-artifact` | `packages/engine/src/__tests__/workflow-malformed-verdict-gate.test.ts`; `packages/engine/src/__tests__/workflow-required-artifact-gate.test.ts`; `packages/engine/src/__tests__/workflow-step-readonly-allowlist.test.ts`; `packages/engine/src/__tests__/executor-workflow-revision-scope.test.ts` | +| `autoMerge:false`, hard-cancel, file-scope, and recovery invariants are preserved under custom workflow execution. | `automerge-hard-cancel-file-scope-recovery` | `packages/engine/src/__tests__/reliability-interactions/workflow-and-file-scope.test.ts`; `packages/engine/src/__tests__/reliability-interactions/workflow-interpreter-cutover.test.ts`; `packages/engine/src/__tests__/self-healing-custom-workflow-recovery.test.ts` | +| Engine/scheduler restart preserves workflow selection and progress, and any recovery emits typed run-audit evidence instead of silent lifecycle mutation. | `restart-selection-progress-run-audit` | `packages/core/src/__tests__/workflow-restart-durability.test.ts`; `packages/engine/src/__tests__/self-healing-custom-workflow-recovery.test.ts` | + +Manual-only checks: **none currently deferred**. If a future release-check item cannot be automated, add it to the manifest's `manual` array with a non-empty `automationDeferredReason`, label it here, and file/link a focused follow-up after confirming there is no duplicate task. diff --git a/docs/testing.md b/docs/testing.md index d94cf63f64..3e2718bc94 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -40,6 +40,9 @@ pnpm verify:workspace # deep opt-in verification: lint -> test:full -> build (N `pnpm test:full` runs each package's default test script with capped worker fanout (`FUSION_TEST_TOTAL_WORKERS=4 FUSION_TEST_CONCURRENCY=2 pnpm -r --workspace-concurrency=2 test`). Do not casually raise worker counts; dashboard/jsdom and integration-heavy packages destabilize when oversubscribed. Use `VITEST_MAX_WORKERS=` only for targeted package-level investigation. + +Custom workflow reliability release signoff has a dedicated on-demand lane: `pnpm test:workflow-release-check` runs the manifest-listed targeted seams from `scripts/lib/workflow-reliability-release-check.json`, while `--dry-run` validates the manifest and prints planned commands and `--json` emits machine-readable item/seam evidence. This lane is **not** part of the merge gate and should not be added to `test:gate` or the `engine-core` allow-list. + Terminal acceptance tasks that require real mobile Safari should use [`docs/ios-acceptance.md`](./ios-acceptance.md) for the `--check` run-vs-NO-OP probe, credential wiring, and physical/cloud real-iOS evidence workflow. diff --git a/package.json b/package.json index 933621814e..41151ed8ab 100644 --- a/package.json +++ b/package.json @@ -32,6 +32,7 @@ "build:exe:all": "pnpm build && pnpm --filter @runfusion/fusion build:exe:all", "test": "node scripts/test-changed.mjs", "test:scripts": "node --test scripts/__tests__/*.test.mjs", + "test:workflow-release-check": "node scripts/workflow-reliability-release-check.mjs", "fn:cache-stats": "node scripts/cache-stats.mjs", "test:full": "node scripts/test-changed.mjs --full --no-cache && pnpm --filter @fusion/engine test:slow", "test:velocity": "node scripts/test-velocity-baseline.mjs", diff --git a/scripts/__tests__/workflow-reliability-release-check.test.mjs b/scripts/__tests__/workflow-reliability-release-check.test.mjs new file mode 100644 index 0000000000..b432b62ae8 --- /dev/null +++ b/scripts/__tests__/workflow-reliability-release-check.test.mjs @@ -0,0 +1,138 @@ +import test from "node:test"; +import assert from "node:assert/strict"; +import path from "node:path"; +import { existsSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { + DEFAULT_MANIFEST_PATH, + loadManifest, + planCommands, + renderReport, + summarize, + validateManifest, +} from "../workflow-reliability-release-check.mjs"; + +const currentFilePath = fileURLToPath(import.meta.url); +const repoRoot = path.resolve(path.dirname(currentFilePath), "../.."); + +const expectedChecklistIds = [ + "author-import-save-discover-reload", + "selected-workflow-execution-fail-closed", + "gate-advisory-readonly-revise-required-artifact", + "automerge-hard-cancel-file-scope-recovery", + "restart-selection-progress-run-audit", +]; + +function clone(value) { + return JSON.parse(JSON.stringify(value)); +} + +test("manifest parses, covers all five release-check items, and references existing seam files", () => { + const manifest = loadManifest(DEFAULT_MANIFEST_PATH, { rootDir: repoRoot }); + + assert.equal(manifest.version, 1); + assert.equal(typeof manifest._comment, "string"); + assert.deepEqual(manifest.checklist.map((item) => item.id), expectedChecklistIds); + assert.equal(manifest.manual.length, 0); + + for (const item of manifest.checklist) { + assert.equal(typeof item.title, "string"); + assert.equal(typeof item.journey, "string"); + assert.ok(item.seams.length >= 1, `${item.id} should map to at least one automated seam`); + for (const seam of item.seams) { + assert.match(seam.package, /^@fusion\/(core|dashboard|engine)$/); + assert.ok(existsSync(path.join(repoRoot, seam.file)), `${seam.file} should exist`); + } + } + + assert.deepEqual(validateManifest(manifest, { repoRoot, existsSync }), { ok: true, errors: [] }); +}); + +test("planned commands are targeted package-scoped vitest invocations", () => { + const manifest = loadManifest(DEFAULT_MANIFEST_PATH, { rootDir: repoRoot }); + const commands = planCommands(manifest); + + assert.deepEqual(commands.map((command) => command.package), ["@fusion/core", "@fusion/dashboard", "@fusion/engine"]); + for (const command of commands) { + assert.equal(command.command, "pnpm"); + assert.deepEqual(command.args.slice(0, 5), ["--filter", command.package, "exec", "vitest", "run"]); + assert.ok(command.args.includes("--silent=passed-only")); + assert.ok(command.args.includes("--reporter=dot")); + assert.ok(command.files.length >= 1); + for (const file of command.files) assert.ok(file.startsWith("packages/")); + } +}); + +test("validateManifest fails closed on dangling seam files and uncovered checklist items", () => { + const manifest = loadManifest(DEFAULT_MANIFEST_PATH, { rootDir: repoRoot }); + const dangling = clone(manifest); + dangling.checklist[0].seams[0].file = "packages/core/src/__tests__/missing-workflow-release-check.test.ts"; + + const danglingResult = validateManifest(dangling, { repoRoot, existsSync: (candidate) => !String(candidate).includes("missing-workflow-release-check") }); + assert.equal(danglingResult.ok, false); + assert.match(danglingResult.errors.join("\n"), /file does not exist/); + + const uncovered = clone(manifest); + uncovered.checklist[1].seams = []; + const uncoveredResult = validateManifest(uncovered, { repoRoot, existsSync: () => true }); + assert.equal(uncoveredResult.ok, false); + assert.match(uncoveredResult.errors.join("\n"), /must have at least one seam or a manual automationDeferredReason/); +}); + +test("validateManifest rejects unknown packages and accepts explicit manual deferrals", () => { + const manifest = loadManifest(DEFAULT_MANIFEST_PATH, { rootDir: repoRoot }); + const unknownPackage = clone(manifest); + unknownPackage.checklist[0].seams[0].package = "@fusion/unknown"; + + const unknownPackageResult = validateManifest(unknownPackage, { repoRoot, existsSync: () => true }); + assert.equal(unknownPackageResult.ok, false); + assert.match(unknownPackageResult.errors.join("\n"), /unknown package/); + + const manual = clone(manifest); + manual.checklist[0].seams = []; + manual.manual = [{ + id: manual.checklist[0].id, + title: manual.checklist[0].title, + automationDeferredReason: "Requires a human-only external signoff artifact.", + }]; + assert.deepEqual(validateManifest(manual, { repoRoot, existsSync: () => true }), { ok: true, errors: [] }); +}); + +test("summarize and renderReport roll up pass, fail, manual, text, and stable JSON", () => { + const summary = summarize([ + { + id: "passed-item", + title: "Passed item", + journey: "A passing synthetic item", + seams: [{ package: "@fusion/core", file: "one.test.ts", status: "PASS" }], + manual: null, + }, + { + id: "failed-item", + title: "Failed item", + journey: "A failing synthetic item", + seams: [{ package: "@fusion/engine", file: "two.test.ts", status: "FAIL", exitCode: 1 }], + manual: null, + }, + { + id: "manual-item", + title: "Manual item", + journey: "A manual synthetic item", + seams: [], + manual: { id: "manual-item", title: "Manual item", automationDeferredReason: "Human inspection only." }, + }, + ]); + + assert.equal(summary.ok, false); + assert.deepEqual(summary.counts, { pass: 1, fail: 1, manual: 1, total: 3 }); + assert.deepEqual(summary.items.map((item) => item.status), ["PASS", "FAIL", "MANUAL"]); + + const text = renderReport(summary); + assert.match(text, /Overall: FAIL \(1 passed, 1 failed, 1 manual, 3 total\)/); + assert.match(text, /FAIL: failed-item/); + assert.match(text, /MANUAL: Human inspection only\./); + + const json = renderReport(summary, { json: true }); + assert.deepEqual(JSON.parse(json), summary); + assert.ok(json.startsWith('{\n "ok": false,')); +}); diff --git a/scripts/lib/workflow-reliability-release-check.json b/scripts/lib/workflow-reliability-release-check.json new file mode 100644 index 0000000000..ff80026f39 --- /dev/null +++ b/scripts/lib/workflow-reliability-release-check.json @@ -0,0 +1,102 @@ +{ + "version": 1, + "_comment": "FNXC:CustomWorkflowReliability 2026-06-19-00:00: FN-6694 requires this manifest to encode goal G-MPW67VQR-0001-97S3's release-check checklist as objective, runnable seam evidence instead of prose-only QA signoff.", + "checklist": [ + { + "id": "author-import-save-discover-reload", + "title": "Custom workflow authoring, import, invalid rejection, save, discovery, selection, and reload", + "journey": "A custom workflow can be authored/imported, rejected on invalid IR, saved, discovered, selected, and reloaded.", + "seams": [ + { + "package": "@fusion/core", + "file": "packages/core/src/__tests__/workflow-definition-store.test.ts" + }, + { + "package": "@fusion/dashboard", + "file": "packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts" + }, + { + "package": "@fusion/dashboard", + "file": "packages/dashboard/src/routes/__tests__/workflow-design-route.test.ts" + }, + { + "package": "@fusion/core", + "file": "packages/core/src/__tests__/workflow-selection-store.test.ts" + } + ] + }, + { + "id": "selected-workflow-execution-fail-closed", + "title": "Selected workflow execution and fail-closed missing custom workflow IDs", + "journey": "A task can execute the selected workflow through runtime primitives, and explicit missing custom workflow IDs fail closed.", + "seams": [ + { + "package": "@fusion/core", + "file": "packages/core/src/__tests__/workflow-selection-store.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/workflow-task-runtime.test.ts" + } + ] + }, + { + "id": "gate-advisory-readonly-revise-required-artifact", + "title": "Gate, advisory, readonly, REVISE, and required-artifact observability", + "journey": "Gate/advisory/readonly/REVISE/required-artifact behavior is observable in task state, workflow results, task documents, and logs.", + "seams": [ + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/workflow-malformed-verdict-gate.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/workflow-required-artifact-gate.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/workflow-step-readonly-allowlist.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/executor-workflow-revision-scope.test.ts" + } + ] + }, + { + "id": "automerge-hard-cancel-file-scope-recovery", + "title": "autoMerge:false, hard-cancel, file-scope, and recovery invariants", + "journey": "autoMerge:false, hard-cancel, file-scope, and recovery invariants are preserved under custom workflow execution.", + "seams": [ + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/reliability-interactions/workflow-and-file-scope.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/reliability-interactions/workflow-interpreter-cutover.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/self-healing-custom-workflow-recovery.test.ts" + } + ] + }, + { + "id": "restart-selection-progress-run-audit", + "title": "Restart durability for workflow selection, progress, and recovery evidence", + "journey": "Engine/scheduler restart preserves workflow selection and progress, and any recovery emits typed run-audit evidence instead of silent lifecycle mutation.", + "seams": [ + { + "package": "@fusion/core", + "file": "packages/core/src/__tests__/workflow-restart-durability.test.ts" + }, + { + "package": "@fusion/engine", + "file": "packages/engine/src/__tests__/self-healing-custom-workflow-recovery.test.ts" + } + ] + } + ], + "manual": [] +} diff --git a/scripts/workflow-reliability-release-check.mjs b/scripts/workflow-reliability-release-check.mjs new file mode 100644 index 0000000000..1f04ef2ae9 --- /dev/null +++ b/scripts/workflow-reliability-release-check.mjs @@ -0,0 +1,312 @@ +#!/usr/bin/env node + +import { existsSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { spawn } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +const currentFilePath = fileURLToPath(import.meta.url); +const repoRoot = path.resolve(path.dirname(currentFilePath), ".."); + +export const DEFAULT_MANIFEST_PATH = "scripts/lib/workflow-reliability-release-check.json"; +export const KNOWN_PACKAGES = new Map([ + ["@fusion/core", "packages/core"], + ["@fusion/dashboard", "packages/dashboard"], + ["@fusion/engine", "packages/engine"], +]); +export const DEFAULT_COMMAND_TIMEOUT_MS = 300_000; + +/** + * FNXC:CustomWorkflowReliability 2026-06-19-00:00: + * FN-6694 makes the custom-workflow reliability release checklist executable for QA without adding it to the merge gate. Keep this harness on-demand, targeted to manifest-listed Vitest seams only, and fail closed when a manifest seam path is missing so release evidence cannot silently drift. + */ +export function loadManifest(manifestPath = DEFAULT_MANIFEST_PATH, { rootDir = repoRoot } = {}) { + const absolutePath = path.isAbsolute(manifestPath) ? manifestPath : path.join(rootDir, manifestPath); + return JSON.parse(readFileSync(absolutePath, "utf8")); +} + +function normalizeManual(manifest) { + const manualEntries = Array.isArray(manifest?.manual) ? manifest.manual : []; + return new Map(manualEntries.map((entry) => [entry?.id, entry])); +} + +function validateSeam(seam, item, index, seamIndex, { repoRoot: rootDir, existsSync: fileExists }) { + const errors = []; + const label = `checklist[${index}] ${item?.id ?? ""} seams[${seamIndex}]`; + if (!seam || typeof seam !== "object") { + errors.push(`${label} must be an object`); + return errors; + } + if (!KNOWN_PACKAGES.has(seam.package)) { + errors.push(`${label} references unknown package ${JSON.stringify(seam.package)}`); + } + if (!seam.file || typeof seam.file !== "string") { + errors.push(`${label} must include a repo-relative file`); + } else { + const normalizedFile = path.normalize(seam.file); + if (path.isAbsolute(seam.file) || normalizedFile.startsWith("..") || normalizedFile.includes(`${path.sep}..${path.sep}`)) { + errors.push(`${label} file must stay within the repository: ${seam.file}`); + } else if (!fileExists(path.join(rootDir, seam.file))) { + errors.push(`${label} file does not exist: ${seam.file}`); + } + } + return errors; +} + +export function validateManifest(manifest, { repoRoot: rootDir = repoRoot, existsSync: fileExists = existsSync } = {}) { + const errors = []; + if (!manifest || typeof manifest !== "object") { + return { ok: false, errors: ["manifest must be an object"] }; + } + if (manifest.version !== 1) errors.push("manifest.version must be 1"); + if (!Array.isArray(manifest.checklist)) errors.push("manifest.checklist must be an array"); + + const manualById = normalizeManual(manifest); + const checklist = Array.isArray(manifest.checklist) ? manifest.checklist : []; + const seenIds = new Set(); + checklist.forEach((item, index) => { + const label = `checklist[${index}]`; + if (!item || typeof item !== "object") { + errors.push(`${label} must be an object`); + return; + } + if (!item.id || typeof item.id !== "string") { + errors.push(`${label} must include an id`); + } else if (seenIds.has(item.id)) { + errors.push(`${label} duplicates id ${item.id}`); + } else { + seenIds.add(item.id); + } + if (!item.title || typeof item.title !== "string") errors.push(`${label} ${item.id ?? ""} must include a title`); + if (!item.journey || typeof item.journey !== "string") errors.push(`${label} ${item.id ?? ""} must include a journey`); + + const seams = Array.isArray(item.seams) ? item.seams : []; + const manual = manualById.get(item.id); + const manualReason = typeof manual?.automationDeferredReason === "string" ? manual.automationDeferredReason.trim() : ""; + if (seams.length === 0 && manualReason.length === 0) { + errors.push(`${label} ${item.id ?? ""} must have at least one seam or a manual automationDeferredReason`); + } + seams.forEach((seam, seamIndex) => { + errors.push(...validateSeam(seam, item, index, seamIndex, { repoRoot: rootDir, existsSync: fileExists })); + }); + }); + + const manual = Array.isArray(manifest.manual) ? manifest.manual : []; + manual.forEach((entry, index) => { + if (!entry?.id || typeof entry.id !== "string") errors.push(`manual[${index}] must include an id`); + if (!entry?.title || typeof entry.title !== "string") errors.push(`manual[${index}] ${entry?.id ?? ""} must include a title`); + if (!entry?.automationDeferredReason || typeof entry.automationDeferredReason !== "string" || entry.automationDeferredReason.trim().length === 0) { + errors.push(`manual[${index}] ${entry?.id ?? ""} must include a non-empty automationDeferredReason`); + } + if (entry?.id && !seenIds.has(entry.id)) errors.push(`manual[${index}] references unknown checklist id ${entry.id}`); + }); + + return { ok: errors.length === 0, errors }; +} + +function distinctSeams(manifest) { + const seen = new Set(); + const seams = []; + for (const item of manifest.checklist ?? []) { + for (const seam of item.seams ?? []) { + const key = `${seam.package}\u0000${seam.file}`; + if (seen.has(key)) continue; + seen.add(key); + seams.push({ package: seam.package, file: seam.file }); + } + } + return seams; +} + +export function planCommands(manifest) { + const byPackage = new Map(); + for (const seam of distinctSeams(manifest)) { + if (!byPackage.has(seam.package)) byPackage.set(seam.package, []); + byPackage.get(seam.package).push(seam.file); + } + + return [...byPackage.entries()].map(([packageName, files]) => { + const packageRoot = KNOWN_PACKAGES.get(packageName); + const packageRelativeFiles = files.map((file) => path.relative(packageRoot, file)); + return { + package: packageName, + files, + command: "pnpm", + args: ["--filter", packageName, "exec", "vitest", "run", ...packageRelativeFiles, "--silent=passed-only", "--reporter=dot"], + }; + }); +} + +function runProcess(command, args, { cwd, timeoutMs, stdout = process.stdout, stderr = process.stderr } = {}) { + return new Promise((resolve) => { + const child = spawn(command, args, { cwd, shell: false, detached: false, stdio: ["ignore", "pipe", "pipe"] }); + let stdoutText = ""; + let stderrText = ""; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGTERM"); + }, timeoutMs); + + child.stdout.on("data", (chunk) => { + const text = String(chunk); + stdoutText += text; + stdout?.write?.(text); + }); + child.stderr.on("data", (chunk) => { + const text = String(chunk); + stderrText += text; + stderr?.write?.(text); + }); + child.on("error", (error) => { + clearTimeout(timer); + resolve({ exitCode: 1, timedOut, stdout: stdoutText, stderr: `${stderrText}${error.message}\n` }); + }); + child.on("close", (exitCode, signal) => { + clearTimeout(timer); + resolve({ exitCode: exitCode ?? 1, signal, timedOut, stdout: stdoutText, stderr: stderrText }); + }); + }); +} + +export async function runReleaseCheck(manifest, { rootDir = repoRoot, timeoutMs = DEFAULT_COMMAND_TIMEOUT_MS, stdout = process.stdout, stderr = process.stderr } = {}) { + const commands = planCommands(manifest); + const seamResults = new Map(); + for (const planned of commands) { + stdout?.write?.(`Running: ${planned.command} ${planned.args.join(" ")}\n`); + const result = await runProcess(planned.command, planned.args, { cwd: rootDir, timeoutMs, stdout, stderr }); + for (const file of planned.files) { + seamResults.set(file, { + package: planned.package, + file, + status: result.exitCode === 0 && !result.timedOut ? "PASS" : "FAIL", + exitCode: result.exitCode, + timedOut: result.timedOut, + command: `${planned.command} ${planned.args.join(" ")}`, + }); + } + } + + return (manifest.checklist ?? []).map((item) => { + const seams = (item.seams ?? []).map((seam) => seamResults.get(seam.file) ?? { ...seam, status: "FAIL", exitCode: null, timedOut: false, command: null }); + const manual = (manifest.manual ?? []).find((entry) => entry.id === item.id) ?? null; + return { id: item.id, title: item.title, journey: item.journey, seams, manual }; + }); +} + +export function summarize(results) { + const items = results.map((result) => { + const seamStatuses = (result.seams ?? []).map((seam) => seam.status); + const hasFailure = seamStatuses.includes("FAIL"); + const hasPass = seamStatuses.includes("PASS"); + const manualReason = typeof result.manual?.automationDeferredReason === "string" ? result.manual.automationDeferredReason.trim() : ""; + const status = hasFailure ? "FAIL" : hasPass ? "PASS" : manualReason ? "MANUAL" : "FAIL"; + return { ...result, status }; + }); + const counts = { + pass: items.filter((item) => item.status === "PASS").length, + fail: items.filter((item) => item.status === "FAIL").length, + manual: items.filter((item) => item.status === "MANUAL").length, + total: items.length, + }; + return { ok: counts.fail === 0, counts, items }; +} + +export function renderReport(summary, { json = false } = {}) { + if (json) return `${JSON.stringify(summary, null, 2)}\n`; + const lines = [ + "Custom workflow reliability release-check", + `Overall: ${summary.ok ? "PASS" : "FAIL"} (${summary.counts.pass} passed, ${summary.counts.fail} failed, ${summary.counts.manual} manual, ${summary.counts.total} total)`, + "", + ]; + for (const item of summary.items) { + lines.push(`${item.status}: ${item.id} — ${item.title}`); + for (const seam of item.seams ?? []) { + lines.push(` - ${seam.status}: ${seam.file} (${seam.package})`); + } + if (item.manual?.automationDeferredReason) { + lines.push(` - MANUAL: ${item.manual.automationDeferredReason}`); + } + } + return `${lines.join("\n")}\n`; +} + +function parseArgs(argv) { + const args = { dryRun: false, json: false, manifestPath: DEFAULT_MANIFEST_PATH, timeoutMs: DEFAULT_COMMAND_TIMEOUT_MS }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--dry-run") args.dryRun = true; + else if (arg === "--json") args.json = true; + else if (arg === "--manifest") args.manifestPath = argv[++index]; + else if (arg === "--timeout-ms") args.timeoutMs = Number(argv[++index]); + else if (arg === "--help" || arg === "-h") args.help = true; + else throw new Error(`Unknown argument: ${arg}`); + } + if (!Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0) throw new Error("--timeout-ms must be a positive number"); + return args; +} + +function renderDryRun(manifest) { + const commands = planCommands(manifest); + const results = (manifest.checklist ?? []).map((item) => ({ + id: item.id, + title: item.title, + journey: item.journey, + seams: (item.seams ?? []).map((seam) => ({ ...seam, status: "PASS", exitCode: 0, timedOut: false, command: null })), + manual: (manifest.manual ?? []).find((entry) => entry.id === item.id) ?? null, + })); + return { commands, summary: summarize(results) }; +} + +export async function main(argv = process.argv.slice(2), { rootDir = repoRoot, stdout = process.stdout, stderr = process.stderr } = {}) { + let args; + try { + args = parseArgs(argv); + } catch (error) { + stderr.write(`${error.message}\n`); + return 1; + } + if (args.help) { + stdout.write("Usage: node scripts/workflow-reliability-release-check.mjs [--dry-run] [--json] [--manifest ] [--timeout-ms ]\n"); + return 0; + } + + let manifest; + try { + manifest = loadManifest(args.manifestPath, { rootDir }); + } catch (error) { + const summary = { ok: false, counts: { pass: 0, fail: 1, manual: 0, total: 1 }, items: [{ id: "manifest", title: "Manifest load", status: "FAIL", seams: [], error: error.message }] }; + stdout.write(renderReport(summary, { json: args.json })); + return 1; + } + + const validation = validateManifest(manifest, { repoRoot: rootDir, existsSync }); + if (!validation.ok) { + const summary = { ok: false, counts: { pass: 0, fail: validation.errors.length, manual: 0, total: validation.errors.length }, items: validation.errors.map((error, index) => ({ id: `manifest-${index + 1}`, title: error, status: "FAIL", seams: [] })) }; + stdout.write(renderReport(summary, { json: args.json })); + return 1; + } + + if (args.dryRun) { + const dryRun = renderDryRun(manifest); + if (args.json) { + stdout.write(`${JSON.stringify({ ok: dryRun.summary.ok, dryRun: true, commands: dryRun.commands, summary: dryRun.summary }, null, 2)}\n`); + } else { + stdout.write("Dry run: manifest is valid. Planned commands:\n"); + for (const command of dryRun.commands) stdout.write(`- ${command.command} ${command.args.join(" ")}\n`); + stdout.write("\n"); + stdout.write(renderReport(dryRun.summary)); + } + return dryRun.summary.ok ? 0 : 1; + } + + const results = await runReleaseCheck(manifest, { rootDir, timeoutMs: args.timeoutMs, stdout: args.json ? { write: () => {} } : stdout, stderr }); + const summary = summarize(results); + stdout.write(renderReport(summary, { json: args.json })); + return summary.ok ? 0 : 1; +} + +if (import.meta.url === `file://${process.argv[1]}`) { + const exitCode = await main(); + process.exitCode = exitCode; +}