ci(release): fail PRs that regress the beta cycle in .changeset/pre.json (#3486)
## Problem PR #3472 resolved a `.changeset/pre.json` rebase conflict against a copy predating the v0.76.0 stable: | | at `v0.77.0-beta.1` | after #3472 | |---|---|---| | `initialVersions["@runfusion/fusion"]` | `0.76.0` | `0.75.1` | | consumed ledger | 67 entries | 158 (the pre-0.76.0 cycle's) | Nothing failed at PR time. Days later `pnpm release` saw the cycle anchored below the shipped `v0.76.0`, fired its stale-cycle re-anchor (`pre exit` → rewrite all 36 `package.json` → `pre enter`), and proposed **`0.77.0-beta.0`** — below the already-published `0.77.0-beta.1`. The re-anchor guard exists to stop a beta numbering under a stable; fed a stale anchor it caused exactly that. `pre.json` is generated by changesets, hand-edited by nobody, and conflicts in nearly every long-lived branch — so a wrong resolution is invisible until release day. This moves the failure to the PR that causes it. ## The check `scripts/check-pre-json-anchor.mjs`, three invariants: - **`anchor-below-stable`** — `initialVersions` must not sit below the newest `v*` stable tag. This is the exact predicate `evaluateBetaCycleAnchor` keys on in `release.mjs`, so green here means the release will *not* re-anchor. - **`ledger-regression`** — the consumed ledger must stay a **superset** of the last `chore(release):` commit's. Deliberately not a count test: #3472's ledger *grew* 67 → 158 while dropping all 67 real entries, so a size comparison would have passed it. - **`dangling-ledger-entry`** — every consumed entry keeps its `.changeset/*.md`, which pre-mode needs to aggregate notes into the eventual stable release. Skips cleanly outside pre-mode (the stable track deletes `pre.json`). ## Wiring Added to the **Lint** job and `pretest`. Job names are unchanged, so no branch-protection update is needed. The Lint checkout takes `fetch-depth: 200` + `fetch-tags` rather than a full 486MB clone — releases land every few days, so that always reaches a baseline; out of range the ledger rule reports `SKIPPED` rather than passing vacuously, and the two local rules still run. ## Verification - Reproducing #3472's exact `pre.json` in the tree → **exit 1** on all three rules, with the 67 dropped entries named. - Clean `main` → exit 0. - 11 unit tests (`scripts/__tests__/check-pre-json-anchor.test.mjs`), including an explicit assertion that the ledger *grew* in the regression case. - `eslint` clean; workflow YAML parses; job names still `Lint, Typecheck, Build, Gate`. No changeset: CI config only, no `@runfusion/fusion` behavior change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added automated validation for beta release-cycle metadata. * Checks anchor versions, consumed changeset records, and corresponding changeset files. * Provides clear success or error messages and skips checks when release history is unavailable or not applicable. * **Chores** * Pull request checks now run the beta-cycle validation automatically. * Added comprehensive coverage for valid, invalid, and skipped validation scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
gsxdsm
145
scripts/__tests__/check-pre-json-anchor.test.mjs
Normal file
145
scripts/__tests__/check-pre-json-anchor.test.mjs
Normal file
@@ -0,0 +1,145 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { evaluatePreJsonInvariants, ANCHOR_PACKAGE } from "../check-pre-json-anchor.mjs";
|
||||
|
||||
const pre = (anchor, changesets = []) => ({
|
||||
mode: "pre",
|
||||
tag: "beta",
|
||||
initialVersions: { [ANCHOR_PACKAGE]: anchor },
|
||||
changesets,
|
||||
});
|
||||
const rules = (result) => result.violations.map((v) => v.rule).sort();
|
||||
|
||||
test("healthy cycle anchored on the shipped stable passes", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.76.0", ["a", "b"]),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["a", "b", "c"],
|
||||
baselinePreState: pre("0.76.0", ["a", "b"]),
|
||||
});
|
||||
assert.deepEqual(result.violations, []);
|
||||
assert.deepEqual(result.skipped, []);
|
||||
});
|
||||
|
||||
test("a cycle that consumed more changesets since the last release still passes", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.76.0", ["a", "b", "c"]),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["a", "b", "c"],
|
||||
baselinePreState: pre("0.76.0", ["a", "b"]),
|
||||
});
|
||||
assert.deepEqual(result.violations, []);
|
||||
});
|
||||
|
||||
/*
|
||||
The PR #3472 shape: pre.json reverted to the pre-v0.76.0 cycle. The anchor fell
|
||||
below the shipped stable AND the ledger swapped an older, LARGER list in for the
|
||||
real entries. Both invariants must fire; the size test that a naive check would
|
||||
use is asserted to be useless here.
|
||||
*/
|
||||
test("reverting pre.json to a pre-stable cycle fires anchor + ledger (the #3472 regression)", () => {
|
||||
const baseline = pre("0.76.0", ["real-1", "real-2"]);
|
||||
const reverted = pre("0.75.1", ["old-1", "old-2", "old-3", "old-4"]);
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: reverted,
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["real-1", "real-2", "old-1", "old-2", "old-3", "old-4"],
|
||||
baselinePreState: baseline,
|
||||
});
|
||||
assert.deepEqual(rules(result), ["anchor-below-stable", "ledger-regression"]);
|
||||
// The ledger GREW while dropping every real entry — a count check passes it.
|
||||
assert.ok(reverted.changesets.length > baseline.changesets.length);
|
||||
const dropped = result.violations.find((v) => v.rule === "ledger-regression");
|
||||
assert.match(dropped.message, /real-1/);
|
||||
assert.match(dropped.message, /real-2/);
|
||||
});
|
||||
|
||||
test("anchor below stable is reported even when the ledger is intact", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.75.1", ["a"]),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["a"],
|
||||
baselinePreState: pre("0.75.1", ["a"]),
|
||||
});
|
||||
assert.deepEqual(rules(result), ["anchor-below-stable"]);
|
||||
});
|
||||
|
||||
test("anchor equal to the shipped stable is fine (release.mjs just re-anchored)", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.76.0", []),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: [],
|
||||
baselinePreState: pre("0.76.0", []),
|
||||
});
|
||||
assert.deepEqual(result.violations, []);
|
||||
});
|
||||
|
||||
test("prerelease anchors compare below their own stable", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.76.0-beta.3", []),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: [],
|
||||
baselinePreState: null,
|
||||
});
|
||||
assert.deepEqual(rules(result), ["anchor-below-stable"]);
|
||||
});
|
||||
|
||||
test("a dropped .changeset/*.md for a consumed entry is a dangling ledger entry", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.76.0", ["a", "gone"]),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["a"],
|
||||
baselinePreState: pre("0.76.0", ["a", "gone"]),
|
||||
});
|
||||
assert.deepEqual(rules(result), ["dangling-ledger-entry"]);
|
||||
assert.match(result.violations[0].message, /gone\.md/);
|
||||
});
|
||||
|
||||
test("missing initialVersions anchor is a violation, not a silent pass", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: { mode: "pre", tag: "beta", initialVersions: {}, changesets: [] },
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: [],
|
||||
baselinePreState: null,
|
||||
});
|
||||
assert.deepEqual(rules(result), ["anchor-below-stable"]);
|
||||
});
|
||||
|
||||
test("no pre.json (stable track, pre exit ran) checks nothing", () => {
|
||||
for (const preState of [null, { mode: "exit", initialVersions: {}, changesets: [] }]) {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState,
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: [],
|
||||
baselinePreState: pre("0.76.0", ["a"]),
|
||||
});
|
||||
assert.deepEqual(result.violations, []);
|
||||
}
|
||||
});
|
||||
|
||||
test("no stable tag yet (fresh repo) skips only the anchor comparison", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.1.0", ["a"]),
|
||||
latestStable: null,
|
||||
changesetFiles: ["a"],
|
||||
baselinePreState: pre("0.1.0", ["a"]),
|
||||
});
|
||||
assert.deepEqual(result.violations, []);
|
||||
});
|
||||
|
||||
/*
|
||||
A shallow clone with no reachable `chore(release):` commit must SKIP the ledger
|
||||
invariant loudly rather than pass it vacuously — the local invariants still run.
|
||||
*/
|
||||
test("absent baseline skips the ledger rule but still enforces the local ones", () => {
|
||||
const result = evaluatePreJsonInvariants({
|
||||
preState: pre("0.75.1", ["a", "gone"]),
|
||||
latestStable: "0.76.0",
|
||||
changesetFiles: ["a"],
|
||||
baselinePreState: null,
|
||||
});
|
||||
assert.deepEqual(rules(result), ["anchor-below-stable", "dangling-ledger-entry"]);
|
||||
assert.equal(result.skipped.length, 1);
|
||||
assert.match(result.skipped[0], /ledger-regression/);
|
||||
});
|
||||
209
scripts/check-pre-json-anchor.mjs
Normal file
209
scripts/check-pre-json-anchor.mjs
Normal file
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env node
|
||||
/*
|
||||
FNXC:UpdateChannels 2026-08-18-07:20:
|
||||
Requirement: a PR must never regress the beta cycle recorded in `.changeset/pre.json`.
|
||||
|
||||
Motivating incident (v0.77.0-beta.2): PR #3472 resolved a pre.json rebase conflict
|
||||
against a copy predating the v0.76.0 stable, reverting `initialVersions` 0.76.0 ->
|
||||
0.75.1 and swapping the consumed-changeset ledger for the older cycle's list. Nothing
|
||||
failed at PR time. Days later `pnpm release` saw the cycle anchored below the shipped
|
||||
v0.76.0, fired its stale-cycle re-anchor (`pre exit` -> rewrite versions -> `pre enter`),
|
||||
and proposed 0.77.0-beta.0 — BELOW the already-published 0.77.0-beta.1. The re-anchor
|
||||
guard exists to stop a beta numbering under a stable; fed a stale anchor it caused
|
||||
exactly that. pre.json is edited by no one and conflicts in nearly every long-lived
|
||||
branch, so a wrong resolution is silent until release day. This check moves that
|
||||
failure to the PR that causes it.
|
||||
|
||||
Three invariants, all cheap and all failing loudly rather than warning:
|
||||
|
||||
1. `anchor-below-stable` — `initialVersions["@runfusion/fusion"]` must not sit below the
|
||||
newest published stable tag. This is the exact predicate `evaluateBetaCycleAnchor`
|
||||
keys on in scripts/release.mjs, so a green check here means the release will NOT
|
||||
re-anchor. Caught #3472 (0.75.1 < 0.76.0).
|
||||
2. `ledger-regression` — the consumed-changeset ledger must remain a SUPERSET of the
|
||||
ledger at the last `chore(release):` commit. Deliberately a superset test, not a
|
||||
count test: #3472's ledger GREW 67 -> 158 while dropping all 67 real entries, so a
|
||||
size comparison would have passed it. Losing an entry means that changeset's notes
|
||||
silently vanish from the eventual stable aggregation.
|
||||
3. `dangling-ledger-entry` — every consumed entry must still have its `.changeset/*.md`
|
||||
file. In pre-mode `changeset version` records consumed changesets and KEEPS the .md
|
||||
so the stable release can aggregate them; a merge that deletes the file while keeping
|
||||
the entry drops it from the release notes.
|
||||
|
||||
Skips cleanly when not in pre-mode (the stable track deletes pre.json via `pre exit`).
|
||||
Invariant 2 needs the last release commit in local history; on a shallow clone with no
|
||||
release commit reachable it reports SKIPPED rather than passing vacuously — invariants 1
|
||||
and 3 are purely local and always run.
|
||||
*/
|
||||
import { existsSync, readFileSync, readdirSync } from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { fileURLToPath, URL } from "node:url";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { compareReleaseVersions, latestStableVersionFromTags } from "./lib/release-version-anchor.mjs";
|
||||
|
||||
export const repoRoot = fileURLToPath(new URL("..", import.meta.url));
|
||||
/** The fixed-group package whose version anchors every X.Y.Z-beta.N. */
|
||||
export const ANCHOR_PACKAGE = "@runfusion/fusion";
|
||||
|
||||
/**
|
||||
* Pure evaluator. All git/filesystem access is done by the caller so the
|
||||
* invariants are testable without a repo fixture.
|
||||
*
|
||||
* @param {object} input
|
||||
* @param {object|null} input.preState parsed .changeset/pre.json, or null when absent
|
||||
* @param {string|null} input.latestStable e.g. "0.76.0", or null when no stable tag exists
|
||||
* @param {string[]} input.changesetFiles `.changeset/*.md` basenames WITHOUT the extension
|
||||
* @param {object|null} input.baselinePreState pre.json at the last `chore(release):` commit
|
||||
* @returns {{ violations: {rule: string, message: string}[], skipped: string[] }}
|
||||
*/
|
||||
export function evaluatePreJsonInvariants({ preState, latestStable, changesetFiles, baselinePreState }) {
|
||||
const violations = [];
|
||||
const skipped = [];
|
||||
|
||||
// Not in a pre-mode cycle: the stable track owns this state and pre.json is absent.
|
||||
if (!preState || preState.mode !== "pre") return { violations, skipped };
|
||||
|
||||
const anchor = preState.initialVersions?.[ANCHOR_PACKAGE] ?? null;
|
||||
const ledger = preState.changesets ?? [];
|
||||
|
||||
// 1. anchor-below-stable
|
||||
if (!anchor) {
|
||||
violations.push({
|
||||
rule: "anchor-below-stable",
|
||||
message: `pre.json has no initialVersions["${ANCHOR_PACKAGE}"]; the beta cycle has no anchor to derive X.Y.Z-beta.N from.`,
|
||||
});
|
||||
} else if (latestStable && compareReleaseVersions(anchor, latestStable) < 0) {
|
||||
violations.push({
|
||||
rule: "anchor-below-stable",
|
||||
message:
|
||||
`pre.json is anchored at ${anchor}, below the shipped stable v${latestStable}.\n` +
|
||||
` scripts/release.mjs will treat this cycle as stale and re-anchor (pre exit -> rewrite versions -> pre enter),\n` +
|
||||
` resetting the ledger and proposing a beta BELOW the betas already published on v${latestStable}.\n` +
|
||||
` Almost always a pre.json merge/rebase conflict resolved against a pre-v${latestStable} copy.\n` +
|
||||
` Fix: git checkout <last chore(release) tag> -- .changeset/pre.json`,
|
||||
});
|
||||
}
|
||||
|
||||
// 2. ledger-regression (needs a baseline from history)
|
||||
if (!baselinePreState) {
|
||||
skipped.push("ledger-regression (no `chore(release):` commit for .changeset/pre.json in local history)");
|
||||
} else {
|
||||
const baselineLedger = baselinePreState.changesets ?? [];
|
||||
const current = new Set(ledger);
|
||||
const dropped = baselineLedger.filter((name) => !current.has(name));
|
||||
if (dropped.length > 0) {
|
||||
violations.push({
|
||||
rule: "ledger-regression",
|
||||
message:
|
||||
`pre.json's consumed-changeset ledger dropped ${dropped.length} entr${dropped.length === 1 ? "y" : "ies"} present at the last release:\n` +
|
||||
dropped.slice(0, 10).map((name) => ` - ${name}`).join("\n") +
|
||||
(dropped.length > 10 ? `\n … and ${dropped.length - 10} more` : "") +
|
||||
`\n A consumed changeset dropped from the ledger loses its notes from the eventual stable release.\n` +
|
||||
` Ledger size alone is not the signal — it can grow while dropping every real entry.`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 3. dangling-ledger-entry
|
||||
const present = new Set(changesetFiles);
|
||||
const dangling = ledger.filter((name) => !present.has(name));
|
||||
if (dangling.length > 0) {
|
||||
violations.push({
|
||||
rule: "dangling-ledger-entry",
|
||||
message:
|
||||
`${dangling.length} consumed changeset${dangling.length === 1 ? "" : "s"} in the ledger no longer ha${dangling.length === 1 ? "s" : "ve"} a .changeset/*.md file:\n` +
|
||||
dangling.slice(0, 10).map((name) => ` - ${name}.md`).join("\n") +
|
||||
(dangling.length > 10 ? `\n … and ${dangling.length - 10} more` : "") +
|
||||
`\n Pre-mode keeps consumed .md files so the stable release can aggregate them; deleting one drops it from the notes.`,
|
||||
});
|
||||
}
|
||||
|
||||
return { violations, skipped };
|
||||
}
|
||||
|
||||
function git(args, { allowFail = true } = {}) {
|
||||
const r = spawnSync("git", args, { cwd: repoRoot, encoding: "utf8" });
|
||||
if (r.status !== 0 && !allowFail) return null;
|
||||
return r.status === 0 ? r.stdout : null;
|
||||
}
|
||||
|
||||
/** Local `v*` tags; falls back to the remote when a shallow CI clone has none. */
|
||||
export function readStableTags() {
|
||||
const local = git(["tag", "--list", "v*"]) ?? "";
|
||||
if (latestStableVersionFromTags(local) !== null) return local;
|
||||
const remote = git(["ls-remote", "--tags", "origin", "v*"]) ?? "";
|
||||
// ls-remote emits "<sha>\trefs/tags/v0.76.0"; reduce to bare tag names.
|
||||
return remote
|
||||
.split("\n")
|
||||
.map((line) => line.split("\t")[1] ?? "")
|
||||
.filter((ref) => ref.startsWith("refs/tags/") && !ref.endsWith("^{}"))
|
||||
.map((ref) => ref.slice("refs/tags/".length))
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
/** pre.json as of the newest `chore(release):` commit that touched it. */
|
||||
export function readBaselinePreState() {
|
||||
const sha = (git(["log", "--format=%H", "--grep=^chore(release):", "-n", "1", "--", ".changeset/pre.json"]) ?? "").trim();
|
||||
if (!sha) return null;
|
||||
const blob = git(["show", `${sha}:.changeset/pre.json`]);
|
||||
if (!blob) return null;
|
||||
try {
|
||||
return { sha, state: JSON.parse(blob) };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function main() {
|
||||
const prePath = join(repoRoot, ".changeset", "pre.json");
|
||||
let preState = null;
|
||||
if (existsSync(prePath)) {
|
||||
try {
|
||||
preState = JSON.parse(readFileSync(prePath, "utf8"));
|
||||
} catch (error) {
|
||||
console.error(`✗ .changeset/pre.json is not valid JSON: ${error.message}`);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
if (!preState || preState.mode !== "pre") {
|
||||
console.log("✓ pre.json anchor: not in changesets pre-mode (stable track) — nothing to check.");
|
||||
return 0;
|
||||
}
|
||||
|
||||
const changesetFiles = readdirSync(join(repoRoot, ".changeset"))
|
||||
.filter((f) => f.endsWith(".md") && f !== "README.md")
|
||||
.map((f) => f.slice(0, -3));
|
||||
const baseline = readBaselinePreState();
|
||||
const latestStable = latestStableVersionFromTags(readStableTags());
|
||||
|
||||
const { violations, skipped } = evaluatePreJsonInvariants({
|
||||
preState,
|
||||
latestStable,
|
||||
changesetFiles,
|
||||
baselinePreState: baseline?.state ?? null,
|
||||
});
|
||||
|
||||
for (const note of skipped) console.log(` SKIPPED: ${note}`);
|
||||
|
||||
if (violations.length > 0) {
|
||||
console.error(`\n✗ .changeset/pre.json regressed the beta release cycle (${violations.length} violation${violations.length === 1 ? "" : "s"}):\n`);
|
||||
for (const v of violations) console.error(` [${v.rule}] ${v.message}\n`);
|
||||
console.error(
|
||||
" pre.json is generated by changesets and hand-edited by no one. If this fired on a rebase,\n" +
|
||||
" take the version from the branch you are merging INTO, never the older side.\n",
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const anchor = preState.initialVersions?.[ANCHOR_PACKAGE];
|
||||
console.log(
|
||||
`✓ pre.json anchor: cycle anchored at ${anchor} (stable v${latestStable ?? "none"}), ` +
|
||||
`${(preState.changesets ?? []).length} consumed changesets intact.`,
|
||||
);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (process.argv[1] && process.argv[1].endsWith("check-pre-json-anchor.mjs")) {
|
||||
process.exit(main());
|
||||
}
|
||||
Reference in New Issue
Block a user