docs: record dashboard api-lane PostgreSQL contention as a suite-infrastructure flake pattern
Three consecutive full-lane runs on the same tree each failed a DIFFERENT file, every one passing in isolation, with hook timeouts arriving alongside PostgreSQL 'Failed query' warnings from two api lanes sharing one database. That is the same class FN-9131 investigated for core's loaded PostgreSQL directory. Deliberately not quarantined: quarantine is file-level and the failing file moves, so it would evict healthy coverage without touching the cause. Recorded with the evidence so the next person does not re-derive it, and so the rescue is aimed at the lane runner's connection/concurrency budget rather than at whichever test lost the race that run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -556,3 +556,40 @@ concurrent 6GB lanes. No timeout was widened, no retry added, no assertion relax
|
||||
of the *same* case is an ordinary on-sight quarantine per the standing rule in AGENTS.md; because the
|
||||
case moves, the honest rescue is a deterministic settle signal in this file's harness rather than a
|
||||
longer wait.
|
||||
|
||||
---
|
||||
|
||||
## Entry: dashboard `api:backfill-*` lanes — PostgreSQL contention under lane concurrency (pattern, not a single test)
|
||||
|
||||
- **Files:** no fixed set. Across three consecutive full-lane runs on the same tree, a DIFFERENT file failed each time:
|
||||
- run 1: `app/components/__tests__/PlanningModeModal.planning-flow.test.tsx`
|
||||
- run 2: `src/__tests__/routes-branch-groups.test.ts`, `src/__tests__/routes-planning.test.ts`
|
||||
- run 3: `src/__tests__/register-signal-routes.test.ts`, `src/__tests__/server-view-preload.test.ts`
|
||||
- **Command:** `pnpm --filter @fusion/dashboard test -- --all --no-fail-fast` (the `run-quality-tests.mjs` lane runner: 15 lanes, concurrency 2, 6144MiB heap per lane).
|
||||
- **Observed tree/SHA:** `cc19584cc4`.
|
||||
- **Every one passes in isolation**, including re-running the exact multi-file command that had just failed.
|
||||
|
||||
Failure shape in run 3 (`api:backfill-1`, `api:backfill-2`):
|
||||
|
||||
```
|
||||
Error: Hook timed out in 15000ms.
|
||||
fnlvl=warn [dashboard-github-tracking-reconciler] … pass failed (other passes still run): Failed query: select … from "project"."tasks" …
|
||||
```
|
||||
|
||||
The hook timeout arrives alongside PostgreSQL `Failed query` warnings, so the proximate cause is
|
||||
database contention/exhaustion when two api lanes and their workers share one PostgreSQL instance —
|
||||
the same class FN-9131 investigated for core's loaded PostgreSQL directory, where a shared
|
||||
connection-budget primitive made a loaded run WORSE and was reverted. This is suite infrastructure,
|
||||
not a defect in any of the five files above, and chasing the file that happened to lose the race on a
|
||||
given run is whack-a-mole.
|
||||
|
||||
Scale for context: run 3 executed roughly 15,200 tests across 15 lanes and failed 2.
|
||||
|
||||
**Not quarantined deliberately.** Quarantine is file-level and the failing file moves, so quarantining
|
||||
would evict healthy coverage without touching the cause. The rescue is a connection/concurrency budget
|
||||
for the api lanes (or per-lane database isolation), owned by whoever owns the lane runner. No timeout
|
||||
was widened, no retry added, and no assertion relaxed anywhere in this investigation.
|
||||
|
||||
One genuinely structural failure WAS found and fixed rather than recorded here: the lane runner's own
|
||||
self-tests spawned `pnpm --filter @fusion/dashboard test`, re-entering the suite from inside it. See
|
||||
`cc19584cc4`.
|
||||
|
||||
Reference in New Issue
Block a user