0540686599
**Problem:** Scheduler was re-reading each task's `task_workflow_selection` once per park-resolution (sweep, hold-release, moved, unpause/wake), causing a nonstop PostgreSQL query storm (~232 idx_scan/s) on idle polling — a major engine CPU hot-spot. **Fix:** Memoize the workflow selection per scheduler tick/event — thread a shared, per-event selection cache through `resolveWorkflowIrForTask` and all park-resolution handlers, then throw it away. Each task resolves its parked columns with at most one read of `task_workflow_selection` per tick. A selection write is always observed on the next event's fresh cache (never a global/infinite LRU). **Includes:** regression test asserting the once-per-tick read invariant, performance changeset + per-tick-cache solution doc, deploy+verify handoff script, and the parallel quarantine-ledger merge (origin FN-9125 + RUFU-072 OOM entries both retained). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance Improvements** * Reduced repeated workflow-selection reads during scheduler ticks and related event processing. * Improved scheduler and health API responsiveness through per-operation caching and read deduplication. * Preserved existing behavior, including retry handling for failed reads and synchronous data-store support. * **Documentation** * Added architectural guidance covering workflow-selection performance, caching behavior, and verification criteria. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Fusion <noreply@runfusion.ai>
93 lines
4.9 KiB
Markdown
93 lines
4.9 KiB
Markdown
---
|
||
category: architecture-patterns
|
||
module: packages/engine/src/scheduler.ts
|
||
date: 2026-08-12
|
||
problem_type: performance
|
||
severity: high
|
||
applies_when:
|
||
- "Looping a per-task store read inside a scheduler/engine pass that visits the same task multiple times"
|
||
- "Seeing Drizzle buildQueryFromSourceParams dominate a CPU profile (~70%) or runaway pg_stat idx_scan counts on a selection/config table"
|
||
- "Adding any caller-owned per-pass cache to a hot engine loop"
|
||
component: scheduler
|
||
tags:
|
||
- performance
|
||
- query-storm
|
||
- workflow-selection
|
||
- drizzle
|
||
- per-pass-cache
|
||
- fnxc-workflowscheduling
|
||
related_components:
|
||
- development_workflow
|
||
- scheduler
|
||
- workflow_resolution
|
||
---
|
||
|
||
# Workflow selection read-once-per-tick in the scheduler (the RUFU-073 query storm)
|
||
|
||
## Symptom
|
||
|
||
Production CPU sat at 62–70% and the health API took 0.77–2.0s. A `cpuprofile` showed **70.6% of the
|
||
node event loop inside Drizzle's `buildQueryFromSourceParams`** (the SQL-string compiler) — the engine
|
||
was *building* identical SQL statements nonstop, not waiting on the database. `pg_stat_user_tables`
|
||
confirmed the loop was a **cache-miss loop on reads**:
|
||
|
||
- `project.task_workflow_selection.idx_scan` grew ~232 q/s nonstop (121M cumulative index-scans, only
|
||
281 inserts).
|
||
- `project.config` `workflow_prompt_overrides` ~108 q/s (127M scans, 0 inserts).
|
||
|
||
The DB was not the bottleneck (PG did 5 selects in 46ms, no locks, ~15 connections) — the node process
|
||
was flooded by per-read Drizzle SQL-string construction.
|
||
|
||
## Root cause
|
||
|
||
`resolveTaskParkedColumns(store, taskId)` resolved the task's workflow IR via
|
||
`resolveWorkflowIrForTask(store, taskId)` **without a selection cache**. That is a separate
|
||
PostgreSQL select of `task_workflow_selection` (plus a Drizzle SQL build) per *call*. In a single
|
||
scheduler event the SAME task is parked-resolved up to ~6×:
|
||
|
||
`task:moved` merge reconciliation, `task:updated` unpause wake, planning-finished wake, approval-cleared
|
||
wake, `task:deleted` dependency reconciliation, and the agent-link rollback path. With 22 active
|
||
projects that composed to ~340 q/s of Drizzle SQL building.
|
||
|
||
## The fix: a caller-owned per-tick selection cache
|
||
|
||
`resolveWorkflowIrForTask(store, taskId, irCache?, selectionCache?)` and its
|
||
`resolveWorkflowIrForTaskWithProvenance(..., selectionCache?)` already accept a
|
||
`WorkflowSelectionCache = Map<taskId, WorkflowSelection | undefined>`. The scheduler now creates a
|
||
**fresh** selection cache at the top of each event/loop scope and threads it through every
|
||
`resolveTaskParkedColumns(store, taskId, selectionCache?)` call. Note the argument positions differ:
|
||
the selection cache is the **3rd argument** of `resolveTaskParkedColumns`, whereas it is the
|
||
**4th argument** (after the optional `irCache`) of `resolveWorkflowIrForTask(store, taskId,
|
||
irCache?, selectionCache?)`. It is also threaded through the `emitHighOverlapFanoutWarnings` escalation
|
||
sweep and the PR-hydration sweep:
|
||
|
||
- Each task's selection is read **at most once per tick** (not once per park-resolution).
|
||
- The cache is **strictly per-call/per-pass**: a fresh Map per event/sweep, discarded at scope end.
|
||
- A selection WRITE by a later pass is always observed, because the next pass creates a fresh Map.
|
||
|
||
### In-flight read coalescing (race the first cut hit)
|
||
|
||
The first iteration put the coalescer only in the sync fallback, and concurrent wake closures sharing
|
||
one cache could each see the cache still-empty (the `.has` check runs before the first `await`
|
||
resolves) and all hit the DB. The fix tracks an in-flight promise per caller-owned cache object in a
|
||
`WeakMap<WorkflowSelectionCache, Map<taskId, Promise>>`: only the first closure performs the read;
|
||
later concurrent closures `await` the same promise. The weak key binds it to that pass, so it is NOT a
|
||
global/infinite cache and auto-releases when the pass cache is GC'd.
|
||
|
||
## Invariant (FNXC:WorkflowScheduling)
|
||
|
||
Selection caches are **per-call/per-scheduler-pass only** and are **strictly invalidated next poll** —
|
||
never a global/infinite LRU. A throwing read is deliberately **not** cached so transient PostgreSQL
|
||
failures are retried next pass; therefore **instrumentation must count reads, not infer them from
|
||
cache-key presence**.
|
||
|
||
## Verification contract
|
||
|
||
- A regression test drives one scheduler tick that clears unpause + planning + approval wakes for the
|
||
same task and asserts the selection is read exactly once (not 3×), that N distinct tasks read ≤ N
|
||
selections, that an empty task set reads zero, and that a **second** tick re-reads (proving the
|
||
cache is pass-scoped, so a selection write between ticks is observed).
|
||
- The existing resolver contract tests (`workflow-ir-selection-cache`, `workflow-ir-resolution-provenance`)
|
||
pin the per-call caching semantics; the sync-only store path dedups the same way.
|
||
- Production verification: `task_workflow_selection` idx_scan growth rate must drop from ~232 q/s to
|
||
below ~30 q/s, health API < 0.5s, CPU < 40%. |