root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(FN-3390): export eval score category type and harden evaluator switch

Browse Source 
Exported the eval score category type from `@fusion/core` and added a defensive guard in the evaluator to prevent edge-case failures in the score evaluation switch.

Fusion-Task-Id: FN-3390
This commit is contained in:
Fusion
2026-05-06 11:09:05 -07:00
committed by gsxdsm
parent e4e0a594d6
commit 6c9f87f71a
10 changed files with 484 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/architecture.md
View File

@@ -195,6 +195,12 @@ Concrete references:
- Data model stores structured scoring/evidence/signal payloads plus durable `taskSnapshot` metadata so historical eval results remain readable even if the live task row later changes or is removed.
- Lifecycle safeguards mirror other core stores: deterministic list ordering, transition guards, terminal immutability for run rows, and active-run conflict protection for scheduled/task-completion triggers.
- `eval_task_results` enforces one row per `(runId, taskId)` via a unique index; store writes use upsert semantics to keep reruns idempotent.
- Canonical scoring contract is documented in `docs/evals.md`; authoritative score computation is centralized in `packages/core/src/eval-scoring.ts`.
Scoring authority boundary:
- Authoritative fields: `categoryScores[].finalScore`, `categoryScores[].band`, `categoryScores[].weight`, and `overallScore` (derived by `computeOverallScore`).
- Advisory/model-authored fields: category `aiScore`, category `rationale`, category `evidence`, and `overallRationale` text.
- Evaluator code (`packages/engine/src/evaluator.ts`) may provide AI category inputs, but must route final score computation through core helpers (`normalizeCategoryScore`, `computeOverallScore`) and must not persist AI-provided overall numbers as source of truth.
Hybrid evaluator pipeline (FN-3389):
- **Batch selection:** `runScheduledEvalBatch` in core computes a deterministic completed-task window (`windowStartExclusive` → `windowEndInclusive`) from the last completed scheduled run.