feat(FN-3392): implement follow-up suggestion contract and policy system

Merges the evaluator follow-up suggestion system (FN-3392, Steps 1–5), which adds a normalized contract, provenance tracking, and lifecycle documentation for AI-generated follow-up tasks, alongside chat UI improvements including unread indicators in header and mobile nav, corrected message routing,

Fusion-Task-Id: FN-3392
This commit is contained in:
Fusion
2026-05-06 19:34:27 -07:00
committed by gsxdsm
parent d791fa90ab
commit 547f024717
11 changed files with 594 additions and 15 deletions

View File

@@ -210,7 +210,8 @@ Hybrid evaluator pipeline (FN-3389/FN-3391):
- **Signal summary:** `collectDeterministicSignals` (`eval-signal-collector.ts`) normalizes timing/workflow/review/log/commit summaries with stable fallbacks for missing metadata.
- **Evidence harvesting:** `collectTaskEvaluationEvidence` (`packages/engine/src/evaluator-evidence.ts`) reads existing task-store/git surfaces (`workflowStepResults`, documents, task activity log, agent logs, run-audit events, merge/PR metadata) and emits a bounded `TaskEvaluationEvidenceBundle` with fixed source-group ordering.
- **AI review:** `HybridEvaluatorService` (`packages/engine/src/evaluator.ts`) injects deterministic signals plus a dedicated `## Evidence` bundle section into a strict JSON prompt, runs a read-only AI session, validates the JSON payload, and merges AI advisory fields into persisted eval output while preserving core score authority.
- **Persistence boundary:** eval rows persist normalized evidence refs plus bounded excerpts/IDs (not full raw logs or unbounded command output). Source drill-down stays in original task/agent/run-audit stores and git history.
- **Follow-up policy engine:** `packages/engine/src/eval-followups.ts` normalizes raw evaluator drafts into canonical follow-up suggestions, applies deterministic suppression/dedupe rules, and (policy permitting) materializes triage tasks through `TaskStore.createTask()` with source provenance back to the parent task and eval run/suggestion IDs.
- **Persistence boundary:** eval rows persist normalized evidence refs plus bounded excerpts/IDs (not full raw logs or unbounded command output) and structured follow-up lifecycle state (`suggested`/`suppressed`/`created`) including suppression reason or created task linkage. Source drill-down stays in original task/agent/run-audit stores and git history.
- **Model resolution (temporary):** evaluator model selection first uses an explicit run override pair (`provider` + `modelId` together only), then falls back to the existing validator lane (`resolveValidatorSettingsModel`) until FN-3393 introduces dedicated evaluator settings.
- **Scheduled execution wiring:** CronRunner intercepts the sentinel command `fn eval --scheduled-batch` and executes in-process, invoking `runScheduledEvalBatch` with `HybridEvaluatorService`; `ProjectEngine` syncs scheduled eval automation on startup and on relevant settings changes.

View File

@@ -99,10 +99,51 @@ Stored references include task/run identifiers and source-specific drill-down fi
`packages/engine/src/evaluator.ts` injects the normalized bundle under a dedicated `## Evidence` prompt section. The evaluator is instructed to cite evidence IDs/labels from this section instead of inventing unsupported claims.
## Follow-up Suggestion Policy
Evaluator follow-ups are normalized into structured `followUps[]` records on each eval result (no freeform-only suggestions).
Each suggestion includes:
- stable `suggestionId` + `dedupeKey`
- `title`, `description`, `priority`, `severity`
- `rationale` and `evidenceRefs[]`
- policy recommendation (`shouldCreate`, `policyQualified`, `reason`)
- lifecycle state: `suggested` | `suppressed` | `created`
- suppression/debug fields when applicable: `suppressedReason`, `matchedTaskId`, `matchedSuggestionId`
- creation linkage when applicable: `createdTaskId`
### Policy modes
Backend policy modes used by evaluator orchestration:
- `persist_only`: persist normalized suggestions for manual review only
- `auto_create_qualified`: auto-create only policy-qualified suggestions
- `create_all_non_duplicates`: auto-create all non-suppressed, non-duplicate suggestions
Current project settings mapping:
- `taskEvaluationFollowUpPolicy = "off" | "suggest"``persist_only`
- `taskEvaluationFollowUpPolicy = "create"``auto_create_qualified`
### Dedupe + suppression guardrails
Suggestions are suppressed when they are:
- empty/generic (`empty_or_generic`)
- missing strong signal (`insufficient_signal`)
- duplicates of an already-open board task (`duplicate_open_task`)
- duplicates of a prior eval suggestion for the same parent task (`duplicate_prior_suggestion`)
Suppression reasons and matched IDs are persisted on the suggestion for auditability.
### Task creation provenance
When policy permits creation, evaluator code uses `TaskStore.createTask()` (no ad hoc file writes). Created tasks:
- are created in `triage`
- set `sourceParentTaskId` to the evaluated task
- set `sourceMetadata` with eval provenance (`type=eval_follow_up`, `runId`, `suggestionId`, `policyMode`, `dedupeKey`)
- include actionable context (problem summary, expected outcome, score/severity, rationale, evidence refs)
## Non-Goals
This contract does not define:
- follow-up task creation policy
- eval settings UX
- eval dashboard/list rendering

View File

@@ -214,7 +214,7 @@ Additional backend notes:
| `research_exports` | Persisted export records for research runs (`runId` FK cascade). Stores format, content, and optional file path. |
| `research_run_events` | Append-only event log for research run lifecycle tracking (`runId` FK cascade, ordered by `seq`). Records status transitions, phase changes, step lifecycle, and failure classifications. |
| `eval_runs` | Eval run lifecycle state (status, trigger, scope, evaluation window boundaries, evaluated task IDs/counts, aggregate scores, provenance). |
| `eval_task_results` | Per-task eval outcomes linked to runs (`runId` FK cascade), including durable task snapshots and structured score payloads. `categoryScores[]` stores canonical per-category fields (`category`, `deterministicScore`, `aiScore`, `finalScore`, `weight`, `band`, `rationale`, `evidence[]`), plus `overallScore` derived from category finals. Also stores deterministic/AI signal payloads, summary rationale, follow-up suggestions, and a bounded `TaskEvaluationEvidenceBundle` (fixed source-order groups, capped entry counts, max 500-char excerpts with truncation marker) embedded in result metadata for backward-compatible persistence. |
| `eval_task_results` | Per-task eval outcomes linked to runs (`runId` FK cascade), including durable task snapshots and structured score payloads. `categoryScores[]` stores canonical per-category fields (`category`, `deterministicScore`, `aiScore`, `finalScore`, `weight`, `band`, `rationale`, `evidence[]`), plus `overallScore` derived from category finals. Also stores deterministic/AI signal payloads, summary rationale, structured follow-up suggestions (`suggestionId`, `dedupeKey`, recommendation, lifecycle state, suppression fields, optional `createdTaskId` linkage), and a bounded `TaskEvaluationEvidenceBundle` (fixed source-order groups, capped entry counts, max 500-char excerpts with truncation marker) embedded in result metadata for backward-compatible persistence. |
| `eval_run_events` | Append-only eval run event trail (`runId` FK cascade, ordered by `seq`) for orchestration/debug auditing and downstream API/UI drill-down. |
Scope boundary note: the `project_auth_*` tables are strictly project-database membership/auth domain data. They do **not** replace or migrate global remote-access credentials/tokens, daemon auth, or model-provider credential settings (which remain in their existing global/project settings stores).