feat(FN-3392): implement follow-up suggestion contract and policy system
Merges the evaluator follow-up suggestion system (FN-3392, Steps 1–5), which adds a normalized contract, provenance tracking, and lifecycle documentation for AI-generated follow-up tasks, alongside chat UI improvements including unread indicators in header and mobile nav, corrected message routing, Fusion-Task-Id: FN-3392
This commit is contained in:
@@ -210,7 +210,8 @@ Hybrid evaluator pipeline (FN-3389/FN-3391):
|
||||
- **Signal summary:** `collectDeterministicSignals` (`eval-signal-collector.ts`) normalizes timing/workflow/review/log/commit summaries with stable fallbacks for missing metadata.
|
||||
- **Evidence harvesting:** `collectTaskEvaluationEvidence` (`packages/engine/src/evaluator-evidence.ts`) reads existing task-store/git surfaces (`workflowStepResults`, documents, task activity log, agent logs, run-audit events, merge/PR metadata) and emits a bounded `TaskEvaluationEvidenceBundle` with fixed source-group ordering.
|
||||
- **AI review:** `HybridEvaluatorService` (`packages/engine/src/evaluator.ts`) injects deterministic signals plus a dedicated `## Evidence` bundle section into a strict JSON prompt, runs a read-only AI session, validates the JSON payload, and merges AI advisory fields into persisted eval output while preserving core score authority.
|
||||
- **Persistence boundary:** eval rows persist normalized evidence refs plus bounded excerpts/IDs (not full raw logs or unbounded command output). Source drill-down stays in original task/agent/run-audit stores and git history.
|
||||
- **Follow-up policy engine:** `packages/engine/src/eval-followups.ts` normalizes raw evaluator drafts into canonical follow-up suggestions, applies deterministic suppression/dedupe rules, and (policy permitting) materializes triage tasks through `TaskStore.createTask()` with source provenance back to the parent task and eval run/suggestion IDs.
|
||||
- **Persistence boundary:** eval rows persist normalized evidence refs plus bounded excerpts/IDs (not full raw logs or unbounded command output) and structured follow-up lifecycle state (`suggested`/`suppressed`/`created`) including suppression reason or created task linkage. Source drill-down stays in original task/agent/run-audit stores and git history.
|
||||
- **Model resolution (temporary):** evaluator model selection first uses an explicit run override pair (`provider` + `modelId` together only), then falls back to the existing validator lane (`resolveValidatorSettingsModel`) until FN-3393 introduces dedicated evaluator settings.
|
||||
- **Scheduled execution wiring:** CronRunner intercepts the sentinel command `fn eval --scheduled-batch` and executes in-process, invoking `runScheduledEvalBatch` with `HybridEvaluatorService`; `ProjectEngine` syncs scheduled eval automation on startup and on relevant settings changes.
|
||||
|
||||
|
||||
@@ -99,10 +99,51 @@ Stored references include task/run identifiers and source-specific drill-down fi
|
||||
|
||||
`packages/engine/src/evaluator.ts` injects the normalized bundle under a dedicated `## Evidence` prompt section. The evaluator is instructed to cite evidence IDs/labels from this section instead of inventing unsupported claims.
|
||||
|
||||
## Follow-up Suggestion Policy
|
||||
|
||||
Evaluator follow-ups are normalized into structured `followUps[]` records on each eval result (no freeform-only suggestions).
|
||||
|
||||
Each suggestion includes:
|
||||
- stable `suggestionId` + `dedupeKey`
|
||||
- `title`, `description`, `priority`, `severity`
|
||||
- `rationale` and `evidenceRefs[]`
|
||||
- policy recommendation (`shouldCreate`, `policyQualified`, `reason`)
|
||||
- lifecycle state: `suggested` | `suppressed` | `created`
|
||||
- suppression/debug fields when applicable: `suppressedReason`, `matchedTaskId`, `matchedSuggestionId`
|
||||
- creation linkage when applicable: `createdTaskId`
|
||||
|
||||
### Policy modes
|
||||
|
||||
Backend policy modes used by evaluator orchestration:
|
||||
- `persist_only`: persist normalized suggestions for manual review only
|
||||
- `auto_create_qualified`: auto-create only policy-qualified suggestions
|
||||
- `create_all_non_duplicates`: auto-create all non-suppressed, non-duplicate suggestions
|
||||
|
||||
Current project settings mapping:
|
||||
- `taskEvaluationFollowUpPolicy = "off" | "suggest"` → `persist_only`
|
||||
- `taskEvaluationFollowUpPolicy = "create"` → `auto_create_qualified`
|
||||
|
||||
### Dedupe + suppression guardrails
|
||||
|
||||
Suggestions are suppressed when they are:
|
||||
- empty/generic (`empty_or_generic`)
|
||||
- missing strong signal (`insufficient_signal`)
|
||||
- duplicates of an already-open board task (`duplicate_open_task`)
|
||||
- duplicates of a prior eval suggestion for the same parent task (`duplicate_prior_suggestion`)
|
||||
|
||||
Suppression reasons and matched IDs are persisted on the suggestion for auditability.
|
||||
|
||||
### Task creation provenance
|
||||
|
||||
When policy permits creation, evaluator code uses `TaskStore.createTask()` (no ad hoc file writes). Created tasks:
|
||||
- are created in `triage`
|
||||
- set `sourceParentTaskId` to the evaluated task
|
||||
- set `sourceMetadata` with eval provenance (`type=eval_follow_up`, `runId`, `suggestionId`, `policyMode`, `dedupeKey`)
|
||||
- include actionable context (problem summary, expected outcome, score/severity, rationale, evidence refs)
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This contract does not define:
|
||||
|
||||
- follow-up task creation policy
|
||||
- eval settings UX
|
||||
- eval dashboard/list rendering
|
||||
|
||||
@@ -214,7 +214,7 @@ Additional backend notes:
|
||||
| `research_exports` | Persisted export records for research runs (`runId` FK cascade). Stores format, content, and optional file path. |
|
||||
| `research_run_events` | Append-only event log for research run lifecycle tracking (`runId` FK cascade, ordered by `seq`). Records status transitions, phase changes, step lifecycle, and failure classifications. |
|
||||
| `eval_runs` | Eval run lifecycle state (status, trigger, scope, evaluation window boundaries, evaluated task IDs/counts, aggregate scores, provenance). |
|
||||
| `eval_task_results` | Per-task eval outcomes linked to runs (`runId` FK cascade), including durable task snapshots and structured score payloads. `categoryScores[]` stores canonical per-category fields (`category`, `deterministicScore`, `aiScore`, `finalScore`, `weight`, `band`, `rationale`, `evidence[]`), plus `overallScore` derived from category finals. Also stores deterministic/AI signal payloads, summary rationale, follow-up suggestions, and a bounded `TaskEvaluationEvidenceBundle` (fixed source-order groups, capped entry counts, max 500-char excerpts with truncation marker) embedded in result metadata for backward-compatible persistence. |
|
||||
| `eval_task_results` | Per-task eval outcomes linked to runs (`runId` FK cascade), including durable task snapshots and structured score payloads. `categoryScores[]` stores canonical per-category fields (`category`, `deterministicScore`, `aiScore`, `finalScore`, `weight`, `band`, `rationale`, `evidence[]`), plus `overallScore` derived from category finals. Also stores deterministic/AI signal payloads, summary rationale, structured follow-up suggestions (`suggestionId`, `dedupeKey`, recommendation, lifecycle state, suppression fields, optional `createdTaskId` linkage), and a bounded `TaskEvaluationEvidenceBundle` (fixed source-order groups, capped entry counts, max 500-char excerpts with truncation marker) embedded in result metadata for backward-compatible persistence. |
|
||||
| `eval_run_events` | Append-only eval run event trail (`runId` FK cascade, ordered by `seq`) for orchestration/debug auditing and downstream API/UI drill-down. |
|
||||
|
||||
Scope boundary note: the `project_auth_*` tables are strictly project-database membership/auth domain data. They do **not** replace or migrate global remote-access credentials/tokens, daemon auth, or model-provider credential settings (which remain in their existing global/project settings stores).
|
||||
|
||||
Reference in New Issue
Block a user