FN-9016: add manager evaluation tools
Enable managers to review report evaluations and record coaching follow-ups. - Add manager-scoped evaluation read and follow-up extension tools. - Enforce management-subtree access through action gating and regression tests. - Document the tool surface and add a minor release changeset. Files changed: .changeset/fn-9016-manager-eval-tools.md | 7 + docs/agent-tool-surface-full-loop.md | 4 +- docs/agents.md | 8 +- packages/cli/skill/fusion/SKILL.md | 2 +- .../cli/skill/fusion/references/extension-tools.md | 22 ++ .../skill/fusion/references/fusion-capabilities.md | 2 + .../__tests__/extension-manager-eval-tools.test.ts | 159 +++++++++++++++ packages/cli/src/extension.ts | 227 +++++++++++++++++++++ .../engine/src/__tests__/agent-action-gate.test.ts | 1 + .../src/__tests__/gating-classifications.test.ts | 1 + .../src/__tests__/permanent-agent-gating.test.ts | 1 + .../engine/src/execution/gating-classifications.ts | 8 + 12 files changed, 438 insertions(+), 4 deletions(-) Fusion-Task-Id: FN-9016 Fusion-Task-Lineage: 1ad7c344-0f3c-4565-aabc-acb51481431d Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-9016-manager-eval-tools.md
Normal file
7
.changeset/fn-9016-manager-eval-tools.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": minor
|
||||
---
|
||||
|
||||
summary: Let managers review and coach evaluation results for agents in their reporting tree.
|
||||
category: feature
|
||||
dev: Adds `fn_agent_read_evaluations` and `fn_agent_evaluation_followup`, reusing the management-subtree boundary and action-gate classification.
|
||||
@@ -32,8 +32,8 @@ This is a source-grounded audit of the runtime factories in `packages/engine/src
|
||||
| --- | --- | --- | --- |
|
||||
| Ideation: diverge and converge | No `fn_ideation_*` factory or registration exists in `packages/engine/src/agent-tools.ts`. Agents can use generic documents, memory, `fn_ask_question`, and chat prose, but none is a persisted ideation operation. | **GAP:** add native, persisted `fn_ideation_diverge` and `fn_ideation_converge` (names provisional until the canonical ideation domain API is designed). The existing `docs/ideation/2026-06-02-open-ideation.md` is an artifact, not a callable pipeline. | No audited Command Center ideation operation was found. Define the domain operation and expose the same operation to its UI and agent callers; do not create a chat-only document shortcut. |
|
||||
| Research | `fn_research_run`, `fn_research_list`, `fn_research_get`, `fn_research_cancel`, and `fn_research_retry` come from `createResearchTools()` in `agent-tools.ts`. Executor only registers them when the research surface is enabled; heartbeat/chat wiring uses the same factories. | **Partial:** tools are gated by `isResearchToolSurfaceEnabled()` / `experimentalFeatures.researchView` in `tool-availability.ts`; disabled or unconfigured calls return setup guidance. Research results do not natively create/update mission hierarchy records. | Research view creates/runs/cancels/retries and exports runs ([Research](./research.md)); the same persisted `ResearchStore` run model is used. The dashboard can create/enrich a task from a finding, but has no research-to-feature/milestone handoff. |
|
||||
| Roadmap read: goals and mission hierarchy | `fn_goal_list` and `fn_goal_show` are created by `createGoalRetrievalTools()`. Task discovery (`fn_task_list`, `fn_task_search`, `fn_task_show`) can show linked task context. No mission-hierarchy reader is in `agent-tools.ts`. | **GAP:** add `fn_mission_list` / `fn_mission_show` (and hierarchy-focused reads as needed) to the shared engine/chat tool surface. | Goals View and Mission Manager expose their canonical read views. Pi extension `fn_mission_list` / `fn_mission_show` already use mission-store operations; reuse those operations and response shapes. |
|
||||
| Roadmap write: Mission → Milestone → Slice → Feature | No `fn_mission_*`, `fn_milestone_*`, `fn_slice_*`, or `fn_feature_*` is created or registered by `agent-tools.ts`. | **Confirmed GAP:** expose canonical mission writes, including create/update/delete, slice activation, feature creation/update/delete, and `fn_feature_link_task`; do not duplicate store semantics in chat. | Mission Manager owns mission/hierarchy editing, slice activation, and feature-to-task linkage. The pi extension already exposes the corresponding `fn_mission_*`, `fn_milestone_*`, `fn_slice_*`, and `fn_feature_*` operations. This is currently dual-access for UI ↔ extension, but not for engine/chat. |
|
||||
| Roadmap read: goals and mission hierarchy | Pi extension tools `fn_goal_list`, `fn_goal_show`, `fn_mission_list`, and `fn_mission_show` use the canonical goal and mission stores. Task discovery (`fn_task_list`, `fn_task_search`, `fn_task_show`) supplies linked delivery context. | **Present:** the mission hierarchy is the roadmap operation surface; no separate roadmap reader is needed. Engine/chat registration remains a deliberate surface-specific decision, not a roadmap-model gap. | Goals View and Mission Manager expose the same canonical read views. |
|
||||
| Roadmap write: Mission → Milestone → Slice → Feature | Pi extension tools provide `fn_mission_*`, `fn_milestone_*`, `fn_slice_*`, `fn_feature_*`, and goal operations, including hierarchy editing, slice activation, feature status/repair, and task linkage. | **Present:** mission, milestone, slice, feature, and goal tools are the roadmap write surface. No separate roadmap tool family is added. The standalone `Roadmap → RoadmapMilestone → RoadmapFeature` plugin model (`plugins/fusion-plugin-roadmap`; see `docs/architecture.md`) intentionally has no agent tools. | Mission Manager owns the same hierarchy editing, slice activation, and feature-to-task linkage through canonical stores. |
|
||||
| Task creation, planning, and assignment | `fn_task_create`, `fn_task_list`, `fn_task_search`, `fn_task_show`, `fn_delegate_task`, and `fn_task_assign` are factory-backed in `agent-tools.ts`; dashboard chat wires the discovery/create/delegate/assign tools. Workflow tools cover selected workflow inspection/authoring; task planner and triage convert a task into executable scope. | **Present, with boundary:** chat has no ambient task, so task documents/logs require explicit IDs. Planning-board tools intentionally provide board read parity. Mission-feature-to-task linkage remains unavailable to engine/chat because roadmap writes are missing. | New Task / Planning Board, task-detail assignment, workflow UI, and Mission Manager feature triage are the human paths. Existing task operations use the same `TaskStore`; task create/assignment is not a weaker chat-only path. |
|
||||
| Execution | The executor owns worktree acquisition, task lifecycle, workflow-step execution, and task-bound file tools. Heartbeats can create/delegate/assign but are not a substitute for an execution worktree. | **Present, guarded:** execution begins through scheduler/workflow dispatch rather than a generic chat “run arbitrary task now” shortcut. No new bypass should skip task state, leases, workflow holds, or action gates. | Board/workflow release and task lifecycle controls invoke the same scheduler/store transitions. Command Center is an observability/control surface, not a separate executor. |
|
||||
| E2E / verification | Task-bound executor sessions expose `fn_run_verification` from `packages/engine/src/run-verification-tool.ts`; it runs bounded test/lint/build/typecheck commands with worktree containment and concurrency controls. | **Partial:** dashboard chat and no-task heartbeat cannot safely call it because they lack an execution worktree. The missing chat control is a safe “request/observe verification on a selected task” operation, not exposing a raw command runner outside executor ownership. | Dashboard task/workflow result surfaces and Command Center funnel show execution state; the same verification result must be surfaced there. A future chat entry point must dispatch the existing task-owned verification operation and return its persisted result, not fork subprocess handling. |
|
||||
|
||||
@@ -120,6 +120,12 @@ The legacy `fn_agent_set_instructions` extension tool remains available for back
|
||||
|
||||
At least one instruction field must be provided. The legacy tool uses the same direct/indirect-report authorization model for agent callers and persists changes through `AgentStore.updateAgent`, so instruction edits are captured as normal agent config revisions.
|
||||
|
||||
### Manager evaluation tools
|
||||
|
||||
`fn_agent_read_evaluations` lets a manager read a named direct or indirect report's rating summary, commented ratings, reflection history, latest reflection, and performance summary. `fn_agent_evaluation_followup` records a 1–5 coaching rating for that same scoped report; the report reads the rating through its existing self-improvement loop. Agent callers may not target themselves, peers, ancestors, or unrelated agents. CLI/user calls without `ctx.agentId` are privileged operator calls and may read or record a follow-up for any durable agent. The follow-up remains subject to the normal agent action policy gate.
|
||||
|
||||
These tools complement, but do not replace, the existing self-only `fn_read_evaluations`: an agent's self-improvement cycle still reads only its own data. Use `fn_task_create` or `fn_delegate_task` to route implementation work discovered from feedback.
|
||||
|
||||
## Agent Field Parity Matrix
|
||||
|
||||
Every first-class editable agent field has a defined create/edit/import/template behavior. This ensures consistent round-tripping across all surfaces.
|
||||
@@ -694,7 +700,7 @@ Operators can disable this per agent in **Agent Detail → Settings → Heartbea
|
||||
|
||||
When `selfImproveEnabled !== false`, heartbeat runs periodically enter a self-improvement phase once `selfImproveIntervalMs` has elapsed since `lastSelfImproveAt` (or first run with available ratings). During that phase the agent is prompted to:
|
||||
|
||||
1. Call `fn_read_evaluations` to inspect ratings/reflections
|
||||
1. Call the self-only `fn_read_evaluations` to inspect its own ratings/reflections (managers use `fn_agent_read_evaluations` for scoped report visibility and `fn_agent_evaluation_followup` for coaching)
|
||||
2. Identify recurring quality issues and trends
|
||||
3. Call `fn_update_identity` to adjust its own `soul`, `instructionsText`, or `memory`
|
||||
4. Record concise improvement decisions
|
||||
|
||||
@@ -32,7 +32,7 @@ Mission → Milestone → Slice → Feature → Task
|
||||
- **GitHub tools** — `fn_task_import_github`, `fn_task_import_github_issue`, `fn_task_browse_github_issues`
|
||||
- **Mission tools** — `fn_mission_create`, `fn_mission_list`, `fn_mission_show`, `fn_mission_list_goals`, `fn_mission_link_goal`, `fn_mission_unlink_goal`, `fn_mission_backfill_assertions`, `fn_mission_delete`, `fn_mission_set_status`, `fn_mission_clear_blocked`, `fn_mission_update`, `fn_milestone_add`, `fn_slice_add`, `fn_feature_add`, `fn_feature_delete`, `fn_slice_delete`, `fn_milestone_delete`, `fn_slice_activate`, `fn_feature_link_task`, `fn_feature_set_status`, `fn_mission_reconcile`, `fn_feature_repair_validation`, `fn_feature_update`, `fn_milestone_update`
|
||||
- **Goal tools** — `fn_goal_list`, `fn_goal_create`, `fn_goal_archive`, `fn_goal_show`
|
||||
- **Agent tools** — `fn_agent_stop`, `fn_agent_start`, `fn_agent_create`, `fn_agent_update`, `fn_agent_set_instructions`, `fn_agent_delete`, `fn_list_agents`, `fn_delegate_task`, `fn_agent_show`, `fn_agent_org_chart`
|
||||
- **Agent tools** — `fn_agent_stop`, `fn_agent_start`, `fn_agent_create`, `fn_agent_update`, `fn_agent_set_instructions`, `fn_agent_read_evaluations`, `fn_agent_evaluation_followup`, `fn_agent_delete`, `fn_list_agents`, `fn_delegate_task`, `fn_agent_show`, `fn_agent_org_chart`
|
||||
- **Skills tools** — `fn_skills_search`, `fn_skills_install`
|
||||
- **Insight tools** — `fn_insight_list`, `fn_insight_show`, `fn_insight_run_list`, `fn_insight_run_show`
|
||||
- **Other tools** — `fn_web_fetch`, `fn_secret_get`, `fn_experiment_finalize`
|
||||
|
||||
@@ -680,6 +680,28 @@ Set the instructionsText and/or instructionsPath of one of the caller's direct o
|
||||
| `instructions_text` | string | — | Inline instructions. Pass an empty string to clear. |
|
||||
| `instructions_path` | string | — | Path to a markdown instructions file. Pass an empty string to clear. |
|
||||
|
||||
### fn_agent_read_evaluations
|
||||
|
||||
Read ratings, feedback, reflections, and performance data for a direct or indirect report.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `agent_id` | string | ✓ | Target agent ID or resolvable name |
|
||||
| `rating_limit` | number | — | Maximum ratings to return (default 10) |
|
||||
| `reflection_limit` | number | — | Maximum reflections to return (default 5) |
|
||||
|
||||
### fn_agent_evaluation_followup
|
||||
|
||||
Record a coaching evaluation follow-up for a direct or indirect report to read through its self-improvement loop.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `agent_id` | string | ✓ | Target agent ID or resolvable name |
|
||||
| `score` | number | ✓ | Evaluation score from 1 to 5 |
|
||||
| `comment` | string | ✓ | Coaching or follow-up note |
|
||||
| `category` | string | — | Optional evaluation category |
|
||||
| `task_id` | string | — | Optional related task ID |
|
||||
|
||||
### fn_agent_delete
|
||||
|
||||
Delete a non-ephemeral agent.
|
||||
|
||||
@@ -87,6 +87,8 @@ All skill/extension tool invocations in this catalog use the public `fn_*` names
|
||||
| `fn_agent_create` | Create a new non-ephemeral agent. |
|
||||
| `fn_agent_update` | Update editable configuration for an existing non-ephemeral agent. Agent callers can only update direct or indirect reports inside their management subtree; user/operator calls are privileged. |
|
||||
| `fn_agent_set_instructions` | Set the instructionsText and/or instructionsPath of one of the caller's direct or indirect reports. At least one of instructions_text or instructions_path is required; pass an empty string to clear a field. The change is persisted and recorded as a config revision. |
|
||||
| `fn_agent_read_evaluations` | Read ratings, feedback, reflections, and performance data for a direct or indirect report. |
|
||||
| `fn_agent_evaluation_followup` | Record a coaching evaluation follow-up for a direct or indirect report to read through its self-improvement loop. |
|
||||
| `fn_agent_delete` | Delete a non-ephemeral agent. |
|
||||
| `fn_list_agents` | List all available agents in the system. Shows each agent's name, role, state, personality (soul), and current assignment. Use this to discover which agents exist and what they specialize in before delegating work. |
|
||||
| `fn_delegate_task` | Create a new task and assign it to a specific agent for execution. The task lands in the selected workflow's ready lane (`todo` on the built-in board, whatever that workflow calls it otherwise) and will be picked up by the target agent on their next heartbeat cycle. Use fn_list_agents first to find available agents and their capabilities. Optionally pass workflow_id to select a workflow at creation time; use fn_workflow_list to discover valid IDs. |
|
||||
|
||||
159
packages/cli/src/__tests__/extension-manager-eval-tools.test.ts
Normal file
159
packages/cli/src/__tests__/extension-manager-eval-tools.test.ts
Normal file
@@ -0,0 +1,159 @@
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, expect, it } from "vitest";
|
||||
import { join } from "node:path";
|
||||
import { AgentStore, ReflectionStore } from "@fusion/core";
|
||||
import {
|
||||
createMockApi,
|
||||
createPgExtensionHarness,
|
||||
pgDescribe,
|
||||
registerExtension,
|
||||
requireTool,
|
||||
} from "./pg-extension-harness";
|
||||
|
||||
/*
|
||||
FNXC:AgentEvaluations 2026-08-12-22:30:
|
||||
Manager evaluation tools must prove their permission invariant through the PostgreSQL-backed
|
||||
extension path: managers may read and coach direct or indirect reports, while self, ancestors,
|
||||
and unrelated agents receive neither evaluation content nor a write.
|
||||
*/
|
||||
const h = createPgExtensionHarness("extension-manager-eval-tools");
|
||||
|
||||
async function withOrg(
|
||||
run: (context: {
|
||||
cwd: string;
|
||||
agentStore: AgentStore;
|
||||
reflectionStore: ReflectionStore;
|
||||
readTool: any;
|
||||
followupTool: any;
|
||||
ids: { manager: string; middle: string; leaf: string; peer: string };
|
||||
}) => Promise<void>,
|
||||
): Promise<void> {
|
||||
const cwd = h.rootDir();
|
||||
const agentStore = new AgentStore({
|
||||
rootDir: join(cwd, ".fusion"),
|
||||
asyncLayer: h.store().getAsyncLayer()!,
|
||||
});
|
||||
const reflectionStore = new ReflectionStore({ rootDir: join(cwd, ".fusion") });
|
||||
try {
|
||||
await agentStore.init();
|
||||
await reflectionStore.init();
|
||||
const manager = await agentStore.createAgent({ name: "manager", role: "engineer", metadata: {} });
|
||||
const middle = await agentStore.createAgent({ name: "middle-manager", role: "engineer", reportsTo: manager.id, metadata: {} });
|
||||
const leaf = await agentStore.createAgent({ name: "leaf-agent", role: "executor", reportsTo: middle.id, metadata: {} });
|
||||
const peer = await agentStore.createAgent({ name: "peer-agent", role: "executor", metadata: {} });
|
||||
const api = createMockApi();
|
||||
registerExtension(api);
|
||||
await run({
|
||||
cwd,
|
||||
agentStore,
|
||||
reflectionStore,
|
||||
readTool: requireTool(api, "fn_agent_read_evaluations"),
|
||||
followupTool: requireTool(api, "fn_agent_evaluation_followup"),
|
||||
ids: { manager: manager.id, middle: middle.id, leaf: leaf.id, peer: peer.id },
|
||||
});
|
||||
} finally {
|
||||
agentStore.close();
|
||||
}
|
||||
}
|
||||
|
||||
const execute = (tool: any, params: Record<string, unknown>, ctx: Record<string, unknown>) =>
|
||||
tool.execute("manager-eval-call", params, undefined, undefined, ctx);
|
||||
|
||||
pgDescribe("manager evaluation tools", () => {
|
||||
beforeAll(h.beforeAll);
|
||||
beforeEach(h.beforeEach);
|
||||
afterEach(h.afterEach);
|
||||
afterAll(h.afterAll);
|
||||
|
||||
it("registers manager evaluation schemas", async () => {
|
||||
await withOrg(async ({ readTool, followupTool }) => {
|
||||
expect(readTool.parameters).toMatchObject({ type: "object", required: ["agent_id"] });
|
||||
expect(followupTool.parameters).toMatchObject({ type: "object", required: ["agent_id", "score", "comment"] });
|
||||
});
|
||||
});
|
||||
|
||||
it("allows a manager to read direct-report ratings and reflections", async () => {
|
||||
await withOrg(async ({ cwd, agentStore, reflectionStore, readTool, ids }) => {
|
||||
await agentStore.addRating(ids.middle, { raterType: "user", score: 4, comment: "solid triage" });
|
||||
await reflectionStore.createReflection({
|
||||
agentId: ids.middle,
|
||||
trigger: "manual",
|
||||
metrics: {},
|
||||
insights: ["Keeps planning focused"],
|
||||
suggestedImprovements: ["Continue concise plans"],
|
||||
summary: "Strong planning reflection",
|
||||
});
|
||||
const result = await execute(readTool, { agent_id: ids.middle }, { cwd, agentId: ids.manager });
|
||||
expect(result.isError).not.toBe(true);
|
||||
expect(result.details).toMatchObject({ outcome: "read", agentId: ids.middle, summary: { averageScore: 4 } });
|
||||
expect(result.content[0].text).toContain("solid triage");
|
||||
expect(result.content[0].text).toContain("Strong planning reflection");
|
||||
});
|
||||
});
|
||||
|
||||
it("allows a manager to read an indirect report", async () => {
|
||||
await withOrg(async ({ cwd, readTool, ids }) => {
|
||||
const result = await execute(readTool, { agent_id: ids.leaf }, { cwd, agentId: ids.manager });
|
||||
expect(result.isError).not.toBe(true);
|
||||
expect(result.details).toMatchObject({ outcome: "empty", agentId: ids.leaf });
|
||||
});
|
||||
});
|
||||
|
||||
it("denies unrelated, self, and ancestor reads without leaking report data", async () => {
|
||||
await withOrg(async ({ cwd, agentStore, readTool, ids }) => {
|
||||
await agentStore.addRating(ids.peer, { raterType: "user", score: 1, comment: "PRIVATE PEER FEEDBACK" });
|
||||
const peerResult = await execute(readTool, { agent_id: ids.peer }, { cwd, agentId: ids.manager });
|
||||
expect(peerResult.isError).toBe(true);
|
||||
expect(peerResult.details).toMatchObject({ outcome: "denied", rule: "direct-or-indirect-reports-only" });
|
||||
expect(JSON.stringify(peerResult)).not.toContain("PRIVATE PEER FEEDBACK");
|
||||
const selfResult = await execute(readTool, { agent_id: ids.manager }, { cwd, agentId: ids.manager });
|
||||
const ancestorResult = await execute(readTool, { agent_id: ids.manager }, { cwd, agentId: ids.leaf });
|
||||
expect(selfResult.details).toMatchObject({ outcome: "denied", rule: "direct-or-indirect-reports-only" });
|
||||
expect(ancestorResult.details).toMatchObject({ outcome: "denied", rule: "direct-or-indirect-reports-only" });
|
||||
});
|
||||
});
|
||||
|
||||
it("allows privileged operators to read any durable agent", async () => {
|
||||
await withOrg(async ({ cwd, agentStore, readTool, ids }) => {
|
||||
await agentStore.addRating(ids.peer, { raterType: "user", score: 5, comment: "operator visible" });
|
||||
const result = await execute(readTool, { agent_id: ids.peer }, { cwd });
|
||||
expect(result.isError).not.toBe(true);
|
||||
expect(result.details).toMatchObject({ outcome: "read", agentId: ids.peer });
|
||||
});
|
||||
});
|
||||
|
||||
it("records subtree follow-ups and rejects out-of-subtree writes", async () => {
|
||||
await withOrg(async ({ cwd, agentStore, followupTool, ids }) => {
|
||||
const recorded = await execute(followupTool, {
|
||||
agent_id: ids.leaf,
|
||||
score: 2,
|
||||
comment: "needs tighter test scoping",
|
||||
category: "testing",
|
||||
}, { cwd, agentId: ids.manager });
|
||||
expect(recorded.details).toMatchObject({ outcome: "recorded", agentId: ids.leaf, score: 2, raterType: "agent" });
|
||||
await expect(agentStore.getRatings(ids.leaf)).resolves.toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ raterType: "agent", raterId: ids.manager, score: 2, comment: "needs tighter test scoping" }),
|
||||
]));
|
||||
|
||||
const peerBefore = await agentStore.getRatings(ids.peer);
|
||||
const denied = await execute(followupTool, { agent_id: ids.peer, score: 3, comment: "unauthorized" }, { cwd, agentId: ids.manager });
|
||||
expect(denied.isError).toBe(true);
|
||||
expect(denied.details).toMatchObject({ outcome: "denied", rule: "direct-or-indirect-reports-only" });
|
||||
await expect(agentStore.getRatings(ids.peer)).resolves.toHaveLength(peerBefore.length);
|
||||
});
|
||||
});
|
||||
|
||||
it("validates follow-up scores and comments before writing", async () => {
|
||||
await withOrg(async ({ cwd, agentStore, followupTool, ids }) => {
|
||||
for (const params of [
|
||||
{ agent_id: ids.leaf, score: 0, comment: "invalid score" },
|
||||
{ agent_id: ids.leaf, score: 6, comment: "invalid score" },
|
||||
{ agent_id: ids.leaf, score: 3, comment: " " },
|
||||
]) {
|
||||
const result = await execute(followupTool, params, { cwd, agentId: ids.manager });
|
||||
expect(result.isError).toBe(true);
|
||||
expect(result.details).toMatchObject({ outcome: "invalid", field: params.comment.trim() ? "score" : "comment" });
|
||||
}
|
||||
await expect(agentStore.getRatings(ids.leaf)).resolves.toHaveLength(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
createTaskStoreForBackend,
|
||||
drizzleSql,
|
||||
AgentStore,
|
||||
ReflectionStore,
|
||||
isEphemeralAgent,
|
||||
evaluateImplementationTaskBind,
|
||||
AGENT_VALID_TRANSITIONS,
|
||||
@@ -24,6 +25,7 @@ import {
|
||||
type InsightStatus,
|
||||
type InsightRunStatus,
|
||||
type InsightRunTrigger,
|
||||
type Agent,
|
||||
type AgentCapability,
|
||||
type AgentUpdateInput,
|
||||
getTaskDuplicateLineage,
|
||||
@@ -681,6 +683,68 @@ async function getAgentStore(cwd: string): Promise<AgentStore> {
|
||||
return new AgentStore({ rootDir: getFusionDir(cwd), asyncLayer: requireProjectLayer(projectStore, "CLI AgentStore") });
|
||||
}
|
||||
|
||||
type ManagerEvaluationToolErrorResult = {
|
||||
content: Array<{ type: "text"; text: string }>;
|
||||
isError: true;
|
||||
details: Record<string, unknown>;
|
||||
};
|
||||
|
||||
type ManagerEvaluationTargetResolution =
|
||||
| { kind: "error"; response: ManagerEvaluationToolErrorResult }
|
||||
| { kind: "allowed"; target: Agent; callerAgentId?: string };
|
||||
|
||||
/** Resolve and authorize a manager's evaluation target under the shared org-subtree boundary. */
|
||||
async function resolveManagerEvaluationTarget(
|
||||
agentStore: AgentStore,
|
||||
agentId: string,
|
||||
ctx: ExtensionCallerContext,
|
||||
): Promise<ManagerEvaluationTargetResolution> {
|
||||
const target = await agentStore.resolveAgent(agentId);
|
||||
if (!target) {
|
||||
return {
|
||||
kind: "error",
|
||||
response: {
|
||||
content: [{ type: "text" as const, text: `Agent '${agentId}' not found` }],
|
||||
isError: true as const,
|
||||
details: { outcome: "not_found", error: "Agent not found", agentId },
|
||||
},
|
||||
};
|
||||
}
|
||||
if (isEphemeralAgent(target)) {
|
||||
return {
|
||||
kind: "error",
|
||||
response: {
|
||||
content: [{ type: "text" as const, text: `ERROR: Cannot evaluate ephemeral/runtime agent ${target.id}` }],
|
||||
isError: true as const,
|
||||
details: { outcome: "invalid", field: "agent_id", agentId: target.id },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const callerAgentId = typeof ctx.agentId === "string" && ctx.agentId ? ctx.agentId : undefined;
|
||||
if (callerAgentId) {
|
||||
const chain = await agentStore.getChainOfCommand(target.id);
|
||||
const callerIndex = chain.findIndex((agent) => agent.id === callerAgentId);
|
||||
if (callerAgentId === target.id || callerIndex < 1) {
|
||||
return {
|
||||
kind: "error",
|
||||
response: {
|
||||
content: [{ type: "text" as const, text: "ERROR: You can only access evaluations for your own direct or indirect reports." }],
|
||||
isError: true as const,
|
||||
details: {
|
||||
outcome: "denied",
|
||||
agentId: target.id,
|
||||
callerAgentId,
|
||||
rule: "direct-or-indirect-reports-only",
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return { kind: "allowed", target, callerAgentId };
|
||||
}
|
||||
|
||||
function emitSecretAudit(
|
||||
store: TaskStore,
|
||||
ctx: { runId?: string; agentId?: string; taskId?: string },
|
||||
@@ -5891,6 +5955,169 @@ export default function kbExtension(pi: ExtensionAPI) {
|
||||
},
|
||||
});
|
||||
|
||||
// ── fn_agent_read_evaluations ──────────────────────────────────
|
||||
|
||||
/**
|
||||
* FNXC:AgentEvaluations 2026-08-12-22:11:
|
||||
* Manager agents need cross-agent evaluation visibility for direct and indirect reports,
|
||||
* while peers, ancestors, and self remain private. This intentionally reuses
|
||||
* getChainOfCommand so the management-subtree boundary has one auditable rule.
|
||||
* Reads use AgentStore and ReflectionStore rather than duplicating evaluation storage.
|
||||
*/
|
||||
pi.registerTool({
|
||||
name: "fn_agent_read_evaluations",
|
||||
label: "fn: Read Report Evaluations",
|
||||
description: "Read ratings, feedback, reflections, and performance data for a direct or indirect report.",
|
||||
promptSnippet: "Review evaluation and reflection results for an agent in your management subtree",
|
||||
promptGuidelines: [
|
||||
"Use to review evaluation data for a direct or indirect report in your management subtree",
|
||||
"Agent callers cannot target themselves, peers, ancestors, or agents outside their subtree",
|
||||
"Operator and CLI calls are privileged and may read any durable agent",
|
||||
],
|
||||
parameters: Type.Object({
|
||||
agent_id: Type.String({ description: "Target agent ID or resolvable name" }),
|
||||
rating_limit: Type.Optional(Type.Number({ minimum: 1, maximum: 50, description: "Maximum ratings to return (default 10)" })),
|
||||
reflection_limit: Type.Optional(Type.Number({ minimum: 1, maximum: 20, description: "Maximum reflections to return (default 5)" })),
|
||||
}),
|
||||
async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
|
||||
const agentStore = await getAgentStore(ctx.cwd);
|
||||
await agentStore.init();
|
||||
const resolved = await resolveManagerEvaluationTarget(agentStore, params.agent_id, ctx as ExtensionCallerContext);
|
||||
if (resolved.kind === "error") return resolved.response;
|
||||
|
||||
const ratingLimit = params.rating_limit ?? 10;
|
||||
const reflectionLimit = params.reflection_limit ?? 5;
|
||||
const reflectionStore = new ReflectionStore({ rootDir: getFusionDir(ctx.cwd) });
|
||||
await reflectionStore.init();
|
||||
const [summary, ratings, latestReflection, reflections, performance] = await Promise.all([
|
||||
agentStore.getRatingSummary(resolved.target.id),
|
||||
agentStore.getRatings(resolved.target.id, { limit: ratingLimit }),
|
||||
reflectionStore.getLatestReflection(resolved.target.id),
|
||||
reflectionStore.getReflections(resolved.target.id, reflectionLimit),
|
||||
reflectionStore.getPerformanceSummary(resolved.target.id),
|
||||
]);
|
||||
|
||||
const hasRatings = ratings.length > 0 || summary.totalRatings > 0;
|
||||
const hasReflections = Boolean(latestReflection) || reflections.length > 0;
|
||||
if (!hasRatings && !hasReflections) {
|
||||
return {
|
||||
content: [{ type: "text" as const, text: `${resolved.target.name} (${resolved.target.id})\n\nNo evaluation data available yet.` }],
|
||||
details: { outcome: "empty", agentId: resolved.target.id, agentName: resolved.target.name, summary, ratings, reflections, latestReflection, performance },
|
||||
};
|
||||
}
|
||||
|
||||
const formatScore = (score: number | null | undefined) =>
|
||||
typeof score === "number" && Number.isFinite(score) ? score.toFixed(2) : "n/a";
|
||||
const lines: string[] = [
|
||||
`Evaluation Summary: ${resolved.target.name} (${resolved.target.id})`,
|
||||
`- Average score: ${formatScore(summary.averageScore)}`,
|
||||
`- Trend: ${summary.trend}`,
|
||||
`- Total ratings: ${summary.totalRatings}`,
|
||||
];
|
||||
const categoryEntries = Object.entries(summary.categoryAverages ?? {});
|
||||
if (categoryEntries.length > 0) {
|
||||
lines.push("", "Category averages:");
|
||||
categoryEntries.forEach(([category, score]) => lines.push(`- ${category}: ${formatScore(score)}`));
|
||||
}
|
||||
const commentedRatings = ratings.filter((rating) => rating.comment?.trim());
|
||||
if (commentedRatings.length > 0) {
|
||||
lines.push("", "Recent rating comments:");
|
||||
commentedRatings.slice(0, 5).forEach((rating) => lines.push(`- [${rating.score}/5] ${rating.comment!.trim()}`));
|
||||
}
|
||||
if (latestReflection) {
|
||||
lines.push("", "Latest reflection:", `- Summary: ${latestReflection.summary}`);
|
||||
if (latestReflection.insights.length > 0) {
|
||||
lines.push("- Insights:");
|
||||
latestReflection.insights.forEach((insight) => lines.push(` - ${insight}`));
|
||||
}
|
||||
if (latestReflection.suggestedImprovements.length > 0) {
|
||||
lines.push("- Suggested improvements:");
|
||||
latestReflection.suggestedImprovements.forEach((item) => lines.push(` - ${item}`));
|
||||
}
|
||||
}
|
||||
if (reflections.length > 0) {
|
||||
lines.push("", "Recent reflection history:");
|
||||
reflections.slice(0, 5).forEach((reflection) => lines.push(`- ${reflection.timestamp}: ${reflection.summary}`));
|
||||
}
|
||||
|
||||
return {
|
||||
content: [{ type: "text" as const, text: lines.join("\n") }],
|
||||
details: { outcome: "read", agentId: resolved.target.id, agentName: resolved.target.name, summary, ratings, reflections, latestReflection, performance },
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
// ── fn_agent_evaluation_followup ───────────────────────────────
|
||||
|
||||
/*
|
||||
* FNXC:AgentEvaluations 2026-08-12-22:11:
|
||||
* Follow-ups reuse AgentStore.addRating, the same evaluation row read by the report's
|
||||
* self-improvement loop and dashboard rating routes, instead of creating a parallel
|
||||
* feedback channel. Task-level routing remains with fn_delegate_task and fn_task_create.
|
||||
*/
|
||||
pi.registerTool({
|
||||
name: "fn_agent_evaluation_followup",
|
||||
label: "fn: Record Evaluation Follow-up",
|
||||
description: "Record a coaching evaluation follow-up for a direct or indirect report to read through its self-improvement loop.",
|
||||
promptSnippet: "Record coaching feedback for an agent in your management subtree",
|
||||
promptGuidelines: [
|
||||
"Use to record actionable coaching feedback for a direct or indirect report",
|
||||
"Agent callers cannot target themselves, peers, ancestors, or agents outside their subtree",
|
||||
"Create or delegate separate implementation work with fn_task_create or fn_delegate_task",
|
||||
],
|
||||
parameters: Type.Object({
|
||||
agent_id: Type.String({ description: "Target agent ID or resolvable name" }),
|
||||
score: Type.Number({ minimum: 1, maximum: 5, description: "Evaluation score from 1 to 5" }),
|
||||
comment: Type.String({ minLength: 1, maxLength: 4000, description: "Coaching or follow-up note" }),
|
||||
category: Type.Optional(Type.String({ maxLength: 100, description: "Optional evaluation category" })),
|
||||
task_id: Type.Optional(Type.String({ description: "Optional related task ID" })),
|
||||
}),
|
||||
async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
|
||||
// FNXC:ToolPermissionGates 2026-08-12-22:11: This mutation intentionally falls through the exempt fallback as task_agent_mutation; do not classify it as a read-only coordination tool.
|
||||
const gated = await applyAgentPolicyGateForExtensionTool(
|
||||
"fn_agent_evaluation_followup",
|
||||
params as Record<string, unknown>,
|
||||
ctx as ExtensionCallerContext,
|
||||
);
|
||||
if (gated) return gated;
|
||||
|
||||
const agentStore = await getAgentStore(ctx.cwd);
|
||||
await agentStore.init();
|
||||
const resolved = await resolveManagerEvaluationTarget(agentStore, params.agent_id, ctx as ExtensionCallerContext);
|
||||
if (resolved.kind === "error") return resolved.response;
|
||||
|
||||
if (!Number.isFinite(params.score) || params.score < 1 || params.score > 5) {
|
||||
return {
|
||||
content: [{ type: "text" as const, text: "ERROR: score must be a finite value from 1 to 5" }],
|
||||
isError: true,
|
||||
details: { outcome: "invalid", field: "score" },
|
||||
};
|
||||
}
|
||||
const comment = params.comment.trim();
|
||||
if (!comment) {
|
||||
return {
|
||||
content: [{ type: "text" as const, text: "ERROR: comment must not be empty" }],
|
||||
isError: true,
|
||||
details: { outcome: "invalid", field: "comment" },
|
||||
};
|
||||
}
|
||||
|
||||
const raterType = resolved.callerAgentId ? "agent" as const : "user" as const;
|
||||
const rating = await agentStore.addRating(resolved.target.id, {
|
||||
raterType,
|
||||
...(resolved.callerAgentId ? { raterId: resolved.callerAgentId } : {}),
|
||||
score: params.score,
|
||||
...(params.category !== undefined ? { category: params.category } : {}),
|
||||
comment,
|
||||
...(params.task_id !== undefined ? { taskId: params.task_id } : {}),
|
||||
});
|
||||
return {
|
||||
content: [{ type: "text" as const, text: `Recorded evaluation follow-up for ${resolved.target.name} (${resolved.target.id}).` }],
|
||||
details: { outcome: "recorded", agentId: resolved.target.id, ratingId: rating.id, score: rating.score, raterType },
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
// ── fn_agent_delete ─────────────────────────────────────────────
|
||||
|
||||
pi.registerTool({
|
||||
|
||||
@@ -29,6 +29,7 @@ const FN_3548_COORDINATION_TOOLS = [
|
||||
"fn_memory_get",
|
||||
"fn_memory_append",
|
||||
"fn_read_evaluations",
|
||||
"fn_agent_read_evaluations",
|
||||
"fn_update_identity",
|
||||
"fn_reflect_on_performance",
|
||||
] as const;
|
||||
|
||||
@@ -130,6 +130,7 @@ describe("gating-classifications parity", () => {
|
||||
[
|
||||
"find",
|
||||
"fn_agent_org_chart",
|
||||
"fn_agent_read_evaluations",
|
||||
"fn_agent_show",
|
||||
"fn_artifact_list",
|
||||
"fn_artifact_register",
|
||||
|
||||
@@ -80,6 +80,7 @@ const FN_3548_COORDINATION_TOOLS = [
|
||||
"fn_memory_get",
|
||||
"fn_memory_append",
|
||||
"fn_read_evaluations",
|
||||
"fn_agent_read_evaluations",
|
||||
"fn_update_identity",
|
||||
"fn_reflect_on_performance",
|
||||
] as const;
|
||||
|
||||
@@ -247,7 +247,13 @@ export const READONLY_FN_TOOLS: ReadonlySet<string> = new Set([
|
||||
"fn_post_room_message",
|
||||
"fn_update_identity",
|
||||
"fn_reflect_on_performance",
|
||||
/*
|
||||
FNXC:ToolGovernance 2026-08-12-22:11:
|
||||
fn_agent_read_evaluations is a subtree-scoped read. Authorization is enforced by
|
||||
the extension tool's getChainOfCommand boundary, not by this action classifier.
|
||||
*/
|
||||
"fn_read_evaluations",
|
||||
"fn_agent_read_evaluations",
|
||||
]);
|
||||
|
||||
export const COORDINATION_EXEMPT_TOOLS = [
|
||||
@@ -308,6 +314,8 @@ export const COORDINATION_EXEMPT_TOOLS = [
|
||||
"fn_post_room_message",
|
||||
"fn_memory_append",
|
||||
"fn_read_evaluations",
|
||||
// FNXC:ToolGovernance 2026-08-12-22:11: The manager read is subtree-authorized in extension.ts; follow-up remains absent so it falls back to task_agent_mutation policy.
|
||||
"fn_agent_read_evaluations",
|
||||
"fn_update_identity",
|
||||
"fn_reflect_on_performance",
|
||||
] as const;
|
||||
|
||||
Reference in New Issue
Block a user