Commit Graph

17 Commits

Author SHA1 Message Date
Berlin Luk
7f87ff05f1 fix: decouple stuck detector timeout 2026-05-11 00:11:23 +08:00
Berlin Luk
873154f82a fix: keep stuck detector active by default 2026-05-10 23:55:36 +08:00
Fusion
2c497b3734 docs(FN-3538): clarify pause-safe stuck timeout behavior
- Update enginePaused setting docs to specify stuck-task timers are suspended while pauses are active
- Document that paused wall-clock time does not count toward taskStuckTimeoutMs, including shared globalPause windows
- Clarify that unpausing restores scheduling and grants active sessions a fresh stuck-task grace window before detection resumes

Fusion-Task-Id: FN-3538
2026-05-06 00:05:55 -07:00
gsxdsm
0923bffc1b fix(engine): respect global pause in reviewer + stuck detector
Reviewer subprocesses were spawned via fn_review_spec / fn_review_step
even with globalPause on, because reviewer.ts had no pause awareness.
Stuck detector also kept running, treating pause-disposed sessions as
inactivity and re-queuing tasks. Pause-transition listeners only called
session.dispose(), which doesn't always interrupt an in-flight LLM
stream — letting reviewer spawns leak through after pause flipped.

- reviewer.ts: re-read settings, return UNAVAILABLE without spawning
  when globalPause/enginePaused is on.
- stuck-task-detector.ts: skip checkStuckTasks() while paused.
- triage.ts / executor.ts: call session.abort() before dispose() in the
  pause-transition listener to interrupt in-flight work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 17:26:01 -07:00
gsxdsm
30b6822b00 feat(FN-2073): merge fusion/fn-2073 2026-04-18 20:56:12 -07:00
gsxdsm
f962be933d feat(FN-1461): fix stuck-task retry for step-session mode
- Add step-scoped tracking key support in StuckTaskDetector for step-session mode
- Update tracking keys to include step session IDs when runStepsInNewSessions is enabled
- Add tests for step-scoped tracking behavior in executor and stuck-task-detector
- Ensure stuck task detection works correctly with per-step retry recovery
2026-04-10 08:58:37 -07:00
gsxdsm
9b30cc323d fix: use canonical task ID in stuck-task-detector for step-session mode
In step-session mode each step is registered with StuckTaskDetector
under a compound key (e.g. "FN-1452-step-1") rather than the bare
task ID. When stuck detection fired, event.taskId was that compound
key, breaking three things simultaneously:

1. beforeRequeue("FN-1452-step-1") → store.getTask() threw (no such
   task) → stuckKillCount never incremented on the real task.

2. markStuckAborted("FN-1452-step-1") → activeStepExecutors.get()
   returned undefined (keyed by "FN-1452") → terminateAllSessions()
   never called, other step sessions kept running.

3. stuckAborted.set("FN-1452-step-1") → executor checks
   stuckAborted.has("FN-1452") → miss → stuckRequeue never set
   → task never moved to todo.

Fix: add a canonicalTaskId field to TrackedTask (defaults to the
tracking key for single-session mode where they are identical).
StepSessionExecutor now passes taskDetail.id as the third arg to
trackTask(). killAndRetry() uses entry.canonicalTaskId for all
external callbacks so they always resolve to the real task ID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 18:16:55 -07:00
gsxdsm
1b98f167a0 feat(FN-978): add diagnostic logging, semaphore resilience, and executor tests
- Add structured diagnostic logging to executor, stuck-task-detector, and pi.ts with subsystem prefixes
- Add defensive guards to AgentSemaphore (limit minimum 1, invalid limit handling)
- Add comprehensive integration tests for agent execution flow (executor.test.ts)
- Add unit tests for semaphore resilience (concurrency.test.ts) and stuck-task-detector (stuck-task-detector.test.ts)
- Fix TypeScript errors in test task objects and duplicate execution test
- Document engine diagnostic logging points in AGENTS.md
2026-04-05 13:37:21 -07:00
gsxdsm
4c2be10d73 feat(FN-882): add loop detection recovery with compact-and-resume
- Add ContextLimitDetector to detect agent loops via repeated tool call patterns
- Implement compact-and-resume strategy: summarize conversation and restart agent from current step
- Add loop recovery to StuckTaskDetector with configurable attempt tracking and retry limits
- Extend executor with automatic loop recovery on context limit detection
- Add loop recovery support to pi executor with same compact-and-resume pattern
- Add comprehensive tests for context-limit-detector, stuck-task-detector loop detection, executor, and pi recovery
- Add changeset for patch bump to @gsxdsm/fusion
- Update README with loop detection and recovery documentation
2026-04-04 19:47:48 -07:00
gsxdsm
de074420d2 fix: defer stuck-kill requeue to executor finally block to prevent race
When the stuck task detector killed a task and immediately called
moveTask("todo"), the scheduler could re-dispatch the task before the
old execution's finally block cleared this.executing. The new execute()
call hit the guard and silently returned, stranding the task in
"in-progress" with no active session or worktree (seen on FN-810/FN-912).

Move the requeue responsibility from StuckTaskDetector.killAndRetry to
the executor's finally block, which runs after this.executing.delete().
The beforeRequeue budget check now runs before session.dispose() and its
result is passed via StuckTaskEvent.shouldRequeue → markStuckAborted().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 12:26:13 -07:00
gsxdsm
866b97a1c2 fix(FN-874): recover stuck-killed tasks reliably 2026-04-04 09:45:45 -07:00
gsxdsm
b2e444ea88 feat(FN-850): add no-progress loop detection and improve stuck-task detector
- Add explicit no-progress loop detection signals to executor agent sessions
- Enhance stuck-task detector with configurable strategies (timeout, no-progress, combined)
- Add comprehensive test coverage for stuck-task detector (371 lines of tests)
- Remove modelFilter test utilities (198 lines of dead test code)
- Simplify modelFilter.ts by removing unused filtering logic
- Clean up dashboard server startup and minor README fix
2026-04-04 07:48:47 -07:00
gsxdsm
eef15f077c feat(engine): harden review pipeline with strict scope, build retry, and E2E tests
Improve the plan→review→approve→merge agent pipeline:

- Harden verdict extraction with JSON block parsing and anchored regexes
- Consolidate legacy/new merger conflict APIs into thin deprecated wrappers
- Add configurable strict scope enforcement (strictScopeEnforcement setting)
- Add build retry with timeout to merger (buildRetryCount, buildTimeoutMs)
- Add handleChangesRequested to PrCommentHandler for review feedback loop
- Remove dead code: handleFsChange, processTaskChange, unused imports/fields
- Add E2E multi-verdict sequence tests for the full review pipeline
- Fix unused parameter warnings across engine and core packages

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 00:29:34 -07:00
gsxdsm
bcb9915968 feat: add SelfHealingManager for unattended multi-day operation
Adds four self-healing subsystems to enable the engine to recover from
common failure modes without human intervention:

- Auto-unpause: clears rate-limit-triggered globalPause with escalating
  backoff (5 min → 60 min cap), resets on sustained recovery
- Stuck kill budget: caps task stuck-kill retries (default 3) to prevent
  infinite stuck→todo→stuck loops
- Periodic maintenance (every 15 min): git worktree prune, orphan cleanup,
  SQLite WAL checkpoint
- Worktree cap enforcement: removes oldest idle worktrees when count
  exceeds 2× maxWorktrees

New settings: autoUnpauseEnabled, autoUnpauseBaseDelayMs,
autoUnpauseMaxDelayMs, maxStuckKills, maintenanceIntervalMs.
New task field: stuckKillCount (schema v8 migration).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 15:35:12 -07:00
gsxdsm
fc0411713c feat(KB-330): rename internal packages from @kb/* to @fusion/*
- Rename @kb/core, @kb/dashboard, @kb/engine to @fusion/* namespace
- Update all import statements across 143+ files to use new package names
- Update workspace dependencies and root package.json references
- Fix bundler configurations (tsup, vite) for new package names
- Update test files and fix typecheck issues
- Add changeset file documenting the package rename
2026-03-31 13:33:44 -07:00
gsxdsm
f94265b639 feat(KB-249): immediate stuck task check on timeout setting change
- Add checkNow() method to StuckTaskDetector for immediate stuck task detection\n- Add comprehensive tests for checkNow() method\n- Wire up settings change handler in dashboard command\n- Update AGENTS.md documentation with immediate check behavior
2026-03-30 23:39:04 -07:00
gsxdsm
b7349a3f1f feat(KB-206): add stuck task detection and auto-recovery
- Add taskStuckTimeoutMs setting to detect tasks with no activity
- Create StuckTaskDetector to poll in-progress tasks every 30s
- Integrate heartbeat tracking into executor via step callbacks
- Implement recovery flow: abort stuck sessions, retry with preserved progress
- Export detector and wire into dashboard for real-time monitoring
- Add comprehensive tests and documentation to AGENTS.md
2026-03-30 17:16:35 -07:00