root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

FN-5943: maintain tasks FTS5 indexes automatically

Browse Source 
Keep the tasks FTS5 index compact and self-healing during routine maintenance.

- add FTS5 trigger helpers, index sizing utilities, and a schema migration that skips no-op searchable-field rewrites
- run merge/optimize/rebuild maintenance from self-healing with thresholds, cadence, and run-audit logging
- add focused FTS maintenance coverage and update storage/architecture docs for the new behavior

Files changed:
 docs/architecture.md                               |   1 +
 docs/storage.md                                    |  14 ++
 packages/core/src/__tests__/db-migrate.test.ts     |  50 ++--
 packages/core/src/__tests__/db.test.ts             |  84 +++----
 packages/core/src/__tests__/goals-schema.test.ts   |   2 +-
 packages/core/src/__tests__/insight-store.test.ts  |  10 +-
 .../src/__tests__/merge-request-record.test.ts     |   2 +-
 packages/core/src/__tests__/mission-store.test.ts  |   2 +-
 packages/core/src/__tests__/run-audit.test.ts      |   2 +-
 .../src/__tests__/store-archive-search.test.ts     |  33 +++
 .../core/src/__tests__/store-merge-queue.test.ts   |   2 +-
 packages/core/src/__tests__/task-documents.test.ts |   2 +-
 packages/core/src/archive-db.ts                    |   6 +-
 packages/core/src/db.ts                            | 162 ++++++++++---
 packages/core/src/store.ts                         |  16 ++
 .../engine/src/__tests__/fts-maintenance.test.ts   | 268 +++++++++++++++++++++
 packages/engine/src/self-healing.ts                |  70 ++++++
 .../src/store/__tests__/roadmap-store.test.ts      |   4 +-
 18 files changed, 622 insertions(+), 108 deletions(-)

Fusion-Task-Id: FN-5943

Fusion-Task-Lineage: 9a851ede-95ee-4b76-bc34-737912f0e3fc
This commit is contained in:
gsxdsm
2026-06-07 14:07:46 -07:00
parent 08d25f041a
commit 294209f64b
18 changed files with 622 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/architecture.md
View File

@@ -669,6 +669,7 @@ Runtime action-gate flow (v1):
- `GridlockDetector` (`gridlock-detector.ts`) — detects all-blocked todo pipelines and emits notification events (plus explicit clear signals when gridlock resolves)
- `TransientErrorDetector` (`transient-error-detector.ts`) — retriable error classification
- `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
- Batch 1 maintenance now includes `fts-maintenance` for the live task search index. When `fts5Available === true`, every maintenance tick runs an incremental `merge`, every 4th tick escalates to `optimize`, and any index larger than `32 MiB` or `1 MiB × live task count` is fully rebuilt. Each pass emits `task:fts-maintenance` run-audit telemetry with before/after byte counts.
- `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`.
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.

14
docs/storage.md
View File

@@ -136,6 +136,20 @@ Important execution nuance:
- done tasks: prefer `mergeDetails.landedFiles`
- in-progress/in-review (or legacy pre-FN-4646 tasks): fall back to `task.modifiedFiles`
## FTS5 task-index maintenance (FN-5943)
- Live task search uses the `tasks_fts` external-content FTS5 table in `fusion.db`; the archive log uses a separate `archived_tasks_fts` table in `archive.db`.
- `tasks_fts_au` is value-aware: even though hot task writes still upsert full rows, the trigger only fires when indexed text actually changes (`id`, `title`, `description`, `comments`, `deletedAt`). Status/step/worktree churn no longer rewrites the FTS row on every update.
- `Database.getFtsIndexBytes()` measures index size via `SELECT SUM(LENGTH(block)) FROM tasks_fts_data`. Fusion intentionally does **not** rely on `dbstat`, because node:sqlite builds do not guarantee `SQLITE_ENABLE_DBSTAT_VTAB`.
- `SelfHealingManager` Batch 1 now runs `fts-maintenance` when `fts5Available === true`:
- every maintenance tick: incremental `merge` compaction
- every 4th maintenance tick: heavier `optimize`
- immediate full `rebuild` when `tasks_fts` exceeds either `32 MiB` absolute or `1 MiB × live task count`
- Each maintenance pass emits run-audit telemetry with `mutationType: "task:fts-maintenance"` and `metadata` including `mode`, `bytesBefore`, `bytesAfter`, `taskCount`, `rebuilt`, and the threshold values.
- `rebuildFts5Index()` and migration 103 also set conservative FTS5 merge policy (`automerge=8`, `crisismerge=16`) so legitimate text edits merge segments sooner without forcing the heaviest optimize path on every write.
- `archived_tasks_fts` is intentionally **not** compacted by this task. The archive DB is effectively append-only for completed tasks, so it does not see the same constant-update churn as the live board; archive FTS compaction is deferred to a follow-up if archive bloat is observed.
- Separate attached-DB recommendation: **defer** moving `tasks_fts*` into a dedicated attached SQLite file. It would isolate FTS bloat/corruption from the main DB, but today it would complicate cross-DB joins in `searchTasks`, widen transaction/backup/checkpoint coordination, and add new multi-instance/polling failure modes on a path that is now bounded by guarded triggers + maintenance. Revisit only if live-main-DB FTS size or corruption remains operationally significant after FN-5943.
## SQLite write-path lock recovery (FN-4042 / FN-4083)
- Every disk-backed SQLite connection that Fusion opens for project storage (`fusion.db`), the central registry (`fusion-central.db`), archives (`archive.db`), and worktree hydration explicitly sets `PRAGMA busy_timeout = 5000` and `PRAGMA journal_mode = WAL` at connection open time before write work begins.