fusion/.fusion/tasks/KB-310/PROMPT.md

Task: KB-310 - Migrate from file-based storage to SQLite (hybrid: DB for metadata, files for blobs)

Created: 2026-03-31 Size: L

Review Level: 3 (Full)

Assessment: This is a foundational infrastructure change affecting all six data stores (TaskStore, Config, Archive, Activity Log, AgentStore, AutomationStore) with complex migration requirements, WAL mode concurrency, and zero-downtime backward compatibility needs. High blast radius, novel SQLite patterns for the codebase, and requires full verification.

Score: 7/8 — Blast radius: 2, Pattern novelty: 2, Security: 1, Reversibility: 2

Mission

Migrate the kb task board system from its current file-based storage to a hybrid SQLite + filesystem architecture. All structured/queryable metadata moves to a SQLite database at .kb/kb.db using Node.js built-in node:sqlite. Large blob files (PROMPT.md, agent.log, attachments) remain on the filesystem under .kb/tasks/{ID}/.

This migration must be seamless: on first run, detect legacy file-based data and auto-migrate it to SQLite while preserving old files as backups. The existing TaskStore, AgentStore, and AutomationStore method signatures must be preserved for backward compatibility.

Dependencies

• None

Context to Read First

1. packages/core/src/types.ts — All data type definitions (Task, Agent, Automation, etc.)
2. packages/core/src/store.ts — Current TaskStore implementation with file-based storage
3. packages/core/src/agent-store.ts — Current AgentStore implementation
4. packages/core/src/automation-store.ts — Current AutomationStore implementation
5. packages/core/src/global-settings.ts — GlobalSettingsStore (remains file-based in ~/.pi/kb/)
6. packages/dashboard/src/sse.ts — SSE endpoint for real-time updates

File Scope

New Files

• packages/core/src/db.ts — Database module with SQLite connection, schema, and migration
• packages/core/src/db-migrate.ts — Migration logic from file-based to SQLite

Modified Files

• packages/core/src/store.ts — Refactor TaskStore to use SQLite for all metadata
• packages/core/src/agent-store.ts — Refactor AgentStore to use SQLite
• packages/core/src/automation-store.ts — Refactor AutomationStore to use SQLite
• packages/core/src/index.ts — Export Database module
• packages/dashboard/src/sse.ts — Update change detection to use polling

Unchanged (for reference)

• packages/core/src/global-settings.ts — Global settings remain in ~/.pi/kb/settings.json
• packages/core/src/types.ts — Types remain unchanged

Steps

Step 1: Database Foundation

Create the Database module with SQLite schema and connection management.

• Create packages/core/src/db.ts with Database class
  • Use node:sqlite (Node 22.5+ built-in module)
  • Synchronous API for simplified transaction handling
  • WAL mode enabled for concurrent reader/writer access
  • Schema version tracking in __meta table
• Define complete schema in db.ts using camelCase column names (matches TypeScript):

  -- Tasks table with JSON columns for nested data
  CREATE TABLE tasks (
    id TEXT PRIMARY KEY,
    title TEXT,
    description TEXT NOT NULL,
    column TEXT NOT NULL,
    status TEXT,
    size TEXT,
    reviewLevel INTEGER,
    currentStep INTEGER DEFAULT 0,
    worktree TEXT,
    blockedBy TEXT,
    paused INTEGER DEFAULT 0,
    baseBranch TEXT,
    modelPresetId TEXT,
    modelProvider TEXT,
    modelId TEXT,
    validatorModelProvider TEXT,
    validatorModelId TEXT,
    mergeRetries INTEGER,
    error TEXT,
    summary TEXT,
    thinkingLevel TEXT,
    createdAt TEXT NOT NULL,
    updatedAt TEXT NOT NULL,
    columnMovedAt TEXT,
    -- JSON columns for nested arrays/objects
    dependencies TEXT DEFAULT '[]',
    steps TEXT DEFAULT '[]',
    log TEXT DEFAULT '[]',
    attachments TEXT DEFAULT '[]',
    steeringComments TEXT DEFAULT '[]',
    workflowStepResults TEXT DEFAULT '[]',
    prInfo TEXT,
    issueInfo TEXT,
    breakIntoSubtasks INTEGER DEFAULT 0,
    enabledWorkflowSteps TEXT DEFAULT '[]'
  );
  
  -- Config table (single row with project settings)
  CREATE TABLE config (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    nextId INTEGER DEFAULT 1,
    nextWorkflowStepId INTEGER DEFAULT 1,
    settings TEXT DEFAULT '{}',
    workflowSteps TEXT DEFAULT '[]',
    updatedAt TEXT
  );
  
  -- Activity log with indexed columns for efficient queries
  CREATE TABLE activityLog (
    id TEXT PRIMARY KEY,
    timestamp TEXT NOT NULL,
    type TEXT NOT NULL,
    taskId TEXT,
    taskTitle TEXT,
    details TEXT NOT NULL,
    metadata TEXT
  );
  CREATE INDEX idxActivityLogTimestamp ON activityLog(timestamp);
  CREATE INDEX idxActivityLogType ON activityLog(type);
  CREATE INDEX idxActivityLogTaskId ON activityLog(taskId);
  
  -- Archived tasks table (migrated from archive.jsonl)
  CREATE TABLE archivedTasks (
    id TEXT PRIMARY KEY,
    data TEXT NOT NULL, -- Full ArchivedTaskEntry as JSON
    archivedAt TEXT NOT NULL
  );
  CREATE INDEX idxArchivedTasksId ON archivedTasks(id);
  
  -- Automations table
  CREATE TABLE automations (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    description TEXT,
    scheduleType TEXT NOT NULL,
    cronExpression TEXT NOT NULL,
    command TEXT NOT NULL,
    enabled INTEGER DEFAULT 1,
    timeoutMs INTEGER,
    steps TEXT, -- JSON array of AutomationStep
    nextRunAt TEXT,
    lastRunAt TEXT,
    lastRunResult TEXT, -- JSON
    runCount INTEGER DEFAULT 0,
    runHistory TEXT DEFAULT '[]', -- JSON array
    createdAt TEXT NOT NULL,
    updatedAt TEXT NOT NULL
  );
  
  -- Agents table
  CREATE TABLE agents (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    role TEXT NOT NULL,
    state TEXT NOT NULL DEFAULT 'idle',
    taskId TEXT,
    createdAt TEXT NOT NULL,
    updatedAt TEXT NOT NULL,
    lastHeartbeatAt TEXT,
    metadata TEXT DEFAULT '{}' -- JSON
  );
  
  -- Agent heartbeat events
  CREATE TABLE agentHeartbeats (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agentId TEXT NOT NULL,
    timestamp TEXT NOT NULL,
    status TEXT NOT NULL,
    runId TEXT NOT NULL,
    FOREIGN KEY (agentId) REFERENCES agents(id) ON DELETE CASCADE
  );
  CREATE INDEX idxAgentHeartbeatsAgentId ON agentHeartbeats(agentId);
  CREATE INDEX idxAgentHeartbeatsRunId ON agentHeartbeats(runId);
  
  -- Schema version tracking
  CREATE TABLE __meta (
    key TEXT PRIMARY KEY,
    value TEXT
  );
  INSERT INTO __meta (key, value) VALUES ('schemaVersion', '1');
  

• Implement change detection polling mechanism:
  • Store lastModified timestamp (epoch ms) in __meta table
  • getLastModified(): number method returns last change timestamp
  • bumpLastModified(): void updates timestamp on every write operation
• Implement helper methods for JSON serialization/deserialization:
  • toJson(value: unknown): string — stringifies arrays/objects, returns '[]' for empty arrays
  • fromJson<T>(json: string | null): T | undefined — parses safely, returns undefined for null/empty
• Implement Database class with these methods:
  • constructor(kbDir: string) — opens database at .kb/kb.db, enables WAL mode
  • init(): void — creates tables if they don't exist
  • close(): void — closes database connection
  • transaction<T>(fn: () => T): T — executes function inside SQLite transaction
  • prepare(sql: string): Statement — returns prepared statement object
  • getLastModified(): number — reads __meta.lastModified
  • bumpLastModified(): void — updates __meta.lastModified to current epoch ms
• Export createDatabase(kbDir: string): Database factory function
• Run pnpm build to verify no TypeScript errors
• Write unit tests in packages/core/src/db.test.ts:
  • Schema creation works
  • WAL mode is enabled
  • Change detection timestamps work
  • JSON serialization helpers work
  • Transactions work atomically

Artifacts:

• packages/core/src/db.ts (new)
• packages/core/src/db.test.ts (new)

Step 2: Migration System

Create auto-migration logic from file-based to SQLite.

• Create packages/core/src/db-migrate.ts with migration functions:
  • detectLegacyData(kbDir: string): boolean — returns true if .kb/tasks/ exists OR .kb/config.json exists OR .kb/agents/ exists OR .kb/automations/ exists OR .kb/activity-log.jsonl exists OR .kb/archive.jsonl exists (but .kb/kb.db does NOT exist)
  • migrateFromLegacy(kbDir: string, db: Database): Promise<void> — performs full migration
• Implement migration steps in order:
  1. Migrate config.json → config table (preserve nextId as nextId, workflowSteps array, settings object)
  2. Iterate .kb/tasks/*/task.json → tasks table (map each JSON field to column, serialize arrays to JSON strings)
  3. Migrate .kb/activity-log.jsonl → activityLog table (parse each line, map fields)
  4. Migrate .kb/archive.jsonl → archivedTasks table (parse each line, store full entry as data JSON)
  5. Migrate .kb/automations/*.json → automations table (read each file, map fields)
  6. Migrate .kb/agents/*.json → agents table (read agent files, map fields)
  7. Migrate .kb/agents/*-heartbeats.jsonl → agentHeartbeats table (parse each heartbeat line)
• Create backup of old files after successful migration:
  • Rename .kb/tasks/ → .kb/tasks.bak/ (if exists)
  • Rename .kb/config.json → .kb/config.json.bak (if exists)
  • Rename .kb/activity-log.jsonl → .kb/activity-log.jsonl.bak (if exists)
  • Rename .kb/archive.jsonl → .kb/archive.jsonl.bak (if exists)
  • Rename .kb/automations/ → .kb/automations.bak/ (if exists)
  • Rename .kb/agents/ → .kb/agents.bak/ (if exists)
• Handle partial/corrupt legacy data gracefully:
  • Skip invalid task.json files with console warning (don't fail entire migration)
  • Skip malformed archive.jsonl lines (continue to next line)
  • Skip malformed activity log lines
  • Wrap each migration step in try/catch, log errors but continue
• Implement getMigrationStatus(kbDir: string): { hasLegacy: boolean; hasDatabase: boolean; needsMigration: boolean }
• Write tests for migration logic:
  • Happy path: all legacy data migrates correctly, data integrity preserved
  • Partial corruption: valid data migrates, corrupt data is skipped with warnings
  • Idempotent: re-running migration with existing SQLite does nothing (detects already migrated)
  • Empty project: fresh project with no legacy data creates empty database correctly
• Run pnpm build and verify tests pass

Artifacts:

• packages/core/src/db-migrate.ts (new)
• packages/core/src/db-migrate.test.ts (new)

Step 3: TaskStore SQLite Migration

Refactor TaskStore to use SQLite for all structured metadata.

• Update TaskStore constructor to accept Database instance and store rootDir only (not kb paths):

  constructor(
    private rootDir: string,
    private db: Database,
    globalSettingsDir?: string
  )
  

• Remove kbDir, tasksDir, configPath, archiveLogPath, activityLogPath — these move to Database
• Remove withTaskLock and withConfigLock promise chains — use db.transaction() instead
• Remove recentlyWritten, debounceTimers — change detection will use polling
• Replace file-based config operations:
  • readConfig() → SELECT * FROM config WHERE id = 1, parse settings/workflowSteps JSON
  • writeConfig() → INSERT OR REPLACE INTO config ...
• Replace task CRUD operations with SQLite:
  • readTaskJson(dir) → SELECT * FROM tasks WHERE id = ?, parse JSON columns
  • atomicWriteTaskJson(dir, task) → INSERT OR REPLACE INTO tasks ... with JSON stringification
  • listTasks() → SELECT * FROM tasks ORDER BY createdAt, parse JSON for each row
  • getTask() → SELECT * FROM tasks WHERE id = ? + read PROMPT.md from disk via taskDir()
• Update allocateId() to use transaction:

  return this.db.transaction(() => {
    const row = this.db.prepare('SELECT nextId, settings FROM config WHERE id = 1').get() as { nextId: number; settings: string };
    const settings = JSON.parse(row.settings || '{}');
    const prefix = settings.taskPrefix || 'KB';
    const id = `${prefix}-${String(row.nextId).padStart(3, '0')}`;
    this.db.prepare('UPDATE config SET nextId = ?').run(row.nextId + 1);
    this.db.bumpLastModified();
    return id;
  });
  

• Update createTask():
  • Insert row into tasks table with JSON-serialized arrays
  • Write PROMPT.md to disk (unchanged)
  • Call db.bumpLastModified()
  • Emit task:created event
• Update updateTask():
  • Use db.prepare().run() to update columns
  • JSON-serialize dependencies, steps, etc.
  • Call db.bumpLastModified()
  • Write PROMPT.md if updates.prompt provided
• Update moveTask(), pauseTask(), updateStep(), logEntry(), addSteeringComment():
  • All use db.transaction() for atomicity
  • Update relevant columns
  • Call db.bumpLastModified() after write
  • Emit appropriate events
• Update updatePrInfo() and updateIssueInfo():
  • Use db.transaction()
  • Read current task with SELECT * FROM tasks WHERE id = ?
  • Compare previous prInfo/issueInfo to detect badgeChanged
  • Update prInfo/issueInfo JSON columns
  • Append to log JSON array
  • Call db.bumpLastModified()
  • Emit task:updated if badgeChanged
• Keep appendAgentLog() file-based (agent.log remains JSONL file):
  • Use same implementation but ensure taskDir() still works
  • taskDir(id) returns join(rootDir, '.kb', 'tasks', id) (computed, not stored)
• Keep getAgentLogs() file-based:
  • Same implementation, just ensure paths work
• Update file-based operations for blob data:
  • taskDir(id) — compute path: join(this.rootDir, '.kb', 'tasks', id)
  • addAttachment() — write file to disk, store metadata in SQLite attachments JSON column
  • getAttachment() — read metadata from SQLite, return disk path
  • deleteAttachment() — delete from disk, update SQLite attachments JSON column
• Replace activity log operations:
  • recordActivity() → INSERT INTO activityLog ..., call db.bumpLastModified()
  • getActivityLog() → SELECT * FROM activityLog ORDER BY timestamp DESC, parse metadata JSON
  • clearActivityLog() → DELETE FROM activityLog
• Replace archive operations:
  • readArchiveLog() → SELECT id, archivedAt, data FROM archivedTasks ORDER BY archivedAt DESC
  • findInArchive() → SELECT * FROM archivedTasks WHERE id = ?
  • archiveTask() → within transaction: insert to archivedTasks, delete from tasks if cleanup
  • cleanupArchivedTasks() → for each archived task with directory: insert compact entry to archivedTasks, delete directory
  • restoreFromArchive() → insert back to tasks from archivedTasks.data, regenerate PROMPT.md
  • unarchiveTask() → check if dir exists (if not restore first), then update column in SQLite
• Replace workflow step operations:
  • createWorkflowStep() → within transaction: read config.workflowSteps, append new step, update config
  • listWorkflowSteps() → read config.workflowSteps, parse JSON array
  • getWorkflowStep() → find in parsed workflowSteps array
  • updateWorkflowStep() → find in array, update, write back to config
  • deleteWorkflowStep() → filter array, write back, then update tasks' enabledWorkflowSteps
• Update duplicateTask(), refineTask():
  • Use SQLite for task creation
  • Copy attachments via disk operations
• Update mergeTask():
  • Use transaction to read/update task
  • Git operations unchanged
  • Call db.bumpLastModified()
• Update deleteTask():
  • SQLite: DELETE FROM tasks WHERE id = ?
  • Disk: delete directory recursively (attachments, PROMPT.md, agent.log)
  • Call db.bumpLastModified()
• Update parseStepsFromPrompt(), parseDependenciesFromPrompt(), parseFileScopeFromPrompt():
  • Ensure they can read PROMPT.md from disk (these read files, unchanged)
• Update change detection (watch(), stopWatching()):
  • Remove fs.watch() based implementation
  • Implement polling: setInterval that checks db.getLastModified() every 1 second
  • When timestamp changes, query for changes and emit events
  • Maintain taskCache Map for diffing to determine event type
  • Preserve event emission: task:created, task:moved, task:updated, task:deleted, task:merged
• Update init():
  • Ensure task directories exist for blobs (.kb/tasks/)
  • db.init() should already be called by caller
• Preserve getRootDir() method (used by dashboard terminal service)
• Run targeted tests: pnpm test packages/core/src/store.test.ts
• Fix any failing tests

Artifacts:

• packages/core/src/store.ts (modified - large refactor)

Step 4: AgentStore SQLite Migration

Refactor AgentStore to use SQLite.

• Update AgentStore constructor:

  constructor(
    options: AgentStoreOptions & { db: Database }
  )
  

• Remove rootDir, agentsDir — these move to Database
• Remove locks Map — use db.transaction() instead
• Replace agent file operations:
  • readAgentFile() → SELECT * FROM agents WHERE id = ?, parse metadata JSON
  • writeAgent() → INSERT OR REPLACE INTO agents ...
  • getAgent() → query SQLite, parse metadata, return Agent
  • listAgents() → SELECT * FROM agents ORDER BY createdAt DESC, apply filters in code
• Replace heartbeat file operations:
  • recordHeartbeat() → within transaction: INSERT INTO agentHeartbeats ..., update agent.lastHeartbeatAt if status === 'ok'
  • getHeartbeatHistory() → SELECT * FROM agentHeartbeats WHERE agentId = ? ORDER BY timestamp DESC LIMIT ?
• Implement heartbeat run reconstruction from SQLite data:
  • getActiveHeartbeatRun(agentId) → query heartbeats for agent, group by runId, find run without terminal status ('missed')
  • getCompletedHeartbeatRuns(agentId) → query heartbeats, group by runId, filter completed/terminated
  • startHeartbeatRun(agentId) → record heartbeat with new runId via recordHeartbeat()
  • endHeartbeatRun(runId, status) → find agent for runId, record terminal heartbeat
• Update createAgent(), updateAgent(), updateAgentState(), assignTask():
  • Use db.transaction() for atomicity
  • Update agents table
  • Call db.bumpLastModified()
  • Emit events (unchanged)
• Update deleteAgent():
  • SQLite handles cascade delete via foreign key
  • DELETE FROM agents WHERE id = ?
  • Call db.bumpLastModified()
• Update listAgents(filter?):
  • Query all, parse metadata, apply filter in code
• Remove withLock() promise chaining — SQLite transactions handle serialization
• Update init() — no longer needs to create agents directory
• Preserve all method signatures and EventEmitter behavior exactly
• Run targeted tests for AgentStore (create packages/core/src/agent-store.test.ts if it doesn't exist)
• Verify agent:created, agent:updated, agent:deleted, agent:heartbeat, agent:stateChanged events still fire

Artifacts:

• packages/core/src/agent-store.ts (modified)
• packages/core/src/agent-store.test.ts (new or modified)

Step 5: AutomationStore SQLite Migration

Refactor AutomationStore to use SQLite.

• Update AutomationStore constructor:

  constructor(
    private rootDir: string,
    private db: Database
  )
  

• Remove automationsDir — automations now in SQLite
• Remove scheduleLocks Map — use db.transaction() instead
• Replace schedule file operations:
  • readScheduleJson() → SELECT * FROM automations WHERE id = ?, parse steps/runHistory JSON
  • atomicWriteScheduleJson() → INSERT OR REPLACE INTO automations ...
  • listSchedules() → SELECT * FROM automations ORDER BY createdAt
• Update createSchedule():
  • Generate UUID using randomUUID() (same as current)
  • Serialize steps and runHistory to JSON
  • Insert row with db.prepare().run()
  • Call db.bumpLastModified()
  • Emit schedule:created
• Update updateSchedule():
  • Use db.transaction()
  • Update columns, recompute nextRunAt if scheduleType/cronExpression changed
  • Call db.bumpLastModified()
  • Emit schedule:updated
• Update reorderSteps():
  • Read schedule, parse steps JSON, reorder, serialize, update
  • Call db.bumpLastModified()
• Update recordRun():
  • Within transaction: read schedule, parse runHistory, unshift new result, cap at MAX_RUN_HISTORY, serialize, update
  • Update lastRunAt, lastRunResult, runCount, recompute nextRunAt
  • Call db.bumpLastModified()
  • Emit schedule:run
• Update getDueSchedules():
  • SELECT * FROM automations WHERE enabled = 1 AND nextRunAt <= ?
  • Pass current ISO timestamp as parameter
• Update deleteSchedule():
  • DELETE FROM automations WHERE id = ?
  • Call db.bumpLastModified()
  • Emit schedule:deleted
• Update computeNextRun() — implementation unchanged (uses cron-parser)
• Remove withScheduleLock() promise chaining — use db.transaction()
• Update init() — no longer needs to create automations directory
• Preserve all method signatures and EventEmitter events exactly
• Run targeted tests: pnpm test packages/core/src/automation-store.test.ts
• Fix any failing tests

Artifacts:

• packages/core/src/automation-store.ts (modified)

Step 6: Dashboard SSE Update

Update the change detection mechanism to use SQLite polling.

• Read packages/dashboard/src/sse.ts to understand current implementation
• The current implementation listens to store events (task:created, task:moved, task:updated, task:deleted, task:merged)
• Verify that Store continues to emit these events (implemented in Step 3 and 4)
• No changes needed to sse.ts if stores continue to emit events correctly
• Update any dashboard API routes that create stores to pass Database instance:
  • Check packages/dashboard/app/routes/ for any route files that instantiate TaskStore, AgentStore, or AutomationStore
  • Update to create Database instance first, pass to store constructors
• Verify dashboard can still:
  • Create tasks (through QuickEntryBox, InlineCreateCard)
  • Move tasks between columns
  • Update task details
  • Archive/unarchive tasks
  • Show real-time updates via SSE
• Run dashboard tests if available

Artifacts:

• packages/dashboard/src/sse.ts (verify no changes needed, or update if required)
• Any affected API routes in packages/dashboard/app/routes/

Step 7: Store Initialization & Entry Points

Update initialization sequence to resolve Database creation before stores.

• Update packages/core/src/index.ts:
  • Export Database class from db.ts
  • Export createDatabase(kbDir: string): Database factory function
  • Export detectLegacyData, migrateFromLegacy, getMigrationStatus from db-migrate.ts
• Update CLI initialization (check packages/cli/src/ for store creation):
  • Find where TaskStore, AgentStore, AutomationStore are instantiated
  • Update to create Database first:

    import { createDatabase, detectLegacyData, migrateFromLegacy } from '@kb/core';
    
    const kbDir = join(rootDir, '.kb');
    const db = createDatabase(kbDir);
    db.init();
    
    if (detectLegacyData(kbDir)) {
      await migrateFromLegacy(kbDir, db);
    }
    
    const store = new TaskStore(rootDir, db, globalSettingsDir);
    const agentStore = new AgentStore({ db, rootDir });
    const automationStore = new AutomationStore(rootDir, db);
    

• Update dashboard server initialization (check packages/dashboard/):
  • Same pattern: create Database, detect legacy, migrate, then create stores
• Verify GlobalSettingsStore remains unchanged (still file-based in ~/.pi/kb/)
• Ensure stores handle the case where they're initialized with fresh database (no data)

Artifacts:

• packages/core/src/index.ts (modified)
• Any CLI initialization code that creates stores
• Any dashboard server initialization code

Step 8: Testing & Verification

ZERO test failures allowed. Full test suite as quality gate.

• Run full test suite: pnpm test
• Verify all tests pass in packages/core/:
  • store.test.ts passes
  • agent-store.test.ts passes (or create if missing)
  • automation-store.test.ts passes
  • db.test.ts passes
  • db-migrate.test.ts passes
• Run pnpm build to ensure no TypeScript errors across all packages
• Manual verification steps (perform these manually or document for QA):
  1. Create a fresh project: mkdir test-project && cd test-project && git init
  2. Run kb init (through CLI or dashboard)
  3. Verify .kb/kb.db is created
  4. Create a task → verify it appears in database and on dashboard
  5. Move task through columns → verify state updates in both DB and UI
  6. Add steering comment → verify persisted
  7. Update PR info → verify badge appears
  8. Archive a task → verify moved to archivedTasks table
  9. Unarchive → verify restored to tasks table
  10. Create an automation → verify in automations table
  11. Create an agent → verify in agents table
  12. Record heartbeat → verify in agentHeartbeats table
  13. View agent logs → verify agent.log file still readable
• Test backward compatibility:
  1. Create project with old file-based version (checkout old commit or manually create file structure)
  2. Create some tasks, agents, automations
  3. Run new version → verify auto-migration triggers
  4. Verify .kb/tasks.bak/ exists with old data
  5. Verify all data correctly migrated (IDs, columns, timestamps, JSON arrays)
  6. Verify tasks work correctly after migration
  7. Restart app → verify doesn't re-migrate (detects already migrated)

Step 9: Documentation & Delivery

• Update AGENTS.md with new section:
  • "SQLite Storage Architecture" — explain hybrid approach (DB for metadata, files for blobs)
  • "Migration Notes" — explain auto-migration, backup files, recovery
  • Schema documentation — list tables and their purposes
• Create changeset file:

  cat > .changeset/sqlite-migration.md << 'EOF'
  ---
  "@dustinbyrne/kb": minor
  ---
  
  Migrate storage from file-based JSON to SQLite (hybrid with files for blobs)
  - Improved performance for large task counts
  - Better concurrent access support (WAL mode)
  - Seamless auto-migration from legacy data
  - Zero breaking changes to store APIs
  EOF
  

• Create follow-up tasks via task_create for any out-of-scope findings:
  • Performance optimization: Add additional indexes if slow queries identified
  • Schema versioning: Upgrade system for future schema changes
  • Global settings: Consider if ~/.pi/kb/settings.json should also migrate

Documentation Requirements

Must Update:

• AGENTS.md — Add "SQLite Storage Architecture" section explaining:
  • Database location at .kb/kb.db
  • Hybrid architecture (SQLite for metadata, files for blobs)
  • Auto-migration behavior and backup files
  • WAL mode for concurrency

Check If Affected:

• packages/core/README.md — Update if it describes storage mechanisms
• packages/dashboard/README.md — Update if it mentions file watching or storage

Completion Criteria

• All steps complete
• All tests passing (pnpm test exits 0)
• Build passes (pnpm build exits 0)
• Auto-migration works seamlessly from file-based storage
• File blobs (PROMPT.md, agent.log, attachments) still work correctly
• Dashboard real-time updates work via store events (SSE unchanged)
• Steering comments, PR info, Issue info all persist correctly
• getRootDir() method preserved on TaskStore
• Documentation updated
• Changeset file created

Git Commit Convention

Commits at step boundaries. All commits include the task ID:

• Step completion: feat(KB-310): complete Step N — description
• Bug fixes: fix(KB-310): description
• Tests: test(KB-310): description
• Documentation: docs(KB-310): description

Do NOT

• Modify GlobalSettingsStore — global settings stay in ~/.pi/kb/settings.json
• Change any public method signatures on stores — preserve backward compatibility
• Use external SQLite dependencies — use only Node.js built-in node:sqlite
• Remove legacy file-based code until migration is verified working
• Delete .kb/*.bak files — they are user data backups
• Skip WAL mode — concurrent access requires it
• Skip transaction handling — data integrity is critical
• Use snake_case column names — use camelCase to match TypeScript
• Remove or rename getRootDir() method — dashboard depends on it