FN-8305: add durable PostgreSQL symbol locks
Introduce durable project-scoped symbol locks backed by PostgreSQL. - Add normalized lease-based lock acquisition, renewal, release, and reconciliation APIs with audit events. - Add PostgreSQL schema migrations and self-healing reconciliation coverage. - Document the lock model and test migration and lock behavior. Files changed: AGENTS.md | 1 + docs/architecture.md | 1 + docs/storage.md | 7 + .../src/__tests__/postgres/schema-applier.test.ts | 115 +++++++++- packages/core/src/__tests__/symbol-locks.test.ts | 91 ++++++++ packages/core/src/index.ts | 17 ++ .../core/src/postgres/migrations/0000_initial.sql | 25 +++ .../src/postgres/migrations/0025_symbol_locks.sql | 63 ++++++ packages/core/src/postgres/schema-applier.ts | 18 +- packages/core/src/postgres/schema/project.ts | 29 +++ packages/core/src/store.ts | 23 ++ packages/core/src/symbol-lock-types.ts | 60 +++++ packages/core/src/task-store/symbol-locks.ts | 244 +++++++++++++++++++++ .../__tests__/symbol-lock-reconciliation.test.ts | 19 ++ packages/engine/src/self-healing.ts | 35 +++ 15 files changed, 745 insertions(+), 3 deletions(-) Fusion-Task-Id: FN-8305 Fusion-Task-Lineage: efd95c73-23e3-4359-8204-dfad374a39bc Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
@@ -280,6 +280,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
- FN-7998: executor emits `task:execution-escalation-retry` when its opt-in, single alternate model/node attempt is persisted after FN-7996 exhaustion, and `task:execution-escalation-exhausted` when that attempt also reaches the terminal park. Metadata remains ids/counts/outcomes-only (`taskId`, graph node id, target booleans, and prior retry count); no model identifiers or prose are persisted in run-audit.
|
||||
- FN-8004: `agent:heartbeat-move-skipped-soft-delete` records a heartbeat move that races a soft-deleted task without parking the durable agent. Metadata remains ids/timestamps/source only (`agentId`, optional `taskId`/`deletedAt`, `moveAttemptedAt`, optional `source`); it never stores error prose.
|
||||
- FN-8141: the executor's `fn_task_done(outcome="blocked", reason=..., blockedBy?=[...])` honest-blocked exit emits `task:execution-blocked-parked` when an executor parks a genuinely-impossible task `failed` (`error = "BLOCKED: <reason>"`) instead of laundering it to `done` by skipping steps. It bypasses the completion/verdict/bulk-completion gates (blocked is not a completion claim), leaves steps in their true statuses, preserves worktree/branch, records `blockedBy` as real `task.dependencies` edges so the task requeues behind the blocker, and does NOT hand off to review — the parked row is honored by the executor's `status === "failed"` post-loop branch and is not auto-recovered into in-review by `recoverStrandedCompletedTodoTasks` (steps are not all done/skipped and `task.error` is set). Metadata stays ids/outcomes-only (`taskId`, `blockedBy` ids, `hasReason` boolean — never the reason prose).
|
||||
- FN-8305: durable symbol-lock operations emit `symbol-lock:acquired`, `symbol-lock:acquire-conflict`, `symbol-lock:renewed`, `symbol-lock:released`, `symbol-lock:reconcile-stale`, and deduplicated `symbol-lock:reconcile-stale-no-action`. Metadata is ids/counts/outcomes-only; normalized opaque symbol keys are permitted IDs, while raw symbol prose is not.
|
||||
|
||||
|
||||
## Reference docs (deeper detail)
|
||||
|
||||
@@ -679,6 +679,7 @@ Runtime action-gate flow (v1):
|
||||
- `SelfHealingManager` (`self-healing.ts`) — auto-unpause/maintenance recovery actions
|
||||
- Batch 1 maintenance includes `reconcile-orphaned-task-dirs` (FN-6783), a paused-safe housekeeping step that calls `TaskStore.reconcileOrphanedTaskDirs()` so valid live `.fusion/tasks/{ID}/task.json` records missing from PostgreSQL become visible without waiting for process restart. The guard skips any ID already present in active, soft-deleted, archived, or tombstoned storage and emits `task:reconcile-orphaned-task-dir` only for recovered rows.
|
||||
- Batch 1 maintenance also includes `reconcile-phantom-committed-reservations` (FN-7069), which calls `TaskStore.reconcilePhantomCommittedReservations()` for committed task-ID reservations that have no live/soft-deleted/archived task row and no `.fusion/tasks/{ID}/task.json`. The sweep prunes orphaned `activityLog` rows and `agents`/cascaded `agentRuns`, preserves `runAuditEvents`, and keeps the reservation `committed` per FN-5105 so the ID is permanently reserved rather than resurrected or handed out again.
|
||||
- Startup recovery and Batch 1 both call `reconcileStaleSymbolLocks()` (FN-8305). It expires only project-scoped held locks whose lease elapsed or whose owner task is terminal/missing, preserves live owners, and does not alter task lifecycle, scheduler admission, worktrees, semaphores, or verification. The audit surface is `symbol-lock:reconcile-stale` plus a deduplicated `symbol-lock:reconcile-stale-no-action` idle signal.
|
||||
- Batch 1 maintenance now includes one `fts-maintenance` step for both search indexes. The live `tasks_fts` branch still runs `merge` every tick, `optimize` every 4th tick, and `rebuild` above `32 MiB` or `1 MiB × live task count`. The archive `archived_tasks_fts` branch is lighter because archive writes are mostly append-only: `merge` every 8th tick, `optimize` every 24th tick, and `rebuild` above `64 MiB` or `512 KiB × archived row count`. Each branch is independently guarded by `fts5Available` and emits `task:fts-maintenance` run-audit telemetry with distinct `target` values (`tasks_fts` vs `archived_tasks_fts`).
|
||||
- AI merge clean-room worktrees are created under the configured worktrees directory's hidden container, `<worktreesDir>/.ai-merge/`, as `fusion-ai-merge-fn-<id>-<random>` detached worktrees. When that container is repo-local, its relative path is added to the repo's local git exclude when possible (alongside the legacy `.fusion/ai-merge/` entry) so an in-flight clean room does not dirty the integration checkout. After `git worktree add` and before the merge/review loop, `runAiMerge` bootstraps the clean room with the shared merge dependency-sync helper: a configured `worktreeInitCommand` is authoritative and always runs, while unset settings infer `pnpm`/`npm`/`yarn`/`bun` installs from lockfiles and can skip only when the `node_modules/.fusion-install-marker` hash still matches. Failures and aborts hard-stop the AI merge before merge agents or verification run, and `merge:ai-deps-sync` records the command, skip state, and duration. Inline cleanup runs from `runAiMerge`'s clean-room `finally` for successful lands, empty/no-op finalization, concurrent-advance retries, and thrown/aborted merges. Cleanup canonicalizes the path, attempts `git worktree remove --force`, always falls back to filesystem removal, then runs `git worktree prune` so stale or partial registrations (including `git worktree add` failures) do not dangle. Cleanup emits `merge:ai-worktree-cleanup` audit events for git-remove, fs-rm, and prune phases; benign already-absent/de-registered paths are treated as idempotent success, while genuine filesystem-removal failures are logged/audited with `success: false` rather than silently swallowed.
|
||||
- Worktrees-dir sweeps that list direct children of `<worktreesDir>` (pool idle scan, orphan cleanup/reap, self-healing unregistered-orphan reap, and cap enforcement) must exclude the `.ai-merge` container by name; those one-level sweeps never inspect or recycle clean rooms beneath it. Batch 1 sweeps stale AI merge clean-room worktrees under the new `<worktreesDir>/.ai-merge/` root and still scans legacy `.fusion/ai-merge/` plus legacy `tmpdir()` locations for pre-relocation leftovers; candidates are bounded to names starting with `fusion-ai-merge-`. `runAiMerge` registers each live clean-room worktree in `activeSessionRegistry` with kind `ai-merge` as soon as the directory exists and keeps both raw and canonical paths registered for the duration of the merge, so the dedicated periodic sweep and pre-merge prune defer when either path is active (including concurrent same-task merge attempts). The default age gate is 2 hours; task-aware cleanup uses a 10-minute grace period for `done`/`archived` tasks and for genuinely missing/deleted task rows, and every removal path is clamped by the same 10-minute minimum-age floor so a freshly created worktree is never reaped. Transient `getTask` lookup failures (for example PostgreSQL availability/transaction errors) are not treated as deletion evidence; they log a warning, emit `lookup-error` only if eventually removed, and retain the conservative 2-hour gate. The sweep canonicalizes paths before checking `activeSessionRegistry`, attempts `git worktree remove --force <path>` before filesystem removal, runs `git worktree prune` after cleanup attempts, and emits `worktree:tempdir-sweep` run-audit telemetry for removal attempts and failures. Fresh directories, active-session paths, and individual removal failures are skipped/logged without aborting the maintenance cycle.
|
||||
|
||||
@@ -17,6 +17,13 @@ See the [2026-07-14 PostgreSQL runtime cutover review](./postgres-migration-revi
|
||||
- Startup/store-open allocator reconciliation bumps each active prefix sequence to `max(current nextSequence, max(tasks suffix)+1, max(archivedTasks suffix)+1, max(reservation sequence)+1)` so stale allocator rows self-heal before local task creation resumes.
|
||||
- Create-class task persistence is intentionally non-destructive: new tasks use plain `INSERT` semantics, while upserts remain update-only. If counters drift and a reserved ID still collides, the create fails and the existing PostgreSQL row / task directory stays intact. A `committed` distributed reservation is valid only with a matching durable task row/directory; failed creates burn the reservation as `aborted` instead of leaving a committed-reservation-without-task phantom.
|
||||
|
||||
## Durable symbol locks (FN-8305)
|
||||
|
||||
- `project.symbol_locks` is the project-scoped, lease-based admission seam for later mission-lineage scheduling. Its composite `(project_id, symbol_key)` identity permits only one current lock row per normalized symbol in a project; ownership records task ID plus optional mission, feature, lineage, node, and agent IDs.
|
||||
- Lock acquire is all-or-nothing over normalized keys. Held unexpired rows owned by another task return their owner as a conflict, while expired/released rows may be reclaimed. Renewal and release are owner-scoped and release is idempotent.
|
||||
- The `0000_initial.sql` baseline defines the table and indexes only. The later `0025_symbol_locks.sql` migration enables and forces RLS, creates `fusion_project_isolation`, and attaches `fusion_assign_project_id` after `0006_project_ownership.sql` creates that function/policy machinery. Both fresh full-applier and upgrade paths therefore end with the same project-isolation contract.
|
||||
- Startup and Batch 1 self-healing expire locks when their lease elapsed or the owner task is terminal/missing. They never move a task or alter scheduler, worktree, semaphore, or verification state. Run-audit events are `symbol-lock:acquired`, `symbol-lock:acquire-conflict`, `symbol-lock:renewed`, `symbol-lock:released`, `symbol-lock:reconcile-stale`, and deduplicated `symbol-lock:reconcile-stale-no-action`; metadata uses only counts/outcomes and normalized opaque keys.
|
||||
|
||||
## Soft-deleted tasks (FN-5105)
|
||||
|
||||
- User-initiated `TaskStore.deleteTask` is a **soft delete**: the task row stays in `tasks` and `deletedAt` is set.
|
||||
|
||||
@@ -65,6 +65,8 @@ import {
|
||||
PROJECT_OWNERSHIP_SCHEMA_VERSION,
|
||||
SESSION_ADVISOR_ENABLED_SCHEMA_VERSION,
|
||||
SQLITE_SCHEMA_PARITY_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
} from "../../postgres/schema-applier.js";
|
||||
import { rekeyFallbackProjectPartition } from "../../postgres/migration-stamping.js";
|
||||
import type { PluginSchemaInitHook } from "../../postgres/plugin-schema-hook.js";
|
||||
@@ -171,6 +173,11 @@ describe("schema-applier: immutable migration identities", () => {
|
||||
expect(Number(SCHEMA_BASELINE_VERSION)).toBeGreaterThanOrEqual(Number(TASK_PROPOSAL_CLAIM_VERSION));
|
||||
});
|
||||
|
||||
it("registers durable symbol locks at the next free migration version", () => {
|
||||
expect(SYMBOL_LOCKS_SCHEMA_VERSION).toBe("0025");
|
||||
expect(Number(SCHEMA_BASELINE_VERSION)).toBeGreaterThanOrEqual(Number(SYMBOL_LOCKS_SCHEMA_VERSION));
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
/*
|
||||
@@ -266,6 +273,79 @@ async function teardownDb(ctx: TestContext | null): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:SymbolLock 2026-07-30-15:15:
|
||||
The baseline declares symbol_locks but cannot attach its ownership trigger before
|
||||
0006 defines fusion_assign_project_id. Both a full fresh apply and an upgraded
|
||||
installation must therefore prove 0025 leaves the final table forced-RLS with
|
||||
its policy and trigger, including actual second-project read/write isolation.
|
||||
*/
|
||||
async function assertSymbolLocksOwnershipContract(ctx: TestContext): Promise<void> {
|
||||
const catalog = (await ctx.db.execute(sql`
|
||||
SELECT c.relrowsecurity AS rls, c.relforcerowsecurity AS forced,
|
||||
EXISTS (
|
||||
SELECT 1 FROM pg_policies
|
||||
WHERE schemaname = 'project' AND tablename = 'symbol_locks'
|
||||
AND policyname = 'fusion_project_isolation'
|
||||
) AS policy,
|
||||
EXISTS (
|
||||
SELECT 1 FROM pg_trigger
|
||||
WHERE tgrelid = 'project.symbol_locks'::regclass
|
||||
AND tgname = 'fusion_assign_project_id' AND NOT tgisinternal
|
||||
) AS trigger
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'project' AND c.relname = 'symbol_locks'
|
||||
`)) as unknown as Array<{ rls: boolean; forced: boolean; policy: boolean; trigger: boolean }>;
|
||||
expect(catalog).toEqual([{ rls: true, forced: true, policy: true, trigger: true }]);
|
||||
|
||||
const role = `fusion_symbol_lock_isolation_${process.pid}_${Math.random().toString(36).slice(2, 8)}`;
|
||||
await ctx.db.execute(sql.raw(`
|
||||
CREATE ROLE ${role} NOLOGIN;
|
||||
GRANT USAGE ON SCHEMA project TO ${role};
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON project.symbol_locks TO ${role};
|
||||
`));
|
||||
try {
|
||||
for (const projectId of ["project-a", "project-b"]) {
|
||||
await ctx.db.transaction(async (tx) => {
|
||||
await tx.execute(sql.raw(`SET LOCAL ROLE ${role}`));
|
||||
await tx.execute(sql`SELECT set_config('fusion.project_id', ${projectId}, true)`);
|
||||
await tx.execute(sql`
|
||||
INSERT INTO project.symbol_locks(
|
||||
symbol_key, owner_task_id, status, acquired_at, renewed_at, expires_at, created_at, updated_at
|
||||
) VALUES (
|
||||
'pkg/shared.ts#export', ${`FN-${projectId}`}, 'held',
|
||||
'2026-07-30T15:15:00.000Z', '2026-07-30T15:15:00.000Z',
|
||||
'2026-07-30T16:15:00.000Z', '2026-07-30T15:15:00.000Z', '2026-07-30T15:15:00.000Z'
|
||||
)
|
||||
`);
|
||||
});
|
||||
}
|
||||
await ctx.db.transaction(async (tx) => {
|
||||
await tx.execute(sql.raw(`SET LOCAL ROLE ${role}`));
|
||||
await tx.execute(sql`SELECT set_config('fusion.project_id', 'project-b', true)`);
|
||||
const visible = (await tx.execute(sql`
|
||||
SELECT project_id, owner_task_id FROM project.symbol_locks ORDER BY project_id
|
||||
`)) as unknown as Array<{ project_id: string; owner_task_id: string }>;
|
||||
expect(visible).toEqual([{ project_id: "project-b", owner_task_id: "FN-project-b" }]);
|
||||
const stolen = (await tx.execute(sql`
|
||||
UPDATE project.symbol_locks SET owner_task_id = 'stolen'
|
||||
WHERE project_id = 'project-a' RETURNING owner_task_id
|
||||
`)) as unknown as Array<{ owner_task_id: string }>;
|
||||
expect(stolen).toEqual([]);
|
||||
});
|
||||
} finally {
|
||||
await ctx.db.execute(sql.raw(`DROP OWNED BY ${role}; DROP ROLE ${role};`));
|
||||
}
|
||||
const rows = (await ctx.db.execute(sql`
|
||||
SELECT project_id, owner_task_id FROM project.symbol_locks ORDER BY project_id
|
||||
`)) as unknown as Array<{ project_id: string; owner_task_id: string }>;
|
||||
expect(rows).toEqual([
|
||||
{ project_id: "project-a", owner_task_id: "FN-project-a" },
|
||||
{ project_id: "project-b", owner_task_id: "FN-project-b" },
|
||||
]);
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:PostgresSchema 2026-06-24-06:30:
|
||||
* Complete enumeration of every CREATE INDEX name from the SQLite final
|
||||
@@ -539,9 +619,10 @@ pgDescribe("schema-applier: VAL-SCHEMA-001 final-schema parity (table counts)",
|
||||
// Project: 87 typed core tables + 2 lossless legacy preservation tables
|
||||
// + 1 import_translation_cache (FNXC:GitHubImportTranslate 2026-07-15-09:30)
|
||||
// + 1 configuration_revisions (FNXC:ConfigVersioning 2026-07-18-14:00)
|
||||
// + 2 ideation_sessions/ideation_candidates (FNXC:Ideation 2026-07-18-13:25 / FN-8295).
|
||||
// + 2 ideation_sessions/ideation_candidates (FNXC:Ideation 2026-07-18-13:25 / FN-8295)
|
||||
// + 1 task_verification_requests + 1 durable symbol_locks table (FN-8305).
|
||||
// Plugin tables are added separately by the hook.
|
||||
expect(bySchema.project).toBe(93);
|
||||
expect(bySchema.project).toBe(95);
|
||||
expect(bySchema.central).toBe(18);
|
||||
expect(bySchema.archive).toBe(1);
|
||||
});
|
||||
@@ -694,6 +775,26 @@ pgDescribe("schema-applier: VAL-SCHEMA-001 final-schema parity (table counts)",
|
||||
expect(rlsGaps).toEqual([]);
|
||||
});
|
||||
|
||||
it("applies symbol-lock RLS, ownership policy, and trigger after a full fresh sequence", async () => {
|
||||
ctx = await setupFreshDb();
|
||||
await applySchemaBaseline(ctx.db, { pluginHooks: [] });
|
||||
await assertSymbolLocksOwnershipContract(ctx);
|
||||
});
|
||||
|
||||
it("repairs an upgraded installation missing the 0025 symbol-lock migration", async () => {
|
||||
ctx = await setupFreshDb();
|
||||
await applySchemaBaseline(ctx.db, { pluginHooks: [] });
|
||||
// Preserve the 0000 marker while removing the later table/marker: this is
|
||||
// the pre-0025 upgraded-install shape, where replay must not rely on baseline SQL.
|
||||
await ctx.db.execute(sql.raw(`
|
||||
DROP TABLE project.symbol_locks;
|
||||
DELETE FROM public.fusion_schema_migrations WHERE version = '0025';
|
||||
`));
|
||||
expect((await applySchemaBaseline(ctx.db, { pluginHooks: [] })).applied).toBe(true);
|
||||
expect(await getAppliedMigrations(ctx.db)).toContain(SYMBOL_LOCKS_SCHEMA_VERSION);
|
||||
await assertSymbolLocksOwnershipContract(ctx);
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:ProjectDataIsolation 2026-07-14-12:10:
|
||||
Exercise the user-visible invariant through a non-superuser role: agents created in one project are invisible and immutable from another project even when application SQL omits project predicates.
|
||||
@@ -1169,6 +1270,8 @@ pgDescribe("schema-applier: automation project-isolation upgrade", () => {
|
||||
CONFIGURATION_REVISIONS_VERSION,
|
||||
IDEATION_SCHEMA_VERSION,
|
||||
RESEARCH_FEATURE_PROVENANCE_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
]);
|
||||
expect((await applySchemaBaseline(ctx.db, { pluginHooks: [] })).applied).toBe(false);
|
||||
});
|
||||
@@ -1218,6 +1321,8 @@ pgDescribe("schema-applier: automation project-isolation upgrade", () => {
|
||||
CONFIGURATION_REVISIONS_VERSION,
|
||||
IDEATION_SCHEMA_VERSION,
|
||||
RESEARCH_FEATURE_PROVENANCE_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -1400,6 +1505,8 @@ pgDescribe("schema-applier: automation project-isolation upgrade", () => {
|
||||
CONFIGURATION_REVISIONS_VERSION,
|
||||
IDEATION_SCHEMA_VERSION,
|
||||
RESEARCH_FEATURE_PROVENANCE_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -1463,6 +1570,8 @@ pgDescribe("schema-applier: automation project-isolation upgrade", () => {
|
||||
CONFIGURATION_REVISIONS_VERSION,
|
||||
IDEATION_SCHEMA_VERSION,
|
||||
RESEARCH_FEATURE_PROVENANCE_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
]);
|
||||
});
|
||||
|
||||
@@ -1526,6 +1635,8 @@ pgDescribe("schema-applier: automation project-isolation upgrade", () => {
|
||||
CONFIGURATION_REVISIONS_VERSION,
|
||||
IDEATION_SCHEMA_VERSION,
|
||||
RESEARCH_FEATURE_PROVENANCE_VERSION,
|
||||
TASK_VERIFICATION_REQUEST_VERSION,
|
||||
SYMBOL_LOCKS_SCHEMA_VERSION,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
91
packages/core/src/__tests__/symbol-locks.test.ts
Normal file
91
packages/core/src/__tests__/symbol-locks.test.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import * as schema from "../postgres/schema/index.js";
|
||||
import { TaskStore } from "../store.js";
|
||||
import { extractSymbolLockIdentity, normalizeSymbolLockKey, symbolLocksConflict } from "../task-store/symbol-locks.js";
|
||||
import { createSharedPgTaskStoreTestHarness, pgDescribe, type SharedPgTaskStoreHarness } from "../__test-utils__/pg-test-harness.js";
|
||||
|
||||
function storeForProject(harness: SharedPgTaskStoreHarness, projectId: string): TaskStore {
|
||||
return new TaskStore(harness.rootDir(), undefined, {
|
||||
asyncLayer: { ...harness.layer(), projectId },
|
||||
});
|
||||
}
|
||||
|
||||
describe("symbol lock normalization", () => {
|
||||
it("normalizes equivalent symbol references deterministically", () => {
|
||||
expect(normalizeSymbolLockKey(" Pkg\\File.ts # Exported.member ")).toBe("pkg/file.ts#exported.member");
|
||||
expect(extractSymbolLockIdentity("project-a", "pkg/file.ts#Exported.member")).toEqual({ projectId: "project-a", normalizedSymbol: "pkg/file.ts#exported.member", symbolKey: "pkg/file.ts#exported.member" });
|
||||
expect(symbolLocksConflict("pkg\\file.ts#Thing", " pkg/file.ts # thing ")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
pgDescribe("TaskStore durable symbol locks", () => {
|
||||
const h: SharedPgTaskStoreHarness = createSharedPgTaskStoreTestHarness({ prefix: "fusion_symbol_lock" });
|
||||
beforeAll(h.beforeAll); afterAll(h.afterAll); beforeEach(h.beforeEach); afterEach(h.afterEach);
|
||||
|
||||
it("acquires all symbols atomically and returns the contending owner without partial acquisition", async () => {
|
||||
const store = h.store();
|
||||
const first = await store.acquireSymbolLocks(["pkg/a.ts#A"], { ownerTaskId: "FN-owner", agentId: "agent-a" }, 60_000);
|
||||
expect(first.acquired).toBe(true);
|
||||
const second = await store.acquireSymbolLocks(["pkg/a.ts#A", "pkg/b.ts#B"], { ownerTaskId: "FN-other" }, 60_000);
|
||||
expect(second).toMatchObject({ acquired: false, locks: [], conflicts: [{ ownerTaskId: "FN-owner", symbolKey: "pkg/a.ts#a" }] });
|
||||
expect(await store.inspectSymbolLockConflicts(["pkg/b.ts#B"])).toEqual([]);
|
||||
});
|
||||
|
||||
/*
|
||||
FNXC:SymbolLock 2026-07-30-15:10:
|
||||
Scheduler admission must partition every symbol-lock operation by project.
|
||||
Two projects may hold the same canonical key, while inspect, reconciliation,
|
||||
and their structured audit records must never make either project contend with
|
||||
or expose the other's lock.
|
||||
*/
|
||||
it("isolates acquire, inspect, reconciliation, and audit metadata by project", async () => {
|
||||
const projectA = storeForProject(h, "project-a");
|
||||
const projectB = storeForProject(h, "project-b");
|
||||
|
||||
expect((await projectA.acquireSymbolLocks(["pkg/shared.ts#Export"], { ownerTaskId: "FN-project-a" }, 60_000)).acquired).toBe(true);
|
||||
expect((await projectB.acquireSymbolLocks(["pkg/shared.ts#Export"], { ownerTaskId: "FN-project-b" }, 60_000)).acquired).toBe(true);
|
||||
expect(await projectA.inspectSymbolLockConflicts(["pkg/shared.ts#Export"])).toMatchObject([
|
||||
{ ownerTaskId: "FN-project-a", symbolKey: "pkg/shared.ts#export" },
|
||||
]);
|
||||
expect(await projectB.inspectSymbolLockConflicts(["pkg/shared.ts#Export"])).toMatchObject([
|
||||
{ ownerTaskId: "FN-project-b", symbolKey: "pkg/shared.ts#export" },
|
||||
]);
|
||||
|
||||
// The missing B task makes only B's lock stale; A remains invisible to B's sweep.
|
||||
expect((await projectB.reconcileStaleSymbolLocks()).reconciled).toEqual(["pkg/shared.ts#export"]);
|
||||
expect(await projectA.inspectSymbolLockConflicts(["pkg/shared.ts#Export"])).toMatchObject([
|
||||
{ ownerTaskId: "FN-project-a", symbolKey: "pkg/shared.ts#export" },
|
||||
]);
|
||||
|
||||
const audits = await h.adminDb().select({ taskId: schema.project.runAuditEvents.taskId, metadata: schema.project.runAuditEvents.metadata })
|
||||
.from(schema.project.runAuditEvents)
|
||||
.where(eq(schema.project.runAuditEvents.mutationType, "symbol-lock:acquired"));
|
||||
expect(audits).toEqual(expect.arrayContaining([
|
||||
{ taskId: "FN-project-a", metadata: { count: 1, symbolKeys: ["pkg/shared.ts#export"], outcome: "acquired" } },
|
||||
{ taskId: "FN-project-b", metadata: { count: 1, symbolKeys: ["pkg/shared.ts#export"], outcome: "acquired" } },
|
||||
]));
|
||||
});
|
||||
|
||||
it("renews and releases only the caller's unexpired locks", async () => {
|
||||
const store = h.store();
|
||||
await store.acquireSymbolLocks(["pkg/a.ts#A"], { ownerTaskId: "FN-owner" }, 60_000);
|
||||
expect(await store.renewSymbolLocks(["pkg/a.ts#A"], "FN-other", 60_000)).toEqual({ renewed: [], lost: ["pkg/a.ts#a"] });
|
||||
expect((await store.releaseSymbolLocks(["pkg/a.ts#A"], "FN-other")).released).toEqual([]);
|
||||
expect((await store.releaseSymbolLocks(["pkg/a.ts#A"], "FN-owner")).released).toEqual(["pkg/a.ts#a"]);
|
||||
expect((await store.releaseSymbolLocks(["pkg/a.ts#A"], "FN-owner")).released).toEqual([]);
|
||||
});
|
||||
|
||||
it("allows expired locks to be acquired and reconciles terminal owners", async () => {
|
||||
const store = h.store(); const layer = h.layer(); const projectId = layer.projectId?.trim() || "__legacy_unscoped__";
|
||||
await store.acquireSymbolLocks(["pkg/expired.ts#A"], { ownerTaskId: "FN-dead" }, 60_000);
|
||||
await layer.db.update(schema.project.symbolLocks).set({ expiresAt: new Date(Date.now() - 1_000).toISOString() }).where(and(eq(schema.project.symbolLocks.projectId, projectId), eq(schema.project.symbolLocks.symbolKey, "pkg/expired.ts#a")));
|
||||
expect((await store.acquireSymbolLocks(["pkg/expired.ts#A"], { ownerTaskId: "FN-new" }, 60_000)).acquired).toBe(true);
|
||||
const owner = await store.createTask({ description: "terminal symbol lock owner" });
|
||||
await store.acquireSymbolLocks(["pkg/terminal.ts#A"], { ownerTaskId: owner.id }, 60_000);
|
||||
await layer.db.update(schema.project.tasks).set({ column: "done" }).where(and(
|
||||
eq(schema.project.tasks.projectId, projectId), eq(schema.project.tasks.id, owner.id),
|
||||
));
|
||||
expect((await store.reconcileStaleSymbolLocks()).reconciled).toContain("pkg/terminal.ts#a");
|
||||
});
|
||||
});
|
||||
File diff suppressed because one or more lines are too long
@@ -243,6 +243,31 @@ CREATE INDEX IF NOT EXISTS "idxDistributedTaskIdReservationsPrefixStatus"
|
||||
CREATE INDEX IF NOT EXISTS "idxDistributedTaskIdReservationsExpiry"
|
||||
ON project.distributed_task_id_reservations(status, expires_at);
|
||||
|
||||
-- FNXC:SymbolLock 2026-07-30-14:10: baseline creates the table only; 0025
|
||||
-- applies RLS after 0006 defines the ownership trigger and policy machinery.
|
||||
CREATE TABLE IF NOT EXISTS project.symbol_locks (
|
||||
project_id text NOT NULL DEFAULT current_setting('fusion.project_id', true),
|
||||
symbol_key text NOT NULL,
|
||||
owner_task_id text NOT NULL,
|
||||
mission_id text,
|
||||
feature_id text,
|
||||
lineage_id text,
|
||||
node_id text,
|
||||
agent_id text,
|
||||
status text NOT NULL,
|
||||
acquired_at text NOT NULL,
|
||||
renewed_at text NOT NULL,
|
||||
expires_at text NOT NULL,
|
||||
created_at text NOT NULL,
|
||||
updated_at text NOT NULL,
|
||||
PRIMARY KEY (project_id, symbol_key),
|
||||
CONSTRAINT symbol_locks_status_check CHECK (status IN ('held', 'released', 'expired'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idxSymbolLocksOwner"
|
||||
ON project.symbol_locks(project_id, owner_task_id);
|
||||
CREATE INDEX IF NOT EXISTS "idxSymbolLocksExpiry"
|
||||
ON project.symbol_locks(status, expires_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS project.workflow_steps (
|
||||
id text PRIMARY KEY,
|
||||
template_id text,
|
||||
|
||||
63
packages/core/src/postgres/migrations/0025_symbol_locks.sql
Normal file
63
packages/core/src/postgres/migrations/0025_symbol_locks.sql
Normal file
@@ -0,0 +1,63 @@
|
||||
-- FNXC:SymbolLock 2026-07-30-14:10:
|
||||
-- Durable mission-lineage admission locks require an upgraded-install table and
|
||||
-- the complete project ownership contract. The baseline intentionally omits
|
||||
-- this block because 0006 creates the trigger function and policy machinery.
|
||||
CREATE TABLE IF NOT EXISTS project.symbol_locks (
|
||||
project_id text NOT NULL DEFAULT current_setting('fusion.project_id', true),
|
||||
symbol_key text NOT NULL,
|
||||
owner_task_id text NOT NULL,
|
||||
mission_id text,
|
||||
feature_id text,
|
||||
lineage_id text,
|
||||
node_id text,
|
||||
agent_id text,
|
||||
status text NOT NULL,
|
||||
acquired_at text NOT NULL,
|
||||
renewed_at text NOT NULL,
|
||||
expires_at text NOT NULL,
|
||||
created_at text NOT NULL,
|
||||
updated_at text NOT NULL,
|
||||
PRIMARY KEY (project_id, symbol_key),
|
||||
CONSTRAINT symbol_locks_status_check CHECK (status IN ('held', 'released', 'expired'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS "idxSymbolLocksOwner"
|
||||
ON project.symbol_locks(project_id, owner_task_id);
|
||||
CREATE INDEX IF NOT EXISTS "idxSymbolLocksExpiry"
|
||||
ON project.symbol_locks(status, expires_at);
|
||||
|
||||
ALTER TABLE project.symbol_locks ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE project.symbol_locks FORCE ROW LEVEL SECURITY;
|
||||
DROP POLICY IF EXISTS fusion_project_isolation ON project.symbol_locks;
|
||||
CREATE POLICY fusion_project_isolation ON project.symbol_locks
|
||||
USING (
|
||||
current_setting('fusion.project_bypass', true) = 'on'
|
||||
OR project_id = current_setting('fusion.project_id', true)
|
||||
)
|
||||
WITH CHECK (
|
||||
current_setting('fusion.project_bypass', true) = 'on'
|
||||
OR project_id = current_setting('fusion.project_id', true)
|
||||
);
|
||||
DROP TRIGGER IF EXISTS fusion_assign_project_id ON project.symbol_locks;
|
||||
CREATE TRIGGER fusion_assign_project_id
|
||||
BEFORE INSERT OR UPDATE OF project_id ON project.symbol_locks
|
||||
FOR EACH ROW EXECUTE FUNCTION project.fusion_assign_project_id();
|
||||
|
||||
-- FNXC:TaskVerificationRequest 2026-07-30-14:30: 0024 was already a
|
||||
-- published migration when its post-0006 ownership omission was discovered;
|
||||
-- repair existing 0024 installations here instead of mutating its identity.
|
||||
ALTER TABLE project.task_verification_requests ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE project.task_verification_requests FORCE ROW LEVEL SECURITY;
|
||||
DROP POLICY IF EXISTS fusion_project_isolation ON project.task_verification_requests;
|
||||
CREATE POLICY fusion_project_isolation ON project.task_verification_requests
|
||||
USING (
|
||||
current_setting('fusion.project_bypass', true) = 'on'
|
||||
OR project_id = current_setting('fusion.project_id', true)
|
||||
)
|
||||
WITH CHECK (
|
||||
current_setting('fusion.project_bypass', true) = 'on'
|
||||
OR project_id = current_setting('fusion.project_id', true)
|
||||
);
|
||||
DROP TRIGGER IF EXISTS fusion_assign_project_id ON project.task_verification_requests;
|
||||
CREATE TRIGGER fusion_assign_project_id
|
||||
BEFORE INSERT OR UPDATE OF project_id ON project.task_verification_requests
|
||||
FOR EACH ROW EXECUTE FUNCTION project.fusion_assign_project_id();
|
||||
@@ -33,7 +33,7 @@ import { acquireSchemaMutationLocks } from "./advisory-locks.js";
|
||||
FNXC:GitHubImportTranslate 2026-07-17-23:48:
|
||||
Advances to 0019 for the import-translation legacy-partition backfill. Per-migration identities above stay fixed; only this latest-version marker moves.
|
||||
*/
|
||||
export const SCHEMA_BASELINE_VERSION = "0024";
|
||||
export const SCHEMA_BASELINE_VERSION = "0025";
|
||||
const INITIAL_SCHEMA_VERSION = "0000";
|
||||
const AUTOMATION_ISOLATION_SCHEMA_VERSION = "0001";
|
||||
const ANALYTICS_ISOLATION_SCHEMA_VERSION = "0002";
|
||||
@@ -114,6 +114,8 @@ export const IDEATION_SCHEMA_VERSION = "0022";
|
||||
export const RESEARCH_FEATURE_PROVENANCE_VERSION = "0023";
|
||||
/** FNXC:TaskVerificationRequest 2026-07-30-00:00: upgrades need the project-scoped chat-to-executor verification queue. */
|
||||
export const TASK_VERIFICATION_REQUEST_VERSION = "0024";
|
||||
/** FNXC:SymbolLock 2026-07-30-14:10: upgraded projects need the durable lock table and RLS contract before scheduler admission can use it. */
|
||||
export const SYMBOL_LOCKS_SCHEMA_VERSION = "0025";
|
||||
|
||||
/** Bookkeeping table for the fresh Drizzle migration history. */
|
||||
export const MIGRATION_BOOKKEEPING_TABLE = "fusion_schema_migrations";
|
||||
@@ -225,6 +227,7 @@ const CONFIGURATION_REVISIONS_MIGRATION_PATH = join(MIGRATIONS_DIR, "0021_config
|
||||
const IDEATION_MIGRATION_PATH = join(MIGRATIONS_DIR, "0022_ideation.sql");
|
||||
const RESEARCH_FEATURE_PROVENANCE_MIGRATION_PATH = join(MIGRATIONS_DIR, "0023_research_feature_provenance.sql");
|
||||
const TASK_VERIFICATION_REQUEST_MIGRATION_PATH = join(MIGRATIONS_DIR, "0024_task_verification_request.sql");
|
||||
const SYMBOL_LOCKS_MIGRATION_PATH = join(MIGRATIONS_DIR, "0025_symbol_locks.sql");
|
||||
|
||||
/**
|
||||
* Ensure the migration bookkeeping table exists. Lives in the public schema so
|
||||
@@ -318,6 +321,7 @@ export async function applySchemaBaseline(
|
||||
const ideationAlreadyApplied = applied.includes(IDEATION_SCHEMA_VERSION);
|
||||
const researchFeatureProvenanceAlreadyApplied = applied.includes(RESEARCH_FEATURE_PROVENANCE_VERSION);
|
||||
const taskVerificationRequestAlreadyApplied = applied.includes(TASK_VERIFICATION_REQUEST_VERSION);
|
||||
const symbolLocksAlreadyApplied = applied.includes(SYMBOL_LOCKS_SCHEMA_VERSION);
|
||||
let schemaChanged = false;
|
||||
|
||||
if (!baselineAlreadyApplied) {
|
||||
@@ -695,6 +699,18 @@ export async function applySchemaBaseline(
|
||||
schemaChanged = true;
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:SymbolLock 2026-07-30-14:10:
|
||||
Migration files are manually registered. Apply 0025 independently so both
|
||||
fresh and upgraded installations gain forced RLS after 0006 owns its setup.
|
||||
*/
|
||||
if (!symbolLocksAlreadyApplied) {
|
||||
const migrationSql = await readFile(SYMBOL_LOCKS_MIGRATION_PATH, "utf8");
|
||||
await tx.execute(sql.raw(migrationSql));
|
||||
await tx.execute(sql`INSERT INTO public.${sql.identifier(MIGRATION_BOOKKEEPING_TABLE)} (version) VALUES (${SYMBOL_LOCKS_SCHEMA_VERSION}) ON CONFLICT (version) DO NOTHING`);
|
||||
schemaChanged = true;
|
||||
}
|
||||
|
||||
if (!importTranslationCacheLegacyPartitionBackfillAlreadyApplied) {
|
||||
const migrationSql = await readFile(IMPORT_TRANSLATION_CACHE_LEGACY_PARTITION_BACKFILL_MIGRATION_PATH, "utf8");
|
||||
await tx.execute(sql.raw(migrationSql));
|
||||
|
||||
@@ -521,6 +521,35 @@ export const distributedTaskIdReservations = projectSchema.table("distributed_ta
|
||||
index("idxDistributedTaskIdReservationsExpiry").on(t.status, t.expiresAt),
|
||||
]);
|
||||
|
||||
// ── Durable symbol locks ─────────────────────────────────────────────
|
||||
/*
|
||||
FNXC:SymbolLock 2026-07-30-14:10:
|
||||
Mission-lineage admission needs one project-scoped row per canonical symbol.
|
||||
A composite primary key retains released/expired ownership history while the
|
||||
atomic store seam may reclaim those rows as held without cross-project conflict.
|
||||
*/
|
||||
export const symbolLocks = projectSchema.table("symbol_locks", {
|
||||
projectId: text("project_id").notNull().default(sql`current_setting('fusion.project_id', true)`),
|
||||
symbolKey: text("symbol_key").notNull(),
|
||||
ownerTaskId: text("owner_task_id").notNull(),
|
||||
missionId: text("mission_id"),
|
||||
featureId: text("feature_id"),
|
||||
lineageId: text("lineage_id"),
|
||||
nodeId: text("node_id"),
|
||||
agentId: text("agent_id"),
|
||||
status: text("status").notNull(),
|
||||
acquiredAt: text("acquired_at").notNull(),
|
||||
renewedAt: text("renewed_at").notNull(),
|
||||
expiresAt: text("expires_at").notNull(),
|
||||
createdAt: text("created_at").notNull(),
|
||||
updatedAt: text("updated_at").notNull(),
|
||||
}, (t) => [
|
||||
primaryKey({ columns: [t.projectId, t.symbolKey] }),
|
||||
check("symbol_locks_status_check", sql`${t.status} IN ('held', 'released', 'expired')`),
|
||||
index("idxSymbolLocksOwner").on(t.projectId, t.ownerTaskId),
|
||||
index("idxSymbolLocksExpiry").on(t.status, t.expiresAt),
|
||||
]);
|
||||
|
||||
// ── Workflow step definitions ────────────────────────────────────────
|
||||
export const workflowSteps = projectSchema.table("workflow_steps", {
|
||||
id: text("id").primaryKey(),
|
||||
|
||||
@@ -113,6 +113,8 @@ import { saveWorkflowRunBranchImpl, clearNearDuplicateReferencesToImpl, selectNe
|
||||
import { taskToArchiveEntryImpl, deleteTaskBackendImpl, archiveTaskBackendImpl, unarchiveTaskImpl, restoreFromArchiveImpl, listArchivedTasksImpl } from "./task-store/archive-lifecycle-2.js";
|
||||
import { pruneOperationalLogsAsync, pruneAgentLogFilesAsync, type OperationalLogPruneResult } from "./task-store/async-maintenance.js";
|
||||
import { reconcilePhantomCommittedReservationsAsync } from "./task-store/async-phantom-reservations.js";
|
||||
import { acquireSymbolLocksAsync, inspectSymbolLockConflictsAsync, reconcileStaleSymbolLocksAsync, releaseSymbolLocksAsync, renewSymbolLocksAsync } from "./task-store/symbol-locks.js";
|
||||
import type { AcquireSymbolLocksResult, ReconcileStaleSymbolLocksResult, ReleaseSymbolLocksResult, RenewSymbolLocksResult, SymbolLockConflict, SymbolLockOwner } from "./symbol-lock-types.js";
|
||||
import { queryRunAuditEvents } from "./task-store/async-audit.js";
|
||||
import { isValidMergeRequestTransitionImpl, enqueueMergeQueueSyncInternalImpl, releaseMergeQueueLeaseImpl, collectMergeDetailsImpl, applyPrMergedTransitionImpl } from "./task-store/merge-queue-ops-2.js";
|
||||
import { upsertWorkflowWorkItemImpl, transitionWorkflowWorkItemImpl, acquireWorkflowWorkItemLeaseImpl } from "./task-store/workflow-workitems-ops-2.js";
|
||||
@@ -704,6 +706,27 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
return reconcileOrphanedTaskDirsImpl(this, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:SymbolLock 2026-07-30-14:10:
|
||||
* The TaskStore owns project binding, so callers cannot accidentally inspect
|
||||
* or mutate an identically named symbol in another project's partition.
|
||||
*/
|
||||
async acquireSymbolLocks(symbols: readonly string[], owner: SymbolLockOwner, leaseMs: number): Promise<AcquireSymbolLocksResult> {
|
||||
return acquireSymbolLocksAsync(this, symbols, owner, leaseMs);
|
||||
}
|
||||
async renewSymbolLocks(symbols: readonly string[], ownerTaskId: string, leaseMs: number): Promise<RenewSymbolLocksResult> {
|
||||
return renewSymbolLocksAsync(this, symbols, ownerTaskId, leaseMs);
|
||||
}
|
||||
async releaseSymbolLocks(symbols: readonly string[], ownerTaskId: string): Promise<ReleaseSymbolLocksResult> {
|
||||
return releaseSymbolLocksAsync(this, symbols, ownerTaskId);
|
||||
}
|
||||
async inspectSymbolLockConflicts(symbols: readonly string[]): Promise<SymbolLockConflict[]> {
|
||||
return inspectSymbolLockConflictsAsync(this, symbols);
|
||||
}
|
||||
async reconcileStaleSymbolLocks(): Promise<ReconcileStaleSymbolLocksResult> {
|
||||
return reconcileStaleSymbolLocksAsync(this);
|
||||
}
|
||||
|
||||
/** Reconcile committed reservations whose task and archive representations are absent. */
|
||||
async reconcilePhantomCommittedReservations(): Promise<{
|
||||
reconciled: string[];
|
||||
|
||||
60
packages/core/src/symbol-lock-types.ts
Normal file
60
packages/core/src/symbol-lock-types.ts
Normal file
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* FNXC:SymbolLock 2026-07-30-14:00:
|
||||
* Mission-lineage scheduler admission needs a durable, project-scoped seam for
|
||||
* protecting code symbols before later scheduling phases consume it. These
|
||||
* opaque normalized keys deliberately avoid storing raw planning prose.
|
||||
*/
|
||||
export type SymbolLockStatus = "held" | "released" | "expired";
|
||||
|
||||
export interface SymbolLockIdentity {
|
||||
projectId: string;
|
||||
symbolKey: string;
|
||||
normalizedSymbol: string;
|
||||
}
|
||||
|
||||
export interface SymbolLockOwner {
|
||||
ownerTaskId: string;
|
||||
missionId?: string;
|
||||
featureId?: string;
|
||||
lineageId?: string;
|
||||
nodeId?: string;
|
||||
agentId?: string;
|
||||
}
|
||||
|
||||
export interface SymbolLockLease {
|
||||
status: SymbolLockStatus;
|
||||
acquiredAt: string;
|
||||
renewedAt: string;
|
||||
expiresAt: string;
|
||||
}
|
||||
|
||||
export interface SymbolLock extends SymbolLockIdentity, SymbolLockOwner, SymbolLockLease {}
|
||||
|
||||
export interface SymbolLockConflict {
|
||||
symbolKey: string;
|
||||
ownerTaskId: string;
|
||||
missionId?: string;
|
||||
featureId?: string;
|
||||
lineageId?: string;
|
||||
nodeId?: string;
|
||||
agentId?: string;
|
||||
expiresAt: string;
|
||||
}
|
||||
|
||||
export type AcquireSymbolLocksResult =
|
||||
| { acquired: true; locks: SymbolLock[]; conflicts: [] }
|
||||
| { acquired: false; locks: []; conflicts: SymbolLockConflict[] };
|
||||
|
||||
export interface RenewSymbolLocksResult {
|
||||
renewed: string[];
|
||||
lost: string[];
|
||||
}
|
||||
|
||||
export interface ReleaseSymbolLocksResult {
|
||||
released: string[];
|
||||
}
|
||||
|
||||
export interface ReconcileStaleSymbolLocksResult {
|
||||
reconciled: string[];
|
||||
skipped: string[];
|
||||
}
|
||||
244
packages/core/src/task-store/symbol-locks.ts
Normal file
244
packages/core/src/task-store/symbol-locks.ts
Normal file
@@ -0,0 +1,244 @@
|
||||
import { and, eq, gt, inArray, sql } from "drizzle-orm";
|
||||
import * as schema from "../postgres/schema/index.js";
|
||||
import { projectOwnershipPartition, recordRunAuditEventWithinTransaction } from "../postgres/data-layer.js";
|
||||
import type { DbTransaction } from "../postgres/data-layer.js";
|
||||
import type { TaskStore } from "../store.js";
|
||||
import type {
|
||||
AcquireSymbolLocksResult,
|
||||
ReconcileStaleSymbolLocksResult,
|
||||
ReleaseSymbolLocksResult,
|
||||
RenewSymbolLocksResult,
|
||||
SymbolLock,
|
||||
SymbolLockConflict,
|
||||
SymbolLockIdentity,
|
||||
SymbolLockOwner,
|
||||
} from "../symbol-lock-types.js";
|
||||
|
||||
/**
|
||||
* FNXC:SymbolLock 2026-07-30-14:00:
|
||||
* Symbol references are an admission key, not a user-facing label. Normalize
|
||||
* whitespace, path separators, and casing deterministically so equivalent
|
||||
* `pkg/file.ts#Exported.member` references contend across agents.
|
||||
*/
|
||||
export function normalizeSymbolLockKey(rawSymbol: string): string {
|
||||
const normalized = rawSymbol
|
||||
.trim()
|
||||
.replaceAll("\\", "/")
|
||||
.replace(/\s+/g, "")
|
||||
.replace(/\/+/g, "/")
|
||||
.replace(/:+/g, ":")
|
||||
.replace(/#+/g, "#")
|
||||
.toLowerCase();
|
||||
if (!normalized) throw new Error("Symbol lock key must not be empty");
|
||||
return normalized;
|
||||
}
|
||||
|
||||
/** Extracts the canonical project-scoped identity from a raw symbol reference. */
|
||||
export function extractSymbolLockIdentity(projectId: string, rawSymbol: string): SymbolLockIdentity {
|
||||
const normalizedSymbol = normalizeSymbolLockKey(rawSymbol);
|
||||
return { projectId, normalizedSymbol, symbolKey: normalizedSymbol };
|
||||
}
|
||||
|
||||
/** Equivalent canonical symbols contend; callers must normalize before storage. */
|
||||
export function symbolLocksConflict(left: string, right: string): boolean {
|
||||
return normalizeSymbolLockKey(left) === normalizeSymbolLockKey(right);
|
||||
}
|
||||
|
||||
function symbolKeys(symbols: readonly string[]): string[] {
|
||||
return [...new Set(symbols.map(normalizeSymbolLockKey))].sort();
|
||||
}
|
||||
|
||||
function toLock(row: typeof schema.project.symbolLocks.$inferSelect): SymbolLock {
|
||||
return {
|
||||
projectId: row.projectId,
|
||||
symbolKey: row.symbolKey,
|
||||
normalizedSymbol: row.symbolKey,
|
||||
ownerTaskId: row.ownerTaskId,
|
||||
missionId: row.missionId ?? undefined,
|
||||
featureId: row.featureId ?? undefined,
|
||||
lineageId: row.lineageId ?? undefined,
|
||||
nodeId: row.nodeId ?? undefined,
|
||||
agentId: row.agentId ?? undefined,
|
||||
status: row.status as SymbolLock["status"],
|
||||
acquiredAt: row.acquiredAt,
|
||||
renewedAt: row.renewedAt,
|
||||
expiresAt: row.expiresAt,
|
||||
};
|
||||
}
|
||||
|
||||
function toConflict(row: typeof schema.project.symbolLocks.$inferSelect): SymbolLockConflict {
|
||||
return {
|
||||
symbolKey: row.symbolKey,
|
||||
ownerTaskId: row.ownerTaskId,
|
||||
missionId: row.missionId ?? undefined,
|
||||
featureId: row.featureId ?? undefined,
|
||||
lineageId: row.lineageId ?? undefined,
|
||||
nodeId: row.nodeId ?? undefined,
|
||||
agentId: row.agentId ?? undefined,
|
||||
expiresAt: row.expiresAt,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:SymbolLock 2026-07-30-14:10:
|
||||
* Advisory transaction locks serialize same-symbol admissions even when no row
|
||||
* exists yet. Acquiring all sorted keys before inspection preserves the
|
||||
* all-or-nothing contract without making later scheduler behavior responsible
|
||||
* for PostgreSQL race recovery.
|
||||
*/
|
||||
async function lockSymbolKeys(tx: DbTransaction, projectId: string, keys: readonly string[]): Promise<void> {
|
||||
for (const key of keys) {
|
||||
await tx.execute(sql`SELECT pg_advisory_xact_lock(hashtextextended(${`${projectId}:${key}`}, 0))`);
|
||||
}
|
||||
}
|
||||
|
||||
export async function acquireSymbolLocksAsync(
|
||||
store: TaskStore,
|
||||
symbols: readonly string[],
|
||||
owner: SymbolLockOwner,
|
||||
leaseMs: number,
|
||||
): Promise<AcquireSymbolLocksResult> {
|
||||
const layer = store.getAsyncLayer();
|
||||
if (!layer) throw new Error("Durable symbol locks require an AsyncDataLayer");
|
||||
if (!Number.isFinite(leaseMs) || leaseMs <= 0) throw new Error("Symbol lock leaseMs must be positive");
|
||||
const projectId = projectOwnershipPartition(layer.projectId);
|
||||
const keys = symbolKeys(symbols);
|
||||
if (keys.length === 0) return { acquired: true, locks: [], conflicts: [] };
|
||||
|
||||
return layer.transactionImmediate(async (tx) => {
|
||||
await lockSymbolKeys(tx, projectId, keys);
|
||||
const now = new Date();
|
||||
const nowIso = now.toISOString();
|
||||
const rows = await tx.select().from(schema.project.symbolLocks).where(and(
|
||||
eq(schema.project.symbolLocks.projectId, projectId),
|
||||
inArray(schema.project.symbolLocks.symbolKey, keys),
|
||||
));
|
||||
const conflicts = rows.filter((row) => row.status === "held" && row.expiresAt > nowIso && row.ownerTaskId !== owner.ownerTaskId);
|
||||
if (conflicts.length > 0) {
|
||||
await recordRunAuditEventWithinTransaction(tx, {
|
||||
taskId: owner.ownerTaskId, agentId: owner.agentId ?? "symbol-lock", runId: `symbol-lock:${owner.ownerTaskId}`,
|
||||
domain: "symbol-lock", mutationType: "symbol-lock:acquire-conflict", target: owner.ownerTaskId,
|
||||
metadata: { count: conflicts.length, symbolKeys: conflicts.map((row) => row.symbolKey).sort(), outcome: "conflict" },
|
||||
});
|
||||
return { acquired: false, locks: [], conflicts: conflicts.map(toConflict) };
|
||||
}
|
||||
const expiresAt = new Date(now.getTime() + leaseMs).toISOString();
|
||||
for (const key of keys) {
|
||||
const existing = rows.find((row) => row.symbolKey === key);
|
||||
const values = {
|
||||
projectId, symbolKey: key, ownerTaskId: owner.ownerTaskId, missionId: owner.missionId ?? null,
|
||||
featureId: owner.featureId ?? null, lineageId: owner.lineageId ?? null, nodeId: owner.nodeId ?? null,
|
||||
agentId: owner.agentId ?? null, status: "held", acquiredAt: nowIso, renewedAt: nowIso,
|
||||
expiresAt, createdAt: nowIso, updatedAt: nowIso,
|
||||
};
|
||||
if (existing) {
|
||||
await tx.update(schema.project.symbolLocks).set(values).where(and(
|
||||
eq(schema.project.symbolLocks.projectId, projectId), eq(schema.project.symbolLocks.symbolKey, key),
|
||||
));
|
||||
} else {
|
||||
await tx.insert(schema.project.symbolLocks).values(values);
|
||||
}
|
||||
}
|
||||
const held = await tx.select().from(schema.project.symbolLocks).where(and(
|
||||
eq(schema.project.symbolLocks.projectId, projectId), inArray(schema.project.symbolLocks.symbolKey, keys),
|
||||
));
|
||||
await recordRunAuditEventWithinTransaction(tx, {
|
||||
taskId: owner.ownerTaskId, agentId: owner.agentId ?? "symbol-lock", runId: `symbol-lock:${owner.ownerTaskId}`,
|
||||
domain: "symbol-lock", mutationType: "symbol-lock:acquired", target: owner.ownerTaskId,
|
||||
metadata: { count: held.length, symbolKeys: keys, outcome: "acquired" },
|
||||
});
|
||||
return { acquired: true, locks: held.map(toLock), conflicts: [] };
|
||||
});
|
||||
}
|
||||
|
||||
export async function renewSymbolLocksAsync(store: TaskStore, symbols: readonly string[], ownerTaskId: string, leaseMs: number): Promise<RenewSymbolLocksResult> {
|
||||
const layer = store.getAsyncLayer();
|
||||
if (!layer) throw new Error("Durable symbol locks require an AsyncDataLayer");
|
||||
if (!Number.isFinite(leaseMs) || leaseMs <= 0) throw new Error("Symbol lock leaseMs must be positive");
|
||||
const projectId = projectOwnershipPartition(layer.projectId);
|
||||
const keys = symbolKeys(symbols);
|
||||
const now = new Date(); const nowIso = now.toISOString(); const expiresAt = new Date(now.getTime() + leaseMs).toISOString();
|
||||
return layer.transactionImmediate(async (tx) => {
|
||||
await lockSymbolKeys(tx, projectId, keys);
|
||||
const rows = keys.length === 0 ? [] : await tx.select().from(schema.project.symbolLocks).where(and(eq(schema.project.symbolLocks.projectId, projectId), inArray(schema.project.symbolLocks.symbolKey, keys)));
|
||||
const renewable = rows.filter((row) => row.ownerTaskId === ownerTaskId && row.status === "held" && row.expiresAt > nowIso).map((row) => row.symbolKey);
|
||||
// FNXC:SymbolLock 2026-07-30-14:45: recheck expiry in the write predicate
|
||||
// so a delayed renewal cannot revive a lease that became expired after read.
|
||||
const renewed = renewable.length === 0 ? [] : (await tx.update(schema.project.symbolLocks)
|
||||
.set({ renewedAt: nowIso, expiresAt, updatedAt: nowIso })
|
||||
.where(and(
|
||||
eq(schema.project.symbolLocks.projectId, projectId),
|
||||
eq(schema.project.symbolLocks.ownerTaskId, ownerTaskId),
|
||||
inArray(schema.project.symbolLocks.symbolKey, renewable),
|
||||
eq(schema.project.symbolLocks.status, "held"),
|
||||
gt(schema.project.symbolLocks.expiresAt, nowIso),
|
||||
))
|
||||
.returning({ symbolKey: schema.project.symbolLocks.symbolKey }))
|
||||
.map((row) => row.symbolKey);
|
||||
const lost = keys.filter((key) => !renewed.includes(key));
|
||||
await recordRunAuditEventWithinTransaction(tx, { taskId: ownerTaskId, agentId: "symbol-lock", runId: `symbol-lock:${ownerTaskId}`, domain: "symbol-lock", mutationType: "symbol-lock:renewed", target: ownerTaskId, metadata: { count: renewed.length, lostCount: lost.length, symbolKeys: renewed, outcome: "renewed" } });
|
||||
return { renewed, lost };
|
||||
});
|
||||
}
|
||||
|
||||
export async function releaseSymbolLocksAsync(store: TaskStore, symbols: readonly string[], ownerTaskId: string): Promise<ReleaseSymbolLocksResult> {
|
||||
const layer = store.getAsyncLayer();
|
||||
if (!layer) throw new Error("Durable symbol locks require an AsyncDataLayer");
|
||||
const projectId = projectOwnershipPartition(layer.projectId); const keys = symbolKeys(symbols); const nowIso = new Date().toISOString();
|
||||
return layer.transactionImmediate(async (tx) => {
|
||||
await lockSymbolKeys(tx, projectId, keys);
|
||||
const released = keys.length === 0 ? [] : await tx.update(schema.project.symbolLocks).set({ status: "released", updatedAt: nowIso }).where(and(eq(schema.project.symbolLocks.projectId, projectId), eq(schema.project.symbolLocks.ownerTaskId, ownerTaskId), eq(schema.project.symbolLocks.status, "held"), inArray(schema.project.symbolLocks.symbolKey, keys))).returning({ symbolKey: schema.project.symbolLocks.symbolKey });
|
||||
await recordRunAuditEventWithinTransaction(tx, { taskId: ownerTaskId, agentId: "symbol-lock", runId: `symbol-lock:${ownerTaskId}`, domain: "symbol-lock", mutationType: "symbol-lock:released", target: ownerTaskId, metadata: { count: released.length, symbolKeys: released.map((row) => row.symbolKey).sort(), outcome: "released" } });
|
||||
return { released: released.map((row) => row.symbolKey) };
|
||||
});
|
||||
}
|
||||
|
||||
export async function inspectSymbolLockConflictsAsync(store: TaskStore, symbols: readonly string[]): Promise<SymbolLockConflict[]> {
|
||||
const layer = store.getAsyncLayer();
|
||||
if (!layer) throw new Error("Durable symbol locks require an AsyncDataLayer");
|
||||
const keys = symbolKeys(symbols); if (!keys.length) return [];
|
||||
const projectId = projectOwnershipPartition(layer.projectId); const nowIso = new Date().toISOString();
|
||||
const rows = await layer.db.select().from(schema.project.symbolLocks).where(and(eq(schema.project.symbolLocks.projectId, projectId), eq(schema.project.symbolLocks.status, "held"), inArray(schema.project.symbolLocks.symbolKey, keys)));
|
||||
return rows.filter((row) => row.expiresAt > nowIso).map(toConflict);
|
||||
}
|
||||
|
||||
export async function reconcileStaleSymbolLocksAsync(store: TaskStore): Promise<ReconcileStaleSymbolLocksResult> {
|
||||
const layer = store.getAsyncLayer();
|
||||
if (!layer) return { reconciled: [], skipped: [] };
|
||||
const projectId = projectOwnershipPartition(layer.projectId); const nowIso = new Date().toISOString();
|
||||
const held = await layer.db.select().from(schema.project.symbolLocks).where(and(eq(schema.project.symbolLocks.projectId, projectId), eq(schema.project.symbolLocks.status, "held")));
|
||||
const stale: Array<{ symbolKey: string; ownerTaskId: string; expiresAt: string }> = [];
|
||||
const skipped: string[] = [];
|
||||
for (const lock of held) {
|
||||
const owner = await store.getTask(lock.ownerTaskId, { includeDeleted: true }).catch(() => undefined);
|
||||
const terminal = !owner || owner.deletedAt != null || owner.column === "done" || owner.column === "archived" || owner.status === "failed";
|
||||
if (lock.expiresAt <= nowIso || terminal) {
|
||||
stale.push({ symbolKey: lock.symbolKey, ownerTaskId: lock.ownerTaskId, expiresAt: lock.expiresAt });
|
||||
} else {
|
||||
skipped.push(lock.symbolKey);
|
||||
}
|
||||
}
|
||||
if (!stale.length) return { reconciled: [], skipped };
|
||||
const reconciled = await layer.transactionImmediate(async (tx) => {
|
||||
// FNXC:SymbolLock 2026-07-30-14:40: stale detection happens before this
|
||||
// transaction, so CAS on the observed owner and lease prevents a sweep from
|
||||
// expiring a lock that was reclaimed by another task in the interim.
|
||||
await lockSymbolKeys(tx, projectId, stale.map((lock) => lock.symbolKey).sort());
|
||||
const expired: string[] = [];
|
||||
for (const lock of stale) {
|
||||
const updated = await tx.update(schema.project.symbolLocks)
|
||||
.set({ status: "expired", updatedAt: nowIso })
|
||||
.where(and(
|
||||
eq(schema.project.symbolLocks.projectId, projectId),
|
||||
eq(schema.project.symbolLocks.symbolKey, lock.symbolKey),
|
||||
eq(schema.project.symbolLocks.ownerTaskId, lock.ownerTaskId),
|
||||
eq(schema.project.symbolLocks.expiresAt, lock.expiresAt),
|
||||
eq(schema.project.symbolLocks.status, "held"),
|
||||
))
|
||||
.returning({ symbolKey: schema.project.symbolLocks.symbolKey });
|
||||
expired.push(...updated.map((row) => row.symbolKey));
|
||||
}
|
||||
return expired;
|
||||
});
|
||||
return { reconciled, skipped };
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { SelfHealingManager } from "../self-healing.js";
|
||||
import type { TaskStore } from "@fusion/core";
|
||||
|
||||
describe("SelfHealingManager symbol-lock reconciliation", () => {
|
||||
it("audits stale lock reclamation and dedupes an idle no-action sweep", async () => {
|
||||
const reconcileStaleSymbolLocks = vi.fn()
|
||||
.mockResolvedValueOnce({ reconciled: ["pkg/a.ts#a"], skipped: ["pkg/live.ts#a"] })
|
||||
.mockResolvedValue({ reconciled: [], skipped: ["pkg/live.ts#a"] });
|
||||
const recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
|
||||
const manager = new SelfHealingManager({ reconcileStaleSymbolLocks, recordRunAuditEvent } as unknown as TaskStore, { rootDir: "/tmp/symbol-lock-test" });
|
||||
await expect(manager.reconcileStaleSymbolLocks()).resolves.toBe(1);
|
||||
await expect(manager.reconcileStaleSymbolLocks()).resolves.toBe(0);
|
||||
await expect(manager.reconcileStaleSymbolLocks()).resolves.toBe(0);
|
||||
expect(recordRunAuditEvent).toHaveBeenCalledTimes(2);
|
||||
expect(recordRunAuditEvent.mock.calls[0]?.[0]).toMatchObject({ mutationType: "symbol-lock:reconcile-stale", metadata: { count: 1, outcome: "reconciled" } });
|
||||
expect(recordRunAuditEvent.mock.calls[1]?.[0]).toMatchObject({ mutationType: "symbol-lock:reconcile-stale-no-action", metadata: { count: 0, outcome: "no-action" } });
|
||||
});
|
||||
});
|
||||
@@ -751,6 +751,8 @@ export class SelfHealingManager {
|
||||
* because the task leaves the promotable state before returning to it.
|
||||
*/
|
||||
private strandedCompletedFailureProvenanceWarned = new Set<string>();
|
||||
/* FNXC:SymbolLock 2026-07-30-14:20: idle symbol-lock sweeps emit one no-action audit until a stale lock re-arms the diagnostic. */
|
||||
private symbolLockNoActionAudited = false;
|
||||
private metaResolvedSkipAuditMemo = new Map<string, string>();
|
||||
private metaStalledSkipAuditMemo = new Map<string, string>();
|
||||
private preservedQueuedOverlapLogged = new Map<string, string>();
|
||||
@@ -1396,6 +1398,7 @@ export class SelfHealingManager {
|
||||
{ name: "clear-stale-blocked-by", fn: () => this.clearStaleBlockedBy().then(() => undefined) },
|
||||
{ name: "reconcile-self-defeating-deps", fn: () => this.reconcileSelfDefeatingDependencies().then(() => undefined) },
|
||||
{ name: "reconcile-dependency-blocking-leases", fn: () => this.reconcileDependencyBlockingLeases().then(() => undefined) },
|
||||
{ name: "reconcile-stale-symbol-locks", fn: () => this.reconcileStaleSymbolLocks().then(() => undefined) },
|
||||
{ name: "reconcile-completed-blocked", fn: () => this.reconcileCompletedBlockedTasks().then(() => undefined) },
|
||||
{ name: "reconcile-in-review-unmet-dependencies", fn: () => this.reconcileInReviewUnmetDependencies().then(() => undefined) },
|
||||
{ name: "reconcile-engine-downtime-active-timing", fn: () => this.reconcileEngineDowntimeActiveTiming().then(() => undefined) },
|
||||
@@ -2512,6 +2515,10 @@ export class SelfHealingManager {
|
||||
return result;
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "reconcile-stale-symbol-locks",
|
||||
fn: () => this.reconcileStaleSymbolLocks(),
|
||||
},
|
||||
{
|
||||
name: "reconcile-phantom-committed-reservations",
|
||||
fn: async () => {
|
||||
@@ -5658,6 +5665,34 @@ export class SelfHealingManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* FNXC:SymbolLock 2026-07-30-14:20:
|
||||
* Crash recovery expires only locks whose owner is terminal/missing or whose
|
||||
* lease elapsed. It is intentionally orthogonal to scheduler admission and
|
||||
* never changes a task, worktree, semaphore, or verification state.
|
||||
*/
|
||||
async reconcileStaleSymbolLocks(): Promise<number> {
|
||||
const result = await this.store.reconcileStaleSymbolLocks();
|
||||
if (result.reconciled.length > 0) {
|
||||
this.symbolLockNoActionAudited = false;
|
||||
await this.store.recordRunAuditEvent({
|
||||
agentId: "self-healing", runId: "symbol-lock-reconcile", domain: "database",
|
||||
mutationType: "symbol-lock:reconcile-stale", target: "symbol-locks",
|
||||
metadata: { count: result.reconciled.length, symbolKeys: result.reconciled, outcome: "reconciled" },
|
||||
});
|
||||
return result.reconciled.length;
|
||||
}
|
||||
if (!this.symbolLockNoActionAudited) {
|
||||
this.symbolLockNoActionAudited = true;
|
||||
await this.store.recordRunAuditEvent({
|
||||
agentId: "self-healing", runId: "symbol-lock-reconcile", domain: "database",
|
||||
mutationType: "symbol-lock:reconcile-stale-no-action", target: "symbol-locks",
|
||||
metadata: { count: 0, outcome: "no-action" },
|
||||
});
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
async reconcileDependencyBlockingLeases(): Promise<number> {
|
||||
const settings = await this.store.getSettings();
|
||||
if (settings.globalPause || settings.enginePaused) return 0;
|
||||
|
||||
Reference in New Issue
Block a user