Keep executor lifecycle transitions moving when run-audit sinks fail or stall. - Add a shared bounded, best-effort executor audit emission seam. - Route executor lifecycle audit writes through sink isolation. - Cover absent, throwing, rejecting, hanging, and late-settling sinks. - Document the emitter policy and publish a patch changeset. Files changed: .changeset/fn-9172-executor-run-audit-isolation.md | 7 + AGENTS.md | 7 + docs/run-audit.md | 6 +- .../src/__tests__/emit-bounded-run-audit.test.ts | 58 +++++ .../executor-run-audit-emitter-isolation.test.ts | 252 +++++++++++++++++++++ .../src/executor/acquire-session-registry-path.ts | 5 +- .../engine/src/executor/completion-finalization.ts | 3 +- .../engine/src/executor/create-task-done-tool.ts | 3 +- .../engine/src/executor/emit-bounded-run-audit.ts | 48 ++++ .../executor/emit-merge-boundary-unproven-audit.ts | 68 ++---- .../engine/src/executor/execute-workflow-graph.ts | 5 +- .../engine/src/executor/handle-graph-failure.ts | 18 +- ...dle-stale-in-review-parse-pause-abort-replay.ts | 7 +- ...ndle-stale-in-review-plan-pause-abort-replay.ts | 7 +- .../maybe-dispatch-workflow-work-engine.ts | 7 +- .../src/executor/no-merge-complete-column.ts | 3 +- .../reenter-paused-aborted-workflow-node.ts | 7 +- .../src/executor/required-artifact-recovery.ts | 3 +- packages/engine/src/executor/run-implementation.ts | 5 +- 19 files changed, 433 insertions(+), 86 deletions(-) Fusion-Task-Id: FN-9172 Fusion-Task-Lineage: 80c67c36-3bbb-4274-ac4c-9808c4cda3ce Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
8.5 KiB
Run-Audit Catalogue
The run-audit catalogue for the S4 Reliability, Durability & Observability delivery-pipeline theme — a durable, single-source-of-truth reference for who did what, when, and why after the fact across the delivery pipeline's reliability/observability event surface.
Status / purpose
This document is the run-audit observability catalogue for the Core Product Vision & Roadmap mission (Mission M-MSL4E01A-0001-Y9QC, Milestone M2 — Roadmap Definition, Slice S4 — Reliability, Durability & Observability roadmap, feature F-MSL72J0A-000M-GIJN), grounded in the M1 vision theme verbatim:
"Reliability, durability, and observability of the delivery pipeline — tasks, agents, and their delivery are recoverable and inspectable."
See the north-star grounding: Core Product Roadmap — §4. Reliability, Durability & Observability and Product Vision — Strategic Themes.
The S4 roadmap near-term item ("Recoverable and inspectable delivery") calls for the pipeline's run-audit behavior to be inspectable after the fact. This catalogue is that surface: it centralizes the delivery-pipeline-focus run-audit event names (finalization, self-healing reconciliation, durable-agent error-state) so an operator or agent can answer "which run-audit events are emitted for delivery-pipeline finalization / durable-agent error-state / self-healing reconciliation, and when?" without grepping source. It is kept truthful by a parity test that enforces lock-step with the typed catalogue module.
How to read / query the run-audit surface
Run-audit events are captured via the run-audit store using the discriminated event union type DatabaseMutationType in the engine (packages/engine/src/util/run-audit.ts). Metadata follows the ids/outcomes-only convention — never description prose. All events named below are literal members of that union; the typed catalogue array packages/engine/src/run-audit/run-audit-catalogue.ts enforces member-validity at compile time, and the parity test (run-audit-catalogue.test.ts) keeps this doc and the module in lock-step so neither can drift from the real union.
Query the stored surface through the audit-store read path (project-scoped historical record of emitted mutation events) filtering by the mutationType names below and the ids/outcomes-only structured metadata each event records. This catalogue documents what each event means; the audit store records when each was emitted in a project's history.
Delivery-pipeline finalization
Events that close a task's delivery: blocked/advanced completion parks, already-merged / already-on-main no-ops, finalize column-mismatch reconciliation, post-finalize verification, finalize-blocking guards, and stale-merger recovery.
| Event | What it records / when it fires |
|---|---|
task:completed-blocked-parked |
A fully implemented task is parked instead of advancing to review because a live completion blocker applies. |
task:completed-blocked-advanced |
The parked completed task's blocker cleared and its work advances to review. |
task:auto-recover-already-merged |
Self-healing finds a task already merged into main and records the no-recovery-needed outcome. |
task:auto-recover-finalize-already-on-main |
A finalize attempt is skipped because the task's changes are already present on main. |
task:auto-merge-skipped-already-done |
Auto-merge is skipped because the task is already done/landed. |
task:auto-merge-finalize-column-mismatch-reconciled |
Finalize found the task in a different live column than its target and reconciled the column. |
task:auto-merge-finalize-column-mismatch-no-action |
Finalize found a column mismatch but took no action (e.g. blocked/no-action per triple-proof). |
task:post-finalize-verification-no-op |
Post-finalize verification ran and found nothing to verify (no-op), recording the check outcome. |
task:no-commits-finalize-blocked-incomplete-steps |
Finalize is blocked for a zero-commit task with incomplete workflow steps (FN-6461 lane). |
task:empty-merge-finalize-blocked-no-landed-proof |
The AI empty-merge lane vetoes a zero-diff no-op finalize with no landed proof (FN-8141). |
task:finalize-unproven-blocked |
Finalize is blocked because finalization has not been proven against the landing truth. |
task:merge-boundary-unproven-parked |
A workflow merge boundary could not be proven and its terminal park is recorded with best-effort, time-bounded telemetry that never blocks or stalls the park. |
task:finalize-lost-work-blocked |
Finalize is blocked because it would discard work (lost-work guard). |
task:auto-recover-stale-merger-status |
Self-healing clears a stale merger status left on a finalize path. |
Self-healing reconciliation events
Reconciliation-scoped auto-recover/reclaim events the self-healing sweep surfaces when it repairs board state after the fact.
| Event | What it records / when it fires |
|---|---|
task:auto-recover-paused-abort-park |
Self-healing clears a benign pause-abort operator park and requeues the task. |
task:auto-rebound-paused-scope-decay |
Self-healing rebounds a task whose paused scope decayed past its floor, unblocking followers. |
task:reclaim-phantom-executor-binding |
Self-healing proves an in-memory executor-active binding is stale and requeues the task. |
task:reconcile-orphaned-pending-step-results |
Self-healing rewrites orphaned pending workflow-step results (no live session) to failed. |
task:reconcile-stale-duplicate-decision |
Self-healing clears a recurring duplicate-decision pause with no canonical target. |
task:reconcile-stale-agent-assignment |
Self-healing clears stale durable Agent.taskId/state drift while preserving file-scope leases. |
task:reconcile-engine-downtime-active-timing |
Self-healing shifts active-task anchors to exclude proven stopped-engine wall-clock. |
task:reconcile-engine-downtime-active-timing-no-action |
Self-healing finds no active task qualifies for downtime-timing reconciliation (no-action). |
task:reconcile-undeclared-column |
Self-healing re-homes a row out of a column its workflow no longer declares. |
task:reconcile-wedged-active-merge |
Self-healing reclaims a wedged single-flight merge entry. |
task:reconcile-stranded-completed-no-action |
A stranded-completed promoter withholds promotion of an all-steps-done/skipped task with a failure-park provenance (no-action). |
task:reconcile-legacy-adoption |
Self-healing startup adopts a pre-cutover legacy task row through the KTD-8 adoption table. |
Durable-agent error-state
Events that make durable-agent error states and their recovery inspectable.
| Event | What it records / when it fires |
|---|---|
agent:auto-recover-error-state |
A recoverable, non-operator-actionable durable-agent error is cleared by the heartbeat/self-healing sweep and retried. |
agent:reset-error-state-on-startup |
An engine restart clears an eligible durable-agent error/exhaustion park and re-arms the heartbeat (startup-only). |
agent:error-retry-exhausted |
A durable-agent error retry budget is exhausted and the agent is parked paused with pauseReason error-retry-exhausted. |
agent:error-parked-unrecoverable |
An operator-actionable durable-agent error parks the agent paused with pauseReason error-unrecoverable for human repair. |
agent:heartbeat-move-skipped-soft-delete |
A heartbeat move races a soft-deleted task and is skipped without parking the durable agent. |
Maintenance contract
Adding a new catalogued run-audit event requires updating both the typed catalogue module (packages/engine/src/run-audit/run-audit-catalogue.ts) and this doc together — the parity test (packages/engine/src/__tests__/run-audit-catalogue.test.ts) fails if the documented event set and the catalogue module's set ever diverge, keeping the observability surface truthful as the real DatabaseMutationType union evolves. Removing an event likewise requires updating both in the same change.
Emit-seam policy
Executor telemetry must use emitBoundedRunAudit. It is best-effort and never load-bearing for lifecycle correctness: absent/non-function, synchronously throwing, rejecting, never-settling, and late-settling sinks are absorbed without altering the owning branch. The seam swallow-logs and bounds each write; it intentionally adds no retry, backoff, or queueing.