fusion/docs/solutions/architecture/postgres-cross-process-task-deleted-observation.md

category, module, tags, problem_type, applies_when

category	module	tags	problem_type	applies_when
architecture	PostgreSQL lifecycle observation	postgresql task-lifecycle cross-process outbox	architecture-decision	A Fusion deployment has more than one TaskStore process sharing a PostgreSQL project and a consumer needs to observe task deletion.

PostgreSQL cross-process task:deleted observation

Decision

Use a transactional PostgreSQL outbox with per-consumer durable cursors for cross-process lifecycle observation. The initial scope is task:deleted; its event contract must be deliberately extended before other lifecycle events join the stream.

The unreachable polling replica is removed by FN-8683. FN-8684 lands the transactional writer and FN-8685 lands the consumer-state migration, explicit identity helper, observed dispatcher, fenced polling seam, and bounded self-healing retention invocation. The consumer remains at-least-once: dispatch precedes the durable receipt/cursor acknowledgement, so crash-window duplicates are expected and observed listeners must remain idempotent.

Problem and current boundary

TaskStore lifecycle events are process-local EventEmitter events. The former SQLite polling replica in packages/core/src/task-store/lifecycle-ops.ts attempted to compare a cache with task rows and re-emit deletions. It cannot operate in the supported runtime: dbImpl in packages/core/src/task-store/task-id-integrity.ts always throws for store.db, with no backend-mode branch, and watchImpl returns after asynchronously warming taskCache for an injected AsyncDataLayer.

Consequently, a delete in one PostgreSQL process is observed only by that writer's store. A dashboard, CLI, child process, or other node holding another TaskStore does not receive task:deleted, and must not be described as having cross-process lifecycle observation until the outbox exists.

The mechanism must ultimately feed these existing bridge and listener surfaces:

• packages/engine/src/project-runtime.ts event contract;
• packages/engine/src/runtimes/in-process-runtime.ts TaskStore-to-runtime forwarding;
• packages/engine/src/runtimes/child-process-runtime.ts IPC receipt and child-process-worker.ts IPC send;
• packages/engine/src/runtimes/remote-node-runtime.ts remote-node transport;
• packages/engine/src/project-manager.ts project-runtime forwarding;
• live consumers including the scheduler, executor, triage, project engine, and dashboard SSE endpoint.

Existing delete ownership

The delete writer owns the transactional soft-delete (deletedAt) and its one task:deleted run-audit row. After commit it also owns the best-effort operator mailbox side effect in task-delete-notice.ts. An observing process must not invoke either writer-owned operation again. Observers only update their local cache and deliver an explicitly observed lifecycle notification to safe consumers.

deletedAt remains the deletion identity: a repeated delete of an already soft-deleted task creates no new lifecycle event. An archived task is a separate archive flow; its delete event retains the pre-delete task snapshot and does not turn an archive move into a second delete.

Options considered

PostgreSQL LISTEN/NOTIFY

• Delivery: at-most-once. Notifications are delivered only to connected listeners after transaction commit; payloads are not a durable queue.
• Restart/drop behavior: a disconnected or restarting consumer misses notifications. It can run a full reconciliation against tasks/deletedAt, but cannot recover an ordered, bounded event history from NOTIFY itself.
• Ordering: commit-order notification is separate from the synchronous in-process emitter. The writer can emit locally after commit, while remote delivery has no exactly-once or global listener order.
• Soft deletes and side effects: payloads can name a task and deletedAt, but subscribers must query to distinguish stale/duplicate data. A subscriber must not write mailbox notices or delete run-audit rows, or it double-fires writer-owned side effects.
• Migration: no table migration is required, but a dedicated connection lifecycle, channel authorization, payload-size handling, and reconciliation path are required.
• Delete-write cost: one pg_notify call in the delete transaction; small, but it provides no durable retry contract.

Rejected: it improves promptness but cannot meet durable delivery across an engine restart or dropped connection without separately building a durable reconciliation mechanism. Adding that mechanism makes a durable outbox the clearer primary record.

Transactional outbox (recommended)

• Delivery: at-least-once to each registered durable consumer; consumers make handling idempotent. The delete and outbox insert commit atomically, so a committed delete cannot be silently absent from the event stream.
• Restart/drop behavior: consumers resume from their stored cursors and scan missed rows. A lost live wake-up only delays the scan; it does not lose the event.
• Ordering: outbox sequence order is the authoritative cross-process order per project. It does not redefine synchronous in-process EventEmitter ordering; local writer listeners remain immediate, while remote consumers process committed sequence order.
• Soft deletes and side effects: one task:deleted event is inserted only for the first deletedAt transition. The observed-event dispatcher must carry originProcessId/writer identity and observed: true, and must forbid mailbox notice, delete run-audit, or delete mutation calls.
• Migration: yes. An outbox table, consumer-cursor table, and dead-letter/attempt state require a numbered migration explicitly registered in packages/core/src/postgres/schema-applier.ts (version constant plus bookkeeping check).
• Delete-write cost: one additional indexed insert in the existing delete transaction. This is bounded and fails atomically with the delete instead of creating a hidden loss window.

Selected because its durable record supplies replay, ordered catch-up, operational visibility, and a bounded recovery path without relying on a continuously connected process.

Engine → dashboard bridge

• Delivery: typically at-most-once unless the bridge develops persistence, acknowledgement, replay, and retry. IPC/SSE alone loses messages on engine/dashboard restart or reconnect.
• Restart/drop behavior: the dashboard can refetch board data, but an engine restart or independent CLI/remote-node consumer has no shared durable history. The bridge also cannot reach a dashboard process that is not connected to that engine.
• Ordering: can preserve a single engine's send order, but not globally across multiple engine nodes or direct API/CLI writers.
• Soft deletes and side effects: the bridge can forward the writer event, but receivers must still avoid repeating writer-owned notice/audit work.
• Migration: no migration for a simple bridge; a durable bridge would need storage equivalent to an outbox.
• Delete-write cost: low for send-only IPC, but it moves reliability cost into process supervision and reconnect handling.

Rejected: it is useful as a transport adapter after durable observation exists, but is not the system-of-record for a multi-node PostgreSQL project and excludes non-engine writers.

Recommended outbox contract

Event shape and identity

Within the delete transaction, insert a project-scoped outbox row such as:

sequence (monotonic per project), project_id, event_type = "task:deleted",
event_id = "task-deleted:<taskId>:<deletedAt>", task_id, deleted_at,
writer_process_id, occurred_at, payload_version

The unique (project_id, event_id) key prevents concurrent delete attempts from producing two events for one deletedAt transition. The payload contains only the minimum post-commit observer data: task ID, deletedAt, prior column/status if consumers need them, event identity, writer identity, and schema version. It never contains mailbox prose, credentials, or free-text run-audit metadata.

Cursor, acknowledgement, and idempotent handling

Create a project-scoped lifecycle_consumer_cursors row keyed by (project_id, consumer_id), holding last_acked_sequence, lease/heartbeat metadata, and current failure state. Every independently durable receiver gets a stable consumer ID (for example dashboard instance role, engine runtime role, or remote-node bridge role); ephemeral SSE clients are fed from their local durable receiver and do not own cursors.

A consumer reads rows with sequence > last_acked_sequence in ascending order. It updates local cache and invokes the observed-event-safe listener path, then advances the cursor in the same consumer transaction where possible. Handler idempotency is mandatory because a process can crash after side effects and before cursor acknowledgement: deduplicate on (consumer_id, event_id) or make every permitted observed handler a no-op when the task is already absent at the recorded deletedAt.

This prevents duplicate delivery from becoming duplicate local work while preserving an at-least-once retry record.

Reconnect catch-up

On startup, connection recovery, or a missed wake-up, a consumer scans from last_acked_sequence + 1. LISTEN/NOTIFY may later wake consumers for low latency, but it is only a hint; cursor scanning is authoritative. Retain normal outbox rows for 30 days after all active durable consumers acknowledge them. A consumer whose cursor is older than the retained floor, missing, or invalid must run full reconciliation: refresh its live task cache from tasks WHERE deleted_at IS NULL, remove locally cached IDs absent from that result, and record its cursor at the current committed sequence before resuming.

This prevents a dropped connection or restart from creating an unrecoverable observation gap; the full reconciliation is the bounded fallback once retained history is no longer available.

Retry and poison handling

For a failed event, do not advance its cursor. Retry with exponential backoff (1s, 5s, 30s, 5m, then 15m) for at most 10 attempts. Persist attempt count, next-attempt time, and last failure class (bounded code only) on a consumer-delivery row or dead-letter table. After attempt 10, atomically park the delivery as dead-letter, emit lifecycle-observation:consumer-poisoned run-audit metadata containing only project/consumer/event IDs and counts, and advance the main cursor only after recording that park.

The consumer continues with later events; a poison event cannot wedge the stream or disappear silently. A repair tool replays a dead-letter event explicitly after the underlying defect is fixed and emits lifecycle-observation:consumer-replayed.

Retention and pruning ownership

SelfHealingManager owns pruning during its scheduled/store-open lifecycle-maintenance sweep. It prunes an acknowledged normal outbox row only when it is older than 30 days and every non-retired durable consumer cursor is at or beyond its sequence. It prunes resolved dead letters after 30 days; unresolved dead letters are retained until an operator resolves or retires their consumer. Consumer retirement is an explicit audited operation, never inferred from a transient missed heartbeat.

This prevents pruning events that an active consumer has not acknowledged and leaves poison evidence available for repair.

Rollout sketch

1. Add schema migration and schema-applier registration for outbox, cursor/delivery, and dead-letter state; include fresh/upgrade/RLS tests.
2. Add a transactional delete-outbox writer beside the existing PostgreSQL soft-delete/audit transaction. Guard the unique event identity and prove idempotent re-delete writes nothing.
3. Implement consumer polling/cursor replay and an observed-event dispatcher. Keep it separate from the writer emitter so it cannot call task-delete-notice.ts, write the delete audit row, or mutate deletion state.
4. Adapt in-process, child-process, remote-node, project-manager, and dashboard bridges to consume observed events from the durable dispatcher. Preserve current payload compatibility for live listeners.
5. Add a NOTIFY wake-up only as an optimization after cursor catch-up is proven; it is never the delivery authority.
6. Enable per deployment, observe cursor lag/dead letters, and retain the old full-cache reconciliation fallback during rollout. The removed SQLite polling replica is not a rollback path because it cannot run in PostgreSQL mode.

Implementation status

FN-8684 landed the transactional writer and FN-8685 landed the consumer-state tables, explicit consumer identity helper, observed dispatch, fenced polling, reconciliation boundary, retry/dead-letter handling, and bounded self-healing retention invocation. Delivery is available only to PostgreSQL TaskStore instances constructed with an explicit durable consumer identity; stores without one intentionally remain observation-disabled. Runtime wiring must source any instance key from a named persisted field, never boot-generated process state.

Consequences

Cross-process task:deleted observation is available through the configured outbox consumer. In-process deletes and their existing listeners remain supported and unchanged. Any feature that needs cross-process lifecycle correctness must depend on the outbox implementation rather than reintroducing store.db polling.