FN-5924: throttle OAuth expiry alerts across restarts

Persist OAuth expiry alert state so repeated provider expiry warnings are suppressed for 12 hours across engine restarts.

- add a persisted OAuth alert state store under ~/.fusion/agent and share it between the expiry monitor and startup validity logger
- throttle repeated oauth-token-expired notifications and startup expiry warnings per provider for 12 hours, while clearing stale state when providers disappear or change
- cover persisted throttling, restart behavior, failure handling, and wiring updates in engine notification tests
- document the persisted 12-hour OAuth alert throttle and add a patch changeset for @runfusion/fusion

Files changed:
 .changeset/fn-5924-oauth-alert-throttle.md         |   5 +
 docs/settings-reference.md                         |   4 +-
 packages/engine/src/__tests__/project-engine-soft-delete-merge-abort.test.ts |   3 +
 packages/engine/src/__tests__/project-engine.test.ts    |  17 ++-
 packages/engine/src/__tests__/reliability-interactions/soft-delete-in-flight-abort.test.ts            |   3 +
 packages/engine/src/auth-storage.ts                |   6 +-
 packages/engine/src/notification/__tests__/oauth-alert-state.test.ts            |  72 ++++++++++
 packages/engine/src/notification/__tests__/oauth-expiry-monitor.test.ts         | 148 ++++++++++++++++---
 packages/engine/src/notification/__tests__/oauth-validity-logger.test.ts        | 159 ++++++++++++++++++---
 packages/engine/src/notification/index.ts          |   3 +
 packages/engine/src/notification/oauth-alert-state.ts   | 144 +++++++++++++++++++
 packages/engine/src/notification/oauth-expiry-monitor.ts       |  12 +-
 packages/engine/src/notification/oauth-validity-logger.ts      |  16 ++-
 packages/engine/src/project-engine.ts              |  13 +-
 14 files changed, 554 insertions(+), 51 deletions(-)

Fusion-Task-Id: FN-5924
Fusion-Task-Lineage: 83255a25-40c7-44d4-8302-b068ae51250e
This commit is contained in:
gsxdsm
2026-06-02 23:13:16 -07:00
parent 669a6ecfdc
commit a1b7556c7b
14 changed files with 558 additions and 55 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
persist the OAuth expiry alert/notification throttle so users are alerted at most once per provider every 12 hours, even across server restarts.

View File

@@ -45,7 +45,7 @@ Defaults from `DEFAULT_GLOBAL_SETTINGS`; key scope from `GLOBAL_SETTINGS_KEYS`.
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication; Fusion also throttles that notification and the matching startup expiry warning to at most once per provider every 12 hours, and the throttle persists across server restarts. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
| `ntfyDashboardHost` | `string` | `undefined` | Dashboard host used to build deep links in notifications. |
| `taskTokenBudget` | `{ soft?: number; hard?: number; perSize?: { S?: { soft?: number; hard?: number }; M?: { soft?: number; hard?: number }; L?: { soft?: number; hard?: number } } }` | `undefined` | Global fallback per-task token budget policy. Project `taskTokenBudget` overrides this. |
| `webhookEnabled` | `boolean` | `false` | Enable webhook notifications for task lifecycle events. Part of the legacy flat settings; prefer `notificationProviders` for new setups. |
@@ -161,7 +161,7 @@ When `id` is `"ntfy"` in `notificationProviders`, the provider `config` supports
| `topic` | `string` | _required_ | ntfy topic name (164 chars, alphanumeric + `-_`). |
| `ntfyBaseUrl` | `string` | `"https://ntfy.sh"` | Optional custom ntfy server URL. |
| `ntfyAccessToken` | `string` | `undefined` | Optional access token. When set, provider sends `Authorization: Bearer <token>` on ntfy publishes. |
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired. |
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired; Fusion suppresses repeat delivery for the same provider for 12 hours even across server restarts, and applies the same persisted window to the startup expiry warning log. |
| `dashboardHost` | `string` | `undefined` | Dashboard host for deep links in notifications. |
Disable daily update checks globally:

View File

@@ -26,11 +26,14 @@ vi.mock("../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(() =>
vi.mock("../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
vi.mock("../auth-storage.js", () => ({
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
}));
vi.mock("../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
vi.mock("../notification/index.js", () => ({
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
}));
vi.mock("../cron-runner.js", () => ({
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),

View File

@@ -4,7 +4,7 @@ import { ProjectEngine } from "../project-engine.js";
import { runtimeLog } from "../logger.js";
import { TunnelProcessManager } from "../remote-access/tunnel-process-manager.js";
import { NtfyNotifier } from "../notifier.js";
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
const mocks = vi.hoisted(() => ({
syncInsightExtractionAutomation: vi.fn(),
@@ -97,6 +97,7 @@ vi.mock("../notification/index.js", () => ({
start: mocks.notificationServiceStart,
stop: mocks.notificationServiceStop,
})),
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({
start: mocks.oauthExpiryMonitorStart,
stop: mocks.oauthExpiryMonitorStop,
@@ -113,6 +114,7 @@ vi.mock("../auth-storage.js", () => ({
getOAuthProviders: vi.fn(() => []),
get: vi.fn(() => undefined),
})),
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
}));
vi.mock("../runtimes/in-process-runtime.js", () => ({
@@ -305,10 +307,19 @@ describe("ProjectEngine notification ownership wiring", () => {
await engine.start();
expect(NotificationService).toHaveBeenCalledTimes(1);
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
const notifierCtorArgs = vi.mocked(NtfyNotifier).mock.calls[0];
expect(notifierCtorArgs?.[2]).toBe(vi.mocked(NotificationService).mock.results[0]?.value);
const alertStateInstance = vi.mocked(OAuthAlertStateStore).mock.results[0]?.value;
expect(vi.mocked(OAuthExpiryMonitor).mock.calls[0]?.[0]).toEqual(
expect.objectContaining({ alertState: alertStateInstance }),
);
expect(vi.mocked(OAuthValidityLogger).mock.calls[0]?.[0]).toEqual(
expect.objectContaining({ alertState: alertStateInstance }),
);
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
@@ -329,7 +340,9 @@ describe("ProjectEngine notification ownership wiring", () => {
// Root cause guard: if ProjectEngine.start is called more than once, it should not
// wire a second NotificationService/NtfyNotifier pair for the same store.
expect(NotificationService).toHaveBeenCalledTimes(1);
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
@@ -344,7 +357,9 @@ describe("ProjectEngine notification ownership wiring", () => {
await engine.start();
expect(NotificationService).not.toHaveBeenCalled();
expect(OAuthAlertStateStore).not.toHaveBeenCalled();
expect(OAuthExpiryMonitor).not.toHaveBeenCalled();
expect(OAuthValidityLogger).not.toHaveBeenCalled();
expect(NtfyNotifier).not.toHaveBeenCalled();
await engine.stop();

View File

@@ -29,11 +29,14 @@ vi.mock("../../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(()
vi.mock("../../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
vi.mock("../../auth-storage.js", () => ({
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
}));
vi.mock("../../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
vi.mock("../../notification/index.js", () => ({
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
}));
vi.mock("../../cron-runner.js", () => ({
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),

View File

@@ -16,7 +16,7 @@ import type { OAuthCredentials } from "@earendil-works/pi-ai/oauth";
type StoredCredential = StoredAuthCredential;
function getHomeDir(): string {
export function getHomeDir(): string {
return process.env.HOME || process.env.USERPROFILE || homedir();
}
@@ -24,6 +24,10 @@ export function getFusionAuthPath(home = getHomeDir()): string {
return join(home, ".fusion", "agent", "auth.json");
}
export function getFusionOAuthAlertStatePath(home = getHomeDir()): string {
return join(home, ".fusion", "agent", "oauth-alert-state.json");
}
export function getFusionModelsPath(home = getHomeDir()): string {
return join(home, ".fusion", "agent", "models.json");
}

View File

@@ -0,0 +1,72 @@
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
const tempDirs: string[] = [];
function createTempStatePath(): string {
const dir = mkdtempSync(join(tmpdir(), "oauth-alert-state-"));
tempDirs.push(dir);
return join(dir, "oauth-alert-state.json");
}
afterEach(() => {
for (const dir of tempDirs.splice(0)) {
rmSync(dir, { force: true, recursive: true });
}
});
describe("OAuthAlertStateStore", () => {
it("round-trips provider alert state using the configured path", () => {
const statePath = createTempStatePath();
const store = new OAuthAlertStateStore({ statePath, clock: () => 1234 });
store.recordAlert("openai-codex", 9999);
const reloaded = new OAuthAlertStateStore({ statePath });
expect(reloaded.get("openai-codex")).toEqual({ expires: 9999, lastAlertAt: 1234 });
});
it("returns empty state when the file is missing or corrupt", () => {
const missingPath = createTempStatePath();
const missingStore = new OAuthAlertStateStore({ statePath: missingPath });
expect(missingStore.get("openai-codex")).toBeUndefined();
const corruptPath = createTempStatePath();
writeFileSync(corruptPath, "{not json", "utf-8");
const corruptStore = new OAuthAlertStateStore({ statePath: corruptPath });
expect(corruptStore.get("openai-codex")).toBeUndefined();
});
it("persists only provider ids with expires and lastAlertAt", () => {
const statePath = createTempStatePath();
const store = new OAuthAlertStateStore({ statePath, clock: () => 5678 });
store.recordAlert("claude", 4321);
expect(JSON.parse(readFileSync(statePath, "utf-8"))).toEqual({
claude: {
expires: 4321,
lastAlertAt: 5678,
},
});
});
it("clears selected providers and all providers", () => {
const statePath = createTempStatePath();
const store = new OAuthAlertStateStore({ statePath, clock: () => 100 });
store.recordAlert("claude", 1_000);
store.recordAlert("openai-codex", 2_000, 200);
store.clear(["claude"]);
const afterSingleClear = new OAuthAlertStateStore({ statePath });
expect(afterSingleClear.get("claude")).toBeUndefined();
expect(afterSingleClear.get("openai-codex")).toEqual({ expires: 2_000, lastAlertAt: 200 });
afterSingleClear.clear();
expect(new OAuthAlertStateStore({ statePath }).get("openai-codex")).toBeUndefined();
});
});

View File

@@ -1,6 +1,18 @@
import { describe, expect, it, vi } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
import { OAuthExpiryMonitor, type AuthStorageLike } from "../oauth-expiry-monitor.js";
const tempDirs: string[] = [];
function createStatePath(): string {
const dir = mkdtempSync(join(tmpdir(), "oauth-expiry-monitor-"));
tempDirs.push(dir);
return join(dir, "oauth-alert-state.json");
}
function createAuthStorage(initialCredential?: { type?: string; expires?: number }): AuthStorageLike & {
credential: { type?: string; expires?: number } | undefined;
} {
@@ -17,17 +29,26 @@ function createAuthStorage(initialCredential?: { type?: string; expires?: number
};
}
afterEach(() => {
vi.useRealTimers();
for (const dir of tempDirs.splice(0)) {
rmSync(dir, { force: true, recursive: true });
}
});
describe("OAuthExpiryMonitor", () => {
it("fires once when an OAuth credential is expired", async () => {
vi.useFakeTimers();
const authStorage = createAuthStorage({ type: "oauth", expires: Date.now() - 1_000 });
const now = Date.now();
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1_000 });
const dispatch = vi.fn(async () => undefined);
const monitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
intervalMs: 100,
clock: () => Date.now(),
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await monitor.start();
@@ -45,7 +66,6 @@ describe("OAuthExpiryMonitor", () => {
}),
);
monitor.stop();
vi.useRealTimers();
});
it("does not fire for non-expired/non-oauth credentials", async () => {
@@ -67,6 +87,7 @@ describe("OAuthExpiryMonitor", () => {
notificationService: { dispatch } as any,
intervalMs: 100,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await monitor.start();
@@ -75,7 +96,6 @@ describe("OAuthExpiryMonitor", () => {
}
expect(dispatch).not.toHaveBeenCalled();
vi.useRealTimers();
});
it("deduplicates dispatches for same provider and expiry", async () => {
@@ -89,6 +109,7 @@ describe("OAuthExpiryMonitor", () => {
notificationService: { dispatch } as any,
intervalMs: 100,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await monitor.start();
@@ -96,12 +117,12 @@ describe("OAuthExpiryMonitor", () => {
expect(dispatch).toHaveBeenCalledTimes(1);
monitor.stop();
vi.useRealTimers();
});
it("re-fires after credential is replaced with a new expiry that later expires", async () => {
vi.useFakeTimers();
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
const dispatch = vi.fn(async () => undefined);
@@ -110,6 +131,7 @@ describe("OAuthExpiryMonitor", () => {
notificationService: { dispatch } as any,
intervalMs: 100,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await monitor.start();
@@ -128,42 +150,126 @@ describe("OAuthExpiryMonitor", () => {
expect(dispatch).toHaveBeenCalledTimes(2);
monitor.stop();
vi.useRealTimers();
});
it("throttles changed expiries until min notify interval elapses", async () => {
it("throttles changed expiries until min notify interval elapses across restarts", async () => {
vi.useFakeTimers();
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
const dispatch = vi.fn(async () => undefined);
const monitor = new OAuthExpiryMonitor({
const firstMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
intervalMs: 100,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await monitor.start();
expect(dispatch).toHaveBeenCalledTimes(1);
authStorage.credential = { type: "oauth", expires: now + 10_000 };
await vi.advanceTimersByTimeAsync(100);
now += 500;
authStorage.credential = { type: "oauth", expires: now - 1 };
await vi.advanceTimersByTimeAsync(100);
await firstMonitor.start();
firstMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(1);
now += 500;
authStorage.credential = { type: "oauth", expires: now - 2 };
await vi.advanceTimersByTimeAsync(100);
const restartedMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await restartedMonitor.start();
restartedMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(1);
now += 500;
authStorage.credential = { type: "oauth", expires: now - 3 };
await restartedMonitor.start();
restartedMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(2);
});
it("does not persist lastAlertAt when dispatch fails", async () => {
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
const dispatch = vi.fn(async () => {
throw new Error("boom");
});
const firstMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await firstMonitor.start();
firstMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(1);
const secondDispatch = vi.fn(async () => undefined);
now += 100;
const restartedMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch: secondDispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await restartedMonitor.start();
restartedMonitor.stop();
expect(secondDispatch).toHaveBeenCalledTimes(1);
});
it("clears persisted state when providers disappear", async () => {
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
const dispatch = vi.fn(async () => undefined);
const firstMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await firstMonitor.start();
firstMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(1);
const noProviderStorage: AuthStorageLike = {
reload: vi.fn(),
getOAuthProviders: () => [],
get: () => undefined,
};
const clearingMonitor = new OAuthExpiryMonitor({
authStorage: noProviderStorage,
notificationService: { dispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await clearingMonitor.start();
clearingMonitor.stop();
now += 100;
const restartedMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: { dispatch } as any,
minNotifyIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await restartedMonitor.start();
restartedMonitor.stop();
expect(dispatch).toHaveBeenCalledTimes(2);
monitor.stop();
vi.useRealTimers();
});
it("stop cancels the interval", async () => {
@@ -177,6 +283,7 @@ describe("OAuthExpiryMonitor", () => {
notificationService: { dispatch } as any,
intervalMs: 100,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await monitor.start();
@@ -184,6 +291,5 @@ describe("OAuthExpiryMonitor", () => {
await vi.advanceTimersByTimeAsync(500);
expect(dispatch).toHaveBeenCalledTimes(1);
vi.useRealTimers();
});
});

View File

@@ -1,7 +1,19 @@
import { describe, expect, it, vi } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
import { OAuthValidityLogger } from "../oauth-validity-logger.js";
import type { AuthStorageLike } from "../oauth-expiry-monitor.js";
const tempDirs: string[] = [];
function createStatePath(): string {
const dir = mkdtempSync(join(tmpdir(), "oauth-validity-logger-"));
tempDirs.push(dir);
return join(dir, "oauth-alert-state.json");
}
function createAuthStorage(providers: Array<{ id: string; name: string }>, credentials: Record<string, any>): AuthStorageLike {
return {
reload: vi.fn(),
@@ -10,6 +22,13 @@ function createAuthStorage(providers: Array<{ id: string; name: string }>, crede
};
}
afterEach(() => {
vi.useRealTimers();
for (const dir of tempDirs.splice(0)) {
rmSync(dir, { force: true, recursive: true });
}
});
describe("OAuthValidityLogger", () => {
it("logs one line per expired oauth credential on start", async () => {
vi.useFakeTimers();
@@ -26,30 +45,79 @@ describe("OAuthValidityLogger", () => {
},
);
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await validityLogger.start();
expect(logger).toHaveBeenCalledTimes(2);
validityLogger.stop();
vi.useRealTimers();
});
it("logs again on interval without dedupe", async () => {
it("skips repeated logs within the throttle window", async () => {
vi.useFakeTimers();
const now = Date.now();
const logger = vi.fn();
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage(
[{ id: "openai-codex", name: "OpenAI Codex" }],
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
);
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
await validityLogger.start();
await vi.advanceTimersByTimeAsync(1_000);
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 100,
minAlertIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
expect(logger).toHaveBeenCalledTimes(2);
await validityLogger.start();
now += 500;
await validityLogger.check();
expect(logger).toHaveBeenCalledTimes(1);
validityLogger.stop();
vi.useRealTimers();
});
it("persists the throttle across a restart and logs again after the window elapses", async () => {
vi.useFakeTimers();
const logger = vi.fn();
let now = Date.now();
const statePath = createStatePath();
const authStorage = createAuthStorage(
[{ id: "openai-codex", name: "OpenAI Codex" }],
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
);
const firstLogger = new OAuthValidityLogger({
authStorage,
logger,
minAlertIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await firstLogger.check();
expect(logger).toHaveBeenCalledTimes(1);
const restartedLogger = new OAuthValidityLogger({
authStorage,
logger,
minAlertIntervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
});
await restartedLogger.check();
expect(logger).toHaveBeenCalledTimes(1);
now += 1_001;
await restartedLogger.check();
expect(logger).toHaveBeenCalledTimes(2);
});
it("does not log for valid oauth, api key, or missing expires", async () => {
@@ -69,30 +137,42 @@ describe("OAuthValidityLogger", () => {
},
);
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await validityLogger.start();
expect(logger).not.toHaveBeenCalled();
validityLogger.stop();
vi.useRealTimers();
});
it("stop cancels the interval", async () => {
vi.useFakeTimers();
const now = Date.now();
let now = Date.now();
const logger = vi.fn();
const authStorage = createAuthStorage(
[{ id: "openai-codex", name: "OpenAI Codex" }],
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
);
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 1_000,
minAlertIntervalMs: 500,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await validityLogger.start();
validityLogger.stop();
now += 5_000;
await vi.advanceTimersByTimeAsync(5_000);
expect(logger).toHaveBeenCalledTimes(1);
vi.useRealTimers();
});
it("continues iterating when one provider throws", async () => {
@@ -113,7 +193,13 @@ describe("OAuthValidityLogger", () => {
},
};
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await validityLogger.start();
expect(logger).toHaveBeenCalledTimes(1);
@@ -122,7 +208,6 @@ describe("OAuthValidityLogger", () => {
expect.objectContaining({ providerId: "claude" }),
);
validityLogger.stop();
vi.useRealTimers();
});
it("never includes token material in log metadata", async () => {
@@ -141,12 +226,50 @@ describe("OAuthValidityLogger", () => {
},
);
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
intervalMs: 1_000,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
});
await validityLogger.start();
const [, meta] = logger.mock.calls[0] ?? [];
expect(Object.keys(meta ?? {}).sort()).toEqual(["expiresAt", "providerId", "providerName"]);
validityLogger.stop();
vi.useRealTimers();
});
it("covers empty, undefined, and populated provider states", async () => {
const logger = vi.fn();
const now = Date.now();
const cases: AuthStorageLike[] = [
{
reload: vi.fn(),
getOAuthProviders: () => [],
get: () => undefined,
},
{
reload: vi.fn(),
getOAuthProviders: () => [{ id: "openai-codex", name: "OpenAI Codex" }],
get: () => undefined,
},
createAuthStorage(
[{ id: "openai-codex", name: "OpenAI Codex" }],
{ "openai-codex": { type: "oauth", expires: now - 1 } },
),
];
for (const [index, authStorage] of cases.entries()) {
const validityLogger = new OAuthValidityLogger({
authStorage,
logger,
clock: () => now,
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now + index }),
});
await validityLogger.check();
}
expect(logger).toHaveBeenCalledTimes(1);
});
});

View File

@@ -7,6 +7,9 @@ export type { WebhookProviderConfig } from "./webhook-provider.js";
export { NotificationService } from "./notification-service.js";
export type { NotificationServiceOptions } from "./notification-service.js";
export { OAuthAlertStateStore } from "./oauth-alert-state.js";
export type { OAuthAlertStateEntry, OAuthAlertStateFs, OAuthAlertStateStoreOptions } from "./oauth-alert-state.js";
export { OAuthExpiryMonitor } from "./oauth-expiry-monitor.js";
export type { AuthStorageLike as OAuthExpiryAuthStorageLike, OAuthExpiryMonitorOptions } from "./oauth-expiry-monitor.js";

View File

@@ -0,0 +1,144 @@
import {
existsSync,
mkdirSync,
readFileSync,
renameSync,
rmSync,
writeFileSync,
} from "node:fs";
import { dirname } from "node:path";
import { getFusionOAuthAlertStatePath } from "../auth-storage.js";
export interface OAuthAlertStateEntry {
expires: number;
lastAlertAt: number;
}
export interface OAuthAlertStateFs {
existsSync(path: string): boolean;
mkdirSync(path: string, options?: { recursive?: boolean }): void;
readFileSync(path: string, encoding: BufferEncoding): string;
renameSync(oldPath: string, newPath: string): void;
rmSync(path: string, options?: { force?: boolean }): void;
writeFileSync(path: string, content: string, encoding: BufferEncoding): void;
}
export interface OAuthAlertStateStoreOptions {
statePath?: string;
clock?: () => number;
fs?: OAuthAlertStateFs;
}
export class OAuthAlertStateStore {
private readonly statePath: string;
private readonly clock: () => number;
private readonly fs: OAuthAlertStateFs;
constructor(options: OAuthAlertStateStoreOptions = {}) {
this.statePath = options.statePath ?? getFusionOAuthAlertStatePath();
this.clock = options.clock ?? Date.now;
this.fs = options.fs ?? {
existsSync: (path) => existsSync(path),
mkdirSync: (path, options) => {
mkdirSync(path, options);
},
readFileSync: (path, encoding) => readFileSync(path, encoding),
renameSync: (oldPath, newPath) => {
renameSync(oldPath, newPath);
},
rmSync: (path, options) => {
rmSync(path, options);
},
writeFileSync: (path, content, encoding) => {
writeFileSync(path, content, encoding);
},
};
}
get(providerId: string): OAuthAlertStateEntry | undefined {
return this.readState()[providerId];
}
getLastAlertAt(providerId: string): number | undefined {
return this.get(providerId)?.lastAlertAt;
}
recordAlert(providerId: string, expires: number, lastAlertAt = this.clock()): void {
const state = this.readState();
state[providerId] = { expires, lastAlertAt };
this.writeState(state);
}
clear(providerIds?: Iterable<string>): void {
if (!providerIds) {
this.writeState({});
return;
}
const state = this.readState();
let changed = false;
for (const providerId of providerIds) {
if (!(providerId in state)) {
continue;
}
delete state[providerId];
changed = true;
}
if (changed) {
this.writeState(state);
}
}
private readState(): Record<string, OAuthAlertStateEntry> {
if (!this.fs.existsSync(this.statePath)) {
return {};
}
try {
const parsed = JSON.parse(this.fs.readFileSync(this.statePath, "utf-8")) as unknown;
return sanitizeState(parsed);
} catch {
return {};
}
}
private writeState(state: Record<string, OAuthAlertStateEntry>): void {
const sanitized = sanitizeState(state);
const dir = dirname(this.statePath);
this.fs.mkdirSync(dir, { recursive: true });
const tempPath = `${this.statePath}.${process.pid}.${this.clock()}.tmp`;
const body = `${JSON.stringify(sanitized, null, 2)}\n`;
this.fs.writeFileSync(tempPath, body, "utf-8");
try {
this.fs.renameSync(tempPath, this.statePath);
} catch (error) {
this.fs.rmSync(tempPath, { force: true });
throw error;
}
}
}
function sanitizeState(parsed: unknown): Record<string, OAuthAlertStateEntry> {
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
return {};
}
const sanitized: Record<string, OAuthAlertStateEntry> = {};
for (const [providerId, value] of Object.entries(parsed)) {
if (!value || typeof value !== "object" || Array.isArray(value)) {
continue;
}
const expires = (value as { expires?: unknown }).expires;
const lastAlertAt = (value as { lastAlertAt?: unknown }).lastAlertAt;
if (typeof expires !== "number" || Number.isNaN(expires)) {
continue;
}
if (typeof lastAlertAt !== "number" || Number.isNaN(lastAlertAt)) {
continue;
}
sanitized[providerId] = { expires, lastAlertAt };
}
return sanitized;
}

View File

@@ -1,5 +1,6 @@
import type { NotificationPayload } from "@fusion/core";
import { schedulerLog } from "../logger.js";
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
import type { NotificationService } from "./notification-service.js";
const DEFAULT_INTERVAL_MS = 5 * 60_000;
@@ -28,6 +29,7 @@ export interface OAuthExpiryMonitorOptions {
clock?: () => number;
warnBeforeMs?: number;
minNotifyIntervalMs?: number;
alertState?: OAuthAlertStateStore;
}
export class OAuthExpiryMonitor {
@@ -35,15 +37,16 @@ export class OAuthExpiryMonitor {
private readonly clock: () => number;
private readonly warnBeforeMs: number;
private readonly minNotifyIntervalMs: number;
private readonly alertState: OAuthAlertStateStore;
private timer: NodeJS.Timeout | null = null;
private readonly dispatchedExpiryKeys = new Set<string>();
private readonly lastNotifiedAt = new Map<string, number>();
constructor(private readonly opts: OAuthExpiryMonitorOptions) {
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
this.clock = opts.clock ?? Date.now;
this.warnBeforeMs = opts.warnBeforeMs ?? 0;
this.minNotifyIntervalMs = opts.minNotifyIntervalMs ?? DEFAULT_MIN_NOTIFY_INTERVAL_MS;
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
}
async start(): Promise<void> {
@@ -73,7 +76,7 @@ export class OAuthExpiryMonitor {
const providers = this.opts.authStorage.getOAuthProviders?.();
if (!providers?.length) {
this.dispatchedExpiryKeys.clear();
this.lastNotifiedAt.clear();
this.alertState.clear();
return;
}
@@ -83,6 +86,7 @@ export class OAuthExpiryMonitor {
for (const provider of providers) {
const credential = this.opts.authStorage.get?.(provider.id);
if (credential?.type !== "oauth" || typeof credential.expires !== "number") {
this.alertState.clear([provider.id]);
continue;
}
@@ -96,7 +100,7 @@ export class OAuthExpiryMonitor {
continue;
}
const previousNotificationAt = this.lastNotifiedAt.get(provider.id);
const previousNotificationAt = this.alertState.getLastAlertAt(provider.id);
if (
typeof previousNotificationAt === "number" &&
now - previousNotificationAt < this.minNotifyIntervalMs
@@ -116,7 +120,7 @@ export class OAuthExpiryMonitor {
try {
await this.opts.notificationService.dispatch("oauth-token-expired", payload);
this.dispatchedExpiryKeys.add(expiryKey);
this.lastNotifiedAt.set(provider.id, now);
this.alertState.recordAlert(provider.id, credential.expires, now);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
schedulerLog.warn(`OAuth expiry notification dispatch failed provider=${provider.id}: ${message}`);

View File

@@ -1,25 +1,33 @@
import { schedulerLog } from "../logger.js";
import type { AuthStorageLike } from "./oauth-expiry-monitor.js";
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
const DEFAULT_INTERVAL_MS = 24 * 60 * 60 * 1000;
const DEFAULT_INTERVAL_MS = 12 * 60 * 60 * 1000;
const DEFAULT_MIN_ALERT_INTERVAL_MS = 12 * 60 * 60 * 1000;
interface OAuthValidityLoggerOptions {
authStorage: AuthStorageLike;
intervalMs?: number;
clock?: () => number;
logger?: (msg: string, meta?: Record<string, unknown>) => void;
alertState?: OAuthAlertStateStore;
minAlertIntervalMs?: number;
}
export class OAuthValidityLogger {
private readonly intervalMs: number;
private readonly clock: () => number;
private readonly logger: (msg: string, meta?: Record<string, unknown>) => void;
private readonly alertState: OAuthAlertStateStore;
private readonly minAlertIntervalMs: number;
private timer: NodeJS.Timeout | null = null;
constructor(private readonly opts: OAuthValidityLoggerOptions) {
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
this.clock = opts.clock ?? Date.now;
this.logger = opts.logger ?? ((message, meta) => schedulerLog.warn(message, meta));
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
this.minAlertIntervalMs = opts.minAlertIntervalMs ?? DEFAULT_MIN_ALERT_INTERVAL_MS;
}
async start(): Promise<void> {
@@ -57,11 +65,17 @@ export class OAuthValidityLogger {
continue;
}
const previousAlertAt = this.alertState.getLastAlertAt(provider.id);
if (typeof previousAlertAt === "number" && now - previousAlertAt < this.minAlertIntervalMs) {
continue;
}
this.logger("oauth credential expired — provider re-login required", {
providerId: provider.id,
providerName: provider.name,
expiresAt: new Date(credential.expires).toISOString(),
});
this.alertState.recordAlert(provider.id, credential.expires, now);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
schedulerLog.warn(`OAuth validity logger failed for provider=${provider.id}: ${message}`);

View File

@@ -19,10 +19,10 @@ import type { ProjectRuntimeConfig } from "./project-runtime.js";
import { PrMonitor } from "./pr-monitor.js";
import { PrCommentHandler } from "./pr-comment-handler.js";
import { NtfyNotifier } from "./notifier.js";
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
import type { NotificationChatStore } from "./notification/notification-service.js";
import { GridlockDetector } from "./gridlock-detector.js";
import { createFusionAuthStorage } from "./auth-storage.js";
import { createFusionAuthStorage, getFusionOAuthAlertStatePath } from "./auth-storage.js";
import { CronRunner, createAiPromptExecutor } from "./cron-runner.js";
import type { RoutineRunner } from "./routine-runner.js";
import { aiMergeTask, sweepStaleAutostashes, VerificationError } from "./merger.js";
@@ -427,12 +427,19 @@ export class ProjectEngine {
});
await this.notificationService.start();
const authStorage = createFusionAuthStorage();
const oauthAlertState = new OAuthAlertStateStore({
statePath: getFusionOAuthAlertStatePath(),
});
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
authStorage,
notificationService: this.notificationService,
alertState: oauthAlertState,
});
await this.oauthExpiryMonitor.start();
this.oauthValidityLogger = new OAuthValidityLogger({ authStorage });
this.oauthValidityLogger = new OAuthValidityLogger({
authStorage,
alertState: oauthAlertState,
});
await this.oauthValidityLogger.start();
// Backward-compatibility shim for gridlock notifications.