FN-5924: throttle OAuth expiry alerts across restarts
Persist OAuth expiry alert state so repeated provider expiry warnings are suppressed for 12 hours across engine restarts. - add a persisted OAuth alert state store under ~/.fusion/agent and share it between the expiry monitor and startup validity logger - throttle repeated oauth-token-expired notifications and startup expiry warnings per provider for 12 hours, while clearing stale state when providers disappear or change - cover persisted throttling, restart behavior, failure handling, and wiring updates in engine notification tests - document the persisted 12-hour OAuth alert throttle and add a patch changeset for @runfusion/fusion Files changed: .changeset/fn-5924-oauth-alert-throttle.md | 5 + docs/settings-reference.md | 4 +- packages/engine/src/__tests__/project-engine-soft-delete-merge-abort.test.ts | 3 + packages/engine/src/__tests__/project-engine.test.ts | 17 ++- packages/engine/src/__tests__/reliability-interactions/soft-delete-in-flight-abort.test.ts | 3 + packages/engine/src/auth-storage.ts | 6 +- packages/engine/src/notification/__tests__/oauth-alert-state.test.ts | 72 ++++++++++ packages/engine/src/notification/__tests__/oauth-expiry-monitor.test.ts | 148 ++++++++++++++++--- packages/engine/src/notification/__tests__/oauth-validity-logger.test.ts | 159 ++++++++++++++++++--- packages/engine/src/notification/index.ts | 3 + packages/engine/src/notification/oauth-alert-state.ts | 144 +++++++++++++++++++ packages/engine/src/notification/oauth-expiry-monitor.ts | 12 +- packages/engine/src/notification/oauth-validity-logger.ts | 16 ++- packages/engine/src/project-engine.ts | 13 +- 14 files changed, 554 insertions(+), 51 deletions(-) Fusion-Task-Id: FN-5924 Fusion-Task-Lineage: 83255a25-40c7-44d4-8302-b068ae51250e
This commit is contained in:
5
.changeset/fn-5924-oauth-alert-throttle.md
Normal file
5
.changeset/fn-5924-oauth-alert-throttle.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
persist the OAuth expiry alert/notification throttle so users are alerted at most once per provider every 12 hours, even across server restarts.
|
||||
@@ -45,7 +45,7 @@ Defaults from `DEFAULT_GLOBAL_SETTINGS`; key scope from `GLOBAL_SETTINGS_KEYS`.
|
||||
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
|
||||
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
|
||||
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |
|
||||
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
|
||||
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication; Fusion also throttles that notification and the matching startup expiry warning to at most once per provider every 12 hours, and the throttle persists across server restarts. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
|
||||
| `ntfyDashboardHost` | `string` | `undefined` | Dashboard host used to build deep links in notifications. |
|
||||
| `taskTokenBudget` | `{ soft?: number; hard?: number; perSize?: { S?: { soft?: number; hard?: number }; M?: { soft?: number; hard?: number }; L?: { soft?: number; hard?: number } } }` | `undefined` | Global fallback per-task token budget policy. Project `taskTokenBudget` overrides this. |
|
||||
| `webhookEnabled` | `boolean` | `false` | Enable webhook notifications for task lifecycle events. Part of the legacy flat settings; prefer `notificationProviders` for new setups. |
|
||||
@@ -161,7 +161,7 @@ When `id` is `"ntfy"` in `notificationProviders`, the provider `config` supports
|
||||
| `topic` | `string` | _required_ | ntfy topic name (1–64 chars, alphanumeric + `-_`). |
|
||||
| `ntfyBaseUrl` | `string` | `"https://ntfy.sh"` | Optional custom ntfy server URL. |
|
||||
| `ntfyAccessToken` | `string` | `undefined` | Optional access token. When set, provider sends `Authorization: Bearer <token>` on ntfy publishes. |
|
||||
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired. |
|
||||
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired; Fusion suppresses repeat delivery for the same provider for 12 hours even across server restarts, and applies the same persisted window to the startup expiry warning log. |
|
||||
| `dashboardHost` | `string` | `undefined` | Dashboard host for deep links in notifications. |
|
||||
|
||||
Disable daily update checks globally:
|
||||
|
||||
@@ -26,11 +26,14 @@ vi.mock("../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(() =>
|
||||
vi.mock("../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
||||
vi.mock("../auth-storage.js", () => ({
|
||||
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
||||
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||
}));
|
||||
vi.mock("../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
||||
vi.mock("../notification/index.js", () => ({
|
||||
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
}));
|
||||
vi.mock("../cron-runner.js", () => ({
|
||||
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
|
||||
@@ -4,7 +4,7 @@ import { ProjectEngine } from "../project-engine.js";
|
||||
import { runtimeLog } from "../logger.js";
|
||||
import { TunnelProcessManager } from "../remote-access/tunnel-process-manager.js";
|
||||
import { NtfyNotifier } from "../notifier.js";
|
||||
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
|
||||
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
syncInsightExtractionAutomation: vi.fn(),
|
||||
@@ -97,6 +97,7 @@ vi.mock("../notification/index.js", () => ({
|
||||
start: mocks.notificationServiceStart,
|
||||
stop: mocks.notificationServiceStop,
|
||||
})),
|
||||
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({
|
||||
start: mocks.oauthExpiryMonitorStart,
|
||||
stop: mocks.oauthExpiryMonitorStop,
|
||||
@@ -113,6 +114,7 @@ vi.mock("../auth-storage.js", () => ({
|
||||
getOAuthProviders: vi.fn(() => []),
|
||||
get: vi.fn(() => undefined),
|
||||
})),
|
||||
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||
}));
|
||||
|
||||
vi.mock("../runtimes/in-process-runtime.js", () => ({
|
||||
@@ -305,10 +307,19 @@ describe("ProjectEngine notification ownership wiring", () => {
|
||||
await engine.start();
|
||||
|
||||
expect(NotificationService).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
|
||||
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
||||
const notifierCtorArgs = vi.mocked(NtfyNotifier).mock.calls[0];
|
||||
expect(notifierCtorArgs?.[2]).toBe(vi.mocked(NotificationService).mock.results[0]?.value);
|
||||
const alertStateInstance = vi.mocked(OAuthAlertStateStore).mock.results[0]?.value;
|
||||
expect(vi.mocked(OAuthExpiryMonitor).mock.calls[0]?.[0]).toEqual(
|
||||
expect.objectContaining({ alertState: alertStateInstance }),
|
||||
);
|
||||
expect(vi.mocked(OAuthValidityLogger).mock.calls[0]?.[0]).toEqual(
|
||||
expect.objectContaining({ alertState: alertStateInstance }),
|
||||
);
|
||||
|
||||
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
||||
@@ -329,7 +340,9 @@ describe("ProjectEngine notification ownership wiring", () => {
|
||||
// Root cause guard: if ProjectEngine.start is called more than once, it should not
|
||||
// wire a second NotificationService/NtfyNotifier pair for the same store.
|
||||
expect(NotificationService).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
||||
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
|
||||
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
||||
@@ -344,7 +357,9 @@ describe("ProjectEngine notification ownership wiring", () => {
|
||||
await engine.start();
|
||||
|
||||
expect(NotificationService).not.toHaveBeenCalled();
|
||||
expect(OAuthAlertStateStore).not.toHaveBeenCalled();
|
||||
expect(OAuthExpiryMonitor).not.toHaveBeenCalled();
|
||||
expect(OAuthValidityLogger).not.toHaveBeenCalled();
|
||||
expect(NtfyNotifier).not.toHaveBeenCalled();
|
||||
|
||||
await engine.stop();
|
||||
|
||||
@@ -29,11 +29,14 @@ vi.mock("../../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(()
|
||||
vi.mock("../../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
||||
vi.mock("../../auth-storage.js", () => ({
|
||||
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
||||
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||
}));
|
||||
vi.mock("../../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
||||
vi.mock("../../notification/index.js", () => ({
|
||||
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
}));
|
||||
vi.mock("../../cron-runner.js", () => ({
|
||||
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||
|
||||
@@ -16,7 +16,7 @@ import type { OAuthCredentials } from "@earendil-works/pi-ai/oauth";
|
||||
|
||||
type StoredCredential = StoredAuthCredential;
|
||||
|
||||
function getHomeDir(): string {
|
||||
export function getHomeDir(): string {
|
||||
return process.env.HOME || process.env.USERPROFILE || homedir();
|
||||
}
|
||||
|
||||
@@ -24,6 +24,10 @@ export function getFusionAuthPath(home = getHomeDir()): string {
|
||||
return join(home, ".fusion", "agent", "auth.json");
|
||||
}
|
||||
|
||||
export function getFusionOAuthAlertStatePath(home = getHomeDir()): string {
|
||||
return join(home, ".fusion", "agent", "oauth-alert-state.json");
|
||||
}
|
||||
|
||||
export function getFusionModelsPath(home = getHomeDir()): string {
|
||||
return join(home, ".fusion", "agent", "models.json");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createTempStatePath(): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "oauth-alert-state-"));
|
||||
tempDirs.push(dir);
|
||||
return join(dir, "oauth-alert-state.json");
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const dir of tempDirs.splice(0)) {
|
||||
rmSync(dir, { force: true, recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe("OAuthAlertStateStore", () => {
|
||||
it("round-trips provider alert state using the configured path", () => {
|
||||
const statePath = createTempStatePath();
|
||||
const store = new OAuthAlertStateStore({ statePath, clock: () => 1234 });
|
||||
|
||||
store.recordAlert("openai-codex", 9999);
|
||||
|
||||
const reloaded = new OAuthAlertStateStore({ statePath });
|
||||
expect(reloaded.get("openai-codex")).toEqual({ expires: 9999, lastAlertAt: 1234 });
|
||||
});
|
||||
|
||||
it("returns empty state when the file is missing or corrupt", () => {
|
||||
const missingPath = createTempStatePath();
|
||||
const missingStore = new OAuthAlertStateStore({ statePath: missingPath });
|
||||
expect(missingStore.get("openai-codex")).toBeUndefined();
|
||||
|
||||
const corruptPath = createTempStatePath();
|
||||
writeFileSync(corruptPath, "{not json", "utf-8");
|
||||
const corruptStore = new OAuthAlertStateStore({ statePath: corruptPath });
|
||||
expect(corruptStore.get("openai-codex")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("persists only provider ids with expires and lastAlertAt", () => {
|
||||
const statePath = createTempStatePath();
|
||||
const store = new OAuthAlertStateStore({ statePath, clock: () => 5678 });
|
||||
|
||||
store.recordAlert("claude", 4321);
|
||||
|
||||
expect(JSON.parse(readFileSync(statePath, "utf-8"))).toEqual({
|
||||
claude: {
|
||||
expires: 4321,
|
||||
lastAlertAt: 5678,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("clears selected providers and all providers", () => {
|
||||
const statePath = createTempStatePath();
|
||||
const store = new OAuthAlertStateStore({ statePath, clock: () => 100 });
|
||||
|
||||
store.recordAlert("claude", 1_000);
|
||||
store.recordAlert("openai-codex", 2_000, 200);
|
||||
store.clear(["claude"]);
|
||||
|
||||
const afterSingleClear = new OAuthAlertStateStore({ statePath });
|
||||
expect(afterSingleClear.get("claude")).toBeUndefined();
|
||||
expect(afterSingleClear.get("openai-codex")).toEqual({ expires: 2_000, lastAlertAt: 200 });
|
||||
|
||||
afterSingleClear.clear();
|
||||
expect(new OAuthAlertStateStore({ statePath }).get("openai-codex")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,18 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||
import { OAuthExpiryMonitor, type AuthStorageLike } from "../oauth-expiry-monitor.js";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createStatePath(): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "oauth-expiry-monitor-"));
|
||||
tempDirs.push(dir);
|
||||
return join(dir, "oauth-alert-state.json");
|
||||
}
|
||||
|
||||
function createAuthStorage(initialCredential?: { type?: string; expires?: number }): AuthStorageLike & {
|
||||
credential: { type?: string; expires?: number } | undefined;
|
||||
} {
|
||||
@@ -17,17 +29,26 @@ function createAuthStorage(initialCredential?: { type?: string; expires?: number
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
for (const dir of tempDirs.splice(0)) {
|
||||
rmSync(dir, { force: true, recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe("OAuthExpiryMonitor", () => {
|
||||
it("fires once when an OAuth credential is expired", async () => {
|
||||
vi.useFakeTimers();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: Date.now() - 1_000 });
|
||||
const now = Date.now();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1_000 });
|
||||
const dispatch = vi.fn(async () => undefined);
|
||||
|
||||
const monitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
clock: () => Date.now(),
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
@@ -45,7 +66,6 @@ describe("OAuthExpiryMonitor", () => {
|
||||
}),
|
||||
);
|
||||
monitor.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("does not fire for non-expired/non-oauth credentials", async () => {
|
||||
@@ -67,6 +87,7 @@ describe("OAuthExpiryMonitor", () => {
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
@@ -75,7 +96,6 @@ describe("OAuthExpiryMonitor", () => {
|
||||
}
|
||||
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("deduplicates dispatches for same provider and expiry", async () => {
|
||||
@@ -89,6 +109,7 @@ describe("OAuthExpiryMonitor", () => {
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
@@ -96,12 +117,12 @@ describe("OAuthExpiryMonitor", () => {
|
||||
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
monitor.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("re-fires after credential is replaced with a new expiry that later expires", async () => {
|
||||
vi.useFakeTimers();
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||
const dispatch = vi.fn(async () => undefined);
|
||||
|
||||
@@ -110,6 +131,7 @@ describe("OAuthExpiryMonitor", () => {
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
@@ -128,42 +150,126 @@ describe("OAuthExpiryMonitor", () => {
|
||||
|
||||
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||
monitor.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("throttles changed expiries until min notify interval elapses", async () => {
|
||||
it("throttles changed expiries until min notify interval elapses across restarts", async () => {
|
||||
vi.useFakeTimers();
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||
const dispatch = vi.fn(async () => undefined);
|
||||
|
||||
const monitor = new OAuthExpiryMonitor({
|
||||
const firstMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
|
||||
authStorage.credential = { type: "oauth", expires: now + 10_000 };
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
now += 500;
|
||||
authStorage.credential = { type: "oauth", expires: now - 1 };
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
|
||||
await firstMonitor.start();
|
||||
firstMonitor.stop();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
|
||||
now += 500;
|
||||
authStorage.credential = { type: "oauth", expires: now - 2 };
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
const restartedMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
|
||||
await restartedMonitor.start();
|
||||
restartedMonitor.stop();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
|
||||
now += 500;
|
||||
authStorage.credential = { type: "oauth", expires: now - 3 };
|
||||
await restartedMonitor.start();
|
||||
restartedMonitor.stop();
|
||||
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("does not persist lastAlertAt when dispatch fails", async () => {
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||
const dispatch = vi.fn(async () => {
|
||||
throw new Error("boom");
|
||||
});
|
||||
|
||||
const firstMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await firstMonitor.start();
|
||||
firstMonitor.stop();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
|
||||
const secondDispatch = vi.fn(async () => undefined);
|
||||
now += 100;
|
||||
const restartedMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch: secondDispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await restartedMonitor.start();
|
||||
restartedMonitor.stop();
|
||||
|
||||
expect(secondDispatch).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("clears persisted state when providers disappear", async () => {
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||
const dispatch = vi.fn(async () => undefined);
|
||||
|
||||
const firstMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await firstMonitor.start();
|
||||
firstMonitor.stop();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
|
||||
const noProviderStorage: AuthStorageLike = {
|
||||
reload: vi.fn(),
|
||||
getOAuthProviders: () => [],
|
||||
get: () => undefined,
|
||||
};
|
||||
const clearingMonitor = new OAuthExpiryMonitor({
|
||||
authStorage: noProviderStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await clearingMonitor.start();
|
||||
clearingMonitor.stop();
|
||||
|
||||
now += 100;
|
||||
const restartedMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: { dispatch } as any,
|
||||
minNotifyIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await restartedMonitor.start();
|
||||
restartedMonitor.stop();
|
||||
|
||||
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||
monitor.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("stop cancels the interval", async () => {
|
||||
@@ -177,6 +283,7 @@ describe("OAuthExpiryMonitor", () => {
|
||||
notificationService: { dispatch } as any,
|
||||
intervalMs: 100,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
|
||||
await monitor.start();
|
||||
@@ -184,6 +291,5 @@ describe("OAuthExpiryMonitor", () => {
|
||||
await vi.advanceTimersByTimeAsync(500);
|
||||
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
vi.useRealTimers();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,7 +1,19 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||
import { OAuthValidityLogger } from "../oauth-validity-logger.js";
|
||||
import type { AuthStorageLike } from "../oauth-expiry-monitor.js";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function createStatePath(): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), "oauth-validity-logger-"));
|
||||
tempDirs.push(dir);
|
||||
return join(dir, "oauth-alert-state.json");
|
||||
}
|
||||
|
||||
function createAuthStorage(providers: Array<{ id: string; name: string }>, credentials: Record<string, any>): AuthStorageLike {
|
||||
return {
|
||||
reload: vi.fn(),
|
||||
@@ -10,6 +22,13 @@ function createAuthStorage(providers: Array<{ id: string; name: string }>, crede
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
for (const dir of tempDirs.splice(0)) {
|
||||
rmSync(dir, { force: true, recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe("OAuthValidityLogger", () => {
|
||||
it("logs one line per expired oauth credential on start", async () => {
|
||||
vi.useFakeTimers();
|
||||
@@ -26,30 +45,79 @@ describe("OAuthValidityLogger", () => {
|
||||
},
|
||||
);
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
await validityLogger.start();
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(2);
|
||||
validityLogger.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("logs again on interval without dedupe", async () => {
|
||||
it("skips repeated logs within the throttle window", async () => {
|
||||
vi.useFakeTimers();
|
||||
const now = Date.now();
|
||||
const logger = vi.fn();
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage(
|
||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||
);
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
await validityLogger.start();
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 100,
|
||||
minAlertIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(2);
|
||||
await validityLogger.start();
|
||||
now += 500;
|
||||
await validityLogger.check();
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
validityLogger.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("persists the throttle across a restart and logs again after the window elapses", async () => {
|
||||
vi.useFakeTimers();
|
||||
const logger = vi.fn();
|
||||
let now = Date.now();
|
||||
const statePath = createStatePath();
|
||||
const authStorage = createAuthStorage(
|
||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||
);
|
||||
|
||||
const firstLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
minAlertIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await firstLogger.check();
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
|
||||
const restartedLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
minAlertIntervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||
});
|
||||
await restartedLogger.check();
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
|
||||
now += 1_001;
|
||||
await restartedLogger.check();
|
||||
expect(logger).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("does not log for valid oauth, api key, or missing expires", async () => {
|
||||
@@ -69,30 +137,42 @@ describe("OAuthValidityLogger", () => {
|
||||
},
|
||||
);
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
await validityLogger.start();
|
||||
|
||||
expect(logger).not.toHaveBeenCalled();
|
||||
validityLogger.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("stop cancels the interval", async () => {
|
||||
vi.useFakeTimers();
|
||||
const now = Date.now();
|
||||
let now = Date.now();
|
||||
const logger = vi.fn();
|
||||
const authStorage = createAuthStorage(
|
||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||
);
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 1_000,
|
||||
minAlertIntervalMs: 500,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
await validityLogger.start();
|
||||
validityLogger.stop();
|
||||
now += 5_000;
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("continues iterating when one provider throws", async () => {
|
||||
@@ -113,7 +193,13 @@ describe("OAuthValidityLogger", () => {
|
||||
},
|
||||
};
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
await validityLogger.start();
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
@@ -122,7 +208,6 @@ describe("OAuthValidityLogger", () => {
|
||||
expect.objectContaining({ providerId: "claude" }),
|
||||
);
|
||||
validityLogger.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("never includes token material in log metadata", async () => {
|
||||
@@ -141,12 +226,50 @@ describe("OAuthValidityLogger", () => {
|
||||
},
|
||||
);
|
||||
|
||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
intervalMs: 1_000,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||
});
|
||||
await validityLogger.start();
|
||||
|
||||
const [, meta] = logger.mock.calls[0] ?? [];
|
||||
expect(Object.keys(meta ?? {}).sort()).toEqual(["expiresAt", "providerId", "providerName"]);
|
||||
validityLogger.stop();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("covers empty, undefined, and populated provider states", async () => {
|
||||
const logger = vi.fn();
|
||||
const now = Date.now();
|
||||
const cases: AuthStorageLike[] = [
|
||||
{
|
||||
reload: vi.fn(),
|
||||
getOAuthProviders: () => [],
|
||||
get: () => undefined,
|
||||
},
|
||||
{
|
||||
reload: vi.fn(),
|
||||
getOAuthProviders: () => [{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||
get: () => undefined,
|
||||
},
|
||||
createAuthStorage(
|
||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||
{ "openai-codex": { type: "oauth", expires: now - 1 } },
|
||||
),
|
||||
];
|
||||
|
||||
for (const [index, authStorage] of cases.entries()) {
|
||||
const validityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
logger,
|
||||
clock: () => now,
|
||||
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now + index }),
|
||||
});
|
||||
await validityLogger.check();
|
||||
}
|
||||
|
||||
expect(logger).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -7,6 +7,9 @@ export type { WebhookProviderConfig } from "./webhook-provider.js";
|
||||
export { NotificationService } from "./notification-service.js";
|
||||
export type { NotificationServiceOptions } from "./notification-service.js";
|
||||
|
||||
export { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||
export type { OAuthAlertStateEntry, OAuthAlertStateFs, OAuthAlertStateStoreOptions } from "./oauth-alert-state.js";
|
||||
|
||||
export { OAuthExpiryMonitor } from "./oauth-expiry-monitor.js";
|
||||
export type { AuthStorageLike as OAuthExpiryAuthStorageLike, OAuthExpiryMonitorOptions } from "./oauth-expiry-monitor.js";
|
||||
|
||||
|
||||
144
packages/engine/src/notification/oauth-alert-state.ts
Normal file
144
packages/engine/src/notification/oauth-alert-state.ts
Normal file
@@ -0,0 +1,144 @@
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { dirname } from "node:path";
|
||||
import { getFusionOAuthAlertStatePath } from "../auth-storage.js";
|
||||
|
||||
export interface OAuthAlertStateEntry {
|
||||
expires: number;
|
||||
lastAlertAt: number;
|
||||
}
|
||||
|
||||
export interface OAuthAlertStateFs {
|
||||
existsSync(path: string): boolean;
|
||||
mkdirSync(path: string, options?: { recursive?: boolean }): void;
|
||||
readFileSync(path: string, encoding: BufferEncoding): string;
|
||||
renameSync(oldPath: string, newPath: string): void;
|
||||
rmSync(path: string, options?: { force?: boolean }): void;
|
||||
writeFileSync(path: string, content: string, encoding: BufferEncoding): void;
|
||||
}
|
||||
|
||||
export interface OAuthAlertStateStoreOptions {
|
||||
statePath?: string;
|
||||
clock?: () => number;
|
||||
fs?: OAuthAlertStateFs;
|
||||
}
|
||||
|
||||
export class OAuthAlertStateStore {
|
||||
private readonly statePath: string;
|
||||
private readonly clock: () => number;
|
||||
private readonly fs: OAuthAlertStateFs;
|
||||
|
||||
constructor(options: OAuthAlertStateStoreOptions = {}) {
|
||||
this.statePath = options.statePath ?? getFusionOAuthAlertStatePath();
|
||||
this.clock = options.clock ?? Date.now;
|
||||
this.fs = options.fs ?? {
|
||||
existsSync: (path) => existsSync(path),
|
||||
mkdirSync: (path, options) => {
|
||||
mkdirSync(path, options);
|
||||
},
|
||||
readFileSync: (path, encoding) => readFileSync(path, encoding),
|
||||
renameSync: (oldPath, newPath) => {
|
||||
renameSync(oldPath, newPath);
|
||||
},
|
||||
rmSync: (path, options) => {
|
||||
rmSync(path, options);
|
||||
},
|
||||
writeFileSync: (path, content, encoding) => {
|
||||
writeFileSync(path, content, encoding);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
get(providerId: string): OAuthAlertStateEntry | undefined {
|
||||
return this.readState()[providerId];
|
||||
}
|
||||
|
||||
getLastAlertAt(providerId: string): number | undefined {
|
||||
return this.get(providerId)?.lastAlertAt;
|
||||
}
|
||||
|
||||
recordAlert(providerId: string, expires: number, lastAlertAt = this.clock()): void {
|
||||
const state = this.readState();
|
||||
state[providerId] = { expires, lastAlertAt };
|
||||
this.writeState(state);
|
||||
}
|
||||
|
||||
clear(providerIds?: Iterable<string>): void {
|
||||
if (!providerIds) {
|
||||
this.writeState({});
|
||||
return;
|
||||
}
|
||||
|
||||
const state = this.readState();
|
||||
let changed = false;
|
||||
for (const providerId of providerIds) {
|
||||
if (!(providerId in state)) {
|
||||
continue;
|
||||
}
|
||||
delete state[providerId];
|
||||
changed = true;
|
||||
}
|
||||
if (changed) {
|
||||
this.writeState(state);
|
||||
}
|
||||
}
|
||||
|
||||
private readState(): Record<string, OAuthAlertStateEntry> {
|
||||
if (!this.fs.existsSync(this.statePath)) {
|
||||
return {};
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(this.fs.readFileSync(this.statePath, "utf-8")) as unknown;
|
||||
return sanitizeState(parsed);
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
private writeState(state: Record<string, OAuthAlertStateEntry>): void {
|
||||
const sanitized = sanitizeState(state);
|
||||
const dir = dirname(this.statePath);
|
||||
this.fs.mkdirSync(dir, { recursive: true });
|
||||
|
||||
const tempPath = `${this.statePath}.${process.pid}.${this.clock()}.tmp`;
|
||||
const body = `${JSON.stringify(sanitized, null, 2)}\n`;
|
||||
this.fs.writeFileSync(tempPath, body, "utf-8");
|
||||
try {
|
||||
this.fs.renameSync(tempPath, this.statePath);
|
||||
} catch (error) {
|
||||
this.fs.rmSync(tempPath, { force: true });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function sanitizeState(parsed: unknown): Record<string, OAuthAlertStateEntry> {
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const sanitized: Record<string, OAuthAlertStateEntry> = {};
|
||||
for (const [providerId, value] of Object.entries(parsed)) {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
||||
continue;
|
||||
}
|
||||
const expires = (value as { expires?: unknown }).expires;
|
||||
const lastAlertAt = (value as { lastAlertAt?: unknown }).lastAlertAt;
|
||||
if (typeof expires !== "number" || Number.isNaN(expires)) {
|
||||
continue;
|
||||
}
|
||||
if (typeof lastAlertAt !== "number" || Number.isNaN(lastAlertAt)) {
|
||||
continue;
|
||||
}
|
||||
sanitized[providerId] = { expires, lastAlertAt };
|
||||
}
|
||||
|
||||
return sanitized;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import type { NotificationPayload } from "@fusion/core";
|
||||
import { schedulerLog } from "../logger.js";
|
||||
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||
import type { NotificationService } from "./notification-service.js";
|
||||
|
||||
const DEFAULT_INTERVAL_MS = 5 * 60_000;
|
||||
@@ -28,6 +29,7 @@ export interface OAuthExpiryMonitorOptions {
|
||||
clock?: () => number;
|
||||
warnBeforeMs?: number;
|
||||
minNotifyIntervalMs?: number;
|
||||
alertState?: OAuthAlertStateStore;
|
||||
}
|
||||
|
||||
export class OAuthExpiryMonitor {
|
||||
@@ -35,15 +37,16 @@ export class OAuthExpiryMonitor {
|
||||
private readonly clock: () => number;
|
||||
private readonly warnBeforeMs: number;
|
||||
private readonly minNotifyIntervalMs: number;
|
||||
private readonly alertState: OAuthAlertStateStore;
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
private readonly dispatchedExpiryKeys = new Set<string>();
|
||||
private readonly lastNotifiedAt = new Map<string, number>();
|
||||
|
||||
constructor(private readonly opts: OAuthExpiryMonitorOptions) {
|
||||
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
||||
this.clock = opts.clock ?? Date.now;
|
||||
this.warnBeforeMs = opts.warnBeforeMs ?? 0;
|
||||
this.minNotifyIntervalMs = opts.minNotifyIntervalMs ?? DEFAULT_MIN_NOTIFY_INTERVAL_MS;
|
||||
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
|
||||
}
|
||||
|
||||
async start(): Promise<void> {
|
||||
@@ -73,7 +76,7 @@ export class OAuthExpiryMonitor {
|
||||
const providers = this.opts.authStorage.getOAuthProviders?.();
|
||||
if (!providers?.length) {
|
||||
this.dispatchedExpiryKeys.clear();
|
||||
this.lastNotifiedAt.clear();
|
||||
this.alertState.clear();
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -83,6 +86,7 @@ export class OAuthExpiryMonitor {
|
||||
for (const provider of providers) {
|
||||
const credential = this.opts.authStorage.get?.(provider.id);
|
||||
if (credential?.type !== "oauth" || typeof credential.expires !== "number") {
|
||||
this.alertState.clear([provider.id]);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -96,7 +100,7 @@ export class OAuthExpiryMonitor {
|
||||
continue;
|
||||
}
|
||||
|
||||
const previousNotificationAt = this.lastNotifiedAt.get(provider.id);
|
||||
const previousNotificationAt = this.alertState.getLastAlertAt(provider.id);
|
||||
if (
|
||||
typeof previousNotificationAt === "number" &&
|
||||
now - previousNotificationAt < this.minNotifyIntervalMs
|
||||
@@ -116,7 +120,7 @@ export class OAuthExpiryMonitor {
|
||||
try {
|
||||
await this.opts.notificationService.dispatch("oauth-token-expired", payload);
|
||||
this.dispatchedExpiryKeys.add(expiryKey);
|
||||
this.lastNotifiedAt.set(provider.id, now);
|
||||
this.alertState.recordAlert(provider.id, credential.expires, now);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
schedulerLog.warn(`OAuth expiry notification dispatch failed provider=${provider.id}: ${message}`);
|
||||
|
||||
@@ -1,25 +1,33 @@
|
||||
import { schedulerLog } from "../logger.js";
|
||||
import type { AuthStorageLike } from "./oauth-expiry-monitor.js";
|
||||
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||
|
||||
const DEFAULT_INTERVAL_MS = 24 * 60 * 60 * 1000;
|
||||
const DEFAULT_INTERVAL_MS = 12 * 60 * 60 * 1000;
|
||||
const DEFAULT_MIN_ALERT_INTERVAL_MS = 12 * 60 * 60 * 1000;
|
||||
|
||||
interface OAuthValidityLoggerOptions {
|
||||
authStorage: AuthStorageLike;
|
||||
intervalMs?: number;
|
||||
clock?: () => number;
|
||||
logger?: (msg: string, meta?: Record<string, unknown>) => void;
|
||||
alertState?: OAuthAlertStateStore;
|
||||
minAlertIntervalMs?: number;
|
||||
}
|
||||
|
||||
export class OAuthValidityLogger {
|
||||
private readonly intervalMs: number;
|
||||
private readonly clock: () => number;
|
||||
private readonly logger: (msg: string, meta?: Record<string, unknown>) => void;
|
||||
private readonly alertState: OAuthAlertStateStore;
|
||||
private readonly minAlertIntervalMs: number;
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
|
||||
constructor(private readonly opts: OAuthValidityLoggerOptions) {
|
||||
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
||||
this.clock = opts.clock ?? Date.now;
|
||||
this.logger = opts.logger ?? ((message, meta) => schedulerLog.warn(message, meta));
|
||||
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
|
||||
this.minAlertIntervalMs = opts.minAlertIntervalMs ?? DEFAULT_MIN_ALERT_INTERVAL_MS;
|
||||
}
|
||||
|
||||
async start(): Promise<void> {
|
||||
@@ -57,11 +65,17 @@ export class OAuthValidityLogger {
|
||||
continue;
|
||||
}
|
||||
|
||||
const previousAlertAt = this.alertState.getLastAlertAt(provider.id);
|
||||
if (typeof previousAlertAt === "number" && now - previousAlertAt < this.minAlertIntervalMs) {
|
||||
continue;
|
||||
}
|
||||
|
||||
this.logger("oauth credential expired — provider re-login required", {
|
||||
providerId: provider.id,
|
||||
providerName: provider.name,
|
||||
expiresAt: new Date(credential.expires).toISOString(),
|
||||
});
|
||||
this.alertState.recordAlert(provider.id, credential.expires, now);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
schedulerLog.warn(`OAuth validity logger failed for provider=${provider.id}: ${message}`);
|
||||
|
||||
@@ -19,10 +19,10 @@ import type { ProjectRuntimeConfig } from "./project-runtime.js";
|
||||
import { PrMonitor } from "./pr-monitor.js";
|
||||
import { PrCommentHandler } from "./pr-comment-handler.js";
|
||||
import { NtfyNotifier } from "./notifier.js";
|
||||
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
|
||||
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
|
||||
import type { NotificationChatStore } from "./notification/notification-service.js";
|
||||
import { GridlockDetector } from "./gridlock-detector.js";
|
||||
import { createFusionAuthStorage } from "./auth-storage.js";
|
||||
import { createFusionAuthStorage, getFusionOAuthAlertStatePath } from "./auth-storage.js";
|
||||
import { CronRunner, createAiPromptExecutor } from "./cron-runner.js";
|
||||
import type { RoutineRunner } from "./routine-runner.js";
|
||||
import { aiMergeTask, sweepStaleAutostashes, VerificationError } from "./merger.js";
|
||||
@@ -427,12 +427,19 @@ export class ProjectEngine {
|
||||
});
|
||||
await this.notificationService.start();
|
||||
const authStorage = createFusionAuthStorage();
|
||||
const oauthAlertState = new OAuthAlertStateStore({
|
||||
statePath: getFusionOAuthAlertStatePath(),
|
||||
});
|
||||
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
|
||||
authStorage,
|
||||
notificationService: this.notificationService,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthExpiryMonitor.start();
|
||||
this.oauthValidityLogger = new OAuthValidityLogger({ authStorage });
|
||||
this.oauthValidityLogger = new OAuthValidityLogger({
|
||||
authStorage,
|
||||
alertState: oauthAlertState,
|
||||
});
|
||||
await this.oauthValidityLogger.start();
|
||||
|
||||
// Backward-compatibility shim for gridlock notifications.
|
||||
|
||||
Reference in New Issue
Block a user