FN-5924: throttle OAuth expiry alerts across restarts
Persist OAuth expiry alert state so repeated provider expiry warnings are suppressed for 12 hours across engine restarts. - add a persisted OAuth alert state store under ~/.fusion/agent and share it between the expiry monitor and startup validity logger - throttle repeated oauth-token-expired notifications and startup expiry warnings per provider for 12 hours, while clearing stale state when providers disappear or change - cover persisted throttling, restart behavior, failure handling, and wiring updates in engine notification tests - document the persisted 12-hour OAuth alert throttle and add a patch changeset for @runfusion/fusion Files changed: .changeset/fn-5924-oauth-alert-throttle.md | 5 + docs/settings-reference.md | 4 +- packages/engine/src/__tests__/project-engine-soft-delete-merge-abort.test.ts | 3 + packages/engine/src/__tests__/project-engine.test.ts | 17 ++- packages/engine/src/__tests__/reliability-interactions/soft-delete-in-flight-abort.test.ts | 3 + packages/engine/src/auth-storage.ts | 6 +- packages/engine/src/notification/__tests__/oauth-alert-state.test.ts | 72 ++++++++++ packages/engine/src/notification/__tests__/oauth-expiry-monitor.test.ts | 148 ++++++++++++++++--- packages/engine/src/notification/__tests__/oauth-validity-logger.test.ts | 159 ++++++++++++++++++--- packages/engine/src/notification/index.ts | 3 + packages/engine/src/notification/oauth-alert-state.ts | 144 +++++++++++++++++++ packages/engine/src/notification/oauth-expiry-monitor.ts | 12 +- packages/engine/src/notification/oauth-validity-logger.ts | 16 ++- packages/engine/src/project-engine.ts | 13 +- 14 files changed, 554 insertions(+), 51 deletions(-) Fusion-Task-Id: FN-5924 Fusion-Task-Lineage: 83255a25-40c7-44d4-8302-b068ae51250e
This commit is contained in:
5
.changeset/fn-5924-oauth-alert-throttle.md
Normal file
5
.changeset/fn-5924-oauth-alert-throttle.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
persist the OAuth expiry alert/notification throttle so users are alerted at most once per provider every 12 hours, even across server restarts.
|
||||||
@@ -45,7 +45,7 @@ Defaults from `DEFAULT_GLOBAL_SETTINGS`; key scope from `GLOBAL_SETTINGS_KEYS`.
|
|||||||
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
|
| `ntfyTopic` | `string` | `undefined` | ntfy topic name. |
|
||||||
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
|
| `ntfyBaseUrl` | `string` | `undefined` | Optional custom ntfy server base URL (must use `http://` or `https://`). If blank/unset, Fusion uses `https://ntfy.sh` for both runtime and test notifications. |
|
||||||
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |
|
| `ntfyAccessToken` | `string` | `undefined` | Optional ntfy access token. When set, Fusion sends `Authorization: Bearer <token>` with ntfy publish requests, including Settings → Notifications test sends. Leave blank/unset to publish without authentication. |
|
||||||
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
|
| `ntfyEvents` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired" \| "token-budget")[]` | `["in-review","merged","failed","awaiting-approval","awaiting-user-review","planning-awaiting-input","gridlock","board-stall-unrecovered","fallback-used","memory-dreams-processed","message:agent-to-user","message:agent-to-agent","message:room","oauth-token-expired","token-budget"]` | Event types that trigger ntfy notifications. `planning-awaiting-input` fires when planning mode is waiting on user input. `gridlock` fires when all schedulable todo tasks are blocked; delivery is cooldown-throttled (first alert immediately, then suppressed for 15 minutes until gridlock resolves). `board-stall-unrecovered` fires only after a board-stall auto-recovery sweep runs and a follow-up verification tick still sees zero progress. `fallback-used` fires when Fusion recovers from a retryable model failure by switching to a configured fallback model. `task-created` fires when an agent creates a new task (requires `sourceAgentId`) and is opt-in/off by default. `memory-dreams-processed` fires when manual dream processing writes a new `DREAMS.md` entry (project and/or agent); disable it via ntfy/webhook event filters if you want to opt out. `message:agent-to-user` fires when an agent sends a direct message to the user. `message:agent-to-agent` fires when an agent sends a message to another agent (including replies). `message:room` fires when an agent posts an assistant reply in a chat room. `oauth-token-expired` fires when a provider OAuth credential reaches its expiry and needs re-authentication; Fusion also throttles that notification and the matching startup expiry warning to at most once per provider every 12 hours, and the throttle persists across server restarts. `token-budget` fires when a task crosses token soft/hard caps. If you use a custom `ntfyEvents` list, these message events must be present (or `ntfyEvents` must be unset so defaults apply) for the corresponding notifications to send. |
|
||||||
| `ntfyDashboardHost` | `string` | `undefined` | Dashboard host used to build deep links in notifications. |
|
| `ntfyDashboardHost` | `string` | `undefined` | Dashboard host used to build deep links in notifications. |
|
||||||
| `taskTokenBudget` | `{ soft?: number; hard?: number; perSize?: { S?: { soft?: number; hard?: number }; M?: { soft?: number; hard?: number }; L?: { soft?: number; hard?: number } } }` | `undefined` | Global fallback per-task token budget policy. Project `taskTokenBudget` overrides this. |
|
| `taskTokenBudget` | `{ soft?: number; hard?: number; perSize?: { S?: { soft?: number; hard?: number }; M?: { soft?: number; hard?: number }; L?: { soft?: number; hard?: number } } }` | `undefined` | Global fallback per-task token budget policy. Project `taskTokenBudget` overrides this. |
|
||||||
| `webhookEnabled` | `boolean` | `false` | Enable webhook notifications for task lifecycle events. Part of the legacy flat settings; prefer `notificationProviders` for new setups. |
|
| `webhookEnabled` | `boolean` | `false` | Enable webhook notifications for task lifecycle events. Part of the legacy flat settings; prefer `notificationProviders` for new setups. |
|
||||||
@@ -161,7 +161,7 @@ When `id` is `"ntfy"` in `notificationProviders`, the provider `config` supports
|
|||||||
| `topic` | `string` | _required_ | ntfy topic name (1–64 chars, alphanumeric + `-_`). |
|
| `topic` | `string` | _required_ | ntfy topic name (1–64 chars, alphanumeric + `-_`). |
|
||||||
| `ntfyBaseUrl` | `string` | `"https://ntfy.sh"` | Optional custom ntfy server URL. |
|
| `ntfyBaseUrl` | `string` | `"https://ntfy.sh"` | Optional custom ntfy server URL. |
|
||||||
| `ntfyAccessToken` | `string` | `undefined` | Optional access token. When set, provider sends `Authorization: Bearer <token>` on ntfy publishes. |
|
| `ntfyAccessToken` | `string` | `undefined` | Optional access token. When set, provider sends `Authorization: Bearer <token>` on ntfy publishes. |
|
||||||
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired. |
|
| `events` | `("in-review" \| "merged" \| "failed" \| "awaiting-approval" \| "awaiting-user-review" \| "planning-awaiting-input" \| "gridlock" \| "board-stall-unrecovered" \| "fallback-used" \| "task-created" \| "memory-dreams-processed" \| "message:agent-to-user" \| "message:agent-to-agent" \| "message:room" \| "oauth-token-expired")[]` | `DEFAULT_NTFY_EVENTS` | Event filter list used by the provider. For `gridlock`, enabled events are still cooldown-throttled at runtime (15-minute suppression window, reset on full resolution). `board-stall-unrecovered` is emitted when board-stall verification fails after an attempted auto-recovery sweep. `task-created` is available as an opt-in event and only fires for agent-created tasks (`sourceAgentId` required). `memory-dreams-processed` is emitted when manual dream processing appends a new project/agent `DREAMS.md` entry. `message:agent-to-user`/`message:agent-to-agent` are emitted for mailbox messages and deep-link to the specific message when `dashboardHost` is configured. `message:room` is emitted for assistant replies in chat rooms and deep-links to the room when `dashboardHost` is configured. `oauth-token-expired` is emitted when a provider OAuth credential has expired; Fusion suppresses repeat delivery for the same provider for 12 hours even across server restarts, and applies the same persisted window to the startup expiry warning log. |
|
||||||
| `dashboardHost` | `string` | `undefined` | Dashboard host for deep links in notifications. |
|
| `dashboardHost` | `string` | `undefined` | Dashboard host for deep links in notifications. |
|
||||||
|
|
||||||
Disable daily update checks globally:
|
Disable daily update checks globally:
|
||||||
|
|||||||
@@ -26,11 +26,14 @@ vi.mock("../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(() =>
|
|||||||
vi.mock("../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
vi.mock("../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
||||||
vi.mock("../auth-storage.js", () => ({
|
vi.mock("../auth-storage.js", () => ({
|
||||||
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
||||||
|
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||||
}));
|
}));
|
||||||
vi.mock("../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
vi.mock("../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
||||||
vi.mock("../notification/index.js", () => ({
|
vi.mock("../notification/index.js", () => ({
|
||||||
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
}));
|
}));
|
||||||
vi.mock("../cron-runner.js", () => ({
|
vi.mock("../cron-runner.js", () => ({
|
||||||
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { ProjectEngine } from "../project-engine.js";
|
|||||||
import { runtimeLog } from "../logger.js";
|
import { runtimeLog } from "../logger.js";
|
||||||
import { TunnelProcessManager } from "../remote-access/tunnel-process-manager.js";
|
import { TunnelProcessManager } from "../remote-access/tunnel-process-manager.js";
|
||||||
import { NtfyNotifier } from "../notifier.js";
|
import { NtfyNotifier } from "../notifier.js";
|
||||||
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
|
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "../notification/index.js";
|
||||||
|
|
||||||
const mocks = vi.hoisted(() => ({
|
const mocks = vi.hoisted(() => ({
|
||||||
syncInsightExtractionAutomation: vi.fn(),
|
syncInsightExtractionAutomation: vi.fn(),
|
||||||
@@ -97,6 +97,7 @@ vi.mock("../notification/index.js", () => ({
|
|||||||
start: mocks.notificationServiceStart,
|
start: mocks.notificationServiceStart,
|
||||||
stop: mocks.notificationServiceStop,
|
stop: mocks.notificationServiceStop,
|
||||||
})),
|
})),
|
||||||
|
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({
|
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({
|
||||||
start: mocks.oauthExpiryMonitorStart,
|
start: mocks.oauthExpiryMonitorStart,
|
||||||
stop: mocks.oauthExpiryMonitorStop,
|
stop: mocks.oauthExpiryMonitorStop,
|
||||||
@@ -113,6 +114,7 @@ vi.mock("../auth-storage.js", () => ({
|
|||||||
getOAuthProviders: vi.fn(() => []),
|
getOAuthProviders: vi.fn(() => []),
|
||||||
get: vi.fn(() => undefined),
|
get: vi.fn(() => undefined),
|
||||||
})),
|
})),
|
||||||
|
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
vi.mock("../runtimes/in-process-runtime.js", () => ({
|
vi.mock("../runtimes/in-process-runtime.js", () => ({
|
||||||
@@ -305,10 +307,19 @@ describe("ProjectEngine notification ownership wiring", () => {
|
|||||||
await engine.start();
|
await engine.start();
|
||||||
|
|
||||||
expect(NotificationService).toHaveBeenCalledTimes(1);
|
expect(NotificationService).toHaveBeenCalledTimes(1);
|
||||||
|
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
|
||||||
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
||||||
|
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
|
||||||
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
||||||
const notifierCtorArgs = vi.mocked(NtfyNotifier).mock.calls[0];
|
const notifierCtorArgs = vi.mocked(NtfyNotifier).mock.calls[0];
|
||||||
expect(notifierCtorArgs?.[2]).toBe(vi.mocked(NotificationService).mock.results[0]?.value);
|
expect(notifierCtorArgs?.[2]).toBe(vi.mocked(NotificationService).mock.results[0]?.value);
|
||||||
|
const alertStateInstance = vi.mocked(OAuthAlertStateStore).mock.results[0]?.value;
|
||||||
|
expect(vi.mocked(OAuthExpiryMonitor).mock.calls[0]?.[0]).toEqual(
|
||||||
|
expect.objectContaining({ alertState: alertStateInstance }),
|
||||||
|
);
|
||||||
|
expect(vi.mocked(OAuthValidityLogger).mock.calls[0]?.[0]).toEqual(
|
||||||
|
expect.objectContaining({ alertState: alertStateInstance }),
|
||||||
|
);
|
||||||
|
|
||||||
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
||||||
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
||||||
@@ -329,7 +340,9 @@ describe("ProjectEngine notification ownership wiring", () => {
|
|||||||
// Root cause guard: if ProjectEngine.start is called more than once, it should not
|
// Root cause guard: if ProjectEngine.start is called more than once, it should not
|
||||||
// wire a second NotificationService/NtfyNotifier pair for the same store.
|
// wire a second NotificationService/NtfyNotifier pair for the same store.
|
||||||
expect(NotificationService).toHaveBeenCalledTimes(1);
|
expect(NotificationService).toHaveBeenCalledTimes(1);
|
||||||
|
expect(OAuthAlertStateStore).toHaveBeenCalledTimes(1);
|
||||||
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
expect(OAuthExpiryMonitor).toHaveBeenCalledTimes(1);
|
||||||
|
expect(OAuthValidityLogger).toHaveBeenCalledTimes(1);
|
||||||
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
expect(NtfyNotifier).toHaveBeenCalledTimes(1);
|
||||||
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
expect(mocks.notificationServiceStart).toHaveBeenCalledTimes(1);
|
||||||
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
expect(mocks.oauthExpiryMonitorStart).toHaveBeenCalledTimes(1);
|
||||||
@@ -344,7 +357,9 @@ describe("ProjectEngine notification ownership wiring", () => {
|
|||||||
await engine.start();
|
await engine.start();
|
||||||
|
|
||||||
expect(NotificationService).not.toHaveBeenCalled();
|
expect(NotificationService).not.toHaveBeenCalled();
|
||||||
|
expect(OAuthAlertStateStore).not.toHaveBeenCalled();
|
||||||
expect(OAuthExpiryMonitor).not.toHaveBeenCalled();
|
expect(OAuthExpiryMonitor).not.toHaveBeenCalled();
|
||||||
|
expect(OAuthValidityLogger).not.toHaveBeenCalled();
|
||||||
expect(NtfyNotifier).not.toHaveBeenCalled();
|
expect(NtfyNotifier).not.toHaveBeenCalled();
|
||||||
|
|
||||||
await engine.stop();
|
await engine.stop();
|
||||||
|
|||||||
@@ -29,11 +29,14 @@ vi.mock("../../pr-monitor.js", () => ({ PrMonitor: vi.fn().mockImplementation(()
|
|||||||
vi.mock("../../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
vi.mock("../../pr-comment-handler.js", () => ({ PrCommentHandler: vi.fn().mockImplementation(() => ({ handleNewComments: vi.fn() })) }));
|
||||||
vi.mock("../../auth-storage.js", () => ({
|
vi.mock("../../auth-storage.js", () => ({
|
||||||
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
createFusionAuthStorage: vi.fn(() => ({ reload: vi.fn(), getOAuthProviders: vi.fn(() => []), get: vi.fn(() => undefined) })),
|
||||||
|
getFusionOAuthAlertStatePath: vi.fn(() => "/tmp/oauth-alert-state.json"),
|
||||||
}));
|
}));
|
||||||
vi.mock("../../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
vi.mock("../../notifier.js", () => ({ NtfyNotifier: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })) }));
|
||||||
vi.mock("../../notification/index.js", () => ({
|
vi.mock("../../notification/index.js", () => ({
|
||||||
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
NotificationService: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
OAuthAlertStateStore: vi.fn().mockImplementation(() => ({})),
|
||||||
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
OAuthExpiryMonitor: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
OAuthValidityLogger: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
}));
|
}));
|
||||||
vi.mock("../../cron-runner.js", () => ({
|
vi.mock("../../cron-runner.js", () => ({
|
||||||
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
CronRunner: vi.fn().mockImplementation(() => ({ start: vi.fn(), stop: vi.fn() })),
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ import type { OAuthCredentials } from "@earendil-works/pi-ai/oauth";
|
|||||||
|
|
||||||
type StoredCredential = StoredAuthCredential;
|
type StoredCredential = StoredAuthCredential;
|
||||||
|
|
||||||
function getHomeDir(): string {
|
export function getHomeDir(): string {
|
||||||
return process.env.HOME || process.env.USERPROFILE || homedir();
|
return process.env.HOME || process.env.USERPROFILE || homedir();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -24,6 +24,10 @@ export function getFusionAuthPath(home = getHomeDir()): string {
|
|||||||
return join(home, ".fusion", "agent", "auth.json");
|
return join(home, ".fusion", "agent", "auth.json");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function getFusionOAuthAlertStatePath(home = getHomeDir()): string {
|
||||||
|
return join(home, ".fusion", "agent", "oauth-alert-state.json");
|
||||||
|
}
|
||||||
|
|
||||||
export function getFusionModelsPath(home = getHomeDir()): string {
|
export function getFusionModelsPath(home = getHomeDir()): string {
|
||||||
return join(home, ".fusion", "agent", "models.json");
|
return join(home, ".fusion", "agent", "models.json");
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, describe, expect, it } from "vitest";
|
||||||
|
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||||
|
|
||||||
|
const tempDirs: string[] = [];
|
||||||
|
|
||||||
|
function createTempStatePath(): string {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "oauth-alert-state-"));
|
||||||
|
tempDirs.push(dir);
|
||||||
|
return join(dir, "oauth-alert-state.json");
|
||||||
|
}
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
for (const dir of tempDirs.splice(0)) {
|
||||||
|
rmSync(dir, { force: true, recursive: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("OAuthAlertStateStore", () => {
|
||||||
|
it("round-trips provider alert state using the configured path", () => {
|
||||||
|
const statePath = createTempStatePath();
|
||||||
|
const store = new OAuthAlertStateStore({ statePath, clock: () => 1234 });
|
||||||
|
|
||||||
|
store.recordAlert("openai-codex", 9999);
|
||||||
|
|
||||||
|
const reloaded = new OAuthAlertStateStore({ statePath });
|
||||||
|
expect(reloaded.get("openai-codex")).toEqual({ expires: 9999, lastAlertAt: 1234 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns empty state when the file is missing or corrupt", () => {
|
||||||
|
const missingPath = createTempStatePath();
|
||||||
|
const missingStore = new OAuthAlertStateStore({ statePath: missingPath });
|
||||||
|
expect(missingStore.get("openai-codex")).toBeUndefined();
|
||||||
|
|
||||||
|
const corruptPath = createTempStatePath();
|
||||||
|
writeFileSync(corruptPath, "{not json", "utf-8");
|
||||||
|
const corruptStore = new OAuthAlertStateStore({ statePath: corruptPath });
|
||||||
|
expect(corruptStore.get("openai-codex")).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("persists only provider ids with expires and lastAlertAt", () => {
|
||||||
|
const statePath = createTempStatePath();
|
||||||
|
const store = new OAuthAlertStateStore({ statePath, clock: () => 5678 });
|
||||||
|
|
||||||
|
store.recordAlert("claude", 4321);
|
||||||
|
|
||||||
|
expect(JSON.parse(readFileSync(statePath, "utf-8"))).toEqual({
|
||||||
|
claude: {
|
||||||
|
expires: 4321,
|
||||||
|
lastAlertAt: 5678,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("clears selected providers and all providers", () => {
|
||||||
|
const statePath = createTempStatePath();
|
||||||
|
const store = new OAuthAlertStateStore({ statePath, clock: () => 100 });
|
||||||
|
|
||||||
|
store.recordAlert("claude", 1_000);
|
||||||
|
store.recordAlert("openai-codex", 2_000, 200);
|
||||||
|
store.clear(["claude"]);
|
||||||
|
|
||||||
|
const afterSingleClear = new OAuthAlertStateStore({ statePath });
|
||||||
|
expect(afterSingleClear.get("claude")).toBeUndefined();
|
||||||
|
expect(afterSingleClear.get("openai-codex")).toEqual({ expires: 2_000, lastAlertAt: 200 });
|
||||||
|
|
||||||
|
afterSingleClear.clear();
|
||||||
|
expect(new OAuthAlertStateStore({ statePath }).get("openai-codex")).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,6 +1,18 @@
|
|||||||
import { describe, expect, it, vi } from "vitest";
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||||
import { OAuthExpiryMonitor, type AuthStorageLike } from "../oauth-expiry-monitor.js";
|
import { OAuthExpiryMonitor, type AuthStorageLike } from "../oauth-expiry-monitor.js";
|
||||||
|
|
||||||
|
const tempDirs: string[] = [];
|
||||||
|
|
||||||
|
function createStatePath(): string {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "oauth-expiry-monitor-"));
|
||||||
|
tempDirs.push(dir);
|
||||||
|
return join(dir, "oauth-alert-state.json");
|
||||||
|
}
|
||||||
|
|
||||||
function createAuthStorage(initialCredential?: { type?: string; expires?: number }): AuthStorageLike & {
|
function createAuthStorage(initialCredential?: { type?: string; expires?: number }): AuthStorageLike & {
|
||||||
credential: { type?: string; expires?: number } | undefined;
|
credential: { type?: string; expires?: number } | undefined;
|
||||||
} {
|
} {
|
||||||
@@ -17,17 +29,26 @@ function createAuthStorage(initialCredential?: { type?: string; expires?: number
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.useRealTimers();
|
||||||
|
for (const dir of tempDirs.splice(0)) {
|
||||||
|
rmSync(dir, { force: true, recursive: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
describe("OAuthExpiryMonitor", () => {
|
describe("OAuthExpiryMonitor", () => {
|
||||||
it("fires once when an OAuth credential is expired", async () => {
|
it("fires once when an OAuth credential is expired", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
const authStorage = createAuthStorage({ type: "oauth", expires: Date.now() - 1_000 });
|
const now = Date.now();
|
||||||
|
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1_000 });
|
||||||
const dispatch = vi.fn(async () => undefined);
|
const dispatch = vi.fn(async () => undefined);
|
||||||
|
|
||||||
const monitor = new OAuthExpiryMonitor({
|
const monitor = new OAuthExpiryMonitor({
|
||||||
authStorage,
|
authStorage,
|
||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
intervalMs: 100,
|
||||||
clock: () => Date.now(),
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await monitor.start();
|
||||||
@@ -45,7 +66,6 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
monitor.stop();
|
monitor.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not fire for non-expired/non-oauth credentials", async () => {
|
it("does not fire for non-expired/non-oauth credentials", async () => {
|
||||||
@@ -67,6 +87,7 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
intervalMs: 100,
|
||||||
clock: () => now,
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await monitor.start();
|
||||||
@@ -75,7 +96,6 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
expect(dispatch).not.toHaveBeenCalled();
|
expect(dispatch).not.toHaveBeenCalled();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("deduplicates dispatches for same provider and expiry", async () => {
|
it("deduplicates dispatches for same provider and expiry", async () => {
|
||||||
@@ -89,6 +109,7 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
intervalMs: 100,
|
||||||
clock: () => now,
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await monitor.start();
|
||||||
@@ -96,12 +117,12 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
|
|
||||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
monitor.stop();
|
monitor.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("re-fires after credential is replaced with a new expiry that later expires", async () => {
|
it("re-fires after credential is replaced with a new expiry that later expires", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
let now = Date.now();
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||||
const dispatch = vi.fn(async () => undefined);
|
const dispatch = vi.fn(async () => undefined);
|
||||||
|
|
||||||
@@ -110,6 +131,7 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
intervalMs: 100,
|
||||||
clock: () => now,
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await monitor.start();
|
||||||
@@ -128,42 +150,126 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
|
|
||||||
expect(dispatch).toHaveBeenCalledTimes(2);
|
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||||
monitor.stop();
|
monitor.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("throttles changed expiries until min notify interval elapses", async () => {
|
it("throttles changed expiries until min notify interval elapses across restarts", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
let now = Date.now();
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||||
const dispatch = vi.fn(async () => undefined);
|
const dispatch = vi.fn(async () => undefined);
|
||||||
|
|
||||||
const monitor = new OAuthExpiryMonitor({
|
const firstMonitor = new OAuthExpiryMonitor({
|
||||||
authStorage,
|
authStorage,
|
||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
|
||||||
minNotifyIntervalMs: 1_000,
|
minNotifyIntervalMs: 1_000,
|
||||||
clock: () => now,
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await firstMonitor.start();
|
||||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
firstMonitor.stop();
|
||||||
|
|
||||||
authStorage.credential = { type: "oauth", expires: now + 10_000 };
|
|
||||||
await vi.advanceTimersByTimeAsync(100);
|
|
||||||
|
|
||||||
now += 500;
|
|
||||||
authStorage.credential = { type: "oauth", expires: now - 1 };
|
|
||||||
await vi.advanceTimersByTimeAsync(100);
|
|
||||||
|
|
||||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
now += 500;
|
now += 500;
|
||||||
authStorage.credential = { type: "oauth", expires: now - 2 };
|
authStorage.credential = { type: "oauth", expires: now - 2 };
|
||||||
await vi.advanceTimersByTimeAsync(100);
|
const restartedMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage,
|
||||||
|
notificationService: { dispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
|
||||||
|
await restartedMonitor.start();
|
||||||
|
restartedMonitor.stop();
|
||||||
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
now += 500;
|
||||||
|
authStorage.credential = { type: "oauth", expires: now - 3 };
|
||||||
|
await restartedMonitor.start();
|
||||||
|
restartedMonitor.stop();
|
||||||
|
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not persist lastAlertAt when dispatch fails", async () => {
|
||||||
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
|
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||||
|
const dispatch = vi.fn(async () => {
|
||||||
|
throw new Error("boom");
|
||||||
|
});
|
||||||
|
|
||||||
|
const firstMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage,
|
||||||
|
notificationService: { dispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await firstMonitor.start();
|
||||||
|
firstMonitor.stop();
|
||||||
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
const secondDispatch = vi.fn(async () => undefined);
|
||||||
|
now += 100;
|
||||||
|
const restartedMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage,
|
||||||
|
notificationService: { dispatch: secondDispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await restartedMonitor.start();
|
||||||
|
restartedMonitor.stop();
|
||||||
|
|
||||||
|
expect(secondDispatch).toHaveBeenCalledTimes(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("clears persisted state when providers disappear", async () => {
|
||||||
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
|
const authStorage = createAuthStorage({ type: "oauth", expires: now - 1 });
|
||||||
|
const dispatch = vi.fn(async () => undefined);
|
||||||
|
|
||||||
|
const firstMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage,
|
||||||
|
notificationService: { dispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await firstMonitor.start();
|
||||||
|
firstMonitor.stop();
|
||||||
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
const noProviderStorage: AuthStorageLike = {
|
||||||
|
reload: vi.fn(),
|
||||||
|
getOAuthProviders: () => [],
|
||||||
|
get: () => undefined,
|
||||||
|
};
|
||||||
|
const clearingMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage: noProviderStorage,
|
||||||
|
notificationService: { dispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await clearingMonitor.start();
|
||||||
|
clearingMonitor.stop();
|
||||||
|
|
||||||
|
now += 100;
|
||||||
|
const restartedMonitor = new OAuthExpiryMonitor({
|
||||||
|
authStorage,
|
||||||
|
notificationService: { dispatch } as any,
|
||||||
|
minNotifyIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await restartedMonitor.start();
|
||||||
|
restartedMonitor.stop();
|
||||||
|
|
||||||
expect(dispatch).toHaveBeenCalledTimes(2);
|
expect(dispatch).toHaveBeenCalledTimes(2);
|
||||||
monitor.stop();
|
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("stop cancels the interval", async () => {
|
it("stop cancels the interval", async () => {
|
||||||
@@ -177,6 +283,7 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
notificationService: { dispatch } as any,
|
notificationService: { dispatch } as any,
|
||||||
intervalMs: 100,
|
intervalMs: 100,
|
||||||
clock: () => now,
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
});
|
});
|
||||||
|
|
||||||
await monitor.start();
|
await monitor.start();
|
||||||
@@ -184,6 +291,5 @@ describe("OAuthExpiryMonitor", () => {
|
|||||||
await vi.advanceTimersByTimeAsync(500);
|
await vi.advanceTimersByTimeAsync(500);
|
||||||
|
|
||||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,7 +1,19 @@
|
|||||||
import { describe, expect, it, vi } from "vitest";
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import { OAuthAlertStateStore } from "../oauth-alert-state.js";
|
||||||
import { OAuthValidityLogger } from "../oauth-validity-logger.js";
|
import { OAuthValidityLogger } from "../oauth-validity-logger.js";
|
||||||
import type { AuthStorageLike } from "../oauth-expiry-monitor.js";
|
import type { AuthStorageLike } from "../oauth-expiry-monitor.js";
|
||||||
|
|
||||||
|
const tempDirs: string[] = [];
|
||||||
|
|
||||||
|
function createStatePath(): string {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), "oauth-validity-logger-"));
|
||||||
|
tempDirs.push(dir);
|
||||||
|
return join(dir, "oauth-alert-state.json");
|
||||||
|
}
|
||||||
|
|
||||||
function createAuthStorage(providers: Array<{ id: string; name: string }>, credentials: Record<string, any>): AuthStorageLike {
|
function createAuthStorage(providers: Array<{ id: string; name: string }>, credentials: Record<string, any>): AuthStorageLike {
|
||||||
return {
|
return {
|
||||||
reload: vi.fn(),
|
reload: vi.fn(),
|
||||||
@@ -10,6 +22,13 @@ function createAuthStorage(providers: Array<{ id: string; name: string }>, crede
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.useRealTimers();
|
||||||
|
for (const dir of tempDirs.splice(0)) {
|
||||||
|
rmSync(dir, { force: true, recursive: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
describe("OAuthValidityLogger", () => {
|
describe("OAuthValidityLogger", () => {
|
||||||
it("logs one line per expired oauth credential on start", async () => {
|
it("logs one line per expired oauth credential on start", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
@@ -26,30 +45,79 @@ describe("OAuthValidityLogger", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
intervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
|
});
|
||||||
await validityLogger.start();
|
await validityLogger.start();
|
||||||
|
|
||||||
expect(logger).toHaveBeenCalledTimes(2);
|
expect(logger).toHaveBeenCalledTimes(2);
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("logs again on interval without dedupe", async () => {
|
it("skips repeated logs within the throttle window", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
const now = Date.now();
|
|
||||||
const logger = vi.fn();
|
const logger = vi.fn();
|
||||||
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
const authStorage = createAuthStorage(
|
const authStorage = createAuthStorage(
|
||||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||||
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||||
);
|
);
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
await validityLogger.start();
|
authStorage,
|
||||||
await vi.advanceTimersByTimeAsync(1_000);
|
logger,
|
||||||
|
intervalMs: 100,
|
||||||
|
minAlertIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
|
||||||
expect(logger).toHaveBeenCalledTimes(2);
|
await validityLogger.start();
|
||||||
|
now += 500;
|
||||||
|
await validityLogger.check();
|
||||||
|
|
||||||
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
vi.useRealTimers();
|
});
|
||||||
|
|
||||||
|
it("persists the throttle across a restart and logs again after the window elapses", async () => {
|
||||||
|
vi.useFakeTimers();
|
||||||
|
const logger = vi.fn();
|
||||||
|
let now = Date.now();
|
||||||
|
const statePath = createStatePath();
|
||||||
|
const authStorage = createAuthStorage(
|
||||||
|
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||||
|
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||||
|
);
|
||||||
|
|
||||||
|
const firstLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
minAlertIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await firstLogger.check();
|
||||||
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
const restartedLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
minAlertIntervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath, clock: () => now }),
|
||||||
|
});
|
||||||
|
await restartedLogger.check();
|
||||||
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
|
|
||||||
|
now += 1_001;
|
||||||
|
await restartedLogger.check();
|
||||||
|
expect(logger).toHaveBeenCalledTimes(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not log for valid oauth, api key, or missing expires", async () => {
|
it("does not log for valid oauth, api key, or missing expires", async () => {
|
||||||
@@ -69,30 +137,42 @@ describe("OAuthValidityLogger", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
intervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
|
});
|
||||||
await validityLogger.start();
|
await validityLogger.start();
|
||||||
|
|
||||||
expect(logger).not.toHaveBeenCalled();
|
expect(logger).not.toHaveBeenCalled();
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("stop cancels the interval", async () => {
|
it("stop cancels the interval", async () => {
|
||||||
vi.useFakeTimers();
|
vi.useFakeTimers();
|
||||||
const now = Date.now();
|
let now = Date.now();
|
||||||
const logger = vi.fn();
|
const logger = vi.fn();
|
||||||
const authStorage = createAuthStorage(
|
const authStorage = createAuthStorage(
|
||||||
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||||
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
{ "openai-codex": { type: "oauth", expires: now - 1_000 } },
|
||||||
);
|
);
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
intervalMs: 1_000,
|
||||||
|
minAlertIntervalMs: 500,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
|
});
|
||||||
await validityLogger.start();
|
await validityLogger.start();
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
|
now += 5_000;
|
||||||
await vi.advanceTimersByTimeAsync(5_000);
|
await vi.advanceTimersByTimeAsync(5_000);
|
||||||
|
|
||||||
expect(logger).toHaveBeenCalledTimes(1);
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("continues iterating when one provider throws", async () => {
|
it("continues iterating when one provider throws", async () => {
|
||||||
@@ -113,7 +193,13 @@ describe("OAuthValidityLogger", () => {
|
|||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
intervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
|
});
|
||||||
await validityLogger.start();
|
await validityLogger.start();
|
||||||
|
|
||||||
expect(logger).toHaveBeenCalledTimes(1);
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
@@ -122,7 +208,6 @@ describe("OAuthValidityLogger", () => {
|
|||||||
expect.objectContaining({ providerId: "claude" }),
|
expect.objectContaining({ providerId: "claude" }),
|
||||||
);
|
);
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("never includes token material in log metadata", async () => {
|
it("never includes token material in log metadata", async () => {
|
||||||
@@ -141,12 +226,50 @@ describe("OAuthValidityLogger", () => {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
const validityLogger = new OAuthValidityLogger({ authStorage, logger, intervalMs: 1_000, clock: () => now });
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
intervalMs: 1_000,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now }),
|
||||||
|
});
|
||||||
await validityLogger.start();
|
await validityLogger.start();
|
||||||
|
|
||||||
const [, meta] = logger.mock.calls[0] ?? [];
|
const [, meta] = logger.mock.calls[0] ?? [];
|
||||||
expect(Object.keys(meta ?? {}).sort()).toEqual(["expiresAt", "providerId", "providerName"]);
|
expect(Object.keys(meta ?? {}).sort()).toEqual(["expiresAt", "providerId", "providerName"]);
|
||||||
validityLogger.stop();
|
validityLogger.stop();
|
||||||
vi.useRealTimers();
|
});
|
||||||
|
|
||||||
|
it("covers empty, undefined, and populated provider states", async () => {
|
||||||
|
const logger = vi.fn();
|
||||||
|
const now = Date.now();
|
||||||
|
const cases: AuthStorageLike[] = [
|
||||||
|
{
|
||||||
|
reload: vi.fn(),
|
||||||
|
getOAuthProviders: () => [],
|
||||||
|
get: () => undefined,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
reload: vi.fn(),
|
||||||
|
getOAuthProviders: () => [{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||||
|
get: () => undefined,
|
||||||
|
},
|
||||||
|
createAuthStorage(
|
||||||
|
[{ id: "openai-codex", name: "OpenAI Codex" }],
|
||||||
|
{ "openai-codex": { type: "oauth", expires: now - 1 } },
|
||||||
|
),
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const [index, authStorage] of cases.entries()) {
|
||||||
|
const validityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
logger,
|
||||||
|
clock: () => now,
|
||||||
|
alertState: new OAuthAlertStateStore({ statePath: createStatePath(), clock: () => now + index }),
|
||||||
|
});
|
||||||
|
await validityLogger.check();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(logger).toHaveBeenCalledTimes(1);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -7,6 +7,9 @@ export type { WebhookProviderConfig } from "./webhook-provider.js";
|
|||||||
export { NotificationService } from "./notification-service.js";
|
export { NotificationService } from "./notification-service.js";
|
||||||
export type { NotificationServiceOptions } from "./notification-service.js";
|
export type { NotificationServiceOptions } from "./notification-service.js";
|
||||||
|
|
||||||
|
export { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||||
|
export type { OAuthAlertStateEntry, OAuthAlertStateFs, OAuthAlertStateStoreOptions } from "./oauth-alert-state.js";
|
||||||
|
|
||||||
export { OAuthExpiryMonitor } from "./oauth-expiry-monitor.js";
|
export { OAuthExpiryMonitor } from "./oauth-expiry-monitor.js";
|
||||||
export type { AuthStorageLike as OAuthExpiryAuthStorageLike, OAuthExpiryMonitorOptions } from "./oauth-expiry-monitor.js";
|
export type { AuthStorageLike as OAuthExpiryAuthStorageLike, OAuthExpiryMonitorOptions } from "./oauth-expiry-monitor.js";
|
||||||
|
|
||||||
|
|||||||
144
packages/engine/src/notification/oauth-alert-state.ts
Normal file
144
packages/engine/src/notification/oauth-alert-state.ts
Normal file
@@ -0,0 +1,144 @@
|
|||||||
|
import {
|
||||||
|
existsSync,
|
||||||
|
mkdirSync,
|
||||||
|
readFileSync,
|
||||||
|
renameSync,
|
||||||
|
rmSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
|
import { dirname } from "node:path";
|
||||||
|
import { getFusionOAuthAlertStatePath } from "../auth-storage.js";
|
||||||
|
|
||||||
|
export interface OAuthAlertStateEntry {
|
||||||
|
expires: number;
|
||||||
|
lastAlertAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OAuthAlertStateFs {
|
||||||
|
existsSync(path: string): boolean;
|
||||||
|
mkdirSync(path: string, options?: { recursive?: boolean }): void;
|
||||||
|
readFileSync(path: string, encoding: BufferEncoding): string;
|
||||||
|
renameSync(oldPath: string, newPath: string): void;
|
||||||
|
rmSync(path: string, options?: { force?: boolean }): void;
|
||||||
|
writeFileSync(path: string, content: string, encoding: BufferEncoding): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OAuthAlertStateStoreOptions {
|
||||||
|
statePath?: string;
|
||||||
|
clock?: () => number;
|
||||||
|
fs?: OAuthAlertStateFs;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class OAuthAlertStateStore {
|
||||||
|
private readonly statePath: string;
|
||||||
|
private readonly clock: () => number;
|
||||||
|
private readonly fs: OAuthAlertStateFs;
|
||||||
|
|
||||||
|
constructor(options: OAuthAlertStateStoreOptions = {}) {
|
||||||
|
this.statePath = options.statePath ?? getFusionOAuthAlertStatePath();
|
||||||
|
this.clock = options.clock ?? Date.now;
|
||||||
|
this.fs = options.fs ?? {
|
||||||
|
existsSync: (path) => existsSync(path),
|
||||||
|
mkdirSync: (path, options) => {
|
||||||
|
mkdirSync(path, options);
|
||||||
|
},
|
||||||
|
readFileSync: (path, encoding) => readFileSync(path, encoding),
|
||||||
|
renameSync: (oldPath, newPath) => {
|
||||||
|
renameSync(oldPath, newPath);
|
||||||
|
},
|
||||||
|
rmSync: (path, options) => {
|
||||||
|
rmSync(path, options);
|
||||||
|
},
|
||||||
|
writeFileSync: (path, content, encoding) => {
|
||||||
|
writeFileSync(path, content, encoding);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
get(providerId: string): OAuthAlertStateEntry | undefined {
|
||||||
|
return this.readState()[providerId];
|
||||||
|
}
|
||||||
|
|
||||||
|
getLastAlertAt(providerId: string): number | undefined {
|
||||||
|
return this.get(providerId)?.lastAlertAt;
|
||||||
|
}
|
||||||
|
|
||||||
|
recordAlert(providerId: string, expires: number, lastAlertAt = this.clock()): void {
|
||||||
|
const state = this.readState();
|
||||||
|
state[providerId] = { expires, lastAlertAt };
|
||||||
|
this.writeState(state);
|
||||||
|
}
|
||||||
|
|
||||||
|
clear(providerIds?: Iterable<string>): void {
|
||||||
|
if (!providerIds) {
|
||||||
|
this.writeState({});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const state = this.readState();
|
||||||
|
let changed = false;
|
||||||
|
for (const providerId of providerIds) {
|
||||||
|
if (!(providerId in state)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
delete state[providerId];
|
||||||
|
changed = true;
|
||||||
|
}
|
||||||
|
if (changed) {
|
||||||
|
this.writeState(state);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private readState(): Record<string, OAuthAlertStateEntry> {
|
||||||
|
if (!this.fs.existsSync(this.statePath)) {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const parsed = JSON.parse(this.fs.readFileSync(this.statePath, "utf-8")) as unknown;
|
||||||
|
return sanitizeState(parsed);
|
||||||
|
} catch {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private writeState(state: Record<string, OAuthAlertStateEntry>): void {
|
||||||
|
const sanitized = sanitizeState(state);
|
||||||
|
const dir = dirname(this.statePath);
|
||||||
|
this.fs.mkdirSync(dir, { recursive: true });
|
||||||
|
|
||||||
|
const tempPath = `${this.statePath}.${process.pid}.${this.clock()}.tmp`;
|
||||||
|
const body = `${JSON.stringify(sanitized, null, 2)}\n`;
|
||||||
|
this.fs.writeFileSync(tempPath, body, "utf-8");
|
||||||
|
try {
|
||||||
|
this.fs.renameSync(tempPath, this.statePath);
|
||||||
|
} catch (error) {
|
||||||
|
this.fs.rmSync(tempPath, { force: true });
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function sanitizeState(parsed: unknown): Record<string, OAuthAlertStateEntry> {
|
||||||
|
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
|
||||||
|
const sanitized: Record<string, OAuthAlertStateEntry> = {};
|
||||||
|
for (const [providerId, value] of Object.entries(parsed)) {
|
||||||
|
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const expires = (value as { expires?: unknown }).expires;
|
||||||
|
const lastAlertAt = (value as { lastAlertAt?: unknown }).lastAlertAt;
|
||||||
|
if (typeof expires !== "number" || Number.isNaN(expires)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (typeof lastAlertAt !== "number" || Number.isNaN(lastAlertAt)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
sanitized[providerId] = { expires, lastAlertAt };
|
||||||
|
}
|
||||||
|
|
||||||
|
return sanitized;
|
||||||
|
}
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
import type { NotificationPayload } from "@fusion/core";
|
import type { NotificationPayload } from "@fusion/core";
|
||||||
import { schedulerLog } from "../logger.js";
|
import { schedulerLog } from "../logger.js";
|
||||||
|
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||||
import type { NotificationService } from "./notification-service.js";
|
import type { NotificationService } from "./notification-service.js";
|
||||||
|
|
||||||
const DEFAULT_INTERVAL_MS = 5 * 60_000;
|
const DEFAULT_INTERVAL_MS = 5 * 60_000;
|
||||||
@@ -28,6 +29,7 @@ export interface OAuthExpiryMonitorOptions {
|
|||||||
clock?: () => number;
|
clock?: () => number;
|
||||||
warnBeforeMs?: number;
|
warnBeforeMs?: number;
|
||||||
minNotifyIntervalMs?: number;
|
minNotifyIntervalMs?: number;
|
||||||
|
alertState?: OAuthAlertStateStore;
|
||||||
}
|
}
|
||||||
|
|
||||||
export class OAuthExpiryMonitor {
|
export class OAuthExpiryMonitor {
|
||||||
@@ -35,15 +37,16 @@ export class OAuthExpiryMonitor {
|
|||||||
private readonly clock: () => number;
|
private readonly clock: () => number;
|
||||||
private readonly warnBeforeMs: number;
|
private readonly warnBeforeMs: number;
|
||||||
private readonly minNotifyIntervalMs: number;
|
private readonly minNotifyIntervalMs: number;
|
||||||
|
private readonly alertState: OAuthAlertStateStore;
|
||||||
private timer: NodeJS.Timeout | null = null;
|
private timer: NodeJS.Timeout | null = null;
|
||||||
private readonly dispatchedExpiryKeys = new Set<string>();
|
private readonly dispatchedExpiryKeys = new Set<string>();
|
||||||
private readonly lastNotifiedAt = new Map<string, number>();
|
|
||||||
|
|
||||||
constructor(private readonly opts: OAuthExpiryMonitorOptions) {
|
constructor(private readonly opts: OAuthExpiryMonitorOptions) {
|
||||||
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
||||||
this.clock = opts.clock ?? Date.now;
|
this.clock = opts.clock ?? Date.now;
|
||||||
this.warnBeforeMs = opts.warnBeforeMs ?? 0;
|
this.warnBeforeMs = opts.warnBeforeMs ?? 0;
|
||||||
this.minNotifyIntervalMs = opts.minNotifyIntervalMs ?? DEFAULT_MIN_NOTIFY_INTERVAL_MS;
|
this.minNotifyIntervalMs = opts.minNotifyIntervalMs ?? DEFAULT_MIN_NOTIFY_INTERVAL_MS;
|
||||||
|
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
|
||||||
}
|
}
|
||||||
|
|
||||||
async start(): Promise<void> {
|
async start(): Promise<void> {
|
||||||
@@ -73,7 +76,7 @@ export class OAuthExpiryMonitor {
|
|||||||
const providers = this.opts.authStorage.getOAuthProviders?.();
|
const providers = this.opts.authStorage.getOAuthProviders?.();
|
||||||
if (!providers?.length) {
|
if (!providers?.length) {
|
||||||
this.dispatchedExpiryKeys.clear();
|
this.dispatchedExpiryKeys.clear();
|
||||||
this.lastNotifiedAt.clear();
|
this.alertState.clear();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -83,6 +86,7 @@ export class OAuthExpiryMonitor {
|
|||||||
for (const provider of providers) {
|
for (const provider of providers) {
|
||||||
const credential = this.opts.authStorage.get?.(provider.id);
|
const credential = this.opts.authStorage.get?.(provider.id);
|
||||||
if (credential?.type !== "oauth" || typeof credential.expires !== "number") {
|
if (credential?.type !== "oauth" || typeof credential.expires !== "number") {
|
||||||
|
this.alertState.clear([provider.id]);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -96,7 +100,7 @@ export class OAuthExpiryMonitor {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
const previousNotificationAt = this.lastNotifiedAt.get(provider.id);
|
const previousNotificationAt = this.alertState.getLastAlertAt(provider.id);
|
||||||
if (
|
if (
|
||||||
typeof previousNotificationAt === "number" &&
|
typeof previousNotificationAt === "number" &&
|
||||||
now - previousNotificationAt < this.minNotifyIntervalMs
|
now - previousNotificationAt < this.minNotifyIntervalMs
|
||||||
@@ -116,7 +120,7 @@ export class OAuthExpiryMonitor {
|
|||||||
try {
|
try {
|
||||||
await this.opts.notificationService.dispatch("oauth-token-expired", payload);
|
await this.opts.notificationService.dispatch("oauth-token-expired", payload);
|
||||||
this.dispatchedExpiryKeys.add(expiryKey);
|
this.dispatchedExpiryKeys.add(expiryKey);
|
||||||
this.lastNotifiedAt.set(provider.id, now);
|
this.alertState.recordAlert(provider.id, credential.expires, now);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
const message = error instanceof Error ? error.message : String(error);
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
schedulerLog.warn(`OAuth expiry notification dispatch failed provider=${provider.id}: ${message}`);
|
schedulerLog.warn(`OAuth expiry notification dispatch failed provider=${provider.id}: ${message}`);
|
||||||
|
|||||||
@@ -1,25 +1,33 @@
|
|||||||
import { schedulerLog } from "../logger.js";
|
import { schedulerLog } from "../logger.js";
|
||||||
import type { AuthStorageLike } from "./oauth-expiry-monitor.js";
|
import type { AuthStorageLike } from "./oauth-expiry-monitor.js";
|
||||||
|
import { OAuthAlertStateStore } from "./oauth-alert-state.js";
|
||||||
|
|
||||||
const DEFAULT_INTERVAL_MS = 24 * 60 * 60 * 1000;
|
const DEFAULT_INTERVAL_MS = 12 * 60 * 60 * 1000;
|
||||||
|
const DEFAULT_MIN_ALERT_INTERVAL_MS = 12 * 60 * 60 * 1000;
|
||||||
|
|
||||||
interface OAuthValidityLoggerOptions {
|
interface OAuthValidityLoggerOptions {
|
||||||
authStorage: AuthStorageLike;
|
authStorage: AuthStorageLike;
|
||||||
intervalMs?: number;
|
intervalMs?: number;
|
||||||
clock?: () => number;
|
clock?: () => number;
|
||||||
logger?: (msg: string, meta?: Record<string, unknown>) => void;
|
logger?: (msg: string, meta?: Record<string, unknown>) => void;
|
||||||
|
alertState?: OAuthAlertStateStore;
|
||||||
|
minAlertIntervalMs?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
export class OAuthValidityLogger {
|
export class OAuthValidityLogger {
|
||||||
private readonly intervalMs: number;
|
private readonly intervalMs: number;
|
||||||
private readonly clock: () => number;
|
private readonly clock: () => number;
|
||||||
private readonly logger: (msg: string, meta?: Record<string, unknown>) => void;
|
private readonly logger: (msg: string, meta?: Record<string, unknown>) => void;
|
||||||
|
private readonly alertState: OAuthAlertStateStore;
|
||||||
|
private readonly minAlertIntervalMs: number;
|
||||||
private timer: NodeJS.Timeout | null = null;
|
private timer: NodeJS.Timeout | null = null;
|
||||||
|
|
||||||
constructor(private readonly opts: OAuthValidityLoggerOptions) {
|
constructor(private readonly opts: OAuthValidityLoggerOptions) {
|
||||||
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
this.intervalMs = opts.intervalMs ?? DEFAULT_INTERVAL_MS;
|
||||||
this.clock = opts.clock ?? Date.now;
|
this.clock = opts.clock ?? Date.now;
|
||||||
this.logger = opts.logger ?? ((message, meta) => schedulerLog.warn(message, meta));
|
this.logger = opts.logger ?? ((message, meta) => schedulerLog.warn(message, meta));
|
||||||
|
this.alertState = opts.alertState ?? new OAuthAlertStateStore({ clock: this.clock });
|
||||||
|
this.minAlertIntervalMs = opts.minAlertIntervalMs ?? DEFAULT_MIN_ALERT_INTERVAL_MS;
|
||||||
}
|
}
|
||||||
|
|
||||||
async start(): Promise<void> {
|
async start(): Promise<void> {
|
||||||
@@ -57,11 +65,17 @@ export class OAuthValidityLogger {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const previousAlertAt = this.alertState.getLastAlertAt(provider.id);
|
||||||
|
if (typeof previousAlertAt === "number" && now - previousAlertAt < this.minAlertIntervalMs) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
this.logger("oauth credential expired — provider re-login required", {
|
this.logger("oauth credential expired — provider re-login required", {
|
||||||
providerId: provider.id,
|
providerId: provider.id,
|
||||||
providerName: provider.name,
|
providerName: provider.name,
|
||||||
expiresAt: new Date(credential.expires).toISOString(),
|
expiresAt: new Date(credential.expires).toISOString(),
|
||||||
});
|
});
|
||||||
|
this.alertState.recordAlert(provider.id, credential.expires, now);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
const message = error instanceof Error ? error.message : String(error);
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
schedulerLog.warn(`OAuth validity logger failed for provider=${provider.id}: ${message}`);
|
schedulerLog.warn(`OAuth validity logger failed for provider=${provider.id}: ${message}`);
|
||||||
|
|||||||
@@ -19,10 +19,10 @@ import type { ProjectRuntimeConfig } from "./project-runtime.js";
|
|||||||
import { PrMonitor } from "./pr-monitor.js";
|
import { PrMonitor } from "./pr-monitor.js";
|
||||||
import { PrCommentHandler } from "./pr-comment-handler.js";
|
import { PrCommentHandler } from "./pr-comment-handler.js";
|
||||||
import { NtfyNotifier } from "./notifier.js";
|
import { NtfyNotifier } from "./notifier.js";
|
||||||
import { NotificationService, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
|
import { NotificationService, OAuthAlertStateStore, OAuthExpiryMonitor, OAuthValidityLogger } from "./notification/index.js";
|
||||||
import type { NotificationChatStore } from "./notification/notification-service.js";
|
import type { NotificationChatStore } from "./notification/notification-service.js";
|
||||||
import { GridlockDetector } from "./gridlock-detector.js";
|
import { GridlockDetector } from "./gridlock-detector.js";
|
||||||
import { createFusionAuthStorage } from "./auth-storage.js";
|
import { createFusionAuthStorage, getFusionOAuthAlertStatePath } from "./auth-storage.js";
|
||||||
import { CronRunner, createAiPromptExecutor } from "./cron-runner.js";
|
import { CronRunner, createAiPromptExecutor } from "./cron-runner.js";
|
||||||
import type { RoutineRunner } from "./routine-runner.js";
|
import type { RoutineRunner } from "./routine-runner.js";
|
||||||
import { aiMergeTask, sweepStaleAutostashes, VerificationError } from "./merger.js";
|
import { aiMergeTask, sweepStaleAutostashes, VerificationError } from "./merger.js";
|
||||||
@@ -427,12 +427,19 @@ export class ProjectEngine {
|
|||||||
});
|
});
|
||||||
await this.notificationService.start();
|
await this.notificationService.start();
|
||||||
const authStorage = createFusionAuthStorage();
|
const authStorage = createFusionAuthStorage();
|
||||||
|
const oauthAlertState = new OAuthAlertStateStore({
|
||||||
|
statePath: getFusionOAuthAlertStatePath(),
|
||||||
|
});
|
||||||
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
|
this.oauthExpiryMonitor = new OAuthExpiryMonitor({
|
||||||
authStorage,
|
authStorage,
|
||||||
notificationService: this.notificationService,
|
notificationService: this.notificationService,
|
||||||
|
alertState: oauthAlertState,
|
||||||
});
|
});
|
||||||
await this.oauthExpiryMonitor.start();
|
await this.oauthExpiryMonitor.start();
|
||||||
this.oauthValidityLogger = new OAuthValidityLogger({ authStorage });
|
this.oauthValidityLogger = new OAuthValidityLogger({
|
||||||
|
authStorage,
|
||||||
|
alertState: oauthAlertState,
|
||||||
|
});
|
||||||
await this.oauthValidityLogger.start();
|
await this.oauthValidityLogger.start();
|
||||||
|
|
||||||
// Backward-compatibility shim for gridlock notifications.
|
// Backward-compatibility shim for gridlock notifications.
|
||||||
|
|||||||
Reference in New Issue
Block a user