fix(ingest): rolling lookback + ongoing-aware watermark to prevent missed sessions

Bug: previous logic advanced watermark to the latest session start_time including
ongoing sessions. PostHog session_recordings filters by start_time, so once a
session was 'seen' as ongoing the watermark moved past its start time and the
session was never re-fetched after it ended. Today 4 auth sessions on
/dashboard/vehicles/* and /dashboard/search (07:17-07:40 UTC) were lost this way.

Fix:
1. ROLLING_LOOKBACK_MINUTES (default 60): every cycle queries date_from =
   min(watermark, now - 60min). Sessions that just finished get re-fetched
   regardless of watermark drift. Upsert dedupes.
2. Track earliestOngoingStart; cap watermark to (earliestOngoingStart - 1s)
   so subsequent cycles re-read that range.

Also added GET/DELETE /api/insights/watermark for manual reset (used to
trigger 24h backfill after this deploy).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Semih
2026-05-14 10:53:38 +00:00
parent 7edbf1d7ad
commit 74f0ff4935
3 changed files with 50 additions and 7 deletions

View File

@@ -0,0 +1,24 @@
import { NextResponse } from "next/server";
import { auth } from "@/lib/auth";
import { headers } from "next/headers";
import { prisma } from "@/lib/db";
export const dynamic = "force-dynamic";
export async function DELETE(req: Request) {
const session = await auth.api.getSession({ headers: await headers() });
if (!session) return NextResponse.json({ ok: false, error: "unauthenticated" }, { status: 401 });
const url = new URL(req.url);
const projectKey = url.searchParams.get("project") ?? "sase";
const deleted = await prisma.ingestionWatermark.deleteMany({ where: { projectKey } });
return NextResponse.json({ ok: true, projectKey, deleted: deleted.count });
}
export async function GET() {
const session = await auth.api.getSession({ headers: await headers() });
if (!session) return NextResponse.json({ ok: false, error: "unauthenticated" }, { status: 401 });
const rows = await prisma.ingestionWatermark.findMany();
return NextResponse.json({ ok: true, rows });
}