fix(ingest): rolling lookback + ongoing-aware watermark to prevent missed sessions

Browse Source

Bug: previous logic advanced watermark to the latest session start_time including
ongoing sessions. PostHog session_recordings filters by start_time, so once a
session was 'seen' as ongoing the watermark moved past its start time and the
session was never re-fetched after it ended. Today 4 auth sessions on
/dashboard/vehicles/* and /dashboard/search (07:17-07:40 UTC) were lost this way.

Fix:
1. ROLLING_LOOKBACK_MINUTES (default 60): every cycle queries date_from =
   min(watermark, now - 60min). Sessions that just finished get re-fetched
   regardless of watermark drift. Upsert dedupes.
2. Track earliestOngoingStart; cap watermark to (earliestOngoingStart - 1s)
   so subsequent cycles re-read that range.

Also added GET/DELETE /api/insights/watermark for manual reset (used to
trigger 24h backfill after this deploy).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

...

This commit is contained in:

Semih

2026-05-14 10:53:38 +00:00

parent 7edbf1d7ad

commit 74f0ff4935

3 changed files with 50 additions and 7 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

2

apps/web/tsconfig.tsbuildinfo

File diff suppressed because one or more lines are too long