p95 was computed over ALL successful rows: cache hits (~10-30ms) set the
baseline, a single lock_wait row (duplicate click waiting on the in-flight
decode of the same VIN) set the spike. On a healthy 3-query window this read
as 'P95 15ms → 2058ms (137× baseline)' (2026-06-11, 4th such false alert).
- p95 now over FRESH decodes only (source='cache', db_hit/redis_positive and
lock_wait rows excluded; null-safe via coalesce)
- both windows must have ≥4 fresh decodes — a 'p95' of fewer is one slow
request, not a tail
- absolute floor raised 1s → 8s (fresh decodes legitimately take seconds;
matches the dashboard's fresh-P95 ok≤8s tone)
Simulated on today's prod data: alert window had 0 fresh decodes → silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5dde24855c