Reproducing insight cmpvfrjgc000114fzc7cdyh66 (a P1 "PL24 timeout" false
positive): trial user typed VW part numbers ("500 907 521", "5Q0 907 521")
into the VIN field on /; client-side regex rejected them with "Geçersiz şase
numarası. 17 karakter olmalı". No provider was called. The pipeline still
tagged the session as `vin_decode_fail_pattern`, routed to `provider_quality`,
and the LLM dutifully invented a PL24 outage.
Root cause spans three files:
1. tagger.ts grouped vin_decode_failed by `provider_attempted ?? source`. When
`provider_attempted` is missing, `source: "landing"` (a UI location) was
treated as a provider name, so a 1-provider set was synthesized and
`vin_decode_fail_pattern` (P1) was emitted.
2. compress.ts formatCustom whitelist excluded `error`, `source`, `vin`. The
LLM therefore never saw "Geçersiz şase numarası" or the offending input.
Pattern 3 mechanical hypothesis told it "check provider health" regardless.
3. prompts.ts pickPromptTag routed any `vin_decode_fail_pattern` straight to
`provider_quality` with no input-quality check, and the v3 system prompt
had no guardrail for client-side validation rejects.
Fix:
- tagger: detect client-side rejects by `error` regex (Turkish + English) and
by VIN shape (length != 17 or contains I/O/Q). When all fails are client
rejects, emit new tag `vin_decode_client_validation_fail` at P3 instead of
`vin_decode_fail_pattern` at P1. Real provider failures now require
`provider_attempted` to be set (no more `source` fallback).
- compress: add `error`, `source`, `vin` to the formatCustom property
whitelist so the LLM can see the actual failure context. Split Pattern 3
into client-reject vs. real-provider-failure branches with distinct
Turkish hypotheses.
- prompts: route `vin_decode_client_validation_fail` to `ux_friction` before
the provider rule. Ship provider_quality v4 with an explicit guardrail
instructing the model to return confidence ≤0.15 and reclassify when the
inlined event properties show client-side rejection.
The seed-runtime upsert path deactivates the active v3 template on next
worker boot and inserts v4 in its place — no manual SQL needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
9a479f9a5b