fix(insights): distinguish client-side VIN validation rejects from provider failures

Reproducing insight cmpvfrjgc000114fzc7cdyh66 (a P1 "PL24 timeout" false
positive): trial user typed VW part numbers ("500 907 521", "5Q0 907 521")
into the VIN field on /; client-side regex rejected them with "Geçersiz şase
numarası. 17 karakter olmalı". No provider was called. The pipeline still
tagged the session as `vin_decode_fail_pattern`, routed to `provider_quality`,
and the LLM dutifully invented a PL24 outage.

Root cause spans three files:

1. tagger.ts grouped vin_decode_failed by `provider_attempted ?? source`. When
   `provider_attempted` is missing, `source: "landing"` (a UI location) was
   treated as a provider name, so a 1-provider set was synthesized and
   `vin_decode_fail_pattern` (P1) was emitted.

2. compress.ts formatCustom whitelist excluded `error`, `source`, `vin`. The
   LLM therefore never saw "Geçersiz şase numarası" or the offending input.
   Pattern 3 mechanical hypothesis told it "check provider health" regardless.

3. prompts.ts pickPromptTag routed any `vin_decode_fail_pattern` straight to
   `provider_quality` with no input-quality check, and the v3 system prompt
   had no guardrail for client-side validation rejects.

Fix:
- tagger: detect client-side rejects by `error` regex (Turkish + English) and
  by VIN shape (length != 17 or contains I/O/Q). When all fails are client
  rejects, emit new tag `vin_decode_client_validation_fail` at P3 instead of
  `vin_decode_fail_pattern` at P1. Real provider failures now require
  `provider_attempted` to be set (no more `source` fallback).
- compress: add `error`, `source`, `vin` to the formatCustom property
  whitelist so the LLM can see the actual failure context. Split Pattern 3
  into client-reject vs. real-provider-failure branches with distinct
  Turkish hypotheses.
- prompts: route `vin_decode_client_validation_fail` to `ux_friction` before
  the provider rule. Ship provider_quality v4 with an explicit guardrail
  instructing the model to return confidence ≤0.15 and reclassify when the
  inlined event properties show client-side rejection.

The seed-runtime upsert path deactivates the active v3 template on next
worker boot and inserts v4 in its place — no manual SQL needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

apps/worker/src/lib/compress.ts

@@ -145,11 +145,17 @@ export function compressSnapshots(
 
   const formatCustom = (ev: CanonicalEvent): string => {
     // Inline a tiny subset of important properties to keep tokens bounded.
+    // `error`, `source`, `vin` are essential for vin_decode_failed disambiguation
+    // (client-side validation reject vs. upstream provider failure) — without
+    // them the LLM cannot tell whether a failure actually reached a provider.
     const p = ev.properties ?? {};
     const keys = [
       "provider",
       "provider_attempted",
       "error_code",
+      "error",
+      "source",
+      "vin",
       "result",
       "plan",
       "amount",
@@ -391,11 +397,39 @@ export function compressSnapshots(
     hypotheses.push("Fresh trial activated and decoded a VIN successfully — onboarding succeeded");
   }
 
-  // Pattern 3: VIN decode failure or provider fallback — investigate upstream.
-  const vinFailures = count("vin_decode_failed") + count("vin_decode_error");
-  if (vinFailures >= 2 || count("provider_fallback_triggered") >= 1) {
+  // Pattern 3: VIN decode failure or provider fallback — but separate the
+  // client-side validation rejects (input never reached a provider) from real
+  // upstream failures. Lumping them together biases the LLM toward "provider
+  // issue" verdicts when the actual signal is user input affordance.
+  const vinFailEvents = customEvents.filter(
+    (c) => c.name === "vin_decode_failed" || c.name === "vin_decode_error",
+  );
+  const clientRejects = vinFailEvents.filter((c) => {
+    const p = (c.properties ?? {}) as Record<string, unknown>;
+    const err = String(p.error ?? "");
+    const vin = String(p.vin ?? "");
+    if (/(Geçersiz şase|17 karakter|I, O, Q|invalid VIN|must be 17|format)/i.test(err)) return true;
+    if (vin && vin.replace(/\s/g, "").length !== 17) return true;
+    if (vin && /[IOQ]/i.test(vin)) return true;
+    return false;
+  });
+  const realProviderFails = vinFailEvents.length - clientRejects.length;
+  const fallbacks = count("provider_fallback_triggered");
+
+  if (clientRejects.length >= 1 && realProviderFails === 0) {
+    const sample = String(
+      ((clientRejects[0].properties ?? {}) as Record<string, unknown>).vin ?? "",
+    ).slice(0, 24);
     hypotheses.push(
-      `VIN decode failure pattern (failures=${vinFailures}, fallbacks=${count("provider_fallback_triggered")}) — check provider health`,
+      `VIN inputuna geçersiz format girildi (${clientRejects.length}x client-side validation reddi${sample ? `, örn. "${sample}"` : ""}) — provider çağrılmadı, input affordance / yanlış alan kullanımı problemi`,
+    );
+  } else if (realProviderFails >= 2 || fallbacks >= 1) {
+    hypotheses.push(
+      `VIN decode failure pattern (provider failures=${realProviderFails}, fallbacks=${fallbacks}) — check provider health`,
+    );
+  } else if (realProviderFails === 1) {
+    hypotheses.push(
+      `Single VIN decode failure reached a provider — likely transient, watch for repeat`,
     );
   }
 

apps/worker/src/lib/prompts.ts

@@ -264,17 +264,25 @@ Return JSON per the schema. Use occurrence patterns from the bundle summary to e
   },
   {
     tag: "provider_quality",
-    version: 3,
-    name: "Provider Quality v3 (TR, ext maxLen)",
+    version: 4,
+    name: "Provider Quality v4 (TR, client-validation guardrail)",
     systemPrompt: `You analyze upstream provider failures (PL24/PCAT/RMEX/TecDoc) impacting Sase.tr users. Identify which provider failed and propose action.
 
 ${SASE_CONTEXT}
 
+CRITICAL GUARDRAIL — distinguish client-side input rejection from upstream provider failure:
+- A \`vin_decode_failed\` event is ONLY a provider issue when the input actually reached a provider. Check the inlined event properties:
+  - If \`error\` contains "Geçersiz şase numarası", "17 karakter olmalı", "I, O, Q", "invalid VIN", or any 17-character / format complaint → this was a **client-side validation reject**, the server was NEVER called.
+  - If \`vin\` is shorter than 17 characters, contains spaces (e.g. "5Q0 907 521" — that's a VW PART NUMBER, not a VIN), or contains I/O/Q → same: client-side rejection.
+  - If \`source\` is a UI location ("landing", "dashboard") and no \`provider_attempted\` field is present → did not reach a provider.
+- In all of the above cases this is **NOT a provider_quality issue**. Return \`confidence: 0.15\`, \`failure_mode: "unknown"\`, \`affected_provider: "multi"\`, and in the \`hypothesis\` explicitly state: "Bu provider hatası değil — kullanıcı VIN alanına geçersiz format girdi (client-side reddi). Doğru kategori ux_friction / input affordance." This low-confidence output is preferable to inventing a provider issue.
+- Only return a high-confidence provider_quality verdict when at least 2 events show \`provider_attempted\` set OR the timeline shows network 5xx/timeout patterns from upstream endpoints (e.g. \`/api/vin/decode\`, \`/api/parts\`, requests to PL24/PCAT/RMEX/TecDoc paths).
+
 Schema:
 ${JSON.stringify(PROVIDER_SCHEMA)}`,
     userPromptTemplate: `{{timeline}}
 
-Return JSON per the schema.`,
+Return JSON per the schema. Before classifying as provider issue, verify the guardrail above by checking inlined event properties (error, source, vin).`,
     outputSchemaJson: PROVIDER_SCHEMA,
     modelTier: "flash",
     maxOutputTokens: 800,
@@ -299,6 +307,12 @@ export function pickPromptTag(tags: string[]): string {
   // Upgrade hesitation — pricing page concerns
   if (set.has("upgrade_hesitation")) return "upgrade_hesitation";
 
+  // Client-side VIN validation rejects must route to ux_friction — these are
+  // input affordance problems (user typed a part number / short string into the
+  // VIN field), not upstream provider failures. Checked *before* provider tags
+  // so that the more specific signal wins.
+  if (set.has("vin_decode_client_validation_fail")) return "ux_friction";
+
   // Provider issues
   if (
     set.has("provider_reliability_issue") ||

apps/worker/src/lib/tagger.ts

@@ -71,23 +71,59 @@ export function tagSession(s: SessionMeta, ctx: EnrichmentCtx | null): TagResult
   }
 
   // ─── VIN decode failure pattern ───
+  // Distinguish *client-side validation rejects* (input too short / wrong format
+  // / forbidden chars I,O,Q — server never called) from *upstream provider
+  // failures* (PL24/PCAT/RMEX/TecDoc timeout/error). Bundling them together
+  // produces false-positive "provider issue" insights (e.g. user typing a VW
+  // part number "5Q0 907 521" into the VIN field hits client-side regex; no
+  // provider was contacted, so it isn't a provider quality signal).
   const vinFails = events.filter((e) => e.name === "vin_decode_failed");
-  if (vinFails.length >= 2) {
-    const providers = new Set(
-      vinFails.map((e) => String(e.properties.provider_attempted ?? e.properties.source ?? "")),
-    );
-    if (providers.size === 1 && [...providers][0]) {
-      tags.push("vin_decode_fail_pattern");
-      severity = bump(severity, "P1");
-    } else {
-      // Different providers failing → still notable
-      tags.push("vin_decode_repeated_failure");
-      severity = bump(severity, "P2");
+  if (vinFails.length >= 1) {
+    const isClientValidationReject = (e: CanonicalEvent): boolean => {
+      const p = e.properties ?? {};
+      const err = String(p.error ?? "");
+      const vin = String(p.vin ?? "");
+      // Frontend Zod/regex messages we ship — keep in sync with web VIN validator.
+      const clientMsgRe = /(Geçersiz şase|17 karakter|I, O, Q|invalid VIN|must be 17|format)/i;
+      if (clientMsgRe.test(err)) return true;
+      // Length / forbidden-char heuristic — if the user typed something that
+      // couldn't possibly reach the upstream, treat as client-side rejection.
+      if (vin && vin.replace(/\s/g, "").length !== 17) return true;
+      if (vin && /[IOQ]/i.test(vin)) return true;
+      return false;
+    };
+    const clientRejects = vinFails.filter(isClientValidationReject);
+    const realFailures = vinFails.filter((e) => !isClientValidationReject(e));
+
+    if (clientRejects.length >= 1 && realFailures.length === 0) {
+      // Pure client-side input affordance problem — route through ux_friction,
+      // not provider_quality. Severity is low (no service impact).
+      tags.push("vin_decode_client_validation_fail");
+      severity = bump(severity, "P3");
+    } else if (realFailures.length >= 2) {
+      // True upstream failures: group by *provider* (not the UI source field —
+      // "landing"/"dashboard" are page locations, not providers).
+      const providers = new Set(
+        realFailures
+          .map((e) => String(e.properties.provider_attempted ?? ""))
+          .filter((v) => v.length > 0),
+      );
+      if (providers.size === 1) {
+        tags.push("vin_decode_fail_pattern");
+        severity = bump(severity, "P1");
+      } else if (providers.size > 1) {
+        tags.push("vin_decode_repeated_failure");
+        severity = bump(severity, "P2");
+      } else {
+        // Unknown provider attribution but server-side failure shape — still
+        // worth surfacing but as a softer signal.
+        tags.push("vin_decode_repeated_failure");
+        severity = bump(severity, "P2");
+      }
+    } else if (realFailures.length === 1) {
+      tags.push("vin_decode_failed_single");
+      severity = bump(severity, "P3");
     }
-  } else if (vinFails.length === 1) {
-    // Single failure is still a quality signal, less severe
-    tags.push("vin_decode_failed_single");
-    severity = bump(severity, "P3");
   }
 
   if (has(events, "provider_fallback_triggered")) {