fix(catalog-source): gate emex parts behind allowlist (safety)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

The catalog-wide bridge in EmexSourceDbService.fetchCategoryParts was
measured against vehicle_parts on 2026-06-01 and found to return 7-114x
more parts than belong to the requesting vehicle, with 49-98 wrong OEM
codes per 100 served. That directly violates the project rule that the
user must never see a wrong OEM.

Per-catalog noiseRatio sample (catalog-wide / per-vehicle):
  RENAULT201910 51x | FFIAT84 45x | VOLVO201410 24x | MB201810 14x
  AU1587 8x | BMW202501 70x (+ gid namespace mismatch ETK vs numeric)
  GM_C201809 114x | MINI202501 12x | LRE201412 7x | MAZDA2020 54x
  GM_OP201809 dump has only 1 wildcard vehicle (unique_key="_") so the
  single Crossland X "owns" all 47k Opel parts — same firehose served
  to any Opel sub-model in sase prod.

All alternative bridges were proven dead:
  SSD eşleştirme        - session-bound, 0/91 sase SSDs match dump
  scrape_queue_v2.vehicle_ssd - same session SSD format
  api_cache replay      - table empty (0 rows)
  wizard_parameters     - table empty (0 rows)
  VIN direct            - no VIN column in dump
The only viable per-vehicle bridge is vehicles.unique_key reconstruction
from raw_data.parsedOptions, but sase currently stores the required 4
wizard fields on just 5/103 emex vehicles (all Renault). That work is
follow-up; this patch only stops the bleeding.

Change:
- Add EMEX_SOURCE_DB_ALLOWED_CATALOGS env (comma-separated, default "")
- EmexSourceDbService.fetchCategoryParts returns null unless catalogCode
  is in the allowlist. Empty allowlist = service is effectively off for
  parts, full fallthrough to live emex.
- Connection pool stays alive so the follow-up per-vehicle bridge /
  schema-only path can use it without flipping env.
- Boot logs warn loudly when connected with an empty allowlist.

Prod was never affected — CATALOG_SOURCE_DB_ENABLED was unset there. This
fixes dev branch behaviour (default-on since commit 3a3a7d3) and keeps
prod safe by default once main is promoted.

Files:
- packages/config/src/index.ts        env schema + audit notes
- apps/api/src/config/configuration.ts parse allowlist into string[]
- apps/api/src/integrations/catalog-source-db/emex-source-db.service.ts
  allowlist field, init logging, fetchCategoryParts gate, class doc
- docker-compose.coolify.yml          env injection for api + worker

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-01 18:39:40 +03:00
parent c01987da7d
commit 3613caa072
4 changed files with 69 additions and 16 deletions

View File

@@ -77,6 +77,13 @@ export default () => ({
// doesn't cover sase's TR-market vehicles; see service comment for details.
emexEnabled: (process.env.EMEX_SOURCE_DB_ENABLED ?? "true") === "true",
pcatEnabled: process.env.PCAT_SOURCE_DB_ENABLED === "true",
// Per-catalog allowlist for emex parts lookup. Empty → service returns
// null for every catalog (safe default after the 2026-06-01 noise audit).
// Populate ONLY after a catalog's per-vehicle bridge is wired & verified.
emexAllowedCatalogs: (process.env.EMEX_SOURCE_DB_ALLOWED_CATALOGS ?? "")
.split(",")
.map((s) => s.trim())
.filter(Boolean),
},
otel: {
enabled: process.env.OTEL_ENABLED === "true",

View File

@@ -11,21 +11,29 @@ import type { EmexHotspot, EmexHotspotArea, EmexPart, EmexPartsResult } from "..
* Bridge: catalogs.code ↔ sase's vehicles.rawData.catalogCode,
* part_groups.group_id (varchar) ↔ sase's categories.externalId (gid).
*
* Why catalog-wide (no per-vehicle filter): the dump's only stable per-vehicle
* key is the EMEX session-state-descriptor (`ssd`), but SSD is regenerated
* every decode session, so the SSD sase stored for its decoded vehicle never
* matches the SSD the dump scraper recorded (0 / 10 sampled — verified). The
* vehicle_id we'd need to join `vehicle_parts` is unreachable. So we return
* all parts in (catalog, group) across every vehicle in the dump for that
* catalog — over-returns variants that may not apply to the user's specific
* vehicle, but parts overlap heavily and the cost is a slightly noisier list.
* Hits: ~71% of sase's 8287 (catalogCode, gid) pairs (verified 2026-06-01).
* SAFETY (2026-06-01 audit): the catalog-wide bridge below (no per-vehicle
* filter) measured a 7-114x noiseRatio across every sampled catalog and
* 49-98 wrong-OEM per 100 parts served, because vehicle_parts is shared
* densely across all variants in a (catalog, group) pair (e.g. Renault
* Mégane vs Duster vs Clio share the same gids). That directly violates the
* "always correct OEM, no exceptions" rule. To prevent leakage, fetchCategoryParts
* now returns null unless the catalogCode is in the EMEX_SOURCE_DB_ALLOWED_CATALOGS
* allowlist — empty by default. A catalog should ONLY be added once a
* per-vehicle bridge (vehicles.unique_key reconstruction from raw_data.parsedOptions)
* lands and has been verified OEM-by-OEM vs live. All alternative bridges (SSD,
* api_cache, wizard_parameters, scrape_queue_v2.vehicle_ssd) were verified
* dead — see memory `sase-emex-source-db-safety.md` for the full inventory.
*
* The connection pool stays alive (master + emex switches default-on) so
* follow-up code can use it for per-vehicle queries / schema-only diagrams
* without having to flip env again.
*/
@Injectable()
export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
private readonly logger = new Logger(EmexSourceDbService.name);
private pool: Pool | null = null;
private enabled = false;
private allowedCatalogs: Set<string> = new Set();
constructor(private readonly config: ConfigService) {}
@@ -33,6 +41,8 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
const masterEnabled = this.config.get<boolean>("catalogSource.enabled");
const emexEnabled = this.config.get<boolean>("catalogSource.emexEnabled");
const url = this.config.get<string>("catalogSource.emexUrl");
const allowed = this.config.get<string[]>("catalogSource.emexAllowedCatalogs") ?? [];
this.allowedCatalogs = new Set(allowed);
if (!masterEnabled || !emexEnabled || !url) {
this.logger.log(
`[emex-src] disabled (master=${masterEnabled}, emex=${emexEnabled}, urlSet=${Boolean(
@@ -48,7 +58,15 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
waitForConnections: true,
});
this.enabled = true;
this.logger.log("[emex-src] connected, lookup-first enabled");
if (this.allowedCatalogs.size === 0) {
this.logger.warn(
"[emex-src] connected but ALLOWLIST EMPTY — all fetchCategoryParts calls return null until EMEX_SOURCE_DB_ALLOWED_CATALOGS is populated (safety: catalog-wide bridge has 7-114x noise; see sase-emex-source-db-safety memory)",
);
} else {
this.logger.log(
`[emex-src] connected, allowlist: [${[...this.allowedCatalogs].sort().join(",")}]`,
);
}
}
async onModuleDestroy() {
@@ -70,6 +88,10 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
): Promise<EmexPartsResult | null> {
if (!this.enabled || !this.pool) return null;
if (!catalogCode || !gid) return null;
// Safety gate: the only path currently implemented (catalog-wide DISTINCT
// on vehicle_parts.group_id, ignoring vehicle_id) is unsafe — see class
// doc. Until a per-vehicle bridge lands, allow no catalogs.
if (!this.allowedCatalogs.has(catalogCode)) return null;
try {
// Resolve (catalog.id, part_group.id) in one round trip.

View File

@@ -82,6 +82,11 @@ services:
# Per-source kill switches. pcat default off (dump doesn't cover TR vehicles).
- EMEX_SOURCE_DB_ENABLED=${EMEX_SOURCE_DB_ENABLED:-true}
- PCAT_SOURCE_DB_ENABLED=${PCAT_SOURCE_DB_ENABLED:-false}
# Per-catalog parts-lookup allowlist (comma-separated). Empty → all
# fetchCategoryParts calls return null. See sase-emex-source-db-safety
# memory: catalog-wide bridge had 7-114x noise; only add a catalog once
# its per-vehicle bridge is wired & OEM-verified.
- EMEX_SOURCE_DB_ALLOWED_CATALOGS=${EMEX_SOURCE_DB_ALLOWED_CATALOGS:-}
depends_on:
sase-redis:
condition: service_healthy
@@ -151,6 +156,11 @@ services:
# Per-source kill switches. pcat default off (dump doesn't cover TR vehicles).
- EMEX_SOURCE_DB_ENABLED=${EMEX_SOURCE_DB_ENABLED:-true}
- PCAT_SOURCE_DB_ENABLED=${PCAT_SOURCE_DB_ENABLED:-false}
# Per-catalog parts-lookup allowlist (comma-separated). Empty → all
# fetchCategoryParts calls return null. See sase-emex-source-db-safety
# memory: catalog-wide bridge had 7-114x noise; only add a catalog once
# its per-vehicle bridge is wired & OEM-verified.
- EMEX_SOURCE_DB_ALLOWED_CATALOGS=${EMEX_SOURCE_DB_ALLOWED_CATALOGS:-}
depends_on:
sase-redis:
condition: service_healthy

View File

@@ -121,12 +121,19 @@ export const envSchema = z.object({
PCAT_SOURCE_DB_URL: z.string().url().optional(),
EMEX_SOURCE_DB_URL: z.string().optional(), // mysql://... — not a strict URL per WHATWG
// Per-source kill switches under the master CATALOG_SOURCE_DB_ENABLED.
// EMEX defaults true (catalog-allowlist tightens which catalogs hit the dump).
// PCAT defaults FALSE — verified 2026-06-01 that the dump's deep-scrape covers
// a US/JDM market subset (Toyota/Nissan/Audi/Chevy/Hyundai) that doesn't
// intersect sase's TR-market vehicle pool (0 / 103 dev carIds had real parts
// data through either bridge). Container stays running for future use cases
// (OEM cross-ref, alt-part search).
// EMEX_SOURCE_DB_ENABLED keeps the connection pool alive but, per the
// 2026-06-01 safety audit, fetchCategoryParts ALWAYS returns null unless the
// requested catalogCode is also in EMEX_SOURCE_DB_ALLOWED_CATALOGS. The
// catalog-wide bridge measured 7-114x noiseRatio across every catalog and
// 49-98 wrong-OEM per 100 served — direct violation of the "always correct
// OEM" rule. Default allowlist is EMPTY → behaviour is safe by default; the
// master/emex switches stay default-on so the service is ready for the
// per-vehicle unique_key bridge (follow-up work).
// PCAT_SOURCE_DB_ENABLED defaults FALSE — verified 2026-06-01 that the dump's
// deep-scrape covers a US/JDM market subset (Toyota/Nissan/Audi/Chevy/Hyundai)
// that doesn't intersect sase's TR-market vehicle pool (0 / 103 dev carIds
// had real parts data through either bridge). Container stays running for
// future use cases (OEM cross-ref, alt-part search).
EMEX_SOURCE_DB_ENABLED: z
.string()
.transform((v) => v === "true")
@@ -135,6 +142,13 @@ export const envSchema = z.object({
.string()
.transform((v) => v === "true")
.default("false"),
// Per-catalog parts-lookup allowlist. Comma-separated catalog codes
// (e.g. "RENAULT201910,FFIAT84"). Empty (default) → fetchCategoryParts
// always returns null → live emex handles every request. A catalog SHOULD
// only be added here AFTER its per-vehicle bridge (unique_key) is wired and
// verified against live OEM-by-OEM on at least 5 sampled vehicles. See
// memory `sase-emex-source-db-safety.md` for the bridge inventory & audit.
EMEX_SOURCE_DB_ALLOWED_CATALOGS: z.string().default(""),
});
export type Env = z.infer<typeof envSchema>;