fix(catalog-source): gate emex parts behind allowlist (safety)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

The catalog-wide bridge in EmexSourceDbService.fetchCategoryParts was
measured against vehicle_parts on 2026-06-01 and found to return 7-114x
more parts than belong to the requesting vehicle, with 49-98 wrong OEM
codes per 100 served. That directly violates the project rule that the
user must never see a wrong OEM.

Per-catalog noiseRatio sample (catalog-wide / per-vehicle):
  RENAULT201910 51x | FFIAT84 45x | VOLVO201410 24x | MB201810 14x
  AU1587 8x | BMW202501 70x (+ gid namespace mismatch ETK vs numeric)
  GM_C201809 114x | MINI202501 12x | LRE201412 7x | MAZDA2020 54x
  GM_OP201809 dump has only 1 wildcard vehicle (unique_key="_") so the
  single Crossland X "owns" all 47k Opel parts — same firehose served
  to any Opel sub-model in sase prod.

All alternative bridges were proven dead:
  SSD eşleştirme        - session-bound, 0/91 sase SSDs match dump
  scrape_queue_v2.vehicle_ssd - same session SSD format
  api_cache replay      - table empty (0 rows)
  wizard_parameters     - table empty (0 rows)
  VIN direct            - no VIN column in dump
The only viable per-vehicle bridge is vehicles.unique_key reconstruction
from raw_data.parsedOptions, but sase currently stores the required 4
wizard fields on just 5/103 emex vehicles (all Renault). That work is
follow-up; this patch only stops the bleeding.

Change:
- Add EMEX_SOURCE_DB_ALLOWED_CATALOGS env (comma-separated, default "")
- EmexSourceDbService.fetchCategoryParts returns null unless catalogCode
  is in the allowlist. Empty allowlist = service is effectively off for
  parts, full fallthrough to live emex.
- Connection pool stays alive so the follow-up per-vehicle bridge /
  schema-only path can use it without flipping env.
- Boot logs warn loudly when connected with an empty allowlist.

Prod was never affected — CATALOG_SOURCE_DB_ENABLED was unset there. This
fixes dev branch behaviour (default-on since commit 3a3a7d3) and keeps
prod safe by default once main is promoted.

Files:
- packages/config/src/index.ts        env schema + audit notes
- apps/api/src/config/configuration.ts parse allowlist into string[]
- apps/api/src/integrations/catalog-source-db/emex-source-db.service.ts
  allowlist field, init logging, fetchCategoryParts gate, class doc
- docker-compose.coolify.yml          env injection for api + worker

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-01 18:39:40 +03:00
parent c01987da7d
commit 3613caa072
4 changed files with 69 additions and 16 deletions

View File

@@ -77,6 +77,13 @@ export default () => ({
// doesn't cover sase's TR-market vehicles; see service comment for details.
emexEnabled: (process.env.EMEX_SOURCE_DB_ENABLED ?? "true") === "true",
pcatEnabled: process.env.PCAT_SOURCE_DB_ENABLED === "true",
// Per-catalog allowlist for emex parts lookup. Empty → service returns
// null for every catalog (safe default after the 2026-06-01 noise audit).
// Populate ONLY after a catalog's per-vehicle bridge is wired & verified.
emexAllowedCatalogs: (process.env.EMEX_SOURCE_DB_ALLOWED_CATALOGS ?? "")
.split(",")
.map((s) => s.trim())
.filter(Boolean),
},
otel: {
enabled: process.env.OTEL_ENABLED === "true",

View File

@@ -11,21 +11,29 @@ import type { EmexHotspot, EmexHotspotArea, EmexPart, EmexPartsResult } from "..
* Bridge: catalogs.code ↔ sase's vehicles.rawData.catalogCode,
* part_groups.group_id (varchar) ↔ sase's categories.externalId (gid).
*
* Why catalog-wide (no per-vehicle filter): the dump's only stable per-vehicle
* key is the EMEX session-state-descriptor (`ssd`), but SSD is regenerated
* every decode session, so the SSD sase stored for its decoded vehicle never
* matches the SSD the dump scraper recorded (0 / 10 sampled — verified). The
* vehicle_id we'd need to join `vehicle_parts` is unreachable. So we return
* all parts in (catalog, group) across every vehicle in the dump for that
* catalog — over-returns variants that may not apply to the user's specific
* vehicle, but parts overlap heavily and the cost is a slightly noisier list.
* Hits: ~71% of sase's 8287 (catalogCode, gid) pairs (verified 2026-06-01).
* SAFETY (2026-06-01 audit): the catalog-wide bridge below (no per-vehicle
* filter) measured a 7-114x noiseRatio across every sampled catalog and
* 49-98 wrong-OEM per 100 parts served, because vehicle_parts is shared
* densely across all variants in a (catalog, group) pair (e.g. Renault
* Mégane vs Duster vs Clio share the same gids). That directly violates the
* "always correct OEM, no exceptions" rule. To prevent leakage, fetchCategoryParts
* now returns null unless the catalogCode is in the EMEX_SOURCE_DB_ALLOWED_CATALOGS
* allowlist — empty by default. A catalog should ONLY be added once a
* per-vehicle bridge (vehicles.unique_key reconstruction from raw_data.parsedOptions)
* lands and has been verified OEM-by-OEM vs live. All alternative bridges (SSD,
* api_cache, wizard_parameters, scrape_queue_v2.vehicle_ssd) were verified
* dead — see memory `sase-emex-source-db-safety.md` for the full inventory.
*
* The connection pool stays alive (master + emex switches default-on) so
* follow-up code can use it for per-vehicle queries / schema-only diagrams
* without having to flip env again.
*/
@Injectable()
export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
private readonly logger = new Logger(EmexSourceDbService.name);
private pool: Pool | null = null;
private enabled = false;
private allowedCatalogs: Set<string> = new Set();
constructor(private readonly config: ConfigService) {}
@@ -33,6 +41,8 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
const masterEnabled = this.config.get<boolean>("catalogSource.enabled");
const emexEnabled = this.config.get<boolean>("catalogSource.emexEnabled");
const url = this.config.get<string>("catalogSource.emexUrl");
const allowed = this.config.get<string[]>("catalogSource.emexAllowedCatalogs") ?? [];
this.allowedCatalogs = new Set(allowed);
if (!masterEnabled || !emexEnabled || !url) {
this.logger.log(
`[emex-src] disabled (master=${masterEnabled}, emex=${emexEnabled}, urlSet=${Boolean(
@@ -48,7 +58,15 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
waitForConnections: true,
});
this.enabled = true;
this.logger.log("[emex-src] connected, lookup-first enabled");
if (this.allowedCatalogs.size === 0) {
this.logger.warn(
"[emex-src] connected but ALLOWLIST EMPTY — all fetchCategoryParts calls return null until EMEX_SOURCE_DB_ALLOWED_CATALOGS is populated (safety: catalog-wide bridge has 7-114x noise; see sase-emex-source-db-safety memory)",
);
} else {
this.logger.log(
`[emex-src] connected, allowlist: [${[...this.allowedCatalogs].sort().join(",")}]`,
);
}
}
async onModuleDestroy() {
@@ -70,6 +88,10 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
): Promise<EmexPartsResult | null> {
if (!this.enabled || !this.pool) return null;
if (!catalogCode || !gid) return null;
// Safety gate: the only path currently implemented (catalog-wide DISTINCT
// on vehicle_parts.group_id, ignoring vehicle_id) is unsafe — see class
// doc. Until a per-vehicle bridge lands, allow no catalogs.
if (!this.allowedCatalogs.has(catalogCode)) return null;
try {
// Resolve (catalog.id, part_group.id) in one round trip.