RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

TL;DR

RAIR is a comprehensive, rule-aware benchmark dataset designed for evaluating the relevance assessment capabilities of models in e-commerce, covering general, long-tail, and visual salience aspects to push the limits of current models including GPT-5.

Contribution

The paper introduces RAIR, a new benchmark dataset with a standardized framework and diverse subsets for thorough evaluation of relevance models in e-commerce.

Findings

RAIR challenges current models, including GPT-5, with its diverse subsets.

Experiments show RAIR's effectiveness in revealing model limitations.

Provides industry-standard relevance evaluation metrics.

Abstract

Search relevance plays a central role in web e-commerce. While large language models (LLMs) have shown significant results on relevance task, existing benchmarks lack sufficient complexity for comprehensive model assessment, resulting in an absence of standardized relevance evaluation metrics across the industry. To address this limitation, we propose Rule-Aware benchmark with Image for Relevance assessment(RAIR), a Chinese dataset derived from real-world scenarios. RAIR established a standardized framework for relevance assessment and provides a set of universal rules, which forms the foundation for standardized evaluation. Additionally, RAIR analyzes essential capabilities required for current relevance models and introduces a comprehensive dataset consists of three subset: (1) a general subset with industry-balanced sampling to evaluate fundamental model competencies; (2) a long-tail hard subset focus on challenging cases to assess performance limits; (3) a visual salience subset for evaluating multimodal understanding capabilities. We conducted experiments on RAIR using 14 open and closed-source models. The results demonstrate that RAIR presents sufficient challenges even for GPT-5, which achieved the best performance. RAIR data are now available, serving as an industry benchmark for relevance assessment while providing new insights into general LLM and Visual Language Model(VLM) evaluation.