ChineseEcomQA: A Scalable E-commerce Concept Evaluation Benchmark for Large Language Models

TL;DR

ChineseEcomQA is a scalable, question-answering benchmark designed to evaluate large language models' understanding of fundamental e-commerce concepts, addressing heterogeneity and balancing generality with specificity.

Contribution

The paper introduces ChineseEcomQA, a novel benchmark for e-commerce concept evaluation that combines LLM validation, RAG validation, and manual annotation to handle diverse tasks.

Findings

Mainstream LLMs show varying performance on ChineseEcomQA.

The benchmark effectively differentiates between general and specific e-commerce concepts.

ChineseEcomQA guides future domain-specific LLM evaluations.

Abstract

With the increasing use of Large Language Models (LLMs) in fields such as e-commerce, domain-specific concept evaluation benchmarks are crucial for assessing their domain capabilities. Existing LLMs may generate factually incorrect information within the complex e-commerce applications. Therefore, it is necessary to build an e-commerce concept benchmark. Existing benchmarks encounter two primary challenges: (1) handle the heterogeneous and diverse nature of tasks, (2) distinguish between generality and specificity within the e-commerce field. To address these problems, we propose \textbf{ChineseEcomQA}, a scalable question-answering benchmark focused on fundamental e-commerce concepts. ChineseEcomQA is built on three core characteristics: \textbf{Focus on Fundamental Concept}, \textbf{E-commerce Generality} and \textbf{E-commerce Expertise}. Fundamental concepts are designed to be applicable across a diverse array of e-commerce tasks, thus addressing the challenge of heterogeneity and diversity. Additionally, by carefully balancing generality and specificity, ChineseEcomQA effectively differentiates between broad e-commerce concepts, allowing for precise validation of domain capabilities. We achieve this through a scalable benchmark construction process that combines LLM validation, Retrieval-Augmented Generation (RAG) validation, and rigorous manual annotation. Based on ChineseEcomQA, we conduct extensive evaluations on mainstream LLMs and provide some valuable insights. We hope that ChineseEcomQA could guide future domain-specific evaluations, and facilitate broader LLM adoption in e-commerce applications.