Towards Reliable Evaluation of Large Language Models for Multilingual and Multimodal E-Commerce Applications

TL;DR

This paper introduces EcomEval, a comprehensive multilingual and multimodal benchmark for evaluating large language models in e-commerce, addressing limitations of existing benchmarks by including diverse, real-world tasks across multiple languages and modalities.

Contribution

The paper presents EcomEval, a new benchmark with 37 tasks across six categories, incorporating authentic data, expert-reviewed answers, and multilingual, multimodal evaluation for e-commerce LLMs.

Findings

EcomEval covers 7 languages, including low-resource ones.

Benchmark includes 8 multimodal tasks reflecting real-world scenarios.

Evaluation scores enable challenge-oriented assessment across model sizes.

Abstract

Large Language Models (LLMs) excel on general-purpose NLP benchmarks, yet their capabilities in specialized domains remain underexplored. In e-commerce, existing evaluations-such as EcomInstruct, ChineseEcomQA, eCeLLM, and Shopping MMLU-suffer from limited task diversity (e.g., lacking product guidance and after-sales issues), limited task modalities (e.g., absence of multimodal data), synthetic or curated data, and a narrow focus on English and Chinese, leaving practitioners without reliable tools to assess models on complex, real-world shopping scenarios. We introduce EcomEval, a comprehensive multilingual and multimodal benchmark for evaluating LLMs in e-commerce. EcomEval covers six categories and 37 tasks (including 8 multimodal tasks), sourced primarily from authentic customer queries and transaction logs, reflecting the noisy and heterogeneous nature of real business interactions. To ensure both quality and scalability of reference answers, we adopt a semi-automatic pipeline in which large models draft candidate responses subsequently reviewed and modified by over 50 expert annotators with strong e-commerce and multilingual expertise. We define difficulty levels for each question and task category by averaging evaluation scores across models with different sizes and capabilities, enabling challenge-oriented and fine-grained assessment. EcomEval also spans seven languages-including five low-resource Southeast Asian languages-offering a multilingual perspective absent from prior work.