EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce

TL;DR

EComStage introduces a comprehensive, stage-wise benchmark for evaluating large language models in e-commerce, covering perception, planning, and action across customer and merchant scenarios, with insights into model strengths and weaknesses.

Contribution

The paper presents EComStage, a novel benchmark that assesses LLMs across multiple reasoning stages and diverse e-commerce scenarios, including merchant-oriented tasks, which were overlooked by prior benchmarks.

Findings

Evaluated over 30 LLMs revealing stage-specific strengths and weaknesses.

Identified performance gaps in perception, planning, and action stages.

Provided actionable insights for optimizing LLMs in real-world e-commerce applications.

Abstract

Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications to assist customer services in tasks such as product inquiries, recommendations, and order management. Existing benchmarks primarily evaluate whether these agents successfully complete the final task, overlooking the intermediate reasoning stages that are crucial for effective decision-making. To address this gap, we propose EComStage, a unified benchmark for evaluating agent-capable LLMs across the comprehensive stage-wise reasoning process: Perception (understanding user intent), Planning (formulating an action plan), and Action (executing the decision). EComStage evaluates LLMs through seven separate representative tasks spanning diverse e-commerce scenarios, with all samples human-annotated and quality-checked. Unlike prior benchmarks that focus only on customer-oriented interactions, EComStage also evaluates merchant-oriented scenarios, including promotion management, content review, and operational support relevant to real-world applications. We evaluate a wide range of over 30 LLMs, spanning from 1B to over 200B parameters, including open-source models and closed-source APIs, revealing stage/orientation-specific strengths and weaknesses. Our results provide fine-grained, actionable insights for designing and optimizing LLM-based agents in real-world e-commerce settings.