ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

TL;DR

ECom-Bench is a new benchmark framework designed to evaluate multimodal large language model agents in complex, real-world e-commerce customer support scenarios, highlighting current model limitations.

Contribution

It introduces the first comprehensive benchmark with realistic tasks and user simulations for assessing LLM agents in e-commerce customer support.

Findings

GPT-4o achieves only 10-20% pass rate on the benchmark.

The benchmark reflects high complexity of real-world e-commerce support.

ECom-Bench is publicly available for further research.

Abstract

In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.