Generative AI in Action: Field Experimental Evidence from Alibaba's Customer Service Operations

TL;DR

This study uses a large-scale field experiment with Alibaba to show that generative AI assistants improve e-commerce customer service speed and subjective quality, especially for low performers, but may affect multitasking and overall quality for top agents.

Contribution

It provides empirical evidence on the impact of generative AI in real-world customer service, highlighting performance improvements and challenges in deployment strategies.

Findings

Gen AI significantly increased service speed and subjective customer ratings.

Low performers benefited most, reducing performance gaps.

Top performers experienced declines in subjective and objective service quality.

Abstract

In collaboration with Alibaba, this study leverages a large-scale field experiment to assess the impact of a generative AI assistant on worker performance in e-commerce after-sales service. Human agents providing digital chat support were randomly assigned with access to a gen AI assistant that offered two core functions: diagnosis of customer issues and solution proposals, presented as text messages. Agents retained discretion to adopt, modify, or disregard AI-generated messages. To evaluate gen AI's impact, we estimate both the intention-to-treat (ITT) effect of gen AI access and the local average treatment effect (LATE) of gen AI usage. Results show that gen AI significantly improved service speed, measured by issue identification time and chat duration. Gen AI also improved subjective service quality reflected in customer ratings and dissatisfaction rates, but it had no significant effect on objective service quality indicated by customer retrial rates. The performance improvements stemmed not only from automation but also from changes in the dynamics of agent-customer interactions: agent communication became more informative and efficient, while customers experienced reduced communication burdens. Low performers achieved the greatest improvements in both service speed and quality, narrowing the performance gap. In contrast, top-performing agents showed little improvement in service speed but experienced declines in both subjective and objective service quality. Evidence suggests that this decline results from increased multitasking tendency, proxied by longer shift-away times across concurrent chats, which slowed customer responses and raised abandonment and retrial rates. These findings suggest that gen AI reshapes work, demanding tailored deployment strategies.