AI Revolution on Chat Bot: Evidence from a Randomized Controlled Experiment

TL;DR

This paper reports on a field randomized controlled trial evaluating the effectiveness of large language model-based chatbots in providing unmonitored support services for information retrieval, highlighting real-world impacts of AI advancements.

Contribution

It provides empirical evidence from a field experiment on the practical effectiveness of LLM-based chatbots in support roles, filling a gap in real-world application studies.

Findings

LLM-based chatbots improve information retrieval efficiency.

Field experiment demonstrates practical benefits of AI in support services.

Results suggest potential for broader deployment in real-world settings.

Abstract

In recent years, generative AI has undergone major advancements, demonstrating significant promise in augmenting human productivity. Notably, large language models (LLM), with ChatGPT-4 as an example, have drawn considerable attention. Numerous articles have examined the impact of LLM-based tools on human productivity in lab settings and designed tasks or in observational studies. Despite recent advances, field experiments applying LLM-based tools in realistic settings are limited. This paper presents the findings of a field randomized controlled trial assessing the effectiveness of LLM-based tools in providing unmonitored support services for information retrieval.