AI Revolution on Chat Bot: Evidence from a Randomized Controlled Experiment
Sida Peng, Wojciech Swiatek, Allen Gao, Paul Cullivan, Haoge Chang

TL;DR
This paper reports on a field randomized controlled trial evaluating the effectiveness of large language model-based chatbots in providing unmonitored support services for information retrieval, highlighting real-world impacts of AI advancements.
Contribution
It provides empirical evidence from a field experiment on the practical effectiveness of LLM-based chatbots in support roles, filling a gap in real-world application studies.
Findings
LLM-based chatbots improve information retrieval efficiency.
Field experiment demonstrates practical benefits of AI in support services.
Results suggest potential for broader deployment in real-world settings.
Abstract
In recent years, generative AI has undergone major advancements, demonstrating significant promise in augmenting human productivity. Notably, large language models (LLM), with ChatGPT-4 as an example, have drawn considerable attention. Numerous articles have examined the impact of LLM-based tools on human productivity in lab settings and designed tasks or in observational studies. Despite recent advances, field experiments applying LLM-based tools in realistic settings are limited. This paper presents the findings of a field randomized controlled trial assessing the effectiveness of LLM-based tools in providing unmonitored support services for information retrieval.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Artificial Intelligence in Healthcare and Education · Topic Modeling
