LLMs in social services: How does chatbot accuracy affect human accuracy?

TL;DR

This study evaluates how the accuracy of LLM-based chatbots influences caseworkers' ability to provide correct guidance in social services, revealing that higher chatbot accuracy significantly improves human performance but also introduces risks when suggestions are incorrect.

Contribution

The paper introduces a benchmark dataset of complex social service questions and experimentally demonstrates how chatbot accuracy levels impact caseworker performance in a real-world setting.

Findings

High-quality chatbots (96-100% accuracy) improve caseworker accuracy by 27 percentage points.

Incorrect chatbot suggestions can reduce caseworker accuracy by two-thirds on easy questions.

Caseworker performance gains plateau at high chatbot accuracy levels, indicating a limit to human reliance.

Abstract

Social service programs like the Supplemental Nutrition Assistance Program (SNAP, or food stamps) have eligibility rules that can be challenging to understand. For nonprofit caseworkers who often support clients in navigating a dozen or more complex programs, LLM-based chatbots may offer a means to provide better, faster help to clients whose situations may be less common. In this paper, we measure the potential effects of LLM-based chatbot suggestions on caseworkers' ability to provide accurate guidance. We first created a 770-question multiple-choice benchmark dataset of difficult, but realistic questions that a caseworker might receive. Next, using these benchmark questions and corresponding expert-verified answers, we conducted a randomized experiment with caseworkers recruited from nonprofit outreach organizations in Los Angeles. Caseworkers in the control condition did not see chatbot suggestions and had a mean accuracy of 49%. Caseworkers in the treatment condition saw chatbot suggestions that we artificially varied to range in aggregate accuracy from low (53%) to high (100%). Caseworker performance significantly improves as chatbot quality improves: high-quality chatbots (96-100% accurate) improved caseworker accuracy by 27 percentage points. At the question-level, incorrect chatbot suggestions substantially reduce caseworker accuracy, with a two-thirds reduction on easy questions where the control group performed best (without chatbot suggestions). Finally, improvements in caseworker accuracy level off as chatbot accuracy increases, a phenomenon that we call the "AI underreliance plateau," which is a concern for real-world deployment and highlights the importance of evaluating human-in-the-loop tools with their users.