Can LLMs Help Allocate Public Health Resources? A Case Study on Childhood Lead Testing

TL;DR

This study evaluates whether large language models can effectively allocate public health resources for childhood lead testing, revealing significant limitations in their reasoning and data retrieval capabilities.

Contribution

The paper introduces a structured evaluation of LLMs' ability to allocate health resources based on vulnerability indicators, highlighting their current shortcomings.

Findings

LLMs often overlooked high-priority neighborhoods with high lead prevalence.

Accuracy of LLM resource allocation averaged 0.46, up to 0.66 with ChatGPT 5 Deep Research.

LLMs frequently cited outdated data and relied on non-empirical narratives.

Abstract

Public health agencies face critical challenges in identifying high-risk neighborhoods for childhood lead exposure with limited resources for outreach and intervention programs. To address this, we develop a Priority Score integrating untested children proportions, elevated blood lead prevalence, and public health coverage patterns to support optimized resource allocation decisions across 136 neighborhoods in Chicago, New York City, and Washington, D.C. We leverage these allocation tasks, which require integrating multiple vulnerability indicators and interpreting empirical evidence, to evaluate whether large language models (LLMs) with agentic reasoning and deep research capabilities can effectively allocate public health resources when presented with structured allocation scenarios. LLMs were tasked with distributing 1,000 test kits within each city based on neighborhood vulnerability indicators. Results reveal significant limitations: LLMs frequently overlooked neighborhoods with highest lead prevalence and largest proportions of untested children, such as West Englewood in Chicago, while allocating disproportionate resources to lower-priority areas like Hunts Point in New York City. Overall accuracy averaged 0.46, reaching a maximum of 0.66 with ChatGPT 5 Deep Research. Despite their marketed deep research capabilities, LLMs struggled with fundamental limitations in information retrieval and evidence-based reasoning, frequently citing outdated data and allowing non-empirical narratives about neighborhood conditions to override quantitative vulnerability indicators.