Street-Level AI: Are Large Language Models Ready for Real-World Judgments?

TL;DR

This paper critically examines whether large language models (LLMs) are suitable for real-world societal judgments, revealing significant inconsistencies and questioning their readiness for high-stakes decision-making.

Contribution

It provides an empirical analysis of LLM alignment with human and societal judgments in resource allocation, highlighting their limitations and potential risks.

Findings

LLMs show high internal and inter-model inconsistency in prioritizations.

LLMs qualitatively align with lay human judgments in pairwise comparisons.

Current LLMs are not ready for high-stakes societal decision-making.

Abstract

A surge of recent work explores the ethical and societal implications of large-scale AI models that make "moral" judgments. Much of this literature focuses either on alignment with human judgments through various thought experiments or on the group fairness implications of AI judgments. However, the most immediate and likely use of AI is to help or fully replace the so-called street-level bureaucrats, the individuals deciding to allocate scarce social resources or approve benefits. There is a rich history underlying how principles of local justice determine how society decides on prioritization mechanisms in such domains. In this paper, we examine how well LLM judgments align with human judgments, as well as with socially and politically determined vulnerability scoring systems currently used in the domain of homelessness resource allocation. Crucially, we use real data on those needing services (maintaining strict confidentiality by only using local large models) to perform our analyses. We find that LLM prioritizations are extremely inconsistent in several ways: internally on different runs, between different LLMs, and between LLMs and the vulnerability scoring systems. At the same time, LLMs demonstrate qualitative consistency with lay human judgments in pairwise testing. Findings call into question the readiness of current generation AI systems for naive integration in high-stakes societal decision-making.