Generative Models, Humans, Predictive Models: Who Is Worse at High-Stakes Decision Making?

TL;DR

This paper evaluates large language models in high-stakes recidivism prediction, revealing they are not suitable tools due to accuracy, bias, and agreement issues, especially when compared to humans and existing models.

Contribution

It systematically assesses LMs' performance, bias, and decision agreement in a high-stakes task, highlighting limitations and unintended effects of bias mitigation techniques.

Findings

LMs show limited accuracy in recidivism prediction.

Providing distractor information affects LM decisions.

Bias mitigation techniques can have unintended consequences.

Abstract

Despite strong advisory against it, large generative models (LMs) are already being used for decision making tasks that were previously done by predictive models or humans. We put popular LMs to the test in a high-stakes decision making task: recidivism prediction. Studying three closed-access and open-source LMs, we analyze the LMs not exclusively in terms of accuracy, but also in terms of agreement with (imperfect, noisy, and sometimes biased) human predictions or existing predictive models. We conduct experiments that assess how providing different types of information, including distractor information such as photos, can influence LM decisions. We also stress test techniques designed to either increase accuracy or mitigate bias in LMs, and find that some to have unintended consequences on LM decisions. Our results provide additional quantitative evidence to the wisdom that current LMs are not the right tools for these types of tasks.