PapersPlease: A Benchmark for Evaluating Motivational Values of Large Language Models Based on ERG Theory

TL;DR

PapersPlease is a benchmark with 3,700 moral dilemmas based on ERG theory, designed to evaluate how large language models prioritize human needs and exhibit biases in decision-making scenarios.

Contribution

This paper introduces a novel benchmark for assessing LLMs' decision-making aligned with ERG theory and analyzes implicit biases across different models and social identities.

Findings

LLMs show significant patterns in prioritizing human needs.

Models exhibit biases, with higher denial rates for marginalized identities.

Incorporating social identities affects model responsiveness.

Abstract

Evaluating the performance and biases of large language models (LLMs) through role-playing scenarios is becoming increasingly common, as LLMs often exhibit biased behaviors in these contexts. Building on this line of research, we introduce PapersPlease, a benchmark consisting of 3,700 moral dilemmas designed to investigate LLMs' decision-making in prioritizing various levels of human needs. In our setup, LLMs act as immigration inspectors deciding whether to approve or deny entry based on the short narratives of people. These narratives are constructed using the Existence, Relatedness, and Growth (ERG) theory, which categorizes human needs into three hierarchical levels. Our analysis of six LLMs reveals statistically significant patterns in decision-making, suggesting that LLMs encode implicit preferences. Additionally, our evaluation of the impact of incorporating social identities into the narratives shows varying responsiveness based on both motivational needs and identity cues, with some models exhibiting higher denial rates for marginalized identities. All data is publicly available at https://github.com/yeonsuuuu28/papers-please.