Evaluating Alignment of Behavioral Dispositions in LLMs

TL;DR

This paper introduces a framework using adapted psychological questionnaires and Situational Judgment Tests to evaluate how well large language models' behavioral dispositions align with human preferences and tendencies.

Contribution

It develops a novel methodology for assessing LLMs' behavioral dispositions through validated SJTs based on psychological questionnaires, enabling systematic comparison with human preferences.

Findings

LLMs often overconfidence in responses when human consensus is low.

Smaller models deviate significantly from human consensus in high-agreement scenarios.

Some LLMs encourage emotional expression contrary to human consensus on composure.

Abstract

As LLMs integrate into our daily lives, understanding their behavior becomes essential. In this work, we focus on behavioral dispositions$-$the underlying tendencies that shape responses in social contexts$-$and introduce a framework to study how closely the dispositions expressed by LLMs align with those of humans. Our approach is grounded in established psychological questionnaires but adapts them for LLMs by transforming human self-report statements into Situational Judgment Tests (SJTs). These SJTs assess behavior by eliciting natural recommendations in realistic user-assistant scenarios. We generate 2,500 SJTs, each validated by three human annotators, and collect preferred actions from 10 annotators per SJT, from a large pool of 550 participants. In a comprehensive study involving 25 LLMs, we find that models often do not reflect the distribution of human preferences: (1) in scenarios with low human consensus, LLMs consistently exhibit overconfidence in a single response; (2) when human consensus is high, smaller models deviate significantly, and even some frontier models do not reflect the consensus in 15-20% of cases; (3) traits can exhibit cross-LLM patterns, e.g., LLMs may encourage emotion expression in contexts where human consensus favors composure. Lastly, mapping psychometric statements directly to behavioral scenarios presents a unique opportunity to evaluate the predictive validity of self-reports, revealing considerable gaps between LLMs' stated values and their revealed behavior.