Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?

TL;DR

This paper introduces ValueActionLens, a framework for evaluating the alignment between LLMs' stated values and their actions, revealing significant gaps and potential risks in current LLM value assessments.

Contribution

The study presents a novel evaluation framework and dataset to measure the value-action gap in LLMs across cultures and topics, highlighting the importance of context-aware assessments.

Findings

Alignment between LLMs' stated values and actions is often sub-optimal.

Leveraging reasoned explanations improves prediction of value-action gaps.

Significant variation in alignment across different models and scenarios.

Abstract

Existing research primarily evaluates the values of LLMs by examining their stated inclinations towards specific values. However, the "Value-Action Gap," a phenomenon rooted in environmental and social psychology, reveals discrepancies between individuals' stated values and their actions in real-world contexts. To what extent do LLMs exhibit a similar gap between their stated values and their actions informed by those values? This study introduces ValueActionLens, an evaluation framework to assess the alignment between LLMs' stated values and their value-informed actions. The framework encompasses the generation of a dataset comprising 14.8k value-informed actions across twelve cultures and eleven social topics, and two tasks to evaluate how well LLMs' stated value inclinations and value-informed actions align across three different alignment measures. Extensive experiments reveal that the alignment between LLMs' stated values and actions is sub-optimal, varying significantly across scenarios and models. Analysis of misaligned results identifies potential harms from certain value-action gaps. To predict the value-action gaps, we also uncover that leveraging reasoned explanations improves performance. These findings underscore the risks of relying solely on the LLMs' stated values to predict their behaviors and emphasize the importance of context-aware evaluations of LLM values and value-action gaps.