Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences

TL;DR

The paper introduces the Deep Value Benchmark (DVB), an evaluation framework to test if large language models genuinely learn human values or just surface-level preferences, revealing models' limited deep value generalization.

Contribution

The paper presents the DVB framework with a novel experimental design to measure models' ability to generalize deep human values over superficial preferences.

Findings

Average DVGR across models is 0.30, below chance.

Larger models tend to have slightly lower DVGR.

Models generally struggle to generalize deep values reliably.

Abstract

We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features -- for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR) -- the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.