Value Alignment Tax: Measuring Value Trade-offs in LLM Alignment

TL;DR

This paper introduces VAT, a framework to measure how alignment interventions in LLMs cause shifts and trade-offs among interconnected values, revealing unintended effects often hidden in traditional evaluations.

Contribution

VAT provides a systematic way to quantify value trade-offs and system-level dynamics of value expression in LLMs under various alignment interventions.

Findings

Alignment causes uneven co-movement among values.

Traditional evaluations miss many unintended value shifts.

VAT reveals systematic trade-offs between target and non-target values.

Abstract

Existing work on value alignment typically characterizes value relations statically, ignoring how alignment interventions, such as prompting, fine-tuning, or preference optimization, reshape the broader value system. In practice, aligning a target value can implicitly shift other values, creating value trade-offs that remain largely unmeasured. We introduce VAT, a framework that quantifies value trade-offs by measuring how alignment-induced changes propagate across interconnected values relative to achieved on-target gain. VAT captures the system-level dynamics of value expression under alignment intervention, enabling evaluation of both intended improvements and unintended side effects. Using a controlled scenario-action dataset grounded in Schwartz value theory, we collect paired pre-post normative judgments and analyze alignment effects across models, values, and interventions. Results show that alignment often produces uneven and structured co-movement among values, revealing systematic trade-offs between target and non-target values. These effects are largely invisible under conventional target-only evaluation, but become evident via VAT, highlighting process-level alignment risks and offering new insights into the dynamic nature of value alignment in LLMs. Dataset and code are open-sourced.