AI Alignment and Social Choice: Fundamental Limitations and Policy Implications

TL;DR

This paper explores the fundamental limitations of using reinforcement learning with human feedback (RLHF) for AI alignment, revealing that universal alignment respecting democratic norms and individual privacy is impossible, with significant policy implications.

Contribution

It demonstrates that RLHF cannot achieve universal AI alignment due to social choice impossibility results, highlighting the need for transparent governance and narrow alignment strategies.

Findings

Universal AI alignment via RLHF is impossible due to social choice limitations.

Aligning with all individuals' values violates private ethical preferences.

Transparent voting rules are essential for accountable AI governance.

Abstract

Aligning AI agents to human intentions and values is a key bottleneck in building safe and deployable AI applications. But whose values should AI agents be aligned with? Reinforcement learning with human feedback (RLHF) has emerged as the key framework for AI alignment. RLHF uses feedback from human reinforcers to fine-tune outputs; all widely deployed large language models (LLMs) use RLHF to align their outputs to human values. It is critical to understand the limitations of RLHF and consider policy challenges arising from these limitations. In this paper, we investigate a specific challenge in building RLHF systems that respect democratic norms. Building on impossibility results in social choice theory, we show that, under fairly broad assumptions, there is no unique voting protocol to universally align AI systems using RLHF through democratic processes. Further, we show that aligning AI agents with the values of all individuals will always violate certain private ethical preferences of an individual user i.e., universal AI alignment using RLHF is impossible. We discuss policy implications for the governance of AI systems built using RLHF: first, the need for mandating transparent voting rules to hold model builders accountable. Second, the need for model builders to focus on developing AI agents that are narrowly aligned to specific user groups.