Should I State or Should I Show? Aligning AI with Human Preferences

TL;DR

This study compares stated and revealed preferences in training AI to align with human choices, finding revealed preferences generally lead to more accurate predictions of human behavior.

Contribution

It provides empirical evidence that revealed preferences improve AI alignment with human choices over stated preferences, highlighting implementation challenges.

Findings

Revealed preferences yield more accurate AI predictions of human choices.

Subjects struggle to accurately translate preferences into written prompts.

When conflicting, AI aligns more with prompts despite lower accuracy.

Abstract

As AI agents become more autonomous, properly aligning their objectives with human preferences becomes increasingly important. We study how effectively an AI agent learns a human principal's preference in choice under risk via stated versus revealed preferences. We conduct an online experiment in which subjects state their preferences through written instructions ("prompts") and reveal them through choices in a series of binary lottery questions ("data"). We find that on average, an AI agent given revealed-preference data predicts subjects' choices more accurately than an AI agent given stated-preference prompts. Further analysis suggests that the gap is driven by subjects' difficulty in translating their own preferences into written instructions. When given a choice between which information source to give to an AI agent, a large portion of subjects fail to select the more informative one. Moreover, when predictions from the two sources conflict, we find that the AI agent aligns more frequently with the prompt, despite its lower accuracy. Overall, these results highlight the revealed preference approach as a powerful mechanism for communicating human preferences to AI agents, but its success depends on careful implementation.