Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback

TL;DR

This paper investigates how human moral preferences evolve over time and the implications for AI alignment, revealing that preferences are often unstable and that current models struggle to adapt, raising challenges for trustworthy AI in high-stakes domains.

Contribution

It provides empirical evidence of moral preference instability over time and analyzes its impact on AI alignment, emphasizing the need to account for dynamic human values.

Findings

Participants change responses 6-20% of the time across sessions.

Significant shifts observed in participants' decision models over time.

Predictive performance of AI models decreases with preference and model instability.

Abstract

Alignment methods in moral domains seek to elicit moral preferences of human stakeholders and incorporate them into AI. This presupposes moral preferences as static targets, but such preferences often evolve over time. Proper alignment of AI to dynamic human preferences should ideally account for "legitimate" changes to moral reasoning, while ignoring changes related to attention deficits, cognitive biases, or other arbitrary factors. However, common AI alignment approaches largely neglect temporal changes in preferences, posing serious challenges to proper alignment, especially in high-stakes applications of AI, e.g., in healthcare domains, where misalignment can jeopardize the trustworthiness of the system and yield serious individual and societal harms. This work investigates the extent to which people's moral preferences change over time, and the impact of such changes on AI alignment. Our study is grounded in the kidney allocation domain, where we elicit responses to pairwise comparisons of hypothetical kidney transplant patients from over 400 participants across 3-5 sessions. We find that, on average, participants change their response to the same scenario presented at different times around 6-20% of the time (exhibiting "response instability"). Additionally, we observe significant shifts in several participants' retrofitted decision-making models over time (capturing "model instability"). The predictive performance of simple AI models decreases as a function of both response and model instability. Moreover, predictive performance diminishes over time, highlighting the importance of accounting for temporal changes in preferences during training. These findings raise fundamental normative and technical challenges relevant to AI alignment, highlighting the need to better understand the object of alignment (what to align to) when user preferences change significantly over time.