On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods

TL;DR

This study investigates the stability of moral preferences over time using repeated surveys, revealing significant response variability that impacts the reliability of moral preference elicitation in AI systems.

Contribution

The paper demonstrates that moral preferences are unstable over time, challenging the assumption that single-survey methods accurately capture true moral values.

Findings

Participants changed responses 10-18% of the time on repeated moral questions.

Response instability correlates positively with response time and decision difficulty.

Implications for moral preference elicitation and stakeholder-AI value alignment are discussed.

Abstract

Preference elicitation frameworks feature heavily in the research on participatory ethical AI tools and provide a viable mechanism to enquire and incorporate the moral values of various stakeholders. As part of the elicitation process, surveys about moral preferences, opinions, and judgments are typically administered only once to each participant. This methodological practice is reasonable if participants' responses are stable over time such that, all other relevant factors being held constant, their responses today will be the same as their responses to the same questions at a later time. However, we do not know how often that is the case. It is possible that participants' true moral preferences change, are subject to temporary moods or whims, or are influenced by environmental factors we don't track. If participants' moral responses are unstable in such ways, it would raise important methodological and theoretical issues for how participants' true moral preferences, opinions, and judgments can be ascertained. We address this possibility here by asking the same survey participants the same moral questions about which patient should receive a kidney when only one is available ten times in ten different sessions over two weeks, varying only presentation order across sessions. We measured how often participants gave different responses to simple (Study One) and more complicated (Study Two) repeated scenarios. On average, the fraction of times participants changed their responses to controversial scenarios was around 10-18% across studies, and this instability is observed to have positive associations with response time and decision-making difficulty. We discuss the implications of these results for the efficacy of moral preference elicitation, highlighting the role of response instability in causing value misalignment between stakeholders and AI tools trained on their moral judgments.