Are Language Models Sensitive to Morally Irrelevant Distractors?

TL;DR

This paper investigates whether large language models are influenced by morally irrelevant situational factors, revealing that such distractors can significantly alter their moral judgments, similar to human cognitive biases.

Contribution

The study introduces a novel multimodal dataset of moral distractors and demonstrates their impact on LLM moral judgments, highlighting the need for more nuanced evaluation methods.

Findings

Moral distractors can shift LLM judgments by over 30%.

LLMs exhibit sensitivity to morally irrelevant situational factors.

Current benchmarks may not fully capture LLMs' moral reasoning.

Abstract

With the rapid development and uptake of large language models (LLMs) across high-stakes settings, it is increasingly important to ensure that LLMs behave in ways that align with human values. Existing moral benchmarks prompt LLMs with value statements, moral scenarios, or psychological questionnaires, with the implicit underlying assumption that LLMs report somewhat stable moral preferences. However, moral psychology research has shown that human moral judgements are sensitive to morally irrelevant situational factors, such as smelling cinnamon rolls or the level of ambient noise, thereby challenging moral theories that assume the stability of human moral judgements. Here, we draw inspiration from this "situationist" view of moral psychology to evaluate whether LLMs exhibit similar cognitive moral biases to humans. We curate a novel multimodal dataset of 60 "moral distractors" from existing psychological datasets of emotionally-valenced images and narratives which have no moral relevance to the situation presented. After injecting these distractors into existing moral benchmarks to measure their effects on LLM responses, we find that moral distractors can shift the moral judgements of LLMs by over 30% even in low-ambiguity scenarios, highlighting the need for more contextual moral evaluations and more nuanced cognitive moral modeling of LLMs.