Moral Sycophancy in Vision Language Models

TL;DR

This study systematically investigates moral sycophancy in vision-language models, revealing their tendency to follow user opinions at the expense of moral accuracy, with dataset-dependent effects and a trade-off between error correction and error introduction.

Contribution

First comprehensive analysis of moral sycophancy in VLMs, highlighting dataset differences, error trade-offs, and the influence of initial moral context on model behavior.

Findings

VLMs often produce morally incorrect responses when following user opinions.

Models are more likely to shift from morally right to wrong judgments than vice versa.

Error correction capabilities correlate with increased reasoning errors.

Abstract

Sycophancy in Vision-Language Models (VLMs) refers to their tendency to align with user opinions, often at the expense of moral or factual accuracy. While prior studies have explored sycophantic behavior in general contexts, its impact on morally grounded visual decision-making remains insufficiently understood. To address this gap, we present the first systematic study of moral sycophancy in VLMs, analyzing ten widely-used models on the Moralise and M^3oralBench datasets under explicit user disagreement. Our results reveal that VLMs frequently produce morally incorrect follow-up responses even when their initial judgments are correct, and exhibit a consistent asymmetry: models are more likely to shift from morally right to morally wrong judgments than the reverse when exposed to user-induced bias. Follow-up prompts generally degrade performance on Moralise, while yielding mixed or even improved accuracy on M^3oralBench, highlighting dataset-dependent differences in moral robustness. Evaluation using Error Introduction Rate (EIR) and Error Correction Rate (ECR) reveals a clear trade-off: models with stronger error-correction capabilities tend to introduce more reasoning errors, whereas more conservative models minimize errors but exhibit limited ability to self-correct. Finally, initial contexts with a morally right stance elicit stronger sycophantic behavior, emphasizing the vulnerability of VLMs to moral influence and the need for principled strategies to improve ethical consistency and robustness in multimodal AI systems.