Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models

TL;DR

This paper investigates the stability of moral judgments in Vision-Language Models (VLMs) under various perturbations, revealing their high fragility and the need for robustness for responsible AI deployment.

Contribution

It introduces the concept of moral robustness in VLMs, systematically evaluates their vulnerabilities, and proposes lightweight interventions to improve moral stability.

Findings

VLMs' moral judgments are highly fragile under simple perturbations.

Stronger instruction-following models are more susceptible to persuasion.

Lightweight interventions can partially restore moral stability.

Abstract

Despite substantial efforts toward improving the moral alignment of Vision-Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs.