SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

TL;DR

This paper investigates how semantic cues influence safety judgments in vision-language models, revealing their reliance on learned associations and exposing vulnerabilities in safety decision mechanisms.

Contribution

Introduces a semantic steering framework and SAVeS benchmark to evaluate and manipulate safety judgments in VLMs using semantic cues.

Findings

Safety decisions are highly sensitive to semantic cues.

VLMs rely on learned visual-linguistic associations rather than grounded understanding.

Automated steering can exploit these semantic vulnerabilities.

Abstract

Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.