Semantic Misalignment in Vision-Language Models under Perceptual Degradation

TL;DR

This paper investigates how vision-language models fail under perceptual degradation, revealing that small visual corruptions can cause significant semantic misalignments affecting safety-critical decisions.

Contribution

It introduces perception-realistic corruptions and language-level metrics to systematically evaluate semantic misalignment in VLMs under perception degradation.

Findings

VLMs exhibit severe semantic failures with moderate perception drops

Current robustness metrics do not predict semantic misalignments

Highlighting the disconnect between perception quality and semantic reliability

Abstract

Vision-Language Models (VLMs) are increasingly deployed in autonomous driving and embodied AI systems, where reliable perception is critical for safe semantic reasoning and decision-making. While recent VLMs demonstrate strong performance on multimodal benchmarks, their robustness to realistic perception degradation remains poorly understood. In this work, we systematically study semantic misalignment in VLMs under controlled degradation of upstream visual perception, using semantic segmentation on the Cityscapes dataset as a representative perception module. We introduce perception-realistic corruptions that induce only moderate drops in conventional segmentation metrics, yet observe severe failures in downstream VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments. To quantify these effects, we propose a set of language-level misalignment metrics that capture hallucination, critical omission, and safety misinterpretation, and analyze their relationship with segmentation quality across multiple contrastive and generative VLMs. Our results reveal a clear disconnect between pixel-level robustness and multimodal semantic reliability, highlighting a critical limitation of current VLM-based systems and motivating the need for evaluation frameworks that explicitly account for perception uncertainty in safety-critical applications.