CLASH: A Benchmark for Cross-Modal Contradiction Detection

TL;DR

CLASH is a new benchmark designed to evaluate and improve models' ability to detect contradictions between images and captions, addressing a critical gap in multimodal understanding and reliability.

Contribution

We introduce CLASH, the first comprehensive benchmark for cross-modal contradiction detection, including a large dataset with controlled contradictions and evaluation protocols.

Findings

State-of-the-art models struggle with contradiction detection.

Fine-tuning on CLASH improves model performance significantly.

Models exhibit modality biases and category-specific weaknesses.

Abstract

Contradictory multimodal inputs are common in real-world settings, yet existing benchmarks typically assume input consistency and fail to evaluate cross-modal contradiction detection - a fundamental capability for preventing hallucinations and ensuring reliability. We introduce CLASH, a novel benchmark for multimodal contradiction detection, featuring COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions. The samples include targeted questions evaluated in both multiple-choice and open-ended formats. The benchmark provides an extensive fine-tuning set filtered through automated quality checks, alongside a smaller human-verified diagnostic set. Our analysis of state-of-the-art models reveals substantial limitations in recognizing cross-modal conflicts, exposing systematic modality biases and category-specific weaknesses. Furthermore, we empirically demonstrate that targeted fine-tuning on CLASH substantially enhances conflict detection capabilities.