Breaking the Visual Shortcuts in Multimodal Knowledge-Based Visual Question Answering

TL;DR

This paper exposes the reliance of current multimodal VQA models on visual shortcuts, introduces a new benchmark called RETINA to challenge this, and proposes MIMIR to improve model robustness by leveraging multiple related images.

Contribution

The paper introduces RETINA, a new benchmark that removes visual shortcuts, and MIMIR, a method that uses multiple images to enhance model understanding in VQA.

Findings

Models rely heavily on visual shortcuts in existing benchmarks.

RETINA significantly degrades existing model performance, revealing shortcut reliance.

MIMIR improves robustness by incorporating multiple related images.

Abstract

Existing Multimodal Knowledge-Based Visual Question Answering (MKB-VQA) benchmarks suffer from "visual shortcuts", as the query image typically matches the primary subject entity of the target document. We demonstrate that models can exploit these shortcuts, achieving comparable results using visual cues alone. To address this, we introduce Relational Entity Text-Image kNowledge Augmented (RETINA) benchmark, automatically constructed using an LLM-driven pipeline, consisting of 120k training and 2k human-curated test set. RETINA contains queries referencing secondary subjects (i.e. related entities) and pairs them with images of these related entities, removing the visual shortcut. When evaluated on RETINA existing models show significantly degraded performance, confirming their reliance on the shortcut. Furthermore, we propose Multi-Image MultImodal Retriever (MIMIR), which enriches document embeddings by augmenting images of multiple related entities, effectively handling RETINA, unlike prior work that uses only a single image per document. Our experiments validate the limitations of existing benchmarks and demonstrate the effectiveness of RETINA and MIMIR. Our project is available at: Project Page.