Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly

TL;DR

This paper reveals that Multimodal Large Language Models often answer incorrectly despite understanding visuals, due to dataset biases and low visual attention, and proposes methods to improve their focus and accuracy.

Contribution

The paper introduces a new benchmark for error analysis in MLLMs and proposes techniques to enhance visual attention and diversify training data.

Findings

MLLMs show lower attention to visual tokens compared to question tokens.

Bias in instruction tuning datasets affects MLLMs' responses to indirect questions.

Proposed methods improve MLLMs' focus on visual content and answer accuracy.

Abstract

Multimodal Large Language Models (MLLMs) have displayed remarkable performance in multi-modal tasks, particularly in visual comprehension. However, we reveal that MLLMs often generate incorrect answers even when they understand the visual content. To this end, we manually construct a benchmark with 12 categories and design evaluation metrics that assess the degree of error in MLLM responses even when the visual content is seemingly understood. Based on this benchmark, we test 15 leading MLLMs and analyze the distribution of attention maps and logits of some MLLMs. Our investigation identifies two primary issues: 1) most instruction tuning datasets predominantly feature questions that 'directly' relate to the visual content, leading to a bias in MLLMs' responses to other indirect questions, and 2) MLLMs' attention to visual tokens is notably lower than to system and question tokens. We further observe that attention scores between questions and visual tokens as well as the model's confidence in the answers are lower in response to misleading questions than to straightforward ones. To address the first challenge, we introduce a paired positive and negative data construction pipeline to diversify the dataset. For the second challenge, we propose to enhance the model's focus on visual content during decoding by refining the text and visual prompt. For the text prompt, we propose a content guided refinement strategy that performs preliminary visual content analysis to generate structured information before answering the question. Additionally, we employ a visual attention refinement strategy that highlights question-relevant visual tokens to increase the model's attention to visual content that aligns with the question. Extensive experiments demonstrate that these challenges can be significantly mitigated with our proposed dataset and techniques.