Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation

TL;DR

This paper introduces Simignore, a method that enhances multimodal large language models' complex reasoning by filtering irrelevant image tokens based on similarity to text, improving interpretability and reasoning performance.

Contribution

The paper proposes a novel image token reduction technique, Simignore, that leverages similarity computation to improve complex reasoning in multimodal large language models.

Findings

Simignore improves reasoning accuracy on complex tasks

Filtering irrelevant image tokens enhances model interpretability

The method is validated through extensive experiments

Abstract

Multimodal large language models have experienced rapid growth, and numerous different models have emerged. The interpretability of LVLMs remains an under-explored area. Especially when faced with more complex tasks such as chain-of-thought reasoning, its internal mechanisms still resemble a black box that is difficult to decipher. By studying the interaction and information flow between images and text, we noticed that in models such as LLaVA1.5, image tokens that are semantically related to text are more likely to have information flow convergence in the LLM decoding layer, and these image tokens receive higher attention scores. However, those image tokens that are less relevant to the text do not have information flow convergence, and they only get very small attention scores. To efficiently utilize the image information, we propose a new image token reduction method, Simignore, which aims to improve the complex reasoning ability of LVLMs by computing the similarity between image and text embeddings and ignoring image tokens that are irrelevant and unimportant to the text. Through extensive experiments, we demonstrate the effectiveness of our method for complex reasoning tasks. The paper's source code can be accessed from \url{https://github.com/FanshuoZeng/Simignore}.