Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow

TL;DR

This paper introduces a method to improve vision-language models by modulating information flow based on token importance, leading to better task performance across multiple datasets.

Contribution

The authors propose a token dynamics-based approach to selectively enhance relevant visual information during inference in VLMs.

Findings

Significant performance improvements on visual question answering and grounding tasks.

Effective identification of important visual tokens using activation patterns.

Enhanced perception accuracy without retraining the models.

Abstract

Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/.