AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering

TL;DR

This paper introduces AVAM, a universal, training-free adaptive visual anchoring method for multimodal large language models, improving multi-image question answering by adaptively compressing irrelevant visual information and balancing global and local visual inputs.

Contribution

The paper proposes a novel adaptive visual anchoring strategy that can be integrated into existing MLLMs, enhancing accuracy and efficiency in MVQA tasks.

Findings

Consistent performance improvements across various MLLMs.

Effective adaptive compression reduces irrelevant visual redundancy.

Enhanced holistic image understanding through collaborative decoding.

Abstract

The advancement of Multimodal Large Language Models (MLLMs) has driven significant progress in Visual Question Answering (VQA), evolving from Single to Multi Image VQA (MVQA). However, the increased number of images in MVQA inevitably introduces substantial visual redundancy that is irrelevant to question answering, negatively impacting both accuracy and efficiency. To address this issue, existing methods lack flexibility in controlling the number of compressed visual tokens and tend to produce discrete visual fragments, which hinder MLLMs' ability to comprehend images holistically. In this paper, we propose a straightforward yet universal Adaptive Visual Anchoring strategy, which can be seamlessly integrated into existing MLLMs, offering significant accuracy improvements through adaptive compression. Meanwhile, to balance the results derived from both global and compressed visual input, we further introduce a novel collaborative decoding mechanism, enabling optimal performance. Extensive experiments validate the effectiveness of our method, demonstrating consistent performance improvements across various MLLMs. The code will be publicly available.