A-VL: Adaptive Attention for Large Vision-Language Models

TL;DR

A-VL introduces a novel adaptive attention mechanism for large vision-language models that dynamically manages attention across modalities, significantly reducing resource consumption while maintaining high performance.

Contribution

The paper presents A-VL, a tailored adaptive attention method for LVLMs that manages visual and language attention separately, improving efficiency during inference.

Findings

Reduces memory and computation in LVLMs

Maintains performance across multiple tasks and datasets

Outperforms existing adaptive attention methods

Abstract

The Large Vision-Language Model (LVLM) integrates computer vision and natural language processing techniques, offering substantial application potential. However, these models demand extensive resources during inference. Adaptive attention techniques can dynamically reduce computational redundancy and thus improve efficiency. Although current adaptive attention methods significantly reduce the memory requirements of Transformer-based language models, they are not tailored for LVLMs. We observe that LVLMs generate responses from both remote image tokens and local text tokens, and different modalities have different attention patterns. This observation inspires us to manage the attention for each modality separately. Specifically, for visual input, we store the cache of potentially useful information but only compute the most critical parts. For language input, we care more about local information. Based on our observation and analysis of vision-language attention patterns, we develop A-VL, a plug-and-play adaptive attention tailored for LVLM inference. Extensive evaluations on three vision-language tasks and five datasets show the effectiveness of our designs. Our approach A-VL outperforms existing adaptive attention methods in reducing memory usage and computational load without compromising performance.