Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

TL;DR

AttentionPack is a novel framework that significantly enhances memory efficiency and inference speed in large vision-language models by employing attention-aware compression and decompression techniques, enabling longer contexts and faster processing.

Contribution

The paper introduces a new attention-aware optimization framework, AttentionPack, which reduces memory overhead and latency in large VLMs during decoding, especially for long-context multi-modal tasks.

Findings

Memory efficiency improved by up to 8x

Faster inference with higher batch sizes

Maintains output quality and longer context capabilities

Abstract

Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.