FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference

TL;DR

FastAV introduces a novel token pruning framework for audio-visual large language models, significantly reducing computational costs while maintaining or enhancing performance through a two-stage attention-based pruning strategy.

Contribution

This work is the first to adapt token pruning specifically for AV-LLMs, employing a two-stage method that improves efficiency without relying on full attention maps.

Findings

Reduces FLOPs by over 40% in AV-LLMs

Maintains or improves model performance after pruning

Compatible with efficient attention mechanisms like FlashAttention

Abstract

In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in standard large language models (LLMs) and vision-language models (LVLMs), its application to AV-LLMs has received little attention, even though multimodal integration substantially increases their token demands. To address this gap, we introduce a pruning strategy that utilizes attention weights to identify tokens emphasized at different stages and estimates their importance. Building on this analysis, FastAV applies a two-stage pruning strategy: (1) global pruning in intermediate layers to remove broadly less influential tokens, and (2) fine pruning in later layers considering the impact on next token generation. Notably, our method does not rely on full attention maps, which makes it fully compatible with efficient attention mechanisms such as FlashAttention. Extensive experiments demonstrate that FastAV reduces FLOPs by more than 40% on two representative AV-LLMs, while preserving or even improving model performance.