Segmentwise Pruning in Audio-Language Models

TL;DR

This paper explores token pruning strategies for audio-language models, introducing a lightweight method that significantly reduces tokens while maintaining near-original performance levels on key benchmarks.

Contribution

It proposes a novel, efficient token pruning approach tailored for audio-language models, incorporating the time dimension for improved effectiveness.

Findings

Retains only a quarter of initial tokens with minimal performance loss

Achieves up to 2% decrease in CIDEr score on Clotho v2

Achieves up to 4% decrease in accuracy on MMAU

Abstract

Recent audio-language models have shown impressive performance across a wide range of audio tasks and are increasingly capable of handling long audio inputs. However, the computing costs in these models heavily depend on sequence length, which can become very large given the nature of audio data. In the vision-language domain, token pruning methods have proven effective in reducing token counts while preserving strong performance on standard benchmarks. In this work, we investigate the relevance and effectiveness of such token selection strategies in the context of audio-language models. We also improve them by proposing a lightweight strategy that takes the time dimension into account. While retaining only a quarter of the initial tokens, our approach results in a relative maximum decrease of 2% in CIDEr on Clotho v2 and a relative maximum decrease of 4% in accuracy on MMAU.