HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models

TL;DR

HeadRouter is a training-free, task-aware token pruning method for large audio language models that selectively retains crucial tokens by recognizing the importance of different attention heads across diverse audio tasks.

Contribution

It introduces a novel head-importance-aware token pruning approach that adapts to task-specific attention head behaviors, improving compression performance without additional training.

Findings

HeadRouter outperforms baseline models in token compression on AudioMarathon and MMAU-Pro benchmarks.

It retains 70% of tokens while surpassing the vanilla model's performance.

The method achieves over 101% of the baseline's accuracy on two large audio models.

Abstract

Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.