AudioKV: KV Cache Eviction in Efficient Large Audio Language Models

TL;DR

AudioKV introduces a novel KV cache management framework for large audio-language models, prioritizing audio-critical attention heads and employing spectral smoothing to improve efficiency and accuracy during long-context inference.

Contribution

The paper presents a new method for KV cache eviction in LALMs that leverages semantic-acoustic alignment and spectral smoothing, outperforming existing compression techniques.

Findings

AudioKV maintains near-full accuracy at 40% compression with only 0.45% accuracy drop.

It significantly outperforms traditional cache compression methods in audio-language models.

The approach enhances computational efficiency while preserving model performance.

Abstract

Large Audio-Language Models (LALMs) have set new benchmarks in speech processing, yet their deployment is hindered by the memory footprint of the Key-Value (KV) cache during long-context inference. While general KV cache compression techniques excel in LLMs, they often fail in the audio domain by overlooking the intrinsic temporal continuity of acoustic signals. To bridge this gap, we propose AudioKV, a novel framework that robustly prioritizes audio-critical attention heads through a hardware-friendly semantic-acoustic alignment mechanism. Specifically, we identify these modality-specialized heads by analyzing attention scores in ASR tasks and dynamically allocate KV cache budgets preferentially to them. Furthermore, we introduce Spectral Score Smoothing (SSS), an FFT-based global filtering strategy designed to suppress high-frequency noise and recover smooth global trends from importance scores, ensuring more balanced token selection with unprecedented precision. Extensive evaluations across multiple LALMs, including Qwen and Gemma series, demonstrate that AudioKV significantly outperforms baselines while enhancing computational efficiency. Notably, at a 40% compression ratio, AudioKV maintains near-full accuracy on Qwen3-Omni-30B with only a 0.45% drop, whereas traditional methods suffer from catastrophic performance degradation and repetition. Our code will be released after acceptance.