ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models

TL;DR

This paper introduces ChronosAudio, a comprehensive benchmark for evaluating long-audio understanding in Audio Large Language Models, revealing critical limitations in current models' ability to process extended audio sequences.

Contribution

It presents the first multi-task benchmark specifically designed for long-audio evaluation in ALLMs, including extensive experiments and analysis of model performance degradation.

Findings

ALLMs show over 90% performance drop from short to long contexts

Performance decline is due to attention diffusion and loss of temporal locality

Current mitigation strategies only recover about 50% of performance

Abstract

Although Audio Large Language Models (ALLMs) have witnessed substantial advancements, their long audio understanding capabilities remain unexplored. A plethora of benchmarks have been proposed for general audio tasks, they predominantly focus on short-form clips, leaving without a consensus on evaluating ALLMs over extended durations. This paper proposes ChronosAudio, the first multi-task benchmark tailored for long-audio understanding in ALLMs. It encompasses six major task categories and comprises 36,000 test instances totaling over 200 hours audio, stratified into short, middle, and long-form categories to comprehensively evaluate length generalization. Extensive experiments on 16 state-of-the-art models using ChronosAudio yield three critical findings: 1.Precipitous Long-Context Collapse: ALLMs exhibit a severe inability to sustain performance, with the transition from short to long contexts triggering a staggering performance degradation of over 90% in specific tasks. 2.Structural Attention Dilution: Performance degradation stems from a fundamental failure in maintaining temporal locality; attention mechanisms suffer from significant diffusion in later sequences. 3.Restorative Ceiling of Mitigation: Current strategies only offer 50% recovery. These findings reveal significant challenges in long-audio, underscoring the urgent need for approaches to achieve robust, document-level audio reasoning.