Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning

TL;DR

Echo introduces a novel audio-interleaved reasoning approach for large audio language models, enabling dynamic re-listening and improved comprehension of complex audio, surpassing existing methods on benchmark tasks.

Contribution

It proposes a two-stage training framework and structured data pipeline to enable LALMs to actively re-listen to audio during reasoning, advancing audio comprehension capabilities.

Findings

Echo outperforms existing models on audio comprehension benchmarks.

The approach demonstrates improved reasoning and generalization in complex audio tasks.

Audio-interleaved reasoning enhances sustained audio engagement and understanding.

Abstract

The maturation of Large Audio Language Models (LALMs) has raised growing expectations for them to comprehend complex audio much like humans. Current efforts primarily replicate text-based reasoning by contextualizing audio content through a one-time encoding, which introduces a critical information bottleneck. Drawing inspiration from human cognition, we propose audio-interleaved reasoning to break through this bottleneck. It treats audio as an active reasoning component, enabling sustained audio engagement and perception-grounded analysis. To instantiate it, we introduce a two-stage training framework, first teaching LALMs to localize salient audio segments through supervised fine-tuning, and then incentivizing proficient re-listening via reinforcement learning. In parallel, a structured data generation pipeline is developed to produce high-quality training data. Consequently, we present Echo, a LALM capable of dynamically re-listening to audio in demand during reasoning. On audio comprehension benchmarks, Echo achieves overall superiority in both challenging expert-level and general-purpose tasks. Comprehensive analysis further confirms the efficiency and generalizability of audio-interleaved reasoning, establishing it as a promising direction for advancing audio comprehension. Project page: https://github.com/wdqqdw/Echo.