Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learning

TL;DR

Audio-Thinker is a reinforcement learning framework that enhances audio language models' reasoning abilities by dynamically guiding their thinking process, leading to improved performance on various audio question answering tasks.

Contribution

It introduces an adaptive think accuracy reward and an external reward model to improve reasoning strategies and consistency in audio language models.

Findings

Outperforms existing reasoning-oriented LALMs on benchmark tasks

Demonstrates superior reasoning and generalization capabilities

Enhances adaptability and consistency in audio reasoning

Abstract

Recent advancements in large language models, multimodal large language models, and large audio language models (LALMs) have significantly improved their reasoning capabilities through reinforcement learning with rule-based rewards. However, the explicit reasoning process has yet to show significant benefits for audio question answering, and effectively leveraging deep reasoning remains an open challenge, with LALMs still falling short of human-level auditory-language reasoning. To address these limitations, we propose Audio-Thinker, a reinforcement learning framework designed to enhance the reasoning capabilities of LALMs, with a focus on improving adaptability, consistency, and effectiveness. Our approach introduces an adaptive think accuracy reward, enabling the model to adjust its reasoning strategies based on task complexity dynamically. Furthermore, we incorporate an external reward model to evaluate the overall consistency and quality of the reasoning process, complemented by think-based rewards that help the model distinguish between valid and flawed reasoning paths during training. Experimental results demonstrate that our Audio-Thinker model outperforms existing reasoning-oriented LALMs across various benchmark tasks, exhibiting superior reasoning and generalization capabilities.