ALARM: Audio-Language Alignment for Reasoning Models

TL;DR

ALARM introduces a novel audio-language alignment method for reasoning models, enhancing auditory understanding and reasoning capabilities while maintaining textual performance at a low training cost.

Contribution

It proposes self-rephrasing for better audio reasoning alignment, fuses multiple encoders, and constructs a large multi-task corpus, leading to superior audio-reasoning performance.

Findings

Outperforms similarly sized models on audio-reasoning benchmarks

Achieves best open-source results on MMAU-speech and MMSU

Ranks third among all evaluated models

Abstract

Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.