Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model

TL;DR

Step-Audio-AQAA is a fully end-to-end large audio language model that enhances natural speech response generation for audio query-answer tasks, integrating advanced tokenization, a large backbone, and speech synthesis.

Contribution

It introduces a novel end-to-end LALM with a dual-codebook tokenizer and a combined training approach, advancing audio query-answering capabilities.

Findings

Outperforms state-of-the-art LALMs on the StepEval-Audio-360 benchmark

Demonstrates superior speech control in AQAA tasks

Highlights the importance of token-based vocoder for performance

Abstract

Large Audio-Language Models (LALMs) have significantly advanced intelligent human-computer interaction, yet their reliance on text-based outputs limits their ability to generate natural speech responses directly, hindering seamless audio interactions. To address this, we introduce Step-Audio-AQAA, a fully end-to-end LALM designed for Audio Query-Audio Answer (AQAA) tasks. The model integrates a dual-codebook audio tokenizer for linguistic and semantic feature extraction, a 130-billion-parameter backbone LLM and a neural vocoder for high-fidelity speech synthesis. Our post-training approach employs interleaved token-output of text and audio to enhance semantic coherence and combines Direct Preference Optimization (DPO) with model merge to improve performance. Evaluations on the StepEval-Audio-360 benchmark demonstrate that Step-Audio-AQAA excels especially in speech control, outperforming the state-of-art LALMs in key areas. This work contributes a promising solution for end-to-end LALMs and highlights the critical role of token-based vocoder in enhancing overall performance for AQAA tasks.