Step-Audio 2 Technical Report

TL;DR

Step-Audio 2 is an advanced multi-modal large language model that integrates audio encoding, reasoning, and retrieval techniques to excel in speech recognition and conversational audio understanding.

Contribution

It introduces a novel end-to-end model combining a latent audio encoder, reinforcement learning, and retrieval-augmented generation for improved audio and speech understanding.

Findings

Achieves state-of-the-art performance on audio understanding benchmarks.

Effectively incorporates paralinguistic features like emotions and speaking styles.

Demonstrates robust conversational capabilities across diverse scenarios.

Abstract

This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.