Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition

TL;DR

Seed-ASR introduces an LLM-based speech recognition framework that effectively handles diverse speech signals and contextual information, outperforming traditional models across multiple languages, domains, and accents with significant error rate reductions.

Contribution

This work presents a novel LLM-based speech recognition model, Seed-ASR, leveraging audio-conditioned LLMs and stage-wise training to enhance diversity handling without extra language models.

Findings

Achieves 10%-40% reduction in word error rates on Chinese and English test sets.

Demonstrates superior performance across multiple domains, accents, and languages.

Supports scenario-specific deployment without additional language models.

Abstract

Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this work, we introduce Seed-ASR, a large language model (LLM) based speech recognition model. Seed-ASR is developed based on the framework of audio conditioned LLM (AcLLM), leveraging the capabilities of LLMs by inputting continuous speech representations together with contextual information into the LLM. Through stage-wise large-scale training and the elicitation of context-aware capabilities in LLM, Seed-ASR demonstrates significant improvement over end-to-end models on comprehensive evaluation sets, including multiple domains, accents/dialects and languages. Additionally, Seed-ASR can be further deployed to support specific needs in various scenarios without requiring extra language models. Compared to recently released large ASR models, Seed-ASR achieves 10%-40% reduction in word (or character, for Chinese) error rates on Chinese and English public test sets, further demonstrating its powerful performance.