ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark

TL;DR

This paper introduces ContextASR-Bench, a large-scale benchmark for evaluating the linguistic and contextual capabilities of speech recognition systems across diverse domains, highlighting the superior performance of Large Audio Language Models over traditional models.

Contribution

The paper presents a new comprehensive benchmark dataset and evaluation framework for assessing contextual and linguistic understanding in ASR systems, addressing a gap in existing benchmarks.

Findings

LALMs outperform conventional ASR models significantly.

The benchmark includes over 40,000 entries with 300,000+ named entities.

Evaluation reveals room for further improvements in context-aware ASR.

Abstract

Automatic Speech Recognition (ASR) has been extensively investigated, yet prior benchmarks have largely focused on assessing the acoustic robustness of ASR models, leaving evaluations of their linguistic capabilities relatively underexplored. This largely stems from the limited parameter sizes and training corpora of conventional ASR models, leaving them with insufficient world knowledge, which is crucial for accurately recognizing named entities across diverse domains. For instance, drug and treatment names in medicine or specialized technical terms in engineering. Recent breakthroughs in Large Language Models (LLMs) and corresponding Large Audio Language Models (LALMs) have markedly enhanced the visibility of advanced context modeling and general artificial intelligence capabilities. Leveraging LLMs, we envision a unified system capable of robust speech recognition across diverse real-world domains, yet existing benchmarks are inadequate for evaluating this objective. To address this gap, we propose ContextASR-Bench: a comprehensive, large-scale benchmark designed to assess the linguistic competence of ASR systems using corpora that feature numerous named entities across multiple domains. It encompasses up to 40,000 data entries with more than 300,000 named entities across over 10 domains. Beyond the audio and its transcription, each sample provides the domain it belongs to and a list of named entities it contains, which are referred to as the context. Based on this, we introduce three evaluation modes to assess how effectively models can exploit such context to improve ASR accuracy. Extensive evaluation on ContextASR-Bench highlights that LALMs outperform conventional ASR models by a large margin thanks to the strong world knowledge and context modeling of LLMs, yet there remains ample room for further improvement. The dataset and evaluation code have been released.