PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech

TL;DR

ProfASR-Bench introduces a comprehensive benchmark for evaluating context-conditioned automatic speech recognition in high-stakes professional domains, revealing a significant gap in current models' utilization of available contextual information.

Contribution

The paper presents a new benchmark dataset and evaluation framework that measures how well ASR systems leverage contextual cues in professional speech, highlighting the underuse of available context.

Findings

Lightweight textual context has minimal impact on WER.

Adversarial prompts do not reliably degrade ASR performance.

Current systems exhibit a context-utilization gap (CUG) in high-stakes settings.

Abstract

Automatic Speech Recognition (ASR) in professional settings faces challenges that existing benchmarks underplay: dense domain terminology, formal register variation, and near-zero tolerance for critical entity errors. We present ProfASR-Bench, a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology. Each example pairs a natural-language prompt (domain cue and/or speaker profile) with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition. The corpus supports conventional ASR metrics alongside entity-aware scores and slice-wise reporting by accent and gender. Using representative families Whisper (encoder-decoder ASR) and Qwen-Omni (audio language models) under matched no-context, profile, domain+profile, oracle, and adversarial conditions, we find a consistent pattern: lightweight textual context produces little to no change in average word error rate (WER), even with oracle prompts, and adversarial prompts do not reliably degrade performance. We term this the context-utilization gap (CUG): current systems are nominally promptable yet underuse readily available side information. ProfASR-Bench provides a standardized context ladder, entity- and slice-aware reporting with confidence intervals, and a reproducible testbed for comparing fusion strategies across model families. Dataset: https://huggingface.co/datasets/prdeepakbabu/ProfASR-Bench Code: https://github.com/prdeepakbabu/ProfASR-Bench