RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

TL;DR

RECOVER is a framework that improves entity recognition in ASR outputs by using multiple hypotheses and LLMs to correct errors, especially in domain-specific contexts, achieving significant reductions in error rates.

Contribution

It introduces a novel agentic correction framework that leverages multiple hypotheses and LLMs for robust entity correction in ASR, especially for rare and domain-specific terms.

Findings

Achieves 8-46% relative reduction in entity-phrase word error rate.

Increases recall of entity detection by up to 22 percentage points.

LLM-Select performs best in overall entity correction while preserving WER.

Abstract

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air traffic control, these errors are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely, 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, and LLM-Select. Evaluated across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points. The LLM-Select achieves the best overall performance in entity correction while maintaining overall WER.