PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding

TL;DR

This paper introduces PMF-CEC, a novel phoneme-augmented multimodal fusion approach that enhances context-aware ASR error correction, especially for homophones, by improving differentiation and reducing bias, while maintaining fast inference.

Contribution

The paper proposes PMF-CEC, an advanced method that improves rare word correction in ASR by integrating phoneme information and a retention mechanism for better error detection.

Findings

PMF-CEC reduces word error rate more effectively than ED-CEC.

The method outperforms other biasing techniques in correcting homophones.

PMF-CEC maintains fast inference speed and robustness against large biasing lists.

Abstract

End-to-end automatic speech recognition (ASR) models often struggle to accurately recognize rare words. Previously, we introduced an ASR postprocessing method called error detection and context-aware error correction (ED-CEC), which leverages contextual information such as named entities and technical terms to improve the accuracy of ASR transcripts. Although ED-CEC achieves a notable success in correcting rare words, its accuracy remains low when dealing with rare words that have similar pronunciations but different spellings. To address this issue, we proposed a phoneme-augmented multimodal fusion method for context-aware error correction (PMF-CEC) method on the basis of ED-CEC, which allowed for better differentiation between target rare words and homophones. Additionally, we observed that the previous ASR error detection module suffers from overdetection. To mitigate this, we introduced a retention probability mechanism to filter out editing operations with confidence scores below a set threshold, preserving the original operation to improve error detection accuracy. Experiments conducted on five datasets demonstrated that our proposed PMF-CEC maintains reasonable inference speed while further reducing the biased word error rate compared with ED-CEC, showing a stronger advantage in correcting homophones. Moreover, our method outperforms other contextual biasing methods, and remains valuable compared with LLM-based methods in terms of faster inference and better robustness under large biasing lists.