Auto Review: Second Stage Error Detection for Highly Accurate Information Extraction from Phone Conversations

TL;DR

Auto Review is a system that automates the second-stage error detection in healthcare phone conversations, significantly reducing manual review effort while maintaining high accuracy despite challenges from noisy transcripts and domain-specific jargon.

Contribution

We introduce Auto Review, a novel postprocessing pipeline that leverages multiple ASR outputs and pseudo-labeling to improve information extraction accuracy in noisy healthcare phone call transcripts.

Findings

Substantial accuracy improvements with multiple ASR alternatives.

Effective pseudo-labeling without manual transcript correction.

Enhanced efficiency of the auto review process.

Abstract

Automating benefit verification phone calls saves time in healthcare and helps patients receive treatment faster. It is critical to obtain highly accurate information in these phone calls, as it can affect a patient's healthcare journey. Given the noise in phone call transcripts, we have a two-stage system that involves a post-call review phase for potentially noisy fields, where human reviewers manually verify the extracted data$\unicode{x2013}$a labor-intensive task. To automate this stage, we introduce Auto Review, which significantly reduces manual effort while maintaining a high bar for accuracy. This system, being highly reliant on call transcripts, suffers a performance bottleneck due to automatic speech recognition (ASR) issues. This problem is further exacerbated by the use of domain-specific jargon in the calls. In this work, we propose a second-stage postprocessing pipeline for accurate information extraction. We improve accuracy by using multiple ASR alternatives and a pseudo-labeling approach that does not require manually corrected transcripts. Experiments with general-purpose large language models and feature-based model pipelines demonstrate substantial improvements in the quality of corrected call transcripts, thereby enhancing the efficiency of Auto Review.