Enhancing conversational quality in language learning chatbots: An evaluation of GPT4 for ASR error correction

TL;DR

This paper investigates GPT4's ability to improve conversational quality in language learning chatbots by correcting ASR errors, showing it enhances dialogue despite higher WER, outperforming traditional methods without domain-specific training.

Contribution

It demonstrates GPT4's effectiveness in ASR error correction for educational chatbots, introducing new evaluation metrics and showing superior performance over standard methods.

Findings

GPT4 improves conversation quality despite increased WER

Semantic similarity and response sensibility metrics are effective for evaluation

GPT4 outperforms standard error correction methods without in-domain data

Abstract

The integration of natural language processing (NLP) technologies into educational applications has shown promising results, particularly in the language learning domain. Recently, many spoken open-domain chatbots have been used as speaking partners, helping language learners improve their language skills. However, one of the significant challenges is the high word-error-rate (WER) when recognizing non-native/non-fluent speech, which interrupts conversation flow and leads to disappointment for learners. This paper explores the use of GPT4 for ASR error correction in conversational settings. In addition to WER, we propose to use semantic textual similarity (STS) and next response sensibility (NRS) metrics to evaluate the impact of error correction models on the quality of the conversation. We find that transcriptions corrected by GPT4 lead to higher conversation quality, despite an increase in WER. GPT4 also outperforms standard error correction methods without the need for in-domain training data.