WER we are and WER we think we are

TL;DR

This paper questions the reported low WERs of modern ASR systems by analyzing real-life conversational data and benchmarks, revealing higher error rates and proposing guidelines for better dataset creation.

Contribution

It critically evaluates current benchmark results, compares commercial ASR systems on real-life data, and offers guidelines for developing more representative datasets.

Findings

Reported WERs are significantly lower than those on real-life data.

Commercial ASR systems perform worse on spontaneous conversations.

Guidelines are proposed for creating high-quality, multi-domain datasets.

Abstract

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.