Improving Character Error Rate Is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-box Acoustic Models

TL;DR

This paper introduces a training scheme for speech enhancement models that optimizes character error rate (CER) directly, even with black-box acoustic models, leading to improved ASR performance without affecting inference efficiency.

Contribution

The paper proposes a novel dual-DNN training approach that enables CER-centric optimization of speech enhancement models with black-box acoustic models, which was not previously possible.

Findings

Achieved 8.8% relative CER reduction with black-box acoustic models.

The method does not increase inference cost or alter network architecture.

Effective across various noise levels.

Abstract

A deep neural network (DNN)-based speech enhancement (SE) aiming to maximize the performance of an automatic speech recognition (ASR) system is proposed in this paper. In order to optimize the DNN-based SE model in terms of the character error rate (CER), which is one of the metric to evaluate the ASR system and generally non-differentiable, our method uses two DNNs: one for speech processing and one for mimicking the output CERs derived through an acoustic model (AM). Then both of DNNs are alternately optimized in the training phase. Even if the AM is a black-box, e.g., like one provided by a third-party, the proposed method enables the DNN-based SE model to be optimized in terms of the CER since the DNN mimicking the AM is differentiable. Consequently, it becomes feasible to build CER-centric SE model that has no negative effect, e.g., additional calculation cost and changing network architecture, on the inference phase since our method is merely a training scheme for the existing DNN-based methods. Experimental results show that our method improved CER by 8.8% relative derived through a black-box AM although certain noise levels are kept.