Analyzing And Improving Neural Speaker Embeddings for ASR

TL;DR

This paper explores integrating neural speaker embeddings into a conformer-based hybrid ASR system, achieving comparable performance to traditional methods and improving WER through model and embedding enhancements.

Contribution

It introduces an improved embedding extraction pipeline and integration method, demonstrating effective use of neural speaker embeddings in hybrid ASR systems with significant WER reductions.

Findings

Neural speaker embeddings can match i-vector performance in ASR.

Switching to a one cycle learning schedule reduces training time by 17%.

Adding speaker embeddings improves WER by approximately 3% on Hub5'00.

Abstract

Neural speaker embeddings encode the speaker's speech characteristics through a DNN model and are prevalent for speaker verification tasks. However, few studies have investigated the usage of neural speaker embeddings for an ASR system. In this work, we present our efforts w.r.t integrating neural speaker embeddings into a conformer based hybrid HMM ASR system. For ASR, our improved embedding extraction pipeline in combination with the Weighted-Simple-Add integration method results in x-vector and c-vector reaching on par performance with i-vectors. We further compare and analyze different speaker embeddings. We present our acoustic model improvements obtained by switching from newbob learning rate schedule to one cycle learning schedule resulting in a ~3% relative WER reduction on Switchboard, additionally reducing the overall training time by 17%. By further adding neural speaker embeddings, we gain additional ~3% relative WER improvement on Hub5'00. Our best Conformer-based hybrid ASR system with speaker embeddings achieves 9.0% WER on Hub5'00 and Hub5'01 with training on SWB 300h.