Robust Unsupervised Adaptation of a Speech Recogniser Using Entropy Minimisation and Speaker Codes

TL;DR

This paper introduces a robust unsupervised adaptation method for speech recognizers using entropy minimisation over multiple hypotheses and speaker codes, achieving significant WER improvements with minimal data.

Contribution

It presents a novel loss function based on conditional entropy over hypotheses and the use of speaker codes for effective unsupervised adaptation.

Findings

20% relative WER reduction on 1 minute of data

29% WER reduction on 10 minutes of data

Effective in noisy, far-field conditions

Abstract

Speech recognisers usually perform optimally only in a specific environment and need to be adapted to work well in another. For adaptation to a new speaker, there is often too little data for fine-tuning to be robust, and that data is usually unlabelled. This paper proposes a combination of approaches to make adaptation to a single minute of data robust. First, instead of estimating the adaptation parameters with cross-entropy on a single error-prone hypothesis or "pseudo-label", this paper proposes a novel loss function, the conditional entropy over complete hypotheses. Using multiple hypotheses makes adaptation more robust to errors in the initial recognition. Second, a "speaker code" characterises a speaker in a vector short enough that it requires little data to estimate. On a far-field noise-augmented version of Common Voice, the proposed scheme yields a 20% relative improvement in word error rate on one minute of adaptation data, increasing on 10 minutes to 29%.