kNN For Whisper And Its Effect On Bias And Speaker Adaptation

TL;DR

This paper explores how token-level k-nearest neighbor search ($k$NN) enhances Whisper speech recognition, particularly in addressing bias and speaker adaptation issues without retraining the model.

Contribution

It demonstrates the application of $k$NN to Whisper, analyzing its effects on bias, gender, accent, and age-related speaker adaptation in speech recognition.

Findings

$k$NN improves Whisper's recognition accuracy across different speaker groups.

Using $k$NN reduces bias related to gender, accent, and age.

The method offers a non-parametric alternative for speaker adaptation.

Abstract

Speech recognition performance varies by language, domain, and speaker characteristics such as accent, but fine-tuning a model on any of these categories may lead to catastrophic forgetting. Token-level $k$ nearest neighbor search ($k$NN), first proposed for neural sequence decoders for natural language generation (NLG) and machine translation (MT), is a non-parametric method that instead adapts using inference-time search in an external datastore, without training the underlying model. We show that Whisper, a transformer end-to-end speech model, benefits from $k$NN. We investigate the differences between the speech and text setups. We discuss implications for speaker adaptation, and analyze improvements by gender, accent, and age.