Can Whisper perform speech-based in-context learning?

TL;DR

This paper explores the in-context learning capabilities of Whisper ASR models, introducing a speech-based in-context learning method that improves recognition accuracy without gradient updates, especially for dialects.

Contribution

It proposes a novel speech-based in-context learning approach for test-time adaptation of Whisper models, enhancing speech recognition across dialects without gradient descent.

Findings

Achieved an average 32.3% relative WER reduction on Chinese dialects.

Further improved WER reduction to 36.4% using k-NN-based example selection.

Demonstrated effective speaker adaptation and continuous speech recognition improvements.

Abstract

This paper investigates the in-context learning abilities of the Whisper automatic speech recognition (ASR) models released by OpenAI. A novel speech-based in-context learning (SICL) approach is proposed for test-time adaptation, which can reduce the word error rates (WERs) with only a small number of labelled speech samples without gradient descent. Language-level adaptation experiments using Chinese dialects showed that when applying SICL to isolated word ASR, consistent and considerable relative WER reductions can be achieved using Whisper models of any size on two dialects, which is on average 32.3%. A k-nearest-neighbours-based in-context example selection technique can be applied to further improve the efficiency of SICL, which can increase the average relative WER reduction to 36.4%. The findings are verified using speaker adaptation or continuous speech recognition tasks, and both achieved considerable relative WER reductions. Detailed quantitative analyses are also provided to shed light on SICL's adaptability to phonological variances and dialect-specific lexical nuances.