Speaker Adaptation for Quantised End-to-End ASR Models

TL;DR

This paper introduces P4Q, a novel speaker adaptation method for quantised end-to-end ASR models, significantly reducing WER while maintaining small model sizes suitable for edge deployment.

Contribution

It proposes a new strategy combining speaker adaptation with quantisation for end-to-end ASR models, improving performance on resource-constrained devices.

Findings

Achieved 15.1% and 23.3% relative WER reductions on Whisper and Conformer models.

Reduced model size by 7 times with minimal speaker-specific parameters.

Demonstrated effectiveness on LibriSpeech and TED-LIUM 3 datasets.

Abstract

End-to-end models have shown superior performance for automatic speech recognition (ASR). However, such models are often very large in size and thus challenging to deploy on resource-constrained edge devices. While quantisation can reduce model sizes, it can lead to increased word error rates (WERs). Although improved quantisation methods were proposed to address the issue of performance degradation, the fact that quantised models deployed on edge devices often target only on a small group of users is under-explored. To this end, we propose personalisation for quantised models (P4Q), a novel strategy that uses speaker adaptation (SA) to improve quantised end-to-end ASR models by fitting them to the characteristics of the target speakers. In this paper, we study the P4Q strategy based on Whisper and Conformer attention-based encoder-decoder (AED) end-to-end ASR models, which leverages a 4-bit block-wise NormalFloat4 (NF4) approach for quantisation and the low-rank adaptation (LoRA) approach for SA. Experimental results on the LibriSpeech and the TED-LIUM 3 corpora show that, with a 7-time reduction in model size and 1% extra speaker-specific parameters, 15.1% and 23.3% relative WER reductions were achieved on quantised Whisper and Conformer AED models respectively, comparing to the full precision models.