Voice Quality and Pitch Features in Transformer-Based Speech Recognition

TL;DR

This paper investigates the integration of voice quality and pitch features into Transformer-based speech recognition, demonstrating that separate convolutional front-ends for prosodic features improve accuracy and reduce word error rates on LibriSpeech.

Contribution

It introduces a novel architectural approach with separate convolutional front-ends for prosodic and spectral features, enhancing Transformer ASR performance.

Findings

Up to 5.6% relative Word Error Rate reduction on LibriSpeech

Separate convolutional front-ends outperform simple concatenation

Voice quality and pitch features improve ASR robustness

Abstract

Jitter and shimmer measurements have shown to be carriers of voice quality and prosodic information which enhance the performance of tasks like speaker recognition, diarization or automatic speech recognition (ASR). However, such features have been seldom used in the context of neural-based ASR, where spectral features often prevail. In this work, we study the effects of incorporating voice quality and pitch features altogether and separately to a Transformer-based ASR model, with the intuition that the attention mechanisms might exploit latent prosodic traits. For doing so, we propose separated convolutional front-ends for prosodic and spectral features, showing that this architectural choice yields better results than simple concatenation of such pitch and voice quality features to mel-spectrogram filterbanks. Furthermore, we find mean Word Error Rate relative reductions of up to 5.6% with the LibriSpeech benchmark. Such findings motivate further research on the application of prosody knowledge for increasing the robustness of Transformer-based ASR.