Dropout Regularization for Self-Supervised Learning of Transformer Encoder Speech Representation

TL;DR

This paper introduces two dropout regularization techniques for transformer-based self-supervised speech learning, which help prevent overfitting and improve downstream task performance.

Contribution

It proposes attention and layer dropout methods specifically designed for transformer speech models, enhancing their ability to utilize global information.

Findings

Improved phoneme classification accuracy.

Enhanced speaker recognition performance.

Dropout methods prevent overfitting in self-supervised learning.

Abstract

Predicting the altered acoustic frames is an effective way of self-supervised learning for speech representation. However, it is challenging to prevent the pretrained model from overfitting. In this paper, we proposed to introduce two dropout regularization methods into the pretraining of transformer encoder: (1) attention dropout, (2) layer dropout. Both of the two dropout methods encourage the model to utilize global speech information, and avoid just copying local spectrum features when reconstructing the masked frames. We evaluated the proposed methods on phoneme classification and speaker recognition tasks. The experiments demonstrate that our dropout approaches achieve competitive results, and improve the performance of classification accuracy on downstream tasks.