Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach

TL;DR

This paper introduces speech-level and phoneme-level masking strategies in speech representation learning, which improve model performance by focusing on meaningful speech segments and entire phonemes during pre-training.

Contribution

The paper proposes novel speech-level and phoneme-level masking methods for pre-training speech models, enhancing downstream task performance over traditional random masking.

Findings

Speech-level masking emphasizes speech segments over silence.

Phoneme-level masking masks entire phonemes, not parts.

Both approaches improve phoneme classification and speaker recognition results.

Abstract

Recovering the masked speech frames is widely applied in speech representation learning. However, most of these models use random masking in the pre-training. In this work, we proposed two kinds of masking approaches: (1) speech-level masking, making the model to mask more speech segments than silence segments, (2) phoneme-level masking, forcing the model to mask the whole frames of the phoneme, instead of phoneme pieces. We pre-trained the model via these two approaches, and evaluated on two downstream tasks, phoneme classification and speaker recognition. The experiments demonstrated that the proposed masking approaches are beneficial to improve the performance of speech representation.