SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?

TL;DR

This paper introduces SpeechGLUE, a benchmark to evaluate how well speech self-supervised learning models capture linguistic knowledge, revealing they acquire some linguistic understanding from unlabeled speech data.

Contribution

The paper presents SpeechGLUE, a new benchmark for assessing linguistic knowledge in speech SSL models, and evaluates their performance across diverse language understanding tasks.

Findings

Speech SSL models outperform baselines in linguistic tasks.

Speech SSL models are less effective than text-based SSL models.

Models can learn some linguistic knowledge from unlabeled speech.

Abstract

Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks, such as speech and speaker recognition. More recently, speech SSL models have also been shown to be beneficial in advancing spoken language understanding tasks, implying that the SSL models have the potential to learn not only acoustic but also linguistic information. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge. For this purpose, we introduce SpeechGLUE, a speech version of the General Language Understanding Evaluation (GLUE) benchmark. Since GLUE comprises a variety of natural language understanding tasks, SpeechGLUE can elucidate the degree of linguistic ability of speech SSL models. Experiments demonstrate that speech SSL models, although inferior to text-based SSL models, perform better than baselines, suggesting that they can acquire a certain amount of general linguistic knowledge from just unlabeled speech data.