Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages

TL;DR

This paper presents a new zero-resource benchmark for evaluating the code-switching capabilities of self-supervised speech encoders across multiple languages, highlighting the performance differences based on pre-training data.

Contribution

It introduces a novel benchmark dataset and baseline system for assessing code-switching in self-supervised speech models without requiring labeled data.

Findings

Multilingual pre-trained models outperform monolingual ones in code-switching tasks.

Speech encoders still have significant room for improvement in code-switching linguistic abilities.

Benchmark enables direct assessment of code-switching capabilities in a zero-resource setting.

Abstract

We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders. We showcase a baseline system of language modeling on discrete units to demonstrate how the code-switching abilities of speech encoders can be assessed in a zero-resource manner. Our experiments encompass a variety of well-known speech encoders, including Wav2vec 2.0, HuBERT, XLSR, etc. We examine the impact of pre-training languages and model size on benchmark performance. Notably, though our results demonstrate that speech encoders with multilingual pre-training, exemplified by XLSR, outperform monolingual variants (Wav2vec 2.0, HuBERT) in code-switching scenarios, there is still substantial room for improvement in their code-switching linguistic abilities.