Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision

TL;DR

This paper evaluates recent foundation models on Mandarin-English code-switched speech tasks, showing self-supervised models perform nearly as well as supervised ones but still face challenges with intra-sentential switching.

Contribution

It provides a comprehensive assessment of large-scale self- and weakly-supervised models on code-switched speech, highlighting their strengths and areas for improvement.

Findings

Self-supervised models nearly match supervised performance

Models struggle with intra-sentential code-switching

Variants of Whisper remain effective in code-switching scenarios

Abstract

This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks.