MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond

TL;DR

The MERaLiON-SpeechEncoder is a new speech foundation model tailored for Singapore, trained on extensive unlabelled data, improving speech recognition benchmarks and supporting diverse speech applications in Southeast Asia.

Contribution

This paper introduces the MERaLiON-SpeechEncoder, a self-supervised speech model specifically designed for Singapore and Southeast Asian languages, with extensive training and evaluation.

Findings

Improves speech recognition benchmarks for Singapore speech data.

Remains competitive across ten diverse speech tasks.

Supports future multilingual expansion.

Abstract

This technical report describes the MERaLiON-SpeechEncoder, a foundation model designed to support a wide range of downstream speech applications. Developed as part of Singapore's National Multimodal Large Language Model Programme, the MERaLiON-SpeechEncoder is tailored to address the speech processing needs in Singapore and the surrounding Southeast Asian region. The model currently supports mainly English, including the variety spoken in Singapore. We are actively expanding our datasets to gradually cover other languages in subsequent releases. The MERaLiON-SpeechEncoder was pre-trained from scratch on 200,000 hours of unlabelled speech data using a self-supervised learning approach based on masked language modelling. We describe our training procedure and hyperparameter tuning experiments in detail below. Our evaluation demonstrates improvements to spontaneous and Singapore speech benchmarks for speech recognition, while remaining competitive to other state-of-the-art speech encoders across ten other speech tasks. We commit to releasing our model, supporting broader research endeavours, both in Singapore and beyond.