Punctuation Restoration for Singaporean Spoken Languages: English, Malay, and Mandarin

TL;DR

This paper introduces a novel slot-filling approach for punctuation restoration in multilingual ASR transcripts, achieving state-of-the-art results for Mandarin and strong performance for English and Malay in Singaporean languages.

Contribution

It is the first system to simultaneously restore punctuation for English, Mandarin, and Malay, using a masked punctuation prediction model and improved tokenization techniques.

Findings

Achieved 73.8% F1-score for Mandarin punctuation restoration.

State-of-the-art performance for Mandarin on IWSLT2022 dataset.

Effective use of Jieba tokenizer improved Mandarin results.

Abstract

This paper presents the work of restoring punctuation for ASR transcripts generated by multilingual ASR systems. The focus languages are English, Mandarin, and Malay which are three of the most popular languages in Singapore. To the best of our knowledge, this is the first system that can tackle punctuation restoration for these three languages simultaneously. Traditional approaches usually treat the task as a sequential labeling task, however, this work adopts a slot-filling approach that predicts the presence and type of punctuation marks at each word boundary. The approach is similar to the Masked-Language Model approach employed during the pre-training stages of BERT, but instead of predicting the masked word, our model predicts masked punctuation. Additionally, we find that using Jieba1 instead of only using the built-in SentencePiece tokenizer of XLM-R can significantly improve the performance of punctuating Mandarin transcripts. Experimental results on English and Mandarin IWSLT2022 datasets and Malay News show that the proposed approach achieved state-of-the-art results for Mandarin with 73.8% F1-score while maintaining a reasonable F1-score for English and Malay, i.e. 74.7% and 78% respectively. Our source code that allows reproducing the results and building a simple web-based application for demonstration purposes is available on Github.