Discriminative Self-training for Punctuation Prediction

TL;DR

This paper introduces a Discriminative Self-Training method with weighted loss and label smoothing to enhance punctuation prediction in speech transcripts, significantly outperforming existing models and establishing new state-of-the-art results.

Contribution

The paper presents a novel Discriminative Self-Training approach that effectively leverages unlabeled speech data for punctuation prediction, outperforming strong baselines and previous state-of-the-art models.

Findings

Achieves significant improvement over baselines on English and Chinese datasets.

Establishes a new state-of-the-art on the IWSLT2011 benchmark with a 1.3% absolute F1 gain.

Outperforms vanilla self-training and existing models like BERT, RoBERTa, and ELECTRA.

Abstract

Punctuation prediction for automatic speech recognition (ASR) output transcripts plays a crucial role for improving the readability of the ASR transcripts and for improving the performance of downstream natural language processing applications. However, achieving good performance on punctuation prediction often requires large amounts of labeled speech transcripts, which is expensive and laborious. In this paper, we propose a Discriminative Self-Training approach with weighted loss and discriminative label smoothing to exploit unlabeled speech transcripts. Experimental results on the English IWSLT2011 benchmark test set and an internal Chinese spoken language dataset demonstrate that the proposed approach achieves significant improvement on punctuation prediction accuracy over strong baselines including BERT, RoBERTa, and ELECTRA models. The proposed Discriminative Self-Training approach outperforms the vanilla self-training approach. We establish a new state-of-the-art (SOTA) on the IWSLT2011 test set, outperforming the current SOTA model by 1.3% absolute gain on F$_1$.