End-to-End Speech Recognition and Disfluency Removal with Acoustic Language Model Pretraining

TL;DR

This paper demonstrates that large-scale audio language model pretraining enables end-to-end speech recognition systems to match or surpass two-stage models in disfluency removal, highlighting the importance of pretraining objectives.

Contribution

It shows that audio-based language models pretrained with weak self-supervised objectives can effectively perform disfluency removal, challenging the dominance of two-stage models.

Findings

Pretrained audio language models match or outperform two-stage models.

Pretraining objectives significantly impact disfluency removal performance.

End-to-end models benefit from recent large-scale audio pretraining advances.

Abstract

The SOTA in transcription of disfluent and conversational speech has in recent years favored two-stage models, with separate transcription and cleaning stages. We believe that previous attempts at end-to-end disfluency removal have fallen short because of the representational advantage that large-scale language model pretraining has given to lexical models. Until recently, the high dimensionality and limited availability of large audio datasets inhibited the development of large-scale self-supervised pretraining objectives for learning effective audio representations, giving a relative advantage to the two-stage approach, which utilises pretrained representations for lexical tokens. In light of recent successes in large scale audio pretraining, we revisit the performance comparison between two-stage and end-to-end model and find that audio based language models pretrained using weak self-supervised objectives match or exceed the performance of similarly trained two-stage models, and further, that the choice of pretraining objective substantially effects a model's ability to be adapted to the disfluency removal task.