UFO2: A unified pre-training framework for online and offline speech recognition

TL;DR

UFO2 introduces a unified pre-training framework that simplifies online and offline speech recognition training, significantly improving word error rates by leveraging joint training and shared representations.

Contribution

It is the first to unify online and offline speech recognition pre-training into a single process with shared models and objectives, enhancing performance with limited annotations.

Findings

UFO2 achieves 29.7% relative WER reduction in offline mode.

UFO2 achieves 18.2% relative WER reduction in online mode.

Unified training simplifies workflows and improves accuracy.

Abstract

In this paper, we propose a Unified pre-training Framework for Online and Offline (UFO2) Automatic Speech Recognition (ASR), which 1) simplifies the two separate training workflows for online and offline modes into one process, and 2) improves the Word Error Rate (WER) performance with limited utterance annotating. Specifically, we extend the conventional offline-mode Self-Supervised Learning (SSL)-based ASR approach to a unified manner, where the model training is conditioned on both the full-context and dynamic-chunked inputs. To enhance the pre-trained representation model, stop-gradient operation is applied to decouple the online-mode objectives to the quantizer. Moreover, in both the pre-training and the downstream fine-tuning stages, joint losses are proposed to train the unified model with full-weight sharing for the two modes. Experimental results on the LibriSpeech dataset show that UFO2 outperforms the SSL-based baseline method by 29.7% and 18.2% relative WER reduction in offline and online modes, respectively.