Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

TL;DR

This paper explores the use of self-supervised learning combined with large language models to improve speech recognition accuracy for Chinese dialects, especially in low-resource settings, achieving state-of-the-art results.

Contribution

It introduces a novel approach of pre-training a Data2vec2 model on extensive unlabeled dialect speech data and systematically analyzes the impact of different projectors and LLMs.

Findings

Achieved state-of-the-art results on multiple dialect datasets.

Demonstrated effectiveness of self-supervised pre-training for Chinese dialect ASR.

Provided open-source tools for reproducibility.

Abstract

Large-scale training corpora have significantly improved the performance of ASR models. Unfortunately, due to the relative scarcity of data, Chinese accents and dialects remain a challenge for most ASR models. Recent advancements in self-supervised learning have shown that self-supervised pre-training, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios. We aim to investigate the effectiveness of this paradigm for Chinese dialects. Specifically, we pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours. Then, we systematically examine the impact of various projectors and LLMs on Mandarin, dialect, and accented speech recognition performance under this paradigm. Our method achieved SOTA results on multiple dialect datasets, including Kespeech. We will open-source our work to promote reproducible research