Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization

TL;DR

This paper introduces an improved end-to-end speech recognition model that directly predicts formatted text with punctuation and capitalization by using knowledge distillation with multi-codebook vector quantization, achieving state-of-the-art results.

Contribution

It presents a novel fully formatted E2E ASR model utilizing knowledge distillation via multi-codebook vector quantization, reducing complexity and improving accuracy.

Findings

Significant reduction in word error rate (WER) compared to previous models.

Achieves state-of-the-art results on LibriSpeech-PC datasets.

Improves punctuation error rate (PER) in formatted transcription.

Abstract

Conventional automatic speech recognition (ASR) models typically produce outputs as normalized texts lacking punctuation and capitalization, necessitating post-processing models to enhance readability. This approach, however, introduces additional complexity and latency due to the cascaded system design. In response to this challenge, there is a growing trend to develop end-to-end (E2E) ASR models capable of directly predicting punctuation and capitalization, though this area remains underexplored. In this paper, we propose an enhanced fully formatted E2E ASR model that leverages knowledge distillation (KD) through multi-codebook vector quantization (MVQ). Experimental results demonstrate that our model significantly outperforms previous works in word error rate (WER) both with and without punctuation and capitalization, and in punctuation error rate (PER). Evaluations on the LibriSpeech-PC test-clean and test-other subsets show that our model achieves state-of-the-art results.