Comparing Discrete and Continuous Space LLMs for Speech Recognition

TL;DR

This paper provides the first comprehensive comparison of discrete and continuous speech representations in LLM-based ASR, analyzing various training methods and model architectures to improve speech recognition accuracy.

Contribution

It introduces a detailed classification of speech representations and models, and presents an open-source approach achieving state-of-the-art WER on LibriSpeech.

Findings

Achieved a WER of 1.69% on LibriSpeech with HuBERT encoder

Provided a comparative analysis of discrete vs. continuous speech representations

First extensive study of speech representations in LLM-based ASR

Abstract

This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.