Bridging the Modality Gap: Softly Discretizing Audio Representation for LLM-based Automatic Speech Recognition

TL;DR

This paper introduces a soft discretization technique using vector quantization to bridge the gap between continuous audio data and discrete LLM inputs, significantly enhancing LLM-based speech recognition especially out-of-domain.

Contribution

The paper proposes a novel soft discretization method that integrates VQ with LLMs for improved speech recognition performance.

Findings

Significant improvement over baseline in out-of-domain conditions

Effective alignment of audio representations with LLM inputs

Demonstrates potential of soft discretization as modality bridge

Abstract

One challenge of integrating speech input with large language models (LLMs) stems from the discrepancy between the continuous nature of audio data and the discrete token-based paradigm of LLMs. To mitigate this gap, we propose a method for integrating vector quantization (VQ) into LLM-based automatic speech recognition (ASR). Using the LLM embedding table as the VQ codebook, the VQ module aligns the continuous representations from the audio encoder with the discrete LLM inputs, enabling the LLM to operate on a discretized audio representation that better reflects the linguistic structure. We further create a soft "discretization" of the audio representation by updating the codebook and performing a weighted sum over the codebook embeddings. Empirical results demonstrate that our proposed method significantly improves upon the LLM-based ASR baseline, particularly in out-of-domain conditions. This work highlights the potential of soft discretization as a modality bridge in LLM-based ASR.