Connecting Speech Encoder and Large Language Model for ASR

TL;DR

This study compares connector structures for integrating speech encoders with large language models in ASR, demonstrating that Q-Former structures significantly improve word error rates and generalize well to out-of-domain data.

Contribution

The paper introduces a comparative analysis of connector structures and proposes a novel segment-level Q-Former for improved long speech recognition in ASR.

Findings

Q-Formers outperform other connectors in WER reduction.

Q-Formers generalize well to out-of-domain datasets.

Segment-level Q-Former improves recognition of long speech segments.

Abstract

The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies attempting to build integrated ASR models by connecting a speech encoder with an LLM. This paper presents a comparative study of three commonly used structures as connectors, including fully connected layers, multi-head cross-attention, and Q-Former. Speech encoders from the Whisper model series as well as LLMs from the Vicuna model series with different model sizes were studied. Experiments were performed on the commonly used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with Q-Formers demonstrated consistent and considerable word error rate (WER) reductions over LLMs with other connector structures. Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard. Moreover, a novel segment-level Q-Former is proposed to enable LLMs to recognise speech segments with a duration exceeding the limitation of the encoders, which results in 17% relative WER reductions over other connector structures on 90-second-long speech data.