Adapting Speech Foundation Models for Unified Multimodal Speech Recognition with Large Language Models

TL;DR

This paper introduces UASR-LLM, a framework that adapts frozen speech foundation models with large language models for unified multimodal speech recognition, achieving superior results across various tasks and conditions.

Contribution

It proposes a novel two-stage training strategy with visual injection modules and LLM integration for unified multimodal speech recognition.

Findings

Outperforms state-of-the-art baselines in VSR, ASR, and AVSR

Effective across clean and noisy conditions

Generalizes well with various SFMs and LLMs

Abstract

While speech foundation models (SFMs) have demonstrated remarkable performance in audio-only tasks, their adaptation to multimodal scenarios remains underexplored. This work presents UASR-LLM, a novel framework that adapts frozen SFMs to unified visual speech recognition (VSR), automatic speech recognition (ASR), and audio-visual speech recognition (AVSR) by leveraging large language models (LLMs) as text decoders. Visual representations are injected into multiple SFM layers via visual injection modules, enabling multimodal fusion and unified representation learning. The augmented SFMs are connected to decoder-only LLMs through a feed-forward adaptor, where concatenated representations and instruction prompts guide transcription. We propose a two-stage training strategy consisting of visual injection pretraining followed by speech recognition finetuning. The pretraining stage aligns audio, visual, and audio-visual representations within the frozen SFM backbone, while the finetuning stage integrates LLMs for unified optimization across speech recognition tasks. Experimental results demonstrate superior performance over state-of-the-art baselines across VSR, ASR, and AVSR under both clean and noisy conditions. Ablation studies further confirm generalization across various SFMs and LLMs, validating the effectiveness of the proposed training strategy.