Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving

TL;DR

This paper introduces MTBI, a multi-task behavior imitation method with speech-text interleaving, significantly improving speech LLMs' generalization capabilities using less supervised speech data.

Contribution

The paper presents a novel MTBI approach that enhances speech LLMs' generalization by leveraging paired speech and transcripts with interleaving, requiring less supervised data.

Findings

MTBI outperforms state-of-the-art models on prompt and task generalization.

MTBI requires less supervised speech data than existing methods.

Interleaving improves alignment efficiency between speech and text.

Abstract

Large language models (LLMs) have shown remarkable generalization across tasks, leading to increased interest in integrating speech with LLMs. These speech LLMs (SLLMs) typically use supervised fine-tuning to align speech with text-based LLMs. However, the lack of annotated speech data across a wide range of tasks hinders alignment efficiency, resulting in poor generalization. To address these issues, we propose a novel multi-task 'behavior imitation' method with speech-text interleaving, called MTBI, which relies solely on paired speech and transcripts. By ensuring the LLM decoder generates equivalent responses to paired speech and text, we achieve a more generalized SLLM. Interleaving is used to further enhance alignment efficiency. We introduce a simple benchmark to evaluate prompt and task generalization across different models. Experimental results demonstrate that our MTBI outperforms SOTA SLLMs on both prompt and task generalization, while requiring less supervised speech data.