DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data

TL;DR

DeSTA2 introduces a method to enhance speech language models with paralinguistic understanding and instruction-following abilities without requiring speech instruction-tuning data, reducing annotation efforts and preserving language skills.

Contribution

It presents an automatic data creation process that injects speech understanding into SLMs while maintaining language capabilities, enabling effective speech tasks without speech instruction-tuning data.

Findings

Achieves strong performance on Dynamic-SUPERB and AIR-Bench-Chat benchmarks.

Demonstrates ability to follow complex instructions and reasoning tasks.

Reduces reliance on extensive annotated datasets.

Abstract

Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) by incorporating pre-trained speech models. However, these SLMs often undergo extensive speech instruction-tuning to bridge the gap between speech and text modalities. This requires significant annotation efforts and risks catastrophic forgetting of the original language capabilities. In this work, we present a simple yet effective automatic process for creating speech-text pair data that carefully injects speech paralinguistic understanding abilities into SLMs while preserving the inherent language capabilities of the text-based LLM. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data, achieving impressive performance on Dynamic-SUPERB and AIR-Bench-Chat benchmarks. Furthermore, our model exhibits the ability to follow complex instructions derived from LLMs, such as specific output formatting and chain-of-thought reasoning. Our approach not only enhances the versatility and effectiveness of SLMs but also reduces reliance on extensive annotated datasets, paving the way for more efficient and capable speech understanding systems.