Self-Powered LLM Modality Expansion for Large Speech-Text Models

TL;DR

This paper introduces a self-powered large speech-text model that reduces speech anchor bias and enhances multimodal integration by leveraging model-generated speech recognition data for instruction tuning.

Contribution

It proposes a novel self-powered approach that uses augmented speech data from the model itself to improve instruction tuning and mitigate bias in large speech-text models.

Findings

Mitigates speech anchor bias effectively

Improves speech-text modality fusion

Enhances instruction-following performance

Abstract

Large language models (LLMs) exhibit remarkable performance across diverse tasks, indicating their potential for expansion into large speech-text models (LSMs) by integrating speech capabilities. Although unified speech-text pre-training and multimodal data instruction-tuning offer considerable benefits, these methods generally entail significant resource demands and tend to overfit specific tasks. This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning. We explore the instruction-following dynamics within LSMs, identifying a critical issue termed speech anchor bias-a tendency for LSMs to over-rely on speech inputs, mistakenly interpreting the entire speech modality as directives, thereby neglecting textual instructions. To counteract this bias, we introduce a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning. Our experiments across a range of speech-based tasks demonstrate that self-powered LSM mitigates speech anchor bias and improves the fusion of speech and text modalities in LSMs. Data, code and scripts are freely available at https://github.com/ytf-philp/Self-powered-LSM.