SLM: Bridge the thin gap between speech and text foundation models

TL;DR

This paper introduces a joint speech and language model that efficiently adapts pretrained foundation models to perform a wide range of tasks, including zero-shot instruction-following, by training a small adapter while preserving the original capabilities.

Contribution

The paper proposes a simple adapter-based approach to bridge speech and text foundation models, enabling diverse tasks and zero-shot instruction-following with minimal additional training.

Findings

Achieves strong performance on speech recognition and translation tasks.

Enables zero-shot instruction-following for unseen tasks.

Efficient adaptation using only 1% of the foundation models' parameters.

Abstract

We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter with just 1\% (156M) of the foundation models' parameters. This adaptation not only leads SLM to achieve strong performance on conventional tasks such as speech recognition (ASR) and speech translation (AST), but also introduces the novel capability of zero-shot instruction-following for more diverse tasks: given a speech input and a text instruction, SLM is able to perform unseen generation tasks including contextual biasing ASR using real-time context, dialog generation, speech continuation, and question answering, etc. Our approach demonstrates that the representational gap between pretrained speech and language models might be narrower than one would expect, and can be bridged by a simple adaptation mechanism. As a result, SLM is not only efficient to train, but also inherits strong capabilities already acquired in foundation models of different modalities.