How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

TL;DR

This paper investigates how different components like Speech Foundation Models, adapters, and Large Language Models influence speech-to-text task performance, highlighting the critical role of the SFM and the variable impact of adapters.

Contribution

It systematically evaluates the effects of various adapters, SFMs, and LLMs on speech-to-text tasks, revealing the dominant influence of SFMs and the context-dependent impact of adapters.

Findings

SFM choice significantly affects downstream performance

Adapter impact varies depending on SFM and LLM used

Performance differences are notable between ASR and Speech Translation tasks

Abstract

The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.