SpeechMapper: Speech-to-text Embedding Projector for LLMs

TL;DR

SpeechMapper introduces a cost-efficient, robust method for integrating speech models with large language models, reducing overfitting and training costs while maintaining high performance across speech tasks.

Contribution

It proposes a novel two-stage training approach that pretrains speech embeddings separately and efficiently attaches them to LLMs, improving generalization and reducing computational requirements.

Findings

SpeechMapper rivals top speech LLMs without task-specific training.

It outperforms existing models in task-specific settings with less data and compute.

The approach is versatile across speech translation and question answering tasks.

Abstract

Current speech LLMs bridge speech foundation models to LLMs using projection layers, training all of these components on speech instruction data. This strategy is computationally intensive and susceptible to task and prompt overfitting. We present SpeechMapper, a cost-efficient speech-to-LLM-embedding training approach that mitigates overfitting, enabling more robust and generalizable models. Our model is first pretrained without the LLM on inexpensive hardware, and then efficiently attached to the target LLM via a brief 1K-step instruction tuning (IT) stage. Through experiments on speech translation and spoken question answering, we demonstrate the versatility of SpeechMapper's pretrained block, presenting results for both task-agnostic IT, an ASR-based adaptation strategy that does not train in the target task, and task-specific IT. In task-agnostic settings, Speechmapper rivals the best instruction-following speech LLM from IWSLT25, despite never being trained on these tasks, while in task-specific settings, it outperforms this model across many datasets, despite requiring less data and compute. Overall, SpeechMapper offers a practical and scalable approach for efficient, generalizable speech-LLM integration without large-scale IT.