AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM

TL;DR

AlignFormer introduces a novel neural adapter to better match speech and text modalities in speech-LLMs, enabling improved zero-shot instruction-following and task performance by reducing sequence length mismatch.

Contribution

The paper proposes AlignFormer, a new neural adapter with CTC and dynamic-window QFormer layers, to effectively align speech and text modalities for speech-LLMs, especially in zero-shot tasks.

Findings

AlignFormer achieves near 100% instruction following rate with audio-first training.

Audio-first training outperforms instruction-first training in instruction following capability.

Speech-LLM with AlignFormer can perform zero-shot speech translation and question answering.

Abstract

Integrating speech into LLM (speech-LLM) has gaining increased attention recently. The mainstream solution is to connect a well-trained speech encoder and LLM with a neural adapter. However, the length mismatch between the speech and text sequences are not well handled, leading to imperfect modality matching between the speech and text. In this work, we propose a novel neural adapter, AlignFormer, to reduce the length gap between the two modalities. AlignFormer consists of CTC and dynamic-window QFormer layers, where the CTC alignment provides the dynamic window information for QFormer. The LLM backbone is frozen in training to preserve its text capability, especially the instruction following capability. When training with ASR data only, the proposed AlignFormer unlocks the instruction following capability for speech-LLM and the model can perform zero-shot speech translation (ST) and speech question answering (SQA) tasks. In fact, speech-LLM with AlignFormer can theoretically perform any tasks that the LLM backbone can deal with in the speech version. To evaluate the effectiveness of the instruction-following speech-LLM, we propose to use instruction following rate (IFR) and offer a systematic perspective for the IFR evaluation. In addition, we find that the audio position in training would affect the instruction following capability of speech-LLM and conduct an in-depth study on it. Our findings show that audio-first training achieves higher IFR than instruction-first training. The AlignFormer can achieve a near 100% IFR with audio-first training and game-changing improvements from zero to non-zero IFR on some evaluation data with instruction-first training. We believe that this study is a big step towards the perfect speech and text modality matching in the LLM embedding space.