Speech-Omni-Lite: Portable Speech Interfaces for Vision-Language Models

TL;DR

Speech-Omni-Lite is a lightweight, cost-effective framework that extends vision-language models with speech understanding and generation capabilities using minimal training data, maintaining high performance and transferability.

Contribution

It introduces a novel plug-and-play module design and a low-cost data construction strategy to add speech functions to VL models without retraining the entire backbone.

Findings

Achieves high-quality spoken QA with only thousands of hours of speech data.

Maintains vision-language performance while adding speech capabilities.

Modules transfer effectively across different VL backbones.

Abstract

While large-scale omni-models have demonstrated impressive capabilities across various modalities, their strong performance heavily relies on massive multimodal data and incurs substantial computational costs. This work introduces Speech-Omni-Lite, a cost-efficient framework for extending pre-trained Visual-Language (VL) backbones with speech understanding and generation capabilities, while fully preserving the backbones' vision-language performance. Specifically, the VL backbone is equipped with two lightweight, trainable plug-and-play modules, a speech projector and a speech token generator, while keeping the VL backbone fully frozen. To mitigate the scarcity of spoken QA corpora, a low-cost data construction strategy is proposed to generate Question-Text Answer-Text-Speech (QTATS) data from existing ASR speech-text pairs, facilitating effective speech generation training. Experimental results show that, even with only thousands of hours of speech training data, Speech-Omni-Lite achieves excellent spoken QA performance, which is comparable to omni-models trained on millions of hours of speech data. Furthermore, the learned speech modules exhibit strong transferability across VL backbones.