SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model

TL;DR

SpeechCLIP introduces a framework that connects speech, images, and text using pre-trained models, enabling improved speech understanding and retrieval without relying on transcribed speech data.

Contribution

It is the first to integrate speech with vision and language models via images, reducing the need for costly transcribed speech data.

Findings

Outperforms previous methods on image-speech retrieval tasks

Achieves zero-shot speech-text retrieval without transcriptions

Can directly retrieve semantically related keywords from speech

Abstract

Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.