Leveraging Pretrained Image-text Models for Improving Audio-Visual Learning

TL;DR

This paper introduces Segmental SpeechCLIP, a hierarchical speech encoder that improves audio-visual learning by leveraging pretrained image and text models like CLIP, DINO, and RoBERTa, achieving significant performance gains.

Contribution

It proposes a novel hierarchical speech encoder that directly learns word-like units and effectively utilizes pretrained multimodal and unimodal models for improved audio-visual learning.

Findings

Significant improvements over cascaded SpeechCLIP.

Audio-only systems perform close to audio-visual systems.

Mapping audio to CLIP vocabulary embeddings enhances semantic understanding.

Abstract

Visually grounded speech systems learn from paired images and their spoken captions. Recently, there have been attempts to utilize the visually grounded models trained from images and their corresponding text captions, such as CLIP, to improve speech-based visually grounded models' performance. However, the majority of these models only utilize the pretrained image encoder. Cascaded SpeechCLIP attempted to generate localized word-level information and utilize both the pretrained image and text encoders. Despite using both, they noticed a substantial drop in retrieval performance. We proposed Segmental SpeechCLIP which used a hierarchical segmental speech encoder to generate sequences of word-like units. We used the pretrained CLIP text encoder on top of these word-like unit representations and showed significant improvements over the cascaded variant of SpeechCLIP. Segmental SpeechCLIP directly learns the word embeddings as input to the CLIP text encoder bypassing the vocabulary embeddings. Here, we explore mapping audio to CLIP vocabulary embeddings via regularization and quantization. As our objective is to distill semantic information into the speech encoders, we explore the usage of large unimodal pretrained language models as the text encoders. Our method enables us to bridge image and text encoders e.g. DINO and RoBERTa trained with uni-modal data. Finally, we extend our framework in audio-only settings where only pairs of semantically related audio are available. Experiments show that audio-only systems perform close to the audio-visual system.