Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models

TL;DR

This paper introduces a novel audio-visual representation learning approach that uses cross-modal knowledge distillation from speech foundation models, significantly improving performance on various speech recognition tasks.

Contribution

The paper proposes a new method leveraging multi-layer representations from speech foundation models as teachers for cross-modal knowledge distillation in audio-visual learning.

Findings

Achieved superior or comparable results to state-of-the-art baselines.

Demonstrated effectiveness across multiple speech recognition tasks.

Validated the approach through extensive ablation studies and visualization.

Abstract

Audio-visual representation learning is crucial for advancing multimodal speech processing tasks, such as lipreading and audio-visual speech recognition. Recently, speech foundation models (SFMs) have shown remarkable generalization capabilities across various speech-related tasks. Building on this progress, we propose an audio-visual representation learning model that leverages cross-modal knowledge distillation from SFMs. In our method, SFMs serve as teachers, from which multi-layer hidden representations are extracted using clean audio inputs. We also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs. A novel representational knowledge distillation loss is employed to train the student during pretraining, which is also applied during finetuning to further enhance the performance on downstream tasks. Our experiments utilized both a self-supervised SFM, WavLM, and a supervised SFM, iFLYTEK-speech. The results demonstrated that our proposed method achieved superior or at least comparable performance to previous state-of-the-art baselines across automatic speech recognition, visual speech recognition, and audio-visual speech recognition tasks. Additionally, comprehensive ablation studies and the visualization of learned representations were conducted to evaluate the effectiveness of our proposed method.