JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

TL;DR

This paper proposes JEP-KD, a novel knowledge distillation method using joint-embedding predictive architecture to enhance visual speech recognition by better leveraging audio features and multimodal training.

Contribution

It introduces a generative network within JEP-KD that improves semantic feature extraction and aligns visual and audio modalities more effectively during training.

Findings

JEP-KD significantly improves VSR model performance.

The approach enhances robustness across different VSR platforms.

Demonstrates versatility in multimodal tasks.

Abstract

Visual Speech Recognition (VSR) tasks are generally recognized to have a lower theoretical performance ceiling than Automatic Speech Recognition (ASR), owing to the inherent limitations of conveying semantic information visually. To mitigate this challenge, this paper introduces an advanced knowledge distillation approach using a Joint-Embedding Predictive Architecture (JEPA), named JEP-KD, designed to more effectively utilize audio features during model training. Central to JEP-KD is the inclusion of a generative network within the embedding layer, which enhances the video encoder's capacity for semantic feature extraction and brings it into closer alignment with the audio features from a pre-trained ASR model's encoder. This approach aims to progressively reduce the performance gap between VSR and ASR. Moreover, a comprehensive multimodal, multistage training regimen for the JEP-KD framework is established, bolstering the robustness and efficacy of the training process. Experiment results demonstrate that JEP-KD significantly improves the performance of VSR models and demonstrates versatility across different VSR platforms, indicating its potential for broader application within other multimodal tasks.