AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model

TL;DR

This paper introduces AKVSR, a novel framework that enhances visual speech recognition by leveraging compressed audio knowledge from a pretrained model, achieving state-of-the-art results without requiring audio input during inference.

Contribution

AKVSR uniquely uses large-scale pretrained audio models and a compact audio memory to improve VSR, enabling effective speech recognition without audio during testing.

Findings

Achieves state-of-the-art performance on LRS3 dataset.

Effectively leverages audio knowledge without audio input during inference.

Demonstrates significant improvement over previous VSR methods.

Abstract

Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this paper, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used LRS3 dataset.