End-to-End Lip Synchronisation Based on Pattern Classification

TL;DR

This paper introduces an end-to-end neural network model that directly predicts audio-video synchronization offsets for talking face videos, outperforming previous proxy-task based methods on standard datasets.

Contribution

The work presents a novel end-to-end training approach that directly predicts synchronization offsets, treating the similarity matrix as an image for pattern classification.

Findings

Outperforms previous methods on LRS2 and LRS3 datasets

Joint training of feature extractor and classifier improves accuracy

Treating similarity matrix as an image enhances pattern recognition

Abstract

The goal of this work is to synchronise audio and video of a talking face using deep neural network models. Existing works have trained networks on proxy tasks such as cross-modal similarity learning, and then computed similarities between audio and video frames using a sliding window approach. While these methods demonstrate satisfactory performance, the networks are not trained directly on the task. To this end, we propose an end-to-end trained network that can directly predict the offset between an audio stream and the corresponding video stream. The similarity matrix between the two modalities is first computed from the features, then the inference of the offset can be considered to be a pattern recognition problem where the matrix is considered equivalent to an image. The feature extractor and the classifier are trained jointly. We demonstrate that the proposed approach outperforms the previous work by a large margin on LRS2 and LRS3 datasets.