# On the Audio-visual Synchronization for Lip-to-Speech Synthesis

## TL;DR

This paper identifies asynchrony issues in lip-to-speech datasets and models, proposing a synchronized model with an automatic mechanism to improve audio-visual alignment and evaluation accuracy.

## Contribution

The paper introduces a synchronized lip-to-speech model with an automatic synchronization mechanism to address data and model asynchrony issues.

## Key findings

- The proposed method improves synchronization in lip-to-speech synthesis.
- Evaluation metrics are enhanced with an audio alignment frontend.
- Synchronization training outperforms state-of-the-art approaches.

## Abstract

Most lip-to-speech (LTS) synthesis models are trained and evaluated under the assumption that the audio-video pairs in the dataset are perfectly synchronized. In this work, we show that the commonly used audio-visual datasets, such as GRID, TCD-TIMIT, and Lip2Wav, can have data asynchrony issues. Training lip-to-speech with such datasets may further cause the model asynchrony issue -- that is, the generated speech and the input video are out of sync. To address these asynchrony issues, we propose a synchronized lip-to-speech (SLTS) model with an automatic synchronization mechanism (ASM) to correct data asynchrony and penalize model asynchrony. We further demonstrate the limitation of the commonly adopted evaluation metrics for LTS with asynchronous test data and introduce an audio alignment frontend before the metrics sensitive to time alignment for better evaluation. We compare our method with state-of-the-art approaches on conventional and time-aligned metrics to show the benefits of synchronization training.