SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer

TL;DR

This paper introduces SwinLip, a lightweight and efficient visual speech encoder based on Swin Transformer, which improves lip reading accuracy and speed while reducing computational complexity, outperforming existing models on multiple datasets.

Contribution

The paper proposes a novel Swin Transformer-based lip reading encoder, SwinLip, that enhances performance and efficiency over traditional CNN-based models, with state-of-the-art results on Mandarin LRW-1000.

Findings

SwinLip improves lip reading accuracy on LRW and LRW-1000 datasets.

SwinLip reduces computational load compared to previous models.

SwinLip achieves state-of-the-art performance on Mandarin LRW-1000.

Abstract

This paper presents an efficient visual speech encoder for lip reading. While most recent lip reading studies have been based on the ResNet architecture and have achieved significant success, they are not sufficiently suitable for efficiently capturing lip reading features due to high computational complexity in modeling spatio-temporal information. Additionally, using a complex visual model not only increases the complexity of lip reading models but also induces delays in the overall network for multi-modal studies (e.g., audio-visual speech recognition, speech enhancement, and speech separation). To overcome the limitations of Convolutional Neural Network (CNN)-based models, we apply the hierarchical structure and window self-attention of the Swin Transformer to lip reading. We configure a new lightweight scale of the Swin Transformer suitable for processing lip reading data and present the SwinLip visual speech encoder, which efficiently reduces computational load by integrating modified Convolution-augmented Transformer (Conformer) temporal embeddings with conventional spatial embeddings in the hierarchical structure. Through extensive experiments, we have validated that our SwinLip successfully improves the performance and inference speed of the lip reading network when applied to various backbones for word and sentence recognition, reducing computational load. In particular, our SwinLip demonstrated robust performance in both English LRW and Mandarin LRW-1000 datasets and achieved state-of-the-art performance on the Mandarin LRW-1000 dataset with less computation compared to the existing state-of-the-art model.