TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading

TL;DR

TD3Net introduces a novel densely connected multi-dilated convolutional network for lipreading, effectively modeling complex temporal features with fewer parameters, achieving competitive accuracy on large datasets.

Contribution

The paper proposes TD3Net, a new architecture combining dense skip connections and multi-dilated convolutions to enhance temporal modeling in lipreading backend networks.

Findings

Achieves comparable or better accuracy than state-of-the-art methods.

Uses fewer parameters and lower computational cost.

Effectively captures diverse temporal features with preserved continuity.

Abstract

The word-level lipreading approach typically employs a two-stage framework with separate frontend and backend architectures to model dynamic lip movements. Each component has been extensively studied, and in the backend architecture, temporal convolutional networks (TCNs) have been widely adopted in state-of-the-art methods. Recently, dense skip connections have been introduced in TCNs to mitigate the limited density of the receptive field, thereby improving the modeling of complex temporal representations. However, their performance remains constrained owing to potential information loss regarding the continuous nature of lip movements, caused by blind spots in the receptive field. To address this limitation, we propose TD3Net, a temporal densely connected multi-dilated convolutional network that combines dense skip connections and multi-dilated temporal convolutions as the backend architecture. TD3Net covers a wide and dense receptive field without blind spots by applying different dilation factors to skip-connected features. Experimental results on a word-level lipreading task using two large publicly available datasets, Lip Reading in the Wild (LRW) and LRW-1000, indicate that the proposed method achieves performance comparable to state-of-the-art methods. It achieved higher accuracy with fewer parameters and lower floating-point operations compared to existing TCN-based backend architectures. Moreover, visualization results suggest that our approach effectively utilizes diverse temporal features while preserving temporal continuity, presenting notable advantages in lipreading systems. The code is available at our GitHub repository (https://github.com/Leebh-kor/TD3Net).