The NPU-ASLP System Description for Visual Speech Recognition in CNVSRC 2024

TL;DR

This paper describes the NPU-ASLP visual speech recognition system for CNVSRC 2024, utilizing advanced data augmentation and an end-to-end model architecture to achieve top results in multiple VSR challenge tracks.

Contribution

Introduction of an end-to-end VSR system with novel architecture components and extensive data augmentation, achieving state-of-the-art performance in CNVSRC 2024.

Findings

Achieved 30.47% CER in Single-Speaker VSR

Secured second place in open track of Single-Speaker Task

First place in three other tracks

Abstract

This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP (Team 237) in the second Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2024), engaging in all four tracks, including the fixed and open tracks of Single-Speaker VSR Task and Multi-Speaker VSR Task. In terms of data processing, we leverage the lip motion extractor from the baseline1 to produce multiscale video data. Besides, various augmentation techniques are applied during training, encompassing speed perturbation, random rotation, horizontal flipping, and color transformation. The VSR model adopts an end-to-end architecture with joint CTC/attention loss, introducing Enhanced ResNet3D visual frontend, E-Branchformer encoder, and Bi-directional Transformer decoder. Our approach yields a 30.47% CER for the Single-Speaker Task and 34.30% CER for the Multi-Speaker Task, securing second place in the open track of the Single-Speaker Task and first place in the other three tracks.