The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023

TL;DR

This paper presents a state-of-the-art visual speech recognition system developed by NPU-ASLP-LiAuto for CNVSRC 2023, utilizing advanced data augmentation and an end-to-end neural architecture to achieve top rankings in multiple VSR tasks.

Contribution

The paper introduces a novel VSR system with multi-scale video processing, extensive data augmentation, and an end-to-end model architecture that outperforms previous approaches in the CNVSRC 2023 challenge.

Findings

Achieved 34.76% CER in Single-Speaker VSR

Achieved 41.06% CER in Multi-Speaker VSR

Ranked first in all three participating tracks

Abstract

This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task. In terms of data processing, we leverage the lip motion extractor from the baseline1 to produce multi-scale video data. Besides, various augmentation techniques are applied during training, encompassing speed perturbation, random rotation, horizontal flipping, and color transformation. The VSR model adopts an end-to-end architecture with joint CTC/attention loss, comprising a ResNet3D visual frontend, an E-Branchformer encoder, and a Transformer decoder. Experiments show that our system achieves 34.76% CER for the Single-Speaker Task and 41.06% CER for the Multi-Speaker Task after multi-system fusion, ranking first place in all three tracks we participate.