DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module

TL;DR

This paper introduces an efficient audio-visual speech recognition model that uses a Dual Conformer Interaction Module and a selective pre-training method to reduce parameters and improve performance in noisy environments.

Contribution

The paper presents a novel Dual Conformer Interaction Module and a pre-training strategy that together enhance AVSR efficiency and accuracy while reducing model complexity.

Findings

Significant reduction in model parameters without sacrificing accuracy.

Improved AVSR performance in noisy conditions.

Enhanced efficiency through selective parameter updating.

Abstract

Speech recognition is the technology that enables machines to interpret and process human speech, converting spoken language into text or commands. This technology is essential for applications such as virtual assistants, transcription services, and communication tools. The Audio-Visual Speech Recognition (AVSR) model enhances traditional speech recognition, particularly in noisy environments, by incorporating visual modalities like lip movements and facial expressions. While traditional AVSR models trained on large-scale datasets with numerous parameters can achieve remarkable accuracy, often surpassing human performance, they also come with high training costs and deployment challenges. To address these issues, we introduce an efficient AVSR model that reduces the number of parameters through the integration of a Dual Conformer Interaction Module (DCIM). In addition, we propose a pre-training method that further optimizes model performance by selectively updating parameters, leading to significant improvements in efficiency. Unlike conventional models that require the system to independently learn the hierarchical relationship between audio and visual modalities, our approach incorporates this distinction directly into the model architecture. This design enhances both efficiency and performance, resulting in a more practical and effective solution for AVSR tasks.