AFNet-M: Adaptive Fusion Network with Masks for 2D+3D Facial Expression Recognition

TL;DR

AFNet-M is an innovative deep learning model that adaptively fuses 2D and 3D facial features using masks and attention mechanisms, significantly improving facial expression recognition accuracy while reducing model complexity.

Contribution

The paper introduces a novel adaptive fusion network with mask-based attention and importance weighting for enhanced 2D+3D facial expression recognition.

Findings

Achieves state-of-the-art results on BU-3DFE and Bosphorus datasets.

Requires fewer parameters than existing models.

Effectively focuses on salient facial regions for better recognition.

Abstract

2D+3D facial expression recognition (FER) can effectively cope with illumination changes and pose variations by simultaneously merging 2D texture and more robust 3D depth information. Most deep learning-based approaches employ the simple fusion strategy that concatenates the multimodal features directly after fully-connected layers, without considering the different degrees of significance for each modality. Meanwhile, how to focus on both 2D and 3D local features in salient regions is still a great challenge. In this letter, we propose the adaptive fusion network with masks (AFNet-M) for 2D+3D FER. To enhance 2D and 3D local features, we take the masks annotating salient regions of the face as prior knowledge and design the mask attention module (MA) which can automatically learn two modulation vectors to adjust the feature maps. Moreover, we introduce a novel fusion strategy that can perform adaptive fusion at convolutional layers through the designed importance weights computing module (IWC). Experimental results demonstrate that our AFNet-M achieves the state-of-the-art performance on BU-3DFE and Bosphorus datasets and requires fewer parameters in comparison with other models.