Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

TL;DR

This paper explores multi-task learning with lightweight neural networks for face identification, attribute classification, and emotion recognition, achieving near state-of-the-art results and improving accuracy in video-based emotion analysis.

Contribution

It introduces multi-task learning models based on MobileNet, EfficientNet, and RexNet architectures for facial attribute and expression recognition, demonstrating their effectiveness across multiple datasets.

Findings

Achieved near state-of-the-art results on UTKFace and AffectNet datasets.

Improved accuracy by 4.5% using trained models as feature extractors.

Models and code are publicly available for further research.

Abstract

In this paper, the multi-task learning of lightweight convolutional neural networks is studied for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. The necessity to fine-tune these networks to predict facial expressions is highlighted. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that they lead to near state-of-the-art results in age, gender and race recognition on the UTKFace dataset and emotion classification on the AffectNet dataset. Moreover, it is shown that the usage of the trained models as feature extractors of facial regions in video frames leads to 4.5% higher accuracy than the previously known state-of-the-art single models for the AFEW and the VGAF datasets from the EmotiW challenges. The models and source code are publicly available at https://github.com/HSE-asavchenko/face-emotion-recognition.