Hand-Assisted Expression Recognition Method from Synthetic Images at the Fourth ABAW Challenge

TL;DR

This paper introduces a hand-assisted expression recognition approach that leverages synthetic images to improve facial expression analysis, addressing the domain gap between synthetic and real images in the ABAW Challenge.

Contribution

It proposes a novel method combining expression recognition and hand prediction modules with decision mode and post-pruning to enhance accuracy.

Findings

Improved F1 score on synthetic images

Effective reduction of synthetic-real domain gap

Enhanced expression recognition accuracy

Abstract

Learning from synthetic images plays an important role in facial expression recognition task due to the difficulties of labeling the real images, and it is challenging because of the gap between the synthetic images and real images. The fourth Affective Behavior Analysis in-the-wild Competition raises the challenge and provides the synthetic images generated from Aff-Wild2 dataset. In this paper, we propose a hand-assisted expression recognition method to reduce the gap between the synthetic data and real data. Our method consists of two parts: expression recognition module and hand prediction module. Expression recognition module extracts expression information and hand prediction module predicts whether the image contains hands. Decision mode is used to combine the results of two modules, and post-pruning is used to improve the result. F1 score is used to verify the effectiveness of our method.