TransFER: Learning Relation-aware Facial Expression Representations with Transformers

TL;DR

TransFER introduces a relation-aware facial expression recognition model using transformers, employing novel attention dropping techniques to enhance local patch diversity and relation learning, leading to superior performance on FER benchmarks.

Contribution

The paper proposes TransFER, a transformer-based FER model with novel attention dropping methods (MAD and MSAD) to improve local patch diversity and relation modeling, advancing state-of-the-art results.

Findings

Outperforms existing FER methods on multiple benchmarks.

Effectively learns diverse local patches despite pose/viewpoint variations.

Enhances relation modeling among facial patches using transformers.

Abstract

Facial expression recognition (FER) has received increasing interest in computer vision. We propose the TransFER model which can learn rich relation-aware local representations. It mainly consists of three components: Multi-Attention Dropping (MAD), ViT-FER, and Multi-head Self-Attention Dropping (MSAD). First, local patches play an important role in distinguishing various expressions, however, few existing works can locate discriminative and diverse local patches. This can cause serious problems when some patches are invisible due to pose variations or viewpoint changes. To address this issue, the MAD is proposed to randomly drop an attention map. Consequently, models are pushed to explore diverse local patches adaptively. Second, to build rich relations between different local patches, the Vision Transformers (ViT) are used in FER, called ViT-FER. Since the global scope is used to reinforce each local patch, a better representation is obtained to boost the FER performance. Thirdly, the multi-head self-attention allows ViT to jointly attend to features from different information subspaces at different positions. Given no explicit guidance, however, multiple self-attentions may extract similar relations. To address this, the MSAD is proposed to randomly drop one self-attention module. As a result, models are forced to learn rich relations among diverse local patches. Our proposed TransFER model outperforms the state-of-the-art methods on several FER benchmarks, showing its effectiveness and usefulness.