Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation

TL;DR

This paper introduces a knowledge-enhanced facial expression recognition method that leverages vision-language models to improve discrimination by transforming facial expressions to neutral states using text embeddings.

Contribution

It proposes a novel approach that uses text embeddings and an emotional-to-neutral transformation to enhance FER accuracy beyond traditional label-based methods.

Findings

Significantly outperforms state-of-the-art FER methods on multiple datasets.

Effectively utilizes vision-language model knowledge for facial expression recognition.

Demonstrates robustness across different pre-trained visual encoders.

Abstract

Existing facial expression recognition (FER) methods typically fine-tune a pre-trained visual encoder using discrete labels. However, this form of supervision limits to specify the emotional concept of different facial expressions. In this paper, we observe that the rich knowledge in text embeddings, generated by vision-language models, is a promising alternative for learning discriminative facial expression representations. Inspired by this, we propose a novel knowledge-enhanced FER method with an emotional-to-neutral transformation. Specifically, we formulate the FER problem as a process to match the similarity between a facial expression representation and text embeddings. Then, we transform the facial expression representation to a neutral representation by simulating the difference in text embeddings from textual facial expression to textual neutral. Finally, a self-contrast objective is introduced to pull the facial expression representation closer to the textual facial expression, while pushing it farther from the neutral representation. We conduct evaluation with diverse pre-trained visual encoders including ResNet-18 and Swin-T on four challenging facial expression datasets. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art FER methods. The code will be publicly available.