ExpLLM: Towards Chain of Thought for Facial Expression Recognition

TL;DR

This paper introduces ExpLLM, a novel approach using large language models to generate a chain of thought for facial expression recognition, improving accuracy and interpretability over existing methods.

Contribution

The paper presents a new method called ExpLLM that leverages LLMs to produce detailed chains of thought for FER, incorporating AU interactions and emotional analysis.

Findings

ExpLLM outperforms state-of-the-art FER methods on RAF-DB and AffectNet datasets.

ExpLLM surpasses GPT-4o in expression CoT generation, especially for micro-expressions.

The approach enhances interpretability and accuracy in facial expression recognition.

Abstract

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails.