Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer

TL;DR

This paper introduces Exp-CLIP, a novel method that enhances zero-shot facial expression recognition by transferring task-specific knowledge from large language models to improve generalization on unseen data.

Contribution

The work proposes a new approach to improve zero-shot FER by integrating LLM-derived semantic knowledge into vision-language models through a specialized projection head.

Findings

Exp-CLIP outperforms CLIP and other LVLMs on seven FER datasets.

The method effectively transfers LLM knowledge to improve zero-shot recognition.

Results demonstrate superior generalization to unseen facial expressions.

Abstract

Current facial expression recognition (FER) models are often designed in a supervised learning manner and thus are constrained by the lack of large-scale facial expression images with high-quality annotations. Consequently, these models often fail to generalize well, performing poorly on unseen images in inference. Vision-language-based zero-shot models demonstrate a promising potential for addressing such challenges. However, these models lack task-specific knowledge and therefore are not optimized for the nuances of recognizing facial expressions. To bridge this gap, this work proposes a novel method, Exp-CLIP, to enhance zero-shot FER by transferring the task knowledge from large language models (LLMs). Specifically, based on the pre-trained vision-language encoders, we incorporate a projection head designed to map the initial joint vision-language space into a space that captures representations of facial actions. To train this projection head for subsequent zero-shot predictions, we propose to align the projected visual representations with task-specific semantic meanings derived from the LLM encoder, and the text instruction-based strategy is employed to customize the LLM knowledge. Given unlabelled facial data and efficient training of the projection head, Exp-CLIP achieves superior zero-shot results to the CLIP models and several other large vision-language models (LVLMs) on seven in-the-wild FER datasets. The code and pre-trained models are available at https://github.com/zengqunzhao/Exp-CLIP.