Multimodal Prompt Alignment for Facial Expression Recognition

TL;DR

This paper introduces MPA-FER, a multimodal prompt alignment framework that enhances facial expression recognition by leveraging detailed descriptions from large language models and aligning visual features with class prototypes for better interpretability and accuracy.

Contribution

The paper proposes a novel multimodal prompt alignment method that integrates LLM-generated detailed prompts and prototype-guided feature alignment to improve FER performance.

Findings

Outperforms state-of-the-art on three FER datasets

Maintains pretrained model benefits with minimal extra computation

Provides more interpretable facial expression representations

Abstract

Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs) like CLIP for various downstream tasks. Despite their success, current VLM-based facial expression recognition (FER) methods struggle to capture fine-grained textual-visual relationships, which are essential for distinguishing subtle differences between facial expressions. To address this challenge, we propose a multimodal prompt alignment framework for FER, called MPA-FER, that provides fine-grained semantic guidance to the learning process of prompted visual features, resulting in more precise and interpretable representations. Specifically, we introduce a multi-granularity hard prompt generation strategy that utilizes a large language model (LLM) like ChatGPT to generate detailed descriptions for each facial expression. The LLM-based external knowledge is injected into the soft prompts by minimizing the feature discrepancy between the soft prompts and the hard prompts. To preserve the generalization abilities of the pretrained CLIP model, our approach incorporates prototype-guided visual feature alignment, ensuring that the prompted visual features from the frozen image encoder align closely with class-specific prototypes. Additionally, we propose a cross-modal global-local alignment module that focuses on expression-relevant facial features, further improving the alignment between textual and visual features. Extensive experiments demonstrate our framework outperforms state-of-the-art methods on three FER benchmark datasets, while retaining the benefits of the pretrained model and minimizing computational costs.