AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion

TL;DR

This paper introduces AU-LLM, a novel framework that leverages Large Language Models with an enhanced feature fusion technique to detect subtle micro-expression Action Units, achieving state-of-the-art results on benchmark datasets.

Contribution

It pioneers the use of LLMs for micro-expression AU detection and proposes the Enhanced Fusion Projector for effective vision-language feature integration.

Findings

Achieves state-of-the-art performance on CASME II and SAMM datasets.

Demonstrates robustness across LOSO and cross-domain protocols.

Validates the effectiveness of LLM-based reasoning in subtle facial expression analysis.

Abstract

The detection of micro-expression Action Units (AUs) is a formidable challenge in affective computing, pivotal for decoding subtle, involuntary human emotions. While Large Language Models (LLMs) demonstrate profound reasoning abilities, their application to the fine-grained, low-intensity domain of micro-expression AU detection remains unexplored. This paper pioneers this direction by introducing \textbf{AU-LLM}, a novel framework that for the first time uses LLM to detect AUs in micro-expression datasets with subtle intensities and the scarcity of data. We specifically address the critical vision-language semantic gap, the \textbf{Enhanced Fusion Projector (EFP)}. The EFP employs a Multi-Layer Perceptron (MLP) to intelligently fuse mid-level (local texture) and high-level (global semantics) visual features from a specialized 3D-CNN backbone into a single, information-dense token. This compact representation effectively empowers the LLM to perform nuanced reasoning over subtle facial muscle movements.Through extensive evaluations on the benchmark CASME II and SAMM datasets, including stringent Leave-One-Subject-Out (LOSO) and cross-domain protocols, AU-LLM establishes a new state-of-the-art, validating the significant potential and robustness of LLM-based reasoning for micro-expression analysis. The codes are available at https://github.com/ZS-liu-JLU/AU-LLMs.