DEFT-LLM: Disentangled Expert Feature Tuning for Micro-Expression Recognition

TL;DR

This paper introduces DEFT-LLM, a novel approach that combines disentangled expert feature tuning with large language models to improve micro-expression recognition by aligning text with facial motion and enhancing interpretability.

Contribution

The paper proposes a new architecture with three experts for decoupling facial dynamics and introduces Uni-MER, a motion-driven instruction dataset for better text-motion alignment in MER.

Findings

Achieves state-of-the-art performance on MER benchmarks.

Enhances interpretability of facial motion modeling.

Effectively captures subtle emotional cues.

Abstract

Micro expression recognition (MER) is crucial for inferring genuine emotion. Applying a multimodal large language model (MLLM) to this task enables spatio-temporal analysis of facial motion and provides interpretable descriptions. However, there are still two core challenges: (1) The entanglement of static appearance and dynamic motion cues prevents the model from focusing on subtle motion; (2) Textual labels in existing MER datasets do not fully correspond to underlying facial muscle movements, creating a semantic gap between text supervision and physical motion. To address these issues, we propose DEFT-LLM, which achieves motion semantic alignment by multi-expert disentanglement. We first introduce Uni-MER, a motion-driven instruction dataset designed to align text with local facial motion. Its construction leverages dual constraints from optical flow and Action Unit (AU) labels to ensure spatio-temporal consistency and reasonable correspondence to the movements. We then design an architecture with three experts to decouple facial dynamics into independent and interpretable representations (structure, dynamic textures, and motion-semantics). By integrating the instruction-aligned knowledge from Uni-MER into DEFT-LLM, our method injects effective physical priors for micro expressions while also leveraging the cross modal reasoning ability of large language models, thus enabling precise capture of subtle emotional cues. Experiments on multiple challenging MER benchmarks demonstrate state-of-the-art performance, as well as a particular advantage in interpretable modeling of local facial motion.