Team RAS in 9th ABAW Competition: Multimodal Compound Expression Recognition Approach

TL;DR

This paper introduces a novel zero-shot multimodal approach for compound expression recognition that combines six heterogeneous modalities and employs advanced semantic understanding techniques, achieving competitive results without domain-specific training.

Contribution

The work presents a new zero-shot multimodal framework with a Multi-Head Probability Fusion module and semantic scene understanding, advancing affective computing for complex emotion detection.

Findings

Achieved F1 scores of 46.95% on AffWild2

Achieved F1 scores of 49.02% on AFEW

Achieved F1 scores of 34.85% on C-EXPR-DB

Abstract

Compound Expression Recognition (CER), a subfield of affective computing, aims to detect complex emotional states formed by combinations of basic emotions. In this work, we present a novel zero-shot multimodal approach for CER that combines six heterogeneous modalities into a single pipeline: static and dynamic facial expressions, scene and label matching, scene context, audio, and text. Unlike previous approaches relying on task-specific training data, our approach uses zero-shot components, including Contrastive Language-Image Pretraining (CLIP)-based label matching and Qwen-VL for semantic scene understanding. We further introduce a Multi-Head Probability Fusion (MHPF) module that dynamically weights modality-specific predictions, followed by a Compound Expressions (CE) transformation module that uses Pair-Wise Probability Aggregation (PPA) and Pair-Wise Feature Similarity Aggregation (PFSA) methods to produce interpretable compound emotion outputs. Evaluated under multi-corpus training, the proposed approach shows F1 scores of 46.95% on AffWild2, 49.02% on Acted Facial Expressions in The Wild (AFEW), and 34.85% on C-EXPR-DB via zero-shot testing, which is comparable to the results of supervised approaches trained on target data. This demonstrates the effectiveness of the proposed approach for capturing CE without domain adaptation. The source code is publicly available.