From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models

TL;DR

This paper evaluates racial biases in large multimodal foundation models for facial expression recognition, revealing significant disparities and emphasizing the need for fairer FER systems.

Contribution

It benchmarks leading LMFMs for racial bias in FER and provides insights into their performance disparities across demographics.

Findings

LMFMs show higher error rates for darker skin tones.

Anger is misclassified as Disgust more often in Black Females.

High accuracy achieved with linear classifiers on CLIP embeddings.

Abstract

This study addresses the racial biases in facial expression recognition (FER) systems within Large Multimodal Foundation Models (LMFMs). Despite advances in deep learning and the availability of diverse datasets, FER systems often exhibit higher error rates for individuals with darker skin tones. Existing research predominantly focuses on traditional FER models (CNNs, RNNs, ViTs), leaving a gap in understanding racial biases in LMFMs. We benchmark four leading LMFMs: GPT-4o, PaliGemma, Gemini, and CLIP to assess their performance in facial emotion detection across different racial demographics. A linear classifier trained on CLIP embeddings obtains accuracies of 95.9\% for RADIATE, 90.3\% for Tarr, and 99.5\% for Chicago Face. Furthermore, we identify that Anger is misclassified as Disgust 2.1 times more often in Black Females than White Females. This study highlights the need for fairer FER systems and establishes a foundation for developing unbiased, accurate FER technologies. Visit https://kvjvhub.github.io/FERRacialBias/ for further information regarding the biases within facial expression recognition.