Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models

TL;DR

This paper investigates bias sources in facial expression recognition datasets and models, revealing that high-performing models like GPT-4o-mini and ViT also exhibit significant bias, highlighting the need for fairness-focused methodologies.

Contribution

The study provides a comprehensive analysis of bias in popular FER datasets and models, comparing CNN and transformer-based architectures in terms of accuracy and fairness.

Findings

AffectNet and ExpW datasets show high generalizability despite imbalances.

GPT-4o-mini and ViT achieve top accuracy but also highest bias levels.

Bias and fairness issues are prominent in state-of-the-art FER models.

Abstract

Building AI systems, including Facial Expression Recognition (FER), involves two critical aspects: data and model design. Both components significantly influence bias and fairness in FER tasks. Issues related to bias and fairness in FER datasets and models remain underexplored. This study investigates bias sources in FER datasets and models. Four common FER datasets--AffectNet, ExpW, Fer2013, and RAF-DB--are analyzed. The findings demonstrate that AffectNet and ExpW exhibit high generalizability despite data imbalances. Additionally, this research evaluates the bias and fairness of six deep models, including three state-of-the-art convolutional neural network (CNN) models: MobileNet, ResNet, XceptionNet, as well as three transformer-based models: ViT, CLIP, and GPT-4o-mini. Experimental results reveal that while GPT-4o-mini and ViT achieve the highest accuracy scores, they also display the highest levels of bias. These findings underscore the urgent need for developing new methodologies to mitigate bias and ensure fairness in datasets and models, particularly in affective computing applications. See our implementation details at https://github.com/MMHosseini/bias_in_FER.