Automatic Classifiers Underdetect Emotions Expressed by Men

TL;DR

This study reveals that current automatic emotion classifiers are biased against men's texts, with higher error rates, highlighting the need for more equitable models in sentiment analysis.

Contribution

The paper introduces a large-scale, self-annotated dataset and a rigorous research design to systematically analyze gender biases in emotion detection models.

Findings

Error rates are higher for men's texts across classifiers and emotions.

Biases could significantly impact downstream applications.

Current models, including large language models, require cautious application regarding gender bias.

Abstract

The widespread adoption of automatic sentiment and emotion classifiers makes it important to ensure that these tools perform reliably across different populations. Yet their reliability is typically assessed using benchmarks that rely on third-party annotators rather than the individuals experiencing the emotions themselves, potentially concealing systematic biases. In this paper, we use a unique, large-scale dataset of more than one million self-annotated posts and a pre-registered research design to investigate gender biases in emotion detection across 414 combinations of models and emotion-related classes. We find that across different types of automatic classifiers and various underlying emotions, error rates are consistently higher for texts authored by men compared to those authored by women. We quantify how this bias could affect results in downstream applications and show that current machine learning tools, including large language models, should be applied with caution when the gender composition of a sample is not known or variable. Our findings demonstrate that sentiment analysis is not yet a solved problem, especially in ensuring equitable model behaviour across demographic groups.