Mapping the Multilingual Margins: Intersectional Biases of Sentiment Analysis Systems in English, Spanish, and Arabic

TL;DR

This paper introduces new multilingual test sets and a statistical framework to measure social biases, including intersectional biases, in NLP systems across English, Spanish, and Arabic, revealing significant biases in existing models.

Contribution

It presents four multilingual bias evaluation corpora and a novel framework for analyzing unisectional and intersectional biases in NLP models.

Findings

Many models show significant social biases

Biases are present across multiple languages and tasks

Intersectional biases are statistically significant

Abstract

As natural language processing systems become more widespread, it is necessary to address fairness issues in their implementation and deployment to ensure that their negative impacts on society are understood and minimized. However, there is limited work that studies fairness using a multilingual and intersectional framework or on downstream tasks. In this paper, we introduce four multilingual Equity Evaluation Corpora, supplementary test sets designed to measure social biases, and a novel statistical framework for studying unisectional and intersectional social biases in natural language processing. We use these tools to measure gender, racial, ethnic, and intersectional social biases across five models trained on emotion regression tasks in English, Spanish, and Arabic. We find that many systems demonstrate statistically significant unisectional and intersectional social biases.