Analyzing Language Bias Between French and English in Conventional Multilingual Sentiment Analysis Models

TL;DR

This study investigates language bias in multilingual sentiment analysis between French and English, revealing biases favoring French and assessing fairness using Fairlearn across different models and datasets.

Contribution

It provides an empirical analysis of language bias in sentiment classification models and evaluates fairness metrics, highlighting the need for equitable multilingual NLP systems.

Findings

French data outperforms English in accuracy, recall, and F1 scores.

Fairlearn indicates near-equity for SVM models across datasets.

Naive Bayes shows greater disparities in demographic parity ratios.

Abstract

Inspired by the 'Bias Considerations in Bilingual Natural Language Processing' report by Statistics Canada, this study delves into potential biases in multilingual sentiment analysis between English and French. Given a 50-50 dataset of French and English, we aim to determine if there exists a language bias and explore how the incorporation of more diverse datasets in the future might affect the equity of multilingual Natural Language Processing (NLP) systems. By employing Support Vector Machine (SVM) and Naive Bayes models on three balanced datasets, we reveal potential biases in multilingual sentiment classification. Utilizing Fairlearn, a tool for assessing bias in machine learning models, our findings indicate nuanced outcomes. With French data outperforming English across accuracy, recall, and F1 score metrics in both models, hinting at a language bias favoring French. However, Fairlearn's metrics suggest that the SVM approaches equitable levels with a demographic parity ratio of 0.963, 0.989, and 0.985 for the three separate datasets, indicating near-equitable treatment across languages. In contrast, Naive Bayes demonstrates greater disparities, evidenced by a demographic parity ratio of 0.813, 0.908, and 0.961. These findings reveal the importance of developing equitable multilingual NLP systems, particularly as we anticipate the inclusion of more datasets in various languages in the future.