Balancing Fairness and Accuracy in Sentiment Detection using Multiple Black Box Models

TL;DR

This paper audits commercial sentiment detection APIs for gender bias and proposes a method to combine multiple black-box models to improve both fairness and accuracy in sentiment analysis.

Contribution

It introduces a novel 'Flexible Fair Regression' approach that jointly learns from multiple black-box models to balance fairness and accuracy.

Findings

Audited APIs reveal gender bias in sentiment detection.

Proposed method achieves better fairness without sacrificing accuracy.

Results demonstrate improved fairness-accuracy trade-offs.

Abstract

Sentiment detection is an important building block for multiple information retrieval tasks such as product recommendation, cyberbullying detection, and misinformation detection. Unsurprisingly, multiple commercial APIs, each with different levels of accuracy and fairness, are now available for sentiment detection. While combining inputs from multiple modalities or black-box models for increasing accuracy is commonly studied in multimedia computing literature, there has been little work on combining different modalities for increasing fairness of the resulting decision. In this work, we audit multiple commercial sentiment detection APIs for the gender bias in two actor news headlines settings and report on the level of bias observed. Next, we propose a "Flexible Fair Regression" approach, which ensures satisfactory accuracy and fairness by jointly learning from multiple black-box models. The results pave way for fair yet accurate sentiment detectors for multiple applications.