Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

TL;DR

This paper unifies and empirically compares various fairness metrics in NLP, revealing how differences in parameter choices influence bias measurement and providing a clearer understanding of social biases in models.

Contribution

It introduces a unified framework for fairness metrics in NLP and systematically analyzes their differences through extensive empirical evaluation.

Findings

Differences in bias measurement are explained by parameter choices.

Unified three generalized fairness metrics.

Empirical comparison clarifies metric similarities and differences.

Abstract

Measuring bias is key for better understanding and addressing unfairness in NLP/ML models. This is often done via fairness metrics which quantify the differences in a model's behaviour across a range of demographic groups. In this work, we shed more light on the differences and similarities between the fairness metrics used in NLP. First, we unify a broad range of existing metrics under three generalized fairness metrics, revealing the connections between them. Next, we carry out an extensive empirical comparison of existing metrics and demonstrate that the observed differences in bias measurement can be systematically explained via differences in parameter choices for our generalized metrics.