Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP

TL;DR

This paper clarifies the landscape of fairness evaluation in NLP, linking methods to theory and addressing model selection challenges to promote meaningful progress in fair learning.

Contribution

It elucidates the relationships among fairness methods and theory, and proposes solutions for model selection issues involving fairness-accuracy trade-offs.

Findings

Clarifies relationships among fairness evaluation methods and theoretical measures.

Addresses practical challenges in model selection balancing fairness and accuracy.

Provides recommendations to guide future fair NLP research.

Abstract

Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. However current progress is hampered by a plurality of definitions of bias, means of quantification, and oftentimes vague relation between debiasing algorithms and theoretical measures of bias. This paper seeks to clarify the current situation and plot a course for meaningful progress in fair learning, with two key contributions: (1) making clear inter-relations among the current gamut of methods, and their relation to fairness theory; and (2) addressing the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. Putting them together, we make several recommendations to help shape future work.