# Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness

## TL;DR

This study explores how machine learning practitioners evaluate fairness in text classification models, revealing that their strategies are influenced by the fairness metrics presented and personal experiences, with implications for designing better evaluation tools.

## Contribution

It provides insights into practitioners' fairness evaluation strategies and offers recommendations for developing interactive fairness assessment tools in text classification.

## Key findings

- Metrics influence fairness judgments
- Practitioners consider risks of under/overprediction
- Personal experiences shape fairness assessments

## Abstract

Mitigating algorithmic bias is a critical task in the development and deployment of machine learning models. While several toolkits exist to aid machine learning practitioners in addressing fairness issues, little is known about the strategies practitioners employ to evaluate model fairness and what factors influence their assessment, particularly in the context of text classification. Two common approaches of evaluating the fairness of a model are group fairness and individual fairness. We run a study with Machine Learning practitioners (n=24) to understand the strategies used to evaluate models. Metrics presented to practitioners (group vs. individual fairness) impact which models they consider fair. Participants focused on risks associated with underpredicting/overpredicting and model sensitivity relative to identity token manipulations. We discover fairness assessment strategies involving personal experiences or how users form groups of identity tokens to test model fairness. We provide recommendations for interactive tools for evaluating fairness in text classification.