Fairness Testing: A Comprehensive Survey and Analysis of Trends

TL;DR

This paper provides a comprehensive survey of fairness testing in machine learning, analyzing existing research, methodologies, datasets, and tools to identify trends and future directions in ensuring ML fairness.

Contribution

It systematically organizes and reviews 100 studies on fairness testing, highlighting current practices, research focus, and open challenges in the field.

Findings

Identification of key datasets and tools used in fairness testing

Analysis of research trends and focus areas in fairness testing

Highlighting promising future directions for fairness testing research

Abstract

Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this paper offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely-adopted datasets and open-source tools for fairness testing.