A First Look at Fairness of Machine Learning Based Code Reviewer Recommendation

TL;DR

This paper investigates the fairness of machine learning models in code reviewer recommendation, revealing existing biases, analyzing their causes, and proposing solutions to improve fairness, especially in imbalanced datasets.

Contribution

First empirical study on ML fairness in software engineering, specifically in code reviewer recommendation, identifying biases and proposing mitigation strategies.

Findings

ML-based reviewer recommendation systems are unfair, favoring male reviewers.

Current mitigation methods can double fairness in balanced datasets.

Effectiveness of mitigation methods is limited on imbalanced or skewed data.

Abstract

The fairness of machine learning (ML) approaches is critical to the reliability of modern artificial intelligence systems. Despite extensive study on this topic, the fairness of ML models in the software engineering (SE) domain has not been well explored yet. As a result, many ML-powered software systems, particularly those utilized in the software engineering community, continue to be prone to fairness issues. Taking one of the typical SE tasks, i.e., code reviewer recommendation, as a subject, this paper conducts the first study toward investigating the issue of fairness of ML applications in the SE domain. Our empirical study demonstrates that current state-of-the-art ML-based code reviewer recommendation techniques exhibit unfairness and discriminating behaviors. Specifically, male reviewers get on average 7.25% more recommendations than female code reviewers compared to their distribution in the reviewer set. This paper also discusses the reasons why the studied ML-based code reviewer recommendation systems are unfair and provides solutions to mitigate the unfairness. Our study further indicates that the existing mitigation methods can enhance fairness by 100% in projects with a similar distribution of protected and privileged groups, but their effectiveness in improving fairness on imbalanced or skewed data is limited. Eventually, we suggest a solution to overcome the drawbacks of existing mitigation techniques and tackle bias in datasets that are imbalanced or skewed.