To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methods

TL;DR

This study investigates how different machine unlearning methods impact fairness in AI models, revealing that non-uniform data deletion can improve fairness with specific methods, highlighting important trade-offs for responsible AI deployment.

Contribution

First comprehensive analysis of fairness implications of machine unlearning methods, providing empirical insights to guide responsible AI practices.

Findings

SISA improves fairness under non-uniform deletion

Initial training and uniform deletion do not significantly affect fairness

Different unlearning methods have varying fairness impacts

Abstract

The right to be forgotten (RTBF) is motivated by the desire of people not to be perpetually disadvantaged by their past deeds. For this, data deletion needs to be deep and permanent, and should be removed from machine learning models. Researchers have proposed machine unlearning algorithms which aim to erase specific data from trained models more efficiently. However, these methods modify how data is fed into the model and how training is done, which may subsequently compromise AI ethics from the fairness perspective. To help software engineers make responsible decisions when adopting these unlearning methods, we present the first study on machine unlearning methods to reveal their fairness implications. We designed and conducted experiments on two typical machine unlearning methods (SISA and AmnesiacML) along with a retraining method (ORTR) as baseline using three fairness datasets under three different deletion strategies. Experimental results show that under non-uniform data deletion, SISA leads to better fairness compared with ORTR and AmnesiacML, while initial training and uniform data deletion do not necessarily affect the fairness of all three methods. These findings have exposed an important research problem in software engineering, and can help practitioners better understand the potential trade-offs on fairness when considering solutions for RTBF.