Predicting Fairness of ML Software Configurations

TL;DR

This paper explores predicting the fairness of machine learning hyperparameters using data-driven models, focusing on group fairness and robustness to distribution shifts, to aid in efficient hyperparameter tuning.

Contribution

It demonstrates that tree regressors and XGBoost outperform neural networks in predicting hyperparameter fairness and provides insights into robustness under distribution shifts.

Findings

Tree regressors and XGBoost outperform neural networks in fairness prediction.

Prediction accuracy varies with dataset, algorithm, and protected attribute.

Tree regressors are robust to certain temporal distribution shifts.

Abstract

This paper investigates the relationships between hyperparameters of machine learning and fairness. Data-driven solutions are increasingly used in critical socio-technical applications where ensuring fairness is important. Rather than explicitly encoding decision logic via control and data structures, the ML developers provide input data, perform some pre-processing, choose ML algorithms, and tune hyperparameters (HPs) to infer a program that encodes the decision logic. Prior works report that the selection of HPs can significantly influence fairness. However, tuning HPs to find an ideal trade-off between accuracy, precision, and fairness has remained an expensive and tedious task. Can we predict fairness of HP configuration for a given dataset? Are the predictions robust to distribution shifts? We focus on group fairness notions and investigate the HP space of 5 training algorithms. We first find that tree regressors and XGBoots significantly outperformed deep neural networks and support vector machines in accurately predicting the fairness of HPs. When predicting the fairness of ML hyperparameters under temporal distribution shift, the tree regressors outperforms the other algorithms with reasonable accuracy. However, the precision depends on the ML training algorithm, dataset, and protected attributes. For example, the tree regressor model was robust for training data shift from 2014 to 2018 on logistic regression and discriminant analysis HPs with sex as the protected attribute; but not for race and other training algorithms. Our method provides a sound framework to efficiently perform fine-tuning of ML training algorithms and understand the relationships between HPs and fairness.