Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework

TL;DR

This paper introduces CFU, a reinforcement learning framework that optimizes multiple fairness metrics and utility simultaneously, addressing the complex fairness-utility trade-off in machine learning decision systems.

Contribution

The work proposes a comprehensive measurement and new metrics for fairness, and develops CFU, a reinforcement learning approach that improves multiple fairness notions without utility loss.

Findings

CFU outperforms existing methods across 6 tasks and 15 measurements.

CFU achieves an average of 37.5% improvement in fairness-utility trade-off.

The framework effectively balances multiple fairness metrics and utility in diverse scenarios.

Abstract

Machine learning is widely used to make decisions with societal impact such as bank loan approving, criminal sentencing, and resume filtering. How to ensure its fairness while maintaining utility is a challenging but crucial issue. Fairness is a complex and context-dependent concept with over 70 different measurement metrics. Since existing regulations are often vague in terms of which metric to use and different organizations may prefer different fairness metrics, it is important to have means of improving fairness comprehensively. Existing mitigation techniques often target at one specific fairness metric and have limitations in improving multiple notions of fairness simultaneously. In this work, we propose CFU (Comprehensive Fairness-Utility), a reinforcement learning-based framework, to efficiently improve the fairness-utility trade-off in machine learning classifiers. A comprehensive measurement that can simultaneously consider multiple fairness notions as well as utility is established, and new metrics are proposed based on an in-depth analysis of the relationship between different fairness metrics. The reward function of CFU is constructed with comprehensive measurement and new metrics. We conduct extensive experiments to evaluate CFU on 6 tasks, 3 machine learning models, and 15 fairness-utility measurements. The results demonstrate that CFU can improve the classifier on multiple fairness metrics without sacrificing its utility. It outperforms all state-of-the-art techniques and has witnessed a 37.5% improvement on average.