Learning Fair Classifiers with Partially Annotated Group Labels

TL;DR

This paper introduces a confidence-based pseudo-labeling strategy to improve fairness and accuracy in classifiers trained with partially annotated group labels, addressing real-world privacy and annotation challenges.

Contribution

It proposes a simple, effective CGL method that enhances fairness-aware learning with partial group labels and can incorporate external datasets.

Findings

CGL improves fairness and accuracy on benchmark datasets.

CGL outperforms vanilla pseudo-labeling strategies.

CGL enables augmentation with external datasets for better fairness and accuracy.

Abstract

Recently, fairness-aware learning have become increasingly crucial, but most of those methods operate by assuming the availability of fully annotated demographic group labels. We emphasize that such assumption is unrealistic for real-world applications since group label annotations are expensive and can conflict with privacy issues. In this paper, we consider a more practical scenario, dubbed as Algorithmic Group Fairness with the Partially annotated Group labels (Fair-PG). We observe that the existing methods to achieve group fairness perform even worse than the vanilla training, which simply uses full data only with target labels, under Fair-PG. To address this problem, we propose a simple Confidence-based Group Label assignment (CGL) strategy that is readily applicable to any fairness-aware learning method. CGL utilizes an auxiliary group classifier to assign pseudo group labels, where random labels are assigned to low confident samples. We first theoretically show that our method design is better than the vanilla pseudo-labeling strategy in terms of fairness criteria. Then, we empirically show on several benchmark datasets that by combining CGL and the state-of-the-art fairness-aware in-processing methods, the target accuracies and the fairness metrics can be jointly improved compared to the baselines. Furthermore, we convincingly show that CGL enables to naturally augment the given group-labeled dataset with external target label-only datasets so that both accuracy and fairness can be improved. Code is available at https://github.com/naver-ai/cgl_fairness.