SHaSaM: Submodular Hard Sample Mining for Fair Facial Attribute Recognition

TL;DR

SHaSaM introduces a submodular hard sample mining approach to improve fairness in facial attribute recognition, effectively addressing data imbalance and reducing bias without sacrificing accuracy.

Contribution

The paper presents a novel two-stage submodular sampling and loss framework for fair facial attribute recognition, outperforming existing methods in fairness and accuracy.

Findings

Achieves up to 2.7 points improvement in fairness (Equalized Odds)

Gains 3.5% in accuracy over baseline methods

Converges faster with fewer training epochs

Abstract

Deep neural networks often inherit social and demographic biases from annotated data during model training, leading to unfair predictions, especially in the presence of sensitive attributes like race, age, gender etc. Existing methods fall prey to the inherent data imbalance between attribute groups and inadvertently emphasize on sensitive attributes, worsening unfairness and performance. To surmount these challenges, we propose SHaSaM (Submodular Hard Sample Mining), a novel combinatorial approach that models fairness-driven representation learning as a submodular hard-sample mining problem. Our two-stage approach comprises of SHaSaM-MINE, which introduces a submodular subset selection strategy to mine hard positives and negatives - effectively mitigating data imbalance, and SHaSaM-LEARN, which introduces a family of combinatorial loss functions based on Submodular Conditional Mutual Information to maximize the decision boundary between target classes while minimizing the influence of sensitive attributes. This unified formulation restricts the model from learning features tied to sensitive attributes, significantly enhancing fairness without sacrificing performance. Experiments on CelebA and UTKFace demonstrate that SHaSaM achieves state-of-the-art results, with up to 2.7 points improvement in model fairness (Equalized Odds) and a 3.5% gain in Accuracy, within fewer epochs as compared to existing methods.