ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

TL;DR

ProtoFair introduces a fairness-aware contrastive loss that enhances demographic fairness in self-supervised visual representations without altering existing SSL objectives.

Contribution

It proposes a novel pseudo-counterfactual pairing method using prototype clustering, compatible with various SSL frameworks, to improve fairness.

Findings

Achieves fairness improvements on CelebA and UTKFace datasets.

Maintains competitive accuracy while reducing demographic bias.

Seamlessly integrates with SimCLR and SupCon frameworks.

Abstract

Self-supervised learning methods learn high-quality visual representations, yet recent studies show that these representations often capture demographic biases present in the training data. Existing fairness-aware methods address this by redesigning the self-supervised objective itself, limiting portability across the rapidly evolving landscape of self-supervised learning (SSL) frameworks. We propose ProtoFair, a fairness-aware contrastive loss designed to work alongside existing SSL objectives without modifying them. ProtoFair leverages unsupervised prototype clustering to identify pseudo-counterfactual pairs: samples sharing the same cluster assignment but belonging to different sensitive groups. By pulling these content-matched, cross-group samples together in the embedding space, ProtoFair encourages the encoder to learn representations that are invariant to the sensitive attribute. The method requires only sensitive attribute annotations, no target labels, and integrates seamlessly with both SimCLR and SupCon. Experiments on CelebA and UTKFace demonstrate consistent fairness improvements while maintaining competitive accuracy.