Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning

TL;DR

This paper introduces Conditional Contrastive Learning (CCL), a method that enhances fairness in self-supervised contrastive learning by conditioning on sensitive attributes, leading to fairer and more effective representations.

Contribution

The paper proposes a novel CCL approach that samples pairs conditioned on sensitive attributes, maximizing conditional mutual information and improving fairness in contrastive SSL.

Findings

Achieves state-of-the-art downstream performance on multiple datasets.

Significantly improves fairness metrics in contrastive SSL models.

Reduces sensitive attribute influence in learned representations.

Abstract

Contrastive self-supervised learning (SSL) learns an embedding space that maps similar data pairs closer and dissimilar data pairs farther apart. Despite its success, one issue has been overlooked: the fairness aspect of representations learned using contrastive SSL. Without mitigation, contrastive SSL techniques can incorporate sensitive information such as gender or race and cause potentially unfair predictions on downstream tasks. In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods. Our approach samples positive and negative pairs from distributions conditioning on the sensitive attribute, or empirically speaking, sampling positive and negative pairs from the same gender or the same race. We show that our approach provably maximizes the conditional mutual information between the learned representations of the positive pairs, and reduces the effect of the sensitive attribute by taking it as the conditional variable. On seven fairness and vision datasets, we empirically demonstrate that the proposed approach achieves state-of-the-art downstream performances compared to unsupervised baselines and significantly improves the fairness of contrastive SSL models on multiple fairness metrics.