Evaluating the fairness of fine-tuning strategies in self-supervised learning

TL;DR

This paper investigates how fine-tuning strategies affect the fairness of contrastive self-supervised learning models, highlighting the importance of Batch Normalization updates and proposing efficient methods to improve fairness with less training time.

Contribution

It introduces a novel approach of updating only Batch Normalization statistics to enhance fairness, achieving comparable results to full fine-tuning with significantly less training.

Findings

Updating BN statistics improves downstream fairness by 36% worst subgroup gap.

Selective BN updates are competitive with supervised learning in fairness.

Training residual skip connections with BN updates reduces training time by 1.33x.

Abstract

In this work we examine how fine-tuning impacts the fairness of contrastive Self-Supervised Learning (SSL) models. Our findings indicate that Batch Normalization (BN) statistics play a crucial role, and that updating only the BN statistics of a pre-trained SSL backbone improves its downstream fairness (36% worst subgroup, 25% mean subgroup gap). This procedure is competitive with supervised learning, while taking 4.4x less time to train and requiring only 0.35% as many parameters to be updated. Finally, inspired by recent work in supervised learning, we find that updating BN statistics and training residual skip connections (12.3% of the parameters) achieves parity with a fully fine-tuned model, while taking 1.33x less time to train.