Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining

TL;DR

This paper investigates the effectiveness of loss weighting during last layer retraining in neural networks, demonstrating that proper weighting can improve fairness and performance when the retraining data is limited and the model is moderately overparameterized.

Contribution

It introduces a theoretical and empirical analysis of loss weighting in the last layer retraining regime, highlighting the importance of considering model overparameterization.

Findings

Loss weighting remains effective in last layer retraining.

Proper weights depend on the degree of model overparameterization.

Empirical results support theoretical predictions.

Abstract

While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights \emph{must} take into account the relative overparameterization of the model.