Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling

TL;DR

This paper introduces a learning-speed aware sampling method for self-supervised learning that enhances robustness against spurious correlations by focusing on samples that learn more slowly, leading to better downstream task performance.

Contribution

It proposes a novel sampling strategy based on learning dynamics to mitigate the impact of spurious correlations in SSL, improving representation robustness.

Findings

LA-SSL improves downstream classification accuracy.

The method reduces reliance on spurious correlations.

Enhanced robustness demonstrated across multiple datasets.

Abstract

Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. The data representations are able to capture many underlying attributes of data, and be useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. cancer is usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of the conspicuous features relevant to those sensitive attributes, despite the presence of other important predictive features for the downstream tasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (e.g. elder patients without cancer). Motivated by these findings, we propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating that it improves the robustness of pretrained representations on downstream classification tasks.