How Robust is Unsupervised Representation Learning to Distribution Shift?

TL;DR

This paper investigates how unsupervised representation learning methods like SSL and AE are more robust to distribution shifts than supervised learning, through extensive experiments on synthetic and real datasets.

Contribution

It provides the first comprehensive evaluation of unsupervised methods' robustness to distribution shift, introducing controllable datasets and a linear head evaluation approach.

Findings

Unsupervised representations outperform supervised ones under distribution shift.

Representations learned from SSL and AE generalize better than supervised models.

The study introduces controllable datasets for realistic distribution shift evaluation.

Abstract

The robustness of machine learning algorithms to distributions shift is primarily discussed in the context of supervised learning (SL). As such, there is a lack of insight on the robustness of the representations learned from unsupervised methods, such as self-supervised learning (SSL) and auto-encoder based algorithms (AE), to distribution shift. We posit that the input-driven objectives of unsupervised algorithms lead to representations that are more robust to distribution shift than the target-driven objective of SL. We verify this by extensively evaluating the performance of SSL and AE on both synthetic and realistic distribution shift datasets. Following observations that the linear layer used for classification itself can be susceptible to spurious correlations, we evaluate the representations using a linear head trained on a small amount of out-of-distribution (OOD) data, to isolate the robustness of the learned representations from that of the linear head. We also develop "controllable" versions of existing realistic domain generalisation datasets with adjustable degrees of distribution shifts. This allows us to study the robustness of different learning algorithms under versatile yet realistic distribution shift conditions. Our experiments show that representations learned from unsupervised learning algorithms generalise better than SL under a wide variety of extreme as well as realistic distribution shifts.