Does Double Descent Occur in Self-Supervised Learning?

TL;DR

This paper investigates whether the double descent phenomenon occurs in self-supervised learning models, finding that it generally does not, unlike in supervised models, based on empirical analysis of autoencoders.

Contribution

It provides the first empirical study of double descent in self-supervised models, showing the phenomenon is absent in standard autoencoders.

Findings

Test loss shows U-shape or monotonic decrease, not double descent.

Double descent is not observed in the studied self-supervised models.

Results suggest different underlying mechanisms from supervised models.

Abstract

Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classical U-shape or to monotonically decrease instead of exhibiting a double-descent curve. We hope that further work on this will help elucidate the theoretical underpinnings of this phenomenon.