Unveiling Multiple Descents in Unsupervised Autoencoders

TL;DR

This paper investigates the occurrence of double and triple descent phenomena in unsupervised autoencoders, revealing their presence in nonlinear models and their impact on performance in various tasks.

Contribution

It is the first to demonstrate double and triple descent in nonlinear unsupervised autoencoders and explores their effects on model performance and robustness.

Findings

Double and triple descent observed in nonlinear AEs

Over-parameterized models improve reconstruction and downstream tasks

Bottleneck size influences the double descent curve

Abstract

The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the importance of bottleneck size in influencing the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-parameterized models not only improve reconstruction but also enhance performance in downstream tasks such as anomaly detection and domain adaptation, highlighting their practical value in complex real-world scenarios.