Do autoencoders need a bottleneck for anomaly detection?

TL;DR

This paper investigates the necessity of bottlenecks in autoencoders for anomaly detection, demonstrating that non-bottlenecked architectures can outperform traditional models by removing the bottleneck through overparameterization and skip connections.

Contribution

The study provides extensive experimental evidence that non-bottlenecked autoencoders, including infinitely-wide variants, can effectively detect anomalies, challenging the conventional belief that a bottleneck is essential.

Findings

Non-bottlenecked AEs outperform bottlenecked ones on CIFAR vs SVHN.

Removing the bottleneck improves AUROC scores significantly.

Infinitely-wide AEs demonstrate the potential of non-bottleneck architectures.

Abstract

A common belief in designing deep autoencoders (AEs), a type of unsupervised neural network, is that a bottleneck is required to prevent learning the identity function. Learning the identity function renders the AEs useless for anomaly detection. In this work, we challenge this limiting belief and investigate the value of non-bottlenecked AEs. The bottleneck can be removed in two ways: (1) overparameterising the latent layer, and (2) introducing skip connections. However, limited works have reported on the use of one of the ways. For the first time, we carry out extensive experiments covering various combinations of bottleneck removal schemes, types of AEs and datasets. In addition, we propose the infinitely-wide AEs as an extreme example of non-bottlenecked AEs. Their improvement over the baseline implies learning the identity function is not trivial as previously assumed. Moreover, we find that non-bottlenecked architectures (highest AUROC=0.857) can outperform their bottlenecked counterparts (highest AUROC=0.696) on the popular task of CIFAR (inliers) vs SVHN (anomalies), among other tasks, shedding light on the potential of developing non-bottlenecked AEs for improving anomaly detection.