OOD Detection with immature Models

TL;DR

This paper shows that early-stage, immature deep generative models can be as effective or better than fully trained models for out-of-distribution detection, challenging assumptions about model maturity and training.

Contribution

It introduces the novel insight that immature models can outperform mature ones in OOD detection, and provides a theoretical explanation based on support overlap.

Findings

Immature models can achieve comparable or superior OOD detection performance.

Using early stopping can reduce training time without sacrificing accuracy.

Support overlap explains the effectiveness of immature models in this task.

Abstract

Likelihood-based deep generative models (DGMs) have gained significant attention for their ability to approximate the distributions of high-dimensional data. However, these models lack a performance guarantee in assigning higher likelihood values to in-distribution (ID) inputs, data the models are trained on, compared to out-of-distribution (OOD) inputs. This counter-intuitive behaviour is particularly pronounced when ID inputs are more complex than OOD data points. One potential approach to address this challenge involves leveraging the gradient of a data point with respect to the parameters of the DGMs. A recent OOD detection framework proposed estimating the joint density of layer-wise gradient norms for a given data point as a model-agnostic method, demonstrating superior performance compared to the Typicality Test across likelihood-based DGMs and image dataset pairs. In particular, most existing methods presuppose access to fully converged models, the training of which is both time-intensive and computationally demanding. In this work, we demonstrate that using immature models,stopped at early stages of training, can mostly achieve equivalent or even superior results on this downstream task compared to mature models capable of generating high-quality samples that closely resemble ID data. This novel finding enhances our understanding of how DGMs learn the distribution of ID data and highlights the potential of leveraging partially trained models for downstream tasks. Furthermore, we offer a possible explanation for this unexpected behaviour through the concept of support overlap.