Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexity

TL;DR

This paper investigates the double descent phenomenon in post-hoc out-of-distribution detection, providing empirical evidence and theoretical explanations, and proposes a method to identify the optimal model complexity for improved OOD detection.

Contribution

It reveals the presence of double descent in post-hoc OOD detection and offers theoretical insights and a practical method to optimize model complexity for better performance.

Findings

Double descent occurs in post-hoc OOD detection.

Overparameterization does not always improve OOD detection.

A method to identify the optimal model regime for OOD detection.

Abstract

Out-of-distribution (OOD) detection is essential for ensuring the reliability and safety of machine learning systems. In recent years, it has received increasing attention, particularly through post-hoc detection and training-based methods. In this paper, we focus on post-hoc OOD detection, which enables identifying OOD samples without altering the model's training procedure or objective. Our primary goal is to investigate the relationship between model capacity and its OOD detection performance. Specifically, we aim to answer the following question: Does the Double Descent phenomenon manifest in post-hoc OOD detection? This question is crucial, as it can reveal whether overparameterization, which is already known to benefit generalization, can also enhance OOD detection. Despite the growing interest in these topics by the classic supervised machine learning community, this intersection remains unexplored for OOD detection. We empirically demonstrate that the Double Descent effect does indeed appear in post-hoc OOD detection. Furthermore, we provide theoretical insights to explain why this phenomenon emerges in such setting. Finally, we show that the overparameterized regime does not yield superior results consistently, and we propose a method to identify the optimal regime for OOD detection based on our observations.