Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection

TL;DR

This paper identifies instability issues in out-of-distribution detection during neural network training and proposes Average of Pruning (AoP), combining model averaging and pruning, to enhance stability and performance.

Contribution

The paper introduces AoP, a novel method that stabilizes OOD detection by integrating model averaging and pruning to reduce overfitting and variability.

Findings

AoP improves OOD detection stability across datasets.

Model averaging smooths performance fluctuations.

Pruning reduces overfitting and redundant features.

Abstract

Detecting Out-of-distribution (OOD) inputs have been a critical issue for neural networks in the open world. However, the unstable behavior of OOD detection along the optimization trajectory during training has not been explored clearly. In this paper, we first find the performance of OOD detection suffers from overfitting and instability during training: 1) the performance could decrease when the training error is near zero, and 2) the performance would vary sharply in the final stage of training. Based on our findings, we propose Average of Pruning (AoP), consisting of model averaging and pruning, to mitigate the unstable behaviors. Specifically, model averaging can help achieve a stable performance by smoothing the landscape, and pruning is certified to eliminate the overfitting by eliminating redundant features. Comprehensive experiments on various datasets and architectures are conducted to verify the effectiveness of our method.