Are Out-of-Distribution Detection Methods Effective on Large-Scale Datasets?

TL;DR

This paper evaluates out-of-distribution detection methods on large-scale datasets like ImageNet, finding that input perturbation and temperature scaling are most effective, while feature space regularization has limited practicality at scale.

Contribution

It systematically compares inference and regularization approaches for out-of-distribution detection on large datasets, identifying optimal combinations and practical insights.

Findings

Input perturbation and temperature scaling perform best on large datasets.

Feature space regularization can help if a suitable background class is available.

Regularization against a background class is impractical for large-scale datasets.

Abstract

Supervised classification methods often assume the train and test data distributions are the same and that all classes in the test set are present in the training set. However, deployed classifiers often require the ability to recognize inputs from outside the training set as unknowns. This problem has been studied under multiple paradigms including out-of-distribution detection and open set recognition. For convolutional neural networks, there have been two major approaches: 1) inference methods to separate knowns from unknowns and 2) feature space regularization strategies to improve model robustness to outlier inputs. There has been little effort to explore the relationship between the two approaches and directly compare performance on anything other than small-scale datasets that have at most 100 categories. Using ImageNet-1K and Places-434, we identify novel combinations of regularization and specialized inference methods that perform best across multiple outlier detection problems of increasing difficulty level. We found that input perturbation and temperature scaling yield the best performance on large scale datasets regardless of the feature space regularization strategy. Improving the feature space by regularizing against a background class can be helpful if an appropriate background class can be found, but this is impractical for large scale image classification datasets.