Continual Improvement of Threshold-Based Novelty Detection

TL;DR

This paper introduces an automatic threshold selection method for novelty detection in neural networks, improving accuracy in open-world scenarios by adapting thresholds dynamically rather than relying on manual tuning.

Contribution

The paper presents a novel automatic threshold selection technique using linear search and leave-one-out validation for improved novelty detection.

Findings

Enhanced accuracy on MNIST, Fashion MNIST, and CIFAR-10 datasets.

Automatic threshold selection outperforms manual tuning methods.

Method adapts to data, improving detection in dynamic environments.

Abstract

When evaluated in dynamic, open-world situations, neural networks struggle to detect unseen classes. This issue complicates the deployment of continual learners in realistic environments where agents are not explicitly informed when novel categories are encountered. A common family of techniques for detecting novelty relies on thresholds of similarity between observed data points and the data used for training. However, these methods often require manually specifying (ahead of time) the value of these thresholds, and are therefore incapable of adapting to the nature of the data. We propose a new method for automatically selecting these thresholds utilizing a linear search and leave-one-out cross-validation on the ID classes. We demonstrate that this novel method for selecting thresholds results in improved total accuracy on MNIST, Fashion MNIST, and CIFAR-10.