Predicting the Reliability of an Image Classifier under Image Distortion

TL;DR

This paper introduces a Gaussian process-based method to predict the reliability of image classifiers under various distortions, addressing class imbalance and outperforming baselines on multiple datasets.

Contribution

The paper proposes a novel Gaussian process approach to predict classifier reliability under distortions, effectively handling imbalanced training data.

Findings

Outperforms baseline methods on six datasets

Effectively handles imbalanced training data

Accurately predicts classifier reliability under distortions

Abstract

In image classification tasks, deep learning models are vulnerable to image distortions i.e. their accuracy significantly drops if the input images are distorted. An image-classifier is considered "reliable" if its accuracy on distorted images is above a user-specified threshold. For a quality control purpose, it is important to predict if the image-classifier is unreliable/reliable under a distortion level. In other words, we want to predict whether a distortion level makes the image-classifier "non-reliable" or "reliable". Our solution is to construct a training set consisting of distortion levels along with their "non-reliable" or "reliable" labels, and train a machine learning predictive model (called distortion-classifier) to classify unseen distortion levels. However, learning an effective distortion-classifier is a challenging problem as the training set is highly imbalanced. To address this problem, we propose a Gaussian process based method to rebalance the training set. We conduct extensive experiments to show that our method significantly outperforms several baselines on six popular image datasets.