Few-shot Algorithm Assurance

TL;DR

This paper introduces a novel method for assessing deep learning model reliability under image distortions, using a Level Set Estimation classifier and synthetic data generation, especially effective in few-sample scenarios.

Contribution

It proposes a new classifier based on Level Set Estimation for model assurance under distortions and extends it to few-sample settings with a novel VAE for synthetic data generation.

Findings

Outperforms strong baselines on five benchmark datasets

Effective in few-sample scenarios with synthetic image generation

Accurately predicts model accuracy above thresholds under distortions

Abstract

In image classification tasks, deep learning models are vulnerable to image distortion. For successful deployment, it is important to identify distortion levels under which the model is usable i.e. its accuracy stays above a stipulated threshold. We refer to this problem as Model Assurance under Image Distortion, and formulate it as a classification task. Given a distortion level, our goal is to predict if the model's accuracy on the set of distorted images is greater than a threshold. We propose a novel classifier based on a Level Set Estimation (LSE) algorithm, which uses the LSE's mean and variance functions to form the classification rule. We further extend our method to a "few sample" setting where we can only acquire few real images to perform the model assurance process. Our idea is to generate extra synthetic images using a novel Conditional Variational Autoencoder model with two new loss functions. We conduct extensive experiments to show that our classification method significantly outperforms strong baselines on five benchmark image datasets.