Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models

TL;DR

This paper introduces a method combining Bayesian optimization and text-to-image models to efficiently identify failure cases of image classifiers by generating targeted synthetic images based on textual prompts.

Contribution

It presents an iterative benchmarking approach that reduces the number of generated images needed to find classifier failures using advanced generative models.

Findings

Efficiently identifies classifier failure modes with fewer generated images.

Combines Bayesian optimization with text-to-image models for targeted failure exploration.

Demonstrates improved failure detection over random sampling methods.

Abstract

Image classifiers should be used with caution in the real world. Performance evaluated on a validation set may not reflect performance in the real world. In particular, classifiers may perform well for conditions that are frequently encountered during training, but poorly for other infrequent conditions. In this study, we hypothesize that recent advances in text-to-image generative models make them valuable for benchmarking computer vision models such as image classifiers: they can generate images conditioned by textual prompts that cause classifier failures, allowing failure conditions to be described with textual attributes. However, their generation cost becomes an issue when a large number of synthetic images need to be generated, which is the case when many different attribute combinations need to be tested. We propose an image classifier benchmarking method as an iterative process that alternates image generation, classifier evaluation, and attribute selection. This method efficiently explores the attributes that ultimately lead to poor behavior detection.