Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation

TL;DR

This paper introduces an end-to-end framework leveraging large language models and CLIP to automatically identify and generate synthetic data for failure modes in deep learning models, significantly improving accuracy on challenging sub-populations.

Contribution

The study presents a novel method combining language and vision models to automatically describe failure modes and generate synthetic data, enhancing model robustness in a few-shot setting.

Findings

Achieved approximately 21% accuracy improvement on hard sub-populations.

Demonstrated effectiveness across 40 different models and multiple datasets.

Enabled automatic failure mode analysis without human intervention.

Abstract

Deep learning models can encounter unexpected failures, especially when dealing with challenging sub-populations. One common reason for these failures is the occurrence of objects in backgrounds that are rarely seen during training. To gain a better understanding of these failure modes, human-interpretable descriptions are crucial for further analysis and improvement which is expensive. In this study, we propose an end-to-end framework that utilizes the capabilities of large language models (ChatGPT) and vision-language deep models (CLIP) to generate text descriptions of failure modes associated with spurious correlations (e.g. rarely seen backgrounds) without human-in-the-loop intervention. These descriptions can be used to generate synthetic data using generative models, such as diffusion models. The model can now use this generated data to learn from its weaknesses and enhance its performance on backgrounds that are uncommon for each class of data. Our approach serves as a broad solution, promising progress in comprehending model failure modes and strengthening deep learning models across a wide range of failure scenarios (e.g. bacckgrounds, colors) automatically in a few-shot manner. Our experiments have shown remarkable \textbf{improvements in accuracy ($\sim \textbf{21%}$)} on hard sub-populations (particularly for wrong background association) across $40$ different models, such as ResNets, EfficientNets, DenseNets, Vision Transformer (ViT), SwAVs, MoCos, DINOs, and CLIPs on various datasets such as ImageNet-1000, CIFAR-10, and CIFAR-100.