DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentation

TL;DR

This paper introduces DILLEMA, a novel framework that combines Large Language Models and Diffusion Models to generate realistic, diverse test cases for evaluating and improving the robustness of vision neural networks.

Contribution

It presents a new method for creating high-fidelity, counterfactual test images from textual descriptions, enhancing robustness testing beyond existing augmentation techniques.

Findings

Generated test cases reveal model weaknesses

Improved model robustness through targeted retraining

High human agreement on image realism

Abstract

Ensuring the robustness of deep learning models requires comprehensive and diverse testing. Existing approaches, often based on simple data augmentation techniques or generative adversarial networks, are limited in producing realistic and varied test cases. To address these limitations, we present a novel framework for testing vision neural networks that leverages Large Language Models and control-conditioned Diffusion Models to generate synthetic, high-fidelity test cases. Our approach begins by translating images into detailed textual descriptions using a captioning model, allowing the language model to identify modifiable aspects of the image and generate counterfactual descriptions. These descriptions are then used to produce new test images through a text-to-image diffusion process that preserves spatial consistency and maintains the critical elements of the scene. We demonstrate the effectiveness of our method using two datasets: ImageNet1K for image classification and SHIFT for semantic segmentation in autonomous driving. The results show that our approach can generate significant test cases that reveal weaknesses and improve the robustness of the model through targeted retraining. We conducted a human assessment using Mechanical Turk to validate the generated images. The responses from the participants confirmed, with high agreement among the voters, that our approach produces valid and realistic images.