Difficulty Controlled Diffusion Model for Synthesizing Effective Training Data

TL;DR

This paper introduces a difficulty-controlled diffusion model that generates challenging training samples to improve model performance efficiently, reducing data and computational costs.

Contribution

It presents a novel method to control sample difficulty during generation, enabling efficient creation of hard samples that enhance training effectiveness.

Findings

Achieves higher performance with only 10% additional synthetic data.

Reduces generation time by 63.4 GPU hours compared to SOTA.

Provides visualizations of category-specific hard factors.

Abstract

Generative models have become a powerful tool for synthesizing training data in computer vision tasks. Current approaches solely focus on aligning generated images with the target dataset distribution. As a result, they capture only the common features in the real dataset and mostly generate 'easy samples', which are already well learned by models trained on real data. In contrast, those rare 'hard samples', with atypical features but crucial for enhancing performance, cannot be effectively generated. Consequently, these approaches must synthesize large volumes of data to yield appreciable performance gains, yet the improvement remains limited. To overcome this limitation, we present a novel method that can learn to control the learning difficulty of samples during generation while also achieving domain alignment. Thus, it can efficiently generate valuable 'hard samples' that yield significant performance improvements for target tasks. This is achieved by incorporating learning difficulty as an additional conditioning signal in generative models, together with a designed encoder structure and training-generation strategy. Experimental results across multiple datasets show that our method can achieve higher performance with lower generation cost. Specifically, we obtain the best performance with only 10% additional synthetic data, saving 63.4 GPU hours of generation time compared to the previous SOTA on ImageNet. Moreover, our method provides insightful visualizations of category-specific hard factors, serving as a tool for analyzing datasets.