Few-shot Defect Image Generation based on Consistency Modeling

TL;DR

This paper introduces DefectDiffu, a text-guided diffusion model that enhances defect image generation by modeling intra- and inter-product consistencies, enabling high-quality, diverse, and zero-shot defect synthesis for improved defect detection.

Contribution

The paper presents a novel diffusion-based method with disentangled architecture and double-free strategy for controlled, diversified defect image generation across multiple products.

Findings

Outperforms state-of-the-art in quality and diversity of defect images

Enables zero-shot defect generation across different products

Improves downstream defect detection performance

Abstract

Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zero-shot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at https://github.com/FFDD-diffusion/DefectDiffu.