Prompt-Free Conditional Diffusion for Multi-object Image Augmentation

TL;DR

This paper introduces a prompt-free conditional diffusion framework for multi-object image augmentation that uses semantic fusion and knowledge injection to improve diversity and reduce deviation from original data, enhancing downstream task performance.

Contribution

The proposed method replaces text prompts with image-based semantics and employs LoRA for knowledge injection, addressing diversity and deviation issues in multi-object image generation.

Findings

Outperforms state-of-the-art baselines in image augmentation tasks.

Enhances downstream task performance and out-of-domain generalization.

Effectively balances diversity and fidelity in generated images.

Abstract

Diffusion models has underpinned much recent advances of dataset augmentation in various computer vision tasks. However, when involving generating multi-object images as real scenarios, most existing methods either rely entirely on text condition, resulting in a deviation between the generated objects and the original data, or rely too much on the original images, resulting in a lack of diversity in the generated images, which is of limited help to downstream tasks. To mitigate both problems with one stone, we propose a prompt-free conditional diffusion framework for multi-object image augmentation. Specifically, we introduce a local-global semantic fusion strategy to extract semantics from images to replace text, and inject knowledge into the diffusion model through LoRA to alleviate the category deviation between the original model and the target dataset. In addition, we design a reward model based counting loss to assist the traditional reconstruction loss for model training. By constraining the object counts of each category instead of pixel-by-pixel constraints, bridging the quantity deviation between the generated data and the original data while improving the diversity of the generated data. Experimental results demonstrate the superiority of the proposed method over several representative state-of-the-art baselines and showcase strong downstream task gain and out-of-domain generalization capabilities. Code is available at \href{https://github.com/00why00/PFCD}{here}.