DiffusionEngine: Diffusion Model is Scalable Data Engine for Object Detection

TL;DR

DiffusionEngine leverages diffusion models to generate diverse, high-quality detection data, significantly improving object detection performance across various scenarios without complex data collection or augmentation.

Contribution

The paper introduces DiffusionEngine, a novel data scaling method using diffusion models and a Detection-Adapter for efficient, diverse detection data generation, along with new datasets COCO-DE and VOC-DE.

Findings

mAP increased by 3.1% on COCO with DE

mAP increased by 7.6% on VOC with DE

mAP increased by 11.5% on Clipart with DE

Abstract

Data is the cornerstone of deep learning. This paper reveals that the recently developed Diffusion Model is a scalable data engine for object detection. Existing methods for scaling up detection-oriented data often require manual collection or generative models to obtain target images, followed by data augmentation and labeling to produce training pairs, which are costly, complex, or lacking diversity. To address these issues, we presentDiffusionEngine (DE), a data scaling-up engine that provides high-quality detection-oriented training pairs in a single stage. DE consists of a pre-trained diffusion model and an effective Detection-Adapter, contributing to generating scalable, diverse and generalizable detection data in a plug-and-play manner. Detection-Adapter is learned to align the implicit semantic and location knowledge in off-the-shelf diffusion models with detection-aware signals to make better bounding-box predictions. Additionally, we contribute two datasets, i.e., COCO-DE and VOC-DE, to scale up existing detection benchmarks for facilitating follow-up research. Extensive experiments demonstrate that data scaling-up via DE can achieve significant improvements in diverse scenarios, such as various detection algorithms, self-supervised pre-training, data-sparse, label-scarce, cross-domain, and semi-supervised learning. For example, when using DE with a DINO-based adapter to scale up data, mAP is improved by 3.1% on COCO, 7.6% on VOC, and 11.5% on Clipart.