GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detection

TL;DR

GenDet introduces a novel object detection framework that formulates detection as an image generation task using a diffusion model, enabling precise bounding box and category prediction within a generative paradigm.

Contribution

It pioneers the use of large-scale pre-trained diffusion models for object detection, integrating semantic constraints for accurate bounding box generation in the image space.

Findings

Achieves competitive accuracy with traditional detectors

Provides flexible and controllable detection outputs

Bridges generative models with discriminative detection tasks

Abstract

This paper presents GenDet, a novel framework that redefines object detection as an image generation task. In contrast to traditional approaches, GenDet adopts a pioneering approach by leveraging generative modeling: it conditions on the input image and directly generates bounding boxes with semantic annotations in the original image space. GenDet establishes a conditional generation architecture built upon the large-scale pre-trained Stable Diffusion model, formulating the detection task as semantic constraints within the latent space. It enables precise control over bounding box positions and category attributes, while preserving the flexibility of the generative model. This novel methodology effectively bridges the gap between generative models and discriminative tasks, providing a fresh perspective for constructing unified visual understanding systems. Systematic experiments demonstrate that GenDet achieves competitive accuracy compared to discriminative detectors, while retaining the flexibility characteristic of generative methods.