Unbiased Object Detection Beyond Frequency with Visually Prompted Image Synthesis

TL;DR

This paper introduces a generation-based debiasing framework for object detection that uses a novel representation score and visual blueprints to improve detection of rare objects and generate high-quality, complex scenes.

Contribution

It proposes a new debiasing method that guides data generation beyond frequency, using representation scores and visual blueprints for better scene synthesis and detection performance.

Findings

Improves detection of rare objects by 4.4/3.6 mAP.

Surpasses prior models by 15.9 mAP in layout accuracy.

Significantly narrows performance gap for underrepresented object groups.

Abstract

This paper presents a generation-based debiasing framework for object detection. Prior debiasing methods are often limited by the representation diversity of samples, while naive generative augmentation often preserves the biases it aims to solve. Moreover, our analysis reveals that simply generating more data for rare classes is suboptimal due to two core issues: i) instance frequency is an incomplete proxy for the true data needs of a model, and ii) current layout-to-image synthesis lacks the fidelity and control to generate high-quality, complex scenes. To overcome this, we introduce the representation score (RS) to diagnose representational gaps beyond mere frequency, guiding the creation of new, unbiased layouts. To ensure high-quality synthesis, we replace ambiguous text prompts with a precise visual blueprint and employ a generative alignment strategy, which fosters communication between the detector and generator. Our method significantly narrows the performance gap for underrepresented object groups, \eg, improving large/rare instances by 4.4/3.6 mAP over the baseline, and surpassing prior L2I synthesis models by 15.9 mAP for layout accuracy in generated images.