ImaginaryNet: Learning Object Detectors without Real Images and Annotations

TL;DR

ImaginaryNet enables object detection without real images or annotations by synthesizing training data through language models and text-to-image generation, achieving competitive performance in a novel learning paradigm.

Contribution

This paper introduces ImaginaryNet, a framework that synthesizes training images from language descriptions to facilitate object detection without real data or annotations.

Findings

Achieves about 70% of weakly supervised performance without real images.

Significantly improves detection performance when combined with other supervision methods.

State-of-the-art or comparable results in zero-annotation object detection.

Abstract

Without the demand of training in reality, humans can easily detect a known concept simply based on its language description. Empowering deep learning with this ability undoubtedly enables the neural network to handle complex vision tasks, e.g., object detection, without collecting and annotating real images. To this end, this paper introduces a novel challenging learning paradigm Imaginary-Supervised Object Detection (ISOD), where neither real images nor manual annotations are allowed for training object detectors. To resolve this challenge, we propose ImaginaryNet, a framework to synthesize images by combining pretrained language model and text-to-image synthesis model. Given a class label, the language model is used to generate a full description of a scene with a target object, and the text-to-image model deployed to generate a photo-realistic image. With the synthesized images and class labels, weakly supervised object detection can then be leveraged to accomplish ISOD. By gradually introducing real images and manual annotations, ImaginaryNet can collaborate with other supervision settings to further boost detection performance. Experiments show that ImaginaryNet can (i) obtain about 70% performance in ISOD compared with the weakly supervised counterpart of the same backbone trained on real data, (ii) significantly improve the baseline while achieving state-of-the-art or comparable performance by incorporating ImaginaryNet with other supervision settings.