On-the-fly Object Detection using StyleGAN with CLIP Guidance

TL;DR

This paper introduces an automated method for satellite object detection that combines StyleGAN and CLIP to generate detectors without human annotations, enabling rapid and scalable analysis of satellite imagery.

Contribution

The work presents a novel approach that leverages StyleGAN and CLIP to build object detectors on-the-fly without requiring labeled data or manual intervention.

Findings

Successfully detects objects in satellite images without annotations

Demonstrates the effectiveness of combining generative models with multi-modal learning

Enables rapid deployment of object detectors in new domains

Abstract

We present a fully automated framework for building object detectors on satellite imagery without requiring any human annotation or intervention. We achieve this by leveraging the combined power of modern generative models (e.g., StyleGAN) and recent advances in multi-modal learning (e.g., CLIP). While deep generative models effectively encode the key semantics pertinent to a data distribution, this information is not immediately accessible for downstream tasks, such as object detection. In this work, we exploit CLIP's ability to associate image features with text descriptions to identify neurons in the generator network, which are subsequently used to build detectors on-the-fly.