GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learning

TL;DR

GridCLIP introduces a one-stage object detection method that leverages grid-level CLIP representations, achieving near two-stage detector performance with significantly faster training and inference, especially improving detection of infrequent categories.

Contribution

It proposes a novel grid-level alignment approach for one-stage detection using CLIP, narrowing the performance gap with two-stage detectors while reducing computational costs.

Findings

Achieves comparable performance to two-stage detectors on LVIS benchmark.

Significantly faster training and inference times than two-stage counterparts.

Improves detection of undersampled and novel categories.

Abstract

A vision-language foundation model pretrained on very large-scale image-text paired data has the potential to provide generalizable knowledge representation for downstream visual recognition and detection tasks, especially on supplementing the undersampled categories in downstream model training. Recent studies utilizing CLIP for object detection have shown that a two-stage detector design typically outperforms a one-stage detector, while requiring more expensive training resources and longer inference time. In this work, we propose a one-stage detector GridCLIP that narrows its performance gap to those of two-stage detectors, with approximately 43 and 5 times faster than its two-stage counterpart (ViLD) in the training and test process respectively. GridCLIP learns grid-level representations to adapt to the intrinsic principle of one-stage detection learning by expanding the conventional CLIP image-text holistic mapping to a more fine-grained, grid-text alignment. This differs from the region-text mapping in two-stage detectors that apply CLIP directly by treating regions as images. Specifically, GridCLIP performs Grid-level Alignment to adapt the CLIP image-level representations to grid-level representations by aligning to CLIP category representations to learn the annotated (especially frequent) categories. To learn generalizable visual representations of broader categories, especially undersampled ones, we perform Image-level Alignment during training to propagate broad pre-learned categories in the CLIP image encoder from the image-level to the grid-level representations. Experiments show that the learned CLIP-based grid-level representations boost the performance of undersampled (infrequent and novel) categories, reaching comparable detection performance on the LVIS benchmark.