F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

TL;DR

F-VLM introduces a simplified open-vocabulary object detection method that leverages frozen vision and language models, achieving state-of-the-art results with minimal training and computational resources.

Contribution

The paper proposes F-VLM, a novel approach that uses frozen vision-language models for open-vocabulary detection, eliminating complex training pipelines and improving performance.

Findings

+6.5 mask AP over previous state-of-the-art on LVIS

Strong performance on COCO open-vocabulary detection

Significant training speed-up and compute savings

Abstract

We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier. We finetune only the detector head and combine the detector and VLM outputs for each region at inference time. F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of the art on novel categories of LVIS open-vocabulary detection benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released at the https://sites.google.com/view/f-vlm/home