Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models

TL;DR

This paper introduces Roboflow100-VL, a comprehensive multi-domain benchmark for evaluating vision-language models on diverse object detection tasks, highlighting their limitations and the potential of few-shot concept alignment.

Contribution

We present Roboflow100-VL, a large-scale multi-modal object detection benchmark with diverse, out-of-distribution concepts, and evaluate model performance across various data regimes.

Findings

VLMs like GroundingDINO achieve less than 2% zero-shot accuracy on medical datasets.

Few-shot concept alignment significantly improves detection performance.

Community participation in CVPR 2025 FSOD competition outperforms baseline models.

Abstract

Vision-language models (VLMs) trained on internet-scale data achieve remarkable zero-shot detection performance on common objects like car, truck, and pedestrian. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training. Rather than simply re-training VLMs on more visual data, we argue that one should align VLMs to new concepts with annotation instructions containing a few visual examples and rich textual descriptions. To this end, we introduce Roboflow100-VL, a large-scale collection of 100 multi-modal object detection datasets with diverse concepts not commonly found in VLM pre-training. We evaluate state-of-the-art models on our benchmark in zero-shot, few-shot, semi-supervised, and fully-supervised settings, allowing for comparison across data regimes. Notably, we find that VLMs like GroundingDINO and Qwen2.5-VL achieve less than 2% zero-shot accuracy on challenging medical imaging datasets within Roboflow100-VL, demonstrating the need for few-shot concept alignment. Lastly, we discuss our recent CVPR 2025 Foundational FSOD competition and share insights from the community. Notably, the winning team significantly outperforms our baseline by 17 mAP! Our code and dataset are available at https://github.com/roboflow/rf100-vl and https://universe.roboflow.com/rf100-vl/.