The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge

TL;DR

This paper presents VLM+ a novel framework that enhances few-shot object detection by integrating multimodal large language models to generate better pseudo-labels, significantly improving detection accuracy.

Contribution

The paper introduces VLM+ which combines vision-language models with large language models to generate refined pseudo-labels for improved few-shot object detection.

Findings

Achieved 32.56 mAP on the test set.

Enhanced pseudo-label quality improves detection performance.

Iterative pseudo-label optimization further boosts results.

Abstract

This report introduces an enhanced method for the Foundational Few-Shot Object Detection (FSOD) task, leveraging the vision-language model (VLM) for object detection. However, on specific datasets, VLM may encounter the problem where the detected targets are misaligned with the target concepts of interest. This misalignment hinders the zero-shot performance of VLM and the application of fine-tuning methods based on pseudo-labels. To address this issue, we propose the VLM+ framework, which integrates the multimodal large language model (MM-LLM). Specifically, we use MM-LLM to generate a series of referential expressions for each category. Based on the VLM predictions and the given annotations, we select the best referential expression for each category by matching the maximum IoU. Subsequently, we use these referential expressions to generate pseudo-labels for all images in the training set and then combine them with the original labeled data to fine-tune the VLM. Additionally, we employ iterative pseudo-label generation and optimization to further enhance the performance of the VLM. Our approach achieve 32.56 mAP in the final test.