Enhancing Novel Object Detection via Cooperative Foundational Models

TL;DR

This paper introduces a cooperative approach leveraging foundational models like CLIP and SAM to enhance novel object detection, transforming closed-set detectors into open-set ones and achieving state-of-the-art results on LVIS and COCO datasets.

Contribution

It proposes a novel cooperative mechanism that integrates foundational models with existing detectors to improve open-set and novel object detection capabilities.

Findings

Achieves 17.42 mAP for novel objects on LVIS

Surpasses state-of-the-art by 7.2 AP50 on COCO OVD split

Establishes new benchmarks in open-set object detection

Abstract

In this work, we address the challenging and emergent problem of novel object detection (NOD), focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed-set, limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models, specifically CLIP and SAM, through our cooperative mechanism. Furthermore, by integrating this mechanism with state-of-the-art open-set detectors such as GDINO, we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split, we surpass the current state-of-the-art by a margin of 7.2 $ \text{AP}_{50} $ for novel classes. Our code is available at https://rohit901.github.io/coop-foundation-models/ .