Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentation

TL;DR

This paper introduces NIDS-Net, a unified framework that leverages large vision models and a novel weight adapter to improve novel instance detection and segmentation, achieving state-of-the-art results across multiple datasets.

Contribution

The paper presents a new framework combining large vision models with a weight adapter for high-quality embeddings, enhancing few-shot novel instance detection and segmentation performance.

Findings

Outperforms current state-of-the-art methods on four detection datasets.

Achieves superior segmentation results on BOP challenge datasets.

Demonstrates effectiveness on real-world images from robots and cameras.

Abstract

Novel Instance Detection and Segmentation (NIDS) aims at detecting and segmenting novel object instances given a few examples of each instance. We propose a unified, simple, yet effective framework (NIDS-Net) comprising object proposal generation, embedding creation for both instance templates and proposal regions, and embedding matching for instance label assignment. Leveraging recent advancements in large vision methods, we utilize Grounding DINO and Segment Anything Model (SAM) to obtain object proposals with accurate bounding boxes and masks. Central to our approach is the generation of high-quality instance embeddings. We utilized foreground feature averages of patch embeddings from the DINOv2 ViT backbone, followed by refinement through a weight adapter mechanism that we introduce. We show experimentally that our weight adapter can adjust the embeddings locally within their feature space and effectively limit overfitting in the few-shot setting. Furthermore, the weight adapter optimizes weights to enhance the distinctiveness of instance embeddings during similarity computation. This methodology enables a straightforward matching strategy that results in significant performance gains. Our framework surpasses current state-of-the-art methods, demonstrating notable improvements in four detection datasets. In the segmentation tasks on seven core datasets of the BOP challenge, our method outperforms the leading published RGB methods and remains competitive with the best RGB-D method. We have also verified our method using real-world images from a Fetch robot and a RealSense camera. Project Page: https://irvlutd.github.io/NIDSNet/