From Local Matches to Global Masks: Template-Guided Instance Detection and Segmentation in Open-World Scenes

TL;DR

This paper introduces L2G-Det, a novel detection framework that uses dense patch matching and a segmentation model to detect and segment novel objects in cluttered, unseen scenes without relying on proposals.

Contribution

The paper presents a local-to-global detection approach that bypasses proposals and integrates with SAM for accurate instance segmentation in open-world environments.

Findings

L2G-Det outperforms proposal-based methods in open-world detection tasks.

Dense patch matching effectively locates objects under occlusion and clutter.

The method reliably reconstructs complete object masks in challenging scenes.

Abstract

Detecting and segmenting novel object instances in open-world environments is a fundamental problem in robotic perception. Given only a small set of template images, a robot must locate and segment a specific object instance in a cluttered, previously unseen scene. Existing proposal-based approaches are highly sensitive to proposal quality and often fail under occlusion and background clutter. We propose L2G-Det, a local-to-global instance detection framework that bypasses explicit object proposals by leveraging dense patch-level matching between templates and the query image. Locally matched patches generate candidate points, which are refined through a candidate selection module to suppress false positives. The filtered points are then used to prompt an augmented Segment Anything Model (SAM) with instance-specific object tokens, enabling reliable reconstruction of complete instance masks. Experiments demonstrate improved performance over proposal-based methods in challenging open-world settings.