One-Shot Instance Segmentation

TL;DR

This paper introduces Siamese Mask R-CNN, a novel approach for one-shot instance segmentation that uses a Siamese backbone to detect and segment objects of a new category based on a single reference image, establishing a strong baseline for this task.

Contribution

It proposes Siamese Mask R-CNN, the first strong baseline for one-shot instance segmentation, extending Mask R-CNN with a Siamese backbone to target specific object categories from a single example.

Findings

Effective transfer of segmentation knowledge to novel categories.

Detection of new object categories remains challenging.

Provides a baseline for future research in one-shot scene analysis.

Abstract

We tackle the problem of one-shot instance segmentation: Given an example image of a novel, previously unknown object category, find and segment all objects of this category within a complex scene. To address this challenging new task, we propose Siamese Mask R-CNN. It extends Mask R-CNN by a Siamese backbone encoding both reference image and scene, allowing it to target detection and segmentation towards the reference category. We demonstrate empirical results on MS Coco highlighting challenges of the one-shot setting: while transferring knowledge about instance segmentation to novel object categories works very well, targeting the detection network towards the reference category appears to be more difficult. Our work provides a first strong baseline for one-shot instance segmentation and will hopefully inspire further research into more powerful and flexible scene analysis algorithms. Code is available at: https://github.com/bethgelab/siamese-mask-rcnn