Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

TL;DR

This paper presents a Bayesian approach to automatically generate training data from unlabeled videos, significantly improving segmentation of unseen object classes and enabling open-world instance segmentation.

Contribution

The paper introduces a novel Bayesian method that creates high-quality training sets from unlabeled videos for unseen class segmentation, outperforming existing video segmentation techniques.

Findings

Generated training sets improve unseen class segmentation performance

Method outperforms existing video segmentation approaches

Enables open-world instance segmentation using Internet videos

Abstract

The ability to localize and segment objects from unseen classes would open the door to new applications, such as autonomous object learning in active vision. Nonetheless, improving the performance on unseen classes requires additional training data, while manually annotating the objects of the unseen classes can be labor-extensive and expensive. In this paper, we explore the use of unlabeled video sequences to automatically generate training data for objects of unseen classes. It is in principle possible to apply existing video segmentation methods to unlabeled videos and automatically obtain object masks, which can then be used as a training set even for classes with no manual labels available. However, our experiments show that these methods do not perform well enough for this purpose. We therefore introduce a Bayesian method that is specifically designed to automatically create such a training set: Our method starts from a set of object proposals and relies on (non-realistic) analysis-by-synthesis to select the correct ones by performing an efficient optimization over all the frames simultaneously. Through extensive experiments, we show that our method can generate a high-quality training set which significantly boosts the performance of segmenting objects of unseen classes. We thus believe that our method could open the door for open-world instance segmentation using abundant Internet videos.