Learning with Free Object Segments for Long-Tailed Instance Segmentation

TL;DR

This paper introduces FreeSeg, a scalable framework that extracts and leverages free object segments from object-centric images to augment training data, significantly improving long-tailed instance segmentation performance especially for rare classes.

Contribution

The paper proposes a novel method to obtain and utilize free object foreground segments from object-centric images, enhancing long-tailed instance segmentation models without additional data collection.

Findings

FreeSeg improves segmentation accuracy for rare object categories.

Augmentation with free segments outperforms traditional data augmentation methods.

State-of-the-art results achieved on long-tailed instance segmentation benchmarks.

Abstract

One fundamental challenge in building an instance segmentation model for a large number of classes in complex scenes is the lack of training examples, especially for rare objects. In this paper, we explore the possibility to increase the training examples without laborious data collection and annotation. We find that an abundance of instance segments can potentially be obtained freely from object-centric images, according to two insights: (i) an object-centric image usually contains one salient object in a simple background; (ii) objects from the same class often share similar appearances or similar contrasts to the background. Motivated by these insights, we propose a simple and scalable framework FreeSeg for extracting and leveraging these "free" object foreground segments to facilitate model training in long-tailed instance segmentation. Concretely, we investigate the similarity among object-centric images of the same class to propose candidate segments of foreground instances, followed by a novel ranking of segment quality. The resulting high-quality object segments can then be used to augment the existing long-tailed datasets, e.g., by copying and pasting the segments onto the original training images. Extensive experiments show that FreeSeg yields substantial improvements on top of strong baselines and achieves state-of-the-art accuracy for segmenting rare object categories.