Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision

TL;DR

This paper introduces UDOS, a novel open-world instance segmentation method that combines top-down learning with bottom-up supervision, improving generalization to unseen categories across multiple datasets.

Contribution

The paper proposes a new open-world segmentation approach that integrates bottom-up class-agnostic segmentation with top-down learning, enhancing generalization and efficiency.

Findings

Significant performance improvements over state-of-the-art methods.

Effective cross-category and cross-dataset transfer capabilities.

Robust instance segmentation on diverse challenging datasets.

Abstract

Many top-down architectures for instance segmentation achieve significant success when trained and tested on pre-defined closed-world taxonomy. However, when deployed in the open world, they exhibit notable bias towards seen classes and suffer from significant performance drop. In this work, we propose a novel approach for open world instance segmentation called bottom-Up and top-Down Open-world Segmentation (UDOS) that combines classical bottom-up segmentation algorithms within a top-down learning framework. UDOS first predicts parts of objects using a top-down network trained with weak supervision from bottom-up segmentations. The bottom-up segmentations are class-agnostic and do not overfit to specific taxonomies. The part-masks are then fed into affinity-based grouping and refinement modules to predict robust instance-level segmentations. UDOS enjoys both the speed and efficiency from the top-down architectures and the generalization ability to unseen categories from bottom-up supervision. We validate the strengths of UDOS on multiple cross-category as well as cross-dataset transfer tasks from 5 challenging datasets including MS-COCO, LVIS, ADE20k, UVO and OpenImages, achieving significant improvements over state-of-the-art across the board. Our code and models are available on our project page.