SOS: Segment Object System for Open-World Instance Segmentation With Object Priors

TL;DR

This paper introduces SOS, a novel system for open-world instance segmentation that leverages foundation models and object priors to improve generalization and precision in segmenting unknown objects across diverse datasets.

Contribution

SOS is the first approach to explicitly incorporate object priors and foundation models for open-world instance segmentation, significantly enhancing generalization and accuracy.

Findings

Achieves up to 81.6% precision improvement over state-of-the-art.

Demonstrates strong generalization on COCO, LVIS, and ADE20k datasets.

Effectively utilizes self-attention maps from Vision Transformers as object priors.

Abstract

We propose an approach for Open-World Instance Segmentation (OWIS), a task that aims to segment arbitrary unknown objects in images by generalizing from a limited set of annotated object classes during training. Our Segment Object System (SOS) explicitly addresses the generalization ability and the low precision of state-of-the-art systems, which often generate background detections. To this end, we generate high-quality pseudo annotations based on the foundation model SAM. We thoroughly study various object priors to generate prompts for SAM, explicitly focusing the foundation model on objects. The strongest object priors were obtained by self-attention maps from self-supervised Vision Transformers, which we utilize for prompting SAM. Finally, the post-processed segments from SAM are used as pseudo annotations to train a standard instance segmentation system. Our approach shows strong generalization capabilities on COCO, LVIS, and ADE20k datasets and improves on the precision by up to 81.6% compared to the state-of-the-art. Source code is available at: https://github.com/chwilms/SOS