Single-Shot Panoptic Segmentation

TL;DR

This paper introduces a fast, end-to-end single-shot panoptic segmentation method that combines object detection and semantic segmentation to achieve near real-time performance, suitable for robotics and other applications.

Contribution

It presents a novel unified network architecture that performs panoptic segmentation in a single shot, relaxing the need for merging multiple models and enabling high-speed processing.

Findings

Achieves 32.6% PQ on MS-COCO dataset

Operates at 23.5 frames per second

Handles overlapping predictions effectively

Abstract

We present a novel end-to-end single-shot method that segments countable object instances (things) as well as background regions (stuff) into a non-overlapping panoptic segmentation at almost video frame rate. Current state-of-the-art methods are far from reaching video frame rate and mostly rely on merging instance segmentation with semantic background segmentation, making them impractical to use in many applications such as robotics. Our approach relaxes this requirement by using an object detector but is still able to resolve inter- and intra-class overlaps to achieve a non-overlapping segmentation. On top of a shared encoder-decoder backbone, we utilize multiple branches for semantic segmentation, object detection, and instance center prediction. Finally, our panoptic head combines all outputs into a panoptic segmentation and can even handle conflicting predictions between branches as well as certain false predictions. Our network achieves 32.6% PQ on MS-COCO at 23.5 FPS, opening up panoptic segmentation to a broader field of applications.