Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network

TL;DR

This paper introduces a unified, faster, and more memory-efficient neural network for panoptic segmentation that combines semantic and instance segmentation in a single architecture, demonstrating competitive results on COCO and Mapillary datasets.

Contribution

The paper proposes the first joint training approach for panoptic segmentation, integrating semantic and instance segmentation into one end-to-end network.

Findings

Achieved a PQ score of 17.6 on Mapillary Vistas validation set.

Achieved a PQ score of 27.2 on COCO test-dev set.

Faster and more memory-efficient than separate networks.

Abstract

We present a single network method for panoptic segmentation. This method combines the predictions from a jointly trained semantic and instance segmentation network using heuristics. Joint training is the first step towards an end-to-end panoptic segmentation network and is faster and more memory efficient than training and predicting with two networks, as done in previous work. The architecture consists of a ResNet-50 feature extractor shared by the semantic segmentation and instance segmentation branch. For instance segmentation, a Mask R-CNN type of architecture is used, while the semantic segmentation branch is augmented with a Pyramid Pooling Module. Results for this method are submitted to the COCO and Mapillary Joint Recognition Challenge 2018. Our approach achieves a PQ score of 17.6 on the Mapillary Vistas validation set and 27.2 on the COCO test-dev set.