Unifying Training and Inference for Panoptic Segmentation

TL;DR

This paper introduces an end-to-end panoptic segmentation network that unifies training and inference, achieving state-of-the-art results on Cityscapes and COCO datasets without requiring post-processing.

Contribution

The proposed network employs a lightweight, end-to-end learned dense instance affinity module to unify training and inference for panoptic segmentation, eliminating the need for post-processing.

Findings

Achieves 61.4 PQ on Cityscapes with ResNet-50

Achieves 43.4 PQ on COCO with ResNet-50

Performs well with or without object mask cues

Abstract

We present an end-to-end network to bridge the gap between training and inference pipeline for panoptic segmentation, a task that seeks to partition an image into semantic regions for "stuff" and object instances for "things". In contrast to recent works, our network exploits a parametrised, yet lightweight panoptic segmentation submodule, powered by an end-to-end learnt dense instance affinity, to capture the probability that any pair of pixels belong to the same instance. This panoptic submodule gives rise to a novel propagation mechanism for panoptic logits and enables the network to output a coherent panoptic segmentation map for both "stuff" and "thing" classes, without any post-processing. Reaping the benefits of end-to-end training, our full system sets new records on the popular street scene dataset, Cityscapes, achieving 61.4 PQ with a ResNet-50 backbone using only the fine annotations. On the challenging COCO dataset, our ResNet-50-based network also delivers state-of-the-art accuracy of 43.4 PQ. Moreover, our network flexibly works with and without object mask cues, performing competitively under both settings, which is of interest for applications with computation budgets.