Amodal Panoptic Segmentation

TL;DR

This paper introduces amodal panoptic segmentation, a new task that combines visible and occluded object segmentation, extends datasets, proposes a new network, and demonstrates state-of-the-art results, advancing robotic perception capabilities.

Contribution

It formulates the novel task of amodal panoptic segmentation, extends datasets with labels, and proposes the APSNet model to address the challenge.

Findings

APSNet achieves state-of-the-art performance on benchmarks.

The new metrics APQ and APC effectively quantify amodal segmentation quality.

The extended datasets facilitate research in amodal perception.

Abstract

Humans have the remarkable ability to perceive objects as a whole, even when parts of them are occluded. This ability of amodal perception forms the basis of our perceptual and cognitive understanding of our world. To enable robots to reason with this capability, we formulate and propose a novel task that we name amodal panoptic segmentation. The goal of this task is to simultaneously predict the pixel-wise semantic segmentation labels of the visible regions of stuff classes and the instance segmentation labels of both the visible and occluded regions of thing classes. To facilitate research on this new task, we extend two established benchmark datasets with pixel-level amodal panoptic segmentation labels that we make publicly available as KITTI-360-APS and BDD100K-APS. We present several strong baselines, along with the amodal panoptic quality (APQ) and amodal parsing coverage (APC) metrics to quantify the performance in an interpretable manner. Furthermore, we propose the novel amodal panoptic segmentation network (APSNet), as a first step towards addressing this task by explicitly modeling the complex relationships between the occluders and occludes. Extensive experimental evaluations demonstrate that APSNet achieves state-of-the-art performance on both benchmarks and more importantly exemplifies the utility of amodal recognition. The benchmarks are available at http://amodal-panoptic.cs.uni-freiburg.de.