Learning to Fuse Things and Stuff

TL;DR

This paper introduces TASCNet, an end-to-end model for panoptic segmentation that unifies instance and semantic segmentation tasks using shared features and cross-task consistency constraints.

Contribution

It presents a novel unified network architecture for panoptic segmentation that combines instance and semantic segmentation with explicit cross-task consistency enforcement.

Findings

Competitive performance on panoptic segmentation benchmarks

Effective unification of instance and semantic segmentation tasks

Improved cross-task consistency in segmentation outputs

Abstract

We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation. Our model, TASCNet, uses feature maps from a shared backbone network to predict in a single feed-forward pass both things and stuff segmentations. We explicitly constrain these two output distributions through a global things and stuff binary mask to enforce cross-task consistency. Our proposed unified network is competitive with the state of the art on several benchmarks for panoptic segmentation as well as on the individual semantic and instance segmentation tasks.