Hierarchical Lov\'asz Embeddings for Proposal-free Panoptic Segmentation

TL;DR

This paper introduces Hierarchical Lovász Embeddings, a unified, proposal-free approach for panoptic segmentation that encodes instance and category information in a single pixel feature space, achieving state-of-the-art results.

Contribution

It proposes a novel hierarchical embedding method that unifies instance and category segmentation without separate branches or proposals, simplifying and improving panoptic segmentation.

Findings

Achieves state-of-the-art results on Cityscapes, COCO, and Mapillary Vistas datasets.

Demonstrates temporal stability in video frame segmentation.

Effectively models both 'thing' and 'stuff' classes with a simple classifier.

Abstract

Panoptic segmentation brings together two separate tasks: instance and semantic segmentation. Although they are related, unifying them faces an apparent paradox: how to learn simultaneously instance-specific and category-specific (i.e. instance-agnostic) representations jointly. Hence, state-of-the-art panoptic segmentation methods use complex models with a distinct stream for each task. In contrast, we propose Hierarchical Lov\'asz Embeddings, per pixel feature vectors that simultaneously encode instance- and category-level discriminative information. We use a hierarchical Lov\'asz hinge loss to learn a low-dimensional embedding space structured into a unified semantic and instance hierarchy without requiring separate network branches or object proposals. Besides modeling instances precisely in a proposal-free manner, our Hierarchical Lov\'asz Embeddings generalize to categories by using a simple Nearest-Class-Mean classifier, including for non-instance "stuff" classes where instance segmentation methods are not applicable. Our simple model achieves state-of-the-art results compared to existing proposal-free panoptic segmentation methods on Cityscapes, COCO, and Mapillary Vistas. Furthermore, our model demonstrates temporal stability between video frames.