Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation

TL;DR

This paper introduces PISR, a relational context encoder that enhances panoptic segmentation by capturing relations among semantic classes and instances, leading to improved performance across multiple benchmarks.

Contribution

The paper proposes a novel PISR module that models relations among semantic and instance contexts, improving panoptic segmentation accuracy and robustness.

Findings

Significant performance improvements on Cityscapes, COCO, and ADE20K benchmarks.

PISR effectively captures relations and focuses on important instances.

Applicable to various existing panoptic segmentation architectures.

Abstract

This paper presents a novel framework to integrate both semantic and instance contexts for panoptic segmentation. In existing works, it is common to use a shared backbone to extract features for both things (countable classes such as vehicles) and stuff (uncountable classes such as roads). This, however, fails to capture the rich relations among them, which can be utilized to enhance visual understanding and segmentation performance. To address this shortcoming, we propose a novel Panoptic, Instance, and Semantic Relations (PISR) module to exploit such contexts. First, we generate panoptic encodings to summarize key features of the semantic classes and predicted instances. A Panoptic Relational Attention (PRA) module is then applied to the encodings and the global feature map from the backbone. It produces a feature map that captures 1) the relations across semantic classes and instances and 2) the relations between these panoptic categories and spatial features. PISR also automatically learns to focus on the more important instances, making it robust to the number of instances used in the relational attention module. Moreover, PISR is a general module that can be applied to any existing panoptic segmentation architecture. Through extensive evaluations on panoptic segmentation benchmarks like Cityscapes, COCO, and ADE20K, we show that PISR attains considerable improvements over existing approaches.