A Simple and Generalist Approach for Panoptic Segmentation

TL;DR

This paper introduces a simple, generalist framework for panoptic segmentation using a deep encoder-shallow decoder architecture, achieving state-of-the-art results by addressing training imbalance with centroid regression in spectral positional embeddings.

Contribution

The paper presents a minimalistic, generalist approach for panoptic segmentation that outperforms specialized methods by effectively handling training imbalance.

Findings

Achieves a panoptic quality (PQ) of 55.1 on MS-COCO

Outperforms previous generalist methods in panoptic segmentation

Introduces centroid regression in spectral positional embeddings to improve training balance

Abstract

Panoptic segmentation is an important computer vision task, where the current state-of-the-art solutions require specialized components to perform well. We propose a simple generalist framework based on a deep encoder - shallow decoder architecture with per-pixel prediction. Essentially fine-tuning a massively pretrained image model with minimal additional components. Naively this method does not yield good results. We show that this is due to imbalance during training and propose a novel method for reducing it - centroid regression in the space of spectral positional embeddings. Our method achieves panoptic quality (PQ) of 55.1 on the challenging MS-COCO dataset, state-of-the-art performance among generalist methods.