Per-Pixel Classification is Not All You Need for Semantic Segmentation

TL;DR

This paper introduces MaskFormer, a unified mask classification approach for semantic and panoptic segmentation that simplifies existing methods and achieves state-of-the-art results by predicting sets of binary masks with associated class labels.

Contribution

The paper proposes MaskFormer, a novel unified mask classification model that handles both semantic and instance segmentation with the same framework, loss, and training procedure.

Findings

MaskFormer outperforms per-pixel classification baselines on large class sets.

Achieves 55.6 mIoU on ADE20K for semantic segmentation.

Achieves 52.7 PQ on COCO for panoptic segmentation.

Abstract

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.