SASFormer: Transformers for Sparsely Annotated Semantic Segmentation

TL;DR

SASFormer introduces a simple, effective framework for sparse annotated semantic segmentation using hierarchical patch attention and affinity loss, achieving state-of-the-art results with less complexity.

Contribution

The paper proposes SASFormer, a novel sparse annotation segmentation method based on segformer, utilizing hierarchical patch attention and affinity loss for improved performance.

Findings

Outperforms existing sparse annotation segmentation methods

Achieves cutting-edge performance on benchmark datasets

Simplifies the training process compared to multi-stage strategies

Abstract

Semantic segmentation based on sparse annotation has advanced in recent years. It labels only part of each object in the image, leaving the remainder unlabeled. Most of the existing approaches are time-consuming and often necessitate a multi-stage training strategy. In this work, we propose a simple yet effective sparse annotated semantic segmentation framework based on segformer, dubbed SASFormer, that achieves remarkable performance. Specifically, the framework first generates hierarchical patch attention maps, which are then multiplied by the network predictions to produce correlated regions separated by valid labels. Besides, we also introduce the affinity loss to ensure consistency between the features of correlation results and network predictions. Extensive experiments showcase that our proposed approach is superior to existing methods and achieves cutting-edge performance. The source code is available at \url{https://github.com/su-hui-zz/SASFormer}.