StructToken : Rethinking Semantic Segmentation with Structural Prior

TL;DR

This paper introduces StructToken, a novel approach to semantic segmentation that incorporates structural prior information through learned tokens, leading to improved performance on multiple benchmarks.

Contribution

It proposes a new structure-aware extraction paradigm that leverages learned structure tokens to enhance segmentation accuracy by capturing object structural information.

Findings

Outperforms state-of-the-art on ADE20K, Cityscapes, and COCO-Stuff-10K.

Demonstrates the effectiveness of structural prior in segmentation.

Validates the approach through extensive experiments.

Abstract

In previous deep-learning-based methods, semantic segmentation has been regarded as a static or dynamic per-pixel classification task, \textit{i.e.,} classify each pixel representation to a specific category. However, these methods only focus on learning better pixel representations or classification kernels while ignoring the structural information of objects, which is critical to human decision-making mechanism. In this paper, we present a new paradigm for semantic segmentation, named structure-aware extraction. Specifically, it generates the segmentation results via the interactions between a set of learned structure tokens and the image feature, which aims to progressively extract the structural information of each category from the feature. Extensive experiments show that our StructToken outperforms the state-of-the-art on three widely-used benchmarks, including ADE20K, Cityscapes, and COCO-Stuff-10K.