Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

TL;DR

This paper introduces ECENet, a novel semantic segmentation approach that explicitly generates and enhances class embeddings based on class masks, improving accuracy and efficiency by revisiting decoding and feature reconstruction processes.

Contribution

The paper proposes ECENet, a new segmentation paradigm that explicitly interacts with class masks to generate meaningful class embeddings and revisits decoding with inverted information flow.

Findings

ECENet outperforms existing methods on ADE20K with less computational cost.

Achieves state-of-the-art results on PASCAL-Context dataset.

Introduces a Feature Reconstruction module to enhance feature discriminability.

Abstract

Semantic segmentation is a computer vision task that associates a label with each pixel in an image. Modern approaches tend to introduce class embeddings into semantic segmentation for deeply utilizing category semantics, and regard supervised class masks as final predictions. In this paper, we explore the mechanism of class embeddings and have an insight that more explicit and meaningful class embeddings can be generated based on class masks purposely. Following this observation, we propose ECENet, a new segmentation paradigm, in which class embeddings are obtained and enhanced explicitly during interacting with multi-stage image features. Based on this, we revisit the traditional decoding process and explore inverted information flow between segmentation masks and class embeddings. Furthermore, to ensure the discriminability and informativity of features from backbone, we propose a Feature Reconstruction module, which combines intrinsic and diverse branches together to ensure the concurrence of diversity and redundancy in features. Experiments show that our ECENet outperforms its counterparts on the ADE20K dataset with much less computational cost and achieves new state-of-the-art results on PASCAL-Context dataset. The code will be released at https://gitee.com/mindspore/models and https://github.com/Carol-lyh/ECENet.