Self-Regulation for Semantic Segmentation

TL;DR

This paper identifies key failure modes in semantic segmentation and introduces Self-Regulation losses that improve detail preservation and semantic understanding without extra data, enhancing model performance.

Contribution

The paper proposes novel Self-Regulation losses that balance detailed features and semantic context, improving semantic segmentation accuracy without additional supervision.

Findings

SR losses improve segmentation of small objects

SR losses enhance semantic consistency in predictions

Method is compatible with various state-of-the-art models

Abstract

In this paper, we seek reasons for the two major failure cases in Semantic Segmentation (SS): 1) missing small objects or minor object parts, and 2) mislabeling minor parts of large objects as wrong classes. We have an interesting finding that Failure-1 is due to the underuse of detailed features and Failure-2 is due to the underuse of visual contexts. To help the model learn a better trade-off, we introduce several Self-Regulation (SR) losses for training SS neural networks. By "self", we mean that the losses are from the model per se without using any additional data or supervision. By applying the SR losses, the deep layer features are regulated by the shallow ones to preserve more details; meanwhile, shallow layer classification logits are regulated by the deep ones to capture more semantics. We conduct extensive experiments on both weakly and fully supervised SS tasks, and the results show that our approach consistently surpasses the baselines. We also validate that SR losses are easy to implement in various state-of-the-art SS models, e.g., SPGNet and OCRNet, incurring little computational overhead during training and none for testing.