A Deep Semantic Segmentation Network with Semantic and Contextual Refinements

TL;DR

This paper introduces a semantic segmentation network with novel refinement modules that improve boundary accuracy and global context understanding, achieving state-of-the-art results with efficient computation.

Contribution

The paper proposes Semantic and Contextual Refinement Modules that enhance segmentation accuracy by addressing misalignment and capturing global context, with validation on multiple datasets.

Findings

Achieved superior performance on Cityscapes, Bdd100K, and ADE20K datasets.

Enhanced boundary delineation through pixel-wise offset learning.

Lightweight model reaches 82.5% mIoU with low computational cost.

Abstract

Semantic segmentation is a fundamental task in multimedia processing, which can be used for analyzing, understanding, editing contents of images and videos, among others. To accelerate the analysis of multimedia data, existing segmentation researches tend to extract semantic information by progressively reducing the spatial resolutions of feature maps. However, this approach introduces a misalignment problem when restoring the resolution of high-level feature maps. In this paper, we design a Semantic Refinement Module (SRM) to address this issue within the segmentation network. Specifically, SRM is designed to learn a transformation offset for each pixel in the upsampled feature maps, guided by high-resolution feature maps and neighboring offsets. By applying these offsets to the upsampled feature maps, SRM enhances the semantic representation of the segmentation network, particularly for pixels around object boundaries. Furthermore, a Contextual Refinement Module (CRM) is presented to capture global context information across both spatial and channel dimensions. To balance dimensions between channel and space, we aggregate the semantic maps from all four stages of the backbone to enrich channel context information. The efficacy of these proposed modules is validated on three widely used datasets-Cityscapes, Bdd100K, and ADE20K-demonstrating superior performance compared to state-of-the-art methods. Additionally, this paper extends these modules to a lightweight segmentation network, achieving an mIoU of 82.5% on the Cityscapes validation set with only 137.9 GFLOPs.