Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation

TL;DR

This paper introduces a novel dynamic neural representational decoder for semantic segmentation that achieves high-resolution predictions with significantly reduced computational cost by representing local label patches with neural networks conditioned on encoder outputs.

Contribution

The paper proposes a new efficient decoder, the neural representational decoder (NRD), which leverages neural representations conditioned on encoder outputs for high-resolution semantic segmentation.

Findings

Outperforms DeeplabV3+ decoder with only 30% of the computational complexity.

Achieves competitive performance with dilated encoder methods at 15% computation.

Demonstrates effectiveness on Cityscapes, ADE20K, and PASCAL Context datasets.

Abstract

Semantic segmentation requires per-pixel prediction for a given image. Typically, the output resolution of a segmentation network is severely reduced due to the downsampling operations in the CNN backbone. Most previous methods employ upsampling decoders to recover the spatial resolution. Various decoders were designed in the literature. Here, we propose a novel decoder, termed dynamic neural representational decoder (NRD), which is simple yet significantly more efficient. As each location on the encoder's output corresponds to a local patch of the semantic labels, in this work, we represent these local patches of labels with compact neural networks. This neural representation enables our decoder to leverage the smoothness prior in the semantic label space, and thus makes our decoder more efficient. Furthermore, these neural representations are dynamically generated and conditioned on the outputs of the encoder networks. The desired semantic labels can be efficiently decoded from the neural representations, resulting in high-resolution semantic segmentation predictions. We empirically show that our proposed decoder can outperform the decoder in DeeplabV3+ with only 30% computational complexity, and achieve competitive performance with the methods using dilated encoders with only 15% computation. Experiments on the Cityscapes, ADE20K, and PASCAL Context datasets demonstrate the effectiveness and efficiency of our proposed method.