Convolutional Feature Masking for Joint Object and Stuff Segmentation

TL;DR

This paper introduces a novel approach for semantic segmentation that masks convolutional features directly, enabling efficient joint object and stuff segmentation with state-of-the-art accuracy and improved computational speed.

Contribution

It proposes a new method of exploiting shape information by masking CNN features, allowing joint object and stuff segmentation without artificial boundaries and with faster computation.

Findings

Achieved state-of-the-art results on PASCAL VOC and PASCAL-CONTEXT datasets.

Demonstrated improved computational efficiency over previous methods.

Effectively handles both objects and 'stuff' in a unified framework.

Abstract

The topic of semantic segmentation has witnessed considerable progress due to the powerful features learned by convolutional neural networks (CNNs). The current leading approaches for semantic segmentation exploit shape information by extracting CNN features from masked image regions. This strategy introduces artificial boundaries on the images and may impact the quality of the extracted features. Besides, the operations on the raw image domain require to compute thousands of networks on a single image, which is time-consuming. In this paper, we propose to exploit shape information via masking convolutional features. The proposal segments (e.g., super-pixels) are treated as masks on the convolutional feature maps. The CNN features of segments are directly masked out from these maps and used to train classifiers for recognition. We further propose a joint method to handle objects and "stuff" (e.g., grass, sky, water) in the same framework. State-of-the-art results are demonstrated on benchmarks of PASCAL VOC and new PASCAL-CONTEXT, with a compelling computational speed.