Coarse-to-fine Semantic Segmentation from Image-level Labels

TL;DR

This paper introduces a recursive coarse-to-fine semantic segmentation framework that uses only image-level labels, reducing annotation costs and handling multi-category images effectively.

Contribution

The proposed method is the first to achieve competitive semantic segmentation performance using only image-level labels and can be extended to foreground object segmentation.

Findings

Achieves comparable performance to state-of-the-art methods on PASCAL VOC.

Successfully extends to foreground object segmentation with competitive results.

Handles multi-category object images with only one label per image.

Abstract

Deep neural network-based semantic segmentation generally requires large-scale cost extensive annotations for training to obtain better performance. To avoid pixel-wise segmentation annotations which are needed for most methods, recently some researchers attempted to use object-level labels (e.g. bounding boxes) or image-level labels (e.g. image categories). In this paper, we propose a novel recursive coarse-to-fine semantic segmentation framework based on only image-level category labels. For each image, an initial coarse mask is first generated by a convolutional neural network-based unsupervised foreground segmentation model and then is enhanced by a graph model. The enhanced coarse mask is fed to a fully convolutional neural network to be recursively refined. Unlike existing image-level label-based semantic segmentation methods which require to label all categories for images contain multiple types of objects, our framework only needs one label for each image and can handle images contains multi-category objects. With only trained on ImageNet, our framework achieves comparable performance on PASCAL VOC dataset as other image-level label-based state-of-the-arts of semantic segmentation. Furthermore, our framework can be easily extended to foreground object segmentation task and achieves comparable performance with the state-of-the-art supervised methods on the Internet Object dataset.