Instance-aware Semantic Segmentation via Multi-task Network Cascades

TL;DR

This paper introduces a multi-task cascade network for instance-aware semantic segmentation, achieving state-of-the-art accuracy and high speed, while also surpassing existing object detection systems.

Contribution

It presents a novel cascaded multi-task network architecture with end-to-end training for efficient and accurate instance-aware semantic segmentation.

Findings

Achieved state-of-the-art accuracy on PASCAL VOC

Processed images in 360ms using VGG-16, much faster than previous methods

Surpassed Fast/Faster R-CNN in object detection results

Abstract

Semantic segmentation research has recently witnessed rapid progress, but many leading methods are unable to identify object instances. In this paper, we present Multi-task Network Cascades for instance-aware semantic segmentation. Our model consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects. These networks form a cascaded structure, and are designed to share their convolutional features. We develop an algorithm for the nontrivial end-to-end training of this causal, cascaded structure. Our solution is a clean, single-step training framework and can be generalized to cascades that have more stages. We demonstrate state-of-the-art instance-aware semantic segmentation accuracy on PASCAL VOC. Meanwhile, our method takes only 360ms testing an image using VGG-16, which is two orders of magnitude faster than previous systems for this challenging problem. As a by product, our method also achieves compelling object detection results which surpass the competitive Fast/Faster R-CNN systems. The method described in this paper is the foundation of our submissions to the MS COCO 2015 segmentation competition, where we won the 1st place.