Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

TL;DR

This paper introduces a method combining deep convolutional neural networks with fully connected CRFs to improve pixel-level semantic image segmentation, achieving state-of-the-art accuracy and efficient computation.

Contribution

The work integrates DCNNs with fully connected CRFs to enhance localization in semantic segmentation, surpassing previous methods in accuracy and efficiency.

Findings

Achieved 71.6% IOU on PASCAL VOC-2012

Combined DCNN responses with CRFs for better boundary localization

Implemented efficient dense computation at 8 fps on GPU

Abstract

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.