Deep Learning Markov Random Field for Semantic Segmentation

TL;DR

This paper introduces Deep Parsing Network (DPN), a CNN-based approach for semantic segmentation that models high-order relations and contextual information efficiently, achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes a novel CNN architecture that approximates mean field inference for MRFs, enabling end-to-end training and high performance in semantic segmentation.

Findings

Achieves state-of-the-art accuracy on PASCAL VOC 2012

Outperforms existing methods on Cityscapes and CamVid datasets

Enables efficient, parallelizable inference with a single model.

Abstract

Semantic segmentation tasks can be well modeled by Markov Random Field (MRF). This paper addresses semantic segmentation by incorporating high-order relations and mixture of label contexts into MRF. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN to model unary terms and additional layers are devised to approximate the mean field (MF) algorithm for pairwise terms. It has several appealing properties. First, different from the recent works that required many iterations of MF during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing models as its special cases. Furthermore, pairwise terms in DPN provide a unified framework to encode rich contextual information in high-dimensional data, such as images and videos. Third, DPN makes MF easier to be parallelized and speeded up, thus enabling efficient inference. DPN is thoroughly evaluated on standard semantic image/video segmentation benchmarks, where a single DPN model yields state-of-the-art segmentation accuracies on PASCAL VOC 2012, Cityscapes dataset and CamVid dataset.