Scene Parsing with Multiscale Feature Learning, Purity Trees, and Optimal Covers

TL;DR

This paper introduces a multiscale feature learning and tree-based segmentation approach for scene parsing, achieving high accuracy and speed by combining dense features, purity-based tree selection, and end-to-end training.

Contribution

It presents a novel scene parsing method that integrates multiscale convolutional features with purity tree selection, trained end-to-end from raw pixels, and significantly improves speed and accuracy.

Findings

Achieved record accuracy on multiple datasets.

System is an order of magnitude faster than competitors.

Produces pixel labels in less than 1 second.

Abstract

Scene parsing, or semantic segmentation, consists in labeling each pixel in an image with the category of the object it belongs to. It is a challenging task that involves the simultaneous detection, segmentation and recognition of all the objects in the image. The scene parsing method proposed here starts by computing a tree of segments from a graph of pixel dissimilarities. Simultaneously, a set of dense feature vectors is computed which encodes regions of multiple sizes centered on each pixel. The feature extractor is a multiscale convolutional network trained from raw pixels. The feature vectors associated with the segments covered by each node in the tree are aggregated and fed to a classifier which produces an estimate of the distribution of object categories contained in the segment. A subset of tree nodes that cover the image are then selected so as to maximize the average "purity" of the class distributions, hence maximizing the overall likelihood that each segment will contain a single object. The convolutional network feature extractor is trained end-to-end from raw pixels, alleviating the need for engineered features. After training, the system is parameter free. The system yields record accuracies on the Stanford Background Dataset (8 classes), the Sift Flow Dataset (33 classes) and the Barcelona Dataset (170 classes) while being an order of magnitude faster than competing approaches, producing a 320 \times 240 image labeling in less than 1 second.