PixelNet: Representation of the pixels, by the pixels, and for the pixels

TL;DR

PixelNet introduces a pixel sampling strategy that enhances learning efficiency and accuracy across various pixel-level prediction tasks, achieving state-of-the-art results by training from scratch.

Contribution

The paper proposes stratified pixel sampling to improve training efficiency and model performance for diverse pixel prediction tasks, enabling training from scratch.

Findings

Achieved state-of-the-art results on PASCAL-Context, NYUDv2, and BSDS datasets.

Stratified sampling speeds up learning and improves accuracy.

Effective training of complex models from scratch.

Abstract

We explore design principles for general pixel-level prediction problems, from low-level edge detection to mid-level surface normal estimation to high-level semantic segmentation. Convolutional predictors, such as the fully-convolutional network (FCN), have achieved remarkable success by exploiting the spatial redundancy of neighboring pixels through convolutional processing. Though computationally efficient, we point out that such approaches are not statistically efficient during learning precisely because spatial redundancy limits the information learned from neighboring pixels. We demonstrate that stratified sampling of pixels allows one to (1) add diversity during batch updates, speeding up learning; (2) explore complex nonlinear predictors, improving accuracy; and (3) efficiently train state-of-the-art models tabula rasa (i.e., "from scratch") for diverse pixel-labeling tasks. Our single architecture produces state-of-the-art results for semantic segmentation on PASCAL-Context dataset, surface normal estimation on NYUDv2 depth dataset, and edge detection on BSDS.