Recalling Holistic Information for Semantic Segmentation

TL;DR

This paper introduces a two-stream neural network that leverages holistic image understanding to improve pixel-level semantic segmentation, achieving state-of-the-art results on multiple datasets.

Contribution

It proposes a novel architecture that integrates holistic image content with local pixel analysis for enhanced segmentation accuracy.

Findings

State-of-the-art performance on PASCAL-Context and NYUDv2 datasets.

Effective in improving segmentation on ADE20K and SIFT-Flow datasets.

Holistic inference enhances detailed pixel labeling.

Abstract

Semantic segmentation requires a detailed labeling of image pixels by object category. Information derived from local image patches is necessary to describe the detailed shape of individual objects. However, this information is ambiguous and can result in noisy labels. Global inference of image content can instead capture the general semantic concepts present. We advocate that high-recall holistic inference of image concepts provides valuable information for detailed pixel labeling. We build a two-stream neural network architecture that facilitates information flow from holistic information to local pixels, while keeping common image features shared among the low-level layers of both the holistic analysis and segmentation branches. We empirically evaluate our network on four standard semantic segmentation datasets. Our network obtains state-of-the-art performance on PASCAL-Context and NYUDv2, and ablation studies verify its effectiveness on ADE20K and SIFT-Flow.