Region-based semantic segmentation with end-to-end training

TL;DR

This paper introduces a novel end-to-end trainable region-based semantic segmentation method that combines the spatial support advantages of region classification with the efficiency of fully convolutional approaches, leading to improved accuracy.

Contribution

It presents a differentiable region-to-pixel layer and a free-form ROI pooling layer enabling end-to-end training for region-based segmentation.

Findings

Achieves 64.0% class-average accuracy on SIFT Flow

Achieves 49.9% class-average accuracy on PASCAL Context

Improves boundary accuracy in segmentation results

Abstract

We propose a novel method for semantic segmentation, the task of labeling each pixel in an image with a semantic class. Our method combines the advantages of the two main competing paradigms. Methods based on region classification offer proper spatial support for appearance measurements, but typically operate in two separate stages, none of which targets pixel labeling performance at the end of the pipeline. More recent fully convolutional methods are capable of end-to-end training for the final pixel labeling, but resort to fixed patches as spatial support. We show how to modify modern region-based approaches to enable end-to-end training for semantic segmentation. This is achieved via a differentiable region-to-pixel layer and a differentiable free-form Region-of-Interest pooling layer. Our method improves the state-of-the-art in terms of class-average accuracy with 64.0% on SIFT Flow and 49.9% on PASCAL Context, and is particularly accurate at object boundaries.