Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding

TL;DR

This paper introduces Affinity CNN, a neural network that predicts pixel pairwise relations to improve image segmentation and figure/ground organization through spectral embedding, offering a direct and effective alternative to traditional methods.

Contribution

The paper presents a CNN that directly predicts affinity matrices for spectral embedding, bypassing intermediate steps like edge detection, enhancing segmentation accuracy.

Findings

Spectral embedding with CNN-predicted affinities improves segmentation.

Direct coupling of CNN and spectral embedding outperforms traditional methods.

Spectral embedding serves as a powerful alternative to CRF-based schemes.

Abstract

Spectral embedding provides a framework for solving perceptual organization problems, including image segmentation and figure/ground organization. From an affinity matrix describing pairwise relationships between pixels, it clusters pixels into regions, and, using a complex-valued extension, orders pixels according to layer. We train a convolutional neural network (CNN) to directly predict the pairwise relationships that define this affinity matrix. Spectral embedding then resolves these predictions into a globally-consistent segmentation and figure/ground organization of the scene. Experiments demonstrate significant benefit to this direct coupling compared to prior works which use explicit intermediate stages, such as edge detection, on the pathway from image to affinities. Our results suggest spectral embedding as a powerful alternative to the conditional random field (CRF)-based globalization schemes typically coupled to deep neural networks.