Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation

TL;DR

Recombinator Networks are a novel deep learning architecture that effectively combines coarse and fine features early in the network to improve pixel-level prediction accuracy, especially in facial keypoint detection.

Contribution

The paper introduces Recombinator Networks, a new model that integrates coarse-to-fine features early in the network, outperforming previous summation-based methods on keypoint detection tasks.

Findings

Reduced error by 30% on AFW and AFLW datasets

Achieved state-of-the-art results on 300W without extra data

Enhanced performance with a novel denoising prediction model

Abstract

Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision. Max-pooling purposefully discards precise spatial information in order to create features that are more robust, and typically organized as lower resolution spatial feature maps. On some tasks, such as whole-image classification, max-pooling derived features are well suited; however, for tasks requiring precise localization, such as pixel level prediction and segmentation, max-pooling destroys exactly the information required to perform well. Precise localization may be preserved by shallow convnets without pooling but at the expense of robustness. Can we have our max-pooled multi-layered cake and eat it too? Several papers have proposed summation and concatenation based methods for combining upsampled coarse, abstract features with finer features to produce robust pixel level predictions. Here we introduce another model --- dubbed Recombinator Networks --- where coarse features inform finer features early in their formation such that finer features can make use of several layers of computation in deciding how to use coarse features. The model is trained once, end-to-end and performs better than summation-based architectures, reducing the error from the previous state of the art on two facial keypoint datasets, AFW and AFLW, by 30\% and beating the current state-of-the-art on 300W without using extra data. We improve performance even further by adding a denoising prediction model based on a novel convnet formulation.