What is Holding Back Convnets for Detection?

TL;DR

This paper investigates the limitations of convolutional neural networks in object detection, revealing their invariance issues and proposing data augmentation with renderings to improve performance.

Contribution

It provides an empirical analysis of convnet invariance issues and demonstrates that architectural changes are necessary beyond data augmentation.

Findings

ConvNets are not invariant to appearance factors.

Increasing training data alone does not mitigate weak points.

Data augmentation with renderings improves detection and view-point estimation.

Abstract

Convolutional neural networks have recently shown excellent results in general object detection and many other tasks. Albeit very effective, they involve many user-defined design choices. In this paper we want to better understand these choices by inspecting two key aspects "what did the network learn?", and "what can the network learn?". We exploit new annotations (Pascal3D+), to enable a new empirical analysis of the R-CNN detector. Despite common belief, our results indicate that existing state-of-the-art convnet architectures are not invariant to various appearance factors. In fact, all considered networks have similar weak points which cannot be mitigated by simply increasing the training data (architectural changes are needed). We show that overall performance can improve when using image renderings for data augmentation. We report the best known results on the Pascal3D+ detection and view-point estimation tasks.