What Do Deep CNNs Learn About Objects?

TL;DR

This paper investigates the invariance of deep CNNs to variations in object class caused by 3D shape, pose, and photorealism, aiming to understand what features these networks learn about objects.

Contribution

The study provides a detailed analysis of CNN invariance to 3D object variations, revealing insights into the learned representations beyond 2D transformations.

Findings

CNNs show limited invariance to 3D shape and pose changes.

Photorealism affects CNN recognition performance.

Deeper layers encode more object-specific features.

Abstract

Deep convolutional neural networks learn extremely powerful image representations, yet most of that power is hidden in the millions of deep-layer parameters. What exactly do these parameters represent? Recent work has started to analyse CNN representations, finding that, e.g., they are invariant to some 2D transformations Fischer et al. (2014), but are confused by particular types of image noise Nguyen et al. (2014). In this work, we delve deeper and ask: how invariant are CNNs to object-class variations caused by 3D shape, pose, and photorealism?