Persistent Evidence of Local Image Properties in Generic ConvNets

TL;DR

This paper reveals that early layers of convolutional networks retain strong spatial information, which can be exploited for various visual tasks, challenging the assumption that training discards auxiliary image details.

Contribution

The study demonstrates persistent local image properties in generic ConvNets' early layers across multiple tasks, highlighting potential for improved training strategies.

Findings

Strong spatial information exists in early ConvNet layers.

Global image descriptors can predict landmarks and pixel labels.

Spatial info can enhance correspondence and training methods.

Abstract

Supervised training of a convolutional network for object classification should make explicit any information related to the class of objects and disregard any auxiliary information associated with the capture of the image or the variation within the object class. Does this happen in practice? Although this seems to pertain to the very final layers in the network, if we look at earlier layers we find that this is not the case. Surprisingly, strong spatial information is implicit. This paper addresses this, in particular, exploiting the image representation at the first fully connected layer, i.e. the global image descriptor which has been recently shown to be most effective in a range of visual recognition tasks. We empirically demonstrate evidences for the finding in the contexts of four different tasks: 2d landmark detection, 2d object keypoints prediction, estimation of the RGB values of input image, and recovery of semantic label of each pixel. We base our investigation on a simple framework with ridge rigression commonly across these tasks, and show results which all support our insight. Such spatial information can be used for computing correspondence of landmarks to a good accuracy, but should potentially be useful for improving the training of the convolutional nets for classification purposes.