Face representation by deep learning: a linear encoding in a parameter space?

TL;DR

This paper reveals that face representations in CNNs can be linearly encoded in a parameter space, challenging the complex feature encoding perspective and offering insights into CNNs' face recognition mechanisms.

Contribution

It demonstrates that CNN face representations can be linearly modeled in a parameter space, achieving comparable or better recognition performance than existing CNNs.

Findings

Linear encoding/decoding of face responses in CNNs in parameter space.

Linear model performs as well or better than state-of-the-art CNNs.

Neuron responses cannot be modeled by the axis model.

Abstract

Recently, Convolutional Neural Networks (CNNs) have achieved tremendous performances on face recognition, and one popular perspective regarding CNNs' success is that CNNs could learn discriminative face representations from face images with complex image feature encoding. However, it is still unclear what is the intrinsic mechanism of face representation in CNNs. In this work, we investigate this problem by formulating face images as points in a shape-appearance parameter space, and our results demonstrate that: (i) The encoding and decoding of the neuron responses (representations) to face images in CNNs could be achieved under a linear model in the parameter space, in agreement with the recent discovery in primate IT face neurons, but different from the aforementioned perspective on CNNs' face representation with complex image feature encoding; (ii) The linear model for face encoding and decoding in the parameter space could achieve close or even better performances on face recognition and verification than state-of-the-art CNNs, which might provide new lights on the design strategies for face recognition systems; (iii) The neuron responses to face images in CNNs could not be adequately modelled by the axis model, a model recently proposed on face modelling in primate IT cortex. All these results might shed some lights on the often complained blackbox nature behind CNNs' tremendous performances on face recognition.