How are attributes expressed in face DCNNs?

TL;DR

This paper introduces a method to measure how much facial attributes like age, sex, and pose are encoded in face recognition DCNNs, revealing attribute expressivity and its evolution during training.

Contribution

It proposes a novel expressivity measure using a secondary neural network to quantify attribute information in face DCNN features, and analyzes how this encoding changes over training.

Findings

Expressivity order: Age > Sex > Yaw in final layers.

Expressivity of attributes decreases as training progresses.

Method can help identify bias sources in face recognition networks.

Abstract

As deep networks become increasingly accurate at recognizing faces, it is vital to understand how these networks process faces. While these networks are solely trained to recognize identities, they also contain face related information such as sex, age, and pose of the face. The networks are not trained to learn these attributes. We introduce expressivity as a measure of how much a feature vector informs us about an attribute, where a feature vector can be from internal or final layers of a network. Expressivity is computed by a second neural network whose inputs are features and attributes. The output of the second neural network approximates the mutual information between feature vectors and an attribute. We investigate the expressivity for two different deep convolutional neural network (DCNN) architectures: a Resnet-101 and an Inception Resnet v2. In the final fully connected layer of the networks, we found the order of expressivity for facial attributes to be Age > Sex > Yaw. Additionally, we studied the changes in the encoding of facial attributes over training iterations. We found that as training progresses, expressivities of yaw, sex, and age decrease. Our technique can be a tool for investigating the sources of bias in a network and a step towards explaining the network's identity decisions.