Identifying individual facial expressions by deconstructing a neural network

TL;DR

This paper introduces a novel explanation method for deep neural networks predicting psychological attributes from face images, revealing important features and assessing model transferability, especially with small datasets.

Contribution

It proposes a new heatmap-based explanation technique for neural networks and demonstrates its effectiveness in analyzing models trained on limited face datasets for psychological attribute prediction.

Findings

Heatmaps highlight key facial regions influencing predictions

Multiclass models capture more diverse features than binary models

Explanation method provides insights into model transferability and feature importance

Abstract

This paper focuses on the problem of explaining predictions of psychological attributes such as attractiveness, happiness, confidence and intelligence from face photographs using deep neural networks. Since psychological attribute datasets typically suffer from small sample sizes, we apply transfer learning with two base models to avoid overfitting. These models were trained on an age and gender prediction task, respectively. Using a novel explanation method we extract heatmaps that highlight the parts of the image most responsible for the prediction. We further observe that the explanation method provides important insights into the nature of features of the base model, which allow one to assess the aptitude of the base model for a given transfer learning task. Finally, we observe that the multiclass model is more feature rich than its binary counterpart. The experimental evaluation is performed on the 2222 images from the 10k US faces dataset containing psychological attribute labels as well as on a subset of KDEF images.