Verifying Deep Learning-based Decisions for Facial Expression Recognition

TL;DR

This paper presents a verification pipeline for facial expression recognition neural networks, combining classification, layer-wise relevance propagation, and region-based explanation quantification to assess model focus and reliability.

Contribution

It introduces a novel verification method that evaluates whether neural networks focus on relevant facial regions, enhancing trustworthiness in high-risk applications.

Findings

Neural network achieves state-of-the-art accuracy.

Visual explanations often omit relevant facial regions.

The method helps identify biases in model focus.

Abstract

Neural networks with high performance can still be biased towards non-relevant features. However, reliability and robustness is especially important for high-risk fields such as clinical pain treatment. We therefore propose a verification pipeline, which consists of three steps. First, we classify facial expressions with a neural network. Next, we apply layer-wise relevance propagation to create pixel-based explanations. Finally, we quantify these visual explanations based on a bounding-box method with respect to facial regions. Although our results show that the neural network achieves state-of-the-art results, the evaluation of the visual explanations reveals that relevant facial regions may not be considered.