Facial Expression Recognition using Visual Saliency and Deep Learning

TL;DR

This paper presents a deep learning approach for facial expression recognition that leverages visual saliency maps and fine-tunes a CNN on multiple datasets, achieving high accuracy and demonstrating generalization across datasets.

Contribution

It introduces a novel combination of visual saliency maps with CNNs for improved facial expression recognition and evaluates cross-dataset generalization.

Findings

Achieved 74.79% and 95.71% accuracy on CFEE and RaFD datasets.

Saliency-enhanced CNN improved generalization to 65.39% accuracy.

Observed human-like confusion patterns between facial expressions.

Abstract

We have developed a convolutional neural network for the purpose of recognizing facial expressions in human beings. We have fine-tuned the existing convolutional neural network model trained on the visual recognition dataset used in the ILSVRC2012 to two widely used facial expression datasets - CFEE and RaFD, which when trained and tested independently yielded test accuracies of 74.79% and 95.71%, respectively. Generalization of results was evident by training on one dataset and testing on the other. Further, the image product of the cropped faces and their visual saliency maps were computed using Deep Multi-Layer Network for saliency prediction and were fed to the facial expression recognition CNN. In the most generalized experiment, we observed the top-1 accuracy in the test set to be 65.39%. General confusion trends between different facial expressions as exhibited by humans were also observed.