Facial Expression Recognition using Convolutional Neural Networks: State of the Art

TL;DR

This paper reviews the current state of CNN-based facial expression recognition, analyzes algorithmic differences, identifies bottlenecks, and demonstrates that modern ensemble CNNs significantly improve accuracy.

Contribution

It provides a comprehensive review of CNN architectures in FER, highlights performance bottlenecks, and shows that ensemble methods with modern CNNs enhance accuracy without extra data.

Findings

Ensemble of modern CNNs achieves 75.2% accuracy on FER2013.

Basic CNN architectures limit performance in FER.

Addressing architectural bottlenecks improves recognition accuracy.

Abstract

The ability to recognize facial expressions automatically enables novel applications in human-computer interaction and other areas. Consequently, there has been active research in this field, with several recent works utilizing Convolutional Neural Networks (CNNs) for feature extraction and inference. These works differ significantly in terms of CNN architectures and other factors. Based on the reported results alone, the performance impact of these factors is unclear. In this paper, we review the state of the art in image-based facial expression recognition using CNNs and highlight algorithmic differences and their performance impact. On this basis, we identify existing bottlenecks and consequently directions for advancing this research field. Furthermore, we demonstrate that overcoming one of these bottlenecks - the comparatively basic architectures of the CNNs utilized in this field - leads to a substantial performance increase. By forming an ensemble of modern deep CNNs, we obtain a FER2013 test accuracy of 75.2%, outperforming previous works without requiring auxiliary training data or face registration.