Local Learning with Deep and Handcrafted Features for Facial Expression Recognition

TL;DR

This paper introduces a novel facial expression recognition method combining deep CNN features and handcrafted BOVW features within a local learning framework, achieving state-of-the-art accuracy on multiple datasets.

Contribution

It is the first to integrate local learning with deep features for facial expression recognition, improving accuracy over existing methods.

Findings

Achieved top accuracy of 75.42% on FER 2013

Surpassed state-of-the-art by more than 1% on all datasets

Demonstrated effectiveness of combining deep and handcrafted features

Abstract

We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pre-trained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-the-art methods by more than 1% on all data sets.