On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study

TL;DR

This paper investigates the challenges and effects of training deep learning models for facial expression recognition using limited and combined datasets, highlighting the importance of data quality, augmentation, and transfer learning.

Contribution

It provides an extensive analysis of how dataset merging, initialization, and synthetic data impact model stability and performance in limited data scenarios for facial expression recognition.

Findings

Transfer learning and synthetic data improve stability but increase variance.

More detailed data leads to more stable performance on new scenarios.

Merging heterogeneous datasets requires careful augmentation and initialization.

Abstract

Deep learning models need large amounts of data for training. In video recognition and classification, significant advances were achieved with the introduction of new large databases. However, the creation of large-databases for training is infeasible in several scenarios. Thus, existing or small collected databases are typically joined and amplified to train these models. Nevertheless, training neural networks on limited data is not straightforward and comes with a set of problems. In this paper, we explore the effects of stacking databases, model initialization, and data amplification techniques when training with limited data on deep learning models' performance. We focused on the problem of Facial Expression Recognition from videos. We performed an extensive study with four databases at a different complexity and nine deep-learning architectures for video classification. We found that (i) complex training sets translate better to more stable test sets when trained with transfer learning and synthetically generated data, but their performance yields a high variance; (ii) training with more detailed data translates to more stable performance on novel scenarios (albeit with lower performance); (iii) merging heterogeneous data is not a straightforward improvement, as the type of augmentation and initialization is crucial; (iv) classical data augmentation cannot fill the holes created by joining largely separated datasets; and (v) inductive biases help to bridge the gap when paired with synthetic data, but this data is not enough when working with standard initialization techniques.