Feature Aggregation for Efficient Continual Learning of Complex Facial Expressions

TL;DR

This paper introduces a hybrid Bayesian approach combining deep features and facial Action Units for continual facial expression recognition, effectively reducing forgetting and improving accuracy in recognizing both basic and compound emotions.

Contribution

It presents a novel hybrid framework that integrates deep features and Action Units with Bayesian Gaussian Mixture Models for continual learning in facial expression recognition.

Findings

Improved recognition accuracy on CFEE dataset

Enhanced knowledge retention and reduced forgetting

Effective recognition of basic and compound expressions

Abstract

As artificial intelligence (AI) systems become increasingly embedded in our daily life, the ability to recognize and adapt to human emotions is essential for effective human-computer interaction. Facial expression recognition (FER) provides a primary channel for inferring affective states, but the dynamic and culturally nuanced nature of emotions requires models that can learn continuously without forgetting prior knowledge. In this work, we propose a hybrid framework for FER in a continual learning setting that mitigates catastrophic forgetting. Our approach integrates two complementary modalities: deep convolutional features and facial Action Units (AUs) derived from the Facial Action Coding System (FACS). The combined representation is modelled through Bayesian Gaussian Mixture Models (BGMMs), which provide a lightweight, probabilistic solution that avoids retraining while offering strong discriminative power. Using the Compound Facial Expression of Emotion (CFEE) dataset, we show that our model can first learn basic expressions and then progressively recognize compound expressions. Experiments demonstrate improved accuracy, stronger knowledge retention, and reduced forgetting. This framework contributes to the development of emotionally intelligent AI systems with applications in education, healthcare, and adaptive user interfaces.