evTransFER: A Transfer Learning Framework for Event-based Facial Expression Recognition

TL;DR

evTransFER introduces a transfer learning framework utilizing event-based cameras for facial expression recognition, achieving high accuracy by encoding facial dynamics and leveraging a novel event representation, outperforming existing methods.

Contribution

The paper presents a novel transfer learning approach with a specialized feature extractor and event-based representation for improved facial expression recognition using event cameras.

Findings

Achieved 93.6% accuracy on e-CK+ dataset.

Reached 76.7% accuracy on NEFER dataset.

Outperformed state-of-the-art methods and models trained from scratch.

Abstract

Event-based cameras are bio-inspired sensors that asynchronously capture pixel intensity changes with microsecond latency, high temporal resolution, and high dynamic range, providing information on the spatiotemporal dynamics of a scene. We propose evTransFER, a transfer learning-based framework for facial expression recognition using event-based cameras. The main contribution is a feature extractor designed to encode facial spatiotemporal dynamics, built by training an adversarial generative method on facial reconstruction and transferring the encoder weights to the facial expression recognition system. We demonstrate that the proposed transfer learning method improves facial expression recognition compared to training a network from scratch. We propose an architecture that incorporates an LSTM to capture longer-term facial expression dynamics and introduces a new event-based representation called TIE. We evaluated the framework using both the synthetic event-based facial expression database e-CK+ and the real neuromorphic dataset NEFER. On e-CK+, evTransFER achieved a recognition rate of 93.6\%, surpassing state-of-the-art methods. For NEFER, which comprises event sequence with real sensor noise and sparse activity, the proposed transfer learning strategy achieved an accuracy of up to 76.7\%. In both datasets, the outcomes surpassed current methodologies and exceeded results when compared with models trained from scratch.