LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis

TL;DR

LibreFace is an open-source toolkit that enables real-time and offline facial expression analysis using deep learning, achieving high accuracy and efficiency for applications in human-computer interaction.

Contribution

It introduces a comprehensive, open-source facial expression analysis toolkit with novel techniques for improved accuracy and speed over existing solutions.

Findings

Achieves a PCC of 0.63 on DISFA, 7% higher than OpenFace 2.0.

Runs twice as fast as OpenFace 2.0 with high accuracy.

Demonstrates competitive performance on multiple facial expression datasets.

Abstract

Facial expression analysis is an important tool for human-computer interaction. In this paper, we introduce LibreFace, an open-source toolkit for facial expression analysis. This open-source toolbox offers real-time and offline analysis of facial behavior through deep learning models, including facial action unit (AU) detection, AU intensity estimation, and facial expression recognition. To accomplish this, we employ several techniques, including the utilization of a large-scale pre-trained network, feature-wise knowledge distillation, and task-specific fine-tuning. These approaches are designed to effectively and accurately analyze facial expressions by leveraging visual information, thereby facilitating the implementation of real-time interactive applications. In terms of Action Unit (AU) intensity estimation, we achieve a Pearson Correlation Coefficient (PCC) of 0.63 on DISFA, which is 7% higher than the performance of OpenFace 2.0 while maintaining highly-efficient inference that runs two times faster than OpenFace 2.0. Despite being compact, our model also demonstrates competitive performance to state-of-the-art facial expression analysis methods on AffecNet, FFHQ, and RAF-DB. Our code will be released at https://github.com/ihp-lab/LibreFace