Micro-expression recognition based on depth map to point cloud

TL;DR

This paper introduces a novel micro-expression recognition method using depth maps transformed into point clouds, leveraging PointNet++ to outperform existing 2D image-based approaches and demonstrate the effectiveness of depth information.

Contribution

The paper proposes a new micro-expression recognition approach based on depth map to point cloud transformation, utilizing PointNet++ for improved accuracy over traditional methods.

Findings

Significantly outperforms existing deep learning methods on CAS(ME)3 dataset.

Depth maps effectively capture motion information for micro-expression recognition.

Point cloud representation enhances robustness against environmental factors.

Abstract

Micro-expressions are nonverbal facial expressions that reveal the covert emotions of individuals, making the micro-expression recognition task receive widespread attention. However, the micro-expression recognition task is challenging due to the subtle facial motion and brevity in duration. Many 2D image-based methods have been developed in recent years to recognize MEs effectively, but, these approaches are restricted by facial texture information and are susceptible to environmental factors, such as lighting. Conversely, depth information can effectively represent motion information related to facial structure changes and is not affected by lighting. Motion information derived from facial structures can describe motion features that pixel textures cannot delineate. We proposed a network for micro-expression recognition based on facial depth information, and our experiments have demonstrated the crucial role of depth maps in the micro-expression recognition task. Initially, we transform the depth map into a point cloud and obtain the motion information for each point by aligning the initiating frame with the apex frame and performing a differential operation. Subsequently, we adjusted all point cloud motion feature input dimensions and used them as inputs for multiple point cloud networks to assess the efficacy of this representation. PointNet++ was chosen as the ultimate outcome for micro-expression recognition due to its superior performance. Our experiments show that our proposed method significantly outperforms the existing deep learning methods, including the baseline, on the $CAS(ME)^3$ dataset, which includes depth information.