Multi-modal Face Pose Estimation with Multi-task Manifold Deep Learning

TL;DR

This paper introduces a novel multi-modal, multi-task deep learning framework called M^2DL for face pose estimation, effectively handling complex backgrounds and various orientations by leveraging manifold regularization and multi-modal data.

Contribution

It proposes a new deep learning framework that combines multi-modal data and multi-task learning with manifold regularization for improved face pose estimation.

Findings

Outperforms existing methods on DPOSE, HPID, and BKHPD datasets

Effectively handles complex backgrounds and diverse face orientations

Demonstrates superior accuracy and robustness in benchmark tests

Abstract

Human face pose estimation aims at estimating the gazing direction or head postures with 2D images. It gives some very important information such as communicative gestures, saliency detection and so on, which attracts plenty of attention recently. However, it is challenging because of complex background, various orientations and face appearance visibility. Therefore, a descriptive representation of face images and mapping it to poses are critical. In this paper, we make use of multi-modal data and propose a novel face pose estimation method that uses a novel deep learning framework named Multi-task Manifold Deep Learning $M^2DL$. It is based on feature extraction with improved deep neural networks and multi-modal mapping relationship with multi-task learning. In the proposed deep learning based framework, Manifold Regularized Convolutional Layers (MRCL) improve traditional convolutional layers by learning the relationship among outputs of neurons. Besides, in the proposed mapping relationship learning method, different modals of face representations are naturally combined to learn the mapping function from face images to poses. In this way, the computed mapping model with multiple tasks is improved. Experimental results on three challenging benchmark datasets DPOSE, HPID and BKHPD demonstrate the outstanding performance of $M^2DL$.