Learning and Fusing Multimodal Features from and for Multi-task Facial Computing

TL;DR

This paper introduces a deep learning-based multimodal feature fusion method for facial computing tasks like recognition, gender, race, and age detection, demonstrating improved accuracy through cross-task feature sharing.

Contribution

It presents a novel multi-task feature fusion approach where features trained for one facial attribute enhance other related tasks, outperforming single-task classifiers.

Findings

Fusion improves ID recognition accuracy by 7.2%.

Fusion improves age recognition accuracy by 20.1%.

Fusion improves race and gender recognition accuracy by over 21%.

Abstract

We propose a deep learning-based feature fusion approach for facial computing including face recognition as well as gender, race and age detection. Instead of training a single classifier on face images to classify them based on the features of the person whose face appears in the image, we first train four different classifiers for classifying face images based on race, age, gender and identification (ID). Multi-task features are then extracted from the trained models and cross-task-feature training is conducted which shows the value of fusing multimodal features extracted from multi-tasks. We have found that features trained for one task can be used for other related tasks. More interestingly, the features trained for a task with more classes (e.g. ID) and then used in another task with fewer classes (e.g. race) outperforms the features trained for the other task itself. The final feature fusion is performed by combining the four types of features extracted from the images by the four classifiers. The feature fusion approach improves the classifications accuracy by a 7.2%, 20.1%, 22.2%, 21.8% margin, respectively, for ID, age, race and gender recognition, over the results of single classifiers trained only on their individual features. The proposed method can be applied to applications in which different types of data or features can be extracted.