Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a Single Image

TL;DR

This paper introduces a low-parameter deep learning framework that uses NMDS to recover 3D face shapes from 2D landmarks in a single image, effectively handling occlusions and complex projections.

Contribution

It is the first to apply NMDS within a deep learning framework for 3D shape recovery from 2D landmarks, incorporating an autoencoder for occlusion removal.

Findings

Achieves comparable accuracy to state-of-the-art methods

Operates efficiently with fewer training parameters

Performs well on synthetic and real-world datasets

Abstract

In this paper, a low parameter deep learning framework utilizing the Non-metric Multi-Dimensional scaling (NMDS) method, is proposed to recover the 3D shape of 2D landmarks on a human face, in a single input image. Hence, NMDS approach is used for the first time to establish a mapping from a 2D landmark space to the corresponding 3D shape space. A deep neural network learns the pairwise dissimilarity among 2D landmarks, used by NMDS approach, whose objective is to learn the pairwise 3D Euclidean distance of the corresponding 2D landmarks on the input image. This scheme results in a symmetric dissimilarity matrix, with the rank larger than 2, leading the NMDS approach toward appropriately recovering the 3D shape of corresponding 2D landmarks. In the case of posed images and complex image formation processes like perspective projection which causes occlusion in the input image, we consider an autoencoder component in the proposed framework, as an occlusion removal part, which turns different input views of the human face into a profile view. The results of a performance evaluation using different synthetic and real-world human face datasets, including Besel Face Model (BFM), CelebA, CoMA - FLAME, and CASIA-3D, indicates the comparable performance of the proposed framework, despite its small number of training parameters, with the related state-of-the-art and powerful 3D reconstruction methods from the literature, in terms of efficiency and accuracy.