RealisticHands: A Hybrid Model for 3D Hand Reconstruction

TL;DR

This paper introduces a hybrid deep learning and differential rendering approach for 3D hand reconstruction from RGB images, enhancing robustness to occlusions and self-similarity, with applications in VR and realistic hand textures.

Contribution

A novel hybrid model combining neural networks and differential rendering for improved 3D hand reconstruction, including egocentric stereo extension for VR applications.

Findings

Achieves superior hand reconstruction accuracy.

Extends to egocentric stereo domain for VR robustness.

Enables realistic virtual hand textures.

Abstract

Estimating 3D hand meshes from RGB images robustly is a highly desirable task, made challenging due to the numerous degrees of freedom, and issues such as self similarity and occlusions. Previous methods generally either use parametric 3D hand models or follow a model-free approach. While the former can be considered more robust, e.g. to occlusions, they are less expressive. We propose a hybrid approach, utilizing a deep neural network and differential rendering based optimization to demonstrably achieve the best of both worlds. In addition, we explore Virtual Reality (VR) as an application. Most VR headsets are nowadays equipped with multiple cameras, which we can leverage by extending our method to the egocentric stereo domain. This extension proves to be more resilient to the above mentioned issues. Finally, as a use-case, we show that the improved image-model alignment can be used to acquire the user's hand texture, which leads to a more realistic virtual hand representation.