Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction

TL;DR

This paper presents a weakly-supervised framework that combines non-parametric mesh fitting with the MANO model to produce accurate and plausible 3D hand shapes, especially in complex interaction scenarios.

Contribution

It introduces an end-to-end joint model that overcomes the accuracy- plausibility tradeoff in 3D hand shape reconstruction.

Findings

Produces well-aligned, high-quality 3D meshes

Effective in challenging two-hand and hand-object interactions

Balances accuracy with shape plausibility

Abstract

Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. Conversely, parametric models like MANO ensure plausible hand shapes but are not as accurate as the non-parametric methods. In this work, we introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion. Our joint model overcomes the tradeoff in accuracy and plausibility to yield well-aligned and high-quality 3D meshes, especially in challenging two-hand and hand-object interaction scenarios.