ORTexME: Occlusion-Robust Human Shape and Pose via Temporal Average Texture and Mesh Encoding

TL;DR

ORTexME introduces a novel occlusion-robust temporal approach for 3D human shape and pose estimation from monocular videos, leveraging average texture learning and mesh guidance to improve accuracy under occlusion.

Contribution

The paper proposes ORTexME, a new method that enhances occlusion handling in human shape and pose estimation by integrating temporal information, average texture learning, and mesh-guided NeRF updates.

Findings

Achieves 1.8 P-MPJPE error reduction on 3DPW dataset.

Outperforms state-of-the-art rendering methods, reducing errors significantly.

Demonstrates robustness to occlusion and segmentation inaccuracies.

Abstract

In 3D human shape and pose estimation from a monocular video, models trained with limited labeled data cannot generalize well to videos with occlusion, which is common in the wild videos. The recent human neural rendering approaches focusing on novel view synthesis initialized by the off-the-shelf human shape and pose methods have the potential to correct the initial human shape. However, the existing methods have some drawbacks such as, erroneous in handling occlusion, sensitive to inaccurate human segmentation, and ineffective loss computation due to the non-regularized opacity field. To address these problems, we introduce ORTexME, an occlusion-robust temporal method that utilizes temporal information from the input video to better regularize the occluded body parts. While our ORTexME is based on NeRF, to determine the reliable regions for the NeRF ray sampling, we utilize our novel average texture learning approach to learn the average appearance of a person, and to infer a mask based on the average texture. In addition, to guide the opacity-field updates in NeRF to suppress blur and noise, we propose the use of human body mesh. The quantitative evaluation demonstrates that our method achieves significant improvement on the challenging multi-person 3DPW dataset, where our method achieves 1.8 P-MPJPE error reduction. The SOTA rendering-based methods fail and enlarge the error up to 5.6 on the same dataset.