FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery

TL;DR

FactorizedHMR is a two-stage hybrid framework that improves video human mesh recovery by separately handling well-constrained torso regions and uncertain limb articulations, especially under occlusion.

Contribution

It introduces a novel two-stage approach with deterministic and probabilistic modules, along with a synthetic data pipeline for diverse supervision, enhancing recovery accuracy under challenging conditions.

Findings

Outperforms strong baselines in occlusion-heavy scenarios

Provides clearer gains in world-space metrics

Maintains competitive performance across benchmarks

Abstract

Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.