Discriminative-Generative Synergy for Occlusion Robust 3D Human Mesh Recovery

TL;DR

This paper introduces a brain-inspired framework combining vision transformers and diffusion models to improve 3D human mesh recovery under occlusion, achieving better accuracy and robustness.

Contribution

It proposes a novel synergistic approach that integrates discriminative and generative models with feature alignment and multi-level fusion for occlusion-robust 3D human mesh recovery.

Findings

Outperforms existing methods on standard benchmarks.

Demonstrates strong robustness in real-world occlusion scenarios.

Achieves higher accuracy in estimating 3D human meshes.

Abstract

3D human mesh recovery from monocular RGB images aims to estimate anatomically plausible 3D human models for downstream applications, but remains challenging under partial or severe occlusions. Regression-based methods are efficient yet often produce implausible or inaccurate results in unconstrained scenarios, while diffusion-based methods provide strong generative priors for occluded regions but may weaken fidelity to rare poses due to over-reliance on generation. To address these limitations, we propose a brain-inspired synergistic framework that integrates the discriminative power of vision transformers with the generative capability of conditional diffusion models. Specifically, the ViT-based pathway extracts deterministic visual cues from visible regions, while the diffusion-based pathway synthesizes structurally coherent human body representations. To effectively bridge the two pathways, we design a diverse-consistent feature learning module to align discriminative features with generative priors, and a cross-attention multi-level fusion mechanism to enable bidirectional interaction across semantic levels. Experiments on standard benchmarks demonstrate that our method achieves superior performance on key metrics and shows strong robustness in complex real-world scenarios.