Cyclic Test-Time Adaptation on Monocular Video for 3D Human Mesh Reconstruction

TL;DR

CycleAdapt introduces a cyclic test-time adaptation method for monocular video that improves 3D human mesh reconstruction by reducing reliance on noisy 2D evidence and progressively refining 3D supervision.

Contribution

It proposes a novel cyclic adaptation framework that jointly refines a human mesh reconstruction network and a motion denoising network using generated 3D supervision from test videos.

Findings

Achieves state-of-the-art performance on 3D human mesh reconstruction.

Effectively handles noisy or missing 2D evidence during test time.

Demonstrates robustness across diverse test videos.

Abstract

Despite recent advances in 3D human mesh reconstruction, domain gap between training and test data is still a major challenge. Several prior works tackle the domain gap problem via test-time adaptation that fine-tunes a network relying on 2D evidence (e.g., 2D human keypoints) from test images. However, the high reliance on 2D evidence during adaptation causes two major issues. First, 2D evidence induces depth ambiguity, preventing the learning of accurate 3D human geometry. Second, 2D evidence is noisy or partially non-existent during test time, and such imperfect 2D evidence leads to erroneous adaptation. To overcome the above issues, we introduce CycleAdapt, which cyclically adapts two networks: a human mesh reconstruction network (HMRNet) and a human motion denoising network (MDNet), given a test video. In our framework, to alleviate high reliance on 2D evidence, we fully supervise HMRNet with generated 3D supervision targets by MDNet. Our cyclic adaptation scheme progressively elaborates the 3D supervision targets, which compensate for imperfect 2D evidence. As a result, our CycleAdapt achieves state-of-the-art performance compared to previous test-time adaptation methods. The codes are available at https://github.com/hygenie1228/CycleAdapt_RELEASE.