GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates

TL;DR

GenFusion introduces a feed-forward method for human performance capture from monocular RGB streams, using a progressively updated canonical space to improve novel view rendering and handle unseen regions.

Contribution

It proposes a novel canonical space update mechanism that accumulates appearance information over time and employs probabilistic regression for better reconstruction.

Findings

Produces sharper reconstructions than deterministic methods.

Enables plausible synthesis in unobserved regions.

Effective on both in-domain and out-of-distribution datasets.

Abstract

We present a feed-forward human performance capture method that renders novel views of a performer from a monocular RGB stream. A key challenge in this setting is the lack of sufficient observations, especially for unseen regions. Assuming the subject moves continuously over time, we take advantage of the fact that more body parts become observable by maintaining a canonical space that is progressively updated with each incoming frame. This canonical space accumulates appearance information over time and serves as a context bank when direct observations are missing in the current live frame. To effectively utilize this context while respecting the deformation of the live state, we formulate the rendering process as probabilistic regression. This resolves conflicts between past and current observations, producing sharper reconstructions than deterministic regression approaches. Furthermore, it enables plausible synthesis even in regions with no prior observations. Experiments on in-domain (4D-Dress) and out-of-distribution (MVHumanNet) datasets demonstrate the effectiveness of our approach.