RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models

TL;DR

RodinHD introduces a novel approach for high-fidelity 3D avatar generation from portraits, addressing detail preservation and generalization issues through innovative training strategies and hierarchical texture integration.

Contribution

The paper proposes a new data scheduling and regularization method to prevent catastrophic forgetting and enhances detail capture by hierarchical texture representation in 3D diffusion models.

Findings

Produces avatars with finer details than previous methods

Successfully generalizes to in-the-wild portrait inputs

Trained on 46K avatars with optimized noise schedule

Abstract

We present RodinHD, which can generate high-fidelity 3D avatars from a portrait image. Existing methods fail to capture intricate details such as hairstyles which we tackle in this paper. We first identify an overlooked problem of catastrophic forgetting that arises when fitting triplanes sequentially on many avatars, caused by the MLP decoder sharing scheme. To overcome this issue, we raise a novel data scheduling strategy and a weight consolidation regularization term, which improves the decoder's capability of rendering sharper details. Additionally, we optimize the guiding effect of the portrait image by computing a finer-grained hierarchical representation that captures rich 2D texture cues, and injecting them to the 3D diffusion model at multiple layers via cross-attention. When trained on 46K avatars with a noise schedule optimized for triplanes, the resulting model can generate 3D avatars with notably better details than previous methods and can generalize to in-the-wild portrait input.