DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields

TL;DR

DeformToon3D introduces a novel framework for 3D toonification that preserves the original GAN latent space, enabling flexible style transfer and editing of stylized 3D faces without extensive retraining.

Contribution

The paper proposes DeformToon3D, a hierarchical 3D GAN-based toonification method with a StyleField for geometry deformation and adaptive texture stylization, improving flexibility and efficiency.

Findings

Enables flexible style degree control.

Allows shape-texture-specific style swapping.

Achieves efficient training without real-world 2D-3D pairs.

Abstract

In this paper, we address the challenging problem of 3D toonification, which involves transferring the style of an artistic domain onto a target 3D face with stylized geometry and texture. Although fine-tuning a pre-trained 3D GAN on the artistic domain can produce reasonable performance, this strategy has limitations in the 3D domain. In particular, fine-tuning can deteriorate the original GAN latent space, which affects subsequent semantic editing, and requires independent optimization and storage for each new style, limiting flexibility and efficient deployment. To overcome these challenges, we propose DeformToon3D, an effective toonification framework tailored for hierarchical 3D GAN. Our approach decomposes 3D toonification into subproblems of geometry and texture stylization to better preserve the original latent space. Specifically, we devise a novel StyleField that predicts conditional 3D deformation to align a real-space NeRF to the style space for geometry stylization. Thanks to the StyleField formulation, which already handles geometry stylization well, texture stylization can be achieved conveniently via adaptive style mixing that injects information of the artistic domain into the decoder of the pre-trained 3D GAN. Due to the unique design, our method enables flexible style degree control and shape-texture-specific style swap. Furthermore, we achieve efficient training without any real-world 2D-3D training pairs but proxy samples synthesized from off-the-shelf 2D toonification models.