TextToon: Real-Time Text Toonify Head Avatar from Single Video

TL;DR

TextToon is a novel method that creates high-quality, stylized, toonified 3D avatars from a single video and text instructions, capable of real-time animation on GPU and mobile devices.

Contribution

It introduces a new approach using conditional embedding Tri-planes and adaptive neural networks for stylized 3D avatar generation from minimal input.

Findings

Achieves real-time avatar rendering at 48 FPS on GPU

Outperforms existing methods in avatar quality and stylization

Supports arbitrary identity-driven animation from minimal input

Abstract

We propose TextToon, a method to generate a drivable toonified avatar. Given a short monocular video sequence and a written instruction about the avatar style, our model can generate a high-fidelity toonified avatar that can be driven in real-time by another video with arbitrary identities. Existing related works heavily rely on multi-view modeling to recover geometry via texture embeddings, presented in a static manner, leading to control limitations. The multi-view video input also makes it difficult to deploy these models in real-world applications. To address these issues, we adopt a conditional embedding Tri-plane to learn realistic and stylized facial representations in a Gaussian deformation field. Additionally, we expand the stylization capabilities of 3D Gaussian Splatting by introducing an adaptive pixel-translation neural network and leveraging patch-aware contrastive learning to achieve high-quality images. To push our work into consumer applications, we develop a real-time system that can operate at 48 FPS on a GPU machine and 15-18 FPS on a mobile machine. Extensive experiments demonstrate the efficacy of our approach in generating textual avatars over existing methods in terms of quality and real-time animation. Please refer to our project page for more details: https://songluchuan.github.io/TextToon/.