Modeling Caricature Expressions by 3D Blendshape and Dynamic Texture

TL;DR

This paper introduces a novel approach to deform caricatures by extending 3D Morphable Models with shape and texture modeling, enabling expressive and identity-preserving caricature animation.

Contribution

It extends traditional 3DMM to caricatures by combining shape optimization and cGAN-based texture generation for flexible, expressive caricature deformation.

Findings

Effective shape and texture modeling for caricatures

Caricatures can be deformed into arbitrary expressions

Visual results show high fidelity in shape and color

Abstract

The problem of deforming an artist-drawn caricature according to a given normal face expression is of interest in applications such as social media, animation and entertainment. This paper presents a solution to the problem, with an emphasis on enhancing the ability to create desired expressions and meanwhile preserve the identity exaggeration style of the caricature, which imposes challenges due to the complicated nature of caricatures. The key of our solution is a novel method to model caricature expression, which extends traditional 3DMM representation to caricature domain. The method consists of shape modelling and texture generation for caricatures. Geometric optimization is developed to create identity-preserving blendshapes for reconstructing accurate and stable geometric shape, and a conditional generative adversarial network (cGAN) is designed for generating dynamic textures under target expressions. The combination of both shape and texture components makes the non-trivial expressions of a caricature be effectively defined by the extension of the popular 3DMM representation and a caricature can thus be flexibly deformed into arbitrary expressions with good results visually in both shape and color spaces. The experiments demonstrate the effectiveness of the proposed method.