Generating Multiple 4D Expression Transitions by Learning Face Landmark Trajectories

TL;DR

This paper introduces a novel approach for generating complex 4D facial expression transitions by modeling landmark trajectories with a GAN and deforming meshes, enabling realistic and identity-independent expression synthesis.

Contribution

The work presents a new model that learns facial expression transitions through landmark trajectories and synthesizes long, composed 4D expressions, advancing beyond simple peak-to-neutral animations.

Findings

Significant improvement over previous methods in expression transition quality.

Model generalizes well to unseen data and diverse expressions.

Effective generation of long, complex 4D facial expression sequences.

Abstract

In this work, we address the problem of 4D facial expressions generation. This is usually addressed by animating a neutral 3D face to reach an expression peak, and then get back to the neutral state. In the real world though, people show more complex expressions, and switch from one expression to another. We thus propose a new model that generates transitions between different expressions, and synthesizes long and composed 4D expressions. This involves three sub-problems: (i) modeling the temporal dynamics of expressions, (ii) learning transitions between them, and (iii) deforming a generic mesh. We propose to encode the temporal evolution of expressions using the motion of a set of 3D landmarks, that we learn to generate by training a manifold-valued GAN (Motion3DGAN). To allow the generation of composed expressions, this model accepts two labels encoding the starting and the ending expressions. The final sequence of meshes is generated by a Sparse2Dense mesh Decoder (S2D-Dec) that maps the landmark displacements to a dense, per-vertex displacement of a known mesh topology. By explicitly working with motion trajectories, the model is totally independent from the identity. Extensive experiments on five public datasets show that our proposed approach brings significant improvements with respect to previous solutions, while retaining good generalization to unseen data.